2023-10-10T18:32:03.633 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-10T18:32:03.635 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370 --verbose --name yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} -- /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/orig.config.yaml 2023-10-10T18:32:03.683 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-10T18:32:03.775 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370 branch: wip-yuri3-testing-2023-10-10-0720-quincy description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: yweinste@redhat.com first_in_suite: false job_id: '7420370' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri3-testing-2023-10-10-0720-quincy ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - Replacing daemon mds sha1: 46d1f59b549730ba7f93d940a3d32b94c45a4cf7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 46d1f59b549730ba7f93d940a3d32b94c45a4cf7 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-10-10-0720-quincy sha1: 46d1f59b549730ba7f93d940a3d32b94c45a4cf7 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 2048 sha1: 46d1f59b549730ba7f93d940a3d32b94c45a4cf7 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri3-testing-2023-10-10-0720-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 46d1f59b549730ba7f93d940a3d32b94c45a4cf7 targets: smithi175.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcuCFH2aQPp6AqsxooAWa9L+2D7lkIlT3KYhmELk9TKiw8Yutr9DYrlPL+/hGDwU2wBsKtoqN/WB7RivMb1/j0= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKAmsrXlRHhVO40XzsNzCCvWbKIzgovODvyFDIx1r5u6lgIzDMu/D40byN68X9lvtywNnIV6lA/+vo2O0KVoaWw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-10_18:16:52 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-10T18:32:03.775 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa; will attempt to use it 2023-10-10T18:32:03.776 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa/tasks 2023-10-10T18:32:03.777 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-10T18:32:03.779 INFO:teuthology.task.internal:Checking packages... 2023-10-10T18:32:03.804 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '46d1f59b549730ba7f93d940a3d32b94c45a4cf7' 2023-10-10T18:32:03.804 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-10T18:32:03.805 INFO:teuthology.packaging:ref: None 2023-10-10T18:32:03.805 INFO:teuthology.packaging:tag: None 2023-10-10T18:32:03.805 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-10-10-0720-quincy 2023-10-10T18:32:03.805 INFO:teuthology.packaging:sha1: 46d1f59b549730ba7f93d940a3d32b94c45a4cf7 2023-10-10T18:32:03.806 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri3-testing-2023-10-10-0720-quincy 2023-10-10T18:32:03.977 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1374.g46d1f59b 2023-10-10T18:32:03.977 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-10T18:32:03.983 INFO:teuthology.task.internal:no buildpackages task found 2023-10-10T18:32:03.984 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-10T18:32:04.004 INFO:teuthology.task.internal:Saving configuration 2023-10-10T18:32:04.022 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-10T18:32:04.029 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-10T18:32:04.053 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 18:26:52.470722', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcuCFH2aQPp6AqsxooAWa9L+2D7lkIlT3KYhmELk9TKiw8Yutr9DYrlPL+/hGDwU2wBsKtoqN/WB7RivMb1/j0='} 2023-10-10T18:32:04.074 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 18:26:52.469006', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKAmsrXlRHhVO40XzsNzCCvWbKIzgovODvyFDIx1r5u6lgIzDMu/D40byN68X9lvtywNnIV6lA/+vo2O0KVoaWw='} 2023-10-10T18:32:04.075 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-10T18:32:04.081 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-10T18:32:04.081 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-10T18:32:04.082 INFO:teuthology.run_tasks:Running task console_log... 2023-10-10T18:32:04.155 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f54eefa4b80>, signals=[15]) 2023-10-10T18:32:04.156 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-10T18:32:04.162 INFO:teuthology.task.internal:Opening connections... 2023-10-10T18:32:04.162 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T18:32:04.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T18:32:04.276 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T18:32:04.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T18:32:04.349 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-10T18:32:04.358 DEBUG:teuthology.orchestra.run.smithi175:> uname -m 2023-10-10T18:32:04.375 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2023-10-10T18:32:04.376 DEBUG:teuthology.orchestra.run.smithi175:> cat /etc/os-release 2023-10-10T18:32:04.431 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="Red Hat Enterprise Linux" 2023-10-10T18:32:04.432 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="8.4 (Ootpa)" 2023-10-10T18:32:04.432 INFO:teuthology.orchestra.run.smithi175.stdout:ID="rhel" 2023-10-10T18:32:04.432 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="fedora" 2023-10-10T18:32:04.432 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="8.4" 2023-10-10T18:32:04.433 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el8" 2023-10-10T18:32:04.433 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-10T18:32:04.433 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31" 2023-10-10T18:32:04.433 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-10T18:32:04.433 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://www.redhat.com/" 2023-10-10T18:32:04.434 INFO:teuthology.orchestra.run.smithi175.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-10T18:32:04.434 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-10T18:32:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:32:04.434 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-10T18:32:04.434 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-10T18:32:04.435 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-10T18:32:04.435 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-10T18:32:04.436 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2023-10-10T18:32:04.461 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2023-10-10T18:32:04.479 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2023-10-10T18:32:04.479 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2023-10-10T18:32:04.538 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="Red Hat Enterprise Linux" 2023-10-10T18:32:04.539 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="8.4 (Ootpa)" 2023-10-10T18:32:04.539 INFO:teuthology.orchestra.run.smithi191.stdout:ID="rhel" 2023-10-10T18:32:04.539 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="fedora" 2023-10-10T18:32:04.539 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="8.4" 2023-10-10T18:32:04.540 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el8" 2023-10-10T18:32:04.540 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-10T18:32:04.540 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2023-10-10T18:32:04.540 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-10T18:32:04.540 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://www.redhat.com/" 2023-10-10T18:32:04.541 INFO:teuthology.orchestra.run.smithi191.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-10T18:32:04.541 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-10T18:32:04.541 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:32:04.541 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-10T18:32:04.541 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-10T18:32:04.542 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-10T18:32:04.542 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-10T18:32:04.542 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2023-10-10T18:32:04.569 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-10T18:32:04.581 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-10T18:32:04.587 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-10T18:32:04.588 DEBUG:teuthology.orchestra.run.smithi175:> test '!' -e /home/ubuntu/cephtest 2023-10-10T18:32:04.590 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2023-10-10T18:32:04.605 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-10T18:32:04.611 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-10T18:32:04.612 DEBUG:teuthology.orchestra.run.smithi175:> test -z $(ls -A /var/lib/ceph) 2023-10-10T18:32:04.648 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2023-10-10T18:32:04.661 INFO:teuthology.orchestra.run.smithi175.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-10T18:32:04.664 INFO:teuthology.orchestra.run.smithi191.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-10T18:32:04.665 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-10T18:32:04.721 INFO:teuthology.run_tasks:Running task kernel... 2023-10-10T18:32:04.736 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-10-10T18:32:04.736 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-10-10T18:32:04.737 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-10-10T18:32:04.737 DEBUG:teuthology.orchestra.run.smithi175:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-10T18:32:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:32:04.754 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2023-10-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi175.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-10T18:32:04.810 INFO:teuthology.task.kernel:Running kernel on smithi175: 4.18.0-305.25.1.el8_4.x86_64 2023-10-10T18:32:04.810 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum install -y kernel 2023-10-10T18:32:06.243 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2023-10-10T18:32:06.244 INFO:teuthology.orchestra.run.smithi175.stdout:Unable to read consumer identity 2023-10-10T18:32:07.694 INFO:teuthology.orchestra.run.smithi175.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-10-10T18:32:08.233 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 32 MB/s | 16 MB 00:00 2023-10-10T18:32:11.537 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 1.7 MB/s | 24 kB 00:00 2023-10-10T18:32:12.282 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-10T18:32:12.282 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-10T18:32:12.282 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-10T18:32:12.443 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-10-10T18:32:12.444 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-10-10T18:32:12.444 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-10-10T18:32:12.553 DEBUG:teuthology.orchestra.run.smithi175:> echo no | sudo yum reinstall kernel || true 2023-10-10T18:32:12.898 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2023-10-10T18:32:12.899 INFO:teuthology.orchestra.run.smithi175.stdout:Unable to read consumer identity 2023-10-10T18:32:13.192 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:02 ago on Tue 10 Oct 2023 06:32:11 PM UTC. 2023-10-10T18:32:13.323 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-10T18:32:13.324 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-10T18:32:13.324 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-10T18:32:13.354 INFO:teuthology.orchestra.run.smithi175.stderr:Error: No packages marked for reinstall. 2023-10-10T18:32:13.404 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum reinstall -y kernel || true 2023-10-10T18:32:13.747 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2023-10-10T18:32:13.748 INFO:teuthology.orchestra.run.smithi175.stdout:Unable to read consumer identity 2023-10-10T18:32:14.029 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:03 ago on Tue 10 Oct 2023 06:32:11 PM UTC. 2023-10-10T18:32:14.164 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-10T18:32:14.164 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-10T18:32:14.165 INFO:teuthology.orchestra.run.smithi175.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-10T18:32:14.177 INFO:teuthology.orchestra.run.smithi175.stderr:Error: No packages marked for reinstall. 2023-10-10T18:32:14.227 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2023-10-10T18:32:14.352 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-10T18:32:14.353 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-10T18:32:14.353 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-10T18:32:14.353 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-10T18:32:14.353 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-10T18:32:14.354 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-10T18:32:14.354 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi175.front.sepia.ceph.com, path=None, version=distro) 2023-10-10T18:32:14.354 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2023-10-10T18:32:14.478 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-10T18:32:14.479 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-10T18:32:14.479 DEBUG:teuthology.orchestra.run.smithi175:> sudo rpm -qi grub2-tools 2023-10-10T18:32:14.506 INFO:teuthology.orchestra.run.smithi175.stdout:Name : grub2-tools 2023-10-10T18:32:14.506 INFO:teuthology.orchestra.run.smithi175.stdout:Epoch : 1 2023-10-10T18:32:14.507 INFO:teuthology.orchestra.run.smithi175.stdout:Version : 2.02 2023-10-10T18:32:14.507 INFO:teuthology.orchestra.run.smithi175.stdout:Release : 99.el8_4.1 2023-10-10T18:32:14.507 INFO:teuthology.orchestra.run.smithi175.stdout:Architecture: x86_64 2023-10-10T18:32:14.507 INFO:teuthology.orchestra.run.smithi175.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-10T18:32:14.507 INFO:teuthology.orchestra.run.smithi175.stdout:Group : System Environment/Base 2023-10-10T18:32:14.508 INFO:teuthology.orchestra.run.smithi175.stdout:Size : 9264631 2023-10-10T18:32:14.508 INFO:teuthology.orchestra.run.smithi175.stdout:License : GPLv3+ 2023-10-10T18:32:14.508 INFO:teuthology.orchestra.run.smithi175.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-10T18:32:14.508 INFO:teuthology.orchestra.run.smithi175.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-10T18:32:14.508 INFO:teuthology.orchestra.run.smithi175.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-10T18:32:14.509 INFO:teuthology.orchestra.run.smithi175.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-10T18:32:14.509 INFO:teuthology.orchestra.run.smithi175.stdout:Relocations : (not relocatable) 2023-10-10T18:32:14.509 INFO:teuthology.orchestra.run.smithi175.stdout:Packager : Red Hat, Inc. 2023-10-10T18:32:14.509 INFO:teuthology.orchestra.run.smithi175.stdout:Vendor : Red Hat, Inc. 2023-10-10T18:32:14.509 INFO:teuthology.orchestra.run.smithi175.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-10T18:32:14.510 INFO:teuthology.orchestra.run.smithi175.stdout:Summary : Support tools for GRUB. 2023-10-10T18:32:14.510 INFO:teuthology.orchestra.run.smithi175.stdout:Description : 2023-10-10T18:32:14.510 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:32:14.510 INFO:teuthology.orchestra.run.smithi175.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-10T18:32:14.510 INFO:teuthology.orchestra.run.smithi175.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-10T18:32:14.511 INFO:teuthology.orchestra.run.smithi175.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-10T18:32:14.511 INFO:teuthology.orchestra.run.smithi175.stdout:hardware devices. 2023-10-10T18:32:14.511 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:32:14.511 INFO:teuthology.orchestra.run.smithi175.stdout:This subpackage provides tools for support of all platforms. 2023-10-10T18:32:14.512 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-10T18:32:14.512 INFO:teuthology.task.kernel:Updating grub on smithi175 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-10T18:32:14.513 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-10T18:32:15.898 INFO:teuthology.orchestra.run.smithi175.stderr:Generating grub configuration file ... 2023-10-10T18:32:17.946 INFO:teuthology.orchestra.run.smithi175.stderr:done 2023-10-10T18:32:17.948 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2023-10-10T18:32:17.963 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.eV1W2u7ejx 2023-10-10T18:32:17.964 DEBUG:teuthology.orchestra.run.smithi175:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eV1W2u7ejx 2023-10-10T18:32:18.033 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.eV1W2u7ejx 2023-10-10T18:32:18.188 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.eV1W2u7ejx is 6KB 2023-10-10T18:32:18.237 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.eV1W2u7ejx 2023-10-10T18:32:18.251 DEBUG:teuthology.orchestra.run.smithi175:> sudo /bin/ls /boot/loader/entries || true 2023-10-10T18:32:18.332 INFO:teuthology.orchestra.run.smithi175.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-10T18:32:18.332 INFO:teuthology.orchestra.run.smithi175.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-10T18:32:18.333 INFO:teuthology.orchestra.run.smithi175.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-10T18:32:18.333 INFO:teuthology.orchestra.run.smithi175.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-10T18:32:18.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-10T18:32:18.519 DEBUG:teuthology.orchestra.run.smithi175:> sudo shutdown -r now 2023-10-10T18:32:48.522 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-10T18:32:48.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-10-10T18:32:48.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T18:33:06.938 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.175 2023-10-10T18:33:15.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-10-10T18:33:15.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T18:33:19.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.175 2023-10-10T18:33:31.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-10-10T18:33:31.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T18:33:49.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.175 2023-10-10T18:34:04.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-10-10T18:34:04.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T18:34:04.790 DEBUG:teuthology.orchestra.run.smithi175:> true 2023-10-10T18:34:05.121 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-10-10T18:34:05.122 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-10T18:34:05.122 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2023-10-10T18:34:05.136 INFO:teuthology.orchestra.run.smithi175.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-10T18:34:05.137 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-10T18:34:05.137 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-10T18:34:05.137 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-10T18:34:06.138 DEBUG:teuthology.parallel:result is None 2023-10-10T18:34:06.139 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-10T18:34:06.153 INFO:teuthology.task.internal:Creating test directory... 2023-10-10T18:34:06.154 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-10T18:34:06.157 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-10T18:34:06.176 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-10T18:34:06.216 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-10T18:34:06.268 INFO:teuthology.task.internal:Creating archive directory... 2023-10-10T18:34:06.269 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-10T18:34:06.271 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-10T18:34:06.304 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-10T18:34:06.314 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-10T18:34:06.314 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-10T18:34:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:34:06.355 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-10T18:34:06.358 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-10T18:34:06.436 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T18:34:06.458 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T18:34:06.492 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T18:34:06.514 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T18:34:06.516 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-10T18:34:06.524 INFO:teuthology.task.internal:Configuring sudo... 2023-10-10T18:34:06.525 DEBUG:teuthology.orchestra.run.smithi175:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-10T18:34:06.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-10T18:34:06.554 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-10T18:34:06.585 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-10T18:34:06.585 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-10T18:34:06.590 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-10T18:34:06.611 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T18:34:06.682 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T18:34:06.718 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T18:34:06.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T18:34:06.833 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:34:06.833 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-10T18:34:06.897 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T18:34:06.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T18:34:06.984 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T18:34:07.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T18:34:07.106 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:34:07.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-10T18:34:07.171 DEBUG:teuthology.orchestra.run.smithi175:> sudo service rsyslog restart 2023-10-10T18:34:07.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2023-10-10T18:34:07.251 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T18:34:07.278 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T18:34:07.630 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-10T18:34:07.638 INFO:teuthology.task.internal:Starting timer... 2023-10-10T18:34:07.638 INFO:teuthology.run_tasks:Running task pcp... 2023-10-10T18:34:07.755 INFO:teuthology.run_tasks:Running task selinux... 2023-10-10T18:34:07.778 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-10-10T18:34:07.779 DEBUG:teuthology.orchestra.run.smithi175:> sudo service auditd rotate 2023-10-10T18:34:08.215 INFO:teuthology.orchestra.run.smithi175.stdout:Rotating logs: [ OK ] 2023-10-10T18:34:08.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2023-10-10T18:34:08.316 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: [ OK ] 2023-10-10T18:34:08.318 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-10T18:34:08.318 DEBUG:teuthology.orchestra.run.smithi175:> /usr/sbin/getenforce 2023-10-10T18:34:08.571 INFO:teuthology.orchestra.run.smithi175.stdout:Permissive 2023-10-10T18:34:08.571 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2023-10-10T18:34:08.592 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2023-10-10T18:34:08.593 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi175.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2023-10-10T18:34:08.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T18:34:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:34:08.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T18:34:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:34:08.766 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-10T18:34:08.766 DEBUG:teuthology.orchestra.run.smithi175:> sudo /usr/sbin/setenforce permissive 2023-10-10T18:34:09.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2023-10-10T18:34:09.119 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-10T18:34:09.128 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-10T18:34:09.135 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-10T18:34:09.283 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-10T18:34:09.301 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-10T18:34:09.302 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi175.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-10T18:42:16.290 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi175.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2023-10-10T18:42:16.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-10-10T18:42:16.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T18:42:16.363 DEBUG:teuthology.orchestra.run.smithi175:> true 2023-10-10T18:42:16.433 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-10-10T18:42:16.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-10-10T18:42:16.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T18:42:16.504 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-10-10T18:42:16.569 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-10-10T18:42:16.569 INFO:teuthology.run_tasks:Running task clock... 2023-10-10T18:42:16.580 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-10T18:42:16.581 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-10T18:42:16.581 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T18:42:16.583 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-10T18:42:16.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T18:42:16.610 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-10T18:42:16.624 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-10T18:42:16.637 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-10T18:42:16.651 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-10T18:42:16.658 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: ntpd: command not found 2023-10-10T18:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout:506 Cannot talk to daemon 2023-10-10T18:42:16.681 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-10T18:42:16.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2023-10-10T18:42:16.691 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2023-10-10T18:42:16.695 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-10T18:42:16.705 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-10T18:42:16.718 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-10T18:42:16.758 INFO:teuthology.orchestra.run.smithi175.stderr:bash: ntpq: command not found 2023-10-10T18:42:16.761 INFO:teuthology.orchestra.run.smithi175.stdout:210 Number of sources = 4 2023-10-10T18:42:16.761 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T18:42:16.762 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2023-10-10T18:42:16.762 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:42:16.762 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:42:16.762 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:42:16.763 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:42:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-10-10T18:42:16.791 INFO:teuthology.orchestra.run.smithi191.stdout:210 Number of sources = 4 2023-10-10T18:42:16.791 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T18:42:16.791 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-10-10T18:42:16.791 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:42:16.792 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:42:16.792 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:42:16.792 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:42:16.793 INFO:teuthology.run_tasks:Running task install... 2023-10-10T18:42:16.801 DEBUG:teuthology.task.install:project ceph 2023-10-10T18:42:16.801 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7'}} 2023-10-10T18:42:16.801 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7'} 2023-10-10T18:42:16.802 INFO:teuthology.task.install:Using flavor: default 2023-10-10T18:42:16.807 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-10T18:42:16.808 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-10T18:42:16.808 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7', 'tag': None, 'wait_for_package': False} 2023-10-10T18:42:16.809 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 2023-10-10T18:42:16.811 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7', 'tag': None, 'wait_for_package': False} 2023-10-10T18:42:16.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 2023-10-10T18:42:16.960 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-10-0720-quincy/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/centos/8/flavors/default/ 2023-10-10T18:42:16.960 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1374.g46d1f59b 2023-10-10T18:42:17.029 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-10-0720-quincy/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/centos/8/flavors/default/ 2023-10-10T18:42:17.030 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1374.g46d1f59b 2023-10-10T18:42:17.072 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-10-0720-quincy/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-10-0720-quincy/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-10-0720-quincy/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-10T18:42:17.072 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:42:17.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-10T18:42:17.098 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-10T18:42:17.098 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-10T18:42:17.132 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-10-0720-quincy/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-10-0720-quincy/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-10-0720-quincy/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-10T18:42:17.133 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:42:17.133 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-10T18:42:17.159 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-10T18:42:17.159 DEBUG:teuthology.orchestra.run.smithi175:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/46d1f59b549730ba7f93d940a3d32b94c45a4cf7/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-10T18:42:17.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-10T18:42:17.226 DEBUG:teuthology.orchestra.run.smithi175:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-10T18:42:17.239 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-10T18:42:17.300 DEBUG:teuthology.orchestra.run.smithi175:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-10T18:42:17.302 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2023-10-10T18:42:17.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2023-10-10T18:42:17.364 INFO:teuthology.orchestra.run.smithi175.stdout:check_obsoletes = 1 2023-10-10T18:42:17.365 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum clean all 2023-10-10T18:42:17.686 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-10-10T18:42:17.753 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2023-10-10T18:42:18.046 INFO:teuthology.orchestra.run.smithi191.stdout:44 files removed 2023-10-10T18:42:18.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-10T18:42:18.139 INFO:teuthology.orchestra.run.smithi175.stdout:44 files removed 2023-10-10T18:42:18.190 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-10T18:42:18.438 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-10-10T18:42:18.542 INFO:teuthology.orchestra.run.smithi175.stdout:Updating Subscription Management repositories. 2023-10-10T18:42:19.214 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 353 kB/s | 78 kB 00:00 2023-10-10T18:42:19.444 INFO:teuthology.orchestra.run.smithi175.stdout:ceph packages for x86_64 233 kB/s | 78 kB 00:00 2023-10-10T18:42:19.560 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-10-10T18:42:19.730 INFO:teuthology.orchestra.run.smithi175.stdout:ceph noarch packages 37 kB/s | 9.2 kB 00:00 2023-10-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 6.0 kB/s | 1.4 kB 00:00 2023-10-10T18:42:19.953 INFO:teuthology.orchestra.run.smithi191.stdout:Copr repo for python3-asyncssh owned by ceph 37 kB/s | 3.5 kB 00:00 2023-10-10T18:42:20.074 INFO:teuthology.orchestra.run.smithi175.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2023-10-10T18:42:20.186 INFO:teuthology.orchestra.run.smithi175.stdout:Copr repo for python3-asyncssh owned by ceph 42 kB/s | 3.5 kB 00:00 2023-10-10T18:42:20.772 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 29 MB/s | 16 MB 00:00 2023-10-10T18:42:21.052 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-10-10T18:42:24.083 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-10-10T18:42:24.277 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-10-10T18:42:24.492 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 116 MB/s | 38 MB 00:00 2023-10-10T18:42:24.684 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 117 MB/s | 38 MB 00:00 2023-10-10T18:42:29.964 INFO:teuthology.orchestra.run.smithi175.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 119 MB/s | 33 MB 00:00 2023-10-10T18:42:30.163 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 MB/s | 33 MB 00:00 2023-10-10T18:42:37.800 INFO:teuthology.orchestra.run.smithi175.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T18:42:37.801 INFO:teuthology.orchestra.run.smithi175.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T18:42:37.811 INFO:teuthology.orchestra.run.smithi175.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-10-10T18:42:37.812 INFO:teuthology.orchestra.run.smithi175.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-10-10T18:42:37.813 INFO:teuthology.orchestra.run.smithi175.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-10-10T18:42:37.814 INFO:teuthology.orchestra.run.smithi175.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-10T18:42:37.815 INFO:teuthology.orchestra.run.smithi175.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-10T18:42:37.818 INFO:teuthology.orchestra.run.smithi175.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-10-10T18:42:37.819 INFO:teuthology.orchestra.run.smithi175.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-10-10T18:42:37.994 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T18:42:37.995 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T18:42:38.005 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-10-10T18:42:38.006 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-10-10T18:42:38.007 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-10-10T18:42:38.008 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-10T18:42:38.010 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-10T18:42:38.012 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-10-10T18:42:38.013 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-10-10T18:42:38.315 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-10-10T18:42:38.327 INFO:teuthology.orchestra.run.smithi175.stdout:====================================================================================================================== 2023-10-10T18:42:38.327 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-10-10T18:42:38.327 INFO:teuthology.orchestra.run.smithi175.stdout:====================================================================================================================== 2023-10-10T18:42:38.328 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-10-10T18:42:38.328 INFO:teuthology.orchestra.run.smithi175.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-10T18:42:38.328 INFO:teuthology.orchestra.run.smithi175.stdout: ceph x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 6.4 k 2023-10-10T18:42:38.328 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 5.2 M 2023-10-10T18:42:38.328 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 876 k 2023-10-10T18:42:38.329 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 161 k 2023-10-10T18:42:38.329 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 1.6 M 2023-10-10T18:42:38.329 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 121 k 2023-10-10T18:42:38.329 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 1.7 M 2023-10-10T18:42:38.329 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 7.4 M 2023-10-10T18:42:38.330 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 47 k 2023-10-10T18:42:38.330 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 17 M 2023-10-10T18:42:38.330 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 52 M 2023-10-10T18:42:38.330 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 267 k 2023-10-10T18:42:38.331 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 80 k 2023-10-10T18:42:38.331 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-mirror x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 255 k 2023-10-10T18:42:38.331 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-top noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 18 k 2023-10-10T18:42:38.331 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-10-10T18:42:38.331 INFO:teuthology.orchestra.run.smithi175.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-10T18:42:38.332 INFO:teuthology.orchestra.run.smithi175.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-10T18:42:38.332 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 25 k 2023-10-10T18:42:38.332 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 797 k 2023-10-10T18:42:38.332 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 130 k 2023-10-10T18:42:38.332 INFO:teuthology.orchestra.run.smithi175.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-10T18:42:38.333 INFO:teuthology.orchestra.run.smithi175.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-10-10T18:42:38.333 INFO:teuthology.orchestra.run.smithi175.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-10T18:42:38.333 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 190 k 2023-10-10T18:42:38.333 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 371 k 2023-10-10T18:42:38.334 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 358 k 2023-10-10T18:42:38.334 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 107 k 2023-10-10T18:42:38.334 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 90 k 2023-10-10T18:42:38.334 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 3.5 M 2023-10-10T18:42:38.334 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 173 k 2023-10-10T18:42:38.335 INFO:teuthology.orchestra.run.smithi175.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-10-10T18:42:38.335 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrading: 2023-10-10T18:42:38.335 INFO:teuthology.orchestra.run.smithi175.stdout: librados2 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 3.5 M 2023-10-10T18:42:38.335 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 3.6 M 2023-10-10T18:42:38.335 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-10-10T18:42:38.336 INFO:teuthology.orchestra.run.smithi175.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-10T18:42:38.336 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 23 M 2023-10-10T18:42:38.336 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 23 k 2023-10-10T18:42:38.336 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 2.3 M 2023-10-10T18:42:38.336 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 238 k 2023-10-10T18:42:38.337 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 3.9 M 2023-10-10T18:42:38.337 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 17 M 2023-10-10T18:42:38.337 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 14 k 2023-10-10T18:42:38.337 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 24 k 2023-10-10T18:42:38.337 INFO:teuthology.orchestra.run.smithi175.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-10T18:42:38.338 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-10T18:42:38.338 INFO:teuthology.orchestra.run.smithi175.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-10T18:42:38.338 INFO:teuthology.orchestra.run.smithi175.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-10T18:42:38.338 INFO:teuthology.orchestra.run.smithi175.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-10T18:42:38.339 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 174 k 2023-10-10T18:42:38.339 INFO:teuthology.orchestra.run.smithi175.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-10-10T18:42:38.339 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-10T18:42:38.339 INFO:teuthology.orchestra.run.smithi175.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-10T18:42:38.339 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-10T18:42:38.340 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-10T18:42:38.340 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 507 k 2023-10-10T18:42:38.340 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-10T18:42:38.340 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 5.6 M 2023-10-10T18:42:38.340 INFO:teuthology.orchestra.run.smithi175.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-10T18:42:38.340 INFO:teuthology.orchestra.run.smithi175.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-10-10T18:42:38.341 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-10T18:42:38.341 INFO:teuthology.orchestra.run.smithi175.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-10T18:42:38.341 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-10T18:42:38.341 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-10T18:42:38.341 INFO:teuthology.orchestra.run.smithi175.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-10T18:42:38.342 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-10T18:42:38.342 INFO:teuthology.orchestra.run.smithi175.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-10T18:42:38.342 INFO:teuthology.orchestra.run.smithi175.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-10T18:42:38.342 INFO:teuthology.orchestra.run.smithi175.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-10T18:42:38.343 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-10T18:42:38.343 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-10T18:42:38.343 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-10T18:42:38.343 INFO:teuthology.orchestra.run.smithi175.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-10T18:42:38.343 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-10T18:42:38.344 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 45 k 2023-10-10T18:42:38.344 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 115 k 2023-10-10T18:42:38.344 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-10T18:42:38.344 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-10T18:42:38.344 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-10T18:42:38.345 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-10T18:42:38.345 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-10T18:42:38.345 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-10T18:42:38.345 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-10T18:42:38.345 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-10T18:42:38.346 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-10T18:42:38.346 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T18:42:38.346 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-10T18:42:38.346 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-10T18:42:38.346 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-10T18:42:38.347 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-10T18:42:38.347 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-10T18:42:38.347 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-10T18:42:38.347 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-10T18:42:38.347 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-10T18:42:38.348 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-10T18:42:38.348 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-10T18:42:38.348 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-10T18:42:38.348 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-10T18:42:38.348 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-10T18:42:38.349 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-10T18:42:38.349 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-10T18:42:38.349 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-10T18:42:38.349 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T18:42:38.349 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-10T18:42:38.350 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-10T18:42:38.350 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-10T18:42:38.350 INFO:teuthology.orchestra.run.smithi175.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-10T18:42:38.350 INFO:teuthology.orchestra.run.smithi175.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-10T18:42:38.350 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-10T18:42:38.351 INFO:teuthology.orchestra.run.smithi175.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-10T18:42:38.351 INFO:teuthology.orchestra.run.smithi175.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-10T18:42:38.351 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-10T18:42:38.351 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-10T18:42:38.351 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-10T18:42:38.352 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-10T18:42:38.352 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-10T18:42:38.352 INFO:teuthology.orchestra.run.smithi175.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-10T18:42:38.352 INFO:teuthology.orchestra.run.smithi175.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-10T18:42:38.352 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-10T18:42:38.353 INFO:teuthology.orchestra.run.smithi175.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-10T18:42:38.353 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:42:38.353 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-10-10T18:42:38.353 INFO:teuthology.orchestra.run.smithi175.stdout:====================================================================================================================== 2023-10-10T18:42:38.353 INFO:teuthology.orchestra.run.smithi175.stdout:Install 118 Packages 2023-10-10T18:42:38.354 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrade 2 Packages 2023-10-10T18:42:38.354 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:42:38.354 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 202 M 2023-10-10T18:42:38.355 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-10-10T18:42:38.494 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-10-10T18:42:38.505 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================================================== 2023-10-10T18:42:38.506 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-10-10T18:42:38.506 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================================================== 2023-10-10T18:42:38.506 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-10-10T18:42:38.506 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-10T18:42:38.506 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 6.4 k 2023-10-10T18:42:38.507 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 5.2 M 2023-10-10T18:42:38.507 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 876 k 2023-10-10T18:42:38.507 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 161 k 2023-10-10T18:42:38.507 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 1.6 M 2023-10-10T18:42:38.507 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 121 k 2023-10-10T18:42:38.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 1.7 M 2023-10-10T18:42:38.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 7.4 M 2023-10-10T18:42:38.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 47 k 2023-10-10T18:42:38.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 17 M 2023-10-10T18:42:38.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 52 M 2023-10-10T18:42:38.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 267 k 2023-10-10T18:42:38.509 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 80 k 2023-10-10T18:42:38.509 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 255 k 2023-10-10T18:42:38.509 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 18 k 2023-10-10T18:42:38.509 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-10-10T18:42:38.509 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-10T18:42:38.510 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-10T18:42:38.510 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 25 k 2023-10-10T18:42:38.510 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 797 k 2023-10-10T18:42:38.510 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 130 k 2023-10-10T18:42:38.511 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-10T18:42:38.511 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-10-10T18:42:38.511 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-10T18:42:38.511 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 190 k 2023-10-10T18:42:38.511 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 371 k 2023-10-10T18:42:38.511 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 358 k 2023-10-10T18:42:38.512 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 107 k 2023-10-10T18:42:38.512 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 90 k 2023-10-10T18:42:38.512 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 3.5 M 2023-10-10T18:42:38.512 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 173 k 2023-10-10T18:42:38.512 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-10-10T18:42:38.512 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2023-10-10T18:42:38.513 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 3.5 M 2023-10-10T18:42:38.513 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 3.6 M 2023-10-10T18:42:38.513 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-10-10T18:42:38.513 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-10T18:42:38.513 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 23 M 2023-10-10T18:42:38.514 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 23 k 2023-10-10T18:42:38.514 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 2.3 M 2023-10-10T18:42:38.514 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 238 k 2023-10-10T18:42:38.514 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 3.9 M 2023-10-10T18:42:38.514 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 17 M 2023-10-10T18:42:38.515 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1374.g46d1f59b.el8 ceph-noarch 14 k 2023-10-10T18:42:38.515 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 24 k 2023-10-10T18:42:38.515 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-10T18:42:38.515 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-10T18:42:38.516 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-10T18:42:38.516 INFO:teuthology.orchestra.run.smithi191.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-10T18:42:38.516 INFO:teuthology.orchestra.run.smithi191.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-10T18:42:38.516 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 174 k 2023-10-10T18:42:38.516 INFO:teuthology.orchestra.run.smithi191.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-10-10T18:42:38.517 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-10T18:42:38.517 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-10T18:42:38.517 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-10T18:42:38.517 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-10T18:42:38.517 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 507 k 2023-10-10T18:42:38.517 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-10T18:42:38.518 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 5.6 M 2023-10-10T18:42:38.518 INFO:teuthology.orchestra.run.smithi191.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-10T18:42:38.518 INFO:teuthology.orchestra.run.smithi191.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-10-10T18:42:38.518 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-10T18:42:38.518 INFO:teuthology.orchestra.run.smithi191.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-10T18:42:38.519 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-10T18:42:38.519 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-10T18:42:38.519 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-10T18:42:38.519 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-10T18:42:38.519 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-10T18:42:38.520 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-10T18:42:38.520 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-10T18:42:38.520 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-10T18:42:38.520 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-10T18:42:38.520 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-10T18:42:38.521 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-10T18:42:38.521 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-10T18:42:38.521 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 45 k 2023-10-10T18:42:38.521 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:17.2.6-1374.g46d1f59b.el8 ceph 115 k 2023-10-10T18:42:38.521 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-10T18:42:38.522 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-10T18:42:38.522 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-10T18:42:38.522 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-10T18:42:38.522 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-10T18:42:38.522 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-10T18:42:38.523 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-10T18:42:38.523 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-10T18:42:38.523 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-10T18:42:38.523 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T18:42:38.523 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-10T18:42:38.524 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-10T18:42:38.524 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-10T18:42:38.524 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-10T18:42:38.524 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-10T18:42:38.524 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-10T18:42:38.525 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-10T18:42:38.525 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-10T18:42:38.525 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-10T18:42:38.525 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-10T18:42:38.525 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-10T18:42:38.526 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-10T18:42:38.526 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-10T18:42:38.526 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-10T18:42:38.526 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-10T18:42:38.527 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-10T18:42:38.527 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T18:42:38.527 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-10T18:42:38.527 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-10T18:42:38.527 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-10T18:42:38.528 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-10T18:42:38.528 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-10T18:42:38.528 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-10T18:42:38.528 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-10T18:42:38.528 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-10T18:42:38.529 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-10T18:42:38.529 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-10T18:42:38.529 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-10T18:42:38.529 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-10T18:42:38.529 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-10T18:42:38.530 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-10T18:42:38.530 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-10T18:42:38.530 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-10T18:42:38.530 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-10T18:42:38.530 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:42:38.531 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-10-10T18:42:38.531 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================================================== 2023-10-10T18:42:38.531 INFO:teuthology.orchestra.run.smithi191.stdout:Install 118 Packages 2023-10-10T18:42:38.531 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2023-10-10T18:42:38.531 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:42:38.532 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 202 M 2023-10-10T18:42:38.532 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-10-10T18:42:38.771 INFO:teuthology.orchestra.run.smithi191.stdout:(1/120): ceph-17.2.6-1374.g46d1f59b.el8.x86_64. 61 kB/s | 6.4 kB 00:00 2023-10-10T18:42:38.780 INFO:teuthology.orchestra.run.smithi175.stdout:(1/120): ceph-17.2.6-1374.g46d1f59b.el8.x86_64. 69 kB/s | 6.4 kB 00:00 2023-10-10T18:42:39.141 INFO:teuthology.orchestra.run.smithi175.stdout:(2/120): ceph-fuse-17.2.6-1374.g46d1f59b.el8.x8 2.4 MB/s | 876 kB 00:00 2023-10-10T18:42:39.143 INFO:teuthology.orchestra.run.smithi191.stdout:(2/120): ceph-fuse-17.2.6-1374.g46d1f59b.el8.x8 2.3 MB/s | 876 kB 00:00 2023-10-10T18:42:39.190 INFO:teuthology.orchestra.run.smithi175.stdout:(3/120): ceph-immutable-object-cache-17.2.6-137 3.1 MB/s | 161 kB 00:00 2023-10-10T18:42:39.214 INFO:teuthology.orchestra.run.smithi191.stdout:(3/120): ceph-immutable-object-cache-17.2.6-137 2.2 MB/s | 161 kB 00:00 2023-10-10T18:42:39.411 INFO:teuthology.orchestra.run.smithi191.stdout:(4/120): ceph-base-17.2.6-1374.g46d1f59b.el8.x8 6.9 MB/s | 5.2 MB 00:00 2023-10-10T18:42:39.625 INFO:teuthology.orchestra.run.smithi191.stdout:(5/120): ceph-mgr-17.2.6-1374.g46d1f59b.el8.x86 7.4 MB/s | 1.6 MB 00:00 2023-10-10T18:42:39.811 INFO:teuthology.orchestra.run.smithi191.stdout:(6/120): ceph-mds-17.2.6-1374.g46d1f59b.el8.x86 3.9 MB/s | 2.3 MB 00:00 2023-10-10T18:42:39.935 INFO:teuthology.orchestra.run.smithi175.stdout:(4/120): ceph-mds-17.2.6-1374.g46d1f59b.el8.x86 3.1 MB/s | 2.3 MB 00:00 2023-10-10T18:42:40.136 INFO:teuthology.orchestra.run.smithi191.stdout:(7/120): ceph-mon-17.2.6-1374.g46d1f59b.el8.x86 7.7 MB/s | 3.9 MB 00:00 2023-10-10T18:42:40.178 INFO:teuthology.orchestra.run.smithi175.stdout:(5/120): ceph-base-17.2.6-1374.g46d1f59b.el8.x8 3.5 MB/s | 5.2 MB 00:01 2023-10-10T18:42:40.465 INFO:teuthology.orchestra.run.smithi175.stdout:(6/120): ceph-mgr-17.2.6-1374.g46d1f59b.el8.x86 3.0 MB/s | 1.6 MB 00:00 2023-10-10T18:42:41.313 INFO:teuthology.orchestra.run.smithi175.stdout:(7/120): ceph-mon-17.2.6-1374.g46d1f59b.el8.x86 3.5 MB/s | 3.9 MB 00:01 2023-10-10T18:42:42.366 INFO:teuthology.orchestra.run.smithi191.stdout:(8/120): ceph-common-17.2.6-1374.g46d1f59b.el8. 6.3 MB/s | 23 MB 00:03 2023-10-10T18:42:42.472 INFO:teuthology.orchestra.run.smithi191.stdout:(9/120): ceph-radosgw-17.2.6-1374.g46d1f59b.el8 7.2 MB/s | 17 MB 00:02 2023-10-10T18:42:42.473 INFO:teuthology.orchestra.run.smithi191.stdout:(10/120): ceph-selinux-17.2.6-1374.g46d1f59b.el 226 kB/s | 24 kB 00:00 2023-10-10T18:42:42.595 INFO:teuthology.orchestra.run.smithi191.stdout:(11/120): cephfs-mirror-17.2.6-1374.g46d1f59b.e 2.1 MB/s | 255 kB 00:00 2023-10-10T18:42:42.657 INFO:teuthology.orchestra.run.smithi191.stdout:(12/120): libcephfs-devel-17.2.6-1374.g46d1f59b 397 kB/s | 25 kB 00:00 2023-10-10T18:42:42.813 INFO:teuthology.orchestra.run.smithi191.stdout:(13/120): libcephfs2-17.2.6-1374.g46d1f59b.el8. 5.0 MB/s | 797 kB 00:00 2023-10-10T18:42:42.884 INFO:teuthology.orchestra.run.smithi191.stdout:(14/120): libcephsqlite-17.2.6-1374.g46d1f59b.e 2.4 MB/s | 174 kB 00:00 2023-10-10T18:42:42.953 INFO:teuthology.orchestra.run.smithi191.stdout:(15/120): librados-devel-17.2.6-1374.g46d1f59b. 1.8 MB/s | 130 kB 00:00 2023-10-10T18:42:43.098 INFO:teuthology.orchestra.run.smithi191.stdout:(16/120): libradosstriper1-17.2.6-1374.g46d1f59 3.4 MB/s | 507 kB 00:00 2023-10-10T18:42:43.989 INFO:teuthology.orchestra.run.smithi191.stdout:(17/120): librgw2-17.2.6-1374.g46d1f59b.el8.x86 6.2 MB/s | 5.6 MB 00:00 2023-10-10T18:42:44.049 INFO:teuthology.orchestra.run.smithi191.stdout:(18/120): python3-ceph-argparse-17.2.6-1374.g46 752 kB/s | 45 kB 00:00 2023-10-10T18:42:44.112 INFO:teuthology.orchestra.run.smithi191.stdout:(19/120): python3-ceph-common-17.2.6-1374.g46d1 1.8 MB/s | 115 kB 00:00 2023-10-10T18:42:44.178 INFO:teuthology.orchestra.run.smithi191.stdout:(20/120): python3-cephfs-17.2.6-1374.g46d1f59b. 2.8 MB/s | 190 kB 00:00 2023-10-10T18:42:44.252 INFO:teuthology.orchestra.run.smithi191.stdout:(21/120): python3-rados-17.2.6-1374.g46d1f59b.e 4.9 MB/s | 371 kB 00:00 2023-10-10T18:42:44.325 INFO:teuthology.orchestra.run.smithi191.stdout:(22/120): python3-rbd-17.2.6-1374.g46d1f59b.el8 4.8 MB/s | 358 kB 00:00 2023-10-10T18:42:44.389 INFO:teuthology.orchestra.run.smithi191.stdout:(23/120): python3-rgw-17.2.6-1374.g46d1f59b.el8 1.6 MB/s | 107 kB 00:00 2023-10-10T18:42:44.562 INFO:teuthology.orchestra.run.smithi191.stdout:(24/120): ceph-osd-17.2.6-1374.g46d1f59b.el8.x8 3.5 MB/s | 17 MB 00:04 2023-10-10T18:42:44.563 INFO:teuthology.orchestra.run.smithi191.stdout:(25/120): rbd-fuse-17.2.6-1374.g46d1f59b.el8.x8 518 kB/s | 90 kB 00:00 2023-10-10T18:42:44.631 INFO:teuthology.orchestra.run.smithi191.stdout:(26/120): rbd-nbd-17.2.6-1374.g46d1f59b.el8.x86 2.5 MB/s | 173 kB 00:00 2023-10-10T18:42:44.692 INFO:teuthology.orchestra.run.smithi191.stdout:(27/120): ceph-grafana-dashboards-17.2.6-1374.g 379 kB/s | 23 kB 00:00 2023-10-10T18:42:44.757 INFO:teuthology.orchestra.run.smithi191.stdout:(28/120): ceph-mgr-cephadm-17.2.6-1374.g46d1f59 1.8 MB/s | 121 kB 00:00 2023-10-10T18:42:44.872 INFO:teuthology.orchestra.run.smithi175.stdout:(8/120): ceph-osd-17.2.6-1374.g46d1f59b.el8.x86 3.8 MB/s | 17 MB 00:04 2023-10-10T18:42:44.979 INFO:teuthology.orchestra.run.smithi191.stdout:(29/120): ceph-mgr-dashboard-17.2.6-1374.g46d1f 7.6 MB/s | 1.7 MB 00:00 2023-10-10T18:42:44.985 INFO:teuthology.orchestra.run.smithi175.stdout:(9/120): ceph-radosgw-17.2.6-1374.g46d1f59b.el8 4.6 MB/s | 17 MB 00:03 2023-10-10T18:42:44.985 INFO:teuthology.orchestra.run.smithi175.stdout:(10/120): ceph-selinux-17.2.6-1374.g46d1f59b.el 214 kB/s | 24 kB 00:00 2023-10-10T18:42:45.128 INFO:teuthology.orchestra.run.smithi175.stdout:(11/120): cephfs-mirror-17.2.6-1374.g46d1f59b.e 1.8 MB/s | 255 kB 00:00 2023-10-10T18:42:45.198 INFO:teuthology.orchestra.run.smithi175.stdout:(12/120): libcephfs-devel-17.2.6-1374.g46d1f59b 355 kB/s | 25 kB 00:00 2023-10-10T18:42:45.344 INFO:teuthology.orchestra.run.smithi175.stdout:(13/120): ceph-common-17.2.6-1374.g46d1f59b.el8 3.5 MB/s | 23 MB 00:06 2023-10-10T18:42:45.421 INFO:teuthology.orchestra.run.smithi175.stdout:(14/120): libcephsqlite-17.2.6-1374.g46d1f59b.e 2.2 MB/s | 174 kB 00:00 2023-10-10T18:42:45.496 INFO:teuthology.orchestra.run.smithi175.stdout:(15/120): librados-devel-17.2.6-1374.g46d1f59b. 1.7 MB/s | 130 kB 00:00 2023-10-10T18:42:45.516 INFO:teuthology.orchestra.run.smithi175.stdout:(16/120): libcephfs2-17.2.6-1374.g46d1f59b.el8. 2.4 MB/s | 797 kB 00:00 2023-10-10T18:42:45.556 INFO:teuthology.orchestra.run.smithi191.stdout:(30/120): rbd-mirror-17.2.6-1374.g46d1f59b.el8. 3.5 MB/s | 3.5 MB 00:00 2023-10-10T18:42:45.705 INFO:teuthology.orchestra.run.smithi191.stdout:(31/120): ceph-mgr-modules-core-17.2.6-1374.g46 1.6 MB/s | 238 kB 00:00 2023-10-10T18:42:45.707 INFO:teuthology.orchestra.run.smithi175.stdout:(17/120): libradosstriper1-17.2.6-1374.g46d1f59 2.4 MB/s | 507 kB 00:00 2023-10-10T18:42:45.770 INFO:teuthology.orchestra.run.smithi191.stdout:(32/120): ceph-mgr-rook-17.2.6-1374.g46d1f59b.e 733 kB/s | 47 kB 00:00 2023-10-10T18:42:45.773 INFO:teuthology.orchestra.run.smithi175.stdout:(18/120): python3-ceph-argparse-17.2.6-1374.g46 681 kB/s | 45 kB 00:00 2023-10-10T18:42:45.834 INFO:teuthology.orchestra.run.smithi191.stdout:(33/120): ceph-prometheus-alerts-17.2.6-1374.g4 225 kB/s | 14 kB 00:00 2023-10-10T18:42:45.840 INFO:teuthology.orchestra.run.smithi175.stdout:(19/120): python3-ceph-common-17.2.6-1374.g46d1 1.7 MB/s | 115 kB 00:00 2023-10-10T18:42:45.918 INFO:teuthology.orchestra.run.smithi175.stdout:(20/120): python3-cephfs-17.2.6-1374.g46d1f59b. 2.4 MB/s | 190 kB 00:00 2023-10-10T18:42:45.978 INFO:teuthology.orchestra.run.smithi191.stdout:(34/120): ceph-volume-17.2.6-1374.g46d1f59b.el8 1.8 MB/s | 267 kB 00:00 2023-10-10T18:42:46.029 INFO:teuthology.orchestra.run.smithi191.stdout:(35/120): ceph-mgr-diskprediction-local-17.2.6- 7.1 MB/s | 7.4 MB 00:01 2023-10-10T18:42:46.047 INFO:teuthology.orchestra.run.smithi191.stdout:(36/120): cephadm-17.2.6-1374.g46d1f59b.el8.noa 1.1 MB/s | 80 kB 00:00 2023-10-10T18:42:46.054 INFO:teuthology.orchestra.run.smithi175.stdout:(21/120): python3-rados-17.2.6-1374.g46d1f59b.e 2.7 MB/s | 371 kB 00:00 2023-10-10T18:42:46.084 INFO:teuthology.orchestra.run.smithi191.stdout:(37/120): cephfs-top-17.2.6-1374.g46d1f59b.el8. 331 kB/s | 18 kB 00:00 2023-10-10T18:42:46.157 INFO:teuthology.orchestra.run.smithi175.stdout:(22/120): python3-rbd-17.2.6-1374.g46d1f59b.el8 3.4 MB/s | 358 kB 00:00 2023-10-10T18:42:46.207 INFO:teuthology.orchestra.run.smithi175.stdout:(23/120): python3-rgw-17.2.6-1374.g46d1f59b.el8 2.1 MB/s | 107 kB 00:00 2023-10-10T18:42:46.255 INFO:teuthology.orchestra.run.smithi175.stdout:(24/120): rbd-fuse-17.2.6-1374.g46d1f59b.el8.x8 1.8 MB/s | 90 kB 00:00 2023-10-10T18:42:46.438 INFO:teuthology.orchestra.run.smithi191.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 166 kB/s | 59 kB 00:00 2023-10-10T18:42:46.603 INFO:teuthology.orchestra.run.smithi191.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 456 kB/s | 75 kB 00:00 2023-10-10T18:42:46.604 INFO:teuthology.orchestra.run.smithi191.stdout:(40/120): gperftools-libs-2.7-9.el8.x86_64.rpm 549 kB/s | 306 kB 00:00 2023-10-10T18:42:46.693 INFO:teuthology.orchestra.run.smithi191.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 491 kB/s | 44 kB 00:00 2023-10-10T18:42:46.809 INFO:teuthology.orchestra.run.smithi191.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 1.6 MB/s | 185 kB 00:00 2023-10-10T18:42:46.857 INFO:teuthology.orchestra.run.smithi175.stdout:(25/120): librgw2-17.2.6-1374.g46d1f59b.el8.x86 4.1 MB/s | 5.6 MB 00:01 2023-10-10T18:42:46.878 INFO:teuthology.orchestra.run.smithi191.stdout:(43/120): python3-asyncssh-2.7.0-2.el8.noarch.r 1.7 MB/s | 476 kB 00:00 2023-10-10T18:42:46.896 INFO:teuthology.orchestra.run.smithi191.stdout:(44/120): python3-cachetools-3.1.1-4.el8.noarch 380 kB/s | 33 kB 00:00 2023-10-10T18:42:46.909 INFO:teuthology.orchestra.run.smithi175.stdout:(26/120): rbd-nbd-17.2.6-1374.g46d1f59b.el8.x86 3.3 MB/s | 173 kB 00:00 2023-10-10T18:42:46.954 INFO:teuthology.orchestra.run.smithi175.stdout:(27/120): ceph-grafana-dashboards-17.2.6-1374.g 506 kB/s | 23 kB 00:00 2023-10-10T18:42:46.961 INFO:teuthology.orchestra.run.smithi191.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 197 kB/s | 16 kB 00:00 2023-10-10T18:42:47.001 INFO:teuthology.orchestra.run.smithi175.stdout:(28/120): ceph-mgr-cephadm-17.2.6-1374.g46d1f59 2.5 MB/s | 121 kB 00:00 2023-10-10T18:42:47.024 INFO:teuthology.orchestra.run.smithi175.stdout:(29/120): rbd-mirror-17.2.6-1374.g46d1f59b.el8. 4.5 MB/s | 3.5 MB 00:00 2023-10-10T18:42:47.187 INFO:teuthology.orchestra.run.smithi175.stdout:(30/120): ceph-mgr-dashboard-17.2.6-1374.g46d1f 9.1 MB/s | 1.7 MB 00:00 2023-10-10T18:42:47.216 INFO:teuthology.orchestra.run.smithi175.stdout:(31/120): ceph-mgr-modules-core-17.2.6-1374.g46 8.0 MB/s | 238 kB 00:00 2023-10-10T18:42:47.238 INFO:teuthology.orchestra.run.smithi175.stdout:(32/120): ceph-mgr-rook-17.2.6-1374.g46d1f59b.e 2.2 MB/s | 47 kB 00:00 2023-10-10T18:42:47.259 INFO:teuthology.orchestra.run.smithi175.stdout:(33/120): ceph-prometheus-alerts-17.2.6-1374.g4 697 kB/s | 14 kB 00:00 2023-10-10T18:42:47.305 INFO:teuthology.orchestra.run.smithi175.stdout:(34/120): ceph-volume-17.2.6-1374.g46d1f59b.el8 5.6 MB/s | 267 kB 00:00 2023-10-10T18:42:47.328 INFO:teuthology.orchestra.run.smithi175.stdout:(35/120): cephadm-17.2.6-1374.g46d1f59b.el8.noa 3.6 MB/s | 80 kB 00:00 2023-10-10T18:42:47.349 INFO:teuthology.orchestra.run.smithi175.stdout:(36/120): cephfs-top-17.2.6-1374.g46d1f59b.el8. 861 kB/s | 18 kB 00:00 2023-10-10T18:42:47.425 INFO:teuthology.orchestra.run.smithi191.stdout:(46/120): ceph-test-17.2.6-1374.g46d1f59b.el8.x 10 MB/s | 52 MB 00:04 2023-10-10T18:42:47.426 INFO:teuthology.orchestra.run.smithi191.stdout:(47/120): python3-cheroot-8.5.2-1.el8.noarch.rp 326 kB/s | 173 kB 00:00 2023-10-10T18:42:47.429 INFO:teuthology.orchestra.run.smithi191.stdout:(48/120): python3-cherrypy-18.4.0-1.el8.noarch. 822 kB/s | 384 kB 00:00 2023-10-10T18:42:47.479 INFO:teuthology.orchestra.run.smithi175.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 2.3 MB/s | 306 kB 00:00 2023-10-10T18:42:47.501 INFO:teuthology.orchestra.run.smithi175.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 2.7 MB/s | 59 kB 00:00 2023-10-10T18:42:47.509 INFO:teuthology.orchestra.run.smithi191.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 134 kB/s | 11 kB 00:00 2023-10-10T18:42:47.512 INFO:teuthology.orchestra.run.smithi191.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 211 kB/s | 18 kB 00:00 2023-10-10T18:42:47.523 INFO:teuthology.orchestra.run.smithi175.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 3.3 MB/s | 75 kB 00:00 2023-10-10T18:42:47.568 INFO:teuthology.orchestra.run.smithi175.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 10 MB/s | 476 kB 00:00 2023-10-10T18:42:47.588 INFO:teuthology.orchestra.run.smithi175.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.2 MB/s | 44 kB 00:00 2023-10-10T18:42:47.599 INFO:teuthology.orchestra.run.smithi191.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 565 kB/s | 49 kB 00:00 2023-10-10T18:42:47.611 INFO:teuthology.orchestra.run.smithi175.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 8.0 MB/s | 185 kB 00:00 2023-10-10T18:42:47.631 INFO:teuthology.orchestra.run.smithi175.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.7 MB/s | 33 kB 00:00 2023-10-10T18:42:47.651 INFO:teuthology.orchestra.run.smithi175.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 823 kB/s | 16 kB 00:00 2023-10-10T18:42:47.673 INFO:teuthology.orchestra.run.smithi175.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 7.6 MB/s | 173 kB 00:00 2023-10-10T18:42:47.687 INFO:teuthology.orchestra.run.smithi191.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 669 kB/s | 59 kB 00:00 2023-10-10T18:42:47.700 INFO:teuthology.orchestra.run.smithi175.stdout:(46/120): python3-cherrypy-18.4.0-1.el8.noarch. 14 MB/s | 384 kB 00:00 2023-10-10T18:42:47.733 INFO:teuthology.orchestra.run.smithi175.stdout:(47/120): ceph-mgr-diskprediction-local-17.2.6- 10 MB/s | 7.4 MB 00:00 2023-10-10T18:42:47.733 INFO:teuthology.orchestra.run.smithi175.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 2.4 MB/s | 81 kB 00:00 2023-10-10T18:42:47.753 INFO:teuthology.orchestra.run.smithi175.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 907 kB/s | 18 kB 00:00 2023-10-10T18:42:47.771 INFO:teuthology.orchestra.run.smithi175.stdout:(50/120): python3-jaraco-6.2-6.el8.noarch.rpm 287 kB/s | 11 kB 00:00 2023-10-10T18:42:47.774 INFO:teuthology.orchestra.run.smithi191.stdout:(53/120): python3-kubernetes-11.0.0-6.el8.noarc 4.4 MB/s | 1.2 MB 00:00 2023-10-10T18:42:47.777 INFO:teuthology.orchestra.run.smithi191.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 666 kB/s | 60 kB 00:00 2023-10-10T18:42:47.798 INFO:teuthology.orchestra.run.smithi175.stdout:(51/120): python3-kubernetes-11.0.0-6.el8.noarc 26 MB/s | 1.2 MB 00:00 2023-10-10T18:42:47.818 INFO:teuthology.orchestra.run.smithi175.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 3.0 MB/s | 59 kB 00:00 2023-10-10T18:42:47.820 INFO:teuthology.orchestra.run.smithi175.stdout:(53/120): python3-logutils-0.3.5-11.el8.noarch. 1.0 MB/s | 49 kB 00:00 2023-10-10T18:42:47.827 INFO:teuthology.orchestra.run.smithi191.stdout:(55/120): python3-google-auth-1.1.1-10.el8.noar 202 kB/s | 81 kB 00:00 2023-10-10T18:42:47.838 INFO:teuthology.orchestra.run.smithi175.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 3.0 MB/s | 60 kB 00:00 2023-10-10T18:42:47.857 INFO:teuthology.orchestra.run.smithi175.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 812 kB/s | 16 kB 00:00 2023-10-10T18:42:47.860 INFO:teuthology.orchestra.run.smithi191.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 188 kB/s | 16 kB 00:00 2023-10-10T18:42:47.870 INFO:teuthology.orchestra.run.smithi191.stdout:(57/120): python3-pecan-1.3.2-9.el8.noarch.rpm 2.9 MB/s | 283 kB 00:00 2023-10-10T18:42:47.877 INFO:teuthology.orchestra.run.smithi175.stdout:(56/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.7 MB/s | 34 kB 00:00 2023-10-10T18:42:47.890 INFO:teuthology.orchestra.run.smithi175.stdout:(57/120): python3-pecan-1.3.2-9.el8.noarch.rpm 4.0 MB/s | 283 kB 00:00 2023-10-10T18:42:47.897 INFO:teuthology.orchestra.run.smithi175.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 9.4 MB/s | 196 kB 00:00 2023-10-10T18:42:47.912 INFO:teuthology.orchestra.run.smithi175.stdout:(59/120): python3-rsa-4.9-2.el8.noarch.rpm 2.9 MB/s | 64 kB 00:00 2023-10-10T18:42:47.917 INFO:teuthology.orchestra.run.smithi175.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 969 kB/s | 19 kB 00:00 2023-10-10T18:42:47.931 INFO:teuthology.orchestra.run.smithi175.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 1.2 MB/s | 24 kB 00:00 2023-10-10T18:42:47.938 INFO:teuthology.orchestra.run.smithi175.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 1.4 MB/s | 29 kB 00:00 2023-10-10T18:42:47.939 INFO:teuthology.orchestra.run.smithi191.stdout:(58/120): python3-repoze-lru-0.7-6.el8.noarch.r 301 kB/s | 34 kB 00:00 2023-10-10T18:42:47.951 INFO:teuthology.orchestra.run.smithi175.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 1.4 MB/s | 27 kB 00:00 2023-10-10T18:42:47.954 INFO:teuthology.orchestra.run.smithi191.stdout:(59/120): python3-rsa-4.9-2.el8.noarch.rpm 773 kB/s | 64 kB 00:00 2023-10-10T18:42:47.958 INFO:teuthology.orchestra.run.smithi175.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 12 MB/s | 255 kB 00:00 2023-10-10T18:42:47.962 INFO:teuthology.orchestra.run.smithi191.stdout:(60/120): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-10-10T18:42:47.978 INFO:teuthology.orchestra.run.smithi175.stdout:(65/120): python3-websocket-client-0.56.0-5.el8 3.1 MB/s | 61 kB 00:00 2023-10-10T18:42:47.980 INFO:teuthology.orchestra.run.smithi175.stdout:(66/120): python3-webob-1.8.5-1.el8.1.noarch.rp 8.4 MB/s | 251 kB 00:00 2023-10-10T18:42:47.997 INFO:teuthology.orchestra.run.smithi175.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 4.3 MB/s | 85 kB 00:00 2023-10-10T18:42:48.000 INFO:teuthology.orchestra.run.smithi175.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 1.2 MB/s | 23 kB 00:00 2023-10-10T18:42:48.005 INFO:teuthology.orchestra.run.smithi175.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-10-10T18:42:48.035 INFO:teuthology.orchestra.run.smithi191.stdout:(61/120): python3-simplegeneric-0.8.1-17.el8.no 195 kB/s | 19 kB 00:00 2023-10-10T18:42:48.036 INFO:teuthology.orchestra.run.smithi191.stdout:(62/120): python3-singledispatch-3.4.0.3-18.el8 291 kB/s | 24 kB 00:00 2023-10-10T18:42:48.037 INFO:teuthology.orchestra.run.smithi175.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 584 kB/s | 19 kB 00:00 2023-10-10T18:42:48.047 INFO:teuthology.orchestra.run.smithi175.stdout:(71/120): thrift-0.13.0-2.el8.x86_64.rpm 35 MB/s | 1.7 MB 00:00 2023-10-10T18:42:48.047 INFO:teuthology.orchestra.run.smithi191.stdout:(63/120): python3-tempora-1.14.1-5.el8.noarch.r 348 kB/s | 29 kB 00:00 2023-10-10T18:42:48.053 INFO:teuthology.orchestra.run.smithi175.stdout:(72/120): python3-jwt-1.6.1-2.el8.noarch.rpm 2.6 MB/s | 43 kB 00:00 2023-10-10T18:42:48.071 INFO:teuthology.orchestra.run.smithi175.stdout:(73/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 12 MB/s | 220 kB 00:00 2023-10-10T18:42:48.079 INFO:teuthology.orchestra.run.smithi175.stdout:(74/120): python3-oauthlib-2.1.0-1.el8.noarch.r 4.7 MB/s | 155 kB 00:00 2023-10-10T18:42:48.086 INFO:teuthology.orchestra.run.smithi175.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.0 MB/s | 48 kB 00:00 2023-10-10T18:42:48.098 INFO:teuthology.orchestra.run.smithi175.stdout:(76/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 12 MB/s | 229 kB 00:00 2023-10-10T18:42:48.101 INFO:teuthology.orchestra.run.smithi175.stdout:(77/120): python3-requests-oauthlib-1.0.0-1.el8 3.0 MB/s | 43 kB 00:00 2023-10-10T18:42:48.116 INFO:teuthology.orchestra.run.smithi175.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 1.1 MB/s | 18 kB 00:00 2023-10-10T18:42:48.128 INFO:teuthology.orchestra.run.smithi191.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 2.7 MB/s | 255 kB 00:00 2023-10-10T18:42:48.141 INFO:teuthology.orchestra.run.smithi191.stdout:(65/120): python3-trustme-0.6.0-4.el8.noarch.rp 262 kB/s | 27 kB 00:00 2023-10-10T18:42:48.141 INFO:teuthology.orchestra.run.smithi175.stdout:(79/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 56 MB/s | 2.3 MB 00:00 2023-10-10T18:42:48.144 INFO:teuthology.orchestra.run.smithi175.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 21 MB/s | 605 kB 00:00 2023-10-10T18:42:48.155 INFO:teuthology.orchestra.run.smithi191.stdout:(66/120): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-10-10T18:42:48.156 INFO:teuthology.orchestra.run.smithi175.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 5.0 MB/s | 83 kB 00:00 2023-10-10T18:42:48.159 INFO:teuthology.orchestra.run.smithi175.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 3.9 MB/s | 59 kB 00:00 2023-10-10T18:42:48.174 INFO:teuthology.orchestra.run.smithi175.stdout:(83/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.6 MB/s | 38 kB 00:00 2023-10-10T18:42:48.175 INFO:teuthology.orchestra.run.smithi175.stdout:(84/120): libquadmath-8.4.1-1.el8.x86_64.rpm 8.8 MB/s | 169 kB 00:00 2023-10-10T18:42:48.188 INFO:teuthology.orchestra.run.smithi175.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.7 MB/s | 86 kB 00:00 2023-10-10T18:42:48.191 INFO:teuthology.orchestra.run.smithi175.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.7 MB/s | 58 kB 00:00 2023-10-10T18:42:48.203 INFO:teuthology.orchestra.run.smithi175.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.2 MB/s | 47 kB 00:00 2023-10-10T18:42:48.214 INFO:teuthology.orchestra.run.smithi191.stdout:(67/120): python3-websocket-client-0.56.0-5.el8 718 kB/s | 61 kB 00:00 2023-10-10T18:42:48.217 INFO:teuthology.orchestra.run.smithi175.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 21 MB/s | 559 kB 00:00 2023-10-10T18:42:48.227 INFO:teuthology.orchestra.run.smithi175.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 27 MB/s | 639 kB 00:00 2023-10-10T18:42:48.235 INFO:teuthology.orchestra.run.smithi175.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 3.9 MB/s | 70 kB 00:00 2023-10-10T18:42:48.241 INFO:teuthology.orchestra.run.smithi191.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 267 kB/s | 23 kB 00:00 2023-10-10T18:42:48.246 INFO:teuthology.orchestra.run.smithi191.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-10-10T18:42:48.247 INFO:teuthology.orchestra.run.smithi175.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 9.7 MB/s | 200 kB 00:00 2023-10-10T18:42:48.256 INFO:teuthology.orchestra.run.smithi175.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 4.6 MB/s | 97 kB 00:00 2023-10-10T18:42:48.272 INFO:teuthology.orchestra.run.smithi175.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 2.8 MB/s | 44 kB 00:00 2023-10-10T18:42:48.285 INFO:teuthology.orchestra.run.smithi191.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 478 kB/s | 19 kB 00:00 2023-10-10T18:42:48.286 INFO:teuthology.orchestra.run.smithi175.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.8 MB/s | 27 kB 00:00 2023-10-10T18:42:48.297 INFO:teuthology.orchestra.run.smithi191.stdout:(71/120): python3-webtest-2.0.33-1.el8.noarch.r 543 kB/s | 85 kB 00:00 2023-10-10T18:42:48.302 INFO:teuthology.orchestra.run.smithi175.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 10 MB/s | 161 kB 00:00 2023-10-10T18:42:48.303 INFO:teuthology.orchestra.run.smithi191.stdout:(72/120): python3-jwt-1.6.1-2.el8.noarch.rpm 2.4 MB/s | 43 kB 00:00 2023-10-10T18:42:48.322 INFO:teuthology.orchestra.run.smithi175.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 16 MB/s | 320 kB 00:00 2023-10-10T18:42:48.323 INFO:teuthology.orchestra.run.smithi191.stdout:(73/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 11 MB/s | 220 kB 00:00 2023-10-10T18:42:48.339 INFO:teuthology.orchestra.run.smithi191.stdout:(74/120): python3-oauthlib-2.1.0-1.el8.noarch.r 3.7 MB/s | 155 kB 00:00 2023-10-10T18:42:48.340 INFO:teuthology.orchestra.run.smithi191.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 2.8 MB/s | 48 kB 00:00 2023-10-10T18:42:48.344 INFO:teuthology.orchestra.run.smithi175.stdout:(97/120): python3-werkzeug-0.12.2-4.el8.noarch. 20 MB/s | 457 kB 00:00 2023-10-10T18:42:48.356 INFO:teuthology.orchestra.run.smithi191.stdout:(76/120): python3-requests-oauthlib-1.0.0-1.el8 2.6 MB/s | 43 kB 00:00 2023-10-10T18:42:48.359 INFO:teuthology.orchestra.run.smithi191.stdout:(77/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 11 MB/s | 229 kB 00:00 2023-10-10T18:42:48.376 INFO:teuthology.orchestra.run.smithi191.stdout:(78/120): thrift-0.13.0-2.el8.x86_64.rpm 11 MB/s | 1.7 MB 00:00 2023-10-10T18:42:48.378 INFO:teuthology.orchestra.run.smithi191.stdout:(79/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 987 kB/s | 18 kB 00:00 2023-10-10T18:42:48.396 INFO:teuthology.orchestra.run.smithi191.stdout:(80/120): fuse-2.9.7-12.el8.x86_64.rpm 4.5 MB/s | 83 kB 00:00 2023-10-10T18:42:48.409 INFO:teuthology.orchestra.run.smithi191.stdout:(81/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 44 MB/s | 2.3 MB 00:00 2023-10-10T18:42:48.416 INFO:teuthology.orchestra.run.smithi191.stdout:(82/120): pcre2-devel-10.32-2.el8.x86_64.rpm 15 MB/s | 605 kB 00:00 2023-10-10T18:42:48.417 INFO:teuthology.orchestra.run.smithi191.stdout:(83/120): elfutils-libelf-devel-0.182-3.el8.x86 2.7 MB/s | 59 kB 00:00 2023-10-10T18:42:48.429 INFO:teuthology.orchestra.run.smithi191.stdout:(84/120): libquadmath-8.4.1-1.el8.x86_64.rpm 8.8 MB/s | 169 kB 00:00 2023-10-10T18:42:48.434 INFO:teuthology.orchestra.run.smithi191.stdout:(85/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.2 MB/s | 38 kB 00:00 2023-10-10T18:42:48.435 INFO:teuthology.orchestra.run.smithi191.stdout:(86/120): libsepol-devel-2.9-2.el8.x86_64.rpm 4.8 MB/s | 86 kB 00:00 2023-10-10T18:42:48.449 INFO:teuthology.orchestra.run.smithi191.stdout:(87/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.8 MB/s | 58 kB 00:00 2023-10-10T18:42:48.450 INFO:teuthology.orchestra.run.smithi191.stdout:(88/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.9 MB/s | 47 kB 00:00 2023-10-10T18:42:48.468 INFO:teuthology.orchestra.run.smithi191.stdout:(89/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 17 MB/s | 559 kB 00:00 2023-10-10T18:42:48.472 INFO:teuthology.orchestra.run.smithi191.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 3.1 MB/s | 70 kB 00:00 2023-10-10T18:42:48.475 INFO:teuthology.orchestra.run.smithi191.stdout:(91/120): libgfortran-8.4.1-1.el8.x86_64.rpm 25 MB/s | 639 kB 00:00 2023-10-10T18:42:48.486 INFO:teuthology.orchestra.run.smithi191.stdout:(92/120): libselinux-devel-2.9-5.el8.x86_64.rpm 11 MB/s | 200 kB 00:00 2023-10-10T18:42:48.490 INFO:teuthology.orchestra.run.smithi191.stdout:(93/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 5.3 MB/s | 97 kB 00:00 2023-10-10T18:42:48.499 INFO:teuthology.orchestra.run.smithi175.stdout:(98/120): python3-scipy-1.0.0-20.module+el8.1.0 58 MB/s | 15 MB 00:00 2023-10-10T18:42:48.503 INFO:teuthology.orchestra.run.smithi191.stdout:(94/120): python3-prettytable-0.7.2-14.el8.noar 2.5 MB/s | 44 kB 00:00 2023-10-10T18:42:48.509 INFO:teuthology.orchestra.run.smithi191.stdout:(95/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.4 MB/s | 27 kB 00:00 2023-10-10T18:42:48.523 INFO:teuthology.orchestra.run.smithi191.stdout:(96/120): jq-1.5-12.el8.x86_64.rpm 8.1 MB/s | 161 kB 00:00 2023-10-10T18:42:48.524 INFO:teuthology.orchestra.run.smithi175.stdout:(99/120): python3-babel-2.5.1-5.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2023-10-10T18:42:48.525 INFO:teuthology.orchestra.run.smithi175.stdout:(100/120): python3-markupsafe-0.23-19.el8.x86_6 1.5 MB/s | 39 kB 00:00 2023-10-10T18:42:48.529 INFO:teuthology.orchestra.run.smithi191.stdout:(97/120): flex-2.6.1-9.el8.x86_64.rpm 16 MB/s | 320 kB 00:00 2023-10-10T18:42:48.545 INFO:teuthology.orchestra.run.smithi175.stdout:(101/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 13 MB/s | 259 kB 00:00 2023-10-10T18:42:48.549 INFO:teuthology.orchestra.run.smithi191.stdout:(98/120): python3-werkzeug-0.12.2-4.el8.noarch. 17 MB/s | 457 kB 00:00 2023-10-10T18:42:48.553 INFO:teuthology.orchestra.run.smithi175.stdout:(102/120): bison-3.0.4-10.el8.x86_64.rpm 23 MB/s | 688 kB 00:00 2023-10-10T18:42:48.565 INFO:teuthology.orchestra.run.smithi175.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 11 MB/s | 225 kB 00:00 2023-10-10T18:42:48.570 INFO:teuthology.orchestra.run.smithi191.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 1.8 MB/s | 39 kB 00:00 2023-10-10T18:42:48.579 INFO:teuthology.orchestra.run.smithi175.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 4.9 MB/s | 126 kB 00:00 2023-10-10T18:42:48.599 INFO:teuthology.orchestra.run.smithi191.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 24 MB/s | 688 kB 00:00 2023-10-10T18:42:48.600 INFO:teuthology.orchestra.run.smithi175.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 8.7 MB/s | 187 kB 00:00 2023-10-10T18:42:48.616 INFO:teuthology.orchestra.run.smithi191.stdout:(101/120): python3-babel-2.5.1-5.el8.noarch.rpm 55 MB/s | 4.8 MB 00:00 2023-10-10T18:42:48.620 INFO:teuthology.orchestra.run.smithi191.stdout:(102/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2023-10-10T18:42:48.622 INFO:teuthology.orchestra.run.smithi175.stdout:(106/120): python3-numpy-1.14.3-9.el8.x86_64.rp 65 MB/s | 3.7 MB 00:00 2023-10-10T18:42:48.628 INFO:teuthology.orchestra.run.smithi175.stdout:(107/120): python3-jinja2-2.10.1-2.el8_0.noarch 19 MB/s | 538 kB 00:00 2023-10-10T18:42:48.638 INFO:teuthology.orchestra.run.smithi191.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 10 MB/s | 225 kB 00:00 2023-10-10T18:42:48.642 INFO:teuthology.orchestra.run.smithi175.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-10-10T18:42:48.646 INFO:teuthology.orchestra.run.smithi175.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 8.5 MB/s | 157 kB 00:00 2023-10-10T18:42:48.648 INFO:teuthology.orchestra.run.smithi191.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 4.4 MB/s | 126 kB 00:00 2023-10-10T18:42:48.660 INFO:teuthology.orchestra.run.smithi175.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 6.1 MB/s | 110 kB 00:00 2023-10-10T18:42:48.672 INFO:teuthology.orchestra.run.smithi175.stdout:(111/120): librdkafka-0.11.4-1.el8.x86_64.rpm 14 MB/s | 353 kB 00:00 2023-10-10T18:42:48.676 INFO:teuthology.orchestra.run.smithi175.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 1.3 MB/s | 20 kB 00:00 2023-10-10T18:42:48.693 INFO:teuthology.orchestra.run.smithi175.stdout:(113/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.2 MB/s | 54 kB 00:00 2023-10-10T18:42:48.695 INFO:teuthology.orchestra.run.smithi175.stdout:(114/120): libtool-2.4.6-25.el8.x86_64.rpm 29 MB/s | 709 kB 00:00 2023-10-10T18:42:48.746 INFO:teuthology.orchestra.run.smithi191.stdout:(105/120): python3-scipy-1.0.0-20.module+el8.1. 54 MB/s | 15 MB 00:00 2023-10-10T18:42:48.749 INFO:teuthology.orchestra.run.smithi191.stdout:(106/120): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-10-10T18:42:48.769 INFO:teuthology.orchestra.run.smithi191.stdout:(107/120): python3-jinja2-2.10.1-2.el8_0.noarch 23 MB/s | 538 kB 00:00 2023-10-10T18:42:48.770 INFO:teuthology.orchestra.run.smithi175.stdout:(115/120): openblas-threads-0.3.12-1.el8.x86_64 61 MB/s | 4.7 MB 00:00 2023-10-10T18:42:48.773 INFO:teuthology.orchestra.run.smithi191.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 12 MB/s | 302 kB 00:00 2023-10-10T18:42:48.784 INFO:teuthology.orchestra.run.smithi175.stdout:(116/120): openblas-0.3.12-1.el8.x86_64.rpm 52 MB/s | 4.6 MB 00:00 2023-10-10T18:42:48.793 INFO:teuthology.orchestra.run.smithi175.stdout:(117/120): automake-1.16.1-7.el8.noarch.rpm 31 MB/s | 713 kB 00:00 2023-10-10T18:42:48.794 INFO:teuthology.orchestra.run.smithi191.stdout:(109/120): python3-numpy-1.14.3-9.el8.x86_64.rp 24 MB/s | 3.7 MB 00:00 2023-10-10T18:42:48.795 INFO:teuthology.orchestra.run.smithi191.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 4.8 MB/s | 110 kB 00:00 2023-10-10T18:42:48.796 INFO:teuthology.orchestra.run.smithi191.stdout:(111/120): python3-mako-1.0.6-13.el8.noarch.rpm 5.6 MB/s | 157 kB 00:00 2023-10-10T18:42:48.812 INFO:teuthology.orchestra.run.smithi191.stdout:(112/120): librdkafka-0.11.4-1.el8.x86_64.rpm 19 MB/s | 353 kB 00:00 2023-10-10T18:42:48.813 INFO:teuthology.orchestra.run.smithi191.stdout:(113/120): python3-jsonpointer-1.10-11.el8.noar 1.1 MB/s | 20 kB 00:00 2023-10-10T18:42:48.821 INFO:teuthology.orchestra.run.smithi191.stdout:(114/120): libtool-2.4.6-25.el8.x86_64.rpm 29 MB/s | 709 kB 00:00 2023-10-10T18:42:48.831 INFO:teuthology.orchestra.run.smithi191.stdout:(115/120): python3-pytz-2017.2-9.el8.noarch.rpm 2.9 MB/s | 54 kB 00:00 2023-10-10T18:42:48.862 INFO:teuthology.orchestra.run.smithi191.stdout:(116/120): automake-1.16.1-7.el8.noarch.rpm 25 MB/s | 713 kB 00:00 2023-10-10T18:42:48.895 INFO:teuthology.orchestra.run.smithi191.stdout:(117/120): openblas-threads-0.3.12-1.el8.x86_64 58 MB/s | 4.7 MB 00:00 2023-10-10T18:42:48.913 INFO:teuthology.orchestra.run.smithi191.stdout:(118/120): openblas-0.3.12-1.el8.x86_64.rpm 50 MB/s | 4.6 MB 00:00 2023-10-10T18:42:49.351 INFO:teuthology.orchestra.run.smithi191.stdout:(119/120): librbd1-17.2.6-1374.g46d1f59b.el8.x8 7.9 MB/s | 3.6 MB 00:00 2023-10-10T18:42:49.440 INFO:teuthology.orchestra.run.smithi175.stdout:(118/120): librados2-17.2.6-1374.g46d1f59b.el8. 5.3 MB/s | 3.5 MB 00:00 2023-10-10T18:42:49.452 INFO:teuthology.orchestra.run.smithi175.stdout:(119/120): librbd1-17.2.6-1374.g46d1f59b.el8.x8 5.4 MB/s | 3.6 MB 00:00 2023-10-10T18:42:49.533 INFO:teuthology.orchestra.run.smithi191.stdout:(120/120): librados2-17.2.6-1374.g46d1f59b.el8. 5.1 MB/s | 3.5 MB 00:00 2023-10-10T18:42:49.536 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-10-10T18:42:49.537 INFO:teuthology.orchestra.run.smithi191.stdout:Total 18 MB/s | 202 MB 00:11 2023-10-10T18:42:49.582 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-10-10T18:42:49.694 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-10-10T18:42:49.694 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-10-10T18:42:50.626 INFO:teuthology.orchestra.run.smithi175.stdout:(120/120): ceph-test-17.2.6-1374.g46d1f59b.el8. 9.2 MB/s | 52 MB 00:05 2023-10-10T18:42:50.630 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-10-10T18:42:50.631 INFO:teuthology.orchestra.run.smithi175.stdout:Total 16 MB/s | 202 MB 00:12 2023-10-10T18:42:50.683 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-10-10T18:42:50.800 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-10-10T18:42:50.800 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-10-10T18:42:51.465 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-10-10T18:42:51.470 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-10-10T18:42:52.474 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-10-10T18:42:52.479 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-10-10T18:42:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-10-10T18:42:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-10-10T18:42:56.505 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-10-10T18:42:56.630 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-10-10T18:42:56.805 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-10-10T18:42:57.116 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-10-10T18:42:57.187 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-10-10T18:42:57.326 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-10-10T18:42:57.421 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:17.2.6-1374.g46d1f59b.el8.x86_64 4/122 2023-10-10T18:42:57.542 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-10-10T18:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-10-10T18:42:57.824 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:17.2.6-1374.g46d1f59b.el8.x86_64 4/122 2023-10-10T18:42:57.892 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-10-10T18:42:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:17.2.6-1374.g46d1f59b.el8.x86_64 5/122 2023-10-10T18:42:58.159 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:17.2.6-1374.g46d1f59b.el8.x86_64 5/122 2023-10-10T18:42:58.275 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-10-10T18:42:58.321 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:17.2.6-1374.g46d1f59b.el8.x86_64 6/122 2023-10-10T18:42:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:17.2.6-1374.g46d1f59b.el8.x86_64 6/122 2023-10-10T18:42:58.525 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librados2-2:17.2.6-1374.g46d1f59b.el8.x86_64 4/122 2023-10-10T18:42:58.610 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:17.2.6-1374.g46d1f59b.el8.x86_64 7/122 2023-10-10T18:42:58.912 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:17.2.6-1374.g46d1f59b.el8.x86_64 4/122 2023-10-10T18:42:58.926 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-10-10T18:42:59.125 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librbd1-2:17.2.6-1374.g46d1f59b.el8.x86_64 5/122 2023-10-10T18:42:59.293 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:17.2.6-1374.g46d1f59b.el8.x86_64 5/122 2023-10-10T18:42:59.483 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs2-2:17.2.6-1374.g46d1f59b.el8.x86_64 6/122 2023-10-10T18:42:59.562 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs2-2:17.2.6-1374.g46d1f59b.el8.x86_64 6/122 2023-10-10T18:42:59.586 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-10-10T18:42:59.746 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rados-2:17.2.6-1374.g46d1f59b.el8.x86_64 7/122 2023-10-10T18:42:59.771 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:17.2.6-1374.g46d1f59b.el8.x86_64 10/122 2023-10-10T18:42:59.880 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:17.2.6-1374.g46d1f59b.el8.x86_64 10/122 2023-10-10T18:43:00.097 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-10-10T18:43:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-10-10T18:43:00.294 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-10-10T18:43:00.521 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:17.2.6-1374.g46d1f59b.el8.x86_64 13/122 2023-10-10T18:43:00.615 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1374.g46d1f59b.el8.x86_64 13/122 2023-10-10T18:43:00.771 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:17.2.6-1374.g46d1f59b.el8.x86 14/122 2023-10-10T18:43:00.801 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-10-10T18:43:00.845 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1374.g46d1f59b.el8.x86 14/122 2023-10-10T18:43:01.008 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librgw2-2:17.2.6-1374.g46d1f59b.el8.x86_64 10/122 2023-10-10T18:43:01.042 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-10-10T18:43:01.130 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librgw2-2:17.2.6-1374.g46d1f59b.el8.x86_64 10/122 2023-10-10T18:43:01.300 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-10-10T18:43:01.412 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-10-10T18:43:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-10-10T18:43:01.581 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-10-10T18:43:01.714 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-10-10T18:43:01.808 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephsqlite-2:17.2.6-1374.g46d1f59b.el8.x86_64 13/122 2023-10-10T18:43:01.871 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-10T18:43:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1374.g46d1f59b.el8.x86_64 13/122 2023-10-10T18:43:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-10T18:43:02.179 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-10T18:43:02.718 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-10T18:43:02.774 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libradosstriper1-2:17.2.6-1374.g46d1f59b.el8.x86 14/122 2023-10-10T18:43:02.862 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1374.g46d1f59b.el8.x86 14/122 2023-10-10T18:43:02.934 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-10T18:43:03.058 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-10-10T18:43:03.347 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-10-10T18:43:03.460 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-10T18:43:03.564 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-10-10T18:43:03.662 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-10-10T18:43:03.738 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-10-10T18:43:03.866 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-10T18:43:04.037 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-10T18:43:04.190 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-10T18:43:04.418 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-10-10T18:43:04.690 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-10-10T18:43:04.742 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-10T18:43:04.924 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-10T18:43:04.928 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-10-10T18:43:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-10-10T18:43:05.443 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-10T18:43:05.615 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-10-10T18:43:05.957 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:17.2.6-1374.g46d1f59b.el8. 26/122 2023-10-10T18:43:06.123 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:17.2.6-1374.g46d1f59b.el 27/122 2023-10-10T18:43:06.329 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-10-10T18:43:06.430 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:17.2.6-1374.g46d1f59b.el8.x86_6 28/122 2023-10-10T18:43:06.619 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-10-10T18:43:06.671 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-10-10T18:43:06.892 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-10-10T18:43:06.969 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-10-10T18:43:07.177 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-10-10T18:43:07.211 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-10T18:43:07.338 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-common-2:17.2.6-1374.g46d1f59b.el8. 26/122 2023-10-10T18:43:07.517 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-argparse-2:17.2.6-1374.g46d1f59b.el 27/122 2023-10-10T18:43:07.774 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cephfs-2:17.2.6-1374.g46d1f59b.el8.x86_6 28/122 2023-10-10T18:43:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-10-10T18:43:08.552 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-10-10T18:43:08.806 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-10T18:43:09.299 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-10T18:43:09.668 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-10-10T18:43:10.719 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-10-10T18:43:10.878 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-10T18:43:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-10-10T18:43:11.246 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-10-10T18:43:11.380 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-10-10T18:43:11.657 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el8.noarch 36/122 2023-10-10T18:43:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-10-10T18:43:12.079 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 38/122 2023-10-10T18:43:12.241 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 39/122 2023-10-10T18:43:12.316 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-10-10T18:43:12.530 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:17.2.6-1374.g46d1f59b.el8.x86_6 40/122 2023-10-10T18:43:12.660 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-10-10T18:43:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-10-10T18:43:12.910 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-10-10T18:43:12.984 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-10-10T18:43:13.154 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rsa-4.9-2.el8.noarch 36/122 2023-10-10T18:43:13.205 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-10-10T18:43:13.390 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-10-10T18:43:13.406 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-10-10T18:43:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 38/122 2023-10-10T18:43:13.611 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-10T18:43:13.690 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-10T18:43:13.786 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 39/122 2023-10-10T18:43:13.869 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-10-10T18:43:14.055 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-10-10T18:43:14.068 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librados-devel-2:17.2.6-1374.g46d1f59b.el8.x86_6 40/122 2023-10-10T18:43:14.212 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-10-10T18:43:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-10-10T18:43:14.414 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-10-10T18:43:14.513 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-10-10T18:43:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-10-10T18:43:14.701 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-10-10T18:43:14.805 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-10-10T18:43:14.934 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-10-10T18:43:14.940 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-10-10T18:43:15.130 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-10T18:43:15.147 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-10-10T18:43:15.211 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-10T18:43:15.348 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-10-10T18:43:15.380 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-10-10T18:43:15.574 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-10-10T18:43:15.638 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-10-10T18:43:15.756 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-10-10T18:43:15.896 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-10-10T18:43:15.951 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-10-10T18:43:16.098 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-10-10T18:43:16.205 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-10-10T18:43:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-10-10T18:43:16.409 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-10-10T18:43:16.484 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-10-10T18:43:16.639 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-10-10T18:43:16.643 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-10-10T18:43:16.792 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-10-10T18:43:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-10-10T18:43:17.064 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-10-10T18:43:17.143 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-10-10T18:43:17.310 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-10-10T18:43:17.436 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-10-10T18:43:17.509 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-10-10T18:43:17.690 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-10-10T18:43:17.761 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-10-10T18:43:17.848 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-10-10T18:43:18.041 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-10-10T18:43:18.054 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-10-10T18:43:18.221 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-10-10T18:43:18.293 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-10-10T18:43:18.418 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-10-10T18:43:18.533 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-10-10T18:43:18.675 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-10-10T18:43:18.739 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-10-10T18:43:18.905 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-10-10T18:43:18.917 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-10-10T18:43:19.112 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-10-10T18:43:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-10-10T18:43:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-10-10T18:43:19.407 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-10-10T18:43:19.466 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-10-10T18:43:19.600 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-10-10T18:43:19.664 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-10-10T18:43:19.901 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-10-10T18:43:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-10-10T18:43:20.090 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-10-10T18:43:20.193 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-10-10T18:43:20.257 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-10-10T18:43:20.371 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-10-10T18:43:20.402 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-10-10T18:43:20.640 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-10-10T18:43:20.657 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-10-10T18:43:20.899 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-10-10T18:43:21.110 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-10-10T18:43:21.399 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-10-10T18:43:21.686 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-10-10T18:43:21.799 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-10-10T18:43:21.861 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-10-10T18:43:22.089 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-10-10T18:43:22.126 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-10-10T18:43:22.335 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-10-10T18:43:22.704 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-10-10T18:43:23.003 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-10-10T18:43:23.299 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-10-10T18:43:23.315 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-10-10T18:43:23.565 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-10-10T18:43:23.618 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-10-10T18:43:23.742 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-10-10T18:43:23.889 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-10-10T18:43:23.976 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-10-10T18:43:24.275 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-10-10T18:43:24.548 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-10-10T18:43:24.869 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-10-10T18:43:25.085 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-10-10T18:43:25.295 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-10-10T18:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-10-10T18:43:28.978 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 87/122 2023-10-10T18:43:29.208 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 87/122 2023-10-10T18:43:29.787 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 87/122 2023-10-10T18:43:30.085 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:17.2.6-1374.g46d1f59b.el8.x86_64 88/122 2023-10-10T18:43:30.172 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:17.2.6-1374.g46d1f59b.el8.x86_64 88/122 2023-10-10T18:43:30.354 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:17.2.6-1374.g46d1f59b.el8.x86_64 89/122 2023-10-10T18:43:30.664 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 87/122 2023-10-10T18:43:30.928 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 87/122 2023-10-10T18:43:31.521 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 87/122 2023-10-10T18:43:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-base-2:17.2.6-1374.g46d1f59b.el8.x86_64 88/122 2023-10-10T18:43:31.946 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-base-2:17.2.6-1374.g46d1f59b.el8.x86_64 88/122 2023-10-10T18:43:32.122 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-selinux-2:17.2.6-1374.g46d1f59b.el8.x86_64 89/122 2023-10-10T18:43:47.915 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1374.g46d1f59b.el8.x86_64 89/122 2023-10-10T18:43:47.915 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2023-10-10T18:43:47.915 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2023-10-10T18:43:47.915 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2023-10-10T18:43:47.916 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /run 2023-10-10T18:43:47.916 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2023-10-10T18:43:47.916 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2023-10-10T18:43:47.916 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2023-10-10T18:43:47.916 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2023-10-10T18:43:47.917 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2023-10-10T18:43:47.917 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:43:48.305 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:17.2.6-1374.g46d1f59b.el8.noarch 90/122 2023-10-10T18:43:49.790 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:17.2.6-1374.g46d1f59b.el8.noarch 90/122 2023-10-10T18:43:49.882 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1374.g46d1f59b.el8.x86_64 89/122 2023-10-10T18:43:49.882 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /sys 2023-10-10T18:43:49.883 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /proc 2023-10-10T18:43:49.883 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /dev 2023-10-10T18:43:49.883 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /run 2023-10-10T18:43:49.883 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /mnt 2023-10-10T18:43:49.884 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /var/tmp 2023-10-10T18:43:49.884 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /home 2023-10-10T18:43:49.884 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /tmp 2023-10-10T18:43:49.884 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /dev 2023-10-10T18:43:49.884 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:43:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:17.2.6-1374.g46d1f59b.el8.x86_64 91/122 2023-10-10T18:43:50.201 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:17.2.6-1374.g46d1f59b.el8.x86_64 91/122 2023-10-10T18:43:50.291 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-volume-2:17.2.6-1374.g46d1f59b.el8.noarch 90/122 2023-10-10T18:43:50.346 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:17.2.6-1374.g46d1f59b.el8.x86_64 92/122 2023-10-10T18:43:50.735 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:17.2.6-1374.g46d1f59b.el8.x86_64 92/122 2023-10-10T18:43:50.896 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:17.2.6-1374.g46d1f59b.el8.x86_64 93/122 2023-10-10T18:43:50.912 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:17.2.6-1374.g46d1f59b.el8.x86_64 93/122 2023-10-10T18:43:51.775 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-volume-2:17.2.6-1374.g46d1f59b.el8.noarch 90/122 2023-10-10T18:43:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-osd-2:17.2.6-1374.g46d1f59b.el8.x86_64 91/122 2023-10-10T18:43:52.189 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-osd-2:17.2.6-1374.g46d1f59b.el8.x86_64 91/122 2023-10-10T18:43:52.324 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mds-2:17.2.6-1374.g46d1f59b.el8.x86_64 92/122 2023-10-10T18:43:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mds-2:17.2.6-1374.g46d1f59b.el8.x86_64 92/122 2023-10-10T18:43:52.882 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mon-2:17.2.6-1374.g46d1f59b.el8.x86_64 93/122 2023-10-10T18:43:52.899 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mon-2:17.2.6-1374.g46d1f59b.el8.x86_64 93/122 2023-10-10T18:43:53.683 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 94/122 2023-10-10T18:43:53.875 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 94/122 2023-10-10T18:43:54.059 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1374.g46d1f59b.e 95/122 2023-10-10T18:43:54.296 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1374.g46d1f59b. 96/122 2023-10-10T18:43:54.479 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1374.g46d1f59b.el8.noa 97/122 2023-10-10T18:43:54.691 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1374.g46d1f59b.el8.noa 97/122 2023-10-10T18:43:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1374.g46d1f59b.el8.n 98/122 2023-10-10T18:43:55.850 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 94/122 2023-10-10T18:43:55.858 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1374.g46d1f59b.el8.n 98/122 2023-10-10T18:43:55.996 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 94/122 2023-10-10T18:43:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1374.g46d 99/122 2023-10-10T18:43:56.115 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1374.g46d 99/122 2023-10-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1374.g46d1f59b.e 95/122 2023-10-10T18:43:56.429 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1374.g46d1f59b. 96/122 2023-10-10T18:43:56.543 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1374.g46d1f59b.el 100/122 2023-10-10T18:43:56.614 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1374.g46d1f59b.el8.noa 97/122 2023-10-10T18:43:56.747 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:17.2.6-1374.g46d1f59b.el8.x86_64 101/122 2023-10-10T18:43:56.783 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1374.g46d1f59b.el8.x86_64 101/122 2023-10-10T18:43:56.832 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1374.g46d1f59b.el8.noa 97/122 2023-10-10T18:43:56.944 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:17.2.6-1374.g46d1f59b.el8.noarch 102/122 2023-10-10T18:43:56.955 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1374.g46d1f59b.el8.noarch 102/122 2023-10-10T18:43:57.050 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1374.g46d1f59b.el8.n 98/122 2023-10-10T18:43:58.020 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1374.g46d1f59b.el8.n 98/122 2023-10-10T18:43:58.213 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1374.g46d 99/122 2023-10-10T18:43:58.312 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1374.g46d 99/122 2023-10-10T18:43:58.547 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:17.2.6-1374.g46d1f59b.el8.x86_64 103/122 2023-10-10T18:43:58.740 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1374.g46d1f59b.el 100/122 2023-10-10T18:43:58.797 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 104/122 2023-10-10T18:43:58.886 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 104/122 2023-10-10T18:43:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-2:17.2.6-1374.g46d1f59b.el8.x86_64 101/122 2023-10-10T18:43:59.004 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1374.g46d1f59b.el8.x86_64 101/122 2023-10-10T18:43:59.036 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1374.g46d1f 105/122 2023-10-10T18:43:59.078 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1374.g46d1f 105/122 2023-10-10T18:43:59.155 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-rook-2:17.2.6-1374.g46d1f59b.el8.noarch 102/122 2023-10-10T18:43:59.167 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1374.g46d1f59b.el8.noarch 102/122 2023-10-10T18:43:59.284 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 106/122 2023-10-10T18:43:59.629 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 106/122 2023-10-10T18:43:59.803 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 107/122 2023-10-10T18:44:00.804 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-2:17.2.6-1374.g46d1f59b.el8.x86_64 103/122 2023-10-10T18:44:01.025 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-radosgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 104/122 2023-10-10T18:44:01.139 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 104/122 2023-10-10T18:44:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1374.g46d1f 105/122 2023-10-10T18:44:01.339 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1374.g46d1f 105/122 2023-10-10T18:44:01.537 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephfs-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 106/122 2023-10-10T18:44:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 106/122 2023-10-10T18:44:02.047 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 107/122 2023-10-10T18:44:04.701 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 107/122 2023-10-10T18:44:05.652 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:17.2.6-1374.g46d1f59b.el8.x86_64 108/122 2023-10-10T18:44:06.037 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-10-10T18:44:06.213 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 110/122 2023-10-10T18:44:06.475 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-10-10T18:44:06.634 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-10-10T18:44:06.645 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-10-10T18:44:06.814 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:17.2.6-1374.g46d1f59b.el8.x86_ 113/122 2023-10-10T18:44:06.972 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-10-10T18:44:06.979 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 107/122 2023-10-10T18:44:07.176 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:17.2.6-1374.g46d1f59b.el8.noarch 115/122 2023-10-10T18:44:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 116/122 2023-10-10T18:44:07.655 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 117/122 2023-10-10T18:44:07.859 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-10-10T18:44:07.911 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-10-10T18:44:07.948 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-test-2:17.2.6-1374.g46d1f59b.el8.x86_64 108/122 2023-10-10T18:44:08.109 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-10-10T18:44:08.118 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-10-10T18:44:08.228 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-10-10T18:44:08.228 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-10T18:44:08.354 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-10-10T18:44:08.470 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-10T18:44:08.470 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-10T18:44:08.565 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 110/122 2023-10-10T18:44:08.809 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-10-10T18:44:09.003 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-10-10T18:44:09.015 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-10-10T18:44:09.183 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs-devel-2:17.2.6-1374.g46d1f59b.el8.x86_ 113/122 2023-10-10T18:44:09.350 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-10-10T18:44:09.354 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-10T18:44:09.354 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:17.2.6-1374.g46d1f59b.el8.x86_64 1/122 2023-10-10T18:44:09.354 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:17.2.6-1374.g46d1f59b.el8.x86_64 2/122 2023-10-10T18:44:09.355 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 3/122 2023-10-10T18:44:09.355 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 4/122 2023-10-10T18:44:09.355 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1374.g46d1f 5/122 2023-10-10T18:44:09.355 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:17.2.6-1374.g46d1f59b.el8.x86_64 6/122 2023-10-10T18:44:09.355 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:17.2.6-1374.g46d1f59b.el8.x86_64 7/122 2023-10-10T18:44:09.356 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:17.2.6-1374.g46d1f59b.el8.x86_64 8/122 2023-10-10T18:44:09.356 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:17.2.6-1374.g46d1f59b.el8.x86_64 9/122 2023-10-10T18:44:09.356 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 10/122 2023-10-10T18:44:09.357 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:17.2.6-1374.g46d1f59b.el8.x86_64 11/122 2023-10-10T18:44:09.357 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:17.2.6-1374.g46d1f59b.el8.x86_64 12/122 2023-10-10T18:44:09.357 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 13/122 2023-10-10T18:44:09.357 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:17.2.6-1374.g46d1f59b.el8.x86_ 14/122 2023-10-10T18:44:09.357 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:17.2.6-1374.g46d1f59b.el8.x86_64 15/122 2023-10-10T18:44:09.358 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:17.2.6-1374.g46d1f59b.el8.x86_64 16/122 2023-10-10T18:44:09.358 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:17.2.6-1374.g46d1f59b.el8.x86_6 17/122 2023-10-10T18:44:09.358 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:17.2.6-1374.g46d1f59b.el8.x86 18/122 2023-10-10T18:44:09.358 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:17.2.6-1374.g46d1f59b.el8.x86_64 19/122 2023-10-10T18:44:09.358 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1374.g46d1f59b.el 20/122 2023-10-10T18:44:09.359 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:17.2.6-1374.g46d1f59b.el8. 21/122 2023-10-10T18:44:09.359 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:17.2.6-1374.g46d1f59b.el8.x86_6 22/122 2023-10-10T18:44:09.359 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:17.2.6-1374.g46d1f59b.el8.x86_64 23/122 2023-10-10T18:44:09.359 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 24/122 2023-10-10T18:44:09.359 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 25/122 2023-10-10T18:44:09.360 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 26/122 2023-10-10T18:44:09.360 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 27/122 2023-10-10T18:44:09.360 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 28/122 2023-10-10T18:44:09.360 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1374.g46d1f59b. 29/122 2023-10-10T18:44:09.360 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1374.g46d1f59b.el8.noa 30/122 2023-10-10T18:44:09.368 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1374.g46d1f59b.el8.n 31/122 2023-10-10T18:44:09.369 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1374.g46d 32/122 2023-10-10T18:44:09.369 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1374.g46d1f59b.el 33/122 2023-10-10T18:44:09.369 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1374.g46d1f59b.el8.noarch 34/122 2023-10-10T18:44:09.369 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1374.g46d1f59b.e 35/122 2023-10-10T18:44:09.370 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:17.2.6-1374.g46d1f59b.el8.noarch 36/122 2023-10-10T18:44:09.370 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 37/122 2023-10-10T18:44:09.370 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:17.2.6-1374.g46d1f59b.el8.noarch 38/122 2023-10-10T18:44:09.370 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-10-10T18:44:09.370 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-10-10T18:44:09.371 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-10-10T18:44:09.371 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-10-10T18:44:09.371 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-10-10T18:44:09.371 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-10-10T18:44:09.371 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-10-10T18:44:09.372 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-10-10T18:44:09.372 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-10-10T18:44:09.372 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-10-10T18:44:09.372 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-10-10T18:44:09.372 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-10-10T18:44:09.373 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-10-10T18:44:09.373 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-10-10T18:44:09.373 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-10-10T18:44:09.373 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-10-10T18:44:09.373 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-10-10T18:44:09.374 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-10-10T18:44:09.374 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-10-10T18:44:09.374 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-10-10T18:44:09.374 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-10-10T18:44:09.374 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/122 2023-10-10T18:44:09.375 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-10-10T18:44:09.375 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-10-10T18:44:09.375 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-10-10T18:44:09.375 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-10-10T18:44:09.375 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-10-10T18:44:09.376 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-10-10T18:44:09.376 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-10-10T18:44:09.376 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-10-10T18:44:09.376 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-10-10T18:44:09.376 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-10-10T18:44:09.377 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-10-10T18:44:09.377 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-10-10T18:44:09.377 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-10-10T18:44:09.377 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-10-10T18:44:09.377 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-10-10T18:44:09.378 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-10-10T18:44:09.378 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-10-10T18:44:09.378 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-10-10T18:44:09.378 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-10-10T18:44:09.378 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-10-10T18:44:09.379 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-10-10T18:44:09.379 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-10-10T18:44:09.379 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-10-10T18:44:09.379 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-10-10T18:44:09.379 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-10-10T18:44:09.379 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-10-10T18:44:09.380 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-10-10T18:44:09.380 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-10-10T18:44:09.380 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-10-10T18:44:09.380 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-10-10T18:44:09.380 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-10-10T18:44:09.381 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-10-10T18:44:09.381 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-10-10T18:44:09.381 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-10-10T18:44:09.381 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-10-10T18:44:09.381 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-10-10T18:44:09.382 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-10-10T18:44:09.382 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-10-10T18:44:09.382 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-10-10T18:44:09.382 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-10-10T18:44:09.382 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-10-10T18:44:09.383 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-10-10T18:44:09.383 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-10-10T18:44:09.383 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-10-10T18:44:09.383 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-10-10T18:44:09.383 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-10-10T18:44:09.384 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-10-10T18:44:09.384 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-10-10T18:44:09.384 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-10-10T18:44:09.384 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-10-10T18:44:09.384 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-10-10T18:44:09.385 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-10-10T18:44:09.385 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-10-10T18:44:09.385 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-10-10T18:44:09.385 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-10-10T18:44:09.386 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-10-10T18:44:09.386 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-10-10T18:44:09.386 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-10-10T18:44:09.386 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:17.2.6-1374.g46d1f59b.el8.x86_64 119/122 2023-10-10T18:44:09.386 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-10-10T18:44:09.387 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:17.2.6-1374.g46d1f59b.el8.x86_64 121/122 2023-10-10T18:44:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephfs-top-2:17.2.6-1374.g46d1f59b.el8.noarch 115/122 2023-10-10T18:44:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 116/122 2023-10-10T18:44:10.016 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-nbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 117/122 2023-10-10T18:44:10.186 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-10-10T18:44:10.241 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-10-10T18:44:10.445 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-10-10T18:44:10.454 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-10-10T18:44:10.581 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-10-10T18:44:10.581 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-10T18:44:10.823 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-10T18:44:10.823 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-10T18:44:11.641 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-10-10T18:44:11.641 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Oct 2023 06:42:30 PM UTC. 2023-10-10T18:44:11.750 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-10T18:44:11.750 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-2:17.2.6-1374.g46d1f59b.el8.x86_64 1/122 2023-10-10T18:44:11.751 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-base-2:17.2.6-1374.g46d1f59b.el8.x86_64 2/122 2023-10-10T18:44:11.751 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 3/122 2023-10-10T18:44:11.751 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 4/122 2023-10-10T18:44:11.751 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1374.g46d1f 5/122 2023-10-10T18:44:11.751 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mds-2:17.2.6-1374.g46d1f59b.el8.x86_64 6/122 2023-10-10T18:44:11.761 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-2:17.2.6-1374.g46d1f59b.el8.x86_64 7/122 2023-10-10T18:44:11.762 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mon-2:17.2.6-1374.g46d1f59b.el8.x86_64 8/122 2023-10-10T18:44:11.762 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-osd-2:17.2.6-1374.g46d1f59b.el8.x86_64 9/122 2023-10-10T18:44:11.762 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-radosgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 10/122 2023-10-10T18:44:11.762 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-selinux-2:17.2.6-1374.g46d1f59b.el8.x86_64 11/122 2023-10-10T18:44:11.763 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-test-2:17.2.6-1374.g46d1f59b.el8.x86_64 12/122 2023-10-10T18:44:11.763 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephfs-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 13/122 2023-10-10T18:44:11.763 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs-devel-2:17.2.6-1374.g46d1f59b.el8.x86_ 14/122 2023-10-10T18:44:11.763 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs2-2:17.2.6-1374.g46d1f59b.el8.x86_64 15/122 2023-10-10T18:44:11.763 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephsqlite-2:17.2.6-1374.g46d1f59b.el8.x86_64 16/122 2023-10-10T18:44:11.764 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados-devel-2:17.2.6-1374.g46d1f59b.el8.x86_6 17/122 2023-10-10T18:44:11.764 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libradosstriper1-2:17.2.6-1374.g46d1f59b.el8.x86 18/122 2023-10-10T18:44:11.764 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librgw2-2:17.2.6-1374.g46d1f59b.el8.x86_64 19/122 2023-10-10T18:44:11.764 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1374.g46d1f59b.el 20/122 2023-10-10T18:44:11.764 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-common-2:17.2.6-1374.g46d1f59b.el8. 21/122 2023-10-10T18:44:11.765 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cephfs-2:17.2.6-1374.g46d1f59b.el8.x86_6 22/122 2023-10-10T18:44:11.765 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rados-2:17.2.6-1374.g46d1f59b.el8.x86_64 23/122 2023-10-10T18:44:11.765 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 24/122 2023-10-10T18:44:11.765 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 25/122 2023-10-10T18:44:11.765 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 26/122 2023-10-10T18:44:11.766 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 27/122 2023-10-10T18:44:11.766 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-nbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 28/122 2023-10-10T18:44:11.766 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1374.g46d1f59b. 29/122 2023-10-10T18:44:11.766 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1374.g46d1f59b.el8.noa 30/122 2023-10-10T18:44:11.766 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1374.g46d1f59b.el8.n 31/122 2023-10-10T18:44:11.766 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1374.g46d 32/122 2023-10-10T18:44:11.767 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1374.g46d1f59b.el 33/122 2023-10-10T18:44:11.767 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1374.g46d1f59b.el8.noarch 34/122 2023-10-10T18:44:11.767 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1374.g46d1f59b.e 35/122 2023-10-10T18:44:11.767 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-volume-2:17.2.6-1374.g46d1f59b.el8.noarch 36/122 2023-10-10T18:44:11.767 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 37/122 2023-10-10T18:44:11.768 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephfs-top-2:17.2.6-1374.g46d1f59b.el8.noarch 38/122 2023-10-10T18:44:11.768 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-10-10T18:44:11.768 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-10-10T18:44:11.768 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-10-10T18:44:11.768 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-10-10T18:44:11.769 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-10-10T18:44:11.769 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-10-10T18:44:11.769 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-10-10T18:44:11.769 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-10-10T18:44:11.769 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-10-10T18:44:11.770 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-10-10T18:44:11.770 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-10-10T18:44:11.770 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-10-10T18:44:11.770 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-10-10T18:44:11.770 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-10-10T18:44:11.771 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-10-10T18:44:11.771 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-10-10T18:44:11.771 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-10-10T18:44:11.771 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-10-10T18:44:11.771 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-10-10T18:44:11.772 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-10-10T18:44:11.772 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-10-10T18:44:11.772 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/122 2023-10-10T18:44:11.772 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-10-10T18:44:11.772 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-10-10T18:44:11.772 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-10-10T18:44:11.773 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-10-10T18:44:11.773 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-10-10T18:44:11.773 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-10-10T18:44:11.773 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-10-10T18:44:11.773 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-10-10T18:44:11.774 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-10-10T18:44:11.774 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-10-10T18:44:11.774 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-10-10T18:44:11.774 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-10-10T18:44:11.774 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-10-10T18:44:11.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-10-10T18:44:11.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-10-10T18:44:11.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-10-10T18:44:11.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-10-10T18:44:11.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-10-10T18:44:11.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-10-10T18:44:11.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-10-10T18:44:11.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-10-10T18:44:11.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-10-10T18:44:11.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-10-10T18:44:11.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-10-10T18:44:11.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-10-10T18:44:11.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-10-10T18:44:11.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-10-10T18:44:11.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-10-10T18:44:11.778 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-10-10T18:44:11.778 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-10-10T18:44:11.778 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-10-10T18:44:11.778 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-10-10T18:44:11.778 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-10-10T18:44:11.779 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-10-10T18:44:11.779 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-10-10T18:44:11.779 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-10-10T18:44:11.779 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-10-10T18:44:11.781 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-10-10T18:44:11.781 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-10-10T18:44:11.781 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-10-10T18:44:11.782 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-10-10T18:44:11.782 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-10-10T18:44:11.782 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-10-10T18:44:11.782 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-10-10T18:44:11.782 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-10-10T18:44:11.783 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-10-10T18:44:11.783 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-10-10T18:44:11.783 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-10-10T18:44:11.783 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-10-10T18:44:11.783 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-10-10T18:44:11.783 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-10-10T18:44:11.784 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-10-10T18:44:11.784 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-10-10T18:44:11.784 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-10-10T18:44:11.784 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-10-10T18:44:11.784 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-10-10T18:44:11.785 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-10-10T18:44:11.785 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-10-10T18:44:11.785 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:17.2.6-1374.g46d1f59b.el8.x86_64 119/122 2023-10-10T18:44:11.785 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-10-10T18:44:11.785 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:17.2.6-1374.g46d1f59b.el8.x86_64 121/122 2023-10-10T18:44:13.738 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2023-10-10T18:44:13.878 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:13.879 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2023-10-10T18:44:13.879 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.879 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.879 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-10-10T18:44:13.880 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.1-7.el8.noarch 2023-10-10T18:44:13.880 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.0.4-10.el8.x86_64 2023-10-10T18:44:13.880 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.880 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.880 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.881 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.881 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.881 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.881 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.881 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.882 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.882 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.882 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.882 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.882 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.884 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.884 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.884 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.884 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.884 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:13.885 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-10-10T18:44:13.885 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.1-9.el8.x86_64 2023-10-10T18:44:13.885 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.7-12.el8.x86_64 2023-10-10T18:44:13.885 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-10T18:44:13.885 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.5-12.el8.x86_64 2023-10-10T18:44:13.886 INFO:teuthology.orchestra.run.smithi191.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-10-10T18:44:13.886 INFO:teuthology.orchestra.run.smithi191.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-10-10T18:44:13.886 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-10T18:44:13.886 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.886 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.887 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.887 INFO:teuthology.orchestra.run.smithi191.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-10-10T18:44:13.887 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-10T18:44:13.887 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-10T18:44:13.887 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-10T18:44:13.888 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-10T18:44:13.888 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.888 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.888 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-10T18:44:13.888 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.889 INFO:teuthology.orchestra.run.smithi191.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-10T18:44:13.889 INFO:teuthology.orchestra.run.smithi191.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-10-10T18:44:13.889 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-10T18:44:13.889 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-10T18:44:13.889 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-10-10T18:44:13.889 INFO:teuthology.orchestra.run.smithi191.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-10T18:44:13.890 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-10T18:44:13.890 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-10T18:44:13.890 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-10T18:44:13.890 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-10T18:44:13.890 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-10-10T18:44:13.891 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-10T18:44:13.891 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-10T18:44:13.891 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-10T18:44:13.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-10T18:44:13.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-10T18:44:13.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-10T18:44:13.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-10T18:44:13.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-10T18:44:13.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-10T18:44:13.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-10T18:44:13.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-10T18:44:13.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-10T18:44:13.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-10T18:44:13.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-10T18:44:13.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-10T18:44:13.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-10T18:44:13.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-10T18:44:13.895 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-10T18:44:13.895 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-10T18:44:13.895 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-10T18:44:13.895 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-10T18:44:13.895 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-10T18:44:13.896 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-10T18:44:13.896 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-10T18:44:13.896 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-10T18:44:13.896 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-10T18:44:13.896 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-10T18:44:13.897 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-10T18:44:13.897 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-2.6-1.el8.noarch 2023-10-10T18:44:13.897 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-10T18:44:13.897 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-10T18:44:13.897 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-10T18:44:13.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-10T18:44:13.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-10T18:44:13.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-10T18:44:13.899 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.899 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-10T18:44:13.899 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-10T18:44:13.899 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-10T18:44:13.899 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-10T18:44:13.900 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-10T18:44:13.900 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-10T18:44:13.900 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-10T18:44:13.900 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-10T18:44:13.900 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-10T18:44:13.900 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-10T18:44:13.901 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-10T18:44:13.901 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-10T18:44:13.901 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-10T18:44:13.901 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.901 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.902 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:13.902 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-10T18:44:13.902 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-10T18:44:13.902 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-10-10T18:44:13.902 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-10T18:44:13.903 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-10T18:44:13.903 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:13.903 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-10-10T18:44:13.980 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-10-10T18:44:13.980 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Oct 2023 06:42:29 PM UTC. 2023-10-10T18:44:14.191 DEBUG:teuthology.parallel:result is None 2023-10-10T18:44:16.073 INFO:teuthology.orchestra.run.smithi175.stdout:Installed products updated. 2023-10-10T18:44:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:16.235 INFO:teuthology.orchestra.run.smithi175.stdout:Upgraded: 2023-10-10T18:44:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: librados2-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.236 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-10-10T18:44:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: automake-1.16.1-7.el8.noarch 2023-10-10T18:44:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: bison-3.0.4-10.el8.x86_64 2023-10-10T18:44:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-top-2:17.2.6-1374.g46d1f59b.el8.noarch 2023-10-10T18:44:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-10-10T18:44:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: flex-2.6.1-9.el8.x86_64 2023-10-10T18:44:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: fuse-2.9.7-12.el8.x86_64 2023-10-10T18:44:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-10T18:44:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: jq-1.5-12.el8.x86_64 2023-10-10T18:44:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-10-10T18:44:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-10-10T18:44:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-10T18:44:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-10-10T18:44:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-10T18:44:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-10T18:44:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-10T18:44:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-10T18:44:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-10T18:44:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-10T18:44:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-10-10T18:44:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-10T18:44:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-10T18:44:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-10-10T18:44:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-10T18:44:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-10T18:44:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-10T18:44:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-10T18:44:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-10T18:44:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-10-10T18:44:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-10T18:44:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-10T18:44:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-10T18:44:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-10T18:44:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-10T18:44:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-10T18:44:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-10T18:44:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-10T18:44:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-10T18:44:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-10T18:44:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-10T18:44:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-10T18:44:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-10T18:44:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-10T18:44:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-10T18:44:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-10T18:44:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-10T18:44:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-10T18:44:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-10T18:44:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-10T18:44:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-10T18:44:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-10T18:44:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-10T18:44:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-10T18:44:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-10T18:44:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-10T18:44:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-10T18:44:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-10T18:44:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend-2.6-1.el8.noarch 2023-10-10T18:44:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-10T18:44:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-10T18:44:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-10T18:44:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-10T18:44:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-10T18:44:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-10T18:44:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-10T18:44:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-10T18:44:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-10T18:44:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-10T18:44:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-10T18:44:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-10T18:44:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-10T18:44:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-10T18:44:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-10T18:44:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-10T18:44:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-10T18:44:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-10T18:44:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-10T18:44:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd-2:17.2.6-1374.g46d1f59b.el8.x86_64 2023-10-10T18:44:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-10T18:44:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-10T18:44:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-10-10T18:44:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-10T18:44:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-10T18:44:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:16.262 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-10-10T18:44:16.550 DEBUG:teuthology.parallel:result is None 2023-10-10T18:44:16.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 2023-10-10T18:44:16.691 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-10T18:44:16.711 INFO:teuthology.orchestra.run.smithi175.stdout:17.2.6-1374.g46d1f59b.el8 2023-10-10T18:44:16.712 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1374.g46d1f59b.el8 2023-10-10T18:44:16.712 INFO:teuthology.task.install:The correct ceph version 17.2.6-1374.g46d1f59b is installed. 2023-10-10T18:44:16.713 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 2023-10-10T18:44:16.854 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-10T18:44:16.878 INFO:teuthology.orchestra.run.smithi191.stdout:17.2.6-1374.g46d1f59b.el8 2023-10-10T18:44:16.879 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1374.g46d1f59b.el8 2023-10-10T18:44:16.879 INFO:teuthology.task.install:The correct ceph version 17.2.6-1374.g46d1f59b is installed. 2023-10-10T18:44:16.880 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-10T18:44:16.880 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:16.881 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-10T18:44:16.918 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:16.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-10T18:44:16.960 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-10T18:44:16.961 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:16.961 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/daemon-helper 2023-10-10T18:44:16.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-10T18:44:17.052 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:17.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2023-10-10T18:44:17.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-10T18:44:17.147 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-10T18:44:17.148 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:17.148 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-10T18:44:17.173 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-10T18:44:17.238 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:17.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-10T18:44:17.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-10T18:44:17.329 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-10T18:44:17.329 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:17.330 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/stdin-killer 2023-10-10T18:44:17.354 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-10T18:44:17.418 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:17.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2023-10-10T18:44:17.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-10T18:44:17.508 INFO:teuthology.run_tasks:Running task ceph... 2023-10-10T18:44:17.576 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-10T18:44:17.576 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 777 /var/log/ceph 2023-10-10T18:44:17.579 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2023-10-10T18:44:17.603 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-10T18:44:17.604 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-10T18:44:17.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-10T18:44:17.670 INFO:tasks.ceph:Creating extra log directories... 2023-10-10T18:44:17.671 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-10T18:44:17.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-10T18:44:17.743 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-10T18:44:17.743 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-10T18:44:17.744 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-10T18:44:17.744 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:17.744 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-10T18:44:17.786 DEBUG:teuthology.orchestra.run.smithi175:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-10T18:44:17.897 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-10T18:44:17.923 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:17.924 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-10T18:44:17.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-10T18:44:18.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-10T18:44:18.096 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-10T18:44:18.096 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-10T18:44:18.096 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370', 'branch': 'wip-yuri3-testing-2023-10-10-0720-quincy', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7420370', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-10-10-0720-quincy'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2023-10-10-0720-quincy', 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 2048, 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-yuri3-testing-2023-10-10-0720-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7', 'targets': {'smithi175.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcuCFH2aQPp6AqsxooAWa9L+2D7lkIlT3KYhmELk9TKiw8Yutr9DYrlPL+/hGDwU2wBsKtoqN/WB7RivMb1/j0=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKAmsrXlRHhVO40XzsNzCCvWbKIzgovODvyFDIx1r5u6lgIzDMu/D40byN68X9lvtywNnIV6lA/+vo2O0KVoaWw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'sha1': '46d1f59b549730ba7f93d940a3d32b94c45a4cf7', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-10_18:16:52', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-10T18:44:18.097 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-10T18:44:18.100 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-10T18:44:18.144 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/run/ceph 2023-10-10T18:44:18.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2023-10-10T18:44:18.208 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:18.209 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/scratch_devs of=/dev/stdout 2023-10-10T18:44:18.232 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-10T18:44:18.233 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_1 2023-10-10T18:44:18.287 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-10T18:44:18.287 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T18:44:18.288 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 221375 Links: 1 2023-10-10T18:44:18.288 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:44:18.288 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T18:44:18.288 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2023-10-10 18:44:11.328672157 +0000 2023-10-10T18:44:18.288 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-10-10 18:41:45.438174029 +0000 2023-10-10T18:44:18.289 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-10-10 18:41:45.438174029 +0000 2023-10-10T18:44:18.289 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2023-10-10T18:44:18.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-10T18:44:18.350 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2023-10-10T18:44:18.351 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2023-10-10T18:44:18.351 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000247583 s, 2.1 MB/s 2023-10-10T18:44:18.354 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-10T18:44:18.409 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_2 2023-10-10T18:44:18.464 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-10T18:44:18.465 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T18:44:18.465 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 222441 Links: 1 2023-10-10T18:44:18.465 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:44:18.465 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T18:44:18.466 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2023-10-10 18:44:11.328672157 +0000 2023-10-10T18:44:18.466 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-10-10 18:41:45.927166753 +0000 2023-10-10T18:44:18.466 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-10-10 18:41:45.927166753 +0000 2023-10-10T18:44:18.466 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2023-10-10T18:44:18.467 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-10T18:44:18.527 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2023-10-10T18:44:18.528 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2023-10-10T18:44:18.528 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000225398 s, 2.3 MB/s 2023-10-10T18:44:18.529 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-10T18:44:18.585 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_3 2023-10-10T18:44:18.641 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-10T18:44:18.641 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T18:44:18.641 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 218582 Links: 1 2023-10-10T18:44:18.642 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:44:18.642 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T18:44:18.642 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2023-10-10 18:44:11.328672157 +0000 2023-10-10T18:44:18.642 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-10-10 18:41:46.417159461 +0000 2023-10-10T18:44:18.642 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-10-10 18:41:46.417159461 +0000 2023-10-10T18:44:18.643 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2023-10-10T18:44:18.643 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-10T18:44:18.704 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2023-10-10T18:44:18.705 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2023-10-10T18:44:18.705 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.00026232 s, 2.0 MB/s 2023-10-10T18:44:18.706 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-10T18:44:18.764 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_4 2023-10-10T18:44:18.819 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-10T18:44:18.820 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T18:44:18.820 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 223347 Links: 1 2023-10-10T18:44:18.820 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:44:18.820 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T18:44:18.821 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2023-10-10 18:44:11.328672157 +0000 2023-10-10T18:44:18.821 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-10-10 18:41:46.902152243 +0000 2023-10-10T18:44:18.821 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-10-10 18:41:46.902152243 +0000 2023-10-10T18:44:18.822 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2023-10-10T18:44:18.822 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-10T18:44:18.883 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2023-10-10T18:44:18.883 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2023-10-10T18:44:18.883 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000197857 s, 2.6 MB/s 2023-10-10T18:44:18.884 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-10T18:44:18.942 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:18.942 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:18.943 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-10-10T18:44:18.959 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-10T18:44:18.960 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2023-10-10T18:44:19.015 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-10T18:44:19.016 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T18:44:19.016 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 217817 Links: 1 2023-10-10T18:44:19.016 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:44:19.017 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T18:44:19.017 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-10-10 18:44:08.972090999 +0000 2023-10-10T18:44:19.017 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-10-10 18:41:48.201454263 +0000 2023-10-10T18:44:19.018 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-10-10 18:41:48.201454263 +0000 2023-10-10T18:44:19.018 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-10-10T18:44:19.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-10T18:44:19.078 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-10-10T18:44:19.078 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-10-10T18:44:19.079 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000230075 s, 2.2 MB/s 2023-10-10T18:44:19.080 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-10T18:44:19.136 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2023-10-10T18:44:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-10T18:44:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T18:44:19.193 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 223542 Links: 1 2023-10-10T18:44:19.193 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:44:19.193 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T18:44:19.194 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-10-10 18:44:08.973090982 +0000 2023-10-10T18:44:19.195 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-10-10 18:41:48.689445985 +0000 2023-10-10T18:44:19.195 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-10-10 18:41:48.689445985 +0000 2023-10-10T18:44:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-10-10T18:44:19.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-10-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-10-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000220366 s, 2.3 MB/s 2023-10-10T18:44:19.256 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-10T18:44:19.313 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2023-10-10T18:44:19.370 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-10T18:44:19.370 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T18:44:19.370 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 223071 Links: 1 2023-10-10T18:44:19.370 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:44:19.371 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T18:44:19.371 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-10-10 18:44:08.973090982 +0000 2023-10-10T18:44:19.371 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-10-10 18:41:49.160437994 +0000 2023-10-10T18:44:19.371 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-10-10 18:41:49.160437994 +0000 2023-10-10T18:44:19.371 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-10-10T18:44:19.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-10T18:44:19.434 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-10-10T18:44:19.434 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-10-10T18:44:19.434 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000255915 s, 2.0 MB/s 2023-10-10T18:44:19.436 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-10T18:44:19.492 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2023-10-10T18:44:19.547 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-10T18:44:19.547 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T18:44:19.547 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 223693 Links: 1 2023-10-10T18:44:19.547 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:44:19.548 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T18:44:19.548 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-10-10 18:44:08.973090982 +0000 2023-10-10T18:44:19.548 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-10-10 18:41:49.641429834 +0000 2023-10-10T18:44:19.548 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-10-10 18:41:49.641429834 +0000 2023-10-10T18:44:19.548 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-10-10T18:44:19.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-10T18:44:19.610 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-10-10T18:44:19.610 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-10-10T18:44:19.610 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000182156 s, 2.8 MB/s 2023-10-10T18:44:19.611 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-10T18:44:19.666 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:19.666 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-10T18:44:19.667 INFO:tasks.ceph:Generating config... 2023-10-10T18:44:19.668 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-10T18:44:19.668 INFO:tasks.ceph:[client] debug client = 20 2023-10-10T18:44:19.669 INFO:tasks.ceph:[client] debug ms = 1 2023-10-10T18:44:19.669 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-10T18:44:19.669 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-10T18:44:19.670 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-10T18:44:19.670 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-10T18:44:19.670 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-10T18:44:19.670 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-10T18:44:19.671 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-10T18:44:19.671 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-10T18:44:19.671 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-10T18:44:19.672 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-10T18:44:19.672 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-10T18:44:19.672 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2023-10-10T18:44:19.672 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-10T18:44:19.673 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-10T18:44:19.673 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-10T18:44:19.673 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-10T18:44:19.673 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-10T18:44:19.674 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-10T18:44:19.674 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-10T18:44:19.674 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-10T18:44:19.674 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-10T18:44:19.675 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-10T18:44:19.675 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-10T18:44:19.675 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-10T18:44:19.675 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-10T18:44:19.676 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-10T18:44:19.676 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-10T18:44:19.676 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-10T18:44:19.677 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-10T18:44:19.677 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-10T18:44:19.677 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-10T18:44:19.677 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-10T18:44:19.678 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-10T18:44:19.678 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-10T18:44:19.678 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-10T18:44:19.678 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-10T18:44:19.679 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-10T18:44:19.679 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-10T18:44:19.679 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-10T18:44:19.680 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-10T18:44:19.680 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-10T18:44:19.680 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-10T18:44:19.681 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-10T18:44:19.681 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-10T18:44:19.681 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-10T18:44:19.681 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-10T18:44:19.682 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-10T18:44:19.682 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-10T18:44:19.682 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-10T18:44:19.683 INFO:tasks.ceph:Setting up mon.a... 2023-10-10T18:44:19.683 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-10T18:44:19.741 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/ceph.keyring 2023-10-10T18:44:19.743 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-10T18:44:19.782 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-10T18:44:19.846 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.175'), ('mon.b', '172.21.15.191'), ('mon.c', '[v2:172.21.15.191:3301,v1:172.21.15.191:6790]')] 2023-10-10T18:44:19.847 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.175,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-10T18:44:19.848 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:19.848 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-10T18:44:19.901 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.175 --add b 172.21.15.191 --addv c '[v2:172.21.15.191:3301,v1:172.21.15.191:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:19.982 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:19.982 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: generated fsid 7f0fe9e3-2088-4ba2-8a54-92a2be521afd 2023-10-10T18:44:19.983 INFO:teuthology.orchestra.run.smithi175.stdout:setting min_mon_release = octopus 2023-10-10T18:44:19.983 INFO:teuthology.orchestra.run.smithi175.stdout:epoch 0 2023-10-10T18:44:19.983 INFO:teuthology.orchestra.run.smithi175.stdout:fsid 7f0fe9e3-2088-4ba2-8a54-92a2be521afd 2023-10-10T18:44:19.983 INFO:teuthology.orchestra.run.smithi175.stdout:last_changed 2023-10-10T18:44:19.982962+0000 2023-10-10T18:44:19.983 INFO:teuthology.orchestra.run.smithi175.stdout:created 2023-10-10T18:44:19.982962+0000 2023-10-10T18:44:19.984 INFO:teuthology.orchestra.run.smithi175.stdout:min_mon_release 15 (octopus) 2023-10-10T18:44:19.984 INFO:teuthology.orchestra.run.smithi175.stdout:election_strategy: 1 2023-10-10T18:44:19.984 INFO:teuthology.orchestra.run.smithi175.stdout:0: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.a 2023-10-10T18:44:19.984 INFO:teuthology.orchestra.run.smithi175.stdout:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-10-10T18:44:19.984 INFO:teuthology.orchestra.run.smithi175.stdout:2: [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] mon.c 2023-10-10T18:44:19.985 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-10T18:44:19.986 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-10T18:44:20.040 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7f0fe9e3-2088-4ba2-8a54-92a2be521afd... 2023-10-10T18:44:20.041 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-10T18:44:20.082 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-10T18:44:20.122 INFO:teuthology.orchestra.run.smithi175.stdout:[global] 2023-10-10T18:44:20.122 INFO:teuthology.orchestra.run.smithi175.stdout: chdir = "" 2023-10-10T18:44:20.122 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-10T18:44:20.122 INFO:teuthology.orchestra.run.smithi175.stdout: auth supported = cephx 2023-10-10T18:44:20.123 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.123 INFO:teuthology.orchestra.run.smithi175.stdout: filestore xattr use omap = true 2023-10-10T18:44:20.123 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.123 INFO:teuthology.orchestra.run.smithi175.stdout: mon clock drift allowed = 1.000 2023-10-10T18:44:20.123 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.124 INFO:teuthology.orchestra.run.smithi175.stdout: osd crush chooseleaf type = 0 2023-10-10T18:44:20.124 INFO:teuthology.orchestra.run.smithi175.stdout: auth debug = true 2023-10-10T18:44:20.124 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.124 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on old message = true 2023-10-10T18:44:20.125 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on bug = true 2023-10-10T18:44:20.125 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.125 INFO:teuthology.orchestra.run.smithi175.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-10T18:44:20.125 INFO:teuthology.orchestra.run.smithi175.stdout: mon pg warn max object skew = 0 2023-10-10T18:44:20.125 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.126 INFO:teuthology.orchestra.run.smithi175.stdout: # disable pg_autoscaler by default for new pools 2023-10-10T18:44:20.126 INFO:teuthology.orchestra.run.smithi175.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-10T18:44:20.126 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.126 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default size = 2 2023-10-10T18:44:20.127 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.127 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow primary affinity = true 2023-10-10T18:44:20.127 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow pg remap = true 2023-10-10T18:44:20.127 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on legacy crush tunables = false 2023-10-10T18:44:20.127 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on crush straw calc version zero = false 2023-10-10T18:44:20.128 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on no sortbitwise = false 2023-10-10T18:44:20.128 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on osd down out interval zero = false 2023-10-10T18:44:20.128 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on too few osds = false 2023-10-10T18:44:20.128 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-10T18:44:20.128 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-10T18:44:20.129 INFO:teuthology.orchestra.run.smithi175.stdout: mon_allow_pool_size_one = true 2023-10-10T18:44:20.129 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.129 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-10T18:44:20.129 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.129 INFO:teuthology.orchestra.run.smithi175.stdout: osd default data pool replay window = 5 2023-10-10T18:44:20.130 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.130 INFO:teuthology.orchestra.run.smithi175.stdout: mon allow pool delete = true 2023-10-10T18:44:20.130 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.130 INFO:teuthology.orchestra.run.smithi175.stdout: mon cluster log file level = debug 2023-10-10T18:44:20.130 INFO:teuthology.orchestra.run.smithi175.stdout: debug asserts on shutdown = true 2023-10-10T18:44:20.131 INFO:teuthology.orchestra.run.smithi175.stdout: mon health detail to clog = false 2023-10-10T18:44:20.131 INFO:teuthology.orchestra.run.smithi175.stdout: mon host = "172.21.15.175,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]" 2023-10-10T18:44:20.131 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on skipped message = False 2023-10-10T18:44:20.131 INFO:teuthology.orchestra.run.smithi175.stdout: fsid = 7f0fe9e3-2088-4ba2-8a54-92a2be521afd 2023-10-10T18:44:20.132 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.132 INFO:teuthology.orchestra.run.smithi175.stdout:[osd] 2023-10-10T18:44:20.132 INFO:teuthology.orchestra.run.smithi175.stdout: osd journal size = 100 2023-10-10T18:44:20.132 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.132 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub load threshold = 5.0 2023-10-10T18:44:20.133 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub max interval = 600 2023-10-10T18:44:20.133 INFO:teuthology.orchestra.run.smithi175.stdout: osd mclock profile = high_recovery_ops 2023-10-10T18:44:20.133 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.133 INFO:teuthology.orchestra.run.smithi175.stdout: osd recover clone overlap = true 2023-10-10T18:44:20.133 INFO:teuthology.orchestra.run.smithi175.stdout: osd recovery max chunk = 1048576 2023-10-10T18:44:20.134 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.134 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug shutdown = true 2023-10-10T18:44:20.134 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug op order = true 2023-10-10T18:44:20.134 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug verify stray on activate = true 2023-10-10T18:44:20.134 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: osd open classes on start = true 2023-10-10T18:44:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug pg log writeout = true 2023-10-10T18:44:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: osd deep scrub update digest min age = 30 2023-10-10T18:44:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: osd map max advance = 10 2023-10-10T18:44:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: journal zero on create = true 2023-10-10T18:44:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.137 INFO:teuthology.orchestra.run.smithi175.stdout: filestore ondisk finisher threads = 3 2023-10-10T18:44:20.137 INFO:teuthology.orchestra.run.smithi175.stdout: filestore apply finisher threads = 3 2023-10-10T18:44:20.137 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.137 INFO:teuthology.orchestra.run.smithi175.stdout: bdev debug aio = true 2023-10-10T18:44:20.137 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug misdirected ops = true 2023-10-10T18:44:20.138 INFO:teuthology.orchestra.run.smithi175.stdout: bluestore block size = 96636764160 2023-10-10T18:44:20.138 INFO:teuthology.orchestra.run.smithi175.stdout: bluestore compression mode = aggressive 2023-10-10T18:44:20.138 INFO:teuthology.orchestra.run.smithi175.stdout: bluestore fsck on mount = True 2023-10-10T18:44:20.138 INFO:teuthology.orchestra.run.smithi175.stdout: debug bluefs = 20 2023-10-10T18:44:20.139 INFO:teuthology.orchestra.run.smithi175.stdout: debug bluestore = 20 2023-10-10T18:44:20.139 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-10-10T18:44:20.139 INFO:teuthology.orchestra.run.smithi175.stdout: debug osd = 20 2023-10-10T18:44:20.139 INFO:teuthology.orchestra.run.smithi175.stdout: debug rocksdb = 10 2023-10-10T18:44:20.139 INFO:teuthology.orchestra.run.smithi175.stdout: mon client directed command retry = 5 2023-10-10T18:44:20.140 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd backfillfull_ratio = 0.85 2023-10-10T18:44:20.140 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd full ratio = 0.9 2023-10-10T18:44:20.140 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd nearfull ratio = 0.8 2023-10-10T18:44:20.140 INFO:teuthology.orchestra.run.smithi175.stdout: ms inject delay max = 1 2023-10-10T18:44:20.140 INFO:teuthology.orchestra.run.smithi175.stdout: ms inject delay probability = 0.005 2023-10-10T18:44:20.140 INFO:teuthology.orchestra.run.smithi175.stdout: ms inject delay type = client mds 2023-10-10T18:44:20.141 INFO:teuthology.orchestra.run.smithi175.stdout: ms inject socket failures = 2500 2023-10-10T18:44:20.141 INFO:teuthology.orchestra.run.smithi175.stdout: osd failsafe full ratio = 0.95 2023-10-10T18:44:20.141 INFO:teuthology.orchestra.run.smithi175.stdout: osd max markdown count = 1000 2023-10-10T18:44:20.141 INFO:teuthology.orchestra.run.smithi175.stdout: osd objectstore = bluestore 2023-10-10T18:44:20.141 INFO:teuthology.orchestra.run.smithi175.stdout: osd op complaint time = 180 2023-10-10T18:44:20.142 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.142 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr] 2023-10-10T18:44:20.142 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-10-10T18:44:20.142 INFO:teuthology.orchestra.run.smithi175.stdout: debug mgr = 20 2023-10-10T18:44:20.142 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2023-10-10T18:44:20.143 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2023-10-10T18:44:20.143 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2023-10-10T18:44:20.143 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2023-10-10T18:44:20.143 INFO:teuthology.orchestra.run.smithi175.stdout: mgr/telemetry/nag = false 2023-10-10T18:44:20.143 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.144 INFO:teuthology.orchestra.run.smithi175.stdout:[mon] 2023-10-10T18:44:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-10-10T18:44:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2023-10-10T18:44:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: debug paxos = 20 2023-10-10T18:44:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2023-10-10T18:44:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: mon data avail warn = 5 2023-10-10T18:44:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: mon mgr mkfs grace = 240 2023-10-10T18:44:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2023-10-10T18:44:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd reporter subtree level = osd 2023-10-10T18:44:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd prime pg temp = true 2023-10-10T18:44:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2023-10-10T18:44:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-10T18:44:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: auth mon ticket ttl = 660 # 11m 2023-10-10T18:44:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: auth service ticket ttl = 240 # 4m 2023-10-10T18:44:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: # don't complain about insecure global_id in the test suite 2023-10-10T18:44:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-10T18:44:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-10T18:44:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: # 1m isn't quite enough 2023-10-10T18:44:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: mon_down_mkfs_grace = 2m 2023-10-10T18:44:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_filestore_osds = false 2023-10-10T18:44:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: mon op complaint time = 120 2023-10-10T18:44:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:20.149 INFO:teuthology.orchestra.run.smithi175.stdout:[client] 2023-10-10T18:44:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: rgw cache enabled = true 2023-10-10T18:44:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable ops log = true 2023-10-10T18:44:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable usage log = true 2023-10-10T18:44:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-10T18:44:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-10T18:44:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: client mount timeout = 600 2023-10-10T18:44:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: debug client = 20 2023-10-10T18:44:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-10-10T18:44:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: rados mon op timeout = 900 2023-10-10T18:44:20.151 INFO:teuthology.orchestra.run.smithi175.stdout: rados osd op timeout = 900 2023-10-10T18:44:20.151 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.a] 2023-10-10T18:44:20.151 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.b] 2023-10-10T18:44:20.151 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.c] 2023-10-10T18:44:20.151 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.a] 2023-10-10T18:44:20.151 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.c] 2023-10-10T18:44:20.152 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.e] 2023-10-10T18:44:20.152 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.b] 2023-10-10T18:44:20.152 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.d] 2023-10-10T18:44:20.152 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.f] 2023-10-10T18:44:20.152 INFO:teuthology.orchestra.run.smithi175.stdout:[mds] 2023-10-10T18:44:20.153 INFO:teuthology.orchestra.run.smithi175.stdout: debug mds = 20 2023-10-10T18:44:20.153 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-10-10T18:44:20.153 INFO:teuthology.orchestra.run.smithi175.stdout: mds bal fragment size max = 10000 2023-10-10T18:44:20.153 INFO:teuthology.orchestra.run.smithi175.stdout: mds bal merge size = 5 2023-10-10T18:44:20.153 INFO:teuthology.orchestra.run.smithi175.stdout: mds bal split bits = 3 2023-10-10T18:44:20.154 INFO:teuthology.orchestra.run.smithi175.stdout: mds bal split size = 100 2023-10-10T18:44:20.154 INFO:teuthology.orchestra.run.smithi175.stdout: mds debug frag = True 2023-10-10T18:44:20.154 INFO:teuthology.orchestra.run.smithi175.stdout: mds debug scatterstat = True 2023-10-10T18:44:20.154 INFO:teuthology.orchestra.run.smithi175.stdout: mds oft prefetch dirfrags = False 2023-10-10T18:44:20.154 INFO:teuthology.orchestra.run.smithi175.stdout: mds op complaint time = 180 2023-10-10T18:44:20.155 INFO:teuthology.orchestra.run.smithi175.stdout: mds verify scatter = True 2023-10-10T18:44:20.155 INFO:teuthology.orchestra.run.smithi175.stdout: mds_sleep_rank_change = 5000000.0 2023-10-10T18:44:20.155 INFO:teuthology.orchestra.run.smithi175.stdout: mon client directed command retry = 5 2023-10-10T18:44:20.155 INFO:teuthology.orchestra.run.smithi175.stdout: ms inject delay max = 1 2023-10-10T18:44:20.155 INFO:teuthology.orchestra.run.smithi175.stdout: ms inject delay probability = 0.005 2023-10-10T18:44:20.156 INFO:teuthology.orchestra.run.smithi175.stdout: ms inject delay type = client mds osd 2023-10-10T18:44:20.156 INFO:teuthology.orchestra.run.smithi175.stdout: ms inject socket failures = 2500 2023-10-10T18:44:20.156 INFO:teuthology.orchestra.run.smithi175.stdout: osd op complaint time = 180 2023-10-10T18:44:20.156 INFO:teuthology.orchestra.run.smithi175.stdout: rados mon op timeout = 900 2023-10-10T18:44:20.156 INFO:teuthology.orchestra.run.smithi175.stdout: rados osd op timeout = 900 2023-10-10T18:44:20.158 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2023-10-10T18:44:20.159 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2023-10-10T18:44:20.159 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-10T18:44:20.159 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2023-10-10T18:44:20.159 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.159 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2023-10-10T18:44:20.160 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.160 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2023-10-10T18:44:20.160 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.160 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2023-10-10T18:44:20.160 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2023-10-10T18:44:20.161 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.161 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2023-10-10T18:44:20.161 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2023-10-10T18:44:20.161 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.161 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-10T18:44:20.162 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2023-10-10T18:44:20.162 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.162 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2023-10-10T18:44:20.162 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-10T18:44:20.162 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.163 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2023-10-10T18:44:20.163 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.163 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2023-10-10T18:44:20.163 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2023-10-10T18:44:20.163 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2023-10-10T18:44:20.164 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2023-10-10T18:44:20.164 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2023-10-10T18:44:20.164 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2023-10-10T18:44:20.164 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2023-10-10T18:44:20.164 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-10T18:44:20.165 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-10T18:44:20.165 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2023-10-10T18:44:20.165 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.165 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-10T18:44:20.165 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.165 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2023-10-10T18:44:20.166 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.166 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2023-10-10T18:44:20.166 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.166 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2023-10-10T18:44:20.166 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2023-10-10T18:44:20.167 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2023-10-10T18:44:20.167 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.175,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]" 2023-10-10T18:44:20.167 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on skipped message = False 2023-10-10T18:44:20.167 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 7f0fe9e3-2088-4ba2-8a54-92a2be521afd 2023-10-10T18:44:20.167 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.168 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2023-10-10T18:44:20.168 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2023-10-10T18:44:20.168 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.168 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2023-10-10T18:44:20.168 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2023-10-10T18:44:20.169 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2023-10-10T18:44:20.169 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.169 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2023-10-10T18:44:20.169 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2023-10-10T18:44:20.169 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.169 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2023-10-10T18:44:20.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2023-10-10T18:44:20.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2023-10-10T18:44:20.170 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2023-10-10T18:44:20.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2023-10-10T18:44:20.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2023-10-10T18:44:20.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2023-10-10T18:44:20.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.172 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2023-10-10T18:44:20.172 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.172 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2023-10-10T18:44:20.172 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2023-10-10T18:44:20.172 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.173 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2023-10-10T18:44:20.173 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2023-10-10T18:44:20.173 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2023-10-10T18:44:20.173 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression mode = aggressive 2023-10-10T18:44:20.173 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2023-10-10T18:44:20.173 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 20 2023-10-10T18:44:20.174 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 20 2023-10-10T18:44:20.174 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-10T18:44:20.174 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2023-10-10T18:44:20.174 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2023-10-10T18:44:20.174 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2023-10-10T18:44:20.175 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2023-10-10T18:44:20.175 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2023-10-10T18:44:20.175 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2023-10-10T18:44:20.175 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay max = 1 2023-10-10T18:44:20.175 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay probability = 0.005 2023-10-10T18:44:20.176 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay type = client mds 2023-10-10T18:44:20.176 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 2500 2023-10-10T18:44:20.176 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2023-10-10T18:44:20.176 INFO:teuthology.orchestra.run.smithi191.stdout: osd max markdown count = 1000 2023-10-10T18:44:20.176 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2023-10-10T18:44:20.177 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2023-10-10T18:44:20.177 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.177 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2023-10-10T18:44:20.177 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-10T18:44:20.177 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2023-10-10T18:44:20.178 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-10-10T18:44:20.178 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-10-10T18:44:20.178 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-10-10T18:44:20.178 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-10-10T18:44:20.178 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2023-10-10T18:44:20.178 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.179 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2023-10-10T18:44:20.179 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-10T18:44:20.179 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-10-10T18:44:20.179 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2023-10-10T18:44:20.179 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-10-10T18:44:20.180 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2023-10-10T18:44:20.180 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2023-10-10T18:44:20.180 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-10-10T18:44:20.180 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2023-10-10T18:44:20.180 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2023-10-10T18:44:20.181 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-10-10T18:44:20.181 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.181 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-10T18:44:20.181 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2023-10-10T18:44:20.181 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2023-10-10T18:44:20.181 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.182 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2023-10-10T18:44:20.182 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-10T18:44:20.182 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-10T18:44:20.182 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.182 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2023-10-10T18:44:20.183 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2023-10-10T18:44:20.183 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.183 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2023-10-10T18:44:20.183 INFO:teuthology.orchestra.run.smithi191.stdout: mon op complaint time = 120 2023-10-10T18:44:20.183 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-10T18:44:20.184 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2023-10-10T18:44:20.184 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2023-10-10T18:44:20.184 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2023-10-10T18:44:20.184 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2023-10-10T18:44:20.184 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-10T18:44:20.185 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-10T18:44:20.185 INFO:teuthology.orchestra.run.smithi191.stdout: client mount timeout = 600 2023-10-10T18:44:20.185 INFO:teuthology.orchestra.run.smithi191.stdout: debug client = 20 2023-10-10T18:44:20.185 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-10T18:44:20.185 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2023-10-10T18:44:20.186 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2023-10-10T18:44:20.186 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2023-10-10T18:44:20.186 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2023-10-10T18:44:20.186 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2023-10-10T18:44:20.186 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2023-10-10T18:44:20.187 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2023-10-10T18:44:20.187 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.e] 2023-10-10T18:44:20.187 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2023-10-10T18:44:20.187 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2023-10-10T18:44:20.187 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.f] 2023-10-10T18:44:20.188 INFO:teuthology.orchestra.run.smithi191.stdout:[mds] 2023-10-10T18:44:20.188 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds = 20 2023-10-10T18:44:20.188 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-10T18:44:20.188 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal fragment size max = 10000 2023-10-10T18:44:20.188 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal merge size = 5 2023-10-10T18:44:20.188 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal split bits = 3 2023-10-10T18:44:20.189 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal split size = 100 2023-10-10T18:44:20.189 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug frag = True 2023-10-10T18:44:20.189 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug scatterstat = True 2023-10-10T18:44:20.189 INFO:teuthology.orchestra.run.smithi191.stdout: mds oft prefetch dirfrags = False 2023-10-10T18:44:20.189 INFO:teuthology.orchestra.run.smithi191.stdout: mds op complaint time = 180 2023-10-10T18:44:20.190 INFO:teuthology.orchestra.run.smithi191.stdout: mds verify scatter = True 2023-10-10T18:44:20.190 INFO:teuthology.orchestra.run.smithi191.stdout: mds_sleep_rank_change = 5000000.0 2023-10-10T18:44:20.190 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2023-10-10T18:44:20.190 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay max = 1 2023-10-10T18:44:20.190 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay probability = 0.005 2023-10-10T18:44:20.191 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay type = client mds osd 2023-10-10T18:44:20.191 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 2500 2023-10-10T18:44:20.191 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2023-10-10T18:44:20.191 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2023-10-10T18:44:20.193 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2023-10-10T18:44:20.194 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-10T18:44:20.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-10T18:44:20.250 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-10T18:44:20.250 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:20.251 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-10T18:44:20.267 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:20.267 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-10T18:44:20.323 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T18:44:20.323 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:20.324 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-10T18:44:20.324 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-10T18:44:20.396 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:20.397 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:20.451 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T18:44:20.451 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:20.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-10T18:44:20.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-10T18:44:20.485 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:20.486 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:20.540 INFO:tasks.ceph:Setting up mon nodes... 2023-10-10T18:44:20.540 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-10T18:44:20.541 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-10T18:44:20.588 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-10T18:44:20.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-10T18:44:20.676 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-10T18:44:20.678 INFO:tasks.ceph:Setting up mds nodes... 2023-10-10T18:44:20.678 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-10T18:44:20.726 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-10T18:44:20.728 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-10T18:44:20.752 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-10T18:44:20.840 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-10T18:44:20.842 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-10T18:44:20.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-10T18:44:20.954 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-10T18:44:20.956 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-10T18:44:20.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-10T18:44:21.028 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-10T18:44:21.031 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-10T18:44:21.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-10T18:44:21.140 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-10T18:44:21.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-10T18:44:21.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-10T18:44:21.252 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-10T18:44:21.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-10T18:44:21.277 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-10T18:44:21.278 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-10T18:44:21.316 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-10T18:44:21.328 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-10T18:44:21.329 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-10T18:44:21.329 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-10T18:44:21.393 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:21.393 INFO:tasks.ceph:role: osd.0 2023-10-10T18:44:21.393 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T18:44:21.394 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-10T18:44:21.465 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-10T18:44:21.465 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-10T18:44:21.465 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-10T18:44:21.465 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 2023-10-10T18:44:21.466 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-10T18:44:21.466 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2023-10-10T18:44:21.466 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-10T18:44:21.466 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-10T18:44:21.466 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-10T18:44:21.467 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-10T18:44:21.902 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2023-10-10T18:44:21.904 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2023-10-10T18:44:21.904 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-10T18:44:22.087 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-10T18:44:22.120 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-10T18:44:22.185 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:22.185 INFO:tasks.ceph:role: osd.1 2023-10-10T18:44:22.185 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T18:44:22.186 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-10T18:44:22.256 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-10T18:44:22.256 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-10T18:44:22.257 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-10T18:44:22.257 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 2023-10-10T18:44:22.257 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-10T18:44:22.257 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2023-10-10T18:44:22.257 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-10T18:44:22.258 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-10T18:44:22.258 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-10T18:44:22.258 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-10T18:44:22.685 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2023-10-10T18:44:22.687 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2023-10-10T18:44:22.687 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-10T18:44:22.800 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-10T18:44:22.829 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-10T18:44:22.894 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:22.894 INFO:tasks.ceph:role: osd.2 2023-10-10T18:44:22.895 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T18:44:22.895 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-10T18:44:22.966 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-10T18:44:22.966 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-10T18:44:22.966 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-10T18:44:22.966 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 2023-10-10T18:44:22.967 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-10T18:44:22.967 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2023-10-10T18:44:22.967 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-10T18:44:22.967 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-10T18:44:22.968 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-10T18:44:22.968 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-10T18:44:23.405 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2023-10-10T18:44:23.406 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2023-10-10T18:44:23.407 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-10T18:44:23.551 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-10T18:44:23.580 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-10T18:44:23.643 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:23.644 INFO:tasks.ceph:role: osd.3 2023-10-10T18:44:23.644 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T18:44:23.644 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-10T18:44:23.715 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-10T18:44:23.715 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-10T18:44:23.716 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-10T18:44:23.716 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 2023-10-10T18:44:23.716 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-10T18:44:23.716 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2023-10-10T18:44:23.716 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-10T18:44:23.716 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-10T18:44:23.717 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-10T18:44:23.717 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-10T18:44:24.160 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2023-10-10T18:44:24.161 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2023-10-10T18:44:24.162 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-10T18:44:24.332 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-10T18:44:24.361 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:24.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:24.454+0000 7fc540978540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-10T18:44:24.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:24.455+0000 7fc540978540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-10T18:44:24.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:24.455+0000 7fc540978540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-10T18:44:24.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:24.455+0000 7fc540978540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-10T18:44:24.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:24.455+0000 7fc540978540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-10T18:44:24.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:24.455+0000 7fc540978540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-10T18:44:24.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:24.982+0000 7fc540978540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:25.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:25.748+0000 7fc540978540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:26.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:26.510+0000 7fc540978540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:27.307 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-10T18:44:27.335 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:27.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:27.420+0000 7fa78de87540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-10T18:44:27.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:27.420+0000 7fa78de87540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-10T18:44:27.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:27.420+0000 7fa78de87540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-10T18:44:27.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:27.420+0000 7fa78de87540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-10T18:44:27.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:27.420+0000 7fa78de87540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-10T18:44:27.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:27.421+0000 7fa78de87540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-10T18:44:27.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:27.953+0000 7fa78de87540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:28.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:28.718+0000 7fa78de87540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:29.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:29.487+0000 7fa78de87540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:30.277 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-10T18:44:30.343 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:30.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:30.425+0000 7f817ee65540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-10T18:44:30.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:30.425+0000 7f817ee65540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-10T18:44:30.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:30.425+0000 7f817ee65540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-10T18:44:30.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:30.425+0000 7f817ee65540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-10T18:44:30.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:30.425+0000 7f817ee65540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-10T18:44:30.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:30.425+0000 7f817ee65540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-10T18:44:30.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:30.956+0000 7f817ee65540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:31.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:31.720+0000 7f817ee65540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:32.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:32.482+0000 7f817ee65540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:33.272 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-10T18:44:33.343 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:33.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:33.427+0000 7f5f94db2540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-10T18:44:33.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:33.428+0000 7f5f94db2540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-10T18:44:33.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:33.428+0000 7f5f94db2540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-10T18:44:33.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:33.428+0000 7f5f94db2540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-10T18:44:33.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:33.428+0000 7f5f94db2540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-10T18:44:33.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:33.428+0000 7f5f94db2540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-10T18:44:33.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:33.954+0000 7f5f94db2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:34.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:34.717+0000 7f5f94db2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:35.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:35.479+0000 7f5f94db2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:36.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-10T18:44:36.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-10T18:44:36.330 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:36.331 INFO:tasks.ceph:role: osd.4 2023-10-10T18:44:36.331 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T18:44:36.331 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-10T18:44:36.405 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-10T18:44:36.405 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-10T18:44:36.405 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-10T18:44:36.406 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 2023-10-10T18:44:36.406 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-10T18:44:36.406 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-10-10T18:44:36.406 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-10T18:44:36.406 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-10T18:44:36.407 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-10T18:44:36.407 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-10T18:44:36.774 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-10-10T18:44:36.776 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-10-10T18:44:36.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-10T18:44:36.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-10T18:44:36.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-10T18:44:37.003 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:37.003 INFO:tasks.ceph:role: osd.5 2023-10-10T18:44:37.004 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T18:44:37.004 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-10T18:44:37.076 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-10T18:44:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-10T18:44:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-10T18:44:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 2023-10-10T18:44:37.077 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-10T18:44:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-10-10T18:44:37.077 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-10T18:44:37.077 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-10T18:44:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-10T18:44:37.078 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-10T18:44:37.435 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-10-10T18:44:37.437 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-10-10T18:44:37.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-10T18:44:37.526 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-10T18:44:37.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-10T18:44:37.621 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:37.621 INFO:tasks.ceph:role: osd.6 2023-10-10T18:44:37.622 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T18:44:37.622 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-10T18:44:37.693 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-10T18:44:37.693 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-10T18:44:37.694 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-10T18:44:37.694 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 2023-10-10T18:44:37.694 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-10T18:44:37.694 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-10-10T18:44:37.695 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-10T18:44:37.695 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-10T18:44:37.695 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-10T18:44:37.695 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-10T18:44:38.057 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-10-10T18:44:38.059 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-10-10T18:44:38.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-10T18:44:38.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-10T18:44:38.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-10T18:44:38.259 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-10T18:44:38.259 INFO:tasks.ceph:role: osd.7 2023-10-10T18:44:38.259 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T18:44:38.259 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-10T18:44:38.332 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-10T18:44:38.332 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-10T18:44:38.332 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-10T18:44:38.333 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 2023-10-10T18:44:38.333 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-10T18:44:38.333 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-10-10T18:44:38.333 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-10T18:44:38.333 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-10T18:44:38.334 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-10T18:44:38.334 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-10T18:44:38.704 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-10-10T18:44:38.705 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-10-10T18:44:38.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-10T18:44:38.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-10T18:44:38.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:38.909 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:38.909+0000 7f8766025540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-10T18:44:38.912 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:38.911+0000 7f8766025540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-10T18:44:38.912 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:38.911+0000 7f8766025540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-10T18:44:38.912 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:38.911+0000 7f8766025540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-10T18:44:38.913 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:38.911+0000 7f8766025540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-10T18:44:38.920 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:38.919+0000 7f8766025540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-10T18:44:39.450 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:39.450+0000 7f8766025540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:40.216 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:40.215+0000 7f8766025540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:40.984 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:40.984+0000 7f8766025540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:41.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-10T18:44:41.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:41.923 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:41.922+0000 7f3a417dc540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-10T18:44:41.924 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:41.922+0000 7f3a417dc540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-10T18:44:41.924 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:41.922+0000 7f3a417dc540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-10T18:44:41.925 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:41.922+0000 7f3a417dc540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-10T18:44:41.925 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:41.922+0000 7f3a417dc540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-10T18:44:41.925 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:41.923+0000 7f3a417dc540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-10T18:44:42.449 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:42.449+0000 7f3a417dc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:43.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:43.213+0000 7f3a417dc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:43.980 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:43.980+0000 7f3a417dc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:44.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-10T18:44:44.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:44.925 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:44.925+0000 7fbe686c3540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-10T18:44:44.926 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:44.925+0000 7fbe686c3540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-10T18:44:44.926 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:44.925+0000 7fbe686c3540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-10T18:44:44.926 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:44.925+0000 7fbe686c3540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-10T18:44:44.927 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:44.925+0000 7fbe686c3540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-10T18:44:44.927 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:44.925+0000 7fbe686c3540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-10T18:44:45.457 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:45.456+0000 7fbe686c3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:46.225 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:46.224+0000 7fbe686c3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:46.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:46.990+0000 7fbe686c3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:47.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-10T18:44:47.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:47.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:47.945+0000 7f88ada22540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-10T18:44:47.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:47.945+0000 7f88ada22540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-10T18:44:47.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:47.945+0000 7f88ada22540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-10T18:44:47.947 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:47.945+0000 7f88ada22540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-10T18:44:47.947 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:47.945+0000 7f88ada22540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-10T18:44:47.947 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:47.946+0000 7f88ada22540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-10T18:44:48.099 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:44:48.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:44:48.476 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:48.476+0000 7f88ada22540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:49.244 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:49.243+0000 7f88ada22540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:50.016 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:50.015+0000 7f88ada22540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-10T18:44:50.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-10T18:44:50.840 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-10T18:44:50.840 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:50.840 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-10T18:44:50.867 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:50.868 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-10T18:44:50.933 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:50.933 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-10T18:44:50.999 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:50.999 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-10T18:44:51.063 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:51.064 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-10T18:44:51.129 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:51.129 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-10T18:44:51.193 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:51.193 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-10T18:44:51.256 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:51.257 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-10T18:44:51.320 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:51.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-10T18:44:51.349 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:51.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-10T18:44:51.417 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:51.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-10T18:44:51.483 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:51.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-10T18:44:51.549 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:51.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-10T18:44:51.614 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:51.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-10T18:44:51.679 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:51.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-10T18:44:51.743 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:51.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-10T18:44:51.807 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:51.807 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-10T18:44:51.822 INFO:tasks.ceph:Adding keys to all mons... 2023-10-10T18:44:51.823 DEBUG:teuthology.orchestra.run.smithi175:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-10T18:44:51.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-10T18:44:51.885 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr.y] 2023-10-10T18:44:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCEmyVlAxYcIxAACFMeayWflud6sweHoZm/uQ== 2023-10-10T18:44:51.886 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.a] 2023-10-10T18:44:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCEmyVlGWhbKxAAXA6Weq/RyRI6k3YyYdYsUw== 2023-10-10T18:44:51.886 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.c] 2023-10-10T18:44:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCEmyVlrI4mMhAAfRXFSR0X8VAoEVzdovbtEw== 2023-10-10T18:44:51.887 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.e] 2023-10-10T18:44:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCEmyVlKJftOBAAnKvGE/It/NWZZTNDzKsb/w== 2023-10-10T18:44:51.887 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2023-10-10T18:44:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCImyVlN2kmGxAAK0q6CRydLVBA4RVdLEMKFg== 2023-10-10T18:44:51.888 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2023-10-10T18:44:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCLmyVlGt8bGRAAUQPbtrqZmfxl4Ze6gSCkAg== 2023-10-10T18:44:51.888 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2023-10-10T18:44:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCOmyVlMX9hGRAAx377DJ05D6HtZWwBo318gQ== 2023-10-10T18:44:51.888 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.3] 2023-10-10T18:44:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCRmyVlkRSLGRAAa+hBaU0gY4iaoxXH82YpJw== 2023-10-10T18:44:51.889 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr.x] 2023-10-10T18:44:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCEmyVlgs5VKBAAulrK2qNU2DhUcDdabcmORw== 2023-10-10T18:44:51.889 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.b] 2023-10-10T18:44:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCFmyVljD/GARAAYyqP4RIAKpPPPbwws/raYA== 2023-10-10T18:44:51.890 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.d] 2023-10-10T18:44:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCFmyVlRnhtCBAAUzIDpP/3kfzsnutaJ1CDXQ== 2023-10-10T18:44:51.890 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.f] 2023-10-10T18:44:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCFmyVlYxQYDxAAefh0XPwqtlxQl5TPXik/bw== 2023-10-10T18:44:51.890 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.4] 2023-10-10T18:44:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCWmyVlbhJGNhAAlLQhqnNeO6kVLFVKnpanBg== 2023-10-10T18:44:51.891 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.5] 2023-10-10T18:44:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCZmyVlvU8PNxAAglZxhaIzjR76w/zjfxeVAw== 2023-10-10T18:44:51.891 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.6] 2023-10-10T18:44:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCcmyVl8yg2NxAA4x+Kweh8utVy4Z+LjqXF/w== 2023-10-10T18:44:51.892 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.7] 2023-10-10T18:44:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCfmyVlb/htOBAAK1LllAWjJ3GZmngiSSG5vQ== 2023-10-10T18:44:51.892 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2023-10-10T18:44:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCFmyVlbHXlEhAAfULlS2EMejNpe2SWkpl9Zw== 2023-10-10T18:44:51.894 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2023-10-10T18:44:51.894 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCEmyVlAxYcIxAACFMeayWflud6sweHoZm/uQ== 2023-10-10T18:44:51.894 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2023-10-10T18:44:51.894 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCEmyVlGWhbKxAAXA6Weq/RyRI6k3YyYdYsUw== 2023-10-10T18:44:51.895 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2023-10-10T18:44:51.895 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCEmyVlrI4mMhAAfRXFSR0X8VAoEVzdovbtEw== 2023-10-10T18:44:51.895 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.e] 2023-10-10T18:44:51.895 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCEmyVlKJftOBAAnKvGE/It/NWZZTNDzKsb/w== 2023-10-10T18:44:51.896 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2023-10-10T18:44:51.896 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCImyVlN2kmGxAAK0q6CRydLVBA4RVdLEMKFg== 2023-10-10T18:44:51.896 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2023-10-10T18:44:51.896 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCLmyVlGt8bGRAAUQPbtrqZmfxl4Ze6gSCkAg== 2023-10-10T18:44:51.896 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2023-10-10T18:44:51.897 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCOmyVlMX9hGRAAx377DJ05D6HtZWwBo318gQ== 2023-10-10T18:44:51.897 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2023-10-10T18:44:51.897 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCRmyVlkRSLGRAAa+hBaU0gY4iaoxXH82YpJw== 2023-10-10T18:44:51.897 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2023-10-10T18:44:51.897 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCEmyVlgs5VKBAAulrK2qNU2DhUcDdabcmORw== 2023-10-10T18:44:51.898 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2023-10-10T18:44:51.898 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCFmyVljD/GARAAYyqP4RIAKpPPPbwws/raYA== 2023-10-10T18:44:51.898 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2023-10-10T18:44:51.898 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCFmyVlRnhtCBAAUzIDpP/3kfzsnutaJ1CDXQ== 2023-10-10T18:44:51.898 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.f] 2023-10-10T18:44:51.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCFmyVlYxQYDxAAefh0XPwqtlxQl5TPXik/bw== 2023-10-10T18:44:51.899 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2023-10-10T18:44:51.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCWmyVlbhJGNhAAlLQhqnNeO6kVLFVKnpanBg== 2023-10-10T18:44:51.899 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2023-10-10T18:44:51.899 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCZmyVlvU8PNxAAglZxhaIzjR76w/zjfxeVAw== 2023-10-10T18:44:51.900 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2023-10-10T18:44:51.900 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCcmyVl8yg2NxAA4x+Kweh8utVy4Z+LjqXF/w== 2023-10-10T18:44:51.900 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2023-10-10T18:44:51.900 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCfmyVlb/htOBAAK1LllAWjJ3GZmngiSSG5vQ== 2023-10-10T18:44:51.900 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2023-10-10T18:44:51.901 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCFmyVlbHXlEhAAfULlS2EMejNpe2SWkpl9Zw== 2023-10-10T18:44:51.901 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-10T18:44:51.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-10T18:44:51.982 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:51.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.034 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.085 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.137 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.186 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.237 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.285 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-10T18:44:52.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-10T18:44:52.383 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.386 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.435 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.487 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-10T18:44:52.539 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.588 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.636 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.684 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-10T18:44:52.733 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-10T18:44:52.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-10T18:44:52.785 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-10T18:44:52.786 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-10T18:44:52.811 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-10T18:44:52.901 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-10T18:44:52.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-10T18:44:52.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-10T18:44:53.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-10T18:44:53.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-10T18:44:53.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-10T18:44:53.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-10T18:44:53.241 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:53.244 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-10T18:44:53.295 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-10T18:44:53.296 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-10T18:44:53.296 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-10T18:44:53.298 INFO:tasks.ceph.mon.a:Started 2023-10-10T18:44:53.299 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-10T18:44:53.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-10T18:44:53.337 INFO:tasks.ceph.mon.b:Started 2023-10-10T18:44:53.338 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-10T18:44:53.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-10T18:44:53.340 INFO:tasks.ceph.mon.c:Started 2023-10-10T18:44:53.341 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-10T18:44:53.341 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-10T18:44:53.342 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-10T18:44:53.344 INFO:tasks.ceph.mgr.y:Started 2023-10-10T18:44:53.344 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-10T18:44:53.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-10T18:44:53.347 INFO:tasks.ceph.mgr.x:Started 2023-10-10T18:44:53.347 INFO:tasks.ceph:Setting crush tunables to default 2023-10-10T18:44:53.347 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd crush tunables default 2023-10-10T18:44:53.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.464+0000 7fe869f1b700 1 Processor -- start 2023-10-10T18:44:53.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.464+0000 7fe869f1b700 1 -- start start 2023-10-10T18:44:53.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe869f1b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe8640c70e0 0x7fe864156c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:53.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe869f1b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 0x7fe864158af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:53.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe869f1b700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe864150950 con 0x7fe8640cc8a0 2023-10-10T18:44:53.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe869f1b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8640cd040 con 0x7fe8640ccc70 2023-10-10T18:44:53.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe869f1b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe864157160 con 0x7fe8640c70e0 2023-10-10T18:44:53.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe863fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 0x7fe864158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:53.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe8637fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe8640cc8a0 0x7fe8640aff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44886/0 (socket says 172.21.15.175:44886) 2023-10-10T18:44:53.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe862ffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe8640c70e0 0x7fe864156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:53.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe863fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 0x7fe864158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40460/0 (socket says 172.21.15.175:40460) 2023-10-10T18:44:53.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe8637fe700 1 -- 172.21.15.175:0/2237402390 learned_addr learned my addr 172.21.15.175:0/2237402390 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:44:53.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe863fff700 1 -- 172.21.15.175:0/2237402390 >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 msgr2=0x7fe864158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-10T18:44:53.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe863fff700 1 -- 172.21.15.175:0/2237402390 >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 msgr2=0x7fe864158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-10T18:44:53.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe862ffd700 1 -- 172.21.15.175:0/2237402390 >> v2:172.21.15.191:3300/0 conn(0x7fe8640c70e0 msgr2=0x7fe864156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-10T18:44:53.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe862ffd700 1 -- 172.21.15.175:0/2237402390 >> v2:172.21.15.191:3300/0 conn(0x7fe8640c70e0 msgr2=0x7fe864156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-10T18:44:53.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe863fff700 1 --2- 172.21.15.175:0/2237402390 >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 0x7fe864158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-10T18:44:53.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe862ffd700 1 --2- 172.21.15.175:0/2237402390 >> v2:172.21.15.191:3300/0 conn(0x7fe8640c70e0 0x7fe864156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-10T18:44:53.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe862ffd700 1 --2- 172.21.15.175:0/2237402390 >> v2:172.21.15.191:3300/0 conn(0x7fe8640c70e0 0x7fe864156c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T18:44:53.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.465+0000 7fe863fff700 1 --2- 172.21.15.175:0/2237402390 >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 0x7fe864158af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T18:44:53.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.653+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869398048 0 0) 0x7fe864150950 con 0x7fe8640cc8a0 2023-10-10T18:44:53.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.654+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe84c003570 con 0x7fe8640cc8a0 2023-10-10T18:44:53.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3142811890 0 0) 0x7fe84c003570 con 0x7fe8640cc8a0 2023-10-10T18:44:53.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 >> v2:172.21.15.191:3300/0 conn(0x7fe8640c70e0 msgr2=0x7fe864156c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T18:44:53.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 --2- 172.21.15.175:0/2237402390 >> v2:172.21.15.191:3300/0 conn(0x7fe8640c70e0 0x7fe864156c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 msgr2=0x7fe864158af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T18:44:53.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 --2- 172.21.15.175:0/2237402390 >> v2:172.21.15.175:3300/0 conn(0x7fe8640ccc70 0x7fe864158af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8640afac0 con 0x7fe8640cc8a0 2023-10-10T18:44:53.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fe8540017f0 con 0x7fe8640cc8a0 2023-10-10T18:44:53.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 >> v1:172.21.15.175:6789/0 conn(0x7fe8640cc8a0 legacy=0x7fe8640aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:53.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe84c003a90 0x7fe84c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:53.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe84c004810 0x7fe84c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:53.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c0051c0 0x7fe84c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:53.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640cd040 con 0x7fe84c004810 2023-10-10T18:44:53.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864157160 con 0x7fe84c003a90 2023-10-10T18:44:53.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c00c150 con 0x7fe84c0051c0 2023-10-10T18:44:53.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe862ffd700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe84c004810 0x7fe84c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:53.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe8637fe700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe84c003a90 0x7fe84c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:53.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.657+0000 7fe863fff700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c0051c0 0x7fe84c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:53.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe862ffd700 1 -- 172.21.15.175:0/2237402390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c0051c0 msgr2=0x7fe84c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:53.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe862ffd700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c0051c0 0x7fe84c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe862ffd700 1 -- 172.21.15.175:0/2237402390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe84c003a90 msgr2=0x7fe84c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:53.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe862ffd700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe84c003a90 0x7fe84c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe862ffd700 1 -- 172.21.15.175:0/2237402390 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c003570 con 0x7fe84c004810 2023-10-10T18:44:53.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe863fff700 1 -- 172.21.15.175:0/2237402390 reap_dead start 2023-10-10T18:44:53.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe862ffd700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe84c004810 0x7fe84c004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe858006f30 tx=0x7fe858008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f66308bde620e54 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:53.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858018070 con 0x7fe84c004810 2023-10-10T18:44:53.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85800d5d0 con 0x7fe84c004810 2023-10-10T18:44:53.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe8627fc700 1 -- 172.21.15.175:0/2237402390 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85800d990 con 0x7fe84c004810 2023-10-10T18:44:53.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe869f1b700 1 -- 172.21.15.175:0/2237402390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe84c004810 msgr2=0x7fe84c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:53.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.658+0000 7fe869f1b700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe84c004810 0x7fe84c004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe858006f30 tx=0x7fe858008040 comp rx=0 tx=0).stop 2023-10-10T18:44:53.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.659+0000 7fe869f1b700 1 -- 172.21.15.175:0/2237402390 shutdown_connections 2023-10-10T18:44:53.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.659+0000 7fe869f1b700 1 --2- 172.21.15.175:0/2237402390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe84c004810 0x7fe84c004c80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.659+0000 7fe869f1b700 1 -- 172.21.15.175:0/2237402390 >> 172.21.15.175:0/2237402390 conn(0x7fe8640682f0 msgr2=0x7fe8640686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:53.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.659+0000 7fe869f1b700 1 -- 172.21.15.175:0/2237402390 shutdown_connections 2023-10-10T18:44:53.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.659+0000 7fe869f1b700 1 -- 172.21.15.175:0/2237402390 wait complete. 2023-10-10T18:44:53.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.660+0000 7fe869f1b700 1 Processor -- start 2023-10-10T18:44:53.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.660+0000 7fe869f1b700 1 -- start start 2023-10-10T18:44:53.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe869f1b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c004810 0x7fe864169ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:53.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe869f1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8640c8b50 0x7fe8640c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:53.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe869f1b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8640c8fc0 0x7fe8640c2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:53.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe869f1b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86416a010 con 0x7fe8640c8b50 2023-10-10T18:44:53.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe869f1b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864068ad0 con 0x7fe8640c8fc0 2023-10-10T18:44:53.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe869f1b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640c07f0 con 0x7fe84c004810 2023-10-10T18:44:53.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8640c8b50 0x7fe8640c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:53.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe8637fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c004810 0x7fe864169ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:53.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8640c8b50 0x7fe8640c2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40486/0 (socket says 172.21.15.175:40486) 2023-10-10T18:44:53.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 -- 172.21.15.175:0/3240706216 learned_addr learned my addr 172.21.15.175:0/3240706216 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:44:53.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe863fff700 1 --2- 172.21.15.175:0/3240706216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8640c8fc0 0x7fe8640c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:53.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 -- 172.21.15.175:0/3240706216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c004810 msgr2=0x7fe864169ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:53.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 --2- 172.21.15.175:0/3240706216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c004810 0x7fe864169ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 -- 172.21.15.175:0/3240706216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8640c8fc0 msgr2=0x7fe8640c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:53.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 --2- 172.21.15.175:0/3240706216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8640c8fc0 0x7fe8640c2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 -- 172.21.15.175:0/3240706216 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8640c1e80 con 0x7fe8640c8b50 2023-10-10T18:44:53.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.661+0000 7fe862ffd700 1 --2- 172.21.15.175:0/3240706216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8640c8b50 0x7fe8640c2310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe85800d440 tx=0x7fe85801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:53.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.663+0000 7fe84b7fe700 1 -- 172.21.15.175:0/3240706216 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858006a90 con 0x7fe8640c8b50 2023-10-10T18:44:53.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.663+0000 7fe84b7fe700 1 -- 172.21.15.175:0/3240706216 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85801f640 con 0x7fe8640c8b50 2023-10-10T18:44:53.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.663+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8640c0510 con 0x7fe8640c8b50 2023-10-10T18:44:53.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.663+0000 7fe84b7fe700 1 -- 172.21.15.175:0/3240706216 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85801fa00 con 0x7fe8640c8b50 2023-10-10T18:44:53.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.663+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe864153780 con 0x7fe8640c8b50 2023-10-10T18:44:53.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.663+0000 7fe84b7fe700 1 -- 172.21.15.175:0/3240706216 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe8580146b0 con 0x7fe8640c8b50 2023-10-10T18:44:53.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.663+0000 7fe84b7fe700 1 -- 172.21.15.175:0/3240706216 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe858027750 con 0x7fe8640c8b50 2023-10-10T18:44:53.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.664+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86406a950 con 0x7fe8640c8b50 2023-10-10T18:44:53.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.667+0000 7fe84b7fe700 1 -- 172.21.15.175:0/3240706216 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe8580466d0 con 0x7fe8640c8b50 2023-10-10T18:44:53.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.743+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe8640c2010 con 0x7fe8640c8b50 2023-10-10T18:44:53.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.749+0000 7fe84b7fe700 1 -- 172.21.15.175:0/3240706216 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe858035020 con 0x7fe8640c8b50 2023-10-10T18:44:53.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8640c8b50 msgr2=0x7fe8640c2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:53.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 --2- 172.21.15.175:0/3240706216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8640c8b50 0x7fe8640c2310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe85800d440 tx=0x7fe85801a000 comp rx=0 tx=0).stop 2023-10-10T18:44:53.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 shutdown_connections 2023-10-10T18:44:53.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 --2- 172.21.15.175:0/3240706216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8640c8fc0 0x7fe8640c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 --2- 172.21.15.175:0/3240706216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe84c004810 0x7fe864169ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 --2- 172.21.15.175:0/3240706216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8640c8b50 0x7fe8640c2310 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 >> 172.21.15.175:0/3240706216 conn(0x7fe8640682f0 msgr2=0x7fe86415cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 shutdown_connections 2023-10-10T18:44:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:53.750+0000 7fe869f1b700 1 -- 172.21.15.175:0/3240706216 wait complete. 2023-10-10T18:44:53.753 INFO:teuthology.orchestra.run.smithi175.stderr:adjusted tunables profile to default 2023-10-10T18:44:53.762 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-10T18:44:53.763 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:53.763 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-10T18:44:53.787 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:53.787 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-10T18:44:53.852 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:53.853 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-10T18:44:53.917 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:44:53.918 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-10T18:44:53.982 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:53.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-10T18:44:54.006 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:54.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-10T18:44:54.070 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:54.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-10T18:44:54.134 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-10T18:44:54.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-10T18:44:54.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 65c64bb9-f0f1-438a-aa17-8dc2335c7070 0 2023-10-10T18:44:54.352 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.352+0000 7fe26df37700 1 Processor -- start 2023-10-10T18:44:54.353 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.352+0000 7fe26df37700 1 -- start start 2023-10-10T18:44:54.355 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.352+0000 7fe26df37700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680cc8a0 0x7fe268156c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.355 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.352+0000 7fe26df37700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe2680ccc70 0x7fe268158af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.355 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.352+0000 7fe26df37700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe268150950 con 0x7fe2680c70e0 2023-10-10T18:44:54.356 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.352+0000 7fe26df37700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe268157160 con 0x7fe2680ccc70 2023-10-10T18:44:54.356 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.352+0000 7fe26df37700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680afac0 con 0x7fe2680cc8a0 2023-10-10T18:44:54.356 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe267fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe2680ccc70 0x7fe268158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.356 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe267fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe2680ccc70 0x7fe268158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44766/0 (socket says 172.21.15.191:44766) 2023-10-10T18:44:54.356 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe267fff700 1 -- 172.21.15.191:0/3600603137 learned_addr learned my addr 172.21.15.191:0/3600603137 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:54.357 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2677fe700 1 --2- 172.21.15.191:0/3600603137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680cc8a0 0x7fe268156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.357 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe266ffd700 1 --1- 172.21.15.191:0/3600603137 >> v1:172.21.15.191:6789/0 conn(0x7fe2680c70e0 0x7fe2680aff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:33880/0 (socket says 172.21.15.191:33880) 2023-10-10T18:44:54.357 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2677fe700 1 -- 172.21.15.191:0/3600603137 >> v2:172.21.15.191:3300/0 conn(0x7fe2680ccc70 msgr2=0x7fe268158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:54.357 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2677fe700 1 --2- 172.21.15.191:0/3600603137 >> v2:172.21.15.191:3300/0 conn(0x7fe2680ccc70 0x7fe268158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.357 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2677fe700 1 -- 172.21.15.191:0/3600603137 >> v1:172.21.15.191:6789/0 conn(0x7fe2680c70e0 legacy=0x7fe2680aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.358 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2677fe700 1 -- 172.21.15.191:0/3600603137 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2680af4e0 con 0x7fe2680cc8a0 2023-10-10T18:44:54.358 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2667fc700 1 -- 172.21.15.191:0/3600603137 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016665883 0 0) 0x7fe268150950 con 0x7fe2680c70e0 2023-10-10T18:44:54.358 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe267fff700 1 --2- 172.21.15.191:0/3600603137 >> v2:172.21.15.191:3300/0 conn(0x7fe2680ccc70 0x7fe268158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:54.358 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2677fe700 1 --2- 172.21.15.191:0/3600603137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680cc8a0 0x7fe268156c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe2580098e0 tx=0x7fe2580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e76df6002aaedb86 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:54.358 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2667fc700 1 -- 172.21.15.191:0/3600603137 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25800b640 con 0x7fe2680cc8a0 2023-10-10T18:44:54.359 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2667fc700 1 -- 172.21.15.191:0/3600603137 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25800b7a0 con 0x7fe2680cc8a0 2023-10-10T18:44:54.359 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.353+0000 7fe2667fc700 1 -- 172.21.15.191:0/3600603137 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25800bb40 con 0x7fe2680cc8a0 2023-10-10T18:44:54.359 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.354+0000 7fe26df37700 1 -- 172.21.15.191:0/3600603137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680cc8a0 msgr2=0x7fe268156c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.359 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.354+0000 7fe26df37700 1 --2- 172.21.15.191:0/3600603137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680cc8a0 0x7fe268156c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe2580098e0 tx=0x7fe2580091a0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.359 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.354+0000 7fe26df37700 1 -- 172.21.15.191:0/3600603137 shutdown_connections 2023-10-10T18:44:54.360 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.354+0000 7fe26df37700 1 --2- 172.21.15.191:0/3600603137 >> v2:172.21.15.191:3300/0 conn(0x7fe2680ccc70 0x7fe268158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.360 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.354+0000 7fe26df37700 1 --2- 172.21.15.191:0/3600603137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680cc8a0 0x7fe268156c20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.360 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.354+0000 7fe26df37700 1 -- 172.21.15.191:0/3600603137 >> 172.21.15.191:0/3600603137 conn(0x7fe2680682f0 msgr2=0x7fe2680686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:54.360 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.354+0000 7fe26df37700 1 -- 172.21.15.191:0/3600603137 shutdown_connections 2023-10-10T18:44:54.360 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.354+0000 7fe26df37700 1 -- 172.21.15.191:0/3600603137 wait complete. 2023-10-10T18:44:54.363 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.355+0000 7fe26df37700 1 Processor -- start 2023-10-10T18:44:54.363 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.355+0000 7fe26df37700 1 -- start start 2023-10-10T18:44:54.363 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe26df37700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680c70e0 0x7fe268152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.364 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe26df37700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2680cc8a0 0x7fe268166570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.364 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe26df37700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2680ccc70 0x7fe268166ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.364 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe26df37700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26814ff80 con 0x7fe2680ccc70 2023-10-10T18:44:54.364 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe26df37700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680c4200 con 0x7fe2680cc8a0 2023-10-10T18:44:54.364 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe26df37700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26814fca0 con 0x7fe2680c70e0 2023-10-10T18:44:54.365 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe266ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2680cc8a0 0x7fe268166570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.365 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe2677fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680c70e0 0x7fe268152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.365 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe266ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2680cc8a0 0x7fe268166570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44770/0 (socket says 172.21.15.191:44770) 2023-10-10T18:44:54.365 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe266ffd700 1 -- 172.21.15.191:0/1827203328 learned_addr learned my addr 172.21.15.191:0/1827203328 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:54.365 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe2677fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680c70e0 0x7fe268152e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:42994/0 (socket says 172.21.15.191:42994) 2023-10-10T18:44:54.366 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe267fff700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2680ccc70 0x7fe268166ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.366 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe2677fe700 1 -- 172.21.15.191:0/1827203328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2680cc8a0 msgr2=0x7fe268166570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.366 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe2677fe700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2680cc8a0 0x7fe268166570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.366 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe2677fe700 1 -- 172.21.15.191:0/1827203328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2680ccc70 msgr2=0x7fe268166ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.366 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe2677fe700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2680ccc70 0x7fe268166ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.367 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe2677fe700 1 -- 172.21.15.191:0/1827203328 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe260009ae0 con 0x7fe2680c70e0 2023-10-10T18:44:54.367 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.356+0000 7fe266ffd700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2680cc8a0 0x7fe268166570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:54.367 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.357+0000 7fe2677fe700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680c70e0 0x7fe268152e60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe258016000 tx=0x7fe258017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:54.367 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.357+0000 7fe24f7fe700 1 -- 172.21.15.191:0/1827203328 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25800bb20 con 0x7fe2680c70e0 2023-10-10T18:44:54.367 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.357+0000 7fe24f7fe700 1 -- 172.21.15.191:0/1827203328 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe258000c90 con 0x7fe2680c70e0 2023-10-10T18:44:54.368 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.357+0000 7fe24f7fe700 1 -- 172.21.15.191:0/1827203328 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258011650 con 0x7fe2680c70e0 2023-10-10T18:44:54.368 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.357+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe258009420 con 0x7fe2680c70e0 2023-10-10T18:44:54.368 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.357+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2680c3f20 con 0x7fe2680c70e0 2023-10-10T18:44:54.368 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.357+0000 7fe24f7fe700 1 -- 172.21.15.191:0/1827203328 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe258011c80 con 0x7fe2680c70e0 2023-10-10T18:44:54.368 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.357+0000 7fe24f7fe700 1 -- 172.21.15.191:0/1827203328 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe258025bf0 con 0x7fe2680c70e0 2023-10-10T18:44:54.368 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.358+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26806a950 con 0x7fe2680c70e0 2023-10-10T18:44:54.369 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.360+0000 7fe24f7fe700 1 -- 172.21.15.191:0/1827203328 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe258029070 con 0x7fe2680c70e0 2023-10-10T18:44:54.444 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.443+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "65c64bb9-f0f1-438a-aa17-8dc2335c7070", "id": 0} v 0) v1 -- 0x7fe268067570 con 0x7fe2680c70e0 2023-10-10T18:44:54.449 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.449+0000 7fe24f7fe700 1 -- 172.21.15.191:0/1827203328 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "65c64bb9-f0f1-438a-aa17-8dc2335c7070", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe258015020 con 0x7fe2680c70e0 2023-10-10T18:44:54.449 INFO:teuthology.orchestra.run.smithi191.stdout:0 2023-10-10T18:44:54.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680c70e0 msgr2=0x7fe268152e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680c70e0 0x7fe268152e60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe258016000 tx=0x7fe258017000 comp rx=0 tx=0).stop 2023-10-10T18:44:54.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 shutdown_connections 2023-10-10T18:44:54.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2680ccc70 0x7fe268166ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2680c70e0 0x7fe268152e60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.452 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 --2- 172.21.15.191:0/1827203328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2680cc8a0 0x7fe268166570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.452 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 >> 172.21.15.191:0/1827203328 conn(0x7fe2680682f0 msgr2=0x7fe2680cbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:54.452 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 shutdown_connections 2023-10-10T18:44:54.452 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.450+0000 7fe26df37700 1 -- 172.21.15.191:0/1827203328 wait complete. 2023-10-10T18:44:54.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new a02d3af7-057b-403b-bb65-aca12bdb262b 1 2023-10-10T18:44:54.575 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.575+0000 7fc7b8ad9700 1 Processor -- start 2023-10-10T18:44:54.576 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.575+0000 7fc7b8ad9700 1 -- start start 2023-10-10T18:44:54.579 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b8ad9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7b4067a40 0x7fc7b4157250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.579 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b8ad9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc7b4159250 0x7fc7b4157790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.579 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b8ad9700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b406b7e0 con 0x7fc7b4067e10 2023-10-10T18:44:54.579 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b8ad9700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b406a8f0 con 0x7fc7b4159250 2023-10-10T18:44:54.580 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b8ad9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b406abd0 con 0x7fc7b4067a40 2023-10-10T18:44:54.580 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b37fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7b4067a40 0x7fc7b4157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.580 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b37fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7b4067a40 0x7fc7b4157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44774/0 (socket says 172.21.15.191:44774) 2023-10-10T18:44:54.580 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b37fe700 1 -- 172.21.15.191:0/588351533 learned_addr learned my addr 172.21.15.191:0/588351533 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:54.580 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b3fff700 1 --2- 172.21.15.191:0/588351533 >> v2:172.21.15.175:3300/0 conn(0x7fc7b4159250 0x7fc7b4157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.581 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b37fe700 1 -- 172.21.15.191:0/588351533 >> v2:172.21.15.175:3300/0 conn(0x7fc7b4159250 msgr2=0x7fc7b4157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:54.581 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b37fe700 1 --2- 172.21.15.191:0/588351533 >> v2:172.21.15.175:3300/0 conn(0x7fc7b4159250 0x7fc7b4157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.581 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b37fe700 1 -- 172.21.15.191:0/588351533 >> v1:172.21.15.175:6789/0 conn(0x7fc7b4067e10 legacy=0x7fc7b414f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.581 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b37fe700 1 -- 172.21.15.191:0/588351533 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b406aeb0 con 0x7fc7b4067a40 2023-10-10T18:44:54.581 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b37fe700 1 --2- 172.21.15.191:0/588351533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b4157250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4064fa0 tx=0x7fc7a4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94e77fce0153bfd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:54.582 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.576+0000 7fc7b27fc700 1 -- 172.21.15.191:0/588351533 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4009e70 con 0x7fc7b4067a40 2023-10-10T18:44:54.582 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b27fc700 1 -- 172.21.15.191:0/588351533 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a400b860 con 0x7fc7b4067a40 2023-10-10T18:44:54.582 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b27fc700 1 -- 172.21.15.191:0/588351533 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a400bc00 con 0x7fc7b4067a40 2023-10-10T18:44:54.582 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/588351533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 msgr2=0x7fc7b4157250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.583 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b8ad9700 1 --2- 172.21.15.191:0/588351533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b4157250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4064fa0 tx=0x7fc7a4009250 comp rx=0 tx=0).stop 2023-10-10T18:44:54.583 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/588351533 shutdown_connections 2023-10-10T18:44:54.583 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b8ad9700 1 --2- 172.21.15.191:0/588351533 >> v2:172.21.15.175:3300/0 conn(0x7fc7b4159250 0x7fc7b4157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.583 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b8ad9700 1 --2- 172.21.15.191:0/588351533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b4157250 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.583 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/588351533 >> 172.21.15.191:0/588351533 conn(0x7fc7b406bf10 msgr2=0x7fc7b406c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:54.584 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/588351533 shutdown_connections 2023-10-10T18:44:54.584 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.577+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/588351533 wait complete. 2023-10-10T18:44:54.584 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.578+0000 7fc7b8ad9700 1 Processor -- start 2023-10-10T18:44:54.584 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.578+0000 7fc7b8ad9700 1 -- start start 2023-10-10T18:44:54.587 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b8ad9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b41666b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.587 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b8ad9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7b4067e10 0x7fc7b4147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.587 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b8ad9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b4159250 0x7fc7b40c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.587 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b8ad9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4166220 con 0x7fc7b4067e10 2023-10-10T18:44:54.587 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b8ad9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4149e40 con 0x7fc7b4067a40 2023-10-10T18:44:54.588 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b8ad9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b414a120 con 0x7fc7b4159250 2023-10-10T18:44:54.588 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b37fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b41666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.588 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b37fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b41666b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44780/0 (socket says 172.21.15.191:44780) 2023-10-10T18:44:54.588 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b37fe700 1 -- 172.21.15.191:0/167358163 learned_addr learned my addr 172.21.15.191:0/167358163 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:54.589 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b2ffd700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7b4067e10 0x7fc7b4147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.589 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b3fff700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b4159250 0x7fc7b40c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.589 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b37fe700 1 -- 172.21.15.191:0/167358163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b4159250 msgr2=0x7fc7b40c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.589 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b37fe700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b4159250 0x7fc7b40c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.589 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b37fe700 1 -- 172.21.15.191:0/167358163 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7b4067e10 msgr2=0x7fc7b4147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.590 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b37fe700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7b4067e10 0x7fc7b4147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.590 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.579+0000 7fc7b37fe700 1 -- 172.21.15.191:0/167358163 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40094f0 con 0x7fc7b4067a40 2023-10-10T18:44:54.590 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.580+0000 7fc7b37fe700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b41666b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc7b406ad10 tx=0x7fc7a4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:54.590 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.580+0000 7fc79b7fe700 1 -- 172.21.15.191:0/167358163 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a400bbe0 con 0x7fc7b4067a40 2023-10-10T18:44:54.590 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.580+0000 7fc79b7fe700 1 -- 172.21.15.191:0/167358163 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a4011420 con 0x7fc7b4067a40 2023-10-10T18:44:54.591 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.580+0000 7fc79b7fe700 1 -- 172.21.15.191:0/167358163 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40117e0 con 0x7fc7b4067a40 2023-10-10T18:44:54.591 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.580+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b4165eb0 con 0x7fc7b4067a40 2023-10-10T18:44:54.591 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.580+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b40c8ca0 con 0x7fc7b4067a40 2023-10-10T18:44:54.591 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.580+0000 7fc79b7fe700 1 -- 172.21.15.191:0/167358163 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc7a402e3f0 con 0x7fc7b4067a40 2023-10-10T18:44:54.592 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.580+0000 7fc79b7fe700 1 -- 172.21.15.191:0/167358163 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fc7a40257b0 con 0x7fc7b4067a40 2023-10-10T18:44:54.592 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.581+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b414dcf0 con 0x7fc7b4067a40 2023-10-10T18:44:54.592 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.583+0000 7fc79b7fe700 1 -- 172.21.15.191:0/167358163 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc7a4033060 con 0x7fc7b4067a40 2023-10-10T18:44:54.666 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.666+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a02d3af7-057b-403b-bb65-aca12bdb262b", "id": 1} v 0) v1 -- 0x7fc7b4066f20 con 0x7fc7b4067a40 2023-10-10T18:44:54.672 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.672+0000 7fc79b7fe700 1 -- 172.21.15.191:0/167358163 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a02d3af7-057b-403b-bb65-aca12bdb262b", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc7a4015020 con 0x7fc7b4067a40 2023-10-10T18:44:54.672 INFO:teuthology.orchestra.run.smithi191.stdout:1 2023-10-10T18:44:54.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 msgr2=0x7fc7b41666b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b41666b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc7b406ad10 tx=0x7fc7a4017000 comp rx=0 tx=0).stop 2023-10-10T18:44:54.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 shutdown_connections 2023-10-10T18:44:54.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b4159250 0x7fc7b40c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4067a40 0x7fc7b41666b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 --2- 172.21.15.191:0/167358163 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7b4067e10 0x7fc7b4147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 >> 172.21.15.191:0/167358163 conn(0x7fc7b406bf10 msgr2=0x7fc7b4158850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:54.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 shutdown_connections 2023-10-10T18:44:54.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.673+0000 7fc7b8ad9700 1 -- 172.21.15.191:0/167358163 wait complete. 2023-10-10T18:44:54.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77 2 2023-10-10T18:44:54.801 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.800+0000 7fe9468c5700 1 Processor -- start 2023-10-10T18:44:54.801 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.800+0000 7fe9468c5700 1 -- start start 2023-10-10T18:44:54.803 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe9468c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400cc8a0 0x7fe940156c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.804 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe9468c5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9400c70e0 0x7fe9400aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.804 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe9468c5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe9400ccc70 0x7fe9400b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.804 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe9468c5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe940157160 con 0x7fe9400ccc70 2023-10-10T18:44:54.804 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe9468c5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe940150950 con 0x7fe9400c70e0 2023-10-10T18:44:54.805 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe9468c5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400afac0 con 0x7fe9400cc8a0 2023-10-10T18:44:54.805 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400cc8a0 0x7fe940156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.805 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93f7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9400c70e0 0x7fe9400aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.805 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400cc8a0 0x7fe940156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43012/0 (socket says 172.21.15.191:43012) 2023-10-10T18:44:54.805 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93f7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9400c70e0 0x7fe9400aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44786/0 (socket says 172.21.15.191:44786) 2023-10-10T18:44:54.806 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93ffff700 1 -- 172.21.15.191:0/768669257 learned_addr learned my addr 172.21.15.191:0/768669257 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:54.806 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe944e62700 1 --2- 172.21.15.191:0/768669257 >> v2:172.21.15.175:3300/0 conn(0x7fe9400ccc70 0x7fe9400b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.806 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93f7fe700 1 -- 172.21.15.191:0/768669257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400cc8a0 msgr2=0x7fe940156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:54.806 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93f7fe700 1 --2- 172.21.15.191:0/768669257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400cc8a0 0x7fe940156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.806 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93f7fe700 1 -- 172.21.15.191:0/768669257 >> v2:172.21.15.175:3300/0 conn(0x7fe9400ccc70 msgr2=0x7fe9400b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:54.807 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93f7fe700 1 --2- 172.21.15.191:0/768669257 >> v2:172.21.15.175:3300/0 conn(0x7fe9400ccc70 0x7fe9400b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.807 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93f7fe700 1 -- 172.21.15.191:0/768669257 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9400af4e0 con 0x7fe9400c70e0 2023-10-10T18:44:54.807 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93ffff700 1 --2- 172.21.15.191:0/768669257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400cc8a0 0x7fe940156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:54.807 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe944e62700 1 --2- 172.21.15.191:0/768669257 >> v2:172.21.15.175:3300/0 conn(0x7fe9400ccc70 0x7fe9400b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:44:54.807 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.801+0000 7fe93f7fe700 1 --2- 172.21.15.191:0/768669257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400c70e0 0x7fe9400aff50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe940071590 tx=0x7fe9300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0ef1f58022e290 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:54.807 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe93effd700 1 -- 172.21.15.191:0/768669257 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930011630 con 0x7fe9400c70e0 2023-10-10T18:44:54.808 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe93effd700 1 -- 172.21.15.191:0/768669257 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe930011790 con 0x7fe9400c70e0 2023-10-10T18:44:54.808 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe93effd700 1 -- 172.21.15.191:0/768669257 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930011b30 con 0x7fe9400c70e0 2023-10-10T18:44:54.808 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 -- 172.21.15.191:0/768669257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400c70e0 msgr2=0x7fe9400aff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.808 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 --2- 172.21.15.191:0/768669257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400c70e0 0x7fe9400aff50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe940071590 tx=0x7fe9300091a0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.808 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 -- 172.21.15.191:0/768669257 shutdown_connections 2023-10-10T18:44:54.809 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 --2- 172.21.15.191:0/768669257 >> v2:172.21.15.175:3300/0 conn(0x7fe9400ccc70 0x7fe9400b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.809 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 --2- 172.21.15.191:0/768669257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400cc8a0 0x7fe940156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.809 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 --2- 172.21.15.191:0/768669257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400c70e0 0x7fe9400aff50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.809 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 -- 172.21.15.191:0/768669257 >> 172.21.15.191:0/768669257 conn(0x7fe9400682f0 msgr2=0x7fe9400686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:54.809 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 -- 172.21.15.191:0/768669257 shutdown_connections 2023-10-10T18:44:54.810 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.802+0000 7fe9468c5700 1 -- 172.21.15.191:0/768669257 wait complete. 2023-10-10T18:44:54.810 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.803+0000 7fe9468c5700 1 Processor -- start 2023-10-10T18:44:54.812 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe9468c5700 1 -- start start 2023-10-10T18:44:54.813 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe9468c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400c70e0 0x7fe940153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.813 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe9468c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9400cc8a0 0x7fe940153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.813 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe9468c5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400ccc70 0x7fe9400c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:54.813 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe9468c5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940166680 con 0x7fe9400cc8a0 2023-10-10T18:44:54.813 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe9468c5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400cc490 con 0x7fe9400ccc70 2023-10-10T18:44:54.814 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe9468c5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400c2230 con 0x7fe9400c70e0 2023-10-10T18:44:54.814 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe93ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400c70e0 0x7fe940153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.814 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe93ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400c70e0 0x7fe940153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43018/0 (socket says 172.21.15.191:43018) 2023-10-10T18:44:54.814 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe93ffff700 1 -- 172.21.15.191:0/446423943 learned_addr learned my addr 172.21.15.191:0/446423943 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:54.814 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe944e62700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400ccc70 0x7fe9400c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.815 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.804+0000 7fe93f7fe700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9400cc8a0 0x7fe940153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:54.815 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe93ffff700 1 -- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400ccc70 msgr2=0x7fe9400c26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.815 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe93ffff700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400ccc70 0x7fe9400c26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.815 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe93ffff700 1 -- 172.21.15.191:0/446423943 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9400cc8a0 msgr2=0x7fe940153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.815 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe93ffff700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9400cc8a0 0x7fe940153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.816 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe93ffff700 1 -- 172.21.15.191:0/446423943 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9340094f0 con 0x7fe9400c70e0 2023-10-10T18:44:54.816 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe944e62700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400ccc70 0x7fe9400c26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:54.816 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe93ffff700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400c70e0 0x7fe940153200 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe93400ec80 tx=0x7fe934009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:54.816 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe927fff700 1 -- 172.21.15.191:0/446423943 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93400c850 con 0x7fe9400c70e0 2023-10-10T18:44:54.816 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe927fff700 1 -- 172.21.15.191:0/446423943 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93400c9b0 con 0x7fe9400c70e0 2023-10-10T18:44:54.816 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe930009420 con 0x7fe9400c70e0 2023-10-10T18:44:54.817 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe927fff700 1 -- 172.21.15.191:0/446423943 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93400cd50 con 0x7fe9400c70e0 2023-10-10T18:44:54.817 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9401663a0 con 0x7fe9400c70e0 2023-10-10T18:44:54.817 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe927fff700 1 -- 172.21.15.191:0/446423943 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe934013900 con 0x7fe9400c70e0 2023-10-10T18:44:54.817 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.805+0000 7fe927fff700 1 -- 172.21.15.191:0/446423943 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fe93401ca60 con 0x7fe9400c70e0 2023-10-10T18:44:54.817 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.806+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9400b07c0 con 0x7fe9400c70e0 2023-10-10T18:44:54.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.808+0000 7fe927fff700 1 -- 172.21.15.191:0/446423943 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe934021060 con 0x7fe9400c70e0 2023-10-10T18:44:54.818 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:54.817+0000 7fe0b1dc4180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T18:44:54.847 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:54.847+0000 7f3fe3830180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T18:44:54.890 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.889+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77", "id": 2} v 0) v1 -- 0x7fe94006a950 con 0x7fe9400c70e0 2023-10-10T18:44:54.894 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.893+0000 7fe927fff700 1 -- 172.21.15.191:0/446423943 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe934017070 con 0x7fe9400c70e0 2023-10-10T18:44:54.894 INFO:teuthology.orchestra.run.smithi191.stdout:2 2023-10-10T18:44:54.895 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400c70e0 msgr2=0x7fe940153200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:54.895 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400c70e0 0x7fe940153200 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe93400ec80 tx=0x7fe934009250 comp rx=0 tx=0).stop 2023-10-10T18:44:54.896 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 shutdown_connections 2023-10-10T18:44:54.896 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9400ccc70 0x7fe9400c26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.896 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9400c70e0 0x7fe940153200 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.896 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 --2- 172.21.15.191:0/446423943 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9400cc8a0 0x7fe940153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:54.896 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 >> 172.21.15.191:0/446423943 conn(0x7fe9400682f0 msgr2=0x7fe9400686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:54.897 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 shutdown_connections 2023-10-10T18:44:54.897 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:54.894+0000 7fe9468c5700 1 -- 172.21.15.191:0/446423943 wait complete. 2023-10-10T18:44:54.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 65e7c04a-f10b-494c-8c18-17f8ecb9b485 3 2023-10-10T18:44:54.948 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:54.949+0000 7fe0b1dc4180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T18:44:54.984 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:54.984+0000 7f3fe3830180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T18:44:55.021 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.020+0000 7fda7bcaf700 1 Processor -- start 2023-10-10T18:44:55.021 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.020+0000 7fda7bcaf700 1 -- start start 2023-10-10T18:44:55.023 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda7bcaf700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda7406dcc0 0x7fda74156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.024 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda7bcaf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.024 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda7bcaf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fda7406e4e0 0x7fda74150240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.024 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda7bcaf700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7406f670 con 0x7fda7406e4e0 2023-10-10T18:44:55.024 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda7bcaf700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda74156e70 con 0x7fda7406dcc0 2023-10-10T18:44:55.025 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda7bcaf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda74063870 con 0x7fda7406e090 2023-10-10T18:44:55.025 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda79a4b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda7406dcc0 0x7fda74156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.025 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda79a4b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda7406dcc0 0x7fda74156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44798/0 (socket says 172.21.15.191:44798) 2023-10-10T18:44:55.025 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda7924a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.025 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda79a4b700 1 -- 172.21.15.191:0/1559233216 learned_addr learned my addr 172.21.15.191:0/1559233216 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:55.026 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.021+0000 7fda7a24c700 1 --2- 172.21.15.191:0/1559233216 >> v2:172.21.15.175:3300/0 conn(0x7fda7406e4e0 0x7fda74150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.026 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda79a4b700 1 -- 172.21.15.191:0/1559233216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 msgr2=0x7fda7414fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:55.026 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda79a4b700 1 --2- 172.21.15.191:0/1559233216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.026 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda79a4b700 1 -- 172.21.15.191:0/1559233216 >> v2:172.21.15.175:3300/0 conn(0x7fda7406e4e0 msgr2=0x7fda74150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:55.026 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda79a4b700 1 --2- 172.21.15.191:0/1559233216 >> v2:172.21.15.175:3300/0 conn(0x7fda7406e4e0 0x7fda74150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda79a4b700 1 -- 172.21.15.191:0/1559233216 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda74063590 con 0x7fda7406dcc0 2023-10-10T18:44:55.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda7924a700 1 --2- 172.21.15.191:0/1559233216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:55.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda79a4b700 1 --2- 172.21.15.191:0/1559233216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406dcc0 0x7fda74156930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda74067240 tx=0x7fda64009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8633a41938125c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda78a49700 1 -- 172.21.15.191:0/1559233216 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6400b640 con 0x7fda7406dcc0 2023-10-10T18:44:55.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda78a49700 1 -- 172.21.15.191:0/1559233216 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6400b7a0 con 0x7fda7406dcc0 2023-10-10T18:44:55.028 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda78a49700 1 -- 172.21.15.191:0/1559233216 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6400bb40 con 0x7fda7406dcc0 2023-10-10T18:44:55.028 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda7bcaf700 1 -- 172.21.15.191:0/1559233216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406dcc0 msgr2=0x7fda74156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.028 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.022+0000 7fda7bcaf700 1 --2- 172.21.15.191:0/1559233216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406dcc0 0x7fda74156930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda74067240 tx=0x7fda64009250 comp rx=0 tx=0).stop 2023-10-10T18:44:55.028 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.023+0000 7fda7bcaf700 1 -- 172.21.15.191:0/1559233216 shutdown_connections 2023-10-10T18:44:55.028 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.023+0000 7fda7bcaf700 1 --2- 172.21.15.191:0/1559233216 >> v2:172.21.15.175:3300/0 conn(0x7fda7406e4e0 0x7fda74150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.029 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.023+0000 7fda7bcaf700 1 --2- 172.21.15.191:0/1559233216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406dcc0 0x7fda74156930 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.029 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.023+0000 7fda7bcaf700 1 --2- 172.21.15.191:0/1559233216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.029 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.023+0000 7fda7bcaf700 1 -- 172.21.15.191:0/1559233216 >> 172.21.15.191:0/1559233216 conn(0x7fda740c8cb0 msgr2=0x7fda740c90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.029 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.023+0000 7fda7bcaf700 1 -- 172.21.15.191:0/1559233216 shutdown_connections 2023-10-10T18:44:55.030 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.023+0000 7fda7bcaf700 1 -- 172.21.15.191:0/1559233216 wait complete. 2023-10-10T18:44:55.030 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:55.024+0000 7fe0b1dc4180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T18:44:55.033 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7bcaf700 1 Processor -- start 2023-10-10T18:44:55.033 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7bcaf700 1 -- start start 2023-10-10T18:44:55.034 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7bcaf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda7406dcc0 0x7fda740c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.034 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7bcaf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.034 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7bcaf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406e4e0 0x7fda74165b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.034 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7bcaf700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda741493f0 con 0x7fda7406dcc0 2023-10-10T18:44:55.034 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7bcaf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda741498e0 con 0x7fda7406e4e0 2023-10-10T18:44:55.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7bcaf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda74169b00 con 0x7fda7406e090 2023-10-10T18:44:55.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7924a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7a24c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406e4e0 0x7fda74165b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7924a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43032/0 (socket says 172.21.15.191:43032) 2023-10-10T18:44:55.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda7924a700 1 -- 172.21.15.191:0/3807507216 learned_addr learned my addr 172.21.15.191:0/3807507216 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:55.036 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.024+0000 7fda79a4b700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda7406dcc0 0x7fda740c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.036 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7924a700 1 -- 172.21.15.191:0/3807507216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406e4e0 msgr2=0x7fda74165b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.036 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7924a700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406e4e0 0x7fda74165b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.036 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7924a700 1 -- 172.21.15.191:0/3807507216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda7406dcc0 msgr2=0x7fda740c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.036 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7924a700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda7406dcc0 0x7fda740c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7924a700 1 -- 172.21.15.191:0/3807507216 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda70009420 con 0x7fda7406e090 2023-10-10T18:44:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda79a4b700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda7406dcc0 0x7fda740c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7a24c700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406e4e0 0x7fda74165b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7924a700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414cb50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fda7000ec80 tx=0x7fda700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda69ffb700 1 -- 172.21.15.191:0/3807507216 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000c920 con 0x7fda7406e090 2023-10-10T18:44:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda69ffb700 1 -- 172.21.15.191:0/3807507216 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000ca80 con 0x7fda7406e090 2023-10-10T18:44:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda640094f0 con 0x7fda7406e090 2023-10-10T18:44:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda69ffb700 1 -- 172.21.15.191:0/3807507216 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000ce00 con 0x7fda7406e090 2023-10-10T18:44:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.025+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda741691d0 con 0x7fda7406e090 2023-10-10T18:44:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.026+0000 7fda69ffb700 1 -- 172.21.15.191:0/3807507216 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fda70013900 con 0x7fda7406e090 2023-10-10T18:44:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.026+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda74067570 con 0x7fda7406e090 2023-10-10T18:44:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.026+0000 7fda69ffb700 1 -- 172.21.15.191:0/3807507216 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7fda70013cf0 con 0x7fda7406e090 2023-10-10T18:44:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.030+0000 7fda69ffb700 1 -- 172.21.15.191:0/3807507216 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fda70021060 con 0x7fda7406e090 2023-10-10T18:44:55.064 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:55.064+0000 7f3fe3830180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T18:44:55.112 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.112+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "65e7c04a-f10b-494c-8c18-17f8ecb9b485", "id": 3} v 0) v1 -- 0x7fda7406d120 con 0x7fda7406e090 2023-10-10T18:44:55.116 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.116+0000 7fda69ffb700 1 -- 172.21.15.191:0/3807507216 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "65e7c04a-f10b-494c-8c18-17f8ecb9b485", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fda70017070 con 0x7fda7406e090 2023-10-10T18:44:55.117 INFO:teuthology.orchestra.run.smithi191.stdout:3 2023-10-10T18:44:55.118 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 msgr2=0x7fda7414cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.118 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414cb50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fda7000ec80 tx=0x7fda700091a0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.119 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 shutdown_connections 2023-10-10T18:44:55.119 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda7406e4e0 0x7fda74165b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.119 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda7406dcc0 0x7fda740c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.119 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 --2- 172.21.15.191:0/3807507216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda7406e090 0x7fda7414cb50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.119 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 >> 172.21.15.191:0/3807507216 conn(0x7fda740c8cb0 msgr2=0x7fda74149d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.120 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 shutdown_connections 2023-10-10T18:44:55.120 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.117+0000 7fda7bcaf700 1 -- 172.21.15.191:0/3807507216 wait complete. 2023-10-10T18:44:55.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 0d263e6f-e89f-4f2f-9f54-b867769fe8f5 4 2023-10-10T18:44:55.209 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:55.209+0000 7fe0b1dc4180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T18:44:55.244 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.243+0000 7f0e08d00700 1 Processor -- start 2023-10-10T18:44:55.244 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.243+0000 7f0e08d00700 1 -- start start 2023-10-10T18:44:55.247 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e08d00700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0e040c2b50 0x7f0e04069010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.247 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e08d00700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2f20 0x7f0e040c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.247 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e08d00700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0e0406f1d0 0x7f0e040c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.248 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e08d00700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e041535e0 con 0x7f0e040c2b50 2023-10-10T18:44:55.248 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e08d00700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e04069af0 con 0x7f0e0406f1d0 2023-10-10T18:44:55.248 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e08d00700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e04152cf0 con 0x7f0e040c2f20 2023-10-10T18:44:55.248 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e01d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2f20 0x7f0e040c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.248 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e02d9d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0e0406f1d0 0x7f0e040c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.249 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e02d9d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0e0406f1d0 0x7f0e040c62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44814/0 (socket says 172.21.15.191:44814) 2023-10-10T18:44:55.249 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e01d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2f20 0x7f0e040c5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43036/0 (socket says 172.21.15.191:43036) 2023-10-10T18:44:55.249 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e02d9d700 1 -- 172.21.15.191:0/3862234372 learned_addr learned my addr 172.21.15.191:0/3862234372 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:55.249 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.244+0000 7f0e0259c700 1 --2- 172.21.15.191:0/3862234372 >> v2:172.21.15.175:3300/0 conn(0x7f0e040c2b50 0x7f0e04069010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.250 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e01d9b700 1 -- 172.21.15.191:0/3862234372 >> v2:172.21.15.191:3300/0 conn(0x7f0e0406f1d0 msgr2=0x7f0e040c62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:55.250 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e01d9b700 1 --2- 172.21.15.191:0/3862234372 >> v2:172.21.15.191:3300/0 conn(0x7f0e0406f1d0 0x7f0e040c62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.250 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e01d9b700 1 -- 172.21.15.191:0/3862234372 >> v2:172.21.15.175:3300/0 conn(0x7f0e040c2b50 msgr2=0x7f0e04069010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:55.250 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e01d9b700 1 --2- 172.21.15.191:0/3862234372 >> v2:172.21.15.175:3300/0 conn(0x7f0e040c2b50 0x7f0e04069010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.250 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e01d9b700 1 -- 172.21.15.191:0/3862234372 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e04153a70 con 0x7f0e040c2f20 2023-10-10T18:44:55.251 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e02d9d700 1 --2- 172.21.15.191:0/3862234372 >> v2:172.21.15.191:3300/0 conn(0x7f0e0406f1d0 0x7f0e040c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:55.251 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e01d9b700 1 --2- 172.21.15.191:0/3862234372 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2f20 0x7f0e040c5d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0e0406f160 tx=0x7f0df40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c50b9d0a66badd7f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.251 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e0159a700 1 -- 172.21.15.191:0/3862234372 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4011630 con 0x7f0e040c2f20 2023-10-10T18:44:55.251 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e0159a700 1 -- 172.21.15.191:0/3862234372 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df40117b0 con 0x7f0e040c2f20 2023-10-10T18:44:55.251 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.245+0000 7f0e0159a700 1 -- 172.21.15.191:0/3862234372 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4011b70 con 0x7f0e040c2f20 2023-10-10T18:44:55.252 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 -- 172.21.15.191:0/3862234372 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2f20 msgr2=0x7f0e040c5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.252 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 --2- 172.21.15.191:0/3862234372 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2f20 0x7f0e040c5d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0e0406f160 tx=0x7f0df40091a0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.252 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 -- 172.21.15.191:0/3862234372 shutdown_connections 2023-10-10T18:44:55.252 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 --2- 172.21.15.191:0/3862234372 >> v2:172.21.15.191:3300/0 conn(0x7f0e0406f1d0 0x7f0e040c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.253 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 --2- 172.21.15.191:0/3862234372 >> v2:172.21.15.175:3300/0 conn(0x7f0e040c2b50 0x7f0e04069010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.253 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 --2- 172.21.15.191:0/3862234372 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2f20 0x7f0e040c5d70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.253 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 -- 172.21.15.191:0/3862234372 >> 172.21.15.191:0/3862234372 conn(0x7f0e04063230 msgr2=0x7f0e04063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.253 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 -- 172.21.15.191:0/3862234372 shutdown_connections 2023-10-10T18:44:55.253 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.246+0000 7f0e08d00700 1 -- 172.21.15.191:0/3862234372 wait complete. 2023-10-10T18:44:55.254 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e08d00700 1 Processor -- start 2023-10-10T18:44:55.254 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e08d00700 1 -- start start 2023-10-10T18:44:55.256 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e08d00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e0406f1d0 0x7f0e041465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.257 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e08d00700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2b50 0x7f0e04147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.257 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e08d00700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0e040c2f20 0x7f0e04164f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.257 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e08d00700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e04154660 con 0x7f0e040c2f20 2023-10-10T18:44:55.257 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e08d00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0414ec80 con 0x7f0e0406f1d0 2023-10-10T18:44:55.258 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e08d00700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0414e070 con 0x7f0e040c2b50 2023-10-10T18:44:55.258 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e01d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2b50 0x7f0e04147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.258 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e0259c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e0406f1d0 0x7f0e041465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.258 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e01d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2b50 0x7f0e04147ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43044/0 (socket says 172.21.15.191:43044) 2023-10-10T18:44:55.259 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e01d9b700 1 -- 172.21.15.191:0/1607015027 learned_addr learned my addr 172.21.15.191:0/1607015027 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:55.259 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.247+0000 7f0e0259c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e0406f1d0 0x7f0e041465e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44816/0 (socket says 172.21.15.191:44816) 2023-10-10T18:44:55.259 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e02d9d700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0e040c2f20 0x7f0e04164f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.259 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e01d9b700 1 -- 172.21.15.191:0/1607015027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e0406f1d0 msgr2=0x7f0e041465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.259 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e01d9b700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e0406f1d0 0x7f0e041465e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.260 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e01d9b700 1 -- 172.21.15.191:0/1607015027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0e040c2f20 msgr2=0x7f0e04164f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.260 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e01d9b700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0e040c2f20 0x7f0e04164f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.260 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e01d9b700 1 -- 172.21.15.191:0/1607015027 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc009ae0 con 0x7f0e040c2b50 2023-10-10T18:44:55.260 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e0259c700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e0406f1d0 0x7f0e041465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:55.261 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e01d9b700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2b50 0x7f0e04147ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0e04152e30 tx=0x7f0df4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.261 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0df27fc700 1 -- 172.21.15.191:0/1607015027 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df401f070 con 0x7f0e040c2b50 2023-10-10T18:44:55.261 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0df27fc700 1 -- 172.21.15.191:0/1607015027 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df4000bc0 con 0x7f0e040c2b50 2023-10-10T18:44:55.261 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df4009420 con 0x7f0e040c2b50 2023-10-10T18:44:55.261 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0df27fc700 1 -- 172.21.15.191:0/1607015027 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4011b50 con 0x7f0e040c2b50 2023-10-10T18:44:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e04165e80 con 0x7f0e040c2b50 2023-10-10T18:44:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0df27fc700 1 -- 172.21.15.191:0/1607015027 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0df4023070 con 0x7f0e040c2b50 2023-10-10T18:44:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.248+0000 7f0df27fc700 1 -- 172.21.15.191:0/1607015027 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f0df400b910 con 0x7f0e040c2b50 2023-10-10T18:44:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.249+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e04165b80 con 0x7f0e040c2b50 2023-10-10T18:44:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.251+0000 7f0df27fc700 1 -- 172.21.15.191:0/1607015027 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0df402a070 con 0x7f0e040c2b50 2023-10-10T18:44:55.263 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:55.256+0000 7f3fe3830180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T18:44:55.333 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.332+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0d263e6f-e89f-4f2f-9f54-b867769fe8f5", "id": 4} v 0) v1 -- 0x7f0e04067570 con 0x7f0e040c2b50 2023-10-10T18:44:55.336 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.336+0000 7f0df27fc700 1 -- 172.21.15.191:0/1607015027 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0d263e6f-e89f-4f2f-9f54-b867769fe8f5", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0df4015020 con 0x7f0e040c2b50 2023-10-10T18:44:55.337 INFO:teuthology.orchestra.run.smithi191.stdout:4 2023-10-10T18:44:55.338 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2b50 msgr2=0x7f0e04147ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.338 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2b50 0x7f0e04147ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0e04152e30 tx=0x7f0df4017000 comp rx=0 tx=0).stop 2023-10-10T18:44:55.339 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 shutdown_connections 2023-10-10T18:44:55.339 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0e040c2f20 0x7f0e04164f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.339 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e0406f1d0 0x7f0e041465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.339 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 --2- 172.21.15.191:0/1607015027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e040c2b50 0x7f0e04147ce0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.339 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 >> 172.21.15.191:0/1607015027 conn(0x7f0e04063230 msgr2=0x7f0e040677f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.340 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 shutdown_connections 2023-10-10T18:44:55.340 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.337+0000 7f0e08d00700 1 -- 172.21.15.191:0/1607015027 wait complete. 2023-10-10T18:44:55.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 98e51172-fbcf-474f-9862-d39dc746a151 5 2023-10-10T18:44:55.381 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:55.381+0000 7fe0b1dc4180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T18:44:55.435 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:55.435+0000 7f3fe3830180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T18:44:55.458 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:55.458+0000 7fe0b1dc4180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T18:44:55.463 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.463+0000 7f09099df700 1 Processor -- start 2023-10-10T18:44:55.463 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.463+0000 7f09099df700 1 -- start start 2023-10-10T18:44:55.466 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.463+0000 7f09099df700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c6530 0x7f09040bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.466 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.463+0000 7f09099df700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f09040c6900 0x7f09040ca380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.466 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.463+0000 7f09099df700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0904157960 con 0x7f09040c4520 2023-10-10T18:44:55.467 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.463+0000 7f09099df700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09040c12e0 con 0x7f09040c6900 2023-10-10T18:44:55.467 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.463+0000 7f09099df700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0904154300 con 0x7f09040c6530 2023-10-10T18:44:55.467 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.463+0000 7f09089dd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c6530 0x7f09040bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.467 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09089dd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c6530 0x7f09040bf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43048/0 (socket says 172.21.15.191:43048) 2023-10-10T18:44:55.468 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09089dd700 1 -- 172.21.15.191:0/794735882 learned_addr learned my addr 172.21.15.191:0/794735882 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:55.468 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09091de700 1 --2- 172.21.15.191:0/794735882 >> v2:172.21.15.175:3300/0 conn(0x7f09040c6900 0x7f09040ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.468 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09089dd700 1 -- 172.21.15.191:0/794735882 >> v2:172.21.15.175:3300/0 conn(0x7f09040c6900 msgr2=0x7f09040ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:55.468 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09089dd700 1 --2- 172.21.15.191:0/794735882 >> v2:172.21.15.175:3300/0 conn(0x7f09040c6900 0x7f09040ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.468 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09089dd700 1 -- 172.21.15.191:0/794735882 >> v1:172.21.15.175:6789/0 conn(0x7f09040c4520 legacy=0x7f0904153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.469 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09089dd700 1 -- 172.21.15.191:0/794735882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040c0d20 con 0x7f09040c6530 2023-10-10T18:44:55.469 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09091de700 1 --2- 172.21.15.191:0/794735882 >> v2:172.21.15.175:3300/0 conn(0x7f09040c6900 0x7f09040ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:44:55.469 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09089dd700 1 --2- 172.21.15.191:0/794735882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c6530 0x7f09040bf1b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f08f80099b0 tx=0x7f08f8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54b7b4107e3fdbc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.469 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09037fe700 1 -- 172.21.15.191:0/794735882 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f800b8f0 con 0x7f09040c6530 2023-10-10T18:44:55.469 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09037fe700 1 -- 172.21.15.191:0/794735882 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f800ba50 con 0x7f09040c6530 2023-10-10T18:44:55.470 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.464+0000 7f09037fe700 1 -- 172.21.15.191:0/794735882 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f800bdf0 con 0x7f09040c6530 2023-10-10T18:44:55.470 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.465+0000 7f09099df700 1 -- 172.21.15.191:0/794735882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c6530 msgr2=0x7f09040bf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.470 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.465+0000 7f09099df700 1 --2- 172.21.15.191:0/794735882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c6530 0x7f09040bf1b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f08f80099b0 tx=0x7f08f8009250 comp rx=0 tx=0).stop 2023-10-10T18:44:55.471 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.465+0000 7f09099df700 1 -- 172.21.15.191:0/794735882 shutdown_connections 2023-10-10T18:44:55.471 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.465+0000 7f09099df700 1 --2- 172.21.15.191:0/794735882 >> v2:172.21.15.175:3300/0 conn(0x7f09040c6900 0x7f09040ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.471 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.465+0000 7f09099df700 1 --2- 172.21.15.191:0/794735882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c6530 0x7f09040bf1b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.471 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.465+0000 7f09099df700 1 -- 172.21.15.191:0/794735882 >> 172.21.15.191:0/794735882 conn(0x7f090406cf90 msgr2=0x7f090406d390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.471 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.465+0000 7f09099df700 1 -- 172.21.15.191:0/794735882 shutdown_connections 2023-10-10T18:44:55.472 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.465+0000 7f09099df700 1 -- 172.21.15.191:0/794735882 wait complete. 2023-10-10T18:44:55.472 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.466+0000 7f09099df700 1 Processor -- start 2023-10-10T18:44:55.472 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.466+0000 7f09099df700 1 -- start start 2023-10-10T18:44:55.474 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09099df700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c4520 0x7f09040a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.475 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09099df700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09040c6530 0x7f0904153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.475 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09099df700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09040c6900 0x7f09040ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.475 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09099df700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09040c39c0 con 0x7f09040c6900 2023-10-10T18:44:55.475 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09099df700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09040af330 con 0x7f09040c6530 2023-10-10T18:44:55.475 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09099df700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09041683b0 con 0x7f09040c4520 2023-10-10T18:44:55.476 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09089dd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c4520 0x7f09040a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.476 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09089dd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c4520 0x7f09040a64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43052/0 (socket says 172.21.15.191:43052) 2023-10-10T18:44:55.476 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0903fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09040c6530 0x7f0904153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.476 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09089dd700 1 -- 172.21.15.191:0/1813515752 learned_addr learned my addr 172.21.15.191:0/1813515752 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:55.477 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09091de700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09040c6900 0x7f09040ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.477 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0903fff700 1 -- 172.21.15.191:0/1813515752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c4520 msgr2=0x7f09040a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.477 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0903fff700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c4520 0x7f09040a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.477 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0903fff700 1 -- 172.21.15.191:0/1813515752 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09040c6900 msgr2=0x7f09040ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.477 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0903fff700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09040c6900 0x7f09040ae200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.478 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0903fff700 1 -- 172.21.15.191:0/1813515752 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f80094f0 con 0x7f09040c6530 2023-10-10T18:44:55.478 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09089dd700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c4520 0x7f09040a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:55.478 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0903fff700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09040c6530 0x7f0904153570 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f08f000c4f0 tx=0x7f08f0007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.478 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0900ff9700 1 -- 172.21.15.191:0/1813515752 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f000ce20 con 0x7f09040c6530 2023-10-10T18:44:55.479 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0900ff9700 1 -- 172.21.15.191:0/1813515752 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f0011590 con 0x7f09040c6530 2023-10-10T18:44:55.479 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09041677a0 con 0x7f09040c6530 2023-10-10T18:44:55.479 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0900ff9700 1 -- 172.21.15.191:0/1813515752 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f0011950 con 0x7f09040c6530 2023-10-10T18:44:55.479 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0904167d60 con 0x7f09040c6530 2023-10-10T18:44:55.479 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.467+0000 7f0900ff9700 1 -- 172.21.15.191:0/1813515752 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f08f001c440 con 0x7f09040c6530 2023-10-10T18:44:55.480 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.468+0000 7f0900ff9700 1 -- 172.21.15.191:0/1813515752 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f08f0000f90 con 0x7f09040c6530 2023-10-10T18:44:55.480 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.468+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09040afec0 con 0x7f09040c6530 2023-10-10T18:44:55.480 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.470+0000 7f0900ff9700 1 -- 172.21.15.191:0/1813515752 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f08f0017070 con 0x7f09040c6530 2023-10-10T18:44:55.515 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:55.515+0000 7f3fe3830180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T18:44:55.552 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.552+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "98e51172-fbcf-474f-9862-d39dc746a151", "id": 5} v 0) v1 -- 0x7f09041684f0 con 0x7f09040c6530 2023-10-10T18:44:55.556 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.556+0000 7f0900ff9700 1 -- 172.21.15.191:0/1813515752 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98e51172-fbcf-474f-9862-d39dc746a151", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f08f001c920 con 0x7f09040c6530 2023-10-10T18:44:55.556 INFO:teuthology.orchestra.run.smithi191.stdout:5 2023-10-10T18:44:55.558 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09040c6530 msgr2=0x7f0904153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.558 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09040c6530 0x7f0904153570 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f08f000c4f0 tx=0x7f08f0007ec0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.558 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 shutdown_connections 2023-10-10T18:44:55.558 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09040c6900 0x7f09040ae200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.558 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09040c4520 0x7f09040a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.559 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 --2- 172.21.15.191:0/1813515752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09040c6530 0x7f0904153570 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.559 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 >> 172.21.15.191:0/1813515752 conn(0x7f090406cf90 msgr2=0x7f0904150df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.559 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 shutdown_connections 2023-10-10T18:44:55.559 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.557+0000 7f09099df700 1 -- 172.21.15.191:0/1813515752 wait complete. 2023-10-10T18:44:55.570 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 5f3ece06-0c14-4bf5-b346-bba2525a7167 6 2023-10-10T18:44:55.604 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:55.605+0000 7fe0b1dc4180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T18:44:55.657 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:55.656+0000 7f3fe3830180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T18:44:55.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.685+0000 7f2fcb924700 1 Processor -- start 2023-10-10T18:44:55.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.685+0000 7f2fcb924700 1 -- start start 2023-10-10T18:44:55.689 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fcb924700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40cc8a0 0x7f2fc4156c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.689 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fcb924700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2fc40ccc70 0x7f2fc4158af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.689 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fcb924700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc4150950 con 0x7f2fc40c70e0 2023-10-10T18:44:55.689 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fcb924700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc4157160 con 0x7f2fc40ccc70 2023-10-10T18:44:55.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fcb924700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40afac0 con 0x7f2fc40cc8a0 2023-10-10T18:44:55.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fc9ec1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2fc40ccc70 0x7f2fc4158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fc96c0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40cc8a0 0x7f2fc4156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fc8ebf700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f2fc40c70e0 0x7f2fc40aff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:33950/0 (socket says 172.21.15.191:33950) 2023-10-10T18:44:55.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.686+0000 7f2fc8ebf700 1 -- 172.21.15.191:0/276514155 learned_addr learned my addr 172.21.15.191:0/276514155 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-10-10T18:44:55.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fc9ec1700 1 -- 172.21.15.191:0/276514155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40cc8a0 msgr2=0x7f2fc4156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:55.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fc9ec1700 1 --2- 172.21.15.191:0/276514155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40cc8a0 0x7f2fc4156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fc9ec1700 1 -- 172.21.15.191:0/276514155 >> v1:172.21.15.191:6789/0 conn(0x7f2fc40c70e0 legacy=0x7f2fc40aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fc9ec1700 1 -- 172.21.15.191:0/276514155 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40af4e0 con 0x7f2fc40ccc70 2023-10-10T18:44:55.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fbbfff700 1 -- 172.21.15.191:0/276514155 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410981345 0 0) 0x7f2fc4150950 con 0x7f2fc40c70e0 2023-10-10T18:44:55.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fc96c0700 1 --2- 172.21.15.191:0/276514155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40cc8a0 0x7f2fc4156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:55.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fc9ec1700 1 --2- 172.21.15.191:0/276514155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40ccc70 0x7f2fc4158af0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009e60 tx=0x7f2fbc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb059d8c6f413d24 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.693 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fbbfff700 1 -- 172.21.15.191:0/276514155 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc00e070 con 0x7f2fc40ccc70 2023-10-10T18:44:55.693 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fbbfff700 1 -- 172.21.15.191:0/276514155 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc00bb60 con 0x7f2fc40ccc70 2023-10-10T18:44:55.693 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fbbfff700 1 -- 172.21.15.191:0/276514155 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc012430 con 0x7f2fc40ccc70 2023-10-10T18:44:55.694 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fcb924700 1 -- 172.21.15.191:0/276514155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40ccc70 msgr2=0x7f2fc4158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.694 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fcb924700 1 --2- 172.21.15.191:0/276514155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40ccc70 0x7f2fc4158af0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009e60 tx=0x7f2fbc009780 comp rx=0 tx=0).stop 2023-10-10T18:44:55.694 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fcb924700 1 -- 172.21.15.191:0/276514155 shutdown_connections 2023-10-10T18:44:55.694 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fcb924700 1 --2- 172.21.15.191:0/276514155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40ccc70 0x7f2fc4158af0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.695 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fcb924700 1 --2- 172.21.15.191:0/276514155 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40cc8a0 0x7f2fc4156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.695 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.687+0000 7f2fcb924700 1 -- 172.21.15.191:0/276514155 >> 172.21.15.191:0/276514155 conn(0x7f2fc40682f0 msgr2=0x7f2fc40686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.695 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.688+0000 7f2fcb924700 1 -- 172.21.15.191:0/276514155 shutdown_connections 2023-10-10T18:44:55.695 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.688+0000 7f2fcb924700 1 -- 172.21.15.191:0/276514155 wait complete. 2023-10-10T18:44:55.695 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.689+0000 7f2fcb924700 1 Processor -- start 2023-10-10T18:44:55.695 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.689+0000 7f2fcb924700 1 -- start start 2023-10-10T18:44:55.698 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.689+0000 7f2fcb924700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fc40c70e0 0x7f2fc4150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.698 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.689+0000 7f2fcb924700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40cc8a0 0x7f2fc40c3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.699 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.689+0000 7f2fcb924700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40ccc70 0x7f2fc414dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.699 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.689+0000 7f2fcb924700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc41514b0 con 0x7f2fc40c70e0 2023-10-10T18:44:55.699 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.689+0000 7f2fcb924700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40c6560 con 0x7f2fc40cc8a0 2023-10-10T18:44:55.699 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.689+0000 7f2fcb924700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40c4eb0 con 0x7f2fc40ccc70 2023-10-10T18:44:55.700 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40ccc70 0x7f2fc414dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.700 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40ccc70 0x7f2fc414dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43066/0 (socket says 172.21.15.191:43066) 2023-10-10T18:44:55.700 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 -- 172.21.15.191:0/3011278352 learned_addr learned my addr 172.21.15.191:0/3011278352 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:55.700 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc8ebf700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40cc8a0 0x7f2fc40c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.700 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc96c0700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fc40c70e0 0x7f2fc4150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.700 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 -- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40cc8a0 msgr2=0x7f2fc40c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.701 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40cc8a0 0x7f2fc40c3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.701 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 -- 172.21.15.191:0/3011278352 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fc40c70e0 msgr2=0x7f2fc4150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.701 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fc40c70e0 0x7f2fc4150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 -- 172.21.15.191:0/3011278352 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb4009420 con 0x7f2fc40ccc70 2023-10-10T18:44:55.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc8ebf700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40cc8a0 0x7f2fc40c3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:55.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fc9ec1700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40ccc70 0x7f2fc414dc50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00b6c0 tx=0x7f2fbc019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fb97fa700 1 -- 172.21.15.191:0/3011278352 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc012e90 con 0x7f2fc40ccc70 2023-10-10T18:44:55.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fb97fa700 1 -- 172.21.15.191:0/3011278352 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc027470 con 0x7f2fc40ccc70 2023-10-10T18:44:55.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fb97fa700 1 -- 172.21.15.191:0/3011278352 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc027830 con 0x7f2fc40ccc70 2023-10-10T18:44:55.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fbc009a20 con 0x7f2fc40ccc70 2023-10-10T18:44:55.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc40c5be0 con 0x7f2fc40ccc70 2023-10-10T18:44:55.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fb97fa700 1 -- 172.21.15.191:0/3011278352 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2fbc024070 con 0x7f2fc40ccc70 2023-10-10T18:44:55.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.690+0000 7f2fb97fa700 1 -- 172.21.15.191:0/3011278352 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f2fbc01dbb0 con 0x7f2fc40ccc70 2023-10-10T18:44:55.705 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.691+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc406a950 con 0x7f2fc40ccc70 2023-10-10T18:44:55.705 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.693+0000 7f2fb97fa700 1 -- 172.21.15.191:0/3011278352 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2fbc02c020 con 0x7f2fc40ccc70 2023-10-10T18:44:55.706 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:55.700+0000 7fe0b1dc4180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T18:44:55.755 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:55.755+0000 7f3fe3830180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T18:44:55.775 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.775+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5f3ece06-0c14-4bf5-b346-bba2525a7167", "id": 6} v 0) v1 -- 0x7f2fc40b1f30 con 0x7f2fc40ccc70 2023-10-10T18:44:55.782 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.781+0000 7f2fb97fa700 1 -- 172.21.15.191:0/3011278352 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5f3ece06-0c14-4bf5-b346-bba2525a7167", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2fbc01de50 con 0x7f2fc40ccc70 2023-10-10T18:44:55.782 INFO:teuthology.orchestra.run.smithi191.stdout:6 2023-10-10T18:44:55.783 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.782+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40ccc70 msgr2=0x7f2fc414dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.784 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.782+0000 7f2fcb924700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40ccc70 0x7f2fc414dc50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00b6c0 tx=0x7f2fbc019000 comp rx=0 tx=0).stop 2023-10-10T18:44:55.784 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.783+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 shutdown_connections 2023-10-10T18:44:55.784 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.783+0000 7f2fcb924700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2fc40ccc70 0x7f2fc414dc50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.784 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.783+0000 7f2fcb924700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fc40c70e0 0x7f2fc4150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.785 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.783+0000 7f2fcb924700 1 --2- 172.21.15.191:0/3011278352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2fc40cc8a0 0x7f2fc40c3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.785 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.783+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 >> 172.21.15.191:0/3011278352 conn(0x7f2fc40682f0 msgr2=0x7f2fc40ca620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.785 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.783+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 shutdown_connections 2023-10-10T18:44:55.785 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.783+0000 7f2fcb924700 1 -- 172.21.15.191:0/3011278352 wait complete. 2023-10-10T18:44:55.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 194ceb5a-d817-4197-bb72-672205ca3331 7 2023-10-10T18:44:55.908 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.908+0000 7f22cd432700 1 Processor -- start 2023-10-10T18:44:55.908 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.908+0000 7f22cd432700 1 -- start start 2023-10-10T18:44:55.911 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.908+0000 7f22cd432700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f22c806eff0 0x7f22c80c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.911 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.908+0000 7f22cd432700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c806ca20 con 0x7f22c8156590 2023-10-10T18:44:55.911 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.908+0000 7f22cd432700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c806cd00 con 0x7f22c806dd00 2023-10-10T18:44:55.911 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.908+0000 7f22cd432700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c80c38e0 con 0x7f22c806eff0 2023-10-10T18:44:55.912 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c77fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f22c806eff0 0x7f22c80c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.912 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c67fc700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f22c806dd00 0x7f22c806e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:33966/0 (socket says 172.21.15.191:33966) 2023-10-10T18:44:55.912 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c77fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f22c806eff0 0x7f22c80c33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44846/0 (socket says 172.21.15.191:44846) 2023-10-10T18:44:55.912 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c67fc700 1 -- 172.21.15.191:0/3648416234 learned_addr learned my addr 172.21.15.191:0/3648416234 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-10-10T18:44:55.912 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c77fe700 1 -- 172.21.15.191:0/3648416234 >> v1:172.21.15.191:6789/0 conn(0x7f22c806dd00 legacy=0x7f22c806e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.913 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c5ffb700 1 -- 172.21.15.191:0/3648416234 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745491221 0 0) 0x7f22c806cd00 con 0x7f22c806dd00 2023-10-10T18:44:55.913 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c77fe700 1 -- 172.21.15.191:0/3648416234 >> v1:172.21.15.175:6789/0 conn(0x7f22c8156590 legacy=0x7f22c8156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.913 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c77fe700 1 -- 172.21.15.191:0/3648416234 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c806c2a0 con 0x7f22c806eff0 2023-10-10T18:44:55.913 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c77fe700 1 --2- 172.21.15.191:0/3648416234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c80c33a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f22c000d0d0 tx=0x7f22c000b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75662938b6a1d625 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.913 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c5ffb700 1 -- 172.21.15.191:0/3648416234 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0010070 con 0x7f22c806eff0 2023-10-10T18:44:55.914 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c5ffb700 1 -- 172.21.15.191:0/3648416234 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c000de80 con 0x7f22c806eff0 2023-10-10T18:44:55.914 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.909+0000 7f22c5ffb700 1 -- 172.21.15.191:0/3648416234 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c000b320 con 0x7f22c806eff0 2023-10-10T18:44:55.914 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.910+0000 7f22cd432700 1 -- 172.21.15.191:0/3648416234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 msgr2=0x7f22c80c33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.914 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.910+0000 7f22cd432700 1 --2- 172.21.15.191:0/3648416234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c80c33a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f22c000d0d0 tx=0x7f22c000b990 comp rx=0 tx=0).stop 2023-10-10T18:44:55.914 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.910+0000 7f22cd432700 1 -- 172.21.15.191:0/3648416234 shutdown_connections 2023-10-10T18:44:55.914 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.910+0000 7f22cd432700 1 --2- 172.21.15.191:0/3648416234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c80c33a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.915 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.910+0000 7f22cd432700 1 -- 172.21.15.191:0/3648416234 >> 172.21.15.191:0/3648416234 conn(0x7f22c80c9f40 msgr2=0x7f22c80c5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:55.915 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.910+0000 7f22cd432700 1 -- 172.21.15.191:0/3648416234 shutdown_connections 2023-10-10T18:44:55.915 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.910+0000 7f22cd432700 1 -- 172.21.15.191:0/3648416234 wait complete. 2023-10-10T18:44:55.918 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.911+0000 7f22cd432700 1 Processor -- start 2023-10-10T18:44:55.918 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.911+0000 7f22cd432700 1 -- start start 2023-10-10T18:44:55.918 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22cd432700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22c806dd00 0x7f22c8162cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.919 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22cd432700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c81558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.919 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22cd432700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22c8156590 0x7f22c8147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:55.919 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22cd432700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c8068760 con 0x7f22c8156590 2023-10-10T18:44:55.919 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22cd432700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c8067890 con 0x7f22c806eff0 2023-10-10T18:44:55.920 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22cd432700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c8067b50 con 0x7f22c806dd00 2023-10-10T18:44:55.920 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c6ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22c806dd00 0x7f22c8162cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.920 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c67fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c81558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.920 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c6ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22c806dd00 0x7f22c8162cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:43076/0 (socket says 172.21.15.191:43076) 2023-10-10T18:44:55.920 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c67fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c81558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:44852/0 (socket says 172.21.15.191:44852) 2023-10-10T18:44:55.921 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c6ffd700 1 -- 172.21.15.191:0/890529762 learned_addr learned my addr 172.21.15.191:0/890529762 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-10T18:44:55.921 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c77fe700 1 --2- 172.21.15.191:0/890529762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22c8156590 0x7f22c8147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:55.921 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c67fc700 1 -- 172.21.15.191:0/890529762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22c806dd00 msgr2=0x7f22c8162cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.921 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c67fc700 1 --2- 172.21.15.191:0/890529762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22c806dd00 0x7f22c8162cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.921 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c67fc700 1 -- 172.21.15.191:0/890529762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22c8156590 msgr2=0x7f22c8147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:55.922 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c67fc700 1 --2- 172.21.15.191:0/890529762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22c8156590 0x7f22c8147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:55.922 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c67fc700 1 -- 172.21.15.191:0/890529762 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c000bc30 con 0x7f22c806eff0 2023-10-10T18:44:55.922 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22c67fc700 1 --2- 172.21.15.191:0/890529762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c81558f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f22c806e140 tx=0x7f22bc007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:55.922 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22aeffd700 1 -- 172.21.15.191:0/890529762 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc00cea0 con 0x7f22c806eff0 2023-10-10T18:44:55.922 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22aeffd700 1 -- 172.21.15.191:0/890529762 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22bc005b30 con 0x7f22c806eff0 2023-10-10T18:44:55.923 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.912+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c8067e30 con 0x7f22c806eff0 2023-10-10T18:44:55.923 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.913+0000 7f22aeffd700 1 -- 172.21.15.191:0/890529762 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc0070d0 con 0x7f22c806eff0 2023-10-10T18:44:55.923 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.913+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c80683f0 con 0x7f22c806eff0 2023-10-10T18:44:55.923 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.913+0000 7f22aeffd700 1 -- 172.21.15.191:0/890529762 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f22bc01c440 con 0x7f22c806eff0 2023-10-10T18:44:55.923 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.913+0000 7f22aeffd700 1 -- 172.21.15.191:0/890529762 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f22bc007780 con 0x7f22c806eff0 2023-10-10T18:44:55.924 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.914+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c80c0840 con 0x7f22c806eff0 2023-10-10T18:44:55.924 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.916+0000 7f22aeffd700 1 -- 172.21.15.191:0/890529762 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f22bc017070 con 0x7f22c806eff0 2023-10-10T18:44:55.997 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:55.997+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "194ceb5a-d817-4197-bb72-672205ca3331", "id": 7} v 0) v1 -- 0x7f22c80688d0 con 0x7f22c806eff0 2023-10-10T18:44:56.004 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.003+0000 7f22aeffd700 1 -- 172.21.15.191:0/890529762 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "194ceb5a-d817-4197-bb72-672205ca3331", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f22bc01c8d0 con 0x7f22c806eff0 2023-10-10T18:44:56.004 INFO:teuthology.orchestra.run.smithi191.stdout:7 2023-10-10T18:44:56.005 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 msgr2=0x7f22c81558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:56.005 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 --2- 172.21.15.191:0/890529762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c81558f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f22c806e140 tx=0x7f22bc007ec0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.005 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 shutdown_connections 2023-10-10T18:44:56.006 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 --2- 172.21.15.191:0/890529762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22c8156590 0x7f22c8147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.006 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 --2- 172.21.15.191:0/890529762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22c806dd00 0x7f22c8162cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.006 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 --2- 172.21.15.191:0/890529762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22c806eff0 0x7f22c81558f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.006 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 >> 172.21.15.191:0/890529762 conn(0x7f22c80c9f40 msgr2=0x7f22c80c80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:56.006 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 shutdown_connections 2023-10-10T18:44:56.007 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-10T18:44:56.004+0000 7f22cd432700 1 -- 172.21.15.191:0/890529762 wait complete. 2023-10-10T18:44:56.016 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-10T18:44:56.017 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-10T18:44:56.019 INFO:tasks.ceph.osd.0:Started 2023-10-10T18:44:56.020 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-10T18:44:56.020 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-10T18:44:56.022 INFO:tasks.ceph.osd.1:Started 2023-10-10T18:44:56.022 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-10T18:44:56.023 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-10T18:44:56.025 INFO:tasks.ceph.osd.2:Started 2023-10-10T18:44:56.025 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-10T18:44:56.025 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-10T18:44:56.028 INFO:tasks.ceph.osd.3:Started 2023-10-10T18:44:56.028 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-10T18:44:56.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-10T18:44:56.031 INFO:tasks.ceph.osd.4:Started 2023-10-10T18:44:56.031 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-10T18:44:56.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-10T18:44:56.034 INFO:tasks.ceph.osd.5:Started 2023-10-10T18:44:56.034 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-10T18:44:56.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-10T18:44:56.039 INFO:tasks.ceph.osd.6:Started 2023-10-10T18:44:56.039 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-10T18:44:56.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-10T18:44:56.042 INFO:tasks.ceph.osd.7:Started 2023-10-10T18:44:56.043 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:44:56.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.184+0000 7f87e759e700 1 Processor -- start 2023-10-10T18:44:56.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.184+0000 7f87e759e700 1 -- start start 2023-10-10T18:44:56.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e759e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:56.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e759e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e8067fa0 con 0x7f87e80cd040 2023-10-10T18:44:56.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e759e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e8156b70 con 0x7f87e808a100 2023-10-10T18:44:56.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e759e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80c5600 con 0x7f87e814fd90 2023-10-10T18:44:56.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e6d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f87e80cd040 0x7f87e80684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51224/0 (socket says 172.21.15.175:51224) 2023-10-10T18:44:56.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e6d9d700 1 -- 172.21.15.175:0/4036999693 learned_addr learned my addr 172.21.15.175:0/4036999693 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:44:56.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e659c700 1 --2- 172.21.15.175:0/4036999693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:56.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e559a700 1 -- 172.21.15.175:0/4036999693 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559901074 0 0) 0x7f87e8067fa0 con 0x7f87e80cd040 2023-10-10T18:44:56.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e559a700 1 -- 172.21.15.175:0/4036999693 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d0003570 con 0x7f87e80cd040 2023-10-10T18:44:56.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e559a700 1 -- 172.21.15.175:0/4036999693 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092061483 0 0) 0x7f87e8156b70 con 0x7f87e808a100 2023-10-10T18:44:56.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e559a700 1 -- 172.21.15.175:0/4036999693 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e8067fa0 con 0x7f87e808a100 2023-10-10T18:44:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e659c700 1 -- 172.21.15.175:0/4036999693 >> v1:172.21.15.191:6789/0 conn(0x7f87e808a100 legacy=0x7f87e80c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e659c700 1 -- 172.21.15.175:0/4036999693 >> v1:172.21.15.175:6789/0 conn(0x7f87e80cd040 legacy=0x7f87e80684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.185+0000 7f87e659c700 1 -- 172.21.15.175:0/4036999693 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e8150160 con 0x7f87e814fd90 2023-10-10T18:44:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e659c700 1 --2- 172.21.15.175:0/4036999693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8159c30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009910 tx=0x7f87d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c11e1adc3a75ae2c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e559a700 1 -- 172.21.15.175:0/4036999693 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8011630 con 0x7f87e814fd90 2023-10-10T18:44:56.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e559a700 1 -- 172.21.15.175:0/4036999693 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8011790 con 0x7f87e814fd90 2023-10-10T18:44:56.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e559a700 1 -- 172.21.15.175:0/4036999693 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8011b30 con 0x7f87e814fd90 2023-10-10T18:44:56.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e759e700 1 -- 172.21.15.175:0/4036999693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 msgr2=0x7f87e8159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:56.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e759e700 1 --2- 172.21.15.175:0/4036999693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8159c30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009910 tx=0x7f87d80091a0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e759e700 1 -- 172.21.15.175:0/4036999693 shutdown_connections 2023-10-10T18:44:56.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e759e700 1 --2- 172.21.15.175:0/4036999693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8159c30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e759e700 1 -- 172.21.15.175:0/4036999693 >> 172.21.15.175:0/4036999693 conn(0x7f87e8063d00 msgr2=0x7f87e80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:56.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e759e700 1 -- 172.21.15.175:0/4036999693 shutdown_connections 2023-10-10T18:44:56.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.186+0000 7f87e759e700 1 -- 172.21.15.175:0/4036999693 wait complete. 2023-10-10T18:44:56.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.187+0000 7f87e759e700 1 Processor -- start 2023-10-10T18:44:56.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e759e700 1 -- start start 2023-10-10T18:44:56.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e759e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87e808a100 0x7f87e8158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:56.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87e80cd040 0x7f87e81527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:56.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e759e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:56.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e759e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e8158eb0 con 0x7f87e80cd040 2023-10-10T18:44:56.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e759e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e81584e0 con 0x7f87e808a100 2023-10-10T18:44:56.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e759e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80c6ef0 con 0x7f87e814fd90 2023-10-10T18:44:56.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e5d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87e80cd040 0x7f87e81527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:56.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e5d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87e80cd040 0x7f87e81527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49608/0 (socket says 172.21.15.175:49608) 2023-10-10T18:44:56.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e5d9b700 1 -- 172.21.15.175:0/590737421 learned_addr learned my addr 172.21.15.175:0/590737421 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:44:56.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e659c700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87e808a100 0x7f87e8158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:56.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.188+0000 7f87e6d9d700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:56.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87e5d9b700 1 -- 172.21.15.175:0/590737421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 msgr2=0x7f87e8154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:56.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87e5d9b700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87e5d9b700 1 -- 172.21.15.175:0/590737421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87e808a100 msgr2=0x7f87e8158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:56.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87e5d9b700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87e808a100 0x7f87e8158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87e5d9b700 1 -- 172.21.15.175:0/590737421 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d8009420 con 0x7f87e80cd040 2023-10-10T18:44:56.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87e5d9b700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87e80cd040 0x7f87e81527b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c790 tx=0x7f87dc007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:56.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87d67fc700 1 -- 172.21.15.175:0/590737421 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc010510 con 0x7f87e80cd040 2023-10-10T18:44:56.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87d67fc700 1 -- 172.21.15.175:0/590737421 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc010670 con 0x7f87e80cd040 2023-10-10T18:44:56.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e814b810 con 0x7f87e80cd040 2023-10-10T18:44:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87d67fc700 1 -- 172.21.15.175:0/590737421 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc010a10 con 0x7f87e80cd040 2023-10-10T18:44:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e814bdd0 con 0x7f87e80cd040 2023-10-10T18:44:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87d67fc700 1 -- 172.21.15.175:0/590737421 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f87dc01c6c0 con 0x7f87e80cd040 2023-10-10T18:44:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.189+0000 7f87d67fc700 1 -- 172.21.15.175:0/590737421 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f87dc010e10 con 0x7f87e80cd040 2023-10-10T18:44:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.190+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e8158690 con 0x7f87e80cd040 2023-10-10T18:44:56.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.192+0000 7f87d67fc700 1 -- 172.21.15.175:0/590737421 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f87e8158690 con 0x7f87e80cd040 2023-10-10T18:44:56.254 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:56.254+0000 7fe0b1dc4180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T18:44:56.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.261+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87e8067570 con 0x7f87e80cd040 2023-10-10T18:44:56.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.262+0000 7f87d67fc700 1 -- 172.21.15.175:0/590737421 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f87dc017070 con 0x7f87e80cd040 2023-10-10T18:44:56.262 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:44:56.262 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":10,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:44:55.999291+0000","last_up_change":"0.000000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:44:56.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87e80cd040 msgr2=0x7f87e81527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:56.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87e80cd040 0x7f87e81527b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c790 tx=0x7f87dc007e10 comp rx=0 tx=0).stop 2023-10-10T18:44:56.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 shutdown_connections 2023-10-10T18:44:56.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87e814fd90 0x7f87e8154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87e808a100 0x7f87e8158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 --2- 172.21.15.175:0/590737421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87e80cd040 0x7f87e81527b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:56.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 >> 172.21.15.175:0/590737421 conn(0x7f87e8063d00 msgr2=0x7f87e814d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:56.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 shutdown_connections 2023-10-10T18:44:56.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:44:56.263+0000 7f87e759e700 1 -- 172.21.15.175:0/590737421 wait complete. 2023-10-10T18:44:56.279 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-10T18:44:56.280 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-10T18:44:56.294 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:56.293+0000 7f3fe3830180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T18:44:56.340 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:56.340+0000 7fe0b1dc4180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T18:44:56.369 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:56.368+0000 7f3fe3830180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T18:44:56.418 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:56.418+0000 7fe0b1dc4180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T18:44:56.447 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:56.446+0000 7f3fe3830180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T18:44:56.630 INFO:tasks.ceph.osd.1.smithi175.stderr:2023-10-10T18:44:56.630+0000 7f93a17cc540 -1 Falling back to public interface 2023-10-10T18:44:56.632 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-10-10T18:44:56.632+0000 7f330d81b540 -1 Falling back to public interface 2023-10-10T18:44:56.638 INFO:tasks.ceph.osd.0.smithi175.stderr:2023-10-10T18:44:56.638+0000 7f2b7f1ca540 -1 Falling back to public interface 2023-10-10T18:44:56.639 INFO:tasks.ceph.osd.2.smithi175.stderr:2023-10-10T18:44:56.639+0000 7f3f91b68540 -1 Falling back to public interface 2023-10-10T18:44:56.645 INFO:tasks.ceph.osd.3.smithi175.stderr:2023-10-10T18:44:56.645+0000 7fa70281e540 -1 Falling back to public interface 2023-10-10T18:44:56.645 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-10-10T18:44:56.645+0000 7fa8b5ff4540 -1 Falling back to public interface 2023-10-10T18:44:56.661 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-10-10T18:44:56.660+0000 7fe71d215540 -1 Falling back to public interface 2023-10-10T18:44:56.665 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-10-10T18:44:56.664+0000 7f59b5939540 -1 Falling back to public interface 2023-10-10T18:44:57.007 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:57.007+0000 7f3fe3830180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T18:44:57.016 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:57.015+0000 7fe0b1dc4180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T18:44:57.086 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-10T18:44:57.142 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:57.141+0000 7f3fe3830180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T18:44:57.157 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:57.156+0000 7fe0b1dc4180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T18:44:57.195 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.194+0000 7fe0ceda3700 1 Processor -- start 2023-10-10T18:44:57.195 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.194+0000 7fe0ceda3700 1 -- start start 2023-10-10T18:44:57.198 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ceda3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe0c81501d0 0x7fe0c80cd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:57.198 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ceda3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81575c0 0x7fe0c8157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:57.198 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ceda3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c80c2730 con 0x7fe0c8159dc0 2023-10-10T18:44:57.198 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ceda3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c80c7350 con 0x7fe0c81501d0 2023-10-10T18:44:57.199 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ceda3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c81505a0 con 0x7fe0c81575c0 2023-10-10T18:44:57.199 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ccb3f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe0c81501d0 0x7fe0c80cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:57.199 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0cd340700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81575c0 0x7fe0c8157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:57.199 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0c7fff700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe0c8159dc0 0x7fe0c80cc650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:36404/0 (socket says 172.21.15.175:36404) 2023-10-10T18:44:57.200 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0cd340700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81575c0 0x7fe0c8157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:34048/0 (socket says 172.21.15.175:34048) 2023-10-10T18:44:57.200 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ccb3f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe0c81501d0 0x7fe0c80cd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:59974/0 (socket says 172.21.15.175:59974) 2023-10-10T18:44:57.200 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0c7fff700 1 -- 172.21.15.175:0/1331614466 learned_addr learned my addr 172.21.15.175:0/1331614466 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:44:57.200 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ccb3f700 1 -- 172.21.15.175:0/1331614466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81575c0 msgr2=0x7fe0c8157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:44:57.201 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ccb3f700 1 --2- 172.21.15.175:0/1331614466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81575c0 0x7fe0c8157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.201 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ccb3f700 1 -- 172.21.15.175:0/1331614466 >> v1:172.21.15.191:6789/0 conn(0x7fe0c8159dc0 legacy=0x7fe0c80cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:57.201 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ccb3f700 1 -- 172.21.15.175:0/1331614466 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c8157f60 con 0x7fe0c81501d0 2023-10-10T18:44:57.201 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0c77fe700 1 -- 172.21.15.175:0/1331614466 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761636020 0 0) 0x7fe0c80c2730 con 0x7fe0c8159dc0 2023-10-10T18:44:57.201 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.195+0000 7fe0ccb3f700 1 --2- 172.21.15.175:0/1331614466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81501d0 0x7fe0c80cd270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0098e0 tx=0x7fe0bc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc3027658a172210 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:57.202 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0c77fe700 1 -- 172.21.15.175:0/1331614466 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc00b6d0 con 0x7fe0c81501d0 2023-10-10T18:44:57.202 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0c77fe700 1 -- 172.21.15.175:0/1331614466 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc00b830 con 0x7fe0c81501d0 2023-10-10T18:44:57.202 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0c77fe700 1 -- 172.21.15.175:0/1331614466 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc00bbd0 con 0x7fe0c81501d0 2023-10-10T18:44:57.203 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0ceda3700 1 -- 172.21.15.175:0/1331614466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81501d0 msgr2=0x7fe0c80cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:57.203 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0ceda3700 1 --2- 172.21.15.175:0/1331614466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81501d0 0x7fe0c80cd270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0098e0 tx=0x7fe0bc0091a0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.203 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0ceda3700 1 -- 172.21.15.175:0/1331614466 shutdown_connections 2023-10-10T18:44:57.203 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0ceda3700 1 --2- 172.21.15.175:0/1331614466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81575c0 0x7fe0c8157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.203 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0ceda3700 1 --2- 172.21.15.175:0/1331614466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81501d0 0x7fe0c80cd270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.204 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.196+0000 7fe0ceda3700 1 -- 172.21.15.175:0/1331614466 >> 172.21.15.175:0/1331614466 conn(0x7fe0c80689f0 msgr2=0x7fe0c8068df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:57.204 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.197+0000 7fe0ceda3700 1 -- 172.21.15.175:0/1331614466 shutdown_connections 2023-10-10T18:44:57.204 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.197+0000 7fe0ceda3700 1 -- 172.21.15.175:0/1331614466 wait complete. 2023-10-10T18:44:57.208 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.198+0000 7fe0ceda3700 1 Processor -- start 2023-10-10T18:44:57.208 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.198+0000 7fe0ceda3700 1 -- start start 2023-10-10T18:44:57.208 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ceda3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81501d0 0x7fe0c81490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:57.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ceda3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81575c0 0x7fe0c81495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:57.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ceda3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0c8159dc0 0x7fe0c8169cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:44:57.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ceda3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c816a880 con 0x7fe0c8159dc0 2023-10-10T18:44:57.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ceda3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c815d550 con 0x7fe0c81575c0 2023-10-10T18:44:57.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ceda3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c8153260 con 0x7fe0c81501d0 2023-10-10T18:44:57.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0cd340700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0c8159dc0 0x7fe0c8169cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:57.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0cd340700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0c8159dc0 0x7fe0c8169cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49612/0 (socket says 172.21.15.175:49612) 2023-10-10T18:44:57.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0c7fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81575c0 0x7fe0c81495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:57.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ccb3f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81501d0 0x7fe0c81490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:44:57.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0cd340700 1 -- 172.21.15.175:0/2336172703 learned_addr learned my addr 172.21.15.175:0/2336172703 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:44:57.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ccb3f700 1 -- 172.21.15.175:0/2336172703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81575c0 msgr2=0x7fe0c81495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:57.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ccb3f700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81575c0 0x7fe0c81495e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ccb3f700 1 -- 172.21.15.175:0/2336172703 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0c8159dc0 msgr2=0x7fe0c8169cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:57.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ccb3f700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0c8159dc0 0x7fe0c8169cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ccb3f700 1 -- 172.21.15.175:0/2336172703 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc009420 con 0x7fe0c81501d0 2023-10-10T18:44:57.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0c7fff700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81575c0 0x7fe0c81495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:57.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0cd340700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0c8159dc0 0x7fe0c8169cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:44:57.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ccb3f700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81501d0 0x7fe0c81490a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc016000 tx=0x7fe0bc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:44:57.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0c4ff9700 1 -- 172.21.15.175:0/2336172703 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc000d10 con 0x7fe0c81501d0 2023-10-10T18:44:57.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0c4ff9700 1 -- 172.21.15.175:0/2336172703 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc00bbb0 con 0x7fe0c81501d0 2023-10-10T18:44:57.213 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c8153540 con 0x7fe0c81501d0 2023-10-10T18:44:57.213 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.200+0000 7fe0c4ff9700 1 -- 172.21.15.175:0/2336172703 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc011650 con 0x7fe0c81501d0 2023-10-10T18:44:57.213 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.201+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c8152bb0 con 0x7fe0c81501d0 2023-10-10T18:44:57.213 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.201+0000 7fe0c4ff9700 1 -- 172.21.15.175:0/2336172703 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe0bc011a00 con 0x7fe0c81501d0 2023-10-10T18:44:57.213 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.201+0000 7fe0c4ff9700 1 -- 172.21.15.175:0/2336172703 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fe0bc025e00 con 0x7fe0c81501d0 2023-10-10T18:44:57.214 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.201+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c805f270 con 0x7fe0c81501d0 2023-10-10T18:44:57.214 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.204+0000 7fe0c4ff9700 1 -- 172.21.15.175:0/2336172703 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe0bc02a060 con 0x7fe0c81501d0 2023-10-10T18:44:57.237 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:57.236+0000 7f3fe3830180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T18:44:57.255 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:57.254+0000 7fe0b1dc4180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T18:44:57.273 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.272+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0c80c1580 con 0x7fe0c81501d0 2023-10-10T18:44:57.274 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.273+0000 7fe0c4ff9700 1 -- 172.21.15.175:0/2336172703 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fe0bc034020 con 0x7fe0c81501d0 2023-10-10T18:44:57.274 INFO:teuthology.misc.health.smithi175.stdout: 2023-10-10T18:44:57.274 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":10,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:44:55.999291+0000","last_up_change":"0.000000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:44:57.275 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81501d0 msgr2=0x7fe0c81490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:44:57.276 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81501d0 0x7fe0c81490a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc016000 tx=0x7fe0bc017000 comp rx=0 tx=0).stop 2023-10-10T18:44:57.276 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 shutdown_connections 2023-10-10T18:44:57.276 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0c8159dc0 0x7fe0c8169cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.276 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe0c81501d0 0x7fe0c81490a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.276 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 --2- 172.21.15.175:0/2336172703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0c81575c0 0x7fe0c81495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:44:57.277 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 >> 172.21.15.175:0/2336172703 conn(0x7fe0c80689f0 msgr2=0x7fe0c80c79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:44:57.277 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 shutdown_connections 2023-10-10T18:44:57.277 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:44:57.274+0000 7fe0ceda3700 1 -- 172.21.15.175:0/2336172703 wait complete. 2023-10-10T18:44:57.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-10T18:44:57.485 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:57.484+0000 7f3fe3830180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T18:44:57.515 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:57.514+0000 7fe0b1dc4180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T18:44:57.561 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:57.560+0000 7f3fe3830180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T18:44:57.592 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:57.591+0000 7fe0b1dc4180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T18:44:58.382 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:58.382+0000 7f3fe3830180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T18:44:58.413 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:58.412+0000 7fe0b1dc4180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T18:44:58.454 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:58.454+0000 7f3fe3830180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T18:44:58.487 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:58.486+0000 7fe0b1dc4180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T18:44:58.526 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:58.525+0000 7f3fe3830180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T18:44:58.561 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:58.560+0000 7fe0b1dc4180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T18:44:58.657 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:58.656+0000 7f3fe3830180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T18:44:58.694 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:58.694+0000 7fe0b1dc4180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T18:44:58.739 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:58.740+0000 7f3fe3830180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T18:44:58.778 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:58.777+0000 7fe0b1dc4180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T18:44:58.887 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:58.887+0000 7f3fe3830180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T18:44:58.930 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:58.929+0000 7fe0b1dc4180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T18:44:58.960 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:58.960+0000 7f3fe3830180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T18:44:59.007 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:59.006+0000 7fe0b1dc4180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T18:44:59.033 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:59.033+0000 7f3fe3830180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T18:44:59.082 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:59.081+0000 7fe0b1dc4180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T18:44:59.106 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:44:59.106+0000 7f3fe3830180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T18:44:59.166 INFO:tasks.ceph.mgr.y.smithi175.stderr:2023-10-10T18:44:59.165+0000 7fe0b1dc4180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T18:45:00.132 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:45:00.132+0000 7f3f9cc32700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-10T18:45:00.132 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-10T18:45:00.133+0000 7f3f9cc32700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-10T18:45:00.141 INFO:tasks.ceph.osd.0.smithi175.stderr:2023-10-10T18:45:00.141+0000 7f2b7f1ca540 -1 osd.0 0 log_to_monitors true 2023-10-10T18:45:00.158 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-10-10T18:45:00.159+0000 7f59b5939540 -1 osd.7 0 log_to_monitors true 2023-10-10T18:45:00.168 INFO:tasks.ceph.osd.1.smithi175.stderr:2023-10-10T18:45:00.167+0000 7f93a17cc540 -1 osd.1 0 log_to_monitors true 2023-10-10T18:45:00.184 INFO:tasks.ceph.osd.3.smithi175.stderr:2023-10-10T18:45:00.183+0000 7fa70281e540 -1 osd.3 0 log_to_monitors true 2023-10-10T18:45:00.185 INFO:tasks.ceph.osd.2.smithi175.stderr:2023-10-10T18:45:00.183+0000 7f3f91b68540 -1 osd.2 0 log_to_monitors true 2023-10-10T18:45:00.191 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-10-10T18:45:00.192+0000 7fa8b5ff4540 -1 osd.6 0 log_to_monitors true 2023-10-10T18:45:00.204 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-10-10T18:45:00.204+0000 7f330d81b540 -1 osd.4 0 log_to_monitors true 2023-10-10T18:45:00.211 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-10-10T18:45:00.212+0000 7fe71d215540 -1 osd.5 0 log_to_monitors true 2023-10-10T18:45:03.037 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-10-10T18:45:03.038+0000 7f59ac55b700 -1 osd.7 0 waiting for initial osdmap 2023-10-10T18:45:03.041 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-10-10T18:45:03.042+0000 7f59a5352700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T18:45:03.047 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-10-10T18:45:03.047+0000 7fa8acc16700 -1 osd.6 0 waiting for initial osdmap 2023-10-10T18:45:03.049 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-10-10T18:45:03.050+0000 7fa8a5a0d700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T18:45:03.052 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-10-10T18:45:03.052+0000 7f3305c40700 -1 osd.4 0 waiting for initial osdmap 2023-10-10T18:45:03.054 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-10-10T18:45:03.055+0000 7f32fd234700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T18:45:03.060 INFO:tasks.ceph.osd.3.smithi175.stderr:2023-10-10T18:45:03.059+0000 7fa6fac43700 -1 osd.3 0 waiting for initial osdmap 2023-10-10T18:45:03.060 INFO:tasks.ceph.osd.1.smithi175.stderr:2023-10-10T18:45:03.060+0000 7f93983ee700 -1 osd.1 0 waiting for initial osdmap 2023-10-10T18:45:03.064 INFO:tasks.ceph.osd.3.smithi175.stderr:2023-10-10T18:45:03.063+0000 7fa6f2237700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T18:45:03.065 INFO:tasks.ceph.osd.1.smithi175.stderr:2023-10-10T18:45:03.065+0000 7f93911e5700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T18:45:03.070 INFO:tasks.ceph.osd.0.smithi175.stderr:2023-10-10T18:45:03.068+0000 7f2b75dec700 -1 osd.0 0 waiting for initial osdmap 2023-10-10T18:45:03.072 INFO:tasks.ceph.osd.0.smithi175.stderr:2023-10-10T18:45:03.070+0000 7f2b6ebe3700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T18:45:03.087 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-10-10T18:45:03.087+0000 7fe713e37700 -1 osd.5 0 waiting for initial osdmap 2023-10-10T18:45:03.087 INFO:tasks.ceph.osd.2.smithi175.stderr:2023-10-10T18:45:03.086+0000 7f3f8878a700 -1 osd.2 0 waiting for initial osdmap 2023-10-10T18:45:03.090 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-10-10T18:45:03.090+0000 7fe70cc2e700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T18:45:03.091 INFO:tasks.ceph.osd.2.smithi175.stderr:2023-10-10T18:45:03.090+0000 7f3f81581700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T18:45:04.095 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:04.205 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.204+0000 7f1dd50d2700 1 Processor -- start 2023-10-10T18:45:04.206 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.204+0000 7f1dd50d2700 1 -- start start 2023-10-10T18:45:04.208 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.205+0000 7f1dd50d2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1dd014c190 0x7f1dd014c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.208 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.205+0000 7f1dd50d2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd0155890 con 0x7f1dd0152cf0 2023-10-10T18:45:04.208 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.205+0000 7f1dd50d2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd0156130 con 0x7f1dd00695d0 2023-10-10T18:45:04.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.205+0000 7f1dd50d2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd00bd810 con 0x7f1dd014c190 2023-10-10T18:45:04.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.205+0000 7f1dc659c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1dd0152cf0 0x7f1dd00699a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51230/0 (socket says 172.21.15.175:51230) 2023-10-10T18:45:04.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.205+0000 7f1dc659c700 1 -- 172.21.15.175:0/2602204987 learned_addr learned my addr 172.21.15.175:0/2602204987 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:04.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.205+0000 7f1dcf59e700 1 --2- 172.21.15.175:0/2602204987 >> v2:172.21.15.191:3300/0 conn(0x7f1dd014c190 0x7f1dd014c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.209 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dce59c700 1 -- 172.21.15.175:0/2602204987 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394637598 0 0) 0x7f1dd0155890 con 0x7f1dd0152cf0 2023-10-10T18:45:04.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dce59c700 1 -- 172.21.15.175:0/2602204987 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dbc003570 con 0x7f1dd0152cf0 2023-10-10T18:45:04.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dcf59e700 1 -- 172.21.15.175:0/2602204987 >> v1:172.21.15.191:6789/0 conn(0x7f1dd00695d0 legacy=0x7f1dd00bd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dcf59e700 1 -- 172.21.15.175:0/2602204987 >> v1:172.21.15.175:6789/0 conn(0x7f1dd0152cf0 legacy=0x7f1dd00699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dcf59e700 1 -- 172.21.15.175:0/2602204987 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd0068470 con 0x7f1dd014c190 2023-10-10T18:45:04.210 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dcf59e700 1 --2- 172.21.15.175:0/2602204987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd014c190 0x7f1dd014c5d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009e60 tx=0x7f1dc8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e44fdd90dd7ab998 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dce59c700 1 -- 172.21.15.175:0/2602204987 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc800e070 con 0x7f1dd014c190 2023-10-10T18:45:04.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dce59c700 1 -- 172.21.15.175:0/2602204987 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc800bbf0 con 0x7f1dd014c190 2023-10-10T18:45:04.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.206+0000 7f1dce59c700 1 -- 172.21.15.175:0/2602204987 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8012430 con 0x7f1dd014c190 2023-10-10T18:45:04.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.207+0000 7f1dd50d2700 1 -- 172.21.15.175:0/2602204987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd014c190 msgr2=0x7f1dd014c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.211 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.207+0000 7f1dd50d2700 1 --2- 172.21.15.175:0/2602204987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd014c190 0x7f1dd014c5d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009e60 tx=0x7f1dc8009780 comp rx=0 tx=0).stop 2023-10-10T18:45:04.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.207+0000 7f1dd50d2700 1 -- 172.21.15.175:0/2602204987 shutdown_connections 2023-10-10T18:45:04.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.207+0000 7f1dd50d2700 1 --2- 172.21.15.175:0/2602204987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd014c190 0x7f1dd014c5d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.207+0000 7f1dd50d2700 1 -- 172.21.15.175:0/2602204987 >> 172.21.15.175:0/2602204987 conn(0x7f1dd00c82b0 msgr2=0x7f1dd00c86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:04.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.207+0000 7f1dd50d2700 1 -- 172.21.15.175:0/2602204987 shutdown_connections 2023-10-10T18:45:04.212 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.207+0000 7f1dd50d2700 1 -- 172.21.15.175:0/2602204987 wait complete. 2023-10-10T18:45:04.215 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.208+0000 7f1dd50d2700 1 Processor -- start 2023-10-10T18:45:04.215 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.208+0000 7f1dd50d2700 1 -- start start 2023-10-10T18:45:04.216 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dd50d2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd00695d0 0x7f1dd014b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.216 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dd50d2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dd014c190 0x7f1dd00ab720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.216 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dd50d2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1dd0152cf0 0x7f1dd00abc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.216 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dd50d2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0165780 con 0x7f1dd0152cf0 2023-10-10T18:45:04.217 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dd50d2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd01659f0 con 0x7f1dd00695d0 2023-10-10T18:45:04.217 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dd50d2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0165cd0 con 0x7f1dd014c190 2023-10-10T18:45:04.217 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1dd0152cf0 0x7f1dd00abc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.217 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1dd0152cf0 0x7f1dd00abc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49650/0 (socket says 172.21.15.175:49650) 2023-10-10T18:45:04.217 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 -- 172.21.15.175:0/757720897 learned_addr learned my addr 172.21.15.175:0/757720897 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:04.218 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dced9d700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd00695d0 0x7f1dd014b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.218 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dc659c700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dd014c190 0x7f1dd00ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.221 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 -- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dd014c190 msgr2=0x7f1dd00ab720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.221 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dd014c190 0x7f1dd00ab720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.222 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 -- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd00695d0 msgr2=0x7f1dd014b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.222 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd00695d0 0x7f1dd014b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.222 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 -- 172.21.15.175:0/757720897 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc8009a20 con 0x7f1dd0152cf0 2023-10-10T18:45:04.222 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dced9d700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd00695d0 0x7f1dd014b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:04.222 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dc659c700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dd014c190 0x7f1dd00ab720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:04.223 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dcf59e700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1dd0152cf0 0x7f1dd00abc60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1dd006cb90 tx=0x7f1dc8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.223 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dc77fe700 1 -- 172.21.15.175:0/757720897 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8012e00 con 0x7f1dd0152cf0 2023-10-10T18:45:04.223 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dc77fe700 1 -- 172.21.15.175:0/757720897 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc801e630 con 0x7f1dd0152cf0 2023-10-10T18:45:04.223 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd0165fb0 con 0x7f1dd0152cf0 2023-10-10T18:45:04.224 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dc77fe700 1 -- 172.21.15.175:0/757720897 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc801e9f0 con 0x7f1dd0152cf0 2023-10-10T18:45:04.224 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.209+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd00c5a30 con 0x7f1dd0152cf0 2023-10-10T18:45:04.224 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.210+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd005f270 con 0x7f1dd0152cf0 2023-10-10T18:45:04.224 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.211+0000 7f1dc77fe700 1 -- 172.21.15.175:0/757720897 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f1dc8024070 con 0x7f1dd0152cf0 2023-10-10T18:45:04.224 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.211+0000 7f1dc77fe700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1db0066ea0 0x7f1db0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.224 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.211+0000 7f1dc77fe700 1 -- 172.21.15.175:0/757720897 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f1dc80868c0 con 0x7f1dd0152cf0 2023-10-10T18:45:04.225 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.211+0000 7f1dced9d700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1db0066ea0 0x7f1db0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.225 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.212+0000 7f1dced9d700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1db0066ea0 0x7f1db0069350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1dc00017a0 tx=0x7f1dc0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.225 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.215+0000 7f1dc77fe700 1 -- 172.21.15.175:0/757720897 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1dc80540e0 con 0x7f1dd0152cf0 2023-10-10T18:45:04.400 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.399+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dd004bb80 con 0x7f1dd0152cf0 2023-10-10T18:45:04.401 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.400+0000 7f1dc77fe700 1 -- 172.21.15.175:0/757720897 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10444 (secure 0 0 0) 0x7f1dc8057700 con 0x7f1dd0152cf0 2023-10-10T18:45:04.401 INFO:teuthology.misc.health.smithi175.stdout: 2023-10-10T18:45:04.402 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":13,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:04.052774+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:04.404 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1db0066ea0 msgr2=0x7f1db0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.404 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1db0066ea0 0x7f1db0069350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1dc00017a0 tx=0x7f1dc0009040 comp rx=0 tx=0).stop 2023-10-10T18:45:04.404 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1dd0152cf0 msgr2=0x7f1dd00abc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.404 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1dd0152cf0 0x7f1dd00abc60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1dd006cb90 tx=0x7f1dc8019000 comp rx=0 tx=0).stop 2023-10-10T18:45:04.405 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 shutdown_connections 2023-10-10T18:45:04.405 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1db0066ea0 0x7f1db0069350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.405 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1dd0152cf0 0x7f1dd00abc60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.405 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1dd00695d0 0x7f1dd014b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.405 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 --2- 172.21.15.175:0/757720897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dd014c190 0x7f1dd00ab720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.406 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 >> 172.21.15.175:0/757720897 conn(0x7f1dd00c82b0 msgr2=0x7f1dd00bd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:04.406 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 shutdown_connections 2023-10-10T18:45:04.406 INFO:teuthology.misc.health.smithi175.stderr:2023-10-10T18:45:04.402+0000 7f1dd50d2700 1 -- 172.21.15.175:0/757720897 wait complete. 2023-10-10T18:45:04.414 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-10T18:45:04.415 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-10T18:45:04.415 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-10T18:45:04.415 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-10T18:45:04.418 INFO:tasks.ceph.mds.a:Started 2023-10-10T18:45:04.418 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-10T18:45:04.418 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-10T18:45:04.420 INFO:tasks.ceph.mds.c:Started 2023-10-10T18:45:04.421 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-10T18:45:04.421 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-10T18:45:04.423 INFO:tasks.ceph.mds.e:Started 2023-10-10T18:45:04.423 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-10T18:45:04.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-10T18:45:04.426 INFO:tasks.ceph.mds.b:Started 2023-10-10T18:45:04.427 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-10T18:45:04.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-10T18:45:04.429 INFO:tasks.ceph.mds.d:Started 2023-10-10T18:45:04.429 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-10T18:45:04.430 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-10T18:45:04.432 INFO:tasks.ceph.mds.f:Started 2023-10-10T18:45:04.432 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-10T18:45:04.433 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:04.493 INFO:tasks.ceph.mds.a.smithi175.stdout:starting mds.a at 2023-10-10T18:45:04.494 INFO:tasks.ceph.mds.c.smithi175.stdout:starting mds.c at 2023-10-10T18:45:04.495 INFO:tasks.ceph.mds.e.smithi175.stdout:starting mds.e at 2023-10-10T18:45:04.512 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-10T18:45:04.512 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-10T18:45:04.513 INFO:tasks.ceph.mds.b.smithi191.stderr:2023-10-10T18:45:04.513+0000 7f4100bd4700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-10-10T18:45:04.513 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T18:45:04.514+0000 7fc56e733700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6836/0: (98) Address already in use 2023-10-10T18:45:04.515 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-10T18:45:04.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.558+0000 7fe98a68f700 1 Processor -- start 2023-10-10T18:45:04.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.558+0000 7fe98a68f700 1 -- start start 2023-10-10T18:45:04.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe98a68f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe9840c11d0 0x7fe984161920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe98a68f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9840c5e30 con 0x7fe9840b01f0 2023-10-10T18:45:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe98a68f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9840c5260 con 0x7fe9840afe20 2023-10-10T18:45:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe98a68f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9840c58b0 con 0x7fe9840c11d0 2023-10-10T18:45:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe988c2c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe9840c11d0 0x7fe984161920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe9837fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe9840b01f0 0x7fe984068880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51252/0 (socket says 172.21.15.175:51252) 2023-10-10T18:45:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe988c2c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe9840c11d0 0x7fe984161920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49704/0 (socket says 172.21.15.175:49704) 2023-10-10T18:45:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe9837fe700 1 -- 172.21.15.175:0/4234830913 learned_addr learned my addr 172.21.15.175:0/4234830913 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe988c2c700 1 -- 172.21.15.175:0/4234830913 >> v1:172.21.15.191:6789/0 conn(0x7fe9840afe20 legacy=0x7fe984159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe988c2c700 1 -- 172.21.15.175:0/4234830913 >> v1:172.21.15.175:6789/0 conn(0x7fe9840b01f0 legacy=0x7fe984068880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe988c2c700 1 -- 172.21.15.175:0/4234830913 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9840c68e0 con 0x7fe9840c11d0 2023-10-10T18:45:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe982ffd700 1 -- 172.21.15.175:0/4234830913 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363998536 0 0) 0x7fe9840c5e30 con 0x7fe9840b01f0 2023-10-10T18:45:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe988c2c700 1 --2- 172.21.15.175:0/4234830913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840c11d0 0x7fe984161920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009e60 tx=0x7fe97c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211213d7cf61a803 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe982ffd700 1 -- 172.21.15.175:0/4234830913 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c00e070 con 0x7fe9840c11d0 2023-10-10T18:45:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe982ffd700 1 -- 172.21.15.175:0/4234830913 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c00bc80 con 0x7fe9840c11d0 2023-10-10T18:45:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.559+0000 7fe982ffd700 1 -- 172.21.15.175:0/4234830913 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c012430 con 0x7fe9840c11d0 2023-10-10T18:45:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.560+0000 7fe98a68f700 1 -- 172.21.15.175:0/4234830913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840c11d0 msgr2=0x7fe984161920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.560+0000 7fe98a68f700 1 --2- 172.21.15.175:0/4234830913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840c11d0 0x7fe984161920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009e60 tx=0x7fe97c009780 comp rx=0 tx=0).stop 2023-10-10T18:45:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.560+0000 7fe98a68f700 1 -- 172.21.15.175:0/4234830913 shutdown_connections 2023-10-10T18:45:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.560+0000 7fe98a68f700 1 --2- 172.21.15.175:0/4234830913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840c11d0 0x7fe984161920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.560+0000 7fe98a68f700 1 -- 172.21.15.175:0/4234830913 >> 172.21.15.175:0/4234830913 conn(0x7fe984063cb0 msgr2=0x7fe98406d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.560+0000 7fe98a68f700 1 -- 172.21.15.175:0/4234830913 shutdown_connections 2023-10-10T18:45:04.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.560+0000 7fe98a68f700 1 -- 172.21.15.175:0/4234830913 wait complete. 2023-10-10T18:45:04.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.561+0000 7fe98a68f700 1 Processor -- start 2023-10-10T18:45:04.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.561+0000 7fe98a68f700 1 -- start start 2023-10-10T18:45:04.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe98a68f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9840afe20 0x7fe984151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe98a68f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840b01f0 0x7fe9841519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe98a68f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9840c11d0 0x7fe9841696e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe98a68f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe984166940 con 0x7fe9840b01f0 2023-10-10T18:45:04.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe98a68f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe984150fd0 con 0x7fe9840c11d0 2023-10-10T18:45:04.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe98a68f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe984166e10 con 0x7fe9840afe20 2023-10-10T18:45:04.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840b01f0 0x7fe9841519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840b01f0 0x7fe9841519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49716/0 (socket says 172.21.15.175:49716) 2023-10-10T18:45:04.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 -- 172.21.15.175:0/316094959 learned_addr learned my addr 172.21.15.175:0/316094959 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:04.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe983fff700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9840afe20 0x7fe984151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe988c2c700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9840c11d0 0x7fe9841696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 -- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9840afe20 msgr2=0x7fe984151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9840afe20 0x7fe984151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 -- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9840c11d0 msgr2=0x7fe9841696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9840c11d0 0x7fe9841696e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 -- 172.21.15.175:0/316094959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c009a20 con 0x7fe9840b01f0 2023-10-10T18:45:04.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe983fff700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9840afe20 0x7fe984151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:04.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe9837fe700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840b01f0 0x7fe9841519a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe97800c7e0 tx=0x7fe978007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe96bfff700 1 -- 172.21.15.175:0/316094959 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe978005b30 con 0x7fe9840b01f0 2023-10-10T18:45:04.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe96bfff700 1 -- 172.21.15.175:0/316094959 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe978005c90 con 0x7fe9840b01f0 2023-10-10T18:45:04.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9841506e0 con 0x7fe9840b01f0 2023-10-10T18:45:04.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe96bfff700 1 -- 172.21.15.175:0/316094959 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe978007300 con 0x7fe9840b01f0 2023-10-10T18:45:04.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.562+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe984150980 con 0x7fe9840b01f0 2023-10-10T18:45:04.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.563+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe984151160 con 0x7fe9840b01f0 2023-10-10T18:45:04.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.564+0000 7fe96bfff700 1 -- 172.21.15.175:0/316094959 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7fe97801b400 con 0x7fe9840b01f0 2023-10-10T18:45:04.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.564+0000 7fe96bfff700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe970066ed0 0x7fe970069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.564+0000 7fe96bfff700 1 -- 172.21.15.175:0/316094959 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7fe978083a90 con 0x7fe9840b01f0 2023-10-10T18:45:04.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.564+0000 7fe983fff700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe970066ed0 0x7fe970069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.565+0000 7fe983fff700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe970066ed0 0x7fe970069380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe974002880 tx=0x7fe974009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.568+0000 7fe96bfff700 1 -- 172.21.15.175:0/316094959 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe978051330 con 0x7fe9840b01f0 2023-10-10T18:45:04.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.750+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe984067570 con 0x7fe9840b01f0 2023-10-10T18:45:04.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.751+0000 7fe96bfff700 1 -- 172.21.15.175:0/316094959 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10444 (secure 0 0 0) 0x7fe978054950 con 0x7fe9840b01f0 2023-10-10T18:45:04.752 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:04.753 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":13,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:04.052774+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:04.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.753+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe970066ed0 msgr2=0x7fe970069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.753+0000 7fe98a68f700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe970066ed0 0x7fe970069380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe974002880 tx=0x7fe974009040 comp rx=0 tx=0).stop 2023-10-10T18:45:04.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840b01f0 msgr2=0x7fe9841519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840b01f0 0x7fe9841519a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe97800c7e0 tx=0x7fe978007df0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 shutdown_connections 2023-10-10T18:45:04.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe970066ed0 0x7fe970069380 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9840c11d0 0x7fe9841696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9840afe20 0x7fe984151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 --2- 172.21.15.175:0/316094959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9840b01f0 0x7fe9841519a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 >> 172.21.15.175:0/316094959 conn(0x7fe984063cb0 msgr2=0x7fe984152620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:04.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 shutdown_connections 2023-10-10T18:45:04.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.754+0000 7fe98a68f700 1 -- 172.21.15.175:0/316094959 wait complete. 2023-10-10T18:45:04.766 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2023-10-10T18:45:04.767 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:04.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.880+0000 7f5f0531f700 1 Processor -- start 2023-10-10T18:45:04.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.880+0000 7f5f0531f700 1 -- start start 2023-10-10T18:45:04.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5f0531f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5f00068cf0 0x7f5f00067b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5f0531f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5f00159dd0 0x7f5f000c2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5f0531f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f00068b30 con 0x7f5f0014fd20 2023-10-10T18:45:04.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5f0531f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f000c1c70 con 0x7f5f00159dd0 2023-10-10T18:45:04.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5f0531f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f000c2640 con 0x7f5f00068cf0 2023-10-10T18:45:04.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5eff7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5f00159dd0 0x7f5f000c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5eff7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5f00159dd0 0x7f5f000c2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50620/0 (socket says 172.21.15.175:50620) 2023-10-10T18:45:04.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5f04b1e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5f0014fd20 0x7f5f000690c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56362/0 (socket says 172.21.15.175:56362) 2023-10-10T18:45:04.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5eff7fe700 1 -- 172.21.15.175:0/360801692 learned_addr learned my addr 172.21.15.175:0/360801692 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:04.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5efffff700 1 --2- 172.21.15.175:0/360801692 >> v2:172.21.15.191:3300/0 conn(0x7f5f00068cf0 0x7f5f00067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5eff7fe700 1 -- 172.21.15.175:0/360801692 >> v2:172.21.15.191:3300/0 conn(0x7f5f00068cf0 msgr2=0x7f5f00067b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:04.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5eff7fe700 1 --2- 172.21.15.175:0/360801692 >> v2:172.21.15.191:3300/0 conn(0x7f5f00068cf0 0x7f5f00067b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5eff7fe700 1 -- 172.21.15.175:0/360801692 >> v1:172.21.15.175:6789/0 conn(0x7f5f0014fd20 legacy=0x7f5f000690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5eff7fe700 1 -- 172.21.15.175:0/360801692 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000cd2b0 con 0x7f5f00159dd0 2023-10-10T18:45:04.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5efffff700 1 --2- 172.21.15.175:0/360801692 >> v2:172.21.15.191:3300/0 conn(0x7f5f00068cf0 0x7f5f00067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:04.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.881+0000 7f5eff7fe700 1 --2- 172.21.15.175:0/360801692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00159dd0 0x7f5f000c2100 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40099b0 tx=0x7f5ef4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20933b0f41c8640c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5efeffd700 1 -- 172.21.15.175:0/360801692 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef400b8b0 con 0x7f5f00159dd0 2023-10-10T18:45:04.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5efeffd700 1 -- 172.21.15.175:0/360801692 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef400ba10 con 0x7f5f00159dd0 2023-10-10T18:45:04.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5efeffd700 1 -- 172.21.15.175:0/360801692 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef400bdb0 con 0x7f5f00159dd0 2023-10-10T18:45:04.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5f0531f700 1 -- 172.21.15.175:0/360801692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00159dd0 msgr2=0x7f5f000c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5f0531f700 1 --2- 172.21.15.175:0/360801692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00159dd0 0x7f5f000c2100 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40099b0 tx=0x7f5ef4009250 comp rx=0 tx=0).stop 2023-10-10T18:45:04.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5f0531f700 1 -- 172.21.15.175:0/360801692 shutdown_connections 2023-10-10T18:45:04.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5f0531f700 1 --2- 172.21.15.175:0/360801692 >> v2:172.21.15.191:3300/0 conn(0x7f5f00068cf0 0x7f5f00067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5f0531f700 1 --2- 172.21.15.175:0/360801692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00159dd0 0x7f5f000c2100 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5f0531f700 1 -- 172.21.15.175:0/360801692 >> 172.21.15.175:0/360801692 conn(0x7f5f00063cc0 msgr2=0x7f5f000cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:04.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5f0531f700 1 -- 172.21.15.175:0/360801692 shutdown_connections 2023-10-10T18:45:04.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.882+0000 7f5f0531f700 1 -- 172.21.15.175:0/360801692 wait complete. 2023-10-10T18:45:04.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.883+0000 7f5f0531f700 1 Processor -- start 2023-10-10T18:45:04.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.883+0000 7f5f0531f700 1 -- start start 2023-10-10T18:45:04.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5f0531f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00068cf0 0x7f5f000cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5f0531f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f0014fd20 0x7f5f000c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5f0531f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f00159dd0 0x7f5f000c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5f0531f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f00166970 con 0x7f5f00068cf0 2023-10-10T18:45:04.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5f0531f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f00165aa0 con 0x7f5f00159dd0 2023-10-10T18:45:04.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5f0531f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f00165d60 con 0x7f5f0014fd20 2023-10-10T18:45:04.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5efffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00068cf0 0x7f5f000cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5efffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00068cf0 0x7f5f000cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50628/0 (socket says 172.21.15.175:50628) 2023-10-10T18:45:04.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.884+0000 7f5efffff700 1 -- 172.21.15.175:0/951197576 learned_addr learned my addr 172.21.15.175:0/951197576 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:04.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5eff7fe700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f0014fd20 0x7f5f000c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5f04b1e700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f00159dd0 0x7f5f000c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5efffff700 1 -- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f0014fd20 msgr2=0x7f5f000c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5efffff700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f0014fd20 0x7f5f000c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5efffff700 1 -- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f00159dd0 msgr2=0x7f5f000c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:04.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5efffff700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f00159dd0 0x7f5f000c42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:04.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5efffff700 1 -- 172.21.15.175:0/951197576 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef40094f0 con 0x7f5f00068cf0 2023-10-10T18:45:04.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5efffff700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00068cf0 0x7f5f000cc270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000c830 tx=0x7f5ef0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5ee7fff700 1 -- 172.21.15.175:0/951197576 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef000a680 con 0x7f5f00068cf0 2023-10-10T18:45:04.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5ee7fff700 1 -- 172.21.15.175:0/951197576 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef000a7e0 con 0x7f5f00068cf0 2023-10-10T18:45:04.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f00166040 con 0x7f5f00068cf0 2023-10-10T18:45:04.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5ee7fff700 1 -- 172.21.15.175:0/951197576 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef000aba0 con 0x7f5f00068cf0 2023-10-10T18:45:04.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.885+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f00166600 con 0x7f5f00068cf0 2023-10-10T18:45:04.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.887+0000 7f5ee7fff700 1 -- 172.21.15.175:0/951197576 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f5ef00113f0 con 0x7f5f00068cf0 2023-10-10T18:45:04.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.887+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f00067570 con 0x7f5f00068cf0 2023-10-10T18:45:04.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.887+0000 7f5ee7fff700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5eec066ea0 0x7f5eec069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:04.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.887+0000 7f5ee7fff700 1 -- 172.21.15.175:0/951197576 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f5ef0084450 con 0x7f5f00068cf0 2023-10-10T18:45:04.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.887+0000 7f5eff7fe700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5eec066ea0 0x7f5eec069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:04.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.888+0000 7f5eff7fe700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5eec066ea0 0x7f5eec069350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40099b0 tx=0x7f5ef4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:04.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:04.891+0000 7f5ee7fff700 1 -- 172.21.15.175:0/951197576 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ef0051c70 con 0x7f5f00068cf0 2023-10-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.124+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f000b18c0 con 0x7f5f00068cf0 2023-10-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.124+0000 7f5ee7fff700 1 -- 172.21.15.175:0/951197576 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1259 (secure 0 0 0) 0x7f5ef0015500 con 0x7f5f00068cf0 2023-10-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-10-10T18:45:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5eec066ea0 msgr2=0x7f5eec069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5eec066ea0 0x7f5eec069350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40099b0 tx=0x7f5ef4017000 comp rx=0 tx=0).stop 2023-10-10T18:45:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00068cf0 msgr2=0x7f5f000cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00068cf0 0x7f5f000cc270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000c830 tx=0x7f5ef0008040 comp rx=0 tx=0).stop 2023-10-10T18:45:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 shutdown_connections 2023-10-10T18:45:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5eec066ea0 0x7f5eec069350 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f00159dd0 0x7f5f000c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f00068cf0 0x7f5f000cc270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 --2- 172.21.15.175:0/951197576 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f0014fd20 0x7f5f000c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:05.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.126+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 >> 172.21.15.175:0/951197576 conn(0x7f5f00063cc0 msgr2=0x7f5f000c92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:05.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.127+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 shutdown_connections 2023-10-10T18:45:05.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:05.127+0000 7f5f0531f700 1 -- 172.21.15.175:0/951197576 wait complete. 2023-10-10T18:45:05.131 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 2 2023-10-10T18:45:07.140 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:07.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.268+0000 7fea793a5700 1 Processor -- start 2023-10-10T18:45:07.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.268+0000 7fea793a5700 1 -- start start 2023-10-10T18:45:07.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea793a5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fea740c7190 0x7fea7415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea793a5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c4580 0x7fea74063c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea793a5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea74157030 con 0x7fea740c6330 2023-10-10T18:45:07.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea793a5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea740c5000 con 0x7fea740c7190 2023-10-10T18:45:07.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea793a5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea74156420 con 0x7fea740c4580 2023-10-10T18:45:07.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea727fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fea740c7190 0x7fea7415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea727fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fea740c7190 0x7fea7415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50640/0 (socket says 172.21.15.175:50640) 2023-10-10T18:45:07.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea727fc700 1 -- 172.21.15.175:0/2402261102 learned_addr learned my addr 172.21.15.175:0/2402261102 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:07.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea737fe700 1 --2- 172.21.15.175:0/2402261102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c4580 0x7fea74063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea727fc700 1 -- 172.21.15.175:0/2402261102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c4580 msgr2=0x7fea74063c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:07.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea727fc700 1 --2- 172.21.15.175:0/2402261102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c4580 0x7fea74063c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea727fc700 1 -- 172.21.15.175:0/2402261102 >> v1:172.21.15.191:6789/0 conn(0x7fea740c6330 legacy=0x7fea740c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea727fc700 1 -- 172.21.15.175:0/2402261102 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74156700 con 0x7fea740c7190 2023-10-10T18:45:07.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea727fc700 1 --2- 172.21.15.175:0/2402261102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea7415a030 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fea680099b0 tx=0x7fea68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06b2966b52c1cde server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:07.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea71ffb700 1 -- 172.21.15.175:0/2402261102 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68009ea0 con 0x7fea740c7190 2023-10-10T18:45:07.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea71ffb700 1 -- 172.21.15.175:0/2402261102 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6800b8f0 con 0x7fea740c7190 2023-10-10T18:45:07.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.269+0000 7fea71ffb700 1 -- 172.21.15.175:0/2402261102 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6800bc90 con 0x7fea740c7190 2023-10-10T18:45:07.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.270+0000 7fea793a5700 1 -- 172.21.15.175:0/2402261102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 msgr2=0x7fea7415a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.270+0000 7fea793a5700 1 --2- 172.21.15.175:0/2402261102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea7415a030 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fea680099b0 tx=0x7fea68009250 comp rx=0 tx=0).stop 2023-10-10T18:45:07.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.270+0000 7fea793a5700 1 -- 172.21.15.175:0/2402261102 shutdown_connections 2023-10-10T18:45:07.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.270+0000 7fea793a5700 1 --2- 172.21.15.175:0/2402261102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c4580 0x7fea74063c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.270+0000 7fea793a5700 1 --2- 172.21.15.175:0/2402261102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea7415a030 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.270+0000 7fea793a5700 1 -- 172.21.15.175:0/2402261102 >> 172.21.15.175:0/2402261102 conn(0x7fea7406ef20 msgr2=0x7fea74063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:07.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.270+0000 7fea793a5700 1 -- 172.21.15.175:0/2402261102 shutdown_connections 2023-10-10T18:45:07.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.270+0000 7fea793a5700 1 -- 172.21.15.175:0/2402261102 wait complete. 2023-10-10T18:45:07.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.271+0000 7fea793a5700 1 Processor -- start 2023-10-10T18:45:07.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.271+0000 7fea793a5700 1 -- start start 2023-10-10T18:45:07.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea793a5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c4580 0x7fea74169650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea793a5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c6330 0x7fea74169b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea793a5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea74165f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea793a5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea740cce20 con 0x7fea740c7190 2023-10-10T18:45:07.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea793a5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea740cd090 con 0x7fea740c4580 2023-10-10T18:45:07.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea793a5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea740c9440 con 0x7fea740c6330 2023-10-10T18:45:07.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea74165f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea727fc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c6330 0x7fea74169b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea74165f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50654/0 (socket says 172.21.15.175:50654) 2023-10-10T18:45:07.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea72ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c4580 0x7fea74169650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 -- 172.21.15.175:0/1372726848 learned_addr learned my addr 172.21.15.175:0/1372726848 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:07.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 -- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c6330 msgr2=0x7fea74169b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c6330 0x7fea74169b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 -- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c4580 msgr2=0x7fea74169650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c4580 0x7fea74169650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 -- 172.21.15.175:0/1372726848 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea680094f0 con 0x7fea740c7190 2023-10-10T18:45:07.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea72ffd700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c4580 0x7fea74169650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:07.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.272+0000 7fea737fe700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea74165f10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009ef0 tx=0x7fea6c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:07.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.273+0000 7fea5affd700 1 -- 172.21.15.175:0/1372726848 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c00cb70 con 0x7fea740c7190 2023-10-10T18:45:07.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.273+0000 7fea5affd700 1 -- 172.21.15.175:0/1372726848 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6c00f040 con 0x7fea740c7190 2023-10-10T18:45:07.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.273+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea740c9720 con 0x7fea740c7190 2023-10-10T18:45:07.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.273+0000 7fea5affd700 1 -- 172.21.15.175:0/1372726848 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c014480 con 0x7fea740c7190 2023-10-10T18:45:07.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.273+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea740cd370 con 0x7fea740c7190 2023-10-10T18:45:07.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.274+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea741550c0 con 0x7fea740c7190 2023-10-10T18:45:07.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.274+0000 7fea5affd700 1 -- 172.21.15.175:0/1372726848 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7fea6c0147f0 con 0x7fea740c7190 2023-10-10T18:45:07.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.274+0000 7fea5affd700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea60066ed0 0x7fea60069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.274+0000 7fea5affd700 1 -- 172.21.15.175:0/1372726848 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fea6c084d80 con 0x7fea740c7190 2023-10-10T18:45:07.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.275+0000 7fea72ffd700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea60066ed0 0x7fea60069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.275+0000 7fea72ffd700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea60066ed0 0x7fea60069380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fea64001dc0 tx=0x7fea64009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:07.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.279+0000 7fea5affd700 1 -- 172.21.15.175:0/1372726848 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea6c052420 con 0x7fea740c7190 2023-10-10T18:45:07.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.519+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea740c9910 con 0x7fea740c7190 2023-10-10T18:45:07.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.519+0000 7fea5affd700 1 -- 172.21.15.175:0/1372726848 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5045 (secure 0 0 0) 0x7fea6c018090 con 0x7fea740c7190 2023-10-10T18:45:07.521 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:07.521 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4250,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-10-10T18:45:07.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea60066ed0 msgr2=0x7fea60069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea60066ed0 0x7fea60069380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fea64001dc0 tx=0x7fea64009040 comp rx=0 tx=0).stop 2023-10-10T18:45:07.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 msgr2=0x7fea74165f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea74165f10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009ef0 tx=0x7fea6c00b040 comp rx=0 tx=0).stop 2023-10-10T18:45:07.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 shutdown_connections 2023-10-10T18:45:07.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea60066ed0 0x7fea60069380 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea740c7190 0x7fea74165f10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c4580 0x7fea74169650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 --2- 172.21.15.175:0/1372726848 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea740c6330 0x7fea74169b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 >> 172.21.15.175:0/1372726848 conn(0x7fea7406ef20 msgr2=0x7fea740c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:07.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 shutdown_connections 2023-10-10T18:45:07.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.521+0000 7fea793a5700 1 -- 172.21.15.175:0/1372726848 wait complete. 2023-10-10T18:45:07.525 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 3 2023-10-10T18:45:07.535 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:07.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.652+0000 7fa45c908700 1 Processor -- start 2023-10-10T18:45:07.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.652+0000 7fa45c908700 1 -- start start 2023-10-10T18:45:07.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa45c908700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa458068c00 0x7fa458159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa45c908700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa458067fa0 con 0x7fa45814fd90 2023-10-10T18:45:07.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa45c908700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa458156b70 con 0x7fa45808a100 2023-10-10T18:45:07.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa45c908700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580c5600 con 0x7fa458068c00 2023-10-10T18:45:07.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa456d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa45814fd90 0x7fa4580c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56376/0 (socket says 172.21.15.175:56376) 2023-10-10T18:45:07.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa456d9d700 1 -- 172.21.15.175:0/1629016728 learned_addr learned my addr 172.21.15.175:0/1629016728 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:07.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa45759e700 1 --2- 172.21.15.175:0/1629016728 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa458068c00 0x7fa458159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa455d9b700 1 -- 172.21.15.175:0/1629016728 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401342908 0 0) 0x7fa458067fa0 con 0x7fa45814fd90 2023-10-10T18:45:07.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa455d9b700 1 -- 172.21.15.175:0/1629016728 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa440003570 con 0x7fa45814fd90 2023-10-10T18:45:07.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa455d9b700 1 -- 172.21.15.175:0/1629016728 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468852261 0 0) 0x7fa458156b70 con 0x7fa45808a100 2023-10-10T18:45:07.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa455d9b700 1 -- 172.21.15.175:0/1629016728 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa458067fa0 con 0x7fa45808a100 2023-10-10T18:45:07.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa45759e700 1 -- 172.21.15.175:0/1629016728 >> v1:172.21.15.191:6789/0 conn(0x7fa45808a100 legacy=0x7fa4580684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa45759e700 1 -- 172.21.15.175:0/1629016728 >> v1:172.21.15.175:6789/0 conn(0x7fa45814fd90 legacy=0x7fa4580c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.653+0000 7fa45759e700 1 -- 172.21.15.175:0/1629016728 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa458150160 con 0x7fa458068c00 2023-10-10T18:45:07.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa45759e700 1 --2- 172.21.15.175:0/1629016728 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa458068c00 0x7fa458159c30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa450009e90 tx=0x7fa450009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2040bb9a72b6ace5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:07.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa455d9b700 1 -- 172.21.15.175:0/1629016728 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45000e070 con 0x7fa458068c00 2023-10-10T18:45:07.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa455d9b700 1 -- 172.21.15.175:0/1629016728 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45000bb80 con 0x7fa458068c00 2023-10-10T18:45:07.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa455d9b700 1 -- 172.21.15.175:0/1629016728 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450012430 con 0x7fa458068c00 2023-10-10T18:45:07.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa45c908700 1 -- 172.21.15.175:0/1629016728 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa458068c00 msgr2=0x7fa458159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa45c908700 1 --2- 172.21.15.175:0/1629016728 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa458068c00 0x7fa458159c30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa450009e90 tx=0x7fa450009780 comp rx=0 tx=0).stop 2023-10-10T18:45:07.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa45c908700 1 -- 172.21.15.175:0/1629016728 shutdown_connections 2023-10-10T18:45:07.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa45c908700 1 --2- 172.21.15.175:0/1629016728 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa458068c00 0x7fa458159c30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa45c908700 1 -- 172.21.15.175:0/1629016728 >> 172.21.15.175:0/1629016728 conn(0x7fa458063d00 msgr2=0x7fa4580cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:07.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa45c908700 1 -- 172.21.15.175:0/1629016728 shutdown_connections 2023-10-10T18:45:07.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.654+0000 7fa45c908700 1 -- 172.21.15.175:0/1629016728 wait complete. 2023-10-10T18:45:07.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.655+0000 7fa45c908700 1 Processor -- start 2023-10-10T18:45:07.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.655+0000 7fa45c908700 1 -- start start 2023-10-10T18:45:07.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa45c908700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa458068c00 0x7fa458153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa45c908700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa45808a100 0x7fa458169170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa45c908700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45814fd90 0x7fa4581696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa45c908700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580a8140 con 0x7fa458068c00 2023-10-10T18:45:07.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa45c908700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580a7c10 con 0x7fa45814fd90 2023-10-10T18:45:07.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa45c908700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa458157eb0 con 0x7fa45808a100 2023-10-10T18:45:07.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa456d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa458068c00 0x7fa458153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa456d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa458068c00 0x7fa458153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50670/0 (socket says 172.21.15.175:50670) 2023-10-10T18:45:07.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa456d9d700 1 -- 172.21.15.175:0/3603837629 learned_addr learned my addr 172.21.15.175:0/3603837629 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:07.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa45659c700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa45808a100 0x7fa458169170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa45759e700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45814fd90 0x7fa4581696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa456d9d700 1 -- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa45808a100 msgr2=0x7fa458169170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa456d9d700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa45808a100 0x7fa458169170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa456d9d700 1 -- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45814fd90 msgr2=0x7fa4581696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.656+0000 7fa456d9d700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45814fd90 0x7fa4581696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.657+0000 7fa456d9d700 1 -- 172.21.15.175:0/3603837629 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa450009a20 con 0x7fa458068c00 2023-10-10T18:45:07.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.657+0000 7fa456d9d700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa458068c00 0x7fa458153030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa44800c790 tx=0x7fa448007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:07.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.657+0000 7fa43effd700 1 -- 172.21.15.175:0/3603837629 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448005b30 con 0x7fa458068c00 2023-10-10T18:45:07.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.657+0000 7fa43effd700 1 -- 172.21.15.175:0/3603837629 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa448005cd0 con 0x7fa458068c00 2023-10-10T18:45:07.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.657+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4580a92e0 con 0x7fa458068c00 2023-10-10T18:45:07.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.657+0000 7fa43effd700 1 -- 172.21.15.175:0/3603837629 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448007320 con 0x7fa458068c00 2023-10-10T18:45:07.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.657+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa458152b30 con 0x7fa458068c00 2023-10-10T18:45:07.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.658+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4581590b0 con 0x7fa458068c00 2023-10-10T18:45:07.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.659+0000 7fa43effd700 1 -- 172.21.15.175:0/3603837629 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7fa448012400 con 0x7fa458068c00 2023-10-10T18:45:07.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.659+0000 7fa43effd700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa444066ea0 0x7fa444069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.659+0000 7fa43effd700 1 -- 172.21.15.175:0/3603837629 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fa448083be0 con 0x7fa458068c00 2023-10-10T18:45:07.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.659+0000 7fa45659c700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa444066ea0 0x7fa444069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.660+0000 7fa45659c700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa444066ea0 0x7fa444069350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa44c007fd0 tx=0x7fa44c007d10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:07.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.663+0000 7fa43effd700 1 -- 172.21.15.175:0/3603837629 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa4480513b0 con 0x7fa458068c00 2023-10-10T18:45:07.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.848+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa458067570 con 0x7fa458068c00 2023-10-10T18:45:07.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.848+0000 7fa43effd700 1 -- 172.21.15.175:0/3603837629 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12188 (secure 0 0 0) 0x7fa448017090 con 0x7fa458068c00 2023-10-10T18:45:07.850 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:07.850 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":16,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:07.060255+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:07.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa444066ea0 msgr2=0x7fa444069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa444066ea0 0x7fa444069350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa44c007fd0 tx=0x7fa44c007d10 comp rx=0 tx=0).stop 2023-10-10T18:45:07.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa458068c00 msgr2=0x7fa458153030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa458068c00 0x7fa458153030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa44800c790 tx=0x7fa448007e80 comp rx=0 tx=0).stop 2023-10-10T18:45:07.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 shutdown_connections 2023-10-10T18:45:07.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa444066ea0 0x7fa444069350 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45814fd90 0x7fa4581696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa458068c00 0x7fa458153030 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 --2- 172.21.15.175:0/3603837629 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa45808a100 0x7fa458169170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 >> 172.21.15.175:0/3603837629 conn(0x7fa458063d00 msgr2=0x7fa4580cbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:07.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 shutdown_connections 2023-10-10T18:45:07.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.850+0000 7fa45c908700 1 -- 172.21.15.175:0/3603837629 wait complete. 2023-10-10T18:45:07.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T18:45:05.161495+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-10T18:45:07.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T18:45:07.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.986+0000 7f3125fed700 1 Processor -- start 2023-10-10T18:45:07.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.986+0000 7f3125fed700 1 -- start start 2023-10-10T18:45:07.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3125fed700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f312006d1e0 0x7f3120155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3125fed700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3120159350 con 0x7f31200c7ae0 2023-10-10T18:45:07.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3125fed700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3120159bf0 con 0x7f31200c4d50 2023-10-10T18:45:07.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3125fed700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31200c6040 con 0x7f312006d1e0 2023-10-10T18:45:07.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3124feb700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f312006d1e0 0x7f3120155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3124feb700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f312006d1e0 0x7f3120155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50680/0 (socket says 172.21.15.175:50680) 2023-10-10T18:45:07.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3124feb700 1 -- 172.21.15.175:0/672799325 learned_addr learned my addr 172.21.15.175:0/672799325 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:07.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3124feb700 1 -- 172.21.15.175:0/672799325 >> v1:172.21.15.191:6789/0 conn(0x7f31200c4d50 legacy=0x7f3120160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3124feb700 1 -- 172.21.15.175:0/672799325 >> v1:172.21.15.175:6789/0 conn(0x7f31200c7ae0 legacy=0x7f3120152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3124feb700 1 -- 172.21.15.175:0/672799325 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3120068aa0 con 0x7f312006d1e0 2023-10-10T18:45:07.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f311f7fe700 1 -- 172.21.15.175:0/672799325 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024525247 0 0) 0x7f3120159350 con 0x7f31200c7ae0 2023-10-10T18:45:07.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f3124feb700 1 --2- 172.21.15.175:0/672799325 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f312006d1e0 0x7f3120155dd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3118009f20 tx=0x7f3118009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3bb01cc0602c5b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:07.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.987+0000 7f311f7fe700 1 -- 172.21.15.175:0/672799325 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311800e070 con 0x7f312006d1e0 2023-10-10T18:45:07.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f311f7fe700 1 -- 172.21.15.175:0/672799325 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311800bb10 con 0x7f312006d1e0 2023-10-10T18:45:07.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f311f7fe700 1 -- 172.21.15.175:0/672799325 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31180092e0 con 0x7f312006d1e0 2023-10-10T18:45:07.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f3125fed700 1 -- 172.21.15.175:0/672799325 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f312006d1e0 msgr2=0x7f3120155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f3125fed700 1 --2- 172.21.15.175:0/672799325 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f312006d1e0 0x7f3120155dd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3118009f20 tx=0x7f3118009840 comp rx=0 tx=0).stop 2023-10-10T18:45:07.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f3125fed700 1 -- 172.21.15.175:0/672799325 shutdown_connections 2023-10-10T18:45:07.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f3125fed700 1 --2- 172.21.15.175:0/672799325 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f312006d1e0 0x7f3120155dd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f3125fed700 1 -- 172.21.15.175:0/672799325 >> 172.21.15.175:0/672799325 conn(0x7f312006bbb0 msgr2=0x7f312006bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:07.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f3125fed700 1 -- 172.21.15.175:0/672799325 shutdown_connections 2023-10-10T18:45:07.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.988+0000 7f3125fed700 1 -- 172.21.15.175:0/672799325 wait complete. 2023-10-10T18:45:07.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.989+0000 7f3125fed700 1 Processor -- start 2023-10-10T18:45:07.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f3125fed700 1 -- start start 2023-10-10T18:45:07.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f3125fed700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f312006d1e0 0x7f31200bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f3125fed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31200c4d50 0x7f31200c3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f3125fed700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31200c7ae0 0x7f312014dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:07.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f3125fed700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31200c4550 con 0x7f31200c4d50 2023-10-10T18:45:07.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f3125fed700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3120166290 con 0x7f312006d1e0 2023-10-10T18:45:07.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f3125fed700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31200bf7f0 con 0x7f31200c7ae0 2023-10-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f311ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31200c4d50 0x7f31200c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f3124feb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f312006d1e0 0x7f31200bf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f311ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31200c4d50 0x7f31200c3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50686/0 (socket says 172.21.15.175:50686) 2023-10-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f311ffff700 1 -- 172.21.15.175:0/489658989 learned_addr learned my addr 172.21.15.175:0/489658989 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.990+0000 7f31257ec700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31200c7ae0 0x7f312014dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:07.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311ffff700 1 -- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31200c7ae0 msgr2=0x7f312014dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311ffff700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31200c7ae0 0x7f312014dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311ffff700 1 -- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f312006d1e0 msgr2=0x7f31200bf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:07.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311ffff700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f312006d1e0 0x7f31200bf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:07.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311ffff700 1 -- 172.21.15.175:0/489658989 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3118009ae0 con 0x7f31200c4d50 2023-10-10T18:45:08.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311ffff700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31200c4d50 0x7f31200c3390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f311000c600 tx=0x7f3110007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:08.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311cff9700 1 -- 172.21.15.175:0/489658989 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110011590 con 0x7f31200c4d50 2023-10-10T18:45:08.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311cff9700 1 -- 172.21.15.175:0/489658989 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110011710 con 0x7f31200c4d50 2023-10-10T18:45:08.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31200c9e30 con 0x7f31200c4d50 2023-10-10T18:45:08.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f311cff9700 1 -- 172.21.15.175:0/489658989 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110011ad0 con 0x7f31200c4d50 2023-10-10T18:45:08.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.991+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3120165c40 con 0x7f31200c4d50 2023-10-10T18:45:08.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.992+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3120165aa0 con 0x7f31200c4d50 2023-10-10T18:45:08.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.993+0000 7f311cff9700 1 -- 172.21.15.175:0/489658989 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f311001b3b0 con 0x7f31200c4d50 2023-10-10T18:45:08.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.993+0000 7f311cff9700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3100066ea0 0x7f3100069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.993+0000 7f311cff9700 1 -- 172.21.15.175:0/489658989 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f31100834f0 con 0x7f31200c4d50 2023-10-10T18:45:08.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.993+0000 7f3124feb700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3100066ea0 0x7f3100069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.993+0000 7f3124feb700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3100066ea0 0x7f3100069350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3118009880 tx=0x7f3118018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:08.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:07.997+0000 7f311cff9700 1 -- 172.21.15.175:0/489658989 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3120165aa0 con 0x7f31200c4d50 2023-10-10T18:45:08.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.183+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3120165aa0 con 0x7f31200c4d50 2023-10-10T18:45:08.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.184+0000 7f311cff9700 1 -- 172.21.15.175:0/489658989 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3120165aa0 con 0x7f31200c4d50 2023-10-10T18:45:08.185 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 1 2023-10-10T18:45:08.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3100066ea0 msgr2=0x7f3100069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3100066ea0 0x7f3100069350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3118009880 tx=0x7f3118018000 comp rx=0 tx=0).stop 2023-10-10T18:45:08.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31200c4d50 msgr2=0x7f31200c3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31200c4d50 0x7f31200c3390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f311000c600 tx=0x7f3110007d70 comp rx=0 tx=0).stop 2023-10-10T18:45:08.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 shutdown_connections 2023-10-10T18:45:08.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3100066ea0 0x7f3100069350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31200c7ae0 0x7f312014dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f312006d1e0 0x7f31200bf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 --2- 172.21.15.175:0/489658989 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31200c4d50 0x7f31200c3390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 >> 172.21.15.175:0/489658989 conn(0x7f312006bbb0 msgr2=0x7f3120151e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:08.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 shutdown_connections 2023-10-10T18:45:08.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.186+0000 7f3125fed700 1 -- 172.21.15.175:0/489658989 wait complete. 2023-10-10T18:45:08.199 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-10T18:45:08.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.313+0000 7f597eb06700 1 Processor -- start 2023-10-10T18:45:08.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.313+0000 7f597eb06700 1 -- start start 2023-10-10T18:45:08.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.314+0000 7f597eb06700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 0x7f5978156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.314+0000 7f597eb06700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59780c9cc0 con 0x7f597806f5f0 2023-10-10T18:45:08.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.314+0000 7f597eb06700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597806e650 con 0x7f59780c64a0 2023-10-10T18:45:08.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.314+0000 7f597eb06700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59781571c0 con 0x7f59780c60d0 2023-10-10T18:45:08.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.314+0000 7f5977fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f597806f5f0 0x7f59780c3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56396/0 (socket says 172.21.15.175:56396) 2023-10-10T18:45:08.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.314+0000 7f5977fff700 1 -- 172.21.15.175:0/911269606 learned_addr learned my addr 172.21.15.175:0/911269606 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:08.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f597c8a2700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 0x7f5978156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227641795 0 0) 0x7f59780c9cc0 con 0x7f597806f5f0 2023-10-10T18:45:08.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5960003570 con 0x7f597806f5f0 2023-10-10T18:45:08.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439062945 0 0) 0x7f5960003570 con 0x7f597806f5f0 2023-10-10T18:45:08.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 msgr2=0x7f5978156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:08.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 0x7f5978156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 >> v1:172.21.15.191:6789/0 conn(0x7f59780c64a0 legacy=0x7f597814cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597806e930 con 0x7f597806f5f0 2023-10-10T18:45:08.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f596c001470 con 0x7f597806f5f0 2023-10-10T18:45:08.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 >> v1:172.21.15.175:6789/0 conn(0x7f597806f5f0 legacy=0x7f59780c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 0x7f5960003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f5960004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960004d50 0x7f59600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59781571c0 con 0x7f5960004010 2023-10-10T18:45:08.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596000bbf0 con 0x7f5960004d50 2023-10-10T18:45:08.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596000dfc0 con 0x7f59780c60d0 2023-10-10T18:45:08.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f5977fff700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f5960004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f597c8a2700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 0x7f5960003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f597d0a3700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960004d50 0x7f59600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f5977fff700 1 -- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 msgr2=0x7f5960003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:08.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f5977fff700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 0x7f5960003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f5977fff700 1 -- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960004d50 msgr2=0x7f59600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:08.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f5977fff700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5960004d50 0x7f59600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f5977fff700 1 -- 172.21.15.175:0/911269606 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5960003570 con 0x7f5960004010 2023-10-10T18:45:08.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f597c8a2700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59780c60d0 0x7f5960003a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:08.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.315+0000 7f5977fff700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f5960004810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f596c001ed0 tx=0x7f5978071740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488741232d6f0e1c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:08.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c00cc00 con 0x7f5960004010 2023-10-10T18:45:08.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c00cd60 con 0x7f5960004010 2023-10-10T18:45:08.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f59777fe700 1 -- 172.21.15.175:0/911269606 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c005b70 con 0x7f5960004010 2023-10-10T18:45:08.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f597eb06700 1 -- 172.21.15.175:0/911269606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 msgr2=0x7f5960004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f597eb06700 1 --2- 172.21.15.175:0/911269606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f5960004810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f596c001ed0 tx=0x7f5978071740 comp rx=0 tx=0).stop 2023-10-10T18:45:08.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f597d0a3700 1 -- 172.21.15.175:0/911269606 reap_dead start 2023-10-10T18:45:08.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f597eb06700 1 -- 172.21.15.175:0/911269606 shutdown_connections 2023-10-10T18:45:08.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f597eb06700 1 -- 172.21.15.175:0/911269606 >> 172.21.15.175:0/911269606 conn(0x7f597806c110 msgr2=0x7f597806c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:08.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f597eb06700 1 -- 172.21.15.175:0/911269606 shutdown_connections 2023-10-10T18:45:08.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.316+0000 7f597eb06700 1 -- 172.21.15.175:0/911269606 wait complete. 2023-10-10T18:45:08.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.317+0000 7f597eb06700 1 Processor -- start 2023-10-10T18:45:08.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.317+0000 7f597eb06700 1 -- start start 2023-10-10T18:45:08.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597eb06700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f59781631f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597eb06700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978068dd0 0x7f5978154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597eb06700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5978143ce0 0x7f5978144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597eb06700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59780c4500 con 0x7f5960004010 2023-10-10T18:45:08.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597eb06700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59780684f0 con 0x7f5978068dd0 2023-10-10T18:45:08.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597eb06700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5978162120 con 0x7f5978143ce0 2023-10-10T18:45:08.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597c8a2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f59781631f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597c8a2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f59781631f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50698/0 (socket says 172.21.15.175:50698) 2023-10-10T18:45:08.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597c8a2700 1 -- 172.21.15.175:0/3585645876 learned_addr learned my addr 172.21.15.175:0/3585645876 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:08.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f5977fff700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978068dd0 0x7f5978154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597d0a3700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5978143ce0 0x7f5978144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597c8a2700 1 -- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5978143ce0 msgr2=0x7f5978144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597c8a2700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5978143ce0 0x7f5978144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597c8a2700 1 -- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978068dd0 msgr2=0x7f5978154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597c8a2700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978068dd0 0x7f5978154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.318+0000 7f597c8a2700 1 -- 172.21.15.175:0/3585645876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978067920 con 0x7f5960004010 2023-10-10T18:45:08.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.319+0000 7f597c8a2700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f59781631f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f596800f390 tx=0x7f596800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:08.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.319+0000 7f5974ff9700 1 -- 172.21.15.175:0/3585645876 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596800e040 con 0x7f5960004010 2023-10-10T18:45:08.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.319+0000 7f5974ff9700 1 -- 172.21.15.175:0/3585645876 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596800f040 con 0x7f5960004010 2023-10-10T18:45:08.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.319+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5978147990 con 0x7f5960004010 2023-10-10T18:45:08.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.319+0000 7f5974ff9700 1 -- 172.21.15.175:0/3585645876 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968015680 con 0x7f5960004010 2023-10-10T18:45:08.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.319+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5978067ea0 con 0x7f5960004010 2023-10-10T18:45:08.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.320+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5978067570 con 0x7f5960004010 2023-10-10T18:45:08.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.320+0000 7f5974ff9700 1 -- 172.21.15.175:0/3585645876 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f5968003020 con 0x7f5960004010 2023-10-10T18:45:08.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.320+0000 7f5974ff9700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5964066ed0 0x7f5964069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.320+0000 7f5974ff9700 1 -- 172.21.15.175:0/3585645876 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f5968084bd0 con 0x7f5960004010 2023-10-10T18:45:08.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.320+0000 7f5977fff700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5964066ed0 0x7f5964069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.321+0000 7f5977fff700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5964066ed0 0x7f5964069380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f596c007cd0 tx=0x7f596c015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:08.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.324+0000 7f5974ff9700 1 -- 172.21.15.175:0/3585645876 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5968052270 con 0x7f5960004010 2023-10-10T18:45:08.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.509+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f597806a490 con 0x7f5960004010 2023-10-10T18:45:08.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.510+0000 7f5974ff9700 1 -- 172.21.15.175:0/3585645876 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5968055890 con 0x7f5960004010 2023-10-10T18:45:08.511 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:08.511 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-10-10T18:45:08.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-10-10T18:45:08.512 INFO:teuthology.orchestra.run.smithi175.stdout: "poolnum": 1, 2023-10-10T18:45:08.512 INFO:teuthology.orchestra.run.smithi175.stdout: "poolname": ".mgr" 2023-10-10T18:45:08.512 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-10-10T18:45:08.512 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-10-10T18:45:08.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5964066ed0 msgr2=0x7f5964069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5964066ed0 0x7f5964069380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f596c007cd0 tx=0x7f596c015000 comp rx=0 tx=0).stop 2023-10-10T18:45:08.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 msgr2=0x7f59781631f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f59781631f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f596800f390 tx=0x7f596800b040 comp rx=0 tx=0).stop 2023-10-10T18:45:08.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 shutdown_connections 2023-10-10T18:45:08.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5964066ed0 0x7f5964069380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5978143ce0 0x7f5978144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5960004010 0x7f59781631f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 --2- 172.21.15.175:0/3585645876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978068dd0 0x7f5978154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 >> 172.21.15.175:0/3585645876 conn(0x7f597806c110 msgr2=0x7f59781568b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:08.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 shutdown_connections 2023-10-10T18:45:08.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.512+0000 7f597eb06700 1 -- 172.21.15.175:0/3585645876 wait complete. 2023-10-10T18:45:08.526 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-10T18:45:08.526 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-10T18:45:08.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.643+0000 7f72de9da700 1 Processor -- start 2023-10-10T18:45:08.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.643+0000 7f72de9da700 1 -- start start 2023-10-10T18:45:08.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72de9da700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c6610 0x7f72d80c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72de9da700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f72d80c4030 0x7f72d8159720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72de9da700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d80c4b60 con 0x7f72d80c6240 2023-10-10T18:45:08.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72de9da700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d8068c50 con 0x7f72d80c4030 2023-10-10T18:45:08.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72de9da700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d8157420 con 0x7f72d80c6610 2023-10-10T18:45:08.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72dcf77700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f72d80c4030 0x7f72d8159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72dcf77700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f72d80c4030 0x7f72d8159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50708/0 (socket says 172.21.15.175:50708) 2023-10-10T18:45:08.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72dcf77700 1 -- 172.21.15.175:0/613329986 learned_addr learned my addr 172.21.15.175:0/613329986 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:08.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.644+0000 7f72d77fe700 1 --2- 172.21.15.175:0/613329986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c6610 0x7f72d80c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72dcf77700 1 -- 172.21.15.175:0/613329986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c6610 msgr2=0x7f72d80c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:08.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72dcf77700 1 --2- 172.21.15.175:0/613329986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c6610 0x7f72d80c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72dcf77700 1 -- 172.21.15.175:0/613329986 >> v1:172.21.15.191:6789/0 conn(0x7f72d80c6240 legacy=0x7f72d80690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72dcf77700 1 -- 172.21.15.175:0/613329986 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d80c4880 con 0x7f72d80c4030 2023-10-10T18:45:08.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72dcf77700 1 --2- 172.21.15.175:0/613329986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c4030 0x7f72d8159720 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009f20 tx=0x7f72d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475de5e42dc96fb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:08.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72d6ffd700 1 -- 172.21.15.175:0/613329986 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d000e070 con 0x7f72d80c4030 2023-10-10T18:45:08.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72d6ffd700 1 -- 172.21.15.175:0/613329986 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d000bb10 con 0x7f72d80c4030 2023-10-10T18:45:08.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72d6ffd700 1 -- 172.21.15.175:0/613329986 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d00092e0 con 0x7f72d80c4030 2023-10-10T18:45:08.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72de9da700 1 -- 172.21.15.175:0/613329986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c4030 msgr2=0x7f72d8159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72de9da700 1 --2- 172.21.15.175:0/613329986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c4030 0x7f72d8159720 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009f20 tx=0x7f72d0009840 comp rx=0 tx=0).stop 2023-10-10T18:45:08.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72de9da700 1 -- 172.21.15.175:0/613329986 shutdown_connections 2023-10-10T18:45:08.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72de9da700 1 --2- 172.21.15.175:0/613329986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c4030 0x7f72d8159720 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72de9da700 1 --2- 172.21.15.175:0/613329986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c6610 0x7f72d80c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72de9da700 1 -- 172.21.15.175:0/613329986 >> 172.21.15.175:0/613329986 conn(0x7f72d8063230 msgr2=0x7f72d8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:08.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72de9da700 1 -- 172.21.15.175:0/613329986 shutdown_connections 2023-10-10T18:45:08.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.645+0000 7f72de9da700 1 -- 172.21.15.175:0/613329986 wait complete. 2023-10-10T18:45:08.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.646+0000 7f72de9da700 1 Processor -- start 2023-10-10T18:45:08.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72de9da700 1 -- start start 2023-10-10T18:45:08.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72de9da700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c4030 0x7f72d8168e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72de9da700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72d80c6240 0x7f72d80c1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72de9da700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c6610 0x7f72d80c2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72de9da700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80cd340 con 0x7f72d80c6610 2023-10-10T18:45:08.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72de9da700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d8169a10 con 0x7f72d80c6240 2023-10-10T18:45:08.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72de9da700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d8169770 con 0x7f72d80c4030 2023-10-10T18:45:08.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72dcf77700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c6610 0x7f72d80c2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72dcf77700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c6610 0x7f72d80c2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50720/0 (socket says 172.21.15.175:50720) 2023-10-10T18:45:08.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72d77fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72d80c6240 0x7f72d80c1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72dcf77700 1 -- 172.21.15.175:0/2517268685 learned_addr learned my addr 172.21.15.175:0/2517268685 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:08.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.647+0000 7f72d7fff700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c4030 0x7f72d8168e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72dcf77700 1 -- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c4030 msgr2=0x7f72d8168e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72dcf77700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c4030 0x7f72d8168e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72dcf77700 1 -- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72d80c6240 msgr2=0x7f72d80c1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72dcf77700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72d80c6240 0x7f72d80c1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72dcf77700 1 -- 172.21.15.175:0/2517268685 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0009ae0 con 0x7f72d80c6610 2023-10-10T18:45:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72dcf77700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c6610 0x7f72d80c2390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009430 tx=0x7f72d0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72bffff700 1 -- 172.21.15.175:0/2517268685 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0008d10 con 0x7f72d80c6610 2023-10-10T18:45:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72bffff700 1 -- 172.21.15.175:0/2517268685 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d0008e70 con 0x7f72d80c6610 2023-10-10T18:45:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d80cd0a0 con 0x7f72d80c6610 2023-10-10T18:45:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72bffff700 1 -- 172.21.15.175:0/2517268685 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d001d630 con 0x7f72d80c6610 2023-10-10T18:45:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.648+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d80c1930 con 0x7f72d80c6610 2023-10-10T18:45:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.649+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d80c1b20 con 0x7f72d80c6610 2023-10-10T18:45:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.649+0000 7f72bffff700 1 -- 172.21.15.175:0/2517268685 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f72d0003040 con 0x7f72d80c6610 2023-10-10T18:45:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.650+0000 7f72bffff700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f72c4066ea0 0x7f72c4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.650+0000 7f72bffff700 1 -- 172.21.15.175:0/2517268685 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f72d0085d20 con 0x7f72d80c6610 2023-10-10T18:45:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.650+0000 7f72d7fff700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f72c4066ea0 0x7f72c4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:08.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.650+0000 7f72d7fff700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f72c4066ea0 0x7f72c4069350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72c8003a80 tx=0x7f72c8007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:08.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.654+0000 7f72bffff700 1 -- 172.21.15.175:0/2517268685 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f72d0053320 con 0x7f72d80c6610 2023-10-10T18:45:08.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:08.837+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f72d8067570 con 0x7f72d80c6610 2023-10-10T18:45:09.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.385+0000 7f72bffff700 1 -- 172.21.15.175:0/2517268685 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f72d0056940 con 0x7f72d80c6610 2023-10-10T18:45:09.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.388+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f72c4066ea0 msgr2=0x7f72c4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:09.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.388+0000 7f72de9da700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f72c4066ea0 0x7f72c4069350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72c8003a80 tx=0x7f72c8007ec0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.388+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c6610 msgr2=0x7f72d80c2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:09.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.388+0000 7f72de9da700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c6610 0x7f72d80c2390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009430 tx=0x7f72d0018000 comp rx=0 tx=0).stop 2023-10-10T18:45:09.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.389+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 shutdown_connections 2023-10-10T18:45:09.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.389+0000 7f72de9da700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f72c4066ea0 0x7f72c4069350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.389+0000 7f72de9da700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72d80c6610 0x7f72d80c2390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.389+0000 7f72de9da700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72d80c4030 0x7f72d8168e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.389+0000 7f72de9da700 1 --2- 172.21.15.175:0/2517268685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72d80c6240 0x7f72d80c1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.389+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 >> 172.21.15.175:0/2517268685 conn(0x7f72d8063230 msgr2=0x7f72d80c7110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:09.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.389+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 shutdown_connections 2023-10-10T18:45:09.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.389+0000 7f72de9da700 1 -- 172.21.15.175:0/2517268685 wait complete. 2023-10-10T18:45:09.393 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'cephfs_metadata' created 2023-10-10T18:45:09.405 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-10T18:45:09.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.528+0000 7f478cf48700 1 Processor -- start 2023-10-10T18:45:09.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.528+0000 7f478cf48700 1 -- start start 2023-10-10T18:45:09.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f478cf48700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47880c6180 0x7f4788155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:09.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f478cf48700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4788157180 0x7f4788068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:09.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f478cf48700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47880c43e0 con 0x7f47880c6550 2023-10-10T18:45:09.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f478cf48700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47880c49a0 con 0x7f4788157180 2023-10-10T18:45:09.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f478cf48700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880c5370 con 0x7f47880c6180 2023-10-10T18:45:09.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f47877fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47880c6180 0x7f4788155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:09.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f4787fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4788157180 0x7f4788068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:09.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f4787fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4788157180 0x7f4788068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:39342/0 (socket says 172.21.15.175:39342) 2023-10-10T18:45:09.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f47877fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47880c6180 0x7f4788155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:43016/0 (socket says 172.21.15.175:43016) 2023-10-10T18:45:09.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f4787fff700 1 -- 172.21.15.175:0/1348793069 learned_addr learned my addr 172.21.15.175:0/1348793069 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:09.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.529+0000 7f4786ffd700 1 --1- 172.21.15.175:0/1348793069 >> v1:172.21.15.191:6789/0 conn(0x7f47880c6550 0x7f47880c39a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:56498/0 (socket says 172.21.15.175:56498) 2023-10-10T18:45:09.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47867fc700 1 -- 172.21.15.175:0/1348793069 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189304533 0 0) 0x7f47880c43e0 con 0x7f47880c6550 2023-10-10T18:45:09.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47867fc700 1 -- 172.21.15.175:0/1348793069 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4770003570 con 0x7f47880c6550 2023-10-10T18:45:09.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47877fe700 1 -- 172.21.15.175:0/1348793069 >> v2:172.21.15.191:3300/0 conn(0x7f4788157180 msgr2=0x7f4788068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:09.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47877fe700 1 --2- 172.21.15.175:0/1348793069 >> v2:172.21.15.191:3300/0 conn(0x7f4788157180 0x7f4788068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47877fe700 1 -- 172.21.15.175:0/1348793069 >> v1:172.21.15.191:6789/0 conn(0x7f47880c6550 legacy=0x7f47880c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:09.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47877fe700 1 -- 172.21.15.175:0/1348793069 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880c46c0 con 0x7f47880c6180 2023-10-10T18:45:09.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f4787fff700 1 --2- 172.21.15.175:0/1348793069 >> v2:172.21.15.191:3300/0 conn(0x7f4788157180 0x7f4788068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:09.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47877fe700 1 --2- 172.21.15.175:0/1348793069 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47880c6180 0x7f4788155d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f47780098e0 tx=0x7f47780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61067386d8fce425 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:09.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47867fc700 1 -- 172.21.15.175:0/1348793069 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778011630 con 0x7f47880c6180 2023-10-10T18:45:09.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47867fc700 1 -- 172.21.15.175:0/1348793069 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4778011790 con 0x7f47880c6180 2023-10-10T18:45:09.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f47867fc700 1 -- 172.21.15.175:0/1348793069 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778011b30 con 0x7f47880c6180 2023-10-10T18:45:09.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f478cf48700 1 -- 172.21.15.175:0/1348793069 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47880c6180 msgr2=0x7f4788155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:09.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f478cf48700 1 --2- 172.21.15.175:0/1348793069 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47880c6180 0x7f4788155d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f47780098e0 tx=0x7f47780091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f478cf48700 1 -- 172.21.15.175:0/1348793069 shutdown_connections 2023-10-10T18:45:09.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f478cf48700 1 --2- 172.21.15.175:0/1348793069 >> v2:172.21.15.191:3300/0 conn(0x7f4788157180 0x7f4788068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f478cf48700 1 --2- 172.21.15.175:0/1348793069 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47880c6180 0x7f4788155d00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f478cf48700 1 -- 172.21.15.175:0/1348793069 >> 172.21.15.175:0/1348793069 conn(0x7f4788063230 msgr2=0x7f4788063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:09.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f478cf48700 1 -- 172.21.15.175:0/1348793069 shutdown_connections 2023-10-10T18:45:09.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.530+0000 7f478cf48700 1 -- 172.21.15.175:0/1348793069 wait complete. 2023-10-10T18:45:09.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f478cf48700 1 Processor -- start 2023-10-10T18:45:09.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f478cf48700 1 -- start start 2023-10-10T18:45:09.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f478cf48700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47880c6180 0x7f4788166580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:09.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f478cf48700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47880c6550 0x7f47881578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:09.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f478cf48700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4788157180 0x7f47880cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:09.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f478cf48700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4788166ac0 con 0x7f47880c6550 2023-10-10T18:45:09.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f478cf48700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4788151b70 con 0x7f47880c6180 2023-10-10T18:45:09.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f478cf48700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880b22c0 con 0x7f4788157180 2023-10-10T18:45:09.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f4786ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47880c6550 0x7f47881578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:09.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f4786ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47880c6550 0x7f47881578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50730/0 (socket says 172.21.15.175:50730) 2023-10-10T18:45:09.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.532+0000 7f4786ffd700 1 -- 172.21.15.175:0/306510469 learned_addr learned my addr 172.21.15.175:0/306510469 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:09.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f47877fe700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47880c6180 0x7f4788166580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:09.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f4787fff700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4788157180 0x7f47880cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:09.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f4786ffd700 1 -- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4788157180 msgr2=0x7f47880cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:09.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f4786ffd700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4788157180 0x7f47880cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f4786ffd700 1 -- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47880c6180 msgr2=0x7f4788166580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:09.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f4786ffd700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47880c6180 0x7f4788166580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:09.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f4786ffd700 1 -- 172.21.15.175:0/306510469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780009ae0 con 0x7f47880c6550 2023-10-10T18:45:09.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f4786ffd700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47880c6550 0x7f47881578c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f477c00c7c0 tx=0x7f477c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:09.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f476f7fe700 1 -- 172.21.15.175:0/306510469 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c005c10 con 0x7f47880c6550 2023-10-10T18:45:09.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f476f7fe700 1 -- 172.21.15.175:0/306510469 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477c005d70 con 0x7f47880c6550 2023-10-10T18:45:09.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4778009420 con 0x7f47880c6550 2023-10-10T18:45:09.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f476f7fe700 1 -- 172.21.15.175:0/306510469 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c007320 con 0x7f47880c6550 2023-10-10T18:45:09.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.533+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47881520a0 con 0x7f47880c6550 2023-10-10T18:45:09.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.534+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4788069530 con 0x7f47880c6550 2023-10-10T18:45:09.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.534+0000 7f476f7fe700 1 -- 172.21.15.175:0/306510469 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f477c01b3b0 con 0x7f47880c6550 2023-10-10T18:45:09.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.535+0000 7f476f7fe700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4774066ed0 0x7f4774069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:09.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.535+0000 7f476f7fe700 1 -- 172.21.15.175:0/306510469 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5223+0+0 (secure 0 0 0) 0x7f477c083920 con 0x7f47880c6550 2023-10-10T18:45:09.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.535+0000 7f47877fe700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4774066ed0 0x7f4774069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:09.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.535+0000 7f47877fe700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4774066ed0 0x7f4774069380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4778011fd0 tx=0x7f4778017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:09.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.539+0000 7f476f7fe700 1 -- 172.21.15.175:0/306510469 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f477c050e50 con 0x7f47880c6550 2023-10-10T18:45:09.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:09.723+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4788153690 con 0x7f47880c6550 2023-10-10T18:45:10.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.093+0000 7f476f7fe700 1 -- 172.21.15.175:0/306510469 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f477c051220 con 0x7f47880c6550 2023-10-10T18:45:10.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.388+0000 7f476f7fe700 1 -- 172.21.15.175:0/306510469 <== mon.0 v2:172.21.15.175:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f477c0548e0 con 0x7f47880c6550 2023-10-10T18:45:10.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.393+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4774066ed0 msgr2=0x7f4774069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:10.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.393+0000 7f478cf48700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4774066ed0 0x7f4774069380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4778011fd0 tx=0x7f4778017000 comp rx=0 tx=0).stop 2023-10-10T18:45:10.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.393+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47880c6550 msgr2=0x7f47881578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:10.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.393+0000 7f478cf48700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47880c6550 0x7f47881578c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f477c00c7c0 tx=0x7f477c007e00 comp rx=0 tx=0).stop 2023-10-10T18:45:10.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.396+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 shutdown_connections 2023-10-10T18:45:10.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.396+0000 7f478cf48700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4774066ed0 0x7f4774069380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:10.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.396+0000 7f478cf48700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4788157180 0x7f47880cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:10.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.396+0000 7f478cf48700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47880c6180 0x7f4788166580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:10.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.396+0000 7f478cf48700 1 --2- 172.21.15.175:0/306510469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47880c6550 0x7f47881578c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:10.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.396+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 >> 172.21.15.175:0/306510469 conn(0x7f4788063230 msgr2=0x7f478814fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:10.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.397+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 shutdown_connections 2023-10-10T18:45:10.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.398+0000 7f478cf48700 1 -- 172.21.15.175:0/306510469 wait complete. 2023-10-10T18:45:10.402 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'cephfs_data' created 2023-10-10T18:45:10.419 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-10T18:45:10.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.535+0000 7f39e61c0700 1 Processor -- start 2023-10-10T18:45:10.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.535+0000 7f39e61c0700 1 -- start start 2023-10-10T18:45:10.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39e61c0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f39e00633f0 0x7f39e0156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39e61c0700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e00c7ca0 con 0x7f39e00c7620 2023-10-10T18:45:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39e61c0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e0156830 con 0x7f39e00637c0 2023-10-10T18:45:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39e61c0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e00af540 con 0x7f39e00633f0 2023-10-10T18:45:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39df7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f39e00633f0 0x7f39e0156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39df7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f39e00633f0 0x7f39e0156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50732/0 (socket says 172.21.15.175:50732) 2023-10-10T18:45:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39df7fe700 1 -- 172.21.15.175:0/474450101 learned_addr learned my addr 172.21.15.175:0/474450101 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39df7fe700 1 -- 172.21.15.175:0/474450101 >> v1:172.21.15.191:6789/0 conn(0x7f39e00637c0 legacy=0x7f39e014f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39df7fe700 1 -- 172.21.15.175:0/474450101 >> v1:172.21.15.175:6789/0 conn(0x7f39e00c7620 legacy=0x7f39e00aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.536+0000 7f39df7fe700 1 -- 172.21.15.175:0/474450101 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0156ad0 con 0x7f39e00633f0 2023-10-10T18:45:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39df7fe700 1 --2- 172.21.15.175:0/474450101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e0156010 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009f20 tx=0x7f39d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f95d82d6f2dc3c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39de7fc700 1 -- 172.21.15.175:0/474450101 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d000e070 con 0x7f39e00633f0 2023-10-10T18:45:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39de7fc700 1 -- 172.21.15.175:0/474450101 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d000bb10 con 0x7f39e00633f0 2023-10-10T18:45:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39de7fc700 1 -- 172.21.15.175:0/474450101 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d00092e0 con 0x7f39e00633f0 2023-10-10T18:45:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39e61c0700 1 -- 172.21.15.175:0/474450101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 msgr2=0x7f39e0156010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39e61c0700 1 --2- 172.21.15.175:0/474450101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e0156010 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009f20 tx=0x7f39d0009840 comp rx=0 tx=0).stop 2023-10-10T18:45:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39e61c0700 1 -- 172.21.15.175:0/474450101 shutdown_connections 2023-10-10T18:45:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39e61c0700 1 --2- 172.21.15.175:0/474450101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e0156010 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39e61c0700 1 -- 172.21.15.175:0/474450101 >> 172.21.15.175:0/474450101 conn(0x7f39e00cc7a0 msgr2=0x7f39e00ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39e61c0700 1 -- 172.21.15.175:0/474450101 shutdown_connections 2023-10-10T18:45:10.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.537+0000 7f39e61c0700 1 -- 172.21.15.175:0/474450101 wait complete. 2023-10-10T18:45:10.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.538+0000 7f39e61c0700 1 Processor -- start 2023-10-10T18:45:10.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.538+0000 7f39e61c0700 1 -- start start 2023-10-10T18:45:10.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39e61c0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e016a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:10.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39e61c0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39e00637c0 0x7f39e00c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:10.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39e61c0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39e00c7620 0x7f39e00c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:10.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39e61c0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0151470 con 0x7f39e00633f0 2023-10-10T18:45:10.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39e61c0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0169d40 con 0x7f39e00c7620 2023-10-10T18:45:10.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39e61c0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0169a10 con 0x7f39e00637c0 2023-10-10T18:45:10.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39df7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e016a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:10.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39df7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e016a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50738/0 (socket says 172.21.15.175:50738) 2023-10-10T18:45:10.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39df7fe700 1 -- 172.21.15.175:0/1403476011 learned_addr learned my addr 172.21.15.175:0/1403476011 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:10.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39dffff700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39e00c7620 0x7f39e00c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:10.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39deffd700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39e00637c0 0x7f39e00c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:10.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39df7fe700 1 -- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39e00637c0 msgr2=0x7f39e00c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:10.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.539+0000 7f39df7fe700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39e00637c0 0x7f39e00c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:10.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39df7fe700 1 -- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39e00c7620 msgr2=0x7f39e00c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:10.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39df7fe700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39e00c7620 0x7f39e00c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:10.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39df7fe700 1 -- 172.21.15.175:0/1403476011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d0009ae0 con 0x7f39e00633f0 2023-10-10T18:45:10.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39dffff700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39e00c7620 0x7f39e00c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:10.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39df7fe700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e016a1d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39e008a0c0 tx=0x7f39d0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:10.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39c77fe700 1 -- 172.21.15.175:0/1403476011 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0008ca0 con 0x7f39e00633f0 2023-10-10T18:45:10.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39c77fe700 1 -- 172.21.15.175:0/1403476011 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d0008e00 con 0x7f39e00633f0 2023-10-10T18:45:10.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e00c3dd0 con 0x7f39e00633f0 2023-10-10T18:45:10.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39c77fe700 1 -- 172.21.15.175:0/1403476011 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d00266b0 con 0x7f39e00633f0 2023-10-10T18:45:10.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.540+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e0150c70 con 0x7f39e00633f0 2023-10-10T18:45:10.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.541+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e016a020 con 0x7f39e00633f0 2023-10-10T18:45:10.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.541+0000 7f39c77fe700 1 -- 172.21.15.175:0/1403476011 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f39d0003040 con 0x7f39e00633f0 2023-10-10T18:45:10.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.541+0000 7f39c77fe700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f39cc066f60 0x7f39cc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:10.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.542+0000 7f39c77fe700 1 -- 172.21.15.175:0/1403476011 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5605+0+0 (secure 0 0 0) 0x7f39d0086430 con 0x7f39e00633f0 2023-10-10T18:45:10.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.542+0000 7f39deffd700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f39cc066f60 0x7f39cc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:10.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.542+0000 7f39deffd700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f39cc066f60 0x7f39cc069410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39e00c69b0 tx=0x7f39d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:10.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.546+0000 7f39c77fe700 1 -- 172.21.15.175:0/1403476011 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39d0053700 con 0x7f39e00633f0 2023-10-10T18:45:10.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:10.772+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f39e0067110 con 0x7f39e00633f0 2023-10-10T18:45:11.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.413+0000 7f39c77fe700 1 -- 172.21.15.175:0/1403476011 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f39d0056d20 con 0x7f39e00633f0 2023-10-10T18:45:11.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f39cc066f60 msgr2=0x7f39cc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f39cc066f60 0x7f39cc069410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39e00c69b0 tx=0x7f39d4009040 comp rx=0 tx=0).stop 2023-10-10T18:45:11.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 msgr2=0x7f39e016a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e016a1d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39e008a0c0 tx=0x7f39d0018000 comp rx=0 tx=0).stop 2023-10-10T18:45:11.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 shutdown_connections 2023-10-10T18:45:11.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f39cc066f60 0x7f39cc069410 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39e00c7620 0x7f39e00c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39e00633f0 0x7f39e016a1d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 --2- 172.21.15.175:0/1403476011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39e00637c0 0x7f39e00c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 >> 172.21.15.175:0/1403476011 conn(0x7f39e00cc7a0 msgr2=0x7f39e00cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:11.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.418+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 shutdown_connections 2023-10-10T18:45:11.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.419+0000 7f39e61c0700 1 -- 172.21.15.175:0/1403476011 wait complete. 2023-10-10T18:45:11.422 INFO:teuthology.orchestra.run.smithi175.stderr:new fs with metadata pool 2 and data pool 3 2023-10-10T18:45:11.436 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:11.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.551+0000 7fc76eb39700 1 Processor -- start 2023-10-10T18:45:11.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.551+0000 7fc76eb39700 1 -- start start 2023-10-10T18:45:11.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76eb39700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc76814fd90 0x7fc768159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76eb39700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7680cd040 0x7fc7680684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76eb39700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc768067fa0 con 0x7fc76808a100 2023-10-10T18:45:11.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76eb39700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7680c5600 con 0x7fc76814fd90 2023-10-10T18:45:11.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76eb39700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc768156b70 con 0x7fc7680cd040 2023-10-10T18:45:11.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76db37700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc76814fd90 0x7fc768159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76db37700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc76814fd90 0x7fc768159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50752/0 (socket says 172.21.15.175:50752) 2023-10-10T18:45:11.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76db37700 1 -- 172.21.15.175:0/2663037791 learned_addr learned my addr 172.21.15.175:0/2663037791 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:11.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76e338700 1 --2- 172.21.15.175:0/2663037791 >> v2:172.21.15.191:3300/0 conn(0x7fc7680cd040 0x7fc7680684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76db37700 1 -- 172.21.15.175:0/2663037791 >> v2:172.21.15.191:3300/0 conn(0x7fc7680cd040 msgr2=0x7fc7680684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:11.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76db37700 1 --2- 172.21.15.175:0/2663037791 >> v2:172.21.15.191:3300/0 conn(0x7fc7680cd040 0x7fc7680684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76db37700 1 -- 172.21.15.175:0/2663037791 >> v1:172.21.15.191:6789/0 conn(0x7fc76808a100 legacy=0x7fc7680c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76db37700 1 -- 172.21.15.175:0/2663037791 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768150160 con 0x7fc76814fd90 2023-10-10T18:45:11.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76db37700 1 --2- 172.21.15.175:0/2663037791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768159c30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc76808a6e0 tx=0x7fc764009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3fcc7c4618a558 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:11.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76cb35700 1 -- 172.21.15.175:0/2663037791 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764009e70 con 0x7fc76814fd90 2023-10-10T18:45:11.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76cb35700 1 -- 172.21.15.175:0/2663037791 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76400b8f0 con 0x7fc76814fd90 2023-10-10T18:45:11.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.552+0000 7fc76cb35700 1 -- 172.21.15.175:0/2663037791 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76400bc90 con 0x7fc76814fd90 2023-10-10T18:45:11.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.553+0000 7fc76eb39700 1 -- 172.21.15.175:0/2663037791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 msgr2=0x7fc768159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.553+0000 7fc76eb39700 1 --2- 172.21.15.175:0/2663037791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768159c30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc76808a6e0 tx=0x7fc764009250 comp rx=0 tx=0).stop 2023-10-10T18:45:11.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.553+0000 7fc76eb39700 1 -- 172.21.15.175:0/2663037791 shutdown_connections 2023-10-10T18:45:11.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.553+0000 7fc76eb39700 1 --2- 172.21.15.175:0/2663037791 >> v2:172.21.15.191:3300/0 conn(0x7fc7680cd040 0x7fc7680684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.553+0000 7fc76eb39700 1 --2- 172.21.15.175:0/2663037791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768159c30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.553+0000 7fc76eb39700 1 -- 172.21.15.175:0/2663037791 >> 172.21.15.175:0/2663037791 conn(0x7fc768063d00 msgr2=0x7fc7680cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:11.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.553+0000 7fc76eb39700 1 -- 172.21.15.175:0/2663037791 shutdown_connections 2023-10-10T18:45:11.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.553+0000 7fc76eb39700 1 -- 172.21.15.175:0/2663037791 wait complete. 2023-10-10T18:45:11.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.554+0000 7fc76eb39700 1 Processor -- start 2023-10-10T18:45:11.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.554+0000 7fc76eb39700 1 -- start start 2023-10-10T18:45:11.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76eb39700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc76808a100 0x7fc7680cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76eb39700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7680cd040 0x7fc768147920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76eb39700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76eb39700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76814f020 con 0x7fc76814fd90 2023-10-10T18:45:11.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76eb39700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76814d920 con 0x7fc76808a100 2023-10-10T18:45:11.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76eb39700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76814d5f0 con 0x7fc7680cd040 2023-10-10T18:45:11.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768147e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50768/0 (socket says 172.21.15.175:50768) 2023-10-10T18:45:11.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 -- 172.21.15.175:0/3762444457 learned_addr learned my addr 172.21.15.175:0/3762444457 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:11.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76db37700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc76808a100 0x7fc7680cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76d336700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7680cd040 0x7fc768147920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 -- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7680cd040 msgr2=0x7fc768147920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7680cd040 0x7fc768147920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 -- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc76808a100 msgr2=0x7fc7680cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc76808a100 0x7fc7680cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 -- 172.21.15.175:0/3762444457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640094f0 con 0x7fc76814fd90 2023-10-10T18:45:11.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76db37700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc76808a100 0x7fc7680cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:11.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.555+0000 7fc76e338700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768147e60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc760009ec0 tx=0x7fc76000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:11.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.556+0000 7fc75dffb700 1 -- 172.21.15.175:0/3762444457 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76000cb70 con 0x7fc76814fd90 2023-10-10T18:45:11.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.556+0000 7fc75dffb700 1 -- 172.21.15.175:0/3762444457 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76000f040 con 0x7fc76814fd90 2023-10-10T18:45:11.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.556+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7680c2f90 con 0x7fc76814fd90 2023-10-10T18:45:11.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.556+0000 7fc75dffb700 1 -- 172.21.15.175:0/3762444457 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760014480 con 0x7fc76814fd90 2023-10-10T18:45:11.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.556+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76814f270 con 0x7fc76814fd90 2023-10-10T18:45:11.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.557+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc768067570 con 0x7fc76814fd90 2023-10-10T18:45:11.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.557+0000 7fc75dffb700 1 -- 172.21.15.175:0/3762444457 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc760014830 con 0x7fc76814fd90 2023-10-10T18:45:11.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.557+0000 7fc75dffb700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc74c066f30 0x7fc74c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.557+0000 7fc75dffb700 1 -- 172.21.15.175:0/3762444457 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc7600850d0 con 0x7fc76814fd90 2023-10-10T18:45:11.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.558+0000 7fc76db37700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc74c066f30 0x7fc74c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.558+0000 7fc76db37700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc74c066f30 0x7fc74c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc76808a6e0 tx=0x7fc764017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:11.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.562+0000 7fc75dffb700 1 -- 172.21.15.175:0/3762444457 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc760052340 con 0x7fc76814fd90 2023-10-10T18:45:11.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.748+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc76814da60 con 0x7fc76814fd90 2023-10-10T18:45:11.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.749+0000 7fc75dffb700 1 -- 172.21.15.175:0/3762444457 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15411 (secure 0 0 0) 0x7fc760055960 con 0x7fc76814fd90 2023-10-10T18:45:11.750 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:11.750 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":20,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:10.774515+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:11.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc74c066f30 msgr2=0x7fc74c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc74c066f30 0x7fc74c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc76808a6e0 tx=0x7fc764017000 comp rx=0 tx=0).stop 2023-10-10T18:45:11.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 msgr2=0x7fc768147e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768147e60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc760009ec0 tx=0x7fc76000b040 comp rx=0 tx=0).stop 2023-10-10T18:45:11.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 shutdown_connections 2023-10-10T18:45:11.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc74c066f30 0x7fc74c0693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc76814fd90 0x7fc768147e60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc76808a100 0x7fc7680cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 --2- 172.21.15.175:0/3762444457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7680cd040 0x7fc768147920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 >> 172.21.15.175:0/3762444457 conn(0x7fc768063d00 msgr2=0x7fc7680c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:11.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 shutdown_connections 2023-10-10T18:45:11.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.751+0000 7fc76eb39700 1 -- 172.21.15.175:0/3762444457 wait complete. 2023-10-10T18:45:11.764 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:11.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.876+0000 7f381be90700 1 Processor -- start 2023-10-10T18:45:11.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.876+0000 7f381be90700 1 -- start start 2023-10-10T18:45:11.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.877+0000 7f381be90700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f381414e140 0x7f3814155270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.877+0000 7f381be90700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38140c1210 0x7f38140bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.877+0000 7f381be90700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381406b740 con 0x7f381406edb0 2023-10-10T18:45:11.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.877+0000 7f381be90700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38140c0960 con 0x7f381414e140 2023-10-10T18:45:11.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.877+0000 7f381be90700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381406ca30 con 0x7f38140c1210 2023-10-10T18:45:11.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.877+0000 7f381a42d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f381406edb0 0x7f3814153df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56422/0 (socket says 172.21.15.175:56422) 2023-10-10T18:45:11.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.877+0000 7f381a42d700 1 -- 172.21.15.175:0/2290838072 learned_addr learned my addr 172.21.15.175:0/2290838072 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:11.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3819c2c700 1 --2- 172.21.15.175:0/2290838072 >> v2:172.21.15.191:3300/0 conn(0x7f381414e140 0x7f3814155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381942b700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38140c1210 0x7f38140bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798223388 0 0) 0x7f381406b740 con 0x7f381406edb0 2023-10-10T18:45:11.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3800003570 con 0x7f381406edb0 2023-10-10T18:45:11.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1156343616 0 0) 0x7f3800003570 con 0x7f381406edb0 2023-10-10T18:45:11.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38140c1210 msgr2=0x7f38140bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:11.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38140c1210 0x7f38140bf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 >> v2:172.21.15.191:3300/0 conn(0x7f381414e140 msgr2=0x7f3814155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:11.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 --2- 172.21.15.175:0/2290838072 >> v2:172.21.15.191:3300/0 conn(0x7f381414e140 0x7f3814155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381406c5a0 con 0x7f381406edb0 2023-10-10T18:45:11.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f380c0017f0 con 0x7f381406edb0 2023-10-10T18:45:11.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 >> v1:172.21.15.175:6789/0 conn(0x7f381406edb0 legacy=0x7f3814153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800003a90 0x7f3800003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38000043e0 0x7f38000048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3800004f20 0x7f380000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38140c0960 con 0x7f3800004f20 2023-10-10T18:45:11.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381406ca30 con 0x7f38000043e0 2023-10-10T18:45:11.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380000bfa0 con 0x7f3800003a90 2023-10-10T18:45:11.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381a42d700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3800004f20 0x7f380000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381942b700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38000043e0 0x7f38000048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f3819c2c700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800003a90 0x7f3800003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381a42d700 1 -- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800003a90 msgr2=0x7f3800003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:11.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381a42d700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800003a90 0x7f3800003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381a42d700 1 -- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38000043e0 msgr2=0x7f38000048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:11.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381a42d700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38000043e0 0x7f38000048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381a42d700 1 -- 172.21.15.175:0/2290838072 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3800003570 con 0x7f3800004f20 2023-10-10T18:45:11.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.878+0000 7f381a42d700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3800004f20 0x7f380000ba60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f380c00a900 tx=0x7f380c0021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9c3559e87840e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:11.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c0070d0 con 0x7f3800004f20 2023-10-10T18:45:11.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c007230 con 0x7f3800004f20 2023-10-10T18:45:11.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f3818c2a700 1 -- 172.21.15.175:0/2290838072 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c0075d0 con 0x7f3800004f20 2023-10-10T18:45:11.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f381be90700 1 -- 172.21.15.175:0/2290838072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3800004f20 msgr2=0x7f380000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f381be90700 1 --2- 172.21.15.175:0/2290838072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3800004f20 0x7f380000ba60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f380c00a900 tx=0x7f380c0021d0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f381a42d700 1 -- 172.21.15.175:0/2290838072 reap_dead start 2023-10-10T18:45:11.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f381be90700 1 -- 172.21.15.175:0/2290838072 shutdown_connections 2023-10-10T18:45:11.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f381be90700 1 -- 172.21.15.175:0/2290838072 >> 172.21.15.175:0/2290838072 conn(0x7f3814063cb0 msgr2=0x7f381406d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:11.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f381be90700 1 -- 172.21.15.175:0/2290838072 shutdown_connections 2023-10-10T18:45:11.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.879+0000 7f381be90700 1 -- 172.21.15.175:0/2290838072 wait complete. 2023-10-10T18:45:11.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.880+0000 7f381be90700 1 Processor -- start 2023-10-10T18:45:11.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.880+0000 7f381be90700 1 -- start start 2023-10-10T18:45:11.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381be90700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800004f20 0x7f38140c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381be90700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38141470b0 0x7f3814147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381be90700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38141474b0 0x7f38140c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381be90700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3814163ea0 con 0x7f38141470b0 2023-10-10T18:45:11.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381be90700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38140c5880 con 0x7f38141474b0 2023-10-10T18:45:11.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381be90700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3814149020 con 0x7f3800004f20 2023-10-10T18:45:11.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38141470b0 0x7f3814147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38141470b0 0x7f3814147fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50774/0 (socket says 172.21.15.175:50774) 2023-10-10T18:45:11.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 -- 172.21.15.175:0/2579066601 learned_addr learned my addr 172.21.15.175:0/2579066601 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:11.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f3819c2c700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800004f20 0x7f38140c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381a42d700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38141474b0 0x7f38140c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 -- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800004f20 msgr2=0x7f38140c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800004f20 0x7f38140c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 -- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38141474b0 msgr2=0x7f38140c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:11.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38141474b0 0x7f38140c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:11.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 -- 172.21.15.175:0/2579066601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381415a430 con 0x7f38141470b0 2023-10-10T18:45:11.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381a42d700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38141474b0 0x7f38140c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:11.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.881+0000 7f381942b700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38141470b0 0x7f3814147fe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3804006ac0 tx=0x7f38040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:11.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.882+0000 7f3809ffb700 1 -- 172.21.15.175:0/2579066601 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380400f940 con 0x7f38141470b0 2023-10-10T18:45:11.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.882+0000 7f3809ffb700 1 -- 172.21.15.175:0/2579066601 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380400faa0 con 0x7f38141470b0 2023-10-10T18:45:11.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.882+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3814163bc0 con 0x7f38141470b0 2023-10-10T18:45:11.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.882+0000 7f3809ffb700 1 -- 172.21.15.175:0/2579066601 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380400fe20 con 0x7f38141470b0 2023-10-10T18:45:11.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.882+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38140c63b0 con 0x7f38141470b0 2023-10-10T18:45:11.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.883+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38140c89b0 con 0x7f38141470b0 2023-10-10T18:45:11.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.883+0000 7f3809ffb700 1 -- 172.21.15.175:0/2579066601 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3804014650 con 0x7f38141470b0 2023-10-10T18:45:11.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.883+0000 7f3809ffb700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f8066f30 0x7f37f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:11.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.884+0000 7f3809ffb700 1 -- 172.21.15.175:0/2579066601 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3804018030 con 0x7f38141470b0 2023-10-10T18:45:11.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.884+0000 7f3819c2c700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f8066f30 0x7f37f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:11.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.884+0000 7f3819c2c700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f8066f30 0x7f37f80693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3810006f00 tx=0x7f38100091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:11.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:11.888+0000 7f3809ffb700 1 -- 172.21.15.175:0/2579066601 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3804052240 con 0x7f38141470b0 2023-10-10T18:45:12.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.071+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3814067570 con 0x7f38141470b0 2023-10-10T18:45:12.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.072+0000 7f3809ffb700 1 -- 172.21.15.175:0/2579066601 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15411 (secure 0 0 0) 0x7f3804022090 con 0x7f38141470b0 2023-10-10T18:45:12.073 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:12.073 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":20,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:10.774515+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:12.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f8066f30 msgr2=0x7f37f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:12.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f8066f30 0x7f37f80693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3810006f00 tx=0x7f38100091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38141470b0 msgr2=0x7f3814147fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:12.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38141470b0 0x7f3814147fe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3804006ac0 tx=0x7f38040091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 shutdown_connections 2023-10-10T18:45:12.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f8066f30 0x7f37f80693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38141474b0 0x7f38140c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3800004f20 0x7f38140c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 --2- 172.21.15.175:0/2579066601 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38141470b0 0x7f3814147fe0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 >> 172.21.15.175:0/2579066601 conn(0x7f3814063cb0 msgr2=0x7f381414d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:12.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 shutdown_connections 2023-10-10T18:45:12.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.074+0000 7f381be90700 1 -- 172.21.15.175:0/2579066601 wait complete. 2023-10-10T18:45:12.087 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-10T18:45:12.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.203+0000 7fe145a63700 1 Processor -- start 2023-10-10T18:45:12.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.203+0000 7fe145a63700 1 -- start start 2023-10-10T18:45:12.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe145a63700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe1400633f0 0x7fe140156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:12.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe145a63700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1400c7ca0 con 0x7fe1400637c0 2023-10-10T18:45:12.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe145a63700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe140156830 con 0x7fe1400c7620 2023-10-10T18:45:12.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe145a63700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1400af540 con 0x7fe1400633f0 2023-10-10T18:45:12.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe13f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe1400633f0 0x7fe140156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:12.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe13f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe1400633f0 0x7fe140156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50778/0 (socket says 172.21.15.175:50778) 2023-10-10T18:45:12.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe13f7fe700 1 -- 172.21.15.175:0/1509686332 learned_addr learned my addr 172.21.15.175:0/1509686332 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:12.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe13ffff700 1 --1- 172.21.15.175:0/1509686332 >> v1:172.21.15.175:6789/0 conn(0x7fe1400637c0 0x7fe14014f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56438/0 (socket says 172.21.15.175:56438) 2023-10-10T18:45:12.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe13f7fe700 1 -- 172.21.15.175:0/1509686332 >> v1:172.21.15.191:6789/0 conn(0x7fe1400c7620 legacy=0x7fe1400aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:12.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe13f7fe700 1 -- 172.21.15.175:0/1509686332 >> v1:172.21.15.175:6789/0 conn(0x7fe1400637c0 legacy=0x7fe14014f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:12.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe13f7fe700 1 -- 172.21.15.175:0/1509686332 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140156ad0 con 0x7fe1400633f0 2023-10-10T18:45:12.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.204+0000 7fe13f7fe700 1 --2- 172.21.15.175:0/1509686332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400633f0 0x7fe140156010 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe130009f20 tx=0x7fe130009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b7362687abb28e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:12.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe13e7fc700 1 -- 172.21.15.175:0/1509686332 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13000e070 con 0x7fe1400633f0 2023-10-10T18:45:12.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe13e7fc700 1 -- 172.21.15.175:0/1509686332 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13000bb10 con 0x7fe1400633f0 2023-10-10T18:45:12.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe13e7fc700 1 -- 172.21.15.175:0/1509686332 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1300092e0 con 0x7fe1400633f0 2023-10-10T18:45:12.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe145a63700 1 -- 172.21.15.175:0/1509686332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400633f0 msgr2=0x7fe140156010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:12.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe145a63700 1 --2- 172.21.15.175:0/1509686332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400633f0 0x7fe140156010 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe130009f20 tx=0x7fe130009840 comp rx=0 tx=0).stop 2023-10-10T18:45:12.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe145a63700 1 -- 172.21.15.175:0/1509686332 shutdown_connections 2023-10-10T18:45:12.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe145a63700 1 --2- 172.21.15.175:0/1509686332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400633f0 0x7fe140156010 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe145a63700 1 -- 172.21.15.175:0/1509686332 >> 172.21.15.175:0/1509686332 conn(0x7fe1400cc7a0 msgr2=0x7fe1400ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:12.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe145a63700 1 -- 172.21.15.175:0/1509686332 shutdown_connections 2023-10-10T18:45:12.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.205+0000 7fe145a63700 1 -- 172.21.15.175:0/1509686332 wait complete. 2023-10-10T18:45:12.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.206+0000 7fe145a63700 1 Processor -- start 2023-10-10T18:45:12.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.206+0000 7fe145a63700 1 -- start start 2023-10-10T18:45:12.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe145a63700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1400633f0 0x7fe1400c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:12.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe145a63700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1400637c0 0x7fe14014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:12.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe145a63700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400c7620 0x7fe140165d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:12.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe145a63700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1400c3cf0 con 0x7fe1400c7620 2023-10-10T18:45:12.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe145a63700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe140153780 con 0x7fe1400633f0 2023-10-10T18:45:12.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe145a63700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1401531c0 con 0x7fe1400637c0 2023-10-10T18:45:12.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe13ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400c7620 0x7fe140165d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:12.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe13ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400c7620 0x7fe140165d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50780/0 (socket says 172.21.15.175:50780) 2023-10-10T18:45:12.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe13ffff700 1 -- 172.21.15.175:0/3802476219 learned_addr learned my addr 172.21.15.175:0/3802476219 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:12.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe13f7fe700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1400633f0 0x7fe1400c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.207+0000 7fe13effd700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1400637c0 0x7fe14014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe13ffff700 1 -- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1400637c0 msgr2=0x7fe14014e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe13ffff700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1400637c0 0x7fe14014e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe13ffff700 1 -- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1400633f0 msgr2=0x7fe1400c37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe13ffff700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1400633f0 0x7fe1400c37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:12.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe13ffff700 1 -- 172.21.15.175:0/3802476219 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe130009ae0 con 0x7fe1400c7620 2023-10-10T18:45:12.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe13ffff700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400c7620 0x7fe140165d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe13800c590 tx=0x7fe138007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:12.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe1277fe700 1 -- 172.21.15.175:0/3802476219 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138005a70 con 0x7fe1400c7620 2023-10-10T18:45:12.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe1277fe700 1 -- 172.21.15.175:0/3802476219 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe138005bf0 con 0x7fe1400c7620 2023-10-10T18:45:12.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1401534a0 con 0x7fe1400c7620 2023-10-10T18:45:12.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe1277fe700 1 -- 172.21.15.175:0/3802476219 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1380070d0 con 0x7fe1400c7620 2023-10-10T18:45:12.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.208+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe140158c50 con 0x7fe1400c7620 2023-10-10T18:45:12.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.209+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe140067110 con 0x7fe1400c7620 2023-10-10T18:45:12.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.209+0000 7fe1277fe700 1 -- 172.21.15.175:0/3802476219 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe138007480 con 0x7fe1400c7620 2023-10-10T18:45:12.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.210+0000 7fe1277fe700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe12c066f60 0x7fe12c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:12.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.210+0000 7fe1277fe700 1 -- 172.21.15.175:0/3802476219 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe138083860 con 0x7fe1400c7620 2023-10-10T18:45:12.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.210+0000 7fe13f7fe700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe12c066f60 0x7fe12c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:12.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.210+0000 7fe13f7fe700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe12c066f60 0x7fe12c069410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe130009880 tx=0x7fe130018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:12.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.214+0000 7fe1277fe700 1 -- 172.21.15.175:0/3802476219 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe138050ad0 con 0x7fe1400c7620 2023-10-10T18:45:12.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:12.439+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe140067570 con 0x7fe1400c7620 2023-10-10T18:45:13.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.426+0000 7fe1277fe700 1 -- 172.21.15.175:0/3802476219 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe138017070 con 0x7fe1400c7620 2023-10-10T18:45:13.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe12c066f60 msgr2=0x7fe12c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:13.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe12c066f60 0x7fe12c069410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe130009880 tx=0x7fe130018000 comp rx=0 tx=0).stop 2023-10-10T18:45:13.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400c7620 msgr2=0x7fe140165d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400c7620 0x7fe140165d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe13800c590 tx=0x7fe138007ef0 comp rx=0 tx=0).stop 2023-10-10T18:45:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 shutdown_connections 2023-10-10T18:45:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe12c066f60 0x7fe12c069410 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1400c7620 0x7fe140165d00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1400633f0 0x7fe1400c37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 --2- 172.21.15.175:0/3802476219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1400637c0 0x7fe14014e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 >> 172.21.15.175:0/3802476219 conn(0x7fe1400cc7a0 msgr2=0x7fe140155dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 shutdown_connections 2023-10-10T18:45:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.429+0000 7fe145a63700 1 -- 172.21.15.175:0/3802476219 wait complete. 2023-10-10T18:45:13.445 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-10-10T18:45:13.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.570+0000 7f954aafb700 1 Processor -- start 2023-10-10T18:45:13.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.570+0000 7f954aafb700 1 -- start start 2023-10-10T18:45:13.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f954aafb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544154760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:13.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f954aafb700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95440c4550 con 0x7f95440c75b0 2023-10-10T18:45:13.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f954aafb700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9544063e80 con 0x7f954406b280 2023-10-10T18:45:13.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f954aafb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95440c49e0 con 0x7f95440c71e0 2023-10-10T18:45:13.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f953bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f95440c75b0 0x7f95440c1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56446/0 (socket says 172.21.15.175:56446) 2023-10-10T18:45:13.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f953bfff700 1 -- 172.21.15.175:0/230284971 learned_addr learned my addr 172.21.15.175:0/230284971 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:13.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f9548897700 1 --2- 172.21.15.175:0/230284971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:13.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f953b7fe700 1 -- 172.21.15.175:0/230284971 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602792598 0 0) 0x7f95440c4550 con 0x7f95440c75b0 2023-10-10T18:45:13.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f953b7fe700 1 -- 172.21.15.175:0/230284971 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f952c003570 con 0x7f95440c75b0 2023-10-10T18:45:13.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f953b7fe700 1 -- 172.21.15.175:0/230284971 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061567994 0 0) 0x7f9544063e80 con 0x7f954406b280 2023-10-10T18:45:13.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f953b7fe700 1 -- 172.21.15.175:0/230284971 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95440c4550 con 0x7f954406b280 2023-10-10T18:45:13.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f9548897700 1 -- 172.21.15.175:0/230284971 >> v1:172.21.15.191:6789/0 conn(0x7f954406b280 legacy=0x7f954414db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:13.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f953b7fe700 1 -- 172.21.15.175:0/230284971 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2493665826 0 0) 0x7f952c003570 con 0x7f95440c75b0 2023-10-10T18:45:13.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f9548897700 1 -- 172.21.15.175:0/230284971 >> v1:172.21.15.175:6789/0 conn(0x7f95440c75b0 legacy=0x7f95440c1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:13.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f9548897700 1 -- 172.21.15.175:0/230284971 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95440c3940 con 0x7f95440c71e0 2023-10-10T18:45:13.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.571+0000 7f9548897700 1 --2- 172.21.15.175:0/230284971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544154760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9540009e90 tx=0x7f9540009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e649a71bc1d11ecd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:13.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f953b7fe700 1 -- 172.21.15.175:0/230284971 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954000e070 con 0x7f95440c71e0 2023-10-10T18:45:13.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f953b7fe700 1 -- 172.21.15.175:0/230284971 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954000b870 con 0x7f95440c71e0 2023-10-10T18:45:13.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f953b7fe700 1 -- 172.21.15.175:0/230284971 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9534001470 con 0x7f95440c71e0 2023-10-10T18:45:13.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f954aafb700 1 -- 172.21.15.175:0/230284971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 msgr2=0x7f9544154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:13.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f954aafb700 1 --2- 172.21.15.175:0/230284971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544154760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9540009e90 tx=0x7f9540009780 comp rx=0 tx=0).stop 2023-10-10T18:45:13.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f954aafb700 1 -- 172.21.15.175:0/230284971 shutdown_connections 2023-10-10T18:45:13.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f954aafb700 1 --2- 172.21.15.175:0/230284971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544154760 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:13.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f954aafb700 1 -- 172.21.15.175:0/230284971 >> 172.21.15.175:0/230284971 conn(0x7f954406dd00 msgr2=0x7f9544063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:13.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f954aafb700 1 -- 172.21.15.175:0/230284971 shutdown_connections 2023-10-10T18:45:13.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.572+0000 7f954aafb700 1 -- 172.21.15.175:0/230284971 wait complete. 2023-10-10T18:45:13.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.573+0000 7f954aafb700 1 Processor -- start 2023-10-10T18:45:13.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.573+0000 7f954aafb700 1 -- start start 2023-10-10T18:45:13.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f954aafb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f954406b280 0x7f95441413c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:13.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f954aafb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:13.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f954aafb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95440c75b0 0x7f9544152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:13.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f954aafb700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9544153ea0 con 0x7f95440c75b0 2023-10-10T18:45:13.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f954aafb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9544153ad0 con 0x7f954406b280 2023-10-10T18:45:13.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f954aafb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95441511a0 con 0x7f95440c71e0 2023-10-10T18:45:13.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f9549098700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95440c75b0 0x7f9544152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:13.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f9549098700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95440c75b0 0x7f9544152f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50794/0 (socket says 172.21.15.175:50794) 2023-10-10T18:45:13.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f9549098700 1 -- 172.21.15.175:0/1332745712 learned_addr learned my addr 172.21.15.175:0/1332745712 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:13.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f9548897700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f954406b280 0x7f95441413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:13.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f953bfff700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:13.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f9549098700 1 -- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 msgr2=0x7f9544152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:13.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f9549098700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:13.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.574+0000 7f9549098700 1 -- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f954406b280 msgr2=0x7f95441413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:13.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.575+0000 7f9549098700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f954406b280 0x7f95441413c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:13.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.575+0000 7f9549098700 1 -- 172.21.15.175:0/1332745712 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9540009a20 con 0x7f95440c75b0 2023-10-10T18:45:13.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.575+0000 7f9549098700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95440c75b0 0x7f9544152f80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f953c00c5c0 tx=0x7f953c007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:13.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.575+0000 7f9538ff9700 1 -- 172.21.15.175:0/1332745712 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c016070 con 0x7f95440c75b0 2023-10-10T18:45:13.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.575+0000 7f9538ff9700 1 -- 172.21.15.175:0/1332745712 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c011590 con 0x7f95440c75b0 2023-10-10T18:45:13.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.575+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9544163830 con 0x7f95440c75b0 2023-10-10T18:45:13.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.575+0000 7f9538ff9700 1 -- 172.21.15.175:0/1332745712 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c011930 con 0x7f95440c75b0 2023-10-10T18:45:13.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.575+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9544150be0 con 0x7f95440c75b0 2023-10-10T18:45:13.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.576+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95441639c0 con 0x7f95440c75b0 2023-10-10T18:45:13.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.576+0000 7f9538ff9700 1 -- 172.21.15.175:0/1332745712 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f953c012420 con 0x7f95440c75b0 2023-10-10T18:45:13.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.576+0000 7f9538ff9700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9530066f30 0x7f95300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:13.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.576+0000 7f9538ff9700 1 -- 172.21.15.175:0/1332745712 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f953c083fe0 con 0x7f95440c75b0 2023-10-10T18:45:13.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.577+0000 7f9548897700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9530066f30 0x7f95300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:13.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.577+0000 7f9548897700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9530066f30 0x7f95300693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f95400097c0 tx=0x7f9540017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:13.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.581+0000 7f9538ff9700 1 -- 172.21.15.175:0/1332745712 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f953c051250 con 0x7f95440c75b0 2023-10-10T18:45:13.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:13.806+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f9544067570 con 0x7f95440c75b0 2023-10-10T18:45:14.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.428+0000 7f9538ff9700 1 -- 172.21.15.175:0/1332745712 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f953c054870 con 0x7f95440c75b0 2023-10-10T18:45:14.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9530066f30 msgr2=0x7f95300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:14.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9530066f30 0x7f95300693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f95400097c0 tx=0x7f9540017000 comp rx=0 tx=0).stop 2023-10-10T18:45:14.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95440c75b0 msgr2=0x7f9544152f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:14.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95440c75b0 0x7f9544152f80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f953c00c5c0 tx=0x7f953c007d80 comp rx=0 tx=0).stop 2023-10-10T18:45:14.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 shutdown_connections 2023-10-10T18:45:14.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9530066f30 0x7f95300693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95440c75b0 0x7f9544152f80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f954406b280 0x7f95441413c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 --2- 172.21.15.175:0/1332745712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95440c71e0 0x7f9544152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 >> 172.21.15.175:0/1332745712 conn(0x7f954406dd00 msgr2=0x7f954414bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:14.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 shutdown_connections 2023-10-10T18:45:14.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.430+0000 7f954aafb700 1 -- 172.21.15.175:0/1332745712 wait complete. 2023-10-10T18:45:14.443 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-10T18:45:14.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.560+0000 7f93e2143700 1 Processor -- start 2023-10-10T18:45:14.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.560+0000 7f93e2143700 1 -- start start 2023-10-10T18:45:14.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e2143700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f93dc06d1e0 0x7f93dc155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:14.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e2143700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f93dc0c4d50 0x7f93dc153430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:14.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e2143700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc159350 con 0x7f93dc0c7ae0 2023-10-10T18:45:14.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e2143700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc159bf0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e2143700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc0c6040 con 0x7f93dc06d1e0 2023-10-10T18:45:14.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e1942700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f93dc0c4d50 0x7f93dc153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:14.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e1942700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f93dc0c4d50 0x7f93dc153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50796/0 (socket says 172.21.15.175:50796) 2023-10-10T18:45:14.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e1942700 1 -- 172.21.15.175:0/2636784915 learned_addr learned my addr 172.21.15.175:0/2636784915 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:14.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e1141700 1 --2- 172.21.15.175:0/2636784915 >> v2:172.21.15.191:3300/0 conn(0x7f93dc06d1e0 0x7f93dc155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:14.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e1942700 1 -- 172.21.15.175:0/2636784915 >> v2:172.21.15.191:3300/0 conn(0x7f93dc06d1e0 msgr2=0x7f93dc155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:14.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e1942700 1 --2- 172.21.15.175:0/2636784915 >> v2:172.21.15.191:3300/0 conn(0x7f93dc06d1e0 0x7f93dc155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e1942700 1 -- 172.21.15.175:0/2636784915 >> v1:172.21.15.175:6789/0 conn(0x7f93dc0c7ae0 legacy=0x7f93dc152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:14.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.561+0000 7f93e1942700 1 -- 172.21.15.175:0/2636784915 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc068aa0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e1942700 1 --2- 172.21.15.175:0/2636784915 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc153430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93dc06ca50 tx=0x7f93d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1e2b254ed00e48 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:14.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93d3fff700 1 -- 172.21.15.175:0/2636784915 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d400b8f0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93d3fff700 1 -- 172.21.15.175:0/2636784915 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d400ba50 con 0x7f93dc0c4d50 2023-10-10T18:45:14.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93d3fff700 1 -- 172.21.15.175:0/2636784915 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d400bdf0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e2143700 1 -- 172.21.15.175:0/2636784915 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 msgr2=0x7f93dc153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:14.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e2143700 1 --2- 172.21.15.175:0/2636784915 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc153430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93dc06ca50 tx=0x7f93d4009250 comp rx=0 tx=0).stop 2023-10-10T18:45:14.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e2143700 1 -- 172.21.15.175:0/2636784915 shutdown_connections 2023-10-10T18:45:14.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e2143700 1 --2- 172.21.15.175:0/2636784915 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc153430 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e2143700 1 --2- 172.21.15.175:0/2636784915 >> v2:172.21.15.191:3300/0 conn(0x7f93dc06d1e0 0x7f93dc155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e2143700 1 -- 172.21.15.175:0/2636784915 >> 172.21.15.175:0/2636784915 conn(0x7f93dc06bbb0 msgr2=0x7f93dc06bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:14.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e2143700 1 -- 172.21.15.175:0/2636784915 shutdown_connections 2023-10-10T18:45:14.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.562+0000 7f93e2143700 1 -- 172.21.15.175:0/2636784915 wait complete. 2023-10-10T18:45:14.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.563+0000 7f93e2143700 1 Processor -- start 2023-10-10T18:45:14.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.563+0000 7f93e2143700 1 -- start start 2023-10-10T18:45:14.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e2143700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93dc06d1e0 0x7f93dc167490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:14.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e2143700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc1679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:14.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e2143700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93dc0c7ae0 0x7f93dc166030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:14.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e2143700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc0cb0b0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e2143700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc0c43c0 con 0x7f93dc06d1e0 2023-10-10T18:45:14.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e2143700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc0c34d0 con 0x7f93dc0c7ae0 2023-10-10T18:45:14.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e0940700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc1679d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:14.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e0940700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc1679d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50798/0 (socket says 172.21.15.175:50798) 2023-10-10T18:45:14.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e0940700 1 -- 172.21.15.175:0/2049111359 learned_addr learned my addr 172.21.15.175:0/2049111359 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:14.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e1141700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93dc06d1e0 0x7f93dc167490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:14.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e1942700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93dc0c7ae0 0x7f93dc166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:14.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e0940700 1 -- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93dc0c7ae0 msgr2=0x7f93dc166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:14.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e0940700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93dc0c7ae0 0x7f93dc166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e0940700 1 -- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93dc06d1e0 msgr2=0x7f93dc167490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:14.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e0940700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93dc06d1e0 0x7f93dc167490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:14.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.564+0000 7f93e0940700 1 -- 172.21.15.175:0/2049111359 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d40094f0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.565+0000 7f93e0940700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc1679d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00c590 tx=0x7f93cc007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:14.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.565+0000 7f93d17fa700 1 -- 172.21.15.175:0/2049111359 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc011590 con 0x7f93dc0c4d50 2023-10-10T18:45:14.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.565+0000 7f93d17fa700 1 -- 172.21.15.175:0/2049111359 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc011710 con 0x7f93dc0c4d50 2023-10-10T18:45:14.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.565+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc0c37b0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.565+0000 7f93d17fa700 1 -- 172.21.15.175:0/2049111359 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc011ad0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.565+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc0c3d70 con 0x7f93dc0c4d50 2023-10-10T18:45:14.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.566+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc066d60 con 0x7f93dc0c4d50 2023-10-10T18:45:14.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.566+0000 7f93d17fa700 1 -- 172.21.15.175:0/2049111359 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f93cc01b3b0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.566+0000 7f93d17fa700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93c0066f60 0x7f93c0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:14.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.567+0000 7f93d17fa700 1 -- 172.21.15.175:0/2049111359 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f93cc0839a0 con 0x7f93dc0c4d50 2023-10-10T18:45:14.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.567+0000 7f93e1141700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93c0066f60 0x7f93c0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:14.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.567+0000 7f93e1141700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93c0066f60 0x7f93c0069410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009fd0 tx=0x7f93d8009880 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:14.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.571+0000 7f93d17fa700 1 -- 172.21.15.175:0/2049111359 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93cc050c10 con 0x7f93dc0c4d50 2023-10-10T18:45:14.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:14.793+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f93dc04bb80 con 0x7f93dc0c4d50 2023-10-10T18:45:15.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.431+0000 7f93d17fa700 1 -- 172.21.15.175:0/2049111359 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f93cc017070 con 0x7f93dc0c4d50 2023-10-10T18:45:15.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93c0066f60 msgr2=0x7f93c0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:15.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93c0066f60 0x7f93c0069410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009fd0 tx=0x7f93d8009880 comp rx=0 tx=0).stop 2023-10-10T18:45:15.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 msgr2=0x7f93dc1679d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:15.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc1679d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00c590 tx=0x7f93cc007f50 comp rx=0 tx=0).stop 2023-10-10T18:45:15.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 shutdown_connections 2023-10-10T18:45:15.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93c0066f60 0x7f93c0069410 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93dc0c7ae0 0x7f93dc166030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93dc06d1e0 0x7f93dc167490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 --2- 172.21.15.175:0/2049111359 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93dc0c4d50 0x7f93dc1679d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 >> 172.21.15.175:0/2049111359 conn(0x7f93dc06bbb0 msgr2=0x7f93dc153c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:15.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 shutdown_connections 2023-10-10T18:45:15.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.434+0000 7f93e2143700 1 -- 172.21.15.175:0/2049111359 wait complete. 2023-10-10T18:45:15.448 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-10T18:45:15.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.567+0000 7fd705681700 1 Processor -- start 2023-10-10T18:45:15.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.567+0000 7fd705681700 1 -- start start 2023-10-10T18:45:15.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.568+0000 7fd705681700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 0x7fd7000aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:15.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.568+0000 7fd705681700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd7000b0490 0x7fd70014f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:15.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.568+0000 7fd705681700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7000c7ca0 con 0x7fd7000633f0 2023-10-10T18:45:15.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.568+0000 7fd705681700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7000af540 con 0x7fd7000b0490 2023-10-10T18:45:15.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.568+0000 7fd705681700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd700156830 con 0x7fd7000c7620 2023-10-10T18:45:15.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.568+0000 7fd6ff7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd7000b0490 0x7fd70014f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:15.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.568+0000 7fd6ff7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd7000b0490 0x7fd70014f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52168/0 (socket says 172.21.15.175:52168) 2023-10-10T18:45:15.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.568+0000 7fd6ff7fe700 1 -- 172.21.15.175:0/1254955297 learned_addr learned my addr 172.21.15.175:0/1254955297 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:15.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6f67fc700 1 --2- 172.21.15.175:0/1254955297 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 0x7fd7000aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:15.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6ff7fe700 1 -- 172.21.15.175:0/1254955297 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 msgr2=0x7fd7000aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:15.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6ff7fe700 1 --2- 172.21.15.175:0/1254955297 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 0x7fd7000aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6ff7fe700 1 -- 172.21.15.175:0/1254955297 >> v1:172.21.15.191:6789/0 conn(0x7fd7000633f0 legacy=0x7fd700156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:15.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6ff7fe700 1 -- 172.21.15.175:0/1254955297 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd700156ad0 con 0x7fd7000b0490 2023-10-10T18:45:15.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6ff7fe700 1 --2- 172.21.15.175:0/1254955297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd70014f7c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8009f20 tx=0x7fd6f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8dc2f4f7bf7fc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:15.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6fe7fc700 1 -- 172.21.15.175:0/1254955297 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f800e070 con 0x7fd7000b0490 2023-10-10T18:45:15.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6fe7fc700 1 -- 172.21.15.175:0/1254955297 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f800bad0 con 0x7fd7000b0490 2023-10-10T18:45:15.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd6fe7fc700 1 -- 172.21.15.175:0/1254955297 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f80092e0 con 0x7fd7000b0490 2023-10-10T18:45:15.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd705681700 1 -- 172.21.15.175:0/1254955297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 msgr2=0x7fd70014f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:15.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd705681700 1 --2- 172.21.15.175:0/1254955297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd70014f7c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8009f20 tx=0x7fd6f8009840 comp rx=0 tx=0).stop 2023-10-10T18:45:15.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd705681700 1 -- 172.21.15.175:0/1254955297 shutdown_connections 2023-10-10T18:45:15.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd705681700 1 --2- 172.21.15.175:0/1254955297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd70014f7c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd705681700 1 --2- 172.21.15.175:0/1254955297 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 0x7fd7000aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd705681700 1 -- 172.21.15.175:0/1254955297 >> 172.21.15.175:0/1254955297 conn(0x7fd7000cc7a0 msgr2=0x7fd7000ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:15.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd705681700 1 -- 172.21.15.175:0/1254955297 shutdown_connections 2023-10-10T18:45:15.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.569+0000 7fd705681700 1 -- 172.21.15.175:0/1254955297 wait complete. 2023-10-10T18:45:15.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd705681700 1 Processor -- start 2023-10-10T18:45:15.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd705681700 1 -- start start 2023-10-10T18:45:15.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd705681700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7000633f0 0x7fd70014b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:15.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd705681700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd7000c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:15.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd705681700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 0x7fd7000c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:15.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd705681700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7001508f0 con 0x7fd7000b0490 2023-10-10T18:45:15.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd705681700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7001518d0 con 0x7fd7000633f0 2023-10-10T18:45:15.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd705681700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7000b0250 con 0x7fd7000c7620 2023-10-10T18:45:15.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd6f67fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd7000c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:15.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd6f67fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd7000c6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52178/0 (socket says 172.21.15.175:52178) 2023-10-10T18:45:15.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.571+0000 7fd6f67fc700 1 -- 172.21.15.175:0/40454240 learned_addr learned my addr 172.21.15.175:0/40454240 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:15.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6ff7fe700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 0x7fd7000c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:15.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6feffd700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7000633f0 0x7fd70014b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:15.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f67fc700 1 -- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 msgr2=0x7fd7000c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:15.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f67fc700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 0x7fd7000c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f67fc700 1 -- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7000633f0 msgr2=0x7fd70014b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:15.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f67fc700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7000633f0 0x7fd70014b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:15.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f67fc700 1 -- 172.21.15.175:0/40454240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f8009ae0 con 0x7fd7000b0490 2023-10-10T18:45:15.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f67fc700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd7000c6580 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800c7d0 tx=0x7fd6e8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:15.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f77fe700 1 -- 172.21.15.175:0/40454240 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8011600 con 0x7fd7000b0490 2023-10-10T18:45:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f77fe700 1 -- 172.21.15.175:0/40454240 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e8011760 con 0x7fd7000b0490 2023-10-10T18:45:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd6f77fe700 1 -- 172.21.15.175:0/40454240 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8011b00 con 0x7fd7000b0490 2023-10-10T18:45:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70014b5a0 con 0x7fd7000b0490 2023-10-10T18:45:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.572+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd700150b00 con 0x7fd7000b0490 2023-10-10T18:45:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.573+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd700153a20 con 0x7fd7000b0490 2023-10-10T18:45:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.573+0000 7fd6f77fe700 1 -- 172.21.15.175:0/40454240 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd6e8010350 con 0x7fd7000b0490 2023-10-10T18:45:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.574+0000 7fd6f77fe700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6e0066f30 0x7fd6e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.574+0000 7fd6f77fe700 1 -- 172.21.15.175:0/40454240 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fd6e8084540 con 0x7fd7000b0490 2023-10-10T18:45:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.574+0000 7fd6feffd700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6e0066f30 0x7fd6e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.574+0000 7fd6feffd700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6e0066f30 0x7fd6e00693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0006fd0 tx=0x7fd6f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:15.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.578+0000 7fd6f77fe700 1 -- 172.21.15.175:0/40454240 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd700153a20 con 0x7fd7000b0490 2023-10-10T18:45:15.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:15.805+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fd700153a20 con 0x7fd7000b0490 2023-10-10T18:45:16.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.441+0000 7fd6f77fe700 1 -- 172.21.15.175:0/40454240 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd700153a20 con 0x7fd7000b0490 2023-10-10T18:45:16.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6e0066f30 msgr2=0x7fd6e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6e0066f30 0x7fd6e00693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0006fd0 tx=0x7fd6f0008040 comp rx=0 tx=0).stop 2023-10-10T18:45:16.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 msgr2=0x7fd7000c6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd7000c6580 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800c7d0 tx=0x7fd6e8008040 comp rx=0 tx=0).stop 2023-10-10T18:45:16.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 shutdown_connections 2023-10-10T18:45:16.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6e0066f30 0x7fd6e00693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7000c7620 0x7fd7000c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7000633f0 0x7fd70014b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 --2- 172.21.15.175:0/40454240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7000b0490 0x7fd7000c6580 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 >> 172.21.15.175:0/40454240 conn(0x7fd7000cc7a0 msgr2=0x7fd7000c9800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:16.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 shutdown_connections 2023-10-10T18:45:16.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.443+0000 7fd705681700 1 -- 172.21.15.175:0/40454240 wait complete. 2023-10-10T18:45:16.458 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:16.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.574+0000 7f4fa4388700 1 Processor -- start 2023-10-10T18:45:16.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.574+0000 7f4fa4388700 1 -- start start 2023-10-10T18:45:16.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa4388700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4f9c0c1fb0 0x7f4f9c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa4388700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4f9c14ddb0 0x7f4f9c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa4388700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9c154e20 con 0x7f4f9c0c1be0 2023-10-10T18:45:16.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa4388700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c06bca0 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa4388700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c154000 con 0x7f4f9c14ddb0 2023-10-10T18:45:16.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa1923700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4f9c0c1fb0 0x7f4f9c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa1923700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4f9c0c1fb0 0x7f4f9c06b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52180/0 (socket says 172.21.15.175:52180) 2023-10-10T18:45:16.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa1923700 1 -- 172.21.15.175:0/369904295 learned_addr learned my addr 172.21.15.175:0/369904295 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:16.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.575+0000 7f4fa2925700 1 --2- 172.21.15.175:0/369904295 >> v2:172.21.15.191:3300/0 conn(0x7f4f9c14ddb0 0x7f4f9c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa1923700 1 -- 172.21.15.175:0/369904295 >> v2:172.21.15.191:3300/0 conn(0x7f4f9c14ddb0 msgr2=0x7f4f9c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:16.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa1923700 1 --2- 172.21.15.175:0/369904295 >> v2:172.21.15.191:3300/0 conn(0x7f4f9c14ddb0 0x7f4f9c14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa1923700 1 -- 172.21.15.175:0/369904295 >> v1:172.21.15.191:6789/0 conn(0x7f4f9c0c1be0 legacy=0x7f4f9c06a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:16.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa1923700 1 -- 172.21.15.175:0/369904295 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c14e730 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa1923700 1 --2- 172.21.15.175:0/369904295 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c06b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0720f0 tx=0x7f4f8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f6bc30f3a1da6e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:16.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa1122700 1 -- 172.21.15.175:0/369904295 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00b820 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa1122700 1 -- 172.21.15.175:0/369904295 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c00b980 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa1122700 1 -- 172.21.15.175:0/369904295 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00bd20 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa4388700 1 -- 172.21.15.175:0/369904295 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 msgr2=0x7f4f9c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa4388700 1 --2- 172.21.15.175:0/369904295 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c06b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0720f0 tx=0x7f4f8c009250 comp rx=0 tx=0).stop 2023-10-10T18:45:16.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa4388700 1 -- 172.21.15.175:0/369904295 shutdown_connections 2023-10-10T18:45:16.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa4388700 1 --2- 172.21.15.175:0/369904295 >> v2:172.21.15.191:3300/0 conn(0x7f4f9c14ddb0 0x7f4f9c14e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa4388700 1 --2- 172.21.15.175:0/369904295 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c06b760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa4388700 1 -- 172.21.15.175:0/369904295 >> 172.21.15.175:0/369904295 conn(0x7f4f9c068280 msgr2=0x7f4f9c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:16.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa4388700 1 -- 172.21.15.175:0/369904295 shutdown_connections 2023-10-10T18:45:16.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.576+0000 7f4fa4388700 1 -- 172.21.15.175:0/369904295 wait complete. 2023-10-10T18:45:16.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa4388700 1 Processor -- start 2023-10-10T18:45:16.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa4388700 1 -- start start 2023-10-10T18:45:16.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa4388700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f9c0c1be0 0x7f4f9c1682b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa4388700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c1491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa4388700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f9c14ddb0 0x7f4f9c14e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa4388700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c149720 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa4388700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c151830 con 0x7f4f9c14ddb0 2023-10-10T18:45:16.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa4388700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c164ac0 con 0x7f4f9c0c1be0 2023-10-10T18:45:16.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa1923700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c1491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa1923700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c1491e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52182/0 (socket says 172.21.15.175:52182) 2023-10-10T18:45:16.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa1923700 1 -- 172.21.15.175:0/2538180568 learned_addr learned my addr 172.21.15.175:0/2538180568 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:16.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa2925700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f9c14ddb0 0x7f4f9c14e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.578+0000 7f4fa2124700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f9c0c1be0 0x7f4f9c1682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4fa1923700 1 -- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f9c0c1be0 msgr2=0x7f4f9c1682b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4fa1923700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f9c0c1be0 0x7f4f9c1682b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4fa1923700 1 -- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f9c14ddb0 msgr2=0x7f4f9c14e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4fa1923700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f9c14ddb0 0x7f4f9c14e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4fa1923700 1 -- 172.21.15.175:0/2538180568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c0094f0 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4fa1923700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c1491e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0030c0 tx=0x7f4f8c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:16.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4f927fc700 1 -- 172.21.15.175:0/2538180568 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c011d00 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4f927fc700 1 -- 172.21.15.175:0/2538180568 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c020070 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c14ee40 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4f927fc700 1 -- 172.21.15.175:0/2538180568 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c01b6a0 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.579+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c1512b0 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.580+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c164c70 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.580+0000 7f4f927fc700 1 -- 172.21.15.175:0/2538180568 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4f8c022030 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.580+0000 7f4f927fc700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f80066f60 0x7f4f80069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.581+0000 7f4f927fc700 1 -- 172.21.15.175:0/2538180568 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4f8c086360 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.581+0000 7f4fa2124700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f80066f60 0x7f4f80069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.581+0000 7f4fa2124700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f80066f60 0x7f4f80069410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800f430 tx=0x7f4f98002ad0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:16.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.585+0000 7f4f927fc700 1 -- 172.21.15.175:0/2538180568 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f9c164c70 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.807+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f9c1687f0 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.808+0000 7f4f927fc700 1 -- 172.21.15.175:0/2538180568 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6115 (secure 0 0 0) 0x7f4f9c1687f0 con 0x7f4f9c0c1fb0 2023-10-10T18:45:16.810 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:16.810 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4250,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:45:16.432213+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:45:16.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f80066f60 msgr2=0x7f4f80069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f80066f60 0x7f4f80069410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800f430 tx=0x7f4f98002ad0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 msgr2=0x7f4f9c1491e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c1491e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0030c0 tx=0x7f4f8c016000 comp rx=0 tx=0).stop 2023-10-10T18:45:16.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 shutdown_connections 2023-10-10T18:45:16.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f80066f60 0x7f4f80069410 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f9c14ddb0 0x7f4f9c14e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f9c0c1be0 0x7f4f9c1682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 --2- 172.21.15.175:0/2538180568 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f9c0c1fb0 0x7f4f9c1491e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 >> 172.21.15.175:0/2538180568 conn(0x7f4f9c068280 msgr2=0x7f4f9c152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:16.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 shutdown_connections 2023-10-10T18:45:16.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.810+0000 7f4fa4388700 1 -- 172.21.15.175:0/2538180568 wait complete. 2023-10-10T18:45:16.815 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 8 2023-10-10T18:45:16.823 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:16.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.936+0000 7f794ccce700 1 Processor -- start 2023-10-10T18:45:16.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.936+0000 7f794ccce700 1 -- start start 2023-10-10T18:45:16.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.937+0000 7f794ccce700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79481573d0 0x7f79480c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.937+0000 7f794ccce700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7948154aa0 con 0x7f79481577a0 2023-10-10T18:45:16.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.937+0000 7f794ccce700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7948063dc0 con 0x7f79480c4f20 2023-10-10T18:45:16.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.937+0000 7f794ccce700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79480bfdf0 con 0x7f79481573d0 2023-10-10T18:45:16.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.937+0000 7f7945d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f79481577a0 0x7f79480bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60480/0 (socket says 172.21.15.175:60480) 2023-10-10T18:45:16.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.937+0000 7f7945d9b700 1 -- 172.21.15.175:0/737740463 learned_addr learned my addr 172.21.15.175:0/737740463 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:16.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794659c700 1 --2- 172.21.15.175:0/737740463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79481573d0 0x7f79480c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794559a700 1 -- 172.21.15.175:0/737740463 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244531799 0 0) 0x7f7948154aa0 con 0x7f79481577a0 2023-10-10T18:45:16.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794559a700 1 -- 172.21.15.175:0/737740463 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7934003570 con 0x7f79481577a0 2023-10-10T18:45:16.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794559a700 1 -- 172.21.15.175:0/737740463 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919426872 0 0) 0x7f7948063dc0 con 0x7f79480c4f20 2023-10-10T18:45:16.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794659c700 1 -- 172.21.15.175:0/737740463 >> v1:172.21.15.191:6789/0 conn(0x7f79480c4f20 legacy=0x7f794814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794659c700 1 -- 172.21.15.175:0/737740463 >> v1:172.21.15.175:6789/0 conn(0x7f79481577a0 legacy=0x7f79480bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794659c700 1 -- 172.21.15.175:0/737740463 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480c5380 con 0x7f79481573d0 2023-10-10T18:45:16.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794659c700 1 --2- 172.21.15.175:0/737740463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79481573d0 0x7f79480c12a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7938009e90 tx=0x7f7938009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7668c2f9d6100bf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:16.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794559a700 1 -- 172.21.15.175:0/737740463 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793800e070 con 0x7f79481573d0 2023-10-10T18:45:16.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794559a700 1 -- 172.21.15.175:0/737740463 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793800bc30 con 0x7f79481573d0 2023-10-10T18:45:16.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.938+0000 7f794559a700 1 -- 172.21.15.175:0/737740463 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938012430 con 0x7f79481573d0 2023-10-10T18:45:16.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.939+0000 7f794ccce700 1 -- 172.21.15.175:0/737740463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79481573d0 msgr2=0x7f79480c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.939+0000 7f794ccce700 1 --2- 172.21.15.175:0/737740463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79481573d0 0x7f79480c12a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7938009e90 tx=0x7f7938009780 comp rx=0 tx=0).stop 2023-10-10T18:45:16.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.939+0000 7f794ccce700 1 -- 172.21.15.175:0/737740463 shutdown_connections 2023-10-10T18:45:16.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.939+0000 7f794ccce700 1 --2- 172.21.15.175:0/737740463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79481573d0 0x7f79480c12a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.939+0000 7f794ccce700 1 -- 172.21.15.175:0/737740463 >> 172.21.15.175:0/737740463 conn(0x7f794806dc70 msgr2=0x7f7948063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:16.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.939+0000 7f794ccce700 1 -- 172.21.15.175:0/737740463 shutdown_connections 2023-10-10T18:45:16.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.939+0000 7f794ccce700 1 -- 172.21.15.175:0/737740463 wait complete. 2023-10-10T18:45:16.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.940+0000 7f794ccce700 1 Processor -- start 2023-10-10T18:45:16.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.940+0000 7f794ccce700 1 -- start start 2023-10-10T18:45:16.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794ccce700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79480c4f20 0x7f7948156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794ccce700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79481573d0 0x7f7948147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794ccce700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79481577a0 0x7f79481480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794ccce700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7948152340 con 0x7f79481573d0 2023-10-10T18:45:16.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794ccce700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7948155b20 con 0x7f79481577a0 2023-10-10T18:45:16.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794ccce700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7948152530 con 0x7f79480c4f20 2023-10-10T18:45:16.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79481573d0 0x7f7948147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79481573d0 0x7f7948147b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52184/0 (socket says 172.21.15.175:52184) 2023-10-10T18:45:16.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 -- 172.21.15.175:0/2302128818 learned_addr learned my addr 172.21.15.175:0/2302128818 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:16.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794659c700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79480c4f20 0x7f7948156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7946d9d700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79481577a0 0x7f79481480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 -- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79480c4f20 msgr2=0x7f7948156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79480c4f20 0x7f7948156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 -- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79481577a0 msgr2=0x7f79481480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:16.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79481577a0 0x7f79481480d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:16.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 -- 172.21.15.175:0/2302128818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7938009a20 con 0x7f79481573d0 2023-10-10T18:45:16.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794659c700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79480c4f20 0x7f7948156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:16.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7946d9d700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79481577a0 0x7f79481480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:16.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f7945d9b700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79481573d0 0x7f7948147b90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f793000c580 tx=0x7f7930007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:16.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f793e7fc700 1 -- 172.21.15.175:0/2302128818 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930005b30 con 0x7f79481573d0 2023-10-10T18:45:16.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f793e7fc700 1 -- 172.21.15.175:0/2302128818 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7930005c90 con 0x7f79481573d0 2023-10-10T18:45:16.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7948155840 con 0x7f79481573d0 2023-10-10T18:45:16.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f793e7fc700 1 -- 172.21.15.175:0/2302128818 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930007320 con 0x7f79481573d0 2023-10-10T18:45:16.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.941+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7948152ab0 con 0x7f79481573d0 2023-10-10T18:45:16.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.942+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7948152fd0 con 0x7f79481573d0 2023-10-10T18:45:16.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.943+0000 7f793e7fc700 1 -- 172.21.15.175:0/2302128818 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7930007690 con 0x7f79481573d0 2023-10-10T18:45:16.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.943+0000 7f793e7fc700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7928066f30 0x7f79280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:16.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.943+0000 7f793e7fc700 1 -- 172.21.15.175:0/2302128818 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7930083a90 con 0x7f79481573d0 2023-10-10T18:45:16.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.943+0000 7f794659c700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7928066f30 0x7f79280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:16.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.944+0000 7f794659c700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7928066f30 0x7f79280693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7938009de0 tx=0x7f7938019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:16.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:16.947+0000 7f793e7fc700 1 -- 172.21.15.175:0/2302128818 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7948152fd0 con 0x7f79481573d0 2023-10-10T18:45:17.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.132+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7948152fd0 con 0x7f79481573d0 2023-10-10T18:45:17.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.133+0000 7f793e7fc700 1 -- 172.21.15.175:0/2302128818 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7f7948152fd0 con 0x7f79481573d0 2023-10-10T18:45:17.134 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:17.135 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:17.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7928066f30 msgr2=0x7f79280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7928066f30 0x7f79280693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7938009de0 tx=0x7f7938019000 comp rx=0 tx=0).stop 2023-10-10T18:45:17.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79481573d0 msgr2=0x7f7948147b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79481573d0 0x7f7948147b90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f793000c580 tx=0x7f7930007df0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 shutdown_connections 2023-10-10T18:45:17.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7928066f30 0x7f79280693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79481577a0 0x7f79481480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79480c4f20 0x7f7948156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 --2- 172.21.15.175:0/2302128818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79481573d0 0x7f7948147b90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 >> 172.21.15.175:0/2302128818 conn(0x7f794806dc70 msgr2=0x7f794814d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:17.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 shutdown_connections 2023-10-10T18:45:17.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.135+0000 7f794ccce700 1 -- 172.21.15.175:0/2302128818 wait complete. 2023-10-10T18:45:17.150 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-10T18:45:17.151 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T18:45:17.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.264+0000 7fb112f68700 1 Processor -- start 2023-10-10T18:45:17.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.264+0000 7fb112f68700 1 -- start start 2023-10-10T18:45:17.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb112f68700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb10c14fd90 0x7fb10c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb112f68700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb10c08a100 0x7fb10c0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb112f68700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10c067fa0 con 0x7fb10c0cd040 2023-10-10T18:45:17.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb112f68700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10c0c5600 con 0x7fb10c14fd90 2023-10-10T18:45:17.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb112f68700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10c156b70 con 0x7fb10c08a100 2023-10-10T18:45:17.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111f66700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb10c14fd90 0x7fb10c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111f66700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb10c14fd90 0x7fb10c159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52194/0 (socket says 172.21.15.175:52194) 2023-10-10T18:45:17.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111f66700 1 -- 172.21.15.175:0/1552457937 learned_addr learned my addr 172.21.15.175:0/1552457937 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:17.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111765700 1 --2- 172.21.15.175:0/1552457937 >> v2:172.21.15.191:3300/0 conn(0x7fb10c08a100 0x7fb10c0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111f66700 1 -- 172.21.15.175:0/1552457937 >> v2:172.21.15.191:3300/0 conn(0x7fb10c08a100 msgr2=0x7fb10c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:17.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111f66700 1 --2- 172.21.15.175:0/1552457937 >> v2:172.21.15.191:3300/0 conn(0x7fb10c08a100 0x7fb10c0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111f66700 1 -- 172.21.15.175:0/1552457937 >> v1:172.21.15.175:6789/0 conn(0x7fb10c0cd040 legacy=0x7fb10c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb110f64700 1 -- 172.21.15.175:0/1552457937 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186616749 0 0) 0x7fb10c067fa0 con 0x7fb10c0cd040 2023-10-10T18:45:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111f66700 1 -- 172.21.15.175:0/1552457937 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c150160 con 0x7fb10c14fd90 2023-10-10T18:45:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb111f66700 1 --2- 172.21.15.175:0/1552457937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c159c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb10c069990 tx=0x7fb108009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416ecac59bcaf74e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb110f64700 1 -- 172.21.15.175:0/1552457937 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10800b8f0 con 0x7fb10c14fd90 2023-10-10T18:45:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb110f64700 1 -- 172.21.15.175:0/1552457937 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10800ba50 con 0x7fb10c14fd90 2023-10-10T18:45:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.265+0000 7fb110f64700 1 -- 172.21.15.175:0/1552457937 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10800bdf0 con 0x7fb10c14fd90 2023-10-10T18:45:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.266+0000 7fb112f68700 1 -- 172.21.15.175:0/1552457937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 msgr2=0x7fb10c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.266+0000 7fb112f68700 1 --2- 172.21.15.175:0/1552457937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c159c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb10c069990 tx=0x7fb108009250 comp rx=0 tx=0).stop 2023-10-10T18:45:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.266+0000 7fb112f68700 1 -- 172.21.15.175:0/1552457937 shutdown_connections 2023-10-10T18:45:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.266+0000 7fb112f68700 1 --2- 172.21.15.175:0/1552457937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c159c30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.266+0000 7fb112f68700 1 --2- 172.21.15.175:0/1552457937 >> v2:172.21.15.191:3300/0 conn(0x7fb10c08a100 0x7fb10c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.266+0000 7fb112f68700 1 -- 172.21.15.175:0/1552457937 >> 172.21.15.175:0/1552457937 conn(0x7fb10c063d00 msgr2=0x7fb10c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:17.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.266+0000 7fb112f68700 1 -- 172.21.15.175:0/1552457937 shutdown_connections 2023-10-10T18:45:17.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.266+0000 7fb112f68700 1 -- 172.21.15.175:0/1552457937 wait complete. 2023-10-10T18:45:17.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.267+0000 7fb112f68700 1 Processor -- start 2023-10-10T18:45:17.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.267+0000 7fb112f68700 1 -- start start 2023-10-10T18:45:17.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112f68700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb10c08a100 0x7fb10c14c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112f68700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb10c0cd040 0x7fb10c155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112f68700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112f68700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c1693c0 con 0x7fb10c14fd90 2023-10-10T18:45:17.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112f68700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c14ca90 con 0x7fb10c0cd040 2023-10-10T18:45:17.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112f68700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c165cc0 con 0x7fb10c08a100 2023-10-10T18:45:17.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52206/0 (socket says 172.21.15.175:52206) 2023-10-10T18:45:17.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 -- 172.21.15.175:0/3990566672 learned_addr learned my addr 172.21.15.175:0/3990566672 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:17.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb111765700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb10c0cd040 0x7fb10c155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb111f66700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb10c08a100 0x7fb10c14c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 -- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb10c08a100 msgr2=0x7fb10c14c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb10c08a100 0x7fb10c14c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 -- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb10c0cd040 msgr2=0x7fb10c155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb10c0cd040 0x7fb10c155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 -- 172.21.15.175:0/3990566672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080094f0 con 0x7fb10c14fd90 2023-10-10T18:45:17.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb111f66700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb10c08a100 0x7fb10c14c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:17.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112767700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c155540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb104001560 tx=0x7fb104001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:17.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb101ffb700 1 -- 172.21.15.175:0/3990566672 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104003030 con 0x7fb10c14fd90 2023-10-10T18:45:17.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb101ffb700 1 -- 172.21.15.175:0/3990566672 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10400a870 con 0x7fb10c14fd90 2023-10-10T18:45:17.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10c169660 con 0x7fb10c14fd90 2023-10-10T18:45:17.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.268+0000 7fb101ffb700 1 -- 172.21.15.175:0/3990566672 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10400ac30 con 0x7fb10c14fd90 2023-10-10T18:45:17.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.269+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c169c20 con 0x7fb10c14fd90 2023-10-10T18:45:17.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.270+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10c067570 con 0x7fb10c14fd90 2023-10-10T18:45:17.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.270+0000 7fb101ffb700 1 -- 172.21.15.175:0/3990566672 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb104012420 con 0x7fb10c14fd90 2023-10-10T18:45:17.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.270+0000 7fb101ffb700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb0f0066f60 0x7fb0f0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.270+0000 7fb101ffb700 1 -- 172.21.15.175:0/3990566672 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb104004070 con 0x7fb10c14fd90 2023-10-10T18:45:17.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.270+0000 7fb111f66700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb0f0066f60 0x7fb0f0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.271+0000 7fb111f66700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb0f0066f60 0x7fb0f0069410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb10c069990 tx=0x7fb108017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:17.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.274+0000 7fb101ffb700 1 -- 172.21.15.175:0/3990566672 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1040516d0 con 0x7fb10c14fd90 2023-10-10T18:45:17.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.456+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb10c155f80 con 0x7fb0f0066f60 2023-10-10T18:45:17.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.465+0000 7fb101ffb700 1 -- 172.21.15.175:0/3990566672 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264601 (secure 0 0 0) 0x7fb10c155f80 con 0x7fb0f0066f60 2023-10-10T18:45:17.471 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:17.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb0f0066f60 msgr2=0x7fb0f0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb0f0066f60 0x7fb0f0069410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb10c069990 tx=0x7fb108017000 comp rx=0 tx=0).stop 2023-10-10T18:45:17.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 msgr2=0x7fb10c155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c155540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb104001560 tx=0x7fb104001770 comp rx=0 tx=0).stop 2023-10-10T18:45:17.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 shutdown_connections 2023-10-10T18:45:17.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb0f0066f60 0x7fb0f0069410 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb10c14fd90 0x7fb10c155540 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb10c08a100 0x7fb10c14c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 --2- 172.21.15.175:0/3990566672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb10c0cd040 0x7fb10c155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 >> 172.21.15.175:0/3990566672 conn(0x7fb10c063d00 msgr2=0x7fb10c0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:17.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 shutdown_connections 2023-10-10T18:45:17.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.468+0000 7fb112f68700 1 -- 172.21.15.175:0/3990566672 wait complete. 2023-10-10T18:45:17.479 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T18:45:17.482 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-10-10T18:45:17.131080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2380128,"kb_used_data":1824,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752594592,"statfs":{"total":773094113280,"available":770656862208,"internally_reserved":0,"allocated":1867776,"data_stored":1783240,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.742298"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429926+0000","last_change":"2023-10-10T18:45:11.419184+0000","last_active":"2023-10-10T18:45:11.429926+0000","last_peered":"2023-10-10T18:45:11.429926+0000","last_clean":"2023-10-10T18:45:11.429926+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T18:45:11.429926+0000","last_undegraded":"2023-10-10T18:45:11.429926+0000","last_fullsized":"2023-10-10T18:45:11.429926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420684+0000","last_change":"2023-10-10T18:45:11.414586+0000","last_active":"2023-10-10T18:45:11.420684+0000","last_peered":"2023-10-10T18:45:11.420684+0000","last_clean":"2023-10-10T18:45:11.420684+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T18:45:11.420684+0000","last_undegraded":"2023-10-10T18:45:11.420684+0000","last_fullsized":"2023-10-10T18:45:11.420684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424203+0000","last_change":"2023-10-10T18:45:11.419119+0000","last_active":"2023-10-10T18:45:11.424203+0000","last_peered":"2023-10-10T18:45:11.424203+0000","last_clean":"2023-10-10T18:45:11.424203+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T18:45:11.424203+0000","last_undegraded":"2023-10-10T18:45:11.424203+0000","last_fullsized":"2023-10-10T18:45:11.424203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425796+0000","last_change":"2023-10-10T18:45:11.421351+0000","last_active":"2023-10-10T18:45:11.425796+0000","last_peered":"2023-10-10T18:45:11.425796+0000","last_clean":"2023-10-10T18:45:11.425796+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T18:45:11.425796+0000","last_undegraded":"2023-10-10T18:45:11.425796+0000","last_fullsized":"2023-10-10T18:45:11.425796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425614+0000","last_change":"2023-10-10T18:45:11.421122+0000","last_active":"2023-10-10T18:45:11.425614+0000","last_peered":"2023-10-10T18:45:11.425614+0000","last_clean":"2023-10-10T18:45:11.425614+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T18:45:11.425614+0000","last_undegraded":"2023-10-10T18:45:11.425614+0000","last_fullsized":"2023-10-10T18:45:11.425614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424542+0000","last_change":"2023-10-10T18:45:11.421729+0000","last_active":"2023-10-10T18:45:11.424542+0000","last_peered":"2023-10-10T18:45:11.424542+0000","last_clean":"2023-10-10T18:45:11.424542+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T18:45:11.424542+0000","last_undegraded":"2023-10-10T18:45:11.424542+0000","last_fullsized":"2023-10-10T18:45:11.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426178+0000","last_change":"2023-10-10T18:45:11.419306+0000","last_active":"2023-10-10T18:45:11.426178+0000","last_peered":"2023-10-10T18:45:11.426178+0000","last_clean":"2023-10-10T18:45:11.426178+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T18:45:11.426178+0000","last_undegraded":"2023-10-10T18:45:11.426178+0000","last_fullsized":"2023-10-10T18:45:11.426178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.422940+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429946+0000","last_change":"2023-10-10T18:45:11.422612+0000","last_active":"2023-10-10T18:45:11.429946+0000","last_peered":"2023-10-10T18:45:11.429946+0000","last_clean":"2023-10-10T18:45:11.429946+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T18:45:11.429946+0000","last_undegraded":"2023-10-10T18:45:11.429946+0000","last_fullsized":"2023-10-10T18:45:11.429946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430763+0000","last_change":"2023-10-10T18:45:11.425754+0000","last_active":"2023-10-10T18:45:11.430763+0000","last_peered":"2023-10-10T18:45:11.430763+0000","last_clean":"2023-10-10T18:45:11.430763+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T18:45:11.430763+0000","last_undegraded":"2023-10-10T18:45:11.430763+0000","last_fullsized":"2023-10-10T18:45:11.430763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430664+0000","last_change":"2023-10-10T18:45:11.425891+0000","last_active":"2023-10-10T18:45:11.430664+0000","last_peered":"2023-10-10T18:45:11.430664+0000","last_clean":"2023-10-10T18:45:11.430664+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T18:45:11.430664+0000","last_undegraded":"2023-10-10T18:45:11.430664+0000","last_fullsized":"2023-10-10T18:45:11.430664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431015+0000","last_change":"2023-10-10T18:45:11.424834+0000","last_active":"2023-10-10T18:45:11.431015+0000","last_peered":"2023-10-10T18:45:11.431015+0000","last_clean":"2023-10-10T18:45:11.431015+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T18:45:11.431015+0000","last_undegraded":"2023-10-10T18:45:11.431015+0000","last_fullsized":"2023-10-10T18:45:11.431015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421085+0000","last_change":"2023-10-10T18:45:11.417796+0000","last_active":"2023-10-10T18:45:11.421085+0000","last_peered":"2023-10-10T18:45:11.421085+0000","last_clean":"2023-10-10T18:45:11.421085+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T18:45:11.421085+0000","last_undegraded":"2023-10-10T18:45:11.421085+0000","last_fullsized":"2023-10-10T18:45:11.421085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419270+0000","last_change":"2023-10-10T18:45:11.414690+0000","last_active":"2023-10-10T18:45:11.419270+0000","last_peered":"2023-10-10T18:45:11.419270+0000","last_clean":"2023-10-10T18:45:11.419270+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T18:45:11.419270+0000","last_undegraded":"2023-10-10T18:45:11.419270+0000","last_fullsized":"2023-10-10T18:45:11.419270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434526+0000","last_change":"2023-10-10T18:45:11.429433+0000","last_active":"2023-10-10T18:45:11.434526+0000","last_peered":"2023-10-10T18:45:11.434526+0000","last_clean":"2023-10-10T18:45:11.434526+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T18:45:11.434526+0000","last_undegraded":"2023-10-10T18:45:11.434526+0000","last_fullsized":"2023-10-10T18:45:11.434526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427193+0000","last_change":"2023-10-10T18:45:11.422935+0000","last_active":"2023-10-10T18:45:11.427193+0000","last_peered":"2023-10-10T18:45:11.427193+0000","last_clean":"2023-10-10T18:45:11.427193+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T18:45:11.427193+0000","last_undegraded":"2023-10-10T18:45:11.427193+0000","last_fullsized":"2023-10-10T18:45:11.427193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434131+0000","last_change":"2023-10-10T18:45:11.431202+0000","last_active":"2023-10-10T18:45:11.434131+0000","last_peered":"2023-10-10T18:45:11.434131+0000","last_clean":"2023-10-10T18:45:11.434131+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T18:45:11.434131+0000","last_undegraded":"2023-10-10T18:45:11.434131+0000","last_fullsized":"2023-10-10T18:45:11.434131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427139+0000","last_change":"2023-10-10T18:45:11.424885+0000","last_active":"2023-10-10T18:45:11.427139+0000","last_peered":"2023-10-10T18:45:11.427139+0000","last_clean":"2023-10-10T18:45:11.427139+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T18:45:11.427139+0000","last_undegraded":"2023-10-10T18:45:11.427139+0000","last_fullsized":"2023-10-10T18:45:11.427139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426206+0000","last_change":"2023-10-10T18:45:11.421039+0000","last_active":"2023-10-10T18:45:11.426206+0000","last_peered":"2023-10-10T18:45:11.426206+0000","last_clean":"2023-10-10T18:45:11.426206+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T18:45:11.426206+0000","last_undegraded":"2023-10-10T18:45:11.426206+0000","last_fullsized":"2023-10-10T18:45:11.426206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405851+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T18:45:11.405851+0000","last_peered":"2023-10-10T18:45:11.405851+0000","last_clean":"2023-10-10T18:45:11.405851+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T18:45:11.405851+0000","last_undegraded":"2023-10-10T18:45:11.405851+0000","last_fullsized":"2023-10-10T18:45:11.405851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421043+0000","last_change":"2023-10-10T18:45:11.417504+0000","last_active":"2023-10-10T18:45:11.421043+0000","last_peered":"2023-10-10T18:45:11.421043+0000","last_clean":"2023-10-10T18:45:11.421043+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T18:45:11.421043+0000","last_undegraded":"2023-10-10T18:45:11.421043+0000","last_fullsized":"2023-10-10T18:45:11.421043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.402480+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T18:45:11.402480+0000","last_peered":"2023-10-10T18:45:11.402480+0000","last_clean":"2023-10-10T18:45:11.402480+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T18:45:11.402480+0000","last_undegraded":"2023-10-10T18:45:11.402480+0000","last_fullsized":"2023-10-10T18:45:11.402480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424178+0000","last_change":"2023-10-10T18:45:11.421896+0000","last_active":"2023-10-10T18:45:11.424178+0000","last_peered":"2023-10-10T18:45:11.424178+0000","last_clean":"2023-10-10T18:45:11.424178+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T18:45:11.424178+0000","last_undegraded":"2023-10-10T18:45:11.424178+0000","last_fullsized":"2023-10-10T18:45:11.424178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.408064+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T18:45:11.408064+0000","last_peered":"2023-10-10T18:45:11.408064+0000","last_clean":"2023-10-10T18:45:11.408064+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T18:45:11.408064+0000","last_undegraded":"2023-10-10T18:45:11.408064+0000","last_fullsized":"2023-10-10T18:45:11.408064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432466+0000","last_change":"2023-10-10T18:45:11.429680+0000","last_active":"2023-10-10T18:45:11.432466+0000","last_peered":"2023-10-10T18:45:11.432466+0000","last_clean":"2023-10-10T18:45:11.432466+0000","last_became_active":"2023-10-10T18:45:11.429397+0000","last_became_peered":"2023-10-10T18:45:11.429397+0000","last_unstale":"2023-10-10T18:45:11.432466+0000","last_undegraded":"2023-10-10T18:45:11.432466+0000","last_fullsized":"2023-10-10T18:45:11.432466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:19:52.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.406417+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T18:45:11.406417+0000","last_peered":"2023-10-10T18:45:11.406417+0000","last_clean":"2023-10-10T18:45:11.406417+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T18:45:11.406417+0000","last_undegraded":"2023-10-10T18:45:11.406417+0000","last_fullsized":"2023-10-10T18:45:11.406417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432439+0000","last_change":"2023-10-10T18:45:11.429408+0000","last_active":"2023-10-10T18:45:11.432439+0000","last_peered":"2023-10-10T18:45:11.432439+0000","last_clean":"2023-10-10T18:45:11.432439+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T18:45:11.432439+0000","last_undegraded":"2023-10-10T18:45:11.432439+0000","last_fullsized":"2023-10-10T18:45:11.432439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.401851+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T18:45:11.401851+0000","last_peered":"2023-10-10T18:45:11.401851+0000","last_clean":"2023-10-10T18:45:11.401851+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T18:45:11.401851+0000","last_undegraded":"2023-10-10T18:45:11.401851+0000","last_fullsized":"2023-10-10T18:45:11.401851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430579+0000","last_change":"2023-10-10T18:45:11.426051+0000","last_active":"2023-10-10T18:45:11.430579+0000","last_peered":"2023-10-10T18:45:11.430579+0000","last_clean":"2023-10-10T18:45:11.430579+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T18:45:11.430579+0000","last_undegraded":"2023-10-10T18:45:11.430579+0000","last_fullsized":"2023-10-10T18:45:11.430579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410880+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T18:45:11.410880+0000","last_peered":"2023-10-10T18:45:11.410880+0000","last_clean":"2023-10-10T18:45:11.410880+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T18:45:11.410880+0000","last_undegraded":"2023-10-10T18:45:11.410880+0000","last_fullsized":"2023-10-10T18:45:11.410880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427223+0000","last_change":"2023-10-10T18:45:11.425183+0000","last_active":"2023-10-10T18:45:11.427223+0000","last_peered":"2023-10-10T18:45:11.427223+0000","last_clean":"2023-10-10T18:45:11.427223+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T18:45:11.427223+0000","last_undegraded":"2023-10-10T18:45:11.427223+0000","last_fullsized":"2023-10-10T18:45:11.427223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.402044+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T18:45:11.402044+0000","last_peered":"2023-10-10T18:45:11.402044+0000","last_clean":"2023-10-10T18:45:11.402044+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T18:45:11.402044+0000","last_undegraded":"2023-10-10T18:45:11.402044+0000","last_fullsized":"2023-10-10T18:45:11.402044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415249+0000","last_change":"2023-10-10T18:45:11.412641+0000","last_active":"2023-10-10T18:45:11.415249+0000","last_peered":"2023-10-10T18:45:11.415249+0000","last_clean":"2023-10-10T18:45:11.415249+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T18:45:11.415249+0000","last_undegraded":"2023-10-10T18:45:11.415249+0000","last_fullsized":"2023-10-10T18:45:11.415249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.413032+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T18:45:11.413032+0000","last_peered":"2023-10-10T18:45:11.413032+0000","last_clean":"2023-10-10T18:45:11.413032+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T18:45:11.413032+0000","last_undegraded":"2023-10-10T18:45:11.413032+0000","last_fullsized":"2023-10-10T18:45:11.413032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420638+0000","last_change":"2023-10-10T18:45:11.413819+0000","last_active":"2023-10-10T18:45:11.420638+0000","last_peered":"2023-10-10T18:45:11.420638+0000","last_clean":"2023-10-10T18:45:11.420638+0000","last_became_active":"2023-10-10T18:45:11.413529+0000","last_became_peered":"2023-10-10T18:45:11.413529+0000","last_unstale":"2023-10-10T18:45:11.420638+0000","last_undegraded":"2023-10-10T18:45:11.420638+0000","last_fullsized":"2023-10-10T18:45:11.420638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.411211+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T18:45:11.411211+0000","last_peered":"2023-10-10T18:45:11.411211+0000","last_clean":"2023-10-10T18:45:11.411211+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T18:45:11.411211+0000","last_undegraded":"2023-10-10T18:45:11.411211+0000","last_fullsized":"2023-10-10T18:45:11.411211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430421+0000","last_change":"2023-10-10T18:45:11.427116+0000","last_active":"2023-10-10T18:45:11.430421+0000","last_peered":"2023-10-10T18:45:11.430421+0000","last_clean":"2023-10-10T18:45:11.430421+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T18:45:11.430421+0000","last_undegraded":"2023-10-10T18:45:11.430421+0000","last_fullsized":"2023-10-10T18:45:11.430421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.416463+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T18:45:11.416463+0000","last_peered":"2023-10-10T18:45:11.416463+0000","last_clean":"2023-10-10T18:45:11.416463+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T18:45:11.416463+0000","last_undegraded":"2023-10-10T18:45:11.416463+0000","last_fullsized":"2023-10-10T18:45:11.416463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424158+0000","last_change":"2023-10-10T18:45:11.421794+0000","last_active":"2023-10-10T18:45:11.424158+0000","last_peered":"2023-10-10T18:45:11.424158+0000","last_clean":"2023-10-10T18:45:11.424158+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T18:45:11.424158+0000","last_undegraded":"2023-10-10T18:45:11.424158+0000","last_fullsized":"2023-10-10T18:45:11.424158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407995+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T18:45:11.407995+0000","last_peered":"2023-10-10T18:45:11.407995+0000","last_clean":"2023-10-10T18:45:11.407995+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T18:45:11.407995+0000","last_undegraded":"2023-10-10T18:45:11.407995+0000","last_fullsized":"2023-10-10T18:45:11.407995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434442+0000","last_change":"2023-10-10T18:45:11.431857+0000","last_active":"2023-10-10T18:45:11.434442+0000","last_peered":"2023-10-10T18:45:11.434442+0000","last_clean":"2023-10-10T18:45:11.434442+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T18:45:11.434442+0000","last_undegraded":"2023-10-10T18:45:11.434442+0000","last_fullsized":"2023-10-10T18:45:11.434442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410980+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T18:45:11.410980+0000","last_peered":"2023-10-10T18:45:11.410980+0000","last_clean":"2023-10-10T18:45:11.410980+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T18:45:11.410980+0000","last_undegraded":"2023-10-10T18:45:11.410980+0000","last_fullsized":"2023-10-10T18:45:11.410980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426196+0000","last_change":"2023-10-10T18:45:11.419774+0000","last_active":"2023-10-10T18:45:11.426196+0000","last_peered":"2023-10-10T18:45:11.426196+0000","last_clean":"2023-10-10T18:45:11.426196+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T18:45:11.426196+0000","last_undegraded":"2023-10-10T18:45:11.426196+0000","last_fullsized":"2023-10-10T18:45:11.426196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405764+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T18:45:11.405764+0000","last_peered":"2023-10-10T18:45:11.405764+0000","last_clean":"2023-10-10T18:45:11.405764+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T18:45:11.405764+0000","last_undegraded":"2023-10-10T18:45:11.405764+0000","last_fullsized":"2023-10-10T18:45:11.405764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424245+0000","last_change":"2023-10-10T18:45:11.420517+0000","last_active":"2023-10-10T18:45:11.424245+0000","last_peered":"2023-10-10T18:45:11.424245+0000","last_clean":"2023-10-10T18:45:11.424245+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T18:45:11.424245+0000","last_undegraded":"2023-10-10T18:45:11.424245+0000","last_fullsized":"2023-10-10T18:45:11.424245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412727+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T18:45:11.412727+0000","last_peered":"2023-10-10T18:45:11.412727+0000","last_clean":"2023-10-10T18:45:11.412727+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T18:45:11.412727+0000","last_undegraded":"2023-10-10T18:45:11.412727+0000","last_fullsized":"2023-10-10T18:45:11.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431834+0000","last_change":"2023-10-10T18:45:11.427109+0000","last_active":"2023-10-10T18:45:11.431834+0000","last_peered":"2023-10-10T18:45:11.431834+0000","last_clean":"2023-10-10T18:45:11.431834+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T18:45:11.431834+0000","last_undegraded":"2023-10-10T18:45:11.431834+0000","last_fullsized":"2023-10-10T18:45:11.431834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410202+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T18:45:11.410202+0000","last_peered":"2023-10-10T18:45:11.410202+0000","last_clean":"2023-10-10T18:45:11.410202+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T18:45:11.410202+0000","last_undegraded":"2023-10-10T18:45:11.410202+0000","last_fullsized":"2023-10-10T18:45:11.410202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430660+0000","last_change":"2023-10-10T18:45:11.416459+0000","last_active":"2023-10-10T18:45:11.430660+0000","last_peered":"2023-10-10T18:45:11.430660+0000","last_clean":"2023-10-10T18:45:11.430660+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T18:45:11.430660+0000","last_undegraded":"2023-10-10T18:45:11.430660+0000","last_fullsized":"2023-10-10T18:45:11.430660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410706+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T18:45:11.410706+0000","last_peered":"2023-10-10T18:45:11.410706+0000","last_clean":"2023-10-10T18:45:11.410706+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T18:45:11.410706+0000","last_undegraded":"2023-10-10T18:45:11.410706+0000","last_fullsized":"2023-10-10T18:45:11.410706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432328+0000","last_change":"2023-10-10T18:45:11.429727+0000","last_active":"2023-10-10T18:45:11.432328+0000","last_peered":"2023-10-10T18:45:11.432328+0000","last_clean":"2023-10-10T18:45:11.432328+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T18:45:11.432328+0000","last_undegraded":"2023-10-10T18:45:11.432328+0000","last_fullsized":"2023-10-10T18:45:11.432328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.422428+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T18:45:11.422428+0000","last_peered":"2023-10-10T18:45:11.422428+0000","last_clean":"2023-10-10T18:45:11.422428+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T18:45:11.422428+0000","last_undegraded":"2023-10-10T18:45:11.422428+0000","last_fullsized":"2023-10-10T18:45:11.422428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431858+0000","last_change":"2023-10-10T18:45:11.427926+0000","last_active":"2023-10-10T18:45:11.431858+0000","last_peered":"2023-10-10T18:45:11.431858+0000","last_clean":"2023-10-10T18:45:11.431858+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T18:45:11.431858+0000","last_undegraded":"2023-10-10T18:45:11.431858+0000","last_fullsized":"2023-10-10T18:45:11.431858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407823+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T18:45:11.407823+0000","last_peered":"2023-10-10T18:45:11.407823+0000","last_clean":"2023-10-10T18:45:11.407823+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T18:45:11.407823+0000","last_undegraded":"2023-10-10T18:45:11.407823+0000","last_fullsized":"2023-10-10T18:45:11.407823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427955+0000","last_change":"2023-10-10T18:45:11.423279+0000","last_active":"2023-10-10T18:45:11.427955+0000","last_peered":"2023-10-10T18:45:11.427955+0000","last_clean":"2023-10-10T18:45:11.427955+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T18:45:11.427955+0000","last_undegraded":"2023-10-10T18:45:11.427955+0000","last_fullsized":"2023-10-10T18:45:11.427955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412668+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T18:45:11.412668+0000","last_peered":"2023-10-10T18:45:11.412668+0000","last_clean":"2023-10-10T18:45:11.412668+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T18:45:11.412668+0000","last_undegraded":"2023-10-10T18:45:11.412668+0000","last_fullsized":"2023-10-10T18:45:11.412668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430728+0000","last_change":"2023-10-10T18:45:11.425339+0000","last_active":"2023-10-10T18:45:11.430728+0000","last_peered":"2023-10-10T18:45:11.430728+0000","last_clean":"2023-10-10T18:45:11.430728+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T18:45:11.430728+0000","last_undegraded":"2023-10-10T18:45:11.430728+0000","last_fullsized":"2023-10-10T18:45:11.430728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421278+0000","last_change":"2023-10-10T18:45:11.418937+0000","last_active":"2023-10-10T18:45:11.421278+0000","last_peered":"2023-10-10T18:45:11.421278+0000","last_clean":"2023-10-10T18:45:11.421278+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T18:45:11.421278+0000","last_undegraded":"2023-10-10T18:45:11.421278+0000","last_fullsized":"2023-10-10T18:45:11.421278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410721+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T18:45:11.410721+0000","last_peered":"2023-10-10T18:45:11.410721+0000","last_clean":"2023-10-10T18:45:11.410721+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T18:45:11.410721+0000","last_undegraded":"2023-10-10T18:45:11.410721+0000","last_fullsized":"2023-10-10T18:45:11.410721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424136+0000","last_change":"2023-10-10T18:45:11.422452+0000","last_active":"2023-10-10T18:45:11.424136+0000","last_peered":"2023-10-10T18:45:11.424136+0000","last_clean":"2023-10-10T18:45:11.424136+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T18:45:11.424136+0000","last_undegraded":"2023-10-10T18:45:11.424136+0000","last_fullsized":"2023-10-10T18:45:11.424136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410662+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T18:45:11.410662+0000","last_peered":"2023-10-10T18:45:11.410662+0000","last_clean":"2023-10-10T18:45:11.410662+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T18:45:11.410662+0000","last_undegraded":"2023-10-10T18:45:11.410662+0000","last_fullsized":"2023-10-10T18:45:11.410662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405467+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T18:45:11.405467+0000","last_peered":"2023-10-10T18:45:11.405467+0000","last_clean":"2023-10-10T18:45:11.405467+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T18:45:11.405467+0000","last_undegraded":"2023-10-10T18:45:11.405467+0000","last_fullsized":"2023-10-10T18:45:11.405467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434404+0000","last_change":"2023-10-10T18:45:11.430233+0000","last_active":"2023-10-10T18:45:11.434404+0000","last_peered":"2023-10-10T18:45:11.434404+0000","last_clean":"2023-10-10T18:45:11.434404+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T18:45:11.434404+0000","last_undegraded":"2023-10-10T18:45:11.434404+0000","last_fullsized":"2023-10-10T18:45:11.434404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405812+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T18:45:11.405812+0000","last_peered":"2023-10-10T18:45:11.405812+0000","last_clean":"2023-10-10T18:45:11.405812+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T18:45:11.405812+0000","last_undegraded":"2023-10-10T18:45:11.405812+0000","last_fullsized":"2023-10-10T18:45:11.405812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.408007+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T18:45:11.408007+0000","last_peered":"2023-10-10T18:45:11.408007+0000","last_clean":"2023-10-10T18:45:11.408007+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T18:45:11.408007+0000","last_undegraded":"2023-10-10T18:45:11.408007+0000","last_fullsized":"2023-10-10T18:45:11.408007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421286+0000","last_change":"2023-10-10T18:45:11.418973+0000","last_active":"2023-10-10T18:45:11.421286+0000","last_peered":"2023-10-10T18:45:11.421286+0000","last_clean":"2023-10-10T18:45:11.421286+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T18:45:11.421286+0000","last_undegraded":"2023-10-10T18:45:11.421286+0000","last_fullsized":"2023-10-10T18:45:11.421286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407798+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T18:45:11.407798+0000","last_peered":"2023-10-10T18:45:11.407798+0000","last_clean":"2023-10-10T18:45:11.407798+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T18:45:11.407798+0000","last_undegraded":"2023-10-10T18:45:11.407798+0000","last_fullsized":"2023-10-10T18:45:11.407798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423727+0000","last_change":"2023-10-10T18:45:11.419846+0000","last_active":"2023-10-10T18:45:11.423727+0000","last_peered":"2023-10-10T18:45:11.423727+0000","last_clean":"2023-10-10T18:45:11.423727+0000","last_became_active":"2023-10-10T18:45:11.419579+0000","last_became_peered":"2023-10-10T18:45:11.419579+0000","last_unstale":"2023-10-10T18:45:11.423727+0000","last_undegraded":"2023-10-10T18:45:11.423727+0000","last_fullsized":"2023-10-10T18:45:11.423727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:58:49.831558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.411120+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T18:45:11.411120+0000","last_peered":"2023-10-10T18:45:11.411120+0000","last_clean":"2023-10-10T18:45:11.411120+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T18:45:11.411120+0000","last_undegraded":"2023-10-10T18:45:11.411120+0000","last_fullsized":"2023-10-10T18:45:11.411120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423663+0000","last_change":"2023-10-10T18:45:11.416217+0000","last_active":"2023-10-10T18:45:11.423663+0000","last_peered":"2023-10-10T18:45:11.423663+0000","last_clean":"2023-10-10T18:45:11.423663+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T18:45:11.423663+0000","last_undegraded":"2023-10-10T18:45:11.423663+0000","last_fullsized":"2023-10-10T18:45:11.423663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405671+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T18:45:11.405671+0000","last_peered":"2023-10-10T18:45:11.405671+0000","last_clean":"2023-10-10T18:45:11.405671+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T18:45:11.405671+0000","last_undegraded":"2023-10-10T18:45:11.405671+0000","last_fullsized":"2023-10-10T18:45:11.405671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431602+0000","last_change":"2023-10-10T18:45:11.427826+0000","last_active":"2023-10-10T18:45:11.431602+0000","last_peered":"2023-10-10T18:45:11.431602+0000","last_clean":"2023-10-10T18:45:11.431602+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T18:45:11.431602+0000","last_undegraded":"2023-10-10T18:45:11.431602+0000","last_fullsized":"2023-10-10T18:45:11.431602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407958+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T18:45:11.407958+0000","last_peered":"2023-10-10T18:45:11.407958+0000","last_clean":"2023-10-10T18:45:11.407958+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T18:45:11.407958+0000","last_undegraded":"2023-10-10T18:45:11.407958+0000","last_fullsized":"2023-10-10T18:45:11.407958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415388+0000","last_change":"2023-10-10T18:45:11.413975+0000","last_active":"2023-10-10T18:45:11.415388+0000","last_peered":"2023-10-10T18:45:11.415388+0000","last_clean":"2023-10-10T18:45:11.415388+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T18:45:11.415388+0000","last_undegraded":"2023-10-10T18:45:11.415388+0000","last_fullsized":"2023-10-10T18:45:11.415388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407755+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T18:45:11.407755+0000","last_peered":"2023-10-10T18:45:11.407755+0000","last_clean":"2023-10-10T18:45:11.407755+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T18:45:11.407755+0000","last_undegraded":"2023-10-10T18:45:11.407755+0000","last_fullsized":"2023-10-10T18:45:11.407755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415058+0000","last_change":"2023-10-10T18:45:11.410768+0000","last_active":"2023-10-10T18:45:11.415058+0000","last_peered":"2023-10-10T18:45:11.415058+0000","last_clean":"2023-10-10T18:45:11.415058+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T18:45:11.415058+0000","last_undegraded":"2023-10-10T18:45:11.415058+0000","last_fullsized":"2023-10-10T18:45:11.415058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407089+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T18:45:11.407089+0000","last_peered":"2023-10-10T18:45:11.407089+0000","last_clean":"2023-10-10T18:45:11.407089+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T18:45:11.407089+0000","last_undegraded":"2023-10-10T18:45:11.407089+0000","last_fullsized":"2023-10-10T18:45:11.407089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430290+0000","last_change":"2023-10-10T18:45:11.425716+0000","last_active":"2023-10-10T18:45:11.430290+0000","last_peered":"2023-10-10T18:45:11.430290+0000","last_clean":"2023-10-10T18:45:11.430290+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T18:45:11.430290+0000","last_undegraded":"2023-10-10T18:45:11.430290+0000","last_fullsized":"2023-10-10T18:45:11.430290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412053+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T18:45:11.412053+0000","last_peered":"2023-10-10T18:45:11.412053+0000","last_clean":"2023-10-10T18:45:11.412053+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T18:45:11.412053+0000","last_undegraded":"2023-10-10T18:45:11.412053+0000","last_fullsized":"2023-10-10T18:45:11.412053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429787+0000","last_change":"2023-10-10T18:45:11.423747+0000","last_active":"2023-10-10T18:45:11.429787+0000","last_peered":"2023-10-10T18:45:11.429787+0000","last_clean":"2023-10-10T18:45:11.429787+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T18:45:11.429787+0000","last_undegraded":"2023-10-10T18:45:11.429787+0000","last_fullsized":"2023-10-10T18:45:11.429787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407940+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T18:45:11.407940+0000","last_peered":"2023-10-10T18:45:11.407940+0000","last_clean":"2023-10-10T18:45:11.407940+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T18:45:11.407940+0000","last_undegraded":"2023-10-10T18:45:11.407940+0000","last_fullsized":"2023-10-10T18:45:11.407940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425816+0000","last_change":"2023-10-10T18:45:11.421872+0000","last_active":"2023-10-10T18:45:11.425816+0000","last_peered":"2023-10-10T18:45:11.425816+0000","last_clean":"2023-10-10T18:45:11.425816+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T18:45:11.425816+0000","last_undegraded":"2023-10-10T18:45:11.425816+0000","last_fullsized":"2023-10-10T18:45:11.425816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412481+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T18:45:11.412481+0000","last_peered":"2023-10-10T18:45:11.412481+0000","last_clean":"2023-10-10T18:45:11.412481+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T18:45:11.412481+0000","last_undegraded":"2023-10-10T18:45:11.412481+0000","last_fullsized":"2023-10-10T18:45:11.412481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429963+0000","last_change":"2023-10-10T18:45:11.425474+0000","last_active":"2023-10-10T18:45:11.429963+0000","last_peered":"2023-10-10T18:45:11.429963+0000","last_clean":"2023-10-10T18:45:11.429963+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T18:45:11.429963+0000","last_undegraded":"2023-10-10T18:45:11.429963+0000","last_fullsized":"2023-10-10T18:45:11.429963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430681+0000","last_change":"2023-10-10T18:45:11.416850+0000","last_active":"2023-10-10T18:45:11.430681+0000","last_peered":"2023-10-10T18:45:11.430681+0000","last_clean":"2023-10-10T18:45:11.430681+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T18:45:11.430681+0000","last_undegraded":"2023-10-10T18:45:11.430681+0000","last_fullsized":"2023-10-10T18:45:11.430681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427108+0000","last_change":"2023-10-10T18:45:11.424440+0000","last_active":"2023-10-10T18:45:11.427108+0000","last_peered":"2023-10-10T18:45:11.427108+0000","last_clean":"2023-10-10T18:45:11.427108+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T18:45:11.427108+0000","last_undegraded":"2023-10-10T18:45:11.427108+0000","last_fullsized":"2023-10-10T18:45:11.427108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.421524+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.420874+0000","last_became_peered":"2023-10-10T18:45:11.420874+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:56:36.851801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419177+0000","last_change":"2023-10-10T18:45:11.412399+0000","last_active":"2023-10-10T18:45:11.419177+0000","last_peered":"2023-10-10T18:45:11.419177+0000","last_clean":"2023-10-10T18:45:11.419177+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T18:45:11.419177+0000","last_undegraded":"2023-10-10T18:45:11.419177+0000","last_fullsized":"2023-10-10T18:45:11.419177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432996+0000","last_change":"2023-10-10T18:45:11.429507+0000","last_active":"2023-10-10T18:45:11.432996+0000","last_peered":"2023-10-10T18:45:11.432996+0000","last_clean":"2023-10-10T18:45:11.432996+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T18:45:11.432996+0000","last_undegraded":"2023-10-10T18:45:11.432996+0000","last_fullsized":"2023-10-10T18:45:11.432996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.418310+0000","last_change":"2023-10-10T18:45:11.413271+0000","last_active":"2023-10-10T18:45:11.418310+0000","last_peered":"2023-10-10T18:45:11.418310+0000","last_clean":"2023-10-10T18:45:11.418310+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T18:45:11.418310+0000","last_undegraded":"2023-10-10T18:45:11.418310+0000","last_fullsized":"2023-10-10T18:45:11.418310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430781+0000","last_change":"2023-10-10T18:45:11.427815+0000","last_active":"2023-10-10T18:45:11.430781+0000","last_peered":"2023-10-10T18:45:11.430781+0000","last_clean":"2023-10-10T18:45:11.430781+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T18:45:11.430781+0000","last_undegraded":"2023-10-10T18:45:11.430781+0000","last_fullsized":"2023-10-10T18:45:11.430781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426972+0000","last_change":"2023-10-10T18:45:11.425458+0000","last_active":"2023-10-10T18:45:11.426972+0000","last_peered":"2023-10-10T18:45:11.426972+0000","last_clean":"2023-10-10T18:45:11.426972+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T18:45:11.426972+0000","last_undegraded":"2023-10-10T18:45:11.426972+0000","last_fullsized":"2023-10-10T18:45:11.426972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424565+0000","last_change":"2023-10-10T18:45:11.422003+0000","last_active":"2023-10-10T18:45:11.424565+0000","last_peered":"2023-10-10T18:45:11.424565+0000","last_clean":"2023-10-10T18:45:11.424565+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T18:45:11.424565+0000","last_undegraded":"2023-10-10T18:45:11.424565+0000","last_fullsized":"2023-10-10T18:45:11.424565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432978+0000","last_change":"2023-10-10T18:45:11.428907+0000","last_active":"2023-10-10T18:45:11.432978+0000","last_peered":"2023-10-10T18:45:11.432978+0000","last_clean":"2023-10-10T18:45:11.432978+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T18:45:11.432978+0000","last_undegraded":"2023-10-10T18:45:11.432978+0000","last_fullsized":"2023-10-10T18:45:11.432978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430538+0000","last_change":"2023-10-10T18:45:11.427241+0000","last_active":"2023-10-10T18:45:11.430538+0000","last_peered":"2023-10-10T18:45:11.430538+0000","last_clean":"2023-10-10T18:45:11.430538+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T18:45:11.430538+0000","last_undegraded":"2023-10-10T18:45:11.430538+0000","last_fullsized":"2023-10-10T18:45:11.430538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427471+0000","last_change":"2023-10-10T18:45:11.426813+0000","last_active":"2023-10-10T18:45:11.427471+0000","last_peered":"2023-10-10T18:45:11.427471+0000","last_clean":"2023-10-10T18:45:11.427471+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T18:45:11.427471+0000","last_undegraded":"2023-10-10T18:45:11.427471+0000","last_fullsized":"2023-10-10T18:45:11.427471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432098+0000","last_change":"2023-10-10T18:45:11.429582+0000","last_active":"2023-10-10T18:45:11.432098+0000","last_peered":"2023-10-10T18:45:11.432098+0000","last_clean":"2023-10-10T18:45:11.432098+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T18:45:11.432098+0000","last_undegraded":"2023-10-10T18:45:11.432098+0000","last_fullsized":"2023-10-10T18:45:11.432098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":282624,"data_stored":665681,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":282624,"data_stored":665681,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T18:45:17.492 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T18:45:17.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.635+0000 7fe44e940700 1 Processor -- start 2023-10-10T18:45:17.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.635+0000 7fe44e940700 1 -- start start 2023-10-10T18:45:17.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe44e940700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe4480b1fc0 0x7fe448155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe44e940700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe4480b2390 0x7fe448068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe44e940700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 0x7fe4480aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe44e940700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe448150770 con 0x7fe4480b1fc0 2023-10-10T18:45:17.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe44e940700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4481574b0 con 0x7fe4480b2390 2023-10-10T18:45:17.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe44e940700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480afad0 con 0x7fe448068bd0 2023-10-10T18:45:17.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe4480b1fc0 0x7fe448155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe4480b1fc0 0x7fe448155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52222/0 (socket says 172.21.15.175:52222) 2023-10-10T18:45:17.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 -- 172.21.15.175:0/2292079746 learned_addr learned my addr 172.21.15.175:0/2292079746 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:17.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe4477fe700 1 --2- 172.21.15.175:0/2292079746 >> v2:172.21.15.191:3300/0 conn(0x7fe4480b2390 0x7fe448068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe44cedd700 1 --2- 172.21.15.175:0/2292079746 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 0x7fe4480aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 -- 172.21.15.175:0/2292079746 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 msgr2=0x7fe4480aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:17.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 --2- 172.21.15.175:0/2292079746 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 0x7fe4480aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 -- 172.21.15.175:0/2292079746 >> v2:172.21.15.191:3300/0 conn(0x7fe4480b2390 msgr2=0x7fe448068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:17.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 --2- 172.21.15.175:0/2292079746 >> v2:172.21.15.191:3300/0 conn(0x7fe4480b2390 0x7fe448068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 -- 172.21.15.175:0/2292079746 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4480af4c0 con 0x7fe4480b1fc0 2023-10-10T18:45:17.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe447fff700 1 --2- 172.21.15.175:0/2292079746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b1fc0 0x7fe448155fb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe4480c0290 tx=0x7fe43c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5764b433cd636 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:17.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe446ffd700 1 -- 172.21.15.175:0/2292079746 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00b8f0 con 0x7fe4480b1fc0 2023-10-10T18:45:17.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe446ffd700 1 -- 172.21.15.175:0/2292079746 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43c00ba50 con 0x7fe4480b1fc0 2023-10-10T18:45:17.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.636+0000 7fe446ffd700 1 -- 172.21.15.175:0/2292079746 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00bdf0 con 0x7fe4480b1fc0 2023-10-10T18:45:17.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 -- 172.21.15.175:0/2292079746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b1fc0 msgr2=0x7fe448155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 --2- 172.21.15.175:0/2292079746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b1fc0 0x7fe448155fb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe4480c0290 tx=0x7fe43c009250 comp rx=0 tx=0).stop 2023-10-10T18:45:17.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 -- 172.21.15.175:0/2292079746 shutdown_connections 2023-10-10T18:45:17.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 --2- 172.21.15.175:0/2292079746 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 0x7fe4480aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 --2- 172.21.15.175:0/2292079746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b1fc0 0x7fe448155fb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 --2- 172.21.15.175:0/2292079746 >> v2:172.21.15.191:3300/0 conn(0x7fe4480b2390 0x7fe448068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 -- 172.21.15.175:0/2292079746 >> 172.21.15.175:0/2292079746 conn(0x7fe448063230 msgr2=0x7fe448063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:17.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 -- 172.21.15.175:0/2292079746 shutdown_connections 2023-10-10T18:45:17.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.637+0000 7fe44e940700 1 -- 172.21.15.175:0/2292079746 wait complete. 2023-10-10T18:45:17.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.638+0000 7fe44e940700 1 Processor -- start 2023-10-10T18:45:17.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.638+0000 7fe44e940700 1 -- start start 2023-10-10T18:45:17.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44e940700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 0x7fe4480ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44e940700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4480b1fc0 0x7fe448154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44e940700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b2390 0x7fe448155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44e940700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe448166d50 con 0x7fe4480b2390 2023-10-10T18:45:17.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44e940700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe448166160 con 0x7fe4480b1fc0 2023-10-10T18:45:17.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44e940700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480cd3f0 con 0x7fe448068bd0 2023-10-10T18:45:17.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b2390 0x7fe448155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b2390 0x7fe448155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52234/0 (socket says 172.21.15.175:52234) 2023-10-10T18:45:17.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 -- 172.21.15.175:0/1420305706 learned_addr learned my addr 172.21.15.175:0/1420305706 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:17.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe447fff700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 0x7fe4480ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe4477fe700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4480b1fc0 0x7fe448154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 -- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 msgr2=0x7fe4480ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 0x7fe4480ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 -- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4480b1fc0 msgr2=0x7fe448154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4480b1fc0 0x7fe448154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 -- 172.21.15.175:0/1420305706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c0094f0 con 0x7fe4480b2390 2023-10-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe44cedd700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b2390 0x7fe448155a80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe440007f20 tx=0x7fe4400098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.639+0000 7fe42ffff700 1 -- 172.21.15.175:0/1420305706 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44000eb90 con 0x7fe4480b2390 2023-10-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.640+0000 7fe42ffff700 1 -- 172.21.15.175:0/1420305706 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44000ed10 con 0x7fe4480b2390 2023-10-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.640+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe448166420 con 0x7fe4480b2390 2023-10-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.640+0000 7fe42ffff700 1 -- 172.21.15.175:0/1420305706 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440014670 con 0x7fe4480b2390 2023-10-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.640+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4481669e0 con 0x7fe4480b2390 2023-10-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.640+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe448067570 con 0x7fe4480b2390 2023-10-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.641+0000 7fe42ffff700 1 -- 172.21.15.175:0/1420305706 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe440014a60 con 0x7fe4480b2390 2023-10-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.641+0000 7fe42ffff700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe434066f30 0x7fe4340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.641+0000 7fe42ffff700 1 -- 172.21.15.175:0/1420305706 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe440084c00 con 0x7fe4480b2390 2023-10-10T18:45:17.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.641+0000 7fe447fff700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe434066f30 0x7fe4340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:17.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.642+0000 7fe447fff700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe434066f30 0x7fe4340693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe4480c0290 tx=0x7fe43c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:17.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.645+0000 7fe42ffff700 1 -- 172.21.15.175:0/1420305706 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe440051dd0 con 0x7fe4480b2390 2023-10-10T18:45:17.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.823+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe448069060 con 0x7fe434066f30 2023-10-10T18:45:17.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.831+0000 7fe42ffff700 1 -- 172.21.15.175:0/1420305706 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264601 (secure 0 0 0) 0x7fe448069060 con 0x7fe434066f30 2023-10-10T18:45:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:17.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.834+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe434066f30 msgr2=0x7fe4340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.834+0000 7fe44e940700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe434066f30 0x7fe4340693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe4480c0290 tx=0x7fe43c017000 comp rx=0 tx=0).stop 2023-10-10T18:45:17.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.834+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b2390 msgr2=0x7fe448155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:17.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.834+0000 7fe44e940700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b2390 0x7fe448155a80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe440007f20 tx=0x7fe4400098a0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.835+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 shutdown_connections 2023-10-10T18:45:17.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.835+0000 7fe44e940700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe434066f30 0x7fe4340693e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.835+0000 7fe44e940700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4480b2390 0x7fe448155a80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.835+0000 7fe44e940700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe448068bd0 0x7fe4480ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.835+0000 7fe44e940700 1 --2- 172.21.15.175:0/1420305706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4480b1fc0 0x7fe448154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:17.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.835+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 >> 172.21.15.175:0/1420305706 conn(0x7fe448063230 msgr2=0x7fe44814fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:17.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.835+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 shutdown_connections 2023-10-10T18:45:17.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:17.835+0000 7fe44e940700 1 -- 172.21.15.175:0/1420305706 wait complete. 2023-10-10T18:45:17.840 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T18:45:17.847 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-10-10T18:45:17.131080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2380128,"kb_used_data":1824,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752594592,"statfs":{"total":773094113280,"available":770656862208,"internally_reserved":0,"allocated":1867776,"data_stored":1783240,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.742298"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429926+0000","last_change":"2023-10-10T18:45:11.419184+0000","last_active":"2023-10-10T18:45:11.429926+0000","last_peered":"2023-10-10T18:45:11.429926+0000","last_clean":"2023-10-10T18:45:11.429926+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T18:45:11.429926+0000","last_undegraded":"2023-10-10T18:45:11.429926+0000","last_fullsized":"2023-10-10T18:45:11.429926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420684+0000","last_change":"2023-10-10T18:45:11.414586+0000","last_active":"2023-10-10T18:45:11.420684+0000","last_peered":"2023-10-10T18:45:11.420684+0000","last_clean":"2023-10-10T18:45:11.420684+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T18:45:11.420684+0000","last_undegraded":"2023-10-10T18:45:11.420684+0000","last_fullsized":"2023-10-10T18:45:11.420684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424203+0000","last_change":"2023-10-10T18:45:11.419119+0000","last_active":"2023-10-10T18:45:11.424203+0000","last_peered":"2023-10-10T18:45:11.424203+0000","last_clean":"2023-10-10T18:45:11.424203+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T18:45:11.424203+0000","last_undegraded":"2023-10-10T18:45:11.424203+0000","last_fullsized":"2023-10-10T18:45:11.424203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425796+0000","last_change":"2023-10-10T18:45:11.421351+0000","last_active":"2023-10-10T18:45:11.425796+0000","last_peered":"2023-10-10T18:45:11.425796+0000","last_clean":"2023-10-10T18:45:11.425796+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T18:45:11.425796+0000","last_undegraded":"2023-10-10T18:45:11.425796+0000","last_fullsized":"2023-10-10T18:45:11.425796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425614+0000","last_change":"2023-10-10T18:45:11.421122+0000","last_active":"2023-10-10T18:45:11.425614+0000","last_peered":"2023-10-10T18:45:11.425614+0000","last_clean":"2023-10-10T18:45:11.425614+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T18:45:11.425614+0000","last_undegraded":"2023-10-10T18:45:11.425614+0000","last_fullsized":"2023-10-10T18:45:11.425614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424542+0000","last_change":"2023-10-10T18:45:11.421729+0000","last_active":"2023-10-10T18:45:11.424542+0000","last_peered":"2023-10-10T18:45:11.424542+0000","last_clean":"2023-10-10T18:45:11.424542+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T18:45:11.424542+0000","last_undegraded":"2023-10-10T18:45:11.424542+0000","last_fullsized":"2023-10-10T18:45:11.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426178+0000","last_change":"2023-10-10T18:45:11.419306+0000","last_active":"2023-10-10T18:45:11.426178+0000","last_peered":"2023-10-10T18:45:11.426178+0000","last_clean":"2023-10-10T18:45:11.426178+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T18:45:11.426178+0000","last_undegraded":"2023-10-10T18:45:11.426178+0000","last_fullsized":"2023-10-10T18:45:11.426178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.422940+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429946+0000","last_change":"2023-10-10T18:45:11.422612+0000","last_active":"2023-10-10T18:45:11.429946+0000","last_peered":"2023-10-10T18:45:11.429946+0000","last_clean":"2023-10-10T18:45:11.429946+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T18:45:11.429946+0000","last_undegraded":"2023-10-10T18:45:11.429946+0000","last_fullsized":"2023-10-10T18:45:11.429946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430763+0000","last_change":"2023-10-10T18:45:11.425754+0000","last_active":"2023-10-10T18:45:11.430763+0000","last_peered":"2023-10-10T18:45:11.430763+0000","last_clean":"2023-10-10T18:45:11.430763+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T18:45:11.430763+0000","last_undegraded":"2023-10-10T18:45:11.430763+0000","last_fullsized":"2023-10-10T18:45:11.430763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430664+0000","last_change":"2023-10-10T18:45:11.425891+0000","last_active":"2023-10-10T18:45:11.430664+0000","last_peered":"2023-10-10T18:45:11.430664+0000","last_clean":"2023-10-10T18:45:11.430664+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T18:45:11.430664+0000","last_undegraded":"2023-10-10T18:45:11.430664+0000","last_fullsized":"2023-10-10T18:45:11.430664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431015+0000","last_change":"2023-10-10T18:45:11.424834+0000","last_active":"2023-10-10T18:45:11.431015+0000","last_peered":"2023-10-10T18:45:11.431015+0000","last_clean":"2023-10-10T18:45:11.431015+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T18:45:11.431015+0000","last_undegraded":"2023-10-10T18:45:11.431015+0000","last_fullsized":"2023-10-10T18:45:11.431015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421085+0000","last_change":"2023-10-10T18:45:11.417796+0000","last_active":"2023-10-10T18:45:11.421085+0000","last_peered":"2023-10-10T18:45:11.421085+0000","last_clean":"2023-10-10T18:45:11.421085+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T18:45:11.421085+0000","last_undegraded":"2023-10-10T18:45:11.421085+0000","last_fullsized":"2023-10-10T18:45:11.421085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419270+0000","last_change":"2023-10-10T18:45:11.414690+0000","last_active":"2023-10-10T18:45:11.419270+0000","last_peered":"2023-10-10T18:45:11.419270+0000","last_clean":"2023-10-10T18:45:11.419270+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T18:45:11.419270+0000","last_undegraded":"2023-10-10T18:45:11.419270+0000","last_fullsized":"2023-10-10T18:45:11.419270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434526+0000","last_change":"2023-10-10T18:45:11.429433+0000","last_active":"2023-10-10T18:45:11.434526+0000","last_peered":"2023-10-10T18:45:11.434526+0000","last_clean":"2023-10-10T18:45:11.434526+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T18:45:11.434526+0000","last_undegraded":"2023-10-10T18:45:11.434526+0000","last_fullsized":"2023-10-10T18:45:11.434526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427193+0000","last_change":"2023-10-10T18:45:11.422935+0000","last_active":"2023-10-10T18:45:11.427193+0000","last_peered":"2023-10-10T18:45:11.427193+0000","last_clean":"2023-10-10T18:45:11.427193+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T18:45:11.427193+0000","last_undegraded":"2023-10-10T18:45:11.427193+0000","last_fullsized":"2023-10-10T18:45:11.427193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434131+0000","last_change":"2023-10-10T18:45:11.431202+0000","last_active":"2023-10-10T18:45:11.434131+0000","last_peered":"2023-10-10T18:45:11.434131+0000","last_clean":"2023-10-10T18:45:11.434131+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T18:45:11.434131+0000","last_undegraded":"2023-10-10T18:45:11.434131+0000","last_fullsized":"2023-10-10T18:45:11.434131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427139+0000","last_change":"2023-10-10T18:45:11.424885+0000","last_active":"2023-10-10T18:45:11.427139+0000","last_peered":"2023-10-10T18:45:11.427139+0000","last_clean":"2023-10-10T18:45:11.427139+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T18:45:11.427139+0000","last_undegraded":"2023-10-10T18:45:11.427139+0000","last_fullsized":"2023-10-10T18:45:11.427139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426206+0000","last_change":"2023-10-10T18:45:11.421039+0000","last_active":"2023-10-10T18:45:11.426206+0000","last_peered":"2023-10-10T18:45:11.426206+0000","last_clean":"2023-10-10T18:45:11.426206+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T18:45:11.426206+0000","last_undegraded":"2023-10-10T18:45:11.426206+0000","last_fullsized":"2023-10-10T18:45:11.426206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405851+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T18:45:11.405851+0000","last_peered":"2023-10-10T18:45:11.405851+0000","last_clean":"2023-10-10T18:45:11.405851+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T18:45:11.405851+0000","last_undegraded":"2023-10-10T18:45:11.405851+0000","last_fullsized":"2023-10-10T18:45:11.405851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421043+0000","last_change":"2023-10-10T18:45:11.417504+0000","last_active":"2023-10-10T18:45:11.421043+0000","last_peered":"2023-10-10T18:45:11.421043+0000","last_clean":"2023-10-10T18:45:11.421043+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T18:45:11.421043+0000","last_undegraded":"2023-10-10T18:45:11.421043+0000","last_fullsized":"2023-10-10T18:45:11.421043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.402480+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T18:45:11.402480+0000","last_peered":"2023-10-10T18:45:11.402480+0000","last_clean":"2023-10-10T18:45:11.402480+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T18:45:11.402480+0000","last_undegraded":"2023-10-10T18:45:11.402480+0000","last_fullsized":"2023-10-10T18:45:11.402480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424178+0000","last_change":"2023-10-10T18:45:11.421896+0000","last_active":"2023-10-10T18:45:11.424178+0000","last_peered":"2023-10-10T18:45:11.424178+0000","last_clean":"2023-10-10T18:45:11.424178+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T18:45:11.424178+0000","last_undegraded":"2023-10-10T18:45:11.424178+0000","last_fullsized":"2023-10-10T18:45:11.424178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.408064+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T18:45:11.408064+0000","last_peered":"2023-10-10T18:45:11.408064+0000","last_clean":"2023-10-10T18:45:11.408064+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T18:45:11.408064+0000","last_undegraded":"2023-10-10T18:45:11.408064+0000","last_fullsized":"2023-10-10T18:45:11.408064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432466+0000","last_change":"2023-10-10T18:45:11.429680+0000","last_active":"2023-10-10T18:45:11.432466+0000","last_peered":"2023-10-10T18:45:11.432466+0000","last_clean":"2023-10-10T18:45:11.432466+0000","last_became_active":"2023-10-10T18:45:11.429397+0000","last_became_peered":"2023-10-10T18:45:11.429397+0000","last_unstale":"2023-10-10T18:45:11.432466+0000","last_undegraded":"2023-10-10T18:45:11.432466+0000","last_fullsized":"2023-10-10T18:45:11.432466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:19:52.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.406417+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T18:45:11.406417+0000","last_peered":"2023-10-10T18:45:11.406417+0000","last_clean":"2023-10-10T18:45:11.406417+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T18:45:11.406417+0000","last_undegraded":"2023-10-10T18:45:11.406417+0000","last_fullsized":"2023-10-10T18:45:11.406417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432439+0000","last_change":"2023-10-10T18:45:11.429408+0000","last_active":"2023-10-10T18:45:11.432439+0000","last_peered":"2023-10-10T18:45:11.432439+0000","last_clean":"2023-10-10T18:45:11.432439+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T18:45:11.432439+0000","last_undegraded":"2023-10-10T18:45:11.432439+0000","last_fullsized":"2023-10-10T18:45:11.432439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.401851+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T18:45:11.401851+0000","last_peered":"2023-10-10T18:45:11.401851+0000","last_clean":"2023-10-10T18:45:11.401851+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T18:45:11.401851+0000","last_undegraded":"2023-10-10T18:45:11.401851+0000","last_fullsized":"2023-10-10T18:45:11.401851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430579+0000","last_change":"2023-10-10T18:45:11.426051+0000","last_active":"2023-10-10T18:45:11.430579+0000","last_peered":"2023-10-10T18:45:11.430579+0000","last_clean":"2023-10-10T18:45:11.430579+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T18:45:11.430579+0000","last_undegraded":"2023-10-10T18:45:11.430579+0000","last_fullsized":"2023-10-10T18:45:11.430579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410880+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T18:45:11.410880+0000","last_peered":"2023-10-10T18:45:11.410880+0000","last_clean":"2023-10-10T18:45:11.410880+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T18:45:11.410880+0000","last_undegraded":"2023-10-10T18:45:11.410880+0000","last_fullsized":"2023-10-10T18:45:11.410880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427223+0000","last_change":"2023-10-10T18:45:11.425183+0000","last_active":"2023-10-10T18:45:11.427223+0000","last_peered":"2023-10-10T18:45:11.427223+0000","last_clean":"2023-10-10T18:45:11.427223+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T18:45:11.427223+0000","last_undegraded":"2023-10-10T18:45:11.427223+0000","last_fullsized":"2023-10-10T18:45:11.427223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.402044+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T18:45:11.402044+0000","last_peered":"2023-10-10T18:45:11.402044+0000","last_clean":"2023-10-10T18:45:11.402044+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T18:45:11.402044+0000","last_undegraded":"2023-10-10T18:45:11.402044+0000","last_fullsized":"2023-10-10T18:45:11.402044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415249+0000","last_change":"2023-10-10T18:45:11.412641+0000","last_active":"2023-10-10T18:45:11.415249+0000","last_peered":"2023-10-10T18:45:11.415249+0000","last_clean":"2023-10-10T18:45:11.415249+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T18:45:11.415249+0000","last_undegraded":"2023-10-10T18:45:11.415249+0000","last_fullsized":"2023-10-10T18:45:11.415249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.413032+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T18:45:11.413032+0000","last_peered":"2023-10-10T18:45:11.413032+0000","last_clean":"2023-10-10T18:45:11.413032+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T18:45:11.413032+0000","last_undegraded":"2023-10-10T18:45:11.413032+0000","last_fullsized":"2023-10-10T18:45:11.413032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420638+0000","last_change":"2023-10-10T18:45:11.413819+0000","last_active":"2023-10-10T18:45:11.420638+0000","last_peered":"2023-10-10T18:45:11.420638+0000","last_clean":"2023-10-10T18:45:11.420638+0000","last_became_active":"2023-10-10T18:45:11.413529+0000","last_became_peered":"2023-10-10T18:45:11.413529+0000","last_unstale":"2023-10-10T18:45:11.420638+0000","last_undegraded":"2023-10-10T18:45:11.420638+0000","last_fullsized":"2023-10-10T18:45:11.420638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.411211+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T18:45:11.411211+0000","last_peered":"2023-10-10T18:45:11.411211+0000","last_clean":"2023-10-10T18:45:11.411211+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T18:45:11.411211+0000","last_undegraded":"2023-10-10T18:45:11.411211+0000","last_fullsized":"2023-10-10T18:45:11.411211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430421+0000","last_change":"2023-10-10T18:45:11.427116+0000","last_active":"2023-10-10T18:45:11.430421+0000","last_peered":"2023-10-10T18:45:11.430421+0000","last_clean":"2023-10-10T18:45:11.430421+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T18:45:11.430421+0000","last_undegraded":"2023-10-10T18:45:11.430421+0000","last_fullsized":"2023-10-10T18:45:11.430421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.416463+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T18:45:11.416463+0000","last_peered":"2023-10-10T18:45:11.416463+0000","last_clean":"2023-10-10T18:45:11.416463+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T18:45:11.416463+0000","last_undegraded":"2023-10-10T18:45:11.416463+0000","last_fullsized":"2023-10-10T18:45:11.416463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424158+0000","last_change":"2023-10-10T18:45:11.421794+0000","last_active":"2023-10-10T18:45:11.424158+0000","last_peered":"2023-10-10T18:45:11.424158+0000","last_clean":"2023-10-10T18:45:11.424158+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T18:45:11.424158+0000","last_undegraded":"2023-10-10T18:45:11.424158+0000","last_fullsized":"2023-10-10T18:45:11.424158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407995+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T18:45:11.407995+0000","last_peered":"2023-10-10T18:45:11.407995+0000","last_clean":"2023-10-10T18:45:11.407995+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T18:45:11.407995+0000","last_undegraded":"2023-10-10T18:45:11.407995+0000","last_fullsized":"2023-10-10T18:45:11.407995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434442+0000","last_change":"2023-10-10T18:45:11.431857+0000","last_active":"2023-10-10T18:45:11.434442+0000","last_peered":"2023-10-10T18:45:11.434442+0000","last_clean":"2023-10-10T18:45:11.434442+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T18:45:11.434442+0000","last_undegraded":"2023-10-10T18:45:11.434442+0000","last_fullsized":"2023-10-10T18:45:11.434442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410980+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T18:45:11.410980+0000","last_peered":"2023-10-10T18:45:11.410980+0000","last_clean":"2023-10-10T18:45:11.410980+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T18:45:11.410980+0000","last_undegraded":"2023-10-10T18:45:11.410980+0000","last_fullsized":"2023-10-10T18:45:11.410980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426196+0000","last_change":"2023-10-10T18:45:11.419774+0000","last_active":"2023-10-10T18:45:11.426196+0000","last_peered":"2023-10-10T18:45:11.426196+0000","last_clean":"2023-10-10T18:45:11.426196+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T18:45:11.426196+0000","last_undegraded":"2023-10-10T18:45:11.426196+0000","last_fullsized":"2023-10-10T18:45:11.426196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405764+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T18:45:11.405764+0000","last_peered":"2023-10-10T18:45:11.405764+0000","last_clean":"2023-10-10T18:45:11.405764+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T18:45:11.405764+0000","last_undegraded":"2023-10-10T18:45:11.405764+0000","last_fullsized":"2023-10-10T18:45:11.405764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424245+0000","last_change":"2023-10-10T18:45:11.420517+0000","last_active":"2023-10-10T18:45:11.424245+0000","last_peered":"2023-10-10T18:45:11.424245+0000","last_clean":"2023-10-10T18:45:11.424245+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T18:45:11.424245+0000","last_undegraded":"2023-10-10T18:45:11.424245+0000","last_fullsized":"2023-10-10T18:45:11.424245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412727+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T18:45:11.412727+0000","last_peered":"2023-10-10T18:45:11.412727+0000","last_clean":"2023-10-10T18:45:11.412727+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T18:45:11.412727+0000","last_undegraded":"2023-10-10T18:45:11.412727+0000","last_fullsized":"2023-10-10T18:45:11.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431834+0000","last_change":"2023-10-10T18:45:11.427109+0000","last_active":"2023-10-10T18:45:11.431834+0000","last_peered":"2023-10-10T18:45:11.431834+0000","last_clean":"2023-10-10T18:45:11.431834+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T18:45:11.431834+0000","last_undegraded":"2023-10-10T18:45:11.431834+0000","last_fullsized":"2023-10-10T18:45:11.431834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410202+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T18:45:11.410202+0000","last_peered":"2023-10-10T18:45:11.410202+0000","last_clean":"2023-10-10T18:45:11.410202+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T18:45:11.410202+0000","last_undegraded":"2023-10-10T18:45:11.410202+0000","last_fullsized":"2023-10-10T18:45:11.410202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430660+0000","last_change":"2023-10-10T18:45:11.416459+0000","last_active":"2023-10-10T18:45:11.430660+0000","last_peered":"2023-10-10T18:45:11.430660+0000","last_clean":"2023-10-10T18:45:11.430660+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T18:45:11.430660+0000","last_undegraded":"2023-10-10T18:45:11.430660+0000","last_fullsized":"2023-10-10T18:45:11.430660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410706+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T18:45:11.410706+0000","last_peered":"2023-10-10T18:45:11.410706+0000","last_clean":"2023-10-10T18:45:11.410706+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T18:45:11.410706+0000","last_undegraded":"2023-10-10T18:45:11.410706+0000","last_fullsized":"2023-10-10T18:45:11.410706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432328+0000","last_change":"2023-10-10T18:45:11.429727+0000","last_active":"2023-10-10T18:45:11.432328+0000","last_peered":"2023-10-10T18:45:11.432328+0000","last_clean":"2023-10-10T18:45:11.432328+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T18:45:11.432328+0000","last_undegraded":"2023-10-10T18:45:11.432328+0000","last_fullsized":"2023-10-10T18:45:11.432328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.422428+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T18:45:11.422428+0000","last_peered":"2023-10-10T18:45:11.422428+0000","last_clean":"2023-10-10T18:45:11.422428+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T18:45:11.422428+0000","last_undegraded":"2023-10-10T18:45:11.422428+0000","last_fullsized":"2023-10-10T18:45:11.422428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431858+0000","last_change":"2023-10-10T18:45:11.427926+0000","last_active":"2023-10-10T18:45:11.431858+0000","last_peered":"2023-10-10T18:45:11.431858+0000","last_clean":"2023-10-10T18:45:11.431858+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T18:45:11.431858+0000","last_undegraded":"2023-10-10T18:45:11.431858+0000","last_fullsized":"2023-10-10T18:45:11.431858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407823+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T18:45:11.407823+0000","last_peered":"2023-10-10T18:45:11.407823+0000","last_clean":"2023-10-10T18:45:11.407823+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T18:45:11.407823+0000","last_undegraded":"2023-10-10T18:45:11.407823+0000","last_fullsized":"2023-10-10T18:45:11.407823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427955+0000","last_change":"2023-10-10T18:45:11.423279+0000","last_active":"2023-10-10T18:45:11.427955+0000","last_peered":"2023-10-10T18:45:11.427955+0000","last_clean":"2023-10-10T18:45:11.427955+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T18:45:11.427955+0000","last_undegraded":"2023-10-10T18:45:11.427955+0000","last_fullsized":"2023-10-10T18:45:11.427955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412668+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T18:45:11.412668+0000","last_peered":"2023-10-10T18:45:11.412668+0000","last_clean":"2023-10-10T18:45:11.412668+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T18:45:11.412668+0000","last_undegraded":"2023-10-10T18:45:11.412668+0000","last_fullsized":"2023-10-10T18:45:11.412668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430728+0000","last_change":"2023-10-10T18:45:11.425339+0000","last_active":"2023-10-10T18:45:11.430728+0000","last_peered":"2023-10-10T18:45:11.430728+0000","last_clean":"2023-10-10T18:45:11.430728+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T18:45:11.430728+0000","last_undegraded":"2023-10-10T18:45:11.430728+0000","last_fullsized":"2023-10-10T18:45:11.430728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421278+0000","last_change":"2023-10-10T18:45:11.418937+0000","last_active":"2023-10-10T18:45:11.421278+0000","last_peered":"2023-10-10T18:45:11.421278+0000","last_clean":"2023-10-10T18:45:11.421278+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T18:45:11.421278+0000","last_undegraded":"2023-10-10T18:45:11.421278+0000","last_fullsized":"2023-10-10T18:45:11.421278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410721+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T18:45:11.410721+0000","last_peered":"2023-10-10T18:45:11.410721+0000","last_clean":"2023-10-10T18:45:11.410721+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T18:45:11.410721+0000","last_undegraded":"2023-10-10T18:45:11.410721+0000","last_fullsized":"2023-10-10T18:45:11.410721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424136+0000","last_change":"2023-10-10T18:45:11.422452+0000","last_active":"2023-10-10T18:45:11.424136+0000","last_peered":"2023-10-10T18:45:11.424136+0000","last_clean":"2023-10-10T18:45:11.424136+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T18:45:11.424136+0000","last_undegraded":"2023-10-10T18:45:11.424136+0000","last_fullsized":"2023-10-10T18:45:11.424136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410662+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T18:45:11.410662+0000","last_peered":"2023-10-10T18:45:11.410662+0000","last_clean":"2023-10-10T18:45:11.410662+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T18:45:11.410662+0000","last_undegraded":"2023-10-10T18:45:11.410662+0000","last_fullsized":"2023-10-10T18:45:11.410662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405467+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T18:45:11.405467+0000","last_peered":"2023-10-10T18:45:11.405467+0000","last_clean":"2023-10-10T18:45:11.405467+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T18:45:11.405467+0000","last_undegraded":"2023-10-10T18:45:11.405467+0000","last_fullsized":"2023-10-10T18:45:11.405467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434404+0000","last_change":"2023-10-10T18:45:11.430233+0000","last_active":"2023-10-10T18:45:11.434404+0000","last_peered":"2023-10-10T18:45:11.434404+0000","last_clean":"2023-10-10T18:45:11.434404+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T18:45:11.434404+0000","last_undegraded":"2023-10-10T18:45:11.434404+0000","last_fullsized":"2023-10-10T18:45:11.434404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405812+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T18:45:11.405812+0000","last_peered":"2023-10-10T18:45:11.405812+0000","last_clean":"2023-10-10T18:45:11.405812+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T18:45:11.405812+0000","last_undegraded":"2023-10-10T18:45:11.405812+0000","last_fullsized":"2023-10-10T18:45:11.405812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.408007+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T18:45:11.408007+0000","last_peered":"2023-10-10T18:45:11.408007+0000","last_clean":"2023-10-10T18:45:11.408007+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T18:45:11.408007+0000","last_undegraded":"2023-10-10T18:45:11.408007+0000","last_fullsized":"2023-10-10T18:45:11.408007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421286+0000","last_change":"2023-10-10T18:45:11.418973+0000","last_active":"2023-10-10T18:45:11.421286+0000","last_peered":"2023-10-10T18:45:11.421286+0000","last_clean":"2023-10-10T18:45:11.421286+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T18:45:11.421286+0000","last_undegraded":"2023-10-10T18:45:11.421286+0000","last_fullsized":"2023-10-10T18:45:11.421286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407798+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T18:45:11.407798+0000","last_peered":"2023-10-10T18:45:11.407798+0000","last_clean":"2023-10-10T18:45:11.407798+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T18:45:11.407798+0000","last_undegraded":"2023-10-10T18:45:11.407798+0000","last_fullsized":"2023-10-10T18:45:11.407798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423727+0000","last_change":"2023-10-10T18:45:11.419846+0000","last_active":"2023-10-10T18:45:11.423727+0000","last_peered":"2023-10-10T18:45:11.423727+0000","last_clean":"2023-10-10T18:45:11.423727+0000","last_became_active":"2023-10-10T18:45:11.419579+0000","last_became_peered":"2023-10-10T18:45:11.419579+0000","last_unstale":"2023-10-10T18:45:11.423727+0000","last_undegraded":"2023-10-10T18:45:11.423727+0000","last_fullsized":"2023-10-10T18:45:11.423727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:58:49.831558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.411120+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T18:45:11.411120+0000","last_peered":"2023-10-10T18:45:11.411120+0000","last_clean":"2023-10-10T18:45:11.411120+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T18:45:11.411120+0000","last_undegraded":"2023-10-10T18:45:11.411120+0000","last_fullsized":"2023-10-10T18:45:11.411120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423663+0000","last_change":"2023-10-10T18:45:11.416217+0000","last_active":"2023-10-10T18:45:11.423663+0000","last_peered":"2023-10-10T18:45:11.423663+0000","last_clean":"2023-10-10T18:45:11.423663+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T18:45:11.423663+0000","last_undegraded":"2023-10-10T18:45:11.423663+0000","last_fullsized":"2023-10-10T18:45:11.423663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405671+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T18:45:11.405671+0000","last_peered":"2023-10-10T18:45:11.405671+0000","last_clean":"2023-10-10T18:45:11.405671+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T18:45:11.405671+0000","last_undegraded":"2023-10-10T18:45:11.405671+0000","last_fullsized":"2023-10-10T18:45:11.405671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431602+0000","last_change":"2023-10-10T18:45:11.427826+0000","last_active":"2023-10-10T18:45:11.431602+0000","last_peered":"2023-10-10T18:45:11.431602+0000","last_clean":"2023-10-10T18:45:11.431602+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T18:45:11.431602+0000","last_undegraded":"2023-10-10T18:45:11.431602+0000","last_fullsized":"2023-10-10T18:45:11.431602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407958+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T18:45:11.407958+0000","last_peered":"2023-10-10T18:45:11.407958+0000","last_clean":"2023-10-10T18:45:11.407958+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T18:45:11.407958+0000","last_undegraded":"2023-10-10T18:45:11.407958+0000","last_fullsized":"2023-10-10T18:45:11.407958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415388+0000","last_change":"2023-10-10T18:45:11.413975+0000","last_active":"2023-10-10T18:45:11.415388+0000","last_peered":"2023-10-10T18:45:11.415388+0000","last_clean":"2023-10-10T18:45:11.415388+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T18:45:11.415388+0000","last_undegraded":"2023-10-10T18:45:11.415388+0000","last_fullsized":"2023-10-10T18:45:11.415388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407755+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T18:45:11.407755+0000","last_peered":"2023-10-10T18:45:11.407755+0000","last_clean":"2023-10-10T18:45:11.407755+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T18:45:11.407755+0000","last_undegraded":"2023-10-10T18:45:11.407755+0000","last_fullsized":"2023-10-10T18:45:11.407755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415058+0000","last_change":"2023-10-10T18:45:11.410768+0000","last_active":"2023-10-10T18:45:11.415058+0000","last_peered":"2023-10-10T18:45:11.415058+0000","last_clean":"2023-10-10T18:45:11.415058+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T18:45:11.415058+0000","last_undegraded":"2023-10-10T18:45:11.415058+0000","last_fullsized":"2023-10-10T18:45:11.415058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407089+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T18:45:11.407089+0000","last_peered":"2023-10-10T18:45:11.407089+0000","last_clean":"2023-10-10T18:45:11.407089+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T18:45:11.407089+0000","last_undegraded":"2023-10-10T18:45:11.407089+0000","last_fullsized":"2023-10-10T18:45:11.407089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430290+0000","last_change":"2023-10-10T18:45:11.425716+0000","last_active":"2023-10-10T18:45:11.430290+0000","last_peered":"2023-10-10T18:45:11.430290+0000","last_clean":"2023-10-10T18:45:11.430290+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T18:45:11.430290+0000","last_undegraded":"2023-10-10T18:45:11.430290+0000","last_fullsized":"2023-10-10T18:45:11.430290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412053+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T18:45:11.412053+0000","last_peered":"2023-10-10T18:45:11.412053+0000","last_clean":"2023-10-10T18:45:11.412053+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T18:45:11.412053+0000","last_undegraded":"2023-10-10T18:45:11.412053+0000","last_fullsized":"2023-10-10T18:45:11.412053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429787+0000","last_change":"2023-10-10T18:45:11.423747+0000","last_active":"2023-10-10T18:45:11.429787+0000","last_peered":"2023-10-10T18:45:11.429787+0000","last_clean":"2023-10-10T18:45:11.429787+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T18:45:11.429787+0000","last_undegraded":"2023-10-10T18:45:11.429787+0000","last_fullsized":"2023-10-10T18:45:11.429787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407940+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T18:45:11.407940+0000","last_peered":"2023-10-10T18:45:11.407940+0000","last_clean":"2023-10-10T18:45:11.407940+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T18:45:11.407940+0000","last_undegraded":"2023-10-10T18:45:11.407940+0000","last_fullsized":"2023-10-10T18:45:11.407940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425816+0000","last_change":"2023-10-10T18:45:11.421872+0000","last_active":"2023-10-10T18:45:11.425816+0000","last_peered":"2023-10-10T18:45:11.425816+0000","last_clean":"2023-10-10T18:45:11.425816+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T18:45:11.425816+0000","last_undegraded":"2023-10-10T18:45:11.425816+0000","last_fullsized":"2023-10-10T18:45:11.425816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412481+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T18:45:11.412481+0000","last_peered":"2023-10-10T18:45:11.412481+0000","last_clean":"2023-10-10T18:45:11.412481+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T18:45:11.412481+0000","last_undegraded":"2023-10-10T18:45:11.412481+0000","last_fullsized":"2023-10-10T18:45:11.412481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429963+0000","last_change":"2023-10-10T18:45:11.425474+0000","last_active":"2023-10-10T18:45:11.429963+0000","last_peered":"2023-10-10T18:45:11.429963+0000","last_clean":"2023-10-10T18:45:11.429963+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T18:45:11.429963+0000","last_undegraded":"2023-10-10T18:45:11.429963+0000","last_fullsized":"2023-10-10T18:45:11.429963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430681+0000","last_change":"2023-10-10T18:45:11.416850+0000","last_active":"2023-10-10T18:45:11.430681+0000","last_peered":"2023-10-10T18:45:11.430681+0000","last_clean":"2023-10-10T18:45:11.430681+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T18:45:11.430681+0000","last_undegraded":"2023-10-10T18:45:11.430681+0000","last_fullsized":"2023-10-10T18:45:11.430681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427108+0000","last_change":"2023-10-10T18:45:11.424440+0000","last_active":"2023-10-10T18:45:11.427108+0000","last_peered":"2023-10-10T18:45:11.427108+0000","last_clean":"2023-10-10T18:45:11.427108+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T18:45:11.427108+0000","last_undegraded":"2023-10-10T18:45:11.427108+0000","last_fullsized":"2023-10-10T18:45:11.427108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.421524+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.420874+0000","last_became_peered":"2023-10-10T18:45:11.420874+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:56:36.851801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419177+0000","last_change":"2023-10-10T18:45:11.412399+0000","last_active":"2023-10-10T18:45:11.419177+0000","last_peered":"2023-10-10T18:45:11.419177+0000","last_clean":"2023-10-10T18:45:11.419177+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T18:45:11.419177+0000","last_undegraded":"2023-10-10T18:45:11.419177+0000","last_fullsized":"2023-10-10T18:45:11.419177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432996+0000","last_change":"2023-10-10T18:45:11.429507+0000","last_active":"2023-10-10T18:45:11.432996+0000","last_peered":"2023-10-10T18:45:11.432996+0000","last_clean":"2023-10-10T18:45:11.432996+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T18:45:11.432996+0000","last_undegraded":"2023-10-10T18:45:11.432996+0000","last_fullsized":"2023-10-10T18:45:11.432996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.418310+0000","last_change":"2023-10-10T18:45:11.413271+0000","last_active":"2023-10-10T18:45:11.418310+0000","last_peered":"2023-10-10T18:45:11.418310+0000","last_clean":"2023-10-10T18:45:11.418310+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T18:45:11.418310+0000","last_undegraded":"2023-10-10T18:45:11.418310+0000","last_fullsized":"2023-10-10T18:45:11.418310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430781+0000","last_change":"2023-10-10T18:45:11.427815+0000","last_active":"2023-10-10T18:45:11.430781+0000","last_peered":"2023-10-10T18:45:11.430781+0000","last_clean":"2023-10-10T18:45:11.430781+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T18:45:11.430781+0000","last_undegraded":"2023-10-10T18:45:11.430781+0000","last_fullsized":"2023-10-10T18:45:11.430781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426972+0000","last_change":"2023-10-10T18:45:11.425458+0000","last_active":"2023-10-10T18:45:11.426972+0000","last_peered":"2023-10-10T18:45:11.426972+0000","last_clean":"2023-10-10T18:45:11.426972+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T18:45:11.426972+0000","last_undegraded":"2023-10-10T18:45:11.426972+0000","last_fullsized":"2023-10-10T18:45:11.426972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424565+0000","last_change":"2023-10-10T18:45:11.422003+0000","last_active":"2023-10-10T18:45:11.424565+0000","last_peered":"2023-10-10T18:45:11.424565+0000","last_clean":"2023-10-10T18:45:11.424565+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T18:45:11.424565+0000","last_undegraded":"2023-10-10T18:45:11.424565+0000","last_fullsized":"2023-10-10T18:45:11.424565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432978+0000","last_change":"2023-10-10T18:45:11.428907+0000","last_active":"2023-10-10T18:45:11.432978+0000","last_peered":"2023-10-10T18:45:11.432978+0000","last_clean":"2023-10-10T18:45:11.432978+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T18:45:11.432978+0000","last_undegraded":"2023-10-10T18:45:11.432978+0000","last_fullsized":"2023-10-10T18:45:11.432978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430538+0000","last_change":"2023-10-10T18:45:11.427241+0000","last_active":"2023-10-10T18:45:11.430538+0000","last_peered":"2023-10-10T18:45:11.430538+0000","last_clean":"2023-10-10T18:45:11.430538+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T18:45:11.430538+0000","last_undegraded":"2023-10-10T18:45:11.430538+0000","last_fullsized":"2023-10-10T18:45:11.430538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427471+0000","last_change":"2023-10-10T18:45:11.426813+0000","last_active":"2023-10-10T18:45:11.427471+0000","last_peered":"2023-10-10T18:45:11.427471+0000","last_clean":"2023-10-10T18:45:11.427471+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T18:45:11.427471+0000","last_undegraded":"2023-10-10T18:45:11.427471+0000","last_fullsized":"2023-10-10T18:45:11.427471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432098+0000","last_change":"2023-10-10T18:45:11.429582+0000","last_active":"2023-10-10T18:45:11.432098+0000","last_peered":"2023-10-10T18:45:11.432098+0000","last_clean":"2023-10-10T18:45:11.432098+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T18:45:11.432098+0000","last_undegraded":"2023-10-10T18:45:11.432098+0000","last_fullsized":"2023-10-10T18:45:11.432098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":282624,"data_stored":665681,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":282624,"data_stored":665681,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T18:45:17.856 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-10T18:45:17.856 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-10T18:45:17.857 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-10T18:45:17.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-10T18:45:17.858 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-10T18:45:18.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.001+0000 7f05fd82c700 1 Processor -- start 2023-10-10T18:45:18.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.001+0000 7f05fd82c700 1 -- start start 2023-10-10T18:45:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.001+0000 7f05fd82c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f05f80b2390 0x7f05f8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.001+0000 7f05fd82c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05f80aff60 0x7f05f80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.001+0000 7f05fd82c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f81574b0 con 0x7f05f80b1fc0 2023-10-10T18:45:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.001+0000 7f05fd82c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f8150770 con 0x7f05f80aff60 2023-10-10T18:45:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.001+0000 7f05fd82c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f80afad0 con 0x7f05f80b2390 2023-10-10T18:45:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05f80aff60 0x7f05f80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05f80aff60 0x7f05f80b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52236/0 (socket says 172.21.15.175:52236) 2023-10-10T18:45:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f77fe700 1 -- 172.21.15.175:0/368965101 learned_addr learned my addr 172.21.15.175:0/368965101 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f67fc700 1 --2- 172.21.15.175:0/368965101 >> v2:172.21.15.191:3300/0 conn(0x7f05f80b2390 0x7f05f8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f77fe700 1 -- 172.21.15.175:0/368965101 >> v2:172.21.15.191:3300/0 conn(0x7f05f80b2390 msgr2=0x7f05f8155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f77fe700 1 --2- 172.21.15.175:0/368965101 >> v2:172.21.15.191:3300/0 conn(0x7f05f80b2390 0x7f05f8155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f77fe700 1 -- 172.21.15.175:0/368965101 >> v1:172.21.15.191:6789/0 conn(0x7f05f80b1fc0 legacy=0x7f05f8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f77fe700 1 -- 172.21.15.175:0/368965101 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f80af4c0 con 0x7f05f80aff60 2023-10-10T18:45:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f77fe700 1 --2- 172.21.15.175:0/368965101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80b03e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f05f80c0290 tx=0x7f05f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a046b9fd05a0ebba server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f5ffb700 1 -- 172.21.15.175:0/368965101 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f000e070 con 0x7f05f80aff60 2023-10-10T18:45:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f5ffb700 1 -- 172.21.15.175:0/368965101 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f000bab0 con 0x7f05f80aff60 2023-10-10T18:45:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.002+0000 7f05f5ffb700 1 -- 172.21.15.175:0/368965101 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f00092e0 con 0x7f05f80aff60 2023-10-10T18:45:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.003+0000 7f05fd82c700 1 -- 172.21.15.175:0/368965101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 msgr2=0x7f05f80b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.003+0000 7f05fd82c700 1 --2- 172.21.15.175:0/368965101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80b03e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f05f80c0290 tx=0x7f05f0009840 comp rx=0 tx=0).stop 2023-10-10T18:45:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.003+0000 7f05fd82c700 1 -- 172.21.15.175:0/368965101 shutdown_connections 2023-10-10T18:45:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.003+0000 7f05fd82c700 1 --2- 172.21.15.175:0/368965101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80b03e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.003+0000 7f05fd82c700 1 --2- 172.21.15.175:0/368965101 >> v2:172.21.15.191:3300/0 conn(0x7f05f80b2390 0x7f05f8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.003+0000 7f05fd82c700 1 -- 172.21.15.175:0/368965101 >> 172.21.15.175:0/368965101 conn(0x7f05f8063230 msgr2=0x7f05f8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:18.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.003+0000 7f05fd82c700 1 -- 172.21.15.175:0/368965101 shutdown_connections 2023-10-10T18:45:18.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.003+0000 7f05fd82c700 1 -- 172.21.15.175:0/368965101 wait complete. 2023-10-10T18:45:18.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.004+0000 7f05fd82c700 1 Processor -- start 2023-10-10T18:45:18.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.004+0000 7f05fd82c700 1 -- start start 2023-10-10T18:45:18.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05fd82c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05fd82c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05f80b1fc0 0x7f05f814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05fd82c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05f80b2390 0x7f05f814ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05fd82c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f80c1240 con 0x7f05f80aff60 2023-10-10T18:45:18.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05fd82c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f80c58a0 con 0x7f05f80b1fc0 2023-10-10T18:45:18.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05fd82c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f80c49f0 con 0x7f05f80b2390 2023-10-10T18:45:18.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80c5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52240/0 (socket says 172.21.15.175:52240) 2023-10-10T18:45:18.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 -- 172.21.15.175:0/422965789 learned_addr learned my addr 172.21.15.175:0/422965789 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:18.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f67fc700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05f80b1fc0 0x7f05f814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f77fe700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05f80b2390 0x7f05f814ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 -- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05f80b2390 msgr2=0x7f05f814ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05f80b2390 0x7f05f814ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 -- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05f80b1fc0 msgr2=0x7f05f814faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05f80b1fc0 0x7f05f814faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 -- 172.21.15.175:0/422965789 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0009ae0 con 0x7f05f80aff60 2023-10-10T18:45:18.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f77fe700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05f80b2390 0x7f05f814ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:18.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05f6ffd700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80c5d30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f05e800c590 tx=0x7f05e800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05deffd700 1 -- 172.21.15.175:0/422965789 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8011590 con 0x7f05f80aff60 2023-10-10T18:45:18.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05deffd700 1 -- 172.21.15.175:0/422965789 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e80116f0 con 0x7f05f80aff60 2023-10-10T18:45:18.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f80c4c90 con 0x7f05f80aff60 2023-10-10T18:45:18.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05deffd700 1 -- 172.21.15.175:0/422965789 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8011ab0 con 0x7f05f80aff60 2023-10-10T18:45:18.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.005+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f80c5250 con 0x7f05f80aff60 2023-10-10T18:45:18.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.006+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f8067570 con 0x7f05f80aff60 2023-10-10T18:45:18.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.007+0000 7f05deffd700 1 -- 172.21.15.175:0/422965789 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f05e801c440 con 0x7f05f80aff60 2023-10-10T18:45:18.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.007+0000 7f05deffd700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05e4066f30 0x7f05e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.007+0000 7f05deffd700 1 -- 172.21.15.175:0/422965789 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f05e8083a80 con 0x7f05f80aff60 2023-10-10T18:45:18.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.007+0000 7f05f67fc700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05e4066f30 0x7f05e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.008+0000 7f05f67fc700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05e4066f30 0x7f05e40693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f05f8153620 tx=0x7f05ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.011+0000 7f05deffd700 1 -- 172.21.15.175:0/422965789 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05e8050cf0 con 0x7f05f80aff60 2023-10-10T18:45:18.200 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:45:18.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:45:18.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.248+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f05f8068da0 con 0x7f05f80aff60 2023-10-10T18:45:18.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.250+0000 7f05deffd700 1 -- 172.21.15.175:0/422965789 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163706 (secure 0 0 0) 0x7f05e8017070 con 0x7f05f80aff60 2023-10-10T18:45:18.255 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:18.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05e4066f30 msgr2=0x7f05e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05e4066f30 0x7f05e40693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f05f8153620 tx=0x7f05ec008040 comp rx=0 tx=0).stop 2023-10-10T18:45:18.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 msgr2=0x7f05f80c5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80c5d30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f05e800c590 tx=0x7f05e800c950 comp rx=0 tx=0).stop 2023-10-10T18:45:18.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 shutdown_connections 2023-10-10T18:45:18.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05e4066f30 0x7f05e40693e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05f80b2390 0x7f05f814ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05f80aff60 0x7f05f80c5d30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 --2- 172.21.15.175:0/422965789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05f80b1fc0 0x7f05f814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 >> 172.21.15.175:0/422965789 conn(0x7f05f8063230 msgr2=0x7f05f8158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:18.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 shutdown_connections 2023-10-10T18:45:18.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.253+0000 7f05fd82c700 1 -- 172.21.15.175:0/422965789 wait complete. 2023-10-10T18:45:18.266 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":5,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":90896},{"type":"v1","addr":"172.21.15.191:6833","nonce":90896}]},"active_addr":"172.21.15.191:6833/90896","active_change":"2023-10-10T18:44:59.121078+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":2497305896}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":4291418324}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":2797752627}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":4207685856}]}]} 2023-10-10T18:45:18.270 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-10T18:45:18.270 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-10T18:45:18.271 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:18.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.422+0000 7fe420fc3700 1 Processor -- start 2023-10-10T18:45:18.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.422+0000 7fe420fc3700 1 -- start start 2023-10-10T18:45:18.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe420fc3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c06f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe420fc3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe41c1597c0 0x7fe41c159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe420fc3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe41c0c7930 0x7fe41c0c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe420fc3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe41c0c6430 con 0x7fe41c1597c0 2023-10-10T18:45:18.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe420fc3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe41c0c34e0 con 0x7fe41c0c7930 2023-10-10T18:45:18.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe420fc3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c150330 con 0x7fe41c1593f0 2023-10-10T18:45:18.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe41c1597c0 0x7fe41c159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe41c1597c0 0x7fe41c159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52244/0 (socket says 172.21.15.175:52244) 2023-10-10T18:45:18.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 -- 172.21.15.175:0/504521939 learned_addr learned my addr 172.21.15.175:0/504521939 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:18.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41b7fe700 1 --2- 172.21.15.175:0/504521939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c06f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41bfff700 1 --2- 172.21.15.175:0/504521939 >> v2:172.21.15.191:3300/0 conn(0x7fe41c0c7930 0x7fe41c0c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 -- 172.21.15.175:0/504521939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 msgr2=0x7fe41c06f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:18.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 --2- 172.21.15.175:0/504521939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c06f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 -- 172.21.15.175:0/504521939 >> v2:172.21.15.191:3300/0 conn(0x7fe41c0c7930 msgr2=0x7fe41c0c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:18.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 --2- 172.21.15.175:0/504521939 >> v2:172.21.15.191:3300/0 conn(0x7fe41c0c7930 0x7fe41c0c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 -- 172.21.15.175:0/504521939 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0c2da0 con 0x7fe41c1597c0 2023-10-10T18:45:18.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41bfff700 1 --2- 172.21.15.175:0/504521939 >> v2:172.21.15.191:3300/0 conn(0x7fe41c0c7930 0x7fe41c0c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:18.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.423+0000 7fe41affd700 1 --2- 172.21.15.175:0/504521939 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 0x7fe41c159bd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe41c06c5e0 tx=0x7fe410009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00a83b4e472c526 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe41a7fc700 1 -- 172.21.15.175:0/504521939 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41000b860 con 0x7fe41c1597c0 2023-10-10T18:45:18.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe41a7fc700 1 -- 172.21.15.175:0/504521939 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41000b9c0 con 0x7fe41c1597c0 2023-10-10T18:45:18.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe41a7fc700 1 -- 172.21.15.175:0/504521939 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41000bd60 con 0x7fe41c1597c0 2023-10-10T18:45:18.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 -- 172.21.15.175:0/504521939 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 msgr2=0x7fe41c159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 --2- 172.21.15.175:0/504521939 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 0x7fe41c159bd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe41c06c5e0 tx=0x7fe410009250 comp rx=0 tx=0).stop 2023-10-10T18:45:18.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 -- 172.21.15.175:0/504521939 shutdown_connections 2023-10-10T18:45:18.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 --2- 172.21.15.175:0/504521939 >> v2:172.21.15.191:3300/0 conn(0x7fe41c0c7930 0x7fe41c0c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 --2- 172.21.15.175:0/504521939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c06f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 --2- 172.21.15.175:0/504521939 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 0x7fe41c159bd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 -- 172.21.15.175:0/504521939 >> 172.21.15.175:0/504521939 conn(0x7fe41c0c9500 msgr2=0x7fe41c0c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:18.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 -- 172.21.15.175:0/504521939 shutdown_connections 2023-10-10T18:45:18.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.424+0000 7fe420fc3700 1 -- 172.21.15.175:0/504521939 wait complete. 2023-10-10T18:45:18.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.425+0000 7fe420fc3700 1 Processor -- start 2023-10-10T18:45:18.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe420fc3700 1 -- start start 2023-10-10T18:45:18.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe420fc3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe41c0c7930 0x7fe41c14ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe420fc3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c14a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe41affd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c14a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe41b7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe41c0c7930 0x7fe41c14ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe420fc3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 0x7fe41c14ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe41affd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c14a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33132/0 (socket says 172.21.15.175:33132) 2023-10-10T18:45:18.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe41affd700 1 -- 172.21.15.175:0/3638016378 learned_addr learned my addr 172.21.15.175:0/3638016378 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:18.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe41b7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe41c0c7930 0x7fe41c14ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37080/0 (socket says 172.21.15.175:37080) 2023-10-10T18:45:18.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c14d030 con 0x7fe41c1597c0 2023-10-10T18:45:18.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0ba440 con 0x7fe41c0c7930 2023-10-10T18:45:18.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.426+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0bf7d0 con 0x7fe41c1593f0 2023-10-10T18:45:18.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe41bfff700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 0x7fe41c14ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe41bfff700 1 -- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 msgr2=0x7fe41c14a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe41bfff700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c14a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe41bfff700 1 -- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe41c0c7930 msgr2=0x7fe41c14ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe41bfff700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe41c0c7930 0x7fe41c14ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe41bfff700 1 -- 172.21.15.175:0/3638016378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4100094f0 con 0x7fe41c1597c0 2023-10-10T18:45:18.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe41bfff700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 0x7fe41c14ac70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe41400e8a0 tx=0x7fe414008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe4037fe700 1 -- 172.21.15.175:0/3638016378 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41400c070 con 0x7fe41c1597c0 2023-10-10T18:45:18.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe4037fe700 1 -- 172.21.15.175:0/3638016378 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe414013410 con 0x7fe41c1597c0 2023-10-10T18:45:18.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c14d690 con 0x7fe41c1597c0 2023-10-10T18:45:18.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe4037fe700 1 -- 172.21.15.175:0/3638016378 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4140137b0 con 0x7fe41c1597c0 2023-10-10T18:45:18.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.427+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c14dc50 con 0x7fe41c1597c0 2023-10-10T18:45:18.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.428+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41c0ba5d0 con 0x7fe41c1597c0 2023-10-10T18:45:18.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.428+0000 7fe4037fe700 1 -- 172.21.15.175:0/3638016378 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe414013b60 con 0x7fe41c1597c0 2023-10-10T18:45:18.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.429+0000 7fe4037fe700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe408066f30 0x7fe4080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.429+0000 7fe4037fe700 1 -- 172.21.15.175:0/3638016378 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe414084210 con 0x7fe41c1597c0 2023-10-10T18:45:18.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.429+0000 7fe41b7fe700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe408066f30 0x7fe4080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.430+0000 7fe41b7fe700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe408066f30 0x7fe4080693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009fd0 tx=0x7fe40c009880 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.433+0000 7fe4037fe700 1 -- 172.21.15.175:0/3638016378 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe414054e80 con 0x7fe41c1597c0 2023-10-10T18:45:18.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.619+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe41c067570 con 0x7fe41c1597c0 2023-10-10T18:45:18.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.620+0000 7fe4037fe700 1 -- 172.21.15.175:0/3638016378 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7fe414054a10 con 0x7fe41c1597c0 2023-10-10T18:45:18.622 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:18.622 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:18.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe408066f30 msgr2=0x7fe4080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe408066f30 0x7fe4080693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009fd0 tx=0x7fe40c009880 comp rx=0 tx=0).stop 2023-10-10T18:45:18.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 msgr2=0x7fe41c14ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 0x7fe41c14ac70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe41400e8a0 tx=0x7fe414008040 comp rx=0 tx=0).stop 2023-10-10T18:45:18.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 shutdown_connections 2023-10-10T18:45:18.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe408066f30 0x7fe4080693e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe41c1597c0 0x7fe41c14ac70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe41c0c7930 0x7fe41c14ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 --2- 172.21.15.175:0/3638016378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe41c1593f0 0x7fe41c14a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 >> 172.21.15.175:0/3638016378 conn(0x7fe41c0c9500 msgr2=0x7fe41c06bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:18.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 shutdown_connections 2023-10-10T18:45:18.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.622+0000 7fe420fc3700 1 -- 172.21.15.175:0/3638016378 wait complete. 2023-10-10T18:45:18.636 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-10T18:45:18.637 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:18.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.750+0000 7ff4a3089700 1 Processor -- start 2023-10-10T18:45:18.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.750+0000 7ff4a3089700 1 -- start start 2023-10-10T18:45:18.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff4a3089700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c0c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff4a3089700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff49c1577a0 0x7ff49c0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff4a3089700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c154aa0 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff4a3089700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49c063dc0 con 0x7ff49c1577a0 2023-10-10T18:45:18.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff4a3089700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c157bf0 con 0x7ff49c1573d0 2023-10-10T18:45:18.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff4a0e25700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c0c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff4a0e25700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c0c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33138/0 (socket says 172.21.15.175:33138) 2023-10-10T18:45:18.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff4a0e25700 1 -- 172.21.15.175:0/1402343393 learned_addr learned my addr 172.21.15.175:0/1402343393 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:18.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.751+0000 7ff493fff700 1 --2- 172.21.15.175:0/1402343393 >> v2:172.21.15.191:3300/0 conn(0x7ff49c1577a0 0x7ff49c0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff4a0e25700 1 -- 172.21.15.175:0/1402343393 >> v2:172.21.15.191:3300/0 conn(0x7ff49c1577a0 msgr2=0x7ff49c0c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:18.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff4a0e25700 1 --2- 172.21.15.175:0/1402343393 >> v2:172.21.15.191:3300/0 conn(0x7ff49c1577a0 0x7ff49c0c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff4a0e25700 1 -- 172.21.15.175:0/1402343393 >> v1:172.21.15.191:6789/0 conn(0x7ff49c0bf6e0 legacy=0x7ff49c0c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff4a0e25700 1 -- 172.21.15.175:0/1402343393 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c157e50 con 0x7ff49c1573d0 2023-10-10T18:45:18.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff493fff700 1 --2- 172.21.15.175:0/1402343393 >> v2:172.21.15.191:3300/0 conn(0x7ff49c1577a0 0x7ff49c0c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:18.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff4a0e25700 1 --2- 172.21.15.175:0/1402343393 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c0c12a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff498009f20 tx=0x7ff498009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c20aba3b948a30a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff4937fe700 1 -- 172.21.15.175:0/1402343393 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49800e070 con 0x7ff49c1573d0 2023-10-10T18:45:18.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff4937fe700 1 -- 172.21.15.175:0/1402343393 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49800b9d0 con 0x7ff49c1573d0 2023-10-10T18:45:18.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.752+0000 7ff4937fe700 1 -- 172.21.15.175:0/1402343393 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4980092e0 con 0x7ff49c1573d0 2023-10-10T18:45:18.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.753+0000 7ff4a3089700 1 -- 172.21.15.175:0/1402343393 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 msgr2=0x7ff49c0c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.753+0000 7ff4a3089700 1 --2- 172.21.15.175:0/1402343393 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c0c12a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff498009f20 tx=0x7ff498009840 comp rx=0 tx=0).stop 2023-10-10T18:45:18.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.753+0000 7ff4a3089700 1 -- 172.21.15.175:0/1402343393 shutdown_connections 2023-10-10T18:45:18.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.753+0000 7ff4a3089700 1 --2- 172.21.15.175:0/1402343393 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c0c12a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.753+0000 7ff4a3089700 1 --2- 172.21.15.175:0/1402343393 >> v2:172.21.15.191:3300/0 conn(0x7ff49c1577a0 0x7ff49c0c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.753+0000 7ff4a3089700 1 -- 172.21.15.175:0/1402343393 >> 172.21.15.175:0/1402343393 conn(0x7ff49c06dc70 msgr2=0x7ff49c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:18.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.753+0000 7ff4a3089700 1 -- 172.21.15.175:0/1402343393 shutdown_connections 2023-10-10T18:45:18.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.753+0000 7ff4a3089700 1 -- 172.21.15.175:0/1402343393 wait complete. 2023-10-10T18:45:18.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.754+0000 7ff4a3089700 1 Processor -- start 2023-10-10T18:45:18.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.754+0000 7ff4a3089700 1 -- start start 2023-10-10T18:45:18.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.754+0000 7ff4a3089700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff49c0bf6e0 0x7ff49c164a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a3089700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c14c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a3089700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff49c1577a0 0x7ff49c167540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a3089700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c14d460 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a3089700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c1424b0 con 0x7ff49c1577a0 2023-10-10T18:45:18.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a3089700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c149cb0 con 0x7ff49c1573d0 2023-10-10T18:45:18.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff49c0bf6e0 0x7ff49c164a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff49c0bf6e0 0x7ff49c164a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52274/0 (socket says 172.21.15.175:52274) 2023-10-10T18:45:18.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 -- 172.21.15.175:0/62331017 learned_addr learned my addr 172.21.15.175:0/62331017 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:18.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff493fff700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c14c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a1626700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff49c1577a0 0x7ff49c167540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 -- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 msgr2=0x7ff49c14c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c14c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 -- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff49c1577a0 msgr2=0x7ff49c167540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff49c1577a0 0x7ff49c167540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 -- 172.21.15.175:0/62331017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c009420 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a0e25700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff49c0bf6e0 0x7ff49c164a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff49800b610 tx=0x7ff498018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff490ff9700 1 -- 172.21.15.175:0/62331017 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498008820 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff490ff9700 1 -- 172.21.15.175:0/62331017 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff498008980 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff498009ae0 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff490ff9700 1 -- 172.21.15.175:0/62331017 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498008d00 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.755+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c14cac0 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.756+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c067570 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.757+0000 7ff490ff9700 1 -- 172.21.15.175:0/62331017 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff498003040 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.757+0000 7ff490ff9700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff488066f30 0x7ff4880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.757+0000 7ff490ff9700 1 -- 172.21.15.175:0/62331017 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff498086180 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.757+0000 7ff493fff700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff488066f30 0x7ff4880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.757+0000 7ff493fff700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff488066f30 0x7ff4880693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009e10 tx=0x7ff48c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.761+0000 7ff490ff9700 1 -- 172.21.15.175:0/62331017 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff4980533f0 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.945+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff49c063ae0 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.945+0000 7ff490ff9700 1 -- 172.21.15.175:0/62331017 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7ff498056a10 con 0x7ff49c0bf6e0 2023-10-10T18:45:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:18.950 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:18.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff488066f30 msgr2=0x7ff4880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff488066f30 0x7ff4880693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009e10 tx=0x7ff48c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff49c0bf6e0 msgr2=0x7ff49c164a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:18.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff49c0bf6e0 0x7ff49c164a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff49800b610 tx=0x7ff498018000 comp rx=0 tx=0).stop 2023-10-10T18:45:18.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 shutdown_connections 2023-10-10T18:45:18.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff488066f30 0x7ff4880693e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff49c1577a0 0x7ff49c167540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff49c0bf6e0 0x7ff49c164a40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 --2- 172.21.15.175:0/62331017 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff49c1573d0 0x7ff49c14c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:18.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 >> 172.21.15.175:0/62331017 conn(0x7ff49c06dc70 msgr2=0x7ff49c0c5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:18.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 shutdown_connections 2023-10-10T18:45:18.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:18.948+0000 7ff4a3089700 1 -- 172.21.15.175:0/62331017 wait complete. 2023-10-10T18:45:18.961 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-10T18:45:18.962 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-10T18:45:18.962 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-10T18:45:18.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-10T18:45:18.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-10T18:45:18.964 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-10T18:45:18.964 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-10T18:45:18.965 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-10T18:45:19.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.155+0000 7fa21a33f700 1 Processor -- start 2023-10-10T18:45:19.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.155+0000 7fa21a33f700 1 -- start start 2023-10-10T18:45:19.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.156+0000 7fa21a33f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa214159fe0 0x7fa21414fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.156+0000 7fa21a33f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2140ca1a0 con 0x7fa2141562b0 2023-10-10T18:45:19.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.156+0000 7fa21a33f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2140c1710 con 0x7fa214150420 2023-10-10T18:45:19.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.156+0000 7fa21a33f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2140b2750 con 0x7fa214159fe0 2023-10-10T18:45:19.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa2188dc700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa214150420 0x7fa2140c1ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52040/0 (socket says 172.21.15.175:52040) 2023-10-10T18:45:19.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa2188dc700 1 -- 172.21.15.175:0/768831099 learned_addr learned my addr 172.21.15.175:0/768831099 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:19.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476133073 0 0) 0x7fa2140c1710 con 0x7fa214150420 2023-10-10T18:45:19.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc003570 con 0x7fa214150420 2023-10-10T18:45:19.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1137165651 0 0) 0x7fa1fc003570 con 0x7fa214150420 2023-10-10T18:45:19.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 >> v2:172.21.15.175:3300/0 conn(0x7fa214159fe0 msgr2=0x7fa21414fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 --2- 172.21.15.175:0/768831099 >> v2:172.21.15.175:3300/0 conn(0x7fa214159fe0 0x7fa21414fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 >> v1:172.21.15.175:6789/0 conn(0x7fa2141562b0 legacy=0x7fa214156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140c9640 con 0x7fa214150420 2023-10-10T18:45:19.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fa20c0017f0 con 0x7fa214150420 2023-10-10T18:45:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.157+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 >> v1:172.21.15.191:6789/0 conn(0x7fa214150420 legacy=0x7fa2140c1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa212ffd700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1fc003a60 0x7fa1fc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa212ffd700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc004810 0x7fa1fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa212ffd700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1fc0052c0 0x7fa1fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140b2750 con 0x7fa1fc0052c0 2023-10-10T18:45:19.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc00c1e0 con 0x7fa1fc003a60 2023-10-10T18:45:19.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc00c3a0 con 0x7fa1fc004810 2023-10-10T18:45:19.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2188dc700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1fc0052c0 0x7fa1fc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa213fff700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1fc003a60 0x7fa1fc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2137fe700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc004810 0x7fa1fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2188dc700 1 -- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc004810 msgr2=0x7fa1fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2188dc700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc004810 0x7fa1fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2188dc700 1 -- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1fc003a60 msgr2=0x7fa1fc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2188dc700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1fc003a60 0x7fa1fc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2188dc700 1 -- 172.21.15.175:0/768831099 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140c9640 con 0x7fa1fc0052c0 2023-10-10T18:45:19.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa213fff700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1fc003a60 0x7fa1fc003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:19.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2188dc700 1 -- 172.21.15.175:0/768831099 reap_dead start 2023-10-10T18:45:19.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.158+0000 7fa2188dc700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1fc0052c0 0x7fa1fc00ba60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa20c00aa80 tx=0x7fa20c00a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69776cb019252f4d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.159+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c00a340 con 0x7fa1fc0052c0 2023-10-10T18:45:19.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.159+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c00c5d0 con 0x7fa1fc0052c0 2023-10-10T18:45:19.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.159+0000 7fa212ffd700 1 -- 172.21.15.175:0/768831099 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c00c990 con 0x7fa1fc0052c0 2023-10-10T18:45:19.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.160+0000 7fa21a33f700 1 -- 172.21.15.175:0/768831099 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1fc0052c0 msgr2=0x7fa1fc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.160+0000 7fa21a33f700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1fc0052c0 0x7fa1fc00ba60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa20c00aa80 tx=0x7fa20c00a0f0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.163+0000 7fa40bc1a700 1 Processor -- start 2023-10-10T18:45:19.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.163+0000 7fa40bc1a700 1 -- start start 2023-10-10T18:45:19.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa40bc1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40414df80 0x7fa4040677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa40bc1a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa404069030 0x7fa40406edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa40bc1a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa404154750 con 0x7fa4040635e0 2023-10-10T18:45:19.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa40bc1a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4040c6530 con 0x7fa404069030 2023-10-10T18:45:19.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa40bc1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa404153ef0 con 0x7fa40414df80 2023-10-10T18:45:19.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7ff475fc0700 1 Processor -- start 2023-10-10T18:45:19.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa4099b6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40414df80 0x7fa4040677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa4099b6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40414df80 0x7fa4040677f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33158/0 (socket says 172.21.15.175:33158) 2023-10-10T18:45:19.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa4099b6700 1 -- 172.21.15.175:0/2491522152 learned_addr learned my addr 172.21.15.175:0/2491522152 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa4099b6700 1 -- 172.21.15.175:0/2491522152 >> v2:172.21.15.175:3300/0 conn(0x7fa404069030 msgr2=0x7fa40406edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa4099b6700 1 --2- 172.21.15.175:0/2491522152 >> v2:172.21.15.175:3300/0 conn(0x7fa404069030 0x7fa40406edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa4099b6700 1 -- 172.21.15.175:0/2491522152 >> v1:172.21.15.191:6789/0 conn(0x7fa4040635e0 legacy=0x7fa404157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa4099b6700 1 -- 172.21.15.175:0/2491522152 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40406d870 con 0x7fa40414df80 2023-10-10T18:45:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7fa4099b6700 1 --2- 172.21.15.175:0/2491522152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40414df80 0x7fa4040677f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa4000099b0 tx=0x7fa400009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e4d20ee7a275a3d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7fa4089b4700 1 -- 172.21.15.175:0/2491522152 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40000b860 con 0x7fa40414df80 2023-10-10T18:45:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7fa4089b4700 1 -- 172.21.15.175:0/2491522152 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40000b9c0 con 0x7fa40414df80 2023-10-10T18:45:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7fa4089b4700 1 -- 172.21.15.175:0/2491522152 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40000bd60 con 0x7fa40414df80 2023-10-10T18:45:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7ff475fc0700 1 -- start start 2023-10-10T18:45:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7fcfa0d27700 1 Processor -- start 2023-10-10T18:45:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7fcfa0d27700 1 -- start start 2023-10-10T18:45:19.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7f8f3ad65700 1 Processor -- start 2023-10-10T18:45:19.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.165+0000 7f8f3ad65700 1 -- start start 2023-10-10T18:45:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa21a33f700 1 -- 172.21.15.175:0/768831099 shutdown_connections 2023-10-10T18:45:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa21a33f700 1 --2- 172.21.15.175:0/768831099 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1fc0052c0 0x7fa1fc00ba60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.164+0000 7fa21a33f700 1 -- 172.21.15.175:0/768831099 >> 172.21.15.175:0/768831099 conn(0x7fa214063230 msgr2=0x7fa214063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7fa21a33f700 1 -- 172.21.15.175:0/768831099 shutdown_connections 2023-10-10T18:45:19.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7fa40bc1a700 1 -- 172.21.15.175:0/2491522152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40414df80 msgr2=0x7fa4040677f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7fa40bc1a700 1 --2- 172.21.15.175:0/2491522152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40414df80 0x7fa4040677f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa4000099b0 tx=0x7fa400009250 comp rx=0 tx=0).stop 2023-10-10T18:45:19.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7fa21a33f700 1 -- 172.21.15.175:0/768831099 wait complete. 2023-10-10T18:45:19.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7fa40bc1a700 1 -- 172.21.15.175:0/2491522152 shutdown_connections 2023-10-10T18:45:19.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7fa40bc1a700 1 --2- 172.21.15.175:0/2491522152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40414df80 0x7fa4040677f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7fa40bc1a700 1 --2- 172.21.15.175:0/2491522152 >> v2:172.21.15.175:3300/0 conn(0x7fa404069030 0x7fa40406edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7fa40bc1a700 1 -- 172.21.15.175:0/2491522152 >> 172.21.15.175:0/2491522152 conn(0x7fa4040c56a0 msgr2=0x7fa4040c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7f8f3ad65700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8f34068e60 0x7f8f34063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7f8f3ad65700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f340c61b0 0x7f8f340c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7f8f3ad65700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f34063770 con 0x7f8f3406de90 2023-10-10T18:45:19.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7f8f3ad65700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3406f1a0 con 0x7f8f34068e60 2023-10-10T18:45:19.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7f8f3ad65700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340682c0 con 0x7f8f340c61b0 2023-10-10T18:45:19.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7ff475fc0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff470156670 0x7ff47006e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.166+0000 7ff475fc0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4700c6b80 0x7ff4700c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7ff475fc0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4700c5b80 con 0x7ff470156a40 2023-10-10T18:45:19.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7ff475fc0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4700ca3b0 con 0x7ff4700c6b80 2023-10-10T18:45:19.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7ff475fc0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700c66f0 con 0x7ff470156670 2023-10-10T18:45:19.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7f8f33fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8f34068e60 0x7f8f34063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7f8f33fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8f34068e60 0x7f8f34063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52316/0 (socket says 172.21.15.175:52316) 2023-10-10T18:45:19.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7f8f33fff700 1 -- 172.21.15.175:0/114415161 learned_addr learned my addr 172.21.15.175:0/114415161 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7ff46f7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff470156670 0x7ff47006e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7ff46f7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff470156670 0x7ff47006e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33164/0 (socket says 172.21.15.175:33164) 2023-10-10T18:45:19.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7ff46f7fe700 1 -- 172.21.15.175:0/1815501879 learned_addr learned my addr 172.21.15.175:0/1815501879 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7f8f33fff700 1 -- 172.21.15.175:0/114415161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f340c61b0 msgr2=0x7f8f340c49e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T18:45:19.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7f8f33fff700 1 --2- 172.21.15.175:0/114415161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f340c61b0 0x7f8f340c49e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7f8f33fff700 1 -- 172.21.15.175:0/114415161 >> v1:172.21.15.191:6789/0 conn(0x7f8f3406de90 legacy=0x7f8f34068750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fa40bc1a700 1 -- 172.21.15.175:0/2491522152 shutdown_connections 2023-10-10T18:45:19.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7f8f33fff700 1 -- 172.21.15.175:0/114415161 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3406d680 con 0x7f8f34068e60 2023-10-10T18:45:19.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcfa0d27700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf9c1505e0 0x7fcf9c0c7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcfa0d27700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c15a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcfa0d27700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcf9c15a540 0x7fcf9c0c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcfa0d27700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9c159890 con 0x7fcf9c15a540 2023-10-10T18:45:19.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcfa0d27700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9c159b70 con 0x7fcf9c1505e0 2023-10-10T18:45:19.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcfa0d27700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c156780 con 0x7fcf9c1562d0 2023-10-10T18:45:19.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcf9a59c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf9c1505e0 0x7fcf9c0c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcf99d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcf9a59c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf9c1505e0 0x7fcf9c0c7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37126/0 (socket says 172.21.15.175:37126) 2023-10-10T18:45:19.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcf9a59c700 1 -- 172.21.15.175:0/92808266 learned_addr learned my addr 172.21.15.175:0/92808266 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fa21a33f700 1 Processor -- start 2023-10-10T18:45:19.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fa40bc1a700 1 -- 172.21.15.175:0/2491522152 wait complete. 2023-10-10T18:45:19.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7f8f33fff700 1 --2- 172.21.15.175:0/114415161 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f34068e60 0x7f8f34063230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f3406d810 tx=0x7f8f24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97d2248e6bab3ae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fa21a33f700 1 -- start start 2023-10-10T18:45:19.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.167+0000 7fcf9ad9d700 1 --2- 172.21.15.175:0/92808266 >> v2:172.21.15.175:3300/0 conn(0x7fcf9c15a540 0x7fcf9c0c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf99d9b700 1 -- 172.21.15.175:0/92808266 >> v2:172.21.15.191:3300/0 conn(0x7fcf9c1505e0 msgr2=0x7fcf9c0c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf99d9b700 1 --2- 172.21.15.175:0/92808266 >> v2:172.21.15.191:3300/0 conn(0x7fcf9c1505e0 0x7fcf9c0c7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf99d9b700 1 -- 172.21.15.175:0/92808266 >> v2:172.21.15.175:3300/0 conn(0x7fcf9c15a540 msgr2=0x7fcf9c0c1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf99d9b700 1 --2- 172.21.15.175:0/92808266 >> v2:172.21.15.175:3300/0 conn(0x7fcf9c15a540 0x7fcf9c0c1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf99d9b700 1 -- 172.21.15.175:0/92808266 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c156a60 con 0x7fcf9c1562d0 2023-10-10T18:45:19.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf99d9b700 1 --2- 172.21.15.175:0/92808266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c15a000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0714a0 tx=0x7fcf900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4310eca90c002f3e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf9959a700 1 -- 172.21.15.175:0/92808266 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90011630 con 0x7fcf9c1562d0 2023-10-10T18:45:19.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095498103 0 0) 0x7ff468001490 con 0x7ff470156a40 2023-10-10T18:45:19.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff458003570 con 0x7ff470156a40 2023-10-10T18:45:19.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf9959a700 1 -- 172.21.15.175:0/92808266 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf900117b0 con 0x7fcf9c1562d0 2023-10-10T18:45:19.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fcf9959a700 1 -- 172.21.15.175:0/92808266 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90011b70 con 0x7fcf9c1562d0 2023-10-10T18:45:19.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1607999123 0 0) 0x7ff458003570 con 0x7ff470156a40 2023-10-10T18:45:19.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff470156670 msgr2=0x7ff47006e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff470156670 0x7ff47006e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 >> v2:172.21.15.191:3300/0 conn(0x7ff4700c6b80 msgr2=0x7ff4700c9e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T18:45:19.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 --2- 172.21.15.175:0/1815501879 >> v2:172.21.15.191:3300/0 conn(0x7ff4700c6b80 0x7ff4700c9e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff470157210 con 0x7ff470156a40 2023-10-10T18:45:19.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7ff468001980 con 0x7ff470156a40 2023-10-10T18:45:19.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 >> v1:172.21.15.191:6789/0 conn(0x7ff470156a40 legacy=0x7ff4700c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff458003a60 0x7ff458003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff458004810 0x7ff458004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4580051c0 0x7ff45800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700ca3b0 con 0x7ff458003a60 2023-10-10T18:45:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700c66f0 con 0x7ff4580051c0 2023-10-10T18:45:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45800c0a0 con 0x7ff458004810 2023-10-10T18:45:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fa21a33f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc0052c0 0x7fa214159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fa21a33f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2141571f0 0x7fa21416cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fa21a33f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2140b0f10 0x7fa21414f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fa21a33f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa214153a00 con 0x7fa2140b0f10 2023-10-10T18:45:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fa21a33f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa214151de0 con 0x7fa2141571f0 2023-10-10T18:45:19.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.168+0000 7fa21a33f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa214153720 con 0x7fa1fc0052c0 2023-10-10T18:45:19.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc0052c0 0x7fa214159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7ff46ffff700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4580051c0 0x7ff45800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc0052c0 0x7fa214159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33182/0 (socket says 172.21.15.175:33182) 2023-10-10T18:45:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 -- 172.21.15.175:0/3259560570 learned_addr learned my addr 172.21.15.175:0/3259560570 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa2137fe700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2141571f0 0x7fa21416cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7ff46effd700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff458004810 0x7ff458004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa40bc1a700 1 Processor -- start 2023-10-10T18:45:19.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7f8f337fe700 1 -- 172.21.15.175:0/114415161 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2400b820 con 0x7f8f34068e60 2023-10-10T18:45:19.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7f8f337fe700 1 -- 172.21.15.175:0/114415161 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2400b980 con 0x7f8f34068e60 2023-10-10T18:45:19.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7ff46f7fe700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff458003a60 0x7ff458003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa2188dc700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2140b0f10 0x7fa21414f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7ff46ffff700 1 -- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff458004810 msgr2=0x7ff458004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7ff46ffff700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff458004810 0x7ff458004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7ff46ffff700 1 -- 172.21.15.175:0/1815501879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff458003a60 msgr2=0x7ff458003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7ff46ffff700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff458003a60 0x7ff458003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa40bc1a700 1 -- start start 2023-10-10T18:45:19.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7f8f337fe700 1 -- 172.21.15.175:0/114415161 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2400bd20 con 0x7f8f34068e60 2023-10-10T18:45:19.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7ff46ffff700 1 -- 172.21.15.175:0/1815501879 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff458003570 con 0x7ff4580051c0 2023-10-10T18:45:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 -- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2141571f0 msgr2=0x7fa21416cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fcfa0d27700 1 -- 172.21.15.175:0/92808266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 msgr2=0x7fcf9c15a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fcfa0d27700 1 --2- 172.21.15.175:0/92808266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c15a000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0714a0 tx=0x7fcf900091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2141571f0 0x7fa21416cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 -- 172.21.15.175:0/3259560570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2140b0f10 msgr2=0x7fa21414f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2140b0f10 0x7fa21414f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 -- 172.21.15.175:0/3259560570 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa214150b80 con 0x7fa1fc0052c0 2023-10-10T18:45:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa2137fe700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2141571f0 0x7fa21416cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fa213fff700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc0052c0 0x7fa214159820 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa20800f930 tx=0x7fa208007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fcfa0d27700 1 -- 172.21.15.175:0/92808266 shutdown_connections 2023-10-10T18:45:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fcfa0d27700 1 --2- 172.21.15.175:0/92808266 >> v2:172.21.15.175:3300/0 conn(0x7fcf9c15a540 0x7fcf9c0c1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fcfa0d27700 1 --2- 172.21.15.175:0/92808266 >> v2:172.21.15.191:3300/0 conn(0x7fcf9c1505e0 0x7fcf9c0c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fcfa0d27700 1 --2- 172.21.15.175:0/92808266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c15a000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.169+0000 7fcfa0d27700 1 -- 172.21.15.175:0/92808266 >> 172.21.15.175:0/92808266 conn(0x7fcf9c063230 msgr2=0x7fcf9c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208006800 con 0x7fa1fc0052c0 2023-10-10T18:45:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa208006960 con 0x7fa1fc0052c0 2023-10-10T18:45:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208006d00 con 0x7fa1fc0052c0 2023-10-10T18:45:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7ff46ffff700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4580051c0 0x7ff45800ba60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff46800a760 tx=0x7ff47006de70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64f571fc484896a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fbe6760e700 1 Processor -- start 2023-10-10T18:45:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468012440 con 0x7ff4580051c0 2023-10-10T18:45:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4680125a0 con 0x7ff4580051c0 2023-10-10T18:45:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7ff46e7fc700 1 -- 172.21.15.175:0/1815501879 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468012960 con 0x7ff4580051c0 2023-10-10T18:45:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa40bc1a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4040635e0 0x7fa40416dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa40bc1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa404069030 0x7fa4040c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa40bc1a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40414df80 0x7fa4040c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa40bc1a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40416d720 con 0x7fa4040635e0 2023-10-10T18:45:19.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa40bc1a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040c02f0 con 0x7fa40414df80 2023-10-10T18:45:19.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa40bc1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040c7590 con 0x7fa404069030 2023-10-10T18:45:19.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fbe6760e700 1 -- start start 2023-10-10T18:45:19.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa4091b5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa404069030 0x7fa4040c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa4091b5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa404069030 0x7fa4040c1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33206/0 (socket says 172.21.15.175:33206) 2023-10-10T18:45:19.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa4091b5700 1 -- 172.21.15.175:0/4046698045 learned_addr learned my addr 172.21.15.175:0/4046698045 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa40a1b7700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40414df80 0x7fa4040c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa4091b5700 1 -- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40414df80 msgr2=0x7fa4040c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa4091b5700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40414df80 0x7fa4040c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa4091b5700 1 -- 172.21.15.175:0/4046698045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4040635e0 msgr2=0x7fa40416dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa4091b5700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4040635e0 0x7fa40416dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7fa4091b5700 1 -- 172.21.15.175:0/4046698045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4000094f0 con 0x7fa404069030 2023-10-10T18:45:19.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fa4091b5700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa404069030 0x7fa4040c1a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400b840 tx=0x7fa3f4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7ff475fc0700 1 -- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4580051c0 msgr2=0x7ff45800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7ff475fc0700 1 --2- 172.21.15.175:0/1815501879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4580051c0 0x7ff45800ba60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff46800a760 tx=0x7ff47006de70 comp rx=0 tx=0).stop 2023-10-10T18:45:19.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7ff46ffff700 1 -- 172.21.15.175:0/1815501879 reap_dead start 2023-10-10T18:45:19.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7f8f3ad65700 1 -- 172.21.15.175:0/114415161 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f34068e60 msgr2=0x7f8f34063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.170+0000 7f8f3ad65700 1 --2- 172.21.15.175:0/114415161 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f34068e60 0x7f8f34063230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f3406d810 tx=0x7f8f24009250 comp rx=0 tx=0).stop 2023-10-10T18:45:19.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fcfa0d27700 1 -- 172.21.15.175:0/92808266 shutdown_connections 2023-10-10T18:45:19.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa214150e60 con 0x7fa1fc0052c0 2023-10-10T18:45:19.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa214154ed0 con 0x7fa1fc0052c0 2023-10-10T18:45:19.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7f8f3ad65700 1 -- 172.21.15.175:0/114415161 shutdown_connections 2023-10-10T18:45:19.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7f8f3ad65700 1 --2- 172.21.15.175:0/114415161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f340c61b0 0x7f8f340c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7f8f3ad65700 1 --2- 172.21.15.175:0/114415161 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f34068e60 0x7f8f34063230 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7f8f3ad65700 1 -- 172.21.15.175:0/114415161 >> 172.21.15.175:0/114415161 conn(0x7f8f340c56a0 msgr2=0x7f8f340c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4010430 con 0x7fa404069030 2023-10-10T18:45:19.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4010590 con 0x7fa404069030 2023-10-10T18:45:19.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4040c0010 con 0x7fa404069030 2023-10-10T18:45:19.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4010950 con 0x7fa404069030 2023-10-10T18:45:19.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe6760e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbe600c3450 0x7fbe600c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe6760e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe600c5070 con 0x7fbe6006be10 2023-10-10T18:45:19.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe6760e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe60150750 con 0x7fbe60150380 2023-10-10T18:45:19.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe6760e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe600c91b0 con 0x7fbe600c3450 2023-10-10T18:45:19.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe64ba9700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbe6006be10 0x7fbe6006adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60508/0 (socket says 172.21.15.175:60508) 2023-10-10T18:45:19.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe64ba9700 1 -- 172.21.15.175:0/2158043843 learned_addr learned my addr 172.21.15.175:0/2158043843 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:19.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe65bab700 1 --2- 172.21.15.175:0/2158043843 >> v2:172.21.15.175:3300/0 conn(0x7fbe600c3450 0x7fbe600c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe57fff700 1 -- 172.21.15.175:0/2158043843 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912710964 0 0) 0x7fbe60150750 con 0x7fbe60150380 2023-10-10T18:45:19.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fbe57fff700 1 -- 172.21.15.175:0/2158043843 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe4c003570 con 0x7fbe60150380 2023-10-10T18:45:19.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.172+0000 7fbe65bab700 1 -- 172.21.15.175:0/2158043843 >> v1:172.21.15.191:6789/0 conn(0x7fbe60150380 legacy=0x7fbe60156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.172+0000 7fbe65bab700 1 -- 172.21.15.175:0/2158043843 >> v1:172.21.15.175:6789/0 conn(0x7fbe6006be10 legacy=0x7fbe6006adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.172+0000 7fbe65bab700 1 -- 172.21.15.175:0/2158043843 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe600c3df0 con 0x7fbe600c3450 2023-10-10T18:45:19.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.172+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa2141509e0 con 0x7fa1fc0052c0 2023-10-10T18:45:19.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.172+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40416e0f0 con 0x7fa404069030 2023-10-10T18:45:19.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.172+0000 7fbe65bab700 1 --2- 172.21.15.175:0/2158043843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe600c3450 0x7fbe600c38b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0098e0 tx=0x7fbe5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2c952de93c06ad server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa404067570 con 0x7fa404069030 2023-10-10T18:45:19.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.171+0000 7fcfa0d27700 1 -- 172.21.15.175:0/92808266 wait complete. 2023-10-10T18:45:19.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fcfa0d27700 1 Processor -- start 2023-10-10T18:45:19.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fcfa0d27700 1 -- start start 2023-10-10T18:45:19.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa3f4010d00 con 0x7fa404069030 2023-10-10T18:45:19.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fbe57fff700 1 -- 172.21.15.175:0/2158043843 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00b640 con 0x7fbe600c3450 2023-10-10T18:45:19.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7f8f3ad65700 1 -- 172.21.15.175:0/114415161 shutdown_connections 2023-10-10T18:45:19.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fbe57fff700 1 -- 172.21.15.175:0/2158043843 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c00b7a0 con 0x7fbe600c3450 2023-10-10T18:45:19.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fbe57fff700 1 -- 172.21.15.175:0/2158043843 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00bb20 con 0x7fbe600c3450 2023-10-10T18:45:19.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fa3f9ffb700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa3e8066f30 0x7fa3e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa208007230 con 0x7fa1fc0052c0 2023-10-10T18:45:19.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa3f4084100 con 0x7fa404069030 2023-10-10T18:45:19.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa4099b6700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa3e8066f30 0x7fa3e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa3f9ffb700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7fa3e806f140 0x7fa3e8071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 --> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3e8071c00 con 0x7fa3e806f140 2023-10-10T18:45:19.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f4086740 con 0x7fa404069030 2023-10-10T18:45:19.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7f394477b700 1 Processor -- start 2023-10-10T18:45:19.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fcfa0d27700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf9c1505e0 0x7fcf9c16a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fcfa0d27700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c0caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fcfa0d27700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf9c15a540 0x7fcf9c0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fcfa0d27700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0cab30 con 0x7fcf9c1505e0 2023-10-10T18:45:19.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fcfa0d27700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c9bf0 con 0x7fcf9c15a540 2023-10-10T18:45:19.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fcfa0d27700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c177160 con 0x7fcf9c1562d0 2023-10-10T18:45:19.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fbe6760e700 1 -- 172.21.15.175:0/2158043843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe600c3450 msgr2=0x7fbe600c38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fbe6760e700 1 --2- 172.21.15.175:0/2158043843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe600c3450 0x7fbe600c38b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0098e0 tx=0x7fbe5c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa4099b6700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa3e8066f30 0x7fa3e80693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa4000099b0 tx=0x7fa400017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa1fbfff700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa200066fe0 0x7fa200069490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa2080843f0 con 0x7fa1fc0052c0 2023-10-10T18:45:19.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa1fbfff700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7fa20006f1f0 0x7fa200071600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 --> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa200071cb0 con 0x7fa20006f1f0 2023-10-10T18:45:19.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fbe6760e700 1 -- 172.21.15.175:0/2158043843 shutdown_connections 2023-10-10T18:45:19.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fbe6760e700 1 --2- 172.21.15.175:0/2158043843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe600c3450 0x7fbe600c38b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa208086930 con 0x7fa1fc0052c0 2023-10-10T18:45:19.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fbe6760e700 1 -- 172.21.15.175:0/2158043843 >> 172.21.15.175:0/2158043843 conn(0x7fbe60068270 msgr2=0x7fbe600c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7ff475fc0700 1 -- 172.21.15.175:0/1815501879 shutdown_connections 2023-10-10T18:45:19.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7ff475fc0700 1 -- 172.21.15.175:0/1815501879 >> 172.21.15.175:0/1815501879 conn(0x7ff47006c110 msgr2=0x7ff47006c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7ff475fc0700 1 -- 172.21.15.175:0/1815501879 shutdown_connections 2023-10-10T18:45:19.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa2188dc700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7fa20006f1f0 0x7fa200071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7fa2137fe700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa200066fe0 0x7fa200069490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.173+0000 7f8f3ad65700 1 -- 172.21.15.175:0/114415161 wait complete. 2023-10-10T18:45:19.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f8f3ad65700 1 Processor -- start 2023-10-10T18:45:19.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7fa2137fe700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa200066fe0 0x7fa200069490 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa2140c9140 tx=0x7fa204009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7fa40a1b7700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7fa3e806f140 0x7fa3e8071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7fa2188dc700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7fa20006f1f0 0x7fa200071600 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7fcf9a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf9c1505e0 0x7fcf9c16a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7fcf9a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf9c1505e0 0x7fcf9c16a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52382/0 (socket says 172.21.15.175:52382) 2023-10-10T18:45:19.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7fcf9a59c700 1 -- 172.21.15.175:0/428767476 learned_addr learned my addr 172.21.15.175:0/428767476 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f8f3ad65700 1 -- start start 2023-10-10T18:45:19.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7fcf99d9b700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c0caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7fcf9ad9d700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf9c15a540 0x7fcf9c0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7f394477b700 1 -- start start 2023-10-10T18:45:19.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f394477b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f393c06a950 0x7f393c153680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f394477b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f393c0c8f90 0x7f393c06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f394477b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393c06d380 con 0x7f393c152c80 2023-10-10T18:45:19.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f394477b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c06df00 con 0x7f393c06a950 2023-10-10T18:45:19.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f394477b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c06b5e0 con 0x7f393c0c8f90 2023-10-10T18:45:19.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f3942517700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f393c06a950 0x7f393c153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f3942517700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f393c06a950 0x7f393c153680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52398/0 (socket says 172.21.15.175:52398) 2023-10-10T18:45:19.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.175+0000 7f3942517700 1 -- 172.21.15.175:0/1985876984 learned_addr learned my addr 172.21.15.175:0/1985876984 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf9a59c700 1 -- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 msgr2=0x7fcf9c0caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf9a59c700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c0caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf9a59c700 1 -- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf9c15a540 msgr2=0x7fcf9c0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf9a59c700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf9c15a540 0x7fcf9c0cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf9a59c700 1 -- 172.21.15.175:0/428767476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90009420 con 0x7fcf9c1505e0 2023-10-10T18:45:19.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f3941d16700 1 --2- 172.21.15.175:0/1985876984 >> v2:172.21.15.191:3300/0 conn(0x7f393c0c8f90 0x7f393c06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf99d9b700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf9c1562d0 0x7fcf9c0caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:19.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf9a59c700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf9c1505e0 0x7fcf9c16a610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e850 tx=0x7fcf8c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f3942517700 1 -- 172.21.15.175:0/1985876984 >> v2:172.21.15.191:3300/0 conn(0x7f393c0c8f90 msgr2=0x7f393c06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f3942517700 1 --2- 172.21.15.175:0/1985876984 >> v2:172.21.15.191:3300/0 conn(0x7f393c0c8f90 0x7f393c06e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f3942517700 1 -- 172.21.15.175:0/1985876984 >> v1:172.21.15.191:6789/0 conn(0x7f393c152c80 legacy=0x7f393c06ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f3942517700 1 -- 172.21.15.175:0/1985876984 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c06d5d0 con 0x7f393c06a950 2023-10-10T18:45:19.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f3941515700 1 -- 172.21.15.175:0/1985876984 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151975369 0 0) 0x7f393c06d380 con 0x7f393c152c80 2023-10-10T18:45:19.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fa40a1b7700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7fa3e806f140 0x7fa3e8071550 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.174+0000 7ff475fc0700 1 -- 172.21.15.175:0/1815501879 wait complete. 2023-10-10T18:45:19.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 <== osd.0 v2:172.21.15.175:6808/90849 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa200071cb0 con 0x7fa20006f1f0 2023-10-10T18:45:19.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7ff475fc0700 1 Processor -- start 2023-10-10T18:45:19.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f3ad65700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f34068e60 0x7f8f340bf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f3ad65700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f3406de90 0x7f8f3416d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f3ad65700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f340c61b0 0x7f8f34152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f3ad65700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3416e450 con 0x7f8f3406de90 2023-10-10T18:45:19.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f3ad65700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3415f5d0 con 0x7f8f340c61b0 2023-10-10T18:45:19.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f3ad65700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3416e190 con 0x7f8f34068e60 2023-10-10T18:45:19.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f33fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f3406de90 0x7f8f3416d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f33fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f3406de90 0x7f8f3416d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52402/0 (socket says 172.21.15.175:52402) 2023-10-10T18:45:19.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f33fff700 1 -- 172.21.15.175:0/75910065 learned_addr learned my addr 172.21.15.175:0/75910065 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f38b01700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f34068e60 0x7f8f340bf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f8f39302700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f340c61b0 0x7f8f34152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c013410 con 0x7fcf9c1505e0 2023-10-10T18:45:19.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7f3942517700 1 --2- 172.21.15.175:0/1985876984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c06a950 0x7f393c153680 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f393c0c83d0 tx=0x7f3934007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279b6be6fb2511b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c177420 con 0x7fcf9c1505e0 2023-10-10T18:45:19.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c013570 con 0x7fcf9c1505e0 2023-10-10T18:45:19.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c013930 con 0x7fcf9c1505e0 2023-10-10T18:45:19.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f8f39302700 1 -- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f34068e60 msgr2=0x7f8f340bf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f8f39302700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f34068e60 0x7f8f340bf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f8f39302700 1 -- 172.21.15.175:0/75910065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f3406de90 msgr2=0x7f8f3416d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f8f39302700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f3406de90 0x7f8f3416d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f8f39302700 1 -- 172.21.15.175:0/75910065 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f240094f0 con 0x7f8f340c61b0 2023-10-10T18:45:19.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f8f39302700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f340c61b0 0x7f8f34152e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800b490 tx=0x7f8f28006f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 <== osd.4 v2:172.21.15.191:6800/91481 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa3e8071c00 con 0x7fa3e806f140 2023-10-10T18:45:19.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.176+0000 7ff475fc0700 1 -- start start 2023-10-10T18:45:19.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff475fc0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4580051c0 0x7ff4700c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff475fc0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4700bd720 0x7ff47013d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff475fc0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4700c4b20 0x7ff4700c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff475fc0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47016d8e0 con 0x7ff4580051c0 2023-10-10T18:45:19.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff475fc0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47016e740 con 0x7ff4700bd720 2023-10-10T18:45:19.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff475fc0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700c7910 con 0x7ff4700c4b20 2023-10-10T18:45:19.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff46effd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4700bd720 0x7ff47013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff46effd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4700bd720 0x7ff47013d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37196/0 (socket says 172.21.15.175:37196) 2023-10-10T18:45:19.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff46effd700 1 -- 172.21.15.175:0/1441884973 learned_addr learned my addr 172.21.15.175:0/1441884973 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f3941515700 1 -- 172.21.15.175:0/1985876984 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934010070 con 0x7f393c06a950 2023-10-10T18:45:19.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f3941515700 1 -- 172.21.15.175:0/1985876984 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393400dbd0 con 0x7f393c06a950 2023-10-10T18:45:19.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7f3941515700 1 -- 172.21.15.175:0/1985876984 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934007b40 con 0x7f393c06a950 2023-10-10T18:45:19.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.177+0000 7ff46ffff700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4700c4b20 0x7ff4700c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c0ca0e0 con 0x7fcf9c1505e0 2023-10-10T18:45:19.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff46effd700 1 -- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4700c4b20 msgr2=0x7ff4700c4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff46effd700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4700c4b20 0x7ff4700c4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff46effd700 1 -- 172.21.15.175:0/1441884973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4580051c0 msgr2=0x7ff4700c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff46effd700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4580051c0 0x7ff4700c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff46effd700 1 -- 172.21.15.175:0/1441884973 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47016db30 con 0x7ff4700bd720 2023-10-10T18:45:19.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff46ffff700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4700c4b20 0x7ff4700c4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:19.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff46effd700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4700bd720 0x7ff47013d020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff46400d5f0 tx=0x7ff464008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf9c067110 con 0x7fcf9c1505e0 2023-10-10T18:45:19.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.179+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcf8c013ca0 con 0x7fcf9c1505e0 2023-10-10T18:45:19.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7f394477b700 1 -- 172.21.15.175:0/1985876984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c06a950 msgr2=0x7f393c153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7f394477b700 1 --2- 172.21.15.175:0/1985876984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c06a950 0x7f393c153680 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f393c0c83d0 tx=0x7f3934007690 comp rx=0 tx=0).stop 2023-10-10T18:45:19.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff464011500 con 0x7ff4700bd720 2023-10-10T18:45:19.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46400f040 con 0x7ff4700bd720 2023-10-10T18:45:19.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4640118d0 con 0x7ff4700bd720 2023-10-10T18:45:19.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7fbe6760e700 1 -- 172.21.15.175:0/2158043843 shutdown_connections 2023-10-10T18:45:19.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2800be90 con 0x7f8f340c61b0 2023-10-10T18:45:19.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2800f510 con 0x7f8f340c61b0 2023-10-10T18:45:19.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.178+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2800f8d0 con 0x7f8f340c61b0 2023-10-10T18:45:19.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7fcf8a7fc700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf7c066f60 0x7fcf7c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fcf8c0841a0 con 0x7fcf9c1505e0 2023-10-10T18:45:19.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7fcf8a7fc700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fcf7c06f170 0x7fcf7c071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 --> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf7c071c30 con 0x7fcf7c06f170 2023-10-10T18:45:19.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf8c0867e0 con 0x7fcf9c1505e0 2023-10-10T18:45:19.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7fcf9ad9d700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fcf7c06f170 0x7fcf7c071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.179+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47016de10 con 0x7ff4700bd720 2023-10-10T18:45:19.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.179+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47016e3d0 con 0x7ff4700bd720 2023-10-10T18:45:19.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff470067570 con 0x7ff4700bd720 2023-10-10T18:45:19.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.179+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f340bd870 con 0x7f8f340c61b0 2023-10-10T18:45:19.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.179+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f340c4360 con 0x7f8f340c61b0 2023-10-10T18:45:19.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f34067570 con 0x7f8f340c61b0 2023-10-10T18:45:19.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.179+0000 7fbe6760e700 1 -- 172.21.15.175:0/2158043843 wait complete. 2023-10-10T18:45:19.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.181+0000 7fbe6760e700 1 Processor -- start 2023-10-10T18:45:19.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.181+0000 7fbe6760e700 1 -- start start 2023-10-10T18:45:19.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.181+0000 7f394477b700 1 -- 172.21.15.175:0/1985876984 shutdown_connections 2023-10-10T18:45:19.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.181+0000 7f394477b700 1 --2- 172.21.15.175:0/1985876984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c06a950 0x7f393c153680 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.181+0000 7f394477b700 1 --2- 172.21.15.175:0/1985876984 >> v2:172.21.15.191:3300/0 conn(0x7f393c0c8f90 0x7f393c06e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.181+0000 7f394477b700 1 -- 172.21.15.175:0/1985876984 >> 172.21.15.175:0/1985876984 conn(0x7f393c068280 msgr2=0x7f393c0c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.181+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8f2800fc80 con 0x7f8f340c61b0 2023-10-10T18:45:19.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.180+0000 7fcf9ad9d700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fcf7c06f170 0x7fcf7c071580 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.181+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 <== osd.1 v2:172.21.15.175:6800/90850 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fcf7c071c30 con 0x7fcf7c06f170 2023-10-10T18:45:19.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7f8f30ff9700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f20067000 0x7f8f200694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8f28083dc0 con 0x7f8f340c61b0 2023-10-10T18:45:19.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7f8f30ff9700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f8f2006f0a0 0x7f8f200714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7f8f38b01700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f20067000 0x7f8f200694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 --> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f20071b60 con 0x7f8f2006f0a0 2023-10-10T18:45:19.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f280863d0 con 0x7f8f340c61b0 2023-10-10T18:45:19.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.183+0000 7f394477b700 1 -- 172.21.15.175:0/1985876984 shutdown_connections 2023-10-10T18:45:19.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.183+0000 7f394477b700 1 -- 172.21.15.175:0/1985876984 wait complete. 2023-10-10T18:45:19.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.183+0000 7f8f38b01700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f20067000 0x7f8f200694b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0075c0 tx=0x7f8f2c009780 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.183+0000 7ffa6079e700 1 Processor -- start 2023-10-10T18:45:19.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.183+0000 7ffa6079e700 1 -- start start 2023-10-10T18:45:19.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7fbe6760e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe6006be10 0x7fbe60157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7fbe6760e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe600c3450 0x7fbe600c1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7fbe6760e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe60150380 0x7fbe60154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7fbe6760e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60153850 con 0x7fbe60150380 2023-10-10T18:45:19.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7fbe6760e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60154830 con 0x7fbe600c3450 2023-10-10T18:45:19.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.182+0000 7fbe6760e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60159410 con 0x7fbe6006be10 2023-10-10T18:45:19.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe600c3450 0x7fbe600c1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe600c3450 0x7fbe600c1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37202/0 (socket says 172.21.15.175:37202) 2023-10-10T18:45:19.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 -- 172.21.15.175:0/4272126999 learned_addr learned my addr 172.21.15.175:0/4272126999 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe65bab700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe60150380 0x7fbe60154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 -- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe6006be10 msgr2=0x7fbe60157bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T18:45:19.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe6006be10 0x7fbe60157bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 -- 172.21.15.175:0/4272126999 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe60150380 msgr2=0x7fbe60154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe60150380 0x7fbe60154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 -- 172.21.15.175:0/4272126999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c009420 con 0x7fbe600c3450 2023-10-10T18:45:19.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7f394477b700 1 Processor -- start 2023-10-10T18:45:19.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe64ba9700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe600c3450 0x7fbe600c1a30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800cc70 tx=0x7fbe58001a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7f394477b700 1 -- start start 2023-10-10T18:45:19.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe557fa700 1 -- 172.21.15.175:0/4272126999 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800a940 con 0x7fbe600c3450 2023-10-10T18:45:19.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa6079e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ffa5814df80 0x7ffa580677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa6079e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa58069030 0x7ffa5806edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa6079e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa58154750 con 0x7ffa580635e0 2023-10-10T18:45:19.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa6079e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa580c6530 con 0x7ffa5814df80 2023-10-10T18:45:19.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa6079e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa58153ef0 con 0x7ffa58069030 2023-10-10T18:45:19.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa5dd39700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa58069030 0x7ffa5806edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa5ed3b700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ffa580635e0 0x7ffa58157420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52084/0 (socket says 172.21.15.175:52084) 2023-10-10T18:45:19.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa5ed3b700 1 -- 172.21.15.175:0/3864477924 learned_addr learned my addr 172.21.15.175:0/3864477924 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:19.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa5dd39700 1 -- 172.21.15.175:0/3864477924 >> v2:172.21.15.191:3300/0 conn(0x7ffa5814df80 msgr2=0x7ffa580677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa5dd39700 1 --2- 172.21.15.175:0/3864477924 >> v2:172.21.15.191:3300/0 conn(0x7ffa5814df80 0x7ffa580677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa5dd39700 1 -- 172.21.15.175:0/3864477924 >> v1:172.21.15.191:6789/0 conn(0x7ffa580635e0 legacy=0x7ffa58157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7f394477b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f393c06a950 0x7f393c158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa5dd39700 1 -- 172.21.15.175:0/3864477924 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5806d870 con 0x7ffa58069030 2023-10-10T18:45:19.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7ffa5d538700 1 -- 172.21.15.175:0/3864477924 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104952519 0 0) 0x7ffa58154750 con 0x7ffa580635e0 2023-10-10T18:45:19.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7f394477b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f393c0c8f90 0x7f393c0c1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7f394477b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c152c80 0x7f393c0c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7f394477b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c155a10 con 0x7f393c152c80 2023-10-10T18:45:19.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7f394477b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1653f0 con 0x7f393c0c8f90 2023-10-10T18:45:19.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7f394477b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1647e0 con 0x7f393c06a950 2023-10-10T18:45:19.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7ffa5dd39700 1 --2- 172.21.15.175:0/3864477924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa58069030 0x7ffa5806edf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffa480098e0 tx=0x7ffa480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b8b7659f3e0f3fb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c152c80 0x7f393c0c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c152c80 0x7f393c0c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52428/0 (socket says 172.21.15.175:52428) 2023-10-10T18:45:19.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 -- 172.21.15.175:0/2463913760 learned_addr learned my addr 172.21.15.175:0/2463913760 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7ffa5d538700 1 -- 172.21.15.175:0/3864477924 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48009e80 con 0x7ffa58069030 2023-10-10T18:45:19.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7ffa5d538700 1 -- 172.21.15.175:0/3864477924 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4800b860 con 0x7ffa58069030 2023-10-10T18:45:19.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7ffa5d538700 1 -- 172.21.15.175:0/3864477924 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4800bc20 con 0x7ffa58069030 2023-10-10T18:45:19.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 -- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f393c06a950 msgr2=0x7f393c158f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T18:45:19.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f393c06a950 0x7f393c158f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 -- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f393c0c8f90 msgr2=0x7f393c0c1620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T18:45:19.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f393c0c8f90 0x7f393c0c1620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 -- 172.21.15.175:0/2463913760 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393400be70 con 0x7f393c152c80 2023-10-10T18:45:19.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f3942d18700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c152c80 0x7f393c0c1b60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f393000c540 tx=0x7f3930007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f392e7fc700 1 -- 172.21.15.175:0/2463913760 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930006820 con 0x7f393c152c80 2023-10-10T18:45:19.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f392e7fc700 1 -- 172.21.15.175:0/2463913760 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3930006980 con 0x7f393c152c80 2023-10-10T18:45:19.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c164ac0 con 0x7f393c152c80 2023-10-10T18:45:19.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.186+0000 7f392e7fc700 1 -- 172.21.15.175:0/2463913760 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930006d40 con 0x7f393c152c80 2023-10-10T18:45:19.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.184+0000 7fbe557fa700 1 -- 172.21.15.175:0/4272126999 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5800aac0 con 0x7fbe600c3450 2023-10-10T18:45:19.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe601596f0 con 0x7fbe600c3450 2023-10-10T18:45:19.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7fbe557fa700 1 -- 172.21.15.175:0/4272126999 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800ae80 con 0x7fbe600c3450 2023-10-10T18:45:19.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.185+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe601544c0 con 0x7fbe600c3450 2023-10-10T18:45:19.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.187+0000 7fbe557fa700 1 -- 172.21.15.175:0/4272126999 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbe58007300 con 0x7fbe600c3450 2023-10-10T18:45:19.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.187+0000 7ffa6079e700 1 -- 172.21.15.175:0/3864477924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa58069030 msgr2=0x7ffa5806edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.187+0000 7ffa6079e700 1 --2- 172.21.15.175:0/3864477924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa58069030 0x7ffa5806edf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffa480098e0 tx=0x7ffa480091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.187+0000 7fbe557fa700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe44066f30 0x7fbe440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.187+0000 7fbe557fa700 1 -- 172.21.15.175:0/4272126999 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbe580848a0 con 0x7fbe600c3450 2023-10-10T18:45:19.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.187+0000 7ffa6079e700 1 -- 172.21.15.175:0/3864477924 shutdown_connections 2023-10-10T18:45:19.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.187+0000 7ffa6079e700 1 --2- 172.21.15.175:0/3864477924 >> v2:172.21.15.191:3300/0 conn(0x7ffa5814df80 0x7ffa580677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.187+0000 7ffa6079e700 1 --2- 172.21.15.175:0/3864477924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa58069030 0x7ffa5806edf0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffa480098e0 tx=0x7ffa480091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.188+0000 7ffa6079e700 1 -- 172.21.15.175:0/3864477924 >> 172.21.15.175:0/3864477924 conn(0x7ffa580c56a0 msgr2=0x7ffa580c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.188+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c165080 con 0x7f393c152c80 2023-10-10T18:45:19.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.188+0000 7fcf99d9b700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf7c066f60 0x7fcf7c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.188+0000 7f8f33fff700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f8f2006f0a0 0x7f8f200714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.188+0000 7fbe653aa700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe44066f30 0x7fbe440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.188+0000 7f392e7fc700 1 -- 172.21.15.175:0/2463913760 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3930007240 con 0x7f393c152c80 2023-10-10T18:45:19.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.188+0000 7fbe653aa700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe44066f30 0x7fbe440693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe50007fd0 tx=0x7fbe50007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7ffa6079e700 1 -- 172.21.15.175:0/3864477924 shutdown_connections 2023-10-10T18:45:19.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7ffa6079e700 1 -- 172.21.15.175:0/3864477924 wait complete. 2023-10-10T18:45:19.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7f392e7fc700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3920066f30 0x7f39200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7f392e7fc700 1 -- 172.21.15.175:0/2463913760 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f39300837b0 con 0x7f393c152c80 2023-10-10T18:45:19.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7f3942517700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3920066f30 0x7f39200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7f394477b700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7f393c157380 0x7f393c14f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 --> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f393c14f850 con 0x7f393c157380 2023-10-10T18:45:19.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7f3941d16700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7f393c157380 0x7f393c14f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.189+0000 7f3941d16700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7f393c157380 0x7f393c14f310 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.190+0000 7fbe6760e700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fbe600c7c90 0x7fbe6014f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.190+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 --> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe600c81e0 con 0x7fbe600c7c90 2023-10-10T18:45:19.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.190+0000 7fbe65bab700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fbe600c7c90 0x7fbe6014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.190+0000 7f392e7fc700 1 -- 172.21.15.175:0/2463913760 <== osd.2 v2:172.21.15.175:6814/90853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f393c14f850 con 0x7f393c157380 2023-10-10T18:45:19.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.190+0000 7fbe65bab700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fbe600c7c90 0x7fbe6014f200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.190+0000 7ffa6079e700 1 Processor -- start 2023-10-10T18:45:19.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.190+0000 7ffa6079e700 1 -- start start 2023-10-10T18:45:19.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.191+0000 7f8f33fff700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f8f2006f0a0 0x7f8f200714b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.191+0000 7fcf99d9b700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf7c066f60 0x7fcf7c069410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0714a0 tx=0x7fcf90017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.191+0000 7fbe557fa700 1 -- 172.21.15.175:0/4272126999 <== osd.5 v2:172.21.15.191:6816/91495 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fbe600c81e0 con 0x7fbe600c7c90 2023-10-10T18:45:19.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.191+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 <== osd.6 v2:172.21.15.191:6808/91482 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f8f20071b60 con 0x7f8f2006f0a0 2023-10-10T18:45:19.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa6079e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa580635e0 0x7ffa581659b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa6079e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa580bf260 0x7ffa580c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa6079e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa580c7c10 0x7ffa5814aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa6079e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa58165520 con 0x7ffa580c7c10 2023-10-10T18:45:19.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa6079e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa580c0300 con 0x7ffa580635e0 2023-10-10T18:45:19.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa6079e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa580c05a0 con 0x7ffa580bf260 2023-10-10T18:45:19.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa580bf260 0x7ffa580c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa580bf260 0x7ffa580c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33258/0 (socket says 172.21.15.175:33258) 2023-10-10T18:45:19.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 -- 172.21.15.175:0/3319007482 learned_addr learned my addr 172.21.15.175:0/3319007482 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5ed3b700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa580c7c10 0x7ffa5814aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 -- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa580635e0 msgr2=0x7ffa581659b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa580635e0 0x7ffa581659b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 -- 172.21.15.175:0/3319007482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa580c7c10 msgr2=0x7ffa5814aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa580c7c10 0x7ffa5814aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 -- 172.21.15.175:0/3319007482 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa48009420 con 0x7ffa580bf260 2023-10-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa5dd39700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa580bf260 0x7ffa580c76d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa58154030 tx=0x7ffa48017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.192+0000 7ffa4e7fc700 1 -- 172.21.15.175:0/3319007482 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4801f070 con 0x7ffa580bf260 2023-10-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ffa4e7fc700 1 -- 172.21.15.175:0/3319007482 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa48000c40 con 0x7ffa580bf260 2023-10-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa580c0860 con 0x7ffa580bf260 2023-10-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ffa4e7fc700 1 -- 172.21.15.175:0/3319007482 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4800bc00 con 0x7ffa580bf260 2023-10-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa580c7460 con 0x7ffa580bf260 2023-10-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff464011cc0 con 0x7ff4700bd720 2023-10-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ff4577fe700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff45c0670b0 0x7ff45c069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff4640846e0 con 0x7ff4700bd720 2023-10-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ff46f7fe700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff45c0670b0 0x7ff45c069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ff4577fe700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7ff45c06f2c0 0x7ff45c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 --> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c071d80 con 0x7ff45c06f2c0 2023-10-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff464084a10 con 0x7ff4700bd720 2023-10-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.195+0000 7ffa4e7fc700 1 -- 172.21.15.175:0/3319007482 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffa48023070 con 0x7ffa580bf260 2023-10-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.195+0000 7ffa4e7fc700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffa44067070 0x7ffa44069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.195+0000 7ffa4e7fc700 1 -- 172.21.15.175:0/3319007482 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ffa48085b20 con 0x7ffa580bf260 2023-10-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.196+0000 7ffa5e53a700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffa44067070 0x7ffa44069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.196+0000 7ffa5e53a700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffa44067070 0x7ffa44069520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffa54006fd0 tx=0x7ffa54006d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.196+0000 7f3942517700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3920066f30 0x7f39200693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f39340076d0 tx=0x7f393401a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.198+0000 7ffa6079e700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7ffa3c001620 0x7ffa3c003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.198+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 --> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa3c006bc0 con 0x7ffa3c001620 2023-10-10T18:45:19.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.193+0000 7ff46ffff700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7ff45c06f2c0 0x7ff45c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.194+0000 7ff46f7fe700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff45c0670b0 0x7ff45c069560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff47006fb90 tx=0x7ff460008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.194+0000 7ff46ffff700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7ff45c06f2c0 0x7ff45c0716d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.200+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 <== osd.3 v2:172.21.15.175:6824/90874 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff45c071d80 con 0x7ff45c06f2c0 2023-10-10T18:45:19.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.201+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 --> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa4040c6250 con 0x7fa3e806f140 2023-10-10T18:45:19.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.202+0000 7fa3f9ffb700 1 -- 172.21.15.175:0/4046698045 <== osd.4 v2:172.21.15.191:6800/91481 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa4040c6250 con 0x7fa3e806f140 2023-10-10T18:45:19.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.204+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 --> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa214159440 con 0x7fa20006f1f0 2023-10-10T18:45:19.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.204+0000 7fa1fbfff700 1 -- 172.21.15.175:0/3259560570 <== osd.0 v2:172.21.15.175:6808/90849 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa214159440 con 0x7fa20006f1f0 2023-10-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.205+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7fa20006f1f0 msgr2=0x7fa200071600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.205+0000 7fa21a33f700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7fa20006f1f0 0x7fa200071600 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa200066fe0 msgr2=0x7fa200069490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa21a33f700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa200066fe0 0x7fa200069490 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa2140c9140 tx=0x7fa204009060 comp rx=0 tx=0).stop 2023-10-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc0052c0 msgr2=0x7fa214159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa21a33f700 1 --2- 172.21.15.175:0/3259560570 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1fc0052c0 0x7fa214159820 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa20800f930 tx=0x7fa208007d10 comp rx=0 tx=0).stop 2023-10-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa2188dc700 1 -- 172.21.15.175:0/3259560570 reap_dead start 2023-10-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 shutdown_connections 2023-10-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 >> 172.21.15.175:0/3259560570 conn(0x7fa214063230 msgr2=0x7fa214063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 shutdown_connections 2023-10-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.209+0000 7fa21a33f700 1 -- 172.21.15.175:0/3259560570 wait complete. 2023-10-10T18:45:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.213+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 --> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcf9c067570 con 0x7fcf7c06f170 2023-10-10T18:45:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.214+0000 7fcf8a7fc700 1 -- 172.21.15.175:0/428767476 <== osd.1 v2:172.21.15.175:6800/90850 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcf9c067570 con 0x7fcf7c06f170 2023-10-10T18:45:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.214+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fcf7c06f170 msgr2=0x7fcf7c071580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.214+0000 7fcfa0d27700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fcf7c06f170 0x7fcf7c071580 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf7c066f60 msgr2=0x7fcf7c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcfa0d27700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf7c066f60 0x7fcf7c069410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0714a0 tx=0x7fcf90017000 comp rx=0 tx=0).stop 2023-10-10T18:45:19.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf9c1505e0 msgr2=0x7fcf9c16a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcfa0d27700 1 --2- 172.21.15.175:0/428767476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf9c1505e0 0x7fcf9c16a610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e850 tx=0x7fcf8c008040 comp rx=0 tx=0).stop 2023-10-10T18:45:19.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcf9ad9d700 1 -- 172.21.15.175:0/428767476 reap_dead start 2023-10-10T18:45:19.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 shutdown_connections 2023-10-10T18:45:19.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 >> 172.21.15.175:0/428767476 conn(0x7fcf9c063230 msgr2=0x7fcf9c14d500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 shutdown_connections 2023-10-10T18:45:19.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fcfa0d27700 1 -- 172.21.15.175:0/428767476 wait complete. 2023-10-10T18:45:19.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.215+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 --> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f393c0693f0 con 0x7f393c157380 2023-10-10T18:45:19.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.216+0000 7f392e7fc700 1 -- 172.21.15.175:0/2463913760 <== osd.2 v2:172.21.15.175:6814/90853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f393c0693f0 con 0x7f393c157380 2023-10-10T18:45:19.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7f393c157380 msgr2=0x7f393c14f310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7f393c157380 0x7f393c14f310 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3920066f30 msgr2=0x7f39200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3920066f30 0x7f39200693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f39340076d0 tx=0x7f393401a000 comp rx=0 tx=0).stop 2023-10-10T18:45:19.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c152c80 msgr2=0x7f393c0c1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 --2- 172.21.15.175:0/2463913760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f393c152c80 0x7f393c0c1b60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f393000c540 tx=0x7f3930007f30 comp rx=0 tx=0).stop 2023-10-10T18:45:19.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f3942d18700 1 -- 172.21.15.175:0/2463913760 reap_dead start 2023-10-10T18:45:19.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 shutdown_connections 2023-10-10T18:45:19.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 >> 172.21.15.175:0/2463913760 conn(0x7f393c068280 msgr2=0x7f393c1544d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 shutdown_connections 2023-10-10T18:45:19.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7f394477b700 1 -- 172.21.15.175:0/2463913760 wait complete. 2023-10-10T18:45:19.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.215+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7fa3e806f140 msgr2=0x7fa3e8071550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.215+0000 7fa40bc1a700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7fa3e806f140 0x7fa3e8071550 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.215+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa3e8066f30 msgr2=0x7fa3e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.215+0000 7fa40bc1a700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa3e8066f30 0x7fa3e80693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa4000099b0 tx=0x7fa400017000 comp rx=0 tx=0).stop 2023-10-10T18:45:19.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.215+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa404069030 msgr2=0x7fa4040c1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.215+0000 7fa40bc1a700 1 --2- 172.21.15.175:0/4046698045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa404069030 0x7fa4040c1a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400b840 tx=0x7fa3f4009250 comp rx=0 tx=0).stop 2023-10-10T18:45:19.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.215+0000 7fa40a1b7700 1 -- 172.21.15.175:0/4046698045 reap_dead start 2023-10-10T18:45:19.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 shutdown_connections 2023-10-10T18:45:19.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 >> 172.21.15.175:0/4046698045 conn(0x7fa4040c56a0 msgr2=0x7fa4040c3140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 shutdown_connections 2023-10-10T18:45:19.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fa40bc1a700 1 -- 172.21.15.175:0/4046698045 wait complete. 2023-10-10T18:45:19.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 --> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbe6006b9c0 con 0x7fbe600c7c90 2023-10-10T18:45:19.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.217+0000 7fbe557fa700 1 -- 172.21.15.175:0/4272126999 <== osd.5 v2:172.21.15.191:6816/91495 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbe6006b9c0 con 0x7fbe600c7c90 2023-10-10T18:45:19.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.200+0000 7ffa5ed3b700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7ffa3c001620 0x7ffa3c003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.200+0000 7ffa5ed3b700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7ffa3c001620 0x7ffa3c003ad0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.201+0000 7ffa4e7fc700 1 -- 172.21.15.175:0/3319007482 <== osd.7 v2:172.21.15.191:6824/91512 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ffa3c006bc0 con 0x7ffa3c001620 2023-10-10T18:45:19.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.220+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 --> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffa40000f90 con 0x7ffa3c001620 2023-10-10T18:45:19.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.221+0000 7ffa4e7fc700 1 -- 172.21.15.175:0/3319007482 <== osd.7 v2:172.21.15.191:6824/91512 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffa40000f90 con 0x7ffa3c001620 2023-10-10T18:45:19.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.221+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 --> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8f340bc330 con 0x7f8f2006f0a0 2023-10-10T18:45:19.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.222+0000 7f8f30ff9700 1 -- 172.21.15.175:0/75910065 <== osd.6 v2:172.21.15.191:6808/91482 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8f24022040 con 0x7f8f2006f0a0 2023-10-10T18:45:19.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.222+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f8f2006f0a0 msgr2=0x7f8f200714b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.222+0000 7f8f3ad65700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f8f2006f0a0 0x7f8f200714b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f20067000 msgr2=0x7f8f200694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7f8f3ad65700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f20067000 0x7f8f200694b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0075c0 tx=0x7f8f2c009780 comp rx=0 tx=0).stop 2023-10-10T18:45:19.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f340c61b0 msgr2=0x7f8f34152e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7f8f3ad65700 1 --2- 172.21.15.175:0/75910065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f340c61b0 0x7f8f34152e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800b490 tx=0x7f8f28006f50 comp rx=0 tx=0).stop 2023-10-10T18:45:19.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fbe600c7c90 msgr2=0x7fbe6014f200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7fbe6760e700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fbe600c7c90 0x7fbe6014f200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe44066f30 msgr2=0x7fbe440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7fbe6760e700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe44066f30 0x7fbe440693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe50007fd0 tx=0x7fbe50007990 comp rx=0 tx=0).stop 2023-10-10T18:45:19.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe600c3450 msgr2=0x7fbe600c1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7fbe6760e700 1 --2- 172.21.15.175:0/4272126999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe600c3450 0x7fbe600c1a30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800cc70 tx=0x7fbe58001a30 comp rx=0 tx=0).stop 2023-10-10T18:45:19.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.223+0000 7fbe65bab700 1 -- 172.21.15.175:0/4272126999 reap_dead start 2023-10-10T18:45:19.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7ffa3c001620 msgr2=0x7ffa3c003ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa6079e700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7ffa3c001620 0x7ffa3c003ad0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffa44067070 msgr2=0x7ffa44069520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa6079e700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffa44067070 0x7ffa44069520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffa54006fd0 tx=0x7ffa54006d30 comp rx=0 tx=0).stop 2023-10-10T18:45:19.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa580bf260 msgr2=0x7ffa580c76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa6079e700 1 --2- 172.21.15.175:0/3319007482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa580bf260 0x7ffa580c76d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa58154030 tx=0x7ffa48017000 comp rx=0 tx=0).stop 2023-10-10T18:45:19.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa5ed3b700 1 -- 172.21.15.175:0/3319007482 reap_dead start 2023-10-10T18:45:19.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 shutdown_connections 2023-10-10T18:45:19.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.224+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 >> 172.21.15.175:0/3319007482 conn(0x7ffa580c56a0 msgr2=0x7ffa580c5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.225+0000 7f8f39302700 1 -- 172.21.15.175:0/75910065 reap_dead start 2023-10-10T18:45:19.364 INFO:teuthology.orchestra.run.smithi175.stdout:55834574855 2023-10-10T18:45:19.364 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-10T18:45:19.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.225+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 shutdown_connections 2023-10-10T18:45:19.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.225+0000 7ffa6079e700 1 -- 172.21.15.175:0/3319007482 wait complete. 2023-10-10T18:45:19.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.229+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 --> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff4700697d0 con 0x7ff45c06f2c0 2023-10-10T18:45:19.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.229+0000 7ff4577fe700 1 -- 172.21.15.175:0/1441884973 <== osd.3 v2:172.21.15.175:6824/90874 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff4700697d0 con 0x7ff45c06f2c0 2023-10-10T18:45:19.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 shutdown_connections 2023-10-10T18:45:19.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 >> 172.21.15.175:0/75910065 conn(0x7f8f340c56a0 msgr2=0x7f8f340c5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7ff45c06f2c0 msgr2=0x7ff45c0716d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7ff475fc0700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7ff45c06f2c0 0x7ff45c0716d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff45c0670b0 msgr2=0x7ff45c069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7ff475fc0700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff45c0670b0 0x7ff45c069560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff47006fb90 tx=0x7ff460008040 comp rx=0 tx=0).stop 2023-10-10T18:45:19.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4700bd720 msgr2=0x7ff47013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7ff475fc0700 1 --2- 172.21.15.175:0/1441884973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4700bd720 0x7ff47013d020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff46400d5f0 tx=0x7ff464008040 comp rx=0 tx=0).stop 2023-10-10T18:45:19.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 shutdown_connections 2023-10-10T18:45:19.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7f8f3ad65700 1 -- 172.21.15.175:0/75910065 wait complete. 2023-10-10T18:45:19.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7ff46ffff700 1 -- 172.21.15.175:0/1441884973 reap_dead start 2023-10-10T18:45:19.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.232+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 shutdown_connections 2023-10-10T18:45:19.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.232+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 >> 172.21.15.175:0/1441884973 conn(0x7ff47006c110 msgr2=0x7ff470153590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.232+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 shutdown_connections 2023-10-10T18:45:19.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.232+0000 7ff475fc0700 1 -- 172.21.15.175:0/1441884973 wait complete. 2023-10-10T18:45:19.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 shutdown_connections 2023-10-10T18:45:19.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.230+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 >> 172.21.15.175:0/4272126999 conn(0x7fbe60068270 msgr2=0x7fbe6006adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.238+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 shutdown_connections 2023-10-10T18:45:19.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.238+0000 7fbe6760e700 1 -- 172.21.15.175:0/4272126999 wait complete. 2023-10-10T18:45:19.372 INFO:teuthology.orchestra.run.smithi175.stdout:55834574853 2023-10-10T18:45:19.373 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-10T18:45:19.374 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:19.374 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-10T18:45:19.375 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:19.375 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-10T18:45:19.376 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:19.377 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-10T18:45:19.378 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:19.378 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-10T18:45:19.379 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:19.379 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-10T18:45:19.380 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:19.381 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-10T18:45:19.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.608+0000 7f351a9d3700 1 Processor -- start 2023-10-10T18:45:19.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.608+0000 7f351a9d3700 1 -- start start 2023-10-10T18:45:19.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.609+0000 7f351a9d3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35140c4920 0x7f3514153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.609+0000 7f351a9d3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35140c50d0 con 0x7f351406b850 2023-10-10T18:45:19.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.609+0000 7f351a9d3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35140bf360 con 0x7f351406b480 2023-10-10T18:45:19.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.609+0000 7f351a9d3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35140c5370 con 0x7f35140c4920 2023-10-10T18:45:19.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35191d0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35140c4920 0x7f3514153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35191d0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35140c4920 0x7f3514153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33264/0 (socket says 172.21.15.175:33264) 2023-10-10T18:45:19.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35191d0700 1 -- 172.21.15.175:0/2887291030 learned_addr learned my addr 172.21.15.175:0/2887291030 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35191d0700 1 -- 172.21.15.175:0/2887291030 >> v1:172.21.15.191:6789/0 conn(0x7f351406b480 legacy=0x7f35140c4210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T18:45:19.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35191d0700 1 -- 172.21.15.175:0/2887291030 >> v1:172.21.15.175:6789/0 conn(0x7f351406b850 legacy=0x7f3514164a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35191d0700 1 -- 172.21.15.175:0/2887291030 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35140bf640 con 0x7f35140c4920 2023-10-10T18:45:19.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35191d0700 1 --2- 172.21.15.175:0/2887291030 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35140c4920 0x7f3514153ed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f35100099e0 tx=0x7f3510009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9732567df6d1e10e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35189cf700 1 -- 172.21.15.175:0/2887291030 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000b7e0 con 0x7f35140c4920 2023-10-10T18:45:19.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.610+0000 7f35189cf700 1 -- 172.21.15.175:0/2887291030 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351000b940 con 0x7f35140c4920 2023-10-10T18:45:19.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.611+0000 7f351a9d3700 1 -- 172.21.15.175:0/2887291030 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35140c4920 msgr2=0x7f3514153ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.611+0000 7f351a9d3700 1 --2- 172.21.15.175:0/2887291030 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35140c4920 0x7f3514153ed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f35100099e0 tx=0x7f3510009250 comp rx=0 tx=0).stop 2023-10-10T18:45:19.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.611+0000 7f351a9d3700 1 -- 172.21.15.175:0/2887291030 shutdown_connections 2023-10-10T18:45:19.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.611+0000 7f351a9d3700 1 --2- 172.21.15.175:0/2887291030 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35140c4920 0x7f3514153ed0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f35100099e0 tx=0x7f3510009250 comp rx=0 tx=0).stop 2023-10-10T18:45:19.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.611+0000 7f351a9d3700 1 -- 172.21.15.175:0/2887291030 >> 172.21.15.175:0/2887291030 conn(0x7f35140c8a70 msgr2=0x7f35140c8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.613+0000 7f351a9d3700 1 -- 172.21.15.175:0/2887291030 shutdown_connections 2023-10-10T18:45:19.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.614+0000 7f351a9d3700 1 -- 172.21.15.175:0/2887291030 wait complete. 2023-10-10T18:45:19.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.614+0000 7feb3e8aa700 1 Processor -- start 2023-10-10T18:45:19.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.614+0000 7feb3e8aa700 1 -- start start 2023-10-10T18:45:19.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.615+0000 7f7509da7700 1 Processor -- start 2023-10-10T18:45:19.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.615+0000 7f7509da7700 1 -- start start 2023-10-10T18:45:19.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7509da7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f75040c4460 0x7f75040c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7509da7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75040ca000 0x7f7504153da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7509da7700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75040c9460 con 0x7f75040c4090 2023-10-10T18:45:19.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7509da7700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75040642f0 con 0x7f75040c4460 2023-10-10T18:45:19.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7509da7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750415a800 con 0x7f75040ca000 2023-10-10T18:45:19.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f351a9d3700 1 Processor -- start 2023-10-10T18:45:19.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb3e8aa700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7feb38068ac0 0x7feb3806d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb3e8aa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb38153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb3e8aa700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3806e080 con 0x7feb38063230 2023-10-10T18:45:19.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb3e8aa700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb380639c0 con 0x7feb38068ac0 2023-10-10T18:45:19.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb3e8aa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb38153e50 con 0x7feb3814d330 2023-10-10T18:45:19.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7502ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f75040c4460 0x7f75040c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7502ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f75040c4460 0x7f75040c5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52438/0 (socket says 172.21.15.175:52438) 2023-10-10T18:45:19.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7502ffd700 1 -- 172.21.15.175:0/48708373 learned_addr learned my addr 172.21.15.175:0/48708373 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb377fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7feb38063230 0x7feb3814cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60518/0 (socket says 172.21.15.175:60518) 2023-10-10T18:45:19.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb37fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7feb38068ac0 0x7feb3806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb377fe700 1 -- 172.21.15.175:0/423263918 learned_addr learned my addr 172.21.15.175:0/423263918 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:19.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f7503fff700 1 --2- 172.21.15.175:0/48708373 >> v2:172.21.15.191:3300/0 conn(0x7f75040ca000 0x7f7504153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb3ce47700 1 --2- 172.21.15.175:0/423263918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb38153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f351a9d3700 1 -- start start 2023-10-10T18:45:19.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f351a9d3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f351406b850 0x7f3514148480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f351a9d3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3514153660 0x7f351416e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f351a9d3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f351416e6b0 0x7f3514143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f351a9d3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3514151d50 con 0x7f3514153660 2023-10-10T18:45:19.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f351a9d3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351416dce0 con 0x7f351406b850 2023-10-10T18:45:19.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f351a9d3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35140bfc90 con 0x7f351416e6b0 2023-10-10T18:45:19.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35191d0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3514153660 0x7f351416e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35191d0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3514153660 0x7f351416e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52450/0 (socket says 172.21.15.175:52450) 2023-10-10T18:45:19.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35191d0700 1 -- 172.21.15.175:0/2575467004 learned_addr learned my addr 172.21.15.175:0/2575467004 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35199d1700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f351406b850 0x7f3514148480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f351a1d2700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f351416e6b0 0x7f3514143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35191d0700 1 -- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f351416e6b0 msgr2=0x7f3514143eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35191d0700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f351416e6b0 0x7f3514143eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35191d0700 1 -- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f351406b850 msgr2=0x7f3514148480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35191d0700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f351406b850 0x7f3514148480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35191d0700 1 -- 172.21.15.175:0/2575467004 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100094f0 con 0x7f3514153660 2023-10-10T18:45:19.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f35199d1700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f351406b850 0x7f3514148480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:19.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f35191d0700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3514153660 0x7f351416e170 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f35100094c0 tx=0x7f3510016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7fe01c72d700 1 Processor -- start 2023-10-10T18:45:19.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7fe01c72d700 1 -- start start 2023-10-10T18:45:19.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01c72d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe01415b0e0 0x7fe01415e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01c72d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe01415b960 0x7fe0141540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01c72d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0140c9320 con 0x7fe01406a950 2023-10-10T18:45:19.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01c72d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0140c2490 con 0x7fe01415b0e0 2023-10-10T18:45:19.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01c72d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01415e960 con 0x7fe01415b960 2023-10-10T18:45:19.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fc3b8b4d700 1 Processor -- start 2023-10-10T18:45:19.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f0620d59700 1 Processor -- start 2023-10-10T18:45:19.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7f0620d59700 1 -- start start 2023-10-10T18:45:19.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f0620d59700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f061c0c3d20 0x7f061c15a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f0620d59700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4d30 0x7f061c15a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f0620d59700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061c0c60b0 con 0x7f061c0c4960 2023-10-10T18:45:19.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f0620d59700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f061c15ac50 con 0x7f061c0c3d20 2023-10-10T18:45:19.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f0620d59700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c15aed0 con 0x7f061c0c4d30 2023-10-10T18:45:19.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f7503fff700 1 -- 172.21.15.175:0/48708373 >> v2:172.21.15.175:3300/0 conn(0x7f75040c4460 msgr2=0x7f75040c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f7503fff700 1 --2- 172.21.15.175:0/48708373 >> v2:172.21.15.175:3300/0 conn(0x7f75040c4460 0x7f75040c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f7503fff700 1 -- 172.21.15.175:0/48708373 >> v1:172.21.15.191:6789/0 conn(0x7f75040c4090 legacy=0x7f75040c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7f7503fff700 1 -- 172.21.15.175:0/48708373 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750415aaa0 con 0x7f75040ca000 2023-10-10T18:45:19.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb37fff700 1 -- 172.21.15.175:0/423263918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 msgr2=0x7feb38153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb37fff700 1 --2- 172.21.15.175:0/423263918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb38153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb37fff700 1 -- 172.21.15.175:0/423263918 >> v1:172.21.15.175:6789/0 conn(0x7feb38063230 legacy=0x7feb3814cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.616+0000 7feb37fff700 1 -- 172.21.15.175:0/423263918 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb380636e0 con 0x7feb38068ac0 2023-10-10T18:45:19.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7feb37fff700 1 --2- 172.21.15.175:0/423263918 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 0x7feb3806d560 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feb30009e90 tx=0x7feb30009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7dcfb071a57de66 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7feb36ffd700 1 -- 172.21.15.175:0/423263918 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3000e070 con 0x7feb38068ac0 2023-10-10T18:45:19.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f061b7fe700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f061c0c4960 0x7f061c06e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52126/0 (socket says 172.21.15.175:52126) 2023-10-10T18:45:19.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f061b7fe700 1 -- 172.21.15.175:0/60009573 learned_addr learned my addr 172.21.15.175:0/60009573 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:19.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f061bfff700 1 --2- 172.21.15.175:0/60009573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4d30 0x7f061c15a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f3509ffb700 1 -- 172.21.15.175:0/2575467004 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510000cc0 con 0x7f3514153660 2023-10-10T18:45:19.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f3509ffb700 1 -- 172.21.15.175:0/2575467004 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3510020070 con 0x7f3514153660 2023-10-10T18:45:19.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f3509ffb700 1 -- 172.21.15.175:0/2575467004 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000bed0 con 0x7f3514153660 2023-10-10T18:45:19.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3514151f80 con 0x7f3514153660 2023-10-10T18:45:19.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35140c2970 con 0x7f3514153660 2023-10-10T18:45:19.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01acca700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe01415b960 0x7fe0141540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01acca700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe01415b960 0x7fe0141540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37254/0 (socket says 172.21.15.175:37254) 2023-10-10T18:45:19.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01acca700 1 -- 172.21.15.175:0/417006347 learned_addr learned my addr 172.21.15.175:0/417006347 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe019cc8700 1 --2- 172.21.15.175:0/417006347 >> v2:172.21.15.175:3300/0 conn(0x7fe01415b0e0 0x7fe01415e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01acca700 1 -- 172.21.15.175:0/417006347 >> v2:172.21.15.175:3300/0 conn(0x7fe01415b0e0 msgr2=0x7fe01415e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01acca700 1 --2- 172.21.15.175:0/417006347 >> v2:172.21.15.175:3300/0 conn(0x7fe01415b0e0 0x7fe01415e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01acca700 1 -- 172.21.15.175:0/417006347 >> v1:172.21.15.191:6789/0 conn(0x7fe01406a950 legacy=0x7fe01406ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fe01acca700 1 -- 172.21.15.175:0/417006347 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01415ac50 con 0x7fe01415b960 2023-10-10T18:45:19.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fe01acca700 1 --2- 172.21.15.175:0/417006347 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 0x7fe0141540d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe014069290 tx=0x7fe010009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d09337121fbf87a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fe0194c7700 1 -- 172.21.15.175:0/417006347 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010009e70 con 0x7fe01415b960 2023-10-10T18:45:19.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fe0194c7700 1 -- 172.21.15.175:0/417006347 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01000b860 con 0x7fe01415b960 2023-10-10T18:45:19.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fe0194c7700 1 -- 172.21.15.175:0/417006347 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01000bc00 con 0x7fe01415b960 2023-10-10T18:45:19.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f061bfff700 1 -- 172.21.15.175:0/60009573 >> v2:172.21.15.175:3300/0 conn(0x7f061c0c3d20 msgr2=0x7f061c15a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7f061affd700 1 --2- 172.21.15.175:0/60009573 >> v2:172.21.15.175:3300/0 conn(0x7f061c0c3d20 0x7f061c15a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f061bfff700 1 --2- 172.21.15.175:0/60009573 >> v2:172.21.15.175:3300/0 conn(0x7f061c0c3d20 0x7f061c15a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f061bfff700 1 -- 172.21.15.175:0/60009573 >> v1:172.21.15.191:6789/0 conn(0x7f061c0c4960 legacy=0x7f061c06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f061bfff700 1 -- 172.21.15.175:0/60009573 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c0c5780 con 0x7f061c0c4d30 2023-10-10T18:45:19.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f061affd700 1 --2- 172.21.15.175:0/60009573 >> v2:172.21.15.175:3300/0 conn(0x7f061c0c3d20 0x7f061c15a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:19.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f061bfff700 1 --2- 172.21.15.175:0/60009573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4d30 0x7f061c15a710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0614009e60 tx=0x7f0614009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51df86e58ca5c96b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7feb36ffd700 1 -- 172.21.15.175:0/423263918 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3000bbf0 con 0x7feb38068ac0 2023-10-10T18:45:19.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.617+0000 7feb36ffd700 1 -- 172.21.15.175:0/423263918 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30012430 con 0x7feb38068ac0 2023-10-10T18:45:19.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7feb3e8aa700 1 -- 172.21.15.175:0/423263918 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 msgr2=0x7feb3806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/423263918 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 0x7feb3806d560 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feb30009e90 tx=0x7feb30009780 comp rx=0 tx=0).stop 2023-10-10T18:45:19.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f061a7fc700 1 -- 172.21.15.175:0/60009573 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061400e070 con 0x7f061c0c4d30 2023-10-10T18:45:19.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f061a7fc700 1 -- 172.21.15.175:0/60009573 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061400bb30 con 0x7f061c0c4d30 2023-10-10T18:45:19.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7f061a7fc700 1 -- 172.21.15.175:0/60009573 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614012430 con 0x7f061c0c4d30 2023-10-10T18:45:19.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fe01c72d700 1 -- 172.21.15.175:0/417006347 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 msgr2=0x7fe0141540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fe01c72d700 1 --2- 172.21.15.175:0/417006347 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 0x7fe0141540d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe014069290 tx=0x7fe010009250 comp rx=0 tx=0).stop 2023-10-10T18:45:19.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fe01c72d700 1 -- 172.21.15.175:0/417006347 shutdown_connections 2023-10-10T18:45:19.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fe01c72d700 1 --2- 172.21.15.175:0/417006347 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 0x7fe0141540d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fe01c72d700 1 --2- 172.21.15.175:0/417006347 >> v2:172.21.15.175:3300/0 conn(0x7fe01415b0e0 0x7fe01415e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fe01c72d700 1 -- 172.21.15.175:0/417006347 >> 172.21.15.175:0/417006347 conn(0x7fe014068280 msgr2=0x7fe0140c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.618+0000 7fc3b8b4d700 1 -- start start 2023-10-10T18:45:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fc3b8b4d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b4153680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fc3b8b4d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc3b406e390 0x7fc3b4152c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fc3b8b4d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b406d380 con 0x7fc3b406a950 2023-10-10T18:45:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fc3b8b4d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b406df00 con 0x7fc3b406e390 2023-10-10T18:45:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.619+0000 7fc3b8b4d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b406e780 con 0x7fc3b40c8f90 2023-10-10T18:45:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b1d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b4153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b1d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b4153680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33304/0 (socket says 172.21.15.175:33304) 2023-10-10T18:45:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b1d9b700 1 -- 172.21.15.175:0/1897648210 learned_addr learned my addr 172.21.15.175:0/1897648210 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b2d9d700 1 --2- 172.21.15.175:0/1897648210 >> v2:172.21.15.191:3300/0 conn(0x7fc3b406e390 0x7fc3b4152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b1d9b700 1 -- 172.21.15.175:0/1897648210 >> v2:172.21.15.191:3300/0 conn(0x7fc3b406e390 msgr2=0x7fc3b4152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b1d9b700 1 --2- 172.21.15.175:0/1897648210 >> v2:172.21.15.191:3300/0 conn(0x7fc3b406e390 0x7fc3b4152c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b1d9b700 1 -- 172.21.15.175:0/1897648210 >> v1:172.21.15.175:6789/0 conn(0x7fc3b406a950 legacy=0x7fc3b406ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b1d9b700 1 -- 172.21.15.175:0/1897648210 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b406b5e0 con 0x7fc3b40c8f90 2023-10-10T18:45:19.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fc3b1d9b700 1 --2- 172.21.15.175:0/1897648210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b4153680 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40089b0 tx=0x7fc3a40082d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebff8b007c0333db server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7f0620d59700 1 -- 172.21.15.175:0/60009573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4d30 msgr2=0x7f061c15a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7f0620d59700 1 --2- 172.21.15.175:0/60009573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4d30 0x7f061c15a710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0614009e60 tx=0x7f0614009780 comp rx=0 tx=0).stop 2023-10-10T18:45:19.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fe01c72d700 1 -- 172.21.15.175:0/417006347 shutdown_connections 2023-10-10T18:45:19.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7fe01c72d700 1 -- 172.21.15.175:0/417006347 wait complete. 2023-10-10T18:45:19.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7feb3e8aa700 1 -- 172.21.15.175:0/423263918 shutdown_connections 2023-10-10T18:45:19.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/423263918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb38153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/423263918 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 0x7feb3806d560 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.620+0000 7feb3e8aa700 1 -- 172.21.15.175:0/423263918 >> 172.21.15.175:0/423263918 conn(0x7feb380c56a0 msgr2=0x7feb380c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f7503fff700 1 --2- 172.21.15.175:0/48708373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040ca000 0x7f7504153da0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75040ca5b0 tx=0x7f74fc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6fd1796683ea47d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f3509ffb700 1 -- 172.21.15.175:0/2575467004 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3510022030 con 0x7f3514153660 2023-10-10T18:45:19.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35140c2b60 con 0x7f3514153660 2023-10-10T18:45:19.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f0620d59700 1 -- 172.21.15.175:0/60009573 shutdown_connections 2023-10-10T18:45:19.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f0620d59700 1 --2- 172.21.15.175:0/60009573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4d30 0x7f061c15a710 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f0620d59700 1 --2- 172.21.15.175:0/60009573 >> v2:172.21.15.175:3300/0 conn(0x7f061c0c3d20 0x7f061c15a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f0620d59700 1 -- 172.21.15.175:0/60009573 >> 172.21.15.175:0/60009573 conn(0x7f061c06be20 msgr2=0x7f061c06c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7feb3e8aa700 1 -- 172.21.15.175:0/423263918 shutdown_connections 2023-10-10T18:45:19.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7feb3e8aa700 1 -- 172.21.15.175:0/423263918 wait complete. 2023-10-10T18:45:19.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7fc3b159a700 1 -- 172.21.15.175:0/1897648210 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4010070 con 0x7fc3b40c8f90 2023-10-10T18:45:19.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f3509ffb700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34f8066f30 0x7f34f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f3509ffb700 1 -- 172.21.15.175:0/2575467004 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3510085df0 con 0x7f3514153660 2023-10-10T18:45:19.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7f35199d1700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34f8066f30 0x7f34f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7fc3b159a700 1 -- 172.21.15.175:0/1897648210 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a400db30 con 0x7fc3b40c8f90 2023-10-10T18:45:19.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7fc3b159a700 1 -- 172.21.15.175:0/1897648210 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4016400 con 0x7fc3b40c8f90 2023-10-10T18:45:19.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1897648210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 msgr2=0x7fc3b4153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1897648210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b4153680 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40089b0 tx=0x7fc3a40082d0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.621+0000 7f79e78de700 1 Processor -- start 2023-10-10T18:45:19.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7f79e78de700 1 -- start start 2023-10-10T18:45:19.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7fe01c72d700 1 Processor -- start 2023-10-10T18:45:19.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7fe01c72d700 1 -- start start 2023-10-10T18:45:19.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7f35199d1700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34f8066f30 0x7f34f80693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f35140c99d0 tx=0x7f35040090f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7f0620d59700 1 -- 172.21.15.175:0/60009573 shutdown_connections 2023-10-10T18:45:19.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.622+0000 7feb3e8aa700 1 Processor -- start 2023-10-10T18:45:19.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1897648210 shutdown_connections 2023-10-10T18:45:19.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1897648210 >> v2:172.21.15.191:3300/0 conn(0x7fc3b406e390 0x7fc3b4152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1897648210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b4153680 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1897648210 >> 172.21.15.175:0/1897648210 conn(0x7fc3b4068280 msgr2=0x7fc3b40c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7feb3e8aa700 1 -- start start 2023-10-10T18:45:19.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f75027fc700 1 -- 172.21.15.175:0/48708373 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc00e070 con 0x7f75040ca000 2023-10-10T18:45:19.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f75027fc700 1 -- 172.21.15.175:0/48708373 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc00bb60 con 0x7f75040ca000 2023-10-10T18:45:19.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f75027fc700 1 -- 172.21.15.175:0/48708373 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc012430 con 0x7f75040ca000 2023-10-10T18:45:19.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f79e78de700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e00bf810 0x7f79e00888a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f79e78de700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f79e006df20 0x7f79e006a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f79e78de700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e006d380 con 0x7f79e006ce90 2023-10-10T18:45:19.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f79e78de700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e0088de0 con 0x7f79e006df20 2023-10-10T18:45:19.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f79e78de700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e006bde0 con 0x7f79e00bf810 2023-10-10T18:45:19.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1897648210 shutdown_connections 2023-10-10T18:45:19.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1897648210 wait complete. 2023-10-10T18:45:19.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01c72d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe01406a950 0x7fe01413fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01c72d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe01415b0e0 0x7fe01413b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01c72d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 0x7fe014144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01c72d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c66b0 con 0x7fe01415b0e0 2023-10-10T18:45:19.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01c72d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01415da50 con 0x7fe01415b960 2023-10-10T18:45:19.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01c72d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe014141090 con 0x7fe01406a950 2023-10-10T18:45:19.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01a4c9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe01406a950 0x7fe01413fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01a4c9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe01406a950 0x7fe01413fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33330/0 (socket says 172.21.15.175:33330) 2023-10-10T18:45:19.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7fe01a4c9700 1 -- 172.21.15.175:0/1441172381 learned_addr learned my addr 172.21.15.175:0/1441172381 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fe01acca700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 0x7fe014144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fe019cc8700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe01415b0e0 0x7fe01413b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fe01acca700 1 -- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe01406a950 msgr2=0x7fe01413fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fe01acca700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe01406a950 0x7fe01413fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fe01acca700 1 -- 172.21.15.175:0/1441172381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe01415b0e0 msgr2=0x7fe01413b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fe01acca700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe01415b0e0 0x7fe01413b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fe01acca700 1 -- 172.21.15.175:0/1441172381 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0100094f0 con 0x7fe01415b960 2023-10-10T18:45:19.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3e8aa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb38063230 0x7feb380bca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3e8aa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 0x7feb3814ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3e8aa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb3813f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3e8aa700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb38167030 con 0x7feb38068ac0 2023-10-10T18:45:19.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3e8aa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb38166160 con 0x7feb38063230 2023-10-10T18:45:19.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3e8aa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb38166420 con 0x7feb3814d330 2023-10-10T18:45:19.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3ce47700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb3813f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3ce47700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb3813f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33336/0 (socket says 172.21.15.175:33336) 2023-10-10T18:45:19.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb3ce47700 1 -- 172.21.15.175:0/2911763456 learned_addr learned my addr 172.21.15.175:0/2911763456 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7feb377fe700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 0x7feb3814ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fe01acca700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 0x7fe014144f40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe014064c40 tx=0x7fe010017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f0620d59700 1 -- 172.21.15.175:0/60009573 wait complete. 2023-10-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7f0620d59700 1 Processor -- start 2023-10-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f79e567a700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f79e006ce90 0x7f79e006d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52132/0 (socket says 172.21.15.175:52132) 2023-10-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f79e567a700 1 -- 172.21.15.175:0/719688253 learned_addr learned my addr 172.21.15.175:0/719688253 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.623+0000 7f79e5e7b700 1 --2- 172.21.15.175:0/719688253 >> v2:172.21.15.191:3300/0 conn(0x7f79e006df20 0x7f79e006a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7f79e4e79700 1 --2- 172.21.15.175:0/719688253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e00bf810 0x7f79e00888a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7f79e5e7b700 1 -- 172.21.15.175:0/719688253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e00bf810 msgr2=0x7f79e00888a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7f79e5e7b700 1 --2- 172.21.15.175:0/719688253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e00bf810 0x7f79e00888a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7f79e5e7b700 1 -- 172.21.15.175:0/719688253 >> v1:172.21.15.191:6789/0 conn(0x7f79e006ce90 legacy=0x7f79e006d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7f79e5e7b700 1 -- 172.21.15.175:0/719688253 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e0166d90 con 0x7f79e006df20 2023-10-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7f79e5e7b700 1 --2- 172.21.15.175:0/719688253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e006df20 0x7f79e006a700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f79d80098e0 tx=0x7f79d80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c2fb899d143f225 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7fc3b8b4d700 1 Processor -- start 2023-10-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.624+0000 7f0620d59700 1 -- start start 2023-10-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7fe00a7fc700 1 -- 172.21.15.175:0/1441172381 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01000baa0 con 0x7fe01415b960 2023-10-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7fc3b8b4d700 1 -- start start 2023-10-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f79d7fff700 1 -- 172.21.15.175:0/719688253 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8009e80 con 0x7f79e006df20 2023-10-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f79d7fff700 1 -- 172.21.15.175:0/719688253 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d800b8b0 con 0x7f79e006df20 2023-10-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f79d7fff700 1 -- 172.21.15.175:0/719688253 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d800bc30 con 0x7f79e006df20 2023-10-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7feb3ce47700 1 -- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb38063230 msgr2=0x7feb380bca50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7feb3ce47700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb38063230 0x7feb380bca50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7feb3ce47700 1 -- 172.21.15.175:0/2911763456 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 msgr2=0x7feb3814ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7feb3ce47700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 0x7feb3814ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7feb3ce47700 1 -- 172.21.15.175:0/2911763456 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb30009a20 con 0x7feb3814d330 2023-10-10T18:45:19.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7fe00a7fc700 1 -- 172.21.15.175:0/1441172381 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe010025420 con 0x7fe01415b960 2023-10-10T18:45:19.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7fe00a7fc700 1 -- 172.21.15.175:0/1441172381 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100257e0 con 0x7fe01415b960 2023-10-10T18:45:19.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7feb3ce47700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb3813f4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb2800ea50 tx=0x7feb280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01415d770 con 0x7fe01415b960 2023-10-10T18:45:19.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe014140d20 con 0x7fe01415b960 2023-10-10T18:45:19.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f0620d59700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f061c0c3d20 0x7f061c1475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f0620d59700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4960 0x7f061c15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f0620d59700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f061c0c4d30 0x7f061c175da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f0620d59700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0adcd0 con 0x7f061c0c4d30 2023-10-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f0620d59700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c15b770 con 0x7f061c0c3d20 2023-10-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.625+0000 7f0620d59700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c15c010 con 0x7f061c0c4960 2023-10-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f061bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f061c0c4d30 0x7f061c175da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f061bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f061c0c4d30 0x7f061c175da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52516/0 (socket says 172.21.15.175:52516) 2023-10-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f061bfff700 1 -- 172.21.15.175:0/2593772489 learned_addr learned my addr 172.21.15.175:0/2593772489 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7fc3b8b4d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b406a950 0x7fc3b40c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7fc3b8b4d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3b406e390 0x7fc3b4152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7fc3b8b4d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b41525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7fc3b8b4d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b416e960 con 0x7fc3b406a950 2023-10-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7fc3b8b4d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b416e680 con 0x7fc3b406e390 2023-10-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7fc3b8b4d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b40c8830 con 0x7fc3b40c8f90 2023-10-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f061b7fe700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f061c0c3d20 0x7f061c1475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7feb1ffff700 1 -- 172.21.15.175:0/2911763456 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800c850 con 0x7feb3814d330 2023-10-10T18:45:19.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7feb1ffff700 1 -- 172.21.15.175:0/2911763456 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2800c9d0 con 0x7feb3814d330 2023-10-10T18:45:19.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7feb1ffff700 1 -- 172.21.15.175:0/2911763456 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800cd90 con 0x7feb3814d330 2023-10-10T18:45:19.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb38166700 con 0x7feb3814d330 2023-10-10T18:45:19.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb38166cc0 con 0x7feb3814d330 2023-10-10T18:45:19.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f7509da7700 1 -- 172.21.15.175:0/48708373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040ca000 msgr2=0x7f7504153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f7509da7700 1 --2- 172.21.15.175:0/48708373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040ca000 0x7f7504153da0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75040ca5b0 tx=0x7f74fc009780 comp rx=0 tx=0).stop 2023-10-10T18:45:19.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f79e78de700 1 -- 172.21.15.175:0/719688253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e006df20 msgr2=0x7f79e006a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f79e78de700 1 --2- 172.21.15.175:0/719688253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e006df20 0x7f79e006a700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f79d80098e0 tx=0x7f79d80091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f061b7fe700 1 -- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4960 msgr2=0x7f061c15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f061b7fe700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4960 0x7f061c15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f061b7fe700 1 -- 172.21.15.175:0/2593772489 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f061c0c4d30 msgr2=0x7f061c175da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.626+0000 7f061b7fe700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f061c0c4d30 0x7f061c175da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7f061b7fe700 1 -- 172.21.15.175:0/2593772489 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0614009a20 con 0x7f061c0c3d20 2023-10-10T18:45:19.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7f061b7fe700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f061c0c3d20 0x7f061c1475f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f060c00c7e0 tx=0x7f060c007e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7f060b7fe700 1 -- 172.21.15.175:0/2593772489 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c005b30 con 0x7f061c0c3d20 2023-10-10T18:45:19.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7f060b7fe700 1 -- 172.21.15.175:0/2593772489 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060c005c90 con 0x7f061c0c3d20 2023-10-10T18:45:19.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7f060b7fe700 1 -- 172.21.15.175:0/2593772489 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c007340 con 0x7f061c0c3d20 2023-10-10T18:45:19.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7fc3b2d9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b41525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7fc3b1d9b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3b406e390 0x7fc3b4152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7fe00a7fc700 1 -- 172.21.15.175:0/1441172381 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe010023070 con 0x7fe01415b960 2023-10-10T18:45:19.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb38067570 con 0x7feb3814d330 2023-10-10T18:45:19.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fe00a7fc700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdff8066fb0 0x7fdff8069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fe00a7fc700 1 -- 172.21.15.175:0/1441172381 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe010085aa0 con 0x7fe01415b960 2023-10-10T18:45:19.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fe01a4c9700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdff8066fb0 0x7fdff8069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7fc3b2d9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b41525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33348/0 (socket says 172.21.15.175:33348) 2023-10-10T18:45:19.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7fc3b2d9d700 1 -- 172.21.15.175:0/1708963724 learned_addr learned my addr 172.21.15.175:0/1708963724 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7fc3b259c700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b406a950 0x7fc3b40c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3b259c700 1 -- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 msgr2=0x7fc3b41525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3b259c700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b41525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3b259c700 1 -- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3b406e390 msgr2=0x7fc3b4152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3b259c700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3b406e390 0x7fc3b4152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3b259c700 1 -- 172.21.15.175:0/1708963724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a4008570 con 0x7fc3b406a950 2023-10-10T18:45:19.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3b259c700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b406a950 0x7fc3b40c73a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800c590 tx=0x7fc3a8007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061c15e0c0 con 0x7f061c0c3d20 2023-10-10T18:45:19.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.627+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061c15b9c0 con 0x7f061c0c3d20 2023-10-10T18:45:19.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061c0641d0 con 0x7f061c0c3d20 2023-10-10T18:45:19.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3a27fc700 1 -- 172.21.15.175:0/1708963724 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8011590 con 0x7fc3b406a950 2023-10-10T18:45:19.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7f7509da7700 1 -- 172.21.15.175:0/48708373 shutdown_connections 2023-10-10T18:45:19.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7f7509da7700 1 --2- 172.21.15.175:0/48708373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040ca000 0x7f7504153da0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7f7509da7700 1 --2- 172.21.15.175:0/48708373 >> v2:172.21.15.175:3300/0 conn(0x7f75040c4460 0x7f75040c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7f7509da7700 1 -- 172.21.15.175:0/48708373 >> 172.21.15.175:0/48708373 conn(0x7f750406c110 msgr2=0x7f750406c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b416dd50 con 0x7fc3b406a950 2023-10-10T18:45:19.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.629+0000 7fe01a4c9700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdff8066fb0 0x7fdff8069460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe00c007fd0 tx=0x7fe00c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.629+0000 7f3509ffb700 1 -- 172.21.15.175:0/2575467004 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3510053060 con 0x7f3514153660 2023-10-10T18:45:19.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.629+0000 7feb1ffff700 1 -- 172.21.15.175:0/2911763456 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feb28013680 con 0x7feb3814d330 2023-10-10T18:45:19.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.629+0000 7f7509da7700 1 -- 172.21.15.175:0/48708373 shutdown_connections 2023-10-10T18:45:19.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3a27fc700 1 -- 172.21.15.175:0/1708963724 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a80116f0 con 0x7fc3b406a950 2023-10-10T18:45:19.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3a27fc700 1 -- 172.21.15.175:0/1708963724 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8011a90 con 0x7fc3b406a950 2023-10-10T18:45:19.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.628+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b416e310 con 0x7fc3b406a950 2023-10-10T18:45:19.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.629+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b416ec40 con 0x7fc3b406a950 2023-10-10T18:45:19.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f79e78de700 1 -- 172.21.15.175:0/719688253 shutdown_connections 2023-10-10T18:45:19.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f79e78de700 1 --2- 172.21.15.175:0/719688253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e006df20 0x7f79e006a700 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f79e78de700 1 --2- 172.21.15.175:0/719688253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e00bf810 0x7f79e00888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f79e78de700 1 -- 172.21.15.175:0/719688253 >> 172.21.15.175:0/719688253 conn(0x7f79e0068280 msgr2=0x7f79e00c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f7509da7700 1 -- 172.21.15.175:0/48708373 wait complete. 2023-10-10T18:45:19.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.629+0000 7feb1ffff700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feb24067030 0x7feb240694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7feb1ffff700 1 -- 172.21.15.175:0/2911763456 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7feb28084e70 con 0x7feb3814d330 2023-10-10T18:45:19.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7feb37fff700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feb24067030 0x7feb240694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f060b7fe700 1 -- 172.21.15.175:0/2593772489 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f060c025430 con 0x7f061c0c3d20 2023-10-10T18:45:19.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7feb37fff700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feb24067030 0x7feb240694e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feb3000b9b0 tx=0x7feb30019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f060b7fe700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0600066f00 0x7f06000693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7fc3a27fc700 1 -- 172.21.15.175:0/1708963724 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc3a8012420 con 0x7fc3b406a950 2023-10-10T18:45:19.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f060b7fe700 1 -- 172.21.15.175:0/2593772489 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f060c084b20 con 0x7f061c0c3d20 2023-10-10T18:45:19.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.630+0000 7f061affd700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0600066f00 0x7f06000693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.631+0000 7fc3a27fc700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc394066f60 0x7fc394069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.631+0000 7f061affd700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0600066f00 0x7f06000693b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f061c06ccf0 tx=0x7f0610008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.631+0000 7fc3a27fc700 1 -- 172.21.15.175:0/1708963724 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc3a80839b0 con 0x7fc3b406a950 2023-10-10T18:45:19.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.631+0000 7fc3b1d9b700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc394066f60 0x7fc394069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.631+0000 7fc3b1d9b700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc394066f60 0x7fc394069410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40089b0 tx=0x7fc3a401c000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.632+0000 7f79e78de700 1 -- 172.21.15.175:0/719688253 shutdown_connections 2023-10-10T18:45:19.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.632+0000 7f79e78de700 1 -- 172.21.15.175:0/719688253 wait complete. 2023-10-10T18:45:19.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.632+0000 7f7509da7700 1 Processor -- start 2023-10-10T18:45:19.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.632+0000 7f7509da7700 1 -- start start 2023-10-10T18:45:19.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f7509da7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75040c4090 0x7f750415e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f7509da7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040c4460 0x7f750415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f7509da7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75040ca000 0x7f750415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f7509da7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750415d7b0 con 0x7f75040c4090 2023-10-10T18:45:19.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f7509da7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750415b7f0 con 0x7f75040c4460 2023-10-10T18:45:19.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f7509da7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750416a320 con 0x7f75040ca000 2023-10-10T18:45:19.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f75037fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75040c4090 0x7f750415e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f75037fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75040c4090 0x7f750415e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52536/0 (socket says 172.21.15.175:52536) 2023-10-10T18:45:19.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f75037fe700 1 -- 172.21.15.175:0/2927128053 learned_addr learned my addr 172.21.15.175:0/2927128053 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f7502ffd700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040c4460 0x7f750415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f7503fff700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75040ca000 0x7f750415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f79e78de700 1 Processor -- start 2023-10-10T18:45:19.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f79e78de700 1 -- start start 2023-10-10T18:45:19.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.634+0000 7feb1ffff700 1 -- 172.21.15.175:0/2911763456 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb28052240 con 0x7feb3814d330 2023-10-10T18:45:19.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.634+0000 7f060b7fe700 1 -- 172.21.15.175:0/2593772489 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f060c051ef0 con 0x7f061c0c3d20 2023-10-10T18:45:19.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f75037fe700 1 -- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75040ca000 msgr2=0x7f750415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f75037fe700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75040ca000 0x7f750415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.633+0000 7f75037fe700 1 -- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040c4460 msgr2=0x7f750415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.634+0000 7f75037fe700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040c4460 0x7f750415bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.634+0000 7f75037fe700 1 -- 172.21.15.175:0/2927128053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc009a20 con 0x7f75040c4090 2023-10-10T18:45:19.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.634+0000 7f75037fe700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75040c4090 0x7f750415e0b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f74f800c5b0 tx=0x7f74f8007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e78de700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79e006ce90 0x7f79e0160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e78de700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e006df20 0x7f79e00b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e78de700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e00bf810 0x7f79e00bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e78de700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e015e550 con 0x7f79e006ce90 2023-10-10T18:45:19.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e78de700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e015f560 con 0x7f79e00bf810 2023-10-10T18:45:19.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e78de700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00b67a0 con 0x7f79e006df20 2023-10-10T18:45:19.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e567a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79e006ce90 0x7f79e0160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e567a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79e006ce90 0x7f79e0160b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52544/0 (socket says 172.21.15.175:52544) 2023-10-10T18:45:19.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e567a700 1 -- 172.21.15.175:0/3194591716 learned_addr learned my addr 172.21.15.175:0/3194591716 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e4e79700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e006df20 0x7f79e00b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e5e7b700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e00bf810 0x7f79e00bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e567a700 1 -- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e006df20 msgr2=0x7f79e00b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e567a700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e006df20 0x7f79e00b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e567a700 1 -- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e00bf810 msgr2=0x7f79e00bdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e567a700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e00bf810 0x7f79e00bdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.635+0000 7f79e567a700 1 -- 172.21.15.175:0/3194591716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8009420 con 0x7f79e006ce90 2023-10-10T18:45:19.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.636+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdffc0052b0 con 0x7fe01415b960 2023-10-10T18:45:19.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.636+0000 7f79e567a700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79e006ce90 0x7f79e0160b00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f79d000b5e0 tx=0x7f79d0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f79d57fa700 1 -- 172.21.15.175:0/3194591716 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0011630 con 0x7f79e006ce90 2023-10-10T18:45:19.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f74eb7fe700 1 -- 172.21.15.175:0/2927128053 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8011590 con 0x7f75040c4090 2023-10-10T18:45:19.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f74eb7fe700 1 -- 172.21.15.175:0/2927128053 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f80116f0 con 0x7f75040c4090 2023-10-10T18:45:19.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750416a5c0 con 0x7f75040c4090 2023-10-10T18:45:19.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7fc3a27fc700 1 -- 172.21.15.175:0/1708963724 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3a8050c30 con 0x7fc3b406a950 2023-10-10T18:45:19.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750416ab80 con 0x7f75040c4090 2023-10-10T18:45:19.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f79d57fa700 1 -- 172.21.15.175:0/3194591716 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d0011790 con 0x7f79e006ce90 2023-10-10T18:45:19.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f79d57fa700 1 -- 172.21.15.175:0/3194591716 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0011b50 con 0x7f79e006ce90 2023-10-10T18:45:19.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e015e730 con 0x7f79e006ce90 2023-10-10T18:45:19.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.637+0000 7f74eb7fe700 1 -- 172.21.15.175:0/2927128053 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8011af0 con 0x7f75040c4090 2023-10-10T18:45:19.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.639+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e40052b0 con 0x7f75040c4090 2023-10-10T18:45:19.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.641+0000 7f74eb7fe700 1 -- 172.21.15.175:0/2927128053 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f74f801b3b0 con 0x7f75040c4090 2023-10-10T18:45:19.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.640+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e015ec70 con 0x7f79e006ce90 2023-10-10T18:45:19.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.641+0000 7f74eb7fe700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74f0066f30 0x7f74f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.641+0000 7f74eb7fe700 1 -- 172.21.15.175:0/2927128053 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f74f8083dc0 con 0x7f75040c4090 2023-10-10T18:45:19.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.641+0000 7f7502ffd700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74f0066f30 0x7f74f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.642+0000 7f7502ffd700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74f0066f30 0x7f74f00693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f75040c4c10 tx=0x7f74f4009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.647+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e015f840 con 0x7f79e006ce90 2023-10-10T18:45:19.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.647+0000 7fe00a7fc700 1 -- 172.21.15.175:0/1441172381 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe01002a080 con 0x7fe01415b960 2023-10-10T18:45:19.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.648+0000 7f74eb7fe700 1 -- 172.21.15.175:0/2927128053 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74f8051140 con 0x7f75040c4090 2023-10-10T18:45:19.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.651+0000 7f79d57fa700 1 -- 172.21.15.175:0/3194591716 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f79d001a400 con 0x7f79e006ce90 2023-10-10T18:45:19.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.652+0000 7f79d57fa700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79c4066f80 0x7f79c4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.652+0000 7f79d57fa700 1 -- 172.21.15.175:0/3194591716 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f79d0084ad0 con 0x7f79e006ce90 2023-10-10T18:45:19.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.652+0000 7f79e4e79700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79c4066f80 0x7f79c4069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.653+0000 7f79e4e79700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79c4066f80 0x7f79c4069430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f79e00c2da0 tx=0x7f79dc0097e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.660+0000 7f79d57fa700 1 -- 172.21.15.175:0/3194591716 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f79d0051dd0 con 0x7f79e006ce90 2023-10-10T18:45:19.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.673+0000 7fba07e51700 1 Processor -- start 2023-10-10T18:45:19.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.673+0000 7fba07e51700 1 -- start start 2023-10-10T18:45:19.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.674+0000 7fba07e51700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fba0006a950 0x7fba0006ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.674+0000 7fba07e51700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006df20 0x7fba000c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.674+0000 7fba07e51700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0006d380 con 0x7fba0006ce90 2023-10-10T18:45:19.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.674+0000 7fba07e51700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0006bde0 con 0x7fba0006a950 2023-10-10T18:45:19.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.674+0000 7fba07e51700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba00166d90 con 0x7fba0006df20 2023-10-10T18:45:19.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.675+0000 7fba063ee700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006df20 0x7fba000c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.675+0000 7fba063ee700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006df20 0x7fba000c1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33372/0 (socket says 172.21.15.175:33372) 2023-10-10T18:45:19.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.675+0000 7fba063ee700 1 -- 172.21.15.175:0/2380722881 learned_addr learned my addr 172.21.15.175:0/2380722881 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.682+0000 7fba04beb700 1 -- 172.21.15.175:0/2380722881 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929621063 0 0) 0x7fba0006d380 con 0x7fba0006ce90 2023-10-10T18:45:19.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.683+0000 7fba04beb700 1 -- 172.21.15.175:0/2380722881 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ec003570 con 0x7fba0006ce90 2023-10-10T18:45:19.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.683+0000 7fba063ee700 1 -- 172.21.15.175:0/2380722881 >> v2:172.21.15.191:3300/0 conn(0x7fba0006a950 msgr2=0x7fba0006ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:19.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.683+0000 7fba04beb700 1 -- 172.21.15.175:0/2380722881 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612486213 0 0) 0x7fb9ec003570 con 0x7fba0006ce90 2023-10-10T18:45:19.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.682+0000 7fba05bed700 1 --2- 172.21.15.175:0/2380722881 >> v2:172.21.15.191:3300/0 conn(0x7fba0006a950 0x7fba0006ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.684+0000 7fba063ee700 1 --2- 172.21.15.175:0/2380722881 >> v2:172.21.15.191:3300/0 conn(0x7fba0006a950 0x7fba0006ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.684+0000 7fba063ee700 1 -- 172.21.15.175:0/2380722881 >> v1:172.21.15.175:6789/0 conn(0x7fba0006ce90 legacy=0x7fba0006d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.684+0000 7fba063ee700 1 -- 172.21.15.175:0/2380722881 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00167050 con 0x7fba0006df20 2023-10-10T18:45:19.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.684+0000 7fba05bed700 1 --2- 172.21.15.175:0/2380722881 >> v2:172.21.15.191:3300/0 conn(0x7fba0006a950 0x7fba0006ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:19.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.692+0000 7fba063ee700 1 --2- 172.21.15.175:0/2380722881 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006df20 0x7fba000c1b00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0098b0 tx=0x7fb9fc007100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0fe1bebb1c15de8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.693+0000 7fba04beb700 1 -- 172.21.15.175:0/2380722881 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f80017f0 con 0x7fba0006df20 2023-10-10T18:45:19.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.693+0000 7fba04beb700 1 -- 172.21.15.175:0/2380722881 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc0090d0 con 0x7fba0006df20 2023-10-10T18:45:19.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.693+0000 7fba04beb700 1 -- 172.21.15.175:0/2380722881 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc007490 con 0x7fba0006df20 2023-10-10T18:45:19.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.694+0000 7fba07e51700 1 -- 172.21.15.175:0/2380722881 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006df20 msgr2=0x7fba000c1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.694+0000 7fba07e51700 1 --2- 172.21.15.175:0/2380722881 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006df20 0x7fba000c1b00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0098b0 tx=0x7fb9fc007100 comp rx=0 tx=0).stop 2023-10-10T18:45:19.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.694+0000 7fba07e51700 1 -- 172.21.15.175:0/2380722881 shutdown_connections 2023-10-10T18:45:19.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.694+0000 7fba07e51700 1 --2- 172.21.15.175:0/2380722881 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006df20 0x7fba000c1b00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.694+0000 7fba07e51700 1 --2- 172.21.15.175:0/2380722881 >> v2:172.21.15.191:3300/0 conn(0x7fba0006a950 0x7fba0006ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.694+0000 7fba07e51700 1 -- 172.21.15.175:0/2380722881 >> 172.21.15.175:0/2380722881 conn(0x7fba00068280 msgr2=0x7fba000c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:19.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.695+0000 7fba07e51700 1 -- 172.21.15.175:0/2380722881 shutdown_connections 2023-10-10T18:45:19.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.695+0000 7fba07e51700 1 -- 172.21.15.175:0/2380722881 wait complete. 2023-10-10T18:45:19.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.696+0000 7fba07e51700 1 Processor -- start 2023-10-10T18:45:19.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.696+0000 7fba07e51700 1 -- start start 2023-10-10T18:45:19.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba07e51700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba0006a950 0x7fba000c3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba07e51700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006ce90 0x7fba0017a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba07e51700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0006df20 0x7fba0017a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba07e51700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba000c2e80 con 0x7fba0006a950 2023-10-10T18:45:19.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba07e51700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba00157ed0 con 0x7fba0006df20 2023-10-10T18:45:19.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba07e51700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba001581b0 con 0x7fba0006ce90 2023-10-10T18:45:19.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba053ec700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006ce90 0x7fba0017a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba05bed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba0006a950 0x7fba000c3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba05bed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba0006a950 0x7fba000c3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52546/0 (socket says 172.21.15.175:52546) 2023-10-10T18:45:19.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.698+0000 7fba05bed700 1 -- 172.21.15.175:0/2241719762 learned_addr learned my addr 172.21.15.175:0/2241719762 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:19.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.699+0000 7fba05bed700 1 -- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006ce90 msgr2=0x7fba0017a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:19.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.699+0000 7fba05bed700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006ce90 0x7fba0017a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.699+0000 7fba05bed700 1 -- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0006df20 msgr2=0x7fba0017a9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T18:45:19.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.699+0000 7fba05bed700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0006df20 0x7fba0017a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:19.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.699+0000 7fba05bed700 1 -- 172.21.15.175:0/2241719762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc00d040 con 0x7fba0006a950 2023-10-10T18:45:19.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.699+0000 7fba05bed700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba0006a950 0x7fba000c3310 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000c850 tx=0x7fb9f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:19.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.700+0000 7fb9f5ffb700 1 -- 172.21.15.175:0/2241719762 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0011600 con 0x7fba0006a950 2023-10-10T18:45:19.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.700+0000 7fb9f5ffb700 1 -- 172.21.15.175:0/2241719762 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f0011760 con 0x7fba0006a950 2023-10-10T18:45:19.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.700+0000 7fb9f5ffb700 1 -- 172.21.15.175:0/2241719762 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0011b20 con 0x7fba0006a950 2023-10-10T18:45:19.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.700+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba00158490 con 0x7fba0006a950 2023-10-10T18:45:19.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.700+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0017b370 con 0x7fba0006a950 2023-10-10T18:45:19.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.703+0000 7fb9f5ffb700 1 -- 172.21.15.175:0/2241719762 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb9f00073b0 con 0x7fba0006a950 2023-10-10T18:45:19.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.704+0000 7fb9f5ffb700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ec0673b0 0x7fb9ec069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:19.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.704+0000 7fb9f5ffb700 1 -- 172.21.15.175:0/2241719762 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb9f0084510 con 0x7fba0006a950 2023-10-10T18:45:19.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.705+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e40052e0 con 0x7fba0006a950 2023-10-10T18:45:19.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.709+0000 7fba053ec700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ec0673b0 0x7fb9ec069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:19.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.713+0000 7fb9f5ffb700 1 -- 172.21.15.175:0/2241719762 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb9f0051760 con 0x7fba0006a950 2023-10-10T18:45:19.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:19.746+0000 7fba053ec700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ec0673b0 0x7fb9ec069860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba000c9290 tx=0x7fb9f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:20.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.004+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7feb38069c40 con 0x7feb3814d330 2023-10-10T18:45:20.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.007+0000 7feb1ffff700 1 -- 172.21.15.175:0/2911763456 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb28055860 con 0x7feb3814d330 2023-10-10T18:45:20.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.007+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f061c0bf990 con 0x7f061c0c3d20 2023-10-10T18:45:20.009 INFO:teuthology.orchestra.run.smithi175.stdout:55834574852 2023-10-10T18:45:20.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.012+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f35140c2b60 con 0x7f3514153660 2023-10-10T18:45:20.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.012+0000 7f3509ffb700 1 -- 172.21.15.175:0/2575467004 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3510056680 con 0x7f3514153660 2023-10-10T18:45:20.017 INFO:teuthology.orchestra.run.smithi175.stdout:55834574853 2023-10-10T18:45:20.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7f060b7fe700 1 -- 172.21.15.175:0/2593772489 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f060c055510 con 0x7f061c0c3d20 2023-10-10T18:45:20.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.017+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34f8066f30 msgr2=0x7f34f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.017+0000 7f351a9d3700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34f8066f30 0x7f34f80693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f35140c99d0 tx=0x7f35040090f0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.017+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3514153660 msgr2=0x7f351416e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.017+0000 7f351a9d3700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3514153660 0x7f351416e170 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f35100094c0 tx=0x7f3510016000 comp rx=0 tx=0).stop 2023-10-10T18:45:20.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feb24067030 msgr2=0x7feb240694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feb24067030 0x7feb240694e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feb3000b9b0 tx=0x7feb30019000 comp rx=0 tx=0).stop 2023-10-10T18:45:20.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 msgr2=0x7feb3813f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb3813f4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb2800ea50 tx=0x7feb280091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 shutdown_connections 2023-10-10T18:45:20.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feb24067030 0x7feb240694e0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feb3000b9b0 tx=0x7feb30019000 comp rx=0 tx=0).stop 2023-10-10T18:45:20.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb3814d330 0x7feb3813f4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb38063230 0x7feb380bca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.016+0000 7feb3e8aa700 1 --2- 172.21.15.175:0/2911763456 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb38068ac0 0x7feb3814ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.017+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 >> 172.21.15.175:0/2911763456 conn(0x7feb380c56a0 msgr2=0x7feb380677f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi175.stdout:55834574852 2023-10-10T18:45:20.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.019+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 shutdown_connections 2023-10-10T18:45:20.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.019+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 shutdown_connections 2023-10-10T18:45:20.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.019+0000 7f351a9d3700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34f8066f30 0x7f34f80693e0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f35140c99d0 tx=0x7f35040090f0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.019+0000 7f351a9d3700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f351416e6b0 0x7f3514143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.019+0000 7f351a9d3700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f351406b850 0x7f3514148480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.019+0000 7f351a9d3700 1 --2- 172.21.15.175:0/2575467004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3514153660 0x7f351416e170 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.019+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 >> 172.21.15.175:0/2575467004 conn(0x7f35140c8a70 msgr2=0x7f35140c8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:20.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.019+0000 7feb3e8aa700 1 -- 172.21.15.175:0/2911763456 wait complete. 2023-10-10T18:45:20.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.021+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc3b4067570 con 0x7fc3b406a950 2023-10-10T18:45:20.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.021+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0600066f00 msgr2=0x7f06000693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.021+0000 7f0620d59700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0600066f00 0x7f06000693b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f061c06ccf0 tx=0x7f0610008040 comp rx=0 tx=0).stop 2023-10-10T18:45:20.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.021+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f061c0c3d20 msgr2=0x7f061c1475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.021+0000 7f0620d59700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f061c0c3d20 0x7f061c1475f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f060c00c7e0 tx=0x7f060c007e40 comp rx=0 tx=0).stop 2023-10-10T18:45:20.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.023+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 shutdown_connections 2023-10-10T18:45:20.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.023+0000 7f0620d59700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0600066f00 0x7f06000693b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.023+0000 7f0620d59700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f061c0c4d30 0x7f061c175da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.023+0000 7f0620d59700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f061c0c3d20 0x7f061c1475f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.023+0000 7f0620d59700 1 --2- 172.21.15.175:0/2593772489 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f061c0c4960 0x7f061c15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.023+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 >> 172.21.15.175:0/2593772489 conn(0x7f061c06be20 msgr2=0x7f061c06f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:20.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.025+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 shutdown_connections 2023-10-10T18:45:20.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.025+0000 7f351a9d3700 1 -- 172.21.15.175:0/2575467004 wait complete. 2023-10-10T18:45:20.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.026+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 shutdown_connections 2023-10-10T18:45:20.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.026+0000 7f0620d59700 1 -- 172.21.15.175:0/2593772489 wait complete. 2023-10-10T18:45:20.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.029+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f74e4005830 con 0x7f75040c4090 2023-10-10T18:45:20.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.033+0000 7fc3a27fc700 1 -- 172.21.15.175:0/1708963724 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc3a8017070 con 0x7fc3b406a950 2023-10-10T18:45:20.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.033+0000 7f74eb7fe700 1 -- 172.21.15.175:0/2927128053 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f74f8021020 con 0x7f75040c4090 2023-10-10T18:45:20.035 INFO:teuthology.orchestra.run.smithi175.stdout:55834574852 2023-10-10T18:45:20.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.036+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdffc005560 con 0x7fe01415b960 2023-10-10T18:45:20.038 INFO:teuthology.orchestra.run.smithi175.stdout:55834574852 2023-10-10T18:45:20.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc394066f60 msgr2=0x7fc394069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc394066f60 0x7fc394069410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40089b0 tx=0x7fc3a401c000 comp rx=0 tx=0).stop 2023-10-10T18:45:20.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b406a950 msgr2=0x7fc3b40c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b406a950 0x7fc3b40c73a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800c590 tx=0x7fc3a8007ec0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74f0066f30 msgr2=0x7f74f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7f7509da7700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74f0066f30 0x7f74f00693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f75040c4c10 tx=0x7f74f4009250 comp rx=0 tx=0).stop 2023-10-10T18:45:20.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75040c4090 msgr2=0x7f750415e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7f7509da7700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75040c4090 0x7f750415e0b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f74f800c5b0 tx=0x7f74f8007f30 comp rx=0 tx=0).stop 2023-10-10T18:45:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 shutdown_connections 2023-10-10T18:45:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc394066f60 0x7fc394069410 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3b40c8f90 0x7fc3b41525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b406a950 0x7fc3b40c73a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 --2- 172.21.15.175:0/1708963724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3b406e390 0x7fc3b4152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 >> 172.21.15.175:0/1708963724 conn(0x7fc3b4068280 msgr2=0x7fc3b40c4420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:20.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f79e0067570 con 0x7f79e006ce90 2023-10-10T18:45:20.042 INFO:teuthology.orchestra.run.smithi175.stdout:55834574851 2023-10-10T18:45:20.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 shutdown_connections 2023-10-10T18:45:20.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f7509da7700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74f0066f30 0x7f74f00693e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f7509da7700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75040ca000 0x7f750415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f7509da7700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75040c4090 0x7f750415e0b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f7509da7700 1 --2- 172.21.15.175:0/2927128053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75040c4460 0x7f750415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 >> 172.21.15.175:0/2927128053 conn(0x7f750406c110 msgr2=0x7f7504152240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 shutdown_connections 2023-10-10T18:45:20.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7fc3b8b4d700 1 -- 172.21.15.175:0/1708963724 wait complete. 2023-10-10T18:45:20.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 shutdown_connections 2023-10-10T18:45:20.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f79d57fa700 1 -- 172.21.15.175:0/3194591716 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f79d0015020 con 0x7f79e006ce90 2023-10-10T18:45:20.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.038+0000 7f7509da7700 1 -- 172.21.15.175:0/2927128053 wait complete. 2023-10-10T18:45:20.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.037+0000 7fe00a7fc700 1 -- 172.21.15.175:0/1441172381 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe010015030 con 0x7fe01415b960 2023-10-10T18:45:20.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.5 2023-10-10T18:45:20.048 INFO:teuthology.orchestra.run.smithi175.stdout:55834574852 2023-10-10T18:45:20.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.041+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79c4066f80 msgr2=0x7f79c4069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.041+0000 7f79e78de700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79c4066f80 0x7f79c4069430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f79e00c2da0 tx=0x7f79dc0097e0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.041+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79e006ce90 msgr2=0x7f79e0160b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.041+0000 7f79e78de700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79e006ce90 0x7f79e0160b00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f79d000b5e0 tx=0x7f79d0009120 comp rx=0 tx=0).stop 2023-10-10T18:45:20.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.042+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdff8066fb0 msgr2=0x7fdff8069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.042+0000 7fe01c72d700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdff8066fb0 0x7fdff8069460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe00c007fd0 tx=0x7fe00c009040 comp rx=0 tx=0).stop 2023-10-10T18:45:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.042+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 msgr2=0x7fe014144f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.042+0000 7fe01c72d700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 0x7fe014144f40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe014064c40 tx=0x7fe010017000 comp rx=0 tx=0).stop 2023-10-10T18:45:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.044+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 shutdown_connections 2023-10-10T18:45:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.044+0000 7fe01c72d700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdff8066fb0 0x7fdff8069460 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.044+0000 7fe01c72d700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe01415b960 0x7fe014144f40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.044+0000 7fe01c72d700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe01406a950 0x7fe01413fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.044+0000 7fe01c72d700 1 --2- 172.21.15.175:0/1441172381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe01415b0e0 0x7fe01413b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.044+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 >> 172.21.15.175:0/1441172381 conn(0x7fe014068280 msgr2=0x7fe01406ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.044+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 shutdown_connections 2023-10-10T18:45:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.044+0000 7fe01c72d700 1 -- 172.21.15.175:0/1441172381 wait complete. 2023-10-10T18:45:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.046+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 shutdown_connections 2023-10-10T18:45:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.046+0000 7f79e78de700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79c4066f80 0x7f79c4069430 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.046+0000 7f79e78de700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e00bf810 0x7f79e00bdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.046+0000 7f79e78de700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79e006ce90 0x7f79e0160b00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.046+0000 7f79e78de700 1 --2- 172.21.15.175:0/3194591716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79e006df20 0x7f79e00b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.046+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 >> 172.21.15.175:0/3194591716 conn(0x7f79e0068280 msgr2=0x7f79e0165310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:20.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.0 2023-10-10T18:45:20.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.050+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 shutdown_connections 2023-10-10T18:45:20.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.050+0000 7f79e78de700 1 -- 172.21.15.175:0/3194591716 wait complete. 2023-10-10T18:45:20.056 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.2 2023-10-10T18:45:20.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.6 2023-10-10T18:45:20.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.1 2023-10-10T18:45:20.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.4 2023-10-10T18:45:20.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.3 2023-10-10T18:45:20.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.073+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb9e4005590 con 0x7fba0006a950 2023-10-10T18:45:20.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.073+0000 7fb9f5ffb700 1 -- 172.21.15.175:0/2241719762 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb9f0054d80 con 0x7fba0006a950 2023-10-10T18:45:20.075 INFO:teuthology.orchestra.run.smithi175.stdout:55834574852 2023-10-10T18:45:20.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ec0673b0 msgr2=0x7fb9ec069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ec0673b0 0x7fb9ec069860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba000c9290 tx=0x7fb9f8009040 comp rx=0 tx=0).stop 2023-10-10T18:45:20.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba0006a950 msgr2=0x7fba000c3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:20.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba0006a950 0x7fba000c3310 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000c850 tx=0x7fb9f0008040 comp rx=0 tx=0).stop 2023-10-10T18:45:20.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 shutdown_connections 2023-10-10T18:45:20.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ec0673b0 0x7fb9ec069860 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0006df20 0x7fba0017a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba0006a950 0x7fba000c3310 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 --2- 172.21.15.175:0/2241719762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba0006ce90 0x7fba0017a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:20.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 >> 172.21.15.175:0/2241719762 conn(0x7fba00068280 msgr2=0x7fba000c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:20.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 shutdown_connections 2023-10-10T18:45:20.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:20.075+0000 7fba07e51700 1 -- 172.21.15.175:0/2241719762 wait complete. 2023-10-10T18:45:20.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.7 2023-10-10T18:45:21.049 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-10T18:45:21.056 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-10T18:45:21.057 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-10T18:45:21.061 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-10T18:45:21.063 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-10T18:45:21.063 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-10T18:45:21.068 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-10T18:45:21.089 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-10T18:45:21.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.234+0000 7f61edc83700 1 Processor -- start 2023-10-10T18:45:21.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.235+0000 7f61edc83700 1 -- start start 2023-10-10T18:45:21.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.236+0000 7f61edc83700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8154870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.236+0000 7f61edc83700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e8064990 con 0x7f61e815e3f0 2023-10-10T18:45:21.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.236+0000 7f61edc83700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e80c9710 con 0x7f61e80c41f0 2023-10-10T18:45:21.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.236+0000 7f61edc83700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e815a590 con 0x7f61e81544a0 2023-10-10T18:45:21.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.237+0000 7f61e7fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f61e815e3f0 0x7f61e815a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60538/0 (socket says 172.21.15.175:60538) 2023-10-10T18:45:21.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.237+0000 7f61e7fff700 1 -- 172.21.15.175:0/1844648171 learned_addr learned my addr 172.21.15.175:0/1844648171 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:21.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.237+0000 7f61e77fe700 1 --2- 172.21.15.175:0/1844648171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.237+0000 7f61e67fc700 1 -- 172.21.15.175:0/1844648171 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265342993 0 0) 0x7f61e80c9710 con 0x7f61e80c41f0 2023-10-10T18:45:21.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.237+0000 7f61e67fc700 1 -- 172.21.15.175:0/1844648171 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61d0003570 con 0x7f61e80c41f0 2023-10-10T18:45:21.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.237+0000 7f61e77fe700 1 -- 172.21.15.175:0/1844648171 >> v1:172.21.15.191:6789/0 conn(0x7f61e80c41f0 legacy=0x7f61e806adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.237+0000 7f61e77fe700 1 -- 172.21.15.175:0/1844648171 >> v1:172.21.15.175:6789/0 conn(0x7f61e815e3f0 legacy=0x7f61e815a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.237+0000 7f61e77fe700 1 -- 172.21.15.175:0/1844648171 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e80c91b0 con 0x7f61e81544a0 2023-10-10T18:45:21.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.238+0000 7f61e77fe700 1 --2- 172.21.15.175:0/1844648171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8154870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009e90 tx=0x7f61e0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e6d6621e6e4a558 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.238+0000 7f61e67fc700 1 -- 172.21.15.175:0/1844648171 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e000e070 con 0x7f61e81544a0 2023-10-10T18:45:21.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.238+0000 7f61e67fc700 1 -- 172.21.15.175:0/1844648171 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e000bc80 con 0x7f61e81544a0 2023-10-10T18:45:21.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.238+0000 7f61e67fc700 1 -- 172.21.15.175:0/1844648171 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e0012430 con 0x7f61e81544a0 2023-10-10T18:45:21.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.239+0000 7f61edc83700 1 -- 172.21.15.175:0/1844648171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 msgr2=0x7f61e8154870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.239+0000 7f61edc83700 1 --2- 172.21.15.175:0/1844648171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8154870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009e90 tx=0x7f61e0009780 comp rx=0 tx=0).stop 2023-10-10T18:45:21.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.239+0000 7f61edc83700 1 -- 172.21.15.175:0/1844648171 shutdown_connections 2023-10-10T18:45:21.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.239+0000 7f61edc83700 1 --2- 172.21.15.175:0/1844648171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8154870 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.239+0000 7f61edc83700 1 -- 172.21.15.175:0/1844648171 >> 172.21.15.175:0/1844648171 conn(0x7f61e8068270 msgr2=0x7f61e80c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.239+0000 7f61edc83700 1 -- 172.21.15.175:0/1844648171 shutdown_connections 2023-10-10T18:45:21.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.240+0000 7f61edc83700 1 -- 172.21.15.175:0/1844648171 wait complete. 2023-10-10T18:45:21.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.241+0000 7f61edc83700 1 Processor -- start 2023-10-10T18:45:21.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.242+0000 7f61edc83700 1 -- start start 2023-10-10T18:45:21.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61edc83700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61e80c41f0 0x7f61e8143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61edc83700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61edc83700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61e815e3f0 0x7f61e80c07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61edc83700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e8159580 con 0x7f61e815e3f0 2023-10-10T18:45:21.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61edc83700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e8156fe0 con 0x7f61e80c41f0 2023-10-10T18:45:21.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61edc83700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e8159030 con 0x7f61e81544a0 2023-10-10T18:45:21.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61e6ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61e6ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33404/0 (socket says 172.21.15.175:33404) 2023-10-10T18:45:21.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61e6ffd700 1 -- 172.21.15.175:0/3440997667 learned_addr learned my addr 172.21.15.175:0/3440997667 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.243+0000 7f61e7fff700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61e815e3f0 0x7f61e80c07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.244+0000 7f61e6ffd700 1 -- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61e80c41f0 msgr2=0x7f61e8143c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T18:45:21.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.244+0000 7f61e6ffd700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61e80c41f0 0x7f61e8143c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.244+0000 7f61e6ffd700 1 -- 172.21.15.175:0/3440997667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61e815e3f0 msgr2=0x7f61e80c07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.244+0000 7f61e6ffd700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61e815e3f0 0x7f61e80c07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.244+0000 7f61e6ffd700 1 -- 172.21.15.175:0/3440997667 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e0009a20 con 0x7f61e81544a0 2023-10-10T18:45:21.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.244+0000 7f61e6ffd700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8155460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61d800c5b0 tx=0x7f61d8007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.245+0000 7f61cf7fe700 1 -- 172.21.15.175:0/3440997667 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8010510 con 0x7f61e81544a0 2023-10-10T18:45:21.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.245+0000 7f61cf7fe700 1 -- 172.21.15.175:0/3440997667 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d8010670 con 0x7f61e81544a0 2023-10-10T18:45:21.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.245+0000 7f61cf7fe700 1 -- 172.21.15.175:0/3440997667 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8010a10 con 0x7f61e81544a0 2023-10-10T18:45:21.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.245+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e81592a0 con 0x7f61e81544a0 2023-10-10T18:45:21.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.245+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e8144d00 con 0x7f61e81544a0 2023-10-10T18:45:21.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.247+0000 7f61cf7fe700 1 -- 172.21.15.175:0/3440997667 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f61d801b400 con 0x7f61e81544a0 2023-10-10T18:45:21.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.248+0000 7f61cf7fe700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61d00674e0 0x7f61d0069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.248+0000 7f61cf7fe700 1 -- 172.21.15.175:0/3440997667 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f61d8084050 con 0x7f61e81544a0 2023-10-10T18:45:21.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.250+0000 7f61e77fe700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61d00674e0 0x7f61d0069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.250+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c80052b0 con 0x7f61e81544a0 2023-10-10T18:45:21.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.250+0000 7f61e77fe700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61d00674e0 0x7f61d0069990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009e90 tx=0x7f61e0019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.255+0000 7f61cf7fe700 1 -- 172.21.15.175:0/3440997667 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61d80512a0 con 0x7f61e81544a0 2023-10-10T18:45:21.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.286+0000 7f5a8de8c700 1 Processor -- start 2023-10-10T18:45:21.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.286+0000 7f5a8de8c700 1 -- start start 2023-10-10T18:45:21.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.287+0000 7f5a8de8c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5a8806a950 0x7f5a88153680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.287+0000 7f5a8de8c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a8806e390 0x7f5a88152c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.287+0000 7f5a8de8c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8806d380 con 0x7f5a880c8f90 2023-10-10T18:45:21.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.287+0000 7f5a8de8c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8806df00 con 0x7f5a8806e390 2023-10-10T18:45:21.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.287+0000 7f5a8de8c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8806e7c0 con 0x7f5a8806a950 2023-10-10T18:45:21.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a87fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a8806e390 0x7f5a88152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a87fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a8806e390 0x7f5a88152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52558/0 (socket says 172.21.15.175:52558) 2023-10-10T18:45:21.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a87fff700 1 -- 172.21.15.175:0/3749991832 learned_addr learned my addr 172.21.15.175:0/3749991832 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a877fe700 1 --2- 172.21.15.175:0/3749991832 >> v2:172.21.15.191:3300/0 conn(0x7f5a8806a950 0x7f5a88153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a87fff700 1 -- 172.21.15.175:0/3749991832 >> v2:172.21.15.191:3300/0 conn(0x7f5a8806a950 msgr2=0x7f5a88153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:21.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a87fff700 1 --2- 172.21.15.175:0/3749991832 >> v2:172.21.15.191:3300/0 conn(0x7f5a8806a950 0x7f5a88153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a87fff700 1 -- 172.21.15.175:0/3749991832 >> v1:172.21.15.191:6789/0 conn(0x7f5a880c8f90 legacy=0x7f5a8806ad20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T18:45:21.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a87fff700 1 -- 172.21.15.175:0/3749991832 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8806b5e0 con 0x7f5a8806e390 2023-10-10T18:45:21.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.288+0000 7f5a87fff700 1 --2- 172.21.15.175:0/3749991832 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 0x7f5a88152c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5a880c5ee0 tx=0x7f5a80009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607fb790bfdbac7f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.289+0000 7f5a867fc700 1 -- 172.21.15.175:0/3749991832 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a800092e0 con 0x7f5a8806e390 2023-10-10T18:45:21.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.289+0000 7f5a867fc700 1 -- 172.21.15.175:0/3749991832 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80010450 con 0x7f5a8806e390 2023-10-10T18:45:21.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.289+0000 7f5a867fc700 1 -- 172.21.15.175:0/3749991832 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80010810 con 0x7f5a8806e390 2023-10-10T18:45:21.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.290+0000 7f5a8de8c700 1 -- 172.21.15.175:0/3749991832 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 msgr2=0x7f5a88152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.290+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/3749991832 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 0x7f5a88152c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5a880c5ee0 tx=0x7f5a80009f90 comp rx=0 tx=0).stop 2023-10-10T18:45:21.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.290+0000 7f5a8de8c700 1 -- 172.21.15.175:0/3749991832 shutdown_connections 2023-10-10T18:45:21.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.290+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/3749991832 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 0x7f5a88152c80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.290+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/3749991832 >> v2:172.21.15.191:3300/0 conn(0x7f5a8806a950 0x7f5a88153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.290+0000 7f5a8de8c700 1 -- 172.21.15.175:0/3749991832 >> 172.21.15.175:0/3749991832 conn(0x7f5a88068280 msgr2=0x7f5a880c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.291+0000 7f5a8de8c700 1 -- 172.21.15.175:0/3749991832 shutdown_connections 2023-10-10T18:45:21.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.292+0000 7f51caf5d700 1 Processor -- start 2023-10-10T18:45:21.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.291+0000 7f5a8de8c700 1 -- 172.21.15.175:0/3749991832 wait complete. 2023-10-10T18:45:21.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.293+0000 7f5a8de8c700 1 Processor -- start 2023-10-10T18:45:21.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.292+0000 7f51caf5d700 1 -- start start 2023-10-10T18:45:21.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.293+0000 7f51caf5d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c406de90 0x7f51c4063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.293+0000 7f51caf5d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f51c40c61b0 0x7f51c4068750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.293+0000 7f51caf5d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c4063770 con 0x7f51c406e260 2023-10-10T18:45:21.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.293+0000 7f51caf5d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c406f1a0 con 0x7f51c40c61b0 2023-10-10T18:45:21.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.293+0000 7f51caf5d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40682c0 con 0x7f51c406de90 2023-10-10T18:45:21.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f51c94fa700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f51c406e260 0x7f51c40c49e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60552/0 (socket says 172.21.15.175:60552) 2023-10-10T18:45:21.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f51c94fa700 1 -- 172.21.15.175:0/3645641862 learned_addr learned my addr 172.21.15.175:0/3645641862 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:21.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.293+0000 7f5a8de8c700 1 -- start start 2023-10-10T18:45:21.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a8de8c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a8806a950 0x7f5a8814e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a8de8c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 0x7f5a88176790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a8de8c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a880c8f90 0x7f5a88173290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a8de8c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8814d1d0 con 0x7f5a8806e390 2023-10-10T18:45:21.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a8de8c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a881727d0 con 0x7f5a880c8f90 2023-10-10T18:45:21.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a8de8c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8814d420 con 0x7f5a8806a950 2023-10-10T18:45:21.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a86ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 0x7f5a88176790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a877fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a8806a950 0x7f5a8814e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a877fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a8806a950 0x7f5a8814e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33416/0 (socket says 172.21.15.175:33416) 2023-10-10T18:45:21.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.294+0000 7f5a877fe700 1 -- 172.21.15.175:0/1420197156 learned_addr learned my addr 172.21.15.175:0/1420197156 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f51c8cf9700 1 --2- 172.21.15.175:0/3645641862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c406de90 0x7f51c4063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a87fff700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a880c8f90 0x7f5a88173290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a877fe700 1 -- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a880c8f90 msgr2=0x7f5a88173290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a877fe700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a880c8f90 0x7f5a88173290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a877fe700 1 -- 172.21.15.175:0/1420197156 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 msgr2=0x7f5a88176790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a877fe700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 0x7f5a88176790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a877fe700 1 -- 172.21.15.175:0/1420197156 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8000c370 con 0x7f5a8806a950 2023-10-10T18:45:21.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f51c8cf9700 1 -- 172.21.15.175:0/3645641862 >> v2:172.21.15.191:3300/0 conn(0x7f51c40c61b0 msgr2=0x7f51c4068750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:21.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f51c8cf9700 1 --2- 172.21.15.175:0/3645641862 >> v2:172.21.15.191:3300/0 conn(0x7f51c40c61b0 0x7f51c4068750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f51c8cf9700 1 -- 172.21.15.175:0/3645641862 >> v1:172.21.15.175:6789/0 conn(0x7f51c406e260 legacy=0x7f51c40c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f51c8cf9700 1 -- 172.21.15.175:0/3645641862 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c406d680 con 0x7f51c406de90 2023-10-10T18:45:21.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a877fe700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a8806a950 0x7f5a8814e440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800ec80 tx=0x7f5a780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a6f7fe700 1 -- 172.21.15.175:0/1420197156 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7800c850 con 0x7f5a8806a950 2023-10-10T18:45:21.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a6f7fe700 1 -- 172.21.15.175:0/1420197156 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7800c9b0 con 0x7f5a8806a950 2023-10-10T18:45:21.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a6f7fe700 1 -- 172.21.15.175:0/1420197156 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7800cd50 con 0x7f5a8806a950 2023-10-10T18:45:21.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8814d700 con 0x7f5a8806a950 2023-10-10T18:45:21.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f51c8cf9700 1 --2- 172.21.15.175:0/3645641862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c406de90 0x7f51c4063230 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51b40098e0 tx=0x7f51b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5086dc99c9dfebb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.295+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a88172d70 con 0x7f5a8806a950 2023-10-10T18:45:21.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.296+0000 7f51c37fe700 1 -- 172.21.15.175:0/3645641862 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4009e80 con 0x7f51c406de90 2023-10-10T18:45:21.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.296+0000 7f51c37fe700 1 -- 172.21.15.175:0/3645641862 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b400b860 con 0x7f51c406de90 2023-10-10T18:45:21.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.296+0000 7f51c37fe700 1 -- 172.21.15.175:0/3645641862 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b400bc20 con 0x7f51c406de90 2023-10-10T18:45:21.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.296+0000 7f1b06f5f700 1 Processor -- start 2023-10-10T18:45:21.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.296+0000 7fa81d052700 1 Processor -- start 2023-10-10T18:45:21.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.296+0000 7f1b06f5f700 1 -- start start 2023-10-10T18:45:21.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f5a6f7fe700 1 -- 172.21.15.175:0/1420197156 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5a78013680 con 0x7f5a8806a950 2023-10-10T18:45:21.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f5a6f7fe700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a74066f30 0x7f5a740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.296+0000 7fa81d052700 1 -- start start 2023-10-10T18:45:21.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7fa81d052700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8181546f0 0x7fa8180c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7fa81d052700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8180c4630 con 0x7fa81806adb0 2023-10-10T18:45:21.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7fa81d052700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8180c95e0 con 0x7fa81815ae60 2023-10-10T18:45:21.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7fa81d052700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8180c4120 con 0x7fa8181546f0 2023-10-10T18:45:21.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f5a6f7fe700 1 -- 172.21.15.175:0/1420197156 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5a780846c0 con 0x7f5a8806a950 2023-10-10T18:45:21.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f5a86ffd700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a74066f30 0x7f5a740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f1b06f5f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b0006f080 0x7f1b00155d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f1b06f5f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1b000bd9c0 0x7f1b000bde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f1b06f5f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0006d290 con 0x7f1b000c2b10 2023-10-10T18:45:21.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f1b06f5f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b000be340 con 0x7f1b000bd9c0 2023-10-10T18:45:21.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.297+0000 7f1b06f5f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b00151f00 con 0x7f1b0006f080 2023-10-10T18:45:21.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0675e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1b000bd9c0 0x7f1b000bde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0675e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1b000bd9c0 0x7f1b000bde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37378/0 (socket says 172.21.15.175:37378) 2023-10-10T18:45:21.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0675e700 1 -- 172.21.15.175:0/1286638593 learned_addr learned my addr 172.21.15.175:0/1286638593 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0575c700 1 --1- 172.21.15.175:0/1286638593 >> v1:172.21.15.191:6789/0 conn(0x7f1b000c2b10 0x7f1b000c4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52154/0 (socket says 172.21.15.175:52154) 2023-10-10T18:45:21.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa81759e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa81806adb0 0x7fa818164dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60566/0 (socket says 172.21.15.175:60566) 2023-10-10T18:45:21.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa81759e700 1 -- 172.21.15.175:0/3910803689 learned_addr learned my addr 172.21.15.175:0/3910803689 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:21.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa816d9d700 1 --2- 172.21.15.175:0/3910803689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8181546f0 0x7fa8180c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b05f5d700 1 --2- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b0006f080 0x7f1b00155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0675e700 1 -- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b0006f080 msgr2=0x7f1b00155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:21.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0675e700 1 --2- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b0006f080 0x7f1b00155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0675e700 1 -- 172.21.15.175:0/1286638593 >> v1:172.21.15.191:6789/0 conn(0x7f1b000c2b10 legacy=0x7f1b000c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0675e700 1 -- 172.21.15.175:0/1286638593 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b001521e0 con 0x7f1b000bd9c0 2023-10-10T18:45:21.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b05f5d700 1 --2- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b0006f080 0x7f1b00155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:21.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa815d9b700 1 -- 172.21.15.175:0/3910803689 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371451878 0 0) 0x7fa8180c4630 con 0x7fa81806adb0 2023-10-10T18:45:21.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa815d9b700 1 -- 172.21.15.175:0/3910803689 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa800003570 con 0x7fa81806adb0 2023-10-10T18:45:21.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f5a86ffd700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a74066f30 0x7f5a740693e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a881738d0 tx=0x7f5a7c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b0675e700 1 --2- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b000bd9c0 0x7f1b000bde00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1af800d1a0 tx=0x7f1af8007690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6fe8ac7269c03a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b04f5b700 1 -- 172.21.15.175:0/1286638593 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8010070 con 0x7f1b000bd9c0 2023-10-10T18:45:21.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b04f5b700 1 -- 172.21.15.175:0/1286638593 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af800dcb0 con 0x7f1b000bd9c0 2023-10-10T18:45:21.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7f1b04f5b700 1 -- 172.21.15.175:0/1286638593 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8007d90 con 0x7f1b000bd9c0 2023-10-10T18:45:21.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f51caf5d700 1 -- 172.21.15.175:0/3645641862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c406de90 msgr2=0x7f51c4063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f51caf5d700 1 --2- 172.21.15.175:0/3645641862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c406de90 0x7f51c4063230 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51b40098e0 tx=0x7f51b40091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa816d9d700 1 -- 172.21.15.175:0/3910803689 >> v1:172.21.15.191:6789/0 conn(0x7fa81815ae60 legacy=0x7fa8180c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa816d9d700 1 -- 172.21.15.175:0/3910803689 >> v1:172.21.15.175:6789/0 conn(0x7fa81806adb0 legacy=0x7fa818164dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa816d9d700 1 -- 172.21.15.175:0/3910803689 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180c4a70 con 0x7fa8181546f0 2023-10-10T18:45:21.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.298+0000 7fa816d9d700 1 --2- 172.21.15.175:0/3910803689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8181546f0 0x7fa8180c90a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009910 tx=0x7fa80c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5116e4dfa7bde0e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7fa815d9b700 1 -- 172.21.15.175:0/3910803689 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00b680 con 0x7fa8181546f0 2023-10-10T18:45:21.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7fa815d9b700 1 -- 172.21.15.175:0/3910803689 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80c00b7e0 con 0x7fa8181546f0 2023-10-10T18:45:21.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7fa815d9b700 1 -- 172.21.15.175:0/3910803689 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00bba0 con 0x7fa8181546f0 2023-10-10T18:45:21.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7fa81d052700 1 -- 172.21.15.175:0/3910803689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8181546f0 msgr2=0x7fa8180c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7fa81d052700 1 --2- 172.21.15.175:0/3910803689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8181546f0 0x7fa8180c90a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009910 tx=0x7fa80c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f1b06f5f700 1 -- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b000bd9c0 msgr2=0x7f1b000bde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b000bd9c0 0x7f1b000bde00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1af800d1a0 tx=0x7f1af8007690 comp rx=0 tx=0).stop 2023-10-10T18:45:21.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f51caf5d700 1 -- 172.21.15.175:0/3645641862 shutdown_connections 2023-10-10T18:45:21.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f51caf5d700 1 --2- 172.21.15.175:0/3645641862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c406de90 0x7f51c4063230 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f51caf5d700 1 --2- 172.21.15.175:0/3645641862 >> v2:172.21.15.191:3300/0 conn(0x7f51c40c61b0 0x7f51c4068750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f51caf5d700 1 -- 172.21.15.175:0/3645641862 >> 172.21.15.175:0/3645641862 conn(0x7f51c40c56a0 msgr2=0x7f51c40c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f1b06f5f700 1 -- 172.21.15.175:0/1286638593 shutdown_connections 2023-10-10T18:45:21.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b000bd9c0 0x7f1b000bde00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/1286638593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b0006f080 0x7f1b00155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.299+0000 7f1b06f5f700 1 -- 172.21.15.175:0/1286638593 >> 172.21.15.175:0/1286638593 conn(0x7f1b0006adc0 msgr2=0x7f1b0006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.300+0000 7f51caf5d700 1 -- 172.21.15.175:0/3645641862 shutdown_connections 2023-10-10T18:45:21.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.300+0000 7f51caf5d700 1 -- 172.21.15.175:0/3645641862 wait complete. 2023-10-10T18:45:21.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.300+0000 7f1b06f5f700 1 -- 172.21.15.175:0/1286638593 shutdown_connections 2023-10-10T18:45:21.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.300+0000 7f1b06f5f700 1 -- 172.21.15.175:0/1286638593 wait complete. 2023-10-10T18:45:21.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.301+0000 7f1b06f5f700 1 Processor -- start 2023-10-10T18:45:21.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.301+0000 7f1b06f5f700 1 -- start start 2023-10-10T18:45:21.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.301+0000 7f51caf5d700 1 Processor -- start 2023-10-10T18:45:21.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.301+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a88067570 con 0x7f5a8806a950 2023-10-10T18:45:21.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.300+0000 7fcfdca4d700 1 Processor -- start 2023-10-10T18:45:21.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.300+0000 7fcfdca4d700 1 -- start start 2023-10-10T18:45:21.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.301+0000 7fa81d052700 1 -- 172.21.15.175:0/3910803689 shutdown_connections 2023-10-10T18:45:21.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fa81d052700 1 --2- 172.21.15.175:0/3910803689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8181546f0 0x7fa8180c90a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fa81d052700 1 -- 172.21.15.175:0/3910803689 >> 172.21.15.175:0/3910803689 conn(0x7fa818068270 msgr2=0x7fa8180c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b06f5f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b0006f080 0x7f1b00173360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b06f5f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b000bd9c0 0x7f1b000c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b06f5f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b000c2b10 0x7f1b000bf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b06f5f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000a7350 con 0x7f1b000c2b10 2023-10-10T18:45:21.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b06f5f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000a64a0 con 0x7f1b0006f080 2023-10-10T18:45:21.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b06f5f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000a6740 con 0x7f1b000bd9c0 2023-10-10T18:45:21.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b0675e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b000c2b10 0x7f1b000bf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b0675e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b000c2b10 0x7f1b000bf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52584/0 (socket says 172.21.15.175:52584) 2023-10-10T18:45:21.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b0675e700 1 -- 172.21.15.175:0/4037378256 learned_addr learned my addr 172.21.15.175:0/4037378256 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b0575c700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b000bd9c0 0x7f1b000c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f1b05f5d700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b0006f080 0x7f1b00173360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.301+0000 7f51caf5d700 1 -- start start 2023-10-10T18:45:21.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f51caf5d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51c406de90 0x7f51c4154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f51caf5d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51c406e260 0x7f51c41576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f51caf5d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c40c61b0 0x7f51c4144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f51caf5d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c415e8d0 con 0x7f51c406e260 2023-10-10T18:45:21.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f51caf5d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4140c50 con 0x7f51c406de90 2023-10-10T18:45:21.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7f51caf5d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c413b380 con 0x7f51c40c61b0 2023-10-10T18:45:21.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c8cf9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51c406de90 0x7f51c4154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c8cf9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51c406de90 0x7f51c4154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37404/0 (socket says 172.21.15.175:37404) 2023-10-10T18:45:21.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c8cf9700 1 -- 172.21.15.175:0/2647847814 learned_addr learned my addr 172.21.15.175:0/2647847814 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c3fff700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51c406e260 0x7f51c41576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c94fa700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c40c61b0 0x7f51c4144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c3fff700 1 -- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c40c61b0 msgr2=0x7f51c4144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c3fff700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c40c61b0 0x7f51c4144590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c3fff700 1 -- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51c406de90 msgr2=0x7f51c4154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c3fff700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51c406de90 0x7f51c4154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c3fff700 1 -- 172.21.15.175:0/2647847814 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b4009420 con 0x7f51c406e260 2023-10-10T18:45:21.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f743eb7f700 1 Processor -- start 2023-10-10T18:45:21.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7fa81d052700 1 -- 172.21.15.175:0/3910803689 shutdown_connections 2023-10-10T18:45:21.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7fa81d052700 1 -- 172.21.15.175:0/3910803689 wait complete. 2023-10-10T18:45:21.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f1b05f5d700 1 -- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b000bd9c0 msgr2=0x7f1b000c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f1b05f5d700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b000bd9c0 0x7f1b000c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f1b05f5d700 1 -- 172.21.15.175:0/4037378256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b000c2b10 msgr2=0x7f1b000bf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f1b05f5d700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b000c2b10 0x7f1b000bf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f1b05f5d700 1 -- 172.21.15.175:0/4037378256 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af800be70 con 0x7f1b0006f080 2023-10-10T18:45:21.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f1b05f5d700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b0006f080 0x7f1b00173360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c880 tx=0x7f1af0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfdca4d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcfd806ce90 0x7fcfd806d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfdca4d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcfd80bb4b0 0x7fcfd806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfdca4d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd806d380 con 0x7fcfd806a8c0 2023-10-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfdca4d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd806b3a0 con 0x7fcfd806ce90 2023-10-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfdca4d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd806bde0 con 0x7fcfd80bb4b0 2023-10-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfd6d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcfd806a8c0 0x7fcfd806ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60568/0 (socket says 172.21.15.175:60568) 2023-10-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfd6d9d700 1 -- 172.21.15.175:0/1893028330 learned_addr learned my addr 172.21.15.175:0/1893028330 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfd659c700 1 --2- 172.21.15.175:0/1893028330 >> v2:172.21.15.175:3300/0 conn(0x7fcfd806ce90 0x7fcfd806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfd5d9b700 1 --2- 172.21.15.175:0/1893028330 >> v2:172.21.15.191:3300/0 conn(0x7fcfd80bb4b0 0x7fcfd806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfd559a700 1 -- 172.21.15.175:0/1893028330 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392474217 0 0) 0x7fcfd806d380 con 0x7fcfd806a8c0 2023-10-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.302+0000 7fcfd559a700 1 -- 172.21.15.175:0/1893028330 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc0003570 con 0x7fcfd806a8c0 2023-10-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7fcfd659c700 1 -- 172.21.15.175:0/1893028330 >> v2:172.21.15.191:3300/0 conn(0x7fcfd80bb4b0 msgr2=0x7fcfd806dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7fcfd659c700 1 --2- 172.21.15.175:0/1893028330 >> v2:172.21.15.191:3300/0 conn(0x7fcfd80bb4b0 0x7fcfd806dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7fcfd659c700 1 -- 172.21.15.175:0/1893028330 >> v1:172.21.15.175:6789/0 conn(0x7fcfd806a8c0 legacy=0x7fcfd806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7fcfd659c700 1 -- 172.21.15.175:0/1893028330 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd8166d90 con 0x7fcfd806ce90 2023-10-10T18:45:21.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f743eb7f700 1 -- start start 2023-10-10T18:45:21.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c3fff700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51c406e260 0x7f51c41576d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f51b800b5e0 tx=0x7f51b80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.303+0000 7f51c0ff9700 1 -- 172.21.15.175:0/2647847814 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800f510 con 0x7f51c406e260 2023-10-10T18:45:21.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.304+0000 7f51c0ff9700 1 -- 172.21.15.175:0/2647847814 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b800f670 con 0x7f51c406e260 2023-10-10T18:45:21.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.304+0000 7f51c0ff9700 1 -- 172.21.15.175:0/2647847814 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800fa30 con 0x7f51c406e260 2023-10-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.304+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c41414b0 con 0x7f51c406e260 2023-10-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.304+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c415e370 con 0x7f51c406e260 2023-10-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f743eb7f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f74380c3460 0x7f7438068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f743eb7f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74380b1020 0x7f7438068b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f743eb7f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743815a980 con 0x7f7438159cd0 2023-10-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f743eb7f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74380c1bd0 con 0x7f74380c3460 2023-10-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f743eb7f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74380af970 con 0x7f74380b1020 2023-10-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f7437fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f74380c3460 0x7f7438068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f7437fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f74380c3460 0x7f7438068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52590/0 (socket says 172.21.15.175:52590) 2023-10-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f7437fff700 1 -- 172.21.15.175:0/2986252171 learned_addr learned my addr 172.21.15.175:0/2986252171 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7fa81d052700 1 Processor -- start 2023-10-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.304+0000 7f1af5ffb700 1 -- 172.21.15.175:0/4037378256 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0011600 con 0x7f1b0006f080 2023-10-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.304+0000 7f1af5ffb700 1 -- 172.21.15.175:0/4037378256 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af0011760 con 0x7f1b0006f080 2023-10-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.304+0000 7f1af5ffb700 1 -- 172.21.15.175:0/4037378256 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0011ae0 con 0x7f1b0006f080 2023-10-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b000a6a20 con 0x7f1b0006f080 2023-10-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b000a6fe0 con 0x7f1b0006f080 2023-10-10T18:45:21.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.304+0000 7fcfd659c700 1 --2- 172.21.15.175:0/1893028330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806ce90 0x7fcfd806d810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8064a30 tx=0x7fcfc80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cd8667ed316250 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f7437fff700 1 -- 172.21.15.175:0/2986252171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74380b1020 msgr2=0x7f7438068b60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T18:45:21.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f7437fff700 1 --2- 172.21.15.175:0/2986252171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74380b1020 0x7f7438068b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f7437fff700 1 -- 172.21.15.175:0/2986252171 >> v1:172.21.15.191:6789/0 conn(0x7f7438159cd0 legacy=0x7f743815ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f7437fff700 1 -- 172.21.15.175:0/2986252171 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74380c1e70 con 0x7f74380c3460 2023-10-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c4141640 con 0x7f51c406e260 2023-10-10T18:45:21.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.305+0000 7fa81d052700 1 -- start start 2023-10-10T18:45:21.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fcfd559a700 1 -- 172.21.15.175:0/1893028330 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8022640 con 0x7fcfd806ce90 2023-10-10T18:45:21.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fcfd559a700 1 -- 172.21.15.175:0/1893028330 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc80227c0 con 0x7fcfd806ce90 2023-10-10T18:45:21.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fcfd559a700 1 -- 172.21.15.175:0/1893028330 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8022b80 con 0x7fcfd806ce90 2023-10-10T18:45:21.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7f7437fff700 1 --2- 172.21.15.175:0/2986252171 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74380c3460 0x7f7438068620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f742c0099b0 tx=0x7f742c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518138ccaf6d2fd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7f74377fe700 1 -- 172.21.15.175:0/2986252171 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c00b820 con 0x7f74380c3460 2023-10-10T18:45:21.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7f74377fe700 1 -- 172.21.15.175:0/2986252171 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c00b980 con 0x7f74380c3460 2023-10-10T18:45:21.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7f74377fe700 1 -- 172.21.15.175:0/2986252171 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c00bd20 con 0x7f74380c3460 2023-10-10T18:45:21.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81d052700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81806adb0 0x7fa8181561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81d052700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa8181546f0 0x7fa818158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81d052700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81815ae60 0x7fa8181588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81d052700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa818157ed0 con 0x7fa8181546f0 2023-10-10T18:45:21.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81d052700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81816b1b0 con 0x7fa81815ae60 2023-10-10T18:45:21.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81d052700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81816b490 con 0x7fa81806adb0 2023-10-10T18:45:21.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81759e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81815ae60 0x7fa8181588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa816d9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81806adb0 0x7fa8181561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81759e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81815ae60 0x7fa8181588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37418/0 (socket says 172.21.15.175:37418) 2023-10-10T18:45:21.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81759e700 1 -- 172.21.15.175:0/288384913 learned_addr learned my addr 172.21.15.175:0/288384913 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.306+0000 7fa81659c700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa8181546f0 0x7fa818158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa81759e700 1 -- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81806adb0 msgr2=0x7fa8181561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa81759e700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81806adb0 0x7fa8181561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa81759e700 1 -- 172.21.15.175:0/288384913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa8181546f0 msgr2=0x7fa818158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa81759e700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa8181546f0 0x7fa818158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa81759e700 1 -- 172.21.15.175:0/288384913 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c009420 con 0x7fa81815ae60 2023-10-10T18:45:21.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa81759e700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81815ae60 0x7fa8181588a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa8080010e0 tx=0x7fa808007dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f743eb7f700 1 -- 172.21.15.175:0/2986252171 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74380c3460 msgr2=0x7f7438068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f743eb7f700 1 --2- 172.21.15.175:0/2986252171 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74380c3460 0x7f7438068620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f742c0099b0 tx=0x7f742c009250 comp rx=0 tx=0).stop 2023-10-10T18:45:21.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa7feffd700 1 -- 172.21.15.175:0/288384913 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808005c00 con 0x7fa81815ae60 2023-10-10T18:45:21.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa7feffd700 1 -- 172.21.15.175:0/288384913 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa808005d60 con 0x7fa81815ae60 2023-10-10T18:45:21.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fa7feffd700 1 -- 172.21.15.175:0/288384913 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8080072e0 con 0x7fa81815ae60 2023-10-10T18:45:21.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0006ba20 con 0x7f1b0006f080 2023-10-10T18:45:21.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f743eb7f700 1 -- 172.21.15.175:0/2986252171 shutdown_connections 2023-10-10T18:45:21.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f743eb7f700 1 --2- 172.21.15.175:0/2986252171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74380b1020 0x7f7438068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f743eb7f700 1 --2- 172.21.15.175:0/2986252171 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74380c3460 0x7f7438068620 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f743eb7f700 1 -- 172.21.15.175:0/2986252171 >> 172.21.15.175:0/2986252171 conn(0x7f74380cc820 msgr2=0x7f74380ccc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f743eb7f700 1 -- 172.21.15.175:0/2986252171 shutdown_connections 2023-10-10T18:45:21.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f1af5ffb700 1 -- 172.21.15.175:0/4037378256 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1af0022350 con 0x7f1b0006f080 2023-10-10T18:45:21.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.308+0000 7f1af5ffb700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ae4067080 0x7f1ae4069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.308+0000 7f1af5ffb700 1 -- 172.21.15.175:0/4037378256 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1af0084420 con 0x7f1b0006f080 2023-10-10T18:45:21.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.308+0000 7f1b0575c700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ae4067080 0x7f1ae4069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f743eb7f700 1 -- 172.21.15.175:0/2986252171 wait complete. 2023-10-10T18:45:21.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.309+0000 7f1b0575c700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ae4067080 0x7f1ae4069530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1b000c9020 tx=0x7f1afc007eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.309+0000 7f743eb7f700 1 Processor -- start 2023-10-10T18:45:21.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.308+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8180bee50 con 0x7fa81815ae60 2023-10-10T18:45:21.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.308+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8180bf410 con 0x7fa81815ae60 2023-10-10T18:45:21.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7f51c0ff9700 1 -- 172.21.15.175:0/2647847814 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f51b801a400 con 0x7f51c406e260 2023-10-10T18:45:21.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.308+0000 7f51c0ff9700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51b0066fe0 0x7f51b0069490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.308+0000 7f51c0ff9700 1 -- 172.21.15.175:0/2647847814 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f51b8084040 con 0x7f51c406e260 2023-10-10T18:45:21.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.309+0000 7f51c8cf9700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51b0066fe0 0x7f51b0069490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fcfdca4d700 1 -- 172.21.15.175:0/1893028330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806ce90 msgr2=0x7fcfd806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/1893028330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806ce90 0x7fcfd806d810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8064a30 tx=0x7fcfc80091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fcfdca4d700 1 -- 172.21.15.175:0/1893028330 shutdown_connections 2023-10-10T18:45:21.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/1893028330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806ce90 0x7fcfd806d810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/1893028330 >> v2:172.21.15.191:3300/0 conn(0x7fcfd80bb4b0 0x7fcfd806dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fcfdca4d700 1 -- 172.21.15.175:0/1893028330 >> 172.21.15.175:0/1893028330 conn(0x7fcfd8068280 msgr2=0x7fcfd80c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fcfdca4d700 1 -- 172.21.15.175:0/1893028330 shutdown_connections 2023-10-10T18:45:21.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.307+0000 7fcfdca4d700 1 -- 172.21.15.175:0/1893028330 wait complete. 2023-10-10T18:45:21.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.309+0000 7fcfdca4d700 1 Processor -- start 2023-10-10T18:45:21.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.309+0000 7fcfdca4d700 1 -- start start 2023-10-10T18:45:21.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.309+0000 7f743eb7f700 1 -- start start 2023-10-10T18:45:21.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.309+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa818158040 con 0x7fa81815ae60 2023-10-10T18:45:21.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.309+0000 7f51c8cf9700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51b0066fe0 0x7f51b0069490 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f51b4013000 tx=0x7f51b4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f5a6f7fe700 1 -- 172.21.15.175:0/1420197156 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a78051930 con 0x7f5a8806a950 2023-10-10T18:45:21.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743eb7f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74380b1020 0x7f743815ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743eb7f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74380c3460 0x7f743806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743eb7f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7438159cd0 0x7f74381579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743eb7f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743806c700 con 0x7f7438159cd0 2023-10-10T18:45:21.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743eb7f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743806a8e0 con 0x7f74380c3460 2023-10-10T18:45:21.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfdca4d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806a8c0 0x7fcfd8176a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfdca4d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd806ce90 0x7fcfd8173420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfdca4d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd80bb4b0 0x7fcfd8165ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfdca4d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd8163390 con 0x7fcfd806a8c0 2023-10-10T18:45:21.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfdca4d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd8162e40 con 0x7fcfd80bb4b0 2023-10-10T18:45:21.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfdca4d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd8173960 con 0x7fcfd806ce90 2023-10-10T18:45:21.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743eb7f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74380c6df0 con 0x7f74380b1020 2023-10-10T18:45:21.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfd659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806a8c0 0x7fcfd8176a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfd659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806a8c0 0x7fcfd8176a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52602/0 (socket says 172.21.15.175:52602) 2023-10-10T18:45:21.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfd659c700 1 -- 172.21.15.175:0/3083677165 learned_addr learned my addr 172.21.15.175:0/3083677165 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7fcfd6d9d700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd80bb4b0 0x7fcfd8165ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743d11c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7438159cd0 0x7f74381579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743d11c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7438159cd0 0x7f74381579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52604/0 (socket says 172.21.15.175:52604) 2023-10-10T18:45:21.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.310+0000 7f743d11c700 1 -- 172.21.15.175:0/1673383440 learned_addr learned my addr 172.21.15.175:0/1673383440 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7f7437fff700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74380c3460 0x7f743806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7fcfd659c700 1 -- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd806ce90 msgr2=0x7fcfd8173420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T18:45:21.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7fcfd659c700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd806ce90 0x7fcfd8173420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7fcfd659c700 1 -- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd80bb4b0 msgr2=0x7fcfd8165ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7fcfd659c700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd80bb4b0 0x7fcfd8165ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7fcfd659c700 1 -- 172.21.15.175:0/3083677165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc8009420 con 0x7fcfd806a8c0 2023-10-10T18:45:21.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7f743d11c700 1 -- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74380b1020 msgr2=0x7f743815ecc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T18:45:21.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7f743d11c700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74380b1020 0x7f743815ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7f743d11c700 1 -- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74380c3460 msgr2=0x7f743806ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7f743d11c700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74380c3460 0x7f743806ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7f743d11c700 1 -- 172.21.15.175:0/1673383440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c0094f0 con 0x7f7438159cd0 2023-10-10T18:45:21.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7fcfd659c700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806a8c0 0x7fcfd8176a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8064cc0 tx=0x7fcfc8028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.311+0000 7f743d11c700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7438159cd0 0x7f74381579f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f742800b4e0 tx=0x7f7428009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.312+0000 7f1af5ffb700 1 -- 172.21.15.175:0/4037378256 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1af0051670 con 0x7f1b0006f080 2023-10-10T18:45:21.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.313+0000 7fa7feffd700 1 -- 172.21.15.175:0/288384913 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa80801b3b0 con 0x7fa81815ae60 2023-10-10T18:45:21.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.314+0000 7fa7feffd700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa804066f30 0x7fa8040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.314+0000 7fa7feffd700 1 -- 172.21.15.175:0/288384913 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa808083de0 con 0x7fa81815ae60 2023-10-10T18:45:21.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.314+0000 7fa816d9d700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa804066f30 0x7fa8040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.314+0000 7fa816d9d700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa804066f30 0x7fa8040693e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0093f0 tx=0x7fa80c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.316+0000 7f51c0ff9700 1 -- 172.21.15.175:0/2647847814 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51b8051290 con 0x7f51c406e260 2023-10-10T18:45:21.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.316+0000 7fcfc67fc700 1 -- 172.21.15.175:0/3083677165 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8030070 con 0x7fcfd806a8c0 2023-10-10T18:45:21.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.316+0000 7fcfc67fc700 1 -- 172.21.15.175:0/3083677165 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc802b5f0 con 0x7fcfd806a8c0 2023-10-10T18:45:21.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.316+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd81630b0 con 0x7fcfd806a8c0 2023-10-10T18:45:21.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.316+0000 7fcfc67fc700 1 -- 172.21.15.175:0/3083677165 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802b970 con 0x7fcfd806a8c0 2023-10-10T18:45:21.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.316+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd8172f00 con 0x7fcfd806a8c0 2023-10-10T18:45:21.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.317+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd8172a80 con 0x7fcfd806a8c0 2023-10-10T18:45:21.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.318+0000 7fa7feffd700 1 -- 172.21.15.175:0/288384913 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa8080510e0 con 0x7fa81815ae60 2023-10-10T18:45:21.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.320+0000 7f7434ff9700 1 -- 172.21.15.175:0/1673383440 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742800f510 con 0x7f7438159cd0 2023-10-10T18:45:21.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.320+0000 7f7434ff9700 1 -- 172.21.15.175:0/1673383440 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742800f690 con 0x7f7438159cd0 2023-10-10T18:45:21.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.320+0000 7f7434ff9700 1 -- 172.21.15.175:0/1673383440 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742800fa50 con 0x7f7438159cd0 2023-10-10T18:45:21.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.321+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74380af1e0 con 0x7f7438159cd0 2023-10-10T18:45:21.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.321+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7438069fb0 con 0x7f7438159cd0 2023-10-10T18:45:21.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.322+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743806abc0 con 0x7f7438159cd0 2023-10-10T18:45:21.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.338+0000 7fcfc67fc700 1 -- 172.21.15.175:0/3083677165 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcfc8034070 con 0x7fcfd806a8c0 2023-10-10T18:45:21.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.338+0000 7fcfc67fc700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb8067080 0x7fcfb8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.338+0000 7fcfc67fc700 1 -- 172.21.15.175:0/3083677165 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fcfc8096ba0 con 0x7fcfd806a8c0 2023-10-10T18:45:21.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.338+0000 7f7434ff9700 1 -- 172.21.15.175:0/1673383440 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7428024430 con 0x7f7438159cd0 2023-10-10T18:45:21.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.339+0000 7f7434ff9700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7424067030 0x7f74240694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.339+0000 7f7434ff9700 1 -- 172.21.15.175:0/1673383440 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7428084b50 con 0x7f7438159cd0 2023-10-10T18:45:21.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.338+0000 7fcfc67fc700 1 -- 172.21.15.175:0/3083677165 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcfc80c1640 con 0x7fcfd806a8c0 2023-10-10T18:45:21.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.339+0000 7f743c91b700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7424067030 0x7f74240694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.340+0000 7f743c91b700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7424067030 0x7f74240694e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f743000ad10 tx=0x7f7430009370 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.346+0000 7f7434ff9700 1 -- 172.21.15.175:0/1673383440 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7428051ed0 con 0x7f7438159cd0 2023-10-10T18:45:21.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.349+0000 7fcfd5d9b700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb8067080 0x7fcfb8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.349+0000 7fcfd5d9b700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb8067080 0x7fcfb8069530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8165260 tx=0x7fcfcc0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.354+0000 7f074fbd2700 1 Processor -- start 2023-10-10T18:45:21.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.354+0000 7f074fbd2700 1 -- start start 2023-10-10T18:45:21.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.355+0000 7f074fbd2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f07480bb4b0 0x7f07480c2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.355+0000 7f074fbd2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074806d380 con 0x7f074806ce90 2023-10-10T18:45:21.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.355+0000 7f074fbd2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0748166d90 con 0x7f074806df20 2023-10-10T18:45:21.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.355+0000 7f074fbd2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0748167050 con 0x7f07480bb4b0 2023-10-10T18:45:21.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.356+0000 7f074e16f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f07480bb4b0 0x7f07480c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.356+0000 7f074e16f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f07480bb4b0 0x7f07480c2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52616/0 (socket says 172.21.15.175:52616) 2023-10-10T18:45:21.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.356+0000 7f074e16f700 1 -- 172.21.15.175:0/2510922200 learned_addr learned my addr 172.21.15.175:0/2510922200 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.356+0000 7f074e16f700 1 -- 172.21.15.175:0/2510922200 >> v1:172.21.15.191:6789/0 conn(0x7f074806df20 legacy=0x7f07480c1b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T18:45:21.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.356+0000 7f074e16f700 1 -- 172.21.15.175:0/2510922200 >> v1:172.21.15.175:6789/0 conn(0x7f074806ce90 legacy=0x7f074806d810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T18:45:21.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.356+0000 7f074e16f700 1 -- 172.21.15.175:0/2510922200 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480c8b40 con 0x7f07480bb4b0 2023-10-10T18:45:21.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.356+0000 7f074e16f700 1 --2- 172.21.15.175:0/2510922200 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07480bb4b0 0x7f07480c2210 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0738013260 tx=0x7f0738011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54cd85eeaef361ac server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.357+0000 7f074c96c700 1 -- 172.21.15.175:0/2510922200 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738016070 con 0x7f07480bb4b0 2023-10-10T18:45:21.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.357+0000 7f074c96c700 1 -- 172.21.15.175:0/2510922200 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07380110d0 con 0x7f07480bb4b0 2023-10-10T18:45:21.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.357+0000 7f074c96c700 1 -- 172.21.15.175:0/2510922200 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738011470 con 0x7f07480bb4b0 2023-10-10T18:45:21.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.358+0000 7f074fbd2700 1 -- 172.21.15.175:0/2510922200 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07480bb4b0 msgr2=0x7f07480c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.358+0000 7f074fbd2700 1 --2- 172.21.15.175:0/2510922200 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07480bb4b0 0x7f07480c2210 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0738013260 tx=0x7f0738011af0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.358+0000 7f074fbd2700 1 -- 172.21.15.175:0/2510922200 shutdown_connections 2023-10-10T18:45:21.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.358+0000 7f074fbd2700 1 --2- 172.21.15.175:0/2510922200 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07480bb4b0 0x7f07480c2210 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.358+0000 7f074fbd2700 1 -- 172.21.15.175:0/2510922200 >> 172.21.15.175:0/2510922200 conn(0x7f0748068280 msgr2=0x7f07480c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.359+0000 7f074fbd2700 1 -- 172.21.15.175:0/2510922200 shutdown_connections 2023-10-10T18:45:21.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.371+0000 7f074fbd2700 1 -- 172.21.15.175:0/2510922200 wait complete. 2023-10-10T18:45:21.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.372+0000 7f074fbd2700 1 Processor -- start 2023-10-10T18:45:21.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.372+0000 7f074fbd2700 1 -- start start 2023-10-10T18:45:21.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074fbd2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f074806ce90 0x7f0748198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074fbd2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07480bb4b0 0x7f07480c3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074fbd2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f07481639a0 0x7f07480c3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074fbd2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07481628c0 con 0x7f074806ce90 2023-10-10T18:45:21.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074fbd2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07480c2ba0 con 0x7f07480bb4b0 2023-10-10T18:45:21.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074fbd2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0748161af0 con 0x7f07481639a0 2023-10-10T18:45:21.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074d96e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f074806ce90 0x7f0748198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074d96e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f074806ce90 0x7f0748198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52632/0 (socket says 172.21.15.175:52632) 2023-10-10T18:45:21.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.374+0000 7f074d96e700 1 -- 172.21.15.175:0/3443665753 learned_addr learned my addr 172.21.15.175:0/3443665753 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f074e16f700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f07481639a0 0x7f07480c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f074d96e700 1 -- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f07481639a0 msgr2=0x7f07480c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f074d96e700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f07481639a0 0x7f07480c3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f074d96e700 1 -- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07480bb4b0 msgr2=0x7f07480c3030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T18:45:21.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f074d96e700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07480bb4b0 0x7f07480c3030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f074d96e700 1 -- 172.21.15.175:0/3443665753 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0738011d90 con 0x7f074806ce90 2023-10-10T18:45:21.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f074d96e700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f074806ce90 0x7f0748198370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f074000bc30 tx=0x7f0740009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f073dffb700 1 -- 172.21.15.175:0/3443665753 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074000c9b0 con 0x7f074806ce90 2023-10-10T18:45:21.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f073dffb700 1 -- 172.21.15.175:0/3443665753 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074000cb70 con 0x7f074806ce90 2023-10-10T18:45:21.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f073dffb700 1 -- 172.21.15.175:0/3443665753 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740014410 con 0x7f074806ce90 2023-10-10T18:45:21.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.375+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0748198db0 con 0x7f074806ce90 2023-10-10T18:45:21.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.376+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0748162070 con 0x7f074806ce90 2023-10-10T18:45:21.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.378+0000 7f073dffb700 1 -- 172.21.15.175:0/3443665753 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0740014840 con 0x7f074806ce90 2023-10-10T18:45:21.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.378+0000 7f073dffb700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f07340671e0 0x7f0734069690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.379+0000 7f073dffb700 1 -- 172.21.15.175:0/3443665753 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0740084790 con 0x7f074806ce90 2023-10-10T18:45:21.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.381+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07280052b0 con 0x7f074806ce90 2023-10-10T18:45:21.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.386+0000 7f074d16d700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f07340671e0 0x7f0734069690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.390+0000 7f073dffb700 1 -- 172.21.15.175:0/3443665753 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f074004c5a0 con 0x7f074806ce90 2023-10-10T18:45:21.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.393+0000 7f074d16d700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f07340671e0 0x7f0734069690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f07480c7cc0 tx=0x7f0744006c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.630+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f61c8005560 con 0x7f61e81544a0 2023-10-10T18:45:21.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.630+0000 7f61cf7fe700 1 -- 172.21.15.175:0/3440997667 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f61d80548c0 con 0x7f61e81544a0 2023-10-10T18:45:21.634 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:21.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61d00674e0 msgr2=0x7f61d0069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61d00674e0 0x7f61d0069990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009e90 tx=0x7f61e0019000 comp rx=0 tx=0).stop 2023-10-10T18:45:21.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 msgr2=0x7f61e8155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8155460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61d800c5b0 tx=0x7f61d8007e10 comp rx=0 tx=0).stop 2023-10-10T18:45:21.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 shutdown_connections 2023-10-10T18:45:21.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61d00674e0 0x7f61d0069990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61e815e3f0 0x7f61e80c07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61e80c41f0 0x7f61e8143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 --2- 172.21.15.175:0/3440997667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61e81544a0 0x7f61e8155460 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 >> 172.21.15.175:0/3440997667 conn(0x7f61e8068270 msgr2=0x7f61e815a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 shutdown_connections 2023-10-10T18:45:21.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.634+0000 7f61edc83700 1 -- 172.21.15.175:0/3440997667 wait complete. 2023-10-10T18:45:21.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2023-10-10T18:45:21.654 DEBUG:teuthology.parallel:result is None 2023-10-10T18:45:21.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.669+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1b000ad8b0 con 0x7f1b0006f080 2023-10-10T18:45:21.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.669+0000 7f1af5ffb700 1 -- 172.21.15.175:0/4037378256 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1af0054c90 con 0x7f1b0006f080 2023-10-10T18:45:21.670 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:21.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.673+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ae4067080 msgr2=0x7f1ae4069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.673+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ae4067080 0x7f1ae4069530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1b000c9020 tx=0x7f1afc007eb0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.673+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b0006f080 msgr2=0x7f1b00173360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.673+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b0006f080 0x7f1b00173360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c880 tx=0x7f1af0008040 comp rx=0 tx=0).stop 2023-10-10T18:45:21.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.674+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 shutdown_connections 2023-10-10T18:45:21.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.674+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ae4067080 0x7f1ae4069530 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.674+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b000c2b10 0x7f1b000bf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.674+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b0006f080 0x7f1b00173360 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.674+0000 7f1b06f5f700 1 --2- 172.21.15.175:0/4037378256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b000bd9c0 0x7f1b000c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.674+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 >> 172.21.15.175:0/4037378256 conn(0x7f1b0006adc0 msgr2=0x7f1b000c4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.675+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 shutdown_connections 2023-10-10T18:45:21.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.675+0000 7f1b06f5f700 1 -- 172.21.15.175:0/4037378256 wait complete. 2023-10-10T18:45:21.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.683+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f51c415e650 con 0x7f51c406e260 2023-10-10T18:45:21.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.684+0000 7f51c0ff9700 1 -- 172.21.15.175:0/2647847814 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f51b80548b0 con 0x7f51c406e260 2023-10-10T18:45:21.685 INFO:teuthology.orchestra.run.smithi175.stdout:55834574855 2023-10-10T18:45:21.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51b0066fe0 msgr2=0x7f51b0069490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51b0066fe0 0x7f51b0069490 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f51b4013000 tx=0x7f51b4017000 comp rx=0 tx=0).stop 2023-10-10T18:45:21.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51c406e260 msgr2=0x7f51c41576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51c406e260 0x7f51c41576d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f51b800b5e0 tx=0x7f51b80090f0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 shutdown_connections 2023-10-10T18:45:21.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51b0066fe0 0x7f51b0069490 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51c40c61b0 0x7f51c4144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51c406de90 0x7f51c4154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 --2- 172.21.15.175:0/2647847814 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51c406e260 0x7f51c41576d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5a880693f0 con 0x7f5a8806a950 2023-10-10T18:45:21.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 >> 172.21.15.175:0/2647847814 conn(0x7f51c40c56a0 msgr2=0x7f51c40bffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 shutdown_connections 2023-10-10T18:45:21.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.687+0000 7f51caf5d700 1 -- 172.21.15.175:0/2647847814 wait complete. 2023-10-10T18:45:21.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2023-10-10T18:45:21.693 DEBUG:teuthology.parallel:result is None 2023-10-10T18:45:21.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.694+0000 7f5a6f7fe700 1 -- 172.21.15.175:0/1420197156 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5a78017090 con 0x7f5a8806a950 2023-10-10T18:45:21.695 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:21.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a74066f30 msgr2=0x7f5a740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a74066f30 0x7f5a740693e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a881738d0 tx=0x7f5a7c008040 comp rx=0 tx=0).stop 2023-10-10T18:45:21.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a8806a950 msgr2=0x7f5a8814e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a8806a950 0x7f5a8814e440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800ec80 tx=0x7f5a780091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa818067570 con 0x7fa81815ae60 2023-10-10T18:45:21.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 shutdown_connections 2023-10-10T18:45:21.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a74066f30 0x7f5a740693e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a880c8f90 0x7f5a88173290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a8806a950 0x7f5a8814e440 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 --2- 172.21.15.175:0/1420197156 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a8806e390 0x7f5a88176790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.698+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 >> 172.21.15.175:0/1420197156 conn(0x7f5a88068280 msgr2=0x7f5a880c1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.699+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 shutdown_connections 2023-10-10T18:45:21.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.699+0000 7f5a8de8c700 1 -- 172.21.15.175:0/1420197156 wait complete. 2023-10-10T18:45:21.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.699+0000 7fa7feffd700 1 -- 172.21.15.175:0/288384913 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa808021020 con 0x7fa81815ae60 2023-10-10T18:45:21.704 INFO:teuthology.orchestra.run.smithi175.stdout:55834574853 2023-10-10T18:45:21.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2023-10-10T18:45:21.705 DEBUG:teuthology.parallel:result is None 2023-10-10T18:45:21.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa804066f30 msgr2=0x7fa8040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa804066f30 0x7fa8040693e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0093f0 tx=0x7fa80c017000 comp rx=0 tx=0).stop 2023-10-10T18:45:21.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81815ae60 msgr2=0x7fa8181588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81815ae60 0x7fa8181588a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa8080010e0 tx=0x7fa808007dd0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 shutdown_connections 2023-10-10T18:45:21.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa804066f30 0x7fa8040693e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81815ae60 0x7fa8181588a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81806adb0 0x7fa8181561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 --2- 172.21.15.175:0/288384913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa8181546f0 0x7fa818158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 >> 172.21.15.175:0/288384913 conn(0x7fa818068270 msgr2=0x7fa8180c3910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 shutdown_connections 2023-10-10T18:45:21.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.703+0000 7fa81d052700 1 -- 172.21.15.175:0/288384913 wait complete. 2023-10-10T18:45:21.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-10-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.715+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7438067570 con 0x7f7438159cd0 2023-10-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.716+0000 7f7434ff9700 1 -- 172.21.15.175:0/1673383440 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f74280554f0 con 0x7f7438159cd0 2023-10-10T18:45:21.718 DEBUG:teuthology.parallel:result is None 2023-10-10T18:45:21.718 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:21.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7424067030 msgr2=0x7f74240694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7424067030 0x7f74240694e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f743000ad10 tx=0x7f7430009370 comp rx=0 tx=0).stop 2023-10-10T18:45:21.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7438159cd0 msgr2=0x7f74381579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7438159cd0 0x7f74381579f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f742800b4e0 tx=0x7f7428009060 comp rx=0 tx=0).stop 2023-10-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 shutdown_connections 2023-10-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7424067030 0x7f74240694e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7438159cd0 0x7f74381579f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74380b1020 0x7f743815ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 --2- 172.21.15.175:0/1673383440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74380c3460 0x7f743806ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 >> 172.21.15.175:0/1673383440 conn(0x7f74380cc820 msgr2=0x7f74381606d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 shutdown_connections 2023-10-10T18:45:21.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.719+0000 7f743eb7f700 1 -- 172.21.15.175:0/1673383440 wait complete. 2023-10-10T18:45:21.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2023-10-10T18:45:21.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.720+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fcfd8172a80 con 0x7fcfd806a8c0 2023-10-10T18:45:21.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.720+0000 7fcfc67fc700 1 -- 172.21.15.175:0/3083677165 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcfd8172a80 con 0x7fcfd806a8c0 2023-10-10T18:45:21.725 DEBUG:teuthology.parallel:result is None 2023-10-10T18:45:21.725 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:21.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb8067080 msgr2=0x7fcfb8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb8067080 0x7fcfb8069530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8165260 tx=0x7fcfcc0091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806a8c0 msgr2=0x7fcfd8176a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806a8c0 0x7fcfd8176a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8064cc0 tx=0x7fcfc8028000 comp rx=0 tx=0).stop 2023-10-10T18:45:21.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 shutdown_connections 2023-10-10T18:45:21.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb8067080 0x7fcfb8069530 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd80bb4b0 0x7fcfd8165ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd806a8c0 0x7fcfd8176a10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 --2- 172.21.15.175:0/3083677165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd806ce90 0x7fcfd8173420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 >> 172.21.15.175:0/3083677165 conn(0x7fcfd8068280 msgr2=0x7fcfd806b540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 shutdown_connections 2023-10-10T18:45:21.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.724+0000 7fcfdca4d700 1 -- 172.21.15.175:0/3083677165 wait complete. 2023-10-10T18:45:21.735 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2023-10-10T18:45:21.735 DEBUG:teuthology.parallel:result is None 2023-10-10T18:45:21.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2023-10-10T18:45:21.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.737+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0728005560 con 0x7f074806ce90 2023-10-10T18:45:21.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.737+0000 7f073dffb700 1 -- 172.21.15.175:0/3443665753 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07400519e0 con 0x7f074806ce90 2023-10-10T18:45:21.740 DEBUG:teuthology.parallel:result is None 2023-10-10T18:45:21.740 INFO:teuthology.orchestra.run.smithi175.stdout:55834574854 2023-10-10T18:45:21.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f07340671e0 msgr2=0x7f0734069690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f07340671e0 0x7f0734069690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f07480c7cc0 tx=0x7f0744006c00 comp rx=0 tx=0).stop 2023-10-10T18:45:21.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f074806ce90 msgr2=0x7f0748198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f074806ce90 0x7f0748198370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f074000bc30 tx=0x7f0740009820 comp rx=0 tx=0).stop 2023-10-10T18:45:21.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 shutdown_connections 2023-10-10T18:45:21.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f07340671e0 0x7f0734069690 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f07481639a0 0x7f07480c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f074806ce90 0x7f0748198370 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 --2- 172.21.15.175:0/3443665753 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07480bb4b0 0x7f07480c3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 >> 172.21.15.175:0/3443665753 conn(0x7f0748068280 msgr2=0x7f0748160190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 shutdown_connections 2023-10-10T18:45:21.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.740+0000 7f074fbd2700 1 -- 172.21.15.175:0/3443665753 wait complete. 2023-10-10T18:45:21.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2023-10-10T18:45:21.753 DEBUG:teuthology.parallel:result is None 2023-10-10T18:45:21.753 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-10T18:45:21.754 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T18:45:21.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.906+0000 7f73e3dd8700 1 Processor -- start 2023-10-10T18:45:21.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.906+0000 7f73e3dd8700 1 -- start start 2023-10-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e3dd8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f73dc0b1fc0 0x7f73dc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e3dd8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e3dd8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc1574b0 con 0x7f73dc0b2390 2023-10-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e3dd8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73dc150770 con 0x7f73dc0b1fc0 2023-10-10T18:45:21.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e3dd8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0afad0 con 0x7f73dc0aff60 2023-10-10T18:45:21.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e1b74700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f73dc0b1fc0 0x7f73dc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e2375700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e1b74700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f73dc0b1fc0 0x7f73dc155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:37436/0 (socket says 172.21.15.175:37436) 2023-10-10T18:45:21.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e2375700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc0b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33516/0 (socket says 172.21.15.175:33516) 2023-10-10T18:45:21.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.907+0000 7f73e1b74700 1 -- 172.21.15.175:0/3066610562 learned_addr learned my addr 172.21.15.175:0/3066610562 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e1b74700 1 -- 172.21.15.175:0/3066610562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 msgr2=0x7f73dc0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:21.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e1b74700 1 --2- 172.21.15.175:0/3066610562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e1b74700 1 -- 172.21.15.175:0/3066610562 >> v1:172.21.15.191:6789/0 conn(0x7f73dc0b2390 legacy=0x7f73dc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e1b74700 1 -- 172.21.15.175:0/3066610562 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc0af4c0 con 0x7f73dc0b1fc0 2023-10-10T18:45:21.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e2375700 1 --2- 172.21.15.175:0/3066610562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:21.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e1b74700 1 --2- 172.21.15.175:0/3066610562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 0x7f73dc155fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009f20 tx=0x7f73cc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=492c35d34653d9a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e0b72700 1 -- 172.21.15.175:0/3066610562 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc00e070 con 0x7f73dc0b1fc0 2023-10-10T18:45:21.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e0b72700 1 -- 172.21.15.175:0/3066610562 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc00ba60 con 0x7f73dc0b1fc0 2023-10-10T18:45:21.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.908+0000 7f73e0b72700 1 -- 172.21.15.175:0/3066610562 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc0092e0 con 0x7f73dc0b1fc0 2023-10-10T18:45:21.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.909+0000 7f73e3dd8700 1 -- 172.21.15.175:0/3066610562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 msgr2=0x7f73dc155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.909+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/3066610562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 0x7f73dc155fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009f20 tx=0x7f73cc009840 comp rx=0 tx=0).stop 2023-10-10T18:45:21.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.909+0000 7f73e3dd8700 1 -- 172.21.15.175:0/3066610562 shutdown_connections 2023-10-10T18:45:21.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.909+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/3066610562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.909+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/3066610562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 0x7f73dc155fb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.909+0000 7f73e3dd8700 1 -- 172.21.15.175:0/3066610562 >> 172.21.15.175:0/3066610562 conn(0x7f73dc063230 msgr2=0x7f73dc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:21.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.909+0000 7f73e3dd8700 1 -- 172.21.15.175:0/3066610562 shutdown_connections 2023-10-10T18:45:21.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.909+0000 7f73e3dd8700 1 -- 172.21.15.175:0/3066610562 wait complete. 2023-10-10T18:45:21.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.910+0000 7f73e3dd8700 1 Processor -- start 2023-10-10T18:45:21.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.910+0000 7f73e3dd8700 1 -- start start 2023-10-10T18:45:21.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e3dd8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e3dd8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 0x7f73dc165db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e3dd8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73dc0b2390 0x7f73dc14d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e3dd8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0cb5e0 con 0x7f73dc0b2390 2023-10-10T18:45:21.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e3dd8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc169d00 con 0x7f73dc0b1fc0 2023-10-10T18:45:21.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e3dd8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc16a190 con 0x7f73dc0aff60 2023-10-10T18:45:21.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73dc0b2390 0x7f73dc14d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73dc0b2390 0x7f73dc14d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52642/0 (socket says 172.21.15.175:52642) 2023-10-10T18:45:21.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 -- 172.21.15.175:0/934248538 learned_addr learned my addr 172.21.15.175:0/934248538 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:21.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e1b74700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e1373700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 0x7f73dc165db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 -- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 msgr2=0x7f73dc15b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc15b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 -- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 msgr2=0x7f73dc165db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:21.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 0x7f73dc165db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:21.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 -- 172.21.15.175:0/934248538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d4009420 con 0x7f73dc0b2390 2023-10-10T18:45:21.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e2375700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73dc0b2390 0x7f73dc14d970 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f73d400ecb0 tx=0x7f73d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73d9ffb700 1 -- 172.21.15.175:0/934248538 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d400c850 con 0x7f73dc0b2390 2023-10-10T18:45:21.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73d9ffb700 1 -- 172.21.15.175:0/934248538 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d400c9b0 con 0x7f73dc0b2390 2023-10-10T18:45:21.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc009ae0 con 0x7f73dc0b2390 2023-10-10T18:45:21.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73d9ffb700 1 -- 172.21.15.175:0/934248538 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d400cd70 con 0x7f73dc0b2390 2023-10-10T18:45:21.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.911+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73dc1693d0 con 0x7f73dc0b2390 2023-10-10T18:45:21.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.912+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73dc067570 con 0x7f73dc0b2390 2023-10-10T18:45:21.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.913+0000 7f73d9ffb700 1 -- 172.21.15.175:0/934248538 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f73d4013680 con 0x7f73dc0b2390 2023-10-10T18:45:21.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.913+0000 7f73d9ffb700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73c8066f30 0x7f73c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:21.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.913+0000 7f73d9ffb700 1 -- 172.21.15.175:0/934248538 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f73d4084910 con 0x7f73dc0b2390 2023-10-10T18:45:21.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.913+0000 7f73e1b74700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73c8066f30 0x7f73c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:21.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.913+0000 7f73e1b74700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73c8066f30 0x7f73c80693e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009880 tx=0x7f73cc018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:21.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:21.917+0000 7f73d9ffb700 1 -- 172.21.15.175:0/934248538 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73d4051b80 con 0x7f73dc0b2390 2023-10-10T18:45:22.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.096+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73dc169e40 con 0x7f73c8066f30 2023-10-10T18:45:22.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.104+0000 7f73d9ffb700 1 -- 172.21.15.175:0/934248538 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264616 (secure 0 0 0) 0x7f73dc169e40 con 0x7f73c8066f30 2023-10-10T18:45:22.108 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:22.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73c8066f30 msgr2=0x7f73c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73c8066f30 0x7f73c80693e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009880 tx=0x7f73cc018000 comp rx=0 tx=0).stop 2023-10-10T18:45:22.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73dc0b2390 msgr2=0x7f73dc14d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73dc0b2390 0x7f73dc14d970 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f73d400ecb0 tx=0x7f73d40091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 shutdown_connections 2023-10-10T18:45:22.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73c8066f30 0x7f73c80693e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73dc0b2390 0x7f73dc14d970 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73dc0aff60 0x7f73dc15b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 --2- 172.21.15.175:0/934248538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73dc0b1fc0 0x7f73dc165db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 >> 172.21.15.175:0/934248538 conn(0x7f73dc063230 msgr2=0x7f73dc14fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:22.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 shutdown_connections 2023-10-10T18:45:22.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.107+0000 7f73e3dd8700 1 -- 172.21.15.175:0/934248538 wait complete. 2023-10-10T18:45:22.112 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T18:45:22.120 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-10T18:45:21.131982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2380192,"kb_used_data":1888,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752594528,"statfs":{"total":773094113280,"available":770656796672,"internally_reserved":0,"allocated":1933312,"data_stored":1787852,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.728648"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429926+0000","last_change":"2023-10-10T18:45:11.419184+0000","last_active":"2023-10-10T18:45:11.429926+0000","last_peered":"2023-10-10T18:45:11.429926+0000","last_clean":"2023-10-10T18:45:11.429926+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T18:45:11.429926+0000","last_undegraded":"2023-10-10T18:45:11.429926+0000","last_fullsized":"2023-10-10T18:45:11.429926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420684+0000","last_change":"2023-10-10T18:45:11.414586+0000","last_active":"2023-10-10T18:45:11.420684+0000","last_peered":"2023-10-10T18:45:11.420684+0000","last_clean":"2023-10-10T18:45:11.420684+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T18:45:11.420684+0000","last_undegraded":"2023-10-10T18:45:11.420684+0000","last_fullsized":"2023-10-10T18:45:11.420684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424203+0000","last_change":"2023-10-10T18:45:11.419119+0000","last_active":"2023-10-10T18:45:11.424203+0000","last_peered":"2023-10-10T18:45:11.424203+0000","last_clean":"2023-10-10T18:45:11.424203+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T18:45:11.424203+0000","last_undegraded":"2023-10-10T18:45:11.424203+0000","last_fullsized":"2023-10-10T18:45:11.424203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425796+0000","last_change":"2023-10-10T18:45:11.421351+0000","last_active":"2023-10-10T18:45:11.425796+0000","last_peered":"2023-10-10T18:45:11.425796+0000","last_clean":"2023-10-10T18:45:11.425796+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T18:45:11.425796+0000","last_undegraded":"2023-10-10T18:45:11.425796+0000","last_fullsized":"2023-10-10T18:45:11.425796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425614+0000","last_change":"2023-10-10T18:45:11.421122+0000","last_active":"2023-10-10T18:45:11.425614+0000","last_peered":"2023-10-10T18:45:11.425614+0000","last_clean":"2023-10-10T18:45:11.425614+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T18:45:11.425614+0000","last_undegraded":"2023-10-10T18:45:11.425614+0000","last_fullsized":"2023-10-10T18:45:11.425614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424542+0000","last_change":"2023-10-10T18:45:11.421729+0000","last_active":"2023-10-10T18:45:11.424542+0000","last_peered":"2023-10-10T18:45:11.424542+0000","last_clean":"2023-10-10T18:45:11.424542+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T18:45:11.424542+0000","last_undegraded":"2023-10-10T18:45:11.424542+0000","last_fullsized":"2023-10-10T18:45:11.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426178+0000","last_change":"2023-10-10T18:45:11.419306+0000","last_active":"2023-10-10T18:45:11.426178+0000","last_peered":"2023-10-10T18:45:11.426178+0000","last_clean":"2023-10-10T18:45:11.426178+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T18:45:11.426178+0000","last_undegraded":"2023-10-10T18:45:11.426178+0000","last_fullsized":"2023-10-10T18:45:11.426178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.422940+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429946+0000","last_change":"2023-10-10T18:45:11.422612+0000","last_active":"2023-10-10T18:45:11.429946+0000","last_peered":"2023-10-10T18:45:11.429946+0000","last_clean":"2023-10-10T18:45:11.429946+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T18:45:11.429946+0000","last_undegraded":"2023-10-10T18:45:11.429946+0000","last_fullsized":"2023-10-10T18:45:11.429946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430763+0000","last_change":"2023-10-10T18:45:11.425754+0000","last_active":"2023-10-10T18:45:11.430763+0000","last_peered":"2023-10-10T18:45:11.430763+0000","last_clean":"2023-10-10T18:45:11.430763+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T18:45:11.430763+0000","last_undegraded":"2023-10-10T18:45:11.430763+0000","last_fullsized":"2023-10-10T18:45:11.430763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430664+0000","last_change":"2023-10-10T18:45:11.425891+0000","last_active":"2023-10-10T18:45:11.430664+0000","last_peered":"2023-10-10T18:45:11.430664+0000","last_clean":"2023-10-10T18:45:11.430664+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T18:45:11.430664+0000","last_undegraded":"2023-10-10T18:45:11.430664+0000","last_fullsized":"2023-10-10T18:45:11.430664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431015+0000","last_change":"2023-10-10T18:45:11.424834+0000","last_active":"2023-10-10T18:45:11.431015+0000","last_peered":"2023-10-10T18:45:11.431015+0000","last_clean":"2023-10-10T18:45:11.431015+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T18:45:11.431015+0000","last_undegraded":"2023-10-10T18:45:11.431015+0000","last_fullsized":"2023-10-10T18:45:11.431015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421085+0000","last_change":"2023-10-10T18:45:11.417796+0000","last_active":"2023-10-10T18:45:11.421085+0000","last_peered":"2023-10-10T18:45:11.421085+0000","last_clean":"2023-10-10T18:45:11.421085+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T18:45:11.421085+0000","last_undegraded":"2023-10-10T18:45:11.421085+0000","last_fullsized":"2023-10-10T18:45:11.421085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419270+0000","last_change":"2023-10-10T18:45:11.414690+0000","last_active":"2023-10-10T18:45:11.419270+0000","last_peered":"2023-10-10T18:45:11.419270+0000","last_clean":"2023-10-10T18:45:11.419270+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T18:45:11.419270+0000","last_undegraded":"2023-10-10T18:45:11.419270+0000","last_fullsized":"2023-10-10T18:45:11.419270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434526+0000","last_change":"2023-10-10T18:45:11.429433+0000","last_active":"2023-10-10T18:45:11.434526+0000","last_peered":"2023-10-10T18:45:11.434526+0000","last_clean":"2023-10-10T18:45:11.434526+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T18:45:11.434526+0000","last_undegraded":"2023-10-10T18:45:11.434526+0000","last_fullsized":"2023-10-10T18:45:11.434526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427193+0000","last_change":"2023-10-10T18:45:11.422935+0000","last_active":"2023-10-10T18:45:11.427193+0000","last_peered":"2023-10-10T18:45:11.427193+0000","last_clean":"2023-10-10T18:45:11.427193+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T18:45:11.427193+0000","last_undegraded":"2023-10-10T18:45:11.427193+0000","last_fullsized":"2023-10-10T18:45:11.427193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434131+0000","last_change":"2023-10-10T18:45:11.431202+0000","last_active":"2023-10-10T18:45:11.434131+0000","last_peered":"2023-10-10T18:45:11.434131+0000","last_clean":"2023-10-10T18:45:11.434131+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T18:45:11.434131+0000","last_undegraded":"2023-10-10T18:45:11.434131+0000","last_fullsized":"2023-10-10T18:45:11.434131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427139+0000","last_change":"2023-10-10T18:45:11.424885+0000","last_active":"2023-10-10T18:45:11.427139+0000","last_peered":"2023-10-10T18:45:11.427139+0000","last_clean":"2023-10-10T18:45:11.427139+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T18:45:11.427139+0000","last_undegraded":"2023-10-10T18:45:11.427139+0000","last_fullsized":"2023-10-10T18:45:11.427139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426206+0000","last_change":"2023-10-10T18:45:11.421039+0000","last_active":"2023-10-10T18:45:11.426206+0000","last_peered":"2023-10-10T18:45:11.426206+0000","last_clean":"2023-10-10T18:45:11.426206+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T18:45:11.426206+0000","last_undegraded":"2023-10-10T18:45:11.426206+0000","last_fullsized":"2023-10-10T18:45:11.426206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421148+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T18:45:16.421148+0000","last_peered":"2023-10-10T18:45:16.421148+0000","last_clean":"2023-10-10T18:45:16.421148+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T18:45:16.421148+0000","last_undegraded":"2023-10-10T18:45:16.421148+0000","last_fullsized":"2023-10-10T18:45:16.421148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421043+0000","last_change":"2023-10-10T18:45:11.417504+0000","last_active":"2023-10-10T18:45:11.421043+0000","last_peered":"2023-10-10T18:45:11.421043+0000","last_clean":"2023-10-10T18:45:11.421043+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T18:45:11.421043+0000","last_undegraded":"2023-10-10T18:45:11.421043+0000","last_fullsized":"2023-10-10T18:45:11.421043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.010024+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T18:45:13.010024+0000","last_peered":"2023-10-10T18:45:13.010024+0000","last_clean":"2023-10-10T18:45:13.010024+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T18:45:13.010024+0000","last_undegraded":"2023-10-10T18:45:13.010024+0000","last_fullsized":"2023-10-10T18:45:13.010024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424178+0000","last_change":"2023-10-10T18:45:11.421896+0000","last_active":"2023-10-10T18:45:11.424178+0000","last_peered":"2023-10-10T18:45:11.424178+0000","last_clean":"2023-10-10T18:45:11.424178+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T18:45:11.424178+0000","last_undegraded":"2023-10-10T18:45:11.424178+0000","last_fullsized":"2023-10-10T18:45:11.424178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.418723+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T18:45:16.418723+0000","last_peered":"2023-10-10T18:45:16.418723+0000","last_clean":"2023-10-10T18:45:16.418723+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T18:45:16.418723+0000","last_undegraded":"2023-10-10T18:45:16.418723+0000","last_fullsized":"2023-10-10T18:45:16.418723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432466+0000","last_change":"2023-10-10T18:45:11.429680+0000","last_active":"2023-10-10T18:45:11.432466+0000","last_peered":"2023-10-10T18:45:11.432466+0000","last_clean":"2023-10-10T18:45:11.432466+0000","last_became_active":"2023-10-10T18:45:11.429397+0000","last_became_peered":"2023-10-10T18:45:11.429397+0000","last_unstale":"2023-10-10T18:45:11.432466+0000","last_undegraded":"2023-10-10T18:45:11.432466+0000","last_fullsized":"2023-10-10T18:45:11.432466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:19:52.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.418579+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T18:45:16.418579+0000","last_peered":"2023-10-10T18:45:16.418579+0000","last_clean":"2023-10-10T18:45:16.418579+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T18:45:16.418579+0000","last_undegraded":"2023-10-10T18:45:16.418579+0000","last_fullsized":"2023-10-10T18:45:16.418579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432439+0000","last_change":"2023-10-10T18:45:11.429408+0000","last_active":"2023-10-10T18:45:11.432439+0000","last_peered":"2023-10-10T18:45:11.432439+0000","last_clean":"2023-10-10T18:45:11.432439+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T18:45:11.432439+0000","last_undegraded":"2023-10-10T18:45:11.432439+0000","last_fullsized":"2023-10-10T18:45:11.432439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.010081+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T18:45:13.010081+0000","last_peered":"2023-10-10T18:45:13.010081+0000","last_clean":"2023-10-10T18:45:13.010081+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T18:45:13.010081+0000","last_undegraded":"2023-10-10T18:45:13.010081+0000","last_fullsized":"2023-10-10T18:45:13.010081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430579+0000","last_change":"2023-10-10T18:45:11.426051+0000","last_active":"2023-10-10T18:45:11.430579+0000","last_peered":"2023-10-10T18:45:11.430579+0000","last_clean":"2023-10-10T18:45:11.430579+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T18:45:11.430579+0000","last_undegraded":"2023-10-10T18:45:11.430579+0000","last_fullsized":"2023-10-10T18:45:11.430579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.009453+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T18:45:13.009453+0000","last_peered":"2023-10-10T18:45:13.009453+0000","last_clean":"2023-10-10T18:45:13.009453+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T18:45:13.009453+0000","last_undegraded":"2023-10-10T18:45:13.009453+0000","last_fullsized":"2023-10-10T18:45:13.009453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427223+0000","last_change":"2023-10-10T18:45:11.425183+0000","last_active":"2023-10-10T18:45:11.427223+0000","last_peered":"2023-10-10T18:45:11.427223+0000","last_clean":"2023-10-10T18:45:11.427223+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T18:45:11.427223+0000","last_undegraded":"2023-10-10T18:45:11.427223+0000","last_fullsized":"2023-10-10T18:45:11.427223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.402044+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T18:45:11.402044+0000","last_peered":"2023-10-10T18:45:11.402044+0000","last_clean":"2023-10-10T18:45:11.402044+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T18:45:11.402044+0000","last_undegraded":"2023-10-10T18:45:11.402044+0000","last_fullsized":"2023-10-10T18:45:11.402044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415249+0000","last_change":"2023-10-10T18:45:11.412641+0000","last_active":"2023-10-10T18:45:11.415249+0000","last_peered":"2023-10-10T18:45:11.415249+0000","last_clean":"2023-10-10T18:45:11.415249+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T18:45:11.415249+0000","last_undegraded":"2023-10-10T18:45:11.415249+0000","last_fullsized":"2023-10-10T18:45:11.415249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.413032+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T18:45:11.413032+0000","last_peered":"2023-10-10T18:45:11.413032+0000","last_clean":"2023-10-10T18:45:11.413032+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T18:45:11.413032+0000","last_undegraded":"2023-10-10T18:45:11.413032+0000","last_fullsized":"2023-10-10T18:45:11.413032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420638+0000","last_change":"2023-10-10T18:45:11.413819+0000","last_active":"2023-10-10T18:45:11.420638+0000","last_peered":"2023-10-10T18:45:11.420638+0000","last_clean":"2023-10-10T18:45:11.420638+0000","last_became_active":"2023-10-10T18:45:11.413529+0000","last_became_peered":"2023-10-10T18:45:11.413529+0000","last_unstale":"2023-10-10T18:45:11.420638+0000","last_undegraded":"2023-10-10T18:45:11.420638+0000","last_fullsized":"2023-10-10T18:45:11.420638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.411211+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T18:45:11.411211+0000","last_peered":"2023-10-10T18:45:11.411211+0000","last_clean":"2023-10-10T18:45:11.411211+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T18:45:11.411211+0000","last_undegraded":"2023-10-10T18:45:11.411211+0000","last_fullsized":"2023-10-10T18:45:11.411211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430421+0000","last_change":"2023-10-10T18:45:11.427116+0000","last_active":"2023-10-10T18:45:11.430421+0000","last_peered":"2023-10-10T18:45:11.430421+0000","last_clean":"2023-10-10T18:45:11.430421+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T18:45:11.430421+0000","last_undegraded":"2023-10-10T18:45:11.430421+0000","last_fullsized":"2023-10-10T18:45:11.430421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.416463+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T18:45:11.416463+0000","last_peered":"2023-10-10T18:45:11.416463+0000","last_clean":"2023-10-10T18:45:11.416463+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T18:45:11.416463+0000","last_undegraded":"2023-10-10T18:45:11.416463+0000","last_fullsized":"2023-10-10T18:45:11.416463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424158+0000","last_change":"2023-10-10T18:45:11.421794+0000","last_active":"2023-10-10T18:45:11.424158+0000","last_peered":"2023-10-10T18:45:11.424158+0000","last_clean":"2023-10-10T18:45:11.424158+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T18:45:11.424158+0000","last_undegraded":"2023-10-10T18:45:11.424158+0000","last_fullsized":"2023-10-10T18:45:11.424158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.418436+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T18:45:16.418436+0000","last_peered":"2023-10-10T18:45:16.418436+0000","last_clean":"2023-10-10T18:45:16.418436+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T18:45:16.418436+0000","last_undegraded":"2023-10-10T18:45:16.418436+0000","last_fullsized":"2023-10-10T18:45:16.418436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434442+0000","last_change":"2023-10-10T18:45:11.431857+0000","last_active":"2023-10-10T18:45:11.434442+0000","last_peered":"2023-10-10T18:45:11.434442+0000","last_clean":"2023-10-10T18:45:11.434442+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T18:45:11.434442+0000","last_undegraded":"2023-10-10T18:45:11.434442+0000","last_fullsized":"2023-10-10T18:45:11.434442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421659+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T18:45:16.421659+0000","last_peered":"2023-10-10T18:45:16.421659+0000","last_clean":"2023-10-10T18:45:16.421659+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T18:45:16.421659+0000","last_undegraded":"2023-10-10T18:45:16.421659+0000","last_fullsized":"2023-10-10T18:45:16.421659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426196+0000","last_change":"2023-10-10T18:45:11.419774+0000","last_active":"2023-10-10T18:45:11.426196+0000","last_peered":"2023-10-10T18:45:11.426196+0000","last_clean":"2023-10-10T18:45:11.426196+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T18:45:11.426196+0000","last_undegraded":"2023-10-10T18:45:11.426196+0000","last_fullsized":"2023-10-10T18:45:11.426196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.419416+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T18:45:16.419416+0000","last_peered":"2023-10-10T18:45:16.419416+0000","last_clean":"2023-10-10T18:45:16.419416+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T18:45:16.419416+0000","last_undegraded":"2023-10-10T18:45:16.419416+0000","last_fullsized":"2023-10-10T18:45:16.419416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424245+0000","last_change":"2023-10-10T18:45:11.420517+0000","last_active":"2023-10-10T18:45:11.424245+0000","last_peered":"2023-10-10T18:45:11.424245+0000","last_clean":"2023-10-10T18:45:11.424245+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T18:45:11.424245+0000","last_undegraded":"2023-10-10T18:45:11.424245+0000","last_fullsized":"2023-10-10T18:45:11.424245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412727+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T18:45:11.412727+0000","last_peered":"2023-10-10T18:45:11.412727+0000","last_clean":"2023-10-10T18:45:11.412727+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T18:45:11.412727+0000","last_undegraded":"2023-10-10T18:45:11.412727+0000","last_fullsized":"2023-10-10T18:45:11.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431834+0000","last_change":"2023-10-10T18:45:11.427109+0000","last_active":"2023-10-10T18:45:11.431834+0000","last_peered":"2023-10-10T18:45:11.431834+0000","last_clean":"2023-10-10T18:45:11.431834+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T18:45:11.431834+0000","last_undegraded":"2023-10-10T18:45:11.431834+0000","last_fullsized":"2023-10-10T18:45:11.431834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410202+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T18:45:11.410202+0000","last_peered":"2023-10-10T18:45:11.410202+0000","last_clean":"2023-10-10T18:45:11.410202+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T18:45:11.410202+0000","last_undegraded":"2023-10-10T18:45:11.410202+0000","last_fullsized":"2023-10-10T18:45:11.410202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430660+0000","last_change":"2023-10-10T18:45:11.416459+0000","last_active":"2023-10-10T18:45:11.430660+0000","last_peered":"2023-10-10T18:45:11.430660+0000","last_clean":"2023-10-10T18:45:11.430660+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T18:45:11.430660+0000","last_undegraded":"2023-10-10T18:45:11.430660+0000","last_fullsized":"2023-10-10T18:45:11.430660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421838+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T18:45:16.421838+0000","last_peered":"2023-10-10T18:45:16.421838+0000","last_clean":"2023-10-10T18:45:16.421838+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T18:45:16.421838+0000","last_undegraded":"2023-10-10T18:45:16.421838+0000","last_fullsized":"2023-10-10T18:45:16.421838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432328+0000","last_change":"2023-10-10T18:45:11.429727+0000","last_active":"2023-10-10T18:45:11.432328+0000","last_peered":"2023-10-10T18:45:11.432328+0000","last_clean":"2023-10-10T18:45:11.432328+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T18:45:11.432328+0000","last_undegraded":"2023-10-10T18:45:11.432328+0000","last_fullsized":"2023-10-10T18:45:11.432328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:12.418575+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T18:45:12.418575+0000","last_peered":"2023-10-10T18:45:12.418575+0000","last_clean":"2023-10-10T18:45:12.418575+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T18:45:12.418575+0000","last_undegraded":"2023-10-10T18:45:12.418575+0000","last_fullsized":"2023-10-10T18:45:12.418575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431858+0000","last_change":"2023-10-10T18:45:11.427926+0000","last_active":"2023-10-10T18:45:11.431858+0000","last_peered":"2023-10-10T18:45:11.431858+0000","last_clean":"2023-10-10T18:45:11.431858+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T18:45:11.431858+0000","last_undegraded":"2023-10-10T18:45:11.431858+0000","last_fullsized":"2023-10-10T18:45:11.431858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407823+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T18:45:11.407823+0000","last_peered":"2023-10-10T18:45:11.407823+0000","last_clean":"2023-10-10T18:45:11.407823+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T18:45:11.407823+0000","last_undegraded":"2023-10-10T18:45:11.407823+0000","last_fullsized":"2023-10-10T18:45:11.407823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427955+0000","last_change":"2023-10-10T18:45:11.423279+0000","last_active":"2023-10-10T18:45:11.427955+0000","last_peered":"2023-10-10T18:45:11.427955+0000","last_clean":"2023-10-10T18:45:11.427955+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T18:45:11.427955+0000","last_undegraded":"2023-10-10T18:45:11.427955+0000","last_fullsized":"2023-10-10T18:45:11.427955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412668+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T18:45:11.412668+0000","last_peered":"2023-10-10T18:45:11.412668+0000","last_clean":"2023-10-10T18:45:11.412668+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T18:45:11.412668+0000","last_undegraded":"2023-10-10T18:45:11.412668+0000","last_fullsized":"2023-10-10T18:45:11.412668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430728+0000","last_change":"2023-10-10T18:45:11.425339+0000","last_active":"2023-10-10T18:45:11.430728+0000","last_peered":"2023-10-10T18:45:11.430728+0000","last_clean":"2023-10-10T18:45:11.430728+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T18:45:11.430728+0000","last_undegraded":"2023-10-10T18:45:11.430728+0000","last_fullsized":"2023-10-10T18:45:11.430728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421278+0000","last_change":"2023-10-10T18:45:11.418937+0000","last_active":"2023-10-10T18:45:11.421278+0000","last_peered":"2023-10-10T18:45:11.421278+0000","last_clean":"2023-10-10T18:45:11.421278+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T18:45:11.421278+0000","last_undegraded":"2023-10-10T18:45:11.421278+0000","last_fullsized":"2023-10-10T18:45:11.421278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410721+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T18:45:11.410721+0000","last_peered":"2023-10-10T18:45:11.410721+0000","last_clean":"2023-10-10T18:45:11.410721+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T18:45:11.410721+0000","last_undegraded":"2023-10-10T18:45:11.410721+0000","last_fullsized":"2023-10-10T18:45:11.410721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424136+0000","last_change":"2023-10-10T18:45:11.422452+0000","last_active":"2023-10-10T18:45:11.424136+0000","last_peered":"2023-10-10T18:45:11.424136+0000","last_clean":"2023-10-10T18:45:11.424136+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T18:45:11.424136+0000","last_undegraded":"2023-10-10T18:45:11.424136+0000","last_fullsized":"2023-10-10T18:45:11.424136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410662+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T18:45:11.410662+0000","last_peered":"2023-10-10T18:45:11.410662+0000","last_clean":"2023-10-10T18:45:11.410662+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T18:45:11.410662+0000","last_undegraded":"2023-10-10T18:45:11.410662+0000","last_fullsized":"2023-10-10T18:45:11.410662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405467+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T18:45:11.405467+0000","last_peered":"2023-10-10T18:45:11.405467+0000","last_clean":"2023-10-10T18:45:11.405467+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T18:45:11.405467+0000","last_undegraded":"2023-10-10T18:45:11.405467+0000","last_fullsized":"2023-10-10T18:45:11.405467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434404+0000","last_change":"2023-10-10T18:45:11.430233+0000","last_active":"2023-10-10T18:45:11.434404+0000","last_peered":"2023-10-10T18:45:11.434404+0000","last_clean":"2023-10-10T18:45:11.434404+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T18:45:11.434404+0000","last_undegraded":"2023-10-10T18:45:11.434404+0000","last_fullsized":"2023-10-10T18:45:11.434404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405812+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T18:45:11.405812+0000","last_peered":"2023-10-10T18:45:11.405812+0000","last_clean":"2023-10-10T18:45:11.405812+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T18:45:11.405812+0000","last_undegraded":"2023-10-10T18:45:11.405812+0000","last_fullsized":"2023-10-10T18:45:11.405812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.408007+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T18:45:11.408007+0000","last_peered":"2023-10-10T18:45:11.408007+0000","last_clean":"2023-10-10T18:45:11.408007+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T18:45:11.408007+0000","last_undegraded":"2023-10-10T18:45:11.408007+0000","last_fullsized":"2023-10-10T18:45:11.408007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421286+0000","last_change":"2023-10-10T18:45:11.418973+0000","last_active":"2023-10-10T18:45:11.421286+0000","last_peered":"2023-10-10T18:45:11.421286+0000","last_clean":"2023-10-10T18:45:11.421286+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T18:45:11.421286+0000","last_undegraded":"2023-10-10T18:45:11.421286+0000","last_fullsized":"2023-10-10T18:45:11.421286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407798+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T18:45:11.407798+0000","last_peered":"2023-10-10T18:45:11.407798+0000","last_clean":"2023-10-10T18:45:11.407798+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T18:45:11.407798+0000","last_undegraded":"2023-10-10T18:45:11.407798+0000","last_fullsized":"2023-10-10T18:45:11.407798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423727+0000","last_change":"2023-10-10T18:45:11.419846+0000","last_active":"2023-10-10T18:45:11.423727+0000","last_peered":"2023-10-10T18:45:11.423727+0000","last_clean":"2023-10-10T18:45:11.423727+0000","last_became_active":"2023-10-10T18:45:11.419579+0000","last_became_peered":"2023-10-10T18:45:11.419579+0000","last_unstale":"2023-10-10T18:45:11.423727+0000","last_undegraded":"2023-10-10T18:45:11.423727+0000","last_fullsized":"2023-10-10T18:45:11.423727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:58:49.831558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.418931+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T18:45:16.418931+0000","last_peered":"2023-10-10T18:45:16.418931+0000","last_clean":"2023-10-10T18:45:16.418931+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T18:45:16.418931+0000","last_undegraded":"2023-10-10T18:45:16.418931+0000","last_fullsized":"2023-10-10T18:45:16.418931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423663+0000","last_change":"2023-10-10T18:45:11.416217+0000","last_active":"2023-10-10T18:45:11.423663+0000","last_peered":"2023-10-10T18:45:11.423663+0000","last_clean":"2023-10-10T18:45:11.423663+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T18:45:11.423663+0000","last_undegraded":"2023-10-10T18:45:11.423663+0000","last_fullsized":"2023-10-10T18:45:11.423663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405671+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T18:45:11.405671+0000","last_peered":"2023-10-10T18:45:11.405671+0000","last_clean":"2023-10-10T18:45:11.405671+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T18:45:11.405671+0000","last_undegraded":"2023-10-10T18:45:11.405671+0000","last_fullsized":"2023-10-10T18:45:11.405671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431602+0000","last_change":"2023-10-10T18:45:11.427826+0000","last_active":"2023-10-10T18:45:11.431602+0000","last_peered":"2023-10-10T18:45:11.431602+0000","last_clean":"2023-10-10T18:45:11.431602+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T18:45:11.431602+0000","last_undegraded":"2023-10-10T18:45:11.431602+0000","last_fullsized":"2023-10-10T18:45:11.431602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407958+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T18:45:11.407958+0000","last_peered":"2023-10-10T18:45:11.407958+0000","last_clean":"2023-10-10T18:45:11.407958+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T18:45:11.407958+0000","last_undegraded":"2023-10-10T18:45:11.407958+0000","last_fullsized":"2023-10-10T18:45:11.407958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415388+0000","last_change":"2023-10-10T18:45:11.413975+0000","last_active":"2023-10-10T18:45:11.415388+0000","last_peered":"2023-10-10T18:45:11.415388+0000","last_clean":"2023-10-10T18:45:11.415388+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T18:45:11.415388+0000","last_undegraded":"2023-10-10T18:45:11.415388+0000","last_fullsized":"2023-10-10T18:45:11.415388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407755+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T18:45:11.407755+0000","last_peered":"2023-10-10T18:45:11.407755+0000","last_clean":"2023-10-10T18:45:11.407755+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T18:45:11.407755+0000","last_undegraded":"2023-10-10T18:45:11.407755+0000","last_fullsized":"2023-10-10T18:45:11.407755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415058+0000","last_change":"2023-10-10T18:45:11.410768+0000","last_active":"2023-10-10T18:45:11.415058+0000","last_peered":"2023-10-10T18:45:11.415058+0000","last_clean":"2023-10-10T18:45:11.415058+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T18:45:11.415058+0000","last_undegraded":"2023-10-10T18:45:11.415058+0000","last_fullsized":"2023-10-10T18:45:11.415058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.419134+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T18:45:16.419134+0000","last_peered":"2023-10-10T18:45:16.419134+0000","last_clean":"2023-10-10T18:45:16.419134+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T18:45:16.419134+0000","last_undegraded":"2023-10-10T18:45:16.419134+0000","last_fullsized":"2023-10-10T18:45:16.419134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430290+0000","last_change":"2023-10-10T18:45:11.425716+0000","last_active":"2023-10-10T18:45:11.430290+0000","last_peered":"2023-10-10T18:45:11.430290+0000","last_clean":"2023-10-10T18:45:11.430290+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T18:45:11.430290+0000","last_undegraded":"2023-10-10T18:45:11.430290+0000","last_fullsized":"2023-10-10T18:45:11.430290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412053+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T18:45:11.412053+0000","last_peered":"2023-10-10T18:45:11.412053+0000","last_clean":"2023-10-10T18:45:11.412053+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T18:45:11.412053+0000","last_undegraded":"2023-10-10T18:45:11.412053+0000","last_fullsized":"2023-10-10T18:45:11.412053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429787+0000","last_change":"2023-10-10T18:45:11.423747+0000","last_active":"2023-10-10T18:45:11.429787+0000","last_peered":"2023-10-10T18:45:11.429787+0000","last_clean":"2023-10-10T18:45:11.429787+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T18:45:11.429787+0000","last_undegraded":"2023-10-10T18:45:11.429787+0000","last_fullsized":"2023-10-10T18:45:11.429787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.009694+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T18:45:13.009694+0000","last_peered":"2023-10-10T18:45:13.009694+0000","last_clean":"2023-10-10T18:45:13.009694+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T18:45:13.009694+0000","last_undegraded":"2023-10-10T18:45:13.009694+0000","last_fullsized":"2023-10-10T18:45:13.009694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425816+0000","last_change":"2023-10-10T18:45:11.421872+0000","last_active":"2023-10-10T18:45:11.425816+0000","last_peered":"2023-10-10T18:45:11.425816+0000","last_clean":"2023-10-10T18:45:11.425816+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T18:45:11.425816+0000","last_undegraded":"2023-10-10T18:45:11.425816+0000","last_fullsized":"2023-10-10T18:45:11.425816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421169+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T18:45:16.421169+0000","last_peered":"2023-10-10T18:45:16.421169+0000","last_clean":"2023-10-10T18:45:16.421169+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T18:45:16.421169+0000","last_undegraded":"2023-10-10T18:45:16.421169+0000","last_fullsized":"2023-10-10T18:45:16.421169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429963+0000","last_change":"2023-10-10T18:45:11.425474+0000","last_active":"2023-10-10T18:45:11.429963+0000","last_peered":"2023-10-10T18:45:11.429963+0000","last_clean":"2023-10-10T18:45:11.429963+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T18:45:11.429963+0000","last_undegraded":"2023-10-10T18:45:11.429963+0000","last_fullsized":"2023-10-10T18:45:11.429963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430681+0000","last_change":"2023-10-10T18:45:11.416850+0000","last_active":"2023-10-10T18:45:11.430681+0000","last_peered":"2023-10-10T18:45:11.430681+0000","last_clean":"2023-10-10T18:45:11.430681+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T18:45:11.430681+0000","last_undegraded":"2023-10-10T18:45:11.430681+0000","last_fullsized":"2023-10-10T18:45:11.430681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427108+0000","last_change":"2023-10-10T18:45:11.424440+0000","last_active":"2023-10-10T18:45:11.427108+0000","last_peered":"2023-10-10T18:45:11.427108+0000","last_clean":"2023-10-10T18:45:11.427108+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T18:45:11.427108+0000","last_undegraded":"2023-10-10T18:45:11.427108+0000","last_fullsized":"2023-10-10T18:45:11.427108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.421524+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.420874+0000","last_became_peered":"2023-10-10T18:45:11.420874+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:56:36.851801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419177+0000","last_change":"2023-10-10T18:45:11.412399+0000","last_active":"2023-10-10T18:45:11.419177+0000","last_peered":"2023-10-10T18:45:11.419177+0000","last_clean":"2023-10-10T18:45:11.419177+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T18:45:11.419177+0000","last_undegraded":"2023-10-10T18:45:11.419177+0000","last_fullsized":"2023-10-10T18:45:11.419177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432996+0000","last_change":"2023-10-10T18:45:11.429507+0000","last_active":"2023-10-10T18:45:11.432996+0000","last_peered":"2023-10-10T18:45:11.432996+0000","last_clean":"2023-10-10T18:45:11.432996+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T18:45:11.432996+0000","last_undegraded":"2023-10-10T18:45:11.432996+0000","last_fullsized":"2023-10-10T18:45:11.432996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.418310+0000","last_change":"2023-10-10T18:45:11.413271+0000","last_active":"2023-10-10T18:45:11.418310+0000","last_peered":"2023-10-10T18:45:11.418310+0000","last_clean":"2023-10-10T18:45:11.418310+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T18:45:11.418310+0000","last_undegraded":"2023-10-10T18:45:11.418310+0000","last_fullsized":"2023-10-10T18:45:11.418310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430781+0000","last_change":"2023-10-10T18:45:11.427815+0000","last_active":"2023-10-10T18:45:11.430781+0000","last_peered":"2023-10-10T18:45:11.430781+0000","last_clean":"2023-10-10T18:45:11.430781+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T18:45:11.430781+0000","last_undegraded":"2023-10-10T18:45:11.430781+0000","last_fullsized":"2023-10-10T18:45:11.430781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426972+0000","last_change":"2023-10-10T18:45:11.425458+0000","last_active":"2023-10-10T18:45:11.426972+0000","last_peered":"2023-10-10T18:45:11.426972+0000","last_clean":"2023-10-10T18:45:11.426972+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T18:45:11.426972+0000","last_undegraded":"2023-10-10T18:45:11.426972+0000","last_fullsized":"2023-10-10T18:45:11.426972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424565+0000","last_change":"2023-10-10T18:45:11.422003+0000","last_active":"2023-10-10T18:45:11.424565+0000","last_peered":"2023-10-10T18:45:11.424565+0000","last_clean":"2023-10-10T18:45:11.424565+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T18:45:11.424565+0000","last_undegraded":"2023-10-10T18:45:11.424565+0000","last_fullsized":"2023-10-10T18:45:11.424565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432978+0000","last_change":"2023-10-10T18:45:11.428907+0000","last_active":"2023-10-10T18:45:11.432978+0000","last_peered":"2023-10-10T18:45:11.432978+0000","last_clean":"2023-10-10T18:45:11.432978+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T18:45:11.432978+0000","last_undegraded":"2023-10-10T18:45:11.432978+0000","last_fullsized":"2023-10-10T18:45:11.432978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430538+0000","last_change":"2023-10-10T18:45:11.427241+0000","last_active":"2023-10-10T18:45:11.430538+0000","last_peered":"2023-10-10T18:45:11.430538+0000","last_clean":"2023-10-10T18:45:11.430538+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T18:45:11.430538+0000","last_undegraded":"2023-10-10T18:45:11.430538+0000","last_fullsized":"2023-10-10T18:45:11.430538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427471+0000","last_change":"2023-10-10T18:45:11.426813+0000","last_active":"2023-10-10T18:45:11.427471+0000","last_peered":"2023-10-10T18:45:11.427471+0000","last_clean":"2023-10-10T18:45:11.427471+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T18:45:11.427471+0000","last_undegraded":"2023-10-10T18:45:11.427471+0000","last_fullsized":"2023-10-10T18:45:11.427471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432098+0000","last_change":"2023-10-10T18:45:11.429582+0000","last_active":"2023-10-10T18:45:11.432098+0000","last_peered":"2023-10-10T18:45:11.432098+0000","last_clean":"2023-10-10T18:45:11.432098+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T18:45:11.432098+0000","last_undegraded":"2023-10-10T18:45:11.432098+0000","last_fullsized":"2023-10-10T18:45:11.432098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297568,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074272,"statfs":{"total":96636764160,"available":96332054528,"internally_reserved":0,"allocated":286720,"data_stored":666235,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":294912,"data_stored":666291,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":75439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":75403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T18:45:22.126 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T18:45:22.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.273+0000 7fa2c66c1700 1 Processor -- start 2023-10-10T18:45:22.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.273+0000 7fa2c66c1700 1 -- start start 2023-10-10T18:45:22.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.274+0000 7fa2c66c1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa2c00c30c0 0x7fa2c014db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.274+0000 7fa2c66c1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa2c00c0270 0x7fa2c014e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.274+0000 7fa2c66c1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c006ec10 con 0x7fa2c00c2cf0 2023-10-10T18:45:22.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.274+0000 7fa2c66c1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c014e600 con 0x7fa2c00c30c0 2023-10-10T18:45:22.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.274+0000 7fa2c66c1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c0153f70 con 0x7fa2c00c0270 2023-10-10T18:45:22.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.274+0000 7fa2bf7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa2c00c30c0 0x7fa2c014db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.274+0000 7fa2bf7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa2c00c30c0 0x7fa2c014db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52646/0 (socket says 172.21.15.175:52646) 2023-10-10T18:45:22.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.274+0000 7fa2bf7fe700 1 -- 172.21.15.175:0/3428467398 learned_addr learned my addr 172.21.15.175:0/3428467398 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:22.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2c4c5e700 1 --2- 172.21.15.175:0/3428467398 >> v2:172.21.15.191:3300/0 conn(0x7fa2c00c0270 0x7fa2c014e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2bf7fe700 1 -- 172.21.15.175:0/3428467398 >> v2:172.21.15.191:3300/0 conn(0x7fa2c00c0270 msgr2=0x7fa2c014e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:22.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2bf7fe700 1 --2- 172.21.15.175:0/3428467398 >> v2:172.21.15.191:3300/0 conn(0x7fa2c00c0270 0x7fa2c014e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2bf7fe700 1 -- 172.21.15.175:0/3428467398 >> v1:172.21.15.191:6789/0 conn(0x7fa2c00c2cf0 legacy=0x7fa2c006e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:22.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2bf7fe700 1 -- 172.21.15.175:0/3428467398 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c0067bd0 con 0x7fa2c00c30c0 2023-10-10T18:45:22.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2bf7fe700 1 --2- 172.21.15.175:0/3428467398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c014db80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0063d60 tx=0x7fa2b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7abd0274339c2c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:22.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2beffd700 1 -- 172.21.15.175:0/3428467398 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b400b8b0 con 0x7fa2c00c30c0 2023-10-10T18:45:22.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2beffd700 1 -- 172.21.15.175:0/3428467398 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b400ba10 con 0x7fa2c00c30c0 2023-10-10T18:45:22.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2beffd700 1 -- 172.21.15.175:0/3428467398 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b400bdb0 con 0x7fa2c00c30c0 2023-10-10T18:45:22.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2c66c1700 1 -- 172.21.15.175:0/3428467398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 msgr2=0x7fa2c014db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.275+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/3428467398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c014db80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0063d60 tx=0x7fa2b4009250 comp rx=0 tx=0).stop 2023-10-10T18:45:22.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.276+0000 7fa2c66c1700 1 -- 172.21.15.175:0/3428467398 shutdown_connections 2023-10-10T18:45:22.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.276+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/3428467398 >> v2:172.21.15.191:3300/0 conn(0x7fa2c00c0270 0x7fa2c014e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.276+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/3428467398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c014db80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.276+0000 7fa2c66c1700 1 -- 172.21.15.175:0/3428467398 >> 172.21.15.175:0/3428467398 conn(0x7fa2c0068900 msgr2=0x7fa2c0068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:22.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.276+0000 7fa2c66c1700 1 -- 172.21.15.175:0/3428467398 shutdown_connections 2023-10-10T18:45:22.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.276+0000 7fa2c66c1700 1 -- 172.21.15.175:0/3428467398 wait complete. 2023-10-10T18:45:22.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.277+0000 7fa2c66c1700 1 Processor -- start 2023-10-10T18:45:22.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.277+0000 7fa2c66c1700 1 -- start start 2023-10-10T18:45:22.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.277+0000 7fa2c66c1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2c00c0270 0x7fa2c0151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.277+0000 7fa2c66c1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2c00c2cf0 0x7fa2c0167840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.277+0000 7fa2c66c1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c01429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.277+0000 7fa2c66c1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0168540 con 0x7fa2c00c30c0 2023-10-10T18:45:22.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.277+0000 7fa2c66c1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c01682a0 con 0x7fa2c00c0270 2023-10-10T18:45:22.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.277+0000 7fa2c66c1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c01487e0 con 0x7fa2c00c2cf0 2023-10-10T18:45:22.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c01429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c01429a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52652/0 (socket says 172.21.15.175:52652) 2023-10-10T18:45:22.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 -- 172.21.15.175:0/1313658623 learned_addr learned my addr 172.21.15.175:0/1313658623 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:22.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2bffff700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2c00c0270 0x7fa2c0151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2bf7fe700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2c00c2cf0 0x7fa2c0167840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 -- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2c00c2cf0 msgr2=0x7fa2c0167840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2c00c2cf0 0x7fa2c0167840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 -- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2c00c0270 msgr2=0x7fa2c0151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2c00c0270 0x7fa2c0151cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 -- 172.21.15.175:0/1313658623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40094f0 con 0x7fa2c00c30c0 2023-10-10T18:45:22.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2bf7fe700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2c00c2cf0 0x7fa2c0167840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:22.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2bffff700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2c00c0270 0x7fa2c0151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:22.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c4c5e700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c01429a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8007f20 tx=0x7fa2b8009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:22.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2a7fff700 1 -- 172.21.15.175:0/1313658623 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b800eca0 con 0x7fa2c00c30c0 2023-10-10T18:45:22.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2a7fff700 1 -- 172.21.15.175:0/1313658623 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b800ee20 con 0x7fa2c00c30c0 2023-10-10T18:45:22.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2a7fff700 1 -- 172.21.15.175:0/1313658623 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8014670 con 0x7fa2c00c30c0 2023-10-10T18:45:22.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c0148ac0 con 0x7fa2c00c30c0 2023-10-10T18:45:22.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.278+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c0149080 con 0x7fa2c00c30c0 2023-10-10T18:45:22.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.279+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c0067570 con 0x7fa2c00c30c0 2023-10-10T18:45:22.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.279+0000 7fa2a7fff700 1 -- 172.21.15.175:0/1313658623 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa2b80149e0 con 0x7fa2c00c30c0 2023-10-10T18:45:22.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.280+0000 7fa2a7fff700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2ac066f30 0x7fa2ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.280+0000 7fa2a7fff700 1 -- 172.21.15.175:0/1313658623 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa2b8084c60 con 0x7fa2c00c30c0 2023-10-10T18:45:22.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.280+0000 7fa2bffff700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2ac066f30 0x7fa2ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.280+0000 7fa2bffff700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2ac066f30 0x7fa2ac0693e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00028b0 tx=0x7fa2b0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:22.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.284+0000 7fa2a7fff700 1 -- 172.21.15.175:0/1313658623 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2b8051ed0 con 0x7fa2c00c30c0 2023-10-10T18:45:22.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.466+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2c00694a0 con 0x7fa2ac066f30 2023-10-10T18:45:22.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.474+0000 7fa2a7fff700 1 -- 172.21.15.175:0/1313658623 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264616 (secure 0 0 0) 0x7fa2c00694a0 con 0x7fa2ac066f30 2023-10-10T18:45:22.479 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:22.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2ac066f30 msgr2=0x7fa2ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2ac066f30 0x7fa2ac0693e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00028b0 tx=0x7fa2b0009040 comp rx=0 tx=0).stop 2023-10-10T18:45:22.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 msgr2=0x7fa2c01429a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c01429a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8007f20 tx=0x7fa2b8009860 comp rx=0 tx=0).stop 2023-10-10T18:45:22.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 shutdown_connections 2023-10-10T18:45:22.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2ac066f30 0x7fa2ac0693e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2c00c30c0 0x7fa2c01429a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2c00c0270 0x7fa2c0151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 --2- 172.21.15.175:0/1313658623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2c00c2cf0 0x7fa2c0167840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 >> 172.21.15.175:0/1313658623 conn(0x7fa2c0068900 msgr2=0x7fa2c006e500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:22.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 shutdown_connections 2023-10-10T18:45:22.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.477+0000 7fa2c66c1700 1 -- 172.21.15.175:0/1313658623 wait complete. 2023-10-10T18:45:22.484 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T18:45:22.490 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-10T18:45:21.131982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2380192,"kb_used_data":1888,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752594528,"statfs":{"total":773094113280,"available":770656796672,"internally_reserved":0,"allocated":1933312,"data_stored":1787852,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.728648"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429926+0000","last_change":"2023-10-10T18:45:11.419184+0000","last_active":"2023-10-10T18:45:11.429926+0000","last_peered":"2023-10-10T18:45:11.429926+0000","last_clean":"2023-10-10T18:45:11.429926+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T18:45:11.429926+0000","last_undegraded":"2023-10-10T18:45:11.429926+0000","last_fullsized":"2023-10-10T18:45:11.429926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420684+0000","last_change":"2023-10-10T18:45:11.414586+0000","last_active":"2023-10-10T18:45:11.420684+0000","last_peered":"2023-10-10T18:45:11.420684+0000","last_clean":"2023-10-10T18:45:11.420684+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T18:45:11.420684+0000","last_undegraded":"2023-10-10T18:45:11.420684+0000","last_fullsized":"2023-10-10T18:45:11.420684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424203+0000","last_change":"2023-10-10T18:45:11.419119+0000","last_active":"2023-10-10T18:45:11.424203+0000","last_peered":"2023-10-10T18:45:11.424203+0000","last_clean":"2023-10-10T18:45:11.424203+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T18:45:11.424203+0000","last_undegraded":"2023-10-10T18:45:11.424203+0000","last_fullsized":"2023-10-10T18:45:11.424203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425796+0000","last_change":"2023-10-10T18:45:11.421351+0000","last_active":"2023-10-10T18:45:11.425796+0000","last_peered":"2023-10-10T18:45:11.425796+0000","last_clean":"2023-10-10T18:45:11.425796+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T18:45:11.425796+0000","last_undegraded":"2023-10-10T18:45:11.425796+0000","last_fullsized":"2023-10-10T18:45:11.425796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425614+0000","last_change":"2023-10-10T18:45:11.421122+0000","last_active":"2023-10-10T18:45:11.425614+0000","last_peered":"2023-10-10T18:45:11.425614+0000","last_clean":"2023-10-10T18:45:11.425614+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T18:45:11.425614+0000","last_undegraded":"2023-10-10T18:45:11.425614+0000","last_fullsized":"2023-10-10T18:45:11.425614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424542+0000","last_change":"2023-10-10T18:45:11.421729+0000","last_active":"2023-10-10T18:45:11.424542+0000","last_peered":"2023-10-10T18:45:11.424542+0000","last_clean":"2023-10-10T18:45:11.424542+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T18:45:11.424542+0000","last_undegraded":"2023-10-10T18:45:11.424542+0000","last_fullsized":"2023-10-10T18:45:11.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426178+0000","last_change":"2023-10-10T18:45:11.419306+0000","last_active":"2023-10-10T18:45:11.426178+0000","last_peered":"2023-10-10T18:45:11.426178+0000","last_clean":"2023-10-10T18:45:11.426178+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T18:45:11.426178+0000","last_undegraded":"2023-10-10T18:45:11.426178+0000","last_fullsized":"2023-10-10T18:45:11.426178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.422940+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429946+0000","last_change":"2023-10-10T18:45:11.422612+0000","last_active":"2023-10-10T18:45:11.429946+0000","last_peered":"2023-10-10T18:45:11.429946+0000","last_clean":"2023-10-10T18:45:11.429946+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T18:45:11.429946+0000","last_undegraded":"2023-10-10T18:45:11.429946+0000","last_fullsized":"2023-10-10T18:45:11.429946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430763+0000","last_change":"2023-10-10T18:45:11.425754+0000","last_active":"2023-10-10T18:45:11.430763+0000","last_peered":"2023-10-10T18:45:11.430763+0000","last_clean":"2023-10-10T18:45:11.430763+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T18:45:11.430763+0000","last_undegraded":"2023-10-10T18:45:11.430763+0000","last_fullsized":"2023-10-10T18:45:11.430763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430664+0000","last_change":"2023-10-10T18:45:11.425891+0000","last_active":"2023-10-10T18:45:11.430664+0000","last_peered":"2023-10-10T18:45:11.430664+0000","last_clean":"2023-10-10T18:45:11.430664+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T18:45:11.430664+0000","last_undegraded":"2023-10-10T18:45:11.430664+0000","last_fullsized":"2023-10-10T18:45:11.430664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431015+0000","last_change":"2023-10-10T18:45:11.424834+0000","last_active":"2023-10-10T18:45:11.431015+0000","last_peered":"2023-10-10T18:45:11.431015+0000","last_clean":"2023-10-10T18:45:11.431015+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T18:45:11.431015+0000","last_undegraded":"2023-10-10T18:45:11.431015+0000","last_fullsized":"2023-10-10T18:45:11.431015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421085+0000","last_change":"2023-10-10T18:45:11.417796+0000","last_active":"2023-10-10T18:45:11.421085+0000","last_peered":"2023-10-10T18:45:11.421085+0000","last_clean":"2023-10-10T18:45:11.421085+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T18:45:11.421085+0000","last_undegraded":"2023-10-10T18:45:11.421085+0000","last_fullsized":"2023-10-10T18:45:11.421085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419270+0000","last_change":"2023-10-10T18:45:11.414690+0000","last_active":"2023-10-10T18:45:11.419270+0000","last_peered":"2023-10-10T18:45:11.419270+0000","last_clean":"2023-10-10T18:45:11.419270+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T18:45:11.419270+0000","last_undegraded":"2023-10-10T18:45:11.419270+0000","last_fullsized":"2023-10-10T18:45:11.419270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434526+0000","last_change":"2023-10-10T18:45:11.429433+0000","last_active":"2023-10-10T18:45:11.434526+0000","last_peered":"2023-10-10T18:45:11.434526+0000","last_clean":"2023-10-10T18:45:11.434526+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T18:45:11.434526+0000","last_undegraded":"2023-10-10T18:45:11.434526+0000","last_fullsized":"2023-10-10T18:45:11.434526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427193+0000","last_change":"2023-10-10T18:45:11.422935+0000","last_active":"2023-10-10T18:45:11.427193+0000","last_peered":"2023-10-10T18:45:11.427193+0000","last_clean":"2023-10-10T18:45:11.427193+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T18:45:11.427193+0000","last_undegraded":"2023-10-10T18:45:11.427193+0000","last_fullsized":"2023-10-10T18:45:11.427193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434131+0000","last_change":"2023-10-10T18:45:11.431202+0000","last_active":"2023-10-10T18:45:11.434131+0000","last_peered":"2023-10-10T18:45:11.434131+0000","last_clean":"2023-10-10T18:45:11.434131+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T18:45:11.434131+0000","last_undegraded":"2023-10-10T18:45:11.434131+0000","last_fullsized":"2023-10-10T18:45:11.434131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427139+0000","last_change":"2023-10-10T18:45:11.424885+0000","last_active":"2023-10-10T18:45:11.427139+0000","last_peered":"2023-10-10T18:45:11.427139+0000","last_clean":"2023-10-10T18:45:11.427139+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T18:45:11.427139+0000","last_undegraded":"2023-10-10T18:45:11.427139+0000","last_fullsized":"2023-10-10T18:45:11.427139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426206+0000","last_change":"2023-10-10T18:45:11.421039+0000","last_active":"2023-10-10T18:45:11.426206+0000","last_peered":"2023-10-10T18:45:11.426206+0000","last_clean":"2023-10-10T18:45:11.426206+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T18:45:11.426206+0000","last_undegraded":"2023-10-10T18:45:11.426206+0000","last_fullsized":"2023-10-10T18:45:11.426206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421148+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T18:45:16.421148+0000","last_peered":"2023-10-10T18:45:16.421148+0000","last_clean":"2023-10-10T18:45:16.421148+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T18:45:16.421148+0000","last_undegraded":"2023-10-10T18:45:16.421148+0000","last_fullsized":"2023-10-10T18:45:16.421148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421043+0000","last_change":"2023-10-10T18:45:11.417504+0000","last_active":"2023-10-10T18:45:11.421043+0000","last_peered":"2023-10-10T18:45:11.421043+0000","last_clean":"2023-10-10T18:45:11.421043+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T18:45:11.421043+0000","last_undegraded":"2023-10-10T18:45:11.421043+0000","last_fullsized":"2023-10-10T18:45:11.421043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.010024+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T18:45:13.010024+0000","last_peered":"2023-10-10T18:45:13.010024+0000","last_clean":"2023-10-10T18:45:13.010024+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T18:45:13.010024+0000","last_undegraded":"2023-10-10T18:45:13.010024+0000","last_fullsized":"2023-10-10T18:45:13.010024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424178+0000","last_change":"2023-10-10T18:45:11.421896+0000","last_active":"2023-10-10T18:45:11.424178+0000","last_peered":"2023-10-10T18:45:11.424178+0000","last_clean":"2023-10-10T18:45:11.424178+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T18:45:11.424178+0000","last_undegraded":"2023-10-10T18:45:11.424178+0000","last_fullsized":"2023-10-10T18:45:11.424178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.418723+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T18:45:16.418723+0000","last_peered":"2023-10-10T18:45:16.418723+0000","last_clean":"2023-10-10T18:45:16.418723+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T18:45:16.418723+0000","last_undegraded":"2023-10-10T18:45:16.418723+0000","last_fullsized":"2023-10-10T18:45:16.418723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432466+0000","last_change":"2023-10-10T18:45:11.429680+0000","last_active":"2023-10-10T18:45:11.432466+0000","last_peered":"2023-10-10T18:45:11.432466+0000","last_clean":"2023-10-10T18:45:11.432466+0000","last_became_active":"2023-10-10T18:45:11.429397+0000","last_became_peered":"2023-10-10T18:45:11.429397+0000","last_unstale":"2023-10-10T18:45:11.432466+0000","last_undegraded":"2023-10-10T18:45:11.432466+0000","last_fullsized":"2023-10-10T18:45:11.432466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:19:52.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.418579+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T18:45:16.418579+0000","last_peered":"2023-10-10T18:45:16.418579+0000","last_clean":"2023-10-10T18:45:16.418579+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T18:45:16.418579+0000","last_undegraded":"2023-10-10T18:45:16.418579+0000","last_fullsized":"2023-10-10T18:45:16.418579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432439+0000","last_change":"2023-10-10T18:45:11.429408+0000","last_active":"2023-10-10T18:45:11.432439+0000","last_peered":"2023-10-10T18:45:11.432439+0000","last_clean":"2023-10-10T18:45:11.432439+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T18:45:11.432439+0000","last_undegraded":"2023-10-10T18:45:11.432439+0000","last_fullsized":"2023-10-10T18:45:11.432439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.010081+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T18:45:13.010081+0000","last_peered":"2023-10-10T18:45:13.010081+0000","last_clean":"2023-10-10T18:45:13.010081+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T18:45:13.010081+0000","last_undegraded":"2023-10-10T18:45:13.010081+0000","last_fullsized":"2023-10-10T18:45:13.010081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430579+0000","last_change":"2023-10-10T18:45:11.426051+0000","last_active":"2023-10-10T18:45:11.430579+0000","last_peered":"2023-10-10T18:45:11.430579+0000","last_clean":"2023-10-10T18:45:11.430579+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T18:45:11.430579+0000","last_undegraded":"2023-10-10T18:45:11.430579+0000","last_fullsized":"2023-10-10T18:45:11.430579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.009453+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T18:45:13.009453+0000","last_peered":"2023-10-10T18:45:13.009453+0000","last_clean":"2023-10-10T18:45:13.009453+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T18:45:13.009453+0000","last_undegraded":"2023-10-10T18:45:13.009453+0000","last_fullsized":"2023-10-10T18:45:13.009453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427223+0000","last_change":"2023-10-10T18:45:11.425183+0000","last_active":"2023-10-10T18:45:11.427223+0000","last_peered":"2023-10-10T18:45:11.427223+0000","last_clean":"2023-10-10T18:45:11.427223+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T18:45:11.427223+0000","last_undegraded":"2023-10-10T18:45:11.427223+0000","last_fullsized":"2023-10-10T18:45:11.427223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.402044+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T18:45:11.402044+0000","last_peered":"2023-10-10T18:45:11.402044+0000","last_clean":"2023-10-10T18:45:11.402044+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T18:45:11.402044+0000","last_undegraded":"2023-10-10T18:45:11.402044+0000","last_fullsized":"2023-10-10T18:45:11.402044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415249+0000","last_change":"2023-10-10T18:45:11.412641+0000","last_active":"2023-10-10T18:45:11.415249+0000","last_peered":"2023-10-10T18:45:11.415249+0000","last_clean":"2023-10-10T18:45:11.415249+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T18:45:11.415249+0000","last_undegraded":"2023-10-10T18:45:11.415249+0000","last_fullsized":"2023-10-10T18:45:11.415249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.413032+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T18:45:11.413032+0000","last_peered":"2023-10-10T18:45:11.413032+0000","last_clean":"2023-10-10T18:45:11.413032+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T18:45:11.413032+0000","last_undegraded":"2023-10-10T18:45:11.413032+0000","last_fullsized":"2023-10-10T18:45:11.413032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420638+0000","last_change":"2023-10-10T18:45:11.413819+0000","last_active":"2023-10-10T18:45:11.420638+0000","last_peered":"2023-10-10T18:45:11.420638+0000","last_clean":"2023-10-10T18:45:11.420638+0000","last_became_active":"2023-10-10T18:45:11.413529+0000","last_became_peered":"2023-10-10T18:45:11.413529+0000","last_unstale":"2023-10-10T18:45:11.420638+0000","last_undegraded":"2023-10-10T18:45:11.420638+0000","last_fullsized":"2023-10-10T18:45:11.420638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.411211+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T18:45:11.411211+0000","last_peered":"2023-10-10T18:45:11.411211+0000","last_clean":"2023-10-10T18:45:11.411211+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T18:45:11.411211+0000","last_undegraded":"2023-10-10T18:45:11.411211+0000","last_fullsized":"2023-10-10T18:45:11.411211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430421+0000","last_change":"2023-10-10T18:45:11.427116+0000","last_active":"2023-10-10T18:45:11.430421+0000","last_peered":"2023-10-10T18:45:11.430421+0000","last_clean":"2023-10-10T18:45:11.430421+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T18:45:11.430421+0000","last_undegraded":"2023-10-10T18:45:11.430421+0000","last_fullsized":"2023-10-10T18:45:11.430421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.416463+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T18:45:11.416463+0000","last_peered":"2023-10-10T18:45:11.416463+0000","last_clean":"2023-10-10T18:45:11.416463+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T18:45:11.416463+0000","last_undegraded":"2023-10-10T18:45:11.416463+0000","last_fullsized":"2023-10-10T18:45:11.416463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424158+0000","last_change":"2023-10-10T18:45:11.421794+0000","last_active":"2023-10-10T18:45:11.424158+0000","last_peered":"2023-10-10T18:45:11.424158+0000","last_clean":"2023-10-10T18:45:11.424158+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T18:45:11.424158+0000","last_undegraded":"2023-10-10T18:45:11.424158+0000","last_fullsized":"2023-10-10T18:45:11.424158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.418436+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T18:45:16.418436+0000","last_peered":"2023-10-10T18:45:16.418436+0000","last_clean":"2023-10-10T18:45:16.418436+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T18:45:16.418436+0000","last_undegraded":"2023-10-10T18:45:16.418436+0000","last_fullsized":"2023-10-10T18:45:16.418436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434442+0000","last_change":"2023-10-10T18:45:11.431857+0000","last_active":"2023-10-10T18:45:11.434442+0000","last_peered":"2023-10-10T18:45:11.434442+0000","last_clean":"2023-10-10T18:45:11.434442+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T18:45:11.434442+0000","last_undegraded":"2023-10-10T18:45:11.434442+0000","last_fullsized":"2023-10-10T18:45:11.434442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421659+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T18:45:16.421659+0000","last_peered":"2023-10-10T18:45:16.421659+0000","last_clean":"2023-10-10T18:45:16.421659+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T18:45:16.421659+0000","last_undegraded":"2023-10-10T18:45:16.421659+0000","last_fullsized":"2023-10-10T18:45:16.421659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426196+0000","last_change":"2023-10-10T18:45:11.419774+0000","last_active":"2023-10-10T18:45:11.426196+0000","last_peered":"2023-10-10T18:45:11.426196+0000","last_clean":"2023-10-10T18:45:11.426196+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T18:45:11.426196+0000","last_undegraded":"2023-10-10T18:45:11.426196+0000","last_fullsized":"2023-10-10T18:45:11.426196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.419416+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T18:45:16.419416+0000","last_peered":"2023-10-10T18:45:16.419416+0000","last_clean":"2023-10-10T18:45:16.419416+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T18:45:16.419416+0000","last_undegraded":"2023-10-10T18:45:16.419416+0000","last_fullsized":"2023-10-10T18:45:16.419416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424245+0000","last_change":"2023-10-10T18:45:11.420517+0000","last_active":"2023-10-10T18:45:11.424245+0000","last_peered":"2023-10-10T18:45:11.424245+0000","last_clean":"2023-10-10T18:45:11.424245+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T18:45:11.424245+0000","last_undegraded":"2023-10-10T18:45:11.424245+0000","last_fullsized":"2023-10-10T18:45:11.424245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412727+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T18:45:11.412727+0000","last_peered":"2023-10-10T18:45:11.412727+0000","last_clean":"2023-10-10T18:45:11.412727+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T18:45:11.412727+0000","last_undegraded":"2023-10-10T18:45:11.412727+0000","last_fullsized":"2023-10-10T18:45:11.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431834+0000","last_change":"2023-10-10T18:45:11.427109+0000","last_active":"2023-10-10T18:45:11.431834+0000","last_peered":"2023-10-10T18:45:11.431834+0000","last_clean":"2023-10-10T18:45:11.431834+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T18:45:11.431834+0000","last_undegraded":"2023-10-10T18:45:11.431834+0000","last_fullsized":"2023-10-10T18:45:11.431834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410202+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T18:45:11.410202+0000","last_peered":"2023-10-10T18:45:11.410202+0000","last_clean":"2023-10-10T18:45:11.410202+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T18:45:11.410202+0000","last_undegraded":"2023-10-10T18:45:11.410202+0000","last_fullsized":"2023-10-10T18:45:11.410202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430660+0000","last_change":"2023-10-10T18:45:11.416459+0000","last_active":"2023-10-10T18:45:11.430660+0000","last_peered":"2023-10-10T18:45:11.430660+0000","last_clean":"2023-10-10T18:45:11.430660+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T18:45:11.430660+0000","last_undegraded":"2023-10-10T18:45:11.430660+0000","last_fullsized":"2023-10-10T18:45:11.430660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421838+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T18:45:16.421838+0000","last_peered":"2023-10-10T18:45:16.421838+0000","last_clean":"2023-10-10T18:45:16.421838+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T18:45:16.421838+0000","last_undegraded":"2023-10-10T18:45:16.421838+0000","last_fullsized":"2023-10-10T18:45:16.421838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432328+0000","last_change":"2023-10-10T18:45:11.429727+0000","last_active":"2023-10-10T18:45:11.432328+0000","last_peered":"2023-10-10T18:45:11.432328+0000","last_clean":"2023-10-10T18:45:11.432328+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T18:45:11.432328+0000","last_undegraded":"2023-10-10T18:45:11.432328+0000","last_fullsized":"2023-10-10T18:45:11.432328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:12.418575+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T18:45:12.418575+0000","last_peered":"2023-10-10T18:45:12.418575+0000","last_clean":"2023-10-10T18:45:12.418575+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T18:45:12.418575+0000","last_undegraded":"2023-10-10T18:45:12.418575+0000","last_fullsized":"2023-10-10T18:45:12.418575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431858+0000","last_change":"2023-10-10T18:45:11.427926+0000","last_active":"2023-10-10T18:45:11.431858+0000","last_peered":"2023-10-10T18:45:11.431858+0000","last_clean":"2023-10-10T18:45:11.431858+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T18:45:11.431858+0000","last_undegraded":"2023-10-10T18:45:11.431858+0000","last_fullsized":"2023-10-10T18:45:11.431858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407823+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T18:45:11.407823+0000","last_peered":"2023-10-10T18:45:11.407823+0000","last_clean":"2023-10-10T18:45:11.407823+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T18:45:11.407823+0000","last_undegraded":"2023-10-10T18:45:11.407823+0000","last_fullsized":"2023-10-10T18:45:11.407823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427955+0000","last_change":"2023-10-10T18:45:11.423279+0000","last_active":"2023-10-10T18:45:11.427955+0000","last_peered":"2023-10-10T18:45:11.427955+0000","last_clean":"2023-10-10T18:45:11.427955+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T18:45:11.427955+0000","last_undegraded":"2023-10-10T18:45:11.427955+0000","last_fullsized":"2023-10-10T18:45:11.427955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412668+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T18:45:11.412668+0000","last_peered":"2023-10-10T18:45:11.412668+0000","last_clean":"2023-10-10T18:45:11.412668+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T18:45:11.412668+0000","last_undegraded":"2023-10-10T18:45:11.412668+0000","last_fullsized":"2023-10-10T18:45:11.412668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430728+0000","last_change":"2023-10-10T18:45:11.425339+0000","last_active":"2023-10-10T18:45:11.430728+0000","last_peered":"2023-10-10T18:45:11.430728+0000","last_clean":"2023-10-10T18:45:11.430728+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T18:45:11.430728+0000","last_undegraded":"2023-10-10T18:45:11.430728+0000","last_fullsized":"2023-10-10T18:45:11.430728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421278+0000","last_change":"2023-10-10T18:45:11.418937+0000","last_active":"2023-10-10T18:45:11.421278+0000","last_peered":"2023-10-10T18:45:11.421278+0000","last_clean":"2023-10-10T18:45:11.421278+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T18:45:11.421278+0000","last_undegraded":"2023-10-10T18:45:11.421278+0000","last_fullsized":"2023-10-10T18:45:11.421278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410721+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T18:45:11.410721+0000","last_peered":"2023-10-10T18:45:11.410721+0000","last_clean":"2023-10-10T18:45:11.410721+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T18:45:11.410721+0000","last_undegraded":"2023-10-10T18:45:11.410721+0000","last_fullsized":"2023-10-10T18:45:11.410721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424136+0000","last_change":"2023-10-10T18:45:11.422452+0000","last_active":"2023-10-10T18:45:11.424136+0000","last_peered":"2023-10-10T18:45:11.424136+0000","last_clean":"2023-10-10T18:45:11.424136+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T18:45:11.424136+0000","last_undegraded":"2023-10-10T18:45:11.424136+0000","last_fullsized":"2023-10-10T18:45:11.424136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.410662+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T18:45:11.410662+0000","last_peered":"2023-10-10T18:45:11.410662+0000","last_clean":"2023-10-10T18:45:11.410662+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T18:45:11.410662+0000","last_undegraded":"2023-10-10T18:45:11.410662+0000","last_fullsized":"2023-10-10T18:45:11.410662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405467+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T18:45:11.405467+0000","last_peered":"2023-10-10T18:45:11.405467+0000","last_clean":"2023-10-10T18:45:11.405467+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T18:45:11.405467+0000","last_undegraded":"2023-10-10T18:45:11.405467+0000","last_fullsized":"2023-10-10T18:45:11.405467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434404+0000","last_change":"2023-10-10T18:45:11.430233+0000","last_active":"2023-10-10T18:45:11.434404+0000","last_peered":"2023-10-10T18:45:11.434404+0000","last_clean":"2023-10-10T18:45:11.434404+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T18:45:11.434404+0000","last_undegraded":"2023-10-10T18:45:11.434404+0000","last_fullsized":"2023-10-10T18:45:11.434404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405812+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T18:45:11.405812+0000","last_peered":"2023-10-10T18:45:11.405812+0000","last_clean":"2023-10-10T18:45:11.405812+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T18:45:11.405812+0000","last_undegraded":"2023-10-10T18:45:11.405812+0000","last_fullsized":"2023-10-10T18:45:11.405812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.408007+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T18:45:11.408007+0000","last_peered":"2023-10-10T18:45:11.408007+0000","last_clean":"2023-10-10T18:45:11.408007+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T18:45:11.408007+0000","last_undegraded":"2023-10-10T18:45:11.408007+0000","last_fullsized":"2023-10-10T18:45:11.408007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421286+0000","last_change":"2023-10-10T18:45:11.418973+0000","last_active":"2023-10-10T18:45:11.421286+0000","last_peered":"2023-10-10T18:45:11.421286+0000","last_clean":"2023-10-10T18:45:11.421286+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T18:45:11.421286+0000","last_undegraded":"2023-10-10T18:45:11.421286+0000","last_fullsized":"2023-10-10T18:45:11.421286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407798+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T18:45:11.407798+0000","last_peered":"2023-10-10T18:45:11.407798+0000","last_clean":"2023-10-10T18:45:11.407798+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T18:45:11.407798+0000","last_undegraded":"2023-10-10T18:45:11.407798+0000","last_fullsized":"2023-10-10T18:45:11.407798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423727+0000","last_change":"2023-10-10T18:45:11.419846+0000","last_active":"2023-10-10T18:45:11.423727+0000","last_peered":"2023-10-10T18:45:11.423727+0000","last_clean":"2023-10-10T18:45:11.423727+0000","last_became_active":"2023-10-10T18:45:11.419579+0000","last_became_peered":"2023-10-10T18:45:11.419579+0000","last_unstale":"2023-10-10T18:45:11.423727+0000","last_undegraded":"2023-10-10T18:45:11.423727+0000","last_fullsized":"2023-10-10T18:45:11.423727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:58:49.831558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.418931+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T18:45:16.418931+0000","last_peered":"2023-10-10T18:45:16.418931+0000","last_clean":"2023-10-10T18:45:16.418931+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T18:45:16.418931+0000","last_undegraded":"2023-10-10T18:45:16.418931+0000","last_fullsized":"2023-10-10T18:45:16.418931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423663+0000","last_change":"2023-10-10T18:45:11.416217+0000","last_active":"2023-10-10T18:45:11.423663+0000","last_peered":"2023-10-10T18:45:11.423663+0000","last_clean":"2023-10-10T18:45:11.423663+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T18:45:11.423663+0000","last_undegraded":"2023-10-10T18:45:11.423663+0000","last_fullsized":"2023-10-10T18:45:11.423663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405671+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T18:45:11.405671+0000","last_peered":"2023-10-10T18:45:11.405671+0000","last_clean":"2023-10-10T18:45:11.405671+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T18:45:11.405671+0000","last_undegraded":"2023-10-10T18:45:11.405671+0000","last_fullsized":"2023-10-10T18:45:11.405671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431602+0000","last_change":"2023-10-10T18:45:11.427826+0000","last_active":"2023-10-10T18:45:11.431602+0000","last_peered":"2023-10-10T18:45:11.431602+0000","last_clean":"2023-10-10T18:45:11.431602+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T18:45:11.431602+0000","last_undegraded":"2023-10-10T18:45:11.431602+0000","last_fullsized":"2023-10-10T18:45:11.431602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407958+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T18:45:11.407958+0000","last_peered":"2023-10-10T18:45:11.407958+0000","last_clean":"2023-10-10T18:45:11.407958+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T18:45:11.407958+0000","last_undegraded":"2023-10-10T18:45:11.407958+0000","last_fullsized":"2023-10-10T18:45:11.407958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415388+0000","last_change":"2023-10-10T18:45:11.413975+0000","last_active":"2023-10-10T18:45:11.415388+0000","last_peered":"2023-10-10T18:45:11.415388+0000","last_clean":"2023-10-10T18:45:11.415388+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T18:45:11.415388+0000","last_undegraded":"2023-10-10T18:45:11.415388+0000","last_fullsized":"2023-10-10T18:45:11.415388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.407755+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T18:45:11.407755+0000","last_peered":"2023-10-10T18:45:11.407755+0000","last_clean":"2023-10-10T18:45:11.407755+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T18:45:11.407755+0000","last_undegraded":"2023-10-10T18:45:11.407755+0000","last_fullsized":"2023-10-10T18:45:11.407755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415058+0000","last_change":"2023-10-10T18:45:11.410768+0000","last_active":"2023-10-10T18:45:11.415058+0000","last_peered":"2023-10-10T18:45:11.415058+0000","last_clean":"2023-10-10T18:45:11.415058+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T18:45:11.415058+0000","last_undegraded":"2023-10-10T18:45:11.415058+0000","last_fullsized":"2023-10-10T18:45:11.415058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.419134+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T18:45:16.419134+0000","last_peered":"2023-10-10T18:45:16.419134+0000","last_clean":"2023-10-10T18:45:16.419134+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T18:45:16.419134+0000","last_undegraded":"2023-10-10T18:45:16.419134+0000","last_fullsized":"2023-10-10T18:45:16.419134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430290+0000","last_change":"2023-10-10T18:45:11.425716+0000","last_active":"2023-10-10T18:45:11.430290+0000","last_peered":"2023-10-10T18:45:11.430290+0000","last_clean":"2023-10-10T18:45:11.430290+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T18:45:11.430290+0000","last_undegraded":"2023-10-10T18:45:11.430290+0000","last_fullsized":"2023-10-10T18:45:11.430290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.412053+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T18:45:11.412053+0000","last_peered":"2023-10-10T18:45:11.412053+0000","last_clean":"2023-10-10T18:45:11.412053+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T18:45:11.412053+0000","last_undegraded":"2023-10-10T18:45:11.412053+0000","last_fullsized":"2023-10-10T18:45:11.412053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429787+0000","last_change":"2023-10-10T18:45:11.423747+0000","last_active":"2023-10-10T18:45:11.429787+0000","last_peered":"2023-10-10T18:45:11.429787+0000","last_clean":"2023-10-10T18:45:11.429787+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T18:45:11.429787+0000","last_undegraded":"2023-10-10T18:45:11.429787+0000","last_fullsized":"2023-10-10T18:45:11.429787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.009694+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T18:45:13.009694+0000","last_peered":"2023-10-10T18:45:13.009694+0000","last_clean":"2023-10-10T18:45:13.009694+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T18:45:13.009694+0000","last_undegraded":"2023-10-10T18:45:13.009694+0000","last_fullsized":"2023-10-10T18:45:13.009694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425816+0000","last_change":"2023-10-10T18:45:11.421872+0000","last_active":"2023-10-10T18:45:11.425816+0000","last_peered":"2023-10-10T18:45:11.425816+0000","last_clean":"2023-10-10T18:45:11.425816+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T18:45:11.425816+0000","last_undegraded":"2023-10-10T18:45:11.425816+0000","last_fullsized":"2023-10-10T18:45:11.425816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421169+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T18:45:16.421169+0000","last_peered":"2023-10-10T18:45:16.421169+0000","last_clean":"2023-10-10T18:45:16.421169+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T18:45:16.421169+0000","last_undegraded":"2023-10-10T18:45:16.421169+0000","last_fullsized":"2023-10-10T18:45:16.421169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429963+0000","last_change":"2023-10-10T18:45:11.425474+0000","last_active":"2023-10-10T18:45:11.429963+0000","last_peered":"2023-10-10T18:45:11.429963+0000","last_clean":"2023-10-10T18:45:11.429963+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T18:45:11.429963+0000","last_undegraded":"2023-10-10T18:45:11.429963+0000","last_fullsized":"2023-10-10T18:45:11.429963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430681+0000","last_change":"2023-10-10T18:45:11.416850+0000","last_active":"2023-10-10T18:45:11.430681+0000","last_peered":"2023-10-10T18:45:11.430681+0000","last_clean":"2023-10-10T18:45:11.430681+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T18:45:11.430681+0000","last_undegraded":"2023-10-10T18:45:11.430681+0000","last_fullsized":"2023-10-10T18:45:11.430681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427108+0000","last_change":"2023-10-10T18:45:11.424440+0000","last_active":"2023-10-10T18:45:11.427108+0000","last_peered":"2023-10-10T18:45:11.427108+0000","last_clean":"2023-10-10T18:45:11.427108+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T18:45:11.427108+0000","last_undegraded":"2023-10-10T18:45:11.427108+0000","last_fullsized":"2023-10-10T18:45:11.427108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.421524+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.420874+0000","last_became_peered":"2023-10-10T18:45:11.420874+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:56:36.851801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419177+0000","last_change":"2023-10-10T18:45:11.412399+0000","last_active":"2023-10-10T18:45:11.419177+0000","last_peered":"2023-10-10T18:45:11.419177+0000","last_clean":"2023-10-10T18:45:11.419177+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T18:45:11.419177+0000","last_undegraded":"2023-10-10T18:45:11.419177+0000","last_fullsized":"2023-10-10T18:45:11.419177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432996+0000","last_change":"2023-10-10T18:45:11.429507+0000","last_active":"2023-10-10T18:45:11.432996+0000","last_peered":"2023-10-10T18:45:11.432996+0000","last_clean":"2023-10-10T18:45:11.432996+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T18:45:11.432996+0000","last_undegraded":"2023-10-10T18:45:11.432996+0000","last_fullsized":"2023-10-10T18:45:11.432996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.418310+0000","last_change":"2023-10-10T18:45:11.413271+0000","last_active":"2023-10-10T18:45:11.418310+0000","last_peered":"2023-10-10T18:45:11.418310+0000","last_clean":"2023-10-10T18:45:11.418310+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T18:45:11.418310+0000","last_undegraded":"2023-10-10T18:45:11.418310+0000","last_fullsized":"2023-10-10T18:45:11.418310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430781+0000","last_change":"2023-10-10T18:45:11.427815+0000","last_active":"2023-10-10T18:45:11.430781+0000","last_peered":"2023-10-10T18:45:11.430781+0000","last_clean":"2023-10-10T18:45:11.430781+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T18:45:11.430781+0000","last_undegraded":"2023-10-10T18:45:11.430781+0000","last_fullsized":"2023-10-10T18:45:11.430781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426972+0000","last_change":"2023-10-10T18:45:11.425458+0000","last_active":"2023-10-10T18:45:11.426972+0000","last_peered":"2023-10-10T18:45:11.426972+0000","last_clean":"2023-10-10T18:45:11.426972+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T18:45:11.426972+0000","last_undegraded":"2023-10-10T18:45:11.426972+0000","last_fullsized":"2023-10-10T18:45:11.426972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424565+0000","last_change":"2023-10-10T18:45:11.422003+0000","last_active":"2023-10-10T18:45:11.424565+0000","last_peered":"2023-10-10T18:45:11.424565+0000","last_clean":"2023-10-10T18:45:11.424565+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T18:45:11.424565+0000","last_undegraded":"2023-10-10T18:45:11.424565+0000","last_fullsized":"2023-10-10T18:45:11.424565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432978+0000","last_change":"2023-10-10T18:45:11.428907+0000","last_active":"2023-10-10T18:45:11.432978+0000","last_peered":"2023-10-10T18:45:11.432978+0000","last_clean":"2023-10-10T18:45:11.432978+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T18:45:11.432978+0000","last_undegraded":"2023-10-10T18:45:11.432978+0000","last_fullsized":"2023-10-10T18:45:11.432978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430538+0000","last_change":"2023-10-10T18:45:11.427241+0000","last_active":"2023-10-10T18:45:11.430538+0000","last_peered":"2023-10-10T18:45:11.430538+0000","last_clean":"2023-10-10T18:45:11.430538+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T18:45:11.430538+0000","last_undegraded":"2023-10-10T18:45:11.430538+0000","last_fullsized":"2023-10-10T18:45:11.430538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427471+0000","last_change":"2023-10-10T18:45:11.426813+0000","last_active":"2023-10-10T18:45:11.427471+0000","last_peered":"2023-10-10T18:45:11.427471+0000","last_clean":"2023-10-10T18:45:11.427471+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T18:45:11.427471+0000","last_undegraded":"2023-10-10T18:45:11.427471+0000","last_fullsized":"2023-10-10T18:45:11.427471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432098+0000","last_change":"2023-10-10T18:45:11.429582+0000","last_active":"2023-10-10T18:45:11.432098+0000","last_peered":"2023-10-10T18:45:11.432098+0000","last_clean":"2023-10-10T18:45:11.432098+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T18:45:11.432098+0000","last_undegraded":"2023-10-10T18:45:11.432098+0000","last_fullsized":"2023-10-10T18:45:11.432098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297568,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074272,"statfs":{"total":96636764160,"available":96332054528,"internally_reserved":0,"allocated":286720,"data_stored":666235,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":294912,"data_stored":666291,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":75439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":75403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T18:45:22.496 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-10T18:45:22.496 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-10T18:45:22.496 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-10T18:45:22.497 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-10T18:45:22.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.643+0000 7f095759e700 1 Processor -- start 2023-10-10T18:45:22.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.643+0000 7f095759e700 1 -- start start 2023-10-10T18:45:22.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095759e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0958157400 0x7f095806bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095759e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09580c4640 0x7f09581577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095759e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095806d8e0 con 0x7f0958157d10 2023-10-10T18:45:22.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095759e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095806e050 con 0x7f0958157400 2023-10-10T18:45:22.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095759e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095806dbc0 con 0x7f09580c4640 2023-10-10T18:45:22.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0958157400 0x7f095806bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0958157400 0x7f095806bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52666/0 (socket says 172.21.15.175:52666) 2023-10-10T18:45:22.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095659c700 1 -- 172.21.15.175:0/235360103 learned_addr learned my addr 172.21.15.175:0/235360103 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:22.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095659c700 1 -- 172.21.15.175:0/235360103 >> v2:172.21.15.191:3300/0 conn(0x7f09580c4640 msgr2=0x7f09581577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:22.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f0955d9b700 1 --2- 172.21.15.175:0/235360103 >> v2:172.21.15.191:3300/0 conn(0x7f09580c4640 0x7f09581577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095659c700 1 --2- 172.21.15.175:0/235360103 >> v2:172.21.15.191:3300/0 conn(0x7f09580c4640 0x7f09581577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095659c700 1 -- 172.21.15.175:0/235360103 >> v1:172.21.15.175:6789/0 conn(0x7f0958157d10 legacy=0x7f095814d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095559a700 1 -- 172.21.15.175:0/235360103 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257100866 0 0) 0x7f095806d8e0 con 0x7f0958157d10 2023-10-10T18:45:22.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095659c700 1 -- 172.21.15.175:0/235360103 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09580c1030 con 0x7f0958157400 2023-10-10T18:45:22.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f0955d9b700 1 --2- 172.21.15.175:0/235360103 >> v2:172.21.15.191:3300/0 conn(0x7f09580c4640 0x7f09581577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:22.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.644+0000 7f095659c700 1 --2- 172.21.15.175:0/235360103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157400 0x7f095806bda0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f09581516e0 tx=0x7f0948009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0fae0cc29a6c33 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:22.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095559a700 1 -- 172.21.15.175:0/235360103 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094800e070 con 0x7f0958157400 2023-10-10T18:45:22.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095559a700 1 -- 172.21.15.175:0/235360103 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094800ba50 con 0x7f0958157400 2023-10-10T18:45:22.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095559a700 1 -- 172.21.15.175:0/235360103 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09480092e0 con 0x7f0958157400 2023-10-10T18:45:22.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095759e700 1 -- 172.21.15.175:0/235360103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157400 msgr2=0x7f095806bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095759e700 1 --2- 172.21.15.175:0/235360103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157400 0x7f095806bda0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f09581516e0 tx=0x7f0948009840 comp rx=0 tx=0).stop 2023-10-10T18:45:22.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095759e700 1 -- 172.21.15.175:0/235360103 shutdown_connections 2023-10-10T18:45:22.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095759e700 1 --2- 172.21.15.175:0/235360103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157400 0x7f095806bda0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095759e700 1 --2- 172.21.15.175:0/235360103 >> v2:172.21.15.191:3300/0 conn(0x7f09580c4640 0x7f09581577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095759e700 1 -- 172.21.15.175:0/235360103 >> 172.21.15.175:0/235360103 conn(0x7f0958063cd0 msgr2=0x7f095806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:22.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095759e700 1 -- 172.21.15.175:0/235360103 shutdown_connections 2023-10-10T18:45:22.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.645+0000 7f095759e700 1 -- 172.21.15.175:0/235360103 wait complete. 2023-10-10T18:45:22.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.646+0000 7f095759e700 1 Processor -- start 2023-10-10T18:45:22.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.646+0000 7f095759e700 1 -- start start 2023-10-10T18:45:22.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f095759e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09580c4640 0x7f09580c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f095759e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0958157400 0x7f09580c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f095759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157d10 0x7f09580c2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f095759e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580c7680 con 0x7f0958157d10 2023-10-10T18:45:22.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f095759e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580c1d00 con 0x7f0958157400 2023-10-10T18:45:22.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f095759e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09580c6de0 con 0x7f09580c4640 2023-10-10T18:45:22.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157d10 0x7f09580c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157d10 0x7f09580c2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52680/0 (socket says 172.21.15.175:52680) 2023-10-10T18:45:22.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 -- 172.21.15.175:0/531267623 learned_addr learned my addr 172.21.15.175:0/531267623 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:22.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0955d9b700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0958157400 0x7f09580c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f095659c700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09580c4640 0x7f09580c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 -- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09580c4640 msgr2=0x7f09580c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09580c4640 0x7f09580c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 -- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0958157400 msgr2=0x7f09580c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0958157400 0x7f09580c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 -- 172.21.15.175:0/531267623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0948009ae0 con 0x7f0958157d10 2023-10-10T18:45:22.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f095659c700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09580c4640 0x7f09580c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:22.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.647+0000 7f0956d9d700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157d10 0x7f09580c2190 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f095000c540 tx=0x7f0950007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:22.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.648+0000 7f09467fc700 1 -- 172.21.15.175:0/531267623 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950011590 con 0x7f0958157d10 2023-10-10T18:45:22.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.648+0000 7f09467fc700 1 -- 172.21.15.175:0/531267623 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09500116f0 con 0x7f0958157d10 2023-10-10T18:45:22.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.648+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09580c70c0 con 0x7f0958157d10 2023-10-10T18:45:22.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.648+0000 7f09467fc700 1 -- 172.21.15.175:0/531267623 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950011ab0 con 0x7f0958157d10 2023-10-10T18:45:22.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.648+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095814c7d0 con 0x7f0958157d10 2023-10-10T18:45:22.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.649+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09580c1e70 con 0x7f0958157d10 2023-10-10T18:45:22.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.649+0000 7f09467fc700 1 -- 172.21.15.175:0/531267623 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f095001b3b0 con 0x7f0958157d10 2023-10-10T18:45:22.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.649+0000 7f09467fc700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0938066f60 0x7f0938069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:22.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.649+0000 7f09467fc700 1 -- 172.21.15.175:0/531267623 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f09500839e0 con 0x7f0958157d10 2023-10-10T18:45:22.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.649+0000 7f095659c700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0938066f60 0x7f0938069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:22.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.650+0000 7f095659c700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0938066f60 0x7f0938069410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0948009880 tx=0x7f0948018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:22.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.653+0000 7f09467fc700 1 -- 172.21.15.175:0/531267623 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0950050c50 con 0x7f0958157d10 2023-10-10T18:45:22.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.879+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0958067570 con 0x7f0958157d10 2023-10-10T18:45:22.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.879+0000 7f09467fc700 1 -- 172.21.15.175:0/531267623 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f0950017020 con 0x7f0958157d10 2023-10-10T18:45:22.881 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:22.881 INFO:teuthology.orchestra.run.smithi175.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-10T18:45:22.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0938066f60 msgr2=0x7f0938069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0938066f60 0x7f0938069410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0948009880 tx=0x7f0948018000 comp rx=0 tx=0).stop 2023-10-10T18:45:22.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157d10 msgr2=0x7f09580c2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:22.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157d10 0x7f09580c2190 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f095000c540 tx=0x7f0950007f30 comp rx=0 tx=0).stop 2023-10-10T18:45:22.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 shutdown_connections 2023-10-10T18:45:22.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0938066f60 0x7f0938069410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0958157d10 0x7f09580c2190 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09580c4640 0x7f09580c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 --2- 172.21.15.175:0/531267623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0958157400 0x7f09580c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:22.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 >> 172.21.15.175:0/531267623 conn(0x7f0958063cd0 msgr2=0x7f095814d360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:22.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.881+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 shutdown_connections 2023-10-10T18:45:22.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:22.882+0000 7f095759e700 1 -- 172.21.15.175:0/531267623 wait complete. 2023-10-10T18:45:25.896 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-10T18:45:26.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.022+0000 7f2d935b0700 1 Processor -- start 2023-10-10T18:45:26.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.022+0000 7f2d935b0700 1 -- start start 2023-10-10T18:45:26.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d935b0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b1fc0 0x7f2d8c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:26.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d935b0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2d8c0aff60 0x7f2d8c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:26.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d935b0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c1574b0 con 0x7f2d8c0b2390 2023-10-10T18:45:26.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d935b0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8c150770 con 0x7f2d8c0aff60 2023-10-10T18:45:26.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d935b0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0afad0 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d91b4d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2d8c0aff60 0x7f2d8c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:26.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d91b4d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2d8c0aff60 0x7f2d8c0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44018/0 (socket says 172.21.15.175:44018) 2023-10-10T18:45:26.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d91b4d700 1 -- 172.21.15.175:0/3558235501 learned_addr learned my addr 172.21.15.175:0/3558235501 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:26.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d9134c700 1 --2- 172.21.15.175:0/3558235501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b1fc0 0x7f2d8c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:26.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d91b4d700 1 -- 172.21.15.175:0/3558235501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b1fc0 msgr2=0x7f2d8c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:26.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d91b4d700 1 --2- 172.21.15.175:0/3558235501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b1fc0 0x7f2d8c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d91b4d700 1 -- 172.21.15.175:0/3558235501 >> v1:172.21.15.191:6789/0 conn(0x7f2d8c0b2390 legacy=0x7f2d8c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:26.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d91b4d700 1 -- 172.21.15.175:0/3558235501 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0af4c0 con 0x7f2d8c0aff60 2023-10-10T18:45:26.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d9134c700 1 --2- 172.21.15.175:0/3558235501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b1fc0 0x7f2d8c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:26.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d91b4d700 1 --2- 172.21.15.175:0/3558235501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0aff60 0x7f2d8c0b03e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009f50 tx=0x7f2d84009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e289a1a469835fe9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:26.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d83fff700 1 -- 172.21.15.175:0/3558235501 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8400e070 con 0x7f2d8c0aff60 2023-10-10T18:45:26.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d83fff700 1 -- 172.21.15.175:0/3558235501 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8400bb00 con 0x7f2d8c0aff60 2023-10-10T18:45:26.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.023+0000 7f2d83fff700 1 -- 172.21.15.175:0/3558235501 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d840092e0 con 0x7f2d8c0aff60 2023-10-10T18:45:26.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.024+0000 7f2d935b0700 1 -- 172.21.15.175:0/3558235501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0aff60 msgr2=0x7f2d8c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:26.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.024+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3558235501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0aff60 0x7f2d8c0b03e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009f50 tx=0x7f2d84009840 comp rx=0 tx=0).stop 2023-10-10T18:45:26.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.024+0000 7f2d935b0700 1 -- 172.21.15.175:0/3558235501 shutdown_connections 2023-10-10T18:45:26.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.024+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3558235501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0aff60 0x7f2d8c0b03e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.024+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3558235501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b1fc0 0x7f2d8c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.024+0000 7f2d935b0700 1 -- 172.21.15.175:0/3558235501 >> 172.21.15.175:0/3558235501 conn(0x7f2d8c063230 msgr2=0x7f2d8c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:26.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.024+0000 7f2d935b0700 1 -- 172.21.15.175:0/3558235501 shutdown_connections 2023-10-10T18:45:26.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.024+0000 7f2d935b0700 1 -- 172.21.15.175:0/3558235501 wait complete. 2023-10-10T18:45:26.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.025+0000 7f2d935b0700 1 Processor -- start 2023-10-10T18:45:26.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.025+0000 7f2d935b0700 1 -- start start 2023-10-10T18:45:26.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d935b0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d8c0aff60 0x7f2d8c0c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:26.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d935b0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0b1fc0 0x7f2d8c0c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:26.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d935b0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b2390 0x7f2d8c0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:26.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d935b0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0c2a10 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d935b0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0c02c0 con 0x7f2d8c0aff60 2023-10-10T18:45:26.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d935b0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0c1db0 con 0x7f2d8c0b2390 2023-10-10T18:45:26.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d90b4b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0b1fc0 0x7f2d8c0c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:26.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d9134c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d8c0aff60 0x7f2d8c0c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:26.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d90b4b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0b1fc0 0x7f2d8c0c4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44030/0 (socket says 172.21.15.175:44030) 2023-10-10T18:45:26.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d90b4b700 1 -- 172.21.15.175:0/3581010574 learned_addr learned my addr 172.21.15.175:0/3581010574 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:26.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d91b4d700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b2390 0x7f2d8c0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:26.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d90b4b700 1 -- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b2390 msgr2=0x7f2d8c0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:26.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d90b4b700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b2390 0x7f2d8c0c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d90b4b700 1 -- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d8c0aff60 msgr2=0x7f2d8c0c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:26.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d90b4b700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d8c0aff60 0x7f2d8c0c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.026+0000 7f2d90b4b700 1 -- 172.21.15.175:0/3581010574 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d84009ae0 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.027+0000 7f2d90b4b700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0b1fc0 0x7f2d8c0c4b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00c4e0 tx=0x7f2d7c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:26.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.027+0000 7f2d817fa700 1 -- 172.21.15.175:0/3581010574 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c011590 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.027+0000 7f2d817fa700 1 -- 172.21.15.175:0/3581010574 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c011710 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.027+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c0c04d0 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.027+0000 7f2d817fa700 1 -- 172.21.15.175:0/3581010574 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c011ad0 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.027+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c0c2350 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.028+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c0c2b50 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.028+0000 7f2d817fa700 1 -- 172.21.15.175:0/3581010574 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2d7c01b3b0 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.029+0000 7f2d817fa700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d70066f30 0x7f2d700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:26.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.029+0000 7f2d817fa700 1 -- 172.21.15.175:0/3581010574 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2d7c083930 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.029+0000 7f2d9134c700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d70066f30 0x7f2d700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:26.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.029+0000 7f2d9134c700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d70066f30 0x7f2d700693e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d88006f00 tx=0x7f2d88008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:26.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.033+0000 7f2d817fa700 1 -- 172.21.15.175:0/3581010574 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d8c0c2b50 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.254+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2d8c0c2b50 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.255+0000 7f2d817fa700 1 -- 172.21.15.175:0/3581010574 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f2d8c0c2b50 con 0x7f2d8c0b1fc0 2023-10-10T18:45:26.255 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:26.256 INFO:teuthology.orchestra.run.smithi175.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-10T18:45:26.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d70066f30 msgr2=0x7f2d700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:26.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d70066f30 0x7f2d700693e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d88006f00 tx=0x7f2d88008040 comp rx=0 tx=0).stop 2023-10-10T18:45:26.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0b1fc0 msgr2=0x7f2d8c0c4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:26.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0b1fc0 0x7f2d8c0c4b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00c4e0 tx=0x7f2d7c00c7f0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 shutdown_connections 2023-10-10T18:45:26.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d70066f30 0x7f2d700693e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d8c0b2390 0x7f2d8c0c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d8c0aff60 0x7f2d8c0c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 --2- 172.21.15.175:0/3581010574 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d8c0b1fc0 0x7f2d8c0c4b20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:26.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 >> 172.21.15.175:0/3581010574 conn(0x7f2d8c063230 msgr2=0x7f2d8c063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:26.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 shutdown_connections 2023-10-10T18:45:26.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:26.257+0000 7f2d935b0700 1 -- 172.21.15.175:0/3581010574 wait complete. 2023-10-10T18:45:26.508 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T18:45:26.508+0000 7fc568727700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-10T18:45:26.509 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T18:45:26.508+0000 7fc568727700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T18:45:29.270 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-10T18:45:29.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.396+0000 7fe67152c700 1 Processor -- start 2023-10-10T18:45:29.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.396+0000 7fe67152c700 1 -- start start 2023-10-10T18:45:29.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe67152c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe66c06d6e0 0x7fe66c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:29.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe67152c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe66c150480 0x7fe66c06e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:29.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe67152c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c0677f0 con 0x7fe66c1500b0 2023-10-10T18:45:29.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe67152c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66c159590 con 0x7fe66c06d6e0 2023-10-10T18:45:29.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe67152c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66c0c4250 con 0x7fe66c150480 2023-10-10T18:45:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe66a7fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe66c06d6e0 0x7fe66c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe66a7fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe66c06d6e0 0x7fe66c06dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44032/0 (socket says 172.21.15.175:44032) 2023-10-10T18:45:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe66a7fc700 1 -- 172.21.15.175:0/231842547 learned_addr learned my addr 172.21.15.175:0/231842547 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe66affd700 1 --1- 172.21.15.175:0/231842547 >> v1:172.21.15.175:6789/0 conn(0x7fe66c1500b0 0x7fe66c0c46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46492/0 (socket says 172.21.15.175:46492) 2023-10-10T18:45:29.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.397+0000 7fe66b7fe700 1 --2- 172.21.15.175:0/231842547 >> v2:172.21.15.191:3300/0 conn(0x7fe66c150480 0x7fe66c06e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:29.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe66a7fc700 1 -- 172.21.15.175:0/231842547 >> v2:172.21.15.191:3300/0 conn(0x7fe66c150480 msgr2=0x7fe66c06e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:29.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe66a7fc700 1 --2- 172.21.15.175:0/231842547 >> v2:172.21.15.191:3300/0 conn(0x7fe66c150480 0x7fe66c06e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe66a7fc700 1 -- 172.21.15.175:0/231842547 >> v1:172.21.15.175:6789/0 conn(0x7fe66c1500b0 legacy=0x7fe66c0c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:29.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe66a7fc700 1 -- 172.21.15.175:0/231842547 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c063760 con 0x7fe66c06d6e0 2023-10-10T18:45:29.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe669ffb700 1 -- 172.21.15.175:0/231842547 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376756064 0 0) 0x7fe66c0677f0 con 0x7fe66c1500b0 2023-10-10T18:45:29.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe66a7fc700 1 --2- 172.21.15.175:0/231842547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c06d6e0 0x7fe66c06dcd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe66c069460 tx=0x7fe660009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03de63efeb93b1f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:29.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe669ffb700 1 -- 172.21.15.175:0/231842547 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660009e70 con 0x7fe66c06d6e0 2023-10-10T18:45:29.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe669ffb700 1 -- 172.21.15.175:0/231842547 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66000b8f0 con 0x7fe66c06d6e0 2023-10-10T18:45:29.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe669ffb700 1 -- 172.21.15.175:0/231842547 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66000bc90 con 0x7fe66c06d6e0 2023-10-10T18:45:29.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe67152c700 1 -- 172.21.15.175:0/231842547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c06d6e0 msgr2=0x7fe66c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:29.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.398+0000 7fe67152c700 1 --2- 172.21.15.175:0/231842547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c06d6e0 0x7fe66c06dcd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe66c069460 tx=0x7fe660009250 comp rx=0 tx=0).stop 2023-10-10T18:45:29.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.399+0000 7fe67152c700 1 -- 172.21.15.175:0/231842547 shutdown_connections 2023-10-10T18:45:29.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.399+0000 7fe67152c700 1 --2- 172.21.15.175:0/231842547 >> v2:172.21.15.191:3300/0 conn(0x7fe66c150480 0x7fe66c06e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.399+0000 7fe67152c700 1 --2- 172.21.15.175:0/231842547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c06d6e0 0x7fe66c06dcd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.399+0000 7fe67152c700 1 -- 172.21.15.175:0/231842547 >> 172.21.15.175:0/231842547 conn(0x7fe66c0c5650 msgr2=0x7fe66c0c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:29.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.399+0000 7fe67152c700 1 -- 172.21.15.175:0/231842547 shutdown_connections 2023-10-10T18:45:29.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.399+0000 7fe67152c700 1 -- 172.21.15.175:0/231842547 wait complete. 2023-10-10T18:45:29.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.400+0000 7fe67152c700 1 Processor -- start 2023-10-10T18:45:29.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.400+0000 7fe67152c700 1 -- start start 2023-10-10T18:45:29.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.400+0000 7fe67152c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe66c06d6e0 0x7fe66c1688a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:29.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.400+0000 7fe67152c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe66c1500b0 0x7fe66c0687e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:29.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.400+0000 7fe67152c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c150480 0x7fe66c14dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:29.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.400+0000 7fe67152c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c1558b0 con 0x7fe66c150480 2023-10-10T18:45:29.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.400+0000 7fe67152c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c168de0 con 0x7fe66c1500b0 2023-10-10T18:45:29.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.400+0000 7fe67152c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c168410 con 0x7fe66c06d6e0 2023-10-10T18:45:29.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c150480 0x7fe66c14dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:29.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c150480 0x7fe66c14dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44042/0 (socket says 172.21.15.175:44042) 2023-10-10T18:45:29.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 -- 172.21.15.175:0/1864872469 learned_addr learned my addr 172.21.15.175:0/1864872469 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:29.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 -- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe66c06d6e0 msgr2=0x7fe66c1688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:29.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe66c06d6e0 0x7fe66c1688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 -- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe66c1500b0 msgr2=0x7fe66c0687e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:29.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66a7fc700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe66c1500b0 0x7fe66c0687e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:29.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe66c1500b0 0x7fe66c0687e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 -- 172.21.15.175:0/1864872469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6600094f0 con 0x7fe66c150480 2023-10-10T18:45:29.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe66b7fe700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c150480 0x7fe66c14dd00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe66400ea80 tx=0x7fe6640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:29.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe652ffd700 1 -- 172.21.15.175:0/1864872469 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66400c8c0 con 0x7fe66c150480 2023-10-10T18:45:29.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe652ffd700 1 -- 172.21.15.175:0/1864872469 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66400ca20 con 0x7fe66c150480 2023-10-10T18:45:29.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66c0c7940 con 0x7fe66c150480 2023-10-10T18:45:29.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe652ffd700 1 -- 172.21.15.175:0/1864872469 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66400cde0 con 0x7fe66c150480 2023-10-10T18:45:29.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.401+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66c0c0fd0 con 0x7fe66c150480 2023-10-10T18:45:29.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.402+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66c067570 con 0x7fe66c150480 2023-10-10T18:45:29.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.402+0000 7fe652ffd700 1 -- 172.21.15.175:0/1864872469 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe664013680 con 0x7fe66c150480 2023-10-10T18:45:29.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.403+0000 7fe652ffd700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe658066f30 0x7fe6580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:29.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.403+0000 7fe652ffd700 1 -- 172.21.15.175:0/1864872469 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe664084800 con 0x7fe66c150480 2023-10-10T18:45:29.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.403+0000 7fe66affd700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe658066f30 0x7fe6580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:29.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.404+0000 7fe66affd700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe658066f30 0x7fe6580693e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe65c002150 tx=0x7fe65c0016b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:29.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.407+0000 7fe652ffd700 1 -- 172.21.15.175:0/1864872469 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe664051a70 con 0x7fe66c150480 2023-10-10T18:45:29.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.631+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe66c168550 con 0x7fe66c150480 2023-10-10T18:45:29.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.631+0000 7fe652ffd700 1 -- 172.21.15.175:0/1864872469 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7fe664055090 con 0x7fe66c150480 2023-10-10T18:45:29.632 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:29.632 INFO:teuthology.orchestra.run.smithi175.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-10T18:45:29.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe658066f30 msgr2=0x7fe6580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:29.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe658066f30 0x7fe6580693e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe65c002150 tx=0x7fe65c0016b0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c150480 msgr2=0x7fe66c14dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:29.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c150480 0x7fe66c14dd00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe66400ea80 tx=0x7fe6640091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 shutdown_connections 2023-10-10T18:45:29.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe658066f30 0x7fe6580693e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe66c150480 0x7fe66c14dd00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe66c06d6e0 0x7fe66c1688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 --2- 172.21.15.175:0/1864872469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe66c1500b0 0x7fe66c0687e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:29.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 >> 172.21.15.175:0/1864872469 conn(0x7fe66c0c5650 msgr2=0x7fe66c15bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:29.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 shutdown_connections 2023-10-10T18:45:29.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:29.633+0000 7fe67152c700 1 -- 172.21.15.175:0/1864872469 wait complete. 2023-10-10T18:45:31.509 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T18:45:31.508+0000 7fc568727700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-10T18:45:32.646 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-10T18:45:32.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.769+0000 7f0eb3829700 1 Processor -- start 2023-10-10T18:45:32.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.770+0000 7f0eb3829700 1 -- start start 2023-10-10T18:45:32.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.770+0000 7f0eb3829700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b1fc0 0x7f0eac155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:32.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.770+0000 7f0eb3829700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0eac0aff60 0x7f0eac0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:32.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.770+0000 7f0eb3829700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac1574b0 con 0x7f0eac0b2390 2023-10-10T18:45:32.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.770+0000 7f0eb3829700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eac150770 con 0x7f0eac0aff60 2023-10-10T18:45:32.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.770+0000 7f0eb3829700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0afad0 con 0x7f0eac0b1fc0 2023-10-10T18:45:32.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.770+0000 7f0eb1dc6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0eac0aff60 0x7f0eac0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:32.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0eb1dc6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0eac0aff60 0x7f0eac0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44052/0 (socket says 172.21.15.175:44052) 2023-10-10T18:45:32.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0eb1dc6700 1 -- 172.21.15.175:0/4238527407 learned_addr learned my addr 172.21.15.175:0/4238527407 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:32.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0eb15c5700 1 --2- 172.21.15.175:0/4238527407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b1fc0 0x7f0eac155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:32.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0eb1dc6700 1 -- 172.21.15.175:0/4238527407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b1fc0 msgr2=0x7f0eac155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:32.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0eb1dc6700 1 --2- 172.21.15.175:0/4238527407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b1fc0 0x7f0eac155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:32.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0eb1dc6700 1 -- 172.21.15.175:0/4238527407 >> v1:172.21.15.175:6789/0 conn(0x7f0eac0b2390 legacy=0x7f0eac068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:32.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0eb1dc6700 1 -- 172.21.15.175:0/4238527407 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0af4c0 con 0x7f0eac0aff60 2023-10-10T18:45:32.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0eb1dc6700 1 --2- 172.21.15.175:0/4238527407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0b03e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40099e0 tx=0x7f0ea4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd68dc558912d467 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:32.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0ea3fff700 1 -- 172.21.15.175:0/4238527407 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400b8b0 con 0x7f0eac0aff60 2023-10-10T18:45:32.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0ea3fff700 1 -- 172.21.15.175:0/4238527407 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea400ba10 con 0x7f0eac0aff60 2023-10-10T18:45:32.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.771+0000 7f0ea3fff700 1 -- 172.21.15.175:0/4238527407 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400bdb0 con 0x7f0eac0aff60 2023-10-10T18:45:32.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.772+0000 7f0eb3829700 1 -- 172.21.15.175:0/4238527407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 msgr2=0x7f0eac0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:32.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.772+0000 7f0eb3829700 1 --2- 172.21.15.175:0/4238527407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0b03e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40099e0 tx=0x7f0ea4009250 comp rx=0 tx=0).stop 2023-10-10T18:45:32.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.772+0000 7f0eb3829700 1 -- 172.21.15.175:0/4238527407 shutdown_connections 2023-10-10T18:45:32.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.772+0000 7f0eb3829700 1 --2- 172.21.15.175:0/4238527407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0b03e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:32.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.772+0000 7f0eb3829700 1 --2- 172.21.15.175:0/4238527407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b1fc0 0x7f0eac155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:32.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.772+0000 7f0eb3829700 1 -- 172.21.15.175:0/4238527407 >> 172.21.15.175:0/4238527407 conn(0x7f0eac063230 msgr2=0x7f0eac063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:32.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.772+0000 7f0eb3829700 1 -- 172.21.15.175:0/4238527407 shutdown_connections 2023-10-10T18:45:32.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.772+0000 7f0eb3829700 1 -- 172.21.15.175:0/4238527407 wait complete. 2023-10-10T18:45:32.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.773+0000 7f0eb3829700 1 Processor -- start 2023-10-10T18:45:32.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.773+0000 7f0eb3829700 1 -- start start 2023-10-10T18:45:32.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb3829700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:32.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb3829700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac0b1fc0 0x7f0eac14def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:32.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb3829700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b2390 0x7f0eac168eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:32.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb3829700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0c6a50 con 0x7f0eac0aff60 2023-10-10T18:45:32.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb3829700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac169ba0 con 0x7f0eac0b1fc0 2023-10-10T18:45:32.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb3829700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0c3b20 con 0x7f0eac0b2390 2023-10-10T18:45:32.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:32.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44068/0 (socket says 172.21.15.175:44068) 2023-10-10T18:45:32.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 -- 172.21.15.175:0/2000273366 learned_addr learned my addr 172.21.15.175:0/2000273366 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:32.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb1dc6700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b2390 0x7f0eac168eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:32.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb0dc4700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac0b1fc0 0x7f0eac14def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:32.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 -- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b2390 msgr2=0x7f0eac168eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:32.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b2390 0x7f0eac168eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:32.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 -- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac0b1fc0 msgr2=0x7f0eac14def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:32.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac0b1fc0 0x7f0eac14def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:32.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 -- 172.21.15.175:0/2000273366 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea40094f0 con 0x7f0eac0aff60 2023-10-10T18:45:32.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb15c5700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0cb720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009ef0 tx=0x7f0ea800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:32.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0ea17fa700 1 -- 172.21.15.175:0/2000273366 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea800cc50 con 0x7f0eac0aff60 2023-10-10T18:45:32.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0ea17fa700 1 -- 172.21.15.175:0/2000273366 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea800f040 con 0x7f0eac0aff60 2023-10-10T18:45:32.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac0c4330 con 0x7f0eac0aff60 2023-10-10T18:45:32.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0ea17fa700 1 -- 172.21.15.175:0/2000273366 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8014480 con 0x7f0eac0aff60 2023-10-10T18:45:32.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.774+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac153520 con 0x7f0eac0aff60 2023-10-10T18:45:32.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.775+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac0c5ab0 con 0x7f0eac0aff60 2023-10-10T18:45:32.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.776+0000 7f0ea17fa700 1 -- 172.21.15.175:0/2000273366 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0ea80147f0 con 0x7f0eac0aff60 2023-10-10T18:45:32.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.776+0000 7f0ea17fa700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0e90066f30 0x7f0e900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:32.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.776+0000 7f0ea17fa700 1 -- 172.21.15.175:0/2000273366 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0ea8085010 con 0x7f0eac0aff60 2023-10-10T18:45:32.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.776+0000 7f0eb0dc4700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0e90066f30 0x7f0e900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:32.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.777+0000 7f0eb0dc4700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0e90066f30 0x7f0e900693e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0eac153350 tx=0x7f0e9c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:32.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:32.780+0000 7f0ea17fa700 1 -- 172.21.15.175:0/2000273366 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0ea8052280 con 0x7f0eac0aff60 2023-10-10T18:45:33.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.001+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0eac067570 con 0x7f0eac0aff60 2023-10-10T18:45:33.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.002+0000 7f0ea17fa700 1 -- 172.21.15.175:0/2000273366 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f0ea80558a0 con 0x7f0eac0aff60 2023-10-10T18:45:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:33.002 INFO:teuthology.orchestra.run.smithi175.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-10T18:45:33.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0e90066f30 msgr2=0x7f0e900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:33.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0e90066f30 0x7f0e900693e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0eac153350 tx=0x7f0e9c009040 comp rx=0 tx=0).stop 2023-10-10T18:45:33.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 msgr2=0x7f0eac0cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:33.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0cb720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009ef0 tx=0x7f0ea800b040 comp rx=0 tx=0).stop 2023-10-10T18:45:33.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 shutdown_connections 2023-10-10T18:45:33.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0e90066f30 0x7f0e900693e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:33.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0eac0b2390 0x7f0eac168eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:33.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0eac0aff60 0x7f0eac0cb720 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:33.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 --2- 172.21.15.175:0/2000273366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac0b1fc0 0x7f0eac14def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:33.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 >> 172.21.15.175:0/2000273366 conn(0x7f0eac063230 msgr2=0x7f0eac14fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:33.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 shutdown_connections 2023-10-10T18:45:33.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:33.004+0000 7f0eb3829700 1 -- 172.21.15.175:0/2000273366 wait complete. 2023-10-10T18:45:36.017 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-10T18:45:36.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.140+0000 7f7fb7b82700 1 Processor -- start 2023-10-10T18:45:36.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.140+0000 7f7fb7b82700 1 -- start start 2023-10-10T18:45:36.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb7b82700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0155fb0 0x7f7fb0159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb7b82700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb01574b0 con 0x7f7fb00b1fc0 2023-10-10T18:45:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb7b82700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb00afad0 con 0x7f7fb00b2390 2023-10-10T18:45:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb7b82700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0150770 con 0x7f7fb0155fb0 2023-10-10T18:45:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb591e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7fb00b1fc0 0x7f7fb0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51284/0 (socket says 172.21.15.175:51284) 2023-10-10T18:45:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb591e700 1 -- 172.21.15.175:0/4124564707 learned_addr learned my addr 172.21.15.175:0/4124564707 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb611f700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0155fb0 0x7f7fb0159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138895419 0 0) 0x7f7fb01574b0 con 0x7f7fb00b1fc0 2023-10-10T18:45:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9c003570 con 0x7f7fb00b1fc0 2023-10-10T18:45:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784019129 0 0) 0x7f7f9c003570 con 0x7f7fb00b1fc0 2023-10-10T18:45:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0155fb0 msgr2=0x7f7fb0159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0155fb0 0x7f7fb0159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 >> v1:172.21.15.191:6789/0 conn(0x7f7fb00b2390 legacy=0x7f7fb00aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb00af4c0 con 0x7f7fb00b1fc0 2023-10-10T18:45:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f7fac001470 con 0x7f7fb00b1fc0 2023-10-10T18:45:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 >> v1:172.21.15.175:6789/0 conn(0x7f7fb00b1fc0 legacy=0x7f7fb0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f9c003a90 0x7f7f9c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7f9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0155fb0 0x7f7f9c0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0150770 con 0x7f7f9c004810 2023-10-10T18:45:36.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c00bb60 con 0x7f7f9c003a90 2023-10-10T18:45:36.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9c00bd20 con 0x7f7fb0155fb0 2023-10-10T18:45:36.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb511d700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7f9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.141+0000 7f7fb591e700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f9c003a90 0x7f7f9c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb611f700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0155fb0 0x7f7f9c0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb511d700 1 -- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0155fb0 msgr2=0x7f7f9c0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb511d700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0155fb0 0x7f7f9c0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb511d700 1 -- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f9c003a90 msgr2=0x7f7f9c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb511d700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f9c003a90 0x7f7f9c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb511d700 1 -- 172.21.15.175:0/4124564707 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c003570 con 0x7f7f9c004810 2023-10-10T18:45:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb511d700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7f9c004c80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7fa00018b0 tx=0x7f7fa0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8eb989703322b85 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0012420 con 0x7f7f9c004810 2023-10-10T18:45:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa0012580 con 0x7f7f9c004810 2023-10-10T18:45:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb491c700 1 -- 172.21.15.175:0/4124564707 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0012920 con 0x7f7f9c004810 2023-10-10T18:45:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb7b82700 1 -- 172.21.15.175:0/4124564707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 msgr2=0x7f7f9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.142+0000 7f7fb7b82700 1 --2- 172.21.15.175:0/4124564707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7f9c004c80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7fa00018b0 tx=0x7f7fa0007d30 comp rx=0 tx=0).stop 2023-10-10T18:45:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.143+0000 7f7fb611f700 1 -- 172.21.15.175:0/4124564707 reap_dead start 2023-10-10T18:45:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.143+0000 7f7fb7b82700 1 -- 172.21.15.175:0/4124564707 shutdown_connections 2023-10-10T18:45:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.143+0000 7f7fb7b82700 1 -- 172.21.15.175:0/4124564707 >> 172.21.15.175:0/4124564707 conn(0x7f7fb0063230 msgr2=0x7f7fb0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.143+0000 7f7fb7b82700 1 -- 172.21.15.175:0/4124564707 shutdown_connections 2023-10-10T18:45:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.143+0000 7f7fb7b82700 1 -- 172.21.15.175:0/4124564707 wait complete. 2023-10-10T18:45:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb7b82700 1 Processor -- start 2023-10-10T18:45:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb7b82700 1 -- start start 2023-10-10T18:45:36.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb7b82700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7fb00c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb7b82700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0157180 0x7f7fb00c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb7b82700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fb00c58a0 0x7f7fb0168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb7b82700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb00c4ed0 con 0x7f7f9c004810 2023-10-10T18:45:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb7b82700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb00c4bd0 con 0x7f7fb00c58a0 2023-10-10T18:45:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb7b82700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb00c5cf0 con 0x7f7fb0157180 2023-10-10T18:45:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb591e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7fb00c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb591e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7fb00c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51542/0 (socket says 172.21.15.175:51542) 2023-10-10T18:45:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.144+0000 7f7fb591e700 1 -- 172.21.15.175:0/2271523517 learned_addr learned my addr 172.21.15.175:0/2271523517 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb511d700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0157180 0x7f7fb00c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb611f700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fb00c58a0 0x7f7fb0168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb591e700 1 -- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0157180 msgr2=0x7f7fb00c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb591e700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0157180 0x7f7fb00c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb591e700 1 -- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fb00c58a0 msgr2=0x7f7fb0168ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb591e700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fb00c58a0 0x7f7fb0168ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb591e700 1 -- 172.21.15.175:0/2271523517 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb00cc220 con 0x7f7f9c004810 2023-10-10T18:45:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb611f700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fb00c58a0 0x7f7fb0168ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb591e700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7fb00c8240 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7fac001200 tx=0x7f7fb00712f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fa5ffb700 1 -- 172.21.15.175:0/2271523517 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac018030 con 0x7f7f9c004810 2023-10-10T18:45:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fa5ffb700 1 -- 172.21.15.175:0/2271523517 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac0077d0 con 0x7f7f9c004810 2023-10-10T18:45:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb00cd200 con 0x7f7f9c004810 2023-10-10T18:45:36.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fa5ffb700 1 -- 172.21.15.175:0/2271523517 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00fac0 con 0x7f7f9c004810 2023-10-10T18:45:36.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.145+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb00cbf30 con 0x7f7f9c004810 2023-10-10T18:45:36.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.146+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb00cd6a0 con 0x7f7f9c004810 2023-10-10T18:45:36.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.147+0000 7f7fa5ffb700 1 -- 172.21.15.175:0/2271523517 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7fac01b070 con 0x7f7f9c004810 2023-10-10T18:45:36.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.147+0000 7f7fa5ffb700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f94066f30 0x7f7f940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.147+0000 7f7fa5ffb700 1 -- 172.21.15.175:0/2271523517 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7fac019070 con 0x7f7f9c004810 2023-10-10T18:45:36.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.147+0000 7f7fb511d700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f94066f30 0x7f7f940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.148+0000 7f7fb511d700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f94066f30 0x7f7f940693e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0153ba0 tx=0x7f7fa0015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.151+0000 7f7fa5ffb700 1 -- 172.21.15.175:0/2271523517 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7fac051750 con 0x7f7f9c004810 2023-10-10T18:45:36.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.374+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7fb0067570 con 0x7f7f9c004810 2023-10-10T18:45:36.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.374+0000 7f7fa5ffb700 1 -- 172.21.15.175:0/2271523517 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7fac054d70 con 0x7f7f9c004810 2023-10-10T18:45:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:36.375 INFO:teuthology.orchestra.run.smithi175.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-10T18:45:36.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f94066f30 msgr2=0x7f7f940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f94066f30 0x7f7f940693e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0153ba0 tx=0x7f7fa0015000 comp rx=0 tx=0).stop 2023-10-10T18:45:36.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 msgr2=0x7f7fb00c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7fb00c8240 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7fac001200 tx=0x7f7fb00712f0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 shutdown_connections 2023-10-10T18:45:36.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f94066f30 0x7f7f940693e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fb00c58a0 0x7f7fb0168ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f9c004810 0x7f7fb00c8240 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 --2- 172.21.15.175:0/2271523517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fb0157180 0x7f7fb00c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 >> 172.21.15.175:0/2271523517 conn(0x7f7fb0063230 msgr2=0x7f7fb01508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:36.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 shutdown_connections 2023-10-10T18:45:36.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.376+0000 7f7fb7b82700 1 -- 172.21.15.175:0/2271523517 wait complete. 2023-10-10T18:45:36.388 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-10T18:45:36.388 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:36.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.503+0000 7fbe70061700 1 Processor -- start 2023-10-10T18:45:36.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.504+0000 7fbe70061700 1 -- start start 2023-10-10T18:45:36.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.504+0000 7fbe70061700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 0x7fbe681547c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.504+0000 7fbe70061700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe680c5280 con 0x7fbe6814e050 2023-10-10T18:45:36.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.504+0000 7fbe70061700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe680c5560 con 0x7fbe680c2a70 2023-10-10T18:45:36.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.504+0000 7fbe70061700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680c4b60 con 0x7fbe680682e0 2023-10-10T18:45:36.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.504+0000 7fbe6ddfd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbe6814e050 0x7fbe68067bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51288/0 (socket says 172.21.15.175:51288) 2023-10-10T18:45:36.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.504+0000 7fbe6ddfd700 1 -- 172.21.15.175:0/945631379 learned_addr learned my addr 172.21.15.175:0/945631379 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:36.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6d5fc700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 0x7fbe681547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008245094 0 0) 0x7fbe680c5280 con 0x7fbe6814e050 2023-10-10T18:45:36.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe50003570 con 0x7fbe6814e050 2023-10-10T18:45:36.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2577885041 0 0) 0x7fbe50003570 con 0x7fbe6814e050 2023-10-10T18:45:36.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 msgr2=0x7fbe681547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 0x7fbe681547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 >> v1:172.21.15.191:6789/0 conn(0x7fbe680c2a70 legacy=0x7fbe680c2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe680c5840 con 0x7fbe6814e050 2023-10-10T18:45:36.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fbe580017f0 con 0x7fbe6814e050 2023-10-10T18:45:36.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 >> v1:172.21.15.175:6789/0 conn(0x7fbe6814e050 legacy=0x7fbe68067bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe50003a90 0x7fbe50003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe50004810 0x7fbe50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 0x7fbe500051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680c4b60 con 0x7fbe50003a90 2023-10-10T18:45:36.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5000bb60 con 0x7fbe50004810 2023-10-10T18:45:36.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5000bd20 con 0x7fbe680682e0 2023-10-10T18:45:36.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6ddfd700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe50003a90 0x7fbe50003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6d5fc700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe50004810 0x7fbe50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6e5fe700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 0x7fbe500051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6ddfd700 1 -- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 msgr2=0x7fbe500051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6ddfd700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 0x7fbe500051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6ddfd700 1 -- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe50004810 msgr2=0x7fbe50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6ddfd700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe50004810 0x7fbe50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6ddfd700 1 -- 172.21.15.175:0/945631379 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe50003570 con 0x7fbe50003a90 2023-10-10T18:45:36.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.505+0000 7fbe6e5fe700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe680682e0 0x7fbe500051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:36.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe6ddfd700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe50003a90 0x7fbe50003ea0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800a930 tx=0x7fbe580021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5a77be8970db17 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe580070d0 con 0x7fbe50003a90 2023-10-10T18:45:36.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe58007230 con 0x7fbe50003a90 2023-10-10T18:45:36.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe6cdfb700 1 -- 172.21.15.175:0/945631379 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe580075d0 con 0x7fbe50003a90 2023-10-10T18:45:36.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe70061700 1 -- 172.21.15.175:0/945631379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe50003a90 msgr2=0x7fbe50003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe70061700 1 --2- 172.21.15.175:0/945631379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe50003a90 0x7fbe50003ea0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800a930 tx=0x7fbe580021d0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe6e5fe700 1 -- 172.21.15.175:0/945631379 reap_dead start 2023-10-10T18:45:36.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe70061700 1 -- 172.21.15.175:0/945631379 shutdown_connections 2023-10-10T18:45:36.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe70061700 1 -- 172.21.15.175:0/945631379 >> 172.21.15.175:0/945631379 conn(0x7fbe6806c110 msgr2=0x7fbe6806c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:36.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe70061700 1 -- 172.21.15.175:0/945631379 shutdown_connections 2023-10-10T18:45:36.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.506+0000 7fbe70061700 1 -- 172.21.15.175:0/945631379 wait complete. 2023-10-10T18:45:36.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.507+0000 7fbe70061700 1 Processor -- start 2023-10-10T18:45:36.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.507+0000 7fbe70061700 1 -- start start 2023-10-10T18:45:36.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe70061700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe50003a90 0x7fbe6816a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe70061700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe6815b5f0 0x7fbe6806e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe70061700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe680c2390 0x7fbe6806ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe70061700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68154d00 con 0x7fbe6815b5f0 2023-10-10T18:45:36.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe70061700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680c1eb0 con 0x7fbe680c2390 2023-10-10T18:45:36.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe70061700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680c0ec0 con 0x7fbe50003a90 2023-10-10T18:45:36.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6d5fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe6815b5f0 0x7fbe6806e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6d5fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe6815b5f0 0x7fbe6806e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51566/0 (socket says 172.21.15.175:51566) 2023-10-10T18:45:36.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6d5fc700 1 -- 172.21.15.175:0/3259432047 learned_addr learned my addr 172.21.15.175:0/3259432047 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:36.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6ddfd700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe50003a90 0x7fbe6816a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6e5fe700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe680c2390 0x7fbe6806ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6d5fc700 1 -- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe50003a90 msgr2=0x7fbe6816a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6d5fc700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe50003a90 0x7fbe6816a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6d5fc700 1 -- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe680c2390 msgr2=0x7fbe6806ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6d5fc700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe680c2390 0x7fbe6806ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.508+0000 7fbe6d5fc700 1 -- 172.21.15.175:0/3259432047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6816a3a0 con 0x7fbe6815b5f0 2023-10-10T18:45:36.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.509+0000 7fbe6d5fc700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe6815b5f0 0x7fbe6806e520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c011d70 tx=0x7fbe5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.509+0000 7fbe65ffb700 1 -- 172.21.15.175:0/3259432047 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c009e20 con 0x7fbe6815b5f0 2023-10-10T18:45:36.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.509+0000 7fbe65ffb700 1 -- 172.21.15.175:0/3259432047 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c00fcd0 con 0x7fbe6815b5f0 2023-10-10T18:45:36.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.509+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6816ad70 con 0x7fbe6815b5f0 2023-10-10T18:45:36.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.509+0000 7fbe65ffb700 1 -- 172.21.15.175:0/3259432047 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c014650 con 0x7fbe6815b5f0 2023-10-10T18:45:36.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.509+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe680c0be0 con 0x7fbe6815b5f0 2023-10-10T18:45:36.524 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T18:45:36.508+0000 7fc568727700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T18:45:36.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.510+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6816a530 con 0x7fbe6815b5f0 2023-10-10T18:45:36.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.510+0000 7fbe65ffb700 1 -- 172.21.15.175:0/3259432047 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbe5c014a80 con 0x7fbe6815b5f0 2023-10-10T18:45:36.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.511+0000 7fbe65ffb700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe54066f60 0x7fbe54069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.511+0000 7fbe65ffb700 1 -- 172.21.15.175:0/3259432047 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbe5c018030 con 0x7fbe6815b5f0 2023-10-10T18:45:36.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.511+0000 7fbe6ddfd700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe54066f60 0x7fbe54069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.511+0000 7fbe6ddfd700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe54066f60 0x7fbe54069410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800afd0 tx=0x7fbe58016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.515+0000 7fbe65ffb700 1 -- 172.21.15.175:0/3259432047 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbe5c052260 con 0x7fbe6815b5f0 2023-10-10T18:45:36.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.701+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe68067570 con 0x7fbe6815b5f0 2023-10-10T18:45:36.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.702+0000 7fbe65ffb700 1 -- 172.21.15.175:0/3259432047 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7fbe5c055880 con 0x7fbe6815b5f0 2023-10-10T18:45:36.702 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:36.702 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:36.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe54066f60 msgr2=0x7fbe54069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe54066f60 0x7fbe54069410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800afd0 tx=0x7fbe58016000 comp rx=0 tx=0).stop 2023-10-10T18:45:36.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe6815b5f0 msgr2=0x7fbe6806e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe6815b5f0 0x7fbe6806e520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c011d70 tx=0x7fbe5c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 shutdown_connections 2023-10-10T18:45:36.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbe54066f60 0x7fbe54069410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe680c2390 0x7fbe6806ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe50003a90 0x7fbe6816a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 --2- 172.21.15.175:0/3259432047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe6815b5f0 0x7fbe6806e520 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 >> 172.21.15.175:0/3259432047 conn(0x7fbe6806c110 msgr2=0x7fbe6814bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:36.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 shutdown_connections 2023-10-10T18:45:36.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.704+0000 7fbe70061700 1 -- 172.21.15.175:0/3259432047 wait complete. 2023-10-10T18:45:36.716 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T18:45:05.161495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T18:45:08.855952+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T18:45:09.742056+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T18:45:36.716 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T18:45:36.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.834+0000 7f3bad117700 1 Processor -- start 2023-10-10T18:45:36.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.834+0000 7f3bad117700 1 -- start start 2023-10-10T18:45:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3bad117700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3ba80633f0 0x7f3ba8156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3bad117700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba80c7620 0x7f3ba80aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3bad117700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba80c7ca0 con 0x7f3ba80b0490 2023-10-10T18:45:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3bad117700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba80af540 con 0x7f3ba80633f0 2023-10-10T18:45:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3bad117700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8156830 con 0x7f3ba80c7620 2023-10-10T18:45:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3ba759e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f3ba80b0490 0x7f3ba8159610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51292/0 (socket says 172.21.15.175:51292) 2023-10-10T18:45:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3ba759e700 1 -- 172.21.15.175:0/1819839053 learned_addr learned my addr 172.21.15.175:0/1819839053 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226026697 0 0) 0x7f3ba80c7ca0 con 0x7f3ba80b0490 2023-10-10T18:45:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3ba6d9d700 1 --2- 172.21.15.175:0/1819839053 >> v2:172.21.15.191:3300/0 conn(0x7f3ba80633f0 0x7f3ba8156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b94003570 con 0x7f3ba80b0490 2023-10-10T18:45:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3b9e59c700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba80c7620 0x7f3ba80aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.835+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591880570 0 0) 0x7f3b94003570 con 0x7f3ba80b0490 2023-10-10T18:45:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba80c7620 msgr2=0x7f3ba80aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba80c7620 0x7f3ba80aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 >> v2:172.21.15.191:3300/0 conn(0x7f3ba80633f0 msgr2=0x7f3ba8156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 --2- 172.21.15.175:0/1819839053 >> v2:172.21.15.191:3300/0 conn(0x7f3ba80633f0 0x7f3ba8156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8156ad0 con 0x7f3ba80b0490 2023-10-10T18:45:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f3ba00017f0 con 0x7f3ba80b0490 2023-10-10T18:45:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 >> v1:172.21.15.175:6789/0 conn(0x7f3ba80b0490 legacy=0x7f3ba8159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba80c7620 0x7f3b94003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94003fe0 0x7f3b94004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b94004d50 0x7f3b940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba80af540 con 0x7f3b94004d50 2023-10-10T18:45:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8156830 con 0x7f3b94003fe0 2023-10-10T18:45:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9400ba90 con 0x7f3ba80c7620 2023-10-10T18:45:36.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba759e700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b94004d50 0x7f3b940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba6d9d700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba80c7620 0x7f3b94003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3b9e59c700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94003fe0 0x7f3b94004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba759e700 1 -- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba80c7620 msgr2=0x7f3b94003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba759e700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba80c7620 0x7f3b94003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba759e700 1 -- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94003fe0 msgr2=0x7f3b94004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:36.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba759e700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94003fe0 0x7f3b94004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba759e700 1 -- 172.21.15.175:0/1819839053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b94003570 con 0x7f3b94004d50 2023-10-10T18:45:36.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba759e700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b94004d50 0x7f3b940051c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000a970 tx=0x7f3ba0002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a7a1d3a0475bd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0012440 con 0x7f3b94004d50 2023-10-10T18:45:36.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba00125c0 con 0x7f3b94004d50 2023-10-10T18:45:36.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.836+0000 7f3ba659c700 1 -- 172.21.15.175:0/1819839053 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0012980 con 0x7f3b94004d50 2023-10-10T18:45:36.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.837+0000 7f3bad117700 1 -- 172.21.15.175:0/1819839053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b94004d50 msgr2=0x7f3b940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.837+0000 7f3bad117700 1 --2- 172.21.15.175:0/1819839053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b94004d50 0x7f3b940051c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000a970 tx=0x7f3ba0002210 comp rx=0 tx=0).stop 2023-10-10T18:45:36.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.837+0000 7f3ba759e700 1 -- 172.21.15.175:0/1819839053 reap_dead start 2023-10-10T18:45:36.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.837+0000 7f3bad117700 1 -- 172.21.15.175:0/1819839053 shutdown_connections 2023-10-10T18:45:36.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.837+0000 7f3bad117700 1 -- 172.21.15.175:0/1819839053 >> 172.21.15.175:0/1819839053 conn(0x7f3ba80cc7a0 msgr2=0x7f3ba80ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:36.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.837+0000 7f3bad117700 1 -- 172.21.15.175:0/1819839053 shutdown_connections 2023-10-10T18:45:36.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.837+0000 7f3bad117700 1 -- 172.21.15.175:0/1819839053 wait complete. 2023-10-10T18:45:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.838+0000 7f3bad117700 1 Processor -- start 2023-10-10T18:45:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.838+0000 7f3bad117700 1 -- start start 2023-10-10T18:45:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3bad117700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94004d50 0x7f3ba8169540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3bad117700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba8069460 0x7f3ba815a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3bad117700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3ba8169a80 0x7f3ba815ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3bad117700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8157bf0 con 0x7f3ba8169a80 2023-10-10T18:45:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3bad117700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8158730 con 0x7f3b94004d50 2023-10-10T18:45:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3bad117700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba816a050 con 0x7f3ba8069460 2023-10-10T18:45:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3ba8169a80 0x7f3ba815ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3ba8169a80 0x7f3ba815ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51596/0 (socket says 172.21.15.175:51596) 2023-10-10T18:45:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 -- 172.21.15.175:0/873370097 learned_addr learned my addr 172.21.15.175:0/873370097 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba6d9d700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94004d50 0x7f3ba8169540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3b9e59c700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba8069460 0x7f3ba815a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 -- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba8069460 msgr2=0x7f3ba815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba8069460 0x7f3ba815a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 -- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94004d50 msgr2=0x7f3ba8169540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94004d50 0x7f3ba8169540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 -- 172.21.15.175:0/873370097 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8157e00 con 0x7f3ba8169a80 2023-10-10T18:45:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3ba759e700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3ba8169a80 0x7f3ba815ae20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0002250 tx=0x7f3ba0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3b9f7fe700 1 -- 172.21.15.175:0/873370097 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0012440 con 0x7f3ba8169a80 2023-10-10T18:45:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3b9f7fe700 1 -- 172.21.15.175:0/873370097 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba00017b0 con 0x7f3ba8169a80 2023-10-10T18:45:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba81580e0 con 0x7f3ba8169a80 2023-10-10T18:45:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3b9f7fe700 1 -- 172.21.15.175:0/873370097 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00012d0 con 0x7f3ba8169a80 2023-10-10T18:45:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.839+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba8159920 con 0x7f3ba8169a80 2023-10-10T18:45:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.841+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba8067110 con 0x7f3ba8169a80 2023-10-10T18:45:36.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.841+0000 7f3b9f7fe700 1 -- 172.21.15.175:0/873370097 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3ba0019030 con 0x7f3ba8169a80 2023-10-10T18:45:36.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.841+0000 7f3b9f7fe700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b88066f30 0x7f3b880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:36.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.841+0000 7f3b9f7fe700 1 -- 172.21.15.175:0/873370097 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3ba0086950 con 0x7f3ba8169a80 2023-10-10T18:45:36.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.841+0000 7f3ba6d9d700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b88066f30 0x7f3b880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:36.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.842+0000 7f3ba6d9d700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b88066f30 0x7f3b880693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3b98006ed0 tx=0x7f3b98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:36.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:36.845+0000 7f3b9f7fe700 1 -- 172.21.15.175:0/873370097 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ba0053bc0 con 0x7f3ba8169a80 2023-10-10T18:45:37.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.027+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3ba8067570 con 0x7f3ba8169a80 2023-10-10T18:45:37.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.027+0000 7f3b9f7fe700 1 -- 172.21.15.175:0/873370097 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ba00170a0 con 0x7f3ba8169a80 2023-10-10T18:45:37.028 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 1 2023-10-10T18:45:37.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b88066f30 msgr2=0x7f3b880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b88066f30 0x7f3b880693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3b98006ed0 tx=0x7f3b98008040 comp rx=0 tx=0).stop 2023-10-10T18:45:37.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3ba8169a80 msgr2=0x7f3ba815ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3ba8169a80 0x7f3ba815ae20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0002250 tx=0x7f3ba0015000 comp rx=0 tx=0).stop 2023-10-10T18:45:37.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 shutdown_connections 2023-10-10T18:45:37.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b88066f30 0x7f3b880693e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3ba8169a80 0x7f3ba815ae20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b94004d50 0x7f3ba8169540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 --2- 172.21.15.175:0/873370097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ba8069460 0x7f3ba815a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 >> 172.21.15.175:0/873370097 conn(0x7f3ba80cc7a0 msgr2=0x7f3ba8155650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:37.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 shutdown_connections 2023-10-10T18:45:37.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.029+0000 7f3bad117700 1 -- 172.21.15.175:0/873370097 wait complete. 2023-10-10T18:45:37.041 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T18:45:37.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.158+0000 7f6e0be27700 1 Processor -- start 2023-10-10T18:45:37.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.158+0000 7f6e0be27700 1 -- start start 2023-10-10T18:45:37.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.159+0000 7f6e0be27700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 0x7f6e040b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.159+0000 7f6e0be27700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040b2320 con 0x7f6e04067b60 2023-10-10T18:45:37.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.159+0000 7f6e0be27700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040c8d20 con 0x7f6e040c71c0 2023-10-10T18:45:37.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.159+0000 7f6e0be27700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040b2840 con 0x7f6e04156780 2023-10-10T18:45:37.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.159+0000 7f6e09bc3700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6e04067b60 0x7f6e0414ff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51302/0 (socket says 172.21.15.175:51302) 2023-10-10T18:45:37.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.159+0000 7f6e09bc3700 1 -- 172.21.15.175:0/2108241802 learned_addr learned my addr 172.21.15.175:0/2108241802 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:37.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e0a3c4700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 0x7f6e040b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489731510 0 0) 0x7f6e00001490 con 0x7f6e04067b60 2023-10-10T18:45:37.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df0003570 con 0x7f6e04067b60 2023-10-10T18:45:37.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 67978207 0 0) 0x7f6df0003570 con 0x7f6e04067b60 2023-10-10T18:45:37.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 msgr2=0x7f6e040b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:37.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 0x7f6e040b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 >> v1:172.21.15.191:6789/0 conn(0x7f6e040c71c0 legacy=0x7f6e04156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e040b2560 con 0x7f6e04067b60 2023-10-10T18:45:37.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f6e00001980 con 0x7f6e04067b60 2023-10-10T18:45:37.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 >> v1:172.21.15.175:6789/0 conn(0x7f6e04067b60 legacy=0x7f6e0414ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df0003a90 0x7f6df0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df0004810 0x7f6df0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 0x7f6df00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040b2840 con 0x7f6df0003a90 2023-10-10T18:45:37.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df000bb40 con 0x7f6df0004810 2023-10-10T18:45:37.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df000bd00 con 0x7f6e04156780 2023-10-10T18:45:37.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e09bc3700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df0003a90 0x7f6df0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e093c2700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df0004810 0x7f6df0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e0a3c4700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 0x7f6df00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e09bc3700 1 -- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 msgr2=0x7f6df00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:37.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e09bc3700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 0x7f6df00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e09bc3700 1 -- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df0004810 msgr2=0x7f6df0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:37.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e09bc3700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df0004810 0x7f6df0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e09bc3700 1 -- 172.21.15.175:0/2108241802 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0003570 con 0x7f6df0003a90 2023-10-10T18:45:37.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e093c2700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df0004810 0x7f6df0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:37.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e0a3c4700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04156780 0x7f6df00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:37.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.160+0000 7f6e09bc3700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df0003a90 0x7f6df0003e60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000aad0 tx=0x7f6e040b1390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc3940b5b4e8235 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000a460 con 0x7f6df0003a90 2023-10-10T18:45:37.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0000c5d0 con 0x7f6df0003a90 2023-10-10T18:45:37.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e08bc1700 1 -- 172.21.15.175:0/2108241802 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000c990 con 0x7f6df0003a90 2023-10-10T18:45:37.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e0be27700 1 -- 172.21.15.175:0/2108241802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df0003a90 msgr2=0x7f6df0003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e0be27700 1 --2- 172.21.15.175:0/2108241802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df0003a90 0x7f6df0003e60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000aad0 tx=0x7f6e040b1390 comp rx=0 tx=0).stop 2023-10-10T18:45:37.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e0a3c4700 1 -- 172.21.15.175:0/2108241802 reap_dead start 2023-10-10T18:45:37.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e0be27700 1 -- 172.21.15.175:0/2108241802 shutdown_connections 2023-10-10T18:45:37.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e0be27700 1 -- 172.21.15.175:0/2108241802 >> 172.21.15.175:0/2108241802 conn(0x7f6e04068700 msgr2=0x7f6e04068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:37.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e0be27700 1 -- 172.21.15.175:0/2108241802 shutdown_connections 2023-10-10T18:45:37.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.161+0000 7f6e0be27700 1 -- 172.21.15.175:0/2108241802 wait complete. 2023-10-10T18:45:37.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.162+0000 7f6e0be27700 1 Processor -- start 2023-10-10T18:45:37.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.162+0000 7f6e0be27700 1 -- start start 2023-10-10T18:45:37.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e0be27700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e0414fdd0 0x7f6e04159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e0be27700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0415a8e0 0x7f6e0415a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e0be27700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e041501a0 0x7f6e041690d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e0be27700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04166580 con 0x7f6e0414fdd0 2023-10-10T18:45:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e0be27700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04166a50 con 0x7f6e041501a0 2023-10-10T18:45:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e0be27700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0415bc50 con 0x7f6e0415a8e0 2023-10-10T18:45:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e0414fdd0 0x7f6e04159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e0414fdd0 0x7f6e04159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51612/0 (socket says 172.21.15.175:51612) 2023-10-10T18:45:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 -- 172.21.15.175:0/2029234336 learned_addr learned my addr 172.21.15.175:0/2029234336 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e0a3c4700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e041501a0 0x7f6e041690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e093c2700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0415a8e0 0x7f6e0415a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 -- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0415a8e0 msgr2=0x7f6e0415a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0415a8e0 0x7f6e0415a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 -- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e041501a0 msgr2=0x7f6e041690d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e041501a0 0x7f6e041690d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 -- 172.21.15.175:0/2029234336 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04166770 con 0x7f6e0414fdd0 2023-10-10T18:45:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e0a3c4700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e041501a0 0x7f6e041690d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e093c2700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0415a8e0 0x7f6e0415a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.163+0000 7f6e09bc3700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e0414fdd0 0x7f6e04159be0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6df000bb10 tx=0x7f6e00016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.164+0000 7f6df9ffb700 1 -- 172.21.15.175:0/2029234336 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00001c20 con 0x7f6e0414fdd0 2023-10-10T18:45:37.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.164+0000 7f6df9ffb700 1 -- 172.21.15.175:0/2029234336 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0001e440 con 0x7f6e0414fdd0 2023-10-10T18:45:37.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.164+0000 7f6df9ffb700 1 -- 172.21.15.175:0/2029234336 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0001e5f0 con 0x7f6e0414fdd0 2023-10-10T18:45:37.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.164+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0415bef0 con 0x7f6e0414fdd0 2023-10-10T18:45:37.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.164+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0408a390 con 0x7f6e0414fdd0 2023-10-10T18:45:37.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.165+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e041598e0 con 0x7f6e0414fdd0 2023-10-10T18:45:37.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.165+0000 7f6df9ffb700 1 -- 172.21.15.175:0/2029234336 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6e00025020 con 0x7f6e0414fdd0 2023-10-10T18:45:37.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.165+0000 7f6df9ffb700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6de8066f30 0x7f6de80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.166+0000 7f6df9ffb700 1 -- 172.21.15.175:0/2029234336 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6e00086b70 con 0x7f6e0414fdd0 2023-10-10T18:45:37.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.166+0000 7f6e093c2700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6de8066f30 0x7f6de80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.166+0000 7f6e093c2700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6de8066f30 0x7f6de80693e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6e0414d9f0 tx=0x7f6df4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.170+0000 7f6df9ffb700 1 -- 172.21.15.175:0/2029234336 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e00053de0 con 0x7f6e0414fdd0 2023-10-10T18:45:37.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.351+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6e040b2320 con 0x7f6e0414fdd0 2023-10-10T18:45:37.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.351+0000 7f6df9ffb700 1 -- 172.21.15.175:0/2029234336 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6e040b2320 con 0x7f6e0414fdd0 2023-10-10T18:45:37.352 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 32 2023-10-10T18:45:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.353+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6de8066f30 msgr2=0x7f6de80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.353+0000 7f6e0be27700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6de8066f30 0x7f6de80693e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6e0414d9f0 tx=0x7f6df4009040 comp rx=0 tx=0).stop 2023-10-10T18:45:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.353+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e0414fdd0 msgr2=0x7f6e04159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.353+0000 7f6e0be27700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e0414fdd0 0x7f6e04159be0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6df000bb10 tx=0x7f6e00016000 comp rx=0 tx=0).stop 2023-10-10T18:45:37.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.354+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 shutdown_connections 2023-10-10T18:45:37.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.354+0000 7f6e0be27700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6de8066f30 0x7f6de80693e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.354+0000 7f6e0be27700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e041501a0 0x7f6e041690d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.354+0000 7f6e0be27700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e0414fdd0 0x7f6e04159be0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.354+0000 7f6e0be27700 1 --2- 172.21.15.175:0/2029234336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0415a8e0 0x7f6e0415a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.354+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 >> 172.21.15.175:0/2029234336 conn(0x7f6e04068700 msgr2=0x7f6e04068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:37.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.354+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 shutdown_connections 2023-10-10T18:45:37.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.354+0000 7f6e0be27700 1 -- 172.21.15.175:0/2029234336 wait complete. 2023-10-10T18:45:37.365 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T18:45:37.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.483+0000 7f774d51e700 1 Processor -- start 2023-10-10T18:45:37.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.483+0000 7f774d51e700 1 -- start start 2023-10-10T18:45:37.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774d51e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f77480c7ae0 0x7f77480c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774d51e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7748156dd0 con 0x7f7748068630 2023-10-10T18:45:37.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774d51e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77480c7650 con 0x7f77480c39d0 2023-10-10T18:45:37.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774d51e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7748156620 con 0x7f77480c7ae0 2023-10-10T18:45:37.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774cd1d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f77480c7ae0 0x7f77480c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f7747fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7748068630 0x7f7748155f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51316/0 (socket says 172.21.15.175:51316) 2023-10-10T18:45:37.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774cd1d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f77480c7ae0 0x7f77480c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51622/0 (socket says 172.21.15.175:51622) 2023-10-10T18:45:37.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f7747fff700 1 -- 172.21.15.175:0/1095350252 learned_addr learned my addr 172.21.15.175:0/1095350252 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:37.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774cd1d700 1 -- 172.21.15.175:0/1095350252 >> v1:172.21.15.191:6789/0 conn(0x7f77480c39d0 legacy=0x7f7748068a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774cd1d700 1 -- 172.21.15.175:0/1095350252 >> v1:172.21.15.175:6789/0 conn(0x7f7748068630 legacy=0x7f7748155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774cd1d700 1 -- 172.21.15.175:0/1095350252 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7748069730 con 0x7f77480c7ae0 2023-10-10T18:45:37.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f7746ffd700 1 -- 172.21.15.175:0/1095350252 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518495199 0 0) 0x7f7748156dd0 con 0x7f7748068630 2023-10-10T18:45:37.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.484+0000 7f774cd1d700 1 --2- 172.21.15.175:0/1095350252 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c7ae0 0x7f77480c9460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7740009e60 tx=0x7f7740009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f4fdd3eeb5e3e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f7746ffd700 1 -- 172.21.15.175:0/1095350252 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774000e070 con 0x7f77480c7ae0 2023-10-10T18:45:37.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f7746ffd700 1 -- 172.21.15.175:0/1095350252 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774000bc30 con 0x7f77480c7ae0 2023-10-10T18:45:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f7746ffd700 1 -- 172.21.15.175:0/1095350252 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7740012430 con 0x7f77480c7ae0 2023-10-10T18:45:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f774d51e700 1 -- 172.21.15.175:0/1095350252 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c7ae0 msgr2=0x7f77480c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f774d51e700 1 --2- 172.21.15.175:0/1095350252 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c7ae0 0x7f77480c9460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7740009e60 tx=0x7f7740009780 comp rx=0 tx=0).stop 2023-10-10T18:45:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f774d51e700 1 -- 172.21.15.175:0/1095350252 shutdown_connections 2023-10-10T18:45:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f774d51e700 1 --2- 172.21.15.175:0/1095350252 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c7ae0 0x7f77480c9460 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f774d51e700 1 -- 172.21.15.175:0/1095350252 >> 172.21.15.175:0/1095350252 conn(0x7f7748063230 msgr2=0x7f7748063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:37.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f774d51e700 1 -- 172.21.15.175:0/1095350252 shutdown_connections 2023-10-10T18:45:37.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.485+0000 7f774d51e700 1 -- 172.21.15.175:0/1095350252 wait complete. 2023-10-10T18:45:37.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.486+0000 7f774d51e700 1 Processor -- start 2023-10-10T18:45:37.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.486+0000 7f774d51e700 1 -- start start 2023-10-10T18:45:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f774d51e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7748068630 0x7f77481560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f774d51e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c39d0 0x7f7748163b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f774d51e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f77480c7ae0 0x7f77480cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f774d51e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7748153b70 con 0x7f77480c39d0 2023-10-10T18:45:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f774d51e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77481546f0 con 0x7f7748068630 2023-10-10T18:45:37.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f774d51e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77481668f0 con 0x7f77480c7ae0 2023-10-10T18:45:37.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f77477fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c39d0 0x7f7748163b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f77477fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c39d0 0x7f7748163b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51638/0 (socket says 172.21.15.175:51638) 2023-10-10T18:45:37.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f77477fe700 1 -- 172.21.15.175:0/3891358220 learned_addr learned my addr 172.21.15.175:0/3891358220 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:37.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f7747fff700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7748068630 0x7f77481560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f774cd1d700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f77480c7ae0 0x7f77480cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f77477fe700 1 -- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f77480c7ae0 msgr2=0x7f77480cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.487+0000 7f77477fe700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f77480c7ae0 0x7f77480cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f77477fe700 1 -- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7748068630 msgr2=0x7f77481560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f77477fe700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7748068630 0x7f77481560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f77477fe700 1 -- 172.21.15.175:0/3891358220 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7740009a20 con 0x7f77480c39d0 2023-10-10T18:45:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f7747fff700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7748068630 0x7f77481560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f774cd1d700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f77480c7ae0 0x7f77480cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f77477fe700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c39d0 0x7f7748163b30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f773c00c590 tx=0x7f773c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f772ffff700 1 -- 172.21.15.175:0/3891358220 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c011590 con 0x7f77480c39d0 2023-10-10T18:45:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f772ffff700 1 -- 172.21.15.175:0/3891358220 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c0116f0 con 0x7f77480c39d0 2023-10-10T18:45:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7748153dc0 con 0x7f77480c39d0 2023-10-10T18:45:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f772ffff700 1 -- 172.21.15.175:0/3891358220 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c011ab0 con 0x7f77480c39d0 2023-10-10T18:45:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.488+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7748154380 con 0x7f77480c39d0 2023-10-10T18:45:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.489+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77481548a0 con 0x7f77480c39d0 2023-10-10T18:45:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.489+0000 7f772ffff700 1 -- 172.21.15.175:0/3891358220 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f773c01b3b0 con 0x7f77480c39d0 2023-10-10T18:45:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.490+0000 7f772ffff700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7734066f30 0x7f77340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.490+0000 7f772ffff700 1 -- 172.21.15.175:0/3891358220 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f773c083ac0 con 0x7f77480c39d0 2023-10-10T18:45:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.490+0000 7f7747fff700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7734066f30 0x7f77340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.490+0000 7f7747fff700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7734066f30 0x7f77340693e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7738001810 tx=0x7f7738009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.494+0000 7f772ffff700 1 -- 172.21.15.175:0/3891358220 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77481548a0 con 0x7f77480c39d0 2023-10-10T18:45:37.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.677+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7748154860 con 0x7f77480c39d0 2023-10-10T18:45:37.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.677+0000 7f772ffff700 1 -- 172.21.15.175:0/3891358220 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f773c017070 con 0x7f77480c39d0 2023-10-10T18:45:37.677 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 64 2023-10-10T18:45:37.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7734066f30 msgr2=0x7f77340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7734066f30 0x7f77340693e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7738001810 tx=0x7f7738009040 comp rx=0 tx=0).stop 2023-10-10T18:45:37.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c39d0 msgr2=0x7f7748163b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c39d0 0x7f7748163b30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f773c00c590 tx=0x7f773c00c950 comp rx=0 tx=0).stop 2023-10-10T18:45:37.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 shutdown_connections 2023-10-10T18:45:37.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7734066f30 0x7f77340693e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f77480c7ae0 0x7f77480cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7748068630 0x7f77481560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 --2- 172.21.15.175:0/3891358220 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f77480c39d0 0x7f7748163b30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 >> 172.21.15.175:0/3891358220 conn(0x7f7748063230 msgr2=0x7f77480ca740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 shutdown_connections 2023-10-10T18:45:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.679+0000 7f774d51e700 1 -- 172.21.15.175:0/3891358220 wait complete. 2023-10-10T18:45:37.691 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:37.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.805+0000 7f40e838b700 1 Processor -- start 2023-10-10T18:45:37.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.805+0000 7f40e838b700 1 -- start start 2023-10-10T18:45:37.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.806+0000 7f40e838b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f40e0068a40 0x7f40e0063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.806+0000 7f40e838b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f40e00cac80 0x7f40e006b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.806+0000 7f40e838b700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e00c4ee0 con 0x7f40e0068e10 2023-10-10T18:45:37.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.806+0000 7f40e838b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e0064180 con 0x7f40e00cac80 2023-10-10T18:45:37.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.806+0000 7f40e838b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e00c5780 con 0x7f40e0068a40 2023-10-10T18:45:37.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5926700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f40e00cac80 0x7f40e006b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e6928700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f40e0068e10 0x7f40e00af910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51320/0 (socket says 172.21.15.175:51320) 2023-10-10T18:45:37.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e6928700 1 -- 172.21.15.175:0/1128294602 learned_addr learned my addr 172.21.15.175:0/1128294602 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:37.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e6127700 1 --2- 172.21.15.175:0/1128294602 >> v2:172.21.15.191:3300/0 conn(0x7f40e0068a40 0x7f40e0063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5926700 1 -- 172.21.15.175:0/1128294602 >> v2:172.21.15.191:3300/0 conn(0x7f40e0068a40 msgr2=0x7f40e0063c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:37.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5926700 1 --2- 172.21.15.175:0/1128294602 >> v2:172.21.15.191:3300/0 conn(0x7f40e0068a40 0x7f40e0063c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5926700 1 -- 172.21.15.175:0/1128294602 >> v1:172.21.15.175:6789/0 conn(0x7f40e0068e10 legacy=0x7f40e00af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5926700 1 -- 172.21.15.175:0/1128294602 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e00c6710 con 0x7f40e00cac80 2023-10-10T18:45:37.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5125700 1 -- 172.21.15.175:0/1128294602 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164416719 0 0) 0x7f40e00c4ee0 con 0x7f40e0068e10 2023-10-10T18:45:37.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5926700 1 --2- 172.21.15.175:0/1128294602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e00cac80 0x7f40e006b720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f40e006b320 tx=0x7f40d0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4500a4edc6968287 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5125700 1 -- 172.21.15.175:0/1128294602 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0005c10 con 0x7f40e00cac80 2023-10-10T18:45:37.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5125700 1 -- 172.21.15.175:0/1128294602 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d0005d70 con 0x7f40e00cac80 2023-10-10T18:45:37.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.807+0000 7f40e5125700 1 -- 172.21.15.175:0/1128294602 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0006280 con 0x7f40e00cac80 2023-10-10T18:45:37.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.808+0000 7f40e838b700 1 -- 172.21.15.175:0/1128294602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e00cac80 msgr2=0x7f40e006b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.808+0000 7f40e838b700 1 --2- 172.21.15.175:0/1128294602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e00cac80 0x7f40e006b720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f40e006b320 tx=0x7f40d0009120 comp rx=0 tx=0).stop 2023-10-10T18:45:37.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.808+0000 7f40e838b700 1 -- 172.21.15.175:0/1128294602 shutdown_connections 2023-10-10T18:45:37.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.808+0000 7f40e838b700 1 --2- 172.21.15.175:0/1128294602 >> v2:172.21.15.191:3300/0 conn(0x7f40e0068a40 0x7f40e0063c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.808+0000 7f40e838b700 1 --2- 172.21.15.175:0/1128294602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e00cac80 0x7f40e006b720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.808+0000 7f40e838b700 1 -- 172.21.15.175:0/1128294602 >> 172.21.15.175:0/1128294602 conn(0x7f40e006dbe0 msgr2=0x7f40e00ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:37.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.808+0000 7f40e838b700 1 -- 172.21.15.175:0/1128294602 shutdown_connections 2023-10-10T18:45:37.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.808+0000 7f40e838b700 1 -- 172.21.15.175:0/1128294602 wait complete. 2023-10-10T18:45:37.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.809+0000 7f40e838b700 1 Processor -- start 2023-10-10T18:45:37.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.809+0000 7f40e838b700 1 -- start start 2023-10-10T18:45:37.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.809+0000 7f40e838b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e0068a40 0x7f40e0167940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.809+0000 7f40e838b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e0068e10 0x7f40e0166ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.809+0000 7f40e838b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40e00cac80 0x7f40e00bdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.809+0000 7f40e838b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00aaee0 con 0x7f40e0068a40 2023-10-10T18:45:37.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.809+0000 7f40e838b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e0167430 con 0x7f40e00cac80 2023-10-10T18:45:37.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.809+0000 7f40e838b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00c35e0 con 0x7f40e0068e10 2023-10-10T18:45:37.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e0068a40 0x7f40e0167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e0068a40 0x7f40e0167940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51654/0 (socket says 172.21.15.175:51654) 2023-10-10T18:45:37.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 -- 172.21.15.175:0/1640497414 learned_addr learned my addr 172.21.15.175:0/1640497414 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:37.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6928700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40e00cac80 0x7f40e00bdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e5926700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e0068e10 0x7f40e0166ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 -- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e0068e10 msgr2=0x7f40e0166ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e0068e10 0x7f40e0166ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 -- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40e00cac80 msgr2=0x7f40e00bdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:37.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40e00cac80 0x7f40e00bdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:37.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 -- 172.21.15.175:0/1640497414 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d00093c0 con 0x7f40e0068a40 2023-10-10T18:45:37.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6928700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40e00cac80 0x7f40e00bdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:37.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e6127700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e0068a40 0x7f40e0167940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00e160 tx=0x7f40dc009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40d67fc700 1 -- 172.21.15.175:0/1640497414 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc00ee00 con 0x7f40e0068a40 2023-10-10T18:45:37.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40d67fc700 1 -- 172.21.15.175:0/1640497414 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40dc014420 con 0x7f40e0068a40 2023-10-10T18:45:37.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e00be4f0 con 0x7f40e0068a40 2023-10-10T18:45:37.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40d67fc700 1 -- 172.21.15.175:0/1640497414 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0147e0 con 0x7f40e0068a40 2023-10-10T18:45:37.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.810+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e00c29d0 con 0x7f40e0068a40 2023-10-10T18:45:37.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.811+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e00c38c0 con 0x7f40e0068a40 2023-10-10T18:45:37.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.811+0000 7f40d67fc700 1 -- 172.21.15.175:0/1640497414 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f40dc014b50 con 0x7f40e0068a40 2023-10-10T18:45:37.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.812+0000 7f40d67fc700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40c4066f30 0x7f40c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:37.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.812+0000 7f40d67fc700 1 -- 172.21.15.175:0/1640497414 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f40dc084d70 con 0x7f40e0068a40 2023-10-10T18:45:37.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.812+0000 7f40e5926700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40c4066f30 0x7f40c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:37.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.812+0000 7f40e5926700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40c4066f30 0x7f40c40693e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009390 tx=0x7f40d0016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:37.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:37.816+0000 7f40d67fc700 1 -- 172.21.15.175:0/1640497414 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f40dc051fe0 con 0x7f40e0068a40 2023-10-10T18:45:38.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.041+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40e0067570 con 0x7f40e0068a40 2023-10-10T18:45:38.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.041+0000 7f40d67fc700 1 -- 172.21.15.175:0/1640497414 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6177 (secure 0 0 0) 0x7f40dc009ab0 con 0x7f40e0068a40 2023-10-10T18:45:38.042 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:38.042 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:45:35.538717+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:45:38.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.043+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40c4066f30 msgr2=0x7f40c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.043+0000 7f40e838b700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40c4066f30 0x7f40c40693e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009390 tx=0x7f40d0016000 comp rx=0 tx=0).stop 2023-10-10T18:45:38.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.043+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e0068a40 msgr2=0x7f40e0167940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.043+0000 7f40e838b700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e0068a40 0x7f40e0167940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00e160 tx=0x7f40dc009970 comp rx=0 tx=0).stop 2023-10-10T18:45:38.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.044+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 shutdown_connections 2023-10-10T18:45:38.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.044+0000 7f40e838b700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40c4066f30 0x7f40c40693e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.044+0000 7f40e838b700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40e00cac80 0x7f40e00bdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.044+0000 7f40e838b700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e0068a40 0x7f40e0167940 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.044+0000 7f40e838b700 1 --2- 172.21.15.175:0/1640497414 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e0068e10 0x7f40e0166ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.044+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 >> 172.21.15.175:0/1640497414 conn(0x7f40e006dbe0 msgr2=0x7f40e0157710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:38.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.044+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 shutdown_connections 2023-10-10T18:45:38.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.044+0000 7f40e838b700 1 -- 172.21.15.175:0/1640497414 wait complete. 2023-10-10T18:45:38.046 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 16 2023-10-10T18:45:38.056 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:38.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.171+0000 7f2f6ca18700 1 Processor -- start 2023-10-10T18:45:38.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.171+0000 7f2f6ca18700 1 -- start start 2023-10-10T18:45:38.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f6ca18700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2f681569a0 0x7f2f6806d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f6ca18700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f680c0f70 0x7f2f68160980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f6ca18700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6806d8a0 con 0x7f2f6806cf90 2023-10-10T18:45:38.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f6ca18700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f680c0990 con 0x7f2f680c0f70 2023-10-10T18:45:38.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f6ca18700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f681595e0 con 0x7f2f681569a0 2023-10-10T18:45:38.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f67fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f680c0f70 0x7f2f68160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f67fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f680c0f70 0x7f2f68160980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51670/0 (socket says 172.21.15.175:51670) 2023-10-10T18:45:38.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f67fff700 1 -- 172.21.15.175:0/3958419150 learned_addr learned my addr 172.21.15.175:0/3958419150 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:38.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f677fe700 1 --1- 172.21.15.175:0/3958419150 >> v1:172.21.15.175:6789/0 conn(0x7f2f6806cf90 0x7f2f68156290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51330/0 (socket says 172.21.15.175:51330) 2023-10-10T18:45:38.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.172+0000 7f2f66ffd700 1 --2- 172.21.15.175:0/3958419150 >> v2:172.21.15.191:3300/0 conn(0x7f2f681569a0 0x7f2f6806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f67fff700 1 -- 172.21.15.175:0/3958419150 >> v2:172.21.15.191:3300/0 conn(0x7f2f681569a0 msgr2=0x7f2f6806d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:38.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f67fff700 1 --2- 172.21.15.175:0/3958419150 >> v2:172.21.15.191:3300/0 conn(0x7f2f681569a0 0x7f2f6806d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f67fff700 1 -- 172.21.15.175:0/3958419150 >> v1:172.21.15.175:6789/0 conn(0x7f2f6806cf90 legacy=0x7f2f68156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f67fff700 1 -- 172.21.15.175:0/3958419150 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6815a440 con 0x7f2f680c0f70 2023-10-10T18:45:38.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f667fc700 1 -- 172.21.15.175:0/3958419150 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170849828 0 0) 0x7f2f6806d8a0 con 0x7f2f6806cf90 2023-10-10T18:45:38.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f67fff700 1 --2- 172.21.15.175:0/3958419150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f680c0f70 0x7f2f68160980 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2f68065040 tx=0x7f2f60009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dab9efbe5f84f86 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f667fc700 1 -- 172.21.15.175:0/3958419150 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000e070 con 0x7f2f680c0f70 2023-10-10T18:45:38.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f667fc700 1 -- 172.21.15.175:0/3958419150 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6000ba50 con 0x7f2f680c0f70 2023-10-10T18:45:38.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f667fc700 1 -- 172.21.15.175:0/3958419150 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f600092e0 con 0x7f2f680c0f70 2023-10-10T18:45:38.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f6ca18700 1 -- 172.21.15.175:0/3958419150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f680c0f70 msgr2=0x7f2f68160980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/3958419150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f680c0f70 0x7f2f68160980 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2f68065040 tx=0x7f2f60009840 comp rx=0 tx=0).stop 2023-10-10T18:45:38.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f6ca18700 1 -- 172.21.15.175:0/3958419150 shutdown_connections 2023-10-10T18:45:38.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/3958419150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f680c0f70 0x7f2f68160980 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/3958419150 >> v2:172.21.15.191:3300/0 conn(0x7f2f681569a0 0x7f2f6806d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f6ca18700 1 -- 172.21.15.175:0/3958419150 >> 172.21.15.175:0/3958419150 conn(0x7f2f6806b310 msgr2=0x7f2f6806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:38.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f6ca18700 1 -- 172.21.15.175:0/3958419150 shutdown_connections 2023-10-10T18:45:38.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.173+0000 7f2f6ca18700 1 -- 172.21.15.175:0/3958419150 wait complete. 2023-10-10T18:45:38.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f6ca18700 1 Processor -- start 2023-10-10T18:45:38.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f6ca18700 1 -- start start 2023-10-10T18:45:38.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f6ca18700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f6806cf90 0x7f2f68155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f6ca18700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f680c0f70 0x7f2f6814ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f6ca18700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f681569a0 0x7f2f6814b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f6ca18700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6816ab20 con 0x7f2f681569a0 2023-10-10T18:45:38.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f6ca18700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6816a5c0 con 0x7f2f6806cf90 2023-10-10T18:45:38.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f6ca18700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f680689d0 con 0x7f2f680c0f70 2023-10-10T18:45:38.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f67fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f681569a0 0x7f2f6814b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f67fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f681569a0 0x7f2f6814b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51680/0 (socket says 172.21.15.175:51680) 2023-10-10T18:45:38.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.175+0000 7f2f67fff700 1 -- 172.21.15.175:0/2472775231 learned_addr learned my addr 172.21.15.175:0/2472775231 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:38.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f677fe700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f6806cf90 0x7f2f68155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f66ffd700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f680c0f70 0x7f2f6814ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f67fff700 1 -- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f680c0f70 msgr2=0x7f2f6814ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f67fff700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f680c0f70 0x7f2f6814ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f67fff700 1 -- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f6806cf90 msgr2=0x7f2f68155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f67fff700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f6806cf90 0x7f2f68155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f67fff700 1 -- 172.21.15.175:0/2472775231 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60009ae0 con 0x7f2f681569a0 2023-10-10T18:45:38.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f66ffd700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f680c0f70 0x7f2f6814ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:38.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f67fff700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f681569a0 0x7f2f6814b250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2f6806e730 tx=0x7f2f60018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f4f7fe700 1 -- 172.21.15.175:0/2472775231 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60008e10 con 0x7f2f681569a0 2023-10-10T18:45:38.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f4f7fe700 1 -- 172.21.15.175:0/2472775231 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f60026460 con 0x7f2f681569a0 2023-10-10T18:45:38.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6806a620 con 0x7f2f681569a0 2023-10-10T18:45:38.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f4f7fe700 1 -- 172.21.15.175:0/2472775231 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60026820 con 0x7f2f681569a0 2023-10-10T18:45:38.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.176+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6806a8c0 con 0x7f2f681569a0 2023-10-10T18:45:38.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.177+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6806a480 con 0x7f2f681569a0 2023-10-10T18:45:38.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.177+0000 7f2f4f7fe700 1 -- 172.21.15.175:0/2472775231 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2f60003040 con 0x7f2f681569a0 2023-10-10T18:45:38.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.178+0000 7f2f4f7fe700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f54066f30 0x7f2f540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.178+0000 7f2f4f7fe700 1 -- 172.21.15.175:0/2472775231 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2f60086590 con 0x7f2f681569a0 2023-10-10T18:45:38.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.178+0000 7f2f677fe700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f54066f30 0x7f2f540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.178+0000 7f2f677fe700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f54066f30 0x7f2f540693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f580012a0 tx=0x7f2f58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.182+0000 7f2f4f7fe700 1 -- 172.21.15.175:0/2472775231 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f6806a480 con 0x7f2f681569a0 2023-10-10T18:45:38.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.364+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f6806a480 con 0x7f2f681569a0 2023-10-10T18:45:38.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.365+0000 7f2f4f7fe700 1 -- 172.21.15.175:0/2472775231 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7f2f6806a480 con 0x7f2f681569a0 2023-10-10T18:45:38.365 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:38.366 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:38.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f54066f30 msgr2=0x7f2f540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f54066f30 0x7f2f540693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f580012a0 tx=0x7f2f58009040 comp rx=0 tx=0).stop 2023-10-10T18:45:38.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f681569a0 msgr2=0x7f2f6814b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f681569a0 0x7f2f6814b250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2f6806e730 tx=0x7f2f60018000 comp rx=0 tx=0).stop 2023-10-10T18:45:38.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 shutdown_connections 2023-10-10T18:45:38.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f54066f30 0x7f2f540693e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f681569a0 0x7f2f6814b250 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f6806cf90 0x7f2f68155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 --2- 172.21.15.175:0/2472775231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f680c0f70 0x7f2f6814ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 >> 172.21.15.175:0/2472775231 conn(0x7f2f6806b310 msgr2=0x7f2f680c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:38.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.367+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 shutdown_connections 2023-10-10T18:45:38.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.368+0000 7f2f6ca18700 1 -- 172.21.15.175:0/2472775231 wait complete. 2023-10-10T18:45:38.379 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T18:45:05.161495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T18:45:08.855952+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T18:45:09.742056+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T18:45:38.380 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T18:45:38.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.498+0000 7f64a3fff700 1 Processor -- start 2023-10-10T18:45:38.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.498+0000 7f64a3fff700 1 -- start start 2023-10-10T18:45:38.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a3fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f64a4068630 0x7f64a40c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a3fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f64a40c7ae0 0x7f64a4155f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a3fff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a4156dd0 con 0x7f64a4068a00 2023-10-10T18:45:38.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a3fff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a40c7650 con 0x7f64a40c7ae0 2023-10-10T18:45:38.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a3fff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a4069730 con 0x7f64a4068630 2023-10-10T18:45:38.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a27fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f64a40c7ae0 0x7f64a4155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a27fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f64a40c7ae0 0x7f64a4155f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51696/0 (socket says 172.21.15.175:51696) 2023-10-10T18:45:38.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a27fc700 1 -- 172.21.15.175:0/1918772285 learned_addr learned my addr 172.21.15.175:0/1918772285 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:38.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a2ffd700 1 --2- 172.21.15.175:0/1918772285 >> v2:172.21.15.191:3300/0 conn(0x7f64a4068630 0x7f64a40c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a27fc700 1 -- 172.21.15.175:0/1918772285 >> v2:172.21.15.191:3300/0 conn(0x7f64a4068630 msgr2=0x7f64a40c39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:38.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a27fc700 1 --2- 172.21.15.175:0/1918772285 >> v2:172.21.15.191:3300/0 conn(0x7f64a4068630 0x7f64a40c39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a27fc700 1 -- 172.21.15.175:0/1918772285 >> v1:172.21.15.191:6789/0 conn(0x7f64a4068a00 legacy=0x7f64a40c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:38.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a27fc700 1 -- 172.21.15.175:0/1918772285 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a40c7f40 con 0x7f64a40c7ae0 2023-10-10T18:45:38.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a27fc700 1 --2- 172.21.15.175:0/1918772285 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a4155f10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64a4089ea0 tx=0x7f6498009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11df8b3f8ee012d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a1ffb700 1 -- 172.21.15.175:0/1918772285 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649800b820 con 0x7f64a40c7ae0 2023-10-10T18:45:38.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a1ffb700 1 -- 172.21.15.175:0/1918772285 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649800b980 con 0x7f64a40c7ae0 2023-10-10T18:45:38.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.499+0000 7f64a1ffb700 1 -- 172.21.15.175:0/1918772285 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649800bd20 con 0x7f64a40c7ae0 2023-10-10T18:45:38.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.500+0000 7f64a3fff700 1 -- 172.21.15.175:0/1918772285 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 msgr2=0x7f64a4155f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.500+0000 7f64a3fff700 1 --2- 172.21.15.175:0/1918772285 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a4155f10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64a4089ea0 tx=0x7f6498009250 comp rx=0 tx=0).stop 2023-10-10T18:45:38.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.500+0000 7f64a3fff700 1 -- 172.21.15.175:0/1918772285 shutdown_connections 2023-10-10T18:45:38.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.500+0000 7f64a3fff700 1 --2- 172.21.15.175:0/1918772285 >> v2:172.21.15.191:3300/0 conn(0x7f64a4068630 0x7f64a40c39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.500+0000 7f64a3fff700 1 --2- 172.21.15.175:0/1918772285 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a4155f10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.500+0000 7f64a3fff700 1 -- 172.21.15.175:0/1918772285 >> 172.21.15.175:0/1918772285 conn(0x7f64a4063230 msgr2=0x7f64a4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:38.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.500+0000 7f64a3fff700 1 -- 172.21.15.175:0/1918772285 shutdown_connections 2023-10-10T18:45:38.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.500+0000 7f64a3fff700 1 -- 172.21.15.175:0/1918772285 wait complete. 2023-10-10T18:45:38.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.501+0000 7f64a3fff700 1 Processor -- start 2023-10-10T18:45:38.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.501+0000 7f64a3fff700 1 -- start start 2023-10-10T18:45:38.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a3fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a4068630 0x7f64a40c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a3fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a4068a00 0x7f64a4152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a40cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a3fff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a40c3070 con 0x7f64a40c7ae0 2023-10-10T18:45:38.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a3fff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a4151ef0 con 0x7f64a4068a00 2023-10-10T18:45:38.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a3fff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a41518e0 con 0x7f64a4068630 2023-10-10T18:45:38.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a40cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a40cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51708/0 (socket says 172.21.15.175:51708) 2023-10-10T18:45:38.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 -- 172.21.15.175:0/2823224982 learned_addr learned my addr 172.21.15.175:0/2823224982 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:38.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a2ffd700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a4068630 0x7f64a40c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a27fc700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a4068a00 0x7f64a4152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 -- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a4068630 msgr2=0x7f64a40c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a4068630 0x7f64a40c0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 -- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a4068a00 msgr2=0x7f64a4152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a4068a00 0x7f64a4152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 -- 172.21.15.175:0/2823224982 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64980094f0 con 0x7f64a40c7ae0 2023-10-10T18:45:38.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f64a37fe700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a40cbcc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c5e0 tx=0x7f649c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.502+0000 7f648affd700 1 -- 172.21.15.175:0/2823224982 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c011590 con 0x7f64a40c7ae0 2023-10-10T18:45:38.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.503+0000 7f648affd700 1 -- 172.21.15.175:0/2823224982 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c011710 con 0x7f64a40c7ae0 2023-10-10T18:45:38.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.503+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a41669d0 con 0x7f64a40c7ae0 2023-10-10T18:45:38.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.503+0000 7f648affd700 1 -- 172.21.15.175:0/2823224982 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c011ad0 con 0x7f64a40c7ae0 2023-10-10T18:45:38.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.503+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a4166cb0 con 0x7f64a40c7ae0 2023-10-10T18:45:38.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.503+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a4156560 con 0x7f64a40c7ae0 2023-10-10T18:45:38.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.504+0000 7f648affd700 1 -- 172.21.15.175:0/2823224982 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f649c01c460 con 0x7f64a40c7ae0 2023-10-10T18:45:38.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.504+0000 7f648affd700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6490066f60 0x7f6490069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.504+0000 7f648affd700 1 -- 172.21.15.175:0/2823224982 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f649c084130 con 0x7f64a40c7ae0 2023-10-10T18:45:38.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.505+0000 7f64a2ffd700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6490066f60 0x7f6490069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.505+0000 7f64a2ffd700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6490066f60 0x7f6490069410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6494009fd0 tx=0x7f6494009880 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.509+0000 7f648affd700 1 -- 172.21.15.175:0/2823224982 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f649c0513a0 con 0x7f64a40c7ae0 2023-10-10T18:45:38.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.693+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64a4071150 con 0x7f64a40c7ae0 2023-10-10T18:45:38.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.693+0000 7f648affd700 1 -- 172.21.15.175:0/2823224982 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f649c0549c0 con 0x7f64a40c7ae0 2023-10-10T18:45:38.694 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 1 2023-10-10T18:45:38.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6490066f60 msgr2=0x7f6490069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6490066f60 0x7f6490069410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6494009fd0 tx=0x7f6494009880 comp rx=0 tx=0).stop 2023-10-10T18:45:38.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 msgr2=0x7f64a40cbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a40cbcc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c5e0 tx=0x7f649c007f60 comp rx=0 tx=0).stop 2023-10-10T18:45:38.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 shutdown_connections 2023-10-10T18:45:38.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6490066f60 0x7f6490069410 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64a40c7ae0 0x7f64a40cbcc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a4068630 0x7f64a40c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 --2- 172.21.15.175:0/2823224982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a4068a00 0x7f64a4152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 >> 172.21.15.175:0/2823224982 conn(0x7f64a4063230 msgr2=0x7f64a4158bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:38.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.695+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 shutdown_connections 2023-10-10T18:45:38.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.696+0000 7f64a3fff700 1 -- 172.21.15.175:0/2823224982 wait complete. 2023-10-10T18:45:38.707 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T18:45:38.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.825+0000 7f6f6c80c700 1 Processor -- start 2023-10-10T18:45:38.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.825+0000 7f6f6c80c700 1 -- start start 2023-10-10T18:45:38.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6c80c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6f680c71c0 0x7f6f680b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6c80c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f68067f30 0x7f6f68156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6c80c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f680b2320 con 0x7f6f68067b60 2023-10-10T18:45:38.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6c80c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f680b2840 con 0x7f6f680c71c0 2023-10-10T18:45:38.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6c80c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f680c91b0 con 0x7f6f68067f30 2023-10-10T18:45:38.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6659c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6f68067b60 0x7f6f6814ff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51344/0 (socket says 172.21.15.175:51344) 2023-10-10T18:45:38.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6659c700 1 -- 172.21.15.175:0/4162030713 learned_addr learned my addr 172.21.15.175:0/4162030713 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:38.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f65d9b700 1 --2- 172.21.15.175:0/4162030713 >> v2:172.21.15.191:3300/0 conn(0x7f6f680c71c0 0x7f6f680b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f66d9d700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f68067f30 0x7f6f68156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010383578 0 0) 0x7f6f58001490 con 0x7f6f68067b60 2023-10-10T18:45:38.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.826+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f50003570 con 0x7f6f68067b60 2023-10-10T18:45:38.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3473218839 0 0) 0x7f6f50003570 con 0x7f6f68067b60 2023-10-10T18:45:38.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f68067f30 msgr2=0x7f6f68156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:38.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f68067f30 0x7f6f68156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 >> v2:172.21.15.191:3300/0 conn(0x7f6f680c71c0 msgr2=0x7f6f680b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:38.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 --2- 172.21.15.175:0/4162030713 >> v2:172.21.15.191:3300/0 conn(0x7f6f680c71c0 0x7f6f680b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f680c8d20 con 0x7f6f68067b60 2023-10-10T18:45:38.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f6f58001980 con 0x7f6f68067b60 2023-10-10T18:45:38.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 >> v1:172.21.15.175:6789/0 conn(0x7f6f68067b60 legacy=0x7f6f6814ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f68067f30 0x7f6f50003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004010 0x7f6f50004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f50004d50 0x7f6f500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f680b2840 con 0x7f6f50004d50 2023-10-10T18:45:38.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f680c91b0 con 0x7f6f50004010 2023-10-10T18:45:38.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5000bab0 con 0x7f6f68067f30 2023-10-10T18:45:38.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f66d9d700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f50004d50 0x7f6f500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6659c700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f68067f30 0x7f6f50003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f65d9b700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004010 0x7f6f50004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f66d9d700 1 -- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f68067f30 msgr2=0x7f6f50003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:38.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f66d9d700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f68067f30 0x7f6f50003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f66d9d700 1 -- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004010 msgr2=0x7f6f50004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:38.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f66d9d700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004010 0x7f6f50004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f66d9d700 1 -- 172.21.15.175:0/4162030713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50003570 con 0x7f6f50004d50 2023-10-10T18:45:38.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f66d9d700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f50004d50 0x7f6f500051c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009c00 tx=0x7f6f600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8146f25e2c8a7952 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.827+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6000e710 con 0x7f6f50004d50 2023-10-10T18:45:38.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6000e870 con 0x7f6f50004d50 2023-10-10T18:45:38.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f6559a700 1 -- 172.21.15.175:0/4162030713 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6000ec30 con 0x7f6f50004d50 2023-10-10T18:45:38.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f6c80c700 1 -- 172.21.15.175:0/4162030713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f50004d50 msgr2=0x7f6f500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f6c80c700 1 --2- 172.21.15.175:0/4162030713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f50004d50 0x7f6f500051c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009c00 tx=0x7f6f600091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f66d9d700 1 -- 172.21.15.175:0/4162030713 reap_dead start 2023-10-10T18:45:38.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f6c80c700 1 -- 172.21.15.175:0/4162030713 shutdown_connections 2023-10-10T18:45:38.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f6c80c700 1 -- 172.21.15.175:0/4162030713 >> 172.21.15.175:0/4162030713 conn(0x7f6f68068700 msgr2=0x7f6f68068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:38.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f6c80c700 1 -- 172.21.15.175:0/4162030713 shutdown_connections 2023-10-10T18:45:38.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.828+0000 7f6f6c80c700 1 -- 172.21.15.175:0/4162030713 wait complete. 2023-10-10T18:45:38.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.829+0000 7f6f6c80c700 1 Processor -- start 2023-10-10T18:45:38.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.829+0000 7f6f6c80c700 1 -- start start 2023-10-10T18:45:38.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f6c80c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004d50 0x7f6f681579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f6c80c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f68157f30 0x7f6f68159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f6c80c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f681599d0 0x7f6f6814aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f6c80c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f680c6f30 con 0x7f6f68157f30 2023-10-10T18:45:38.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f6c80c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f68159000 con 0x7f6f50004d50 2023-10-10T18:45:38.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f6c80c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f68159e70 con 0x7f6f681599d0 2023-10-10T18:45:38.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f68157f30 0x7f6f68159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f68157f30 0x7f6f68159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51734/0 (socket says 172.21.15.175:51734) 2023-10-10T18:45:38.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 -- 172.21.15.175:0/1701459273 learned_addr learned my addr 172.21.15.175:0/1701459273 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:38.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f6659c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004d50 0x7f6f681579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f66d9d700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f681599d0 0x7f6f6814aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 -- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f681599d0 msgr2=0x7f6f6814aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f681599d0 0x7f6f6814aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 -- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004d50 msgr2=0x7f6f681579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:38.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004d50 0x7f6f681579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:38.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 -- 172.21.15.175:0/1701459273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f68152d00 con 0x7f6f68157f30 2023-10-10T18:45:38.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f65d9b700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f68157f30 0x7f6f68159490 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009fd0 tx=0x7f6f5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.830+0000 7f6f567fc700 1 -- 172.21.15.175:0/1701459273 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c00f940 con 0x7f6f68157f30 2023-10-10T18:45:38.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.831+0000 7f6f567fc700 1 -- 172.21.15.175:0/1701459273 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5c00faa0 con 0x7f6f68157f30 2023-10-10T18:45:38.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.831+0000 7f6f567fc700 1 -- 172.21.15.175:0/1701459273 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c00fe20 con 0x7f6f68157f30 2023-10-10T18:45:38.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.831+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f680c8480 con 0x7f6f68157f30 2023-10-10T18:45:38.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.831+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f681583e0 con 0x7f6f68157f30 2023-10-10T18:45:38.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.831+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f68067570 con 0x7f6f68157f30 2023-10-10T18:45:38.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.832+0000 7f6f567fc700 1 -- 172.21.15.175:0/1701459273 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6f5c014650 con 0x7f6f68157f30 2023-10-10T18:45:38.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.832+0000 7f6f567fc700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f48066f60 0x7f6f48069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:38.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.832+0000 7f6f567fc700 1 -- 172.21.15.175:0/1701459273 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6f5c018030 con 0x7f6f68157f30 2023-10-10T18:45:38.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.832+0000 7f6f6659c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f48066f60 0x7f6f48069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:38.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.833+0000 7f6f6659c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f48066f60 0x7f6f48069410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800a2a0 tx=0x7f6f58006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:38.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:38.836+0000 7f6f567fc700 1 -- 172.21.15.175:0/1701459273 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f5c0520b0 con 0x7f6f68157f30 2023-10-10T18:45:39.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.017+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f68159140 con 0x7f6f68157f30 2023-10-10T18:45:39.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.017+0000 7f6f567fc700 1 -- 172.21.15.175:0/1701459273 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f5c022020 con 0x7f6f68157f30 2023-10-10T18:45:39.018 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 32 2023-10-10T18:45:39.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f48066f60 msgr2=0x7f6f48069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f48066f60 0x7f6f48069410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800a2a0 tx=0x7f6f58006040 comp rx=0 tx=0).stop 2023-10-10T18:45:39.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f68157f30 msgr2=0x7f6f68159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f68157f30 0x7f6f68159490 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009fd0 tx=0x7f6f5c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 shutdown_connections 2023-10-10T18:45:39.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f48066f60 0x7f6f48069410 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f681599d0 0x7f6f6814aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f50004d50 0x7f6f681579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 --2- 172.21.15.175:0/1701459273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f68157f30 0x7f6f68159490 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 >> 172.21.15.175:0/1701459273 conn(0x7f6f68068700 msgr2=0x7f6f68151c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:39.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 shutdown_connections 2023-10-10T18:45:39.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.019+0000 7f6f6c80c700 1 -- 172.21.15.175:0/1701459273 wait complete. 2023-10-10T18:45:39.031 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T18:45:39.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.149+0000 7f818a96b700 1 Processor -- start 2023-10-10T18:45:39.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.149+0000 7f818a96b700 1 -- start start 2023-10-10T18:45:39.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.150+0000 7f818a96b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406c0b0 0x7f81841560a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.150+0000 7f818a96b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f818406dda0 0x7f8184153510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.150+0000 7f818a96b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8184153a50 con 0x7f818406c480 2023-10-10T18:45:39.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.150+0000 7f818a96b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8184152b20 con 0x7f818406dda0 2023-10-10T18:45:39.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.150+0000 7f818a96b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8184156750 con 0x7f818406c0b0 2023-10-10T18:45:39.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.150+0000 7f8189168700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f818406dda0 0x7f8184153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.150+0000 7f8189969700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406c0b0 0x7f81841560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189168700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f818406dda0 0x7f8184153510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:48744/0 (socket says 172.21.15.175:48744) 2023-10-10T18:45:39.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189969700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406c0b0 0x7f81841560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:45348/0 (socket says 172.21.15.175:45348) 2023-10-10T18:45:39.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189168700 1 -- 172.21.15.175:0/409991483 learned_addr learned my addr 172.21.15.175:0/409991483 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:39.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189168700 1 -- 172.21.15.175:0/409991483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406c0b0 msgr2=0x7f81841560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:39.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189168700 1 --2- 172.21.15.175:0/409991483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406c0b0 0x7f81841560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189168700 1 -- 172.21.15.175:0/409991483 >> v1:172.21.15.191:6789/0 conn(0x7f818406c480 legacy=0x7f818414fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189168700 1 -- 172.21.15.175:0/409991483 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8184156a30 con 0x7f818406dda0 2023-10-10T18:45:39.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189969700 1 --2- 172.21.15.175:0/409991483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406c0b0 0x7f81841560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:39.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8189168700 1 --2- 172.21.15.175:0/409991483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406dda0 0x7f8184153510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81740099b0 tx=0x7f8174009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe6c1fa2df7523a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8188967700 1 -- 172.21.15.175:0/409991483 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8174009ea0 con 0x7f818406dda0 2023-10-10T18:45:39.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8188967700 1 -- 172.21.15.175:0/409991483 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817400b860 con 0x7f818406dda0 2023-10-10T18:45:39.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.151+0000 7f8188967700 1 -- 172.21.15.175:0/409991483 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400bc00 con 0x7f818406dda0 2023-10-10T18:45:39.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.152+0000 7f818a96b700 1 -- 172.21.15.175:0/409991483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406dda0 msgr2=0x7f8184153510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.152+0000 7f818a96b700 1 --2- 172.21.15.175:0/409991483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406dda0 0x7f8184153510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81740099b0 tx=0x7f8174009250 comp rx=0 tx=0).stop 2023-10-10T18:45:39.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.152+0000 7f818a96b700 1 -- 172.21.15.175:0/409991483 shutdown_connections 2023-10-10T18:45:39.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.152+0000 7f818a96b700 1 --2- 172.21.15.175:0/409991483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406c0b0 0x7f81841560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.152+0000 7f818a96b700 1 --2- 172.21.15.175:0/409991483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406dda0 0x7f8184153510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.152+0000 7f818a96b700 1 -- 172.21.15.175:0/409991483 >> 172.21.15.175:0/409991483 conn(0x7f818406e360 msgr2=0x7f818406e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:39.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.152+0000 7f818a96b700 1 -- 172.21.15.175:0/409991483 shutdown_connections 2023-10-10T18:45:39.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.152+0000 7f818a96b700 1 -- 172.21.15.175:0/409991483 wait complete. 2023-10-10T18:45:39.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.153+0000 7f818a96b700 1 Processor -- start 2023-10-10T18:45:39.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.153+0000 7f818a96b700 1 -- start start 2023-10-10T18:45:39.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f818a96b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f818406c0b0 0x7f818414a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f818a96b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406c480 0x7f81840677f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f818a96b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406dda0 0x7f8184067d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f818a96b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81840be060 con 0x7f818406c0b0 2023-10-10T18:45:39.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f818a96b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81840bf2e0 con 0x7f818406c480 2023-10-10T18:45:39.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f818a96b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81840c8230 con 0x7f818406dda0 2023-10-10T18:45:39.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f818406c0b0 0x7f818414a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f818406c0b0 0x7f818414a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51750/0 (socket says 172.21.15.175:51750) 2023-10-10T18:45:39.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 -- 172.21.15.175:0/162784761 learned_addr learned my addr 172.21.15.175:0/162784761 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:39.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189168700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406c480 0x7f81840677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f818a16a700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406dda0 0x7f8184067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 -- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406dda0 msgr2=0x7f8184067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406dda0 0x7f8184067d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 -- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406c480 msgr2=0x7f81840677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406c480 0x7f81840677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 -- 172.21.15.175:0/162784761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8180009a20 con 0x7f818406c0b0 2023-10-10T18:45:39.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8189969700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f818406c0b0 0x7f818414a370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f818000c0c0 tx=0x7f818000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8179ffb700 1 -- 172.21.15.175:0/162784761 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818000cc00 con 0x7f818406c0b0 2023-10-10T18:45:39.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f8179ffb700 1 -- 172.21.15.175:0/162784761 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818000f040 con 0x7f818406c0b0 2023-10-10T18:45:39.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.154+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81740094f0 con 0x7f818406c0b0 2023-10-10T18:45:39.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.155+0000 7f8179ffb700 1 -- 172.21.15.175:0/162784761 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180014480 con 0x7f818406c0b0 2023-10-10T18:45:39.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.155+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81840bf5a0 con 0x7f818406c0b0 2023-10-10T18:45:39.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.156+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81840bcb00 con 0x7f818406c0b0 2023-10-10T18:45:39.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.156+0000 7f8179ffb700 1 -- 172.21.15.175:0/162784761 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f81800147f0 con 0x7f818406c0b0 2023-10-10T18:45:39.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.157+0000 7f8179ffb700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8168066f30 0x7f81680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.157+0000 7f8179ffb700 1 -- 172.21.15.175:0/162784761 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8180085480 con 0x7f818406c0b0 2023-10-10T18:45:39.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.157+0000 7f8189168700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8168066f30 0x7f81680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.157+0000 7f8189168700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8168066f30 0x7f81680693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81740099b0 tx=0x7f8174017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.161+0000 7f8179ffb700 1 -- 172.21.15.175:0/162784761 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8180052650 con 0x7f818406c0b0 2023-10-10T18:45:39.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.341+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f81840c5b00 con 0x7f818406c0b0 2023-10-10T18:45:39.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.342+0000 7f8179ffb700 1 -- 172.21.15.175:0/162784761 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8180055c70 con 0x7f818406c0b0 2023-10-10T18:45:39.342 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 64 2023-10-10T18:45:39.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8168066f30 msgr2=0x7f81680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8168066f30 0x7f81680693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81740099b0 tx=0x7f8174017000 comp rx=0 tx=0).stop 2023-10-10T18:45:39.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f818406c0b0 msgr2=0x7f818414a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f818406c0b0 0x7f818414a370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f818000c0c0 tx=0x7f818000b040 comp rx=0 tx=0).stop 2023-10-10T18:45:39.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 shutdown_connections 2023-10-10T18:45:39.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8168066f30 0x7f81680693e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f818406dda0 0x7f8184067d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f818406c0b0 0x7f818414a370 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 --2- 172.21.15.175:0/162784761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f818406c480 0x7f81840677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 >> 172.21.15.175:0/162784761 conn(0x7f818406e360 msgr2=0x7f8184151150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:39.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 shutdown_connections 2023-10-10T18:45:39.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.344+0000 7f818a96b700 1 -- 172.21.15.175:0/162784761 wait complete. 2023-10-10T18:45:39.356 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:39.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.471+0000 7f210ebc9700 1 Processor -- start 2023-10-10T18:45:39.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.471+0000 7f210ebc9700 1 -- start start 2023-10-10T18:45:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f210ebc9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f21080b2390 0x7f2108155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f210ebc9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21080aff60 0x7f21080b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f210ebc9700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21081574b0 con 0x7f21080b1fc0 2023-10-10T18:45:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f210ebc9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2108150770 con 0x7f21080b2390 2023-10-10T18:45:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f210ebc9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21080afad0 con 0x7f21080aff60 2023-10-10T18:45:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f210c965700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f21080b1fc0 0x7f2108068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51346/0 (socket says 172.21.15.175:51346) 2023-10-10T18:45:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f210c965700 1 -- 172.21.15.175:0/1797070051 learned_addr learned my addr 172.21.15.175:0/1797070051 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f210d166700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21080aff60 0x7f21080b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f2107fff700 1 --2- 172.21.15.175:0/1797070051 >> v2:172.21.15.191:3300/0 conn(0x7f21080b2390 0x7f2108155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489413440 0 0) 0x7f21081574b0 con 0x7f21080b1fc0 2023-10-10T18:45:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4003570 con 0x7f21080b1fc0 2023-10-10T18:45:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 374684515 0 0) 0x7f20f4003570 con 0x7f21080b1fc0 2023-10-10T18:45:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21080aff60 msgr2=0x7f21080b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21080aff60 0x7f21080b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 >> v2:172.21.15.191:3300/0 conn(0x7f21080b2390 msgr2=0x7f2108155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 --2- 172.21.15.175:0/1797070051 >> v2:172.21.15.191:3300/0 conn(0x7f21080b2390 0x7f2108155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21080af4c0 con 0x7f21080b1fc0 2023-10-10T18:45:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f21000017f0 con 0x7f21080b1fc0 2023-10-10T18:45:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 >> v1:172.21.15.175:6789/0 conn(0x7f21080b1fc0 legacy=0x7f2108068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20f4003a60 0x7f20f4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21080aff60 0x7f20f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20f4004d50 0x7f20f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2108150770 con 0x7f20f4004d50 2023-10-10T18:45:39.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21080afad0 con 0x7f20f4003a60 2023-10-10T18:45:39.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.472+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f400ba60 con 0x7f21080aff60 2023-10-10T18:45:39.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f210c965700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20f4003a60 0x7f20f4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f210d166700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20f4004d50 0x7f20f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f2107fff700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21080aff60 0x7f20f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f210d166700 1 -- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21080aff60 msgr2=0x7f20f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:39.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f210d166700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21080aff60 0x7f20f4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f210d166700 1 -- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20f4003a60 msgr2=0x7f20f4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:39.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f210d166700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20f4003a60 0x7f20f4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f210d166700 1 -- 172.21.15.175:0/1797070051 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f4003570 con 0x7f20f4004d50 2023-10-10T18:45:39.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f210d166700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20f4004d50 0x7f20f40051c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f20f0009c00 tx=0x7f20f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc8803b69953e9b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f000e6b0 con 0x7f20f4004d50 2023-10-10T18:45:39.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f000e810 con 0x7f20f4004d50 2023-10-10T18:45:39.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.473+0000 7f21077fe700 1 -- 172.21.15.175:0/1797070051 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f000ebd0 con 0x7f20f4004d50 2023-10-10T18:45:39.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.474+0000 7f210ebc9700 1 -- 172.21.15.175:0/1797070051 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20f4004d50 msgr2=0x7f20f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.474+0000 7f210ebc9700 1 --2- 172.21.15.175:0/1797070051 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20f4004d50 0x7f20f40051c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f20f0009c00 tx=0x7f20f00091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.474+0000 7f210d166700 1 -- 172.21.15.175:0/1797070051 reap_dead start 2023-10-10T18:45:39.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.474+0000 7f210ebc9700 1 -- 172.21.15.175:0/1797070051 shutdown_connections 2023-10-10T18:45:39.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.474+0000 7f210ebc9700 1 -- 172.21.15.175:0/1797070051 >> 172.21.15.175:0/1797070051 conn(0x7f2108063230 msgr2=0x7f2108063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:39.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.474+0000 7f210ebc9700 1 -- 172.21.15.175:0/1797070051 shutdown_connections 2023-10-10T18:45:39.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.474+0000 7f210ebc9700 1 -- 172.21.15.175:0/1797070051 wait complete. 2023-10-10T18:45:39.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.475+0000 7f210ebc9700 1 Processor -- start 2023-10-10T18:45:39.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.475+0000 7f210ebc9700 1 -- start start 2023-10-10T18:45:39.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.475+0000 7f210ebc9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f210814e710 0x7f2108158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.475+0000 7f210ebc9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f210814eae0 0x7f2108159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.475+0000 7f210ebc9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2108157180 0x7f21081517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.475+0000 7f210ebc9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2108168c30 con 0x7f210814eae0 2023-10-10T18:45:39.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.475+0000 7f210ebc9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21081697b0 con 0x7f210814e710 2023-10-10T18:45:39.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.475+0000 7f210ebc9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21080af030 con 0x7f2108157180 2023-10-10T18:45:39.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f210814eae0 0x7f2108159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f210814eae0 0x7f2108159380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51774/0 (socket says 172.21.15.175:51774) 2023-10-10T18:45:39.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 -- 172.21.15.175:0/723372514 learned_addr learned my addr 172.21.15.175:0/723372514 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:39.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f210d166700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2108157180 0x7f21081517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f210c965700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f210814e710 0x7f2108158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 -- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2108157180 msgr2=0x7f21081517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2108157180 0x7f21081517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 -- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f210814e710 msgr2=0x7f2108158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f210814e710 0x7f2108158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 -- 172.21.15.175:0/723372514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21081576c0 con 0x7f210814eae0 2023-10-10T18:45:39.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f210c965700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f210814e710 0x7f2108158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:39.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f210d166700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2108157180 0x7f21081517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:39.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2107fff700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f210814eae0 0x7f2108159380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f20f8011b10 tx=0x7f20f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2104ff9700 1 -- 172.21.15.175:0/723372514 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f800f990 con 0x7f210814eae0 2023-10-10T18:45:39.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2104ff9700 1 -- 172.21.15.175:0/723372514 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f800faf0 con 0x7f210814eae0 2023-10-10T18:45:39.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2108168e80 con 0x7f210814eae0 2023-10-10T18:45:39.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f2104ff9700 1 -- 172.21.15.175:0/723372514 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f800fe70 con 0x7f210814eae0 2023-10-10T18:45:39.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.476+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2108169440 con 0x7f210814eae0 2023-10-10T18:45:39.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.477+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2108169600 con 0x7f210814eae0 2023-10-10T18:45:39.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.477+0000 7f2104ff9700 1 -- 172.21.15.175:0/723372514 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f20f8014650 con 0x7f210814eae0 2023-10-10T18:45:39.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.478+0000 7f2104ff9700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f20e8066f30 0x7f20e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.478+0000 7f2104ff9700 1 -- 172.21.15.175:0/723372514 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f20f8018030 con 0x7f210814eae0 2023-10-10T18:45:39.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.478+0000 7f210c965700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f20e8066f30 0x7f20e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.478+0000 7f210c965700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f20e8066f30 0x7f20e80693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2100001580 tx=0x7f2100006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.482+0000 7f2104ff9700 1 -- 172.21.15.175:0/723372514 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f20f8055ca0 con 0x7f210814eae0 2023-10-10T18:45:39.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.707+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2108067570 con 0x7f210814eae0 2023-10-10T18:45:39.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.708+0000 7f2104ff9700 1 -- 172.21.15.175:0/723372514 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6177 (secure 0 0 0) 0x7f20f8022090 con 0x7f210814eae0 2023-10-10T18:45:39.708 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:39.708 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:45:35.538717+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:45:39.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f20e8066f30 msgr2=0x7f20e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f20e8066f30 0x7f20e80693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2100001580 tx=0x7f2100006040 comp rx=0 tx=0).stop 2023-10-10T18:45:39.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f210814eae0 msgr2=0x7f2108159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f210814eae0 0x7f2108159380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f20f8011b10 tx=0x7f20f80091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 shutdown_connections 2023-10-10T18:45:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f20e8066f30 0x7f20e80693e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2108157180 0x7f21081517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f210814e710 0x7f2108158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 --2- 172.21.15.175:0/723372514 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f210814eae0 0x7f2108159380 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 >> 172.21.15.175:0/723372514 conn(0x7f2108063230 msgr2=0x7f210815c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 shutdown_connections 2023-10-10T18:45:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.710+0000 7f210ebc9700 1 -- 172.21.15.175:0/723372514 wait complete. 2023-10-10T18:45:39.713 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 16 2023-10-10T18:45:39.722 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:39.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.836+0000 7f9bac227700 1 Processor -- start 2023-10-10T18:45:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.836+0000 7f9bac227700 1 -- start start 2023-10-10T18:45:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9bac227700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9ba414ffc0 0x7f9ba4159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9bac227700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9ba40681e0 0x7f9ba40c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9bac227700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba4156800 con 0x7f9ba414fbf0 2023-10-10T18:45:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9bac227700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba40c59e0 con 0x7f9ba40681e0 2023-10-10T18:45:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9bac227700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba415a2b0 con 0x7f9ba414ffc0 2023-10-10T18:45:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9baa7c4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9ba40681e0 0x7f9ba40c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9baa7c4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9ba40681e0 0x7f9ba40c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51782/0 (socket says 172.21.15.175:51782) 2023-10-10T18:45:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9baa7c4700 1 -- 172.21.15.175:0/2567113378 learned_addr learned my addr 172.21.15.175:0/2567113378 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.837+0000 7f9ba97c2700 1 --2- 172.21.15.175:0/2567113378 >> v2:172.21.15.191:3300/0 conn(0x7f9ba414ffc0 0x7f9ba4159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9baa7c4700 1 -- 172.21.15.175:0/2567113378 >> v2:172.21.15.191:3300/0 conn(0x7f9ba414ffc0 msgr2=0x7f9ba4159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9baa7c4700 1 --2- 172.21.15.175:0/2567113378 >> v2:172.21.15.191:3300/0 conn(0x7f9ba414ffc0 0x7f9ba4159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9baa7c4700 1 -- 172.21.15.175:0/2567113378 >> v1:172.21.15.191:6789/0 conn(0x7f9ba414fbf0 legacy=0x7f9ba4063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9baa7c4700 1 -- 172.21.15.175:0/2567113378 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba4063e30 con 0x7f9ba40681e0 2023-10-10T18:45:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9ba97c2700 1 --2- 172.21.15.175:0/2567113378 >> v2:172.21.15.191:3300/0 conn(0x7f9ba414ffc0 0x7f9ba4159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9baa7c4700 1 --2- 172.21.15.175:0/2567113378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40681e0 0x7f9ba40c5fb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4089b70 tx=0x7f9b9c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47c7f39211a5bec server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9ba8fc1700 1 -- 172.21.15.175:0/2567113378 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c00e070 con 0x7f9ba40681e0 2023-10-10T18:45:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9ba8fc1700 1 -- 172.21.15.175:0/2567113378 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c00ba00 con 0x7f9ba40681e0 2023-10-10T18:45:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9ba8fc1700 1 -- 172.21.15.175:0/2567113378 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0092e0 con 0x7f9ba40681e0 2023-10-10T18:45:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9bac227700 1 -- 172.21.15.175:0/2567113378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40681e0 msgr2=0x7f9ba40c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9bac227700 1 --2- 172.21.15.175:0/2567113378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40681e0 0x7f9ba40c5fb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4089b70 tx=0x7f9b9c009840 comp rx=0 tx=0).stop 2023-10-10T18:45:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9bac227700 1 -- 172.21.15.175:0/2567113378 shutdown_connections 2023-10-10T18:45:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9bac227700 1 --2- 172.21.15.175:0/2567113378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40681e0 0x7f9ba40c5fb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9bac227700 1 --2- 172.21.15.175:0/2567113378 >> v2:172.21.15.191:3300/0 conn(0x7f9ba414ffc0 0x7f9ba4159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9bac227700 1 -- 172.21.15.175:0/2567113378 >> 172.21.15.175:0/2567113378 conn(0x7f9ba40cc760 msgr2=0x7f9ba40ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:39.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.838+0000 7f9bac227700 1 -- 172.21.15.175:0/2567113378 shutdown_connections 2023-10-10T18:45:39.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.839+0000 7f9bac227700 1 -- 172.21.15.175:0/2567113378 wait complete. 2023-10-10T18:45:39.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.839+0000 7f9bac227700 1 Processor -- start 2023-10-10T18:45:39.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9bac227700 1 -- start start 2023-10-10T18:45:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9bac227700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40681e0 0x7f9ba40b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9bac227700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba414fbf0 0x7f9ba4151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9bac227700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba414ffc0 0x7f9ba41515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9bac227700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba4166c50 con 0x7f9ba414ffc0 2023-10-10T18:45:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9bac227700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba4150640 con 0x7f9ba414fbf0 2023-10-10T18:45:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9bac227700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba4150c00 con 0x7f9ba40681e0 2023-10-10T18:45:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9baa7c4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba414ffc0 0x7f9ba41515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9baa7c4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba414ffc0 0x7f9ba41515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51796/0 (socket says 172.21.15.175:51796) 2023-10-10T18:45:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9baa7c4700 1 -- 172.21.15.175:0/2280929112 learned_addr learned my addr 172.21.15.175:0/2280929112 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9ba9fc3700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40681e0 0x7f9ba40b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.840+0000 7f9ba97c2700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba414fbf0 0x7f9ba4151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9baa7c4700 1 -- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40681e0 msgr2=0x7f9ba40b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9baa7c4700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40681e0 0x7f9ba40b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9baa7c4700 1 -- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba414fbf0 msgr2=0x7f9ba4151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9baa7c4700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba414fbf0 0x7f9ba4151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9baa7c4700 1 -- 172.21.15.175:0/2280929112 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c009ae0 con 0x7f9ba414ffc0 2023-10-10T18:45:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9baa7c4700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba414ffc0 0x7f9ba41515d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c012970 tx=0x7f9b9c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9b99ffb700 1 -- 172.21.15.175:0/2280929112 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0032f0 con 0x7f9ba414ffc0 2023-10-10T18:45:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9b99ffb700 1 -- 172.21.15.175:0/2280929112 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c012d60 con 0x7f9ba414ffc0 2023-10-10T18:45:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba4152590 con 0x7f9ba414ffc0 2023-10-10T18:45:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9b99ffb700 1 -- 172.21.15.175:0/2280929112 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c01d650 con 0x7f9ba414ffc0 2023-10-10T18:45:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.841+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba4150890 con 0x7f9ba414ffc0 2023-10-10T18:45:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.842+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba4067570 con 0x7f9ba414ffc0 2023-10-10T18:45:39.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.842+0000 7f9b99ffb700 1 -- 172.21.15.175:0/2280929112 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9b9c01da40 con 0x7f9ba414ffc0 2023-10-10T18:45:39.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.843+0000 7f9b99ffb700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b88066f30 0x7f9b880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:39.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.843+0000 7f9b99ffb700 1 -- 172.21.15.175:0/2280929112 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f9b9c085d70 con 0x7f9ba414ffc0 2023-10-10T18:45:39.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.843+0000 7f9ba9fc3700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b88066f30 0x7f9b880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:39.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.844+0000 7f9ba9fc3700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b88066f30 0x7f9b880693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009b40 tx=0x7f9ba00090f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:39.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:39.847+0000 7f9b99ffb700 1 -- 172.21.15.175:0/2280929112 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b9c052fe0 con 0x7f9ba414ffc0 2023-10-10T18:45:40.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.029+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ba4150d40 con 0x7f9ba414ffc0 2023-10-10T18:45:40.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.030+0000 7f9b99ffb700 1 -- 172.21.15.175:0/2280929112 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7f9b9c009940 con 0x7f9ba414ffc0 2023-10-10T18:45:40.030 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:40.030 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:40.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b88066f30 msgr2=0x7f9b880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:40.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b88066f30 0x7f9b880693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009b40 tx=0x7f9ba00090f0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba414ffc0 msgr2=0x7f9ba41515d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:40.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba414ffc0 0x7f9ba41515d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c012970 tx=0x7f9b9c017000 comp rx=0 tx=0).stop 2023-10-10T18:45:40.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 shutdown_connections 2023-10-10T18:45:40.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b88066f30 0x7f9b880693e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba414ffc0 0x7f9ba41515d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40681e0 0x7f9ba40b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 --2- 172.21.15.175:0/2280929112 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba414fbf0 0x7f9ba4151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.032+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 >> 172.21.15.175:0/2280929112 conn(0x7f9ba40cc760 msgr2=0x7f9ba4153fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:40.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.033+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 shutdown_connections 2023-10-10T18:45:40.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.033+0000 7f9bac227700 1 -- 172.21.15.175:0/2280929112 wait complete. 2023-10-10T18:45:40.045 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:40.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.159+0000 7fdeb3fff700 1 Processor -- start 2023-10-10T18:45:40.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.159+0000 7fdeb3fff700 1 -- start start 2023-10-10T18:45:40.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb3fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdeb414fd90 0x7fdeb4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:40.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb3fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb408a100 0x7fdeb40c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:40.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb3fff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb4067fa0 con 0x7fdeb40cd040 2023-10-10T18:45:40.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb3fff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb40c5600 con 0x7fdeb414fd90 2023-10-10T18:45:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb3fff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb4156b70 con 0x7fdeb408a100 2023-10-10T18:45:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb27fc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb408a100 0x7fdeb40c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb2ffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdeb414fd90 0x7fdeb4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb27fc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb408a100 0x7fdeb40c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:45410/0 (socket says 172.21.15.175:45410) 2023-10-10T18:45:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.160+0000 7fdeb27fc700 1 -- 172.21.15.175:0/1032354602 learned_addr learned my addr 172.21.15.175:0/1032354602 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:40.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb2ffd700 1 -- 172.21.15.175:0/1032354602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb408a100 msgr2=0x7fdeb40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:40.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb2ffd700 1 --2- 172.21.15.175:0/1032354602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb408a100 0x7fdeb40c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb2ffd700 1 -- 172.21.15.175:0/1032354602 >> v1:172.21.15.191:6789/0 conn(0x7fdeb40cd040 legacy=0x7fdeb40684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:40.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb2ffd700 1 -- 172.21.15.175:0/1032354602 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb4150160 con 0x7fdeb414fd90 2023-10-10T18:45:40.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb2ffd700 1 --2- 172.21.15.175:0/1032354602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 0x7fdeb4159c30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009e60 tx=0x7fdea4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=660a5df84d16c299 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:40.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb1ffb700 1 -- 172.21.15.175:0/1032354602 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea400e070 con 0x7fdeb414fd90 2023-10-10T18:45:40.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb1ffb700 1 -- 172.21.15.175:0/1032354602 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea400bb60 con 0x7fdeb414fd90 2023-10-10T18:45:40.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb1ffb700 1 -- 172.21.15.175:0/1032354602 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4012430 con 0x7fdeb414fd90 2023-10-10T18:45:40.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb3fff700 1 -- 172.21.15.175:0/1032354602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 msgr2=0x7fdeb4159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:40.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.161+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/1032354602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 0x7fdeb4159c30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009e60 tx=0x7fdea4009780 comp rx=0 tx=0).stop 2023-10-10T18:45:40.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.162+0000 7fdeb3fff700 1 -- 172.21.15.175:0/1032354602 shutdown_connections 2023-10-10T18:45:40.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.162+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/1032354602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 0x7fdeb4159c30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.162+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/1032354602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb408a100 0x7fdeb40c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.162+0000 7fdeb3fff700 1 -- 172.21.15.175:0/1032354602 >> 172.21.15.175:0/1032354602 conn(0x7fdeb4063d00 msgr2=0x7fdeb40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:40.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.162+0000 7fdeb3fff700 1 -- 172.21.15.175:0/1032354602 shutdown_connections 2023-10-10T18:45:40.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.162+0000 7fdeb3fff700 1 -- 172.21.15.175:0/1032354602 wait complete. 2023-10-10T18:45:40.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.163+0000 7fdeb3fff700 1 Processor -- start 2023-10-10T18:45:40.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.163+0000 7fdeb3fff700 1 -- start start 2023-10-10T18:45:40.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb408a100 0x7fdeb4152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:40.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb3fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb40cd040 0x7fdeb4152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:40.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb3fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 0x7fdeb40c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:40.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb3fff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb40a7c10 con 0x7fdeb408a100 2023-10-10T18:45:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb3fff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb40a8790 con 0x7fdeb414fd90 2023-10-10T18:45:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb3fff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb4166d30 con 0x7fdeb40cd040 2023-10-10T18:45:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb408a100 0x7fdeb4152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb408a100 0x7fdeb4152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51800/0 (socket says 172.21.15.175:51800) 2023-10-10T18:45:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 -- 172.21.15.175:0/804228427 learned_addr learned my addr 172.21.15.175:0/804228427 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb27fc700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb40cd040 0x7fdeb4152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:40.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb37fe700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 0x7fdeb40c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:40.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 -- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb40cd040 msgr2=0x7fdeb4152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:40.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb40cd040 0x7fdeb4152a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 -- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 msgr2=0x7fdeb40c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:40.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 0x7fdeb40c6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 -- 172.21.15.175:0/804228427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea4009a20 con 0x7fdeb408a100 2023-10-10T18:45:40.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb37fe700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 0x7fdeb40c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:40.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb2ffd700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb408a100 0x7fdeb4152510 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdea400b6c0 tx=0x7fdea4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:40.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeaaffd700 1 -- 172.21.15.175:0/804228427 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4012e50 con 0x7fdeb408a100 2023-10-10T18:45:40.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeaaffd700 1 -- 172.21.15.175:0/804228427 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea4027470 con 0x7fdeb408a100 2023-10-10T18:45:40.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb40a7e60 con 0x7fdeb408a100 2023-10-10T18:45:40.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeaaffd700 1 -- 172.21.15.175:0/804228427 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4027830 con 0x7fdeb408a100 2023-10-10T18:45:40.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.164+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb40a8420 con 0x7fdeb408a100 2023-10-10T18:45:40.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.165+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb4067570 con 0x7fdeb408a100 2023-10-10T18:45:40.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.166+0000 7fdeaaffd700 1 -- 172.21.15.175:0/804228427 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdea4024070 con 0x7fdeb408a100 2023-10-10T18:45:40.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.166+0000 7fdeaaffd700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde94066f30 0x7fde940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:40.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.166+0000 7fdeaaffd700 1 -- 172.21.15.175:0/804228427 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fdea4086ba0 con 0x7fdeb408a100 2023-10-10T18:45:40.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.166+0000 7fdeb27fc700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde94066f30 0x7fde940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:40.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.166+0000 7fdeb27fc700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde94066f30 0x7fde940693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdeb414e880 tx=0x7fde9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:40.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.170+0000 7fdeaaffd700 1 -- 172.21.15.175:0/804228427 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdea4053e10 con 0x7fdeb408a100 2023-10-10T18:45:40.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.394+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdeb4155f80 con 0x7fdeb408a100 2023-10-10T18:45:40.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.395+0000 7fdeaaffd700 1 -- 172.21.15.175:0/804228427 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6177 (secure 0 0 0) 0x7fdea4057430 con 0x7fdeb408a100 2023-10-10T18:45:40.395 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:40.395 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:45:35.538717+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:45:40.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde94066f30 msgr2=0x7fde940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:40.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde94066f30 0x7fde940693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdeb414e880 tx=0x7fde9c008040 comp rx=0 tx=0).stop 2023-10-10T18:45:40.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb408a100 msgr2=0x7fdeb4152510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb408a100 0x7fdeb4152510 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdea400b6c0 tx=0x7fdea4019000 comp rx=0 tx=0).stop 2023-10-10T18:45:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 shutdown_connections 2023-10-10T18:45:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde94066f30 0x7fde940693e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb414fd90 0x7fdeb40c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb408a100 0x7fdeb4152510 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 --2- 172.21.15.175:0/804228427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb40cd040 0x7fdeb4152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 >> 172.21.15.175:0/804228427 conn(0x7fdeb4063d00 msgr2=0x7fdeb40cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 shutdown_connections 2023-10-10T18:45:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:40.397+0000 7fdeb3fff700 1 -- 172.21.15.175:0/804228427 wait complete. 2023-10-10T18:45:40.400 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 16 2023-10-10T18:45:41.012 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-10T18:45:10.774833+0000', 'modified': '2023-10-10T18:45:35.538717+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4263, 'mds_1': 4251, 'mds_2': 4250, 'mds_3': 4248, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.191:6836/3007475551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6835', 'nonce': 3007475551}, {'type': 'v1', 'addr': '172.21.15.191:6836', 'nonce': 3007475551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.175:6836/647425682', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6833', 'nonce': 647425682}, {'type': 'v1', 'addr': '172.21.15.175:6836', 'nonce': 647425682}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4250': {'gid': 4250, 'name': 'f', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.191:6839/246221790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 246221790}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 246221790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4251': {'gid': 4251, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.175:6837/2966961140', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6835', 'nonce': 2966961140}, {'type': 'v1', 'addr': '172.21.15.175:6837', 'nonce': 2966961140}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4263': {'gid': 4263, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6837/3870390806', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3870390806}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 3870390806}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-10T18:45:41.013 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4247:up:creating 2023-10-10T18:45:42.014 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:42.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.142+0000 7f220f0ca700 1 Processor -- start 2023-10-10T18:45:42.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.142+0000 7f220f0ca700 1 -- start start 2023-10-10T18:45:42.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f220f0ca700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f220814e160 0x7f22080c1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:42.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f220f0ca700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f22081577b0 0x7f220806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:42.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f220f0ca700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220806ac70 con 0x7f22081573e0 2023-10-10T18:45:42.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f220f0ca700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220806af50 con 0x7f220814e160 2023-10-10T18:45:42.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f220f0ca700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22080c3e70 con 0x7f22081577b0 2023-10-10T18:45:42.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f2207fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f220814e160 0x7f22080c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:42.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f220ce66700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f22081573e0 0x7f220814da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51360/0 (socket says 172.21.15.175:51360) 2023-10-10T18:45:42.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f220ce66700 1 -- 172.21.15.175:0/3674011698 learned_addr learned my addr 172.21.15.175:0/3674011698 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:42.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f220d667700 1 --2- 172.21.15.175:0/3674011698 >> v2:172.21.15.191:3300/0 conn(0x7f22081577b0 0x7f220806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:42.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f22077fe700 1 -- 172.21.15.175:0/3674011698 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138943779 0 0) 0x7f220806ac70 con 0x7f22081573e0 2023-10-10T18:45:42.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f22077fe700 1 -- 172.21.15.175:0/3674011698 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f8003570 con 0x7f22081573e0 2023-10-10T18:45:42.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f2207fff700 1 -- 172.21.15.175:0/3674011698 >> v2:172.21.15.191:3300/0 conn(0x7f22081577b0 msgr2=0x7f220806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:42.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f2207fff700 1 --2- 172.21.15.175:0/3674011698 >> v2:172.21.15.191:3300/0 conn(0x7f22081577b0 0x7f220806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f2207fff700 1 -- 172.21.15.175:0/3674011698 >> v1:172.21.15.175:6789/0 conn(0x7f22081573e0 legacy=0x7f220814da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:42.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f2207fff700 1 -- 172.21.15.175:0/3674011698 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22080c4150 con 0x7f220814e160 2023-10-10T18:45:42.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f2207fff700 1 --2- 172.21.15.175:0/3674011698 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f220814e160 0x7f22080c1930 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22080c17d0 tx=0x7f21f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf4963a5d9720a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:42.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f22077fe700 1 -- 172.21.15.175:0/3674011698 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4011630 con 0x7f220814e160 2023-10-10T18:45:42.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f22077fe700 1 -- 172.21.15.175:0/3674011698 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f40117b0 con 0x7f220814e160 2023-10-10T18:45:42.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.143+0000 7f22077fe700 1 -- 172.21.15.175:0/3674011698 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4011b70 con 0x7f220814e160 2023-10-10T18:45:42.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.144+0000 7f220f0ca700 1 -- 172.21.15.175:0/3674011698 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f220814e160 msgr2=0x7f22080c1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:42.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.144+0000 7f220f0ca700 1 --2- 172.21.15.175:0/3674011698 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f220814e160 0x7f22080c1930 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22080c17d0 tx=0x7f21f40091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.144+0000 7f220f0ca700 1 -- 172.21.15.175:0/3674011698 shutdown_connections 2023-10-10T18:45:42.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.144+0000 7f220f0ca700 1 --2- 172.21.15.175:0/3674011698 >> v2:172.21.15.191:3300/0 conn(0x7f22081577b0 0x7f220806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.144+0000 7f220f0ca700 1 --2- 172.21.15.175:0/3674011698 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f220814e160 0x7f22080c1930 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.144+0000 7f220f0ca700 1 -- 172.21.15.175:0/3674011698 >> 172.21.15.175:0/3674011698 conn(0x7f2208063230 msgr2=0x7f2208063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:42.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.144+0000 7f220f0ca700 1 -- 172.21.15.175:0/3674011698 shutdown_connections 2023-10-10T18:45:42.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.144+0000 7f220f0ca700 1 -- 172.21.15.175:0/3674011698 wait complete. 2023-10-10T18:45:42.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.145+0000 7f220f0ca700 1 Processor -- start 2023-10-10T18:45:42.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.145+0000 7f220f0ca700 1 -- start start 2023-10-10T18:45:42.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f220f0ca700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f220814e160 0x7f22080c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:42.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f220f0ca700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22081573e0 0x7f2208167290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:42.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f220f0ca700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22081577b0 0x7f22080c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:42.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f220f0ca700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22080c87b0 con 0x7f22081573e0 2023-10-10T18:45:42.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f220f0ca700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22080c9400 con 0x7f220814e160 2023-10-10T18:45:42.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f220f0ca700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22080c8480 con 0x7f22081577b0 2023-10-10T18:45:42.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22081573e0 0x7f2208167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:42.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22081573e0 0x7f2208167290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51830/0 (socket says 172.21.15.175:51830) 2023-10-10T18:45:42.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 -- 172.21.15.175:0/2231615980 learned_addr learned my addr 172.21.15.175:0/2231615980 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:42.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 -- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22081577b0 msgr2=0x7f22080c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:42.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22081577b0 0x7f22080c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 -- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f220814e160 msgr2=0x7f22080c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:42.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f220814e160 0x7f22080c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 -- 172.21.15.175:0/2231615980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f4009420 con 0x7f22081573e0 2023-10-10T18:45:42.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.146+0000 7f2207fff700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22081573e0 0x7f2208167290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f21f4013000 tx=0x7f21f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:42.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.147+0000 7f2204ff9700 1 -- 172.21.15.175:0/2231615980 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f401f070 con 0x7f22081573e0 2023-10-10T18:45:42.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.147+0000 7f2204ff9700 1 -- 172.21.15.175:0/2231615980 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f4000c90 con 0x7f22081573e0 2023-10-10T18:45:42.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.147+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220814e970 con 0x7f22081573e0 2023-10-10T18:45:42.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.147+0000 7f2204ff9700 1 -- 172.21.15.175:0/2231615980 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4011b50 con 0x7f22081573e0 2023-10-10T18:45:42.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.147+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22080c9980 con 0x7f22081573e0 2023-10-10T18:45:42.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.147+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2208067570 con 0x7f22081573e0 2023-10-10T18:45:42.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.148+0000 7f2204ff9700 1 -- 172.21.15.175:0/2231615980 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f21f4023070 con 0x7f22081573e0 2023-10-10T18:45:42.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.148+0000 7f2204ff9700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f21ec066f60 0x7f21ec069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:42.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.148+0000 7f2204ff9700 1 -- 172.21.15.175:0/2231615980 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f21f40858c0 con 0x7f22081573e0 2023-10-10T18:45:42.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.148+0000 7f220ce66700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f21ec066f60 0x7f21ec069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:42.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.149+0000 7f220ce66700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f21ec066f60 0x7f21ec069410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0058d0 tx=0x7f21fc0079d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:42.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.152+0000 7f2204ff9700 1 -- 172.21.15.175:0/2231615980 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21f4052b30 con 0x7f22081573e0 2023-10-10T18:45:42.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.376+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22080c88f0 con 0x7f22081573e0 2023-10-10T18:45:42.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.376+0000 7f2204ff9700 1 -- 172.21.15.175:0/2231615980 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6176 (secure 0 0 0) 0x7f21f402a090 con 0x7f22081573e0 2023-10-10T18:45:42.376 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:42.377 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:45:41.554584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:45:42.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f21ec066f60 msgr2=0x7f21ec069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:42.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f21ec066f60 0x7f21ec069410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0058d0 tx=0x7f21fc0079d0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22081573e0 msgr2=0x7f2208167290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:42.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22081573e0 0x7f2208167290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f21f4013000 tx=0x7f21f4017000 comp rx=0 tx=0).stop 2023-10-10T18:45:42.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 shutdown_connections 2023-10-10T18:45:42.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f21ec066f60 0x7f21ec069410 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22081577b0 0x7f22080c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f220814e160 0x7f22080c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 --2- 172.21.15.175:0/2231615980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22081573e0 0x7f2208167290 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:42.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.378+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 >> 172.21.15.175:0/2231615980 conn(0x7f2208063230 msgr2=0x7f22080c4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:42.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.379+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 shutdown_connections 2023-10-10T18:45:42.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:42.379+0000 7f220f0ca700 1 -- 172.21.15.175:0/2231615980 wait complete. 2023-10-10T18:45:42.381 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 17 2023-10-10T18:45:42.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-10T18:45:10.774833+0000', 'modified': '2023-10-10T18:45:41.554584+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4263, 'mds_1': 4251, 'mds_2': 4250, 'mds_3': 4248, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.191:6836/3007475551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6835', 'nonce': 3007475551}, {'type': 'v1', 'addr': '172.21.15.191:6836', 'nonce': 3007475551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.175:6836/647425682', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6833', 'nonce': 647425682}, {'type': 'v1', 'addr': '172.21.15.175:6836', 'nonce': 647425682}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4250': {'gid': 4250, 'name': 'f', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.191:6839/246221790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 246221790}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 246221790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4251': {'gid': 4251, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.175:6837/2966961140', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6835', 'nonce': 2966961140}, {'type': 'v1', 'addr': '172.21.15.175:6837', 'nonce': 2966961140}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4263': {'gid': 4263, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6837/3870390806', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3870390806}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 3870390806}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-10T18:45:42.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-10-10T18:45:42.995 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-10T18:45:43.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.113+0000 7fc428a51700 1 Processor -- start 2023-10-10T18:45:43.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.113+0000 7fc428a51700 1 -- start start 2023-10-10T18:45:43.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc428a51700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424157960 0x7fc424155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc428a51700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc424067e10 0x7fc4241561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc428a51700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42406b7e0 con 0x7fc424067a40 2023-10-10T18:45:43.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc428a51700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42406a8f0 con 0x7fc424067e10 2023-10-10T18:45:43.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc428a51700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42406abd0 con 0x7fc424157960 2023-10-10T18:45:43.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42359e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc424067e10 0x7fc4241561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42359e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc424067e10 0x7fc4241561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51832/0 (socket says 172.21.15.175:51832) 2023-10-10T18:45:43.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42359e700 1 -- 172.21.15.175:0/313360067 learned_addr learned my addr 172.21.15.175:0/313360067 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:43.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42259c700 1 --2- 172.21.15.175:0/313360067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424157960 0x7fc424155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42359e700 1 -- 172.21.15.175:0/313360067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424157960 msgr2=0x7fc424155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:43.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42359e700 1 --2- 172.21.15.175:0/313360067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424157960 0x7fc424155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42359e700 1 -- 172.21.15.175:0/313360067 >> v1:172.21.15.191:6789/0 conn(0x7fc424067a40 legacy=0x7fc424157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:43.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42359e700 1 -- 172.21.15.175:0/313360067 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42406aeb0 con 0x7fc424067e10 2023-10-10T18:45:43.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42259c700 1 --2- 172.21.15.175:0/313360067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424157960 0x7fc424155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:43.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc42359e700 1 --2- 172.21.15.175:0/313360067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc4241561b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009f20 tx=0x7fc41c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c1da971ce72bdd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:43.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc421d9b700 1 -- 172.21.15.175:0/313360067 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c00e070 con 0x7fc424067e10 2023-10-10T18:45:43.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc421d9b700 1 -- 172.21.15.175:0/313360067 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c00ba80 con 0x7fc424067e10 2023-10-10T18:45:43.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.114+0000 7fc421d9b700 1 -- 172.21.15.175:0/313360067 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c0092e0 con 0x7fc424067e10 2023-10-10T18:45:43.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.115+0000 7fc428a51700 1 -- 172.21.15.175:0/313360067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 msgr2=0x7fc4241561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:43.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.115+0000 7fc428a51700 1 --2- 172.21.15.175:0/313360067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc4241561b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009f20 tx=0x7fc41c009840 comp rx=0 tx=0).stop 2023-10-10T18:45:43.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.115+0000 7fc428a51700 1 -- 172.21.15.175:0/313360067 shutdown_connections 2023-10-10T18:45:43.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.115+0000 7fc428a51700 1 --2- 172.21.15.175:0/313360067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc4241561b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.115+0000 7fc428a51700 1 --2- 172.21.15.175:0/313360067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424157960 0x7fc424155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.115+0000 7fc428a51700 1 -- 172.21.15.175:0/313360067 >> 172.21.15.175:0/313360067 conn(0x7fc42406bf10 msgr2=0x7fc42406c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:43.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.115+0000 7fc428a51700 1 -- 172.21.15.175:0/313360067 shutdown_connections 2023-10-10T18:45:43.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.115+0000 7fc428a51700 1 -- 172.21.15.175:0/313360067 wait complete. 2023-10-10T18:45:43.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.116+0000 7fc428a51700 1 Processor -- start 2023-10-10T18:45:43.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.116+0000 7fc428a51700 1 -- start start 2023-10-10T18:45:43.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc428a51700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424067a40 0x7fc42415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc428a51700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc424169040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc428a51700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc424157960 0x7fc4241406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc428a51700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc424150ae0 con 0x7fc424067e10 2023-10-10T18:45:43.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc428a51700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc424150f70 con 0x7fc424157960 2023-10-10T18:45:43.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc428a51700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4241501a0 con 0x7fc424067a40 2023-10-10T18:45:43.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc424169040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc424169040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51838/0 (socket says 172.21.15.175:51838) 2023-10-10T18:45:43.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 -- 172.21.15.175:0/1787863845 learned_addr learned my addr 172.21.15.175:0/1787863845 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:43.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc422d9d700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424067a40 0x7fc42415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42359e700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc424157960 0x7fc4241406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 -- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424067a40 msgr2=0x7fc42415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:43.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424067a40 0x7fc42415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 -- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc424157960 msgr2=0x7fc4241406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:43.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc424157960 0x7fc4241406b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 -- 172.21.15.175:0/1787863845 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c009ae0 con 0x7fc424067e10 2023-10-10T18:45:43.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc42259c700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc424169040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc41800d860 tx=0x7fc418008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:43.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc40affd700 1 -- 172.21.15.175:0/1787863845 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418005a00 con 0x7fc424067e10 2023-10-10T18:45:43.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc40affd700 1 -- 172.21.15.175:0/1787863845 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418005b60 con 0x7fc424067e10 2023-10-10T18:45:43.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc428a51700 1 -- 172.21.15.175:0/1787863845 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4240c89e0 con 0x7fc424067e10 2023-10-10T18:45:43.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.117+0000 7fc40affd700 1 -- 172.21.15.175:0/1787863845 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4180125f0 con 0x7fc424067e10 2023-10-10T18:45:43.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.118+0000 7fc428a51700 1 -- 172.21.15.175:0/1787863845 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc424150770 con 0x7fc424067e10 2023-10-10T18:45:43.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.119+0000 7fc40affd700 1 -- 172.21.15.175:0/1787863845 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc418012a20 con 0x7fc424067e10 2023-10-10T18:45:43.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.119+0000 7fc40affd700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc410066f30 0x7fc4100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.119+0000 7fc40affd700 1 -- 172.21.15.175:0/1787863845 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc418084670 con 0x7fc424067e10 2023-10-10T18:45:43.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.119+0000 7fc422d9d700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc410066f30 0x7fc4100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.120+0000 7fc422d9d700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc410066f30 0x7fc4100693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc414003a80 tx=0x7fc414007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:43.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.121+0000 7fc43254bb80 1 Processor -- start 2023-10-10T18:45:43.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.121+0000 7fc43254bb80 1 -- start start 2023-10-10T18:45:43.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cf77210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf77750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf767d0 0x560c1cf6c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc43254bb80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1ce8ba70 con 0x560c1cf37f30 2023-10-10T18:45:43.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc43254bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1ce89270 con 0x560c1cf767d0 2023-10-10T18:45:43.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc43254bb80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c1ce85060 con 0x560c1cf76400 2023-10-10T18:45:43.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf77750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf77750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51852/0 (socket says 172.21.15.175:51852) 2023-10-10T18:45:43.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 -- 172.21.15.175:0/2910545373 learned_addr learned my addr 172.21.15.175:0/2910545373 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:43.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc422d9d700 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cf77210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42359e700 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf767d0 0x560c1cf6c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 -- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 msgr2=0x560c1cf77210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:43.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cf77210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 -- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf767d0 msgr2=0x560c1cf6c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:43.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf767d0 0x560c1cf6c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 -- 172.21.15.175:0/2910545373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c1cd1f270 con 0x560c1cf37f30 2023-10-10T18:45:43.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc422d9d700 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cf77210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:43.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc42259c700 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf77750 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc418082c40 tx=0x7fc418082af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e58a9cd648cedb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:43.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc403fff700 1 -- 172.21.15.175:0/2910545373 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41805c540 con 0x560c1cf37f30 2023-10-10T18:45:43.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc403fff700 1 -- 172.21.15.175:0/2910545373 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418082ca0 con 0x560c1cf37f30 2023-10-10T18:45:43.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.122+0000 7fc403fff700 1 -- 172.21.15.175:0/2910545373 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4180539f0 con 0x560c1cf37f30 2023-10-10T18:45:43.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 -- 172.21.15.175:0/2910545373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 msgr2=0x560c1cf77750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:43.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf77750 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc418082c40 tx=0x7fc418082af0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 -- 172.21.15.175:0/2910545373 shutdown_connections 2023-10-10T18:45:43.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf767d0 0x560c1cf6c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cf77210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 --2- 172.21.15.175:0/2910545373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf77750 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 -- 172.21.15.175:0/2910545373 >> 172.21.15.175:0/2910545373 conn(0x560c1cf344a0 msgr2=0x560c1cf34870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:43.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 -- 172.21.15.175:0/2910545373 shutdown_connections 2023-10-10T18:45:43.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.123+0000 7fc43254bb80 1 -- 172.21.15.175:0/2910545373 wait complete. 2023-10-10T18:45:43.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.124+0000 7fc43254bb80 1 -- start start 2023-10-10T18:45:43.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.124+0000 7fc43254bb80 1 Processor -- start 2023-10-10T18:45:43.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf76400 0x560c1cf73f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf767d0 0x560c1cf8a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1ce30890 con 0x560c1cf37f30 2023-10-10T18:45:43.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1ce4bea0 con 0x560c1cf76400 2023-10-10T18:45:43.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c1cd07e30 con 0x560c1cf767d0 2023-10-10T18:45:43.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 10 client.0 _ll_register_callbacks cb 0x560c1cef82c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:43.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51856/0 (socket says 172.21.15.175:51856) 2023-10-10T18:45:43.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 -- 172.21.15.175:0/658218507 learned_addr learned my addr 172.21.15.175:0/658218507 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:43.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc42359e700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf767d0 0x560c1cf8a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc42259c700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf76400 0x560c1cf73f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 -- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf767d0 msgr2=0x560c1cf8a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:43.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf767d0 0x560c1cf8a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 -- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf76400 msgr2=0x560c1cf73f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:43.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf76400 0x560c1cf73f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:43.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 -- 172.21.15.175:0/658218507 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc424150770 con 0x560c1cf37f30 2023-10-10T18:45:43.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc42259c700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf76400 0x560c1cf73f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:43.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc422d9d700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6d1e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc414002a30 tx=0x7fc414015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae87af43e1f056e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:43.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc4017fa700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:43.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc4017fa700 1 -- 172.21.15.175:0/658218507 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414028070 con 0x560c1cf37f30 2023-10-10T18:45:43.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 1 -- 172.21.15.175:0/658218507 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560c1cf29cd0 con 0x560c1cf37f30 2023-10-10T18:45:43.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc4017fa700 1 -- 172.21.15.175:0/658218507 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc414014660 con 0x560c1cf37f30 2023-10-10T18:45:43.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc4017fa700 1 -- 172.21.15.175:0/658218507 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc414014a30 con 0x560c1cf37f30 2023-10-10T18:45:43.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 10 client.4680 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:43.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.125+0000 7fc43254bb80 1 -- 172.21.15.175:0/658218507 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x560c1ccc1fe0 con 0x560c1cf37f30 2023-10-10T18:45:43.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc4017fa700 1 -- 172.21.15.175:0/658218507 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560c1cf29cd0 con 0x560c1cf37f30 2023-10-10T18:45:43.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc4017fa700 1 -- 172.21.15.175:0/658218507 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7fc414014b90 con 0x560c1cf37f30 2023-10-10T18:45:43.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc43254bb80 10 client.4680 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:43.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc43254bb80 10 client.4680 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-10T18:45:43.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc43254bb80 20 client.4680 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:43.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc43254bb80 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:43.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc43254bb80 4 client.4680 mds_command: new command op to 4247 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T18:45:43.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc43254bb80 1 -- 172.21.15.175:0/658218507 --> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c1cf06aa0 con 0x560c1cf651e0 2023-10-10T18:45:43.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc42259c700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:43.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc4017fa700 10 client.4680 ms_handle_connect on v2:172.21.15.191:6835/3007475551 2023-10-10T18:45:43.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:43.126+0000 7fc42259c700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=286a7c1c2c4ae0d9 server_cookie=4ceab971d8c04673 in_seq=0 out_seq=0 2023-10-10T18:45:46.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.561+0000 7fc4017fa700 1 -- 172.21.15.175:0/658218507 <== mds.4 v2:172.21.15.191:6835/3007475551 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x560c1cf06aa0 con 0x560c1cf651e0 2023-10-10T18:45:46.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.561+0000 7fc4017fa700 10 client.4680 handle_command_reply: tid=0 2023-10-10T18:45:46.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.561+0000 7fc42259c700 1 -- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 msgr2=0x560c1cf655b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2023-10-10T18:45:46.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.561+0000 7fc42259c700 1 -- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 msgr2=0x560c1cf655b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-10T18:45:46.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.561+0000 7fc42259c700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-10T18:45:46.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.561+0000 7fc43254bb80 1 client.4680 shutdown 2023-10-10T18:45:46.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.561+0000 7fc42259c700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2023-10-10T18:45:46.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 -- 172.21.15.175:0/658218507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 msgr2=0x560c1cf6d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6d1e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc414002a30 tx=0x7fc414015000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc42259c700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc42259c700 -1 monclient: get_auth_request but no auth handler is set up 2023-10-10T18:45:46.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc42259c700 0 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 unknown :-1 s=AUTH_CONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -13 2023-10-10T18:45:46.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc42259c700 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 unknown :-1 s=AUTH_CONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc4017fa700 0 client.4680 ms_handle_reset on v2:172.21.15.191:6835/3007475551 2023-10-10T18:45:46.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 -- 172.21.15.175:0/658218507 shutdown_connections 2023-10-10T18:45:46.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf767d0 0x560c1cf8a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf651e0 0x560c1cf655b0 unknown :-1 s=CLOSED pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6d1e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 --2- 172.21.15.175:0/658218507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf76400 0x560c1cf73f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 -- 172.21.15.175:0/658218507 >> 172.21.15.175:0/658218507 conn(0x560c1cf344a0 msgr2=0x560c1cf68890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 -- 172.21.15.175:0/658218507 shutdown_connections 2023-10-10T18:45:46.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 1 -- 172.21.15.175:0/658218507 wait complete. 2023-10-10T18:45:46.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.562+0000 7fc43254bb80 20 client.4680 trim_cache size 0 max 16384 2023-10-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.580+0000 7fc43254bb80 1 Processor -- start 2023-10-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.580+0000 7fc43254bb80 1 -- start start 2023-10-10T18:45:46.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf37f30 0x560c1cf65c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cf75710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf76400 0x560c1cf75c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc43254bb80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1cf2c580 con 0x560c1cf76400 2023-10-10T18:45:46.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc43254bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1cf2b5c0 con 0x560c1cf651e0 2023-10-10T18:45:46.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc43254bb80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c1cf6d370 con 0x560c1cf37f30 2023-10-10T18:45:46.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf76400 0x560c1cf75c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf76400 0x560c1cf75c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37412/0 (socket says 172.21.15.175:37412) 2023-10-10T18:45:46.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 -- 172.21.15.175:0/1647617063 learned_addr learned my addr 172.21.15.175:0/1647617063 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc422d9d700 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf37f30 0x560c1cf65c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42259c700 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cf75710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 -- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf37f30 msgr2=0x560c1cf65c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf37f30 0x560c1cf65c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 -- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 msgr2=0x560c1cf75710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cf75710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 -- 172.21.15.175:0/1647617063 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c1cf6b3b0 con 0x560c1cf76400 2023-10-10T18:45:46.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc422d9d700 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf37f30 0x560c1cf65c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:46.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42259c700 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cf75710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:46.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.581+0000 7fc42359e700 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf76400 0x560c1cf75c50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00b6c0 tx=0x7fc41c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ce9b60953ca4e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc408ff9700 1 -- 172.21.15.175:0/1647617063 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c023040 con 0x560c1cf76400 2023-10-10T18:45:46.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc408ff9700 1 -- 172.21.15.175:0/1647617063 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c008120 con 0x560c1cf76400 2023-10-10T18:45:46.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc408ff9700 1 -- 172.21.15.175:0/1647617063 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c0084e0 con 0x560c1cf76400 2023-10-10T18:45:46.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc43254bb80 1 -- 172.21.15.175:0/1647617063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf76400 msgr2=0x560c1cf75c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc43254bb80 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf76400 0x560c1cf75c50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00b6c0 tx=0x7fc41c018000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc43254bb80 1 -- 172.21.15.175:0/1647617063 shutdown_connections 2023-10-10T18:45:46.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc43254bb80 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf76400 0x560c1cf75c50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc43254bb80 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf37f30 0x560c1cf65c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.582+0000 7fc43254bb80 1 --2- 172.21.15.175:0/1647617063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cf75710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.583+0000 7fc43254bb80 1 -- 172.21.15.175:0/1647617063 >> 172.21.15.175:0/1647617063 conn(0x560c1cf344a0 msgr2=0x560c1cf6dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.583+0000 7fc43254bb80 1 -- 172.21.15.175:0/1647617063 shutdown_connections 2023-10-10T18:45:46.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.583+0000 7fc43254bb80 1 -- 172.21.15.175:0/1647617063 wait complete. 2023-10-10T18:45:46.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.583+0000 7fc43254bb80 1 -- start start 2023-10-10T18:45:46.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.583+0000 7fc43254bb80 1 Processor -- start 2023-10-10T18:45:46.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cfd6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc43254bb80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cfd67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc43254bb80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1cf7a330 con 0x560c1cf37f30 2023-10-10T18:45:46.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc43254bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1cf36980 con 0x560c1cf651e0 2023-10-10T18:45:46.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc43254bb80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c1cf6e1f0 con 0x560c1cf76400 2023-10-10T18:45:46.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc43254bb80 10 client.0 _ll_register_callbacks cb 0x560c1cef82c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:46.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc422d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc422d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37428/0 (socket says 172.21.15.175:37428) 2023-10-10T18:45:46.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc422d9d700 1 -- 172.21.15.175:0/399638424 learned_addr learned my addr 172.21.15.175:0/399638424 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc42259c700 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cfd6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.584+0000 7fc42359e700 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cfd67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc422d9d700 1 -- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 msgr2=0x560c1cfd67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc422d9d700 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cfd67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc422d9d700 1 -- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 msgr2=0x560c1cfd6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc422d9d700 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cfd6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc422d9d700 1 -- 172.21.15.175:0/399638424 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42406aeb0 con 0x560c1cf37f30 2023-10-10T18:45:46.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc422d9d700 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6dcb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc414014740 tx=0x7fc414029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f6fd2fd4f5766d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc4017fa700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:46.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc4017fa700 1 -- 172.21.15.175:0/399638424 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41403e070 con 0x560c1cf37f30 2023-10-10T18:45:46.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc4017fa700 1 -- 172.21.15.175:0/399638424 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc4140024f0 con 0x560c1cf37f30 2023-10-10T18:45:46.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc43254bb80 1 -- 172.21.15.175:0/399638424 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc418082ca0 con 0x560c1cf37f30 2023-10-10T18:45:46.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc4017fa700 1 -- 172.21.15.175:0/399638424 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc414014c40 con 0x560c1cf37f30 2023-10-10T18:45:46.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc43254bb80 10 client.4686 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:46.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc43254bb80 1 -- 172.21.15.175:0/399638424 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x560c1ce53780 con 0x560c1cf37f30 2023-10-10T18:45:46.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.585+0000 7fc4017fa700 1 -- 172.21.15.175:0/399638424 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418082ca0 con 0x560c1cf37f30 2023-10-10T18:45:46.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc4017fa700 1 -- 172.21.15.175:0/399638424 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7fc41402ba60 con 0x560c1cf37f30 2023-10-10T18:45:46.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc43254bb80 10 client.4686 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:46.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc43254bb80 10 client.4686 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-10T18:45:46.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc43254bb80 20 client.4686 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:46.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc43254bb80 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf767d0 0x560c1cfdfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc43254bb80 4 client.4686 mds_command: new command op to 4247 tid=0[{"prefix": "status"}] 2023-10-10T18:45:46.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc43254bb80 1 -- 172.21.15.175:0/399638424 --> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560c1ce27e00 con 0x560c1cf767d0 2023-10-10T18:45:46.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc42259c700 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf767d0 0x560c1cfdfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc4017fa700 10 client.4686 ms_handle_connect on v2:172.21.15.191:6835/3007475551 2023-10-10T18:45:46.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc42259c700 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf767d0 0x560c1cfdfc80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=de691dd994aca3a5 server_cookie=f39b8dd7048fd12a in_seq=0 out_seq=0 2023-10-10T18:45:46.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc4017fa700 1 -- 172.21.15.175:0/399638424 <== mds.4 v2:172.21.15.191:6835/3007475551 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x560c1ce27e00 con 0x560c1cf767d0 2023-10-10T18:45:46.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc4017fa700 10 client.4686 handle_command_reply: tid=0 2023-10-10T18:45:46.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.586+0000 7fc43254bb80 1 client.4686 shutdown 2023-10-10T18:45:46.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 -- 172.21.15.175:0/399638424 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 msgr2=0x560c1cf6dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6dcb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc414014740 tx=0x7fc414029000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 -- 172.21.15.175:0/399638424 shutdown_connections 2023-10-10T18:45:46.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x560c1cf76400 0x560c1cfd67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:6835/3007475551,v1:172.21.15.191:6836/3007475551] conn(0x560c1cf767d0 0x560c1cfdfc80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560c1cf37f30 0x560c1cf6dcb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 --2- 172.21.15.175:0/399638424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c1cf651e0 0x560c1cfd6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc4017fa700 0 client.4686 ms_handle_reset on v2:172.21.15.191:6835/3007475551 2023-10-10T18:45:46.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 -- 172.21.15.175:0/399638424 >> 172.21.15.175:0/399638424 conn(0x560c1cf344a0 msgr2=0x560c1cf759c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 -- 172.21.15.175:0/399638424 shutdown_connections 2023-10-10T18:45:46.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 1 -- 172.21.15.175:0/399638424 wait complete. 2023-10-10T18:45:46.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc43254bb80 20 client.4686 trim_cache size 0 max 16384 2023-10-10T18:45:46.604 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T18:45:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "cluster_fsid": "7f0fe9e3-2088-4ba2-8a54-92a2be521afd", 2023-10-10T18:45:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "whoami": 4, 2023-10-10T18:45:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 4247, 2023-10-10T18:45:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "want_state": "up:active", 2023-10-10T18:45:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "state": "up:active", 2023-10-10T18:45:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs", 2023-10-10T18:45:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "rank_uptime": 11.044514088, 2023-10-10T18:45:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "mdsmap_epoch": 17, 2023-10-10T18:45:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch": 21, 2023-10-10T18:45:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch_barrier": 21, 2023-10-10T18:45:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "uptime": 42.072971334999998 2023-10-10T18:45:46.606 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T18:45:46.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc428a51700 1 -- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc410066f30 msgr2=0x7fc4100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc428a51700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc410066f30 0x7fc4100693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc414003a80 tx=0x7fc414007ec0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc428a51700 1 -- 172.21.15.175:0/1787863845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 msgr2=0x7fc424169040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc428a51700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc424169040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc41800d860 tx=0x7fc418008040 comp rx=0 tx=0).stop 2023-10-10T18:45:46.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc428a51700 1 -- 172.21.15.175:0/1787863845 shutdown_connections 2023-10-10T18:45:46.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.587+0000 7fc428a51700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc410066f30 0x7fc4100693e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.588+0000 7fc428a51700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc424157960 0x7fc4241406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.588+0000 7fc428a51700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc424067a40 0x7fc42415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.588+0000 7fc428a51700 1 --2- 172.21.15.175:0/1787863845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc424067e10 0x7fc424169040 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.588+0000 7fc428a51700 1 -- 172.21.15.175:0/1787863845 >> 172.21.15.175:0/1787863845 conn(0x7fc42406bf10 msgr2=0x7fc424143f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.588+0000 7fc428a51700 1 -- 172.21.15.175:0/1787863845 shutdown_connections 2023-10-10T18:45:46.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.588+0000 7fc428a51700 1 -- 172.21.15.175:0/1787863845 wait complete. 2023-10-10T18:45:46.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-10T18:45:46.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.732+0000 7fa45553e700 1 Processor -- start 2023-10-10T18:45:46.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.732+0000 7fa45553e700 1 -- start start 2023-10-10T18:45:46.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.733+0000 7fa45553e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa450156850 0x7fa450156c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.733+0000 7fa45553e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45006dcc0 con 0x7fa45006d870 2023-10-10T18:45:46.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.733+0000 7fa45553e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa450068880 con 0x7fa4501599d0 2023-10-10T18:45:46.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.733+0000 7fa45553e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa450069c90 con 0x7fa450156850 2023-10-10T18:45:46.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.733+0000 7fa44f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa450156850 0x7fa450156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.733+0000 7fa44f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa450156850 0x7fa450156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37436/0 (socket says 172.21.15.175:37436) 2023-10-10T18:45:46.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.733+0000 7fa44e7fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa45006d870 0x7fa45006bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51474/0 (socket says 172.21.15.175:51474) 2023-10-10T18:45:46.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.733+0000 7fa44f7fe700 1 -- 172.21.15.175:0/2451276742 learned_addr learned my addr 172.21.15.175:0/2451276742 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa44f7fe700 1 -- 172.21.15.175:0/2451276742 >> v1:172.21.15.191:6789/0 conn(0x7fa4501599d0 legacy=0x7fa450159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa44f7fe700 1 -- 172.21.15.175:0/2451276742 >> v1:172.21.15.175:6789/0 conn(0x7fa45006d870 legacy=0x7fa45006bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa44f7fe700 1 -- 172.21.15.175:0/2451276742 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa450068b60 con 0x7fa450156850 2023-10-10T18:45:46.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa44dffb700 1 -- 172.21.15.175:0/2451276742 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272788339 0 0) 0x7fa438001110 con 0x7fa45006d870 2023-10-10T18:45:46.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/2451276742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa450156850 0x7fa450156c40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa4480099b0 tx=0x7fa448009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2628afffda0f8c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa44dffb700 1 -- 172.21.15.175:0/2451276742 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800b860 con 0x7fa450156850 2023-10-10T18:45:46.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa44dffb700 1 -- 172.21.15.175:0/2451276742 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44800b9c0 con 0x7fa450156850 2023-10-10T18:45:46.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa44dffb700 1 -- 172.21.15.175:0/2451276742 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800bd60 con 0x7fa450156850 2023-10-10T18:45:46.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa45553e700 1 -- 172.21.15.175:0/2451276742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa450156850 msgr2=0x7fa450156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa45553e700 1 --2- 172.21.15.175:0/2451276742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa450156850 0x7fa450156c40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa4480099b0 tx=0x7fa448009250 comp rx=0 tx=0).stop 2023-10-10T18:45:46.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa45553e700 1 -- 172.21.15.175:0/2451276742 shutdown_connections 2023-10-10T18:45:46.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa45553e700 1 --2- 172.21.15.175:0/2451276742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa450156850 0x7fa450156c40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa45553e700 1 -- 172.21.15.175:0/2451276742 >> 172.21.15.175:0/2451276742 conn(0x7fa450063cc0 msgr2=0x7fa45006d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa45553e700 1 -- 172.21.15.175:0/2451276742 shutdown_connections 2023-10-10T18:45:46.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.734+0000 7fa45553e700 1 -- 172.21.15.175:0/2451276742 wait complete. 2023-10-10T18:45:46.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa45553e700 1 Processor -- start 2023-10-10T18:45:46.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa45553e700 1 -- start start 2023-10-10T18:45:46.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa45553e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45006d870 0x7fa450158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa45553e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa450156850 0x7fa4501548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa45553e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4501599d0 0x7fa450157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa45553e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45015c5d0 con 0x7fa4501599d0 2023-10-10T18:45:46.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa45553e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500c1400 con 0x7fa45006d870 2023-10-10T18:45:46.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa45553e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45014c5f0 con 0x7fa450156850 2023-10-10T18:45:46.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa44f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4501599d0 0x7fa450157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa44f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4501599d0 0x7fa450157470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37438/0 (socket says 172.21.15.175:37438) 2023-10-10T18:45:46.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.736+0000 7fa44f7fe700 1 -- 172.21.15.175:0/346357711 learned_addr learned my addr 172.21.15.175:0/346357711 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa450156850 0x7fa4501548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44f7fe700 1 -- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa450156850 msgr2=0x7fa4501548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa450156850 0x7fa4501548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44effd700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45006d870 0x7fa450158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44f7fe700 1 -- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45006d870 msgr2=0x7fa450158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45006d870 0x7fa450158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44f7fe700 1 -- 172.21.15.175:0/346357711 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4480094f0 con 0x7fa4501599d0 2023-10-10T18:45:46.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa450156850 0x7fa4501548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:46.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4501599d0 0x7fa450157470 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa4500641b0 tx=0x7fa448017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa446ffd700 1 -- 172.21.15.175:0/346357711 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44801f070 con 0x7fa4501599d0 2023-10-10T18:45:46.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa446ffd700 1 -- 172.21.15.175:0/346357711 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa448002f70 con 0x7fa4501599d0 2023-10-10T18:45:46.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa446ffd700 1 -- 172.21.15.175:0/346357711 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448025650 con 0x7fa4501599d0 2023-10-10T18:45:46.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa45553e700 1 -- 172.21.15.175:0/346357711 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa450159470 con 0x7fa4501599d0 2023-10-10T18:45:46.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.737+0000 7fa45553e700 1 -- 172.21.15.175:0/346357711 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45015b5c0 con 0x7fa4501599d0 2023-10-10T18:45:46.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.738+0000 7fa446ffd700 1 -- 172.21.15.175:0/346357711 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa448023070 con 0x7fa4501599d0 2023-10-10T18:45:46.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.739+0000 7fa446ffd700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa430066f30 0x7fa4300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.739+0000 7fa446ffd700 1 -- 172.21.15.175:0/346357711 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa448085e00 con 0x7fa4501599d0 2023-10-10T18:45:46.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.739+0000 7fa44effd700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa430066f30 0x7fa4300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.740+0000 7fa44effd700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa430066f30 0x7fa4300693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa440007fd0 tx=0x7fa440009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.741+0000 7fa45e837b80 1 Processor -- start 2023-10-10T18:45:46.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.741+0000 7fa45e837b80 1 -- start start 2023-10-10T18:45:46.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 0x562ce6fd4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce6fd4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa45e837b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce6f2ba70 con 0x562ce700c200 2023-10-10T18:45:46.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa45e837b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce6f29270 con 0x562ce700c5d0 2023-10-10T18:45:46.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa45e837b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ce6f25060 con 0x562ce6fd88a0 2023-10-10T18:45:46.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37440/0 (socket says 172.21.15.175:37440) 2023-10-10T18:45:46.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 -- 172.21.15.175:0/1690485988 learned_addr learned my addr 172.21.15.175:0/1690485988 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 0x562ce6fd4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce6fd4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 -- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 msgr2=0x562ce6fd4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 0x562ce6fd4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 -- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 msgr2=0x562ce6fd4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce6fd4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 -- 172.21.15.175:0/1690485988 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ce6dbf270 con 0x562ce700c200 2023-10-10T18:45:46.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce6fd4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:46.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa44effd700 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd1800 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa44000db70 tx=0x7fa440014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab3434ad341e641 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa42ffff700 1 -- 172.21.15.175:0/1690485988 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440014ac0 con 0x562ce700c200 2023-10-10T18:45:46.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa42ffff700 1 -- 172.21.15.175:0/1690485988 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa440014c20 con 0x562ce700c200 2023-10-10T18:45:46.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.742+0000 7fa42ffff700 1 -- 172.21.15.175:0/1690485988 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44001f400 con 0x562ce700c200 2023-10-10T18:45:46.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 -- 172.21.15.175:0/1690485988 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 msgr2=0x562ce6fd1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd1800 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa44000db70 tx=0x7fa440014000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 -- 172.21.15.175:0/1690485988 shutdown_connections 2023-10-10T18:45:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce6fd4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd1800 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 --2- 172.21.15.175:0/1690485988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 0x562ce6fd4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 -- 172.21.15.175:0/1690485988 >> 172.21.15.175:0/1690485988 conn(0x562ce6fd5ba0 msgr2=0x562ce6fd6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 -- 172.21.15.175:0/1690485988 shutdown_connections 2023-10-10T18:45:46.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.743+0000 7fa45e837b80 1 -- 172.21.15.175:0/1690485988 wait complete. 2023-10-10T18:45:46.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.744+0000 7fa45e837b80 1 -- start start 2023-10-10T18:45:46.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.744+0000 7fa45e837b80 1 Processor -- start 2023-10-10T18:45:46.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.744+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce6fd88a0 0x562ce6fd0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.744+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c200 0x562ce701a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.744+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.744+0000 7fa45e837b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce6ed0890 con 0x562ce6fd88a0 2023-10-10T18:45:46.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.744+0000 7fa45e837b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce6eebea0 con 0x562ce700c200 2023-10-10T18:45:46.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.744+0000 7fa45e837b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ce6da7e30 con 0x562ce700c5d0 2023-10-10T18:45:46.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 10 client.0 _ll_register_callbacks cb 0x562ce6f98110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:46.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce6fd88a0 0x562ce6fd0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce6fd88a0 0x562ce6fd0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37450/0 (socket says 172.21.15.175:37450) 2023-10-10T18:45:46.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 -- 172.21.15.175:0/2352132708 learned_addr learned my addr 172.21.15.175:0/2352132708 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c200 0x562ce701a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 -- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 msgr2=0x562ce701aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 -- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c200 msgr2=0x562ce701a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c200 0x562ce701a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 -- 172.21.15.175:0/2352132708 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa450068880 con 0x562ce6fd88a0 2023-10-10T18:45:46.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:46.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c200 0x562ce701a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:46.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa44effd700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce6fd88a0 0x562ce6fd0000 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x562ce6fd30f0 tx=0x7fa440016030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c581ef8b9526fbe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa42d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:46.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa42d7fa700 1 -- 172.21.15.175:0/2352132708 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44001f980 con 0x562ce6fd88a0 2023-10-10T18:45:46.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa42d7fa700 1 -- 172.21.15.175:0/2352132708 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa44001fae0 con 0x562ce6fd88a0 2023-10-10T18:45:46.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 1 -- 172.21.15.175:0/2352132708 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ce6fc9b70 con 0x562ce6fd88a0 2023-10-10T18:45:46.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa42d7fa700 1 -- 172.21.15.175:0/2352132708 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa440029410 con 0x562ce6fd88a0 2023-10-10T18:45:46.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 10 client.4701 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:46.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 1 -- 172.21.15.175:0/2352132708 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x562ce6d61fe0 con 0x562ce6fd88a0 2023-10-10T18:45:46.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa42d7fa700 1 -- 172.21.15.175:0/2352132708 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562ce6fc9b70 con 0x562ce6fd88a0 2023-10-10T18:45:46.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa42d7fa700 1 -- 172.21.15.175:0/2352132708 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7fa440029570 con 0x562ce6fd88a0 2023-10-10T18:45:46.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 10 client.4701 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:46.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 10 client.4701 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-10T18:45:46.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 20 client.4701 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:46.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] conn(0x562ce6ffff20 0x562ce70002f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 4 client.4701 mds_command: new command op to 4248 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T18:45:46.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.745+0000 7fa45e837b80 1 -- 172.21.15.175:0/2352132708 --> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ce6fa6aa0 con 0x562ce6ffff20 2023-10-10T18:45:46.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.746+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] conn(0x562ce6ffff20 0x562ce70002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.746+0000 7fa42d7fa700 10 client.4701 ms_handle_connect on v2:172.21.15.175:6833/647425682 2023-10-10T18:45:46.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.746+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] conn(0x562ce6ffff20 0x562ce70002f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ae75de1d3ebdcb86 server_cookie=177eaa109ba3f405 in_seq=0 out_seq=0 2023-10-10T18:45:46.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.746+0000 7fa42d7fa700 1 -- 172.21.15.175:0/2352132708 <== mds.3 v2:172.21.15.175:6833/647425682 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x562ce6fa6aa0 con 0x562ce6ffff20 2023-10-10T18:45:46.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.746+0000 7fa42d7fa700 10 client.4701 handle_command_reply: tid=0 2023-10-10T18:45:46.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.746+0000 7fa45e837b80 1 client.4701 shutdown 2023-10-10T18:45:46.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.746+0000 7fa45e837b80 1 -- 172.21.15.175:0/2352132708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce6fd88a0 msgr2=0x562ce6fd0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.746+0000 7fa45e837b80 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce6fd88a0 0x562ce6fd0000 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x562ce6fd30f0 tx=0x7fa440016030 comp rx=0 tx=0).stop 2023-10-10T18:45:46.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 1 -- 172.21.15.175:0/2352132708 shutdown_connections 2023-10-10T18:45:46.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] conn(0x562ce6ffff20 0x562ce70002f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa42d7fa700 0 client.4701 ms_handle_reset on v2:172.21.15.175:6833/647425682 2023-10-10T18:45:46.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce6fd88a0 0x562ce6fd0000 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 1 --2- 172.21.15.175:0/2352132708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c200 0x562ce701a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 1 -- 172.21.15.175:0/2352132708 >> 172.21.15.175:0/2352132708 conn(0x562ce6fd5ba0 msgr2=0x562ce6fd0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 1 -- 172.21.15.175:0/2352132708 shutdown_connections 2023-10-10T18:45:46.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 1 -- 172.21.15.175:0/2352132708 wait complete. 2023-10-10T18:45:46.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.747+0000 7fa45e837b80 20 client.4701 trim_cache size 0 max 16384 2023-10-10T18:45:46.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.763+0000 7fa45e837b80 1 Processor -- start 2023-10-10T18:45:46.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.763+0000 7fa45e837b80 1 -- start start 2023-10-10T18:45:46.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce6fd88a0 0x562ce7005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce701a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa45e837b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce700a570 con 0x562ce700c200 2023-10-10T18:45:46.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa45e837b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce7009b40 con 0x562ce6fd88a0 2023-10-10T18:45:46.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa45e837b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ce6fd72d0 con 0x562ce700c5d0 2023-10-10T18:45:46.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce701a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce701a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37456/0 (socket says 172.21.15.175:37456) 2023-10-10T18:45:46.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44e7fc700 1 -- 172.21.15.175:0/3307752165 learned_addr learned my addr 172.21.15.175:0/3307752165 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44effd700 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce6fd88a0 0x562ce7005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44e7fc700 1 -- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 msgr2=0x562ce701a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44e7fc700 1 -- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce6fd88a0 msgr2=0x562ce7005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce6fd88a0 0x562ce7005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44e7fc700 1 -- 172.21.15.175:0/3307752165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ce6ef3780 con 0x562ce700c200 2023-10-10T18:45:46.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.764+0000 7fa44effd700 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce6fd88a0 0x562ce7005b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:46.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce701a270 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa438000fd0 tx=0x7fa438007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d821754b98b4f882 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa444ff9700 1 -- 172.21.15.175:0/3307752165 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa438005e20 con 0x562ce700c200 2023-10-10T18:45:46.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa444ff9700 1 -- 172.21.15.175:0/3307752165 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43801a070 con 0x562ce700c200 2023-10-10T18:45:46.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa444ff9700 1 -- 172.21.15.175:0/3307752165 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa438007310 con 0x562ce700c200 2023-10-10T18:45:46.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 -- 172.21.15.175:0/3307752165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 msgr2=0x562ce701a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce701a270 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa438000fd0 tx=0x7fa438007d30 comp rx=0 tx=0).stop 2023-10-10T18:45:46.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 -- 172.21.15.175:0/3307752165 shutdown_connections 2023-10-10T18:45:46.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce700c5d0 0x562ce701a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce6fd88a0 0x562ce7005b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 --2- 172.21.15.175:0/3307752165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce701a270 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 -- 172.21.15.175:0/3307752165 >> 172.21.15.175:0/3307752165 conn(0x562ce6fd5ba0 msgr2=0x562ce700b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 -- 172.21.15.175:0/3307752165 shutdown_connections 2023-10-10T18:45:46.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.765+0000 7fa45e837b80 1 -- 172.21.15.175:0/3307752165 wait complete. 2023-10-10T18:45:46.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.766+0000 7fa45e837b80 1 -- start start 2023-10-10T18:45:46.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.766+0000 7fa45e837b80 1 Processor -- start 2023-10-10T18:45:46.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 0x562ce6fff940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce7004320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce7002680 con 0x562ce700c200 2023-10-10T18:45:46.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ce6fd0fa0 con 0x562ce700c5d0 2023-10-10T18:45:46.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ce707cf60 con 0x562ce6fd88a0 2023-10-10T18:45:46.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 10 client.0 _ll_register_callbacks cb 0x562ce6f98110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:46.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44effd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 0x562ce6fff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37460/0 (socket says 172.21.15.175:37460) 2023-10-10T18:45:46.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 -- 172.21.15.175:0/138480844 learned_addr learned my addr 172.21.15.175:0/138480844 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44f7fe700 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce7004320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 -- 172.21.15.175:0/138480844 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 msgr2=0x562ce6fff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 0x562ce6fff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 -- 172.21.15.175:0/138480844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 msgr2=0x562ce7004320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce7004320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 -- 172.21.15.175:0/138480844 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4380017b0 con 0x562ce700c200 2023-10-10T18:45:46.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa42d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:46.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa44e7fc700 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd0a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa4501557f0 tx=0x7fa438009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0df1e828d463c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa42d7fa700 1 -- 172.21.15.175:0/138480844 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa438026070 con 0x562ce700c200 2023-10-10T18:45:46.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa42d7fa700 1 -- 172.21.15.175:0/138480844 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa43800c040 con 0x562ce700c200 2023-10-10T18:45:46.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 1 -- 172.21.15.175:0/138480844 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa440014c20 con 0x562ce700c200 2023-10-10T18:45:46.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa42d7fa700 1 -- 172.21.15.175:0/138480844 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa438016590 con 0x562ce700c200 2023-10-10T18:45:46.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 10 client.4707 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:46.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.767+0000 7fa45e837b80 1 -- 172.21.15.175:0/138480844 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x562ce6eda470 con 0x562ce700c200 2023-10-10T18:45:46.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa42d7fa700 1 -- 172.21.15.175:0/138480844 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa440014c20 con 0x562ce700c200 2023-10-10T18:45:46.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa42d7fa700 1 -- 172.21.15.175:0/138480844 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7fa4380166f0 con 0x562ce700c200 2023-10-10T18:45:46.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 10 client.4707 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:46.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 10 client.4707 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-10T18:45:46.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 20 client.4707 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:46.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] conn(0x562ce7004e50 0x562ce7076300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 4 client.4707 mds_command: new command op to 4248 tid=0[{"prefix": "status"}] 2023-10-10T18:45:46.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 1 -- 172.21.15.175:0/138480844 --> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562ce6ec7e00 con 0x562ce7004e50 2023-10-10T18:45:46.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa44effd700 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] conn(0x562ce7004e50 0x562ce7076300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa42d7fa700 10 client.4707 ms_handle_connect on v2:172.21.15.175:6833/647425682 2023-10-10T18:45:46.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa44effd700 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] conn(0x562ce7004e50 0x562ce7076300 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4a2635f67caa0f69 server_cookie=48894ac4b8f37b4d in_seq=0 out_seq=0 2023-10-10T18:45:46.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa42d7fa700 1 -- 172.21.15.175:0/138480844 <== mds.3 v2:172.21.15.175:6833/647425682 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x562ce6ec7e00 con 0x562ce7004e50 2023-10-10T18:45:46.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa42d7fa700 10 client.4707 handle_command_reply: tid=0 2023-10-10T18:45:46.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 1 client.4707 shutdown 2023-10-10T18:45:46.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 1 -- 172.21.15.175:0/138480844 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 msgr2=0x562ce6fd0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd0a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa4501557f0 tx=0x7fa438009000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 1 -- 172.21.15.175:0/138480844 shutdown_connections 2023-10-10T18:45:46.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.768+0000 7fa45e837b80 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.175:6833/647425682,v1:172.21.15.175:6836/647425682] conn(0x562ce7004e50 0x562ce7076300 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45e837b80 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562ce700c5d0 0x562ce7004320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45e837b80 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562ce6fd88a0 0x562ce6fff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45e837b80 1 --2- 172.21.15.175:0/138480844 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562ce700c200 0x562ce6fd0a60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa42d7fa700 0 client.4707 ms_handle_reset on v2:172.21.15.175:6833/647425682 2023-10-10T18:45:46.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45e837b80 1 -- 172.21.15.175:0/138480844 >> 172.21.15.175:0/138480844 conn(0x562ce6fd5ba0 msgr2=0x562ce6fd41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45e837b80 1 -- 172.21.15.175:0/138480844 shutdown_connections 2023-10-10T18:45:46.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45e837b80 1 -- 172.21.15.175:0/138480844 wait complete. 2023-10-10T18:45:46.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45e837b80 20 client.4707 trim_cache size 0 max 16384 2023-10-10T18:45:46.797 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T18:45:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "cluster_fsid": "7f0fe9e3-2088-4ba2-8a54-92a2be521afd", 2023-10-10T18:45:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "whoami": 3, 2023-10-10T18:45:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 4248, 2023-10-10T18:45:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "want_state": "up:active", 2023-10-10T18:45:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "state": "up:active", 2023-10-10T18:45:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs", 2023-10-10T18:45:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "rank_uptime": 17.249029617000001, 2023-10-10T18:45:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "mdsmap_epoch": 17, 2023-10-10T18:45:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch": 21, 2023-10-10T18:45:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch_barrier": 21, 2023-10-10T18:45:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: "uptime": 42.272352357000003 2023-10-10T18:45:46.800 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T18:45:46.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 -- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa430066f30 msgr2=0x7fa4300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa430066f30 0x7fa4300693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa440007fd0 tx=0x7fa440009040 comp rx=0 tx=0).stop 2023-10-10T18:45:46.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 -- 172.21.15.175:0/346357711 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4501599d0 msgr2=0x7fa450157470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4501599d0 0x7fa450157470 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa4500641b0 tx=0x7fa448017000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 -- 172.21.15.175:0/346357711 shutdown_connections 2023-10-10T18:45:46.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa430066f30 0x7fa4300693e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4501599d0 0x7fa450157470 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45006d870 0x7fa450158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 --2- 172.21.15.175:0/346357711 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa450156850 0x7fa4501548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 -- 172.21.15.175:0/346357711 >> 172.21.15.175:0/346357711 conn(0x7fa450063cc0 msgr2=0x7fa4500c84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 -- 172.21.15.175:0/346357711 shutdown_connections 2023-10-10T18:45:46.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.769+0000 7fa45553e700 1 -- 172.21.15.175:0/346357711 wait complete. 2023-10-10T18:45:46.804 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-10-10T18:45:46.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.919+0000 7f69b88cc700 1 Processor -- start 2023-10-10T18:45:46.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.919+0000 7f69b88cc700 1 -- start start 2023-10-10T18:45:46.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b88cc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40bf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b88cc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b40c6c20 con 0x7f69b406b080 2023-10-10T18:45:46.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b88cc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b4158010 con 0x7f69b40c57f0 2023-10-10T18:45:46.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b88cc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b40c4d70 con 0x7f69b40b0080 2023-10-10T18:45:46.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b2ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f69b406b080 0x7f69b4157900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51478/0 (socket says 172.21.15.175:51478) 2023-10-10T18:45:46.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b2ffd700 1 -- 172.21.15.175:0/1387797841 learned_addr learned my addr 172.21.15.175:0/1387797841 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:46.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b3fff700 1 --2- 172.21.15.175:0/1387797841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40bf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b27fc700 1 -- 172.21.15.175:0/1387797841 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226355577 0 0) 0x7f69b40c6c20 con 0x7f69b406b080 2023-10-10T18:45:46.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b27fc700 1 -- 172.21.15.175:0/1387797841 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f699c003570 con 0x7f69b406b080 2023-10-10T18:45:46.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b27fc700 1 -- 172.21.15.175:0/1387797841 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491121401 0 0) 0x7f69b4158010 con 0x7f69b40c57f0 2023-10-10T18:45:46.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b27fc700 1 -- 172.21.15.175:0/1387797841 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b40c6c20 con 0x7f69b40c57f0 2023-10-10T18:45:46.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b3fff700 1 -- 172.21.15.175:0/1387797841 >> v1:172.21.15.191:6789/0 conn(0x7f69b40c57f0 legacy=0x7f69b41571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b3fff700 1 -- 172.21.15.175:0/1387797841 >> v1:172.21.15.175:6789/0 conn(0x7f69b406b080 legacy=0x7f69b4157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.920+0000 7f69b3fff700 1 -- 172.21.15.175:0/1387797841 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b40c5260 con 0x7f69b40b0080 2023-10-10T18:45:46.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b3fff700 1 --2- 172.21.15.175:0/1387797841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40bf270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009910 tx=0x7f69ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9a9682f5b4b37dd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b27fc700 1 -- 172.21.15.175:0/1387797841 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00b690 con 0x7f69b40b0080 2023-10-10T18:45:46.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b27fc700 1 -- 172.21.15.175:0/1387797841 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac00b7f0 con 0x7f69b40b0080 2023-10-10T18:45:46.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b27fc700 1 -- 172.21.15.175:0/1387797841 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00bb90 con 0x7f69b40b0080 2023-10-10T18:45:46.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b88cc700 1 -- 172.21.15.175:0/1387797841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 msgr2=0x7f69b40bf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b88cc700 1 --2- 172.21.15.175:0/1387797841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40bf270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009910 tx=0x7f69ac0091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b88cc700 1 -- 172.21.15.175:0/1387797841 shutdown_connections 2023-10-10T18:45:46.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b88cc700 1 --2- 172.21.15.175:0/1387797841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40bf270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b88cc700 1 -- 172.21.15.175:0/1387797841 >> 172.21.15.175:0/1387797841 conn(0x7f69b406c640 msgr2=0x7f69b4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b88cc700 1 -- 172.21.15.175:0/1387797841 shutdown_connections 2023-10-10T18:45:46.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.921+0000 7f69b88cc700 1 -- 172.21.15.175:0/1387797841 wait complete. 2023-10-10T18:45:46.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b88cc700 1 Processor -- start 2023-10-10T18:45:46.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b88cc700 1 -- start start 2023-10-10T18:45:46.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b88cc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69b406b080 0x7f69b40c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b88cc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b88cc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b40c57f0 0x7f69b40ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b88cc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b40c2830 con 0x7f69b40c57f0 2023-10-10T18:45:46.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b88cc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b40c9db0 con 0x7f69b406b080 2023-10-10T18:45:46.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b88cc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b40c16e0 con 0x7f69b40b0080 2023-10-10T18:45:46.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b40c57f0 0x7f69b40ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b40c57f0 0x7f69b40ca200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37464/0 (socket says 172.21.15.175:37464) 2023-10-10T18:45:46.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b3fff700 1 -- 172.21.15.175:0/4225189770 learned_addr learned my addr 172.21.15.175:0/4225189770 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b37fe700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69b406b080 0x7f69b40c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.923+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f69b3fff700 1 -- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 msgr2=0x7f69b40c35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f69b3fff700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40c35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f69b3fff700 1 -- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69b406b080 msgr2=0x7f69b40c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f69b3fff700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69b406b080 0x7f69b40c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f69b3fff700 1 -- 172.21.15.175:0/4225189770 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac009420 con 0x7f69b40c57f0 2023-10-10T18:45:46.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f69b3fff700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b40c57f0 0x7f69b40ca200 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f69b406cc30 tx=0x7f69ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f699b7fe700 1 -- 172.21.15.175:0/4225189770 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac01f070 con 0x7f69b40c57f0 2023-10-10T18:45:46.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f699b7fe700 1 -- 172.21.15.175:0/4225189770 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac002f70 con 0x7f69b40c57f0 2023-10-10T18:45:46.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f69b88cc700 1 -- 172.21.15.175:0/4225189770 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b4150290 con 0x7f69b40c57f0 2023-10-10T18:45:46.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f699b7fe700 1 -- 172.21.15.175:0/4225189770 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00b690 con 0x7f69b40c57f0 2023-10-10T18:45:46.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.924+0000 7f69b88cc700 1 -- 172.21.15.175:0/4225189770 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b414fc50 con 0x7f69b40c57f0 2023-10-10T18:45:46.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.925+0000 7f699b7fe700 1 -- 172.21.15.175:0/4225189770 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f69ac0118c0 con 0x7f69b40c57f0 2023-10-10T18:45:46.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.926+0000 7f699b7fe700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f69a0066f60 0x7f69a0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.926+0000 7f699b7fe700 1 -- 172.21.15.175:0/4225189770 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f69ac085910 con 0x7f69b40c57f0 2023-10-10T18:45:46.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.926+0000 7f69b37fe700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f69a0066f60 0x7f69a0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.926+0000 7f69b37fe700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f69a0066f60 0x7f69a0069410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f69a40040a0 tx=0x7f69a4002690 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.928+0000 7f69c2e27b80 1 Processor -- start 2023-10-10T18:45:46.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.928+0000 7f69c2e27b80 1 -- start start 2023-10-10T18:45:46.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.928+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 0x5653ac247210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.928+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac247750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.928+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac2467d0 0x5653ac23c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.928+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ac15ba70 con 0x5653ac207f30 2023-10-10T18:45:46.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.928+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ac159270 con 0x5653ac2467d0 2023-10-10T18:45:46.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.928+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653ac155060 con 0x5653ac246400 2023-10-10T18:45:46.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac247750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac247750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37480/0 (socket says 172.21.15.175:37480) 2023-10-10T18:45:46.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b37fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 0x5653ac247210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 -- 172.21.15.175:0/3089033605 learned_addr learned my addr 172.21.15.175:0/3089033605 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b3fff700 1 --2- 172.21.15.175:0/3089033605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac2467d0 0x5653ac23c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 -- 172.21.15.175:0/3089033605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 msgr2=0x5653ac247210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/3089033605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 0x5653ac247210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 -- 172.21.15.175:0/3089033605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac2467d0 msgr2=0x5653ac23c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/3089033605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac2467d0 0x5653ac23c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 -- 172.21.15.175:0/3089033605 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653abfef270 con 0x5653ac207f30 2023-10-10T18:45:46.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/3089033605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac247750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f69a80017c0 tx=0x7f69a8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9395edf2621c4287 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f6998ff9700 1 -- 172.21.15.175:0/3089033605 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8005b10 con 0x5653ac207f30 2023-10-10T18:45:46.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f6998ff9700 1 -- 172.21.15.175:0/3089033605 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a8005c70 con 0x5653ac207f30 2023-10-10T18:45:46.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.929+0000 7f6998ff9700 1 -- 172.21.15.175:0/3089033605 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a80070d0 con 0x5653ac207f30 2023-10-10T18:45:46.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 -- 172.21.15.175:0/3089033605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 msgr2=0x5653ac247750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/3089033605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac247750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f69a80017c0 tx=0x7f69a8009040 comp rx=0 tx=0).stop 2023-10-10T18:45:46.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 -- 172.21.15.175:0/3089033605 shutdown_connections 2023-10-10T18:45:46.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/3089033605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac2467d0 0x5653ac23c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/3089033605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 0x5653ac247210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/3089033605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac247750 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 -- 172.21.15.175:0/3089033605 >> 172.21.15.175:0/3089033605 conn(0x5653ac2044a0 msgr2=0x5653ac204870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 -- 172.21.15.175:0/3089033605 shutdown_connections 2023-10-10T18:45:46.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 -- 172.21.15.175:0/3089033605 wait complete. 2023-10-10T18:45:46.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 -- start start 2023-10-10T18:45:46.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.930+0000 7f69c2e27b80 1 Processor -- start 2023-10-10T18:45:46.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac23af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac257140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac2467d0 0x5653ac25a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ac100890 con 0x5653ac207f30 2023-10-10T18:45:46.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ac11bea0 con 0x5653ac246400 2023-10-10T18:45:46.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653abfd7e30 con 0x5653ac2467d0 2023-10-10T18:45:46.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69c2e27b80 10 client.0 _ll_register_callbacks cb 0x5653ac1c82c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:46.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac23af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac23af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37488/0 (socket says 172.21.15.175:37488) 2023-10-10T18:45:46.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b37fe700 1 -- 172.21.15.175:0/1322843733 learned_addr learned my addr 172.21.15.175:0/1322843733 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac257140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b3fff700 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac2467d0 0x5653ac25a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b37fe700 1 -- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac2467d0 msgr2=0x5653ac25a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b37fe700 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac2467d0 0x5653ac25a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b37fe700 1 -- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 msgr2=0x5653ac257140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b37fe700 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac257140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b37fe700 1 -- 172.21.15.175:0/1322843733 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c003570 con 0x5653ac207f30 2023-10-10T18:45:46.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.931+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac257140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:46.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69b37fe700 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac23af70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f69a4003050 tx=0x7f69a4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40427a2a17bc16a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f6991ffb700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:46.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f6991ffb700 1 -- 172.21.15.175:0/1322843733 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a4006ba0 con 0x5653ac207f30 2023-10-10T18:45:46.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f6991ffb700 1 -- 172.21.15.175:0/1322843733 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f69a4019070 con 0x5653ac207f30 2023-10-10T18:45:46.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 1 -- 172.21.15.175:0/1322843733 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5653ac1f9cd0 con 0x5653ac207f30 2023-10-10T18:45:46.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f6991ffb700 1 -- 172.21.15.175:0/1322843733 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69a4014450 con 0x5653ac207f30 2023-10-10T18:45:46.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 10 client.4719 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:46.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 1 -- 172.21.15.175:0/1322843733 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5653abf91fe0 con 0x5653ac207f30 2023-10-10T18:45:46.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f6991ffb700 1 -- 172.21.15.175:0/1322843733 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5653ac1f9cd0 con 0x5653ac207f30 2023-10-10T18:45:46.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f6991ffb700 1 -- 172.21.15.175:0/1322843733 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7f69a40145b0 con 0x5653ac207f30 2023-10-10T18:45:46.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 10 client.4719 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:46.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 10 client.4719 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-10T18:45:46.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 20 client.4719 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:46.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] conn(0x5653ac234b90 0x5653ac234f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 4 client.4719 mds_command: new command op to 4250 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T18:45:46.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69c2e27b80 1 -- 172.21.15.175:0/1322843733 --> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653ac1d6aa0 con 0x5653ac234b90 2023-10-10T18:45:46.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.932+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] conn(0x5653ac234b90 0x5653ac234f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] conn(0x5653ac234b90 0x5653ac234f60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=205637a47765a5df server_cookie=c78942246a892937 in_seq=0 out_seq=0 2023-10-10T18:45:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f6991ffb700 10 client.4719 ms_handle_connect on v2:172.21.15.191:6838/246221790 2023-10-10T18:45:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f6991ffb700 1 -- 172.21.15.175:0/1322843733 <== mds.2 v2:172.21.15.191:6838/246221790 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x5653ac1d6aa0 con 0x5653ac234b90 2023-10-10T18:45:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f6991ffb700 10 client.4719 handle_command_reply: tid=0 2023-10-10T18:45:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f69c2e27b80 1 client.4719 shutdown 2023-10-10T18:45:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f69c2e27b80 1 -- 172.21.15.175:0/1322843733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 msgr2=0x5653ac23af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac23af70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f69a4003050 tx=0x7f69a4015000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f69c2e27b80 1 -- 172.21.15.175:0/1322843733 shutdown_connections 2023-10-10T18:45:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] conn(0x5653ac234b90 0x5653ac234f60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac2467d0 0x5653ac25a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f6991ffb700 0 client.4719 ms_handle_reset on v2:172.21.15.191:6838/246221790 2023-10-10T18:45:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.933+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac207f30 0x5653ac23af70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.934+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/1322843733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac257140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.934+0000 7f69c2e27b80 1 -- 172.21.15.175:0/1322843733 >> 172.21.15.175:0/1322843733 conn(0x5653ac2044a0 msgr2=0x5653ac205150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.934+0000 7f69c2e27b80 1 -- 172.21.15.175:0/1322843733 shutdown_connections 2023-10-10T18:45:46.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.934+0000 7f69c2e27b80 1 -- 172.21.15.175:0/1322843733 wait complete. 2023-10-10T18:45:46.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.934+0000 7f69c2e27b80 20 client.4719 trim_cache size 0 max 16384 2023-10-10T18:45:46.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69c2e27b80 1 Processor -- start 2023-10-10T18:45:46.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69c2e27b80 1 -- start start 2023-10-10T18:45:46.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac207f30 0x5653ac235360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 0x5653ac205150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac205690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ac1fc580 con 0x5653ac2467d0 2023-10-10T18:45:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ac24ac60 con 0x5653ac207f30 2023-10-10T18:45:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653ac1fd9e0 con 0x5653ac246400 2023-10-10T18:45:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69b3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac205690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69b3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac205690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37498/0 (socket says 172.21.15.175:37498) 2023-10-10T18:45:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69b3fff700 1 -- 172.21.15.175:0/154830413 learned_addr learned my addr 172.21.15.175:0/154830413 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.951+0000 7f69b37fe700 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac207f30 0x5653ac235360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 0x5653ac205150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69b3fff700 1 -- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 msgr2=0x5653ac205150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69b3fff700 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 0x5653ac205150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69b3fff700 1 -- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac207f30 msgr2=0x5653ac235360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69b3fff700 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac207f30 0x5653ac235360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69b3fff700 1 -- 172.21.15.175:0/154830413 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653ac123780 con 0x5653ac2467d0 2023-10-10T18:45:46.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69b37fe700 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac207f30 0x5653ac235360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:46.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69b3fff700 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac205690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009860 tx=0x7f69ac059040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0d386d0bb5e922 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69997fa700 1 -- 172.21.15.175:0/154830413 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac06b070 con 0x5653ac2467d0 2023-10-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69997fa700 1 -- 172.21.15.175:0/154830413 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac056f40 con 0x5653ac2467d0 2023-10-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69997fa700 1 -- 172.21.15.175:0/154830413 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac056840 con 0x5653ac2467d0 2023-10-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 -- 172.21.15.175:0/154830413 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 msgr2=0x5653ac205690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac205690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009860 tx=0x7f69ac059040 comp rx=0 tx=0).stop 2023-10-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 -- 172.21.15.175:0/154830413 shutdown_connections 2023-10-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac205690 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac207f30 0x5653ac235360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/154830413 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac246400 0x5653ac205150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 -- 172.21.15.175:0/154830413 >> 172.21.15.175:0/154830413 conn(0x5653ac2044a0 msgr2=0x5653ac2440a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 -- 172.21.15.175:0/154830413 shutdown_connections 2023-10-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.952+0000 7f69c2e27b80 1 -- 172.21.15.175:0/154830413 wait complete. 2023-10-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.953+0000 7f69c2e27b80 1 -- start start 2023-10-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.953+0000 7f69c2e27b80 1 Processor -- start 2023-10-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac207f30 0x5653ac206890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac255f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac2a6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ac2b11f0 con 0x5653ac2467d0 2023-10-10T18:45:46.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ac23b730 con 0x5653ac246400 2023-10-10T18:45:46.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653ac2072a0 con 0x5653ac207f30 2023-10-10T18:45:46.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 10 client.0 _ll_register_callbacks cb 0x5653ac1c82c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:46.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac2a6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac2a6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37508/0 (socket says 172.21.15.175:37508) 2023-10-10T18:45:46.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 -- 172.21.15.175:0/2868739102 learned_addr learned my addr 172.21.15.175:0/2868739102 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:46.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b37fe700 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac207f30 0x5653ac206890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b2ffd700 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac255f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 -- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac207f30 msgr2=0x5653ac206890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac207f30 0x5653ac206890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 -- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 msgr2=0x5653ac255f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac255f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 -- 172.21.15.175:0/2868739102 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b414fc50 con 0x5653ac2467d0 2023-10-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69b3fff700 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac2a6bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0567d0 tx=0x7f69ac02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd4e2ca1b9ab06e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f6991ffb700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f6991ffb700 1 -- 172.21.15.175:0/2868739102 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac06b070 con 0x5653ac2467d0 2023-10-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f6991ffb700 1 -- 172.21.15.175:0/2868739102 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f69ac05d7e0 con 0x5653ac2467d0 2023-10-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 1 -- 172.21.15.175:0/2868739102 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69a8005c70 con 0x5653ac2467d0 2023-10-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f6991ffb700 1 -- 172.21.15.175:0/2868739102 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69ac072490 con 0x5653ac2467d0 2023-10-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 10 client.4725 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.954+0000 7f69c2e27b80 1 -- 172.21.15.175:0/2868739102 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5653ac10a470 con 0x5653ac2467d0 2023-10-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f6991ffb700 1 -- 172.21.15.175:0/2868739102 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac065080 con 0x5653ac2467d0 2023-10-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f6991ffb700 1 -- 172.21.15.175:0/2868739102 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7f69ac055d30 con 0x5653ac2467d0 2023-10-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69c2e27b80 10 client.4725 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69c2e27b80 10 client.4725 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69c2e27b80 20 client.4725 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] conn(0x5653ac2acf30 0x5653ac2ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69c2e27b80 4 client.4725 mds_command: new command op to 4250 tid=0[{"prefix": "status"}] 2023-10-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69c2e27b80 1 -- 172.21.15.175:0/2868739102 --> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5653ac0f7e00 con 0x5653ac2acf30 2023-10-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69b37fe700 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] conn(0x5653ac2acf30 0x5653ac2ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f6991ffb700 10 client.4725 ms_handle_connect on v2:172.21.15.191:6838/246221790 2023-10-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69b37fe700 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] conn(0x5653ac2acf30 0x5653ac2ad300 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=153c69b00d8bf44f server_cookie=ce3cbc76b20281ca in_seq=0 out_seq=0 2023-10-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f6991ffb700 1 -- 172.21.15.175:0/2868739102 <== mds.2 v2:172.21.15.191:6838/246221790 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5653ac0f7e00 con 0x5653ac2acf30 2023-10-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f6991ffb700 10 client.4725 handle_command_reply: tid=0 2023-10-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.955+0000 7f69c2e27b80 1 client.4725 shutdown 2023-10-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 -- 172.21.15.175:0/2868739102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 msgr2=0x5653ac2a6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac2a6bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0567d0 tx=0x7f69ac02f000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 -- 172.21.15.175:0/2868739102 shutdown_connections 2023-10-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:6838/246221790,v1:172.21.15.191:6839/246221790] conn(0x5653ac2acf30 0x5653ac2ad300 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5653ac2467d0 0x5653ac2a6bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5653ac207f30 0x5653ac206890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 --2- 172.21.15.175:0/2868739102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653ac246400 0x5653ac255f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f6991ffb700 0 client.4725 ms_handle_reset on v2:172.21.15.191:6838/246221790 2023-10-10T18:45:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 -- 172.21.15.175:0/2868739102 >> 172.21.15.175:0/2868739102 conn(0x5653ac2044a0 msgr2=0x5653ac235360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 -- 172.21.15.175:0/2868739102 shutdown_connections 2023-10-10T18:45:46.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 1 -- 172.21.15.175:0/2868739102 wait complete. 2023-10-10T18:45:46.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69c2e27b80 20 client.4725 trim_cache size 0 max 16384 2023-10-10T18:45:46.991 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T18:45:46.992 INFO:teuthology.orchestra.run.smithi175.stdout: "cluster_fsid": "7f0fe9e3-2088-4ba2-8a54-92a2be521afd", 2023-10-10T18:45:46.992 INFO:teuthology.orchestra.run.smithi175.stdout: "whoami": 2, 2023-10-10T18:45:46.992 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 4250, 2023-10-10T18:45:46.992 INFO:teuthology.orchestra.run.smithi175.stdout: "want_state": "up:active", 2023-10-10T18:45:46.993 INFO:teuthology.orchestra.run.smithi175.stdout: "state": "up:active", 2023-10-10T18:45:46.993 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs", 2023-10-10T18:45:46.993 INFO:teuthology.orchestra.run.smithi175.stdout: "rank_uptime": 23.445157966, 2023-10-10T18:45:46.993 INFO:teuthology.orchestra.run.smithi175.stdout: "mdsmap_epoch": 17, 2023-10-10T18:45:46.994 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch": 21, 2023-10-10T18:45:46.994 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch_barrier": 21, 2023-10-10T18:45:46.994 INFO:teuthology.orchestra.run.smithi175.stdout: "uptime": 42.438381333999999 2023-10-10T18:45:46.994 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T18:45:46.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 -- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f69a0066f60 msgr2=0x7f69a0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f69a0066f60 0x7f69a0069410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f69a40040a0 tx=0x7f69a4002690 comp rx=0 tx=0).stop 2023-10-10T18:45:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 -- 172.21.15.175:0/4225189770 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b40c57f0 msgr2=0x7f69b40ca200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b40c57f0 0x7f69b40ca200 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f69b406cc30 tx=0x7f69ac017000 comp rx=0 tx=0).stop 2023-10-10T18:45:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 -- 172.21.15.175:0/4225189770 shutdown_connections 2023-10-10T18:45:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f69a0066f60 0x7f69a0069410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b40c57f0 0x7f69b40ca200 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69b406b080 0x7f69b40c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 --2- 172.21.15.175:0/4225189770 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69b40b0080 0x7f69b40c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 -- 172.21.15.175:0/4225189770 >> 172.21.15.175:0/4225189770 conn(0x7f69b406c640 msgr2=0x7f69b4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:46.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 -- 172.21.15.175:0/4225189770 shutdown_connections 2023-10-10T18:45:46.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:46.956+0000 7f69b88cc700 1 -- 172.21.15.175:0/4225189770 wait complete. 2023-10-10T18:45:46.999 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-10-10T18:45:47.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.114+0000 7f1734b9d700 1 Processor -- start 2023-10-10T18:45:47.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.114+0000 7f1734b9d700 1 -- start start 2023-10-10T18:45:47.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f1734b9d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1730155fd0 0x7f17300ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f1734b9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17301563a0 0x7f17300c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f1734b9d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17300c21b0 con 0x7f17301568d0 2023-10-10T18:45:47.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f1734b9d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17300cd5f0 con 0x7f1730155fd0 2023-10-10T18:45:47.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f1734b9d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17300c2640 con 0x7f17301563a0 2023-10-10T18:45:47.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f172effd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f17301568d0 0x7f17301571b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51490/0 (socket says 172.21.15.175:51490) 2023-10-10T18:45:47.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f172effd700 1 -- 172.21.15.175:0/3986533949 learned_addr learned my addr 172.21.15.175:0/3986533949 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:47.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f172f7fe700 1 --2- 172.21.15.175:0/3986533949 >> v2:172.21.15.191:3300/0 conn(0x7f1730155fd0 0x7f17300ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f172ffff700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17301563a0 0x7f17300c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915689367 0 0) 0x7f17300c21b0 con 0x7f17301568d0 2023-10-10T18:45:47.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.115+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1718003570 con 0x7f17301568d0 2023-10-10T18:45:47.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1264419118 0 0) 0x7f1718003570 con 0x7f17301568d0 2023-10-10T18:45:47.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17301563a0 msgr2=0x7f17300c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17301563a0 0x7f17300c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 >> v2:172.21.15.191:3300/0 conn(0x7f1730155fd0 msgr2=0x7f17300ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 --2- 172.21.15.175:0/3986533949 >> v2:172.21.15.191:3300/0 conn(0x7f1730155fd0 0x7f17300ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17300c15a0 con 0x7f17301568d0 2023-10-10T18:45:47.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f1724001470 con 0x7f17301568d0 2023-10-10T18:45:47.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172f7fe700 1 --2- 172.21.15.175:0/3986533949 >> v2:172.21.15.191:3300/0 conn(0x7f1730155fd0 0x7f17300ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:47.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 >> v1:172.21.15.175:6789/0 conn(0x7f17301568d0 legacy=0x7f17301571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1718003a90 0x7f1718003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1718004810 0x7f1718004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17301563a0 0x7f17180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17300cd5f0 con 0x7f1718004810 2023-10-10T18:45:47.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17300c2640 con 0x7f1718003a90 2023-10-10T18:45:47.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171800ba60 con 0x7f17301563a0 2023-10-10T18:45:47.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172effd700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1718004810 0x7f1718004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172ffff700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17301563a0 0x7f17180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172f7fe700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1718003a90 0x7f1718003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172effd700 1 -- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17301563a0 msgr2=0x7f17180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172effd700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17301563a0 0x7f17180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172effd700 1 -- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1718003a90 msgr2=0x7f1718003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172effd700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1718003a90 0x7f1718003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172effd700 1 -- 172.21.15.175:0/3986533949 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718003570 con 0x7f1718004810 2023-10-10T18:45:47.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172effd700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1718004810 0x7f1718004c80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f172400c3f0 tx=0x7f1724001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58867cde68c713a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400cc80 con 0x7f1718004810 2023-10-10T18:45:47.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172400cde0 con 0x7f1718004810 2023-10-10T18:45:47.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.116+0000 7f172e7fc700 1 -- 172.21.15.175:0/3986533949 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724005b70 con 0x7f1718004810 2023-10-10T18:45:47.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.117+0000 7f1734b9d700 1 -- 172.21.15.175:0/3986533949 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1718004810 msgr2=0x7f1718004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.117+0000 7f1734b9d700 1 --2- 172.21.15.175:0/3986533949 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1718004810 0x7f1718004c80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f172400c3f0 tx=0x7f1724001e50 comp rx=0 tx=0).stop 2023-10-10T18:45:47.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.117+0000 7f172ffff700 1 -- 172.21.15.175:0/3986533949 reap_dead start 2023-10-10T18:45:47.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.117+0000 7f1734b9d700 1 -- 172.21.15.175:0/3986533949 shutdown_connections 2023-10-10T18:45:47.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.117+0000 7f1734b9d700 1 -- 172.21.15.175:0/3986533949 >> 172.21.15.175:0/3986533949 conn(0x7f1730068a20 msgr2=0x7f1730068e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.117+0000 7f1734b9d700 1 -- 172.21.15.175:0/3986533949 shutdown_connections 2023-10-10T18:45:47.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.117+0000 7f1734b9d700 1 -- 172.21.15.175:0/3986533949 wait complete. 2023-10-10T18:45:47.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.118+0000 7f1734b9d700 1 Processor -- start 2023-10-10T18:45:47.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.118+0000 7f1734b9d700 1 -- start start 2023-10-10T18:45:47.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f1734b9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1718004810 0x7f17300ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f1734b9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1730155320 0x7f17300ae720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f1734b9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17300b0ad0 0x7f17300c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f1734b9d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730152790 con 0x7f17300b0ad0 2023-10-10T18:45:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f1734b9d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730152520 con 0x7f1730155320 2023-10-10T18:45:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f1734b9d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1730169c90 con 0x7f1718004810 2023-10-10T18:45:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17300b0ad0 0x7f17300c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17300b0ad0 0x7f17300c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37526/0 (socket says 172.21.15.175:37526) 2023-10-10T18:45:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 -- 172.21.15.175:0/4290921207 learned_addr learned my addr 172.21.15.175:0/4290921207 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172f7fe700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1718004810 0x7f17300ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172effd700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1730155320 0x7f17300ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 -- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1718004810 msgr2=0x7f17300ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1718004810 0x7f17300ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 -- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1730155320 msgr2=0x7f17300ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1730155320 0x7f17300ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 -- 172.21.15.175:0/4290921207 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730169f70 con 0x7f17300b0ad0 2023-10-10T18:45:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172f7fe700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1718004810 0x7f17300ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f172ffff700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17300b0ad0 0x7f17300c5770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1728006f00 tx=0x7f17280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f17177fe700 1 -- 172.21.15.175:0/4290921207 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172800f940 con 0x7f17300b0ad0 2023-10-10T18:45:47.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f17177fe700 1 -- 172.21.15.175:0/4290921207 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172800faa0 con 0x7f17300b0ad0 2023-10-10T18:45:47.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f17177fe700 1 -- 172.21.15.175:0/4290921207 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172800fe40 con 0x7f17300b0ad0 2023-10-10T18:45:47.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.119+0000 7f1734b9d700 1 -- 172.21.15.175:0/4290921207 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17300cc5e0 con 0x7f17300b0ad0 2023-10-10T18:45:47.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.120+0000 7f1734b9d700 1 -- 172.21.15.175:0/4290921207 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17300c5cb0 con 0x7f17300b0ad0 2023-10-10T18:45:47.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.121+0000 7f17177fe700 1 -- 172.21.15.175:0/4290921207 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1728014650 con 0x7f17300b0ad0 2023-10-10T18:45:47.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.121+0000 7f17177fe700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f171c066f60 0x7f171c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.121+0000 7f17177fe700 1 -- 172.21.15.175:0/4290921207 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1728084a90 con 0x7f17300b0ad0 2023-10-10T18:45:47.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.122+0000 7f172f7fe700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f171c066f60 0x7f171c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.122+0000 7f172f7fe700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f171c066f60 0x7f171c069410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17200094d0 tx=0x7f172000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.123+0000 7f173f0f8b80 1 Processor -- start 2023-10-10T18:45:47.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.123+0000 7f173f0f8b80 1 -- start start 2023-10-10T18:45:47.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c24400 0x562597c25210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c25750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c247d0 0x562597c1a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597b39a70 con 0x562597c24400 2023-10-10T18:45:47.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597b37270 con 0x562597be5f30 2023-10-10T18:45:47.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x562597b33060 con 0x562597c247d0 2023-10-10T18:45:47.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c24400 0x562597c25210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c24400 0x562597c25210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37536/0 (socket says 172.21.15.175:37536) 2023-10-10T18:45:47.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172f7fe700 1 -- 172.21.15.175:0/1018193640 learned_addr learned my addr 172.21.15.175:0/1018193640 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172effd700 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c25750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172ffff700 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c247d0 0x562597c1a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172f7fe700 1 -- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c247d0 msgr2=0x562597c1a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c247d0 0x562597c1a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172f7fe700 1 -- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 msgr2=0x562597c25750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c25750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172f7fe700 1 -- 172.21.15.175:0/1018193640 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625979cd270 con 0x562597c24400 2023-10-10T18:45:47.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.124+0000 7f172effd700 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c25750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c24400 0x562597c25210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1720016890 tx=0x7f172001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7c316a2932e8e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f1714ff9700 1 -- 172.21.15.175:0/1018193640 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720027070 con 0x562597c24400 2023-10-10T18:45:47.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f1714ff9700 1 -- 172.21.15.175:0/1018193640 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172000f040 con 0x562597c24400 2023-10-10T18:45:47.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f1714ff9700 1 -- 172.21.15.175:0/1018193640 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172000e040 con 0x562597c24400 2023-10-10T18:45:47.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1018193640 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c24400 msgr2=0x562597c25210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c24400 0x562597c25210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1720016890 tx=0x7f172001a000 comp rx=0 tx=0).stop 2023-10-10T18:45:47.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1018193640 shutdown_connections 2023-10-10T18:45:47.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c247d0 0x562597c1a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c24400 0x562597c25210 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1018193640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c25750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1018193640 >> 172.21.15.175:0/1018193640 conn(0x562597be24a0 msgr2=0x562597be2870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.125+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1018193640 shutdown_connections 2023-10-10T18:45:47.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.126+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1018193640 wait complete. 2023-10-10T18:45:47.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.126+0000 7f173f0f8b80 1 -- start start 2023-10-10T18:45:47.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.126+0000 7f173f0f8b80 1 Processor -- start 2023-10-10T18:45:47.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597c18460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c38790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c251b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597ade890 con 0x562597be5f30 2023-10-10T18:45:47.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597af9ea0 con 0x562597c247d0 2023-10-10T18:45:47.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625979b5e30 con 0x562597c24400 2023-10-10T18:45:47.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f173f0f8b80 10 client.0 _ll_register_callbacks cb 0x562597ba62c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:47.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597c18460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597c18460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37552/0 (socket says 172.21.15.175:37552) 2023-10-10T18:45:47.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 -- 172.21.15.175:0/1928275031 learned_addr learned my addr 172.21.15.175:0/1928275031 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172ffff700 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c251b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172effd700 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c38790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 -- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 msgr2=0x562597c38790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c38790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 -- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 msgr2=0x562597c251b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c251b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 -- 172.21.15.175:0/1928275031 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1720016570 con 0x562597be5f30 2023-10-10T18:45:47.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172effd700 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c38790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597c18460 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1720015390 tx=0x7f172001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e124a7fed695916 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f170dffb700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:47.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f170dffb700 1 -- 172.21.15.175:0/1928275031 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172000e030 con 0x562597be5f30 2023-10-10T18:45:47.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f170dffb700 1 -- 172.21.15.175:0/1928275031 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1720023650 con 0x562597be5f30 2023-10-10T18:45:47.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.127+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1928275031 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562597bd7cd0 con 0x562597be5f30 2023-10-10T18:45:47.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f170dffb700 1 -- 172.21.15.175:0/1928275031 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1720027040 con 0x562597be5f30 2023-10-10T18:45:47.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f173f0f8b80 10 client.4737 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:47.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1928275031 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x56259796ffe0 con 0x562597be5f30 2023-10-10T18:45:47.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f170dffb700 1 -- 172.21.15.175:0/1928275031 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562597bd7cd0 con 0x562597be5f30 2023-10-10T18:45:47.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f170dffb700 1 -- 172.21.15.175:0/1928275031 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7f1720023a40 con 0x562597be5f30 2023-10-10T18:45:47.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f173f0f8b80 10 client.4737 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:47.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f173f0f8b80 10 client.4737 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-10T18:45:47.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f173f0f8b80 20 client.4737 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:47.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] conn(0x562597c1e410 0x562597c1e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f173f0f8b80 4 client.4737 mds_command: new command op to 4251 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T18:45:47.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1928275031 --> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562597bb4aa0 con 0x562597c1e410 2023-10-10T18:45:47.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f172effd700 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] conn(0x562597c1e410 0x562597c1e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f170dffb700 10 client.4737 ms_handle_connect on v2:172.21.15.175:6835/2966961140 2023-10-10T18:45:47.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.128+0000 7f172effd700 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] conn(0x562597c1e410 0x562597c1e7e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ecea3f505a214a9 server_cookie=5e4c48244af8ef60 in_seq=0 out_seq=0 2023-10-10T18:45:47.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f170dffb700 1 -- 172.21.15.175:0/1928275031 <== mds.1 v2:172.21.15.175:6835/2966961140 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x562597bb4aa0 con 0x562597c1e410 2023-10-10T18:45:47.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f170dffb700 10 client.4737 handle_command_reply: tid=0 2023-10-10T18:45:47.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 client.4737 shutdown 2023-10-10T18:45:47.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1928275031 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 msgr2=0x562597c18460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597c18460 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1720015390 tx=0x7f172001d000 comp rx=0 tx=0).stop 2023-10-10T18:45:47.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1928275031 shutdown_connections 2023-10-10T18:45:47.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] conn(0x562597c1e410 0x562597c1e7e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c251b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f170dffb700 0 client.4737 ms_handle_reset on v2:172.21.15.175:6835/2966961140 2023-10-10T18:45:47.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597c18460 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1928275031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c38790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1928275031 >> 172.21.15.175:0/1928275031 conn(0x562597be24a0 msgr2=0x562597c25cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1928275031 shutdown_connections 2023-10-10T18:45:47.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1928275031 wait complete. 2023-10-10T18:45:47.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.129+0000 7f173f0f8b80 20 client.4737 trim_cache size 0 max 16384 2023-10-10T18:45:47.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.146+0000 7f173f0f8b80 1 Processor -- start 2023-10-10T18:45:47.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.146+0000 7f173f0f8b80 1 -- start start 2023-10-10T18:45:47.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597bd9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c23710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c23c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597be5bf0 con 0x562597be5f30 2023-10-10T18:45:47.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597c3ef50 con 0x562597c247d0 2023-10-10T18:45:47.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x562597bd9fe0 con 0x562597c24400 2023-10-10T18:45:47.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597bd9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597bd9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37562/0 (socket says 172.21.15.175:37562) 2023-10-10T18:45:47.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 -- 172.21.15.175:0/3223197245 learned_addr learned my addr 172.21.15.175:0/3223197245 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172effd700 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c23710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172ffff700 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c23c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 -- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 msgr2=0x562597c23710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c23710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 -- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 msgr2=0x562597c23c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c23c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 -- 172.21.15.175:0/3223197245 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562597b01780 con 0x562597be5f30 2023-10-10T18:45:47.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172effd700 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c23710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:47.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172ffff700 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c23c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:47.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f172f7fe700 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597bd9110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f17200167e0 tx=0x7f172002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2943cdf9a003b67 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f17157fa700 1 -- 172.21.15.175:0/3223197245 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172000e030 con 0x562597be5f30 2023-10-10T18:45:47.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f17157fa700 1 -- 172.21.15.175:0/3223197245 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1720023a40 con 0x562597be5f30 2023-10-10T18:45:47.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.147+0000 7f17157fa700 1 -- 172.21.15.175:0/3223197245 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720027040 con 0x562597be5f30 2023-10-10T18:45:47.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 -- 172.21.15.175:0/3223197245 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 msgr2=0x562597bd9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597bd9110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f17200167e0 tx=0x7f172002f000 comp rx=0 tx=0).stop 2023-10-10T18:45:47.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 -- 172.21.15.175:0/3223197245 shutdown_connections 2023-10-10T18:45:47.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597c247d0 0x562597c23c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597be5f30 0x562597bd9110 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/3223197245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c23710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 -- 172.21.15.175:0/3223197245 >> 172.21.15.175:0/3223197245 conn(0x562597be24a0 msgr2=0x562597c19e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 -- 172.21.15.175:0/3223197245 shutdown_connections 2023-10-10T18:45:47.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 -- 172.21.15.175:0/3223197245 wait complete. 2023-10-10T18:45:47.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 -- start start 2023-10-10T18:45:47.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.148+0000 7f173f0f8b80 1 Processor -- start 2023-10-10T18:45:47.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c8ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c37e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f173f0f8b80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c247d0 0x562597c84060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597c13940 con 0x562597c247d0 2023-10-10T18:45:47.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597c8d800 con 0x562597be5f30 2023-10-10T18:45:47.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f173f0f8b80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x562597c282d0 con 0x562597c24400 2023-10-10T18:45:47.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f173f0f8b80 10 client.0 _ll_register_callbacks cb 0x562597ba62c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:47.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c247d0 0x562597c84060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c247d0 0x562597c84060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37574/0 (socket says 172.21.15.175:37574) 2023-10-10T18:45:47.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172ffff700 1 -- 172.21.15.175:0/1587215011 learned_addr learned my addr 172.21.15.175:0/1587215011 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172effd700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c37e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c8ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172ffff700 1 -- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 msgr2=0x562597c37e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172ffff700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c37e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172ffff700 1 -- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 msgr2=0x562597c8ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172ffff700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c8ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172ffff700 1 -- 172.21.15.175:0/1587215011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56259796ffe0 con 0x562597c247d0 2023-10-10T18:45:47.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c8ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.149+0000 7f172effd700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c37e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f172ffff700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c247d0 0x562597c84060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1728082fb0 tx=0x7f1728058040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a28607af90b9382 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f170dffb700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:47.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f170dffb700 1 -- 172.21.15.175:0/1587215011 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172805c9e0 con 0x562597c247d0 2023-10-10T18:45:47.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f170dffb700 1 -- 172.21.15.175:0/1587215011 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1728006c50 con 0x562597c247d0 2023-10-10T18:45:47.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1587215011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f172000f040 con 0x562597c247d0 2023-10-10T18:45:47.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f170dffb700 1 -- 172.21.15.175:0/1587215011 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f172805c550 con 0x562597c247d0 2023-10-10T18:45:47.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 10 client.4743 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:47.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1587215011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x562597ae8470 con 0x562597c247d0 2023-10-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f170dffb700 1 -- 172.21.15.175:0/1587215011 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172000f040 con 0x562597c247d0 2023-10-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f170dffb700 1 -- 172.21.15.175:0/1587215011 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7f1728051830 con 0x562597c247d0 2023-10-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 10 client.4743 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 10 client.4743 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 20 client.4743 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] conn(0x562597c8a550 0x562597c8a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 4 client.4743 mds_command: new command op to 4251 tid=0[{"prefix": "status"}] 2023-10-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1587215011 --> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562597ad5e00 con 0x562597c8a550 2023-10-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] conn(0x562597c8a550 0x562597c8a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f170dffb700 10 client.4743 ms_handle_connect on v2:172.21.15.175:6835/2966961140 2023-10-10T18:45:47.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.150+0000 7f172f7fe700 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] conn(0x562597c8a550 0x562597c8a920 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cea0faec1af2b0a8 server_cookie=79e90d21ff17f71f in_seq=0 out_seq=0 2023-10-10T18:45:47.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f170dffb700 1 -- 172.21.15.175:0/1587215011 <== mds.1 v2:172.21.15.175:6835/2966961140 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x562597ad5e00 con 0x562597c8a550 2023-10-10T18:45:47.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f170dffb700 10 client.4743 handle_command_reply: tid=0 2023-10-10T18:45:47.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 client.4743 shutdown 2023-10-10T18:45:47.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1587215011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c247d0 msgr2=0x562597c84060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c247d0 0x562597c84060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1728082fb0 tx=0x7f1728058040 comp rx=0 tx=0).stop 2023-10-10T18:45:47.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1587215011 shutdown_connections 2023-10-10T18:45:47.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.175:6835/2966961140,v1:172.21.15.175:6837/2966961140] conn(0x562597c8a550 0x562597c8a920 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x562597c247d0 0x562597c84060 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562597be5f30 0x562597c8ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f170dffb700 0 client.4743 ms_handle_reset on v2:172.21.15.175:6835/2966961140 2023-10-10T18:45:47.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 --2- 172.21.15.175:0/1587215011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x562597c24400 0x562597c37e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1587215011 >> 172.21.15.175:0/1587215011 conn(0x562597be24a0 msgr2=0x562597c23710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1587215011 shutdown_connections 2023-10-10T18:45:47.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 1 -- 172.21.15.175:0/1587215011 wait complete. 2023-10-10T18:45:47.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f173f0f8b80 20 client.4743 trim_cache size 0 max 16384 2023-10-10T18:45:47.190 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T18:45:47.190 INFO:teuthology.orchestra.run.smithi175.stdout: "cluster_fsid": "7f0fe9e3-2088-4ba2-8a54-92a2be521afd", 2023-10-10T18:45:47.191 INFO:teuthology.orchestra.run.smithi175.stdout: "whoami": 1, 2023-10-10T18:45:47.191 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 4251, 2023-10-10T18:45:47.191 INFO:teuthology.orchestra.run.smithi175.stdout: "want_state": "up:active", 2023-10-10T18:45:47.191 INFO:teuthology.orchestra.run.smithi175.stdout: "state": "up:active", 2023-10-10T18:45:47.191 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs", 2023-10-10T18:45:47.192 INFO:teuthology.orchestra.run.smithi175.stdout: "rank_uptime": 29.700609074999999, 2023-10-10T18:45:47.192 INFO:teuthology.orchestra.run.smithi175.stdout: "mdsmap_epoch": 17, 2023-10-10T18:45:47.192 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch": 21, 2023-10-10T18:45:47.192 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch_barrier": 21, 2023-10-10T18:45:47.192 INFO:teuthology.orchestra.run.smithi175.stdout: "uptime": 42.654362061 2023-10-10T18:45:47.192 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T18:45:47.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 -- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f171c066f60 msgr2=0x7f171c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f171c066f60 0x7f171c069410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17200094d0 tx=0x7f172000b040 comp rx=0 tx=0).stop 2023-10-10T18:45:47.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 -- 172.21.15.175:0/4290921207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17300b0ad0 msgr2=0x7f17300c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17300b0ad0 0x7f17300c5770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1728006f00 tx=0x7f17280091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 -- 172.21.15.175:0/4290921207 shutdown_connections 2023-10-10T18:45:47.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f171c066f60 0x7f171c069410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17300b0ad0 0x7f17300c5770 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1718004810 0x7f17300ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 --2- 172.21.15.175:0/4290921207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1730155320 0x7f17300ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 -- 172.21.15.175:0/4290921207 >> 172.21.15.175:0/4290921207 conn(0x7f1730068a20 msgr2=0x7f1730158020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 -- 172.21.15.175:0/4290921207 shutdown_connections 2023-10-10T18:45:47.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.151+0000 7f1734b9d700 1 -- 172.21.15.175:0/4290921207 wait complete. 2023-10-10T18:45:47.197 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-10T18:45:47.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.313+0000 7f6dc3e51700 1 Processor -- start 2023-10-10T18:45:47.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.313+0000 7f6dc3e51700 1 -- start start 2023-10-10T18:45:47.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc3e51700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 0x7f6dbc0c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc3e51700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6dbc15a2d0 0x7f6dbc14fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc3e51700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc0c74b0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc3e51700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dbc0683c0 con 0x7f6dbc15a2d0 2023-10-10T18:45:47.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc3e51700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc068f40 con 0x7f6dbc1597f0 2023-10-10T18:45:47.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc23ee700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6dbc15a2d0 0x7f6dbc14fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc23ee700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6dbc15a2d0 0x7f6dbc14fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37582/0 (socket says 172.21.15.175:37582) 2023-10-10T18:45:47.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc23ee700 1 -- 172.21.15.175:0/3865352551 learned_addr learned my addr 172.21.15.175:0/3865352551 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc13ec700 1 --1- 172.21.15.175:0/3865352551 >> v1:172.21.15.175:6789/0 conn(0x7f6dbc0c1f60 0x7f6dbc159bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51500/0 (socket says 172.21.15.175:51500) 2023-10-10T18:45:47.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/3865352551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 0x7f6dbc0c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc23ee700 1 -- 172.21.15.175:0/3865352551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 msgr2=0x7f6dbc0c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/3865352551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 0x7f6dbc0c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc23ee700 1 -- 172.21.15.175:0/3865352551 >> v1:172.21.15.175:6789/0 conn(0x7f6dbc0c1f60 legacy=0x7f6dbc159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc23ee700 1 -- 172.21.15.175:0/3865352551 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc0693d0 con 0x7f6dbc15a2d0 2023-10-10T18:45:47.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.314+0000 7f6dc0beb700 1 -- 172.21.15.175:0/3865352551 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059462103 0 0) 0x7f6dbc0c74b0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/3865352551 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc15a2d0 0x7f6dbc14fc70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009f20 tx=0x7f6db4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7925889439bf3cf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc0beb700 1 -- 172.21.15.175:0/3865352551 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db400e070 con 0x7f6dbc15a2d0 2023-10-10T18:45:47.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc0beb700 1 -- 172.21.15.175:0/3865352551 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db400b9d0 con 0x7f6dbc15a2d0 2023-10-10T18:45:47.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc0beb700 1 -- 172.21.15.175:0/3865352551 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db40092e0 con 0x7f6dbc15a2d0 2023-10-10T18:45:47.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc3e51700 1 -- 172.21.15.175:0/3865352551 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc15a2d0 msgr2=0x7f6dbc14fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/3865352551 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc15a2d0 0x7f6dbc14fc70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009f20 tx=0x7f6db4009840 comp rx=0 tx=0).stop 2023-10-10T18:45:47.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc3e51700 1 -- 172.21.15.175:0/3865352551 shutdown_connections 2023-10-10T18:45:47.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/3865352551 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc15a2d0 0x7f6dbc14fc70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/3865352551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 0x7f6dbc0c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc3e51700 1 -- 172.21.15.175:0/3865352551 >> 172.21.15.175:0/3865352551 conn(0x7f6dbc063230 msgr2=0x7f6dbc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc3e51700 1 -- 172.21.15.175:0/3865352551 shutdown_connections 2023-10-10T18:45:47.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.315+0000 7f6dc3e51700 1 -- 172.21.15.175:0/3865352551 wait complete. 2023-10-10T18:45:47.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.316+0000 7f6dc3e51700 1 Processor -- start 2023-10-10T18:45:47.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.316+0000 7f6dc3e51700 1 -- start start 2023-10-10T18:45:47.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc3e51700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc0c1f60 0x7f6dbc0cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc3e51700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 0x7f6dbc0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc3e51700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dbc153f20 0x7f6dbc0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc3e51700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc154ad0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc3e51700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc14bbb0 con 0x7f6dbc153f20 2023-10-10T18:45:47.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc3e51700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc0ca5c0 con 0x7f6dbc1597f0 2023-10-10T18:45:47.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc1bed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc0c1f60 0x7f6dbc0cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc1bed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc0c1f60 0x7f6dbc0cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37588/0 (socket says 172.21.15.175:37588) 2023-10-10T18:45:47.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc1bed700 1 -- 172.21.15.175:0/1301511359 learned_addr learned my addr 172.21.15.175:0/1301511359 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 0x7f6dbc0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dbc153f20 0x7f6dbc0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc1bed700 1 -- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 msgr2=0x7f6dbc0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 0x7f6dbc0cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc1bed700 1 -- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dbc153f20 msgr2=0x7f6dbc0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.317+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dbc153f20 0x7f6dbc0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.318+0000 7f6dc1bed700 1 -- 172.21.15.175:0/1301511359 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db4009ae0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.318+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dbc153f20 0x7f6dbc0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.318+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc0c1f60 0x7f6dbc0cad30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6db800ece0 tx=0x7f6db80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.318+0000 7f6db1ffb700 1 -- 172.21.15.175:0/1301511359 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800c9b0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.318+0000 7f6db1ffb700 1 -- 172.21.15.175:0/1301511359 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db800cb10 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.318+0000 7f6dc3e51700 1 -- 172.21.15.175:0/1301511359 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dbc0ca8a0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.318+0000 7f6db1ffb700 1 -- 172.21.15.175:0/1301511359 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800ceb0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.318+0000 7f6dc3e51700 1 -- 172.21.15.175:0/1301511359 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc1565a0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.319+0000 7f6db1ffb700 1 -- 172.21.15.175:0/1301511359 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6db8013680 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.320+0000 7f6db1ffb700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6da0066f60 0x7f6da0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.320+0000 7f6db1ffb700 1 -- 172.21.15.175:0/1301511359 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6db80848a0 con 0x7f6dbc0c1f60 2023-10-10T18:45:47.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.320+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6da0066f60 0x7f6da0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.320+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6da0066f60 0x7f6da0069410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc15c780 tx=0x7f6dac007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.321+0000 7f6dcd14ab80 1 Processor -- start 2023-10-10T18:45:47.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.321+0000 7f6dcd14ab80 1 -- start start 2023-10-10T18:45:47.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d200 0x56371b0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 0x56371b0d5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d5d0 0x56371b0d5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371b02ca70 con 0x56371b10d5d0 2023-10-10T18:45:47.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371b02a270 con 0x56371b0d98a0 2023-10-10T18:45:47.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x56371b026060 con 0x56371b10d200 2023-10-10T18:45:47.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc23ee700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d5d0 0x56371b0d5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc23ee700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d5d0 0x56371b0d5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37596/0 (socket says 172.21.15.175:37596) 2023-10-10T18:45:47.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc23ee700 1 -- 172.21.15.175:0/371725071 learned_addr learned my addr 172.21.15.175:0/371725071 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 0x56371b0d5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d200 0x56371b0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc23ee700 1 -- 172.21.15.175:0/371725071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d200 msgr2=0x56371b0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d200 0x56371b0d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc23ee700 1 -- 172.21.15.175:0/371725071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 msgr2=0x56371b0d5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 0x56371b0d5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.322+0000 7f6dc23ee700 1 -- 172.21.15.175:0/371725071 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56371aec0270 con 0x56371b10d5d0 2023-10-10T18:45:47.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d5d0 0x56371b0d5850 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6db400b610 tx=0x7f6db4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd333179e67c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6da6ffd700 1 -- 172.21.15.175:0/371725071 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db400bc10 con 0x56371b10d5d0 2023-10-10T18:45:47.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6da6ffd700 1 -- 172.21.15.175:0/371725071 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db401e430 con 0x56371b10d5d0 2023-10-10T18:45:47.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6da6ffd700 1 -- 172.21.15.175:0/371725071 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db401e7f0 con 0x56371b10d5d0 2023-10-10T18:45:47.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/371725071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d5d0 msgr2=0x56371b0d5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d5d0 0x56371b0d5850 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6db400b610 tx=0x7f6db4018000 comp rx=0 tx=0).stop 2023-10-10T18:45:47.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/371725071 shutdown_connections 2023-10-10T18:45:47.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d5d0 0x56371b0d5850 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d200 0x56371b0d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/371725071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 0x56371b0d5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/371725071 >> 172.21.15.175:0/371725071 conn(0x56371b0d6ba0 msgr2=0x56371b0d7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/371725071 shutdown_connections 2023-10-10T18:45:47.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.323+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/371725071 wait complete. 2023-10-10T18:45:47.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.324+0000 7f6dcd14ab80 1 -- start start 2023-10-10T18:45:47.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.324+0000 7f6dcd14ab80 1 Processor -- start 2023-10-10T18:45:47.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 0x56371b111430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d200 0x56371b127b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d5d0 0x56371b104280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371afd1890 con 0x56371b10d200 2023-10-10T18:45:47.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371afecea0 con 0x56371b0d98a0 2023-10-10T18:45:47.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x56371aea8e30 con 0x56371b10d5d0 2023-10-10T18:45:47.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dcd14ab80 10 client.0 _ll_register_callbacks cb 0x56371b099110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:47.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d200 0x56371b127b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d200 0x56371b127b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37608/0 (socket says 172.21.15.175:37608) 2023-10-10T18:45:47.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 -- 172.21.15.175:0/3140591958 learned_addr learned my addr 172.21.15.175:0/3140591958 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc23ee700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d5d0 0x56371b104280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 0x56371b111430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 -- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d5d0 msgr2=0x56371b104280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d5d0 0x56371b104280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 -- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 msgr2=0x56371b111430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 0x56371b111430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 -- 172.21.15.175:0/3140591958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc0693d0 con 0x56371b10d200 2023-10-10T18:45:47.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d200 0x56371b127b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6dac013520 tx=0x7f6dac014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8be5480cc47cb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.325+0000 7f6d9bfff700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:47.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6d9bfff700 1 -- 172.21.15.175:0/3140591958 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac013bf0 con 0x56371b10d200 2023-10-10T18:45:47.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6d9bfff700 1 -- 172.21.15.175:0/3140591958 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6dac01b070 con 0x56371b10d200 2023-10-10T18:45:47.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/3140591958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56371b0cab70 con 0x56371b10d200 2023-10-10T18:45:47.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6d9bfff700 1 -- 172.21.15.175:0/3140591958 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dac021410 con 0x56371b10d200 2023-10-10T18:45:47.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 10 client.4758 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:47.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/3140591958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x56371ae62fe0 con 0x56371b10d200 2023-10-10T18:45:47.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6d9bfff700 1 -- 172.21.15.175:0/3140591958 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56371b0cab70 con 0x56371b10d200 2023-10-10T18:45:47.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6d9bfff700 1 -- 172.21.15.175:0/3140591958 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7f6dac021570 con 0x56371b10d200 2023-10-10T18:45:47.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 10 client.4758 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:47.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 10 client.4758 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-10T18:45:47.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 20 client.4758 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:47.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] conn(0x56371b100f20 0x56371b1012f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 4 client.4758 mds_command: new command op to 4263 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T18:45:47.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/3140591958 --> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56371b0a7aa0 con 0x56371b100f20 2023-10-10T18:45:47.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.326+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] conn(0x56371b100f20 0x56371b1012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6d9bfff700 10 client.4758 ms_handle_connect on v2:172.21.15.191:6834/3870390806 2023-10-10T18:45:47.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] conn(0x56371b100f20 0x56371b1012f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94d3c2a761d45f1f server_cookie=a559ae84b03cf699 in_seq=0 out_seq=0 2023-10-10T18:45:47.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6d9bfff700 1 -- 172.21.15.175:0/3140591958 <== mds.0 v2:172.21.15.191:6834/3870390806 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x56371b0a7aa0 con 0x56371b100f20 2023-10-10T18:45:47.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6d9bfff700 10 client.4758 handle_command_reply: tid=0 2023-10-10T18:45:47.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 client.4758 shutdown 2023-10-10T18:45:47.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/3140591958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d200 msgr2=0x56371b127b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d200 0x56371b127b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6dac013520 tx=0x7f6dac014000 comp rx=0 tx=0).stop 2023-10-10T18:45:47.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/3140591958 shutdown_connections 2023-10-10T18:45:47.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] conn(0x56371b100f20 0x56371b1012f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b10d5d0 0x56371b104280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6d9bfff700 0 client.4758 ms_handle_reset on v2:172.21.15.191:6834/3870390806 2023-10-10T18:45:47.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b0d98a0 0x56371b111430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/3140591958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b10d200 0x56371b127b70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/3140591958 >> 172.21.15.175:0/3140591958 conn(0x56371b0d6ba0 msgr2=0x56371b0d53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.327+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/3140591958 shutdown_connections 2023-10-10T18:45:47.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.328+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/3140591958 wait complete. 2023-10-10T18:45:47.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.328+0000 7f6dcd14ab80 20 client.4758 trim_cache size 0 max 16384 2023-10-10T18:45:47.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.344+0000 7f6dcd14ab80 1 Processor -- start 2023-10-10T18:45:47.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.344+0000 7f6dcd14ab80 1 -- start start 2023-10-10T18:45:47.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b0d98a0 0x56371b105b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b100f20 0x56371b1060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b109fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371b0d05a0 con 0x56371b0d98a0 2023-10-10T18:45:47.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371b0d1560 con 0x56371b10d200 2023-10-10T18:45:47.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x56371b118430 con 0x56371b100f20 2023-10-10T18:45:47.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b0d98a0 0x56371b105b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b0d98a0 0x56371b105b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37622/0 (socket says 172.21.15.175:37622) 2023-10-10T18:45:47.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 -- 172.21.15.175:0/2187175605 learned_addr learned my addr 172.21.15.175:0/2187175605 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 -- 172.21.15.175:0/2187175605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b100f20 msgr2=0x56371b1060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b100f20 0x56371b1060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 -- 172.21.15.175:0/2187175605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 msgr2=0x56371b109fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b109fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b109fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 -- 172.21.15.175:0/2187175605 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56371aff4780 con 0x56371b0d98a0 2023-10-10T18:45:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b0d98a0 0x56371b105b70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6db8051cd0 tx=0x7f6db8015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcdfe7446c501a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.345+0000 7f6da77fe700 1 -- 172.21.15.175:0/2187175605 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8082ed0 con 0x56371b0d98a0 2023-10-10T18:45:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6da77fe700 1 -- 172.21.15.175:0/2187175605 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db8053910 con 0x56371b0d98a0 2023-10-10T18:45:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6da77fe700 1 -- 172.21.15.175:0/2187175605 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8053b70 con 0x56371b0d98a0 2023-10-10T18:45:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2187175605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b0d98a0 msgr2=0x56371b105b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b0d98a0 0x56371b105b70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6db8051cd0 tx=0x7f6db8015040 comp rx=0 tx=0).stop 2023-10-10T18:45:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2187175605 shutdown_connections 2023-10-10T18:45:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b0d98a0 0x56371b105b70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2187175605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b100f20 0x56371b1060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2187175605 >> 172.21.15.175:0/2187175605 conn(0x56371b0d6ba0 msgr2=0x56371b0d7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2187175605 shutdown_connections 2023-10-10T18:45:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.346+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2187175605 wait complete. 2023-10-10T18:45:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.347+0000 7f6dcd14ab80 1 -- start start 2023-10-10T18:45:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.347+0000 7f6dcd14ab80 1 Processor -- start 2023-10-10T18:45:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b0d98a0 0x56371b0d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b100f20 0x56371b177cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b0ffde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371b104640 con 0x56371b100f20 2023-10-10T18:45:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371b0d1f80 con 0x56371b10d200 2023-10-10T18:45:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x56371b108990 con 0x56371b0d98a0 2023-10-10T18:45:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 10 client.0 _ll_register_callbacks cb 0x56371b0d7780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T18:45:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b100f20 0x56371b177cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b100f20 0x56371b177cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37628/0 (socket says 172.21.15.175:37628) 2023-10-10T18:45:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 -- 172.21.15.175:0/2532977996 learned_addr learned my addr 172.21.15.175:0/2532977996 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b0d98a0 0x56371b0d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b0ffde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 -- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b0d98a0 msgr2=0x56371b0d16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b0d98a0 0x56371b0d16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 -- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 msgr2=0x56371b0ffde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b0ffde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 -- 172.21.15.175:0/2532977996 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc1565a0 con 0x56371b100f20 2023-10-10T18:45:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc23ee700 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b0ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dc13ec700 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b100f20 0x56371b177cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6dac001ed0 tx=0x7f6dac015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa45f16822c41490 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6d9bfff700 10 client.0 ms_handle_connect on v2:172.21.15.175:3300/0 2023-10-10T18:45:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6d9bfff700 1 -- 172.21.15.175:0/2532977996 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac013830 con 0x56371b100f20 2023-10-10T18:45:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6d9bfff700 1 -- 172.21.15.175:0/2532977996 <== mon.0 v2:172.21.15.175:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6dac021ad0 con 0x56371b100f20 2023-10-10T18:45:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2532977996 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6db401e430 con 0x56371b100f20 2023-10-10T18:45:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6d9bfff700 1 -- 172.21.15.175:0/2532977996 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dac03c430 con 0x56371b100f20 2023-10-10T18:45:47.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 10 client.4764 fetch_fsmap learned FSMap version 17 2023-10-10T18:45:47.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2532977996 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x56371afdb470 con 0x56371b100f20 2023-10-10T18:45:47.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.348+0000 7f6d9bfff700 1 -- 172.21.15.175:0/2532977996 <== mon.0 v2:172.21.15.175:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db401e430 con 0x56371b100f20 2023-10-10T18:45:47.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6d9bfff700 1 -- 172.21.15.175:0/2532977996 <== mon.0 v2:172.21.15.175:3300/0 5 ==== fsmap(e 17) v1 ==== 4238+0+0 (secure 0 0 0) 0x7f6dac03c590 con 0x56371b100f20 2023-10-10T18:45:47.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 10 client.4764 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T18:45:47.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 10 client.4764 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-10T18:45:47.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 20 client.4764 populate_metadata read hostname 'smithi175' 2023-10-10T18:45:47.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] conn(0x56371b10d5d0 0x56371b10af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 4 client.4764 mds_command: new command op to 4263 tid=0[{"prefix": "status"}] 2023-10-10T18:45:47.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2532977996 --> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56371afc8e00 con 0x56371b10d5d0 2023-10-10T18:45:47.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] conn(0x56371b10d5d0 0x56371b10af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6d9bfff700 10 client.4764 ms_handle_connect on v2:172.21.15.191:6834/3870390806 2023-10-10T18:45:47.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dc1bed700 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] conn(0x56371b10d5d0 0x56371b10af50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a115a70aa4cb687 server_cookie=b467a3c72ec73469 in_seq=0 out_seq=0 2023-10-10T18:45:47.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6d9bfff700 1 -- 172.21.15.175:0/2532977996 <== mds.0 v2:172.21.15.191:6834/3870390806 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56371afc8e00 con 0x56371b10d5d0 2023-10-10T18:45:47.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6d9bfff700 10 client.4764 handle_command_reply: tid=0 2023-10-10T18:45:47.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 client.4764 shutdown 2023-10-10T18:45:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2532977996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b100f20 msgr2=0x56371b177cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b100f20 0x56371b177cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6dac001ed0 tx=0x7f6dac015000 comp rx=0 tx=0).stop 2023-10-10T18:45:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2532977996 shutdown_connections 2023-10-10T18:45:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:6834/3870390806,v1:172.21.15.191:6837/3870390806] conn(0x56371b10d5d0 0x56371b10af50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56371b10d200 0x56371b0ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6d9bfff700 0 client.4764 ms_handle_reset on v2:172.21.15.191:6834/3870390806 2023-10-10T18:45:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56371b0d98a0 0x56371b0d16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 --2- 172.21.15.175:0/2532977996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56371b100f20 0x56371b177cf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.349+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2532977996 >> 172.21.15.175:0/2532977996 conn(0x56371b0d6ba0 msgr2=0x56371b105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2532977996 shutdown_connections 2023-10-10T18:45:47.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dcd14ab80 1 -- 172.21.15.175:0/2532977996 wait complete. 2023-10-10T18:45:47.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dcd14ab80 20 client.4764 trim_cache size 0 max 16384 2023-10-10T18:45:47.378 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T18:45:47.379 INFO:teuthology.orchestra.run.smithi175.stdout: "cluster_fsid": "7f0fe9e3-2088-4ba2-8a54-92a2be521afd", 2023-10-10T18:45:47.379 INFO:teuthology.orchestra.run.smithi175.stdout: "whoami": 0, 2023-10-10T18:45:47.379 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 4263, 2023-10-10T18:45:47.379 INFO:teuthology.orchestra.run.smithi175.stdout: "want_state": "up:active", 2023-10-10T18:45:47.379 INFO:teuthology.orchestra.run.smithi175.stdout: "state": "up:active", 2023-10-10T18:45:47.380 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs", 2023-10-10T18:45:47.380 INFO:teuthology.orchestra.run.smithi175.stdout: "rank_uptime": 35.940473910999998, 2023-10-10T18:45:47.380 INFO:teuthology.orchestra.run.smithi175.stdout: "mdsmap_epoch": 17, 2023-10-10T18:45:47.380 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch": 21, 2023-10-10T18:45:47.381 INFO:teuthology.orchestra.run.smithi175.stdout: "osdmap_epoch_barrier": 21, 2023-10-10T18:45:47.381 INFO:teuthology.orchestra.run.smithi175.stdout: "uptime": 42.835775235 2023-10-10T18:45:47.381 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T18:45:47.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 -- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6da0066f60 msgr2=0x7f6da0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6da0066f60 0x7f6da0069410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc15c780 tx=0x7f6dac007ef0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 -- 172.21.15.175:0/1301511359 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc0c1f60 msgr2=0x7f6dbc0cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc0c1f60 0x7f6dbc0cad30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6db800ece0 tx=0x7f6db80091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 -- 172.21.15.175:0/1301511359 shutdown_connections 2023-10-10T18:45:47.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6da0066f60 0x7f6da0069410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dbc153f20 0x7f6dbc0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6dbc0c1f60 0x7f6dbc0cad30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 --2- 172.21.15.175:0/1301511359 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6dbc1597f0 0x7f6dbc0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 -- 172.21.15.175:0/1301511359 >> 172.21.15.175:0/1301511359 conn(0x7f6dbc063230 msgr2=0x7f6dbc0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 -- 172.21.15.175:0/1301511359 shutdown_connections 2023-10-10T18:45:47.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.350+0000 7f6dc3e51700 1 -- 172.21.15.175:0/1301511359 wait complete. 2023-10-10T18:45:47.386 INFO:teuthology.run_tasks:Running task kclient... 2023-10-10T18:45:47.408 INFO:tasks.kclient:Mounting kernel clients... 2023-10-10T18:45:47.408 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-10T18:45:47.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:47.409 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2023-10-10T18:45:47.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:47.425 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link delete ceph-brx 2023-10-10T18:45:47.491 INFO:teuthology.orchestra.run.smithi175.stderr:Cannot find device "ceph-brx" 2023-10-10T18:45:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:45:47.493 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-10T18:45:47.493 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-10T18:45:47.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.649+0000 7fae05303700 1 Processor -- start 2023-10-10T18:45:47.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.649+0000 7fae05303700 1 -- start start 2023-10-10T18:45:47.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fae05303700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae0008a100 0x7fae00159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fae05303700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae00067fa0 con 0x7fae0014fd90 2023-10-10T18:45:47.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fae05303700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae00156b70 con 0x7fae000cd040 2023-10-10T18:45:47.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fae05303700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae000c5600 con 0x7fae0008a100 2023-10-10T18:45:47.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fae0014fd90 0x7fae000c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51512/0 (socket says 172.21.15.175:51512) 2023-10-10T18:45:47.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfffff700 1 -- 172.21.15.175:0/3028152835 learned_addr learned my addr 172.21.15.175:0/3028152835 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:47.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadff7fe700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae0008a100 0x7fae00159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618221681 0 0) 0x7fae00067fa0 con 0x7fae0014fd90 2023-10-10T18:45:47.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade8003570 con 0x7fae0014fd90 2023-10-10T18:45:47.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3780573168 0 0) 0x7fade8003570 con 0x7fae0014fd90 2023-10-10T18:45:47.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae0008a100 msgr2=0x7fae00159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae0008a100 0x7fae00159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 >> v1:172.21.15.191:6789/0 conn(0x7fae000cd040 legacy=0x7fae000684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00150160 con 0x7fae0014fd90 2023-10-10T18:45:47.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fadf0001470 con 0x7fae0014fd90 2023-10-10T18:45:47.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 >> v1:172.21.15.175:6789/0 conn(0x7fae0014fd90 legacy=0x7fae000c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fade8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.650+0000 7fadfeffd700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fade8004810 0x7fade8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfeffd700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae0008a100 0x7fade80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae000c5600 con 0x7fade8003a90 2023-10-10T18:45:47.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade800bb10 con 0x7fade8004810 2023-10-10T18:45:47.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade800bcd0 con 0x7fae0008a100 2023-10-10T18:45:47.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfffff700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fade8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadff7fe700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fade8004810 0x7fade8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fae04b02700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae0008a100 0x7fade80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfffff700 1 -- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae0008a100 msgr2=0x7fade80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfffff700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae0008a100 0x7fade80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfffff700 1 -- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fade8004810 msgr2=0x7fade8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:47.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfffff700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fade8004810 0x7fade8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfffff700 1 -- 172.21.15.175:0/3028152835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade8003570 con 0x7fade8003a90 2023-10-10T18:45:47.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfffff700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fade8003e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fadf000c3c0 tx=0x7fae000c5dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=badee41893294574 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf000cc90 con 0x7fade8003a90 2023-10-10T18:45:47.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf000ce10 con 0x7fade8003a90 2023-10-10T18:45:47.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.651+0000 7fadfeffd700 1 -- 172.21.15.175:0/3028152835 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0005b70 con 0x7fade8003a90 2023-10-10T18:45:47.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.652+0000 7fae05303700 1 -- 172.21.15.175:0/3028152835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 msgr2=0x7fade8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.652+0000 7fae05303700 1 --2- 172.21.15.175:0/3028152835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fade8003e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fadf000c3c0 tx=0x7fae000c5dd0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.652+0000 7fae04b02700 1 -- 172.21.15.175:0/3028152835 reap_dead start 2023-10-10T18:45:47.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.652+0000 7fae05303700 1 -- 172.21.15.175:0/3028152835 shutdown_connections 2023-10-10T18:45:47.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.652+0000 7fae05303700 1 -- 172.21.15.175:0/3028152835 >> 172.21.15.175:0/3028152835 conn(0x7fae00063d00 msgr2=0x7fae000cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.652+0000 7fae05303700 1 -- 172.21.15.175:0/3028152835 shutdown_connections 2023-10-10T18:45:47.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.652+0000 7fae05303700 1 -- 172.21.15.175:0/3028152835 wait complete. 2023-10-10T18:45:47.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.653+0000 7fae05303700 1 Processor -- start 2023-10-10T18:45:47.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.653+0000 7fae05303700 1 -- start start 2023-10-10T18:45:47.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.653+0000 7fae05303700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fae0014bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.653+0000 7fae05303700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae00149580 0x7fae0014dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fae05303700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae0014e170 0x7fae00148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fae05303700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae00166150 con 0x7fade8003a90 2023-10-10T18:45:47.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fae05303700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae00165860 con 0x7fae0014e170 2023-10-10T18:45:47.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fae05303700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae001665e0 con 0x7fae00149580 2023-10-10T18:45:47.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fae0014bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fae0014bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37634/0 (socket says 172.21.15.175:37634) 2023-10-10T18:45:47.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 -- 172.21.15.175:0/3512011137 learned_addr learned my addr 172.21.15.175:0/3512011137 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:47.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadff7fe700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae00149580 0x7fae0014dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fae04b02700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae0014e170 0x7fae00148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 -- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae00149580 msgr2=0x7fae0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae00149580 0x7fae0014dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 -- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae0014e170 msgr2=0x7fae00148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae0014e170 0x7fae00148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 -- 172.21.15.175:0/3512011137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0014f2c0 con 0x7fade8003a90 2023-10-10T18:45:47.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadff7fe700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae00149580 0x7fae0014dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fae04b02700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae0014e170 0x7fae00148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:47.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fadfffff700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fae0014bb00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fade8004770 tx=0x7fadf0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fade7fff700 1 -- 172.21.15.175:0/3512011137 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf00070d0 con 0x7fade8003a90 2023-10-10T18:45:47.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fade7fff700 1 -- 172.21.15.175:0/3512011137 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf0007250 con 0x7fade8003a90 2023-10-10T18:45:47.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fade7fff700 1 -- 172.21.15.175:0/3512011137 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0007610 con 0x7fade8003a90 2023-10-10T18:45:47.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae00165b00 con 0x7fade8003a90 2023-10-10T18:45:47.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.654+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0014f5a0 con 0x7fade8003a90 2023-10-10T18:45:47.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.655+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae00165fa0 con 0x7fade8003a90 2023-10-10T18:45:47.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.656+0000 7fade7fff700 1 -- 172.21.15.175:0/3512011137 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fadf0019030 con 0x7fade8003a90 2023-10-10T18:45:47.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.656+0000 7fade7fff700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fadec066f30 0x7fadec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:47.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.656+0000 7fade7fff700 1 -- 172.21.15.175:0/3512011137 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fadf00855c0 con 0x7fade8003a90 2023-10-10T18:45:47.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.656+0000 7fadff7fe700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fadec066f30 0x7fadec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:47.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.657+0000 7fadff7fe700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fadec066f30 0x7fadec0693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadf4006dc0 tx=0x7fadf4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:47.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.660+0000 7fade7fff700 1 -- 172.21.15.175:0/3512011137 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fadf0052830 con 0x7fade8003a90 2023-10-10T18:45:47.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.887+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fae00166430 con 0x7fade8003a90 2023-10-10T18:45:47.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.887+0000 7fade7fff700 1 -- 172.21.15.175:0/3512011137 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fadf0055e50 con 0x7fade8003a90 2023-10-10T18:45:47.888 INFO:teuthology.orchestra.run.smithi175.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-10T18:45:47.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fadec066f30 msgr2=0x7fadec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fadec066f30 0x7fadec0693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadf4006dc0 tx=0x7fadf4008040 comp rx=0 tx=0).stop 2023-10-10T18:45:47.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 msgr2=0x7fae0014bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:47.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fae0014bb00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fade8004770 tx=0x7fadf0012000 comp rx=0 tx=0).stop 2023-10-10T18:45:47.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 shutdown_connections 2023-10-10T18:45:47.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fadec066f30 0x7fadec0693e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae0014e170 0x7fae00148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fade8003a90 0x7fae0014bb00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 --2- 172.21.15.175:0/3512011137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae00149580 0x7fae0014dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:47.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 >> 172.21.15.175:0/3512011137 conn(0x7fae00063d00 msgr2=0x7fae00159b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:47.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 shutdown_connections 2023-10-10T18:45:47.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:47.889+0000 7fae05303700 1 -- 172.21.15.175:0/3512011137 wait complete. 2023-10-10T18:45:47.901 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-10T18:45:47.901 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-10T18:45:47.902 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi175.front.sepia.ceph.com 2023-10-10T18:45:47.902 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-10T18:45:47.902 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-10T18:45:47.903 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-10T18:45:47.903 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-10T18:45:47.903 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-10T18:45:47.904 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-10T18:45:47.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:47.904 DEBUG:teuthology.orchestra.run.smithi175:> ip addr 2023-10-10T18:45:47.919 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-10T18:45:47.920 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-10T18:45:47.920 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2023-10-10T18:45:47.920 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2023-10-10T18:45:47.920 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2023-10-10T18:45:47.921 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2023-10-10T18:45:47.921 INFO:teuthology.orchestra.run.smithi175.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T18:45:47.921 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ee brd ff:ff:ff:ff:ff:ff 2023-10-10T18:45:47.922 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f0 2023-10-10T18:45:47.922 INFO:teuthology.orchestra.run.smithi175.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T18:45:47.923 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ef brd ff:ff:ff:ff:ff:ff 2023-10-10T18:45:47.923 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f1 2023-10-10T18:45:47.923 INFO:teuthology.orchestra.run.smithi175.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T18:45:47.923 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f0 brd ff:ff:ff:ff:ff:ff 2023-10-10T18:45:47.923 INFO:teuthology.orchestra.run.smithi175.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-10T18:45:47.924 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f1 brd ff:ff:ff:ff:ff:ff 2023-10-10T18:45:47.924 INFO:teuthology.orchestra.run.smithi175.stdout: inet 172.21.15.175/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-10T18:45:47.924 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft 42491sec preferred_lft 42491sec 2023-10-10T18:45:47.924 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::ec4:7aff:fe88:71f1/64 scope link 2023-10-10T18:45:47.924 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2023-10-10T18:45:47.925 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-10T18:45:47.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:47.926 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2023-10-10T18:45:47.926 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2023-10-10T18:45:47.926 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link add name ceph-brx type bridge 2023-10-10T18:45:47.926 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip addr flush dev ceph-brx 2023-10-10T18:45:47.927 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set ceph-brx up 2023-10-10T18:45:47.927 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-10T18:45:47.927 DEBUG:teuthology.orchestra.run.smithi175:> ') 2023-10-10T18:45:48.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:48.084 DEBUG:teuthology.orchestra.run.smithi175:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-10T18:45:48.124 INFO:teuthology.orchestra.run.smithi175.stdout:1 2023-10-10T18:45:48.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:48.126 DEBUG:teuthology.orchestra.run.smithi175:> route 2023-10-10T18:45:48.194 INFO:teuthology.orchestra.run.smithi175.stdout:Kernel IP routing table 2023-10-10T18:45:48.194 INFO:teuthology.orchestra.run.smithi175.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-10T18:45:48.194 INFO:teuthology.orchestra.run.smithi175.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-10T18:45:48.195 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-10T18:45:48.195 INFO:teuthology.orchestra.run.smithi175.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-10T18:45:48.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:48.196 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2023-10-10T18:45:48.196 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2023-10-10T18:45:48.196 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-10T18:45:48.197 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-10T18:45:48.197 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-10T18:45:48.197 DEBUG:teuthology.orchestra.run.smithi175:> ') 2023-10-10T18:45:48.261 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:45:48.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:45:48.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:48.485 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2023-10-10T18:45:48.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:48.501 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list-id 2023-10-10T18:45:48.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:48.556 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2023-10-10T18:45:48.556 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2023-10-10T18:45:48.557 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-10T18:45:48.557 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-10T18:45:48.557 DEBUG:teuthology.orchestra.run.smithi175:> ') 2023-10-10T18:45:48.645 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-10T18:45:48.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:48.646 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2023-10-10T18:45:48.646 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2023-10-10T18:45:48.646 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-10T18:45:48.647 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-10T18:45:48.647 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-10T18:45:48.647 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-10T18:45:48.648 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-10T18:45:48.648 DEBUG:teuthology.orchestra.run.smithi175:> ') 2023-10-10T18:45:48.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:48.831 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2023-10-10T18:45:48.831 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2023-10-10T18:45:48.831 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set brx.0 up 2023-10-10T18:45:48.832 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set dev brx.0 master ceph-brx 2023-10-10T18:45:48.832 DEBUG:teuthology.orchestra.run.smithi175:> ') 2023-10-10T18:45:48.924 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-10T18:45:48.924 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-10T18:45:48.939 INFO:teuthology.orchestra.run.smithi175.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-10T18:45:48.940 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-10T18:45:48.940 DEBUG:teuthology.orchestra.run.smithi175:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-10T18:45:48.995 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-10T18:45:48.995 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-10T18:45:48.995 DEBUG:teuthology.orchestra.run.smithi175:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-10T18:45:49.093 INFO:teuthology.orchestra.run.smithi175.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-10T18:45:49.094 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-10T18:45:49.094 INFO:teuthology.orchestra.run.smithi175.stdout:invalid new device string format 2023-10-10T18:45:49.165 INFO:teuthology.orchestra.run.smithi175.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-10T18:45:49.165 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-10T18:45:49.165 INFO:teuthology.orchestra.run.smithi175.stdout:invalid new device string format 2023-10-10T18:45:49.166 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: resolved to: "172.21.15.175:3300,172.21.15.191:3300,172.21.15.191:3301" 2023-10-10T18:45:49.166 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-10T18:45:49.187 INFO:teuthology.orchestra.run.smithi175.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-10T18:45:49.188 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-10T18:45:49.188 INFO:teuthology.orchestra.run.smithi175.stdout:invalid new device string format 2023-10-10T18:45:49.188 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: resolved to: "172.21.15.175:3300,172.21.15.191:3300,172.21.15.191:3301" 2023-10-10T18:45:49.188 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-10T18:45:49.189 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: resolved to: "172.21.15.175:6789,172.21.15.191:6789,172.21.15.191:6790" 2023-10-10T18:45:49.189 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.175:6789,172.21.15.191:6789,172.21.15.191:6790:/ 2023-10-10T18:45:49.189 INFO:teuthology.orchestra.run.smithi175.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=7f0fe9e3-2088-4ba2-8a54-92a2be521afd" will pass to kernel 2023-10-10T18:45:49.190 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-10T18:45:49.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T18:45:49.191 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-10T18:45:49.993 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-10T18:45:50.009 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:50.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.132+0000 7f890527e700 1 Processor -- start 2023-10-10T18:45:50.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.132+0000 7f890527e700 1 -- start start 2023-10-10T18:45:50.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f890527e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89000c5850 0x7f89000c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f890527e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c4f20 0x7f89000c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f890527e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89000c6fb0 con 0x7f89000c12d0 2023-10-10T18:45:50.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f890527e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8900063ea0 con 0x7f89000c5850 2023-10-10T18:45:50.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f890527e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8900155f30 con 0x7f89000c4f20 2023-10-10T18:45:50.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88feffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89000c5850 0x7f89000c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88ff7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c4f20 0x7f89000c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88feffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89000c5850 0x7f89000c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:34996/0 (socket says 172.21.15.175:34996) 2023-10-10T18:45:50.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88feffd700 1 -- 172.21.15.175:0/1173818283 learned_addr learned my addr 172.21.15.175:0/1173818283 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:50.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88feffd700 1 -- 172.21.15.175:0/1173818283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c4f20 msgr2=0x7f89000c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:50.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88feffd700 1 --2- 172.21.15.175:0/1173818283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c4f20 0x7f89000c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88feffd700 1 -- 172.21.15.175:0/1173818283 >> v1:172.21.15.191:6789/0 conn(0x7f89000c12d0 legacy=0x7f8900156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88feffd700 1 -- 172.21.15.175:0/1173818283 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89000c6160 con 0x7f89000c5850 2023-10-10T18:45:50.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.133+0000 7f88ff7fe700 1 --2- 172.21.15.175:0/1173818283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c4f20 0x7f89000c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:45:50.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f88feffd700 1 --2- 172.21.15.175:0/1173818283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c5850 0x7f89000c5c20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009f20 tx=0x7f88f8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=defa55a90e35f4b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f88fdffb700 1 -- 172.21.15.175:0/1173818283 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f800e070 con 0x7f89000c5850 2023-10-10T18:45:50.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f88fdffb700 1 -- 172.21.15.175:0/1173818283 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f800b9d0 con 0x7f89000c5850 2023-10-10T18:45:50.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f88fdffb700 1 -- 172.21.15.175:0/1173818283 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80092e0 con 0x7f89000c5850 2023-10-10T18:45:50.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f890527e700 1 -- 172.21.15.175:0/1173818283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c5850 msgr2=0x7f89000c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f890527e700 1 --2- 172.21.15.175:0/1173818283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c5850 0x7f89000c5c20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009f20 tx=0x7f88f8009840 comp rx=0 tx=0).stop 2023-10-10T18:45:50.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f890527e700 1 -- 172.21.15.175:0/1173818283 shutdown_connections 2023-10-10T18:45:50.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f890527e700 1 --2- 172.21.15.175:0/1173818283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c4f20 0x7f89000c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f890527e700 1 --2- 172.21.15.175:0/1173818283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c5850 0x7f89000c5c20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f890527e700 1 -- 172.21.15.175:0/1173818283 >> 172.21.15.175:0/1173818283 conn(0x7f890006dc70 msgr2=0x7f8900063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:50.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f890527e700 1 -- 172.21.15.175:0/1173818283 shutdown_connections 2023-10-10T18:45:50.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.134+0000 7f890527e700 1 -- 172.21.15.175:0/1173818283 wait complete. 2023-10-10T18:45:50.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f890527e700 1 Processor -- start 2023-10-10T18:45:50.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f890527e700 1 -- start start 2023-10-10T18:45:50.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f890527e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89000c12d0 0x7f890014b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f890527e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c4f20 0x7f890014b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f890527e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c5850 0x7f89000c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f890527e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000aff50 con 0x7f89000c12d0 2023-10-10T18:45:50.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f890527e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890014abb0 con 0x7f89000c4f20 2023-10-10T18:45:50.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f890527e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8900169fe0 con 0x7f89000c5850 2023-10-10T18:45:50.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f88feffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89000c12d0 0x7f890014b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f88feffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89000c12d0 0x7f890014b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37642/0 (socket says 172.21.15.175:37642) 2023-10-10T18:45:50.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.136+0000 7f88feffd700 1 -- 172.21.15.175:0/314840614 learned_addr learned my addr 172.21.15.175:0/314840614 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:50.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88fe7fc700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c4f20 0x7f890014b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88ff7fe700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c5850 0x7f89000c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88feffd700 1 -- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c5850 msgr2=0x7f89000c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88feffd700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c5850 0x7f89000c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88feffd700 1 -- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c4f20 msgr2=0x7f890014b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88feffd700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c4f20 0x7f890014b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88feffd700 1 -- 172.21.15.175:0/314840614 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f0009420 con 0x7f89000c12d0 2023-10-10T18:45:50.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88feffd700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89000c12d0 0x7f890014b2c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f88f800b610 tx=0x7f88f8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88e6ffd700 1 -- 172.21.15.175:0/314840614 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80089a0 con 0x7f89000c12d0 2023-10-10T18:45:50.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88e6ffd700 1 -- 172.21.15.175:0/314840614 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f8008b00 con 0x7f89000c12d0 2023-10-10T18:45:50.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f8009ae0 con 0x7f89000c12d0 2023-10-10T18:45:50.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f88e6ffd700 1 -- 172.21.15.175:0/314840614 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8026460 con 0x7f89000c12d0 2023-10-10T18:45:50.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.137+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890014c1a0 con 0x7f89000c12d0 2023-10-10T18:45:50.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.138+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8900067570 con 0x7f89000c12d0 2023-10-10T18:45:50.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.138+0000 7f88e6ffd700 1 -- 172.21.15.175:0/314840614 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f88f8003040 con 0x7f89000c12d0 2023-10-10T18:45:50.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.139+0000 7f88e6ffd700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f88ec066f30 0x7f88ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.139+0000 7f88e6ffd700 1 -- 172.21.15.175:0/314840614 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f88f80862b0 con 0x7f89000c12d0 2023-10-10T18:45:50.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.139+0000 7f88fe7fc700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f88ec066f30 0x7f88ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.140+0000 7f88fe7fc700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f88ec066f30 0x7f88ec0693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89000c0290 tx=0x7f88f4007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.143+0000 7f88e6ffd700 1 -- 172.21.15.175:0/314840614 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f88f8053540 con 0x7f89000c12d0 2023-10-10T18:45:50.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.327+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89000bffe0 con 0x7f89000c12d0 2023-10-10T18:45:50.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.328+0000 7f88e6ffd700 1 -- 172.21.15.175:0/314840614 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7f88f8056b60 con 0x7f89000c12d0 2023-10-10T18:45:50.328 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:50.329 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:50.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f88ec066f30 msgr2=0x7f88ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f88ec066f30 0x7f88ec0693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89000c0290 tx=0x7f88f4007ec0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89000c12d0 msgr2=0x7f890014b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89000c12d0 0x7f890014b2c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f88f800b610 tx=0x7f88f8018000 comp rx=0 tx=0).stop 2023-10-10T18:45:50.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 shutdown_connections 2023-10-10T18:45:50.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f88ec066f30 0x7f88ec0693e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89000c5850 0x7f89000c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89000c12d0 0x7f890014b2c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 --2- 172.21.15.175:0/314840614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89000c4f20 0x7f890014b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 >> 172.21.15.175:0/314840614 conn(0x7f890006dc70 msgr2=0x7f89000bf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:50.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 shutdown_connections 2023-10-10T18:45:50.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.331+0000 7f890527e700 1 -- 172.21.15.175:0/314840614 wait complete. 2023-10-10T18:45:50.343 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T18:45:05.161495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T18:45:08.855952+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T18:45:09.742056+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T18:45:50.344 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T18:45:50.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.463+0000 7f699ee0b700 1 Processor -- start 2023-10-10T18:45:50.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.463+0000 7f699ee0b700 1 -- start start 2023-10-10T18:45:50.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699ee0b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6998150130 0x7f699815a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699ee0b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f69980c5010 0x7f69980c4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699ee0b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69980c3e90 con 0x7f699806d5b0 2023-10-10T18:45:50.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699ee0b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6998063950 con 0x7f6998150130 2023-10-10T18:45:50.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699ee0b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69980c2ae0 con 0x7f69980c5010 2023-10-10T18:45:50.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699cba7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6998150130 0x7f699815a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699cba7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6998150130 0x7f699815a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37656/0 (socket says 172.21.15.175:37656) 2023-10-10T18:45:50.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699cba7700 1 -- 172.21.15.175:0/3642678763 learned_addr learned my addr 172.21.15.175:0/3642678763 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:50.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.464+0000 7f699d3a8700 1 --2- 172.21.15.175:0/3642678763 >> v2:172.21.15.191:3300/0 conn(0x7f69980c5010 0x7f69980c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f699cba7700 1 -- 172.21.15.175:0/3642678763 >> v2:172.21.15.191:3300/0 conn(0x7f69980c5010 msgr2=0x7f69980c4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:50.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f699cba7700 1 --2- 172.21.15.175:0/3642678763 >> v2:172.21.15.191:3300/0 conn(0x7f69980c5010 0x7f69980c4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f699cba7700 1 -- 172.21.15.175:0/3642678763 >> v1:172.21.15.191:6789/0 conn(0x7f699806d5b0 legacy=0x7f699806ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f699cba7700 1 -- 172.21.15.175:0/3642678763 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69980c6540 con 0x7f6998150130 2023-10-10T18:45:50.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f699cba7700 1 --2- 172.21.15.175:0/3642678763 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f699815a010 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69980c9d50 tx=0x7f6994009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dcec2a8ab8c978 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f698f7fe700 1 -- 172.21.15.175:0/3642678763 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699400e070 con 0x7f6998150130 2023-10-10T18:45:50.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f698f7fe700 1 -- 172.21.15.175:0/3642678763 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699400bab0 con 0x7f6998150130 2023-10-10T18:45:50.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f698f7fe700 1 -- 172.21.15.175:0/3642678763 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69940092e0 con 0x7f6998150130 2023-10-10T18:45:50.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f699ee0b700 1 -- 172.21.15.175:0/3642678763 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 msgr2=0x7f699815a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.465+0000 7f699ee0b700 1 --2- 172.21.15.175:0/3642678763 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f699815a010 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69980c9d50 tx=0x7f6994009840 comp rx=0 tx=0).stop 2023-10-10T18:45:50.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.466+0000 7f699ee0b700 1 -- 172.21.15.175:0/3642678763 shutdown_connections 2023-10-10T18:45:50.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.466+0000 7f699ee0b700 1 --2- 172.21.15.175:0/3642678763 >> v2:172.21.15.191:3300/0 conn(0x7f69980c5010 0x7f69980c4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.466+0000 7f699ee0b700 1 --2- 172.21.15.175:0/3642678763 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f699815a010 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.466+0000 7f699ee0b700 1 -- 172.21.15.175:0/3642678763 >> 172.21.15.175:0/3642678763 conn(0x7f69980c9770 msgr2=0x7f69980c56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:50.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.466+0000 7f699ee0b700 1 -- 172.21.15.175:0/3642678763 shutdown_connections 2023-10-10T18:45:50.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.466+0000 7f699ee0b700 1 -- 172.21.15.175:0/3642678763 wait complete. 2023-10-10T18:45:50.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.467+0000 7f699ee0b700 1 Processor -- start 2023-10-10T18:45:50.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.467+0000 7f699ee0b700 1 -- start start 2023-10-10T18:45:50.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699ee0b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699806d5b0 0x7f6998156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699ee0b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c5010 0x7f6998156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699ee0b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f69981514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699ee0b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6998155320 con 0x7f6998150130 2023-10-10T18:45:50.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699ee0b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6998167310 con 0x7f69980c5010 2023-10-10T18:45:50.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699ee0b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6998151030 con 0x7f699806d5b0 2023-10-10T18:45:50.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f69981514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f69981514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37668/0 (socket says 172.21.15.175:37668) 2023-10-10T18:45:50.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 -- 172.21.15.175:0/2844938034 learned_addr learned my addr 172.21.15.175:0/2844938034 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:50.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699cba7700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699806d5b0 0x7f6998156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f698ffff700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c5010 0x7f6998156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 -- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699806d5b0 msgr2=0x7f6998156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699806d5b0 0x7f6998156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 -- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c5010 msgr2=0x7f6998156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c5010 0x7f6998156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 -- 172.21.15.175:0/2844938034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994009ae0 con 0x7f6998150130 2023-10-10T18:45:50.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699cba7700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699806d5b0 0x7f6998156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:50.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f698ffff700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c5010 0x7f6998156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:50.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699d3a8700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f69981514c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f699000eab0 tx=0x7f69900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f698cff9700 1 -- 172.21.15.175:0/2844938034 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699000c920 con 0x7f6998150130 2023-10-10T18:45:50.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f698cff9700 1 -- 172.21.15.175:0/2844938034 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699000ca80 con 0x7f6998150130 2023-10-10T18:45:50.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6998166de0 con 0x7f6998150130 2023-10-10T18:45:50.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f698cff9700 1 -- 172.21.15.175:0/2844938034 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699000ce40 con 0x7f6998150130 2023-10-10T18:45:50.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.468+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6998167030 con 0x7f6998150130 2023-10-10T18:45:50.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.469+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6998150880 con 0x7f6998150130 2023-10-10T18:45:50.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.470+0000 7f698cff9700 1 -- 172.21.15.175:0/2844938034 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6990013680 con 0x7f6998150130 2023-10-10T18:45:50.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.470+0000 7f698cff9700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6984066f60 0x7f6984069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.470+0000 7f698cff9700 1 -- 172.21.15.175:0/2844938034 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6990084730 con 0x7f6998150130 2023-10-10T18:45:50.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.470+0000 7f699cba7700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6984066f60 0x7f6984069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.471+0000 7f699cba7700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6984066f60 0x7f6984069410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6994009880 tx=0x7f6994018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.474+0000 7f698cff9700 1 -- 172.21.15.175:0/2844938034 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6998150880 con 0x7f6998150130 2023-10-10T18:45:50.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.658+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6998150880 con 0x7f6998150130 2023-10-10T18:45:50.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.659+0000 7f698cff9700 1 -- 172.21.15.175:0/2844938034 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6998150880 con 0x7f6998150130 2023-10-10T18:45:50.659 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 1 2023-10-10T18:45:50.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6984066f60 msgr2=0x7f6984069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6984066f60 0x7f6984069410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6994009880 tx=0x7f6994018000 comp rx=0 tx=0).stop 2023-10-10T18:45:50.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 msgr2=0x7f69981514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f69981514c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f699000eab0 tx=0x7f69900091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 shutdown_connections 2023-10-10T18:45:50.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6984066f60 0x7f6984069410 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6998150130 0x7f69981514c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f699806d5b0 0x7f6998156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 --2- 172.21.15.175:0/2844938034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69980c5010 0x7f6998156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 >> 172.21.15.175:0/2844938034 conn(0x7f69980c9770 msgr2=0x7f69980c73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:50.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 shutdown_connections 2023-10-10T18:45:50.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.661+0000 7f699ee0b700 1 -- 172.21.15.175:0/2844938034 wait complete. 2023-10-10T18:45:50.673 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T18:45:50.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.791+0000 7f2f3382c700 1 Processor -- start 2023-10-10T18:45:50.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f3382c700 1 -- start start 2023-10-10T18:45:50.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f3382c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f2c0c71e0 0x7f2f2c06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f3382c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2f2c0c75b0 0x7f2f2c063bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f3382c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2c0c66e0 con 0x7f2f2c06bf30 2023-10-10T18:45:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f3382c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2c0c69c0 con 0x7f2f2c0c71e0 2023-10-10T18:45:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f3382c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2c0c6ca0 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f315c8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f2c0c71e0 0x7f2f2c06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f31dc9700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2f2c06bf30 0x7f2f2c06c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51528/0 (socket says 172.21.15.175:51528) 2023-10-10T18:45:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.792+0000 7f2f31dc9700 1 -- 172.21.15.175:0/42842116 learned_addr learned my addr 172.21.15.175:0/42842116 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f30dc7700 1 --2- 172.21.15.175:0/42842116 >> v2:172.21.15.191:3300/0 conn(0x7f2f2c0c75b0 0x7f2f2c063bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f315c8700 1 -- 172.21.15.175:0/42842116 >> v2:172.21.15.191:3300/0 conn(0x7f2f2c0c75b0 msgr2=0x7f2f2c063bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f315c8700 1 --2- 172.21.15.175:0/42842116 >> v2:172.21.15.191:3300/0 conn(0x7f2f2c0c75b0 0x7f2f2c063bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f315c8700 1 -- 172.21.15.175:0/42842116 >> v1:172.21.15.175:6789/0 conn(0x7f2f2c06bf30 legacy=0x7f2f2c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f315c8700 1 -- 172.21.15.175:0/42842116 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c068920 con 0x7f2f2c0c71e0 2023-10-10T18:45:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f2bfff700 1 -- 172.21.15.175:0/42842116 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017854085 0 0) 0x7f2f2c0c66e0 con 0x7f2f2c06bf30 2023-10-10T18:45:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f315c8700 1 --2- 172.21.15.175:0/42842116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c71e0 0x7f2f2c06a540 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c06e1f0 tx=0x7f2f1c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae20c5731ac2b46 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f2bfff700 1 -- 172.21.15.175:0/42842116 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c00e070 con 0x7f2f2c0c71e0 2023-10-10T18:45:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f2bfff700 1 -- 172.21.15.175:0/42842116 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c00bc30 con 0x7f2f2c0c71e0 2023-10-10T18:45:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f2bfff700 1 -- 172.21.15.175:0/42842116 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c012430 con 0x7f2f2c0c71e0 2023-10-10T18:45:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f3382c700 1 -- 172.21.15.175:0/42842116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c71e0 msgr2=0x7f2f2c06a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.793+0000 7f2f3382c700 1 --2- 172.21.15.175:0/42842116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c71e0 0x7f2f2c06a540 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c06e1f0 tx=0x7f2f1c009780 comp rx=0 tx=0).stop 2023-10-10T18:45:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.794+0000 7f2f3382c700 1 -- 172.21.15.175:0/42842116 shutdown_connections 2023-10-10T18:45:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.794+0000 7f2f3382c700 1 --2- 172.21.15.175:0/42842116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c71e0 0x7f2f2c06a540 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.794+0000 7f2f3382c700 1 --2- 172.21.15.175:0/42842116 >> v2:172.21.15.191:3300/0 conn(0x7f2f2c0c75b0 0x7f2f2c063bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.794+0000 7f2f3382c700 1 -- 172.21.15.175:0/42842116 >> 172.21.15.175:0/42842116 conn(0x7f2f2c06dd00 msgr2=0x7f2f2c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:50.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.794+0000 7f2f3382c700 1 -- 172.21.15.175:0/42842116 shutdown_connections 2023-10-10T18:45:50.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.794+0000 7f2f3382c700 1 -- 172.21.15.175:0/42842116 wait complete. 2023-10-10T18:45:50.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.795+0000 7f2f3382c700 1 Processor -- start 2023-10-10T18:45:50.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.795+0000 7f2f3382c700 1 -- start start 2023-10-10T18:45:50.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.795+0000 7f2f3382c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f2c06bf30 0x7f2f2c152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.795+0000 7f2f3382c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f2c0c71e0 0x7f2f2c155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.795+0000 7f2f3382c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c75b0 0x7f2f2c169360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.795+0000 7f2f3382c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c151c50 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.795+0000 7f2f3382c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c1666f0 con 0x7f2f2c0c71e0 2023-10-10T18:45:50.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.795+0000 7f2f3382c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c0caa00 con 0x7f2f2c06bf30 2023-10-10T18:45:50.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c75b0 0x7f2f2c169360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c75b0 0x7f2f2c169360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37698/0 (socket says 172.21.15.175:37698) 2023-10-10T18:45:50.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 -- 172.21.15.175:0/2235160681 learned_addr learned my addr 172.21.15.175:0/2235160681 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:50.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f30dc7700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f2c0c71e0 0x7f2f2c155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f315c8700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f2c06bf30 0x7f2f2c152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 -- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f2c06bf30 msgr2=0x7f2f2c152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f2c06bf30 0x7f2f2c152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 -- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f2c0c71e0 msgr2=0x7f2f2c155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f2c0c71e0 0x7f2f2c155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 -- 172.21.15.175:0/2235160681 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c009a20 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f30dc7700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f2c0c71e0 0x7f2f2c155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:50.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f315c8700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f2c06bf30 0x7f2f2c152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:50.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f31dc9700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c75b0 0x7f2f2c169360 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400c7e0 tx=0x7f2f24007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f297fa700 1 -- 172.21.15.175:0/2235160681 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24005c00 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f297fa700 1 -- 172.21.15.175:0/2235160681 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f24005d60 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2c165de0 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f297fa700 1 -- 172.21.15.175:0/2235160681 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24004a80 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.796+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2c1660a0 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.797+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2c1668a0 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.798+0000 7f2f297fa700 1 -- 172.21.15.175:0/2235160681 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2f24005030 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.798+0000 7f2f297fa700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f18066f30 0x7f2f180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:50.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.798+0000 7f2f297fa700 1 -- 172.21.15.175:0/2235160681 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2f240839a0 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.798+0000 7f2f315c8700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f18066f30 0x7f2f180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:50.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.799+0000 7f2f315c8700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f18066f30 0x7f2f180693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c06e1f0 tx=0x7f2f1c019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:50.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.802+0000 7f2f297fa700 1 -- 172.21.15.175:0/2235160681 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f2404b5a0 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.983+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f2c067570 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.983+0000 7f2f297fa700 1 -- 172.21.15.175:0/2235160681 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f2c1668a0 con 0x7f2f2c0c75b0 2023-10-10T18:45:50.984 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 32 2023-10-10T18:45:50.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f18066f30 msgr2=0x7f2f180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f18066f30 0x7f2f180693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c06e1f0 tx=0x7f2f1c019000 comp rx=0 tx=0).stop 2023-10-10T18:45:50.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c75b0 msgr2=0x7f2f2c169360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:50.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c75b0 0x7f2f2c169360 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400c7e0 tx=0x7f2f24007d90 comp rx=0 tx=0).stop 2023-10-10T18:45:50.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 shutdown_connections 2023-10-10T18:45:50.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f18066f30 0x7f2f180693e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f2c0c75b0 0x7f2f2c169360 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f2c06bf30 0x7f2f2c152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 --2- 172.21.15.175:0/2235160681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f2c0c71e0 0x7f2f2c155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:50.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 >> 172.21.15.175:0/2235160681 conn(0x7f2f2c06dd00 msgr2=0x7f2f2c0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:50.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 shutdown_connections 2023-10-10T18:45:50.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:50.985+0000 7f2f3382c700 1 -- 172.21.15.175:0/2235160681 wait complete. 2023-10-10T18:45:50.997 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T18:45:51.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.114+0000 7f8e8d918700 1 Processor -- start 2023-10-10T18:45:51.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.114+0000 7f8e8d918700 1 -- start start 2023-10-10T18:45:51.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8d918700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8e88068630 0x7f8e880c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8d918700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88155f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8d918700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e88156dd0 con 0x7f8e88068a00 2023-10-10T18:45:51.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8d918700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e880c7650 con 0x7f8e88068630 2023-10-10T18:45:51.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8d918700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88069730 con 0x7f8e880c7ae0 2023-10-10T18:45:51.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8c916700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8e88068630 0x7f8e880c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8d117700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8e88068a00 0x7f8e880c9460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51530/0 (socket says 172.21.15.175:51530) 2023-10-10T18:45:51.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8d117700 1 -- 172.21.15.175:0/3089926080 learned_addr learned my addr 172.21.15.175:0/3089926080 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:51.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e87fff700 1 --2- 172.21.15.175:0/3089926080 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8c916700 1 -- 172.21.15.175:0/3089926080 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 msgr2=0x7f8e88155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8c916700 1 --2- 172.21.15.175:0/3089926080 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8c916700 1 -- 172.21.15.175:0/3089926080 >> v1:172.21.15.175:6789/0 conn(0x7f8e88068a00 legacy=0x7f8e880c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e877fe700 1 -- 172.21.15.175:0/3089926080 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940606826 0 0) 0x7f8e88156dd0 con 0x7f8e88068a00 2023-10-10T18:45:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8c916700 1 -- 172.21.15.175:0/3089926080 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e880c7f10 con 0x7f8e88068630 2023-10-10T18:45:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e8c916700 1 --2- 172.21.15.175:0/3089926080 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880c39d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0098e0 tx=0x7f8e7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34bbb3604b0cf4c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e877fe700 1 -- 172.21.15.175:0/3089926080 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c009e80 con 0x7f8e88068630 2023-10-10T18:45:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e877fe700 1 -- 172.21.15.175:0/3089926080 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c00b860 con 0x7f8e88068630 2023-10-10T18:45:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.115+0000 7f8e877fe700 1 -- 172.21.15.175:0/3089926080 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c00bbe0 con 0x7f8e88068630 2023-10-10T18:45:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.116+0000 7f8e8d918700 1 -- 172.21.15.175:0/3089926080 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 msgr2=0x7f8e880c39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.116+0000 7f8e8d918700 1 --2- 172.21.15.175:0/3089926080 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880c39d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0098e0 tx=0x7f8e7c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.116+0000 7f8e8d918700 1 -- 172.21.15.175:0/3089926080 shutdown_connections 2023-10-10T18:45:51.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.116+0000 7f8e8d918700 1 --2- 172.21.15.175:0/3089926080 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880c39d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.116+0000 7f8e8d918700 1 --2- 172.21.15.175:0/3089926080 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.116+0000 7f8e8d918700 1 -- 172.21.15.175:0/3089926080 >> 172.21.15.175:0/3089926080 conn(0x7f8e88063230 msgr2=0x7f8e88063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:51.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.116+0000 7f8e8d918700 1 -- 172.21.15.175:0/3089926080 shutdown_connections 2023-10-10T18:45:51.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.116+0000 7f8e8d918700 1 -- 172.21.15.175:0/3089926080 wait complete. 2023-10-10T18:45:51.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.117+0000 7f8e8d918700 1 Processor -- start 2023-10-10T18:45:51.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.117+0000 7f8e8d918700 1 -- start start 2023-10-10T18:45:51.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8d918700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880bfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8d918700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e88068a00 0x7f8e88151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8d918700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8d918700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e880c0650 con 0x7f8e88068630 2023-10-10T18:45:51.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8d918700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88165b10 con 0x7f8e88068a00 2023-10-10T18:45:51.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8d918700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88166910 con 0x7f8e880c7ae0 2023-10-10T18:45:51.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880bfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880bfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37718/0 (socket says 172.21.15.175:37718) 2023-10-10T18:45:51.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 -- 172.21.15.175:0/1939513142 learned_addr learned my addr 172.21.15.175:0/1939513142 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:51.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8d117700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e87fff700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e88068a00 0x7f8e88151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 -- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 msgr2=0x7f8e88152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 -- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e88068a00 msgr2=0x7f8e88151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e88068a00 0x7f8e88151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 -- 172.21.15.175:0/1939513142 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c009420 con 0x7f8e88068630 2023-10-10T18:45:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8d117700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e87fff700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e88068a00 0x7f8e88151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.118+0000 7f8e8c916700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880bfc50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8e88089810 tx=0x7f8e7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.119+0000 7f8e84ff9700 1 -- 172.21.15.175:0/1939513142 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c00bbe0 con 0x7f8e88068630 2023-10-10T18:45:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.119+0000 7f8e84ff9700 1 -- 172.21.15.175:0/1939513142 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c011420 con 0x7f8e88068630 2023-10-10T18:45:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.119+0000 7f8e84ff9700 1 -- 172.21.15.175:0/1939513142 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0117e0 con 0x7f8e88068630 2023-10-10T18:45:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.119+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e880c0290 con 0x7f8e88068630 2023-10-10T18:45:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.119+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e88165db0 con 0x7f8e88068630 2023-10-10T18:45:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.119+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e880c5320 con 0x7f8e88068630 2023-10-10T18:45:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.120+0000 7f8e84ff9700 1 -- 172.21.15.175:0/1939513142 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8e7c011b90 con 0x7f8e88068630 2023-10-10T18:45:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.120+0000 7f8e84ff9700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8e6c066f30 0x7f8e6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.120+0000 7f8e84ff9700 1 -- 172.21.15.175:0/1939513142 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8e7c085eb0 con 0x7f8e88068630 2023-10-10T18:45:51.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.120+0000 7f8e87fff700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8e6c066f30 0x7f8e6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.121+0000 7f8e87fff700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8e6c066f30 0x7f8e6c0693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8e88166bd0 tx=0x7f8e74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.125+0000 7f8e84ff9700 1 -- 172.21.15.175:0/1939513142 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8e7c053120 con 0x7f8e88068630 2023-10-10T18:45:51.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.304+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8e880715d0 con 0x7f8e88068630 2023-10-10T18:45:51.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.304+0000 7f8e84ff9700 1 -- 172.21.15.175:0/1939513142 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8e7c056740 con 0x7f8e88068630 2023-10-10T18:45:51.305 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 64 2023-10-10T18:45:51.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8e6c066f30 msgr2=0x7f8e6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8e6c066f30 0x7f8e6c0693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8e88166bd0 tx=0x7f8e74008040 comp rx=0 tx=0).stop 2023-10-10T18:45:51.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 msgr2=0x7f8e880bfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880bfc50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8e88089810 tx=0x7f8e7c017000 comp rx=0 tx=0).stop 2023-10-10T18:45:51.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 shutdown_connections 2023-10-10T18:45:51.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8e6c066f30 0x7f8e6c0693e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e880c7ae0 0x7f8e88152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e88068630 0x7f8e880bfc50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 --2- 172.21.15.175:0/1939513142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e88068a00 0x7f8e88151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 >> 172.21.15.175:0/1939513142 conn(0x7f8e88063230 msgr2=0x7f8e88158900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:51.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 shutdown_connections 2023-10-10T18:45:51.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.306+0000 7f8e8d918700 1 -- 172.21.15.175:0/1939513142 wait complete. 2023-10-10T18:45:51.318 INFO:tasks.mds_thrash:mds thrasher using random seed: 1696963551 2023-10-10T18:45:51.319 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:51.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.435+0000 7ff3f5a5f700 1 Processor -- start 2023-10-10T18:45:51.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.435+0000 7ff3f5a5f700 1 -- start start 2023-10-10T18:45:51.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3f5a5f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff3f0157400 0x7ff3f006bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3f5a5f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff3f00c4640 0x7ff3f01577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3f5a5f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 0x7ff3f014d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3f5a5f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f006e050 con 0x7ff3f00c4640 2023-10-10T18:45:51.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3f5a5f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f006d8e0 con 0x7ff3f0157400 2023-10-10T18:45:51.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3f5a5f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f006dbc0 con 0x7ff3f0157d10 2023-10-10T18:45:51.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3e7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff3f00c4640 0x7ff3f01577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3e7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff3f00c4640 0x7ff3f01577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37724/0 (socket says 172.21.15.175:37724) 2023-10-10T18:45:51.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3e7fff700 1 -- 172.21.15.175:0/4167951313 learned_addr learned my addr 172.21.15.175:0/4167951313 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:51.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3f4a5d700 1 --2- 172.21.15.175:0/4167951313 >> v2:172.21.15.191:3300/0 conn(0x7ff3f0157400 0x7ff3f006bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.436+0000 7ff3f525e700 1 --2- 172.21.15.175:0/4167951313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 0x7ff3f014d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e7fff700 1 -- 172.21.15.175:0/4167951313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 msgr2=0x7ff3f014d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:51.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e7fff700 1 --2- 172.21.15.175:0/4167951313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 0x7ff3f014d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e7fff700 1 -- 172.21.15.175:0/4167951313 >> v2:172.21.15.191:3300/0 conn(0x7ff3f0157400 msgr2=0x7ff3f006bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:51.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e7fff700 1 --2- 172.21.15.175:0/4167951313 >> v2:172.21.15.191:3300/0 conn(0x7ff3f0157400 0x7ff3f006bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e7fff700 1 -- 172.21.15.175:0/4167951313 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f00c1030 con 0x7ff3f00c4640 2023-10-10T18:45:51.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e7fff700 1 --2- 172.21.15.175:0/4167951313 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f01577d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0067120 tx=0x7ff3e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe698ac10c86f789 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e77fe700 1 -- 172.21.15.175:0/4167951313 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e000b640 con 0x7ff3f00c4640 2023-10-10T18:45:51.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e77fe700 1 -- 172.21.15.175:0/4167951313 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e000b7a0 con 0x7ff3f00c4640 2023-10-10T18:45:51.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3e77fe700 1 -- 172.21.15.175:0/4167951313 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e000bb60 con 0x7ff3f00c4640 2023-10-10T18:45:51.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/4167951313 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 msgr2=0x7ff3f01577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/4167951313 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f01577d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0067120 tx=0x7ff3e0009250 comp rx=0 tx=0).stop 2023-10-10T18:45:51.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/4167951313 shutdown_connections 2023-10-10T18:45:51.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/4167951313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 0x7ff3f014d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/4167951313 >> v2:172.21.15.191:3300/0 conn(0x7ff3f0157400 0x7ff3f006bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/4167951313 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f01577d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.437+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/4167951313 >> 172.21.15.175:0/4167951313 conn(0x7ff3f0063cd0 msgr2=0x7ff3f006d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:51.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.438+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/4167951313 shutdown_connections 2023-10-10T18:45:51.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.438+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/4167951313 wait complete. 2023-10-10T18:45:51.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.439+0000 7ff3f5a5f700 1 Processor -- start 2023-10-10T18:45:51.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.439+0000 7ff3f5a5f700 1 -- start start 2023-10-10T18:45:51.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.439+0000 7ff3f5a5f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f0068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.439+0000 7ff3f5a5f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f0157400 0x7ff3f00c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.439+0000 7ff3f5a5f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 0x7ff3f0169980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f5a5f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00c96f0 con 0x7ff3f00c4640 2023-10-10T18:45:51.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f5a5f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00c9920 con 0x7ff3f0157400 2023-10-10T18:45:51.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f5a5f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0154410 con 0x7ff3f0157d10 2023-10-10T18:45:51.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f0068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f0068900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37738/0 (socket says 172.21.15.175:37738) 2023-10-10T18:45:51.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 -- 172.21.15.175:0/3667331283 learned_addr learned my addr 172.21.15.175:0/3667331283 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:51.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f525e700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 0x7ff3f0169980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3e7fff700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f0157400 0x7ff3f00c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 -- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 msgr2=0x7ff3f0169980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 0x7ff3f0169980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 -- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f0157400 msgr2=0x7ff3f00c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f0157400 0x7ff3f00c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 -- 172.21.15.175:0/3667331283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e00094f0 con 0x7ff3f00c4640 2023-10-10T18:45:51.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f4a5d700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f0068900 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009ef0 tx=0x7ff3ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3e4ff9700 1 -- 172.21.15.175:0/3667331283 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec00cc50 con 0x7ff3f00c4640 2023-10-10T18:45:51.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3e4ff9700 1 -- 172.21.15.175:0/3667331283 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec00f040 con 0x7ff3f00c4640 2023-10-10T18:45:51.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f0068e40 con 0x7ff3f00c4640 2023-10-10T18:45:51.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3e4ff9700 1 -- 172.21.15.175:0/3667331283 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec014480 con 0x7ff3f00c4640 2023-10-10T18:45:51.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.440+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f01540a0 con 0x7ff3f00c4640 2023-10-10T18:45:51.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.441+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f01546f0 con 0x7ff3f00c4640 2023-10-10T18:45:51.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.441+0000 7ff3e4ff9700 1 -- 172.21.15.175:0/3667331283 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff3ec0147f0 con 0x7ff3f00c4640 2023-10-10T18:45:51.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.442+0000 7ff3e4ff9700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff3dc066f30 0x7ff3dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.442+0000 7ff3e4ff9700 1 -- 172.21.15.175:0/3667331283 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff3ec085280 con 0x7ff3f00c4640 2023-10-10T18:45:51.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.442+0000 7ff3e7fff700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff3dc066f30 0x7ff3dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.443+0000 7ff3e7fff700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff3dc066f30 0x7ff3dc0693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0067120 tx=0x7ff3e0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.446+0000 7ff3e4ff9700 1 -- 172.21.15.175:0/3667331283 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3ec0524f0 con 0x7ff3f00c4640 2023-10-10T18:45:51.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.632+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3f0067570 con 0x7ff3f00c4640 2023-10-10T18:45:51.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.633+0000 7ff3e4ff9700 1 -- 172.21.15.175:0/3667331283 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7ff3ec055b10 con 0x7ff3f00c4640 2023-10-10T18:45:51.633 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:51.633 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:51.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff3dc066f30 msgr2=0x7ff3dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff3dc066f30 0x7ff3dc0693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0067120 tx=0x7ff3e0017000 comp rx=0 tx=0).stop 2023-10-10T18:45:51.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 msgr2=0x7ff3f0068900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f0068900 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009ef0 tx=0x7ff3ec00b040 comp rx=0 tx=0).stop 2023-10-10T18:45:51.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 shutdown_connections 2023-10-10T18:45:51.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff3dc066f30 0x7ff3dc0693e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f0157d10 0x7ff3f0169980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3f00c4640 0x7ff3f0068900 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 --2- 172.21.15.175:0/3667331283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f0157400 0x7ff3f00c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 >> 172.21.15.175:0/3667331283 conn(0x7ff3f0063cd0 msgr2=0x7ff3f00c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:51.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 shutdown_connections 2023-10-10T18:45:51.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.635+0000 7ff3f5a5f700 1 -- 172.21.15.175:0/3667331283 wait complete. 2023-10-10T18:45:51.647 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T18:45:05.161495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T18:45:08.855952+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T18:45:09.742056+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T18:45:51.647 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T18:45:51.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.765+0000 7f84aaad2700 1 Processor -- start 2023-10-10T18:45:51.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.765+0000 7f84aaad2700 1 -- start start 2023-10-10T18:45:51.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.766+0000 7f84aaad2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84a406f4f0 0x7f84a414cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.766+0000 7f84aaad2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a40c94f0 con 0x7f84a4068b50 2023-10-10T18:45:51.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.766+0000 7f84aaad2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a4152d50 con 0x7f84a4068f20 2023-10-10T18:45:51.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.766+0000 7f84aaad2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a4152fb0 con 0x7f84a406f4f0 2023-10-10T18:45:51.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.766+0000 7f84a886e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f84a4068b50 0x7f84a40c6560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51536/0 (socket says 172.21.15.175:51536) 2023-10-10T18:45:51.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.766+0000 7f84a886e700 1 -- 172.21.15.175:0/2067156756 learned_addr learned my addr 172.21.15.175:0/2067156756 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:51.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a3fff700 1 --2- 172.21.15.175:0/2067156756 >> v2:172.21.15.191:3300/0 conn(0x7f84a406f4f0 0x7f84a414cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896419594 0 0) 0x7f84a40c94f0 con 0x7f84a4068b50 2023-10-10T18:45:51.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8490003570 con 0x7f84a4068b50 2023-10-10T18:45:51.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336580854 0 0) 0x7f8490003570 con 0x7f84a4068b50 2023-10-10T18:45:51.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 >> v2:172.21.15.191:3300/0 conn(0x7f84a406f4f0 msgr2=0x7f84a414cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:51.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 --2- 172.21.15.175:0/2067156756 >> v2:172.21.15.191:3300/0 conn(0x7f84a406f4f0 0x7f84a414cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 >> v1:172.21.15.191:6789/0 conn(0x7f84a4068f20 legacy=0x7f84a40c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4153270 con 0x7f84a4068b50 2023-10-10T18:45:51.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f8494001470 con 0x7f84a4068b50 2023-10-10T18:45:51.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 >> v1:172.21.15.175:6789/0 conn(0x7f84a4068b50 legacy=0x7f84a40c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8490003a60 0x7f8490003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8490004810 0x7f8490004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84900051c0 0x7f849000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4152fb0 con 0x7f8490004810 2023-10-10T18:45:51.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849000c0d0 con 0x7f84900051c0 2023-10-10T18:45:51.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849000c290 con 0x7f8490003a60 2023-10-10T18:45:51.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a3fff700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8490004810 0x7f8490004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a886e700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8490003a60 0x7f8490003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a906f700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84900051c0 0x7f849000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a3fff700 1 -- 172.21.15.175:0/2067156756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8490003a60 msgr2=0x7f8490003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:51.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a3fff700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8490003a60 0x7f8490003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a3fff700 1 -- 172.21.15.175:0/2067156756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84900051c0 msgr2=0x7f849000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:51.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a3fff700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84900051c0 0x7f849000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a3fff700 1 -- 172.21.15.175:0/2067156756 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490003570 con 0x7f8490004810 2023-10-10T18:45:51.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a906f700 1 -- 172.21.15.175:0/2067156756 reap_dead start 2023-10-10T18:45:51.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.767+0000 7f84a3fff700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8490004810 0x7f8490004c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f848c00e210 tx=0x7f848c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89101c5d5d2e63aa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c00eac0 con 0x7f8490004810 2023-10-10T18:45:51.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c00f040 con 0x7f8490004810 2023-10-10T18:45:51.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84a37fe700 1 -- 172.21.15.175:0/2067156756 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c00ee70 con 0x7f8490004810 2023-10-10T18:45:51.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84aaad2700 1 -- 172.21.15.175:0/2067156756 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8490004810 msgr2=0x7f8490004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84aaad2700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8490004810 0x7f8490004c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f848c00e210 tx=0x7f848c00b040 comp rx=0 tx=0).stop 2023-10-10T18:45:51.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84aaad2700 1 -- 172.21.15.175:0/2067156756 shutdown_connections 2023-10-10T18:45:51.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84aaad2700 1 --2- 172.21.15.175:0/2067156756 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8490004810 0x7f8490004c80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84aaad2700 1 -- 172.21.15.175:0/2067156756 >> 172.21.15.175:0/2067156756 conn(0x7f84a4063230 msgr2=0x7f84a406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:51.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84aaad2700 1 -- 172.21.15.175:0/2067156756 shutdown_connections 2023-10-10T18:45:51.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.768+0000 7f84aaad2700 1 -- 172.21.15.175:0/2067156756 wait complete. 2023-10-10T18:45:51.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.769+0000 7f84aaad2700 1 Processor -- start 2023-10-10T18:45:51.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.769+0000 7f84aaad2700 1 -- start start 2023-10-10T18:45:51.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84aaad2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8490004810 0x7f84a414f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84aaad2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84a4158ac0 0x7f84a41635e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84aaad2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84a4163b20 0x7f84a41656e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84aaad2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4157e00 con 0x7f84a4163b20 2023-10-10T18:45:51.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84aaad2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4163fa0 con 0x7f8490004810 2023-10-10T18:45:51.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84aaad2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4153ec0 con 0x7f84a4158ac0 2023-10-10T18:45:51.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a906f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84a4163b20 0x7f84a41656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a906f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84a4163b20 0x7f84a41656e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37762/0 (socket says 172.21.15.175:37762) 2023-10-10T18:45:51.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a906f700 1 -- 172.21.15.175:0/2815221910 learned_addr learned my addr 172.21.15.175:0/2815221910 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:51.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a886e700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8490004810 0x7f84a414f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a3fff700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84a4158ac0 0x7f84a41635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a906f700 1 -- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84a4158ac0 msgr2=0x7f84a41635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a906f700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84a4158ac0 0x7f84a41635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a906f700 1 -- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8490004810 msgr2=0x7f84a414f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a906f700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8490004810 0x7f84a414f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.770+0000 7f84a906f700 1 -- 172.21.15.175:0/2815221910 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4154ce0 con 0x7f84a4163b20 2023-10-10T18:45:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.771+0000 7f84a906f700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84a4163b20 0x7f84a41656e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84a40c3a40 tx=0x7f849c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.771+0000 7f84a0ff9700 1 -- 172.21.15.175:0/2815221910 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c006a30 con 0x7f84a4163b20 2023-10-10T18:45:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.771+0000 7f84a0ff9700 1 -- 172.21.15.175:0/2815221910 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849c006bb0 con 0x7f84a4163b20 2023-10-10T18:45:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.771+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a40c2900 con 0x7f84a4163b20 2023-10-10T18:45:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.771+0000 7f84a0ff9700 1 -- 172.21.15.175:0/2815221910 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c007030 con 0x7f84a4163b20 2023-10-10T18:45:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.771+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a41543b0 con 0x7f84a4163b20 2023-10-10T18:45:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.772+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a4154e90 con 0x7f84a4163b20 2023-10-10T18:45:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.772+0000 7f84a0ff9700 1 -- 172.21.15.175:0/2815221910 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f849c007420 con 0x7f84a4163b20 2023-10-10T18:45:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.772+0000 7f84a0ff9700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8484066f60 0x7f8484069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.772+0000 7f84a0ff9700 1 -- 172.21.15.175:0/2815221910 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f849c083c60 con 0x7f84a4163b20 2023-10-10T18:45:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.772+0000 7f84a886e700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8484066f60 0x7f8484069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.773+0000 7f84a886e700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8484066f60 0x7f8484069410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f84940015a0 tx=0x7f84a40c4310 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:51.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.776+0000 7f84a0ff9700 1 -- 172.21.15.175:0/2815221910 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84a4154e90 con 0x7f84a4163b20 2023-10-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.959+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84a4154e90 con 0x7f84a4163b20 2023-10-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.959+0000 7f84a0ff9700 1 -- 172.21.15.175:0/2815221910 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f84a4154e90 con 0x7f84a4163b20 2023-10-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 1 2023-10-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8484066f60 msgr2=0x7f8484069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8484066f60 0x7f8484069410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f84940015a0 tx=0x7f84a40c4310 comp rx=0 tx=0).stop 2023-10-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84a4163b20 msgr2=0x7f84a41656e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84a4163b20 0x7f84a41656e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84a40c3a40 tx=0x7f849c007d30 comp rx=0 tx=0).stop 2023-10-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 shutdown_connections 2023-10-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8484066f60 0x7f8484069410 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84a4163b20 0x7f84a41656e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8490004810 0x7f84a414f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 --2- 172.21.15.175:0/2815221910 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84a4158ac0 0x7f84a41635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 >> 172.21.15.175:0/2815221910 conn(0x7f84a4063230 msgr2=0x7f84a4151310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 shutdown_connections 2023-10-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:51.961+0000 7f84aaad2700 1 -- 172.21.15.175:0/2815221910 wait complete. 2023-10-10T18:45:51.973 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T18:45:52.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.090+0000 7f18356cb700 1 Processor -- start 2023-10-10T18:45:52.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.090+0000 7f18356cb700 1 -- start start 2023-10-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f18356cb700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f18300c6b70 0x7f1830063cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f18356cb700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f183006a540 0x7f183006b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f18356cb700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183006c480 con 0x7f18300c67a0 2023-10-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f18356cb700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18300c7bb0 con 0x7f183006a540 2023-10-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f18356cb700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18300c5360 con 0x7f18300c6b70 2023-10-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f182f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f183006a540 0x7f183006b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f182f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f183006a540 0x7f183006b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37778/0 (socket says 172.21.15.175:37778) 2023-10-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f182f7fe700 1 -- 172.21.15.175:0/2475403286 learned_addr learned my addr 172.21.15.175:0/2475403286 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:52.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f182e7fc700 1 --2- 172.21.15.175:0/2475403286 >> v2:172.21.15.191:3300/0 conn(0x7f18300c6b70 0x7f1830063cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f182f7fe700 1 -- 172.21.15.175:0/2475403286 >> v2:172.21.15.191:3300/0 conn(0x7f18300c6b70 msgr2=0x7f1830063cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:52.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f182f7fe700 1 --2- 172.21.15.175:0/2475403286 >> v2:172.21.15.191:3300/0 conn(0x7f18300c6b70 0x7f1830063cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.091+0000 7f182f7fe700 1 -- 172.21.15.175:0/2475403286 >> v1:172.21.15.191:6789/0 conn(0x7f18300c67a0 legacy=0x7f18300c1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:52.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f182f7fe700 1 -- 172.21.15.175:0/2475403286 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1830068be0 con 0x7f183006a540 2023-10-10T18:45:52.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f182e7fc700 1 --2- 172.21.15.175:0/2475403286 >> v2:172.21.15.191:3300/0 conn(0x7f18300c6b70 0x7f1830063cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:52.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f182f7fe700 1 --2- 172.21.15.175:0/2475403286 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f183006b3e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f183006ee90 tx=0x7f1828009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb59e73b49536921 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f182dffb700 1 -- 172.21.15.175:0/2475403286 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182800b820 con 0x7f183006a540 2023-10-10T18:45:52.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f182dffb700 1 -- 172.21.15.175:0/2475403286 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182800b980 con 0x7f183006a540 2023-10-10T18:45:52.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f182dffb700 1 -- 172.21.15.175:0/2475403286 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182800bd20 con 0x7f183006a540 2023-10-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f18356cb700 1 -- 172.21.15.175:0/2475403286 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 msgr2=0x7f183006b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f18356cb700 1 --2- 172.21.15.175:0/2475403286 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f183006b3e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f183006ee90 tx=0x7f1828009250 comp rx=0 tx=0).stop 2023-10-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f18356cb700 1 -- 172.21.15.175:0/2475403286 shutdown_connections 2023-10-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f18356cb700 1 --2- 172.21.15.175:0/2475403286 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f183006b3e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f18356cb700 1 --2- 172.21.15.175:0/2475403286 >> v2:172.21.15.191:3300/0 conn(0x7f18300c6b70 0x7f1830063cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f18356cb700 1 -- 172.21.15.175:0/2475403286 >> 172.21.15.175:0/2475403286 conn(0x7f183006dcc0 msgr2=0x7f1830063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f18356cb700 1 -- 172.21.15.175:0/2475403286 shutdown_connections 2023-10-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.092+0000 7f18356cb700 1 -- 172.21.15.175:0/2475403286 wait complete. 2023-10-10T18:45:52.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f18356cb700 1 Processor -- start 2023-10-10T18:45:52.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f18356cb700 1 -- start start 2023-10-10T18:45:52.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f18356cb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f18300c2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f18356cb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18300c67a0 0x7f1830169810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f18356cb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18300c6b70 0x7f1830157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f18356cb700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18300af990 con 0x7f183006a540 2023-10-10T18:45:52.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f18356cb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18300af3d0 con 0x7f18300c6b70 2023-10-10T18:45:52.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f18356cb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18300af6b0 con 0x7f18300c67a0 2023-10-10T18:45:52.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f182effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f18300c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f182effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f18300c2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37794/0 (socket says 172.21.15.175:37794) 2023-10-10T18:45:52.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.094+0000 7f182effd700 1 -- 172.21.15.175:0/1985607971 learned_addr learned my addr 172.21.15.175:0/1985607971 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:52.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f182f7fe700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18300c6b70 0x7f1830157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f182e7fc700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18300c67a0 0x7f1830169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f182effd700 1 -- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18300c67a0 msgr2=0x7f1830169810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f182effd700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18300c67a0 0x7f1830169810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f182effd700 1 -- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18300c6b70 msgr2=0x7f1830157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f182effd700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18300c6b70 0x7f1830157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f182effd700 1 -- 172.21.15.175:0/1985607971 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18280094f0 con 0x7f183006a540 2023-10-10T18:45:52.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f182effd700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f18300c2430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f182000c950 tx=0x7f1820002760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f1816ffd700 1 -- 172.21.15.175:0/1985607971 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182000a7e0 con 0x7f183006a540 2023-10-10T18:45:52.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f1816ffd700 1 -- 172.21.15.175:0/1985607971 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182000a940 con 0x7f183006a540 2023-10-10T18:45:52.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f1816ffd700 1 -- 172.21.15.175:0/1985607971 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182000ad00 con 0x7f183006a540 2023-10-10T18:45:52.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18300aa9f0 con 0x7f183006a540 2023-10-10T18:45:52.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.095+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18300c29e0 con 0x7f183006a540 2023-10-10T18:45:52.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.096+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1830165ae0 con 0x7f183006a540 2023-10-10T18:45:52.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.096+0000 7f1816ffd700 1 -- 172.21.15.175:0/1985607971 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1820012630 con 0x7f183006a540 2023-10-10T18:45:52.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.097+0000 7f1816ffd700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f181c066f30 0x7f181c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.097+0000 7f1816ffd700 1 -- 172.21.15.175:0/1985607971 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1820084a50 con 0x7f183006a540 2023-10-10T18:45:52.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.097+0000 7f182e7fc700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f181c066f30 0x7f181c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.097+0000 7f182e7fc700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f181c066f30 0x7f181c0693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f18300c2970 tx=0x7f1824006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.101+0000 7f1816ffd700 1 -- 172.21.15.175:0/1985607971 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1820051cc0 con 0x7f183006a540 2023-10-10T18:45:52.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.283+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f18300c7e10 con 0x7f183006a540 2023-10-10T18:45:52.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.283+0000 7f1816ffd700 1 -- 172.21.15.175:0/1985607971 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f182000ae60 con 0x7f183006a540 2023-10-10T18:45:52.283 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 32 2023-10-10T18:45:52.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f181c066f30 msgr2=0x7f181c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f181c066f30 0x7f181c0693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f18300c2970 tx=0x7f1824006f90 comp rx=0 tx=0).stop 2023-10-10T18:45:52.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 msgr2=0x7f18300c2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f18300c2430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f182000c950 tx=0x7f1820002760 comp rx=0 tx=0).stop 2023-10-10T18:45:52.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 shutdown_connections 2023-10-10T18:45:52.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f181c066f30 0x7f181c0693e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18300c6b70 0x7f1830157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183006a540 0x7f18300c2430 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 --2- 172.21.15.175:0/1985607971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18300c67a0 0x7f1830169810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 >> 172.21.15.175:0/1985607971 conn(0x7f183006dcc0 msgr2=0x7f18300c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:52.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 shutdown_connections 2023-10-10T18:45:52.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.285+0000 7f18356cb700 1 -- 172.21.15.175:0/1985607971 wait complete. 2023-10-10T18:45:52.297 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T18:45:52.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.416+0000 7f6e76ab4700 1 Processor -- start 2023-10-10T18:45:52.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.416+0000 7f6e76ab4700 1 -- start start 2023-10-10T18:45:52.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e76ab4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6e700c43b0 0x7f6e700bd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e76ab4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6e700c5780 0x7f6e700bd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e76ab4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7006fbb0 con 0x7f6e70150110 2023-10-10T18:45:52.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e76ab4700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e700be2d0 con 0x7f6e700c43b0 2023-10-10T18:45:52.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e76ab4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e700be590 con 0x7f6e700c5780 2023-10-10T18:45:52.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e752b1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6e700c43b0 0x7f6e700bd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e752b1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6e700c43b0 0x7f6e700bd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37806/0 (socket says 172.21.15.175:37806) 2023-10-10T18:45:52.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e752b1700 1 -- 172.21.15.175:0/2086876339 learned_addr learned my addr 172.21.15.175:0/2086876339 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:52.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e762b3700 1 --2- 172.21.15.175:0/2086876339 >> v2:172.21.15.191:3300/0 conn(0x7f6e700c5780 0x7f6e700bd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e752b1700 1 -- 172.21.15.175:0/2086876339 >> v2:172.21.15.191:3300/0 conn(0x7f6e700c5780 msgr2=0x7f6e700bd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:52.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e752b1700 1 --2- 172.21.15.175:0/2086876339 >> v2:172.21.15.191:3300/0 conn(0x7f6e700c5780 0x7f6e700bd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e752b1700 1 -- 172.21.15.175:0/2086876339 >> v1:172.21.15.191:6789/0 conn(0x7f6e70150110 legacy=0x7f6e7006e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:52.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e752b1700 1 -- 172.21.15.175:0/2086876339 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e700be870 con 0x7f6e700c43b0 2023-10-10T18:45:52.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.417+0000 7f6e752b1700 1 --2- 172.21.15.175:0/2086876339 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c43b0 0x7f6e700bd450 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6e7006d030 tx=0x7f6e60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1bec90bc20aa91b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e74ab0700 1 -- 172.21.15.175:0/2086876339 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000b820 con 0x7f6e700c43b0 2023-10-10T18:45:52.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e74ab0700 1 -- 172.21.15.175:0/2086876339 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6000b980 con 0x7f6e700c43b0 2023-10-10T18:45:52.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e74ab0700 1 -- 172.21.15.175:0/2086876339 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000bd20 con 0x7f6e700c43b0 2023-10-10T18:45:52.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2086876339 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c43b0 msgr2=0x7f6e700bd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2086876339 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c43b0 0x7f6e700bd450 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6e7006d030 tx=0x7f6e60009250 comp rx=0 tx=0).stop 2023-10-10T18:45:52.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2086876339 shutdown_connections 2023-10-10T18:45:52.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2086876339 >> v2:172.21.15.191:3300/0 conn(0x7f6e700c5780 0x7f6e700bd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2086876339 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c43b0 0x7f6e700bd450 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2086876339 >> 172.21.15.175:0/2086876339 conn(0x7f6e700677f0 msgr2=0x7f6e70067bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:52.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2086876339 shutdown_connections 2023-10-10T18:45:52.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.418+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2086876339 wait complete. 2023-10-10T18:45:52.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.420+0000 7f6e76ab4700 1 Processor -- start 2023-10-10T18:45:52.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.420+0000 7f6e76ab4700 1 -- start start 2023-10-10T18:45:52.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.420+0000 7f6e76ab4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e700c43b0 0x7f6e700c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.420+0000 7f6e76ab4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c5780 0x7f6e700c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.420+0000 7f6e76ab4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e70150110 0x7f6e70162c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.420+0000 7f6e76ab4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700c0100 con 0x7f6e700c5780 2023-10-10T18:45:52.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.420+0000 7f6e76ab4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70162550 con 0x7f6e70150110 2023-10-10T18:45:52.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.420+0000 7f6e76ab4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e701627f0 con 0x7f6e700c43b0 2023-10-10T18:45:52.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c5780 0x7f6e700c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c5780 0x7f6e700c1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37816/0 (socket says 172.21.15.175:37816) 2023-10-10T18:45:52.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 -- 172.21.15.175:0/2003198260 learned_addr learned my addr 172.21.15.175:0/2003198260 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:52.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e762b3700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e70150110 0x7f6e70162c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e75ab2700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e700c43b0 0x7f6e700c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 -- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e700c43b0 msgr2=0x7f6e700c0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e700c43b0 0x7f6e700c0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 -- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e70150110 msgr2=0x7f6e70162c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e70150110 0x7f6e70162c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 -- 172.21.15.175:0/2003198260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e600094f0 con 0x7f6e700c5780 2023-10-10T18:45:52.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e762b3700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e70150110 0x7f6e70162c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:52.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e752b1700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c5780 0x7f6e700c1ca0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009290 tx=0x7f6e60016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e65ffb700 1 -- 172.21.15.175:0/2003198260 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60011e20 con 0x7f6e700c5780 2023-10-10T18:45:52.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e65ffb700 1 -- 172.21.15.175:0/2003198260 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60020070 con 0x7f6e700c5780 2023-10-10T18:45:52.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e701557e0 con 0x7f6e700c5780 2023-10-10T18:45:52.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e65ffb700 1 -- 172.21.15.175:0/2003198260 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6001b6a0 con 0x7f6e700c5780 2023-10-10T18:45:52.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.421+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e70155390 con 0x7f6e700c5780 2023-10-10T18:45:52.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.422+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7004bb80 con 0x7f6e700c5780 2023-10-10T18:45:52.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.422+0000 7f6e65ffb700 1 -- 172.21.15.175:0/2003198260 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6e60022030 con 0x7f6e700c5780 2023-10-10T18:45:52.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.423+0000 7f6e65ffb700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e54066f30 0x7f6e540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.423+0000 7f6e65ffb700 1 -- 172.21.15.175:0/2003198260 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6e600864d0 con 0x7f6e700c5780 2023-10-10T18:45:52.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.423+0000 7f6e75ab2700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e54066f30 0x7f6e540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.423+0000 7f6e75ab2700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e54066f30 0x7f6e540693e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c006fd0 tx=0x7f6e6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.427+0000 7f6e65ffb700 1 -- 172.21.15.175:0/2003198260 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e600537f0 con 0x7f6e700c5780 2023-10-10T18:45:52.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.609+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6e700c4190 con 0x7f6e700c5780 2023-10-10T18:45:52.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.610+0000 7f6e65ffb700 1 -- 172.21.15.175:0/2003198260 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6e60056e10 con 0x7f6e700c5780 2023-10-10T18:45:52.610 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 64 2023-10-10T18:45:52.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e54066f30 msgr2=0x7f6e540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e54066f30 0x7f6e540693e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c006fd0 tx=0x7f6e6c008040 comp rx=0 tx=0).stop 2023-10-10T18:45:52.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c5780 msgr2=0x7f6e700c1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c5780 0x7f6e700c1ca0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009290 tx=0x7f6e60016000 comp rx=0 tx=0).stop 2023-10-10T18:45:52.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 shutdown_connections 2023-10-10T18:45:52.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e54066f30 0x7f6e540693e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e70150110 0x7f6e70162c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e700c43b0 0x7f6e700c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 --2- 172.21.15.175:0/2003198260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e700c5780 0x7f6e700c1ca0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 >> 172.21.15.175:0/2003198260 conn(0x7f6e700677f0 msgr2=0x7f6e70067bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:52.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 shutdown_connections 2023-10-10T18:45:52.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.612+0000 7f6e76ab4700 1 -- 172.21.15.175:0/2003198260 wait complete. 2023-10-10T18:45:52.623 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-10T18:45:52.624 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:52.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.741+0000 7f5557961700 1 Processor -- start 2023-10-10T18:45:52.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.741+0000 7f5557961700 1 -- start start 2023-10-10T18:45:52.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5557961700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f555014fbf0 0x7f5550159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5557961700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55500681e0 0x7f55500c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5557961700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5550156800 con 0x7f555014ffc0 2023-10-10T18:45:52.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5557961700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55500c59e0 con 0x7f555014fbf0 2023-10-10T18:45:52.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5557961700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555015a2b0 con 0x7f55500681e0 2023-10-10T18:45:52.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f55556fd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f555014fbf0 0x7f5550159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5554efc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f555014ffc0 0x7f5550063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51550/0 (socket says 172.21.15.175:51550) 2023-10-10T18:45:52.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5554efc700 1 -- 172.21.15.175:0/2885173549 learned_addr learned my addr 172.21.15.175:0/2885173549 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:52.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f55556fd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f555014fbf0 0x7f5550159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37828/0 (socket says 172.21.15.175:37828) 2023-10-10T18:45:52.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5555efe700 1 --2- 172.21.15.175:0/2885173549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55500681e0 0x7f55500c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f55556fd700 1 -- 172.21.15.175:0/2885173549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55500681e0 msgr2=0x7f55500c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:52.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f55556fd700 1 --2- 172.21.15.175:0/2885173549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55500681e0 0x7f55500c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f55556fd700 1 -- 172.21.15.175:0/2885173549 >> v1:172.21.15.175:6789/0 conn(0x7f555014ffc0 legacy=0x7f5550063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f5547fff700 1 -- 172.21.15.175:0/2885173549 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223194431 0 0) 0x7f5550156800 con 0x7f555014ffc0 2023-10-10T18:45:52.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f55556fd700 1 -- 172.21.15.175:0/2885173549 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5550063e30 con 0x7f555014fbf0 2023-10-10T18:45:52.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.742+0000 7f55556fd700 1 --2- 172.21.15.175:0/2885173549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550159d70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f554c0098d0 tx=0x7f554c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b01cbe519dd7953 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5547fff700 1 -- 172.21.15.175:0/2885173549 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c009e70 con 0x7f555014fbf0 2023-10-10T18:45:52.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5547fff700 1 -- 172.21.15.175:0/2885173549 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c005c10 con 0x7f555014fbf0 2023-10-10T18:45:52.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5547fff700 1 -- 172.21.15.175:0/2885173549 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c00b650 con 0x7f555014fbf0 2023-10-10T18:45:52.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5557961700 1 -- 172.21.15.175:0/2885173549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 msgr2=0x7f5550159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5557961700 1 --2- 172.21.15.175:0/2885173549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550159d70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f554c0098d0 tx=0x7f554c009120 comp rx=0 tx=0).stop 2023-10-10T18:45:52.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5557961700 1 -- 172.21.15.175:0/2885173549 shutdown_connections 2023-10-10T18:45:52.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5557961700 1 --2- 172.21.15.175:0/2885173549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55500681e0 0x7f55500c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5557961700 1 --2- 172.21.15.175:0/2885173549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550159d70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5557961700 1 -- 172.21.15.175:0/2885173549 >> 172.21.15.175:0/2885173549 conn(0x7f55500cc760 msgr2=0x7f55500ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:52.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5557961700 1 -- 172.21.15.175:0/2885173549 shutdown_connections 2023-10-10T18:45:52.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.743+0000 7f5557961700 1 -- 172.21.15.175:0/2885173549 wait complete. 2023-10-10T18:45:52.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.744+0000 7f5557961700 1 Processor -- start 2023-10-10T18:45:52.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.744+0000 7f5557961700 1 -- start start 2023-10-10T18:45:52.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5557961700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55500681e0 0x7f555014d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5557961700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5557961700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f555014ffc0 0x7f5550155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5557961700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5550165c30 con 0x7f555014fbf0 2023-10-10T18:45:52.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5557961700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55500c89c0 con 0x7f55500681e0 2023-10-10T18:45:52.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5557961700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5550151ff0 con 0x7f555014ffc0 2023-10-10T18:45:52.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5554efc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5554efc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37832/0 (socket says 172.21.15.175:37832) 2023-10-10T18:45:52.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5554efc700 1 -- 172.21.15.175:0/1747872431 learned_addr learned my addr 172.21.15.175:0/1747872431 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:52.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f5555efe700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f555014ffc0 0x7f5550155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.745+0000 7f55556fd700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55500681e0 0x7f555014d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f5554efc700 1 -- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f555014ffc0 msgr2=0x7f5550155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f5554efc700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f555014ffc0 0x7f5550155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f5554efc700 1 -- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55500681e0 msgr2=0x7f555014d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f5554efc700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55500681e0 0x7f555014d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f5554efc700 1 -- 172.21.15.175:0/1747872431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c0093c0 con 0x7f555014fbf0 2023-10-10T18:45:52.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f5554efc700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550154ae0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f554000c5b0 tx=0x7f5540001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f55457fa700 1 -- 172.21.15.175:0/1747872431 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540005c00 con 0x7f555014fbf0 2023-10-10T18:45:52.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f55457fa700 1 -- 172.21.15.175:0/1747872431 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5540005d60 con 0x7f555014fbf0 2023-10-10T18:45:52.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f55457fa700 1 -- 172.21.15.175:0/1747872431 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540007300 con 0x7f555014fbf0 2023-10-10T18:45:52.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5550165eb0 con 0x7f555014fbf0 2023-10-10T18:45:52.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.746+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5550166470 con 0x7f555014fbf0 2023-10-10T18:45:52.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.747+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5550067570 con 0x7f555014fbf0 2023-10-10T18:45:52.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.747+0000 7f55457fa700 1 -- 172.21.15.175:0/1747872431 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f554001b3b0 con 0x7f555014fbf0 2023-10-10T18:45:52.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.748+0000 7f55457fa700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5534066fb0 0x7f5534069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:52.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.748+0000 7f55457fa700 1 -- 172.21.15.175:0/1747872431 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5540083ba0 con 0x7f555014fbf0 2023-10-10T18:45:52.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.748+0000 7f55556fd700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5534066fb0 0x7f5534069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:52.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.748+0000 7f55556fd700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5534066fb0 0x7f5534069460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f554c009160 tx=0x7f554c016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:52.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.752+0000 7f55457fa700 1 -- 172.21.15.175:0/1747872431 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5540050d70 con 0x7f555014fbf0 2023-10-10T18:45:52.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.976+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5550069780 con 0x7f555014fbf0 2023-10-10T18:45:52.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.976+0000 7f55457fa700 1 -- 172.21.15.175:0/1747872431 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6176 (secure 0 0 0) 0x7f5540017020 con 0x7f555014fbf0 2023-10-10T18:45:52.977 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:52.977 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:45:41.554584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:45:52.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.978+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5534066fb0 msgr2=0x7f5534069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.978+0000 7f5557961700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5534066fb0 0x7f5534069460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f554c009160 tx=0x7f554c016000 comp rx=0 tx=0).stop 2023-10-10T18:45:52.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.978+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 msgr2=0x7f5550154ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:52.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.978+0000 7f5557961700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550154ae0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f554000c5b0 tx=0x7f5540001070 comp rx=0 tx=0).stop 2023-10-10T18:45:52.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.979+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 shutdown_connections 2023-10-10T18:45:52.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.979+0000 7f5557961700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5534066fb0 0x7f5534069460 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.979+0000 7f5557961700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f555014ffc0 0x7f5550155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.979+0000 7f5557961700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55500681e0 0x7f555014d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.979+0000 7f5557961700 1 --2- 172.21.15.175:0/1747872431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f555014fbf0 0x7f5550154ae0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:52.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.979+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 >> 172.21.15.175:0/1747872431 conn(0x7f55500cc760 msgr2=0x7f55501599f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:52.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.979+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 shutdown_connections 2023-10-10T18:45:52.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:52.979+0000 7f5557961700 1 -- 172.21.15.175:0/1747872431 wait complete. 2023-10-10T18:45:52.981 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 17 2023-10-10T18:45:52.990 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-10T18:45:52.991 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-10T18:45:52.991 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T18:45:53.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.105+0000 7fc38f0b3700 1 Processor -- start 2023-10-10T18:45:53.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.105+0000 7fc38f0b3700 1 -- start start 2023-10-10T18:45:53.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc38f0b3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc38806e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc38f0b3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc3880c30c0 0x7fc38814db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc38f0b3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc388153f70 con 0x7fc3880c0270 2023-10-10T18:45:53.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc38f0b3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc388067bd0 con 0x7fc3880c30c0 2023-10-10T18:45:53.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc38f0b3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc388154210 con 0x7fc3880c2cf0 2023-10-10T18:45:53.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc37ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc3880c30c0 0x7fc38814db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc38ce4f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc38806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc38ce4f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc38806e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:35226/0 (socket says 172.21.15.175:35226) 2023-10-10T18:45:53.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc37ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc3880c30c0 0x7fc38814db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:35168/0 (socket says 172.21.15.175:35168) 2023-10-10T18:45:53.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.106+0000 7fc38ce4f700 1 -- 172.21.15.175:0/131265529 learned_addr learned my addr 172.21.15.175:0/131265529 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:53.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.107+0000 7fc38ce4f700 1 -- 172.21.15.175:0/131265529 >> v2:172.21.15.191:3300/0 conn(0x7fc3880c30c0 msgr2=0x7fc38814db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:53.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.107+0000 7fc38ce4f700 1 --2- 172.21.15.175:0/131265529 >> v2:172.21.15.191:3300/0 conn(0x7fc3880c30c0 0x7fc38814db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.107+0000 7fc38ce4f700 1 -- 172.21.15.175:0/131265529 >> v1:172.21.15.191:6789/0 conn(0x7fc3880c0270 legacy=0x7fc388157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.107+0000 7fc38ce4f700 1 -- 172.21.15.175:0/131265529 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3881544f0 con 0x7fc3880c2cf0 2023-10-10T18:45:53.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.107+0000 7fc38ce4f700 1 --2- 172.21.15.175:0/131265529 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc38806e500 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc384009990 tx=0x7fc384009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a96d642573e0de52 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.107+0000 7fc37f7fe700 1 -- 172.21.15.175:0/131265529 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384009e80 con 0x7fc3880c2cf0 2023-10-10T18:45:53.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.107+0000 7fc37f7fe700 1 -- 172.21.15.175:0/131265529 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384005c10 con 0x7fc3880c2cf0 2023-10-10T18:45:53.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.107+0000 7fc37f7fe700 1 -- 172.21.15.175:0/131265529 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38400b6e0 con 0x7fc3880c2cf0 2023-10-10T18:45:53.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.108+0000 7fc38f0b3700 1 -- 172.21.15.175:0/131265529 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 msgr2=0x7fc38806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.108+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/131265529 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc38806e500 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc384009990 tx=0x7fc384009220 comp rx=0 tx=0).stop 2023-10-10T18:45:53.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.108+0000 7fc38f0b3700 1 -- 172.21.15.175:0/131265529 shutdown_connections 2023-10-10T18:45:53.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.108+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/131265529 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc38806e500 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.108+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/131265529 >> v2:172.21.15.191:3300/0 conn(0x7fc3880c30c0 0x7fc38814db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.108+0000 7fc38f0b3700 1 -- 172.21.15.175:0/131265529 >> 172.21.15.175:0/131265529 conn(0x7fc388068900 msgr2=0x7fc388068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:53.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.108+0000 7fc38f0b3700 1 -- 172.21.15.175:0/131265529 shutdown_connections 2023-10-10T18:45:53.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.108+0000 7fc38f0b3700 1 -- 172.21.15.175:0/131265529 wait complete. 2023-10-10T18:45:53.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.109+0000 7fc38f0b3700 1 Processor -- start 2023-10-10T18:45:53.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.109+0000 7fc38f0b3700 1 -- start start 2023-10-10T18:45:53.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.109+0000 7fc38f0b3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3880c0270 0x7fc388165c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.109+0000 7fc38f0b3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc3881500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38f0b3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3880c30c0 0x7fc3881505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38f0b3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3881490e0 con 0x7fc3880c0270 2023-10-10T18:45:53.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38f0b3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38806cd30 con 0x7fc3880c30c0 2023-10-10T18:45:53.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38f0b3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38806c8d0 con 0x7fc3880c2cf0 2023-10-10T18:45:53.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3880c0270 0x7fc388165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3880c0270 0x7fc388165c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37848/0 (socket says 172.21.15.175:37848) 2023-10-10T18:45:53.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 -- 172.21.15.175:0/1143961494 learned_addr learned my addr 172.21.15.175:0/1143961494 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:53.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc37ffff700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc3881500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38d650700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3880c30c0 0x7fc3881505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 -- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 msgr2=0x7fc3881500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc3881500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 -- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3880c30c0 msgr2=0x7fc3881505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3880c30c0 0x7fc3881505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 -- 172.21.15.175:0/1143961494 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc384009480 con 0x7fc3880c0270 2023-10-10T18:45:53.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38d650700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3880c30c0 0x7fc3881505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:45:53.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38ce4f700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3880c0270 0x7fc388165c60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc388154350 tx=0x7fc384016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc37cff9700 1 -- 172.21.15.175:0/1143961494 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384011d90 con 0x7fc3880c0270 2023-10-10T18:45:53.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc37cff9700 1 -- 172.21.15.175:0/1143961494 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38401b430 con 0x7fc3880c0270 2023-10-10T18:45:53.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc37cff9700 1 -- 172.21.15.175:0/1143961494 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38401b7d0 con 0x7fc3880c0270 2023-10-10T18:45:53.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc388169260 con 0x7fc3880c0270 2023-10-10T18:45:53.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.110+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc388169520 con 0x7fc3880c0270 2023-10-10T18:45:53.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.111+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc388067570 con 0x7fc3880c0270 2023-10-10T18:45:53.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.112+0000 7fc37cff9700 1 -- 172.21.15.175:0/1143961494 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc38401bbc0 con 0x7fc3880c0270 2023-10-10T18:45:53.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.112+0000 7fc37cff9700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc374066f30 0x7fc3740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.112+0000 7fc37cff9700 1 -- 172.21.15.175:0/1143961494 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc384086070 con 0x7fc3880c0270 2023-10-10T18:45:53.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.112+0000 7fc37ffff700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc374066f30 0x7fc3740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.113+0000 7fc37ffff700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc374066f30 0x7fc3740693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc38806f500 tx=0x7fc378008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.116+0000 7fc37cff9700 1 -- 172.21.15.175:0/1143961494 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3840532e0 con 0x7fc3880c0270 2023-10-10T18:45:53.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.297+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3880694a0 con 0x7fc374066f30 2023-10-10T18:45:53.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.305+0000 7fc37cff9700 1 -- 172.21.15.175:0/1143961494 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264694 (secure 0 0 0) 0x7fc3880694a0 con 0x7fc374066f30 2023-10-10T18:45:53.308 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:53.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc374066f30 msgr2=0x7fc3740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc374066f30 0x7fc3740693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc38806f500 tx=0x7fc378008040 comp rx=0 tx=0).stop 2023-10-10T18:45:53.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3880c0270 msgr2=0x7fc388165c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3880c0270 0x7fc388165c60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc388154350 tx=0x7fc384016000 comp rx=0 tx=0).stop 2023-10-10T18:45:53.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 shutdown_connections 2023-10-10T18:45:53.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc374066f30 0x7fc3740693e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3880c30c0 0x7fc3881505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3880c0270 0x7fc388165c60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 --2- 172.21.15.175:0/1143961494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3880c2cf0 0x7fc3881500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 >> 172.21.15.175:0/1143961494 conn(0x7fc388068900 msgr2=0x7fc38814e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:53.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 shutdown_connections 2023-10-10T18:45:53.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.308+0000 7fc38f0b3700 1 -- 172.21.15.175:0/1143961494 wait complete. 2023-10-10T18:45:53.312 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T18:45:53.320 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2023-10-10T18:45:53.140876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597994,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2380384,"kb_used_data":2080,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752594336,"statfs":{"total":773094113280,"available":770656600064,"internally_reserved":0,"allocated":2129920,"data_stored":1798492,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1330,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003287"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429926+0000","last_change":"2023-10-10T18:45:11.419184+0000","last_active":"2023-10-10T18:45:11.429926+0000","last_peered":"2023-10-10T18:45:11.429926+0000","last_clean":"2023-10-10T18:45:11.429926+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T18:45:11.429926+0000","last_undegraded":"2023-10-10T18:45:11.429926+0000","last_fullsized":"2023-10-10T18:45:11.429926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420684+0000","last_change":"2023-10-10T18:45:11.414586+0000","last_active":"2023-10-10T18:45:11.420684+0000","last_peered":"2023-10-10T18:45:11.420684+0000","last_clean":"2023-10-10T18:45:11.420684+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T18:45:11.420684+0000","last_undegraded":"2023-10-10T18:45:11.420684+0000","last_fullsized":"2023-10-10T18:45:11.420684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424203+0000","last_change":"2023-10-10T18:45:11.419119+0000","last_active":"2023-10-10T18:45:11.424203+0000","last_peered":"2023-10-10T18:45:11.424203+0000","last_clean":"2023-10-10T18:45:11.424203+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T18:45:11.424203+0000","last_undegraded":"2023-10-10T18:45:11.424203+0000","last_fullsized":"2023-10-10T18:45:11.424203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425796+0000","last_change":"2023-10-10T18:45:11.421351+0000","last_active":"2023-10-10T18:45:11.425796+0000","last_peered":"2023-10-10T18:45:11.425796+0000","last_clean":"2023-10-10T18:45:11.425796+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T18:45:11.425796+0000","last_undegraded":"2023-10-10T18:45:11.425796+0000","last_fullsized":"2023-10-10T18:45:11.425796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425614+0000","last_change":"2023-10-10T18:45:11.421122+0000","last_active":"2023-10-10T18:45:11.425614+0000","last_peered":"2023-10-10T18:45:11.425614+0000","last_clean":"2023-10-10T18:45:11.425614+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T18:45:11.425614+0000","last_undegraded":"2023-10-10T18:45:11.425614+0000","last_fullsized":"2023-10-10T18:45:11.425614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424542+0000","last_change":"2023-10-10T18:45:11.421729+0000","last_active":"2023-10-10T18:45:11.424542+0000","last_peered":"2023-10-10T18:45:11.424542+0000","last_clean":"2023-10-10T18:45:11.424542+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T18:45:11.424542+0000","last_undegraded":"2023-10-10T18:45:11.424542+0000","last_fullsized":"2023-10-10T18:45:11.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426178+0000","last_change":"2023-10-10T18:45:11.419306+0000","last_active":"2023-10-10T18:45:11.426178+0000","last_peered":"2023-10-10T18:45:11.426178+0000","last_clean":"2023-10-10T18:45:11.426178+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T18:45:11.426178+0000","last_undegraded":"2023-10-10T18:45:11.426178+0000","last_fullsized":"2023-10-10T18:45:11.426178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.422940+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429946+0000","last_change":"2023-10-10T18:45:11.422612+0000","last_active":"2023-10-10T18:45:11.429946+0000","last_peered":"2023-10-10T18:45:11.429946+0000","last_clean":"2023-10-10T18:45:11.429946+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T18:45:11.429946+0000","last_undegraded":"2023-10-10T18:45:11.429946+0000","last_fullsized":"2023-10-10T18:45:11.429946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430763+0000","last_change":"2023-10-10T18:45:11.425754+0000","last_active":"2023-10-10T18:45:11.430763+0000","last_peered":"2023-10-10T18:45:11.430763+0000","last_clean":"2023-10-10T18:45:11.430763+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T18:45:11.430763+0000","last_undegraded":"2023-10-10T18:45:11.430763+0000","last_fullsized":"2023-10-10T18:45:11.430763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430664+0000","last_change":"2023-10-10T18:45:11.425891+0000","last_active":"2023-10-10T18:45:11.430664+0000","last_peered":"2023-10-10T18:45:11.430664+0000","last_clean":"2023-10-10T18:45:11.430664+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T18:45:11.430664+0000","last_undegraded":"2023-10-10T18:45:11.430664+0000","last_fullsized":"2023-10-10T18:45:11.430664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431015+0000","last_change":"2023-10-10T18:45:11.424834+0000","last_active":"2023-10-10T18:45:11.431015+0000","last_peered":"2023-10-10T18:45:11.431015+0000","last_clean":"2023-10-10T18:45:11.431015+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T18:45:11.431015+0000","last_undegraded":"2023-10-10T18:45:11.431015+0000","last_fullsized":"2023-10-10T18:45:11.431015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421085+0000","last_change":"2023-10-10T18:45:11.417796+0000","last_active":"2023-10-10T18:45:11.421085+0000","last_peered":"2023-10-10T18:45:11.421085+0000","last_clean":"2023-10-10T18:45:11.421085+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T18:45:11.421085+0000","last_undegraded":"2023-10-10T18:45:11.421085+0000","last_fullsized":"2023-10-10T18:45:11.421085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419270+0000","last_change":"2023-10-10T18:45:11.414690+0000","last_active":"2023-10-10T18:45:11.419270+0000","last_peered":"2023-10-10T18:45:11.419270+0000","last_clean":"2023-10-10T18:45:11.419270+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T18:45:11.419270+0000","last_undegraded":"2023-10-10T18:45:11.419270+0000","last_fullsized":"2023-10-10T18:45:11.419270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434526+0000","last_change":"2023-10-10T18:45:11.429433+0000","last_active":"2023-10-10T18:45:11.434526+0000","last_peered":"2023-10-10T18:45:11.434526+0000","last_clean":"2023-10-10T18:45:11.434526+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T18:45:11.434526+0000","last_undegraded":"2023-10-10T18:45:11.434526+0000","last_fullsized":"2023-10-10T18:45:11.434526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427193+0000","last_change":"2023-10-10T18:45:11.422935+0000","last_active":"2023-10-10T18:45:11.427193+0000","last_peered":"2023-10-10T18:45:11.427193+0000","last_clean":"2023-10-10T18:45:11.427193+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T18:45:11.427193+0000","last_undegraded":"2023-10-10T18:45:11.427193+0000","last_fullsized":"2023-10-10T18:45:11.427193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434131+0000","last_change":"2023-10-10T18:45:11.431202+0000","last_active":"2023-10-10T18:45:11.434131+0000","last_peered":"2023-10-10T18:45:11.434131+0000","last_clean":"2023-10-10T18:45:11.434131+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T18:45:11.434131+0000","last_undegraded":"2023-10-10T18:45:11.434131+0000","last_fullsized":"2023-10-10T18:45:11.434131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427139+0000","last_change":"2023-10-10T18:45:11.424885+0000","last_active":"2023-10-10T18:45:11.427139+0000","last_peered":"2023-10-10T18:45:11.427139+0000","last_clean":"2023-10-10T18:45:11.427139+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T18:45:11.427139+0000","last_undegraded":"2023-10-10T18:45:11.427139+0000","last_fullsized":"2023-10-10T18:45:11.427139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426206+0000","last_change":"2023-10-10T18:45:11.421039+0000","last_active":"2023-10-10T18:45:11.426206+0000","last_peered":"2023-10-10T18:45:11.426206+0000","last_clean":"2023-10-10T18:45:11.426206+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T18:45:11.426206+0000","last_undegraded":"2023-10-10T18:45:11.426206+0000","last_fullsized":"2023-10-10T18:45:11.426206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.554010+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T18:45:40.554010+0000","last_peered":"2023-10-10T18:45:40.554010+0000","last_clean":"2023-10-10T18:45:40.554010+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T18:45:40.554010+0000","last_undegraded":"2023-10-10T18:45:40.554010+0000","last_fullsized":"2023-10-10T18:45:40.554010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421043+0000","last_change":"2023-10-10T18:45:11.417504+0000","last_active":"2023-10-10T18:45:11.421043+0000","last_peered":"2023-10-10T18:45:11.421043+0000","last_clean":"2023-10-10T18:45:11.421043+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T18:45:11.421043+0000","last_undegraded":"2023-10-10T18:45:11.421043+0000","last_fullsized":"2023-10-10T18:45:11.421043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421448+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T18:45:16.421448+0000","last_peered":"2023-10-10T18:45:16.421448+0000","last_clean":"2023-10-10T18:45:16.421448+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T18:45:16.421448+0000","last_undegraded":"2023-10-10T18:45:16.421448+0000","last_fullsized":"2023-10-10T18:45:16.421448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424178+0000","last_change":"2023-10-10T18:45:11.421896+0000","last_active":"2023-10-10T18:45:11.424178+0000","last_peered":"2023-10-10T18:45:11.424178+0000","last_clean":"2023-10-10T18:45:11.424178+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T18:45:11.424178+0000","last_undegraded":"2023-10-10T18:45:11.424178+0000","last_fullsized":"2023-10-10T18:45:11.424178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552890+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T18:45:40.552890+0000","last_peered":"2023-10-10T18:45:40.552890+0000","last_clean":"2023-10-10T18:45:40.552890+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T18:45:40.552890+0000","last_undegraded":"2023-10-10T18:45:40.552890+0000","last_fullsized":"2023-10-10T18:45:40.552890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432466+0000","last_change":"2023-10-10T18:45:11.429680+0000","last_active":"2023-10-10T18:45:11.432466+0000","last_peered":"2023-10-10T18:45:11.432466+0000","last_clean":"2023-10-10T18:45:11.432466+0000","last_became_active":"2023-10-10T18:45:11.429397+0000","last_became_peered":"2023-10-10T18:45:11.429397+0000","last_unstale":"2023-10-10T18:45:11.432466+0000","last_undegraded":"2023-10-10T18:45:11.432466+0000","last_fullsized":"2023-10-10T18:45:11.432466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:19:52.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.551576+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T18:45:40.551576+0000","last_peered":"2023-10-10T18:45:40.551576+0000","last_clean":"2023-10-10T18:45:40.551576+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T18:45:40.551576+0000","last_undegraded":"2023-10-10T18:45:40.551576+0000","last_fullsized":"2023-10-10T18:45:40.551576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432439+0000","last_change":"2023-10-10T18:45:11.429408+0000","last_active":"2023-10-10T18:45:11.432439+0000","last_peered":"2023-10-10T18:45:11.432439+0000","last_clean":"2023-10-10T18:45:11.432439+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T18:45:11.432439+0000","last_undegraded":"2023-10-10T18:45:11.432439+0000","last_fullsized":"2023-10-10T18:45:11.432439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.457325+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T18:45:22.457325+0000","last_peered":"2023-10-10T18:45:22.457325+0000","last_clean":"2023-10-10T18:45:22.457325+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T18:45:22.457325+0000","last_undegraded":"2023-10-10T18:45:22.457325+0000","last_fullsized":"2023-10-10T18:45:22.457325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430579+0000","last_change":"2023-10-10T18:45:11.426051+0000","last_active":"2023-10-10T18:45:11.430579+0000","last_peered":"2023-10-10T18:45:11.430579+0000","last_clean":"2023-10-10T18:45:11.430579+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T18:45:11.430579+0000","last_undegraded":"2023-10-10T18:45:11.430579+0000","last_fullsized":"2023-10-10T18:45:11.430579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531752+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T18:45:34.531752+0000","last_peered":"2023-10-10T18:45:34.531752+0000","last_clean":"2023-10-10T18:45:34.531752+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T18:45:34.531752+0000","last_undegraded":"2023-10-10T18:45:34.531752+0000","last_fullsized":"2023-10-10T18:45:34.531752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427223+0000","last_change":"2023-10-10T18:45:11.425183+0000","last_active":"2023-10-10T18:45:11.427223+0000","last_peered":"2023-10-10T18:45:11.427223+0000","last_clean":"2023-10-10T18:45:11.427223+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T18:45:11.427223+0000","last_undegraded":"2023-10-10T18:45:11.427223+0000","last_fullsized":"2023-10-10T18:45:11.427223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.523628+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T18:45:28.523628+0000","last_peered":"2023-10-10T18:45:28.523628+0000","last_clean":"2023-10-10T18:45:28.523628+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T18:45:28.523628+0000","last_undegraded":"2023-10-10T18:45:28.523628+0000","last_fullsized":"2023-10-10T18:45:28.523628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415249+0000","last_change":"2023-10-10T18:45:11.412641+0000","last_active":"2023-10-10T18:45:11.415249+0000","last_peered":"2023-10-10T18:45:11.415249+0000","last_clean":"2023-10-10T18:45:11.415249+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T18:45:11.415249+0000","last_undegraded":"2023-10-10T18:45:11.415249+0000","last_fullsized":"2023-10-10T18:45:11.415249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531323+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T18:45:34.531323+0000","last_peered":"2023-10-10T18:45:34.531323+0000","last_clean":"2023-10-10T18:45:34.531323+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T18:45:34.531323+0000","last_undegraded":"2023-10-10T18:45:34.531323+0000","last_fullsized":"2023-10-10T18:45:34.531323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420638+0000","last_change":"2023-10-10T18:45:11.413819+0000","last_active":"2023-10-10T18:45:11.420638+0000","last_peered":"2023-10-10T18:45:11.420638+0000","last_clean":"2023-10-10T18:45:11.420638+0000","last_became_active":"2023-10-10T18:45:11.413529+0000","last_became_peered":"2023-10-10T18:45:11.413529+0000","last_unstale":"2023-10-10T18:45:11.420638+0000","last_undegraded":"2023-10-10T18:45:11.420638+0000","last_fullsized":"2023-10-10T18:45:11.420638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.460216+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T18:45:22.460216+0000","last_peered":"2023-10-10T18:45:22.460216+0000","last_clean":"2023-10-10T18:45:22.460216+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T18:45:22.460216+0000","last_undegraded":"2023-10-10T18:45:22.460216+0000","last_fullsized":"2023-10-10T18:45:22.460216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430421+0000","last_change":"2023-10-10T18:45:11.427116+0000","last_active":"2023-10-10T18:45:11.430421+0000","last_peered":"2023-10-10T18:45:11.430421+0000","last_clean":"2023-10-10T18:45:11.430421+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T18:45:11.430421+0000","last_undegraded":"2023-10-10T18:45:11.430421+0000","last_fullsized":"2023-10-10T18:45:11.430421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552523+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T18:45:40.552523+0000","last_peered":"2023-10-10T18:45:40.552523+0000","last_clean":"2023-10-10T18:45:40.552523+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T18:45:40.552523+0000","last_undegraded":"2023-10-10T18:45:40.552523+0000","last_fullsized":"2023-10-10T18:45:40.552523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424158+0000","last_change":"2023-10-10T18:45:11.421794+0000","last_active":"2023-10-10T18:45:11.424158+0000","last_peered":"2023-10-10T18:45:11.424158+0000","last_clean":"2023-10-10T18:45:11.424158+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T18:45:11.424158+0000","last_undegraded":"2023-10-10T18:45:11.424158+0000","last_fullsized":"2023-10-10T18:45:11.424158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.459519+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T18:45:22.459519+0000","last_peered":"2023-10-10T18:45:22.459519+0000","last_clean":"2023-10-10T18:45:22.459519+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T18:45:22.459519+0000","last_undegraded":"2023-10-10T18:45:22.459519+0000","last_fullsized":"2023-10-10T18:45:22.459519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434442+0000","last_change":"2023-10-10T18:45:11.431857+0000","last_active":"2023-10-10T18:45:11.434442+0000","last_peered":"2023-10-10T18:45:11.434442+0000","last_clean":"2023-10-10T18:45:11.434442+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T18:45:11.434442+0000","last_undegraded":"2023-10-10T18:45:11.434442+0000","last_fullsized":"2023-10-10T18:45:11.434442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.553208+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T18:45:40.553208+0000","last_peered":"2023-10-10T18:45:40.553208+0000","last_clean":"2023-10-10T18:45:40.553208+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T18:45:40.553208+0000","last_undegraded":"2023-10-10T18:45:40.553208+0000","last_fullsized":"2023-10-10T18:45:40.553208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426196+0000","last_change":"2023-10-10T18:45:11.419774+0000","last_active":"2023-10-10T18:45:11.426196+0000","last_peered":"2023-10-10T18:45:11.426196+0000","last_clean":"2023-10-10T18:45:11.426196+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T18:45:11.426196+0000","last_undegraded":"2023-10-10T18:45:11.426196+0000","last_fullsized":"2023-10-10T18:45:11.426196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.551374+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T18:45:40.551374+0000","last_peered":"2023-10-10T18:45:40.551374+0000","last_clean":"2023-10-10T18:45:40.551374+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T18:45:40.551374+0000","last_undegraded":"2023-10-10T18:45:40.551374+0000","last_fullsized":"2023-10-10T18:45:40.551374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424245+0000","last_change":"2023-10-10T18:45:11.420517+0000","last_active":"2023-10-10T18:45:11.424245+0000","last_peered":"2023-10-10T18:45:11.424245+0000","last_clean":"2023-10-10T18:45:11.424245+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T18:45:11.424245+0000","last_undegraded":"2023-10-10T18:45:11.424245+0000","last_fullsized":"2023-10-10T18:45:11.424245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.527081+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T18:45:34.527081+0000","last_peered":"2023-10-10T18:45:34.527081+0000","last_clean":"2023-10-10T18:45:34.527081+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T18:45:34.527081+0000","last_undegraded":"2023-10-10T18:45:34.527081+0000","last_fullsized":"2023-10-10T18:45:34.527081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431834+0000","last_change":"2023-10-10T18:45:11.427109+0000","last_active":"2023-10-10T18:45:11.431834+0000","last_peered":"2023-10-10T18:45:11.431834+0000","last_clean":"2023-10-10T18:45:11.431834+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T18:45:11.431834+0000","last_undegraded":"2023-10-10T18:45:11.431834+0000","last_fullsized":"2023-10-10T18:45:11.431834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529442+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T18:45:34.529442+0000","last_peered":"2023-10-10T18:45:34.529442+0000","last_clean":"2023-10-10T18:45:34.529442+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T18:45:34.529442+0000","last_undegraded":"2023-10-10T18:45:34.529442+0000","last_fullsized":"2023-10-10T18:45:34.529442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430660+0000","last_change":"2023-10-10T18:45:11.416459+0000","last_active":"2023-10-10T18:45:11.430660+0000","last_peered":"2023-10-10T18:45:11.430660+0000","last_clean":"2023-10-10T18:45:11.430660+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T18:45:11.430660+0000","last_undegraded":"2023-10-10T18:45:11.430660+0000","last_fullsized":"2023-10-10T18:45:11.430660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.527437+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T18:45:34.527437+0000","last_peered":"2023-10-10T18:45:34.527437+0000","last_clean":"2023-10-10T18:45:34.527437+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T18:45:34.527437+0000","last_undegraded":"2023-10-10T18:45:34.527437+0000","last_fullsized":"2023-10-10T18:45:34.527437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432328+0000","last_change":"2023-10-10T18:45:11.429727+0000","last_active":"2023-10-10T18:45:11.432328+0000","last_peered":"2023-10-10T18:45:11.432328+0000","last_clean":"2023-10-10T18:45:11.432328+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T18:45:11.432328+0000","last_undegraded":"2023-10-10T18:45:11.432328+0000","last_fullsized":"2023-10-10T18:45:11.432328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421569+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T18:45:16.421569+0000","last_peered":"2023-10-10T18:45:16.421569+0000","last_clean":"2023-10-10T18:45:16.421569+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T18:45:16.421569+0000","last_undegraded":"2023-10-10T18:45:16.421569+0000","last_fullsized":"2023-10-10T18:45:16.421569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431858+0000","last_change":"2023-10-10T18:45:11.427926+0000","last_active":"2023-10-10T18:45:11.431858+0000","last_peered":"2023-10-10T18:45:11.431858+0000","last_clean":"2023-10-10T18:45:11.431858+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T18:45:11.431858+0000","last_undegraded":"2023-10-10T18:45:11.431858+0000","last_fullsized":"2023-10-10T18:45:11.431858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.009459+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T18:45:13.009459+0000","last_peered":"2023-10-10T18:45:13.009459+0000","last_clean":"2023-10-10T18:45:13.009459+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T18:45:13.009459+0000","last_undegraded":"2023-10-10T18:45:13.009459+0000","last_fullsized":"2023-10-10T18:45:13.009459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427955+0000","last_change":"2023-10-10T18:45:11.423279+0000","last_active":"2023-10-10T18:45:11.427955+0000","last_peered":"2023-10-10T18:45:11.427955+0000","last_clean":"2023-10-10T18:45:11.427955+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T18:45:11.427955+0000","last_undegraded":"2023-10-10T18:45:11.427955+0000","last_fullsized":"2023-10-10T18:45:11.427955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.554631+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T18:45:40.554631+0000","last_peered":"2023-10-10T18:45:40.554631+0000","last_clean":"2023-10-10T18:45:40.554631+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T18:45:40.554631+0000","last_undegraded":"2023-10-10T18:45:40.554631+0000","last_fullsized":"2023-10-10T18:45:40.554631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430728+0000","last_change":"2023-10-10T18:45:11.425339+0000","last_active":"2023-10-10T18:45:11.430728+0000","last_peered":"2023-10-10T18:45:11.430728+0000","last_clean":"2023-10-10T18:45:11.430728+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T18:45:11.430728+0000","last_undegraded":"2023-10-10T18:45:11.430728+0000","last_fullsized":"2023-10-10T18:45:11.430728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421278+0000","last_change":"2023-10-10T18:45:11.418937+0000","last_active":"2023-10-10T18:45:11.421278+0000","last_peered":"2023-10-10T18:45:11.421278+0000","last_clean":"2023-10-10T18:45:11.421278+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T18:45:11.421278+0000","last_undegraded":"2023-10-10T18:45:11.421278+0000","last_fullsized":"2023-10-10T18:45:11.421278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.524806+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T18:45:28.524806+0000","last_peered":"2023-10-10T18:45:28.524806+0000","last_clean":"2023-10-10T18:45:28.524806+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T18:45:28.524806+0000","last_undegraded":"2023-10-10T18:45:28.524806+0000","last_fullsized":"2023-10-10T18:45:28.524806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424136+0000","last_change":"2023-10-10T18:45:11.422452+0000","last_active":"2023-10-10T18:45:11.424136+0000","last_peered":"2023-10-10T18:45:11.424136+0000","last_clean":"2023-10-10T18:45:11.424136+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T18:45:11.424136+0000","last_undegraded":"2023-10-10T18:45:11.424136+0000","last_fullsized":"2023-10-10T18:45:11.424136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529502+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T18:45:34.529502+0000","last_peered":"2023-10-10T18:45:34.529502+0000","last_clean":"2023-10-10T18:45:34.529502+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T18:45:34.529502+0000","last_undegraded":"2023-10-10T18:45:34.529502+0000","last_fullsized":"2023-10-10T18:45:34.529502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405467+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T18:45:11.405467+0000","last_peered":"2023-10-10T18:45:11.405467+0000","last_clean":"2023-10-10T18:45:11.405467+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T18:45:11.405467+0000","last_undegraded":"2023-10-10T18:45:11.405467+0000","last_fullsized":"2023-10-10T18:45:11.405467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434404+0000","last_change":"2023-10-10T18:45:11.430233+0000","last_active":"2023-10-10T18:45:11.434404+0000","last_peered":"2023-10-10T18:45:11.434404+0000","last_clean":"2023-10-10T18:45:11.434404+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T18:45:11.434404+0000","last_undegraded":"2023-10-10T18:45:11.434404+0000","last_fullsized":"2023-10-10T18:45:11.434404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.520084+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T18:45:28.520084+0000","last_peered":"2023-10-10T18:45:28.520084+0000","last_clean":"2023-10-10T18:45:28.520084+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T18:45:28.520084+0000","last_undegraded":"2023-10-10T18:45:28.520084+0000","last_fullsized":"2023-10-10T18:45:28.520084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531458+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T18:45:34.531458+0000","last_peered":"2023-10-10T18:45:34.531458+0000","last_clean":"2023-10-10T18:45:34.531458+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T18:45:34.531458+0000","last_undegraded":"2023-10-10T18:45:34.531458+0000","last_fullsized":"2023-10-10T18:45:34.531458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421286+0000","last_change":"2023-10-10T18:45:11.418973+0000","last_active":"2023-10-10T18:45:11.421286+0000","last_peered":"2023-10-10T18:45:11.421286+0000","last_clean":"2023-10-10T18:45:11.421286+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T18:45:11.421286+0000","last_undegraded":"2023-10-10T18:45:11.421286+0000","last_fullsized":"2023-10-10T18:45:11.421286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531687+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T18:45:34.531687+0000","last_peered":"2023-10-10T18:45:34.531687+0000","last_clean":"2023-10-10T18:45:34.531687+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T18:45:34.531687+0000","last_undegraded":"2023-10-10T18:45:34.531687+0000","last_fullsized":"2023-10-10T18:45:34.531687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423727+0000","last_change":"2023-10-10T18:45:11.419846+0000","last_active":"2023-10-10T18:45:11.423727+0000","last_peered":"2023-10-10T18:45:11.423727+0000","last_clean":"2023-10-10T18:45:11.423727+0000","last_became_active":"2023-10-10T18:45:11.419579+0000","last_became_peered":"2023-10-10T18:45:11.419579+0000","last_unstale":"2023-10-10T18:45:11.423727+0000","last_undegraded":"2023-10-10T18:45:11.423727+0000","last_fullsized":"2023-10-10T18:45:11.423727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:58:49.831558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552259+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T18:45:40.552259+0000","last_peered":"2023-10-10T18:45:40.552259+0000","last_clean":"2023-10-10T18:45:40.552259+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T18:45:40.552259+0000","last_undegraded":"2023-10-10T18:45:40.552259+0000","last_fullsized":"2023-10-10T18:45:40.552259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423663+0000","last_change":"2023-10-10T18:45:11.416217+0000","last_active":"2023-10-10T18:45:11.423663+0000","last_peered":"2023-10-10T18:45:11.423663+0000","last_clean":"2023-10-10T18:45:11.423663+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T18:45:11.423663+0000","last_undegraded":"2023-10-10T18:45:11.423663+0000","last_fullsized":"2023-10-10T18:45:11.423663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.659888+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T18:45:22.659888+0000","last_peered":"2023-10-10T18:45:22.659888+0000","last_clean":"2023-10-10T18:45:22.659888+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T18:45:22.659888+0000","last_undegraded":"2023-10-10T18:45:22.659888+0000","last_fullsized":"2023-10-10T18:45:22.659888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431602+0000","last_change":"2023-10-10T18:45:11.427826+0000","last_active":"2023-10-10T18:45:11.431602+0000","last_peered":"2023-10-10T18:45:11.431602+0000","last_clean":"2023-10-10T18:45:11.431602+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T18:45:11.431602+0000","last_undegraded":"2023-10-10T18:45:11.431602+0000","last_fullsized":"2023-10-10T18:45:11.431602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552130+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T18:45:40.552130+0000","last_peered":"2023-10-10T18:45:40.552130+0000","last_clean":"2023-10-10T18:45:40.552130+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T18:45:40.552130+0000","last_undegraded":"2023-10-10T18:45:40.552130+0000","last_fullsized":"2023-10-10T18:45:40.552130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415388+0000","last_change":"2023-10-10T18:45:11.413975+0000","last_active":"2023-10-10T18:45:11.415388+0000","last_peered":"2023-10-10T18:45:11.415388+0000","last_clean":"2023-10-10T18:45:11.415388+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T18:45:11.415388+0000","last_undegraded":"2023-10-10T18:45:11.415388+0000","last_fullsized":"2023-10-10T18:45:11.415388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529028+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T18:45:34.529028+0000","last_peered":"2023-10-10T18:45:34.529028+0000","last_clean":"2023-10-10T18:45:34.529028+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T18:45:34.529028+0000","last_undegraded":"2023-10-10T18:45:34.529028+0000","last_fullsized":"2023-10-10T18:45:34.529028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415058+0000","last_change":"2023-10-10T18:45:11.410768+0000","last_active":"2023-10-10T18:45:11.415058+0000","last_peered":"2023-10-10T18:45:11.415058+0000","last_clean":"2023-10-10T18:45:11.415058+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T18:45:11.415058+0000","last_undegraded":"2023-10-10T18:45:11.415058+0000","last_fullsized":"2023-10-10T18:45:11.415058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.660373+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T18:45:22.660373+0000","last_peered":"2023-10-10T18:45:22.660373+0000","last_clean":"2023-10-10T18:45:22.660373+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T18:45:22.660373+0000","last_undegraded":"2023-10-10T18:45:22.660373+0000","last_fullsized":"2023-10-10T18:45:22.660373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430290+0000","last_change":"2023-10-10T18:45:11.425716+0000","last_active":"2023-10-10T18:45:11.430290+0000","last_peered":"2023-10-10T18:45:11.430290+0000","last_clean":"2023-10-10T18:45:11.430290+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T18:45:11.430290+0000","last_undegraded":"2023-10-10T18:45:11.430290+0000","last_fullsized":"2023-10-10T18:45:11.430290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.530979+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T18:45:34.530979+0000","last_peered":"2023-10-10T18:45:34.530979+0000","last_clean":"2023-10-10T18:45:34.530979+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T18:45:34.530979+0000","last_undegraded":"2023-10-10T18:45:34.530979+0000","last_fullsized":"2023-10-10T18:45:34.530979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429787+0000","last_change":"2023-10-10T18:45:11.423747+0000","last_active":"2023-10-10T18:45:11.429787+0000","last_peered":"2023-10-10T18:45:11.429787+0000","last_clean":"2023-10-10T18:45:11.429787+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T18:45:11.429787+0000","last_undegraded":"2023-10-10T18:45:11.429787+0000","last_fullsized":"2023-10-10T18:45:11.429787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.532321+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T18:45:34.532321+0000","last_peered":"2023-10-10T18:45:34.532321+0000","last_clean":"2023-10-10T18:45:34.532321+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T18:45:34.532321+0000","last_undegraded":"2023-10-10T18:45:34.532321+0000","last_fullsized":"2023-10-10T18:45:34.532321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425816+0000","last_change":"2023-10-10T18:45:11.421872+0000","last_active":"2023-10-10T18:45:11.425816+0000","last_peered":"2023-10-10T18:45:11.425816+0000","last_clean":"2023-10-10T18:45:11.425816+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T18:45:11.425816+0000","last_undegraded":"2023-10-10T18:45:11.425816+0000","last_fullsized":"2023-10-10T18:45:11.425816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.550580+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T18:45:40.550580+0000","last_peered":"2023-10-10T18:45:40.550580+0000","last_clean":"2023-10-10T18:45:40.550580+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T18:45:40.550580+0000","last_undegraded":"2023-10-10T18:45:40.550580+0000","last_fullsized":"2023-10-10T18:45:40.550580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429963+0000","last_change":"2023-10-10T18:45:11.425474+0000","last_active":"2023-10-10T18:45:11.429963+0000","last_peered":"2023-10-10T18:45:11.429963+0000","last_clean":"2023-10-10T18:45:11.429963+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T18:45:11.429963+0000","last_undegraded":"2023-10-10T18:45:11.429963+0000","last_fullsized":"2023-10-10T18:45:11.429963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430681+0000","last_change":"2023-10-10T18:45:11.416850+0000","last_active":"2023-10-10T18:45:11.430681+0000","last_peered":"2023-10-10T18:45:11.430681+0000","last_clean":"2023-10-10T18:45:11.430681+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T18:45:11.430681+0000","last_undegraded":"2023-10-10T18:45:11.430681+0000","last_fullsized":"2023-10-10T18:45:11.430681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427108+0000","last_change":"2023-10-10T18:45:11.424440+0000","last_active":"2023-10-10T18:45:11.427108+0000","last_peered":"2023-10-10T18:45:11.427108+0000","last_clean":"2023-10-10T18:45:11.427108+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T18:45:11.427108+0000","last_undegraded":"2023-10-10T18:45:11.427108+0000","last_fullsized":"2023-10-10T18:45:11.427108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.421524+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.420874+0000","last_became_peered":"2023-10-10T18:45:11.420874+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:56:36.851801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419177+0000","last_change":"2023-10-10T18:45:11.412399+0000","last_active":"2023-10-10T18:45:11.419177+0000","last_peered":"2023-10-10T18:45:11.419177+0000","last_clean":"2023-10-10T18:45:11.419177+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T18:45:11.419177+0000","last_undegraded":"2023-10-10T18:45:11.419177+0000","last_fullsized":"2023-10-10T18:45:11.419177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432996+0000","last_change":"2023-10-10T18:45:11.429507+0000","last_active":"2023-10-10T18:45:11.432996+0000","last_peered":"2023-10-10T18:45:11.432996+0000","last_clean":"2023-10-10T18:45:11.432996+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T18:45:11.432996+0000","last_undegraded":"2023-10-10T18:45:11.432996+0000","last_fullsized":"2023-10-10T18:45:11.432996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.418310+0000","last_change":"2023-10-10T18:45:11.413271+0000","last_active":"2023-10-10T18:45:11.418310+0000","last_peered":"2023-10-10T18:45:11.418310+0000","last_clean":"2023-10-10T18:45:11.418310+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T18:45:11.418310+0000","last_undegraded":"2023-10-10T18:45:11.418310+0000","last_fullsized":"2023-10-10T18:45:11.418310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430781+0000","last_change":"2023-10-10T18:45:11.427815+0000","last_active":"2023-10-10T18:45:11.430781+0000","last_peered":"2023-10-10T18:45:11.430781+0000","last_clean":"2023-10-10T18:45:11.430781+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T18:45:11.430781+0000","last_undegraded":"2023-10-10T18:45:11.430781+0000","last_fullsized":"2023-10-10T18:45:11.430781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426972+0000","last_change":"2023-10-10T18:45:11.425458+0000","last_active":"2023-10-10T18:45:11.426972+0000","last_peered":"2023-10-10T18:45:11.426972+0000","last_clean":"2023-10-10T18:45:11.426972+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T18:45:11.426972+0000","last_undegraded":"2023-10-10T18:45:11.426972+0000","last_fullsized":"2023-10-10T18:45:11.426972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424565+0000","last_change":"2023-10-10T18:45:11.422003+0000","last_active":"2023-10-10T18:45:11.424565+0000","last_peered":"2023-10-10T18:45:11.424565+0000","last_clean":"2023-10-10T18:45:11.424565+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T18:45:11.424565+0000","last_undegraded":"2023-10-10T18:45:11.424565+0000","last_fullsized":"2023-10-10T18:45:11.424565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432978+0000","last_change":"2023-10-10T18:45:11.428907+0000","last_active":"2023-10-10T18:45:11.432978+0000","last_peered":"2023-10-10T18:45:11.432978+0000","last_clean":"2023-10-10T18:45:11.432978+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T18:45:11.432978+0000","last_undegraded":"2023-10-10T18:45:11.432978+0000","last_fullsized":"2023-10-10T18:45:11.432978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430538+0000","last_change":"2023-10-10T18:45:11.427241+0000","last_active":"2023-10-10T18:45:11.430538+0000","last_peered":"2023-10-10T18:45:11.430538+0000","last_clean":"2023-10-10T18:45:11.430538+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T18:45:11.430538+0000","last_undegraded":"2023-10-10T18:45:11.430538+0000","last_fullsized":"2023-10-10T18:45:11.430538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427471+0000","last_change":"2023-10-10T18:45:11.426813+0000","last_active":"2023-10-10T18:45:11.427471+0000","last_peered":"2023-10-10T18:45:11.427471+0000","last_clean":"2023-10-10T18:45:11.427471+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T18:45:11.427471+0000","last_undegraded":"2023-10-10T18:45:11.427471+0000","last_fullsized":"2023-10-10T18:45:11.427471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432098+0000","last_change":"2023-10-10T18:45:11.429582+0000","last_active":"2023-10-10T18:45:11.432098+0000","last_peered":"2023-10-10T18:45:11.432098+0000","last_clean":"2023-10-10T18:45:11.432098+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T18:45:11.432098+0000","last_undegraded":"2023-10-10T18:45:11.432098+0000","last_fullsized":"2023-10-10T18:45:11.432098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7626,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":15252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074240,"statfs":{"total":96636764160,"available":96332021760,"internally_reserved":0,"allocated":319488,"data_stored":667677,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":245760,"data_stored":77591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":245760,"data_stored":76769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":666527,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":249856,"data_stored":76699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":249856,"data_stored":77625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297524,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074316,"statfs":{"total":96636764160,"available":96332099584,"internally_reserved":0,"allocated":241664,"data_stored":76567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":79037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T18:45:53.326 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T18:45:53.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.474+0000 7f1f76cac700 1 Processor -- start 2023-10-10T18:45:53.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.474+0000 7f1f76cac700 1 -- start start 2023-10-10T18:45:53.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f76cac700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70068cf0 0x7f1f70067b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f76cac700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1f7014fd20 0x7f1f701501c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f76cac700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f70068b30 con 0x7f1f70159dd0 2023-10-10T18:45:53.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f76cac700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f700c1c70 con 0x7f1f7014fd20 2023-10-10T18:45:53.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f76cac700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700cd2b0 con 0x7f1f70068cf0 2023-10-10T18:45:53.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f764ab700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1f7014fd20 0x7f1f701501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f754a9700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1f70159dd0 0x7f1f700c2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51564/0 (socket says 172.21.15.175:51564) 2023-10-10T18:45:53.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f754a9700 1 -- 172.21.15.175:0/2139330853 learned_addr learned my addr 172.21.15.175:0/2139330853 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:53.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f75caa700 1 --2- 172.21.15.175:0/2139330853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70068cf0 0x7f1f70067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f764ab700 1 -- 172.21.15.175:0/2139330853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70068cf0 msgr2=0x7f1f70067b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:53.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f764ab700 1 --2- 172.21.15.175:0/2139330853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70068cf0 0x7f1f70067b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f764ab700 1 -- 172.21.15.175:0/2139330853 >> v1:172.21.15.175:6789/0 conn(0x7f1f70159dd0 legacy=0x7f1f700c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f74ca8700 1 -- 172.21.15.175:0/2139330853 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025916700 0 0) 0x7f1f70068b30 con 0x7f1f70159dd0 2023-10-10T18:45:53.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f764ab700 1 -- 172.21.15.175:0/2139330853 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f700cd590 con 0x7f1f7014fd20 2023-10-10T18:45:53.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.475+0000 7f1f764ab700 1 --2- 172.21.15.175:0/2139330853 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f7014fd20 0x7f1f701501c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1f680098e0 tx=0x7f1f680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a93990dd61e6d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f74ca8700 1 -- 172.21.15.175:0/2139330853 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68009e80 con 0x7f1f7014fd20 2023-10-10T18:45:53.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f74ca8700 1 -- 172.21.15.175:0/2139330853 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6800b8f0 con 0x7f1f7014fd20 2023-10-10T18:45:53.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f74ca8700 1 -- 172.21.15.175:0/2139330853 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800bc70 con 0x7f1f7014fd20 2023-10-10T18:45:53.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f76cac700 1 -- 172.21.15.175:0/2139330853 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f7014fd20 msgr2=0x7f1f701501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f76cac700 1 --2- 172.21.15.175:0/2139330853 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f7014fd20 0x7f1f701501c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1f680098e0 tx=0x7f1f680091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f76cac700 1 -- 172.21.15.175:0/2139330853 shutdown_connections 2023-10-10T18:45:53.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f76cac700 1 --2- 172.21.15.175:0/2139330853 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f7014fd20 0x7f1f701501c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f76cac700 1 --2- 172.21.15.175:0/2139330853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70068cf0 0x7f1f70067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f76cac700 1 -- 172.21.15.175:0/2139330853 >> 172.21.15.175:0/2139330853 conn(0x7f1f70063cc0 msgr2=0x7f1f700cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:53.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f76cac700 1 -- 172.21.15.175:0/2139330853 shutdown_connections 2023-10-10T18:45:53.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.476+0000 7f1f76cac700 1 -- 172.21.15.175:0/2139330853 wait complete. 2023-10-10T18:45:53.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.477+0000 7f1f76cac700 1 Processor -- start 2023-10-10T18:45:53.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.477+0000 7f1f76cac700 1 -- start start 2023-10-10T18:45:53.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f76cac700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f70068cf0 0x7f1f7015cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f76cac700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f7014fd20 0x7f1f700c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f76cac700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70159dd0 0x7f1f700c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f76cac700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700c6190 con 0x7f1f70068cf0 2023-10-10T18:45:53.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f76cac700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700c6960 con 0x7f1f7014fd20 2023-10-10T18:45:53.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f76cac700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70166490 con 0x7f1f70159dd0 2023-10-10T18:45:53.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f75caa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f70068cf0 0x7f1f7015cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f75caa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f70068cf0 0x7f1f7015cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37864/0 (socket says 172.21.15.175:37864) 2023-10-10T18:45:53.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f75caa700 1 -- 172.21.15.175:0/1294910917 learned_addr learned my addr 172.21.15.175:0/1294910917 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:53.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f754a9700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f7014fd20 0x7f1f700c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f764ab700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70159dd0 0x7f1f700c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f75caa700 1 -- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70159dd0 msgr2=0x7f1f700c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f75caa700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70159dd0 0x7f1f700c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f75caa700 1 -- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f7014fd20 msgr2=0x7f1f700c3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f75caa700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f7014fd20 0x7f1f700c3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.478+0000 7f1f75caa700 1 -- 172.21.15.175:0/1294910917 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f68009420 con 0x7f1f70068cf0 2023-10-10T18:45:53.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.479+0000 7f1f75caa700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f70068cf0 0x7f1f7015cce0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000e190 tx=0x7f1f60009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.479+0000 7f1f6dffb700 1 -- 172.21.15.175:0/1294910917 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6000ecd0 con 0x7f1f70068cf0 2023-10-10T18:45:53.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.479+0000 7f1f6dffb700 1 -- 172.21.15.175:0/1294910917 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6000ee30 con 0x7f1f70068cf0 2023-10-10T18:45:53.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.479+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f701537b0 con 0x7f1f70068cf0 2023-10-10T18:45:53.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.479+0000 7f1f6dffb700 1 -- 172.21.15.175:0/1294910917 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60014650 con 0x7f1f70068cf0 2023-10-10T18:45:53.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.479+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f70166710 con 0x7f1f70068cf0 2023-10-10T18:45:53.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.480+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f70067570 con 0x7f1f70068cf0 2023-10-10T18:45:53.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.480+0000 7f1f6dffb700 1 -- 172.21.15.175:0/1294910917 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1f600149c0 con 0x7f1f70068cf0 2023-10-10T18:45:53.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.481+0000 7f1f6dffb700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f5c066f30 0x7f1f5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.481+0000 7f1f6dffb700 1 -- 172.21.15.175:0/1294910917 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1f60084b20 con 0x7f1f70068cf0 2023-10-10T18:45:53.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.481+0000 7f1f754a9700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f5c066f30 0x7f1f5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.481+0000 7f1f754a9700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f5c066f30 0x7f1f5c0693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1f64007160 tx=0x7f1f64007f50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.485+0000 7f1f6dffb700 1 -- 172.21.15.175:0/1294910917 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f60051d90 con 0x7f1f70068cf0 2023-10-10T18:45:53.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.666+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f70065140 con 0x7f1f5c066f30 2023-10-10T18:45:53.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.673+0000 7f1f6dffb700 1 -- 172.21.15.175:0/1294910917 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264694 (secure 0 0 0) 0x7f1f70065140 con 0x7f1f5c066f30 2023-10-10T18:45:53.677 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:53.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f5c066f30 msgr2=0x7f1f5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f5c066f30 0x7f1f5c0693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1f64007160 tx=0x7f1f64007f50 comp rx=0 tx=0).stop 2023-10-10T18:45:53.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f70068cf0 msgr2=0x7f1f7015cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f70068cf0 0x7f1f7015cce0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000e190 tx=0x7f1f60009920 comp rx=0 tx=0).stop 2023-10-10T18:45:53.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 shutdown_connections 2023-10-10T18:45:53.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f5c066f30 0x7f1f5c0693e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f70159dd0 0x7f1f700c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f70068cf0 0x7f1f7015cce0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 --2- 172.21.15.175:0/1294910917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f7014fd20 0x7f1f700c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 >> 172.21.15.175:0/1294910917 conn(0x7f1f70063cc0 msgr2=0x7f1f700b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:53.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 shutdown_connections 2023-10-10T18:45:53.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.677+0000 7f1f76cac700 1 -- 172.21.15.175:0/1294910917 wait complete. 2023-10-10T18:45:53.682 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T18:45:53.689 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2023-10-10T18:45:53.140876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597994,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2380384,"kb_used_data":2080,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752594336,"statfs":{"total":773094113280,"available":770656600064,"internally_reserved":0,"allocated":2129920,"data_stored":1798492,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1330,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003287"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429926+0000","last_change":"2023-10-10T18:45:11.419184+0000","last_active":"2023-10-10T18:45:11.429926+0000","last_peered":"2023-10-10T18:45:11.429926+0000","last_clean":"2023-10-10T18:45:11.429926+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T18:45:11.429926+0000","last_undegraded":"2023-10-10T18:45:11.429926+0000","last_fullsized":"2023-10-10T18:45:11.429926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420684+0000","last_change":"2023-10-10T18:45:11.414586+0000","last_active":"2023-10-10T18:45:11.420684+0000","last_peered":"2023-10-10T18:45:11.420684+0000","last_clean":"2023-10-10T18:45:11.420684+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T18:45:11.420684+0000","last_undegraded":"2023-10-10T18:45:11.420684+0000","last_fullsized":"2023-10-10T18:45:11.420684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424203+0000","last_change":"2023-10-10T18:45:11.419119+0000","last_active":"2023-10-10T18:45:11.424203+0000","last_peered":"2023-10-10T18:45:11.424203+0000","last_clean":"2023-10-10T18:45:11.424203+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T18:45:11.424203+0000","last_undegraded":"2023-10-10T18:45:11.424203+0000","last_fullsized":"2023-10-10T18:45:11.424203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425796+0000","last_change":"2023-10-10T18:45:11.421351+0000","last_active":"2023-10-10T18:45:11.425796+0000","last_peered":"2023-10-10T18:45:11.425796+0000","last_clean":"2023-10-10T18:45:11.425796+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T18:45:11.425796+0000","last_undegraded":"2023-10-10T18:45:11.425796+0000","last_fullsized":"2023-10-10T18:45:11.425796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425614+0000","last_change":"2023-10-10T18:45:11.421122+0000","last_active":"2023-10-10T18:45:11.425614+0000","last_peered":"2023-10-10T18:45:11.425614+0000","last_clean":"2023-10-10T18:45:11.425614+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T18:45:11.425614+0000","last_undegraded":"2023-10-10T18:45:11.425614+0000","last_fullsized":"2023-10-10T18:45:11.425614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424542+0000","last_change":"2023-10-10T18:45:11.421729+0000","last_active":"2023-10-10T18:45:11.424542+0000","last_peered":"2023-10-10T18:45:11.424542+0000","last_clean":"2023-10-10T18:45:11.424542+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T18:45:11.424542+0000","last_undegraded":"2023-10-10T18:45:11.424542+0000","last_fullsized":"2023-10-10T18:45:11.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426178+0000","last_change":"2023-10-10T18:45:11.419306+0000","last_active":"2023-10-10T18:45:11.426178+0000","last_peered":"2023-10-10T18:45:11.426178+0000","last_clean":"2023-10-10T18:45:11.426178+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T18:45:11.426178+0000","last_undegraded":"2023-10-10T18:45:11.426178+0000","last_fullsized":"2023-10-10T18:45:11.426178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.422940+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429946+0000","last_change":"2023-10-10T18:45:11.422612+0000","last_active":"2023-10-10T18:45:11.429946+0000","last_peered":"2023-10-10T18:45:11.429946+0000","last_clean":"2023-10-10T18:45:11.429946+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T18:45:11.429946+0000","last_undegraded":"2023-10-10T18:45:11.429946+0000","last_fullsized":"2023-10-10T18:45:11.429946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430763+0000","last_change":"2023-10-10T18:45:11.425754+0000","last_active":"2023-10-10T18:45:11.430763+0000","last_peered":"2023-10-10T18:45:11.430763+0000","last_clean":"2023-10-10T18:45:11.430763+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T18:45:11.430763+0000","last_undegraded":"2023-10-10T18:45:11.430763+0000","last_fullsized":"2023-10-10T18:45:11.430763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430664+0000","last_change":"2023-10-10T18:45:11.425891+0000","last_active":"2023-10-10T18:45:11.430664+0000","last_peered":"2023-10-10T18:45:11.430664+0000","last_clean":"2023-10-10T18:45:11.430664+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T18:45:11.430664+0000","last_undegraded":"2023-10-10T18:45:11.430664+0000","last_fullsized":"2023-10-10T18:45:11.430664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431015+0000","last_change":"2023-10-10T18:45:11.424834+0000","last_active":"2023-10-10T18:45:11.431015+0000","last_peered":"2023-10-10T18:45:11.431015+0000","last_clean":"2023-10-10T18:45:11.431015+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T18:45:11.431015+0000","last_undegraded":"2023-10-10T18:45:11.431015+0000","last_fullsized":"2023-10-10T18:45:11.431015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421085+0000","last_change":"2023-10-10T18:45:11.417796+0000","last_active":"2023-10-10T18:45:11.421085+0000","last_peered":"2023-10-10T18:45:11.421085+0000","last_clean":"2023-10-10T18:45:11.421085+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T18:45:11.421085+0000","last_undegraded":"2023-10-10T18:45:11.421085+0000","last_fullsized":"2023-10-10T18:45:11.421085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419270+0000","last_change":"2023-10-10T18:45:11.414690+0000","last_active":"2023-10-10T18:45:11.419270+0000","last_peered":"2023-10-10T18:45:11.419270+0000","last_clean":"2023-10-10T18:45:11.419270+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T18:45:11.419270+0000","last_undegraded":"2023-10-10T18:45:11.419270+0000","last_fullsized":"2023-10-10T18:45:11.419270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434526+0000","last_change":"2023-10-10T18:45:11.429433+0000","last_active":"2023-10-10T18:45:11.434526+0000","last_peered":"2023-10-10T18:45:11.434526+0000","last_clean":"2023-10-10T18:45:11.434526+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T18:45:11.434526+0000","last_undegraded":"2023-10-10T18:45:11.434526+0000","last_fullsized":"2023-10-10T18:45:11.434526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427193+0000","last_change":"2023-10-10T18:45:11.422935+0000","last_active":"2023-10-10T18:45:11.427193+0000","last_peered":"2023-10-10T18:45:11.427193+0000","last_clean":"2023-10-10T18:45:11.427193+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T18:45:11.427193+0000","last_undegraded":"2023-10-10T18:45:11.427193+0000","last_fullsized":"2023-10-10T18:45:11.427193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434131+0000","last_change":"2023-10-10T18:45:11.431202+0000","last_active":"2023-10-10T18:45:11.434131+0000","last_peered":"2023-10-10T18:45:11.434131+0000","last_clean":"2023-10-10T18:45:11.434131+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T18:45:11.434131+0000","last_undegraded":"2023-10-10T18:45:11.434131+0000","last_fullsized":"2023-10-10T18:45:11.434131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427139+0000","last_change":"2023-10-10T18:45:11.424885+0000","last_active":"2023-10-10T18:45:11.427139+0000","last_peered":"2023-10-10T18:45:11.427139+0000","last_clean":"2023-10-10T18:45:11.427139+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T18:45:11.427139+0000","last_undegraded":"2023-10-10T18:45:11.427139+0000","last_fullsized":"2023-10-10T18:45:11.427139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426206+0000","last_change":"2023-10-10T18:45:11.421039+0000","last_active":"2023-10-10T18:45:11.426206+0000","last_peered":"2023-10-10T18:45:11.426206+0000","last_clean":"2023-10-10T18:45:11.426206+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T18:45:11.426206+0000","last_undegraded":"2023-10-10T18:45:11.426206+0000","last_fullsized":"2023-10-10T18:45:11.426206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.554010+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T18:45:40.554010+0000","last_peered":"2023-10-10T18:45:40.554010+0000","last_clean":"2023-10-10T18:45:40.554010+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T18:45:40.554010+0000","last_undegraded":"2023-10-10T18:45:40.554010+0000","last_fullsized":"2023-10-10T18:45:40.554010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421043+0000","last_change":"2023-10-10T18:45:11.417504+0000","last_active":"2023-10-10T18:45:11.421043+0000","last_peered":"2023-10-10T18:45:11.421043+0000","last_clean":"2023-10-10T18:45:11.421043+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T18:45:11.421043+0000","last_undegraded":"2023-10-10T18:45:11.421043+0000","last_fullsized":"2023-10-10T18:45:11.421043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421448+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T18:45:16.421448+0000","last_peered":"2023-10-10T18:45:16.421448+0000","last_clean":"2023-10-10T18:45:16.421448+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T18:45:16.421448+0000","last_undegraded":"2023-10-10T18:45:16.421448+0000","last_fullsized":"2023-10-10T18:45:16.421448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424178+0000","last_change":"2023-10-10T18:45:11.421896+0000","last_active":"2023-10-10T18:45:11.424178+0000","last_peered":"2023-10-10T18:45:11.424178+0000","last_clean":"2023-10-10T18:45:11.424178+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T18:45:11.424178+0000","last_undegraded":"2023-10-10T18:45:11.424178+0000","last_fullsized":"2023-10-10T18:45:11.424178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552890+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T18:45:40.552890+0000","last_peered":"2023-10-10T18:45:40.552890+0000","last_clean":"2023-10-10T18:45:40.552890+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T18:45:40.552890+0000","last_undegraded":"2023-10-10T18:45:40.552890+0000","last_fullsized":"2023-10-10T18:45:40.552890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432466+0000","last_change":"2023-10-10T18:45:11.429680+0000","last_active":"2023-10-10T18:45:11.432466+0000","last_peered":"2023-10-10T18:45:11.432466+0000","last_clean":"2023-10-10T18:45:11.432466+0000","last_became_active":"2023-10-10T18:45:11.429397+0000","last_became_peered":"2023-10-10T18:45:11.429397+0000","last_unstale":"2023-10-10T18:45:11.432466+0000","last_undegraded":"2023-10-10T18:45:11.432466+0000","last_fullsized":"2023-10-10T18:45:11.432466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:19:52.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.551576+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T18:45:40.551576+0000","last_peered":"2023-10-10T18:45:40.551576+0000","last_clean":"2023-10-10T18:45:40.551576+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T18:45:40.551576+0000","last_undegraded":"2023-10-10T18:45:40.551576+0000","last_fullsized":"2023-10-10T18:45:40.551576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432439+0000","last_change":"2023-10-10T18:45:11.429408+0000","last_active":"2023-10-10T18:45:11.432439+0000","last_peered":"2023-10-10T18:45:11.432439+0000","last_clean":"2023-10-10T18:45:11.432439+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T18:45:11.432439+0000","last_undegraded":"2023-10-10T18:45:11.432439+0000","last_fullsized":"2023-10-10T18:45:11.432439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.457325+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T18:45:22.457325+0000","last_peered":"2023-10-10T18:45:22.457325+0000","last_clean":"2023-10-10T18:45:22.457325+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T18:45:22.457325+0000","last_undegraded":"2023-10-10T18:45:22.457325+0000","last_fullsized":"2023-10-10T18:45:22.457325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430579+0000","last_change":"2023-10-10T18:45:11.426051+0000","last_active":"2023-10-10T18:45:11.430579+0000","last_peered":"2023-10-10T18:45:11.430579+0000","last_clean":"2023-10-10T18:45:11.430579+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T18:45:11.430579+0000","last_undegraded":"2023-10-10T18:45:11.430579+0000","last_fullsized":"2023-10-10T18:45:11.430579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531752+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T18:45:34.531752+0000","last_peered":"2023-10-10T18:45:34.531752+0000","last_clean":"2023-10-10T18:45:34.531752+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T18:45:34.531752+0000","last_undegraded":"2023-10-10T18:45:34.531752+0000","last_fullsized":"2023-10-10T18:45:34.531752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427223+0000","last_change":"2023-10-10T18:45:11.425183+0000","last_active":"2023-10-10T18:45:11.427223+0000","last_peered":"2023-10-10T18:45:11.427223+0000","last_clean":"2023-10-10T18:45:11.427223+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T18:45:11.427223+0000","last_undegraded":"2023-10-10T18:45:11.427223+0000","last_fullsized":"2023-10-10T18:45:11.427223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.523628+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T18:45:28.523628+0000","last_peered":"2023-10-10T18:45:28.523628+0000","last_clean":"2023-10-10T18:45:28.523628+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T18:45:28.523628+0000","last_undegraded":"2023-10-10T18:45:28.523628+0000","last_fullsized":"2023-10-10T18:45:28.523628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415249+0000","last_change":"2023-10-10T18:45:11.412641+0000","last_active":"2023-10-10T18:45:11.415249+0000","last_peered":"2023-10-10T18:45:11.415249+0000","last_clean":"2023-10-10T18:45:11.415249+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T18:45:11.415249+0000","last_undegraded":"2023-10-10T18:45:11.415249+0000","last_fullsized":"2023-10-10T18:45:11.415249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531323+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T18:45:34.531323+0000","last_peered":"2023-10-10T18:45:34.531323+0000","last_clean":"2023-10-10T18:45:34.531323+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T18:45:34.531323+0000","last_undegraded":"2023-10-10T18:45:34.531323+0000","last_fullsized":"2023-10-10T18:45:34.531323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420638+0000","last_change":"2023-10-10T18:45:11.413819+0000","last_active":"2023-10-10T18:45:11.420638+0000","last_peered":"2023-10-10T18:45:11.420638+0000","last_clean":"2023-10-10T18:45:11.420638+0000","last_became_active":"2023-10-10T18:45:11.413529+0000","last_became_peered":"2023-10-10T18:45:11.413529+0000","last_unstale":"2023-10-10T18:45:11.420638+0000","last_undegraded":"2023-10-10T18:45:11.420638+0000","last_fullsized":"2023-10-10T18:45:11.420638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.460216+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T18:45:22.460216+0000","last_peered":"2023-10-10T18:45:22.460216+0000","last_clean":"2023-10-10T18:45:22.460216+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T18:45:22.460216+0000","last_undegraded":"2023-10-10T18:45:22.460216+0000","last_fullsized":"2023-10-10T18:45:22.460216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430421+0000","last_change":"2023-10-10T18:45:11.427116+0000","last_active":"2023-10-10T18:45:11.430421+0000","last_peered":"2023-10-10T18:45:11.430421+0000","last_clean":"2023-10-10T18:45:11.430421+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T18:45:11.430421+0000","last_undegraded":"2023-10-10T18:45:11.430421+0000","last_fullsized":"2023-10-10T18:45:11.430421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552523+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T18:45:40.552523+0000","last_peered":"2023-10-10T18:45:40.552523+0000","last_clean":"2023-10-10T18:45:40.552523+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T18:45:40.552523+0000","last_undegraded":"2023-10-10T18:45:40.552523+0000","last_fullsized":"2023-10-10T18:45:40.552523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424158+0000","last_change":"2023-10-10T18:45:11.421794+0000","last_active":"2023-10-10T18:45:11.424158+0000","last_peered":"2023-10-10T18:45:11.424158+0000","last_clean":"2023-10-10T18:45:11.424158+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T18:45:11.424158+0000","last_undegraded":"2023-10-10T18:45:11.424158+0000","last_fullsized":"2023-10-10T18:45:11.424158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.459519+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T18:45:22.459519+0000","last_peered":"2023-10-10T18:45:22.459519+0000","last_clean":"2023-10-10T18:45:22.459519+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T18:45:22.459519+0000","last_undegraded":"2023-10-10T18:45:22.459519+0000","last_fullsized":"2023-10-10T18:45:22.459519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434442+0000","last_change":"2023-10-10T18:45:11.431857+0000","last_active":"2023-10-10T18:45:11.434442+0000","last_peered":"2023-10-10T18:45:11.434442+0000","last_clean":"2023-10-10T18:45:11.434442+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T18:45:11.434442+0000","last_undegraded":"2023-10-10T18:45:11.434442+0000","last_fullsized":"2023-10-10T18:45:11.434442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.553208+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T18:45:40.553208+0000","last_peered":"2023-10-10T18:45:40.553208+0000","last_clean":"2023-10-10T18:45:40.553208+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T18:45:40.553208+0000","last_undegraded":"2023-10-10T18:45:40.553208+0000","last_fullsized":"2023-10-10T18:45:40.553208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426196+0000","last_change":"2023-10-10T18:45:11.419774+0000","last_active":"2023-10-10T18:45:11.426196+0000","last_peered":"2023-10-10T18:45:11.426196+0000","last_clean":"2023-10-10T18:45:11.426196+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T18:45:11.426196+0000","last_undegraded":"2023-10-10T18:45:11.426196+0000","last_fullsized":"2023-10-10T18:45:11.426196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.551374+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T18:45:40.551374+0000","last_peered":"2023-10-10T18:45:40.551374+0000","last_clean":"2023-10-10T18:45:40.551374+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T18:45:40.551374+0000","last_undegraded":"2023-10-10T18:45:40.551374+0000","last_fullsized":"2023-10-10T18:45:40.551374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424245+0000","last_change":"2023-10-10T18:45:11.420517+0000","last_active":"2023-10-10T18:45:11.424245+0000","last_peered":"2023-10-10T18:45:11.424245+0000","last_clean":"2023-10-10T18:45:11.424245+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T18:45:11.424245+0000","last_undegraded":"2023-10-10T18:45:11.424245+0000","last_fullsized":"2023-10-10T18:45:11.424245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.527081+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T18:45:34.527081+0000","last_peered":"2023-10-10T18:45:34.527081+0000","last_clean":"2023-10-10T18:45:34.527081+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T18:45:34.527081+0000","last_undegraded":"2023-10-10T18:45:34.527081+0000","last_fullsized":"2023-10-10T18:45:34.527081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431834+0000","last_change":"2023-10-10T18:45:11.427109+0000","last_active":"2023-10-10T18:45:11.431834+0000","last_peered":"2023-10-10T18:45:11.431834+0000","last_clean":"2023-10-10T18:45:11.431834+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T18:45:11.431834+0000","last_undegraded":"2023-10-10T18:45:11.431834+0000","last_fullsized":"2023-10-10T18:45:11.431834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529442+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T18:45:34.529442+0000","last_peered":"2023-10-10T18:45:34.529442+0000","last_clean":"2023-10-10T18:45:34.529442+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T18:45:34.529442+0000","last_undegraded":"2023-10-10T18:45:34.529442+0000","last_fullsized":"2023-10-10T18:45:34.529442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430660+0000","last_change":"2023-10-10T18:45:11.416459+0000","last_active":"2023-10-10T18:45:11.430660+0000","last_peered":"2023-10-10T18:45:11.430660+0000","last_clean":"2023-10-10T18:45:11.430660+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T18:45:11.430660+0000","last_undegraded":"2023-10-10T18:45:11.430660+0000","last_fullsized":"2023-10-10T18:45:11.430660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.527437+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T18:45:34.527437+0000","last_peered":"2023-10-10T18:45:34.527437+0000","last_clean":"2023-10-10T18:45:34.527437+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T18:45:34.527437+0000","last_undegraded":"2023-10-10T18:45:34.527437+0000","last_fullsized":"2023-10-10T18:45:34.527437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432328+0000","last_change":"2023-10-10T18:45:11.429727+0000","last_active":"2023-10-10T18:45:11.432328+0000","last_peered":"2023-10-10T18:45:11.432328+0000","last_clean":"2023-10-10T18:45:11.432328+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T18:45:11.432328+0000","last_undegraded":"2023-10-10T18:45:11.432328+0000","last_fullsized":"2023-10-10T18:45:11.432328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421569+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T18:45:16.421569+0000","last_peered":"2023-10-10T18:45:16.421569+0000","last_clean":"2023-10-10T18:45:16.421569+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T18:45:16.421569+0000","last_undegraded":"2023-10-10T18:45:16.421569+0000","last_fullsized":"2023-10-10T18:45:16.421569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431858+0000","last_change":"2023-10-10T18:45:11.427926+0000","last_active":"2023-10-10T18:45:11.431858+0000","last_peered":"2023-10-10T18:45:11.431858+0000","last_clean":"2023-10-10T18:45:11.431858+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T18:45:11.431858+0000","last_undegraded":"2023-10-10T18:45:11.431858+0000","last_fullsized":"2023-10-10T18:45:11.431858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.009459+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T18:45:13.009459+0000","last_peered":"2023-10-10T18:45:13.009459+0000","last_clean":"2023-10-10T18:45:13.009459+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T18:45:13.009459+0000","last_undegraded":"2023-10-10T18:45:13.009459+0000","last_fullsized":"2023-10-10T18:45:13.009459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427955+0000","last_change":"2023-10-10T18:45:11.423279+0000","last_active":"2023-10-10T18:45:11.427955+0000","last_peered":"2023-10-10T18:45:11.427955+0000","last_clean":"2023-10-10T18:45:11.427955+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T18:45:11.427955+0000","last_undegraded":"2023-10-10T18:45:11.427955+0000","last_fullsized":"2023-10-10T18:45:11.427955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.554631+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T18:45:40.554631+0000","last_peered":"2023-10-10T18:45:40.554631+0000","last_clean":"2023-10-10T18:45:40.554631+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T18:45:40.554631+0000","last_undegraded":"2023-10-10T18:45:40.554631+0000","last_fullsized":"2023-10-10T18:45:40.554631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430728+0000","last_change":"2023-10-10T18:45:11.425339+0000","last_active":"2023-10-10T18:45:11.430728+0000","last_peered":"2023-10-10T18:45:11.430728+0000","last_clean":"2023-10-10T18:45:11.430728+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T18:45:11.430728+0000","last_undegraded":"2023-10-10T18:45:11.430728+0000","last_fullsized":"2023-10-10T18:45:11.430728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421278+0000","last_change":"2023-10-10T18:45:11.418937+0000","last_active":"2023-10-10T18:45:11.421278+0000","last_peered":"2023-10-10T18:45:11.421278+0000","last_clean":"2023-10-10T18:45:11.421278+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T18:45:11.421278+0000","last_undegraded":"2023-10-10T18:45:11.421278+0000","last_fullsized":"2023-10-10T18:45:11.421278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.524806+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T18:45:28.524806+0000","last_peered":"2023-10-10T18:45:28.524806+0000","last_clean":"2023-10-10T18:45:28.524806+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T18:45:28.524806+0000","last_undegraded":"2023-10-10T18:45:28.524806+0000","last_fullsized":"2023-10-10T18:45:28.524806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424136+0000","last_change":"2023-10-10T18:45:11.422452+0000","last_active":"2023-10-10T18:45:11.424136+0000","last_peered":"2023-10-10T18:45:11.424136+0000","last_clean":"2023-10-10T18:45:11.424136+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T18:45:11.424136+0000","last_undegraded":"2023-10-10T18:45:11.424136+0000","last_fullsized":"2023-10-10T18:45:11.424136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529502+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T18:45:34.529502+0000","last_peered":"2023-10-10T18:45:34.529502+0000","last_clean":"2023-10-10T18:45:34.529502+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T18:45:34.529502+0000","last_undegraded":"2023-10-10T18:45:34.529502+0000","last_fullsized":"2023-10-10T18:45:34.529502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405467+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T18:45:11.405467+0000","last_peered":"2023-10-10T18:45:11.405467+0000","last_clean":"2023-10-10T18:45:11.405467+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T18:45:11.405467+0000","last_undegraded":"2023-10-10T18:45:11.405467+0000","last_fullsized":"2023-10-10T18:45:11.405467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434404+0000","last_change":"2023-10-10T18:45:11.430233+0000","last_active":"2023-10-10T18:45:11.434404+0000","last_peered":"2023-10-10T18:45:11.434404+0000","last_clean":"2023-10-10T18:45:11.434404+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T18:45:11.434404+0000","last_undegraded":"2023-10-10T18:45:11.434404+0000","last_fullsized":"2023-10-10T18:45:11.434404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.520084+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T18:45:28.520084+0000","last_peered":"2023-10-10T18:45:28.520084+0000","last_clean":"2023-10-10T18:45:28.520084+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T18:45:28.520084+0000","last_undegraded":"2023-10-10T18:45:28.520084+0000","last_fullsized":"2023-10-10T18:45:28.520084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531458+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T18:45:34.531458+0000","last_peered":"2023-10-10T18:45:34.531458+0000","last_clean":"2023-10-10T18:45:34.531458+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T18:45:34.531458+0000","last_undegraded":"2023-10-10T18:45:34.531458+0000","last_fullsized":"2023-10-10T18:45:34.531458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421286+0000","last_change":"2023-10-10T18:45:11.418973+0000","last_active":"2023-10-10T18:45:11.421286+0000","last_peered":"2023-10-10T18:45:11.421286+0000","last_clean":"2023-10-10T18:45:11.421286+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T18:45:11.421286+0000","last_undegraded":"2023-10-10T18:45:11.421286+0000","last_fullsized":"2023-10-10T18:45:11.421286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531687+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T18:45:34.531687+0000","last_peered":"2023-10-10T18:45:34.531687+0000","last_clean":"2023-10-10T18:45:34.531687+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T18:45:34.531687+0000","last_undegraded":"2023-10-10T18:45:34.531687+0000","last_fullsized":"2023-10-10T18:45:34.531687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423727+0000","last_change":"2023-10-10T18:45:11.419846+0000","last_active":"2023-10-10T18:45:11.423727+0000","last_peered":"2023-10-10T18:45:11.423727+0000","last_clean":"2023-10-10T18:45:11.423727+0000","last_became_active":"2023-10-10T18:45:11.419579+0000","last_became_peered":"2023-10-10T18:45:11.419579+0000","last_unstale":"2023-10-10T18:45:11.423727+0000","last_undegraded":"2023-10-10T18:45:11.423727+0000","last_fullsized":"2023-10-10T18:45:11.423727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:58:49.831558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552259+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T18:45:40.552259+0000","last_peered":"2023-10-10T18:45:40.552259+0000","last_clean":"2023-10-10T18:45:40.552259+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T18:45:40.552259+0000","last_undegraded":"2023-10-10T18:45:40.552259+0000","last_fullsized":"2023-10-10T18:45:40.552259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423663+0000","last_change":"2023-10-10T18:45:11.416217+0000","last_active":"2023-10-10T18:45:11.423663+0000","last_peered":"2023-10-10T18:45:11.423663+0000","last_clean":"2023-10-10T18:45:11.423663+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T18:45:11.423663+0000","last_undegraded":"2023-10-10T18:45:11.423663+0000","last_fullsized":"2023-10-10T18:45:11.423663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.659888+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T18:45:22.659888+0000","last_peered":"2023-10-10T18:45:22.659888+0000","last_clean":"2023-10-10T18:45:22.659888+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T18:45:22.659888+0000","last_undegraded":"2023-10-10T18:45:22.659888+0000","last_fullsized":"2023-10-10T18:45:22.659888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431602+0000","last_change":"2023-10-10T18:45:11.427826+0000","last_active":"2023-10-10T18:45:11.431602+0000","last_peered":"2023-10-10T18:45:11.431602+0000","last_clean":"2023-10-10T18:45:11.431602+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T18:45:11.431602+0000","last_undegraded":"2023-10-10T18:45:11.431602+0000","last_fullsized":"2023-10-10T18:45:11.431602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552130+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T18:45:40.552130+0000","last_peered":"2023-10-10T18:45:40.552130+0000","last_clean":"2023-10-10T18:45:40.552130+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T18:45:40.552130+0000","last_undegraded":"2023-10-10T18:45:40.552130+0000","last_fullsized":"2023-10-10T18:45:40.552130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415388+0000","last_change":"2023-10-10T18:45:11.413975+0000","last_active":"2023-10-10T18:45:11.415388+0000","last_peered":"2023-10-10T18:45:11.415388+0000","last_clean":"2023-10-10T18:45:11.415388+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T18:45:11.415388+0000","last_undegraded":"2023-10-10T18:45:11.415388+0000","last_fullsized":"2023-10-10T18:45:11.415388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529028+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T18:45:34.529028+0000","last_peered":"2023-10-10T18:45:34.529028+0000","last_clean":"2023-10-10T18:45:34.529028+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T18:45:34.529028+0000","last_undegraded":"2023-10-10T18:45:34.529028+0000","last_fullsized":"2023-10-10T18:45:34.529028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415058+0000","last_change":"2023-10-10T18:45:11.410768+0000","last_active":"2023-10-10T18:45:11.415058+0000","last_peered":"2023-10-10T18:45:11.415058+0000","last_clean":"2023-10-10T18:45:11.415058+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T18:45:11.415058+0000","last_undegraded":"2023-10-10T18:45:11.415058+0000","last_fullsized":"2023-10-10T18:45:11.415058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.660373+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T18:45:22.660373+0000","last_peered":"2023-10-10T18:45:22.660373+0000","last_clean":"2023-10-10T18:45:22.660373+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T18:45:22.660373+0000","last_undegraded":"2023-10-10T18:45:22.660373+0000","last_fullsized":"2023-10-10T18:45:22.660373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430290+0000","last_change":"2023-10-10T18:45:11.425716+0000","last_active":"2023-10-10T18:45:11.430290+0000","last_peered":"2023-10-10T18:45:11.430290+0000","last_clean":"2023-10-10T18:45:11.430290+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T18:45:11.430290+0000","last_undegraded":"2023-10-10T18:45:11.430290+0000","last_fullsized":"2023-10-10T18:45:11.430290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.530979+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T18:45:34.530979+0000","last_peered":"2023-10-10T18:45:34.530979+0000","last_clean":"2023-10-10T18:45:34.530979+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T18:45:34.530979+0000","last_undegraded":"2023-10-10T18:45:34.530979+0000","last_fullsized":"2023-10-10T18:45:34.530979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429787+0000","last_change":"2023-10-10T18:45:11.423747+0000","last_active":"2023-10-10T18:45:11.429787+0000","last_peered":"2023-10-10T18:45:11.429787+0000","last_clean":"2023-10-10T18:45:11.429787+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T18:45:11.429787+0000","last_undegraded":"2023-10-10T18:45:11.429787+0000","last_fullsized":"2023-10-10T18:45:11.429787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.532321+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T18:45:34.532321+0000","last_peered":"2023-10-10T18:45:34.532321+0000","last_clean":"2023-10-10T18:45:34.532321+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T18:45:34.532321+0000","last_undegraded":"2023-10-10T18:45:34.532321+0000","last_fullsized":"2023-10-10T18:45:34.532321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425816+0000","last_change":"2023-10-10T18:45:11.421872+0000","last_active":"2023-10-10T18:45:11.425816+0000","last_peered":"2023-10-10T18:45:11.425816+0000","last_clean":"2023-10-10T18:45:11.425816+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T18:45:11.425816+0000","last_undegraded":"2023-10-10T18:45:11.425816+0000","last_fullsized":"2023-10-10T18:45:11.425816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.550580+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T18:45:40.550580+0000","last_peered":"2023-10-10T18:45:40.550580+0000","last_clean":"2023-10-10T18:45:40.550580+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T18:45:40.550580+0000","last_undegraded":"2023-10-10T18:45:40.550580+0000","last_fullsized":"2023-10-10T18:45:40.550580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429963+0000","last_change":"2023-10-10T18:45:11.425474+0000","last_active":"2023-10-10T18:45:11.429963+0000","last_peered":"2023-10-10T18:45:11.429963+0000","last_clean":"2023-10-10T18:45:11.429963+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T18:45:11.429963+0000","last_undegraded":"2023-10-10T18:45:11.429963+0000","last_fullsized":"2023-10-10T18:45:11.429963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430681+0000","last_change":"2023-10-10T18:45:11.416850+0000","last_active":"2023-10-10T18:45:11.430681+0000","last_peered":"2023-10-10T18:45:11.430681+0000","last_clean":"2023-10-10T18:45:11.430681+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T18:45:11.430681+0000","last_undegraded":"2023-10-10T18:45:11.430681+0000","last_fullsized":"2023-10-10T18:45:11.430681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427108+0000","last_change":"2023-10-10T18:45:11.424440+0000","last_active":"2023-10-10T18:45:11.427108+0000","last_peered":"2023-10-10T18:45:11.427108+0000","last_clean":"2023-10-10T18:45:11.427108+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T18:45:11.427108+0000","last_undegraded":"2023-10-10T18:45:11.427108+0000","last_fullsized":"2023-10-10T18:45:11.427108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.421524+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.420874+0000","last_became_peered":"2023-10-10T18:45:11.420874+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:56:36.851801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419177+0000","last_change":"2023-10-10T18:45:11.412399+0000","last_active":"2023-10-10T18:45:11.419177+0000","last_peered":"2023-10-10T18:45:11.419177+0000","last_clean":"2023-10-10T18:45:11.419177+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T18:45:11.419177+0000","last_undegraded":"2023-10-10T18:45:11.419177+0000","last_fullsized":"2023-10-10T18:45:11.419177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432996+0000","last_change":"2023-10-10T18:45:11.429507+0000","last_active":"2023-10-10T18:45:11.432996+0000","last_peered":"2023-10-10T18:45:11.432996+0000","last_clean":"2023-10-10T18:45:11.432996+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T18:45:11.432996+0000","last_undegraded":"2023-10-10T18:45:11.432996+0000","last_fullsized":"2023-10-10T18:45:11.432996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.418310+0000","last_change":"2023-10-10T18:45:11.413271+0000","last_active":"2023-10-10T18:45:11.418310+0000","last_peered":"2023-10-10T18:45:11.418310+0000","last_clean":"2023-10-10T18:45:11.418310+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T18:45:11.418310+0000","last_undegraded":"2023-10-10T18:45:11.418310+0000","last_fullsized":"2023-10-10T18:45:11.418310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430781+0000","last_change":"2023-10-10T18:45:11.427815+0000","last_active":"2023-10-10T18:45:11.430781+0000","last_peered":"2023-10-10T18:45:11.430781+0000","last_clean":"2023-10-10T18:45:11.430781+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T18:45:11.430781+0000","last_undegraded":"2023-10-10T18:45:11.430781+0000","last_fullsized":"2023-10-10T18:45:11.430781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426972+0000","last_change":"2023-10-10T18:45:11.425458+0000","last_active":"2023-10-10T18:45:11.426972+0000","last_peered":"2023-10-10T18:45:11.426972+0000","last_clean":"2023-10-10T18:45:11.426972+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T18:45:11.426972+0000","last_undegraded":"2023-10-10T18:45:11.426972+0000","last_fullsized":"2023-10-10T18:45:11.426972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424565+0000","last_change":"2023-10-10T18:45:11.422003+0000","last_active":"2023-10-10T18:45:11.424565+0000","last_peered":"2023-10-10T18:45:11.424565+0000","last_clean":"2023-10-10T18:45:11.424565+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T18:45:11.424565+0000","last_undegraded":"2023-10-10T18:45:11.424565+0000","last_fullsized":"2023-10-10T18:45:11.424565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432978+0000","last_change":"2023-10-10T18:45:11.428907+0000","last_active":"2023-10-10T18:45:11.432978+0000","last_peered":"2023-10-10T18:45:11.432978+0000","last_clean":"2023-10-10T18:45:11.432978+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T18:45:11.432978+0000","last_undegraded":"2023-10-10T18:45:11.432978+0000","last_fullsized":"2023-10-10T18:45:11.432978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430538+0000","last_change":"2023-10-10T18:45:11.427241+0000","last_active":"2023-10-10T18:45:11.430538+0000","last_peered":"2023-10-10T18:45:11.430538+0000","last_clean":"2023-10-10T18:45:11.430538+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T18:45:11.430538+0000","last_undegraded":"2023-10-10T18:45:11.430538+0000","last_fullsized":"2023-10-10T18:45:11.430538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427471+0000","last_change":"2023-10-10T18:45:11.426813+0000","last_active":"2023-10-10T18:45:11.427471+0000","last_peered":"2023-10-10T18:45:11.427471+0000","last_clean":"2023-10-10T18:45:11.427471+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T18:45:11.427471+0000","last_undegraded":"2023-10-10T18:45:11.427471+0000","last_fullsized":"2023-10-10T18:45:11.427471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432098+0000","last_change":"2023-10-10T18:45:11.429582+0000","last_active":"2023-10-10T18:45:11.432098+0000","last_peered":"2023-10-10T18:45:11.432098+0000","last_clean":"2023-10-10T18:45:11.432098+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T18:45:11.432098+0000","last_undegraded":"2023-10-10T18:45:11.432098+0000","last_fullsized":"2023-10-10T18:45:11.432098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7626,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":15252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074240,"statfs":{"total":96636764160,"available":96332021760,"internally_reserved":0,"allocated":319488,"data_stored":667677,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":245760,"data_stored":77591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":245760,"data_stored":76769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":666527,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":249856,"data_stored":76699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":249856,"data_stored":77625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297524,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074316,"statfs":{"total":96636764160,"available":96332099584,"internally_reserved":0,"allocated":241664,"data_stored":76567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":79037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T18:45:53.694 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-10T18:45:53.695 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T18:45:53.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.843+0000 7f1f4308d700 1 Processor -- start 2023-10-10T18:45:53.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.843+0000 7f1f4308d700 1 -- start start 2023-10-10T18:45:53.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f4308d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 0x7f1f3c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f4308d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1f3c0aff60 0x7f1f3c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f4308d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c1574b0 con 0x7f1f3c0b2390 2023-10-10T18:45:53.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f4308d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c150770 con 0x7f1f3c0aff60 2023-10-10T18:45:53.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f4308d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0afad0 con 0x7f1f3c0b1fc0 2023-10-10T18:45:53.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f4162a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1f3c0aff60 0x7f1f3c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f4162a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1f3c0aff60 0x7f1f3c0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37866/0 (socket says 172.21.15.175:37866) 2023-10-10T18:45:53.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f4162a700 1 -- 172.21.15.175:0/1228517027 learned_addr learned my addr 172.21.15.175:0/1228517027 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:53.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f3bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1f3c0b2390 0x7f1f3c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51574/0 (socket says 172.21.15.175:51574) 2023-10-10T18:45:53.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.844+0000 7f1f40e29700 1 --2- 172.21.15.175:0/1228517027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 0x7f1f3c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f4162a700 1 -- 172.21.15.175:0/1228517027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 msgr2=0x7f1f3c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:53.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f4162a700 1 --2- 172.21.15.175:0/1228517027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 0x7f1f3c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f4162a700 1 -- 172.21.15.175:0/1228517027 >> v1:172.21.15.175:6789/0 conn(0x7f1f3c0b2390 legacy=0x7f1f3c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f4162a700 1 -- 172.21.15.175:0/1228517027 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c0af4c0 con 0x7f1f3c0aff60 2023-10-10T18:45:53.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f3b7fe700 1 -- 172.21.15.175:0/1228517027 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491223701 0 0) 0x7f1f3c1574b0 con 0x7f1f3c0b2390 2023-10-10T18:45:53.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f4162a700 1 --2- 172.21.15.175:0/1228517027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0b03e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f340099e0 tx=0x7f1f34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31254b5fca5a9f76 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f3b7fe700 1 -- 172.21.15.175:0/1228517027 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400b8b0 con 0x7f1f3c0aff60 2023-10-10T18:45:53.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f3b7fe700 1 -- 172.21.15.175:0/1228517027 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3400ba10 con 0x7f1f3c0aff60 2023-10-10T18:45:53.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f3b7fe700 1 -- 172.21.15.175:0/1228517027 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400bdb0 con 0x7f1f3c0aff60 2023-10-10T18:45:53.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f4308d700 1 -- 172.21.15.175:0/1228517027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 msgr2=0x7f1f3c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.845+0000 7f1f4308d700 1 --2- 172.21.15.175:0/1228517027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0b03e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f340099e0 tx=0x7f1f34009250 comp rx=0 tx=0).stop 2023-10-10T18:45:53.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.846+0000 7f1f4308d700 1 -- 172.21.15.175:0/1228517027 shutdown_connections 2023-10-10T18:45:53.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.846+0000 7f1f4308d700 1 --2- 172.21.15.175:0/1228517027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0b03e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.846+0000 7f1f4308d700 1 --2- 172.21.15.175:0/1228517027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 0x7f1f3c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.846+0000 7f1f4308d700 1 -- 172.21.15.175:0/1228517027 >> 172.21.15.175:0/1228517027 conn(0x7f1f3c063230 msgr2=0x7f1f3c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:53.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.846+0000 7f1f4308d700 1 -- 172.21.15.175:0/1228517027 shutdown_connections 2023-10-10T18:45:53.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.846+0000 7f1f4308d700 1 -- 172.21.15.175:0/1228517027 wait complete. 2023-10-10T18:45:53.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.847+0000 7f1f4308d700 1 Processor -- start 2023-10-10T18:45:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.847+0000 7f1f4308d700 1 -- start start 2023-10-10T18:45:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.847+0000 7f1f4308d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.847+0000 7f1f4308d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 0x7f1f3c0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.847+0000 7f1f4308d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f3c0b2390 0x7f1f3c168f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.847+0000 7f1f4308d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c157180 con 0x7f1f3c0aff60 2023-10-10T18:45:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.847+0000 7f1f4308d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0c39a0 con 0x7f1f3c0b2390 2023-10-10T18:45:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.847+0000 7f1f4308d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0c3c80 con 0x7f1f3c0b1fc0 2023-10-10T18:45:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37882/0 (socket says 172.21.15.175:37882) 2023-10-10T18:45:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 -- 172.21.15.175:0/446615256 learned_addr learned my addr 172.21.15.175:0/446615256 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f3bfff700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 0x7f1f3c0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f4162a700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f3c0b2390 0x7f1f3c168f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 -- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 msgr2=0x7f1f3c0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 0x7f1f3c0c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 -- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f3c0b2390 msgr2=0x7f1f3c168f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f3c0b2390 0x7f1f3c168f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 -- 172.21.15.175:0/446615256 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f340094f0 con 0x7f1f3c0aff60 2023-10-10T18:45:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f40e29700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0c5820 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009ef0 tx=0x7f1f2c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f38ff9700 1 -- 172.21.15.175:0/446615256 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c00cc50 con 0x7f1f3c0aff60 2023-10-10T18:45:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f38ff9700 1 -- 172.21.15.175:0/446615256 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c00f040 con 0x7f1f3c0aff60 2023-10-10T18:45:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c1573d0 con 0x7f1f3c0aff60 2023-10-10T18:45:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f38ff9700 1 -- 172.21.15.175:0/446615256 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c014480 con 0x7f1f3c0aff60 2023-10-10T18:45:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.848+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c068960 con 0x7f1f3c0aff60 2023-10-10T18:45:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.849+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c067570 con 0x7f1f3c0aff60 2023-10-10T18:45:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.850+0000 7f1f38ff9700 1 -- 172.21.15.175:0/446615256 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1f2c0147f0 con 0x7f1f3c0aff60 2023-10-10T18:45:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.850+0000 7f1f38ff9700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f28066f30 0x7f1f280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.850+0000 7f1f38ff9700 1 -- 172.21.15.175:0/446615256 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1f2c085320 con 0x7f1f3c0aff60 2023-10-10T18:45:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.850+0000 7f1f3bfff700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f28066f30 0x7f1f280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.851+0000 7f1f3bfff700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f28066f30 0x7f1f280693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c15a590 tx=0x7f1f30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:53.854+0000 7f1f38ff9700 1 -- 172.21.15.175:0/446615256 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f2c052590 con 0x7f1f3c0aff60 2023-10-10T18:45:54.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.031+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f3c068da0 con 0x7f1f28066f30 2023-10-10T18:45:54.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.039+0000 7f1f38ff9700 1 -- 172.21.15.175:0/446615256 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264694 (secure 0 0 0) 0x7f1f3c068da0 con 0x7f1f28066f30 2023-10-10T18:45:54.042 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.042+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f28066f30 msgr2=0x7f1f280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.042+0000 7f1f4308d700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f28066f30 0x7f1f280693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c15a590 tx=0x7f1f30009040 comp rx=0 tx=0).stop 2023-10-10T18:45:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.042+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 msgr2=0x7f1f3c0c5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.042+0000 7f1f4308d700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0c5820 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009ef0 tx=0x7f1f2c00b040 comp rx=0 tx=0).stop 2023-10-10T18:45:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.043+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 shutdown_connections 2023-10-10T18:45:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.043+0000 7f1f4308d700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f28066f30 0x7f1f280693e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.043+0000 7f1f4308d700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f3c0b2390 0x7f1f3c168f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.043+0000 7f1f4308d700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f3c0aff60 0x7f1f3c0c5820 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.043+0000 7f1f4308d700 1 --2- 172.21.15.175:0/446615256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f3c0b1fc0 0x7f1f3c0c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.043+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 >> 172.21.15.175:0/446615256 conn(0x7f1f3c063230 msgr2=0x7f1f3c153ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.043+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 shutdown_connections 2023-10-10T18:45:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.043+0000 7f1f4308d700 1 -- 172.21.15.175:0/446615256 wait complete. 2023-10-10T18:45:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T18:45:54.054 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2023-10-10T18:45:53.140876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597994,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2380384,"kb_used_data":2080,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752594336,"statfs":{"total":773094113280,"available":770656600064,"internally_reserved":0,"allocated":2129920,"data_stored":1798492,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1330,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003287"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429926+0000","last_change":"2023-10-10T18:45:11.419184+0000","last_active":"2023-10-10T18:45:11.429926+0000","last_peered":"2023-10-10T18:45:11.429926+0000","last_clean":"2023-10-10T18:45:11.429926+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T18:45:11.429926+0000","last_undegraded":"2023-10-10T18:45:11.429926+0000","last_fullsized":"2023-10-10T18:45:11.429926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420684+0000","last_change":"2023-10-10T18:45:11.414586+0000","last_active":"2023-10-10T18:45:11.420684+0000","last_peered":"2023-10-10T18:45:11.420684+0000","last_clean":"2023-10-10T18:45:11.420684+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T18:45:11.420684+0000","last_undegraded":"2023-10-10T18:45:11.420684+0000","last_fullsized":"2023-10-10T18:45:11.420684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424203+0000","last_change":"2023-10-10T18:45:11.419119+0000","last_active":"2023-10-10T18:45:11.424203+0000","last_peered":"2023-10-10T18:45:11.424203+0000","last_clean":"2023-10-10T18:45:11.424203+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T18:45:11.424203+0000","last_undegraded":"2023-10-10T18:45:11.424203+0000","last_fullsized":"2023-10-10T18:45:11.424203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425796+0000","last_change":"2023-10-10T18:45:11.421351+0000","last_active":"2023-10-10T18:45:11.425796+0000","last_peered":"2023-10-10T18:45:11.425796+0000","last_clean":"2023-10-10T18:45:11.425796+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T18:45:11.425796+0000","last_undegraded":"2023-10-10T18:45:11.425796+0000","last_fullsized":"2023-10-10T18:45:11.425796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425614+0000","last_change":"2023-10-10T18:45:11.421122+0000","last_active":"2023-10-10T18:45:11.425614+0000","last_peered":"2023-10-10T18:45:11.425614+0000","last_clean":"2023-10-10T18:45:11.425614+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T18:45:11.425614+0000","last_undegraded":"2023-10-10T18:45:11.425614+0000","last_fullsized":"2023-10-10T18:45:11.425614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424542+0000","last_change":"2023-10-10T18:45:11.421729+0000","last_active":"2023-10-10T18:45:11.424542+0000","last_peered":"2023-10-10T18:45:11.424542+0000","last_clean":"2023-10-10T18:45:11.424542+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T18:45:11.424542+0000","last_undegraded":"2023-10-10T18:45:11.424542+0000","last_fullsized":"2023-10-10T18:45:11.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426178+0000","last_change":"2023-10-10T18:45:11.419306+0000","last_active":"2023-10-10T18:45:11.426178+0000","last_peered":"2023-10-10T18:45:11.426178+0000","last_clean":"2023-10-10T18:45:11.426178+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T18:45:11.426178+0000","last_undegraded":"2023-10-10T18:45:11.426178+0000","last_fullsized":"2023-10-10T18:45:11.426178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.422940+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429946+0000","last_change":"2023-10-10T18:45:11.422612+0000","last_active":"2023-10-10T18:45:11.429946+0000","last_peered":"2023-10-10T18:45:11.429946+0000","last_clean":"2023-10-10T18:45:11.429946+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T18:45:11.429946+0000","last_undegraded":"2023-10-10T18:45:11.429946+0000","last_fullsized":"2023-10-10T18:45:11.429946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430763+0000","last_change":"2023-10-10T18:45:11.425754+0000","last_active":"2023-10-10T18:45:11.430763+0000","last_peered":"2023-10-10T18:45:11.430763+0000","last_clean":"2023-10-10T18:45:11.430763+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T18:45:11.430763+0000","last_undegraded":"2023-10-10T18:45:11.430763+0000","last_fullsized":"2023-10-10T18:45:11.430763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430664+0000","last_change":"2023-10-10T18:45:11.425891+0000","last_active":"2023-10-10T18:45:11.430664+0000","last_peered":"2023-10-10T18:45:11.430664+0000","last_clean":"2023-10-10T18:45:11.430664+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T18:45:11.430664+0000","last_undegraded":"2023-10-10T18:45:11.430664+0000","last_fullsized":"2023-10-10T18:45:11.430664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431015+0000","last_change":"2023-10-10T18:45:11.424834+0000","last_active":"2023-10-10T18:45:11.431015+0000","last_peered":"2023-10-10T18:45:11.431015+0000","last_clean":"2023-10-10T18:45:11.431015+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T18:45:11.431015+0000","last_undegraded":"2023-10-10T18:45:11.431015+0000","last_fullsized":"2023-10-10T18:45:11.431015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421085+0000","last_change":"2023-10-10T18:45:11.417796+0000","last_active":"2023-10-10T18:45:11.421085+0000","last_peered":"2023-10-10T18:45:11.421085+0000","last_clean":"2023-10-10T18:45:11.421085+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T18:45:11.421085+0000","last_undegraded":"2023-10-10T18:45:11.421085+0000","last_fullsized":"2023-10-10T18:45:11.421085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419270+0000","last_change":"2023-10-10T18:45:11.414690+0000","last_active":"2023-10-10T18:45:11.419270+0000","last_peered":"2023-10-10T18:45:11.419270+0000","last_clean":"2023-10-10T18:45:11.419270+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T18:45:11.419270+0000","last_undegraded":"2023-10-10T18:45:11.419270+0000","last_fullsized":"2023-10-10T18:45:11.419270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434526+0000","last_change":"2023-10-10T18:45:11.429433+0000","last_active":"2023-10-10T18:45:11.434526+0000","last_peered":"2023-10-10T18:45:11.434526+0000","last_clean":"2023-10-10T18:45:11.434526+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T18:45:11.434526+0000","last_undegraded":"2023-10-10T18:45:11.434526+0000","last_fullsized":"2023-10-10T18:45:11.434526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427193+0000","last_change":"2023-10-10T18:45:11.422935+0000","last_active":"2023-10-10T18:45:11.427193+0000","last_peered":"2023-10-10T18:45:11.427193+0000","last_clean":"2023-10-10T18:45:11.427193+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T18:45:11.427193+0000","last_undegraded":"2023-10-10T18:45:11.427193+0000","last_fullsized":"2023-10-10T18:45:11.427193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434131+0000","last_change":"2023-10-10T18:45:11.431202+0000","last_active":"2023-10-10T18:45:11.434131+0000","last_peered":"2023-10-10T18:45:11.434131+0000","last_clean":"2023-10-10T18:45:11.434131+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T18:45:11.434131+0000","last_undegraded":"2023-10-10T18:45:11.434131+0000","last_fullsized":"2023-10-10T18:45:11.434131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427139+0000","last_change":"2023-10-10T18:45:11.424885+0000","last_active":"2023-10-10T18:45:11.427139+0000","last_peered":"2023-10-10T18:45:11.427139+0000","last_clean":"2023-10-10T18:45:11.427139+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T18:45:11.427139+0000","last_undegraded":"2023-10-10T18:45:11.427139+0000","last_fullsized":"2023-10-10T18:45:11.427139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426206+0000","last_change":"2023-10-10T18:45:11.421039+0000","last_active":"2023-10-10T18:45:11.426206+0000","last_peered":"2023-10-10T18:45:11.426206+0000","last_clean":"2023-10-10T18:45:11.426206+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T18:45:11.426206+0000","last_undegraded":"2023-10-10T18:45:11.426206+0000","last_fullsized":"2023-10-10T18:45:11.426206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.554010+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T18:45:40.554010+0000","last_peered":"2023-10-10T18:45:40.554010+0000","last_clean":"2023-10-10T18:45:40.554010+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T18:45:40.554010+0000","last_undegraded":"2023-10-10T18:45:40.554010+0000","last_fullsized":"2023-10-10T18:45:40.554010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421043+0000","last_change":"2023-10-10T18:45:11.417504+0000","last_active":"2023-10-10T18:45:11.421043+0000","last_peered":"2023-10-10T18:45:11.421043+0000","last_clean":"2023-10-10T18:45:11.421043+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T18:45:11.421043+0000","last_undegraded":"2023-10-10T18:45:11.421043+0000","last_fullsized":"2023-10-10T18:45:11.421043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421448+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T18:45:16.421448+0000","last_peered":"2023-10-10T18:45:16.421448+0000","last_clean":"2023-10-10T18:45:16.421448+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T18:45:16.421448+0000","last_undegraded":"2023-10-10T18:45:16.421448+0000","last_fullsized":"2023-10-10T18:45:16.421448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424178+0000","last_change":"2023-10-10T18:45:11.421896+0000","last_active":"2023-10-10T18:45:11.424178+0000","last_peered":"2023-10-10T18:45:11.424178+0000","last_clean":"2023-10-10T18:45:11.424178+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T18:45:11.424178+0000","last_undegraded":"2023-10-10T18:45:11.424178+0000","last_fullsized":"2023-10-10T18:45:11.424178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552890+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T18:45:40.552890+0000","last_peered":"2023-10-10T18:45:40.552890+0000","last_clean":"2023-10-10T18:45:40.552890+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T18:45:40.552890+0000","last_undegraded":"2023-10-10T18:45:40.552890+0000","last_fullsized":"2023-10-10T18:45:40.552890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432466+0000","last_change":"2023-10-10T18:45:11.429680+0000","last_active":"2023-10-10T18:45:11.432466+0000","last_peered":"2023-10-10T18:45:11.432466+0000","last_clean":"2023-10-10T18:45:11.432466+0000","last_became_active":"2023-10-10T18:45:11.429397+0000","last_became_peered":"2023-10-10T18:45:11.429397+0000","last_unstale":"2023-10-10T18:45:11.432466+0000","last_undegraded":"2023-10-10T18:45:11.432466+0000","last_fullsized":"2023-10-10T18:45:11.432466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:19:52.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.551576+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T18:45:40.551576+0000","last_peered":"2023-10-10T18:45:40.551576+0000","last_clean":"2023-10-10T18:45:40.551576+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T18:45:40.551576+0000","last_undegraded":"2023-10-10T18:45:40.551576+0000","last_fullsized":"2023-10-10T18:45:40.551576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432439+0000","last_change":"2023-10-10T18:45:11.429408+0000","last_active":"2023-10-10T18:45:11.432439+0000","last_peered":"2023-10-10T18:45:11.432439+0000","last_clean":"2023-10-10T18:45:11.432439+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T18:45:11.432439+0000","last_undegraded":"2023-10-10T18:45:11.432439+0000","last_fullsized":"2023-10-10T18:45:11.432439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.457325+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T18:45:22.457325+0000","last_peered":"2023-10-10T18:45:22.457325+0000","last_clean":"2023-10-10T18:45:22.457325+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T18:45:22.457325+0000","last_undegraded":"2023-10-10T18:45:22.457325+0000","last_fullsized":"2023-10-10T18:45:22.457325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430579+0000","last_change":"2023-10-10T18:45:11.426051+0000","last_active":"2023-10-10T18:45:11.430579+0000","last_peered":"2023-10-10T18:45:11.430579+0000","last_clean":"2023-10-10T18:45:11.430579+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T18:45:11.430579+0000","last_undegraded":"2023-10-10T18:45:11.430579+0000","last_fullsized":"2023-10-10T18:45:11.430579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531752+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T18:45:34.531752+0000","last_peered":"2023-10-10T18:45:34.531752+0000","last_clean":"2023-10-10T18:45:34.531752+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T18:45:34.531752+0000","last_undegraded":"2023-10-10T18:45:34.531752+0000","last_fullsized":"2023-10-10T18:45:34.531752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427223+0000","last_change":"2023-10-10T18:45:11.425183+0000","last_active":"2023-10-10T18:45:11.427223+0000","last_peered":"2023-10-10T18:45:11.427223+0000","last_clean":"2023-10-10T18:45:11.427223+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T18:45:11.427223+0000","last_undegraded":"2023-10-10T18:45:11.427223+0000","last_fullsized":"2023-10-10T18:45:11.427223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.523628+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T18:45:28.523628+0000","last_peered":"2023-10-10T18:45:28.523628+0000","last_clean":"2023-10-10T18:45:28.523628+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T18:45:28.523628+0000","last_undegraded":"2023-10-10T18:45:28.523628+0000","last_fullsized":"2023-10-10T18:45:28.523628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415249+0000","last_change":"2023-10-10T18:45:11.412641+0000","last_active":"2023-10-10T18:45:11.415249+0000","last_peered":"2023-10-10T18:45:11.415249+0000","last_clean":"2023-10-10T18:45:11.415249+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T18:45:11.415249+0000","last_undegraded":"2023-10-10T18:45:11.415249+0000","last_fullsized":"2023-10-10T18:45:11.415249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531323+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T18:45:34.531323+0000","last_peered":"2023-10-10T18:45:34.531323+0000","last_clean":"2023-10-10T18:45:34.531323+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T18:45:34.531323+0000","last_undegraded":"2023-10-10T18:45:34.531323+0000","last_fullsized":"2023-10-10T18:45:34.531323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.420638+0000","last_change":"2023-10-10T18:45:11.413819+0000","last_active":"2023-10-10T18:45:11.420638+0000","last_peered":"2023-10-10T18:45:11.420638+0000","last_clean":"2023-10-10T18:45:11.420638+0000","last_became_active":"2023-10-10T18:45:11.413529+0000","last_became_peered":"2023-10-10T18:45:11.413529+0000","last_unstale":"2023-10-10T18:45:11.420638+0000","last_undegraded":"2023-10-10T18:45:11.420638+0000","last_fullsized":"2023-10-10T18:45:11.420638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.460216+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T18:45:22.460216+0000","last_peered":"2023-10-10T18:45:22.460216+0000","last_clean":"2023-10-10T18:45:22.460216+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T18:45:22.460216+0000","last_undegraded":"2023-10-10T18:45:22.460216+0000","last_fullsized":"2023-10-10T18:45:22.460216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430421+0000","last_change":"2023-10-10T18:45:11.427116+0000","last_active":"2023-10-10T18:45:11.430421+0000","last_peered":"2023-10-10T18:45:11.430421+0000","last_clean":"2023-10-10T18:45:11.430421+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T18:45:11.430421+0000","last_undegraded":"2023-10-10T18:45:11.430421+0000","last_fullsized":"2023-10-10T18:45:11.430421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552523+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T18:45:40.552523+0000","last_peered":"2023-10-10T18:45:40.552523+0000","last_clean":"2023-10-10T18:45:40.552523+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T18:45:40.552523+0000","last_undegraded":"2023-10-10T18:45:40.552523+0000","last_fullsized":"2023-10-10T18:45:40.552523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424158+0000","last_change":"2023-10-10T18:45:11.421794+0000","last_active":"2023-10-10T18:45:11.424158+0000","last_peered":"2023-10-10T18:45:11.424158+0000","last_clean":"2023-10-10T18:45:11.424158+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T18:45:11.424158+0000","last_undegraded":"2023-10-10T18:45:11.424158+0000","last_fullsized":"2023-10-10T18:45:11.424158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.459519+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T18:45:22.459519+0000","last_peered":"2023-10-10T18:45:22.459519+0000","last_clean":"2023-10-10T18:45:22.459519+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T18:45:22.459519+0000","last_undegraded":"2023-10-10T18:45:22.459519+0000","last_fullsized":"2023-10-10T18:45:22.459519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434442+0000","last_change":"2023-10-10T18:45:11.431857+0000","last_active":"2023-10-10T18:45:11.434442+0000","last_peered":"2023-10-10T18:45:11.434442+0000","last_clean":"2023-10-10T18:45:11.434442+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T18:45:11.434442+0000","last_undegraded":"2023-10-10T18:45:11.434442+0000","last_fullsized":"2023-10-10T18:45:11.434442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.553208+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T18:45:40.553208+0000","last_peered":"2023-10-10T18:45:40.553208+0000","last_clean":"2023-10-10T18:45:40.553208+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T18:45:40.553208+0000","last_undegraded":"2023-10-10T18:45:40.553208+0000","last_fullsized":"2023-10-10T18:45:40.553208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426196+0000","last_change":"2023-10-10T18:45:11.419774+0000","last_active":"2023-10-10T18:45:11.426196+0000","last_peered":"2023-10-10T18:45:11.426196+0000","last_clean":"2023-10-10T18:45:11.426196+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T18:45:11.426196+0000","last_undegraded":"2023-10-10T18:45:11.426196+0000","last_fullsized":"2023-10-10T18:45:11.426196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.551374+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T18:45:40.551374+0000","last_peered":"2023-10-10T18:45:40.551374+0000","last_clean":"2023-10-10T18:45:40.551374+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T18:45:40.551374+0000","last_undegraded":"2023-10-10T18:45:40.551374+0000","last_fullsized":"2023-10-10T18:45:40.551374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424245+0000","last_change":"2023-10-10T18:45:11.420517+0000","last_active":"2023-10-10T18:45:11.424245+0000","last_peered":"2023-10-10T18:45:11.424245+0000","last_clean":"2023-10-10T18:45:11.424245+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T18:45:11.424245+0000","last_undegraded":"2023-10-10T18:45:11.424245+0000","last_fullsized":"2023-10-10T18:45:11.424245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.527081+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T18:45:34.527081+0000","last_peered":"2023-10-10T18:45:34.527081+0000","last_clean":"2023-10-10T18:45:34.527081+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T18:45:34.527081+0000","last_undegraded":"2023-10-10T18:45:34.527081+0000","last_fullsized":"2023-10-10T18:45:34.527081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431834+0000","last_change":"2023-10-10T18:45:11.427109+0000","last_active":"2023-10-10T18:45:11.431834+0000","last_peered":"2023-10-10T18:45:11.431834+0000","last_clean":"2023-10-10T18:45:11.431834+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T18:45:11.431834+0000","last_undegraded":"2023-10-10T18:45:11.431834+0000","last_fullsized":"2023-10-10T18:45:11.431834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529442+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T18:45:34.529442+0000","last_peered":"2023-10-10T18:45:34.529442+0000","last_clean":"2023-10-10T18:45:34.529442+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T18:45:34.529442+0000","last_undegraded":"2023-10-10T18:45:34.529442+0000","last_fullsized":"2023-10-10T18:45:34.529442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430660+0000","last_change":"2023-10-10T18:45:11.416459+0000","last_active":"2023-10-10T18:45:11.430660+0000","last_peered":"2023-10-10T18:45:11.430660+0000","last_clean":"2023-10-10T18:45:11.430660+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T18:45:11.430660+0000","last_undegraded":"2023-10-10T18:45:11.430660+0000","last_fullsized":"2023-10-10T18:45:11.430660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.527437+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T18:45:34.527437+0000","last_peered":"2023-10-10T18:45:34.527437+0000","last_clean":"2023-10-10T18:45:34.527437+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T18:45:34.527437+0000","last_undegraded":"2023-10-10T18:45:34.527437+0000","last_fullsized":"2023-10-10T18:45:34.527437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432328+0000","last_change":"2023-10-10T18:45:11.429727+0000","last_active":"2023-10-10T18:45:11.432328+0000","last_peered":"2023-10-10T18:45:11.432328+0000","last_clean":"2023-10-10T18:45:11.432328+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T18:45:11.432328+0000","last_undegraded":"2023-10-10T18:45:11.432328+0000","last_fullsized":"2023-10-10T18:45:11.432328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:16.421569+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T18:45:16.421569+0000","last_peered":"2023-10-10T18:45:16.421569+0000","last_clean":"2023-10-10T18:45:16.421569+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T18:45:16.421569+0000","last_undegraded":"2023-10-10T18:45:16.421569+0000","last_fullsized":"2023-10-10T18:45:16.421569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431858+0000","last_change":"2023-10-10T18:45:11.427926+0000","last_active":"2023-10-10T18:45:11.431858+0000","last_peered":"2023-10-10T18:45:11.431858+0000","last_clean":"2023-10-10T18:45:11.431858+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T18:45:11.431858+0000","last_undegraded":"2023-10-10T18:45:11.431858+0000","last_fullsized":"2023-10-10T18:45:11.431858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:13.009459+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T18:45:13.009459+0000","last_peered":"2023-10-10T18:45:13.009459+0000","last_clean":"2023-10-10T18:45:13.009459+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T18:45:13.009459+0000","last_undegraded":"2023-10-10T18:45:13.009459+0000","last_fullsized":"2023-10-10T18:45:13.009459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427955+0000","last_change":"2023-10-10T18:45:11.423279+0000","last_active":"2023-10-10T18:45:11.427955+0000","last_peered":"2023-10-10T18:45:11.427955+0000","last_clean":"2023-10-10T18:45:11.427955+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T18:45:11.427955+0000","last_undegraded":"2023-10-10T18:45:11.427955+0000","last_fullsized":"2023-10-10T18:45:11.427955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.554631+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T18:45:40.554631+0000","last_peered":"2023-10-10T18:45:40.554631+0000","last_clean":"2023-10-10T18:45:40.554631+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T18:45:40.554631+0000","last_undegraded":"2023-10-10T18:45:40.554631+0000","last_fullsized":"2023-10-10T18:45:40.554631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430728+0000","last_change":"2023-10-10T18:45:11.425339+0000","last_active":"2023-10-10T18:45:11.430728+0000","last_peered":"2023-10-10T18:45:11.430728+0000","last_clean":"2023-10-10T18:45:11.430728+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T18:45:11.430728+0000","last_undegraded":"2023-10-10T18:45:11.430728+0000","last_fullsized":"2023-10-10T18:45:11.430728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421278+0000","last_change":"2023-10-10T18:45:11.418937+0000","last_active":"2023-10-10T18:45:11.421278+0000","last_peered":"2023-10-10T18:45:11.421278+0000","last_clean":"2023-10-10T18:45:11.421278+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T18:45:11.421278+0000","last_undegraded":"2023-10-10T18:45:11.421278+0000","last_fullsized":"2023-10-10T18:45:11.421278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.524806+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T18:45:28.524806+0000","last_peered":"2023-10-10T18:45:28.524806+0000","last_clean":"2023-10-10T18:45:28.524806+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T18:45:28.524806+0000","last_undegraded":"2023-10-10T18:45:28.524806+0000","last_fullsized":"2023-10-10T18:45:28.524806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424136+0000","last_change":"2023-10-10T18:45:11.422452+0000","last_active":"2023-10-10T18:45:11.424136+0000","last_peered":"2023-10-10T18:45:11.424136+0000","last_clean":"2023-10-10T18:45:11.424136+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T18:45:11.424136+0000","last_undegraded":"2023-10-10T18:45:11.424136+0000","last_fullsized":"2023-10-10T18:45:11.424136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529502+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T18:45:34.529502+0000","last_peered":"2023-10-10T18:45:34.529502+0000","last_clean":"2023-10-10T18:45:34.529502+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T18:45:34.529502+0000","last_undegraded":"2023-10-10T18:45:34.529502+0000","last_fullsized":"2023-10-10T18:45:34.529502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.405467+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T18:45:11.405467+0000","last_peered":"2023-10-10T18:45:11.405467+0000","last_clean":"2023-10-10T18:45:11.405467+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T18:45:11.405467+0000","last_undegraded":"2023-10-10T18:45:11.405467+0000","last_fullsized":"2023-10-10T18:45:11.405467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.434404+0000","last_change":"2023-10-10T18:45:11.430233+0000","last_active":"2023-10-10T18:45:11.434404+0000","last_peered":"2023-10-10T18:45:11.434404+0000","last_clean":"2023-10-10T18:45:11.434404+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T18:45:11.434404+0000","last_undegraded":"2023-10-10T18:45:11.434404+0000","last_fullsized":"2023-10-10T18:45:11.434404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:28.520084+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T18:45:28.520084+0000","last_peered":"2023-10-10T18:45:28.520084+0000","last_clean":"2023-10-10T18:45:28.520084+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T18:45:28.520084+0000","last_undegraded":"2023-10-10T18:45:28.520084+0000","last_fullsized":"2023-10-10T18:45:28.520084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531458+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T18:45:34.531458+0000","last_peered":"2023-10-10T18:45:34.531458+0000","last_clean":"2023-10-10T18:45:34.531458+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T18:45:34.531458+0000","last_undegraded":"2023-10-10T18:45:34.531458+0000","last_fullsized":"2023-10-10T18:45:34.531458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.421286+0000","last_change":"2023-10-10T18:45:11.418973+0000","last_active":"2023-10-10T18:45:11.421286+0000","last_peered":"2023-10-10T18:45:11.421286+0000","last_clean":"2023-10-10T18:45:11.421286+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T18:45:11.421286+0000","last_undegraded":"2023-10-10T18:45:11.421286+0000","last_fullsized":"2023-10-10T18:45:11.421286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.531687+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T18:45:34.531687+0000","last_peered":"2023-10-10T18:45:34.531687+0000","last_clean":"2023-10-10T18:45:34.531687+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T18:45:34.531687+0000","last_undegraded":"2023-10-10T18:45:34.531687+0000","last_fullsized":"2023-10-10T18:45:34.531687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423727+0000","last_change":"2023-10-10T18:45:11.419846+0000","last_active":"2023-10-10T18:45:11.423727+0000","last_peered":"2023-10-10T18:45:11.423727+0000","last_clean":"2023-10-10T18:45:11.423727+0000","last_became_active":"2023-10-10T18:45:11.419579+0000","last_became_peered":"2023-10-10T18:45:11.419579+0000","last_unstale":"2023-10-10T18:45:11.423727+0000","last_undegraded":"2023-10-10T18:45:11.423727+0000","last_fullsized":"2023-10-10T18:45:11.423727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:58:49.831558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552259+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T18:45:40.552259+0000","last_peered":"2023-10-10T18:45:40.552259+0000","last_clean":"2023-10-10T18:45:40.552259+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T18:45:40.552259+0000","last_undegraded":"2023-10-10T18:45:40.552259+0000","last_fullsized":"2023-10-10T18:45:40.552259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.423663+0000","last_change":"2023-10-10T18:45:11.416217+0000","last_active":"2023-10-10T18:45:11.423663+0000","last_peered":"2023-10-10T18:45:11.423663+0000","last_clean":"2023-10-10T18:45:11.423663+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T18:45:11.423663+0000","last_undegraded":"2023-10-10T18:45:11.423663+0000","last_fullsized":"2023-10-10T18:45:11.423663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.659888+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T18:45:22.659888+0000","last_peered":"2023-10-10T18:45:22.659888+0000","last_clean":"2023-10-10T18:45:22.659888+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T18:45:22.659888+0000","last_undegraded":"2023-10-10T18:45:22.659888+0000","last_fullsized":"2023-10-10T18:45:22.659888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.431602+0000","last_change":"2023-10-10T18:45:11.427826+0000","last_active":"2023-10-10T18:45:11.431602+0000","last_peered":"2023-10-10T18:45:11.431602+0000","last_clean":"2023-10-10T18:45:11.431602+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T18:45:11.431602+0000","last_undegraded":"2023-10-10T18:45:11.431602+0000","last_fullsized":"2023-10-10T18:45:11.431602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.552130+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T18:45:40.552130+0000","last_peered":"2023-10-10T18:45:40.552130+0000","last_clean":"2023-10-10T18:45:40.552130+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T18:45:40.552130+0000","last_undegraded":"2023-10-10T18:45:40.552130+0000","last_fullsized":"2023-10-10T18:45:40.552130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415388+0000","last_change":"2023-10-10T18:45:11.413975+0000","last_active":"2023-10-10T18:45:11.415388+0000","last_peered":"2023-10-10T18:45:11.415388+0000","last_clean":"2023-10-10T18:45:11.415388+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T18:45:11.415388+0000","last_undegraded":"2023-10-10T18:45:11.415388+0000","last_fullsized":"2023-10-10T18:45:11.415388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.529028+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T18:45:34.529028+0000","last_peered":"2023-10-10T18:45:34.529028+0000","last_clean":"2023-10-10T18:45:34.529028+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T18:45:34.529028+0000","last_undegraded":"2023-10-10T18:45:34.529028+0000","last_fullsized":"2023-10-10T18:45:34.529028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.415058+0000","last_change":"2023-10-10T18:45:11.410768+0000","last_active":"2023-10-10T18:45:11.415058+0000","last_peered":"2023-10-10T18:45:11.415058+0000","last_clean":"2023-10-10T18:45:11.415058+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T18:45:11.415058+0000","last_undegraded":"2023-10-10T18:45:11.415058+0000","last_fullsized":"2023-10-10T18:45:11.415058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:22.660373+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T18:45:22.660373+0000","last_peered":"2023-10-10T18:45:22.660373+0000","last_clean":"2023-10-10T18:45:22.660373+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T18:45:22.660373+0000","last_undegraded":"2023-10-10T18:45:22.660373+0000","last_fullsized":"2023-10-10T18:45:22.660373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430290+0000","last_change":"2023-10-10T18:45:11.425716+0000","last_active":"2023-10-10T18:45:11.430290+0000","last_peered":"2023-10-10T18:45:11.430290+0000","last_clean":"2023-10-10T18:45:11.430290+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T18:45:11.430290+0000","last_undegraded":"2023-10-10T18:45:11.430290+0000","last_fullsized":"2023-10-10T18:45:11.430290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.530979+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T18:45:34.530979+0000","last_peered":"2023-10-10T18:45:34.530979+0000","last_clean":"2023-10-10T18:45:34.530979+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T18:45:34.530979+0000","last_undegraded":"2023-10-10T18:45:34.530979+0000","last_fullsized":"2023-10-10T18:45:34.530979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429787+0000","last_change":"2023-10-10T18:45:11.423747+0000","last_active":"2023-10-10T18:45:11.429787+0000","last_peered":"2023-10-10T18:45:11.429787+0000","last_clean":"2023-10-10T18:45:11.429787+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T18:45:11.429787+0000","last_undegraded":"2023-10-10T18:45:11.429787+0000","last_fullsized":"2023-10-10T18:45:11.429787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:34.532321+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T18:45:34.532321+0000","last_peered":"2023-10-10T18:45:34.532321+0000","last_clean":"2023-10-10T18:45:34.532321+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T18:45:34.532321+0000","last_undegraded":"2023-10-10T18:45:34.532321+0000","last_fullsized":"2023-10-10T18:45:34.532321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.425816+0000","last_change":"2023-10-10T18:45:11.421872+0000","last_active":"2023-10-10T18:45:11.425816+0000","last_peered":"2023-10-10T18:45:11.425816+0000","last_clean":"2023-10-10T18:45:11.425816+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T18:45:11.425816+0000","last_undegraded":"2023-10-10T18:45:11.425816+0000","last_fullsized":"2023-10-10T18:45:11.425816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-10T18:45:40.550580+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T18:45:40.550580+0000","last_peered":"2023-10-10T18:45:40.550580+0000","last_clean":"2023-10-10T18:45:40.550580+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T18:45:40.550580+0000","last_undegraded":"2023-10-10T18:45:40.550580+0000","last_fullsized":"2023-10-10T18:45:40.550580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.429963+0000","last_change":"2023-10-10T18:45:11.425474+0000","last_active":"2023-10-10T18:45:11.429963+0000","last_peered":"2023-10-10T18:45:11.429963+0000","last_clean":"2023-10-10T18:45:11.429963+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T18:45:11.429963+0000","last_undegraded":"2023-10-10T18:45:11.429963+0000","last_fullsized":"2023-10-10T18:45:11.429963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430681+0000","last_change":"2023-10-10T18:45:11.416850+0000","last_active":"2023-10-10T18:45:11.430681+0000","last_peered":"2023-10-10T18:45:11.430681+0000","last_clean":"2023-10-10T18:45:11.430681+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T18:45:11.430681+0000","last_undegraded":"2023-10-10T18:45:11.430681+0000","last_fullsized":"2023-10-10T18:45:11.430681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427108+0000","last_change":"2023-10-10T18:45:11.424440+0000","last_active":"2023-10-10T18:45:11.427108+0000","last_peered":"2023-10-10T18:45:11.427108+0000","last_clean":"2023-10-10T18:45:11.427108+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T18:45:11.427108+0000","last_undegraded":"2023-10-10T18:45:11.427108+0000","last_fullsized":"2023-10-10T18:45:11.427108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424524+0000","last_change":"2023-10-10T18:45:11.421524+0000","last_active":"2023-10-10T18:45:11.424524+0000","last_peered":"2023-10-10T18:45:11.424524+0000","last_clean":"2023-10-10T18:45:11.424524+0000","last_became_active":"2023-10-10T18:45:11.420874+0000","last_became_peered":"2023-10-10T18:45:11.420874+0000","last_unstale":"2023-10-10T18:45:11.424524+0000","last_undegraded":"2023-10-10T18:45:11.424524+0000","last_fullsized":"2023-10-10T18:45:11.424524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:56:36.851801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.419177+0000","last_change":"2023-10-10T18:45:11.412399+0000","last_active":"2023-10-10T18:45:11.419177+0000","last_peered":"2023-10-10T18:45:11.419177+0000","last_clean":"2023-10-10T18:45:11.419177+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T18:45:11.419177+0000","last_undegraded":"2023-10-10T18:45:11.419177+0000","last_fullsized":"2023-10-10T18:45:11.419177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432996+0000","last_change":"2023-10-10T18:45:11.429507+0000","last_active":"2023-10-10T18:45:11.432996+0000","last_peered":"2023-10-10T18:45:11.432996+0000","last_clean":"2023-10-10T18:45:11.432996+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T18:45:11.432996+0000","last_undegraded":"2023-10-10T18:45:11.432996+0000","last_fullsized":"2023-10-10T18:45:11.432996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.418310+0000","last_change":"2023-10-10T18:45:11.413271+0000","last_active":"2023-10-10T18:45:11.418310+0000","last_peered":"2023-10-10T18:45:11.418310+0000","last_clean":"2023-10-10T18:45:11.418310+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T18:45:11.418310+0000","last_undegraded":"2023-10-10T18:45:11.418310+0000","last_fullsized":"2023-10-10T18:45:11.418310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430781+0000","last_change":"2023-10-10T18:45:11.427815+0000","last_active":"2023-10-10T18:45:11.430781+0000","last_peered":"2023-10-10T18:45:11.430781+0000","last_clean":"2023-10-10T18:45:11.430781+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T18:45:11.430781+0000","last_undegraded":"2023-10-10T18:45:11.430781+0000","last_fullsized":"2023-10-10T18:45:11.430781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.426972+0000","last_change":"2023-10-10T18:45:11.425458+0000","last_active":"2023-10-10T18:45:11.426972+0000","last_peered":"2023-10-10T18:45:11.426972+0000","last_clean":"2023-10-10T18:45:11.426972+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T18:45:11.426972+0000","last_undegraded":"2023-10-10T18:45:11.426972+0000","last_fullsized":"2023-10-10T18:45:11.426972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.424565+0000","last_change":"2023-10-10T18:45:11.422003+0000","last_active":"2023-10-10T18:45:11.424565+0000","last_peered":"2023-10-10T18:45:11.424565+0000","last_clean":"2023-10-10T18:45:11.424565+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T18:45:11.424565+0000","last_undegraded":"2023-10-10T18:45:11.424565+0000","last_fullsized":"2023-10-10T18:45:11.424565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432978+0000","last_change":"2023-10-10T18:45:11.428907+0000","last_active":"2023-10-10T18:45:11.432978+0000","last_peered":"2023-10-10T18:45:11.432978+0000","last_clean":"2023-10-10T18:45:11.432978+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T18:45:11.432978+0000","last_undegraded":"2023-10-10T18:45:11.432978+0000","last_fullsized":"2023-10-10T18:45:11.432978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.430538+0000","last_change":"2023-10-10T18:45:11.427241+0000","last_active":"2023-10-10T18:45:11.430538+0000","last_peered":"2023-10-10T18:45:11.430538+0000","last_clean":"2023-10-10T18:45:11.430538+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T18:45:11.430538+0000","last_undegraded":"2023-10-10T18:45:11.430538+0000","last_fullsized":"2023-10-10T18:45:11.430538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.427471+0000","last_change":"2023-10-10T18:45:11.426813+0000","last_active":"2023-10-10T18:45:11.427471+0000","last_peered":"2023-10-10T18:45:11.427471+0000","last_clean":"2023-10-10T18:45:11.427471+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T18:45:11.427471+0000","last_undegraded":"2023-10-10T18:45:11.427471+0000","last_fullsized":"2023-10-10T18:45:11.427471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-10T18:45:11.432098+0000","last_change":"2023-10-10T18:45:11.429582+0000","last_active":"2023-10-10T18:45:11.432098+0000","last_peered":"2023-10-10T18:45:11.432098+0000","last_clean":"2023-10-10T18:45:11.432098+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T18:45:11.432098+0000","last_undegraded":"2023-10-10T18:45:11.432098+0000","last_fullsized":"2023-10-10T18:45:11.432098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7626,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":15252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074240,"statfs":{"total":96636764160,"available":96332021760,"internally_reserved":0,"allocated":319488,"data_stored":667677,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":245760,"data_stored":77591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":245760,"data_stored":76769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":666527,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":249856,"data_stored":76699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":249856,"data_stored":77625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297524,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074316,"statfs":{"total":96636764160,"available":96332099584,"internally_reserved":0,"allocated":241664,"data_stored":76567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":79037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T18:45:54.060 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:54.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.209+0000 7f0d7a105700 1 Processor -- start 2023-10-10T18:45:54.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d7a105700 1 -- start start 2023-10-10T18:45:54.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d7a105700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741573d0 0x7f0d740c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d7a105700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0d741577a0 0x7f0d740c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d7a105700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d74154aa0 con 0x7f0d740bf6e0 2023-10-10T18:45:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d7a105700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d74063dc0 con 0x7f0d741577a0 2023-10-10T18:45:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d7a105700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d74157bf0 con 0x7f0d741573d0 2023-10-10T18:45:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d72ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0d741577a0 0x7f0d740c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d73fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f0d740bf6e0 0x7f0d740c6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51586/0 (socket says 172.21.15.175:51586) 2023-10-10T18:45:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.210+0000 7f0d73fff700 1 -- 172.21.15.175:0/3274565365 learned_addr learned my addr 172.21.15.175:0/3274565365 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:54.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d737fe700 1 --2- 172.21.15.175:0/3274565365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741573d0 0x7f0d740c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d72ffd700 1 -- 172.21.15.175:0/3274565365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741573d0 msgr2=0x7f0d740c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:54.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d72ffd700 1 --2- 172.21.15.175:0/3274565365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741573d0 0x7f0d740c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d72ffd700 1 -- 172.21.15.175:0/3274565365 >> v1:172.21.15.175:6789/0 conn(0x7f0d740bf6e0 legacy=0x7f0d740c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d727fc700 1 -- 172.21.15.175:0/3274565365 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293457730 0 0) 0x7f0d74154aa0 con 0x7f0d740bf6e0 2023-10-10T18:45:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d72ffd700 1 -- 172.21.15.175:0/3274565365 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d74157e50 con 0x7f0d741577a0 2023-10-10T18:45:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d72ffd700 1 --2- 172.21.15.175:0/3274565365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d741577a0 0x7f0d740c4f20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0d680098e0 tx=0x7f0d680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8513fb3b3158241a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d727fc700 1 -- 172.21.15.175:0/3274565365 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68009e80 con 0x7f0d741577a0 2023-10-10T18:45:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d727fc700 1 -- 172.21.15.175:0/3274565365 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6800b860 con 0x7f0d741577a0 2023-10-10T18:45:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d727fc700 1 -- 172.21.15.175:0/3274565365 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6800bbe0 con 0x7f0d741577a0 2023-10-10T18:45:54.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d7a105700 1 -- 172.21.15.175:0/3274565365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d741577a0 msgr2=0x7f0d740c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.211+0000 7f0d7a105700 1 --2- 172.21.15.175:0/3274565365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d741577a0 0x7f0d740c4f20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0d680098e0 tx=0x7f0d680091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.212+0000 7f0d7a105700 1 -- 172.21.15.175:0/3274565365 shutdown_connections 2023-10-10T18:45:54.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.212+0000 7f0d7a105700 1 --2- 172.21.15.175:0/3274565365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741573d0 0x7f0d740c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.212+0000 7f0d7a105700 1 --2- 172.21.15.175:0/3274565365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d741577a0 0x7f0d740c4f20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.212+0000 7f0d7a105700 1 -- 172.21.15.175:0/3274565365 >> 172.21.15.175:0/3274565365 conn(0x7f0d7406dc70 msgr2=0x7f0d74063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:54.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.212+0000 7f0d7a105700 1 -- 172.21.15.175:0/3274565365 shutdown_connections 2023-10-10T18:45:54.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.212+0000 7f0d7a105700 1 -- 172.21.15.175:0/3274565365 wait complete. 2023-10-10T18:45:54.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.213+0000 7f0d7a105700 1 Processor -- start 2023-10-10T18:45:54.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.213+0000 7f0d7a105700 1 -- start start 2023-10-10T18:45:54.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.213+0000 7f0d7a105700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d740bf6e0 0x7f0d740ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d7a105700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d741573d0 0x7f0d741660b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d7a105700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741577a0 0x7f0d74168e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d7a105700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d740c92b0 con 0x7f0d740bf6e0 2023-10-10T18:45:54.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d7a105700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d740c8f50 con 0x7f0d741573d0 2023-10-10T18:45:54.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d7a105700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d74165c20 con 0x7f0d741577a0 2023-10-10T18:45:54.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d740bf6e0 0x7f0d740ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d740bf6e0 0x7f0d740ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37902/0 (socket says 172.21.15.175:37902) 2023-10-10T18:45:54.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 -- 172.21.15.175:0/291820810 learned_addr learned my addr 172.21.15.175:0/291820810 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:54.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 -- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741577a0 msgr2=0x7f0d74168e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741577a0 0x7f0d74168e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 -- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d741573d0 msgr2=0x7f0d741660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d741573d0 0x7f0d741660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 -- 172.21.15.175:0/291820810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d68009420 con 0x7f0d740bf6e0 2023-10-10T18:45:54.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d737fe700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d740bf6e0 0x7f0d740ca5f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0d64007fc0 tx=0x7f0d640078b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d5b7fe700 1 -- 172.21.15.175:0/291820810 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64016430 con 0x7f0d740bf6e0 2023-10-10T18:45:54.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d5b7fe700 1 -- 172.21.15.175:0/291820810 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d64016590 con 0x7f0d740bf6e0 2023-10-10T18:45:54.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d5b7fe700 1 -- 172.21.15.175:0/291820810 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64016930 con 0x7f0d740bf6e0 2023-10-10T18:45:54.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d74153db0 con 0x7f0d740bf6e0 2023-10-10T18:45:54.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.214+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7406b050 con 0x7f0d740bf6e0 2023-10-10T18:45:54.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.215+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d74067570 con 0x7f0d740bf6e0 2023-10-10T18:45:54.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.215+0000 7f0d5b7fe700 1 -- 172.21.15.175:0/291820810 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0d64003020 con 0x7f0d740bf6e0 2023-10-10T18:45:54.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.216+0000 7f0d5b7fe700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d60066f30 0x7f0d600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.216+0000 7f0d5b7fe700 1 -- 172.21.15.175:0/291820810 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0d64012070 con 0x7f0d740bf6e0 2023-10-10T18:45:54.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.216+0000 7f0d72ffd700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d60066f30 0x7f0d600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.217+0000 7f0d72ffd700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d60066f30 0x7f0d600693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0d7406f970 tx=0x7f0d68017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.220+0000 7f0d5b7fe700 1 -- 172.21.15.175:0/291820810 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0d640526f0 con 0x7f0d740bf6e0 2023-10-10T18:45:54.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.405+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d74063ae0 con 0x7f0d740bf6e0 2023-10-10T18:45:54.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.406+0000 7f0d5b7fe700 1 -- 172.21.15.175:0/291820810 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7f0d64055d10 con 0x7f0d740bf6e0 2023-10-10T18:45:54.406 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:54.407 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:54.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d60066f30 msgr2=0x7f0d600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d60066f30 0x7f0d600693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0d7406f970 tx=0x7f0d68017000 comp rx=0 tx=0).stop 2023-10-10T18:45:54.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d740bf6e0 msgr2=0x7f0d740ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d740bf6e0 0x7f0d740ca5f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0d64007fc0 tx=0x7f0d640078b0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 shutdown_connections 2023-10-10T18:45:54.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d60066f30 0x7f0d600693e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d741577a0 0x7f0d74168e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d740bf6e0 0x7f0d740ca5f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 --2- 172.21.15.175:0/291820810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d741573d0 0x7f0d741660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 >> 172.21.15.175:0/291820810 conn(0x7f0d7406dc70 msgr2=0x7f0d74147c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:54.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 shutdown_connections 2023-10-10T18:45:54.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.408+0000 7f0d7a105700 1 -- 172.21.15.175:0/291820810 wait complete. 2023-10-10T18:45:54.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T18:45:05.161495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T18:45:08.855952+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T18:45:09.742056+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T18:45:54.421 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T18:45:54.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.539+0000 7f52ecafe700 1 Processor -- start 2023-10-10T18:45:54.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.539+0000 7f52ecafe700 1 -- start start 2023-10-10T18:45:54.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52ecafe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80633f0 0x7f52e8156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52ecafe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52e80637c0 0x7f52e814f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52ecafe700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e80c7ca0 con 0x7f52e80c7620 2023-10-10T18:45:54.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52ecafe700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e80af540 con 0x7f52e80637c0 2023-10-10T18:45:54.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52ecafe700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e8156830 con 0x7f52e80633f0 2023-10-10T18:45:54.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e6d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52e80637c0 0x7f52e814f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e5d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f52e80c7620 0x7f52e80aff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51600/0 (socket says 172.21.15.175:51600) 2023-10-10T18:45:54.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e5d9b700 1 -- 172.21.15.175:0/1677596366 learned_addr learned my addr 172.21.15.175:0/1677596366 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:54.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e659c700 1 --2- 172.21.15.175:0/1677596366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80633f0 0x7f52e8156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e6d9d700 1 -- 172.21.15.175:0/1677596366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80633f0 msgr2=0x7f52e8156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:54.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e6d9d700 1 --2- 172.21.15.175:0/1677596366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80633f0 0x7f52e8156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e6d9d700 1 -- 172.21.15.175:0/1677596366 >> v1:172.21.15.175:6789/0 conn(0x7f52e80c7620 legacy=0x7f52e80aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e6d9d700 1 -- 172.21.15.175:0/1677596366 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e8156ad0 con 0x7f52e80637c0 2023-10-10T18:45:54.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e6d9d700 1 --2- 172.21.15.175:0/1677596366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80637c0 0x7f52e814f7c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f52d80098e0 tx=0x7f52d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4218183a6f5df928 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e559a700 1 -- 172.21.15.175:0/1677596366 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d8009e80 con 0x7f52e80637c0 2023-10-10T18:45:54.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.540+0000 7f52e559a700 1 -- 172.21.15.175:0/1677596366 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d800b8f0 con 0x7f52e80637c0 2023-10-10T18:45:54.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52e559a700 1 -- 172.21.15.175:0/1677596366 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d800bc70 con 0x7f52e80637c0 2023-10-10T18:45:54.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52ecafe700 1 -- 172.21.15.175:0/1677596366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80637c0 msgr2=0x7f52e814f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1677596366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80637c0 0x7f52e814f7c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f52d80098e0 tx=0x7f52d80091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52ecafe700 1 -- 172.21.15.175:0/1677596366 shutdown_connections 2023-10-10T18:45:54.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1677596366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80637c0 0x7f52e814f7c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1677596366 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80633f0 0x7f52e8156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52ecafe700 1 -- 172.21.15.175:0/1677596366 >> 172.21.15.175:0/1677596366 conn(0x7f52e80cc7a0 msgr2=0x7f52e80ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:54.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52ecafe700 1 -- 172.21.15.175:0/1677596366 shutdown_connections 2023-10-10T18:45:54.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.541+0000 7f52ecafe700 1 -- 172.21.15.175:0/1677596366 wait complete. 2023-10-10T18:45:54.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.542+0000 7f52ecafe700 1 Processor -- start 2023-10-10T18:45:54.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.542+0000 7f52ecafe700 1 -- start start 2023-10-10T18:45:54.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52ecafe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80633f0 0x7f52e8166140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52ecafe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52e80637c0 0x7f52e81523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52ecafe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80c7620 0x7f52e8152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52ecafe700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e8151f50 con 0x7f52e80633f0 2023-10-10T18:45:54.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52ecafe700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e8151b60 con 0x7f52e80637c0 2023-10-10T18:45:54.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52ecafe700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e8150fe0 con 0x7f52e80c7620 2023-10-10T18:45:54.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80633f0 0x7f52e8166140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80633f0 0x7f52e8166140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37916/0 (socket says 172.21.15.175:37916) 2023-10-10T18:45:54.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 -- 172.21.15.175:0/1172686535 learned_addr learned my addr 172.21.15.175:0/1172686535 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:54.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e6d9d700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80c7620 0x7f52e8152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e5d9b700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52e80637c0 0x7f52e81523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 -- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80c7620 msgr2=0x7f52e8152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80c7620 0x7f52e8152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 -- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52e80637c0 msgr2=0x7f52e81523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52e80637c0 0x7f52e81523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 -- 172.21.15.175:0/1172686535 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d8009420 con 0x7f52e80633f0 2023-10-10T18:45:54.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52e659c700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80633f0 0x7f52e8166140 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f52e000ce10 tx=0x7f52e0007810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52d67fc700 1 -- 172.21.15.175:0/1172686535 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e000ab90 con 0x7f52e80633f0 2023-10-10T18:45:54.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.543+0000 7f52d67fc700 1 -- 172.21.15.175:0/1172686535 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e000acf0 con 0x7f52e80633f0 2023-10-10T18:45:54.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.544+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e81512c0 con 0x7f52e80633f0 2023-10-10T18:45:54.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.544+0000 7f52d67fc700 1 -- 172.21.15.175:0/1172686535 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0017680 con 0x7f52e80633f0 2023-10-10T18:45:54.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.544+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e8151880 con 0x7f52e80633f0 2023-10-10T18:45:54.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.545+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e814def0 con 0x7f52e80633f0 2023-10-10T18:45:54.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.545+0000 7f52d67fc700 1 -- 172.21.15.175:0/1172686535 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f52e0003020 con 0x7f52e80633f0 2023-10-10T18:45:54.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.545+0000 7f52d67fc700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f52c8066f30 0x7f52c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.545+0000 7f52d67fc700 1 -- 172.21.15.175:0/1172686535 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f52e0013070 con 0x7f52e80633f0 2023-10-10T18:45:54.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.545+0000 7f52e5d9b700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f52c8066f30 0x7f52c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.546+0000 7f52e5d9b700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f52c8066f30 0x7f52c80693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f52e80c7250 tx=0x7f52dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.549+0000 7f52d67fc700 1 -- 172.21.15.175:0/1172686535 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f52e0051d00 con 0x7f52e80633f0 2023-10-10T18:45:54.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.732+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52e8067110 con 0x7f52e80633f0 2023-10-10T18:45:54.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.733+0000 7f52d67fc700 1 -- 172.21.15.175:0/1172686535 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52e001c070 con 0x7f52e80633f0 2023-10-10T18:45:54.733 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 1 2023-10-10T18:45:54.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.734+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f52c8066f30 msgr2=0x7f52c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.734+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f52c8066f30 0x7f52c80693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f52e80c7250 tx=0x7f52dc009040 comp rx=0 tx=0).stop 2023-10-10T18:45:54.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80633f0 msgr2=0x7f52e8166140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80633f0 0x7f52e8166140 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f52e000ce10 tx=0x7f52e0007810 comp rx=0 tx=0).stop 2023-10-10T18:45:54.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 shutdown_connections 2023-10-10T18:45:54.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f52c8066f30 0x7f52c80693e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52e80c7620 0x7f52e8152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52e80633f0 0x7f52e8166140 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 --2- 172.21.15.175:0/1172686535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52e80637c0 0x7f52e81523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 >> 172.21.15.175:0/1172686535 conn(0x7f52e80cc7a0 msgr2=0x7f52e815be00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:54.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 shutdown_connections 2023-10-10T18:45:54.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.735+0000 7f52ecafe700 1 -- 172.21.15.175:0/1172686535 wait complete. 2023-10-10T18:45:54.746 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T18:45:54.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.865+0000 7f3640976700 1 Processor -- start 2023-10-10T18:45:54.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.865+0000 7f3640976700 1 -- start start 2023-10-10T18:45:54.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.866+0000 7f3640976700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f363c158e40 0x7f363c155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.866+0000 7f3640976700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c1567c0 con 0x7f363c158a70 2023-10-10T18:45:54.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.866+0000 7f3640976700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c156aa0 con 0x7f363c1562c0 2023-10-10T18:45:54.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.866+0000 7f3640976700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363c069510 con 0x7f363c158e40 2023-10-10T18:45:54.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.866+0000 7f3639d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f363c158e40 0x7f363c155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.866+0000 7f363a59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f363c158a70 0x7f363c068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41978/0 (socket says 172.21.15.175:41978) 2023-10-10T18:45:54.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.866+0000 7f363a59c700 1 -- 172.21.15.175:0/4123373229 learned_addr learned my addr 172.21.15.175:0/4123373229 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:54.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.866+0000 7f3639d9b700 1 -- 172.21.15.175:0/4123373229 >> v1:172.21.15.191:6789/0 conn(0x7f363c1562c0 legacy=0x7f363c0c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:54.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3639d9b700 1 -- 172.21.15.175:0/4123373229 >> v1:172.21.15.175:6789/0 conn(0x7f363c158a70 legacy=0x7f363c068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3639d9b700 1 -- 172.21.15.175:0/4123373229 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c068320 con 0x7f363c158e40 2023-10-10T18:45:54.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f363959a700 1 -- 172.21.15.175:0/4123373229 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796203847 0 0) 0x7f363c1567c0 con 0x7f363c158a70 2023-10-10T18:45:54.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3639d9b700 1 --2- 172.21.15.175:0/4123373229 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c155d80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f36240098e0 tx=0x7f36240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0d481853bed0bd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f363959a700 1 -- 172.21.15.175:0/4123373229 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624009e80 con 0x7f363c158e40 2023-10-10T18:45:54.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f363959a700 1 -- 172.21.15.175:0/4123373229 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362400b8b0 con 0x7f363c158e40 2023-10-10T18:45:54.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f363959a700 1 -- 172.21.15.175:0/4123373229 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362400bc30 con 0x7f363c158e40 2023-10-10T18:45:54.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3640976700 1 -- 172.21.15.175:0/4123373229 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 msgr2=0x7f363c155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3640976700 1 --2- 172.21.15.175:0/4123373229 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c155d80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f36240098e0 tx=0x7f36240091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3640976700 1 -- 172.21.15.175:0/4123373229 shutdown_connections 2023-10-10T18:45:54.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3640976700 1 --2- 172.21.15.175:0/4123373229 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c155d80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3640976700 1 -- 172.21.15.175:0/4123373229 >> 172.21.15.175:0/4123373229 conn(0x7f363c063230 msgr2=0x7f363c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:54.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3640976700 1 -- 172.21.15.175:0/4123373229 shutdown_connections 2023-10-10T18:45:54.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.867+0000 7f3640976700 1 -- 172.21.15.175:0/4123373229 wait complete. 2023-10-10T18:45:54.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.868+0000 7f3640976700 1 Processor -- start 2023-10-10T18:45:54.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.868+0000 7f3640976700 1 -- start start 2023-10-10T18:45:54.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f3640976700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f363c1562c0 0x7f363c169190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f3640976700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363c158a70 0x7f363c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f3640976700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f3640976700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c1696d0 con 0x7f363c158e40 2023-10-10T18:45:54.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f3640976700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0cc450 con 0x7f363c1562c0 2023-10-10T18:45:54.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f3640976700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0cb4c0 con 0x7f363c158a70 2023-10-10T18:45:54.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f363ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f363ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c154b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57846/0 (socket says 172.21.15.175:57846) 2023-10-10T18:45:54.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f363ad9d700 1 -- 172.21.15.175:0/2078960301 learned_addr learned my addr 172.21.15.175:0/2078960301 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:54.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f363a59c700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f363c1562c0 0x7f363c169190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.869+0000 7f3639d9b700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363c158a70 0x7f363c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f363ad9d700 1 -- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363c158a70 msgr2=0x7f363c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f363ad9d700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363c158a70 0x7f363c0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f363ad9d700 1 -- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f363c1562c0 msgr2=0x7f363c169190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:54.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f363ad9d700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f363c1562c0 0x7f363c169190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:54.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f363ad9d700 1 -- 172.21.15.175:0/2078960301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624009420 con 0x7f363c158e40 2023-10-10T18:45:54.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f3639d9b700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363c158a70 0x7f363c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:54.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f363a59c700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f363c1562c0 0x7f363c169190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:54.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f363ad9d700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c154b30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f363400b510 tx=0x7f3634006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f36327fc700 1 -- 172.21.15.175:0/2078960301 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363400f510 con 0x7f363c158e40 2023-10-10T18:45:54.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f36327fc700 1 -- 172.21.15.175:0/2078960301 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363400f690 con 0x7f363c158e40 2023-10-10T18:45:54.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c0cba80 con 0x7f363c158e40 2023-10-10T18:45:54.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f36327fc700 1 -- 172.21.15.175:0/2078960301 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363400fa50 con 0x7f363c158e40 2023-10-10T18:45:54.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.870+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c169c50 con 0x7f363c158e40 2023-10-10T18:45:54.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.871+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363c067570 con 0x7f363c158e40 2023-10-10T18:45:54.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.871+0000 7f36327fc700 1 -- 172.21.15.175:0/2078960301 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f363401a400 con 0x7f363c158e40 2023-10-10T18:45:54.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.871+0000 7f36327fc700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f361c066f60 0x7f361c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:54.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.872+0000 7f36327fc700 1 -- 172.21.15.175:0/2078960301 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3634083f20 con 0x7f363c158e40 2023-10-10T18:45:54.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.872+0000 7f363a59c700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f361c066f60 0x7f361c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:54.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.872+0000 7f363a59c700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f361c066f60 0x7f361c069410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f362c001560 tx=0x7f362c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:54.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:54.876+0000 7f36327fc700 1 -- 172.21.15.175:0/2078960301 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3634051210 con 0x7f363c158e40 2023-10-10T18:45:55.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.056+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f363c169f70 con 0x7f363c158e40 2023-10-10T18:45:55.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.056+0000 7f36327fc700 1 -- 172.21.15.175:0/2078960301 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3634058e60 con 0x7f363c158e40 2023-10-10T18:45:55.057 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 32 2023-10-10T18:45:55.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f361c066f60 msgr2=0x7f361c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f361c066f60 0x7f361c069410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f362c001560 tx=0x7f362c009040 comp rx=0 tx=0).stop 2023-10-10T18:45:55.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 msgr2=0x7f363c154b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c154b30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f363400b510 tx=0x7f3634006f90 comp rx=0 tx=0).stop 2023-10-10T18:45:55.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 shutdown_connections 2023-10-10T18:45:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f361c066f60 0x7f361c069410 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f363c158e40 0x7f363c154b30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f363c1562c0 0x7f363c169190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 --2- 172.21.15.175:0/2078960301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363c158a70 0x7f363c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.058+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 >> 172.21.15.175:0/2078960301 conn(0x7f363c063230 msgr2=0x7f363c14fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:55.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.059+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 shutdown_connections 2023-10-10T18:45:55.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.059+0000 7f3640976700 1 -- 172.21.15.175:0/2078960301 wait complete. 2023-10-10T18:45:55.070 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T18:45:55.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.188+0000 7f6c41709700 1 Processor -- start 2023-10-10T18:45:55.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.188+0000 7f6c41709700 1 -- start start 2023-10-10T18:45:55.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.188+0000 7f6c41709700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6c3c0633f0 0x7f6c3c156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.188+0000 7f6c41709700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0c7620 0x7f6c3c0aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.188+0000 7f6c41709700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0c7ca0 con 0x7f6c3c0b0490 2023-10-10T18:45:55.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.188+0000 7f6c41709700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c0af540 con 0x7f6c3c0633f0 2023-10-10T18:45:55.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.188+0000 7f6c41709700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c156830 con 0x7f6c3c0c7620 2023-10-10T18:45:55.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.188+0000 7f6c3affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6c3c0633f0 0x7f6c3c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c3affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6c3c0633f0 0x7f6c3c156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57850/0 (socket says 172.21.15.175:57850) 2023-10-10T18:45:55.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c3affd700 1 -- 172.21.15.175:0/828746381 learned_addr learned my addr 172.21.15.175:0/828746381 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:55.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c3a7fc700 1 --2- 172.21.15.175:0/828746381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0c7620 0x7f6c3c0aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c3affd700 1 -- 172.21.15.175:0/828746381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0c7620 msgr2=0x7f6c3c0aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:55.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c3affd700 1 --2- 172.21.15.175:0/828746381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0c7620 0x7f6c3c0aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c3affd700 1 -- 172.21.15.175:0/828746381 >> v1:172.21.15.191:6789/0 conn(0x7f6c3c0b0490 legacy=0x7f6c3c159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c3affd700 1 -- 172.21.15.175:0/828746381 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c156ad0 con 0x7f6c3c0633f0 2023-10-10T18:45:55.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c3affd700 1 --2- 172.21.15.175:0/828746381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0633f0 0x7f6c3c156010 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009f20 tx=0x7f6c2c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5665973075bce4bd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c39ffb700 1 -- 172.21.15.175:0/828746381 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c00e070 con 0x7f6c3c0633f0 2023-10-10T18:45:55.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c39ffb700 1 -- 172.21.15.175:0/828746381 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c00bb10 con 0x7f6c3c0633f0 2023-10-10T18:45:55.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.189+0000 7f6c39ffb700 1 -- 172.21.15.175:0/828746381 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c0092e0 con 0x7f6c3c0633f0 2023-10-10T18:45:55.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.190+0000 7f6c41709700 1 -- 172.21.15.175:0/828746381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0633f0 msgr2=0x7f6c3c156010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.190+0000 7f6c41709700 1 --2- 172.21.15.175:0/828746381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0633f0 0x7f6c3c156010 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009f20 tx=0x7f6c2c009840 comp rx=0 tx=0).stop 2023-10-10T18:45:55.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.190+0000 7f6c41709700 1 -- 172.21.15.175:0/828746381 shutdown_connections 2023-10-10T18:45:55.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.190+0000 7f6c41709700 1 --2- 172.21.15.175:0/828746381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0633f0 0x7f6c3c156010 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.190+0000 7f6c41709700 1 --2- 172.21.15.175:0/828746381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0c7620 0x7f6c3c0aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.190+0000 7f6c41709700 1 -- 172.21.15.175:0/828746381 >> 172.21.15.175:0/828746381 conn(0x7f6c3c0cc7a0 msgr2=0x7f6c3c0ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:55.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.190+0000 7f6c41709700 1 -- 172.21.15.175:0/828746381 shutdown_connections 2023-10-10T18:45:55.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.190+0000 7f6c41709700 1 -- 172.21.15.175:0/828746381 wait complete. 2023-10-10T18:45:55.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.191+0000 7f6c41709700 1 Processor -- start 2023-10-10T18:45:55.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.191+0000 7f6c41709700 1 -- start start 2023-10-10T18:45:55.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c41709700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0633f0 0x7f6c3c14a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c41709700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0b0490 0x7f6c3c159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c41709700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c3c0c7620 0x7f6c3c159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c41709700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c151660 con 0x7f6c3c0b0490 2023-10-10T18:45:55.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c41709700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c150ab0 con 0x7f6c3c0c7620 2023-10-10T18:45:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c41709700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c150d50 con 0x7f6c3c0633f0 2023-10-10T18:45:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0b0490 0x7f6c3c159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0b0490 0x7f6c3c159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57866/0 (socket says 172.21.15.175:57866) 2023-10-10T18:45:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 -- 172.21.15.175:0/3983944182 learned_addr learned my addr 172.21.15.175:0/3983944182 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3affd700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0633f0 0x7f6c3c14a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3b7fe700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c3c0c7620 0x7f6c3c159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 -- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0633f0 msgr2=0x7f6c3c14a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0633f0 0x7f6c3c14a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 -- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c3c0c7620 msgr2=0x7f6c3c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c3c0c7620 0x7f6c3c159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 -- 172.21.15.175:0/3983944182 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c009ae0 con 0x7f6c3c0b0490 2023-10-10T18:45:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3b7fe700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c3c0c7620 0x7f6c3c159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3affd700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0633f0 0x7f6c3c14a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c3a7fc700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0b0490 0x7f6c3c159540 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400d890 tx=0x7f6c24008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c32ffd700 1 -- 172.21.15.175:0/3983944182 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2400b6c0 con 0x7f6c3c0b0490 2023-10-10T18:45:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c32ffd700 1 -- 172.21.15.175:0/3983944182 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2400b840 con 0x7f6c3c0b0490 2023-10-10T18:45:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c151010 con 0x7f6c3c0b0490 2023-10-10T18:45:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c32ffd700 1 -- 172.21.15.175:0/3983944182 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2400bc00 con 0x7f6c3c0b0490 2023-10-10T18:45:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.192+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c0c9010 con 0x7f6c3c0b0490 2023-10-10T18:45:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.193+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c067110 con 0x7f6c3c0b0490 2023-10-10T18:45:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.194+0000 7f6c32ffd700 1 -- 172.21.15.175:0/3983944182 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6c2401e460 con 0x7f6c3c0b0490 2023-10-10T18:45:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.194+0000 7f6c32ffd700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6c1c066f60 0x7f6c1c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.194+0000 7f6c32ffd700 1 -- 172.21.15.175:0/3983944182 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6c24084c90 con 0x7f6c3c0b0490 2023-10-10T18:45:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.194+0000 7f6c3affd700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6c1c066f60 0x7f6c1c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.195+0000 7f6c3affd700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6c1c066f60 0x7f6c1c069410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009880 tx=0x7f6c2c018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.198+0000 7f6c32ffd700 1 -- 172.21.15.175:0/3983944182 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6c24051f00 con 0x7f6c3c0b0490 2023-10-10T18:45:55.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.379+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c3c067570 con 0x7f6c3c0b0490 2023-10-10T18:45:55.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.380+0000 7f6c32ffd700 1 -- 172.21.15.175:0/3983944182 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c24055520 con 0x7f6c3c0b0490 2023-10-10T18:45:55.380 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 64 2023-10-10T18:45:55.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6c1c066f60 msgr2=0x7f6c1c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6c1c066f60 0x7f6c1c069410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009880 tx=0x7f6c2c018000 comp rx=0 tx=0).stop 2023-10-10T18:45:55.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0b0490 msgr2=0x7f6c3c159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0b0490 0x7f6c3c159540 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400d890 tx=0x7f6c24008040 comp rx=0 tx=0).stop 2023-10-10T18:45:55.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 shutdown_connections 2023-10-10T18:45:55.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6c1c066f60 0x7f6c1c069410 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c3c0c7620 0x7f6c3c159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c3c0633f0 0x7f6c3c14a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 --2- 172.21.15.175:0/3983944182 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6c3c0b0490 0x7f6c3c159540 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 >> 172.21.15.175:0/3983944182 conn(0x7f6c3c0cc7a0 msgr2=0x7f6c3c0c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:55.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 shutdown_connections 2023-10-10T18:45:55.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.382+0000 7f6c41709700 1 -- 172.21.15.175:0/3983944182 wait complete. 2023-10-10T18:45:55.394 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:45:55.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.508+0000 7fde332f0700 1 Processor -- start 2023-10-10T18:45:55.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.508+0000 7fde332f0700 1 -- start start 2023-10-10T18:45:55.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde332f0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde332f0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fde2c0aff60 0x7fde2c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde332f0700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c1574b0 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde332f0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2c150770 con 0x7fde2c0aff60 2023-10-10T18:45:55.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde332f0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0afad0 con 0x7fde2c0b2390 2023-10-10T18:45:55.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3188d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fde2c0aff60 0x7fde2c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3108c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fde2c0b1fc0 0x7fde2c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41990/0 (socket says 172.21.15.175:41990) 2023-10-10T18:45:55.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3108c700 1 -- 172.21.15.175:0/3629230434 learned_addr learned my addr 172.21.15.175:0/3629230434 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:45:55.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3088b700 1 --2- 172.21.15.175:0/3629230434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3188d700 1 -- 172.21.15.175:0/3629230434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 msgr2=0x7fde2c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:55.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3188d700 1 --2- 172.21.15.175:0/3629230434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3188d700 1 -- 172.21.15.175:0/3629230434 >> v1:172.21.15.175:6789/0 conn(0x7fde2c0b1fc0 legacy=0x7fde2c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde23fff700 1 -- 172.21.15.175:0/3629230434 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432265608 0 0) 0x7fde2c1574b0 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3188d700 1 -- 172.21.15.175:0/3629230434 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0af4c0 con 0x7fde2c0aff60 2023-10-10T18:45:55.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.509+0000 7fde3088b700 1 --2- 172.21.15.175:0/3629230434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:45:55.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde3188d700 1 --2- 172.21.15.175:0/3629230434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0aff60 0x7fde2c0b03e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fde24009910 tx=0x7fde240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432f4f84215ed3ab server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde23fff700 1 -- 172.21.15.175:0/3629230434 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2400b8b0 con 0x7fde2c0aff60 2023-10-10T18:45:55.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde23fff700 1 -- 172.21.15.175:0/3629230434 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2400ba10 con 0x7fde2c0aff60 2023-10-10T18:45:55.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde23fff700 1 -- 172.21.15.175:0/3629230434 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2400bdb0 con 0x7fde2c0aff60 2023-10-10T18:45:55.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde332f0700 1 -- 172.21.15.175:0/3629230434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0aff60 msgr2=0x7fde2c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde332f0700 1 --2- 172.21.15.175:0/3629230434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0aff60 0x7fde2c0b03e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fde24009910 tx=0x7fde240091a0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde332f0700 1 -- 172.21.15.175:0/3629230434 shutdown_connections 2023-10-10T18:45:55.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde332f0700 1 --2- 172.21.15.175:0/3629230434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0aff60 0x7fde2c0b03e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde332f0700 1 --2- 172.21.15.175:0/3629230434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde332f0700 1 -- 172.21.15.175:0/3629230434 >> 172.21.15.175:0/3629230434 conn(0x7fde2c063230 msgr2=0x7fde2c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:55.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde332f0700 1 -- 172.21.15.175:0/3629230434 shutdown_connections 2023-10-10T18:45:55.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.510+0000 7fde332f0700 1 -- 172.21.15.175:0/3629230434 wait complete. 2023-10-10T18:45:55.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.511+0000 7fde332f0700 1 Processor -- start 2023-10-10T18:45:55.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde332f0700 1 -- start start 2023-10-10T18:45:55.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde332f0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde2c0aff60 0x7fde2c0c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde332f0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0b1fc0 0x7fde2c169340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde332f0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c169880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde332f0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0c6af0 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde332f0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c15a280 con 0x7fde2c0aff60 2023-10-10T18:45:55.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde332f0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0c3e40 con 0x7fde2c0b2390 2023-10-10T18:45:55.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde3088b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0b1fc0 0x7fde2c169340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde3088b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0b1fc0 0x7fde2c169340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57894/0 (socket says 172.21.15.175:57894) 2023-10-10T18:45:55.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.512+0000 7fde3088b700 1 -- 172.21.15.175:0/2439056761 learned_addr learned my addr 172.21.15.175:0/2439056761 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:55.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde3108c700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde2c0aff60 0x7fde2c0c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde3188d700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c169880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde3088b700 1 -- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 msgr2=0x7fde2c169880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde3088b700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c169880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde3088b700 1 -- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde2c0aff60 msgr2=0x7fde2c0c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde3088b700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde2c0aff60 0x7fde2c0c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde3088b700 1 -- 172.21.15.175:0/2439056761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde24009420 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde3088b700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0b1fc0 0x7fde2c169340 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00d860 tx=0x7fde1c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde217fa700 1 -- 172.21.15.175:0/2439056761 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c0125f0 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde217fa700 1 -- 172.21.15.175:0/2439056761 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1c012750 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde217fa700 1 -- 172.21.15.175:0/2439056761 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c012af0 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c0c57f0 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.513+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c0c5db0 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.514+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2c067570 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.515+0000 7fde217fa700 1 -- 172.21.15.175:0/2439056761 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fde1c014430 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.515+0000 7fde217fa700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde10066f30 0x7fde100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.515+0000 7fde217fa700 1 -- 172.21.15.175:0/2439056761 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fde1c0847e0 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.515+0000 7fde3108c700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde10066f30 0x7fde100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.515+0000 7fde3108c700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde10066f30 0x7fde100693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fde28002250 tx=0x7fde28001a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.519+0000 7fde217fa700 1 -- 172.21.15.175:0/2439056761 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde1c051a50 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.744+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde2c0c4120 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.745+0000 7fde217fa700 1 -- 172.21.15.175:0/2439056761 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6176 (secure 0 0 0) 0x7fde1c055070 con 0x7fde2c0b1fc0 2023-10-10T18:45:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:55.746 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:45:41.554584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:45:55.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde10066f30 msgr2=0x7fde100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde10066f30 0x7fde100693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fde28002250 tx=0x7fde28001a20 comp rx=0 tx=0).stop 2023-10-10T18:45:55.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0b1fc0 msgr2=0x7fde2c169340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0b1fc0 0x7fde2c169340 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00d860 tx=0x7fde1c008040 comp rx=0 tx=0).stop 2023-10-10T18:45:55.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 shutdown_connections 2023-10-10T18:45:55.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde10066f30 0x7fde100693e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde2c0b2390 0x7fde2c169880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde2c0aff60 0x7fde2c0c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 --2- 172.21.15.175:0/2439056761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde2c0b1fc0 0x7fde2c169340 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 >> 172.21.15.175:0/2439056761 conn(0x7fde2c063230 msgr2=0x7fde2c0c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:55.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 shutdown_connections 2023-10-10T18:45:55.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.747+0000 7fde332f0700 1 -- 172.21.15.175:0/2439056761 wait complete. 2023-10-10T18:45:55.750 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 17 2023-10-10T18:45:55.759 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T18:45:55.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.874+0000 7fbef8780700 1 Processor -- start 2023-10-10T18:45:55.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.874+0000 7fbef8780700 1 -- start start 2023-10-10T18:45:55.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef8780700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef00b1fc0 0x7fbef0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef8780700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbef00b2390 0x7fbef0068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef8780700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbef0068bd0 0x7fbef00aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef8780700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef0150770 con 0x7fbef00b2390 2023-10-10T18:45:55.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef8780700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef01574b0 con 0x7fbef0068bd0 2023-10-10T18:45:55.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef8780700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00afad0 con 0x7fbef00b1fc0 2023-10-10T18:45:55.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbef00b2390 0x7fbef0068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbef00b2390 0x7fbef0068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57904/0 (socket says 172.21.15.175:57904) 2023-10-10T18:45:55.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 -- 172.21.15.175:0/1804538852 learned_addr learned my addr 172.21.15.175:0/1804538852 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:55.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef651c700 1 --2- 172.21.15.175:0/1804538852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef00b1fc0 0x7fbef0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef6d1d700 1 --2- 172.21.15.175:0/1804538852 >> v2:172.21.15.191:3300/0 conn(0x7fbef0068bd0 0x7fbef00aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 -- 172.21.15.175:0/1804538852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef00b1fc0 msgr2=0x7fbef0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:55.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 --2- 172.21.15.175:0/1804538852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef00b1fc0 0x7fbef0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 -- 172.21.15.175:0/1804538852 >> v2:172.21.15.191:3300/0 conn(0x7fbef0068bd0 msgr2=0x7fbef00aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:45:55.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 --2- 172.21.15.175:0/1804538852 >> v2:172.21.15.191:3300/0 conn(0x7fbef0068bd0 0x7fbef00aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 -- 172.21.15.175:0/1804538852 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00af4c0 con 0x7fbef00b2390 2023-10-10T18:45:55.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef5d1b700 1 --2- 172.21.15.175:0/1804538852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b2390 0x7fbef0068690 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbef00afa40 tx=0x7fbee4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fbe889c0cfdb015 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef551a700 1 -- 172.21.15.175:0/1804538852 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee400b8f0 con 0x7fbef00b2390 2023-10-10T18:45:55.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef551a700 1 -- 172.21.15.175:0/1804538852 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee400ba50 con 0x7fbef00b2390 2023-10-10T18:45:55.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.875+0000 7fbef551a700 1 -- 172.21.15.175:0/1804538852 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee400bdf0 con 0x7fbef00b2390 2023-10-10T18:45:55.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 -- 172.21.15.175:0/1804538852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b2390 msgr2=0x7fbef0068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 --2- 172.21.15.175:0/1804538852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b2390 0x7fbef0068690 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbef00afa40 tx=0x7fbee4009250 comp rx=0 tx=0).stop 2023-10-10T18:45:55.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 -- 172.21.15.175:0/1804538852 shutdown_connections 2023-10-10T18:45:55.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 --2- 172.21.15.175:0/1804538852 >> v2:172.21.15.191:3300/0 conn(0x7fbef0068bd0 0x7fbef00aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 --2- 172.21.15.175:0/1804538852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef00b1fc0 0x7fbef0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 --2- 172.21.15.175:0/1804538852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b2390 0x7fbef0068690 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 -- 172.21.15.175:0/1804538852 >> 172.21.15.175:0/1804538852 conn(0x7fbef0063230 msgr2=0x7fbef0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:55.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 -- 172.21.15.175:0/1804538852 shutdown_connections 2023-10-10T18:45:55.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.876+0000 7fbef8780700 1 -- 172.21.15.175:0/1804538852 wait complete. 2023-10-10T18:45:55.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.877+0000 7fbef8780700 1 Processor -- start 2023-10-10T18:45:55.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.877+0000 7fbef8780700 1 -- start start 2023-10-10T18:45:55.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef8780700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef0068bd0 0x7fbef0157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef8780700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b1fc0 0x7fbef0155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef8780700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbef00b2390 0x7fbef01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef8780700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00cd740 con 0x7fbef00b1fc0 2023-10-10T18:45:55.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef8780700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef008a4c0 con 0x7fbef00b2390 2023-10-10T18:45:55.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef8780700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00cd460 con 0x7fbef0068bd0 2023-10-10T18:45:55.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b1fc0 0x7fbef0155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b1fc0 0x7fbef0155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57918/0 (socket says 172.21.15.175:57918) 2023-10-10T18:45:55.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 -- 172.21.15.175:0/3273250365 learned_addr learned my addr 172.21.15.175:0/3273250365 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:45:55.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef651c700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef0068bd0 0x7fbef0157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef6d1d700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbef00b2390 0x7fbef01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 -- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef0068bd0 msgr2=0x7fbef0157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef0068bd0 0x7fbef0157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 -- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbef00b2390 msgr2=0x7fbef01563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbef00b2390 0x7fbef01563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 -- 172.21.15.175:0/3273250365 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee40094f0 con 0x7fbef00b1fc0 2023-10-10T18:45:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef5d1b700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b1fc0 0x7fbef0155e90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbef006fd90 tx=0x7fbee4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbeee7fc700 1 -- 172.21.15.175:0/3273250365 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4011e60 con 0x7fbef00b1fc0 2023-10-10T18:45:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbeee7fc700 1 -- 172.21.15.175:0/3273250365 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee401a660 con 0x7fbef00b1fc0 2023-10-10T18:45:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbeee7fc700 1 -- 172.21.15.175:0/3273250365 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee401aa20 con 0x7fbef00b1fc0 2023-10-10T18:45:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef01550d0 con 0x7fbef00b1fc0 2023-10-10T18:45:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.878+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef0155690 con 0x7fbef00b1fc0 2023-10-10T18:45:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.879+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef0067570 con 0x7fbef00b1fc0 2023-10-10T18:45:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.880+0000 7fbeee7fc700 1 -- 172.21.15.175:0/3273250365 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbee4025440 con 0x7fbef00b1fc0 2023-10-10T18:45:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.880+0000 7fbeee7fc700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbedc066f30 0x7fbedc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:45:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.880+0000 7fbeee7fc700 1 -- 172.21.15.175:0/3273250365 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbee40862d0 con 0x7fbef00b1fc0 2023-10-10T18:45:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.880+0000 7fbef651c700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbedc066f30 0x7fbedc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:45:55.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.881+0000 7fbef651c700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbedc066f30 0x7fbedc0693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbee80075a0 tx=0x7fbee80097a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:45:55.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:55.884+0000 7fbeee7fc700 1 -- 172.21.15.175:0/3273250365 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbee4053540 con 0x7fbef00b1fc0 2023-10-10T18:45:56.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.067+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbef0069060 con 0x7fbef00b1fc0 2023-10-10T18:45:56.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.068+0000 7fbeee7fc700 1 -- 172.21.15.175:0/3273250365 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15398 (secure 0 0 0) 0x7fbee4056b60 con 0x7fbef00b1fc0 2023-10-10T18:45:56.068 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:45:56.068 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":21,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T18:45:12.409273+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T18:45:56.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbedc066f30 msgr2=0x7fbedc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:56.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbedc066f30 0x7fbedc0693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbee80075a0 tx=0x7fbee80097a0 comp rx=0 tx=0).stop 2023-10-10T18:45:56.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b1fc0 msgr2=0x7fbef0155e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:45:56.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b1fc0 0x7fbef0155e90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbef006fd90 tx=0x7fbee4017000 comp rx=0 tx=0).stop 2023-10-10T18:45:56.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 shutdown_connections 2023-10-10T18:45:56.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbedc066f30 0x7fbedc0693e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:56.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbef00b2390 0x7fbef01563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:56.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbef0068bd0 0x7fbef0157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:56.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 --2- 172.21.15.175:0/3273250365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbef00b1fc0 0x7fbef0155e90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:45:56.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 >> 172.21.15.175:0/3273250365 conn(0x7fbef0063230 msgr2=0x7fbef014fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:45:56.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 shutdown_connections 2023-10-10T18:45:56.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:45:56.070+0000 7fbef8780700 1 -- 172.21.15.175:0/3273250365 wait complete. 2023-10-10T18:45:56.082 DEBUG:tasks.mds_thrash:Yielding 2023-10-10T18:45:56.082 INFO:teuthology.run_tasks:Running task workunit... 2023-10-10T18:45:56.097 INFO:tasks.workunit:Pulling workunits from ref 46d1f59b549730ba7f93d940a3d32b94c45a4cf7 2023-10-10T18:45:56.097 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-10T18:45:56.098 INFO:tasks.workunit:timeout=3h 2023-10-10T18:45:56.098 INFO:tasks.workunit:cleanup=True 2023-10-10T18:45:56.099 DEBUG:teuthology.orchestra.run.smithi175:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-10T18:45:56.099 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-10-10T18:45:56.100 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-10-10T18:45:56.116 INFO:teuthology.orchestra.run.smithi175.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-10T18:45:56.117 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-10T18:45:56.117 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 2eh/46d Inode: 1 Links: 2 2023-10-10T18:45:56.118 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T18:45:56.118 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:cephfs_t:s0 2023-10-10T18:45:56.119 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-10T18:45:56.119 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-10-10 18:45:16.410828942 +0000 2023-10-10T18:45:56.119 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-10-10 18:45:49.214229396 +0000 2023-10-10T18:45:56.120 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2023-10-10T18:45:56.120 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-10T18:45:56.121 DEBUG:teuthology.orchestra.run.smithi175:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-10T18:45:56.186 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 46d1f59b549730ba7f93d940a3d32b94c45a4cf7 2023-10-10T18:45:56.241 INFO:tasks.workunit.client.0.smithi175.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-10T18:46:18.335 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:46:18.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:46:41.240 INFO:tasks.workunit.client.0.smithi175.stderr:Note: switching to '46d1f59b549730ba7f93d940a3d32b94c45a4cf7'. 2023-10-10T18:46:41.241 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T18:46:41.241 INFO:tasks.workunit.client.0.smithi175.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-10T18:46:41.241 INFO:tasks.workunit.client.0.smithi175.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-10T18:46:41.241 INFO:tasks.workunit.client.0.smithi175.stderr:state without impacting any branches by switching back to a branch. 2023-10-10T18:46:41.242 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T18:46:41.242 INFO:tasks.workunit.client.0.smithi175.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-10T18:46:41.242 INFO:tasks.workunit.client.0.smithi175.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-10T18:46:41.242 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T18:46:41.242 INFO:tasks.workunit.client.0.smithi175.stderr: git switch -c 2023-10-10T18:46:41.243 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T18:46:41.243 INFO:tasks.workunit.client.0.smithi175.stderr:Or undo this operation with: 2023-10-10T18:46:41.243 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T18:46:41.243 INFO:tasks.workunit.client.0.smithi175.stderr: git switch - 2023-10-10T18:46:41.243 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T18:46:41.244 INFO:tasks.workunit.client.0.smithi175.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-10T18:46:41.244 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T18:46:41.244 INFO:tasks.workunit.client.0.smithi175.stderr:HEAD is now at 46d1f59b549 Merge branch 'wip-63081-quincy' of https://github.com/batrick/ceph into wip-yuri3-testing-2023-10-10-0720-quincy 2023-10-10T18:46:41.246 DEBUG:teuthology.orchestra.run.smithi175:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-10T18:46:41.350 INFO:tasks.workunit.client.0.smithi175.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-10T18:46:41.352 INFO:tasks.workunit.client.0.smithi175.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-10T18:46:41.352 INFO:tasks.workunit.client.0.smithi175.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-10T18:46:41.908 INFO:tasks.workunit.client.0.smithi175.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-10T18:46:41.962 INFO:tasks.workunit.client.0.smithi175.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-10T18:46:41.999 INFO:tasks.workunit.client.0.smithi175.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-10T18:46:42.000 INFO:tasks.workunit.client.0.smithi175.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-10T18:46:42.000 INFO:tasks.workunit.client.0.smithi175.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-10T18:46:42.036 INFO:tasks.workunit.client.0.smithi175.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-10T18:46:42.039 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-10-10T18:46:42.040 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-10T18:46:42.054 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-10-10T18:46:42.055 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-10-10T18:46:42.056 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-10-10T18:46:42.116 INFO:tasks.workunit.client.0.smithi175.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-10T18:46:44.465 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir foo 2023-10-10T18:46:44.470 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-10-10T18:46:44.520 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir foo/.snap/barsnap 2023-10-10T18:46:44.543 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-10-10T18:46:44.558 INFO:tasks.workunit.client.0.smithi175.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-10-10T18:46:44.767 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 'passed: files are identical' 2023-10-10T18:46:44.768 INFO:tasks.workunit.client.0.smithi175.stdout:passed: files are identical 2023-10-10T18:46:44.768 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir foo/.snap/barsnap 2023-10-10T18:46:44.777 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T18:46:44.778 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T18:46:44.778 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T18:46:44.779 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T18:46:44.855 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-10-10T18:46:44.856 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-10-10T18:46:44.916 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 1 2023-10-10T18:46:44.917 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 2 2023-10-10T18:46:44.918 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 3 2023-10-10T18:46:44.919 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' -e file4 ']' 2023-10-10T18:46:44.919 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/snap1 2023-10-10T18:46:44.944 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 4 2023-10-10T18:46:44.945 INFO:tasks.workunit.client.0.smithi175.stderr:++ ls 2023-10-10T18:46:44.949 INFO:tasks.workunit.client.0.smithi175.stderr:+ now='file1 2023-10-10T18:46:44.949 INFO:tasks.workunit.client.0.smithi175.stderr:file2 2023-10-10T18:46:44.949 INFO:tasks.workunit.client.0.smithi175.stderr:file3 2023-10-10T18:46:44.950 INFO:tasks.workunit.client.0.smithi175.stderr:file4' 2023-10-10T18:46:44.950 INFO:tasks.workunit.client.0.smithi175.stderr:++ ls .snap/snap1 2023-10-10T18:46:45.701 INFO:tasks.workunit.client.0.smithi175.stderr:+ then='file1 2023-10-10T18:46:45.702 INFO:tasks.workunit.client.0.smithi175.stderr:file2 2023-10-10T18:46:45.702 INFO:tasks.workunit.client.0.smithi175.stderr:file3' 2023-10-10T18:46:45.702 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/snap1 2023-10-10T18:46:45.707 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' 'file1 2023-10-10T18:46:45.707 INFO:tasks.workunit.client.0.smithi175.stderr:file2 2023-10-10T18:46:45.707 INFO:tasks.workunit.client.0.smithi175.stderr:file3 2023-10-10T18:46:45.707 INFO:tasks.workunit.client.0.smithi175.stderr:file4' = 'file1 2023-10-10T18:46:45.708 INFO:tasks.workunit.client.0.smithi175.stderr:file2 2023-10-10T18:46:45.708 INFO:tasks.workunit.client.0.smithi175.stderr:file3' ']' 2023-10-10T18:46:45.708 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 1 2023-10-10T18:46:45.712 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 2 2023-10-10T18:46:45.717 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 3 2023-10-10T18:46:45.722 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/snap1 2023-10-10T18:46:45.746 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 4 2023-10-10T18:46:45.751 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/snap1 2023-10-10T18:46:45.757 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm file1 file2 file3 file4 2023-10-10T18:46:45.766 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T18:46:45.767 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T18:46:45.767 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T18:46:45.768 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T18:46:45.844 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-10-10T18:46:45.844 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-10-10T18:46:47.461 INFO:tasks.workunit.client.0.smithi175.stdout:Create dir 100 to 199 ... 2023-10-10T18:46:47.473 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 100 2023-10-10T18:46:47.487 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 101 2023-10-10T18:46:47.500 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 102 2023-10-10T18:46:48.035 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 103 2023-10-10T18:46:48.049 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 104 2023-10-10T18:46:48.062 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 105 2023-10-10T18:46:48.076 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 106 2023-10-10T18:46:48.090 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 107 2023-10-10T18:46:48.102 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 108 2023-10-10T18:46:48.116 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 109 2023-10-10T18:46:48.129 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 110 2023-10-10T18:46:48.142 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 111 2023-10-10T18:46:48.153 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 112 2023-10-10T18:46:48.164 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 113 2023-10-10T18:46:48.176 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 114 2023-10-10T18:46:48.188 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 115 2023-10-10T18:46:48.200 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 116 2023-10-10T18:46:48.211 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 117 2023-10-10T18:46:48.225 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 118 2023-10-10T18:46:48.237 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 119 2023-10-10T18:46:48.248 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 120 2023-10-10T18:46:48.260 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 121 2023-10-10T18:46:48.272 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 122 2023-10-10T18:46:48.283 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 123 2023-10-10T18:46:48.295 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 124 2023-10-10T18:46:48.308 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 125 2023-10-10T18:46:48.412 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:46:48.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:46:48.584 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 126 2023-10-10T18:46:48.598 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 127 2023-10-10T18:46:48.609 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 128 2023-10-10T18:46:48.622 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 129 2023-10-10T18:46:48.635 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 130 2023-10-10T18:46:48.647 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 131 2023-10-10T18:46:49.278 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 132 2023-10-10T18:46:49.292 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 133 2023-10-10T18:46:49.306 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 134 2023-10-10T18:46:49.320 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 135 2023-10-10T18:46:49.333 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 136 2023-10-10T18:46:49.346 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 137 2023-10-10T18:46:49.360 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 138 2023-10-10T18:46:49.374 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 139 2023-10-10T18:46:49.388 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 140 2023-10-10T18:46:49.403 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 141 2023-10-10T18:46:49.417 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 142 2023-10-10T18:46:49.431 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 143 2023-10-10T18:46:49.444 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 144 2023-10-10T18:46:49.457 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 145 2023-10-10T18:46:49.469 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 146 2023-10-10T18:46:49.482 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 147 2023-10-10T18:46:49.494 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 148 2023-10-10T18:46:49.507 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 149 2023-10-10T18:46:49.518 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 150 2023-10-10T18:46:49.529 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 151 2023-10-10T18:46:49.540 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 152 2023-10-10T18:46:49.552 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 153 2023-10-10T18:46:49.564 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 154 2023-10-10T18:46:49.575 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 155 2023-10-10T18:46:49.586 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 156 2023-10-10T18:46:49.598 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 157 2023-10-10T18:46:49.611 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 158 2023-10-10T18:46:50.583 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 159 2023-10-10T18:46:50.863 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 160 2023-10-10T18:46:50.912 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 161 2023-10-10T18:46:51.410 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 162 2023-10-10T18:46:51.424 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 163 2023-10-10T18:46:51.439 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 164 2023-10-10T18:46:51.453 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 165 2023-10-10T18:46:51.467 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 166 2023-10-10T18:46:51.480 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 167 2023-10-10T18:46:51.492 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 168 2023-10-10T18:46:51.505 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 169 2023-10-10T18:46:51.519 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 170 2023-10-10T18:46:51.532 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 171 2023-10-10T18:46:51.547 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 172 2023-10-10T18:46:51.559 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 173 2023-10-10T18:46:51.570 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 174 2023-10-10T18:46:51.599 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 175 2023-10-10T18:46:51.874 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 176 2023-10-10T18:46:51.886 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 177 2023-10-10T18:46:51.899 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 178 2023-10-10T18:46:51.910 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 179 2023-10-10T18:46:51.922 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 180 2023-10-10T18:46:51.934 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 181 2023-10-10T18:46:51.946 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 182 2023-10-10T18:46:51.959 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 183 2023-10-10T18:46:51.973 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 184 2023-10-10T18:46:51.987 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 185 2023-10-10T18:46:51.999 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 186 2023-10-10T18:46:52.012 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 187 2023-10-10T18:46:52.025 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 188 2023-10-10T18:46:52.038 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 189 2023-10-10T18:46:52.053 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 190 2023-10-10T18:46:52.063 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 191 2023-10-10T18:46:52.076 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 192 2023-10-10T18:46:52.088 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 193 2023-10-10T18:46:52.101 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 194 2023-10-10T18:46:52.114 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 195 2023-10-10T18:46:52.127 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 196 2023-10-10T18:46:52.140 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 197 2023-10-10T18:46:52.152 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 198 2023-10-10T18:46:52.163 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 199 2023-10-10T18:46:52.175 INFO:tasks.workunit.client.0.smithi175.stdout:Take first snapshot .snap/test1 2023-10-10T18:46:53.167 INFO:tasks.workunit.client.0.smithi175.stdout:Create dir 200 to 299 ... 2023-10-10T18:46:53.168 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 200 2023-10-10T18:46:53.182 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 201 2023-10-10T18:46:53.195 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 202 2023-10-10T18:46:53.208 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 203 2023-10-10T18:46:53.220 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 204 2023-10-10T18:46:53.232 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 205 2023-10-10T18:46:53.244 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 206 2023-10-10T18:46:53.257 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 207 2023-10-10T18:46:53.547 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 208 2023-10-10T18:46:53.559 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 209 2023-10-10T18:46:53.572 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 210 2023-10-10T18:46:53.584 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 211 2023-10-10T18:46:53.820 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 212 2023-10-10T18:46:53.833 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 213 2023-10-10T18:46:53.845 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 214 2023-10-10T18:46:53.857 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 215 2023-10-10T18:46:53.869 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 216 2023-10-10T18:46:53.883 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 217 2023-10-10T18:46:53.895 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 218 2023-10-10T18:46:54.206 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 219 2023-10-10T18:46:54.218 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 220 2023-10-10T18:46:54.230 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 221 2023-10-10T18:46:54.243 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 222 2023-10-10T18:46:54.256 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 223 2023-10-10T18:46:54.269 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 224 2023-10-10T18:46:54.282 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 225 2023-10-10T18:46:54.293 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 226 2023-10-10T18:46:54.306 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 227 2023-10-10T18:46:54.318 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 228 2023-10-10T18:46:54.329 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 229 2023-10-10T18:46:54.342 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 230 2023-10-10T18:46:54.355 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 231 2023-10-10T18:46:54.366 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 232 2023-10-10T18:46:54.377 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 233 2023-10-10T18:46:54.389 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 234 2023-10-10T18:46:54.401 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 235 2023-10-10T18:46:54.410 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 236 2023-10-10T18:46:54.420 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 237 2023-10-10T18:46:54.431 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 238 2023-10-10T18:46:54.441 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 239 2023-10-10T18:46:54.453 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 240 2023-10-10T18:46:54.464 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 241 2023-10-10T18:46:54.974 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 242 2023-10-10T18:46:54.987 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 243 2023-10-10T18:46:54.997 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 244 2023-10-10T18:46:55.009 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 245 2023-10-10T18:46:55.286 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 246 2023-10-10T18:46:55.298 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 247 2023-10-10T18:46:55.571 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 248 2023-10-10T18:46:55.583 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 249 2023-10-10T18:46:55.595 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 250 2023-10-10T18:46:55.611 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 251 2023-10-10T18:46:58.126 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 252 2023-10-10T18:46:58.138 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 253 2023-10-10T18:46:58.151 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 254 2023-10-10T18:46:58.164 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 255 2023-10-10T18:46:58.177 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 256 2023-10-10T18:46:58.190 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 257 2023-10-10T18:46:58.203 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 258 2023-10-10T18:46:58.216 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 259 2023-10-10T18:46:58.228 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 260 2023-10-10T18:46:58.240 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 261 2023-10-10T18:46:58.252 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 262 2023-10-10T18:46:58.263 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 263 2023-10-10T18:46:58.275 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 264 2023-10-10T18:46:58.287 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 265 2023-10-10T18:46:58.298 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 266 2023-10-10T18:46:58.309 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 267 2023-10-10T18:46:58.322 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 268 2023-10-10T18:46:58.334 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 269 2023-10-10T18:46:58.347 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 270 2023-10-10T18:46:58.358 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 271 2023-10-10T18:46:58.369 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 272 2023-10-10T18:46:58.381 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 273 2023-10-10T18:46:58.391 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 274 2023-10-10T18:46:58.403 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 275 2023-10-10T18:46:58.418 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 276 2023-10-10T18:46:58.430 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 277 2023-10-10T18:46:58.441 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 278 2023-10-10T18:46:58.730 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 279 2023-10-10T18:46:58.782 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 280 2023-10-10T18:46:58.813 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 281 2023-10-10T18:46:58.908 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 282 2023-10-10T18:46:58.919 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 283 2023-10-10T18:46:58.929 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 284 2023-10-10T18:46:58.939 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 285 2023-10-10T18:46:59.213 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 286 2023-10-10T18:46:59.224 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 287 2023-10-10T18:46:59.236 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 288 2023-10-10T18:46:59.248 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 289 2023-10-10T18:46:59.259 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 290 2023-10-10T18:46:59.270 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 291 2023-10-10T18:46:59.280 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 292 2023-10-10T18:46:59.292 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 293 2023-10-10T18:46:59.303 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 294 2023-10-10T18:46:59.313 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 295 2023-10-10T18:46:59.323 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 296 2023-10-10T18:46:59.334 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 297 2023-10-10T18:46:59.345 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 298 2023-10-10T18:46:59.356 INFO:tasks.workunit.client.0.smithi175.stdout: create dir 299 2023-10-10T18:46:59.368 INFO:tasks.workunit.client.0.smithi175.stdout:Create a snapshot in every first level dir ... 2023-10-10T18:47:00.338 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/.snap/snap-subdir-test 2023-10-10T18:47:00.461 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/30 file after the snapshot 2023-10-10T18:47:00.462 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/31 file after the snapshot 2023-10-10T18:47:00.463 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/32 file after the snapshot 2023-10-10T18:47:00.465 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/33 file after the snapshot 2023-10-10T18:47:00.466 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/34 file after the snapshot 2023-10-10T18:47:00.467 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/35 file after the snapshot 2023-10-10T18:47:00.469 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/36 file after the snapshot 2023-10-10T18:47:00.470 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/37 file after the snapshot 2023-10-10T18:47:00.475 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/38 file after the snapshot 2023-10-10T18:47:00.476 INFO:tasks.workunit.client.0.smithi175.stdout: create 100/39 file after the snapshot 2023-10-10T18:47:00.476 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/.snap/snap-subdir-test 2023-10-10T18:47:01.389 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/30 file after the snapshot 2023-10-10T18:47:01.390 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/31 file after the snapshot 2023-10-10T18:47:01.391 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/32 file after the snapshot 2023-10-10T18:47:01.392 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/33 file after the snapshot 2023-10-10T18:47:01.393 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/34 file after the snapshot 2023-10-10T18:47:01.394 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/35 file after the snapshot 2023-10-10T18:47:01.395 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/36 file after the snapshot 2023-10-10T18:47:01.396 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/37 file after the snapshot 2023-10-10T18:47:01.398 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/38 file after the snapshot 2023-10-10T18:47:01.399 INFO:tasks.workunit.client.0.smithi175.stdout: create 101/39 file after the snapshot 2023-10-10T18:47:01.400 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/.snap/snap-subdir-test 2023-10-10T18:47:01.439 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/30 file after the snapshot 2023-10-10T18:47:01.440 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/31 file after the snapshot 2023-10-10T18:47:01.441 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/32 file after the snapshot 2023-10-10T18:47:01.442 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/33 file after the snapshot 2023-10-10T18:47:01.443 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/34 file after the snapshot 2023-10-10T18:47:01.444 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/35 file after the snapshot 2023-10-10T18:47:01.445 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/36 file after the snapshot 2023-10-10T18:47:01.446 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/37 file after the snapshot 2023-10-10T18:47:01.447 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/38 file after the snapshot 2023-10-10T18:47:01.448 INFO:tasks.workunit.client.0.smithi175.stdout: create 102/39 file after the snapshot 2023-10-10T18:47:01.449 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/.snap/snap-subdir-test 2023-10-10T18:47:01.486 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/30 file after the snapshot 2023-10-10T18:47:01.487 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/31 file after the snapshot 2023-10-10T18:47:01.488 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/32 file after the snapshot 2023-10-10T18:47:01.489 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/33 file after the snapshot 2023-10-10T18:47:01.490 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/34 file after the snapshot 2023-10-10T18:47:01.491 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/35 file after the snapshot 2023-10-10T18:47:01.492 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/36 file after the snapshot 2023-10-10T18:47:01.493 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/37 file after the snapshot 2023-10-10T18:47:01.494 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/38 file after the snapshot 2023-10-10T18:47:01.495 INFO:tasks.workunit.client.0.smithi175.stdout: create 103/39 file after the snapshot 2023-10-10T18:47:01.496 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/.snap/snap-subdir-test 2023-10-10T18:47:01.530 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/30 file after the snapshot 2023-10-10T18:47:01.532 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/31 file after the snapshot 2023-10-10T18:47:01.532 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/32 file after the snapshot 2023-10-10T18:47:01.533 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/33 file after the snapshot 2023-10-10T18:47:01.534 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/34 file after the snapshot 2023-10-10T18:47:01.535 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/35 file after the snapshot 2023-10-10T18:47:01.536 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/36 file after the snapshot 2023-10-10T18:47:01.536 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/37 file after the snapshot 2023-10-10T18:47:01.537 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/38 file after the snapshot 2023-10-10T18:47:01.538 INFO:tasks.workunit.client.0.smithi175.stdout: create 104/39 file after the snapshot 2023-10-10T18:47:01.539 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/.snap/snap-subdir-test 2023-10-10T18:47:01.574 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/30 file after the snapshot 2023-10-10T18:47:01.575 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/31 file after the snapshot 2023-10-10T18:47:01.576 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/32 file after the snapshot 2023-10-10T18:47:01.577 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/33 file after the snapshot 2023-10-10T18:47:01.578 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/34 file after the snapshot 2023-10-10T18:47:01.579 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/35 file after the snapshot 2023-10-10T18:47:01.580 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/36 file after the snapshot 2023-10-10T18:47:01.581 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/37 file after the snapshot 2023-10-10T18:47:01.582 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/38 file after the snapshot 2023-10-10T18:47:01.582 INFO:tasks.workunit.client.0.smithi175.stdout: create 105/39 file after the snapshot 2023-10-10T18:47:01.584 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/.snap/snap-subdir-test 2023-10-10T18:47:01.618 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/30 file after the snapshot 2023-10-10T18:47:01.620 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/31 file after the snapshot 2023-10-10T18:47:01.621 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/32 file after the snapshot 2023-10-10T18:47:01.622 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/33 file after the snapshot 2023-10-10T18:47:01.623 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/34 file after the snapshot 2023-10-10T18:47:01.624 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/35 file after the snapshot 2023-10-10T18:47:01.625 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/36 file after the snapshot 2023-10-10T18:47:01.626 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/37 file after the snapshot 2023-10-10T18:47:01.627 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/38 file after the snapshot 2023-10-10T18:47:01.628 INFO:tasks.workunit.client.0.smithi175.stdout: create 106/39 file after the snapshot 2023-10-10T18:47:01.629 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/.snap/snap-subdir-test 2023-10-10T18:47:01.664 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/30 file after the snapshot 2023-10-10T18:47:01.665 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/31 file after the snapshot 2023-10-10T18:47:01.666 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/32 file after the snapshot 2023-10-10T18:47:01.667 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/33 file after the snapshot 2023-10-10T18:47:01.668 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/34 file after the snapshot 2023-10-10T18:47:01.669 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/35 file after the snapshot 2023-10-10T18:47:01.670 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/36 file after the snapshot 2023-10-10T18:47:01.671 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/37 file after the snapshot 2023-10-10T18:47:01.671 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/38 file after the snapshot 2023-10-10T18:47:01.672 INFO:tasks.workunit.client.0.smithi175.stdout: create 107/39 file after the snapshot 2023-10-10T18:47:01.673 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/.snap/snap-subdir-test 2023-10-10T18:47:01.708 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/30 file after the snapshot 2023-10-10T18:47:01.709 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/31 file after the snapshot 2023-10-10T18:47:01.710 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/32 file after the snapshot 2023-10-10T18:47:01.711 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/33 file after the snapshot 2023-10-10T18:47:01.711 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/34 file after the snapshot 2023-10-10T18:47:01.712 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/35 file after the snapshot 2023-10-10T18:47:01.713 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/36 file after the snapshot 2023-10-10T18:47:01.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/37 file after the snapshot 2023-10-10T18:47:01.715 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/38 file after the snapshot 2023-10-10T18:47:01.716 INFO:tasks.workunit.client.0.smithi175.stdout: create 108/39 file after the snapshot 2023-10-10T18:47:01.716 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/.snap/snap-subdir-test 2023-10-10T18:47:01.751 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/30 file after the snapshot 2023-10-10T18:47:01.752 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/31 file after the snapshot 2023-10-10T18:47:01.753 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/32 file after the snapshot 2023-10-10T18:47:01.753 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/33 file after the snapshot 2023-10-10T18:47:01.754 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/34 file after the snapshot 2023-10-10T18:47:01.755 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/35 file after the snapshot 2023-10-10T18:47:01.756 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/36 file after the snapshot 2023-10-10T18:47:01.757 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/37 file after the snapshot 2023-10-10T18:47:01.758 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/38 file after the snapshot 2023-10-10T18:47:01.759 INFO:tasks.workunit.client.0.smithi175.stdout: create 109/39 file after the snapshot 2023-10-10T18:47:01.760 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/.snap/snap-subdir-test 2023-10-10T18:47:01.786 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/30 file after the snapshot 2023-10-10T18:47:01.787 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/31 file after the snapshot 2023-10-10T18:47:01.788 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/32 file after the snapshot 2023-10-10T18:47:01.789 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/33 file after the snapshot 2023-10-10T18:47:01.790 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/34 file after the snapshot 2023-10-10T18:47:01.791 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/35 file after the snapshot 2023-10-10T18:47:01.792 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/36 file after the snapshot 2023-10-10T18:47:01.792 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/37 file after the snapshot 2023-10-10T18:47:01.793 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/38 file after the snapshot 2023-10-10T18:47:01.794 INFO:tasks.workunit.client.0.smithi175.stdout: create 110/39 file after the snapshot 2023-10-10T18:47:01.795 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/.snap/snap-subdir-test 2023-10-10T18:47:01.821 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/30 file after the snapshot 2023-10-10T18:47:01.822 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/31 file after the snapshot 2023-10-10T18:47:01.823 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/32 file after the snapshot 2023-10-10T18:47:01.824 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/33 file after the snapshot 2023-10-10T18:47:01.825 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/34 file after the snapshot 2023-10-10T18:47:01.825 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/35 file after the snapshot 2023-10-10T18:47:01.826 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/36 file after the snapshot 2023-10-10T18:47:01.827 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/37 file after the snapshot 2023-10-10T18:47:01.827 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/38 file after the snapshot 2023-10-10T18:47:01.828 INFO:tasks.workunit.client.0.smithi175.stdout: create 111/39 file after the snapshot 2023-10-10T18:47:01.829 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/.snap/snap-subdir-test 2023-10-10T18:47:01.855 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/30 file after the snapshot 2023-10-10T18:47:01.857 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/31 file after the snapshot 2023-10-10T18:47:01.858 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/32 file after the snapshot 2023-10-10T18:47:01.858 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/33 file after the snapshot 2023-10-10T18:47:01.859 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/34 file after the snapshot 2023-10-10T18:47:01.860 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/35 file after the snapshot 2023-10-10T18:47:01.860 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/36 file after the snapshot 2023-10-10T18:47:01.861 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/37 file after the snapshot 2023-10-10T18:47:01.862 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/38 file after the snapshot 2023-10-10T18:47:01.863 INFO:tasks.workunit.client.0.smithi175.stdout: create 112/39 file after the snapshot 2023-10-10T18:47:01.864 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/.snap/snap-subdir-test 2023-10-10T18:47:01.891 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/30 file after the snapshot 2023-10-10T18:47:01.892 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/31 file after the snapshot 2023-10-10T18:47:01.893 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/32 file after the snapshot 2023-10-10T18:47:01.894 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/33 file after the snapshot 2023-10-10T18:47:01.895 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/34 file after the snapshot 2023-10-10T18:47:01.896 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/35 file after the snapshot 2023-10-10T18:47:01.897 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/36 file after the snapshot 2023-10-10T18:47:01.898 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/37 file after the snapshot 2023-10-10T18:47:01.900 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/38 file after the snapshot 2023-10-10T18:47:01.901 INFO:tasks.workunit.client.0.smithi175.stdout: create 113/39 file after the snapshot 2023-10-10T18:47:01.902 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/.snap/snap-subdir-test 2023-10-10T18:47:01.930 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/30 file after the snapshot 2023-10-10T18:47:01.931 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/31 file after the snapshot 2023-10-10T18:47:01.932 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/32 file after the snapshot 2023-10-10T18:47:01.933 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/33 file after the snapshot 2023-10-10T18:47:01.934 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/34 file after the snapshot 2023-10-10T18:47:01.935 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/35 file after the snapshot 2023-10-10T18:47:01.936 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/36 file after the snapshot 2023-10-10T18:47:01.937 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/37 file after the snapshot 2023-10-10T18:47:01.938 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/38 file after the snapshot 2023-10-10T18:47:01.939 INFO:tasks.workunit.client.0.smithi175.stdout: create 114/39 file after the snapshot 2023-10-10T18:47:01.940 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/.snap/snap-subdir-test 2023-10-10T18:47:01.975 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/30 file after the snapshot 2023-10-10T18:47:01.976 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/31 file after the snapshot 2023-10-10T18:47:01.977 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/32 file after the snapshot 2023-10-10T18:47:01.978 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/33 file after the snapshot 2023-10-10T18:47:01.979 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/34 file after the snapshot 2023-10-10T18:47:01.980 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/35 file after the snapshot 2023-10-10T18:47:01.981 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/36 file after the snapshot 2023-10-10T18:47:01.982 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/37 file after the snapshot 2023-10-10T18:47:01.983 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/38 file after the snapshot 2023-10-10T18:47:01.984 INFO:tasks.workunit.client.0.smithi175.stdout: create 115/39 file after the snapshot 2023-10-10T18:47:01.985 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/.snap/snap-subdir-test 2023-10-10T18:47:02.013 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/30 file after the snapshot 2023-10-10T18:47:02.014 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/31 file after the snapshot 2023-10-10T18:47:02.015 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/32 file after the snapshot 2023-10-10T18:47:02.016 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/33 file after the snapshot 2023-10-10T18:47:02.017 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/34 file after the snapshot 2023-10-10T18:47:02.018 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/35 file after the snapshot 2023-10-10T18:47:02.019 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/36 file after the snapshot 2023-10-10T18:47:02.020 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/37 file after the snapshot 2023-10-10T18:47:02.021 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/38 file after the snapshot 2023-10-10T18:47:02.021 INFO:tasks.workunit.client.0.smithi175.stdout: create 116/39 file after the snapshot 2023-10-10T18:47:02.022 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/.snap/snap-subdir-test 2023-10-10T18:47:02.049 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/30 file after the snapshot 2023-10-10T18:47:02.050 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/31 file after the snapshot 2023-10-10T18:47:02.051 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/32 file after the snapshot 2023-10-10T18:47:02.052 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/33 file after the snapshot 2023-10-10T18:47:02.053 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/34 file after the snapshot 2023-10-10T18:47:02.054 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/35 file after the snapshot 2023-10-10T18:47:02.054 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/36 file after the snapshot 2023-10-10T18:47:02.055 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/37 file after the snapshot 2023-10-10T18:47:02.056 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/38 file after the snapshot 2023-10-10T18:47:02.057 INFO:tasks.workunit.client.0.smithi175.stdout: create 117/39 file after the snapshot 2023-10-10T18:47:02.058 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/.snap/snap-subdir-test 2023-10-10T18:47:02.085 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/30 file after the snapshot 2023-10-10T18:47:02.086 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/31 file after the snapshot 2023-10-10T18:47:02.087 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/32 file after the snapshot 2023-10-10T18:47:02.088 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/33 file after the snapshot 2023-10-10T18:47:02.089 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/34 file after the snapshot 2023-10-10T18:47:02.090 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/35 file after the snapshot 2023-10-10T18:47:02.090 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/36 file after the snapshot 2023-10-10T18:47:02.091 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/37 file after the snapshot 2023-10-10T18:47:02.092 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/38 file after the snapshot 2023-10-10T18:47:02.093 INFO:tasks.workunit.client.0.smithi175.stdout: create 118/39 file after the snapshot 2023-10-10T18:47:02.094 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/.snap/snap-subdir-test 2023-10-10T18:47:02.121 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/30 file after the snapshot 2023-10-10T18:47:02.122 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/31 file after the snapshot 2023-10-10T18:47:02.123 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/32 file after the snapshot 2023-10-10T18:47:02.124 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/33 file after the snapshot 2023-10-10T18:47:02.125 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/34 file after the snapshot 2023-10-10T18:47:02.126 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/35 file after the snapshot 2023-10-10T18:47:02.126 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/36 file after the snapshot 2023-10-10T18:47:02.127 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/37 file after the snapshot 2023-10-10T18:47:02.128 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/38 file after the snapshot 2023-10-10T18:47:02.129 INFO:tasks.workunit.client.0.smithi175.stdout: create 119/39 file after the snapshot 2023-10-10T18:47:02.130 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/.snap/snap-subdir-test 2023-10-10T18:47:02.157 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/30 file after the snapshot 2023-10-10T18:47:02.158 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/31 file after the snapshot 2023-10-10T18:47:02.159 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/32 file after the snapshot 2023-10-10T18:47:02.159 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/33 file after the snapshot 2023-10-10T18:47:02.160 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/34 file after the snapshot 2023-10-10T18:47:02.161 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/35 file after the snapshot 2023-10-10T18:47:02.162 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/36 file after the snapshot 2023-10-10T18:47:02.163 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/37 file after the snapshot 2023-10-10T18:47:02.607 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/38 file after the snapshot 2023-10-10T18:47:02.942 INFO:tasks.workunit.client.0.smithi175.stdout: create 120/39 file after the snapshot 2023-10-10T18:47:02.943 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/.snap/snap-subdir-test 2023-10-10T18:47:02.970 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/30 file after the snapshot 2023-10-10T18:47:02.971 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/31 file after the snapshot 2023-10-10T18:47:02.972 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/32 file after the snapshot 2023-10-10T18:47:02.972 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/33 file after the snapshot 2023-10-10T18:47:02.973 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/34 file after the snapshot 2023-10-10T18:47:02.974 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/35 file after the snapshot 2023-10-10T18:47:02.975 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/36 file after the snapshot 2023-10-10T18:47:02.976 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/37 file after the snapshot 2023-10-10T18:47:02.977 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/38 file after the snapshot 2023-10-10T18:47:02.978 INFO:tasks.workunit.client.0.smithi175.stdout: create 121/39 file after the snapshot 2023-10-10T18:47:02.979 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/.snap/snap-subdir-test 2023-10-10T18:47:03.008 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/30 file after the snapshot 2023-10-10T18:47:03.009 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/31 file after the snapshot 2023-10-10T18:47:03.010 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/32 file after the snapshot 2023-10-10T18:47:03.011 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/33 file after the snapshot 2023-10-10T18:47:03.012 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/34 file after the snapshot 2023-10-10T18:47:03.013 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/35 file after the snapshot 2023-10-10T18:47:03.014 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/36 file after the snapshot 2023-10-10T18:47:03.015 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/37 file after the snapshot 2023-10-10T18:47:03.016 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/38 file after the snapshot 2023-10-10T18:47:03.017 INFO:tasks.workunit.client.0.smithi175.stdout: create 122/39 file after the snapshot 2023-10-10T18:47:03.018 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/.snap/snap-subdir-test 2023-10-10T18:47:03.046 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/30 file after the snapshot 2023-10-10T18:47:03.047 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/31 file after the snapshot 2023-10-10T18:47:03.048 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/32 file after the snapshot 2023-10-10T18:47:03.049 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/33 file after the snapshot 2023-10-10T18:47:03.050 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/34 file after the snapshot 2023-10-10T18:47:03.050 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/35 file after the snapshot 2023-10-10T18:47:03.051 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/36 file after the snapshot 2023-10-10T18:47:03.052 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/37 file after the snapshot 2023-10-10T18:47:03.053 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/38 file after the snapshot 2023-10-10T18:47:03.053 INFO:tasks.workunit.client.0.smithi175.stdout: create 123/39 file after the snapshot 2023-10-10T18:47:03.054 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/.snap/snap-subdir-test 2023-10-10T18:47:03.081 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/30 file after the snapshot 2023-10-10T18:47:03.082 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/31 file after the snapshot 2023-10-10T18:47:03.083 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/32 file after the snapshot 2023-10-10T18:47:03.084 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/33 file after the snapshot 2023-10-10T18:47:03.085 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/34 file after the snapshot 2023-10-10T18:47:03.086 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/35 file after the snapshot 2023-10-10T18:47:03.087 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/36 file after the snapshot 2023-10-10T18:47:03.088 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/37 file after the snapshot 2023-10-10T18:47:03.088 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/38 file after the snapshot 2023-10-10T18:47:03.089 INFO:tasks.workunit.client.0.smithi175.stdout: create 124/39 file after the snapshot 2023-10-10T18:47:03.090 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/.snap/snap-subdir-test 2023-10-10T18:47:03.117 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/30 file after the snapshot 2023-10-10T18:47:03.118 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/31 file after the snapshot 2023-10-10T18:47:03.119 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/32 file after the snapshot 2023-10-10T18:47:03.120 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/33 file after the snapshot 2023-10-10T18:47:03.121 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/34 file after the snapshot 2023-10-10T18:47:03.122 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/35 file after the snapshot 2023-10-10T18:47:03.123 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/36 file after the snapshot 2023-10-10T18:47:03.124 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/37 file after the snapshot 2023-10-10T18:47:03.124 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/38 file after the snapshot 2023-10-10T18:47:03.125 INFO:tasks.workunit.client.0.smithi175.stdout: create 125/39 file after the snapshot 2023-10-10T18:47:03.126 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/.snap/snap-subdir-test 2023-10-10T18:47:03.153 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/30 file after the snapshot 2023-10-10T18:47:03.154 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/31 file after the snapshot 2023-10-10T18:47:03.155 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/32 file after the snapshot 2023-10-10T18:47:03.156 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/33 file after the snapshot 2023-10-10T18:47:03.156 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/34 file after the snapshot 2023-10-10T18:47:03.158 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/35 file after the snapshot 2023-10-10T18:47:03.159 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/36 file after the snapshot 2023-10-10T18:47:03.159 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/37 file after the snapshot 2023-10-10T18:47:03.160 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/38 file after the snapshot 2023-10-10T18:47:03.161 INFO:tasks.workunit.client.0.smithi175.stdout: create 126/39 file after the snapshot 2023-10-10T18:47:03.162 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/.snap/snap-subdir-test 2023-10-10T18:47:03.188 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/30 file after the snapshot 2023-10-10T18:47:03.189 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/31 file after the snapshot 2023-10-10T18:47:03.190 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/32 file after the snapshot 2023-10-10T18:47:03.191 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/33 file after the snapshot 2023-10-10T18:47:03.191 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/34 file after the snapshot 2023-10-10T18:47:03.192 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/35 file after the snapshot 2023-10-10T18:47:03.193 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/36 file after the snapshot 2023-10-10T18:47:03.193 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/37 file after the snapshot 2023-10-10T18:47:03.194 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/38 file after the snapshot 2023-10-10T18:47:03.195 INFO:tasks.workunit.client.0.smithi175.stdout: create 127/39 file after the snapshot 2023-10-10T18:47:03.196 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/.snap/snap-subdir-test 2023-10-10T18:47:03.222 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/30 file after the snapshot 2023-10-10T18:47:03.223 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/31 file after the snapshot 2023-10-10T18:47:03.224 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/32 file after the snapshot 2023-10-10T18:47:03.225 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/33 file after the snapshot 2023-10-10T18:47:03.225 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/34 file after the snapshot 2023-10-10T18:47:03.226 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/35 file after the snapshot 2023-10-10T18:47:03.227 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/36 file after the snapshot 2023-10-10T18:47:03.228 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/37 file after the snapshot 2023-10-10T18:47:03.229 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/38 file after the snapshot 2023-10-10T18:47:03.230 INFO:tasks.workunit.client.0.smithi175.stdout: create 128/39 file after the snapshot 2023-10-10T18:47:03.230 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/.snap/snap-subdir-test 2023-10-10T18:47:03.258 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/30 file after the snapshot 2023-10-10T18:47:03.259 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/31 file after the snapshot 2023-10-10T18:47:03.260 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/32 file after the snapshot 2023-10-10T18:47:03.261 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/33 file after the snapshot 2023-10-10T18:47:03.262 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/34 file after the snapshot 2023-10-10T18:47:03.263 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/35 file after the snapshot 2023-10-10T18:47:03.264 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/36 file after the snapshot 2023-10-10T18:47:03.265 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/37 file after the snapshot 2023-10-10T18:47:03.266 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/38 file after the snapshot 2023-10-10T18:47:03.266 INFO:tasks.workunit.client.0.smithi175.stdout: create 129/39 file after the snapshot 2023-10-10T18:47:03.267 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/.snap/snap-subdir-test 2023-10-10T18:47:03.451 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/30 file after the snapshot 2023-10-10T18:47:03.454 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/31 file after the snapshot 2023-10-10T18:47:03.455 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/32 file after the snapshot 2023-10-10T18:47:03.455 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/33 file after the snapshot 2023-10-10T18:47:03.456 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/34 file after the snapshot 2023-10-10T18:47:03.462 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/35 file after the snapshot 2023-10-10T18:47:03.462 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/36 file after the snapshot 2023-10-10T18:47:03.463 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/37 file after the snapshot 2023-10-10T18:47:03.463 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/38 file after the snapshot 2023-10-10T18:47:03.464 INFO:tasks.workunit.client.0.smithi175.stdout: create 130/39 file after the snapshot 2023-10-10T18:47:03.465 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/.snap/snap-subdir-test 2023-10-10T18:47:03.937 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/30 file after the snapshot 2023-10-10T18:47:03.938 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/31 file after the snapshot 2023-10-10T18:47:03.939 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/32 file after the snapshot 2023-10-10T18:47:03.940 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/33 file after the snapshot 2023-10-10T18:47:03.941 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/34 file after the snapshot 2023-10-10T18:47:03.942 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/35 file after the snapshot 2023-10-10T18:47:03.943 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/36 file after the snapshot 2023-10-10T18:47:03.943 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/37 file after the snapshot 2023-10-10T18:47:03.944 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/38 file after the snapshot 2023-10-10T18:47:03.945 INFO:tasks.workunit.client.0.smithi175.stdout: create 131/39 file after the snapshot 2023-10-10T18:47:03.946 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/.snap/snap-subdir-test 2023-10-10T18:47:04.602 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/30 file after the snapshot 2023-10-10T18:47:04.603 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/31 file after the snapshot 2023-10-10T18:47:04.604 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/32 file after the snapshot 2023-10-10T18:47:04.605 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/33 file after the snapshot 2023-10-10T18:47:04.606 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/34 file after the snapshot 2023-10-10T18:47:04.607 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/35 file after the snapshot 2023-10-10T18:47:04.608 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/36 file after the snapshot 2023-10-10T18:47:04.609 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/37 file after the snapshot 2023-10-10T18:47:04.609 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/38 file after the snapshot 2023-10-10T18:47:04.611 INFO:tasks.workunit.client.0.smithi175.stdout: create 132/39 file after the snapshot 2023-10-10T18:47:04.612 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/.snap/snap-subdir-test 2023-10-10T18:47:04.637 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/30 file after the snapshot 2023-10-10T18:47:04.640 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/31 file after the snapshot 2023-10-10T18:47:04.641 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/32 file after the snapshot 2023-10-10T18:47:04.642 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/33 file after the snapshot 2023-10-10T18:47:04.643 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/34 file after the snapshot 2023-10-10T18:47:04.644 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/35 file after the snapshot 2023-10-10T18:47:04.645 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/36 file after the snapshot 2023-10-10T18:47:04.646 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/37 file after the snapshot 2023-10-10T18:47:04.648 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/38 file after the snapshot 2023-10-10T18:47:04.649 INFO:tasks.workunit.client.0.smithi175.stdout: create 133/39 file after the snapshot 2023-10-10T18:47:04.650 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/.snap/snap-subdir-test 2023-10-10T18:47:04.672 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/30 file after the snapshot 2023-10-10T18:47:04.674 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/31 file after the snapshot 2023-10-10T18:47:04.676 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/32 file after the snapshot 2023-10-10T18:47:04.676 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/33 file after the snapshot 2023-10-10T18:47:04.677 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/34 file after the snapshot 2023-10-10T18:47:04.677 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/35 file after the snapshot 2023-10-10T18:47:04.678 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/36 file after the snapshot 2023-10-10T18:47:04.678 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/37 file after the snapshot 2023-10-10T18:47:04.679 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/38 file after the snapshot 2023-10-10T18:47:04.680 INFO:tasks.workunit.client.0.smithi175.stdout: create 134/39 file after the snapshot 2023-10-10T18:47:04.681 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/.snap/snap-subdir-test 2023-10-10T18:47:04.708 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/30 file after the snapshot 2023-10-10T18:47:04.709 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/31 file after the snapshot 2023-10-10T18:47:04.710 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/32 file after the snapshot 2023-10-10T18:47:04.711 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/33 file after the snapshot 2023-10-10T18:47:04.711 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/34 file after the snapshot 2023-10-10T18:47:04.712 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/35 file after the snapshot 2023-10-10T18:47:04.713 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/36 file after the snapshot 2023-10-10T18:47:04.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/37 file after the snapshot 2023-10-10T18:47:04.715 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/38 file after the snapshot 2023-10-10T18:47:04.716 INFO:tasks.workunit.client.0.smithi175.stdout: create 135/39 file after the snapshot 2023-10-10T18:47:04.717 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/.snap/snap-subdir-test 2023-10-10T18:47:04.741 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/30 file after the snapshot 2023-10-10T18:47:04.742 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/31 file after the snapshot 2023-10-10T18:47:04.743 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/32 file after the snapshot 2023-10-10T18:47:04.744 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/33 file after the snapshot 2023-10-10T18:47:04.746 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/34 file after the snapshot 2023-10-10T18:47:04.747 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/35 file after the snapshot 2023-10-10T18:47:04.748 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/36 file after the snapshot 2023-10-10T18:47:04.750 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/37 file after the snapshot 2023-10-10T18:47:04.751 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/38 file after the snapshot 2023-10-10T18:47:04.752 INFO:tasks.workunit.client.0.smithi175.stdout: create 136/39 file after the snapshot 2023-10-10T18:47:04.753 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/.snap/snap-subdir-test 2023-10-10T18:47:04.774 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/30 file after the snapshot 2023-10-10T18:47:04.775 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/31 file after the snapshot 2023-10-10T18:47:04.776 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/32 file after the snapshot 2023-10-10T18:47:04.777 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/33 file after the snapshot 2023-10-10T18:47:04.778 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/34 file after the snapshot 2023-10-10T18:47:04.779 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/35 file after the snapshot 2023-10-10T18:47:04.780 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/36 file after the snapshot 2023-10-10T18:47:04.781 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/37 file after the snapshot 2023-10-10T18:47:04.782 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/38 file after the snapshot 2023-10-10T18:47:04.783 INFO:tasks.workunit.client.0.smithi175.stdout: create 137/39 file after the snapshot 2023-10-10T18:47:04.784 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/.snap/snap-subdir-test 2023-10-10T18:47:04.809 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/30 file after the snapshot 2023-10-10T18:47:04.809 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/31 file after the snapshot 2023-10-10T18:47:04.810 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/32 file after the snapshot 2023-10-10T18:47:04.810 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/33 file after the snapshot 2023-10-10T18:47:04.811 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/34 file after the snapshot 2023-10-10T18:47:04.812 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/35 file after the snapshot 2023-10-10T18:47:04.812 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/36 file after the snapshot 2023-10-10T18:47:04.813 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/37 file after the snapshot 2023-10-10T18:47:04.813 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/38 file after the snapshot 2023-10-10T18:47:04.814 INFO:tasks.workunit.client.0.smithi175.stdout: create 138/39 file after the snapshot 2023-10-10T18:47:04.815 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/.snap/snap-subdir-test 2023-10-10T18:47:04.836 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/30 file after the snapshot 2023-10-10T18:47:04.837 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/31 file after the snapshot 2023-10-10T18:47:04.838 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/32 file after the snapshot 2023-10-10T18:47:04.839 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/33 file after the snapshot 2023-10-10T18:47:04.840 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/34 file after the snapshot 2023-10-10T18:47:04.841 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/35 file after the snapshot 2023-10-10T18:47:04.842 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/36 file after the snapshot 2023-10-10T18:47:04.843 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/37 file after the snapshot 2023-10-10T18:47:04.844 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/38 file after the snapshot 2023-10-10T18:47:04.845 INFO:tasks.workunit.client.0.smithi175.stdout: create 139/39 file after the snapshot 2023-10-10T18:47:04.846 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/.snap/snap-subdir-test 2023-10-10T18:47:04.875 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/30 file after the snapshot 2023-10-10T18:47:04.877 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/31 file after the snapshot 2023-10-10T18:47:04.878 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/32 file after the snapshot 2023-10-10T18:47:04.879 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/33 file after the snapshot 2023-10-10T18:47:04.880 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/34 file after the snapshot 2023-10-10T18:47:04.881 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/35 file after the snapshot 2023-10-10T18:47:04.882 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/36 file after the snapshot 2023-10-10T18:47:04.883 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/37 file after the snapshot 2023-10-10T18:47:04.884 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/38 file after the snapshot 2023-10-10T18:47:04.885 INFO:tasks.workunit.client.0.smithi175.stdout: create 140/39 file after the snapshot 2023-10-10T18:47:04.885 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/.snap/snap-subdir-test 2023-10-10T18:47:04.908 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/30 file after the snapshot 2023-10-10T18:47:04.909 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/31 file after the snapshot 2023-10-10T18:47:04.910 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/32 file after the snapshot 2023-10-10T18:47:04.911 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/33 file after the snapshot 2023-10-10T18:47:04.912 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/34 file after the snapshot 2023-10-10T18:47:04.913 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/35 file after the snapshot 2023-10-10T18:47:04.914 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/36 file after the snapshot 2023-10-10T18:47:04.915 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/37 file after the snapshot 2023-10-10T18:47:04.916 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/38 file after the snapshot 2023-10-10T18:47:04.917 INFO:tasks.workunit.client.0.smithi175.stdout: create 141/39 file after the snapshot 2023-10-10T18:47:04.918 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/.snap/snap-subdir-test 2023-10-10T18:47:04.945 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/30 file after the snapshot 2023-10-10T18:47:04.947 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/31 file after the snapshot 2023-10-10T18:47:04.947 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/32 file after the snapshot 2023-10-10T18:47:04.947 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/33 file after the snapshot 2023-10-10T18:47:04.949 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/34 file after the snapshot 2023-10-10T18:47:04.949 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/35 file after the snapshot 2023-10-10T18:47:04.950 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/36 file after the snapshot 2023-10-10T18:47:04.950 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/37 file after the snapshot 2023-10-10T18:47:04.951 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/38 file after the snapshot 2023-10-10T18:47:04.951 INFO:tasks.workunit.client.0.smithi175.stdout: create 142/39 file after the snapshot 2023-10-10T18:47:04.952 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/.snap/snap-subdir-test 2023-10-10T18:47:04.972 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/30 file after the snapshot 2023-10-10T18:47:04.974 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/31 file after the snapshot 2023-10-10T18:47:04.977 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/32 file after the snapshot 2023-10-10T18:47:04.977 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/33 file after the snapshot 2023-10-10T18:47:04.978 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/34 file after the snapshot 2023-10-10T18:47:04.978 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/35 file after the snapshot 2023-10-10T18:47:04.979 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/36 file after the snapshot 2023-10-10T18:47:04.979 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/37 file after the snapshot 2023-10-10T18:47:04.980 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/38 file after the snapshot 2023-10-10T18:47:04.981 INFO:tasks.workunit.client.0.smithi175.stdout: create 143/39 file after the snapshot 2023-10-10T18:47:04.981 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/.snap/snap-subdir-test 2023-10-10T18:47:05.005 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/30 file after the snapshot 2023-10-10T18:47:05.006 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/31 file after the snapshot 2023-10-10T18:47:05.006 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/32 file after the snapshot 2023-10-10T18:47:05.007 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/33 file after the snapshot 2023-10-10T18:47:05.008 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/34 file after the snapshot 2023-10-10T18:47:05.008 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/35 file after the snapshot 2023-10-10T18:47:05.009 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/36 file after the snapshot 2023-10-10T18:47:05.010 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/37 file after the snapshot 2023-10-10T18:47:05.011 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/38 file after the snapshot 2023-10-10T18:47:05.012 INFO:tasks.workunit.client.0.smithi175.stdout: create 144/39 file after the snapshot 2023-10-10T18:47:05.012 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/.snap/snap-subdir-test 2023-10-10T18:47:05.038 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/30 file after the snapshot 2023-10-10T18:47:05.039 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/31 file after the snapshot 2023-10-10T18:47:05.040 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/32 file after the snapshot 2023-10-10T18:47:05.040 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/33 file after the snapshot 2023-10-10T18:47:05.041 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/34 file after the snapshot 2023-10-10T18:47:05.042 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/35 file after the snapshot 2023-10-10T18:47:05.043 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/36 file after the snapshot 2023-10-10T18:47:05.043 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/37 file after the snapshot 2023-10-10T18:47:05.044 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/38 file after the snapshot 2023-10-10T18:47:05.045 INFO:tasks.workunit.client.0.smithi175.stdout: create 145/39 file after the snapshot 2023-10-10T18:47:05.046 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/.snap/snap-subdir-test 2023-10-10T18:47:05.070 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/30 file after the snapshot 2023-10-10T18:47:05.071 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/31 file after the snapshot 2023-10-10T18:47:05.072 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/32 file after the snapshot 2023-10-10T18:47:05.074 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/33 file after the snapshot 2023-10-10T18:47:05.075 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/34 file after the snapshot 2023-10-10T18:47:05.076 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/35 file after the snapshot 2023-10-10T18:47:05.077 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/36 file after the snapshot 2023-10-10T18:47:05.078 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/37 file after the snapshot 2023-10-10T18:47:05.079 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/38 file after the snapshot 2023-10-10T18:47:05.080 INFO:tasks.workunit.client.0.smithi175.stdout: create 146/39 file after the snapshot 2023-10-10T18:47:05.081 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/.snap/snap-subdir-test 2023-10-10T18:47:05.108 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/30 file after the snapshot 2023-10-10T18:47:05.109 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/31 file after the snapshot 2023-10-10T18:47:05.110 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/32 file after the snapshot 2023-10-10T18:47:05.110 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/33 file after the snapshot 2023-10-10T18:47:05.111 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/34 file after the snapshot 2023-10-10T18:47:05.112 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/35 file after the snapshot 2023-10-10T18:47:05.112 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/36 file after the snapshot 2023-10-10T18:47:05.113 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/37 file after the snapshot 2023-10-10T18:47:05.114 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/38 file after the snapshot 2023-10-10T18:47:05.114 INFO:tasks.workunit.client.0.smithi175.stdout: create 147/39 file after the snapshot 2023-10-10T18:47:05.115 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/.snap/snap-subdir-test 2023-10-10T18:47:05.136 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/30 file after the snapshot 2023-10-10T18:47:05.137 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/31 file after the snapshot 2023-10-10T18:47:05.138 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/32 file after the snapshot 2023-10-10T18:47:05.139 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/33 file after the snapshot 2023-10-10T18:47:05.140 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/34 file after the snapshot 2023-10-10T18:47:05.140 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/35 file after the snapshot 2023-10-10T18:47:05.141 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/36 file after the snapshot 2023-10-10T18:47:05.142 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/37 file after the snapshot 2023-10-10T18:47:05.143 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/38 file after the snapshot 2023-10-10T18:47:05.143 INFO:tasks.workunit.client.0.smithi175.stdout: create 148/39 file after the snapshot 2023-10-10T18:47:05.144 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/.snap/snap-subdir-test 2023-10-10T18:47:05.171 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/30 file after the snapshot 2023-10-10T18:47:05.171 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/31 file after the snapshot 2023-10-10T18:47:05.172 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/32 file after the snapshot 2023-10-10T18:47:05.173 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/33 file after the snapshot 2023-10-10T18:47:05.173 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/34 file after the snapshot 2023-10-10T18:47:05.174 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/35 file after the snapshot 2023-10-10T18:47:05.175 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/36 file after the snapshot 2023-10-10T18:47:05.175 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/37 file after the snapshot 2023-10-10T18:47:05.176 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/38 file after the snapshot 2023-10-10T18:47:05.177 INFO:tasks.workunit.client.0.smithi175.stdout: create 149/39 file after the snapshot 2023-10-10T18:47:05.177 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/.snap/snap-subdir-test 2023-10-10T18:47:05.202 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/30 file after the snapshot 2023-10-10T18:47:05.202 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/31 file after the snapshot 2023-10-10T18:47:05.203 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/32 file after the snapshot 2023-10-10T18:47:05.204 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/33 file after the snapshot 2023-10-10T18:47:05.204 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/34 file after the snapshot 2023-10-10T18:47:05.205 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/35 file after the snapshot 2023-10-10T18:47:05.206 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/36 file after the snapshot 2023-10-10T18:47:05.206 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/37 file after the snapshot 2023-10-10T18:47:05.207 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/38 file after the snapshot 2023-10-10T18:47:05.207 INFO:tasks.workunit.client.0.smithi175.stdout: create 150/39 file after the snapshot 2023-10-10T18:47:05.208 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/.snap/snap-subdir-test 2023-10-10T18:47:05.228 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/30 file after the snapshot 2023-10-10T18:47:05.229 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/31 file after the snapshot 2023-10-10T18:47:05.230 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/32 file after the snapshot 2023-10-10T18:47:05.230 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/33 file after the snapshot 2023-10-10T18:47:05.231 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/34 file after the snapshot 2023-10-10T18:47:05.231 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/35 file after the snapshot 2023-10-10T18:47:05.232 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/36 file after the snapshot 2023-10-10T18:47:05.233 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/37 file after the snapshot 2023-10-10T18:47:05.233 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/38 file after the snapshot 2023-10-10T18:47:05.234 INFO:tasks.workunit.client.0.smithi175.stdout: create 151/39 file after the snapshot 2023-10-10T18:47:05.234 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/.snap/snap-subdir-test 2023-10-10T18:47:05.257 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/30 file after the snapshot 2023-10-10T18:47:05.258 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/31 file after the snapshot 2023-10-10T18:47:05.259 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/32 file after the snapshot 2023-10-10T18:47:05.260 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/33 file after the snapshot 2023-10-10T18:47:05.261 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/34 file after the snapshot 2023-10-10T18:47:05.262 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/35 file after the snapshot 2023-10-10T18:47:05.263 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/36 file after the snapshot 2023-10-10T18:47:05.264 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/37 file after the snapshot 2023-10-10T18:47:05.265 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/38 file after the snapshot 2023-10-10T18:47:05.266 INFO:tasks.workunit.client.0.smithi175.stdout: create 152/39 file after the snapshot 2023-10-10T18:47:05.266 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/.snap/snap-subdir-test 2023-10-10T18:47:05.292 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/30 file after the snapshot 2023-10-10T18:47:05.294 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/31 file after the snapshot 2023-10-10T18:47:05.295 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/32 file after the snapshot 2023-10-10T18:47:05.295 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/33 file after the snapshot 2023-10-10T18:47:05.296 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/34 file after the snapshot 2023-10-10T18:47:05.296 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/35 file after the snapshot 2023-10-10T18:47:05.297 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/36 file after the snapshot 2023-10-10T18:47:05.298 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/37 file after the snapshot 2023-10-10T18:47:05.299 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/38 file after the snapshot 2023-10-10T18:47:05.300 INFO:tasks.workunit.client.0.smithi175.stdout: create 153/39 file after the snapshot 2023-10-10T18:47:05.301 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/.snap/snap-subdir-test 2023-10-10T18:47:05.326 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/30 file after the snapshot 2023-10-10T18:47:05.327 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/31 file after the snapshot 2023-10-10T18:47:05.328 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/32 file after the snapshot 2023-10-10T18:47:05.329 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/33 file after the snapshot 2023-10-10T18:47:05.330 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/34 file after the snapshot 2023-10-10T18:47:05.331 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/35 file after the snapshot 2023-10-10T18:47:05.331 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/36 file after the snapshot 2023-10-10T18:47:05.332 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/37 file after the snapshot 2023-10-10T18:47:05.333 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/38 file after the snapshot 2023-10-10T18:47:05.334 INFO:tasks.workunit.client.0.smithi175.stdout: create 154/39 file after the snapshot 2023-10-10T18:47:05.335 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/.snap/snap-subdir-test 2023-10-10T18:47:05.358 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/30 file after the snapshot 2023-10-10T18:47:05.359 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/31 file after the snapshot 2023-10-10T18:47:05.360 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/32 file after the snapshot 2023-10-10T18:47:05.361 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/33 file after the snapshot 2023-10-10T18:47:05.362 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/34 file after the snapshot 2023-10-10T18:47:05.363 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/35 file after the snapshot 2023-10-10T18:47:05.364 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/36 file after the snapshot 2023-10-10T18:47:05.365 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/37 file after the snapshot 2023-10-10T18:47:05.365 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/38 file after the snapshot 2023-10-10T18:47:05.366 INFO:tasks.workunit.client.0.smithi175.stdout: create 155/39 file after the snapshot 2023-10-10T18:47:05.367 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/.snap/snap-subdir-test 2023-10-10T18:47:05.387 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/30 file after the snapshot 2023-10-10T18:47:05.388 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/31 file after the snapshot 2023-10-10T18:47:05.389 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/32 file after the snapshot 2023-10-10T18:47:05.390 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/33 file after the snapshot 2023-10-10T18:47:05.391 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/34 file after the snapshot 2023-10-10T18:47:05.392 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/35 file after the snapshot 2023-10-10T18:47:05.393 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/36 file after the snapshot 2023-10-10T18:47:05.394 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/37 file after the snapshot 2023-10-10T18:47:05.395 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/38 file after the snapshot 2023-10-10T18:47:05.395 INFO:tasks.workunit.client.0.smithi175.stdout: create 156/39 file after the snapshot 2023-10-10T18:47:05.396 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/.snap/snap-subdir-test 2023-10-10T18:47:06.620 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/30 file after the snapshot 2023-10-10T18:47:06.621 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/31 file after the snapshot 2023-10-10T18:47:06.622 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/32 file after the snapshot 2023-10-10T18:47:06.622 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/33 file after the snapshot 2023-10-10T18:47:06.623 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/34 file after the snapshot 2023-10-10T18:47:06.624 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/35 file after the snapshot 2023-10-10T18:47:06.625 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/36 file after the snapshot 2023-10-10T18:47:06.625 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/37 file after the snapshot 2023-10-10T18:47:06.626 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/38 file after the snapshot 2023-10-10T18:47:06.627 INFO:tasks.workunit.client.0.smithi175.stdout: create 157/39 file after the snapshot 2023-10-10T18:47:06.628 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/.snap/snap-subdir-test 2023-10-10T18:47:06.656 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/30 file after the snapshot 2023-10-10T18:47:06.657 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/31 file after the snapshot 2023-10-10T18:47:06.657 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/32 file after the snapshot 2023-10-10T18:47:06.658 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/33 file after the snapshot 2023-10-10T18:47:06.659 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/34 file after the snapshot 2023-10-10T18:47:06.659 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/35 file after the snapshot 2023-10-10T18:47:06.660 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/36 file after the snapshot 2023-10-10T18:47:06.661 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/37 file after the snapshot 2023-10-10T18:47:06.661 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/38 file after the snapshot 2023-10-10T18:47:06.662 INFO:tasks.workunit.client.0.smithi175.stdout: create 158/39 file after the snapshot 2023-10-10T18:47:06.663 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/.snap/snap-subdir-test 2023-10-10T18:47:06.692 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/30 file after the snapshot 2023-10-10T18:47:06.693 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/31 file after the snapshot 2023-10-10T18:47:06.694 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/32 file after the snapshot 2023-10-10T18:47:06.695 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/33 file after the snapshot 2023-10-10T18:47:06.696 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/34 file after the snapshot 2023-10-10T18:47:06.697 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/35 file after the snapshot 2023-10-10T18:47:06.698 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/36 file after the snapshot 2023-10-10T18:47:06.699 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/37 file after the snapshot 2023-10-10T18:47:06.700 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/38 file after the snapshot 2023-10-10T18:47:06.701 INFO:tasks.workunit.client.0.smithi175.stdout: create 159/39 file after the snapshot 2023-10-10T18:47:06.702 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/.snap/snap-subdir-test 2023-10-10T18:47:06.731 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/30 file after the snapshot 2023-10-10T18:47:06.732 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/31 file after the snapshot 2023-10-10T18:47:06.733 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/32 file after the snapshot 2023-10-10T18:47:06.734 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/33 file after the snapshot 2023-10-10T18:47:06.736 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/34 file after the snapshot 2023-10-10T18:47:06.737 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/35 file after the snapshot 2023-10-10T18:47:06.738 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/36 file after the snapshot 2023-10-10T18:47:06.739 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/37 file after the snapshot 2023-10-10T18:47:06.740 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/38 file after the snapshot 2023-10-10T18:47:06.742 INFO:tasks.workunit.client.0.smithi175.stdout: create 160/39 file after the snapshot 2023-10-10T18:47:06.743 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/.snap/snap-subdir-test 2023-10-10T18:47:06.774 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/30 file after the snapshot 2023-10-10T18:47:07.038 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/31 file after the snapshot 2023-10-10T18:47:07.039 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/32 file after the snapshot 2023-10-10T18:47:07.040 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/33 file after the snapshot 2023-10-10T18:47:07.041 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/34 file after the snapshot 2023-10-10T18:47:07.042 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/35 file after the snapshot 2023-10-10T18:47:07.043 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/36 file after the snapshot 2023-10-10T18:47:07.044 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/37 file after the snapshot 2023-10-10T18:47:07.045 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/38 file after the snapshot 2023-10-10T18:47:07.046 INFO:tasks.workunit.client.0.smithi175.stdout: create 161/39 file after the snapshot 2023-10-10T18:47:07.047 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/.snap/snap-subdir-test 2023-10-10T18:47:07.080 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/30 file after the snapshot 2023-10-10T18:47:07.081 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/31 file after the snapshot 2023-10-10T18:47:07.082 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/32 file after the snapshot 2023-10-10T18:47:07.083 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/33 file after the snapshot 2023-10-10T18:47:07.084 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/34 file after the snapshot 2023-10-10T18:47:07.085 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/35 file after the snapshot 2023-10-10T18:47:07.086 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/36 file after the snapshot 2023-10-10T18:47:07.087 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/37 file after the snapshot 2023-10-10T18:47:07.088 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/38 file after the snapshot 2023-10-10T18:47:07.089 INFO:tasks.workunit.client.0.smithi175.stdout: create 162/39 file after the snapshot 2023-10-10T18:47:07.090 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/.snap/snap-subdir-test 2023-10-10T18:47:07.427 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/30 file after the snapshot 2023-10-10T18:47:07.428 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/31 file after the snapshot 2023-10-10T18:47:07.429 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/32 file after the snapshot 2023-10-10T18:47:07.430 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/33 file after the snapshot 2023-10-10T18:47:07.431 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/34 file after the snapshot 2023-10-10T18:47:07.432 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/35 file after the snapshot 2023-10-10T18:47:07.433 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/36 file after the snapshot 2023-10-10T18:47:07.434 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/37 file after the snapshot 2023-10-10T18:47:07.599 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/38 file after the snapshot 2023-10-10T18:47:07.738 INFO:tasks.workunit.client.0.smithi175.stdout: create 163/39 file after the snapshot 2023-10-10T18:47:07.739 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/.snap/snap-subdir-test 2023-10-10T18:47:07.766 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/30 file after the snapshot 2023-10-10T18:47:08.440 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/31 file after the snapshot 2023-10-10T18:47:08.700 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/32 file after the snapshot 2023-10-10T18:47:08.700 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/33 file after the snapshot 2023-10-10T18:47:08.701 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/34 file after the snapshot 2023-10-10T18:47:08.701 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/35 file after the snapshot 2023-10-10T18:47:08.702 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/36 file after the snapshot 2023-10-10T18:47:08.703 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/37 file after the snapshot 2023-10-10T18:47:08.703 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/38 file after the snapshot 2023-10-10T18:47:08.704 INFO:tasks.workunit.client.0.smithi175.stdout: create 164/39 file after the snapshot 2023-10-10T18:47:08.705 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/.snap/snap-subdir-test 2023-10-10T18:47:08.735 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/30 file after the snapshot 2023-10-10T18:47:08.736 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/31 file after the snapshot 2023-10-10T18:47:08.736 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/32 file after the snapshot 2023-10-10T18:47:08.737 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/33 file after the snapshot 2023-10-10T18:47:08.738 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/34 file after the snapshot 2023-10-10T18:47:08.739 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/35 file after the snapshot 2023-10-10T18:47:08.740 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/36 file after the snapshot 2023-10-10T18:47:08.741 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/37 file after the snapshot 2023-10-10T18:47:08.741 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/38 file after the snapshot 2023-10-10T18:47:08.742 INFO:tasks.workunit.client.0.smithi175.stdout: create 165/39 file after the snapshot 2023-10-10T18:47:08.743 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/.snap/snap-subdir-test 2023-10-10T18:47:08.774 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/30 file after the snapshot 2023-10-10T18:47:08.774 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/31 file after the snapshot 2023-10-10T18:47:08.775 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/32 file after the snapshot 2023-10-10T18:47:08.776 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/33 file after the snapshot 2023-10-10T18:47:08.777 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/34 file after the snapshot 2023-10-10T18:47:08.777 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/35 file after the snapshot 2023-10-10T18:47:08.778 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/36 file after the snapshot 2023-10-10T18:47:08.779 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/37 file after the snapshot 2023-10-10T18:47:08.779 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/38 file after the snapshot 2023-10-10T18:47:08.780 INFO:tasks.workunit.client.0.smithi175.stdout: create 166/39 file after the snapshot 2023-10-10T18:47:08.781 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/.snap/snap-subdir-test 2023-10-10T18:47:08.801 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/30 file after the snapshot 2023-10-10T18:47:08.802 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/31 file after the snapshot 2023-10-10T18:47:08.802 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/32 file after the snapshot 2023-10-10T18:47:08.803 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/33 file after the snapshot 2023-10-10T18:47:08.804 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/34 file after the snapshot 2023-10-10T18:47:08.804 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/35 file after the snapshot 2023-10-10T18:47:08.805 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/36 file after the snapshot 2023-10-10T18:47:08.805 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/37 file after the snapshot 2023-10-10T18:47:08.806 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/38 file after the snapshot 2023-10-10T18:47:08.807 INFO:tasks.workunit.client.0.smithi175.stdout: create 167/39 file after the snapshot 2023-10-10T18:47:08.807 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/.snap/snap-subdir-test 2023-10-10T18:47:09.331 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/30 file after the snapshot 2023-10-10T18:47:09.332 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/31 file after the snapshot 2023-10-10T18:47:09.333 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/32 file after the snapshot 2023-10-10T18:47:09.335 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/33 file after the snapshot 2023-10-10T18:47:09.336 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/34 file after the snapshot 2023-10-10T18:47:09.337 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/35 file after the snapshot 2023-10-10T18:47:09.338 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/36 file after the snapshot 2023-10-10T18:47:09.339 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/37 file after the snapshot 2023-10-10T18:47:09.340 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/38 file after the snapshot 2023-10-10T18:47:09.341 INFO:tasks.workunit.client.0.smithi175.stdout: create 168/39 file after the snapshot 2023-10-10T18:47:09.342 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/.snap/snap-subdir-test 2023-10-10T18:47:09.364 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/30 file after the snapshot 2023-10-10T18:47:09.366 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/31 file after the snapshot 2023-10-10T18:47:09.367 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/32 file after the snapshot 2023-10-10T18:47:09.368 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/33 file after the snapshot 2023-10-10T18:47:09.369 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/34 file after the snapshot 2023-10-10T18:47:09.369 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/35 file after the snapshot 2023-10-10T18:47:09.370 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/36 file after the snapshot 2023-10-10T18:47:09.370 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/37 file after the snapshot 2023-10-10T18:47:09.371 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/38 file after the snapshot 2023-10-10T18:47:09.371 INFO:tasks.workunit.client.0.smithi175.stdout: create 169/39 file after the snapshot 2023-10-10T18:47:09.372 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/.snap/snap-subdir-test 2023-10-10T18:47:09.393 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/30 file after the snapshot 2023-10-10T18:47:09.393 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/31 file after the snapshot 2023-10-10T18:47:09.394 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/32 file after the snapshot 2023-10-10T18:47:09.395 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/33 file after the snapshot 2023-10-10T18:47:09.395 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/34 file after the snapshot 2023-10-10T18:47:09.396 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/35 file after the snapshot 2023-10-10T18:47:09.397 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/36 file after the snapshot 2023-10-10T18:47:09.397 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/37 file after the snapshot 2023-10-10T18:47:09.398 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/38 file after the snapshot 2023-10-10T18:47:09.662 INFO:tasks.workunit.client.0.smithi175.stdout: create 170/39 file after the snapshot 2023-10-10T18:47:09.663 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/.snap/snap-subdir-test 2023-10-10T18:47:09.683 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/30 file after the snapshot 2023-10-10T18:47:09.686 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/31 file after the snapshot 2023-10-10T18:47:09.686 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/32 file after the snapshot 2023-10-10T18:47:09.686 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/33 file after the snapshot 2023-10-10T18:47:09.687 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/34 file after the snapshot 2023-10-10T18:47:09.687 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/35 file after the snapshot 2023-10-10T18:47:09.687 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/36 file after the snapshot 2023-10-10T18:47:09.688 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/37 file after the snapshot 2023-10-10T18:47:09.688 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/38 file after the snapshot 2023-10-10T18:47:09.689 INFO:tasks.workunit.client.0.smithi175.stdout: create 171/39 file after the snapshot 2023-10-10T18:47:09.689 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/.snap/snap-subdir-test 2023-10-10T18:47:09.710 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/30 file after the snapshot 2023-10-10T18:47:09.712 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/31 file after the snapshot 2023-10-10T18:47:09.712 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/32 file after the snapshot 2023-10-10T18:47:09.713 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/33 file after the snapshot 2023-10-10T18:47:09.713 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/34 file after the snapshot 2023-10-10T18:47:09.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/35 file after the snapshot 2023-10-10T18:47:09.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/36 file after the snapshot 2023-10-10T18:47:09.715 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/37 file after the snapshot 2023-10-10T18:47:09.715 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/38 file after the snapshot 2023-10-10T18:47:09.716 INFO:tasks.workunit.client.0.smithi175.stdout: create 172/39 file after the snapshot 2023-10-10T18:47:09.717 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/.snap/snap-subdir-test 2023-10-10T18:47:09.736 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/30 file after the snapshot 2023-10-10T18:47:09.737 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/31 file after the snapshot 2023-10-10T18:47:09.738 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/32 file after the snapshot 2023-10-10T18:47:09.739 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/33 file after the snapshot 2023-10-10T18:47:09.740 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/34 file after the snapshot 2023-10-10T18:47:09.741 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/35 file after the snapshot 2023-10-10T18:47:09.742 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/36 file after the snapshot 2023-10-10T18:47:09.743 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/37 file after the snapshot 2023-10-10T18:47:09.744 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/38 file after the snapshot 2023-10-10T18:47:09.745 INFO:tasks.workunit.client.0.smithi175.stdout: create 173/39 file after the snapshot 2023-10-10T18:47:09.746 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/.snap/snap-subdir-test 2023-10-10T18:47:09.768 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/30 file after the snapshot 2023-10-10T18:47:09.769 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/31 file after the snapshot 2023-10-10T18:47:09.770 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/32 file after the snapshot 2023-10-10T18:47:09.771 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/33 file after the snapshot 2023-10-10T18:47:09.772 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/34 file after the snapshot 2023-10-10T18:47:09.773 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/35 file after the snapshot 2023-10-10T18:47:09.774 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/36 file after the snapshot 2023-10-10T18:47:09.775 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/37 file after the snapshot 2023-10-10T18:47:09.776 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/38 file after the snapshot 2023-10-10T18:47:09.777 INFO:tasks.workunit.client.0.smithi175.stdout: create 174/39 file after the snapshot 2023-10-10T18:47:09.778 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/.snap/snap-subdir-test 2023-10-10T18:47:09.798 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/30 file after the snapshot 2023-10-10T18:47:09.799 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/31 file after the snapshot 2023-10-10T18:47:09.800 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/32 file after the snapshot 2023-10-10T18:47:09.801 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/33 file after the snapshot 2023-10-10T18:47:09.802 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/34 file after the snapshot 2023-10-10T18:47:09.803 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/35 file after the snapshot 2023-10-10T18:47:09.804 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/36 file after the snapshot 2023-10-10T18:47:09.805 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/37 file after the snapshot 2023-10-10T18:47:09.806 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/38 file after the snapshot 2023-10-10T18:47:09.807 INFO:tasks.workunit.client.0.smithi175.stdout: create 175/39 file after the snapshot 2023-10-10T18:47:09.808 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/.snap/snap-subdir-test 2023-10-10T18:47:09.827 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/30 file after the snapshot 2023-10-10T18:47:09.829 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/31 file after the snapshot 2023-10-10T18:47:09.829 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/32 file after the snapshot 2023-10-10T18:47:09.830 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/33 file after the snapshot 2023-10-10T18:47:09.831 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/34 file after the snapshot 2023-10-10T18:47:09.832 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/35 file after the snapshot 2023-10-10T18:47:09.832 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/36 file after the snapshot 2023-10-10T18:47:09.833 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/37 file after the snapshot 2023-10-10T18:47:09.834 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/38 file after the snapshot 2023-10-10T18:47:09.835 INFO:tasks.workunit.client.0.smithi175.stdout: create 176/39 file after the snapshot 2023-10-10T18:47:09.835 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/.snap/snap-subdir-test 2023-10-10T18:47:10.556 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/30 file after the snapshot 2023-10-10T18:47:10.557 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/31 file after the snapshot 2023-10-10T18:47:10.558 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/32 file after the snapshot 2023-10-10T18:47:10.559 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/33 file after the snapshot 2023-10-10T18:47:10.559 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/34 file after the snapshot 2023-10-10T18:47:10.560 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/35 file after the snapshot 2023-10-10T18:47:10.561 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/36 file after the snapshot 2023-10-10T18:47:10.561 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/37 file after the snapshot 2023-10-10T18:47:10.562 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/38 file after the snapshot 2023-10-10T18:47:10.563 INFO:tasks.workunit.client.0.smithi175.stdout: create 177/39 file after the snapshot 2023-10-10T18:47:10.563 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/.snap/snap-subdir-test 2023-10-10T18:47:10.663 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/30 file after the snapshot 2023-10-10T18:47:10.665 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/31 file after the snapshot 2023-10-10T18:47:10.666 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/32 file after the snapshot 2023-10-10T18:47:10.667 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/33 file after the snapshot 2023-10-10T18:47:10.669 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/34 file after the snapshot 2023-10-10T18:47:10.670 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/35 file after the snapshot 2023-10-10T18:47:10.671 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/36 file after the snapshot 2023-10-10T18:47:10.672 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/37 file after the snapshot 2023-10-10T18:47:10.673 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/38 file after the snapshot 2023-10-10T18:47:10.674 INFO:tasks.workunit.client.0.smithi175.stdout: create 178/39 file after the snapshot 2023-10-10T18:47:10.675 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/.snap/snap-subdir-test 2023-10-10T18:47:10.701 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/30 file after the snapshot 2023-10-10T18:47:10.703 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/31 file after the snapshot 2023-10-10T18:47:10.704 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/32 file after the snapshot 2023-10-10T18:47:10.705 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/33 file after the snapshot 2023-10-10T18:47:10.706 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/34 file after the snapshot 2023-10-10T18:47:10.709 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/35 file after the snapshot 2023-10-10T18:47:10.709 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/36 file after the snapshot 2023-10-10T18:47:10.710 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/37 file after the snapshot 2023-10-10T18:47:10.711 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/38 file after the snapshot 2023-10-10T18:47:10.712 INFO:tasks.workunit.client.0.smithi175.stdout: create 179/39 file after the snapshot 2023-10-10T18:47:10.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/.snap/snap-subdir-test 2023-10-10T18:47:10.740 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/30 file after the snapshot 2023-10-10T18:47:10.741 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/31 file after the snapshot 2023-10-10T18:47:10.743 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/32 file after the snapshot 2023-10-10T18:47:10.744 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/33 file after the snapshot 2023-10-10T18:47:10.745 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/34 file after the snapshot 2023-10-10T18:47:10.745 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/35 file after the snapshot 2023-10-10T18:47:10.746 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/36 file after the snapshot 2023-10-10T18:47:10.747 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/37 file after the snapshot 2023-10-10T18:47:10.748 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/38 file after the snapshot 2023-10-10T18:47:10.749 INFO:tasks.workunit.client.0.smithi175.stdout: create 180/39 file after the snapshot 2023-10-10T18:47:10.750 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/.snap/snap-subdir-test 2023-10-10T18:47:10.774 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/30 file after the snapshot 2023-10-10T18:47:10.775 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/31 file after the snapshot 2023-10-10T18:47:10.776 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/32 file after the snapshot 2023-10-10T18:47:10.777 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/33 file after the snapshot 2023-10-10T18:47:10.778 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/34 file after the snapshot 2023-10-10T18:47:10.779 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/35 file after the snapshot 2023-10-10T18:47:10.780 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/36 file after the snapshot 2023-10-10T18:47:10.781 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/37 file after the snapshot 2023-10-10T18:47:10.782 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/38 file after the snapshot 2023-10-10T18:47:11.046 INFO:tasks.workunit.client.0.smithi175.stdout: create 181/39 file after the snapshot 2023-10-10T18:47:11.047 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/.snap/snap-subdir-test 2023-10-10T18:47:11.067 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/30 file after the snapshot 2023-10-10T18:47:11.069 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/31 file after the snapshot 2023-10-10T18:47:11.070 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/32 file after the snapshot 2023-10-10T18:47:11.071 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/33 file after the snapshot 2023-10-10T18:47:11.072 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/34 file after the snapshot 2023-10-10T18:47:11.073 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/35 file after the snapshot 2023-10-10T18:47:11.075 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/36 file after the snapshot 2023-10-10T18:47:11.075 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/37 file after the snapshot 2023-10-10T18:47:11.076 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/38 file after the snapshot 2023-10-10T18:47:11.077 INFO:tasks.workunit.client.0.smithi175.stdout: create 182/39 file after the snapshot 2023-10-10T18:47:11.078 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/.snap/snap-subdir-test 2023-10-10T18:47:11.353 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/30 file after the snapshot 2023-10-10T18:47:11.353 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/31 file after the snapshot 2023-10-10T18:47:11.354 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/32 file after the snapshot 2023-10-10T18:47:11.355 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/33 file after the snapshot 2023-10-10T18:47:11.356 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/34 file after the snapshot 2023-10-10T18:47:11.357 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/35 file after the snapshot 2023-10-10T18:47:11.358 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/36 file after the snapshot 2023-10-10T18:47:11.360 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/37 file after the snapshot 2023-10-10T18:47:11.360 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/38 file after the snapshot 2023-10-10T18:47:11.361 INFO:tasks.workunit.client.0.smithi175.stdout: create 183/39 file after the snapshot 2023-10-10T18:47:11.362 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/.snap/snap-subdir-test 2023-10-10T18:47:11.385 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/30 file after the snapshot 2023-10-10T18:47:11.386 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/31 file after the snapshot 2023-10-10T18:47:11.387 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/32 file after the snapshot 2023-10-10T18:47:11.388 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/33 file after the snapshot 2023-10-10T18:47:11.389 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/34 file after the snapshot 2023-10-10T18:47:11.389 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/35 file after the snapshot 2023-10-10T18:47:11.390 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/36 file after the snapshot 2023-10-10T18:47:11.391 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/37 file after the snapshot 2023-10-10T18:47:11.392 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/38 file after the snapshot 2023-10-10T18:47:11.393 INFO:tasks.workunit.client.0.smithi175.stdout: create 184/39 file after the snapshot 2023-10-10T18:47:11.394 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/.snap/snap-subdir-test 2023-10-10T18:47:11.421 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/30 file after the snapshot 2023-10-10T18:47:11.421 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/31 file after the snapshot 2023-10-10T18:47:11.422 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/32 file after the snapshot 2023-10-10T18:47:11.423 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/33 file after the snapshot 2023-10-10T18:47:11.423 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/34 file after the snapshot 2023-10-10T18:47:11.424 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/35 file after the snapshot 2023-10-10T18:47:11.425 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/36 file after the snapshot 2023-10-10T18:47:11.425 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/37 file after the snapshot 2023-10-10T18:47:11.426 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/38 file after the snapshot 2023-10-10T18:47:11.427 INFO:tasks.workunit.client.0.smithi175.stdout: create 185/39 file after the snapshot 2023-10-10T18:47:11.427 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/.snap/snap-subdir-test 2023-10-10T18:47:11.450 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/30 file after the snapshot 2023-10-10T18:47:11.451 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/31 file after the snapshot 2023-10-10T18:47:11.452 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/32 file after the snapshot 2023-10-10T18:47:11.453 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/33 file after the snapshot 2023-10-10T18:47:11.454 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/34 file after the snapshot 2023-10-10T18:47:11.455 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/35 file after the snapshot 2023-10-10T18:47:11.456 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/36 file after the snapshot 2023-10-10T18:47:11.457 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/37 file after the snapshot 2023-10-10T18:47:11.458 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/38 file after the snapshot 2023-10-10T18:47:11.459 INFO:tasks.workunit.client.0.smithi175.stdout: create 186/39 file after the snapshot 2023-10-10T18:47:11.460 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/.snap/snap-subdir-test 2023-10-10T18:47:11.478 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/30 file after the snapshot 2023-10-10T18:47:11.479 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/31 file after the snapshot 2023-10-10T18:47:11.480 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/32 file after the snapshot 2023-10-10T18:47:11.481 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/33 file after the snapshot 2023-10-10T18:47:11.482 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/34 file after the snapshot 2023-10-10T18:47:11.483 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/35 file after the snapshot 2023-10-10T18:47:11.484 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/36 file after the snapshot 2023-10-10T18:47:11.485 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/37 file after the snapshot 2023-10-10T18:47:11.486 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/38 file after the snapshot 2023-10-10T18:47:11.487 INFO:tasks.workunit.client.0.smithi175.stdout: create 187/39 file after the snapshot 2023-10-10T18:47:11.488 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/.snap/snap-subdir-test 2023-10-10T18:47:11.507 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/30 file after the snapshot 2023-10-10T18:47:11.508 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/31 file after the snapshot 2023-10-10T18:47:11.509 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/32 file after the snapshot 2023-10-10T18:47:11.510 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/33 file after the snapshot 2023-10-10T18:47:11.510 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/34 file after the snapshot 2023-10-10T18:47:11.511 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/35 file after the snapshot 2023-10-10T18:47:11.512 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/36 file after the snapshot 2023-10-10T18:47:11.512 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/37 file after the snapshot 2023-10-10T18:47:11.513 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/38 file after the snapshot 2023-10-10T18:47:11.513 INFO:tasks.workunit.client.0.smithi175.stdout: create 188/39 file after the snapshot 2023-10-10T18:47:11.514 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/.snap/snap-subdir-test 2023-10-10T18:47:11.532 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/30 file after the snapshot 2023-10-10T18:47:11.533 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/31 file after the snapshot 2023-10-10T18:47:11.533 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/32 file after the snapshot 2023-10-10T18:47:11.534 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/33 file after the snapshot 2023-10-10T18:47:11.534 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/34 file after the snapshot 2023-10-10T18:47:11.535 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/35 file after the snapshot 2023-10-10T18:47:11.535 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/36 file after the snapshot 2023-10-10T18:47:11.536 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/37 file after the snapshot 2023-10-10T18:47:11.536 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/38 file after the snapshot 2023-10-10T18:47:11.537 INFO:tasks.workunit.client.0.smithi175.stdout: create 189/39 file after the snapshot 2023-10-10T18:47:11.538 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/.snap/snap-subdir-test 2023-10-10T18:47:11.564 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/30 file after the snapshot 2023-10-10T18:47:11.565 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/31 file after the snapshot 2023-10-10T18:47:11.565 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/32 file after the snapshot 2023-10-10T18:47:11.566 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/33 file after the snapshot 2023-10-10T18:47:11.567 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/34 file after the snapshot 2023-10-10T18:47:11.567 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/35 file after the snapshot 2023-10-10T18:47:11.568 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/36 file after the snapshot 2023-10-10T18:47:11.569 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/37 file after the snapshot 2023-10-10T18:47:11.569 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/38 file after the snapshot 2023-10-10T18:47:11.570 INFO:tasks.workunit.client.0.smithi175.stdout: create 190/39 file after the snapshot 2023-10-10T18:47:11.571 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/.snap/snap-subdir-test 2023-10-10T18:47:12.320 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/30 file after the snapshot 2023-10-10T18:47:12.321 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/31 file after the snapshot 2023-10-10T18:47:12.321 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/32 file after the snapshot 2023-10-10T18:47:12.322 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/33 file after the snapshot 2023-10-10T18:47:12.323 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/34 file after the snapshot 2023-10-10T18:47:12.323 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/35 file after the snapshot 2023-10-10T18:47:12.324 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/36 file after the snapshot 2023-10-10T18:47:12.324 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/37 file after the snapshot 2023-10-10T18:47:12.325 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/38 file after the snapshot 2023-10-10T18:47:12.326 INFO:tasks.workunit.client.0.smithi175.stdout: create 191/39 file after the snapshot 2023-10-10T18:47:12.326 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/.snap/snap-subdir-test 2023-10-10T18:47:12.354 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/30 file after the snapshot 2023-10-10T18:47:12.354 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/31 file after the snapshot 2023-10-10T18:47:12.355 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/32 file after the snapshot 2023-10-10T18:47:12.355 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/33 file after the snapshot 2023-10-10T18:47:12.356 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/34 file after the snapshot 2023-10-10T18:47:12.357 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/35 file after the snapshot 2023-10-10T18:47:12.357 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/36 file after the snapshot 2023-10-10T18:47:12.359 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/37 file after the snapshot 2023-10-10T18:47:12.359 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/38 file after the snapshot 2023-10-10T18:47:12.359 INFO:tasks.workunit.client.0.smithi175.stdout: create 192/39 file after the snapshot 2023-10-10T18:47:12.360 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/.snap/snap-subdir-test 2023-10-10T18:47:12.387 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/30 file after the snapshot 2023-10-10T18:47:12.388 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/31 file after the snapshot 2023-10-10T18:47:12.388 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/32 file after the snapshot 2023-10-10T18:47:12.389 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/33 file after the snapshot 2023-10-10T18:47:12.389 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/34 file after the snapshot 2023-10-10T18:47:12.390 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/35 file after the snapshot 2023-10-10T18:47:12.391 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/36 file after the snapshot 2023-10-10T18:47:12.391 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/37 file after the snapshot 2023-10-10T18:47:12.392 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/38 file after the snapshot 2023-10-10T18:47:12.393 INFO:tasks.workunit.client.0.smithi175.stdout: create 193/39 file after the snapshot 2023-10-10T18:47:12.394 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/.snap/snap-subdir-test 2023-10-10T18:47:12.479 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/30 file after the snapshot 2023-10-10T18:47:12.480 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/31 file after the snapshot 2023-10-10T18:47:12.480 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/32 file after the snapshot 2023-10-10T18:47:12.952 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/33 file after the snapshot 2023-10-10T18:47:12.968 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/34 file after the snapshot 2023-10-10T18:47:12.995 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/35 file after the snapshot 2023-10-10T18:47:13.115 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/36 file after the snapshot 2023-10-10T18:47:13.804 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/37 file after the snapshot 2023-10-10T18:47:14.131 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/38 file after the snapshot 2023-10-10T18:47:14.131 INFO:tasks.workunit.client.0.smithi175.stdout: create 194/39 file after the snapshot 2023-10-10T18:47:14.132 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/.snap/snap-subdir-test 2023-10-10T18:47:18.370 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/30 file after the snapshot 2023-10-10T18:47:18.372 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/31 file after the snapshot 2023-10-10T18:47:18.372 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/32 file after the snapshot 2023-10-10T18:47:18.373 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/33 file after the snapshot 2023-10-10T18:47:18.374 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/34 file after the snapshot 2023-10-10T18:47:18.375 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/35 file after the snapshot 2023-10-10T18:47:18.375 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/36 file after the snapshot 2023-10-10T18:47:18.376 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/37 file after the snapshot 2023-10-10T18:47:18.377 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/38 file after the snapshot 2023-10-10T18:47:18.378 INFO:tasks.workunit.client.0.smithi175.stdout: create 195/39 file after the snapshot 2023-10-10T18:47:18.378 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/.snap/snap-subdir-test 2023-10-10T18:47:18.479 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:47:18.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:47:18.608 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/30 file after the snapshot 2023-10-10T18:47:18.608 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/31 file after the snapshot 2023-10-10T18:47:18.609 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/32 file after the snapshot 2023-10-10T18:47:18.610 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/33 file after the snapshot 2023-10-10T18:47:18.610 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/34 file after the snapshot 2023-10-10T18:47:18.872 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/35 file after the snapshot 2023-10-10T18:47:18.872 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/36 file after the snapshot 2023-10-10T18:47:18.873 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/37 file after the snapshot 2023-10-10T18:47:18.873 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/38 file after the snapshot 2023-10-10T18:47:18.873 INFO:tasks.workunit.client.0.smithi175.stdout: create 196/39 file after the snapshot 2023-10-10T18:47:18.874 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/.snap/snap-subdir-test 2023-10-10T18:47:18.899 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/30 file after the snapshot 2023-10-10T18:47:18.899 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/31 file after the snapshot 2023-10-10T18:47:18.900 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/32 file after the snapshot 2023-10-10T18:47:18.901 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/33 file after the snapshot 2023-10-10T18:47:18.901 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/34 file after the snapshot 2023-10-10T18:47:18.902 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/35 file after the snapshot 2023-10-10T18:47:18.905 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/36 file after the snapshot 2023-10-10T18:47:18.906 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/37 file after the snapshot 2023-10-10T18:47:18.906 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/38 file after the snapshot 2023-10-10T18:47:18.906 INFO:tasks.workunit.client.0.smithi175.stdout: create 197/39 file after the snapshot 2023-10-10T18:47:18.906 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/.snap/snap-subdir-test 2023-10-10T18:47:18.932 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/30 file after the snapshot 2023-10-10T18:47:18.932 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/31 file after the snapshot 2023-10-10T18:47:18.934 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/32 file after the snapshot 2023-10-10T18:47:18.934 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/33 file after the snapshot 2023-10-10T18:47:18.934 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/34 file after the snapshot 2023-10-10T18:47:18.935 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/35 file after the snapshot 2023-10-10T18:47:18.936 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/36 file after the snapshot 2023-10-10T18:47:18.936 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/37 file after the snapshot 2023-10-10T18:47:18.937 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/38 file after the snapshot 2023-10-10T18:47:18.937 INFO:tasks.workunit.client.0.smithi175.stdout: create 198/39 file after the snapshot 2023-10-10T18:47:18.938 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/.snap/snap-subdir-test 2023-10-10T18:47:18.964 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/30 file after the snapshot 2023-10-10T18:47:18.965 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/31 file after the snapshot 2023-10-10T18:47:18.966 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/32 file after the snapshot 2023-10-10T18:47:18.966 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/33 file after the snapshot 2023-10-10T18:47:18.967 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/34 file after the snapshot 2023-10-10T18:47:18.967 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/35 file after the snapshot 2023-10-10T18:47:18.968 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/36 file after the snapshot 2023-10-10T18:47:18.969 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/37 file after the snapshot 2023-10-10T18:47:18.969 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/38 file after the snapshot 2023-10-10T18:47:18.970 INFO:tasks.workunit.client.0.smithi175.stdout: create 199/39 file after the snapshot 2023-10-10T18:47:18.971 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/.snap/snap-subdir-test 2023-10-10T18:47:19.005 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/30 file after the snapshot 2023-10-10T18:47:19.006 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/31 file after the snapshot 2023-10-10T18:47:19.006 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/32 file after the snapshot 2023-10-10T18:47:19.007 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/33 file after the snapshot 2023-10-10T18:47:19.008 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/34 file after the snapshot 2023-10-10T18:47:19.008 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/35 file after the snapshot 2023-10-10T18:47:19.009 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/36 file after the snapshot 2023-10-10T18:47:19.009 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/37 file after the snapshot 2023-10-10T18:47:19.010 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/38 file after the snapshot 2023-10-10T18:47:19.011 INFO:tasks.workunit.client.0.smithi175.stdout: create 200/39 file after the snapshot 2023-10-10T18:47:19.011 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/.snap/snap-subdir-test 2023-10-10T18:47:20.335 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/30 file after the snapshot 2023-10-10T18:47:20.339 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/31 file after the snapshot 2023-10-10T18:47:20.340 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/32 file after the snapshot 2023-10-10T18:47:20.341 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/33 file after the snapshot 2023-10-10T18:47:20.342 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/34 file after the snapshot 2023-10-10T18:47:20.342 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/35 file after the snapshot 2023-10-10T18:47:20.344 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/36 file after the snapshot 2023-10-10T18:47:20.345 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/37 file after the snapshot 2023-10-10T18:47:20.346 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/38 file after the snapshot 2023-10-10T18:47:20.348 INFO:tasks.workunit.client.0.smithi175.stdout: create 201/39 file after the snapshot 2023-10-10T18:47:20.349 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/.snap/snap-subdir-test 2023-10-10T18:47:20.378 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/30 file after the snapshot 2023-10-10T18:47:20.379 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/31 file after the snapshot 2023-10-10T18:47:20.380 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/32 file after the snapshot 2023-10-10T18:47:20.381 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/33 file after the snapshot 2023-10-10T18:47:20.382 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/34 file after the snapshot 2023-10-10T18:47:20.383 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/35 file after the snapshot 2023-10-10T18:47:20.384 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/36 file after the snapshot 2023-10-10T18:47:20.385 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/37 file after the snapshot 2023-10-10T18:47:20.386 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/38 file after the snapshot 2023-10-10T18:47:20.387 INFO:tasks.workunit.client.0.smithi175.stdout: create 202/39 file after the snapshot 2023-10-10T18:47:20.388 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/.snap/snap-subdir-test 2023-10-10T18:47:20.498 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/30 file after the snapshot 2023-10-10T18:47:20.499 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/31 file after the snapshot 2023-10-10T18:47:20.500 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/32 file after the snapshot 2023-10-10T18:47:20.502 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/33 file after the snapshot 2023-10-10T18:47:20.503 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/34 file after the snapshot 2023-10-10T18:47:20.504 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/35 file after the snapshot 2023-10-10T18:47:21.390 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/36 file after the snapshot 2023-10-10T18:47:21.391 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/37 file after the snapshot 2023-10-10T18:47:21.391 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/38 file after the snapshot 2023-10-10T18:47:21.392 INFO:tasks.workunit.client.0.smithi175.stdout: create 203/39 file after the snapshot 2023-10-10T18:47:21.393 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/.snap/snap-subdir-test 2023-10-10T18:47:21.423 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/30 file after the snapshot 2023-10-10T18:47:21.424 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/31 file after the snapshot 2023-10-10T18:47:21.425 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/32 file after the snapshot 2023-10-10T18:47:21.426 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/33 file after the snapshot 2023-10-10T18:47:21.427 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/34 file after the snapshot 2023-10-10T18:47:21.428 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/35 file after the snapshot 2023-10-10T18:47:21.429 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/36 file after the snapshot 2023-10-10T18:47:21.430 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/37 file after the snapshot 2023-10-10T18:47:21.431 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/38 file after the snapshot 2023-10-10T18:47:21.432 INFO:tasks.workunit.client.0.smithi175.stdout: create 204/39 file after the snapshot 2023-10-10T18:47:21.433 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/.snap/snap-subdir-test 2023-10-10T18:47:21.458 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/30 file after the snapshot 2023-10-10T18:47:21.460 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/31 file after the snapshot 2023-10-10T18:47:21.460 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/32 file after the snapshot 2023-10-10T18:47:21.461 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/33 file after the snapshot 2023-10-10T18:47:21.462 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/34 file after the snapshot 2023-10-10T18:47:21.463 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/35 file after the snapshot 2023-10-10T18:47:21.464 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/36 file after the snapshot 2023-10-10T18:47:21.465 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/37 file after the snapshot 2023-10-10T18:47:21.466 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/38 file after the snapshot 2023-10-10T18:47:21.467 INFO:tasks.workunit.client.0.smithi175.stdout: create 205/39 file after the snapshot 2023-10-10T18:47:21.468 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/.snap/snap-subdir-test 2023-10-10T18:47:21.488 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/30 file after the snapshot 2023-10-10T18:47:21.628 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/31 file after the snapshot 2023-10-10T18:47:21.628 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/32 file after the snapshot 2023-10-10T18:47:21.628 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/33 file after the snapshot 2023-10-10T18:47:21.628 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/34 file after the snapshot 2023-10-10T18:47:21.629 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/35 file after the snapshot 2023-10-10T18:47:21.629 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/36 file after the snapshot 2023-10-10T18:47:21.629 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/37 file after the snapshot 2023-10-10T18:47:21.629 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/38 file after the snapshot 2023-10-10T18:47:21.629 INFO:tasks.workunit.client.0.smithi175.stdout: create 206/39 file after the snapshot 2023-10-10T18:47:21.630 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/.snap/snap-subdir-test 2023-10-10T18:47:21.630 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/30 file after the snapshot 2023-10-10T18:47:21.782 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/31 file after the snapshot 2023-10-10T18:47:21.783 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/32 file after the snapshot 2023-10-10T18:47:21.783 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/33 file after the snapshot 2023-10-10T18:47:21.784 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/34 file after the snapshot 2023-10-10T18:47:21.785 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/35 file after the snapshot 2023-10-10T18:47:21.785 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/36 file after the snapshot 2023-10-10T18:47:21.786 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/37 file after the snapshot 2023-10-10T18:47:21.787 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/38 file after the snapshot 2023-10-10T18:47:21.788 INFO:tasks.workunit.client.0.smithi175.stdout: create 207/39 file after the snapshot 2023-10-10T18:47:21.789 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/.snap/snap-subdir-test 2023-10-10T18:47:21.813 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/30 file after the snapshot 2023-10-10T18:47:21.815 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/31 file after the snapshot 2023-10-10T18:47:21.816 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/32 file after the snapshot 2023-10-10T18:47:21.817 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/33 file after the snapshot 2023-10-10T18:47:21.818 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/34 file after the snapshot 2023-10-10T18:47:21.819 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/35 file after the snapshot 2023-10-10T18:47:21.820 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/36 file after the snapshot 2023-10-10T18:47:21.822 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/37 file after the snapshot 2023-10-10T18:47:21.822 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/38 file after the snapshot 2023-10-10T18:47:21.823 INFO:tasks.workunit.client.0.smithi175.stdout: create 208/39 file after the snapshot 2023-10-10T18:47:21.824 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/.snap/snap-subdir-test 2023-10-10T18:47:21.847 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/30 file after the snapshot 2023-10-10T18:47:21.848 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/31 file after the snapshot 2023-10-10T18:47:21.849 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/32 file after the snapshot 2023-10-10T18:47:21.850 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/33 file after the snapshot 2023-10-10T18:47:21.851 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/34 file after the snapshot 2023-10-10T18:47:21.852 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/35 file after the snapshot 2023-10-10T18:47:21.853 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/36 file after the snapshot 2023-10-10T18:47:21.854 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/37 file after the snapshot 2023-10-10T18:47:21.855 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/38 file after the snapshot 2023-10-10T18:47:21.856 INFO:tasks.workunit.client.0.smithi175.stdout: create 209/39 file after the snapshot 2023-10-10T18:47:21.857 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/.snap/snap-subdir-test 2023-10-10T18:47:22.488 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/30 file after the snapshot 2023-10-10T18:47:22.489 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/31 file after the snapshot 2023-10-10T18:47:22.490 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/32 file after the snapshot 2023-10-10T18:47:22.491 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/33 file after the snapshot 2023-10-10T18:47:22.491 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/34 file after the snapshot 2023-10-10T18:47:22.492 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/35 file after the snapshot 2023-10-10T18:47:22.493 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/36 file after the snapshot 2023-10-10T18:47:22.494 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/37 file after the snapshot 2023-10-10T18:47:22.495 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/38 file after the snapshot 2023-10-10T18:47:22.496 INFO:tasks.workunit.client.0.smithi175.stdout: create 210/39 file after the snapshot 2023-10-10T18:47:22.497 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/.snap/snap-subdir-test 2023-10-10T18:47:22.523 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/30 file after the snapshot 2023-10-10T18:47:22.524 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/31 file after the snapshot 2023-10-10T18:47:22.525 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/32 file after the snapshot 2023-10-10T18:47:22.526 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/33 file after the snapshot 2023-10-10T18:47:22.527 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/34 file after the snapshot 2023-10-10T18:47:22.528 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/35 file after the snapshot 2023-10-10T18:47:22.529 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/36 file after the snapshot 2023-10-10T18:47:22.530 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/37 file after the snapshot 2023-10-10T18:47:22.532 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/38 file after the snapshot 2023-10-10T18:47:22.533 INFO:tasks.workunit.client.0.smithi175.stdout: create 211/39 file after the snapshot 2023-10-10T18:47:22.534 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/.snap/snap-subdir-test 2023-10-10T18:47:22.557 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/30 file after the snapshot 2023-10-10T18:47:22.557 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/31 file after the snapshot 2023-10-10T18:47:22.558 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/32 file after the snapshot 2023-10-10T18:47:22.559 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/33 file after the snapshot 2023-10-10T18:47:22.560 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/34 file after the snapshot 2023-10-10T18:47:22.560 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/35 file after the snapshot 2023-10-10T18:47:22.561 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/36 file after the snapshot 2023-10-10T18:47:22.562 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/37 file after the snapshot 2023-10-10T18:47:22.562 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/38 file after the snapshot 2023-10-10T18:47:22.563 INFO:tasks.workunit.client.0.smithi175.stdout: create 212/39 file after the snapshot 2023-10-10T18:47:22.564 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/.snap/snap-subdir-test 2023-10-10T18:47:22.920 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/30 file after the snapshot 2023-10-10T18:47:22.921 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/31 file after the snapshot 2023-10-10T18:47:22.922 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/32 file after the snapshot 2023-10-10T18:47:22.922 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/33 file after the snapshot 2023-10-10T18:47:22.923 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/34 file after the snapshot 2023-10-10T18:47:22.925 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/35 file after the snapshot 2023-10-10T18:47:22.925 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/36 file after the snapshot 2023-10-10T18:47:22.925 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/37 file after the snapshot 2023-10-10T18:47:22.926 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/38 file after the snapshot 2023-10-10T18:47:22.927 INFO:tasks.workunit.client.0.smithi175.stdout: create 213/39 file after the snapshot 2023-10-10T18:47:22.928 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/.snap/snap-subdir-test 2023-10-10T18:47:22.951 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/30 file after the snapshot 2023-10-10T18:47:22.953 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/31 file after the snapshot 2023-10-10T18:47:22.954 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/32 file after the snapshot 2023-10-10T18:47:22.956 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/33 file after the snapshot 2023-10-10T18:47:22.957 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/34 file after the snapshot 2023-10-10T18:47:22.958 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/35 file after the snapshot 2023-10-10T18:47:22.959 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/36 file after the snapshot 2023-10-10T18:47:22.961 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/37 file after the snapshot 2023-10-10T18:47:22.962 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/38 file after the snapshot 2023-10-10T18:47:22.963 INFO:tasks.workunit.client.0.smithi175.stdout: create 214/39 file after the snapshot 2023-10-10T18:47:22.965 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/.snap/snap-subdir-test 2023-10-10T18:47:22.995 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/30 file after the snapshot 2023-10-10T18:47:22.996 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/31 file after the snapshot 2023-10-10T18:47:22.997 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/32 file after the snapshot 2023-10-10T18:47:22.998 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/33 file after the snapshot 2023-10-10T18:47:22.999 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/34 file after the snapshot 2023-10-10T18:47:23.001 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/35 file after the snapshot 2023-10-10T18:47:23.002 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/36 file after the snapshot 2023-10-10T18:47:23.003 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/37 file after the snapshot 2023-10-10T18:47:23.004 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/38 file after the snapshot 2023-10-10T18:47:23.005 INFO:tasks.workunit.client.0.smithi175.stdout: create 215/39 file after the snapshot 2023-10-10T18:47:23.006 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/.snap/snap-subdir-test 2023-10-10T18:47:23.035 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/30 file after the snapshot 2023-10-10T18:47:23.035 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/31 file after the snapshot 2023-10-10T18:47:23.036 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/32 file after the snapshot 2023-10-10T18:47:23.037 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/33 file after the snapshot 2023-10-10T18:47:23.037 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/34 file after the snapshot 2023-10-10T18:47:23.038 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/35 file after the snapshot 2023-10-10T18:47:23.039 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/36 file after the snapshot 2023-10-10T18:47:23.039 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/37 file after the snapshot 2023-10-10T18:47:23.041 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/38 file after the snapshot 2023-10-10T18:47:23.041 INFO:tasks.workunit.client.0.smithi175.stdout: create 216/39 file after the snapshot 2023-10-10T18:47:23.041 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/.snap/snap-subdir-test 2023-10-10T18:47:23.068 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/30 file after the snapshot 2023-10-10T18:47:23.068 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/31 file after the snapshot 2023-10-10T18:47:23.069 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/32 file after the snapshot 2023-10-10T18:47:23.070 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/33 file after the snapshot 2023-10-10T18:47:23.070 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/34 file after the snapshot 2023-10-10T18:47:23.071 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/35 file after the snapshot 2023-10-10T18:47:23.072 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/36 file after the snapshot 2023-10-10T18:47:23.072 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/37 file after the snapshot 2023-10-10T18:47:23.073 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/38 file after the snapshot 2023-10-10T18:47:23.074 INFO:tasks.workunit.client.0.smithi175.stdout: create 217/39 file after the snapshot 2023-10-10T18:47:23.074 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/.snap/snap-subdir-test 2023-10-10T18:47:23.103 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/30 file after the snapshot 2023-10-10T18:47:23.104 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/31 file after the snapshot 2023-10-10T18:47:23.105 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/32 file after the snapshot 2023-10-10T18:47:23.105 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/33 file after the snapshot 2023-10-10T18:47:23.106 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/34 file after the snapshot 2023-10-10T18:47:23.107 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/35 file after the snapshot 2023-10-10T18:47:23.107 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/36 file after the snapshot 2023-10-10T18:47:23.108 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/37 file after the snapshot 2023-10-10T18:47:23.109 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/38 file after the snapshot 2023-10-10T18:47:23.109 INFO:tasks.workunit.client.0.smithi175.stdout: create 218/39 file after the snapshot 2023-10-10T18:47:23.110 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/.snap/snap-subdir-test 2023-10-10T18:47:23.913 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/30 file after the snapshot 2023-10-10T18:47:23.914 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/31 file after the snapshot 2023-10-10T18:47:23.916 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/32 file after the snapshot 2023-10-10T18:47:23.917 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/33 file after the snapshot 2023-10-10T18:47:23.918 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/34 file after the snapshot 2023-10-10T18:47:23.918 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/35 file after the snapshot 2023-10-10T18:47:23.919 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/36 file after the snapshot 2023-10-10T18:47:23.920 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/37 file after the snapshot 2023-10-10T18:47:23.921 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/38 file after the snapshot 2023-10-10T18:47:23.922 INFO:tasks.workunit.client.0.smithi175.stdout: create 219/39 file after the snapshot 2023-10-10T18:47:23.923 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/.snap/snap-subdir-test 2023-10-10T18:47:23.949 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/30 file after the snapshot 2023-10-10T18:47:23.950 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/31 file after the snapshot 2023-10-10T18:47:23.951 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/32 file after the snapshot 2023-10-10T18:47:23.952 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/33 file after the snapshot 2023-10-10T18:47:23.952 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/34 file after the snapshot 2023-10-10T18:47:23.953 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/35 file after the snapshot 2023-10-10T18:47:23.954 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/36 file after the snapshot 2023-10-10T18:47:23.954 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/37 file after the snapshot 2023-10-10T18:47:23.955 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/38 file after the snapshot 2023-10-10T18:47:23.956 INFO:tasks.workunit.client.0.smithi175.stdout: create 220/39 file after the snapshot 2023-10-10T18:47:23.957 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/.snap/snap-subdir-test 2023-10-10T18:47:23.985 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/30 file after the snapshot 2023-10-10T18:47:24.246 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/31 file after the snapshot 2023-10-10T18:47:24.247 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/32 file after the snapshot 2023-10-10T18:47:24.248 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/33 file after the snapshot 2023-10-10T18:47:24.249 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/34 file after the snapshot 2023-10-10T18:47:24.250 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/35 file after the snapshot 2023-10-10T18:47:24.252 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/36 file after the snapshot 2023-10-10T18:47:24.253 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/37 file after the snapshot 2023-10-10T18:47:24.254 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/38 file after the snapshot 2023-10-10T18:47:24.255 INFO:tasks.workunit.client.0.smithi175.stdout: create 221/39 file after the snapshot 2023-10-10T18:47:24.257 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/.snap/snap-subdir-test 2023-10-10T18:47:24.285 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/30 file after the snapshot 2023-10-10T18:47:24.286 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/31 file after the snapshot 2023-10-10T18:47:24.286 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/32 file after the snapshot 2023-10-10T18:47:24.287 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/33 file after the snapshot 2023-10-10T18:47:24.288 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/34 file after the snapshot 2023-10-10T18:47:24.288 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/35 file after the snapshot 2023-10-10T18:47:24.289 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/36 file after the snapshot 2023-10-10T18:47:24.289 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/37 file after the snapshot 2023-10-10T18:47:24.290 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/38 file after the snapshot 2023-10-10T18:47:24.291 INFO:tasks.workunit.client.0.smithi175.stdout: create 222/39 file after the snapshot 2023-10-10T18:47:24.291 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/.snap/snap-subdir-test 2023-10-10T18:47:24.318 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/30 file after the snapshot 2023-10-10T18:47:24.318 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/31 file after the snapshot 2023-10-10T18:47:24.319 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/32 file after the snapshot 2023-10-10T18:47:24.320 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/33 file after the snapshot 2023-10-10T18:47:24.320 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/34 file after the snapshot 2023-10-10T18:47:24.321 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/35 file after the snapshot 2023-10-10T18:47:24.322 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/36 file after the snapshot 2023-10-10T18:47:24.322 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/37 file after the snapshot 2023-10-10T18:47:24.323 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/38 file after the snapshot 2023-10-10T18:47:24.324 INFO:tasks.workunit.client.0.smithi175.stdout: create 223/39 file after the snapshot 2023-10-10T18:47:24.324 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/.snap/snap-subdir-test 2023-10-10T18:47:24.351 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/30 file after the snapshot 2023-10-10T18:47:24.351 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/31 file after the snapshot 2023-10-10T18:47:24.614 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/32 file after the snapshot 2023-10-10T18:47:24.615 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/33 file after the snapshot 2023-10-10T18:47:24.615 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/34 file after the snapshot 2023-10-10T18:47:24.616 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/35 file after the snapshot 2023-10-10T18:47:24.616 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/36 file after the snapshot 2023-10-10T18:47:24.617 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/37 file after the snapshot 2023-10-10T18:47:24.618 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/38 file after the snapshot 2023-10-10T18:47:24.618 INFO:tasks.workunit.client.0.smithi175.stdout: create 224/39 file after the snapshot 2023-10-10T18:47:24.619 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/.snap/snap-subdir-test 2023-10-10T18:47:24.641 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/30 file after the snapshot 2023-10-10T18:47:24.642 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/31 file after the snapshot 2023-10-10T18:47:24.643 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/32 file after the snapshot 2023-10-10T18:47:24.644 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/33 file after the snapshot 2023-10-10T18:47:24.645 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/34 file after the snapshot 2023-10-10T18:47:24.646 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/35 file after the snapshot 2023-10-10T18:47:24.650 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/36 file after the snapshot 2023-10-10T18:47:24.650 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/37 file after the snapshot 2023-10-10T18:47:24.650 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/38 file after the snapshot 2023-10-10T18:47:24.650 INFO:tasks.workunit.client.0.smithi175.stdout: create 225/39 file after the snapshot 2023-10-10T18:47:24.651 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/.snap/snap-subdir-test 2023-10-10T18:47:25.101 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:25.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.217+0000 7f2b04454700 1 Processor -- start 2023-10-10T18:47:25.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.217+0000 7f2b04454700 1 -- start start 2023-10-10T18:47:25.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b04454700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b1fc0 0x7f2afc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b04454700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2afc0b2390 0x7f2afc068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b04454700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc1574b0 con 0x7f2afc068bd0 2023-10-10T18:47:25.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b04454700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2afc150770 con 0x7f2afc0b2390 2023-10-10T18:47:25.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b04454700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc0afad0 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b019ef700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2afc0b2390 0x7f2afc068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b019ef700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2afc0b2390 0x7f2afc068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58348/0 (socket says 172.21.15.175:58348) 2023-10-10T18:47:25.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b029f1700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2afc068bd0 0x7f2afc0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35778/0 (socket says 172.21.15.175:35778) 2023-10-10T18:47:25.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b019ef700 1 -- 172.21.15.175:0/1565463387 learned_addr learned my addr 172.21.15.175:0/1565463387 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:25.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b021f0700 1 --2- 172.21.15.175:0/1565463387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b1fc0 0x7f2afc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b019ef700 1 -- 172.21.15.175:0/1565463387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b1fc0 msgr2=0x7f2afc155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:25.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b019ef700 1 --2- 172.21.15.175:0/1565463387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b1fc0 0x7f2afc155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b019ef700 1 -- 172.21.15.175:0/1565463387 >> v1:172.21.15.175:6789/0 conn(0x7f2afc068bd0 legacy=0x7f2afc0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b011ee700 1 -- 172.21.15.175:0/1565463387 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715772739 0 0) 0x7f2afc1574b0 con 0x7f2afc068bd0 2023-10-10T18:47:25.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.218+0000 7f2b019ef700 1 -- 172.21.15.175:0/1565463387 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc0af4c0 con 0x7f2afc0b2390 2023-10-10T18:47:25.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b019ef700 1 --2- 172.21.15.175:0/1565463387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b2390 0x7f2afc068690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0099b0 tx=0x7f2aec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a1aa6442ef2818 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:25.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b011ee700 1 -- 172.21.15.175:0/1565463387 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec00b8f0 con 0x7f2afc0b2390 2023-10-10T18:47:25.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b011ee700 1 -- 172.21.15.175:0/1565463387 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec00ba50 con 0x7f2afc0b2390 2023-10-10T18:47:25.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b011ee700 1 -- 172.21.15.175:0/1565463387 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec00bdf0 con 0x7f2afc0b2390 2023-10-10T18:47:25.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b04454700 1 -- 172.21.15.175:0/1565463387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b2390 msgr2=0x7f2afc068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b04454700 1 --2- 172.21.15.175:0/1565463387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b2390 0x7f2afc068690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0099b0 tx=0x7f2aec009250 comp rx=0 tx=0).stop 2023-10-10T18:47:25.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b04454700 1 -- 172.21.15.175:0/1565463387 shutdown_connections 2023-10-10T18:47:25.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b04454700 1 --2- 172.21.15.175:0/1565463387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b1fc0 0x7f2afc155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b04454700 1 --2- 172.21.15.175:0/1565463387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b2390 0x7f2afc068690 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b04454700 1 -- 172.21.15.175:0/1565463387 >> 172.21.15.175:0/1565463387 conn(0x7f2afc063230 msgr2=0x7f2afc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:25.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b04454700 1 -- 172.21.15.175:0/1565463387 shutdown_connections 2023-10-10T18:47:25.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.219+0000 7f2b04454700 1 -- 172.21.15.175:0/1565463387 wait complete. 2023-10-10T18:47:25.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.220+0000 7f2b04454700 1 Processor -- start 2023-10-10T18:47:25.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.220+0000 7f2b04454700 1 -- start start 2023-10-10T18:47:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b04454700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2afc068bd0 0x7f2afc0cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b04454700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b1fc0 0x7f2afc165d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b04454700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b2390 0x7f2afc166270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b04454700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc0c4570 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b04454700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc0cc570 con 0x7f2afc068bd0 2023-10-10T18:47:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b04454700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc0cd660 con 0x7f2afc0b2390 2023-10-10T18:47:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b019ef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b1fc0 0x7f2afc165d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b019ef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b1fc0 0x7f2afc165d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58362/0 (socket says 172.21.15.175:58362) 2023-10-10T18:47:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b019ef700 1 -- 172.21.15.175:0/350163683 learned_addr learned my addr 172.21.15.175:0/350163683 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b021f0700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2afc068bd0 0x7f2afc0cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b029f1700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b2390 0x7f2afc166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b019ef700 1 -- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b2390 msgr2=0x7f2afc166270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b019ef700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b2390 0x7f2afc166270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b019ef700 1 -- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2afc068bd0 msgr2=0x7f2afc0cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b019ef700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2afc068bd0 0x7f2afc0cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.221+0000 7f2b019ef700 1 -- 172.21.15.175:0/350163683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec0094f0 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.222+0000 7f2b019ef700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b1fc0 0x7f2afc165d30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2afc06fd90 tx=0x7f2aec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:25.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.222+0000 7f2af27fc700 1 -- 172.21.15.175:0/350163683 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec011ea0 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.222+0000 7f2af27fc700 1 -- 172.21.15.175:0/350163683 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec01a660 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.222+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2afc0cbc80 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.222+0000 7f2af27fc700 1 -- 172.21.15.175:0/350163683 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec01aa20 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.222+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2afc0cbf20 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.223+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2afc0cc900 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.223+0000 7f2af27fc700 1 -- 172.21.15.175:0/350163683 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2aec01b430 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.224+0000 7f2af27fc700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ae0066f30 0x7f2ae00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.224+0000 7f2af27fc700 1 -- 172.21.15.175:0/350163683 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f2aec0862c0 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.224+0000 7f2b021f0700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ae0066f30 0x7f2ae00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.224+0000 7f2b021f0700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ae0066f30 0x7f2ae00693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2af800ab30 tx=0x7f2af8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.228+0000 7f2af27fc700 1 -- 172.21.15.175:0/350163683 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2aec053550 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.449+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2afc0cc6b0 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.450+0000 7f2af27fc700 1 -- 172.21.15.175:0/350163683 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7f2aec056b70 con 0x7f2afc0b1fc0 2023-10-10T18:47:25.451 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:25.451 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:25.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ae0066f30 msgr2=0x7f2ae00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ae0066f30 0x7f2ae00693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2af800ab30 tx=0x7f2af8008040 comp rx=0 tx=0).stop 2023-10-10T18:47:25.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b1fc0 msgr2=0x7f2afc165d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b1fc0 0x7f2afc165d30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2afc06fd90 tx=0x7f2aec017000 comp rx=0 tx=0).stop 2023-10-10T18:47:25.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 shutdown_connections 2023-10-10T18:47:25.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ae0066f30 0x7f2ae00693e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2afc0b2390 0x7f2afc166270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2afc068bd0 0x7f2afc0cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 --2- 172.21.15.175:0/350163683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2afc0b1fc0 0x7f2afc165d30 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 >> 172.21.15.175:0/350163683 conn(0x7f2afc063230 msgr2=0x7f2afc152540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:25.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 shutdown_connections 2023-10-10T18:47:25.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.452+0000 7f2b04454700 1 -- 172.21.15.175:0/350163683 wait complete. 2023-10-10T18:47:25.456 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:25.465 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2023-10-10T18:47:25.465 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-10T18:47:25.465 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T18:47:25.477 INFO:tasks.ceph.mds.b:Stopped 2023-10-10T18:47:25.478 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:25.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.594+0000 7fd78d837700 1 Processor -- start 2023-10-10T18:47:25.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.594+0000 7fd78d837700 1 -- start start 2023-10-10T18:47:25.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78d837700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd78808a100 0x7fd788159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78d837700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7880cd040 0x7fd7880684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78d837700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd788067fa0 con 0x7fd78814fd90 2023-10-10T18:47:25.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78d837700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7880c5600 con 0x7fd78808a100 2023-10-10T18:47:25.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78d837700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd788156b70 con 0x7fd7880cd040 2023-10-10T18:47:25.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd787fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd78808a100 0x7fd788159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78c835700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd78814fd90 0x7fd7880c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35790/0 (socket says 172.21.15.175:35790) 2023-10-10T18:47:25.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78c835700 1 -- 172.21.15.175:0/2174802016 learned_addr learned my addr 172.21.15.175:0/2174802016 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:25.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78d036700 1 --2- 172.21.15.175:0/2174802016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7880cd040 0x7fd7880684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd787fff700 1 -- 172.21.15.175:0/2174802016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7880cd040 msgr2=0x7fd7880684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:25.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd787fff700 1 --2- 172.21.15.175:0/2174802016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7880cd040 0x7fd7880684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd787fff700 1 -- 172.21.15.175:0/2174802016 >> v1:172.21.15.175:6789/0 conn(0x7fd78814fd90 legacy=0x7fd7880c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd7877fe700 1 -- 172.21.15.175:0/2174802016 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586839908 0 0) 0x7fd788067fa0 con 0x7fd78814fd90 2023-10-10T18:47:25.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd787fff700 1 -- 172.21.15.175:0/2174802016 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788150160 con 0x7fd78808a100 2023-10-10T18:47:25.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.595+0000 7fd78d036700 1 --2- 172.21.15.175:0/2174802016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7880cd040 0x7fd7880684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:47:25.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd787fff700 1 --2- 172.21.15.175:0/2174802016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78808a100 0x7fd788159c30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd7780098e0 tx=0x7fd7780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a788236e177e40d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:25.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd7877fe700 1 -- 172.21.15.175:0/2174802016 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd778009e80 con 0x7fd78808a100 2023-10-10T18:47:25.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd7877fe700 1 -- 172.21.15.175:0/2174802016 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77800b860 con 0x7fd78808a100 2023-10-10T18:47:25.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd7877fe700 1 -- 172.21.15.175:0/2174802016 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77800bbe0 con 0x7fd78808a100 2023-10-10T18:47:25.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd78d837700 1 -- 172.21.15.175:0/2174802016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78808a100 msgr2=0x7fd788159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd78d837700 1 --2- 172.21.15.175:0/2174802016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78808a100 0x7fd788159c30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd7780098e0 tx=0x7fd7780091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd78d837700 1 -- 172.21.15.175:0/2174802016 shutdown_connections 2023-10-10T18:47:25.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd78d837700 1 --2- 172.21.15.175:0/2174802016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7880cd040 0x7fd7880684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd78d837700 1 --2- 172.21.15.175:0/2174802016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78808a100 0x7fd788159c30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd78d837700 1 -- 172.21.15.175:0/2174802016 >> 172.21.15.175:0/2174802016 conn(0x7fd788063d00 msgr2=0x7fd7880cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:25.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd78d837700 1 -- 172.21.15.175:0/2174802016 shutdown_connections 2023-10-10T18:47:25.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.596+0000 7fd78d837700 1 -- 172.21.15.175:0/2174802016 wait complete. 2023-10-10T18:47:25.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.597+0000 7fd78d837700 1 Processor -- start 2023-10-10T18:47:25.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.597+0000 7fd78d837700 1 -- start start 2023-10-10T18:47:25.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d837700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd78808a100 0x7fd7881661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d837700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7880cd040 0x7fd7881577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d837700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78814fd90 0x7fd788158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d837700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880b1c00 con 0x7fd78814fd90 2023-10-10T18:47:25.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d837700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7881666f0 con 0x7fd7880cd040 2023-10-10T18:47:25.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d837700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880c7a60 con 0x7fd78808a100 2023-10-10T18:47:25.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d036700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78814fd90 0x7fd788158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d036700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78814fd90 0x7fd788158190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58380/0 (socket says 172.21.15.175:58380) 2023-10-10T18:47:25.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78d036700 1 -- 172.21.15.175:0/1596164127 learned_addr learned my addr 172.21.15.175:0/1596164127 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:25.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd78c835700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd78808a100 0x7fd7881661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.598+0000 7fd787fff700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7880cd040 0x7fd7881577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd78d036700 1 -- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd78808a100 msgr2=0x7fd7881661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd78d036700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd78808a100 0x7fd7881661b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd78d036700 1 -- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7880cd040 msgr2=0x7fd7881577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd78d036700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7880cd040 0x7fd7881577f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd78d036700 1 -- 172.21.15.175:0/1596164127 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd778009420 con 0x7fd78814fd90 2023-10-10T18:47:25.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd78d036700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78814fd90 0x7fd788158190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd78000a380 tx=0x7fd780007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:25.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd784ff9700 1 -- 172.21.15.175:0/1596164127 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780015440 con 0x7fd78814fd90 2023-10-10T18:47:25.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd784ff9700 1 -- 172.21.15.175:0/1596164127 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7800155a0 con 0x7fd78814fd90 2023-10-10T18:47:25.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd784ff9700 1 -- 172.21.15.175:0/1596164127 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780015960 con 0x7fd78814fd90 2023-10-10T18:47:25.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd788156da0 con 0x7fd78814fd90 2023-10-10T18:47:25.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.599+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd788157360 con 0x7fd78814fd90 2023-10-10T18:47:25.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.600+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7881574f0 con 0x7fd78814fd90 2023-10-10T18:47:25.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.600+0000 7fd784ff9700 1 -- 172.21.15.175:0/1596164127 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd78000d440 con 0x7fd78814fd90 2023-10-10T18:47:25.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.601+0000 7fd784ff9700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd768066f30 0x7fd7680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:25.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.601+0000 7fd784ff9700 1 -- 172.21.15.175:0/1596164127 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fd7800850c0 con 0x7fd78814fd90 2023-10-10T18:47:25.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.601+0000 7fd78c835700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd768066f30 0x7fd7680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:25.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.601+0000 7fd78c835700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd768066f30 0x7fd7680693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd770007160 tx=0x7fd770007f50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:25.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.605+0000 7fd784ff9700 1 -- 172.21.15.175:0/1596164127 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7800522a0 con 0x7fd78814fd90 2023-10-10T18:47:25.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.830+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd788067570 con 0x7fd78814fd90 2023-10-10T18:47:25.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.830+0000 7fd784ff9700 1 -- 172.21.15.175:0/1596164127 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7fd7800558c0 con 0x7fd78814fd90 2023-10-10T18:47:25.832 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:25.832 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:25.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd768066f30 msgr2=0x7fd7680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd768066f30 0x7fd7680693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd770007160 tx=0x7fd770007f50 comp rx=0 tx=0).stop 2023-10-10T18:47:25.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78814fd90 msgr2=0x7fd788158190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:25.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78814fd90 0x7fd788158190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd78000a380 tx=0x7fd780007900 comp rx=0 tx=0).stop 2023-10-10T18:47:25.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 shutdown_connections 2023-10-10T18:47:25.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd768066f30 0x7fd7680693e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd78814fd90 0x7fd788158190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd78808a100 0x7fd7881661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 --2- 172.21.15.175:0/1596164127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7880cd040 0x7fd7881577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:25.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 >> 172.21.15.175:0/1596164127 conn(0x7fd788063d00 msgr2=0x7fd78814ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:25.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 shutdown_connections 2023-10-10T18:47:25.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:25.832+0000 7fd78d837700 1 -- 172.21.15.175:0/1596164127 wait complete. 2023-10-10T18:47:25.836 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:25.845 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:47:27.846 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:27.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.969+0000 7fa61d145700 1 Processor -- start 2023-10-10T18:47:27.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.969+0000 7fa61d145700 1 -- start start 2023-10-10T18:47:27.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.970+0000 7fa61d145700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa6181573d0 0x7fa6180c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:27.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.970+0000 7fa61d145700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa6180c4f20 0x7fa61814d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:27.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.970+0000 7fa61d145700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa618154aa0 con 0x7fa6181577a0 2023-10-10T18:47:27.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.970+0000 7fa61d145700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6180bfdf0 con 0x7fa6181573d0 2023-10-10T18:47:27.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.970+0000 7fa61d145700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa618063dc0 con 0x7fa6180c4f20 2023-10-10T18:47:27.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.970+0000 7fa616d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa6181573d0 0x7fa6180c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:27.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.970+0000 7fa616d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa6181573d0 0x7fa6180c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58396/0 (socket says 172.21.15.175:58396) 2023-10-10T18:47:27.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.970+0000 7fa616d9d700 1 -- 172.21.15.175:0/3075481869 learned_addr learned my addr 172.21.15.175:0/3075481869 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:27.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa61759e700 1 --2- 172.21.15.175:0/3075481869 >> v2:172.21.15.191:3300/0 conn(0x7fa6180c4f20 0x7fa61814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:27.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa616d9d700 1 -- 172.21.15.175:0/3075481869 >> v2:172.21.15.191:3300/0 conn(0x7fa6180c4f20 msgr2=0x7fa61814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:27.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa616d9d700 1 --2- 172.21.15.175:0/3075481869 >> v2:172.21.15.191:3300/0 conn(0x7fa6180c4f20 0x7fa61814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:27.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa616d9d700 1 -- 172.21.15.175:0/3075481869 >> v1:172.21.15.191:6789/0 conn(0x7fa6181577a0 legacy=0x7fa6180bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:27.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa616d9d700 1 -- 172.21.15.175:0/3075481869 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618157c20 con 0x7fa6181573d0 2023-10-10T18:47:27.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa616d9d700 1 --2- 172.21.15.175:0/3075481869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa6180c12a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa6180c6ee0 tx=0x7fa608009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beace0a494f02f79 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:27.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa615d9b700 1 -- 172.21.15.175:0/3075481869 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60800e070 con 0x7fa6181573d0 2023-10-10T18:47:27.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa615d9b700 1 -- 172.21.15.175:0/3075481869 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60800ba10 con 0x7fa6181573d0 2023-10-10T18:47:27.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa615d9b700 1 -- 172.21.15.175:0/3075481869 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6080092e0 con 0x7fa6181573d0 2023-10-10T18:47:27.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa61d145700 1 -- 172.21.15.175:0/3075481869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 msgr2=0x7fa6180c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:27.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa61d145700 1 --2- 172.21.15.175:0/3075481869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa6180c12a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa6180c6ee0 tx=0x7fa608009840 comp rx=0 tx=0).stop 2023-10-10T18:47:27.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa61d145700 1 -- 172.21.15.175:0/3075481869 shutdown_connections 2023-10-10T18:47:27.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa61d145700 1 --2- 172.21.15.175:0/3075481869 >> v2:172.21.15.191:3300/0 conn(0x7fa6180c4f20 0x7fa61814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:27.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa61d145700 1 --2- 172.21.15.175:0/3075481869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa6180c12a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:27.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa61d145700 1 -- 172.21.15.175:0/3075481869 >> 172.21.15.175:0/3075481869 conn(0x7fa61806dc70 msgr2=0x7fa618063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:27.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.971+0000 7fa61d145700 1 -- 172.21.15.175:0/3075481869 shutdown_connections 2023-10-10T18:47:27.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.972+0000 7fa61d145700 1 -- 172.21.15.175:0/3075481869 wait complete. 2023-10-10T18:47:27.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61d145700 1 Processor -- start 2023-10-10T18:47:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61d145700 1 -- start start 2023-10-10T18:47:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61d145700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6180c4f20 0x7fa618152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61d145700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa618164160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61d145700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6181577a0 0x7fa6180ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61d145700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c8e00 con 0x7fa6181573d0 2023-10-10T18:47:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61d145700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c9290 con 0x7fa6180c4f20 2023-10-10T18:47:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61d145700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa618163cd0 con 0x7fa6181577a0 2023-10-10T18:47:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.973+0000 7fa61659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa618164160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa618164160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58406/0 (socket says 172.21.15.175:58406) 2023-10-10T18:47:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61659c700 1 -- 172.21.15.175:0/470448015 learned_addr learned my addr 172.21.15.175:0/470448015 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:27.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa616d9d700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6180c4f20 0x7fa618152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:27.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61759e700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6181577a0 0x7fa6180ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:27.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61659c700 1 -- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6181577a0 msgr2=0x7fa6180ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:27.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61659c700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6181577a0 0x7fa6180ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:27.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61659c700 1 -- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6180c4f20 msgr2=0x7fa618152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:27.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61659c700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6180c4f20 0x7fa618152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:27.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61659c700 1 -- 172.21.15.175:0/470448015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa608009ae0 con 0x7fa6181573d0 2023-10-10T18:47:27.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61659c700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa618164160 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00c600 tx=0x7fa60c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:27.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa5feffd700 1 -- 172.21.15.175:0/470448015 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c011590 con 0x7fa6181573d0 2023-10-10T18:47:27.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa5feffd700 1 -- 172.21.15.175:0/470448015 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c0116f0 con 0x7fa6181573d0 2023-10-10T18:47:27.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa5feffd700 1 -- 172.21.15.175:0/470448015 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c011ab0 con 0x7fa6181573d0 2023-10-10T18:47:27.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61814b480 con 0x7fa6181573d0 2023-10-10T18:47:27.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.974+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa618151b40 con 0x7fa6181573d0 2023-10-10T18:47:27.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.975+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa618067570 con 0x7fa6181573d0 2023-10-10T18:47:27.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.975+0000 7fa5feffd700 1 -- 172.21.15.175:0/470448015 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa60c01c460 con 0x7fa6181573d0 2023-10-10T18:47:27.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.976+0000 7fa5feffd700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa604066f30 0x7fa6040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:27.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.976+0000 7fa5feffd700 1 -- 172.21.15.175:0/470448015 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fa60c084140 con 0x7fa6181573d0 2023-10-10T18:47:27.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.976+0000 7fa616d9d700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa604066f30 0x7fa6040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:27.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.976+0000 7fa616d9d700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa604066f30 0x7fa6040693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa608009880 tx=0x7fa608017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:27.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:27.980+0000 7fa5feffd700 1 -- 172.21.15.175:0/470448015 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa60c051320 con 0x7fa6181573d0 2023-10-10T18:47:28.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.205+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa61814d130 con 0x7fa6181573d0 2023-10-10T18:47:28.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.205+0000 7fa5feffd700 1 -- 172.21.15.175:0/470448015 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7fa60c054940 con 0x7fa6181573d0 2023-10-10T18:47:28.207 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:28.207 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:28.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa604066f30 msgr2=0x7fa6040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:28.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa604066f30 0x7fa6040693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa608009880 tx=0x7fa608017000 comp rx=0 tx=0).stop 2023-10-10T18:47:28.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 msgr2=0x7fa618164160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:28.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa618164160 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00c600 tx=0x7fa60c007f60 comp rx=0 tx=0).stop 2023-10-10T18:47:28.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 shutdown_connections 2023-10-10T18:47:28.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa604066f30 0x7fa6040693e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:28.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6181577a0 0x7fa6180ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:28.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6180c4f20 0x7fa618152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:28.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 --2- 172.21.15.175:0/470448015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa6181573d0 0x7fa618164160 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:28.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 >> 172.21.15.175:0/470448015 conn(0x7fa61806dc70 msgr2=0x7fa6180c6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:28.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 shutdown_connections 2023-10-10T18:47:28.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:28.207+0000 7fa61d145700 1 -- 172.21.15.175:0/470448015 wait complete. 2023-10-10T18:47:28.211 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:28.220 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:47:30.221 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:30.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.348+0000 7fb7c7935700 1 Processor -- start 2023-10-10T18:47:30.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.348+0000 7fb7c7935700 1 -- start start 2023-10-10T18:47:30.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c7935700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c015a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:30.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c7935700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb7c00c4580 0x7fb7c0063c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:30.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c7935700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c0157030 con 0x7fb7c00c7190 2023-10-10T18:47:30.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c7935700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c00c5000 con 0x7fb7c00c4580 2023-10-10T18:47:30.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c7935700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c0156420 con 0x7fb7c00c6330 2023-10-10T18:47:30.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c5ed2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb7c00c4580 0x7fb7c0063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:30.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c4ed0700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb7c00c7190 0x7fb7c00c3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35800/0 (socket says 172.21.15.175:35800) 2023-10-10T18:47:30.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c4ed0700 1 -- 172.21.15.175:0/2358300708 learned_addr learned my addr 172.21.15.175:0/2358300708 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:30.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c56d1700 1 --2- 172.21.15.175:0/2358300708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:30.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c5ed2700 1 -- 172.21.15.175:0/2358300708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 msgr2=0x7fb7c015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:30.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c5ed2700 1 --2- 172.21.15.175:0/2358300708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c015a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c5ed2700 1 -- 172.21.15.175:0/2358300708 >> v1:172.21.15.175:6789/0 conn(0x7fb7c00c7190 legacy=0x7fb7c00c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:30.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7b7fff700 1 -- 172.21.15.175:0/2358300708 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213246744 0 0) 0x7fb7c0157030 con 0x7fb7c00c7190 2023-10-10T18:47:30.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c5ed2700 1 -- 172.21.15.175:0/2358300708 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c0156700 con 0x7fb7c00c4580 2023-10-10T18:47:30.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.349+0000 7fb7c56d1700 1 --2- 172.21.15.175:0/2358300708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:47:30.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c5ed2700 1 --2- 172.21.15.175:0/2358300708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c0063c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009e60 tx=0x7fb7b8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59791737d3e3c090 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:30.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7b7fff700 1 -- 172.21.15.175:0/2358300708 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b800e070 con 0x7fb7c00c4580 2023-10-10T18:47:30.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7b7fff700 1 -- 172.21.15.175:0/2358300708 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b800bcc0 con 0x7fb7c00c4580 2023-10-10T18:47:30.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7b7fff700 1 -- 172.21.15.175:0/2358300708 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8012680 con 0x7fb7c00c4580 2023-10-10T18:47:30.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c7935700 1 -- 172.21.15.175:0/2358300708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 msgr2=0x7fb7c0063c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c7935700 1 --2- 172.21.15.175:0/2358300708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c0063c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009e60 tx=0x7fb7b8009780 comp rx=0 tx=0).stop 2023-10-10T18:47:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c7935700 1 -- 172.21.15.175:0/2358300708 shutdown_connections 2023-10-10T18:47:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c7935700 1 --2- 172.21.15.175:0/2358300708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c0063c20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c7935700 1 --2- 172.21.15.175:0/2358300708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c7935700 1 -- 172.21.15.175:0/2358300708 >> 172.21.15.175:0/2358300708 conn(0x7fb7c006ef20 msgr2=0x7fb7c0063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:30.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c7935700 1 -- 172.21.15.175:0/2358300708 shutdown_connections 2023-10-10T18:47:30.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.350+0000 7fb7c7935700 1 -- 172.21.15.175:0/2358300708 wait complete. 2023-10-10T18:47:30.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.351+0000 7fb7c7935700 1 Processor -- start 2023-10-10T18:47:30.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.351+0000 7fb7c7935700 1 -- start start 2023-10-10T18:47:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c7935700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c014be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c7935700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c014b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c7935700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c00c7190 0x7fb7c00cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c7935700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c0159180 con 0x7fb7c00c4580 2023-10-10T18:47:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c7935700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00c8ea0 con 0x7fb7c00c7190 2023-10-10T18:47:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c7935700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00cd560 con 0x7fb7c00c6330 2023-10-10T18:47:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c56d1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c014be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c56d1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c014be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58436/0 (socket says 172.21.15.175:58436) 2023-10-10T18:47:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.352+0000 7fb7c56d1700 1 -- 172.21.15.175:0/73656571 learned_addr learned my addr 172.21.15.175:0/73656571 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c4ed0700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c014b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c5ed2700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c00c7190 0x7fb7c00cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c56d1700 1 -- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 msgr2=0x7fb7c014b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c56d1700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c014b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c56d1700 1 -- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c00c7190 msgr2=0x7fb7c00cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c56d1700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c00c7190 0x7fb7c00cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c56d1700 1 -- 172.21.15.175:0/73656571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8009a20 con 0x7fb7c00c4580 2023-10-10T18:47:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c56d1700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c014be30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c830 tx=0x7fb7bc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7b57fa700 1 -- 172.21.15.175:0/73656571 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc011600 con 0x7fb7c00c4580 2023-10-10T18:47:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7b57fa700 1 -- 172.21.15.175:0/73656571 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc011760 con 0x7fb7c00c4580 2023-10-10T18:47:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7b57fa700 1 -- 172.21.15.175:0/73656571 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc011b20 con 0x7fb7c00c4580 2023-10-10T18:47:30.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c0157350 con 0x7fb7c00c4580 2023-10-10T18:47:30.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.353+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c00c8b30 con 0x7fb7c00c4580 2023-10-10T18:47:30.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.354+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c0067570 con 0x7fb7c00c4580 2023-10-10T18:47:30.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.354+0000 7fb7b57fa700 1 -- 172.21.15.175:0/73656571 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb7bc018360 con 0x7fb7c00c4580 2023-10-10T18:47:30.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.355+0000 7fb7b57fa700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7a4066f30 0x7fb7a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:30.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.355+0000 7fb7b57fa700 1 -- 172.21.15.175:0/73656571 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fb7bc084550 con 0x7fb7c00c4580 2023-10-10T18:47:30.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.355+0000 7fb7c4ed0700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7a4066f30 0x7fb7a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:30.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.355+0000 7fb7c4ed0700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7a4066f30 0x7fb7a40693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00c4ab0 tx=0x7fb7b0007f50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:30.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.359+0000 7fb7b57fa700 1 -- 172.21.15.175:0/73656571 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7bc051730 con 0x7fb7c00c4580 2023-10-10T18:47:30.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.588+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7c01569e0 con 0x7fb7c00c4580 2023-10-10T18:47:30.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.588+0000 7fb7b57fa700 1 -- 172.21.15.175:0/73656571 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7fb7bc054d50 con 0x7fb7c00c4580 2023-10-10T18:47:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:30.590 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:30.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.590+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7a4066f30 msgr2=0x7fb7a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:30.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.590+0000 7fb7c7935700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7a4066f30 0x7fb7a40693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00c4ab0 tx=0x7fb7b0007f50 comp rx=0 tx=0).stop 2023-10-10T18:47:30.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.590+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 msgr2=0x7fb7c014be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:30.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.590+0000 7fb7c7935700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c014be30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c830 tx=0x7fb7bc008040 comp rx=0 tx=0).stop 2023-10-10T18:47:30.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.591+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 shutdown_connections 2023-10-10T18:47:30.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.591+0000 7fb7c7935700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7a4066f30 0x7fb7a40693e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.591+0000 7fb7c7935700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c00c7190 0x7fb7c00cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.591+0000 7fb7c7935700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c00c4580 0x7fb7c014be30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.591+0000 7fb7c7935700 1 --2- 172.21.15.175:0/73656571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c00c6330 0x7fb7c014b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.591+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 >> 172.21.15.175:0/73656571 conn(0x7fb7c006ef20 msgr2=0x7fb7c014f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.591+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 shutdown_connections 2023-10-10T18:47:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:30.591+0000 7fb7c7935700 1 -- 172.21.15.175:0/73656571 wait complete. 2023-10-10T18:47:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:30.603 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:47:32.604 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:32.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.728+0000 7fd94c053700 1 Processor -- start 2023-10-10T18:47:32.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd94c053700 1 -- start start 2023-10-10T18:47:32.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd94c053700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441573d0 0x7fd9440c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:32.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd94c053700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd9441577a0 0x7fd9440c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:32.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd94c053700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd944154aa0 con 0x7fd9440bf6e0 2023-10-10T18:47:32.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd94c053700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd944063dc0 con 0x7fd9441577a0 2023-10-10T18:47:32.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd94c053700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd944157bf0 con 0x7fd9441573d0 2023-10-10T18:47:32.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd9495ee700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd9441577a0 0x7fd9440c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:32.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd94a5f0700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd9440bf6e0 0x7fd9440c6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35808/0 (socket says 172.21.15.175:35808) 2023-10-10T18:47:32.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.729+0000 7fd94a5f0700 1 -- 172.21.15.175:0/973879268 learned_addr learned my addr 172.21.15.175:0/973879268 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:32.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd949def700 1 --2- 172.21.15.175:0/973879268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441573d0 0x7fd9440c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:32.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd9495ee700 1 -- 172.21.15.175:0/973879268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441573d0 msgr2=0x7fd9440c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:32.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd9495ee700 1 --2- 172.21.15.175:0/973879268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441573d0 0x7fd9440c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd9495ee700 1 -- 172.21.15.175:0/973879268 >> v1:172.21.15.175:6789/0 conn(0x7fd9440bf6e0 legacy=0x7fd9440c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:32.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd9495ee700 1 -- 172.21.15.175:0/973879268 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd944157e50 con 0x7fd9441577a0 2023-10-10T18:47:32.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd948ded700 1 -- 172.21.15.175:0/973879268 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070063566 0 0) 0x7fd944154aa0 con 0x7fd9440bf6e0 2023-10-10T18:47:32.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd9495ee700 1 --2- 172.21.15.175:0/973879268 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9441577a0 0x7fd9440c4f20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd9380098e0 tx=0x7fd9380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4aa7b15764761d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:32.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd948ded700 1 -- 172.21.15.175:0/973879268 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd938009e80 con 0x7fd9441577a0 2023-10-10T18:47:32.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd948ded700 1 -- 172.21.15.175:0/973879268 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93800b860 con 0x7fd9441577a0 2023-10-10T18:47:32.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.730+0000 7fd948ded700 1 -- 172.21.15.175:0/973879268 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93800bbe0 con 0x7fd9441577a0 2023-10-10T18:47:32.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.731+0000 7fd94c053700 1 -- 172.21.15.175:0/973879268 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9441577a0 msgr2=0x7fd9440c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:32.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.731+0000 7fd94c053700 1 --2- 172.21.15.175:0/973879268 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9441577a0 0x7fd9440c4f20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd9380098e0 tx=0x7fd9380091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.731+0000 7fd94c053700 1 -- 172.21.15.175:0/973879268 shutdown_connections 2023-10-10T18:47:32.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.731+0000 7fd94c053700 1 --2- 172.21.15.175:0/973879268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441573d0 0x7fd9440c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.731+0000 7fd94c053700 1 --2- 172.21.15.175:0/973879268 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9441577a0 0x7fd9440c4f20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.731+0000 7fd94c053700 1 -- 172.21.15.175:0/973879268 >> 172.21.15.175:0/973879268 conn(0x7fd94406dc70 msgr2=0x7fd944063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:32.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.731+0000 7fd94c053700 1 -- 172.21.15.175:0/973879268 shutdown_connections 2023-10-10T18:47:32.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.731+0000 7fd94c053700 1 -- 172.21.15.175:0/973879268 wait complete. 2023-10-10T18:47:32.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.732+0000 7fd94c053700 1 Processor -- start 2023-10-10T18:47:32.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.732+0000 7fd94c053700 1 -- start start 2023-10-10T18:47:32.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94c053700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9440bf6e0 0x7fd9440c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94c053700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9441573d0 0x7fd9440c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94c053700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441577a0 0x7fd944166e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94c053700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd944153790 con 0x7fd9440bf6e0 2023-10-10T18:47:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94c053700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd944153c80 con 0x7fd9441573d0 2023-10-10T18:47:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94c053700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440c3150 con 0x7fd9441577a0 2023-10-10T18:47:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9440bf6e0 0x7fd9440c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9440bf6e0 0x7fd9440c35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58456/0 (socket says 172.21.15.175:58456) 2023-10-10T18:47:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 -- 172.21.15.175:0/2704479691 learned_addr learned my addr 172.21.15.175:0/2704479691 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd9495ee700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9441573d0 0x7fd9440c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94a5f0700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441577a0 0x7fd944166e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 -- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441577a0 msgr2=0x7fd944166e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441577a0 0x7fd944166e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 -- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9441573d0 msgr2=0x7fd9440c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9441573d0 0x7fd9440c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 -- 172.21.15.175:0/2704479691 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd938009420 con 0x7fd9440bf6e0 2023-10-10T18:47:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd9495ee700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9441573d0 0x7fd9440c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:47:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94a5f0700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441577a0 0x7fd944166e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:47:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd949def700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9440bf6e0 0x7fd9440c35e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd93400e190 tx=0x7fd934009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd941ffb700 1 -- 172.21.15.175:0/2704479691 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93400ee30 con 0x7fd9440bf6e0 2023-10-10T18:47:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd941ffb700 1 -- 172.21.15.175:0/2704479691 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd934014420 con 0x7fd9440bf6e0 2023-10-10T18:47:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9441539a0 con 0x7fd9440bf6e0 2023-10-10T18:47:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.733+0000 7fd941ffb700 1 -- 172.21.15.175:0/2704479691 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9340147e0 con 0x7fd9440bf6e0 2023-10-10T18:47:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.734+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9440c2de0 con 0x7fd9440bf6e0 2023-10-10T18:47:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.735+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd944067570 con 0x7fd9440bf6e0 2023-10-10T18:47:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.735+0000 7fd941ffb700 1 -- 172.21.15.175:0/2704479691 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd934014bd0 con 0x7fd9440bf6e0 2023-10-10T18:47:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.735+0000 7fd941ffb700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd930066f30 0x7fd9300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.735+0000 7fd941ffb700 1 -- 172.21.15.175:0/2704479691 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fd934084e00 con 0x7fd9440bf6e0 2023-10-10T18:47:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.735+0000 7fd9495ee700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd930066f30 0x7fd9300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.736+0000 7fd9495ee700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd930066f30 0x7fd9300693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd9380091e0 tx=0x7fd938017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.739+0000 7fd941ffb700 1 -- 172.21.15.175:0/2704479691 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd934051fe0 con 0x7fd9440bf6e0 2023-10-10T18:47:32.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.965+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd944063ae0 con 0x7fd9440bf6e0 2023-10-10T18:47:32.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.966+0000 7fd941ffb700 1 -- 172.21.15.175:0/2704479691 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7fd934055600 con 0x7fd9440bf6e0 2023-10-10T18:47:32.967 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:32.967 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:32.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd930066f30 msgr2=0x7fd9300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:32.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd930066f30 0x7fd9300693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd9380091e0 tx=0x7fd938017000 comp rx=0 tx=0).stop 2023-10-10T18:47:32.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9440bf6e0 msgr2=0x7fd9440c35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:32.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9440bf6e0 0x7fd9440c35e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd93400e190 tx=0x7fd934009920 comp rx=0 tx=0).stop 2023-10-10T18:47:32.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 shutdown_connections 2023-10-10T18:47:32.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd930066f30 0x7fd9300693e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd9441577a0 0x7fd944166e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9440bf6e0 0x7fd9440c35e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 --2- 172.21.15.175:0/2704479691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9441573d0 0x7fd9440c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:32.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 >> 172.21.15.175:0/2704479691 conn(0x7fd94406dc70 msgr2=0x7fd944147c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:32.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 shutdown_connections 2023-10-10T18:47:32.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:32.968+0000 7fd94c053700 1 -- 172.21.15.175:0/2704479691 wait complete. 2023-10-10T18:47:32.972 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:32.981 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:47:34.982 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:35.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.107+0000 7f4916473700 1 Processor -- start 2023-10-10T18:47:35.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.107+0000 7f4916473700 1 -- start start 2023-10-10T18:47:35.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f4916473700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f49100b1fc0 0x7f4910155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:35.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f4916473700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f49100b2390 0x7f4910068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:35.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f4916473700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49101574b0 con 0x7f4910068bd0 2023-10-10T18:47:35.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f4916473700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4910150770 con 0x7f49100b2390 2023-10-10T18:47:35.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f4916473700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49100afad0 con 0x7f49100b1fc0 2023-10-10T18:47:35.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f49100b2390 0x7f4910068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:35.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f49100b2390 0x7f4910068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52336/0 (socket says 172.21.15.175:52336) 2023-10-10T18:47:35.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490f7fe700 1 -- 172.21.15.175:0/3620239387 learned_addr learned my addr 172.21.15.175:0/3620239387 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:35.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490ffff700 1 --2- 172.21.15.175:0/3620239387 >> v2:172.21.15.191:3300/0 conn(0x7f49100b1fc0 0x7f4910155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:35.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490f7fe700 1 -- 172.21.15.175:0/3620239387 >> v2:172.21.15.191:3300/0 conn(0x7f49100b1fc0 msgr2=0x7f4910155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:35.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490f7fe700 1 --2- 172.21.15.175:0/3620239387 >> v2:172.21.15.191:3300/0 conn(0x7f49100b1fc0 0x7f4910155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490f7fe700 1 -- 172.21.15.175:0/3620239387 >> v1:172.21.15.191:6789/0 conn(0x7f4910068bd0 legacy=0x7f49100aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:35.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490f7fe700 1 -- 172.21.15.175:0/3620239387 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100af4c0 con 0x7f49100b2390 2023-10-10T18:47:35.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490f7fe700 1 --2- 172.21.15.175:0/3620239387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b2390 0x7f4910068690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f49100b28c0 tx=0x7f4904009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d8d04839441527 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:35.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490effd700 1 -- 172.21.15.175:0/3620239387 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904009e70 con 0x7f49100b2390 2023-10-10T18:47:35.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490effd700 1 -- 172.21.15.175:0/3620239387 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490400b8f0 con 0x7f49100b2390 2023-10-10T18:47:35.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.108+0000 7f490effd700 1 -- 172.21.15.175:0/3620239387 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490400bc90 con 0x7f49100b2390 2023-10-10T18:47:35.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.109+0000 7f4916473700 1 -- 172.21.15.175:0/3620239387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b2390 msgr2=0x7f4910068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:35.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.109+0000 7f4916473700 1 --2- 172.21.15.175:0/3620239387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b2390 0x7f4910068690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f49100b28c0 tx=0x7f4904009250 comp rx=0 tx=0).stop 2023-10-10T18:47:35.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.109+0000 7f4916473700 1 -- 172.21.15.175:0/3620239387 shutdown_connections 2023-10-10T18:47:35.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.109+0000 7f4916473700 1 --2- 172.21.15.175:0/3620239387 >> v2:172.21.15.191:3300/0 conn(0x7f49100b1fc0 0x7f4910155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.109+0000 7f4916473700 1 --2- 172.21.15.175:0/3620239387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b2390 0x7f4910068690 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.109+0000 7f4916473700 1 -- 172.21.15.175:0/3620239387 >> 172.21.15.175:0/3620239387 conn(0x7f4910063230 msgr2=0x7f4910063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:35.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.109+0000 7f4916473700 1 -- 172.21.15.175:0/3620239387 shutdown_connections 2023-10-10T18:47:35.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.109+0000 7f4916473700 1 -- 172.21.15.175:0/3620239387 wait complete. 2023-10-10T18:47:35.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.110+0000 7f4916473700 1 Processor -- start 2023-10-10T18:47:35.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.110+0000 7f4916473700 1 -- start start 2023-10-10T18:47:35.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f4916473700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4910068bd0 0x7f49100cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:35.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f4916473700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b1fc0 0x7f49100cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:35.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f4916473700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49100b2390 0x7f49100caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f4916473700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100cd260 con 0x7f49100b1fc0 2023-10-10T18:47:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f4916473700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49101564d0 con 0x7f4910068bd0 2023-10-10T18:47:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f4916473700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100c34d0 con 0x7f49100b2390 2023-10-10T18:47:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b1fc0 0x7f49100cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b1fc0 0x7f49100cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52346/0 (socket says 172.21.15.175:52346) 2023-10-10T18:47:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490f7fe700 1 -- 172.21.15.175:0/366078584 learned_addr learned my addr 172.21.15.175:0/366078584 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490ffff700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4910068bd0 0x7f49100cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f4914a10700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49100b2390 0x7f49100caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490f7fe700 1 -- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49100b2390 msgr2=0x7f49100caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:35.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490f7fe700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49100b2390 0x7f49100caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490f7fe700 1 -- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4910068bd0 msgr2=0x7f49100cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490f7fe700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4910068bd0 0x7f49100cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.111+0000 7f490f7fe700 1 -- 172.21.15.175:0/366078584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49040094f0 con 0x7f49100b1fc0 2023-10-10T18:47:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.112+0000 7f490f7fe700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b1fc0 0x7f49100cc2f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f491006fd90 tx=0x7f4904017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.112+0000 7f48f7fff700 1 -- 172.21.15.175:0/366078584 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490400bc70 con 0x7f49100b1fc0 2023-10-10T18:47:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.112+0000 7f48f7fff700 1 -- 172.21.15.175:0/366078584 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490401a450 con 0x7f49100b1fc0 2023-10-10T18:47:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.112+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4910166c60 con 0x7f49100b1fc0 2023-10-10T18:47:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.112+0000 7f48f7fff700 1 -- 172.21.15.175:0/366078584 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490401a7f0 con 0x7f49100b1fc0 2023-10-10T18:47:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.112+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49101561f0 con 0x7f49100b1fc0 2023-10-10T18:47:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.113+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49100c3680 con 0x7f49100b1fc0 2023-10-10T18:47:35.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.113+0000 7f48f7fff700 1 -- 172.21.15.175:0/366078584 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f490401ab60 con 0x7f49100b1fc0 2023-10-10T18:47:35.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.114+0000 7f48f7fff700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f48fc066f60 0x7f48fc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:35.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.114+0000 7f48f7fff700 1 -- 172.21.15.175:0/366078584 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f4904085f60 con 0x7f49100b1fc0 2023-10-10T18:47:35.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.114+0000 7f490ffff700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f48fc066f60 0x7f48fc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:35.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.114+0000 7f490ffff700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f48fc066f60 0x7f48fc069410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4900009db0 tx=0x7f49000091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:35.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.118+0000 7f48f7fff700 1 -- 172.21.15.175:0/366078584 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f49100c3680 con 0x7f49100b1fc0 2023-10-10T18:47:35.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.343+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49100c3680 con 0x7f49100b1fc0 2023-10-10T18:47:35.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.344+0000 7f48f7fff700 1 -- 172.21.15.175:0/366078584 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7f49100c3680 con 0x7f49100b1fc0 2023-10-10T18:47:35.345 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:35.345 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:35.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f48fc066f60 msgr2=0x7f48fc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:35.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f48fc066f60 0x7f48fc069410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4900009db0 tx=0x7f49000091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b1fc0 msgr2=0x7f49100cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:35.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b1fc0 0x7f49100cc2f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f491006fd90 tx=0x7f4904017000 comp rx=0 tx=0).stop 2023-10-10T18:47:35.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 shutdown_connections 2023-10-10T18:47:35.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f48fc066f60 0x7f48fc069410 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49100b2390 0x7f49100caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4910068bd0 0x7f49100cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 --2- 172.21.15.175:0/366078584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f49100b1fc0 0x7f49100cc2f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:35.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 >> 172.21.15.175:0/366078584 conn(0x7f4910063230 msgr2=0x7f4910150ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:35.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 shutdown_connections 2023-10-10T18:47:35.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:35.346+0000 7f4916473700 1 -- 172.21.15.175:0/366078584 wait complete. 2023-10-10T18:47:35.350 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:35.358 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:47:37.360 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:37.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.485+0000 7fd9112b6700 1 Processor -- start 2023-10-10T18:47:37.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.485+0000 7fd9112b6700 1 -- start start 2023-10-10T18:47:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd9112b6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd90c0b2390 0x7fd90c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd9112b6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd90c0aff60 0x7fd90c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd9112b6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c1574b0 con 0x7fd90c0b1fc0 2023-10-10T18:47:37.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd9112b6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c150770 con 0x7fd90c0aff60 2023-10-10T18:47:37.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd9112b6700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c0afad0 con 0x7fd90c0b2390 2023-10-10T18:47:37.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd90b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd90c0aff60 0x7fd90c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:37.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd90b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd90c0aff60 0x7fd90c0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52356/0 (socket says 172.21.15.175:52356) 2023-10-10T18:47:37.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd90b7fe700 1 -- 172.21.15.175:0/321843819 learned_addr learned my addr 172.21.15.175:0/321843819 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:37.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd903fff700 1 --2- 172.21.15.175:0/321843819 >> v2:172.21.15.191:3300/0 conn(0x7fd90c0b2390 0x7fd90c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:37.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd90b7fe700 1 -- 172.21.15.175:0/321843819 >> v2:172.21.15.191:3300/0 conn(0x7fd90c0b2390 msgr2=0x7fd90c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:37.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd90b7fe700 1 --2- 172.21.15.175:0/321843819 >> v2:172.21.15.191:3300/0 conn(0x7fd90c0b2390 0x7fd90c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd90b7fe700 1 -- 172.21.15.175:0/321843819 >> v1:172.21.15.191:6789/0 conn(0x7fd90c0b1fc0 legacy=0x7fd90c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:37.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd90b7fe700 1 -- 172.21.15.175:0/321843819 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c0af4c0 con 0x7fd90c0aff60 2023-10-10T18:47:37.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.486+0000 7fd90b7fe700 1 --2- 172.21.15.175:0/321843819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c0b03e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0c0290 tx=0x7fd904009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d3d81a16d562a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:37.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd90a7fc700 1 -- 172.21.15.175:0/321843819 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90400e070 con 0x7fd90c0aff60 2023-10-10T18:47:37.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd90a7fc700 1 -- 172.21.15.175:0/321843819 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90400baa0 con 0x7fd90c0aff60 2023-10-10T18:47:37.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd90a7fc700 1 -- 172.21.15.175:0/321843819 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9040092e0 con 0x7fd90c0aff60 2023-10-10T18:47:37.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd9112b6700 1 -- 172.21.15.175:0/321843819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 msgr2=0x7fd90c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:37.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd9112b6700 1 --2- 172.21.15.175:0/321843819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c0b03e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0c0290 tx=0x7fd904009840 comp rx=0 tx=0).stop 2023-10-10T18:47:37.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd9112b6700 1 -- 172.21.15.175:0/321843819 shutdown_connections 2023-10-10T18:47:37.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd9112b6700 1 --2- 172.21.15.175:0/321843819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c0b03e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd9112b6700 1 --2- 172.21.15.175:0/321843819 >> v2:172.21.15.191:3300/0 conn(0x7fd90c0b2390 0x7fd90c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd9112b6700 1 -- 172.21.15.175:0/321843819 >> 172.21.15.175:0/321843819 conn(0x7fd90c063230 msgr2=0x7fd90c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd9112b6700 1 -- 172.21.15.175:0/321843819 shutdown_connections 2023-10-10T18:47:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.487+0000 7fd9112b6700 1 -- 172.21.15.175:0/321843819 wait complete. 2023-10-10T18:47:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.488+0000 7fd9112b6700 1 Processor -- start 2023-10-10T18:47:37.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.488+0000 7fd9112b6700 1 -- start start 2023-10-10T18:47:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd9112b6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c169d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd9112b6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd90c0b1fc0 0x7fd90c165d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd9112b6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd90c0b2390 0x7fd90c1595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd9112b6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c169890 con 0x7fd90c0aff60 2023-10-10T18:47:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd9112b6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c169280 con 0x7fd90c0b1fc0 2023-10-10T18:47:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd9112b6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c154740 con 0x7fd90c0b2390 2023-10-10T18:47:37.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c169d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c169d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52358/0 (socket says 172.21.15.175:52358) 2023-10-10T18:47:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 -- 172.21.15.175:0/2235024352 learned_addr learned my addr 172.21.15.175:0/2235024352 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd903fff700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd90c0b1fc0 0x7fd90c165d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90b7fe700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd90c0b2390 0x7fd90c1595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 -- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd90c0b2390 msgr2=0x7fd90c1595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd90c0b2390 0x7fd90c1595d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 -- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd90c0b1fc0 msgr2=0x7fd90c165d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd90c0b1fc0 0x7fd90c165d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 -- 172.21.15.175:0/2235024352 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904009ae0 con 0x7fd90c0aff60 2023-10-10T18:47:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd903fff700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd90c0b1fc0 0x7fd90c165d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:47:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90b7fe700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd90c0b2390 0x7fd90c1595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:47:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.489+0000 7fd90affd700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c169d20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00b600 tx=0x7fd8fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.490+0000 7fd902ffd700 1 -- 172.21.15.175:0/2235024352 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc00f510 con 0x7fd90c0aff60 2023-10-10T18:47:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.490+0000 7fd902ffd700 1 -- 172.21.15.175:0/2235024352 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc00f670 con 0x7fd90c0aff60 2023-10-10T18:47:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.490+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c154a20 con 0x7fd90c0aff60 2023-10-10T18:47:37.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.490+0000 7fd902ffd700 1 -- 172.21.15.175:0/2235024352 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc00fa30 con 0x7fd90c0aff60 2023-10-10T18:47:37.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.490+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c169520 con 0x7fd90c0aff60 2023-10-10T18:47:37.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.491+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c0713a0 con 0x7fd90c0aff60 2023-10-10T18:47:37.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.491+0000 7fd902ffd700 1 -- 172.21.15.175:0/2235024352 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd8fc01a400 con 0x7fd90c0aff60 2023-10-10T18:47:37.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.491+0000 7fd902ffd700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8ec066f30 0x7fd8ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:37.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.492+0000 7fd902ffd700 1 -- 172.21.15.175:0/2235024352 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fd8fc084230 con 0x7fd90c0aff60 2023-10-10T18:47:37.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.492+0000 7fd903fff700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8ec066f30 0x7fd8ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:37.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.492+0000 7fd903fff700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8ec066f30 0x7fd8ec0693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4006fd0 tx=0x7fd8f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:37.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.496+0000 7fd902ffd700 1 -- 172.21.15.175:0/2235024352 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd90c0713a0 con 0x7fd90c0aff60 2023-10-10T18:47:37.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.721+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd90c0713a0 con 0x7fd90c0aff60 2023-10-10T18:47:37.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.722+0000 7fd902ffd700 1 -- 172.21.15.175:0/2235024352 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7fd90c0713a0 con 0x7fd90c0aff60 2023-10-10T18:47:37.723 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:37.723 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:37.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.723+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8ec066f30 msgr2=0x7fd8ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:37.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.723+0000 7fd9112b6700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8ec066f30 0x7fd8ec0693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4006fd0 tx=0x7fd8f4008040 comp rx=0 tx=0).stop 2023-10-10T18:47:37.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.723+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 msgr2=0x7fd90c169d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:37.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.723+0000 7fd9112b6700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c169d20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00b600 tx=0x7fd8fc0091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.724+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 shutdown_connections 2023-10-10T18:47:37.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.724+0000 7fd9112b6700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8ec066f30 0x7fd8ec0693e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.724+0000 7fd9112b6700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd90c0b2390 0x7fd90c1595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.724+0000 7fd9112b6700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd90c0aff60 0x7fd90c169d20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.724+0000 7fd9112b6700 1 --2- 172.21.15.175:0/2235024352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd90c0b1fc0 0x7fd90c165d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:37.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.724+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 >> 172.21.15.175:0/2235024352 conn(0x7fd90c063230 msgr2=0x7fd90c1508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:37.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.724+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 shutdown_connections 2023-10-10T18:47:37.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:37.724+0000 7fd9112b6700 1 -- 172.21.15.175:0/2235024352 wait complete. 2023-10-10T18:47:37.728 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:37.736 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:47:39.737 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:39.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.863+0000 7f16bf80f700 1 Processor -- start 2023-10-10T18:47:39.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.864+0000 7f16bf80f700 1 -- start start 2023-10-10T18:47:39.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.864+0000 7f16bf80f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b806b640 0x7f16b814cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:39.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.864+0000 7f16bf80f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b81533e0 con 0x7f16b80c3700 2023-10-10T18:47:39.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.864+0000 7f16bf80f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b806e130 con 0x7f16b80c3330 2023-10-10T18:47:39.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.864+0000 7f16bf80f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806a970 con 0x7f16b806b640 2023-10-10T18:47:39.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.864+0000 7f16bddac700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f16b80c3700 0x7f16b815d8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:59038/0 (socket says 172.21.15.175:59038) 2023-10-10T18:47:39.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.864+0000 7f16bddac700 1 -- 172.21.15.175:0/330526627 learned_addr learned my addr 172.21.15.175:0/330526627 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:39.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b806b640 0x7f16b814cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:39.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586429938 0 0) 0x7f16b81533e0 con 0x7f16b80c3700 2023-10-10T18:47:39.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4003570 con 0x7f16b80c3700 2023-10-10T18:47:39.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1054330684 0 0) 0x7f16a4003570 con 0x7f16b80c3700 2023-10-10T18:47:39.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b806b640 msgr2=0x7f16b814cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:39.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b806b640 0x7f16b814cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 >> v1:172.21.15.191:6789/0 conn(0x7f16b80c3330 legacy=0x7f16b80c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80c43b0 con 0x7f16b80c3700 2023-10-10T18:47:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f16b0001470 con 0x7f16b80c3700 2023-10-10T18:47:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 >> v1:172.21.15.175:6789/0 conn(0x7f16b80c3700 legacy=0x7f16b815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16a4003a90 0x7f16a4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:39.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16a40043e0 0x7f16a40048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:39.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16a40050b0 0x7f16a400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:39.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806a970 con 0x7f16a40043e0 2023-10-10T18:47:39.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a400bfa0 con 0x7f16a40050b0 2023-10-10T18:47:39.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a400c160 con 0x7f16a4003a90 2023-10-10T18:47:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b806b640 0x7f16b814cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:47:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16a40043e0 0x7f16a40048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16bd5ab700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16a4003a90 0x7f16a4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.865+0000 7f16bddac700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16a40050b0 0x7f16a400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bcdaa700 1 -- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16a4003a90 msgr2=0x7f16a4003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:39.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16a4003a90 0x7f16a4003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:39.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bcdaa700 1 -- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16a40050b0 msgr2=0x7f16a400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:39.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16a40050b0 0x7f16a400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:39.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bcdaa700 1 -- 172.21.15.175:0/330526627 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4003570 con 0x7f16a40043e0 2023-10-10T18:47:39.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16a40043e0 0x7f16a40048b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f16b806ab50 tx=0x7f16a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c7fede833ee32e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a800e6e0 con 0x7f16a40043e0 2023-10-10T18:47:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a800e840 con 0x7f16a40043e0 2023-10-10T18:47:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16affff700 1 -- 172.21.15.175:0/330526627 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a800ebe0 con 0x7f16a40043e0 2023-10-10T18:47:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bf80f700 1 -- 172.21.15.175:0/330526627 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16a40043e0 msgr2=0x7f16a40048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bf80f700 1 --2- 172.21.15.175:0/330526627 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16a40043e0 0x7f16a40048b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f16b806ab50 tx=0x7f16a80091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:39.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bddac700 1 -- 172.21.15.175:0/330526627 reap_dead start 2023-10-10T18:47:39.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bf80f700 1 -- 172.21.15.175:0/330526627 shutdown_connections 2023-10-10T18:47:39.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.866+0000 7f16bf80f700 1 -- 172.21.15.175:0/330526627 >> 172.21.15.175:0/330526627 conn(0x7f16b8068280 msgr2=0x7f16b806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:39.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.867+0000 7f16bf80f700 1 -- 172.21.15.175:0/330526627 shutdown_connections 2023-10-10T18:47:39.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.867+0000 7f16bf80f700 1 -- 172.21.15.175:0/330526627 wait complete. 2023-10-10T18:47:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.868+0000 7f16bf80f700 1 Processor -- start 2023-10-10T18:47:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.868+0000 7f16bf80f700 1 -- start start 2023-10-10T18:47:39.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.868+0000 7f16bf80f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16b80c74c0 0x7f16b81518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:39.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.868+0000 7f16bf80f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16b8152090 0x7f16b80c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:39.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.868+0000 7f16bf80f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b80b5190 0x7f16b8149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.868+0000 7f16bf80f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8150920 con 0x7f16b80c74c0 2023-10-10T18:47:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.868+0000 7f16bf80f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8151e30 con 0x7f16b8152090 2023-10-10T18:47:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.868+0000 7f16bf80f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8151460 con 0x7f16b80b5190 2023-10-10T18:47:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16b80c74c0 0x7f16b81518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16b80c74c0 0x7f16b81518f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52372/0 (socket says 172.21.15.175:52372) 2023-10-10T18:47:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 -- 172.21.15.175:0/2002140558 learned_addr learned my addr 172.21.15.175:0/2002140558 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bddac700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b80b5190 0x7f16b8149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16b8152090 0x7f16b80c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 -- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b80b5190 msgr2=0x7f16b8149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b80b5190 0x7f16b8149a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:39.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 -- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16b8152090 msgr2=0x7f16b80c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:39.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16b8152090 0x7f16b80c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:39.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 -- 172.21.15.175:0/2002140558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8161ab0 con 0x7f16b80c74c0 2023-10-10T18:47:39.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bd5ab700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16b80c74c0 0x7f16b81518f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f16b400e750 tx=0x7f16b4001ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:39.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16ad7fa700 1 -- 172.21.15.175:0/2002140558 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4012730 con 0x7f16b80c74c0 2023-10-10T18:47:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16ad7fa700 1 -- 172.21.15.175:0/2002140558 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b4012890 con 0x7f16b80c74c0 2023-10-10T18:47:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b8150b30 con 0x7f16b80c74c0 2023-10-10T18:47:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16ad7fa700 1 -- 172.21.15.175:0/2002140558 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4012c50 con 0x7f16b80c74c0 2023-10-10T18:47:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.869+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b81510f0 con 0x7f16b80c74c0 2023-10-10T18:47:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.870+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b8151740 con 0x7f16b80c74c0 2023-10-10T18:47:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.871+0000 7f16ad7fa700 1 -- 172.21.15.175:0/2002140558 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f16b40113e0 con 0x7f16b80c74c0 2023-10-10T18:47:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.871+0000 7f16ad7fa700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f169c066f60 0x7f169c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.871+0000 7f16ad7fa700 1 -- 172.21.15.175:0/2002140558 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f16b4087530 con 0x7f16b80c74c0 2023-10-10T18:47:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.871+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f169c066f60 0x7f169c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.872+0000 7f16bcdaa700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f169c066f60 0x7f169c069410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f16a80091e0 tx=0x7f16a8019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:39.876+0000 7f16ad7fa700 1 -- 172.21.15.175:0/2002140558 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16b40547c0 con 0x7f16b80c74c0 2023-10-10T18:47:40.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.098+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16b8067570 con 0x7f16b80c74c0 2023-10-10T18:47:40.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.099+0000 7f16ad7fa700 1 -- 172.21.15.175:0/2002140558 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7f16b4057de0 con 0x7f16b80c74c0 2023-10-10T18:47:40.100 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:40.101 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:40.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f169c066f60 msgr2=0x7f169c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:40.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f169c066f60 0x7f169c069410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f16a80091e0 tx=0x7f16a8019000 comp rx=0 tx=0).stop 2023-10-10T18:47:40.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16b80c74c0 msgr2=0x7f16b81518f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:40.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16b80c74c0 0x7f16b81518f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f16b400e750 tx=0x7f16b4001ba0 comp rx=0 tx=0).stop 2023-10-10T18:47:40.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 shutdown_connections 2023-10-10T18:47:40.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f169c066f60 0x7f169c069410 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:40.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16b80b5190 0x7f16b8149a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:40.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16b80c74c0 0x7f16b81518f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:40.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 --2- 172.21.15.175:0/2002140558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16b8152090 0x7f16b80c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:40.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 >> 172.21.15.175:0/2002140558 conn(0x7f16b8068280 msgr2=0x7f16b814cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:40.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 shutdown_connections 2023-10-10T18:47:40.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:40.101+0000 7f16bf80f700 1 -- 172.21.15.175:0/2002140558 wait complete. 2023-10-10T18:47:40.107 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:40.114 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:47:42.115 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:42.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.237+0000 7f8c74ae7700 1 Processor -- start 2023-10-10T18:47:42.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.237+0000 7f8c74ae7700 1 -- start start 2023-10-10T18:47:42.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.238+0000 7f8c74ae7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8c70155fb0 0x7f8c70159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:42.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.238+0000 7f8c74ae7700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c701574b0 con 0x7f8c700b1fc0 2023-10-10T18:47:42.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.238+0000 7f8c74ae7700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c700afad0 con 0x7f8c700b2390 2023-10-10T18:47:42.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.238+0000 7f8c74ae7700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c70150770 con 0x7f8c70155fb0 2023-10-10T18:47:42.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8c70155fb0 0x7f8c70159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:42.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6e59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8c700b1fc0 0x7f8c70068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:59042/0 (socket says 172.21.15.175:59042) 2023-10-10T18:47:42.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6e59c700 1 -- 172.21.15.175:0/3118114673 learned_addr learned my addr 172.21.15.175:0/3118114673 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:42.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6ed9d700 1 -- 172.21.15.175:0/3118114673 >> v1:172.21.15.191:6789/0 conn(0x7f8c700b2390 legacy=0x7f8c700aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:42.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6ed9d700 1 -- 172.21.15.175:0/3118114673 >> v1:172.21.15.175:6789/0 conn(0x7f8c700b1fc0 legacy=0x7f8c70068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:42.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6d59a700 1 -- 172.21.15.175:0/3118114673 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790980322 0 0) 0x7f8c701574b0 con 0x7f8c700b1fc0 2023-10-10T18:47:42.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6ed9d700 1 -- 172.21.15.175:0/3118114673 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c700af4c0 con 0x7f8c70155fb0 2023-10-10T18:47:42.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6ed9d700 1 --2- 172.21.15.175:0/3118114673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c70155fb0 0x7f8c70159920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8c680098e0 tx=0x7f8c680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6507ba8fa8b50bdb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:42.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6d59a700 1 -- 172.21.15.175:0/3118114673 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68009e80 con 0x7f8c70155fb0 2023-10-10T18:47:42.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6d59a700 1 -- 172.21.15.175:0/3118114673 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6800b8f0 con 0x7f8c70155fb0 2023-10-10T18:47:42.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.239+0000 7f8c6d59a700 1 -- 172.21.15.175:0/3118114673 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6800bc70 con 0x7f8c70155fb0 2023-10-10T18:47:42.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.240+0000 7f8c74ae7700 1 -- 172.21.15.175:0/3118114673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c70155fb0 msgr2=0x7f8c70159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:42.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.240+0000 7f8c74ae7700 1 --2- 172.21.15.175:0/3118114673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c70155fb0 0x7f8c70159920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8c680098e0 tx=0x7f8c680091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:42.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.240+0000 7f8c74ae7700 1 -- 172.21.15.175:0/3118114673 shutdown_connections 2023-10-10T18:47:42.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.240+0000 7f8c74ae7700 1 --2- 172.21.15.175:0/3118114673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c70155fb0 0x7f8c70159920 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:42.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.240+0000 7f8c74ae7700 1 -- 172.21.15.175:0/3118114673 >> 172.21.15.175:0/3118114673 conn(0x7f8c70063230 msgr2=0x7f8c70063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:42.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.240+0000 7f8c74ae7700 1 -- 172.21.15.175:0/3118114673 shutdown_connections 2023-10-10T18:47:42.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.240+0000 7f8c74ae7700 1 -- 172.21.15.175:0/3118114673 wait complete. 2023-10-10T18:47:42.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.241+0000 7f8c74ae7700 1 Processor -- start 2023-10-10T18:47:42.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.241+0000 7f8c74ae7700 1 -- start start 2023-10-10T18:47:42.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c74ae7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c700b1fc0 0x7f8c700cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:42.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c74ae7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c700b2390 0x7f8c700cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:42.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c74ae7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c70155fb0 0x7f8c70169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:42.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c74ae7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700688a0 con 0x7f8c700b1fc0 2023-10-10T18:47:42.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c74ae7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700c3870 con 0x7f8c70155fb0 2023-10-10T18:47:42.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c74ae7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c70169e00 con 0x7f8c700b2390 2023-10-10T18:47:42.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c700b1fc0 0x7f8c700cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:42.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c700b1fc0 0x7f8c700cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52402/0 (socket says 172.21.15.175:52402) 2023-10-10T18:47:42.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 -- 172.21.15.175:0/1716152638 learned_addr learned my addr 172.21.15.175:0/1716152638 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:42.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6ed9d700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c70155fb0 0x7f8c70169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:42.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6dd9b700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c700b2390 0x7f8c700cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:42.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 -- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c700b2390 msgr2=0x7f8c700cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:42.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c700b2390 0x7f8c700cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:42.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 -- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c70155fb0 msgr2=0x7f8c70169310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:42.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c70155fb0 0x7f8c70169310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:42.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 -- 172.21.15.175:0/1716152638 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68009420 con 0x7f8c700b1fc0 2023-10-10T18:47:42.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c6e59c700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c700b1fc0 0x7f8c700cb2d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000ccc0 tx=0x7f8c60001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:42.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c5e7fc700 1 -- 172.21.15.175:0/1716152638 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6000ab70 con 0x7f8c700b1fc0 2023-10-10T18:47:42.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c5e7fc700 1 -- 172.21.15.175:0/1716152638 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6000acd0 con 0x7f8c700b1fc0 2023-10-10T18:47:42.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c700c3b30 con 0x7f8c700b1fc0 2023-10-10T18:47:42.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c5e7fc700 1 -- 172.21.15.175:0/1716152638 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c60007340 con 0x7f8c700b1fc0 2023-10-10T18:47:42.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.242+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c700cbb50 con 0x7f8c700b1fc0 2023-10-10T18:47:42.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.243+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c700cbd70 con 0x7f8c700b1fc0 2023-10-10T18:47:42.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.244+0000 7f8c5e7fc700 1 -- 172.21.15.175:0/1716152638 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8c60007770 con 0x7f8c700b1fc0 2023-10-10T18:47:42.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.244+0000 7f8c5e7fc700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8c50066f60 0x7f8c50069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:42.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.244+0000 7f8c5e7fc700 1 -- 172.21.15.175:0/1716152638 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f8c60084a20 con 0x7f8c700b1fc0 2023-10-10T18:47:42.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.244+0000 7f8c6dd9b700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8c50066f60 0x7f8c50069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:42.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.245+0000 7f8c6dd9b700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8c50066f60 0x7f8c50069410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8c70071e90 tx=0x7f8c64009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:42.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.248+0000 7f8c5e7fc700 1 -- 172.21.15.175:0/1716152638 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c60051c80 con 0x7f8c700b1fc0 2023-10-10T18:47:42.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.475+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7015a120 con 0x7f8c700b1fc0 2023-10-10T18:47:42.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.476+0000 7f8c5e7fc700 1 -- 172.21.15.175:0/1716152638 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6184 (secure 0 0 0) 0x7f8c600552a0 con 0x7f8c700b1fc0 2023-10-10T18:47:42.476 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:42.476 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:22.476646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.191:6836/3007475551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3007475551},{"type":"v1","addr":"172.21.15.191:6836","nonce":3007475551}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:42.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.477+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8c50066f60 msgr2=0x7f8c50069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:42.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.477+0000 7f8c74ae7700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8c50066f60 0x7f8c50069410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8c70071e90 tx=0x7f8c64009040 comp rx=0 tx=0).stop 2023-10-10T18:47:42.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c700b1fc0 msgr2=0x7f8c700cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:42.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c700b1fc0 0x7f8c700cb2d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000ccc0 tx=0x7f8c60001a80 comp rx=0 tx=0).stop 2023-10-10T18:47:42.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 shutdown_connections 2023-10-10T18:47:42.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8c50066f60 0x7f8c50069410 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:42.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c70155fb0 0x7f8c70169310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:42.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c700b1fc0 0x7f8c700cb2d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:42.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 --2- 172.21.15.175:0/1716152638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c700b2390 0x7f8c700cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:42.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 >> 172.21.15.175:0/1716152638 conn(0x7f8c70063230 msgr2=0x7f8c70151700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:42.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 shutdown_connections 2023-10-10T18:47:42.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:42.478+0000 7f8c74ae7700 1 -- 172.21.15.175:0/1716152638 wait complete. 2023-10-10T18:47:42.481 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 22 2023-10-10T18:47:42.489 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:47:43.439 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/30 file after the snapshot 2023-10-10T18:47:43.441 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/31 file after the snapshot 2023-10-10T18:47:43.442 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/32 file after the snapshot 2023-10-10T18:47:43.443 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/33 file after the snapshot 2023-10-10T18:47:43.444 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/34 file after the snapshot 2023-10-10T18:47:43.445 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/35 file after the snapshot 2023-10-10T18:47:43.446 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/36 file after the snapshot 2023-10-10T18:47:43.447 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/37 file after the snapshot 2023-10-10T18:47:43.449 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/38 file after the snapshot 2023-10-10T18:47:43.450 INFO:tasks.workunit.client.0.smithi175.stdout: create 226/39 file after the snapshot 2023-10-10T18:47:43.451 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/.snap/snap-subdir-test 2023-10-10T18:47:44.490 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:44.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.615+0000 7fe00d0b4700 1 Processor -- start 2023-10-10T18:47:44.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.615+0000 7fe00d0b4700 1 -- start start 2023-10-10T18:47:44.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.616+0000 7fe00d0b4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe008159d70 0x7fe0080c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.616+0000 7fe00d0b4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008156800 con 0x7fe00814fbf0 2023-10-10T18:47:44.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.616+0000 7fe00d0b4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008063e30 con 0x7fe00814ffc0 2023-10-10T18:47:44.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.616+0000 7fe00d0b4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0080c59e0 con 0x7fe008159d70 2023-10-10T18:47:44.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.616+0000 7fe006d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe00814fbf0 0x7fe008063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45786/0 (socket says 172.21.15.175:45786) 2023-10-10T18:47:44.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.616+0000 7fe006d9d700 1 -- 172.21.15.175:0/1576418141 learned_addr learned my addr 172.21.15.175:0/1576418141 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:44.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00759e700 1 --2- 172.21.15.175:0/1576418141 >> v2:172.21.15.191:3300/0 conn(0x7fe008159d70 0x7fe0080c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378255800 0 0) 0x7fe008156800 con 0x7fe00814fbf0 2023-10-10T18:47:44.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff4003570 con 0x7fe00814fbf0 2023-10-10T18:47:44.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1492565218 0 0) 0x7fdff4003570 con 0x7fe00814fbf0 2023-10-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 >> v2:172.21.15.191:3300/0 conn(0x7fe008159d70 msgr2=0x7fe0080c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 --2- 172.21.15.175:0/1576418141 >> v2:172.21.15.191:3300/0 conn(0x7fe008159d70 0x7fe0080c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 >> v1:172.21.15.191:6789/0 conn(0x7fe00814ffc0 legacy=0x7fe0080681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe008156c90 con 0x7fe00814fbf0 2023-10-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fdff80015f0 con 0x7fe00814fbf0 2023-10-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 >> v1:172.21.15.175:6789/0 conn(0x7fe00814fbf0 legacy=0x7fe008063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdff4003a60 0x7fdff4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff4004810 0x7fdff4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdff40051c0 0x7fdff400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0080c59e0 con 0x7fdff40051c0 2023-10-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff400c0d0 con 0x7fdff4004810 2023-10-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff400c290 con 0x7fdff4003a60 2023-10-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00759e700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdff40051c0 0x7fdff400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe006d9d700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdff4003a60 0x7fdff4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fdffe59c700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff4004810 0x7fdff4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00759e700 1 -- 172.21.15.175:0/1576418141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdff4003a60 msgr2=0x7fdff4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00759e700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdff4003a60 0x7fdff4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00759e700 1 -- 172.21.15.175:0/1576418141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff4004810 msgr2=0x7fdff4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00759e700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff4004810 0x7fdff4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00759e700 1 -- 172.21.15.175:0/1576418141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff4003570 con 0x7fdff40051c0 2023-10-10T18:47:44.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.617+0000 7fe00759e700 1 -- 172.21.15.175:0/1576418141 reap_dead start 2023-10-10T18:47:44.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00759e700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdff40051c0 0x7fdff400ba60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe00000e210 tx=0x7fe00000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac28205a4c5c68f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:44.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00000eac0 con 0x7fdff40051c0 2023-10-10T18:47:44.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00000f040 con 0x7fdff40051c0 2023-10-10T18:47:44.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00659c700 1 -- 172.21.15.175:0/1576418141 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00000ee70 con 0x7fdff40051c0 2023-10-10T18:47:44.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1576418141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdff40051c0 msgr2=0x7fdff400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00d0b4700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdff40051c0 0x7fdff400ba60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe00000e210 tx=0x7fe00000b040 comp rx=0 tx=0).stop 2023-10-10T18:47:44.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1576418141 shutdown_connections 2023-10-10T18:47:44.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00d0b4700 1 --2- 172.21.15.175:0/1576418141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdff40051c0 0x7fdff400ba60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1576418141 >> 172.21.15.175:0/1576418141 conn(0x7fe0080cc760 msgr2=0x7fe0080ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:44.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1576418141 shutdown_connections 2023-10-10T18:47:44.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.618+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1576418141 wait complete. 2023-10-10T18:47:44.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.619+0000 7fe00d0b4700 1 Processor -- start 2023-10-10T18:47:44.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.619+0000 7fe00d0b4700 1 -- start start 2023-10-10T18:47:44.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00d0b4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff40051c0 0x7fe0080a9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00d0b4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe00814d9c0 0x7fe0080a7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00d0b4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0080a82a0 0x7fe0080a8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00d0b4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0080b2530 con 0x7fe0080a82a0 2023-10-10T18:47:44.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00d0b4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0080c9030 con 0x7fdff40051c0 2023-10-10T18:47:44.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00d0b4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0080c6da0 con 0x7fe00814d9c0 2023-10-10T18:47:44.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0080a82a0 0x7fe0080a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0080a82a0 0x7fe0080a8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54148/0 (socket says 172.21.15.175:54148) 2023-10-10T18:47:44.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00759e700 1 -- 172.21.15.175:0/1692962507 learned_addr learned my addr 172.21.15.175:0/1692962507 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:44.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe006d9d700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff40051c0 0x7fe0080a9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fdffe59c700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe00814d9c0 0x7fe0080a7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00759e700 1 -- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe00814d9c0 msgr2=0x7fe0080a7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00759e700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe00814d9c0 0x7fe0080a7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00759e700 1 -- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff40051c0 msgr2=0x7fe0080a9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00759e700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff40051c0 0x7fe0080a9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.620+0000 7fe00759e700 1 -- 172.21.15.175:0/1692962507 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe008166490 con 0x7fe0080a82a0 2023-10-10T18:47:44.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.621+0000 7fe00759e700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0080a82a0 0x7fe0080a8740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe0080b2840 tx=0x7fe000019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:44.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.621+0000 7fdfff7fe700 1 -- 172.21.15.175:0/1692962507 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00001d660 con 0x7fe0080a82a0 2023-10-10T18:47:44.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.621+0000 7fdfff7fe700 1 -- 172.21.15.175:0/1692962507 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe000022070 con 0x7fe0080a82a0 2023-10-10T18:47:44.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.621+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0080c9240 con 0x7fe0080a82a0 2023-10-10T18:47:44.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.621+0000 7fdfff7fe700 1 -- 172.21.15.175:0/1692962507 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00001da50 con 0x7fe0080a82a0 2023-10-10T18:47:44.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.621+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0080c9bf0 con 0x7fe0080a82a0 2023-10-10T18:47:44.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.622+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0080c9de0 con 0x7fe0080a82a0 2023-10-10T18:47:44.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.622+0000 7fdfff7fe700 1 -- 172.21.15.175:0/1692962507 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe000003020 con 0x7fe0080a82a0 2023-10-10T18:47:44.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.622+0000 7fdfff7fe700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfe8066f60 0x7fdfe8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.623+0000 7fdfff7fe700 1 -- 172.21.15.175:0/1692962507 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fe000024070 con 0x7fe0080a82a0 2023-10-10T18:47:44.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.623+0000 7fe006d9d700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfe8066f60 0x7fdfe8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.623+0000 7fe006d9d700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfe8066f60 0x7fdfe8069410 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdff8007a30 tx=0x7fdff8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:44.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.627+0000 7fdfff7fe700 1 -- 172.21.15.175:0/1692962507 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe000053530 con 0x7fe0080a82a0 2023-10-10T18:47:44.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.851+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe008067570 con 0x7fe0080a82a0 2023-10-10T18:47:44.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.852+0000 7fdfff7fe700 1 -- 172.21.15.175:0/1692962507 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5425 (secure 0 0 0) 0x7fe000056b50 con 0x7fe0080a82a0 2023-10-10T18:47:44.852 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:44.853 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:43.409578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:44.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfe8066f60 msgr2=0x7fdfe8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfe8066f60 0x7fdfe8069410 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdff8007a30 tx=0x7fdff8009040 comp rx=0 tx=0).stop 2023-10-10T18:47:44.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0080a82a0 msgr2=0x7fe0080a8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0080a82a0 0x7fe0080a8740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe0080b2840 tx=0x7fe000019000 comp rx=0 tx=0).stop 2023-10-10T18:47:44.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 shutdown_connections 2023-10-10T18:47:44.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfe8066f60 0x7fdfe8069410 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0080a82a0 0x7fe0080a8740 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdff40051c0 0x7fe0080a9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 --2- 172.21.15.175:0/1692962507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe00814d9c0 0x7fe0080a7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 >> 172.21.15.175:0/1692962507 conn(0x7fe0080cc760 msgr2=0x7fe0081594c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:44.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 shutdown_connections 2023-10-10T18:47:44.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.854+0000 7fe00d0b4700 1 -- 172.21.15.175:0/1692962507 wait complete. 2023-10-10T18:47:44.857 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 23 2023-10-10T18:47:44.866 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-10T18:47:44.866 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T18:47:44.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:44.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.981+0000 7ff2a9e14700 1 Processor -- start 2023-10-10T18:47:44.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.981+0000 7ff2a9e14700 1 -- start start 2023-10-10T18:47:44.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a9e14700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a4155fb0 0x7ff2a4159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a9e14700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a41574b0 con 0x7ff2a40b1fc0 2023-10-10T18:47:44.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a9e14700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a40afad0 con 0x7ff2a40b2390 2023-10-10T18:47:44.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a9e14700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4150770 con 0x7ff2a4155fb0 2023-10-10T18:47:44.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a37fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff2a40b1fc0 0x7ff2a4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45792/0 (socket says 172.21.15.175:45792) 2023-10-10T18:47:44.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a37fe700 1 -- 172.21.15.175:0/2816155260 learned_addr learned my addr 172.21.15.175:0/2816155260 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:44.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a3fff700 1 --2- 172.21.15.175:0/2816155260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a4155fb0 0x7ff2a4159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a27fc700 1 -- 172.21.15.175:0/2816155260 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266026406 0 0) 0x7ff2a41574b0 con 0x7ff2a40b1fc0 2023-10-10T18:47:44.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a27fc700 1 -- 172.21.15.175:0/2816155260 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff28c003570 con 0x7ff2a40b1fc0 2023-10-10T18:47:44.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a27fc700 1 -- 172.21.15.175:0/2816155260 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028407834 0 0) 0x7ff2a40afad0 con 0x7ff2a40b2390 2023-10-10T18:47:44.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a27fc700 1 -- 172.21.15.175:0/2816155260 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a41574b0 con 0x7ff2a40b2390 2023-10-10T18:47:44.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a3fff700 1 -- 172.21.15.175:0/2816155260 >> v1:172.21.15.191:6789/0 conn(0x7ff2a40b2390 legacy=0x7ff2a40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a3fff700 1 -- 172.21.15.175:0/2816155260 >> v1:172.21.15.175:6789/0 conn(0x7ff2a40b1fc0 legacy=0x7ff2a4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.982+0000 7ff2a3fff700 1 -- 172.21.15.175:0/2816155260 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40af4c0 con 0x7ff2a4155fb0 2023-10-10T18:47:44.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a3fff700 1 --2- 172.21.15.175:0/2816155260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a4155fb0 0x7ff2a4159920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff29c009e90 tx=0x7ff29c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af26dd738491207 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:44.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a27fc700 1 -- 172.21.15.175:0/2816155260 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c00e070 con 0x7ff2a4155fb0 2023-10-10T18:47:44.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a27fc700 1 -- 172.21.15.175:0/2816155260 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29c00bc10 con 0x7ff2a4155fb0 2023-10-10T18:47:44.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a27fc700 1 -- 172.21.15.175:0/2816155260 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c012430 con 0x7ff2a4155fb0 2023-10-10T18:47:44.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2816155260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a4155fb0 msgr2=0x7ff2a4159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a9e14700 1 --2- 172.21.15.175:0/2816155260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a4155fb0 0x7ff2a4159920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff29c009e90 tx=0x7ff29c009780 comp rx=0 tx=0).stop 2023-10-10T18:47:44.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2816155260 shutdown_connections 2023-10-10T18:47:44.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a9e14700 1 --2- 172.21.15.175:0/2816155260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a4155fb0 0x7ff2a4159920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2816155260 >> 172.21.15.175:0/2816155260 conn(0x7ff2a4063230 msgr2=0x7ff2a4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:44.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2816155260 shutdown_connections 2023-10-10T18:47:44.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.983+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2816155260 wait complete. 2023-10-10T18:47:44.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.984+0000 7ff2a9e14700 1 Processor -- start 2023-10-10T18:47:44.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a9e14700 1 -- start start 2023-10-10T18:47:44.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a9e14700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a40b1fc0 0x7ff2a40cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a9e14700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a40b2390 0x7ff2a414f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a9e14700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2a4155fb0 0x7ff2a414f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a9e14700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40cc2a0 con 0x7ff2a4155fb0 2023-10-10T18:47:44.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a9e14700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4159b20 con 0x7ff2a40b2390 2023-10-10T18:47:44.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a9e14700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4169740 con 0x7ff2a40b1fc0 2023-10-10T18:47:44.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2a4155fb0 0x7ff2a414f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a37fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a40b1fc0 0x7ff2a40cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2a4155fb0 0x7ff2a414f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54162/0 (socket says 172.21.15.175:54162) 2023-10-10T18:47:44.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a3fff700 1 -- 172.21.15.175:0/2158356483 learned_addr learned my addr 172.21.15.175:0/2158356483 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:44.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.985+0000 7ff2a2ffd700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a40b2390 0x7ff2a414f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff2a3fff700 1 -- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a40b1fc0 msgr2=0x7ff2a40cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff2a3fff700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a40b1fc0 0x7ff2a40cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff2a3fff700 1 -- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a40b2390 msgr2=0x7ff2a414f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:44.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff2a3fff700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a40b2390 0x7ff2a414f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:44.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff2a3fff700 1 -- 172.21.15.175:0/2158356483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c009a20 con 0x7ff2a4155fb0 2023-10-10T18:47:44.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff2a3fff700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2a4155fb0 0x7ff2a414f870 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00b610 tx=0x7ff29c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:44.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff28b7fe700 1 -- 172.21.15.175:0/2158356483 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c026480 con 0x7ff2a4155fb0 2023-10-10T18:47:44.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff28b7fe700 1 -- 172.21.15.175:0/2158356483 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29c026600 con 0x7ff2a4155fb0 2023-10-10T18:47:44.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff28b7fe700 1 -- 172.21.15.175:0/2158356483 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c0269c0 con 0x7ff2a4155fb0 2023-10-10T18:47:44.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a4159840 con 0x7ff2a4155fb0 2023-10-10T18:47:44.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.986+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a41693d0 con 0x7ff2a4155fb0 2023-10-10T18:47:44.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.987+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a4067570 con 0x7ff2a4155fb0 2023-10-10T18:47:44.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.987+0000 7ff28b7fe700 1 -- 172.21.15.175:0/2158356483 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff29c01c400 con 0x7ff2a4155fb0 2023-10-10T18:47:44.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.988+0000 7ff28b7fe700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff290066fb0 0x7ff290069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:44.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.988+0000 7ff28b7fe700 1 -- 172.21.15.175:0/2158356483 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7ff29c0863c0 con 0x7ff2a4155fb0 2023-10-10T18:47:44.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.988+0000 7ff2a37fe700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff290066fb0 0x7ff290069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:44.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.989+0000 7ff2a37fe700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff290066fb0 0x7ff290069460 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff294007fd0 tx=0x7ff294009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:44.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:44.992+0000 7ff28b7fe700 1 -- 172.21.15.175:0/2158356483 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff29c0534c0 con 0x7ff2a4155fb0 2023-10-10T18:47:45.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.216+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2a4169880 con 0x7ff2a4155fb0 2023-10-10T18:47:45.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.216+0000 7ff28b7fe700 1 -- 172.21.15.175:0/2158356483 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5425 (secure 0 0 0) 0x7ff29c056ae0 con 0x7ff2a4155fb0 2023-10-10T18:47:45.217 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:45.217 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:43.409578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:45.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff290066fb0 msgr2=0x7ff290069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:45.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff290066fb0 0x7ff290069460 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff294007fd0 tx=0x7ff294009040 comp rx=0 tx=0).stop 2023-10-10T18:47:45.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2a4155fb0 msgr2=0x7ff2a414f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:45.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2a4155fb0 0x7ff2a414f870 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00b610 tx=0x7ff29c019000 comp rx=0 tx=0).stop 2023-10-10T18:47:45.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 shutdown_connections 2023-10-10T18:47:45.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff290066fb0 0x7ff290069460 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:45.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2a4155fb0 0x7ff2a414f870 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:45.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2a40b1fc0 0x7ff2a40cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:45.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 --2- 172.21.15.175:0/2158356483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a40b2390 0x7ff2a414f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:45.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 >> 172.21.15.175:0/2158356483 conn(0x7ff2a4063230 msgr2=0x7ff2a4154db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:45.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 shutdown_connections 2023-10-10T18:47:45.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:45.219+0000 7ff2a9e14700 1 -- 172.21.15.175:0/2158356483 wait complete. 2023-10-10T18:47:45.222 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 23 2023-10-10T18:47:45.231 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4245": { "addr": "172.21.15.175:6834/688515950", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 688515950, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 688515950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4245, "incarnation": 23, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4248": { "addr": "172.21.15.175:6836/647425682", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 647425682, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 647425682, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4248, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4250": { "addr": "172.21.15.191:6839/246221790", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 246221790, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 246221790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4250, "incarnation": 12, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:47:43.409578+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 4250, "mds_3": 4248, "mds_4": 4245 } } } ], "standbys": [] } 2023-10-10T18:47:47.233 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:47.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.359+0000 7f06f1118700 1 Processor -- start 2023-10-10T18:47:47.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.359+0000 7f06f1118700 1 -- start start 2023-10-10T18:47:47.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.360+0000 7f06f1118700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f06ec0b2390 0x7f06ec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.360+0000 7f06f1118700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec1574b0 con 0x7f06ec0b1fc0 2023-10-10T18:47:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.360+0000 7f06f1118700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec0afad0 con 0x7f06ec0aff60 2023-10-10T18:47:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.360+0000 7f06f1118700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec150770 con 0x7f06ec0b2390 2023-10-10T18:47:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.360+0000 7f06ead9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f06ec0b1fc0 0x7f06ec068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45794/0 (socket says 172.21.15.175:45794) 2023-10-10T18:47:47.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.360+0000 7f06ead9d700 1 -- 172.21.15.175:0/1311181213 learned_addr learned my addr 172.21.15.175:0/1311181213 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.360+0000 7f06ea59c700 1 --2- 172.21.15.175:0/1311181213 >> v2:172.21.15.191:3300/0 conn(0x7f06ec0b2390 0x7f06ec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759348993 0 0) 0x7f06ec1574b0 con 0x7f06ec0b1fc0 2023-10-10T18:47:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d4003570 con 0x7f06ec0b1fc0 2023-10-10T18:47:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2423511101 0 0) 0x7f06d4003570 con 0x7f06ec0b1fc0 2023-10-10T18:47:47.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 >> v2:172.21.15.191:3300/0 conn(0x7f06ec0b2390 msgr2=0x7f06ec155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 --2- 172.21.15.175:0/1311181213 >> v2:172.21.15.191:3300/0 conn(0x7f06ec0b2390 0x7f06ec155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 >> v1:172.21.15.191:6789/0 conn(0x7f06ec0aff60 legacy=0x7f06ec159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0af4c0 con 0x7f06ec0b1fc0 2023-10-10T18:47:47.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f06dc0017f0 con 0x7f06ec0b1fc0 2023-10-10T18:47:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 >> v1:172.21.15.175:6789/0 conn(0x7f06ec0b1fc0 legacy=0x7f06ec068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06d4003a60 0x7f06d4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06d4004810 0x7f06d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:47.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06d400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec150770 con 0x7f06d40051c0 2023-10-10T18:47:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d400c0d0 con 0x7f06d4004810 2023-10-10T18:47:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d400c290 con 0x7f06d4003a60 2023-10-10T18:47:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06eb59e700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06d400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:47.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06ead9d700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06d4003a60 0x7f06d4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06ea59c700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06d4004810 0x7f06d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06eb59e700 1 -- 172.21.15.175:0/1311181213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06d4003a60 msgr2=0x7f06d4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06eb59e700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06d4003a60 0x7f06d4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06eb59e700 1 -- 172.21.15.175:0/1311181213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06d4004810 msgr2=0x7f06d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:47.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06eb59e700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06d4004810 0x7f06d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06eb59e700 1 -- 172.21.15.175:0/1311181213 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d4003570 con 0x7f06d40051c0 2023-10-10T18:47:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06eb59e700 1 -- 172.21.15.175:0/1311181213 reap_dead start 2023-10-10T18:47:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.361+0000 7f06eb59e700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06d400ba60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f06e40018b0 tx=0x7f06e4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91991cb7702855 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4012420 con 0x7f06d40051c0 2023-10-10T18:47:47.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e40125a0 con 0x7f06d40051c0 2023-10-10T18:47:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06e9d9b700 1 -- 172.21.15.175:0/1311181213 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4012960 con 0x7f06d40051c0 2023-10-10T18:47:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06f1118700 1 -- 172.21.15.175:0/1311181213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 msgr2=0x7f06d400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06f1118700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06d400ba60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f06e40018b0 tx=0x7f06e4007d30 comp rx=0 tx=0).stop 2023-10-10T18:47:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06f1118700 1 -- 172.21.15.175:0/1311181213 shutdown_connections 2023-10-10T18:47:47.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06f1118700 1 --2- 172.21.15.175:0/1311181213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06d400ba60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06f1118700 1 -- 172.21.15.175:0/1311181213 >> 172.21.15.175:0/1311181213 conn(0x7f06ec063230 msgr2=0x7f06ec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06f1118700 1 -- 172.21.15.175:0/1311181213 shutdown_connections 2023-10-10T18:47:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.362+0000 7f06f1118700 1 -- 172.21.15.175:0/1311181213 wait complete. 2023-10-10T18:47:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.363+0000 7f06f1118700 1 Processor -- start 2023-10-10T18:47:47.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.363+0000 7f06f1118700 1 -- start start 2023-10-10T18:47:47.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06f1118700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06ec156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:47.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06f1118700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06ec159a20 0x7f06ec0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06f1118700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06ec159e90 0x7f06ec0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06f1118700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0c0d50 con 0x7f06d40051c0 2023-10-10T18:47:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06f1118700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1667a0 con 0x7f06ec159e90 2023-10-10T18:47:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06f1118700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec159260 con 0x7f06ec159a20 2023-10-10T18:47:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06ead9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06ec156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:47.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06ead9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06ec156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54174/0 (socket says 172.21.15.175:54174) 2023-10-10T18:47:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06ead9d700 1 -- 172.21.15.175:0/2848561856 learned_addr learned my addr 172.21.15.175:0/2848561856 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06ea59c700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06ec159a20 0x7f06ec0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06eb59e700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06ec159e90 0x7f06ec0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06ead9d700 1 -- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06ec159a20 msgr2=0x7f06ec0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:47.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.364+0000 7f06ead9d700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06ec159a20 0x7f06ec0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06ead9d700 1 -- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06ec159e90 msgr2=0x7f06ec0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:47.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06ead9d700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06ec159e90 0x7f06ec0cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06ead9d700 1 -- 172.21.15.175:0/2848561856 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec071d40 con 0x7f06d40051c0 2023-10-10T18:47:47.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06ead9d700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06ec156b20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00a0e0 tx=0x7f06ec0af650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:47.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06d2ffd700 1 -- 172.21.15.175:0/2848561856 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc00acc0 con 0x7f06d40051c0 2023-10-10T18:47:47.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06d2ffd700 1 -- 172.21.15.175:0/2848561856 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc00ae60 con 0x7f06d40051c0 2023-10-10T18:47:47.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec15a970 con 0x7f06d40051c0 2023-10-10T18:47:47.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06d2ffd700 1 -- 172.21.15.175:0/2848561856 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc013640 con 0x7f06d40051c0 2023-10-10T18:47:47.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.365+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec159500 con 0x7f06d40051c0 2023-10-10T18:47:47.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.366+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec067570 con 0x7f06d40051c0 2023-10-10T18:47:47.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.366+0000 7f06d2ffd700 1 -- 172.21.15.175:0/2848561856 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f06dc0139b0 con 0x7f06d40051c0 2023-10-10T18:47:47.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.367+0000 7f06d2ffd700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f06d8066f30 0x7f06d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:47.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.367+0000 7f06d2ffd700 1 -- 172.21.15.175:0/2848561856 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f06dc084fc0 con 0x7f06d40051c0 2023-10-10T18:47:47.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.367+0000 7f06ea59c700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f06d8066f30 0x7f06d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:47.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.367+0000 7f06ea59c700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f06d8066f30 0x7f06d80693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0712f0 tx=0x7f06e00091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:47.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.371+0000 7f06d2ffd700 1 -- 172.21.15.175:0/2848561856 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06dc052160 con 0x7f06d40051c0 2023-10-10T18:47:47.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.598+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06ec1596f0 con 0x7f06d40051c0 2023-10-10T18:47:47.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.599+0000 7f06d2ffd700 1 -- 172.21.15.175:0/2848561856 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5425 (secure 0 0 0) 0x7f06dc018090 con 0x7f06d40051c0 2023-10-10T18:47:47.599 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:47.599 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:43.409578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:47.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f06d8066f30 msgr2=0x7f06d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:47.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f06d8066f30 0x7f06d80693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0712f0 tx=0x7f06e00091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 msgr2=0x7f06ec156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:47.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06ec156b20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00a0e0 tx=0x7f06ec0af650 comp rx=0 tx=0).stop 2023-10-10T18:47:47.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 shutdown_connections 2023-10-10T18:47:47.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f06d8066f30 0x7f06d80693e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06ec159e90 0x7f06ec0cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f06d40051c0 0x7f06ec156b20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 --2- 172.21.15.175:0/2848561856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06ec159a20 0x7f06ec0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:47.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 >> 172.21.15.175:0/2848561856 conn(0x7f06ec063230 msgr2=0x7f06ec150d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:47.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 shutdown_connections 2023-10-10T18:47:47.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:47.601+0000 7f06f1118700 1 -- 172.21.15.175:0/2848561856 wait complete. 2023-10-10T18:47:47.605 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 23 2023-10-10T18:47:47.614 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:47:48.543 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:47:48.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:47:49.615 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:49.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.736+0000 7faadb8b8700 1 Processor -- start 2023-10-10T18:47:49.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.736+0000 7faadb8b8700 1 -- start start 2023-10-10T18:47:49.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.737+0000 7faadb8b8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faad4068a40 0x7faad4063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:49.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.737+0000 7faadb8b8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faad40cac80 0x7faad406b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:49.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.737+0000 7faadb8b8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad40c4ee0 con 0x7faad4068e10 2023-10-10T18:47:49.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.737+0000 7faadb8b8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad4064180 con 0x7faad40cac80 2023-10-10T18:47:49.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.737+0000 7faadb8b8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad40c5780 con 0x7faad4068a40 2023-10-10T18:47:49.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.737+0000 7faad8e53700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faad40cac80 0x7faad406b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:49.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.737+0000 7faad8e53700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faad40cac80 0x7faad406b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54176/0 (socket says 172.21.15.175:54176) 2023-10-10T18:47:49.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.737+0000 7faad8e53700 1 -- 172.21.15.175:0/1510508745 learned_addr learned my addr 172.21.15.175:0/1510508745 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:49.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faad9654700 1 --2- 172.21.15.175:0/1510508745 >> v2:172.21.15.191:3300/0 conn(0x7faad4068a40 0x7faad4063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:49.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faad8e53700 1 -- 172.21.15.175:0/1510508745 >> v2:172.21.15.191:3300/0 conn(0x7faad4068a40 msgr2=0x7faad4063c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:49.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faad8e53700 1 --2- 172.21.15.175:0/1510508745 >> v2:172.21.15.191:3300/0 conn(0x7faad4068a40 0x7faad4063c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faad8e53700 1 -- 172.21.15.175:0/1510508745 >> v1:172.21.15.191:6789/0 conn(0x7faad4068e10 legacy=0x7faad40af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:49.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faad8e53700 1 -- 172.21.15.175:0/1510508745 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40c6710 con 0x7faad40cac80 2023-10-10T18:47:49.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faad8e53700 1 --2- 172.21.15.175:0/1510508745 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad40cac80 0x7faad406b720 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faad406b320 tx=0x7faac4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9efedc40609d92a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:49.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faacbfff700 1 -- 172.21.15.175:0/1510508745 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4005c10 con 0x7faad40cac80 2023-10-10T18:47:49.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faacbfff700 1 -- 172.21.15.175:0/1510508745 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac4005d70 con 0x7faad40cac80 2023-10-10T18:47:49.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faacbfff700 1 -- 172.21.15.175:0/1510508745 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac400b860 con 0x7faad40cac80 2023-10-10T18:47:49.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faadb8b8700 1 -- 172.21.15.175:0/1510508745 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad40cac80 msgr2=0x7faad406b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:49.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faadb8b8700 1 --2- 172.21.15.175:0/1510508745 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad40cac80 0x7faad406b720 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faad406b320 tx=0x7faac4009220 comp rx=0 tx=0).stop 2023-10-10T18:47:49.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faadb8b8700 1 -- 172.21.15.175:0/1510508745 shutdown_connections 2023-10-10T18:47:49.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faadb8b8700 1 --2- 172.21.15.175:0/1510508745 >> v2:172.21.15.191:3300/0 conn(0x7faad4068a40 0x7faad4063c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faadb8b8700 1 --2- 172.21.15.175:0/1510508745 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad40cac80 0x7faad406b720 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.738+0000 7faadb8b8700 1 -- 172.21.15.175:0/1510508745 >> 172.21.15.175:0/1510508745 conn(0x7faad406dbe0 msgr2=0x7faad40ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:49.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.739+0000 7faadb8b8700 1 -- 172.21.15.175:0/1510508745 shutdown_connections 2023-10-10T18:47:49.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.739+0000 7faadb8b8700 1 -- 172.21.15.175:0/1510508745 wait complete. 2023-10-10T18:47:49.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.740+0000 7faadb8b8700 1 Processor -- start 2023-10-10T18:47:49.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.740+0000 7faadb8b8700 1 -- start start 2023-10-10T18:47:49.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.740+0000 7faadb8b8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faad4068a40 0x7faad41676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:49.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.740+0000 7faadb8b8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad4068e10 0x7faad41699f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:49.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faadb8b8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faad40cac80 0x7faad4169f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:49.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faadb8b8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad40c2b90 con 0x7faad4068e10 2023-10-10T18:47:49.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faadb8b8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad416a580 con 0x7faad40cac80 2023-10-10T18:47:49.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faadb8b8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad40aba20 con 0x7faad4068a40 2023-10-10T18:47:49.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad4068e10 0x7faad41699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:49.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad4068e10 0x7faad41699f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54178/0 (socket says 172.21.15.175:54178) 2023-10-10T18:47:49.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 -- 172.21.15.175:0/3029043623 learned_addr learned my addr 172.21.15.175:0/3029043623 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:49.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad9e55700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faad40cac80 0x7faad4169f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:49.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad9654700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faad4068a40 0x7faad41676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:49.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 -- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faad4068a40 msgr2=0x7faad41676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:49.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faad4068a40 0x7faad41676d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 -- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faad40cac80 msgr2=0x7faad4169f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:49.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faad40cac80 0x7faad4169f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 -- 172.21.15.175:0/3029043623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac4009480 con 0x7faad4068e10 2023-10-10T18:47:49.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad9e55700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faad40cac80 0x7faad4169f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:47:49.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faad8e53700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad4068e10 0x7faad41699f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7faac40030a0 tx=0x7faac4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:49.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faac97fa700 1 -- 172.21.15.175:0/3029043623 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4006580 con 0x7faad4068e10 2023-10-10T18:47:49.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faac97fa700 1 -- 172.21.15.175:0/3029043623 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac4006700 con 0x7faad4068e10 2023-10-10T18:47:49.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad40be5d0 con 0x7faad4068e10 2023-10-10T18:47:49.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faac97fa700 1 -- 172.21.15.175:0/3029043623 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac401c420 con 0x7faad4068e10 2023-10-10T18:47:49.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.741+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad40c2540 con 0x7faad4068e10 2023-10-10T18:47:49.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.742+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad40c2d00 con 0x7faad4068e10 2023-10-10T18:47:49.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.743+0000 7faac97fa700 1 -- 172.21.15.175:0/3029043623 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faac401c810 con 0x7faad4068e10 2023-10-10T18:47:49.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.743+0000 7faac97fa700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faab8066f30 0x7faab80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:49.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.743+0000 7faac97fa700 1 -- 172.21.15.175:0/3029043623 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7faac4086490 con 0x7faad4068e10 2023-10-10T18:47:49.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.743+0000 7faad9654700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faab8066f30 0x7faab80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:49.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.744+0000 7faad9654700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faab8066f30 0x7faab80693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faad00075e0 tx=0x7faad00097f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:49.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.747+0000 7faac97fa700 1 -- 172.21.15.175:0/3029043623 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faac4053630 con 0x7faad4068e10 2023-10-10T18:47:49.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.971+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faad4067570 con 0x7faad4068e10 2023-10-10T18:47:49.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.972+0000 7faac97fa700 1 -- 172.21.15.175:0/3029043623 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5427 (secure 0 0 0) 0x7faac4056c50 con 0x7faad4068e10 2023-10-10T18:47:49.972 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:49.972 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:49.552967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:resolve","state_seq":43,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:49.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.973+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faab8066f30 msgr2=0x7faab80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:49.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.973+0000 7faadb8b8700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faab8066f30 0x7faab80693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faad00075e0 tx=0x7faad00097f0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.973+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad4068e10 msgr2=0x7faad41699f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:49.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.973+0000 7faadb8b8700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad4068e10 0x7faad41699f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7faac40030a0 tx=0x7faac4015000 comp rx=0 tx=0).stop 2023-10-10T18:47:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.974+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 shutdown_connections 2023-10-10T18:47:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.974+0000 7faadb8b8700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faab8066f30 0x7faab80693e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.974+0000 7faadb8b8700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faad40cac80 0x7faad4169f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.974+0000 7faadb8b8700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faad4068a40 0x7faad41676d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.974+0000 7faadb8b8700 1 --2- 172.21.15.175:0/3029043623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faad4068e10 0x7faad41699f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:49.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.974+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 >> 172.21.15.175:0/3029043623 conn(0x7faad406dbe0 msgr2=0x7faad4156c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:49.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.974+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 shutdown_connections 2023-10-10T18:47:49.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:49.974+0000 7faadb8b8700 1 -- 172.21.15.175:0/3029043623 wait complete. 2023-10-10T18:47:49.976 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 24 2023-10-10T18:47:49.985 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:47:51.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:52.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.113+0000 7f46c46d4700 1 Processor -- start 2023-10-10T18:47:52.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.113+0000 7f46c46d4700 1 -- start start 2023-10-10T18:47:52.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.114+0000 7f46c46d4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f46bc0c12d0 0x7f46bc0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:52.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.114+0000 7f46c46d4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc0c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:52.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.114+0000 7f46c46d4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc0c6fb0 con 0x7f46bc0c5850 2023-10-10T18:47:52.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.114+0000 7f46c46d4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46bc063ea0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.114+0000 7f46c46d4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc155f30 con 0x7f46bc0c4f20 2023-10-10T18:47:52.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c2c71700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc0c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:52.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c1c6f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f46bc0c12d0 0x7f46bc0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:52.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c2c71700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc0c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:46132/0 (socket says 172.21.15.175:46132) 2023-10-10T18:47:52.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c2c71700 1 -- 172.21.15.175:0/3512053935 learned_addr learned my addr 172.21.15.175:0/3512053935 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:52.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c1c6f700 1 -- 172.21.15.175:0/3512053935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 msgr2=0x7f46bc0c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:52.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c1c6f700 1 --2- 172.21.15.175:0/3512053935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc0c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c1c6f700 1 -- 172.21.15.175:0/3512053935 >> v1:172.21.15.191:6789/0 conn(0x7f46bc0c5850 legacy=0x7f46bc156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:52.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c1c6f700 1 -- 172.21.15.175:0/3512053935 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0c6160 con 0x7f46bc0c12d0 2023-10-10T18:47:52.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c2c71700 1 --2- 172.21.15.175:0/3512053935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc0c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:47:52.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c1c6f700 1 --2- 172.21.15.175:0/3512053935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c12d0 0x7f46bc0c5c20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0098e0 tx=0x7f46ac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40b4eaff488db3ae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:52.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c146e700 1 -- 172.21.15.175:0/3512053935 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00b640 con 0x7f46bc0c12d0 2023-10-10T18:47:52.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c146e700 1 -- 172.21.15.175:0/3512053935 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac00b7a0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.115+0000 7f46c146e700 1 -- 172.21.15.175:0/3512053935 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00bb40 con 0x7f46bc0c12d0 2023-10-10T18:47:52.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.116+0000 7f46c46d4700 1 -- 172.21.15.175:0/3512053935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c12d0 msgr2=0x7f46bc0c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:52.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.116+0000 7f46c46d4700 1 --2- 172.21.15.175:0/3512053935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c12d0 0x7f46bc0c5c20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0098e0 tx=0x7f46ac0091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.116+0000 7f46c46d4700 1 -- 172.21.15.175:0/3512053935 shutdown_connections 2023-10-10T18:47:52.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.116+0000 7f46c46d4700 1 --2- 172.21.15.175:0/3512053935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc0c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.116+0000 7f46c46d4700 1 --2- 172.21.15.175:0/3512053935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c12d0 0x7f46bc0c5c20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.116+0000 7f46c46d4700 1 -- 172.21.15.175:0/3512053935 >> 172.21.15.175:0/3512053935 conn(0x7f46bc06dc70 msgr2=0x7f46bc063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:52.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.116+0000 7f46c46d4700 1 -- 172.21.15.175:0/3512053935 shutdown_connections 2023-10-10T18:47:52.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.116+0000 7f46c46d4700 1 -- 172.21.15.175:0/3512053935 wait complete. 2023-10-10T18:47:52.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.117+0000 7f46c46d4700 1 Processor -- start 2023-10-10T18:47:52.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.117+0000 7f46c46d4700 1 -- start start 2023-10-10T18:47:52.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c46d4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46bc0c12d0 0x7f46bc167140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:52.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c46d4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc165e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:52.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c46d4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c5850 0x7f46bc169240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:52.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c46d4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0c2d00 con 0x7f46bc0c12d0 2023-10-10T18:47:52.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c46d4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc166b90 con 0x7f46bc0c5850 2023-10-10T18:47:52.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c46d4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc1668f0 con 0x7f46bc0c4f20 2023-10-10T18:47:52.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46bc0c12d0 0x7f46bc167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:52.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46bc0c12d0 0x7f46bc167140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54180/0 (socket says 172.21.15.175:54180) 2023-10-10T18:47:52.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 -- 172.21.15.175:0/2313197900 learned_addr learned my addr 172.21.15.175:0/2313197900 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:52.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2c71700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c5850 0x7f46bc169240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:52.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c1c6f700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:52.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 -- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 msgr2=0x7f46bc165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:52.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 -- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c5850 msgr2=0x7f46bc169240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:52.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c5850 0x7f46bc169240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 -- 172.21.15.175:0/2313197900 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b40094f0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46c2470700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46bc0c12d0 0x7f46bc167140 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f46b800cb10 tx=0x7f46b80017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:52.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.118+0000 7f46b27fc700 1 -- 172.21.15.175:0/2313197900 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b800ce20 con 0x7f46bc0c12d0 2023-10-10T18:47:52.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.119+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ac009420 con 0x7f46bc0c12d0 2023-10-10T18:47:52.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.119+0000 7f46b27fc700 1 -- 172.21.15.175:0/2313197900 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b800a7e0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.119+0000 7f46b27fc700 1 -- 172.21.15.175:0/2313197900 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b800aba0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.119+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc0c26b0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.119+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46bc067570 con 0x7f46bc0c12d0 2023-10-10T18:47:52.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.120+0000 7f46b27fc700 1 -- 172.21.15.175:0/2313197900 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f46b8003030 con 0x7f46bc0c12d0 2023-10-10T18:47:52.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.120+0000 7f46b27fc700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f46a0066f60 0x7f46a0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:52.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.120+0000 7f46b27fc700 1 -- 172.21.15.175:0/2313197900 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f46b8084620 con 0x7f46bc0c12d0 2023-10-10T18:47:52.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.121+0000 7f46c1c6f700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f46a0066f60 0x7f46a0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:52.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.121+0000 7f46c1c6f700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f46a0066f60 0x7f46a0069410 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0093f0 tx=0x7f46ac017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:52.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.125+0000 7f46b27fc700 1 -- 172.21.15.175:0/2313197900 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46b80517c0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.354+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46bc0c6bc0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.355+0000 7f46b27fc700 1 -- 172.21.15.175:0/2313197900 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5427 (secure 0 0 0) 0x7f46b8054de0 con 0x7f46bc0c12d0 2023-10-10T18:47:52.355 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:52.355 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:49.552967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:resolve","state_seq":43,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:52.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f46a0066f60 msgr2=0x7f46a0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:52.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f46a0066f60 0x7f46a0069410 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0093f0 tx=0x7f46ac017000 comp rx=0 tx=0).stop 2023-10-10T18:47:52.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46bc0c12d0 msgr2=0x7f46bc167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:52.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46bc0c12d0 0x7f46bc167140 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f46b800cb10 tx=0x7f46b80017b0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 shutdown_connections 2023-10-10T18:47:52.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f46a0066f60 0x7f46a0069410 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46bc0c5850 0x7f46bc169240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46bc0c12d0 0x7f46bc167140 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 --2- 172.21.15.175:0/2313197900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46bc0c4f20 0x7f46bc165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:52.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 >> 172.21.15.175:0/2313197900 conn(0x7f46bc06dc70 msgr2=0x7f46bc154a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:52.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 shutdown_connections 2023-10-10T18:47:52.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:52.357+0000 7f46c46d4700 1 -- 172.21.15.175:0/2313197900 wait complete. 2023-10-10T18:47:52.360 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 24 2023-10-10T18:47:52.368 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:47:54.370 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:54.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.498+0000 7faec37b4700 1 Processor -- start 2023-10-10T18:47:54.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.498+0000 7faec37b4700 1 -- start start 2023-10-10T18:47:54.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faec37b4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc1573d0 0x7faebc0c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:54.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faec37b4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faebc154aa0 con 0x7faebc0c4f20 2023-10-10T18:47:54.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faec37b4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faebc063dc0 con 0x7faebc1577a0 2023-10-10T18:47:54.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faec37b4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0bfdf0 con 0x7faebc1573d0 2023-10-10T18:47:54.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faec1d51700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7faebc0c4f20 0x7faebc14d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45828/0 (socket says 172.21.15.175:45828) 2023-10-10T18:47:54.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faec1d51700 1 -- 172.21.15.175:0/2394136659 learned_addr learned my addr 172.21.15.175:0/2394136659 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:54.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faec1550700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc1573d0 0x7faebc0c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:54.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539052227 0 0) 0x7faebc154aa0 con 0x7faebc0c4f20 2023-10-10T18:47:54.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea8003570 con 0x7faebc0c4f20 2023-10-10T18:47:54.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421371042 0 0) 0x7faea8003570 con 0x7faebc0c4f20 2023-10-10T18:47:54.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc1573d0 msgr2=0x7faebc0c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:54.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc1573d0 0x7faebc0c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 >> v1:172.21.15.191:6789/0 conn(0x7faebc1577a0 legacy=0x7faebc0bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:54.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc0c5380 con 0x7faebc0c4f20 2023-10-10T18:47:54.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7faeb40017f0 con 0x7faebc0c4f20 2023-10-10T18:47:54.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.499+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 >> v1:172.21.15.175:6789/0 conn(0x7faebc0c4f20 legacy=0x7faebc14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:54.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc1573d0 0x7faea8003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:54.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faea8003fd0 0x7faea8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:54.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faea80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:54.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea800bb10 con 0x7faea8004d50 2023-10-10T18:47:54.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea800bc80 con 0x7faea8003fd0 2023-10-10T18:47:54.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea800be40 con 0x7faebc1573d0 2023-10-10T18:47:54.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec1d51700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faea80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:54.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec1550700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc1573d0 0x7faea8003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:54.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec0d4f700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faea8003fd0 0x7faea8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:54.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec1d51700 1 -- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc1573d0 msgr2=0x7faea8003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:54.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec1d51700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc1573d0 0x7faea8003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec1d51700 1 -- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faea8003fd0 msgr2=0x7faea8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:54.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec1d51700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faea8003fd0 0x7faea8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec1d51700 1 -- 172.21.15.175:0/2394136659 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea8003570 con 0x7faea8004d50 2023-10-10T18:47:54.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faec1d51700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faea80051c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faeb4014f60 tx=0x7faebc06f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3c2eb7bf501604 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:54.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb400abd0 con 0x7faea8004d50 2023-10-10T18:47:54.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb400ad30 con 0x7faea8004d50 2023-10-10T18:47:54.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.500+0000 7faeb3fff700 1 -- 172.21.15.175:0/2394136659 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb400aea0 con 0x7faea8004d50 2023-10-10T18:47:54.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.501+0000 7faec37b4700 1 -- 172.21.15.175:0/2394136659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 msgr2=0x7faea80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:54.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.501+0000 7faec37b4700 1 --2- 172.21.15.175:0/2394136659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faea80051c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faeb4014f60 tx=0x7faebc06f1f0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.501+0000 7faec1d51700 1 -- 172.21.15.175:0/2394136659 reap_dead start 2023-10-10T18:47:54.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.501+0000 7faec37b4700 1 -- 172.21.15.175:0/2394136659 shutdown_connections 2023-10-10T18:47:54.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.501+0000 7faec37b4700 1 -- 172.21.15.175:0/2394136659 >> 172.21.15.175:0/2394136659 conn(0x7faebc06dc70 msgr2=0x7faebc063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:54.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.501+0000 7faec37b4700 1 -- 172.21.15.175:0/2394136659 shutdown_connections 2023-10-10T18:47:54.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.501+0000 7faec37b4700 1 -- 172.21.15.175:0/2394136659 wait complete. 2023-10-10T18:47:54.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.502+0000 7faec37b4700 1 Processor -- start 2023-10-10T18:47:54.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.502+0000 7faec37b4700 1 -- start start 2023-10-10T18:47:54.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec37b4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faebc14c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:54.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec37b4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faebc0ca2c0 0x7faebc144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:54.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec37b4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc143e30 0x7faebc0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:54.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec37b4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc14b740 con 0x7faea8004d50 2023-10-10T18:47:54.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec37b4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0cad90 con 0x7faebc0ca2c0 2023-10-10T18:47:54.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec37b4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0cb070 con 0x7faebc143e30 2023-10-10T18:47:54.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faebc14c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:54.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faebc14c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54190/0 (socket says 172.21.15.175:54190) 2023-10-10T18:47:54.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 -- 172.21.15.175:0/1101552266 learned_addr learned my addr 172.21.15.175:0/1101552266 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:54.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1d51700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc143e30 0x7faebc0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:54.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec0d4f700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faebc0ca2c0 0x7faebc144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:54.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 -- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc143e30 msgr2=0x7faebc0c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:54.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc143e30 0x7faebc0c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 -- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faebc0ca2c0 msgr2=0x7faebc144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:54.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faebc0ca2c0 0x7faebc144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 -- 172.21.15.175:0/1101552266 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc14b0f0 con 0x7faea8004d50 2023-10-10T18:47:54.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.503+0000 7faec1550700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faebc14c9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faeb8006ac0 tx=0x7faeb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:54.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.504+0000 7faeb17fa700 1 -- 172.21.15.175:0/1101552266 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb800f940 con 0x7faea8004d50 2023-10-10T18:47:54.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.504+0000 7faeb17fa700 1 -- 172.21.15.175:0/1101552266 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb800faa0 con 0x7faea8004d50 2023-10-10T18:47:54.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.504+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faebc14b3d0 con 0x7faea8004d50 2023-10-10T18:47:54.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.504+0000 7faeb17fa700 1 -- 172.21.15.175:0/1101552266 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb800fe20 con 0x7faea8004d50 2023-10-10T18:47:54.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.504+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc14cee0 con 0x7faea8004d50 2023-10-10T18:47:54.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.504+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faebc14d190 con 0x7faea8004d50 2023-10-10T18:47:54.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.505+0000 7faeb17fa700 1 -- 172.21.15.175:0/1101552266 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faeb8014650 con 0x7faea8004d50 2023-10-10T18:47:54.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.505+0000 7faeb17fa700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faea0066f60 0x7faea0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:54.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.506+0000 7faeb17fa700 1 -- 172.21.15.175:0/1101552266 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7faeb8018030 con 0x7faea8004d50 2023-10-10T18:47:54.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.506+0000 7faec0d4f700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faea0066f60 0x7faea0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:54.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.506+0000 7faec0d4f700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faea0066f60 0x7faea0069410 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faebc0c42b0 tx=0x7faeac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:54.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.510+0000 7faeb17fa700 1 -- 172.21.15.175:0/1101552266 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faeb8089070 con 0x7faea8004d50 2023-10-10T18:47:54.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.742+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faebc067570 con 0x7faea8004d50 2023-10-10T18:47:54.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.742+0000 7faeb17fa700 1 -- 172.21.15.175:0/1101552266 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5427 (secure 0 0 0) 0x7faeb8022090 con 0x7faea8004d50 2023-10-10T18:47:54.742 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:54.743 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:49.552967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:resolve","state_seq":43,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:54.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faea0066f60 msgr2=0x7faea0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:54.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faea0066f60 0x7faea0069410 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faebc0c42b0 tx=0x7faeac009040 comp rx=0 tx=0).stop 2023-10-10T18:47:54.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 msgr2=0x7faebc14c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:54.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faebc14c9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faeb8006ac0 tx=0x7faeb80091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 shutdown_connections 2023-10-10T18:47:54.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faea0066f60 0x7faea0069410 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faebc143e30 0x7faebc0c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faea8004d50 0x7faebc14c9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 --2- 172.21.15.175:0/1101552266 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faebc0ca2c0 0x7faebc144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:54.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 >> 172.21.15.175:0/1101552266 conn(0x7faebc06dc70 msgr2=0x7faebc0c78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:54.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 shutdown_connections 2023-10-10T18:47:54.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:54.744+0000 7faec37b4700 1 -- 172.21.15.175:0/1101552266 wait complete. 2023-10-10T18:47:54.747 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 24 2023-10-10T18:47:54.756 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:47:56.757 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:56.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.885+0000 7fca0e367700 1 Processor -- start 2023-10-10T18:47:56.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.885+0000 7fca0e367700 1 -- start start 2023-10-10T18:47:56.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca0e367700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca08068c00 0x7fca08159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:56.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca0e367700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca08067fa0 con 0x7fca0808a100 2023-10-10T18:47:56.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca0e367700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca08156b70 con 0x7fca0814fd90 2023-10-10T18:47:56.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca0e367700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca080c5600 con 0x7fca08068c00 2023-10-10T18:47:56.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca0cb64700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fca0808a100 0x7fca080684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53288/0 (socket says 172.21.15.175:53288) 2023-10-10T18:47:56.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca0cb64700 1 -- 172.21.15.175:0/746980554 learned_addr learned my addr 172.21.15.175:0/746980554 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:56.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca0db66700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca08068c00 0x7fca08159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:56.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937847773 0 0) 0x7fca08067fa0 con 0x7fca0808a100 2023-10-10T18:47:56.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.886+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f0003570 con 0x7fca0808a100 2023-10-10T18:47:56.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1834455200 0 0) 0x7fc9f0003570 con 0x7fca0808a100 2023-10-10T18:47:56.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca08068c00 msgr2=0x7fca08159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:56.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca08068c00 0x7fca08159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:56.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 >> v1:172.21.15.191:6789/0 conn(0x7fca0814fd90 legacy=0x7fca080c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:56.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08150160 con 0x7fca0808a100 2023-10-10T18:47:56.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fc9fc001470 con 0x7fca0808a100 2023-10-10T18:47:56.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 >> v1:172.21.15.175:6789/0 conn(0x7fca0808a100 legacy=0x7fca080684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:56.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9f0003a90 0x7fc9f0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:56.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0004810 0x7fc9f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:56.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca08068c00 0x7fc9f00051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:56.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca080c5600 con 0x7fc9f0003a90 2023-10-10T18:47:56.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f000bb60 con 0x7fc9f0004810 2023-10-10T18:47:56.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f000bd20 con 0x7fca08068c00 2023-10-10T18:47:56.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0d365700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9f0003a90 0x7fc9f0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:56.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0cb64700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0004810 0x7fc9f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:56.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0db66700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca08068c00 0x7fc9f00051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:56.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0d365700 1 -- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca08068c00 msgr2=0x7fc9f00051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:56.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0d365700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca08068c00 0x7fc9f00051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:56.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0d365700 1 -- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0004810 msgr2=0x7fc9f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:47:56.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0d365700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0004810 0x7fc9f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:56.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0d365700 1 -- 172.21.15.175:0/746980554 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f0003570 con 0x7fc9f0003a90 2023-10-10T18:47:56.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca0d365700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9f0003a90 0x7fc9f0003ea0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fca00001ff0 tx=0x7fca00001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16d9f28e224c00a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:56.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca00001980 con 0x7fc9f0003a90 2023-10-10T18:47:56.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0000c8b0 con 0x7fc9f0003a90 2023-10-10T18:47:56.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.887+0000 7fca07fff700 1 -- 172.21.15.175:0/746980554 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0000cc50 con 0x7fc9f0003a90 2023-10-10T18:47:56.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.888+0000 7fca0e367700 1 -- 172.21.15.175:0/746980554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9f0003a90 msgr2=0x7fc9f0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:56.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.888+0000 7fca0e367700 1 --2- 172.21.15.175:0/746980554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9f0003a90 0x7fc9f0003ea0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fca00001ff0 tx=0x7fca00001670 comp rx=0 tx=0).stop 2023-10-10T18:47:56.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.888+0000 7fca0db66700 1 -- 172.21.15.175:0/746980554 reap_dead start 2023-10-10T18:47:56.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.888+0000 7fca0e367700 1 -- 172.21.15.175:0/746980554 shutdown_connections 2023-10-10T18:47:56.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.888+0000 7fca0e367700 1 -- 172.21.15.175:0/746980554 >> 172.21.15.175:0/746980554 conn(0x7fca08063d00 msgr2=0x7fca080cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:56.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.888+0000 7fca0e367700 1 -- 172.21.15.175:0/746980554 shutdown_connections 2023-10-10T18:47:56.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.888+0000 7fca0e367700 1 -- 172.21.15.175:0/746980554 wait complete. 2023-10-10T18:47:56.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.889+0000 7fca0e367700 1 Processor -- start 2023-10-10T18:47:56.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.889+0000 7fca0e367700 1 -- start start 2023-10-10T18:47:56.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0e367700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0003a90 0x7fca08169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:56.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0e367700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca081669d0 0x7fca08152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:56.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0e367700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca080b1fd0 0x7fca08152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:56.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0e367700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0816a1e0 con 0x7fca080b1fd0 2023-10-10T18:47:56.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0e367700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca08165ce0 con 0x7fc9f0003a90 2023-10-10T18:47:56.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0e367700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca08166540 con 0x7fca081669d0 2023-10-10T18:47:56.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0db66700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca080b1fd0 0x7fca08152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:56.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0d365700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0003a90 0x7fca08169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:56.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0db66700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca080b1fd0 0x7fca08152f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44112/0 (socket says 172.21.15.175:44112) 2023-10-10T18:47:56.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0db66700 1 -- 172.21.15.175:0/2911397759 learned_addr learned my addr 172.21.15.175:0/2911397759 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:56.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.890+0000 7fca0cb64700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca081669d0 0x7fca08152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:56.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0db66700 1 -- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca081669d0 msgr2=0x7fca08152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:56.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0db66700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca081669d0 0x7fca08152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:56.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0db66700 1 -- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0003a90 msgr2=0x7fca08169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:56.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0db66700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0003a90 0x7fca08169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:56.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0db66700 1 -- 172.21.15.175:0/2911397759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08150550 con 0x7fca080b1fd0 2023-10-10T18:47:56.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0d365700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0003a90 0x7fca08169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:47:56.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0db66700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca080b1fd0 0x7fca08152f90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8006f00 tx=0x7fc9f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:56.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca057fa700 1 -- 172.21.15.175:0/2911397759 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f800f7d0 con 0x7fca080b1fd0 2023-10-10T18:47:56.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca057fa700 1 -- 172.21.15.175:0/2911397759 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f800f930 con 0x7fca080b1fd0 2023-10-10T18:47:56.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca08150830 con 0x7fca080b1fd0 2023-10-10T18:47:56.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca057fa700 1 -- 172.21.15.175:0/2911397759 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f800fcd0 con 0x7fca080b1fd0 2023-10-10T18:47:56.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.891+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca081661d0 con 0x7fca080b1fd0 2023-10-10T18:47:56.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.892+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca081666f0 con 0x7fca080b1fd0 2023-10-10T18:47:56.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.892+0000 7fca057fa700 1 -- 172.21.15.175:0/2911397759 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc9f8014650 con 0x7fca080b1fd0 2023-10-10T18:47:56.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.893+0000 7fca057fa700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9f4066f30 0x7fc9f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:56.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.893+0000 7fca0d365700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9f4066f30 0x7fc9f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:56.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.893+0000 7fca057fa700 1 -- 172.21.15.175:0/2911397759 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fc9f8084bb0 con 0x7fca080b1fd0 2023-10-10T18:47:56.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.893+0000 7fca0d365700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9f4066f30 0x7fc9f40693e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fca00004290 tx=0x7fca00015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:56.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:56.897+0000 7fca057fa700 1 -- 172.21.15.175:0/2911397759 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca081666f0 con 0x7fca080b1fd0 2023-10-10T18:47:57.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.121+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca081666f0 con 0x7fca080b1fd0 2023-10-10T18:47:57.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.122+0000 7fca057fa700 1 -- 172.21.15.175:0/2911397759 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5429 (secure 0 0 0) 0x7fca081666f0 con 0x7fca080b1fd0 2023-10-10T18:47:57.122 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:57.122 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:55.567925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:reconnect","state_seq":45,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:57.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9f4066f30 msgr2=0x7fc9f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:57.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9f4066f30 0x7fc9f40693e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fca00004290 tx=0x7fca00015000 comp rx=0 tx=0).stop 2023-10-10T18:47:57.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca080b1fd0 msgr2=0x7fca08152f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:57.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca080b1fd0 0x7fca08152f90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8006f00 tx=0x7fc9f80091a0 comp rx=0 tx=0).stop 2023-10-10T18:47:57.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 shutdown_connections 2023-10-10T18:47:57.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9f4066f30 0x7fc9f40693e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:57.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca080b1fd0 0x7fca08152f90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:57.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9f0003a90 0x7fca08169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:57.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 --2- 172.21.15.175:0/2911397759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca081669d0 0x7fca08152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:57.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 >> 172.21.15.175:0/2911397759 conn(0x7fca08063d00 msgr2=0x7fca0814dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:57.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 shutdown_connections 2023-10-10T18:47:57.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:57.124+0000 7fca0e367700 1 -- 172.21.15.175:0/2911397759 wait complete. 2023-10-10T18:47:57.126 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 25 2023-10-10T18:47:57.136 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4245": { "addr": "172.21.15.175:6834/688515950", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 688515950, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 688515950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4245, "incarnation": 23, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:reconnect", "state_seq": 45 }, "gid_4248": { "addr": "172.21.15.175:6836/647425682", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 647425682, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 647425682, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4248, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4250": { "addr": "172.21.15.191:6839/246221790", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 246221790, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 246221790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4250, "incarnation": 12, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:47:55.567925+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 4250, "mds_3": 4248, "mds_4": 4245 } } } ], "standbys": [] } 2023-10-10T18:47:59.137 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:47:59.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.262+0000 7f31d118f700 1 Processor -- start 2023-10-10T18:47:59.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.262+0000 7f31d118f700 1 -- start start 2023-10-10T18:47:59.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.262+0000 7f31d118f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f31cc0c4f20 0x7f31cc0c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:59.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.262+0000 7f31d118f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc0c6fb0 con 0x7f31cc0c12d0 2023-10-10T18:47:59.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.262+0000 7f31d118f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc063940 con 0x7f31cc0c5850 2023-10-10T18:47:59.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.262+0000 7f31d118f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc063ea0 con 0x7f31cc0c4f20 2023-10-10T18:47:59.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31c259c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f31cc0c12d0 0x7f31cc0c7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53292/0 (socket says 172.21.15.175:53292) 2023-10-10T18:47:59.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31c259c700 1 -- 172.21.15.175:0/3965784735 learned_addr learned my addr 172.21.15.175:0/3965784735 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:47:59.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31cb59e700 1 --2- 172.21.15.175:0/3965784735 >> v2:172.21.15.191:3300/0 conn(0x7f31cc0c4f20 0x7f31cc0c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:59.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31ca59c700 1 -- 172.21.15.175:0/3965784735 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800192455 0 0) 0x7f31cc0c6fb0 con 0x7f31cc0c12d0 2023-10-10T18:47:59.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31ca59c700 1 -- 172.21.15.175:0/3965784735 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b8003570 con 0x7f31cc0c12d0 2023-10-10T18:47:59.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31ca59c700 1 -- 172.21.15.175:0/3965784735 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1184309156 0 0) 0x7f31b8003570 con 0x7f31cc0c12d0 2023-10-10T18:47:59.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31cb59e700 1 -- 172.21.15.175:0/3965784735 >> v1:172.21.15.191:6789/0 conn(0x7f31cc0c5850 legacy=0x7f31cc156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:59.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31cb59e700 1 -- 172.21.15.175:0/3965784735 >> v1:172.21.15.175:6789/0 conn(0x7f31cc0c12d0 legacy=0x7f31cc0c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:59.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31cb59e700 1 -- 172.21.15.175:0/3965784735 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc063bc0 con 0x7f31cc0c4f20 2023-10-10T18:47:59.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31cb59e700 1 --2- 172.21.15.175:0/3965784735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c4f20 0x7f31cc0c7a80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009e60 tx=0x7f31c4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b417fc1c5514cd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:59.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31ca59c700 1 -- 172.21.15.175:0/3965784735 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c400e070 con 0x7f31cc0c4f20 2023-10-10T18:47:59.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.263+0000 7f31ca59c700 1 -- 172.21.15.175:0/3965784735 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c400b8e0 con 0x7f31cc0c4f20 2023-10-10T18:47:59.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.264+0000 7f31ca59c700 1 -- 172.21.15.175:0/3965784735 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b4001470 con 0x7f31cc0c4f20 2023-10-10T18:47:59.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.264+0000 7f31d118f700 1 -- 172.21.15.175:0/3965784735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c4f20 msgr2=0x7f31cc0c7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:59.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.264+0000 7f31d118f700 1 --2- 172.21.15.175:0/3965784735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c4f20 0x7f31cc0c7a80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009e60 tx=0x7f31c4009780 comp rx=0 tx=0).stop 2023-10-10T18:47:59.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.264+0000 7f31d118f700 1 -- 172.21.15.175:0/3965784735 shutdown_connections 2023-10-10T18:47:59.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.264+0000 7f31d118f700 1 --2- 172.21.15.175:0/3965784735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c4f20 0x7f31cc0c7a80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:59.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.264+0000 7f31d118f700 1 -- 172.21.15.175:0/3965784735 >> 172.21.15.175:0/3965784735 conn(0x7f31cc06dc70 msgr2=0x7f31cc063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:59.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.264+0000 7f31d118f700 1 -- 172.21.15.175:0/3965784735 shutdown_connections 2023-10-10T18:47:59.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.264+0000 7f31d118f700 1 -- 172.21.15.175:0/3965784735 wait complete. 2023-10-10T18:47:59.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.265+0000 7f31d118f700 1 Processor -- start 2023-10-10T18:47:59.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.265+0000 7f31d118f700 1 -- start start 2023-10-10T18:47:59.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31d118f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c12d0 0x7f31cc0c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:59.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31d118f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31cc0c4f20 0x7f31cc1512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:59.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31d118f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31cc0c5850 0x7f31cc14d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:59.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31d118f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc168fa0 con 0x7f31cc0c5850 2023-10-10T18:47:59.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31d118f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc157020 con 0x7f31cc0c12d0 2023-10-10T18:47:59.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31d118f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0c9e50 con 0x7f31cc0c4f20 2023-10-10T18:47:59.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cb59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31cc0c5850 0x7f31cc14d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:59.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cb59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31cc0c5850 0x7f31cc14d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44128/0 (socket says 172.21.15.175:44128) 2023-10-10T18:47:59.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cb59e700 1 -- 172.21.15.175:0/288108385 learned_addr learned my addr 172.21.15.175:0/288108385 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:47:59.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cad9d700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c12d0 0x7f31cc0c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:59.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31c259c700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31cc0c4f20 0x7f31cc1512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:59.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cb59e700 1 -- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31cc0c4f20 msgr2=0x7f31cc1512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:59.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cb59e700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31cc0c4f20 0x7f31cc1512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:59.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cb59e700 1 -- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c12d0 msgr2=0x7f31cc0c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:59.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cb59e700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c12d0 0x7f31cc0c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:59.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.266+0000 7f31cb59e700 1 -- 172.21.15.175:0/288108385 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c4009a20 con 0x7f31cc0c5850 2023-10-10T18:47:59.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.267+0000 7f31cb59e700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31cc0c5850 0x7f31cc14d5b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f31cc06e6e0 tx=0x7f31c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:59.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.267+0000 7f31c37fe700 1 -- 172.21.15.175:0/288108385 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4012c80 con 0x7f31cc0c5850 2023-10-10T18:47:59.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.267+0000 7f31c37fe700 1 -- 172.21.15.175:0/288108385 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c4012de0 con 0x7f31cc0c5850 2023-10-10T18:47:59.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.267+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc06b6e0 con 0x7f31cc0c5850 2023-10-10T18:47:59.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.267+0000 7f31c37fe700 1 -- 172.21.15.175:0/288108385 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c401e6a0 con 0x7f31cc0c5850 2023-10-10T18:47:59.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.267+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc168cc0 con 0x7f31cc0c5850 2023-10-10T18:47:59.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.268+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc067570 con 0x7f31cc0c5850 2023-10-10T18:47:59.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.268+0000 7f31c37fe700 1 -- 172.21.15.175:0/288108385 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f31c401ea10 con 0x7f31cc0c5850 2023-10-10T18:47:59.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.268+0000 7f31c37fe700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31ac066f30 0x7f31ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:47:59.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.268+0000 7f31c37fe700 1 -- 172.21.15.175:0/288108385 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f31c4086750 con 0x7f31cc0c5850 2023-10-10T18:47:59.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.269+0000 7f31cad9d700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31ac066f30 0x7f31ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:47:59.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.269+0000 7f31cad9d700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31ac066f30 0x7f31ac0693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f31bc003a80 tx=0x7f31bc007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:47:59.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.273+0000 7f31c37fe700 1 -- 172.21.15.175:0/288108385 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31c40538f0 con 0x7f31cc0c5850 2023-10-10T18:47:59.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.500+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31cc0c9f90 con 0x7f31cc0c5850 2023-10-10T18:47:59.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.501+0000 7f31c37fe700 1 -- 172.21.15.175:0/288108385 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5429 (secure 0 0 0) 0x7f31c4056f10 con 0x7f31cc0c5850 2023-10-10T18:47:59.501 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:47:59.501 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:47:55.567925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:reconnect","state_seq":45,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:47:59.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31ac066f30 msgr2=0x7f31ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:59.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31ac066f30 0x7f31ac0693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f31bc003a80 tx=0x7f31bc007ec0 comp rx=0 tx=0).stop 2023-10-10T18:47:59.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31cc0c5850 msgr2=0x7f31cc14d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:47:59.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31cc0c5850 0x7f31cc14d5b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f31cc06e6e0 tx=0x7f31c4017000 comp rx=0 tx=0).stop 2023-10-10T18:47:59.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 shutdown_connections 2023-10-10T18:47:59.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31ac066f30 0x7f31ac0693e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:59.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31cc0c5850 0x7f31cc14d5b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:59.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31cc0c12d0 0x7f31cc0c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:59.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 --2- 172.21.15.175:0/288108385 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31cc0c4f20 0x7f31cc1512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:47:59.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 >> 172.21.15.175:0/288108385 conn(0x7f31cc06dc70 msgr2=0x7f31cc0c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:47:59.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 shutdown_connections 2023-10-10T18:47:59.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:47:59.503+0000 7f31d118f700 1 -- 172.21.15.175:0/288108385 wait complete. 2023-10-10T18:47:59.506 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 25 2023-10-10T18:47:59.515 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:48:01.516 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:48:01.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.642+0000 7f8413fc1700 1 Processor -- start 2023-10-10T18:48:01.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.643+0000 7f8413fc1700 1 -- start start 2023-10-10T18:48:01.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.643+0000 7f8413fc1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c159d70 0x7f840c0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:01.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.643+0000 7f8413fc1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c156800 con 0x7f840c14ffc0 2023-10-10T18:48:01.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.643+0000 7f8413fc1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c063e30 con 0x7f840c14fbf0 2023-10-10T18:48:01.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.643+0000 7f8413fc1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0c59e0 con 0x7f840c159d70 2023-10-10T18:48:01.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f841155c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f840c14ffc0 0x7f840c0681e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53296/0 (socket says 172.21.15.175:53296) 2023-10-10T18:48:01.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f841155c700 1 -- 172.21.15.175:0/3773777327 learned_addr learned my addr 172.21.15.175:0/3773777327 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:48:01.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f841255e700 1 --2- 172.21.15.175:0/3773777327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c159d70 0x7f840c0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:01.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f8410d5b700 1 -- 172.21.15.175:0/3773777327 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516397333 0 0) 0x7f840c156800 con 0x7f840c14ffc0 2023-10-10T18:48:01.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f8410d5b700 1 -- 172.21.15.175:0/3773777327 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f8003570 con 0x7f840c14ffc0 2023-10-10T18:48:01.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f8410d5b700 1 -- 172.21.15.175:0/3773777327 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508717969 0 0) 0x7f840c063e30 con 0x7f840c14fbf0 2023-10-10T18:48:01.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f8410d5b700 1 -- 172.21.15.175:0/3773777327 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c156800 con 0x7f840c14fbf0 2023-10-10T18:48:01.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f8410d5b700 1 -- 172.21.15.175:0/3773777327 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 668004428 0 0) 0x7f83f8003570 con 0x7f840c14ffc0 2023-10-10T18:48:01.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f841255e700 1 -- 172.21.15.175:0/3773777327 >> v1:172.21.15.191:6789/0 conn(0x7f840c14fbf0 legacy=0x7f840c063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:01.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f841255e700 1 -- 172.21.15.175:0/3773777327 >> v1:172.21.15.175:6789/0 conn(0x7f840c14ffc0 legacy=0x7f840c0681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:01.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f841255e700 1 -- 172.21.15.175:0/3773777327 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c156c90 con 0x7f840c159d70 2023-10-10T18:48:01.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f841255e700 1 --2- 172.21.15.175:0/3773777327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c159d70 0x7f840c0c5fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8404009e90 tx=0x7f8404009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6137e5d934c827b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:01.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.644+0000 7f8410d5b700 1 -- 172.21.15.175:0/3773777327 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840400e070 con 0x7f840c159d70 2023-10-10T18:48:01.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8410d5b700 1 -- 172.21.15.175:0/3773777327 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840400b870 con 0x7f840c159d70 2023-10-10T18:48:01.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8410d5b700 1 -- 172.21.15.175:0/3773777327 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc001470 con 0x7f840c159d70 2023-10-10T18:48:01.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8413fc1700 1 -- 172.21.15.175:0/3773777327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c159d70 msgr2=0x7f840c0c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:01.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8413fc1700 1 --2- 172.21.15.175:0/3773777327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c159d70 0x7f840c0c5fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8404009e90 tx=0x7f8404009780 comp rx=0 tx=0).stop 2023-10-10T18:48:01.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8413fc1700 1 -- 172.21.15.175:0/3773777327 shutdown_connections 2023-10-10T18:48:01.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8413fc1700 1 --2- 172.21.15.175:0/3773777327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c159d70 0x7f840c0c5fb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8413fc1700 1 -- 172.21.15.175:0/3773777327 >> 172.21.15.175:0/3773777327 conn(0x7f840c0cc760 msgr2=0x7f840c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:01.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8413fc1700 1 -- 172.21.15.175:0/3773777327 shutdown_connections 2023-10-10T18:48:01.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.645+0000 7f8413fc1700 1 -- 172.21.15.175:0/3773777327 wait complete. 2023-10-10T18:48:01.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.646+0000 7f8413fc1700 1 Processor -- start 2023-10-10T18:48:01.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.646+0000 7f8413fc1700 1 -- start start 2023-10-10T18:48:01.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8413fc1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f840c14fbf0 0x7f840c0cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:01.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8413fc1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c14ffc0 0x7f840c1690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:01.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8413fc1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f840c159d70 0x7f840c14c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:01.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8413fc1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1546d0 con 0x7f840c14fbf0 2023-10-10T18:48:01.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8413fc1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1664e0 con 0x7f840c159d70 2023-10-10T18:48:01.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8413fc1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840c155550 con 0x7f840c14ffc0 2023-10-10T18:48:01.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f840c14fbf0 0x7f840c0cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:01.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f840c14fbf0 0x7f840c0cbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44134/0 (socket says 172.21.15.175:44134) 2023-10-10T18:48:01.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 -- 172.21.15.175:0/2084049373 learned_addr learned my addr 172.21.15.175:0/2084049373 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:01.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f841255e700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f840c159d70 0x7f840c14c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:01.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f841155c700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c14ffc0 0x7f840c1690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:01.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 -- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c14ffc0 msgr2=0x7f840c1690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:01.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c14ffc0 0x7f840c1690c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 -- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f840c159d70 msgr2=0x7f840c14c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:01.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f840c159d70 0x7f840c14c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 -- 172.21.15.175:0/2084049373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8404009a20 con 0x7f840c14fbf0 2023-10-10T18:48:01.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.647+0000 7f8411d5d700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f840c14fbf0 0x7f840c0cbb50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f840800c7c0 tx=0x7f8408007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:01.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.648+0000 7f8401ffb700 1 -- 172.21.15.175:0/2084049373 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408001570 con 0x7f840c14fbf0 2023-10-10T18:48:01.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.648+0000 7f8401ffb700 1 -- 172.21.15.175:0/2084049373 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84080016f0 con 0x7f840c14fbf0 2023-10-10T18:48:01.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.648+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840c1557f0 con 0x7f840c14fbf0 2023-10-10T18:48:01.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.648+0000 7f8401ffb700 1 -- 172.21.15.175:0/2084049373 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408001ab0 con 0x7f840c14fbf0 2023-10-10T18:48:01.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.648+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840c165e90 con 0x7f840c14fbf0 2023-10-10T18:48:01.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.649+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840c067570 con 0x7f840c14fbf0 2023-10-10T18:48:01.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.649+0000 7f8401ffb700 1 -- 172.21.15.175:0/2084049373 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f840801b3b0 con 0x7f840c14fbf0 2023-10-10T18:48:01.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.650+0000 7f8401ffb700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83f0066f30 0x7f83f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:01.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.650+0000 7f8401ffb700 1 -- 172.21.15.175:0/2084049373 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f8408083e00 con 0x7f840c14fbf0 2023-10-10T18:48:01.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.650+0000 7f841155c700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83f0066f30 0x7f83f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:01.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.650+0000 7f841155c700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83f0066f30 0x7f83f00693e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f840c1554e0 tx=0x7f83fc000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:01.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.654+0000 7f8401ffb700 1 -- 172.21.15.175:0/2084049373 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8408050fa0 con 0x7f840c14fbf0 2023-10-10T18:48:01.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.876+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f840c069780 con 0x7f840c14fbf0 2023-10-10T18:48:01.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.876+0000 7f8401ffb700 1 -- 172.21.15.175:0/2084049373 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5426 (secure 0 0 0) 0x7f8408017020 con 0x7f840c14fbf0 2023-10-10T18:48:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:48:01.877 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:48:00.592089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:rejoin","state_seq":47,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:48:01.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83f0066f30 msgr2=0x7f83f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:01.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83f0066f30 0x7f83f00693e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f840c1554e0 tx=0x7f83fc000bc0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f840c14fbf0 msgr2=0x7f840c0cbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:01.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f840c14fbf0 0x7f840c0cbb50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f840800c7c0 tx=0x7f8408007dc0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 shutdown_connections 2023-10-10T18:48:01.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83f0066f30 0x7f83f00693e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f840c159d70 0x7f840c14c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f840c14fbf0 0x7f840c0cbb50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 --2- 172.21.15.175:0/2084049373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f840c14ffc0 0x7f840c1690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:01.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 >> 172.21.15.175:0/2084049373 conn(0x7f840c0cc760 msgr2=0x7f840c0ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:01.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 shutdown_connections 2023-10-10T18:48:01.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:01.878+0000 7f8413fc1700 1 -- 172.21.15.175:0/2084049373 wait complete. 2023-10-10T18:48:01.881 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 26 2023-10-10T18:48:01.890 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:48:03.891 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:48:04.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.018+0000 7f2c8bfff700 1 Processor -- start 2023-10-10T18:48:04.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.018+0000 7f2c8bfff700 1 -- start start 2023-10-10T18:48:04.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2c8c14fd90 0x7f2c8c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:04.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8bfff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2c8c08a100 0x7f2c8c0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:04.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8bfff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8c067fa0 con 0x7f2c8c0cd040 2023-10-10T18:48:04.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8bfff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8c0c5600 con 0x7f2c8c14fd90 2023-10-10T18:48:04.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8bfff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8c156b70 con 0x7f2c8c08a100 2023-10-10T18:48:04.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2c8c14fd90 0x7f2c8c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:04.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2c8c0cd040 0x7f2c8c0684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53308/0 (socket says 172.21.15.175:53308) 2023-10-10T18:48:04.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8b7fe700 1 -- 172.21.15.175:0/464262224 learned_addr learned my addr 172.21.15.175:0/464262224 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:48:04.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c827fc700 1 --2- 172.21.15.175:0/464262224 >> v2:172.21.15.191:3300/0 conn(0x7f2c8c08a100 0x7f2c8c0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:04.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8affd700 1 -- 172.21.15.175:0/464262224 >> v2:172.21.15.191:3300/0 conn(0x7f2c8c08a100 msgr2=0x7f2c8c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:04.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8affd700 1 --2- 172.21.15.175:0/464262224 >> v2:172.21.15.191:3300/0 conn(0x7f2c8c08a100 0x7f2c8c0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8affd700 1 -- 172.21.15.175:0/464262224 >> v1:172.21.15.175:6789/0 conn(0x7f2c8c0cd040 legacy=0x7f2c8c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:04.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8affd700 1 -- 172.21.15.175:0/464262224 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c150160 con 0x7f2c8c14fd90 2023-10-10T18:48:04.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c8a7fc700 1 -- 172.21.15.175:0/464262224 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716226344 0 0) 0x7f2c8c067fa0 con 0x7f2c8c0cd040 2023-10-10T18:48:04.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.019+0000 7f2c827fc700 1 --2- 172.21.15.175:0/464262224 >> v2:172.21.15.191:3300/0 conn(0x7f2c8c08a100 0x7f2c8c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:48:04.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8affd700 1 --2- 172.21.15.175:0/464262224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c159c30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c069990 tx=0x7f2c7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e98cece2a1c5ba server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:04.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8a7fc700 1 -- 172.21.15.175:0/464262224 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c00b740 con 0x7f2c8c14fd90 2023-10-10T18:48:04.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8a7fc700 1 -- 172.21.15.175:0/464262224 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c00b8a0 con 0x7f2c8c14fd90 2023-10-10T18:48:04.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8a7fc700 1 -- 172.21.15.175:0/464262224 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c00bc40 con 0x7f2c8c14fd90 2023-10-10T18:48:04.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8bfff700 1 -- 172.21.15.175:0/464262224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 msgr2=0x7f2c8c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:04.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/464262224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c159c30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c069990 tx=0x7f2c7c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8bfff700 1 -- 172.21.15.175:0/464262224 shutdown_connections 2023-10-10T18:48:04.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/464262224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c159c30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/464262224 >> v2:172.21.15.191:3300/0 conn(0x7f2c8c08a100 0x7f2c8c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8bfff700 1 -- 172.21.15.175:0/464262224 >> 172.21.15.175:0/464262224 conn(0x7f2c8c063d00 msgr2=0x7f2c8c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:04.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8bfff700 1 -- 172.21.15.175:0/464262224 shutdown_connections 2023-10-10T18:48:04.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.020+0000 7f2c8bfff700 1 -- 172.21.15.175:0/464262224 wait complete. 2023-10-10T18:48:04.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.021+0000 7f2c8bfff700 1 Processor -- start 2023-10-10T18:48:04.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8bfff700 1 -- start start 2023-10-10T18:48:04.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8c08a100 0x7f2c8c154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:04.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8bfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0cd040 0x7f2c8c156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:04.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c1574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:04.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8bfff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c157a20 con 0x7f2c8c14fd90 2023-10-10T18:48:04.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8bfff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c1518f0 con 0x7f2c8c0cd040 2023-10-10T18:48:04.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8bfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c165b70 con 0x7f2c8c08a100 2023-10-10T18:48:04.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c1574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:04.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c1574e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44156/0 (socket says 172.21.15.175:44156) 2023-10-10T18:48:04.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8b7fe700 1 -- 172.21.15.175:0/3945554620 learned_addr learned my addr 172.21.15.175:0/3945554620 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:04.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c827fc700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0cd040 0x7f2c8c156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:04.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.022+0000 7f2c8affd700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8c08a100 0x7f2c8c154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:04.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c8b7fe700 1 -- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8c08a100 msgr2=0x7f2c8c154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:04.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c8b7fe700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8c08a100 0x7f2c8c154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c8b7fe700 1 -- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0cd040 msgr2=0x7f2c8c156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:04.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c8b7fe700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0cd040 0x7f2c8c156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c8b7fe700 1 -- 172.21.15.175:0/3945554620 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c009420 con 0x7f2c8c14fd90 2023-10-10T18:48:04.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c8b7fe700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c1574e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400cc90 tx=0x7f2c84001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:04.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c837fe700 1 -- 172.21.15.175:0/3945554620 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8400a8b0 con 0x7f2c8c14fd90 2023-10-10T18:48:04.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c837fe700 1 -- 172.21.15.175:0/3945554620 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8400aa10 con 0x7f2c8c14fd90 2023-10-10T18:48:04.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c165e10 con 0x7f2c8c14fd90 2023-10-10T18:48:04.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c837fe700 1 -- 172.21.15.175:0/3945554620 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8400add0 con 0x7f2c8c14fd90 2023-10-10T18:48:04.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.023+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8c166670 con 0x7f2c8c14fd90 2023-10-10T18:48:04.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.024+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c8c156e10 con 0x7f2c8c14fd90 2023-10-10T18:48:04.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.024+0000 7f2c837fe700 1 -- 172.21.15.175:0/3945554620 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2c84007300 con 0x7f2c8c14fd90 2023-10-10T18:48:04.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.025+0000 7f2c837fe700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c6c066f30 0x7f2c6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:04.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.025+0000 7f2c837fe700 1 -- 172.21.15.175:0/3945554620 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f2c84084960 con 0x7f2c8c14fd90 2023-10-10T18:48:04.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.025+0000 7f2c8affd700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c6c066f30 0x7f2c6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:04.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.026+0000 7f2c8affd700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c6c066f30 0x7f2c6c0693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c0093f0 tx=0x7f2c7c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:04.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.029+0000 7f2c837fe700 1 -- 172.21.15.175:0/3945554620 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c84051b80 con 0x7f2c8c14fd90 2023-10-10T18:48:04.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.255+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c8c067570 con 0x7f2c8c14fd90 2023-10-10T18:48:04.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.255+0000 7f2c837fe700 1 -- 172.21.15.175:0/3945554620 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5426 (secure 0 0 0) 0x7f2c840551a0 con 0x7f2c8c14fd90 2023-10-10T18:48:04.256 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:48:04.256 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:48:00.592089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:rejoin","state_seq":47,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:48:04.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c6c066f30 msgr2=0x7f2c6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:04.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c6c066f30 0x7f2c6c0693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c0093f0 tx=0x7f2c7c017000 comp rx=0 tx=0).stop 2023-10-10T18:48:04.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 msgr2=0x7f2c8c1574e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:04.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c1574e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400cc90 tx=0x7f2c84001a80 comp rx=0 tx=0).stop 2023-10-10T18:48:04.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 shutdown_connections 2023-10-10T18:48:04.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c6c066f30 0x7f2c6c0693e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8c14fd90 0x7f2c8c1574e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8c08a100 0x7f2c8c154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 --2- 172.21.15.175:0/3945554620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0cd040 0x7f2c8c156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:04.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 >> 172.21.15.175:0/3945554620 conn(0x7f2c8c063d00 msgr2=0x7f2c8c0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:04.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 shutdown_connections 2023-10-10T18:48:04.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:04.257+0000 7f2c8bfff700 1 -- 172.21.15.175:0/3945554620 wait complete. 2023-10-10T18:48:04.260 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 26 2023-10-10T18:48:04.269 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:48:06.270 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:48:06.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.397+0000 7f47b6415700 1 Processor -- start 2023-10-10T18:48:06.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.397+0000 7f47b6415700 1 -- start start 2023-10-10T18:48:06.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47b6415700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:06.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47b6415700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f47b014cc10 0x7f47b00c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:06.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47b6415700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b0155ca0 con 0x7f47b00c59d0 2023-10-10T18:48:06.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47b6415700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b006bd50 con 0x7f47b014cc10 2023-10-10T18:48:06.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47b6415700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0156b00 con 0x7f47b014c840 2023-10-10T18:48:06.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47b49b2700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f47b00c59d0 0x7f47b006b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52778/0 (socket says 172.21.15.175:52778) 2023-10-10T18:48:06.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47b49b2700 1 -- 172.21.15.175:0/1873411610 learned_addr learned my addr 172.21.15.175:0/1873411610 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:48:06.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47affff700 1 --2- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:06.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.398+0000 7f47af7fe700 1 --2- 172.21.15.175:0/1873411610 >> v2:172.21.15.191:3300/0 conn(0x7f47b014cc10 0x7f47b00c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:06.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47aeffd700 1 -- 172.21.15.175:0/1873411610 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133288864 0 0) 0x7f47b0155ca0 con 0x7f47b00c59d0 2023-10-10T18:48:06.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47aeffd700 1 -- 172.21.15.175:0/1873411610 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4798003570 con 0x7f47b00c59d0 2023-10-10T18:48:06.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47af7fe700 1 -- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 msgr2=0x7f47b006e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:06.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47af7fe700 1 --2- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b006e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47af7fe700 1 -- 172.21.15.175:0/1873411610 >> v1:172.21.15.175:6789/0 conn(0x7f47b00c59d0 legacy=0x7f47b006b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:06.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47af7fe700 1 -- 172.21.15.175:0/1873411610 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b006d700 con 0x7f47b014cc10 2023-10-10T18:48:06.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47affff700 1 --2- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b006e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:48:06.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47af7fe700 1 --2- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b014cc10 0x7f47b00c4b90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47a40098e0 tx=0x7f47a40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=632c3563ca1b073b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:06.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47aeffd700 1 -- 172.21.15.175:0/1873411610 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4011630 con 0x7f47b014cc10 2023-10-10T18:48:06.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47aeffd700 1 -- 172.21.15.175:0/1873411610 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a4011790 con 0x7f47b014cc10 2023-10-10T18:48:06.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.399+0000 7f47aeffd700 1 -- 172.21.15.175:0/1873411610 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4011b30 con 0x7f47b014cc10 2023-10-10T18:48:06.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.400+0000 7f47b6415700 1 -- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b014cc10 msgr2=0x7f47b00c4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:06.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.400+0000 7f47b6415700 1 --2- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b014cc10 0x7f47b00c4b90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47a40098e0 tx=0x7f47a40091a0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.400+0000 7f47b6415700 1 -- 172.21.15.175:0/1873411610 shutdown_connections 2023-10-10T18:48:06.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.400+0000 7f47b6415700 1 --2- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b006e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.400+0000 7f47b6415700 1 --2- 172.21.15.175:0/1873411610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b014cc10 0x7f47b00c4b90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.400+0000 7f47b6415700 1 -- 172.21.15.175:0/1873411610 >> 172.21.15.175:0/1873411610 conn(0x7f47b0068280 msgr2=0x7f47b006af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:06.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.400+0000 7f47b6415700 1 -- 172.21.15.175:0/1873411610 shutdown_connections 2023-10-10T18:48:06.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.400+0000 7f47b6415700 1 -- 172.21.15.175:0/1873411610 wait complete. 2023-10-10T18:48:06.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.401+0000 7f47b6415700 1 Processor -- start 2023-10-10T18:48:06.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.401+0000 7f47b6415700 1 -- start start 2023-10-10T18:48:06.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b6415700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b00c59d0 0x7f47b0146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:06.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b6415700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b01442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:06.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b6415700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47b014cc10 0x7f47b0144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:06.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b6415700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b01523d0 con 0x7f47b014cc10 2023-10-10T18:48:06.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b6415700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0151a90 con 0x7f47b00c59d0 2023-10-10T18:48:06.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b6415700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0145ef0 con 0x7f47b014c840 2023-10-10T18:48:06.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47b014cc10 0x7f47b0144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:06.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47b014cc10 0x7f47b0144800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37884/0 (socket says 172.21.15.175:37884) 2023-10-10T18:48:06.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 -- 172.21.15.175:0/1860380963 learned_addr learned my addr 172.21.15.175:0/1860380963 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:06.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47af7fe700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b01442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:06.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47affff700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b00c59d0 0x7f47b0146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:06.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 -- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 msgr2=0x7f47b01442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:06.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b01442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 -- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b00c59d0 msgr2=0x7f47b0146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:06.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b00c59d0 0x7f47b0146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 -- 172.21.15.175:0/1860380963 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00093c0 con 0x7f47b014cc10 2023-10-10T18:48:06.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f47b49b2700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47b014cc10 0x7f47b0144800 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f47a800cc40 tx=0x7f47a8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:06.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f4797fff700 1 -- 172.21.15.175:0/1860380963 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800a830 con 0x7f47b014cc10 2023-10-10T18:48:06.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f4797fff700 1 -- 172.21.15.175:0/1860380963 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a800a990 con 0x7f47b014cc10 2023-10-10T18:48:06.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.402+0000 7f4797fff700 1 -- 172.21.15.175:0/1860380963 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800ad10 con 0x7f47b014cc10 2023-10-10T18:48:06.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.403+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a4009420 con 0x7f47b014cc10 2023-10-10T18:48:06.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.403+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b01520f0 con 0x7f47b014cc10 2023-10-10T18:48:06.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.404+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b0067570 con 0x7f47b014cc10 2023-10-10T18:48:06.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.404+0000 7f4797fff700 1 -- 172.21.15.175:0/1860380963 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f47a8007300 con 0x7f47b014cc10 2023-10-10T18:48:06.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.404+0000 7f4797fff700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f479c066f30 0x7f479c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:06.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.405+0000 7f4797fff700 1 -- 172.21.15.175:0/1860380963 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f47a8084870 con 0x7f47b014cc10 2023-10-10T18:48:06.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.405+0000 7f47affff700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f479c066f30 0x7f479c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:06.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.405+0000 7f47affff700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f479c066f30 0x7f479c0693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009f30 tx=0x7f47a0009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:06.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.409+0000 7f4797fff700 1 -- 172.21.15.175:0/1860380963 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47a8051a90 con 0x7f47b014cc10 2023-10-10T18:48:06.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.640+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47b00c9260 con 0x7f47b014cc10 2023-10-10T18:48:06.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.640+0000 7f4797fff700 1 -- 172.21.15.175:0/1860380963 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5432 (secure 0 0 0) 0x7f47a80550b0 con 0x7f47b014cc10 2023-10-10T18:48:06.641 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:48:06.641 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:48:06.610992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:clientreplay","state_seq":49,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:48:06.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f479c066f30 msgr2=0x7f479c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:06.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f479c066f30 0x7f479c0693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009f30 tx=0x7f47a0009120 comp rx=0 tx=0).stop 2023-10-10T18:48:06.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47b014cc10 msgr2=0x7f47b0144800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:06.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47b014cc10 0x7f47b0144800 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f47a800cc40 tx=0x7f47a8001a40 comp rx=0 tx=0).stop 2023-10-10T18:48:06.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 shutdown_connections 2023-10-10T18:48:06.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f479c066f30 0x7f479c0693e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47b014cc10 0x7f47b0144800 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47b00c59d0 0x7f47b0146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 --2- 172.21.15.175:0/1860380963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47b014c840 0x7f47b01442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:06.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 >> 172.21.15.175:0/1860380963 conn(0x7f47b0068280 msgr2=0x7f47b00c29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:06.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 shutdown_connections 2023-10-10T18:48:06.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:06.642+0000 7f47b6415700 1 -- 172.21.15.175:0/1860380963 wait complete. 2023-10-10T18:48:06.646 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 27 2023-10-10T18:48:06.654 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:48:08.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:48:08.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.783+0000 7f7f8d663700 1 Processor -- start 2023-10-10T18:48:08.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.783+0000 7f7f8d663700 1 -- start start 2023-10-10T18:48:08.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f8d663700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f8814fd90 0x7f7f88159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:08.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f8d663700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7f8808a100 0x7f7f880c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:08.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f8d663700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f88067fa0 con 0x7f7f880cd040 2023-10-10T18:48:08.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f8d663700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f880c5600 con 0x7f7f8808a100 2023-10-10T18:48:08.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f8d663700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f88156b70 con 0x7f7f8814fd90 2023-10-10T18:48:08.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f877fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7f8808a100 0x7f7f880c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:08.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f87fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f8814fd90 0x7f7f88159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:08.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f877fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7f8808a100 0x7f7f880c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:60426/0 (socket says 172.21.15.175:60426) 2023-10-10T18:48:08.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f87fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f8814fd90 0x7f7f88159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:58496/0 (socket says 172.21.15.175:58496) 2023-10-10T18:48:08.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.784+0000 7f7f877fe700 1 -- 172.21.15.175:0/2848566067 learned_addr learned my addr 172.21.15.175:0/2848566067 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:08.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f877fe700 1 -- 172.21.15.175:0/2848566067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f8814fd90 msgr2=0x7f7f88159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:08.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f877fe700 1 --2- 172.21.15.175:0/2848566067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f8814fd90 0x7f7f88159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:08.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f877fe700 1 -- 172.21.15.175:0/2848566067 >> v1:172.21.15.191:6789/0 conn(0x7f7f880cd040 legacy=0x7f7f880684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:08.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f877fe700 1 -- 172.21.15.175:0/2848566067 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f88150160 con 0x7f7f8808a100 2023-10-10T18:48:08.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f87fff700 1 --2- 172.21.15.175:0/2848566067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f8814fd90 0x7f7f88159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:48:08.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f877fe700 1 --2- 172.21.15.175:0/2848566067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 0x7f7f880c5eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0099b0 tx=0x7f7f7c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4708382bbd5d9bd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:08.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f86ffd700 1 -- 172.21.15.175:0/2848566067 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c009ea0 con 0x7f7f8808a100 2023-10-10T18:48:08.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f86ffd700 1 -- 172.21.15.175:0/2848566067 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c00b860 con 0x7f7f8808a100 2023-10-10T18:48:08.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.785+0000 7f7f86ffd700 1 -- 172.21.15.175:0/2848566067 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00bc00 con 0x7f7f8808a100 2023-10-10T18:48:08.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.786+0000 7f7f8d663700 1 -- 172.21.15.175:0/2848566067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 msgr2=0x7f7f880c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:08.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.786+0000 7f7f8d663700 1 --2- 172.21.15.175:0/2848566067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 0x7f7f880c5eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0099b0 tx=0x7f7f7c009250 comp rx=0 tx=0).stop 2023-10-10T18:48:08.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.786+0000 7f7f8d663700 1 -- 172.21.15.175:0/2848566067 shutdown_connections 2023-10-10T18:48:08.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.786+0000 7f7f8d663700 1 --2- 172.21.15.175:0/2848566067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f8814fd90 0x7f7f88159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:08.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.786+0000 7f7f8d663700 1 --2- 172.21.15.175:0/2848566067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 0x7f7f880c5eb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:08.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.786+0000 7f7f8d663700 1 -- 172.21.15.175:0/2848566067 >> 172.21.15.175:0/2848566067 conn(0x7f7f88063d00 msgr2=0x7f7f880cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:08.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.786+0000 7f7f8d663700 1 -- 172.21.15.175:0/2848566067 shutdown_connections 2023-10-10T18:48:08.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.786+0000 7f7f8d663700 1 -- 172.21.15.175:0/2848566067 wait complete. 2023-10-10T18:48:08.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.787+0000 7f7f8d663700 1 Processor -- start 2023-10-10T18:48:08.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.787+0000 7f7f8d663700 1 -- start start 2023-10-10T18:48:08.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8d663700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 0x7f7f88166060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:08.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8d663700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f880cd040 0x7f7f88167280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:08.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8d663700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f8814fd90 0x7f7f881497b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:08.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8d663700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880c8950 con 0x7f7f8814fd90 2023-10-10T18:48:08.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8d663700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8815af70 con 0x7f7f8808a100 2023-10-10T18:48:08.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8d663700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880c84c0 con 0x7f7f880cd040 2023-10-10T18:48:08.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f8814fd90 0x7f7f881497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:08.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f8814fd90 0x7f7f881497b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37894/0 (socket says 172.21.15.175:37894) 2023-10-10T18:48:08.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f877fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f880cd040 0x7f7f88167280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:08.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f87fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 0x7f7f88166060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:08.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 -- 172.21.15.175:0/1106584955 learned_addr learned my addr 172.21.15.175:0/1106584955 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:08.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 -- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f880cd040 msgr2=0x7f7f88167280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:08.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f880cd040 0x7f7f88167280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:08.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 -- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 msgr2=0x7f7f88166060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:08.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 0x7f7f88166060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:08.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 -- 172.21.15.175:0/1106584955 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f78009420 con 0x7f7f8814fd90 2023-10-10T18:48:08.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f87fff700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 0x7f7f88166060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:48:08.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8ce62700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f8814fd90 0x7f7f881497b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c910 tx=0x7f7f80001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:08.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f6ffff700 1 -- 172.21.15.175:0/1106584955 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80012630 con 0x7f7f8814fd90 2023-10-10T18:48:08.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f6ffff700 1 -- 172.21.15.175:0/1106584955 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f800127b0 con 0x7f7f8814fd90 2023-10-10T18:48:08.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f6ffff700 1 -- 172.21.15.175:0/1106584955 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80012b70 con 0x7f7f8814fd90 2023-10-10T18:48:08.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7c0094f0 con 0x7f7f8814fd90 2023-10-10T18:48:08.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.788+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f88166ca0 con 0x7f7f8814fd90 2023-10-10T18:48:08.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.789+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f88159fb0 con 0x7f7f8814fd90 2023-10-10T18:48:08.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.790+0000 7f7f6ffff700 1 -- 172.21.15.175:0/1106584955 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7f80003030 con 0x7f7f8814fd90 2023-10-10T18:48:08.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.790+0000 7f7f6ffff700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f74066f30 0x7f7f740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:08.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.790+0000 7f7f6ffff700 1 -- 172.21.15.175:0/1106584955 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f7f80084530 con 0x7f7f8814fd90 2023-10-10T18:48:08.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.791+0000 7f7f87fff700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f74066f30 0x7f7f740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:08.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.791+0000 7f7f87fff700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f74066f30 0x7f7f740693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7f78011560 tx=0x7f7f780091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:08.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:08.795+0000 7f7f6ffff700 1 -- 172.21.15.175:0/1106584955 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f800516d0 con 0x7f7f8814fd90 2023-10-10T18:48:09.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.019+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f880c8600 con 0x7f7f8814fd90 2023-10-10T18:48:09.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.020+0000 7f7f6ffff700 1 -- 172.21.15.175:0/1106584955 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5432 (secure 0 0 0) 0x7f7f80054cf0 con 0x7f7f8814fd90 2023-10-10T18:48:09.020 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:48:09.020 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:48:06.610992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:clientreplay","state_seq":49,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:48:09.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f74066f30 msgr2=0x7f7f740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:09.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f74066f30 0x7f7f740693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7f78011560 tx=0x7f7f780091a0 comp rx=0 tx=0).stop 2023-10-10T18:48:09.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f8814fd90 msgr2=0x7f7f881497b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:09.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f8814fd90 0x7f7f881497b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c910 tx=0x7f7f80001710 comp rx=0 tx=0).stop 2023-10-10T18:48:09.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 shutdown_connections 2023-10-10T18:48:09.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7f74066f30 0x7f7f740693e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:09.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f8814fd90 0x7f7f881497b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:09.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f8808a100 0x7f7f88166060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:09.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 --2- 172.21.15.175:0/1106584955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f880cd040 0x7f7f88167280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:09.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 >> 172.21.15.175:0/1106584955 conn(0x7f7f88063d00 msgr2=0x7f7f880c4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:09.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 shutdown_connections 2023-10-10T18:48:09.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:09.022+0000 7f7f8d663700 1 -- 172.21.15.175:0/1106584955 wait complete. 2023-10-10T18:48:09.025 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 27 2023-10-10T18:48:09.035 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4245": { "addr": "172.21.15.175:6834/688515950", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 688515950, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 688515950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4245, "incarnation": 23, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:clientreplay", "state_seq": 49 }, "gid_4248": { "addr": "172.21.15.175:6836/647425682", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 647425682, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 647425682, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4248, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4250": { "addr": "172.21.15.191:6839/246221790", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 246221790, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 246221790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4250, "incarnation": 12, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:48:06.610992+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 4250, "mds_3": 4248, "mds_4": 4245 } } } ], "standbys": [] } 2023-10-10T18:48:11.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:48:11.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.164+0000 7f43ac1b0700 1 Processor -- start 2023-10-10T18:48:11.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.164+0000 7f43ac1b0700 1 -- start start 2023-10-10T18:48:11.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43ac1b0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f43a41573d0 0x7f43a40c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:11.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43ac1b0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 0x7f43a40c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:11.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43ac1b0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43a40bf6e0 0x7f43a40c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:11.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43ac1b0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a4154aa0 con 0x7f43a41573d0 2023-10-10T18:48:11.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43ac1b0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a4063dc0 con 0x7f43a40bf6e0 2023-10-10T18:48:11.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43ac1b0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a4157bf0 con 0x7f43a41577a0 2023-10-10T18:48:11.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f43a41573d0 0x7f43a40c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:11.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f43a41573d0 0x7f43a40c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37902/0 (socket says 172.21.15.175:37902) 2023-10-10T18:48:11.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 -- 172.21.15.175:0/161402928 learned_addr learned my addr 172.21.15.175:0/161402928 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:11.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a974b700 1 --2- 172.21.15.175:0/161402928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 0x7f43a40c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:11.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43aa74d700 1 --2- 172.21.15.175:0/161402928 >> v2:172.21.15.191:3300/0 conn(0x7f43a40bf6e0 0x7f43a40c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:11.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 -- 172.21.15.175:0/161402928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 msgr2=0x7f43a40c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:11.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 --2- 172.21.15.175:0/161402928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 0x7f43a40c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 -- 172.21.15.175:0/161402928 >> v2:172.21.15.191:3300/0 conn(0x7f43a40bf6e0 msgr2=0x7f43a40c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:11.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 --2- 172.21.15.175:0/161402928 >> v2:172.21.15.191:3300/0 conn(0x7f43a40bf6e0 0x7f43a40c6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 -- 172.21.15.175:0/161402928 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4157e50 con 0x7f43a41573d0 2023-10-10T18:48:11.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a9f4c700 1 --2- 172.21.15.175:0/161402928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a41573d0 0x7f43a40c12a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f43a406ea20 tx=0x7f43a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6618081ccf1a9eb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:11.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a8f4a700 1 -- 172.21.15.175:0/161402928 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a000b640 con 0x7f43a41573d0 2023-10-10T18:48:11.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a8f4a700 1 -- 172.21.15.175:0/161402928 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a000b7a0 con 0x7f43a41573d0 2023-10-10T18:48:11.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.165+0000 7f43a8f4a700 1 -- 172.21.15.175:0/161402928 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a000bb60 con 0x7f43a41573d0 2023-10-10T18:48:11.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 -- 172.21.15.175:0/161402928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a41573d0 msgr2=0x7f43a40c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:11.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/161402928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a41573d0 0x7f43a40c12a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f43a406ea20 tx=0x7f43a0009250 comp rx=0 tx=0).stop 2023-10-10T18:48:11.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 -- 172.21.15.175:0/161402928 shutdown_connections 2023-10-10T18:48:11.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/161402928 >> v2:172.21.15.191:3300/0 conn(0x7f43a40bf6e0 0x7f43a40c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/161402928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a41573d0 0x7f43a40c12a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/161402928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 0x7f43a40c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 -- 172.21.15.175:0/161402928 >> 172.21.15.175:0/161402928 conn(0x7f43a406dc70 msgr2=0x7f43a4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:11.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 -- 172.21.15.175:0/161402928 shutdown_connections 2023-10-10T18:48:11.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.166+0000 7f43ac1b0700 1 -- 172.21.15.175:0/161402928 wait complete. 2023-10-10T18:48:11.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.167+0000 7f43ac1b0700 1 Processor -- start 2023-10-10T18:48:11.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.167+0000 7f43ac1b0700 1 -- start start 2023-10-10T18:48:11.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43ac1b0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a40bf6e0 0x7f43a40caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:11.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43ac1b0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a41573d0 0x7f43a40c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:11.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43ac1b0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 0x7f43a4152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:11.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43ac1b0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c89d0 con 0x7f43a40bf6e0 2023-10-10T18:48:11.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43ac1b0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c5a70 con 0x7f43a41573d0 2023-10-10T18:48:11.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43ac1b0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c6af0 con 0x7f43a41577a0 2023-10-10T18:48:11.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a40bf6e0 0x7f43a40caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:11.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a40bf6e0 0x7f43a40caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37908/0 (socket says 172.21.15.175:37908) 2023-10-10T18:48:11.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 -- 172.21.15.175:0/1415312915 learned_addr learned my addr 172.21.15.175:0/1415312915 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:11.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a974b700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a41573d0 0x7f43a40c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:11.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43aa74d700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 0x7f43a4152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:11.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 -- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 msgr2=0x7f43a4152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:11.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 0x7f43a4152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 -- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a41573d0 msgr2=0x7f43a40c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:11.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a41573d0 0x7f43a40c8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 -- 172.21.15.175:0/1415312915 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a00094f0 con 0x7f43a40bf6e0 2023-10-10T18:48:11.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.168+0000 7f43a9f4c700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a40bf6e0 0x7f43a40caad0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f43a0006fd0 tx=0x7f43a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:11.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.169+0000 7f4399ffb700 1 -- 172.21.15.175:0/1415312915 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a000b910 con 0x7f43a40bf6e0 2023-10-10T18:48:11.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.169+0000 7f4399ffb700 1 -- 172.21.15.175:0/1415312915 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a000ba70 con 0x7f43a40bf6e0 2023-10-10T18:48:11.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.169+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a40c6d90 con 0x7f43a40bf6e0 2023-10-10T18:48:11.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.169+0000 7f4399ffb700 1 -- 172.21.15.175:0/1415312915 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0011650 con 0x7f43a40bf6e0 2023-10-10T18:48:11.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.169+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a40c7120 con 0x7f43a40bf6e0 2023-10-10T18:48:11.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.170+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a40c6f20 con 0x7f43a40bf6e0 2023-10-10T18:48:11.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.170+0000 7f4399ffb700 1 -- 172.21.15.175:0/1415312915 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f43a00119c0 con 0x7f43a40bf6e0 2023-10-10T18:48:11.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.171+0000 7f4399ffb700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4388066f60 0x7f4388069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:11.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.171+0000 7f4399ffb700 1 -- 172.21.15.175:0/1415312915 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f43a0085d20 con 0x7f43a40bf6e0 2023-10-10T18:48:11.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.171+0000 7f43a974b700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4388066f60 0x7f4388069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:11.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.171+0000 7f43a974b700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4388066f60 0x7f4388069410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f43a4154350 tx=0x7f43940091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:11.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.175+0000 7f4399ffb700 1 -- 172.21.15.175:0/1415312915 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f43a0052e20 con 0x7f43a40bf6e0 2023-10-10T18:48:11.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.398+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43a40675d0 con 0x7f43a40bf6e0 2023-10-10T18:48:11.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.399+0000 7f4399ffb700 1 -- 172.21.15.175:0/1415312915 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5432 (secure 0 0 0) 0x7f43a0056440 con 0x7f43a40bf6e0 2023-10-10T18:48:11.399 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:48:11.399 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:48:06.610992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:clientreplay","state_seq":49,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:48:11.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4388066f60 msgr2=0x7f4388069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:11.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4388066f60 0x7f4388069410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f43a4154350 tx=0x7f43940091a0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a40bf6e0 msgr2=0x7f43a40caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:11.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a40bf6e0 0x7f43a40caad0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f43a0006fd0 tx=0x7f43a0017000 comp rx=0 tx=0).stop 2023-10-10T18:48:11.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 shutdown_connections 2023-10-10T18:48:11.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4388066f60 0x7f4388069410 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43a41577a0 0x7f43a4152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43a40bf6e0 0x7f43a40caad0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 --2- 172.21.15.175:0/1415312915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a41573d0 0x7f43a40c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:11.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 >> 172.21.15.175:0/1415312915 conn(0x7f43a406dc70 msgr2=0x7f43a406acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:11.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 shutdown_connections 2023-10-10T18:48:11.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:11.401+0000 7f43ac1b0700 1 -- 172.21.15.175:0/1415312915 wait complete. 2023-10-10T18:48:11.403 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 27 2023-10-10T18:48:11.412 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:48:13.413 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:48:13.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.539+0000 7f66f781a700 1 Processor -- start 2023-10-10T18:48:13.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.540+0000 7f66f781a700 1 -- start start 2023-10-10T18:48:13.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.540+0000 7f66f781a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f66f014cc10 0x7f66f006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:13.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.540+0000 7f66f781a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f0155ca0 con 0x7f66f00c52a0 2023-10-10T18:48:13.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.540+0000 7f66f781a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f0156b00 con 0x7f66f014c840 2023-10-10T18:48:13.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.540+0000 7f66f781a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f006d700 con 0x7f66f014cc10 2023-10-10T18:48:13.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.540+0000 7f66f5db7700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f66f00c52a0 0x7f66f0152970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52784/0 (socket says 172.21.15.175:52784) 2023-10-10T18:48:13.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.540+0000 7f66f5db7700 1 -- 172.21.15.175:0/3148966317 learned_addr learned my addr 172.21.15.175:0/3148966317 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:48:13.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66f4db5700 1 --2- 172.21.15.175:0/3148966317 >> v2:172.21.15.191:3300/0 conn(0x7f66f014cc10 0x7f66f006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:13.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173818437 0 0) 0x7f66f0155ca0 con 0x7f66f00c52a0 2023-10-10T18:48:13.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66dc003570 con 0x7f66f00c52a0 2023-10-10T18:48:13.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059919935 0 0) 0x7f66dc003570 con 0x7f66f00c52a0 2023-10-10T18:48:13.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 >> v2:172.21.15.191:3300/0 conn(0x7f66f014cc10 msgr2=0x7f66f006e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:13.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 --2- 172.21.15.175:0/3148966317 >> v2:172.21.15.191:3300/0 conn(0x7f66f014cc10 0x7f66f006e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 >> v1:172.21.15.191:6789/0 conn(0x7f66f014c840 legacy=0x7f66f00c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:13.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0155ef0 con 0x7f66f00c52a0 2023-10-10T18:48:13.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f66ec0017f0 con 0x7f66f00c52a0 2023-10-10T18:48:13.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 >> v1:172.21.15.175:6789/0 conn(0x7f66f00c52a0 legacy=0x7f66f0152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:13.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66dc003a60 0x7f66dc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:13.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66dc004810 0x7f66dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:13.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66dc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:13.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f006d700 con 0x7f66dc0051c0 2023-10-10T18:48:13.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc00d1e0 con 0x7f66dc004810 2023-10-10T18:48:13.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc00d3a0 con 0x7f66dc003a60 2023-10-10T18:48:13.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66f4db5700 1 --2- 172.21.15.175:0/3148966317 >> v2:172.21.15.191:3300/0 conn(0x7f66f014cc10 0x7f66f006e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:48:13.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66f5db7700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66dc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:13.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66f55b6700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66dc003a60 0x7f66dc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:13.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66f4db5700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66dc004810 0x7f66dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:13.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66f5db7700 1 -- 172.21.15.175:0/3148966317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66dc003a60 msgr2=0x7f66dc003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:13.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66f5db7700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66dc003a60 0x7f66dc003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.541+0000 7f66f5db7700 1 -- 172.21.15.175:0/3148966317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66dc004810 msgr2=0x7f66dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:13.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f5db7700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66dc004810 0x7f66dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f5db7700 1 -- 172.21.15.175:0/3148966317 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc003570 con 0x7f66dc0051c0 2023-10-10T18:48:13.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f5db7700 1 -- 172.21.15.175:0/3148966317 reap_dead start 2023-10-10T18:48:13.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f5db7700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66dc00ba60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00a930 tx=0x7f66ec0021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863cec53fa8da185 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:13.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec0070d0 con 0x7f66dc0051c0 2023-10-10T18:48:13.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ec007270 con 0x7f66dc0051c0 2023-10-10T18:48:13.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66e7fff700 1 -- 172.21.15.175:0/3148966317 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec007630 con 0x7f66dc0051c0 2023-10-10T18:48:13.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f781a700 1 -- 172.21.15.175:0/3148966317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 msgr2=0x7f66dc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:13.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f781a700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66dc00ba60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00a930 tx=0x7f66ec0021d0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f781a700 1 -- 172.21.15.175:0/3148966317 shutdown_connections 2023-10-10T18:48:13.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f781a700 1 --2- 172.21.15.175:0/3148966317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66dc00ba60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f781a700 1 -- 172.21.15.175:0/3148966317 >> 172.21.15.175:0/3148966317 conn(0x7f66f0068280 msgr2=0x7f66f006af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:13.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f781a700 1 -- 172.21.15.175:0/3148966317 shutdown_connections 2023-10-10T18:48:13.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.542+0000 7f66f781a700 1 -- 172.21.15.175:0/3148966317 wait complete. 2023-10-10T18:48:13.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.544+0000 7f66f781a700 1 Processor -- start 2023-10-10T18:48:13.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.544+0000 7f66f781a700 1 -- start start 2023-10-10T18:48:13.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f781a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66f01442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:13.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f781a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66f01502e0 0x7f66f0144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:13.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f781a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f0147750 0x7f66f0161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:13.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f781a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0147d00 con 0x7f66dc0051c0 2023-10-10T18:48:13.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f781a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0142940 con 0x7f66f0147750 2023-10-10T18:48:13.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f781a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0142c20 con 0x7f66f01502e0 2023-10-10T18:48:13.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66f01442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:13.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66f01442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37928/0 (socket says 172.21.15.175:37928) 2023-10-10T18:48:13.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 -- 172.21.15.175:0/2205211227 learned_addr learned my addr 172.21.15.175:0/2205211227 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:13.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f4db5700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66f01502e0 0x7f66f0144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:13.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f5db7700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f0147750 0x7f66f0161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:13.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 -- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66f01502e0 msgr2=0x7f66f0144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:13.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66f01502e0 0x7f66f0144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 -- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f0147750 msgr2=0x7f66f0161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:13.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f0147750 0x7f66f0161bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 -- 172.21.15.175:0/2205211227 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0142f00 con 0x7f66dc0051c0 2023-10-10T18:48:13.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f4db5700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66f01502e0 0x7f66f0144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:48:13.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f5db7700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f0147750 0x7f66f0161bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:48:13.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f55b6700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66f01442b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f66e8002970 tx=0x7f66e8007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:13.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66e57fa700 1 -- 172.21.15.175:0/2205211227 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e800a740 con 0x7f66dc0051c0 2023-10-10T18:48:13.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66e57fa700 1 -- 172.21.15.175:0/2205211227 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e800a8a0 con 0x7f66dc0051c0 2023-10-10T18:48:13.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f01431e0 con 0x7f66dc0051c0 2023-10-10T18:48:13.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66e57fa700 1 -- 172.21.15.175:0/2205211227 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e800ac20 con 0x7f66dc0051c0 2023-10-10T18:48:13.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.545+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f01452e0 con 0x7f66dc0051c0 2023-10-10T18:48:13.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.546+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f0067570 con 0x7f66dc0051c0 2023-10-10T18:48:13.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.547+0000 7f66e57fa700 1 -- 172.21.15.175:0/2205211227 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f66e801f460 con 0x7f66dc0051c0 2023-10-10T18:48:13.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.547+0000 7f66e57fa700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f66d4066f30 0x7f66d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:13.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.547+0000 7f66e57fa700 1 -- 172.21.15.175:0/2205211227 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f66e8086ce0 con 0x7f66dc0051c0 2023-10-10T18:48:13.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.547+0000 7f66f4db5700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f66d4066f30 0x7f66d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:13.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.548+0000 7f66f4db5700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f66d4066f30 0x7f66d40693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f66f0162980 tx=0x7f66e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:13.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.551+0000 7f66e57fa700 1 -- 172.21.15.175:0/2205211227 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66e8053e80 con 0x7f66dc0051c0 2023-10-10T18:48:13.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.782+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66f006da40 con 0x7f66dc0051c0 2023-10-10T18:48:13.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.783+0000 7f66e57fa700 1 -- 172.21.15.175:0/2205211227 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5426 (secure 0 0 0) 0x7f66e80574a0 con 0x7f66dc0051c0 2023-10-10T18:48:13.783 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:48:13.783 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:48:13.636752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:48:13.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f66d4066f30 msgr2=0x7f66d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:13.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f66d4066f30 0x7f66d40693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f66f0162980 tx=0x7f66e0008040 comp rx=0 tx=0).stop 2023-10-10T18:48:13.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 msgr2=0x7f66f01442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:13.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66f01442b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f66e8002970 tx=0x7f66e8007d80 comp rx=0 tx=0).stop 2023-10-10T18:48:13.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 shutdown_connections 2023-10-10T18:48:13.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f66d4066f30 0x7f66d40693e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f0147750 0x7f66f0161bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66dc0051c0 0x7f66f01442b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 --2- 172.21.15.175:0/2205211227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66f01502e0 0x7f66f0144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:13.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 >> 172.21.15.175:0/2205211227 conn(0x7f66f0068280 msgr2=0x7f66f00c4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:13.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 shutdown_connections 2023-10-10T18:48:13.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:13.785+0000 7f66f781a700 1 -- 172.21.15.175:0/2205211227 wait complete. 2023-10-10T18:48:13.788 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 28 2023-10-10T18:48:13.797 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=4, replacing gid=4247 2023-10-10T18:48:13.797 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b 2023-10-10T18:48:16.519 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T18:48:16.519+0000 7fc568727700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-10T18:48:18.629 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:48:18.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:48:18.681 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/30 file after the snapshot 2023-10-10T18:48:18.682 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/31 file after the snapshot 2023-10-10T18:48:18.683 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/32 file after the snapshot 2023-10-10T18:48:18.683 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/33 file after the snapshot 2023-10-10T18:48:18.684 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/34 file after the snapshot 2023-10-10T18:48:18.685 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/35 file after the snapshot 2023-10-10T18:48:18.686 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/36 file after the snapshot 2023-10-10T18:48:18.686 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/37 file after the snapshot 2023-10-10T18:48:18.687 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/38 file after the snapshot 2023-10-10T18:48:18.688 INFO:tasks.workunit.client.0.smithi175.stdout: create 227/39 file after the snapshot 2023-10-10T18:48:18.688 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/.snap/snap-subdir-test 2023-10-10T18:48:18.713 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/30 file after the snapshot 2023-10-10T18:48:18.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/31 file after the snapshot 2023-10-10T18:48:18.715 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/32 file after the snapshot 2023-10-10T18:48:18.716 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/33 file after the snapshot 2023-10-10T18:48:18.716 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/34 file after the snapshot 2023-10-10T18:48:18.717 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/35 file after the snapshot 2023-10-10T18:48:18.718 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/36 file after the snapshot 2023-10-10T18:48:18.719 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/37 file after the snapshot 2023-10-10T18:48:18.720 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/38 file after the snapshot 2023-10-10T18:48:18.720 INFO:tasks.workunit.client.0.smithi175.stdout: create 228/39 file after the snapshot 2023-10-10T18:48:18.721 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/.snap/snap-subdir-test 2023-10-10T18:48:18.748 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/30 file after the snapshot 2023-10-10T18:48:18.749 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/31 file after the snapshot 2023-10-10T18:48:18.750 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/32 file after the snapshot 2023-10-10T18:48:18.750 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/33 file after the snapshot 2023-10-10T18:48:18.751 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/34 file after the snapshot 2023-10-10T18:48:18.752 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/35 file after the snapshot 2023-10-10T18:48:18.752 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/36 file after the snapshot 2023-10-10T18:48:18.753 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/37 file after the snapshot 2023-10-10T18:48:18.754 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/38 file after the snapshot 2023-10-10T18:48:18.754 INFO:tasks.workunit.client.0.smithi175.stdout: create 229/39 file after the snapshot 2023-10-10T18:48:18.755 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/.snap/snap-subdir-test 2023-10-10T18:48:18.782 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/30 file after the snapshot 2023-10-10T18:48:18.784 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/31 file after the snapshot 2023-10-10T18:48:18.785 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/32 file after the snapshot 2023-10-10T18:48:18.786 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/33 file after the snapshot 2023-10-10T18:48:18.787 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/34 file after the snapshot 2023-10-10T18:48:18.788 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/35 file after the snapshot 2023-10-10T18:48:18.789 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/36 file after the snapshot 2023-10-10T18:48:18.790 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/37 file after the snapshot 2023-10-10T18:48:18.791 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/38 file after the snapshot 2023-10-10T18:48:18.792 INFO:tasks.workunit.client.0.smithi175.stdout: create 230/39 file after the snapshot 2023-10-10T18:48:18.793 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/.snap/snap-subdir-test 2023-10-10T18:48:19.079 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/30 file after the snapshot 2023-10-10T18:48:19.080 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/31 file after the snapshot 2023-10-10T18:48:19.081 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/32 file after the snapshot 2023-10-10T18:48:19.082 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/33 file after the snapshot 2023-10-10T18:48:19.083 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/34 file after the snapshot 2023-10-10T18:48:19.083 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/35 file after the snapshot 2023-10-10T18:48:19.085 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/36 file after the snapshot 2023-10-10T18:48:19.086 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/37 file after the snapshot 2023-10-10T18:48:19.086 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/38 file after the snapshot 2023-10-10T18:48:19.087 INFO:tasks.workunit.client.0.smithi175.stdout: create 231/39 file after the snapshot 2023-10-10T18:48:19.088 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/.snap/snap-subdir-test 2023-10-10T18:48:19.116 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/30 file after the snapshot 2023-10-10T18:48:19.117 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/31 file after the snapshot 2023-10-10T18:48:19.118 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/32 file after the snapshot 2023-10-10T18:48:19.119 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/33 file after the snapshot 2023-10-10T18:48:19.120 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/34 file after the snapshot 2023-10-10T18:48:19.121 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/35 file after the snapshot 2023-10-10T18:48:19.122 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/36 file after the snapshot 2023-10-10T18:48:19.123 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/37 file after the snapshot 2023-10-10T18:48:19.124 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/38 file after the snapshot 2023-10-10T18:48:19.125 INFO:tasks.workunit.client.0.smithi175.stdout: create 232/39 file after the snapshot 2023-10-10T18:48:19.126 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/.snap/snap-subdir-test 2023-10-10T18:48:19.152 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/30 file after the snapshot 2023-10-10T18:48:19.153 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/31 file after the snapshot 2023-10-10T18:48:19.154 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/32 file after the snapshot 2023-10-10T18:48:19.155 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/33 file after the snapshot 2023-10-10T18:48:19.155 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/34 file after the snapshot 2023-10-10T18:48:19.156 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/35 file after the snapshot 2023-10-10T18:48:19.157 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/36 file after the snapshot 2023-10-10T18:48:19.158 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/37 file after the snapshot 2023-10-10T18:48:19.159 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/38 file after the snapshot 2023-10-10T18:48:19.160 INFO:tasks.workunit.client.0.smithi175.stdout: create 233/39 file after the snapshot 2023-10-10T18:48:19.161 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/.snap/snap-subdir-test 2023-10-10T18:48:19.189 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/30 file after the snapshot 2023-10-10T18:48:19.190 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/31 file after the snapshot 2023-10-10T18:48:19.190 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/32 file after the snapshot 2023-10-10T18:48:19.191 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/33 file after the snapshot 2023-10-10T18:48:19.192 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/34 file after the snapshot 2023-10-10T18:48:19.192 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/35 file after the snapshot 2023-10-10T18:48:19.193 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/36 file after the snapshot 2023-10-10T18:48:19.193 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/37 file after the snapshot 2023-10-10T18:48:19.194 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/38 file after the snapshot 2023-10-10T18:48:19.195 INFO:tasks.workunit.client.0.smithi175.stdout: create 234/39 file after the snapshot 2023-10-10T18:48:19.195 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/.snap/snap-subdir-test 2023-10-10T18:48:19.218 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/30 file after the snapshot 2023-10-10T18:48:19.219 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/31 file after the snapshot 2023-10-10T18:48:19.220 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/32 file after the snapshot 2023-10-10T18:48:19.220 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/33 file after the snapshot 2023-10-10T18:48:19.221 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/34 file after the snapshot 2023-10-10T18:48:19.222 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/35 file after the snapshot 2023-10-10T18:48:19.222 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/36 file after the snapshot 2023-10-10T18:48:19.223 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/37 file after the snapshot 2023-10-10T18:48:19.223 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/38 file after the snapshot 2023-10-10T18:48:19.224 INFO:tasks.workunit.client.0.smithi175.stdout: create 235/39 file after the snapshot 2023-10-10T18:48:19.225 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/.snap/snap-subdir-test 2023-10-10T18:48:19.250 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/30 file after the snapshot 2023-10-10T18:48:19.251 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/31 file after the snapshot 2023-10-10T18:48:19.252 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/32 file after the snapshot 2023-10-10T18:48:19.252 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/33 file after the snapshot 2023-10-10T18:48:19.253 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/34 file after the snapshot 2023-10-10T18:48:19.253 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/35 file after the snapshot 2023-10-10T18:48:19.254 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/36 file after the snapshot 2023-10-10T18:48:19.255 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/37 file after the snapshot 2023-10-10T18:48:19.255 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/38 file after the snapshot 2023-10-10T18:48:19.256 INFO:tasks.workunit.client.0.smithi175.stdout: create 236/39 file after the snapshot 2023-10-10T18:48:19.257 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/.snap/snap-subdir-test 2023-10-10T18:48:19.274 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/30 file after the snapshot 2023-10-10T18:48:19.274 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/31 file after the snapshot 2023-10-10T18:48:19.275 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/32 file after the snapshot 2023-10-10T18:48:19.276 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/33 file after the snapshot 2023-10-10T18:48:19.276 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/34 file after the snapshot 2023-10-10T18:48:19.277 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/35 file after the snapshot 2023-10-10T18:48:19.278 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/36 file after the snapshot 2023-10-10T18:48:19.278 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/37 file after the snapshot 2023-10-10T18:48:19.279 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/38 file after the snapshot 2023-10-10T18:48:19.280 INFO:tasks.workunit.client.0.smithi175.stdout: create 237/39 file after the snapshot 2023-10-10T18:48:19.280 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/.snap/snap-subdir-test 2023-10-10T18:48:19.301 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/30 file after the snapshot 2023-10-10T18:48:19.303 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/31 file after the snapshot 2023-10-10T18:48:19.304 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/32 file after the snapshot 2023-10-10T18:48:19.305 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/33 file after the snapshot 2023-10-10T18:48:19.305 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/34 file after the snapshot 2023-10-10T18:48:19.306 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/35 file after the snapshot 2023-10-10T18:48:19.307 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/36 file after the snapshot 2023-10-10T18:48:19.535 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/37 file after the snapshot 2023-10-10T18:48:19.536 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/38 file after the snapshot 2023-10-10T18:48:19.537 INFO:tasks.workunit.client.0.smithi175.stdout: create 238/39 file after the snapshot 2023-10-10T18:48:19.538 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/.snap/snap-subdir-test 2023-10-10T18:48:19.557 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/30 file after the snapshot 2023-10-10T18:48:19.558 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/31 file after the snapshot 2023-10-10T18:48:19.559 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/32 file after the snapshot 2023-10-10T18:48:19.560 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/33 file after the snapshot 2023-10-10T18:48:19.560 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/34 file after the snapshot 2023-10-10T18:48:19.561 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/35 file after the snapshot 2023-10-10T18:48:19.562 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/36 file after the snapshot 2023-10-10T18:48:19.563 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/37 file after the snapshot 2023-10-10T18:48:19.563 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/38 file after the snapshot 2023-10-10T18:48:19.564 INFO:tasks.workunit.client.0.smithi175.stdout: create 239/39 file after the snapshot 2023-10-10T18:48:19.564 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/.snap/snap-subdir-test 2023-10-10T18:48:19.583 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/30 file after the snapshot 2023-10-10T18:48:19.584 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/31 file after the snapshot 2023-10-10T18:48:19.585 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/32 file after the snapshot 2023-10-10T18:48:19.585 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/33 file after the snapshot 2023-10-10T18:48:19.586 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/34 file after the snapshot 2023-10-10T18:48:19.586 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/35 file after the snapshot 2023-10-10T18:48:19.587 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/36 file after the snapshot 2023-10-10T18:48:19.588 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/37 file after the snapshot 2023-10-10T18:48:19.588 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/38 file after the snapshot 2023-10-10T18:48:19.589 INFO:tasks.workunit.client.0.smithi175.stdout: create 240/39 file after the snapshot 2023-10-10T18:48:19.590 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/.snap/snap-subdir-test 2023-10-10T18:48:19.608 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/30 file after the snapshot 2023-10-10T18:48:19.609 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/31 file after the snapshot 2023-10-10T18:48:19.610 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/32 file after the snapshot 2023-10-10T18:48:19.611 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/33 file after the snapshot 2023-10-10T18:48:19.611 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/34 file after the snapshot 2023-10-10T18:48:19.612 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/35 file after the snapshot 2023-10-10T18:48:19.612 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/36 file after the snapshot 2023-10-10T18:48:19.613 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/37 file after the snapshot 2023-10-10T18:48:19.614 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/38 file after the snapshot 2023-10-10T18:48:19.614 INFO:tasks.workunit.client.0.smithi175.stdout: create 241/39 file after the snapshot 2023-10-10T18:48:19.615 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/.snap/snap-subdir-test 2023-10-10T18:48:19.634 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/30 file after the snapshot 2023-10-10T18:48:19.635 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/31 file after the snapshot 2023-10-10T18:48:19.636 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/32 file after the snapshot 2023-10-10T18:48:19.636 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/33 file after the snapshot 2023-10-10T18:48:19.637 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/34 file after the snapshot 2023-10-10T18:48:19.637 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/35 file after the snapshot 2023-10-10T18:48:19.638 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/36 file after the snapshot 2023-10-10T18:48:19.639 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/37 file after the snapshot 2023-10-10T18:48:19.639 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/38 file after the snapshot 2023-10-10T18:48:19.640 INFO:tasks.workunit.client.0.smithi175.stdout: create 242/39 file after the snapshot 2023-10-10T18:48:19.640 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/.snap/snap-subdir-test 2023-10-10T18:48:19.660 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/30 file after the snapshot 2023-10-10T18:48:19.661 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/31 file after the snapshot 2023-10-10T18:48:19.662 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/32 file after the snapshot 2023-10-10T18:48:19.663 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/33 file after the snapshot 2023-10-10T18:48:19.663 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/34 file after the snapshot 2023-10-10T18:48:19.664 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/35 file after the snapshot 2023-10-10T18:48:19.664 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/36 file after the snapshot 2023-10-10T18:48:19.665 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/37 file after the snapshot 2023-10-10T18:48:19.665 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/38 file after the snapshot 2023-10-10T18:48:19.666 INFO:tasks.workunit.client.0.smithi175.stdout: create 243/39 file after the snapshot 2023-10-10T18:48:19.666 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/.snap/snap-subdir-test 2023-10-10T18:48:19.683 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/30 file after the snapshot 2023-10-10T18:48:19.685 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/31 file after the snapshot 2023-10-10T18:48:19.685 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/32 file after the snapshot 2023-10-10T18:48:19.686 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/33 file after the snapshot 2023-10-10T18:48:19.687 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/34 file after the snapshot 2023-10-10T18:48:19.688 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/35 file after the snapshot 2023-10-10T18:48:19.689 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/36 file after the snapshot 2023-10-10T18:48:19.690 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/37 file after the snapshot 2023-10-10T18:48:19.691 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/38 file after the snapshot 2023-10-10T18:48:19.692 INFO:tasks.workunit.client.0.smithi175.stdout: create 244/39 file after the snapshot 2023-10-10T18:48:19.693 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/.snap/snap-subdir-test 2023-10-10T18:48:19.712 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/30 file after the snapshot 2023-10-10T18:48:19.713 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/31 file after the snapshot 2023-10-10T18:48:19.713 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/32 file after the snapshot 2023-10-10T18:48:19.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/33 file after the snapshot 2023-10-10T18:48:19.715 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/34 file after the snapshot 2023-10-10T18:48:19.715 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/35 file after the snapshot 2023-10-10T18:48:19.716 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/36 file after the snapshot 2023-10-10T18:48:19.982 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/37 file after the snapshot 2023-10-10T18:48:19.982 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/38 file after the snapshot 2023-10-10T18:48:19.983 INFO:tasks.workunit.client.0.smithi175.stdout: create 245/39 file after the snapshot 2023-10-10T18:48:19.983 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/.snap/snap-subdir-test 2023-10-10T18:48:20.005 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/30 file after the snapshot 2023-10-10T18:48:20.006 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/31 file after the snapshot 2023-10-10T18:48:20.007 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/32 file after the snapshot 2023-10-10T18:48:20.008 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/33 file after the snapshot 2023-10-10T18:48:20.010 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/34 file after the snapshot 2023-10-10T18:48:20.011 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/35 file after the snapshot 2023-10-10T18:48:20.363 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/36 file after the snapshot 2023-10-10T18:48:20.364 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/37 file after the snapshot 2023-10-10T18:48:20.365 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/38 file after the snapshot 2023-10-10T18:48:20.366 INFO:tasks.workunit.client.0.smithi175.stdout: create 246/39 file after the snapshot 2023-10-10T18:48:20.367 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/.snap/snap-subdir-test 2023-10-10T18:48:20.387 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/30 file after the snapshot 2023-10-10T18:48:20.388 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/31 file after the snapshot 2023-10-10T18:48:20.389 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/32 file after the snapshot 2023-10-10T18:48:20.390 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/33 file after the snapshot 2023-10-10T18:48:20.391 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/34 file after the snapshot 2023-10-10T18:48:20.392 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/35 file after the snapshot 2023-10-10T18:48:20.393 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/36 file after the snapshot 2023-10-10T18:48:20.394 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/37 file after the snapshot 2023-10-10T18:48:20.655 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/38 file after the snapshot 2023-10-10T18:48:20.656 INFO:tasks.workunit.client.0.smithi175.stdout: create 247/39 file after the snapshot 2023-10-10T18:48:20.656 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/.snap/snap-subdir-test 2023-10-10T18:48:20.676 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/30 file after the snapshot 2023-10-10T18:48:20.677 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/31 file after the snapshot 2023-10-10T18:48:20.678 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/32 file after the snapshot 2023-10-10T18:48:20.679 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/33 file after the snapshot 2023-10-10T18:48:20.680 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/34 file after the snapshot 2023-10-10T18:48:20.681 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/35 file after the snapshot 2023-10-10T18:48:20.682 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/36 file after the snapshot 2023-10-10T18:48:20.683 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/37 file after the snapshot 2023-10-10T18:48:20.684 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/38 file after the snapshot 2023-10-10T18:48:20.685 INFO:tasks.workunit.client.0.smithi175.stdout: create 248/39 file after the snapshot 2023-10-10T18:48:20.686 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/.snap/snap-subdir-test 2023-10-10T18:48:20.708 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/30 file after the snapshot 2023-10-10T18:48:20.709 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/31 file after the snapshot 2023-10-10T18:48:20.710 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/32 file after the snapshot 2023-10-10T18:48:20.712 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/33 file after the snapshot 2023-10-10T18:48:20.713 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/34 file after the snapshot 2023-10-10T18:48:20.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/35 file after the snapshot 2023-10-10T18:48:20.714 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/36 file after the snapshot 2023-10-10T18:48:20.715 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/37 file after the snapshot 2023-10-10T18:48:20.716 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/38 file after the snapshot 2023-10-10T18:48:20.717 INFO:tasks.workunit.client.0.smithi175.stdout: create 249/39 file after the snapshot 2023-10-10T18:48:20.717 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/.snap/snap-subdir-test 2023-10-10T18:48:20.738 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/30 file after the snapshot 2023-10-10T18:48:20.739 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/31 file after the snapshot 2023-10-10T18:48:20.740 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/32 file after the snapshot 2023-10-10T18:48:20.741 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/33 file after the snapshot 2023-10-10T18:48:20.742 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/34 file after the snapshot 2023-10-10T18:48:20.743 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/35 file after the snapshot 2023-10-10T18:48:20.744 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/36 file after the snapshot 2023-10-10T18:48:20.745 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/37 file after the snapshot 2023-10-10T18:48:20.746 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/38 file after the snapshot 2023-10-10T18:48:20.747 INFO:tasks.workunit.client.0.smithi175.stdout: create 250/39 file after the snapshot 2023-10-10T18:48:20.748 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/.snap/snap-subdir-test 2023-10-10T18:48:20.798 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-10T18:48:20.799 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-10T18:48:20.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-10T18:48:20.803 INFO:tasks.ceph.mds.b:Started 2023-10-10T18:48:20.804 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:48:20.881 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-10T18:48:20.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.917+0000 7f8a7645a700 1 Processor -- start 2023-10-10T18:48:20.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.917+0000 7f8a7645a700 1 -- start start 2023-10-10T18:48:20.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a7645a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 0x7f8a70155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:20.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a7645a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8a700aff60 0x7f8a700b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:20.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a7645a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a701574b0 con 0x7f8a700b1fc0 2023-10-10T18:48:20.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a7645a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a70150770 con 0x7f8a700aff60 2023-10-10T18:48:20.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a7645a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700afad0 con 0x7f8a700b2390 2023-10-10T18:48:20.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a749f7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8a700aff60 0x7f8a700b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:20.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a6ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8a700b1fc0 0x7f8a70068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41282/0 (socket says 172.21.15.175:41282) 2023-10-10T18:48:20.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a749f7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8a700aff60 0x7f8a700b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35304/0 (socket says 172.21.15.175:35304) 2023-10-10T18:48:20.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a6ffff700 1 -- 172.21.15.175:0/3346774224 learned_addr learned my addr 172.21.15.175:0/3346774224 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:48:20.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.918+0000 7f8a6f7fe700 1 --2- 172.21.15.175:0/3346774224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 0x7f8a70155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:20.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a749f7700 1 -- 172.21.15.175:0/3346774224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 msgr2=0x7f8a70155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:20.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a749f7700 1 --2- 172.21.15.175:0/3346774224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 0x7f8a70155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:20.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a749f7700 1 -- 172.21.15.175:0/3346774224 >> v1:172.21.15.175:6789/0 conn(0x7f8a700b1fc0 legacy=0x7f8a70068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:20.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a749f7700 1 -- 172.21.15.175:0/3346774224 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700af4c0 con 0x7f8a700aff60 2023-10-10T18:48:20.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a6effd700 1 -- 172.21.15.175:0/3346774224 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873484952 0 0) 0x7f8a701574b0 con 0x7f8a700b1fc0 2023-10-10T18:48:20.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a749f7700 1 --2- 172.21.15.175:0/3346774224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a700b03e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009e90 tx=0x7f8a68009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d412c3c7e79f59d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:20.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a6effd700 1 -- 172.21.15.175:0/3346774224 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800e070 con 0x7f8a700aff60 2023-10-10T18:48:20.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a6effd700 1 -- 172.21.15.175:0/3346774224 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6800bc50 con 0x7f8a700aff60 2023-10-10T18:48:20.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a6effd700 1 -- 172.21.15.175:0/3346774224 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68012430 con 0x7f8a700aff60 2023-10-10T18:48:20.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a7645a700 1 -- 172.21.15.175:0/3346774224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 msgr2=0x7f8a700b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:20.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3346774224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a700b03e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009e90 tx=0x7f8a68009780 comp rx=0 tx=0).stop 2023-10-10T18:48:20.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a7645a700 1 -- 172.21.15.175:0/3346774224 shutdown_connections 2023-10-10T18:48:20.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3346774224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a700b03e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:20.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3346774224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 0x7f8a70155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:20.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a7645a700 1 -- 172.21.15.175:0/3346774224 >> 172.21.15.175:0/3346774224 conn(0x7f8a70063230 msgr2=0x7f8a70063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:20.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a7645a700 1 -- 172.21.15.175:0/3346774224 shutdown_connections 2023-10-10T18:48:20.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.919+0000 7f8a7645a700 1 -- 172.21.15.175:0/3346774224 wait complete. 2023-10-10T18:48:20.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.920+0000 7f8a7645a700 1 Processor -- start 2023-10-10T18:48:20.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a7645a700 1 -- start start 2023-10-10T18:48:20.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a7645a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a70159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:20.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a7645a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a700b1fc0 0x7f8a700c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:20.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a7645a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 0x7f8a700c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:20.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a7645a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700c49f0 con 0x7f8a700aff60 2023-10-10T18:48:20.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a7645a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700bfe60 con 0x7f8a700b1fc0 2023-10-10T18:48:20.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a7645a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700c0090 con 0x7f8a700b2390 2023-10-10T18:48:20.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a6ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a70159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:20.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.921+0000 7f8a6ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a70159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35320/0 (socket says 172.21.15.175:35320) 2023-10-10T18:48:20.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a6ffff700 1 -- 172.21.15.175:0/3232977531 learned_addr learned my addr 172.21.15.175:0/3232977531 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:20.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a749f7700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 0x7f8a700c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:20.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a6f7fe700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a700b1fc0 0x7f8a700c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:20.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a6ffff700 1 -- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 msgr2=0x7f8a700c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:20.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a6ffff700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 0x7f8a700c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:20.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a6ffff700 1 -- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a700b1fc0 msgr2=0x7f8a700c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:20.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a6ffff700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a700b1fc0 0x7f8a700c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:20.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a6ffff700 1 -- 172.21.15.175:0/3232977531 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a68009a20 con 0x7f8a700aff60 2023-10-10T18:48:20.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a6ffff700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a70159b40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000c810 tx=0x7f8a60007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:20.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a57fff700 1 -- 172.21.15.175:0/3232977531 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60005b30 con 0x7f8a700aff60 2023-10-10T18:48:20.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a57fff700 1 -- 172.21.15.175:0/3232977531 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a60005c90 con 0x7f8a700aff60 2023-10-10T18:48:20.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a700c3c40 con 0x7f8a700aff60 2023-10-10T18:48:20.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a57fff700 1 -- 172.21.15.175:0/3232977531 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60007300 con 0x7f8a700aff60 2023-10-10T18:48:20.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.922+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a700cc990 con 0x7f8a700aff60 2023-10-10T18:48:20.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.923+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a70071de0 con 0x7f8a700aff60 2023-10-10T18:48:20.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.923+0000 7f8a57fff700 1 -- 172.21.15.175:0/3232977531 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8a6001b3b0 con 0x7f8a700aff60 2023-10-10T18:48:20.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.924+0000 7f8a57fff700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8a5c066f60 0x7f8a5c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:20.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.924+0000 7f8a57fff700 1 -- 172.21.15.175:0/3232977531 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f8a60083ba0 con 0x7f8a700aff60 2023-10-10T18:48:20.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.924+0000 7f8a6f7fe700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8a5c066f60 0x7f8a5c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:20.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.924+0000 7f8a6f7fe700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8a5c066f60 0x7f8a5c069410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a70153ba0 tx=0x7f8a64008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:20.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:20.928+0000 7f8a57fff700 1 -- 172.21.15.175:0/3232977531 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a60050d40 con 0x7f8a700aff60 2023-10-10T18:48:20.995 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/30 file after the snapshot 2023-10-10T18:48:20.996 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/31 file after the snapshot 2023-10-10T18:48:20.997 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/32 file after the snapshot 2023-10-10T18:48:21.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.157+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a700c5320 con 0x7f8a700aff60 2023-10-10T18:48:21.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.158+0000 7f8a57fff700 1 -- 172.21.15.175:0/3232977531 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5426 (secure 0 0 0) 0x7f8a60017020 con 0x7f8a700aff60 2023-10-10T18:48:21.159 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:48:21.159 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:48:13.636752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:48:21.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8a5c066f60 msgr2=0x7f8a5c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:21.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8a5c066f60 0x7f8a5c069410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a70153ba0 tx=0x7f8a64008040 comp rx=0 tx=0).stop 2023-10-10T18:48:21.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 msgr2=0x7f8a70159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:21.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a70159b40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000c810 tx=0x7f8a60007df0 comp rx=0 tx=0).stop 2023-10-10T18:48:21.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 shutdown_connections 2023-10-10T18:48:21.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8a5c066f60 0x7f8a5c069410 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:21.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a700b2390 0x7f8a700c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:21.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8a700aff60 0x7f8a70159b40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:21.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 --2- 172.21.15.175:0/3232977531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a700b1fc0 0x7f8a700c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:21.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 >> 172.21.15.175:0/3232977531 conn(0x7f8a70063230 msgr2=0x7f8a70152380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:21.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 shutdown_connections 2023-10-10T18:48:21.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:21.160+0000 7f8a7645a700 1 -- 172.21.15.175:0/3232977531 wait complete. 2023-10-10T18:48:21.164 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 28 2023-10-10T18:48:21.173 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-10T18:48:21.833 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/33 file after the snapshot 2023-10-10T18:48:21.834 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/34 file after the snapshot 2023-10-10T18:48:21.834 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/35 file after the snapshot 2023-10-10T18:48:21.835 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/36 file after the snapshot 2023-10-10T18:48:21.836 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/37 file after the snapshot 2023-10-10T18:48:21.836 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/38 file after the snapshot 2023-10-10T18:48:21.837 INFO:tasks.workunit.client.0.smithi175.stdout: create 251/39 file after the snapshot 2023-10-10T18:48:21.838 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/.snap/snap-subdir-test 2023-10-10T18:48:21.858 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/30 file after the snapshot 2023-10-10T18:48:21.859 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/31 file after the snapshot 2023-10-10T18:48:21.860 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/32 file after the snapshot 2023-10-10T18:48:21.861 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/33 file after the snapshot 2023-10-10T18:48:21.861 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/34 file after the snapshot 2023-10-10T18:48:21.862 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/35 file after the snapshot 2023-10-10T18:48:21.863 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/36 file after the snapshot 2023-10-10T18:48:21.863 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/37 file after the snapshot 2023-10-10T18:48:21.864 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/38 file after the snapshot 2023-10-10T18:48:21.865 INFO:tasks.workunit.client.0.smithi175.stdout: create 252/39 file after the snapshot 2023-10-10T18:48:21.865 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/.snap/snap-subdir-test 2023-10-10T18:48:21.886 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/30 file after the snapshot 2023-10-10T18:48:21.888 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/31 file after the snapshot 2023-10-10T18:48:21.889 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/32 file after the snapshot 2023-10-10T18:48:21.890 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/33 file after the snapshot 2023-10-10T18:48:21.891 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/34 file after the snapshot 2023-10-10T18:48:21.892 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/35 file after the snapshot 2023-10-10T18:48:21.893 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/36 file after the snapshot 2023-10-10T18:48:21.894 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/37 file after the snapshot 2023-10-10T18:48:21.895 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/38 file after the snapshot 2023-10-10T18:48:21.896 INFO:tasks.workunit.client.0.smithi175.stdout: create 253/39 file after the snapshot 2023-10-10T18:48:21.897 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/.snap/snap-subdir-test 2023-10-10T18:48:21.916 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/30 file after the snapshot 2023-10-10T18:48:21.917 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/31 file after the snapshot 2023-10-10T18:48:21.918 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/32 file after the snapshot 2023-10-10T18:48:21.918 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/33 file after the snapshot 2023-10-10T18:48:21.919 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/34 file after the snapshot 2023-10-10T18:48:21.920 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/35 file after the snapshot 2023-10-10T18:48:21.920 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/36 file after the snapshot 2023-10-10T18:48:21.921 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/37 file after the snapshot 2023-10-10T18:48:21.922 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/38 file after the snapshot 2023-10-10T18:48:21.922 INFO:tasks.workunit.client.0.smithi175.stdout: create 254/39 file after the snapshot 2023-10-10T18:48:21.923 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/.snap/snap-subdir-test 2023-10-10T18:48:21.944 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/30 file after the snapshot 2023-10-10T18:48:21.945 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/31 file after the snapshot 2023-10-10T18:48:21.946 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/32 file after the snapshot 2023-10-10T18:48:21.946 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/33 file after the snapshot 2023-10-10T18:48:21.947 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/34 file after the snapshot 2023-10-10T18:48:21.947 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/35 file after the snapshot 2023-10-10T18:48:21.948 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/36 file after the snapshot 2023-10-10T18:48:21.949 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/37 file after the snapshot 2023-10-10T18:48:21.949 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/38 file after the snapshot 2023-10-10T18:48:21.950 INFO:tasks.workunit.client.0.smithi175.stdout: create 255/39 file after the snapshot 2023-10-10T18:48:21.951 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/.snap/snap-subdir-test 2023-10-10T18:48:22.961 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/30 file after the snapshot 2023-10-10T18:48:22.962 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/31 file after the snapshot 2023-10-10T18:48:22.962 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/32 file after the snapshot 2023-10-10T18:48:22.963 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/33 file after the snapshot 2023-10-10T18:48:22.964 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/34 file after the snapshot 2023-10-10T18:48:22.965 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/35 file after the snapshot 2023-10-10T18:48:22.965 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/36 file after the snapshot 2023-10-10T18:48:22.966 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/37 file after the snapshot 2023-10-10T18:48:22.967 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/38 file after the snapshot 2023-10-10T18:48:22.968 INFO:tasks.workunit.client.0.smithi175.stdout: create 256/39 file after the snapshot 2023-10-10T18:48:22.969 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/.snap/snap-subdir-test 2023-10-10T18:48:22.990 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/30 file after the snapshot 2023-10-10T18:48:22.991 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/31 file after the snapshot 2023-10-10T18:48:22.992 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/32 file after the snapshot 2023-10-10T18:48:22.994 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/33 file after the snapshot 2023-10-10T18:48:22.994 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/34 file after the snapshot 2023-10-10T18:48:22.995 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/35 file after the snapshot 2023-10-10T18:48:22.995 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/36 file after the snapshot 2023-10-10T18:48:22.996 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/37 file after the snapshot 2023-10-10T18:48:22.996 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/38 file after the snapshot 2023-10-10T18:48:22.997 INFO:tasks.workunit.client.0.smithi175.stdout: create 257/39 file after the snapshot 2023-10-10T18:48:22.998 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/.snap/snap-subdir-test 2023-10-10T18:48:23.019 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/30 file after the snapshot 2023-10-10T18:48:23.020 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/31 file after the snapshot 2023-10-10T18:48:23.021 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/32 file after the snapshot 2023-10-10T18:48:23.022 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/33 file after the snapshot 2023-10-10T18:48:23.023 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/34 file after the snapshot 2023-10-10T18:48:23.024 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/35 file after the snapshot 2023-10-10T18:48:23.025 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/36 file after the snapshot 2023-10-10T18:48:23.026 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/37 file after the snapshot 2023-10-10T18:48:23.027 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/38 file after the snapshot 2023-10-10T18:48:23.028 INFO:tasks.workunit.client.0.smithi175.stdout: create 258/39 file after the snapshot 2023-10-10T18:48:23.029 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/.snap/snap-subdir-test 2023-10-10T18:48:23.173 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:48:23.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.291+0000 7f8f1b327700 1 Processor -- start 2023-10-10T18:48:23.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.291+0000 7f8f1b327700 1 -- start start 2023-10-10T18:48:23.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f1b327700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f140b1fc0 0x7f8f14155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f1b327700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8f140b2390 0x7f8f14068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f1b327700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f141574b0 con 0x7f8f14068bd0 2023-10-10T18:48:23.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f1b327700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f14150770 con 0x7f8f140b2390 2023-10-10T18:48:23.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f1b327700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140afad0 con 0x7f8f140b1fc0 2023-10-10T18:48:23.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f198c4700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8f14068bd0 0x7f8f140aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41286/0 (socket says 172.21.15.175:41286) 2023-10-10T18:48:23.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f198c4700 1 -- 172.21.15.175:0/1361276391 learned_addr learned my addr 172.21.15.175:0/1361276391 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:48:23.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f190c3700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f140b1fc0 0x7f8f14155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872672225 0 0) 0x7f8f141574b0 con 0x7f8f14068bd0 2023-10-10T18:48:23.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f188c2700 1 --2- 172.21.15.175:0/1361276391 >> v2:172.21.15.191:3300/0 conn(0x7f8f140b2390 0x7f8f14068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.292+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f00003570 con 0x7f8f14068bd0 2023-10-10T18:48:23.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1271752944 0 0) 0x7f8f00003570 con 0x7f8f14068bd0 2023-10-10T18:48:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f140b1fc0 msgr2=0x7f8f14155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f140b1fc0 0x7f8f14155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 >> v2:172.21.15.191:3300/0 conn(0x7f8f140b2390 msgr2=0x7f8f14068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 --2- 172.21.15.175:0/1361276391 >> v2:172.21.15.191:3300/0 conn(0x7f8f140b2390 0x7f8f14068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f140af4c0 con 0x7f8f14068bd0 2023-10-10T18:48:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f8f0c001470 con 0x7f8f14068bd0 2023-10-10T18:48:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 >> v1:172.21.15.175:6789/0 conn(0x7f8f14068bd0 legacy=0x7f8f140aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f00003a60 0x7f8f00003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f00004810 0x7f8f00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f140b1fc0 0x7f8f000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f14150770 con 0x7f8f00003a60 2023-10-10T18:48:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140afad0 con 0x7f8f00004810 2023-10-10T18:48:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0000ba60 con 0x7f8f140b1fc0 2023-10-10T18:48:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f190c3700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f00003a60 0x7f8f00003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f188c2700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f00004810 0x7f8f00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f198c4700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f140b1fc0 0x7f8f000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f190c3700 1 -- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f140b1fc0 msgr2=0x7f8f000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f190c3700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f140b1fc0 0x7f8f000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f190c3700 1 -- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f00004810 msgr2=0x7f8f00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:48:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f190c3700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f00004810 0x7f8f00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f190c3700 1 -- 172.21.15.175:0/1361276391 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f00003570 con 0x7f8f00003a60 2023-10-10T18:48:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f190c3700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f00003a60 0x7f8f00003e70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000e240 tx=0x7f8f1000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785488c40a6d9854 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1000eaf0 con 0x7f8f00003a60 2023-10-10T18:48:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1000f040 con 0x7f8f00003a60 2023-10-10T18:48:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.293+0000 7f8f0bfff700 1 -- 172.21.15.175:0/1361276391 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1000eea0 con 0x7f8f00003a60 2023-10-10T18:48:23.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.294+0000 7f8f1b327700 1 -- 172.21.15.175:0/1361276391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f00003a60 msgr2=0x7f8f00003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:23.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.294+0000 7f8f1b327700 1 --2- 172.21.15.175:0/1361276391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f00003a60 0x7f8f00003e70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000e240 tx=0x7f8f1000b040 comp rx=0 tx=0).stop 2023-10-10T18:48:23.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.294+0000 7f8f198c4700 1 -- 172.21.15.175:0/1361276391 reap_dead start 2023-10-10T18:48:23.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.294+0000 7f8f1b327700 1 -- 172.21.15.175:0/1361276391 shutdown_connections 2023-10-10T18:48:23.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.294+0000 7f8f1b327700 1 -- 172.21.15.175:0/1361276391 >> 172.21.15.175:0/1361276391 conn(0x7f8f14063230 msgr2=0x7f8f14063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:23.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.294+0000 7f8f1b327700 1 -- 172.21.15.175:0/1361276391 shutdown_connections 2023-10-10T18:48:23.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.294+0000 7f8f1b327700 1 -- 172.21.15.175:0/1361276391 wait complete. 2023-10-10T18:48:23.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.295+0000 7f8f1b327700 1 Processor -- start 2023-10-10T18:48:23.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.295+0000 7f8f1b327700 1 -- start start 2023-10-10T18:48:23.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f1b327700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f00003a60 0x7f8f140cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f1b327700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f140cd170 0x7f8f140c1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f1b327700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f140caac0 0x7f8f140cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f1b327700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140ca630 con 0x7f8f140cd170 2023-10-10T18:48:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f1b327700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140c17d0 con 0x7f8f140caac0 2023-10-10T18:48:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f1b327700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1416a0a0 con 0x7f8f00003a60 2023-10-10T18:48:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f188c2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f140cd170 0x7f8f140c1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f188c2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f140cd170 0x7f8f140c1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35344/0 (socket says 172.21.15.175:35344) 2023-10-10T18:48:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f188c2700 1 -- 172.21.15.175:0/1565107083 learned_addr learned my addr 172.21.15.175:0/1565107083 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:48:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f190c3700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f00003a60 0x7f8f140cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f198c4700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f140caac0 0x7f8f140cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f188c2700 1 -- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f00003a60 msgr2=0x7f8f140cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f188c2700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f00003a60 0x7f8f140cbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f188c2700 1 -- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f140caac0 msgr2=0x7f8f140cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f188c2700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f140caac0 0x7f8f140cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.296+0000 7f8f188c2700 1 -- 172.21.15.175:0/1565107083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f14159e50 con 0x7f8f140cd170 2023-10-10T18:48:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.297+0000 7f8f188c2700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f140cd170 0x7f8f140c1290 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8f04011d70 tx=0x7f8f040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.297+0000 7f8f097fa700 1 -- 172.21.15.175:0/1565107083 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0400fac0 con 0x7f8f140cd170 2023-10-10T18:48:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.297+0000 7f8f097fa700 1 -- 172.21.15.175:0/1565107083 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0400fc20 con 0x7f8f140cd170 2023-10-10T18:48:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.297+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1415a130 con 0x7f8f140cd170 2023-10-10T18:48:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.297+0000 7f8f097fa700 1 -- 172.21.15.175:0/1565107083 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04014440 con 0x7f8f140cd170 2023-10-10T18:48:23.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.297+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f14157180 con 0x7f8f140cd170 2023-10-10T18:48:23.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.298+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f14159cb0 con 0x7f8f140cd170 2023-10-10T18:48:23.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.298+0000 7f8f097fa700 1 -- 172.21.15.175:0/1565107083 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8f04014870 con 0x7f8f140cd170 2023-10-10T18:48:23.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.298+0000 7f8f097fa700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8ef8066f60 0x7f8ef8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:48:23.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.298+0000 7f8f097fa700 1 -- 172.21.15.175:0/1565107083 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f8f04018030 con 0x7f8f140cd170 2023-10-10T18:48:23.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.299+0000 7f8f190c3700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8ef8066f60 0x7f8ef8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:48:23.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.299+0000 7f8f190c3700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8ef8066f60 0x7f8ef8069410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009c30 tx=0x7f8f1001a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:48:23.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.302+0000 7f8f097fa700 1 -- 172.21.15.175:0/1565107083 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f14159cb0 con 0x7f8f140cd170 2023-10-10T18:48:23.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.536+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f140ca7e0 con 0x7f8f140cd170 2023-10-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.537+0000 7f8f097fa700 1 -- 172.21.15.175:0/1565107083 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6185 (secure 0 0 0) 0x7f8f140ca7e0 con 0x7f8f140cd170 2023-10-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:48:23.539 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:48:13.636752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:48:23.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.539+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8ef8066f60 msgr2=0x7f8ef8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:23.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.539+0000 7f8f1b327700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8ef8066f60 0x7f8ef8069410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009c30 tx=0x7f8f1001a000 comp rx=0 tx=0).stop 2023-10-10T18:48:23.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.539+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f140cd170 msgr2=0x7f8f140c1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:48:23.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.539+0000 7f8f1b327700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f140cd170 0x7f8f140c1290 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8f04011d70 tx=0x7f8f040091a0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.540+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 shutdown_connections 2023-10-10T18:48:23.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.540+0000 7f8f1b327700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8ef8066f60 0x7f8ef8069410 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.540+0000 7f8f1b327700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f140caac0 0x7f8f140cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.540+0000 7f8f1b327700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f00003a60 0x7f8f140cbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.540+0000 7f8f1b327700 1 --2- 172.21.15.175:0/1565107083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f140cd170 0x7f8f140c1290 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:48:23.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.540+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 >> 172.21.15.175:0/1565107083 conn(0x7f8f14063230 msgr2=0x7f8f1415c800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:48:23.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.540+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 shutdown_connections 2023-10-10T18:48:23.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:48:23.540+0000 7f8f1b327700 1 -- 172.21.15.175:0/1565107083 wait complete. 2023-10-10T18:48:23.543 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 29 2023-10-10T18:48:23.552 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-10T18:48:23.553 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2023-10-10T18:48:24.050 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/30 file after the snapshot 2023-10-10T18:48:24.053 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/31 file after the snapshot 2023-10-10T18:48:24.056 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/32 file after the snapshot 2023-10-10T18:48:24.063 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/33 file after the snapshot 2023-10-10T18:48:24.065 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/34 file after the snapshot 2023-10-10T18:48:24.069 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/35 file after the snapshot 2023-10-10T18:48:24.076 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/36 file after the snapshot 2023-10-10T18:48:24.077 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/37 file after the snapshot 2023-10-10T18:48:24.078 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/38 file after the snapshot 2023-10-10T18:48:24.079 INFO:tasks.workunit.client.0.smithi175.stdout: create 259/39 file after the snapshot 2023-10-10T18:48:24.081 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/.snap/snap-subdir-test 2023-10-10T18:48:25.466 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/30 file after the snapshot 2023-10-10T18:48:25.467 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/31 file after the snapshot 2023-10-10T18:48:25.468 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/32 file after the snapshot 2023-10-10T18:48:25.469 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/33 file after the snapshot 2023-10-10T18:48:25.470 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/34 file after the snapshot 2023-10-10T18:48:25.471 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/35 file after the snapshot 2023-10-10T18:48:25.472 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/36 file after the snapshot 2023-10-10T18:48:25.473 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/37 file after the snapshot 2023-10-10T18:48:25.474 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/38 file after the snapshot 2023-10-10T18:48:25.475 INFO:tasks.workunit.client.0.smithi175.stdout: create 260/39 file after the snapshot 2023-10-10T18:48:25.476 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/.snap/snap-subdir-test 2023-10-10T18:48:25.501 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/30 file after the snapshot 2023-10-10T18:48:25.502 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/31 file after the snapshot 2023-10-10T18:48:25.502 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/32 file after the snapshot 2023-10-10T18:48:25.898 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/33 file after the snapshot 2023-10-10T18:48:25.899 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/34 file after the snapshot 2023-10-10T18:48:25.900 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/35 file after the snapshot 2023-10-10T18:48:25.900 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/36 file after the snapshot 2023-10-10T18:48:25.901 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/37 file after the snapshot 2023-10-10T18:48:25.902 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/38 file after the snapshot 2023-10-10T18:48:25.902 INFO:tasks.workunit.client.0.smithi175.stdout: create 261/39 file after the snapshot 2023-10-10T18:48:25.903 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/.snap/snap-subdir-test 2023-10-10T18:48:25.931 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/30 file after the snapshot 2023-10-10T18:48:25.932 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/31 file after the snapshot 2023-10-10T18:48:25.934 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/32 file after the snapshot 2023-10-10T18:48:25.935 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/33 file after the snapshot 2023-10-10T18:48:25.936 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/34 file after the snapshot 2023-10-10T18:48:25.937 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/35 file after the snapshot 2023-10-10T18:48:25.938 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/36 file after the snapshot 2023-10-10T18:48:26.198 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/37 file after the snapshot 2023-10-10T18:48:26.199 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/38 file after the snapshot 2023-10-10T18:48:26.200 INFO:tasks.workunit.client.0.smithi175.stdout: create 262/39 file after the snapshot 2023-10-10T18:48:26.201 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/.snap/snap-subdir-test 2023-10-10T18:48:26.479 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/30 file after the snapshot 2023-10-10T18:48:26.480 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/31 file after the snapshot 2023-10-10T18:48:26.481 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/32 file after the snapshot 2023-10-10T18:48:26.481 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/33 file after the snapshot 2023-10-10T18:48:26.482 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/34 file after the snapshot 2023-10-10T18:48:26.483 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/35 file after the snapshot 2023-10-10T18:48:26.483 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/36 file after the snapshot 2023-10-10T18:48:26.484 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/37 file after the snapshot 2023-10-10T18:48:26.485 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/38 file after the snapshot 2023-10-10T18:48:26.485 INFO:tasks.workunit.client.0.smithi175.stdout: create 263/39 file after the snapshot 2023-10-10T18:48:26.486 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/.snap/snap-subdir-test 2023-10-10T18:48:26.513 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/30 file after the snapshot 2023-10-10T18:48:26.514 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/31 file after the snapshot 2023-10-10T18:48:26.515 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/32 file after the snapshot 2023-10-10T18:48:26.516 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/33 file after the snapshot 2023-10-10T18:48:26.516 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/34 file after the snapshot 2023-10-10T18:48:26.517 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/35 file after the snapshot 2023-10-10T18:48:26.518 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/36 file after the snapshot 2023-10-10T18:48:26.518 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/37 file after the snapshot 2023-10-10T18:48:26.519 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/38 file after the snapshot 2023-10-10T18:48:26.520 INFO:tasks.workunit.client.0.smithi175.stdout: create 264/39 file after the snapshot 2023-10-10T18:48:26.521 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/.snap/snap-subdir-test 2023-10-10T18:48:26.545 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/30 file after the snapshot 2023-10-10T18:48:26.545 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/31 file after the snapshot 2023-10-10T18:48:26.546 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/32 file after the snapshot 2023-10-10T18:48:26.547 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/33 file after the snapshot 2023-10-10T18:48:26.547 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/34 file after the snapshot 2023-10-10T18:48:26.548 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/35 file after the snapshot 2023-10-10T18:48:26.549 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/36 file after the snapshot 2023-10-10T18:48:26.549 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/37 file after the snapshot 2023-10-10T18:48:26.550 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/38 file after the snapshot 2023-10-10T18:48:26.551 INFO:tasks.workunit.client.0.smithi175.stdout: create 265/39 file after the snapshot 2023-10-10T18:48:26.551 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/.snap/snap-subdir-test 2023-10-10T18:48:26.580 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/30 file after the snapshot 2023-10-10T18:48:26.581 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/31 file after the snapshot 2023-10-10T18:48:26.582 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/32 file after the snapshot 2023-10-10T18:48:26.582 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/33 file after the snapshot 2023-10-10T18:48:26.583 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/34 file after the snapshot 2023-10-10T18:48:26.584 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/35 file after the snapshot 2023-10-10T18:48:26.584 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/36 file after the snapshot 2023-10-10T18:48:26.585 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/37 file after the snapshot 2023-10-10T18:48:26.585 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/38 file after the snapshot 2023-10-10T18:48:26.586 INFO:tasks.workunit.client.0.smithi175.stdout: create 266/39 file after the snapshot 2023-10-10T18:48:26.587 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/.snap/snap-subdir-test 2023-10-10T18:48:26.614 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/30 file after the snapshot 2023-10-10T18:48:26.878 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/31 file after the snapshot 2023-10-10T18:48:26.879 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/32 file after the snapshot 2023-10-10T18:48:26.880 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/33 file after the snapshot 2023-10-10T18:48:26.881 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/34 file after the snapshot 2023-10-10T18:48:26.882 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/35 file after the snapshot 2023-10-10T18:48:26.883 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/36 file after the snapshot 2023-10-10T18:48:26.884 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/37 file after the snapshot 2023-10-10T18:48:26.885 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/38 file after the snapshot 2023-10-10T18:48:26.886 INFO:tasks.workunit.client.0.smithi175.stdout: create 267/39 file after the snapshot 2023-10-10T18:48:26.887 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/.snap/snap-subdir-test 2023-10-10T18:48:26.914 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/30 file after the snapshot 2023-10-10T18:48:26.916 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/31 file after the snapshot 2023-10-10T18:48:26.917 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/32 file after the snapshot 2023-10-10T18:48:26.918 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/33 file after the snapshot 2023-10-10T18:48:26.919 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/34 file after the snapshot 2023-10-10T18:48:26.920 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/35 file after the snapshot 2023-10-10T18:48:26.921 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/36 file after the snapshot 2023-10-10T18:48:26.922 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/37 file after the snapshot 2023-10-10T18:48:26.923 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/38 file after the snapshot 2023-10-10T18:48:26.924 INFO:tasks.workunit.client.0.smithi175.stdout: create 268/39 file after the snapshot 2023-10-10T18:48:26.925 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/.snap/snap-subdir-test 2023-10-10T18:48:26.944 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/30 file after the snapshot 2023-10-10T18:48:26.945 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/31 file after the snapshot 2023-10-10T18:48:26.946 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/32 file after the snapshot 2023-10-10T18:48:26.947 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/33 file after the snapshot 2023-10-10T18:48:26.948 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/34 file after the snapshot 2023-10-10T18:48:26.949 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/35 file after the snapshot 2023-10-10T18:48:26.950 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/36 file after the snapshot 2023-10-10T18:48:26.951 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/37 file after the snapshot 2023-10-10T18:48:26.952 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/38 file after the snapshot 2023-10-10T18:48:26.953 INFO:tasks.workunit.client.0.smithi175.stdout: create 269/39 file after the snapshot 2023-10-10T18:48:26.954 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/.snap/snap-subdir-test 2023-10-10T18:48:26.974 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/30 file after the snapshot 2023-10-10T18:48:26.975 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/31 file after the snapshot 2023-10-10T18:48:26.976 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/32 file after the snapshot 2023-10-10T18:48:26.977 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/33 file after the snapshot 2023-10-10T18:48:26.978 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/34 file after the snapshot 2023-10-10T18:48:26.979 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/35 file after the snapshot 2023-10-10T18:48:26.980 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/36 file after the snapshot 2023-10-10T18:48:26.981 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/37 file after the snapshot 2023-10-10T18:48:26.982 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/38 file after the snapshot 2023-10-10T18:48:26.982 INFO:tasks.workunit.client.0.smithi175.stdout: create 270/39 file after the snapshot 2023-10-10T18:48:26.984 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/.snap/snap-subdir-test 2023-10-10T18:48:27.002 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/30 file after the snapshot 2023-10-10T18:48:27.003 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/31 file after the snapshot 2023-10-10T18:48:27.004 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/32 file after the snapshot 2023-10-10T18:48:27.005 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/33 file after the snapshot 2023-10-10T18:48:27.005 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/34 file after the snapshot 2023-10-10T18:48:27.006 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/35 file after the snapshot 2023-10-10T18:48:27.006 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/36 file after the snapshot 2023-10-10T18:48:27.007 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/37 file after the snapshot 2023-10-10T18:48:27.008 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/38 file after the snapshot 2023-10-10T18:48:27.008 INFO:tasks.workunit.client.0.smithi175.stdout: create 271/39 file after the snapshot 2023-10-10T18:48:27.010 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/.snap/snap-subdir-test 2023-10-10T18:48:27.028 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/30 file after the snapshot 2023-10-10T18:48:27.029 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/31 file after the snapshot 2023-10-10T18:48:27.029 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/32 file after the snapshot 2023-10-10T18:48:27.031 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/33 file after the snapshot 2023-10-10T18:48:27.031 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/34 file after the snapshot 2023-10-10T18:48:27.032 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/35 file after the snapshot 2023-10-10T18:48:27.032 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/36 file after the snapshot 2023-10-10T18:48:27.033 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/37 file after the snapshot 2023-10-10T18:48:27.034 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/38 file after the snapshot 2023-10-10T18:48:27.035 INFO:tasks.workunit.client.0.smithi175.stdout: create 272/39 file after the snapshot 2023-10-10T18:48:27.035 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/.snap/snap-subdir-test 2023-10-10T18:48:27.053 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/30 file after the snapshot 2023-10-10T18:48:27.054 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/31 file after the snapshot 2023-10-10T18:48:27.055 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/32 file after the snapshot 2023-10-10T18:48:27.055 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/33 file after the snapshot 2023-10-10T18:48:27.056 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/34 file after the snapshot 2023-10-10T18:48:27.057 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/35 file after the snapshot 2023-10-10T18:48:27.057 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/36 file after the snapshot 2023-10-10T18:48:27.058 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/37 file after the snapshot 2023-10-10T18:48:27.059 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/38 file after the snapshot 2023-10-10T18:48:27.059 INFO:tasks.workunit.client.0.smithi175.stdout: create 273/39 file after the snapshot 2023-10-10T18:48:27.060 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/.snap/snap-subdir-test 2023-10-10T18:48:27.081 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/30 file after the snapshot 2023-10-10T18:48:27.082 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/31 file after the snapshot 2023-10-10T18:48:27.083 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/32 file after the snapshot 2023-10-10T18:48:27.083 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/33 file after the snapshot 2023-10-10T18:48:27.084 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/34 file after the snapshot 2023-10-10T18:48:27.085 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/35 file after the snapshot 2023-10-10T18:48:27.085 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/36 file after the snapshot 2023-10-10T18:48:27.086 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/37 file after the snapshot 2023-10-10T18:48:27.087 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/38 file after the snapshot 2023-10-10T18:48:27.087 INFO:tasks.workunit.client.0.smithi175.stdout: create 274/39 file after the snapshot 2023-10-10T18:48:27.088 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/.snap/snap-subdir-test 2023-10-10T18:48:27.107 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/30 file after the snapshot 2023-10-10T18:48:27.108 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/31 file after the snapshot 2023-10-10T18:48:27.108 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/32 file after the snapshot 2023-10-10T18:48:27.109 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/33 file after the snapshot 2023-10-10T18:48:27.109 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/34 file after the snapshot 2023-10-10T18:48:27.110 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/35 file after the snapshot 2023-10-10T18:48:27.111 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/36 file after the snapshot 2023-10-10T18:48:27.111 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/37 file after the snapshot 2023-10-10T18:48:27.112 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/38 file after the snapshot 2023-10-10T18:48:27.113 INFO:tasks.workunit.client.0.smithi175.stdout: create 275/39 file after the snapshot 2023-10-10T18:48:27.113 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/.snap/snap-subdir-test 2023-10-10T18:48:27.132 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/30 file after the snapshot 2023-10-10T18:48:27.133 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/31 file after the snapshot 2023-10-10T18:48:27.133 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/32 file after the snapshot 2023-10-10T18:48:27.134 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/33 file after the snapshot 2023-10-10T18:48:27.134 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/34 file after the snapshot 2023-10-10T18:48:27.135 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/35 file after the snapshot 2023-10-10T18:48:27.136 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/36 file after the snapshot 2023-10-10T18:48:27.136 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/37 file after the snapshot 2023-10-10T18:48:27.137 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/38 file after the snapshot 2023-10-10T18:48:27.137 INFO:tasks.workunit.client.0.smithi175.stdout: create 276/39 file after the snapshot 2023-10-10T18:48:27.138 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/.snap/snap-subdir-test 2023-10-10T18:48:27.157 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/30 file after the snapshot 2023-10-10T18:48:27.158 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/31 file after the snapshot 2023-10-10T18:48:27.158 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/32 file after the snapshot 2023-10-10T18:48:27.159 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/33 file after the snapshot 2023-10-10T18:48:27.160 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/34 file after the snapshot 2023-10-10T18:48:27.160 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/35 file after the snapshot 2023-10-10T18:48:27.161 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/36 file after the snapshot 2023-10-10T18:48:27.161 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/37 file after the snapshot 2023-10-10T18:48:27.162 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/38 file after the snapshot 2023-10-10T18:48:27.163 INFO:tasks.workunit.client.0.smithi175.stdout: create 277/39 file after the snapshot 2023-10-10T18:48:27.163 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/.snap/snap-subdir-test 2023-10-10T18:48:27.182 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/30 file after the snapshot 2023-10-10T18:48:27.183 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/31 file after the snapshot 2023-10-10T18:48:27.184 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/32 file after the snapshot 2023-10-10T18:48:27.184 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/33 file after the snapshot 2023-10-10T18:48:27.185 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/34 file after the snapshot 2023-10-10T18:48:27.185 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/35 file after the snapshot 2023-10-10T18:48:27.186 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/36 file after the snapshot 2023-10-10T18:48:27.187 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/37 file after the snapshot 2023-10-10T18:48:27.187 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/38 file after the snapshot 2023-10-10T18:48:27.188 INFO:tasks.workunit.client.0.smithi175.stdout: create 278/39 file after the snapshot 2023-10-10T18:48:27.189 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/.snap/snap-subdir-test 2023-10-10T18:48:27.208 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/30 file after the snapshot 2023-10-10T18:48:27.208 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/31 file after the snapshot 2023-10-10T18:48:27.209 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/32 file after the snapshot 2023-10-10T18:48:27.210 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/33 file after the snapshot 2023-10-10T18:48:27.210 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/34 file after the snapshot 2023-10-10T18:48:27.211 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/35 file after the snapshot 2023-10-10T18:48:27.212 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/36 file after the snapshot 2023-10-10T18:48:27.212 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/37 file after the snapshot 2023-10-10T18:48:27.213 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/38 file after the snapshot 2023-10-10T18:48:27.214 INFO:tasks.workunit.client.0.smithi175.stdout: create 279/39 file after the snapshot 2023-10-10T18:48:27.214 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/.snap/snap-subdir-test 2023-10-10T18:48:27.234 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/30 file after the snapshot 2023-10-10T18:48:27.234 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/31 file after the snapshot 2023-10-10T18:48:27.235 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/32 file after the snapshot 2023-10-10T18:48:27.236 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/33 file after the snapshot 2023-10-10T18:48:27.236 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/34 file after the snapshot 2023-10-10T18:48:27.237 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/35 file after the snapshot 2023-10-10T18:48:27.237 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/36 file after the snapshot 2023-10-10T18:48:27.238 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/37 file after the snapshot 2023-10-10T18:48:27.239 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/38 file after the snapshot 2023-10-10T18:48:27.239 INFO:tasks.workunit.client.0.smithi175.stdout: create 280/39 file after the snapshot 2023-10-10T18:48:27.240 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/.snap/snap-subdir-test 2023-10-10T18:48:27.260 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/30 file after the snapshot 2023-10-10T18:48:27.262 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/31 file after the snapshot 2023-10-10T18:48:27.263 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/32 file after the snapshot 2023-10-10T18:48:27.263 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/33 file after the snapshot 2023-10-10T18:48:27.264 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/34 file after the snapshot 2023-10-10T18:48:27.265 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/35 file after the snapshot 2023-10-10T18:48:27.266 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/36 file after the snapshot 2023-10-10T18:48:27.267 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/37 file after the snapshot 2023-10-10T18:48:27.268 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/38 file after the snapshot 2023-10-10T18:48:27.269 INFO:tasks.workunit.client.0.smithi175.stdout: create 281/39 file after the snapshot 2023-10-10T18:48:27.270 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/.snap/snap-subdir-test 2023-10-10T18:48:27.720 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/30 file after the snapshot 2023-10-10T18:48:27.721 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/31 file after the snapshot 2023-10-10T18:48:27.721 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/32 file after the snapshot 2023-10-10T18:48:27.722 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/33 file after the snapshot 2023-10-10T18:48:27.723 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/34 file after the snapshot 2023-10-10T18:48:27.724 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/35 file after the snapshot 2023-10-10T18:48:27.724 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/36 file after the snapshot 2023-10-10T18:48:27.725 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/37 file after the snapshot 2023-10-10T18:48:27.726 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/38 file after the snapshot 2023-10-10T18:48:27.726 INFO:tasks.workunit.client.0.smithi175.stdout: create 282/39 file after the snapshot 2023-10-10T18:48:27.727 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/.snap/snap-subdir-test 2023-10-10T18:48:28.221 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/30 file after the snapshot 2023-10-10T18:48:28.223 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/31 file after the snapshot 2023-10-10T18:48:28.224 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/32 file after the snapshot 2023-10-10T18:48:28.225 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/33 file after the snapshot 2023-10-10T18:48:28.226 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/34 file after the snapshot 2023-10-10T18:48:28.227 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/35 file after the snapshot 2023-10-10T18:48:28.228 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/36 file after the snapshot 2023-10-10T18:48:28.229 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/37 file after the snapshot 2023-10-10T18:48:28.230 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/38 file after the snapshot 2023-10-10T18:48:28.231 INFO:tasks.workunit.client.0.smithi175.stdout: create 283/39 file after the snapshot 2023-10-10T18:48:28.232 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/.snap/snap-subdir-test 2023-10-10T18:48:28.254 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/30 file after the snapshot 2023-10-10T18:48:28.255 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/31 file after the snapshot 2023-10-10T18:48:28.255 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/32 file after the snapshot 2023-10-10T18:48:28.256 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/33 file after the snapshot 2023-10-10T18:48:28.257 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/34 file after the snapshot 2023-10-10T18:48:28.258 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/35 file after the snapshot 2023-10-10T18:48:28.258 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/36 file after the snapshot 2023-10-10T18:48:28.259 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/37 file after the snapshot 2023-10-10T18:48:28.260 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/38 file after the snapshot 2023-10-10T18:48:28.260 INFO:tasks.workunit.client.0.smithi175.stdout: create 284/39 file after the snapshot 2023-10-10T18:48:28.261 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/.snap/snap-subdir-test 2023-10-10T18:48:28.284 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/30 file after the snapshot 2023-10-10T18:48:28.285 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/31 file after the snapshot 2023-10-10T18:48:28.286 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/32 file after the snapshot 2023-10-10T18:48:28.287 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/33 file after the snapshot 2023-10-10T18:48:28.288 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/34 file after the snapshot 2023-10-10T18:48:28.289 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/35 file after the snapshot 2023-10-10T18:48:28.290 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/36 file after the snapshot 2023-10-10T18:48:28.291 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/37 file after the snapshot 2023-10-10T18:48:28.292 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/38 file after the snapshot 2023-10-10T18:48:28.293 INFO:tasks.workunit.client.0.smithi175.stdout: create 285/39 file after the snapshot 2023-10-10T18:48:28.294 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/.snap/snap-subdir-test 2023-10-10T18:48:28.314 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/30 file after the snapshot 2023-10-10T18:48:28.315 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/31 file after the snapshot 2023-10-10T18:48:28.316 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/32 file after the snapshot 2023-10-10T18:48:28.317 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/33 file after the snapshot 2023-10-10T18:48:28.317 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/34 file after the snapshot 2023-10-10T18:48:28.318 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/35 file after the snapshot 2023-10-10T18:48:28.319 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/36 file after the snapshot 2023-10-10T18:48:28.319 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/37 file after the snapshot 2023-10-10T18:48:28.320 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/38 file after the snapshot 2023-10-10T18:48:28.321 INFO:tasks.workunit.client.0.smithi175.stdout: create 286/39 file after the snapshot 2023-10-10T18:48:28.321 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/.snap/snap-subdir-test 2023-10-10T18:48:28.533 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/30 file after the snapshot 2023-10-10T18:48:28.534 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/31 file after the snapshot 2023-10-10T18:48:28.534 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/32 file after the snapshot 2023-10-10T18:48:28.535 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/33 file after the snapshot 2023-10-10T18:48:28.536 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/34 file after the snapshot 2023-10-10T18:48:28.536 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/35 file after the snapshot 2023-10-10T18:48:28.537 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/36 file after the snapshot 2023-10-10T18:48:28.538 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/37 file after the snapshot 2023-10-10T18:48:28.538 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/38 file after the snapshot 2023-10-10T18:48:28.539 INFO:tasks.workunit.client.0.smithi175.stdout: create 287/39 file after the snapshot 2023-10-10T18:48:28.540 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/.snap/snap-subdir-test 2023-10-10T18:48:28.722 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/30 file after the snapshot 2023-10-10T18:48:28.723 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/31 file after the snapshot 2023-10-10T18:48:28.724 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/32 file after the snapshot 2023-10-10T18:48:28.725 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/33 file after the snapshot 2023-10-10T18:48:28.726 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/34 file after the snapshot 2023-10-10T18:48:28.727 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/35 file after the snapshot 2023-10-10T18:48:28.728 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/36 file after the snapshot 2023-10-10T18:48:28.729 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/37 file after the snapshot 2023-10-10T18:48:28.730 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/38 file after the snapshot 2023-10-10T18:48:28.731 INFO:tasks.workunit.client.0.smithi175.stdout: create 288/39 file after the snapshot 2023-10-10T18:48:28.732 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/.snap/snap-subdir-test 2023-10-10T18:48:28.754 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/30 file after the snapshot 2023-10-10T18:48:28.755 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/31 file after the snapshot 2023-10-10T18:48:28.756 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/32 file after the snapshot 2023-10-10T18:48:28.757 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/33 file after the snapshot 2023-10-10T18:48:28.758 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/34 file after the snapshot 2023-10-10T18:48:28.759 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/35 file after the snapshot 2023-10-10T18:48:28.760 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/36 file after the snapshot 2023-10-10T18:48:28.761 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/37 file after the snapshot 2023-10-10T18:48:28.762 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/38 file after the snapshot 2023-10-10T18:48:28.763 INFO:tasks.workunit.client.0.smithi175.stdout: create 289/39 file after the snapshot 2023-10-10T18:48:28.764 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/.snap/snap-subdir-test 2023-10-10T18:48:28.782 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/30 file after the snapshot 2023-10-10T18:48:28.783 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/31 file after the snapshot 2023-10-10T18:48:28.783 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/32 file after the snapshot 2023-10-10T18:48:28.784 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/33 file after the snapshot 2023-10-10T18:48:28.785 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/34 file after the snapshot 2023-10-10T18:48:28.785 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/35 file after the snapshot 2023-10-10T18:48:28.786 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/36 file after the snapshot 2023-10-10T18:48:28.787 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/37 file after the snapshot 2023-10-10T18:48:28.787 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/38 file after the snapshot 2023-10-10T18:48:28.788 INFO:tasks.workunit.client.0.smithi175.stdout: create 290/39 file after the snapshot 2023-10-10T18:48:28.789 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/.snap/snap-subdir-test 2023-10-10T18:48:28.810 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/30 file after the snapshot 2023-10-10T18:48:28.811 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/31 file after the snapshot 2023-10-10T18:48:28.813 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/32 file after the snapshot 2023-10-10T18:48:28.814 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/33 file after the snapshot 2023-10-10T18:48:28.815 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/34 file after the snapshot 2023-10-10T18:48:28.816 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/35 file after the snapshot 2023-10-10T18:48:28.818 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/36 file after the snapshot 2023-10-10T18:48:28.819 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/37 file after the snapshot 2023-10-10T18:48:28.820 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/38 file after the snapshot 2023-10-10T18:48:28.821 INFO:tasks.workunit.client.0.smithi175.stdout: create 291/39 file after the snapshot 2023-10-10T18:48:28.823 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/.snap/snap-subdir-test 2023-10-10T18:48:28.844 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/30 file after the snapshot 2023-10-10T18:48:28.845 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/31 file after the snapshot 2023-10-10T18:48:28.846 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/32 file after the snapshot 2023-10-10T18:48:28.847 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/33 file after the snapshot 2023-10-10T18:48:28.848 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/34 file after the snapshot 2023-10-10T18:48:28.849 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/35 file after the snapshot 2023-10-10T18:48:28.850 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/36 file after the snapshot 2023-10-10T18:48:28.851 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/37 file after the snapshot 2023-10-10T18:48:28.852 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/38 file after the snapshot 2023-10-10T18:48:28.853 INFO:tasks.workunit.client.0.smithi175.stdout: create 292/39 file after the snapshot 2023-10-10T18:48:28.854 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/.snap/snap-subdir-test 2023-10-10T18:48:28.874 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/30 file after the snapshot 2023-10-10T18:48:28.875 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/31 file after the snapshot 2023-10-10T18:48:28.876 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/32 file after the snapshot 2023-10-10T18:48:28.877 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/33 file after the snapshot 2023-10-10T18:48:28.878 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/34 file after the snapshot 2023-10-10T18:48:28.879 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/35 file after the snapshot 2023-10-10T18:48:28.879 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/36 file after the snapshot 2023-10-10T18:48:28.880 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/37 file after the snapshot 2023-10-10T18:48:28.881 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/38 file after the snapshot 2023-10-10T18:48:28.882 INFO:tasks.workunit.client.0.smithi175.stdout: create 293/39 file after the snapshot 2023-10-10T18:48:28.883 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/.snap/snap-subdir-test 2023-10-10T18:48:28.902 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/30 file after the snapshot 2023-10-10T18:48:28.903 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/31 file after the snapshot 2023-10-10T18:48:28.903 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/32 file after the snapshot 2023-10-10T18:48:28.904 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/33 file after the snapshot 2023-10-10T18:48:28.905 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/34 file after the snapshot 2023-10-10T18:48:28.905 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/35 file after the snapshot 2023-10-10T18:48:28.906 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/36 file after the snapshot 2023-10-10T18:48:28.907 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/37 file after the snapshot 2023-10-10T18:48:28.907 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/38 file after the snapshot 2023-10-10T18:48:28.908 INFO:tasks.workunit.client.0.smithi175.stdout: create 294/39 file after the snapshot 2023-10-10T18:48:28.909 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/.snap/snap-subdir-test 2023-10-10T18:48:28.931 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/30 file after the snapshot 2023-10-10T18:48:28.931 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/31 file after the snapshot 2023-10-10T18:48:28.932 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/32 file after the snapshot 2023-10-10T18:48:28.933 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/33 file after the snapshot 2023-10-10T18:48:28.933 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/34 file after the snapshot 2023-10-10T18:48:28.934 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/35 file after the snapshot 2023-10-10T18:48:28.935 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/36 file after the snapshot 2023-10-10T18:48:28.935 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/37 file after the snapshot 2023-10-10T18:48:28.936 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/38 file after the snapshot 2023-10-10T18:48:28.936 INFO:tasks.workunit.client.0.smithi175.stdout: create 295/39 file after the snapshot 2023-10-10T18:48:28.937 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/.snap/snap-subdir-test 2023-10-10T18:48:29.492 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/30 file after the snapshot 2023-10-10T18:48:29.492 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/31 file after the snapshot 2023-10-10T18:48:29.493 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/32 file after the snapshot 2023-10-10T18:48:29.494 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/33 file after the snapshot 2023-10-10T18:48:29.494 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/34 file after the snapshot 2023-10-10T18:48:29.495 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/35 file after the snapshot 2023-10-10T18:48:29.496 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/36 file after the snapshot 2023-10-10T18:48:29.496 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/37 file after the snapshot 2023-10-10T18:48:29.497 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/38 file after the snapshot 2023-10-10T18:48:29.497 INFO:tasks.workunit.client.0.smithi175.stdout: create 296/39 file after the snapshot 2023-10-10T18:48:29.498 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/.snap/snap-subdir-test 2023-10-10T18:48:29.526 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/30 file after the snapshot 2023-10-10T18:48:29.527 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/31 file after the snapshot 2023-10-10T18:48:29.528 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/32 file after the snapshot 2023-10-10T18:48:29.529 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/33 file after the snapshot 2023-10-10T18:48:29.530 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/34 file after the snapshot 2023-10-10T18:48:29.531 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/35 file after the snapshot 2023-10-10T18:48:29.532 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/36 file after the snapshot 2023-10-10T18:48:29.533 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/37 file after the snapshot 2023-10-10T18:48:29.534 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/38 file after the snapshot 2023-10-10T18:48:29.535 INFO:tasks.workunit.client.0.smithi175.stdout: create 297/39 file after the snapshot 2023-10-10T18:48:29.536 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/.snap/snap-subdir-test 2023-10-10T18:48:29.563 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/30 file after the snapshot 2023-10-10T18:48:29.564 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/31 file after the snapshot 2023-10-10T18:48:29.565 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/32 file after the snapshot 2023-10-10T18:48:29.566 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/33 file after the snapshot 2023-10-10T18:48:29.567 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/34 file after the snapshot 2023-10-10T18:48:29.568 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/35 file after the snapshot 2023-10-10T18:48:29.569 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/36 file after the snapshot 2023-10-10T18:48:29.570 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/37 file after the snapshot 2023-10-10T18:48:29.570 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/38 file after the snapshot 2023-10-10T18:48:29.571 INFO:tasks.workunit.client.0.smithi175.stdout: create 298/39 file after the snapshot 2023-10-10T18:48:29.572 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/.snap/snap-subdir-test 2023-10-10T18:48:29.598 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/30 file after the snapshot 2023-10-10T18:48:29.599 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/31 file after the snapshot 2023-10-10T18:48:29.600 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/32 file after the snapshot 2023-10-10T18:48:29.601 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/33 file after the snapshot 2023-10-10T18:48:29.602 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/34 file after the snapshot 2023-10-10T18:48:29.603 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/35 file after the snapshot 2023-10-10T18:48:29.604 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/36 file after the snapshot 2023-10-10T18:48:29.605 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/37 file after the snapshot 2023-10-10T18:48:29.606 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/38 file after the snapshot 2023-10-10T18:48:29.607 INFO:tasks.workunit.client.0.smithi175.stdout: create 299/39 file after the snapshot 2023-10-10T18:48:29.608 INFO:tasks.workunit.client.0.smithi175.stdout:Take second snapshot .snap/test2 2023-10-10T18:48:29.640 INFO:tasks.workunit.client.0.smithi175.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-10-10T18:48:30.207 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-10-10T18:48:30.207 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-10-10T18:48:30.207 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-10-10T18:48:30.207 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-10-10T18:48:30.208 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-10-10T18:48:30.208 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-10-10T18:48:30.208 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-10-10T18:48:30.208 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-10-10T18:48:30.208 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-10-10T18:48:30.209 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-10-10T18:48:30.209 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-10-10T18:48:30.209 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-10-10T18:48:30.209 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-10-10T18:48:30.209 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-10-10T18:48:30.210 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-10-10T18:48:30.210 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-10-10T18:48:30.210 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-10-10T18:48:30.210 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-10-10T18:48:30.210 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-10-10T18:48:30.211 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-10-10T18:48:30.211 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-10-10T18:48:30.211 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-10-10T18:48:30.211 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-10-10T18:48:30.211 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-10-10T18:48:30.211 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-10-10T18:48:30.212 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-10-10T18:48:30.212 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-10-10T18:48:30.212 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-10-10T18:48:30.212 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-10-10T18:48:30.212 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-10-10T18:48:30.213 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-10-10T18:48:30.213 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-10-10T18:48:30.213 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-10-10T18:48:30.213 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-10-10T18:48:30.213 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-10-10T18:48:30.214 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-10-10T18:48:30.214 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-10-10T18:48:30.214 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-10-10T18:48:30.214 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-10-10T18:48:30.214 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-10-10T18:48:30.215 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-10-10T18:48:30.215 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-10-10T18:48:30.215 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-10-10T18:48:30.215 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-10-10T18:48:30.215 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-10-10T18:48:30.216 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-10-10T18:48:30.216 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-10-10T18:48:30.216 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-10-10T18:48:30.216 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-10-10T18:48:30.216 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-10-10T18:48:30.217 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-10-10T18:48:30.217 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-10-10T18:48:30.217 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-10-10T18:48:30.217 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-10-10T18:48:30.217 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-10-10T18:48:30.217 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-10-10T18:48:30.218 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-10-10T18:48:30.218 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-10-10T18:48:30.218 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-10-10T18:48:30.218 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-10-10T18:48:30.218 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-10-10T18:48:30.219 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-10-10T18:48:30.219 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-10-10T18:48:30.219 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-10-10T18:48:30.219 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-10-10T18:48:30.219 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-10-10T18:48:30.220 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-10-10T18:48:30.220 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-10-10T18:48:30.220 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-10-10T18:48:30.220 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-10-10T18:48:30.220 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-10-10T18:48:30.221 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-10-10T18:48:30.221 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-10-10T18:48:30.221 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-10-10T18:48:30.221 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-10-10T18:48:30.221 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-10-10T18:48:30.221 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-10-10T18:48:30.222 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-10-10T18:48:30.222 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-10-10T18:48:30.222 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-10-10T18:48:30.222 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-10-10T18:48:30.873 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-10-10T18:48:30.873 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-10-10T18:48:30.874 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-10-10T18:48:30.874 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-10-10T18:48:30.874 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-10-10T18:48:30.874 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-10-10T18:48:30.874 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-10-10T18:48:30.875 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-10-10T18:48:30.875 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-10-10T18:48:30.875 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-10-10T18:48:30.875 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-10-10T18:48:30.875 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-10-10T18:48:30.876 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-10-10T18:48:30.876 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-10-10T18:48:30.876 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-10-10T18:48:30.876 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-10-10T18:48:30.876 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-10-10T18:48:30.877 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-10-10T18:48:30.877 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-10-10T18:48:30.877 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-10-10T18:48:30.877 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-10-10T18:48:30.877 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-10-10T18:48:30.878 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-10-10T18:48:30.878 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-10-10T18:48:30.878 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-10-10T18:48:30.878 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-10-10T18:48:30.878 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-10-10T18:48:30.879 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-10-10T18:48:30.879 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-10-10T18:48:30.879 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-10-10T18:48:30.879 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-10-10T18:48:30.879 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-10-10T18:48:30.879 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-10-10T18:48:30.880 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-10-10T18:48:30.880 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-10-10T18:48:30.880 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-10-10T18:48:30.880 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-10-10T18:48:30.880 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-10-10T18:48:30.881 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-10-10T18:48:30.881 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-10-10T18:48:30.881 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-10-10T18:48:30.881 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-10-10T18:48:30.881 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-10-10T18:48:30.882 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-10-10T18:48:30.882 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-10-10T18:48:30.882 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-10-10T18:48:30.882 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-10-10T18:48:30.882 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-10-10T18:48:30.883 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-10-10T18:48:30.883 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-10-10T18:48:30.883 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-10-10T18:48:30.883 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-10-10T18:48:30.883 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-10-10T18:48:30.884 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-10-10T18:48:30.884 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-10-10T18:48:30.884 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-10-10T18:48:30.884 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-10-10T18:48:30.884 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-10-10T18:48:30.884 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-10-10T18:48:30.885 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-10-10T18:48:30.885 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-10-10T18:48:30.885 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-10-10T18:48:30.885 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-10-10T18:48:30.885 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-10-10T18:48:30.886 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-10-10T18:48:30.886 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-10-10T18:48:30.886 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-10-10T18:48:30.886 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-10-10T18:48:30.886 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-10-10T18:48:30.887 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-10-10T18:48:30.887 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-10-10T18:48:30.887 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-10-10T18:48:30.887 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-10-10T18:48:30.887 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-10-10T18:48:30.888 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-10-10T18:48:30.888 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-10-10T18:48:30.888 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-10-10T18:48:30.888 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-10-10T18:48:30.888 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-10-10T18:48:30.889 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-10-10T18:48:30.889 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-10-10T18:48:32.782 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-10-10T18:48:32.782 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-10-10T18:48:32.782 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-10-10T18:48:32.783 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-10-10T18:48:32.783 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-10-10T18:48:32.783 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-10-10T18:48:32.783 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-10-10T18:48:32.783 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-10-10T18:48:32.784 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-10-10T18:48:32.784 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-10-10T18:48:32.784 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-10-10T18:48:32.784 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-10-10T18:48:32.785 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-10-10T18:48:32.785 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-10-10T18:48:32.785 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-10-10T18:48:32.785 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-10-10T18:48:32.785 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-10-10T18:48:32.786 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-10-10T18:48:32.786 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-10-10T18:48:32.786 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-10-10T18:48:32.786 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-10-10T18:48:32.786 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-10-10T18:48:32.787 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-10-10T18:48:32.787 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-10-10T18:48:32.787 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-10-10T18:48:32.787 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-10-10T18:48:32.787 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-10-10T18:48:32.788 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-10-10T18:48:32.788 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-10-10T18:48:32.788 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-10-10T18:48:32.788 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-10-10T18:48:32.788 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-10-10T18:48:32.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-10-10T18:48:32.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-10-10T18:48:32.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-10-10T18:48:32.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-10-10T18:48:32.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-10-10T18:48:32.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-10-10T18:48:32.790 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-10-10T18:48:32.790 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-10-10T18:48:32.790 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-10-10T18:48:32.790 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-10-10T18:48:32.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-10-10T18:48:32.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-10-10T18:48:32.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-10-10T18:48:32.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-10-10T18:48:32.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-10-10T18:48:32.792 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-10-10T18:48:32.792 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-10-10T18:48:32.792 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-10-10T18:48:32.792 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-10-10T18:48:32.793 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-10-10T18:48:32.793 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-10-10T18:48:32.794 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-10-10T18:48:32.794 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-10-10T18:48:32.794 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-10-10T18:48:32.794 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-10-10T18:48:32.795 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-10-10T18:48:32.797 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-10-10T18:48:32.797 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-10-10T18:48:32.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-10-10T18:48:32.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-10-10T18:48:32.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-10-10T18:48:32.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-10-10T18:48:32.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-10-10T18:48:32.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-10-10T18:48:32.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-10-10T18:48:32.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-10-10T18:48:32.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-10-10T18:48:32.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-10-10T18:48:32.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-10-10T18:48:32.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-10-10T18:48:32.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-10-10T18:48:32.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-10-10T18:48:32.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-10-10T18:48:32.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-10-10T18:48:32.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-10-10T18:48:32.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-10-10T18:48:32.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-10-10T18:48:32.802 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-10-10T18:48:32.802 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-10-10T18:48:32.928 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-10-10T18:48:32.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-10-10T18:48:32.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-10-10T18:48:32.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-10-10T18:48:32.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-10-10T18:48:32.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-10-10T18:48:32.930 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-10-10T18:48:32.930 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-10-10T18:48:32.930 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-10-10T18:48:32.930 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-10-10T18:48:32.930 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-10-10T18:48:32.931 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-10-10T18:48:32.931 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-10-10T18:48:32.931 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-10-10T18:48:32.931 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-10-10T18:48:32.931 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-10-10T18:48:32.932 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-10-10T18:48:32.932 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-10-10T18:48:32.932 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-10-10T18:48:32.932 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-10-10T18:48:32.932 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-10-10T18:48:32.933 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-10-10T18:48:32.933 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-10-10T18:48:32.933 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-10-10T18:48:32.933 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-10-10T18:48:32.933 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-10-10T18:48:32.934 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-10-10T18:48:32.934 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-10-10T18:48:32.934 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-10-10T18:48:32.934 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-10-10T18:48:32.934 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-10-10T18:48:32.935 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-10-10T18:48:32.935 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-10-10T18:48:32.935 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-10-10T18:48:32.935 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-10-10T18:48:32.935 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-10-10T18:48:32.936 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-10-10T18:48:32.936 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-10-10T18:48:32.936 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-10-10T18:48:32.936 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-10-10T18:48:32.936 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-10-10T18:48:32.937 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-10-10T18:48:32.937 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-10-10T18:48:32.937 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-10-10T18:48:32.937 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-10-10T18:48:32.937 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-10-10T18:48:32.938 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-10-10T18:48:32.938 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-10-10T18:48:32.938 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-10-10T18:48:32.938 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-10-10T18:48:32.939 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-10-10T18:48:32.939 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-10-10T18:48:32.939 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-10-10T18:48:32.939 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-10-10T18:48:32.939 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-10-10T18:48:32.939 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-10-10T18:48:32.940 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-10-10T18:48:32.940 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-10-10T18:48:32.940 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-10-10T18:48:32.940 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-10-10T18:48:32.941 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-10-10T18:48:32.941 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-10-10T18:48:32.941 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-10-10T18:48:32.941 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-10-10T18:48:32.941 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-10-10T18:48:32.941 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-10-10T18:48:32.942 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-10-10T18:48:32.942 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-10-10T18:48:32.942 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-10-10T18:48:32.942 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-10-10T18:48:32.942 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-10-10T18:48:32.943 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-10-10T18:48:32.943 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-10-10T18:48:32.943 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-10-10T18:48:32.943 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-10-10T18:48:32.943 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-10-10T18:48:32.944 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-10-10T18:48:32.944 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-10-10T18:48:32.944 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-10-10T18:48:32.944 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-10-10T18:48:32.944 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-10-10T18:48:34.461 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-10-10T18:48:34.461 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-10-10T18:48:34.461 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-10-10T18:48:34.461 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-10-10T18:48:34.462 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-10-10T18:48:34.462 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-10-10T18:48:34.462 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-10-10T18:48:34.462 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-10-10T18:48:34.462 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-10-10T18:48:34.463 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-10-10T18:48:34.463 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-10-10T18:48:34.463 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-10-10T18:48:34.463 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-10-10T18:48:34.463 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-10-10T18:48:34.464 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-10-10T18:48:34.464 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-10-10T18:48:34.464 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-10-10T18:48:34.464 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-10-10T18:48:34.464 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-10-10T18:48:34.465 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-10-10T18:48:34.465 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-10-10T18:48:34.465 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-10-10T18:48:34.465 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-10-10T18:48:34.465 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-10-10T18:48:34.465 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-10-10T18:48:34.466 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-10-10T18:48:34.466 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-10-10T18:48:34.466 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-10-10T18:48:34.466 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-10-10T18:48:34.467 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-10-10T18:48:34.467 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-10-10T18:48:34.467 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-10-10T18:48:34.467 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-10-10T18:48:34.467 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-10-10T18:48:34.468 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-10-10T18:48:34.468 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-10-10T18:48:34.468 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-10-10T18:48:34.468 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-10-10T18:48:34.468 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-10-10T18:48:34.468 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-10-10T18:48:34.469 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-10-10T18:48:34.469 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-10-10T18:48:34.469 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-10-10T18:48:34.469 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-10-10T18:48:34.469 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-10-10T18:48:34.470 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-10-10T18:48:34.470 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-10-10T18:48:34.470 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-10-10T18:48:34.470 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-10-10T18:48:34.470 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-10-10T18:48:34.471 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-10-10T18:48:34.471 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-10-10T18:48:34.471 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-10-10T18:48:34.471 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-10-10T18:48:34.471 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-10-10T18:48:34.472 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-10-10T18:48:34.472 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-10-10T18:48:34.472 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-10-10T18:48:34.472 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-10-10T18:48:34.472 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-10-10T18:48:34.473 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-10-10T18:48:34.473 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-10-10T18:48:34.473 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-10-10T18:48:34.473 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-10-10T18:48:34.473 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-10-10T18:48:34.474 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-10-10T18:48:34.474 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-10-10T18:48:34.474 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-10-10T18:48:34.474 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-10-10T18:48:34.474 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-10-10T18:48:34.475 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-10-10T18:48:34.475 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-10-10T18:48:34.475 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-10-10T18:48:34.475 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-10-10T18:48:34.475 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-10-10T18:48:34.476 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-10-10T18:48:34.476 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-10-10T18:48:34.476 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-10-10T18:48:34.476 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-10-10T18:48:34.476 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-10-10T18:48:34.477 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-10-10T18:48:35.596 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-10-10T18:48:35.596 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-10-10T18:48:35.596 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-10-10T18:48:35.596 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-10-10T18:48:35.597 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-10-10T18:48:35.597 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-10-10T18:48:35.597 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-10-10T18:48:35.597 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-10-10T18:48:35.597 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-10-10T18:48:35.598 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-10-10T18:48:35.598 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-10-10T18:48:35.598 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-10-10T18:48:35.598 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-10-10T18:48:35.599 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-10-10T18:48:35.599 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-10-10T18:48:35.599 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-10-10T18:48:35.599 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-10-10T18:48:35.599 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-10-10T18:48:35.600 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-10-10T18:48:35.600 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-10-10T18:48:35.600 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-10-10T18:48:35.600 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-10-10T18:48:35.600 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-10-10T18:48:35.601 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-10-10T18:48:35.601 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-10-10T18:48:35.601 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-10-10T18:48:35.601 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-10-10T18:48:35.601 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-10-10T18:48:35.602 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-10-10T18:48:35.602 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-10-10T18:48:35.602 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-10-10T18:48:35.602 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-10-10T18:48:35.602 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-10-10T18:48:35.602 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-10-10T18:48:35.603 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-10-10T18:48:35.603 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-10-10T18:48:35.603 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-10-10T18:48:35.603 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-10-10T18:48:35.603 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-10-10T18:48:35.604 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-10-10T18:48:35.604 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-10-10T18:48:35.604 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-10-10T18:48:35.604 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-10-10T18:48:35.604 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-10-10T18:48:35.605 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-10-10T18:48:35.605 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-10-10T18:48:35.605 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-10-10T18:48:35.605 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-10-10T18:48:35.605 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-10-10T18:48:35.605 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-10-10T18:48:35.606 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-10-10T18:48:35.606 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-10-10T18:48:35.606 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-10-10T18:48:35.606 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-10-10T18:48:35.606 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-10-10T18:48:35.607 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-10-10T18:48:35.607 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-10-10T18:48:35.607 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-10-10T18:48:35.607 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-10-10T18:48:35.608 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-10-10T18:48:35.608 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-10-10T18:48:35.608 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-10-10T18:48:35.608 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-10-10T18:48:35.608 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-10-10T18:48:35.609 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-10-10T18:48:35.609 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-10-10T18:48:35.609 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-10-10T18:48:35.609 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-10-10T18:48:35.609 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-10-10T18:48:35.609 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-10-10T18:48:35.610 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-10-10T18:48:35.610 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-10-10T18:48:35.610 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-10-10T18:48:35.611 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-10-10T18:48:35.611 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-10-10T18:48:35.611 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-10-10T18:48:35.611 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-10-10T18:48:35.611 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-10-10T18:48:35.612 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-10-10T18:48:35.612 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-10-10T18:48:35.612 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-10-10T18:48:36.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-10-10T18:48:36.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-10-10T18:48:36.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-10-10T18:48:36.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-10-10T18:48:36.404 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-10-10T18:48:36.404 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-10-10T18:48:36.404 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-10-10T18:48:36.404 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-10-10T18:48:36.404 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-10-10T18:48:36.405 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-10-10T18:48:36.405 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-10-10T18:48:36.405 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-10-10T18:48:36.405 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-10-10T18:48:36.405 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-10-10T18:48:36.406 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-10-10T18:48:36.406 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-10-10T18:48:36.406 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-10-10T18:48:36.406 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-10-10T18:48:36.406 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-10-10T18:48:36.407 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-10-10T18:48:36.407 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-10-10T18:48:36.407 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-10-10T18:48:36.407 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-10-10T18:48:36.407 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-10-10T18:48:36.408 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-10-10T18:48:36.408 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-10-10T18:48:36.408 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-10-10T18:48:36.408 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-10-10T18:48:36.408 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-10-10T18:48:36.409 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-10-10T18:48:36.409 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-10-10T18:48:36.409 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-10-10T18:48:36.409 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-10-10T18:48:36.409 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-10-10T18:48:36.410 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-10-10T18:48:36.410 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-10-10T18:48:36.410 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-10-10T18:48:36.410 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-10-10T18:48:36.410 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-10-10T18:48:36.410 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-10-10T18:48:36.411 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-10-10T18:48:36.411 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-10-10T18:48:36.411 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-10-10T18:48:36.411 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-10-10T18:48:36.411 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-10-10T18:48:36.412 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-10-10T18:48:36.412 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-10-10T18:48:36.412 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-10-10T18:48:36.412 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-10-10T18:48:36.412 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-10-10T18:48:36.413 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-10-10T18:48:36.413 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-10-10T18:48:36.413 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-10-10T18:48:36.413 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-10-10T18:48:36.413 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-10-10T18:48:36.413 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-10-10T18:48:36.414 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-10-10T18:48:36.414 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-10-10T18:48:36.414 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-10-10T18:48:36.414 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-10-10T18:48:36.414 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-10-10T18:48:36.415 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-10-10T18:48:36.415 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-10-10T18:48:36.415 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-10-10T18:48:36.415 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-10-10T18:48:36.415 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-10-10T18:48:36.416 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-10-10T18:48:36.416 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-10-10T18:48:36.416 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-10-10T18:48:36.416 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-10-10T18:48:36.416 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-10-10T18:48:36.417 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-10-10T18:48:36.417 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-10-10T18:48:36.417 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-10-10T18:48:36.417 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-10-10T18:48:36.417 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-10-10T18:48:36.418 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-10-10T18:48:36.418 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-10-10T18:48:36.418 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-10-10T18:48:36.418 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-10-10T18:48:36.418 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-10-10T18:48:36.419 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-10-10T18:48:37.612 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-10-10T18:48:37.612 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-10-10T18:48:37.612 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-10-10T18:48:37.613 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-10-10T18:48:37.613 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-10-10T18:48:37.613 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-10-10T18:48:37.613 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-10-10T18:48:37.613 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-10-10T18:48:37.614 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-10-10T18:48:37.614 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-10-10T18:48:37.614 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-10-10T18:48:37.614 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-10-10T18:48:37.615 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-10-10T18:48:37.615 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-10-10T18:48:37.615 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-10-10T18:48:37.615 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-10-10T18:48:37.615 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-10-10T18:48:37.616 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-10-10T18:48:37.616 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-10-10T18:48:37.616 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-10-10T18:48:37.616 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-10-10T18:48:37.616 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-10-10T18:48:37.617 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-10-10T18:48:37.617 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-10-10T18:48:37.617 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-10-10T18:48:37.617 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-10-10T18:48:37.617 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-10-10T18:48:37.618 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-10-10T18:48:37.618 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-10-10T18:48:37.618 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-10-10T18:48:37.618 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-10-10T18:48:37.618 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-10-10T18:48:37.619 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-10-10T18:48:37.619 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-10-10T18:48:37.619 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-10-10T18:48:37.619 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-10-10T18:48:37.619 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-10-10T18:48:37.620 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-10-10T18:48:37.621 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-10-10T18:48:37.621 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-10-10T18:48:37.621 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-10-10T18:48:37.622 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-10-10T18:48:37.622 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-10-10T18:48:37.622 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-10-10T18:48:37.622 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-10-10T18:48:37.622 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-10-10T18:48:37.623 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-10-10T18:48:37.623 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-10-10T18:48:37.623 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-10-10T18:48:37.623 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-10-10T18:48:37.623 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-10-10T18:48:37.624 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-10-10T18:48:37.624 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-10-10T18:48:37.624 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-10-10T18:48:37.624 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-10-10T18:48:37.624 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-10-10T18:48:37.625 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-10-10T18:48:37.625 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-10-10T18:48:37.625 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-10-10T18:48:37.625 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-10-10T18:48:37.625 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-10-10T18:48:37.626 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-10-10T18:48:37.626 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-10-10T18:48:37.626 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-10-10T18:48:37.627 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-10-10T18:48:37.627 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-10-10T18:48:37.627 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-10-10T18:48:37.627 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-10-10T18:48:37.627 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-10-10T18:48:37.628 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-10-10T18:48:37.628 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-10-10T18:48:37.628 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-10-10T18:48:37.628 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-10-10T18:48:37.628 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-10-10T18:48:37.629 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-10-10T18:48:37.629 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-10-10T18:48:37.629 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-10-10T18:48:37.629 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-10-10T18:48:37.629 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-10-10T18:48:37.629 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-10-10T18:48:37.630 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-10-10T18:48:37.955 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-10-10T18:48:37.956 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-10-10T18:48:37.956 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-10-10T18:48:37.957 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-10-10T18:48:37.957 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-10-10T18:48:37.957 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-10-10T18:48:37.957 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-10-10T18:48:37.957 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-10-10T18:48:37.958 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-10-10T18:48:37.958 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-10-10T18:48:37.958 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-10-10T18:48:37.958 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-10-10T18:48:37.958 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-10-10T18:48:37.959 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-10-10T18:48:37.959 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-10-10T18:48:37.959 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-10-10T18:48:37.959 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-10-10T18:48:37.959 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-10-10T18:48:37.960 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-10-10T18:48:37.960 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-10-10T18:48:37.960 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-10-10T18:48:37.960 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-10-10T18:48:37.960 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-10-10T18:48:37.960 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-10-10T18:48:37.961 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-10-10T18:48:37.961 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-10-10T18:48:37.961 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-10-10T18:48:37.961 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-10-10T18:48:37.961 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-10-10T18:48:37.961 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-10-10T18:48:37.962 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-10-10T18:48:37.962 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-10-10T18:48:37.962 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-10-10T18:48:37.962 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-10-10T18:48:37.962 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-10-10T18:48:37.963 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-10-10T18:48:37.963 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-10-10T18:48:37.963 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-10-10T18:48:37.963 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-10-10T18:48:37.963 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-10-10T18:48:37.964 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-10-10T18:48:37.964 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-10-10T18:48:37.964 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-10-10T18:48:37.964 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-10-10T18:48:37.964 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-10-10T18:48:37.965 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-10-10T18:48:37.965 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-10-10T18:48:37.965 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-10-10T18:48:37.965 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-10-10T18:48:37.965 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-10-10T18:48:37.965 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-10-10T18:48:37.966 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-10-10T18:48:37.966 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-10-10T18:48:37.966 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-10-10T18:48:37.966 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-10-10T18:48:37.966 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-10-10T18:48:37.967 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-10-10T18:48:37.967 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-10-10T18:48:37.967 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-10-10T18:48:37.967 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-10-10T18:48:37.967 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-10-10T18:48:37.968 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-10-10T18:48:37.968 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-10-10T18:48:37.968 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-10-10T18:48:37.968 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-10-10T18:48:37.968 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-10-10T18:48:37.969 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-10-10T18:48:37.969 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-10-10T18:48:37.969 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-10-10T18:48:37.969 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-10-10T18:48:37.969 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-10-10T18:48:37.970 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-10-10T18:48:37.970 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-10-10T18:48:37.970 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-10-10T18:48:37.970 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-10-10T18:48:37.970 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-10-10T18:48:37.971 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-10-10T18:48:37.971 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-10-10T18:48:37.971 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-10-10T18:48:37.971 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-10-10T18:48:37.972 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-10-10T18:48:40.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-10-10T18:48:40.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-10-10T18:48:40.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-10-10T18:48:40.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-10-10T18:48:40.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-10-10T18:48:40.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-10-10T18:48:40.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-10-10T18:48:40.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-10-10T18:48:40.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-10-10T18:48:40.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-10-10T18:48:40.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-10-10T18:48:40.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-10-10T18:48:40.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-10-10T18:48:40.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-10-10T18:48:40.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-10-10T18:48:40.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-10-10T18:48:40.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-10-10T18:48:40.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-10-10T18:48:40.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-10-10T18:48:40.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-10-10T18:48:40.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-10-10T18:48:40.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-10-10T18:48:40.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-10-10T18:48:40.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-10-10T18:48:40.393 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-10-10T18:48:40.393 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-10-10T18:48:40.393 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-10-10T18:48:40.393 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-10-10T18:48:40.393 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-10-10T18:48:40.394 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-10-10T18:48:40.394 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-10-10T18:48:40.394 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-10-10T18:48:40.394 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-10-10T18:48:40.394 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-10-10T18:48:40.395 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-10-10T18:48:40.395 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-10-10T18:48:40.395 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-10-10T18:48:40.395 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-10-10T18:48:40.395 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-10-10T18:48:40.396 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-10-10T18:48:40.396 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-10-10T18:48:40.396 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-10-10T18:48:40.396 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-10-10T18:48:40.396 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-10-10T18:48:40.397 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-10-10T18:48:40.397 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-10-10T18:48:40.397 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-10-10T18:48:40.397 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-10-10T18:48:40.397 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-10-10T18:48:40.398 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-10-10T18:48:40.398 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-10-10T18:48:40.398 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-10-10T18:48:40.398 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-10-10T18:48:40.398 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-10-10T18:48:40.399 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-10-10T18:48:40.399 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-10-10T18:48:40.399 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-10-10T18:48:40.399 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-10-10T18:48:40.400 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-10-10T18:48:40.400 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-10-10T18:48:40.400 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-10-10T18:48:40.400 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-10-10T18:48:40.400 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-10-10T18:48:40.401 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-10-10T18:48:40.401 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-10-10T18:48:40.401 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-10-10T18:48:40.401 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-10-10T18:48:40.401 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-10-10T18:48:40.402 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-10-10T18:48:40.402 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-10-10T18:48:40.402 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-10-10T18:48:40.402 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-10-10T18:48:40.402 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-10-10T18:48:40.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-10-10T18:48:40.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-10-10T18:48:40.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-10-10T18:48:40.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-10-10T18:48:40.403 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-10-10T18:48:40.404 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-10-10T18:48:40.404 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-10-10T18:48:40.404 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-10-10T18:48:41.375 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-10-10T18:48:41.376 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-10-10T18:48:41.376 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-10-10T18:48:41.376 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-10-10T18:48:41.376 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-10-10T18:48:41.376 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-10-10T18:48:41.377 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-10-10T18:48:41.377 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-10-10T18:48:41.377 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-10-10T18:48:41.377 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-10-10T18:48:41.377 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-10-10T18:48:41.378 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-10-10T18:48:41.378 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-10-10T18:48:41.378 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-10-10T18:48:41.378 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-10-10T18:48:41.379 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-10-10T18:48:41.379 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-10-10T18:48:41.379 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-10-10T18:48:41.379 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-10-10T18:48:41.379 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-10-10T18:48:41.380 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-10-10T18:48:41.380 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-10-10T18:48:41.380 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-10-10T18:48:41.380 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-10-10T18:48:41.380 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-10-10T18:48:41.381 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-10-10T18:48:41.381 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-10-10T18:48:41.381 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-10-10T18:48:41.381 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-10-10T18:48:41.381 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-10-10T18:48:41.382 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-10-10T18:48:41.382 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-10-10T18:48:41.382 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-10-10T18:48:41.383 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-10-10T18:48:41.383 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-10-10T18:48:41.383 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-10-10T18:48:41.384 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-10-10T18:48:41.384 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-10-10T18:48:41.384 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-10-10T18:48:41.384 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-10-10T18:48:41.384 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-10-10T18:48:41.385 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-10-10T18:48:41.385 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-10-10T18:48:41.385 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-10-10T18:48:41.385 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-10-10T18:48:41.385 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-10-10T18:48:41.386 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-10-10T18:48:41.386 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-10-10T18:48:41.386 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-10-10T18:48:41.386 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-10-10T18:48:41.386 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-10-10T18:48:41.387 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-10-10T18:48:41.387 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-10-10T18:48:41.387 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-10-10T18:48:41.387 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-10-10T18:48:41.387 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-10-10T18:48:41.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-10-10T18:48:41.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-10-10T18:48:41.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-10-10T18:48:41.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-10-10T18:48:41.388 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-10-10T18:48:41.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-10-10T18:48:41.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-10-10T18:48:41.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-10-10T18:48:41.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-10-10T18:48:41.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-10-10T18:48:41.389 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-10-10T18:48:41.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-10-10T18:48:41.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-10-10T18:48:41.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-10-10T18:48:41.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-10-10T18:48:41.390 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-10-10T18:48:41.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-10-10T18:48:41.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-10-10T18:48:41.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-10-10T18:48:41.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-10-10T18:48:41.391 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-10-10T18:48:41.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-10-10T18:48:41.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-10-10T18:48:41.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-10-10T18:48:41.392 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-10-10T18:48:41.520 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-10-10T18:48:41.520 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-10-10T18:48:41.520 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-10-10T18:48:41.521 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-10-10T18:48:41.521 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-10-10T18:48:41.521 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-10-10T18:48:41.521 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-10-10T18:48:41.522 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-10-10T18:48:41.522 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-10-10T18:48:41.522 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-10-10T18:48:41.522 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-10-10T18:48:41.522 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-10-10T18:48:41.523 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-10-10T18:48:41.523 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-10-10T18:48:41.523 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-10-10T18:48:41.523 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-10-10T18:48:41.523 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-10-10T18:48:41.524 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-10-10T18:48:41.524 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-10-10T18:48:41.524 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-10-10T18:48:41.524 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-10-10T18:48:41.524 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-10-10T18:48:41.525 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-10-10T18:48:41.525 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-10-10T18:48:41.525 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-10-10T18:48:41.525 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-10-10T18:48:41.525 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-10-10T18:48:41.525 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-10-10T18:48:41.526 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-10-10T18:48:41.526 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-10-10T18:48:41.526 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-10-10T18:48:41.526 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-10-10T18:48:41.527 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-10-10T18:48:41.527 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-10-10T18:48:41.527 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-10-10T18:48:41.527 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-10-10T18:48:41.527 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-10-10T18:48:41.527 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-10-10T18:48:41.528 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-10-10T18:48:41.528 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-10-10T18:48:41.528 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-10-10T18:48:41.528 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-10-10T18:48:41.528 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-10-10T18:48:41.529 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-10-10T18:48:41.529 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-10-10T18:48:41.529 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-10-10T18:48:41.529 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-10-10T18:48:41.529 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-10-10T18:48:41.530 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-10-10T18:48:41.530 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-10-10T18:48:41.530 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-10-10T18:48:41.530 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-10-10T18:48:41.530 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-10-10T18:48:41.531 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-10-10T18:48:41.531 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-10-10T18:48:41.531 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-10-10T18:48:41.531 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-10-10T18:48:41.531 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-10-10T18:48:41.531 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-10-10T18:48:41.532 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-10-10T18:48:41.532 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-10-10T18:48:41.532 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-10-10T18:48:41.532 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-10-10T18:48:41.532 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-10-10T18:48:41.533 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-10-10T18:48:41.533 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-10-10T18:48:41.533 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-10-10T18:48:41.533 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-10-10T18:48:41.533 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-10-10T18:48:41.534 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-10-10T18:48:41.534 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-10-10T18:48:41.534 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-10-10T18:48:41.534 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-10-10T18:48:41.534 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-10-10T18:48:41.535 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-10-10T18:48:41.535 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-10-10T18:48:41.535 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-10-10T18:48:41.535 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-10-10T18:48:41.535 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-10-10T18:48:41.536 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-10-10T18:48:41.536 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-10-10T18:48:41.914 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-10-10T18:48:41.914 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-10-10T18:48:41.915 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-10-10T18:48:41.915 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-10-10T18:48:41.915 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-10-10T18:48:41.915 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-10-10T18:48:41.915 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-10-10T18:48:41.916 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-10-10T18:48:41.916 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-10-10T18:48:41.916 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-10-10T18:48:41.916 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-10-10T18:48:41.916 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-10-10T18:48:41.917 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-10-10T18:48:41.917 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-10-10T18:48:41.917 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-10-10T18:48:41.917 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-10-10T18:48:41.917 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-10-10T18:48:41.918 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-10-10T18:48:41.918 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-10-10T18:48:41.918 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-10-10T18:48:41.918 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-10-10T18:48:41.918 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-10-10T18:48:41.919 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-10-10T18:48:41.919 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-10-10T18:48:41.919 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-10-10T18:48:41.919 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-10-10T18:48:41.919 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-10-10T18:48:41.920 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-10-10T18:48:41.920 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-10-10T18:48:41.920 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-10-10T18:48:41.920 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-10-10T18:48:41.920 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-10-10T18:48:41.921 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-10-10T18:48:41.921 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-10-10T18:48:41.921 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-10-10T18:48:41.921 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-10-10T18:48:41.921 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-10-10T18:48:41.922 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-10-10T18:48:41.922 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-10-10T18:48:41.922 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-10-10T18:48:41.922 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-10-10T18:48:41.922 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-10-10T18:48:41.923 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-10-10T18:48:41.923 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-10-10T18:48:41.923 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-10-10T18:48:41.923 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-10-10T18:48:41.923 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-10-10T18:48:41.924 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-10-10T18:48:41.924 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-10-10T18:48:41.924 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-10-10T18:48:41.924 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-10-10T18:48:41.924 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-10-10T18:48:41.924 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-10-10T18:48:41.925 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-10-10T18:48:41.925 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-10-10T18:48:41.925 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-10-10T18:48:41.925 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-10-10T18:48:41.925 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-10-10T18:48:41.926 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-10-10T18:48:41.926 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-10-10T18:48:41.926 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-10-10T18:48:41.926 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-10-10T18:48:41.926 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-10-10T18:48:41.927 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-10-10T18:48:41.927 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-10-10T18:48:41.927 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-10-10T18:48:41.927 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-10-10T18:48:41.927 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-10-10T18:48:41.928 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-10-10T18:48:41.928 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-10-10T18:48:41.928 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-10-10T18:48:41.928 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-10-10T18:48:41.928 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-10-10T18:48:41.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-10-10T18:48:41.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-10-10T18:48:41.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-10-10T18:48:41.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-10-10T18:48:41.929 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-10-10T18:48:41.930 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-10-10T18:48:41.930 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-10-10T18:48:41.930 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-10-10T18:48:42.668 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-10-10T18:48:42.668 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-10-10T18:48:42.668 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-10-10T18:48:42.668 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-10-10T18:48:42.669 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-10-10T18:48:42.669 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-10-10T18:48:42.669 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-10-10T18:48:42.669 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-10-10T18:48:42.669 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-10-10T18:48:42.670 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-10-10T18:48:42.670 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-10-10T18:48:42.670 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-10-10T18:48:42.670 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-10-10T18:48:42.670 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-10-10T18:48:42.671 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-10-10T18:48:42.671 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-10-10T18:48:42.671 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-10-10T18:48:42.671 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-10-10T18:48:42.672 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-10-10T18:48:42.672 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-10-10T18:48:42.672 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-10-10T18:48:42.672 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-10-10T18:48:42.672 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-10-10T18:48:42.673 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-10-10T18:48:42.673 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-10-10T18:48:42.673 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-10-10T18:48:42.673 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-10-10T18:48:42.673 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-10-10T18:48:42.673 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-10-10T18:48:42.674 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-10-10T18:48:42.674 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-10-10T18:48:42.674 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-10-10T18:48:42.674 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-10-10T18:48:42.674 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-10-10T18:48:42.675 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-10-10T18:48:42.675 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-10-10T18:48:42.675 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-10-10T18:48:42.675 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-10-10T18:48:42.675 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-10-10T18:48:42.676 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-10-10T18:48:42.676 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-10-10T18:48:42.676 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-10-10T18:48:42.676 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-10-10T18:48:42.676 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-10-10T18:48:42.677 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-10-10T18:48:42.677 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-10-10T18:48:42.677 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-10-10T18:48:42.677 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-10-10T18:48:42.677 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-10-10T18:48:42.678 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-10-10T18:48:42.678 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-10-10T18:48:42.678 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-10-10T18:48:42.678 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-10-10T18:48:42.678 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-10-10T18:48:42.679 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-10-10T18:48:42.679 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-10-10T18:48:42.679 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-10-10T18:48:42.679 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-10-10T18:48:42.679 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-10-10T18:48:42.680 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-10-10T18:48:42.680 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-10-10T18:48:42.680 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-10-10T18:48:42.680 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-10-10T18:48:42.680 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-10-10T18:48:42.681 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-10-10T18:48:42.681 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-10-10T18:48:42.681 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-10-10T18:48:42.681 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-10-10T18:48:42.681 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-10-10T18:48:42.682 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-10-10T18:48:42.682 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-10-10T18:48:42.682 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-10-10T18:48:42.682 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-10-10T18:48:42.682 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-10-10T18:48:42.683 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-10-10T18:48:42.683 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-10-10T18:48:42.683 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-10-10T18:48:42.683 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-10-10T18:48:42.683 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-10-10T18:48:42.684 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-10-10T18:48:42.684 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-10-10T18:48:42.788 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-10-10T18:48:42.788 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-10-10T18:48:42.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-10-10T18:48:42.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-10-10T18:48:42.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-10-10T18:48:42.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-10-10T18:48:42.789 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-10-10T18:48:42.790 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-10-10T18:48:42.790 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-10-10T18:48:42.790 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-10-10T18:48:42.790 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-10-10T18:48:42.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-10-10T18:48:42.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-10-10T18:48:42.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-10-10T18:48:42.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-10-10T18:48:42.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-10-10T18:48:42.791 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-10-10T18:48:42.792 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-10-10T18:48:42.792 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-10-10T18:48:42.792 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-10-10T18:48:42.792 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-10-10T18:48:42.793 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-10-10T18:48:42.793 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-10-10T18:48:42.793 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-10-10T18:48:42.793 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-10-10T18:48:42.793 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-10-10T18:48:42.794 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-10-10T18:48:42.794 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-10-10T18:48:42.794 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-10-10T18:48:42.794 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-10-10T18:48:42.795 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-10-10T18:48:42.795 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-10-10T18:48:42.795 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-10-10T18:48:42.795 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-10-10T18:48:42.795 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-10-10T18:48:42.796 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-10-10T18:48:42.796 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-10-10T18:48:42.796 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-10-10T18:48:42.796 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-10-10T18:48:42.797 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-10-10T18:48:42.797 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-10-10T18:48:42.797 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-10-10T18:48:42.797 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-10-10T18:48:42.797 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-10-10T18:48:42.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-10-10T18:48:42.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-10-10T18:48:42.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-10-10T18:48:42.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-10-10T18:48:42.798 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-10-10T18:48:42.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-10-10T18:48:42.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-10-10T18:48:42.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-10-10T18:48:42.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-10-10T18:48:42.799 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-10-10T18:48:42.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-10-10T18:48:42.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-10-10T18:48:42.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-10-10T18:48:42.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-10-10T18:48:42.800 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-10-10T18:48:42.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-10-10T18:48:42.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-10-10T18:48:42.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-10-10T18:48:42.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-10-10T18:48:42.801 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-10-10T18:48:42.802 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-10-10T18:48:42.802 INFO:tasks.workunit.client.0.smithi175.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-10-10T18:48:42.802 INFO:tasks.workunit.client.0.smithi175.stdout:Take third snapshot .snap/test3 2023-10-10T18:48:43.259 INFO:tasks.workunit.client.0.smithi175.stdout:Delete the snapshots... 2023-10-10T18:48:45.510 INFO:tasks.workunit.client.0.smithi175.stdout:.//.snap/test1 2023-10-10T18:48:45.510 INFO:tasks.workunit.client.0.smithi175.stdout:.//.snap/test2 2023-10-10T18:48:45.511 INFO:tasks.workunit.client.0.smithi175.stdout:.//.snap/test3 2023-10-10T18:48:45.516 INFO:tasks.workunit.client.0.smithi175.stdout:./274/.snap/snap-subdir-test 2023-10-10T18:48:45.523 INFO:tasks.workunit.client.0.smithi175.stdout:./166/.snap/snap-subdir-test 2023-10-10T18:48:45.528 INFO:tasks.workunit.client.0.smithi175.stdout:./256/.snap/snap-subdir-test 2023-10-10T18:48:45.534 INFO:tasks.workunit.client.0.smithi175.stdout:./247/.snap/snap-subdir-test 2023-10-10T18:48:45.537 INFO:tasks.workunit.client.0.smithi175.stdout:./248/.snap/snap-subdir-test 2023-10-10T18:48:45.544 INFO:tasks.workunit.client.0.smithi175.stdout:./147/.snap/snap-subdir-test 2023-10-10T18:48:45.552 INFO:tasks.workunit.client.0.smithi175.stdout:./151/.snap/snap-subdir-test 2023-10-10T18:48:45.557 INFO:tasks.workunit.client.0.smithi175.stdout:./170/.snap/snap-subdir-test 2023-10-10T18:48:45.563 INFO:tasks.workunit.client.0.smithi175.stdout:./167/.snap/snap-subdir-test 2023-10-10T18:48:45.568 INFO:tasks.workunit.client.0.smithi175.stdout:./140/.snap/snap-subdir-test 2023-10-10T18:48:45.574 INFO:tasks.workunit.client.0.smithi175.stdout:./142/.snap/snap-subdir-test 2023-10-10T18:48:46.123 INFO:tasks.workunit.client.0.smithi175.stdout:./150/.snap/snap-subdir-test 2023-10-10T18:48:46.129 INFO:tasks.workunit.client.0.smithi175.stdout:./273/.snap/snap-subdir-test 2023-10-10T18:48:46.192 INFO:tasks.workunit.client.0.smithi175.stdout:./269/.snap/snap-subdir-test 2023-10-10T18:48:47.007 INFO:tasks.workunit.client.0.smithi175.stdout:./105/.snap/snap-subdir-test 2023-10-10T18:48:47.162 INFO:tasks.workunit.client.0.smithi175.stdout:./210/.snap/snap-subdir-test 2023-10-10T18:48:47.168 INFO:tasks.workunit.client.0.smithi175.stdout:./265/.snap/snap-subdir-test 2023-10-10T18:48:47.176 INFO:tasks.workunit.client.0.smithi175.stdout:./218/.snap/snap-subdir-test 2023-10-10T18:48:47.184 INFO:tasks.workunit.client.0.smithi175.stdout:./290/.snap/snap-subdir-test 2023-10-10T18:48:47.191 INFO:tasks.workunit.client.0.smithi175.stdout:./149/.snap/snap-subdir-test 2023-10-10T18:48:47.199 INFO:tasks.workunit.client.0.smithi175.stdout:./109/.snap/snap-subdir-test 2023-10-10T18:48:47.206 INFO:tasks.workunit.client.0.smithi175.stdout:./292/.snap/snap-subdir-test 2023-10-10T18:48:47.212 INFO:tasks.workunit.client.0.smithi175.stdout:./242/.snap/snap-subdir-test 2023-10-10T18:48:47.576 INFO:tasks.workunit.client.0.smithi175.stdout:./158/.snap/snap-subdir-test 2023-10-10T18:48:47.847 INFO:tasks.workunit.client.0.smithi175.stdout:./243/.snap/snap-subdir-test 2023-10-10T18:48:47.857 INFO:tasks.workunit.client.0.smithi175.stdout:./106/.snap/snap-subdir-test 2023-10-10T18:48:47.920 INFO:tasks.workunit.client.0.smithi175.stdout:./250/.snap/snap-subdir-test 2023-10-10T18:48:47.927 INFO:tasks.workunit.client.0.smithi175.stdout:./221/.snap/snap-subdir-test 2023-10-10T18:48:48.238 INFO:tasks.workunit.client.0.smithi175.stdout:./244/.snap/snap-subdir-test 2023-10-10T18:48:48.245 INFO:tasks.workunit.client.0.smithi175.stdout:./258/.snap/snap-subdir-test 2023-10-10T18:48:48.254 INFO:tasks.workunit.client.0.smithi175.stdout:./268/.snap/snap-subdir-test 2023-10-10T18:48:48.259 INFO:tasks.workunit.client.0.smithi175.stdout:./189/.snap/snap-subdir-test 2023-10-10T18:48:48.266 INFO:tasks.workunit.client.0.smithi175.stdout:./285/.snap/snap-subdir-test 2023-10-10T18:48:48.272 INFO:tasks.workunit.client.0.smithi175.stdout:./208/.snap/snap-subdir-test 2023-10-10T18:48:48.278 INFO:tasks.workunit.client.0.smithi175.stdout:./288/.snap/snap-subdir-test 2023-10-10T18:48:48.286 INFO:tasks.workunit.client.0.smithi175.stdout:./209/.snap/snap-subdir-test 2023-10-10T18:48:48.293 INFO:tasks.workunit.client.0.smithi175.stdout:./253/.snap/snap-subdir-test 2023-10-10T18:48:48.298 INFO:tasks.workunit.client.0.smithi175.stdout:./249/.snap/snap-subdir-test 2023-10-10T18:48:48.306 INFO:tasks.workunit.client.0.smithi175.stdout:./133/.snap/snap-subdir-test 2023-10-10T18:48:48.313 INFO:tasks.workunit.client.0.smithi175.stdout:./291/.snap/snap-subdir-test 2023-10-10T18:48:48.320 INFO:tasks.workunit.client.0.smithi175.stdout:./226/.snap/snap-subdir-test 2023-10-10T18:48:48.328 INFO:tasks.workunit.client.0.smithi175.stdout:./173/.snap/snap-subdir-test 2023-10-10T18:48:48.335 INFO:tasks.workunit.client.0.smithi175.stdout:./174/.snap/snap-subdir-test 2023-10-10T18:48:48.730 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:48:48.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:48:48.830 INFO:tasks.workunit.client.0.smithi175.stdout:./299/.snap/snap-subdir-test 2023-10-10T18:48:48.837 INFO:tasks.workunit.client.0.smithi175.stdout:./262/.snap/snap-subdir-test 2023-10-10T18:48:48.844 INFO:tasks.workunit.client.0.smithi175.stdout:./231/.snap/snap-subdir-test 2023-10-10T18:48:49.144 INFO:tasks.workunit.client.0.smithi175.stdout:./214/.snap/snap-subdir-test 2023-10-10T18:48:49.152 INFO:tasks.workunit.client.0.smithi175.stdout:./281/.snap/snap-subdir-test 2023-10-10T18:48:49.159 INFO:tasks.workunit.client.0.smithi175.stdout:./289/.snap/snap-subdir-test 2023-10-10T18:48:49.165 INFO:tasks.workunit.client.0.smithi175.stdout:./283/.snap/snap-subdir-test 2023-10-10T18:48:49.173 INFO:tasks.workunit.client.0.smithi175.stdout:./232/.snap/snap-subdir-test 2023-10-10T18:48:49.179 INFO:tasks.workunit.client.0.smithi175.stdout:./298/.snap/snap-subdir-test 2023-10-10T18:48:49.187 INFO:tasks.workunit.client.0.smithi175.stdout:./233/.snap/snap-subdir-test 2023-10-10T18:48:49.196 INFO:tasks.workunit.client.0.smithi175.stdout:./155/.snap/snap-subdir-test 2023-10-10T18:48:51.135 INFO:tasks.workunit.client.0.smithi175.stdout:./114/.snap/snap-subdir-test 2023-10-10T18:48:51.142 INFO:tasks.workunit.client.0.smithi175.stdout:./159/.snap/snap-subdir-test 2023-10-10T18:48:51.147 INFO:tasks.workunit.client.0.smithi175.stdout:./187/.snap/snap-subdir-test 2023-10-10T18:48:51.150 INFO:tasks.workunit.client.0.smithi175.stdout:./185/.snap/snap-subdir-test 2023-10-10T18:48:51.156 INFO:tasks.workunit.client.0.smithi175.stdout:./293/.snap/snap-subdir-test 2023-10-10T18:48:52.135 INFO:tasks.workunit.client.0.smithi175.stdout:./228/.snap/snap-subdir-test 2023-10-10T18:48:52.145 INFO:tasks.workunit.client.0.smithi175.stdout:./120/.snap/snap-subdir-test 2023-10-10T18:48:52.245 INFO:tasks.workunit.client.0.smithi175.stdout:./110/.snap/snap-subdir-test 2023-10-10T18:48:52.253 INFO:tasks.workunit.client.0.smithi175.stdout:./267/.snap/snap-subdir-test 2023-10-10T18:48:53.070 INFO:tasks.workunit.client.0.smithi175.stdout:./202/.snap/snap-subdir-test 2023-10-10T18:48:53.073 INFO:tasks.workunit.client.0.smithi175.stdout:./137/.snap/snap-subdir-test 2023-10-10T18:48:53.081 INFO:tasks.workunit.client.0.smithi175.stdout:./246/.snap/snap-subdir-test 2023-10-10T18:48:53.089 INFO:tasks.workunit.client.0.smithi175.stdout:./219/.snap/snap-subdir-test 2023-10-10T18:48:53.094 INFO:tasks.workunit.client.0.smithi175.stdout:./198/.snap/snap-subdir-test 2023-10-10T18:48:53.101 INFO:tasks.workunit.client.0.smithi175.stdout:./136/.snap/snap-subdir-test 2023-10-10T18:48:53.108 INFO:tasks.workunit.client.0.smithi175.stdout:./270/.snap/snap-subdir-test 2023-10-10T18:48:54.009 INFO:tasks.workunit.client.0.smithi175.stdout:./125/.snap/snap-subdir-test 2023-10-10T18:48:54.015 INFO:tasks.workunit.client.0.smithi175.stdout:./194/.snap/snap-subdir-test 2023-10-10T18:48:54.023 INFO:tasks.workunit.client.0.smithi175.stdout:./119/.snap/snap-subdir-test 2023-10-10T18:48:54.029 INFO:tasks.workunit.client.0.smithi175.stdout:./297/.snap/snap-subdir-test 2023-10-10T18:48:54.036 INFO:tasks.workunit.client.0.smithi175.stdout:./230/.snap/snap-subdir-test 2023-10-10T18:48:54.105 INFO:tasks.workunit.client.0.smithi175.stdout:./260/.snap/snap-subdir-test 2023-10-10T18:48:54.112 INFO:tasks.workunit.client.0.smithi175.stdout:./157/.snap/snap-subdir-test 2023-10-10T18:48:54.842 INFO:tasks.workunit.client.0.smithi175.stdout:./172/.snap/snap-subdir-test 2023-10-10T18:48:55.574 INFO:tasks.workunit.client.0.smithi175.stdout:./277/.snap/snap-subdir-test 2023-10-10T18:48:55.582 INFO:tasks.workunit.client.0.smithi175.stdout:./186/.snap/snap-subdir-test 2023-10-10T18:48:55.589 INFO:tasks.workunit.client.0.smithi175.stdout:./237/.snap/snap-subdir-test 2023-10-10T18:48:56.405 INFO:tasks.workunit.client.0.smithi175.stdout:./190/.snap/snap-subdir-test 2023-10-10T18:48:56.414 INFO:tasks.workunit.client.0.smithi175.stdout:./213/.snap/snap-subdir-test 2023-10-10T18:48:56.600 INFO:tasks.workunit.client.0.smithi175.stdout:./104/.snap/snap-subdir-test 2023-10-10T18:48:56.611 INFO:tasks.workunit.client.0.smithi175.stdout:./164/.snap/snap-subdir-test 2023-10-10T18:48:56.619 INFO:tasks.workunit.client.0.smithi175.stdout:./112/.snap/snap-subdir-test 2023-10-10T18:48:56.826 INFO:tasks.workunit.client.0.smithi175.stdout:./295/.snap/snap-subdir-test 2023-10-10T18:48:56.834 INFO:tasks.workunit.client.0.smithi175.stdout:./272/.snap/snap-subdir-test 2023-10-10T18:48:56.840 INFO:tasks.workunit.client.0.smithi175.stdout:./100/.snap/snap-subdir-test 2023-10-10T18:48:56.846 INFO:tasks.workunit.client.0.smithi175.stdout:./134/.snap/snap-subdir-test 2023-10-10T18:48:56.853 INFO:tasks.workunit.client.0.smithi175.stdout:./200/.snap/snap-subdir-test 2023-10-10T18:48:56.861 INFO:tasks.workunit.client.0.smithi175.stdout:./115/.snap/snap-subdir-test 2023-10-10T18:48:56.866 INFO:tasks.workunit.client.0.smithi175.stdout:./284/.snap/snap-subdir-test 2023-10-10T18:48:56.872 INFO:tasks.workunit.client.0.smithi175.stdout:./280/.snap/snap-subdir-test 2023-10-10T18:48:56.879 INFO:tasks.workunit.client.0.smithi175.stdout:./116/.snap/snap-subdir-test 2023-10-10T18:48:56.886 INFO:tasks.workunit.client.0.smithi175.stdout:./245/.snap/snap-subdir-test 2023-10-10T18:48:57.039 INFO:tasks.workunit.client.0.smithi175.stdout:./195/.snap/snap-subdir-test 2023-10-10T18:48:57.047 INFO:tasks.workunit.client.0.smithi175.stdout:./138/.snap/snap-subdir-test 2023-10-10T18:48:57.983 INFO:tasks.workunit.client.0.smithi175.stdout:./135/.snap/snap-subdir-test 2023-10-10T18:48:57.990 INFO:tasks.workunit.client.0.smithi175.stdout:./117/.snap/snap-subdir-test 2023-10-10T18:48:57.998 INFO:tasks.workunit.client.0.smithi175.stdout:./241/.snap/snap-subdir-test 2023-10-10T18:48:58.003 INFO:tasks.workunit.client.0.smithi175.stdout:./143/.snap/snap-subdir-test 2023-10-10T18:48:58.008 INFO:tasks.workunit.client.0.smithi175.stdout:./180/.snap/snap-subdir-test 2023-10-10T18:48:58.015 INFO:tasks.workunit.client.0.smithi175.stdout:./126/.snap/snap-subdir-test 2023-10-10T18:48:58.018 INFO:tasks.workunit.client.0.smithi175.stdout:./161/.snap/snap-subdir-test 2023-10-10T18:48:58.026 INFO:tasks.workunit.client.0.smithi175.stdout:./204/.snap/snap-subdir-test 2023-10-10T18:48:58.030 INFO:tasks.workunit.client.0.smithi175.stdout:./176/.snap/snap-subdir-test 2023-10-10T18:48:58.496 INFO:tasks.workunit.client.0.smithi175.stdout:./255/.snap/snap-subdir-test 2023-10-10T18:48:58.502 INFO:tasks.workunit.client.0.smithi175.stdout:./215/.snap/snap-subdir-test 2023-10-10T18:48:58.768 INFO:tasks.workunit.client.0.smithi175.stdout:./111/.snap/snap-subdir-test 2023-10-10T18:48:58.775 INFO:tasks.workunit.client.0.smithi175.stdout:./238/.snap/snap-subdir-test 2023-10-10T18:48:58.778 INFO:tasks.workunit.client.0.smithi175.stdout:./191/.snap/snap-subdir-test 2023-10-10T18:48:58.782 INFO:tasks.workunit.client.0.smithi175.stdout:./184/.snap/snap-subdir-test 2023-10-10T18:48:58.786 INFO:tasks.workunit.client.0.smithi175.stdout:./192/.snap/snap-subdir-test 2023-10-10T18:48:58.793 INFO:tasks.workunit.client.0.smithi175.stdout:./103/.snap/snap-subdir-test 2023-10-10T18:48:58.800 INFO:tasks.workunit.client.0.smithi175.stdout:./128/.snap/snap-subdir-test 2023-10-10T18:48:59.329 INFO:tasks.workunit.client.0.smithi175.stdout:./229/.snap/snap-subdir-test 2023-10-10T18:48:59.995 INFO:tasks.workunit.client.0.smithi175.stdout:./144/.snap/snap-subdir-test 2023-10-10T18:49:00.003 INFO:tasks.workunit.client.0.smithi175.stdout:./146/.snap/snap-subdir-test 2023-10-10T18:49:00.263 INFO:tasks.workunit.client.0.smithi175.stdout:./227/.snap/snap-subdir-test 2023-10-10T18:49:00.495 INFO:tasks.workunit.client.0.smithi175.stdout:./278/.snap/snap-subdir-test 2023-10-10T18:49:00.501 INFO:tasks.workunit.client.0.smithi175.stdout:./141/.snap/snap-subdir-test 2023-10-10T18:49:00.509 INFO:tasks.workunit.client.0.smithi175.stdout:./234/.snap/snap-subdir-test 2023-10-10T18:49:00.523 INFO:tasks.workunit.client.0.smithi175.stdout:./225/.snap/snap-subdir-test 2023-10-10T18:49:00.526 INFO:tasks.workunit.client.0.smithi175.stdout:./182/.snap/snap-subdir-test 2023-10-10T18:49:01.112 INFO:tasks.workunit.client.0.smithi175.stdout:./107/.snap/snap-subdir-test 2023-10-10T18:49:01.116 INFO:tasks.workunit.client.0.smithi175.stdout:./178/.snap/snap-subdir-test 2023-10-10T18:49:01.121 INFO:tasks.workunit.client.0.smithi175.stdout:./266/.snap/snap-subdir-test 2023-10-10T18:49:01.130 INFO:tasks.workunit.client.0.smithi175.stdout:./240/.snap/snap-subdir-test 2023-10-10T18:49:01.133 INFO:tasks.workunit.client.0.smithi175.stdout:./156/.snap/snap-subdir-test 2023-10-10T18:49:01.399 INFO:tasks.workunit.client.0.smithi175.stdout:./131/.snap/snap-subdir-test 2023-10-10T18:49:01.407 INFO:tasks.workunit.client.0.smithi175.stdout:./123/.snap/snap-subdir-test 2023-10-10T18:49:01.412 INFO:tasks.workunit.client.0.smithi175.stdout:./160/.snap/snap-subdir-test 2023-10-10T18:49:01.415 INFO:tasks.workunit.client.0.smithi175.stdout:./165/.snap/snap-subdir-test 2023-10-10T18:49:01.419 INFO:tasks.workunit.client.0.smithi175.stdout:./101/.snap/snap-subdir-test 2023-10-10T18:49:01.424 INFO:tasks.workunit.client.0.smithi175.stdout:./148/.snap/snap-subdir-test 2023-10-10T18:49:01.428 INFO:tasks.workunit.client.0.smithi175.stdout:./168/.snap/snap-subdir-test 2023-10-10T18:49:01.432 INFO:tasks.workunit.client.0.smithi175.stdout:./179/.snap/snap-subdir-test 2023-10-10T18:49:01.439 INFO:tasks.workunit.client.0.smithi175.stdout:./257/.snap/snap-subdir-test 2023-10-10T18:49:01.447 INFO:tasks.workunit.client.0.smithi175.stdout:./252/.snap/snap-subdir-test 2023-10-10T18:49:01.452 INFO:tasks.workunit.client.0.smithi175.stdout:./132/.snap/snap-subdir-test 2023-10-10T18:49:02.169 INFO:tasks.workunit.client.0.smithi175.stdout:./236/.snap/snap-subdir-test 2023-10-10T18:49:02.180 INFO:tasks.workunit.client.0.smithi175.stdout:./211/.snap/snap-subdir-test 2023-10-10T18:49:02.185 INFO:tasks.workunit.client.0.smithi175.stdout:./199/.snap/snap-subdir-test 2023-10-10T18:49:02.195 INFO:tasks.workunit.client.0.smithi175.stdout:./205/.snap/snap-subdir-test 2023-10-10T18:49:02.199 INFO:tasks.workunit.client.0.smithi175.stdout:./163/.snap/snap-subdir-test 2023-10-10T18:49:02.207 INFO:tasks.workunit.client.0.smithi175.stdout:./261/.snap/snap-subdir-test 2023-10-10T18:49:02.217 INFO:tasks.workunit.client.0.smithi175.stdout:./201/.snap/snap-subdir-test 2023-10-10T18:49:02.452 INFO:tasks.workunit.client.0.smithi175.stdout:./282/.snap/snap-subdir-test 2023-10-10T18:49:02.524 INFO:tasks.workunit.client.0.smithi175.stdout:./296/.snap/snap-subdir-test 2023-10-10T18:49:02.978 INFO:tasks.workunit.client.0.smithi175.stdout:./217/.snap/snap-subdir-test 2023-10-10T18:49:02.985 INFO:tasks.workunit.client.0.smithi175.stdout:./153/.snap/snap-subdir-test 2023-10-10T18:49:02.990 INFO:tasks.workunit.client.0.smithi175.stdout:./206/.snap/snap-subdir-test 2023-10-10T18:49:03.570 INFO:tasks.workunit.client.0.smithi175.stdout:./122/.snap/snap-subdir-test 2023-10-10T18:49:03.576 INFO:tasks.workunit.client.0.smithi175.stdout:./162/.snap/snap-subdir-test 2023-10-10T18:49:03.580 INFO:tasks.workunit.client.0.smithi175.stdout:./197/.snap/snap-subdir-test 2023-10-10T18:49:03.588 INFO:tasks.workunit.client.0.smithi175.stdout:./216/.snap/snap-subdir-test 2023-10-10T18:49:10.585 INFO:tasks.workunit.client.0.smithi175.stdout:./254/.snap/snap-subdir-test 2023-10-10T18:49:10.593 INFO:tasks.workunit.client.0.smithi175.stdout:./113/.snap/snap-subdir-test 2023-10-10T18:49:10.600 INFO:tasks.workunit.client.0.smithi175.stdout:./152/.snap/snap-subdir-test 2023-10-10T18:49:10.607 INFO:tasks.workunit.client.0.smithi175.stdout:./121/.snap/snap-subdir-test 2023-10-10T18:49:10.613 INFO:tasks.workunit.client.0.smithi175.stdout:./118/.snap/snap-subdir-test 2023-10-10T18:49:10.615 INFO:tasks.workunit.client.0.smithi175.stdout:./196/.snap/snap-subdir-test 2023-10-10T18:49:10.883 INFO:tasks.workunit.client.0.smithi175.stdout:./124/.snap/snap-subdir-test 2023-10-10T18:49:10.891 INFO:tasks.workunit.client.0.smithi175.stdout:./239/.snap/snap-subdir-test 2023-10-10T18:49:11.519 INFO:tasks.workunit.client.0.smithi175.stdout:./188/.snap/snap-subdir-test 2023-10-10T18:49:11.527 INFO:tasks.workunit.client.0.smithi175.stdout:./287/.snap/snap-subdir-test 2023-10-10T18:49:11.534 INFO:tasks.workunit.client.0.smithi175.stdout:./275/.snap/snap-subdir-test 2023-10-10T18:49:11.543 INFO:tasks.workunit.client.0.smithi175.stdout:./259/.snap/snap-subdir-test 2023-10-10T18:49:11.551 INFO:tasks.workunit.client.0.smithi175.stdout:./224/.snap/snap-subdir-test 2023-10-10T18:49:11.768 INFO:tasks.workunit.client.0.smithi175.stdout:./264/.snap/snap-subdir-test 2023-10-10T18:49:11.778 INFO:tasks.workunit.client.0.smithi175.stdout:./145/.snap/snap-subdir-test 2023-10-10T18:49:11.785 INFO:tasks.workunit.client.0.smithi175.stdout:./279/.snap/snap-subdir-test 2023-10-10T18:49:11.791 INFO:tasks.workunit.client.0.smithi175.stdout:./263/.snap/snap-subdir-test 2023-10-10T18:49:11.798 INFO:tasks.workunit.client.0.smithi175.stdout:./130/.snap/snap-subdir-test 2023-10-10T18:49:11.802 INFO:tasks.workunit.client.0.smithi175.stdout:./193/.snap/snap-subdir-test 2023-10-10T18:49:11.809 INFO:tasks.workunit.client.0.smithi175.stdout:./223/.snap/snap-subdir-test 2023-10-10T18:49:12.170 INFO:tasks.workunit.client.0.smithi175.stdout:./207/.snap/snap-subdir-test 2023-10-10T18:49:12.177 INFO:tasks.workunit.client.0.smithi175.stdout:./276/.snap/snap-subdir-test 2023-10-10T18:49:12.185 INFO:tasks.workunit.client.0.smithi175.stdout:./271/.snap/snap-subdir-test 2023-10-10T18:49:12.191 INFO:tasks.workunit.client.0.smithi175.stdout:./251/.snap/snap-subdir-test 2023-10-10T18:49:12.198 INFO:tasks.workunit.client.0.smithi175.stdout:./175/.snap/snap-subdir-test 2023-10-10T18:49:12.530 INFO:tasks.workunit.client.0.smithi175.stdout:./222/.snap/snap-subdir-test 2023-10-10T18:49:13.482 INFO:tasks.workunit.client.0.smithi175.stdout:./220/.snap/snap-subdir-test 2023-10-10T18:49:13.492 INFO:tasks.workunit.client.0.smithi175.stdout:./203/.snap/snap-subdir-test 2023-10-10T18:49:13.500 INFO:tasks.workunit.client.0.smithi175.stdout:./181/.snap/snap-subdir-test 2023-10-10T18:49:13.509 INFO:tasks.workunit.client.0.smithi175.stdout:./286/.snap/snap-subdir-test 2023-10-10T18:49:13.517 INFO:tasks.workunit.client.0.smithi175.stdout:./127/.snap/snap-subdir-test 2023-10-10T18:49:13.524 INFO:tasks.workunit.client.0.smithi175.stdout:./108/.snap/snap-subdir-test 2023-10-10T18:49:13.533 INFO:tasks.workunit.client.0.smithi175.stdout:./183/.snap/snap-subdir-test 2023-10-10T18:49:13.705 INFO:tasks.workunit.client.0.smithi175.stdout:./294/.snap/snap-subdir-test 2023-10-10T18:49:13.712 INFO:tasks.workunit.client.0.smithi175.stdout:./235/.snap/snap-subdir-test 2023-10-10T18:49:13.983 INFO:tasks.workunit.client.0.smithi175.stdout:./177/.snap/snap-subdir-test 2023-10-10T18:49:13.992 INFO:tasks.workunit.client.0.smithi175.stdout:./169/.snap/snap-subdir-test 2023-10-10T18:49:13.999 INFO:tasks.workunit.client.0.smithi175.stdout:./212/.snap/snap-subdir-test 2023-10-10T18:49:14.006 INFO:tasks.workunit.client.0.smithi175.stdout:./154/.snap/snap-subdir-test 2023-10-10T18:49:14.879 INFO:tasks.workunit.client.0.smithi175.stdout:./171/.snap/snap-subdir-test 2023-10-10T18:49:14.887 INFO:tasks.workunit.client.0.smithi175.stdout:./139/.snap/snap-subdir-test 2023-10-10T18:49:14.896 INFO:tasks.workunit.client.0.smithi175.stdout:./102/.snap/snap-subdir-test 2023-10-10T18:49:14.904 INFO:tasks.workunit.client.0.smithi175.stdout:./129/.snap/snap-subdir-test 2023-10-10T18:49:18.553 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:18.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.700+0000 7f0212974700 1 Processor -- start 2023-10-10T18:49:18.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.700+0000 7f0212974700 1 -- start start 2023-10-10T18:49:18.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f0212974700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c0c3330 0x7f020c0c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:18.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f0212974700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c1533e0 con 0x7f020c06b640 2023-10-10T18:49:18.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f0212974700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c06e130 con 0x7f020c0c41e0 2023-10-10T18:49:18.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f0212974700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c06a970 con 0x7f020c0c3330 2023-10-10T18:49:18.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f0210f11700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f020c06b640 0x7f020c15d8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40062/0 (socket says 172.21.15.175:40062) 2023-10-10T18:49:18.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f0210f11700 1 -- 172.21.15.175:0/3826752697 learned_addr learned my addr 172.21.15.175:0/3826752697 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:49:18.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020bfff700 1 --2- 172.21.15.175:0/3826752697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c0c3330 0x7f020c0c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:18.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020affd700 1 -- 172.21.15.175:0/3826752697 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783783832 0 0) 0x7f020c1533e0 con 0x7f020c06b640 2023-10-10T18:49:18.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020affd700 1 -- 172.21.15.175:0/3826752697 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f4003570 con 0x7f020c06b640 2023-10-10T18:49:18.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020affd700 1 -- 172.21.15.175:0/3826752697 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354054178 0 0) 0x7f020c06e130 con 0x7f020c0c41e0 2023-10-10T18:49:18.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020affd700 1 -- 172.21.15.175:0/3826752697 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c1533e0 con 0x7f020c0c41e0 2023-10-10T18:49:18.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020bfff700 1 -- 172.21.15.175:0/3826752697 >> v1:172.21.15.191:6789/0 conn(0x7f020c0c41e0 legacy=0x7f020c14cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:18.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020bfff700 1 -- 172.21.15.175:0/3826752697 >> v1:172.21.15.175:6789/0 conn(0x7f020c06b640 legacy=0x7f020c15d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:18.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020bfff700 1 -- 172.21.15.175:0/3826752697 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c06d520 con 0x7f020c0c3330 2023-10-10T18:49:18.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020bfff700 1 --2- 172.21.15.175:0/3826752697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c0c3330 0x7f020c0c3ca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009900 tx=0x7f01fc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0dba57c853054af server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:18.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.701+0000 7f020affd700 1 -- 172.21.15.175:0/3826752697 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc005c10 con 0x7f020c0c3330 2023-10-10T18:49:18.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f020affd700 1 -- 172.21.15.175:0/3826752697 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc005d70 con 0x7f020c0c3330 2023-10-10T18:49:18.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f020affd700 1 -- 172.21.15.175:0/3826752697 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc006260 con 0x7f020c0c3330 2023-10-10T18:49:18.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f0212974700 1 -- 172.21.15.175:0/3826752697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c0c3330 msgr2=0x7f020c0c3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:18.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f0212974700 1 --2- 172.21.15.175:0/3826752697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c0c3330 0x7f020c0c3ca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009900 tx=0x7f01fc009120 comp rx=0 tx=0).stop 2023-10-10T18:49:18.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f0212974700 1 -- 172.21.15.175:0/3826752697 shutdown_connections 2023-10-10T18:49:18.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f0212974700 1 --2- 172.21.15.175:0/3826752697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c0c3330 0x7f020c0c3ca0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:18.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f0212974700 1 -- 172.21.15.175:0/3826752697 >> 172.21.15.175:0/3826752697 conn(0x7f020c068280 msgr2=0x7f020c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:18.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f0212974700 1 -- 172.21.15.175:0/3826752697 shutdown_connections 2023-10-10T18:49:18.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.702+0000 7f0212974700 1 -- 172.21.15.175:0/3826752697 wait complete. 2023-10-10T18:49:18.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.703+0000 7f0212974700 1 Processor -- start 2023-10-10T18:49:18.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.703+0000 7f0212974700 1 -- start start 2023-10-10T18:49:18.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f0212974700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c06b640 0x7f020c1477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:18.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f0212974700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f020c0c3330 0x7f020c069580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:18.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f0212974700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f020c0c41e0 0x7f020c069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:18.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f0212974700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c155d20 con 0x7f020c0c3330 2023-10-10T18:49:18.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f0212974700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c152cb0 con 0x7f020c0c41e0 2023-10-10T18:49:18.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f0212974700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c1509e0 con 0x7f020c06b640 2023-10-10T18:49:18.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f020c0c3330 0x7f020c069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:18.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f020c0c3330 0x7f020c069580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46178/0 (socket says 172.21.15.175:46178) 2023-10-10T18:49:18.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c06b640 0x7f020c1477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:18.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020b7fe700 1 -- 172.21.15.175:0/1956115376 learned_addr learned my addr 172.21.15.175:0/1956115376 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:18.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f0210f11700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f020c0c41e0 0x7f020c069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:18.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020b7fe700 1 -- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c06b640 msgr2=0x7f020c1477f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:18.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020b7fe700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c06b640 0x7f020c1477f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:18.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020b7fe700 1 -- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f020c0c41e0 msgr2=0x7f020c069ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:18.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020b7fe700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f020c0c41e0 0x7f020c069ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:18.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.704+0000 7f020b7fe700 1 -- 172.21.15.175:0/1956115376 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc0093c0 con 0x7f020c0c3330 2023-10-10T18:49:18.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.705+0000 7f020b7fe700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f020c0c3330 0x7f020c069580 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f020000c580 tx=0x7f0200007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:18.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.705+0000 7f01f3fff700 1 -- 172.21.15.175:0/1956115376 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200010510 con 0x7f020c0c3330 2023-10-10T18:49:18.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.705+0000 7f01f3fff700 1 -- 172.21.15.175:0/1956115376 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200010670 con 0x7f020c0c3330 2023-10-10T18:49:18.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.705+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c1520a0 con 0x7f020c0c3330 2023-10-10T18:49:18.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.705+0000 7f01f3fff700 1 -- 172.21.15.175:0/1956115376 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200010a30 con 0x7f020c0c3330 2023-10-10T18:49:18.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.705+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c152660 con 0x7f020c0c3330 2023-10-10T18:49:18.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.706+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020c067570 con 0x7f020c0c3330 2023-10-10T18:49:18.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.706+0000 7f01f3fff700 1 -- 172.21.15.175:0/1956115376 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f020001b400 con 0x7f020c0c3330 2023-10-10T18:49:18.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.706+0000 7f01f3fff700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01f8066f30 0x7f01f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:18.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.707+0000 7f01f3fff700 1 -- 172.21.15.175:0/1956115376 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6107+0+0 (secure 0 0 0) 0x7f0200084250 con 0x7f020c0c3330 2023-10-10T18:49:18.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.707+0000 7f020bfff700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01f8066f30 0x7f01f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:18.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.707+0000 7f020bfff700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01f8066f30 0x7f01f80693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009390 tx=0x7f01fc016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:18.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.711+0000 7f01f3fff700 1 -- 172.21.15.175:0/1956115376 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0200051330 con 0x7f020c0c3330 2023-10-10T18:49:18.803 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:49:18.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:49:18.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.975+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f020c0c1a30 con 0x7f020c0c3330 2023-10-10T18:49:18.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.976+0000 7f01f3fff700 1 -- 172.21.15.175:0/1956115376 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6194 (secure 0 0 0) 0x7f0200054950 con 0x7f020c0c3330 2023-10-10T18:49:18.976 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:01.486855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:18.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01f8066f30 msgr2=0x7f01f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:18.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01f8066f30 0x7f01f80693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009390 tx=0x7f01fc016000 comp rx=0 tx=0).stop 2023-10-10T18:49:18.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f020c0c3330 msgr2=0x7f020c069580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:18.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f020c0c3330 0x7f020c069580 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f020000c580 tx=0x7f0200007e10 comp rx=0 tx=0).stop 2023-10-10T18:49:18.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 shutdown_connections 2023-10-10T18:49:18.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01f8066f30 0x7f01f80693e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:18.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f020c0c41e0 0x7f020c069ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:18.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f020c06b640 0x7f020c1477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:18.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 --2- 172.21.15.175:0/1956115376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f020c0c3330 0x7f020c069580 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:18.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 >> 172.21.15.175:0/1956115376 conn(0x7f020c068280 msgr2=0x7f020c14cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:18.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 shutdown_connections 2023-10-10T18:49:18.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:18.978+0000 7f0212974700 1 -- 172.21.15.175:0/1956115376 wait complete. 2023-10-10T18:49:18.981 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 33 2023-10-10T18:49:18.990 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=4) 2023-10-10T18:49:18.991 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-10T18:49:18.991 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T18:49:19.002 INFO:tasks.ceph.mds.a:Stopped 2023-10-10T18:49:19.002 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:19.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.156+0000 7fd25c93e700 1 Processor -- start 2023-10-10T18:49:19.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.156+0000 7fd25c93e700 1 -- start start 2023-10-10T18:49:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd25c93e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd258068c00 0x7fd258159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd25c93e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd258067fa0 con 0x7fd25814fd90 2023-10-10T18:49:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd25c93e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd258156b70 con 0x7fd25808a100 2023-10-10T18:49:19.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd25c93e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2580c5600 con 0x7fd258068c00 2023-10-10T18:49:19.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2577fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd25814fd90 0x7fd2580c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40066/0 (socket says 172.21.15.175:40066) 2023-10-10T18:49:19.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2577fe700 1 -- 172.21.15.175:0/390259123 learned_addr learned my addr 172.21.15.175:0/390259123 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:49:19.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd257fff700 1 --2- 172.21.15.175:0/390259123 >> v2:172.21.15.191:3300/0 conn(0x7fd258068c00 0x7fd258159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:19.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956650145 0 0) 0x7fd258067fa0 con 0x7fd25814fd90 2023-10-10T18:49:19.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd240003570 con 0x7fd25814fd90 2023-10-10T18:49:19.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 109453973 0 0) 0x7fd240003570 con 0x7fd25814fd90 2023-10-10T18:49:19.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 >> v2:172.21.15.191:3300/0 conn(0x7fd258068c00 msgr2=0x7fd258159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:19.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2567fc700 1 --2- 172.21.15.175:0/390259123 >> v2:172.21.15.191:3300/0 conn(0x7fd258068c00 0x7fd258159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 >> v1:172.21.15.191:6789/0 conn(0x7fd25808a100 legacy=0x7fd2580684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:19.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.157+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258150160 con 0x7fd25814fd90 2023-10-10T18:49:19.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fd2480017f0 con 0x7fd25814fd90 2023-10-10T18:49:19.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd257fff700 1 --2- 172.21.15.175:0/390259123 >> v2:172.21.15.191:3300/0 conn(0x7fd258068c00 0x7fd258159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:49:19.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 >> v1:172.21.15.175:6789/0 conn(0x7fd25814fd90 legacy=0x7fd2580c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:19.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd240003a60 0x7fd240003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:19.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd240004810 0x7fd240004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:19.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2400051c0 0x7fd24000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:19.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2580c5600 con 0x7fd2400051c0 2023-10-10T18:49:19.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24000c0d0 con 0x7fd240003a60 2023-10-10T18:49:19.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24000c290 con 0x7fd240004810 2023-10-10T18:49:19.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2577fe700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd240003a60 0x7fd240003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:19.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd257fff700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2400051c0 0x7fd24000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:19.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd256ffd700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd240004810 0x7fd240004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:19.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2577fe700 1 -- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd240004810 msgr2=0x7fd240004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:19.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2577fe700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd240004810 0x7fd240004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2577fe700 1 -- 172.21.15.175:0/390259123 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2400051c0 msgr2=0x7fd24000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:19.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2577fe700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2400051c0 0x7fd24000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd257fff700 1 -- 172.21.15.175:0/390259123 reap_dead start 2023-10-10T18:49:19.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2577fe700 1 -- 172.21.15.175:0/390259123 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd240003570 con 0x7fd240003a60 2023-10-10T18:49:19.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd256ffd700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd240004810 0x7fd240004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:49:19.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2577fe700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd240003a60 0x7fd240003e70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd24800a980 tx=0x7fd258156ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd1ea7b27ab033fd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:19.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2480070f0 con 0x7fd240003a60 2023-10-10T18:49:19.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd248007250 con 0x7fd240003a60 2023-10-10T18:49:19.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.158+0000 7fd2567fc700 1 -- 172.21.15.175:0/390259123 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248007610 con 0x7fd240003a60 2023-10-10T18:49:19.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.159+0000 7fd25c93e700 1 -- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd240003a60 msgr2=0x7fd240003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.159+0000 7fd25c93e700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd240003a60 0x7fd240003e70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd24800a980 tx=0x7fd258156ae0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.159+0000 7fd25c93e700 1 -- 172.21.15.175:0/390259123 shutdown_connections 2023-10-10T18:49:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.159+0000 7fd25c93e700 1 --2- 172.21.15.175:0/390259123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd240003a60 0x7fd240003e70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.159+0000 7fd25c93e700 1 -- 172.21.15.175:0/390259123 >> 172.21.15.175:0/390259123 conn(0x7fd258063d00 msgr2=0x7fd2580cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:19.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.159+0000 7fd25c93e700 1 -- 172.21.15.175:0/390259123 shutdown_connections 2023-10-10T18:49:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.159+0000 7fd25c93e700 1 -- 172.21.15.175:0/390259123 wait complete. 2023-10-10T18:49:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.160+0000 7fd25c93e700 1 Processor -- start 2023-10-10T18:49:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.160+0000 7fd25c93e700 1 -- start start 2023-10-10T18:49:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd25c93e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd240003a60 0x7fd258153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:19.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd25c93e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd258150730 0x7fd25814c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd25c93e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd258150b80 0x7fd25814ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd25c93e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd258165bd0 con 0x7fd240003a60 2023-10-10T18:49:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd25c93e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25814d1f0 con 0x7fd258150b80 2023-10-10T18:49:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd25c93e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2580b20f0 con 0x7fd258150730 2023-10-10T18:49:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd240003a60 0x7fd258153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:19.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd240003a60 0x7fd258153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46186/0 (socket says 172.21.15.175:46186) 2023-10-10T18:49:19.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 -- 172.21.15.175:0/3379104615 learned_addr learned my addr 172.21.15.175:0/3379104615 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:19.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd257fff700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd258150b80 0x7fd25814ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:19.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd256ffd700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd258150730 0x7fd25814c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:19.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 -- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd258150730 msgr2=0x7fd25814c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd258150730 0x7fd25814c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 -- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd258150b80 msgr2=0x7fd25814ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd258150b80 0x7fd25814ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 -- 172.21.15.175:0/3379104615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258165e90 con 0x7fd240003a60 2023-10-10T18:49:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd2577fe700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd240003a60 0x7fd258153070 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd24800af70 tx=0x7fd248013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:19.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd23f7fe700 1 -- 172.21.15.175:0/3379104615 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248007070 con 0x7fd240003a60 2023-10-10T18:49:19.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.161+0000 7fd23f7fe700 1 -- 172.21.15.175:0/3379104615 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2480019d0 con 0x7fd240003a60 2023-10-10T18:49:19.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.162+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd258152590 con 0x7fd240003a60 2023-10-10T18:49:19.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.162+0000 7fd23f7fe700 1 -- 172.21.15.175:0/3379104615 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248001b40 con 0x7fd240003a60 2023-10-10T18:49:19.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.162+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd258152b50 con 0x7fd240003a60 2023-10-10T18:49:19.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.163+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd258067570 con 0x7fd240003a60 2023-10-10T18:49:19.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.163+0000 7fd23f7fe700 1 -- 172.21.15.175:0/3379104615 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd248024070 con 0x7fd240003a60 2023-10-10T18:49:19.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.163+0000 7fd23f7fe700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd244066f30 0x7fd2440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:19.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.163+0000 7fd23f7fe700 1 -- 172.21.15.175:0/3379104615 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6107+0+0 (secure 0 0 0) 0x7fd248086f80 con 0x7fd240003a60 2023-10-10T18:49:19.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.163+0000 7fd256ffd700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd244066f30 0x7fd2440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:19.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.164+0000 7fd256ffd700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd244066f30 0x7fd2440693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd24c007bc0 tx=0x7fd24c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:19.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.167+0000 7fd23f7fe700 1 -- 172.21.15.175:0/3379104615 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2480b85a0 con 0x7fd240003a60 2023-10-10T18:49:19.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.392+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd258155f80 con 0x7fd240003a60 2023-10-10T18:49:19.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.392+0000 7fd23f7fe700 1 -- 172.21.15.175:0/3379104615 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6194 (secure 0 0 0) 0x7fd248054060 con 0x7fd240003a60 2023-10-10T18:49:19.393 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:19.393 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:01.486855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:19.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd244066f30 msgr2=0x7fd2440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:19.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd244066f30 0x7fd2440693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd24c007bc0 tx=0x7fd24c009040 comp rx=0 tx=0).stop 2023-10-10T18:49:19.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd240003a60 msgr2=0x7fd258153070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:19.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd240003a60 0x7fd258153070 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd24800af70 tx=0x7fd248013000 comp rx=0 tx=0).stop 2023-10-10T18:49:19.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 shutdown_connections 2023-10-10T18:49:19.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd244066f30 0x7fd2440693e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd258150b80 0x7fd25814ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd240003a60 0x7fd258153070 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 --2- 172.21.15.175:0/3379104615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd258150730 0x7fd25814c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:19.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.394+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 >> 172.21.15.175:0/3379104615 conn(0x7fd258063d00 msgr2=0x7fd25814ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:19.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.395+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 shutdown_connections 2023-10-10T18:49:19.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:19.395+0000 7fd25c93e700 1 -- 172.21.15.175:0/3379104615 wait complete. 2023-10-10T18:49:19.397 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 33 2023-10-10T18:49:19.407 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T18:49:21.407 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:21.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.524+0000 7f2f468a8700 1 Processor -- start 2023-10-10T18:49:21.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.524+0000 7f2f468a8700 1 -- start start 2023-10-10T18:49:21.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.524+0000 7f2f468a8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 0x7f2f40155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:21.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.524+0000 7f2f468a8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2f400b2390 0x7f2f40068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:21.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.524+0000 7f2f468a8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f40068bd0 0x7f2f400aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:21.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.524+0000 7f2f468a8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f40150770 con 0x7f2f40068bd0 2023-10-10T18:49:21.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.524+0000 7f2f468a8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f401574b0 con 0x7f2f400b2390 2023-10-10T18:49:21.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.524+0000 7f2f468a8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400afad0 con 0x7f2f400b1fc0 2023-10-10T18:49:21.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f40068bd0 0x7f2f400aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:21.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f40068bd0 0x7f2f400aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46202/0 (socket says 172.21.15.175:46202) 2023-10-10T18:49:21.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 -- 172.21.15.175:0/3641367332 learned_addr learned my addr 172.21.15.175:0/3641367332 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:21.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f3f7fe700 1 --2- 172.21.15.175:0/3641367332 >> v2:172.21.15.191:3300/0 conn(0x7f2f400b2390 0x7f2f40068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:21.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f3ffff700 1 --2- 172.21.15.175:0/3641367332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 0x7f2f40155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:21.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 -- 172.21.15.175:0/3641367332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 msgr2=0x7f2f40155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:21.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 --2- 172.21.15.175:0/3641367332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 0x7f2f40155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 -- 172.21.15.175:0/3641367332 >> v2:172.21.15.191:3300/0 conn(0x7f2f400b2390 msgr2=0x7f2f40068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:21.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 --2- 172.21.15.175:0/3641367332 >> v2:172.21.15.191:3300/0 conn(0x7f2f400b2390 0x7f2f40068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 -- 172.21.15.175:0/3641367332 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f400af4c0 con 0x7f2f40068bd0 2023-10-10T18:49:21.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f44e45700 1 --2- 172.21.15.175:0/3641367332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f40068bd0 0x7f2f400aff60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2f40071640 tx=0x7f2f38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3975e431efda4ab1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:21.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f3effd700 1 -- 172.21.15.175:0/3641367332 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800e070 con 0x7f2f40068bd0 2023-10-10T18:49:21.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f3effd700 1 -- 172.21.15.175:0/3641367332 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3800ba00 con 0x7f2f40068bd0 2023-10-10T18:49:21.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.525+0000 7f2f3effd700 1 -- 172.21.15.175:0/3641367332 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f380092e0 con 0x7f2f40068bd0 2023-10-10T18:49:21.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 -- 172.21.15.175:0/3641367332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f40068bd0 msgr2=0x7f2f400aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:21.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3641367332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f40068bd0 0x7f2f400aff60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2f40071640 tx=0x7f2f38009840 comp rx=0 tx=0).stop 2023-10-10T18:49:21.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 -- 172.21.15.175:0/3641367332 shutdown_connections 2023-10-10T18:49:21.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3641367332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f40068bd0 0x7f2f400aff60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3641367332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 0x7f2f40155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3641367332 >> v2:172.21.15.191:3300/0 conn(0x7f2f400b2390 0x7f2f40068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 -- 172.21.15.175:0/3641367332 >> 172.21.15.175:0/3641367332 conn(0x7f2f40063230 msgr2=0x7f2f40063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:21.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 -- 172.21.15.175:0/3641367332 shutdown_connections 2023-10-10T18:49:21.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.526+0000 7f2f468a8700 1 -- 172.21.15.175:0/3641367332 wait complete. 2023-10-10T18:49:21.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.527+0000 7f2f468a8700 1 Processor -- start 2023-10-10T18:49:21.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.527+0000 7f2f468a8700 1 -- start start 2023-10-10T18:49:21.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f468a8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f40068bd0 0x7f2f40165630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:21.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f468a8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 0x7f2f4014fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:21.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f468a8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f400b2390 0x7f2f4014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:21.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f468a8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40166200 con 0x7f2f400b2390 2023-10-10T18:49:21.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f468a8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400caf60 con 0x7f2f40068bd0 2023-10-10T18:49:21.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f468a8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40156010 con 0x7f2f400b1fc0 2023-10-10T18:49:21.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f3ffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f40068bd0 0x7f2f40165630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:21.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f44e45700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f400b2390 0x7f2f4014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:21.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f3ffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f40068bd0 0x7f2f40165630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:44118/0 (socket says 172.21.15.175:44118) 2023-10-10T18:49:21.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f3f7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 0x7f2f4014fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:21.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f3ffff700 1 -- 172.21.15.175:0/3372140547 learned_addr learned my addr 172.21.15.175:0/3372140547 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:21.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f44e45700 1 -- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 msgr2=0x7f2f4014fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:21.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f44e45700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 0x7f2f4014fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f44e45700 1 -- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f40068bd0 msgr2=0x7f2f40165630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:21.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f44e45700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f40068bd0 0x7f2f40165630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f44e45700 1 -- 172.21.15.175:0/3372140547 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f38009ae0 con 0x7f2f400b2390 2023-10-10T18:49:21.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f44e45700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f400b2390 0x7f2f4014fff0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2f400afa40 tx=0x7f2f38018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:21.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f27fff700 1 -- 172.21.15.175:0/3372140547 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38008ad0 con 0x7f2f400b2390 2023-10-10T18:49:21.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.528+0000 7f2f27fff700 1 -- 172.21.15.175:0/3372140547 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f38008c90 con 0x7f2f400b2390 2023-10-10T18:49:21.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.529+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f40165f20 con 0x7f2f400b2390 2023-10-10T18:49:21.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.529+0000 7f2f27fff700 1 -- 172.21.15.175:0/3372140547 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3801d670 con 0x7f2f400b2390 2023-10-10T18:49:21.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.529+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f400cb430 con 0x7f2f400b2390 2023-10-10T18:49:21.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.530+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f40067570 con 0x7f2f400b2390 2023-10-10T18:49:21.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.530+0000 7f2f27fff700 1 -- 172.21.15.175:0/3372140547 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2f38003040 con 0x7f2f400b2390 2023-10-10T18:49:21.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.530+0000 7f2f27fff700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f2c066f30 0x7f2f2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:21.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.530+0000 7f2f27fff700 1 -- 172.21.15.175:0/3372140547 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6635+0+0 (secure 0 0 0) 0x7f2f380865d0 con 0x7f2f400b2390 2023-10-10T18:49:21.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.530+0000 7f2f3ffff700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f2c066f30 0x7f2f2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:21.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.531+0000 7f2f3ffff700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f2c066f30 0x7f2f2c0693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009e70 tx=0x7f2f300091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:21.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.535+0000 7f2f27fff700 1 -- 172.21.15.175:0/3372140547 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f380534a0 con 0x7f2f400b2390 2023-10-10T18:49:21.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.761+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f40069060 con 0x7f2f400b2390 2023-10-10T18:49:21.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.762+0000 7f2f27fff700 1 -- 172.21.15.175:0/3372140547 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6194 (secure 0 0 0) 0x7f2f38056ac0 con 0x7f2f400b2390 2023-10-10T18:49:21.762 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:21.762 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:01.486855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:21.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f2c066f30 msgr2=0x7f2f2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:21.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f2c066f30 0x7f2f2c0693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009e70 tx=0x7f2f300091a0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f400b2390 msgr2=0x7f2f4014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:21.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f400b2390 0x7f2f4014fff0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2f400afa40 tx=0x7f2f38018000 comp rx=0 tx=0).stop 2023-10-10T18:49:21.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 shutdown_connections 2023-10-10T18:49:21.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2f2c066f30 0x7f2f2c0693e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f400b2390 0x7f2f4014fff0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f40068bd0 0x7f2f40165630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 --2- 172.21.15.175:0/3372140547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f400b1fc0 0x7f2f4014fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:21.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 >> 172.21.15.175:0/3372140547 conn(0x7f2f40063230 msgr2=0x7f2f40151f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:21.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 shutdown_connections 2023-10-10T18:49:21.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:21.764+0000 7f2f468a8700 1 -- 172.21.15.175:0/3372140547 wait complete. 2023-10-10T18:49:21.767 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 33 2023-10-10T18:49:21.776 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T18:49:23.777 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:23.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.904+0000 7f52bde1f700 1 Processor -- start 2023-10-10T18:49:23.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.904+0000 7f52bde1f700 1 -- start start 2023-10-10T18:49:23.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.904+0000 7f52bde1f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52b80b2390 0x7f52b8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.904+0000 7f52bde1f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52b80aff60 0x7f52b80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.904+0000 7f52bde1f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b81574b0 con 0x7f52b80b1fc0 2023-10-10T18:49:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.904+0000 7f52bde1f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b8150770 con 0x7f52b80aff60 2023-10-10T18:49:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.904+0000 7f52bde1f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b80afad0 con 0x7f52b80b2390 2023-10-10T18:49:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52b80aff60 0x7f52b80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52b80aff60 0x7f52b80b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46228/0 (socket says 172.21.15.175:46228) 2023-10-10T18:49:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b7fff700 1 -- 172.21.15.175:0/645958904 learned_addr learned my addr 172.21.15.175:0/645958904 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52aeffd700 1 --2- 172.21.15.175:0/645958904 >> v2:172.21.15.191:3300/0 conn(0x7f52b80b2390 0x7f52b8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b7fff700 1 -- 172.21.15.175:0/645958904 >> v2:172.21.15.191:3300/0 conn(0x7f52b80b2390 msgr2=0x7f52b8155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:23.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b7fff700 1 --2- 172.21.15.175:0/645958904 >> v2:172.21.15.191:3300/0 conn(0x7f52b80b2390 0x7f52b8155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:23.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b7fff700 1 -- 172.21.15.175:0/645958904 >> v1:172.21.15.191:6789/0 conn(0x7f52b80b1fc0 legacy=0x7f52b8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:23.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b7fff700 1 -- 172.21.15.175:0/645958904 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80af4c0 con 0x7f52b80aff60 2023-10-10T18:49:23.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b7fff700 1 --2- 172.21.15.175:0/645958904 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b80b03e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f52b80c0290 tx=0x7f52b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8839cb9ef426db server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:23.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b6ffd700 1 -- 172.21.15.175:0/645958904 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b000e070 con 0x7f52b80aff60 2023-10-10T18:49:23.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b6ffd700 1 -- 172.21.15.175:0/645958904 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b000baa0 con 0x7f52b80aff60 2023-10-10T18:49:23.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.905+0000 7f52b6ffd700 1 -- 172.21.15.175:0/645958904 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00092e0 con 0x7f52b80aff60 2023-10-10T18:49:23.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.906+0000 7f52bde1f700 1 -- 172.21.15.175:0/645958904 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 msgr2=0x7f52b80b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:23.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.906+0000 7f52bde1f700 1 --2- 172.21.15.175:0/645958904 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b80b03e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f52b80c0290 tx=0x7f52b0009840 comp rx=0 tx=0).stop 2023-10-10T18:49:23.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.906+0000 7f52bde1f700 1 -- 172.21.15.175:0/645958904 shutdown_connections 2023-10-10T18:49:23.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.906+0000 7f52bde1f700 1 --2- 172.21.15.175:0/645958904 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b80b03e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:23.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.906+0000 7f52bde1f700 1 --2- 172.21.15.175:0/645958904 >> v2:172.21.15.191:3300/0 conn(0x7f52b80b2390 0x7f52b8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:23.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.906+0000 7f52bde1f700 1 -- 172.21.15.175:0/645958904 >> 172.21.15.175:0/645958904 conn(0x7f52b8063230 msgr2=0x7f52b8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:23.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.906+0000 7f52bde1f700 1 -- 172.21.15.175:0/645958904 shutdown_connections 2023-10-10T18:49:23.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.906+0000 7f52bde1f700 1 -- 172.21.15.175:0/645958904 wait complete. 2023-10-10T18:49:23.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.907+0000 7f52bde1f700 1 Processor -- start 2023-10-10T18:49:23.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.907+0000 7f52bde1f700 1 -- start start 2023-10-10T18:49:23.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52bde1f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b8166380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52bde1f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b80b1fc0 0x7f52b8155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52bde1f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80b2390 0x7f52b81595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52bde1f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b80c3b20 con 0x7f52b80aff60 2023-10-10T18:49:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52bde1f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b81668c0 con 0x7f52b80b1fc0 2023-10-10T18:49:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52bde1f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b80cbd20 con 0x7f52b80b2390 2023-10-10T18:49:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b8166380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b8166380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46244/0 (socket says 172.21.15.175:46244) 2023-10-10T18:49:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 -- 172.21.15.175:0/55974630 learned_addr learned my addr 172.21.15.175:0/55974630 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52aeffd700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b80b1fc0 0x7f52b8155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 -- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80b2390 msgr2=0x7f52b81595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80b2390 0x7f52b81595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 -- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b80b1fc0 msgr2=0x7f52b8155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b80b1fc0 0x7f52b8155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:23.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 -- 172.21.15.175:0/55974630 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0009ae0 con 0x7f52b80aff60 2023-10-10T18:49:23.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52b77fe700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b8166380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f52a800b600 tx=0x7f52a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:23.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52affff700 1 -- 172.21.15.175:0/55974630 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a800f510 con 0x7f52b80aff60 2023-10-10T18:49:23.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52affff700 1 -- 172.21.15.175:0/55974630 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a800f690 con 0x7f52b80aff60 2023-10-10T18:49:23.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b8155dc0 con 0x7f52b80aff60 2023-10-10T18:49:23.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.908+0000 7f52affff700 1 -- 172.21.15.175:0/55974630 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a800fa50 con 0x7f52b80aff60 2023-10-10T18:49:23.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.909+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b80ccf50 con 0x7f52b80aff60 2023-10-10T18:49:23.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.910+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b80713a0 con 0x7f52b80aff60 2023-10-10T18:49:23.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.910+0000 7f52affff700 1 -- 172.21.15.175:0/55974630 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f52a801a400 con 0x7f52b80aff60 2023-10-10T18:49:23.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.910+0000 7f52affff700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f529c066f30 0x7f529c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:23.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.910+0000 7f52affff700 1 -- 172.21.15.175:0/55974630 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6395+0+0 (secure 0 0 0) 0x7f52a8084200 con 0x7f52b80aff60 2023-10-10T18:49:23.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.911+0000 7f52aeffd700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f529c066f30 0x7f529c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:23.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.911+0000 7f52aeffd700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f529c066f30 0x7f529c0693e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f52a0006fd0 tx=0x7f52a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:23.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:23.915+0000 7f52affff700 1 -- 172.21.15.175:0/55974630 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f52b80713a0 con 0x7f52b80aff60 2023-10-10T18:49:24.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.139+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52b80713a0 con 0x7f52b80aff60 2023-10-10T18:49:24.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.139+0000 7f52affff700 1 -- 172.21.15.175:0/55974630 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6194 (secure 0 0 0) 0x7f52b80713a0 con 0x7f52b80aff60 2023-10-10T18:49:24.140 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:24.140 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:01.486855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:24.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f529c066f30 msgr2=0x7f529c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:24.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f529c066f30 0x7f529c0693e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f52a0006fd0 tx=0x7f52a0008040 comp rx=0 tx=0).stop 2023-10-10T18:49:24.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 msgr2=0x7f52b8166380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:24.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b8166380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f52a800b600 tx=0x7f52a80091a0 comp rx=0 tx=0).stop 2023-10-10T18:49:24.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 shutdown_connections 2023-10-10T18:49:24.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f529c066f30 0x7f529c0693e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:24.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80b2390 0x7f52b81595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:24.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52b80aff60 0x7f52b8166380 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:24.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 --2- 172.21.15.175:0/55974630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b80b1fc0 0x7f52b8155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:24.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 >> 172.21.15.175:0/55974630 conn(0x7f52b8063230 msgr2=0x7f52b814fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:24.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 shutdown_connections 2023-10-10T18:49:24.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:24.142+0000 7f52bde1f700 1 -- 172.21.15.175:0/55974630 wait complete. 2023-10-10T18:49:24.145 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 33 2023-10-10T18:49:24.154 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T18:49:26.155 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:26.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.282+0000 7f38a3f91700 1 Processor -- start 2023-10-10T18:49:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.283+0000 7f38a3f91700 1 -- start start 2023-10-10T18:49:26.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.283+0000 7f38a3f91700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c1577a0 0x7f389c0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:26.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.283+0000 7f38a3f91700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c154aa0 con 0x7f389c0c12a0 2023-10-10T18:49:26.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.283+0000 7f38a3f91700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c063dc0 con 0x7f389c1573d0 2023-10-10T18:49:26.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.283+0000 7f38a3f91700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389c0bfdf0 con 0x7f389c1577a0 2023-10-10T18:49:26.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.283+0000 7f38a152c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f389c0c12a0 0x7f389c14d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37504/0 (socket says 172.21.15.175:37504) 2023-10-10T18:49:26.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a152c700 1 -- 172.21.15.175:0/3075732524 learned_addr learned my addr 172.21.15.175:0/3075732524 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:49:26.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a252e700 1 --2- 172.21.15.175:0/3075732524 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c1577a0 0x7f389c0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:26.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a0d2b700 1 -- 172.21.15.175:0/3075732524 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328060907 0 0) 0x7f389c154aa0 con 0x7f389c0c12a0 2023-10-10T18:49:26.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a0d2b700 1 -- 172.21.15.175:0/3075732524 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3888003570 con 0x7f389c0c12a0 2023-10-10T18:49:26.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a0d2b700 1 -- 172.21.15.175:0/3075732524 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674736028 0 0) 0x7f389c063dc0 con 0x7f389c1573d0 2023-10-10T18:49:26.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a0d2b700 1 -- 172.21.15.175:0/3075732524 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c154aa0 con 0x7f389c1573d0 2023-10-10T18:49:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a252e700 1 -- 172.21.15.175:0/3075732524 >> v1:172.21.15.191:6789/0 conn(0x7f389c1573d0 legacy=0x7f389c0bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a252e700 1 -- 172.21.15.175:0/3075732524 >> v1:172.21.15.175:6789/0 conn(0x7f389c0c12a0 legacy=0x7f389c14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a252e700 1 -- 172.21.15.175:0/3075732524 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c157bc0 con 0x7f389c1577a0 2023-10-10T18:49:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a252e700 1 --2- 172.21.15.175:0/3075732524 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c1577a0 0x7f389c0c4f20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3894009e90 tx=0x7f3894009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=171097ae286a7d41 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a0d2b700 1 -- 172.21.15.175:0/3075732524 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389400e070 con 0x7f389c1577a0 2023-10-10T18:49:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a0d2b700 1 -- 172.21.15.175:0/3075732524 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389400bb80 con 0x7f389c1577a0 2023-10-10T18:49:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.284+0000 7f38a0d2b700 1 -- 172.21.15.175:0/3075732524 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894012430 con 0x7f389c1577a0 2023-10-10T18:49:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.285+0000 7f38a3f91700 1 -- 172.21.15.175:0/3075732524 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c1577a0 msgr2=0x7f389c0c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.285+0000 7f38a3f91700 1 --2- 172.21.15.175:0/3075732524 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c1577a0 0x7f389c0c4f20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3894009e90 tx=0x7f3894009780 comp rx=0 tx=0).stop 2023-10-10T18:49:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.285+0000 7f38a3f91700 1 -- 172.21.15.175:0/3075732524 shutdown_connections 2023-10-10T18:49:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.285+0000 7f38a3f91700 1 --2- 172.21.15.175:0/3075732524 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c1577a0 0x7f389c0c4f20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.285+0000 7f38a3f91700 1 -- 172.21.15.175:0/3075732524 >> 172.21.15.175:0/3075732524 conn(0x7f389c06dc70 msgr2=0x7f389c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.285+0000 7f38a3f91700 1 -- 172.21.15.175:0/3075732524 shutdown_connections 2023-10-10T18:49:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.285+0000 7f38a3f91700 1 -- 172.21.15.175:0/3075732524 wait complete. 2023-10-10T18:49:26.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.286+0000 7f38a3f91700 1 Processor -- start 2023-10-10T18:49:26.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.286+0000 7f38a3f91700 1 -- start start 2023-10-10T18:49:26.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a3f91700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c0c12a0 0x7f389c167420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:26.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a3f91700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f389c1573d0 0x7f389c14c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:26.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a3f91700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f389c1577a0 0x7f389c14c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:26.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a3f91700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c154160 con 0x7f389c1573d0 2023-10-10T18:49:26.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a3f91700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c166f90 con 0x7f389c1577a0 2023-10-10T18:49:26.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a3f91700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389c1669d0 con 0x7f389c0c12a0 2023-10-10T18:49:26.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f389c1573d0 0x7f389c14c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:26.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f389c1573d0 0x7f389c14c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56408/0 (socket says 172.21.15.175:56408) 2023-10-10T18:49:26.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 -- 172.21.15.175:0/3111022928 learned_addr learned my addr 172.21.15.175:0/3111022928 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:26.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a1d2d700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c0c12a0 0x7f389c167420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:26.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a252e700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f389c1577a0 0x7f389c14c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:26.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 -- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c0c12a0 msgr2=0x7f389c167420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:26.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c0c12a0 0x7f389c167420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:26.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 -- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f389c1577a0 msgr2=0x7f389c14c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:26.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f389c1577a0 0x7f389c14c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:26.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 -- 172.21.15.175:0/3111022928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3894009a20 con 0x7f389c1573d0 2023-10-10T18:49:26.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a152c700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f389c1573d0 0x7f389c14c1d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c7c0 tx=0x7f388c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:26.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f3891ffb700 1 -- 172.21.15.175:0/3111022928 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c005b30 con 0x7f389c1573d0 2023-10-10T18:49:26.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f3891ffb700 1 -- 172.21.15.175:0/3111022928 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388c005c90 con 0x7f389c1573d0 2023-10-10T18:49:26.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.287+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c156280 con 0x7f389c1573d0 2023-10-10T18:49:26.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.288+0000 7f3891ffb700 1 -- 172.21.15.175:0/3111022928 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c007300 con 0x7f389c1573d0 2023-10-10T18:49:26.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.288+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c1567c0 con 0x7f389c1573d0 2023-10-10T18:49:26.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.289+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c067570 con 0x7f389c1573d0 2023-10-10T18:49:26.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.289+0000 7f3891ffb700 1 -- 172.21.15.175:0/3111022928 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f388c007670 con 0x7f389c1573d0 2023-10-10T18:49:26.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.289+0000 7f3891ffb700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3880066f30 0x7f38800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:26.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.290+0000 7f3891ffb700 1 -- 172.21.15.175:0/3111022928 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6395+0+0 (secure 0 0 0) 0x7f388c083cd0 con 0x7f389c1573d0 2023-10-10T18:49:26.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.290+0000 7f38a1d2d700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3880066f30 0x7f38800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:26.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.290+0000 7f38a1d2d700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3880066f30 0x7f38800693e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3898007fd0 tx=0x7f3898007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:26.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.294+0000 7f3891ffb700 1 -- 172.21.15.175:0/3111022928 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f388c050c90 con 0x7f389c1573d0 2023-10-10T18:49:26.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.522+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f389c157e50 con 0x7f389c1573d0 2023-10-10T18:49:26.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.523+0000 7f3891ffb700 1 -- 172.21.15.175:0/3111022928 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6194 (secure 0 0 0) 0x7f388c017020 con 0x7f389c1573d0 2023-10-10T18:49:26.523 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:26.524 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:01.486855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:26.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3880066f30 msgr2=0x7f38800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:26.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3880066f30 0x7f38800693e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3898007fd0 tx=0x7f3898007d50 comp rx=0 tx=0).stop 2023-10-10T18:49:26.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f389c1573d0 msgr2=0x7f389c14c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:26.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f389c1573d0 0x7f389c14c1d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c7c0 tx=0x7f388c007ea0 comp rx=0 tx=0).stop 2023-10-10T18:49:26.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 shutdown_connections 2023-10-10T18:49:26.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3880066f30 0x7f38800693e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:26.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f389c1577a0 0x7f389c14c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:26.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f389c0c12a0 0x7f389c167420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:26.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 --2- 172.21.15.175:0/3111022928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f389c1573d0 0x7f389c14c1d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:26.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 >> 172.21.15.175:0/3111022928 conn(0x7f389c06dc70 msgr2=0x7f389c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:26.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 shutdown_connections 2023-10-10T18:49:26.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:26.525+0000 7f38a3f91700 1 -- 172.21.15.175:0/3111022928 wait complete. 2023-10-10T18:49:26.528 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 33 2023-10-10T18:49:26.537 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T18:49:28.538 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:28.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.665+0000 7fa1acd1b700 1 Processor -- start 2023-10-10T18:49:28.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.665+0000 7fa1acd1b700 1 -- start start 2023-10-10T18:49:28.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.666+0000 7fa1acd1b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa1a80b2390 0x7fa1a8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:28.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.666+0000 7fa1acd1b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1a80aff60 0x7fa1a80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:28.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.666+0000 7fa1acd1b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a81574b0 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.666+0000 7fa1acd1b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a8150770 con 0x7fa1a80b2390 2023-10-10T18:49:28.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.666+0000 7fa1acd1b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a80afad0 con 0x7fa1a80aff60 2023-10-10T18:49:28.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a5d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa1a80b2390 0x7fa1a8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:28.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a5d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa1a80b2390 0x7fa1a8155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56420/0 (socket says 172.21.15.175:56420) 2023-10-10T18:49:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a5d9b700 1 -- 172.21.15.175:0/2713619318 learned_addr learned my addr 172.21.15.175:0/2713619318 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a6d9d700 1 --2- 172.21.15.175:0/2713619318 >> v2:172.21.15.191:3300/0 conn(0x7fa1a80aff60 0x7fa1a80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a5d9b700 1 -- 172.21.15.175:0/2713619318 >> v2:172.21.15.191:3300/0 conn(0x7fa1a80aff60 msgr2=0x7fa1a80b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a5d9b700 1 --2- 172.21.15.175:0/2713619318 >> v2:172.21.15.191:3300/0 conn(0x7fa1a80aff60 0x7fa1a80b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a5d9b700 1 -- 172.21.15.175:0/2713619318 >> v1:172.21.15.191:6789/0 conn(0x7fa1a80b1fc0 legacy=0x7fa1a8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a5d9b700 1 -- 172.21.15.175:0/2713619318 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a80af4c0 con 0x7fa1a80b2390 2023-10-10T18:49:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a5d9b700 1 --2- 172.21.15.175:0/2713619318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b2390 0x7fa1a8155fb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80641f0 tx=0x7fa19c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f176167539c7cbd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a559a700 1 -- 172.21.15.175:0/2713619318 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c009e70 con 0x7fa1a80b2390 2023-10-10T18:49:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a559a700 1 -- 172.21.15.175:0/2713619318 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c00b8f0 con 0x7fa1a80b2390 2023-10-10T18:49:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.667+0000 7fa1a559a700 1 -- 172.21.15.175:0/2713619318 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c00bc90 con 0x7fa1a80b2390 2023-10-10T18:49:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.668+0000 7fa1acd1b700 1 -- 172.21.15.175:0/2713619318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b2390 msgr2=0x7fa1a8155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.668+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/2713619318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b2390 0x7fa1a8155fb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80641f0 tx=0x7fa19c009250 comp rx=0 tx=0).stop 2023-10-10T18:49:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.668+0000 7fa1acd1b700 1 -- 172.21.15.175:0/2713619318 shutdown_connections 2023-10-10T18:49:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.668+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/2713619318 >> v2:172.21.15.191:3300/0 conn(0x7fa1a80aff60 0x7fa1a80b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.668+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/2713619318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b2390 0x7fa1a8155fb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.668+0000 7fa1acd1b700 1 -- 172.21.15.175:0/2713619318 >> 172.21.15.175:0/2713619318 conn(0x7fa1a8063230 msgr2=0x7fa1a8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.668+0000 7fa1acd1b700 1 -- 172.21.15.175:0/2713619318 shutdown_connections 2023-10-10T18:49:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.668+0000 7fa1acd1b700 1 -- 172.21.15.175:0/2713619318 wait complete. 2023-10-10T18:49:28.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.669+0000 7fa1acd1b700 1 Processor -- start 2023-10-10T18:49:28.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.669+0000 7fa1acd1b700 1 -- start start 2023-10-10T18:49:28.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1acd1b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1a80aff60 0x7fa1a80ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:28.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1acd1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b1fc0 0x7fa1a80cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:28.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1acd1b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1a80b2390 0x7fa1a80cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:28.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1acd1b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8166c10 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1acd1b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8156ba0 con 0x7fa1a80aff60 2023-10-10T18:49:28.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1acd1b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8166930 con 0x7fa1a80b2390 2023-10-10T18:49:28.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b1fc0 0x7fa1a80cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:28.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b1fc0 0x7fa1a80cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56434/0 (socket says 172.21.15.175:56434) 2023-10-10T18:49:28.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 -- 172.21.15.175:0/1151057913 learned_addr learned my addr 172.21.15.175:0/1151057913 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:28.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a659c700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1a80aff60 0x7fa1a80ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:28.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a6d9d700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1a80b2390 0x7fa1a80cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:28.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 -- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1a80b2390 msgr2=0x7fa1a80cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:28.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1a80b2390 0x7fa1a80cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 -- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1a80aff60 msgr2=0x7fa1a80ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:28.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1a80aff60 0x7fa1a80ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 -- 172.21.15.175:0/1151057913 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c0094f0 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a659c700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1a80aff60 0x7fa1a80ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:49:28.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a6d9d700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1a80b2390 0x7fa1a80cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:49:28.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1a5d9b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b1fc0 0x7fa1a80cc300 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa19c006fd0 tx=0x7fa19c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:28.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1967fc700 1 -- 172.21.15.175:0/1151057913 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c01a450 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1967fc700 1 -- 172.21.15.175:0/1151057913 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c01a5d0 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a81565e0 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1967fc700 1 -- 172.21.15.175:0/1151057913 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c01a990 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.670+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a80cbde0 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.671+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a8067570 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.672+0000 7fa1967fc700 1 -- 172.21.15.175:0/1151057913 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa19c01ad00 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.672+0000 7fa1967fc700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa188066f60 0x7fa188069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:28.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.672+0000 7fa1967fc700 1 -- 172.21.15.175:0/1151057913 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6239+0+0 (secure 0 0 0) 0x7fa19c086300 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.672+0000 7fa1a659c700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa188066f60 0x7fa188069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:28.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.673+0000 7fa1a659c700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa188066f60 0x7fa188069410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0007fd0 tx=0x7fa1a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:28.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.676+0000 7fa1967fc700 1 -- 172.21.15.175:0/1151057913 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa19c053360 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.901+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1a8068da0 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.901+0000 7fa1967fc700 1 -- 172.21.15.175:0/1151057913 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6194 (secure 0 0 0) 0x7fa19c056980 con 0x7fa1a80b1fc0 2023-10-10T18:49:28.903 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:28.903 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:01.486855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:28.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa188066f60 msgr2=0x7fa188069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:28.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa188066f60 0x7fa188069410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0007fd0 tx=0x7fa1a0009040 comp rx=0 tx=0).stop 2023-10-10T18:49:28.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b1fc0 msgr2=0x7fa1a80cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:28.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b1fc0 0x7fa1a80cc300 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa19c006fd0 tx=0x7fa19c017000 comp rx=0 tx=0).stop 2023-10-10T18:49:28.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 shutdown_connections 2023-10-10T18:49:28.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa188066f60 0x7fa188069410 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1a80b2390 0x7fa1a80cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1a80aff60 0x7fa1a80ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 --2- 172.21.15.175:0/1151057913 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1a80b1fc0 0x7fa1a80cc300 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:28.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 >> 172.21.15.175:0/1151057913 conn(0x7fa1a8063230 msgr2=0x7fa1a814fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:28.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 shutdown_connections 2023-10-10T18:49:28.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:28.903+0000 7fa1acd1b700 1 -- 172.21.15.175:0/1151057913 wait complete. 2023-10-10T18:49:28.907 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 33 2023-10-10T18:49:28.916 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T18:49:30.916 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:31.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.041+0000 7f6550459700 1 Processor -- start 2023-10-10T18:49:31.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.041+0000 7f6550459700 1 -- start start 2023-10-10T18:49:31.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.042+0000 7f6550459700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814fbf0 0x7f6548159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:31.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.042+0000 7f6550459700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f654814ffc0 0x7f6548063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:31.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.042+0000 7f6550459700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6548156800 con 0x7f65480681e0 2023-10-10T18:49:31.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.042+0000 7f6550459700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65480c59e0 con 0x7f654814ffc0 2023-10-10T18:49:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.042+0000 7f6550459700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654815a2b0 con 0x7f654814fbf0 2023-10-10T18:49:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.042+0000 7f654d9f4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f654814ffc0 0x7f6548063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.042+0000 7f654d9f4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f654814ffc0 0x7f6548063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56446/0 (socket says 172.21.15.175:56446) 2023-10-10T18:49:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.042+0000 7f654d9f4700 1 -- 172.21.15.175:0/2143510250 learned_addr learned my addr 172.21.15.175:0/2143510250 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654e1f5700 1 --2- 172.21.15.175:0/2143510250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814fbf0 0x7f6548159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654d9f4700 1 -- 172.21.15.175:0/2143510250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814fbf0 msgr2=0x7f6548159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654d9f4700 1 --2- 172.21.15.175:0/2143510250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814fbf0 0x7f6548159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654d9f4700 1 -- 172.21.15.175:0/2143510250 >> v1:172.21.15.175:6789/0 conn(0x7f65480681e0 legacy=0x7f65480c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654d9f4700 1 -- 172.21.15.175:0/2143510250 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6548063e30 con 0x7f654814ffc0 2023-10-10T18:49:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654d9f4700 1 --2- 172.21.15.175:0/2143510250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814ffc0 0x7f6548063230 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f65380099e0 tx=0x7f6538009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab5ccb2804d429e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654d1f3700 1 -- 172.21.15.175:0/2143510250 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653800b820 con 0x7f654814ffc0 2023-10-10T18:49:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654d1f3700 1 -- 172.21.15.175:0/2143510250 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653800b980 con 0x7f654814ffc0 2023-10-10T18:49:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.043+0000 7f654d1f3700 1 -- 172.21.15.175:0/2143510250 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653800bd20 con 0x7f654814ffc0 2023-10-10T18:49:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.044+0000 7f6550459700 1 -- 172.21.15.175:0/2143510250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814ffc0 msgr2=0x7f6548063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.044+0000 7f6550459700 1 --2- 172.21.15.175:0/2143510250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814ffc0 0x7f6548063230 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f65380099e0 tx=0x7f6538009250 comp rx=0 tx=0).stop 2023-10-10T18:49:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.044+0000 7f6550459700 1 -- 172.21.15.175:0/2143510250 shutdown_connections 2023-10-10T18:49:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.044+0000 7f6550459700 1 --2- 172.21.15.175:0/2143510250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814fbf0 0x7f6548159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.044+0000 7f6550459700 1 --2- 172.21.15.175:0/2143510250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814ffc0 0x7f6548063230 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.044+0000 7f6550459700 1 -- 172.21.15.175:0/2143510250 >> 172.21.15.175:0/2143510250 conn(0x7f65480cc760 msgr2=0x7f65480ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.044+0000 7f6550459700 1 -- 172.21.15.175:0/2143510250 shutdown_connections 2023-10-10T18:49:31.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.044+0000 7f6550459700 1 -- 172.21.15.175:0/2143510250 wait complete. 2023-10-10T18:49:31.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.045+0000 7f6550459700 1 Processor -- start 2023-10-10T18:49:31.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.045+0000 7f6550459700 1 -- start start 2023-10-10T18:49:31.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.045+0000 7f6550459700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65480681e0 0x7f6548158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:31.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.045+0000 7f6550459700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814fbf0 0x7f65481586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:31.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.045+0000 7f6550459700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814ffc0 0x7f65480a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:31.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.045+0000 7f6550459700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480aa700 con 0x7f654814fbf0 2023-10-10T18:49:31.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.045+0000 7f6550459700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480a8180 con 0x7f65480681e0 2023-10-10T18:49:31.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.045+0000 7f6550459700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6548165f50 con 0x7f654814ffc0 2023-10-10T18:49:31.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814fbf0 0x7f65481586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:31.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814fbf0 0x7f65481586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56462/0 (socket says 172.21.15.175:56462) 2023-10-10T18:49:31.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 -- 172.21.15.175:0/3892937440 learned_addr learned my addr 172.21.15.175:0/3892937440 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:31.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654e1f5700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65480681e0 0x7f6548158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:31.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654e9f6700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814ffc0 0x7f65480a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:31.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 -- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814ffc0 msgr2=0x7f65480a8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:31.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814ffc0 0x7f65480a8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 -- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65480681e0 msgr2=0x7f6548158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:31.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65480681e0 0x7f6548158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 -- 172.21.15.175:0/3892937440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65380094f0 con 0x7f654814fbf0 2023-10-10T18:49:31.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f654d9f4700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814fbf0 0x7f65481586b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6548089f50 tx=0x7f6538017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:31.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f653e7fc700 1 -- 172.21.15.175:0/3892937440 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538011d10 con 0x7f654814fbf0 2023-10-10T18:49:31.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f653e7fc700 1 -- 172.21.15.175:0/3892937440 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6538011e90 con 0x7f654814fbf0 2023-10-10T18:49:31.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6548166230 con 0x7f654814fbf0 2023-10-10T18:49:31.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f653e7fc700 1 -- 172.21.15.175:0/3892937440 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65380266a0 con 0x7f654814fbf0 2023-10-10T18:49:31.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.046+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65481667f0 con 0x7f654814fbf0 2023-10-10T18:49:31.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.047+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65480a82f0 con 0x7f654814fbf0 2023-10-10T18:49:31.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.048+0000 7f653e7fc700 1 -- 172.21.15.175:0/3892937440 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f653801a420 con 0x7f654814fbf0 2023-10-10T18:49:31.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.048+0000 7f653e7fc700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f652c066f30 0x7f652c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:31.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.048+0000 7f653e7fc700 1 -- 172.21.15.175:0/3892937440 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6239+0+0 (secure 0 0 0) 0x7f6538086250 con 0x7f654814fbf0 2023-10-10T18:49:31.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.048+0000 7f654e1f5700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f652c066f30 0x7f652c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:31.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.049+0000 7f654e1f5700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f652c066f30 0x7f652c0693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6544007680 tx=0x7f65440098d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:31.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.052+0000 7f653e7fc700 1 -- 172.21.15.175:0/3892937440 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6538053330 con 0x7f654814fbf0 2023-10-10T18:49:31.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.277+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65481669e0 con 0x7f654814fbf0 2023-10-10T18:49:31.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.278+0000 7f653e7fc700 1 -- 172.21.15.175:0/3892937440 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6194 (secure 0 0 0) 0x7f6538056950 con 0x7f654814fbf0 2023-10-10T18:49:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:31.279 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4760,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:01.486855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":4,"incarnation":23,"state":"up:active","state_seq":51,"addr":"172.21.15.175:6834/688515950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":688515950},{"type":"v1","addr":"172.21.15.175:6834","nonce":688515950}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:31.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f652c066f30 msgr2=0x7f652c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:31.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f652c066f30 0x7f652c0693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6544007680 tx=0x7f65440098d0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814fbf0 msgr2=0x7f65481586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:31.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814fbf0 0x7f65481586b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6548089f50 tx=0x7f6538017000 comp rx=0 tx=0).stop 2023-10-10T18:49:31.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 shutdown_connections 2023-10-10T18:49:31.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f652c066f30 0x7f652c0693e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f654814ffc0 0x7f65480a8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65480681e0 0x7f6548158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 --2- 172.21.15.175:0/3892937440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f654814fbf0 0x7f65481586b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:31.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 >> 172.21.15.175:0/3892937440 conn(0x7f65480cc760 msgr2=0x7f6548154660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:31.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 shutdown_connections 2023-10-10T18:49:31.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:31.280+0000 7f6550459700 1 -- 172.21.15.175:0/3892937440 wait complete. 2023-10-10T18:49:31.284 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 33 2023-10-10T18:49:31.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T18:49:33.294 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:33.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.420+0000 7fc8d18a8700 1 Processor -- start 2023-10-10T18:49:33.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.420+0000 7fc8d18a8700 1 -- start start 2023-10-10T18:49:33.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8d18a8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc8cc14cc10 0x7fc8cc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8d18a8700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc155ca0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8d18a8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc156b00 con 0x7fc8cc14c840 2023-10-10T18:49:33.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8d18a8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc06d700 con 0x7fc8cc14cc10 2023-10-10T18:49:33.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8cb7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc8cc0c52a0 0x7fc8cc152970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37526/0 (socket says 172.21.15.175:37526) 2023-10-10T18:49:33.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8cb7fe700 1 -- 172.21.15.175:0/1458791106 learned_addr learned my addr 172.21.15.175:0/1458791106 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:49:33.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8c3fff700 1 --2- 172.21.15.175:0/1458791106 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc14cc10 0x7fc8cc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1458791106 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377601806 0 0) 0x7fc8cc155ca0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.421+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1458791106 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b8003570 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8c3fff700 1 -- 172.21.15.175:0/1458791106 >> v1:172.21.15.191:6789/0 conn(0x7fc8cc14c840 legacy=0x7fc8cc0c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8c3fff700 1 -- 172.21.15.175:0/1458791106 >> v1:172.21.15.175:6789/0 conn(0x7fc8cc0c52a0 legacy=0x7fc8cc152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8c3fff700 1 -- 172.21.15.175:0/1458791106 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc155ef0 con 0x7fc8cc14cc10 2023-10-10T18:49:33.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8c3fff700 1 --2- 172.21.15.175:0/1458791106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14cc10 0x7fc8cc06e7a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400aa90 tx=0x7fc8b400a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94549295a7774368 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:33.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1458791106 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b400ca00 con 0x7fc8cc14cc10 2023-10-10T18:49:33.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1458791106 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b400cb60 con 0x7fc8cc14cc10 2023-10-10T18:49:33.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1458791106 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4018400 con 0x7fc8cc14cc10 2023-10-10T18:49:33.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8d18a8700 1 -- 172.21.15.175:0/1458791106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14cc10 msgr2=0x7fc8cc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8d18a8700 1 --2- 172.21.15.175:0/1458791106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14cc10 0x7fc8cc06e7a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400aa90 tx=0x7fc8b400a330 comp rx=0 tx=0).stop 2023-10-10T18:49:33.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8d18a8700 1 -- 172.21.15.175:0/1458791106 shutdown_connections 2023-10-10T18:49:33.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8d18a8700 1 --2- 172.21.15.175:0/1458791106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14cc10 0x7fc8cc06e7a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.422+0000 7fc8d18a8700 1 -- 172.21.15.175:0/1458791106 >> 172.21.15.175:0/1458791106 conn(0x7fc8cc068280 msgr2=0x7fc8cc06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:33.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.423+0000 7fc8d18a8700 1 -- 172.21.15.175:0/1458791106 shutdown_connections 2023-10-10T18:49:33.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.423+0000 7fc8d18a8700 1 -- 172.21.15.175:0/1458791106 wait complete. 2023-10-10T18:49:33.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8d18a8700 1 Processor -- start 2023-10-10T18:49:33.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8d18a8700 1 -- start start 2023-10-10T18:49:33.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8d18a8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0c52a0 0x7fc8cc16a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8d18a8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14c840 0x7fc8cc1446d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8d18a8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc14cc10 0x7fc8cc0c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8d18a8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc16a5a0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8d18a8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc144c10 con 0x7fc8cc14c840 2023-10-10T18:49:33.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8d18a8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc1669b0 con 0x7fc8cc14cc10 2023-10-10T18:49:33.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8caffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0c52a0 0x7fc8cc16a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8caffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0c52a0 0x7fc8cc16a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56476/0 (socket says 172.21.15.175:56476) 2023-10-10T18:49:33.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.424+0000 7fc8caffd700 1 -- 172.21.15.175:0/380943291 learned_addr learned my addr 172.21.15.175:0/380943291 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:33.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8cb7fe700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc14cc10 0x7fc8cc0c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8c3fff700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14c840 0x7fc8cc1446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8caffd700 1 -- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc14cc10 msgr2=0x7fc8cc0c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8caffd700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc14cc10 0x7fc8cc0c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8caffd700 1 -- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14c840 msgr2=0x7fc8cc1446d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8caffd700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14c840 0x7fc8cc1446d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8caffd700 1 -- 172.21.15.175:0/380943291 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b400a5d0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8c3fff700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14c840 0x7fc8cc1446d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:49:33.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8cb7fe700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc14cc10 0x7fc8cc0c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:49:33.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8caffd700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0c52a0 0x7fc8cc16a060 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00c7f0 tx=0x7fc8bc007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:33.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8c2ffd700 1 -- 172.21.15.175:0/380943291 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc012630 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8c2ffd700 1 -- 172.21.15.175:0/380943291 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc0127b0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc1660a0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8c2ffd700 1 -- 172.21.15.175:0/380943291 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc012b70 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.425+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc145110 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.426+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc0c03e0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.426+0000 7fc8c2ffd700 1 -- 172.21.15.175:0/380943291 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc8bc01b400 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.427+0000 7fc8c2ffd700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8ac066f30 0x7fc8ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.427+0000 7fc8c2ffd700 1 -- 172.21.15.175:0/380943291 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6239+0+0 (secure 0 0 0) 0x7fc8bc0842f0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.427+0000 7fc8c3fff700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8ac066f30 0x7fc8ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.427+0000 7fc8c3fff700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8ac066f30 0x7fc8ac0693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400a5a0 tx=0x7fc8b401a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:33.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.431+0000 7fc8c2ffd700 1 -- 172.21.15.175:0/380943291 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8bc0512b0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.666+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8cc067570 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.666+0000 7fc8c2ffd700 1 -- 172.21.15.175:0/380943291 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5436 (secure 0 0 0) 0x7fc8bc0548d0 con 0x7fc8cc0c52a0 2023-10-10T18:49:33.668 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:33.668 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:33.433308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:33.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.668+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8ac066f30 msgr2=0x7fc8ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.668+0000 7fc8d18a8700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8ac066f30 0x7fc8ac0693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400a5a0 tx=0x7fc8b401a000 comp rx=0 tx=0).stop 2023-10-10T18:49:33.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.668+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0c52a0 msgr2=0x7fc8cc16a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.668+0000 7fc8d18a8700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0c52a0 0x7fc8cc16a060 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00c7f0 tx=0x7fc8bc007f70 comp rx=0 tx=0).stop 2023-10-10T18:49:33.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.669+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 shutdown_connections 2023-10-10T18:49:33.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.669+0000 7fc8d18a8700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8ac066f30 0x7fc8ac0693e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.669+0000 7fc8d18a8700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc14cc10 0x7fc8cc0c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.669+0000 7fc8d18a8700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0c52a0 0x7fc8cc16a060 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.669+0000 7fc8d18a8700 1 --2- 172.21.15.175:0/380943291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc14c840 0x7fc8cc1446d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.669+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 >> 172.21.15.175:0/380943291 conn(0x7fc8cc068280 msgr2=0x7fc8cc0c0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:33.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.669+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 shutdown_connections 2023-10-10T18:49:33.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.669+0000 7fc8d18a8700 1 -- 172.21.15.175:0/380943291 wait complete. 2023-10-10T18:49:33.672 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 34 2023-10-10T18:49:33.681 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-10T18:49:33.682 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T18:49:33.682 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:33.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.795+0000 7f7db49fc700 1 Processor -- start 2023-10-10T18:49:33.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.795+0000 7f7db49fc700 1 -- start start 2023-10-10T18:49:33.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7db49fc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7db0159dd0 0x7f7db0067b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7db49fc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db0068b30 con 0x7f7db0068cf0 2023-10-10T18:49:33.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7db49fc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00c1c70 con 0x7f7db014fd20 2023-10-10T18:49:33.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7db49fc700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db00cd2b0 con 0x7f7db0159dd0 2023-10-10T18:49:33.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7daed9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7db0068cf0 0x7f7db00c2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37528/0 (socket says 172.21.15.175:37528) 2023-10-10T18:49:33.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7daed9d700 1 -- 172.21.15.175:0/774552434 learned_addr learned my addr 172.21.15.175:0/774552434 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:49:33.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7da659c700 1 --2- 172.21.15.175:0/774552434 >> v2:172.21.15.191:3300/0 conn(0x7f7db0159dd0 0x7f7db0067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904295655 0 0) 0x7f7db0068b30 con 0x7f7db0068cf0 2023-10-10T18:49:33.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9c003570 con 0x7f7db0068cf0 2023-10-10T18:49:33.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609734579 0 0) 0x7f7d9c003570 con 0x7f7db0068cf0 2023-10-10T18:49:33.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 >> v2:172.21.15.191:3300/0 conn(0x7f7db0159dd0 msgr2=0x7f7db0067b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:33.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 --2- 172.21.15.175:0/774552434 >> v2:172.21.15.191:3300/0 conn(0x7f7db0159dd0 0x7f7db0067b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 >> v1:172.21.15.191:6789/0 conn(0x7f7db014fd20 legacy=0x7f7db00690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db00cd590 con 0x7f7db0068cf0 2023-10-10T18:49:33.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f7da0001470 con 0x7f7db0068cf0 2023-10-10T18:49:33.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 >> v1:172.21.15.175:6789/0 conn(0x7f7db0068cf0 legacy=0x7f7db00c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.796+0000 7f7dae59c700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d9c003a90 0x7f7d9c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7dae59c700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d9c004810 0x7f7d9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7dae59c700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7d9c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00cd2b0 con 0x7f7d9c005210 2023-10-10T18:49:33.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c00c130 con 0x7f7d9c004810 2023-10-10T18:49:33.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c00c2c0 con 0x7f7d9c003a90 2023-10-10T18:49:33.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7da659c700 1 --2- 172.21.15.175:0/774552434 >> v2:172.21.15.191:3300/0 conn(0x7f7db0159dd0 0x7f7db0067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:49:33.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daf59e700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7d9c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daed9d700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d9c003a90 0x7f7d9c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7da659c700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d9c004810 0x7f7d9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daf59e700 1 -- 172.21.15.175:0/774552434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d9c003a90 msgr2=0x7f7d9c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:33.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daf59e700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d9c003a90 0x7f7d9c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daf59e700 1 -- 172.21.15.175:0/774552434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d9c004810 msgr2=0x7f7d9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:33.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daf59e700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d9c004810 0x7f7d9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daf59e700 1 -- 172.21.15.175:0/774552434 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c003570 con 0x7f7d9c005210 2023-10-10T18:49:33.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daf59e700 1 -- 172.21.15.175:0/774552434 reap_dead start 2023-10-10T18:49:33.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7daf59e700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7d9c00ba60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7da8001ff0 tx=0x7f7da8001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee6da8af093b1e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:33.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8001950 con 0x7f7d9c005210 2023-10-10T18:49:33.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da800c870 con 0x7f7d9c005210 2023-10-10T18:49:33.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.797+0000 7f7dae59c700 1 -- 172.21.15.175:0/774552434 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da800cc10 con 0x7f7d9c005210 2023-10-10T18:49:33.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.798+0000 7f7db49fc700 1 -- 172.21.15.175:0/774552434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 msgr2=0x7f7d9c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.798+0000 7f7db49fc700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7d9c00ba60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7da8001ff0 tx=0x7f7da8001640 comp rx=0 tx=0).stop 2023-10-10T18:49:33.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.798+0000 7f7db49fc700 1 -- 172.21.15.175:0/774552434 shutdown_connections 2023-10-10T18:49:33.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.798+0000 7f7db49fc700 1 --2- 172.21.15.175:0/774552434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7d9c00ba60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.798+0000 7f7db49fc700 1 -- 172.21.15.175:0/774552434 >> 172.21.15.175:0/774552434 conn(0x7f7db0063cc0 msgr2=0x7f7db00cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:33.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.798+0000 7f7db49fc700 1 -- 172.21.15.175:0/774552434 shutdown_connections 2023-10-10T18:49:33.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.798+0000 7f7db49fc700 1 -- 172.21.15.175:0/774552434 wait complete. 2023-10-10T18:49:33.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.799+0000 7f7db49fc700 1 Processor -- start 2023-10-10T18:49:33.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.799+0000 7f7db49fc700 1 -- start start 2023-10-10T18:49:33.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7db49fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7db00cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7db49fc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7db00c47d0 0x7f7db014bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7db49fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7db0157530 0x7f7db0168d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7db49fc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db0169c70 con 0x7f7d9c005210 2023-10-10T18:49:33.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7db49fc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db014b230 con 0x7f7db0157530 2023-10-10T18:49:33.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7db49fc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db0157f40 con 0x7f7db00c47d0 2023-10-10T18:49:33.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7db00cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7db00cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56496/0 (socket says 172.21.15.175:56496) 2023-10-10T18:49:33.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 -- 172.21.15.175:0/73680622 learned_addr learned my addr 172.21.15.175:0/73680622 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:33.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7da659c700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7db00c47d0 0x7f7db014bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daf59e700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7db0157530 0x7f7db0168d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 -- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7db00c47d0 msgr2=0x7f7db014bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7db00c47d0 0x7f7db014bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 -- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7db0157530 msgr2=0x7f7db0168d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:33.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7db0157530 0x7f7db0168d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:33.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 -- 172.21.15.175:0/73680622 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db0147060 con 0x7f7d9c005210 2023-10-10T18:49:33.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7da659c700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7db00c47d0 0x7f7db014bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:49:33.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7daed9d700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7db00cc550 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7da00073c0 tx=0x7f7db00c3480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:33.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7da77fe700 1 -- 172.21.15.175:0/73680622 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000f900 con 0x7f7d9c005210 2023-10-10T18:49:33.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7da77fe700 1 -- 172.21.15.175:0/73680622 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da000fa60 con 0x7f7d9c005210 2023-10-10T18:49:33.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db014b760 con 0x7f7d9c005210 2023-10-10T18:49:33.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7da77fe700 1 -- 172.21.15.175:0/73680622 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000fe20 con 0x7f7d9c005210 2023-10-10T18:49:33.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.800+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db014b430 con 0x7f7d9c005210 2023-10-10T18:49:33.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.801+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db0067570 con 0x7f7d9c005210 2023-10-10T18:49:33.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.802+0000 7f7da77fe700 1 -- 172.21.15.175:0/73680622 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7da00135c0 con 0x7f7d9c005210 2023-10-10T18:49:33.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.802+0000 7f7da77fe700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7d90066f60 0x7f7d90069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:33.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.802+0000 7f7da77fe700 1 -- 172.21.15.175:0/73680622 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5993+0+0 (secure 0 0 0) 0x7f7da0084480 con 0x7f7d9c005210 2023-10-10T18:49:33.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.802+0000 7f7da659c700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7d90066f60 0x7f7d90069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:33.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.803+0000 7f7da659c700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7d90066f60 0x7f7d90069410 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7d980093f0 tx=0x7f7d98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:33.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:33.806+0000 7f7da77fe700 1 -- 172.21.15.175:0/73680622 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7da00515d0 con 0x7f7d9c005210 2023-10-10T18:49:34.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.029+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7db01699b0 con 0x7f7d9c005210 2023-10-10T18:49:34.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.030+0000 7f7da77fe700 1 -- 172.21.15.175:0/73680622 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5436 (secure 0 0 0) 0x7f7da001a090 con 0x7f7d9c005210 2023-10-10T18:49:34.031 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:34.031 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:33.433308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:34.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7d90066f60 msgr2=0x7f7d90069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:34.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7d90066f60 0x7f7d90069410 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7d980093f0 tx=0x7f7d98008040 comp rx=0 tx=0).stop 2023-10-10T18:49:34.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 msgr2=0x7f7db00cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:34.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7db00cc550 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7da00073c0 tx=0x7f7db00c3480 comp rx=0 tx=0).stop 2023-10-10T18:49:34.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 shutdown_connections 2023-10-10T18:49:34.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7d90066f60 0x7f7d90069410 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:34.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7db0157530 0x7f7db0168d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:34.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d9c005210 0x7f7db00cc550 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:34.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 --2- 172.21.15.175:0/73680622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7db00c47d0 0x7f7db014bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:34.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 >> 172.21.15.175:0/73680622 conn(0x7f7db0063cc0 msgr2=0x7f7db014f260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:34.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 shutdown_connections 2023-10-10T18:49:34.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:34.032+0000 7f7db49fc700 1 -- 172.21.15.175:0/73680622 wait complete. 2023-10-10T18:49:34.037 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 34 2023-10-10T18:49:34.045 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4248": { "addr": "172.21.15.175:6836/647425682", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 647425682, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 647425682, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4248, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4250": { "addr": "172.21.15.191:6839/246221790", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 246221790, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 246221790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4250, "incarnation": 12, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:49:33.433308+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 4250, "mds_3": 4248, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:49:36.046 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:36.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.171+0000 7f60c27b3700 1 Processor -- start 2023-10-10T18:49:36.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60c27b3700 1 -- start start 2023-10-10T18:49:36.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60c27b3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f60bc0b1fc0 0x7f60bc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:36.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60c27b3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f60bc0aff60 0x7f60bc0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:36.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60c27b3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60bc1574b0 con 0x7f60bc0b2390 2023-10-10T18:49:36.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60c27b3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc150770 con 0x7f60bc0b1fc0 2023-10-10T18:49:36.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60c27b3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc0afad0 con 0x7f60bc0aff60 2023-10-10T18:49:36.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60bbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f60bc0b1fc0 0x7f60bc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:36.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60bbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f60bc0b1fc0 0x7f60bc155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43852/0 (socket says 172.21.15.175:43852) 2023-10-10T18:49:36.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60bbfff700 1 -- 172.21.15.175:0/604357045 learned_addr learned my addr 172.21.15.175:0/604357045 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:36.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.172+0000 7f60bb7fe700 1 --1- 172.21.15.175:0/604357045 >> v1:172.21.15.175:6789/0 conn(0x7f60bc0b2390 0x7f60bc068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37288/0 (socket says 172.21.15.175:37288) 2023-10-10T18:49:36.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60c0d50700 1 --2- 172.21.15.175:0/604357045 >> v2:172.21.15.191:3300/0 conn(0x7f60bc0aff60 0x7f60bc0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:36.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60bbfff700 1 -- 172.21.15.175:0/604357045 >> v2:172.21.15.191:3300/0 conn(0x7f60bc0aff60 msgr2=0x7f60bc0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:36.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60bbfff700 1 --2- 172.21.15.175:0/604357045 >> v2:172.21.15.191:3300/0 conn(0x7f60bc0aff60 0x7f60bc0b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60bbfff700 1 -- 172.21.15.175:0/604357045 >> v1:172.21.15.175:6789/0 conn(0x7f60bc0b2390 legacy=0x7f60bc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:36.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60bbfff700 1 -- 172.21.15.175:0/604357045 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc0af4c0 con 0x7f60bc0b1fc0 2023-10-10T18:49:36.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60c0d50700 1 --2- 172.21.15.175:0/604357045 >> v2:172.21.15.191:3300/0 conn(0x7f60bc0aff60 0x7f60bc0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:49:36.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60bbfff700 1 --2- 172.21.15.175:0/604357045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0b1fc0 0x7f60bc155fb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0641f0 tx=0x7f60ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79aa6c5160dbd0e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:36.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60baffd700 1 -- 172.21.15.175:0/604357045 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00b8f0 con 0x7f60bc0b1fc0 2023-10-10T18:49:36.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60baffd700 1 -- 172.21.15.175:0/604357045 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac00ba50 con 0x7f60bc0b1fc0 2023-10-10T18:49:36.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.173+0000 7f60baffd700 1 -- 172.21.15.175:0/604357045 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00bdf0 con 0x7f60bc0b1fc0 2023-10-10T18:49:36.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.174+0000 7f60c27b3700 1 -- 172.21.15.175:0/604357045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0b1fc0 msgr2=0x7f60bc155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:36.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.174+0000 7f60c27b3700 1 --2- 172.21.15.175:0/604357045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0b1fc0 0x7f60bc155fb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0641f0 tx=0x7f60ac009250 comp rx=0 tx=0).stop 2023-10-10T18:49:36.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.174+0000 7f60c27b3700 1 -- 172.21.15.175:0/604357045 shutdown_connections 2023-10-10T18:49:36.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.174+0000 7f60c27b3700 1 --2- 172.21.15.175:0/604357045 >> v2:172.21.15.191:3300/0 conn(0x7f60bc0aff60 0x7f60bc0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.174+0000 7f60c27b3700 1 --2- 172.21.15.175:0/604357045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0b1fc0 0x7f60bc155fb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.174+0000 7f60c27b3700 1 -- 172.21.15.175:0/604357045 >> 172.21.15.175:0/604357045 conn(0x7f60bc063230 msgr2=0x7f60bc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:36.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.174+0000 7f60c27b3700 1 -- 172.21.15.175:0/604357045 shutdown_connections 2023-10-10T18:49:36.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.174+0000 7f60c27b3700 1 -- 172.21.15.175:0/604357045 wait complete. 2023-10-10T18:49:36.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.175+0000 7f60c27b3700 1 Processor -- start 2023-10-10T18:49:36.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.175+0000 7f60c27b3700 1 -- start start 2023-10-10T18:49:36.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.175+0000 7f60c27b3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0aff60 0x7f60bc14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:36.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.175+0000 7f60c27b3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc0b1fc0 0x7f60bc169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:36.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.175+0000 7f60c27b3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc0b2390 0x7f60bc16a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:36.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.175+0000 7f60c27b3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0c62c0 con 0x7f60bc0aff60 2023-10-10T18:49:36.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60c27b3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc169850 con 0x7f60bc0b2390 2023-10-10T18:49:36.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60c27b3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0cca40 con 0x7f60bc0b1fc0 2023-10-10T18:49:36.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0aff60 0x7f60bc14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:36.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0aff60 0x7f60bc14dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43864/0 (socket says 172.21.15.175:43864) 2023-10-10T18:49:36.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 -- 172.21.15.175:0/2425773242 learned_addr learned my addr 172.21.15.175:0/2425773242 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:36.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 -- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc0b1fc0 msgr2=0x7f60bc169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:36.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc0b1fc0 0x7f60bc169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 -- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc0b2390 msgr2=0x7f60bc16a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:36.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60c0d50700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc0b2390 0x7f60bc16a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:36.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc0b2390 0x7f60bc16a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 -- 172.21.15.175:0/2425773242 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac0094f0 con 0x7f60bc0aff60 2023-10-10T18:49:36.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60bbfff700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0aff60 0x7f60bc14dce0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009290 tx=0x7f60ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:36.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60a3fff700 1 -- 172.21.15.175:0/2425773242 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac006bc0 con 0x7f60bc0aff60 2023-10-10T18:49:36.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60a3fff700 1 -- 172.21.15.175:0/2425773242 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac011e60 con 0x7f60bc0aff60 2023-10-10T18:49:36.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60a3fff700 1 -- 172.21.15.175:0/2425773242 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac01a8b0 con 0x7f60bc0aff60 2023-10-10T18:49:36.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc0ccce0 con 0x7f60bc0aff60 2023-10-10T18:49:36.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.176+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc0cd2a0 con 0x7f60bc0aff60 2023-10-10T18:49:36.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.177+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc0713a0 con 0x7f60bc0aff60 2023-10-10T18:49:36.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.177+0000 7f60a3fff700 1 -- 172.21.15.175:0/2425773242 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f60ac01ac20 con 0x7f60bc0aff60 2023-10-10T18:49:36.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.178+0000 7f60a3fff700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f60a8066f30 0x7f60a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:36.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.178+0000 7f60a3fff700 1 -- 172.21.15.175:0/2425773242 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5993+0+0 (secure 0 0 0) 0x7f60ac085eb0 con 0x7f60bc0aff60 2023-10-10T18:49:36.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.178+0000 7f60bb7fe700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f60a8066f30 0x7f60a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:36.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.179+0000 7f60bb7fe700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f60a8066f30 0x7f60a80693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f60bc14fc70 tx=0x7f60b0007b30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:36.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.182+0000 7f60a3fff700 1 -- 172.21.15.175:0/2425773242 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f60bc0713a0 con 0x7f60bc0aff60 2023-10-10T18:49:36.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.406+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60bc0713a0 con 0x7f60bc0aff60 2023-10-10T18:49:36.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.407+0000 7f60a3fff700 1 -- 172.21.15.175:0/2425773242 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5436 (secure 0 0 0) 0x7f60bc0713a0 con 0x7f60bc0aff60 2023-10-10T18:49:36.408 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:36.408 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:33.433308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:36.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.408+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f60a8066f30 msgr2=0x7f60a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:36.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.408+0000 7f60c27b3700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f60a8066f30 0x7f60a80693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f60bc14fc70 tx=0x7f60b0007b30 comp rx=0 tx=0).stop 2023-10-10T18:49:36.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.408+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0aff60 msgr2=0x7f60bc14dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:36.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.408+0000 7f60c27b3700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0aff60 0x7f60bc14dce0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009290 tx=0x7f60ac017000 comp rx=0 tx=0).stop 2023-10-10T18:49:36.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.408+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 shutdown_connections 2023-10-10T18:49:36.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.409+0000 7f60c27b3700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f60a8066f30 0x7f60a80693e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.409+0000 7f60c27b3700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc0b2390 0x7f60bc16a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.409+0000 7f60c27b3700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc0aff60 0x7f60bc14dce0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.409+0000 7f60c27b3700 1 --2- 172.21.15.175:0/2425773242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc0b1fc0 0x7f60bc169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:36.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.409+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 >> 172.21.15.175:0/2425773242 conn(0x7f60bc063230 msgr2=0x7f60bc14fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:36.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.409+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 shutdown_connections 2023-10-10T18:49:36.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:36.409+0000 7f60c27b3700 1 -- 172.21.15.175:0/2425773242 wait complete. 2023-10-10T18:49:36.412 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 34 2023-10-10T18:49:36.421 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:49:38.422 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:38.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.543+0000 7fdf17226700 1 Processor -- start 2023-10-10T18:49:38.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.543+0000 7fdf17226700 1 -- start start 2023-10-10T18:49:38.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf17226700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdf1014fbf0 0x7fdf10159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:38.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf17226700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf100681e0 0x7fdf100c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:38.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf17226700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf10156800 con 0x7fdf1014ffc0 2023-10-10T18:49:38.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf17226700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf100c59e0 con 0x7fdf1014fbf0 2023-10-10T18:49:38.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf17226700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1015a2b0 con 0x7fdf100681e0 2023-10-10T18:49:38.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf14fc2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdf1014fbf0 0x7fdf10159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:38.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf14fc2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdf1014fbf0 0x7fdf10159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43866/0 (socket says 172.21.15.175:43866) 2023-10-10T18:49:38.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf14fc2700 1 -- 172.21.15.175:0/4235837783 learned_addr learned my addr 172.21.15.175:0/4235837783 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:38.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf157c3700 1 --2- 172.21.15.175:0/4235837783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf100681e0 0x7fdf100c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:38.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf14fc2700 1 -- 172.21.15.175:0/4235837783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf100681e0 msgr2=0x7fdf100c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:38.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf14fc2700 1 --2- 172.21.15.175:0/4235837783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf100681e0 0x7fdf100c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf14fc2700 1 -- 172.21.15.175:0/4235837783 >> v1:172.21.15.175:6789/0 conn(0x7fdf1014ffc0 legacy=0x7fdf10063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:38.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf14fc2700 1 -- 172.21.15.175:0/4235837783 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf10063e30 con 0x7fdf1014fbf0 2023-10-10T18:49:38.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf14fc2700 1 --2- 172.21.15.175:0/4235837783 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf1014fbf0 0x7fdf10159d70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009990 tx=0x7fdf04009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1203d6c52c04806 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:38.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.544+0000 7fdf0f7fe700 1 -- 172.21.15.175:0/4235837783 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04005c10 con 0x7fdf1014fbf0 2023-10-10T18:49:38.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf0f7fe700 1 -- 172.21.15.175:0/4235837783 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf04005d70 con 0x7fdf1014fbf0 2023-10-10T18:49:38.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf0f7fe700 1 -- 172.21.15.175:0/4235837783 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0400b880 con 0x7fdf1014fbf0 2023-10-10T18:49:38.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf17226700 1 -- 172.21.15.175:0/4235837783 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf1014fbf0 msgr2=0x7fdf10159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:38.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf17226700 1 --2- 172.21.15.175:0/4235837783 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf1014fbf0 0x7fdf10159d70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009990 tx=0x7fdf04009220 comp rx=0 tx=0).stop 2023-10-10T18:49:38.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf17226700 1 -- 172.21.15.175:0/4235837783 shutdown_connections 2023-10-10T18:49:38.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf17226700 1 --2- 172.21.15.175:0/4235837783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf100681e0 0x7fdf100c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf17226700 1 --2- 172.21.15.175:0/4235837783 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf1014fbf0 0x7fdf10159d70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf17226700 1 -- 172.21.15.175:0/4235837783 >> 172.21.15.175:0/4235837783 conn(0x7fdf100cc760 msgr2=0x7fdf100ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:38.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf17226700 1 -- 172.21.15.175:0/4235837783 shutdown_connections 2023-10-10T18:49:38.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.545+0000 7fdf17226700 1 -- 172.21.15.175:0/4235837783 wait complete. 2023-10-10T18:49:38.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.546+0000 7fdf17226700 1 Processor -- start 2023-10-10T18:49:38.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.546+0000 7fdf17226700 1 -- start start 2023-10-10T18:49:38.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf17226700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf100681e0 0x7fdf10166620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:38.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf17226700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1014fbf0 0x7fdf1014dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:38.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf17226700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf1014ffc0 0x7fdf100c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:38.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf17226700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf10166190 con 0x7fdf100681e0 2023-10-10T18:49:38.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf17226700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf10150530 con 0x7fdf1014fbf0 2023-10-10T18:49:38.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf17226700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf101507f0 con 0x7fdf1014ffc0 2023-10-10T18:49:38.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf14fc2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf100681e0 0x7fdf10166620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:38.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf14fc2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf100681e0 0x7fdf10166620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43874/0 (socket says 172.21.15.175:43874) 2023-10-10T18:49:38.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf14fc2700 1 -- 172.21.15.175:0/1599376384 learned_addr learned my addr 172.21.15.175:0/1599376384 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:38.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf157c3700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf1014ffc0 0x7fdf100c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:38.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf0ffff700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1014fbf0 0x7fdf1014dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:38.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf14fc2700 1 -- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf1014ffc0 msgr2=0x7fdf100c8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:38.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf14fc2700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf1014ffc0 0x7fdf100c8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf14fc2700 1 -- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1014fbf0 msgr2=0x7fdf1014dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:38.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf14fc2700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1014fbf0 0x7fdf1014dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.547+0000 7fdf14fc2700 1 -- 172.21.15.175:0/1599376384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04009480 con 0x7fdf100681e0 2023-10-10T18:49:38.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.548+0000 7fdf14fc2700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf100681e0 0x7fdf10166620 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400b3c0 tx=0x7fdf04016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:38.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.548+0000 7fdf0cff9700 1 -- 172.21.15.175:0/1599376384 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf040118c0 con 0x7fdf100681e0 2023-10-10T18:49:38.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.548+0000 7fdf0cff9700 1 -- 172.21.15.175:0/1599376384 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf04020070 con 0x7fdf100681e0 2023-10-10T18:49:38.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.548+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf10150ad0 con 0x7fdf100681e0 2023-10-10T18:49:38.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.548+0000 7fdf0cff9700 1 -- 172.21.15.175:0/1599376384 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04011c50 con 0x7fdf100681e0 2023-10-10T18:49:38.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.548+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf100c73a0 con 0x7fdf100681e0 2023-10-10T18:49:38.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.549+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf10165fe0 con 0x7fdf100681e0 2023-10-10T18:49:38.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.549+0000 7fdf0cff9700 1 -- 172.21.15.175:0/1599376384 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdf04022030 con 0x7fdf100681e0 2023-10-10T18:49:38.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.549+0000 7fdf0cff9700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdef4066f30 0x7fdef40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:38.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.550+0000 7fdf0cff9700 1 -- 172.21.15.175:0/1599376384 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fdf040863c0 con 0x7fdf100681e0 2023-10-10T18:49:38.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.550+0000 7fdf0ffff700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdef4066f30 0x7fdef40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:38.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.550+0000 7fdf0ffff700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdef4066f30 0x7fdef40693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdf100cbc90 tx=0x7fdefc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:38.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.554+0000 7fdf0cff9700 1 -- 172.21.15.175:0/1599376384 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdf04053460 con 0x7fdf100681e0 2023-10-10T18:49:38.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.784+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf10067570 con 0x7fdf100681e0 2023-10-10T18:49:38.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.785+0000 7fdf0cff9700 1 -- 172.21.15.175:0/1599376384 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5436 (secure 0 0 0) 0x7fdf04011db0 con 0x7fdf100681e0 2023-10-10T18:49:38.786 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:38.786 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:33.433308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:38.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdef4066f30 msgr2=0x7fdef40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:38.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdef4066f30 0x7fdef40693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdf100cbc90 tx=0x7fdefc009040 comp rx=0 tx=0).stop 2023-10-10T18:49:38.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf100681e0 msgr2=0x7fdf10166620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:38.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf100681e0 0x7fdf10166620 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400b3c0 tx=0x7fdf04016000 comp rx=0 tx=0).stop 2023-10-10T18:49:38.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 shutdown_connections 2023-10-10T18:49:38.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdef4066f30 0x7fdef40693e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf1014ffc0 0x7fdf100c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdf100681e0 0x7fdf10166620 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 --2- 172.21.15.175:0/1599376384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1014fbf0 0x7fdf1014dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:38.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 >> 172.21.15.175:0/1599376384 conn(0x7fdf100cc760 msgr2=0x7fdf10158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:38.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 shutdown_connections 2023-10-10T18:49:38.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:38.787+0000 7fdf17226700 1 -- 172.21.15.175:0/1599376384 wait complete. 2023-10-10T18:49:38.792 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 34 2023-10-10T18:49:38.800 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:49:40.801 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:40.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.928+0000 7f57a84b9700 1 Processor -- start 2023-10-10T18:49:40.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.928+0000 7f57a84b9700 1 -- start start 2023-10-10T18:49:40.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a84b9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f57a014ffc0 0x7f57a0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:40.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a84b9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:40.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a84b9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a0156800 con 0x7f57a014fbf0 2023-10-10T18:49:40.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a84b9700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a00c59e0 con 0x7f57a014ffc0 2023-10-10T18:49:40.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a84b9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a015a2b0 con 0x7f57a00681e0 2023-10-10T18:49:40.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a5a54700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f57a014ffc0 0x7f57a0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:40.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a5a54700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f57a014ffc0 0x7f57a0159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43886/0 (socket says 172.21.15.175:43886) 2023-10-10T18:49:40.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a5a54700 1 -- 172.21.15.175:0/751099719 learned_addr learned my addr 172.21.15.175:0/751099719 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:40.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a6a56700 1 --2- 172.21.15.175:0/751099719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:40.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a5a54700 1 -- 172.21.15.175:0/751099719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 msgr2=0x7f57a00c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:40.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a5a54700 1 --2- 172.21.15.175:0/751099719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:40.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a5a54700 1 -- 172.21.15.175:0/751099719 >> v1:172.21.15.191:6789/0 conn(0x7f57a014fbf0 legacy=0x7f57a0063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:40.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a5a54700 1 -- 172.21.15.175:0/751099719 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a0063e30 con 0x7f57a014ffc0 2023-10-10T18:49:40.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a6a56700 1 --2- 172.21.15.175:0/751099719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:49:40.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.929+0000 7f57a5a54700 1 --2- 172.21.15.175:0/751099719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a0159d70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f57900099b0 tx=0x7f5790009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b253dc4478295a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:40.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a5253700 1 -- 172.21.15.175:0/751099719 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579000b820 con 0x7f57a014ffc0 2023-10-10T18:49:40.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a5253700 1 -- 172.21.15.175:0/751099719 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579000b980 con 0x7f57a014ffc0 2023-10-10T18:49:40.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a5253700 1 -- 172.21.15.175:0/751099719 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579000bd20 con 0x7f57a014ffc0 2023-10-10T18:49:40.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a84b9700 1 -- 172.21.15.175:0/751099719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 msgr2=0x7f57a0159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:40.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a84b9700 1 --2- 172.21.15.175:0/751099719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a0159d70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f57900099b0 tx=0x7f5790009250 comp rx=0 tx=0).stop 2023-10-10T18:49:40.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a84b9700 1 -- 172.21.15.175:0/751099719 shutdown_connections 2023-10-10T18:49:40.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a84b9700 1 --2- 172.21.15.175:0/751099719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:40.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a84b9700 1 --2- 172.21.15.175:0/751099719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a0159d70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:40.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a84b9700 1 -- 172.21.15.175:0/751099719 >> 172.21.15.175:0/751099719 conn(0x7f57a00cc760 msgr2=0x7f57a00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:40.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a84b9700 1 -- 172.21.15.175:0/751099719 shutdown_connections 2023-10-10T18:49:40.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.930+0000 7f57a84b9700 1 -- 172.21.15.175:0/751099719 wait complete. 2023-10-10T18:49:40.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.931+0000 7f57a84b9700 1 Processor -- start 2023-10-10T18:49:40.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.931+0000 7f57a84b9700 1 -- start start 2023-10-10T18:49:40.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a84b9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:40.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a84b9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57a014fbf0 0x7f57a01698a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:40.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a84b9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a014bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:40.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a84b9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a016a530 con 0x7f57a014ffc0 2023-10-10T18:49:40.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a84b9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0152a50 con 0x7f57a014fbf0 2023-10-10T18:49:40.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a84b9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a016a250 con 0x7f57a00681e0 2023-10-10T18:49:40.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a6a56700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a014bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:40.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a6a56700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a014bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43890/0 (socket says 172.21.15.175:43890) 2023-10-10T18:49:40.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a6a56700 1 -- 172.21.15.175:0/3526086518 learned_addr learned my addr 172.21.15.175:0/3526086518 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:40.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a6255700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:40.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.932+0000 7f57a5a54700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57a014fbf0 0x7f57a01698a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:40.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57a6a56700 1 -- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 msgr2=0x7f57a00c11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:40.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57a6a56700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:40.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57a6a56700 1 -- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57a014fbf0 msgr2=0x7f57a01698a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:40.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57a6a56700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57a014fbf0 0x7f57a01698a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:40.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57a6a56700 1 -- 172.21.15.175:0/3526086518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57900094f0 con 0x7f57a014ffc0 2023-10-10T18:49:40.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57a6a56700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a014bb40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f579800a380 tx=0x7f5798007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:40.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57967fc700 1 -- 172.21.15.175:0/3526086518 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798015440 con 0x7f57a014ffc0 2023-10-10T18:49:40.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57967fc700 1 -- 172.21.15.175:0/3526086518 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57980155a0 con 0x7f57a014ffc0 2023-10-10T18:49:40.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a0152120 con 0x7f57a014ffc0 2023-10-10T18:49:40.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57967fc700 1 -- 172.21.15.175:0/3526086518 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798015960 con 0x7f57a014ffc0 2023-10-10T18:49:40.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.933+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a01526e0 con 0x7f57a014ffc0 2023-10-10T18:49:40.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.934+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a0067570 con 0x7f57a014ffc0 2023-10-10T18:49:40.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.934+0000 7f57967fc700 1 -- 172.21.15.175:0/3526086518 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f579800d440 con 0x7f57a014ffc0 2023-10-10T18:49:40.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.934+0000 7f57967fc700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5784066f30 0x7f57840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:40.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.935+0000 7f57967fc700 1 -- 172.21.15.175:0/3526086518 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f5798085290 con 0x7f57a014ffc0 2023-10-10T18:49:40.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.935+0000 7f57a6255700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5784066f30 0x7f57840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:40.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.935+0000 7f57a6255700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5784066f30 0x7f57840693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f579c003a70 tx=0x7f579c007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:40.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:40.939+0000 7f57967fc700 1 -- 172.21.15.175:0/3526086518 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5798052330 con 0x7f57a014ffc0 2023-10-10T18:49:41.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.161+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57a0063830 con 0x7f57a014ffc0 2023-10-10T18:49:41.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.162+0000 7f57967fc700 1 -- 172.21.15.175:0/3526086518 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5438 (secure 0 0 0) 0x7f5798055950 con 0x7f57a014ffc0 2023-10-10T18:49:41.163 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:41.163 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:40.482454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:resolve","state_seq":21,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:41.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5784066f30 msgr2=0x7f57840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:41.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5784066f30 0x7f57840693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f579c003a70 tx=0x7f579c007ef0 comp rx=0 tx=0).stop 2023-10-10T18:49:41.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 msgr2=0x7f57a014bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:41.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a014bb40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f579800a380 tx=0x7f5798007900 comp rx=0 tx=0).stop 2023-10-10T18:49:41.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 shutdown_connections 2023-10-10T18:49:41.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5784066f30 0x7f57840693e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:41.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57a014ffc0 0x7f57a014bb40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:41.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57a00681e0 0x7f57a00c11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:41.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 --2- 172.21.15.175:0/3526086518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57a014fbf0 0x7f57a01698a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:41.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 >> 172.21.15.175:0/3526086518 conn(0x7f57a00cc760 msgr2=0x7f57a0159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:41.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 shutdown_connections 2023-10-10T18:49:41.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:41.164+0000 7f57a84b9700 1 -- 172.21.15.175:0/3526086518 wait complete. 2023-10-10T18:49:41.168 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 35 2023-10-10T18:49:41.176 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:49:43.178 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:43.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.302+0000 7f349e0f1700 1 Processor -- start 2023-10-10T18:49:43.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.302+0000 7f349e0f1700 1 -- start start 2023-10-10T18:49:43.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.303+0000 7f349e0f1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34980c2cf0 0x7f349806e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:43.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.303+0000 7f349e0f1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3498153f70 con 0x7f34980c30c0 2023-10-10T18:49:43.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.303+0000 7f349e0f1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3498067bd0 con 0x7f349814e290 2023-10-10T18:49:43.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.303+0000 7f349e0f1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3498154210 con 0x7f34980c2cf0 2023-10-10T18:49:43.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.303+0000 7f34977fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34980c2cf0 0x7f349806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:43.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.303+0000 7f34977fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34980c2cf0 0x7f349806e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43894/0 (socket says 172.21.15.175:43894) 2023-10-10T18:49:43.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.303+0000 7f34977fe700 1 -- 172.21.15.175:0/1038092433 learned_addr learned my addr 172.21.15.175:0/1038092433 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:43.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f34977fe700 1 -- 172.21.15.175:0/1038092433 >> v1:172.21.15.191:6789/0 conn(0x7f349814e290 legacy=0x7f349815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:43.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f34977fe700 1 -- 172.21.15.175:0/1038092433 >> v1:172.21.15.175:6789/0 conn(0x7f34980c30c0 legacy=0x7f349814db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:43.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f34977fe700 1 -- 172.21.15.175:0/1038092433 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34981544f0 con 0x7f34980c2cf0 2023-10-10T18:49:43.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f34977fe700 1 --2- 172.21.15.175:0/1038092433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34980c2cf0 0x7f349806e500 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3488009f70 tx=0x7f3488009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70da2e13c4a749c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:43.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f34967fc700 1 -- 172.21.15.175:0/1038092433 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348800e070 con 0x7f34980c2cf0 2023-10-10T18:49:43.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f34967fc700 1 -- 172.21.15.175:0/1038092433 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348800bb20 con 0x7f34980c2cf0 2023-10-10T18:49:43.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f34967fc700 1 -- 172.21.15.175:0/1038092433 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880092e0 con 0x7f34980c2cf0 2023-10-10T18:49:43.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f349e0f1700 1 -- 172.21.15.175:0/1038092433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34980c2cf0 msgr2=0x7f349806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:43.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.304+0000 7f349e0f1700 1 --2- 172.21.15.175:0/1038092433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34980c2cf0 0x7f349806e500 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3488009f70 tx=0x7f3488009840 comp rx=0 tx=0).stop 2023-10-10T18:49:43.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.305+0000 7f349e0f1700 1 -- 172.21.15.175:0/1038092433 shutdown_connections 2023-10-10T18:49:43.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.305+0000 7f349e0f1700 1 --2- 172.21.15.175:0/1038092433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34980c2cf0 0x7f349806e500 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:43.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.305+0000 7f349e0f1700 1 -- 172.21.15.175:0/1038092433 >> 172.21.15.175:0/1038092433 conn(0x7f3498068900 msgr2=0x7f3498068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:43.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.305+0000 7f349e0f1700 1 -- 172.21.15.175:0/1038092433 shutdown_connections 2023-10-10T18:49:43.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.305+0000 7f349e0f1700 1 -- 172.21.15.175:0/1038092433 wait complete. 2023-10-10T18:49:43.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.306+0000 7f349e0f1700 1 Processor -- start 2023-10-10T18:49:43.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.306+0000 7f349e0f1700 1 -- start start 2023-10-10T18:49:43.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.306+0000 7f349e0f1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c2cf0 0x7f3498150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:43.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f349e0f1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980c30c0 0x7f3498166400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:43.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f349e0f1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f349814e290 0x7f3498166940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:43.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f349e0f1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34981509b0 con 0x7f349814e290 2023-10-10T18:49:43.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f349e0f1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3498165fb0 con 0x7f34980c2cf0 2023-10-10T18:49:43.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f349e0f1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349814a480 con 0x7f34980c30c0 2023-10-10T18:49:43.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f349814e290 0x7f3498166940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:43.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f349814e290 0x7f3498166940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43900/0 (socket says 172.21.15.175:43900) 2023-10-10T18:49:43.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 -- 172.21.15.175:0/1608761015 learned_addr learned my addr 172.21.15.175:0/1608761015 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:43.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f34977fe700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c2cf0 0x7f3498150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:43.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3496ffd700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980c30c0 0x7f3498166400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:43.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 -- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980c30c0 msgr2=0x7f3498166400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:43.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980c30c0 0x7f3498166400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:43.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 -- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c2cf0 msgr2=0x7f3498150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:43.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c2cf0 0x7f3498150470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:43.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 -- 172.21.15.175:0/1608761015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3488009ae0 con 0x7f349814e290 2023-10-10T18:49:43.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f3497fff700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f349814e290 0x7f3498166940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f349000c5b0 tx=0x7f3490007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:43.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f347f7fe700 1 -- 172.21.15.175:0/1608761015 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490011590 con 0x7f349814e290 2023-10-10T18:49:43.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f347f7fe700 1 -- 172.21.15.175:0/1608761015 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3490011710 con 0x7f349814e290 2023-10-10T18:49:43.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349814a6d0 con 0x7f349814e290 2023-10-10T18:49:43.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f347f7fe700 1 -- 172.21.15.175:0/1608761015 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490011ad0 con 0x7f349814e290 2023-10-10T18:49:43.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.307+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349814ac90 con 0x7f349814e290 2023-10-10T18:49:43.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.308+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3498166120 con 0x7f349814e290 2023-10-10T18:49:43.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.309+0000 7f347f7fe700 1 -- 172.21.15.175:0/1608761015 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f349001c460 con 0x7f349814e290 2023-10-10T18:49:43.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.309+0000 7f347f7fe700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3484066f30 0x7f34840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:43.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.309+0000 7f347f7fe700 1 -- 172.21.15.175:0/1608761015 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f34900841f0 con 0x7f349814e290 2023-10-10T18:49:43.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.309+0000 7f34977fe700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3484066f30 0x7f34840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:43.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.310+0000 7f34977fe700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3484066f30 0x7f34840693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3488009880 tx=0x7f3488017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:43.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.313+0000 7f347f7fe700 1 -- 172.21.15.175:0/1608761015 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3498166120 con 0x7f349814e290 2023-10-10T18:49:43.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.548+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3498166120 con 0x7f349814e290 2023-10-10T18:49:43.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.549+0000 7f347f7fe700 1 -- 172.21.15.175:0/1608761015 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5438 (secure 0 0 0) 0x7f3498166120 con 0x7f349814e290 2023-10-10T18:49:43.550 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:43.551 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:40.482454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:resolve","state_seq":21,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:43.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3484066f30 msgr2=0x7f34840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:43.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3484066f30 0x7f34840693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3488009880 tx=0x7f3488017000 comp rx=0 tx=0).stop 2023-10-10T18:49:43.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f349814e290 msgr2=0x7f3498166940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:43.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f349814e290 0x7f3498166940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f349000c5b0 tx=0x7f3490007f60 comp rx=0 tx=0).stop 2023-10-10T18:49:43.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 shutdown_connections 2023-10-10T18:49:43.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3484066f30 0x7f34840693e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:43.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f349814e290 0x7f3498166940 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:43.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c2cf0 0x7f3498150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:43.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 --2- 172.21.15.175:0/1608761015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980c30c0 0x7f3498166400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:43.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 >> 172.21.15.175:0/1608761015 conn(0x7f3498068900 msgr2=0x7f3498153110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:43.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 shutdown_connections 2023-10-10T18:49:43.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:43.551+0000 7f349e0f1700 1 -- 172.21.15.175:0/1608761015 wait complete. 2023-10-10T18:49:43.555 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 35 2023-10-10T18:49:43.564 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:49:45.565 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:45.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.690+0000 7f52b2dd5700 1 Processor -- start 2023-10-10T18:49:45.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.690+0000 7f52b2dd5700 1 -- start start 2023-10-10T18:49:45.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b2dd5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52ac155fb0 0x7f52ac159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:45.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b2dd5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac1574b0 con 0x7f52ac0b2390 2023-10-10T18:49:45.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b2dd5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac0afad0 con 0x7f52ac0b1fc0 2023-10-10T18:49:45.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b2dd5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ac150770 con 0x7f52ac155fb0 2023-10-10T18:49:45.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b1372700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52ac155fb0 0x7f52ac159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:45.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b1372700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52ac155fb0 0x7f52ac159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57042/0 (socket says 172.21.15.175:57042) 2023-10-10T18:49:45.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b1372700 1 -- 172.21.15.175:0/1257091822 learned_addr learned my addr 172.21.15.175:0/1257091822 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:45.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b1372700 1 -- 172.21.15.175:0/1257091822 >> v1:172.21.15.191:6789/0 conn(0x7f52ac0b1fc0 legacy=0x7f52ac068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:45.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b1372700 1 -- 172.21.15.175:0/1257091822 >> v1:172.21.15.175:6789/0 conn(0x7f52ac0b2390 legacy=0x7f52ac0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:45.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b1372700 1 -- 172.21.15.175:0/1257091822 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac0af4c0 con 0x7f52ac155fb0 2023-10-10T18:49:45.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52b1372700 1 --2- 172.21.15.175:0/1257091822 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac155fb0 0x7f52ac159920 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009f20 tx=0x7f52a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fc9040f8d54ac5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:45.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52ab7fe700 1 -- 172.21.15.175:0/1257091822 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a400e070 con 0x7f52ac155fb0 2023-10-10T18:49:45.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52ab7fe700 1 -- 172.21.15.175:0/1257091822 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a400bb10 con 0x7f52ac155fb0 2023-10-10T18:49:45.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.691+0000 7f52ab7fe700 1 -- 172.21.15.175:0/1257091822 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40092e0 con 0x7f52ac155fb0 2023-10-10T18:49:45.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.692+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1257091822 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac155fb0 msgr2=0x7f52ac159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:45.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.692+0000 7f52b2dd5700 1 --2- 172.21.15.175:0/1257091822 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac155fb0 0x7f52ac159920 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009f20 tx=0x7f52a4009840 comp rx=0 tx=0).stop 2023-10-10T18:49:45.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.692+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1257091822 shutdown_connections 2023-10-10T18:49:45.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.692+0000 7f52b2dd5700 1 --2- 172.21.15.175:0/1257091822 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac155fb0 0x7f52ac159920 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:45.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.692+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1257091822 >> 172.21.15.175:0/1257091822 conn(0x7f52ac063230 msgr2=0x7f52ac063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:45.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.692+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1257091822 shutdown_connections 2023-10-10T18:49:45.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.692+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1257091822 wait complete. 2023-10-10T18:49:45.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.693+0000 7f52b2dd5700 1 Processor -- start 2023-10-10T18:49:45.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.693+0000 7f52b2dd5700 1 -- start start 2023-10-10T18:49:45.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52b2dd5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ac0b1fc0 0x7f52ac0ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:45.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52b2dd5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac0b2390 0x7f52ac1518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:45.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52b2dd5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ac155fb0 0x7f52ac166390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:45.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52b2dd5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac165f00 con 0x7f52ac0b2390 2023-10-10T18:49:45.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52b2dd5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac150a50 con 0x7f52ac0b1fc0 2023-10-10T18:49:45.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52b2dd5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac165bb0 con 0x7f52ac155fb0 2023-10-10T18:49:45.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac0b2390 0x7f52ac1518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:45.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac0b2390 0x7f52ac1518c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57048/0 (socket says 172.21.15.175:57048) 2023-10-10T18:49:45.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 -- 172.21.15.175:0/1459946827 learned_addr learned my addr 172.21.15.175:0/1459946827 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:45.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52b0b71700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ac0b1fc0 0x7f52ac0ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:45.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52b1372700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ac155fb0 0x7f52ac166390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:45.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 -- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ac155fb0 msgr2=0x7f52ac166390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:45.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ac155fb0 0x7f52ac166390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:45.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 -- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ac0b1fc0 msgr2=0x7f52ac0ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:45.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ac0b1fc0 0x7f52ac0ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:45.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 -- 172.21.15.175:0/1459946827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a4009ae0 con 0x7f52ac0b2390 2023-10-10T18:49:45.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.694+0000 7f52abfff700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac0b2390 0x7f52ac1518c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f529800c590 tx=0x7f529800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:45.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.695+0000 7f52a8ff9700 1 -- 172.21.15.175:0/1459946827 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298011590 con 0x7f52ac0b2390 2023-10-10T18:49:45.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.695+0000 7f52a8ff9700 1 -- 172.21.15.175:0/1459946827 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52980116f0 con 0x7f52ac0b2390 2023-10-10T18:49:45.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.695+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ac150cf0 con 0x7f52ac0b2390 2023-10-10T18:49:45.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.695+0000 7f52a8ff9700 1 -- 172.21.15.175:0/1459946827 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298011ab0 con 0x7f52ac0b2390 2023-10-10T18:49:45.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.695+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ac1512b0 con 0x7f52ac0b2390 2023-10-10T18:49:45.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.696+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ac067570 con 0x7f52ac0b2390 2023-10-10T18:49:45.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.696+0000 7f52a8ff9700 1 -- 172.21.15.175:0/1459946827 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f529801b3b0 con 0x7f52ac0b2390 2023-10-10T18:49:45.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.696+0000 7f52a8ff9700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5290066f30 0x7f52900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:45.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.696+0000 7f52a8ff9700 1 -- 172.21.15.175:0/1459946827 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6117+0+0 (secure 0 0 0) 0x7f5298083cd0 con 0x7f52ac0b2390 2023-10-10T18:49:45.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.697+0000 7f52b0b71700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5290066f30 0x7f52900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:45.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.697+0000 7f52b0b71700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5290066f30 0x7f52900693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f52a0003860 tx=0x7f52a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:45.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.701+0000 7f52a8ff9700 1 -- 172.21.15.175:0/1459946827 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5298050d00 con 0x7f52ac0b2390 2023-10-10T18:49:45.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.924+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52ac068470 con 0x7f52ac0b2390 2023-10-10T18:49:45.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.925+0000 7f52a8ff9700 1 -- 172.21.15.175:0/1459946827 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5438 (secure 0 0 0) 0x7f5298017020 con 0x7f52ac0b2390 2023-10-10T18:49:45.926 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:45.926 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:40.482454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:resolve","state_seq":21,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:45.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5290066f30 msgr2=0x7f52900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:45.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5290066f30 0x7f52900693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f52a0003860 tx=0x7f52a0009040 comp rx=0 tx=0).stop 2023-10-10T18:49:45.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac0b2390 msgr2=0x7f52ac1518c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:45.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac0b2390 0x7f52ac1518c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f529800c590 tx=0x7f529800c950 comp rx=0 tx=0).stop 2023-10-10T18:49:45.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 shutdown_connections 2023-10-10T18:49:45.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5290066f30 0x7f52900693e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:45.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ac155fb0 0x7f52ac166390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:45.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ac0b1fc0 0x7f52ac0ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:45.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 --2- 172.21.15.175:0/1459946827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52ac0b2390 0x7f52ac1518c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:45.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 >> 172.21.15.175:0/1459946827 conn(0x7f52ac063230 msgr2=0x7f52ac0c8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:45.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 shutdown_connections 2023-10-10T18:49:45.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:45.927+0000 7f52b2dd5700 1 -- 172.21.15.175:0/1459946827 wait complete. 2023-10-10T18:49:45.931 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 35 2023-10-10T18:49:45.941 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4248": { "addr": "172.21.15.175:6836/647425682", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 647425682, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 647425682, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4248, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4250": { "addr": "172.21.15.191:6839/246221790", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 246221790, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 246221790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4250, "incarnation": 12, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:resolve", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:49:40.482454+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 4250, "mds_3": 4248, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:49:47.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:48.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.065+0000 7ff95ecf9700 1 Processor -- start 2023-10-10T18:49:48.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.065+0000 7ff95ecf9700 1 -- start start 2023-10-10T18:49:48.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff95ecf9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff9580b2390 0x7ff958155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:48.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff95ecf9700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9581574b0 con 0x7ff9580aff60 2023-10-10T18:49:48.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff95ecf9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9580afad0 con 0x7ff9580b1fc0 2023-10-10T18:49:48.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff95ecf9700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff958150770 con 0x7ff9580b2390 2023-10-10T18:49:48.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff957fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff9580b2390 0x7ff958155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:48.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff957fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff9580b2390 0x7ff958155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57060/0 (socket says 172.21.15.175:57060) 2023-10-10T18:49:48.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff957fff700 1 -- 172.21.15.175:0/673610205 learned_addr learned my addr 172.21.15.175:0/673610205 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:48.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff957fff700 1 -- 172.21.15.175:0/673610205 >> v1:172.21.15.191:6789/0 conn(0x7ff9580b1fc0 legacy=0x7ff958068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:48.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff957fff700 1 -- 172.21.15.175:0/673610205 >> v1:172.21.15.175:6789/0 conn(0x7ff9580aff60 legacy=0x7ff958159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:48.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff957fff700 1 -- 172.21.15.175:0/673610205 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9580af4c0 con 0x7ff9580b2390 2023-10-10T18:49:48.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff957fff700 1 --2- 172.21.15.175:0/673610205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b2390 0x7ff958155fb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff9440099b0 tx=0x7ff944009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e53a449f169fbc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:48.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff9577fe700 1 -- 172.21.15.175:0/673610205 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94400b8f0 con 0x7ff9580b2390 2023-10-10T18:49:48.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff9577fe700 1 -- 172.21.15.175:0/673610205 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94400ba50 con 0x7ff9580b2390 2023-10-10T18:49:48.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.066+0000 7ff9577fe700 1 -- 172.21.15.175:0/673610205 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94400bdf0 con 0x7ff9580b2390 2023-10-10T18:49:48.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.067+0000 7ff95ecf9700 1 -- 172.21.15.175:0/673610205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b2390 msgr2=0x7ff958155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:48.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.067+0000 7ff95ecf9700 1 --2- 172.21.15.175:0/673610205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b2390 0x7ff958155fb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff9440099b0 tx=0x7ff944009250 comp rx=0 tx=0).stop 2023-10-10T18:49:48.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.067+0000 7ff95ecf9700 1 -- 172.21.15.175:0/673610205 shutdown_connections 2023-10-10T18:49:48.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.067+0000 7ff95ecf9700 1 --2- 172.21.15.175:0/673610205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b2390 0x7ff958155fb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:48.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.067+0000 7ff95ecf9700 1 -- 172.21.15.175:0/673610205 >> 172.21.15.175:0/673610205 conn(0x7ff958063230 msgr2=0x7ff958063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:48.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.067+0000 7ff95ecf9700 1 -- 172.21.15.175:0/673610205 shutdown_connections 2023-10-10T18:49:48.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.067+0000 7ff95ecf9700 1 -- 172.21.15.175:0/673610205 wait complete. 2023-10-10T18:49:48.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.068+0000 7ff95ecf9700 1 Processor -- start 2023-10-10T18:49:48.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.068+0000 7ff95ecf9700 1 -- start start 2023-10-10T18:49:48.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95ecf9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9580aff60 0x7ff9580c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:48.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95ecf9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b1fc0 0x7ff9580c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:48.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95ecf9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9580b2390 0x7ff958169820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:48.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95ecf9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c35f0 con 0x7ff9580b1fc0 2023-10-10T18:49:48.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95ecf9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c5dd0 con 0x7ff9580b2390 2023-10-10T18:49:48.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95ecf9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c60b0 con 0x7ff9580aff60 2023-10-10T18:49:48.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b1fc0 0x7ff9580c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:48.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b1fc0 0x7ff9580c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57076/0 (socket says 172.21.15.175:57076) 2023-10-10T18:49:48.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 -- 172.21.15.175:0/3015953678 learned_addr learned my addr 172.21.15.175:0/3015953678 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:48.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95d296700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9580b2390 0x7ff958169820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:48.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95ca95700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9580aff60 0x7ff9580c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:48.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 -- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9580aff60 msgr2=0x7ff9580c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:48.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9580aff60 0x7ff9580c44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:48.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 -- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9580b2390 msgr2=0x7ff958169820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:48.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9580b2390 0x7ff958169820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:48.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 -- 172.21.15.175:0/3015953678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440094f0 con 0x7ff9580b1fc0 2023-10-10T18:49:48.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff95ca95700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9580aff60 0x7ff9580c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:49:48.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.069+0000 7ff957fff700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b1fc0 0x7ff9580c4a30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff95806fd90 tx=0x7ff944017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:48.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.070+0000 7ff954ff9700 1 -- 172.21.15.175:0/3015953678 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94401a660 con 0x7ff9580b1fc0 2023-10-10T18:49:48.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.070+0000 7ff954ff9700 1 -- 172.21.15.175:0/3015953678 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94401a7c0 con 0x7ff9580b1fc0 2023-10-10T18:49:48.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.070+0000 7ff954ff9700 1 -- 172.21.15.175:0/3015953678 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94401ab80 con 0x7ff9580b1fc0 2023-10-10T18:49:48.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.070+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9580c3a10 con 0x7ff9580b1fc0 2023-10-10T18:49:48.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.070+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9580c3fd0 con 0x7ff9580b1fc0 2023-10-10T18:49:48.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.071+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9580c3830 con 0x7ff9580b1fc0 2023-10-10T18:49:48.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.071+0000 7ff954ff9700 1 -- 172.21.15.175:0/3015953678 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff944025440 con 0x7ff9580b1fc0 2023-10-10T18:49:48.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.071+0000 7ff954ff9700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff93c066f30 0x7ff93c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:48.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.071+0000 7ff954ff9700 1 -- 172.21.15.175:0/3015953678 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6117+0+0 (secure 0 0 0) 0x7ff944085e20 con 0x7ff9580b1fc0 2023-10-10T18:49:48.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.072+0000 7ff95ca95700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff93c066f30 0x7ff93c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:48.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.072+0000 7ff95ca95700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff93c066f30 0x7ff93c0693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff94c007fd0 tx=0x7ff94c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:48.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.075+0000 7ff954ff9700 1 -- 172.21.15.175:0/3015953678 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff944052fa0 con 0x7ff9580b1fc0 2023-10-10T18:49:48.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.300+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9580c41c0 con 0x7ff9580b1fc0 2023-10-10T18:49:48.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.301+0000 7ff954ff9700 1 -- 172.21.15.175:0/3015953678 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5440 (secure 0 0 0) 0x7ff944015090 con 0x7ff9580b1fc0 2023-10-10T18:49:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:48.302 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:46.439469+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:48.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff93c066f30 msgr2=0x7ff93c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:48.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff93c066f30 0x7ff93c0693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff94c007fd0 tx=0x7ff94c009040 comp rx=0 tx=0).stop 2023-10-10T18:49:48.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b1fc0 msgr2=0x7ff9580c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:48.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b1fc0 0x7ff9580c4a30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff95806fd90 tx=0x7ff944017000 comp rx=0 tx=0).stop 2023-10-10T18:49:48.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 shutdown_connections 2023-10-10T18:49:48.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff93c066f30 0x7ff93c0693e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:48.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9580b2390 0x7ff958169820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:48.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9580aff60 0x7ff9580c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:48.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 --2- 172.21.15.175:0/3015953678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9580b1fc0 0x7ff9580c4a30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:48.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 >> 172.21.15.175:0/3015953678 conn(0x7ff958063230 msgr2=0x7ff95814fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:48.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 shutdown_connections 2023-10-10T18:49:48.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:48.303+0000 7ff95ecf9700 1 -- 172.21.15.175:0/3015953678 wait complete. 2023-10-10T18:49:48.307 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 36 2023-10-10T18:49:48.315 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:49:48.890 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:49:48.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:49:50.316 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:50.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.438+0000 7f3d56750700 1 Processor -- start 2023-10-10T18:49:50.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.438+0000 7f3d56750700 1 -- start start 2023-10-10T18:49:50.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.439+0000 7f3d56750700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d500c1210 0x7f3d500bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:50.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.439+0000 7f3d56750700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3d5006edb0 0x7f3d500bf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:50.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.439+0000 7f3d56750700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5006b740 con 0x7f3d5014e140 2023-10-10T18:49:50.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.439+0000 7f3d56750700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d500c0960 con 0x7f3d5006edb0 2023-10-10T18:49:50.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.439+0000 7f3d56750700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5006ca30 con 0x7f3d500c1210 2023-10-10T18:49:50.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d54ced700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3d5006edb0 0x7f3d500bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:50.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d54ced700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3d5006edb0 0x7f3d500bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:38666/0 (socket says 172.21.15.175:38666) 2023-10-10T18:49:50.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d54ced700 1 -- 172.21.15.175:0/4216972644 learned_addr learned my addr 172.21.15.175:0/4216972644 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:50.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d4f7fe700 1 --2- 172.21.15.175:0/4216972644 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d500c1210 0x7f3d500bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:50.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d54ced700 1 -- 172.21.15.175:0/4216972644 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d500c1210 msgr2=0x7f3d500bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:50.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d54ced700 1 --2- 172.21.15.175:0/4216972644 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d500c1210 0x7f3d500bf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d54ced700 1 -- 172.21.15.175:0/4216972644 >> v1:172.21.15.191:6789/0 conn(0x7f3d5014e140 legacy=0x7f3d50155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:50.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d54ced700 1 -- 172.21.15.175:0/4216972644 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5006c5a0 con 0x7f3d5006edb0 2023-10-10T18:49:50.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d54ced700 1 --2- 172.21.15.175:0/4216972644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d5006edb0 0x7f3d500bf870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009f50 tx=0x7f3d48009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2f2c08bd2b5bf28 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:50.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d4effd700 1 -- 172.21.15.175:0/4216972644 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4800e070 con 0x7f3d5006edb0 2023-10-10T18:49:50.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d4effd700 1 -- 172.21.15.175:0/4216972644 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4800b9c0 con 0x7f3d5006edb0 2023-10-10T18:49:50.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.440+0000 7f3d4effd700 1 -- 172.21.15.175:0/4216972644 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480092e0 con 0x7f3d5006edb0 2023-10-10T18:49:50.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.441+0000 7f3d56750700 1 -- 172.21.15.175:0/4216972644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d5006edb0 msgr2=0x7f3d500bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:50.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.441+0000 7f3d56750700 1 --2- 172.21.15.175:0/4216972644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d5006edb0 0x7f3d500bf870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009f50 tx=0x7f3d48009840 comp rx=0 tx=0).stop 2023-10-10T18:49:50.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.441+0000 7f3d56750700 1 -- 172.21.15.175:0/4216972644 shutdown_connections 2023-10-10T18:49:50.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.441+0000 7f3d56750700 1 --2- 172.21.15.175:0/4216972644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d5006edb0 0x7f3d500bf870 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.441+0000 7f3d56750700 1 --2- 172.21.15.175:0/4216972644 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d500c1210 0x7f3d500bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.441+0000 7f3d56750700 1 -- 172.21.15.175:0/4216972644 >> 172.21.15.175:0/4216972644 conn(0x7f3d50063cb0 msgr2=0x7f3d5006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:50.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.441+0000 7f3d56750700 1 -- 172.21.15.175:0/4216972644 shutdown_connections 2023-10-10T18:49:50.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.441+0000 7f3d56750700 1 -- 172.21.15.175:0/4216972644 wait complete. 2023-10-10T18:49:50.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.442+0000 7f3d56750700 1 Processor -- start 2023-10-10T18:49:50.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.442+0000 7f3d56750700 1 -- start start 2023-10-10T18:49:50.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d56750700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d5006edb0 0x7f3d5014ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:50.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d56750700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d5014e140 0x7f3d50146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:50.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d56750700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d50164bc0 0x7f3d501473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:50.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d56750700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d501650e0 con 0x7f3d5006edb0 2023-10-10T18:49:50.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d56750700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d501559c0 con 0x7f3d50164bc0 2023-10-10T18:49:50.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d56750700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d50155400 con 0x7f3d5014e140 2023-10-10T18:49:50.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d5006edb0 0x7f3d5014ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:50.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d5006edb0 0x7f3d5014ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57084/0 (socket says 172.21.15.175:57084) 2023-10-10T18:49:50.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 -- 172.21.15.175:0/2818397436 learned_addr learned my addr 172.21.15.175:0/2818397436 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:50.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d54ced700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d50164bc0 0x7f3d501473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:50.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4f7fe700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d5014e140 0x7f3d50146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:50.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 -- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d5014e140 msgr2=0x7f3d50146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:50.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d5014e140 0x7f3d50146e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 -- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d50164bc0 msgr2=0x7f3d501473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:50.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d50164bc0 0x7f3d501473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 -- 172.21.15.175:0/2818397436 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d48009ae0 con 0x7f3d5006edb0 2023-10-10T18:49:50.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d54ced700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d50164bc0 0x7f3d501473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:49:50.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.443+0000 7f3d4ffff700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d5006edb0 0x7f3d5014ede0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000c4e0 tx=0x7f3d4000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:50.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.444+0000 7f3d37fff700 1 -- 172.21.15.175:0/2818397436 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40011590 con 0x7f3d5006edb0 2023-10-10T18:49:50.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.444+0000 7f3d37fff700 1 -- 172.21.15.175:0/2818397436 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d40011710 con 0x7f3d5006edb0 2023-10-10T18:49:50.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.444+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d501556e0 con 0x7f3d5006edb0 2023-10-10T18:49:50.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.444+0000 7f3d37fff700 1 -- 172.21.15.175:0/2818397436 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40011ad0 con 0x7f3d5006edb0 2023-10-10T18:49:50.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.444+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5014f9b0 con 0x7f3d5006edb0 2023-10-10T18:49:50.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.444+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d500c45e0 con 0x7f3d5006edb0 2023-10-10T18:49:50.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.445+0000 7f3d37fff700 1 -- 172.21.15.175:0/2818397436 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3d4001b3b0 con 0x7f3d5006edb0 2023-10-10T18:49:50.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.445+0000 7f3d37fff700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d3c066f30 0x7f3d3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:50.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.445+0000 7f3d37fff700 1 -- 172.21.15.175:0/2818397436 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6117+0+0 (secure 0 0 0) 0x7f3d40083c40 con 0x7f3d5006edb0 2023-10-10T18:49:50.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.445+0000 7f3d4f7fe700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d3c066f30 0x7f3d3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:50.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.446+0000 7f3d4f7fe700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d3c066f30 0x7f3d3c0693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3d500c7bf0 tx=0x7f3d44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:50.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.450+0000 7f3d37fff700 1 -- 172.21.15.175:0/2818397436 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d40050d10 con 0x7f3d5006edb0 2023-10-10T18:49:50.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.683+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d50067570 con 0x7f3d5006edb0 2023-10-10T18:49:50.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.683+0000 7f3d37fff700 1 -- 172.21.15.175:0/2818397436 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5438 (secure 0 0 0) 0x7f3d40017070 con 0x7f3d5006edb0 2023-10-10T18:49:50.685 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:50.685 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:48.448753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:50.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d3c066f30 msgr2=0x7f3d3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:50.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d3c066f30 0x7f3d3c0693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3d500c7bf0 tx=0x7f3d44008040 comp rx=0 tx=0).stop 2023-10-10T18:49:50.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d5006edb0 msgr2=0x7f3d5014ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:50.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d5006edb0 0x7f3d5014ede0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000c4e0 tx=0x7f3d4000c8a0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 shutdown_connections 2023-10-10T18:49:50.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d3c066f30 0x7f3d3c0693e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d50164bc0 0x7f3d501473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d5006edb0 0x7f3d5014ede0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 --2- 172.21.15.175:0/2818397436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d5014e140 0x7f3d50146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:50.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 >> 172.21.15.175:0/2818397436 conn(0x7f3d50063cb0 msgr2=0x7f3d5014b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:50.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 shutdown_connections 2023-10-10T18:49:50.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:50.685+0000 7f3d56750700 1 -- 172.21.15.175:0/2818397436 wait complete. 2023-10-10T18:49:50.690 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 37 2023-10-10T18:49:50.698 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:49:52.699 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:52.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.819+0000 7fe075ba2700 1 Processor -- start 2023-10-10T18:49:52.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.819+0000 7fe075ba2700 1 -- start start 2023-10-10T18:49:52.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe075ba2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe07008a100 0x7fe070159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:52.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe075ba2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe070067fa0 con 0x7fe0700cd040 2023-10-10T18:49:52.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe075ba2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe070156b70 con 0x7fe07014fd90 2023-10-10T18:49:52.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe075ba2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0700c5600 con 0x7fe07008a100 2023-10-10T18:49:52.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe0753a1700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe0700cd040 0x7fe0700684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53920/0 (socket says 172.21.15.175:53920) 2023-10-10T18:49:52.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe0753a1700 1 -- 172.21.15.175:0/1387067144 learned_addr learned my addr 172.21.15.175:0/1387067144 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:49:52.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe06ffff700 1 --2- 172.21.15.175:0/1387067144 >> v2:172.21.15.191:3300/0 conn(0x7fe07008a100 0x7fe070159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:52.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe06f7fe700 1 -- 172.21.15.175:0/1387067144 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697173682 0 0) 0x7fe070067fa0 con 0x7fe0700cd040 2023-10-10T18:49:52.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe06f7fe700 1 -- 172.21.15.175:0/1387067144 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe060003570 con 0x7fe0700cd040 2023-10-10T18:49:52.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe06ffff700 1 -- 172.21.15.175:0/1387067144 >> v1:172.21.15.191:6789/0 conn(0x7fe07014fd90 legacy=0x7fe0700c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:52.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe06ffff700 1 -- 172.21.15.175:0/1387067144 >> v1:172.21.15.175:6789/0 conn(0x7fe0700cd040 legacy=0x7fe0700684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:52.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe06f7fe700 1 -- 172.21.15.175:0/1387067144 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1894856060 0 0) 0x7fe060003570 con 0x7fe0700cd040 2023-10-10T18:49:52.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe06ffff700 1 -- 172.21.15.175:0/1387067144 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070150160 con 0x7fe07008a100 2023-10-10T18:49:52.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.820+0000 7fe06ffff700 1 --2- 172.21.15.175:0/1387067144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 0x7fe070159c30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0098e0 tx=0x7fe05c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=814e92141578c5f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:52.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe06f7fe700 1 -- 172.21.15.175:0/1387067144 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0680017f0 con 0x7fe07008a100 2023-10-10T18:49:52.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe06f7fe700 1 -- 172.21.15.175:0/1387067144 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c011420 con 0x7fe07008a100 2023-10-10T18:49:52.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe06f7fe700 1 -- 172.21.15.175:0/1387067144 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0117c0 con 0x7fe07008a100 2023-10-10T18:49:52.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe075ba2700 1 -- 172.21.15.175:0/1387067144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 msgr2=0x7fe070159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:52.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe075ba2700 1 --2- 172.21.15.175:0/1387067144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 0x7fe070159c30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0098e0 tx=0x7fe05c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:49:52.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe075ba2700 1 -- 172.21.15.175:0/1387067144 shutdown_connections 2023-10-10T18:49:52.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe075ba2700 1 --2- 172.21.15.175:0/1387067144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 0x7fe070159c30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:52.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe075ba2700 1 -- 172.21.15.175:0/1387067144 >> 172.21.15.175:0/1387067144 conn(0x7fe070063d00 msgr2=0x7fe0700cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:52.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe075ba2700 1 -- 172.21.15.175:0/1387067144 shutdown_connections 2023-10-10T18:49:52.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.821+0000 7fe075ba2700 1 -- 172.21.15.175:0/1387067144 wait complete. 2023-10-10T18:49:52.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.822+0000 7fe075ba2700 1 Processor -- start 2023-10-10T18:49:52.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.822+0000 7fe075ba2700 1 -- start start 2023-10-10T18:49:52.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe075ba2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 0x7fe070157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:52.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe075ba2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0700cd040 0x7fe070159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:52.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe075ba2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe07014fd90 0x7fe070169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:52.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe075ba2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070069280 con 0x7fe0700cd040 2023-10-10T18:49:52.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe075ba2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070151260 con 0x7fe07008a100 2023-10-10T18:49:52.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe075ba2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe070150c30 con 0x7fe07014fd90 2023-10-10T18:49:52.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe06ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0700cd040 0x7fe070159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:52.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe06ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0700cd040 0x7fe070159190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57088/0 (socket says 172.21.15.175:57088) 2023-10-10T18:49:52.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe06ffff700 1 -- 172.21.15.175:0/2059495433 learned_addr learned my addr 172.21.15.175:0/2059495433 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:52.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe074ba0700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 0x7fe070157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:52.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.823+0000 7fe0753a1700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe07014fd90 0x7fe070169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:52.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06ffff700 1 -- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe07014fd90 msgr2=0x7fe070169990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:52.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06ffff700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe07014fd90 0x7fe070169990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:52.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06ffff700 1 -- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 msgr2=0x7fe070157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:52.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06ffff700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 0x7fe070157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:52.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06ffff700 1 -- 172.21.15.175:0/2059495433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c009420 con 0x7fe0700cd040 2023-10-10T18:49:52.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe0753a1700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe07014fd90 0x7fe070169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:49:52.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06ffff700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0700cd040 0x7fe070159190 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0091e0 tx=0x7fe05c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:52.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06cff9700 1 -- 172.21.15.175:0/2059495433 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c01cbe0 con 0x7fe0700cd040 2023-10-10T18:49:52.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06cff9700 1 -- 172.21.15.175:0/2059495433 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c01cd40 con 0x7fe0700cd040 2023-10-10T18:49:52.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe070169500 con 0x7fe0700cd040 2023-10-10T18:49:52.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe06cff9700 1 -- 172.21.15.175:0/2059495433 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c01a6a0 con 0x7fe0700cd040 2023-10-10T18:49:52.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.824+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe070156fb0 con 0x7fe0700cd040 2023-10-10T18:49:52.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.825+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070169690 con 0x7fe0700cd040 2023-10-10T18:49:52.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.825+0000 7fe06cff9700 1 -- 172.21.15.175:0/2059495433 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe05c01aa10 con 0x7fe0700cd040 2023-10-10T18:49:52.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.825+0000 7fe06cff9700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe054066f30 0x7fe0540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:52.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.826+0000 7fe06cff9700 1 -- 172.21.15.175:0/2059495433 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fe05c0861a0 con 0x7fe0700cd040 2023-10-10T18:49:52.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.826+0000 7fe074ba0700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe054066f30 0x7fe0540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:52.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.826+0000 7fe074ba0700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe054066f30 0x7fe0540693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe0640039e0 tx=0x7fe064007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:52.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:52.830+0000 7fe06cff9700 1 -- 172.21.15.175:0/2059495433 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe05c053270 con 0x7fe0700cd040 2023-10-10T18:49:53.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.053+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe070067570 con 0x7fe0700cd040 2023-10-10T18:49:53.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.053+0000 7fe06cff9700 1 -- 172.21.15.175:0/2059495433 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5433 (secure 0 0 0) 0x7fe05c056890 con 0x7fe0700cd040 2023-10-10T18:49:53.055 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:53.055 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:52.466616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:53.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe054066f30 msgr2=0x7fe0540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:53.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe054066f30 0x7fe0540693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe0640039e0 tx=0x7fe064007ec0 comp rx=0 tx=0).stop 2023-10-10T18:49:53.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0700cd040 msgr2=0x7fe070159190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:53.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0700cd040 0x7fe070159190 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0091e0 tx=0x7fe05c016000 comp rx=0 tx=0).stop 2023-10-10T18:49:53.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 shutdown_connections 2023-10-10T18:49:53.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe054066f30 0x7fe0540693e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:53.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe07014fd90 0x7fe070169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:53.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe07008a100 0x7fe070157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:53.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 --2- 172.21.15.175:0/2059495433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0700cd040 0x7fe070159190 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:53.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 >> 172.21.15.175:0/2059495433 conn(0x7fe070063d00 msgr2=0x7fe0700c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:53.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 shutdown_connections 2023-10-10T18:49:53.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:53.055+0000 7fe075ba2700 1 -- 172.21.15.175:0/2059495433 wait complete. 2023-10-10T18:49:53.059 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 39 2023-10-10T18:49:53.068 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:49:55.069 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:55.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.194+0000 7efd183cd700 1 Processor -- start 2023-10-10T18:49:55.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.194+0000 7efd183cd700 1 -- start start 2023-10-10T18:49:55.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd183cd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd100b1fc0 0x7efd10155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:55.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd183cd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efd100b2390 0x7efd10068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:55.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd183cd700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd101574b0 con 0x7efd10068bd0 2023-10-10T18:49:55.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd183cd700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd10150770 con 0x7efd100b2390 2023-10-10T18:49:55.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd183cd700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd100afad0 con 0x7efd100b1fc0 2023-10-10T18:49:55.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd15968700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efd100b2390 0x7efd10068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:55.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd15968700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efd100b2390 0x7efd10068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:49722/0 (socket says 172.21.15.175:49722) 2023-10-10T18:49:55.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd16169700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd100b1fc0 0x7efd10155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:55.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd15968700 1 -- 172.21.15.175:0/654085423 learned_addr learned my addr 172.21.15.175:0/654085423 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:55.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd15968700 1 -- 172.21.15.175:0/654085423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd100b1fc0 msgr2=0x7efd10155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:55.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd15968700 1 --2- 172.21.15.175:0/654085423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd100b1fc0 0x7efd10155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd15968700 1 -- 172.21.15.175:0/654085423 >> v1:172.21.15.191:6789/0 conn(0x7efd10068bd0 legacy=0x7efd100aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:55.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd15968700 1 -- 172.21.15.175:0/654085423 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd100af4c0 con 0x7efd100b2390 2023-10-10T18:49:55.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.195+0000 7efd16169700 1 --2- 172.21.15.175:0/654085423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd100b1fc0 0x7efd10155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:49:55.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd15968700 1 --2- 172.21.15.175:0/654085423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 0x7efd10068690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efd040099b0 tx=0x7efd04009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f77c9528ca5ae8d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd15167700 1 -- 172.21.15.175:0/654085423 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04009ea0 con 0x7efd100b2390 2023-10-10T18:49:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd15167700 1 -- 172.21.15.175:0/654085423 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0400b8f0 con 0x7efd100b2390 2023-10-10T18:49:55.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd15167700 1 -- 172.21.15.175:0/654085423 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0400bc90 con 0x7efd100b2390 2023-10-10T18:49:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd183cd700 1 -- 172.21.15.175:0/654085423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 msgr2=0x7efd10068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd183cd700 1 --2- 172.21.15.175:0/654085423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 0x7efd10068690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efd040099b0 tx=0x7efd04009250 comp rx=0 tx=0).stop 2023-10-10T18:49:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd183cd700 1 -- 172.21.15.175:0/654085423 shutdown_connections 2023-10-10T18:49:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd183cd700 1 --2- 172.21.15.175:0/654085423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd100b1fc0 0x7efd10155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd183cd700 1 --2- 172.21.15.175:0/654085423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 0x7efd10068690 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd183cd700 1 -- 172.21.15.175:0/654085423 >> 172.21.15.175:0/654085423 conn(0x7efd10063230 msgr2=0x7efd10063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd183cd700 1 -- 172.21.15.175:0/654085423 shutdown_connections 2023-10-10T18:49:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.196+0000 7efd183cd700 1 -- 172.21.15.175:0/654085423 wait complete. 2023-10-10T18:49:55.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.197+0000 7efd183cd700 1 Processor -- start 2023-10-10T18:49:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.197+0000 7efd183cd700 1 -- start start 2023-10-10T18:49:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd183cd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd10068bd0 0x7efd10166f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd183cd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efd100b1fc0 0x7efd10169190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd183cd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 0x7efd101696d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:55.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd183cd700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd10169da0 con 0x7efd100b1fc0 2023-10-10T18:49:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd183cd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd100cc950 con 0x7efd100b2390 2023-10-10T18:49:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd183cd700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd10068820 con 0x7efd10068bd0 2023-10-10T18:49:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd15968700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efd100b1fc0 0x7efd10169190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd15968700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efd100b1fc0 0x7efd10169190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36784/0 (socket says 172.21.15.175:36784) 2023-10-10T18:49:55.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd15968700 1 -- 172.21.15.175:0/452077706 learned_addr learned my addr 172.21.15.175:0/452077706 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:55.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd1696a700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 0x7efd101696d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:55.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.198+0000 7efd16169700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd10068bd0 0x7efd10166f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:55.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd15968700 1 -- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd10068bd0 msgr2=0x7efd10166f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:55.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd15968700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd10068bd0 0x7efd10166f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd15968700 1 -- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 msgr2=0x7efd101696d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:55.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd15968700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 0x7efd101696d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd15968700 1 -- 172.21.15.175:0/452077706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00009a20 con 0x7efd100b1fc0 2023-10-10T18:49:55.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd15968700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efd100b1fc0 0x7efd10169190 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efd04006fd0 tx=0x7efd04017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:55.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd0e7fc700 1 -- 172.21.15.175:0/452077706 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04000d10 con 0x7efd100b1fc0 2023-10-10T18:49:55.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd0e7fc700 1 -- 172.21.15.175:0/452077706 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0400bc70 con 0x7efd100b1fc0 2023-10-10T18:49:55.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd0e7fc700 1 -- 172.21.15.175:0/452077706 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04025670 con 0x7efd100b1fc0 2023-10-10T18:49:55.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd040094f0 con 0x7efd100b1fc0 2023-10-10T18:49:55.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.199+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd100cc020 con 0x7efd100b1fc0 2023-10-10T18:49:55.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.200+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd100ccae0 con 0x7efd100b1fc0 2023-10-10T18:49:55.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.200+0000 7efd0e7fc700 1 -- 172.21.15.175:0/452077706 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efd0402e440 con 0x7efd100b1fc0 2023-10-10T18:49:55.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.201+0000 7efd0e7fc700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcfc066f30 0x7efcfc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:55.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.201+0000 7efd0e7fc700 1 -- 172.21.15.175:0/452077706 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6117+0+0 (secure 0 0 0) 0x7efd040862e0 con 0x7efd100b1fc0 2023-10-10T18:49:55.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.201+0000 7efd16169700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcfc066f30 0x7efcfc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:55.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.201+0000 7efd16169700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcfc066f30 0x7efcfc0693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efd00009fd0 tx=0x7efd0000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:55.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.206+0000 7efd0e7fc700 1 -- 172.21.15.175:0/452077706 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efd04053390 con 0x7efd100b1fc0 2023-10-10T18:49:55.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.429+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd10069060 con 0x7efd100b1fc0 2023-10-10T18:49:55.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.429+0000 7efd0e7fc700 1 -- 172.21.15.175:0/452077706 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5433 (secure 0 0 0) 0x7efd040569b0 con 0x7efd100b1fc0 2023-10-10T18:49:55.430 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:55.431 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:52.466616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:55.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcfc066f30 msgr2=0x7efcfc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:55.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcfc066f30 0x7efcfc0693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efd00009fd0 tx=0x7efd0000b040 comp rx=0 tx=0).stop 2023-10-10T18:49:55.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efd100b1fc0 msgr2=0x7efd10169190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:55.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efd100b1fc0 0x7efd10169190 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efd04006fd0 tx=0x7efd04017000 comp rx=0 tx=0).stop 2023-10-10T18:49:55.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 shutdown_connections 2023-10-10T18:49:55.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcfc066f30 0x7efcfc0693e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd100b2390 0x7efd101696d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd10068bd0 0x7efd10166f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 --2- 172.21.15.175:0/452077706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efd100b1fc0 0x7efd10169190 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:55.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 >> 172.21.15.175:0/452077706 conn(0x7efd10063230 msgr2=0x7efd1014fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:55.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 shutdown_connections 2023-10-10T18:49:55.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:55.432+0000 7efd183cd700 1 -- 172.21.15.175:0/452077706 wait complete. 2023-10-10T18:49:55.435 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 39 2023-10-10T18:49:55.444 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:49:57.445 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:49:57.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.571+0000 7fcf76c6f700 1 Processor -- start 2023-10-10T18:49:57.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.571+0000 7fcf76c6f700 1 -- start start 2023-10-10T18:49:57.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.571+0000 7fcf76c6f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf700b2390 0x7fcf70155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:57.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.571+0000 7fcf76c6f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700aff60 0x7fcf700b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:57.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.571+0000 7fcf76c6f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf701574b0 con 0x7fcf700b1fc0 2023-10-10T18:49:57.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.571+0000 7fcf76c6f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf70150770 con 0x7fcf700b2390 2023-10-10T18:49:57.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.571+0000 7fcf76c6f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf700afad0 con 0x7fcf700aff60 2023-10-10T18:49:57.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf6ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf700b2390 0x7fcf70155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:57.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf7520c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700aff60 0x7fcf700b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:57.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf6ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf700b2390 0x7fcf70155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:49742/0 (socket says 172.21.15.175:49742) 2023-10-10T18:49:57.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf6ffff700 1 -- 172.21.15.175:0/3878856746 learned_addr learned my addr 172.21.15.175:0/3878856746 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:57.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf7520c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700aff60 0x7fcf700b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:40496/0 (socket says 172.21.15.175:40496) 2023-10-10T18:49:57.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf7520c700 1 -- 172.21.15.175:0/3878856746 >> v2:172.21.15.191:3300/0 conn(0x7fcf700b2390 msgr2=0x7fcf70155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:49:57.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf7520c700 1 --2- 172.21.15.175:0/3878856746 >> v2:172.21.15.191:3300/0 conn(0x7fcf700b2390 0x7fcf70155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf7520c700 1 -- 172.21.15.175:0/3878856746 >> v1:172.21.15.191:6789/0 conn(0x7fcf700b1fc0 legacy=0x7fcf70068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:57.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf7520c700 1 -- 172.21.15.175:0/3878856746 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf700af4c0 con 0x7fcf700aff60 2023-10-10T18:49:57.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf6ffff700 1 --2- 172.21.15.175:0/3878856746 >> v2:172.21.15.191:3300/0 conn(0x7fcf700b2390 0x7fcf70155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:49:57.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf7520c700 1 --2- 172.21.15.175:0/3878856746 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700aff60 0x7fcf700b0400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf680098e0 tx=0x7fcf680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60e0c86187857c4d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:57.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.572+0000 7fcf6f7fe700 1 -- 172.21.15.175:0/3878856746 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6800b6d0 con 0x7fcf700aff60 2023-10-10T18:49:57.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf6f7fe700 1 -- 172.21.15.175:0/3878856746 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6800b830 con 0x7fcf700aff60 2023-10-10T18:49:57.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf6f7fe700 1 -- 172.21.15.175:0/3878856746 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6800bbd0 con 0x7fcf700aff60 2023-10-10T18:49:57.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf76c6f700 1 -- 172.21.15.175:0/3878856746 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700aff60 msgr2=0x7fcf700b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:57.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/3878856746 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700aff60 0x7fcf700b0400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf680098e0 tx=0x7fcf680091a0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf76c6f700 1 -- 172.21.15.175:0/3878856746 shutdown_connections 2023-10-10T18:49:57.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/3878856746 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700aff60 0x7fcf700b0400 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/3878856746 >> v2:172.21.15.191:3300/0 conn(0x7fcf700b2390 0x7fcf70155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf76c6f700 1 -- 172.21.15.175:0/3878856746 >> 172.21.15.175:0/3878856746 conn(0x7fcf70063230 msgr2=0x7fcf70063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:57.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf76c6f700 1 -- 172.21.15.175:0/3878856746 shutdown_connections 2023-10-10T18:49:57.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.573+0000 7fcf76c6f700 1 -- 172.21.15.175:0/3878856746 wait complete. 2023-10-10T18:49:57.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.574+0000 7fcf76c6f700 1 Processor -- start 2023-10-10T18:49:57.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.574+0000 7fcf76c6f700 1 -- start start 2023-10-10T18:49:57.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf76c6f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf700aff60 0x7fcf70169210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:57.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf76c6f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700b1fc0 0x7fcf700cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:57.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf76c6f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf700b2390 0x7fcf7014f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:57.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf76c6f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70156780 con 0x7fcf700b2390 2023-10-10T18:49:57.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf76c6f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70169f50 con 0x7fcf700aff60 2023-10-10T18:49:57.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf76c6f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf701545d0 con 0x7fcf700b1fc0 2023-10-10T18:49:57.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf7520c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf700b2390 0x7fcf7014f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:57.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf6ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700b1fc0 0x7fcf700cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:57.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf74a0b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf700aff60 0x7fcf70169210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:57.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf7520c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf700b2390 0x7fcf7014f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36796/0 (socket says 172.21.15.175:36796) 2023-10-10T18:49:57.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.575+0000 7fcf7520c700 1 -- 172.21.15.175:0/2191444593 learned_addr learned my addr 172.21.15.175:0/2191444593 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:49:57.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6ffff700 1 -- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf700aff60 msgr2=0x7fcf70169210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:57.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6ffff700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf700aff60 0x7fcf70169210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6ffff700 1 -- 172.21.15.175:0/2191444593 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf700b2390 msgr2=0x7fcf7014f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:57.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6ffff700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf700b2390 0x7fcf7014f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6ffff700 1 -- 172.21.15.175:0/2191444593 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf640094f0 con 0x7fcf700b1fc0 2023-10-10T18:49:57.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf7520c700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf700b2390 0x7fcf7014f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:49:57.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6ffff700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700b1fc0 0x7fcf700cb570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf64006fd0 tx=0x7fcf64009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:57.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6cff9700 1 -- 172.21.15.175:0/2191444593 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6400c6a0 con 0x7fcf700b1fc0 2023-10-10T18:49:57.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6cff9700 1 -- 172.21.15.175:0/2191444593 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6400c800 con 0x7fcf700b1fc0 2023-10-10T18:49:57.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf68009420 con 0x7fcf700b1fc0 2023-10-10T18:49:57.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf6cff9700 1 -- 172.21.15.175:0/2191444593 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6400cb80 con 0x7fcf700b1fc0 2023-10-10T18:49:57.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.576+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf70156bd0 con 0x7fcf700b1fc0 2023-10-10T18:49:57.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.577+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf70154430 con 0x7fcf700b1fc0 2023-10-10T18:49:57.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.577+0000 7fcf6cff9700 1 -- 172.21.15.175:0/2191444593 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcf64005a00 con 0x7fcf700b1fc0 2023-10-10T18:49:57.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.578+0000 7fcf6cff9700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf5c066f30 0x7fcf5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:49:57.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.578+0000 7fcf6cff9700 1 -- 172.21.15.175:0/2191444593 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fcf640847d0 con 0x7fcf700b1fc0 2023-10-10T18:49:57.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.578+0000 7fcf74a0b700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf5c066f30 0x7fcf5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:49:57.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.579+0000 7fcf74a0b700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf5c066f30 0x7fcf5c0693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcf60007fd0 tx=0x7fcf600016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:49:57.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.582+0000 7fcf6cff9700 1 -- 172.21.15.175:0/2191444593 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf64051880 con 0x7fcf700b1fc0 2023-10-10T18:49:57.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.807+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf70154430 con 0x7fcf700b1fc0 2023-10-10T18:49:57.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.807+0000 7fcf6cff9700 1 -- 172.21.15.175:0/2191444593 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5430 (secure 0 0 0) 0x7fcf70154430 con 0x7fcf700b1fc0 2023-10-10T18:49:57.808 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:49:57.808 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:57.482323+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:49:57.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf5c066f30 msgr2=0x7fcf5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:57.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf5c066f30 0x7fcf5c0693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcf60007fd0 tx=0x7fcf600016f0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700b1fc0 msgr2=0x7fcf700cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:49:57.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700b1fc0 0x7fcf700cb570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf64006fd0 tx=0x7fcf64009250 comp rx=0 tx=0).stop 2023-10-10T18:49:57.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 shutdown_connections 2023-10-10T18:49:57.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf5c066f30 0x7fcf5c0693e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf700b2390 0x7fcf7014f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf700aff60 0x7fcf70169210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 --2- 172.21.15.175:0/2191444593 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf700b1fc0 0x7fcf700cb570 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:49:57.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 >> 172.21.15.175:0/2191444593 conn(0x7fcf70063230 msgr2=0x7fcf70158d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:49:57.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 shutdown_connections 2023-10-10T18:49:57.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:49:57.810+0000 7fcf76c6f700 1 -- 172.21.15.175:0/2191444593 wait complete. 2023-10-10T18:49:57.813 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 40 2023-10-10T18:49:57.822 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=4, replacing gid=4245 2023-10-10T18:49:57.822 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.a 2023-10-10T18:50:01.525 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T18:50:01.525+0000 7fc568727700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-10T18:50:06.822 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-10T18:50:06.823 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-10T18:50:06.823 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-10T18:50:06.826 INFO:tasks.ceph.mds.a:Started 2023-10-10T18:50:06.827 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:50:06.910 INFO:tasks.ceph.mds.a.smithi175.stdout:starting mds.a at 2023-10-10T18:50:06.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.958+0000 7f091d580700 1 Processor -- start 2023-10-10T18:50:06.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.959+0000 7f091d580700 1 -- start start 2023-10-10T18:50:06.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.959+0000 7f091d580700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09181577a0 0x7f09180c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:06.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.959+0000 7f091d580700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0918154aa0 con 0x7f09180c12a0 2023-10-10T18:50:06.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.959+0000 7f091d580700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0918063dc0 con 0x7f09181573d0 2023-10-10T18:50:06.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.959+0000 7f091d580700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09180bfdf0 con 0x7f09181577a0 2023-10-10T18:50:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.959+0000 7f090ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f09180c12a0 0x7f091814d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47156/0 (socket says 172.21.15.175:47156) 2023-10-10T18:50:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f090ffff700 1 -- 172.21.15.175:0/3138690008 learned_addr learned my addr 172.21.15.175:0/3138690008 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:50:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09177fe700 1 --2- 172.21.15.175:0/3138690008 >> v2:172.21.15.191:3300/0 conn(0x7f09181577a0 0x7f09180c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09167fc700 1 -- 172.21.15.175:0/3138690008 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537419799 0 0) 0x7f0918154aa0 con 0x7f09180c12a0 2023-10-10T18:50:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09167fc700 1 -- 172.21.15.175:0/3138690008 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904003570 con 0x7f09180c12a0 2023-10-10T18:50:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09177fe700 1 -- 172.21.15.175:0/3138690008 >> v1:172.21.15.191:6789/0 conn(0x7f09181573d0 legacy=0x7f09180bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09177fe700 1 -- 172.21.15.175:0/3138690008 >> v1:172.21.15.175:6789/0 conn(0x7f09180c12a0 legacy=0x7f091814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09177fe700 1 -- 172.21.15.175:0/3138690008 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918157ba0 con 0x7f09181577a0 2023-10-10T18:50:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09177fe700 1 --2- 172.21.15.175:0/3138690008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 0x7f09180c4f20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0910009e60 tx=0x7f0910009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec2aec55bb5bd599 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:50:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09167fc700 1 -- 172.21.15.175:0/3138690008 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091000e070 con 0x7f09181577a0 2023-10-10T18:50:06.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09167fc700 1 -- 172.21.15.175:0/3138690008 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091000bb60 con 0x7f09181577a0 2023-10-10T18:50:06.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.960+0000 7f09167fc700 1 -- 172.21.15.175:0/3138690008 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910012430 con 0x7f09181577a0 2023-10-10T18:50:06.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.961+0000 7f091d580700 1 -- 172.21.15.175:0/3138690008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 msgr2=0x7f09180c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:06.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.961+0000 7f091d580700 1 --2- 172.21.15.175:0/3138690008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 0x7f09180c4f20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0910009e60 tx=0x7f0910009780 comp rx=0 tx=0).stop 2023-10-10T18:50:06.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.961+0000 7f091d580700 1 -- 172.21.15.175:0/3138690008 shutdown_connections 2023-10-10T18:50:06.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.961+0000 7f091d580700 1 --2- 172.21.15.175:0/3138690008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 0x7f09180c4f20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:06.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.961+0000 7f091d580700 1 -- 172.21.15.175:0/3138690008 >> 172.21.15.175:0/3138690008 conn(0x7f091806dc70 msgr2=0x7f0918063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:50:06.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.961+0000 7f091d580700 1 -- 172.21.15.175:0/3138690008 shutdown_connections 2023-10-10T18:50:06.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.961+0000 7f091d580700 1 -- 172.21.15.175:0/3138690008 wait complete. 2023-10-10T18:50:06.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.962+0000 7f091d580700 1 Processor -- start 2023-10-10T18:50:06.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.962+0000 7f091d580700 1 -- start start 2023-10-10T18:50:06.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f091d580700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09180c12a0 0x7f09181503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:06.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f091d580700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09181573d0 0x7f0918150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:06.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f091d580700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 0x7f0918166e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:06.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f091d580700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091814ff50 con 0x7f09181573d0 2023-10-10T18:50:06.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f091d580700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0918153980 con 0x7f09181577a0 2023-10-10T18:50:06.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f091d580700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0918154190 con 0x7f09180c12a0 2023-10-10T18:50:06.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f090ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09181573d0 0x7f0918150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:06.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f090ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09181573d0 0x7f0918150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46994/0 (socket says 172.21.15.175:46994) 2023-10-10T18:50:06.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f090ffff700 1 -- 172.21.15.175:0/3121395137 learned_addr learned my addr 172.21.15.175:0/3121395137 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:50:06.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f09177fe700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 0x7f0918166e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:06.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f0916ffd700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09180c12a0 0x7f09181503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:06.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f090ffff700 1 -- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09180c12a0 msgr2=0x7f09181503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:06.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f090ffff700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09180c12a0 0x7f09181503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:06.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f090ffff700 1 -- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 msgr2=0x7f0918166e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:06.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f090ffff700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 0x7f0918166e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:06.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.963+0000 7f090ffff700 1 -- 172.21.15.175:0/3121395137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0910009a20 con 0x7f09181573d0 2023-10-10T18:50:06.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.964+0000 7f0916ffd700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09180c12a0 0x7f09181503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:50:06.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.964+0000 7f090ffff700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09181573d0 0x7f0918150920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f090000c7c0 tx=0x7f0900007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:50:06.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.964+0000 7f090effd700 1 -- 172.21.15.175:0/3121395137 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900005b30 con 0x7f09181573d0 2023-10-10T18:50:06.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.964+0000 7f090effd700 1 -- 172.21.15.175:0/3121395137 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0900005cb0 con 0x7f09181573d0 2023-10-10T18:50:06.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.964+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091814f940 con 0x7f09181573d0 2023-10-10T18:50:06.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.964+0000 7f090effd700 1 -- 172.21.15.175:0/3121395137 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900007320 con 0x7f09181573d0 2023-10-10T18:50:06.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.964+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0918153eb0 con 0x7f09181573d0 2023-10-10T18:50:06.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.965+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0918067570 con 0x7f09181573d0 2023-10-10T18:50:06.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.965+0000 7f090effd700 1 -- 172.21.15.175:0/3121395137 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f090001b3b0 con 0x7f09181573d0 2023-10-10T18:50:06.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.966+0000 7f090effd700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f08f8066f30 0x7f08f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:06.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.966+0000 7f090effd700 1 -- 172.21.15.175:0/3121395137 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6117+0+0 (secure 0 0 0) 0x7f0900083d00 con 0x7f09181573d0 2023-10-10T18:50:06.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.966+0000 7f0916ffd700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f08f8066f30 0x7f08f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:06.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.966+0000 7f0916ffd700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f08f8066f30 0x7f08f80693e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0908007fd0 tx=0x7f0908009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:50:06.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:06.970+0000 7f090effd700 1 -- 172.21.15.175:0/3121395137 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0900050d40 con 0x7f09181573d0 2023-10-10T18:50:07.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.199+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0918157e50 con 0x7f09181573d0 2023-10-10T18:50:07.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.200+0000 7f090effd700 1 -- 172.21.15.175:0/3121395137 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5430 (secure 0 0 0) 0x7f0900017020 con 0x7f09181573d0 2023-10-10T18:50:07.202 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:50:07.202 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:57.482323+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:50:07.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f08f8066f30 msgr2=0x7f08f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:07.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f08f8066f30 0x7f08f80693e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0908007fd0 tx=0x7f0908009040 comp rx=0 tx=0).stop 2023-10-10T18:50:07.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09181573d0 msgr2=0x7f0918150920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:07.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09181573d0 0x7f0918150920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f090000c7c0 tx=0x7f0900007e00 comp rx=0 tx=0).stop 2023-10-10T18:50:07.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 shutdown_connections 2023-10-10T18:50:07.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f08f8066f30 0x7f08f80693e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:07.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09181577a0 0x7f0918166e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:07.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09180c12a0 0x7f09181503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:07.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 --2- 172.21.15.175:0/3121395137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09181573d0 0x7f0918150920 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:07.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 >> 172.21.15.175:0/3121395137 conn(0x7f091806dc70 msgr2=0x7f091814b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:50:07.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 shutdown_connections 2023-10-10T18:50:07.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:07.202+0000 7f091d580700 1 -- 172.21.15.175:0/3121395137 wait complete. 2023-10-10T18:50:07.205 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 40 2023-10-10T18:50:07.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-10T18:50:09.216 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:50:09.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.331+0000 7fb91a737700 1 Processor -- start 2023-10-10T18:50:09.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.331+0000 7fb91a737700 1 -- start start 2023-10-10T18:50:09.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb91a737700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb9140b1fc0 0x7fb914155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:09.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb91a737700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb9140b2390 0x7fb914068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:09.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb91a737700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9141574b0 con 0x7fb914068bd0 2023-10-10T18:50:09.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb91a737700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb914150770 con 0x7fb9140b2390 2023-10-10T18:50:09.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb91a737700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9140afad0 con 0x7fb9140b1fc0 2023-10-10T18:50:09.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb9137fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb9140b2390 0x7fb914068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:09.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb9137fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb9140b2390 0x7fb914068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47008/0 (socket says 172.21.15.175:47008) 2023-10-10T18:50:09.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb9137fe700 1 -- 172.21.15.175:0/1637931763 learned_addr learned my addr 172.21.15.175:0/1637931763 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:50:09.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb913fff700 1 --2- 172.21.15.175:0/1637931763 >> v2:172.21.15.191:3300/0 conn(0x7fb9140b1fc0 0x7fb914155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:09.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb9137fe700 1 -- 172.21.15.175:0/1637931763 >> v2:172.21.15.191:3300/0 conn(0x7fb9140b1fc0 msgr2=0x7fb914155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:50:09.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb9137fe700 1 --2- 172.21.15.175:0/1637931763 >> v2:172.21.15.191:3300/0 conn(0x7fb9140b1fc0 0x7fb914155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb9137fe700 1 -- 172.21.15.175:0/1637931763 >> v1:172.21.15.191:6789/0 conn(0x7fb914068bd0 legacy=0x7fb9140aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:50:09.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb9137fe700 1 -- 172.21.15.175:0/1637931763 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140af4c0 con 0x7fb9140b2390 2023-10-10T18:50:09.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb9137fe700 1 --2- 172.21.15.175:0/1637931763 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9140b2390 0x7fb914068690 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb9140b28c0 tx=0x7fb904009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5493d8d19ab13e69 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:50:09.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb912ffd700 1 -- 172.21.15.175:0/1637931763 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90400b8b0 con 0x7fb9140b2390 2023-10-10T18:50:09.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb912ffd700 1 -- 172.21.15.175:0/1637931763 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90400ba10 con 0x7fb9140b2390 2023-10-10T18:50:09.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.332+0000 7fb912ffd700 1 -- 172.21.15.175:0/1637931763 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90400bdb0 con 0x7fb9140b2390 2023-10-10T18:50:09.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.333+0000 7fb91a737700 1 -- 172.21.15.175:0/1637931763 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9140b2390 msgr2=0x7fb914068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:09.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.333+0000 7fb91a737700 1 --2- 172.21.15.175:0/1637931763 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9140b2390 0x7fb914068690 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb9140b28c0 tx=0x7fb904009250 comp rx=0 tx=0).stop 2023-10-10T18:50:09.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.333+0000 7fb91a737700 1 -- 172.21.15.175:0/1637931763 shutdown_connections 2023-10-10T18:50:09.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.333+0000 7fb91a737700 1 --2- 172.21.15.175:0/1637931763 >> v2:172.21.15.191:3300/0 conn(0x7fb9140b1fc0 0x7fb914155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.333+0000 7fb91a737700 1 --2- 172.21.15.175:0/1637931763 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9140b2390 0x7fb914068690 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.333+0000 7fb91a737700 1 -- 172.21.15.175:0/1637931763 >> 172.21.15.175:0/1637931763 conn(0x7fb914063230 msgr2=0x7fb914063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:50:09.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.333+0000 7fb91a737700 1 -- 172.21.15.175:0/1637931763 shutdown_connections 2023-10-10T18:50:09.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.333+0000 7fb91a737700 1 -- 172.21.15.175:0/1637931763 wait complete. 2023-10-10T18:50:09.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.334+0000 7fb91a737700 1 Processor -- start 2023-10-10T18:50:09.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.334+0000 7fb91a737700 1 -- start start 2023-10-10T18:50:09.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb91a737700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb914068bd0 0x7fb9140cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:09.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb91a737700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9140b1fc0 0x7fb9141691c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:09.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb91a737700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9140b2390 0x7fb914169700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:09.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb91a737700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91416a080 con 0x7fb914068bd0 2023-10-10T18:50:09.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb91a737700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140cb6a0 con 0x7fb9140b2390 2023-10-10T18:50:09.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb91a737700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb914169de0 con 0x7fb9140b1fc0 2023-10-10T18:50:09.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb914068bd0 0x7fb9140cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:09.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb914068bd0 0x7fb9140cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47022/0 (socket says 172.21.15.175:47022) 2023-10-10T18:50:09.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 -- 172.21.15.175:0/8339027 learned_addr learned my addr 172.21.15.175:0/8339027 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:50:09.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb918cd4700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9140b2390 0x7fb914169700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:09.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 -- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9140b1fc0 msgr2=0x7fb9141691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:09.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9140b1fc0 0x7fb9141691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 -- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9140b2390 msgr2=0x7fb914169700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:09.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9140b2390 0x7fb914169700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 -- 172.21.15.175:0/8339027 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9040094f0 con 0x7fb914068bd0 2023-10-10T18:50:09.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb913fff700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb914068bd0 0x7fb9140cbb30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb90800d8c0 tx=0x7fb908008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:50:09.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb8fbfff700 1 -- 172.21.15.175:0/8339027 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90800de60 con 0x7fb914068bd0 2023-10-10T18:50:09.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.335+0000 7fb8fbfff700 1 -- 172.21.15.175:0/8339027 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90800b780 con 0x7fb914068bd0 2023-10-10T18:50:09.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.336+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9140cc7b0 con 0x7fb914068bd0 2023-10-10T18:50:09.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.336+0000 7fb8fbfff700 1 -- 172.21.15.175:0/8339027 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90800bb20 con 0x7fb914068bd0 2023-10-10T18:50:09.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.336+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9140cb050 con 0x7fb914068bd0 2023-10-10T18:50:09.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.337+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9140cb980 con 0x7fb914068bd0 2023-10-10T18:50:09.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.337+0000 7fb8fbfff700 1 -- 172.21.15.175:0/8339027 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb90801e460 con 0x7fb914068bd0 2023-10-10T18:50:09.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.337+0000 7fb8fbfff700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:50:09.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.337+0000 7fb8fbfff700 1 -- 172.21.15.175:0/8339027 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fb908084fa0 con 0x7fb914068bd0 2023-10-10T18:50:09.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.337+0000 7fb9137fe700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:50:09.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.338+0000 7fb9137fe700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb9140b28c0 tx=0x7fb904016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:50:09.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.342+0000 7fb8fbfff700 1 -- 172.21.15.175:0/8339027 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb908051e30 con 0x7fb914068bd0 2023-10-10T18:50:09.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.590+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9140c9680 con 0x7fb914068bd0 2023-10-10T18:50:09.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.591+0000 7fb8fbfff700 1 -- 172.21.15.175:0/8339027 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6189 (secure 0 0 0) 0x7fb908055450 con 0x7fb914068bd0 2023-10-10T18:50:09.595 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:50:09.595 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:49:57.482323+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:50:09.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.592+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 msgr2=0x7fb9000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:09.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.592+0000 7fb91a737700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb9140b28c0 tx=0x7fb904016000 comp rx=0 tx=0).stop 2023-10-10T18:50:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.592+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb914068bd0 msgr2=0x7fb9140cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:50:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.592+0000 7fb91a737700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb914068bd0 0x7fb9140cbb30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb90800d8c0 tx=0x7fb908008040 comp rx=0 tx=0).stop 2023-10-10T18:50:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.593+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 shutdown_connections 2023-10-10T18:50:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.593+0000 7fb91a737700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.593+0000 7fb91a737700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9140b2390 0x7fb914169700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.593+0000 7fb91a737700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb914068bd0 0x7fb9140cbb30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.593+0000 7fb91a737700 1 --2- 172.21.15.175:0/8339027 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9140b1fc0 0x7fb9141691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:50:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.593+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 >> 172.21.15.175:0/8339027 conn(0x7fb914063230 msgr2=0x7fb91414fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:50:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.593+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 shutdown_connections 2023-10-10T18:50:09.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:50:09.593+0000 7fb91a737700 1 -- 172.21.15.175:0/8339027 wait complete. 2023-10-10T18:50:09.599 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 41 2023-10-10T18:50:09.604 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-10T18:50:09.605 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 119 secs before thrashing 2023-10-10T18:50:18.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:50:18.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:50:21.753 INFO:tasks.workunit.client.0.smithi175.stdout:Delete all the files and directories ... 2023-10-10T18:50:25.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/19' 2023-10-10T18:50:25.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/15' 2023-10-10T18:50:25.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/31' 2023-10-10T18:50:25.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/36' 2023-10-10T18:50:25.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/20' 2023-10-10T18:50:25.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/12' 2023-10-10T18:50:25.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/39' 2023-10-10T18:50:25.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/10' 2023-10-10T18:50:25.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/34' 2023-10-10T18:50:25.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/38' 2023-10-10T18:50:25.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/11' 2023-10-10T18:50:25.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/16' 2023-10-10T18:50:25.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/32' 2023-10-10T18:50:25.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/30' 2023-10-10T18:50:25.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/14' 2023-10-10T18:50:25.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/18' 2023-10-10T18:50:25.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/13' 2023-10-10T18:50:25.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/33' 2023-10-10T18:50:25.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/17' 2023-10-10T18:50:25.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/35' 2023-10-10T18:50:25.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './100/37' 2023-10-10T18:50:25.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './100' 2023-10-10T18:50:25.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/19' 2023-10-10T18:50:25.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/15' 2023-10-10T18:50:25.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/31' 2023-10-10T18:50:25.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/36' 2023-10-10T18:50:25.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/20' 2023-10-10T18:50:25.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/12' 2023-10-10T18:50:25.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/39' 2023-10-10T18:50:25.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/10' 2023-10-10T18:50:25.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/34' 2023-10-10T18:50:25.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/38' 2023-10-10T18:50:25.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/11' 2023-10-10T18:50:25.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/16' 2023-10-10T18:50:25.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/32' 2023-10-10T18:50:25.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/30' 2023-10-10T18:50:25.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/14' 2023-10-10T18:50:25.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/18' 2023-10-10T18:50:25.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/13' 2023-10-10T18:50:25.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/33' 2023-10-10T18:50:25.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/17' 2023-10-10T18:50:25.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/35' 2023-10-10T18:50:25.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './101/37' 2023-10-10T18:50:25.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './101' 2023-10-10T18:50:25.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/19' 2023-10-10T18:50:25.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/15' 2023-10-10T18:50:25.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/31' 2023-10-10T18:50:25.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/36' 2023-10-10T18:50:25.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/20' 2023-10-10T18:50:25.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/12' 2023-10-10T18:50:25.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/39' 2023-10-10T18:50:25.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/10' 2023-10-10T18:50:25.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/34' 2023-10-10T18:50:25.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/38' 2023-10-10T18:50:25.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/11' 2023-10-10T18:50:25.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/16' 2023-10-10T18:50:25.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/32' 2023-10-10T18:50:25.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/30' 2023-10-10T18:50:25.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/14' 2023-10-10T18:50:25.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/18' 2023-10-10T18:50:25.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/13' 2023-10-10T18:50:25.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/33' 2023-10-10T18:50:25.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/17' 2023-10-10T18:50:25.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/35' 2023-10-10T18:50:25.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './102/37' 2023-10-10T18:50:25.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './102' 2023-10-10T18:50:25.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/19' 2023-10-10T18:50:25.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/15' 2023-10-10T18:50:25.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/31' 2023-10-10T18:50:25.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/36' 2023-10-10T18:50:25.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/20' 2023-10-10T18:50:25.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/12' 2023-10-10T18:50:25.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/39' 2023-10-10T18:50:25.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/10' 2023-10-10T18:50:25.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/34' 2023-10-10T18:50:25.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/38' 2023-10-10T18:50:25.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/11' 2023-10-10T18:50:25.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/16' 2023-10-10T18:50:25.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/32' 2023-10-10T18:50:25.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/30' 2023-10-10T18:50:25.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/14' 2023-10-10T18:50:25.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/18' 2023-10-10T18:50:25.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/13' 2023-10-10T18:50:25.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/33' 2023-10-10T18:50:25.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/17' 2023-10-10T18:50:25.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/35' 2023-10-10T18:50:25.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './103/37' 2023-10-10T18:50:25.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './103' 2023-10-10T18:50:25.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/19' 2023-10-10T18:50:25.996 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/15' 2023-10-10T18:50:25.996 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/31' 2023-10-10T18:50:25.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/36' 2023-10-10T18:50:25.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/20' 2023-10-10T18:50:25.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/12' 2023-10-10T18:50:25.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/39' 2023-10-10T18:50:25.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/10' 2023-10-10T18:50:25.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/34' 2023-10-10T18:50:25.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/38' 2023-10-10T18:50:25.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/11' 2023-10-10T18:50:25.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/16' 2023-10-10T18:50:25.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/32' 2023-10-10T18:50:25.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/30' 2023-10-10T18:50:25.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/14' 2023-10-10T18:50:25.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/18' 2023-10-10T18:50:25.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/13' 2023-10-10T18:50:25.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/33' 2023-10-10T18:50:26.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/17' 2023-10-10T18:50:26.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/35' 2023-10-10T18:50:26.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './104/37' 2023-10-10T18:50:26.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './104' 2023-10-10T18:50:26.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/19' 2023-10-10T18:50:26.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/15' 2023-10-10T18:50:26.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/31' 2023-10-10T18:50:26.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/36' 2023-10-10T18:50:26.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/20' 2023-10-10T18:50:26.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/12' 2023-10-10T18:50:26.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/39' 2023-10-10T18:50:26.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/10' 2023-10-10T18:50:26.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/34' 2023-10-10T18:50:26.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/38' 2023-10-10T18:50:26.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/11' 2023-10-10T18:50:26.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/16' 2023-10-10T18:50:26.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/32' 2023-10-10T18:50:26.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/30' 2023-10-10T18:50:26.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/14' 2023-10-10T18:50:26.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/18' 2023-10-10T18:50:26.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/13' 2023-10-10T18:50:26.004 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/33' 2023-10-10T18:50:26.004 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/17' 2023-10-10T18:50:26.004 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/35' 2023-10-10T18:50:26.004 INFO:tasks.workunit.client.0.smithi175.stdout:removed './105/37' 2023-10-10T18:50:26.004 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './105' 2023-10-10T18:50:26.005 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/19' 2023-10-10T18:50:26.005 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/15' 2023-10-10T18:50:26.005 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/31' 2023-10-10T18:50:26.005 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/36' 2023-10-10T18:50:26.005 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/20' 2023-10-10T18:50:26.006 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/12' 2023-10-10T18:50:26.006 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/39' 2023-10-10T18:50:26.006 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/10' 2023-10-10T18:50:26.006 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/34' 2023-10-10T18:50:26.006 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/38' 2023-10-10T18:50:26.007 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/11' 2023-10-10T18:50:26.007 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/16' 2023-10-10T18:50:26.007 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/32' 2023-10-10T18:50:26.007 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/30' 2023-10-10T18:50:26.007 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/14' 2023-10-10T18:50:26.007 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/18' 2023-10-10T18:50:26.008 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/13' 2023-10-10T18:50:26.008 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/33' 2023-10-10T18:50:26.008 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/17' 2023-10-10T18:50:26.008 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/35' 2023-10-10T18:50:26.008 INFO:tasks.workunit.client.0.smithi175.stdout:removed './106/37' 2023-10-10T18:50:26.009 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './106' 2023-10-10T18:50:26.009 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/19' 2023-10-10T18:50:26.009 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/15' 2023-10-10T18:50:26.009 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/31' 2023-10-10T18:50:26.009 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/36' 2023-10-10T18:50:26.010 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/20' 2023-10-10T18:50:26.010 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/12' 2023-10-10T18:50:26.010 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/39' 2023-10-10T18:50:26.010 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/10' 2023-10-10T18:50:26.010 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/34' 2023-10-10T18:50:26.011 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/38' 2023-10-10T18:50:26.011 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/11' 2023-10-10T18:50:26.011 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/16' 2023-10-10T18:50:26.011 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/32' 2023-10-10T18:50:26.011 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/30' 2023-10-10T18:50:26.011 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/14' 2023-10-10T18:50:26.012 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/18' 2023-10-10T18:50:26.012 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/13' 2023-10-10T18:50:26.012 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/33' 2023-10-10T18:50:26.012 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/17' 2023-10-10T18:50:26.012 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/35' 2023-10-10T18:50:26.013 INFO:tasks.workunit.client.0.smithi175.stdout:removed './107/37' 2023-10-10T18:50:26.013 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './107' 2023-10-10T18:50:26.013 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/19' 2023-10-10T18:50:26.013 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/15' 2023-10-10T18:50:26.013 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/31' 2023-10-10T18:50:26.014 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/36' 2023-10-10T18:50:26.014 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/20' 2023-10-10T18:50:26.014 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/12' 2023-10-10T18:50:26.014 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/39' 2023-10-10T18:50:26.014 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/10' 2023-10-10T18:50:26.015 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/34' 2023-10-10T18:50:26.015 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/38' 2023-10-10T18:50:26.015 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/11' 2023-10-10T18:50:26.015 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/16' 2023-10-10T18:50:26.015 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/32' 2023-10-10T18:50:26.015 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/30' 2023-10-10T18:50:26.016 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/14' 2023-10-10T18:50:26.016 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/18' 2023-10-10T18:50:26.016 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/13' 2023-10-10T18:50:26.016 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/33' 2023-10-10T18:50:26.016 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/17' 2023-10-10T18:50:26.017 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/35' 2023-10-10T18:50:26.017 INFO:tasks.workunit.client.0.smithi175.stdout:removed './108/37' 2023-10-10T18:50:26.017 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './108' 2023-10-10T18:50:26.017 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/19' 2023-10-10T18:50:26.017 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/15' 2023-10-10T18:50:26.018 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/31' 2023-10-10T18:50:26.018 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/36' 2023-10-10T18:50:26.018 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/20' 2023-10-10T18:50:26.018 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/12' 2023-10-10T18:50:26.018 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/39' 2023-10-10T18:50:26.018 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/10' 2023-10-10T18:50:26.019 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/34' 2023-10-10T18:50:26.019 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/38' 2023-10-10T18:50:26.019 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/11' 2023-10-10T18:50:26.019 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/16' 2023-10-10T18:50:26.019 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/32' 2023-10-10T18:50:26.020 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/30' 2023-10-10T18:50:29.692 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/14' 2023-10-10T18:50:29.692 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/18' 2023-10-10T18:50:29.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/13' 2023-10-10T18:50:29.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/33' 2023-10-10T18:50:29.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/17' 2023-10-10T18:50:29.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/35' 2023-10-10T18:50:29.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed './109/37' 2023-10-10T18:50:29.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './109' 2023-10-10T18:50:29.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/19' 2023-10-10T18:50:29.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/15' 2023-10-10T18:50:29.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/31' 2023-10-10T18:50:29.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/36' 2023-10-10T18:50:29.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/20' 2023-10-10T18:50:29.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/12' 2023-10-10T18:50:29.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/39' 2023-10-10T18:50:29.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/10' 2023-10-10T18:50:29.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/34' 2023-10-10T18:50:29.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/38' 2023-10-10T18:50:29.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/11' 2023-10-10T18:50:29.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/16' 2023-10-10T18:50:29.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/32' 2023-10-10T18:50:29.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/30' 2023-10-10T18:50:29.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/14' 2023-10-10T18:50:29.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/18' 2023-10-10T18:50:29.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/13' 2023-10-10T18:50:29.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/33' 2023-10-10T18:50:29.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/17' 2023-10-10T18:50:29.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/35' 2023-10-10T18:50:29.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './110/37' 2023-10-10T18:50:29.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './110' 2023-10-10T18:50:29.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/19' 2023-10-10T18:50:29.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/15' 2023-10-10T18:50:29.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/31' 2023-10-10T18:50:29.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/36' 2023-10-10T18:50:29.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/20' 2023-10-10T18:50:29.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/12' 2023-10-10T18:50:29.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/39' 2023-10-10T18:50:29.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/10' 2023-10-10T18:50:29.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/34' 2023-10-10T18:50:29.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/38' 2023-10-10T18:50:29.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/11' 2023-10-10T18:50:29.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/16' 2023-10-10T18:50:29.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/32' 2023-10-10T18:50:29.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/30' 2023-10-10T18:50:29.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/14' 2023-10-10T18:50:29.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/18' 2023-10-10T18:50:29.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/13' 2023-10-10T18:50:29.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/33' 2023-10-10T18:50:29.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/17' 2023-10-10T18:50:29.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/35' 2023-10-10T18:50:29.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './111/37' 2023-10-10T18:50:29.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './111' 2023-10-10T18:50:29.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/19' 2023-10-10T18:50:29.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/15' 2023-10-10T18:50:29.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/31' 2023-10-10T18:50:29.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/36' 2023-10-10T18:50:29.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/20' 2023-10-10T18:50:29.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/12' 2023-10-10T18:50:29.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/39' 2023-10-10T18:50:29.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/10' 2023-10-10T18:50:29.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/34' 2023-10-10T18:50:29.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/38' 2023-10-10T18:50:29.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/11' 2023-10-10T18:50:29.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/16' 2023-10-10T18:50:29.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/32' 2023-10-10T18:50:29.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/30' 2023-10-10T18:50:29.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/14' 2023-10-10T18:50:29.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/18' 2023-10-10T18:50:29.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/13' 2023-10-10T18:50:29.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/33' 2023-10-10T18:50:29.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/17' 2023-10-10T18:50:29.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/35' 2023-10-10T18:50:29.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './112/37' 2023-10-10T18:50:29.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './112' 2023-10-10T18:50:29.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/19' 2023-10-10T18:50:29.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/15' 2023-10-10T18:50:29.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/31' 2023-10-10T18:50:29.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/36' 2023-10-10T18:50:29.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/20' 2023-10-10T18:50:29.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/12' 2023-10-10T18:50:29.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/39' 2023-10-10T18:50:29.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/10' 2023-10-10T18:50:29.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/34' 2023-10-10T18:50:29.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/38' 2023-10-10T18:50:29.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/11' 2023-10-10T18:50:29.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/16' 2023-10-10T18:50:29.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/32' 2023-10-10T18:50:29.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/30' 2023-10-10T18:50:29.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/14' 2023-10-10T18:50:29.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/18' 2023-10-10T18:50:29.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/13' 2023-10-10T18:50:29.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/33' 2023-10-10T18:50:29.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/17' 2023-10-10T18:50:29.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/35' 2023-10-10T18:50:29.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './113/37' 2023-10-10T18:50:29.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './113' 2023-10-10T18:50:29.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/19' 2023-10-10T18:50:29.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/15' 2023-10-10T18:50:29.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/31' 2023-10-10T18:50:29.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/36' 2023-10-10T18:50:29.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/20' 2023-10-10T18:50:29.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/12' 2023-10-10T18:50:29.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/39' 2023-10-10T18:50:29.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/10' 2023-10-10T18:50:29.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/34' 2023-10-10T18:50:29.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/38' 2023-10-10T18:50:29.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/11' 2023-10-10T18:50:29.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/16' 2023-10-10T18:50:29.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/32' 2023-10-10T18:50:29.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/30' 2023-10-10T18:50:29.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/14' 2023-10-10T18:50:29.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/18' 2023-10-10T18:50:29.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/13' 2023-10-10T18:50:29.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/33' 2023-10-10T18:50:29.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/17' 2023-10-10T18:50:29.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/35' 2023-10-10T18:50:29.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './114/37' 2023-10-10T18:50:29.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './114' 2023-10-10T18:50:29.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/19' 2023-10-10T18:50:29.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/15' 2023-10-10T18:50:29.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/31' 2023-10-10T18:50:29.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/36' 2023-10-10T18:50:29.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/20' 2023-10-10T18:50:29.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/12' 2023-10-10T18:50:29.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/39' 2023-10-10T18:50:29.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/10' 2023-10-10T18:50:29.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/34' 2023-10-10T18:50:29.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/38' 2023-10-10T18:50:29.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/11' 2023-10-10T18:50:29.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/16' 2023-10-10T18:50:29.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/32' 2023-10-10T18:50:29.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/30' 2023-10-10T18:50:29.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/14' 2023-10-10T18:50:29.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/18' 2023-10-10T18:50:29.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/13' 2023-10-10T18:50:29.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/33' 2023-10-10T18:50:29.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/17' 2023-10-10T18:50:29.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/35' 2023-10-10T18:50:29.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed './115/37' 2023-10-10T18:50:29.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './115' 2023-10-10T18:50:29.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/19' 2023-10-10T18:50:29.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/15' 2023-10-10T18:50:29.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/31' 2023-10-10T18:50:29.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/36' 2023-10-10T18:50:29.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/20' 2023-10-10T18:50:29.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/12' 2023-10-10T18:50:29.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/39' 2023-10-10T18:50:29.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/10' 2023-10-10T18:50:29.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/34' 2023-10-10T18:50:29.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/38' 2023-10-10T18:50:29.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/11' 2023-10-10T18:50:29.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/16' 2023-10-10T18:50:29.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/32' 2023-10-10T18:50:29.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/30' 2023-10-10T18:50:29.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/14' 2023-10-10T18:50:29.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/18' 2023-10-10T18:50:29.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/13' 2023-10-10T18:50:29.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/33' 2023-10-10T18:50:29.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/17' 2023-10-10T18:50:29.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/35' 2023-10-10T18:50:29.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './116/37' 2023-10-10T18:50:29.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './116' 2023-10-10T18:50:29.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/19' 2023-10-10T18:50:29.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/15' 2023-10-10T18:50:29.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/31' 2023-10-10T18:50:29.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/36' 2023-10-10T18:50:29.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/20' 2023-10-10T18:50:29.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/12' 2023-10-10T18:50:29.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/39' 2023-10-10T18:50:29.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/10' 2023-10-10T18:50:29.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/34' 2023-10-10T18:50:29.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/38' 2023-10-10T18:50:29.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/11' 2023-10-10T18:50:29.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/16' 2023-10-10T18:50:29.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/32' 2023-10-10T18:50:29.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/30' 2023-10-10T18:50:29.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/14' 2023-10-10T18:50:29.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/18' 2023-10-10T18:50:29.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/13' 2023-10-10T18:50:29.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/33' 2023-10-10T18:50:29.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/17' 2023-10-10T18:50:29.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/35' 2023-10-10T18:50:29.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './117/37' 2023-10-10T18:50:29.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './117' 2023-10-10T18:50:29.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/19' 2023-10-10T18:50:29.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/15' 2023-10-10T18:50:29.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/31' 2023-10-10T18:50:29.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/36' 2023-10-10T18:50:29.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/20' 2023-10-10T18:50:29.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/12' 2023-10-10T18:50:29.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/39' 2023-10-10T18:50:29.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/10' 2023-10-10T18:50:29.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/34' 2023-10-10T18:50:29.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/38' 2023-10-10T18:50:29.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/11' 2023-10-10T18:50:29.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/16' 2023-10-10T18:50:29.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/32' 2023-10-10T18:50:29.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/30' 2023-10-10T18:50:29.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/14' 2023-10-10T18:50:29.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/18' 2023-10-10T18:50:29.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/13' 2023-10-10T18:50:29.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/33' 2023-10-10T18:50:29.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/17' 2023-10-10T18:50:29.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/35' 2023-10-10T18:50:29.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './118/37' 2023-10-10T18:50:29.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './118' 2023-10-10T18:50:29.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/19' 2023-10-10T18:50:29.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/15' 2023-10-10T18:50:29.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/31' 2023-10-10T18:50:29.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/36' 2023-10-10T18:50:29.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/20' 2023-10-10T18:50:29.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/12' 2023-10-10T18:50:32.563 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/39' 2023-10-10T18:50:32.563 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/10' 2023-10-10T18:50:32.563 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/34' 2023-10-10T18:50:32.563 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/38' 2023-10-10T18:50:32.564 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/11' 2023-10-10T18:50:32.564 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/16' 2023-10-10T18:50:32.564 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/32' 2023-10-10T18:50:32.564 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/30' 2023-10-10T18:50:32.564 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/14' 2023-10-10T18:50:32.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/18' 2023-10-10T18:50:32.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/13' 2023-10-10T18:50:32.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/33' 2023-10-10T18:50:32.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/17' 2023-10-10T18:50:32.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/35' 2023-10-10T18:50:32.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './119/37' 2023-10-10T18:50:32.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './119' 2023-10-10T18:50:32.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/19' 2023-10-10T18:50:32.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/15' 2023-10-10T18:50:32.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/31' 2023-10-10T18:50:32.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/36' 2023-10-10T18:50:32.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/20' 2023-10-10T18:50:32.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/12' 2023-10-10T18:50:32.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/39' 2023-10-10T18:50:32.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/10' 2023-10-10T18:50:32.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/34' 2023-10-10T18:50:32.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/38' 2023-10-10T18:50:32.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/11' 2023-10-10T18:50:32.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/16' 2023-10-10T18:50:32.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/32' 2023-10-10T18:50:32.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/30' 2023-10-10T18:50:32.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/14' 2023-10-10T18:50:32.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/18' 2023-10-10T18:50:32.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/13' 2023-10-10T18:50:32.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/33' 2023-10-10T18:50:32.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/17' 2023-10-10T18:50:32.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/35' 2023-10-10T18:50:32.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './120/37' 2023-10-10T18:50:32.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './120' 2023-10-10T18:50:32.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/19' 2023-10-10T18:50:32.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/15' 2023-10-10T18:50:32.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/31' 2023-10-10T18:50:32.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/36' 2023-10-10T18:50:32.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/20' 2023-10-10T18:50:32.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/12' 2023-10-10T18:50:32.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/39' 2023-10-10T18:50:32.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/10' 2023-10-10T18:50:32.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/34' 2023-10-10T18:50:32.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/38' 2023-10-10T18:50:32.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/11' 2023-10-10T18:50:32.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/16' 2023-10-10T18:50:32.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/32' 2023-10-10T18:50:32.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/30' 2023-10-10T18:50:32.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/14' 2023-10-10T18:50:32.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/18' 2023-10-10T18:50:32.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/13' 2023-10-10T18:50:32.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/33' 2023-10-10T18:50:32.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/17' 2023-10-10T18:50:32.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/35' 2023-10-10T18:50:32.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './121/37' 2023-10-10T18:50:32.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './121' 2023-10-10T18:50:32.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/19' 2023-10-10T18:50:32.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/15' 2023-10-10T18:50:32.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/31' 2023-10-10T18:50:32.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/36' 2023-10-10T18:50:32.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/20' 2023-10-10T18:50:32.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/12' 2023-10-10T18:50:32.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/39' 2023-10-10T18:50:32.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/10' 2023-10-10T18:50:32.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/34' 2023-10-10T18:50:32.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/38' 2023-10-10T18:50:32.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/11' 2023-10-10T18:50:32.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/16' 2023-10-10T18:50:32.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/32' 2023-10-10T18:50:32.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/30' 2023-10-10T18:50:32.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/14' 2023-10-10T18:50:32.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/18' 2023-10-10T18:50:32.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/13' 2023-10-10T18:50:32.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/33' 2023-10-10T18:50:32.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/17' 2023-10-10T18:50:32.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/35' 2023-10-10T18:50:32.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './122/37' 2023-10-10T18:50:32.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './122' 2023-10-10T18:50:32.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/19' 2023-10-10T18:50:32.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/15' 2023-10-10T18:50:32.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/31' 2023-10-10T18:50:32.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/36' 2023-10-10T18:50:32.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/20' 2023-10-10T18:50:32.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/12' 2023-10-10T18:50:32.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/39' 2023-10-10T18:50:32.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/10' 2023-10-10T18:50:32.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/34' 2023-10-10T18:50:32.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/38' 2023-10-10T18:50:32.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/11' 2023-10-10T18:50:32.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/16' 2023-10-10T18:50:32.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/32' 2023-10-10T18:50:32.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/30' 2023-10-10T18:50:32.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/14' 2023-10-10T18:50:32.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/18' 2023-10-10T18:50:32.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/13' 2023-10-10T18:50:32.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/33' 2023-10-10T18:50:32.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/17' 2023-10-10T18:50:32.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/35' 2023-10-10T18:50:32.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './123/37' 2023-10-10T18:50:32.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './123' 2023-10-10T18:50:32.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/19' 2023-10-10T18:50:32.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/15' 2023-10-10T18:50:32.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/31' 2023-10-10T18:50:32.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/36' 2023-10-10T18:50:32.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/20' 2023-10-10T18:50:32.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/12' 2023-10-10T18:50:32.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/39' 2023-10-10T18:50:32.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/10' 2023-10-10T18:50:32.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/34' 2023-10-10T18:50:32.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/38' 2023-10-10T18:50:32.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/11' 2023-10-10T18:50:32.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/16' 2023-10-10T18:50:32.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/32' 2023-10-10T18:50:32.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/30' 2023-10-10T18:50:32.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/14' 2023-10-10T18:50:32.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/18' 2023-10-10T18:50:32.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/13' 2023-10-10T18:50:32.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/33' 2023-10-10T18:50:32.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/17' 2023-10-10T18:50:32.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/35' 2023-10-10T18:50:32.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './124/37' 2023-10-10T18:50:32.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './124' 2023-10-10T18:50:32.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/19' 2023-10-10T18:50:32.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/15' 2023-10-10T18:50:32.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/31' 2023-10-10T18:50:32.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/36' 2023-10-10T18:50:32.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/20' 2023-10-10T18:50:32.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/12' 2023-10-10T18:50:32.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/39' 2023-10-10T18:50:32.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/10' 2023-10-10T18:50:32.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/34' 2023-10-10T18:50:32.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/38' 2023-10-10T18:50:32.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/11' 2023-10-10T18:50:32.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/16' 2023-10-10T18:50:32.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/32' 2023-10-10T18:50:32.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/30' 2023-10-10T18:50:32.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/14' 2023-10-10T18:50:32.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/18' 2023-10-10T18:50:32.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/13' 2023-10-10T18:50:32.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/33' 2023-10-10T18:50:32.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/17' 2023-10-10T18:50:32.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/35' 2023-10-10T18:50:32.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './125/37' 2023-10-10T18:50:32.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './125' 2023-10-10T18:50:32.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/19' 2023-10-10T18:50:32.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/15' 2023-10-10T18:50:32.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/31' 2023-10-10T18:50:32.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/36' 2023-10-10T18:50:32.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/20' 2023-10-10T18:50:32.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/12' 2023-10-10T18:50:32.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/39' 2023-10-10T18:50:32.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/10' 2023-10-10T18:50:32.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/34' 2023-10-10T18:50:32.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/38' 2023-10-10T18:50:32.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/11' 2023-10-10T18:50:32.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/16' 2023-10-10T18:50:32.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/32' 2023-10-10T18:50:32.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/30' 2023-10-10T18:50:32.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/14' 2023-10-10T18:50:32.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/18' 2023-10-10T18:50:32.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/13' 2023-10-10T18:50:32.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/33' 2023-10-10T18:50:32.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/17' 2023-10-10T18:50:32.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/35' 2023-10-10T18:50:32.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './126/37' 2023-10-10T18:50:32.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './126' 2023-10-10T18:50:32.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/19' 2023-10-10T18:50:32.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/15' 2023-10-10T18:50:32.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/31' 2023-10-10T18:50:32.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/36' 2023-10-10T18:50:32.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/20' 2023-10-10T18:50:32.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/12' 2023-10-10T18:50:32.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/39' 2023-10-10T18:50:32.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/10' 2023-10-10T18:50:32.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/34' 2023-10-10T18:50:32.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/38' 2023-10-10T18:50:32.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/11' 2023-10-10T18:50:32.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/16' 2023-10-10T18:50:32.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/32' 2023-10-10T18:50:32.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/30' 2023-10-10T18:50:32.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/14' 2023-10-10T18:50:32.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/18' 2023-10-10T18:50:32.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/13' 2023-10-10T18:50:32.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/33' 2023-10-10T18:50:32.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/17' 2023-10-10T18:50:32.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/35' 2023-10-10T18:50:32.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './127/37' 2023-10-10T18:50:32.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './127' 2023-10-10T18:50:32.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/19' 2023-10-10T18:50:32.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/15' 2023-10-10T18:50:32.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/31' 2023-10-10T18:50:32.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/36' 2023-10-10T18:50:32.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/20' 2023-10-10T18:50:32.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/12' 2023-10-10T18:50:32.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/39' 2023-10-10T18:50:32.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/10' 2023-10-10T18:50:32.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/34' 2023-10-10T18:50:32.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/38' 2023-10-10T18:50:32.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/11' 2023-10-10T18:50:32.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/16' 2023-10-10T18:50:32.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/32' 2023-10-10T18:50:32.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/30' 2023-10-10T18:50:32.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/14' 2023-10-10T18:50:32.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/18' 2023-10-10T18:50:32.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/13' 2023-10-10T18:50:32.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/33' 2023-10-10T18:50:32.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/17' 2023-10-10T18:50:32.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/35' 2023-10-10T18:50:39.211 INFO:tasks.workunit.client.0.smithi175.stdout:removed './128/37' 2023-10-10T18:50:39.212 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './128' 2023-10-10T18:50:39.212 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/19' 2023-10-10T18:50:39.212 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/15' 2023-10-10T18:50:39.213 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/31' 2023-10-10T18:50:39.213 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/36' 2023-10-10T18:50:39.213 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/20' 2023-10-10T18:50:39.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/12' 2023-10-10T18:50:39.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/39' 2023-10-10T18:50:39.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/10' 2023-10-10T18:50:39.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/34' 2023-10-10T18:50:39.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/38' 2023-10-10T18:50:39.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/11' 2023-10-10T18:50:39.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/16' 2023-10-10T18:50:39.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/32' 2023-10-10T18:50:39.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/30' 2023-10-10T18:50:39.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/14' 2023-10-10T18:50:39.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/18' 2023-10-10T18:50:39.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/13' 2023-10-10T18:50:39.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/33' 2023-10-10T18:50:39.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/17' 2023-10-10T18:50:39.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/35' 2023-10-10T18:50:39.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './129/37' 2023-10-10T18:50:39.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './129' 2023-10-10T18:50:39.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/19' 2023-10-10T18:50:39.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/15' 2023-10-10T18:50:39.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/31' 2023-10-10T18:50:39.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/36' 2023-10-10T18:50:39.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/20' 2023-10-10T18:50:39.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/12' 2023-10-10T18:50:39.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/39' 2023-10-10T18:50:39.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/10' 2023-10-10T18:50:39.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/34' 2023-10-10T18:50:39.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/38' 2023-10-10T18:50:39.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/11' 2023-10-10T18:50:39.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/16' 2023-10-10T18:50:39.220 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/32' 2023-10-10T18:50:39.220 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/30' 2023-10-10T18:50:39.220 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/14' 2023-10-10T18:50:39.220 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/18' 2023-10-10T18:50:39.220 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/13' 2023-10-10T18:50:39.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/33' 2023-10-10T18:50:39.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/17' 2023-10-10T18:50:39.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/35' 2023-10-10T18:50:39.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed './130/37' 2023-10-10T18:50:39.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './130' 2023-10-10T18:50:39.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/19' 2023-10-10T18:50:39.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/15' 2023-10-10T18:50:39.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/31' 2023-10-10T18:50:39.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/36' 2023-10-10T18:50:39.223 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/20' 2023-10-10T18:50:39.223 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/12' 2023-10-10T18:50:39.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/39' 2023-10-10T18:50:39.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/10' 2023-10-10T18:50:39.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/34' 2023-10-10T18:50:39.225 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/38' 2023-10-10T18:50:39.225 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/11' 2023-10-10T18:50:39.225 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/16' 2023-10-10T18:50:39.226 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/32' 2023-10-10T18:50:39.226 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/30' 2023-10-10T18:50:39.226 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/14' 2023-10-10T18:50:39.227 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/18' 2023-10-10T18:50:39.227 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/13' 2023-10-10T18:50:39.228 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/33' 2023-10-10T18:50:39.228 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/17' 2023-10-10T18:50:39.228 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/35' 2023-10-10T18:50:39.229 INFO:tasks.workunit.client.0.smithi175.stdout:removed './131/37' 2023-10-10T18:50:39.229 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './131' 2023-10-10T18:50:39.229 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/19' 2023-10-10T18:50:39.230 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/15' 2023-10-10T18:50:39.230 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/31' 2023-10-10T18:50:39.230 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/36' 2023-10-10T18:50:39.231 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/20' 2023-10-10T18:50:39.231 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/12' 2023-10-10T18:50:39.231 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/39' 2023-10-10T18:50:39.232 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/10' 2023-10-10T18:50:39.232 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/34' 2023-10-10T18:50:39.232 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/38' 2023-10-10T18:50:39.233 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/11' 2023-10-10T18:50:39.233 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/16' 2023-10-10T18:50:39.233 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/32' 2023-10-10T18:50:39.234 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/30' 2023-10-10T18:50:39.234 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/14' 2023-10-10T18:50:39.234 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/18' 2023-10-10T18:50:39.235 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/13' 2023-10-10T18:50:39.235 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/33' 2023-10-10T18:50:39.235 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/17' 2023-10-10T18:50:39.236 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/35' 2023-10-10T18:50:39.236 INFO:tasks.workunit.client.0.smithi175.stdout:removed './132/37' 2023-10-10T18:50:39.236 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './132' 2023-10-10T18:50:39.237 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/19' 2023-10-10T18:50:39.237 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/15' 2023-10-10T18:50:39.237 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/31' 2023-10-10T18:50:39.238 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/36' 2023-10-10T18:50:39.238 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/20' 2023-10-10T18:50:39.238 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/12' 2023-10-10T18:50:39.239 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/39' 2023-10-10T18:50:39.239 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/10' 2023-10-10T18:50:39.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/34' 2023-10-10T18:50:39.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/38' 2023-10-10T18:50:39.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/11' 2023-10-10T18:50:39.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/16' 2023-10-10T18:50:39.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/32' 2023-10-10T18:50:39.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/30' 2023-10-10T18:50:39.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/14' 2023-10-10T18:50:39.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/18' 2023-10-10T18:50:39.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/13' 2023-10-10T18:50:39.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/33' 2023-10-10T18:50:39.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/17' 2023-10-10T18:50:39.243 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/35' 2023-10-10T18:50:39.243 INFO:tasks.workunit.client.0.smithi175.stdout:removed './133/37' 2023-10-10T18:50:39.243 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './133' 2023-10-10T18:50:39.244 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/19' 2023-10-10T18:50:39.244 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/15' 2023-10-10T18:50:39.244 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/31' 2023-10-10T18:50:39.245 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/36' 2023-10-10T18:50:39.245 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/20' 2023-10-10T18:50:39.245 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/12' 2023-10-10T18:50:39.246 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/39' 2023-10-10T18:50:39.246 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/10' 2023-10-10T18:50:39.246 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/34' 2023-10-10T18:50:39.247 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/38' 2023-10-10T18:50:39.247 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/11' 2023-10-10T18:50:39.247 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/16' 2023-10-10T18:50:39.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/32' 2023-10-10T18:50:39.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/30' 2023-10-10T18:50:39.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/14' 2023-10-10T18:50:39.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/18' 2023-10-10T18:50:39.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/13' 2023-10-10T18:50:39.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/33' 2023-10-10T18:50:39.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/17' 2023-10-10T18:50:39.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/35' 2023-10-10T18:50:39.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed './134/37' 2023-10-10T18:50:39.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './134' 2023-10-10T18:50:39.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/19' 2023-10-10T18:50:39.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/15' 2023-10-10T18:50:39.251 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/31' 2023-10-10T18:50:39.251 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/36' 2023-10-10T18:50:39.251 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/20' 2023-10-10T18:50:39.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/12' 2023-10-10T18:50:39.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/39' 2023-10-10T18:50:39.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/10' 2023-10-10T18:50:39.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/34' 2023-10-10T18:50:39.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/38' 2023-10-10T18:50:39.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/11' 2023-10-10T18:50:39.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/16' 2023-10-10T18:50:39.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/32' 2023-10-10T18:50:39.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/30' 2023-10-10T18:50:39.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/14' 2023-10-10T18:50:39.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/18' 2023-10-10T18:50:39.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/13' 2023-10-10T18:50:39.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/33' 2023-10-10T18:50:39.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/17' 2023-10-10T18:50:39.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/35' 2023-10-10T18:50:39.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './135/37' 2023-10-10T18:50:39.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './135' 2023-10-10T18:50:39.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/19' 2023-10-10T18:50:39.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/15' 2023-10-10T18:50:39.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/31' 2023-10-10T18:50:39.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/36' 2023-10-10T18:50:39.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/20' 2023-10-10T18:50:39.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/12' 2023-10-10T18:50:39.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/39' 2023-10-10T18:50:39.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/10' 2023-10-10T18:50:39.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/34' 2023-10-10T18:50:39.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/38' 2023-10-10T18:50:39.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/11' 2023-10-10T18:50:39.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/16' 2023-10-10T18:50:39.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/32' 2023-10-10T18:50:39.258 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/30' 2023-10-10T18:50:39.258 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/14' 2023-10-10T18:50:39.258 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/18' 2023-10-10T18:50:39.259 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/13' 2023-10-10T18:50:39.259 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/33' 2023-10-10T18:50:39.259 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/17' 2023-10-10T18:50:39.260 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/35' 2023-10-10T18:50:39.260 INFO:tasks.workunit.client.0.smithi175.stdout:removed './136/37' 2023-10-10T18:50:39.260 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './136' 2023-10-10T18:50:39.260 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/19' 2023-10-10T18:50:39.260 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/15' 2023-10-10T18:50:39.260 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/31' 2023-10-10T18:50:39.261 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/36' 2023-10-10T18:50:39.261 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/20' 2023-10-10T18:50:39.261 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/12' 2023-10-10T18:50:39.261 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/39' 2023-10-10T18:50:39.261 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/10' 2023-10-10T18:50:39.262 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/34' 2023-10-10T18:50:39.262 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/38' 2023-10-10T18:50:39.262 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/11' 2023-10-10T18:50:39.262 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/16' 2023-10-10T18:50:39.263 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/32' 2023-10-10T18:50:39.263 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/30' 2023-10-10T18:50:39.263 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/14' 2023-10-10T18:50:39.263 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/18' 2023-10-10T18:50:39.263 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/13' 2023-10-10T18:50:39.264 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/33' 2023-10-10T18:50:39.264 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/17' 2023-10-10T18:50:39.264 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/35' 2023-10-10T18:50:39.264 INFO:tasks.workunit.client.0.smithi175.stdout:removed './137/37' 2023-10-10T18:50:39.264 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './137' 2023-10-10T18:50:39.265 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/19' 2023-10-10T18:50:39.265 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/15' 2023-10-10T18:50:39.265 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/31' 2023-10-10T18:50:39.265 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/36' 2023-10-10T18:50:39.265 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/20' 2023-10-10T18:50:39.265 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/12' 2023-10-10T18:50:39.266 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/39' 2023-10-10T18:50:39.266 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/10' 2023-10-10T18:50:39.266 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/34' 2023-10-10T18:50:39.266 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/38' 2023-10-10T18:50:39.266 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/11' 2023-10-10T18:50:39.267 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/16' 2023-10-10T18:50:42.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/32' 2023-10-10T18:50:42.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/30' 2023-10-10T18:50:42.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/14' 2023-10-10T18:50:42.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/18' 2023-10-10T18:50:42.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/13' 2023-10-10T18:50:42.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/33' 2023-10-10T18:50:42.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/17' 2023-10-10T18:50:42.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/35' 2023-10-10T18:50:42.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './138/37' 2023-10-10T18:50:42.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './138' 2023-10-10T18:50:42.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/19' 2023-10-10T18:50:42.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/15' 2023-10-10T18:50:42.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/31' 2023-10-10T18:50:42.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/36' 2023-10-10T18:50:42.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/20' 2023-10-10T18:50:42.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/12' 2023-10-10T18:50:42.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/39' 2023-10-10T18:50:42.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/10' 2023-10-10T18:50:42.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/34' 2023-10-10T18:50:42.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/38' 2023-10-10T18:50:42.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/11' 2023-10-10T18:50:42.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/16' 2023-10-10T18:50:42.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/32' 2023-10-10T18:50:42.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/30' 2023-10-10T18:50:42.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/14' 2023-10-10T18:50:42.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/18' 2023-10-10T18:50:42.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/13' 2023-10-10T18:50:42.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/33' 2023-10-10T18:50:42.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/17' 2023-10-10T18:50:42.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/35' 2023-10-10T18:50:42.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './139/37' 2023-10-10T18:50:42.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './139' 2023-10-10T18:50:42.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/19' 2023-10-10T18:50:42.735 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/15' 2023-10-10T18:50:42.735 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/31' 2023-10-10T18:50:42.735 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/36' 2023-10-10T18:50:42.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/20' 2023-10-10T18:50:42.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/12' 2023-10-10T18:50:42.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/39' 2023-10-10T18:50:42.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/10' 2023-10-10T18:50:42.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/34' 2023-10-10T18:50:42.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/38' 2023-10-10T18:50:42.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/11' 2023-10-10T18:50:42.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/16' 2023-10-10T18:50:42.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/32' 2023-10-10T18:50:42.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/30' 2023-10-10T18:50:42.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/14' 2023-10-10T18:50:42.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/18' 2023-10-10T18:50:42.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/13' 2023-10-10T18:50:42.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/33' 2023-10-10T18:50:42.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/17' 2023-10-10T18:50:42.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/35' 2023-10-10T18:50:42.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './140/37' 2023-10-10T18:50:42.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './140' 2023-10-10T18:50:42.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/19' 2023-10-10T18:50:42.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/15' 2023-10-10T18:50:42.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/31' 2023-10-10T18:50:42.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/36' 2023-10-10T18:50:42.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/20' 2023-10-10T18:50:42.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/12' 2023-10-10T18:50:42.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/39' 2023-10-10T18:50:42.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/10' 2023-10-10T18:50:42.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/34' 2023-10-10T18:50:42.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/38' 2023-10-10T18:50:42.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/11' 2023-10-10T18:50:42.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/16' 2023-10-10T18:50:42.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/32' 2023-10-10T18:50:42.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/30' 2023-10-10T18:50:42.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/14' 2023-10-10T18:50:42.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/18' 2023-10-10T18:50:42.743 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/13' 2023-10-10T18:50:42.743 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/33' 2023-10-10T18:50:42.743 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/17' 2023-10-10T18:50:42.743 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/35' 2023-10-10T18:50:42.743 INFO:tasks.workunit.client.0.smithi175.stdout:removed './141/37' 2023-10-10T18:50:42.743 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './141' 2023-10-10T18:50:42.744 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/19' 2023-10-10T18:50:42.744 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/15' 2023-10-10T18:50:42.744 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/31' 2023-10-10T18:50:42.744 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/36' 2023-10-10T18:50:42.744 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/20' 2023-10-10T18:50:42.745 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/12' 2023-10-10T18:50:42.745 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/39' 2023-10-10T18:50:42.745 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/10' 2023-10-10T18:50:42.745 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/34' 2023-10-10T18:50:42.745 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/38' 2023-10-10T18:50:42.746 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/11' 2023-10-10T18:50:42.746 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/16' 2023-10-10T18:50:42.746 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/32' 2023-10-10T18:50:42.746 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/30' 2023-10-10T18:50:42.747 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/14' 2023-10-10T18:50:42.747 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/18' 2023-10-10T18:50:42.747 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/13' 2023-10-10T18:50:42.747 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/33' 2023-10-10T18:50:42.747 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/17' 2023-10-10T18:50:42.748 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/35' 2023-10-10T18:50:42.748 INFO:tasks.workunit.client.0.smithi175.stdout:removed './142/37' 2023-10-10T18:50:42.748 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './142' 2023-10-10T18:50:42.748 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/19' 2023-10-10T18:50:42.748 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/15' 2023-10-10T18:50:42.749 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/31' 2023-10-10T18:50:42.749 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/36' 2023-10-10T18:50:42.749 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/20' 2023-10-10T18:50:42.749 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/12' 2023-10-10T18:50:42.749 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/39' 2023-10-10T18:50:42.750 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/10' 2023-10-10T18:50:42.750 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/34' 2023-10-10T18:50:42.750 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/38' 2023-10-10T18:50:42.750 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/11' 2023-10-10T18:50:42.750 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/16' 2023-10-10T18:50:42.751 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/32' 2023-10-10T18:50:42.751 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/30' 2023-10-10T18:50:42.751 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/14' 2023-10-10T18:50:42.751 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/18' 2023-10-10T18:50:42.751 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/13' 2023-10-10T18:50:42.752 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/33' 2023-10-10T18:50:42.752 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/17' 2023-10-10T18:50:42.752 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/35' 2023-10-10T18:50:42.752 INFO:tasks.workunit.client.0.smithi175.stdout:removed './143/37' 2023-10-10T18:50:42.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './143' 2023-10-10T18:50:42.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/19' 2023-10-10T18:50:42.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/15' 2023-10-10T18:50:42.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/31' 2023-10-10T18:50:42.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/36' 2023-10-10T18:50:42.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/20' 2023-10-10T18:50:42.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/12' 2023-10-10T18:50:42.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/39' 2023-10-10T18:50:42.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/10' 2023-10-10T18:50:42.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/34' 2023-10-10T18:50:42.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/38' 2023-10-10T18:50:42.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/11' 2023-10-10T18:50:42.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/16' 2023-10-10T18:50:42.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/32' 2023-10-10T18:50:42.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/30' 2023-10-10T18:50:42.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/14' 2023-10-10T18:50:42.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/18' 2023-10-10T18:50:42.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/13' 2023-10-10T18:50:42.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/33' 2023-10-10T18:50:42.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/17' 2023-10-10T18:50:42.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/35' 2023-10-10T18:50:42.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './144/37' 2023-10-10T18:50:42.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './144' 2023-10-10T18:50:42.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/19' 2023-10-10T18:50:42.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/15' 2023-10-10T18:50:42.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/31' 2023-10-10T18:50:42.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/36' 2023-10-10T18:50:42.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/20' 2023-10-10T18:50:42.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/12' 2023-10-10T18:50:42.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/39' 2023-10-10T18:50:42.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/10' 2023-10-10T18:50:42.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/34' 2023-10-10T18:50:42.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/38' 2023-10-10T18:50:42.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/11' 2023-10-10T18:50:42.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/16' 2023-10-10T18:50:42.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/32' 2023-10-10T18:50:42.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/30' 2023-10-10T18:50:42.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/14' 2023-10-10T18:50:42.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/18' 2023-10-10T18:50:42.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/13' 2023-10-10T18:50:42.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/33' 2023-10-10T18:50:42.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/17' 2023-10-10T18:50:42.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/35' 2023-10-10T18:50:42.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './145/37' 2023-10-10T18:50:42.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './145' 2023-10-10T18:50:42.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/19' 2023-10-10T18:50:42.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/15' 2023-10-10T18:50:42.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/31' 2023-10-10T18:50:42.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/36' 2023-10-10T18:50:42.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/20' 2023-10-10T18:50:42.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/12' 2023-10-10T18:50:42.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/39' 2023-10-10T18:50:42.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/10' 2023-10-10T18:50:42.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/34' 2023-10-10T18:50:42.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/38' 2023-10-10T18:50:42.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/11' 2023-10-10T18:50:42.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/16' 2023-10-10T18:50:42.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/32' 2023-10-10T18:50:42.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/30' 2023-10-10T18:50:42.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/14' 2023-10-10T18:50:42.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/18' 2023-10-10T18:50:42.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/13' 2023-10-10T18:50:42.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/33' 2023-10-10T18:50:42.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/17' 2023-10-10T18:50:42.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/35' 2023-10-10T18:50:42.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './146/37' 2023-10-10T18:50:42.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './146' 2023-10-10T18:50:42.766 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/19' 2023-10-10T18:50:42.766 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/15' 2023-10-10T18:50:42.766 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/31' 2023-10-10T18:50:42.766 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/36' 2023-10-10T18:50:42.766 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/20' 2023-10-10T18:50:42.767 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/12' 2023-10-10T18:50:42.767 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/39' 2023-10-10T18:50:42.767 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/10' 2023-10-10T18:50:42.767 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/34' 2023-10-10T18:50:42.767 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/38' 2023-10-10T18:50:42.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/11' 2023-10-10T18:50:42.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/16' 2023-10-10T18:50:42.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/32' 2023-10-10T18:50:42.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/30' 2023-10-10T18:50:42.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/14' 2023-10-10T18:50:42.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/18' 2023-10-10T18:50:42.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/13' 2023-10-10T18:50:42.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/33' 2023-10-10T18:50:42.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/17' 2023-10-10T18:50:42.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/35' 2023-10-10T18:50:42.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed './147/37' 2023-10-10T18:50:42.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './147' 2023-10-10T18:50:42.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/19' 2023-10-10T18:50:42.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/15' 2023-10-10T18:50:42.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/31' 2023-10-10T18:50:42.771 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/36' 2023-10-10T18:50:45.564 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/20' 2023-10-10T18:50:45.564 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/12' 2023-10-10T18:50:45.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/39' 2023-10-10T18:50:45.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/10' 2023-10-10T18:50:45.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/34' 2023-10-10T18:50:45.565 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/38' 2023-10-10T18:50:45.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/11' 2023-10-10T18:50:45.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/16' 2023-10-10T18:50:45.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/32' 2023-10-10T18:50:45.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/30' 2023-10-10T18:50:45.566 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/14' 2023-10-10T18:50:45.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/18' 2023-10-10T18:50:45.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/13' 2023-10-10T18:50:45.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/33' 2023-10-10T18:50:45.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/17' 2023-10-10T18:50:45.567 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/35' 2023-10-10T18:50:45.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './148/37' 2023-10-10T18:50:45.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './148' 2023-10-10T18:50:45.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/19' 2023-10-10T18:50:45.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/15' 2023-10-10T18:50:45.568 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/31' 2023-10-10T18:50:45.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/36' 2023-10-10T18:50:45.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/20' 2023-10-10T18:50:45.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/12' 2023-10-10T18:50:45.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/39' 2023-10-10T18:50:45.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/10' 2023-10-10T18:50:45.569 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/34' 2023-10-10T18:50:45.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/38' 2023-10-10T18:50:45.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/11' 2023-10-10T18:50:45.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/16' 2023-10-10T18:50:45.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/32' 2023-10-10T18:50:45.570 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/30' 2023-10-10T18:50:45.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/14' 2023-10-10T18:50:45.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/18' 2023-10-10T18:50:45.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/13' 2023-10-10T18:50:45.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/33' 2023-10-10T18:50:45.571 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/17' 2023-10-10T18:50:45.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/35' 2023-10-10T18:50:45.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed './149/37' 2023-10-10T18:50:45.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './149' 2023-10-10T18:50:45.572 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/19' 2023-10-10T18:50:45.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/15' 2023-10-10T18:50:45.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/31' 2023-10-10T18:50:45.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/36' 2023-10-10T18:50:45.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/20' 2023-10-10T18:50:45.573 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/12' 2023-10-10T18:50:45.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/39' 2023-10-10T18:50:45.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/10' 2023-10-10T18:50:45.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/34' 2023-10-10T18:50:45.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/38' 2023-10-10T18:50:45.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/11' 2023-10-10T18:50:45.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/16' 2023-10-10T18:50:45.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/32' 2023-10-10T18:50:45.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/30' 2023-10-10T18:50:45.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/14' 2023-10-10T18:50:45.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/18' 2023-10-10T18:50:45.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/13' 2023-10-10T18:50:45.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/33' 2023-10-10T18:50:45.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/17' 2023-10-10T18:50:45.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/35' 2023-10-10T18:50:45.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './150/37' 2023-10-10T18:50:45.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './150' 2023-10-10T18:50:45.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/19' 2023-10-10T18:50:45.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/15' 2023-10-10T18:50:45.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/31' 2023-10-10T18:50:45.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/36' 2023-10-10T18:50:45.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/20' 2023-10-10T18:50:45.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/12' 2023-10-10T18:50:45.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/39' 2023-10-10T18:50:45.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/10' 2023-10-10T18:50:45.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/34' 2023-10-10T18:50:45.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/38' 2023-10-10T18:50:45.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/11' 2023-10-10T18:50:45.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/16' 2023-10-10T18:50:45.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/32' 2023-10-10T18:50:45.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/30' 2023-10-10T18:50:45.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/14' 2023-10-10T18:50:45.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/18' 2023-10-10T18:50:45.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/13' 2023-10-10T18:50:45.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/33' 2023-10-10T18:50:45.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/17' 2023-10-10T18:50:45.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/35' 2023-10-10T18:50:45.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './151/37' 2023-10-10T18:50:45.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './151' 2023-10-10T18:50:45.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/19' 2023-10-10T18:50:45.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/15' 2023-10-10T18:50:45.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/31' 2023-10-10T18:50:45.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/36' 2023-10-10T18:50:45.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/20' 2023-10-10T18:50:45.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/12' 2023-10-10T18:50:45.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/39' 2023-10-10T18:50:45.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/10' 2023-10-10T18:50:45.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/34' 2023-10-10T18:50:45.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/38' 2023-10-10T18:50:45.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/11' 2023-10-10T18:50:45.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/16' 2023-10-10T18:50:45.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/32' 2023-10-10T18:50:45.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/30' 2023-10-10T18:50:45.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/14' 2023-10-10T18:50:45.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/18' 2023-10-10T18:50:45.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/13' 2023-10-10T18:50:45.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/33' 2023-10-10T18:50:45.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/17' 2023-10-10T18:50:45.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/35' 2023-10-10T18:50:45.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './152/37' 2023-10-10T18:50:45.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './152' 2023-10-10T18:50:45.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/19' 2023-10-10T18:50:45.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/15' 2023-10-10T18:50:45.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/31' 2023-10-10T18:50:45.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/36' 2023-10-10T18:50:45.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/20' 2023-10-10T18:50:45.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/12' 2023-10-10T18:50:45.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/39' 2023-10-10T18:50:45.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/10' 2023-10-10T18:50:45.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/34' 2023-10-10T18:50:45.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/38' 2023-10-10T18:50:45.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/11' 2023-10-10T18:50:45.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/16' 2023-10-10T18:50:45.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/32' 2023-10-10T18:50:45.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/30' 2023-10-10T18:50:45.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/14' 2023-10-10T18:50:45.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/18' 2023-10-10T18:50:45.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/13' 2023-10-10T18:50:45.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/33' 2023-10-10T18:50:45.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/17' 2023-10-10T18:50:45.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/35' 2023-10-10T18:50:45.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './153/37' 2023-10-10T18:50:45.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './153' 2023-10-10T18:50:45.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/19' 2023-10-10T18:50:45.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/15' 2023-10-10T18:50:45.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/31' 2023-10-10T18:50:45.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/36' 2023-10-10T18:50:45.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/20' 2023-10-10T18:50:45.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/12' 2023-10-10T18:50:45.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/39' 2023-10-10T18:50:45.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/10' 2023-10-10T18:50:45.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/34' 2023-10-10T18:50:45.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/38' 2023-10-10T18:50:45.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/11' 2023-10-10T18:50:45.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/16' 2023-10-10T18:50:45.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/32' 2023-10-10T18:50:45.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/30' 2023-10-10T18:50:45.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/14' 2023-10-10T18:50:45.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/18' 2023-10-10T18:50:45.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/13' 2023-10-10T18:50:45.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/33' 2023-10-10T18:50:45.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/17' 2023-10-10T18:50:45.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/35' 2023-10-10T18:50:45.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './154/37' 2023-10-10T18:50:45.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './154' 2023-10-10T18:50:45.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/19' 2023-10-10T18:50:45.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/15' 2023-10-10T18:50:45.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/31' 2023-10-10T18:50:45.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/36' 2023-10-10T18:50:45.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/20' 2023-10-10T18:50:45.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/12' 2023-10-10T18:50:45.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/39' 2023-10-10T18:50:45.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/10' 2023-10-10T18:50:45.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/34' 2023-10-10T18:50:45.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/38' 2023-10-10T18:50:45.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/11' 2023-10-10T18:50:45.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/16' 2023-10-10T18:50:45.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/32' 2023-10-10T18:50:45.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/30' 2023-10-10T18:50:45.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/14' 2023-10-10T18:50:45.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/18' 2023-10-10T18:50:45.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/13' 2023-10-10T18:50:45.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/33' 2023-10-10T18:50:45.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/17' 2023-10-10T18:50:45.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/35' 2023-10-10T18:50:45.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './155/37' 2023-10-10T18:50:45.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './155' 2023-10-10T18:50:45.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/19' 2023-10-10T18:50:45.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/15' 2023-10-10T18:50:45.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/31' 2023-10-10T18:50:45.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/36' 2023-10-10T18:50:45.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/20' 2023-10-10T18:50:45.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/12' 2023-10-10T18:50:45.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/39' 2023-10-10T18:50:45.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/10' 2023-10-10T18:50:45.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/34' 2023-10-10T18:50:45.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/38' 2023-10-10T18:50:45.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/11' 2023-10-10T18:50:45.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/16' 2023-10-10T18:50:45.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/32' 2023-10-10T18:50:45.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/30' 2023-10-10T18:50:45.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/14' 2023-10-10T18:50:45.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/18' 2023-10-10T18:50:45.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/13' 2023-10-10T18:50:45.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/33' 2023-10-10T18:50:45.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/17' 2023-10-10T18:50:45.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/35' 2023-10-10T18:50:45.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './156/37' 2023-10-10T18:50:45.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './156' 2023-10-10T18:50:45.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/19' 2023-10-10T18:50:45.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/15' 2023-10-10T18:50:45.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/31' 2023-10-10T18:50:45.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/36' 2023-10-10T18:50:45.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/20' 2023-10-10T18:50:45.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/12' 2023-10-10T18:50:45.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/39' 2023-10-10T18:50:45.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/10' 2023-10-10T18:50:45.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/34' 2023-10-10T18:50:45.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/38' 2023-10-10T18:50:45.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/11' 2023-10-10T18:50:45.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/16' 2023-10-10T18:50:45.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/32' 2023-10-10T18:50:45.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/30' 2023-10-10T18:50:45.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/14' 2023-10-10T18:50:45.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/18' 2023-10-10T18:50:45.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/13' 2023-10-10T18:50:45.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/33' 2023-10-10T18:50:47.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/17' 2023-10-10T18:50:47.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/35' 2023-10-10T18:50:47.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './157/37' 2023-10-10T18:50:47.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './157' 2023-10-10T18:50:47.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/19' 2023-10-10T18:50:47.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/15' 2023-10-10T18:50:47.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/31' 2023-10-10T18:50:47.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/36' 2023-10-10T18:50:47.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/20' 2023-10-10T18:50:47.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/12' 2023-10-10T18:50:47.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/39' 2023-10-10T18:50:47.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/10' 2023-10-10T18:50:47.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/34' 2023-10-10T18:50:47.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/38' 2023-10-10T18:50:47.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/11' 2023-10-10T18:50:47.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/16' 2023-10-10T18:50:47.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/32' 2023-10-10T18:50:47.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/30' 2023-10-10T18:50:47.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/14' 2023-10-10T18:50:47.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/18' 2023-10-10T18:50:47.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/13' 2023-10-10T18:50:47.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/33' 2023-10-10T18:50:47.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/17' 2023-10-10T18:50:47.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/35' 2023-10-10T18:50:47.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './158/37' 2023-10-10T18:50:47.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './158' 2023-10-10T18:50:47.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/19' 2023-10-10T18:50:47.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/15' 2023-10-10T18:50:47.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/31' 2023-10-10T18:50:47.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/36' 2023-10-10T18:50:47.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/20' 2023-10-10T18:50:47.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/12' 2023-10-10T18:50:47.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/39' 2023-10-10T18:50:47.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/10' 2023-10-10T18:50:47.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/34' 2023-10-10T18:50:47.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/38' 2023-10-10T18:50:47.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/11' 2023-10-10T18:50:47.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/16' 2023-10-10T18:50:47.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/32' 2023-10-10T18:50:47.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/30' 2023-10-10T18:50:47.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/14' 2023-10-10T18:50:47.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/18' 2023-10-10T18:50:47.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/13' 2023-10-10T18:50:47.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/33' 2023-10-10T18:50:47.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/17' 2023-10-10T18:50:47.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/35' 2023-10-10T18:50:47.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './159/37' 2023-10-10T18:50:47.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './159' 2023-10-10T18:50:47.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/19' 2023-10-10T18:50:47.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/15' 2023-10-10T18:50:47.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/31' 2023-10-10T18:50:47.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/36' 2023-10-10T18:50:47.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/20' 2023-10-10T18:50:47.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/12' 2023-10-10T18:50:47.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/39' 2023-10-10T18:50:47.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/10' 2023-10-10T18:50:47.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/34' 2023-10-10T18:50:47.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/38' 2023-10-10T18:50:47.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/11' 2023-10-10T18:50:47.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/16' 2023-10-10T18:50:47.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/32' 2023-10-10T18:50:47.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/30' 2023-10-10T18:50:47.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/14' 2023-10-10T18:50:47.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/18' 2023-10-10T18:50:47.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/13' 2023-10-10T18:50:47.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/33' 2023-10-10T18:50:47.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/17' 2023-10-10T18:50:47.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/35' 2023-10-10T18:50:47.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './160/37' 2023-10-10T18:50:47.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './160' 2023-10-10T18:50:47.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/19' 2023-10-10T18:50:47.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/15' 2023-10-10T18:50:47.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/31' 2023-10-10T18:50:47.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/36' 2023-10-10T18:50:47.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/20' 2023-10-10T18:50:47.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/12' 2023-10-10T18:50:47.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/39' 2023-10-10T18:50:47.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/10' 2023-10-10T18:50:47.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/34' 2023-10-10T18:50:47.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/38' 2023-10-10T18:50:47.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/11' 2023-10-10T18:50:47.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/16' 2023-10-10T18:50:47.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/32' 2023-10-10T18:50:47.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/30' 2023-10-10T18:50:47.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/14' 2023-10-10T18:50:47.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/18' 2023-10-10T18:50:47.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/13' 2023-10-10T18:50:47.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/33' 2023-10-10T18:50:47.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/17' 2023-10-10T18:50:47.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/35' 2023-10-10T18:50:47.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './161/37' 2023-10-10T18:50:47.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './161' 2023-10-10T18:50:47.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/19' 2023-10-10T18:50:47.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/15' 2023-10-10T18:50:47.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/31' 2023-10-10T18:50:47.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/36' 2023-10-10T18:50:47.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/20' 2023-10-10T18:50:47.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/12' 2023-10-10T18:50:47.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/39' 2023-10-10T18:50:47.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/10' 2023-10-10T18:50:47.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/34' 2023-10-10T18:50:47.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/38' 2023-10-10T18:50:47.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/11' 2023-10-10T18:50:47.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/16' 2023-10-10T18:50:47.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/32' 2023-10-10T18:50:47.683 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/30' 2023-10-10T18:50:47.683 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/14' 2023-10-10T18:50:47.683 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/18' 2023-10-10T18:50:47.683 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/13' 2023-10-10T18:50:47.683 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/33' 2023-10-10T18:50:47.684 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/17' 2023-10-10T18:50:47.684 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/35' 2023-10-10T18:50:47.684 INFO:tasks.workunit.client.0.smithi175.stdout:removed './162/37' 2023-10-10T18:50:47.684 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './162' 2023-10-10T18:50:47.684 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/19' 2023-10-10T18:50:47.684 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/15' 2023-10-10T18:50:47.685 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/31' 2023-10-10T18:50:47.685 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/36' 2023-10-10T18:50:47.685 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/20' 2023-10-10T18:50:47.685 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/12' 2023-10-10T18:50:47.685 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/39' 2023-10-10T18:50:47.686 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/10' 2023-10-10T18:50:47.686 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/34' 2023-10-10T18:50:47.686 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/38' 2023-10-10T18:50:47.686 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/11' 2023-10-10T18:50:47.686 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/16' 2023-10-10T18:50:47.687 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/32' 2023-10-10T18:50:47.687 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/30' 2023-10-10T18:50:47.687 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/14' 2023-10-10T18:50:47.687 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/18' 2023-10-10T18:50:47.687 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/13' 2023-10-10T18:50:47.688 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/33' 2023-10-10T18:50:47.688 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/17' 2023-10-10T18:50:47.688 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/35' 2023-10-10T18:50:47.688 INFO:tasks.workunit.client.0.smithi175.stdout:removed './163/37' 2023-10-10T18:50:47.688 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './163' 2023-10-10T18:50:47.689 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/19' 2023-10-10T18:50:47.689 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/15' 2023-10-10T18:50:47.689 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/31' 2023-10-10T18:50:47.689 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/36' 2023-10-10T18:50:47.689 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/20' 2023-10-10T18:50:47.689 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/12' 2023-10-10T18:50:47.690 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/39' 2023-10-10T18:50:47.690 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/10' 2023-10-10T18:50:47.690 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/34' 2023-10-10T18:50:47.690 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/38' 2023-10-10T18:50:47.690 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/11' 2023-10-10T18:50:47.691 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/16' 2023-10-10T18:50:47.691 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/32' 2023-10-10T18:50:47.691 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/30' 2023-10-10T18:50:47.691 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/14' 2023-10-10T18:50:47.691 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/18' 2023-10-10T18:50:47.692 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/13' 2023-10-10T18:50:47.692 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/33' 2023-10-10T18:50:47.692 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/17' 2023-10-10T18:50:47.692 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/35' 2023-10-10T18:50:47.692 INFO:tasks.workunit.client.0.smithi175.stdout:removed './164/37' 2023-10-10T18:50:47.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './164' 2023-10-10T18:50:47.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/19' 2023-10-10T18:50:47.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/15' 2023-10-10T18:50:47.693 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/31' 2023-10-10T18:50:47.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/36' 2023-10-10T18:50:47.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/20' 2023-10-10T18:50:47.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/12' 2023-10-10T18:50:47.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/39' 2023-10-10T18:50:47.694 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/10' 2023-10-10T18:50:47.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/34' 2023-10-10T18:50:47.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/38' 2023-10-10T18:50:47.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/11' 2023-10-10T18:50:47.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/16' 2023-10-10T18:50:47.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/32' 2023-10-10T18:50:47.695 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/30' 2023-10-10T18:50:47.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/14' 2023-10-10T18:50:47.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/18' 2023-10-10T18:50:47.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/13' 2023-10-10T18:50:47.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/33' 2023-10-10T18:50:47.696 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/17' 2023-10-10T18:50:47.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/35' 2023-10-10T18:50:47.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './165/37' 2023-10-10T18:50:47.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './165' 2023-10-10T18:50:47.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/19' 2023-10-10T18:50:47.697 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/15' 2023-10-10T18:50:47.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/31' 2023-10-10T18:50:47.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/36' 2023-10-10T18:50:47.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/20' 2023-10-10T18:50:47.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/12' 2023-10-10T18:50:47.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/39' 2023-10-10T18:50:47.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/10' 2023-10-10T18:50:47.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/34' 2023-10-10T18:50:47.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/38' 2023-10-10T18:50:47.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/11' 2023-10-10T18:50:47.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/16' 2023-10-10T18:50:47.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/32' 2023-10-10T18:50:47.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/30' 2023-10-10T18:50:47.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/14' 2023-10-10T18:50:47.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/18' 2023-10-10T18:50:47.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/13' 2023-10-10T18:50:47.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/33' 2023-10-10T18:50:47.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/17' 2023-10-10T18:50:47.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/35' 2023-10-10T18:50:47.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './166/37' 2023-10-10T18:50:47.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './166' 2023-10-10T18:50:47.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/19' 2023-10-10T18:50:47.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/15' 2023-10-10T18:50:47.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/31' 2023-10-10T18:50:47.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/36' 2023-10-10T18:50:47.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/20' 2023-10-10T18:50:47.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/12' 2023-10-10T18:50:47.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/39' 2023-10-10T18:50:47.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/10' 2023-10-10T18:50:47.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/34' 2023-10-10T18:50:47.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/38' 2023-10-10T18:50:49.041 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:50:49.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:50:59.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/11' 2023-10-10T18:50:59.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/16' 2023-10-10T18:50:59.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/32' 2023-10-10T18:50:59.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/30' 2023-10-10T18:50:59.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/14' 2023-10-10T18:50:59.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/18' 2023-10-10T18:50:59.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/13' 2023-10-10T18:50:59.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/33' 2023-10-10T18:50:59.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/17' 2023-10-10T18:50:59.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/35' 2023-10-10T18:50:59.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed './167/37' 2023-10-10T18:50:59.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './167' 2023-10-10T18:50:59.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/19' 2023-10-10T18:50:59.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/15' 2023-10-10T18:50:59.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/31' 2023-10-10T18:50:59.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/36' 2023-10-10T18:50:59.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/20' 2023-10-10T18:50:59.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/12' 2023-10-10T18:50:59.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/39' 2023-10-10T18:50:59.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/10' 2023-10-10T18:50:59.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/34' 2023-10-10T18:50:59.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/38' 2023-10-10T18:50:59.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/11' 2023-10-10T18:50:59.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/16' 2023-10-10T18:50:59.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/32' 2023-10-10T18:50:59.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/30' 2023-10-10T18:50:59.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/14' 2023-10-10T18:50:59.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/18' 2023-10-10T18:50:59.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/13' 2023-10-10T18:50:59.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/33' 2023-10-10T18:50:59.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/17' 2023-10-10T18:50:59.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/35' 2023-10-10T18:50:59.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './168/37' 2023-10-10T18:50:59.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './168' 2023-10-10T18:50:59.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/19' 2023-10-10T18:50:59.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/15' 2023-10-10T18:50:59.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/31' 2023-10-10T18:50:59.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/36' 2023-10-10T18:50:59.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/20' 2023-10-10T18:50:59.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/12' 2023-10-10T18:50:59.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/39' 2023-10-10T18:50:59.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/10' 2023-10-10T18:50:59.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/34' 2023-10-10T18:50:59.639 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/38' 2023-10-10T18:50:59.639 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/11' 2023-10-10T18:50:59.639 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/16' 2023-10-10T18:50:59.639 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/32' 2023-10-10T18:50:59.639 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/30' 2023-10-10T18:50:59.640 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/14' 2023-10-10T18:50:59.640 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/18' 2023-10-10T18:50:59.640 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/13' 2023-10-10T18:50:59.640 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/33' 2023-10-10T18:50:59.640 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/17' 2023-10-10T18:50:59.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/35' 2023-10-10T18:50:59.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed './169/37' 2023-10-10T18:50:59.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './169' 2023-10-10T18:50:59.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/19' 2023-10-10T18:50:59.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/15' 2023-10-10T18:50:59.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/31' 2023-10-10T18:50:59.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/36' 2023-10-10T18:50:59.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/20' 2023-10-10T18:50:59.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/12' 2023-10-10T18:50:59.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/39' 2023-10-10T18:50:59.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/10' 2023-10-10T18:50:59.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/34' 2023-10-10T18:50:59.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/38' 2023-10-10T18:50:59.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/11' 2023-10-10T18:50:59.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/16' 2023-10-10T18:50:59.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/32' 2023-10-10T18:50:59.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/30' 2023-10-10T18:50:59.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/14' 2023-10-10T18:50:59.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/18' 2023-10-10T18:50:59.645 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/13' 2023-10-10T18:50:59.645 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/33' 2023-10-10T18:50:59.645 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/17' 2023-10-10T18:50:59.645 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/35' 2023-10-10T18:50:59.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './170/37' 2023-10-10T18:50:59.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './170' 2023-10-10T18:50:59.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/19' 2023-10-10T18:50:59.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/15' 2023-10-10T18:50:59.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/31' 2023-10-10T18:50:59.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/36' 2023-10-10T18:50:59.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/20' 2023-10-10T18:50:59.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/12' 2023-10-10T18:50:59.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/39' 2023-10-10T18:50:59.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/10' 2023-10-10T18:50:59.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/34' 2023-10-10T18:50:59.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/38' 2023-10-10T18:50:59.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/11' 2023-10-10T18:50:59.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/16' 2023-10-10T18:50:59.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/32' 2023-10-10T18:50:59.649 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/30' 2023-10-10T18:50:59.649 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/14' 2023-10-10T18:50:59.649 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/18' 2023-10-10T18:50:59.650 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/13' 2023-10-10T18:50:59.650 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/33' 2023-10-10T18:50:59.650 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/17' 2023-10-10T18:50:59.651 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/35' 2023-10-10T18:50:59.651 INFO:tasks.workunit.client.0.smithi175.stdout:removed './171/37' 2023-10-10T18:50:59.651 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './171' 2023-10-10T18:50:59.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/19' 2023-10-10T18:50:59.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/15' 2023-10-10T18:50:59.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/31' 2023-10-10T18:50:59.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/36' 2023-10-10T18:50:59.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/20' 2023-10-10T18:50:59.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/12' 2023-10-10T18:50:59.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/39' 2023-10-10T18:50:59.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/10' 2023-10-10T18:50:59.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/34' 2023-10-10T18:50:59.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/38' 2023-10-10T18:50:59.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/11' 2023-10-10T18:50:59.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/16' 2023-10-10T18:50:59.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/32' 2023-10-10T18:50:59.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/30' 2023-10-10T18:50:59.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/14' 2023-10-10T18:50:59.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/18' 2023-10-10T18:50:59.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/13' 2023-10-10T18:50:59.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/33' 2023-10-10T18:50:59.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/17' 2023-10-10T18:50:59.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/35' 2023-10-10T18:50:59.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed './172/37' 2023-10-10T18:50:59.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './172' 2023-10-10T18:50:59.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/19' 2023-10-10T18:50:59.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/15' 2023-10-10T18:50:59.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/31' 2023-10-10T18:50:59.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/36' 2023-10-10T18:50:59.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/20' 2023-10-10T18:50:59.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/12' 2023-10-10T18:50:59.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/39' 2023-10-10T18:50:59.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/10' 2023-10-10T18:50:59.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/34' 2023-10-10T18:50:59.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/38' 2023-10-10T18:50:59.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/11' 2023-10-10T18:50:59.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/16' 2023-10-10T18:50:59.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/32' 2023-10-10T18:50:59.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/30' 2023-10-10T18:50:59.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/14' 2023-10-10T18:50:59.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/18' 2023-10-10T18:50:59.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/13' 2023-10-10T18:50:59.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/33' 2023-10-10T18:50:59.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/17' 2023-10-10T18:50:59.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/35' 2023-10-10T18:50:59.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './173/37' 2023-10-10T18:50:59.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './173' 2023-10-10T18:50:59.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/19' 2023-10-10T18:50:59.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/15' 2023-10-10T18:50:59.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/31' 2023-10-10T18:50:59.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/36' 2023-10-10T18:50:59.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/20' 2023-10-10T18:50:59.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/12' 2023-10-10T18:50:59.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/39' 2023-10-10T18:50:59.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/10' 2023-10-10T18:50:59.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/34' 2023-10-10T18:50:59.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/38' 2023-10-10T18:50:59.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/11' 2023-10-10T18:50:59.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/16' 2023-10-10T18:50:59.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/32' 2023-10-10T18:50:59.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/30' 2023-10-10T18:50:59.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/14' 2023-10-10T18:50:59.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/18' 2023-10-10T18:50:59.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/13' 2023-10-10T18:50:59.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/33' 2023-10-10T18:50:59.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/17' 2023-10-10T18:50:59.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/35' 2023-10-10T18:50:59.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './174/37' 2023-10-10T18:50:59.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './174' 2023-10-10T18:50:59.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/19' 2023-10-10T18:50:59.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/15' 2023-10-10T18:50:59.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/31' 2023-10-10T18:50:59.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/36' 2023-10-10T18:50:59.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/20' 2023-10-10T18:50:59.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/12' 2023-10-10T18:50:59.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/39' 2023-10-10T18:50:59.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/10' 2023-10-10T18:50:59.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/34' 2023-10-10T18:50:59.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/38' 2023-10-10T18:50:59.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/11' 2023-10-10T18:50:59.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/16' 2023-10-10T18:50:59.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/32' 2023-10-10T18:50:59.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/30' 2023-10-10T18:50:59.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/14' 2023-10-10T18:50:59.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/18' 2023-10-10T18:50:59.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/13' 2023-10-10T18:50:59.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/33' 2023-10-10T18:50:59.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/17' 2023-10-10T18:50:59.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/35' 2023-10-10T18:50:59.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './175/37' 2023-10-10T18:50:59.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './175' 2023-10-10T18:50:59.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/19' 2023-10-10T18:50:59.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/15' 2023-10-10T18:50:59.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/31' 2023-10-10T18:50:59.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/36' 2023-10-10T18:50:59.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/20' 2023-10-10T18:50:59.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/12' 2023-10-10T18:50:59.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/39' 2023-10-10T18:50:59.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/10' 2023-10-10T18:50:59.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/34' 2023-10-10T18:50:59.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/38' 2023-10-10T18:50:59.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/11' 2023-10-10T18:50:59.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/16' 2023-10-10T18:50:59.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/32' 2023-10-10T18:50:59.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/30' 2023-10-10T18:50:59.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/14' 2023-10-10T18:50:59.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/18' 2023-10-10T18:50:59.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/13' 2023-10-10T18:50:59.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/33' 2023-10-10T18:50:59.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/17' 2023-10-10T18:50:59.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/35' 2023-10-10T18:50:59.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './176/37' 2023-10-10T18:50:59.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './176' 2023-10-10T18:50:59.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/19' 2023-10-10T18:50:59.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/15' 2023-10-10T18:51:05.080 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/31' 2023-10-10T18:51:05.080 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/36' 2023-10-10T18:51:05.080 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/20' 2023-10-10T18:51:05.081 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/12' 2023-10-10T18:51:05.081 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/39' 2023-10-10T18:51:05.081 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/10' 2023-10-10T18:51:05.081 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/34' 2023-10-10T18:51:05.082 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/38' 2023-10-10T18:51:05.082 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/11' 2023-10-10T18:51:05.082 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/16' 2023-10-10T18:51:05.083 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/32' 2023-10-10T18:51:05.083 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/30' 2023-10-10T18:51:05.083 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/14' 2023-10-10T18:51:05.083 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/18' 2023-10-10T18:51:05.084 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/13' 2023-10-10T18:51:05.084 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/33' 2023-10-10T18:51:05.084 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/17' 2023-10-10T18:51:05.084 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/35' 2023-10-10T18:51:05.084 INFO:tasks.workunit.client.0.smithi175.stdout:removed './177/37' 2023-10-10T18:51:05.085 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './177' 2023-10-10T18:51:05.085 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/19' 2023-10-10T18:51:05.085 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/15' 2023-10-10T18:51:05.085 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/31' 2023-10-10T18:51:05.085 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/36' 2023-10-10T18:51:05.086 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/20' 2023-10-10T18:51:05.086 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/12' 2023-10-10T18:51:05.086 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/39' 2023-10-10T18:51:05.086 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/10' 2023-10-10T18:51:05.087 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/34' 2023-10-10T18:51:05.087 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/38' 2023-10-10T18:51:05.087 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/11' 2023-10-10T18:51:05.087 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/16' 2023-10-10T18:51:05.087 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/32' 2023-10-10T18:51:05.087 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/30' 2023-10-10T18:51:05.088 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/14' 2023-10-10T18:51:05.088 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/18' 2023-10-10T18:51:05.088 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/13' 2023-10-10T18:51:05.088 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/33' 2023-10-10T18:51:05.088 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/17' 2023-10-10T18:51:05.089 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/35' 2023-10-10T18:51:05.089 INFO:tasks.workunit.client.0.smithi175.stdout:removed './178/37' 2023-10-10T18:51:05.089 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './178' 2023-10-10T18:51:05.089 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/19' 2023-10-10T18:51:05.089 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/15' 2023-10-10T18:51:05.090 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/31' 2023-10-10T18:51:05.090 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/36' 2023-10-10T18:51:05.090 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/20' 2023-10-10T18:51:05.090 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/12' 2023-10-10T18:51:05.090 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/39' 2023-10-10T18:51:05.091 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/10' 2023-10-10T18:51:05.091 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/34' 2023-10-10T18:51:05.091 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/38' 2023-10-10T18:51:05.091 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/11' 2023-10-10T18:51:05.091 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/16' 2023-10-10T18:51:05.092 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/32' 2023-10-10T18:51:05.092 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/30' 2023-10-10T18:51:05.092 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/14' 2023-10-10T18:51:05.092 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/18' 2023-10-10T18:51:05.092 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/13' 2023-10-10T18:51:05.093 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/33' 2023-10-10T18:51:05.093 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/17' 2023-10-10T18:51:05.093 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/35' 2023-10-10T18:51:05.093 INFO:tasks.workunit.client.0.smithi175.stdout:removed './179/37' 2023-10-10T18:51:05.093 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './179' 2023-10-10T18:51:05.094 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/19' 2023-10-10T18:51:05.094 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/15' 2023-10-10T18:51:05.094 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/31' 2023-10-10T18:51:05.095 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/36' 2023-10-10T18:51:05.095 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/20' 2023-10-10T18:51:05.095 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/12' 2023-10-10T18:51:05.095 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/39' 2023-10-10T18:51:05.095 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/10' 2023-10-10T18:51:05.096 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/34' 2023-10-10T18:51:05.096 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/38' 2023-10-10T18:51:05.096 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/11' 2023-10-10T18:51:05.096 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/16' 2023-10-10T18:51:05.096 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/32' 2023-10-10T18:51:05.097 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/30' 2023-10-10T18:51:05.097 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/14' 2023-10-10T18:51:05.097 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/18' 2023-10-10T18:51:05.097 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/13' 2023-10-10T18:51:05.097 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/33' 2023-10-10T18:51:05.098 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/17' 2023-10-10T18:51:05.098 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/35' 2023-10-10T18:51:05.098 INFO:tasks.workunit.client.0.smithi175.stdout:removed './180/37' 2023-10-10T18:51:05.098 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './180' 2023-10-10T18:51:05.099 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/19' 2023-10-10T18:51:05.099 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/15' 2023-10-10T18:51:05.099 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/31' 2023-10-10T18:51:05.099 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/36' 2023-10-10T18:51:05.100 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/20' 2023-10-10T18:51:05.100 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/12' 2023-10-10T18:51:05.100 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/39' 2023-10-10T18:51:05.100 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/10' 2023-10-10T18:51:05.100 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/34' 2023-10-10T18:51:05.101 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/38' 2023-10-10T18:51:05.101 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/11' 2023-10-10T18:51:05.101 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/16' 2023-10-10T18:51:05.101 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/32' 2023-10-10T18:51:05.102 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/30' 2023-10-10T18:51:05.102 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/14' 2023-10-10T18:51:05.102 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/18' 2023-10-10T18:51:05.102 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/13' 2023-10-10T18:51:05.103 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/33' 2023-10-10T18:51:05.103 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/17' 2023-10-10T18:51:05.103 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/35' 2023-10-10T18:51:05.103 INFO:tasks.workunit.client.0.smithi175.stdout:removed './181/37' 2023-10-10T18:51:05.103 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './181' 2023-10-10T18:51:05.104 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/19' 2023-10-10T18:51:05.104 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/15' 2023-10-10T18:51:05.104 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/31' 2023-10-10T18:51:05.104 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/36' 2023-10-10T18:51:05.105 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/20' 2023-10-10T18:51:05.105 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/12' 2023-10-10T18:51:05.105 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/39' 2023-10-10T18:51:05.105 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/10' 2023-10-10T18:51:05.105 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/34' 2023-10-10T18:51:05.106 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/38' 2023-10-10T18:51:05.106 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/11' 2023-10-10T18:51:05.106 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/16' 2023-10-10T18:51:05.106 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/32' 2023-10-10T18:51:05.107 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/30' 2023-10-10T18:51:05.107 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/14' 2023-10-10T18:51:05.107 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/18' 2023-10-10T18:51:05.107 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/13' 2023-10-10T18:51:05.107 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/33' 2023-10-10T18:51:05.107 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/17' 2023-10-10T18:51:05.108 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/35' 2023-10-10T18:51:05.108 INFO:tasks.workunit.client.0.smithi175.stdout:removed './182/37' 2023-10-10T18:51:05.108 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './182' 2023-10-10T18:51:05.108 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/19' 2023-10-10T18:51:05.109 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/15' 2023-10-10T18:51:05.109 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/31' 2023-10-10T18:51:05.109 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/36' 2023-10-10T18:51:05.109 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/20' 2023-10-10T18:51:05.109 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/12' 2023-10-10T18:51:05.109 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/39' 2023-10-10T18:51:05.110 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/10' 2023-10-10T18:51:05.110 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/34' 2023-10-10T18:51:05.110 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/38' 2023-10-10T18:51:05.110 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/11' 2023-10-10T18:51:05.110 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/16' 2023-10-10T18:51:05.111 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/32' 2023-10-10T18:51:05.111 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/30' 2023-10-10T18:51:05.111 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/14' 2023-10-10T18:51:05.111 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/18' 2023-10-10T18:51:05.111 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/13' 2023-10-10T18:51:05.112 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/33' 2023-10-10T18:51:05.112 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/17' 2023-10-10T18:51:05.112 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/35' 2023-10-10T18:51:05.112 INFO:tasks.workunit.client.0.smithi175.stdout:removed './183/37' 2023-10-10T18:51:05.112 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './183' 2023-10-10T18:51:05.113 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/19' 2023-10-10T18:51:05.113 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/15' 2023-10-10T18:51:05.113 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/31' 2023-10-10T18:51:05.114 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/36' 2023-10-10T18:51:05.114 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/20' 2023-10-10T18:51:05.114 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/12' 2023-10-10T18:51:05.115 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/39' 2023-10-10T18:51:05.115 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/10' 2023-10-10T18:51:05.115 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/34' 2023-10-10T18:51:05.115 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/38' 2023-10-10T18:51:05.116 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/11' 2023-10-10T18:51:05.116 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/16' 2023-10-10T18:51:05.116 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/32' 2023-10-10T18:51:05.116 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/30' 2023-10-10T18:51:05.117 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/14' 2023-10-10T18:51:05.117 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/18' 2023-10-10T18:51:05.117 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/13' 2023-10-10T18:51:05.117 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/33' 2023-10-10T18:51:05.117 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/17' 2023-10-10T18:51:05.118 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/35' 2023-10-10T18:51:05.118 INFO:tasks.workunit.client.0.smithi175.stdout:removed './184/37' 2023-10-10T18:51:05.118 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './184' 2023-10-10T18:51:05.118 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/19' 2023-10-10T18:51:05.118 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/15' 2023-10-10T18:51:05.119 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/31' 2023-10-10T18:51:05.119 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/36' 2023-10-10T18:51:05.119 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/20' 2023-10-10T18:51:05.119 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/12' 2023-10-10T18:51:05.119 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/39' 2023-10-10T18:51:05.120 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/10' 2023-10-10T18:51:05.120 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/34' 2023-10-10T18:51:05.120 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/38' 2023-10-10T18:51:05.120 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/11' 2023-10-10T18:51:05.120 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/16' 2023-10-10T18:51:05.120 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/32' 2023-10-10T18:51:05.121 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/30' 2023-10-10T18:51:05.121 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/14' 2023-10-10T18:51:05.121 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/18' 2023-10-10T18:51:05.121 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/13' 2023-10-10T18:51:05.121 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/33' 2023-10-10T18:51:05.122 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/17' 2023-10-10T18:51:05.122 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/35' 2023-10-10T18:51:05.122 INFO:tasks.workunit.client.0.smithi175.stdout:removed './185/37' 2023-10-10T18:51:05.122 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './185' 2023-10-10T18:51:05.123 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/19' 2023-10-10T18:51:05.123 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/15' 2023-10-10T18:51:05.123 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/31' 2023-10-10T18:51:05.123 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/36' 2023-10-10T18:51:05.123 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/20' 2023-10-10T18:51:05.124 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/12' 2023-10-10T18:51:05.124 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/39' 2023-10-10T18:51:05.124 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/10' 2023-10-10T18:51:05.124 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/34' 2023-10-10T18:51:05.125 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/38' 2023-10-10T18:51:05.125 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/11' 2023-10-10T18:51:05.125 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/16' 2023-10-10T18:51:05.125 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/32' 2023-10-10T18:51:05.125 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/30' 2023-10-10T18:51:05.126 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/14' 2023-10-10T18:51:05.126 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/18' 2023-10-10T18:51:15.181 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/13' 2023-10-10T18:51:15.182 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/33' 2023-10-10T18:51:15.182 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/17' 2023-10-10T18:51:15.182 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/35' 2023-10-10T18:51:15.182 INFO:tasks.workunit.client.0.smithi175.stdout:removed './186/37' 2023-10-10T18:51:15.182 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './186' 2023-10-10T18:51:15.183 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/19' 2023-10-10T18:51:15.183 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/15' 2023-10-10T18:51:15.183 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/31' 2023-10-10T18:51:15.183 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/36' 2023-10-10T18:51:15.183 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/20' 2023-10-10T18:51:15.184 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/12' 2023-10-10T18:51:15.184 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/39' 2023-10-10T18:51:15.184 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/10' 2023-10-10T18:51:15.184 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/34' 2023-10-10T18:51:15.184 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/38' 2023-10-10T18:51:15.185 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/11' 2023-10-10T18:51:15.185 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/16' 2023-10-10T18:51:15.185 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/32' 2023-10-10T18:51:15.185 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/30' 2023-10-10T18:51:15.185 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/14' 2023-10-10T18:51:15.186 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/18' 2023-10-10T18:51:15.186 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/13' 2023-10-10T18:51:15.186 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/33' 2023-10-10T18:51:15.187 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/17' 2023-10-10T18:51:15.187 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/35' 2023-10-10T18:51:15.187 INFO:tasks.workunit.client.0.smithi175.stdout:removed './187/37' 2023-10-10T18:51:15.187 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './187' 2023-10-10T18:51:15.187 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/19' 2023-10-10T18:51:15.188 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/15' 2023-10-10T18:51:15.188 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/31' 2023-10-10T18:51:15.188 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/36' 2023-10-10T18:51:15.188 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/20' 2023-10-10T18:51:15.188 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/12' 2023-10-10T18:51:15.188 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/39' 2023-10-10T18:51:15.189 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/10' 2023-10-10T18:51:15.189 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/34' 2023-10-10T18:51:15.189 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/38' 2023-10-10T18:51:15.189 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/11' 2023-10-10T18:51:15.189 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/16' 2023-10-10T18:51:15.190 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/32' 2023-10-10T18:51:15.190 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/30' 2023-10-10T18:51:15.190 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/14' 2023-10-10T18:51:15.190 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/18' 2023-10-10T18:51:15.191 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/13' 2023-10-10T18:51:15.191 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/33' 2023-10-10T18:51:15.191 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/17' 2023-10-10T18:51:15.191 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/35' 2023-10-10T18:51:15.191 INFO:tasks.workunit.client.0.smithi175.stdout:removed './188/37' 2023-10-10T18:51:15.191 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './188' 2023-10-10T18:51:15.192 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/19' 2023-10-10T18:51:15.192 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/15' 2023-10-10T18:51:15.192 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/31' 2023-10-10T18:51:15.192 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/36' 2023-10-10T18:51:15.192 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/20' 2023-10-10T18:51:15.193 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/12' 2023-10-10T18:51:15.193 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/39' 2023-10-10T18:51:15.193 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/10' 2023-10-10T18:51:15.193 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/34' 2023-10-10T18:51:15.193 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/38' 2023-10-10T18:51:15.194 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/11' 2023-10-10T18:51:15.194 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/16' 2023-10-10T18:51:15.194 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/32' 2023-10-10T18:51:15.194 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/30' 2023-10-10T18:51:15.194 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/14' 2023-10-10T18:51:15.195 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/18' 2023-10-10T18:51:15.195 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/13' 2023-10-10T18:51:15.195 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/33' 2023-10-10T18:51:15.195 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/17' 2023-10-10T18:51:15.195 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/35' 2023-10-10T18:51:15.196 INFO:tasks.workunit.client.0.smithi175.stdout:removed './189/37' 2023-10-10T18:51:15.196 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './189' 2023-10-10T18:51:15.196 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/19' 2023-10-10T18:51:15.196 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/15' 2023-10-10T18:51:15.196 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/31' 2023-10-10T18:51:15.197 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/36' 2023-10-10T18:51:15.197 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/20' 2023-10-10T18:51:15.197 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/12' 2023-10-10T18:51:15.197 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/39' 2023-10-10T18:51:15.197 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/10' 2023-10-10T18:51:15.198 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/34' 2023-10-10T18:51:15.198 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/38' 2023-10-10T18:51:15.198 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/11' 2023-10-10T18:51:15.198 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/16' 2023-10-10T18:51:15.198 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/32' 2023-10-10T18:51:15.199 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/30' 2023-10-10T18:51:15.199 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/14' 2023-10-10T18:51:15.199 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/18' 2023-10-10T18:51:15.199 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/13' 2023-10-10T18:51:15.199 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/33' 2023-10-10T18:51:15.200 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/17' 2023-10-10T18:51:15.200 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/35' 2023-10-10T18:51:15.200 INFO:tasks.workunit.client.0.smithi175.stdout:removed './190/37' 2023-10-10T18:51:15.200 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './190' 2023-10-10T18:51:15.200 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/19' 2023-10-10T18:51:15.200 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/15' 2023-10-10T18:51:15.201 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/31' 2023-10-10T18:51:15.201 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/36' 2023-10-10T18:51:15.201 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/20' 2023-10-10T18:51:15.201 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/12' 2023-10-10T18:51:15.201 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/39' 2023-10-10T18:51:15.202 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/10' 2023-10-10T18:51:15.202 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/34' 2023-10-10T18:51:15.202 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/38' 2023-10-10T18:51:15.202 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/11' 2023-10-10T18:51:15.202 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/16' 2023-10-10T18:51:15.203 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/32' 2023-10-10T18:51:15.203 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/30' 2023-10-10T18:51:15.203 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/14' 2023-10-10T18:51:15.203 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/18' 2023-10-10T18:51:15.203 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/13' 2023-10-10T18:51:15.204 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/33' 2023-10-10T18:51:15.204 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/17' 2023-10-10T18:51:15.204 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/35' 2023-10-10T18:51:15.204 INFO:tasks.workunit.client.0.smithi175.stdout:removed './191/37' 2023-10-10T18:51:15.204 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './191' 2023-10-10T18:51:15.205 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/19' 2023-10-10T18:51:15.205 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/15' 2023-10-10T18:51:15.205 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/31' 2023-10-10T18:51:15.205 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/36' 2023-10-10T18:51:15.205 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/20' 2023-10-10T18:51:15.206 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/12' 2023-10-10T18:51:15.206 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/39' 2023-10-10T18:51:15.206 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/10' 2023-10-10T18:51:15.206 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/34' 2023-10-10T18:51:15.206 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/38' 2023-10-10T18:51:15.207 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/11' 2023-10-10T18:51:15.207 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/16' 2023-10-10T18:51:15.207 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/32' 2023-10-10T18:51:15.207 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/30' 2023-10-10T18:51:15.207 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/14' 2023-10-10T18:51:15.208 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/18' 2023-10-10T18:51:15.208 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/13' 2023-10-10T18:51:15.208 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/33' 2023-10-10T18:51:15.208 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/17' 2023-10-10T18:51:15.208 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/35' 2023-10-10T18:51:15.209 INFO:tasks.workunit.client.0.smithi175.stdout:removed './192/37' 2023-10-10T18:51:15.209 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './192' 2023-10-10T18:51:15.209 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/19' 2023-10-10T18:51:15.209 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/15' 2023-10-10T18:51:15.209 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/31' 2023-10-10T18:51:15.210 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/36' 2023-10-10T18:51:15.210 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/20' 2023-10-10T18:51:15.210 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/12' 2023-10-10T18:51:15.210 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/39' 2023-10-10T18:51:15.210 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/10' 2023-10-10T18:51:15.211 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/34' 2023-10-10T18:51:15.211 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/38' 2023-10-10T18:51:15.211 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/11' 2023-10-10T18:51:15.211 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/16' 2023-10-10T18:51:15.211 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/32' 2023-10-10T18:51:15.212 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/30' 2023-10-10T18:51:15.212 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/14' 2023-10-10T18:51:15.212 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/18' 2023-10-10T18:51:15.212 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/13' 2023-10-10T18:51:15.213 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/33' 2023-10-10T18:51:15.213 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/17' 2023-10-10T18:51:15.213 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/35' 2023-10-10T18:51:15.213 INFO:tasks.workunit.client.0.smithi175.stdout:removed './193/37' 2023-10-10T18:51:15.213 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './193' 2023-10-10T18:51:15.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/19' 2023-10-10T18:51:15.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/15' 2023-10-10T18:51:15.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/31' 2023-10-10T18:51:15.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/36' 2023-10-10T18:51:15.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/20' 2023-10-10T18:51:15.214 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/12' 2023-10-10T18:51:15.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/39' 2023-10-10T18:51:15.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/10' 2023-10-10T18:51:15.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/34' 2023-10-10T18:51:15.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/38' 2023-10-10T18:51:15.215 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/11' 2023-10-10T18:51:15.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/16' 2023-10-10T18:51:15.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/32' 2023-10-10T18:51:15.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/30' 2023-10-10T18:51:15.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/14' 2023-10-10T18:51:15.216 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/18' 2023-10-10T18:51:15.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/13' 2023-10-10T18:51:15.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/33' 2023-10-10T18:51:15.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/17' 2023-10-10T18:51:15.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/35' 2023-10-10T18:51:15.217 INFO:tasks.workunit.client.0.smithi175.stdout:removed './194/37' 2023-10-10T18:51:15.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './194' 2023-10-10T18:51:15.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/19' 2023-10-10T18:51:15.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/15' 2023-10-10T18:51:15.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/31' 2023-10-10T18:51:15.218 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/36' 2023-10-10T18:51:15.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/20' 2023-10-10T18:51:15.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/12' 2023-10-10T18:51:15.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/39' 2023-10-10T18:51:15.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/10' 2023-10-10T18:51:15.219 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/34' 2023-10-10T18:51:15.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/38' 2023-10-10T18:51:15.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/11' 2023-10-10T18:51:15.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/16' 2023-10-10T18:51:15.221 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/32' 2023-10-10T18:51:15.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/30' 2023-10-10T18:51:15.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/14' 2023-10-10T18:51:15.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/18' 2023-10-10T18:51:15.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/13' 2023-10-10T18:51:15.222 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/33' 2023-10-10T18:51:15.223 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/17' 2023-10-10T18:51:15.223 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/35' 2023-10-10T18:51:15.223 INFO:tasks.workunit.client.0.smithi175.stdout:removed './195/37' 2023-10-10T18:51:15.223 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './195' 2023-10-10T18:51:15.223 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/19' 2023-10-10T18:51:15.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/15' 2023-10-10T18:51:15.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/31' 2023-10-10T18:51:15.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/36' 2023-10-10T18:51:15.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/20' 2023-10-10T18:51:15.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/12' 2023-10-10T18:51:15.224 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/39' 2023-10-10T18:51:15.225 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/10' 2023-10-10T18:51:16.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/34' 2023-10-10T18:51:16.574 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/38' 2023-10-10T18:51:16.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/11' 2023-10-10T18:51:16.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/16' 2023-10-10T18:51:16.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/32' 2023-10-10T18:51:16.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/30' 2023-10-10T18:51:16.575 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/14' 2023-10-10T18:51:16.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/18' 2023-10-10T18:51:16.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/13' 2023-10-10T18:51:16.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/33' 2023-10-10T18:51:16.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/17' 2023-10-10T18:51:16.576 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/35' 2023-10-10T18:51:16.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './196/37' 2023-10-10T18:51:16.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './196' 2023-10-10T18:51:16.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/19' 2023-10-10T18:51:16.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/15' 2023-10-10T18:51:16.577 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/31' 2023-10-10T18:51:16.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/36' 2023-10-10T18:51:16.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/20' 2023-10-10T18:51:16.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/12' 2023-10-10T18:51:16.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/39' 2023-10-10T18:51:16.578 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/10' 2023-10-10T18:51:16.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/34' 2023-10-10T18:51:16.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/38' 2023-10-10T18:51:16.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/11' 2023-10-10T18:51:16.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/16' 2023-10-10T18:51:16.579 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/32' 2023-10-10T18:51:16.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/30' 2023-10-10T18:51:16.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/14' 2023-10-10T18:51:16.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/18' 2023-10-10T18:51:16.580 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/13' 2023-10-10T18:51:16.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/33' 2023-10-10T18:51:16.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/17' 2023-10-10T18:51:16.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/35' 2023-10-10T18:51:16.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed './197/37' 2023-10-10T18:51:16.581 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './197' 2023-10-10T18:51:16.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/19' 2023-10-10T18:51:16.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/15' 2023-10-10T18:51:16.582 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/31' 2023-10-10T18:51:16.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/36' 2023-10-10T18:51:16.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/20' 2023-10-10T18:51:16.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/12' 2023-10-10T18:51:16.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/39' 2023-10-10T18:51:16.583 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/10' 2023-10-10T18:51:16.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/34' 2023-10-10T18:51:16.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/38' 2023-10-10T18:51:16.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/11' 2023-10-10T18:51:16.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/16' 2023-10-10T18:51:16.584 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/32' 2023-10-10T18:51:16.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/30' 2023-10-10T18:51:16.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/14' 2023-10-10T18:51:16.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/18' 2023-10-10T18:51:16.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/13' 2023-10-10T18:51:16.585 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/33' 2023-10-10T18:51:16.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/17' 2023-10-10T18:51:16.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/35' 2023-10-10T18:51:16.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './198/37' 2023-10-10T18:51:16.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './198' 2023-10-10T18:51:16.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/19' 2023-10-10T18:51:16.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/15' 2023-10-10T18:51:16.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/31' 2023-10-10T18:51:16.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/36' 2023-10-10T18:51:16.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/20' 2023-10-10T18:51:16.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/12' 2023-10-10T18:51:16.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/39' 2023-10-10T18:51:16.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/10' 2023-10-10T18:51:16.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/34' 2023-10-10T18:51:16.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/38' 2023-10-10T18:51:16.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/11' 2023-10-10T18:51:16.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/16' 2023-10-10T18:51:16.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/32' 2023-10-10T18:51:16.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/30' 2023-10-10T18:51:16.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/14' 2023-10-10T18:51:16.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/18' 2023-10-10T18:51:16.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/13' 2023-10-10T18:51:16.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/33' 2023-10-10T18:51:16.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/17' 2023-10-10T18:51:16.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/35' 2023-10-10T18:51:16.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './199/37' 2023-10-10T18:51:16.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './199' 2023-10-10T18:51:16.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/21' 2023-10-10T18:51:16.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/27' 2023-10-10T18:51:16.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/31' 2023-10-10T18:51:16.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/36' 2023-10-10T18:51:16.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/20' 2023-10-10T18:51:16.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/39' 2023-10-10T18:51:16.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/34' 2023-10-10T18:51:16.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/38' 2023-10-10T18:51:16.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/28' 2023-10-10T18:51:16.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/24' 2023-10-10T18:51:16.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/29' 2023-10-10T18:51:16.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/32' 2023-10-10T18:51:16.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/22' 2023-10-10T18:51:16.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/25' 2023-10-10T18:51:16.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/30' 2023-10-10T18:51:16.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/33' 2023-10-10T18:51:16.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/23' 2023-10-10T18:51:16.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/35' 2023-10-10T18:51:16.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/26' 2023-10-10T18:51:16.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './200/37' 2023-10-10T18:51:16.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './200' 2023-10-10T18:51:16.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/21' 2023-10-10T18:51:16.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/27' 2023-10-10T18:51:16.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/31' 2023-10-10T18:51:16.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/36' 2023-10-10T18:51:16.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/20' 2023-10-10T18:51:16.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/39' 2023-10-10T18:51:16.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/34' 2023-10-10T18:51:16.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/38' 2023-10-10T18:51:16.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/28' 2023-10-10T18:51:16.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/24' 2023-10-10T18:51:16.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/29' 2023-10-10T18:51:16.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/32' 2023-10-10T18:51:16.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/22' 2023-10-10T18:51:16.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/25' 2023-10-10T18:51:16.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/30' 2023-10-10T18:51:16.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/33' 2023-10-10T18:51:16.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/23' 2023-10-10T18:51:16.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/35' 2023-10-10T18:51:16.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/26' 2023-10-10T18:51:16.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './201/37' 2023-10-10T18:51:16.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './201' 2023-10-10T18:51:16.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/21' 2023-10-10T18:51:16.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/27' 2023-10-10T18:51:16.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/31' 2023-10-10T18:51:16.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/36' 2023-10-10T18:51:16.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/20' 2023-10-10T18:51:16.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/39' 2023-10-10T18:51:16.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/34' 2023-10-10T18:51:16.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/38' 2023-10-10T18:51:16.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/28' 2023-10-10T18:51:16.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/24' 2023-10-10T18:51:16.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/29' 2023-10-10T18:51:16.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/32' 2023-10-10T18:51:16.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/22' 2023-10-10T18:51:16.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/25' 2023-10-10T18:51:16.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/30' 2023-10-10T18:51:16.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/33' 2023-10-10T18:51:16.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/23' 2023-10-10T18:51:16.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/35' 2023-10-10T18:51:16.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/26' 2023-10-10T18:51:16.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './202/37' 2023-10-10T18:51:16.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './202' 2023-10-10T18:51:16.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/21' 2023-10-10T18:51:16.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/27' 2023-10-10T18:51:16.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/31' 2023-10-10T18:51:16.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/36' 2023-10-10T18:51:16.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/20' 2023-10-10T18:51:16.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/39' 2023-10-10T18:51:16.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/34' 2023-10-10T18:51:16.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/38' 2023-10-10T18:51:16.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/28' 2023-10-10T18:51:16.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/24' 2023-10-10T18:51:16.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/29' 2023-10-10T18:51:16.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/32' 2023-10-10T18:51:16.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/22' 2023-10-10T18:51:16.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/25' 2023-10-10T18:51:16.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/30' 2023-10-10T18:51:16.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/33' 2023-10-10T18:51:16.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/23' 2023-10-10T18:51:16.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/35' 2023-10-10T18:51:16.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/26' 2023-10-10T18:51:16.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './203/37' 2023-10-10T18:51:16.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './203' 2023-10-10T18:51:16.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/21' 2023-10-10T18:51:16.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/27' 2023-10-10T18:51:16.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/31' 2023-10-10T18:51:16.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/36' 2023-10-10T18:51:16.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/20' 2023-10-10T18:51:16.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/39' 2023-10-10T18:51:16.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/34' 2023-10-10T18:51:16.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/38' 2023-10-10T18:51:16.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/28' 2023-10-10T18:51:16.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/24' 2023-10-10T18:51:16.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/29' 2023-10-10T18:51:16.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/32' 2023-10-10T18:51:16.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/22' 2023-10-10T18:51:16.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/25' 2023-10-10T18:51:16.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/30' 2023-10-10T18:51:16.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/33' 2023-10-10T18:51:16.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/23' 2023-10-10T18:51:16.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/35' 2023-10-10T18:51:16.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/26' 2023-10-10T18:51:16.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './204/37' 2023-10-10T18:51:16.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './204' 2023-10-10T18:51:16.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/21' 2023-10-10T18:51:16.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/27' 2023-10-10T18:51:16.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/31' 2023-10-10T18:51:16.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/36' 2023-10-10T18:51:16.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/20' 2023-10-10T18:51:16.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/39' 2023-10-10T18:51:16.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/34' 2023-10-10T18:51:16.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/38' 2023-10-10T18:51:16.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/28' 2023-10-10T18:51:16.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/24' 2023-10-10T18:51:16.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/29' 2023-10-10T18:51:16.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/32' 2023-10-10T18:51:16.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/22' 2023-10-10T18:51:16.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/25' 2023-10-10T18:51:16.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/30' 2023-10-10T18:51:16.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/33' 2023-10-10T18:51:16.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/23' 2023-10-10T18:51:16.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/35' 2023-10-10T18:51:16.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/26' 2023-10-10T18:51:16.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './205/37' 2023-10-10T18:51:16.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './205' 2023-10-10T18:51:16.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/21' 2023-10-10T18:51:16.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/27' 2023-10-10T18:51:16.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/31' 2023-10-10T18:51:16.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/36' 2023-10-10T18:51:16.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/20' 2023-10-10T18:51:16.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/39' 2023-10-10T18:51:19.107 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:51:19.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:51:19.640 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/34' 2023-10-10T18:51:19.640 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/38' 2023-10-10T18:51:19.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/28' 2023-10-10T18:51:19.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/24' 2023-10-10T18:51:19.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/29' 2023-10-10T18:51:19.641 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/32' 2023-10-10T18:51:19.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/22' 2023-10-10T18:51:19.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/25' 2023-10-10T18:51:19.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/30' 2023-10-10T18:51:19.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/33' 2023-10-10T18:51:19.642 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/23' 2023-10-10T18:51:19.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/35' 2023-10-10T18:51:19.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/26' 2023-10-10T18:51:19.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './206/37' 2023-10-10T18:51:19.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './206' 2023-10-10T18:51:19.643 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/21' 2023-10-10T18:51:19.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/27' 2023-10-10T18:51:19.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/31' 2023-10-10T18:51:19.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/36' 2023-10-10T18:51:19.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/20' 2023-10-10T18:51:19.644 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/39' 2023-10-10T18:51:19.645 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/34' 2023-10-10T18:51:19.645 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/38' 2023-10-10T18:51:19.645 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/28' 2023-10-10T18:51:19.645 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/24' 2023-10-10T18:51:19.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/29' 2023-10-10T18:51:19.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/32' 2023-10-10T18:51:19.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/22' 2023-10-10T18:51:19.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/25' 2023-10-10T18:51:19.646 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/30' 2023-10-10T18:51:19.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/33' 2023-10-10T18:51:19.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/23' 2023-10-10T18:51:19.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/35' 2023-10-10T18:51:19.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/26' 2023-10-10T18:51:19.647 INFO:tasks.workunit.client.0.smithi175.stdout:removed './207/37' 2023-10-10T18:51:19.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './207' 2023-10-10T18:51:19.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/21' 2023-10-10T18:51:19.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/27' 2023-10-10T18:51:19.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/31' 2023-10-10T18:51:19.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/36' 2023-10-10T18:51:19.648 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/20' 2023-10-10T18:51:19.649 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/39' 2023-10-10T18:51:19.649 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/34' 2023-10-10T18:51:19.649 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/38' 2023-10-10T18:51:19.649 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/28' 2023-10-10T18:51:19.650 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/24' 2023-10-10T18:51:19.650 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/29' 2023-10-10T18:51:19.650 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/32' 2023-10-10T18:51:19.650 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/22' 2023-10-10T18:51:19.651 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/25' 2023-10-10T18:51:19.651 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/30' 2023-10-10T18:51:19.651 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/33' 2023-10-10T18:51:19.651 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/23' 2023-10-10T18:51:19.651 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/35' 2023-10-10T18:51:19.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/26' 2023-10-10T18:51:19.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed './208/37' 2023-10-10T18:51:19.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './208' 2023-10-10T18:51:19.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/21' 2023-10-10T18:51:19.652 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/27' 2023-10-10T18:51:19.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/31' 2023-10-10T18:51:19.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/36' 2023-10-10T18:51:19.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/20' 2023-10-10T18:51:19.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/39' 2023-10-10T18:51:19.653 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/34' 2023-10-10T18:51:19.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/38' 2023-10-10T18:51:19.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/28' 2023-10-10T18:51:19.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/24' 2023-10-10T18:51:19.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/29' 2023-10-10T18:51:19.654 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/32' 2023-10-10T18:51:19.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/22' 2023-10-10T18:51:19.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/25' 2023-10-10T18:51:19.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/30' 2023-10-10T18:51:19.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/33' 2023-10-10T18:51:19.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/23' 2023-10-10T18:51:19.655 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/35' 2023-10-10T18:51:19.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/26' 2023-10-10T18:51:19.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed './209/37' 2023-10-10T18:51:19.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './209' 2023-10-10T18:51:19.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/21' 2023-10-10T18:51:19.656 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/27' 2023-10-10T18:51:19.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/31' 2023-10-10T18:51:19.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/36' 2023-10-10T18:51:19.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/20' 2023-10-10T18:51:19.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/39' 2023-10-10T18:51:19.657 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/34' 2023-10-10T18:51:19.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/38' 2023-10-10T18:51:19.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/28' 2023-10-10T18:51:19.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/24' 2023-10-10T18:51:19.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/29' 2023-10-10T18:51:19.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/32' 2023-10-10T18:51:19.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/22' 2023-10-10T18:51:19.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/25' 2023-10-10T18:51:19.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/30' 2023-10-10T18:51:19.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/33' 2023-10-10T18:51:19.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/23' 2023-10-10T18:51:19.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/35' 2023-10-10T18:51:19.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/26' 2023-10-10T18:51:19.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './210/37' 2023-10-10T18:51:19.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './210' 2023-10-10T18:51:19.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/21' 2023-10-10T18:51:19.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/27' 2023-10-10T18:51:19.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/31' 2023-10-10T18:51:19.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/36' 2023-10-10T18:51:19.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/20' 2023-10-10T18:51:19.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/39' 2023-10-10T18:51:19.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/34' 2023-10-10T18:51:19.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/38' 2023-10-10T18:51:19.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/28' 2023-10-10T18:51:19.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/24' 2023-10-10T18:51:19.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/29' 2023-10-10T18:51:19.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/32' 2023-10-10T18:51:19.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/22' 2023-10-10T18:51:19.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/25' 2023-10-10T18:51:19.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/30' 2023-10-10T18:51:19.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/33' 2023-10-10T18:51:19.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/23' 2023-10-10T18:51:19.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/35' 2023-10-10T18:51:19.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/26' 2023-10-10T18:51:19.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './211/37' 2023-10-10T18:51:19.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './211' 2023-10-10T18:51:19.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/21' 2023-10-10T18:51:19.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/27' 2023-10-10T18:51:19.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/31' 2023-10-10T18:51:19.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/36' 2023-10-10T18:51:19.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/20' 2023-10-10T18:51:19.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/39' 2023-10-10T18:51:19.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/34' 2023-10-10T18:51:19.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/38' 2023-10-10T18:51:19.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/28' 2023-10-10T18:51:19.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/24' 2023-10-10T18:51:19.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/29' 2023-10-10T18:51:19.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/32' 2023-10-10T18:51:19.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/22' 2023-10-10T18:51:19.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/25' 2023-10-10T18:51:19.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/30' 2023-10-10T18:51:19.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/33' 2023-10-10T18:51:19.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/23' 2023-10-10T18:51:19.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/35' 2023-10-10T18:51:19.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/26' 2023-10-10T18:51:19.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './212/37' 2023-10-10T18:51:19.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './212' 2023-10-10T18:51:19.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/21' 2023-10-10T18:51:19.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/27' 2023-10-10T18:51:19.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/31' 2023-10-10T18:51:19.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/36' 2023-10-10T18:51:19.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/20' 2023-10-10T18:51:19.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/39' 2023-10-10T18:51:19.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/34' 2023-10-10T18:51:19.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/38' 2023-10-10T18:51:19.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/28' 2023-10-10T18:51:19.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/24' 2023-10-10T18:51:19.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/29' 2023-10-10T18:51:19.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/32' 2023-10-10T18:51:19.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/22' 2023-10-10T18:51:19.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/25' 2023-10-10T18:51:19.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/30' 2023-10-10T18:51:19.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/33' 2023-10-10T18:51:19.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/23' 2023-10-10T18:51:19.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/35' 2023-10-10T18:51:19.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/26' 2023-10-10T18:51:19.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './213/37' 2023-10-10T18:51:19.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './213' 2023-10-10T18:51:19.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/21' 2023-10-10T18:51:19.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/27' 2023-10-10T18:51:19.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/31' 2023-10-10T18:51:19.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/36' 2023-10-10T18:51:19.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/20' 2023-10-10T18:51:19.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/39' 2023-10-10T18:51:19.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/34' 2023-10-10T18:51:19.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/38' 2023-10-10T18:51:19.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/28' 2023-10-10T18:51:19.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/24' 2023-10-10T18:51:19.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/29' 2023-10-10T18:51:19.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/32' 2023-10-10T18:51:19.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/22' 2023-10-10T18:51:19.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/25' 2023-10-10T18:51:19.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/30' 2023-10-10T18:51:19.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/33' 2023-10-10T18:51:19.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/23' 2023-10-10T18:51:19.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/35' 2023-10-10T18:51:19.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/26' 2023-10-10T18:51:19.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './214/37' 2023-10-10T18:51:19.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './214' 2023-10-10T18:51:19.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/21' 2023-10-10T18:51:19.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/27' 2023-10-10T18:51:19.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/31' 2023-10-10T18:51:19.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/36' 2023-10-10T18:51:19.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/20' 2023-10-10T18:51:19.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/39' 2023-10-10T18:51:19.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/34' 2023-10-10T18:51:19.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/38' 2023-10-10T18:51:19.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/28' 2023-10-10T18:51:19.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/24' 2023-10-10T18:51:19.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/29' 2023-10-10T18:51:19.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/32' 2023-10-10T18:51:19.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/22' 2023-10-10T18:51:19.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/25' 2023-10-10T18:51:19.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/30' 2023-10-10T18:51:19.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/33' 2023-10-10T18:51:19.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/23' 2023-10-10T18:51:19.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/35' 2023-10-10T18:51:19.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/26' 2023-10-10T18:51:19.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './215/37' 2023-10-10T18:51:19.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './215' 2023-10-10T18:51:19.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/21' 2023-10-10T18:51:19.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/27' 2023-10-10T18:51:19.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/31' 2023-10-10T18:51:19.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/36' 2023-10-10T18:51:19.682 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/20' 2023-10-10T18:51:19.683 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/39' 2023-10-10T18:51:19.683 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/34' 2023-10-10T18:51:19.683 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/38' 2023-10-10T18:51:23.375 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/28' 2023-10-10T18:51:23.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/24' 2023-10-10T18:51:23.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/29' 2023-10-10T18:51:23.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/32' 2023-10-10T18:51:23.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/22' 2023-10-10T18:51:23.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/25' 2023-10-10T18:51:23.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/30' 2023-10-10T18:51:23.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/33' 2023-10-10T18:51:23.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/23' 2023-10-10T18:51:23.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/35' 2023-10-10T18:51:23.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/26' 2023-10-10T18:51:23.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './216/37' 2023-10-10T18:51:23.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './216' 2023-10-10T18:51:23.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/21' 2023-10-10T18:51:23.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/27' 2023-10-10T18:51:23.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/31' 2023-10-10T18:51:23.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/36' 2023-10-10T18:51:23.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/20' 2023-10-10T18:51:23.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/39' 2023-10-10T18:51:23.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/34' 2023-10-10T18:51:23.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/38' 2023-10-10T18:51:23.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/28' 2023-10-10T18:51:23.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/24' 2023-10-10T18:51:23.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/29' 2023-10-10T18:51:23.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/32' 2023-10-10T18:51:23.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/22' 2023-10-10T18:51:23.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/25' 2023-10-10T18:51:23.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/30' 2023-10-10T18:51:23.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/33' 2023-10-10T18:51:23.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/23' 2023-10-10T18:51:23.382 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/35' 2023-10-10T18:51:23.382 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/26' 2023-10-10T18:51:23.382 INFO:tasks.workunit.client.0.smithi175.stdout:removed './217/37' 2023-10-10T18:51:23.382 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './217' 2023-10-10T18:51:23.382 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/21' 2023-10-10T18:51:23.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/27' 2023-10-10T18:51:23.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/31' 2023-10-10T18:51:23.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/36' 2023-10-10T18:51:23.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/20' 2023-10-10T18:51:23.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/39' 2023-10-10T18:51:23.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/34' 2023-10-10T18:51:23.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/38' 2023-10-10T18:51:23.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/28' 2023-10-10T18:51:23.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/24' 2023-10-10T18:51:23.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/29' 2023-10-10T18:51:23.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/32' 2023-10-10T18:51:23.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/22' 2023-10-10T18:51:23.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/25' 2023-10-10T18:51:23.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/30' 2023-10-10T18:51:23.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/33' 2023-10-10T18:51:23.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/23' 2023-10-10T18:51:23.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/35' 2023-10-10T18:51:23.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/26' 2023-10-10T18:51:23.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './218/37' 2023-10-10T18:51:23.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './218' 2023-10-10T18:51:23.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/21' 2023-10-10T18:51:23.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/27' 2023-10-10T18:51:23.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/31' 2023-10-10T18:51:23.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/36' 2023-10-10T18:51:23.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/20' 2023-10-10T18:51:23.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/39' 2023-10-10T18:51:23.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/34' 2023-10-10T18:51:23.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/38' 2023-10-10T18:51:23.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/28' 2023-10-10T18:51:23.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/24' 2023-10-10T18:51:23.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/29' 2023-10-10T18:51:23.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/32' 2023-10-10T18:51:23.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/22' 2023-10-10T18:51:23.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/25' 2023-10-10T18:51:23.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/30' 2023-10-10T18:51:23.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/33' 2023-10-10T18:51:23.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/23' 2023-10-10T18:51:23.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/35' 2023-10-10T18:51:23.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/26' 2023-10-10T18:51:23.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './219/37' 2023-10-10T18:51:23.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './219' 2023-10-10T18:51:23.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/21' 2023-10-10T18:51:23.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/27' 2023-10-10T18:51:23.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/31' 2023-10-10T18:51:23.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/36' 2023-10-10T18:51:23.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/20' 2023-10-10T18:51:23.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/39' 2023-10-10T18:51:23.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/34' 2023-10-10T18:51:23.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/38' 2023-10-10T18:51:23.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/28' 2023-10-10T18:51:23.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/24' 2023-10-10T18:51:23.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/29' 2023-10-10T18:51:23.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/32' 2023-10-10T18:51:23.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/22' 2023-10-10T18:51:23.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/25' 2023-10-10T18:51:23.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/30' 2023-10-10T18:51:23.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/33' 2023-10-10T18:51:23.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/23' 2023-10-10T18:51:23.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/35' 2023-10-10T18:51:23.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/26' 2023-10-10T18:51:23.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './220/37' 2023-10-10T18:51:23.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './220' 2023-10-10T18:51:23.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/21' 2023-10-10T18:51:23.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/27' 2023-10-10T18:51:23.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/31' 2023-10-10T18:51:23.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/36' 2023-10-10T18:51:23.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/20' 2023-10-10T18:51:23.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/39' 2023-10-10T18:51:23.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/34' 2023-10-10T18:51:23.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/38' 2023-10-10T18:51:23.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/28' 2023-10-10T18:51:23.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/24' 2023-10-10T18:51:23.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/29' 2023-10-10T18:51:23.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/32' 2023-10-10T18:51:23.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/22' 2023-10-10T18:51:23.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/25' 2023-10-10T18:51:23.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/30' 2023-10-10T18:51:23.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/33' 2023-10-10T18:51:23.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/23' 2023-10-10T18:51:23.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/35' 2023-10-10T18:51:23.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/26' 2023-10-10T18:51:23.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed './221/37' 2023-10-10T18:51:23.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './221' 2023-10-10T18:51:23.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/21' 2023-10-10T18:51:23.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/27' 2023-10-10T18:51:23.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/31' 2023-10-10T18:51:23.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/36' 2023-10-10T18:51:23.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/20' 2023-10-10T18:51:23.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/39' 2023-10-10T18:51:23.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/34' 2023-10-10T18:51:23.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/38' 2023-10-10T18:51:23.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/28' 2023-10-10T18:51:23.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/24' 2023-10-10T18:51:23.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/29' 2023-10-10T18:51:23.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/32' 2023-10-10T18:51:23.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/22' 2023-10-10T18:51:23.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/25' 2023-10-10T18:51:23.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/30' 2023-10-10T18:51:23.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/33' 2023-10-10T18:51:23.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/23' 2023-10-10T18:51:23.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/35' 2023-10-10T18:51:23.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/26' 2023-10-10T18:51:23.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './222/37' 2023-10-10T18:51:23.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './222' 2023-10-10T18:51:23.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/21' 2023-10-10T18:51:23.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/27' 2023-10-10T18:51:23.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/31' 2023-10-10T18:51:23.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/36' 2023-10-10T18:51:23.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/20' 2023-10-10T18:51:23.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/39' 2023-10-10T18:51:23.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/34' 2023-10-10T18:51:23.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/38' 2023-10-10T18:51:23.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/28' 2023-10-10T18:51:23.406 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/24' 2023-10-10T18:51:23.406 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/29' 2023-10-10T18:51:23.406 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/32' 2023-10-10T18:51:23.406 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/22' 2023-10-10T18:51:23.406 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/25' 2023-10-10T18:51:23.406 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/30' 2023-10-10T18:51:23.407 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/33' 2023-10-10T18:51:23.407 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/23' 2023-10-10T18:51:23.407 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/35' 2023-10-10T18:51:23.407 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/26' 2023-10-10T18:51:23.407 INFO:tasks.workunit.client.0.smithi175.stdout:removed './223/37' 2023-10-10T18:51:23.408 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './223' 2023-10-10T18:51:23.408 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/21' 2023-10-10T18:51:23.408 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/27' 2023-10-10T18:51:23.408 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/31' 2023-10-10T18:51:23.408 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/36' 2023-10-10T18:51:23.409 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/20' 2023-10-10T18:51:23.409 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/39' 2023-10-10T18:51:23.409 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/34' 2023-10-10T18:51:23.409 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/38' 2023-10-10T18:51:23.409 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/28' 2023-10-10T18:51:23.410 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/24' 2023-10-10T18:51:23.410 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/29' 2023-10-10T18:51:23.410 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/32' 2023-10-10T18:51:23.410 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/22' 2023-10-10T18:51:23.410 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/25' 2023-10-10T18:51:23.411 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/30' 2023-10-10T18:51:23.411 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/33' 2023-10-10T18:51:23.411 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/23' 2023-10-10T18:51:23.411 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/35' 2023-10-10T18:51:23.411 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/26' 2023-10-10T18:51:23.412 INFO:tasks.workunit.client.0.smithi175.stdout:removed './224/37' 2023-10-10T18:51:23.412 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './224' 2023-10-10T18:51:23.412 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/21' 2023-10-10T18:51:23.412 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/27' 2023-10-10T18:51:23.412 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/31' 2023-10-10T18:51:23.413 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/36' 2023-10-10T18:51:23.413 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/20' 2023-10-10T18:51:23.413 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/39' 2023-10-10T18:51:23.413 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/34' 2023-10-10T18:51:23.413 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/38' 2023-10-10T18:51:23.414 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/28' 2023-10-10T18:51:23.414 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/24' 2023-10-10T18:51:23.414 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/29' 2023-10-10T18:51:23.414 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/32' 2023-10-10T18:51:23.414 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/22' 2023-10-10T18:51:23.415 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/25' 2023-10-10T18:51:23.415 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/30' 2023-10-10T18:51:23.415 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/33' 2023-10-10T18:51:23.415 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/23' 2023-10-10T18:51:23.415 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/35' 2023-10-10T18:51:23.416 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/26' 2023-10-10T18:51:23.416 INFO:tasks.workunit.client.0.smithi175.stdout:removed './225/37' 2023-10-10T18:51:23.416 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './225' 2023-10-10T18:51:23.416 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/21' 2023-10-10T18:51:23.416 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/27' 2023-10-10T18:51:23.417 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/31' 2023-10-10T18:51:23.417 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/36' 2023-10-10T18:51:23.417 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/20' 2023-10-10T18:51:23.417 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/39' 2023-10-10T18:51:23.417 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/34' 2023-10-10T18:51:23.418 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/38' 2023-10-10T18:51:23.418 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/28' 2023-10-10T18:51:23.418 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/24' 2023-10-10T18:51:35.961 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/29' 2023-10-10T18:51:35.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/32' 2023-10-10T18:51:35.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/22' 2023-10-10T18:51:35.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/25' 2023-10-10T18:51:35.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/30' 2023-10-10T18:51:35.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/33' 2023-10-10T18:51:35.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/23' 2023-10-10T18:51:35.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/35' 2023-10-10T18:51:35.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/26' 2023-10-10T18:51:35.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './226/37' 2023-10-10T18:51:35.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './226' 2023-10-10T18:51:35.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/21' 2023-10-10T18:51:35.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/27' 2023-10-10T18:51:35.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/31' 2023-10-10T18:51:35.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/36' 2023-10-10T18:51:35.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/20' 2023-10-10T18:51:35.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/39' 2023-10-10T18:51:35.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/34' 2023-10-10T18:51:35.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/38' 2023-10-10T18:51:35.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/28' 2023-10-10T18:51:35.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/24' 2023-10-10T18:51:35.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/29' 2023-10-10T18:51:35.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/32' 2023-10-10T18:51:35.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/22' 2023-10-10T18:51:35.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/25' 2023-10-10T18:51:35.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/30' 2023-10-10T18:51:35.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/33' 2023-10-10T18:51:35.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/23' 2023-10-10T18:51:35.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/35' 2023-10-10T18:51:35.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/26' 2023-10-10T18:51:35.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed './227/37' 2023-10-10T18:51:35.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './227' 2023-10-10T18:51:35.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/21' 2023-10-10T18:51:35.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/27' 2023-10-10T18:51:35.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/31' 2023-10-10T18:51:35.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/36' 2023-10-10T18:51:35.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/20' 2023-10-10T18:51:35.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/39' 2023-10-10T18:51:35.970 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/34' 2023-10-10T18:51:35.970 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/38' 2023-10-10T18:51:35.970 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/28' 2023-10-10T18:51:35.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/24' 2023-10-10T18:51:35.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/29' 2023-10-10T18:51:35.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/32' 2023-10-10T18:51:35.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/22' 2023-10-10T18:51:35.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/25' 2023-10-10T18:51:35.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/30' 2023-10-10T18:51:35.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/33' 2023-10-10T18:51:35.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/23' 2023-10-10T18:51:35.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/35' 2023-10-10T18:51:35.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/26' 2023-10-10T18:51:35.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './228/37' 2023-10-10T18:51:35.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './228' 2023-10-10T18:51:35.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/21' 2023-10-10T18:51:35.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/27' 2023-10-10T18:51:35.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/31' 2023-10-10T18:51:35.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/36' 2023-10-10T18:51:35.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/20' 2023-10-10T18:51:35.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/39' 2023-10-10T18:51:35.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/34' 2023-10-10T18:51:35.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/38' 2023-10-10T18:51:35.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/28' 2023-10-10T18:51:35.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/24' 2023-10-10T18:51:35.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/29' 2023-10-10T18:51:35.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/32' 2023-10-10T18:51:35.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/22' 2023-10-10T18:51:35.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/25' 2023-10-10T18:51:35.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/30' 2023-10-10T18:51:35.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/33' 2023-10-10T18:51:35.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/23' 2023-10-10T18:51:35.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/35' 2023-10-10T18:51:35.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/26' 2023-10-10T18:51:35.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './229/37' 2023-10-10T18:51:35.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './229' 2023-10-10T18:51:35.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/21' 2023-10-10T18:51:35.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/27' 2023-10-10T18:51:35.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/31' 2023-10-10T18:51:35.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/36' 2023-10-10T18:51:35.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/20' 2023-10-10T18:51:35.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/39' 2023-10-10T18:51:35.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/34' 2023-10-10T18:51:35.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/38' 2023-10-10T18:51:35.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/28' 2023-10-10T18:51:35.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/24' 2023-10-10T18:51:35.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/29' 2023-10-10T18:51:35.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/32' 2023-10-10T18:51:35.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/22' 2023-10-10T18:51:35.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/25' 2023-10-10T18:51:35.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/30' 2023-10-10T18:51:35.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/33' 2023-10-10T18:51:35.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/23' 2023-10-10T18:51:35.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/35' 2023-10-10T18:51:35.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/26' 2023-10-10T18:51:35.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './230/37' 2023-10-10T18:51:35.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './230' 2023-10-10T18:51:35.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/21' 2023-10-10T18:51:35.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/27' 2023-10-10T18:51:35.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/31' 2023-10-10T18:51:35.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/36' 2023-10-10T18:51:35.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/20' 2023-10-10T18:51:35.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/39' 2023-10-10T18:51:35.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/34' 2023-10-10T18:51:35.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/38' 2023-10-10T18:51:35.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/28' 2023-10-10T18:51:35.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/24' 2023-10-10T18:51:35.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/29' 2023-10-10T18:51:35.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/32' 2023-10-10T18:51:35.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/22' 2023-10-10T18:51:35.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/25' 2023-10-10T18:51:35.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/30' 2023-10-10T18:51:35.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/33' 2023-10-10T18:51:35.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/23' 2023-10-10T18:51:35.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/35' 2023-10-10T18:51:35.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/26' 2023-10-10T18:51:35.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './231/37' 2023-10-10T18:51:35.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './231' 2023-10-10T18:51:35.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/21' 2023-10-10T18:51:35.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/27' 2023-10-10T18:51:35.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/31' 2023-10-10T18:51:35.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/36' 2023-10-10T18:51:35.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/20' 2023-10-10T18:51:35.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/39' 2023-10-10T18:51:35.986 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/34' 2023-10-10T18:51:35.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/38' 2023-10-10T18:51:35.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/28' 2023-10-10T18:51:35.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/24' 2023-10-10T18:51:35.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/29' 2023-10-10T18:51:35.987 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/32' 2023-10-10T18:51:35.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/22' 2023-10-10T18:51:35.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/25' 2023-10-10T18:51:35.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/30' 2023-10-10T18:51:35.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/33' 2023-10-10T18:51:35.988 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/23' 2023-10-10T18:51:35.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/35' 2023-10-10T18:51:35.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/26' 2023-10-10T18:51:35.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './232/37' 2023-10-10T18:51:35.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './232' 2023-10-10T18:51:35.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/21' 2023-10-10T18:51:35.989 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/27' 2023-10-10T18:51:35.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/31' 2023-10-10T18:51:35.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/36' 2023-10-10T18:51:35.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/20' 2023-10-10T18:51:35.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/39' 2023-10-10T18:51:35.990 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/34' 2023-10-10T18:51:35.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/38' 2023-10-10T18:51:35.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/28' 2023-10-10T18:51:35.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/24' 2023-10-10T18:51:35.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/29' 2023-10-10T18:51:35.991 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/32' 2023-10-10T18:51:35.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/22' 2023-10-10T18:51:35.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/25' 2023-10-10T18:51:35.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/30' 2023-10-10T18:51:35.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/33' 2023-10-10T18:51:35.992 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/23' 2023-10-10T18:51:35.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/35' 2023-10-10T18:51:35.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/26' 2023-10-10T18:51:35.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './233/37' 2023-10-10T18:51:35.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './233' 2023-10-10T18:51:35.993 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/21' 2023-10-10T18:51:35.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/27' 2023-10-10T18:51:35.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/31' 2023-10-10T18:51:35.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/36' 2023-10-10T18:51:35.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/20' 2023-10-10T18:51:35.994 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/39' 2023-10-10T18:51:35.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/34' 2023-10-10T18:51:35.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/38' 2023-10-10T18:51:35.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/28' 2023-10-10T18:51:35.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/24' 2023-10-10T18:51:35.995 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/29' 2023-10-10T18:51:35.996 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/32' 2023-10-10T18:51:35.996 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/22' 2023-10-10T18:51:35.996 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/25' 2023-10-10T18:51:35.996 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/30' 2023-10-10T18:51:35.996 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/33' 2023-10-10T18:51:35.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/23' 2023-10-10T18:51:35.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/35' 2023-10-10T18:51:35.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/26' 2023-10-10T18:51:35.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './234/37' 2023-10-10T18:51:35.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './234' 2023-10-10T18:51:35.997 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/21' 2023-10-10T18:51:35.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/27' 2023-10-10T18:51:35.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/31' 2023-10-10T18:51:35.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/36' 2023-10-10T18:51:35.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/20' 2023-10-10T18:51:35.998 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/39' 2023-10-10T18:51:35.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/34' 2023-10-10T18:51:35.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/38' 2023-10-10T18:51:35.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/28' 2023-10-10T18:51:35.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/24' 2023-10-10T18:51:35.999 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/29' 2023-10-10T18:51:36.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/32' 2023-10-10T18:51:36.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/22' 2023-10-10T18:51:36.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/25' 2023-10-10T18:51:36.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/30' 2023-10-10T18:51:36.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/33' 2023-10-10T18:51:36.000 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/23' 2023-10-10T18:51:36.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/35' 2023-10-10T18:51:36.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/26' 2023-10-10T18:51:36.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './235/37' 2023-10-10T18:51:36.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './235' 2023-10-10T18:51:36.001 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/21' 2023-10-10T18:51:36.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/27' 2023-10-10T18:51:36.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/31' 2023-10-10T18:51:36.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/36' 2023-10-10T18:51:36.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/20' 2023-10-10T18:51:36.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/39' 2023-10-10T18:51:36.002 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/34' 2023-10-10T18:51:36.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/38' 2023-10-10T18:51:36.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/28' 2023-10-10T18:51:36.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/24' 2023-10-10T18:51:36.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/29' 2023-10-10T18:51:36.003 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/32' 2023-10-10T18:51:40.752 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/22' 2023-10-10T18:51:40.752 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/25' 2023-10-10T18:51:40.752 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/30' 2023-10-10T18:51:40.752 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/33' 2023-10-10T18:51:40.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/23' 2023-10-10T18:51:40.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/35' 2023-10-10T18:51:40.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/26' 2023-10-10T18:51:40.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed './236/37' 2023-10-10T18:51:40.753 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './236' 2023-10-10T18:51:40.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/21' 2023-10-10T18:51:40.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/27' 2023-10-10T18:51:40.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/31' 2023-10-10T18:51:40.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/36' 2023-10-10T18:51:40.754 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/20' 2023-10-10T18:51:40.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/39' 2023-10-10T18:51:40.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/34' 2023-10-10T18:51:40.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/38' 2023-10-10T18:51:40.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/28' 2023-10-10T18:51:40.755 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/24' 2023-10-10T18:51:40.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/29' 2023-10-10T18:51:40.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/32' 2023-10-10T18:51:40.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/22' 2023-10-10T18:51:40.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/25' 2023-10-10T18:51:40.756 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/30' 2023-10-10T18:51:40.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/33' 2023-10-10T18:51:40.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/23' 2023-10-10T18:51:40.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/35' 2023-10-10T18:51:40.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/26' 2023-10-10T18:51:40.757 INFO:tasks.workunit.client.0.smithi175.stdout:removed './237/37' 2023-10-10T18:51:40.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './237' 2023-10-10T18:51:40.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/21' 2023-10-10T18:51:40.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/27' 2023-10-10T18:51:40.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/31' 2023-10-10T18:51:40.758 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/36' 2023-10-10T18:51:40.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/20' 2023-10-10T18:51:40.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/39' 2023-10-10T18:51:40.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/34' 2023-10-10T18:51:40.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/38' 2023-10-10T18:51:40.759 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/28' 2023-10-10T18:51:40.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/24' 2023-10-10T18:51:40.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/29' 2023-10-10T18:51:40.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/32' 2023-10-10T18:51:40.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/22' 2023-10-10T18:51:40.760 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/25' 2023-10-10T18:51:40.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/30' 2023-10-10T18:51:40.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/33' 2023-10-10T18:51:40.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/23' 2023-10-10T18:51:40.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/35' 2023-10-10T18:51:40.761 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/26' 2023-10-10T18:51:40.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed './238/37' 2023-10-10T18:51:40.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './238' 2023-10-10T18:51:40.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/21' 2023-10-10T18:51:40.762 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/27' 2023-10-10T18:51:40.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/31' 2023-10-10T18:51:40.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/36' 2023-10-10T18:51:40.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/20' 2023-10-10T18:51:40.763 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/39' 2023-10-10T18:51:40.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/34' 2023-10-10T18:51:40.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/38' 2023-10-10T18:51:40.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/28' 2023-10-10T18:51:40.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/24' 2023-10-10T18:51:40.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/29' 2023-10-10T18:51:40.764 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/32' 2023-10-10T18:51:40.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/22' 2023-10-10T18:51:40.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/25' 2023-10-10T18:51:40.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/30' 2023-10-10T18:51:40.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/33' 2023-10-10T18:51:40.765 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/23' 2023-10-10T18:51:40.766 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/35' 2023-10-10T18:51:40.766 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/26' 2023-10-10T18:51:40.767 INFO:tasks.workunit.client.0.smithi175.stdout:removed './239/37' 2023-10-10T18:51:40.767 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './239' 2023-10-10T18:51:40.767 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/21' 2023-10-10T18:51:40.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/27' 2023-10-10T18:51:40.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/31' 2023-10-10T18:51:40.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/36' 2023-10-10T18:51:40.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/20' 2023-10-10T18:51:40.768 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/39' 2023-10-10T18:51:40.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/34' 2023-10-10T18:51:40.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/38' 2023-10-10T18:51:40.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/28' 2023-10-10T18:51:40.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/24' 2023-10-10T18:51:40.769 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/29' 2023-10-10T18:51:40.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/32' 2023-10-10T18:51:40.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/22' 2023-10-10T18:51:40.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/25' 2023-10-10T18:51:40.770 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/30' 2023-10-10T18:51:40.771 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/33' 2023-10-10T18:51:40.771 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/23' 2023-10-10T18:51:40.771 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/35' 2023-10-10T18:51:40.771 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/26' 2023-10-10T18:51:40.771 INFO:tasks.workunit.client.0.smithi175.stdout:removed './240/37' 2023-10-10T18:51:40.772 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './240' 2023-10-10T18:51:40.772 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/21' 2023-10-10T18:51:40.772 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/27' 2023-10-10T18:51:40.772 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/31' 2023-10-10T18:51:40.772 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/36' 2023-10-10T18:51:40.773 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/20' 2023-10-10T18:51:40.773 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/39' 2023-10-10T18:51:40.773 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/34' 2023-10-10T18:51:40.773 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/38' 2023-10-10T18:51:40.773 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/28' 2023-10-10T18:51:40.774 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/24' 2023-10-10T18:51:40.774 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/29' 2023-10-10T18:51:40.774 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/32' 2023-10-10T18:51:40.774 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/22' 2023-10-10T18:51:40.774 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/25' 2023-10-10T18:51:40.775 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/30' 2023-10-10T18:51:40.775 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/33' 2023-10-10T18:51:40.775 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/23' 2023-10-10T18:51:40.775 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/35' 2023-10-10T18:51:40.775 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/26' 2023-10-10T18:51:40.776 INFO:tasks.workunit.client.0.smithi175.stdout:removed './241/37' 2023-10-10T18:51:40.776 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './241' 2023-10-10T18:51:40.776 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/21' 2023-10-10T18:51:40.776 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/27' 2023-10-10T18:51:40.776 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/31' 2023-10-10T18:51:40.777 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/36' 2023-10-10T18:51:40.777 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/20' 2023-10-10T18:51:40.777 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/39' 2023-10-10T18:51:40.777 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/34' 2023-10-10T18:51:40.777 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/38' 2023-10-10T18:51:40.778 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/28' 2023-10-10T18:51:40.778 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/24' 2023-10-10T18:51:40.778 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/29' 2023-10-10T18:51:40.778 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/32' 2023-10-10T18:51:40.778 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/22' 2023-10-10T18:51:40.779 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/25' 2023-10-10T18:51:40.779 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/30' 2023-10-10T18:51:40.779 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/33' 2023-10-10T18:51:40.779 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/23' 2023-10-10T18:51:40.779 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/35' 2023-10-10T18:51:40.779 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/26' 2023-10-10T18:51:40.780 INFO:tasks.workunit.client.0.smithi175.stdout:removed './242/37' 2023-10-10T18:51:40.780 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './242' 2023-10-10T18:51:40.780 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/21' 2023-10-10T18:51:40.780 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/27' 2023-10-10T18:51:40.781 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/31' 2023-10-10T18:51:40.781 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/36' 2023-10-10T18:51:40.781 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/20' 2023-10-10T18:51:40.781 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/39' 2023-10-10T18:51:40.781 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/34' 2023-10-10T18:51:40.782 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/38' 2023-10-10T18:51:40.782 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/28' 2023-10-10T18:51:40.782 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/24' 2023-10-10T18:51:40.782 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/29' 2023-10-10T18:51:40.783 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/32' 2023-10-10T18:51:40.783 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/22' 2023-10-10T18:51:40.783 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/25' 2023-10-10T18:51:40.783 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/30' 2023-10-10T18:51:40.783 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/33' 2023-10-10T18:51:40.783 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/23' 2023-10-10T18:51:40.784 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/35' 2023-10-10T18:51:40.784 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/26' 2023-10-10T18:51:40.784 INFO:tasks.workunit.client.0.smithi175.stdout:removed './243/37' 2023-10-10T18:51:40.784 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './243' 2023-10-10T18:51:40.784 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/21' 2023-10-10T18:51:40.785 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/27' 2023-10-10T18:51:40.785 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/31' 2023-10-10T18:51:40.785 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/36' 2023-10-10T18:51:40.785 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/20' 2023-10-10T18:51:40.785 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/39' 2023-10-10T18:51:40.785 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/34' 2023-10-10T18:51:40.786 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/38' 2023-10-10T18:51:40.786 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/28' 2023-10-10T18:51:40.786 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/24' 2023-10-10T18:51:40.786 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/29' 2023-10-10T18:51:40.786 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/32' 2023-10-10T18:51:40.787 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/22' 2023-10-10T18:51:40.787 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/25' 2023-10-10T18:51:40.787 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/30' 2023-10-10T18:51:40.787 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/33' 2023-10-10T18:51:40.787 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/23' 2023-10-10T18:51:40.788 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/35' 2023-10-10T18:51:40.788 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/26' 2023-10-10T18:51:40.788 INFO:tasks.workunit.client.0.smithi175.stdout:removed './244/37' 2023-10-10T18:51:40.788 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './244' 2023-10-10T18:51:40.788 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/21' 2023-10-10T18:51:40.789 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/27' 2023-10-10T18:51:40.789 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/31' 2023-10-10T18:51:40.789 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/36' 2023-10-10T18:51:40.789 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/20' 2023-10-10T18:51:40.789 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/39' 2023-10-10T18:51:40.789 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/34' 2023-10-10T18:51:40.790 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/38' 2023-10-10T18:51:40.790 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/28' 2023-10-10T18:51:40.790 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/24' 2023-10-10T18:51:40.790 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/29' 2023-10-10T18:51:40.790 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/32' 2023-10-10T18:51:40.791 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/22' 2023-10-10T18:51:40.791 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/25' 2023-10-10T18:51:40.791 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/30' 2023-10-10T18:51:40.791 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/33' 2023-10-10T18:51:40.791 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/23' 2023-10-10T18:51:40.792 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/35' 2023-10-10T18:51:40.792 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/26' 2023-10-10T18:51:40.792 INFO:tasks.workunit.client.0.smithi175.stdout:removed './245/37' 2023-10-10T18:51:40.792 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './245' 2023-10-10T18:51:40.793 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/21' 2023-10-10T18:51:40.793 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/27' 2023-10-10T18:51:40.793 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/31' 2023-10-10T18:51:40.793 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/36' 2023-10-10T18:51:40.794 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/20' 2023-10-10T18:51:40.794 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/39' 2023-10-10T18:51:40.794 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/34' 2023-10-10T18:51:40.794 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/38' 2023-10-10T18:51:40.794 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/28' 2023-10-10T18:51:40.795 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/24' 2023-10-10T18:51:40.795 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/29' 2023-10-10T18:51:40.795 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/32' 2023-10-10T18:51:40.795 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/22' 2023-10-10T18:51:43.363 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/25' 2023-10-10T18:51:43.364 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/30' 2023-10-10T18:51:43.364 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/33' 2023-10-10T18:51:43.364 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/23' 2023-10-10T18:51:43.364 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/35' 2023-10-10T18:51:43.364 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/26' 2023-10-10T18:51:43.365 INFO:tasks.workunit.client.0.smithi175.stdout:removed './246/37' 2023-10-10T18:51:43.365 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './246' 2023-10-10T18:51:43.365 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/21' 2023-10-10T18:51:43.365 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/27' 2023-10-10T18:51:43.365 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/31' 2023-10-10T18:51:43.366 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/36' 2023-10-10T18:51:43.366 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/20' 2023-10-10T18:51:43.366 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/39' 2023-10-10T18:51:43.366 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/34' 2023-10-10T18:51:43.366 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/38' 2023-10-10T18:51:43.367 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/28' 2023-10-10T18:51:43.367 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/24' 2023-10-10T18:51:43.367 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/29' 2023-10-10T18:51:43.367 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/32' 2023-10-10T18:51:43.367 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/22' 2023-10-10T18:51:43.368 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/25' 2023-10-10T18:51:43.368 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/30' 2023-10-10T18:51:43.368 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/33' 2023-10-10T18:51:43.368 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/23' 2023-10-10T18:51:43.369 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/35' 2023-10-10T18:51:43.369 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/26' 2023-10-10T18:51:43.369 INFO:tasks.workunit.client.0.smithi175.stdout:removed './247/37' 2023-10-10T18:51:43.369 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './247' 2023-10-10T18:51:43.369 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/21' 2023-10-10T18:51:43.370 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/27' 2023-10-10T18:51:43.370 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/31' 2023-10-10T18:51:43.370 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/36' 2023-10-10T18:51:43.370 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/20' 2023-10-10T18:51:43.370 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/39' 2023-10-10T18:51:43.371 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/34' 2023-10-10T18:51:43.371 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/38' 2023-10-10T18:51:43.371 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/28' 2023-10-10T18:51:43.372 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/24' 2023-10-10T18:51:43.372 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/29' 2023-10-10T18:51:43.372 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/32' 2023-10-10T18:51:43.372 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/22' 2023-10-10T18:51:43.372 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/25' 2023-10-10T18:51:43.373 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/30' 2023-10-10T18:51:43.373 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/33' 2023-10-10T18:51:43.373 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/23' 2023-10-10T18:51:43.373 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/35' 2023-10-10T18:51:43.373 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/26' 2023-10-10T18:51:43.374 INFO:tasks.workunit.client.0.smithi175.stdout:removed './248/37' 2023-10-10T18:51:43.374 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './248' 2023-10-10T18:51:43.374 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/21' 2023-10-10T18:51:43.374 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/27' 2023-10-10T18:51:43.375 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/31' 2023-10-10T18:51:43.375 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/36' 2023-10-10T18:51:43.375 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/20' 2023-10-10T18:51:43.375 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/39' 2023-10-10T18:51:43.375 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/34' 2023-10-10T18:51:43.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/38' 2023-10-10T18:51:43.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/28' 2023-10-10T18:51:43.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/24' 2023-10-10T18:51:43.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/29' 2023-10-10T18:51:43.376 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/32' 2023-10-10T18:51:43.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/22' 2023-10-10T18:51:43.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/25' 2023-10-10T18:51:43.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/30' 2023-10-10T18:51:43.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/33' 2023-10-10T18:51:43.377 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/23' 2023-10-10T18:51:43.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/35' 2023-10-10T18:51:43.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/26' 2023-10-10T18:51:43.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './249/37' 2023-10-10T18:51:43.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './249' 2023-10-10T18:51:43.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/21' 2023-10-10T18:51:43.378 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/27' 2023-10-10T18:51:43.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/31' 2023-10-10T18:51:43.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/36' 2023-10-10T18:51:43.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/20' 2023-10-10T18:51:43.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/39' 2023-10-10T18:51:43.379 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/34' 2023-10-10T18:51:43.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/38' 2023-10-10T18:51:43.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/28' 2023-10-10T18:51:43.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/24' 2023-10-10T18:51:43.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/29' 2023-10-10T18:51:43.380 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/32' 2023-10-10T18:51:43.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/22' 2023-10-10T18:51:43.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/25' 2023-10-10T18:51:43.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/30' 2023-10-10T18:51:43.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/33' 2023-10-10T18:51:43.381 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/23' 2023-10-10T18:51:43.382 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/35' 2023-10-10T18:51:43.382 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/26' 2023-10-10T18:51:43.382 INFO:tasks.workunit.client.0.smithi175.stdout:removed './250/37' 2023-10-10T18:51:43.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './250' 2023-10-10T18:51:43.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/21' 2023-10-10T18:51:43.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/27' 2023-10-10T18:51:43.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/31' 2023-10-10T18:51:43.383 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/36' 2023-10-10T18:51:43.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/20' 2023-10-10T18:51:43.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/39' 2023-10-10T18:51:43.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/34' 2023-10-10T18:51:43.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/38' 2023-10-10T18:51:43.384 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/28' 2023-10-10T18:51:43.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/24' 2023-10-10T18:51:43.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/29' 2023-10-10T18:51:43.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/32' 2023-10-10T18:51:43.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/22' 2023-10-10T18:51:43.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/25' 2023-10-10T18:51:43.385 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/30' 2023-10-10T18:51:43.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/33' 2023-10-10T18:51:43.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/23' 2023-10-10T18:51:43.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/35' 2023-10-10T18:51:43.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/26' 2023-10-10T18:51:43.386 INFO:tasks.workunit.client.0.smithi175.stdout:removed './251/37' 2023-10-10T18:51:43.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './251' 2023-10-10T18:51:43.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/21' 2023-10-10T18:51:43.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/27' 2023-10-10T18:51:43.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/31' 2023-10-10T18:51:43.387 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/36' 2023-10-10T18:51:43.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/20' 2023-10-10T18:51:43.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/39' 2023-10-10T18:51:43.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/34' 2023-10-10T18:51:43.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/38' 2023-10-10T18:51:43.388 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/28' 2023-10-10T18:51:43.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/24' 2023-10-10T18:51:43.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/29' 2023-10-10T18:51:43.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/32' 2023-10-10T18:51:43.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/22' 2023-10-10T18:51:43.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/25' 2023-10-10T18:51:43.389 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/30' 2023-10-10T18:51:43.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/33' 2023-10-10T18:51:43.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/23' 2023-10-10T18:51:43.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/35' 2023-10-10T18:51:43.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/26' 2023-10-10T18:51:43.390 INFO:tasks.workunit.client.0.smithi175.stdout:removed './252/37' 2023-10-10T18:51:43.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './252' 2023-10-10T18:51:43.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/21' 2023-10-10T18:51:43.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/27' 2023-10-10T18:51:43.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/31' 2023-10-10T18:51:43.391 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/36' 2023-10-10T18:51:43.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/20' 2023-10-10T18:51:43.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/39' 2023-10-10T18:51:43.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/34' 2023-10-10T18:51:43.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/38' 2023-10-10T18:51:43.392 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/28' 2023-10-10T18:51:43.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/24' 2023-10-10T18:51:43.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/29' 2023-10-10T18:51:43.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/32' 2023-10-10T18:51:43.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/22' 2023-10-10T18:51:43.393 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/25' 2023-10-10T18:51:43.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/30' 2023-10-10T18:51:43.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/33' 2023-10-10T18:51:43.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/23' 2023-10-10T18:51:43.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/35' 2023-10-10T18:51:43.394 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/26' 2023-10-10T18:51:43.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './253/37' 2023-10-10T18:51:43.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './253' 2023-10-10T18:51:43.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/21' 2023-10-10T18:51:43.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/27' 2023-10-10T18:51:43.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/31' 2023-10-10T18:51:43.395 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/36' 2023-10-10T18:51:43.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/20' 2023-10-10T18:51:43.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/39' 2023-10-10T18:51:43.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/34' 2023-10-10T18:51:43.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/38' 2023-10-10T18:51:43.396 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/28' 2023-10-10T18:51:43.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/24' 2023-10-10T18:51:43.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/29' 2023-10-10T18:51:43.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/32' 2023-10-10T18:51:43.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/22' 2023-10-10T18:51:43.397 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/25' 2023-10-10T18:51:43.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/30' 2023-10-10T18:51:43.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/33' 2023-10-10T18:51:43.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/23' 2023-10-10T18:51:43.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/35' 2023-10-10T18:51:43.398 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/26' 2023-10-10T18:51:43.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed './254/37' 2023-10-10T18:51:43.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './254' 2023-10-10T18:51:43.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/21' 2023-10-10T18:51:43.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/27' 2023-10-10T18:51:43.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/31' 2023-10-10T18:51:43.399 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/36' 2023-10-10T18:51:43.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/20' 2023-10-10T18:51:43.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/39' 2023-10-10T18:51:43.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/34' 2023-10-10T18:51:43.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/38' 2023-10-10T18:51:43.400 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/28' 2023-10-10T18:51:43.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/24' 2023-10-10T18:51:43.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/29' 2023-10-10T18:51:43.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/32' 2023-10-10T18:51:43.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/22' 2023-10-10T18:51:43.401 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/25' 2023-10-10T18:51:43.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/30' 2023-10-10T18:51:43.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/33' 2023-10-10T18:51:43.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/23' 2023-10-10T18:51:43.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/35' 2023-10-10T18:51:43.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/26' 2023-10-10T18:51:43.402 INFO:tasks.workunit.client.0.smithi175.stdout:removed './255/37' 2023-10-10T18:51:43.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './255' 2023-10-10T18:51:43.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/21' 2023-10-10T18:51:43.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/27' 2023-10-10T18:51:43.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/31' 2023-10-10T18:51:43.403 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/36' 2023-10-10T18:51:43.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/20' 2023-10-10T18:51:43.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/39' 2023-10-10T18:51:43.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/34' 2023-10-10T18:51:43.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/38' 2023-10-10T18:51:43.404 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/28' 2023-10-10T18:51:43.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/24' 2023-10-10T18:51:43.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/29' 2023-10-10T18:51:43.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/32' 2023-10-10T18:51:43.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/22' 2023-10-10T18:51:43.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/25' 2023-10-10T18:51:43.405 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/30' 2023-10-10T18:51:49.131 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/33' 2023-10-10T18:51:49.131 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/23' 2023-10-10T18:51:49.132 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/35' 2023-10-10T18:51:49.132 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/26' 2023-10-10T18:51:49.132 INFO:tasks.workunit.client.0.smithi175.stdout:removed './256/37' 2023-10-10T18:51:49.133 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './256' 2023-10-10T18:51:49.133 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/21' 2023-10-10T18:51:49.133 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/27' 2023-10-10T18:51:49.134 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/31' 2023-10-10T18:51:49.134 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/36' 2023-10-10T18:51:49.134 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/20' 2023-10-10T18:51:49.135 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/39' 2023-10-10T18:51:49.135 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/34' 2023-10-10T18:51:49.135 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/38' 2023-10-10T18:51:49.136 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/28' 2023-10-10T18:51:49.136 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/24' 2023-10-10T18:51:49.136 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/29' 2023-10-10T18:51:49.137 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/32' 2023-10-10T18:51:49.137 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/22' 2023-10-10T18:51:49.137 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/25' 2023-10-10T18:51:49.138 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/30' 2023-10-10T18:51:49.138 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/33' 2023-10-10T18:51:49.138 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/23' 2023-10-10T18:51:49.139 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/35' 2023-10-10T18:51:49.139 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/26' 2023-10-10T18:51:49.139 INFO:tasks.workunit.client.0.smithi175.stdout:removed './257/37' 2023-10-10T18:51:49.140 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './257' 2023-10-10T18:51:49.140 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/21' 2023-10-10T18:51:49.140 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/27' 2023-10-10T18:51:49.140 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/31' 2023-10-10T18:51:49.141 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/36' 2023-10-10T18:51:49.141 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/20' 2023-10-10T18:51:49.141 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/39' 2023-10-10T18:51:49.141 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/34' 2023-10-10T18:51:49.141 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/38' 2023-10-10T18:51:49.142 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/28' 2023-10-10T18:51:49.142 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/24' 2023-10-10T18:51:49.142 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/29' 2023-10-10T18:51:49.142 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/32' 2023-10-10T18:51:49.143 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/22' 2023-10-10T18:51:49.143 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/25' 2023-10-10T18:51:49.143 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/30' 2023-10-10T18:51:49.143 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/33' 2023-10-10T18:51:49.144 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/23' 2023-10-10T18:51:49.144 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/35' 2023-10-10T18:51:49.144 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/26' 2023-10-10T18:51:49.144 INFO:tasks.workunit.client.0.smithi175.stdout:removed './258/37' 2023-10-10T18:51:49.145 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './258' 2023-10-10T18:51:49.145 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/21' 2023-10-10T18:51:49.145 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/27' 2023-10-10T18:51:49.145 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/31' 2023-10-10T18:51:49.145 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/36' 2023-10-10T18:51:49.146 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/20' 2023-10-10T18:51:49.146 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/39' 2023-10-10T18:51:49.146 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/34' 2023-10-10T18:51:49.146 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/38' 2023-10-10T18:51:49.146 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/28' 2023-10-10T18:51:49.147 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/24' 2023-10-10T18:51:49.147 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/29' 2023-10-10T18:51:49.147 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/32' 2023-10-10T18:51:49.147 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/22' 2023-10-10T18:51:49.147 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/25' 2023-10-10T18:51:49.147 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/30' 2023-10-10T18:51:49.148 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/33' 2023-10-10T18:51:49.148 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/23' 2023-10-10T18:51:49.148 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/35' 2023-10-10T18:51:49.148 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/26' 2023-10-10T18:51:49.148 INFO:tasks.workunit.client.0.smithi175.stdout:removed './259/37' 2023-10-10T18:51:49.149 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './259' 2023-10-10T18:51:49.149 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/21' 2023-10-10T18:51:49.149 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/27' 2023-10-10T18:51:49.149 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/31' 2023-10-10T18:51:49.149 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/36' 2023-10-10T18:51:49.150 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/20' 2023-10-10T18:51:49.150 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/39' 2023-10-10T18:51:49.150 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/34' 2023-10-10T18:51:49.150 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/38' 2023-10-10T18:51:49.150 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/28' 2023-10-10T18:51:49.151 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/24' 2023-10-10T18:51:49.151 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/29' 2023-10-10T18:51:49.151 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/32' 2023-10-10T18:51:49.151 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/22' 2023-10-10T18:51:49.151 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/25' 2023-10-10T18:51:49.152 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/30' 2023-10-10T18:51:49.152 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/33' 2023-10-10T18:51:49.152 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/23' 2023-10-10T18:51:49.152 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/35' 2023-10-10T18:51:49.152 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/26' 2023-10-10T18:51:49.152 INFO:tasks.workunit.client.0.smithi175.stdout:removed './260/37' 2023-10-10T18:51:49.153 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './260' 2023-10-10T18:51:49.153 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/21' 2023-10-10T18:51:49.153 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/27' 2023-10-10T18:51:49.153 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/31' 2023-10-10T18:51:49.153 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/36' 2023-10-10T18:51:49.154 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/20' 2023-10-10T18:51:49.154 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/39' 2023-10-10T18:51:49.154 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/34' 2023-10-10T18:51:49.154 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/38' 2023-10-10T18:51:49.154 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/28' 2023-10-10T18:51:49.155 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/24' 2023-10-10T18:51:49.155 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/29' 2023-10-10T18:51:49.155 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/32' 2023-10-10T18:51:49.155 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/22' 2023-10-10T18:51:49.155 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/25' 2023-10-10T18:51:49.156 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/30' 2023-10-10T18:51:49.156 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/33' 2023-10-10T18:51:49.156 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/23' 2023-10-10T18:51:49.156 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/35' 2023-10-10T18:51:49.156 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/26' 2023-10-10T18:51:49.157 INFO:tasks.workunit.client.0.smithi175.stdout:removed './261/37' 2023-10-10T18:51:49.157 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './261' 2023-10-10T18:51:49.157 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/21' 2023-10-10T18:51:49.157 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/27' 2023-10-10T18:51:49.157 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/31' 2023-10-10T18:51:49.158 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/36' 2023-10-10T18:51:49.158 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/20' 2023-10-10T18:51:49.158 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/39' 2023-10-10T18:51:49.158 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/34' 2023-10-10T18:51:49.159 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/38' 2023-10-10T18:51:49.159 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/28' 2023-10-10T18:51:49.159 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/24' 2023-10-10T18:51:49.159 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/29' 2023-10-10T18:51:49.159 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/32' 2023-10-10T18:51:49.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/22' 2023-10-10T18:51:49.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/25' 2023-10-10T18:51:49.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/30' 2023-10-10T18:51:49.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/33' 2023-10-10T18:51:49.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/23' 2023-10-10T18:51:49.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/35' 2023-10-10T18:51:49.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/26' 2023-10-10T18:51:49.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './262/37' 2023-10-10T18:51:49.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './262' 2023-10-10T18:51:49.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/21' 2023-10-10T18:51:49.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/27' 2023-10-10T18:51:49.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/31' 2023-10-10T18:51:49.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/36' 2023-10-10T18:51:49.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/20' 2023-10-10T18:51:49.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/39' 2023-10-10T18:51:49.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/34' 2023-10-10T18:51:49.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/38' 2023-10-10T18:51:49.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/28' 2023-10-10T18:51:49.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/24' 2023-10-10T18:51:49.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/29' 2023-10-10T18:51:49.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/32' 2023-10-10T18:51:49.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/22' 2023-10-10T18:51:49.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/25' 2023-10-10T18:51:49.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/30' 2023-10-10T18:51:49.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/33' 2023-10-10T18:51:49.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/23' 2023-10-10T18:51:49.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/35' 2023-10-10T18:51:49.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/26' 2023-10-10T18:51:49.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './263/37' 2023-10-10T18:51:49.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './263' 2023-10-10T18:51:49.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/21' 2023-10-10T18:51:49.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/27' 2023-10-10T18:51:49.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/31' 2023-10-10T18:51:49.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/36' 2023-10-10T18:51:49.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/20' 2023-10-10T18:51:49.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/39' 2023-10-10T18:51:49.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/34' 2023-10-10T18:51:49.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/38' 2023-10-10T18:51:49.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/28' 2023-10-10T18:51:49.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/24' 2023-10-10T18:51:49.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/29' 2023-10-10T18:51:49.168 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/32' 2023-10-10T18:51:49.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/22' 2023-10-10T18:51:49.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/25' 2023-10-10T18:51:49.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/30' 2023-10-10T18:51:49.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/33' 2023-10-10T18:51:49.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/23' 2023-10-10T18:51:49.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/35' 2023-10-10T18:51:49.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/26' 2023-10-10T18:51:49.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './264/37' 2023-10-10T18:51:49.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './264' 2023-10-10T18:51:49.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/21' 2023-10-10T18:51:49.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/27' 2023-10-10T18:51:49.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/31' 2023-10-10T18:51:49.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/36' 2023-10-10T18:51:49.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/20' 2023-10-10T18:51:49.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/39' 2023-10-10T18:51:49.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/34' 2023-10-10T18:51:49.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/38' 2023-10-10T18:51:49.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/28' 2023-10-10T18:51:49.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/24' 2023-10-10T18:51:49.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/29' 2023-10-10T18:51:49.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/32' 2023-10-10T18:51:49.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/22' 2023-10-10T18:51:49.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/25' 2023-10-10T18:51:49.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/30' 2023-10-10T18:51:49.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/33' 2023-10-10T18:51:49.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/23' 2023-10-10T18:51:49.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/35' 2023-10-10T18:51:49.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/26' 2023-10-10T18:51:49.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './265/37' 2023-10-10T18:51:49.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './265' 2023-10-10T18:51:49.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/21' 2023-10-10T18:51:49.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/27' 2023-10-10T18:51:49.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/31' 2023-10-10T18:51:49.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/36' 2023-10-10T18:51:49.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/20' 2023-10-10T18:51:49.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/39' 2023-10-10T18:51:49.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/34' 2023-10-10T18:51:49.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/38' 2023-10-10T18:51:49.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/28' 2023-10-10T18:51:49.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/24' 2023-10-10T18:51:49.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/29' 2023-10-10T18:51:49.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/32' 2023-10-10T18:51:49.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/22' 2023-10-10T18:51:49.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/25' 2023-10-10T18:51:49.178 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/30' 2023-10-10T18:51:49.178 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/33' 2023-10-10T18:51:49.178 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/23' 2023-10-10T18:51:49.203 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:51:49.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:51:53.917 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/35' 2023-10-10T18:51:53.917 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/26' 2023-10-10T18:51:53.917 INFO:tasks.workunit.client.0.smithi175.stdout:removed './266/37' 2023-10-10T18:51:53.918 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './266' 2023-10-10T18:51:53.918 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/21' 2023-10-10T18:51:53.918 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/27' 2023-10-10T18:51:53.918 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/31' 2023-10-10T18:51:53.918 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/36' 2023-10-10T18:51:53.919 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/20' 2023-10-10T18:51:53.919 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/39' 2023-10-10T18:51:53.919 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/34' 2023-10-10T18:51:53.919 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/38' 2023-10-10T18:51:53.920 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/28' 2023-10-10T18:51:53.920 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/24' 2023-10-10T18:51:53.920 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/29' 2023-10-10T18:51:53.920 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/32' 2023-10-10T18:51:53.920 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/22' 2023-10-10T18:51:53.921 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/25' 2023-10-10T18:51:53.921 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/30' 2023-10-10T18:51:53.921 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/33' 2023-10-10T18:51:53.921 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/23' 2023-10-10T18:51:53.921 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/35' 2023-10-10T18:51:53.922 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/26' 2023-10-10T18:51:53.922 INFO:tasks.workunit.client.0.smithi175.stdout:removed './267/37' 2023-10-10T18:51:53.922 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './267' 2023-10-10T18:51:53.922 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/21' 2023-10-10T18:51:53.922 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/27' 2023-10-10T18:51:53.923 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/31' 2023-10-10T18:51:53.923 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/36' 2023-10-10T18:51:53.923 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/20' 2023-10-10T18:51:53.923 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/39' 2023-10-10T18:51:53.923 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/34' 2023-10-10T18:51:53.924 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/38' 2023-10-10T18:51:53.924 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/28' 2023-10-10T18:51:53.924 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/24' 2023-10-10T18:51:53.924 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/29' 2023-10-10T18:51:53.925 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/32' 2023-10-10T18:51:53.925 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/22' 2023-10-10T18:51:53.925 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/25' 2023-10-10T18:51:53.925 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/30' 2023-10-10T18:51:53.925 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/33' 2023-10-10T18:51:53.926 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/23' 2023-10-10T18:51:53.926 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/35' 2023-10-10T18:51:53.926 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/26' 2023-10-10T18:51:53.926 INFO:tasks.workunit.client.0.smithi175.stdout:removed './268/37' 2023-10-10T18:51:53.926 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './268' 2023-10-10T18:51:53.927 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/21' 2023-10-10T18:51:53.927 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/27' 2023-10-10T18:51:53.927 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/31' 2023-10-10T18:51:53.927 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/36' 2023-10-10T18:51:53.927 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/20' 2023-10-10T18:51:53.928 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/39' 2023-10-10T18:51:53.928 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/34' 2023-10-10T18:51:53.928 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/38' 2023-10-10T18:51:53.928 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/28' 2023-10-10T18:51:53.928 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/24' 2023-10-10T18:51:53.928 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/29' 2023-10-10T18:51:53.929 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/32' 2023-10-10T18:51:53.929 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/22' 2023-10-10T18:51:53.929 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/25' 2023-10-10T18:51:53.929 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/30' 2023-10-10T18:51:53.929 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/33' 2023-10-10T18:51:53.929 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/23' 2023-10-10T18:51:53.930 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/35' 2023-10-10T18:51:53.930 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/26' 2023-10-10T18:51:53.930 INFO:tasks.workunit.client.0.smithi175.stdout:removed './269/37' 2023-10-10T18:51:53.930 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './269' 2023-10-10T18:51:53.930 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/21' 2023-10-10T18:51:53.931 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/27' 2023-10-10T18:51:53.931 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/31' 2023-10-10T18:51:53.931 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/36' 2023-10-10T18:51:53.931 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/20' 2023-10-10T18:51:53.931 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/39' 2023-10-10T18:51:53.932 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/34' 2023-10-10T18:51:53.932 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/38' 2023-10-10T18:51:53.932 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/28' 2023-10-10T18:51:53.932 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/24' 2023-10-10T18:51:53.932 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/29' 2023-10-10T18:51:53.932 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/32' 2023-10-10T18:51:53.933 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/22' 2023-10-10T18:51:53.933 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/25' 2023-10-10T18:51:53.933 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/30' 2023-10-10T18:51:53.933 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/33' 2023-10-10T18:51:53.933 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/23' 2023-10-10T18:51:53.934 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/35' 2023-10-10T18:51:53.934 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/26' 2023-10-10T18:51:53.934 INFO:tasks.workunit.client.0.smithi175.stdout:removed './270/37' 2023-10-10T18:51:53.934 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './270' 2023-10-10T18:51:53.934 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/21' 2023-10-10T18:51:53.935 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/27' 2023-10-10T18:51:53.935 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/31' 2023-10-10T18:51:53.935 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/36' 2023-10-10T18:51:53.935 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/20' 2023-10-10T18:51:53.935 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/39' 2023-10-10T18:51:53.936 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/34' 2023-10-10T18:51:53.936 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/38' 2023-10-10T18:51:53.936 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/28' 2023-10-10T18:51:53.936 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/24' 2023-10-10T18:51:53.936 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/29' 2023-10-10T18:51:53.937 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/32' 2023-10-10T18:51:53.937 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/22' 2023-10-10T18:51:53.937 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/25' 2023-10-10T18:51:53.937 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/30' 2023-10-10T18:51:53.937 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/33' 2023-10-10T18:51:53.938 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/23' 2023-10-10T18:51:53.938 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/35' 2023-10-10T18:51:53.938 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/26' 2023-10-10T18:51:53.938 INFO:tasks.workunit.client.0.smithi175.stdout:removed './271/37' 2023-10-10T18:51:53.938 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './271' 2023-10-10T18:51:53.938 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/21' 2023-10-10T18:51:53.939 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/27' 2023-10-10T18:51:53.939 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/31' 2023-10-10T18:51:53.939 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/36' 2023-10-10T18:51:53.939 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/20' 2023-10-10T18:51:53.939 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/39' 2023-10-10T18:51:53.940 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/34' 2023-10-10T18:51:53.940 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/38' 2023-10-10T18:51:53.940 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/28' 2023-10-10T18:51:53.940 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/24' 2023-10-10T18:51:53.940 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/29' 2023-10-10T18:51:53.940 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/32' 2023-10-10T18:51:53.941 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/22' 2023-10-10T18:51:53.941 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/25' 2023-10-10T18:51:53.941 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/30' 2023-10-10T18:51:53.941 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/33' 2023-10-10T18:51:53.941 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/23' 2023-10-10T18:51:53.942 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/35' 2023-10-10T18:51:53.942 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/26' 2023-10-10T18:51:53.942 INFO:tasks.workunit.client.0.smithi175.stdout:removed './272/37' 2023-10-10T18:51:53.942 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './272' 2023-10-10T18:51:53.942 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/21' 2023-10-10T18:51:53.943 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/27' 2023-10-10T18:51:53.943 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/31' 2023-10-10T18:51:53.943 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/36' 2023-10-10T18:51:53.943 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/20' 2023-10-10T18:51:53.943 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/39' 2023-10-10T18:51:53.944 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/34' 2023-10-10T18:51:53.944 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/38' 2023-10-10T18:51:53.944 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/28' 2023-10-10T18:51:53.944 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/24' 2023-10-10T18:51:53.944 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/29' 2023-10-10T18:51:53.944 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/32' 2023-10-10T18:51:53.945 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/22' 2023-10-10T18:51:53.945 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/25' 2023-10-10T18:51:53.945 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/30' 2023-10-10T18:51:53.945 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/33' 2023-10-10T18:51:53.945 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/23' 2023-10-10T18:51:53.946 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/35' 2023-10-10T18:51:53.946 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/26' 2023-10-10T18:51:53.946 INFO:tasks.workunit.client.0.smithi175.stdout:removed './273/37' 2023-10-10T18:51:53.946 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './273' 2023-10-10T18:51:53.947 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/21' 2023-10-10T18:51:53.947 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/27' 2023-10-10T18:51:53.947 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/31' 2023-10-10T18:51:53.947 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/36' 2023-10-10T18:51:53.947 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/20' 2023-10-10T18:51:53.948 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/39' 2023-10-10T18:51:53.948 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/34' 2023-10-10T18:51:53.948 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/38' 2023-10-10T18:51:53.948 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/28' 2023-10-10T18:51:53.948 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/24' 2023-10-10T18:51:53.949 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/29' 2023-10-10T18:51:53.949 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/32' 2023-10-10T18:51:53.949 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/22' 2023-10-10T18:51:53.949 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/25' 2023-10-10T18:51:53.949 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/30' 2023-10-10T18:51:53.950 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/33' 2023-10-10T18:51:53.950 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/23' 2023-10-10T18:51:53.950 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/35' 2023-10-10T18:51:53.950 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/26' 2023-10-10T18:51:53.950 INFO:tasks.workunit.client.0.smithi175.stdout:removed './274/37' 2023-10-10T18:51:53.951 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './274' 2023-10-10T18:51:53.951 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/21' 2023-10-10T18:51:53.951 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/27' 2023-10-10T18:51:53.951 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/31' 2023-10-10T18:51:53.951 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/36' 2023-10-10T18:51:53.952 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/20' 2023-10-10T18:51:53.952 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/39' 2023-10-10T18:51:53.952 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/34' 2023-10-10T18:51:53.952 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/38' 2023-10-10T18:51:53.952 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/28' 2023-10-10T18:51:53.953 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/24' 2023-10-10T18:51:53.953 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/29' 2023-10-10T18:51:53.953 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/32' 2023-10-10T18:51:53.953 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/22' 2023-10-10T18:51:53.953 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/25' 2023-10-10T18:51:53.954 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/30' 2023-10-10T18:51:53.954 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/33' 2023-10-10T18:51:53.954 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/23' 2023-10-10T18:51:53.954 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/35' 2023-10-10T18:51:53.954 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/26' 2023-10-10T18:51:53.955 INFO:tasks.workunit.client.0.smithi175.stdout:removed './275/37' 2023-10-10T18:51:53.955 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './275' 2023-10-10T18:51:53.955 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/21' 2023-10-10T18:51:53.955 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/27' 2023-10-10T18:51:53.956 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/31' 2023-10-10T18:51:53.956 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/36' 2023-10-10T18:51:53.956 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/20' 2023-10-10T18:51:53.956 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/39' 2023-10-10T18:51:53.956 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/34' 2023-10-10T18:51:53.957 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/38' 2023-10-10T18:51:53.957 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/28' 2023-10-10T18:51:53.957 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/24' 2023-10-10T18:51:53.957 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/29' 2023-10-10T18:51:53.957 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/32' 2023-10-10T18:51:53.958 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/22' 2023-10-10T18:51:53.958 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/25' 2023-10-10T18:51:53.958 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/30' 2023-10-10T18:51:53.958 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/33' 2023-10-10T18:51:53.958 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/23' 2023-10-10T18:51:53.959 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/35' 2023-10-10T18:51:53.959 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/26' 2023-10-10T18:51:56.586 INFO:tasks.workunit.client.0.smithi175.stdout:removed './276/37' 2023-10-10T18:51:56.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './276' 2023-10-10T18:51:56.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/21' 2023-10-10T18:51:56.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/27' 2023-10-10T18:51:56.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/31' 2023-10-10T18:51:56.587 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/36' 2023-10-10T18:51:56.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/20' 2023-10-10T18:51:56.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/39' 2023-10-10T18:51:56.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/34' 2023-10-10T18:51:56.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/38' 2023-10-10T18:51:56.588 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/28' 2023-10-10T18:51:56.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/24' 2023-10-10T18:51:56.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/29' 2023-10-10T18:51:56.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/32' 2023-10-10T18:51:56.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/22' 2023-10-10T18:51:56.589 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/25' 2023-10-10T18:51:56.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/30' 2023-10-10T18:51:56.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/33' 2023-10-10T18:51:56.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/23' 2023-10-10T18:51:56.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/35' 2023-10-10T18:51:56.590 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/26' 2023-10-10T18:51:56.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './277/37' 2023-10-10T18:51:56.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './277' 2023-10-10T18:51:56.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/21' 2023-10-10T18:51:56.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/27' 2023-10-10T18:51:56.591 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/31' 2023-10-10T18:51:56.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/36' 2023-10-10T18:51:56.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/20' 2023-10-10T18:51:56.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/39' 2023-10-10T18:51:56.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/34' 2023-10-10T18:51:56.592 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/38' 2023-10-10T18:51:56.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/28' 2023-10-10T18:51:56.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/24' 2023-10-10T18:51:56.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/29' 2023-10-10T18:51:56.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/32' 2023-10-10T18:51:56.593 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/22' 2023-10-10T18:51:56.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/25' 2023-10-10T18:51:56.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/30' 2023-10-10T18:51:56.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/33' 2023-10-10T18:51:56.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/23' 2023-10-10T18:51:56.594 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/35' 2023-10-10T18:51:56.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/26' 2023-10-10T18:51:56.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './278/37' 2023-10-10T18:51:56.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './278' 2023-10-10T18:51:56.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/21' 2023-10-10T18:51:56.595 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/27' 2023-10-10T18:51:56.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/31' 2023-10-10T18:51:56.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/36' 2023-10-10T18:51:56.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/20' 2023-10-10T18:51:56.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/39' 2023-10-10T18:51:56.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/34' 2023-10-10T18:51:56.596 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/38' 2023-10-10T18:51:56.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/28' 2023-10-10T18:51:56.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/24' 2023-10-10T18:51:56.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/29' 2023-10-10T18:51:56.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/32' 2023-10-10T18:51:56.597 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/22' 2023-10-10T18:51:56.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/25' 2023-10-10T18:51:56.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/30' 2023-10-10T18:51:56.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/33' 2023-10-10T18:51:56.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/23' 2023-10-10T18:51:56.598 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/35' 2023-10-10T18:51:56.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/26' 2023-10-10T18:51:56.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed './279/37' 2023-10-10T18:51:56.599 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './279' 2023-10-10T18:51:56.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/21' 2023-10-10T18:51:56.600 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/27' 2023-10-10T18:51:56.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/31' 2023-10-10T18:51:56.601 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/36' 2023-10-10T18:51:56.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/20' 2023-10-10T18:51:56.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/39' 2023-10-10T18:51:56.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/34' 2023-10-10T18:51:56.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/38' 2023-10-10T18:51:56.602 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/28' 2023-10-10T18:51:56.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/24' 2023-10-10T18:51:56.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/29' 2023-10-10T18:51:56.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/32' 2023-10-10T18:51:56.603 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/22' 2023-10-10T18:51:56.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/25' 2023-10-10T18:51:56.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/30' 2023-10-10T18:51:56.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/33' 2023-10-10T18:51:56.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/23' 2023-10-10T18:51:56.604 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/35' 2023-10-10T18:51:56.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/26' 2023-10-10T18:51:56.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './280/37' 2023-10-10T18:51:56.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './280' 2023-10-10T18:51:56.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/21' 2023-10-10T18:51:56.605 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/27' 2023-10-10T18:51:56.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/31' 2023-10-10T18:51:56.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/36' 2023-10-10T18:51:56.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/20' 2023-10-10T18:51:56.606 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/39' 2023-10-10T18:51:56.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/34' 2023-10-10T18:51:56.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/38' 2023-10-10T18:51:56.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/28' 2023-10-10T18:51:56.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/24' 2023-10-10T18:51:56.607 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/29' 2023-10-10T18:51:56.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/32' 2023-10-10T18:51:56.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/22' 2023-10-10T18:51:56.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/25' 2023-10-10T18:51:56.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/30' 2023-10-10T18:51:56.608 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/33' 2023-10-10T18:51:56.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/23' 2023-10-10T18:51:56.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/35' 2023-10-10T18:51:56.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/26' 2023-10-10T18:51:56.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed './281/37' 2023-10-10T18:51:56.609 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './281' 2023-10-10T18:51:56.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/21' 2023-10-10T18:51:56.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/27' 2023-10-10T18:51:56.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/31' 2023-10-10T18:51:56.610 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/36' 2023-10-10T18:51:56.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/20' 2023-10-10T18:51:56.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/39' 2023-10-10T18:51:56.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/34' 2023-10-10T18:51:56.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/38' 2023-10-10T18:51:56.611 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/28' 2023-10-10T18:51:56.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/24' 2023-10-10T18:51:56.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/29' 2023-10-10T18:51:56.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/32' 2023-10-10T18:51:56.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/22' 2023-10-10T18:51:56.612 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/25' 2023-10-10T18:51:56.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/30' 2023-10-10T18:51:56.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/33' 2023-10-10T18:51:56.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/23' 2023-10-10T18:51:56.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/35' 2023-10-10T18:51:56.613 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/26' 2023-10-10T18:51:56.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed './282/37' 2023-10-10T18:51:56.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './282' 2023-10-10T18:51:56.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/21' 2023-10-10T18:51:56.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/27' 2023-10-10T18:51:56.614 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/31' 2023-10-10T18:51:56.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/36' 2023-10-10T18:51:56.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/20' 2023-10-10T18:51:56.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/39' 2023-10-10T18:51:56.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/34' 2023-10-10T18:51:56.615 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/38' 2023-10-10T18:51:56.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/28' 2023-10-10T18:51:56.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/24' 2023-10-10T18:51:56.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/29' 2023-10-10T18:51:56.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/32' 2023-10-10T18:51:56.616 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/22' 2023-10-10T18:51:56.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/25' 2023-10-10T18:51:56.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/30' 2023-10-10T18:51:56.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/33' 2023-10-10T18:51:56.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/23' 2023-10-10T18:51:56.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/35' 2023-10-10T18:51:56.617 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/26' 2023-10-10T18:51:56.618 INFO:tasks.workunit.client.0.smithi175.stdout:removed './283/37' 2023-10-10T18:51:56.618 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './283' 2023-10-10T18:51:56.618 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/21' 2023-10-10T18:51:56.618 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/27' 2023-10-10T18:51:56.618 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/31' 2023-10-10T18:51:56.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/36' 2023-10-10T18:51:56.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/20' 2023-10-10T18:51:56.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/39' 2023-10-10T18:51:56.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/34' 2023-10-10T18:51:56.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/38' 2023-10-10T18:51:56.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/28' 2023-10-10T18:51:56.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/24' 2023-10-10T18:51:56.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/29' 2023-10-10T18:51:56.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/32' 2023-10-10T18:51:56.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/22' 2023-10-10T18:51:56.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/25' 2023-10-10T18:51:56.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/30' 2023-10-10T18:51:56.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/33' 2023-10-10T18:51:56.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/23' 2023-10-10T18:51:56.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/35' 2023-10-10T18:51:56.622 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/26' 2023-10-10T18:51:56.622 INFO:tasks.workunit.client.0.smithi175.stdout:removed './284/37' 2023-10-10T18:51:56.622 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './284' 2023-10-10T18:51:56.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/21' 2023-10-10T18:51:56.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/27' 2023-10-10T18:51:56.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/31' 2023-10-10T18:51:56.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/36' 2023-10-10T18:51:56.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/20' 2023-10-10T18:51:56.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/39' 2023-10-10T18:51:56.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/34' 2023-10-10T18:51:56.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/38' 2023-10-10T18:51:56.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/28' 2023-10-10T18:51:56.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/24' 2023-10-10T18:51:56.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/29' 2023-10-10T18:51:56.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/32' 2023-10-10T18:51:56.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/22' 2023-10-10T18:51:56.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/25' 2023-10-10T18:51:56.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/30' 2023-10-10T18:51:56.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/33' 2023-10-10T18:51:56.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/23' 2023-10-10T18:51:56.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/35' 2023-10-10T18:51:56.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/26' 2023-10-10T18:51:56.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './285/37' 2023-10-10T18:51:56.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './285' 2023-10-10T18:51:56.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/21' 2023-10-10T18:51:56.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/27' 2023-10-10T18:51:56.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/31' 2023-10-10T18:51:56.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/36' 2023-10-10T18:51:56.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/20' 2023-10-10T18:51:56.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/39' 2023-10-10T18:51:56.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/34' 2023-10-10T18:51:56.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/38' 2023-10-10T18:51:56.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/28' 2023-10-10T18:51:56.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/24' 2023-10-10T18:51:56.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/29' 2023-10-10T18:51:56.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/32' 2023-10-10T18:51:56.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/22' 2023-10-10T18:51:56.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/25' 2023-10-10T18:51:56.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/30' 2023-10-10T18:51:56.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/33' 2023-10-10T18:51:56.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/23' 2023-10-10T18:51:56.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/35' 2023-10-10T18:51:56.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/26' 2023-10-10T18:51:56.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './286/37' 2023-10-10T18:51:56.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './286' 2023-10-10T18:52:05.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/21' 2023-10-10T18:52:05.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/27' 2023-10-10T18:52:05.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/31' 2023-10-10T18:52:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/36' 2023-10-10T18:52:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/20' 2023-10-10T18:52:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/39' 2023-10-10T18:52:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/34' 2023-10-10T18:52:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/38' 2023-10-10T18:52:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/28' 2023-10-10T18:52:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/24' 2023-10-10T18:52:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/29' 2023-10-10T18:52:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/32' 2023-10-10T18:52:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/22' 2023-10-10T18:52:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/25' 2023-10-10T18:52:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/30' 2023-10-10T18:52:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/33' 2023-10-10T18:52:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/23' 2023-10-10T18:52:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/35' 2023-10-10T18:52:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/26' 2023-10-10T18:52:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './287/37' 2023-10-10T18:52:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './287' 2023-10-10T18:52:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/21' 2023-10-10T18:52:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/27' 2023-10-10T18:52:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/31' 2023-10-10T18:52:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/36' 2023-10-10T18:52:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/20' 2023-10-10T18:52:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/39' 2023-10-10T18:52:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/34' 2023-10-10T18:52:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/38' 2023-10-10T18:52:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/28' 2023-10-10T18:52:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/24' 2023-10-10T18:52:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/29' 2023-10-10T18:52:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/32' 2023-10-10T18:52:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/22' 2023-10-10T18:52:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/25' 2023-10-10T18:52:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/30' 2023-10-10T18:52:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/33' 2023-10-10T18:52:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/23' 2023-10-10T18:52:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/35' 2023-10-10T18:52:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/26' 2023-10-10T18:52:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './288/37' 2023-10-10T18:52:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './288' 2023-10-10T18:52:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/21' 2023-10-10T18:52:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/27' 2023-10-10T18:52:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/31' 2023-10-10T18:52:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/36' 2023-10-10T18:52:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/20' 2023-10-10T18:52:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/39' 2023-10-10T18:52:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/34' 2023-10-10T18:52:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/38' 2023-10-10T18:52:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/28' 2023-10-10T18:52:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/24' 2023-10-10T18:52:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/29' 2023-10-10T18:52:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/32' 2023-10-10T18:52:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/22' 2023-10-10T18:52:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/25' 2023-10-10T18:52:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/30' 2023-10-10T18:52:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/33' 2023-10-10T18:52:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/23' 2023-10-10T18:52:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/35' 2023-10-10T18:52:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/26' 2023-10-10T18:52:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './289/37' 2023-10-10T18:52:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './289' 2023-10-10T18:52:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/21' 2023-10-10T18:52:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/27' 2023-10-10T18:52:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/31' 2023-10-10T18:52:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/36' 2023-10-10T18:52:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/20' 2023-10-10T18:52:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/39' 2023-10-10T18:52:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/34' 2023-10-10T18:52:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/38' 2023-10-10T18:52:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/28' 2023-10-10T18:52:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/24' 2023-10-10T18:52:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/29' 2023-10-10T18:52:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/32' 2023-10-10T18:52:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/22' 2023-10-10T18:52:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/25' 2023-10-10T18:52:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/30' 2023-10-10T18:52:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/33' 2023-10-10T18:52:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/23' 2023-10-10T18:52:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/35' 2023-10-10T18:52:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/26' 2023-10-10T18:52:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './290/37' 2023-10-10T18:52:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './290' 2023-10-10T18:52:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/21' 2023-10-10T18:52:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/27' 2023-10-10T18:52:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/31' 2023-10-10T18:52:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/36' 2023-10-10T18:52:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/20' 2023-10-10T18:52:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/39' 2023-10-10T18:52:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/34' 2023-10-10T18:52:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/38' 2023-10-10T18:52:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/28' 2023-10-10T18:52:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/24' 2023-10-10T18:52:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/29' 2023-10-10T18:52:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/32' 2023-10-10T18:52:05.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/22' 2023-10-10T18:52:05.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/25' 2023-10-10T18:52:05.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/30' 2023-10-10T18:52:05.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/33' 2023-10-10T18:52:05.720 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/23' 2023-10-10T18:52:05.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/35' 2023-10-10T18:52:05.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/26' 2023-10-10T18:52:05.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed './291/37' 2023-10-10T18:52:05.721 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './291' 2023-10-10T18:52:05.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/21' 2023-10-10T18:52:05.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/27' 2023-10-10T18:52:05.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/31' 2023-10-10T18:52:05.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/36' 2023-10-10T18:52:05.722 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/20' 2023-10-10T18:52:05.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/39' 2023-10-10T18:52:05.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/34' 2023-10-10T18:52:05.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/38' 2023-10-10T18:52:05.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/28' 2023-10-10T18:52:05.723 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/24' 2023-10-10T18:52:05.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/29' 2023-10-10T18:52:05.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/32' 2023-10-10T18:52:05.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/22' 2023-10-10T18:52:05.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/25' 2023-10-10T18:52:05.724 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/30' 2023-10-10T18:52:05.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/33' 2023-10-10T18:52:05.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/23' 2023-10-10T18:52:05.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/35' 2023-10-10T18:52:05.725 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/26' 2023-10-10T18:52:05.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './292/37' 2023-10-10T18:52:05.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './292' 2023-10-10T18:52:05.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/21' 2023-10-10T18:52:05.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/27' 2023-10-10T18:52:05.726 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/31' 2023-10-10T18:52:05.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/36' 2023-10-10T18:52:05.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/20' 2023-10-10T18:52:05.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/39' 2023-10-10T18:52:05.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/34' 2023-10-10T18:52:05.727 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/38' 2023-10-10T18:52:05.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/28' 2023-10-10T18:52:05.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/24' 2023-10-10T18:52:05.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/29' 2023-10-10T18:52:05.728 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/32' 2023-10-10T18:52:05.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/22' 2023-10-10T18:52:05.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/25' 2023-10-10T18:52:05.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/30' 2023-10-10T18:52:05.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/33' 2023-10-10T18:52:05.729 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/23' 2023-10-10T18:52:05.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/35' 2023-10-10T18:52:05.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/26' 2023-10-10T18:52:05.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './293/37' 2023-10-10T18:52:05.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './293' 2023-10-10T18:52:05.730 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/21' 2023-10-10T18:52:05.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/27' 2023-10-10T18:52:05.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/31' 2023-10-10T18:52:05.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/36' 2023-10-10T18:52:05.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/20' 2023-10-10T18:52:05.731 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/39' 2023-10-10T18:52:05.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/34' 2023-10-10T18:52:05.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/38' 2023-10-10T18:52:05.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/28' 2023-10-10T18:52:05.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/24' 2023-10-10T18:52:05.732 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/29' 2023-10-10T18:52:05.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/32' 2023-10-10T18:52:05.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/22' 2023-10-10T18:52:05.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/25' 2023-10-10T18:52:05.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/30' 2023-10-10T18:52:05.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/33' 2023-10-10T18:52:05.733 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/23' 2023-10-10T18:52:05.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/35' 2023-10-10T18:52:05.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/26' 2023-10-10T18:52:05.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './294/37' 2023-10-10T18:52:05.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './294' 2023-10-10T18:52:05.734 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/21' 2023-10-10T18:52:05.735 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/27' 2023-10-10T18:52:05.735 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/31' 2023-10-10T18:52:05.735 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/36' 2023-10-10T18:52:05.735 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/20' 2023-10-10T18:52:05.735 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/39' 2023-10-10T18:52:05.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/34' 2023-10-10T18:52:05.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/38' 2023-10-10T18:52:05.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/28' 2023-10-10T18:52:05.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/24' 2023-10-10T18:52:05.736 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/29' 2023-10-10T18:52:05.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/32' 2023-10-10T18:52:05.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/22' 2023-10-10T18:52:05.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/25' 2023-10-10T18:52:05.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/30' 2023-10-10T18:52:05.737 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/33' 2023-10-10T18:52:05.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/23' 2023-10-10T18:52:05.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/35' 2023-10-10T18:52:05.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/26' 2023-10-10T18:52:05.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed './295/37' 2023-10-10T18:52:05.738 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './295' 2023-10-10T18:52:05.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/21' 2023-10-10T18:52:05.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/27' 2023-10-10T18:52:05.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/31' 2023-10-10T18:52:05.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/36' 2023-10-10T18:52:05.739 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/20' 2023-10-10T18:52:05.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/39' 2023-10-10T18:52:05.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/34' 2023-10-10T18:52:05.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/38' 2023-10-10T18:52:05.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/28' 2023-10-10T18:52:05.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/24' 2023-10-10T18:52:05.740 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/29' 2023-10-10T18:52:05.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/32' 2023-10-10T18:52:05.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/22' 2023-10-10T18:52:05.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/25' 2023-10-10T18:52:05.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/30' 2023-10-10T18:52:05.741 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/33' 2023-10-10T18:52:05.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/23' 2023-10-10T18:52:05.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/35' 2023-10-10T18:52:05.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/26' 2023-10-10T18:52:05.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed './296/37' 2023-10-10T18:52:05.742 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './296' 2023-10-10T18:52:05.743 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/21' 2023-10-10T18:52:05.743 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/27' 2023-10-10T18:52:08.606 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:08.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.732+0000 7efe3759e700 1 Processor -- start 2023-10-10T18:52:08.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.732+0000 7efe3759e700 1 -- start start 2023-10-10T18:52:08.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe3759e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efe3814fd90 0x7efe38159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:08.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe3759e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe380cd040 0x7efe380684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:08.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe3759e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe38067fa0 con 0x7efe3808a100 2023-10-10T18:52:08.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe3759e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe380c5600 con 0x7efe3814fd90 2023-10-10T18:52:08.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe3759e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe38156b70 con 0x7efe380cd040 2023-10-10T18:52:08.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe35d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7efe3808a100 0x7efe380c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:34802/0 (socket says 172.21.15.175:34802) 2023-10-10T18:52:08.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe35d9b700 1 -- 172.21.15.175:0/3693212815 learned_addr learned my addr 172.21.15.175:0/3693212815 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:08.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe36d9d700 1 --2- 172.21.15.175:0/3693212815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe380cd040 0x7efe380684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:08.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe3659c700 1 --2- 172.21.15.175:0/3693212815 >> v2:172.21.15.191:3300/0 conn(0x7efe3814fd90 0x7efe38159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe3559a700 1 -- 172.21.15.175:0/3693212815 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822973405 0 0) 0x7efe38067fa0 con 0x7efe3808a100 2023-10-10T18:52:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe3559a700 1 -- 172.21.15.175:0/3693212815 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe24003570 con 0x7efe3808a100 2023-10-10T18:52:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe36d9d700 1 -- 172.21.15.175:0/3693212815 >> v2:172.21.15.191:3300/0 conn(0x7efe3814fd90 msgr2=0x7efe38159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe36d9d700 1 --2- 172.21.15.175:0/3693212815 >> v2:172.21.15.191:3300/0 conn(0x7efe3814fd90 0x7efe38159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe36d9d700 1 -- 172.21.15.175:0/3693212815 >> v1:172.21.15.175:6789/0 conn(0x7efe3808a100 legacy=0x7efe380c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.733+0000 7efe36d9d700 1 -- 172.21.15.175:0/3693212815 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe38150160 con 0x7efe380cd040 2023-10-10T18:52:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3659c700 1 --2- 172.21.15.175:0/3693212815 >> v2:172.21.15.191:3300/0 conn(0x7efe3814fd90 0x7efe38159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:52:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe36d9d700 1 --2- 172.21.15.175:0/3693212815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe380cd040 0x7efe380684f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efe280098e0 tx=0x7efe280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e02dfc25555589c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3559a700 1 -- 172.21.15.175:0/3693212815 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe28011630 con 0x7efe380cd040 2023-10-10T18:52:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3559a700 1 -- 172.21.15.175:0/3693212815 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe28011790 con 0x7efe380cd040 2023-10-10T18:52:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3559a700 1 -- 172.21.15.175:0/3693212815 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe28011b10 con 0x7efe380cd040 2023-10-10T18:52:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3759e700 1 -- 172.21.15.175:0/3693212815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe380cd040 msgr2=0x7efe380684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3759e700 1 --2- 172.21.15.175:0/3693212815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe380cd040 0x7efe380684f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efe280098e0 tx=0x7efe280091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3759e700 1 -- 172.21.15.175:0/3693212815 shutdown_connections 2023-10-10T18:52:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3759e700 1 --2- 172.21.15.175:0/3693212815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe380cd040 0x7efe380684f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3759e700 1 --2- 172.21.15.175:0/3693212815 >> v2:172.21.15.191:3300/0 conn(0x7efe3814fd90 0x7efe38159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.734+0000 7efe3759e700 1 -- 172.21.15.175:0/3693212815 >> 172.21.15.175:0/3693212815 conn(0x7efe38063d00 msgr2=0x7efe380cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.735+0000 7efe3759e700 1 -- 172.21.15.175:0/3693212815 shutdown_connections 2023-10-10T18:52:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.735+0000 7efe3759e700 1 -- 172.21.15.175:0/3693212815 wait complete. 2023-10-10T18:52:08.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3759e700 1 Processor -- start 2023-10-10T18:52:08.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3759e700 1 -- start start 2023-10-10T18:52:08.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3808a100 0x7efe380ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:08.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3759e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe380cd040 0x7efe3815b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3759e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe3814fd90 0x7efe38168c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3759e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38153380 con 0x7efe3808a100 2023-10-10T18:52:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3759e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38153640 con 0x7efe380cd040 2023-10-10T18:52:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3759e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe38151ba0 con 0x7efe3814fd90 2023-10-10T18:52:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.736+0000 7efe3659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3808a100 0x7efe380ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3808a100 0x7efe380ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52962/0 (socket says 172.21.15.175:52962) 2023-10-10T18:52:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3659c700 1 -- 172.21.15.175:0/2078888878 learned_addr learned my addr 172.21.15.175:0/2078888878 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe36d9d700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe3814fd90 0x7efe38168c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe35d9b700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe380cd040 0x7efe3815b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3659c700 1 -- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe3814fd90 msgr2=0x7efe38168c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3659c700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe3814fd90 0x7efe38168c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3659c700 1 -- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe380cd040 msgr2=0x7efe3815b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3659c700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe380cd040 0x7efe3815b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3659c700 1 -- 172.21.15.175:0/2078888878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe30009a20 con 0x7efe3808a100 2023-10-10T18:52:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3659c700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3808a100 0x7efe380ca420 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c0c0 tx=0x7efe3000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe2e7fc700 1 -- 172.21.15.175:0/2078888878 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3000cc90 con 0x7efe3808a100 2023-10-10T18:52:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe2e7fc700 1 -- 172.21.15.175:0/2078888878 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3000f040 con 0x7efe3808a100 2023-10-10T18:52:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe2e7fc700 1 -- 172.21.15.175:0/2078888878 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe30014480 con 0x7efe3808a100 2023-10-10T18:52:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe28009420 con 0x7efe3808a100 2023-10-10T18:52:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.737+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe38152160 con 0x7efe3808a100 2023-10-10T18:52:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.738+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe38067570 con 0x7efe3808a100 2023-10-10T18:52:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.738+0000 7efe2e7fc700 1 -- 172.21.15.175:0/2078888878 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efe300147f0 con 0x7efe3808a100 2023-10-10T18:52:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.739+0000 7efe2e7fc700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efe18066f60 0x7efe18069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.739+0000 7efe2e7fc700 1 -- 172.21.15.175:0/2078888878 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6549+0+0 (secure 0 0 0) 0x7efe30085610 con 0x7efe3808a100 2023-10-10T18:52:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.739+0000 7efe35d9b700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efe18066f60 0x7efe18069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:08.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.739+0000 7efe35d9b700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efe18066f60 0x7efe18069410 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe380cc090 tx=0x7efe20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:08.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.743+0000 7efe2e7fc700 1 -- 172.21.15.175:0/2078888878 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe30052530 con 0x7efe3808a100 2023-10-10T18:52:08.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.967+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe38155f80 con 0x7efe3808a100 2023-10-10T18:52:08.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.967+0000 7efe2e7fc700 1 -- 172.21.15.175:0/2078888878 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6214 (secure 0 0 0) 0x7efe30055b50 con 0x7efe3808a100 2023-10-10T18:52:08.969 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:08.969 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:51:57.583479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:08.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.969+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efe18066f60 msgr2=0x7efe18069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:08.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.969+0000 7efe3759e700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efe18066f60 0x7efe18069410 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efe380cc090 tx=0x7efe20009040 comp rx=0 tx=0).stop 2023-10-10T18:52:08.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3808a100 msgr2=0x7efe380ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:08.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3808a100 0x7efe380ca420 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c0c0 tx=0x7efe3000b040 comp rx=0 tx=0).stop 2023-10-10T18:52:08.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 shutdown_connections 2023-10-10T18:52:08.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efe18066f60 0x7efe18069410 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe3814fd90 0x7efe38168c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3808a100 0x7efe380ca420 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 --2- 172.21.15.175:0/2078888878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe380cd040 0x7efe3815b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:08.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 >> 172.21.15.175:0/2078888878 conn(0x7efe38063d00 msgr2=0x7efe380684f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:08.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 shutdown_connections 2023-10-10T18:52:08.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:08.970+0000 7efe3759e700 1 -- 172.21.15.175:0/2078888878 wait complete. 2023-10-10T18:52:08.973 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 56 2023-10-10T18:52:08.982 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=3) 2023-10-10T18:52:08.983 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-10T18:52:08.983 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T18:52:08.993 INFO:tasks.ceph.mds.c:Stopped 2023-10-10T18:52:08.993 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:09.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.146+0000 7f87c1649700 1 Processor -- start 2023-10-10T18:52:09.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.146+0000 7f87c1649700 1 -- start start 2023-10-10T18:52:09.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.146+0000 7f87c1649700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 0x7f87bc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:09.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.146+0000 7f87c1649700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f87bc0b2390 0x7f87bc068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:09.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.146+0000 7f87c1649700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87bc1574b0 con 0x7f87bc068bd0 2023-10-10T18:52:09.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.146+0000 7f87c1649700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc150770 con 0x7f87bc0b2390 2023-10-10T18:52:09.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.146+0000 7f87c1649700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc0afad0 con 0x7f87bc0b1fc0 2023-10-10T18:52:09.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87ba7fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f87bc0b2390 0x7f87bc068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:09.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87bb7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f87bc068bd0 0x7f87bc0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:34810/0 (socket says 172.21.15.175:34810) 2023-10-10T18:52:09.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87bb7fe700 1 -- 172.21.15.175:0/2499925894 learned_addr learned my addr 172.21.15.175:0/2499925894 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:09.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87baffd700 1 --2- 172.21.15.175:0/2499925894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 0x7f87bc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:09.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87b9ffb700 1 -- 172.21.15.175:0/2499925894 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253996447 0 0) 0x7f87bc1574b0 con 0x7f87bc068bd0 2023-10-10T18:52:09.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87b9ffb700 1 -- 172.21.15.175:0/2499925894 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a8003570 con 0x7f87bc068bd0 2023-10-10T18:52:09.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87ba7fc700 1 -- 172.21.15.175:0/2499925894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 msgr2=0x7f87bc155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:09.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87ba7fc700 1 --2- 172.21.15.175:0/2499925894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 0x7f87bc155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87ba7fc700 1 -- 172.21.15.175:0/2499925894 >> v1:172.21.15.175:6789/0 conn(0x7f87bc068bd0 legacy=0x7f87bc0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:09.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87ba7fc700 1 -- 172.21.15.175:0/2499925894 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc0af4c0 con 0x7f87bc0b2390 2023-10-10T18:52:09.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87ba7fc700 1 --2- 172.21.15.175:0/2499925894 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc068690 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f87a40098e0 tx=0x7f87a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319c2e60c2e1c13a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:09.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87b9ffb700 1 -- 172.21.15.175:0/2499925894 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4011630 con 0x7f87bc0b2390 2023-10-10T18:52:09.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87b9ffb700 1 -- 172.21.15.175:0/2499925894 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a4011790 con 0x7f87bc0b2390 2023-10-10T18:52:09.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.147+0000 7f87b9ffb700 1 -- 172.21.15.175:0/2499925894 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4011b30 con 0x7f87bc0b2390 2023-10-10T18:52:09.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.148+0000 7f87c1649700 1 -- 172.21.15.175:0/2499925894 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 msgr2=0x7f87bc068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:09.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.148+0000 7f87c1649700 1 --2- 172.21.15.175:0/2499925894 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc068690 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f87a40098e0 tx=0x7f87a40091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.148+0000 7f87c1649700 1 -- 172.21.15.175:0/2499925894 shutdown_connections 2023-10-10T18:52:09.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.148+0000 7f87c1649700 1 --2- 172.21.15.175:0/2499925894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 0x7f87bc155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.148+0000 7f87c1649700 1 --2- 172.21.15.175:0/2499925894 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc068690 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.148+0000 7f87c1649700 1 -- 172.21.15.175:0/2499925894 >> 172.21.15.175:0/2499925894 conn(0x7f87bc063230 msgr2=0x7f87bc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:09.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.148+0000 7f87c1649700 1 -- 172.21.15.175:0/2499925894 shutdown_connections 2023-10-10T18:52:09.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.148+0000 7f87c1649700 1 -- 172.21.15.175:0/2499925894 wait complete. 2023-10-10T18:52:09.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.149+0000 7f87c1649700 1 Processor -- start 2023-10-10T18:52:09.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.149+0000 7f87c1649700 1 -- start start 2023-10-10T18:52:09.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87c1649700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc068bd0 0x7f87bc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:09.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87c1649700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 0x7f87bc150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:09.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87c1649700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:09.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87c1649700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc15a330 con 0x7f87bc0b2390 2023-10-10T18:52:09.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87c1649700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc0cc060 con 0x7f87bc068bd0 2023-10-10T18:52:09.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87c1649700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc0cc5e0 con 0x7f87bc0b1fc0 2023-10-10T18:52:09.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:09.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52990/0 (socket says 172.21.15.175:52990) 2023-10-10T18:52:09.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 -- 172.21.15.175:0/622810688 learned_addr learned my addr 172.21.15.175:0/622810688 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:09.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87ba7fc700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 0x7f87bc150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:09.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87baffd700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc068bd0 0x7f87bc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:09.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 -- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 msgr2=0x7f87bc150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:09.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 0x7f87bc150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 -- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc068bd0 msgr2=0x7f87bc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:09.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc068bd0 0x7f87bc0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 -- 172.21.15.175:0/622810688 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a4009420 con 0x7f87bc0b2390 2023-10-10T18:52:09.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87bb7fe700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc158510 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c790 tx=0x7f87b4007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:09.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87b2ffd700 1 -- 172.21.15.175:0/622810688 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4005c00 con 0x7f87bc0b2390 2023-10-10T18:52:09.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87b2ffd700 1 -- 172.21.15.175:0/622810688 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b4005d60 con 0x7f87bc0b2390 2023-10-10T18:52:09.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87bc159d70 con 0x7f87bc0b2390 2023-10-10T18:52:09.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.150+0000 7f87b2ffd700 1 -- 172.21.15.175:0/622810688 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b40072e0 con 0x7f87bc0b2390 2023-10-10T18:52:09.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.151+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87bc0cc270 con 0x7f87bc0b2390 2023-10-10T18:52:09.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.151+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87bc067570 con 0x7f87bc0b2390 2023-10-10T18:52:09.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.152+0000 7f87b2ffd700 1 -- 172.21.15.175:0/622810688 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f87b401b3b0 con 0x7f87bc0b2390 2023-10-10T18:52:09.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.152+0000 7f87b2ffd700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f879c066fb0 0x7f879c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:09.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.152+0000 7f87b2ffd700 1 -- 172.21.15.175:0/622810688 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6549+0+0 (secure 0 0 0) 0x7f87b4083d60 con 0x7f87bc0b2390 2023-10-10T18:52:09.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.152+0000 7f87baffd700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f879c066fb0 0x7f879c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:09.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.153+0000 7f87baffd700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f879c066fb0 0x7f879c069460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0076a0 tx=0x7f87ac00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:09.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.156+0000 7f87b2ffd700 1 -- 172.21.15.175:0/622810688 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87b404b5a0 con 0x7f87bc0b2390 2023-10-10T18:52:09.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.381+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87bc069060 con 0x7f87bc0b2390 2023-10-10T18:52:09.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.381+0000 7f87b2ffd700 1 -- 172.21.15.175:0/622810688 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6214 (secure 0 0 0) 0x7f87b4017020 con 0x7f87bc0b2390 2023-10-10T18:52:09.382 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:09.383 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:51:57.583479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:09.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f879c066fb0 msgr2=0x7f879c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:09.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f879c066fb0 0x7f879c069460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0076a0 tx=0x7f87ac00b040 comp rx=0 tx=0).stop 2023-10-10T18:52:09.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 msgr2=0x7f87bc158510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:09.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc158510 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c790 tx=0x7f87b4007d90 comp rx=0 tx=0).stop 2023-10-10T18:52:09.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 shutdown_connections 2023-10-10T18:52:09.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f879c066fb0 0x7f879c069460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87bc0b2390 0x7f87bc158510 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc068bd0 0x7f87bc0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 --2- 172.21.15.175:0/622810688 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87bc0b1fc0 0x7f87bc150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:09.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 >> 172.21.15.175:0/622810688 conn(0x7f87bc063230 msgr2=0x7f87bc1541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:09.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 shutdown_connections 2023-10-10T18:52:09.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:09.383+0000 7f87c1649700 1 -- 172.21.15.175:0/622810688 wait complete. 2023-10-10T18:52:09.387 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 56 2023-10-10T18:52:09.396 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T18:52:11.397 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:11.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.525+0000 7f86f60e9700 1 Processor -- start 2023-10-10T18:52:11.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.525+0000 7f86f60e9700 1 -- start start 2023-10-10T18:52:11.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f60e9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86f014fd90 0x7f86f0159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:11.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f60e9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f008a100 0x7f86f00c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:11.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f60e9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f0067fa0 con 0x7f86f00cd040 2023-10-10T18:52:11.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f60e9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f00c5600 con 0x7f86f014fd90 2023-10-10T18:52:11.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f60e9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0156b70 con 0x7f86f008a100 2023-10-10T18:52:11.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f48e6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f008a100 0x7f86f00c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:11.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f50e7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86f014fd90 0x7f86f0159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:11.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f50e7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86f014fd90 0x7f86f0159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:35572/0 (socket says 172.21.15.175:35572) 2023-10-10T18:52:11.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.526+0000 7f86f50e7700 1 -- 172.21.15.175:0/1355393340 learned_addr learned my addr 172.21.15.175:0/1355393340 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:11.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86f50e7700 1 -- 172.21.15.175:0/1355393340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f008a100 msgr2=0x7f86f00c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:11.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86f50e7700 1 --2- 172.21.15.175:0/1355393340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f008a100 0x7f86f00c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86f50e7700 1 -- 172.21.15.175:0/1355393340 >> v1:172.21.15.191:6789/0 conn(0x7f86f00cd040 legacy=0x7f86f00684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:11.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86f50e7700 1 -- 172.21.15.175:0/1355393340 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0150160 con 0x7f86f014fd90 2023-10-10T18:52:11.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86f48e6700 1 --2- 172.21.15.175:0/1355393340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f008a100 0x7f86f00c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:52:11.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86f50e7700 1 --2- 172.21.15.175:0/1355393340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 0x7f86f0159c30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f86ec009f20 tx=0x7f86ec009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72ce0b709f6e2e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:11.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86e7fff700 1 -- 172.21.15.175:0/1355393340 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00e070 con 0x7f86f014fd90 2023-10-10T18:52:11.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86e7fff700 1 -- 172.21.15.175:0/1355393340 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ec00b9d0 con 0x7f86f014fd90 2023-10-10T18:52:11.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.527+0000 7f86e7fff700 1 -- 172.21.15.175:0/1355393340 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec0092e0 con 0x7f86f014fd90 2023-10-10T18:52:11.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.528+0000 7f86f60e9700 1 -- 172.21.15.175:0/1355393340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 msgr2=0x7f86f0159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:11.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.528+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1355393340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 0x7f86f0159c30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f86ec009f20 tx=0x7f86ec009840 comp rx=0 tx=0).stop 2023-10-10T18:52:11.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.528+0000 7f86f60e9700 1 -- 172.21.15.175:0/1355393340 shutdown_connections 2023-10-10T18:52:11.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.528+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1355393340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 0x7f86f0159c30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.528+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1355393340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f008a100 0x7f86f00c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.528+0000 7f86f60e9700 1 -- 172.21.15.175:0/1355393340 >> 172.21.15.175:0/1355393340 conn(0x7f86f0063d00 msgr2=0x7f86f00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:11.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.528+0000 7f86f60e9700 1 -- 172.21.15.175:0/1355393340 shutdown_connections 2023-10-10T18:52:11.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.528+0000 7f86f60e9700 1 -- 172.21.15.175:0/1355393340 wait complete. 2023-10-10T18:52:11.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.529+0000 7f86f60e9700 1 Processor -- start 2023-10-10T18:52:11.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.529+0000 7f86f60e9700 1 -- start start 2023-10-10T18:52:11.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f60e9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86f008a100 0x7f86f00cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:11.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f60e9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f00cd040 0x7f86f0169620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:11.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f60e9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 0x7f86f0169b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:11.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f60e9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00c2fa0 con 0x7f86f008a100 2023-10-10T18:52:11.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f60e9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00c9f80 con 0x7f86f014fd90 2023-10-10T18:52:11.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f60e9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0169190 con 0x7f86f00cd040 2023-10-10T18:52:11.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86f008a100 0x7f86f00cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:11.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86f008a100 0x7f86f00cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53000/0 (socket says 172.21.15.175:53000) 2023-10-10T18:52:11.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 -- 172.21.15.175:0/1799742240 learned_addr learned my addr 172.21.15.175:0/1799742240 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:11.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f48e6700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f00cd040 0x7f86f0169620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:11.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f58e8700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 0x7f86f0169b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:11.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 -- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f00cd040 msgr2=0x7f86f0169620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:11.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f00cd040 0x7f86f0169620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 -- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 msgr2=0x7f86f0169b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:11.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 0x7f86f0169b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 -- 172.21.15.175:0/1799742240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e0009420 con 0x7f86f008a100 2023-10-10T18:52:11.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f48e6700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f00cd040 0x7f86f0169620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:11.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.530+0000 7f86f50e7700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86f008a100 0x7f86f00cc440 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00b610 tx=0x7f86ec018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:11.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.531+0000 7f86e57fa700 1 -- 172.21.15.175:0/1799742240 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec008a50 con 0x7f86f008a100 2023-10-10T18:52:11.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.531+0000 7f86e57fa700 1 -- 172.21.15.175:0/1799742240 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ec008bb0 con 0x7f86f008a100 2023-10-10T18:52:11.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.531+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec009ae0 con 0x7f86f008a100 2023-10-10T18:52:11.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.531+0000 7f86e57fa700 1 -- 172.21.15.175:0/1799742240 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec025460 con 0x7f86f008a100 2023-10-10T18:52:11.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.531+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f014ef60 con 0x7f86f008a100 2023-10-10T18:52:11.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.532+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f0067570 con 0x7f86f008a100 2023-10-10T18:52:11.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.532+0000 7f86e57fa700 1 -- 172.21.15.175:0/1799742240 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f86ec003040 con 0x7f86f008a100 2023-10-10T18:52:11.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.532+0000 7f86e57fa700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86d4066f60 0x7f86d4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:11.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.532+0000 7f86e57fa700 1 -- 172.21.15.175:0/1799742240 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6549+0+0 (secure 0 0 0) 0x7f86ec085d30 con 0x7f86f008a100 2023-10-10T18:52:11.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.533+0000 7f86f48e6700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86d4066f60 0x7f86d4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:11.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.533+0000 7f86f48e6700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86d4066f60 0x7f86d4069410 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f86f0159360 tx=0x7f86e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:11.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.537+0000 7f86e57fa700 1 -- 172.21.15.175:0/1799742240 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f86ec052d00 con 0x7f86f008a100 2023-10-10T18:52:11.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.765+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86f0155f80 con 0x7f86f008a100 2023-10-10T18:52:11.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.766+0000 7f86e57fa700 1 -- 172.21.15.175:0/1799742240 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6214 (secure 0 0 0) 0x7f86ec056320 con 0x7f86f008a100 2023-10-10T18:52:11.766 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:11.767 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:51:57.583479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:11.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86d4066f60 msgr2=0x7f86d4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:11.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86d4066f60 0x7f86d4069410 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f86f0159360 tx=0x7f86e0008040 comp rx=0 tx=0).stop 2023-10-10T18:52:11.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86f008a100 msgr2=0x7f86f00cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:11.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86f008a100 0x7f86f00cc440 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00b610 tx=0x7f86ec018000 comp rx=0 tx=0).stop 2023-10-10T18:52:11.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 shutdown_connections 2023-10-10T18:52:11.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86d4066f60 0x7f86d4069410 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86f014fd90 0x7f86f0169b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86f008a100 0x7f86f00cc440 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 --2- 172.21.15.175:0/1799742240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86f00cd040 0x7f86f0169620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:11.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 >> 172.21.15.175:0/1799742240 conn(0x7f86f0063d00 msgr2=0x7f86f014b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:11.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 shutdown_connections 2023-10-10T18:52:11.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:11.768+0000 7f86f60e9700 1 -- 172.21.15.175:0/1799742240 wait complete. 2023-10-10T18:52:11.771 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 56 2023-10-10T18:52:11.780 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T18:52:13.781 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:13.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.909+0000 7f5834fbf700 1 Processor -- start 2023-10-10T18:52:13.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.909+0000 7f5834fbf700 1 -- start start 2023-10-10T18:52:13.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.909+0000 7f5834fbf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b1fc0 0x7f5830155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:13.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f5834fbf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f58300aff60 0x7f58300b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:13.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f5834fbf700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58301574b0 con 0x7f58300b2390 2023-10-10T18:52:13.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f5834fbf700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5830150770 con 0x7f58300aff60 2023-10-10T18:52:13.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f5834fbf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58300afad0 con 0x7f58300b1fc0 2023-10-10T18:52:13.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f58300aff60 0x7f58300b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:13.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582dd9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f58300b2390 0x7f5830068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:34818/0 (socket says 172.21.15.175:34818) 2023-10-10T18:52:13.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582dd9b700 1 -- 172.21.15.175:0/1255058705 learned_addr learned my addr 172.21.15.175:0/1255058705 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:13.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582e59c700 1 --2- 172.21.15.175:0/1255058705 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b1fc0 0x7f5830155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:13.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582ed9d700 1 -- 172.21.15.175:0/1255058705 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b1fc0 msgr2=0x7f5830155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:13.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582ed9d700 1 --2- 172.21.15.175:0/1255058705 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b1fc0 0x7f5830155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:13.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582ed9d700 1 -- 172.21.15.175:0/1255058705 >> v1:172.21.15.175:6789/0 conn(0x7f58300b2390 legacy=0x7f5830068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:13.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582ed9d700 1 -- 172.21.15.175:0/1255058705 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58300af4c0 con 0x7f58300aff60 2023-10-10T18:52:13.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582d59a700 1 -- 172.21.15.175:0/1255058705 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988100701 0 0) 0x7f58301574b0 con 0x7f58300b2390 2023-10-10T18:52:13.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582ed9d700 1 --2- 172.21.15.175:0/1255058705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300aff60 0x7f58300b03e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5828009910 tx=0x7f58280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545bd2595f2ff780 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:13.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582d59a700 1 -- 172.21.15.175:0/1255058705 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582800b8b0 con 0x7f58300aff60 2023-10-10T18:52:13.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582d59a700 1 -- 172.21.15.175:0/1255058705 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582800ba10 con 0x7f58300aff60 2023-10-10T18:52:13.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.910+0000 7f582d59a700 1 -- 172.21.15.175:0/1255058705 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582800bdb0 con 0x7f58300aff60 2023-10-10T18:52:13.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.911+0000 7f5834fbf700 1 -- 172.21.15.175:0/1255058705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300aff60 msgr2=0x7f58300b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:13.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.911+0000 7f5834fbf700 1 --2- 172.21.15.175:0/1255058705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300aff60 0x7f58300b03e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5828009910 tx=0x7f58280091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:13.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.911+0000 7f5834fbf700 1 -- 172.21.15.175:0/1255058705 shutdown_connections 2023-10-10T18:52:13.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.911+0000 7f5834fbf700 1 --2- 172.21.15.175:0/1255058705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300aff60 0x7f58300b03e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:13.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.911+0000 7f5834fbf700 1 --2- 172.21.15.175:0/1255058705 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b1fc0 0x7f5830155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:13.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.911+0000 7f5834fbf700 1 -- 172.21.15.175:0/1255058705 >> 172.21.15.175:0/1255058705 conn(0x7f5830063230 msgr2=0x7f5830063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:13.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.911+0000 7f5834fbf700 1 -- 172.21.15.175:0/1255058705 shutdown_connections 2023-10-10T18:52:13.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.911+0000 7f5834fbf700 1 -- 172.21.15.175:0/1255058705 wait complete. 2023-10-10T18:52:13.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.912+0000 7f5834fbf700 1 Processor -- start 2023-10-10T18:52:13.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.912+0000 7f5834fbf700 1 -- start start 2023-10-10T18:52:13.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f5834fbf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58300aff60 0x7f5830158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:13.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f5834fbf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300b1fc0 0x7f58301529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:13.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f5834fbf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b2390 0x7f58300cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:13.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f5834fbf700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5830166cf0 con 0x7f58300b1fc0 2023-10-10T18:52:13.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f5834fbf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5830158730 con 0x7f58300aff60 2023-10-10T18:52:13.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f5834fbf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58300c4070 con 0x7f58300b2390 2023-10-10T18:52:13.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300b1fc0 0x7f58301529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:13.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300b1fc0 0x7f58301529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53026/0 (socket says 172.21.15.175:53026) 2023-10-10T18:52:13.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 -- 172.21.15.175:0/3704994017 learned_addr learned my addr 172.21.15.175:0/3704994017 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:13.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582ed9d700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b2390 0x7f58300cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:13.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582e59c700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58300aff60 0x7f5830158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:13.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 -- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b2390 msgr2=0x7f58300cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:13.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b2390 0x7f58300cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:13.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 -- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58300aff60 msgr2=0x7f5830158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:13.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58300aff60 0x7f5830158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:13.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 -- 172.21.15.175:0/3704994017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5828009420 con 0x7f58300b1fc0 2023-10-10T18:52:13.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.913+0000 7f582dd9b700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300b1fc0 0x7f58301529e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f582400c5b0 tx=0x7f5824001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:13.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.914+0000 7f581e7fc700 1 -- 172.21.15.175:0/3704994017 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5824005b30 con 0x7f58300b1fc0 2023-10-10T18:52:13.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.914+0000 7f581e7fc700 1 -- 172.21.15.175:0/3704994017 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5824005c90 con 0x7f58300b1fc0 2023-10-10T18:52:13.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.914+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5830158400 con 0x7f58300b1fc0 2023-10-10T18:52:13.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.914+0000 7f581e7fc700 1 -- 172.21.15.175:0/3704994017 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5824007300 con 0x7f58300b1fc0 2023-10-10T18:52:13.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.914+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5830159a60 con 0x7f58300b1fc0 2023-10-10T18:52:13.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.915+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5830159e00 con 0x7f58300b1fc0 2023-10-10T18:52:13.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.915+0000 7f581e7fc700 1 -- 172.21.15.175:0/3704994017 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f582401b3b0 con 0x7f58300b1fc0 2023-10-10T18:52:13.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.915+0000 7f581e7fc700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5810066f30 0x7f58100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:13.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.915+0000 7f581e7fc700 1 -- 172.21.15.175:0/3704994017 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6549+0+0 (secure 0 0 0) 0x7f5824083db0 con 0x7f58300b1fc0 2023-10-10T18:52:13.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.916+0000 7f582e59c700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5810066f30 0x7f58100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:13.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.916+0000 7f582e59c700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5810066f30 0x7f58100693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f58200075a0 tx=0x7f58200097a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:13.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:13.920+0000 7f581e7fc700 1 -- 172.21.15.175:0/3704994017 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5824050cd0 con 0x7f58300b1fc0 2023-10-10T18:52:14.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.144+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5830067570 con 0x7f58300b1fc0 2023-10-10T18:52:14.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.144+0000 7f581e7fc700 1 -- 172.21.15.175:0/3704994017 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6214 (secure 0 0 0) 0x7f5824017090 con 0x7f58300b1fc0 2023-10-10T18:52:14.144 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:14.145 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:51:57.583479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:14.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5810066f30 msgr2=0x7f58100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5810066f30 0x7f58100693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f58200075a0 tx=0x7f58200097a0 comp rx=0 tx=0).stop 2023-10-10T18:52:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300b1fc0 msgr2=0x7f58301529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300b1fc0 0x7f58301529e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f582400c5b0 tx=0x7f5824001070 comp rx=0 tx=0).stop 2023-10-10T18:52:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 shutdown_connections 2023-10-10T18:52:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5810066f30 0x7f58100693e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58300b2390 0x7f58300cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58300aff60 0x7f5830158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 --2- 172.21.15.175:0/3704994017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58300b1fc0 0x7f58301529e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 >> 172.21.15.175:0/3704994017 conn(0x7f5830063230 msgr2=0x7f583014fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:14.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 shutdown_connections 2023-10-10T18:52:14.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:14.146+0000 7f5834fbf700 1 -- 172.21.15.175:0/3704994017 wait complete. 2023-10-10T18:52:14.149 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 56 2023-10-10T18:52:14.158 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T18:52:16.160 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:16.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.287+0000 7fa89f067700 1 Processor -- start 2023-10-10T18:52:16.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.287+0000 7fa89f067700 1 -- start start 2023-10-10T18:52:16.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89f067700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa89806cf90 0x7fa89806d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:16.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89f067700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa8980c0f70 0x7fa898156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:16.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89f067700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980bf2b0 0x7fa8981567d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:16.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89f067700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8980c0990 con 0x7fa89806cf90 2023-10-10T18:52:16.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89f067700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa89806d8a0 con 0x7fa8980c0f70 2023-10-10T18:52:16.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89f067700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8981595e0 con 0x7fa8980bf2b0 2023-10-10T18:52:16.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89e065700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa89806cf90 0x7fa89806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:16.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89e065700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa89806cf90 0x7fa89806d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59706/0 (socket says 172.21.15.175:59706) 2023-10-10T18:52:16.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.288+0000 7fa89e065700 1 -- 172.21.15.175:0/2935645469 learned_addr learned my addr 172.21.15.175:0/2935645469 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:16.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89e866700 1 --2- 172.21.15.175:0/2935645469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980bf2b0 0x7fa8981567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:16.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89d864700 1 --2- 172.21.15.175:0/2935645469 >> v2:172.21.15.191:3300/0 conn(0x7fa8980c0f70 0x7fa898156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:16.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89e065700 1 -- 172.21.15.175:0/2935645469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980bf2b0 msgr2=0x7fa8981567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:16.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89e065700 1 --2- 172.21.15.175:0/2935645469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980bf2b0 0x7fa8981567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89e065700 1 -- 172.21.15.175:0/2935645469 >> v2:172.21.15.191:3300/0 conn(0x7fa8980c0f70 msgr2=0x7fa898156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:16.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89e065700 1 --2- 172.21.15.175:0/2935645469 >> v2:172.21.15.191:3300/0 conn(0x7fa8980c0f70 0x7fa898156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89e065700 1 -- 172.21.15.175:0/2935645469 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89815a440 con 0x7fa89806cf90 2023-10-10T18:52:16.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89e065700 1 --2- 172.21.15.175:0/2935645469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa89806d360 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa898065040 tx=0x7fa894009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d974e955af3874f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:16.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89d063700 1 -- 172.21.15.175:0/2935645469 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894005c10 con 0x7fa89806cf90 2023-10-10T18:52:16.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89d063700 1 -- 172.21.15.175:0/2935645469 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894005d70 con 0x7fa89806cf90 2023-10-10T18:52:16.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.289+0000 7fa89d063700 1 -- 172.21.15.175:0/2935645469 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89400b880 con 0x7fa89806cf90 2023-10-10T18:52:16.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 -- 172.21.15.175:0/2935645469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 msgr2=0x7fa89806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:16.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 --2- 172.21.15.175:0/2935645469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa89806d360 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa898065040 tx=0x7fa894009220 comp rx=0 tx=0).stop 2023-10-10T18:52:16.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 -- 172.21.15.175:0/2935645469 shutdown_connections 2023-10-10T18:52:16.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 --2- 172.21.15.175:0/2935645469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980bf2b0 0x7fa8981567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 --2- 172.21.15.175:0/2935645469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa89806d360 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 --2- 172.21.15.175:0/2935645469 >> v2:172.21.15.191:3300/0 conn(0x7fa8980c0f70 0x7fa898156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 -- 172.21.15.175:0/2935645469 >> 172.21.15.175:0/2935645469 conn(0x7fa89806b310 msgr2=0x7fa89806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:16.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 -- 172.21.15.175:0/2935645469 shutdown_connections 2023-10-10T18:52:16.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.290+0000 7fa89f067700 1 -- 172.21.15.175:0/2935645469 wait complete. 2023-10-10T18:52:16.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.291+0000 7fa89f067700 1 Processor -- start 2023-10-10T18:52:16.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.291+0000 7fa89f067700 1 -- start start 2023-10-10T18:52:16.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89f067700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa898169bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:16.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89f067700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980bf2b0 0x7fa898069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:16.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89f067700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980c0f70 0x7fa898069c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:16.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89f067700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa898158e00 con 0x7fa89806cf90 2023-10-10T18:52:16.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89f067700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa898069030 con 0x7fa8980bf2b0 2023-10-10T18:52:16.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89f067700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89816a6b0 con 0x7fa8980c0f70 2023-10-10T18:52:16.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa898169bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:16.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa898169bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59716/0 (socket says 172.21.15.175:59716) 2023-10-10T18:52:16.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 -- 172.21.15.175:0/3889761187 learned_addr learned my addr 172.21.15.175:0/3889761187 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:16.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89d864700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980bf2b0 0x7fa898069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:16.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e866700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980c0f70 0x7fa898069c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:16.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 -- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980c0f70 msgr2=0x7fa898069c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:16.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980c0f70 0x7fa898069c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 -- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980bf2b0 msgr2=0x7fa898069740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:16.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980bf2b0 0x7fa898069740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 -- 172.21.15.175:0/3889761187 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa894009480 con 0x7fa89806cf90 2023-10-10T18:52:16.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89d864700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980bf2b0 0x7fa898069740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:52:16.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e866700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980c0f70 0x7fa898069c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:52:16.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89e065700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa898169bc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa89400b3c0 tx=0x7fa894016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:16.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa88e7fc700 1 -- 172.21.15.175:0/3889761187 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894011920 con 0x7fa89806cf90 2023-10-10T18:52:16.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa88e7fc700 1 -- 172.21.15.175:0/3889761187 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894020070 con 0x7fa89806cf90 2023-10-10T18:52:16.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89814eb20 con 0x7fa89806cf90 2023-10-10T18:52:16.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa88e7fc700 1 -- 172.21.15.175:0/3889761187 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894011cb0 con 0x7fa89806cf90 2023-10-10T18:52:16.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.292+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89814e810 con 0x7fa89806cf90 2023-10-10T18:52:16.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.293+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa898067570 con 0x7fa89806cf90 2023-10-10T18:52:16.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.294+0000 7fa88e7fc700 1 -- 172.21.15.175:0/3889761187 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa894022030 con 0x7fa89806cf90 2023-10-10T18:52:16.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.294+0000 7fa88e7fc700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa87c066f30 0x7fa87c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:16.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.294+0000 7fa88e7fc700 1 -- 172.21.15.175:0/3889761187 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6549+0+0 (secure 0 0 0) 0x7fa894086510 con 0x7fa89806cf90 2023-10-10T18:52:16.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.294+0000 7fa89d864700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa87c066f30 0x7fa87c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:16.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.295+0000 7fa89d864700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa87c066f30 0x7fa87c0693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa898156790 tx=0x7fa888008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:16.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.298+0000 7fa88e7fc700 1 -- 172.21.15.175:0/3889761187 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa894053430 con 0x7fa89806cf90 2023-10-10T18:52:16.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.524+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8980c04b0 con 0x7fa89806cf90 2023-10-10T18:52:16.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.525+0000 7fa88e7fc700 1 -- 172.21.15.175:0/3889761187 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6214 (secure 0 0 0) 0x7fa894056a50 con 0x7fa89806cf90 2023-10-10T18:52:16.525 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:16.525 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:51:57.583479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:16.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa87c066f30 msgr2=0x7fa87c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:16.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa87c066f30 0x7fa87c0693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa898156790 tx=0x7fa888008040 comp rx=0 tx=0).stop 2023-10-10T18:52:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 msgr2=0x7fa898169bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa898169bc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa89400b3c0 tx=0x7fa894016000 comp rx=0 tx=0).stop 2023-10-10T18:52:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 shutdown_connections 2023-10-10T18:52:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa87c066f30 0x7fa87c0693e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa8980c0f70 0x7fa898069c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa89806cf90 0x7fa898169bc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 --2- 172.21.15.175:0/3889761187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980bf2b0 0x7fa898069740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 >> 172.21.15.175:0/3889761187 conn(0x7fa89806b310 msgr2=0x7fa8980c6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 shutdown_connections 2023-10-10T18:52:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:16.527+0000 7fa89f067700 1 -- 172.21.15.175:0/3889761187 wait complete. 2023-10-10T18:52:16.530 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 56 2023-10-10T18:52:16.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T18:52:18.540 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:18.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.668+0000 7f0c363b4700 1 Processor -- start 2023-10-10T18:52:18.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.668+0000 7f0c363b4700 1 -- start start 2023-10-10T18:52:18.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c363b4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0c3014ff10 0x7f0c301569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:18.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c363b4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0c301502e0 0x7f0c301572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:18.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c363b4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c30156510 con 0x7f0c3006e280 2023-10-10T18:52:18.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c363b4700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c300c90a0 con 0x7f0c301502e0 2023-10-10T18:52:18.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c363b4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c300c9380 con 0x7f0c3014ff10 2023-10-10T18:52:18.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c34951700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f0c3006e280 0x7f0c3006aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48548/0 (socket says 172.21.15.175:48548) 2023-10-10T18:52:18.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c34951700 1 -- 172.21.15.175:0/3161913416 learned_addr learned my addr 172.21.15.175:0/3161913416 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:18.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c2f7fe700 1 --2- 172.21.15.175:0/3161913416 >> v2:172.21.15.175:3300/0 conn(0x7f0c301502e0 0x7f0c301572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:18.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c2ffff700 1 --2- 172.21.15.175:0/3161913416 >> v2:172.21.15.191:3300/0 conn(0x7f0c3014ff10 0x7f0c301569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:18.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c2effd700 1 -- 172.21.15.175:0/3161913416 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142869323 0 0) 0x7f0c30156510 con 0x7f0c3006e280 2023-10-10T18:52:18.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c2effd700 1 -- 172.21.15.175:0/3161913416 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c18003570 con 0x7f0c3006e280 2023-10-10T18:52:18.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c2f7fe700 1 -- 172.21.15.175:0/3161913416 >> v2:172.21.15.191:3300/0 conn(0x7f0c3014ff10 msgr2=0x7f0c301569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:18.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c2f7fe700 1 --2- 172.21.15.175:0/3161913416 >> v2:172.21.15.191:3300/0 conn(0x7f0c3014ff10 0x7f0c301569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c2f7fe700 1 -- 172.21.15.175:0/3161913416 >> v1:172.21.15.175:6789/0 conn(0x7f0c3006e280 legacy=0x7f0c3006aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:18.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.669+0000 7f0c2f7fe700 1 -- 172.21.15.175:0/3161913416 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30155ec0 con 0x7f0c301502e0 2023-10-10T18:52:18.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c2f7fe700 1 --2- 172.21.15.175:0/3161913416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c301572b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0c30071450 tx=0x7f0c240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69bfe46cc35d4194 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:18.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c2effd700 1 -- 172.21.15.175:0/3161913416 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24011630 con 0x7f0c301502e0 2023-10-10T18:52:18.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c2effd700 1 -- 172.21.15.175:0/3161913416 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c240117b0 con 0x7f0c301502e0 2023-10-10T18:52:18.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c2effd700 1 -- 172.21.15.175:0/3161913416 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24011b70 con 0x7f0c301502e0 2023-10-10T18:52:18.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c363b4700 1 -- 172.21.15.175:0/3161913416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 msgr2=0x7f0c301572b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:18.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c363b4700 1 --2- 172.21.15.175:0/3161913416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c301572b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0c30071450 tx=0x7f0c240091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c363b4700 1 -- 172.21.15.175:0/3161913416 shutdown_connections 2023-10-10T18:52:18.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c363b4700 1 --2- 172.21.15.175:0/3161913416 >> v2:172.21.15.191:3300/0 conn(0x7f0c3014ff10 0x7f0c301569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c363b4700 1 --2- 172.21.15.175:0/3161913416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c301572b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c363b4700 1 -- 172.21.15.175:0/3161913416 >> 172.21.15.175:0/3161913416 conn(0x7f0c300682c0 msgr2=0x7f0c300c87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:18.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c363b4700 1 -- 172.21.15.175:0/3161913416 shutdown_connections 2023-10-10T18:52:18.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.670+0000 7f0c363b4700 1 -- 172.21.15.175:0/3161913416 wait complete. 2023-10-10T18:52:18.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.671+0000 7f0c363b4700 1 Processor -- start 2023-10-10T18:52:18.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.672+0000 7f0c363b4700 1 -- start start 2023-10-10T18:52:18.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.672+0000 7f0c363b4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0c3006e280 0x7f0c300c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:18.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.672+0000 7f0c363b4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c3014ff10 0x7f0c30167130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:18.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.672+0000 7f0c363b4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c30165ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:18.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.672+0000 7f0c363b4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301531a0 con 0x7f0c301502e0 2023-10-10T18:52:18.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.672+0000 7f0c363b4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30166ad0 con 0x7f0c3014ff10 2023-10-10T18:52:18.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.672+0000 7f0c363b4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30166860 con 0x7f0c3006e280 2023-10-10T18:52:18.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c30165ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:18.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c30165ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59736/0 (socket says 172.21.15.175:59736) 2023-10-10T18:52:18.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 -- 172.21.15.175:0/820798887 learned_addr learned my addr 172.21.15.175:0/820798887 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:18.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c2f7fe700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c3014ff10 0x7f0c30167130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:18.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c2ffff700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0c3006e280 0x7f0c300c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:18.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 -- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0c3006e280 msgr2=0x7f0c300c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:18.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0c3006e280 0x7f0c300c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 -- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c3014ff10 msgr2=0x7f0c30167130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:18.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c3014ff10 0x7f0c30167130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 -- 172.21.15.175:0/820798887 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c24009420 con 0x7f0c301502e0 2023-10-10T18:52:18.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c2ffff700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0c3006e280 0x7f0c300c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:18.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c34951700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c30165ee0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800cc70 tx=0x7f0c28001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:18.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c17fff700 1 -- 172.21.15.175:0/820798887 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800ce80 con 0x7f0c301502e0 2023-10-10T18:52:18.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c17fff700 1 -- 172.21.15.175:0/820798887 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2800ab30 con 0x7f0c301502e0 2023-10-10T18:52:18.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c300c7110 con 0x7f0c301502e0 2023-10-10T18:52:18.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c17fff700 1 -- 172.21.15.175:0/820798887 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280070f0 con 0x7f0c301502e0 2023-10-10T18:52:18.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.673+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c300c76d0 con 0x7f0c301502e0 2023-10-10T18:52:18.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.674+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c300c7d20 con 0x7f0c301502e0 2023-10-10T18:52:18.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.675+0000 7f0c17fff700 1 -- 172.21.15.175:0/820798887 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0c28007520 con 0x7f0c301502e0 2023-10-10T18:52:18.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.675+0000 7f0c17fff700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0c1c066f30 0x7f0c1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:18.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.675+0000 7f0c17fff700 1 -- 172.21.15.175:0/820798887 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6549+0+0 (secure 0 0 0) 0x7f0c28084dd0 con 0x7f0c301502e0 2023-10-10T18:52:18.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.675+0000 7f0c2ffff700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0c1c066f30 0x7f0c1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:18.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.676+0000 7f0c2ffff700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0c1c066f30 0x7f0c1c0693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0c20006f00 tx=0x7f0c20008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:18.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.679+0000 7f0c17fff700 1 -- 172.21.15.175:0/820798887 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c28051ce0 con 0x7f0c301502e0 2023-10-10T18:52:18.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.904+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c30067570 con 0x7f0c301502e0 2023-10-10T18:52:18.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.904+0000 7f0c17fff700 1 -- 172.21.15.175:0/820798887 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6214 (secure 0 0 0) 0x7f0c28055300 con 0x7f0c301502e0 2023-10-10T18:52:18.905 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:18.905 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:51:57.583479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:18.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0c1c066f30 msgr2=0x7f0c1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:18.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0c1c066f30 0x7f0c1c0693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0c20006f00 tx=0x7f0c20008040 comp rx=0 tx=0).stop 2023-10-10T18:52:18.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 msgr2=0x7f0c30165ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c30165ee0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800cc70 tx=0x7f0c28001aa0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 shutdown_connections 2023-10-10T18:52:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0c1c066f30 0x7f0c1c0693e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0c301502e0 0x7f0c30165ee0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0c3006e280 0x7f0c300c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 --2- 172.21.15.175:0/820798887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c3014ff10 0x7f0c30167130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:18.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 >> 172.21.15.175:0/820798887 conn(0x7f0c300682c0 msgr2=0x7f0c3014f320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:18.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 shutdown_connections 2023-10-10T18:52:18.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:18.907+0000 7f0c363b4700 1 -- 172.21.15.175:0/820798887 wait complete. 2023-10-10T18:52:18.910 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 56 2023-10-10T18:52:18.919 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T18:52:19.264 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:52:19.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:52:20.920 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:21.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.047+0000 7fa967503700 1 Processor -- start 2023-10-10T18:52:21.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.047+0000 7fa967503700 1 -- start start 2023-10-10T18:52:21.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.048+0000 7fa967503700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa9600c5850 0x7fa9600c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:21.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.048+0000 7fa967503700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9600c6fb0 con 0x7fa9600c4f20 2023-10-10T18:52:21.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.048+0000 7fa967503700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9600c6160 con 0x7fa9600c12d0 2023-10-10T18:52:21.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.048+0000 7fa967503700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa960063ea0 con 0x7fa9600c5850 2023-10-10T18:52:21.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.048+0000 7fa96529f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa9600c5850 0x7fa9600c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:21.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.048+0000 7fa965aa0700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa9600c4f20 0x7fa9600c7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48556/0 (socket says 172.21.15.175:48556) 2023-10-10T18:52:21.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.048+0000 7fa965aa0700 1 -- 172.21.15.175:0/3599175650 learned_addr learned my addr 172.21.15.175:0/3599175650 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:21.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.049+0000 7fa96529f700 1 -- 172.21.15.175:0/3599175650 >> v1:172.21.15.191:6789/0 conn(0x7fa9600c12d0 legacy=0x7fa960156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:21.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.049+0000 7fa96529f700 1 -- 172.21.15.175:0/3599175650 >> v1:172.21.15.175:6789/0 conn(0x7fa9600c4f20 legacy=0x7fa9600c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:21.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.049+0000 7fa96529f700 1 -- 172.21.15.175:0/3599175650 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa960063940 con 0x7fa9600c5850 2023-10-10T18:52:21.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.049+0000 7fa957fff700 1 -- 172.21.15.175:0/3599175650 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907984107 0 0) 0x7fa9600c6fb0 con 0x7fa9600c4f20 2023-10-10T18:52:21.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.049+0000 7fa96529f700 1 --2- 172.21.15.175:0/3599175650 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c5850 0x7fa9600c5c20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009e60 tx=0x7fa95c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3982baa2054b971e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:21.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.049+0000 7fa957fff700 1 -- 172.21.15.175:0/3599175650 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00e070 con 0x7fa9600c5850 2023-10-10T18:52:21.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.049+0000 7fa957fff700 1 -- 172.21.15.175:0/3599175650 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c00bcc0 con 0x7fa9600c5850 2023-10-10T18:52:21.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.049+0000 7fa957fff700 1 -- 172.21.15.175:0/3599175650 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c012680 con 0x7fa9600c5850 2023-10-10T18:52:21.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.050+0000 7fa967503700 1 -- 172.21.15.175:0/3599175650 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c5850 msgr2=0x7fa9600c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:21.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.050+0000 7fa967503700 1 --2- 172.21.15.175:0/3599175650 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c5850 0x7fa9600c5c20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009e60 tx=0x7fa95c009780 comp rx=0 tx=0).stop 2023-10-10T18:52:21.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.050+0000 7fa967503700 1 -- 172.21.15.175:0/3599175650 shutdown_connections 2023-10-10T18:52:21.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.050+0000 7fa967503700 1 --2- 172.21.15.175:0/3599175650 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c5850 0x7fa9600c5c20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:21.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.050+0000 7fa967503700 1 -- 172.21.15.175:0/3599175650 >> 172.21.15.175:0/3599175650 conn(0x7fa96006dc70 msgr2=0x7fa960063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:21.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.050+0000 7fa967503700 1 -- 172.21.15.175:0/3599175650 shutdown_connections 2023-10-10T18:52:21.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.050+0000 7fa967503700 1 -- 172.21.15.175:0/3599175650 wait complete. 2023-10-10T18:52:21.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.051+0000 7fa967503700 1 Processor -- start 2023-10-10T18:52:21.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.051+0000 7fa967503700 1 -- start start 2023-10-10T18:52:21.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa967503700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9600c12d0 0x7fa960151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:21.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa967503700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c4f20 0x7fa9601581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:21.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa967503700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9600c5850 0x7fa960158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:21.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa967503700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9600c0e40 con 0x7fa9600c4f20 2023-10-10T18:52:21.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa967503700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9601525d0 con 0x7fa9600c5850 2023-10-10T18:52:21.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa967503700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9601528b0 con 0x7fa9600c12d0 2023-10-10T18:52:21.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c4f20 0x7fa9601581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:21.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c4f20 0x7fa9601581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59752/0 (socket says 172.21.15.175:59752) 2023-10-10T18:52:21.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 -- 172.21.15.175:0/833120958 learned_addr learned my addr 172.21.15.175:0/833120958 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:21.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa96529f700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9600c12d0 0x7fa960151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:21.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa965aa0700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9600c5850 0x7fa960158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:21.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 -- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9600c12d0 msgr2=0x7fa960151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:21.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9600c12d0 0x7fa960151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:21.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 -- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9600c5850 msgr2=0x7fa960158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:21.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9600c5850 0x7fa960158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:21.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 -- 172.21.15.175:0/833120958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c009a20 con 0x7fa9600c4f20 2023-10-10T18:52:21.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa964a9e700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c4f20 0x7fa9601581e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa95000c4e0 tx=0x7fa95000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:21.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa9557fa700 1 -- 172.21.15.175:0/833120958 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa950011590 con 0x7fa9600c4f20 2023-10-10T18:52:21.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa9557fa700 1 -- 172.21.15.175:0/833120958 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa950011710 con 0x7fa9600c4f20 2023-10-10T18:52:21.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa960152b90 con 0x7fa9600c4f20 2023-10-10T18:52:21.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.052+0000 7fa9557fa700 1 -- 172.21.15.175:0/833120958 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa950011ad0 con 0x7fa9600c4f20 2023-10-10T18:52:21.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.053+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9601579e0 con 0x7fa9600c4f20 2023-10-10T18:52:21.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.053+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa960067570 con 0x7fa9600c4f20 2023-10-10T18:52:21.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.054+0000 7fa9557fa700 1 -- 172.21.15.175:0/833120958 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa95001b3b0 con 0x7fa9600c4f20 2023-10-10T18:52:21.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.054+0000 7fa9557fa700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa944066f30 0x7fa9440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:21.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.054+0000 7fa9557fa700 1 -- 172.21.15.175:0/833120958 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6549+0+0 (secure 0 0 0) 0x7fa950083cb0 con 0x7fa9600c4f20 2023-10-10T18:52:21.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.054+0000 7fa96529f700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa944066f30 0x7fa9440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:21.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.055+0000 7fa96529f700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa944066f30 0x7fa9440693e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0099f0 tx=0x7fa95c019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:21.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.058+0000 7fa9557fa700 1 -- 172.21.15.175:0/833120958 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa950050bd0 con 0x7fa9600c4f20 2023-10-10T18:52:21.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.281+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa960066b60 con 0x7fa9600c4f20 2023-10-10T18:52:21.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.282+0000 7fa9557fa700 1 -- 172.21.15.175:0/833120958 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6214 (secure 0 0 0) 0x7fa950017070 con 0x7fa9600c4f20 2023-10-10T18:52:21.283 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:21.283 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:51:57.583479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":4248,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.175:6836/647425682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":647425682},{"type":"v1","addr":"172.21.15.175:6836","nonce":647425682}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:21.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.283+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa944066f30 msgr2=0x7fa9440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:21.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.283+0000 7fa967503700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa944066f30 0x7fa9440693e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0099f0 tx=0x7fa95c019000 comp rx=0 tx=0).stop 2023-10-10T18:52:21.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.283+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c4f20 msgr2=0x7fa9601581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:21.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.283+0000 7fa967503700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c4f20 0x7fa9601581e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa95000c4e0 tx=0x7fa95000c7f0 comp rx=0 tx=0).stop 2023-10-10T18:52:21.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.284+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 shutdown_connections 2023-10-10T18:52:21.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.284+0000 7fa967503700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa944066f30 0x7fa9440693e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:21.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.284+0000 7fa967503700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9600c5850 0x7fa960158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:21.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.284+0000 7fa967503700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9600c12d0 0x7fa960151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:21.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.284+0000 7fa967503700 1 --2- 172.21.15.175:0/833120958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9600c4f20 0x7fa9601581e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:21.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.284+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 >> 172.21.15.175:0/833120958 conn(0x7fa96006dc70 msgr2=0x7fa9600bf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:21.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.284+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 shutdown_connections 2023-10-10T18:52:21.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:21.284+0000 7fa967503700 1 -- 172.21.15.175:0/833120958 wait complete. 2023-10-10T18:52:21.288 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 56 2023-10-10T18:52:21.295 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T18:52:23.297 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:23.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.424+0000 7f93f27ce700 1 Processor -- start 2023-10-10T18:52:23.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.424+0000 7f93f27ce700 1 -- start start 2023-10-10T18:52:23.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93f27ce700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f93ec155fb0 0x7f93ec159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93f27ce700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec1574b0 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93f27ce700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec0afad0 con 0x7f93ec0b2390 2023-10-10T18:52:23.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93f27ce700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec150770 con 0x7f93ec155fb0 2023-10-10T18:52:23.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93f0d6b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f93ec155fb0 0x7f93ec159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93ebfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f93ec0b1fc0 0x7f93ec068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48560/0 (socket says 172.21.15.175:48560) 2023-10-10T18:52:23.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93ebfff700 1 -- 172.21.15.175:0/82782592 learned_addr learned my addr 172.21.15.175:0/82782592 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:23.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93f0d6b700 1 -- 172.21.15.175:0/82782592 >> v1:172.21.15.191:6789/0 conn(0x7f93ec0b2390 legacy=0x7f93ec0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93f0d6b700 1 -- 172.21.15.175:0/82782592 >> v1:172.21.15.175:6789/0 conn(0x7f93ec0b1fc0 legacy=0x7f93ec068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93f0d6b700 1 -- 172.21.15.175:0/82782592 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec0af4c0 con 0x7f93ec155fb0 2023-10-10T18:52:23.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.425+0000 7f93eaffd700 1 -- 172.21.15.175:0/82782592 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937576031 0 0) 0x7f93ec1574b0 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93f0d6b700 1 --2- 172.21.15.175:0/82782592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec155fb0 0x7f93ec159920 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009e60 tx=0x7f93e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9290dd36825b43 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:23.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93eaffd700 1 -- 172.21.15.175:0/82782592 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e400e070 con 0x7f93ec155fb0 2023-10-10T18:52:23.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93eaffd700 1 -- 172.21.15.175:0/82782592 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e400bcc0 con 0x7f93ec155fb0 2023-10-10T18:52:23.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93eaffd700 1 -- 172.21.15.175:0/82782592 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4012680 con 0x7f93ec155fb0 2023-10-10T18:52:23.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93f27ce700 1 -- 172.21.15.175:0/82782592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec155fb0 msgr2=0x7f93ec159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93f27ce700 1 --2- 172.21.15.175:0/82782592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec155fb0 0x7f93ec159920 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009e60 tx=0x7f93e4009780 comp rx=0 tx=0).stop 2023-10-10T18:52:23.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93f27ce700 1 -- 172.21.15.175:0/82782592 shutdown_connections 2023-10-10T18:52:23.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93f27ce700 1 --2- 172.21.15.175:0/82782592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec155fb0 0x7f93ec159920 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93f27ce700 1 -- 172.21.15.175:0/82782592 >> 172.21.15.175:0/82782592 conn(0x7f93ec063230 msgr2=0x7f93ec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:23.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93f27ce700 1 -- 172.21.15.175:0/82782592 shutdown_connections 2023-10-10T18:52:23.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.426+0000 7f93f27ce700 1 -- 172.21.15.175:0/82782592 wait complete. 2023-10-10T18:52:23.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.427+0000 7f93f27ce700 1 Processor -- start 2023-10-10T18:52:23.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.428+0000 7f93f27ce700 1 -- start start 2023-10-10T18:52:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.428+0000 7f93f27ce700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec0b1fc0 0x7f93ec0cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.428+0000 7f93f27ce700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93ec0b2390 0x7f93ec1666e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.428+0000 7f93f27ce700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93ec155fb0 0x7f93ec14dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.428+0000 7f93f27ce700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec14d710 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.428+0000 7f93f27ce700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0cb300 con 0x7f93ec155fb0 2023-10-10T18:52:23.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.428+0000 7f93f27ce700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0cb5e0 con 0x7f93ec0b2390 2023-10-10T18:52:23.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.428+0000 7f93ebfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec0b1fc0 0x7f93ec0cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93ebfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec0b1fc0 0x7f93ec0cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59772/0 (socket says 172.21.15.175:59772) 2023-10-10T18:52:23.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93ebfff700 1 -- 172.21.15.175:0/3822787762 learned_addr learned my addr 172.21.15.175:0/3822787762 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:23.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93eb7fe700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93ec0b2390 0x7f93ec1666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93f0d6b700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93ec155fb0 0x7f93ec14dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93ebfff700 1 -- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93ec0b2390 msgr2=0x7f93ec1666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93ebfff700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93ec0b2390 0x7f93ec1666e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93ebfff700 1 -- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93ec155fb0 msgr2=0x7f93ec14dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93ebfff700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93ec155fb0 0x7f93ec14dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93ebfff700 1 -- 172.21.15.175:0/3822787762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e4009a20 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93eb7fe700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93ec0b2390 0x7f93ec1666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93ebfff700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec0b1fc0 0x7f93ec0cc0e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00c7e0 tx=0x7f93dc007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93d3fff700 1 -- 172.21.15.175:0/3822787762 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc005b30 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93d3fff700 1 -- 172.21.15.175:0/3822787762 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc005c90 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93d3fff700 1 -- 172.21.15.175:0/3822787762 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc007300 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec0cb8c0 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.429+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec14d3a0 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.430+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec14d9f0 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.431+0000 7f93d3fff700 1 -- 172.21.15.175:0/3822787762 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f93dc01b400 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.431+0000 7f93d3fff700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93d8066f30 0x7f93d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.431+0000 7f93d3fff700 1 -- 172.21.15.175:0/3822787762 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6549+0+0 (secure 0 0 0) 0x7f93dc0843a0 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.431+0000 7f93eb7fe700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93d8066f30 0x7f93d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.432+0000 7f93eb7fe700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93d8066f30 0x7f93d80693e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f93ec14fc50 tx=0x7f93e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:23.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.436+0000 7f93d3fff700 1 -- 172.21.15.175:0/3822787762 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93dc051340 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.669+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93ec067570 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.670+0000 7f93d3fff700 1 -- 172.21.15.175:0/3822787762 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5453 (secure 0 0 0) 0x7f93dc054960 con 0x7f93ec0b1fc0 2023-10-10T18:52:23.670 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:23.671 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:23.469100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:23.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93d8066f30 msgr2=0x7f93d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93d8066f30 0x7f93d80693e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f93ec14fc50 tx=0x7f93e0009040 comp rx=0 tx=0).stop 2023-10-10T18:52:23.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec0b1fc0 msgr2=0x7f93ec0cc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec0b1fc0 0x7f93ec0cc0e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00c7e0 tx=0x7f93dc007df0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 shutdown_connections 2023-10-10T18:52:23.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f93d8066f30 0x7f93d80693e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93ec155fb0 0x7f93ec14dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f93ec0b1fc0 0x7f93ec0cc0e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 --2- 172.21.15.175:0/3822787762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93ec0b2390 0x7f93ec1666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 >> 172.21.15.175:0/3822787762 conn(0x7f93ec063230 msgr2=0x7f93ec0c7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:23.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 shutdown_connections 2023-10-10T18:52:23.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.672+0000 7f93f27ce700 1 -- 172.21.15.175:0/3822787762 wait complete. 2023-10-10T18:52:23.675 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 57 2023-10-10T18:52:23.685 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-10T18:52:23.685 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T18:52:23.685 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:23.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.801+0000 7fa309798700 1 Processor -- start 2023-10-10T18:52:23.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.801+0000 7fa309798700 1 -- start start 2023-10-10T18:52:23.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa309798700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30414fd90 0x7fa304159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa309798700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa3040cd040 0x7fa3040684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa309798700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa304067fa0 con 0x7fa30408a100 2023-10-10T18:52:23.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa309798700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3040c5600 con 0x7fa3040cd040 2023-10-10T18:52:23.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa309798700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa304156b70 con 0x7fa30414fd90 2023-10-10T18:52:23.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa308f97700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa3040cd040 0x7fa3040684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa303fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30414fd90 0x7fa304159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa308f97700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa3040cd040 0x7fa3040684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:42976/0 (socket says 172.21.15.175:42976) 2023-10-10T18:52:23.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa303fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30414fd90 0x7fa304159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:42982/0 (socket says 172.21.15.175:42982) 2023-10-10T18:52:23.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa308f97700 1 -- 172.21.15.175:0/3065530517 learned_addr learned my addr 172.21.15.175:0/3065530517 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:23.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa308f97700 1 -- 172.21.15.175:0/3065530517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30414fd90 msgr2=0x7fa304159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:23.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa308f97700 1 --2- 172.21.15.175:0/3065530517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30414fd90 0x7fa304159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa308f97700 1 -- 172.21.15.175:0/3065530517 >> v1:172.21.15.191:6789/0 conn(0x7fa30408a100 legacy=0x7fa3040c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa308f97700 1 -- 172.21.15.175:0/3065530517 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304150160 con 0x7fa3040cd040 2023-10-10T18:52:23.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.802+0000 7fa303fff700 1 --2- 172.21.15.175:0/3065530517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30414fd90 0x7fa304159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:52:23.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa308f97700 1 --2- 172.21.15.175:0/3065530517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3040cd040 0x7fa3040684f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009f20 tx=0x7fa2fc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=734071b44ad1fa24 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:23.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa302ffd700 1 -- 172.21.15.175:0/3065530517 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00e070 con 0x7fa3040cd040 2023-10-10T18:52:23.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa302ffd700 1 -- 172.21.15.175:0/3065530517 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc00b9d0 con 0x7fa3040cd040 2023-10-10T18:52:23.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa302ffd700 1 -- 172.21.15.175:0/3065530517 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0092e0 con 0x7fa3040cd040 2023-10-10T18:52:23.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa309798700 1 -- 172.21.15.175:0/3065530517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3040cd040 msgr2=0x7fa3040684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa309798700 1 --2- 172.21.15.175:0/3065530517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3040cd040 0x7fa3040684f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009f20 tx=0x7fa2fc009840 comp rx=0 tx=0).stop 2023-10-10T18:52:23.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa309798700 1 -- 172.21.15.175:0/3065530517 shutdown_connections 2023-10-10T18:52:23.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa309798700 1 --2- 172.21.15.175:0/3065530517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3040cd040 0x7fa3040684f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa309798700 1 --2- 172.21.15.175:0/3065530517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa30414fd90 0x7fa304159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa309798700 1 -- 172.21.15.175:0/3065530517 >> 172.21.15.175:0/3065530517 conn(0x7fa304063d00 msgr2=0x7fa3040cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:23.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa309798700 1 -- 172.21.15.175:0/3065530517 shutdown_connections 2023-10-10T18:52:23.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.803+0000 7fa309798700 1 -- 172.21.15.175:0/3065530517 wait complete. 2023-10-10T18:52:23.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.804+0000 7fa309798700 1 Processor -- start 2023-10-10T18:52:23.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.805+0000 7fa309798700 1 -- start start 2023-10-10T18:52:23.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.805+0000 7fa309798700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30408a100 0x7fa3040cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.805+0000 7fa309798700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3040cd040 0x7fa304150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.805+0000 7fa309798700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa30414fd90 0x7fa304150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.805+0000 7fa309798700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3041659f0 con 0x7fa30414fd90 2023-10-10T18:52:23.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.805+0000 7fa309798700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa304150640 con 0x7fa30408a100 2023-10-10T18:52:23.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.805+0000 7fa309798700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3041519c0 con 0x7fa3040cd040 2023-10-10T18:52:23.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.805+0000 7fa308f97700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa30414fd90 0x7fa304150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa308f97700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa30414fd90 0x7fa304150fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59782/0 (socket says 172.21.15.175:59782) 2023-10-10T18:52:23.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa308f97700 1 -- 172.21.15.175:0/2043651111 learned_addr learned my addr 172.21.15.175:0/2043651111 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:23.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa303fff700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30408a100 0x7fa3040cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa3037fe700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3040cd040 0x7fa304150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa308f97700 1 -- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3040cd040 msgr2=0x7fa304150a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa308f97700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3040cd040 0x7fa304150a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa308f97700 1 -- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30408a100 msgr2=0x7fa3040cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:23.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa308f97700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30408a100 0x7fa3040cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:23.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa308f97700 1 -- 172.21.15.175:0/2043651111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4009420 con 0x7fa30414fd90 2023-10-10T18:52:23.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa308f97700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa30414fd90 0x7fa304150fd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa304064c20 tx=0x7fa2fc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:23.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa2ebfff700 1 -- 172.21.15.175:0/2043651111 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc008c30 con 0x7fa30414fd90 2023-10-10T18:52:23.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa2ebfff700 1 -- 172.21.15.175:0/2043651111 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc008dd0 con 0x7fa30414fd90 2023-10-10T18:52:23.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc009ae0 con 0x7fa30414fd90 2023-10-10T18:52:23.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa2ebfff700 1 -- 172.21.15.175:0/2043651111 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0266b0 con 0x7fa30414fd90 2023-10-10T18:52:23.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.806+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa304169a20 con 0x7fa30414fd90 2023-10-10T18:52:23.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.807+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3041507b0 con 0x7fa30414fd90 2023-10-10T18:52:23.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.808+0000 7fa2ebfff700 1 -- 172.21.15.175:0/2043651111 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa2fc003040 con 0x7fa30414fd90 2023-10-10T18:52:23.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.808+0000 7fa2ebfff700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2f0066f30 0x7fa2f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:23.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.808+0000 7fa2ebfff700 1 -- 172.21.15.175:0/2043651111 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7fa2fc086740 con 0x7fa30414fd90 2023-10-10T18:52:23.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.808+0000 7fa303fff700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2f0066f30 0x7fa2f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:23.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.809+0000 7fa303fff700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2f0066f30 0x7fa2f00693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa2f40093f0 tx=0x7fa2f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:23.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:23.812+0000 7fa2ebfff700 1 -- 172.21.15.175:0/2043651111 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa3041507b0 con 0x7fa30414fd90 2023-10-10T18:52:24.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.036+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3041507b0 con 0x7fa30414fd90 2023-10-10T18:52:24.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.036+0000 7fa2ebfff700 1 -- 172.21.15.175:0/2043651111 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5453 (secure 0 0 0) 0x7fa3041507b0 con 0x7fa30414fd90 2023-10-10T18:52:24.036 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:24.037 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:23.469100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:24.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2f0066f30 msgr2=0x7fa2f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:24.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2f0066f30 0x7fa2f00693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa2f40093f0 tx=0x7fa2f4008040 comp rx=0 tx=0).stop 2023-10-10T18:52:24.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa30414fd90 msgr2=0x7fa304150fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:24.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa30414fd90 0x7fa304150fd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa304064c20 tx=0x7fa2fc018000 comp rx=0 tx=0).stop 2023-10-10T18:52:24.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 shutdown_connections 2023-10-10T18:52:24.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa2f0066f30 0x7fa2f00693e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:24.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa30414fd90 0x7fa304150fd0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:24.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30408a100 0x7fa3040cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:24.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 --2- 172.21.15.175:0/2043651111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3040cd040 0x7fa304150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:24.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 >> 172.21.15.175:0/2043651111 conn(0x7fa304063d00 msgr2=0x7fa3040c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:24.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 shutdown_connections 2023-10-10T18:52:24.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:24.038+0000 7fa309798700 1 -- 172.21.15.175:0/2043651111 wait complete. 2023-10-10T18:52:24.041 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 57 2023-10-10T18:52:24.051 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 57, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 57, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4250": { "addr": "172.21.15.191:6839/246221790", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 246221790, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 246221790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4250, "incarnation": 12, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:52:23.469100+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 4250, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:52:26.052 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:26.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.180+0000 7faab3af2700 1 Processor -- start 2023-10-10T18:52:26.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.181+0000 7faab3af2700 1 -- start start 2023-10-10T18:52:26.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.181+0000 7faab3af2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faaac0c1be0 0x7faaac06b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:26.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.181+0000 7faab3af2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 0x7faaac06a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:26.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.181+0000 7faab3af2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faaac14ddb0 0x7faaac14e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:26.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.181+0000 7faab3af2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaac06bca0 con 0x7faaac0c1be0 2023-10-10T18:52:26.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.181+0000 7faab3af2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaac154000 con 0x7faaac14ddb0 2023-10-10T18:52:26.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.181+0000 7faab3af2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaac154e20 con 0x7faaac0c1fb0 2023-10-10T18:52:26.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.181+0000 7faab188e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faaac0c1be0 0x7faaac06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:26.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab188e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faaac0c1be0 0x7faaac06b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51254/0 (socket says 172.21.15.175:51254) 2023-10-10T18:52:26.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab188e700 1 -- 172.21.15.175:0/355836992 learned_addr learned my addr 172.21.15.175:0/355836992 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:26.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab208f700 1 --2- 172.21.15.175:0/355836992 >> v2:172.21.15.191:3300/0 conn(0x7faaac14ddb0 0x7faaac14e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:26.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab108d700 1 --2- 172.21.15.175:0/355836992 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 0x7faaac06a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:26.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab188e700 1 -- 172.21.15.175:0/355836992 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 msgr2=0x7faaac06a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:26.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab188e700 1 --2- 172.21.15.175:0/355836992 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 0x7faaac06a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab188e700 1 -- 172.21.15.175:0/355836992 >> v2:172.21.15.191:3300/0 conn(0x7faaac14ddb0 msgr2=0x7faaac14e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:26.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab188e700 1 --2- 172.21.15.175:0/355836992 >> v2:172.21.15.191:3300/0 conn(0x7faaac14ddb0 0x7faaac14e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab188e700 1 -- 172.21.15.175:0/355836992 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac14e750 con 0x7faaac0c1be0 2023-10-10T18:52:26.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab188e700 1 --2- 172.21.15.175:0/355836992 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac06b760 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faaac06d380 tx=0x7faaa8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd10bf48d8d9c19 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:26.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab088c700 1 -- 172.21.15.175:0/355836992 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8005c10 con 0x7faaac0c1be0 2023-10-10T18:52:26.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab088c700 1 -- 172.21.15.175:0/355836992 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa8005d70 con 0x7faaac0c1be0 2023-10-10T18:52:26.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.182+0000 7faab088c700 1 -- 172.21.15.175:0/355836992 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa800b880 con 0x7faaac0c1be0 2023-10-10T18:52:26.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 -- 172.21.15.175:0/355836992 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 msgr2=0x7faaac06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:26.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 --2- 172.21.15.175:0/355836992 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac06b760 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faaac06d380 tx=0x7faaa8009220 comp rx=0 tx=0).stop 2023-10-10T18:52:26.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 -- 172.21.15.175:0/355836992 shutdown_connections 2023-10-10T18:52:26.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 --2- 172.21.15.175:0/355836992 >> v2:172.21.15.191:3300/0 conn(0x7faaac14ddb0 0x7faaac14e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 --2- 172.21.15.175:0/355836992 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac06b760 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 --2- 172.21.15.175:0/355836992 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 0x7faaac06a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 -- 172.21.15.175:0/355836992 >> 172.21.15.175:0/355836992 conn(0x7faaac068280 msgr2=0x7faaac06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:26.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 -- 172.21.15.175:0/355836992 shutdown_connections 2023-10-10T18:52:26.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.183+0000 7faab3af2700 1 -- 172.21.15.175:0/355836992 wait complete. 2023-10-10T18:52:26.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.184+0000 7faab3af2700 1 Processor -- start 2023-10-10T18:52:26.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.184+0000 7faab3af2700 1 -- start start 2023-10-10T18:52:26.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.184+0000 7faab3af2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac14ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:26.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.184+0000 7faab3af2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 0x7faaac141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:26.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.184+0000 7faab3af2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac14ddb0 0x7faaac141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:26.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.184+0000 7faab3af2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaac1672f0 con 0x7faaac0c1be0 2023-10-10T18:52:26.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.184+0000 7faab3af2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaac167e70 con 0x7faaac14ddb0 2023-10-10T18:52:26.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.184+0000 7faab3af2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaac14e480 con 0x7faaac0c1fb0 2023-10-10T18:52:26.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac14ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:26.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac14ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51264/0 (socket says 172.21.15.175:51264) 2023-10-10T18:52:26.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 -- 172.21.15.175:0/3573689651 learned_addr learned my addr 172.21.15.175:0/3573689651 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:26.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab208f700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac14ddb0 0x7faaac141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:26.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab108d700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 0x7faaac141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:26.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 -- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 msgr2=0x7faaac141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:26.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 0x7faaac141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 -- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac14ddb0 msgr2=0x7faaac141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac14ddb0 0x7faaac141580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 -- 172.21.15.175:0/3573689651 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa8009480 con 0x7faaac0c1be0 2023-10-10T18:52:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab188e700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac14ac60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009fd0 tx=0x7faaa8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faaa1ffb700 1 -- 172.21.15.175:0/3573689651 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8011910 con 0x7faaac0c1be0 2023-10-10T18:52:26.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faaa1ffb700 1 -- 172.21.15.175:0/3573689651 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa8020070 con 0x7faaac0c1be0 2023-10-10T18:52:26.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaac14ca30 con 0x7faaac0c1be0 2023-10-10T18:52:26.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faaa1ffb700 1 -- 172.21.15.175:0/3573689651 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8011ca0 con 0x7faaac0c1be0 2023-10-10T18:52:26.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.185+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaac167820 con 0x7faaac0c1be0 2023-10-10T18:52:26.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.186+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaac067570 con 0x7faaac0c1be0 2023-10-10T18:52:26.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.187+0000 7faaa1ffb700 1 -- 172.21.15.175:0/3573689651 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faaa8022030 con 0x7faaac0c1be0 2023-10-10T18:52:26.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.187+0000 7faaa1ffb700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faa90066f30 0x7faa900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:26.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.187+0000 7faaa1ffb700 1 -- 172.21.15.175:0/3573689651 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7faaa8086310 con 0x7faaac0c1be0 2023-10-10T18:52:26.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.187+0000 7faab108d700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faa90066f30 0x7faa900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:26.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.188+0000 7faab108d700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faa90066f30 0x7faa900693e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faaac1422c0 tx=0x7faa9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:26.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.191+0000 7faaa1ffb700 1 -- 172.21.15.175:0/3573689651 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faaa8053410 con 0x7faaac0c1be0 2023-10-10T18:52:26.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.416+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaac168150 con 0x7faaac0c1be0 2023-10-10T18:52:26.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.416+0000 7faaa1ffb700 1 -- 172.21.15.175:0/3573689651 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5453 (secure 0 0 0) 0x7faaa8056a30 con 0x7faaac0c1be0 2023-10-10T18:52:26.417 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:26.417 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:23.469100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:26.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.418+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faa90066f30 msgr2=0x7faa900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:26.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.418+0000 7faab3af2700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faa90066f30 0x7faa900693e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faaac1422c0 tx=0x7faa9c008040 comp rx=0 tx=0).stop 2023-10-10T18:52:26.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.418+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 msgr2=0x7faaac14ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:26.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.418+0000 7faab3af2700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac14ac60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009fd0 tx=0x7faaa8016000 comp rx=0 tx=0).stop 2023-10-10T18:52:26.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.419+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 shutdown_connections 2023-10-10T18:52:26.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.419+0000 7faab3af2700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faa90066f30 0x7faa900693e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.419+0000 7faab3af2700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaac14ddb0 0x7faaac141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.419+0000 7faab3af2700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaac0c1be0 0x7faaac14ac60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.419+0000 7faab3af2700 1 --2- 172.21.15.175:0/3573689651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faaac0c1fb0 0x7faaac141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:26.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.419+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 >> 172.21.15.175:0/3573689651 conn(0x7faaac068280 msgr2=0x7faaac06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:26.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.419+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 shutdown_connections 2023-10-10T18:52:26.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:26.419+0000 7faab3af2700 1 -- 172.21.15.175:0/3573689651 wait complete. 2023-10-10T18:52:26.421 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 57 2023-10-10T18:52:26.431 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:28.432 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:28.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.589+0000 7f5284cbd700 1 Processor -- start 2023-10-10T18:52:28.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.589+0000 7f5284cbd700 1 -- start start 2023-10-10T18:52:28.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f5284cbd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52800c12a0 0x7f52801577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:28.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f5284cbd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52800c4f20 0x7f528014d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:28.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f5284cbd700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5280154aa0 con 0x7f52801573d0 2023-10-10T18:52:28.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f5284cbd700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52800bfdf0 con 0x7f52800c4f20 2023-10-10T18:52:28.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f5284cbd700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5280063dc0 con 0x7f52800c12a0 2023-10-10T18:52:28.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52800c4f20 0x7f528014d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:28.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527e59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f52801573d0 0x7f52800bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40934/0 (socket says 172.21.15.175:40934) 2023-10-10T18:52:28.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527e59c700 1 -- 172.21.15.175:0/1196819812 learned_addr learned my addr 172.21.15.175:0/1196819812 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:28.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527dd9b700 1 --2- 172.21.15.175:0/1196819812 >> v2:172.21.15.191:3300/0 conn(0x7f52800c12a0 0x7f52801577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:28.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527ed9d700 1 -- 172.21.15.175:0/1196819812 >> v2:172.21.15.191:3300/0 conn(0x7f52800c12a0 msgr2=0x7f52801577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:28.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527ed9d700 1 --2- 172.21.15.175:0/1196819812 >> v2:172.21.15.191:3300/0 conn(0x7f52800c12a0 0x7f52801577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527ed9d700 1 -- 172.21.15.175:0/1196819812 >> v1:172.21.15.175:6789/0 conn(0x7f52801573d0 legacy=0x7f52800bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:28.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527ed9d700 1 -- 172.21.15.175:0/1196819812 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52800c5320 con 0x7f52800c4f20 2023-10-10T18:52:28.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527d59a700 1 -- 172.21.15.175:0/1196819812 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735999081 0 0) 0x7f5280154aa0 con 0x7f52801573d0 2023-10-10T18:52:28.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527ed9d700 1 --2- 172.21.15.175:0/1196819812 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f528014d450 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f52800c6ee0 tx=0x7f5278009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a83f2804f508a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:28.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.591+0000 7f527d59a700 1 -- 172.21.15.175:0/1196819812 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527800e070 con 0x7f52800c4f20 2023-10-10T18:52:28.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f527d59a700 1 -- 172.21.15.175:0/1196819812 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527800bb60 con 0x7f52800c4f20 2023-10-10T18:52:28.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f527d59a700 1 -- 172.21.15.175:0/1196819812 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278012430 con 0x7f52800c4f20 2023-10-10T18:52:28.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f5284cbd700 1 -- 172.21.15.175:0/1196819812 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 msgr2=0x7f528014d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:28.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f5284cbd700 1 --2- 172.21.15.175:0/1196819812 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f528014d450 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f52800c6ee0 tx=0x7f5278009780 comp rx=0 tx=0).stop 2023-10-10T18:52:28.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f5284cbd700 1 -- 172.21.15.175:0/1196819812 shutdown_connections 2023-10-10T18:52:28.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f5284cbd700 1 --2- 172.21.15.175:0/1196819812 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f528014d450 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f5284cbd700 1 --2- 172.21.15.175:0/1196819812 >> v2:172.21.15.191:3300/0 conn(0x7f52800c12a0 0x7f52801577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f5284cbd700 1 -- 172.21.15.175:0/1196819812 >> 172.21.15.175:0/1196819812 conn(0x7f528006dc70 msgr2=0x7f5280063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:28.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f5284cbd700 1 -- 172.21.15.175:0/1196819812 shutdown_connections 2023-10-10T18:52:28.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.592+0000 7f5284cbd700 1 -- 172.21.15.175:0/1196819812 wait complete. 2023-10-10T18:52:28.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.593+0000 7f5284cbd700 1 Processor -- start 2023-10-10T18:52:28.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.593+0000 7f5284cbd700 1 -- start start 2023-10-10T18:52:28.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f5284cbd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52800c12a0 0x7f5280154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:28.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f5284cbd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f52801487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:28.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f5284cbd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52801573d0 0x7f5280148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:28.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f5284cbd700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528013ffb0 con 0x7f52800c4f20 2023-10-10T18:52:28.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f5284cbd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5280152880 con 0x7f52800c12a0 2023-10-10T18:52:28.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f5284cbd700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52801483c0 con 0x7f52801573d0 2023-10-10T18:52:28.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f52801487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:28.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f52801487d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51286/0 (socket says 172.21.15.175:51286) 2023-10-10T18:52:28.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 -- 172.21.15.175:0/2409534995 learned_addr learned my addr 172.21.15.175:0/2409534995 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:28.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527e59c700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52800c12a0 0x7f5280154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:28.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527ed9d700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52801573d0 0x7f5280148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:28.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 -- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52801573d0 msgr2=0x7f5280148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:28.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52801573d0 0x7f5280148d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 -- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52800c12a0 msgr2=0x7f5280154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:28.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52800c12a0 0x7f5280154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 -- 172.21.15.175:0/2409534995 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5278009a20 con 0x7f52800c4f20 2023-10-10T18:52:28.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527ed9d700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52801573d0 0x7f5280148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:28.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.594+0000 7f527dd9b700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f52801487d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f527400ece0 tx=0x7f52740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:28.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.595+0000 7f526e7fc700 1 -- 172.21.15.175:0/2409534995 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527400ca60 con 0x7f52800c4f20 2023-10-10T18:52:28.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.595+0000 7f526e7fc700 1 -- 172.21.15.175:0/2409534995 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527400cbc0 con 0x7f52800c4f20 2023-10-10T18:52:28.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.595+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5280155290 con 0x7f52800c4f20 2023-10-10T18:52:28.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.595+0000 7f526e7fc700 1 -- 172.21.15.175:0/2409534995 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5274013470 con 0x7f52800c4f20 2023-10-10T18:52:28.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.595+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52801480e0 con 0x7f52800c4f20 2023-10-10T18:52:28.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.595+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52800c6ce0 con 0x7f52800c4f20 2023-10-10T18:52:28.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.596+0000 7f526e7fc700 1 -- 172.21.15.175:0/2409534995 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5274013820 con 0x7f52800c4f20 2023-10-10T18:52:28.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.596+0000 7f526e7fc700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5260066f60 0x7f5260069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:28.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.597+0000 7f526e7fc700 1 -- 172.21.15.175:0/2409534995 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f5274084a60 con 0x7f52800c4f20 2023-10-10T18:52:28.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.597+0000 7f527e59c700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5260066f60 0x7f5260069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:28.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.597+0000 7f527e59c700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5260066f60 0x7f5260069410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f527000c8f0 tx=0x7f5270009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:28.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.601+0000 7f526e7fc700 1 -- 172.21.15.175:0/2409534995 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5274051b60 con 0x7f52800c4f20 2023-10-10T18:52:28.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.834+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52801529c0 con 0x7f52800c4f20 2023-10-10T18:52:28.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.835+0000 7f526e7fc700 1 -- 172.21.15.175:0/2409534995 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5453 (secure 0 0 0) 0x7f5274055180 con 0x7f52800c4f20 2023-10-10T18:52:28.835 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:28.835 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:23.469100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:28.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5260066f60 msgr2=0x7f5260069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:28.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5260066f60 0x7f5260069410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f527000c8f0 tx=0x7f5270009040 comp rx=0 tx=0).stop 2023-10-10T18:52:28.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 msgr2=0x7f52801487d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:28.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f52801487d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f527400ece0 tx=0x7f52740091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 shutdown_connections 2023-10-10T18:52:28.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5260066f60 0x7f5260069410 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52801573d0 0x7f5280148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52800c12a0 0x7f5280154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 --2- 172.21.15.175:0/2409534995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52800c4f20 0x7f52801487d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:28.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 >> 172.21.15.175:0/2409534995 conn(0x7f528006dc70 msgr2=0x7f528006b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:28.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 shutdown_connections 2023-10-10T18:52:28.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:28.838+0000 7f5284cbd700 1 -- 172.21.15.175:0/2409534995 wait complete. 2023-10-10T18:52:28.841 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 57 2023-10-10T18:52:28.850 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:30.851 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:30.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.979+0000 7f9edb07a700 1 Processor -- start 2023-10-10T18:52:30.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.979+0000 7f9edb07a700 1 -- start start 2023-10-10T18:52:30.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.979+0000 7f9edb07a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed40b1fc0 0x7f9ed4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:30.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.979+0000 7f9edb07a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9ed40b2390 0x7f9ed4068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:30.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.979+0000 7f9edb07a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed41574b0 con 0x7f9ed4068bd0 2023-10-10T18:52:30.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.979+0000 7f9edb07a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed4150770 con 0x7f9ed40b2390 2023-10-10T18:52:30.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.979+0000 7f9edb07a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40afad0 con 0x7f9ed40b1fc0 2023-10-10T18:52:30.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed3fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9ed40b2390 0x7f9ed4068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:30.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed9617700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9ed4068bd0 0x7f9ed40aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40948/0 (socket says 172.21.15.175:40948) 2023-10-10T18:52:30.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed9617700 1 -- 172.21.15.175:0/564913774 learned_addr learned my addr 172.21.15.175:0/564913774 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:30.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed8e16700 1 --2- 172.21.15.175:0/564913774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed40b1fc0 0x7f9ed4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:30.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed3fff700 1 -- 172.21.15.175:0/564913774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed40b1fc0 msgr2=0x7f9ed4155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:30.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed3fff700 1 --2- 172.21.15.175:0/564913774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed40b1fc0 0x7f9ed4155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:30.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed3fff700 1 -- 172.21.15.175:0/564913774 >> v1:172.21.15.175:6789/0 conn(0x7f9ed4068bd0 legacy=0x7f9ed40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:30.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed3fff700 1 -- 172.21.15.175:0/564913774 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed40af4c0 con 0x7f9ed40b2390 2023-10-10T18:52:30.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed37fe700 1 -- 172.21.15.175:0/564913774 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512838932 0 0) 0x7f9ed41574b0 con 0x7f9ed4068bd0 2023-10-10T18:52:30.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed8e16700 1 --2- 172.21.15.175:0/564913774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed40b1fc0 0x7f9ed4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:30.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed3fff700 1 --2- 172.21.15.175:0/564913774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed4068690 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80098e0 tx=0x7f9ec80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d5b4766b1f8846 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:30.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed37fe700 1 -- 172.21.15.175:0/564913774 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8009e80 con 0x7f9ed40b2390 2023-10-10T18:52:30.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed37fe700 1 -- 172.21.15.175:0/564913774 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec800b8f0 con 0x7f9ed40b2390 2023-10-10T18:52:30.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.980+0000 7f9ed37fe700 1 -- 172.21.15.175:0/564913774 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800bc70 con 0x7f9ed40b2390 2023-10-10T18:52:30.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.981+0000 7f9edb07a700 1 -- 172.21.15.175:0/564913774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 msgr2=0x7f9ed4068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:30.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.981+0000 7f9edb07a700 1 --2- 172.21.15.175:0/564913774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed4068690 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80098e0 tx=0x7f9ec80091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:30.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.981+0000 7f9edb07a700 1 -- 172.21.15.175:0/564913774 shutdown_connections 2023-10-10T18:52:30.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.981+0000 7f9edb07a700 1 --2- 172.21.15.175:0/564913774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed40b1fc0 0x7f9ed4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:30.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.981+0000 7f9edb07a700 1 --2- 172.21.15.175:0/564913774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed4068690 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:30.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.981+0000 7f9edb07a700 1 -- 172.21.15.175:0/564913774 >> 172.21.15.175:0/564913774 conn(0x7f9ed4063230 msgr2=0x7f9ed4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:30.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.981+0000 7f9edb07a700 1 -- 172.21.15.175:0/564913774 shutdown_connections 2023-10-10T18:52:30.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.981+0000 7f9edb07a700 1 -- 172.21.15.175:0/564913774 wait complete. 2023-10-10T18:52:30.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.982+0000 7f9edb07a700 1 Processor -- start 2023-10-10T18:52:30.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.982+0000 7f9edb07a700 1 -- start start 2023-10-10T18:52:30.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9edb07a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed4068bd0 0x7f9ed40cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:30.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9edb07a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed40b1fc0 0x7f9ed4165420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:30.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9edb07a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed40c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:31.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9edb07a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40cc260 con 0x7f9ed40b2390 2023-10-10T18:52:31.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9edb07a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40ca5a0 con 0x7f9ed40b1fc0 2023-10-10T18:52:31.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9edb07a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40ca840 con 0x7f9ed4068bd0 2023-10-10T18:52:31.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed40c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:31.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed40c6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51308/0 (socket says 172.21.15.175:51308) 2023-10-10T18:52:31.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 -- 172.21.15.175:0/3135491188 learned_addr learned my addr 172.21.15.175:0/3135491188 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:31.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed8e16700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed4068bd0 0x7f9ed40cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:31.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed3fff700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed40b1fc0 0x7f9ed4165420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:31.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 -- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed4068bd0 msgr2=0x7f9ed40cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:31.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed4068bd0 0x7f9ed40cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:31.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 -- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed40b1fc0 msgr2=0x7f9ed4165420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:31.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed40b1fc0 0x7f9ed4165420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:31.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 -- 172.21.15.175:0/3135491188 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec8009420 con 0x7f9ed40b2390 2023-10-10T18:52:31.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.983+0000 7f9ed9617700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed40c6060 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c5b0 tx=0x7f9ecc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:31.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.984+0000 7f9ed0ff9700 1 -- 172.21.15.175:0/3135491188 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc005b30 con 0x7f9ed40b2390 2023-10-10T18:52:31.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.984+0000 7f9ed0ff9700 1 -- 172.21.15.175:0/3135491188 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc005c90 con 0x7f9ed40b2390 2023-10-10T18:52:31.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.984+0000 7f9ed0ff9700 1 -- 172.21.15.175:0/3135491188 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc007300 con 0x7f9ed40b2390 2023-10-10T18:52:31.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.984+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed40cab20 con 0x7f9ed40b2390 2023-10-10T18:52:31.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.984+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed40cbef0 con 0x7f9ed40b2390 2023-10-10T18:52:31.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.985+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed4067570 con 0x7f9ed40b2390 2023-10-10T18:52:31.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.985+0000 7f9ed0ff9700 1 -- 172.21.15.175:0/3135491188 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9ecc007670 con 0x7f9ed40b2390 2023-10-10T18:52:31.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.985+0000 7f9ed0ff9700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ec0066f30 0x7f9ec00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:31.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.985+0000 7f9ed0ff9700 1 -- 172.21.15.175:0/3135491188 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f9ecc084190 con 0x7f9ed40b2390 2023-10-10T18:52:31.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.985+0000 7f9ed8e16700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ec0066f30 0x7f9ec00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:31.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.986+0000 7f9ed8e16700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ec0066f30 0x7f9ec00693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400e650 tx=0x7f9ec4007710 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:31.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:30.989+0000 7f9ed0ff9700 1 -- 172.21.15.175:0/3135491188 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ecc051310 con 0x7f9ed40b2390 2023-10-10T18:52:31.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.213+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ed4069060 con 0x7f9ed40b2390 2023-10-10T18:52:31.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.214+0000 7f9ed0ff9700 1 -- 172.21.15.175:0/3135491188 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5455 (secure 0 0 0) 0x7f9ecc054930 con 0x7f9ed40b2390 2023-10-10T18:52:31.214 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:31.214 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:29.649116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:resolve","state_seq":37,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:31.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ec0066f30 msgr2=0x7f9ec00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:31.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ec0066f30 0x7f9ec00693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400e650 tx=0x7f9ec4007710 comp rx=0 tx=0).stop 2023-10-10T18:52:31.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 msgr2=0x7f9ed40c6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:31.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed40c6060 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c5b0 tx=0x7f9ecc007d30 comp rx=0 tx=0).stop 2023-10-10T18:52:31.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 shutdown_connections 2023-10-10T18:52:31.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ec0066f30 0x7f9ec00693e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:31.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ed40b2390 0x7f9ed40c6060 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:31.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ed4068bd0 0x7f9ed40cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:31.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 --2- 172.21.15.175:0/3135491188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed40b1fc0 0x7f9ed4165420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:31.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 >> 172.21.15.175:0/3135491188 conn(0x7f9ed4063230 msgr2=0x7f9ed4152620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:31.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 shutdown_connections 2023-10-10T18:52:31.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:31.216+0000 7f9edb07a700 1 -- 172.21.15.175:0/3135491188 wait complete. 2023-10-10T18:52:31.218 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 58 2023-10-10T18:52:31.228 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:33.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:33.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.358+0000 7f683c753700 1 Processor -- start 2023-10-10T18:52:33.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.358+0000 7f683c753700 1 -- start start 2023-10-10T18:52:33.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683c753700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6834155fb0 0x7f6834159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:33.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683c753700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68341574b0 con 0x7f68340b2390 2023-10-10T18:52:33.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683c753700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340afad0 con 0x7f68340b1fc0 2023-10-10T18:52:33.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683c753700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6834150770 con 0x7f6834155fb0 2023-10-10T18:52:33.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683acf0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6834155fb0 0x7f6834159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:33.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f6839cee700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f68340b2390 0x7f68340aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40964/0 (socket says 172.21.15.175:40964) 2023-10-10T18:52:33.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f6839cee700 1 -- 172.21.15.175:0/3573553212 learned_addr learned my addr 172.21.15.175:0/3573553212 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:33.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683acf0700 1 -- 172.21.15.175:0/3573553212 >> v1:172.21.15.191:6789/0 conn(0x7f68340b1fc0 legacy=0x7f6834068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:33.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683acf0700 1 -- 172.21.15.175:0/3573553212 >> v1:172.21.15.175:6789/0 conn(0x7f68340b2390 legacy=0x7f68340aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:33.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683acf0700 1 -- 172.21.15.175:0/3573553212 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68340af4c0 con 0x7f6834155fb0 2023-10-10T18:52:33.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f683acf0700 1 --2- 172.21.15.175:0/3573553212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6834155fb0 0x7f6834159920 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f68240098e0 tx=0x7f68240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee12c3406b6ba8a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:33.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f68394ed700 1 -- 172.21.15.175:0/3573553212 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824009e80 con 0x7f6834155fb0 2023-10-10T18:52:33.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f68394ed700 1 -- 172.21.15.175:0/3573553212 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682400b8f0 con 0x7f6834155fb0 2023-10-10T18:52:33.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.359+0000 7f68394ed700 1 -- 172.21.15.175:0/3573553212 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682400bc70 con 0x7f6834155fb0 2023-10-10T18:52:33.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.360+0000 7f683c753700 1 -- 172.21.15.175:0/3573553212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6834155fb0 msgr2=0x7f6834159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:33.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.360+0000 7f683c753700 1 --2- 172.21.15.175:0/3573553212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6834155fb0 0x7f6834159920 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f68240098e0 tx=0x7f68240091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.360+0000 7f683c753700 1 -- 172.21.15.175:0/3573553212 shutdown_connections 2023-10-10T18:52:33.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.360+0000 7f683c753700 1 --2- 172.21.15.175:0/3573553212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6834155fb0 0x7f6834159920 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.360+0000 7f683c753700 1 -- 172.21.15.175:0/3573553212 >> 172.21.15.175:0/3573553212 conn(0x7f6834063230 msgr2=0x7f6834063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:33.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.360+0000 7f683c753700 1 -- 172.21.15.175:0/3573553212 shutdown_connections 2023-10-10T18:52:33.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.360+0000 7f683c753700 1 -- 172.21.15.175:0/3573553212 wait complete. 2023-10-10T18:52:33.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.361+0000 7f683c753700 1 Processor -- start 2023-10-10T18:52:33.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.361+0000 7f683c753700 1 -- start start 2023-10-10T18:52:33.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683c753700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68340b1fc0 0x7f68340ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:33.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683c753700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68340b2390 0x7f68341518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:33.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683c753700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6834155fb0 0x7f6834166390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:33.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683c753700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834165f00 con 0x7f68340b1fc0 2023-10-10T18:52:33.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683c753700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834150a50 con 0x7f6834155fb0 2023-10-10T18:52:33.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683c753700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6834165bb0 con 0x7f68340b2390 2023-10-10T18:52:33.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68340b1fc0 0x7f68340ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:33.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68340b1fc0 0x7f68340ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51330/0 (socket says 172.21.15.175:51330) 2023-10-10T18:52:33.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 -- 172.21.15.175:0/2523189161 learned_addr learned my addr 172.21.15.175:0/2523189161 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:33.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683acf0700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6834155fb0 0x7f6834166390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:33.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f6839cee700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68340b2390 0x7f68341518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:33.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 -- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68340b2390 msgr2=0x7f68341518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:33.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68340b2390 0x7f68341518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 -- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6834155fb0 msgr2=0x7f6834166390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:33.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6834155fb0 0x7f6834166390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 -- 172.21.15.175:0/2523189161 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824009420 con 0x7f68340b1fc0 2023-10-10T18:52:33.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f683a4ef700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68340b1fc0 0x7f68340ccd30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f682c00cb10 tx=0x7f682c0017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:33.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.362+0000 7f68327fc700 1 -- 172.21.15.175:0/2523189161 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c00ce20 con 0x7f68340b1fc0 2023-10-10T18:52:33.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.363+0000 7f68327fc700 1 -- 172.21.15.175:0/2523189161 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c00a7e0 con 0x7f68340b1fc0 2023-10-10T18:52:33.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.363+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6834150cf0 con 0x7f68340b1fc0 2023-10-10T18:52:33.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.363+0000 7f68327fc700 1 -- 172.21.15.175:0/2523189161 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c00aba0 con 0x7f68340b1fc0 2023-10-10T18:52:33.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.363+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68341512b0 con 0x7f68340b1fc0 2023-10-10T18:52:33.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.363+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68341661e0 con 0x7f68340b1fc0 2023-10-10T18:52:33.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.364+0000 7f68327fc700 1 -- 172.21.15.175:0/2523189161 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f682c003030 con 0x7f68340b1fc0 2023-10-10T18:52:33.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.364+0000 7f68327fc700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6820066f30 0x7f68200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:33.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.364+0000 7f68327fc700 1 -- 172.21.15.175:0/2523189161 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f682c084630 con 0x7f68340b1fc0 2023-10-10T18:52:33.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.365+0000 7f6839cee700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6820066f30 0x7f68200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:33.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.365+0000 7f6839cee700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6820066f30 0x7f68200693e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6834068a10 tx=0x7f6828007eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:33.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.369+0000 7f68327fc700 1 -- 172.21.15.175:0/2523189161 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f682c051730 con 0x7f68340b1fc0 2023-10-10T18:52:33.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.592+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6834067570 con 0x7f68340b1fc0 2023-10-10T18:52:33.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.592+0000 7f68327fc700 1 -- 172.21.15.175:0/2523189161 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5453 (secure 0 0 0) 0x7f682c054d50 con 0x7f68340b1fc0 2023-10-10T18:52:33.593 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:33.593 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:32.659064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:resolve","state_seq":37,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:33.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6820066f30 msgr2=0x7f68200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:33.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6820066f30 0x7f68200693e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6834068a10 tx=0x7f6828007eb0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68340b1fc0 msgr2=0x7f68340ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:33.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68340b1fc0 0x7f68340ccd30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f682c00cb10 tx=0x7f682c0017b0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 shutdown_connections 2023-10-10T18:52:33.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6820066f30 0x7f68200693e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6834155fb0 0x7f6834166390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68340b1fc0 0x7f68340ccd30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 --2- 172.21.15.175:0/2523189161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68340b2390 0x7f68341518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:33.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.594+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 >> 172.21.15.175:0/2523189161 conn(0x7f6834063230 msgr2=0x7f68340c8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:33.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.595+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 shutdown_connections 2023-10-10T18:52:33.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:33.595+0000 7f683c753700 1 -- 172.21.15.175:0/2523189161 wait complete. 2023-10-10T18:52:33.597 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 59 2023-10-10T18:52:33.606 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:35.607 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:35.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.736+0000 7f844ee35700 1 Processor -- start 2023-10-10T18:52:35.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.736+0000 7f844ee35700 1 -- start start 2023-10-10T18:52:35.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.736+0000 7f844ee35700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f84480c35e0 0x7f8448159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:35.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.736+0000 7f844ee35700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84480c9390 0x7f84480c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:35.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.736+0000 7f844ee35700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84480c5e40 con 0x7f84480c3210 2023-10-10T18:52:35.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.736+0000 7f844ee35700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84480c4c50 con 0x7f84480c35e0 2023-10-10T18:52:35.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.736+0000 7f844ee35700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84480ca400 con 0x7f84480c9390 2023-10-10T18:52:35.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f84480c35e0 0x7f8448159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:35.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f84480c35e0 0x7f8448159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59220/0 (socket says 172.21.15.175:59220) 2023-10-10T18:52:35.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843ffff700 1 -- 172.21.15.175:0/3715342720 learned_addr learned my addr 172.21.15.175:0/3715342720 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:35.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f844d3d2700 1 --2- 172.21.15.175:0/3715342720 >> v2:172.21.15.191:3300/0 conn(0x7f84480c9390 0x7f84480c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:35.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843ffff700 1 -- 172.21.15.175:0/3715342720 >> v2:172.21.15.191:3300/0 conn(0x7f84480c9390 msgr2=0x7f84480c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:35.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843ffff700 1 --2- 172.21.15.175:0/3715342720 >> v2:172.21.15.191:3300/0 conn(0x7f84480c9390 0x7f84480c62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843ffff700 1 -- 172.21.15.175:0/3715342720 >> v1:172.21.15.191:6789/0 conn(0x7f84480c3210 legacy=0x7f84480c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:35.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843ffff700 1 -- 172.21.15.175:0/3715342720 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84480c9880 con 0x7f84480c35e0 2023-10-10T18:52:35.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843ffff700 1 --2- 172.21.15.175:0/3715342720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448159fa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8448067050 tx=0x7f8438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad0b9ca4f6913aa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:35.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843f7fe700 1 -- 172.21.15.175:0/3715342720 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843800b820 con 0x7f84480c35e0 2023-10-10T18:52:35.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843f7fe700 1 -- 172.21.15.175:0/3715342720 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843800b980 con 0x7f84480c35e0 2023-10-10T18:52:35.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.737+0000 7f843f7fe700 1 -- 172.21.15.175:0/3715342720 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843800bd20 con 0x7f84480c35e0 2023-10-10T18:52:35.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.738+0000 7f844ee35700 1 -- 172.21.15.175:0/3715342720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 msgr2=0x7f8448159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:35.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.738+0000 7f844ee35700 1 --2- 172.21.15.175:0/3715342720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448159fa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8448067050 tx=0x7f8438009250 comp rx=0 tx=0).stop 2023-10-10T18:52:35.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.738+0000 7f844ee35700 1 -- 172.21.15.175:0/3715342720 shutdown_connections 2023-10-10T18:52:35.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.738+0000 7f844ee35700 1 --2- 172.21.15.175:0/3715342720 >> v2:172.21.15.191:3300/0 conn(0x7f84480c9390 0x7f84480c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.738+0000 7f844ee35700 1 --2- 172.21.15.175:0/3715342720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448159fa0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.738+0000 7f844ee35700 1 -- 172.21.15.175:0/3715342720 >> 172.21.15.175:0/3715342720 conn(0x7f844806c110 msgr2=0x7f844806c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:35.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.738+0000 7f844ee35700 1 -- 172.21.15.175:0/3715342720 shutdown_connections 2023-10-10T18:52:35.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.738+0000 7f844ee35700 1 -- 172.21.15.175:0/3715342720 wait complete. 2023-10-10T18:52:35.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.739+0000 7f844ee35700 1 Processor -- start 2023-10-10T18:52:35.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.739+0000 7f844ee35700 1 -- start start 2023-10-10T18:52:35.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844ee35700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84480c3210 0x7f8448152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:35.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844ee35700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:35.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844ee35700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84480c9390 0x7f84480bfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:35.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844ee35700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84480c5840 con 0x7f84480c35e0 2023-10-10T18:52:35.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844ee35700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8448167310 con 0x7f84480c9390 2023-10-10T18:52:35.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844ee35700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84481525c0 con 0x7f84480c3210 2023-10-10T18:52:35.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:35.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448152f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59232/0 (socket says 172.21.15.175:59232) 2023-10-10T18:52:35.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 -- 172.21.15.175:0/2073886268 learned_addr learned my addr 172.21.15.175:0/2073886268 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:35.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844d3d2700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84480c9390 0x7f84480bfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:35.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844cbd1700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84480c3210 0x7f8448152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:35.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 -- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84480c3210 msgr2=0x7f8448152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:35.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84480c3210 0x7f8448152a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 -- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84480c9390 msgr2=0x7f84480bfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:35.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84480c9390 0x7f84480bfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 -- 172.21.15.175:0/2073886268 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84380094f0 con 0x7f84480c35e0 2023-10-10T18:52:35.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f844d3d2700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84480c9390 0x7f84480bfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:35.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843ffff700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448152f90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8438009290 tx=0x7f8438017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:35.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.740+0000 7f843cff9700 1 -- 172.21.15.175:0/2073886268 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843800bd00 con 0x7f84480c35e0 2023-10-10T18:52:35.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.741+0000 7f843cff9700 1 -- 172.21.15.175:0/2073886268 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8438024440 con 0x7f84480c35e0 2023-10-10T18:52:35.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.741+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8448167030 con 0x7f84480c35e0 2023-10-10T18:52:35.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.741+0000 7f843cff9700 1 -- 172.21.15.175:0/2073886268 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84380247c0 con 0x7f84480c35e0 2023-10-10T18:52:35.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.741+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8448156d70 con 0x7f84480c35e0 2023-10-10T18:52:35.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.741+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8448152750 con 0x7f84480c35e0 2023-10-10T18:52:35.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.742+0000 7f843cff9700 1 -- 172.21.15.175:0/2073886268 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f843801a420 con 0x7f84480c35e0 2023-10-10T18:52:35.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.742+0000 7f843cff9700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8434066f60 0x7f8434069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:35.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.742+0000 7f843cff9700 1 -- 172.21.15.175:0/2073886268 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f84380865b0 con 0x7f84480c35e0 2023-10-10T18:52:35.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.742+0000 7f844cbd1700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8434066f60 0x7f8434069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:35.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.743+0000 7f844cbd1700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8434066f60 0x7f8434069410 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8444007fd0 tx=0x7f8444009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:35.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.747+0000 7f843cff9700 1 -- 172.21.15.175:0/2073886268 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8438053730 con 0x7f84480c35e0 2023-10-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.970+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8448067570 con 0x7f84480c35e0 2023-10-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.971+0000 7f843cff9700 1 -- 172.21.15.175:0/2073886268 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5453 (secure 0 0 0) 0x7f8438056d50 con 0x7f84480c35e0 2023-10-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:32.659064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:resolve","state_seq":37,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:35.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8434066f60 msgr2=0x7f8434069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:35.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8434066f60 0x7f8434069410 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8444007fd0 tx=0x7f8444009040 comp rx=0 tx=0).stop 2023-10-10T18:52:35.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 msgr2=0x7f8448152f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:35.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448152f90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8438009290 tx=0x7f8438017000 comp rx=0 tx=0).stop 2023-10-10T18:52:35.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 shutdown_connections 2023-10-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8434066f60 0x7f8434069410 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84480c9390 0x7f84480bfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84480c3210 0x7f8448152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 --2- 172.21.15.175:0/2073886268 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84480c35e0 0x7f8448152f90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:35.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 >> 172.21.15.175:0/2073886268 conn(0x7f844806c110 msgr2=0x7f84480c1830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:35.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 shutdown_connections 2023-10-10T18:52:35.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:35.973+0000 7f844ee35700 1 -- 172.21.15.175:0/2073886268 wait complete. 2023-10-10T18:52:35.976 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 59 2023-10-10T18:52:35.985 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 59, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 59, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4250": { "addr": "172.21.15.191:6839/246221790", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 246221790, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 246221790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4250, "incarnation": 12, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:resolve", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:52:32.659064+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 4250, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:52:37.986 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:38.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.113+0000 7f75b883d700 1 Processor -- start 2023-10-10T18:52:38.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.113+0000 7f75b883d700 1 -- start start 2023-10-10T18:52:38.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b883d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75b408a100 0x7f75b4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:38.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b883d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:38.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b883d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b4067fa0 con 0x7f75b414fd90 2023-10-10T18:52:38.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b883d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b40c5600 con 0x7f75b408a100 2023-10-10T18:52:38.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b883d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4156b70 con 0x7f75b40cd040 2023-10-10T18:52:38.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b359e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:38.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75aa59c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75b408a100 0x7f75b4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:38.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75aa59c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75b408a100 0x7f75b4159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:48346/0 (socket says 172.21.15.175:48346) 2023-10-10T18:52:38.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75aa59c700 1 -- 172.21.15.175:0/1136768158 learned_addr learned my addr 172.21.15.175:0/1136768158 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:38.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b359e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:50970/0 (socket says 172.21.15.175:50970) 2023-10-10T18:52:38.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b359e700 1 -- 172.21.15.175:0/1136768158 >> v2:172.21.15.191:3300/0 conn(0x7f75b408a100 msgr2=0x7f75b4159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:38.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b359e700 1 --2- 172.21.15.175:0/1136768158 >> v2:172.21.15.191:3300/0 conn(0x7f75b408a100 0x7f75b4159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b359e700 1 -- 172.21.15.175:0/1136768158 >> v1:172.21.15.191:6789/0 conn(0x7f75b414fd90 legacy=0x7f75b40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:38.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75b359e700 1 -- 172.21.15.175:0/1136768158 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b4150160 con 0x7f75b40cd040 2023-10-10T18:52:38.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.114+0000 7f75aa59c700 1 --2- 172.21.15.175:0/1136768158 >> v2:172.21.15.191:3300/0 conn(0x7f75b408a100 0x7f75b4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:52:38.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b359e700 1 --2- 172.21.15.175:0/1136768158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40684f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009e60 tx=0x7f75ac009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43ddb0fa8fa6a41f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:38.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b259c700 1 -- 172.21.15.175:0/1136768158 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00e070 con 0x7f75b40cd040 2023-10-10T18:52:38.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b259c700 1 -- 172.21.15.175:0/1136768158 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac00bb60 con 0x7f75b40cd040 2023-10-10T18:52:38.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b259c700 1 -- 172.21.15.175:0/1136768158 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac012430 con 0x7f75b40cd040 2023-10-10T18:52:38.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b883d700 1 -- 172.21.15.175:0/1136768158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 msgr2=0x7f75b40684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:38.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b883d700 1 --2- 172.21.15.175:0/1136768158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40684f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009e60 tx=0x7f75ac009780 comp rx=0 tx=0).stop 2023-10-10T18:52:38.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b883d700 1 -- 172.21.15.175:0/1136768158 shutdown_connections 2023-10-10T18:52:38.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b883d700 1 --2- 172.21.15.175:0/1136768158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40684f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b883d700 1 --2- 172.21.15.175:0/1136768158 >> v2:172.21.15.191:3300/0 conn(0x7f75b408a100 0x7f75b4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b883d700 1 -- 172.21.15.175:0/1136768158 >> 172.21.15.175:0/1136768158 conn(0x7f75b4063d00 msgr2=0x7f75b40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:38.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b883d700 1 -- 172.21.15.175:0/1136768158 shutdown_connections 2023-10-10T18:52:38.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.115+0000 7f75b883d700 1 -- 172.21.15.175:0/1136768158 wait complete. 2023-10-10T18:52:38.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b883d700 1 Processor -- start 2023-10-10T18:52:38.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b883d700 1 -- start start 2023-10-10T18:52:38.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b883d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75b408a100 0x7f75b4150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:38.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b883d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:38.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b883d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b414fd90 0x7f75b4152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:38.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b883d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4150820 con 0x7f75b408a100 2023-10-10T18:52:38.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b883d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b41520f0 con 0x7f75b414fd90 2023-10-10T18:52:38.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b883d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b41655f0 con 0x7f75b40cd040 2023-10-10T18:52:38.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75aa59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:38.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b2d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75b408a100 0x7f75b4150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:38.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75aa59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40c7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:50986/0 (socket says 172.21.15.175:50986) 2023-10-10T18:52:38.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75aa59c700 1 -- 172.21.15.175:0/3287881467 learned_addr learned my addr 172.21.15.175:0/3287881467 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:38.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.117+0000 7f75b359e700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b414fd90 0x7f75b4152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:38.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b2d9d700 1 -- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 msgr2=0x7f75b40c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:38.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b2d9d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b2d9d700 1 -- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b414fd90 msgr2=0x7f75b4152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:38.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b2d9d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b414fd90 0x7f75b4152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b2d9d700 1 -- 172.21.15.175:0/3287881467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c0094f0 con 0x7f75b408a100 2023-10-10T18:52:38.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b359e700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b414fd90 0x7f75b4152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:52:38.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b2d9d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75b408a100 0x7f75b4150ba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f75a400c490 tx=0x7f75a4007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:38.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75ab7fe700 1 -- 172.21.15.175:0/3287881467 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400ce40 con 0x7f75b408a100 2023-10-10T18:52:38.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75ab7fe700 1 -- 172.21.15.175:0/3287881467 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a4011590 con 0x7f75b408a100 2023-10-10T18:52:38.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac009a20 con 0x7f75b408a100 2023-10-10T18:52:38.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75ab7fe700 1 -- 172.21.15.175:0/3287881467 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4011950 con 0x7f75b408a100 2023-10-10T18:52:38.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.118+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b41517c0 con 0x7f75b408a100 2023-10-10T18:52:38.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.119+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b4067570 con 0x7f75b408a100 2023-10-10T18:52:38.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.119+0000 7f75ab7fe700 1 -- 172.21.15.175:0/3287881467 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f75a4011cc0 con 0x7f75b408a100 2023-10-10T18:52:38.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.120+0000 7f75ab7fe700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7594066f30 0x7f75940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:38.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.120+0000 7f75ab7fe700 1 -- 172.21.15.175:0/3287881467 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f75a4083990 con 0x7f75b408a100 2023-10-10T18:52:38.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.120+0000 7f75aa59c700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7594066f30 0x7f75940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:38.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.120+0000 7f75aa59c700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7594066f30 0x7f75940693e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f75b414f540 tx=0x7f759c009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:38.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.124+0000 7f75ab7fe700 1 -- 172.21.15.175:0/3287881467 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f75a4050a90 con 0x7f75b408a100 2023-10-10T18:52:38.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.347+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75b4155f80 con 0x7f75b408a100 2023-10-10T18:52:38.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.347+0000 7f75ab7fe700 1 -- 172.21.15.175:0/3287881467 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5451 (secure 0 0 0) 0x7f75a4017070 con 0x7f75b408a100 2023-10-10T18:52:38.348 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:38.348 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:37.672390+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:resolve","state_seq":37,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:38.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7594066f30 msgr2=0x7f75940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:38.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7594066f30 0x7f75940693e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f75b414f540 tx=0x7f759c009250 comp rx=0 tx=0).stop 2023-10-10T18:52:38.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75b408a100 msgr2=0x7f75b4150ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:38.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75b408a100 0x7f75b4150ba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f75a400c490 tx=0x7f75a4007ec0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 shutdown_connections 2023-10-10T18:52:38.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7594066f30 0x7f75940693e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75b414fd90 0x7f75b4152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75b408a100 0x7f75b4150ba0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 --2- 172.21.15.175:0/3287881467 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75b40cd040 0x7f75b40c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:38.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 >> 172.21.15.175:0/3287881467 conn(0x7f75b4063d00 msgr2=0x7f75b40b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:38.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 shutdown_connections 2023-10-10T18:52:38.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:38.349+0000 7f75b883d700 1 -- 172.21.15.175:0/3287881467 wait complete. 2023-10-10T18:52:38.352 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 60 2023-10-10T18:52:38.361 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:40.362 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:40.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.491+0000 7f56bb154700 1 Processor -- start 2023-10-10T18:52:40.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.491+0000 7f56bb154700 1 -- start start 2023-10-10T18:52:40.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56bb154700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:40.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56bb154700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56b40aff60 0x7f56b40b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:40.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56bb154700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b41574b0 con 0x7f56b40b2390 2023-10-10T18:52:40.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56bb154700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b4150770 con 0x7f56b40aff60 2023-10-10T18:52:40.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56bb154700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40afad0 con 0x7f56b40b1fc0 2023-10-10T18:52:40.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b96f1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56b40aff60 0x7f56b40b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:40.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b8ef0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:40.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b8ef0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:51002/0 (socket says 172.21.15.175:51002) 2023-10-10T18:52:40.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b96f1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56b40aff60 0x7f56b40b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:48370/0 (socket says 172.21.15.175:48370) 2023-10-10T18:52:40.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b8ef0700 1 -- 172.21.15.175:0/2345775790 learned_addr learned my addr 172.21.15.175:0/2345775790 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:40.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b8ef0700 1 -- 172.21.15.175:0/2345775790 >> v2:172.21.15.191:3300/0 conn(0x7f56b40aff60 msgr2=0x7f56b40b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:40.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b8ef0700 1 --2- 172.21.15.175:0/2345775790 >> v2:172.21.15.191:3300/0 conn(0x7f56b40aff60 0x7f56b40b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b8ef0700 1 -- 172.21.15.175:0/2345775790 >> v1:172.21.15.191:6789/0 conn(0x7f56b40b2390 legacy=0x7f56b4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:40.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b8ef0700 1 -- 172.21.15.175:0/2345775790 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b40af4c0 con 0x7f56b40b1fc0 2023-10-10T18:52:40.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.492+0000 7f56b96f1700 1 --2- 172.21.15.175:0/2345775790 >> v2:172.21.15.191:3300/0 conn(0x7f56b40aff60 0x7f56b40b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:52:40.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56b8ef0700 1 --2- 172.21.15.175:0/2345775790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4155fb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56b00099e0 tx=0x7f56b0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57594ccc721e38cf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:40.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56ab7fe700 1 -- 172.21.15.175:0/2345775790 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000b8b0 con 0x7f56b40b1fc0 2023-10-10T18:52:40.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56ab7fe700 1 -- 172.21.15.175:0/2345775790 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b000ba10 con 0x7f56b40b1fc0 2023-10-10T18:52:40.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56ab7fe700 1 -- 172.21.15.175:0/2345775790 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000bdb0 con 0x7f56b40b1fc0 2023-10-10T18:52:40.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56bb154700 1 -- 172.21.15.175:0/2345775790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 msgr2=0x7f56b4155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:40.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56bb154700 1 --2- 172.21.15.175:0/2345775790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4155fb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56b00099e0 tx=0x7f56b0009250 comp rx=0 tx=0).stop 2023-10-10T18:52:40.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56bb154700 1 -- 172.21.15.175:0/2345775790 shutdown_connections 2023-10-10T18:52:40.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56bb154700 1 --2- 172.21.15.175:0/2345775790 >> v2:172.21.15.191:3300/0 conn(0x7f56b40aff60 0x7f56b40b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56bb154700 1 --2- 172.21.15.175:0/2345775790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4155fb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56bb154700 1 -- 172.21.15.175:0/2345775790 >> 172.21.15.175:0/2345775790 conn(0x7f56b4063230 msgr2=0x7f56b4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:40.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56bb154700 1 -- 172.21.15.175:0/2345775790 shutdown_connections 2023-10-10T18:52:40.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.493+0000 7f56bb154700 1 -- 172.21.15.175:0/2345775790 wait complete. 2023-10-10T18:52:40.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.494+0000 7f56bb154700 1 Processor -- start 2023-10-10T18:52:40.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56bb154700 1 -- start start 2023-10-10T18:52:40.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56bb154700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40aff60 0x7f56b415bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:40.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56bb154700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4165e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:40.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56bb154700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b40b2390 0x7f56b4166390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:40.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56bb154700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40cb090 con 0x7f56b40b2390 2023-10-10T18:52:40.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56bb154700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4169ef0 con 0x7f56b40aff60 2023-10-10T18:52:40.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56bb154700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b41692e0 con 0x7f56b40b1fc0 2023-10-10T18:52:40.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56b96f1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b40b2390 0x7f56b4166390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:40.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56b96f1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b40b2390 0x7f56b4166390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59248/0 (socket says 172.21.15.175:59248) 2023-10-10T18:52:40.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.495+0000 7f56b96f1700 1 -- 172.21.15.175:0/2815396911 learned_addr learned my addr 172.21.15.175:0/2815396911 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:40.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56abfff700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4165e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:40.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56b8ef0700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40aff60 0x7f56b415bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:40.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56b96f1700 1 -- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 msgr2=0x7f56b4165e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:40.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56b96f1700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4165e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56b96f1700 1 -- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40aff60 msgr2=0x7f56b415bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:40.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56b96f1700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40aff60 0x7f56b415bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56b96f1700 1 -- 172.21.15.175:0/2815396911 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac009a20 con 0x7f56b40b2390 2023-10-10T18:52:40.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56abfff700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4165e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:52:40.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56b96f1700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b40b2390 0x7f56b4166390 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f56ac009fc0 tx=0x7f56ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:40.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56a8ff9700 1 -- 172.21.15.175:0/2815396911 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac00cbd0 con 0x7f56b40b2390 2023-10-10T18:52:40.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56a8ff9700 1 -- 172.21.15.175:0/2815396911 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac00f040 con 0x7f56b40b2390 2023-10-10T18:52:40.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b00094f0 con 0x7f56b40b2390 2023-10-10T18:52:40.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56a8ff9700 1 -- 172.21.15.175:0/2815396911 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac014480 con 0x7f56b40b2390 2023-10-10T18:52:40.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.496+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b41698a0 con 0x7f56b40b2390 2023-10-10T18:52:40.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.497+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b4067570 con 0x7f56b40b2390 2023-10-10T18:52:40.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.498+0000 7f56a8ff9700 1 -- 172.21.15.175:0/2815396911 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f56ac003020 con 0x7f56b40b2390 2023-10-10T18:52:40.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.498+0000 7f56a8ff9700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56a0066f30 0x7f56a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:40.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.498+0000 7f56a8ff9700 1 -- 172.21.15.175:0/2815396911 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f56ac084fc0 con 0x7f56b40b2390 2023-10-10T18:52:40.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.498+0000 7f56b8ef0700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56a0066f30 0x7f56a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:40.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.499+0000 7f56b8ef0700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56a0066f30 0x7f56a00693e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f56b00099e0 tx=0x7f56b0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:40.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.502+0000 7f56a8ff9700 1 -- 172.21.15.175:0/2815396911 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56ac0520c0 con 0x7f56b40b2390 2023-10-10T18:52:40.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.729+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56b4068da0 con 0x7f56b40b2390 2023-10-10T18:52:40.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.729+0000 7f56a8ff9700 1 -- 172.21.15.175:0/2815396911 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5453 (secure 0 0 0) 0x7f56ac0556e0 con 0x7f56b40b2390 2023-10-10T18:52:40.729 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:40.730 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:39.473192+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:40.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56a0066f30 msgr2=0x7f56a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:40.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56a0066f30 0x7f56a00693e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f56b00099e0 tx=0x7f56b0017000 comp rx=0 tx=0).stop 2023-10-10T18:52:40.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b40b2390 msgr2=0x7f56b4166390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:40.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b40b2390 0x7f56b4166390 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f56ac009fc0 tx=0x7f56ac00b040 comp rx=0 tx=0).stop 2023-10-10T18:52:40.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 shutdown_connections 2023-10-10T18:52:40.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56a0066f30 0x7f56a00693e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b40b2390 0x7f56b4166390 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40aff60 0x7f56b415bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 --2- 172.21.15.175:0/2815396911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40b1fc0 0x7f56b4165e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:40.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 >> 172.21.15.175:0/2815396911 conn(0x7f56b4063230 msgr2=0x7f56b414fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:40.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 shutdown_connections 2023-10-10T18:52:40.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:40.731+0000 7f56bb154700 1 -- 172.21.15.175:0/2815396911 wait complete. 2023-10-10T18:52:40.734 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 61 2023-10-10T18:52:40.743 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:42.745 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:42.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.871+0000 7f9472ba0700 1 Processor -- start 2023-10-10T18:52:42.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.871+0000 7f9472ba0700 1 -- start start 2023-10-10T18:52:42.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9472ba0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f946c0b1fc0 0x7f946c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:42.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9472ba0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f946c0b2390 0x7f946c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:42.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9472ba0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c068bd0 0x7f946c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:42.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9472ba0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946c150770 con 0x7f946c0b2390 2023-10-10T18:52:42.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9472ba0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946c1574b0 con 0x7f946c0b1fc0 2023-10-10T18:52:42.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9472ba0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0afad0 con 0x7f946c068bd0 2023-10-10T18:52:42.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9463fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f946c0b2390 0x7f946c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:42.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9463fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f946c0b2390 0x7f946c068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59260/0 (socket says 172.21.15.175:59260) 2023-10-10T18:52:42.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f9463fff700 1 -- 172.21.15.175:0/1067090193 learned_addr learned my addr 172.21.15.175:0/1067090193 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:42.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f947093c700 1 --2- 172.21.15.175:0/1067090193 >> v2:172.21.15.191:3300/0 conn(0x7f946c0b1fc0 0x7f946c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:42.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.872+0000 7f947113d700 1 --2- 172.21.15.175:0/1067090193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c068bd0 0x7f946c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:42.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9463fff700 1 -- 172.21.15.175:0/1067090193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c068bd0 msgr2=0x7f946c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:42.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9463fff700 1 --2- 172.21.15.175:0/1067090193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c068bd0 0x7f946c0aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:42.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9463fff700 1 -- 172.21.15.175:0/1067090193 >> v2:172.21.15.191:3300/0 conn(0x7f946c0b1fc0 msgr2=0x7f946c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:42.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9463fff700 1 --2- 172.21.15.175:0/1067090193 >> v2:172.21.15.191:3300/0 conn(0x7f946c0b1fc0 0x7f946c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:42.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9463fff700 1 -- 172.21.15.175:0/1067090193 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c0af4c0 con 0x7f946c0b2390 2023-10-10T18:52:42.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f947113d700 1 --2- 172.21.15.175:0/1067090193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c068bd0 0x7f946c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:42.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f947093c700 1 --2- 172.21.15.175:0/1067090193 >> v2:172.21.15.191:3300/0 conn(0x7f946c0b1fc0 0x7f946c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:42.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9463fff700 1 --2- 172.21.15.175:0/1067090193 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c0b2390 0x7f946c068690 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f946c0b28c0 tx=0x7f945c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a76928c535bb35 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:42.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f94637fe700 1 -- 172.21.15.175:0/1067090193 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00b8f0 con 0x7f946c0b2390 2023-10-10T18:52:42.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f94637fe700 1 -- 172.21.15.175:0/1067090193 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c00ba50 con 0x7f946c0b2390 2023-10-10T18:52:42.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f94637fe700 1 -- 172.21.15.175:0/1067090193 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00bdf0 con 0x7f946c0b2390 2023-10-10T18:52:42.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9472ba0700 1 -- 172.21.15.175:0/1067090193 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c0b2390 msgr2=0x7f946c068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:42.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9472ba0700 1 --2- 172.21.15.175:0/1067090193 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c0b2390 0x7f946c068690 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f946c0b28c0 tx=0x7f945c009250 comp rx=0 tx=0).stop 2023-10-10T18:52:42.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9472ba0700 1 -- 172.21.15.175:0/1067090193 shutdown_connections 2023-10-10T18:52:42.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9472ba0700 1 --2- 172.21.15.175:0/1067090193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c068bd0 0x7f946c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:42.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9472ba0700 1 --2- 172.21.15.175:0/1067090193 >> v2:172.21.15.191:3300/0 conn(0x7f946c0b1fc0 0x7f946c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:42.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9472ba0700 1 --2- 172.21.15.175:0/1067090193 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c0b2390 0x7f946c068690 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:42.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.873+0000 7f9472ba0700 1 -- 172.21.15.175:0/1067090193 >> 172.21.15.175:0/1067090193 conn(0x7f946c063230 msgr2=0x7f946c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:42.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.874+0000 7f9472ba0700 1 -- 172.21.15.175:0/1067090193 shutdown_connections 2023-10-10T18:52:42.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.874+0000 7f9472ba0700 1 -- 172.21.15.175:0/1067090193 wait complete. 2023-10-10T18:52:42.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.875+0000 7f9472ba0700 1 Processor -- start 2023-10-10T18:52:42.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.875+0000 7f9472ba0700 1 -- start start 2023-10-10T18:52:42.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.875+0000 7f9472ba0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c068bd0 0x7f946c1567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:42.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.875+0000 7f9472ba0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c0b1fc0 0x7f946c154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:42.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.875+0000 7f9472ba0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f946c0b2390 0x7f946c16d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:42.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.875+0000 7f9472ba0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0bfab0 con 0x7f946c068bd0 2023-10-10T18:52:42.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.875+0000 7f9472ba0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0c6ae0 con 0x7f946c0b2390 2023-10-10T18:52:42.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.875+0000 7f9472ba0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0c39f0 con 0x7f946c0b1fc0 2023-10-10T18:52:42.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c068bd0 0x7f946c1567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:42.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c068bd0 0x7f946c1567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59262/0 (socket says 172.21.15.175:59262) 2023-10-10T18:52:42.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 -- 172.21.15.175:0/2142383491 learned_addr learned my addr 172.21.15.175:0/2142383491 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:42.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f9463fff700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c0b1fc0 0x7f946c154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:42.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947113d700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f946c0b2390 0x7f946c16d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:42.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 -- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c0b1fc0 msgr2=0x7f946c154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:42.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c0b1fc0 0x7f946c154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:42.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 -- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f946c0b2390 msgr2=0x7f946c16d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:42.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f946c0b2390 0x7f946c16d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:42.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 -- 172.21.15.175:0/2142383491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0094f0 con 0x7f946c068bd0 2023-10-10T18:52:42.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f947093c700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c068bd0 0x7f946c1567c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f946800c830 tx=0x7f9468008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:42.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f9460ff9700 1 -- 172.21.15.175:0/2142383491 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468011600 con 0x7f946c068bd0 2023-10-10T18:52:42.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f9460ff9700 1 -- 172.21.15.175:0/2142383491 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9468011760 con 0x7f946c068bd0 2023-10-10T18:52:42.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c0bfcc0 con 0x7f946c068bd0 2023-10-10T18:52:42.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f9460ff9700 1 -- 172.21.15.175:0/2142383491 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468011b20 con 0x7f946c068bd0 2023-10-10T18:52:42.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.876+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c0c65c0 con 0x7f946c068bd0 2023-10-10T18:52:42.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.877+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c067570 con 0x7f946c068bd0 2023-10-10T18:52:42.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.878+0000 7f9460ff9700 1 -- 172.21.15.175:0/2142383491 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9468018360 con 0x7f946c068bd0 2023-10-10T18:52:42.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.878+0000 7f9460ff9700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9458066f30 0x7f94580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:42.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.878+0000 7f9460ff9700 1 -- 172.21.15.175:0/2142383491 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f94680846d0 con 0x7f946c068bd0 2023-10-10T18:52:42.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.878+0000 7f9463fff700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9458066f30 0x7f94580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:42.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.879+0000 7f9463fff700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9458066f30 0x7f94580693e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f946c0b28c0 tx=0x7f945c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:42.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:42.882+0000 7f9460ff9700 1 -- 172.21.15.175:0/2142383491 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f94680517d0 con 0x7f946c068bd0 2023-10-10T18:52:43.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.105+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f946c069060 con 0x7f946c068bd0 2023-10-10T18:52:43.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.105+0000 7f9460ff9700 1 -- 172.21.15.175:0/2142383491 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5453 (secure 0 0 0) 0x7f9468054df0 con 0x7f946c068bd0 2023-10-10T18:52:43.106 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:43.106 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:39.473192+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:43.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9458066f30 msgr2=0x7f94580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:43.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9458066f30 0x7f94580693e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f946c0b28c0 tx=0x7f945c017000 comp rx=0 tx=0).stop 2023-10-10T18:52:43.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c068bd0 msgr2=0x7f946c1567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:43.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c068bd0 0x7f946c1567c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f946800c830 tx=0x7f9468008040 comp rx=0 tx=0).stop 2023-10-10T18:52:43.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 shutdown_connections 2023-10-10T18:52:43.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9458066f30 0x7f94580693e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:43.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f946c0b2390 0x7f946c16d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:43.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f946c068bd0 0x7f946c1567c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:43.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 --2- 172.21.15.175:0/2142383491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f946c0b1fc0 0x7f946c154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:43.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 >> 172.21.15.175:0/2142383491 conn(0x7f946c063230 msgr2=0x7f946c1575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:43.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 shutdown_connections 2023-10-10T18:52:43.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:43.108+0000 7f9472ba0700 1 -- 172.21.15.175:0/2142383491 wait complete. 2023-10-10T18:52:43.110 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 61 2023-10-10T18:52:43.120 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:45.120 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:45.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.247+0000 7fb01f196700 1 Processor -- start 2023-10-10T18:52:45.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.248+0000 7fb01f196700 1 -- start start 2023-10-10T18:52:45.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.248+0000 7fb01f196700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb01814fbf0 0x7fb018159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:45.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.248+0000 7fb01f196700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb0180681e0 0x7fb0180c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:45.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.248+0000 7fb01f196700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb018156800 con 0x7fb01814ffc0 2023-10-10T18:52:45.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.248+0000 7fb01f196700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180c59e0 con 0x7fb0180681e0 2023-10-10T18:52:45.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.248+0000 7fb01f196700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01815a2b0 con 0x7fb01814fbf0 2023-10-10T18:52:45.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01d733700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb0180681e0 0x7fb0180c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:45.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01cf32700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb01814fbf0 0x7fb018159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:45.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01cf32700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb01814fbf0 0x7fb018159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:45438/0 (socket says 172.21.15.175:45438) 2023-10-10T18:52:45.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01cf32700 1 -- 172.21.15.175:0/1456628645 learned_addr learned my addr 172.21.15.175:0/1456628645 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:45.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01d733700 1 -- 172.21.15.175:0/1456628645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb01814fbf0 msgr2=0x7fb018159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:45.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01d733700 1 --2- 172.21.15.175:0/1456628645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb01814fbf0 0x7fb018159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01d733700 1 -- 172.21.15.175:0/1456628645 >> v1:172.21.15.191:6789/0 conn(0x7fb01814ffc0 legacy=0x7fb018063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:45.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01d733700 1 -- 172.21.15.175:0/1456628645 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb018063e30 con 0x7fb0180681e0 2023-10-10T18:52:45.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01cf32700 1 --2- 172.21.15.175:0/1456628645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb01814fbf0 0x7fb018159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:52:45.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb01d733700 1 --2- 172.21.15.175:0/1456628645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0180681e0 0x7fb0180c5fb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb010009910 tx=0x7fb0100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baea76f482a84d18 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:45.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.249+0000 7fb0177fe700 1 -- 172.21.15.175:0/1456628645 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01000b640 con 0x7fb0180681e0 2023-10-10T18:52:45.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb0177fe700 1 -- 172.21.15.175:0/1456628645 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01000b7a0 con 0x7fb0180681e0 2023-10-10T18:52:45.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb0177fe700 1 -- 172.21.15.175:0/1456628645 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01000bb60 con 0x7fb0180681e0 2023-10-10T18:52:45.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb01f196700 1 -- 172.21.15.175:0/1456628645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0180681e0 msgr2=0x7fb0180c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:45.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb01f196700 1 --2- 172.21.15.175:0/1456628645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0180681e0 0x7fb0180c5fb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb010009910 tx=0x7fb0100091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb01f196700 1 -- 172.21.15.175:0/1456628645 shutdown_connections 2023-10-10T18:52:45.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb01f196700 1 --2- 172.21.15.175:0/1456628645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0180681e0 0x7fb0180c5fb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb01f196700 1 --2- 172.21.15.175:0/1456628645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb01814fbf0 0x7fb018159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb01f196700 1 -- 172.21.15.175:0/1456628645 >> 172.21.15.175:0/1456628645 conn(0x7fb0180cc760 msgr2=0x7fb0180ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:45.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb01f196700 1 -- 172.21.15.175:0/1456628645 shutdown_connections 2023-10-10T18:52:45.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.250+0000 7fb01f196700 1 -- 172.21.15.175:0/1456628645 wait complete. 2023-10-10T18:52:45.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.251+0000 7fb01f196700 1 Processor -- start 2023-10-10T18:52:45.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.251+0000 7fb01f196700 1 -- start start 2023-10-10T18:52:45.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01f196700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0180681e0 0x7fb018167020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:45.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01f196700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb01814fbf0 0x7fb0180b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:45.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01f196700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb01814ffc0 0x7fb0180c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:45.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01f196700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb018166b90 con 0x7fb01814ffc0 2023-10-10T18:52:45.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01f196700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb018167560 con 0x7fb01814fbf0 2023-10-10T18:52:45.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01f196700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb018166650 con 0x7fb0180681e0 2023-10-10T18:52:45.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01d733700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb01814ffc0 0x7fb0180c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:45.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01d733700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb01814ffc0 0x7fb0180c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42250/0 (socket says 172.21.15.175:42250) 2023-10-10T18:52:45.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01d733700 1 -- 172.21.15.175:0/2290392001 learned_addr learned my addr 172.21.15.175:0/2290392001 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:45.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01cf32700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0180681e0 0x7fb018167020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:45.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb017fff700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb01814fbf0 0x7fb0180b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:45.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01d733700 1 -- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0180681e0 msgr2=0x7fb018167020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:45.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01d733700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0180681e0 0x7fb018167020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01d733700 1 -- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb01814fbf0 msgr2=0x7fb0180b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:45.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01d733700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb01814fbf0 0x7fb0180b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.252+0000 7fb01d733700 1 -- 172.21.15.175:0/2290392001 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008009ae0 con 0x7fb01814ffc0 2023-10-10T18:52:45.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.253+0000 7fb01d733700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb01814ffc0 0x7fb0180c5ed0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb01001af80 tx=0x7fb010017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:45.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.253+0000 7fb014ff9700 1 -- 172.21.15.175:0/2290392001 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01001f070 con 0x7fb01814ffc0 2023-10-10T18:52:45.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.253+0000 7fb014ff9700 1 -- 172.21.15.175:0/2290392001 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01000bb20 con 0x7fb01814ffc0 2023-10-10T18:52:45.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.253+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb010009420 con 0x7fb01814ffc0 2023-10-10T18:52:45.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.253+0000 7fb014ff9700 1 -- 172.21.15.175:0/2290392001 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb010011690 con 0x7fb01814ffc0 2023-10-10T18:52:45.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.253+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0181668d0 con 0x7fb01814ffc0 2023-10-10T18:52:45.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.254+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb018071dd0 con 0x7fb01814ffc0 2023-10-10T18:52:45.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.254+0000 7fb014ff9700 1 -- 172.21.15.175:0/2290392001 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb010011a00 con 0x7fb01814ffc0 2023-10-10T18:52:45.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.254+0000 7fb014ff9700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb004066f30 0x7fb0040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:45.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.255+0000 7fb014ff9700 1 -- 172.21.15.175:0/2290392001 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7fb0100855c0 con 0x7fb01814ffc0 2023-10-10T18:52:45.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.255+0000 7fb01cf32700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb004066f30 0x7fb0040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:45.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.255+0000 7fb01cf32700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb004066f30 0x7fb0040693e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb0080072d0 tx=0x7fb00800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:45.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.259+0000 7fb014ff9700 1 -- 172.21.15.175:0/2290392001 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0100526c0 con 0x7fb01814ffc0 2023-10-10T18:52:45.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.484+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb018067570 con 0x7fb01814ffc0 2023-10-10T18:52:45.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.485+0000 7fb014ff9700 1 -- 172.21.15.175:0/2290392001 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5450 (secure 0 0 0) 0x7fb010011cb0 con 0x7fb01814ffc0 2023-10-10T18:52:45.485 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:45.485 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:44.486071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:rejoin","state_seq":42,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:45.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb004066f30 msgr2=0x7fb0040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:45.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb004066f30 0x7fb0040693e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb0080072d0 tx=0x7fb00800b040 comp rx=0 tx=0).stop 2023-10-10T18:52:45.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb01814ffc0 msgr2=0x7fb0180c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:45.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb01814ffc0 0x7fb0180c5ed0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb01001af80 tx=0x7fb010017000 comp rx=0 tx=0).stop 2023-10-10T18:52:45.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 shutdown_connections 2023-10-10T18:52:45.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb004066f30 0x7fb0040693e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb01814ffc0 0x7fb0180c5ed0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0180681e0 0x7fb018167020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 --2- 172.21.15.175:0/2290392001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb01814fbf0 0x7fb0180b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:45.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 >> 172.21.15.175:0/2290392001 conn(0x7fb0180cc760 msgr2=0x7fb018158d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:45.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 shutdown_connections 2023-10-10T18:52:45.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:45.487+0000 7fb01f196700 1 -- 172.21.15.175:0/2290392001 wait complete. 2023-10-10T18:52:45.491 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 62 2023-10-10T18:52:45.499 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:47.500 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:47.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.627+0000 7f5296729700 1 Processor -- start 2023-10-10T18:52:47.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.627+0000 7f5296729700 1 -- start start 2023-10-10T18:52:47.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.628+0000 7f5296729700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52900b1fc0 0x7f5290155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:47.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.628+0000 7f5296729700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52901574b0 con 0x7f52900b2390 2023-10-10T18:52:47.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.628+0000 7f5296729700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52900afad0 con 0x7f52900aff60 2023-10-10T18:52:47.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.628+0000 7f5296729700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5290150770 con 0x7f52900b1fc0 2023-10-10T18:52:47.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.628+0000 7f528f7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f52900b2390 0x7f5290068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55162/0 (socket says 172.21.15.175:55162) 2023-10-10T18:52:47.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.628+0000 7f528f7fe700 1 -- 172.21.15.175:0/3320802485 learned_addr learned my addr 172.21.15.175:0/3320802485 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:47.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.628+0000 7f528ffff700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52900b1fc0 0x7f5290155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:47.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838590688 0 0) 0x7f52901574b0 con 0x7f52900b2390 2023-10-10T18:52:47.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5278003570 con 0x7f52900b2390 2023-10-10T18:52:47.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151386255 0 0) 0x7f52900afad0 con 0x7f52900aff60 2023-10-10T18:52:47.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52901574b0 con 0x7f52900aff60 2023-10-10T18:52:47.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2580565597 0 0) 0x7f5278003570 con 0x7f52900b2390 2023-10-10T18:52:47.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52900b1fc0 msgr2=0x7f5290155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:47.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52900b1fc0 0x7f5290155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 >> v1:172.21.15.191:6789/0 conn(0x7f52900aff60 legacy=0x7f5290159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:47.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900af4c0 con 0x7f52900b2390 2023-10-10T18:52:47.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f5284001470 con 0x7f52900b2390 2023-10-10T18:52:47.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 >> v1:172.21.15.175:6789/0 conn(0x7f52900b2390 legacy=0x7f5290068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:47.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52900b1fc0 0x7f5278003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:47.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52780041d0 0x7f5278004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:47.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278004d50 0x7f52780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:47.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5290150770 con 0x7f52780041d0 2023-10-10T18:52:47.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527800bbf0 con 0x7f5278004d50 2023-10-10T18:52:47.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527800bdb0 con 0x7f52900b1fc0 2023-10-10T18:52:47.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528f7fe700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52780041d0 0x7f5278004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:47.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528ffff700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52900b1fc0 0x7f5278003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:47.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f5294cc6700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278004d50 0x7f52780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:47.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528f7fe700 1 -- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52900b1fc0 msgr2=0x7f5278003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:47.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528f7fe700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52900b1fc0 0x7f5278003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528f7fe700 1 -- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278004d50 msgr2=0x7f52780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:47.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528f7fe700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278004d50 0x7f52780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528f7fe700 1 -- 172.21.15.175:0/3320802485 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5278003570 con 0x7f52780041d0 2023-10-10T18:52:47.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f5294cc6700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278004d50 0x7f52780051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:52:47.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.629+0000 7f528f7fe700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52780041d0 0x7f5278004810 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f528400c1e0 tx=0x7f52900afa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69049255b26c1eb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:47.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528400cab0 con 0x7f52780041d0 2023-10-10T18:52:47.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528400cc10 con 0x7f52780041d0 2023-10-10T18:52:47.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f528effd700 1 -- 172.21.15.175:0/3320802485 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284005920 con 0x7f52780041d0 2023-10-10T18:52:47.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f5296729700 1 -- 172.21.15.175:0/3320802485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52780041d0 msgr2=0x7f5278004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:47.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f5296729700 1 --2- 172.21.15.175:0/3320802485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52780041d0 0x7f5278004810 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f528400c1e0 tx=0x7f52900afa40 comp rx=0 tx=0).stop 2023-10-10T18:52:47.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f5294cc6700 1 -- 172.21.15.175:0/3320802485 reap_dead start 2023-10-10T18:52:47.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f5296729700 1 -- 172.21.15.175:0/3320802485 shutdown_connections 2023-10-10T18:52:47.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f5296729700 1 -- 172.21.15.175:0/3320802485 >> 172.21.15.175:0/3320802485 conn(0x7f5290063230 msgr2=0x7f5290063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:47.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f5296729700 1 -- 172.21.15.175:0/3320802485 shutdown_connections 2023-10-10T18:52:47.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.630+0000 7f5296729700 1 -- 172.21.15.175:0/3320802485 wait complete. 2023-10-10T18:52:47.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.631+0000 7f5296729700 1 Processor -- start 2023-10-10T18:52:47.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f5296729700 1 -- start start 2023-10-10T18:52:47.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f5296729700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52780041d0 0x7f52901666c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:47.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f5296729700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52900c0f40 0x7f52900c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:47.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f5296729700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5290155de0 0x7f5290156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:47.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f5296729700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900c19b0 con 0x7f52900c0f40 2023-10-10T18:52:47.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f5296729700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900c0ab0 con 0x7f5290155de0 2023-10-10T18:52:47.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f5296729700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52900c6680 con 0x7f52780041d0 2023-10-10T18:52:47.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f528f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52900c0f40 0x7f52900c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:47.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f528f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52900c0f40 0x7f52900c1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42276/0 (socket says 172.21.15.175:42276) 2023-10-10T18:52:47.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f528f7fe700 1 -- 172.21.15.175:0/243269858 learned_addr learned my addr 172.21.15.175:0/243269858 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:47.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.632+0000 7f528ffff700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52780041d0 0x7f52901666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:47.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f5294cc6700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5290155de0 0x7f5290156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:47.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f528f7fe700 1 -- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52780041d0 msgr2=0x7f52901666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:47.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f528f7fe700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52780041d0 0x7f52901666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f528f7fe700 1 -- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5290155de0 msgr2=0x7f5290156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:47.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f528f7fe700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5290155de0 0x7f5290156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f528f7fe700 1 -- 172.21.15.175:0/243269858 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900c68f0 con 0x7f52900c0f40 2023-10-10T18:52:47.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f528f7fe700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52900c0f40 0x7f52900c1b40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f528400d010 tx=0x7f5284015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:47.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f5277fff700 1 -- 172.21.15.175:0/243269858 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284004a40 con 0x7f52900c0f40 2023-10-10T18:52:47.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f5277fff700 1 -- 172.21.15.175:0/243269858 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5284004ba0 con 0x7f52900c0f40 2023-10-10T18:52:47.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52900c6bd0 con 0x7f52900c0f40 2023-10-10T18:52:47.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f5277fff700 1 -- 172.21.15.175:0/243269858 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52840070d0 con 0x7f52900c0f40 2023-10-10T18:52:47.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.633+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52900c0740 con 0x7f52900c0f40 2023-10-10T18:52:47.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.634+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52900c02c0 con 0x7f52900c0f40 2023-10-10T18:52:47.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.635+0000 7f5277fff700 1 -- 172.21.15.175:0/243269858 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5284007480 con 0x7f52900c0f40 2023-10-10T18:52:47.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.635+0000 7f5277fff700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f527c066f30 0x7f527c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:47.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.635+0000 7f5277fff700 1 -- 172.21.15.175:0/243269858 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f52840855d0 con 0x7f52900c0f40 2023-10-10T18:52:47.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.635+0000 7f528ffff700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f527c066f30 0x7f527c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:47.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.635+0000 7f528ffff700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f527c066f30 0x7f527c0693e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5280006f00 tx=0x7f52800091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:47.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.639+0000 7f5277fff700 1 -- 172.21.15.175:0/243269858 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f52900c02c0 con 0x7f52900c0f40 2023-10-10T18:52:47.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.868+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52900c02c0 con 0x7f52900c0f40 2023-10-10T18:52:47.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.868+0000 7f5277fff700 1 -- 172.21.15.175:0/243269858 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5450 (secure 0 0 0) 0x7f52900c02c0 con 0x7f52900c0f40 2023-10-10T18:52:47.869 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:47.869 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:44.486071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:rejoin","state_seq":42,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:47.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f527c066f30 msgr2=0x7f527c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:47.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f527c066f30 0x7f527c0693e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5280006f00 tx=0x7f52800091a0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52900c0f40 msgr2=0x7f52900c1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:47.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52900c0f40 0x7f52900c1b40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f528400d010 tx=0x7f5284015000 comp rx=0 tx=0).stop 2023-10-10T18:52:47.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 shutdown_connections 2023-10-10T18:52:47.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f527c066f30 0x7f527c0693e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5290155de0 0x7f5290156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52780041d0 0x7f52901666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 --2- 172.21.15.175:0/243269858 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52900c0f40 0x7f52900c1b40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:47.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 >> 172.21.15.175:0/243269858 conn(0x7f5290063230 msgr2=0x7f529014fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:47.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 shutdown_connections 2023-10-10T18:52:47.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:47.870+0000 7f5296729700 1 -- 172.21.15.175:0/243269858 wait complete. 2023-10-10T18:52:47.873 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 62 2023-10-10T18:52:47.883 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4250": { "addr": "172.21.15.191:6839/246221790", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 246221790, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 246221790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4250, "incarnation": 12, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:rejoin", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:52:44.486071+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 4250, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:52:49.325 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:52:49.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:52:49.884 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:50.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.004+0000 7fd8ff123700 1 Processor -- start 2023-10-10T18:52:50.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.004+0000 7fd8ff123700 1 -- start start 2023-10-10T18:52:50.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.004+0000 7fd8ff123700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd8f8068a40 0x7fd8f8063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:50.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.004+0000 7fd8ff123700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd8f80cac80 0x7fd8f806b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:50.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.004+0000 7fd8ff123700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80c4ee0 con 0x7fd8f8068e10 2023-10-10T18:52:50.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.004+0000 7fd8ff123700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f8064180 con 0x7fd8f8068a40 2023-10-10T18:52:50.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.004+0000 7fd8ff123700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f80c5780 con 0x7fd8f80cac80 2023-10-10T18:52:50.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8fcebf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd8f8068a40 0x7fd8f8063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:50.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8fd6c0700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd8f8068e10 0x7fd8f80af910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55164/0 (socket says 172.21.15.175:55164) 2023-10-10T18:52:50.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8fd6c0700 1 -- 172.21.15.175:0/329262780 learned_addr learned my addr 172.21.15.175:0/329262780 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:50.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8effff700 1 --2- 172.21.15.175:0/329262780 >> v2:172.21.15.191:3300/0 conn(0x7fd8f80cac80 0x7fd8f806b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:50.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8fcebf700 1 -- 172.21.15.175:0/329262780 >> v2:172.21.15.191:3300/0 conn(0x7fd8f80cac80 msgr2=0x7fd8f806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:50.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8fcebf700 1 --2- 172.21.15.175:0/329262780 >> v2:172.21.15.191:3300/0 conn(0x7fd8f80cac80 0x7fd8f806b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8fcebf700 1 -- 172.21.15.175:0/329262780 >> v1:172.21.15.175:6789/0 conn(0x7fd8f8068e10 legacy=0x7fd8f80af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:50.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8fcebf700 1 -- 172.21.15.175:0/329262780 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f80c6710 con 0x7fd8f8068a40 2023-10-10T18:52:50.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8ef7fe700 1 -- 172.21.15.175:0/329262780 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967960428 0 0) 0x7fd8f80c4ee0 con 0x7fd8f8068e10 2023-10-10T18:52:50.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8fcebf700 1 --2- 172.21.15.175:0/329262780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f8068a40 0x7fd8f8063c40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd8f806e700 tx=0x7fd8f4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6122f6c4f8638e2e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:50.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8ef7fe700 1 -- 172.21.15.175:0/329262780 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400e070 con 0x7fd8f8068a40 2023-10-10T18:52:50.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8ef7fe700 1 -- 172.21.15.175:0/329262780 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f400bb60 con 0x7fd8f8068a40 2023-10-10T18:52:50.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.005+0000 7fd8ef7fe700 1 -- 172.21.15.175:0/329262780 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4012430 con 0x7fd8f8068a40 2023-10-10T18:52:50.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.006+0000 7fd8ff123700 1 -- 172.21.15.175:0/329262780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f8068a40 msgr2=0x7fd8f8063c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:50.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.006+0000 7fd8ff123700 1 --2- 172.21.15.175:0/329262780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f8068a40 0x7fd8f8063c40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd8f806e700 tx=0x7fd8f4009780 comp rx=0 tx=0).stop 2023-10-10T18:52:50.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.006+0000 7fd8ff123700 1 -- 172.21.15.175:0/329262780 shutdown_connections 2023-10-10T18:52:50.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.006+0000 7fd8ff123700 1 --2- 172.21.15.175:0/329262780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f8068a40 0x7fd8f8063c40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.006+0000 7fd8ff123700 1 --2- 172.21.15.175:0/329262780 >> v2:172.21.15.191:3300/0 conn(0x7fd8f80cac80 0x7fd8f806b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.006+0000 7fd8ff123700 1 -- 172.21.15.175:0/329262780 >> 172.21.15.175:0/329262780 conn(0x7fd8f806dbe0 msgr2=0x7fd8f80ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:50.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.006+0000 7fd8ff123700 1 -- 172.21.15.175:0/329262780 shutdown_connections 2023-10-10T18:52:50.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.006+0000 7fd8ff123700 1 -- 172.21.15.175:0/329262780 wait complete. 2023-10-10T18:52:50.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.007+0000 7fd8ff123700 1 Processor -- start 2023-10-10T18:52:50.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.007+0000 7fd8ff123700 1 -- start start 2023-10-10T18:52:50.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ff123700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8f8068a40 0x7fd8f8156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:50.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ff123700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8f8068e10 0x7fd8f80aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:50.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ff123700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f80cac80 0x7fd8f80aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:50.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ff123700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f8157490 con 0x7fd8f80cac80 2023-10-10T18:52:50.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ff123700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f81662f0 con 0x7fd8f8068e10 2023-10-10T18:52:50.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ff123700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f81665d0 con 0x7fd8f8068a40 2023-10-10T18:52:50.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f80cac80 0x7fd8f80aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:50.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f80cac80 0x7fd8f80aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42294/0 (socket says 172.21.15.175:42294) 2023-10-10T18:52:50.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 -- 172.21.15.175:0/323161084 learned_addr learned my addr 172.21.15.175:0/323161084 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:50.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8effff700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8f8068e10 0x7fd8f80aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:50.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fcebf700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8f8068a40 0x7fd8f8156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:50.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 -- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8f8068a40 msgr2=0x7fd8f8156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:50.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8f8068a40 0x7fd8f8156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 -- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8f8068e10 msgr2=0x7fd8f80aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:50.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8f8068e10 0x7fd8f80aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 -- 172.21.15.175:0/323161084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f4009a20 con 0x7fd8f80cac80 2023-10-10T18:52:50.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8fd6c0700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f80cac80 0x7fd8f80aae00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00011e0 tx=0x7fd8f0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:50.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ecff9700 1 -- 172.21.15.175:0/323161084 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0005b30 con 0x7fd8f80cac80 2023-10-10T18:52:50.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ecff9700 1 -- 172.21.15.175:0/323161084 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0005c90 con 0x7fd8f80cac80 2023-10-10T18:52:50.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f81668b0 con 0x7fd8f80cac80 2023-10-10T18:52:50.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ecff9700 1 -- 172.21.15.175:0/323161084 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0007300 con 0x7fd8f80cac80 2023-10-10T18:52:50.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.008+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f8166e70 con 0x7fd8f80cac80 2023-10-10T18:52:50.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.009+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f8067570 con 0x7fd8f80cac80 2023-10-10T18:52:50.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.010+0000 7fd8ecff9700 1 -- 172.21.15.175:0/323161084 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd8f0007670 con 0x7fd8f80cac80 2023-10-10T18:52:50.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.010+0000 7fd8ecff9700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8e4066f60 0x7fd8e4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:50.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.010+0000 7fd8ecff9700 1 -- 172.21.15.175:0/323161084 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7fd8f0083be0 con 0x7fd8f80cac80 2023-10-10T18:52:50.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.010+0000 7fd8fcebf700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8e4066f60 0x7fd8e4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:50.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.011+0000 7fd8fcebf700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8e4066f60 0x7fd8e4069410 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd8f40099f0 tx=0x7fd8f4019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:50.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.014+0000 7fd8ecff9700 1 -- 172.21.15.175:0/323161084 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8f0050ce0 con 0x7fd8f80cac80 2023-10-10T18:52:50.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.238+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8f80c6990 con 0x7fd8f80cac80 2023-10-10T18:52:50.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.238+0000 7fd8ecff9700 1 -- 172.21.15.175:0/323161084 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5448 (secure 0 0 0) 0x7fd8f0017070 con 0x7fd8f80cac80 2023-10-10T18:52:50.240 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:50.240 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:48.499464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:rejoin","state_seq":42,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:50.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8e4066f60 msgr2=0x7fd8e4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:50.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8e4066f60 0x7fd8e4069410 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd8f40099f0 tx=0x7fd8f4019000 comp rx=0 tx=0).stop 2023-10-10T18:52:50.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f80cac80 msgr2=0x7fd8f80aae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:50.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f80cac80 0x7fd8f80aae00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00011e0 tx=0x7fd8f0007d30 comp rx=0 tx=0).stop 2023-10-10T18:52:50.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 shutdown_connections 2023-10-10T18:52:50.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd8e4066f60 0x7fd8e4069410 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8f80cac80 0x7fd8f80aae00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd8f8068a40 0x7fd8f8156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 --2- 172.21.15.175:0/323161084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8f8068e10 0x7fd8f80aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:50.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.240+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 >> 172.21.15.175:0/323161084 conn(0x7fd8f806dbe0 msgr2=0x7fd8f80c46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:50.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.241+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 shutdown_connections 2023-10-10T18:52:50.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:50.241+0000 7fd8ff123700 1 -- 172.21.15.175:0/323161084 wait complete. 2023-10-10T18:52:50.245 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 63 2023-10-10T18:52:50.254 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:52.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:52.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.381+0000 7f0a58bb6700 1 Processor -- start 2023-10-10T18:52:52.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.381+0000 7f0a58bb6700 1 -- start start 2023-10-10T18:52:52.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a58bb6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0a540c5850 0x7f0a540c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:52.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a58bb6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a540c12d0 0x7f0a54156660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:52.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a58bb6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0a54156ba0 0x7f0a540c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:52.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a58bb6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a54063ea0 con 0x7f0a54156ba0 2023-10-10T18:52:52.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a58bb6700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a54155f30 con 0x7f0a540c5850 2023-10-10T18:52:52.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a58bb6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a540c6fb0 con 0x7f0a540c12d0 2023-10-10T18:52:52.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a52d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0a54156ba0 0x7f0a540c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:52.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a52d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0a54156ba0 0x7f0a540c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42298/0 (socket says 172.21.15.175:42298) 2023-10-10T18:52:52.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a52d9d700 1 -- 172.21.15.175:0/3905108374 learned_addr learned my addr 172.21.15.175:0/3905108374 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:52.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a51d9b700 1 --2- 172.21.15.175:0/3905108374 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a540c12d0 0x7f0a54156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:52.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.382+0000 7f0a5259c700 1 --2- 172.21.15.175:0/3905108374 >> v2:172.21.15.191:3300/0 conn(0x7f0a540c5850 0x7f0a540c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:52.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a52d9d700 1 -- 172.21.15.175:0/3905108374 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a540c12d0 msgr2=0x7f0a54156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:52.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a52d9d700 1 --2- 172.21.15.175:0/3905108374 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a540c12d0 0x7f0a54156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a52d9d700 1 -- 172.21.15.175:0/3905108374 >> v2:172.21.15.191:3300/0 conn(0x7f0a540c5850 msgr2=0x7f0a540c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:52.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a52d9d700 1 --2- 172.21.15.175:0/3905108374 >> v2:172.21.15.191:3300/0 conn(0x7f0a540c5850 0x7f0a540c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a52d9d700 1 -- 172.21.15.175:0/3905108374 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540c6160 con 0x7f0a54156ba0 2023-10-10T18:52:52.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a52d9d700 1 --2- 172.21.15.175:0/3905108374 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a54156ba0 0x7f0a540c7370 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0a540cb5c0 tx=0x7f0a4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261591a00bc539c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:52.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a5159a700 1 -- 172.21.15.175:0/3905108374 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c00b6d0 con 0x7f0a54156ba0 2023-10-10T18:52:52.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a5159a700 1 -- 172.21.15.175:0/3905108374 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4c00b830 con 0x7f0a54156ba0 2023-10-10T18:52:52.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a5159a700 1 -- 172.21.15.175:0/3905108374 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c00bbd0 con 0x7f0a54156ba0 2023-10-10T18:52:52.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.383+0000 7f0a58bb6700 1 -- 172.21.15.175:0/3905108374 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a54156ba0 msgr2=0x7f0a540c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:52.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.384+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/3905108374 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a54156ba0 0x7f0a540c7370 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0a540cb5c0 tx=0x7f0a4c009250 comp rx=0 tx=0).stop 2023-10-10T18:52:52.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.384+0000 7f0a58bb6700 1 -- 172.21.15.175:0/3905108374 shutdown_connections 2023-10-10T18:52:52.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.384+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/3905108374 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a54156ba0 0x7f0a540c7370 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.384+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/3905108374 >> v2:172.21.15.191:3300/0 conn(0x7f0a540c5850 0x7f0a540c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.384+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/3905108374 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a540c12d0 0x7f0a54156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.384+0000 7f0a58bb6700 1 -- 172.21.15.175:0/3905108374 >> 172.21.15.175:0/3905108374 conn(0x7f0a5406dc70 msgr2=0x7f0a54063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:52.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.384+0000 7f0a58bb6700 1 -- 172.21.15.175:0/3905108374 shutdown_connections 2023-10-10T18:52:52.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.384+0000 7f0a58bb6700 1 -- 172.21.15.175:0/3905108374 wait complete. 2023-10-10T18:52:52.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a58bb6700 1 Processor -- start 2023-10-10T18:52:52.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a58bb6700 1 -- start start 2023-10-10T18:52:52.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a58bb6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a540c12d0 0x7f0a5414e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:52.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a58bb6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a540c5850 0x7f0a5415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:52.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a58bb6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a54156ba0 0x7f0a541593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:52.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a58bb6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5415c1f0 con 0x7f0a540c12d0 2023-10-10T18:52:52.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a58bb6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5415c720 con 0x7f0a540c5850 2023-10-10T18:52:52.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a58bb6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5415ab60 con 0x7f0a54156ba0 2023-10-10T18:52:52.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.385+0000 7f0a5259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a540c12d0 0x7f0a5414e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:52.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a5259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a540c12d0 0x7f0a5414e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42300/0 (socket says 172.21.15.175:42300) 2023-10-10T18:52:52.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a5259c700 1 -- 172.21.15.175:0/2900538525 learned_addr learned my addr 172.21.15.175:0/2900538525 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:52.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a51d9b700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a540c5850 0x7f0a5415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:52.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a52d9d700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a54156ba0 0x7f0a541593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:52.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a5259c700 1 -- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a54156ba0 msgr2=0x7f0a541593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:52.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a5259c700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a54156ba0 0x7f0a541593d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a5259c700 1 -- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a540c5850 msgr2=0x7f0a5415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:52.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a5259c700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a540c5850 0x7f0a5415aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a5259c700 1 -- 172.21.15.175:0/2900538525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c0094f0 con 0x7f0a540c12d0 2023-10-10T18:52:52.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a51d9b700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a540c5850 0x7f0a5415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:52:52.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a5259c700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a540c12d0 0x7f0a5414e4d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009eb0 tx=0x7f0a4400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:52.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a4a7fc700 1 -- 172.21.15.175:0/2900538525 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4400ccb0 con 0x7f0a540c12d0 2023-10-10T18:52:52.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a4a7fc700 1 -- 172.21.15.175:0/2900538525 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4400f040 con 0x7f0a540c12d0 2023-10-10T18:52:52.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a540c9890 con 0x7f0a540c12d0 2023-10-10T18:52:52.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a4a7fc700 1 -- 172.21.15.175:0/2900538525 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44014480 con 0x7f0a540c12d0 2023-10-10T18:52:52.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.386+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5415c440 con 0x7f0a540c12d0 2023-10-10T18:52:52.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.387+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a54063890 con 0x7f0a540c12d0 2023-10-10T18:52:52.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.387+0000 7f0a4a7fc700 1 -- 172.21.15.175:0/2900538525 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0a440147f0 con 0x7f0a540c12d0 2023-10-10T18:52:52.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.388+0000 7f0a4a7fc700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a34066f30 0x7f0a340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:52.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.388+0000 7f0a4a7fc700 1 -- 172.21.15.175:0/2900538525 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f0a44085110 con 0x7f0a540c12d0 2023-10-10T18:52:52.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.388+0000 7f0a51d9b700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a34066f30 0x7f0a340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:52.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.388+0000 7f0a51d9b700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a34066f30 0x7f0a340693e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c006f00 tx=0x7f0a3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:52.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.392+0000 7f0a4a7fc700 1 -- 172.21.15.175:0/2900538525 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a44052210 con 0x7f0a540c12d0 2023-10-10T18:52:52.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.621+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a54067570 con 0x7f0a540c12d0 2023-10-10T18:52:52.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.621+0000 7f0a4a7fc700 1 -- 172.21.15.175:0/2900538525 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5454 (secure 0 0 0) 0x7f0a44059e60 con 0x7f0a540c12d0 2023-10-10T18:52:52.623 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:52.624 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:50.507611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:clientreplay","state_seq":44,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:52.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a34066f30 msgr2=0x7f0a340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:52.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a34066f30 0x7f0a340693e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c006f00 tx=0x7f0a3c008040 comp rx=0 tx=0).stop 2023-10-10T18:52:52.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a540c12d0 msgr2=0x7f0a5414e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a540c12d0 0x7f0a5414e4d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009eb0 tx=0x7f0a4400b040 comp rx=0 tx=0).stop 2023-10-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 shutdown_connections 2023-10-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a34066f30 0x7f0a340693e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a54156ba0 0x7f0a541593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0a540c12d0 0x7f0a5414e4d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 --2- 172.21.15.175:0/2900538525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a540c5850 0x7f0a5415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 >> 172.21.15.175:0/2900538525 conn(0x7f0a5406dc70 msgr2=0x7f0a54152eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 shutdown_connections 2023-10-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:52.624+0000 7f0a58bb6700 1 -- 172.21.15.175:0/2900538525 wait complete. 2023-10-10T18:52:52.628 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 64 2023-10-10T18:52:52.636 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:54.637 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:54.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.765+0000 7fef6dbe5700 1 Processor -- start 2023-10-10T18:52:54.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.765+0000 7fef6dbe5700 1 -- start start 2023-10-10T18:52:54.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef6dbe5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6806e520 0x7fef681569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef6dbe5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fef6806e8f0 0x7fef6806c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef6dbe5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6806cca0 con 0x7fef680c9610 2023-10-10T18:52:54.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef6dbe5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef680c33c0 con 0x7fef6806e8f0 2023-10-10T18:52:54.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef6dbe5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef680c07f0 con 0x7fef6806e520 2023-10-10T18:52:54.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef6d3e4700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fef680c9610 0x7fef680c9a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58938/0 (socket says 172.21.15.175:58938) 2023-10-10T18:52:54.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef6d3e4700 1 -- 172.21.15.175:0/1791393395 learned_addr learned my addr 172.21.15.175:0/1791393395 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:54.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef6cbe3700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6806e520 0x7fef681569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef5ffff700 1 --2- 172.21.15.175:0/1791393395 >> v2:172.21.15.191:3300/0 conn(0x7fef6806e8f0 0x7fef6806c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467755103 0 0) 0x7fef6806cca0 con 0x7fef680c9610 2023-10-10T18:52:54.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.766+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef50003570 con 0x7fef680c9610 2023-10-10T18:52:54.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866935 0 0) 0x7fef50003570 con 0x7fef680c9610 2023-10-10T18:52:54.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6806e520 msgr2=0x7fef681569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:54.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6806e520 0x7fef681569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:54.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 >> v2:172.21.15.191:3300/0 conn(0x7fef6806e8f0 msgr2=0x7fef6806c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:54.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 --2- 172.21.15.175:0/1791393395 >> v2:172.21.15.191:3300/0 conn(0x7fef6806e8f0 0x7fef6806c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:54.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68152b70 con 0x7fef680c9610 2023-10-10T18:52:54.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fef60001470 con 0x7fef680c9610 2023-10-10T18:52:54.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 >> v1:172.21.15.175:6789/0 conn(0x7fef680c9610 legacy=0x7fef680c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:54.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef50003a60 0x7fef50003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004810 0x7fef50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6806e520 0x7fef500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef680c07f0 con 0x7fef50003a60 2023-10-10T18:52:54.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5000ba60 con 0x7fef50004810 2023-10-10T18:52:54.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5000bba0 con 0x7fef6806e520 2023-10-10T18:52:54.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef6cbe3700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef50003a60 0x7fef50003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef5ffff700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004810 0x7fef50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef6d3e4700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6806e520 0x7fef500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef6cbe3700 1 -- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6806e520 msgr2=0x7fef500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:54.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef6cbe3700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6806e520 0x7fef500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:54.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef6cbe3700 1 -- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004810 msgr2=0x7fef50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:54.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef6cbe3700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004810 0x7fef50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:54.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef6cbe3700 1 -- 172.21.15.175:0/1791393395 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef50003570 con 0x7fef50003a60 2023-10-10T18:52:54.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.767+0000 7fef6cbe3700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef50003a60 0x7fef50003e70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fef640129d0 tx=0x7fef64015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea95d86bfd1395fb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:54.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef64018040 con 0x7fef50003a60 2023-10-10T18:52:54.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6401c400 con 0x7fef50003a60 2023-10-10T18:52:54.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef5f7fe700 1 -- 172.21.15.175:0/1791393395 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6401c7c0 con 0x7fef50003a60 2023-10-10T18:52:54.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef6dbe5700 1 -- 172.21.15.175:0/1791393395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef50003a60 msgr2=0x7fef50003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:54.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef6dbe5700 1 --2- 172.21.15.175:0/1791393395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef50003a60 0x7fef50003e70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fef640129d0 tx=0x7fef64015040 comp rx=0 tx=0).stop 2023-10-10T18:52:54.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef6d3e4700 1 -- 172.21.15.175:0/1791393395 reap_dead start 2023-10-10T18:52:54.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef6dbe5700 1 -- 172.21.15.175:0/1791393395 shutdown_connections 2023-10-10T18:52:54.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef6dbe5700 1 -- 172.21.15.175:0/1791393395 >> 172.21.15.175:0/1791393395 conn(0x7fef68063230 msgr2=0x7fef6806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:54.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef6dbe5700 1 -- 172.21.15.175:0/1791393395 shutdown_connections 2023-10-10T18:52:54.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.768+0000 7fef6dbe5700 1 -- 172.21.15.175:0/1791393395 wait complete. 2023-10-10T18:52:54.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.769+0000 7fef6dbe5700 1 Processor -- start 2023-10-10T18:52:54.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.769+0000 7fef6dbe5700 1 -- start start 2023-10-10T18:52:54.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef6dbe5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef50003a60 0x7fef680c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef6dbe5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef6814ba10 0x7fef6814be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef6dbe5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680b4f10 0x7fef680b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef6dbe5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef680b58c0 con 0x7fef6814ba10 2023-10-10T18:52:54.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef6dbe5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef680c03c0 con 0x7fef680b4f10 2023-10-10T18:52:54.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef6dbe5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6814f4f0 con 0x7fef50003a60 2023-10-10T18:52:54.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef5ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef6814ba10 0x7fef6814be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef5ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef6814ba10 0x7fef6814be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53726/0 (socket says 172.21.15.175:53726) 2023-10-10T18:52:54.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef6cbe3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef50003a60 0x7fef680c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef5ffff700 1 -- 172.21.15.175:0/529532420 learned_addr learned my addr 172.21.15.175:0/529532420 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:54.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef6d3e4700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680b4f10 0x7fef680b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef5ffff700 1 -- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef50003a60 msgr2=0x7fef680c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:54.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef5ffff700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef50003a60 0x7fef680c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:54.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef5ffff700 1 -- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680b4f10 msgr2=0x7fef680b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:54.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef5ffff700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680b4f10 0x7fef680b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:54.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.770+0000 7fef5ffff700 1 -- 172.21.15.175:0/529532420 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef681610d0 con 0x7fef6814ba10 2023-10-10T18:52:54.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.771+0000 7fef5ffff700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef6814ba10 0x7fef6814be80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fef5800cb00 tx=0x7fef5800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:54.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.771+0000 7fef5cff9700 1 -- 172.21.15.175:0/529532420 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5800b810 con 0x7fef6814ba10 2023-10-10T18:52:54.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.771+0000 7fef5cff9700 1 -- 172.21.15.175:0/529532420 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef58013040 con 0x7fef6814ba10 2023-10-10T18:52:54.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.771+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6814e8e0 con 0x7fef6814ba10 2023-10-10T18:52:54.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.771+0000 7fef5cff9700 1 -- 172.21.15.175:0/529532420 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5800bbc0 con 0x7fef6814ba10 2023-10-10T18:52:54.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.771+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6814eea0 con 0x7fef6814ba10 2023-10-10T18:52:54.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.772+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6814f880 con 0x7fef6814ba10 2023-10-10T18:52:54.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.772+0000 7fef5cff9700 1 -- 172.21.15.175:0/529532420 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fef58021460 con 0x7fef6814ba10 2023-10-10T18:52:54.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.772+0000 7fef5cff9700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef54066f30 0x7fef540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:54.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.773+0000 7fef5cff9700 1 -- 172.21.15.175:0/529532420 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7fef58084750 con 0x7fef6814ba10 2023-10-10T18:52:54.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.773+0000 7fef6cbe3700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef54066f30 0x7fef540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:54.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.773+0000 7fef6cbe3700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef54066f30 0x7fef540693e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fef64013fd0 tx=0x7fef64022000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:54.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:54.777+0000 7fef5cff9700 1 -- 172.21.15.175:0/529532420 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef58051850 con 0x7fef6814ba10 2023-10-10T18:52:55.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.000+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef680c0b50 con 0x7fef6814ba10 2023-10-10T18:52:55.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.001+0000 7fef5cff9700 1 -- 172.21.15.175:0/529532420 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5454 (secure 0 0 0) 0x7fef58054e70 con 0x7fef6814ba10 2023-10-10T18:52:55.002 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:55.002 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:50.507611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:clientreplay","state_seq":44,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:55.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef54066f30 msgr2=0x7fef540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:55.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef54066f30 0x7fef540693e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fef64013fd0 tx=0x7fef64022000 comp rx=0 tx=0).stop 2023-10-10T18:52:55.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef6814ba10 msgr2=0x7fef6814be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:55.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef6814ba10 0x7fef6814be80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fef5800cb00 tx=0x7fef5800f040 comp rx=0 tx=0).stop 2023-10-10T18:52:55.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 shutdown_connections 2023-10-10T18:52:55.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef54066f30 0x7fef540693e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:55.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef680b4f10 0x7fef680b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:55.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef50003a60 0x7fef680c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:55.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 --2- 172.21.15.175:0/529532420 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef6814ba10 0x7fef6814be80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:55.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 >> 172.21.15.175:0/529532420 conn(0x7fef68063230 msgr2=0x7fef680c3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:55.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 shutdown_connections 2023-10-10T18:52:55.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:55.003+0000 7fef6dbe5700 1 -- 172.21.15.175:0/529532420 wait complete. 2023-10-10T18:52:55.007 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 64 2023-10-10T18:52:55.015 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:52:57.017 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:52:57.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.140+0000 7f89c0fe0700 1 Processor -- start 2023-10-10T18:52:57.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.140+0000 7f89c0fe0700 1 -- start start 2023-10-10T18:52:57.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.140+0000 7f89c0fe0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89bc08a100 0x7f89bc159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:57.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.140+0000 7f89c0fe0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f89bc0cd040 0x7f89bc0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:57.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89c0fe0700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc067fa0 con 0x7f89bc14fd90 2023-10-10T18:52:57.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89c0fe0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc0c5600 con 0x7f89bc0cd040 2023-10-10T18:52:57.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89c0fe0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc156b70 con 0x7f89bc08a100 2023-10-10T18:52:57.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89bbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f89bc0cd040 0x7f89bc0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:57.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89bb7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f89bc14fd90 0x7f89bc0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58944/0 (socket says 172.21.15.175:58944) 2023-10-10T18:52:57.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89bb7fe700 1 -- 172.21.15.175:0/1710894973 learned_addr learned my addr 172.21.15.175:0/1710894973 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:52:57.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89baffd700 1 --2- 172.21.15.175:0/1710894973 >> v2:172.21.15.191:3300/0 conn(0x7f89bc08a100 0x7f89bc159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:57.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89bbfff700 1 -- 172.21.15.175:0/1710894973 >> v2:172.21.15.191:3300/0 conn(0x7f89bc08a100 msgr2=0x7f89bc159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:52:57.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89bbfff700 1 --2- 172.21.15.175:0/1710894973 >> v2:172.21.15.191:3300/0 conn(0x7f89bc08a100 0x7f89bc159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89bbfff700 1 -- 172.21.15.175:0/1710894973 >> v1:172.21.15.175:6789/0 conn(0x7f89bc14fd90 legacy=0x7f89bc0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:57.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89bbfff700 1 -- 172.21.15.175:0/1710894973 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc150160 con 0x7f89bc0cd040 2023-10-10T18:52:57.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89ba7fc700 1 -- 172.21.15.175:0/1710894973 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690383969 0 0) 0x7f89bc067fa0 con 0x7f89bc14fd90 2023-10-10T18:52:57.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89bbfff700 1 --2- 172.21.15.175:0/1710894973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc0cd040 0x7f89bc0684f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f89bc069990 tx=0x7f89b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2278a319b255d887 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:57.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89ba7fc700 1 -- 172.21.15.175:0/1710894973 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b400e070 con 0x7f89bc0cd040 2023-10-10T18:52:57.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89ba7fc700 1 -- 172.21.15.175:0/1710894973 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b400bcc0 con 0x7f89bc0cd040 2023-10-10T18:52:57.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.141+0000 7f89ba7fc700 1 -- 172.21.15.175:0/1710894973 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4012680 con 0x7f89bc0cd040 2023-10-10T18:52:57.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.142+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1710894973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc0cd040 msgr2=0x7f89bc0684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:57.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.142+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1710894973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc0cd040 0x7f89bc0684f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f89bc069990 tx=0x7f89b4009780 comp rx=0 tx=0).stop 2023-10-10T18:52:57.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.142+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1710894973 shutdown_connections 2023-10-10T18:52:57.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.142+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1710894973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc0cd040 0x7f89bc0684f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.142+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1710894973 >> v2:172.21.15.191:3300/0 conn(0x7f89bc08a100 0x7f89bc159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.142+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1710894973 >> 172.21.15.175:0/1710894973 conn(0x7f89bc063d00 msgr2=0x7f89bc0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:57.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.142+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1710894973 shutdown_connections 2023-10-10T18:52:57.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.142+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1710894973 wait complete. 2023-10-10T18:52:57.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.143+0000 7f89c0fe0700 1 Processor -- start 2023-10-10T18:52:57.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.143+0000 7f89c0fe0700 1 -- start start 2023-10-10T18:52:57.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89c0fe0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc08a100 0x7f89bc1510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:57.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89c0fe0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89bc0cd040 0x7f89bc1531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:57.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89c0fe0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89bc14fd90 0x7f89bc154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:57.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89c0fe0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc150a60 con 0x7f89bc08a100 2023-10-10T18:52:57.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89c0fe0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0cade0 con 0x7f89bc0cd040 2023-10-10T18:52:57.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89c0fe0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc14eed0 con 0x7f89bc14fd90 2023-10-10T18:52:57.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc08a100 0x7f89bc1510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:57.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc08a100 0x7f89bc1510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53754/0 (socket says 172.21.15.175:53754) 2023-10-10T18:52:57.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 -- 172.21.15.175:0/1679108332 learned_addr learned my addr 172.21.15.175:0/1679108332 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:52:57.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bbfff700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89bc14fd90 0x7f89bc154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:57.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89baffd700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89bc0cd040 0x7f89bc1531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:57.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 -- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89bc14fd90 msgr2=0x7f89bc154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:57.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89bc14fd90 0x7f89bc154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 -- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89bc0cd040 msgr2=0x7f89bc1531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:57.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89bc0cd040 0x7f89bc1531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 -- 172.21.15.175:0/1679108332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b4009a20 con 0x7f89bc08a100 2023-10-10T18:52:57.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.144+0000 7f89bb7fe700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc08a100 0x7f89bc1510b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f89a400c5e0 tx=0x7f89a4001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:57.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.145+0000 7f89b37fe700 1 -- 172.21.15.175:0/1679108332 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4005b30 con 0x7f89bc08a100 2023-10-10T18:52:57.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.145+0000 7f89b37fe700 1 -- 172.21.15.175:0/1679108332 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a4005c90 con 0x7f89bc08a100 2023-10-10T18:52:57.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.145+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc14f1b0 con 0x7f89bc08a100 2023-10-10T18:52:57.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.145+0000 7f89b37fe700 1 -- 172.21.15.175:0/1679108332 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40072e0 con 0x7f89bc08a100 2023-10-10T18:52:57.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.145+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc0cb2b0 con 0x7f89bc08a100 2023-10-10T18:52:57.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.146+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc067570 con 0x7f89bc08a100 2023-10-10T18:52:57.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.146+0000 7f89b37fe700 1 -- 172.21.15.175:0/1679108332 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f89a401b3b0 con 0x7f89bc08a100 2023-10-10T18:52:57.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.146+0000 7f89b37fe700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f899c066f60 0x7f899c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:52:57.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.147+0000 7f89b37fe700 1 -- 172.21.15.175:0/1679108332 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f89a4083ad0 con 0x7f89bc08a100 2023-10-10T18:52:57.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.147+0000 7f89baffd700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f899c066f60 0x7f899c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:52:57.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.147+0000 7f89baffd700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f899c066f60 0x7f899c069410 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0ccb10 tx=0x7f89ac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:52:57.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.151+0000 7f89b37fe700 1 -- 172.21.15.175:0/1679108332 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89a404b5a0 con 0x7f89bc08a100 2023-10-10T18:52:57.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.373+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89bc155f80 con 0x7f89bc08a100 2023-10-10T18:52:57.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.374+0000 7f89b37fe700 1 -- 172.21.15.175:0/1679108332 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5448 (secure 0 0 0) 0x7f89a4017020 con 0x7f89bc08a100 2023-10-10T18:52:57.375 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:52:57.375 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:56.539778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:52:57.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f899c066f60 msgr2=0x7f899c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:57.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f899c066f60 0x7f899c069410 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0ccb10 tx=0x7f89ac008040 comp rx=0 tx=0).stop 2023-10-10T18:52:57.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc08a100 msgr2=0x7f89bc1510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:52:57.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc08a100 0x7f89bc1510b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f89a400c5e0 tx=0x7f89a4001070 comp rx=0 tx=0).stop 2023-10-10T18:52:57.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 shutdown_connections 2023-10-10T18:52:57.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f899c066f60 0x7f899c069410 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89bc14fd90 0x7f89bc154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89bc08a100 0x7f89bc1510b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 --2- 172.21.15.175:0/1679108332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89bc0cd040 0x7f89bc1531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:52:57.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 >> 172.21.15.175:0/1679108332 conn(0x7f89bc063d00 msgr2=0x7f89bc0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:52:57.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 shutdown_connections 2023-10-10T18:52:57.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:52:57.376+0000 7f89c0fe0700 1 -- 172.21.15.175:0/1679108332 wait complete. 2023-10-10T18:52:57.379 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 65 2023-10-10T18:52:57.388 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=3, replacing gid=4248 2023-10-10T18:52:57.389 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.c 2023-10-10T18:53:01.390 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-10T18:53:01.390 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-10T18:53:01.390 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-10T18:53:01.393 INFO:tasks.ceph.mds.c:Started 2023-10-10T18:53:01.393 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:01.474 INFO:tasks.ceph.mds.c.smithi175.stdout:starting mds.c at 2023-10-10T18:53:01.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.521+0000 7fb9c52cc700 1 Processor -- start 2023-10-10T18:53:01.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.521+0000 7fb9c52cc700 1 -- start start 2023-10-10T18:53:01.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9c52cc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9c52cc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c01574b0 con 0x7fb9c00b1fc0 2023-10-10T18:53:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9c52cc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00afad0 con 0x7fb9c00b2390 2023-10-10T18:53:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9c52cc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0150770 con 0x7fb9c0155fb0 2023-10-10T18:53:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9beffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb9c00b1fc0 0x7fb9c0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58960/0 (socket says 172.21.15.175:58960) 2023-10-10T18:53:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9beffd700 1 -- 172.21.15.175:0/4240870730 learned_addr learned my addr 172.21.15.175:0/4240870730 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bf7fe700 1 --2- 172.21.15.175:0/4240870730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bdffb700 1 -- 172.21.15.175:0/4240870730 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476937165 0 0) 0x7fb9c01574b0 con 0x7fb9c00b1fc0 2023-10-10T18:53:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bdffb700 1 -- 172.21.15.175:0/4240870730 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a8003570 con 0x7fb9c00b1fc0 2023-10-10T18:53:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bdffb700 1 -- 172.21.15.175:0/4240870730 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924148083 0 0) 0x7fb9c00afad0 con 0x7fb9c00b2390 2023-10-10T18:53:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bdffb700 1 -- 172.21.15.175:0/4240870730 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c01574b0 con 0x7fb9c00b2390 2023-10-10T18:53:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bdffb700 1 -- 172.21.15.175:0/4240870730 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580444842 0 0) 0x7fb9a8003570 con 0x7fb9c00b1fc0 2023-10-10T18:53:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bf7fe700 1 -- 172.21.15.175:0/4240870730 >> v1:172.21.15.191:6789/0 conn(0x7fb9c00b2390 legacy=0x7fb9c00aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bf7fe700 1 -- 172.21.15.175:0/4240870730 >> v1:172.21.15.175:6789/0 conn(0x7fb9c00b1fc0 legacy=0x7fb9c0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bf7fe700 1 -- 172.21.15.175:0/4240870730 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00af4c0 con 0x7fb9c0155fb0 2023-10-10T18:53:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.522+0000 7fb9bf7fe700 1 --2- 172.21.15.175:0/4240870730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0159920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009e90 tx=0x7fb9b8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=448cddab6627e157 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9bdffb700 1 -- 172.21.15.175:0/4240870730 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b800e070 con 0x7fb9c0155fb0 2023-10-10T18:53:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9bdffb700 1 -- 172.21.15.175:0/4240870730 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b800b900 con 0x7fb9c0155fb0 2023-10-10T18:53:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9bdffb700 1 -- 172.21.15.175:0/4240870730 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4001470 con 0x7fb9c0155fb0 2023-10-10T18:53:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9c52cc700 1 -- 172.21.15.175:0/4240870730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 msgr2=0x7fb9c0159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9c52cc700 1 --2- 172.21.15.175:0/4240870730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0159920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009e90 tx=0x7fb9b8009780 comp rx=0 tx=0).stop 2023-10-10T18:53:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9c52cc700 1 -- 172.21.15.175:0/4240870730 shutdown_connections 2023-10-10T18:53:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9c52cc700 1 --2- 172.21.15.175:0/4240870730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0159920 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9c52cc700 1 -- 172.21.15.175:0/4240870730 >> 172.21.15.175:0/4240870730 conn(0x7fb9c0063230 msgr2=0x7fb9c0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9c52cc700 1 -- 172.21.15.175:0/4240870730 shutdown_connections 2023-10-10T18:53:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.523+0000 7fb9c52cc700 1 -- 172.21.15.175:0/4240870730 wait complete. 2023-10-10T18:53:01.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.524+0000 7fb9c52cc700 1 Processor -- start 2023-10-10T18:53:01.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9c52cc700 1 -- start start 2023-10-10T18:53:01.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9c52cc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9c00b1fc0 0x7fb9c014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:01.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9c52cc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c00b2390 0x7fb9c014d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:01.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9c52cc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:01.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9c52cc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00cc690 con 0x7fb9c00b2390 2023-10-10T18:53:01.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9c52cc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00cbaa0 con 0x7fb9c00b1fc0 2023-10-10T18:53:01.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9c52cc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c014eaf0 con 0x7fb9c0155fb0 2023-10-10T18:53:01.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9be7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c00b2390 0x7fb9c014d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:01.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9be7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c00b2390 0x7fb9c014d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53780/0 (socket says 172.21.15.175:53780) 2023-10-10T18:53:01.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9be7fc700 1 -- 172.21.15.175:0/85681406 learned_addr learned my addr 172.21.15.175:0/85681406 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:01.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9beffd700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9c00b1fc0 0x7fb9c014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:01.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.525+0000 7fb9bf7fe700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:01.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9be7fc700 1 -- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 msgr2=0x7fb9c0148c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:01.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9be7fc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0148c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:01.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9be7fc700 1 -- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9c00b1fc0 msgr2=0x7fb9c014f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:01.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9be7fc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9c00b1fc0 0x7fb9c014f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:01.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9be7fc700 1 -- 172.21.15.175:0/85681406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8009a20 con 0x7fb9c00b2390 2023-10-10T18:53:01.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9be7fc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c00b2390 0x7fb9c014d380 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c790 tx=0x7fb9b0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:01.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9a6ffd700 1 -- 172.21.15.175:0/85681406 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0010510 con 0x7fb9c00b2390 2023-10-10T18:53:01.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9a6ffd700 1 -- 172.21.15.175:0/85681406 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b0010670 con 0x7fb9c00b2390 2023-10-10T18:53:01.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c00cbd60 con 0x7fb9c00b2390 2023-10-10T18:53:01.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9a6ffd700 1 -- 172.21.15.175:0/85681406 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0010a10 con 0x7fb9c00b2390 2023-10-10T18:53:01.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.526+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c00cc320 con 0x7fb9c00b2390 2023-10-10T18:53:01.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.527+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c00cc4e0 con 0x7fb9c00b2390 2023-10-10T18:53:01.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.527+0000 7fb9a6ffd700 1 -- 172.21.15.175:0/85681406 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb9b001b3b0 con 0x7fb9c00b2390 2023-10-10T18:53:01.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.528+0000 7fb9a6ffd700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ac066f30 0x7fb9ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:01.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.528+0000 7fb9a6ffd700 1 -- 172.21.15.175:0/85681406 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7fb9b0083ef0 con 0x7fb9c00b2390 2023-10-10T18:53:01.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.528+0000 7fb9beffd700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ac066f30 0x7fb9ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:01.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.528+0000 7fb9beffd700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ac066f30 0x7fb9ac0693e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4004de0 tx=0x7fb9b4000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:01.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.532+0000 7fb9a6ffd700 1 -- 172.21.15.175:0/85681406 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb9b0050ff0 con 0x7fb9c00b2390 2023-10-10T18:53:01.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.773+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9c0067570 con 0x7fb9c00b2390 2023-10-10T18:53:01.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.773+0000 7fb9a6ffd700 1 -- 172.21.15.175:0/85681406 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6207 (secure 0 0 0) 0x7fb9b0017020 con 0x7fb9c00b2390 2023-10-10T18:53:01.775 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:01.775 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5367,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:52:56.539778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:01.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ac066f30 msgr2=0x7fb9ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:01.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ac066f30 0x7fb9ac0693e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4004de0 tx=0x7fb9b4000bc0 comp rx=0 tx=0).stop 2023-10-10T18:53:01.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c00b2390 msgr2=0x7fb9c014d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:01.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c00b2390 0x7fb9c014d380 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c790 tx=0x7fb9b0007e10 comp rx=0 tx=0).stop 2023-10-10T18:53:01.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 shutdown_connections 2023-10-10T18:53:01.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb9ac066f30 0x7fb9ac0693e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:01.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9c0155fb0 0x7fb9c0148c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:01.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9c00b1fc0 0x7fb9c014f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:01.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 --2- 172.21.15.175:0/85681406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c00b2390 0x7fb9c014d380 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:01.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 >> 172.21.15.175:0/85681406 conn(0x7fb9c0063230 msgr2=0x7fb9c00c8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:01.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 shutdown_connections 2023-10-10T18:53:01.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:01.775+0000 7fb9c52cc700 1 -- 172.21.15.175:0/85681406 wait complete. 2023-10-10T18:53:01.779 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 66 2023-10-10T18:53:01.788 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-10T18:53:01.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 18 secs before thrashing 2023-10-10T18:53:03.958 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/31' 2023-10-10T18:53:03.958 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/36' 2023-10-10T18:53:03.958 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/20' 2023-10-10T18:53:03.959 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/39' 2023-10-10T18:53:03.959 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/34' 2023-10-10T18:53:03.959 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/38' 2023-10-10T18:53:03.959 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/28' 2023-10-10T18:53:03.959 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/24' 2023-10-10T18:53:03.960 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/29' 2023-10-10T18:53:03.960 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/32' 2023-10-10T18:53:03.960 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/22' 2023-10-10T18:53:03.960 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/25' 2023-10-10T18:53:03.960 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/30' 2023-10-10T18:53:03.961 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/33' 2023-10-10T18:53:03.961 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/23' 2023-10-10T18:53:03.961 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/35' 2023-10-10T18:53:03.961 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/26' 2023-10-10T18:53:03.961 INFO:tasks.workunit.client.0.smithi175.stdout:removed './297/37' 2023-10-10T18:53:03.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './297' 2023-10-10T18:53:03.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/21' 2023-10-10T18:53:03.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/27' 2023-10-10T18:53:03.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/31' 2023-10-10T18:53:03.962 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/36' 2023-10-10T18:53:03.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/20' 2023-10-10T18:53:03.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/39' 2023-10-10T18:53:03.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/34' 2023-10-10T18:53:03.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/38' 2023-10-10T18:53:03.963 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/28' 2023-10-10T18:53:03.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/24' 2023-10-10T18:53:03.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/29' 2023-10-10T18:53:03.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/32' 2023-10-10T18:53:03.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/22' 2023-10-10T18:53:03.964 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/25' 2023-10-10T18:53:03.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/30' 2023-10-10T18:53:03.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/33' 2023-10-10T18:53:03.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/23' 2023-10-10T18:53:03.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/35' 2023-10-10T18:53:03.965 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/26' 2023-10-10T18:53:03.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './298/37' 2023-10-10T18:53:03.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './298' 2023-10-10T18:53:03.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/21' 2023-10-10T18:53:03.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/27' 2023-10-10T18:53:03.966 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/31' 2023-10-10T18:53:03.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/36' 2023-10-10T18:53:03.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/20' 2023-10-10T18:53:03.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/39' 2023-10-10T18:53:03.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/34' 2023-10-10T18:53:03.967 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/38' 2023-10-10T18:53:03.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/28' 2023-10-10T18:53:03.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/24' 2023-10-10T18:53:03.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/29' 2023-10-10T18:53:03.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/32' 2023-10-10T18:53:03.968 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/22' 2023-10-10T18:53:03.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/25' 2023-10-10T18:53:03.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/30' 2023-10-10T18:53:03.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/33' 2023-10-10T18:53:03.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/23' 2023-10-10T18:53:03.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/35' 2023-10-10T18:53:03.969 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/26' 2023-10-10T18:53:03.970 INFO:tasks.workunit.client.0.smithi175.stdout:removed './299/37' 2023-10-10T18:53:03.970 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './299' 2023-10-10T18:53:03.970 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/19' 2023-10-10T18:53:03.970 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/15' 2023-10-10T18:53:03.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/20' 2023-10-10T18:53:03.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/12' 2023-10-10T18:53:03.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/10' 2023-10-10T18:53:03.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/11' 2023-10-10T18:53:03.971 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/16' 2023-10-10T18:53:03.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/14' 2023-10-10T18:53:03.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/18' 2023-10-10T18:53:03.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/13' 2023-10-10T18:53:03.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/166/17' 2023-10-10T18:53:03.972 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/166' 2023-10-10T18:53:03.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/19' 2023-10-10T18:53:03.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/15' 2023-10-10T18:53:03.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/20' 2023-10-10T18:53:03.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/12' 2023-10-10T18:53:03.973 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/10' 2023-10-10T18:53:03.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/11' 2023-10-10T18:53:03.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/16' 2023-10-10T18:53:03.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/14' 2023-10-10T18:53:03.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/18' 2023-10-10T18:53:03.974 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/13' 2023-10-10T18:53:03.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/147/17' 2023-10-10T18:53:03.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/147' 2023-10-10T18:53:03.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/19' 2023-10-10T18:53:03.975 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/15' 2023-10-10T18:53:03.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/20' 2023-10-10T18:53:03.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/12' 2023-10-10T18:53:03.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/10' 2023-10-10T18:53:03.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/11' 2023-10-10T18:53:03.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/16' 2023-10-10T18:53:03.976 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/14' 2023-10-10T18:53:03.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/18' 2023-10-10T18:53:03.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/13' 2023-10-10T18:53:03.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/151/17' 2023-10-10T18:53:03.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/151' 2023-10-10T18:53:03.977 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/19' 2023-10-10T18:53:03.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/15' 2023-10-10T18:53:03.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/20' 2023-10-10T18:53:03.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/12' 2023-10-10T18:53:03.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/10' 2023-10-10T18:53:03.978 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/11' 2023-10-10T18:53:03.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/16' 2023-10-10T18:53:03.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/14' 2023-10-10T18:53:03.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/18' 2023-10-10T18:53:03.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/13' 2023-10-10T18:53:03.979 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/170/17' 2023-10-10T18:53:03.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/170' 2023-10-10T18:53:03.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/19' 2023-10-10T18:53:03.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/15' 2023-10-10T18:53:03.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/20' 2023-10-10T18:53:03.980 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/12' 2023-10-10T18:53:03.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/10' 2023-10-10T18:53:03.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/11' 2023-10-10T18:53:03.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/16' 2023-10-10T18:53:03.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/14' 2023-10-10T18:53:03.981 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/18' 2023-10-10T18:53:03.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/13' 2023-10-10T18:53:03.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/167/17' 2023-10-10T18:53:03.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/167' 2023-10-10T18:53:03.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/19' 2023-10-10T18:53:03.982 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/15' 2023-10-10T18:53:03.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/20' 2023-10-10T18:53:03.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/12' 2023-10-10T18:53:03.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/10' 2023-10-10T18:53:03.983 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/11' 2023-10-10T18:53:03.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/16' 2023-10-10T18:53:03.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/14' 2023-10-10T18:53:03.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/18' 2023-10-10T18:53:03.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/13' 2023-10-10T18:53:03.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/140/17' 2023-10-10T18:53:03.984 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/140' 2023-10-10T18:53:03.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/19' 2023-10-10T18:53:03.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/15' 2023-10-10T18:53:03.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/20' 2023-10-10T18:53:03.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/12' 2023-10-10T18:53:03.985 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/10' 2023-10-10T18:53:05.698 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/11' 2023-10-10T18:53:05.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/16' 2023-10-10T18:53:05.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/14' 2023-10-10T18:53:05.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/18' 2023-10-10T18:53:05.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/13' 2023-10-10T18:53:05.699 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/142/17' 2023-10-10T18:53:05.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/142' 2023-10-10T18:53:05.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/19' 2023-10-10T18:53:05.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/15' 2023-10-10T18:53:05.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/20' 2023-10-10T18:53:05.700 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/12' 2023-10-10T18:53:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/10' 2023-10-10T18:53:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/11' 2023-10-10T18:53:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/16' 2023-10-10T18:53:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/14' 2023-10-10T18:53:05.701 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/18' 2023-10-10T18:53:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/13' 2023-10-10T18:53:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/150/17' 2023-10-10T18:53:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/150' 2023-10-10T18:53:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/19' 2023-10-10T18:53:05.702 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/15' 2023-10-10T18:53:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/20' 2023-10-10T18:53:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/12' 2023-10-10T18:53:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/10' 2023-10-10T18:53:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/11' 2023-10-10T18:53:05.703 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/16' 2023-10-10T18:53:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/14' 2023-10-10T18:53:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/18' 2023-10-10T18:53:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/13' 2023-10-10T18:53:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/105/17' 2023-10-10T18:53:05.704 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/105' 2023-10-10T18:53:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/19' 2023-10-10T18:53:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/15' 2023-10-10T18:53:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/20' 2023-10-10T18:53:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/12' 2023-10-10T18:53:05.705 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/10' 2023-10-10T18:53:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/11' 2023-10-10T18:53:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/16' 2023-10-10T18:53:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/14' 2023-10-10T18:53:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/18' 2023-10-10T18:53:05.706 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/13' 2023-10-10T18:53:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/149/17' 2023-10-10T18:53:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/149' 2023-10-10T18:53:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/19' 2023-10-10T18:53:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/15' 2023-10-10T18:53:05.707 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/20' 2023-10-10T18:53:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/12' 2023-10-10T18:53:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/10' 2023-10-10T18:53:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/11' 2023-10-10T18:53:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/16' 2023-10-10T18:53:05.708 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/14' 2023-10-10T18:53:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/18' 2023-10-10T18:53:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/13' 2023-10-10T18:53:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/109/17' 2023-10-10T18:53:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/109' 2023-10-10T18:53:05.709 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/19' 2023-10-10T18:53:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/15' 2023-10-10T18:53:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/20' 2023-10-10T18:53:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/12' 2023-10-10T18:53:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/10' 2023-10-10T18:53:05.710 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/11' 2023-10-10T18:53:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/16' 2023-10-10T18:53:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/14' 2023-10-10T18:53:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/18' 2023-10-10T18:53:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/13' 2023-10-10T18:53:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/158/17' 2023-10-10T18:53:05.711 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/158' 2023-10-10T18:53:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/19' 2023-10-10T18:53:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/15' 2023-10-10T18:53:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/20' 2023-10-10T18:53:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/12' 2023-10-10T18:53:05.712 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/10' 2023-10-10T18:53:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/11' 2023-10-10T18:53:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/16' 2023-10-10T18:53:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/14' 2023-10-10T18:53:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/18' 2023-10-10T18:53:05.713 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/13' 2023-10-10T18:53:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/106/17' 2023-10-10T18:53:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/106' 2023-10-10T18:53:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/19' 2023-10-10T18:53:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/15' 2023-10-10T18:53:05.714 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/20' 2023-10-10T18:53:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/12' 2023-10-10T18:53:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/10' 2023-10-10T18:53:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/11' 2023-10-10T18:53:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/16' 2023-10-10T18:53:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/14' 2023-10-10T18:53:05.715 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/18' 2023-10-10T18:53:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/13' 2023-10-10T18:53:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/189/17' 2023-10-10T18:53:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/189' 2023-10-10T18:53:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/19' 2023-10-10T18:53:05.716 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/15' 2023-10-10T18:53:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/20' 2023-10-10T18:53:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/12' 2023-10-10T18:53:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/10' 2023-10-10T18:53:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/11' 2023-10-10T18:53:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/16' 2023-10-10T18:53:05.717 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/14' 2023-10-10T18:53:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/18' 2023-10-10T18:53:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/13' 2023-10-10T18:53:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/133/17' 2023-10-10T18:53:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/133' 2023-10-10T18:53:05.718 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/19' 2023-10-10T18:53:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/15' 2023-10-10T18:53:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/20' 2023-10-10T18:53:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/12' 2023-10-10T18:53:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/10' 2023-10-10T18:53:05.719 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/11' 2023-10-10T18:53:12.295 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/16' 2023-10-10T18:53:12.295 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/14' 2023-10-10T18:53:12.295 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/18' 2023-10-10T18:53:12.296 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/13' 2023-10-10T18:53:12.296 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/173/17' 2023-10-10T18:53:12.296 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/173' 2023-10-10T18:53:12.296 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/19' 2023-10-10T18:53:12.296 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/15' 2023-10-10T18:53:12.297 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/20' 2023-10-10T18:53:12.297 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/12' 2023-10-10T18:53:12.297 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/10' 2023-10-10T18:53:12.297 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/11' 2023-10-10T18:53:12.297 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/16' 2023-10-10T18:53:12.298 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/14' 2023-10-10T18:53:12.298 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/18' 2023-10-10T18:53:12.298 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/13' 2023-10-10T18:53:12.298 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/174/17' 2023-10-10T18:53:12.298 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/174' 2023-10-10T18:53:12.299 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/19' 2023-10-10T18:53:12.299 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/15' 2023-10-10T18:53:12.299 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/20' 2023-10-10T18:53:12.299 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/12' 2023-10-10T18:53:12.299 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/10' 2023-10-10T18:53:12.300 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/11' 2023-10-10T18:53:12.300 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/16' 2023-10-10T18:53:12.300 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/14' 2023-10-10T18:53:12.300 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/18' 2023-10-10T18:53:12.300 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/13' 2023-10-10T18:53:12.301 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/155/17' 2023-10-10T18:53:12.301 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/155' 2023-10-10T18:53:12.301 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/19' 2023-10-10T18:53:12.301 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/15' 2023-10-10T18:53:12.301 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/20' 2023-10-10T18:53:12.302 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/12' 2023-10-10T18:53:12.302 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/10' 2023-10-10T18:53:12.302 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/11' 2023-10-10T18:53:12.302 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/16' 2023-10-10T18:53:12.302 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/14' 2023-10-10T18:53:12.303 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/18' 2023-10-10T18:53:12.303 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/13' 2023-10-10T18:53:12.303 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/114/17' 2023-10-10T18:53:12.303 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/114' 2023-10-10T18:53:12.303 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/19' 2023-10-10T18:53:12.304 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/15' 2023-10-10T18:53:12.304 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/20' 2023-10-10T18:53:12.304 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/12' 2023-10-10T18:53:12.304 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/10' 2023-10-10T18:53:12.304 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/11' 2023-10-10T18:53:12.305 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/16' 2023-10-10T18:53:12.305 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/14' 2023-10-10T18:53:12.305 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/18' 2023-10-10T18:53:12.305 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/13' 2023-10-10T18:53:12.305 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/159/17' 2023-10-10T18:53:12.305 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/159' 2023-10-10T18:53:12.306 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/19' 2023-10-10T18:53:12.306 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/15' 2023-10-10T18:53:12.306 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/20' 2023-10-10T18:53:12.306 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/12' 2023-10-10T18:53:12.306 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/10' 2023-10-10T18:53:12.307 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/11' 2023-10-10T18:53:12.307 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/16' 2023-10-10T18:53:12.307 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/14' 2023-10-10T18:53:12.307 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/18' 2023-10-10T18:53:12.307 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/13' 2023-10-10T18:53:12.307 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/187/17' 2023-10-10T18:53:12.308 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/187' 2023-10-10T18:53:12.308 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/19' 2023-10-10T18:53:12.308 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/15' 2023-10-10T18:53:12.308 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/20' 2023-10-10T18:53:12.308 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/12' 2023-10-10T18:53:12.309 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/10' 2023-10-10T18:53:12.309 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/11' 2023-10-10T18:53:12.309 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/16' 2023-10-10T18:53:12.309 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/14' 2023-10-10T18:53:12.309 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/18' 2023-10-10T18:53:12.309 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/13' 2023-10-10T18:53:12.310 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/185/17' 2023-10-10T18:53:12.310 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/185' 2023-10-10T18:53:12.310 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/19' 2023-10-10T18:53:12.310 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/15' 2023-10-10T18:53:12.310 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/20' 2023-10-10T18:53:12.311 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/12' 2023-10-10T18:53:12.311 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/10' 2023-10-10T18:53:12.311 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/11' 2023-10-10T18:53:12.311 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/16' 2023-10-10T18:53:12.311 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/14' 2023-10-10T18:53:12.312 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/18' 2023-10-10T18:53:12.312 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/13' 2023-10-10T18:53:12.312 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/120/17' 2023-10-10T18:53:12.312 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/120' 2023-10-10T18:53:12.312 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/19' 2023-10-10T18:53:12.312 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/15' 2023-10-10T18:53:12.313 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/20' 2023-10-10T18:53:12.313 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/12' 2023-10-10T18:53:12.313 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/10' 2023-10-10T18:53:12.313 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/11' 2023-10-10T18:53:12.313 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/16' 2023-10-10T18:53:12.314 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/14' 2023-10-10T18:53:12.314 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/18' 2023-10-10T18:53:12.314 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/13' 2023-10-10T18:53:12.314 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/110/17' 2023-10-10T18:53:12.314 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/110' 2023-10-10T18:53:12.315 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/19' 2023-10-10T18:53:12.315 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/15' 2023-10-10T18:53:12.315 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/20' 2023-10-10T18:53:12.315 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/12' 2023-10-10T18:53:12.315 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/10' 2023-10-10T18:53:12.316 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/11' 2023-10-10T18:53:12.316 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/16' 2023-10-10T18:53:12.618 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/14' 2023-10-10T18:53:12.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/18' 2023-10-10T18:53:12.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/13' 2023-10-10T18:53:12.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/137/17' 2023-10-10T18:53:12.619 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/137' 2023-10-10T18:53:12.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/19' 2023-10-10T18:53:12.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/15' 2023-10-10T18:53:12.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/20' 2023-10-10T18:53:12.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/12' 2023-10-10T18:53:12.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/10' 2023-10-10T18:53:12.620 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/11' 2023-10-10T18:53:12.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/16' 2023-10-10T18:53:12.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/14' 2023-10-10T18:53:12.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/18' 2023-10-10T18:53:12.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/13' 2023-10-10T18:53:12.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/198/17' 2023-10-10T18:53:12.621 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/198' 2023-10-10T18:53:12.622 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/19' 2023-10-10T18:53:12.622 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/15' 2023-10-10T18:53:12.622 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/20' 2023-10-10T18:53:12.622 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/12' 2023-10-10T18:53:12.622 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/10' 2023-10-10T18:53:12.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/11' 2023-10-10T18:53:12.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/16' 2023-10-10T18:53:12.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/14' 2023-10-10T18:53:12.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/18' 2023-10-10T18:53:12.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/13' 2023-10-10T18:53:12.623 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/136/17' 2023-10-10T18:53:12.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/136' 2023-10-10T18:53:12.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/19' 2023-10-10T18:53:12.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/15' 2023-10-10T18:53:12.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/20' 2023-10-10T18:53:12.624 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/12' 2023-10-10T18:53:12.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/10' 2023-10-10T18:53:12.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/11' 2023-10-10T18:53:12.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/16' 2023-10-10T18:53:12.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/14' 2023-10-10T18:53:12.625 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/18' 2023-10-10T18:53:12.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/13' 2023-10-10T18:53:12.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/125/17' 2023-10-10T18:53:12.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/125' 2023-10-10T18:53:12.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/19' 2023-10-10T18:53:12.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/15' 2023-10-10T18:53:12.626 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/20' 2023-10-10T18:53:12.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/12' 2023-10-10T18:53:12.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/10' 2023-10-10T18:53:12.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/11' 2023-10-10T18:53:12.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/16' 2023-10-10T18:53:12.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/14' 2023-10-10T18:53:12.627 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/18' 2023-10-10T18:53:12.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/13' 2023-10-10T18:53:12.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/194/17' 2023-10-10T18:53:12.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/194' 2023-10-10T18:53:12.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/19' 2023-10-10T18:53:12.628 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/15' 2023-10-10T18:53:12.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/20' 2023-10-10T18:53:12.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/12' 2023-10-10T18:53:12.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/10' 2023-10-10T18:53:12.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/11' 2023-10-10T18:53:12.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/16' 2023-10-10T18:53:12.629 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/14' 2023-10-10T18:53:12.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/18' 2023-10-10T18:53:12.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/13' 2023-10-10T18:53:12.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/119/17' 2023-10-10T18:53:12.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/119' 2023-10-10T18:53:12.630 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/19' 2023-10-10T18:53:12.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/15' 2023-10-10T18:53:12.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/20' 2023-10-10T18:53:12.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/12' 2023-10-10T18:53:12.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/10' 2023-10-10T18:53:12.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/11' 2023-10-10T18:53:12.631 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/16' 2023-10-10T18:53:12.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/14' 2023-10-10T18:53:12.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/18' 2023-10-10T18:53:12.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/13' 2023-10-10T18:53:12.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/157/17' 2023-10-10T18:53:12.632 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/157' 2023-10-10T18:53:12.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/19' 2023-10-10T18:53:12.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/15' 2023-10-10T18:53:12.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/20' 2023-10-10T18:53:12.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/12' 2023-10-10T18:53:12.633 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/10' 2023-10-10T18:53:12.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/11' 2023-10-10T18:53:12.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/16' 2023-10-10T18:53:12.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/14' 2023-10-10T18:53:12.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/18' 2023-10-10T18:53:12.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/13' 2023-10-10T18:53:12.634 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/172/17' 2023-10-10T18:53:12.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/172' 2023-10-10T18:53:12.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/19' 2023-10-10T18:53:12.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/15' 2023-10-10T18:53:12.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/20' 2023-10-10T18:53:12.635 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/12' 2023-10-10T18:53:12.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/10' 2023-10-10T18:53:12.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/11' 2023-10-10T18:53:12.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/16' 2023-10-10T18:53:12.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/14' 2023-10-10T18:53:12.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/18' 2023-10-10T18:53:12.636 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/13' 2023-10-10T18:53:12.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/186/17' 2023-10-10T18:53:12.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/186' 2023-10-10T18:53:12.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/19' 2023-10-10T18:53:12.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/15' 2023-10-10T18:53:12.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/20' 2023-10-10T18:53:12.637 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/12' 2023-10-10T18:53:12.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/10' 2023-10-10T18:53:12.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/11' 2023-10-10T18:53:12.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/16' 2023-10-10T18:53:12.638 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/14' 2023-10-10T18:53:17.805 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/18' 2023-10-10T18:53:17.806 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/13' 2023-10-10T18:53:17.806 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/190/17' 2023-10-10T18:53:17.806 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/190' 2023-10-10T18:53:17.806 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/19' 2023-10-10T18:53:17.806 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/15' 2023-10-10T18:53:17.807 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/20' 2023-10-10T18:53:17.807 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/12' 2023-10-10T18:53:17.807 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/10' 2023-10-10T18:53:17.807 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/11' 2023-10-10T18:53:17.807 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/16' 2023-10-10T18:53:17.808 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/14' 2023-10-10T18:53:17.808 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/18' 2023-10-10T18:53:17.808 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/13' 2023-10-10T18:53:17.808 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/104/17' 2023-10-10T18:53:17.808 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/104' 2023-10-10T18:53:17.809 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/19' 2023-10-10T18:53:17.809 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/15' 2023-10-10T18:53:17.809 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/20' 2023-10-10T18:53:17.809 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/12' 2023-10-10T18:53:17.809 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/10' 2023-10-10T18:53:17.810 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/11' 2023-10-10T18:53:17.810 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/16' 2023-10-10T18:53:17.810 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/14' 2023-10-10T18:53:17.810 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/18' 2023-10-10T18:53:17.810 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/13' 2023-10-10T18:53:17.811 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/164/17' 2023-10-10T18:53:17.811 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/164' 2023-10-10T18:53:17.811 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/19' 2023-10-10T18:53:17.811 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/15' 2023-10-10T18:53:17.811 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/20' 2023-10-10T18:53:17.811 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/12' 2023-10-10T18:53:17.812 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/10' 2023-10-10T18:53:17.812 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/11' 2023-10-10T18:53:17.812 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/16' 2023-10-10T18:53:17.812 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/14' 2023-10-10T18:53:17.812 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/18' 2023-10-10T18:53:17.813 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/13' 2023-10-10T18:53:17.813 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/112/17' 2023-10-10T18:53:17.813 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/112' 2023-10-10T18:53:17.813 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/19' 2023-10-10T18:53:17.813 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/15' 2023-10-10T18:53:17.814 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/20' 2023-10-10T18:53:17.814 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/12' 2023-10-10T18:53:17.814 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/10' 2023-10-10T18:53:17.814 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/11' 2023-10-10T18:53:17.814 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/16' 2023-10-10T18:53:17.815 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/14' 2023-10-10T18:53:17.815 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/18' 2023-10-10T18:53:17.815 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/13' 2023-10-10T18:53:17.815 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/100/17' 2023-10-10T18:53:17.815 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/100' 2023-10-10T18:53:17.816 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/19' 2023-10-10T18:53:17.816 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/15' 2023-10-10T18:53:17.816 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/20' 2023-10-10T18:53:17.816 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/12' 2023-10-10T18:53:17.816 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/10' 2023-10-10T18:53:17.817 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/11' 2023-10-10T18:53:17.817 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/16' 2023-10-10T18:53:17.817 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/14' 2023-10-10T18:53:17.817 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/18' 2023-10-10T18:53:17.817 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/13' 2023-10-10T18:53:17.818 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/134/17' 2023-10-10T18:53:17.818 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/134' 2023-10-10T18:53:17.818 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/19' 2023-10-10T18:53:17.818 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/15' 2023-10-10T18:53:17.818 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/20' 2023-10-10T18:53:17.819 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/12' 2023-10-10T18:53:17.819 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/10' 2023-10-10T18:53:17.819 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/11' 2023-10-10T18:53:17.819 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/16' 2023-10-10T18:53:17.819 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/14' 2023-10-10T18:53:17.819 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/18' 2023-10-10T18:53:17.820 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/13' 2023-10-10T18:53:17.820 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/115/17' 2023-10-10T18:53:17.820 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/115' 2023-10-10T18:53:17.820 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/19' 2023-10-10T18:53:17.820 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/15' 2023-10-10T18:53:17.821 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/20' 2023-10-10T18:53:17.821 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/12' 2023-10-10T18:53:17.821 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/10' 2023-10-10T18:53:17.821 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/11' 2023-10-10T18:53:17.821 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/16' 2023-10-10T18:53:17.822 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/14' 2023-10-10T18:53:17.822 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/18' 2023-10-10T18:53:17.822 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/13' 2023-10-10T18:53:17.822 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/116/17' 2023-10-10T18:53:17.822 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/116' 2023-10-10T18:53:17.823 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/19' 2023-10-10T18:53:17.823 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/15' 2023-10-10T18:53:17.823 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/20' 2023-10-10T18:53:17.823 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/12' 2023-10-10T18:53:17.823 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/10' 2023-10-10T18:53:17.823 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/11' 2023-10-10T18:53:17.824 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/16' 2023-10-10T18:53:17.824 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/14' 2023-10-10T18:53:17.824 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/18' 2023-10-10T18:53:17.824 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/13' 2023-10-10T18:53:17.824 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/195/17' 2023-10-10T18:53:17.825 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/195' 2023-10-10T18:53:17.825 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/19' 2023-10-10T18:53:17.825 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/15' 2023-10-10T18:53:17.825 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/20' 2023-10-10T18:53:17.825 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/12' 2023-10-10T18:53:17.826 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/10' 2023-10-10T18:53:17.826 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/11' 2023-10-10T18:53:17.826 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/16' 2023-10-10T18:53:17.826 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/14' 2023-10-10T18:53:17.826 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/18' 2023-10-10T18:53:19.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/13' 2023-10-10T18:53:19.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/138/17' 2023-10-10T18:53:19.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/138' 2023-10-10T18:53:19.160 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/19' 2023-10-10T18:53:19.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/15' 2023-10-10T18:53:19.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/20' 2023-10-10T18:53:19.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/12' 2023-10-10T18:53:19.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/10' 2023-10-10T18:53:19.161 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/11' 2023-10-10T18:53:19.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/16' 2023-10-10T18:53:19.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/14' 2023-10-10T18:53:19.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/18' 2023-10-10T18:53:19.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/13' 2023-10-10T18:53:19.162 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/135/17' 2023-10-10T18:53:19.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/135' 2023-10-10T18:53:19.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/19' 2023-10-10T18:53:19.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/15' 2023-10-10T18:53:19.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/20' 2023-10-10T18:53:19.163 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/12' 2023-10-10T18:53:19.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/10' 2023-10-10T18:53:19.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/11' 2023-10-10T18:53:19.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/16' 2023-10-10T18:53:19.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/14' 2023-10-10T18:53:19.164 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/18' 2023-10-10T18:53:19.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/13' 2023-10-10T18:53:19.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/117/17' 2023-10-10T18:53:19.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/117' 2023-10-10T18:53:19.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/19' 2023-10-10T18:53:19.165 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/15' 2023-10-10T18:53:19.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/20' 2023-10-10T18:53:19.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/12' 2023-10-10T18:53:19.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/10' 2023-10-10T18:53:19.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/11' 2023-10-10T18:53:19.166 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/16' 2023-10-10T18:53:19.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/14' 2023-10-10T18:53:19.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/18' 2023-10-10T18:53:19.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/13' 2023-10-10T18:53:19.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/143/17' 2023-10-10T18:53:19.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/143' 2023-10-10T18:53:19.167 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/19' 2023-10-10T18:53:19.168 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/15' 2023-10-10T18:53:19.168 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/20' 2023-10-10T18:53:19.168 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/12' 2023-10-10T18:53:19.168 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/10' 2023-10-10T18:53:19.168 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/11' 2023-10-10T18:53:19.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/16' 2023-10-10T18:53:19.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/14' 2023-10-10T18:53:19.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/18' 2023-10-10T18:53:19.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/13' 2023-10-10T18:53:19.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/180/17' 2023-10-10T18:53:19.169 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/180' 2023-10-10T18:53:19.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/19' 2023-10-10T18:53:19.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/15' 2023-10-10T18:53:19.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/20' 2023-10-10T18:53:19.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/12' 2023-10-10T18:53:19.170 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/10' 2023-10-10T18:53:19.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/11' 2023-10-10T18:53:19.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/16' 2023-10-10T18:53:19.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/14' 2023-10-10T18:53:19.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/18' 2023-10-10T18:53:19.171 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/13' 2023-10-10T18:53:19.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/126/17' 2023-10-10T18:53:19.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/126' 2023-10-10T18:53:19.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/19' 2023-10-10T18:53:19.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/15' 2023-10-10T18:53:19.172 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/20' 2023-10-10T18:53:19.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/12' 2023-10-10T18:53:19.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/10' 2023-10-10T18:53:19.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/11' 2023-10-10T18:53:19.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/16' 2023-10-10T18:53:19.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/14' 2023-10-10T18:53:19.173 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/18' 2023-10-10T18:53:19.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/13' 2023-10-10T18:53:19.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/161/17' 2023-10-10T18:53:19.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/161' 2023-10-10T18:53:19.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/19' 2023-10-10T18:53:19.174 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/15' 2023-10-10T18:53:19.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/20' 2023-10-10T18:53:19.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/12' 2023-10-10T18:53:19.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/10' 2023-10-10T18:53:19.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/11' 2023-10-10T18:53:19.175 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/16' 2023-10-10T18:53:19.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/14' 2023-10-10T18:53:19.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/18' 2023-10-10T18:53:19.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/13' 2023-10-10T18:53:19.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/176/17' 2023-10-10T18:53:19.176 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/176' 2023-10-10T18:53:19.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/19' 2023-10-10T18:53:19.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/15' 2023-10-10T18:53:19.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/20' 2023-10-10T18:53:19.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/12' 2023-10-10T18:53:19.177 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/10' 2023-10-10T18:53:19.178 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/11' 2023-10-10T18:53:19.178 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/16' 2023-10-10T18:53:19.178 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/14' 2023-10-10T18:53:19.178 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/18' 2023-10-10T18:53:19.178 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/13' 2023-10-10T18:53:19.179 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/111/17' 2023-10-10T18:53:19.179 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/111' 2023-10-10T18:53:19.179 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/19' 2023-10-10T18:53:19.179 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/15' 2023-10-10T18:53:19.179 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/20' 2023-10-10T18:53:19.180 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/12' 2023-10-10T18:53:19.180 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/10' 2023-10-10T18:53:19.180 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/11' 2023-10-10T18:53:19.180 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/16' 2023-10-10T18:53:19.180 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/14' 2023-10-10T18:53:19.180 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/18' 2023-10-10T18:53:19.181 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/13' 2023-10-10T18:53:19.388 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:53:19.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:53:19.790 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:19.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.907+0000 7f0aab06d700 1 Processor -- start 2023-10-10T18:53:19.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.907+0000 7f0aab06d700 1 -- start start 2023-10-10T18:53:19.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.908+0000 7f0aab06d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 0x7f0aa40c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:19.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.908+0000 7f0aab06d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0aa40c41e0 0x7f0aa414cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:19.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.908+0000 7f0aab06d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa41533e0 con 0x7f0aa406b640 2023-10-10T18:53:19.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.908+0000 7f0aab06d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa406e130 con 0x7f0aa40c41e0 2023-10-10T18:53:19.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.908+0000 7f0aab06d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa406a970 con 0x7f0aa40c3330 2023-10-10T18:53:19.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.908+0000 7f0a9bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0aa40c41e0 0x7f0aa414cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:19.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.908+0000 7f0a9bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0aa40c41e0 0x7f0aa414cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59888/0 (socket says 172.21.15.175:59888) 2023-10-10T18:53:19.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.908+0000 7f0a9bfff700 1 -- 172.21.15.175:0/3778456317 learned_addr learned my addr 172.21.15.175:0/3778456317 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:19.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0aa8e09700 1 --2- 172.21.15.175:0/3778456317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 0x7f0aa40c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:19.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0a9bfff700 1 -- 172.21.15.175:0/3778456317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 msgr2=0x7f0aa40c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:19.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0a9bfff700 1 --2- 172.21.15.175:0/3778456317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 0x7f0aa40c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:19.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0a9bfff700 1 -- 172.21.15.175:0/3778456317 >> v1:172.21.15.191:6789/0 conn(0x7f0aa406b640 legacy=0x7f0aa4156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:19.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0a9bfff700 1 -- 172.21.15.175:0/3778456317 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa406d520 con 0x7f0aa40c41e0 2023-10-10T18:53:19.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0a9bfff700 1 --2- 172.21.15.175:0/3778456317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa414cda0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a940099b0 tx=0x7f0a94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241f7ab20ce335ce server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:19.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0a9b7fe700 1 -- 172.21.15.175:0/3778456317 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94009ea0 con 0x7f0aa40c41e0 2023-10-10T18:53:19.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0a9b7fe700 1 -- 172.21.15.175:0/3778456317 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9400b860 con 0x7f0aa40c41e0 2023-10-10T18:53:19.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0a9b7fe700 1 -- 172.21.15.175:0/3778456317 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9400bc00 con 0x7f0aa40c41e0 2023-10-10T18:53:19.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0aab06d700 1 -- 172.21.15.175:0/3778456317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 msgr2=0x7f0aa414cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:19.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.909+0000 7f0aab06d700 1 --2- 172.21.15.175:0/3778456317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa414cda0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a940099b0 tx=0x7f0a94009250 comp rx=0 tx=0).stop 2023-10-10T18:53:19.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.910+0000 7f0aab06d700 1 -- 172.21.15.175:0/3778456317 shutdown_connections 2023-10-10T18:53:19.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.910+0000 7f0aab06d700 1 --2- 172.21.15.175:0/3778456317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 0x7f0aa40c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:19.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.910+0000 7f0aab06d700 1 --2- 172.21.15.175:0/3778456317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa414cda0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:19.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.910+0000 7f0aab06d700 1 -- 172.21.15.175:0/3778456317 >> 172.21.15.175:0/3778456317 conn(0x7f0aa4068280 msgr2=0x7f0aa406af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:19.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.910+0000 7f0aab06d700 1 -- 172.21.15.175:0/3778456317 shutdown_connections 2023-10-10T18:53:19.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.910+0000 7f0aab06d700 1 -- 172.21.15.175:0/3778456317 wait complete. 2023-10-10T18:53:19.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.911+0000 7f0aab06d700 1 Processor -- start 2023-10-10T18:53:19.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.911+0000 7f0aab06d700 1 -- start start 2023-10-10T18:53:19.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.911+0000 7f0aab06d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aa406b640 0x7f0aa41550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.911+0000 7f0aab06d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 0x7f0aa4164010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.911+0000 7f0aab06d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa41477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.911+0000 7f0aab06d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa4155b50 con 0x7f0aa40c41e0 2023-10-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.911+0000 7f0aab06d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa4159510 con 0x7f0aa406b640 2023-10-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.911+0000 7f0aab06d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa41503a0 con 0x7f0aa40c3330 2023-10-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa41477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa41477f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59898/0 (socket says 172.21.15.175:59898) 2023-10-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 -- 172.21.15.175:0/1684104610 learned_addr learned my addr 172.21.15.175:0/1684104610 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa8e09700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aa406b640 0x7f0aa41550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:19.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0a9bfff700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 0x7f0aa4164010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:19.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 -- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 msgr2=0x7f0aa4164010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:19.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 0x7f0aa4164010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:19.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 -- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aa406b640 msgr2=0x7f0aa41550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:19.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aa406b640 0x7f0aa41550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:19.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 -- 172.21.15.175:0/1684104610 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a940094f0 con 0x7f0aa40c41e0 2023-10-10T18:53:19.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aa960a700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa41477f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c940 tx=0x7f0a9c0016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:19.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0a98ff9700 1 -- 172.21.15.175:0/1684104610 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c012630 con 0x7f0aa40c41e0 2023-10-10T18:53:19.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0a98ff9700 1 -- 172.21.15.175:0/1684104610 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c0127b0 con 0x7f0aa40c41e0 2023-10-10T18:53:19.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa4154650 con 0x7f0aa40c41e0 2023-10-10T18:53:19.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0a98ff9700 1 -- 172.21.15.175:0/1684104610 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c012b70 con 0x7f0aa40c41e0 2023-10-10T18:53:19.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.912+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa4154c10 con 0x7f0aa40c41e0 2023-10-10T18:53:19.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.913+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa4154da0 con 0x7f0aa40c41e0 2023-10-10T18:53:19.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.914+0000 7f0a98ff9700 1 -- 172.21.15.175:0/1684104610 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0a9c01a3a0 con 0x7f0aa40c41e0 2023-10-10T18:53:19.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.914+0000 7f0a98ff9700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a90066f30 0x7f0a900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:19.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.914+0000 7f0a98ff9700 1 -- 172.21.15.175:0/1684104610 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f0a9c0846c0 con 0x7f0aa40c41e0 2023-10-10T18:53:19.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.914+0000 7f0aa8e09700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a90066f30 0x7f0a900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:19.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.914+0000 7f0aa8e09700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a90066f30 0x7f0a900693e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009da0 tx=0x7f0aa0009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:19.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:19.918+0000 7f0a98ff9700 1 -- 172.21.15.175:0/1684104610 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a9c0517c0 con 0x7f0aa40c41e0 2023-10-10T18:53:20.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.143+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aa4067570 con 0x7f0aa40c41e0 2023-10-10T18:53:20.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.143+0000 7f0a98ff9700 1 -- 172.21.15.175:0/1684104610 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+6214 (secure 0 0 0) 0x7f0a9c054de0 con 0x7f0aa40c41e0 2023-10-10T18:53:20.174 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:20.175 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5367,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:11.582418+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:20.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a90066f30 msgr2=0x7f0a900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a90066f30 0x7f0a900693e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009da0 tx=0x7f0aa0009120 comp rx=0 tx=0).stop 2023-10-10T18:53:20.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 msgr2=0x7f0aa41477f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa41477f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c940 tx=0x7f0a9c0016b0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 shutdown_connections 2023-10-10T18:53:20.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a90066f30 0x7f0a900693e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aa40c41e0 0x7f0aa41477f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aa406b640 0x7f0aa41550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 --2- 172.21.15.175:0/1684104610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aa40c3330 0x7f0aa4164010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 >> 172.21.15.175:0/1684104610 conn(0x7f0aa4068280 msgr2=0x7f0aa406af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:20.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 shutdown_connections 2023-10-10T18:53:20.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.145+0000 7f0aab06d700 1 -- 172.21.15.175:0/1684104610 wait complete. 2023-10-10T18:53:20.179 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 68 2023-10-10T18:53:20.180 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=2) 2023-10-10T18:53:20.180 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-10T18:53:20.181 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T18:53:20.191 INFO:tasks.ceph.mds.f:Stopped 2023-10-10T18:53:20.191 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:20.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.306+0000 7f85093c6700 1 Processor -- start 2023-10-10T18:53:20.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.306+0000 7f85093c6700 1 -- start start 2023-10-10T18:53:20.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.307+0000 7f85093c6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 0x7f8504159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:20.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.307+0000 7f85093c6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8504156800 con 0x7f850414ffc0 2023-10-10T18:53:20.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.307+0000 7f85093c6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8504063e30 con 0x7f85040681e0 2023-10-10T18:53:20.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.307+0000 7f85093c6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85040c59e0 con 0x7f850414fbf0 2023-10-10T18:53:20.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.307+0000 7f84fa7fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f850414ffc0 0x7f8504063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38102/0 (socket says 172.21.15.175:38102) 2023-10-10T18:53:20.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.307+0000 7f84fa7fc700 1 -- 172.21.15.175:0/3009941067 learned_addr learned my addr 172.21.15.175:0/3009941067 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:20.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.307+0000 7f8502ffd700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 0x7f8504159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:20.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387199076 0 0) 0x7f8504156800 con 0x7f850414ffc0 2023-10-10T18:53:20.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f0003570 con 0x7f850414ffc0 2023-10-10T18:53:20.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454733734 0 0) 0x7f8504063e30 con 0x7f85040681e0 2023-10-10T18:53:20.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8504156800 con 0x7f85040681e0 2023-10-10T18:53:20.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400096932 0 0) 0x7f84f0003570 con 0x7f850414ffc0 2023-10-10T18:53:20.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 msgr2=0x7f8504159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:20.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 0x7f8504159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 >> v1:172.21.15.191:6789/0 conn(0x7f85040681e0 legacy=0x7f85040c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8504156c90 con 0x7f850414ffc0 2023-10-10T18:53:20.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f84ec001470 con 0x7f850414ffc0 2023-10-10T18:53:20.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f8502ffd700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 0x7f8504159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:20.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 >> v1:172.21.15.175:6789/0 conn(0x7f850414ffc0 legacy=0x7f8504063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84f0003c20 0x7f84f0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:20.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 0x7f84f0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:20.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0004d50 0x7f84f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:20.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040c59e0 con 0x7f84f0003c20 2023-10-10T18:53:20.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f000bb40 con 0x7f84f0004d50 2023-10-10T18:53:20.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f000bd00 con 0x7f850414fbf0 2023-10-10T18:53:20.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f8502ffd700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84f0003c20 0x7f84f0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:20.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f84fa7fc700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 0x7f84f0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:20.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f85037fe700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0004d50 0x7f84f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:20.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f8502ffd700 1 -- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 msgr2=0x7f84f0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:20.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f8502ffd700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f850414fbf0 0x7f84f0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f8502ffd700 1 -- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0004d50 msgr2=0x7f84f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:20.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f8502ffd700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0004d50 0x7f84f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.308+0000 7f8502ffd700 1 -- 172.21.15.175:0/3009941067 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f40093c0 con 0x7f84f0003c20 2023-10-10T18:53:20.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f8502ffd700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84f0003c20 0x7f84f0004010 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009de0 tx=0x7f84f4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644770578b24eee6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:20.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4009730 con 0x7f84f0003c20 2023-10-10T18:53:20.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f4005a00 con 0x7f84f0003c20 2023-10-10T18:53:20.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85027fc700 1 -- 172.21.15.175:0/3009941067 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4005dc0 con 0x7f84f0003c20 2023-10-10T18:53:20.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85093c6700 1 -- 172.21.15.175:0/3009941067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84f0003c20 msgr2=0x7f84f0004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85093c6700 1 --2- 172.21.15.175:0/3009941067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84f0003c20 0x7f84f0004010 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009de0 tx=0x7f84f4009120 comp rx=0 tx=0).stop 2023-10-10T18:53:20.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85037fe700 1 -- 172.21.15.175:0/3009941067 reap_dead start 2023-10-10T18:53:20.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85093c6700 1 -- 172.21.15.175:0/3009941067 shutdown_connections 2023-10-10T18:53:20.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85093c6700 1 -- 172.21.15.175:0/3009941067 >> 172.21.15.175:0/3009941067 conn(0x7f85040cc760 msgr2=0x7f85040ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:20.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85093c6700 1 -- 172.21.15.175:0/3009941067 shutdown_connections 2023-10-10T18:53:20.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.309+0000 7f85093c6700 1 -- 172.21.15.175:0/3009941067 wait complete. 2023-10-10T18:53:20.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.310+0000 7f85093c6700 1 Processor -- start 2023-10-10T18:53:20.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.310+0000 7f85093c6700 1 -- start start 2023-10-10T18:53:20.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85093c6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0003c20 0x7f85040c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:20.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85093c6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85040c2ee0 0x7f85040c2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:20.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85093c6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8504165db0 0x7f8504166230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:20.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85093c6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040c0c80 con 0x7f8504165db0 2023-10-10T18:53:20.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85093c6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040c1b00 con 0x7f84f0003c20 2023-10-10T18:53:20.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85093c6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85041509c0 con 0x7f85040c2ee0 2023-10-10T18:53:20.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85037fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8504165db0 0x7f8504166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:20.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85037fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8504165db0 0x7f8504166230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59908/0 (socket says 172.21.15.175:59908) 2023-10-10T18:53:20.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85037fe700 1 -- 172.21.15.175:0/445359023 learned_addr learned my addr 172.21.15.175:0/445359023 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:20.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f84fa7fc700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85040c2ee0 0x7f85040c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:20.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f8502ffd700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0003c20 0x7f85040c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:20.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85037fe700 1 -- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85040c2ee0 msgr2=0x7f85040c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85037fe700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85040c2ee0 0x7f85040c2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85037fe700 1 -- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0003c20 msgr2=0x7f85040c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85037fe700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0003c20 0x7f85040c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.311+0000 7f85037fe700 1 -- 172.21.15.175:0/445359023 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8504150ca0 con 0x7f8504165db0 2023-10-10T18:53:20.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.312+0000 7f85037fe700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8504165db0 0x7f8504166230 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f85040642b0 tx=0x7f84fc0017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:20.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.312+0000 7f84fb7fe700 1 -- 172.21.15.175:0/445359023 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc006ba0 con 0x7f8504165db0 2023-10-10T18:53:20.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.312+0000 7f84fb7fe700 1 -- 172.21.15.175:0/445359023 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc006d00 con 0x7f8504165db0 2023-10-10T18:53:20.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.312+0000 7f84fb7fe700 1 -- 172.21.15.175:0/445359023 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc00ca70 con 0x7f8504165db0 2023-10-10T18:53:20.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.312+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8504151390 con 0x7f8504165db0 2023-10-10T18:53:20.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.312+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85040c3420 con 0x7f8504165db0 2023-10-10T18:53:20.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.313+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85040c1c70 con 0x7f8504165db0 2023-10-10T18:53:20.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.313+0000 7f84fb7fe700 1 -- 172.21.15.175:0/445359023 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f84fc014480 con 0x7f8504165db0 2023-10-10T18:53:20.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.313+0000 7f84fb7fe700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f84e4066f60 0x7f84e4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:20.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.314+0000 7f84fb7fe700 1 -- 172.21.15.175:0/445359023 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f84fc083f30 con 0x7f8504165db0 2023-10-10T18:53:20.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.314+0000 7f8502ffd700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f84e4066f60 0x7f84e4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:20.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.314+0000 7f8502ffd700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f84e4066f60 0x7f84e4069410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009390 tx=0x7f84f4019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:20.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.318+0000 7f84fb7fe700 1 -- 172.21.15.175:0/445359023 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84fc04c5a0 con 0x7f8504165db0 2023-10-10T18:53:20.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.550+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8504067570 con 0x7f8504165db0 2023-10-10T18:53:20.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.551+0000 7f84fb7fe700 1 -- 172.21.15.175:0/445359023 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+6214 (secure 0 0 0) 0x7f85040c1c70 con 0x7f8504165db0 2023-10-10T18:53:20.552 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:20.552 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5367,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:11.582418+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:20.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f84e4066f60 msgr2=0x7f84e4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f84e4066f60 0x7f84e4069410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009390 tx=0x7f84f4019000 comp rx=0 tx=0).stop 2023-10-10T18:53:20.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8504165db0 msgr2=0x7f8504166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:20.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8504165db0 0x7f8504166230 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f85040642b0 tx=0x7f84fc0017a0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 shutdown_connections 2023-10-10T18:53:20.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f84e4066f60 0x7f84e4069410 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8504165db0 0x7f8504166230 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84f0003c20 0x7f85040c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 --2- 172.21.15.175:0/445359023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85040c2ee0 0x7f85040c2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:20.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 >> 172.21.15.175:0/445359023 conn(0x7f85040cc760 msgr2=0x7f8504154a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:20.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 shutdown_connections 2023-10-10T18:53:20.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:20.553+0000 7f85093c6700 1 -- 172.21.15.175:0/445359023 wait complete. 2023-10-10T18:53:20.557 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 68 2023-10-10T18:53:20.565 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T18:53:21.237 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/191/17' 2023-10-10T18:53:21.238 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/191' 2023-10-10T18:53:21.238 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/19' 2023-10-10T18:53:21.238 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/15' 2023-10-10T18:53:21.238 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/20' 2023-10-10T18:53:21.239 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/12' 2023-10-10T18:53:21.239 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/10' 2023-10-10T18:53:21.239 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/11' 2023-10-10T18:53:21.239 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/16' 2023-10-10T18:53:21.239 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/14' 2023-10-10T18:53:21.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/18' 2023-10-10T18:53:21.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/13' 2023-10-10T18:53:21.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/184/17' 2023-10-10T18:53:21.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/184' 2023-10-10T18:53:21.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/19' 2023-10-10T18:53:21.240 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/15' 2023-10-10T18:53:21.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/20' 2023-10-10T18:53:21.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/12' 2023-10-10T18:53:21.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/10' 2023-10-10T18:53:21.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/11' 2023-10-10T18:53:21.241 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/16' 2023-10-10T18:53:21.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/14' 2023-10-10T18:53:21.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/18' 2023-10-10T18:53:21.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/13' 2023-10-10T18:53:21.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/192/17' 2023-10-10T18:53:21.242 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/192' 2023-10-10T18:53:21.243 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/19' 2023-10-10T18:53:21.243 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/15' 2023-10-10T18:53:21.243 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/20' 2023-10-10T18:53:21.243 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/12' 2023-10-10T18:53:21.243 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/10' 2023-10-10T18:53:21.244 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/11' 2023-10-10T18:53:21.244 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/16' 2023-10-10T18:53:21.244 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/14' 2023-10-10T18:53:21.244 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/18' 2023-10-10T18:53:21.244 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/13' 2023-10-10T18:53:21.245 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/103/17' 2023-10-10T18:53:21.245 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/103' 2023-10-10T18:53:21.245 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/19' 2023-10-10T18:53:21.245 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/15' 2023-10-10T18:53:21.246 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/20' 2023-10-10T18:53:21.246 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/12' 2023-10-10T18:53:21.246 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/10' 2023-10-10T18:53:21.246 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/11' 2023-10-10T18:53:21.246 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/16' 2023-10-10T18:53:21.247 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/14' 2023-10-10T18:53:21.247 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/18' 2023-10-10T18:53:21.247 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/13' 2023-10-10T18:53:21.247 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/128/17' 2023-10-10T18:53:21.247 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/128' 2023-10-10T18:53:21.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/19' 2023-10-10T18:53:21.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/15' 2023-10-10T18:53:21.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/20' 2023-10-10T18:53:21.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/12' 2023-10-10T18:53:21.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/10' 2023-10-10T18:53:21.248 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/11' 2023-10-10T18:53:21.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/16' 2023-10-10T18:53:21.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/14' 2023-10-10T18:53:21.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/18' 2023-10-10T18:53:21.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/13' 2023-10-10T18:53:21.249 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/144/17' 2023-10-10T18:53:21.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/144' 2023-10-10T18:53:21.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/19' 2023-10-10T18:53:21.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/15' 2023-10-10T18:53:21.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/20' 2023-10-10T18:53:21.250 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/12' 2023-10-10T18:53:21.251 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/10' 2023-10-10T18:53:21.251 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/11' 2023-10-10T18:53:21.251 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/16' 2023-10-10T18:53:21.251 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/14' 2023-10-10T18:53:21.251 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/18' 2023-10-10T18:53:21.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/13' 2023-10-10T18:53:21.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/146/17' 2023-10-10T18:53:21.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/146' 2023-10-10T18:53:21.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/19' 2023-10-10T18:53:21.252 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/15' 2023-10-10T18:53:21.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/20' 2023-10-10T18:53:21.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/12' 2023-10-10T18:53:21.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/10' 2023-10-10T18:53:21.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/11' 2023-10-10T18:53:21.253 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/16' 2023-10-10T18:53:21.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/14' 2023-10-10T18:53:21.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/18' 2023-10-10T18:53:21.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/13' 2023-10-10T18:53:21.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/141/17' 2023-10-10T18:53:21.254 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/141' 2023-10-10T18:53:21.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/19' 2023-10-10T18:53:21.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/15' 2023-10-10T18:53:21.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/20' 2023-10-10T18:53:21.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/12' 2023-10-10T18:53:21.255 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/10' 2023-10-10T18:53:21.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/11' 2023-10-10T18:53:21.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/16' 2023-10-10T18:53:21.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/14' 2023-10-10T18:53:21.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/18' 2023-10-10T18:53:21.256 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/13' 2023-10-10T18:53:21.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/182/17' 2023-10-10T18:53:21.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/182' 2023-10-10T18:53:21.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/19' 2023-10-10T18:53:21.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/15' 2023-10-10T18:53:21.257 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/20' 2023-10-10T18:53:21.258 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/12' 2023-10-10T18:53:21.258 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/10' 2023-10-10T18:53:21.258 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/11' 2023-10-10T18:53:21.258 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/16' 2023-10-10T18:53:21.258 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/14' 2023-10-10T18:53:21.259 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/18' 2023-10-10T18:53:21.259 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/13' 2023-10-10T18:53:21.259 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/107/17' 2023-10-10T18:53:22.271 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/107' 2023-10-10T18:53:22.271 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/19' 2023-10-10T18:53:22.272 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/15' 2023-10-10T18:53:22.272 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/20' 2023-10-10T18:53:22.272 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/12' 2023-10-10T18:53:22.272 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/10' 2023-10-10T18:53:22.273 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/11' 2023-10-10T18:53:22.273 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/16' 2023-10-10T18:53:22.273 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/14' 2023-10-10T18:53:22.273 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/18' 2023-10-10T18:53:22.273 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/13' 2023-10-10T18:53:22.274 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/178/17' 2023-10-10T18:53:22.274 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/178' 2023-10-10T18:53:22.274 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/19' 2023-10-10T18:53:22.274 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/15' 2023-10-10T18:53:22.274 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/20' 2023-10-10T18:53:22.275 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/12' 2023-10-10T18:53:22.275 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/10' 2023-10-10T18:53:22.275 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/11' 2023-10-10T18:53:22.275 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/16' 2023-10-10T18:53:22.275 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/14' 2023-10-10T18:53:22.276 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/18' 2023-10-10T18:53:22.276 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/13' 2023-10-10T18:53:22.276 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/156/17' 2023-10-10T18:53:22.276 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/156' 2023-10-10T18:53:22.276 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/19' 2023-10-10T18:53:22.277 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/15' 2023-10-10T18:53:22.277 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/20' 2023-10-10T18:53:22.277 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/12' 2023-10-10T18:53:22.277 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/10' 2023-10-10T18:53:22.277 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/11' 2023-10-10T18:53:22.278 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/16' 2023-10-10T18:53:22.278 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/14' 2023-10-10T18:53:22.278 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/18' 2023-10-10T18:53:22.278 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/13' 2023-10-10T18:53:22.278 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/131/17' 2023-10-10T18:53:22.279 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/131' 2023-10-10T18:53:22.279 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/19' 2023-10-10T18:53:22.279 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/15' 2023-10-10T18:53:22.279 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/20' 2023-10-10T18:53:22.279 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/12' 2023-10-10T18:53:22.280 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/10' 2023-10-10T18:53:22.280 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/11' 2023-10-10T18:53:22.280 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/16' 2023-10-10T18:53:22.280 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/14' 2023-10-10T18:53:22.280 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/18' 2023-10-10T18:53:22.280 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/13' 2023-10-10T18:53:22.281 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/123/17' 2023-10-10T18:53:22.281 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/123' 2023-10-10T18:53:22.281 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/19' 2023-10-10T18:53:22.281 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/15' 2023-10-10T18:53:22.281 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/20' 2023-10-10T18:53:22.282 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/12' 2023-10-10T18:53:22.282 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/10' 2023-10-10T18:53:22.282 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/11' 2023-10-10T18:53:22.282 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/16' 2023-10-10T18:53:22.282 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/14' 2023-10-10T18:53:22.283 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/18' 2023-10-10T18:53:22.283 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/13' 2023-10-10T18:53:22.283 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/160/17' 2023-10-10T18:53:22.283 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/160' 2023-10-10T18:53:22.283 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/19' 2023-10-10T18:53:22.284 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/15' 2023-10-10T18:53:22.284 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/20' 2023-10-10T18:53:22.284 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/12' 2023-10-10T18:53:22.284 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/10' 2023-10-10T18:53:22.284 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/11' 2023-10-10T18:53:22.285 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/16' 2023-10-10T18:53:22.285 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/14' 2023-10-10T18:53:22.285 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/18' 2023-10-10T18:53:22.285 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/13' 2023-10-10T18:53:22.286 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/165/17' 2023-10-10T18:53:22.286 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/165' 2023-10-10T18:53:22.286 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/19' 2023-10-10T18:53:22.286 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/15' 2023-10-10T18:53:22.286 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/20' 2023-10-10T18:53:22.287 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/12' 2023-10-10T18:53:22.287 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/10' 2023-10-10T18:53:22.287 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/11' 2023-10-10T18:53:22.287 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/16' 2023-10-10T18:53:22.287 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/14' 2023-10-10T18:53:22.288 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/18' 2023-10-10T18:53:22.288 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/13' 2023-10-10T18:53:22.288 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/101/17' 2023-10-10T18:53:22.288 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/101' 2023-10-10T18:53:22.288 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/19' 2023-10-10T18:53:22.289 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/15' 2023-10-10T18:53:22.289 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/20' 2023-10-10T18:53:22.289 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/12' 2023-10-10T18:53:22.289 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/10' 2023-10-10T18:53:22.289 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/11' 2023-10-10T18:53:22.290 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/16' 2023-10-10T18:53:22.290 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/14' 2023-10-10T18:53:22.290 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/18' 2023-10-10T18:53:22.290 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/13' 2023-10-10T18:53:22.290 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/148/17' 2023-10-10T18:53:22.291 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/148' 2023-10-10T18:53:22.291 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/19' 2023-10-10T18:53:22.291 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/15' 2023-10-10T18:53:22.291 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/20' 2023-10-10T18:53:22.291 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/12' 2023-10-10T18:53:22.292 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/10' 2023-10-10T18:53:22.292 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/11' 2023-10-10T18:53:22.292 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/16' 2023-10-10T18:53:22.292 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/14' 2023-10-10T18:53:22.292 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/18' 2023-10-10T18:53:22.293 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/13' 2023-10-10T18:53:22.293 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/168/17' 2023-10-10T18:53:22.293 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/168' 2023-10-10T18:53:22.566 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:22.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.699+0000 7f1553952700 1 Processor -- start 2023-10-10T18:53:22.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.699+0000 7f1553952700 1 -- start start 2023-10-10T18:53:22.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f1553952700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f154c1500b0 0x7f154c0c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:22.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f1553952700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 0x7f154c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:22.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f1553952700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f154c0c4c20 0x7f154c06e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:22.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f1553952700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c0677f0 con 0x7f154c1500b0 2023-10-10T18:53:22.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f1553952700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c159590 con 0x7f154c0c4c20 2023-10-10T18:53:22.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f1553952700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0c4250 con 0x7f154c06d6e0 2023-10-10T18:53:22.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f15516ee700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f154c1500b0 0x7f154c0c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:22.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f15516ee700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f154c1500b0 0x7f154c0c46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59912/0 (socket says 172.21.15.175:59912) 2023-10-10T18:53:22.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f15516ee700 1 -- 172.21.15.175:0/1150479346 learned_addr learned my addr 172.21.15.175:0/1150479346 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:22.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f1551eef700 1 --2- 172.21.15.175:0/1150479346 >> v2:172.21.15.191:3300/0 conn(0x7f154c0c4c20 0x7f154c06e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:22.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.700+0000 7f1550eed700 1 --2- 172.21.15.175:0/1150479346 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 0x7f154c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:22.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f15516ee700 1 -- 172.21.15.175:0/1150479346 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 msgr2=0x7f154c06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:22.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f15516ee700 1 --2- 172.21.15.175:0/1150479346 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 0x7f154c06dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f15516ee700 1 -- 172.21.15.175:0/1150479346 >> v2:172.21.15.191:3300/0 conn(0x7f154c0c4c20 msgr2=0x7f154c06e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:22.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f15516ee700 1 --2- 172.21.15.175:0/1150479346 >> v2:172.21.15.191:3300/0 conn(0x7f154c0c4c20 0x7f154c06e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f15516ee700 1 -- 172.21.15.175:0/1150479346 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c063760 con 0x7f154c1500b0 2023-10-10T18:53:22.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f15516ee700 1 --2- 172.21.15.175:0/1150479346 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c1500b0 0x7f154c0c46e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f154c0c6a70 tx=0x7f1548009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a638c8039ac65334 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:22.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f1543fff700 1 -- 172.21.15.175:0/1150479346 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154800b6d0 con 0x7f154c1500b0 2023-10-10T18:53:22.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f1543fff700 1 -- 172.21.15.175:0/1150479346 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154800b830 con 0x7f154c1500b0 2023-10-10T18:53:22.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f1543fff700 1 -- 172.21.15.175:0/1150479346 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154800bbf0 con 0x7f154c1500b0 2023-10-10T18:53:22.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f1553952700 1 -- 172.21.15.175:0/1150479346 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c1500b0 msgr2=0x7f154c0c46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:22.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.701+0000 7f1553952700 1 --2- 172.21.15.175:0/1150479346 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c1500b0 0x7f154c0c46e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f154c0c6a70 tx=0x7f1548009250 comp rx=0 tx=0).stop 2023-10-10T18:53:22.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.702+0000 7f1553952700 1 -- 172.21.15.175:0/1150479346 shutdown_connections 2023-10-10T18:53:22.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.702+0000 7f1553952700 1 --2- 172.21.15.175:0/1150479346 >> v2:172.21.15.191:3300/0 conn(0x7f154c0c4c20 0x7f154c06e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.702+0000 7f1553952700 1 --2- 172.21.15.175:0/1150479346 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c1500b0 0x7f154c0c46e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.702+0000 7f1553952700 1 --2- 172.21.15.175:0/1150479346 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 0x7f154c06dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.702+0000 7f1553952700 1 -- 172.21.15.175:0/1150479346 >> 172.21.15.175:0/1150479346 conn(0x7f154c0c5650 msgr2=0x7f154c0c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:22.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.702+0000 7f1553952700 1 -- 172.21.15.175:0/1150479346 shutdown_connections 2023-10-10T18:53:22.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.702+0000 7f1553952700 1 -- 172.21.15.175:0/1150479346 wait complete. 2023-10-10T18:53:22.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.703+0000 7f1553952700 1 Processor -- start 2023-10-10T18:53:22.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.703+0000 7f1553952700 1 -- start start 2023-10-10T18:53:22.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.703+0000 7f1553952700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 0x7f154c0bbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:22.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1553952700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c0c4c20 0x7f154c0b9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:22.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1553952700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f154c1500b0 0x7f154c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:22.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1553952700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0b9a60 con 0x7f154c0c4c20 2023-10-10T18:53:22.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1553952700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c1569f0 con 0x7f154c1500b0 2023-10-10T18:53:22.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1553952700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c154000 con 0x7f154c06d6e0 2023-10-10T18:53:22.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c0c4c20 0x7f154c0b9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:22.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c0c4c20 0x7f154c0b9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59916/0 (socket says 172.21.15.175:59916) 2023-10-10T18:53:22.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 -- 172.21.15.175:0/2023370031 learned_addr learned my addr 172.21.15.175:0/2023370031 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:22.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1551eef700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f154c1500b0 0x7f154c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:22.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f15516ee700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 0x7f154c0bbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:22.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 -- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 msgr2=0x7f154c0bbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:22.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 0x7f154c0bbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 -- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f154c1500b0 msgr2=0x7f154c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:22.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f154c1500b0 0x7f154c15d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 -- 172.21.15.175:0/2023370031 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480094f0 con 0x7f154c0c4c20 2023-10-10T18:53:22.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1550eed700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c0c4c20 0x7f154c0b9520 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f153c00e930 tx=0x7f153c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:22.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f15417fa700 1 -- 172.21.15.175:0/2023370031 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c00eed0 con 0x7f154c0c4c20 2023-10-10T18:53:22.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f15417fa700 1 -- 172.21.15.175:0/2023370031 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c00c8e0 con 0x7f154c0c4c20 2023-10-10T18:53:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f15417fa700 1 -- 172.21.15.175:0/2023370031 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c00cc80 con 0x7f154c0c4c20 2023-10-10T18:53:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c0bc260 con 0x7f154c0c4c20 2023-10-10T18:53:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.704+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c156f90 con 0x7f154c0c4c20 2023-10-10T18:53:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.705+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c154220 con 0x7f154c0c4c20 2023-10-10T18:53:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.706+0000 7f15417fa700 1 -- 172.21.15.175:0/2023370031 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f153c013680 con 0x7f154c0c4c20 2023-10-10T18:53:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.706+0000 7f15417fa700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1530066f30 0x7f15300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:22.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.706+0000 7f15417fa700 1 -- 172.21.15.175:0/2023370031 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f153c0848d0 con 0x7f154c0c4c20 2023-10-10T18:53:22.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.706+0000 7f15516ee700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1530066f30 0x7f15300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:22.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.707+0000 7f15516ee700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1530066f30 0x7f15300693e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f154c0c6a70 tx=0x7f1548017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:22.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.710+0000 7f15417fa700 1 -- 172.21.15.175:0/2023370031 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f154c154220 con 0x7f154c0c4c20 2023-10-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.937+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f154c069100 con 0x7f154c0c4c20 2023-10-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.937+0000 7f15417fa700 1 -- 172.21.15.175:0/2023370031 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+6214 (secure 0 0 0) 0x7f153c0519d0 con 0x7f154c0c4c20 2023-10-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5367,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:11.582418+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1530066f30 msgr2=0x7f15300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1530066f30 0x7f15300693e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f154c0c6a70 tx=0x7f1548017000 comp rx=0 tx=0).stop 2023-10-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c0c4c20 msgr2=0x7f154c0b9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c0c4c20 0x7f154c0b9520 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f153c00e930 tx=0x7f153c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 shutdown_connections 2023-10-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1530066f30 0x7f15300693e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f154c1500b0 0x7f154c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f154c06d6e0 0x7f154c0bbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 --2- 172.21.15.175:0/2023370031 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f154c0c4c20 0x7f154c0b9520 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 >> 172.21.15.175:0/2023370031 conn(0x7f154c0c5650 msgr2=0x7f154c0c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 shutdown_connections 2023-10-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:22.939+0000 7f1553952700 1 -- 172.21.15.175:0/2023370031 wait complete. 2023-10-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 68 2023-10-10T18:53:22.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T18:53:24.952 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:25.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.079+0000 7ff5bf8e2700 1 Processor -- start 2023-10-10T18:53:25.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.079+0000 7ff5bf8e2700 1 -- start start 2023-10-10T18:53:25.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bf8e2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b2390 0x7ff5b8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:25.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bf8e2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff5b80aff60 0x7ff5b80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:25.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bf8e2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b81574b0 con 0x7ff5b80b1fc0 2023-10-10T18:53:25.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bf8e2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b8150770 con 0x7ff5b80aff60 2023-10-10T18:53:25.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bf8e2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b80afad0 con 0x7ff5b80b2390 2023-10-10T18:53:25.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bde7f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff5b80aff60 0x7ff5b80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:25.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bd67e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff5b80b1fc0 0x7ff5b8068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46154/0 (socket says 172.21.15.175:46154) 2023-10-10T18:53:25.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bd67e700 1 -- 172.21.15.175:0/1242950644 learned_addr learned my addr 172.21.15.175:0/1242950644 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:25.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.080+0000 7ff5bce7d700 1 --2- 172.21.15.175:0/1242950644 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b2390 0x7ff5b8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:25.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bde7f700 1 -- 172.21.15.175:0/1242950644 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b2390 msgr2=0x7ff5b8155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:25.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bde7f700 1 --2- 172.21.15.175:0/1242950644 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b2390 0x7ff5b8155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bde7f700 1 -- 172.21.15.175:0/1242950644 >> v1:172.21.15.175:6789/0 conn(0x7ff5b80b1fc0 legacy=0x7ff5b8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:25.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bde7f700 1 -- 172.21.15.175:0/1242950644 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b80af4c0 con 0x7ff5b80aff60 2023-10-10T18:53:25.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5affff700 1 -- 172.21.15.175:0/1242950644 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341389752 0 0) 0x7ff5b81574b0 con 0x7ff5b80b1fc0 2023-10-10T18:53:25.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bde7f700 1 --2- 172.21.15.175:0/1242950644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80aff60 0x7ff5b80b03e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009910 tx=0x7ff5b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a68f0b055a94a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:25.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5affff700 1 -- 172.21.15.175:0/1242950644 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b000b8b0 con 0x7ff5b80aff60 2023-10-10T18:53:25.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5affff700 1 -- 172.21.15.175:0/1242950644 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b000ba10 con 0x7ff5b80aff60 2023-10-10T18:53:25.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5affff700 1 -- 172.21.15.175:0/1242950644 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b000bdb0 con 0x7ff5b80aff60 2023-10-10T18:53:25.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/1242950644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80aff60 msgr2=0x7ff5b80b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:25.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/1242950644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80aff60 0x7ff5b80b03e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009910 tx=0x7ff5b00091a0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/1242950644 shutdown_connections 2023-10-10T18:53:25.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/1242950644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80aff60 0x7ff5b80b03e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/1242950644 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b2390 0x7ff5b8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.081+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/1242950644 >> 172.21.15.175:0/1242950644 conn(0x7ff5b8063230 msgr2=0x7ff5b8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:25.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.082+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/1242950644 shutdown_connections 2023-10-10T18:53:25.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.082+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/1242950644 wait complete. 2023-10-10T18:53:25.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.083+0000 7ff5bf8e2700 1 Processor -- start 2023-10-10T18:53:25.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.083+0000 7ff5bf8e2700 1 -- start start 2023-10-10T18:53:25.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.083+0000 7ff5bf8e2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5b80aff60 0x7ff5b80cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:25.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.083+0000 7ff5bf8e2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b1fc0 0x7ff5b80c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:25.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.083+0000 7ff5bf8e2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80b2390 0x7ff5b8159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:25.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.083+0000 7ff5bf8e2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b816a180 con 0x7ff5b80b2390 2023-10-10T18:53:25.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.083+0000 7ff5bf8e2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b8157e90 con 0x7ff5b80aff60 2023-10-10T18:53:25.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.083+0000 7ff5bf8e2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b81692b0 con 0x7ff5b80b1fc0 2023-10-10T18:53:25.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80b2390 0x7ff5b8159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:25.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80b2390 0x7ff5b8159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49734/0 (socket says 172.21.15.175:49734) 2023-10-10T18:53:25.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 -- 172.21.15.175:0/164365653 learned_addr learned my addr 172.21.15.175:0/164365653 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:25.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bce7d700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b1fc0 0x7ff5b80c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:25.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bd67e700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5b80aff60 0x7ff5b80cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:25.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 -- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b1fc0 msgr2=0x7ff5b80c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:25.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b1fc0 0x7ff5b80c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 -- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5b80aff60 msgr2=0x7ff5b80cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:25.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5b80aff60 0x7ff5b80cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 -- 172.21.15.175:0/164365653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b0009420 con 0x7ff5b80b2390 2023-10-10T18:53:25.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bce7d700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b1fc0 0x7ff5b80c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:53:25.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bd67e700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5b80aff60 0x7ff5b80cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:53:25.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bde7f700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80b2390 0x7ff5b8159d00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00098e0 tx=0x7ff5b0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:25.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5ad7fa700 1 -- 172.21.15.175:0/164365653 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b001a630 con 0x7ff5b80b2390 2023-10-10T18:53:25.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5ad7fa700 1 -- 172.21.15.175:0/164365653 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b001a7b0 con 0x7ff5b80b2390 2023-10-10T18:53:25.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b8169570 con 0x7ff5b80b2390 2023-10-10T18:53:25.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5ad7fa700 1 -- 172.21.15.175:0/164365653 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b001ab70 con 0x7ff5b80b2390 2023-10-10T18:53:25.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.084+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b8169b30 con 0x7ff5b80b2390 2023-10-10T18:53:25.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.085+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b80c53d0 con 0x7ff5b80b2390 2023-10-10T18:53:25.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.086+0000 7ff5ad7fa700 1 -- 172.21.15.175:0/164365653 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff5b00233e0 con 0x7ff5b80b2390 2023-10-10T18:53:25.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.086+0000 7ff5ad7fa700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff59c066f60 0x7ff59c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:25.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.086+0000 7ff5ad7fa700 1 -- 172.21.15.175:0/164365653 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7ff5b0085ea0 con 0x7ff5b80b2390 2023-10-10T18:53:25.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.086+0000 7ff5bd67e700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff59c066f60 0x7ff59c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:25.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.087+0000 7ff5bd67e700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff59c066f60 0x7ff59c069410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4001560 tx=0x7ff5b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:25.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.090+0000 7ff5ad7fa700 1 -- 172.21.15.175:0/164365653 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5b0052fa0 con 0x7ff5b80b2390 2023-10-10T18:53:25.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.313+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5b8067570 con 0x7ff5b80b2390 2023-10-10T18:53:25.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.314+0000 7ff5ad7fa700 1 -- 172.21.15.175:0/164365653 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+6214 (secure 0 0 0) 0x7ff5b0015090 con 0x7ff5b80b2390 2023-10-10T18:53:25.315 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:25.315 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5367,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:11.582418+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:25.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff59c066f60 msgr2=0x7ff59c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:25.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff59c066f60 0x7ff59c069410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4001560 tx=0x7ff5b4009040 comp rx=0 tx=0).stop 2023-10-10T18:53:25.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80b2390 msgr2=0x7ff5b8159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:25.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80b2390 0x7ff5b8159d00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00098e0 tx=0x7ff5b0017000 comp rx=0 tx=0).stop 2023-10-10T18:53:25.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 shutdown_connections 2023-10-10T18:53:25.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff59c066f60 0x7ff59c069410 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff5b80b2390 0x7ff5b8159d00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5b80aff60 0x7ff5b80cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 --2- 172.21.15.175:0/164365653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5b80b1fc0 0x7ff5b80c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:25.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 >> 172.21.15.175:0/164365653 conn(0x7ff5b8063230 msgr2=0x7ff5b815ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:25.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 shutdown_connections 2023-10-10T18:53:25.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:25.316+0000 7ff5bf8e2700 1 -- 172.21.15.175:0/164365653 wait complete. 2023-10-10T18:53:25.319 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 68 2023-10-10T18:53:25.328 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T18:53:27.328 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:27.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.457+0000 7f2833302700 1 Processor -- start 2023-10-10T18:53:27.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.457+0000 7f2833302700 1 -- start start 2023-10-10T18:53:27.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f2833302700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f282c0b2390 0x7f282c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:27.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f2833302700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f282c0aff60 0x7f282c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:27.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f2833302700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c1574b0 con 0x7f282c0b1fc0 2023-10-10T18:53:27.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f2833302700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f282c150770 con 0x7f282c0b2390 2023-10-10T18:53:27.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f2833302700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f282c0afad0 con 0x7f282c0aff60 2023-10-10T18:53:27.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283089d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f282c0b2390 0x7f282c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:27.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283189f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f282c0aff60 0x7f282c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:27.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283109e700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f282c0b1fc0 0x7f282c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:50888/0 (socket says 172.21.15.175:50888) 2023-10-10T18:53:27.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283089d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f282c0b2390 0x7f282c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49736/0 (socket says 172.21.15.175:49736) 2023-10-10T18:53:27.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283109e700 1 -- 172.21.15.175:0/595959314 learned_addr learned my addr 172.21.15.175:0/595959314 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:27.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283089d700 1 -- 172.21.15.175:0/595959314 >> v2:172.21.15.191:3300/0 conn(0x7f282c0aff60 msgr2=0x7f282c0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:27.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283089d700 1 --2- 172.21.15.175:0/595959314 >> v2:172.21.15.191:3300/0 conn(0x7f282c0aff60 0x7f282c0b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283089d700 1 -- 172.21.15.175:0/595959314 >> v1:172.21.15.191:6789/0 conn(0x7f282c0b1fc0 legacy=0x7f282c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:27.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.458+0000 7f283089d700 1 -- 172.21.15.175:0/595959314 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c0af4c0 con 0x7f282c0b2390 2023-10-10T18:53:27.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f283089d700 1 --2- 172.21.15.175:0/595959314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155fb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f282c0641f0 tx=0x7f28200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc557dfab030a9ee server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:27.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f281ffff700 1 -- 172.21.15.175:0/595959314 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282000b690 con 0x7f282c0b2390 2023-10-10T18:53:27.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f281ffff700 1 -- 172.21.15.175:0/595959314 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282000b7f0 con 0x7f282c0b2390 2023-10-10T18:53:27.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f281ffff700 1 -- 172.21.15.175:0/595959314 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282000bb90 con 0x7f282c0b2390 2023-10-10T18:53:27.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f2833302700 1 -- 172.21.15.175:0/595959314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 msgr2=0x7f282c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:27.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f2833302700 1 --2- 172.21.15.175:0/595959314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155fb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f282c0641f0 tx=0x7f28200091a0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f2833302700 1 -- 172.21.15.175:0/595959314 shutdown_connections 2023-10-10T18:53:27.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f2833302700 1 --2- 172.21.15.175:0/595959314 >> v2:172.21.15.191:3300/0 conn(0x7f282c0aff60 0x7f282c0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f2833302700 1 --2- 172.21.15.175:0/595959314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155fb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.459+0000 7f2833302700 1 -- 172.21.15.175:0/595959314 >> 172.21.15.175:0/595959314 conn(0x7f282c063230 msgr2=0x7f282c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:27.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.460+0000 7f2833302700 1 -- 172.21.15.175:0/595959314 shutdown_connections 2023-10-10T18:53:27.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.460+0000 7f2833302700 1 -- 172.21.15.175:0/595959314 wait complete. 2023-10-10T18:53:27.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.461+0000 7f2833302700 1 Processor -- start 2023-10-10T18:53:27.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.461+0000 7f2833302700 1 -- start start 2023-10-10T18:53:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.461+0000 7f2833302700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f282c0aff60 0x7f282c166930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.461+0000 7f2833302700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f282c0b1fc0 0x7f282c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.461+0000 7f2833302700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f2833302700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0c3ca0 con 0x7f282c0b2390 2023-10-10T18:53:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f2833302700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0cc680 con 0x7f282c0aff60 2023-10-10T18:53:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f2833302700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282c166e70 con 0x7f282c0b1fc0 2023-10-10T18:53:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283109e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f282c0aff60 0x7f282c166930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49740/0 (socket says 172.21.15.175:49740) 2023-10-10T18:53:27.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 -- 172.21.15.175:0/2535885545 learned_addr learned my addr 172.21.15.175:0/2535885545 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283089d700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f282c0b1fc0 0x7f282c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 -- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f282c0b1fc0 msgr2=0x7f282c15b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f282c0b1fc0 0x7f282c15b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 -- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f282c0aff60 msgr2=0x7f282c166930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f282c0aff60 0x7f282c166930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 -- 172.21.15.175:0/2535885545 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2820009420 con 0x7f282c0b2390 2023-10-10T18:53:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283109e700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f282c0aff60 0x7f282c166930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f283189f700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2824009ec0 tx=0x7f282400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f281d7fa700 1 -- 172.21.15.175:0/2535885545 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282400cc00 con 0x7f282c0b2390 2023-10-10T18:53:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f281d7fa700 1 -- 172.21.15.175:0/2535885545 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282400f040 con 0x7f282c0b2390 2023-10-10T18:53:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f281d7fa700 1 -- 172.21.15.175:0/2535885545 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824014480 con 0x7f282c0b2390 2023-10-10T18:53:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282c0c3330 con 0x7f282c0b2390 2023-10-10T18:53:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.462+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282c0cc1f0 con 0x7f282c0b2390 2023-10-10T18:53:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.463+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282c0cc380 con 0x7f282c0b2390 2023-10-10T18:53:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.464+0000 7f281d7fa700 1 -- 172.21.15.175:0/2535885545 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f28240147f0 con 0x7f282c0b2390 2023-10-10T18:53:27.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.464+0000 7f281d7fa700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2810066f30 0x7f28100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:27.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.464+0000 7f281d7fa700 1 -- 172.21.15.175:0/2535885545 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f2824085330 con 0x7f282c0b2390 2023-10-10T18:53:27.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.464+0000 7f283109e700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2810066f30 0x7f28100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:27.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.465+0000 7f283109e700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2810066f30 0x7f28100693e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2828000c00 tx=0x7f2828009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:27.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.468+0000 7f281d7fa700 1 -- 172.21.15.175:0/2535885545 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2824052430 con 0x7f282c0b2390 2023-10-10T18:53:27.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.694+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f282c067570 con 0x7f282c0b2390 2023-10-10T18:53:27.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.694+0000 7f281d7fa700 1 -- 172.21.15.175:0/2535885545 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6213 (secure 0 0 0) 0x7f2824055a50 con 0x7f282c0b2390 2023-10-10T18:53:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:27.695 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5367,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:27.623204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:27.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.696+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2810066f30 msgr2=0x7f28100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:27.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.696+0000 7f2833302700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2810066f30 0x7f28100693e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2828000c00 tx=0x7f2828009040 comp rx=0 tx=0).stop 2023-10-10T18:53:27.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.696+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 msgr2=0x7f282c155880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:27.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.696+0000 7f2833302700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2824009ec0 tx=0x7f282400b040 comp rx=0 tx=0).stop 2023-10-10T18:53:27.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.697+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 shutdown_connections 2023-10-10T18:53:27.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.697+0000 7f2833302700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2810066f30 0x7f28100693e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.697+0000 7f2833302700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f282c0b2390 0x7f282c155880 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.697+0000 7f2833302700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f282c0aff60 0x7f282c166930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.697+0000 7f2833302700 1 --2- 172.21.15.175:0/2535885545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f282c0b1fc0 0x7f282c15b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:27.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.697+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 >> 172.21.15.175:0/2535885545 conn(0x7f282c063230 msgr2=0x7f282c14fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:27.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.697+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 shutdown_connections 2023-10-10T18:53:27.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:27.697+0000 7f2833302700 1 -- 172.21.15.175:0/2535885545 wait complete. 2023-10-10T18:53:27.699 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 69 2023-10-10T18:53:27.708 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T18:53:28.519 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/19' 2023-10-10T18:53:28.520 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/15' 2023-10-10T18:53:28.520 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/20' 2023-10-10T18:53:28.520 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/12' 2023-10-10T18:53:28.520 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/10' 2023-10-10T18:53:28.521 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/11' 2023-10-10T18:53:28.521 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/16' 2023-10-10T18:53:28.521 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/14' 2023-10-10T18:53:28.521 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/18' 2023-10-10T18:53:28.521 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/13' 2023-10-10T18:53:28.522 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/179/17' 2023-10-10T18:53:28.522 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/179' 2023-10-10T18:53:28.522 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/19' 2023-10-10T18:53:28.522 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/15' 2023-10-10T18:53:28.523 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/20' 2023-10-10T18:53:28.523 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/12' 2023-10-10T18:53:28.523 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/10' 2023-10-10T18:53:28.523 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/11' 2023-10-10T18:53:28.523 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/16' 2023-10-10T18:53:28.523 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/14' 2023-10-10T18:53:28.524 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/18' 2023-10-10T18:53:28.524 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/13' 2023-10-10T18:53:28.524 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/132/17' 2023-10-10T18:53:28.524 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/132' 2023-10-10T18:53:28.524 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/19' 2023-10-10T18:53:28.525 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/15' 2023-10-10T18:53:28.525 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/20' 2023-10-10T18:53:28.525 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/12' 2023-10-10T18:53:28.525 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/10' 2023-10-10T18:53:28.525 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/11' 2023-10-10T18:53:28.526 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/16' 2023-10-10T18:53:28.526 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/14' 2023-10-10T18:53:28.526 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/18' 2023-10-10T18:53:28.526 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/13' 2023-10-10T18:53:28.526 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/199/17' 2023-10-10T18:53:28.527 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/199' 2023-10-10T18:53:28.527 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/19' 2023-10-10T18:53:28.527 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/15' 2023-10-10T18:53:28.527 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/20' 2023-10-10T18:53:28.527 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/12' 2023-10-10T18:53:28.528 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/10' 2023-10-10T18:53:28.528 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/11' 2023-10-10T18:53:28.528 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/16' 2023-10-10T18:53:28.528 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/14' 2023-10-10T18:53:28.528 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/18' 2023-10-10T18:53:28.529 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/13' 2023-10-10T18:53:28.529 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/163/17' 2023-10-10T18:53:28.529 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/163' 2023-10-10T18:53:28.529 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/19' 2023-10-10T18:53:28.529 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/15' 2023-10-10T18:53:28.530 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/20' 2023-10-10T18:53:28.530 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/12' 2023-10-10T18:53:28.530 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/10' 2023-10-10T18:53:28.530 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/11' 2023-10-10T18:53:28.530 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/16' 2023-10-10T18:53:28.531 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/14' 2023-10-10T18:53:28.531 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/18' 2023-10-10T18:53:28.531 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/13' 2023-10-10T18:53:28.531 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/153/17' 2023-10-10T18:53:28.531 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/153' 2023-10-10T18:53:28.532 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/19' 2023-10-10T18:53:28.532 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/15' 2023-10-10T18:53:28.532 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/20' 2023-10-10T18:53:28.532 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/12' 2023-10-10T18:53:28.532 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/10' 2023-10-10T18:53:28.532 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/11' 2023-10-10T18:53:28.533 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/16' 2023-10-10T18:53:28.533 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/14' 2023-10-10T18:53:28.533 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/18' 2023-10-10T18:53:28.533 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/13' 2023-10-10T18:53:28.533 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/122/17' 2023-10-10T18:53:28.534 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/122' 2023-10-10T18:53:28.534 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/19' 2023-10-10T18:53:28.534 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/15' 2023-10-10T18:53:28.534 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/20' 2023-10-10T18:53:28.534 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/12' 2023-10-10T18:53:28.535 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/10' 2023-10-10T18:53:28.535 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/11' 2023-10-10T18:53:28.535 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/16' 2023-10-10T18:53:28.535 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/14' 2023-10-10T18:53:28.535 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/18' 2023-10-10T18:53:28.536 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/13' 2023-10-10T18:53:28.536 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/162/17' 2023-10-10T18:53:28.536 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/162' 2023-10-10T18:53:28.536 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/19' 2023-10-10T18:53:28.536 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/15' 2023-10-10T18:53:28.536 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/20' 2023-10-10T18:53:28.537 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/12' 2023-10-10T18:53:28.537 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/10' 2023-10-10T18:53:28.537 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/11' 2023-10-10T18:53:28.537 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/16' 2023-10-10T18:53:28.537 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/14' 2023-10-10T18:53:28.538 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/18' 2023-10-10T18:53:28.538 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/13' 2023-10-10T18:53:28.538 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/197/17' 2023-10-10T18:53:28.538 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/197' 2023-10-10T18:53:28.538 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/19' 2023-10-10T18:53:28.539 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/15' 2023-10-10T18:53:28.539 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/20' 2023-10-10T18:53:28.539 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/12' 2023-10-10T18:53:28.539 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/10' 2023-10-10T18:53:28.539 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/11' 2023-10-10T18:53:28.540 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/16' 2023-10-10T18:53:28.540 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/14' 2023-10-10T18:53:28.540 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/18' 2023-10-10T18:53:28.540 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/13' 2023-10-10T18:53:28.540 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/113/17' 2023-10-10T18:53:28.541 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/113' 2023-10-10T18:53:28.541 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/19' 2023-10-10T18:53:29.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/15' 2023-10-10T18:53:29.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/20' 2023-10-10T18:53:29.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/12' 2023-10-10T18:53:29.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/10' 2023-10-10T18:53:29.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/11' 2023-10-10T18:53:29.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/16' 2023-10-10T18:53:29.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/14' 2023-10-10T18:53:29.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/18' 2023-10-10T18:53:29.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/13' 2023-10-10T18:53:29.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/152/17' 2023-10-10T18:53:29.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/152' 2023-10-10T18:53:29.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/19' 2023-10-10T18:53:29.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/15' 2023-10-10T18:53:29.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/20' 2023-10-10T18:53:29.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/12' 2023-10-10T18:53:29.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/10' 2023-10-10T18:53:29.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/11' 2023-10-10T18:53:29.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/16' 2023-10-10T18:53:29.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/14' 2023-10-10T18:53:29.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/18' 2023-10-10T18:53:29.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/13' 2023-10-10T18:53:29.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/121/17' 2023-10-10T18:53:29.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/121' 2023-10-10T18:53:29.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/19' 2023-10-10T18:53:29.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/15' 2023-10-10T18:53:29.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/20' 2023-10-10T18:53:29.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/12' 2023-10-10T18:53:29.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/10' 2023-10-10T18:53:29.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/11' 2023-10-10T18:53:29.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/16' 2023-10-10T18:53:29.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/14' 2023-10-10T18:53:29.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/18' 2023-10-10T18:53:29.665 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/13' 2023-10-10T18:53:29.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/118/17' 2023-10-10T18:53:29.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/118' 2023-10-10T18:53:29.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/19' 2023-10-10T18:53:29.666 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/15' 2023-10-10T18:53:29.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/20' 2023-10-10T18:53:29.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/12' 2023-10-10T18:53:29.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/10' 2023-10-10T18:53:29.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/11' 2023-10-10T18:53:29.667 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/16' 2023-10-10T18:53:29.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/14' 2023-10-10T18:53:29.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/18' 2023-10-10T18:53:29.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/13' 2023-10-10T18:53:29.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/196/17' 2023-10-10T18:53:29.668 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/196' 2023-10-10T18:53:29.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/19' 2023-10-10T18:53:29.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/15' 2023-10-10T18:53:29.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/20' 2023-10-10T18:53:29.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/12' 2023-10-10T18:53:29.669 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/10' 2023-10-10T18:53:29.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/11' 2023-10-10T18:53:29.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/16' 2023-10-10T18:53:29.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/14' 2023-10-10T18:53:29.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/18' 2023-10-10T18:53:29.670 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/13' 2023-10-10T18:53:29.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/124/17' 2023-10-10T18:53:29.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/124' 2023-10-10T18:53:29.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/19' 2023-10-10T18:53:29.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/15' 2023-10-10T18:53:29.671 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/20' 2023-10-10T18:53:29.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/12' 2023-10-10T18:53:29.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/10' 2023-10-10T18:53:29.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/11' 2023-10-10T18:53:29.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/16' 2023-10-10T18:53:29.672 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/14' 2023-10-10T18:53:29.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/18' 2023-10-10T18:53:29.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/13' 2023-10-10T18:53:29.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/188/17' 2023-10-10T18:53:29.673 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/188' 2023-10-10T18:53:29.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/19' 2023-10-10T18:53:29.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/15' 2023-10-10T18:53:29.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/20' 2023-10-10T18:53:29.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/12' 2023-10-10T18:53:29.674 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/10' 2023-10-10T18:53:29.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/11' 2023-10-10T18:53:29.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/16' 2023-10-10T18:53:29.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/14' 2023-10-10T18:53:29.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/18' 2023-10-10T18:53:29.675 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/13' 2023-10-10T18:53:29.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/145/17' 2023-10-10T18:53:29.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/145' 2023-10-10T18:53:29.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/19' 2023-10-10T18:53:29.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/15' 2023-10-10T18:53:29.676 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/20' 2023-10-10T18:53:29.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/12' 2023-10-10T18:53:29.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/10' 2023-10-10T18:53:29.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/11' 2023-10-10T18:53:29.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/16' 2023-10-10T18:53:29.677 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/14' 2023-10-10T18:53:29.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/18' 2023-10-10T18:53:29.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/13' 2023-10-10T18:53:29.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/130/17' 2023-10-10T18:53:29.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/130' 2023-10-10T18:53:29.678 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/19' 2023-10-10T18:53:29.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/15' 2023-10-10T18:53:29.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/20' 2023-10-10T18:53:29.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/12' 2023-10-10T18:53:29.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/10' 2023-10-10T18:53:29.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/11' 2023-10-10T18:53:29.679 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/16' 2023-10-10T18:53:29.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/14' 2023-10-10T18:53:29.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/18' 2023-10-10T18:53:29.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/13' 2023-10-10T18:53:29.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/193/17' 2023-10-10T18:53:29.680 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/193' 2023-10-10T18:53:29.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/19' 2023-10-10T18:53:29.681 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/15' 2023-10-10T18:53:29.709 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:29.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.839+0000 7f91688d3700 1 Processor -- start 2023-10-10T18:53:29.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.839+0000 7f91688d3700 1 -- start start 2023-10-10T18:53:29.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.839+0000 7f91688d3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9164155fb0 0x7f9164159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:29.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.839+0000 7f91688d3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91641574b0 con 0x7f91640b1fc0 2023-10-10T18:53:29.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.839+0000 7f91688d3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91640afad0 con 0x7f91640b2390 2023-10-10T18:53:29.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.839+0000 7f91688d3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9164150770 con 0x7f9164155fb0 2023-10-10T18:53:29.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916259c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f91640b1fc0 0x7f9164068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46160/0 (socket says 172.21.15.175:46160) 2023-10-10T18:53:29.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916259c700 1 -- 172.21.15.175:0/2924423049 learned_addr learned my addr 172.21.15.175:0/2924423049 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:29.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f9162d9d700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9164155fb0 0x7f9164159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:29.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639249418 0 0) 0x7f91641574b0 con 0x7f91640b1fc0 2023-10-10T18:53:29.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f914c003570 con 0x7f91640b1fc0 2023-10-10T18:53:29.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937258827 0 0) 0x7f91640afad0 con 0x7f91640b2390 2023-10-10T18:53:29.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91641574b0 con 0x7f91640b2390 2023-10-10T18:53:29.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4149483632 0 0) 0x7f914c003570 con 0x7f91640b1fc0 2023-10-10T18:53:29.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9164155fb0 msgr2=0x7f9164159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:29.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9164155fb0 0x7f9164159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:29.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 >> v1:172.21.15.191:6789/0 conn(0x7f91640b2390 legacy=0x7f91640aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:29.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640af4c0 con 0x7f91640b1fc0 2023-10-10T18:53:29.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f91540017f0 con 0x7f91640b1fc0 2023-10-10T18:53:29.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 >> v1:172.21.15.175:6789/0 conn(0x7f91640b1fc0 legacy=0x7f9164068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:29.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f914c003c20 0x7f914c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:29.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f914c004810 0x7f914c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:29.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c0051c0 0x7f914c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:29.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9164150770 con 0x7f914c003c20 2023-10-10T18:53:29.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c00c020 con 0x7f914c004810 2023-10-10T18:53:29.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c00c1e0 con 0x7f914c0051c0 2023-10-10T18:53:29.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f9162d9d700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9164155fb0 0x7f9164159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:29.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f916259c700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f914c003c20 0x7f914c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:29.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.840+0000 7f9161d9b700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f914c004810 0x7f914c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:29.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f9162d9d700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c0051c0 0x7f914c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:29.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916259c700 1 -- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c0051c0 msgr2=0x7f914c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:29.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916259c700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c0051c0 0x7f914c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:29.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916259c700 1 -- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f914c004810 msgr2=0x7f914c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:29.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916259c700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f914c004810 0x7f914c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:29.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916259c700 1 -- 172.21.15.175:0/2924423049 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c003570 con 0x7f914c003c20 2023-10-10T18:53:29.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916259c700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f914c003c20 0x7f914c004010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f915400a8d0 tx=0x7f91640712f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87eac6b26dcaa9b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:29.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91540070d0 con 0x7f914c003c20 2023-10-10T18:53:29.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9154007270 con 0x7f914c003c20 2023-10-10T18:53:29.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f916159a700 1 -- 172.21.15.175:0/2924423049 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154007630 con 0x7f914c003c20 2023-10-10T18:53:29.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f91688d3700 1 -- 172.21.15.175:0/2924423049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f914c003c20 msgr2=0x7f914c004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:29.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f91688d3700 1 --2- 172.21.15.175:0/2924423049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f914c003c20 0x7f914c004010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f915400a8d0 tx=0x7f91640712f0 comp rx=0 tx=0).stop 2023-10-10T18:53:29.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f9162d9d700 1 -- 172.21.15.175:0/2924423049 reap_dead start 2023-10-10T18:53:29.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.841+0000 7f91688d3700 1 -- 172.21.15.175:0/2924423049 shutdown_connections 2023-10-10T18:53:29.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.842+0000 7f91688d3700 1 -- 172.21.15.175:0/2924423049 >> 172.21.15.175:0/2924423049 conn(0x7f9164063230 msgr2=0x7f9164063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:29.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.842+0000 7f91688d3700 1 -- 172.21.15.175:0/2924423049 shutdown_connections 2023-10-10T18:53:29.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.842+0000 7f91688d3700 1 -- 172.21.15.175:0/2924423049 wait complete. 2023-10-10T18:53:29.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f91688d3700 1 Processor -- start 2023-10-10T18:53:29.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f91688d3700 1 -- start start 2023-10-10T18:53:29.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f91688d3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c003c20 0x7f9164150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:29.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f91688d3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91640cba90 0x7f91640cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:29.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f91688d3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9164169290 0x7f9164169700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:29.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f91688d3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640c6a40 con 0x7f91640cba90 2023-10-10T18:53:29.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f91688d3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640c4460 con 0x7f9164169290 2023-10-10T18:53:29.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f91688d3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9164159d10 con 0x7f914c003c20 2023-10-10T18:53:29.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f9161d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91640cba90 0x7f91640cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:29.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f916259c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c003c20 0x7f9164150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:29.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f9161d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91640cba90 0x7f91640cbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49758/0 (socket says 172.21.15.175:49758) 2023-10-10T18:53:29.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f9161d9b700 1 -- 172.21.15.175:0/1804201119 learned_addr learned my addr 172.21.15.175:0/1804201119 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:29.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.843+0000 7f9162d9d700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9164169290 0x7f9164169700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:29.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f9161d9b700 1 -- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c003c20 msgr2=0x7f9164150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:29.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f9161d9b700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c003c20 0x7f9164150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:29.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f9161d9b700 1 -- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9164169290 msgr2=0x7f9164169700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:29.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f9161d9b700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9164169290 0x7f9164169700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:29.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f9161d9b700 1 -- 172.21.15.175:0/1804201119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640cc440 con 0x7f91640cba90 2023-10-10T18:53:29.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f916259c700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c003c20 0x7f9164150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:29.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f9162d9d700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9164169290 0x7f9164169700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:29.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f9161d9b700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91640cba90 0x7f91640cbf00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f915800f930 tx=0x7f9158007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:29.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f91527fc700 1 -- 172.21.15.175:0/1804201119 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158006820 con 0x7f91640cba90 2023-10-10T18:53:29.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f91527fc700 1 -- 172.21.15.175:0/1804201119 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158006980 con 0x7f91640cba90 2023-10-10T18:53:29.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f91527fc700 1 -- 172.21.15.175:0/1804201119 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158006d40 con 0x7f91640cba90 2023-10-10T18:53:29.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91640c3ea0 con 0x7f91640cba90 2023-10-10T18:53:29.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.844+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91641599a0 con 0x7f91640cba90 2023-10-10T18:53:29.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.845+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91640cb7b0 con 0x7f91640cba90 2023-10-10T18:53:29.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.845+0000 7f91527fc700 1 -- 172.21.15.175:0/1804201119 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9158007240 con 0x7f91640cba90 2023-10-10T18:53:29.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.846+0000 7f91527fc700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9144066f30 0x7f91440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:29.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.846+0000 7f91527fc700 1 -- 172.21.15.175:0/1804201119 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f9158084600 con 0x7f91640cba90 2023-10-10T18:53:29.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.846+0000 7f916259c700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9144066f30 0x7f91440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:29.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.846+0000 7f916259c700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9144066f30 0x7f91440693e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f915400a390 tx=0x7f9154016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:29.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:29.850+0000 7f91527fc700 1 -- 172.21.15.175:0/1804201119 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91640cb7b0 con 0x7f91640cba90 2023-10-10T18:53:30.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.075+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91640cb7b0 con 0x7f91640cba90 2023-10-10T18:53:30.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.076+0000 7f91527fc700 1 -- 172.21.15.175:0/1804201119 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6213 (secure 0 0 0) 0x7f91640cb7b0 con 0x7f91640cba90 2023-10-10T18:53:30.076 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:30.076 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5367,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:27.623204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:30.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9144066f30 msgr2=0x7f91440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:30.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9144066f30 0x7f91440693e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f915400a390 tx=0x7f9154016000 comp rx=0 tx=0).stop 2023-10-10T18:53:30.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91640cba90 msgr2=0x7f91640cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:30.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91640cba90 0x7f91640cbf00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f915800f930 tx=0x7f9158007d10 comp rx=0 tx=0).stop 2023-10-10T18:53:30.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 shutdown_connections 2023-10-10T18:53:30.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9144066f30 0x7f91440693e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:30.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9164169290 0x7f9164169700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:30.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f914c003c20 0x7f9164150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:30.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 --2- 172.21.15.175:0/1804201119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91640cba90 0x7f91640cbf00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:30.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 >> 172.21.15.175:0/1804201119 conn(0x7f9164063230 msgr2=0x7f9164158120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:30.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 shutdown_connections 2023-10-10T18:53:30.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:30.078+0000 7f91688d3700 1 -- 172.21.15.175:0/1804201119 wait complete. 2023-10-10T18:53:30.081 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 69 2023-10-10T18:53:30.090 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T18:53:32.091 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:32.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.219+0000 7f87a54a3700 1 Processor -- start 2023-10-10T18:53:32.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.219+0000 7f87a54a3700 1 -- start start 2023-10-10T18:53:32.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.219+0000 7f87a54a3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 0x7f87a0159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:32.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.219+0000 7f87a54a3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a01574b0 con 0x7f87a00b1fc0 2023-10-10T18:53:32.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.219+0000 7f87a54a3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a00afad0 con 0x7f87a00b2390 2023-10-10T18:53:32.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.219+0000 7f87a54a3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0150770 con 0x7f87a0155fb0 2023-10-10T18:53:32.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879effd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f87a00b1fc0 0x7f87a0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46174/0 (socket says 172.21.15.175:46174) 2023-10-10T18:53:32.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879effd700 1 -- 172.21.15.175:0/2237967547 learned_addr learned my addr 172.21.15.175:0/2237967547 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:32.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879f7fe700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 0x7f87a0159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:32.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121999926 0 0) 0x7f87a01574b0 con 0x7f87a00b1fc0 2023-10-10T18:53:32.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878c003570 con 0x7f87a00b1fc0 2023-10-10T18:53:32.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183565358 0 0) 0x7f878c003570 con 0x7f87a00b1fc0 2023-10-10T18:53:32.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 msgr2=0x7f87a0159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:32.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 0x7f87a0159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 >> v1:172.21.15.191:6789/0 conn(0x7f87a00b2390 legacy=0x7f87a00aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:32.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a00af4c0 con 0x7f87a00b1fc0 2023-10-10T18:53:32.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f8790001470 con 0x7f87a00b1fc0 2023-10-10T18:53:32.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 >> v1:172.21.15.175:6789/0 conn(0x7f87a00b1fc0 legacy=0x7f87a0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:32.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f878c003a90 0x7f878c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:32.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f878c004810 0x7f878c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:32.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 0x7f878c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:32.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0150770 con 0x7f878c004810 2023-10-10T18:53:32.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c00bb10 con 0x7f878c003a90 2023-10-10T18:53:32.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c00bcd0 con 0x7f87a0155fb0 2023-10-10T18:53:32.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879effd700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f878c003a90 0x7f878c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:32.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f87967fc700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f878c004810 0x7f878c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:32.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.220+0000 7f879f7fe700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 0x7f878c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:32.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f87967fc700 1 -- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 msgr2=0x7f878c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:32.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f87967fc700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 0x7f878c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f87967fc700 1 -- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f878c003a90 msgr2=0x7f878c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:32.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f87967fc700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f878c003a90 0x7f878c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f87967fc700 1 -- 172.21.15.175:0/2237967547 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c003570 con 0x7f878c004810 2023-10-10T18:53:32.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f879effd700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f878c003a90 0x7f878c003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:32.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f879f7fe700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a0155fb0 0x7f878c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:32.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f87967fc700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f878c004810 0x7f878c004c80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f87880019a0 tx=0x7f8788007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3402bb106ebef67 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:32.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878800c5d0 con 0x7f878c004810 2023-10-10T18:53:32.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878800c750 con 0x7f878c004810 2023-10-10T18:53:32.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f879e7fc700 1 -- 172.21.15.175:0/2237967547 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878800cb10 con 0x7f878c004810 2023-10-10T18:53:32.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f87a54a3700 1 -- 172.21.15.175:0/2237967547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f878c004810 msgr2=0x7f878c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:32.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f87a54a3700 1 --2- 172.21.15.175:0/2237967547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f878c004810 0x7f878c004c80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f87880019a0 tx=0x7f8788007d10 comp rx=0 tx=0).stop 2023-10-10T18:53:32.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.221+0000 7f879f7fe700 1 -- 172.21.15.175:0/2237967547 reap_dead start 2023-10-10T18:53:32.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.222+0000 7f87a54a3700 1 -- 172.21.15.175:0/2237967547 shutdown_connections 2023-10-10T18:53:32.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.222+0000 7f87a54a3700 1 -- 172.21.15.175:0/2237967547 >> 172.21.15.175:0/2237967547 conn(0x7f87a0063230 msgr2=0x7f87a0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:32.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.222+0000 7f87a54a3700 1 -- 172.21.15.175:0/2237967547 shutdown_connections 2023-10-10T18:53:32.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.222+0000 7f87a54a3700 1 -- 172.21.15.175:0/2237967547 wait complete. 2023-10-10T18:53:32.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.223+0000 7f87a54a3700 1 Processor -- start 2023-10-10T18:53:32.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.223+0000 7f87a54a3700 1 -- start start 2023-10-10T18:53:32.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.223+0000 7f87a54a3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f878c004810 0x7f87a00c4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:32.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.223+0000 7f87a54a3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a0166860 0x7f87a015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:32.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.223+0000 7f87a54a3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a0152000 0x7f87a014f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:32.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.223+0000 7f87a54a3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0157180 con 0x7f87a0166860 2023-10-10T18:53:32.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.223+0000 7f87a54a3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0166cd0 con 0x7f87a0152000 2023-10-10T18:53:32.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.223+0000 7f87a54a3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a015bf10 con 0x7f878c004810 2023-10-10T18:53:32.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a0166860 0x7f87a015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:32.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a0166860 0x7f87a015c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49784/0 (socket says 172.21.15.175:49784) 2023-10-10T18:53:32.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 -- 172.21.15.175:0/169400671 learned_addr learned my addr 172.21.15.175:0/169400671 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:32.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f879f7fe700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a0152000 0x7f87a014f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:32.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f879effd700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f878c004810 0x7f87a00c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:32.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 -- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f878c004810 msgr2=0x7f87a00c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:32.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f878c004810 0x7f87a00c4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 -- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a0152000 msgr2=0x7f87a014f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:32.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a0152000 0x7f87a014f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 -- 172.21.15.175:0/169400671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a0157390 con 0x7f87a0166860 2023-10-10T18:53:32.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f879f7fe700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a0152000 0x7f87a014f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:53:32.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87967fc700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a0166860 0x7f87a015c3a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f878800c3c0 tx=0x7f8788015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:32.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87977fe700 1 -- 172.21.15.175:0/169400671 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878800c5d0 con 0x7f87a0166860 2023-10-10T18:53:32.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87977fe700 1 -- 172.21.15.175:0/169400671 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878800ce80 con 0x7f87a0166860 2023-10-10T18:53:32.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87977fe700 1 -- 172.21.15.175:0/169400671 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878800caf0 con 0x7f87a0166860 2023-10-10T18:53:32.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a0157670 con 0x7f87a0166860 2023-10-10T18:53:32.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.224+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a015bba0 con 0x7f87a0166860 2023-10-10T18:53:32.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.225+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a015bd60 con 0x7f87a0166860 2023-10-10T18:53:32.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.225+0000 7f87977fe700 1 -- 172.21.15.175:0/169400671 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8788019030 con 0x7f87a0166860 2023-10-10T18:53:32.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.226+0000 7f87977fe700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8780066f30 0x7f87800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:32.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.226+0000 7f87977fe700 1 -- 172.21.15.175:0/169400671 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f8788085be0 con 0x7f87a0166860 2023-10-10T18:53:32.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.226+0000 7f879effd700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8780066f30 0x7f87800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:32.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.226+0000 7f879effd700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8780066f30 0x7f87800693e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8790007fd0 tx=0x7f8790009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:32.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.230+0000 7f87977fe700 1 -- 172.21.15.175:0/169400671 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8788052da0 con 0x7f87a0166860 2023-10-10T18:53:32.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.455+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87a015c0c0 con 0x7f87a0166860 2023-10-10T18:53:32.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.456+0000 7f87977fe700 1 -- 172.21.15.175:0/169400671 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6213 (secure 0 0 0) 0x7f87a015c0c0 con 0x7f87a0166860 2023-10-10T18:53:32.456 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:32.456 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5367,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:27.623204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":4250,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"f","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6839/246221790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":246221790},{"type":"v1","addr":"172.21.15.191:6839","nonce":246221790}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:32.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8780066f30 msgr2=0x7f87800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:32.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8780066f30 0x7f87800693e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8790007fd0 tx=0x7f8790009040 comp rx=0 tx=0).stop 2023-10-10T18:53:32.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a0166860 msgr2=0x7f87a015c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:32.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a0166860 0x7f87a015c3a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f878800c3c0 tx=0x7f8788015000 comp rx=0 tx=0).stop 2023-10-10T18:53:32.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 shutdown_connections 2023-10-10T18:53:32.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8780066f30 0x7f87800693e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a0152000 0x7f87a014f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f878c004810 0x7f87a00c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 --2- 172.21.15.175:0/169400671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a0166860 0x7f87a015c3a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:32.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 >> 172.21.15.175:0/169400671 conn(0x7f87a0063230 msgr2=0x7f87a0068690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:32.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 shutdown_connections 2023-10-10T18:53:32.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:32.458+0000 7f87a54a3700 1 -- 172.21.15.175:0/169400671 wait complete. 2023-10-10T18:53:32.461 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 69 2023-10-10T18:53:32.470 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T18:53:33.315 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/20' 2023-10-10T18:53:33.316 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/12' 2023-10-10T18:53:33.316 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/10' 2023-10-10T18:53:33.316 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/11' 2023-10-10T18:53:33.316 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/16' 2023-10-10T18:53:33.317 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/14' 2023-10-10T18:53:33.317 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/18' 2023-10-10T18:53:33.317 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/13' 2023-10-10T18:53:33.317 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/175/17' 2023-10-10T18:53:33.317 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/175' 2023-10-10T18:53:33.318 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/19' 2023-10-10T18:53:33.318 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/15' 2023-10-10T18:53:33.318 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/20' 2023-10-10T18:53:33.318 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/12' 2023-10-10T18:53:33.318 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/10' 2023-10-10T18:53:33.319 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/11' 2023-10-10T18:53:33.319 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/16' 2023-10-10T18:53:33.319 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/14' 2023-10-10T18:53:33.319 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/18' 2023-10-10T18:53:33.319 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/13' 2023-10-10T18:53:33.320 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/181/17' 2023-10-10T18:53:33.320 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/181' 2023-10-10T18:53:33.320 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/19' 2023-10-10T18:53:33.320 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/15' 2023-10-10T18:53:33.320 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/20' 2023-10-10T18:53:33.321 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/12' 2023-10-10T18:53:33.321 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/10' 2023-10-10T18:53:33.321 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/11' 2023-10-10T18:53:33.321 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/16' 2023-10-10T18:53:33.322 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/14' 2023-10-10T18:53:33.322 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/18' 2023-10-10T18:53:33.322 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/13' 2023-10-10T18:53:33.322 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/127/17' 2023-10-10T18:53:33.323 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/127' 2023-10-10T18:53:33.323 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/19' 2023-10-10T18:53:33.323 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/15' 2023-10-10T18:53:33.323 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/20' 2023-10-10T18:53:33.323 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/12' 2023-10-10T18:53:33.324 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/10' 2023-10-10T18:53:33.324 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/11' 2023-10-10T18:53:33.324 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/16' 2023-10-10T18:53:33.324 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/14' 2023-10-10T18:53:33.324 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/18' 2023-10-10T18:53:33.325 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/13' 2023-10-10T18:53:33.325 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/108/17' 2023-10-10T18:53:33.325 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/108' 2023-10-10T18:53:33.325 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/19' 2023-10-10T18:53:33.326 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/15' 2023-10-10T18:53:33.326 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/20' 2023-10-10T18:53:33.326 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/12' 2023-10-10T18:53:33.326 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/10' 2023-10-10T18:53:33.326 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/11' 2023-10-10T18:53:33.327 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/16' 2023-10-10T18:53:33.327 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/14' 2023-10-10T18:53:33.327 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/18' 2023-10-10T18:53:33.327 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/13' 2023-10-10T18:53:33.327 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/183/17' 2023-10-10T18:53:33.328 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/183' 2023-10-10T18:53:33.328 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/19' 2023-10-10T18:53:33.328 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/15' 2023-10-10T18:53:33.328 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/20' 2023-10-10T18:53:33.328 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/12' 2023-10-10T18:53:33.328 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/10' 2023-10-10T18:53:33.329 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/11' 2023-10-10T18:53:33.329 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/16' 2023-10-10T18:53:33.329 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/14' 2023-10-10T18:53:33.329 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/18' 2023-10-10T18:53:33.330 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/13' 2023-10-10T18:53:33.330 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/177/17' 2023-10-10T18:53:33.330 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/177' 2023-10-10T18:53:33.330 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/19' 2023-10-10T18:53:33.330 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/15' 2023-10-10T18:53:33.331 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/20' 2023-10-10T18:53:33.331 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/12' 2023-10-10T18:53:33.331 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/10' 2023-10-10T18:53:33.331 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/11' 2023-10-10T18:53:33.332 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/16' 2023-10-10T18:53:33.332 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/14' 2023-10-10T18:53:33.332 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/18' 2023-10-10T18:53:33.332 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/13' 2023-10-10T18:53:33.332 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/169/17' 2023-10-10T18:53:33.332 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/169' 2023-10-10T18:53:33.333 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/19' 2023-10-10T18:53:33.333 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/15' 2023-10-10T18:53:33.333 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/20' 2023-10-10T18:53:33.333 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/12' 2023-10-10T18:53:33.334 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/10' 2023-10-10T18:53:33.334 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/11' 2023-10-10T18:53:33.334 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/16' 2023-10-10T18:53:33.335 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/14' 2023-10-10T18:53:33.335 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/18' 2023-10-10T18:53:33.335 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/13' 2023-10-10T18:53:33.335 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/154/17' 2023-10-10T18:53:33.335 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/154' 2023-10-10T18:53:33.336 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/19' 2023-10-10T18:53:33.336 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/15' 2023-10-10T18:53:33.336 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/20' 2023-10-10T18:53:33.336 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/12' 2023-10-10T18:53:33.336 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/10' 2023-10-10T18:53:33.337 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/11' 2023-10-10T18:53:33.337 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/16' 2023-10-10T18:53:33.337 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/14' 2023-10-10T18:53:33.337 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/18' 2023-10-10T18:53:33.337 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/13' 2023-10-10T18:53:33.338 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/171/17' 2023-10-10T18:53:33.338 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/171' 2023-10-10T18:53:33.338 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/19' 2023-10-10T18:53:33.338 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/15' 2023-10-10T18:53:33.339 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/20' 2023-10-10T18:53:34.471 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:34.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.590+0000 7fe35cc9e700 1 Processor -- start 2023-10-10T18:53:34.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.590+0000 7fe35cc9e700 1 -- start start 2023-10-10T18:53:34.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.591+0000 7fe35cc9e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3580b1fc0 0x7fe358155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.591+0000 7fe35cc9e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580aff60 0x7fe3580b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.591+0000 7fe35cc9e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3581574b0 con 0x7fe3580b2390 2023-10-10T18:53:34.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.591+0000 7fe35cc9e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe358150770 con 0x7fe3580b1fc0 2023-10-10T18:53:34.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.591+0000 7fe35cc9e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3580afad0 con 0x7fe3580aff60 2023-10-10T18:53:34.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.591+0000 7fe35659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3580b1fc0 0x7fe358155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.591+0000 7fe35659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3580b1fc0 0x7fe358155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56762/0 (socket says 172.21.15.175:56762) 2023-10-10T18:53:34.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.591+0000 7fe35659c700 1 -- 172.21.15.175:0/1018984211 learned_addr learned my addr 172.21.15.175:0/1018984211 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:34.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe356d9d700 1 --2- 172.21.15.175:0/1018984211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580aff60 0x7fe3580b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35659c700 1 -- 172.21.15.175:0/1018984211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580aff60 msgr2=0x7fe3580b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:34.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35659c700 1 --2- 172.21.15.175:0/1018984211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580aff60 0x7fe3580b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35659c700 1 -- 172.21.15.175:0/1018984211 >> v1:172.21.15.191:6789/0 conn(0x7fe3580b2390 legacy=0x7fe358068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35659c700 1 -- 172.21.15.175:0/1018984211 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3580af4c0 con 0x7fe3580b1fc0 2023-10-10T18:53:34.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35659c700 1 --2- 172.21.15.175:0/1018984211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580b1fc0 0x7fe358155fb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe3480099b0 tx=0x7fe348009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f0684fbee0d545 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:34.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35559a700 1 -- 172.21.15.175:0/1018984211 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34800b8f0 con 0x7fe3580b1fc0 2023-10-10T18:53:34.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35559a700 1 -- 172.21.15.175:0/1018984211 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34800ba50 con 0x7fe3580b1fc0 2023-10-10T18:53:34.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35559a700 1 -- 172.21.15.175:0/1018984211 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34800bdf0 con 0x7fe3580b1fc0 2023-10-10T18:53:34.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35cc9e700 1 -- 172.21.15.175:0/1018984211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580b1fc0 msgr2=0x7fe358155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.592+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/1018984211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580b1fc0 0x7fe358155fb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe3480099b0 tx=0x7fe348009250 comp rx=0 tx=0).stop 2023-10-10T18:53:34.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.593+0000 7fe35cc9e700 1 -- 172.21.15.175:0/1018984211 shutdown_connections 2023-10-10T18:53:34.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.593+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/1018984211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580aff60 0x7fe3580b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.593+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/1018984211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580b1fc0 0x7fe358155fb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.593+0000 7fe35cc9e700 1 -- 172.21.15.175:0/1018984211 >> 172.21.15.175:0/1018984211 conn(0x7fe358063230 msgr2=0x7fe358063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:34.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.593+0000 7fe35cc9e700 1 -- 172.21.15.175:0/1018984211 shutdown_connections 2023-10-10T18:53:34.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.593+0000 7fe35cc9e700 1 -- 172.21.15.175:0/1018984211 wait complete. 2023-10-10T18:53:34.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35cc9e700 1 Processor -- start 2023-10-10T18:53:34.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35cc9e700 1 -- start start 2023-10-10T18:53:34.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35cc9e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580aff60 0x7fe3581599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35cc9e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3580b1fc0 0x7fe3581666e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35cc9e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580b2390 0x7fe358156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35cc9e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe358166250 con 0x7fe3580aff60 2023-10-10T18:53:34.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35cc9e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3581508d0 con 0x7fe3580b1fc0 2023-10-10T18:53:34.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35cc9e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe358165f00 con 0x7fe3580b2390 2023-10-10T18:53:34.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580aff60 0x7fe3581599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580aff60 0x7fe3581599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56770/0 (socket says 172.21.15.175:56770) 2023-10-10T18:53:34.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe35659c700 1 -- 172.21.15.175:0/2810414863 learned_addr learned my addr 172.21.15.175:0/2810414863 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:34.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe356d9d700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580b2390 0x7fe358156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.594+0000 7fe355d9b700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3580b1fc0 0x7fe3581666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe35659c700 1 -- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580b2390 msgr2=0x7fe358156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe35659c700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580b2390 0x7fe358156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe35659c700 1 -- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3580b1fc0 msgr2=0x7fe3581666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe35659c700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3580b1fc0 0x7fe3581666e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe35659c700 1 -- 172.21.15.175:0/2810414863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3480094f0 con 0x7fe3580aff60 2023-10-10T18:53:34.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe35659c700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580aff60 0x7fe3581599e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe35806fd90 tx=0x7fe348017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:34.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe3467fc700 1 -- 172.21.15.175:0/2810414863 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348011e40 con 0x7fe3580aff60 2023-10-10T18:53:34.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe3467fc700 1 -- 172.21.15.175:0/2810414863 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348024440 con 0x7fe3580aff60 2023-10-10T18:53:34.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe3467fc700 1 -- 172.21.15.175:0/2810414863 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348024800 con 0x7fe3580aff60 2023-10-10T18:53:34.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe358150b20 con 0x7fe3580aff60 2023-10-10T18:53:34.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.595+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe358169e40 con 0x7fe3580aff60 2023-10-10T18:53:34.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.596+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe358166530 con 0x7fe3580aff60 2023-10-10T18:53:34.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.596+0000 7fe3467fc700 1 -- 172.21.15.175:0/2810414863 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe34801a420 con 0x7fe3580aff60 2023-10-10T18:53:34.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.597+0000 7fe3467fc700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe338066f00 0x7fe3380693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.597+0000 7fe3467fc700 1 -- 172.21.15.175:0/2810414863 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fe3480864e0 con 0x7fe3580aff60 2023-10-10T18:53:34.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.597+0000 7fe355d9b700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe338066f00 0x7fe3380693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.597+0000 7fe355d9b700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe338066f00 0x7fe3380693b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe358158720 tx=0x7fe34c007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:34.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.601+0000 7fe3467fc700 1 -- 172.21.15.175:0/2810414863 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe348053580 con 0x7fe3580aff60 2023-10-10T18:53:34.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.833+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe35815bdb0 con 0x7fe3580aff60 2023-10-10T18:53:34.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.833+0000 7fe3467fc700 1 -- 172.21.15.175:0/2810414863 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5450 (secure 0 0 0) 0x7fe348056ba0 con 0x7fe3580aff60 2023-10-10T18:53:34.833 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:34.834 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:33.482137+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:34.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe338066f00 msgr2=0x7fe3380693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe338066f00 0x7fe3380693b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe358158720 tx=0x7fe34c007ec0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580aff60 msgr2=0x7fe3581599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580aff60 0x7fe3581599e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe35806fd90 tx=0x7fe348017000 comp rx=0 tx=0).stop 2023-10-10T18:53:34.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 shutdown_connections 2023-10-10T18:53:34.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe338066f00 0x7fe3380693b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3580b2390 0x7fe358156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3580aff60 0x7fe3581599e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 --2- 172.21.15.175:0/2810414863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3580b1fc0 0x7fe3581666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 >> 172.21.15.175:0/2810414863 conn(0x7fe358063230 msgr2=0x7fe35814fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:34.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 shutdown_connections 2023-10-10T18:53:34.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.835+0000 7fe35cc9e700 1 -- 172.21.15.175:0/2810414863 wait complete. 2023-10-10T18:53:34.838 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 70 2023-10-10T18:53:34.847 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-10T18:53:34.847 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T18:53:34.847 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:34.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.961+0000 7fbf535c5700 1 Processor -- start 2023-10-10T18:53:34.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.961+0000 7fbf535c5700 1 -- start start 2023-10-10T18:53:34.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf535c5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbf4c0b1fc0 0x7fbf4c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf535c5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbf4c0aff60 0x7fbf4c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf535c5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4c1574b0 con 0x7fbf4c0b2390 2023-10-10T18:53:34.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf535c5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4c150770 con 0x7fbf4c0aff60 2023-10-10T18:53:34.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf535c5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4c0afad0 con 0x7fbf4c0b1fc0 2023-10-10T18:53:34.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf51b62700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbf4c0aff60 0x7fbf4c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf50b60700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbf4c0b2390 0x7fbf4c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52676/0 (socket says 172.21.15.175:52676) 2023-10-10T18:53:34.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf50b60700 1 -- 172.21.15.175:0/1983060682 learned_addr learned my addr 172.21.15.175:0/1983060682 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:34.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf51361700 1 --2- 172.21.15.175:0/1983060682 >> v2:172.21.15.191:3300/0 conn(0x7fbf4c0b1fc0 0x7fbf4c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf51b62700 1 -- 172.21.15.175:0/1983060682 >> v2:172.21.15.191:3300/0 conn(0x7fbf4c0b1fc0 msgr2=0x7fbf4c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:34.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf51b62700 1 --2- 172.21.15.175:0/1983060682 >> v2:172.21.15.191:3300/0 conn(0x7fbf4c0b1fc0 0x7fbf4c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf51b62700 1 -- 172.21.15.175:0/1983060682 >> v1:172.21.15.175:6789/0 conn(0x7fbf4c0b2390 legacy=0x7fbf4c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf43fff700 1 -- 172.21.15.175:0/1983060682 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268989408 0 0) 0x7fbf4c1574b0 con 0x7fbf4c0b2390 2023-10-10T18:53:34.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf51b62700 1 -- 172.21.15.175:0/1983060682 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c0af4c0 con 0x7fbf4c0aff60 2023-10-10T18:53:34.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.962+0000 7fbf51361700 1 --2- 172.21.15.175:0/1983060682 >> v2:172.21.15.191:3300/0 conn(0x7fbf4c0b1fc0 0x7fbf4c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:53:34.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf51b62700 1 --2- 172.21.15.175:0/1983060682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0aff60 0x7fbf4c0b03e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0b28c0 tx=0x7fbf44009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94da5fcf462944b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:34.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf43fff700 1 -- 172.21.15.175:0/1983060682 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4400e070 con 0x7fbf4c0aff60 2023-10-10T18:53:34.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf43fff700 1 -- 172.21.15.175:0/1983060682 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4400bcc0 con 0x7fbf4c0aff60 2023-10-10T18:53:34.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf43fff700 1 -- 172.21.15.175:0/1983060682 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44012680 con 0x7fbf4c0aff60 2023-10-10T18:53:34.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf535c5700 1 -- 172.21.15.175:0/1983060682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0aff60 msgr2=0x7fbf4c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf535c5700 1 --2- 172.21.15.175:0/1983060682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0aff60 0x7fbf4c0b03e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0b28c0 tx=0x7fbf44009780 comp rx=0 tx=0).stop 2023-10-10T18:53:34.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf535c5700 1 -- 172.21.15.175:0/1983060682 shutdown_connections 2023-10-10T18:53:34.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf535c5700 1 --2- 172.21.15.175:0/1983060682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0aff60 0x7fbf4c0b03e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf535c5700 1 --2- 172.21.15.175:0/1983060682 >> v2:172.21.15.191:3300/0 conn(0x7fbf4c0b1fc0 0x7fbf4c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf535c5700 1 -- 172.21.15.175:0/1983060682 >> 172.21.15.175:0/1983060682 conn(0x7fbf4c063230 msgr2=0x7fbf4c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:34.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf535c5700 1 -- 172.21.15.175:0/1983060682 shutdown_connections 2023-10-10T18:53:34.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.963+0000 7fbf535c5700 1 -- 172.21.15.175:0/1983060682 wait complete. 2023-10-10T18:53:34.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.964+0000 7fbf535c5700 1 Processor -- start 2023-10-10T18:53:34.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.964+0000 7fbf535c5700 1 -- start start 2023-10-10T18:53:34.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf535c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf4c0aff60 0x7fbf4c0cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf535c5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf4c0b1fc0 0x7fbf4c166420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf535c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0b2390 0x7fbf4c166960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf535c5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c0cd1a0 con 0x7fbf4c0b2390 2023-10-10T18:53:34.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf535c5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c165f90 con 0x7fbf4c0b1fc0 2023-10-10T18:53:34.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf535c5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c068770 con 0x7fbf4c0aff60 2023-10-10T18:53:34.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf51b62700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0b2390 0x7fbf4c166960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf51b62700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0b2390 0x7fbf4c166960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56786/0 (socket says 172.21.15.175:56786) 2023-10-10T18:53:34.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.965+0000 7fbf51b62700 1 -- 172.21.15.175:0/2621717830 learned_addr learned my addr 172.21.15.175:0/2621717830 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:34.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf51361700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf4c0aff60 0x7fbf4c0cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf50b60700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf4c0b1fc0 0x7fbf4c166420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf51b62700 1 -- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf4c0aff60 msgr2=0x7fbf4c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf51b62700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf4c0aff60 0x7fbf4c0cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf51b62700 1 -- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf4c0b1fc0 msgr2=0x7fbf4c166420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:34.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf51b62700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf4c0b1fc0 0x7fbf4c166420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:34.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf51b62700 1 -- 172.21.15.175:0/2621717830 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf44009a20 con 0x7fbf4c0b2390 2023-10-10T18:53:34.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf50b60700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf4c0b1fc0 0x7fbf4c166420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:53:34.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf51b62700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0b2390 0x7fbf4c166960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009fd0 tx=0x7fbf44019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:34.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf417fa700 1 -- 172.21.15.175:0/2621717830 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44027470 con 0x7fbf4c0b2390 2023-10-10T18:53:34.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf417fa700 1 -- 172.21.15.175:0/2621717830 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf440275f0 con 0x7fbf4c0b2390 2023-10-10T18:53:34.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4c0c6280 con 0x7fbf4c0b2390 2023-10-10T18:53:34.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf417fa700 1 -- 172.21.15.175:0/2621717830 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf440279b0 con 0x7fbf4c0b2390 2023-10-10T18:53:34.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.966+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4c0c6520 con 0x7fbf4c0b2390 2023-10-10T18:53:34.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.967+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4c166140 con 0x7fbf4c0b2390 2023-10-10T18:53:34.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.967+0000 7fbf417fa700 1 -- 172.21.15.175:0/2621717830 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbf44024070 con 0x7fbf4c0b2390 2023-10-10T18:53:34.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.968+0000 7fbf417fa700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f30 0x7fbf300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:34.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.968+0000 7fbf417fa700 1 -- 172.21.15.175:0/2621717830 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fbf44087220 con 0x7fbf4c0b2390 2023-10-10T18:53:34.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.968+0000 7fbf51361700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f30 0x7fbf300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:34.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.968+0000 7fbf51361700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f30 0x7fbf300693e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbf48006f00 tx=0x7fbf48008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:34.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:34.972+0000 7fbf417fa700 1 -- 172.21.15.175:0/2621717830 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf4c166140 con 0x7fbf4c0b2390 2023-10-10T18:53:35.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.194+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf4c166140 con 0x7fbf4c0b2390 2023-10-10T18:53:35.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.195+0000 7fbf417fa700 1 -- 172.21.15.175:0/2621717830 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5450 (secure 0 0 0) 0x7fbf4c166140 con 0x7fbf4c0b2390 2023-10-10T18:53:35.195 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:35.195 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:33.482137+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:35.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.196+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f30 msgr2=0x7fbf300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:35.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.196+0000 7fbf535c5700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f30 0x7fbf300693e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbf48006f00 tx=0x7fbf48008040 comp rx=0 tx=0).stop 2023-10-10T18:53:35.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.196+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0b2390 msgr2=0x7fbf4c166960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:35.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.196+0000 7fbf535c5700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0b2390 0x7fbf4c166960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009fd0 tx=0x7fbf44019000 comp rx=0 tx=0).stop 2023-10-10T18:53:35.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.197+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 shutdown_connections 2023-10-10T18:53:35.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.197+0000 7fbf535c5700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f30 0x7fbf300693e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:35.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.197+0000 7fbf535c5700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf4c0b2390 0x7fbf4c166960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:35.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.197+0000 7fbf535c5700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf4c0aff60 0x7fbf4c0cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:35.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.197+0000 7fbf535c5700 1 --2- 172.21.15.175:0/2621717830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf4c0b1fc0 0x7fbf4c166420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:35.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.197+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 >> 172.21.15.175:0/2621717830 conn(0x7fbf4c063230 msgr2=0x7fbf4c153db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:35.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.197+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 shutdown_connections 2023-10-10T18:53:35.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:35.197+0000 7fbf535c5700 1 -- 172.21.15.175:0/2621717830 wait complete. 2023-10-10T18:53:35.199 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 70 2023-10-10T18:53:35.209 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 70, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:53:33.482137+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 5367, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:53:37.210 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:37.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.337+0000 7f604140e700 1 Processor -- start 2023-10-10T18:53:37.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.337+0000 7f604140e700 1 -- start start 2023-10-10T18:53:37.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.338+0000 7f604140e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:37.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.338+0000 7f604140e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c1574b0 con 0x7f603c0aff60 2023-10-10T18:53:37.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.338+0000 7f604140e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c0afad0 con 0x7f603c0b2390 2023-10-10T18:53:37.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.338+0000 7f604140e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c150770 con 0x7f603c0b1fc0 2023-10-10T18:53:37.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.338+0000 7f603b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f603c0aff60 0x7f603c159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52684/0 (socket says 172.21.15.175:52684) 2023-10-10T18:53:37.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.338+0000 7f603b7fe700 1 -- 172.21.15.175:0/4234886660 learned_addr learned my addr 172.21.15.175:0/4234886660 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:37.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.338+0000 7f603affd700 1 --2- 172.21.15.175:0/4234886660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:37.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603a7fc700 1 -- 172.21.15.175:0/4234886660 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822036431 0 0) 0x7f603c1574b0 con 0x7f603c0aff60 2023-10-10T18:53:37.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603a7fc700 1 -- 172.21.15.175:0/4234886660 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6028003570 con 0x7f603c0aff60 2023-10-10T18:53:37.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603a7fc700 1 -- 172.21.15.175:0/4234886660 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302952846 0 0) 0x7f603c0afad0 con 0x7f603c0b2390 2023-10-10T18:53:37.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603a7fc700 1 -- 172.21.15.175:0/4234886660 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f603c1574b0 con 0x7f603c0b2390 2023-10-10T18:53:37.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603affd700 1 -- 172.21.15.175:0/4234886660 >> v1:172.21.15.191:6789/0 conn(0x7f603c0b2390 legacy=0x7f603c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:37.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603a7fc700 1 -- 172.21.15.175:0/4234886660 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619326431 0 0) 0x7f6028003570 con 0x7f603c0aff60 2023-10-10T18:53:37.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603affd700 1 -- 172.21.15.175:0/4234886660 >> v1:172.21.15.175:6789/0 conn(0x7f603c0aff60 legacy=0x7f603c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:37.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603affd700 1 -- 172.21.15.175:0/4234886660 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0af4c0 con 0x7f603c0b1fc0 2023-10-10T18:53:37.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603affd700 1 --2- 172.21.15.175:0/4234886660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c155fb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f602c009940 tx=0x7f602c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7ecfea0f0d1cdfc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:37.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603a7fc700 1 -- 172.21.15.175:0/4234886660 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60340017f0 con 0x7f603c0b1fc0 2023-10-10T18:53:37.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603a7fc700 1 -- 172.21.15.175:0/4234886660 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c011420 con 0x7f603c0b1fc0 2023-10-10T18:53:37.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.339+0000 7f603a7fc700 1 -- 172.21.15.175:0/4234886660 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c0117c0 con 0x7f603c0b1fc0 2023-10-10T18:53:37.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.340+0000 7f604140e700 1 -- 172.21.15.175:0/4234886660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 msgr2=0x7f603c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:37.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.340+0000 7f604140e700 1 --2- 172.21.15.175:0/4234886660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c155fb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f602c009940 tx=0x7f602c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:53:37.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.340+0000 7f604140e700 1 -- 172.21.15.175:0/4234886660 shutdown_connections 2023-10-10T18:53:37.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.340+0000 7f604140e700 1 --2- 172.21.15.175:0/4234886660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c155fb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:37.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.340+0000 7f604140e700 1 -- 172.21.15.175:0/4234886660 >> 172.21.15.175:0/4234886660 conn(0x7f603c063230 msgr2=0x7f603c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:37.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.340+0000 7f604140e700 1 -- 172.21.15.175:0/4234886660 shutdown_connections 2023-10-10T18:53:37.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.340+0000 7f604140e700 1 -- 172.21.15.175:0/4234886660 wait complete. 2023-10-10T18:53:37.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.341+0000 7f604140e700 1 Processor -- start 2023-10-10T18:53:37.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.341+0000 7f604140e700 1 -- start start 2023-10-10T18:53:37.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f604140e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f603c0aff60 0x7f603c165e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:37.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f604140e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c0c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:37.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f604140e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f603c0b2390 0x7f603c0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:37.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f604140e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c063ea0 con 0x7f603c0aff60 2023-10-10T18:53:37.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f604140e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0c5aa0 con 0x7f603c0b2390 2023-10-10T18:53:37.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f604140e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0c3bd0 con 0x7f603c0b1fc0 2023-10-10T18:53:37.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f603c0aff60 0x7f603c165e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:37.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f603c0aff60 0x7f603c165e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56790/0 (socket says 172.21.15.175:56790) 2023-10-10T18:53:37.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 -- 172.21.15.175:0/260885476 learned_addr learned my addr 172.21.15.175:0/260885476 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:37.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f60327fc700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c0c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:37.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603b7fe700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f603c0b2390 0x7f603c0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:37.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 -- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 msgr2=0x7f603c0c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:37.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c0c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:37.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 -- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f603c0b2390 msgr2=0x7f603c0c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:37.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f603c0b2390 0x7f603c0c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:37.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 -- 172.21.15.175:0/260885476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c009420 con 0x7f603c0aff60 2023-10-10T18:53:37.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f603affd700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f603c0aff60 0x7f603c165e80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f602c009860 tx=0x7f602c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:37.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f60337fe700 1 -- 172.21.15.175:0/260885476 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c01bb20 con 0x7f603c0aff60 2023-10-10T18:53:37.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f60337fe700 1 -- 172.21.15.175:0/260885476 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c01bc80 con 0x7f603c0aff60 2023-10-10T18:53:37.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c0c3eb0 con 0x7f603c0aff60 2023-10-10T18:53:37.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.342+0000 7f60337fe700 1 -- 172.21.15.175:0/260885476 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c01c450 con 0x7f603c0aff60 2023-10-10T18:53:37.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.343+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c0c4470 con 0x7f603c0aff60 2023-10-10T18:53:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.343+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c067570 con 0x7f603c0aff60 2023-10-10T18:53:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.344+0000 7f60337fe700 1 -- 172.21.15.175:0/260885476 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f602c01c840 con 0x7f603c0aff60 2023-10-10T18:53:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.344+0000 7f60337fe700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f601c066f30 0x7f601c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.344+0000 7f60337fe700 1 -- 172.21.15.175:0/260885476 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f602c085cf0 con 0x7f603c0aff60 2023-10-10T18:53:37.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.344+0000 7f60327fc700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f601c066f30 0x7f601c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:37.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.345+0000 7f60327fc700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f601c066f30 0x7f601c0693e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f603c0c5820 tx=0x7f6024007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:37.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.348+0000 7f60337fe700 1 -- 172.21.15.175:0/260885476 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f602c08a070 con 0x7f603c0aff60 2023-10-10T18:53:37.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.574+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f603c068da0 con 0x7f603c0aff60 2023-10-10T18:53:37.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.574+0000 7f60337fe700 1 -- 172.21.15.175:0/260885476 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5450 (secure 0 0 0) 0x7f602c02b090 con 0x7f603c0aff60 2023-10-10T18:53:37.575 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:37.575 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:33.482137+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:37.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f601c066f30 msgr2=0x7f601c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:37.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f601c066f30 0x7f601c0693e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f603c0c5820 tx=0x7f6024007d50 comp rx=0 tx=0).stop 2023-10-10T18:53:37.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f603c0aff60 msgr2=0x7f603c165e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f603c0aff60 0x7f603c165e80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f602c009860 tx=0x7f602c016000 comp rx=0 tx=0).stop 2023-10-10T18:53:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 shutdown_connections 2023-10-10T18:53:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f601c066f30 0x7f601c0693e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f603c0b2390 0x7f603c0c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f603c0aff60 0x7f603c165e80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:37.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 --2- 172.21.15.175:0/260885476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f603c0b1fc0 0x7f603c0c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:37.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 >> 172.21.15.175:0/260885476 conn(0x7f603c063230 msgr2=0x7f603c14fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:37.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 shutdown_connections 2023-10-10T18:53:37.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:37.576+0000 7f604140e700 1 -- 172.21.15.175:0/260885476 wait complete. 2023-10-10T18:53:37.579 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 70 2023-10-10T18:53:37.588 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:53:39.589 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:39.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.707+0000 7fefa4352700 1 Processor -- start 2023-10-10T18:53:39.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.707+0000 7fefa4352700 1 -- start start 2023-10-10T18:53:39.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.707+0000 7fefa4352700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fef9c0b1fc0 0x7fef9c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:39.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa4352700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b2390 0x7fef9c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:39.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa4352700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c1574b0 con 0x7fef9c068bd0 2023-10-10T18:53:39.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa4352700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c150770 con 0x7fef9c0b1fc0 2023-10-10T18:53:39.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa4352700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0afad0 con 0x7fef9c0b2390 2023-10-10T18:53:39.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa20ee700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fef9c0b1fc0 0x7fef9c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:39.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa20ee700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fef9c0b1fc0 0x7fef9c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:53384/0 (socket says 172.21.15.175:53384) 2023-10-10T18:53:39.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa18ed700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b2390 0x7fef9c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa20ee700 1 -- 172.21.15.175:0/1194145356 learned_addr learned my addr 172.21.15.175:0/1194145356 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa20ee700 1 -- 172.21.15.175:0/1194145356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b2390 msgr2=0x7fef9c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa20ee700 1 --2- 172.21.15.175:0/1194145356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b2390 0x7fef9c068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa20ee700 1 -- 172.21.15.175:0/1194145356 >> v1:172.21.15.191:6789/0 conn(0x7fef9c068bd0 legacy=0x7fef9c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:39.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa20ee700 1 -- 172.21.15.175:0/1194145356 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0af4c0 con 0x7fef9c0b1fc0 2023-10-10T18:53:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa18ed700 1 --2- 172.21.15.175:0/1194145356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b2390 0x7fef9c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.708+0000 7fefa20ee700 1 --2- 172.21.15.175:0/1194145356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c0b1fc0 0x7fef9c155fb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef940099b0 tx=0x7fef94009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dc1e0b85009bf02 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa10ec700 1 -- 172.21.15.175:0/1194145356 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94009ea0 con 0x7fef9c0b1fc0 2023-10-10T18:53:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa10ec700 1 -- 172.21.15.175:0/1194145356 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9400b8f0 con 0x7fef9c0b1fc0 2023-10-10T18:53:39.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa10ec700 1 -- 172.21.15.175:0/1194145356 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400bc90 con 0x7fef9c0b1fc0 2023-10-10T18:53:39.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa4352700 1 -- 172.21.15.175:0/1194145356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c0b1fc0 msgr2=0x7fef9c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:39.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa4352700 1 --2- 172.21.15.175:0/1194145356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c0b1fc0 0x7fef9c155fb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef940099b0 tx=0x7fef94009250 comp rx=0 tx=0).stop 2023-10-10T18:53:39.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa4352700 1 -- 172.21.15.175:0/1194145356 shutdown_connections 2023-10-10T18:53:39.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa4352700 1 --2- 172.21.15.175:0/1194145356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c0b1fc0 0x7fef9c155fb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa4352700 1 --2- 172.21.15.175:0/1194145356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b2390 0x7fef9c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa4352700 1 -- 172.21.15.175:0/1194145356 >> 172.21.15.175:0/1194145356 conn(0x7fef9c063230 msgr2=0x7fef9c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:39.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa4352700 1 -- 172.21.15.175:0/1194145356 shutdown_connections 2023-10-10T18:53:39.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.709+0000 7fefa4352700 1 -- 172.21.15.175:0/1194145356 wait complete. 2023-10-10T18:53:39.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.710+0000 7fefa4352700 1 Processor -- start 2023-10-10T18:53:39.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.710+0000 7fefa4352700 1 -- start start 2023-10-10T18:53:39.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa4352700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c068bd0 0x7fef9c14ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:39.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa4352700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b1fc0 0x7fef9c0c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:39.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa4352700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef9c0b2390 0x7fef9c14db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:39.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa4352700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c156e60 con 0x7fef9c0b2390 2023-10-10T18:53:39.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa4352700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c155a70 con 0x7fef9c068bd0 2023-10-10T18:53:39.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa4352700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c169fc0 con 0x7fef9c0b1fc0 2023-10-10T18:53:39.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa28ef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef9c0b2390 0x7fef9c14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:39.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa20ee700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c068bd0 0x7fef9c14ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:39.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa28ef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef9c0b2390 0x7fef9c14db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56798/0 (socket says 172.21.15.175:56798) 2023-10-10T18:53:39.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa28ef700 1 -- 172.21.15.175:0/1958568305 learned_addr learned my addr 172.21.15.175:0/1958568305 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:39.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa18ed700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b1fc0 0x7fef9c0c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:39.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa20ee700 1 -- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b1fc0 msgr2=0x7fef9c0c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:39.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa20ee700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b1fc0 0x7fef9c0c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa20ee700 1 -- 172.21.15.175:0/1958568305 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef9c0b2390 msgr2=0x7fef9c14db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:39.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa20ee700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef9c0b2390 0x7fef9c14db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.711+0000 7fefa20ee700 1 -- 172.21.15.175:0/1958568305 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c009420 con 0x7fef9c068bd0 2023-10-10T18:53:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.712+0000 7fefa20ee700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c068bd0 0x7fef9c14ee10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fef94013000 tx=0x7fef94017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.712+0000 7fef927fc700 1 -- 172.21.15.175:0/1958568305 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400bc70 con 0x7fef9c068bd0 2023-10-10T18:53:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.712+0000 7fef927fc700 1 -- 172.21.15.175:0/1958568305 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef94011420 con 0x7fef9c068bd0 2023-10-10T18:53:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.712+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef940094f0 con 0x7fef9c068bd0 2023-10-10T18:53:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.712+0000 7fef927fc700 1 -- 172.21.15.175:0/1958568305 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef940117c0 con 0x7fef9c068bd0 2023-10-10T18:53:39.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.712+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c155140 con 0x7fef9c068bd0 2023-10-10T18:53:39.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.713+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c067570 con 0x7fef9c068bd0 2023-10-10T18:53:39.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.713+0000 7fef927fc700 1 -- 172.21.15.175:0/1958568305 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fef94011bf0 con 0x7fef9c068bd0 2023-10-10T18:53:39.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.713+0000 7fef927fc700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef80066f30 0x7fef800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:39.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.713+0000 7fef927fc700 1 -- 172.21.15.175:0/1958568305 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fef94085f60 con 0x7fef9c068bd0 2023-10-10T18:53:39.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.714+0000 7fefa18ed700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef80066f30 0x7fef800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:39.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.714+0000 7fefa18ed700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef80066f30 0x7fef800693e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef9c069b30 tx=0x7fef8c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:39.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.718+0000 7fef927fc700 1 -- 172.21.15.175:0/1958568305 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef94053000 con 0x7fef9c068bd0 2023-10-10T18:53:39.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.946+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef9c069060 con 0x7fef9c068bd0 2023-10-10T18:53:39.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.947+0000 7fef927fc700 1 -- 172.21.15.175:0/1958568305 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5452 (secure 0 0 0) 0x7fef94015090 con 0x7fef9c068bd0 2023-10-10T18:53:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:39.948 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:39.650890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:39.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef80066f30 msgr2=0x7fef800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:39.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef80066f30 0x7fef800693e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef9c069b30 tx=0x7fef8c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c068bd0 msgr2=0x7fef9c14ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:39.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c068bd0 0x7fef9c14ee10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fef94013000 tx=0x7fef94017000 comp rx=0 tx=0).stop 2023-10-10T18:53:39.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 shutdown_connections 2023-10-10T18:53:39.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fef80066f30 0x7fef800693e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fef9c0b2390 0x7fef9c14db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef9c068bd0 0x7fef9c14ee10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 --2- 172.21.15.175:0/1958568305 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef9c0b1fc0 0x7fef9c0c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 >> 172.21.15.175:0/1958568305 conn(0x7fef9c063230 msgr2=0x7fef9c159310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 shutdown_connections 2023-10-10T18:53:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:39.950+0000 7fefa4352700 1 -- 172.21.15.175:0/1958568305 wait complete. 2023-10-10T18:53:39.953 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 71 2023-10-10T18:53:39.961 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:53:41.962 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:42.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.089+0000 7f37d461c700 1 Processor -- start 2023-10-10T18:53:42.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.089+0000 7f37d461c700 1 -- start start 2023-10-10T18:53:42.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d461c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37cc0b1fc0 0x7f37cc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:42.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d461c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0aff60 0x7f37cc0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:42.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d461c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc1574b0 con 0x7f37cc0b2390 2023-10-10T18:53:42.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d461c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc150770 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d461c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0afad0 con 0x7f37cc0aff60 2023-10-10T18:53:42.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d23b8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37cc0b1fc0 0x7f37cc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:42.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d23b8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37cc0b1fc0 0x7f37cc155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56812/0 (socket says 172.21.15.175:56812) 2023-10-10T18:53:42.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d23b8700 1 -- 172.21.15.175:0/2245759148 learned_addr learned my addr 172.21.15.175:0/2245759148 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:42.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d2bb9700 1 --2- 172.21.15.175:0/2245759148 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0aff60 0x7f37cc0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:42.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d23b8700 1 -- 172.21.15.175:0/2245759148 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0aff60 msgr2=0x7f37cc0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:42.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d23b8700 1 --2- 172.21.15.175:0/2245759148 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0aff60 0x7f37cc0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d23b8700 1 -- 172.21.15.175:0/2245759148 >> v1:172.21.15.191:6789/0 conn(0x7f37cc0b2390 legacy=0x7f37cc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:42.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d23b8700 1 -- 172.21.15.175:0/2245759148 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc0af4c0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d2bb9700 1 --2- 172.21.15.175:0/2245759148 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0aff60 0x7f37cc0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:53:42.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d23b8700 1 --2- 172.21.15.175:0/2245759148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc155fb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f37c80099b0 tx=0x7f37c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5998d6e49c26afb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:42.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d13b6700 1 -- 172.21.15.175:0/2245759148 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c800b8f0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.090+0000 7f37d13b6700 1 -- 172.21.15.175:0/2245759148 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c800ba50 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d13b6700 1 -- 172.21.15.175:0/2245759148 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c800bdf0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d461c700 1 -- 172.21.15.175:0/2245759148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 msgr2=0x7f37cc155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:42.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d461c700 1 --2- 172.21.15.175:0/2245759148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc155fb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f37c80099b0 tx=0x7f37c8009250 comp rx=0 tx=0).stop 2023-10-10T18:53:42.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d461c700 1 -- 172.21.15.175:0/2245759148 shutdown_connections 2023-10-10T18:53:42.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d461c700 1 --2- 172.21.15.175:0/2245759148 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0aff60 0x7f37cc0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d461c700 1 --2- 172.21.15.175:0/2245759148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc155fb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d461c700 1 -- 172.21.15.175:0/2245759148 >> 172.21.15.175:0/2245759148 conn(0x7f37cc063230 msgr2=0x7f37cc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:42.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d461c700 1 -- 172.21.15.175:0/2245759148 shutdown_connections 2023-10-10T18:53:42.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.091+0000 7f37d461c700 1 -- 172.21.15.175:0/2245759148 wait complete. 2023-10-10T18:53:42.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.092+0000 7f37d461c700 1 Processor -- start 2023-10-10T18:53:42.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.092+0000 7f37d461c700 1 -- start start 2023-10-10T18:53:42.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d461c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37cc0aff60 0x7f37cc14de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:42.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d461c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc0c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:42.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d461c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0b2390 0x7f37cc0c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:42.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d461c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc166b40 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d461c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0c5920 con 0x7f37cc0aff60 2023-10-10T18:53:42.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d461c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0c4610 con 0x7f37cc0b2390 2023-10-10T18:53:42.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc0c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:42.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc0c5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56828/0 (socket says 172.21.15.175:56828) 2023-10-10T18:53:42.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 -- 172.21.15.175:0/612730284 learned_addr learned my addr 172.21.15.175:0/612730284 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:42.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d2bb9700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0b2390 0x7f37cc0c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:42.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d23b8700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37cc0aff60 0x7f37cc14de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:42.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 -- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0b2390 msgr2=0x7f37cc0c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:42.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0b2390 0x7f37cc0c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 -- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37cc0aff60 msgr2=0x7f37cc14de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:42.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37cc0aff60 0x7f37cc14de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 -- 172.21.15.175:0/612730284 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c80094f0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.093+0000 7f37d1bb7700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc0c5db0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00c4e0 tx=0x7f37bc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:42.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.094+0000 7f37c27fc700 1 -- 172.21.15.175:0/612730284 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc011590 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.094+0000 7f37c27fc700 1 -- 172.21.15.175:0/612730284 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc011710 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.094+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc1665b0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.094+0000 7f37c27fc700 1 -- 172.21.15.175:0/612730284 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc011ad0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.094+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc0c4bc0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.094+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc0c56f0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.095+0000 7f37c27fc700 1 -- 172.21.15.175:0/612730284 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f37bc01b3b0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.095+0000 7f37c27fc700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37b0066f00 0x7f37b00693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:42.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.096+0000 7f37c27fc700 1 -- 172.21.15.175:0/612730284 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f37bc083a70 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.096+0000 7f37d23b8700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37b0066f00 0x7f37b00693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:42.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.096+0000 7f37d23b8700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37b0066f00 0x7f37b00693b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f37c80099b0 tx=0x7f37c8017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:42.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.100+0000 7f37c27fc700 1 -- 172.21.15.175:0/612730284 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37cc0c56f0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.325+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37cc0c56f0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.326+0000 7f37c27fc700 1 -- 172.21.15.175:0/612730284 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5452 (secure 0 0 0) 0x7f37cc0c56f0 con 0x7f37cc0b1fc0 2023-10-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:39.650890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37b0066f00 msgr2=0x7f37b00693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37b0066f00 0x7f37b00693b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f37c80099b0 tx=0x7f37c8017000 comp rx=0 tx=0).stop 2023-10-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 msgr2=0x7f37cc0c5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc0c5db0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00c4e0 tx=0x7f37bc00c7f0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 shutdown_connections 2023-10-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37b0066f00 0x7f37b00693b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37cc0b2390 0x7f37cc0c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37cc0aff60 0x7f37cc14de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 --2- 172.21.15.175:0/612730284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc0b1fc0 0x7f37cc0c5db0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:42.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 >> 172.21.15.175:0/612730284 conn(0x7f37cc063230 msgr2=0x7f37cc0c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:42.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 shutdown_connections 2023-10-10T18:53:42.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:42.328+0000 7f37d461c700 1 -- 172.21.15.175:0/612730284 wait complete. 2023-10-10T18:53:42.331 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 71 2023-10-10T18:53:42.339 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:53:44.340 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:44.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.466+0000 7fa53c92b700 1 Processor -- start 2023-10-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.467+0000 7fa53c92b700 1 -- start start 2023-10-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.467+0000 7fa53c92b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa5380b1fc0 0x7fa538155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.467+0000 7fa53c92b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa5380b2390 0x7fa538068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.467+0000 7fa53c92b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5381574b0 con 0x7fa538068bd0 2023-10-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.467+0000 7fa53c92b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa538150770 con 0x7fa5380b2390 2023-10-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.467+0000 7fa53c92b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5380afad0 con 0x7fa5380b1fc0 2023-10-10T18:53:44.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa535d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa5380b2390 0x7fa538068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:44.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa535d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa5380b2390 0x7fa538068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56842/0 (socket says 172.21.15.175:56842) 2023-10-10T18:53:44.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa535d9b700 1 -- 172.21.15.175:0/4172096392 learned_addr learned my addr 172.21.15.175:0/4172096392 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:44.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa53659c700 1 --2- 172.21.15.175:0/4172096392 >> v2:172.21.15.191:3300/0 conn(0x7fa5380b1fc0 0x7fa538155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa535d9b700 1 -- 172.21.15.175:0/4172096392 >> v2:172.21.15.191:3300/0 conn(0x7fa5380b1fc0 msgr2=0x7fa538155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa535d9b700 1 --2- 172.21.15.175:0/4172096392 >> v2:172.21.15.191:3300/0 conn(0x7fa5380b1fc0 0x7fa538155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa535d9b700 1 -- 172.21.15.175:0/4172096392 >> v1:172.21.15.191:6789/0 conn(0x7fa538068bd0 legacy=0x7fa5380aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa535d9b700 1 -- 172.21.15.175:0/4172096392 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5380af4c0 con 0x7fa5380b2390 2023-10-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa535d9b700 1 --2- 172.21.15.175:0/4172096392 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5380b2390 0x7fa538068690 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa5380b28c0 tx=0x7fa52c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbced407a09233 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:44.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa53559a700 1 -- 172.21.15.175:0/4172096392 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c009e70 con 0x7fa5380b2390 2023-10-10T18:53:44.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa53559a700 1 -- 172.21.15.175:0/4172096392 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c00b8f0 con 0x7fa5380b2390 2023-10-10T18:53:44.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.468+0000 7fa53559a700 1 -- 172.21.15.175:0/4172096392 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c00bc90 con 0x7fa5380b2390 2023-10-10T18:53:44.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.469+0000 7fa53c92b700 1 -- 172.21.15.175:0/4172096392 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5380b2390 msgr2=0x7fa538068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:44.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.469+0000 7fa53c92b700 1 --2- 172.21.15.175:0/4172096392 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5380b2390 0x7fa538068690 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa5380b28c0 tx=0x7fa52c009250 comp rx=0 tx=0).stop 2023-10-10T18:53:44.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.469+0000 7fa53c92b700 1 -- 172.21.15.175:0/4172096392 shutdown_connections 2023-10-10T18:53:44.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.469+0000 7fa53c92b700 1 --2- 172.21.15.175:0/4172096392 >> v2:172.21.15.191:3300/0 conn(0x7fa5380b1fc0 0x7fa538155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.469+0000 7fa53c92b700 1 --2- 172.21.15.175:0/4172096392 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5380b2390 0x7fa538068690 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.469+0000 7fa53c92b700 1 -- 172.21.15.175:0/4172096392 >> 172.21.15.175:0/4172096392 conn(0x7fa538063230 msgr2=0x7fa538063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:44.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.469+0000 7fa53c92b700 1 -- 172.21.15.175:0/4172096392 shutdown_connections 2023-10-10T18:53:44.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.469+0000 7fa53c92b700 1 -- 172.21.15.175:0/4172096392 wait complete. 2023-10-10T18:53:44.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.470+0000 7fa53c92b700 1 Processor -- start 2023-10-10T18:53:44.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.470+0000 7fa53c92b700 1 -- start start 2023-10-10T18:53:44.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53c92b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa538068bd0 0x7fa538157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:44.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53c92b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5380b1fc0 0x7fa5380cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:44.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53c92b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5380b2390 0x7fa5380cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:44.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53c92b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5380cb6e0 con 0x7fa538068bd0 2023-10-10T18:53:44.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53c92b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5380cbf40 con 0x7fa5380b2390 2023-10-10T18:53:44.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53c92b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5380cab60 con 0x7fa5380b1fc0 2023-10-10T18:53:44.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa538068bd0 0x7fa538157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:44.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa538068bd0 0x7fa538157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56852/0 (socket says 172.21.15.175:56852) 2023-10-10T18:53:44.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 -- 172.21.15.175:0/2564604661 learned_addr learned my addr 172.21.15.175:0/2564604661 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:44.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa535d9b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5380b1fc0 0x7fa5380cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:44.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa536d9d700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5380b2390 0x7fa5380cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:44.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 -- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5380b1fc0 msgr2=0x7fa5380cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:44.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5380b1fc0 0x7fa5380cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 -- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5380b2390 msgr2=0x7fa5380cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:44.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5380b2390 0x7fa5380cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 -- 172.21.15.175:0/2564604661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c0094f0 con 0x7fa538068bd0 2023-10-10T18:53:44.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa535d9b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5380b1fc0 0x7fa5380cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:53:44.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa536d9d700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5380b2390 0x7fa5380cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:53:44.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53659c700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa538068bd0 0x7fa538157890 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa528009ef0 tx=0x7fa52800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:44.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa5267fc700 1 -- 172.21.15.175:0/2564604661 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52800cc50 con 0x7fa538068bd0 2023-10-10T18:53:44.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa5267fc700 1 -- 172.21.15.175:0/2564604661 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52800f040 con 0x7fa538068bd0 2023-10-10T18:53:44.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5380cae40 con 0x7fa538068bd0 2023-10-10T18:53:44.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa5267fc700 1 -- 172.21.15.175:0/2564604661 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528014480 con 0x7fa538068bd0 2023-10-10T18:53:44.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.471+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5380cb400 con 0x7fa538068bd0 2023-10-10T18:53:44.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.472+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5380c36a0 con 0x7fa538068bd0 2023-10-10T18:53:44.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.473+0000 7fa5267fc700 1 -- 172.21.15.175:0/2564604661 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa5280147f0 con 0x7fa538068bd0 2023-10-10T18:53:44.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.473+0000 7fa5267fc700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa518066f30 0x7fa5180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:44.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.473+0000 7fa5267fc700 1 -- 172.21.15.175:0/2564604661 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fa528085230 con 0x7fa538068bd0 2023-10-10T18:53:44.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.473+0000 7fa535d9b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa518066f30 0x7fa5180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:44.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.474+0000 7fa535d9b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa518066f30 0x7fa5180693e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa5380b28c0 tx=0x7fa52c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:44.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.477+0000 7fa5267fc700 1 -- 172.21.15.175:0/2564604661 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5280522d0 con 0x7fa538068bd0 2023-10-10T18:53:44.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.704+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5380cc100 con 0x7fa538068bd0 2023-10-10T18:53:44.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.705+0000 7fa5267fc700 1 -- 172.21.15.175:0/2564604661 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5452 (secure 0 0 0) 0x7fa5280558f0 con 0x7fa538068bd0 2023-10-10T18:53:44.705 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:44.705 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:39.650890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:44.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa518066f30 msgr2=0x7fa5180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:44.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa518066f30 0x7fa5180693e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa5380b28c0 tx=0x7fa52c017000 comp rx=0 tx=0).stop 2023-10-10T18:53:44.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa538068bd0 msgr2=0x7fa538157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:44.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa538068bd0 0x7fa538157890 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa528009ef0 tx=0x7fa52800b040 comp rx=0 tx=0).stop 2023-10-10T18:53:44.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 shutdown_connections 2023-10-10T18:53:44.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa518066f30 0x7fa5180693e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5380b2390 0x7fa5380cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa538068bd0 0x7fa538157890 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 --2- 172.21.15.175:0/2564604661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5380b1fc0 0x7fa5380cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:44.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 >> 172.21.15.175:0/2564604661 conn(0x7fa538063230 msgr2=0x7fa5380c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:44.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 shutdown_connections 2023-10-10T18:53:44.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:44.707+0000 7fa53c92b700 1 -- 172.21.15.175:0/2564604661 wait complete. 2023-10-10T18:53:44.709 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 71 2023-10-10T18:53:44.718 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:53:46.720 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:46.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.848+0000 7f809b364700 1 Processor -- start 2023-10-10T18:53:46.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.849+0000 7f809b364700 1 -- start start 2023-10-10T18:53:46.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.849+0000 7f809b364700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:46.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.849+0000 7f809b364700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f80940c4f20 0x7f80940c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:46.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.849+0000 7f809b364700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940c6fb0 con 0x7f80940c12d0 2023-10-10T18:53:46.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.849+0000 7f809b364700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8094063ea0 con 0x7f80940c4f20 2023-10-10T18:53:46.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.849+0000 7f809b364700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8094155f30 con 0x7f80940c5850 2023-10-10T18:53:46.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099901700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f80940c4f20 0x7f80940c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:46.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099901700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f80940c4f20 0x7f80940c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48838/0 (socket says 172.21.15.175:48838) 2023-10-10T18:53:46.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099901700 1 -- 172.21.15.175:0/4007674909 learned_addr learned my addr 172.21.15.175:0/4007674909 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:46.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f80988ff700 1 --1- 172.21.15.175:0/4007674909 >> v1:172.21.15.175:6789/0 conn(0x7f80940c12d0 0x7f8094156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54076/0 (socket says 172.21.15.175:54076) 2023-10-10T18:53:46.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099100700 1 --2- 172.21.15.175:0/4007674909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:46.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099901700 1 -- 172.21.15.175:0/4007674909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 msgr2=0x7f80940c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:46.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099901700 1 --2- 172.21.15.175:0/4007674909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:46.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099901700 1 -- 172.21.15.175:0/4007674909 >> v1:172.21.15.175:6789/0 conn(0x7f80940c12d0 legacy=0x7f8094156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:46.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099901700 1 -- 172.21.15.175:0/4007674909 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940c6160 con 0x7f80940c4f20 2023-10-10T18:53:46.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099100700 1 --2- 172.21.15.175:0/4007674909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:53:46.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f8099901700 1 --2- 172.21.15.175:0/4007674909 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80940c7370 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f808c009f20 tx=0x7f808c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f911b2dc0f1066 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:46.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f808bfff700 1 -- 172.21.15.175:0/4007674909 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c00e070 con 0x7f80940c4f20 2023-10-10T18:53:46.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f808bfff700 1 -- 172.21.15.175:0/4007674909 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c00bb10 con 0x7f80940c4f20 2023-10-10T18:53:46.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.850+0000 7f808bfff700 1 -- 172.21.15.175:0/4007674909 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c0092e0 con 0x7f80940c4f20 2023-10-10T18:53:46.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.851+0000 7f809b364700 1 -- 172.21.15.175:0/4007674909 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 msgr2=0x7f80940c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:46.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.851+0000 7f809b364700 1 --2- 172.21.15.175:0/4007674909 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80940c7370 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f808c009f20 tx=0x7f808c009840 comp rx=0 tx=0).stop 2023-10-10T18:53:46.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.851+0000 7f809b364700 1 -- 172.21.15.175:0/4007674909 shutdown_connections 2023-10-10T18:53:46.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.851+0000 7f809b364700 1 --2- 172.21.15.175:0/4007674909 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80940c7370 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:46.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.851+0000 7f809b364700 1 --2- 172.21.15.175:0/4007674909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:46.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.851+0000 7f809b364700 1 -- 172.21.15.175:0/4007674909 >> 172.21.15.175:0/4007674909 conn(0x7f809406dc70 msgr2=0x7f8094063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:46.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.851+0000 7f809b364700 1 -- 172.21.15.175:0/4007674909 shutdown_connections 2023-10-10T18:53:46.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.851+0000 7f809b364700 1 -- 172.21.15.175:0/4007674909 wait complete. 2023-10-10T18:53:46.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.852+0000 7f809b364700 1 Processor -- start 2023-10-10T18:53:46.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.852+0000 7f809b364700 1 -- start start 2023-10-10T18:53:46.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f809b364700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80940c12d0 0x7f8094152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:46.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f809b364700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80941507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:46.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f809b364700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:46.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f809b364700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80940c2a20 con 0x7f80940c4f20 2023-10-10T18:53:46.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f809b364700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8094150420 con 0x7f80940c12d0 2023-10-10T18:53:46.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f809b364700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809414fb80 con 0x7f80940c5850 2023-10-10T18:53:46.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80941507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:46.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80941507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48850/0 (socket says 172.21.15.175:48850) 2023-10-10T18:53:46.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 -- 172.21.15.175:0/3438292053 learned_addr learned my addr 172.21.15.175:0/3438292053 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:46.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f8099100700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80940c12d0 0x7f8094152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:46.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f8099901700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:46.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 -- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 msgr2=0x7f80940c2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:46.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:46.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 -- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80940c12d0 msgr2=0x7f8094152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:46.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80940c12d0 0x7f8094152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:46.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 -- 172.21.15.175:0/3438292053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808c009ae0 con 0x7f80940c4f20 2023-10-10T18:53:46.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f8099100700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80940c12d0 0x7f8094152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:53:46.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f8099901700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:53:46.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80988ff700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80941507a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f808400c560 tx=0x7f8084007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:46.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80897fa700 1 -- 172.21.15.175:0/3438292053 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084005a70 con 0x7f80940c4f20 2023-10-10T18:53:46.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80897fa700 1 -- 172.21.15.175:0/3438292053 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8084005bd0 con 0x7f80940c4f20 2023-10-10T18:53:46.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f80897fa700 1 -- 172.21.15.175:0/3438292053 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80840070d0 con 0x7f80940c4f20 2023-10-10T18:53:46.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.853+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809414fe60 con 0x7f80940c4f20 2023-10-10T18:53:46.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.854+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80940c26b0 con 0x7f80940c4f20 2023-10-10T18:53:46.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.855+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8094067570 con 0x7f80940c4f20 2023-10-10T18:53:46.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.855+0000 7f80897fa700 1 -- 172.21.15.175:0/3438292053 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8084007480 con 0x7f80940c4f20 2023-10-10T18:53:46.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.855+0000 7f80897fa700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8078066f60 0x7f8078069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:46.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.855+0000 7f80897fa700 1 -- 172.21.15.175:0/3438292053 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f8084083ca0 con 0x7f80940c4f20 2023-10-10T18:53:46.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.855+0000 7f8099100700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8078066f60 0x7f8078069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:46.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.856+0000 7f8099100700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8078066f60 0x7f8078069410 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8090006f00 tx=0x7f8090008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:46.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:46.859+0000 7f80897fa700 1 -- 172.21.15.175:0/3438292053 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8084050d40 con 0x7f80940c4f20 2023-10-10T18:53:47.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.083+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f809414f870 con 0x7f80940c4f20 2023-10-10T18:53:47.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.083+0000 7f80897fa700 1 -- 172.21.15.175:0/3438292053 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5454 (secure 0 0 0) 0x7f8084017070 con 0x7f80940c4f20 2023-10-10T18:53:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:47.084 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:45.663943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:47.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8078066f60 msgr2=0x7f8078069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:47.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8078066f60 0x7f8078069410 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8090006f00 tx=0x7f8090008040 comp rx=0 tx=0).stop 2023-10-10T18:53:47.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 msgr2=0x7f80941507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:47.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80941507a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f808400c560 tx=0x7f8084007f70 comp rx=0 tx=0).stop 2023-10-10T18:53:47.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 shutdown_connections 2023-10-10T18:53:47.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8078066f60 0x7f8078069410 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:47.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80940c5850 0x7f80940c2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:47.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80940c12d0 0x7f8094152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:47.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 --2- 172.21.15.175:0/3438292053 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80940c4f20 0x7f80941507a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:47.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 >> 172.21.15.175:0/3438292053 conn(0x7f809406dc70 msgr2=0x7f8094063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:47.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 shutdown_connections 2023-10-10T18:53:47.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:47.085+0000 7f809b364700 1 -- 172.21.15.175:0/3438292053 wait complete. 2023-10-10T18:53:47.088 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 72 2023-10-10T18:53:47.097 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 72, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 72, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4251": { "addr": "172.21.15.175:6837/2966961140", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 2966961140, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 2966961140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4251, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:53:45.663943+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 4251, "mds_2": 5367, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:53:49.099 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:49.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.228+0000 7f9c3db9c700 1 Processor -- start 2023-10-10T18:53:49.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.228+0000 7f9c3db9c700 1 -- start start 2023-10-10T18:53:49.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.228+0000 7f9c3db9c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c3806bf00 0x7f9c380c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.228+0000 7f9c3db9c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9c38156520 0x7f9c381569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.228+0000 7f9c3db9c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c38156f00 con 0x7f9c380c5e40 2023-10-10T18:53:49.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.228+0000 7f9c3db9c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c381560d0 con 0x7f9c38156520 2023-10-10T18:53:49.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.228+0000 7f9c3db9c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c380c6df0 con 0x7f9c3806bf00 2023-10-10T18:53:49.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c377fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9c380c5e40 0x7f9c380c6210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54080/0 (socket says 172.21.15.175:54080) 2023-10-10T18:53:49.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c377fe700 1 -- 172.21.15.175:0/2292310727 learned_addr learned my addr 172.21.15.175:0/2292310727 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:49.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c2effd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c3806bf00 0x7f9c380c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c37fff700 1 --2- 172.21.15.175:0/2292310727 >> v2:172.21.15.191:3300/0 conn(0x7f9c38156520 0x7f9c381569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618284900 0 0) 0x7f9c38156f00 con 0x7f9c380c5e40 2023-10-10T18:53:49.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c24003570 con 0x7f9c380c5e40 2023-10-10T18:53:49.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029562544 0 0) 0x7f9c24003570 con 0x7f9c380c5e40 2023-10-10T18:53:49.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c3806bf00 msgr2=0x7f9c380c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:49.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c3806bf00 0x7f9c380c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 >> v2:172.21.15.191:3300/0 conn(0x7f9c38156520 msgr2=0x7f9c381569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:49.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 --2- 172.21.15.175:0/2292310727 >> v2:172.21.15.191:3300/0 conn(0x7f9c38156520 0x7f9c381569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c380c6920 con 0x7f9c380c5e40 2023-10-10T18:53:49.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f9c280015f0 con 0x7f9c380c5e40 2023-10-10T18:53:49.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 >> v1:172.21.15.175:6789/0 conn(0x7f9c380c5e40 legacy=0x7f9c380c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:49.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c3806bf00 0x7f9c24003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c24003fe0 0x7f9c24004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24004d50 0x7f9c240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c381560d0 con 0x7f9c24003fe0 2023-10-10T18:53:49.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c380c6df0 con 0x7f9c24004d50 2023-10-10T18:53:49.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2400ba90 con 0x7f9c3806bf00 2023-10-10T18:53:49.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c2effd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c24003fe0 0x7f9c24004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c377fe700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c3806bf00 0x7f9c24003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.229+0000 7f9c37fff700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24004d50 0x7f9c240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c2effd700 1 -- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c3806bf00 msgr2=0x7f9c24003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:49.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c2effd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c3806bf00 0x7f9c24003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c2effd700 1 -- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24004d50 msgr2=0x7f9c240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:49.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c2effd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24004d50 0x7f9c240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c2effd700 1 -- 172.21.15.175:0/2292310727 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24003570 con 0x7f9c24003fe0 2023-10-10T18:53:49.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c2effd700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c24003fe0 0x7f9c24004810 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9c380c4d10 tx=0x7f9c200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40a75e0455b6ecc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:49.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2000e6b0 con 0x7f9c24003fe0 2023-10-10T18:53:49.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2000e810 con 0x7f9c24003fe0 2023-10-10T18:53:49.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c36ffd700 1 -- 172.21.15.175:0/2292310727 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2000ebd0 con 0x7f9c24003fe0 2023-10-10T18:53:49.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2292310727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c24003fe0 msgr2=0x7f9c24004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:49.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c3db9c700 1 --2- 172.21.15.175:0/2292310727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c24003fe0 0x7f9c24004810 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9c380c4d10 tx=0x7f9c200091a0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c37fff700 1 -- 172.21.15.175:0/2292310727 reap_dead start 2023-10-10T18:53:49.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2292310727 shutdown_connections 2023-10-10T18:53:49.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2292310727 >> 172.21.15.175:0/2292310727 conn(0x7f9c38068900 msgr2=0x7f9c38068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:49.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2292310727 shutdown_connections 2023-10-10T18:53:49.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.230+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2292310727 wait complete. 2023-10-10T18:53:49.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.231+0000 7f9c3db9c700 1 Processor -- start 2023-10-10T18:53:49.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.232+0000 7f9c3db9c700 1 -- start start 2023-10-10T18:53:49.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.232+0000 7f9c3db9c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24003fe0 0x7f9c38139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.232+0000 7f9c3db9c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c380c8b90 0x7f9c3813d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.232+0000 7f9c3db9c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c380c8530 0x7f9c3813dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.232+0000 7f9c3db9c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3813af20 con 0x7f9c380c8b90 2023-10-10T18:53:49.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.232+0000 7f9c3db9c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c380c0ff0 con 0x7f9c24003fe0 2023-10-10T18:53:49.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.232+0000 7f9c3db9c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38166da0 con 0x7f9c380c8530 2023-10-10T18:53:49.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.232+0000 7f9c2effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c380c8b90 0x7f9c3813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c380c8b90 0x7f9c3813d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48864/0 (socket says 172.21.15.175:48864) 2023-10-10T18:53:49.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2effd700 1 -- 172.21.15.175:0/2387757059 learned_addr learned my addr 172.21.15.175:0/2387757059 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:49.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c37fff700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c380c8530 0x7f9c3813dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c377fe700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24003fe0 0x7f9c38139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2effd700 1 -- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c380c8530 msgr2=0x7f9c3813dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:49.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2effd700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c380c8530 0x7f9c3813dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2effd700 1 -- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24003fe0 msgr2=0x7f9c38139c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:49.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2effd700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24003fe0 0x7f9c38139c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2effd700 1 -- 172.21.15.175:0/2387757059 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3813a980 con 0x7f9c380c8b90 2023-10-10T18:53:49.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2effd700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c380c8b90 0x7f9c3813d7a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9c200091e0 tx=0x7f9c2001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:49.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2ffff700 1 -- 172.21.15.175:0/2387757059 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20023070 con 0x7f9c380c8b90 2023-10-10T18:53:49.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2ffff700 1 -- 172.21.15.175:0/2387757059 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2000e980 con 0x7f9c380c8b90 2023-10-10T18:53:49.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c38167040 con 0x7f9c380c8b90 2023-10-10T18:53:49.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c2ffff700 1 -- 172.21.15.175:0/2387757059 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2000e6b0 con 0x7f9c380c8b90 2023-10-10T18:53:49.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.233+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c380c0c80 con 0x7f9c380c8b90 2023-10-10T18:53:49.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.234+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c380c0ae0 con 0x7f9c380c8b90 2023-10-10T18:53:49.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.234+0000 7f9c2ffff700 1 -- 172.21.15.175:0/2387757059 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9c20013470 con 0x7f9c380c8b90 2023-10-10T18:53:49.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.235+0000 7f9c2ffff700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c18066f60 0x7f9c18069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:49.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.235+0000 7f9c2ffff700 1 -- 172.21.15.175:0/2387757059 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f9c20085b80 con 0x7f9c380c8b90 2023-10-10T18:53:49.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.235+0000 7f9c377fe700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c18066f60 0x7f9c18069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:49.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.235+0000 7f9c377fe700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c18066f60 0x7f9c18069410 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9c28001360 tx=0x7f9c28009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:49.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.239+0000 7f9c2ffff700 1 -- 172.21.15.175:0/2387757059 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c380c0ae0 con 0x7f9c380c8b90 2023-10-10T18:53:49.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:53:49.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:53:49.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.464+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c380c11a0 con 0x7f9c380c8b90 2023-10-10T18:53:49.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.465+0000 7f9c2ffff700 1 -- 172.21.15.175:0/2387757059 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5450 (secure 0 0 0) 0x7f9c380c11a0 con 0x7f9c380c8b90 2023-10-10T18:53:49.465 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:49.465 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:47.673068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c18066f60 msgr2=0x7f9c18069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c18066f60 0x7f9c18069410 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9c28001360 tx=0x7f9c28009040 comp rx=0 tx=0).stop 2023-10-10T18:53:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c380c8b90 msgr2=0x7f9c3813d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c380c8b90 0x7f9c3813d7a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9c200091e0 tx=0x7f9c2001a000 comp rx=0 tx=0).stop 2023-10-10T18:53:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 shutdown_connections 2023-10-10T18:53:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c18066f60 0x7f9c18069410 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c380c8530 0x7f9c3813dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c24003fe0 0x7f9c38139c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 --2- 172.21.15.175:0/2387757059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c380c8b90 0x7f9c3813d7a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 >> 172.21.15.175:0/2387757059 conn(0x7f9c38068900 msgr2=0x7f9c3806f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 shutdown_connections 2023-10-10T18:53:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:49.468+0000 7f9c3db9c700 1 -- 172.21.15.175:0/2387757059 wait complete. 2023-10-10T18:53:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 73 2023-10-10T18:53:49.482 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:53:51.484 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:51.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.614+0000 7f9e30e38700 1 Processor -- start 2023-10-10T18:53:51.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.614+0000 7f9e30e38700 1 -- start start 2023-10-10T18:53:51.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e30e38700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9e2c14cdc0 0x7f9e2c0c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:51.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e30e38700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9e2c14d190 0x7f9e2c0c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:51.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e30e38700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 0x7f9e2c156300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:51.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e30e38700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2c06e940 con 0x7f9e2c14d190 2023-10-10T18:53:51.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e30e38700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2c0c1c90 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e30e38700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c0c6e20 con 0x7f9e2c06f620 2023-10-10T18:53:51.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e29d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9e2c14d190 0x7f9e2c0c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:51.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e29d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9e2c14d190 0x7f9e2c0c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48872/0 (socket says 172.21.15.175:48872) 2023-10-10T18:53:51.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e29d9b700 1 -- 172.21.15.175:0/57584395 learned_addr learned my addr 172.21.15.175:0/57584395 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:51.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e2a59c700 1 --2- 172.21.15.175:0/57584395 >> v2:172.21.15.191:3300/0 conn(0x7f9e2c14cdc0 0x7f9e2c0c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:51.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e2ad9d700 1 --2- 172.21.15.175:0/57584395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 0x7f9e2c156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:51.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e29d9b700 1 -- 172.21.15.175:0/57584395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 msgr2=0x7f9e2c156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:51.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e29d9b700 1 --2- 172.21.15.175:0/57584395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 0x7f9e2c156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e29d9b700 1 -- 172.21.15.175:0/57584395 >> v2:172.21.15.191:3300/0 conn(0x7f9e2c14cdc0 msgr2=0x7f9e2c0c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:51.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e29d9b700 1 --2- 172.21.15.175:0/57584395 >> v2:172.21.15.191:3300/0 conn(0x7f9e2c14cdc0 0x7f9e2c0c26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.615+0000 7f9e29d9b700 1 -- 172.21.15.175:0/57584395 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c06ca40 con 0x7f9e2c14d190 2023-10-10T18:53:51.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e29d9b700 1 --2- 172.21.15.175:0/57584395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14d190 0x7f9e2c0c2c20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000bc50 tx=0x7f9e2000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb6cb49eae11654 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:51.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e2959a700 1 -- 172.21.15.175:0/57584395 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000db00 con 0x7f9e2c14d190 2023-10-10T18:53:51.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e2959a700 1 -- 172.21.15.175:0/57584395 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2000dc60 con 0x7f9e2c14d190 2023-10-10T18:53:51.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e2959a700 1 -- 172.21.15.175:0/57584395 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e20019480 con 0x7f9e2c14d190 2023-10-10T18:53:51.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 -- 172.21.15.175:0/57584395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14d190 msgr2=0x7f9e2c0c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:51.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 --2- 172.21.15.175:0/57584395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14d190 0x7f9e2c0c2c20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000bc50 tx=0x7f9e2000b4f0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 -- 172.21.15.175:0/57584395 shutdown_connections 2023-10-10T18:53:51.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 --2- 172.21.15.175:0/57584395 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 0x7f9e2c156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 --2- 172.21.15.175:0/57584395 >> v2:172.21.15.191:3300/0 conn(0x7f9e2c14cdc0 0x7f9e2c0c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 --2- 172.21.15.175:0/57584395 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14d190 0x7f9e2c0c2c20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 -- 172.21.15.175:0/57584395 >> 172.21.15.175:0/57584395 conn(0x7f9e2c068900 msgr2=0x7f9e2c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:51.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 -- 172.21.15.175:0/57584395 shutdown_connections 2023-10-10T18:53:51.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.616+0000 7f9e30e38700 1 -- 172.21.15.175:0/57584395 wait complete. 2023-10-10T18:53:51.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.617+0000 7f9e30e38700 1 Processor -- start 2023-10-10T18:53:51.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.617+0000 7f9e30e38700 1 -- start start 2023-10-10T18:53:51.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e30e38700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 0x7f9e2c1502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:51.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e30e38700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14cdc0 0x7f9e2c168de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:51.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e30e38700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e2c14d190 0x7f9e2c163da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:51.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e30e38700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c168950 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e30e38700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c16a100 con 0x7f9e2c14d190 2023-10-10T18:53:51.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e30e38700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c168020 con 0x7f9e2c06f620 2023-10-10T18:53:51.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e29d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14cdc0 0x7f9e2c168de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:51.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e29d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14cdc0 0x7f9e2c168de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48876/0 (socket says 172.21.15.175:48876) 2023-10-10T18:53:51.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e29d9b700 1 -- 172.21.15.175:0/1629739412 learned_addr learned my addr 172.21.15.175:0/1629739412 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:51.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e2a59c700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 0x7f9e2c1502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:51.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.618+0000 7f9e2ad9d700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e2c14d190 0x7f9e2c163da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:51.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e29d9b700 1 -- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 msgr2=0x7f9e2c1502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:51.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e29d9b700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 0x7f9e2c1502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e29d9b700 1 -- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e2c14d190 msgr2=0x7f9e2c163da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:51.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e29d9b700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e2c14d190 0x7f9e2c163da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e29d9b700 1 -- 172.21.15.175:0/1629739412 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2000b790 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e29d9b700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14cdc0 0x7f9e2c168de0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000b760 tx=0x7f9e2001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:51.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e1a7fc700 1 -- 172.21.15.175:0/1629739412 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e20010680 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e1a7fc700 1 -- 172.21.15.175:0/1629739412 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2001e040 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c168300 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e1a7fc700 1 -- 172.21.15.175:0/1629739412 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e200109f0 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.619+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2c153b60 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.620+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2c0c63a0 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.620+0000 7f9e1a7fc700 1 -- 172.21.15.175:0/1629739412 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9e200183e0 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.621+0000 7f9e1a7fc700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9e0c066f60 0x7f9e0c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:51.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.621+0000 7f9e1a7fc700 1 -- 172.21.15.175:0/1629739412 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f9e20014070 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.621+0000 7f9e2a59c700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9e0c066f60 0x7f9e0c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:51.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.621+0000 7f9e2a59c700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9e0c066f60 0x7f9e0c069410 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009da0 tx=0x7f9e1c009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:51.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.625+0000 7f9e1a7fc700 1 -- 172.21.15.175:0/1629739412 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e20053010 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.851+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e2c067570 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.852+0000 7f9e1a7fc700 1 -- 172.21.15.175:0/1629739412 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5447 (secure 0 0 0) 0x7f9e20027090 con 0x7f9e2c14cdc0 2023-10-10T18:53:51.852 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:51.852 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:50.687188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:51.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9e0c066f60 msgr2=0x7f9e0c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:51.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9e0c066f60 0x7f9e0c069410 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009da0 tx=0x7f9e1c009120 comp rx=0 tx=0).stop 2023-10-10T18:53:51.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14cdc0 msgr2=0x7f9e2c168de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:51.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14cdc0 0x7f9e2c168de0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000b760 tx=0x7f9e2001b000 comp rx=0 tx=0).stop 2023-10-10T18:53:51.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 shutdown_connections 2023-10-10T18:53:51.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9e0c066f60 0x7f9e0c069410 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e2c14d190 0x7f9e2c163da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2c06f620 0x7f9e2c1502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 --2- 172.21.15.175:0/1629739412 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9e2c14cdc0 0x7f9e2c168de0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:51.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 >> 172.21.15.175:0/1629739412 conn(0x7f9e2c068900 msgr2=0x7f9e2c0c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:51.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 shutdown_connections 2023-10-10T18:53:51.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:51.854+0000 7f9e30e38700 1 -- 172.21.15.175:0/1629739412 wait complete. 2023-10-10T18:53:51.857 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 74 2023-10-10T18:53:51.866 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:53:53.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:53.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.994+0000 7f5cd1653700 1 Processor -- start 2023-10-10T18:53:53.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.994+0000 7f5cd1653700 1 -- start start 2023-10-10T18:53:53.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.995+0000 7f5cd1653700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5ccc0c35e0 0x7f5ccc159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:53.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.995+0000 7f5cd1653700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5ccc0c9390 0x7f5ccc0c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:53.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.995+0000 7f5cd1653700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0c5e40 con 0x7f5ccc0c3210 2023-10-10T18:53:53.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.995+0000 7f5cd1653700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc0c4c50 con 0x7f5ccc0c9390 2023-10-10T18:53:53.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.995+0000 7f5cd1653700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc0ca400 con 0x7f5ccc0c35e0 2023-10-10T18:53:53.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.995+0000 7f5ccb7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5ccc0c9390 0x7f5ccc0c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:53.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.995+0000 7f5ccb7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5ccc0c9390 0x7f5ccc0c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48880/0 (socket says 172.21.15.175:48880) 2023-10-10T18:53:53.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.995+0000 7f5ccb7fe700 1 -- 172.21.15.175:0/1527004774 learned_addr learned my addr 172.21.15.175:0/1527004774 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:53.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5cc27fc700 1 --2- 172.21.15.175:0/1527004774 >> v2:172.21.15.191:3300/0 conn(0x7f5ccc0c35e0 0x7f5ccc159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:53.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5ccb7fe700 1 -- 172.21.15.175:0/1527004774 >> v2:172.21.15.191:3300/0 conn(0x7f5ccc0c35e0 msgr2=0x7f5ccc159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:53.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5ccb7fe700 1 --2- 172.21.15.175:0/1527004774 >> v2:172.21.15.191:3300/0 conn(0x7f5ccc0c35e0 0x7f5ccc159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5ccb7fe700 1 -- 172.21.15.175:0/1527004774 >> v1:172.21.15.191:6789/0 conn(0x7f5ccc0c3210 legacy=0x7f5ccc0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:54.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5ccb7fe700 1 -- 172.21.15.175:0/1527004774 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0c9880 con 0x7f5ccc0c9390 2023-10-10T18:53:54.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5ccb7fe700 1 --2- 172.21.15.175:0/1527004774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc0c62d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc064890 tx=0x7f5cc4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e295cb6e28d8b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:54.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5cca7fc700 1 -- 172.21.15.175:0/1527004774 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc400e070 con 0x7f5ccc0c9390 2023-10-10T18:53:54.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5cca7fc700 1 -- 172.21.15.175:0/1527004774 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc400ba40 con 0x7f5ccc0c9390 2023-10-10T18:53:54.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.996+0000 7f5cca7fc700 1 -- 172.21.15.175:0/1527004774 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40092e0 con 0x7f5ccc0c9390 2023-10-10T18:53:54.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.997+0000 7f5cd1653700 1 -- 172.21.15.175:0/1527004774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 msgr2=0x7f5ccc0c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:54.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.997+0000 7f5cd1653700 1 --2- 172.21.15.175:0/1527004774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc0c62d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc064890 tx=0x7f5cc4009840 comp rx=0 tx=0).stop 2023-10-10T18:53:54.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.997+0000 7f5cd1653700 1 -- 172.21.15.175:0/1527004774 shutdown_connections 2023-10-10T18:53:54.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.997+0000 7f5cd1653700 1 --2- 172.21.15.175:0/1527004774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc0c62d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.997+0000 7f5cd1653700 1 --2- 172.21.15.175:0/1527004774 >> v2:172.21.15.191:3300/0 conn(0x7f5ccc0c35e0 0x7f5ccc159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.997+0000 7f5cd1653700 1 -- 172.21.15.175:0/1527004774 >> 172.21.15.175:0/1527004774 conn(0x7f5ccc06c110 msgr2=0x7f5ccc06c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:54.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.997+0000 7f5cd1653700 1 -- 172.21.15.175:0/1527004774 shutdown_connections 2023-10-10T18:53:54.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.997+0000 7f5cd1653700 1 -- 172.21.15.175:0/1527004774 wait complete. 2023-10-10T18:53:54.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.998+0000 7f5cd1653700 1 Processor -- start 2023-10-10T18:53:54.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.998+0000 7f5cd1653700 1 -- start start 2023-10-10T18:53:54.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cd1653700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ccc0c3210 0x7f5ccc153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:54.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cd1653700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ccc0c35e0 0x7f5ccc06c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:54.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cd1653700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:54.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cd1653700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0c5840 con 0x7f5ccc0c9390 2023-10-10T18:53:54.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cd1653700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc1563b0 con 0x7f5ccc0c3210 2023-10-10T18:53:54.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cd1653700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc155af0 con 0x7f5ccc0c35e0 2023-10-10T18:53:54.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:54.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48896/0 (socket says 172.21.15.175:48896) 2023-10-10T18:53:54.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 -- 172.21.15.175:0/3641210313 learned_addr learned my addr 172.21.15.175:0/3641210313 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:54.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccaffd700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ccc0c3210 0x7f5ccc153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:54.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 -- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ccc0c35e0 msgr2=0x7f5ccc06c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:54.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ccc0c35e0 0x7f5ccc06c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 -- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ccc0c3210 msgr2=0x7f5ccc153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:54.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ccc0c3210 0x7f5ccc153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 -- 172.21.15.175:0/3641210313 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc4009ae0 con 0x7f5ccc0c9390 2023-10-10T18:53:54.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5ccb7fe700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc158c50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4012000 tx=0x7f5cc4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:54.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cc37fe700 1 -- 172.21.15.175:0/3641210313 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4008e40 con 0x7f5ccc0c9390 2023-10-10T18:53:54.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cc37fe700 1 -- 172.21.15.175:0/3641210313 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc400be50 con 0x7f5ccc0c9390 2023-10-10T18:53:54.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc156f70 con 0x7f5ccc0c9390 2023-10-10T18:53:54.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cc37fe700 1 -- 172.21.15.175:0/3641210313 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc401c640 con 0x7f5ccc0c9390 2023-10-10T18:53:54.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:53.999+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc155d60 con 0x7f5ccc0c9390 2023-10-10T18:53:54.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.000+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc156560 con 0x7f5ccc0c9390 2023-10-10T18:53:54.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.001+0000 7f5cc37fe700 1 -- 172.21.15.175:0/3641210313 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5cc4003040 con 0x7f5ccc0c9390 2023-10-10T18:53:54.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.001+0000 7f5cc37fe700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5cac066f30 0x7f5cac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:54.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.001+0000 7f5cc37fe700 1 -- 172.21.15.175:0/3641210313 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f5cc4086030 con 0x7f5ccc0c9390 2023-10-10T18:53:54.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.001+0000 7f5ccaffd700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5cac066f30 0x7f5cac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:54.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.002+0000 7f5ccaffd700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5cac066f30 0x7f5cac0693e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc001760 tx=0x7f5cbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:54.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.005+0000 7f5cc37fe700 1 -- 172.21.15.175:0/3641210313 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ccc156560 con 0x7f5ccc0c9390 2023-10-10T18:53:54.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.232+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ccc156560 con 0x7f5ccc0c9390 2023-10-10T18:53:54.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.233+0000 7f5cc37fe700 1 -- 172.21.15.175:0/3641210313 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5447 (secure 0 0 0) 0x7f5ccc156560 con 0x7f5ccc0c9390 2023-10-10T18:53:54.233 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:54.233 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:50.687188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5cac066f30 msgr2=0x7f5cac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5cac066f30 0x7f5cac0693e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc001760 tx=0x7f5cbc009040 comp rx=0 tx=0).stop 2023-10-10T18:53:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 msgr2=0x7f5ccc158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc158c50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4012000 tx=0x7f5cc4018000 comp rx=0 tx=0).stop 2023-10-10T18:53:54.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 shutdown_connections 2023-10-10T18:53:54.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5cac066f30 0x7f5cac0693e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ccc0c9390 0x7f5ccc158c50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ccc0c3210 0x7f5ccc153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 --2- 172.21.15.175:0/3641210313 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ccc0c35e0 0x7f5ccc06c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:54.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 >> 172.21.15.175:0/3641210313 conn(0x7f5ccc06c110 msgr2=0x7f5ccc0c8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:54.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 shutdown_connections 2023-10-10T18:53:54.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:54.235+0000 7f5cd1653700 1 -- 172.21.15.175:0/3641210313 wait complete. 2023-10-10T18:53:54.238 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 74 2023-10-10T18:53:54.247 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:53:56.248 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:56.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.376+0000 7f15bfbe1700 1 Processor -- start 2023-10-10T18:53:56.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.376+0000 7f15bfbe1700 1 -- start start 2023-10-10T18:53:56.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15bfbe1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80b1fc0 0x7f15b8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:56.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15bfbe1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f15b80aff60 0x7f15b80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:56.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15bfbe1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b81574b0 con 0x7f15b80b2390 2023-10-10T18:53:56.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15bfbe1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b8150770 con 0x7f15b80aff60 2023-10-10T18:53:56.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15bfbe1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80afad0 con 0x7f15b80b1fc0 2023-10-10T18:53:56.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15bd97d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80b1fc0 0x7f15b8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:56.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15be17e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f15b80aff60 0x7f15b80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:56.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15bd17c700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f15b80b2390 0x7f15b8068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:57020/0 (socket says 172.21.15.175:57020) 2023-10-10T18:53:56.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.377+0000 7f15bd17c700 1 -- 172.21.15.175:0/1244326723 learned_addr learned my addr 172.21.15.175:0/1244326723 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:56.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bc97b700 1 -- 172.21.15.175:0/1244326723 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733298000 0 0) 0x7f15b81574b0 con 0x7f15b80b2390 2023-10-10T18:53:56.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bc97b700 1 -- 172.21.15.175:0/1244326723 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a4003570 con 0x7f15b80b2390 2023-10-10T18:53:56.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bd97d700 1 -- 172.21.15.175:0/1244326723 >> v2:172.21.15.191:3300/0 conn(0x7f15b80aff60 msgr2=0x7f15b80b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:56.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bd97d700 1 --2- 172.21.15.175:0/1244326723 >> v2:172.21.15.191:3300/0 conn(0x7f15b80aff60 0x7f15b80b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bd97d700 1 -- 172.21.15.175:0/1244326723 >> v1:172.21.15.191:6789/0 conn(0x7f15b80b2390 legacy=0x7f15b8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:56.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bd97d700 1 -- 172.21.15.175:0/1244326723 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b80af4c0 con 0x7f15b80b1fc0 2023-10-10T18:53:56.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15be17e700 1 --2- 172.21.15.175:0/1244326723 >> v2:172.21.15.191:3300/0 conn(0x7f15b80aff60 0x7f15b80b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:53:56.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bd97d700 1 --2- 172.21.15.175:0/1244326723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80b1fc0 0x7f15b8155fb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009910 tx=0x7f15b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39dfff745a7933d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:56.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bc97b700 1 -- 172.21.15.175:0/1244326723 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b4011630 con 0x7f15b80b1fc0 2023-10-10T18:53:56.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bc97b700 1 -- 172.21.15.175:0/1244326723 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b4011790 con 0x7f15b80b1fc0 2023-10-10T18:53:56.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.378+0000 7f15bc97b700 1 -- 172.21.15.175:0/1244326723 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b4011b10 con 0x7f15b80b1fc0 2023-10-10T18:53:56.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.379+0000 7f15bfbe1700 1 -- 172.21.15.175:0/1244326723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80b1fc0 msgr2=0x7f15b8155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:56.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.379+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/1244326723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80b1fc0 0x7f15b8155fb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009910 tx=0x7f15b40091a0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.379+0000 7f15bfbe1700 1 -- 172.21.15.175:0/1244326723 shutdown_connections 2023-10-10T18:53:56.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.379+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/1244326723 >> v2:172.21.15.191:3300/0 conn(0x7f15b80aff60 0x7f15b80b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.379+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/1244326723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80b1fc0 0x7f15b8155fb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.379+0000 7f15bfbe1700 1 -- 172.21.15.175:0/1244326723 >> 172.21.15.175:0/1244326723 conn(0x7f15b8063230 msgr2=0x7f15b8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:56.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.379+0000 7f15bfbe1700 1 -- 172.21.15.175:0/1244326723 shutdown_connections 2023-10-10T18:53:56.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.379+0000 7f15bfbe1700 1 -- 172.21.15.175:0/1244326723 wait complete. 2023-10-10T18:53:56.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.380+0000 7f15bfbe1700 1 Processor -- start 2023-10-10T18:53:56.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.380+0000 7f15bfbe1700 1 -- start start 2023-10-10T18:53:56.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15bfbe1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80aff60 0x7f15b8157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:56.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15bfbe1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15b80b1fc0 0x7f15b81690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:56.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15bfbe1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15b80b2390 0x7f15b81695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:56.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15bfbe1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80cd0e0 con 0x7f15b80b2390 2023-10-10T18:53:56.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15bfbe1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8159960 con 0x7f15b80b1fc0 2023-10-10T18:53:56.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15bfbe1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8158d90 con 0x7f15b80aff60 2023-10-10T18:53:56.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15b80b2390 0x7f15b81695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:56.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15b80b2390 0x7f15b81695e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39706/0 (socket says 172.21.15.175:39706) 2023-10-10T18:53:56.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 -- 172.21.15.175:0/2392780041 learned_addr learned my addr 172.21.15.175:0/2392780041 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:56.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15bd97d700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80aff60 0x7f15b8157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:56.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15bd17c700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15b80b1fc0 0x7f15b81690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:56.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 -- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80aff60 msgr2=0x7f15b8157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:56.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80aff60 0x7f15b8157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 -- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15b80b1fc0 msgr2=0x7f15b81690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:56.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15b80b1fc0 0x7f15b81690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 -- 172.21.15.175:0/2392780041 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b0009a20 con 0x7f15b80b2390 2023-10-10T18:53:56.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15be17e700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15b80b2390 0x7f15b81695e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009fc0 tx=0x7f15b000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:56.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.381+0000 7f15adffb700 1 -- 172.21.15.175:0/2392780041 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b000cbd0 con 0x7f15b80b2390 2023-10-10T18:53:56.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.382+0000 7f15adffb700 1 -- 172.21.15.175:0/2392780041 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b000f040 con 0x7f15b80b2390 2023-10-10T18:53:56.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.382+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b4009420 con 0x7f15b80b2390 2023-10-10T18:53:56.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.382+0000 7f15adffb700 1 -- 172.21.15.175:0/2392780041 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b0014480 con 0x7f15b80b2390 2023-10-10T18:53:56.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.382+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b8159310 con 0x7f15b80b2390 2023-10-10T18:53:56.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.383+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b8159b10 con 0x7f15b80b2390 2023-10-10T18:53:56.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.383+0000 7f15adffb700 1 -- 172.21.15.175:0/2392780041 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f15b0003020 con 0x7f15b80b2390 2023-10-10T18:53:56.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.383+0000 7f15adffb700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f159c066f30 0x7f159c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:56.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.383+0000 7f15adffb700 1 -- 172.21.15.175:0/2392780041 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f15b0085060 con 0x7f15b80b2390 2023-10-10T18:53:56.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.383+0000 7f15bd97d700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f159c066f30 0x7f159c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:56.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.384+0000 7f15bd97d700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f159c066f30 0x7f159c0693e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f15b40098e0 tx=0x7f15b4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:56.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.387+0000 7f15adffb700 1 -- 172.21.15.175:0/2392780041 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f15b0052100 con 0x7f15b80b2390 2023-10-10T18:53:56.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.614+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15b8067570 con 0x7f15b80b2390 2023-10-10T18:53:56.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.614+0000 7f15adffb700 1 -- 172.21.15.175:0/2392780041 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5447 (secure 0 0 0) 0x7f15b0055720 con 0x7f15b80b2390 2023-10-10T18:53:56.615 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:56.615 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:50.687188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:56.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f159c066f30 msgr2=0x7f159c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:56.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f159c066f30 0x7f159c0693e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f15b40098e0 tx=0x7f15b4017000 comp rx=0 tx=0).stop 2023-10-10T18:53:56.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15b80b2390 msgr2=0x7f15b81695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:56.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15b80b2390 0x7f15b81695e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009fc0 tx=0x7f15b000b040 comp rx=0 tx=0).stop 2023-10-10T18:53:56.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 shutdown_connections 2023-10-10T18:53:56.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f159c066f30 0x7f159c0693e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15b80b2390 0x7f15b81695e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15b80aff60 0x7f15b8157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 --2- 172.21.15.175:0/2392780041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15b80b1fc0 0x7f15b81690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:56.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 >> 172.21.15.175:0/2392780041 conn(0x7f15b8063230 msgr2=0x7f15b815a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:56.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 shutdown_connections 2023-10-10T18:53:56.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:56.616+0000 7f15bfbe1700 1 -- 172.21.15.175:0/2392780041 wait complete. 2023-10-10T18:53:56.619 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 74 2023-10-10T18:53:56.628 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:53:56.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/12' 2023-10-10T18:53:56.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/10' 2023-10-10T18:53:56.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/11' 2023-10-10T18:53:56.658 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/16' 2023-10-10T18:53:56.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/14' 2023-10-10T18:53:56.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/18' 2023-10-10T18:53:56.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/13' 2023-10-10T18:53:56.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/139/17' 2023-10-10T18:53:56.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/139' 2023-10-10T18:53:56.659 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/19' 2023-10-10T18:53:56.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/15' 2023-10-10T18:53:56.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/20' 2023-10-10T18:53:56.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/12' 2023-10-10T18:53:56.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/10' 2023-10-10T18:53:56.660 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/11' 2023-10-10T18:53:56.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/16' 2023-10-10T18:53:56.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/14' 2023-10-10T18:53:56.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/18' 2023-10-10T18:53:56.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/13' 2023-10-10T18:53:56.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/102/17' 2023-10-10T18:53:56.661 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/102' 2023-10-10T18:53:56.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/19' 2023-10-10T18:53:56.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/15' 2023-10-10T18:53:56.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/20' 2023-10-10T18:53:56.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/12' 2023-10-10T18:53:56.662 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/10' 2023-10-10T18:53:56.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/11' 2023-10-10T18:53:56.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/16' 2023-10-10T18:53:56.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/14' 2023-10-10T18:53:56.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/18' 2023-10-10T18:53:56.663 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/13' 2023-10-10T18:53:56.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed './copyofsnap1/test1/129/17' 2023-10-10T18:53:56.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1/129' 2023-10-10T18:53:56.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1/test1' 2023-10-10T18:53:56.664 INFO:tasks.workunit.client.0.smithi175.stdout:removed directory './copyofsnap1' 2023-10-10T18:53:56.664 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T18:53:56.665 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T18:53:56.665 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T18:53:56.742 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-10-10T18:53:56.742 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-10-10T18:53:56.808 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T18:53:56.809 INFO:tasks.workunit.client.0.smithi175.stderr:+ touch foo 2023-10-10T18:53:56.812 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod +x foo 2023-10-10T18:53:56.812 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T18:53:58.629 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:53:58.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.758+0000 7f3575906700 1 Processor -- start 2023-10-10T18:53:58.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.758+0000 7f3575906700 1 -- start start 2023-10-10T18:53:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f3575906700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f35700c5c50 0x7f3570155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f3575906700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3570068280 con 0x7f357006dd70 2023-10-10T18:53:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f3575906700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3570159b70 con 0x7f3570063230 2023-10-10T18:53:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f3575906700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f357015a0d0 con 0x7f35700c5c50 2023-10-10T18:53:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f356effd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f357006dd70 0x7f3570068710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:43620/0 (socket says 172.21.15.175:43620) 2023-10-10T18:53:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f356effd700 1 -- 172.21.15.175:0/525511464 learned_addr learned my addr 172.21.15.175:0/525511464 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:53:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139465363 0 0) 0x7f3570068280 con 0x7f357006dd70 2023-10-10T18:53:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f356e7fc700 1 --2- 172.21.15.175:0/525511464 >> v2:172.21.15.191:3300/0 conn(0x7f35700c5c50 0x7f3570155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.759+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3558003570 con 0x7f357006dd70 2023-10-10T18:53:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426124288 0 0) 0x7f3558003570 con 0x7f357006dd70 2023-10-10T18:53:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 >> v2:172.21.15.191:3300/0 conn(0x7f35700c5c50 msgr2=0x7f3570155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 --2- 172.21.15.175:0/525511464 >> v2:172.21.15.191:3300/0 conn(0x7f35700c5c50 0x7f3570155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 >> v1:172.21.15.191:6789/0 conn(0x7f3570063230 legacy=0x7f3570156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570067990 con 0x7f357006dd70 2023-10-10T18:53:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f3568001470 con 0x7f357006dd70 2023-10-10T18:53:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 >> v1:172.21.15.175:6789/0 conn(0x7f357006dd70 legacy=0x7f3570068710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3558003a90 0x7f3558003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3558004810 0x7f3558004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558005210 0x7f355800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357015a0d0 con 0x7f3558003a90 2023-10-10T18:53:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355800c130 con 0x7f3558004810 2023-10-10T18:53:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355800c2c0 con 0x7f3558005210 2023-10-10T18:53:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356effd700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3558003a90 0x7f3558003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356e7fc700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3558004810 0x7f3558004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356f7fe700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558005210 0x7f355800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356effd700 1 -- 172.21.15.175:0/525511464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558005210 msgr2=0x7f355800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:58.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356effd700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558005210 0x7f355800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:58.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356effd700 1 -- 172.21.15.175:0/525511464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3558004810 msgr2=0x7f3558004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:53:58.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356effd700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3558004810 0x7f3558004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:58.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356effd700 1 -- 172.21.15.175:0/525511464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3558003570 con 0x7f3558003a90 2023-10-10T18:53:58.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356f7fe700 1 -- 172.21.15.175:0/525511464 reap_dead start 2023-10-10T18:53:58.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356effd700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3558003a90 0x7f3558003ea0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f356800c3c0 tx=0x7f3570067b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d541f2fc646addba server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:58.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356800cc50 con 0x7f3558003a90 2023-10-10T18:53:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356800cdd0 con 0x7f3558003a90 2023-10-10T18:53:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.760+0000 7f356dffb700 1 -- 172.21.15.175:0/525511464 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568005b70 con 0x7f3558003a90 2023-10-10T18:53:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.761+0000 7f3575906700 1 -- 172.21.15.175:0/525511464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3558003a90 msgr2=0x7f3558003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.761+0000 7f3575906700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3558003a90 0x7f3558003ea0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f356800c3c0 tx=0x7f3570067b20 comp rx=0 tx=0).stop 2023-10-10T18:53:58.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.761+0000 7f3575906700 1 -- 172.21.15.175:0/525511464 shutdown_connections 2023-10-10T18:53:58.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.761+0000 7f3575906700 1 --2- 172.21.15.175:0/525511464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3558003a90 0x7f3558003ea0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:58.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.761+0000 7f3575906700 1 -- 172.21.15.175:0/525511464 >> 172.21.15.175:0/525511464 conn(0x7f35700c95a0 msgr2=0x7f35700c5540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:58.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.761+0000 7f3575906700 1 -- 172.21.15.175:0/525511464 shutdown_connections 2023-10-10T18:53:58.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.761+0000 7f3575906700 1 -- 172.21.15.175:0/525511464 wait complete. 2023-10-10T18:53:58.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.762+0000 7f3575906700 1 Processor -- start 2023-10-10T18:53:58.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.762+0000 7f3575906700 1 -- start start 2023-10-10T18:53:58.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f3575906700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558003a90 0x7f35700c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:58.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f3575906700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3570159100 0x7f35700bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:58.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f3575906700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3570169990 0x7f357015cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:58.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f3575906700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570159530 con 0x7f3570159100 2023-10-10T18:53:58.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f3575906700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35700c25e0 con 0x7f3570169990 2023-10-10T18:53:58.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f3575906700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3570159720 con 0x7f3558003a90 2023-10-10T18:53:58.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3570159100 0x7f35700bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:58.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3570159100 0x7f35700bfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39726/0 (socket says 172.21.15.175:39726) 2023-10-10T18:53:58.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 -- 172.21.15.175:0/1034159655 learned_addr learned my addr 172.21.15.175:0/1034159655 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:53:58.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356effd700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558003a90 0x7f35700c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:58.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356f7fe700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3570169990 0x7f357015cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:58.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 -- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558003a90 msgr2=0x7f35700c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:58.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558003a90 0x7f35700c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:58.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 -- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3570169990 msgr2=0x7f357015cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:58.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3570169990 0x7f357015cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:58.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 -- 172.21.15.175:0/1034159655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570156d50 con 0x7f3570159100 2023-10-10T18:53:58.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.763+0000 7f356e7fc700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3570159100 0x7f35700bfd30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3564011b70 tx=0x7f35640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:58.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.764+0000 7f3556ffd700 1 -- 172.21.15.175:0/1034159655 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356400f9d0 con 0x7f3570159100 2023-10-10T18:53:58.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.764+0000 7f3556ffd700 1 -- 172.21.15.175:0/1034159655 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356400fb30 con 0x7f3570159100 2023-10-10T18:53:58.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.764+0000 7f3556ffd700 1 -- 172.21.15.175:0/1034159655 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356400feb0 con 0x7f3570159100 2023-10-10T18:53:58.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.764+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35700c19d0 con 0x7f3570159100 2023-10-10T18:53:58.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.764+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35700c1f90 con 0x7f3570159100 2023-10-10T18:53:58.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.765+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35700c2790 con 0x7f3570159100 2023-10-10T18:53:58.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.765+0000 7f3556ffd700 1 -- 172.21.15.175:0/1034159655 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3564014650 con 0x7f3570159100 2023-10-10T18:53:58.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.765+0000 7f3556ffd700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f355c066f60 0x7f355c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:53:58.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.765+0000 7f3556ffd700 1 -- 172.21.15.175:0/1034159655 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f3564018030 con 0x7f3570159100 2023-10-10T18:53:58.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.765+0000 7f356effd700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f355c066f60 0x7f355c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:53:58.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.766+0000 7f356effd700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f355c066f60 0x7f355c069410 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3568001ed0 tx=0x7f3568015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:53:58.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.770+0000 7f3556ffd700 1 -- 172.21.15.175:0/1034159655 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35700c2790 con 0x7f3570159100 2023-10-10T18:53:58.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.995+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35700c2790 con 0x7f3570159100 2023-10-10T18:53:58.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.996+0000 7f3556ffd700 1 -- 172.21.15.175:0/1034159655 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5447 (secure 0 0 0) 0x7f35700c2790 con 0x7f3570159100 2023-10-10T18:53:58.997 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:53:58.997 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:53:56.704235+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:53:58.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f355c066f60 msgr2=0x7f355c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:58.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f355c066f60 0x7f355c069410 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3568001ed0 tx=0x7f3568015000 comp rx=0 tx=0).stop 2023-10-10T18:53:58.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3570159100 msgr2=0x7f35700bfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:53:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3570159100 0x7f35700bfd30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3564011b70 tx=0x7f35640091a0 comp rx=0 tx=0).stop 2023-10-10T18:53:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 shutdown_connections 2023-10-10T18:53:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f355c066f60 0x7f355c069410 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3570169990 0x7f357015cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3558003a90 0x7f35700c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 --2- 172.21.15.175:0/1034159655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3570159100 0x7f35700bfd30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:53:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 >> 172.21.15.175:0/1034159655 conn(0x7f35700c95a0 msgr2=0x7f3570068710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:53:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 shutdown_connections 2023-10-10T18:53:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:53:58.998+0000 7f3575906700 1 -- 172.21.15.175:0/1034159655 wait complete. 2023-10-10T18:53:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 75 2023-10-10T18:53:59.010 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4250 2023-10-10T18:53:59.011 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.f 2023-10-10T18:54:01.728 INFO:tasks.workunit.client.0.smithi175.stderr:+ find .snap/s/foo -executable 2023-10-10T18:54:01.729 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep foo 2023-10-10T18:54:01.734 INFO:tasks.workunit.client.0.smithi175.stdout:.snap/s/foo 2023-10-10T18:54:01.735 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T18:54:01.741 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm foo 2023-10-10T18:54:02.235 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T18:54:02.236 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T18:54:02.237 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T18:54:02.237 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T18:54:02.275 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-10-10T18:54:02.276 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-10-10T18:54:02.346 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T18:54:02.346 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir foo 2023-10-10T18:54:02.347 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo x 2023-10-10T18:54:02.348 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:03.006 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir foo/.snap/ss 2023-10-10T18:54:03.033 INFO:tasks.workunit.client.0.smithi175.stderr:+ ln foo/x foo/xx 2023-10-10T18:54:03.035 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat foo/.snap/ss/x 2023-10-10T18:54:03.043 INFO:tasks.workunit.client.0.smithi175.stdout:x 2023-10-10T18:54:03.043 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir foo/.snap/ss 2023-10-10T18:54:03.050 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo a 2023-10-10T18:54:03.051 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo b 2023-10-10T18:54:03.052 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir foo/.snap/s 2023-10-10T18:54:03.079 INFO:tasks.workunit.client.0.smithi175.stderr:++ cat foo/.snap/s/a 2023-10-10T18:54:03.086 INFO:tasks.workunit.client.0.smithi175.stderr:+ r=a 2023-10-10T18:54:03.086 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' -z a ']' 2023-10-10T18:54:03.086 INFO:tasks.workunit.client.0.smithi175.stderr:+ ln foo/b foo/b2 2023-10-10T18:54:03.088 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat foo/.snap/s/b 2023-10-10T18:54:03.101 INFO:tasks.workunit.client.0.smithi175.stdout:b 2023-10-10T18:54:03.102 INFO:tasks.workunit.client.0.smithi175.stdout:this used to hang: 2023-10-10T18:54:03.102 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 'this used to hang:' 2023-10-10T18:54:03.102 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo more 2023-10-10T18:54:03.103 INFO:tasks.workunit.client.0.smithi175.stdout:oh, it didn't hang! good job. 2023-10-10T18:54:03.103 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2023-10-10T18:54:03.104 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat foo/b 2023-10-10T18:54:03.104 INFO:tasks.workunit.client.0.smithi175.stdout:b 2023-10-10T18:54:03.104 INFO:tasks.workunit.client.0.smithi175.stdout:more 2023-10-10T18:54:03.105 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir foo/.snap/s 2023-10-10T18:54:03.108 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -r foo 2023-10-10T18:54:03.131 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T18:54:03.131 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T18:54:03.132 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T18:54:03.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T18:54:03.207 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-10-10T18:54:03.207 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-10-10T18:54:03.275 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T18:54:03.275 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:03.277 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/bar 2023-10-10T18:54:03.425 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/foo 2023-10-10T18:54:03.447 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/foo 2023-10-10T18:54:03.453 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:03.456 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/bar/a 2023-10-10T18:54:06.036 INFO:tasks.workunit.client.0.smithi175.stdout: File: .snap/bar/a 2023-10-10T18:54:06.037 INFO:tasks.workunit.client.0.smithi175.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-10T18:54:06.037 INFO:tasks.workunit.client.0.smithi175.stdout:Device: feh/254d Inode: 5497558139588 Links: 2 2023-10-10T18:54:06.037 INFO:tasks.workunit.client.0.smithi175.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1275/ ubuntu) 2023-10-10T18:54:06.037 INFO:tasks.workunit.client.0.smithi175.stdout:Context: unconfined_u:object_r:cephfs_t:s0 2023-10-10T18:54:06.037 INFO:tasks.workunit.client.0.smithi175.stdout:Access: 2023-10-10 18:54:03.275903791 +0000 2023-10-10T18:54:06.038 INFO:tasks.workunit.client.0.smithi175.stdout:Modify: 2023-10-10 18:54:03.275903791 +0000 2023-10-10T18:54:06.038 INFO:tasks.workunit.client.0.smithi175.stdout:Change: 2023-10-10 18:54:03.275903791 +0000 2023-10-10T18:54:06.038 INFO:tasks.workunit.client.0.smithi175.stdout: Birth: - 2023-10-10T18:54:06.038 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/bar 2023-10-10T18:54:06.044 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T18:54:06.045 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T18:54:06.045 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T18:54:06.046 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T18:54:06.998 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-10-10T18:54:06.999 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-10-10T18:54:07.030 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T18:54:07.030 INFO:tasks.workunit.client.0.smithi175.stderr:++ seq 1 20 2023-10-10T18:54:07.031 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:07.031 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:07.033 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:07.033 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:07.058 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:07.079 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:07.080 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:07.190 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:07.202 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:07.202 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:07.202 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:07.202 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:07.203 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:07.203 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:07.203 INFO:tasks.workunit.client.0.smithi175.stdout: 105808 ? 00:00:00 ps 2023-10-10T18:54:07.203 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:07.393 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:07.396 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:07.404 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:07.405 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:07.406 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:07.407 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:07.787 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:07.824 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:07.825 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:08.007 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:08.011 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-10T18:54:08.012 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-10T18:54:08.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-10T18:54:08.015 INFO:tasks.ceph.mds.f:Started 2023-10-10T18:54:08.015 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:54:08.018 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:08.018 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:08.018 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:08.019 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:08.019 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:08.019 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:08.019 INFO:tasks.workunit.client.0.smithi175.stdout: 105819 ? 00:00:00 ps 2023-10-10T18:54:08.019 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:08.090 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-10T18:54:08.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.133+0000 7faff15e3700 1 Processor -- start 2023-10-10T18:54:08.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.133+0000 7faff15e3700 1 -- start start 2023-10-10T18:54:08.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7faff15e3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b1fc0 0x7fafec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:08.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7faff15e3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fafec0b2390 0x7fafec068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:08.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7faff15e3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafec1574b0 con 0x7fafec068bd0 2023-10-10T18:54:08.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7faff15e3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafec150770 con 0x7fafec0b2390 2023-10-10T18:54:08.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7faff15e3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0afad0 con 0x7fafec0b1fc0 2023-10-10T18:54:08.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafe27fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fafec0b2390 0x7fafec068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:08.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafeb7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fafec068bd0 0x7fafec0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60978/0 (socket says 172.21.15.175:60978) 2023-10-10T18:54:08.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafeb7fe700 1 -- 172.21.15.175:0/1947147365 learned_addr learned my addr 172.21.15.175:0/1947147365 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:54:08.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafeaffd700 1 --2- 172.21.15.175:0/1947147365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b1fc0 0x7fafec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:08.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafe27fc700 1 -- 172.21.15.175:0/1947147365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b1fc0 msgr2=0x7fafec155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:54:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafe27fc700 1 --2- 172.21.15.175:0/1947147365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b1fc0 0x7fafec155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafe27fc700 1 -- 172.21.15.175:0/1947147365 >> v1:172.21.15.175:6789/0 conn(0x7fafec068bd0 legacy=0x7fafec0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafea7fc700 1 -- 172.21.15.175:0/1947147365 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371543428 0 0) 0x7fafec1574b0 con 0x7fafec068bd0 2023-10-10T18:54:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.134+0000 7fafe27fc700 1 -- 172.21.15.175:0/1947147365 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec0af4c0 con 0x7fafec0b2390 2023-10-10T18:54:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7fafe27fc700 1 --2- 172.21.15.175:0/1947147365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec0b2390 0x7fafec068690 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fafd40098e0 tx=0x7fafd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0dbdedf029e423f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:54:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7fafea7fc700 1 -- 172.21.15.175:0/1947147365 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd400b6d0 con 0x7fafec0b2390 2023-10-10T18:54:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7fafea7fc700 1 -- 172.21.15.175:0/1947147365 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd400b830 con 0x7fafec0b2390 2023-10-10T18:54:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7fafea7fc700 1 -- 172.21.15.175:0/1947147365 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd400bbd0 con 0x7fafec0b2390 2023-10-10T18:54:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7faff15e3700 1 -- 172.21.15.175:0/1947147365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec0b2390 msgr2=0x7fafec068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7faff15e3700 1 --2- 172.21.15.175:0/1947147365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec0b2390 0x7fafec068690 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fafd40098e0 tx=0x7fafd40091a0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7faff15e3700 1 -- 172.21.15.175:0/1947147365 shutdown_connections 2023-10-10T18:54:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7faff15e3700 1 --2- 172.21.15.175:0/1947147365 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b1fc0 0x7fafec155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7faff15e3700 1 --2- 172.21.15.175:0/1947147365 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec0b2390 0x7fafec068690 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7faff15e3700 1 -- 172.21.15.175:0/1947147365 >> 172.21.15.175:0/1947147365 conn(0x7fafec063230 msgr2=0x7fafec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:54:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7faff15e3700 1 -- 172.21.15.175:0/1947147365 shutdown_connections 2023-10-10T18:54:08.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.135+0000 7faff15e3700 1 -- 172.21.15.175:0/1947147365 wait complete. 2023-10-10T18:54:08.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.136+0000 7faff15e3700 1 Processor -- start 2023-10-10T18:54:08.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.136+0000 7faff15e3700 1 -- start start 2023-10-10T18:54:08.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7faff15e3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec068bd0 0x7fafec0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:08.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7faff15e3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafec0b1fc0 0x7fafec1667b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:08.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7faff15e3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b2390 0x7fafec159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:08.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7faff15e3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec15a2e0 con 0x7fafec068bd0 2023-10-10T18:54:08.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7faff15e3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0cbd60 con 0x7fafec0b1fc0 2023-10-10T18:54:08.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7faff15e3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0c4360 con 0x7fafec0b2390 2023-10-10T18:54:08.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafeaffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec068bd0 0x7fafec0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:08.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafeaffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec068bd0 0x7fafec0cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52066/0 (socket says 172.21.15.175:52066) 2023-10-10T18:54:08.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafeaffd700 1 -- 172.21.15.175:0/3298495677 learned_addr learned my addr 172.21.15.175:0/3298495677 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:54:08.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafe27fc700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafec0b1fc0 0x7fafec1667b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:08.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafeb7fe700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b2390 0x7fafec159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:08.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafeaffd700 1 -- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b2390 msgr2=0x7fafec159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:08.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafeaffd700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b2390 0x7fafec159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafeaffd700 1 -- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafec0b1fc0 msgr2=0x7fafec1667b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:08.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.137+0000 7fafeaffd700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafec0b1fc0 0x7fafec1667b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.138+0000 7fafeaffd700 1 -- 172.21.15.175:0/3298495677 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd4009420 con 0x7fafec068bd0 2023-10-10T18:54:08.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.138+0000 7fafeaffd700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec068bd0 0x7fafec0cc1f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00ec80 tx=0x7fafdc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:54:08.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.138+0000 7fafe37fe700 1 -- 172.21.15.175:0/3298495677 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc00c850 con 0x7fafec068bd0 2023-10-10T18:54:08.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.138+0000 7fafe37fe700 1 -- 172.21.15.175:0/3298495677 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc00c9b0 con 0x7fafec068bd0 2023-10-10T18:54:08.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.138+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec0ccc60 con 0x7fafec068bd0 2023-10-10T18:54:08.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.138+0000 7fafe37fe700 1 -- 172.21.15.175:0/3298495677 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc00cd50 con 0x7fafec068bd0 2023-10-10T18:54:08.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.138+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec0cd1c0 con 0x7fafec068bd0 2023-10-10T18:54:08.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.139+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec067570 con 0x7fafec068bd0 2023-10-10T18:54:08.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.139+0000 7fafe37fe700 1 -- 172.21.15.175:0/3298495677 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fafdc013680 con 0x7fafec068bd0 2023-10-10T18:54:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.139+0000 7fafe37fe700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafcc066f30 0x7fafcc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.140+0000 7fafe37fe700 1 -- 172.21.15.175:0/3298495677 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fafdc084970 con 0x7fafec068bd0 2023-10-10T18:54:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.140+0000 7fafe27fc700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafcc066f30 0x7fafcc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.140+0000 7fafe27fc700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafcc066f30 0x7fafcc0693e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fafd40093f0 tx=0x7fafd4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:54:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.144+0000 7fafe37fe700 1 -- 172.21.15.175:0/3298495677 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafdc0519c0 con 0x7fafec068bd0 2023-10-10T18:54:08.155 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:08.158 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:08.164 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:08.164 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:08.166 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:08.166 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:08.190 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:08.211 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:08.212 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:08.309 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:08.316 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:08.316 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:08.316 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:08.316 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:08.317 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:08.317 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:08.317 INFO:tasks.workunit.client.0.smithi175.stdout: 105869 ? 00:00:00 ps 2023-10-10T18:54:08.317 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:08.330 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:08.331 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:08.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.375+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafec069060 con 0x7fafec068bd0 2023-10-10T18:54:08.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.376+0000 7fafe37fe700 1 -- 172.21.15.175:0/3298495677 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5443 (secure 0 0 0) 0x7fafdc054fe0 con 0x7fafec068bd0 2023-10-10T18:54:08.377 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:54:08.378 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:54:04.492024+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:54:08.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafcc066f30 msgr2=0x7fafcc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:08.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafcc066f30 0x7fafcc0693e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fafd40093f0 tx=0x7fafd4017000 comp rx=0 tx=0).stop 2023-10-10T18:54:08.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec068bd0 msgr2=0x7fafec0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:08.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec068bd0 0x7fafec0cc1f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00ec80 tx=0x7fafdc0091a0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 shutdown_connections 2023-10-10T18:54:08.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafcc066f30 0x7fafcc0693e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafec0b2390 0x7fafec159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafec068bd0 0x7fafec0cc1f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 --2- 172.21.15.175:0/3298495677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafec0b1fc0 0x7fafec1667b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:08.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 >> 172.21.15.175:0/3298495677 conn(0x7fafec063230 msgr2=0x7fafec14fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:54:08.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 shutdown_connections 2023-10-10T18:54:08.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:08.378+0000 7faff15e3700 1 -- 172.21.15.175:0/3298495677 wait complete. 2023-10-10T18:54:08.382 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 76 2023-10-10T18:54:08.391 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-10T18:54:08.905 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:08.906 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:08.909 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:08.909 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:08.933 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:08.954 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:08.955 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:09.077 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:09.084 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:09.084 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:09.084 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:09.085 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:09.085 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:09.085 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:09.085 INFO:tasks.workunit.client.0.smithi175.stdout: 105881 ? 00:00:00 ps 2023-10-10T18:54:09.086 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:09.096 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:09.098 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:09.106 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:09.106 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:09.108 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:09.108 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:09.135 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:09.157 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:09.158 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:09.216 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:09.222 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:09.223 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:09.223 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:09.223 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:09.224 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:09.224 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:09.224 INFO:tasks.workunit.client.0.smithi175.stdout: 105891 ? 00:00:00 ps 2023-10-10T18:54:09.225 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:09.236 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:09.239 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:09.246 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:09.246 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:09.248 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:09.248 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:10.013 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:10.392 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:54:10.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.513+0000 7f5e2ef23700 1 Processor -- start 2023-10-10T18:54:10.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.513+0000 7f5e2ef23700 1 -- start start 2023-10-10T18:54:10.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e2ef23700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e280c12a0 0x7f5e281577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:10.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e2ef23700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e28154aa0 con 0x7f5e281573d0 2023-10-10T18:54:10.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e2ef23700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e28063dc0 con 0x7f5e280c4f20 2023-10-10T18:54:10.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e2ef23700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e280bfdf0 con 0x7f5e280c12a0 2023-10-10T18:54:10.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e2ccbf700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5e281573d0 0x7f5e280bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60982/0 (socket says 172.21.15.175:60982) 2023-10-10T18:54:10.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e2ccbf700 1 -- 172.21.15.175:0/1282439586 learned_addr learned my addr 172.21.15.175:0/1282439586 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:54:10.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e27fff700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e280c12a0 0x7f5e281577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:10.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153049318 0 0) 0x7f5e28154aa0 con 0x7f5e281573d0 2023-10-10T18:54:10.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e18003570 con 0x7f5e281573d0 2023-10-10T18:54:10.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 289496415 0 0) 0x7f5e18003570 con 0x7f5e281573d0 2023-10-10T18:54:10.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e280c12a0 msgr2=0x7f5e281577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:54:10.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e277fe700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e280c12a0 0x7f5e281577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 >> v1:172.21.15.191:6789/0 conn(0x7f5e280c4f20 legacy=0x7f5e2814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e280c5360 con 0x7f5e281573d0 2023-10-10T18:54:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.514+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f5e1c001470 con 0x7f5e281573d0 2023-10-10T18:54:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 >> v1:172.21.15.175:6789/0 conn(0x7f5e281573d0 legacy=0x7f5e280bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e18003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e280c12a0 0x7f5e18004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e18004d50 0x7f5e180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1800bb70 con 0x7f5e18003a90 2023-10-10T18:54:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1800bce0 con 0x7f5e18004d50 2023-10-10T18:54:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1800bea0 con 0x7f5e280c12a0 2023-10-10T18:54:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e2ccbf700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e18003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e27fff700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e280c12a0 0x7f5e18004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e2ccbf700 1 -- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e280c12a0 msgr2=0x7f5e18004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:54:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e2ccbf700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e280c12a0 0x7f5e18004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e2ccbf700 1 -- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e18004d50 msgr2=0x7f5e180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:54:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e2ccbf700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e18004d50 0x7f5e180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e2ccbf700 1 -- 172.21.15.175:0/1282439586 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e18003570 con 0x7f5e18003a90 2023-10-10T18:54:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e2ccbf700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e18003ea0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00c340 tx=0x7f5e2806f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc27ed971222a69c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:54:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00cc70 con 0x7f5e18003a90 2023-10-10T18:54:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c00cdf0 con 0x7f5e18003a90 2023-10-10T18:54:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.515+0000 7f5e277fe700 1 -- 172.21.15.175:0/1282439586 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c005b70 con 0x7f5e18003a90 2023-10-10T18:54:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.516+0000 7f5e2ef23700 1 -- 172.21.15.175:0/1282439586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 msgr2=0x7f5e18003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.516+0000 7f5e2ef23700 1 --2- 172.21.15.175:0/1282439586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e18003ea0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00c340 tx=0x7f5e2806f1f0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.516+0000 7f5e2d4c0700 1 -- 172.21.15.175:0/1282439586 reap_dead start 2023-10-10T18:54:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.516+0000 7f5e2ef23700 1 -- 172.21.15.175:0/1282439586 shutdown_connections 2023-10-10T18:54:10.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.516+0000 7f5e2ef23700 1 -- 172.21.15.175:0/1282439586 >> 172.21.15.175:0/1282439586 conn(0x7f5e2806dc70 msgr2=0x7f5e28063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:54:10.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.516+0000 7f5e2ef23700 1 -- 172.21.15.175:0/1282439586 shutdown_connections 2023-10-10T18:54:10.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.516+0000 7f5e2ef23700 1 -- 172.21.15.175:0/1282439586 wait complete. 2023-10-10T18:54:10.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.517+0000 7f5e2ef23700 1 Processor -- start 2023-10-10T18:54:10.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.517+0000 7f5e2ef23700 1 -- start start 2023-10-10T18:54:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ef23700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e28164720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ef23700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e281519c0 0x7f5e28159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ef23700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e281536f0 0x7f5e28167290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ef23700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e28151710 con 0x7f5e18003a90 2023-10-10T18:54:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ef23700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e280cb3c0 con 0x7f5e281519c0 2023-10-10T18:54:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ef23700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e28163d10 con 0x7f5e281536f0 2023-10-10T18:54:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e28164720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e28164720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52086/0 (socket says 172.21.15.175:52086) 2023-10-10T18:54:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 -- 172.21.15.175:0/3763091086 learned_addr learned my addr 172.21.15.175:0/3763091086 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:54:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e27fff700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e281519c0 0x7f5e28159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2d4c0700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e281536f0 0x7f5e28167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 -- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e281536f0 msgr2=0x7f5e28167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e281536f0 0x7f5e28167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 -- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e281519c0 msgr2=0x7f5e28159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e281519c0 0x7f5e28159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 -- 172.21.15.175:0/3763091086 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e28164290 con 0x7f5e18003a90 2023-10-10T18:54:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.518+0000 7f5e2ccbf700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e28164720 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800bcb0 tx=0x7f5e1c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:54:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.519+0000 7f5e24ff9700 1 -- 172.21.15.175:0/3763091086 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0070d0 con 0x7f5e18003a90 2023-10-10T18:54:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.519+0000 7f5e24ff9700 1 -- 172.21.15.175:0/3763091086 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c007230 con 0x7f5e18003a90 2023-10-10T18:54:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.519+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e28164c60 con 0x7f5e18003a90 2023-10-10T18:54:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.519+0000 7f5e24ff9700 1 -- 172.21.15.175:0/3763091086 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0075d0 con 0x7f5e18003a90 2023-10-10T18:54:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.519+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e28152ff0 con 0x7f5e18003a90 2023-10-10T18:54:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.520+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e28067570 con 0x7f5e18003a90 2023-10-10T18:54:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.520+0000 7f5e24ff9700 1 -- 172.21.15.175:0/3763091086 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5e1c019030 con 0x7f5e18003a90 2023-10-10T18:54:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.520+0000 7f5e24ff9700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e0c066f30 0x7f5e0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:54:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.520+0000 7f5e24ff9700 1 -- 172.21.15.175:0/3763091086 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6237+0+0 (secure 0 0 0) 0x7f5e1c085660 con 0x7f5e18003a90 2023-10-10T18:54:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.521+0000 7f5e27fff700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e0c066f30 0x7f5e0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:54:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.521+0000 7f5e27fff700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e0c066f30 0x7f5e0c0693e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e2814b820 tx=0x7f5e14008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:54:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.525+0000 7f5e24ff9700 1 -- 172.21.15.175:0/3763091086 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e1c0526c0 con 0x7f5e18003a90 2023-10-10T18:54:10.636 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:10.637 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:10.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.773+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e28164c60 con 0x7f5e18003a90 2023-10-10T18:54:10.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.774+0000 7f5e24ff9700 1 -- 172.21.15.175:0/3763091086 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6202 (secure 0 0 0) 0x7f5e28164c60 con 0x7f5e18003a90 2023-10-10T18:54:10.775 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:54:10.775 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5332,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:54:04.492024+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:54:10.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e0c066f30 msgr2=0x7f5e0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:10.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e0c066f30 0x7f5e0c0693e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e2814b820 tx=0x7f5e14008040 comp rx=0 tx=0).stop 2023-10-10T18:54:10.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 msgr2=0x7f5e28164720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:54:10.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e28164720 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800bcb0 tx=0x7f5e1c012000 comp rx=0 tx=0).stop 2023-10-10T18:54:10.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 shutdown_connections 2023-10-10T18:54:10.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e0c066f30 0x7f5e0c0693e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e281536f0 0x7f5e28167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e18003a90 0x7f5e28164720 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 --2- 172.21.15.175:0/3763091086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e281519c0 0x7f5e28159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:54:10.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 >> 172.21.15.175:0/3763091086 conn(0x7f5e2806dc70 msgr2=0x7f5e2814bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:54:10.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 shutdown_connections 2023-10-10T18:54:10.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:54:10.776+0000 7f5e2ef23700 1 -- 172.21.15.175:0/3763091086 wait complete. 2023-10-10T18:54:10.780 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 77 2023-10-10T18:54:10.788 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-10T18:54:10.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2023-10-10T18:54:10.908 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:10.914 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:10.914 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:10.914 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:10.915 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:10.915 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:10.915 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:10.915 INFO:tasks.workunit.client.0.smithi175.stdout: 105943 ? 00:00:00 ps 2023-10-10T18:54:10.916 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:11.335 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:11.337 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:11.345 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:11.345 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:11.348 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:11.348 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:11.995 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:12.012 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:12.012 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:12.109 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:12.115 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:12.115 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:12.116 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:12.116 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:12.116 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:12.116 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:12.116 INFO:tasks.workunit.client.0.smithi175.stdout: 105953 ? 00:00:00 ps 2023-10-10T18:54:12.117 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:12.129 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:12.132 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:12.140 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:12.140 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:12.142 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:12.142 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:12.173 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:12.195 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:12.196 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:12.258 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:12.265 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:12.265 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:12.265 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:12.265 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:12.266 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:12.266 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:12.266 INFO:tasks.workunit.client.0.smithi175.stdout: 105964 ? 00:00:00 ps 2023-10-10T18:54:12.267 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:12.637 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:12.713 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:13.123 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:13.124 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:13.126 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:13.126 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:13.153 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:13.176 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:13.176 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:13.364 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:13.371 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:13.371 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:13.371 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:13.371 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:13.372 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:13.372 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:13.372 INFO:tasks.workunit.client.0.smithi175.stdout: 105974 ? 00:00:00 ps 2023-10-10T18:54:13.373 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:13.386 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:13.389 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:13.400 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:13.400 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:13.402 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:13.402 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:13.430 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:14.143 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:14.144 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:14.269 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:14.275 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:14.275 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:14.276 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:14.276 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:14.276 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:14.277 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:14.277 INFO:tasks.workunit.client.0.smithi175.stdout: 105984 ? 00:00:00 ps 2023-10-10T18:54:14.278 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:14.292 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:14.295 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:14.303 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:14.303 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:14.305 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:14.306 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:14.338 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:14.361 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:14.362 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:14.468 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:14.474 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:14.474 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:14.475 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:14.475 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:14.475 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:14.475 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:14.475 INFO:tasks.workunit.client.0.smithi175.stdout: 105994 ? 00:00:00 ps 2023-10-10T18:54:14.476 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:14.488 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:14.490 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:14.509 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:14.509 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:14.512 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:14.512 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:14.547 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:14.570 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:14.571 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:14.677 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:14.683 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:14.683 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:14.683 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:14.684 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:14.684 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:14.684 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:14.684 INFO:tasks.workunit.client.0.smithi175.stdout: 106004 ? 00:00:00 ps 2023-10-10T18:54:14.685 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:14.700 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:15.425 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:16.177 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:16.177 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:16.180 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:16.180 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:16.202 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:16.218 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:16.218 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:16.404 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:16.410 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:16.411 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:16.411 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:16.411 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:16.411 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:16.411 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:16.412 INFO:tasks.workunit.client.0.smithi175.stdout: 106014 ? 00:00:00 ps 2023-10-10T18:54:16.412 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:17.465 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:17.530 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:18.232 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:18.232 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:18.234 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:18.235 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:18.268 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:18.291 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:18.292 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:18.691 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:18.699 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:18.699 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:18.700 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:18.700 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:18.700 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:18.700 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:18.701 INFO:tasks.workunit.client.0.smithi175.stdout: 106024 ? 00:00:00 ps 2023-10-10T18:54:18.701 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:18.726 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:18.728 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:18.737 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:18.737 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:18.739 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:18.739 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:18.767 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:18.790 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:18.791 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:18.895 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:18.901 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:18.901 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:18.902 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:18.902 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:18.902 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:18.902 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:18.902 INFO:tasks.workunit.client.0.smithi175.stdout: 106034 ? 00:00:00 ps 2023-10-10T18:54:18.903 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:18.916 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:18.919 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:19.284 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:19.284 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:19.287 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:19.287 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:19.316 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:19.338 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:19.339 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:19.464 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:19.471 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:19.471 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:19.471 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:19.471 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:19.471 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:19.472 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:19.472 INFO:tasks.workunit.client.0.smithi175.stdout: 106044 ? 00:00:00 ps 2023-10-10T18:54:19.472 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:19.496 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:19.502 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:19.521 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:54:19.522 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:19.523 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:19.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:54:19.525 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:19.526 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:19.566 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:19.890 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:19.891 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:20.006 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:20.012 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:20.012 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:20.013 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:20.013 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:20.013 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:20.013 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:20.013 INFO:tasks.workunit.client.0.smithi175.stdout: 106071 ? 00:00:00 ps 2023-10-10T18:54:20.014 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:21.498 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:21.505 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:21.698 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:21.698 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:21.701 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:21.701 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:21.734 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:21.757 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:21.757 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:21.978 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:21.984 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:21.984 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:21.985 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:21.985 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:21.985 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:21.985 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:21.986 INFO:tasks.workunit.client.0.smithi175.stdout: 106081 ? 00:00:00 ps 2023-10-10T18:54:21.986 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:22.000 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:22.003 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:22.011 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:22.012 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:22.014 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:22.014 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:22.299 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:22.315 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:22.316 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:22.396 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:22.402 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:22.402 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:22.403 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:22.403 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:22.403 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:22.403 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:22.404 INFO:tasks.workunit.client.0.smithi175.stdout: 106091 ? 00:00:00 ps 2023-10-10T18:54:22.404 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:22.418 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:22.421 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:22.429 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T18:54:22.429 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T18:54:22.431 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/one 2023-10-10T18:54:22.432 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat 2023-10-10T18:54:22.453 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/two 2023-10-10T18:54:22.482 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a/foo 2023-10-10T18:54:22.482 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T18:54:22.554 INFO:tasks.workunit.client.0.smithi175.stderr:+ ps 2023-10-10T18:54:22.561 INFO:tasks.workunit.client.0.smithi175.stdout: PID TTY TIME CMD 2023-10-10T18:54:22.561 INFO:tasks.workunit.client.0.smithi175.stdout: 4607 ? 00:00:00 systemd 2023-10-10T18:54:22.561 INFO:tasks.workunit.client.0.smithi175.stdout: 4610 ? 00:00:00 (sd-pam) 2023-10-10T18:54:22.562 INFO:tasks.workunit.client.0.smithi175.stdout: 78445 ? 00:00:00 sshd 2023-10-10T18:54:22.562 INFO:tasks.workunit.client.0.smithi175.stdout: 105783 ? 00:00:00 timeout 2023-10-10T18:54:22.562 INFO:tasks.workunit.client.0.smithi175.stdout: 105800 ? 00:00:00 snaptest-double 2023-10-10T18:54:22.563 INFO:tasks.workunit.client.0.smithi175.stdout: 106101 ? 00:00:00 ps 2023-10-10T18:54:22.563 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-10T18:54:22.852 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a/foo 2023-10-10T18:54:22.855 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a 2023-10-10T18:54:22.865 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T18:54:22.866 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T18:54:22.867 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T18:54:22.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T18:54:23.800 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-10-10T18:54:23.800 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-10-10T18:54:27.707 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/foo 2023-10-10T18:54:28.739 INFO:tasks.workunit.client.0.smithi175.stdout:We want ENOENT, not ESTALE, here. 2023-10-10T18:54:28.740 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2023-10-10T18:54:28.740 INFO:tasks.workunit.client.0.smithi175.stderr:++ seq 1 100 2023-10-10T18:54:28.741 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.741 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/1 2023-10-10T18:54:28.742 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.744 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/1': No such file or directory 2023-10-10T18:54:28.744 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.745 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/2 2023-10-10T18:54:28.745 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.745 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/2': No such file or directory 2023-10-10T18:54:28.746 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.746 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/3 2023-10-10T18:54:28.746 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.747 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/3': No such file or directory 2023-10-10T18:54:28.748 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.748 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/4 2023-10-10T18:54:28.748 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.749 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/4': No such file or directory 2023-10-10T18:54:28.749 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.750 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/5 2023-10-10T18:54:28.750 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.750 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/5': No such file or directory 2023-10-10T18:54:28.751 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.751 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/6 2023-10-10T18:54:28.751 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.752 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/6': No such file or directory 2023-10-10T18:54:28.752 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.753 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/7 2023-10-10T18:54:28.753 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.753 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/7': No such file or directory 2023-10-10T18:54:28.754 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.754 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/8 2023-10-10T18:54:28.755 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.755 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/8': No such file or directory 2023-10-10T18:54:28.756 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.756 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/9 2023-10-10T18:54:28.756 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.757 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/9': No such file or directory 2023-10-10T18:54:28.757 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.758 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/10 2023-10-10T18:54:28.758 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.758 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/10': No such file or directory 2023-10-10T18:54:28.759 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.759 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/11 2023-10-10T18:54:28.759 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.760 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/11': No such file or directory 2023-10-10T18:54:28.760 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.761 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/12 2023-10-10T18:54:28.761 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.761 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/12': No such file or directory 2023-10-10T18:54:28.762 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.762 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/13 2023-10-10T18:54:28.762 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.763 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/13': No such file or directory 2023-10-10T18:54:28.764 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.764 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/14 2023-10-10T18:54:28.764 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.764 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/14': No such file or directory 2023-10-10T18:54:28.765 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.765 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/15 2023-10-10T18:54:28.766 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.766 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/15': No such file or directory 2023-10-10T18:54:28.767 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.767 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/16 2023-10-10T18:54:28.767 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.768 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/16': No such file or directory 2023-10-10T18:54:28.768 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.768 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/17 2023-10-10T18:54:28.769 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.769 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/17': No such file or directory 2023-10-10T18:54:28.770 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.770 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/18 2023-10-10T18:54:28.770 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.771 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/18': No such file or directory 2023-10-10T18:54:28.771 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.772 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/19 2023-10-10T18:54:28.772 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.772 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/19': No such file or directory 2023-10-10T18:54:28.773 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.773 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/20 2023-10-10T18:54:28.773 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.774 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/20': No such file or directory 2023-10-10T18:54:28.774 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.775 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/21 2023-10-10T18:54:28.775 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.775 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/21': No such file or directory 2023-10-10T18:54:28.776 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.776 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/22 2023-10-10T18:54:28.776 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.777 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/22': No such file or directory 2023-10-10T18:54:28.778 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.778 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/23 2023-10-10T18:54:28.778 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.778 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/23': No such file or directory 2023-10-10T18:54:28.779 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.779 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/24 2023-10-10T18:54:28.779 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.780 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/24': No such file or directory 2023-10-10T18:54:28.781 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.781 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/25 2023-10-10T18:54:28.781 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.782 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/25': No such file or directory 2023-10-10T18:54:28.782 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.782 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/26 2023-10-10T18:54:28.783 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.783 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/26': No such file or directory 2023-10-10T18:54:28.784 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.784 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/27 2023-10-10T18:54:28.784 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.785 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/27': No such file or directory 2023-10-10T18:54:28.785 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.785 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/28 2023-10-10T18:54:28.786 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.786 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/28': No such file or directory 2023-10-10T18:54:28.787 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.787 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/29 2023-10-10T18:54:28.787 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.787 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/29': No such file or directory 2023-10-10T18:54:28.788 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.788 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/30 2023-10-10T18:54:28.789 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.789 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/30': No such file or directory 2023-10-10T18:54:28.790 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.790 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/31 2023-10-10T18:54:28.790 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.790 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/31': No such file or directory 2023-10-10T18:54:28.791 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.791 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/32 2023-10-10T18:54:28.792 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.792 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/32': No such file or directory 2023-10-10T18:54:28.793 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.793 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/33 2023-10-10T18:54:28.793 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.794 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/33': No such file or directory 2023-10-10T18:54:28.794 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.794 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/34 2023-10-10T18:54:28.795 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.795 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/34': No such file or directory 2023-10-10T18:54:28.796 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.796 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/35 2023-10-10T18:54:28.796 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.796 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/35': No such file or directory 2023-10-10T18:54:28.797 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.797 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/36 2023-10-10T18:54:28.798 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.798 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/36': No such file or directory 2023-10-10T18:54:28.799 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.799 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/37 2023-10-10T18:54:28.799 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.799 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/37': No such file or directory 2023-10-10T18:54:28.800 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.800 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/38 2023-10-10T18:54:28.800 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.801 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/38': No such file or directory 2023-10-10T18:54:28.801 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.802 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/39 2023-10-10T18:54:28.802 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.802 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/39': No such file or directory 2023-10-10T18:54:28.803 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.803 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/40 2023-10-10T18:54:28.803 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.804 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/40': No such file or directory 2023-10-10T18:54:28.804 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.805 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/41 2023-10-10T18:54:28.805 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.805 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/41': No such file or directory 2023-10-10T18:54:28.806 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.806 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/42 2023-10-10T18:54:28.806 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.807 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/42': No such file or directory 2023-10-10T18:54:28.807 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.807 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/43 2023-10-10T18:54:28.808 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.809 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/43': No such file or directory 2023-10-10T18:54:28.810 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.810 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/44 2023-10-10T18:54:28.811 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.811 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/44': No such file or directory 2023-10-10T18:54:28.812 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.812 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/45 2023-10-10T18:54:28.812 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.813 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/45': No such file or directory 2023-10-10T18:54:28.813 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.814 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/46 2023-10-10T18:54:28.814 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.814 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/46': No such file or directory 2023-10-10T18:54:28.815 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.815 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/47 2023-10-10T18:54:28.815 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.816 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/47': No such file or directory 2023-10-10T18:54:28.817 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.817 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/48 2023-10-10T18:54:28.817 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.817 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/48': No such file or directory 2023-10-10T18:54:28.818 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.818 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/49 2023-10-10T18:54:28.819 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.819 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/49': No such file or directory 2023-10-10T18:54:28.820 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.820 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/50 2023-10-10T18:54:28.820 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.821 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/50': No such file or directory 2023-10-10T18:54:28.821 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.822 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/51 2023-10-10T18:54:28.822 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.822 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/51': No such file or directory 2023-10-10T18:54:28.823 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.823 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/52 2023-10-10T18:54:28.823 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.824 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/52': No such file or directory 2023-10-10T18:54:28.824 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.825 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/53 2023-10-10T18:54:28.825 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.825 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/53': No such file or directory 2023-10-10T18:54:28.826 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.826 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/54 2023-10-10T18:54:28.826 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.827 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/54': No such file or directory 2023-10-10T18:54:28.827 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.828 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/55 2023-10-10T18:54:28.828 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.828 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/55': No such file or directory 2023-10-10T18:54:28.829 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.829 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/56 2023-10-10T18:54:28.830 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.830 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/56': No such file or directory 2023-10-10T18:54:28.831 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.831 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/57 2023-10-10T18:54:28.831 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.831 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/57': No such file or directory 2023-10-10T18:54:28.832 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.832 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/58 2023-10-10T18:54:28.833 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.833 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/58': No such file or directory 2023-10-10T18:54:28.834 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.834 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/59 2023-10-10T18:54:28.834 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.835 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/59': No such file or directory 2023-10-10T18:54:28.835 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.836 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/60 2023-10-10T18:54:28.836 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.836 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/60': No such file or directory 2023-10-10T18:54:28.837 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.837 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/61 2023-10-10T18:54:28.838 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.838 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/61': No such file or directory 2023-10-10T18:54:28.839 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.839 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/62 2023-10-10T18:54:28.839 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.840 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/62': No such file or directory 2023-10-10T18:54:28.840 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.841 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/63 2023-10-10T18:54:28.841 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.841 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/63': No such file or directory 2023-10-10T18:54:28.842 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.842 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/64 2023-10-10T18:54:28.843 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.843 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/64': No such file or directory 2023-10-10T18:54:28.844 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.844 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/65 2023-10-10T18:54:28.844 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.845 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/65': No such file or directory 2023-10-10T18:54:28.845 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.846 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/66 2023-10-10T18:54:28.846 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.846 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/66': No such file or directory 2023-10-10T18:54:28.847 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.847 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/67 2023-10-10T18:54:28.847 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.848 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/67': No such file or directory 2023-10-10T18:54:28.849 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.849 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/68 2023-10-10T18:54:28.849 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.849 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/68': No such file or directory 2023-10-10T18:54:28.850 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.850 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/69 2023-10-10T18:54:28.851 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.851 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/69': No such file or directory 2023-10-10T18:54:28.852 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.852 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/70 2023-10-10T18:54:28.852 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.852 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/70': No such file or directory 2023-10-10T18:54:28.853 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.853 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/71 2023-10-10T18:54:28.854 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.854 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/71': No such file or directory 2023-10-10T18:54:28.855 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.855 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/72 2023-10-10T18:54:28.855 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.856 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/72': No such file or directory 2023-10-10T18:54:28.856 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.856 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/73 2023-10-10T18:54:28.857 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.857 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/73': No such file or directory 2023-10-10T18:54:28.858 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.858 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/74 2023-10-10T18:54:28.858 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.858 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/74': No such file or directory 2023-10-10T18:54:28.859 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.859 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/75 2023-10-10T18:54:28.860 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.860 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/75': No such file or directory 2023-10-10T18:54:28.861 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.861 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/76 2023-10-10T18:54:28.861 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.862 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/76': No such file or directory 2023-10-10T18:54:28.862 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.863 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/77 2023-10-10T18:54:28.863 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.864 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/77': No such file or directory 2023-10-10T18:54:28.864 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.864 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/78 2023-10-10T18:54:28.865 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.865 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/78': No such file or directory 2023-10-10T18:54:28.866 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.866 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/79 2023-10-10T18:54:28.866 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.867 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/79': No such file or directory 2023-10-10T18:54:28.867 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.867 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/80 2023-10-10T18:54:28.868 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.868 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/80': No such file or directory 2023-10-10T18:54:28.869 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.869 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/81 2023-10-10T18:54:28.869 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.870 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/81': No such file or directory 2023-10-10T18:54:28.870 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.870 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/82 2023-10-10T18:54:28.871 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.871 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/82': No such file or directory 2023-10-10T18:54:28.872 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.872 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/83 2023-10-10T18:54:28.872 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.873 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/83': No such file or directory 2023-10-10T18:54:28.873 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.873 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/84 2023-10-10T18:54:28.874 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.874 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/84': No such file or directory 2023-10-10T18:54:28.875 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.875 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/85 2023-10-10T18:54:28.875 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.876 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/85': No such file or directory 2023-10-10T18:54:28.876 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.877 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/86 2023-10-10T18:54:28.877 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.877 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/86': No such file or directory 2023-10-10T18:54:28.878 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.878 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/87 2023-10-10T18:54:28.879 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.879 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/87': No such file or directory 2023-10-10T18:54:28.880 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.880 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/88 2023-10-10T18:54:28.880 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.880 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/88': No such file or directory 2023-10-10T18:54:28.881 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.881 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/89 2023-10-10T18:54:28.881 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.882 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/89': No such file or directory 2023-10-10T18:54:28.883 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.883 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/90 2023-10-10T18:54:28.883 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.883 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/90': No such file or directory 2023-10-10T18:54:28.884 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.884 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/91 2023-10-10T18:54:28.885 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.885 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/91': No such file or directory 2023-10-10T18:54:28.886 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.886 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/92 2023-10-10T18:54:28.886 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.886 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/92': No such file or directory 2023-10-10T18:54:28.887 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.887 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/93 2023-10-10T18:54:28.887 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.888 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/93': No such file or directory 2023-10-10T18:54:28.888 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.889 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/94 2023-10-10T18:54:28.889 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.889 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/94': No such file or directory 2023-10-10T18:54:28.890 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.890 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/95 2023-10-10T18:54:28.890 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.891 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/95': No such file or directory 2023-10-10T18:54:28.891 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.892 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/96 2023-10-10T18:54:28.892 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.892 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/96': No such file or directory 2023-10-10T18:54:28.893 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.893 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/97 2023-10-10T18:54:28.893 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.894 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/97': No such file or directory 2023-10-10T18:54:28.894 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.894 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/98 2023-10-10T18:54:28.895 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.895 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/98': No such file or directory 2023-10-10T18:54:28.896 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.896 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/99 2023-10-10T18:54:28.896 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.897 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/99': No such file or directory 2023-10-10T18:54:28.897 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 100` 2023-10-10T18:54:28.897 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/foo/100 2023-10-10T18:54:28.898 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'No such file' 2023-10-10T18:54:28.898 INFO:tasks.workunit.client.0.smithi175.stdout:stat: cannot stat '.snap/foo/100': No such file or directory 2023-10-10T18:54:28.898 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/foo 2023-10-10T18:54:28.899 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T18:54:28.900 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T18:54:28.900 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T18:54:28.900 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T18:54:29.802 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-10-10T18:54:29.803 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-10-10T18:54:32.521 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T18:54:32.522 INFO:tasks.workunit.client.0.smithi175.stderr:+ retried=false 2023-10-10T18:54:32.522 INFO:tasks.workunit.client.0.smithi175.stderr:+ trap -- retry EXIT 2023-10-10T18:54:32.522 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -rf ceph 2023-10-10T18:54:32.523 INFO:tasks.workunit.client.0.smithi175.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-10-10T18:54:32.525 INFO:tasks.workunit.client.0.smithi175.stderr:Cloning into 'ceph'... 2023-10-10T18:54:49.602 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:54:49.606 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:55:19.723 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:55:19.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:55:25.789 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:26.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.054+0000 7fd74d7c5700 1 Processor -- start 2023-10-10T18:55:26.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.054+0000 7fd74d7c5700 1 -- start start 2023-10-10T18:55:26.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.056+0000 7fd74d7c5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd7480c36b0 0x7fd7480ad740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.056+0000 7fd74d7c5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7481541c0 con 0x7fd7480c2240 2023-10-10T18:55:26.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.056+0000 7fd74d7c5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7480beab0 con 0x7fd7480c32e0 2023-10-10T18:55:26.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.056+0000 7fd74d7c5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7480c8a80 con 0x7fd7480c36b0 2023-10-10T18:55:26.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.056+0000 7fd7467fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd7480c36b0 0x7fd7480ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.056+0000 7fd7477fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd7480c2240 0x7fd74815e3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36028/0 (socket says 172.21.15.175:36028) 2023-10-10T18:55:26.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.056+0000 7fd7477fe700 1 -- 172.21.15.175:0/3703246147 learned_addr learned my addr 172.21.15.175:0/3703246147 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:26.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.057+0000 7fd7467fc700 1 -- 172.21.15.175:0/3703246147 >> v1:172.21.15.191:6789/0 conn(0x7fd7480c32e0 legacy=0x7fd748153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:26.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.057+0000 7fd7467fc700 1 -- 172.21.15.175:0/3703246147 >> v1:172.21.15.175:6789/0 conn(0x7fd7480c2240 legacy=0x7fd74815e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.057+0000 7fd745ffb700 1 -- 172.21.15.175:0/3703246147 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023078762 0 0) 0x7fd7481541c0 con 0x7fd7480c2240 2023-10-10T18:55:26.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.057+0000 7fd7467fc700 1 -- 172.21.15.175:0/3703246147 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7480bd070 con 0x7fd7480c36b0 2023-10-10T18:55:26.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.057+0000 7fd7467fc700 1 --2- 172.21.15.175:0/3703246147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c36b0 0x7fd7480ad740 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd7380098e0 tx=0x7fd7380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a205722ba24116 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:26.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.057+0000 7fd745ffb700 1 -- 172.21.15.175:0/3703246147 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738009e80 con 0x7fd7480c36b0 2023-10-10T18:55:26.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.057+0000 7fd745ffb700 1 -- 172.21.15.175:0/3703246147 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73800b8b0 con 0x7fd7480c36b0 2023-10-10T18:55:26.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.057+0000 7fd745ffb700 1 -- 172.21.15.175:0/3703246147 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73800bc30 con 0x7fd7480c36b0 2023-10-10T18:55:26.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.058+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3703246147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c36b0 msgr2=0x7fd7480ad740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.058+0000 7fd74d7c5700 1 --2- 172.21.15.175:0/3703246147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c36b0 0x7fd7480ad740 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd7380098e0 tx=0x7fd7380091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.058+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3703246147 shutdown_connections 2023-10-10T18:55:26.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.058+0000 7fd74d7c5700 1 --2- 172.21.15.175:0/3703246147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c36b0 0x7fd7480ad740 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.058+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3703246147 >> 172.21.15.175:0/3703246147 conn(0x7fd7480682c0 msgr2=0x7fd7480c8220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:26.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.058+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3703246147 shutdown_connections 2023-10-10T18:55:26.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.058+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3703246147 wait complete. 2023-10-10T18:55:26.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.060+0000 7fd74d7c5700 1 Processor -- start 2023-10-10T18:55:26.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.060+0000 7fd74d7c5700 1 -- start start 2023-10-10T18:55:26.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.061+0000 7fd74d7c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c2240 0x7fd748158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.061+0000 7fd74d7c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7480c32e0 0x7fd7480a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.061+0000 7fd74d7c5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7480c36b0 0x7fd748068b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.061+0000 7fd74d7c5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd748150c60 con 0x7fd7480c2240 2023-10-10T18:55:26.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.061+0000 7fd74d7c5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd748156810 con 0x7fd7480c36b0 2023-10-10T18:55:26.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.061+0000 7fd74d7c5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd748158990 con 0x7fd7480c32e0 2023-10-10T18:55:26.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c2240 0x7fd748158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c2240 0x7fd748158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49642/0 (socket says 172.21.15.175:49642) 2023-10-10T18:55:26.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 -- 172.21.15.175:0/3827273554 learned_addr learned my addr 172.21.15.175:0/3827273554 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:26.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd7477fe700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7480c36b0 0x7fd748068b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd7467fc700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7480c32e0 0x7fd7480a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 -- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7480c32e0 msgr2=0x7fd7480a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7480c32e0 0x7fd7480a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 -- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7480c36b0 msgr2=0x7fd748068b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7480c36b0 0x7fd748068b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 -- 172.21.15.175:0/3827273554 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd738009420 con 0x7fd7480c2240 2023-10-10T18:55:26.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd746ffd700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c2240 0x7fd748158450 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00b630 tx=0x7fd73c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:26.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd72effd700 1 -- 172.21.15.175:0/3827273554 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c00f510 con 0x7fd7480c2240 2023-10-10T18:55:26.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7480ad760 con 0x7fd7480c2240 2023-10-10T18:55:26.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.062+0000 7fd72effd700 1 -- 172.21.15.175:0/3827273554 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c00f690 con 0x7fd7480c2240 2023-10-10T18:55:26.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.063+0000 7fd72effd700 1 -- 172.21.15.175:0/3827273554 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c00fa50 con 0x7fd7480c2240 2023-10-10T18:55:26.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.063+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7480ad1a0 con 0x7fd7480c2240 2023-10-10T18:55:26.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.064+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd748067570 con 0x7fd7480c2240 2023-10-10T18:55:26.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.064+0000 7fd72effd700 1 -- 172.21.15.175:0/3827273554 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd73c01a400 con 0x7fd7480c2240 2023-10-10T18:55:26.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.065+0000 7fd72effd700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd734066f60 0x7fd734069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.065+0000 7fd72effd700 1 -- 172.21.15.175:0/3827273554 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6193+0+0 (secure 0 0 0) 0x7fd73c084280 con 0x7fd7480c2240 2023-10-10T18:55:26.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.066+0000 7fd7467fc700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd734066f60 0x7fd734069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.067+0000 7fd7467fc700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd734066f60 0x7fd734069410 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd7380030c0 tx=0x7fd738016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:26.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.074+0000 7fd72effd700 1 -- 172.21.15.175:0/3827273554 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd73c051300 con 0x7fd7480c2240 2023-10-10T18:55:26.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.359+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7480c8f40 con 0x7fd7480c2240 2023-10-10T18:55:26.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.359+0000 7fd72effd700 1 -- 172.21.15.175:0/3827273554 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6195 (secure 0 0 0) 0x7fd73c054920 con 0x7fd7480c2240 2023-10-10T18:55:26.360 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:26.361 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5332,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:07.537641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:26.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd734066f60 msgr2=0x7fd734069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd734066f60 0x7fd734069410 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd7380030c0 tx=0x7fd738016000 comp rx=0 tx=0).stop 2023-10-10T18:55:26.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c2240 msgr2=0x7fd748158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c2240 0x7fd748158450 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00b630 tx=0x7fd73c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 shutdown_connections 2023-10-10T18:55:26.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd734066f60 0x7fd734069410 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7480c36b0 0x7fd748068b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7480c2240 0x7fd748158450 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.361+0000 7fd74d7c5700 1 --2- 172.21.15.175:0/3827273554 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7480c32e0 0x7fd7480a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.362+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 >> 172.21.15.175:0/3827273554 conn(0x7fd7480682c0 msgr2=0x7fd748153800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:26.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.362+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 shutdown_connections 2023-10-10T18:55:26.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.362+0000 7fd74d7c5700 1 -- 172.21.15.175:0/3827273554 wait complete. 2023-10-10T18:55:26.365 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 81 2023-10-10T18:55:26.375 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=1) 2023-10-10T18:55:26.376 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-10T18:55:26.376 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T18:55:26.389 INFO:tasks.ceph.mds.e:Stopped 2023-10-10T18:55:26.390 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:26.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.548+0000 7f18aa92a700 1 Processor -- start 2023-10-10T18:55:26.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.548+0000 7f18aa92a700 1 -- start start 2023-10-10T18:55:26.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.549+0000 7f18aa92a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18a41593f0 0x7f18a406f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.549+0000 7f18aa92a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a40c34e0 con 0x7f18a41597c0 2023-10-10T18:55:26.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.549+0000 7f18aa92a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a4150330 con 0x7f18a40c8040 2023-10-10T18:55:26.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.549+0000 7f18aa92a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a40c6430 con 0x7f18a41593f0 2023-10-10T18:55:26.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.549+0000 7f18a9928700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18a41593f0 0x7f18a406f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.549+0000 7f18a9127700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f18a41597c0 0x7f18a40c7930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36038/0 (socket says 172.21.15.175:36038) 2023-10-10T18:55:26.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.549+0000 7f18a9127700 1 -- 172.21.15.175:0/2580214917 learned_addr learned my addr 172.21.15.175:0/2580214917 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:26.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.550+0000 7f18a8926700 1 -- 172.21.15.175:0/2580214917 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945798068 0 0) 0x7f18a4150330 con 0x7f18a40c8040 2023-10-10T18:55:26.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.550+0000 7f18a9928700 1 -- 172.21.15.175:0/2580214917 >> v1:172.21.15.191:6789/0 conn(0x7f18a40c8040 legacy=0x7f18a40c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.550+0000 7f18a9928700 1 -- 172.21.15.175:0/2580214917 >> v1:172.21.15.175:6789/0 conn(0x7f18a41597c0 legacy=0x7f18a40c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.550+0000 7f18a9928700 1 -- 172.21.15.175:0/2580214917 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a40c2da0 con 0x7f18a41593f0 2023-10-10T18:55:26.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.550+0000 7f18a9928700 1 --2- 172.21.15.175:0/2580214917 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a41593f0 0x7f18a406f7e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f18a00098e0 tx=0x7f18a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddcf750f471c6f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:26.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.550+0000 7f18a8926700 1 -- 172.21.15.175:0/2580214917 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0009e80 con 0x7f18a41593f0 2023-10-10T18:55:26.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.550+0000 7f18a8926700 1 -- 172.21.15.175:0/2580214917 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a000b860 con 0x7f18a41593f0 2023-10-10T18:55:26.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.550+0000 7f18a8926700 1 -- 172.21.15.175:0/2580214917 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a000bbe0 con 0x7f18a41593f0 2023-10-10T18:55:26.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.551+0000 7f18aa92a700 1 -- 172.21.15.175:0/2580214917 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a41593f0 msgr2=0x7f18a406f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.551+0000 7f18aa92a700 1 --2- 172.21.15.175:0/2580214917 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a41593f0 0x7f18a406f7e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f18a00098e0 tx=0x7f18a00091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.551+0000 7f18aa92a700 1 -- 172.21.15.175:0/2580214917 shutdown_connections 2023-10-10T18:55:26.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.551+0000 7f18aa92a700 1 --2- 172.21.15.175:0/2580214917 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a41593f0 0x7f18a406f7e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.551+0000 7f18aa92a700 1 -- 172.21.15.175:0/2580214917 >> 172.21.15.175:0/2580214917 conn(0x7f18a40c9500 msgr2=0x7f18a40c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.551+0000 7f18aa92a700 1 -- 172.21.15.175:0/2580214917 shutdown_connections 2023-10-10T18:55:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.551+0000 7f18aa92a700 1 -- 172.21.15.175:0/2580214917 wait complete. 2023-10-10T18:55:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.552+0000 7f18aa92a700 1 Processor -- start 2023-10-10T18:55:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.552+0000 7f18aa92a700 1 -- start start 2023-10-10T18:55:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18aa92a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a40c8040 0x7f18a40c1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18aa92a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18a41593f0 0x7f18a414b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18aa92a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18a41597c0 0x7f18a414b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18aa92a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40bc260 con 0x7f18a40c8040 2023-10-10T18:55:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18aa92a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40688e0 con 0x7f18a41593f0 2023-10-10T18:55:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18aa92a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40bb650 con 0x7f18a41597c0 2023-10-10T18:55:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a40c8040 0x7f18a40c1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a40c8040 0x7f18a40c1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49668/0 (socket says 172.21.15.175:49668) 2023-10-10T18:55:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 -- 172.21.15.175:0/2488040160 learned_addr learned my addr 172.21.15.175:0/2488040160 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9127700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18a41593f0 0x7f18a414b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18aa129700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18a41597c0 0x7f18a414b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 -- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18a41597c0 msgr2=0x7f18a414b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18a41597c0 0x7f18a414b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 -- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18a41593f0 msgr2=0x7f18a414b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18a41593f0 0x7f18a414b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 -- 172.21.15.175:0/2488040160 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0009420 con 0x7f18a40c8040 2023-10-10T18:55:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18a9928700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a40c8040 0x7f18a40c1090 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f18a406cf00 tx=0x7f18a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f1899ffb700 1 -- 172.21.15.175:0/2488040160 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a001f070 con 0x7f18a40c8040 2023-10-10T18:55:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f1899ffb700 1 -- 172.21.15.175:0/2488040160 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a0011420 con 0x7f18a40c8040 2023-10-10T18:55:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a40bb930 con 0x7f18a40c8040 2023-10-10T18:55:26.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.554+0000 7f1899ffb700 1 -- 172.21.15.175:0/2488040160 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a000bbe0 con 0x7f18a40c8040 2023-10-10T18:55:26.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.555+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a40bbef0 con 0x7f18a40c8040 2023-10-10T18:55:26.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.556+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a4067570 con 0x7f18a40c8040 2023-10-10T18:55:26.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.556+0000 7f1899ffb700 1 -- 172.21.15.175:0/2488040160 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f18a0011a10 con 0x7f18a40c8040 2023-10-10T18:55:26.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.557+0000 7f1899ffb700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1888066f60 0x7f1888069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:26.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.557+0000 7f1899ffb700 1 -- 172.21.15.175:0/2488040160 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6193+0+0 (secure 0 0 0) 0x7f18a00860c0 con 0x7f18a40c8040 2023-10-10T18:55:26.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.557+0000 7f18a9127700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1888066f60 0x7f1888069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:26.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.557+0000 7f18a9127700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1888066f60 0x7f1888069410 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f18a40c7a70 tx=0x7f1894009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:26.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.564+0000 7f1899ffb700 1 -- 172.21.15.175:0/2488040160 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f18a0053140 con 0x7f18a40c8040 2023-10-10T18:55:26.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.798+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18a4165890 con 0x7f18a40c8040 2023-10-10T18:55:26.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.798+0000 7f1899ffb700 1 -- 172.21.15.175:0/2488040160 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6195 (secure 0 0 0) 0x7f18a0015090 con 0x7f18a40c8040 2023-10-10T18:55:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:26.800 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5332,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:07.537641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:26.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1888066f60 msgr2=0x7f1888069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1888066f60 0x7f1888069410 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f18a40c7a70 tx=0x7f1894009040 comp rx=0 tx=0).stop 2023-10-10T18:55:26.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a40c8040 msgr2=0x7f18a40c1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:26.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a40c8040 0x7f18a40c1090 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f18a406cf00 tx=0x7f18a0017000 comp rx=0 tx=0).stop 2023-10-10T18:55:26.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 shutdown_connections 2023-10-10T18:55:26.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1888066f60 0x7f1888069410 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18a41597c0 0x7f18a414b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18a40c8040 0x7f18a40c1090 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 --2- 172.21.15.175:0/2488040160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18a41593f0 0x7f18a414b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:26.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 >> 172.21.15.175:0/2488040160 conn(0x7f18a40c9500 msgr2=0x7f18a40c3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:26.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 shutdown_connections 2023-10-10T18:55:26.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:26.801+0000 7f18aa92a700 1 -- 172.21.15.175:0/2488040160 wait complete. 2023-10-10T18:55:26.805 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 81 2023-10-10T18:55:26.817 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T18:55:28.818 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:28.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.936+0000 7f71ab59e700 1 Processor -- start 2023-10-10T18:55:28.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.936+0000 7f71ab59e700 1 -- start start 2023-10-10T18:55:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.937+0000 7f71ab59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac06e520 0x7f71ac153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.937+0000 7f71ab59e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac06c5e0 con 0x7f71ac0c2b50 2023-10-10T18:55:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71ab59e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac0c3100 con 0x7f71ac06e8f0 2023-10-10T18:55:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71ab59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0c33e0 con 0x7f71ac06e520 2023-10-10T18:55:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71aad9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f71ac0c2b50 0x7f71ac14d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36040/0 (socket says 172.21.15.175:36040) 2023-10-10T18:55:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71aad9d700 1 -- 172.21.15.175:0/81163074 learned_addr learned my addr 172.21.15.175:0/81163074 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71aa59c700 1 --2- 172.21.15.175:0/81163074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac06e520 0x7f71ac153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71a959a700 1 -- 172.21.15.175:0/81163074 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483543218 0 0) 0x7f71ac0c3100 con 0x7f71ac06e8f0 2023-10-10T18:55:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71a959a700 1 -- 172.21.15.175:0/81163074 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7194003570 con 0x7f71ac06e8f0 2023-10-10T18:55:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71a959a700 1 -- 172.21.15.175:0/81163074 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17169807 0 0) 0x7f71ac06c5e0 con 0x7f71ac0c2b50 2023-10-10T18:55:28.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71a959a700 1 -- 172.21.15.175:0/81163074 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ac0c3100 con 0x7f71ac0c2b50 2023-10-10T18:55:28.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71aa59c700 1 -- 172.21.15.175:0/81163074 >> v1:172.21.15.191:6789/0 conn(0x7f71ac06e8f0 legacy=0x7f71ac0c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:28.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71aa59c700 1 -- 172.21.15.175:0/81163074 >> v1:172.21.15.175:6789/0 conn(0x7f71ac0c2b50 legacy=0x7f71ac14d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:28.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71aa59c700 1 -- 172.21.15.175:0/81163074 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac0c36c0 con 0x7f71ac06e520 2023-10-10T18:55:28.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.938+0000 7f71aa59c700 1 --2- 172.21.15.175:0/81163074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac06e520 0x7f71ac153bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f719c009910 tx=0x7f719c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=456f71e79981b13d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:28.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.939+0000 7f71a959a700 1 -- 172.21.15.175:0/81163074 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c011630 con 0x7f71ac06e520 2023-10-10T18:55:28.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.939+0000 7f71a959a700 1 -- 172.21.15.175:0/81163074 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c011790 con 0x7f71ac06e520 2023-10-10T18:55:28.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.939+0000 7f71a959a700 1 -- 172.21.15.175:0/81163074 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c011b30 con 0x7f71ac06e520 2023-10-10T18:55:28.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.939+0000 7f71ab59e700 1 -- 172.21.15.175:0/81163074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac06e520 msgr2=0x7f71ac153bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:28.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.939+0000 7f71ab59e700 1 --2- 172.21.15.175:0/81163074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac06e520 0x7f71ac153bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f719c009910 tx=0x7f719c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.940+0000 7f71ab59e700 1 -- 172.21.15.175:0/81163074 shutdown_connections 2023-10-10T18:55:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.940+0000 7f71ab59e700 1 --2- 172.21.15.175:0/81163074 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac06e520 0x7f71ac153bd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.940+0000 7f71ab59e700 1 -- 172.21.15.175:0/81163074 >> 172.21.15.175:0/81163074 conn(0x7f71ac063230 msgr2=0x7f71ac06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.940+0000 7f71ab59e700 1 -- 172.21.15.175:0/81163074 shutdown_connections 2023-10-10T18:55:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.940+0000 7f71ab59e700 1 -- 172.21.15.175:0/81163074 wait complete. 2023-10-10T18:55:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.941+0000 7f71ab59e700 1 Processor -- start 2023-10-10T18:55:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.941+0000 7f71ab59e700 1 -- start start 2023-10-10T18:55:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71ab59e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71ac06e520 0x7f71ac142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71ab59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71ac06e8f0 0x7f71ac1535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71ab59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac0c2b50 0x7f71ac150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71ab59e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac15a060 con 0x7f71ac06e8f0 2023-10-10T18:55:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71ab59e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac142660 con 0x7f71ac06e520 2023-10-10T18:55:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71ab59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac1570b0 con 0x7f71ac0c2b50 2023-10-10T18:55:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71ac06e8f0 0x7f71ac1535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:28.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71ac06e8f0 0x7f71ac1535d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49682/0 (socket says 172.21.15.175:49682) 2023-10-10T18:55:28.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 -- 172.21.15.175:0/266408454 learned_addr learned my addr 172.21.15.175:0/266408454 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:28.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71aa59c700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71ac06e520 0x7f71ac142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:28.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71aad9d700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac0c2b50 0x7f71ac150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:28.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 -- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac0c2b50 msgr2=0x7f71ac150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:28.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac0c2b50 0x7f71ac150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:28.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 -- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71ac06e520 msgr2=0x7f71ac142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:28.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71ac06e520 0x7f71ac142120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:28.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 -- 172.21.15.175:0/266408454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c009420 con 0x7f71ac06e8f0 2023-10-10T18:55:28.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71aad9d700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac0c2b50 0x7f71ac150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:55:28.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f71a9d9b700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71ac06e8f0 0x7f71ac1535d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f71a000c790 tx=0x7f71a0007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:28.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f719a7fc700 1 -- 172.21.15.175:0/266408454 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a0010510 con 0x7f71ac06e8f0 2023-10-10T18:55:28.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f719a7fc700 1 -- 172.21.15.175:0/266408454 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a0010670 con 0x7f71ac06e8f0 2023-10-10T18:55:28.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.943+0000 7f719a7fc700 1 -- 172.21.15.175:0/266408454 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a0010a10 con 0x7f71ac06e8f0 2023-10-10T18:55:28.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.944+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac157350 con 0x7f71ac06e8f0 2023-10-10T18:55:28.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.944+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac159aa0 con 0x7f71ac06e8f0 2023-10-10T18:55:28.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.945+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ac067570 con 0x7f71ac06e8f0 2023-10-10T18:55:28.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.946+0000 7f719a7fc700 1 -- 172.21.15.175:0/266408454 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f71a001b400 con 0x7f71ac06e8f0 2023-10-10T18:55:28.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.946+0000 7f719a7fc700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f718c066f60 0x7f718c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:28.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.946+0000 7f719a7fc700 1 -- 172.21.15.175:0/266408454 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6193+0+0 (secure 0 0 0) 0x7f71a0084490 con 0x7f71ac06e8f0 2023-10-10T18:55:28.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.946+0000 7f71aa59c700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f718c066f60 0x7f718c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:28.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.947+0000 7f71aa59c700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f718c066f60 0x7f718c069410 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f719c009860 tx=0x7f719c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:28.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:28.954+0000 7f719a7fc700 1 -- 172.21.15.175:0/266408454 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71a0051510 con 0x7f71ac06e8f0 2023-10-10T18:55:29.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.189+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71ac14a710 con 0x7f71ac06e8f0 2023-10-10T18:55:29.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.190+0000 7f719a7fc700 1 -- 172.21.15.175:0/266408454 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6195 (secure 0 0 0) 0x7f71a0054b30 con 0x7f71ac06e8f0 2023-10-10T18:55:29.191 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:29.191 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5332,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:07.537641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:29.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f718c066f60 msgr2=0x7f718c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:29.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f718c066f60 0x7f718c069410 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f719c009860 tx=0x7f719c017000 comp rx=0 tx=0).stop 2023-10-10T18:55:29.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71ac06e8f0 msgr2=0x7f71ac1535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:29.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71ac06e8f0 0x7f71ac1535d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f71a000c790 tx=0x7f71a0007e30 comp rx=0 tx=0).stop 2023-10-10T18:55:29.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 shutdown_connections 2023-10-10T18:55:29.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f718c066f60 0x7f718c069410 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:29.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71ac0c2b50 0x7f71ac150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:29.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71ac06e520 0x7f71ac142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:29.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 --2- 172.21.15.175:0/266408454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71ac06e8f0 0x7f71ac1535d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:29.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 >> 172.21.15.175:0/266408454 conn(0x7f71ac063230 msgr2=0x7f71ac154f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:29.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 shutdown_connections 2023-10-10T18:55:29.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:29.193+0000 7f71ab59e700 1 -- 172.21.15.175:0/266408454 wait complete. 2023-10-10T18:55:29.196 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 81 2023-10-10T18:55:29.210 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T18:55:31.211 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:31.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.330+0000 7fc5749f3700 1 Processor -- start 2023-10-10T18:55:31.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.330+0000 7fc5749f3700 1 -- start start 2023-10-10T18:55:31.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.330+0000 7fc5749f3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc5701593f0 0x7fc57006f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:31.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.330+0000 7fc5749f3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc5700c8040 0x7fc5700c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:31.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.330+0000 7fc5749f3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5700c34e0 con 0x7fc5701597c0 2023-10-10T18:55:31.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.330+0000 7fc5749f3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5700c6430 con 0x7fc5701593f0 2023-10-10T18:55:31.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.330+0000 7fc5749f3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc570150330 con 0x7fc5700c8040 2023-10-10T18:55:31.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc5701593f0 0x7fc57006f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:31.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc5701593f0 0x7fc57006f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49696/0 (socket says 172.21.15.175:49696) 2023-10-10T18:55:31.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56ed9d700 1 -- 172.21.15.175:0/3600374148 learned_addr learned my addr 172.21.15.175:0/3600374148 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:31.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56f59e700 1 --2- 172.21.15.175:0/3600374148 >> v2:172.21.15.191:3300/0 conn(0x7fc5700c8040 0x7fc5700c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:31.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56ed9d700 1 -- 172.21.15.175:0/3600374148 >> v2:172.21.15.191:3300/0 conn(0x7fc5700c8040 msgr2=0x7fc5700c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:31.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56ed9d700 1 --2- 172.21.15.175:0/3600374148 >> v2:172.21.15.191:3300/0 conn(0x7fc5700c8040 0x7fc5700c3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56ed9d700 1 -- 172.21.15.175:0/3600374148 >> v1:172.21.15.191:6789/0 conn(0x7fc5701597c0 legacy=0x7fc5700c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:31.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56ed9d700 1 -- 172.21.15.175:0/3600374148 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700c2da0 con 0x7fc5701593f0 2023-10-10T18:55:31.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56ed9d700 1 --2- 172.21.15.175:0/3600374148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5701593f0 0x7fc57006f7e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc57006d560 tx=0x7fc560009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cffaefb98749cc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:31.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56dd9b700 1 -- 172.21.15.175:0/3600374148 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560009e70 con 0x7fc5701593f0 2023-10-10T18:55:31.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56dd9b700 1 -- 172.21.15.175:0/3600374148 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56000b860 con 0x7fc5701593f0 2023-10-10T18:55:31.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.331+0000 7fc56dd9b700 1 -- 172.21.15.175:0/3600374148 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56000bc00 con 0x7fc5701593f0 2023-10-10T18:55:31.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.332+0000 7fc5749f3700 1 -- 172.21.15.175:0/3600374148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5701593f0 msgr2=0x7fc57006f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:31.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.332+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3600374148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5701593f0 0x7fc57006f7e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc57006d560 tx=0x7fc560009250 comp rx=0 tx=0).stop 2023-10-10T18:55:31.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.332+0000 7fc5749f3700 1 -- 172.21.15.175:0/3600374148 shutdown_connections 2023-10-10T18:55:31.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.332+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3600374148 >> v2:172.21.15.191:3300/0 conn(0x7fc5700c8040 0x7fc5700c3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.332+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3600374148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5701593f0 0x7fc57006f7e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.332+0000 7fc5749f3700 1 -- 172.21.15.175:0/3600374148 >> 172.21.15.175:0/3600374148 conn(0x7fc5700c9500 msgr2=0x7fc5700c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:31.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.332+0000 7fc5749f3700 1 -- 172.21.15.175:0/3600374148 shutdown_connections 2023-10-10T18:55:31.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.332+0000 7fc5749f3700 1 -- 172.21.15.175:0/3600374148 wait complete. 2023-10-10T18:55:31.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.333+0000 7fc5749f3700 1 Processor -- start 2023-10-10T18:55:31.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.333+0000 7fc5749f3700 1 -- start start 2023-10-10T18:55:31.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc5749f3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5700c8040 0x7fc570150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:31.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc5749f3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5701593f0 0x7fc570150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:31.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc5749f3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5701597c0 0x7fc570151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:31.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc5749f3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701661e0 con 0x7fc5700c8040 2023-10-10T18:55:31.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc5749f3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc570151670 con 0x7fc5701597c0 2023-10-10T18:55:31.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc5749f3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701666b0 con 0x7fc5701593f0 2023-10-10T18:55:31.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5700c8040 0x7fc570150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:31.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5700c8040 0x7fc570150a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49702/0 (socket says 172.21.15.175:49702) 2023-10-10T18:55:31.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 -- 172.21.15.175:0/3936275237 learned_addr learned my addr 172.21.15.175:0/3936275237 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:31.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56f59e700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5701597c0 0x7fc570151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:31.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56e59c700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5701593f0 0x7fc570150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 -- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5701593f0 msgr2=0x7fc570150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5701593f0 0x7fc570150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 -- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5701597c0 msgr2=0x7fc570151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5701597c0 0x7fc570151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 -- 172.21.15.175:0/3936275237 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600094f0 con 0x7fc5700c8040 2023-10-10T18:55:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.334+0000 7fc56ed9d700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5700c8040 0x7fc570150a20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc560006fd0 tx=0x7fc560017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.335+0000 7fc556ffd700 1 -- 172.21.15.175:0/3936275237 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56000bbe0 con 0x7fc5700c8040 2023-10-10T18:55:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.335+0000 7fc556ffd700 1 -- 172.21.15.175:0/3936275237 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc560011420 con 0x7fc5700c8040 2023-10-10T18:55:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.335+0000 7fc556ffd700 1 -- 172.21.15.175:0/3936275237 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5600117c0 con 0x7fc5700c8040 2023-10-10T18:55:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.335+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5700688f0 con 0x7fc5700c8040 2023-10-10T18:55:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.335+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc570068b90 con 0x7fc5700c8040 2023-10-10T18:55:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.336+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc570151950 con 0x7fc5700c8040 2023-10-10T18:55:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.336+0000 7fc556ffd700 1 -- 172.21.15.175:0/3936275237 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc560011bf0 con 0x7fc5700c8040 2023-10-10T18:55:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.337+0000 7fc556ffd700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc55c066f60 0x7fc55c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.337+0000 7fc556ffd700 1 -- 172.21.15.175:0/3936275237 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6193+0+0 (secure 0 0 0) 0x7fc560085f80 con 0x7fc5700c8040 2023-10-10T18:55:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.337+0000 7fc56e59c700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc55c066f60 0x7fc55c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:31.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.337+0000 7fc56e59c700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc55c066f60 0x7fc55c069410 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc570156c60 tx=0x7fc564007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:31.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.345+0000 7fc556ffd700 1 -- 172.21.15.175:0/3936275237 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5600b75a0 con 0x7fc5700c8040 2023-10-10T18:55:31.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.601+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5701514a0 con 0x7fc5700c8040 2023-10-10T18:55:31.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.602+0000 7fc556ffd700 1 -- 172.21.15.175:0/3936275237 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6195 (secure 0 0 0) 0x7fc560053000 con 0x7fc5700c8040 2023-10-10T18:55:31.602 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:31.602 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5332,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:07.537641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:31.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.604+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc55c066f60 msgr2=0x7fc55c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.604+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc55c066f60 0x7fc55c069410 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc570156c60 tx=0x7fc564007ec0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.604+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5700c8040 msgr2=0x7fc570150a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.604+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5700c8040 0x7fc570150a20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc560006fd0 tx=0x7fc560017000 comp rx=0 tx=0).stop 2023-10-10T18:55:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.605+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 shutdown_connections 2023-10-10T18:55:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.605+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc55c066f60 0x7fc55c069410 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.605+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5701597c0 0x7fc570151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.605+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5700c8040 0x7fc570150a20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.605+0000 7fc5749f3700 1 --2- 172.21.15.175:0/3936275237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5701593f0 0x7fc570150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:31.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.605+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 >> 172.21.15.175:0/3936275237 conn(0x7fc5700c9500 msgr2=0x7fc5700c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:31.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.605+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 shutdown_connections 2023-10-10T18:55:31.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:31.605+0000 7fc5749f3700 1 -- 172.21.15.175:0/3936275237 wait complete. 2023-10-10T18:55:31.607 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 81 2023-10-10T18:55:31.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T18:55:33.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:33.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.739+0000 7f8d0d987700 1 Processor -- start 2023-10-10T18:55:33.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.739+0000 7f8d0d987700 1 -- start start 2023-10-10T18:55:33.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d0d987700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c7cf0 0x7f8d081533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:33.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d0d987700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8d080c5d30 0x7f8d0806d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:33.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d0d987700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8d0806def0 0x7f8d0814c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:33.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d0d987700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0806f770 con 0x7f8d080c5d30 2023-10-10T18:55:33.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d0d987700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d081568b0 con 0x7f8d0806def0 2023-10-10T18:55:33.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d0d987700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d080c4950 con 0x7f8d080c7cf0 2023-10-10T18:55:33.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d07fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8d080c5d30 0x7f8d0806d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:33.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d07fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8d080c5d30 0x7f8d0806d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49718/0 (socket says 172.21.15.175:49718) 2023-10-10T18:55:33.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d07fff700 1 -- 172.21.15.175:0/2322937401 learned_addr learned my addr 172.21.15.175:0/2322937401 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:33.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d0c985700 1 --2- 172.21.15.175:0/2322937401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c7cf0 0x7f8d081533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:33.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.740+0000 7f8d0d186700 1 --2- 172.21.15.175:0/2322937401 >> v2:172.21.15.191:3300/0 conn(0x7f8d0806def0 0x7f8d0814c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:33.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d07fff700 1 -- 172.21.15.175:0/2322937401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c7cf0 msgr2=0x7f8d081533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:33.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d07fff700 1 --2- 172.21.15.175:0/2322937401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c7cf0 0x7f8d081533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d07fff700 1 -- 172.21.15.175:0/2322937401 >> v2:172.21.15.191:3300/0 conn(0x7f8d0806def0 msgr2=0x7f8d0814c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:33.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d07fff700 1 --2- 172.21.15.175:0/2322937401 >> v2:172.21.15.191:3300/0 conn(0x7f8d0806def0 0x7f8d0814c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d07fff700 1 -- 172.21.15.175:0/2322937401 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08156d40 con 0x7f8d080c5d30 2023-10-10T18:55:33.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d07fff700 1 --2- 172.21.15.175:0/2322937401 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d080c5d30 0x7f8d0806d9b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8d080bfa30 tx=0x7f8cf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3141c619e5634757 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:33.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d077fe700 1 -- 172.21.15.175:0/2322937401 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf400b860 con 0x7f8d080c5d30 2023-10-10T18:55:33.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d077fe700 1 -- 172.21.15.175:0/2322937401 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf400b9c0 con 0x7f8d080c5d30 2023-10-10T18:55:33.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.741+0000 7f8d077fe700 1 -- 172.21.15.175:0/2322937401 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf400bd60 con 0x7f8d080c5d30 2023-10-10T18:55:33.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 -- 172.21.15.175:0/2322937401 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d080c5d30 msgr2=0x7f8d0806d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:33.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2322937401 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d080c5d30 0x7f8d0806d9b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8d080bfa30 tx=0x7f8cf4009250 comp rx=0 tx=0).stop 2023-10-10T18:55:33.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 -- 172.21.15.175:0/2322937401 shutdown_connections 2023-10-10T18:55:33.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2322937401 >> v2:172.21.15.191:3300/0 conn(0x7f8d0806def0 0x7f8d0814c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2322937401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c7cf0 0x7f8d081533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2322937401 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d080c5d30 0x7f8d0806d9b0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 -- 172.21.15.175:0/2322937401 >> 172.21.15.175:0/2322937401 conn(0x7f8d080c9500 msgr2=0x7f8d080c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:33.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 -- 172.21.15.175:0/2322937401 shutdown_connections 2023-10-10T18:55:33.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.742+0000 7f8d0d987700 1 -- 172.21.15.175:0/2322937401 wait complete. 2023-10-10T18:55:33.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.743+0000 7f8d0d987700 1 Processor -- start 2023-10-10T18:55:33.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.743+0000 7f8d0d987700 1 -- start start 2023-10-10T18:55:33.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0d987700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0806def0 0x7f8d08147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:33.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0d987700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c5d30 0x7f8d08163cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:33.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0d987700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d080c7cf0 0x7f8d08154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:33.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0d987700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d08068650 con 0x7f8d0806def0 2023-10-10T18:55:33.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0d987700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d081634d0 con 0x7f8d080c7cf0 2023-10-10T18:55:33.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0d987700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d08167590 con 0x7f8d080c5d30 2023-10-10T18:55:33.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0806def0 0x7f8d08147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:33.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0806def0 0x7f8d08147880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49720/0 (socket says 172.21.15.175:49720) 2023-10-10T18:55:33.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 -- 172.21.15.175:0/2741596859 learned_addr learned my addr 172.21.15.175:0/2741596859 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:33.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d07fff700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c5d30 0x7f8d08163cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:33.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0d186700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d080c7cf0 0x7f8d08154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:33.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 -- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c5d30 msgr2=0x7f8d08163cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:33.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c5d30 0x7f8d08163cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 -- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d080c7cf0 msgr2=0x7f8d08154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:33.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d080c7cf0 0x7f8d08154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 -- 172.21.15.175:0/2741596859 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf40094f0 con 0x7f8d0806def0 2023-10-10T18:55:33.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.744+0000 7f8d0c985700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0806def0 0x7f8d08147880 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00d860 tx=0x7f8cfc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:33.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.745+0000 7f8d04ff9700 1 -- 172.21.15.175:0/2741596859 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc0125f0 con 0x7f8d0806def0 2023-10-10T18:55:33.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.745+0000 7f8d04ff9700 1 -- 172.21.15.175:0/2741596859 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc012750 con 0x7f8d0806def0 2023-10-10T18:55:33.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.745+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d080688f0 con 0x7f8d0806def0 2023-10-10T18:55:33.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.745+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0814dd40 con 0x7f8d0806def0 2023-10-10T18:55:33.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.745+0000 7f8d04ff9700 1 -- 172.21.15.175:0/2741596859 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc012b10 con 0x7f8d0806def0 2023-10-10T18:55:33.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.746+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce80052b0 con 0x7f8d0806def0 2023-10-10T18:55:33.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.746+0000 7f8d04ff9700 1 -- 172.21.15.175:0/2741596859 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8cfc01e460 con 0x7f8d0806def0 2023-10-10T18:55:33.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.747+0000 7f8d04ff9700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cec067030 0x7f8cec0694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:33.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.747+0000 7f8d04ff9700 1 -- 172.21.15.175:0/2741596859 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6193+0+0 (secure 0 0 0) 0x7f8cfc084cc0 con 0x7f8d0806def0 2023-10-10T18:55:33.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.747+0000 7f8d07fff700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cec067030 0x7f8cec0694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:33.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.747+0000 7f8d07fff700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cec067030 0x7f8cec0694e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8d080bfa30 tx=0x7f8cf4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:33.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.755+0000 7f8d04ff9700 1 -- 172.21.15.175:0/2741596859 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8cfc055850 con 0x7f8d0806def0 2023-10-10T18:55:33.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.988+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ce8005560 con 0x7f8d0806def0 2023-10-10T18:55:33.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.989+0000 7f8d04ff9700 1 -- 172.21.15.175:0/2741596859 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6195 (secure 0 0 0) 0x7f8cfc0553e0 con 0x7f8d0806def0 2023-10-10T18:55:33.989 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:33.989 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5332,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:07.537641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:33.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.991+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cec067030 msgr2=0x7f8cec0694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:33.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.991+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cec067030 0x7f8cec0694e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8d080bfa30 tx=0x7f8cf4017000 comp rx=0 tx=0).stop 2023-10-10T18:55:33.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.991+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0806def0 msgr2=0x7f8d08147880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:33.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.991+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0806def0 0x7f8d08147880 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00d860 tx=0x7f8cfc008040 comp rx=0 tx=0).stop 2023-10-10T18:55:33.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.992+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 shutdown_connections 2023-10-10T18:55:33.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.992+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cec067030 0x7f8cec0694e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.992+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d080c7cf0 0x7f8d08154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.992+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0806def0 0x7f8d08147880 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.992+0000 7f8d0d987700 1 --2- 172.21.15.175:0/2741596859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d080c5d30 0x7f8d08163cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:33.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.992+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 >> 172.21.15.175:0/2741596859 conn(0x7f8d080c9500 msgr2=0x7f8d080c3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:33.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.992+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 shutdown_connections 2023-10-10T18:55:33.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:33.992+0000 7f8d0d987700 1 -- 172.21.15.175:0/2741596859 wait complete. 2023-10-10T18:55:33.994 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 81 2023-10-10T18:55:34.004 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T18:55:36.005 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:36.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.123+0000 7f4fa5669700 1 Processor -- start 2023-10-10T18:55:36.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.123+0000 7f4fa5669700 1 -- start start 2023-10-10T18:55:36.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4fa5669700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 0x7f4fa00c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:36.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4fa5669700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa00c40a0 con 0x7f4fa014bf20 2023-10-10T18:55:36.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4fa5669700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa00c4c20 con 0x7f4fa006e390 2023-10-10T18:55:36.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4fa5669700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa006f200 con 0x7f4fa00c50b0 2023-10-10T18:55:36.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4f9ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f4fa014bf20 0x7f4fa00c6890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40414/0 (socket says 172.21.15.175:40414) 2023-10-10T18:55:36.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4f9ffff700 1 -- 172.21.15.175:0/2071063873 learned_addr learned my addr 172.21.15.175:0/2071063873 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:36.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4f9f7fe700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 0x7f4fa00c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:36.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560170246 0 0) 0x7f4fa00c40a0 con 0x7f4fa014bf20 2023-10-10T18:55:36.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f88003570 con 0x7f4fa014bf20 2023-10-10T18:55:36.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.124+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223591038 0 0) 0x7f4fa00c4c20 con 0x7f4fa006e390 2023-10-10T18:55:36.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa00c40a0 con 0x7f4fa006e390 2023-10-10T18:55:36.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2995884961 0 0) 0x7f4f88003570 con 0x7f4fa014bf20 2023-10-10T18:55:36.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 msgr2=0x7f4fa00c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:36.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 0x7f4fa00c3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 >> v1:172.21.15.191:6789/0 conn(0x7f4fa006e390 legacy=0x7f4fa0155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:36.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa006e900 con 0x7f4fa014bf20 2023-10-10T18:55:36.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f4f90001470 con 0x7f4fa014bf20 2023-10-10T18:55:36.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9f7fe700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 0x7f4fa00c3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:55:36.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 >> v1:172.21.15.175:6789/0 conn(0x7f4fa014bf20 legacy=0x7f4fa00c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:36.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f88003c20 0x7f4f88004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:36.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f88004810 0x7f4f88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:36.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 0x7f4f880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:36.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa006f200 con 0x7f4f88004810 2023-10-10T18:55:36.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8800bb10 con 0x7f4f88003c20 2023-10-10T18:55:36.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8800bcd0 con 0x7f4fa00c50b0 2023-10-10T18:55:36.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9f7fe700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f88004810 0x7f4f88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:36.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4fa4e68700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 0x7f4f880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:36.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9ffff700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f88003c20 0x7f4f88004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:36.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9f7fe700 1 -- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 msgr2=0x7f4f880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:36.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9f7fe700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fa00c50b0 0x7f4f880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9f7fe700 1 -- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f88003c20 msgr2=0x7f4f88004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:36.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9f7fe700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f88003c20 0x7f4f88004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9f7fe700 1 -- 172.21.15.175:0/2071063873 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f94009420 con 0x7f4f88004810 2023-10-10T18:55:36.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.125+0000 7f4f9f7fe700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f88004810 0x7f4f88004c80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00c54a0 tx=0x7f4f940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f65aa12196d0fe5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:36.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f94009740 con 0x7f4f88004810 2023-10-10T18:55:36.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9400e920 con 0x7f4f88004810 2023-10-10T18:55:36.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4f9effd700 1 -- 172.21.15.175:0/2071063873 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9400ece0 con 0x7f4f88004810 2023-10-10T18:55:36.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4fa5669700 1 -- 172.21.15.175:0/2071063873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f88004810 msgr2=0x7f4f88004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:36.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4fa5669700 1 --2- 172.21.15.175:0/2071063873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4f88004810 0x7f4f88004c80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00c54a0 tx=0x7f4f940091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4fa4e68700 1 -- 172.21.15.175:0/2071063873 reap_dead start 2023-10-10T18:55:36.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4fa5669700 1 -- 172.21.15.175:0/2071063873 shutdown_connections 2023-10-10T18:55:36.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4fa5669700 1 -- 172.21.15.175:0/2071063873 >> 172.21.15.175:0/2071063873 conn(0x7f4fa00c9500 msgr2=0x7f4fa00c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:36.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4fa5669700 1 -- 172.21.15.175:0/2071063873 shutdown_connections 2023-10-10T18:55:36.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.126+0000 7f4fa5669700 1 -- 172.21.15.175:0/2071063873 wait complete. 2023-10-10T18:55:36.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.128+0000 7f4fa5669700 1 Processor -- start 2023-10-10T18:55:36.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.128+0000 7f4fa5669700 1 -- start start 2023-10-10T18:55:36.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.129+0000 7f4fa5669700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f88004810 0x7f4fa0165410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:36.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.129+0000 7f4fa5669700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa006d2b0 0x7f4fa0165950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:36.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.129+0000 7f4fa5669700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4fa0147ca0 0x7f4fa0148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:36.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.129+0000 7f4fa5669700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa0153880 con 0x7f4fa0147ca0 2023-10-10T18:55:36.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.129+0000 7f4fa5669700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa00c7d30 con 0x7f4fa006d2b0 2023-10-10T18:55:36.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.129+0000 7f4fa5669700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa0148680 con 0x7f4f88004810 2023-10-10T18:55:36.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4fa0147ca0 0x7f4fa0148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:36.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4fa0147ca0 0x7f4fa0148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43198/0 (socket says 172.21.15.175:43198) 2023-10-10T18:55:36.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 -- 172.21.15.175:0/3673439453 learned_addr learned my addr 172.21.15.175:0/3673439453 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:36.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4f9f7fe700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa006d2b0 0x7f4fa0165950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:36.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4f9ffff700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f88004810 0x7f4fa0165410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:36.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 -- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f88004810 msgr2=0x7f4fa0165410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f88004810 0x7f4fa0165410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 -- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa006d2b0 msgr2=0x7f4fa0165950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa006d2b0 0x7f4fa0165950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 -- 172.21.15.175:0/3673439453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa0068b50 con 0x7f4fa0147ca0 2023-10-10T18:55:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4f9ffff700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f88004810 0x7f4fa0165410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:55:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa4e68700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4fa0147ca0 0x7f4fa0148140 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00c6570 tx=0x7f4f980017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4f87fff700 1 -- 172.21.15.175:0/3673439453 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98017070 con 0x7f4fa0147ca0 2023-10-10T18:55:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4f87fff700 1 -- 172.21.15.175:0/3673439453 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f98006ba0 con 0x7f4fa0147ca0 2023-10-10T18:55:36.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa0153ad0 con 0x7f4fa0147ca0 2023-10-10T18:55:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4f87fff700 1 -- 172.21.15.175:0/3673439453 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9800c820 con 0x7f4fa0147ca0 2023-10-10T18:55:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.130+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa014feb0 con 0x7f4fa0147ca0 2023-10-10T18:55:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.131+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa00c7b80 con 0x7f4fa0147ca0 2023-10-10T18:55:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.132+0000 7f4f87fff700 1 -- 172.21.15.175:0/3673439453 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4f9800cc50 con 0x7f4fa0147ca0 2023-10-10T18:55:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.132+0000 7f4f87fff700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f8c066f30 0x7f4f8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.132+0000 7f4f87fff700 1 -- 172.21.15.175:0/3673439453 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6193+0+0 (secure 0 0 0) 0x7f4f98084200 con 0x7f4fa0147ca0 2023-10-10T18:55:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.132+0000 7f4f9ffff700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f8c066f30 0x7f4f8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.133+0000 7f4f9ffff700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f8c066f30 0x7f4f8c0693e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4f90001200 tx=0x7f4f90009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.140+0000 7f4f87fff700 1 -- 172.21.15.175:0/3673439453 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f98051280 con 0x7f4fa0147ca0 2023-10-10T18:55:36.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.382+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fa0153d10 con 0x7f4fa0147ca0 2023-10-10T18:55:36.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.382+0000 7f4f87fff700 1 -- 172.21.15.175:0/3673439453 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6195 (secure 0 0 0) 0x7f4fa0153d10 con 0x7f4fa0147ca0 2023-10-10T18:55:36.383 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:36.383 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5332,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:07.537641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":4251,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.175:6837/2966961140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":2966961140},{"type":"v1","addr":"172.21.15.175:6837","nonce":2966961140}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:36.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f8c066f30 msgr2=0x7f4f8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:36.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f8c066f30 0x7f4f8c0693e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4f90001200 tx=0x7f4f90009040 comp rx=0 tx=0).stop 2023-10-10T18:55:36.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4fa0147ca0 msgr2=0x7f4fa0148140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:36.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4fa0147ca0 0x7f4fa0148140 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00c6570 tx=0x7f4f980017a0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 shutdown_connections 2023-10-10T18:55:36.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4f8c066f30 0x7f4f8c0693e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4fa0147ca0 0x7f4fa0148140 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f88004810 0x7f4fa0165410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 --2- 172.21.15.175:0/3673439453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa006d2b0 0x7f4fa0165950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:36.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 >> 172.21.15.175:0/3673439453 conn(0x7f4fa00c9500 msgr2=0x7f4fa00677f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:36.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 shutdown_connections 2023-10-10T18:55:36.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:36.385+0000 7f4fa5669700 1 -- 172.21.15.175:0/3673439453 wait complete. 2023-10-10T18:55:36.388 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 81 2023-10-10T18:55:36.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T18:55:38.399 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:38.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.528+0000 7f1b6bfff700 1 Processor -- start 2023-10-10T18:55:38.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.528+0000 7f1b6bfff700 1 -- start start 2023-10-10T18:55:38.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b6bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c0c4640 0x7f1b6c1577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b6bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1b6c157d10 0x7f1b6c14d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b6bfff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c06d8e0 con 0x7f1b6c157400 2023-10-10T18:55:38.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b6bfff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c06e050 con 0x7f1b6c157d10 2023-10-10T18:55:38.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b6bfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c06dbc0 con 0x7f1b6c0c4640 2023-10-10T18:55:38.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b6b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1b6c157d10 0x7f1b6c14d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b6b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1b6c157d10 0x7f1b6c14d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43204/0 (socket says 172.21.15.175:43204) 2023-10-10T18:55:38.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b6b7fe700 1 -- 172.21.15.175:0/3973416016 learned_addr learned my addr 172.21.15.175:0/3973416016 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:38.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.529+0000 7f1b627fc700 1 --2- 172.21.15.175:0/3973416016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c0c4640 0x7f1b6c1577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6b7fe700 1 -- 172.21.15.175:0/3973416016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c0c4640 msgr2=0x7f1b6c1577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:38.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6b7fe700 1 --2- 172.21.15.175:0/3973416016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c0c4640 0x7f1b6c1577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6b7fe700 1 -- 172.21.15.175:0/3973416016 >> v1:172.21.15.191:6789/0 conn(0x7f1b6c157400 legacy=0x7f1b6c06bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6b7fe700 1 -- 172.21.15.175:0/3973416016 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c0c1030 con 0x7f1b6c157d10 2023-10-10T18:55:38.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6b7fe700 1 --2- 172.21.15.175:0/3973416016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157d10 0x7f1b6c14d590 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009f20 tx=0x7f1b64009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc83e5737c8e430a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:38.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6a7fc700 1 -- 172.21.15.175:0/3973416016 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6400e070 con 0x7f1b6c157d10 2023-10-10T18:55:38.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6a7fc700 1 -- 172.21.15.175:0/3973416016 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6400b9d0 con 0x7f1b6c157d10 2023-10-10T18:55:38.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6a7fc700 1 -- 172.21.15.175:0/3973416016 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b640092e0 con 0x7f1b6c157d10 2023-10-10T18:55:38.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3973416016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157d10 msgr2=0x7f1b6c14d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3973416016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157d10 0x7f1b6c14d590 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009f20 tx=0x7f1b64009840 comp rx=0 tx=0).stop 2023-10-10T18:55:38.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3973416016 shutdown_connections 2023-10-10T18:55:38.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3973416016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157d10 0x7f1b6c14d590 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3973416016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c0c4640 0x7f1b6c1577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3973416016 >> 172.21.15.175:0/3973416016 conn(0x7f1b6c063cd0 msgr2=0x7f1b6c06d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:38.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.530+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3973416016 shutdown_connections 2023-10-10T18:55:38.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.531+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3973416016 wait complete. 2023-10-10T18:55:38.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b6bfff700 1 Processor -- start 2023-10-10T18:55:38.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b6bfff700 1 -- start start 2023-10-10T18:55:38.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b6bfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b6c0c4640 0x7f1b6c1643a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b6bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157400 0x7f1b6c0c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b6bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c157d10 0x7f1b6c0c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b6bfff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c1551a0 con 0x7f1b6c157400 2023-10-10T18:55:38.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b6bfff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c1648e0 con 0x7f1b6c0c4640 2023-10-10T18:55:38.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b6bfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c0c2610 con 0x7f1b6c157d10 2023-10-10T18:55:38.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.532+0000 7f1b627fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157400 0x7f1b6c0c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b627fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157400 0x7f1b6c0c18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43212/0 (socket says 172.21.15.175:43212) 2023-10-10T18:55:38.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b627fc700 1 -- 172.21.15.175:0/3592582864 learned_addr learned my addr 172.21.15.175:0/3592582864 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:38.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b6affd700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b6c0c4640 0x7f1b6c1643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b6b7fe700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c157d10 0x7f1b6c0c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b627fc700 1 -- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c157d10 msgr2=0x7f1b6c0c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b627fc700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c157d10 0x7f1b6c0c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b627fc700 1 -- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b6c0c4640 msgr2=0x7f1b6c1643a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b627fc700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b6c0c4640 0x7f1b6c1643a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b627fc700 1 -- 172.21.15.175:0/3592582864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64009ae0 con 0x7f1b6c157400 2023-10-10T18:55:38.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b627fc700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157400 0x7f1b6c0c18c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400ece0 tx=0x7f1b540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:38.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b637fe700 1 -- 172.21.15.175:0/3592582864 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5400ca60 con 0x7f1b6c157400 2023-10-10T18:55:38.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b637fe700 1 -- 172.21.15.175:0/3592582864 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5400cbc0 con 0x7f1b6c157400 2023-10-10T18:55:38.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c0c6bb0 con 0x7f1b6c157400 2023-10-10T18:55:38.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b637fe700 1 -- 172.21.15.175:0/3592582864 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54013470 con 0x7f1b6c157400 2023-10-10T18:55:38.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.533+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c0c1e00 con 0x7f1b6c157400 2023-10-10T18:55:38.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.534+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c070a30 con 0x7f1b6c157400 2023-10-10T18:55:38.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.534+0000 7f1b637fe700 1 -- 172.21.15.175:0/3592582864 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1b54013820 con 0x7f1b6c157400 2023-10-10T18:55:38.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.535+0000 7f1b637fe700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b4c066f60 0x7f1b4c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.535+0000 7f1b637fe700 1 -- 172.21.15.175:0/3592582864 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f1b54084b40 con 0x7f1b6c157400 2023-10-10T18:55:38.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.535+0000 7f1b6affd700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b4c066f60 0x7f1b4c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.535+0000 7f1b6affd700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b4c066f60 0x7f1b4c069410 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c003860 tx=0x7f1b5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:38.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.539+0000 7f1b637fe700 1 -- 172.21.15.175:0/3592582864 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b54051b90 con 0x7f1b6c157400 2023-10-10T18:55:38.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.763+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b6c067570 con 0x7f1b6c157400 2023-10-10T18:55:38.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.764+0000 7f1b637fe700 1 -- 172.21.15.175:0/3592582864 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5436 (secure 0 0 0) 0x7f1b540551b0 con 0x7f1b6c157400 2023-10-10T18:55:38.767 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:38.767 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:38.510133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:38.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b4c066f60 msgr2=0x7f1b4c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b4c066f60 0x7f1b4c069410 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c003860 tx=0x7f1b5c009040 comp rx=0 tx=0).stop 2023-10-10T18:55:38.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157400 msgr2=0x7f1b6c0c18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157400 0x7f1b6c0c18c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400ece0 tx=0x7f1b540091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 shutdown_connections 2023-10-10T18:55:38.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b4c066f60 0x7f1b4c069410 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b6c157d10 0x7f1b6c0c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b6c0c4640 0x7f1b6c1643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 --2- 172.21.15.175:0/3592582864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1b6c157400 0x7f1b6c0c18c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 >> 172.21.15.175:0/3592582864 conn(0x7f1b6c063cd0 msgr2=0x7f1b6c14b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:38.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 shutdown_connections 2023-10-10T18:55:38.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.766+0000 7f1b6bfff700 1 -- 172.21.15.175:0/3592582864 wait complete. 2023-10-10T18:55:38.771 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 82 2023-10-10T18:55:38.777 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-10T18:55:38.778 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T18:55:38.778 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:38.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.893+0000 7f059f72f700 1 Processor -- start 2023-10-10T18:55:38.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.893+0000 7f059f72f700 1 -- start start 2023-10-10T18:55:38.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059f72f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f05980c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059f72f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f05980c4c20 0x7f0598160930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059f72f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05980677f0 con 0x7f059806d6e0 2023-10-10T18:55:38.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059f72f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0598159590 con 0x7f05980c4c20 2023-10-10T18:55:38.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059f72f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05980c4250 con 0x7f05981500b0 2023-10-10T18:55:38.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059d4cb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f05980c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059dccc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f05980c4c20 0x7f0598160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059d4cb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f05980c46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:46228/0 (socket says 172.21.15.175:46228) 2023-10-10T18:55:38.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059ccca700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f059806d6e0 0x7f059806dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:36220/0 (socket says 172.21.15.175:36220) 2023-10-10T18:55:38.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059d4cb700 1 -- 172.21.15.175:0/1827104463 learned_addr learned my addr 172.21.15.175:0/1827104463 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:38.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059d4cb700 1 -- 172.21.15.175:0/1827104463 >> v2:172.21.15.191:3300/0 conn(0x7f05980c4c20 msgr2=0x7f0598160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:38.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059d4cb700 1 --2- 172.21.15.175:0/1827104463 >> v2:172.21.15.191:3300/0 conn(0x7f05980c4c20 0x7f0598160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059d4cb700 1 -- 172.21.15.175:0/1827104463 >> v1:172.21.15.191:6789/0 conn(0x7f059806d6e0 legacy=0x7f059806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059d4cb700 1 -- 172.21.15.175:0/1827104463 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598063760 con 0x7f05981500b0 2023-10-10T18:55:38.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.894+0000 7f059dccc700 1 --2- 172.21.15.175:0/1827104463 >> v2:172.21.15.191:3300/0 conn(0x7f05980c4c20 0x7f0598160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:55:38.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059d4cb700 1 --2- 172.21.15.175:0/1827104463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f05980c46e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f05900099b0 tx=0x7f0590009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92d744e09a7b2c8f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:38.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f0597fff700 1 -- 172.21.15.175:0/1827104463 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590009ea0 con 0x7f05981500b0 2023-10-10T18:55:38.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f0597fff700 1 -- 172.21.15.175:0/1827104463 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059000b8f0 con 0x7f05981500b0 2023-10-10T18:55:38.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f0597fff700 1 -- 172.21.15.175:0/1827104463 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059000bc90 con 0x7f05981500b0 2023-10-10T18:55:38.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059f72f700 1 -- 172.21.15.175:0/1827104463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 msgr2=0x7f05980c46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059f72f700 1 --2- 172.21.15.175:0/1827104463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f05980c46e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f05900099b0 tx=0x7f0590009250 comp rx=0 tx=0).stop 2023-10-10T18:55:38.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059f72f700 1 -- 172.21.15.175:0/1827104463 shutdown_connections 2023-10-10T18:55:38.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059f72f700 1 --2- 172.21.15.175:0/1827104463 >> v2:172.21.15.191:3300/0 conn(0x7f05980c4c20 0x7f0598160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059f72f700 1 --2- 172.21.15.175:0/1827104463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f05980c46e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059f72f700 1 -- 172.21.15.175:0/1827104463 >> 172.21.15.175:0/1827104463 conn(0x7f05980c5650 msgr2=0x7f05980c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:38.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059f72f700 1 -- 172.21.15.175:0/1827104463 shutdown_connections 2023-10-10T18:55:38.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.895+0000 7f059f72f700 1 -- 172.21.15.175:0/1827104463 wait complete. 2023-10-10T18:55:38.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.896+0000 7f059f72f700 1 Processor -- start 2023-10-10T18:55:38.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.896+0000 7f059f72f700 1 -- start start 2023-10-10T18:55:38.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059f72f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f059806d6e0 0x7f05981671d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059f72f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05980c4c20 0x7f059814d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059f72f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f0598169290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059f72f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05981542f0 con 0x7f059806d6e0 2023-10-10T18:55:38.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059f72f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05980690b0 con 0x7f05980c4c20 2023-10-10T18:55:38.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059f72f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05980c8130 con 0x7f05981500b0 2023-10-10T18:55:38.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059d4cb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f059806d6e0 0x7f05981671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059d4cb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f059806d6e0 0x7f05981671d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43222/0 (socket says 172.21.15.175:43222) 2023-10-10T18:55:38.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059d4cb700 1 -- 172.21.15.175:0/1965948766 learned_addr learned my addr 172.21.15.175:0/1965948766 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:38.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059dccc700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f0598169290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.897+0000 7f059ccca700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05980c4c20 0x7f059814d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f059d4cb700 1 -- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 msgr2=0x7f0598169290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f059d4cb700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f0598169290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f059d4cb700 1 -- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05980c4c20 msgr2=0x7f059814d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:38.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f059d4cb700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05980c4c20 0x7f059814d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:38.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f059d4cb700 1 -- 172.21.15.175:0/1965948766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0588009a20 con 0x7f059806d6e0 2023-10-10T18:55:38.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f059d4cb700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f059806d6e0 0x7f05981671d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f05980c4390 tx=0x7f0590017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:38.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f05957fa700 1 -- 172.21.15.175:0/1965948766 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590000d10 con 0x7f059806d6e0 2023-10-10T18:55:38.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05900094f0 con 0x7f059806d6e0 2023-10-10T18:55:38.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f05957fa700 1 -- 172.21.15.175:0/1965948766 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059000bc70 con 0x7f059806d6e0 2023-10-10T18:55:38.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f05957fa700 1 -- 172.21.15.175:0/1965948766 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059001a6c0 con 0x7f059806d6e0 2023-10-10T18:55:38.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.898+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05981522f0 con 0x7f059806d6e0 2023-10-10T18:55:38.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.899+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0598067570 con 0x7f059806d6e0 2023-10-10T18:55:38.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.899+0000 7f05957fa700 1 -- 172.21.15.175:0/1965948766 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f059001aa70 con 0x7f059806d6e0 2023-10-10T18:55:38.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.900+0000 7f05957fa700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0584066f30 0x7f05840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:38.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.900+0000 7f05957fa700 1 -- 172.21.15.175:0/1965948766 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f0590086120 con 0x7f059806d6e0 2023-10-10T18:55:38.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.900+0000 7f059ccca700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0584066f30 0x7f05840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:38.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.900+0000 7f059ccca700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0584066f30 0x7f05840693e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f05981537e0 tx=0x7f058c000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:38.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:38.904+0000 7f05957fa700 1 -- 172.21.15.175:0/1965948766 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0590053170 con 0x7f059806d6e0 2023-10-10T18:55:39.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.131+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05980c6c50 con 0x7f059806d6e0 2023-10-10T18:55:39.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.132+0000 7f05957fa700 1 -- 172.21.15.175:0/1965948766 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5436 (secure 0 0 0) 0x7f0590056790 con 0x7f059806d6e0 2023-10-10T18:55:39.132 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:39.132 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:38.510133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:39.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0584066f30 msgr2=0x7f05840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:39.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0584066f30 0x7f05840693e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f05981537e0 tx=0x7f058c000bc0 comp rx=0 tx=0).stop 2023-10-10T18:55:39.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f059806d6e0 msgr2=0x7f05981671d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:39.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f059806d6e0 0x7f05981671d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f05980c4390 tx=0x7f0590017000 comp rx=0 tx=0).stop 2023-10-10T18:55:39.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 shutdown_connections 2023-10-10T18:55:39.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0584066f30 0x7f05840693e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:39.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05981500b0 0x7f0598169290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:39.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f059806d6e0 0x7f05981671d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:39.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 --2- 172.21.15.175:0/1965948766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05980c4c20 0x7f059814d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:39.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 >> 172.21.15.175:0/1965948766 conn(0x7f05980c5650 msgr2=0x7f05980c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:39.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 shutdown_connections 2023-10-10T18:55:39.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:39.134+0000 7f059f72f700 1 -- 172.21.15.175:0/1965948766 wait complete. 2023-10-10T18:55:39.136 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 82 2023-10-10T18:55:39.146 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 82, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 82, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:55:38.510133+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 5332, "mds_2": 5367, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:55:41.147 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:41.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.275+0000 7f1e26070700 1 Processor -- start 2023-10-10T18:55:41.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.275+0000 7f1e26070700 1 -- start start 2023-10-10T18:55:41.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e26070700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e200c60d0 0x7f1e20156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:41.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e26070700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c64a0 0x7f1e200c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:41.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e26070700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e200c9cc0 con 0x7f1e2006f5f0 2023-10-10T18:55:41.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e26070700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e201571c0 con 0x7f1e200c60d0 2023-10-10T18:55:41.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e26070700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200c4500 con 0x7f1e200c64a0 2023-10-10T18:55:41.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e200c60d0 0x7f1e20156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:41.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1effd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1e2006f5f0 0x7f1e200c3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40430/0 (socket says 172.21.15.175:40430) 2023-10-10T18:55:41.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1effd700 1 -- 172.21.15.175:0/1727036919 learned_addr learned my addr 172.21.15.175:0/1727036919 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:41.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1ffff700 1 --2- 172.21.15.175:0/1727036919 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c64a0 0x7f1e200c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:41.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1e7fc700 1 -- 172.21.15.175:0/1727036919 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935526746 0 0) 0x7f1e200c9cc0 con 0x7f1e2006f5f0 2023-10-10T18:55:41.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1e7fc700 1 -- 172.21.15.175:0/1727036919 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e08003570 con 0x7f1e2006f5f0 2023-10-10T18:55:41.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1f7fe700 1 -- 172.21.15.175:0/1727036919 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c64a0 msgr2=0x7f1e200c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:41.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1f7fe700 1 --2- 172.21.15.175:0/1727036919 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c64a0 0x7f1e200c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1f7fe700 1 -- 172.21.15.175:0/1727036919 >> v1:172.21.15.175:6789/0 conn(0x7f1e2006f5f0 legacy=0x7f1e200c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:41.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.276+0000 7f1e1f7fe700 1 -- 172.21.15.175:0/1727036919 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2006e650 con 0x7f1e200c60d0 2023-10-10T18:55:41.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e1f7fe700 1 --2- 172.21.15.175:0/1727036919 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e200c60d0 0x7f1e20156c80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009e60 tx=0x7f1e10009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f55256960c397da server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:41.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e1e7fc700 1 -- 172.21.15.175:0/1727036919 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1000e070 con 0x7f1e200c60d0 2023-10-10T18:55:41.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e1e7fc700 1 -- 172.21.15.175:0/1727036919 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1000bb60 con 0x7f1e200c60d0 2023-10-10T18:55:41.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e1e7fc700 1 -- 172.21.15.175:0/1727036919 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e10012430 con 0x7f1e200c60d0 2023-10-10T18:55:41.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e26070700 1 -- 172.21.15.175:0/1727036919 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e200c60d0 msgr2=0x7f1e20156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:41.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e26070700 1 --2- 172.21.15.175:0/1727036919 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e200c60d0 0x7f1e20156c80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009e60 tx=0x7f1e10009780 comp rx=0 tx=0).stop 2023-10-10T18:55:41.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e26070700 1 -- 172.21.15.175:0/1727036919 shutdown_connections 2023-10-10T18:55:41.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e26070700 1 --2- 172.21.15.175:0/1727036919 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c64a0 0x7f1e200c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e26070700 1 --2- 172.21.15.175:0/1727036919 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e200c60d0 0x7f1e20156c80 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e26070700 1 -- 172.21.15.175:0/1727036919 >> 172.21.15.175:0/1727036919 conn(0x7f1e2006c110 msgr2=0x7f1e2006c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:41.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e26070700 1 -- 172.21.15.175:0/1727036919 shutdown_connections 2023-10-10T18:55:41.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.277+0000 7f1e26070700 1 -- 172.21.15.175:0/1727036919 wait complete. 2023-10-10T18:55:41.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e26070700 1 Processor -- start 2023-10-10T18:55:41.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e26070700 1 -- start start 2023-10-10T18:55:41.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e26070700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e2006f5f0 0x7f1e201503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:41.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e26070700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c60d0 0x7f1e200c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:41.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e26070700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e200c64a0 0x7f1e200c2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:41.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e26070700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20147c50 con 0x7f1e2006f5f0 2023-10-10T18:55:41.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e26070700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20168230 con 0x7f1e200c64a0 2023-10-10T18:55:41.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e26070700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20167680 con 0x7f1e200c60d0 2023-10-10T18:55:41.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e1f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e2006f5f0 0x7f1e201503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:41.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e1f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e2006f5f0 0x7f1e201503c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43252/0 (socket says 172.21.15.175:43252) 2023-10-10T18:55:41.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e1f7fe700 1 -- 172.21.15.175:0/145297203 learned_addr learned my addr 172.21.15.175:0/145297203 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:41.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e1effd700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c60d0 0x7f1e200c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:41.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.279+0000 7f1e1ffff700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e200c64a0 0x7f1e200c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:41.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e1f7fe700 1 -- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c60d0 msgr2=0x7f1e200c1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:41.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e1f7fe700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c60d0 0x7f1e200c1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e1f7fe700 1 -- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e200c64a0 msgr2=0x7f1e200c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:41.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e1f7fe700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e200c64a0 0x7f1e200c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e1f7fe700 1 -- 172.21.15.175:0/145297203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e10009a20 con 0x7f1e2006f5f0 2023-10-10T18:55:41.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e1f7fe700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e2006f5f0 0x7f1e201503c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000b6c0 tx=0x7f1e10019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:41.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e077fe700 1 -- 172.21.15.175:0/145297203 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e10012cf0 con 0x7f1e2006f5f0 2023-10-10T18:55:41.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e077fe700 1 -- 172.21.15.175:0/145297203 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e10012e50 con 0x7f1e2006f5f0 2023-10-10T18:55:41.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e20164c30 con 0x7f1e2006f5f0 2023-10-10T18:55:41.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e077fe700 1 -- 172.21.15.175:0/145297203 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e100276c0 con 0x7f1e2006f5f0 2023-10-10T18:55:41.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.280+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e20167980 con 0x7f1e2006f5f0 2023-10-10T18:55:41.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.281+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e20067570 con 0x7f1e2006f5f0 2023-10-10T18:55:41.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.281+0000 7f1e077fe700 1 -- 172.21.15.175:0/145297203 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1e10024070 con 0x7f1e2006f5f0 2023-10-10T18:55:41.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.282+0000 7f1e077fe700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e0c066f30 0x7f1e0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:41.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.282+0000 7f1e077fe700 1 -- 172.21.15.175:0/145297203 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f1e10086d30 con 0x7f1e2006f5f0 2023-10-10T18:55:41.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.282+0000 7f1e1effd700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e0c066f30 0x7f1e0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:41.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.283+0000 7f1e1effd700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e0c066f30 0x7f1e0c0693e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1e201492c0 tx=0x7f1e14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:41.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.286+0000 7f1e077fe700 1 -- 172.21.15.175:0/145297203 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1e10053d80 con 0x7f1e2006f5f0 2023-10-10T18:55:41.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.510+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e2006a490 con 0x7f1e2006f5f0 2023-10-10T18:55:41.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.511+0000 7f1e077fe700 1 -- 172.21.15.175:0/145297203 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5436 (secure 0 0 0) 0x7f1e100573a0 con 0x7f1e2006f5f0 2023-10-10T18:55:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:41.511 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:38.510133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:41.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e0c066f30 msgr2=0x7f1e0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:41.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e0c066f30 0x7f1e0c0693e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1e201492c0 tx=0x7f1e14009040 comp rx=0 tx=0).stop 2023-10-10T18:55:41.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e2006f5f0 msgr2=0x7f1e201503c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:41.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e2006f5f0 0x7f1e201503c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000b6c0 tx=0x7f1e10019000 comp rx=0 tx=0).stop 2023-10-10T18:55:41.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 shutdown_connections 2023-10-10T18:55:41.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e0c066f30 0x7f1e0c0693e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e200c64a0 0x7f1e200c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e2006f5f0 0x7f1e201503c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 --2- 172.21.15.175:0/145297203 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e200c60d0 0x7f1e200c1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:41.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 >> 172.21.15.175:0/145297203 conn(0x7f1e2006c110 msgr2=0x7f1e20144e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:41.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 shutdown_connections 2023-10-10T18:55:41.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:41.513+0000 7f1e26070700 1 -- 172.21.15.175:0/145297203 wait complete. 2023-10-10T18:55:41.516 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 82 2023-10-10T18:55:41.525 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:55:43.526 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:43.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.649+0000 7f9c5af29700 1 Processor -- start 2023-10-10T18:55:43.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.649+0000 7f9c5af29700 1 -- start start 2023-10-10T18:55:43.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c5af29700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9c5408a100 0x7f9c54159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:43.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c5af29700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54067fa0 con 0x7f9c540cd040 2023-10-10T18:55:43.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c5af29700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54156b70 con 0x7f9c5414fd90 2023-10-10T18:55:43.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c5af29700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c540c5600 con 0x7f9c5408a100 2023-10-10T18:55:43.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c59726700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9c5408a100 0x7f9c54159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:43.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c5a728700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9c540cd040 0x7f9c540684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40434/0 (socket says 172.21.15.175:40434) 2023-10-10T18:55:43.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c5a728700 1 -- 172.21.15.175:0/1184962635 learned_addr learned my addr 172.21.15.175:0/1184962635 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:43.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c59726700 1 -- 172.21.15.175:0/1184962635 >> v1:172.21.15.191:6789/0 conn(0x7f9c5414fd90 legacy=0x7f9c540c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:43.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c59726700 1 -- 172.21.15.175:0/1184962635 >> v1:172.21.15.175:6789/0 conn(0x7f9c540cd040 legacy=0x7f9c540684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:43.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c58f25700 1 -- 172.21.15.175:0/1184962635 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801879187 0 0) 0x7f9c54067fa0 con 0x7f9c540cd040 2023-10-10T18:55:43.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.650+0000 7f9c59726700 1 -- 172.21.15.175:0/1184962635 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54150160 con 0x7f9c5408a100 2023-10-10T18:55:43.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c59726700 1 --2- 172.21.15.175:0/1184962635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c5408a100 0x7f9c54159c30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9c440098e0 tx=0x7f9c440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7052db3ede50a8b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:43.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c58f25700 1 -- 172.21.15.175:0/1184962635 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44009e80 con 0x7f9c5408a100 2023-10-10T18:55:43.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c58f25700 1 -- 172.21.15.175:0/1184962635 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4400b8f0 con 0x7f9c5408a100 2023-10-10T18:55:43.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c58f25700 1 -- 172.21.15.175:0/1184962635 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4400bc70 con 0x7f9c5408a100 2023-10-10T18:55:43.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c5af29700 1 -- 172.21.15.175:0/1184962635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c5408a100 msgr2=0x7f9c54159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:43.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c5af29700 1 --2- 172.21.15.175:0/1184962635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c5408a100 0x7f9c54159c30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9c440098e0 tx=0x7f9c440091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:43.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c5af29700 1 -- 172.21.15.175:0/1184962635 shutdown_connections 2023-10-10T18:55:43.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c5af29700 1 --2- 172.21.15.175:0/1184962635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c5408a100 0x7f9c54159c30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:43.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c5af29700 1 -- 172.21.15.175:0/1184962635 >> 172.21.15.175:0/1184962635 conn(0x7f9c54063d00 msgr2=0x7f9c540cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:43.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c5af29700 1 -- 172.21.15.175:0/1184962635 shutdown_connections 2023-10-10T18:55:43.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.651+0000 7f9c5af29700 1 -- 172.21.15.175:0/1184962635 wait complete. 2023-10-10T18:55:43.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.652+0000 7f9c5af29700 1 Processor -- start 2023-10-10T18:55:43.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c5af29700 1 -- start start 2023-10-10T18:55:43.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c5af29700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c5408a100 0x7f9c54151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:43.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c5af29700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c540cd040 0x7f9c54068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:43.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c5af29700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c5414fd90 0x7f9c5414c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:43.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c5af29700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540a8350 con 0x7f9c540cd040 2023-10-10T18:55:43.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c5af29700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541691c0 con 0x7f9c5408a100 2023-10-10T18:55:43.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c5af29700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540a8db0 con 0x7f9c5414fd90 2023-10-10T18:55:43.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c59726700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c540cd040 0x7f9c54068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:43.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c59726700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c540cd040 0x7f9c54068e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43270/0 (socket says 172.21.15.175:43270) 2023-10-10T18:55:43.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c59f27700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c5408a100 0x7f9c54151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:43.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c59726700 1 -- 172.21.15.175:0/2736260795 learned_addr learned my addr 172.21.15.175:0/2736260795 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:43.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.653+0000 7f9c5a728700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c5414fd90 0x7f9c5414c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:43.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.654+0000 7f9c59726700 1 -- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c5414fd90 msgr2=0x7f9c5414c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:43.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.654+0000 7f9c59726700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c5414fd90 0x7f9c5414c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:43.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.654+0000 7f9c59726700 1 -- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c5408a100 msgr2=0x7f9c54151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:43.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.654+0000 7f9c59726700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c5408a100 0x7f9c54151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:43.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.654+0000 7f9c59726700 1 -- 172.21.15.175:0/2736260795 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c44009420 con 0x7f9c540cd040 2023-10-10T18:55:43.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.654+0000 7f9c59726700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c540cd040 0x7f9c54068e10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9c54064c20 tx=0x7f9c44017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:43.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.655+0000 7f9c49ffb700 1 -- 172.21.15.175:0/2736260795 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4401f070 con 0x7f9c540cd040 2023-10-10T18:55:43.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.655+0000 7f9c49ffb700 1 -- 172.21.15.175:0/2736260795 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c44000c90 con 0x7f9c540cd040 2023-10-10T18:55:43.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.655+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5414f180 con 0x7f9c540cd040 2023-10-10T18:55:43.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.655+0000 7f9c49ffb700 1 -- 172.21.15.175:0/2736260795 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4400bc70 con 0x7f9c540cd040 2023-10-10T18:55:43.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.655+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c54169410 con 0x7f9c540cd040 2023-10-10T18:55:43.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.656+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c54067570 con 0x7f9c540cd040 2023-10-10T18:55:43.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.657+0000 7f9c49ffb700 1 -- 172.21.15.175:0/2736260795 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9c4401a8c0 con 0x7f9c540cd040 2023-10-10T18:55:43.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.657+0000 7f9c49ffb700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c38066f30 0x7f9c380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:43.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.657+0000 7f9c49ffb700 1 -- 172.21.15.175:0/2736260795 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f9c44085b80 con 0x7f9c540cd040 2023-10-10T18:55:43.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.657+0000 7f9c59f27700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c38066f30 0x7f9c380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:43.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.658+0000 7f9c59f27700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c38066f30 0x7f9c380693e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9c50007fd0 tx=0x7f9c50001650 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:43.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.661+0000 7f9c49ffb700 1 -- 172.21.15.175:0/2736260795 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c44052bd0 con 0x7f9c540cd040 2023-10-10T18:55:43.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.887+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c54155f80 con 0x7f9c540cd040 2023-10-10T18:55:43.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.888+0000 7f9c49ffb700 1 -- 172.21.15.175:0/2736260795 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5436 (secure 0 0 0) 0x7f9c44015090 con 0x7f9c540cd040 2023-10-10T18:55:43.888 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:43.888 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:38.510133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:43.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c38066f30 msgr2=0x7f9c380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:43.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c38066f30 0x7f9c380693e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9c50007fd0 tx=0x7f9c50001650 comp rx=0 tx=0).stop 2023-10-10T18:55:43.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c540cd040 msgr2=0x7f9c54068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:43.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c540cd040 0x7f9c54068e10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9c54064c20 tx=0x7f9c44017000 comp rx=0 tx=0).stop 2023-10-10T18:55:43.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 shutdown_connections 2023-10-10T18:55:43.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c38066f30 0x7f9c380693e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:43.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c5414fd90 0x7f9c5414c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:43.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c5408a100 0x7f9c54151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:43.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 --2- 172.21.15.175:0/2736260795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9c540cd040 0x7f9c54068e10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:43.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 >> 172.21.15.175:0/2736260795 conn(0x7f9c54063d00 msgr2=0x7f9c540c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:43.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 shutdown_connections 2023-10-10T18:55:43.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:43.890+0000 7f9c5af29700 1 -- 172.21.15.175:0/2736260795 wait complete. 2023-10-10T18:55:43.893 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 82 2023-10-10T18:55:43.902 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:55:45.903 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:46.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.022+0000 7f9d32381700 1 Processor -- start 2023-10-10T18:55:46.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.022+0000 7f9d32381700 1 -- start start 2023-10-10T18:55:46.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d32381700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c14fd90 0x7f9d2c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:46.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d32381700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2c067fa0 con 0x7f9d2c08a100 2023-10-10T18:55:46.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d32381700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2c156b70 con 0x7f9d2c0cd040 2023-10-10T18:55:46.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d32381700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c0c5600 con 0x7f9d2c14fd90 2023-10-10T18:55:46.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d30b7e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9d2c08a100 0x7f9d2c0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42874/0 (socket says 172.21.15.175:42874) 2023-10-10T18:55:46.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d30b7e700 1 -- 172.21.15.175:0/335225335 learned_addr learned my addr 172.21.15.175:0/335225335 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:46.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d3137f700 1 --2- 172.21.15.175:0/335225335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c14fd90 0x7f9d2c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:46.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d23fff700 1 -- 172.21.15.175:0/335225335 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142448247 0 0) 0x7f9d2c067fa0 con 0x7f9d2c08a100 2023-10-10T18:55:46.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d23fff700 1 -- 172.21.15.175:0/335225335 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d18003570 con 0x7f9d2c08a100 2023-10-10T18:55:46.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d23fff700 1 -- 172.21.15.175:0/335225335 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240333564 0 0) 0x7f9d2c156b70 con 0x7f9d2c0cd040 2023-10-10T18:55:46.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.023+0000 7f9d23fff700 1 -- 172.21.15.175:0/335225335 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2c067fa0 con 0x7f9d2c0cd040 2023-10-10T18:55:46.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d3137f700 1 -- 172.21.15.175:0/335225335 >> v1:172.21.15.191:6789/0 conn(0x7f9d2c0cd040 legacy=0x7f9d2c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:46.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d3137f700 1 -- 172.21.15.175:0/335225335 >> v1:172.21.15.175:6789/0 conn(0x7f9d2c08a100 legacy=0x7f9d2c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:46.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d3137f700 1 -- 172.21.15.175:0/335225335 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c150160 con 0x7f9d2c14fd90 2023-10-10T18:55:46.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d3137f700 1 --2- 172.21.15.175:0/335225335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c14fd90 0x7f9d2c159c30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009910 tx=0x7f9d280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41b256873fd56202 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:46.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d23fff700 1 -- 172.21.15.175:0/335225335 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28011630 con 0x7f9d2c14fd90 2023-10-10T18:55:46.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d23fff700 1 -- 172.21.15.175:0/335225335 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28011790 con 0x7f9d2c14fd90 2023-10-10T18:55:46.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d23fff700 1 -- 172.21.15.175:0/335225335 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28011b30 con 0x7f9d2c14fd90 2023-10-10T18:55:46.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d32381700 1 -- 172.21.15.175:0/335225335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c14fd90 msgr2=0x7f9d2c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:46.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.024+0000 7f9d32381700 1 --2- 172.21.15.175:0/335225335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c14fd90 0x7f9d2c159c30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009910 tx=0x7f9d280091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.025+0000 7f9d32381700 1 -- 172.21.15.175:0/335225335 shutdown_connections 2023-10-10T18:55:46.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.025+0000 7f9d32381700 1 --2- 172.21.15.175:0/335225335 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c14fd90 0x7f9d2c159c30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.025+0000 7f9d32381700 1 -- 172.21.15.175:0/335225335 >> 172.21.15.175:0/335225335 conn(0x7f9d2c063d00 msgr2=0x7f9d2c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:46.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.025+0000 7f9d32381700 1 -- 172.21.15.175:0/335225335 shutdown_connections 2023-10-10T18:55:46.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.025+0000 7f9d32381700 1 -- 172.21.15.175:0/335225335 wait complete. 2023-10-10T18:55:46.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.026+0000 7f9d32381700 1 Processor -- start 2023-10-10T18:55:46.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.026+0000 7f9d32381700 1 -- start start 2023-10-10T18:55:46.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.026+0000 7f9d32381700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d2c08a100 0x7f9d2c1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:46.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.026+0000 7f9d32381700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c0cd040 0x7f9d2c165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:46.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.026+0000 7f9d32381700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d2c14fd90 0x7f9d2c1497b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:46.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.026+0000 7f9d32381700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c166ba0 con 0x7f9d2c08a100 2023-10-10T18:55:46.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.026+0000 7f9d32381700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c150550 con 0x7f9d2c14fd90 2023-10-10T18:55:46.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.026+0000 7f9d32381700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c152d30 con 0x7f9d2c0cd040 2023-10-10T18:55:46.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d2c08a100 0x7f9d2c1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:46.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d2c08a100 0x7f9d2c1531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50274/0 (socket says 172.21.15.175:50274) 2023-10-10T18:55:46.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 -- 172.21.15.175:0/988196928 learned_addr learned my addr 172.21.15.175:0/988196928 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:46.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 -- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c0cd040 msgr2=0x7f9d2c165ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:46.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d31b80700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d2c14fd90 0x7f9d2c1497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:46.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c0cd040 0x7f9d2c165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 -- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d2c14fd90 msgr2=0x7f9d2c1497b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:46.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d2c14fd90 0x7f9d2c1497b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 -- 172.21.15.175:0/988196928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d28009420 con 0x7f9d2c08a100 2023-10-10T18:55:46.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d3137f700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d2c08a100 0x7f9d2c1531c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c064c20 tx=0x7f9d28017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:46.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d217fa700 1 -- 172.21.15.175:0/988196928 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2801f070 con 0x7f9d2c08a100 2023-10-10T18:55:46.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d217fa700 1 -- 172.21.15.175:0/988196928 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28000c90 con 0x7f9d2c08a100 2023-10-10T18:55:46.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2c155cf0 con 0x7f9d2c08a100 2023-10-10T18:55:46.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d217fa700 1 -- 172.21.15.175:0/988196928 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28025420 con 0x7f9d2c08a100 2023-10-10T18:55:46.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.027+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2c166900 con 0x7f9d2c08a100 2023-10-10T18:55:46.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.028+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2c153700 con 0x7f9d2c08a100 2023-10-10T18:55:46.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.029+0000 7f9d217fa700 1 -- 172.21.15.175:0/988196928 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9d28023070 con 0x7f9d2c08a100 2023-10-10T18:55:46.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.029+0000 7f9d217fa700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d10066f60 0x7f9d10069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:46.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.029+0000 7f9d217fa700 1 -- 172.21.15.175:0/988196928 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f9d280859f0 con 0x7f9d2c08a100 2023-10-10T18:55:46.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.029+0000 7f9d30b7e700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d10066f60 0x7f9d10069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:46.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.030+0000 7f9d30b7e700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d10066f60 0x7f9d10069410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c0058d0 tx=0x7f9d1c0079f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:46.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.033+0000 7f9d217fa700 1 -- 172.21.15.175:0/988196928 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d28052af0 con 0x7f9d2c08a100 2023-10-10T18:55:46.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.257+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d2c067570 con 0x7f9d2c08a100 2023-10-10T18:55:46.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.258+0000 7f9d217fa700 1 -- 172.21.15.175:0/988196928 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5438 (secure 0 0 0) 0x7f9d2802a090 con 0x7f9d2c08a100 2023-10-10T18:55:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:46.258 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:44.654827+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:resolve","state_seq":25,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:46.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d10066f60 msgr2=0x7f9d10069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:46.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d10066f60 0x7f9d10069410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c0058d0 tx=0x7f9d1c0079f0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d2c08a100 msgr2=0x7f9d2c1531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:46.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d2c08a100 0x7f9d2c1531c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c064c20 tx=0x7f9d28017000 comp rx=0 tx=0).stop 2023-10-10T18:55:46.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 shutdown_connections 2023-10-10T18:55:46.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d10066f60 0x7f9d10069410 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d2c14fd90 0x7f9d2c1497b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d2c08a100 0x7f9d2c1531c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 --2- 172.21.15.175:0/988196928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d2c0cd040 0x7f9d2c165ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:46.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 >> 172.21.15.175:0/988196928 conn(0x7f9d2c063d00 msgr2=0x7f9d2c14edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:46.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 shutdown_connections 2023-10-10T18:55:46.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:46.260+0000 7f9d32381700 1 -- 172.21.15.175:0/988196928 wait complete. 2023-10-10T18:55:46.263 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 83 2023-10-10T18:55:46.272 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:55:48.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:48.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.399+0000 7fb81c8eb700 1 Processor -- start 2023-10-10T18:55:48.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.399+0000 7fb81c8eb700 1 -- start start 2023-10-10T18:55:48.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb81c8eb700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb81808a100 0x7fb818159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:48.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb81c8eb700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb818067fa0 con 0x7fb8180cd040 2023-10-10T18:55:48.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb81c8eb700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb818156b70 con 0x7fb81814fd90 2023-10-10T18:55:48.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb81c8eb700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8180c5600 con 0x7fb81808a100 2023-10-10T18:55:48.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb81759e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb8180cd040 0x7fb8180684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42880/0 (socket says 172.21.15.175:42880) 2023-10-10T18:55:48.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb81759e700 1 -- 172.21.15.175:0/2062491648 learned_addr learned my addr 172.21.15.175:0/2062491648 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:48.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb80e59c700 1 --2- 172.21.15.175:0/2062491648 >> v2:172.21.15.191:3300/0 conn(0x7fb81808a100 0x7fb818159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:48.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb81659c700 1 -- 172.21.15.175:0/2062491648 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393389429 0 0) 0x7fb818067fa0 con 0x7fb8180cd040 2023-10-10T18:55:48.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb81659c700 1 -- 172.21.15.175:0/2062491648 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb804003570 con 0x7fb8180cd040 2023-10-10T18:55:48.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb80e59c700 1 -- 172.21.15.175:0/2062491648 >> v1:172.21.15.191:6789/0 conn(0x7fb81814fd90 legacy=0x7fb8180c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:48.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.400+0000 7fb80e59c700 1 -- 172.21.15.175:0/2062491648 >> v1:172.21.15.175:6789/0 conn(0x7fb8180cd040 legacy=0x7fb8180684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:48.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb80e59c700 1 -- 172.21.15.175:0/2062491648 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818150160 con 0x7fb81808a100 2023-10-10T18:55:48.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb80e59c700 1 --2- 172.21.15.175:0/2062491648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81808a100 0x7fb818159c30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb8000098e0 tx=0x7fb8000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4c2aecabfbadc3a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:48.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81659c700 1 -- 172.21.15.175:0/2062491648 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80000b6d0 con 0x7fb81808a100 2023-10-10T18:55:48.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81659c700 1 -- 172.21.15.175:0/2062491648 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80000b830 con 0x7fb81808a100 2023-10-10T18:55:48.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81659c700 1 -- 172.21.15.175:0/2062491648 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80000bbd0 con 0x7fb81808a100 2023-10-10T18:55:48.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81c8eb700 1 -- 172.21.15.175:0/2062491648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81808a100 msgr2=0x7fb818159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:48.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81c8eb700 1 --2- 172.21.15.175:0/2062491648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81808a100 0x7fb818159c30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb8000098e0 tx=0x7fb8000091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81c8eb700 1 -- 172.21.15.175:0/2062491648 shutdown_connections 2023-10-10T18:55:48.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81c8eb700 1 --2- 172.21.15.175:0/2062491648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81808a100 0x7fb818159c30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81c8eb700 1 -- 172.21.15.175:0/2062491648 >> 172.21.15.175:0/2062491648 conn(0x7fb818063d00 msgr2=0x7fb8180cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:48.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81c8eb700 1 -- 172.21.15.175:0/2062491648 shutdown_connections 2023-10-10T18:55:48.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.401+0000 7fb81c8eb700 1 -- 172.21.15.175:0/2062491648 wait complete. 2023-10-10T18:55:48.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb81c8eb700 1 Processor -- start 2023-10-10T18:55:48.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb81c8eb700 1 -- start start 2023-10-10T18:55:48.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb81c8eb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb81808a100 0x7fb81814ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:48.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb81c8eb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180cd040 0x7fb818166620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:48.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb81c8eb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81814fd90 0x7fb81814c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:48.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb81c8eb700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8180a8000 con 0x7fb81808a100 2023-10-10T18:55:48.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb81c8eb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8180a7d20 con 0x7fb81814fd90 2023-10-10T18:55:48.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb81c8eb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb818169e60 con 0x7fb8180cd040 2023-10-10T18:55:48.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb816d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb81808a100 0x7fb81814ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:48.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb816d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb81808a100 0x7fb81814ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50278/0 (socket says 172.21.15.175:50278) 2023-10-10T18:55:48.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.403+0000 7fb816d9d700 1 -- 172.21.15.175:0/1884471082 learned_addr learned my addr 172.21.15.175:0/1884471082 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:48.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb80e59c700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180cd040 0x7fb818166620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:48.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb81759e700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81814fd90 0x7fb81814c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:48.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb816d9d700 1 -- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180cd040 msgr2=0x7fb818166620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:48.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb816d9d700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180cd040 0x7fb818166620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb816d9d700 1 -- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81814fd90 msgr2=0x7fb81814c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:48.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb816d9d700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81814fd90 0x7fb81814c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb816d9d700 1 -- 172.21.15.175:0/1884471082 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800009420 con 0x7fb81808a100 2023-10-10T18:55:48.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb816d9d700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb81808a100 0x7fb81814ef40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c4e0 tx=0x7fb80800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:48.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb80f7fe700 1 -- 172.21.15.175:0/1884471082 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808011590 con 0x7fb81808a100 2023-10-10T18:55:48.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb80f7fe700 1 -- 172.21.15.175:0/1884471082 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb808011710 con 0x7fb81808a100 2023-10-10T18:55:48.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81816a140 con 0x7fb81808a100 2023-10-10T18:55:48.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb80f7fe700 1 -- 172.21.15.175:0/1884471082 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808011ad0 con 0x7fb81808a100 2023-10-10T18:55:48.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.404+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb818157420 con 0x7fb81808a100 2023-10-10T18:55:48.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.405+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8181594f0 con 0x7fb81808a100 2023-10-10T18:55:48.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.405+0000 7fb80f7fe700 1 -- 172.21.15.175:0/1884471082 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb80801b3b0 con 0x7fb81808a100 2023-10-10T18:55:48.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.406+0000 7fb80f7fe700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7f8066f60 0x7fb7f8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:48.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.406+0000 7fb80f7fe700 1 -- 172.21.15.175:0/1884471082 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fb808083ce0 con 0x7fb81808a100 2023-10-10T18:55:48.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.406+0000 7fb80e59c700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7f8066f60 0x7fb7f8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:48.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.406+0000 7fb80e59c700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7f8066f60 0x7fb7f8069410 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb8000093f0 tx=0x7fb800017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:48.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.410+0000 7fb80f7fe700 1 -- 172.21.15.175:0/1884471082 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb808050d30 con 0x7fb81808a100 2023-10-10T18:55:48.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.635+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb818067570 con 0x7fb81808a100 2023-10-10T18:55:48.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.636+0000 7fb80f7fe700 1 -- 172.21.15.175:0/1884471082 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5438 (secure 0 0 0) 0x7fb808017070 con 0x7fb81808a100 2023-10-10T18:55:48.636 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:48.636 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:44.654827+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:resolve","state_seq":25,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:48.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7f8066f60 msgr2=0x7fb7f8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:48.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7f8066f60 0x7fb7f8069410 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb8000093f0 tx=0x7fb800017000 comp rx=0 tx=0).stop 2023-10-10T18:55:48.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb81808a100 msgr2=0x7fb81814ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:48.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb81808a100 0x7fb81814ef40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c4e0 tx=0x7fb80800c8a0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 shutdown_connections 2023-10-10T18:55:48.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7f8066f60 0x7fb7f8069410 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb81814fd90 0x7fb81814c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb81808a100 0x7fb81814ef40 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 --2- 172.21.15.175:0/1884471082 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180cd040 0x7fb818166620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:48.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 >> 172.21.15.175:0/1884471082 conn(0x7fb818063d00 msgr2=0x7fb818153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:48.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 shutdown_connections 2023-10-10T18:55:48.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:48.638+0000 7fb81c8eb700 1 -- 172.21.15.175:0/1884471082 wait complete. 2023-10-10T18:55:48.641 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 83 2023-10-10T18:55:48.650 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:55:49.795 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:55:49.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:55:50.653 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:50.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.782+0000 7f279f59e700 1 Processor -- start 2023-10-10T18:55:50.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.782+0000 7f279f59e700 1 -- start start 2023-10-10T18:55:50.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279f59e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f27a006d730 0x7f27a0154450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:50.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279f59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00687a0 0x7f27a006f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:50.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279f59e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a0067c00 con 0x7f27a006d360 2023-10-10T18:55:50.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279f59e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a006fcc0 con 0x7f27a006d730 2023-10-10T18:55:50.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279f59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a006faa0 con 0x7f27a00687a0 2023-10-10T18:55:50.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279e59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f27a006d360 0x7f27a0068090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42890/0 (socket says 172.21.15.175:42890) 2023-10-10T18:55:50.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279e59c700 1 -- 172.21.15.175:0/1390528126 learned_addr learned my addr 172.21.15.175:0/1390528126 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:50.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f2795d9b700 1 --2- 172.21.15.175:0/1390528126 >> v2:172.21.15.175:3300/0 conn(0x7f27a006d730 0x7f27a0154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:50.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279ed9d700 1 --2- 172.21.15.175:0/1390528126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00687a0 0x7f27a006f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:50.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279dd9b700 1 -- 172.21.15.175:0/1390528126 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140297176 0 0) 0x7f27a0067c00 con 0x7f27a006d360 2023-10-10T18:55:50.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f279dd9b700 1 -- 172.21.15.175:0/1390528126 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c003570 con 0x7f27a006d360 2023-10-10T18:55:50.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f2795d9b700 1 -- 172.21.15.175:0/1390528126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00687a0 msgr2=0x7f27a006f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:50.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f2795d9b700 1 --2- 172.21.15.175:0/1390528126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00687a0 0x7f27a006f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:50.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f2795d9b700 1 -- 172.21.15.175:0/1390528126 >> v1:172.21.15.175:6789/0 conn(0x7f27a006d360 legacy=0x7f27a0068090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:50.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.783+0000 7f2795d9b700 1 -- 172.21.15.175:0/1390528126 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a00c5550 con 0x7f27a006d730 2023-10-10T18:55:50.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f2795d9b700 1 --2- 172.21.15.175:0/1390528126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a006d730 0x7f27a0154450 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2788009910 tx=0x7f27880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1916e8d1f9ecead server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:50.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279dd9b700 1 -- 172.21.15.175:0/1390528126 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2788011630 con 0x7f27a006d730 2023-10-10T18:55:50.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279dd9b700 1 -- 172.21.15.175:0/1390528126 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2788011790 con 0x7f27a006d730 2023-10-10T18:55:50.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279dd9b700 1 -- 172.21.15.175:0/1390528126 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2788011b30 con 0x7f27a006d730 2023-10-10T18:55:50.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279f59e700 1 -- 172.21.15.175:0/1390528126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a006d730 msgr2=0x7f27a0154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:50.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279f59e700 1 --2- 172.21.15.175:0/1390528126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a006d730 0x7f27a0154450 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2788009910 tx=0x7f27880091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:50.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279f59e700 1 -- 172.21.15.175:0/1390528126 shutdown_connections 2023-10-10T18:55:50.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279f59e700 1 --2- 172.21.15.175:0/1390528126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00687a0 0x7f27a006f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:50.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279f59e700 1 --2- 172.21.15.175:0/1390528126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a006d730 0x7f27a0154450 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:50.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279f59e700 1 -- 172.21.15.175:0/1390528126 >> 172.21.15.175:0/1390528126 conn(0x7f27a00c8d00 msgr2=0x7f27a00c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:50.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279f59e700 1 -- 172.21.15.175:0/1390528126 shutdown_connections 2023-10-10T18:55:50.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.784+0000 7f279f59e700 1 -- 172.21.15.175:0/1390528126 wait complete. 2023-10-10T18:55:50.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.785+0000 7f279f59e700 1 Processor -- start 2023-10-10T18:55:50.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279f59e700 1 -- start start 2023-10-10T18:55:50.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279f59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a00687a0 0x7f27a0147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:50.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279f59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a006d360 0x7f27a0158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:50.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279f59e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a006d730 0x7f27a00bf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:50.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279f59e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00c4e10 con 0x7f27a00687a0 2023-10-10T18:55:50.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279f59e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01640c0 con 0x7f27a006d730 2023-10-10T18:55:50.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279f59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0163c30 con 0x7f27a006d360 2023-10-10T18:55:50.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a00687a0 0x7f27a0147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a00687a0 0x7f27a0147a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50302/0 (socket says 172.21.15.175:50302) 2023-10-10T18:55:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279e59c700 1 -- 172.21.15.175:0/764133137 learned_addr learned my addr 172.21.15.175:0/764133137 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f279ed9d700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a006d730 0x7f27a00bf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.786+0000 7f2795d9b700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a006d360 0x7f27a0158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:50.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f279e59c700 1 -- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a006d360 msgr2=0x7f27a0158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f279e59c700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a006d360 0x7f27a0158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f279e59c700 1 -- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a006d730 msgr2=0x7f27a00bf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f279e59c700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a006d730 0x7f27a00bf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f279e59c700 1 -- 172.21.15.175:0/764133137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788009420 con 0x7f27a00687a0 2023-10-10T18:55:50.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f279e59c700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a00687a0 0x7f27a0147a70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f279000c7c0 tx=0x7f2790007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f2796ffd700 1 -- 172.21.15.175:0/764133137 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790005b30 con 0x7f27a00687a0 2023-10-10T18:55:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f2796ffd700 1 -- 172.21.15.175:0/764133137 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2790005c90 con 0x7f27a00687a0 2023-10-10T18:55:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a00c22c0 con 0x7f27a00687a0 2023-10-10T18:55:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f2796ffd700 1 -- 172.21.15.175:0/764133137 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790007300 con 0x7f27a00687a0 2023-10-10T18:55:50.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.787+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a0163300 con 0x7f27a00687a0 2023-10-10T18:55:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.788+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a0163f10 con 0x7f27a00687a0 2023-10-10T18:55:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.788+0000 7f2796ffd700 1 -- 172.21.15.175:0/764133137 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f279001b3b0 con 0x7f27a00687a0 2023-10-10T18:55:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.789+0000 7f2796ffd700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2780066f60 0x7f2780069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.789+0000 7f2796ffd700 1 -- 172.21.15.175:0/764133137 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f2790083c00 con 0x7f27a00687a0 2023-10-10T18:55:50.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.789+0000 7f2795d9b700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2780066f60 0x7f2780069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:50.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.789+0000 7f2795d9b700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2780066f60 0x7f2780069410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f27a00c1f10 tx=0x7f2788017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:50.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:50.793+0000 7f2796ffd700 1 -- 172.21.15.175:0/764133137 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f279008a080 con 0x7f27a00687a0 2023-10-10T18:55:51.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.021+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27a0067570 con 0x7f27a00687a0 2023-10-10T18:55:51.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.021+0000 7f2796ffd700 1 -- 172.21.15.175:0/764133137 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5440 (secure 0 0 0) 0x7f2790017020 con 0x7f27a00687a0 2023-10-10T18:55:51.022 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:51.022 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:49.670902+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:51.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2780066f60 msgr2=0x7f2780069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:51.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2780066f60 0x7f2780069410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f27a00c1f10 tx=0x7f2788017000 comp rx=0 tx=0).stop 2023-10-10T18:55:51.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a00687a0 msgr2=0x7f27a0147a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:51.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a00687a0 0x7f27a0147a70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f279000c7c0 tx=0x7f2790007e00 comp rx=0 tx=0).stop 2023-10-10T18:55:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 shutdown_connections 2023-10-10T18:55:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2780066f60 0x7f2780069410 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a006d730 0x7f27a00bf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a00687a0 0x7f27a0147a70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 --2- 172.21.15.175:0/764133137 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a006d360 0x7f27a0158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:51.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 >> 172.21.15.175:0/764133137 conn(0x7f27a00c8d00 msgr2=0x7f27a00c6940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:51.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 shutdown_connections 2023-10-10T18:55:51.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:51.024+0000 7f279f59e700 1 -- 172.21.15.175:0/764133137 wait complete. 2023-10-10T18:55:51.027 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 84 2023-10-10T18:55:51.036 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 84, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4263": { "addr": "172.21.15.191:6837/3870390806", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3870390806, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 3870390806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4263, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:reconnect", "state_seq": 27 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:55:49.670902+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4263, "mds_1": 5332, "mds_2": 5367, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:55:53.038 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:53.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.164+0000 7f877ce33700 1 Processor -- start 2023-10-10T18:55:53.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.164+0000 7f877ce33700 1 -- start start 2023-10-10T18:55:53.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.165+0000 7f877ce33700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f87780c5e40 0x7f87780c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:53.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.165+0000 7f877ce33700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f877806bf00 0x7f87780c6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:53.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.165+0000 7f877ce33700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87780c6750 0x7f8778156520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:53.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.165+0000 7f877ce33700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87781560d0 con 0x7f877806bf00 2023-10-10T18:55:53.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.165+0000 7f877ce33700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87780c6df0 con 0x7f87780c5e40 2023-10-10T18:55:53.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.165+0000 7f877ce33700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87781597e0 con 0x7f87780c6750 2023-10-10T18:55:53.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f877806bf00 0x7f87780c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:53.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f877806bf00 0x7f87780c6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50318/0 (socket says 172.21.15.175:50318) 2023-10-10T18:55:53.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 -- 172.21.15.175:0/3662739777 learned_addr learned my addr 172.21.15.175:0/3662739777 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:53.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f877659c700 1 --2- 172.21.15.175:0/3662739777 >> v2:172.21.15.191:3300/0 conn(0x7f87780c5e40 0x7f87780c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:53.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8776d9d700 1 --2- 172.21.15.175:0/3662739777 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87780c6750 0x7f8778156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:53.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 -- 172.21.15.175:0/3662739777 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87780c6750 msgr2=0x7f8778156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:53.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 --2- 172.21.15.175:0/3662739777 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87780c6750 0x7f8778156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 -- 172.21.15.175:0/3662739777 >> v2:172.21.15.191:3300/0 conn(0x7f87780c5e40 msgr2=0x7f87780c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:53.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 --2- 172.21.15.175:0/3662739777 >> v2:172.21.15.191:3300/0 conn(0x7f87780c5e40 0x7f87780c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 -- 172.21.15.175:0/3662739777 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8778159ac0 con 0x7f877806bf00 2023-10-10T18:55:53.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f8775d9b700 1 --2- 172.21.15.175:0/3662739777 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f877806bf00 0x7f87780c6210 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f87780650c0 tx=0x7f8760009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb65bac5b9ff233e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:53.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f877559a700 1 -- 172.21.15.175:0/3662739777 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876000b8f0 con 0x7f877806bf00 2023-10-10T18:55:53.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f877559a700 1 -- 172.21.15.175:0/3662739777 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876000ba50 con 0x7f877806bf00 2023-10-10T18:55:53.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.166+0000 7f877559a700 1 -- 172.21.15.175:0/3662739777 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876000bdf0 con 0x7f877806bf00 2023-10-10T18:55:53.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 -- 172.21.15.175:0/3662739777 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f877806bf00 msgr2=0x7f87780c6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:53.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 --2- 172.21.15.175:0/3662739777 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f877806bf00 0x7f87780c6210 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f87780650c0 tx=0x7f8760009250 comp rx=0 tx=0).stop 2023-10-10T18:55:53.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 -- 172.21.15.175:0/3662739777 shutdown_connections 2023-10-10T18:55:53.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 --2- 172.21.15.175:0/3662739777 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87780c6750 0x7f8778156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 --2- 172.21.15.175:0/3662739777 >> v2:172.21.15.191:3300/0 conn(0x7f87780c5e40 0x7f87780c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 --2- 172.21.15.175:0/3662739777 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f877806bf00 0x7f87780c6210 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 -- 172.21.15.175:0/3662739777 >> 172.21.15.175:0/3662739777 conn(0x7f8778068900 msgr2=0x7f8778068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:53.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 -- 172.21.15.175:0/3662739777 shutdown_connections 2023-10-10T18:55:53.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.167+0000 7f877ce33700 1 -- 172.21.15.175:0/3662739777 wait complete. 2023-10-10T18:55:53.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.168+0000 7f877ce33700 1 Processor -- start 2023-10-10T18:55:53.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.168+0000 7f877ce33700 1 -- start start 2023-10-10T18:55:53.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877ce33700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f877806bf00 0x7f877813c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:53.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877ce33700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87780c5e40 0x7f87781396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:53.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877ce33700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8778138b60 0x7f8778139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:53.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877ce33700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87780c8ff0 con 0x7f87780c5e40 2023-10-10T18:55:53.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877ce33700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8778153ee0 con 0x7f8778138b60 2023-10-10T18:55:53.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877ce33700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87780ca350 con 0x7f877806bf00 2023-10-10T18:55:53.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87780c5e40 0x7f87781396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:53.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87780c5e40 0x7f87781396d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50332/0 (socket says 172.21.15.175:50332) 2023-10-10T18:55:53.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 -- 172.21.15.175:0/2634885124 learned_addr learned my addr 172.21.15.175:0/2634885124 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:53.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8776d9d700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8778138b60 0x7f8778139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:53.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877659c700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f877806bf00 0x7f877813c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:53.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 -- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f877806bf00 msgr2=0x7f877813c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:53.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f877806bf00 0x7f877813c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 -- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8778138b60 msgr2=0x7f8778139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:53.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8778138b60 0x7f8778139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 -- 172.21.15.175:0/2634885124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87600094f0 con 0x7f87780c5e40 2023-10-10T18:55:53.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f8775d9b700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87780c5e40 0x7f87781396d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f877806b450 tx=0x7f8760017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:53.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f876e7fc700 1 -- 172.21.15.175:0/2634885124 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8760024440 con 0x7f87780c5e40 2023-10-10T18:55:53.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f876e7fc700 1 -- 172.21.15.175:0/2634885124 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87600245a0 con 0x7f87780c5e40 2023-10-10T18:55:53.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87780c7ca0 con 0x7f87780c5e40 2023-10-10T18:55:53.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f876e7fc700 1 -- 172.21.15.175:0/2634885124 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8760024960 con 0x7f87780c5e40 2023-10-10T18:55:53.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.169+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87780c8240 con 0x7f87780c5e40 2023-10-10T18:55:53.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.170+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87780c91a0 con 0x7f87780c5e40 2023-10-10T18:55:53.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.171+0000 7f876e7fc700 1 -- 172.21.15.175:0/2634885124 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f876001a420 con 0x7f87780c5e40 2023-10-10T18:55:53.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.171+0000 7f876e7fc700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8758066f30 0x7f87580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:53.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.171+0000 7f876e7fc700 1 -- 172.21.15.175:0/2634885124 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f8760085820 con 0x7f87780c5e40 2023-10-10T18:55:53.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.171+0000 7f877659c700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8758066f30 0x7f87580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:53.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.172+0000 7f877659c700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8758066f30 0x7f87580693e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8768007680 tx=0x7f87680098d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:53.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.176+0000 7f876e7fc700 1 -- 172.21.15.175:0/2634885124 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87780c91a0 con 0x7f87780c5e40 2023-10-10T18:55:53.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.402+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8778067570 con 0x7f87780c5e40 2023-10-10T18:55:53.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.402+0000 7f876e7fc700 1 -- 172.21.15.175:0/2634885124 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5436 (secure 0 0 0) 0x7f8760052870 con 0x7f87780c5e40 2023-10-10T18:55:53.402 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:53.403 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:51.677734+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:53.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8758066f30 msgr2=0x7f87580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:53.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8758066f30 0x7f87580693e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8768007680 tx=0x7f87680098d0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87780c5e40 msgr2=0x7f87781396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:53.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87780c5e40 0x7f87781396d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f877806b450 tx=0x7f8760017000 comp rx=0 tx=0).stop 2023-10-10T18:55:53.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 shutdown_connections 2023-10-10T18:55:53.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8758066f30 0x7f87580693e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8778138b60 0x7f8778139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f877806bf00 0x7f877813c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.404+0000 7f877ce33700 1 --2- 172.21.15.175:0/2634885124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87780c5e40 0x7f87781396d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:53.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.405+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 >> 172.21.15.175:0/2634885124 conn(0x7f8778068900 msgr2=0x7f8778068cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:53.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.405+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 shutdown_connections 2023-10-10T18:55:53.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:53.405+0000 7f877ce33700 1 -- 172.21.15.175:0/2634885124 wait complete. 2023-10-10T18:55:53.407 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 85 2023-10-10T18:55:53.417 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:55:55.418 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:55.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.545+0000 7f151adb8700 1 Processor -- start 2023-10-10T18:55:55.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.545+0000 7f151adb8700 1 -- start start 2023-10-10T18:55:55.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.546+0000 7f151adb8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f15140c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:55.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.546+0000 7f151adb8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1514156c90 0x7f15140c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:55.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.546+0000 7f151adb8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151415a6c0 con 0x7f15140c2b80 2023-10-10T18:55:55.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.546+0000 7f151adb8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151414d4a0 con 0x7f1514156c90 2023-10-10T18:55:55.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.546+0000 7f151adb8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1514159830 con 0x7f15140c35a0 2023-10-10T18:55:55.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f15140c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:55.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f1519355700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1514156c90 0x7f15140c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:55.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f15140c6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:43972/0 (socket says 172.21.15.175:43972) 2023-10-10T18:55:55.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f1519355700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1514156c90 0x7f15140c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:47560/0 (socket says 172.21.15.175:47560) 2023-10-10T18:55:55.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150bfff700 1 -- 172.21.15.175:0/2540481327 learned_addr learned my addr 172.21.15.175:0/2540481327 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:55.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f1518b54700 1 --1- 172.21.15.175:0/2540481327 >> v1:172.21.15.191:6789/0 conn(0x7f15140c2b80 0x7f1514156580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:53638/0 (socket says 172.21.15.175:53638) 2023-10-10T18:55:55.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150bfff700 1 -- 172.21.15.175:0/2540481327 >> v2:172.21.15.191:3300/0 conn(0x7f1514156c90 msgr2=0x7f15140c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:55.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150bfff700 1 --2- 172.21.15.175:0/2540481327 >> v2:172.21.15.191:3300/0 conn(0x7f1514156c90 0x7f15140c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150bfff700 1 -- 172.21.15.175:0/2540481327 >> v1:172.21.15.191:6789/0 conn(0x7f15140c2b80 legacy=0x7f1514156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:55.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150bfff700 1 -- 172.21.15.175:0/2540481327 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1514159ab0 con 0x7f15140c35a0 2023-10-10T18:55:55.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f1519355700 1 --2- 172.21.15.175:0/2540481327 >> v2:172.21.15.191:3300/0 conn(0x7f1514156c90 0x7f15140c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:55:55.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150b7fe700 1 -- 172.21.15.175:0/2540481327 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343566266 0 0) 0x7f151415a6c0 con 0x7f15140c2b80 2023-10-10T18:55:55.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150bfff700 1 --2- 172.21.15.175:0/2540481327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f15140c6a30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f15040099b0 tx=0x7f1504009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6272a1751dc379c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:55.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150b7fe700 1 -- 172.21.15.175:0/2540481327 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504009ea0 con 0x7f15140c35a0 2023-10-10T18:55:55.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150b7fe700 1 -- 172.21.15.175:0/2540481327 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150400b8f0 con 0x7f15140c35a0 2023-10-10T18:55:55.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.547+0000 7f150b7fe700 1 -- 172.21.15.175:0/2540481327 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150400bc90 con 0x7f15140c35a0 2023-10-10T18:55:55.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.548+0000 7f151adb8700 1 -- 172.21.15.175:0/2540481327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 msgr2=0x7f15140c6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:55.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.548+0000 7f151adb8700 1 --2- 172.21.15.175:0/2540481327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f15140c6a30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f15040099b0 tx=0x7f1504009250 comp rx=0 tx=0).stop 2023-10-10T18:55:55.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.548+0000 7f151adb8700 1 -- 172.21.15.175:0/2540481327 shutdown_connections 2023-10-10T18:55:55.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.548+0000 7f151adb8700 1 --2- 172.21.15.175:0/2540481327 >> v2:172.21.15.191:3300/0 conn(0x7f1514156c90 0x7f15140c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.548+0000 7f151adb8700 1 --2- 172.21.15.175:0/2540481327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f15140c6a30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.548+0000 7f151adb8700 1 -- 172.21.15.175:0/2540481327 >> 172.21.15.175:0/2540481327 conn(0x7f15140c56a0 msgr2=0x7f15140c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:55.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.548+0000 7f151adb8700 1 -- 172.21.15.175:0/2540481327 shutdown_connections 2023-10-10T18:55:55.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.548+0000 7f151adb8700 1 -- 172.21.15.175:0/2540481327 wait complete. 2023-10-10T18:55:55.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.549+0000 7f151adb8700 1 Processor -- start 2023-10-10T18:55:55.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.549+0000 7f151adb8700 1 -- start start 2023-10-10T18:55:55.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f151adb8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15140c2b80 0x7f15140b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:55.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f151adb8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f151406e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:55.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f151adb8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1514156c90 0x7f15141696c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:55.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f151adb8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15140ba360 con 0x7f1514156c90 2023-10-10T18:55:55.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f151adb8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1514166e00 con 0x7f15140c2b80 2023-10-10T18:55:55.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f151adb8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15140ba510 con 0x7f15140c35a0 2023-10-10T18:55:55.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1519355700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1514156c90 0x7f15141696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:55.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1519355700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1514156c90 0x7f15141696c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56246/0 (socket says 172.21.15.175:56246) 2023-10-10T18:55:55.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1519355700 1 -- 172.21.15.175:0/2841366090 learned_addr learned my addr 172.21.15.175:0/2841366090 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:55.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1518b54700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15140c2b80 0x7f15140b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:55.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f150bfff700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f151406e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:55.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1519355700 1 -- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 msgr2=0x7f151406e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:55.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1519355700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f151406e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1519355700 1 -- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15140c2b80 msgr2=0x7f15140b9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:55.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1519355700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15140c2b80 0x7f15140b9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.550+0000 7f1519355700 1 -- 172.21.15.175:0/2841366090 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c009420 con 0x7f1514156c90 2023-10-10T18:55:55.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.551+0000 7f1519355700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1514156c90 0x7f15141696c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f150c00ec50 tx=0x7f150c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:55.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.551+0000 7f1508ff9700 1 -- 172.21.15.175:0/2841366090 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c00c850 con 0x7f1514156c90 2023-10-10T18:55:55.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.551+0000 7f1508ff9700 1 -- 172.21.15.175:0/2841366090 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c00c9b0 con 0x7f1514156c90 2023-10-10T18:55:55.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.551+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15040094f0 con 0x7f1514156c90 2023-10-10T18:55:55.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.551+0000 7f1508ff9700 1 -- 172.21.15.175:0/2841366090 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c00cd30 con 0x7f1514156c90 2023-10-10T18:55:55.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.551+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1514166180 con 0x7f1514156c90 2023-10-10T18:55:55.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.552+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1514166f40 con 0x7f1514156c90 2023-10-10T18:55:55.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.552+0000 7f1508ff9700 1 -- 172.21.15.175:0/2841366090 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f150c013680 con 0x7f1514156c90 2023-10-10T18:55:55.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.552+0000 7f1508ff9700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1500066f30 0x7f15000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:55.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.553+0000 7f1508ff9700 1 -- 172.21.15.175:0/2841366090 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f150c084a40 con 0x7f1514156c90 2023-10-10T18:55:55.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.553+0000 7f1518b54700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1500066f30 0x7f15000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:55.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.553+0000 7f1518b54700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1500066f30 0x7f15000693e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1510002090 tx=0x7f1510001a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:55.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.557+0000 7f1508ff9700 1 -- 172.21.15.175:0/2841366090 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f150c086e60 con 0x7f1514156c90 2023-10-10T18:55:55.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.783+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f151406a140 con 0x7f1514156c90 2023-10-10T18:55:55.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.783+0000 7f1508ff9700 1 -- 172.21.15.175:0/2841366090 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5433 (secure 0 0 0) 0x7f150c017090 con 0x7f1514156c90 2023-10-10T18:55:55.784 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:55.784 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:54.688290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:55.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1500066f30 msgr2=0x7f15000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:55.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1500066f30 0x7f15000693e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1510002090 tx=0x7f1510001a30 comp rx=0 tx=0).stop 2023-10-10T18:55:55.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1514156c90 msgr2=0x7f15141696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:55.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1514156c90 0x7f15141696c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f150c00ec50 tx=0x7f150c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 shutdown_connections 2023-10-10T18:55:55.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1500066f30 0x7f15000693e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1514156c90 0x7f15141696c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15140c2b80 0x7f15140b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 --2- 172.21.15.175:0/2841366090 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15140c35a0 0x7f151406e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:55.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 >> 172.21.15.175:0/2841366090 conn(0x7f15140c56a0 msgr2=0x7f15140c7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:55.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 shutdown_connections 2023-10-10T18:55:55.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:55.786+0000 7f151adb8700 1 -- 172.21.15.175:0/2841366090 wait complete. 2023-10-10T18:55:55.788 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 86 2023-10-10T18:55:55.798 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:55:57.799 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:55:57.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.925+0000 7fc7b08b4700 1 Processor -- start 2023-10-10T18:55:57.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.926+0000 7fc7b08b4700 1 -- start start 2023-10-10T18:55:57.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.926+0000 7fc7b08b4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0c9950 0x7fc7ac155f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:57.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.926+0000 7fc7b08b4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7ac0b04a0 0x7fc7ac0c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:57.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7b08b4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac0b0010 con 0x7fc7ac0c9d20 2023-10-10T18:55:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7b08b4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac15a000 con 0x7fc7ac0b04a0 2023-10-10T18:55:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7b08b4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac15a2e0 con 0x7fc7ac0c9950 2023-10-10T18:55:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7a9d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc7ac0c9d20 0x7fc7ac159550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55628/0 (socket says 172.21.15.175:55628) 2023-10-10T18:55:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7a9d9b700 1 -- 172.21.15.175:0/1586378930 learned_addr learned my addr 172.21.15.175:0/1586378930 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:55:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7aad9d700 1 --2- 172.21.15.175:0/1586378930 >> v2:172.21.15.191:3300/0 conn(0x7fc7ac0b04a0 0x7fc7ac0c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:57.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7aa59c700 1 --2- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0c9950 0x7fc7ac155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:57.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7a959a700 1 -- 172.21.15.175:0/1586378930 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403834348 0 0) 0x7fc7ac0b0010 con 0x7fc7ac0c9d20 2023-10-10T18:55:57.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7a959a700 1 -- 172.21.15.175:0/1586378930 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc794003570 con 0x7fc7ac0c9d20 2023-10-10T18:55:57.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7aad9d700 1 -- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0c9950 msgr2=0x7fc7ac155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:55:57.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7a959a700 1 -- 172.21.15.175:0/1586378930 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3192674980 0 0) 0x7fc794003570 con 0x7fc7ac0c9d20 2023-10-10T18:55:57.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7aad9d700 1 --2- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0c9950 0x7fc7ac155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:57.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7aad9d700 1 -- 172.21.15.175:0/1586378930 >> v1:172.21.15.175:6789/0 conn(0x7fc7ac0c9d20 legacy=0x7fc7ac159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:57.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7aad9d700 1 -- 172.21.15.175:0/1586378930 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0af6e0 con 0x7fc7ac0b04a0 2023-10-10T18:55:57.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7aa59c700 1 --2- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0c9950 0x7fc7ac155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:55:57.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.927+0000 7fc7aad9d700 1 --2- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0b04a0 0x7fc7ac0c8d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009e60 tx=0x7fc7a4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0c2b95a3d26acc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:57.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7a959a700 1 -- 172.21.15.175:0/1586378930 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a400e070 con 0x7fc7ac0b04a0 2023-10-10T18:55:57.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7a959a700 1 -- 172.21.15.175:0/1586378930 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a400b830 con 0x7fc7ac0b04a0 2023-10-10T18:55:57.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7a959a700 1 -- 172.21.15.175:0/1586378930 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0001470 con 0x7fc7ac0b04a0 2023-10-10T18:55:57.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7b08b4700 1 -- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0b04a0 msgr2=0x7fc7ac0c8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:57.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0b04a0 0x7fc7ac0c8d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009e60 tx=0x7fc7a4009780 comp rx=0 tx=0).stop 2023-10-10T18:55:57.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7b08b4700 1 -- 172.21.15.175:0/1586378930 shutdown_connections 2023-10-10T18:55:57.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0b04a0 0x7fc7ac0c8d40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:57.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/1586378930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0c9950 0x7fc7ac155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:57.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7b08b4700 1 -- 172.21.15.175:0/1586378930 >> 172.21.15.175:0/1586378930 conn(0x7fc7ac063230 msgr2=0x7fc7ac063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:57.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7b08b4700 1 -- 172.21.15.175:0/1586378930 shutdown_connections 2023-10-10T18:55:57.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.928+0000 7fc7b08b4700 1 -- 172.21.15.175:0/1586378930 wait complete. 2023-10-10T18:55:57.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.929+0000 7fc7b08b4700 1 Processor -- start 2023-10-10T18:55:57.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.929+0000 7fc7b08b4700 1 -- start start 2023-10-10T18:55:57.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7b08b4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0b04a0 0x7fc7ac169460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:57.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7b08b4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7ac0c9950 0x7fc7ac1699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:57.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7b08b4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0c9d20 0x7fc7ac14a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:57.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7b08b4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0c3d30 con 0x7fc7ac0c9950 2023-10-10T18:55:57.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7b08b4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac1521b0 con 0x7fc7ac0c9d20 2023-10-10T18:55:57.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7b08b4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac151da0 con 0x7fc7ac0b04a0 2023-10-10T18:55:57.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7a9d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7ac0c9950 0x7fc7ac1699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:57.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7a9d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7ac0c9950 0x7fc7ac1699a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56256/0 (socket says 172.21.15.175:56256) 2023-10-10T18:55:57.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7a9d9b700 1 -- 172.21.15.175:0/412278703 learned_addr learned my addr 172.21.15.175:0/412278703 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:55:57.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7aad9d700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0c9d20 0x7fc7ac14a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:57.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7aa59c700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0b04a0 0x7fc7ac169460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:57.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7a9d9b700 1 -- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0b04a0 msgr2=0x7fc7ac169460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:57.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7a9d9b700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0b04a0 0x7fc7ac169460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:57.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7a9d9b700 1 -- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0c9d20 msgr2=0x7fc7ac14a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:57.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7a9d9b700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0c9d20 0x7fc7ac14a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:57.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7a9d9b700 1 -- 172.21.15.175:0/412278703 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c009420 con 0x7fc7ac0c9950 2023-10-10T18:55:57.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.930+0000 7fc7aa59c700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0b04a0 0x7fc7ac169460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:55:57.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.931+0000 7fc7a9d9b700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7ac0c9950 0x7fc7ac1699a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c990 tx=0x7fc7a0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:57.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.931+0000 7fc79a7fc700 1 -- 172.21.15.175:0/412278703 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0005b30 con 0x7fc7ac0c9950 2023-10-10T18:55:57.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.931+0000 7fc79a7fc700 1 -- 172.21.15.175:0/412278703 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a0005c90 con 0x7fc7ac0c9950 2023-10-10T18:55:57.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.931+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a4009a20 con 0x7fc7ac0c9950 2023-10-10T18:55:57.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.931+0000 7fc79a7fc700 1 -- 172.21.15.175:0/412278703 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0007320 con 0x7fc7ac0c9950 2023-10-10T18:55:57.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.931+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac152400 con 0x7fc7ac0c9950 2023-10-10T18:55:57.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.932+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac067570 con 0x7fc7ac0c9950 2023-10-10T18:55:57.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.932+0000 7fc79a7fc700 1 -- 172.21.15.175:0/412278703 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc7a000a590 con 0x7fc7ac0c9950 2023-10-10T18:55:57.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.932+0000 7fc79a7fc700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc78c066f30 0x7fc78c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:55:57.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.933+0000 7fc79a7fc700 1 -- 172.21.15.175:0/412278703 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fc7a00844e0 con 0x7fc7ac0c9950 2023-10-10T18:55:57.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.933+0000 7fc7aa59c700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc78c066f30 0x7fc78c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:55:57.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.933+0000 7fc7aa59c700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc78c066f30 0x7fc78c0693e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0093f0 tx=0x7fc79c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:55:57.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:57.937+0000 7fc79a7fc700 1 -- 172.21.15.175:0/412278703 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7a0051660 con 0x7fc7ac0c9950 2023-10-10T18:55:58.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.160+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7ac0cbd80 con 0x7fc7ac0c9950 2023-10-10T18:55:58.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.161+0000 7fc79a7fc700 1 -- 172.21.15.175:0/412278703 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5433 (secure 0 0 0) 0x7fc7a0054c80 con 0x7fc7ac0c9950 2023-10-10T18:55:58.161 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:55:58.161 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:54.688290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:55:58.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc78c066f30 msgr2=0x7fc78c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:58.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc78c066f30 0x7fc78c0693e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0093f0 tx=0x7fc79c008040 comp rx=0 tx=0).stop 2023-10-10T18:55:58.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7ac0c9950 msgr2=0x7fc7ac1699a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:55:58.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7ac0c9950 0x7fc7ac1699a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c990 tx=0x7fc7a0001a00 comp rx=0 tx=0).stop 2023-10-10T18:55:58.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 shutdown_connections 2023-10-10T18:55:58.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc78c066f30 0x7fc78c0693e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:58.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac0c9d20 0x7fc7ac14a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:58.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7ac0b04a0 0x7fc7ac169460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:58.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 --2- 172.21.15.175:0/412278703 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7ac0c9950 0x7fc7ac1699a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:55:58.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 >> 172.21.15.175:0/412278703 conn(0x7fc7ac063230 msgr2=0x7fc7ac068f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:55:58.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 shutdown_connections 2023-10-10T18:55:58.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:55:58.163+0000 7fc7b08b4700 1 -- 172.21.15.175:0/412278703 wait complete. 2023-10-10T18:55:58.165 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 86 2023-10-10T18:55:58.175 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:56:00.176 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:00.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.303+0000 7f1848ec4700 1 Processor -- start 2023-10-10T18:56:00.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.303+0000 7f1848ec4700 1 -- start start 2023-10-10T18:56:00.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1848ec4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18441513a0 0x7f18440c0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:00.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1848ec4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18440b01d0 0x7f1844153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:00.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1848ec4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1844154300 0x7f184406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:00.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1848ec4700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1844151770 con 0x7f18440b01d0 2023-10-10T18:56:00.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1848ec4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18440bf2b0 con 0x7f1844154300 2023-10-10T18:56:00.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1848ec4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18440bf580 con 0x7f18441513a0 2023-10-10T18:56:00.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1842ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18440b01d0 0x7f1844153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:00.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1842ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18440b01d0 0x7f1844153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56258/0 (socket says 172.21.15.175:56258) 2023-10-10T18:56:00.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f18437fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18441513a0 0x7f18440c0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:00.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1842ffd700 1 -- 172.21.15.175:0/356249910 learned_addr learned my addr 172.21.15.175:0/356249910 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:00.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1843fff700 1 --2- 172.21.15.175:0/356249910 >> v2:172.21.15.191:3300/0 conn(0x7f1844154300 0x7f184406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:00.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1842ffd700 1 -- 172.21.15.175:0/356249910 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18441513a0 msgr2=0x7f18440c0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:00.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1842ffd700 1 --2- 172.21.15.175:0/356249910 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18441513a0 0x7f18440c0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1842ffd700 1 -- 172.21.15.175:0/356249910 >> v2:172.21.15.191:3300/0 conn(0x7f1844154300 msgr2=0x7f184406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:00.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.304+0000 7f1842ffd700 1 --2- 172.21.15.175:0/356249910 >> v2:172.21.15.191:3300/0 conn(0x7f1844154300 0x7f184406cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1842ffd700 1 -- 172.21.15.175:0/356249910 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18440bf860 con 0x7f18440b01d0 2023-10-10T18:56:00.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1842ffd700 1 --2- 172.21.15.175:0/356249910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18440b01d0 0x7f1844153dc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f184406cd00 tx=0x7f1838009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9a7a969b641b43 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:00.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f18427fc700 1 -- 172.21.15.175:0/356249910 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838005c10 con 0x7f18440b01d0 2023-10-10T18:56:00.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f18427fc700 1 -- 172.21.15.175:0/356249910 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1838005d70 con 0x7f18440b01d0 2023-10-10T18:56:00.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f18427fc700 1 -- 172.21.15.175:0/356249910 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838006260 con 0x7f18440b01d0 2023-10-10T18:56:00.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 -- 172.21.15.175:0/356249910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18440b01d0 msgr2=0x7f1844153dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:00.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 --2- 172.21.15.175:0/356249910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18440b01d0 0x7f1844153dc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f184406cd00 tx=0x7f1838009220 comp rx=0 tx=0).stop 2023-10-10T18:56:00.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 -- 172.21.15.175:0/356249910 shutdown_connections 2023-10-10T18:56:00.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 --2- 172.21.15.175:0/356249910 >> v2:172.21.15.191:3300/0 conn(0x7f1844154300 0x7f184406cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 --2- 172.21.15.175:0/356249910 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18441513a0 0x7f18440c0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 --2- 172.21.15.175:0/356249910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18440b01d0 0x7f1844153dc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 -- 172.21.15.175:0/356249910 >> 172.21.15.175:0/356249910 conn(0x7f184406b310 msgr2=0x7f184406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:00.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 -- 172.21.15.175:0/356249910 shutdown_connections 2023-10-10T18:56:00.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.305+0000 7f1848ec4700 1 -- 172.21.15.175:0/356249910 wait complete. 2023-10-10T18:56:00.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.307+0000 7f1848ec4700 1 Processor -- start 2023-10-10T18:56:00.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.307+0000 7f1848ec4700 1 -- start start 2023-10-10T18:56:00.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.307+0000 7f1848ec4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18441513a0 0x7f18441470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:00.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.307+0000 7f1848ec4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1844154300 0x7f1844167070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:00.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.307+0000 7f1848ec4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1844147990 0x7f1844147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:00.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.307+0000 7f1848ec4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18440c9da0 con 0x7f18441513a0 2023-10-10T18:56:00.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.307+0000 7f1848ec4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1844163af0 con 0x7f1844147990 2023-10-10T18:56:00.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.307+0000 7f1848ec4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18440c97e0 con 0x7f1844154300 2023-10-10T18:56:00.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f18437fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18441513a0 0x7f18441470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:00.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f1842ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1844154300 0x7f1844167070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:00.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f1842ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1844154300 0x7f1844167070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:44026/0 (socket says 172.21.15.175:44026) 2023-10-10T18:56:00.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f1843fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1844147990 0x7f1844147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:00.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f1842ffd700 1 -- 172.21.15.175:0/535743451 learned_addr learned my addr 172.21.15.175:0/535743451 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:00.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f18437fe700 1 -- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1844154300 msgr2=0x7f1844167070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:00.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f18437fe700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1844154300 0x7f1844167070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f18437fe700 1 -- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1844147990 msgr2=0x7f1844147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:00.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f18437fe700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1844147990 0x7f1844147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f18437fe700 1 -- 172.21.15.175:0/535743451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1838009480 con 0x7f18441513a0 2023-10-10T18:56:00.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f1842ffd700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1844154300 0x7f1844167070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:56:00.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f18437fe700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18441513a0 0x7f18441470d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f183400ea50 tx=0x7f1834009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:00.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f182b7fe700 1 -- 172.21.15.175:0/535743451 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183400c640 con 0x7f18441513a0 2023-10-10T18:56:00.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f182b7fe700 1 -- 172.21.15.175:0/535743451 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183400c7a0 con 0x7f18441513a0 2023-10-10T18:56:00.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18440c9ac0 con 0x7f18441513a0 2023-10-10T18:56:00.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f182b7fe700 1 -- 172.21.15.175:0/535743451 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183400cb20 con 0x7f18441513a0 2023-10-10T18:56:00.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.308+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1844163fe0 con 0x7f18441513a0 2023-10-10T18:56:00.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.309+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1844067570 con 0x7f18441513a0 2023-10-10T18:56:00.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.309+0000 7f182b7fe700 1 -- 172.21.15.175:0/535743451 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1834013470 con 0x7f18441513a0 2023-10-10T18:56:00.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.310+0000 7f182b7fe700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1830066f30 0x7f18300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:00.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.310+0000 7f182b7fe700 1 -- 172.21.15.175:0/535743451 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f1834084910 con 0x7f18441513a0 2023-10-10T18:56:00.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.310+0000 7f1842ffd700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1830066f30 0x7f18300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:00.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.310+0000 7f1842ffd700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1830066f30 0x7f18300693e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f184406d980 tx=0x7f1838016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:00.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.314+0000 7f182b7fe700 1 -- 172.21.15.175:0/535743451 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1834051960 con 0x7f18441513a0 2023-10-10T18:56:00.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.545+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1844150a20 con 0x7f18441513a0 2023-10-10T18:56:00.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.545+0000 7f182b7fe700 1 -- 172.21.15.175:0/535743451 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5433 (secure 0 0 0) 0x7f1834054f80 con 0x7f18441513a0 2023-10-10T18:56:00.546 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:00.546 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:55:54.688290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:00.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1830066f30 msgr2=0x7f18300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:00.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1830066f30 0x7f18300693e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f184406d980 tx=0x7f1838016000 comp rx=0 tx=0).stop 2023-10-10T18:56:00.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18441513a0 msgr2=0x7f18441470d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:00.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18441513a0 0x7f18441470d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f183400ea50 tx=0x7f1834009120 comp rx=0 tx=0).stop 2023-10-10T18:56:00.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 shutdown_connections 2023-10-10T18:56:00.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1830066f30 0x7f18300693e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1844147990 0x7f1844147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18441513a0 0x7f18441470d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 --2- 172.21.15.175:0/535743451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1844154300 0x7f1844167070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:00.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.547+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 >> 172.21.15.175:0/535743451 conn(0x7f184406b310 msgr2=0x7f184414f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:00.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.548+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 shutdown_connections 2023-10-10T18:56:00.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:00.548+0000 7f1848ec4700 1 -- 172.21.15.175:0/535743451 wait complete. 2023-10-10T18:56:00.551 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 86 2023-10-10T18:56:00.559 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:56:01.546 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T18:56:01.546+0000 7fc568727700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T18:56:02.560 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:02.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.684+0000 7ff0c7f22700 1 Processor -- start 2023-10-10T18:56:02.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.685+0000 7ff0c7f22700 1 -- start start 2023-10-10T18:56:02.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.685+0000 7ff0c7f22700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01573d0 0x7ff0c00c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:02.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.685+0000 7ff0c7f22700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff0c00c4f20 0x7ff0c014d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:02.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.685+0000 7ff0c7f22700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c0154aa0 con 0x7ff0c01577a0 2023-10-10T18:56:02.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.685+0000 7ff0c7f22700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c00bfdf0 con 0x7ff0c00c4f20 2023-10-10T18:56:02.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.685+0000 7ff0c7f22700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c0063dc0 con 0x7ff0c01573d0 2023-10-10T18:56:02.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c54bd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff0c01577a0 0x7ff0c00bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55644/0 (socket says 172.21.15.175:55644) 2023-10-10T18:56:02.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c54bd700 1 -- 172.21.15.175:0/1290271278 learned_addr learned my addr 172.21.15.175:0/1290271278 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:56:02.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c64bf700 1 --2- 172.21.15.175:0/1290271278 >> v2:172.21.15.191:3300/0 conn(0x7ff0c00c4f20 0x7ff0c014d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:02.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c5cbe700 1 --2- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01573d0 0x7ff0c00c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:02.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c4cbc700 1 -- 172.21.15.175:0/1290271278 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844881669 0 0) 0x7ff0c0154aa0 con 0x7ff0c01577a0 2023-10-10T18:56:02.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c4cbc700 1 -- 172.21.15.175:0/1290271278 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ac003570 con 0x7ff0c01577a0 2023-10-10T18:56:02.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c64bf700 1 -- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01573d0 msgr2=0x7ff0c00c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:02.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c64bf700 1 --2- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01573d0 0x7ff0c00c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c64bf700 1 -- 172.21.15.175:0/1290271278 >> v1:172.21.15.175:6789/0 conn(0x7ff0c01577a0 legacy=0x7ff0c00bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:02.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c64bf700 1 -- 172.21.15.175:0/1290271278 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c00c5380 con 0x7ff0c00c4f20 2023-10-10T18:56:02.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c5cbe700 1 --2- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01573d0 0x7ff0c00c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:56:02.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c64bf700 1 --2- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c014d450 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80098e0 tx=0x7ff0b80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb4f2286a38ddd59 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:02.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c4cbc700 1 -- 172.21.15.175:0/1290271278 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8011630 con 0x7ff0c00c4f20 2023-10-10T18:56:02.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c4cbc700 1 -- 172.21.15.175:0/1290271278 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b8011790 con 0x7ff0c00c4f20 2023-10-10T18:56:02.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.686+0000 7ff0c4cbc700 1 -- 172.21.15.175:0/1290271278 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8011b10 con 0x7ff0c00c4f20 2023-10-10T18:56:02.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.687+0000 7ff0c7f22700 1 -- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 msgr2=0x7ff0c014d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:02.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.687+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c014d450 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80098e0 tx=0x7ff0b80091a0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.687+0000 7ff0c7f22700 1 -- 172.21.15.175:0/1290271278 shutdown_connections 2023-10-10T18:56:02.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.687+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c014d450 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.687+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/1290271278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01573d0 0x7ff0c00c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.687+0000 7ff0c7f22700 1 -- 172.21.15.175:0/1290271278 >> 172.21.15.175:0/1290271278 conn(0x7ff0c006dc70 msgr2=0x7ff0c0063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:02.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.687+0000 7ff0c7f22700 1 -- 172.21.15.175:0/1290271278 shutdown_connections 2023-10-10T18:56:02.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.687+0000 7ff0c7f22700 1 -- 172.21.15.175:0/1290271278 wait complete. 2023-10-10T18:56:02.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.688+0000 7ff0c7f22700 1 Processor -- start 2023-10-10T18:56:02.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.688+0000 7ff0c7f22700 1 -- start start 2023-10-10T18:56:02.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c7f22700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c0166ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:02.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c7f22700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff0c01573d0 0x7ff0c01671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:02.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c7f22700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01577a0 0x7ff0c0165530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:02.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c7f22700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00c7b10 con 0x7ff0c01573d0 2023-10-10T18:56:02.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c7f22700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c01650a0 con 0x7ff0c00c4f20 2023-10-10T18:56:02.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c7f22700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00c45a0 con 0x7ff0c01577a0 2023-10-10T18:56:02.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c5cbe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c0166ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:02.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c5cbe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c0166ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:47626/0 (socket says 172.21.15.175:47626) 2023-10-10T18:56:02.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c5cbe700 1 -- 172.21.15.175:0/951227956 learned_addr learned my addr 172.21.15.175:0/951227956 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:02.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c64bf700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01577a0 0x7ff0c0165530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:02.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c54bd700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff0c01573d0 0x7ff0c01671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:02.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c5cbe700 1 -- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01577a0 msgr2=0x7ff0c0165530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:02.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c5cbe700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01577a0 0x7ff0c0165530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c5cbe700 1 -- 172.21.15.175:0/951227956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff0c01573d0 msgr2=0x7ff0c01671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:02.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.689+0000 7ff0c5cbe700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff0c01573d0 0x7ff0c01671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.690+0000 7ff0c5cbe700 1 -- 172.21.15.175:0/951227956 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc009a20 con 0x7ff0c00c4f20 2023-10-10T18:56:02.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.690+0000 7ff0c5cbe700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c0166ca0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c170 tx=0x7ff0bc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:02.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.690+0000 7ff0b5ffb700 1 -- 172.21.15.175:0/951227956 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc00ccb0 con 0x7ff0c00c4f20 2023-10-10T18:56:02.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.690+0000 7ff0b5ffb700 1 -- 172.21.15.175:0/951227956 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc00f040 con 0x7ff0c00c4f20 2023-10-10T18:56:02.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.690+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b8009420 con 0x7ff0c00c4f20 2023-10-10T18:56:02.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.690+0000 7ff0b5ffb700 1 -- 172.21.15.175:0/951227956 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc014480 con 0x7ff0c00c4f20 2023-10-10T18:56:02.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.690+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c0164a50 con 0x7ff0c00c4f20 2023-10-10T18:56:02.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.691+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c0067570 con 0x7ff0c00c4f20 2023-10-10T18:56:02.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.692+0000 7ff0b5ffb700 1 -- 172.21.15.175:0/951227956 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff0bc0147f0 con 0x7ff0c00c4f20 2023-10-10T18:56:02.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.692+0000 7ff0b5ffb700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0a4066f60 0x7ff0a4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:02.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.692+0000 7ff0b5ffb700 1 -- 172.21.15.175:0/951227956 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7ff0bc085530 con 0x7ff0c00c4f20 2023-10-10T18:56:02.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.692+0000 7ff0c54bd700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0a4066f60 0x7ff0a4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:02.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.692+0000 7ff0c54bd700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0a4066f60 0x7ff0a4069410 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00aff60 tx=0x7ff0b0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:02.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.696+0000 7ff0b5ffb700 1 -- 172.21.15.175:0/951227956 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff0bc0524e0 con 0x7ff0c00c4f20 2023-10-10T18:56:02.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.924+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0c006bc30 con 0x7ff0c00c4f20 2023-10-10T18:56:02.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.925+0000 7ff0b5ffb700 1 -- 172.21.15.175:0/951227956 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5433 (secure 0 0 0) 0x7ff0bc055b00 con 0x7ff0c00c4f20 2023-10-10T18:56:02.925 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:02.925 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:00.704700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:02.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0a4066f60 msgr2=0x7ff0a4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:02.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0a4066f60 0x7ff0a4069410 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00aff60 tx=0x7ff0b0009040 comp rx=0 tx=0).stop 2023-10-10T18:56:02.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 msgr2=0x7ff0c0166ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:02.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c0166ca0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c170 tx=0x7ff0bc00b040 comp rx=0 tx=0).stop 2023-10-10T18:56:02.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 shutdown_connections 2023-10-10T18:56:02.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0a4066f60 0x7ff0a4069410 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0c01577a0 0x7ff0c0165530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0c00c4f20 0x7ff0c0166ca0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 --2- 172.21.15.175:0/951227956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff0c01573d0 0x7ff0c01671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:02.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 >> 172.21.15.175:0/951227956 conn(0x7ff0c006dc70 msgr2=0x7ff0c014c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:02.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 shutdown_connections 2023-10-10T18:56:02.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:02.927+0000 7ff0c7f22700 1 -- 172.21.15.175:0/951227956 wait complete. 2023-10-10T18:56:02.930 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 87 2023-10-10T18:56:02.939 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=1, replacing gid=4251 2023-10-10T18:56:02.939 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.e 2023-10-10T18:56:02.939 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-10T18:56:02.940 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-10T18:56:02.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-10T18:56:02.942 INFO:tasks.ceph.mds.e:Started 2023-10-10T18:56:02.943 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:03.014 INFO:tasks.ceph.mds.e.smithi175.stdout:starting mds.e at 2023-10-10T18:56:03.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.059+0000 7f9282242700 1 Processor -- start 2023-10-10T18:56:03.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.059+0000 7f9282242700 1 -- start start 2023-10-10T18:56:03.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f9282242700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f927c0b2390 0x7f927c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:03.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f9282242700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:03.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f9282242700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c1574b0 con 0x7f927c0b1fc0 2023-10-10T18:56:03.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f9282242700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927c150770 con 0x7f927c0b2390 2023-10-10T18:56:03.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f9282242700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c0afad0 con 0x7f927c0aff60 2023-10-10T18:56:03.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f927bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:03.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f9273fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f927c0b2390 0x7f927c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:03.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f927bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c0b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:44058/0 (socket says 172.21.15.175:44058) 2023-10-10T18:56:03.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f927bfff700 1 -- 172.21.15.175:0/2305163388 learned_addr learned my addr 172.21.15.175:0/2305163388 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:03.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.060+0000 7f927bfff700 1 -- 172.21.15.175:0/2305163388 >> v2:172.21.15.191:3300/0 conn(0x7f927c0b2390 msgr2=0x7f927c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:03.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f927bfff700 1 --2- 172.21.15.175:0/2305163388 >> v2:172.21.15.191:3300/0 conn(0x7f927c0b2390 0x7f927c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f927bfff700 1 -- 172.21.15.175:0/2305163388 >> v1:172.21.15.191:6789/0 conn(0x7f927c0b1fc0 legacy=0x7f927c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:03.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f927bfff700 1 -- 172.21.15.175:0/2305163388 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c0af4c0 con 0x7f927c0aff60 2023-10-10T18:56:03.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f9273fff700 1 --2- 172.21.15.175:0/2305163388 >> v2:172.21.15.191:3300/0 conn(0x7f927c0b2390 0x7f927c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:56:03.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f927bfff700 1 --2- 172.21.15.175:0/2305163388 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c0b0400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9274009f20 tx=0x7f9274009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2758bcf1c891143 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:03.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f927affd700 1 -- 172.21.15.175:0/2305163388 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400e070 con 0x7f927c0aff60 2023-10-10T18:56:03.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f927affd700 1 -- 172.21.15.175:0/2305163388 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927400ba60 con 0x7f927c0aff60 2023-10-10T18:56:03.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f927affd700 1 -- 172.21.15.175:0/2305163388 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740092e0 con 0x7f927c0aff60 2023-10-10T18:56:03.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f9282242700 1 -- 172.21.15.175:0/2305163388 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 msgr2=0x7f927c0b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:03.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.061+0000 7f9282242700 1 --2- 172.21.15.175:0/2305163388 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c0b0400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9274009f20 tx=0x7f9274009840 comp rx=0 tx=0).stop 2023-10-10T18:56:03.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.062+0000 7f9282242700 1 -- 172.21.15.175:0/2305163388 shutdown_connections 2023-10-10T18:56:03.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.062+0000 7f9282242700 1 --2- 172.21.15.175:0/2305163388 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c0b0400 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.062+0000 7f9282242700 1 --2- 172.21.15.175:0/2305163388 >> v2:172.21.15.191:3300/0 conn(0x7f927c0b2390 0x7f927c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.062+0000 7f9282242700 1 -- 172.21.15.175:0/2305163388 >> 172.21.15.175:0/2305163388 conn(0x7f927c063230 msgr2=0x7f927c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:03.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.062+0000 7f9282242700 1 -- 172.21.15.175:0/2305163388 shutdown_connections 2023-10-10T18:56:03.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.062+0000 7f9282242700 1 -- 172.21.15.175:0/2305163388 wait complete. 2023-10-10T18:56:03.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9282242700 1 Processor -- start 2023-10-10T18:56:03.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9282242700 1 -- start start 2023-10-10T18:56:03.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9282242700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c1532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:03.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9282242700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f927c0b1fc0 0x7f927c167060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:03.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9282242700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f927c0b2390 0x7f927c1675a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:03.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9282242700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c14f110 con 0x7f927c0b1fc0 2023-10-10T18:56:03.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9282242700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c14fc20 con 0x7f927c0b2390 2023-10-10T18:56:03.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9282242700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c1526c0 con 0x7f927c0aff60 2023-10-10T18:56:03.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9273fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f927c0b1fc0 0x7f927c167060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:03.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.063+0000 7f9273fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f927c0b1fc0 0x7f927c167060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56300/0 (socket says 172.21.15.175:56300) 2023-10-10T18:56:03.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9273fff700 1 -- 172.21.15.175:0/2507948580 learned_addr learned my addr 172.21.15.175:0/2507948580 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:03.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f927bfff700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f927c0b2390 0x7f927c1675a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:03.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f927b7fe700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c1532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:03.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9273fff700 1 -- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 msgr2=0x7f927c1532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:03.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9273fff700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c1532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9273fff700 1 -- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f927c0b2390 msgr2=0x7f927c1675a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:03.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9273fff700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f927c0b2390 0x7f927c1675a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9273fff700 1 -- 172.21.15.175:0/2507948580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9264009420 con 0x7f927c0b1fc0 2023-10-10T18:56:03.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9273fff700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f927c0b1fc0 0x7f927c167060 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f926400ec80 tx=0x7f92640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:03.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f92737fe700 1 -- 172.21.15.175:0/2507948580 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926400c850 con 0x7f927c0b1fc0 2023-10-10T18:56:03.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f92737fe700 1 -- 172.21.15.175:0/2507948580 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926400c9b0 con 0x7f927c0b1fc0 2023-10-10T18:56:03.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f92737fe700 1 -- 172.21.15.175:0/2507948580 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926400cd30 con 0x7f927c0b1fc0 2023-10-10T18:56:03.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9274009ae0 con 0x7f927c0b1fc0 2023-10-10T18:56:03.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.064+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c14f5d0 con 0x7f927c0b1fc0 2023-10-10T18:56:03.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.065+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927c067570 con 0x7f927c0b1fc0 2023-10-10T18:56:03.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.065+0000 7f92737fe700 1 -- 172.21.15.175:0/2507948580 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9264013680 con 0x7f927c0b1fc0 2023-10-10T18:56:03.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.066+0000 7f92737fe700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f925c066f30 0x7f925c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:03.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.066+0000 7f92737fe700 1 -- 172.21.15.175:0/2507948580 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f9264084940 con 0x7f927c0b1fc0 2023-10-10T18:56:03.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.066+0000 7f927b7fe700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f925c066f30 0x7f925c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:03.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.066+0000 7f927b7fe700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f925c066f30 0x7f925c0693e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f926c003a80 tx=0x7f926c007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:03.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.070+0000 7f92737fe700 1 -- 172.21.15.175:0/2507948580 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9264055420 con 0x7f927c0b1fc0 2023-10-10T18:56:03.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.294+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f927c068da0 con 0x7f927c0b1fc0 2023-10-10T18:56:03.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.295+0000 7f92737fe700 1 -- 172.21.15.175:0/2507948580 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5433 (secure 0 0 0) 0x7f9264054fb0 con 0x7f927c0b1fc0 2023-10-10T18:56:03.295 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:03.295 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:00.704700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:03.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f925c066f30 msgr2=0x7f925c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:03.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f925c066f30 0x7f925c0693e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f926c003a80 tx=0x7f926c007ec0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f927c0b1fc0 msgr2=0x7f927c167060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:03.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f927c0b1fc0 0x7f927c167060 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f926400ec80 tx=0x7f92640091a0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 shutdown_connections 2023-10-10T18:56:03.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f925c066f30 0x7f925c0693e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f927c0b2390 0x7f927c1675a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f927c0aff60 0x7f927c1532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 --2- 172.21.15.175:0/2507948580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f927c0b1fc0 0x7f927c167060 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:03.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 >> 172.21.15.175:0/2507948580 conn(0x7f927c063230 msgr2=0x7f927c063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:03.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 shutdown_connections 2023-10-10T18:56:03.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:03.297+0000 7f9282242700 1 -- 172.21.15.175:0/2507948580 wait complete. 2023-10-10T18:56:03.300 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 87 2023-10-10T18:56:03.309 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-10T18:56:05.310 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:05.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.435+0000 7f37eaac1700 1 Processor -- start 2023-10-10T18:56:05.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.436+0000 7f37eaac1700 1 -- start start 2023-10-10T18:56:05.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.436+0000 7f37eaac1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37e406a650 0x7f37e40c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:05.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.436+0000 7f37eaac1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e4068c90 con 0x7f37e40c5630 2023-10-10T18:56:05.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.436+0000 7f37eaac1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e4069120 con 0x7f37e4159a70 2023-10-10T18:56:05.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.436+0000 7f37eaac1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e40c5b90 con 0x7f37e406a650 2023-10-10T18:56:05.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.436+0000 7f37dbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37e406a650 0x7f37e40c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:05.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.436+0000 7f37e905e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f37e40c5630 0x7f37e4156470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58596/0 (socket says 172.21.15.175:58596) 2023-10-10T18:56:05.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.436+0000 7f37e905e700 1 -- 172.21.15.175:0/2326569026 learned_addr learned my addr 172.21.15.175:0/2326569026 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:56:05.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37dbfff700 1 -- 172.21.15.175:0/2326569026 >> v1:172.21.15.191:6789/0 conn(0x7f37e4159a70 legacy=0x7f37e4159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:05.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37dbfff700 1 -- 172.21.15.175:0/2326569026 >> v1:172.21.15.175:6789/0 conn(0x7f37e40c5630 legacy=0x7f37e4156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:05.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37dbfff700 1 -- 172.21.15.175:0/2326569026 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40639e0 con 0x7f37e406a650 2023-10-10T18:56:05.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37db7fe700 1 -- 172.21.15.175:0/2326569026 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384731428 0 0) 0x7f37e4068c90 con 0x7f37e40c5630 2023-10-10T18:56:05.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37dbfff700 1 --2- 172.21.15.175:0/2326569026 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c7060 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f37d40098e0 tx=0x7f37d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469eecc33d198867 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:05.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37db7fe700 1 -- 172.21.15.175:0/2326569026 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4009e80 con 0x7f37e406a650 2023-10-10T18:56:05.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37db7fe700 1 -- 172.21.15.175:0/2326569026 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d400b8f0 con 0x7f37e406a650 2023-10-10T18:56:05.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37db7fe700 1 -- 172.21.15.175:0/2326569026 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d400bc70 con 0x7f37e406a650 2023-10-10T18:56:05.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37eaac1700 1 -- 172.21.15.175:0/2326569026 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 msgr2=0x7f37e40c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:05.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.437+0000 7f37eaac1700 1 --2- 172.21.15.175:0/2326569026 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c7060 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f37d40098e0 tx=0x7f37d40091a0 comp rx=0 tx=0).stop 2023-10-10T18:56:05.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.438+0000 7f37eaac1700 1 -- 172.21.15.175:0/2326569026 shutdown_connections 2023-10-10T18:56:05.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.438+0000 7f37eaac1700 1 --2- 172.21.15.175:0/2326569026 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c7060 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:05.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.438+0000 7f37eaac1700 1 -- 172.21.15.175:0/2326569026 >> 172.21.15.175:0/2326569026 conn(0x7f37e406dc80 msgr2=0x7f37e4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:05.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.438+0000 7f37eaac1700 1 -- 172.21.15.175:0/2326569026 shutdown_connections 2023-10-10T18:56:05.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.438+0000 7f37eaac1700 1 -- 172.21.15.175:0/2326569026 wait complete. 2023-10-10T18:56:05.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.439+0000 7f37eaac1700 1 Processor -- start 2023-10-10T18:56:05.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.439+0000 7f37eaac1700 1 -- start start 2023-10-10T18:56:05.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.439+0000 7f37eaac1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:05.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.439+0000 7f37eaac1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37e40c5630 0x7f37e4151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:05.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.439+0000 7f37eaac1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37e4159a70 0x7f37e4166f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:05.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.439+0000 7f37eaac1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40c3c40 con 0x7f37e406a650 2023-10-10T18:56:05.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.439+0000 7f37eaac1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40c48c0 con 0x7f37e40c5630 2023-10-10T18:56:05.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.439+0000 7f37eaac1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40c2d50 con 0x7f37e4159a70 2023-10-10T18:56:05.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:05.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34248/0 (socket says 172.21.15.175:34248) 2023-10-10T18:56:05.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 -- 172.21.15.175:0/2503181692 learned_addr learned my addr 172.21.15.175:0/2503181692 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:05.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37dbfff700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37e40c5630 0x7f37e4151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:05.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e905e700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37e4159a70 0x7f37e4166f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:05.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 -- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37e4159a70 msgr2=0x7f37e4166f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:05.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37e4159a70 0x7f37e4166f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:05.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 -- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37e40c5630 msgr2=0x7f37e4151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:05.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37e40c5630 0x7f37e4151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:05.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 -- 172.21.15.175:0/2503181692 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d4009420 con 0x7f37e406a650 2023-10-10T18:56:05.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e905e700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37e4159a70 0x7f37e4166f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:56:05.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37e885d700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c6ac0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f37e000c910 tx=0x7f37e0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:05.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37d8ff9700 1 -- 172.21.15.175:0/2503181692 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0012630 con 0x7f37e406a650 2023-10-10T18:56:05.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37d8ff9700 1 -- 172.21.15.175:0/2503181692 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e0012790 con 0x7f37e406a650 2023-10-10T18:56:05.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e40c2fc0 con 0x7f37e406a650 2023-10-10T18:56:05.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37d8ff9700 1 -- 172.21.15.175:0/2503181692 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0012b50 con 0x7f37e406a650 2023-10-10T18:56:05.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.440+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e40c3580 con 0x7f37e406a650 2023-10-10T18:56:05.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.441+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e4067570 con 0x7f37e406a650 2023-10-10T18:56:05.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.442+0000 7f37d8ff9700 1 -- 172.21.15.175:0/2503181692 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f37e0003030 con 0x7f37e406a650 2023-10-10T18:56:05.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.442+0000 7f37d8ff9700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37d0066f60 0x7f37d0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:05.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.442+0000 7f37d8ff9700 1 -- 172.21.15.175:0/2503181692 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f37e0084730 con 0x7f37e406a650 2023-10-10T18:56:05.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.442+0000 7f37dbfff700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37d0066f60 0x7f37d0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:05.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.443+0000 7f37dbfff700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37d0066f60 0x7f37d0069410 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f37e406aa60 tx=0x7f37d4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:05.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.446+0000 7f37d8ff9700 1 -- 172.21.15.175:0/2503181692 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37e0051780 con 0x7f37e406a650 2023-10-10T18:56:05.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.669+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37e4063cc0 con 0x7f37e406a650 2023-10-10T18:56:05.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.669+0000 7f37d8ff9700 1 -- 172.21.15.175:0/2503181692 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6192 (secure 0 0 0) 0x7f37e0054da0 con 0x7f37e406a650 2023-10-10T18:56:05.669 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:05.670 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:00.704700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:05.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37d0066f60 msgr2=0x7f37d0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:05.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37d0066f60 0x7f37d0069410 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f37e406aa60 tx=0x7f37d4017000 comp rx=0 tx=0).stop 2023-10-10T18:56:05.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 msgr2=0x7f37e40c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:05.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c6ac0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f37e000c910 tx=0x7f37e0001710 comp rx=0 tx=0).stop 2023-10-10T18:56:05.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 shutdown_connections 2023-10-10T18:56:05.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37d0066f60 0x7f37d0069410 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:05.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37e4159a70 0x7f37e4166f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:05.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37e406a650 0x7f37e40c6ac0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:05.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 --2- 172.21.15.175:0/2503181692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37e40c5630 0x7f37e4151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:05.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 >> 172.21.15.175:0/2503181692 conn(0x7f37e406dc80 msgr2=0x7f37e40c9430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:05.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 shutdown_connections 2023-10-10T18:56:05.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:05.671+0000 7f37eaac1700 1 -- 172.21.15.175:0/2503181692 wait complete. 2023-10-10T18:56:05.674 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 88 2023-10-10T18:56:05.683 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-10T18:56:05.683 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-10-10T18:56:19.865 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:56:19.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:56:49.684 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:49.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.813+0000 7fb4fe831700 1 Processor -- start 2023-10-10T18:56:49.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.813+0000 7fb4fe831700 1 -- start start 2023-10-10T18:56:49.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.814+0000 7fb4fe831700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814c840 0x7fb4f806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:49.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.814+0000 7fb4fe831700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb4f814cc10 0x7fb4f80c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:49.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.814+0000 7fb4fe831700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f8155ca0 con 0x7fb4f80c59d0 2023-10-10T18:56:49.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.814+0000 7fb4fe831700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f8156b00 con 0x7fb4f814cc10 2023-10-10T18:56:49.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.814+0000 7fb4fe831700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f806d700 con 0x7fb4f814c840 2023-10-10T18:56:49.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.814+0000 7fb4f77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb4f814cc10 0x7fb4f80c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:49.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.814+0000 7fb4f77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb4f814cc10 0x7fb4f80c4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37962/0 (socket says 172.21.15.175:37962) 2023-10-10T18:56:49.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.814+0000 7fb4f77fe700 1 -- 172.21.15.175:0/1883245866 learned_addr learned my addr 172.21.15.175:0/1883245866 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:49.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f7fff700 1 --2- 172.21.15.175:0/1883245866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814c840 0x7fb4f806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:49.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f77fe700 1 -- 172.21.15.175:0/1883245866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814c840 msgr2=0x7fb4f806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:49.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f77fe700 1 --2- 172.21.15.175:0/1883245866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814c840 0x7fb4f806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:49.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f77fe700 1 -- 172.21.15.175:0/1883245866 >> v1:172.21.15.191:6789/0 conn(0x7fb4f80c59d0 legacy=0x7fb4f8152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:49.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f77fe700 1 -- 172.21.15.175:0/1883245866 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f8155ef0 con 0x7fb4f814cc10 2023-10-10T18:56:49.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f77fe700 1 --2- 172.21.15.175:0/1883245866 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814cc10 0x7fb4f80c4b90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00bc50 tx=0x7fb4ec00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2bced1e97f68ef8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:49.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f6ffd700 1 -- 172.21.15.175:0/1883245866 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec00de90 con 0x7fb4f814cc10 2023-10-10T18:56:49.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f6ffd700 1 -- 172.21.15.175:0/1883245866 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec00be60 con 0x7fb4f814cc10 2023-10-10T18:56:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.815+0000 7fb4f6ffd700 1 -- 172.21.15.175:0/1883245866 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec0196b0 con 0x7fb4f814cc10 2023-10-10T18:56:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.816+0000 7fb4fe831700 1 -- 172.21.15.175:0/1883245866 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814cc10 msgr2=0x7fb4f80c4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.816+0000 7fb4fe831700 1 --2- 172.21.15.175:0/1883245866 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814cc10 0x7fb4f80c4b90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00bc50 tx=0x7fb4ec00b4f0 comp rx=0 tx=0).stop 2023-10-10T18:56:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.816+0000 7fb4fe831700 1 -- 172.21.15.175:0/1883245866 shutdown_connections 2023-10-10T18:56:49.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.816+0000 7fb4fe831700 1 --2- 172.21.15.175:0/1883245866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814c840 0x7fb4f806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:49.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.816+0000 7fb4fe831700 1 --2- 172.21.15.175:0/1883245866 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814cc10 0x7fb4f80c4b90 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:49.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.816+0000 7fb4fe831700 1 -- 172.21.15.175:0/1883245866 >> 172.21.15.175:0/1883245866 conn(0x7fb4f8068280 msgr2=0x7fb4f806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:49.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.816+0000 7fb4fe831700 1 -- 172.21.15.175:0/1883245866 shutdown_connections 2023-10-10T18:56:49.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.816+0000 7fb4fe831700 1 -- 172.21.15.175:0/1883245866 wait complete. 2023-10-10T18:56:49.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.817+0000 7fb4fe831700 1 Processor -- start 2023-10-10T18:56:49.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.817+0000 7fb4fe831700 1 -- start start 2023-10-10T18:56:49.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fe831700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4f80c59d0 0x7fb4f80c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:49.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fe831700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814c840 0x7fb4f8147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:49.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fe831700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814cc10 0x7fb4f8150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:49.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fe831700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f81559d0 con 0x7fb4f814c840 2023-10-10T18:56:49.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fe831700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f81499e0 con 0x7fb4f80c59d0 2023-10-10T18:56:49.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fe831700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f814aae0 con 0x7fb4f814cc10 2023-10-10T18:56:49.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814c840 0x7fb4f8147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:49.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814c840 0x7fb4f8147ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37974/0 (socket says 172.21.15.175:37974) 2023-10-10T18:56:49.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 -- 172.21.15.175:0/3299111732 learned_addr learned my addr 172.21.15.175:0/3299111732 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:49.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f7fff700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4f80c59d0 0x7fb4f80c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:49.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fcdce700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814cc10 0x7fb4f8150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:49.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 -- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814cc10 msgr2=0x7fb4f8150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:49.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814cc10 0x7fb4f8150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:49.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 -- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4f80c59d0 msgr2=0x7fb4f80c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:49.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4f80c59d0 0x7fb4f80c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:49.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 -- 172.21.15.175:0/3299111732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec00b790 con 0x7fb4f814c840 2023-10-10T18:56:49.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fcdce700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814cc10 0x7fb4f8150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:56:49.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4f77fe700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814c840 0x7fb4f8147ac0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00b760 tx=0x7fb4ec01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:49.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4dffff700 1 -- 172.21.15.175:0/3299111732 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec0105c0 con 0x7fb4f814c840 2023-10-10T18:56:49.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4dffff700 1 -- 172.21.15.175:0/3299111732 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec010720 con 0x7fb4f814c840 2023-10-10T18:56:49.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4dffff700 1 -- 172.21.15.175:0/3299111732 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec010ac0 con 0x7fb4f814c840 2023-10-10T18:56:49.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.818+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f81556f0 con 0x7fb4f814c840 2023-10-10T18:56:49.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.819+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f80c8230 con 0x7fb4f814c840 2023-10-10T18:56:49.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.819+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f80c8060 con 0x7fb4f814c840 2023-10-10T18:56:49.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.820+0000 7fb4dffff700 1 -- 172.21.15.175:0/3299111732 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb4ec022070 con 0x7fb4f814c840 2023-10-10T18:56:49.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.820+0000 7fb4dffff700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb4e4066f30 0x7fb4e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:49.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.820+0000 7fb4dffff700 1 -- 172.21.15.175:0/3299111732 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fb4ec014070 con 0x7fb4f814c840 2023-10-10T18:56:49.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.820+0000 7fb4f7fff700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb4e4066f30 0x7fb4e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:49.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.821+0000 7fb4f7fff700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb4e4066f30 0x7fb4e40693e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8006aa0 tx=0x7fb4e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:49.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:49.824+0000 7fb4dffff700 1 -- 172.21.15.175:0/3299111732 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4ec055300 con 0x7fb4f814c840 2023-10-10T18:56:49.934 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:56:49.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:56:50.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.055+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4f80c74c0 con 0x7fb4f814c840 2023-10-10T18:56:50.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.055+0000 7fb4dffff700 1 -- 172.21.15.175:0/3299111732 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7fb4ec058920 con 0x7fb4f814c840 2023-10-10T18:56:50.056 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:50.056 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:50.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb4e4066f30 msgr2=0x7fb4e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:50.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb4e4066f30 0x7fb4e40693e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8006aa0 tx=0x7fb4e8008040 comp rx=0 tx=0).stop 2023-10-10T18:56:50.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814c840 msgr2=0x7fb4f8147ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:50.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814c840 0x7fb4f8147ac0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00b760 tx=0x7fb4ec01b000 comp rx=0 tx=0).stop 2023-10-10T18:56:50.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 shutdown_connections 2023-10-10T18:56:50.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb4e4066f30 0x7fb4e40693e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4f814cc10 0x7fb4f8150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4f80c59d0 0x7fb4f80c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 --2- 172.21.15.175:0/3299111732 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4f814c840 0x7fb4f8147ac0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.057+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 >> 172.21.15.175:0/3299111732 conn(0x7fb4f8068280 msgr2=0x7fb4f80c0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:50.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.058+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 shutdown_connections 2023-10-10T18:56:50.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.058+0000 7fb4fe831700 1 -- 172.21.15.175:0/3299111732 wait complete. 2023-10-10T18:56:50.060 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:56:50.069 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-10T18:56:50.069 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-10T18:56:50.070 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T18:56:50.081 INFO:tasks.ceph.mds.d:Stopped 2023-10-10T18:56:50.082 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:50.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.197+0000 7f2cfa814700 1 Processor -- start 2023-10-10T18:56:50.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.197+0000 7f2cfa814700 1 -- start start 2023-10-10T18:56:50.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa814700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf408a100 0x7f2cf4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:50.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa814700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2cf40cd040 0x7f2cf40684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:50.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa814700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf4067fa0 con 0x7f2cf414fd90 2023-10-10T18:56:50.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa814700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf40c5600 con 0x7f2cf40cd040 2023-10-10T18:56:50.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa814700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4156b70 con 0x7f2cf408a100 2023-10-10T18:56:50.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cf9812700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2cf414fd90 0x7f2cf40c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40032/0 (socket says 172.21.15.175:40032) 2023-10-10T18:56:50.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cf9812700 1 -- 172.21.15.175:0/590670507 learned_addr learned my addr 172.21.15.175:0/590670507 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:56:50.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa013700 1 --2- 172.21.15.175:0/590670507 >> v2:172.21.15.175:3300/0 conn(0x7f2cf40cd040 0x7f2cf40684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:50.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cf9011700 1 --2- 172.21.15.175:0/590670507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf408a100 0x7f2cf4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:50.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cf8810700 1 -- 172.21.15.175:0/590670507 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848825584 0 0) 0x7f2cf4067fa0 con 0x7f2cf414fd90 2023-10-10T18:56:50.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cf8810700 1 -- 172.21.15.175:0/590670507 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce0003570 con 0x7f2cf414fd90 2023-10-10T18:56:50.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa013700 1 -- 172.21.15.175:0/590670507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf408a100 msgr2=0x7f2cf4159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:50.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa013700 1 --2- 172.21.15.175:0/590670507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf408a100 0x7f2cf4159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa013700 1 -- 172.21.15.175:0/590670507 >> v1:172.21.15.175:6789/0 conn(0x7f2cf414fd90 legacy=0x7f2cf40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:50.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa013700 1 -- 172.21.15.175:0/590670507 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf4150160 con 0x7f2cf40cd040 2023-10-10T18:56:50.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cfa013700 1 --2- 172.21.15.175:0/590670507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf40cd040 0x7f2cf40684f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2cf00098e0 tx=0x7f2cf00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55e9e33f851b1f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:50.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cf8810700 1 -- 172.21.15.175:0/590670507 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf000b640 con 0x7f2cf40cd040 2023-10-10T18:56:50.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cf8810700 1 -- 172.21.15.175:0/590670507 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf000b7a0 con 0x7f2cf40cd040 2023-10-10T18:56:50.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.198+0000 7f2cf8810700 1 -- 172.21.15.175:0/590670507 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf000bb40 con 0x7f2cf40cd040 2023-10-10T18:56:50.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.199+0000 7f2cfa814700 1 -- 172.21.15.175:0/590670507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf40cd040 msgr2=0x7f2cf40684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:50.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.199+0000 7f2cfa814700 1 --2- 172.21.15.175:0/590670507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf40cd040 0x7f2cf40684f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2cf00098e0 tx=0x7f2cf00091a0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.199+0000 7f2cfa814700 1 -- 172.21.15.175:0/590670507 shutdown_connections 2023-10-10T18:56:50.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.199+0000 7f2cfa814700 1 --2- 172.21.15.175:0/590670507 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf40cd040 0x7f2cf40684f0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.199+0000 7f2cfa814700 1 --2- 172.21.15.175:0/590670507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf408a100 0x7f2cf4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.199+0000 7f2cfa814700 1 -- 172.21.15.175:0/590670507 >> 172.21.15.175:0/590670507 conn(0x7f2cf4063d00 msgr2=0x7f2cf40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:50.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.199+0000 7f2cfa814700 1 -- 172.21.15.175:0/590670507 shutdown_connections 2023-10-10T18:56:50.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.199+0000 7f2cfa814700 1 -- 172.21.15.175:0/590670507 wait complete. 2023-10-10T18:56:50.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.200+0000 7f2cfa814700 1 Processor -- start 2023-10-10T18:56:50.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.200+0000 7f2cfa814700 1 -- start start 2023-10-10T18:56:50.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa814700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf408a100 0x7f2cf4153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:50.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa814700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf40cd040 0x7f2cf40c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:50.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa814700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf414fd90 0x7f2cf4165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:50.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa814700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4151a10 con 0x7f2cf414fd90 2023-10-10T18:56:50.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa814700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4165b10 con 0x7f2cf408a100 2023-10-10T18:56:50.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa814700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40c8bc0 con 0x7f2cf40cd040 2023-10-10T18:56:50.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf414fd90 0x7f2cf4165fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:50.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf414fd90 0x7f2cf4165fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37988/0 (socket says 172.21.15.175:37988) 2023-10-10T18:56:50.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 -- 172.21.15.175:0/4289025878 learned_addr learned my addr 172.21.15.175:0/4289025878 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:50.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cf9812700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf408a100 0x7f2cf4153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:50.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cf9011700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf40cd040 0x7f2cf40c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:50.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 -- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf40cd040 msgr2=0x7f2cf40c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:50.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf40cd040 0x7f2cf40c7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 -- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf408a100 msgr2=0x7f2cf4153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:50.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf408a100 0x7f2cf4153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 -- 172.21.15.175:0/4289025878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf0009420 con 0x7f2cf414fd90 2023-10-10T18:56:50.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa013700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf414fd90 0x7f2cf4165fa0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4064c20 tx=0x7f2cf0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:50.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2ce9ffb700 1 -- 172.21.15.175:0/4289025878 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf000b640 con 0x7f2cf414fd90 2023-10-10T18:56:50.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2ce9ffb700 1 -- 172.21.15.175:0/4289025878 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf0024440 con 0x7f2cf414fd90 2023-10-10T18:56:50.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.201+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf40c8ea0 con 0x7f2cf414fd90 2023-10-10T18:56:50.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.202+0000 7f2ce9ffb700 1 -- 172.21.15.175:0/4289025878 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf00247e0 con 0x7f2cf414fd90 2023-10-10T18:56:50.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.202+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf40c9460 con 0x7f2cf414fd90 2023-10-10T18:56:50.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.203+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf4067570 con 0x7f2cf414fd90 2023-10-10T18:56:50.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.203+0000 7f2ce9ffb700 1 -- 172.21.15.175:0/4289025878 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2cf002d440 con 0x7f2cf414fd90 2023-10-10T18:56:50.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.203+0000 7f2ce9ffb700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2cd8066f30 0x7f2cd80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:50.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.203+0000 7f2ce9ffb700 1 -- 172.21.15.175:0/4289025878 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f2cf0086050 con 0x7f2cf414fd90 2023-10-10T18:56:50.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.203+0000 7f2cf9812700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2cd8066f30 0x7f2cd80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:50.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.204+0000 7f2cf9812700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2cd8066f30 0x7f2cd80693e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2cec005c50 tx=0x7f2cec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:50.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.207+0000 7f2ce9ffb700 1 -- 172.21.15.175:0/4289025878 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2cf00530a0 con 0x7f2cf414fd90 2023-10-10T18:56:50.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.431+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cf4155f80 con 0x7f2cf414fd90 2023-10-10T18:56:50.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.432+0000 7f2ce9ffb700 1 -- 172.21.15.175:0/4289025878 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7f2cf00566c0 con 0x7f2cf414fd90 2023-10-10T18:56:50.432 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:50.432 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:50.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2cd8066f30 msgr2=0x7f2cd80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:50.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2cd8066f30 0x7f2cd80693e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2cec005c50 tx=0x7f2cec009040 comp rx=0 tx=0).stop 2023-10-10T18:56:50.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf414fd90 msgr2=0x7f2cf4165fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:50.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf414fd90 0x7f2cf4165fa0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4064c20 tx=0x7f2cf0017000 comp rx=0 tx=0).stop 2023-10-10T18:56:50.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 shutdown_connections 2023-10-10T18:56:50.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2cd8066f30 0x7f2cd80693e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2cf414fd90 0x7f2cf4165fa0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf408a100 0x7f2cf4153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 --2- 172.21.15.175:0/4289025878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf40cd040 0x7f2cf40c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:50.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 >> 172.21.15.175:0/4289025878 conn(0x7f2cf4063d00 msgr2=0x7f2cf415a540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:50.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 shutdown_connections 2023-10-10T18:56:50.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:50.434+0000 7f2cfa814700 1 -- 172.21.15.175:0/4289025878 wait complete. 2023-10-10T18:56:50.436 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:56:50.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T18:56:52.447 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:52.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.566+0000 7fcc824ad700 1 Processor -- start 2023-10-10T18:56:52.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.566+0000 7fcc824ad700 1 -- start start 2023-10-10T18:56:52.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.567+0000 7fcc824ad700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcc7c0b2390 0x7fcc7c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:52.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.567+0000 7fcc824ad700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c1574b0 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.567+0000 7fcc824ad700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0afad0 con 0x7fcc7c0aff60 2023-10-10T18:56:52.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.567+0000 7fcc824ad700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c150770 con 0x7fcc7c0b2390 2023-10-10T18:56:52.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.567+0000 7fcc7b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcc7c0b2390 0x7fcc7c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:52.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.567+0000 7fcc7bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcc7c0b1fc0 0x7fcc7c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40040/0 (socket says 172.21.15.175:40040) 2023-10-10T18:56:52.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.567+0000 7fcc7bfff700 1 -- 172.21.15.175:0/1812249 learned_addr learned my addr 172.21.15.175:0/1812249 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:56:52.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.567+0000 7fcc7b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcc7c0b2390 0x7fcc7c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37992/0 (socket says 172.21.15.175:37992) 2023-10-10T18:56:52.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc7b7fe700 1 -- 172.21.15.175:0/1812249 >> v1:172.21.15.191:6789/0 conn(0x7fcc7c0aff60 legacy=0x7fcc7c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:52.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc7b7fe700 1 -- 172.21.15.175:0/1812249 >> v1:172.21.15.175:6789/0 conn(0x7fcc7c0b1fc0 legacy=0x7fcc7c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:52.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc7b7fe700 1 -- 172.21.15.175:0/1812249 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c0af4c0 con 0x7fcc7c0b2390 2023-10-10T18:56:52.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc7affd700 1 -- 172.21.15.175:0/1812249 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081363552 0 0) 0x7fcc7c1574b0 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc7b7fe700 1 --2- 172.21.15.175:0/1812249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b2390 0x7fcc7c155fb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc700098e0 tx=0x7fcc700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f235cca38b489c58 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:52.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc7affd700 1 -- 172.21.15.175:0/1812249 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70009e80 con 0x7fcc7c0b2390 2023-10-10T18:56:52.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc7affd700 1 -- 172.21.15.175:0/1812249 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7000b8f0 con 0x7fcc7c0b2390 2023-10-10T18:56:52.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc7affd700 1 -- 172.21.15.175:0/1812249 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000bc70 con 0x7fcc7c0b2390 2023-10-10T18:56:52.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc824ad700 1 -- 172.21.15.175:0/1812249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b2390 msgr2=0x7fcc7c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:52.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc824ad700 1 --2- 172.21.15.175:0/1812249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b2390 0x7fcc7c155fb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc700098e0 tx=0x7fcc700091a0 comp rx=0 tx=0).stop 2023-10-10T18:56:52.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc824ad700 1 -- 172.21.15.175:0/1812249 shutdown_connections 2023-10-10T18:56:52.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc824ad700 1 --2- 172.21.15.175:0/1812249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b2390 0x7fcc7c155fb0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:52.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.568+0000 7fcc824ad700 1 -- 172.21.15.175:0/1812249 >> 172.21.15.175:0/1812249 conn(0x7fcc7c063230 msgr2=0x7fcc7c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:52.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.569+0000 7fcc824ad700 1 -- 172.21.15.175:0/1812249 shutdown_connections 2023-10-10T18:56:52.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.569+0000 7fcc824ad700 1 -- 172.21.15.175:0/1812249 wait complete. 2023-10-10T18:56:52.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.570+0000 7fcc824ad700 1 Processor -- start 2023-10-10T18:56:52.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.570+0000 7fcc824ad700 1 -- start start 2023-10-10T18:56:52.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.570+0000 7fcc824ad700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc7c0aff60 0x7fcc7c169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:52.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.570+0000 7fcc824ad700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b1fc0 0x7fcc7c14c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:52.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.570+0000 7fcc824ad700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc7c0b2390 0x7fcc7c14c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:52.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.570+0000 7fcc824ad700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0cc860 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.570+0000 7fcc824ad700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c147ad0 con 0x7fcc7c0b2390 2023-10-10T18:56:52.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.570+0000 7fcc824ad700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c155cb0 con 0x7fcc7c0aff60 2023-10-10T18:56:52.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b1fc0 0x7fcc7c14c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:52.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b1fc0 0x7fcc7c14c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37998/0 (socket says 172.21.15.175:37998) 2023-10-10T18:56:52.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 -- 172.21.15.175:0/2553561496 learned_addr learned my addr 172.21.15.175:0/2553561496 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:52.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc80a4a700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc7c0b2390 0x7fcc7c14c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:52.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7bfff700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc7c0aff60 0x7fcc7c169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:52.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 -- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc7c0aff60 msgr2=0x7fcc7c169960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:52.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc7c0aff60 0x7fcc7c169960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:52.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 -- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc7c0b2390 msgr2=0x7fcc7c14c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:52.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc7c0b2390 0x7fcc7c14c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:52.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 -- 172.21.15.175:0/2553561496 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc70009420 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc80a4a700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc7c0b2390 0x7fcc7c14c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:56:52.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc7b7fe700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b1fc0 0x7fcc7c14c450 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c06fd90 tx=0x7fcc70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:52.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc63fff700 1 -- 172.21.15.175:0/2553561496 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000bc30 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc63fff700 1 -- 172.21.15.175:0/2553561496 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7001a630 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c0cac90 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc63fff700 1 -- 172.21.15.175:0/2553561496 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7001a9d0 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.571+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c169160 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.572+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c067570 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.572+0000 7fcc63fff700 1 -- 172.21.15.175:0/2553561496 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcc7002d430 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.573+0000 7fcc63fff700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc68066f60 0x7fcc68069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:52.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.573+0000 7fcc63fff700 1 -- 172.21.15.175:0/2553561496 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fcc700862e0 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.573+0000 7fcc7bfff700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc68066f60 0x7fcc68069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:52.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.573+0000 7fcc7bfff700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc68066f60 0x7fcc68069410 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c000c00 tx=0x7fcc6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:52.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.577+0000 7fcc63fff700 1 -- 172.21.15.175:0/2553561496 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc70053330 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.805+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc7c068da0 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.806+0000 7fcc63fff700 1 -- 172.21.15.175:0/2553561496 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7fcc70056950 con 0x7fcc7c0b1fc0 2023-10-10T18:56:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:52.806 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:52.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc68066f60 msgr2=0x7fcc68069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:52.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc68066f60 0x7fcc68069410 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c000c00 tx=0x7fcc6c009040 comp rx=0 tx=0).stop 2023-10-10T18:56:52.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b1fc0 msgr2=0x7fcc7c14c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:52.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b1fc0 0x7fcc7c14c450 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c06fd90 tx=0x7fcc70017000 comp rx=0 tx=0).stop 2023-10-10T18:56:52.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 shutdown_connections 2023-10-10T18:56:52.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc68066f60 0x7fcc68069410 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:52.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc7c0b2390 0x7fcc7c14c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:52.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc7c0aff60 0x7fcc7c169960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:52.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 --2- 172.21.15.175:0/2553561496 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc7c0b1fc0 0x7fcc7c14c450 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:52.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 >> 172.21.15.175:0/2553561496 conn(0x7fcc7c063230 msgr2=0x7fcc7c0b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:52.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 shutdown_connections 2023-10-10T18:56:52.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:52.808+0000 7fcc824ad700 1 -- 172.21.15.175:0/2553561496 wait complete. 2023-10-10T18:56:52.810 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:56:52.819 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T18:56:54.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:54.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.944+0000 7f488cfdc700 1 Processor -- start 2023-10-10T18:56:54.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.944+0000 7f488cfdc700 1 -- start start 2023-10-10T18:56:54.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f488cfdc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4888155fb0 0x7f4888159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:54.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f488cfdc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48881574b0 con 0x7f48880b1fc0 2023-10-10T18:56:54.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f488cfdc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48880afad0 con 0x7f48880b2390 2023-10-10T18:56:54.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f488cfdc700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4888150770 con 0x7f4888155fb0 2023-10-10T18:56:54.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f488659c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f48880b1fc0 0x7f4888068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39334/0 (socket says 172.21.15.175:39334) 2023-10-10T18:56:54.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f488659c700 1 -- 172.21.15.175:0/3115742706 learned_addr learned my addr 172.21.15.175:0/3115742706 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:56:54.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4886d9d700 1 --2- 172.21.15.175:0/3115742706 >> v2:172.21.15.191:3300/0 conn(0x7f4888155fb0 0x7f4888159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:54.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102072520 0 0) 0x7f48881574b0 con 0x7f48880b1fc0 2023-10-10T18:56:54.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4874003570 con 0x7f48880b1fc0 2023-10-10T18:56:54.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2368719824 0 0) 0x7f4874003570 con 0x7f48880b1fc0 2023-10-10T18:56:54.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 >> v2:172.21.15.191:3300/0 conn(0x7f4888155fb0 msgr2=0x7f4888159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:54.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4885d9b700 1 --2- 172.21.15.175:0/3115742706 >> v2:172.21.15.191:3300/0 conn(0x7f4888155fb0 0x7f4888159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:54.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 >> v1:172.21.15.191:6789/0 conn(0x7f48880b2390 legacy=0x7f48880aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:54.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48880af4c0 con 0x7f48880b1fc0 2023-10-10T18:56:54.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.945+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f4878001470 con 0x7f48880b1fc0 2023-10-10T18:56:54.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 >> v1:172.21.15.175:6789/0 conn(0x7f48880b1fc0 legacy=0x7f4888068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:54.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4874003a60 0x7f4874003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:54.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4874004810 0x7f4874004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:54.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48740051c0 0x7f487400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:54.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888150770 con 0x7f48740051c0 2023-10-10T18:56:54.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487400c0d0 con 0x7f4874004810 2023-10-10T18:56:54.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487400c290 con 0x7f4874003a60 2023-10-10T18:56:54.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4886d9d700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48740051c0 0x7f487400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:54.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f487dd9b700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4874004810 0x7f4874004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:54.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f488659c700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4874003a60 0x7f4874003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:54.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4886d9d700 1 -- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4874003a60 msgr2=0x7f4874003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:54.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4886d9d700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4874003a60 0x7f4874003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:54.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4886d9d700 1 -- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4874004810 msgr2=0x7f4874004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:54.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4886d9d700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4874004810 0x7f4874004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:54.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4886d9d700 1 -- 172.21.15.175:0/3115742706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4874003570 con 0x7f48740051c0 2023-10-10T18:56:54.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f487dd9b700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4874004810 0x7f4874004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:56:54.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4886d9d700 1 -- 172.21.15.175:0/3115742706 reap_dead start 2023-10-10T18:56:54.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4886d9d700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48740051c0 0x7f487400ba60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f488000e210 tx=0x7f488000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef6f5827308867a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:54.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488000eac0 con 0x7f48740051c0 2023-10-10T18:56:54.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488000f040 con 0x7f48740051c0 2023-10-10T18:56:54.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.946+0000 7f4885d9b700 1 -- 172.21.15.175:0/3115742706 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488000ee70 con 0x7f48740051c0 2023-10-10T18:56:54.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.947+0000 7f488cfdc700 1 -- 172.21.15.175:0/3115742706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48740051c0 msgr2=0x7f487400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:54.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.947+0000 7f488cfdc700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48740051c0 0x7f487400ba60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f488000e210 tx=0x7f488000b040 comp rx=0 tx=0).stop 2023-10-10T18:56:54.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.947+0000 7f488cfdc700 1 -- 172.21.15.175:0/3115742706 shutdown_connections 2023-10-10T18:56:54.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.947+0000 7f488cfdc700 1 --2- 172.21.15.175:0/3115742706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48740051c0 0x7f487400ba60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:54.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.947+0000 7f488cfdc700 1 -- 172.21.15.175:0/3115742706 >> 172.21.15.175:0/3115742706 conn(0x7f4888063230 msgr2=0x7f4888063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:54.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.947+0000 7f488cfdc700 1 -- 172.21.15.175:0/3115742706 shutdown_connections 2023-10-10T18:56:54.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.947+0000 7f488cfdc700 1 -- 172.21.15.175:0/3115742706 wait complete. 2023-10-10T18:56:54.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.948+0000 7f488cfdc700 1 Processor -- start 2023-10-10T18:56:54.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.948+0000 7f488cfdc700 1 -- start start 2023-10-10T18:56:54.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f488cfdc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48740051c0 0x7f4888165cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:54.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f488cfdc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4888151120 0x7f48881508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:54.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f488cfdc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4888169c90 0x7f48880ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:54.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f488cfdc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48881578a0 con 0x7f4888169c90 2023-10-10T18:56:54.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f488cfdc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888150f10 con 0x7f4888151120 2023-10-10T18:56:54.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f488cfdc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4888169800 con 0x7f48740051c0 2023-10-10T18:56:54.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4888169c90 0x7f48880ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:54.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4888169c90 0x7f48880ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42008/0 (socket says 172.21.15.175:42008) 2023-10-10T18:56:54.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 -- 172.21.15.175:0/4149574384 learned_addr learned my addr 172.21.15.175:0/4149574384 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:54.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f487dd9b700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4888151120 0x7f48881508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:54.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f488659c700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48740051c0 0x7f4888165cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:54.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 -- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48740051c0 msgr2=0x7f4888165cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:54.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48740051c0 0x7f4888165cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:54.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 -- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4888151120 msgr2=0x7f48881508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:54.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4888151120 0x7f48881508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:54.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 -- 172.21.15.175:0/4149574384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4888157320 con 0x7f4888169c90 2023-10-10T18:56:54.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f4886d9d700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4888169c90 0x7f48880ca760 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f488808a030 tx=0x7f4880019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:54.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f487effd700 1 -- 172.21.15.175:0/4149574384 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488001d660 con 0x7f4888169c90 2023-10-10T18:56:54.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f487effd700 1 -- 172.21.15.175:0/4149574384 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880022070 con 0x7f4888169c90 2023-10-10T18:56:54.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48881575c0 con 0x7f4888169c90 2023-10-10T18:56:54.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.949+0000 7f487effd700 1 -- 172.21.15.175:0/4149574384 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488001da10 con 0x7f4888169c90 2023-10-10T18:56:54.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.950+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48880aff60 con 0x7f4888169c90 2023-10-10T18:56:54.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.951+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48881692f0 con 0x7f4888169c90 2023-10-10T18:56:54.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.951+0000 7f487effd700 1 -- 172.21.15.175:0/4149574384 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4880003020 con 0x7f4888169c90 2023-10-10T18:56:54.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.951+0000 7f487effd700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4868066f30 0x7f48680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:54.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.951+0000 7f487effd700 1 -- 172.21.15.175:0/4149574384 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f4880024070 con 0x7f4888169c90 2023-10-10T18:56:54.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.951+0000 7f488659c700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4868066f30 0x7f48680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:54.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.952+0000 7f488659c700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4868066f30 0x7f48680693e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4878001050 tx=0x7f48880af650 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:54.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:54.956+0000 7f487effd700 1 -- 172.21.15.175:0/4149574384 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f48881692f0 con 0x7f4888169c90 2023-10-10T18:56:55.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.182+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48881692f0 con 0x7f4888169c90 2023-10-10T18:56:55.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.183+0000 7f487effd700 1 -- 172.21.15.175:0/4149574384 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7f48881692f0 con 0x7f4888169c90 2023-10-10T18:56:55.183 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:55.183 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:55.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4868066f30 msgr2=0x7f48680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:55.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4868066f30 0x7f48680693e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4878001050 tx=0x7f48880af650 comp rx=0 tx=0).stop 2023-10-10T18:56:55.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4888169c90 msgr2=0x7f48880ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:55.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4888169c90 0x7f48880ca760 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f488808a030 tx=0x7f4880019000 comp rx=0 tx=0).stop 2023-10-10T18:56:55.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 shutdown_connections 2023-10-10T18:56:55.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4868066f30 0x7f48680693e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:55.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4888169c90 0x7f48880ca760 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:55.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48740051c0 0x7f4888165cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:55.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 --2- 172.21.15.175:0/4149574384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4888151120 0x7f48881508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:55.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 >> 172.21.15.175:0/4149574384 conn(0x7f4888063230 msgr2=0x7f48880cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:55.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 shutdown_connections 2023-10-10T18:56:55.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:55.185+0000 7f488cfdc700 1 -- 172.21.15.175:0/4149574384 wait complete. 2023-10-10T18:56:55.188 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:56:55.197 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T18:56:57.198 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:57.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.326+0000 7fa068d84700 1 Processor -- start 2023-10-10T18:56:57.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.326+0000 7fa068d84700 1 -- start start 2023-10-10T18:56:57.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa068d84700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa0640b2390 0x7fa064155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:57.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa068d84700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa0640aff60 0x7fa0640b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:57.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa068d84700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0641574b0 con 0x7fa0640b1fc0 2023-10-10T18:56:57.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa068d84700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa064150770 con 0x7fa0640b2390 2023-10-10T18:56:57.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa068d84700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0640afad0 con 0x7fa0640aff60 2023-10-10T18:56:57.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa06259c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa0640b1fc0 0x7fa064068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39346/0 (socket says 172.21.15.175:39346) 2023-10-10T18:56:57.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa061d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa0640b2390 0x7fa064155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:57.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa06259c700 1 -- 172.21.15.175:0/726320068 learned_addr learned my addr 172.21.15.175:0/726320068 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:56:57.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.327+0000 7fa062d9d700 1 --2- 172.21.15.175:0/726320068 >> v2:172.21.15.191:3300/0 conn(0x7fa0640aff60 0x7fa0640b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:57.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa061d9b700 1 -- 172.21.15.175:0/726320068 >> v2:172.21.15.191:3300/0 conn(0x7fa0640aff60 msgr2=0x7fa0640b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:57.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa061d9b700 1 --2- 172.21.15.175:0/726320068 >> v2:172.21.15.191:3300/0 conn(0x7fa0640aff60 0x7fa0640b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa061d9b700 1 -- 172.21.15.175:0/726320068 >> v1:172.21.15.175:6789/0 conn(0x7fa0640b1fc0 legacy=0x7fa064068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:57.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa061d9b700 1 -- 172.21.15.175:0/726320068 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640af4c0 con 0x7fa0640b2390 2023-10-10T18:56:57.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa06159a700 1 -- 172.21.15.175:0/726320068 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37797890 0 0) 0x7fa0641574b0 con 0x7fa0640b1fc0 2023-10-10T18:56:57.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa061d9b700 1 --2- 172.21.15.175:0/726320068 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b2390 0x7fa064155fb0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa0640641f0 tx=0x7fa0580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7fea316fd25ef4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:57.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa06159a700 1 -- 172.21.15.175:0/726320068 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800b690 con 0x7fa0640b2390 2023-10-10T18:56:57.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa06159a700 1 -- 172.21.15.175:0/726320068 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05800b7f0 con 0x7fa0640b2390 2023-10-10T18:56:57.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.328+0000 7fa06159a700 1 -- 172.21.15.175:0/726320068 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800bb90 con 0x7fa0640b2390 2023-10-10T18:56:57.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.329+0000 7fa068d84700 1 -- 172.21.15.175:0/726320068 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b2390 msgr2=0x7fa064155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:57.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.329+0000 7fa068d84700 1 --2- 172.21.15.175:0/726320068 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b2390 0x7fa064155fb0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa0640641f0 tx=0x7fa0580091a0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.329+0000 7fa068d84700 1 -- 172.21.15.175:0/726320068 shutdown_connections 2023-10-10T18:56:57.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.329+0000 7fa068d84700 1 --2- 172.21.15.175:0/726320068 >> v2:172.21.15.191:3300/0 conn(0x7fa0640aff60 0x7fa0640b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.329+0000 7fa068d84700 1 --2- 172.21.15.175:0/726320068 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b2390 0x7fa064155fb0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.329+0000 7fa068d84700 1 -- 172.21.15.175:0/726320068 >> 172.21.15.175:0/726320068 conn(0x7fa064063230 msgr2=0x7fa064063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:57.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.329+0000 7fa068d84700 1 -- 172.21.15.175:0/726320068 shutdown_connections 2023-10-10T18:56:57.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.329+0000 7fa068d84700 1 -- 172.21.15.175:0/726320068 wait complete. 2023-10-10T18:56:57.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.330+0000 7fa068d84700 1 Processor -- start 2023-10-10T18:56:57.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.330+0000 7fa068d84700 1 -- start start 2023-10-10T18:56:57.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.330+0000 7fa068d84700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa0640aff60 0x7fa0641690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:57.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.330+0000 7fa068d84700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b1fc0 0x7fa0640cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:57.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.330+0000 7fa068d84700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0640b2390 0x7fa0641659a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:57.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.330+0000 7fa068d84700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640c3ed0 con 0x7fa0640b1fc0 2023-10-10T18:56:57.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.330+0000 7fa068d84700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa064159bf0 con 0x7fa0640b2390 2023-10-10T18:56:57.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.330+0000 7fa068d84700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa064159760 con 0x7fa0640aff60 2023-10-10T18:56:57.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b1fc0 0x7fa0640cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:57.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b1fc0 0x7fa0640cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42020/0 (socket says 172.21.15.175:42020) 2023-10-10T18:56:57.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 -- 172.21.15.175:0/1045068203 learned_addr learned my addr 172.21.15.175:0/1045068203 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:57.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa062d9d700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0640b2390 0x7fa0641659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:57.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa06259c700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa0640aff60 0x7fa0641690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:57.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 -- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa0640aff60 msgr2=0x7fa0641690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:57.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa0640aff60 0x7fa0641690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 -- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0640b2390 msgr2=0x7fa0641659a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:57.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0640b2390 0x7fa0641659a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 -- 172.21.15.175:0/1045068203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa058009420 con 0x7fa0640b1fc0 2023-10-10T18:56:57.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa062d9d700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0640b2390 0x7fa0641659a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:56:57.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa061d9b700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b1fc0 0x7fa0640cb570 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa05800bfd0 tx=0x7fa058017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:57.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa0527fc700 1 -- 172.21.15.175:0/1045068203 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800bb70 con 0x7fa0640b1fc0 2023-10-10T18:56:57.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa0527fc700 1 -- 172.21.15.175:0/1045068203 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa058011420 con 0x7fa0640b1fc0 2023-10-10T18:56:57.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0641580f0 con 0x7fa0640b1fc0 2023-10-10T18:56:57.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa0527fc700 1 -- 172.21.15.175:0/1045068203 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0580117c0 con 0x7fa0640b1fc0 2023-10-10T18:56:57.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.331+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa064153dc0 con 0x7fa0640b1fc0 2023-10-10T18:56:57.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.332+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa064159910 con 0x7fa0640b1fc0 2023-10-10T18:56:57.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.333+0000 7fa0527fc700 1 -- 172.21.15.175:0/1045068203 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa058011bb0 con 0x7fa0640b1fc0 2023-10-10T18:56:57.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.334+0000 7fa0527fc700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa044066f60 0x7fa044069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:57.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.334+0000 7fa0527fc700 1 -- 172.21.15.175:0/1045068203 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fa058086490 con 0x7fa0640b1fc0 2023-10-10T18:56:57.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.334+0000 7fa06259c700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa044066f60 0x7fa044069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:57.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.334+0000 7fa06259c700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa044066f60 0x7fa044069410 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa0540021a0 tx=0x7fa054009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:57.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.337+0000 7fa0527fc700 1 -- 172.21.15.175:0/1045068203 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa0580534c0 con 0x7fa0640b1fc0 2023-10-10T18:56:57.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.567+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa064159910 con 0x7fa0640b1fc0 2023-10-10T18:56:57.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.567+0000 7fa0527fc700 1 -- 172.21.15.175:0/1045068203 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7fa064159910 con 0x7fa0640b1fc0 2023-10-10T18:56:57.568 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:57.568 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:57.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.569+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa044066f60 msgr2=0x7fa044069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:57.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.569+0000 7fa068d84700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa044066f60 0x7fa044069410 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa0540021a0 tx=0x7fa054009040 comp rx=0 tx=0).stop 2023-10-10T18:56:57.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.569+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b1fc0 msgr2=0x7fa0640cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:57.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.569+0000 7fa068d84700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b1fc0 0x7fa0640cb570 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa05800bfd0 tx=0x7fa058017000 comp rx=0 tx=0).stop 2023-10-10T18:56:57.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.570+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 shutdown_connections 2023-10-10T18:56:57.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.570+0000 7fa068d84700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa044066f60 0x7fa044069410 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.570+0000 7fa068d84700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0640b2390 0x7fa0641659a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.570+0000 7fa068d84700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa0640aff60 0x7fa0641690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.570+0000 7fa068d84700 1 --2- 172.21.15.175:0/1045068203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0640b1fc0 0x7fa0640cb570 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:57.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.570+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 >> 172.21.15.175:0/1045068203 conn(0x7fa064063230 msgr2=0x7fa064152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:57.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.570+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 shutdown_connections 2023-10-10T18:56:57.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:57.570+0000 7fa068d84700 1 -- 172.21.15.175:0/1045068203 wait complete. 2023-10-10T18:56:57.572 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:56:57.581 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T18:56:59.582 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:56:59.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.710+0000 7f401f997700 1 Processor -- start 2023-10-10T18:56:59.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.710+0000 7f401f997700 1 -- start start 2023-10-10T18:56:59.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.710+0000 7f401f997700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f40180b1fc0 0x7f4018155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:59.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401f997700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f40180b2390 0x7f4018068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:59.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401f997700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4018068bd0 0x7f40180aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:59.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401f997700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4018150770 con 0x7f40180b2390 2023-10-10T18:56:59.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401f997700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40181574b0 con 0x7f40180b1fc0 2023-10-10T18:56:59.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401f997700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40180afad0 con 0x7f4018068bd0 2023-10-10T18:56:59.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f40180b2390 0x7f4018068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:59.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f40180b2390 0x7f4018068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42036/0 (socket says 172.21.15.175:42036) 2023-10-10T18:56:59.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 -- 172.21.15.175:0/824502611 learned_addr learned my addr 172.21.15.175:0/824502611 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:59.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401d733700 1 --2- 172.21.15.175:0/824502611 >> v2:172.21.15.191:3300/0 conn(0x7f40180b1fc0 0x7f4018155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:59.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401df34700 1 --2- 172.21.15.175:0/824502611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4018068bd0 0x7f40180aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:59.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 -- 172.21.15.175:0/824502611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4018068bd0 msgr2=0x7f40180aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:59.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 --2- 172.21.15.175:0/824502611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4018068bd0 0x7f40180aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 -- 172.21.15.175:0/824502611 >> v2:172.21.15.191:3300/0 conn(0x7f40180b1fc0 msgr2=0x7f4018155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:56:59.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 --2- 172.21.15.175:0/824502611 >> v2:172.21.15.191:3300/0 conn(0x7f40180b1fc0 0x7f4018155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 -- 172.21.15.175:0/824502611 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40180af4c0 con 0x7f40180b2390 2023-10-10T18:56:59.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f401cf32700 1 --2- 172.21.15.175:0/824502611 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40180b2390 0x7f4018068690 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f40180b28c0 tx=0x7f400c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744e629dbb20d9a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:59.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f4017fff700 1 -- 172.21.15.175:0/824502611 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00b8f0 con 0x7f40180b2390 2023-10-10T18:56:59.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f4017fff700 1 -- 172.21.15.175:0/824502611 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c00ba50 con 0x7f40180b2390 2023-10-10T18:56:59.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.711+0000 7f4017fff700 1 -- 172.21.15.175:0/824502611 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00bdf0 con 0x7f40180b2390 2023-10-10T18:56:59.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 -- 172.21.15.175:0/824502611 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40180b2390 msgr2=0x7f4018068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:59.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 --2- 172.21.15.175:0/824502611 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40180b2390 0x7f4018068690 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f40180b28c0 tx=0x7f400c009250 comp rx=0 tx=0).stop 2023-10-10T18:56:59.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 -- 172.21.15.175:0/824502611 shutdown_connections 2023-10-10T18:56:59.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 --2- 172.21.15.175:0/824502611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4018068bd0 0x7f40180aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 --2- 172.21.15.175:0/824502611 >> v2:172.21.15.191:3300/0 conn(0x7f40180b1fc0 0x7f4018155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 --2- 172.21.15.175:0/824502611 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40180b2390 0x7f4018068690 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 -- 172.21.15.175:0/824502611 >> 172.21.15.175:0/824502611 conn(0x7f4018063230 msgr2=0x7f4018063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:59.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 -- 172.21.15.175:0/824502611 shutdown_connections 2023-10-10T18:56:59.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.712+0000 7f401f997700 1 -- 172.21.15.175:0/824502611 wait complete. 2023-10-10T18:56:59.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.713+0000 7f401f997700 1 Processor -- start 2023-10-10T18:56:59.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.713+0000 7f401f997700 1 -- start start 2023-10-10T18:56:59.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401f997700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4018068bd0 0x7f401814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:59.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401f997700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40180b1fc0 0x7f40181491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:59.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401f997700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40180b2390 0x7f4018149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:59.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401f997700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40180cb6b0 con 0x7f4018068bd0 2023-10-10T18:56:59.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401f997700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4018148d40 con 0x7f40180b2390 2023-10-10T18:56:59.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401f997700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40181481f0 con 0x7f40180b1fc0 2023-10-10T18:56:59.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401d733700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4018068bd0 0x7f401814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:59.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401d733700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4018068bd0 0x7f401814fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42048/0 (socket says 172.21.15.175:42048) 2023-10-10T18:56:59.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401d733700 1 -- 172.21.15.175:0/506926257 learned_addr learned my addr 172.21.15.175:0/506926257 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:56:59.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401df34700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40180b2390 0x7f4018149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:59.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401cf32700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40180b1fc0 0x7f40181491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:59.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401d733700 1 -- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40180b1fc0 msgr2=0x7f40181491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:59.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401d733700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40180b1fc0 0x7f40181491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401d733700 1 -- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40180b2390 msgr2=0x7f4018149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:59.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401d733700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40180b2390 0x7f4018149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.714+0000 7f401d733700 1 -- 172.21.15.175:0/506926257 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c0094f0 con 0x7f4018068bd0 2023-10-10T18:56:59.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.715+0000 7f401d733700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4018068bd0 0x7f401814fc60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f400800d8c0 tx=0x7f4008008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:59.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.715+0000 7f40157fa700 1 -- 172.21.15.175:0/506926257 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40080125f0 con 0x7f4018068bd0 2023-10-10T18:56:59.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.715+0000 7f40157fa700 1 -- 172.21.15.175:0/506926257 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4008012750 con 0x7f4018068bd0 2023-10-10T18:56:59.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.715+0000 7f40157fa700 1 -- 172.21.15.175:0/506926257 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008012af0 con 0x7f4018068bd0 2023-10-10T18:56:59.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.715+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401814af30 con 0x7f4018068bd0 2023-10-10T18:56:59.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.715+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40180cb8e0 con 0x7f4018068bd0 2023-10-10T18:56:59.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.716+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4018149020 con 0x7f4018068bd0 2023-10-10T18:56:59.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.716+0000 7f40157fa700 1 -- 172.21.15.175:0/506926257 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f400801e460 con 0x7f4018068bd0 2023-10-10T18:56:59.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.716+0000 7f40157fa700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4004066f30 0x7f40040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:56:59.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.717+0000 7f40157fa700 1 -- 172.21.15.175:0/506926257 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f4008084d70 con 0x7f4018068bd0 2023-10-10T18:56:59.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.717+0000 7f401cf32700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4004066f30 0x7f40040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:56:59.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.717+0000 7f401cf32700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4004066f30 0x7f40040693e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f40180b28c0 tx=0x7f400c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:56:59.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.721+0000 7f40157fa700 1 -- 172.21.15.175:0/506926257 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4008051d20 con 0x7f4018068bd0 2023-10-10T18:56:59.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.944+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4018067570 con 0x7f4018068bd0 2023-10-10T18:56:59.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.945+0000 7f40157fa700 1 -- 172.21.15.175:0/506926257 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7f4008055340 con 0x7f4018068bd0 2023-10-10T18:56:59.945 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:56:59.945 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:56:59.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4004066f30 msgr2=0x7f40040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:59.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4004066f30 0x7f40040693e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f40180b28c0 tx=0x7f400c017000 comp rx=0 tx=0).stop 2023-10-10T18:56:59.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4018068bd0 msgr2=0x7f401814fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:56:59.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4018068bd0 0x7f401814fc60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f400800d8c0 tx=0x7f4008008040 comp rx=0 tx=0).stop 2023-10-10T18:56:59.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 shutdown_connections 2023-10-10T18:56:59.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4004066f30 0x7f40040693e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40180b2390 0x7f4018149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4018068bd0 0x7f401814fc60 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 --2- 172.21.15.175:0/506926257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40180b1fc0 0x7f40181491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:56:59.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 >> 172.21.15.175:0/506926257 conn(0x7f4018063230 msgr2=0x7f4018063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:56:59.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 shutdown_connections 2023-10-10T18:56:59.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:56:59.947+0000 7f401f997700 1 -- 172.21.15.175:0/506926257 wait complete. 2023-10-10T18:56:59.950 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:56:59.959 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T18:57:01.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:02.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.087+0000 7fc2c62a7700 1 Processor -- start 2023-10-10T18:57:02.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.087+0000 7fc2c62a7700 1 -- start start 2023-10-10T18:57:02.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c62a7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b2390 0x7fc2c0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:02.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c62a7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc2c00aff60 0x7fc2c00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:02.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c62a7700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c01574b0 con 0x7fc2c00b1fc0 2023-10-10T18:57:02.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c62a7700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c0150770 con 0x7fc2c00aff60 2023-10-10T18:57:02.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c62a7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00afad0 con 0x7fc2c00b2390 2023-10-10T18:57:02.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c4844700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc2c00aff60 0x7fc2c00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:02.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2bffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc2c00b1fc0 0x7fc2c0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39354/0 (socket says 172.21.15.175:39354) 2023-10-10T18:57:02.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2bffff700 1 -- 172.21.15.175:0/1867428177 learned_addr learned my addr 172.21.15.175:0/1867428177 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:57:02.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2bf7fe700 1 --2- 172.21.15.175:0/1867428177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b2390 0x7fc2c0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:02.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c4844700 1 -- 172.21.15.175:0/1867428177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b2390 msgr2=0x7fc2c0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:02.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c4844700 1 --2- 172.21.15.175:0/1867428177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b2390 0x7fc2c0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c4844700 1 -- 172.21.15.175:0/1867428177 >> v1:172.21.15.175:6789/0 conn(0x7fc2c00b1fc0 legacy=0x7fc2c0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:02.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2beffd700 1 -- 172.21.15.175:0/1867428177 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073688381 0 0) 0x7fc2c01574b0 con 0x7fc2c00b1fc0 2023-10-10T18:57:02.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.088+0000 7fc2c4844700 1 -- 172.21.15.175:0/1867428177 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c00af4c0 con 0x7fc2c00aff60 2023-10-10T18:57:02.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c4844700 1 --2- 172.21.15.175:0/1867428177 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00aff60 0x7fc2c00b03e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009e90 tx=0x7fc2b8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ca31b7a1e3e245 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:02.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2beffd700 1 -- 172.21.15.175:0/1867428177 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b800e070 con 0x7fc2c00aff60 2023-10-10T18:57:02.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2beffd700 1 -- 172.21.15.175:0/1867428177 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b800bc50 con 0x7fc2c00aff60 2023-10-10T18:57:02.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2beffd700 1 -- 172.21.15.175:0/1867428177 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8012430 con 0x7fc2c00aff60 2023-10-10T18:57:02.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1867428177 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00aff60 msgr2=0x7fc2c00b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:02.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1867428177 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00aff60 0x7fc2c00b03e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009e90 tx=0x7fc2b8009780 comp rx=0 tx=0).stop 2023-10-10T18:57:02.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1867428177 shutdown_connections 2023-10-10T18:57:02.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1867428177 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00aff60 0x7fc2c00b03e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1867428177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b2390 0x7fc2c0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1867428177 >> 172.21.15.175:0/1867428177 conn(0x7fc2c0063230 msgr2=0x7fc2c0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:02.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1867428177 shutdown_connections 2023-10-10T18:57:02.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.089+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1867428177 wait complete. 2023-10-10T18:57:02.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.090+0000 7fc2c62a7700 1 Processor -- start 2023-10-10T18:57:02.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c62a7700 1 -- start start 2023-10-10T18:57:02.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c62a7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2c00aff60 0x7fc2c014e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:02.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c62a7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b1fc0 0x7fc2c0169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:02.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c62a7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00b2390 0x7fc2c00cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:02.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c62a7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00c4ae0 con 0x7fc2c00b2390 2023-10-10T18:57:02.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c62a7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00cd210 con 0x7fc2c00aff60 2023-10-10T18:57:02.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c62a7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00c4160 con 0x7fc2c00b1fc0 2023-10-10T18:57:02.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c4844700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00b2390 0x7fc2c00cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:02.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c4844700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00b2390 0x7fc2c00cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42058/0 (socket says 172.21.15.175:42058) 2023-10-10T18:57:02.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2c4844700 1 -- 172.21.15.175:0/1770678877 learned_addr learned my addr 172.21.15.175:0/1770678877 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:02.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2bf7fe700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b1fc0 0x7fc2c0169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:02.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.091+0000 7fc2bffff700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2c00aff60 0x7fc2c014e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:02.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2c4844700 1 -- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b1fc0 msgr2=0x7fc2c0169310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:02.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2c4844700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b1fc0 0x7fc2c0169310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2c4844700 1 -- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2c00aff60 msgr2=0x7fc2c014e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:02.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2c4844700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2c00aff60 0x7fc2c014e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2c4844700 1 -- 172.21.15.175:0/1770678877 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b8009a20 con 0x7fc2c00b2390 2023-10-10T18:57:02.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2c4844700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00b2390 0x7fc2c00cc020 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8013000 tx=0x7fc2b8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:02.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2a7fff700 1 -- 172.21.15.175:0/1770678877 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8012da0 con 0x7fc2c00b2390 2023-10-10T18:57:02.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2a7fff700 1 -- 172.21.15.175:0/1770678877 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b8021070 con 0x7fc2c00b2390 2023-10-10T18:57:02.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c00c3870 con 0x7fc2c00b2390 2023-10-10T18:57:02.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2a7fff700 1 -- 172.21.15.175:0/1770678877 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8026680 con 0x7fc2c00b2390 2023-10-10T18:57:02.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.092+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c00c3b10 con 0x7fc2c00b2390 2023-10-10T18:57:02.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.093+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c0067570 con 0x7fc2c00b2390 2023-10-10T18:57:02.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.093+0000 7fc2a7fff700 1 -- 172.21.15.175:0/1770678877 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc2b8023030 con 0x7fc2c00b2390 2023-10-10T18:57:02.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.094+0000 7fc2a7fff700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc2ac066f30 0x7fc2ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:02.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.094+0000 7fc2a7fff700 1 -- 172.21.15.175:0/1770678877 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fc2b80869c0 con 0x7fc2c00b2390 2023-10-10T18:57:02.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.094+0000 7fc2bffff700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc2ac066f30 0x7fc2ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:02.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.095+0000 7fc2bffff700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc2ac066f30 0x7fc2ac0693e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0007fd0 tx=0x7fc2b0007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:02.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.098+0000 7fc2a7fff700 1 -- 172.21.15.175:0/1770678877 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2b8053a10 con 0x7fc2c00b2390 2023-10-10T18:57:02.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.324+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2c0068da0 con 0x7fc2c00b2390 2023-10-10T18:57:02.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.324+0000 7fc2a7fff700 1 -- 172.21.15.175:0/1770678877 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7fc2b8057030 con 0x7fc2c00b2390 2023-10-10T18:57:02.325 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:02.326 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:02.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc2ac066f30 msgr2=0x7fc2ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:02.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc2ac066f30 0x7fc2ac0693e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0007fd0 tx=0x7fc2b0007d30 comp rx=0 tx=0).stop 2023-10-10T18:57:02.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00b2390 msgr2=0x7fc2c00cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:02.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00b2390 0x7fc2c00cc020 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8013000 tx=0x7fc2b8019000 comp rx=0 tx=0).stop 2023-10-10T18:57:02.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 shutdown_connections 2023-10-10T18:57:02.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc2ac066f30 0x7fc2ac0693e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2c00b2390 0x7fc2c00cc020 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2c00aff60 0x7fc2c014e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 --2- 172.21.15.175:0/1770678877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc2c00b1fc0 0x7fc2c0169310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:02.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.326+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 >> 172.21.15.175:0/1770678877 conn(0x7fc2c0063230 msgr2=0x7fc2c01508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:02.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.327+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 shutdown_connections 2023-10-10T18:57:02.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:02.327+0000 7fc2c62a7700 1 -- 172.21.15.175:0/1770678877 wait complete. 2023-10-10T18:57:02.333 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:57:02.339 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T18:57:04.340 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:04.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.467+0000 7ff2705f3700 1 Processor -- start 2023-10-10T18:57:04.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.467+0000 7ff2705f3700 1 -- start start 2023-10-10T18:57:04.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.468+0000 7ff2705f3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff2680b1fc0 0x7ff268155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:04.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.468+0000 7ff2705f3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff2680b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:04.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.468+0000 7ff2705f3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2681574b0 con 0x7ff2680b2390 2023-10-10T18:57:04.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.468+0000 7ff2705f3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff268150770 con 0x7ff2680b1fc0 2023-10-10T18:57:04.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.468+0000 7ff2705f3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680afad0 con 0x7ff2680aff60 2023-10-10T18:57:04.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26e38f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff2680b1fc0 0x7ff268155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:04.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26eb90700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff2680b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:04.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26e38f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff2680b1fc0 0x7ff268155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:33912/0 (socket says 172.21.15.175:33912) 2023-10-10T18:57:04.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26eb90700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff2680b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:50270/0 (socket says 172.21.15.175:50270) 2023-10-10T18:57:04.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26db8e700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff2680b2390 0x7ff268068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:48954/0 (socket says 172.21.15.175:48954) 2023-10-10T18:57:04.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26e38f700 1 -- 172.21.15.175:0/1120686488 learned_addr learned my addr 172.21.15.175:0/1120686488 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:04.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26eb90700 1 -- 172.21.15.175:0/1120686488 >> v2:172.21.15.191:3300/0 conn(0x7ff2680b1fc0 msgr2=0x7ff268155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:04.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26eb90700 1 --2- 172.21.15.175:0/1120686488 >> v2:172.21.15.191:3300/0 conn(0x7ff2680b1fc0 0x7ff268155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26eb90700 1 -- 172.21.15.175:0/1120686488 >> v1:172.21.15.191:6789/0 conn(0x7ff2680b2390 legacy=0x7ff268068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:04.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26eb90700 1 -- 172.21.15.175:0/1120686488 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2680af4c0 con 0x7ff2680aff60 2023-10-10T18:57:04.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26e38f700 1 --2- 172.21.15.175:0/1120686488 >> v2:172.21.15.191:3300/0 conn(0x7ff2680b1fc0 0x7ff268155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:57:04.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26eb90700 1 --2- 172.21.15.175:0/1120686488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff2680b0400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff2600098e0 tx=0x7ff2600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab827a15d833497a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:04.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26d38d700 1 -- 172.21.15.175:0/1120686488 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26000b6d0 con 0x7ff2680aff60 2023-10-10T18:57:04.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26d38d700 1 -- 172.21.15.175:0/1120686488 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26000b830 con 0x7ff2680aff60 2023-10-10T18:57:04.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.469+0000 7ff26d38d700 1 -- 172.21.15.175:0/1120686488 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26000bbd0 con 0x7ff2680aff60 2023-10-10T18:57:04.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.470+0000 7ff2705f3700 1 -- 172.21.15.175:0/1120686488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 msgr2=0x7ff2680b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:04.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.470+0000 7ff2705f3700 1 --2- 172.21.15.175:0/1120686488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff2680b0400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff2600098e0 tx=0x7ff2600091a0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.470+0000 7ff2705f3700 1 -- 172.21.15.175:0/1120686488 shutdown_connections 2023-10-10T18:57:04.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.470+0000 7ff2705f3700 1 --2- 172.21.15.175:0/1120686488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff2680b0400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.470+0000 7ff2705f3700 1 --2- 172.21.15.175:0/1120686488 >> v2:172.21.15.191:3300/0 conn(0x7ff2680b1fc0 0x7ff268155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.470+0000 7ff2705f3700 1 -- 172.21.15.175:0/1120686488 >> 172.21.15.175:0/1120686488 conn(0x7ff268063230 msgr2=0x7ff268063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:04.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.470+0000 7ff2705f3700 1 -- 172.21.15.175:0/1120686488 shutdown_connections 2023-10-10T18:57:04.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.470+0000 7ff2705f3700 1 -- 172.21.15.175:0/1120686488 wait complete. 2023-10-10T18:57:04.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.471+0000 7ff2705f3700 1 Processor -- start 2023-10-10T18:57:04.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.471+0000 7ff2705f3700 1 -- start start 2023-10-10T18:57:04.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff2705f3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff268154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:04.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff2705f3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2680b1fc0 0x7ff2681567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:04.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff2705f3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2680b2390 0x7ff2680c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:04.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff2705f3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680c40f0 con 0x7ff2680b2390 2023-10-10T18:57:04.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff2705f3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff268155c80 con 0x7ff2680b1fc0 2023-10-10T18:57:04.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff2705f3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2681556b0 con 0x7ff2680aff60 2023-10-10T18:57:04.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2680b2390 0x7ff2680c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:04.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2680b2390 0x7ff2680c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42064/0 (socket says 172.21.15.175:42064) 2023-10-10T18:57:04.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 -- 172.21.15.175:0/2339157131 learned_addr learned my addr 172.21.15.175:0/2339157131 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:04.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26db8e700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2680b1fc0 0x7ff2681567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:04.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26e38f700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff268154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:04.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 -- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 msgr2=0x7ff268154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:04.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff268154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 -- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2680b1fc0 msgr2=0x7ff2681567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:04.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2680b1fc0 0x7ff2681567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 -- 172.21.15.175:0/2339157131 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff264009ae0 con 0x7ff2680b2390 2023-10-10T18:57:04.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.472+0000 7ff26eb90700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2680b2390 0x7ff2680c5670 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff2680afc10 tx=0x7ff260017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:04.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.473+0000 7ff25e7fc700 1 -- 172.21.15.175:0/2339157131 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260011420 con 0x7ff2680b2390 2023-10-10T18:57:04.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.473+0000 7ff25e7fc700 1 -- 172.21.15.175:0/2339157131 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff260011580 con 0x7ff2680b2390 2023-10-10T18:57:04.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.473+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff260009420 con 0x7ff2680b2390 2023-10-10T18:57:04.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.473+0000 7ff25e7fc700 1 -- 172.21.15.175:0/2339157131 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260011940 con 0x7ff2680b2390 2023-10-10T18:57:04.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.473+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2680cd560 con 0x7ff2680b2390 2023-10-10T18:57:04.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.473+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff268067570 con 0x7ff2680b2390 2023-10-10T18:57:04.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.474+0000 7ff25e7fc700 1 -- 172.21.15.175:0/2339157131 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff260011cb0 con 0x7ff2680b2390 2023-10-10T18:57:04.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.474+0000 7ff25e7fc700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff24c066f60 0x7ff24c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:04.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.474+0000 7ff25e7fc700 1 -- 172.21.15.175:0/2339157131 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7ff260086270 con 0x7ff2680b2390 2023-10-10T18:57:04.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.474+0000 7ff26e38f700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff24c066f60 0x7ff24c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:04.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.475+0000 7ff26e38f700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff24c066f60 0x7ff24c069410 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff2640072d0 tx=0x7ff26400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:04.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.479+0000 7ff25e7fc700 1 -- 172.21.15.175:0/2339157131 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff2600532c0 con 0x7ff2680b2390 2023-10-10T18:57:04.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.705+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2680c52a0 con 0x7ff2680b2390 2023-10-10T18:57:04.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.706+0000 7ff25e7fc700 1 -- 172.21.15.175:0/2339157131 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7ff2600568e0 con 0x7ff2680b2390 2023-10-10T18:57:04.706 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:04.707 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:04.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff24c066f60 msgr2=0x7ff24c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:04.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff24c066f60 0x7ff24c069410 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff2640072d0 tx=0x7ff26400b040 comp rx=0 tx=0).stop 2023-10-10T18:57:04.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2680b2390 msgr2=0x7ff2680c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:04.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2680b2390 0x7ff2680c5670 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff2680afc10 tx=0x7ff260017000 comp rx=0 tx=0).stop 2023-10-10T18:57:04.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 shutdown_connections 2023-10-10T18:57:04.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff24c066f60 0x7ff24c069410 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2680b2390 0x7ff2680c5670 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2680aff60 0x7ff268154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 --2- 172.21.15.175:0/2339157131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2680b1fc0 0x7ff2681567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:04.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 >> 172.21.15.175:0/2339157131 conn(0x7ff268063230 msgr2=0x7ff2681508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:04.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 shutdown_connections 2023-10-10T18:57:04.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:04.708+0000 7ff2705f3700 1 -- 172.21.15.175:0/2339157131 wait complete. 2023-10-10T18:57:04.711 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:57:04.720 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T18:57:06.721 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:06.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.848+0000 7fe1d489f700 1 Processor -- start 2023-10-10T18:57:06.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.848+0000 7fe1d489f700 1 -- start start 2023-10-10T18:57:06.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1d489f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0068cf0 0x7fe1d0067b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:06.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1d489f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe1d014fd20 0x7fe1d01501c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:06.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1d489f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d0068b30 con 0x7fe1d0159dd0 2023-10-10T18:57:06.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1d489f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d00c1c70 con 0x7fe1d014fd20 2023-10-10T18:57:06.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1d489f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d00cd2b0 con 0x7fe1d0068cf0 2023-10-10T18:57:06.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1cf59e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe1d014fd20 0x7fe1d01501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:06.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1cf59e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe1d014fd20 0x7fe1d01501c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58376/0 (socket says 172.21.15.175:58376) 2023-10-10T18:57:06.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1cf59e700 1 -- 172.21.15.175:0/2509977761 learned_addr learned my addr 172.21.15.175:0/2509977761 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:06.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1ced9d700 1 --2- 172.21.15.175:0/2509977761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0068cf0 0x7fe1d0067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:06.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1cf59e700 1 -- 172.21.15.175:0/2509977761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0068cf0 msgr2=0x7fe1d0067b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:06.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1cf59e700 1 --2- 172.21.15.175:0/2509977761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0068cf0 0x7fe1d0067b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:06.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1cf59e700 1 -- 172.21.15.175:0/2509977761 >> v1:172.21.15.191:6789/0 conn(0x7fe1d0159dd0 legacy=0x7fe1d00c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:06.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.849+0000 7fe1cf59e700 1 -- 172.21.15.175:0/2509977761 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00cd590 con 0x7fe1d014fd20 2023-10-10T18:57:06.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1cf59e700 1 --2- 172.21.15.175:0/2509977761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d014fd20 0x7fe1d01501c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009f20 tx=0x7fe1c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942fa20535b009a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:06.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1ce59c700 1 -- 172.21.15.175:0/2509977761 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c800e070 con 0x7fe1d014fd20 2023-10-10T18:57:06.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1ce59c700 1 -- 172.21.15.175:0/2509977761 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c800ba60 con 0x7fe1d014fd20 2023-10-10T18:57:06.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1ce59c700 1 -- 172.21.15.175:0/2509977761 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c80092e0 con 0x7fe1d014fd20 2023-10-10T18:57:06.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1d489f700 1 -- 172.21.15.175:0/2509977761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d014fd20 msgr2=0x7fe1d01501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:06.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1d489f700 1 --2- 172.21.15.175:0/2509977761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d014fd20 0x7fe1d01501c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009f20 tx=0x7fe1c8009840 comp rx=0 tx=0).stop 2023-10-10T18:57:06.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1d489f700 1 -- 172.21.15.175:0/2509977761 shutdown_connections 2023-10-10T18:57:06.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1d489f700 1 --2- 172.21.15.175:0/2509977761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d014fd20 0x7fe1d01501c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:06.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1d489f700 1 --2- 172.21.15.175:0/2509977761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0068cf0 0x7fe1d0067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:06.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1d489f700 1 -- 172.21.15.175:0/2509977761 >> 172.21.15.175:0/2509977761 conn(0x7fe1d0063cc0 msgr2=0x7fe1d00cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:06.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1d489f700 1 -- 172.21.15.175:0/2509977761 shutdown_connections 2023-10-10T18:57:06.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.850+0000 7fe1d489f700 1 -- 172.21.15.175:0/2509977761 wait complete. 2023-10-10T18:57:06.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.851+0000 7fe1d489f700 1 Processor -- start 2023-10-10T18:57:06.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1d489f700 1 -- start start 2023-10-10T18:57:06.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1d489f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d0068cf0 0x7fe1d014ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:06.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1d489f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1d014fd20 0x7fe1d01525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:06.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1d489f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0159dd0 0x7fe1d0152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:06.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1d489f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d0151f60 con 0x7fe1d0068cf0 2023-10-10T18:57:06.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1d489f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d01507f0 con 0x7fe1d014fd20 2023-10-10T18:57:06.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1d489f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d0159120 con 0x7fe1d0159dd0 2023-10-10T18:57:06.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1ced9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d0068cf0 0x7fe1d014ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:06.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1ced9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d0068cf0 0x7fe1d014ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58392/0 (socket says 172.21.15.175:58392) 2023-10-10T18:57:06.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1ced9d700 1 -- 172.21.15.175:0/5785374 learned_addr learned my addr 172.21.15.175:0/5785374 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:06.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1c659c700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1d014fd20 0x7fe1d01525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:06.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.852+0000 7fe1cf59e700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0159dd0 0x7fe1d0152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:06.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1ced9d700 1 -- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0159dd0 msgr2=0x7fe1d0152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:06.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1ced9d700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0159dd0 0x7fe1d0152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:06.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1ced9d700 1 -- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1d014fd20 msgr2=0x7fe1d01525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:06.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1ced9d700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1d014fd20 0x7fe1d01525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:06.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1ced9d700 1 -- 172.21.15.175:0/5785374 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8009ae0 con 0x7fe1d0068cf0 2023-10-10T18:57:06.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1ced9d700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d0068cf0 0x7fe1d014ebb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000ecb0 tx=0x7fe1c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:06.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1c77fe700 1 -- 172.21.15.175:0/5785374 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000c850 con 0x7fe1d0068cf0 2023-10-10T18:57:06.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1c77fe700 1 -- 172.21.15.175:0/5785374 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c000c9b0 con 0x7fe1d0068cf0 2023-10-10T18:57:06.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d0150a40 con 0x7fe1d0068cf0 2023-10-10T18:57:06.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1c77fe700 1 -- 172.21.15.175:0/5785374 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000cd50 con 0x7fe1d0068cf0 2023-10-10T18:57:06.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.853+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d00c3ca0 con 0x7fe1d0068cf0 2023-10-10T18:57:06.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.854+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d00c42f0 con 0x7fe1d0068cf0 2023-10-10T18:57:06.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.854+0000 7fe1c77fe700 1 -- 172.21.15.175:0/5785374 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe1c0013680 con 0x7fe1d0068cf0 2023-10-10T18:57:06.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.855+0000 7fe1c77fe700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b0066f30 0x7fe1b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:06.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.855+0000 7fe1c77fe700 1 -- 172.21.15.175:0/5785374 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fe1c0084980 con 0x7fe1d0068cf0 2023-10-10T18:57:06.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.855+0000 7fe1c659c700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b0066f30 0x7fe1b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:06.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.855+0000 7fe1c659c700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b0066f30 0x7fe1b00693e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00afe50 tx=0x7fe1b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:06.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:06.859+0000 7fe1c77fe700 1 -- 172.21.15.175:0/5785374 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1c00519d0 con 0x7fe1d0068cf0 2023-10-10T18:57:07.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.082+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1d0067570 con 0x7fe1d0068cf0 2023-10-10T18:57:07.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.082+0000 7fe1c77fe700 1 -- 172.21.15.175:0/5785374 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6197 (secure 0 0 0) 0x7fe1c0054ff0 con 0x7fe1d0068cf0 2023-10-10T18:57:07.083 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:07.083 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5634,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:56:27.792348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4263,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6837/3870390806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3870390806},{"type":"v1","addr":"172.21.15.191:6837","nonce":3870390806}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:07.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b0066f30 msgr2=0x7fe1b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:07.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b0066f30 0x7fe1b00693e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00afe50 tx=0x7fe1b8009040 comp rx=0 tx=0).stop 2023-10-10T18:57:07.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d0068cf0 msgr2=0x7fe1d014ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:07.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d0068cf0 0x7fe1d014ebb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000ecb0 tx=0x7fe1c00091a0 comp rx=0 tx=0).stop 2023-10-10T18:57:07.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 shutdown_connections 2023-10-10T18:57:07.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b0066f30 0x7fe1b00693e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:07.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1d0159dd0 0x7fe1d0152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:07.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1d0068cf0 0x7fe1d014ebb0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:07.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 --2- 172.21.15.175:0/5785374 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1d014fd20 0x7fe1d01525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:07.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 >> 172.21.15.175:0/5785374 conn(0x7fe1d0063cc0 msgr2=0x7fe1d00690c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:07.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 shutdown_connections 2023-10-10T18:57:07.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:07.084+0000 7fe1d489f700 1 -- 172.21.15.175:0/5785374 wait complete. 2023-10-10T18:57:07.087 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 93 2023-10-10T18:57:07.096 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T18:57:09.097 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:09.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.219+0000 7fd7eb6bc700 1 Processor -- start 2023-10-10T18:57:09.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.219+0000 7fd7eb6bc700 1 -- start start 2023-10-10T18:57:09.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7eb6bc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e41577a0 0x7fd7e40c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7eb6bc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e4154aa0 con 0x7fd7e40c12a0 2023-10-10T18:57:09.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7eb6bc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e4063dc0 con 0x7fd7e41573d0 2023-10-10T18:57:09.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7eb6bc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e40bfdf0 con 0x7fd7e41577a0 2023-10-10T18:57:09.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7e8c57700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd7e40c12a0 0x7fd7e414d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39540/0 (socket says 172.21.15.175:39540) 2023-10-10T18:57:09.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7e8c57700 1 -- 172.21.15.175:0/928414061 learned_addr learned my addr 172.21.15.175:0/928414061 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:57:09.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171503271 0 0) 0x7fd7e4154aa0 con 0x7fd7e40c12a0 2023-10-10T18:57:09.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d0003570 con 0x7fd7e40c12a0 2023-10-10T18:57:09.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7e9c59700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e41577a0 0x7fd7e40c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2514734110 0 0) 0x7fd7d0003570 con 0x7fd7e40c12a0 2023-10-10T18:57:09.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e41577a0 msgr2=0x7fd7e40c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:09.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e41577a0 0x7fd7e40c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 >> v1:172.21.15.191:6789/0 conn(0x7fd7e41573d0 legacy=0x7fd7e40bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e4157bc0 con 0x7fd7e40c12a0 2023-10-10T18:57:09.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fd7d4001470 con 0x7fd7e40c12a0 2023-10-10T18:57:09.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 >> v1:172.21.15.175:6789/0 conn(0x7fd7e40c12a0 legacy=0x7fd7e414d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7d0003a90 0x7fd7d0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e41577a0 0x7fd7d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.220+0000 7fd7dbfff700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7d0004d50 0x7fd7d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d000bb10 con 0x7fd7d0004d50 2023-10-10T18:57:09.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d000bc50 con 0x7fd7d0003a90 2023-10-10T18:57:09.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d000be10 con 0x7fd7e41577a0 2023-10-10T18:57:09.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e9c59700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7d0004d50 0x7fd7d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e9458700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7d0003a90 0x7fd7d0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e8c57700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e41577a0 0x7fd7d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e9c59700 1 -- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e41577a0 msgr2=0x7fd7d0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:09.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e9c59700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e41577a0 0x7fd7d0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e9c59700 1 -- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7d0003a90 msgr2=0x7fd7d0003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:09.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e9c59700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7d0003a90 0x7fd7d0003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e9c59700 1 -- 172.21.15.175:0/928414061 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d0003570 con 0x7fd7d0004d50 2023-10-10T18:57:09.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7e9c59700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7d0004d50 0x7fd7d00051c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009c00 tx=0x7fd7e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51111e6eb186b131 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:09.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e000e710 con 0x7fd7d0004d50 2023-10-10T18:57:09.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e000e870 con 0x7fd7d0004d50 2023-10-10T18:57:09.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.221+0000 7fd7dbfff700 1 -- 172.21.15.175:0/928414061 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e000ec30 con 0x7fd7d0004d50 2023-10-10T18:57:09.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.222+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/928414061 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7d0004d50 msgr2=0x7fd7d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.222+0000 7fd7eb6bc700 1 --2- 172.21.15.175:0/928414061 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7d0004d50 0x7fd7d00051c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009c00 tx=0x7fd7e00091a0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.222+0000 7fd7e9c59700 1 -- 172.21.15.175:0/928414061 reap_dead start 2023-10-10T18:57:09.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.222+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/928414061 shutdown_connections 2023-10-10T18:57:09.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.222+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/928414061 >> 172.21.15.175:0/928414061 conn(0x7fd7e406dc70 msgr2=0x7fd7e4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:09.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.222+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/928414061 shutdown_connections 2023-10-10T18:57:09.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.222+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/928414061 wait complete. 2023-10-10T18:57:09.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.223+0000 7fd7eb6bc700 1 Processor -- start 2023-10-10T18:57:09.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.223+0000 7fd7eb6bc700 1 -- start start 2023-10-10T18:57:09.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7eb6bc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7e4147290 0x7fd7e40c2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7eb6bc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7e4069bb0 0x7fd7e41689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7eb6bc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e4069fe0 0x7fd7e4168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7eb6bc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e41697b0 con 0x7fd7e4069bb0 2023-10-10T18:57:09.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7eb6bc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e40c1ca0 con 0x7fd7e4147290 2023-10-10T18:57:09.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7eb6bc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e4069720 con 0x7fd7e4069fe0 2023-10-10T18:57:09.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7e4069bb0 0x7fd7e41689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7e4069bb0 0x7fd7e41689b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58402/0 (socket says 172.21.15.175:58402) 2023-10-10T18:57:09.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 -- 172.21.15.175:0/769281042 learned_addr learned my addr 172.21.15.175:0/769281042 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:09.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e9458700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7e4147290 0x7fd7e40c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e9c59700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e4069fe0 0x7fd7e4168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 -- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e4069fe0 msgr2=0x7fd7e4168ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e4069fe0 0x7fd7e4168ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 -- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7e4147290 msgr2=0x7fd7e40c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7e4147290 0x7fd7e40c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 -- 172.21.15.175:0/769281042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e41694f0 con 0x7fd7e4069bb0 2023-10-10T18:57:09.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e9458700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7e4147290 0x7fd7e40c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:57:09.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7e8c57700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7e4069bb0 0x7fd7e41689b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4004fd0 tx=0x7fd7e40c11c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:09.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7d97fa700 1 -- 172.21.15.175:0/769281042 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4018030 con 0x7fd7e4069bb0 2023-10-10T18:57:09.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7d97fa700 1 -- 172.21.15.175:0/769281042 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d400f8d0 con 0x7fd7e4069bb0 2023-10-10T18:57:09.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e40c19c0 con 0x7fd7e4069bb0 2023-10-10T18:57:09.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7d97fa700 1 -- 172.21.15.175:0/769281042 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d400fc70 con 0x7fd7e4069bb0 2023-10-10T18:57:09.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.224+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e40c32d0 con 0x7fd7e4069bb0 2023-10-10T18:57:09.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.225+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e4069570 con 0x7fd7e4069bb0 2023-10-10T18:57:09.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.226+0000 7fd7d97fa700 1 -- 172.21.15.175:0/769281042 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd7d401b070 con 0x7fd7e4069bb0 2023-10-10T18:57:09.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.226+0000 7fd7d97fa700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7c8066f30 0x7fd7c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.226+0000 7fd7d97fa700 1 -- 172.21.15.175:0/769281042 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fd7d4019070 con 0x7fd7e4069bb0 2023-10-10T18:57:09.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.226+0000 7fd7e9458700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7c8066f30 0x7fd7c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.227+0000 7fd7e9458700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7c8066f30 0x7fd7c80693e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc000f80 tx=0x7fd7dc0016d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:09.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.230+0000 7fd7d97fa700 1 -- 172.21.15.175:0/769281042 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7d40516e0 con 0x7fd7e4069bb0 2023-10-10T18:57:09.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.455+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7e4067570 con 0x7fd7e4069bb0 2023-10-10T18:57:09.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.455+0000 7fd7d97fa700 1 -- 172.21.15.175:0/769281042 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5439 (secure 0 0 0) 0x7fd7d4054d00 con 0x7fd7e4069bb0 2023-10-10T18:57:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:09.456 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:08.527691+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:09.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.457+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7c8066f30 msgr2=0x7fd7c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.457+0000 7fd7eb6bc700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7c8066f30 0x7fd7c80693e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc000f80 tx=0x7fd7dc0016d0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.457+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7e4069bb0 msgr2=0x7fd7e41689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.457+0000 7fd7eb6bc700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7e4069bb0 0x7fd7e41689b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4004fd0 tx=0x7fd7e40c11c0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.458+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 shutdown_connections 2023-10-10T18:57:09.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.458+0000 7fd7eb6bc700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7c8066f30 0x7fd7c80693e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.458+0000 7fd7eb6bc700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7e4069fe0 0x7fd7e4168ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.458+0000 7fd7eb6bc700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7e4147290 0x7fd7e40c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.458+0000 7fd7eb6bc700 1 --2- 172.21.15.175:0/769281042 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7e4069bb0 0x7fd7e41689b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.458+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 >> 172.21.15.175:0/769281042 conn(0x7fd7e406dc70 msgr2=0x7fd7e4155720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:09.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.458+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 shutdown_connections 2023-10-10T18:57:09.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.458+0000 7fd7eb6bc700 1 -- 172.21.15.175:0/769281042 wait complete. 2023-10-10T18:57:09.460 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 94 2023-10-10T18:57:09.469 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-10T18:57:09.469 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T18:57:09.470 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:09.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926f135700 1 Processor -- start 2023-10-10T18:57:09.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926f135700 1 -- start start 2023-10-10T18:57:09.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926f135700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9268155fb0 0x7f9268159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926f135700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92681574b0 con 0x7f92680b1fc0 2023-10-10T18:57:09.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926f135700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680afad0 con 0x7f92680b2390 2023-10-10T18:57:09.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926f135700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9268150770 con 0x7f9268155fb0 2023-10-10T18:57:09.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926d6d2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9268155fb0 0x7f9268159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926ced1700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f92680b1fc0 0x7f9268068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39556/0 (socket says 172.21.15.175:39556) 2023-10-10T18:57:09.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.586+0000 7f926ced1700 1 -- 172.21.15.175:0/2719775407 learned_addr learned my addr 172.21.15.175:0/2719775407 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:57:09.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.587+0000 7f926d6d2700 1 -- 172.21.15.175:0/2719775407 >> v1:172.21.15.191:6789/0 conn(0x7f92680b2390 legacy=0x7f92680aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.587+0000 7f926d6d2700 1 -- 172.21.15.175:0/2719775407 >> v1:172.21.15.175:6789/0 conn(0x7f92680b1fc0 legacy=0x7f9268068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.587+0000 7f926d6d2700 1 -- 172.21.15.175:0/2719775407 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680af4c0 con 0x7f9268155fb0 2023-10-10T18:57:09.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.587+0000 7f92677fe700 1 -- 172.21.15.175:0/2719775407 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879187851 0 0) 0x7f92681574b0 con 0x7f92680b1fc0 2023-10-10T18:57:09.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.587+0000 7f926d6d2700 1 --2- 172.21.15.175:0/2719775407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9268155fb0 0x7f9268159920 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9260009e60 tx=0x7f9260009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ca6ecaa136f74c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:09.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.587+0000 7f92677fe700 1 -- 172.21.15.175:0/2719775407 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926000e070 con 0x7f9268155fb0 2023-10-10T18:57:09.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.587+0000 7f92677fe700 1 -- 172.21.15.175:0/2719775407 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926000bcc0 con 0x7f9268155fb0 2023-10-10T18:57:09.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.587+0000 7f92677fe700 1 -- 172.21.15.175:0/2719775407 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260012680 con 0x7f9268155fb0 2023-10-10T18:57:09.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.588+0000 7f926f135700 1 -- 172.21.15.175:0/2719775407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9268155fb0 msgr2=0x7f9268159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.588+0000 7f926f135700 1 --2- 172.21.15.175:0/2719775407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9268155fb0 0x7f9268159920 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9260009e60 tx=0x7f9260009780 comp rx=0 tx=0).stop 2023-10-10T18:57:09.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.588+0000 7f926f135700 1 -- 172.21.15.175:0/2719775407 shutdown_connections 2023-10-10T18:57:09.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.588+0000 7f926f135700 1 --2- 172.21.15.175:0/2719775407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9268155fb0 0x7f9268159920 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.588+0000 7f926f135700 1 -- 172.21.15.175:0/2719775407 >> 172.21.15.175:0/2719775407 conn(0x7f9268063230 msgr2=0x7f9268063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:09.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.588+0000 7f926f135700 1 -- 172.21.15.175:0/2719775407 shutdown_connections 2023-10-10T18:57:09.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.588+0000 7f926f135700 1 -- 172.21.15.175:0/2719775407 wait complete. 2023-10-10T18:57:09.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.589+0000 7f926f135700 1 Processor -- start 2023-10-10T18:57:09.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.589+0000 7f926f135700 1 -- start start 2023-10-10T18:57:09.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926f135700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92680b1fc0 0x7f92680cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926f135700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92680b2390 0x7f92681666e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926f135700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268155fb0 0x7f926814dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926f135700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926814d710 con 0x7f92680b2390 2023-10-10T18:57:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926f135700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680cb300 con 0x7f9268155fb0 2023-10-10T18:57:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926f135700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92680cb5e0 con 0x7f92680b1fc0 2023-10-10T18:57:09.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92680b2390 0x7f92681666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92680b2390 0x7f92681666e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58408/0 (socket says 172.21.15.175:58408) 2023-10-10T18:57:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 -- 172.21.15.175:0/795176259 learned_addr learned my addr 172.21.15.175:0/795176259 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926d6d2700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268155fb0 0x7f926814dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926ced1700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92680b1fc0 0x7f92680cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 -- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92680b1fc0 msgr2=0x7f92680cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92680b1fc0 0x7f92680cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 -- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268155fb0 msgr2=0x7f926814dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268155fb0 0x7f926814dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 -- 172.21.15.175:0/795176259 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9260009a20 con 0x7f92680b2390 2023-10-10T18:57:09.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9267fff700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92680b2390 0x7f92681666e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f925400c590 tx=0x7f925400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:09.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9264ff9700 1 -- 172.21.15.175:0/795176259 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9254011590 con 0x7f92680b2390 2023-10-10T18:57:09.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9264ff9700 1 -- 172.21.15.175:0/795176259 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9254011710 con 0x7f92680b2390 2023-10-10T18:57:09.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92680cb8c0 con 0x7f92680b2390 2023-10-10T18:57:09.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f9264ff9700 1 -- 172.21.15.175:0/795176259 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9254011ad0 con 0x7f92680b2390 2023-10-10T18:57:09.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.590+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926814d3a0 con 0x7f92680b2390 2023-10-10T18:57:09.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.591+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9268067570 con 0x7f92680b2390 2023-10-10T18:57:09.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.592+0000 7f9264ff9700 1 -- 172.21.15.175:0/795176259 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f925401b3b0 con 0x7f92680b2390 2023-10-10T18:57:09.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.592+0000 7f9264ff9700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f924c066f30 0x7f924c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:09.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.592+0000 7f9264ff9700 1 -- 172.21.15.175:0/795176259 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f9254083be0 con 0x7f92680b2390 2023-10-10T18:57:09.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.592+0000 7f926ced1700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f924c066f30 0x7f924c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:09.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.593+0000 7f926ced1700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f924c066f30 0x7f924c0693e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f925c007fd0 tx=0x7f925c007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:09.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.596+0000 7f9264ff9700 1 -- 172.21.15.175:0/795176259 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9254050bd0 con 0x7f92680b2390 2023-10-10T18:57:09.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.828+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9268068da0 con 0x7f92680b2390 2023-10-10T18:57:09.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.828+0000 7f9264ff9700 1 -- 172.21.15.175:0/795176259 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5439 (secure 0 0 0) 0x7f9254017070 con 0x7f92680b2390 2023-10-10T18:57:09.829 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:09.829 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:08.527691+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:09.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f924c066f30 msgr2=0x7f924c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f924c066f30 0x7f924c0693e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f925c007fd0 tx=0x7f925c007d30 comp rx=0 tx=0).stop 2023-10-10T18:57:09.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92680b2390 msgr2=0x7f92681666e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92680b2390 0x7f92681666e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f925400c590 tx=0x7f925400c8a0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 shutdown_connections 2023-10-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f924c066f30 0x7f924c0693e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268155fb0 0x7f926814dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92680b1fc0 0x7f92680cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 --2- 172.21.15.175:0/795176259 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92680b2390 0x7f92681666e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:09.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 >> 172.21.15.175:0/795176259 conn(0x7f9268063230 msgr2=0x7f92680c7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:09.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 shutdown_connections 2023-10-10T18:57:09.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:09.831+0000 7f926f135700 1 -- 172.21.15.175:0/795176259 wait complete. 2023-10-10T18:57:09.834 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 94 2023-10-10T18:57:09.843 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 31 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:57:08.527691+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 5332, "mds_2": 5367, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:57:11.845 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:11.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.972+0000 7f0fd759e700 1 Processor -- start 2023-10-10T18:57:11.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.972+0000 7f0fd759e700 1 -- start start 2023-10-10T18:57:11.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd759e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 0x7f0fd8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:11.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd759e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0fd808a100 0x7f0fd80c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:11.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd759e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0fd80cd040 0x7f0fd80c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:11.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd759e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd80c5600 con 0x7f0fd80cd040 2023-10-10T18:57:11.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd759e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd8067fa0 con 0x7f0fd808a100 2023-10-10T18:57:11.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd759e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8156b70 con 0x7f0fd814fd90 2023-10-10T18:57:11.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0fd80cd040 0x7f0fd80c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:11.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fcdd9b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0fd808a100 0x7f0fd80c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:11.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0fd80cd040 0x7f0fd80c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58422/0 (socket says 172.21.15.175:58422) 2023-10-10T18:57:11.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 -- 172.21.15.175:0/4057652737 learned_addr learned my addr 172.21.15.175:0/4057652737 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:11.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd659c700 1 --2- 172.21.15.175:0/4057652737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 0x7f0fd8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:11.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 -- 172.21.15.175:0/4057652737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 msgr2=0x7f0fd8159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:11.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 --2- 172.21.15.175:0/4057652737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 0x7f0fd8159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:11.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 -- 172.21.15.175:0/4057652737 >> v2:172.21.15.191:3300/0 conn(0x7f0fd808a100 msgr2=0x7f0fd80c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:11.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 --2- 172.21.15.175:0/4057652737 >> v2:172.21.15.191:3300/0 conn(0x7f0fd808a100 0x7f0fd80c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:11.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 -- 172.21.15.175:0/4057652737 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd8150160 con 0x7f0fd80cd040 2023-10-10T18:57:11.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd6d9d700 1 --2- 172.21.15.175:0/4057652737 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd80c63f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80898c0 tx=0x7f0fd0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f414e46d406199 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:11.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.973+0000 7f0fd5d9b700 1 -- 172.21.15.175:0/4057652737 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000e070 con 0x7f0fd80cd040 2023-10-10T18:57:11.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd5d9b700 1 -- 172.21.15.175:0/4057652737 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd000b970 con 0x7f0fd80cd040 2023-10-10T18:57:11.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd5d9b700 1 -- 172.21.15.175:0/4057652737 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00092e0 con 0x7f0fd80cd040 2023-10-10T18:57:11.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 -- 172.21.15.175:0/4057652737 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 msgr2=0x7f0fd80c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:11.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 --2- 172.21.15.175:0/4057652737 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd80c63f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80898c0 tx=0x7f0fd0009840 comp rx=0 tx=0).stop 2023-10-10T18:57:11.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 -- 172.21.15.175:0/4057652737 shutdown_connections 2023-10-10T18:57:11.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 --2- 172.21.15.175:0/4057652737 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd80c63f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:11.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 --2- 172.21.15.175:0/4057652737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 0x7f0fd8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:11.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 --2- 172.21.15.175:0/4057652737 >> v2:172.21.15.191:3300/0 conn(0x7f0fd808a100 0x7f0fd80c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:11.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 -- 172.21.15.175:0/4057652737 >> 172.21.15.175:0/4057652737 conn(0x7f0fd8063d00 msgr2=0x7f0fd80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:11.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 -- 172.21.15.175:0/4057652737 shutdown_connections 2023-10-10T18:57:11.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.974+0000 7f0fd759e700 1 -- 172.21.15.175:0/4057652737 wait complete. 2023-10-10T18:57:11.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.975+0000 7f0fd759e700 1 Processor -- start 2023-10-10T18:57:11.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.975+0000 7f0fd759e700 1 -- start start 2023-10-10T18:57:11.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fd759e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd808a100 0x7f0fd80c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:11.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fd759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd81504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:11.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fd759e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 0x7f0fd815bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:11.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fd759e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8151700 con 0x7f0fd80cd040 2023-10-10T18:57:11.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fd759e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8165e90 con 0x7f0fd808a100 2023-10-10T18:57:11.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fd759e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd81520e0 con 0x7f0fd814fd90 2023-10-10T18:57:11.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fcdd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd81504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:11.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fcdd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd81504b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58432/0 (socket says 172.21.15.175:58432) 2023-10-10T18:57:11.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fcdd9b700 1 -- 172.21.15.175:0/1447451511 learned_addr learned my addr 172.21.15.175:0/1447451511 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:11.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fd659c700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd808a100 0x7f0fd80c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:11.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fd6d9d700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 0x7f0fd815bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:11.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.976+0000 7f0fcdd9b700 1 -- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 msgr2=0x7f0fd815bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:11.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fcdd9b700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 0x7f0fd815bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:11.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fcdd9b700 1 -- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd808a100 msgr2=0x7f0fd80c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:11.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fcdd9b700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd808a100 0x7f0fd80c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:11.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fcdd9b700 1 -- 172.21.15.175:0/1447451511 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd0009ae0 con 0x7f0fd80cd040 2023-10-10T18:57:11.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fcdd9b700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd81504b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000e930 tx=0x7f0fc00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:11.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fceffd700 1 -- 172.21.15.175:0/1447451511 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0013680 con 0x7f0fd80cd040 2023-10-10T18:57:11.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fceffd700 1 -- 172.21.15.175:0/1447451511 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc00137e0 con 0x7f0fd80cd040 2023-10-10T18:57:11.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fceffd700 1 -- 172.21.15.175:0/1447451511 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0013b80 con 0x7f0fd80cd040 2023-10-10T18:57:11.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd8151b20 con 0x7f0fd80cd040 2023-10-10T18:57:11.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.977+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd8165840 con 0x7f0fd80cd040 2023-10-10T18:57:11.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.978+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd8067570 con 0x7f0fd80cd040 2023-10-10T18:57:11.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.978+0000 7f0fceffd700 1 -- 172.21.15.175:0/1447451511 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0fc000c3e0 con 0x7f0fd80cd040 2023-10-10T18:57:11.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.979+0000 7f0fceffd700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fb8066f30 0x7f0fb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:11.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.979+0000 7f0fceffd700 1 -- 172.21.15.175:0/1447451511 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f0fc0084860 con 0x7f0fd80cd040 2023-10-10T18:57:11.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.979+0000 7f0fd659c700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fb8066f30 0x7f0fb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:11.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.979+0000 7f0fd659c700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fb8066f30 0x7f0fb80693e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009db0 tx=0x7f0fc80091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:11.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:11.983+0000 7f0fceffd700 1 -- 172.21.15.175:0/1447451511 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0fc0051850 con 0x7f0fd80cd040 2023-10-10T18:57:12.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.210+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fd8155f80 con 0x7f0fd80cd040 2023-10-10T18:57:12.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.210+0000 7f0fceffd700 1 -- 172.21.15.175:0/1447451511 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5439 (secure 0 0 0) 0x7f0fc0054e70 con 0x7f0fd80cd040 2023-10-10T18:57:12.210 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:12.211 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:08.527691+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:12.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fb8066f30 msgr2=0x7f0fb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:12.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fb8066f30 0x7f0fb80693e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009db0 tx=0x7f0fc80091a0 comp rx=0 tx=0).stop 2023-10-10T18:57:12.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 msgr2=0x7f0fd81504b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:12.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd81504b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000e930 tx=0x7f0fc00091a0 comp rx=0 tx=0).stop 2023-10-10T18:57:12.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 shutdown_connections 2023-10-10T18:57:12.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fb8066f30 0x7f0fb80693e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:12.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd814fd90 0x7f0fd815bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:12.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd808a100 0x7f0fd80c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:12.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 --2- 172.21.15.175:0/1447451511 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fd80cd040 0x7f0fd81504b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:12.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 >> 172.21.15.175:0/1447451511 conn(0x7f0fd8063d00 msgr2=0x7f0fd8154be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:12.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 shutdown_connections 2023-10-10T18:57:12.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:12.212+0000 7f0fd759e700 1 -- 172.21.15.175:0/1447451511 wait complete. 2023-10-10T18:57:12.215 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 94 2023-10-10T18:57:12.224 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:57:14.225 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:14.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.343+0000 7f7b17fff700 1 Processor -- start 2023-10-10T18:57:14.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.343+0000 7f7b17fff700 1 -- start start 2023-10-10T18:57:14.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b17fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7b180b0240 0x7f7b1806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:14.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b17fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b1806bda0 0x7f7b180c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:14.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b17fff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1806a8c0 con 0x7f7b181595a0 2023-10-10T18:57:14.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b17fff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b18159fb0 con 0x7f7b180b0240 2023-10-10T18:57:14.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b17fff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b180c5ab0 con 0x7f7b1806bda0 2023-10-10T18:57:14.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b167fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7b180b0240 0x7f7b1806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:14.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b167fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7b180b0240 0x7f7b1806dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58446/0 (socket says 172.21.15.175:58446) 2023-10-10T18:57:14.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b16ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7b181595a0 0x7f7b180afb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39572/0 (socket says 172.21.15.175:39572) 2023-10-10T18:57:14.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b167fc700 1 -- 172.21.15.175:0/4015319475 learned_addr learned my addr 172.21.15.175:0/4015319475 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:14.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b177fe700 1 --2- 172.21.15.175:0/4015319475 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b1806bda0 0x7f7b180c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:14.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b167fc700 1 -- 172.21.15.175:0/4015319475 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b1806bda0 msgr2=0x7f7b180c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:14.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b167fc700 1 --2- 172.21.15.175:0/4015319475 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b1806bda0 0x7f7b180c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b167fc700 1 -- 172.21.15.175:0/4015319475 >> v1:172.21.15.175:6789/0 conn(0x7f7b181595a0 legacy=0x7f7b180afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:14.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b15ffb700 1 -- 172.21.15.175:0/4015319475 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848752248 0 0) 0x7f7b1806a8c0 con 0x7f7b181595a0 2023-10-10T18:57:14.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.344+0000 7f7b167fc700 1 -- 172.21.15.175:0/4015319475 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b180c57d0 con 0x7f7b180b0240 2023-10-10T18:57:14.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b167fc700 1 --2- 172.21.15.175:0/4015319475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b180b0240 0x7f7b1806dc90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c0099b0 tx=0x7f7b0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b8e8c43a93a68a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:14.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b15ffb700 1 -- 172.21.15.175:0/4015319475 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c009ea0 con 0x7f7b180b0240 2023-10-10T18:57:14.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b15ffb700 1 -- 172.21.15.175:0/4015319475 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c00b8f0 con 0x7f7b180b0240 2023-10-10T18:57:14.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b15ffb700 1 -- 172.21.15.175:0/4015319475 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c00bc90 con 0x7f7b180b0240 2023-10-10T18:57:14.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b17fff700 1 -- 172.21.15.175:0/4015319475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b180b0240 msgr2=0x7f7b1806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:14.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b17fff700 1 --2- 172.21.15.175:0/4015319475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b180b0240 0x7f7b1806dc90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c0099b0 tx=0x7f7b0c009250 comp rx=0 tx=0).stop 2023-10-10T18:57:14.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b17fff700 1 -- 172.21.15.175:0/4015319475 shutdown_connections 2023-10-10T18:57:14.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b17fff700 1 --2- 172.21.15.175:0/4015319475 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b1806bda0 0x7f7b180c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b17fff700 1 --2- 172.21.15.175:0/4015319475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b180b0240 0x7f7b1806dc90 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b17fff700 1 -- 172.21.15.175:0/4015319475 >> 172.21.15.175:0/4015319475 conn(0x7f7b18063cd0 msgr2=0x7f7b1806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:14.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b17fff700 1 -- 172.21.15.175:0/4015319475 shutdown_connections 2023-10-10T18:57:14.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.345+0000 7f7b17fff700 1 -- 172.21.15.175:0/4015319475 wait complete. 2023-10-10T18:57:14.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.347+0000 7f7b17fff700 1 Processor -- start 2023-10-10T18:57:14.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.347+0000 7f7b17fff700 1 -- start start 2023-10-10T18:57:14.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.348+0000 7f7b17fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b1806bda0 0x7f7b18155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:14.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.348+0000 7f7b17fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b180b0240 0x7f7b181581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:14.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.348+0000 7f7b17fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b181595a0 0x7f7b181586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:14.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.348+0000 7f7b17fff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b181577a0 con 0x7f7b1806bda0 2023-10-10T18:57:14.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.348+0000 7f7b17fff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b18169d00 con 0x7f7b180b0240 2023-10-10T18:57:14.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.348+0000 7f7b17fff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b18157150 con 0x7f7b181595a0 2023-10-10T18:57:14.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.348+0000 7f7b16ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b1806bda0 0x7f7b18155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:14.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b16ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b1806bda0 0x7f7b18155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58448/0 (socket says 172.21.15.175:58448) 2023-10-10T18:57:14.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b16ffd700 1 -- 172.21.15.175:0/3781490587 learned_addr learned my addr 172.21.15.175:0/3781490587 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:14.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b167fc700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b180b0240 0x7f7b181581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:14.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b177fe700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b181595a0 0x7f7b181586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:14.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b16ffd700 1 -- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b181595a0 msgr2=0x7f7b181586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:14.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b16ffd700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b181595a0 0x7f7b181586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b16ffd700 1 -- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b180b0240 msgr2=0x7f7b181581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:14.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b16ffd700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b180b0240 0x7f7b181581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b16ffd700 1 -- 172.21.15.175:0/3781490587 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c0094f0 con 0x7f7b1806bda0 2023-10-10T18:57:14.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b167fc700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b180b0240 0x7f7b181581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:57:14.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b177fe700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b181595a0 0x7f7b181586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:57:14.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b16ffd700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b1806bda0 0x7f7b18155520 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800c5b0 tx=0x7f7b08001040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:14.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7afeffd700 1 -- 172.21.15.175:0/3781490587 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08005a70 con 0x7f7b1806bda0 2023-10-10T18:57:14.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7afeffd700 1 -- 172.21.15.175:0/3781490587 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b08005bd0 con 0x7f7b1806bda0 2023-10-10T18:57:14.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7afeffd700 1 -- 172.21.15.175:0/3781490587 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b080070f0 con 0x7f7b1806bda0 2023-10-10T18:57:14.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b18157430 con 0x7f7b1806bda0 2023-10-10T18:57:14.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.349+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b18156290 con 0x7f7b1806bda0 2023-10-10T18:57:14.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.350+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b18156fb0 con 0x7f7b1806bda0 2023-10-10T18:57:14.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.351+0000 7f7afeffd700 1 -- 172.21.15.175:0/3781490587 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7b08007460 con 0x7f7b1806bda0 2023-10-10T18:57:14.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.351+0000 7f7afeffd700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b04066f30 0x7f7b040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:14.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.351+0000 7f7afeffd700 1 -- 172.21.15.175:0/3781490587 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f7b08084290 con 0x7f7b1806bda0 2023-10-10T18:57:14.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.351+0000 7f7b167fc700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b04066f30 0x7f7b040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:14.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.352+0000 7f7b167fc700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b04066f30 0x7f7b040693e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c0099b0 tx=0x7f7b0c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:14.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.356+0000 7f7afeffd700 1 -- 172.21.15.175:0/3781490587 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b18156fb0 con 0x7f7b1806bda0 2023-10-10T18:57:14.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.582+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b18156fb0 con 0x7f7b1806bda0 2023-10-10T18:57:14.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.582+0000 7f7afeffd700 1 -- 172.21.15.175:0/3781490587 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5439 (secure 0 0 0) 0x7f7b18156fb0 con 0x7f7b1806bda0 2023-10-10T18:57:14.584 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:14.584 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:08.527691+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:14.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b04066f30 msgr2=0x7f7b040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:14.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b04066f30 0x7f7b040693e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c0099b0 tx=0x7f7b0c017000 comp rx=0 tx=0).stop 2023-10-10T18:57:14.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b1806bda0 msgr2=0x7f7b18155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:14.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b1806bda0 0x7f7b18155520 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800c5b0 tx=0x7f7b08001040 comp rx=0 tx=0).stop 2023-10-10T18:57:14.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 shutdown_connections 2023-10-10T18:57:14.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b04066f30 0x7f7b040693e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b181595a0 0x7f7b181586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b1806bda0 0x7f7b18155520 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 --2- 172.21.15.175:0/3781490587 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b180b0240 0x7f7b181581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:14.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 >> 172.21.15.175:0/3781490587 conn(0x7f7b18063cd0 msgr2=0x7f7b180bfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:14.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 shutdown_connections 2023-10-10T18:57:14.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:14.584+0000 7f7b17fff700 1 -- 172.21.15.175:0/3781490587 wait complete. 2023-10-10T18:57:14.588 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 94 2023-10-10T18:57:14.597 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:57:16.599 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:16.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.723+0000 7fac3a12e700 1 Processor -- start 2023-10-10T18:57:16.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac3a12e700 1 -- start start 2023-10-10T18:57:16.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac3a12e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fac340b2390 0x7fac34155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:16.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac3a12e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fac340aff60 0x7fac340b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:16.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac3a12e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac341574b0 con 0x7fac340b1fc0 2023-10-10T18:57:16.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac3a12e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac34150770 con 0x7fac340b2390 2023-10-10T18:57:16.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac3a12e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac340afad0 con 0x7fac340aff60 2023-10-10T18:57:16.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac32ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fac340b2390 0x7fac34155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:16.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac32ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fac340b2390 0x7fac34155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42666/0 (socket says 172.21.15.175:42666) 2023-10-10T18:57:16.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.724+0000 7fac32ffd700 1 -- 172.21.15.175:0/4148383693 learned_addr learned my addr 172.21.15.175:0/4148383693 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:16.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac33fff700 1 --2- 172.21.15.175:0/4148383693 >> v2:172.21.15.191:3300/0 conn(0x7fac340aff60 0x7fac340b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:16.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac32ffd700 1 -- 172.21.15.175:0/4148383693 >> v2:172.21.15.191:3300/0 conn(0x7fac340aff60 msgr2=0x7fac340b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:16.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac32ffd700 1 --2- 172.21.15.175:0/4148383693 >> v2:172.21.15.191:3300/0 conn(0x7fac340aff60 0x7fac340b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac32ffd700 1 -- 172.21.15.175:0/4148383693 >> v1:172.21.15.191:6789/0 conn(0x7fac340b1fc0 legacy=0x7fac34068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:16.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac32ffd700 1 -- 172.21.15.175:0/4148383693 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac340af4c0 con 0x7fac340b2390 2023-10-10T18:57:16.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac32ffd700 1 --2- 172.21.15.175:0/4148383693 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b2390 0x7fac34155fb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fac340641f0 tx=0x7fac28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81a0e3d0cd52542 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:16.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac327fc700 1 -- 172.21.15.175:0/4148383693 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2800b8b0 con 0x7fac340b2390 2023-10-10T18:57:16.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac327fc700 1 -- 172.21.15.175:0/4148383693 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2800ba10 con 0x7fac340b2390 2023-10-10T18:57:16.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.725+0000 7fac327fc700 1 -- 172.21.15.175:0/4148383693 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2800bdb0 con 0x7fac340b2390 2023-10-10T18:57:16.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.726+0000 7fac3a12e700 1 -- 172.21.15.175:0/4148383693 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b2390 msgr2=0x7fac34155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:16.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.726+0000 7fac3a12e700 1 --2- 172.21.15.175:0/4148383693 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b2390 0x7fac34155fb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fac340641f0 tx=0x7fac28009250 comp rx=0 tx=0).stop 2023-10-10T18:57:16.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.726+0000 7fac3a12e700 1 -- 172.21.15.175:0/4148383693 shutdown_connections 2023-10-10T18:57:16.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.726+0000 7fac3a12e700 1 --2- 172.21.15.175:0/4148383693 >> v2:172.21.15.191:3300/0 conn(0x7fac340aff60 0x7fac340b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.726+0000 7fac3a12e700 1 --2- 172.21.15.175:0/4148383693 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b2390 0x7fac34155fb0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.726+0000 7fac3a12e700 1 -- 172.21.15.175:0/4148383693 >> 172.21.15.175:0/4148383693 conn(0x7fac34063230 msgr2=0x7fac34063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:16.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.726+0000 7fac3a12e700 1 -- 172.21.15.175:0/4148383693 shutdown_connections 2023-10-10T18:57:16.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.726+0000 7fac3a12e700 1 -- 172.21.15.175:0/4148383693 wait complete. 2023-10-10T18:57:16.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.727+0000 7fac3a12e700 1 Processor -- start 2023-10-10T18:57:16.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.727+0000 7fac3a12e700 1 -- start start 2023-10-10T18:57:16.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.727+0000 7fac3a12e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac340aff60 0x7fac341547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:16.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac3a12e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b1fc0 0x7fac34154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:16.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac3a12e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac340b2390 0x7fac340cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:16.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac3a12e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac340c2f20 con 0x7fac340b1fc0 2023-10-10T18:57:16.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac3a12e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac34166c40 con 0x7fac340aff60 2023-10-10T18:57:16.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac3a12e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac340c3130 con 0x7fac340b2390 2023-10-10T18:57:16.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b1fc0 0x7fac34154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:16.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b1fc0 0x7fac34154d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42672/0 (socket says 172.21.15.175:42672) 2023-10-10T18:57:16.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 -- 172.21.15.175:0/3588597303 learned_addr learned my addr 172.21.15.175:0/3588597303 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:16.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac337fe700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac340aff60 0x7fac341547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:16.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac33fff700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac340b2390 0x7fac340cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:16.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 -- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac340b2390 msgr2=0x7fac340cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:16.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac340b2390 0x7fac340cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 -- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac340aff60 msgr2=0x7fac341547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:16.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac340aff60 0x7fac341547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 -- 172.21.15.175:0/3588597303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac280094f0 con 0x7fac340b1fc0 2023-10-10T18:57:16.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac32ffd700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b1fc0 0x7fac34154d10 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fac28009290 tx=0x7fac28016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:16.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac1b7fe700 1 -- 172.21.15.175:0/3588597303 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28011e60 con 0x7fac340b1fc0 2023-10-10T18:57:16.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac1b7fe700 1 -- 172.21.15.175:0/3588597303 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2801c670 con 0x7fac340b1fc0 2023-10-10T18:57:16.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac340c3410 con 0x7fac340b1fc0 2023-10-10T18:57:16.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac1b7fe700 1 -- 172.21.15.175:0/3588597303 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2801ca50 con 0x7fac340b1fc0 2023-10-10T18:57:16.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.728+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac341542b0 con 0x7fac340b1fc0 2023-10-10T18:57:16.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.729+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac341544a0 con 0x7fac340b1fc0 2023-10-10T18:57:16.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.730+0000 7fac1b7fe700 1 -- 172.21.15.175:0/3588597303 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fac2801a450 con 0x7fac340b1fc0 2023-10-10T18:57:16.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.730+0000 7fac1b7fe700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac20066f30 0x7fac200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:16.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.730+0000 7fac1b7fe700 1 -- 172.21.15.175:0/3588597303 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fac28086090 con 0x7fac340b1fc0 2023-10-10T18:57:16.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.731+0000 7fac337fe700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac20066f30 0x7fac200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:16.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.731+0000 7fac337fe700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac20066f30 0x7fac200693e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fac24007fd0 tx=0x7fac24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:16.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.735+0000 7fac1b7fe700 1 -- 172.21.15.175:0/3588597303 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac28053080 con 0x7fac340b1fc0 2023-10-10T18:57:16.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.960+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac34067570 con 0x7fac340b1fc0 2023-10-10T18:57:16.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.961+0000 7fac1b7fe700 1 -- 172.21.15.175:0/3588597303 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5441 (secure 0 0 0) 0x7fac280566a0 con 0x7fac340b1fc0 2023-10-10T18:57:16.962 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:16.962 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:14.591775+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:16.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac20066f30 msgr2=0x7fac200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:16.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac20066f30 0x7fac200693e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fac24007fd0 tx=0x7fac24009040 comp rx=0 tx=0).stop 2023-10-10T18:57:16.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b1fc0 msgr2=0x7fac34154d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:16.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b1fc0 0x7fac34154d10 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fac28009290 tx=0x7fac28016000 comp rx=0 tx=0).stop 2023-10-10T18:57:16.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 shutdown_connections 2023-10-10T18:57:16.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac20066f30 0x7fac200693e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac340b2390 0x7fac340cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac340aff60 0x7fac341547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 --2- 172.21.15.175:0/3588597303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac340b1fc0 0x7fac34154d10 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:16.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 >> 172.21.15.175:0/3588597303 conn(0x7fac34063230 msgr2=0x7fac3414fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:16.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 shutdown_connections 2023-10-10T18:57:16.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:16.963+0000 7fac3a12e700 1 -- 172.21.15.175:0/3588597303 wait complete. 2023-10-10T18:57:16.967 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 95 2023-10-10T18:57:16.976 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:57:18.976 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:19.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.103+0000 7f6e080f2700 1 Processor -- start 2023-10-10T18:57:19.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.104+0000 7f6e080f2700 1 -- start start 2023-10-10T18:57:19.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.104+0000 7f6e080f2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6e000b1fc0 0x7f6e00155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:19.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.104+0000 7f6e080f2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e000b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:19.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.104+0000 7f6e080f2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e001574b0 con 0x7f6e000b2390 2023-10-10T18:57:19.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.104+0000 7f6e080f2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e00150770 con 0x7f6e000b1fc0 2023-10-10T18:57:19.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.104+0000 7f6e080f2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000afad0 con 0x7f6e000aff60 2023-10-10T18:57:19.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.104+0000 7f6e0568d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6e000b2390 0x7f6e00068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60212/0 (socket says 172.21.15.175:60212) 2023-10-10T18:57:19.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.104+0000 7f6e0568d700 1 -- 172.21.15.175:0/3656003875 learned_addr learned my addr 172.21.15.175:0/3656003875 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:57:19.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e05e8e700 1 --2- 172.21.15.175:0/3656003875 >> v2:172.21.15.191:3300/0 conn(0x7f6e000b1fc0 0x7f6e00155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:19.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e0668f700 1 --2- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e000b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:19.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e04e8c700 1 -- 172.21.15.175:0/3656003875 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103836798 0 0) 0x7f6e001574b0 con 0x7f6e000b2390 2023-10-10T18:57:19.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e04e8c700 1 -- 172.21.15.175:0/3656003875 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de8003570 con 0x7f6e000b2390 2023-10-10T18:57:19.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e05e8e700 1 -- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 msgr2=0x7f6e000b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:19.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e05e8e700 1 --2- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e000b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e05e8e700 1 -- 172.21.15.175:0/3656003875 >> v1:172.21.15.175:6789/0 conn(0x7f6e000b2390 legacy=0x7f6e00068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:19.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e05e8e700 1 -- 172.21.15.175:0/3656003875 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e000af4c0 con 0x7f6e000b1fc0 2023-10-10T18:57:19.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e0668f700 1 --2- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e000b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:57:19.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e05e8e700 1 --2- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b1fc0 0x7f6e00155fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6df00098e0 tx=0x7f6df00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a0519eb21bf6a1b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:19.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e04e8c700 1 -- 172.21.15.175:0/3656003875 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0011630 con 0x7f6e000b1fc0 2023-10-10T18:57:19.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e04e8c700 1 -- 172.21.15.175:0/3656003875 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df0011790 con 0x7f6e000b1fc0 2023-10-10T18:57:19.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.105+0000 7f6e04e8c700 1 -- 172.21.15.175:0/3656003875 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0011b30 con 0x7f6e000b1fc0 2023-10-10T18:57:19.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.106+0000 7f6e080f2700 1 -- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b1fc0 msgr2=0x7f6e00155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:19.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.106+0000 7f6e080f2700 1 --2- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b1fc0 0x7f6e00155fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6df00098e0 tx=0x7f6df00091a0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.106+0000 7f6e080f2700 1 -- 172.21.15.175:0/3656003875 shutdown_connections 2023-10-10T18:57:19.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.106+0000 7f6e080f2700 1 --2- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e000b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.106+0000 7f6e080f2700 1 --2- 172.21.15.175:0/3656003875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b1fc0 0x7f6e00155fb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.106+0000 7f6e080f2700 1 -- 172.21.15.175:0/3656003875 >> 172.21.15.175:0/3656003875 conn(0x7f6e00063230 msgr2=0x7f6e00063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:19.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.106+0000 7f6e080f2700 1 -- 172.21.15.175:0/3656003875 shutdown_connections 2023-10-10T18:57:19.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.106+0000 7f6e080f2700 1 -- 172.21.15.175:0/3656003875 wait complete. 2023-10-10T18:57:19.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.107+0000 7f6e080f2700 1 Processor -- start 2023-10-10T18:57:19.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.107+0000 7f6e080f2700 1 -- start start 2023-10-10T18:57:19.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e080f2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e001599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:19.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e080f2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e000b1fc0 0x7f6e00150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:19.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e080f2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b2390 0x7f6e0014d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:19.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e080f2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000cd330 con 0x7f6e000b1fc0 2023-10-10T18:57:19.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e080f2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e00158fe0 con 0x7f6e000b2390 2023-10-10T18:57:19.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e080f2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000c4440 con 0x7f6e000aff60 2023-10-10T18:57:19.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e000b1fc0 0x7f6e00150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:19.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e000b1fc0 0x7f6e00150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42682/0 (socket says 172.21.15.175:42682) 2023-10-10T18:57:19.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 -- 172.21.15.175:0/1870076885 learned_addr learned my addr 172.21.15.175:0/1870076885 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:19.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0668f700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b2390 0x7f6e0014d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:19.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e05e8e700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e001599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:19.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 -- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 msgr2=0x7f6e001599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:19.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e001599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 -- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b2390 msgr2=0x7f6e0014d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:19.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b2390 0x7f6e0014d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 -- 172.21.15.175:0/1870076885 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df8009a20 con 0x7f6e000b1fc0 2023-10-10T18:57:19.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e0568d700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e000b1fc0 0x7f6e00150fa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c590 tx=0x7f6df4007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:19.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6dfdffb700 1 -- 172.21.15.175:0/1870076885 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4005b30 con 0x7f6e000b1fc0 2023-10-10T18:57:19.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6dfdffb700 1 -- 172.21.15.175:0/1870076885 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df4005c90 con 0x7f6e000b1fc0 2023-10-10T18:57:19.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df0009420 con 0x7f6e000b1fc0 2023-10-10T18:57:19.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.108+0000 7f6dfdffb700 1 -- 172.21.15.175:0/1870076885 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4007320 con 0x7f6e000b1fc0 2023-10-10T18:57:19.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.109+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e00158670 con 0x7f6e000b1fc0 2023-10-10T18:57:19.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.110+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e00067570 con 0x7f6e000b1fc0 2023-10-10T18:57:19.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.110+0000 7f6dfdffb700 1 -- 172.21.15.175:0/1870076885 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6df401b3b0 con 0x7f6e000b1fc0 2023-10-10T18:57:19.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.110+0000 7f6dfdffb700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dec066f30 0x7f6dec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:19.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.111+0000 7f6dfdffb700 1 -- 172.21.15.175:0/1870076885 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f6df4083cc0 con 0x7f6e000b1fc0 2023-10-10T18:57:19.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.111+0000 7f6e05e8e700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dec066f30 0x7f6dec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:19.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.111+0000 7f6e05e8e700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dec066f30 0x7f6dec0693e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6df00091e0 tx=0x7f6df0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:19.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.115+0000 7f6dfdffb700 1 -- 172.21.15.175:0/1870076885 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6df408a080 con 0x7f6e000b1fc0 2023-10-10T18:57:19.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.340+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e00068da0 con 0x7f6e000b1fc0 2023-10-10T18:57:19.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.341+0000 7f6dfdffb700 1 -- 172.21.15.175:0/1870076885 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5441 (secure 0 0 0) 0x7f6df4017020 con 0x7f6e000b1fc0 2023-10-10T18:57:19.342 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:19.343 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:14.591775+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:19.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dec066f30 msgr2=0x7f6dec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dec066f30 0x7f6dec0693e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6df00091e0 tx=0x7f6df0017000 comp rx=0 tx=0).stop 2023-10-10T18:57:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e000b1fc0 msgr2=0x7f6e00150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e000b1fc0 0x7f6e00150fa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c590 tx=0x7f6df4007e00 comp rx=0 tx=0).stop 2023-10-10T18:57:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 shutdown_connections 2023-10-10T18:57:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dec066f30 0x7f6dec0693e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e000b2390 0x7f6e0014d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e000aff60 0x7f6e001599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 --2- 172.21.15.175:0/1870076885 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e000b1fc0 0x7f6e00150fa0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:19.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 >> 172.21.15.175:0/1870076885 conn(0x7f6e00063230 msgr2=0x7f6e0014fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:19.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 shutdown_connections 2023-10-10T18:57:19.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:19.343+0000 7f6e080f2700 1 -- 172.21.15.175:0/1870076885 wait complete. 2023-10-10T18:57:19.347 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 95 2023-10-10T18:57:19.356 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:57:20.000 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:57:20.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:57:21.357 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:21.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.509+0000 7f8b598f8700 1 Processor -- start 2023-10-10T18:57:21.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.509+0000 7f8b598f8700 1 -- start start 2023-10-10T18:57:21.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.509+0000 7f8b598f8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b540b2390 0x7f8b54155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:21.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.509+0000 7f8b598f8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8b540aff60 0x7f8b540b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:21.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.509+0000 7f8b598f8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b541574b0 con 0x7f8b540b1fc0 2023-10-10T18:57:21.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.509+0000 7f8b598f8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b54150770 con 0x7f8b540b2390 2023-10-10T18:57:21.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.509+0000 7f8b598f8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b540afad0 con 0x7f8b540aff60 2023-10-10T18:57:21.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b4a7fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b540b2390 0x7f8b54155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:21.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b4a7fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b540b2390 0x7f8b54155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42690/0 (socket says 172.21.15.175:42690) 2023-10-10T18:57:21.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b4a7fc700 1 -- 172.21.15.175:0/602439231 learned_addr learned my addr 172.21.15.175:0/602439231 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:21.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b537fe700 1 --2- 172.21.15.175:0/602439231 >> v2:172.21.15.191:3300/0 conn(0x7f8b540aff60 0x7f8b540b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:21.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b4a7fc700 1 -- 172.21.15.175:0/602439231 >> v2:172.21.15.191:3300/0 conn(0x7f8b540aff60 msgr2=0x7f8b540b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:21.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b4a7fc700 1 --2- 172.21.15.175:0/602439231 >> v2:172.21.15.191:3300/0 conn(0x7f8b540aff60 0x7f8b540b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b4a7fc700 1 -- 172.21.15.175:0/602439231 >> v1:172.21.15.191:6789/0 conn(0x7f8b540b1fc0 legacy=0x7f8b54068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:21.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b4a7fc700 1 -- 172.21.15.175:0/602439231 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540af4c0 con 0x7f8b540b2390 2023-10-10T18:57:21.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b4a7fc700 1 --2- 172.21.15.175:0/602439231 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540b2390 0x7f8b54155fb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8b540641f0 tx=0x7f8b3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42432588039ebbe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:21.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b527fc700 1 -- 172.21.15.175:0/602439231 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c00b8b0 con 0x7f8b540b2390 2023-10-10T18:57:21.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b527fc700 1 -- 172.21.15.175:0/602439231 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c00ba10 con 0x7f8b540b2390 2023-10-10T18:57:21.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.510+0000 7f8b527fc700 1 -- 172.21.15.175:0/602439231 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c00bdb0 con 0x7f8b540b2390 2023-10-10T18:57:21.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.511+0000 7f8b598f8700 1 -- 172.21.15.175:0/602439231 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540b2390 msgr2=0x7f8b54155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:21.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.511+0000 7f8b598f8700 1 --2- 172.21.15.175:0/602439231 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540b2390 0x7f8b54155fb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8b540641f0 tx=0x7f8b3c009250 comp rx=0 tx=0).stop 2023-10-10T18:57:21.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.511+0000 7f8b598f8700 1 -- 172.21.15.175:0/602439231 shutdown_connections 2023-10-10T18:57:21.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.511+0000 7f8b598f8700 1 --2- 172.21.15.175:0/602439231 >> v2:172.21.15.191:3300/0 conn(0x7f8b540aff60 0x7f8b540b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.511+0000 7f8b598f8700 1 --2- 172.21.15.175:0/602439231 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540b2390 0x7f8b54155fb0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.511+0000 7f8b598f8700 1 -- 172.21.15.175:0/602439231 >> 172.21.15.175:0/602439231 conn(0x7f8b54063230 msgr2=0x7f8b54063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:21.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.511+0000 7f8b598f8700 1 -- 172.21.15.175:0/602439231 shutdown_connections 2023-10-10T18:57:21.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.511+0000 7f8b598f8700 1 -- 172.21.15.175:0/602439231 wait complete. 2023-10-10T18:57:21.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.512+0000 7f8b598f8700 1 Processor -- start 2023-10-10T18:57:21.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.512+0000 7f8b598f8700 1 -- start start 2023-10-10T18:57:21.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b598f8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540aff60 0x7f8b540ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:21.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b598f8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b540b1fc0 0x7f8b54169660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:21.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b598f8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b540b2390 0x7f8b5414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:21.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b598f8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540cd7a0 con 0x7f8b540aff60 2023-10-10T18:57:21.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b598f8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540c3870 con 0x7f8b540b2390 2023-10-10T18:57:21.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b598f8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540cd140 con 0x7f8b540b1fc0 2023-10-10T18:57:21.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540aff60 0x7f8b540ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:21.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540aff60 0x7f8b540ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42702/0 (socket says 172.21.15.175:42702) 2023-10-10T18:57:21.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 -- 172.21.15.175:0/3670159332 learned_addr learned my addr 172.21.15.175:0/3670159332 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:21.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b537fe700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b540b2390 0x7f8b5414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:21.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b4a7fc700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b540b1fc0 0x7f8b54169660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:21.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 -- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b540b1fc0 msgr2=0x7f8b54169660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:21.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b540b1fc0 0x7f8b54169660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 -- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b540b2390 msgr2=0x7f8b5414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:21.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b540b2390 0x7f8b5414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 -- 172.21.15.175:0/3670159332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0094f0 con 0x7f8b540aff60 2023-10-10T18:57:21.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b52ffd700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540aff60 0x7f8b540ccc00 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b4400c5d0 tx=0x7f8b44007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:21.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3670159332 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44011590 con 0x7f8b540aff60 2023-10-10T18:57:21.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3670159332 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b44011710 con 0x7f8b540aff60 2023-10-10T18:57:21.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5416a050 con 0x7f8b540aff60 2023-10-10T18:57:21.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3670159332 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44011ad0 con 0x7f8b540aff60 2023-10-10T18:57:21.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.513+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b540cc400 con 0x7f8b540aff60 2023-10-10T18:57:21.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.514+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b540cbf80 con 0x7f8b540aff60 2023-10-10T18:57:21.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.515+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3670159332 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8b4401c460 con 0x7f8b540aff60 2023-10-10T18:57:21.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.515+0000 7f8b4b7fe700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b34066f30 0x7f8b340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:21.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.515+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3670159332 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f8b44084520 con 0x7f8b540aff60 2023-10-10T18:57:21.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.515+0000 7f8b4a7fc700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b34066f30 0x7f8b340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:21.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.516+0000 7f8b4a7fc700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b34066f30 0x7f8b340693e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8b540641f0 tx=0x7f8b3c016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:21.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.519+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3670159332 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b540cbf80 con 0x7f8b540aff60 2023-10-10T18:57:21.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.757+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b540cbf80 con 0x7f8b540aff60 2023-10-10T18:57:21.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.758+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3670159332 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5443 (secure 0 0 0) 0x7f8b540cbf80 con 0x7f8b540aff60 2023-10-10T18:57:21.760 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:21.760 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:20.611209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:21.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b34066f30 msgr2=0x7f8b340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:21.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b34066f30 0x7f8b340693e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8b540641f0 tx=0x7f8b3c016000 comp rx=0 tx=0).stop 2023-10-10T18:57:21.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540aff60 msgr2=0x7f8b540ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:21.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540aff60 0x7f8b540ccc00 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b4400c5d0 tx=0x7f8b44007f60 comp rx=0 tx=0).stop 2023-10-10T18:57:21.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 shutdown_connections 2023-10-10T18:57:21.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b34066f30 0x7f8b340693e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b540b2390 0x7f8b5414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b540aff60 0x7f8b540ccc00 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 --2- 172.21.15.175:0/3670159332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b540b1fc0 0x7f8b54169660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:21.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 >> 172.21.15.175:0/3670159332 conn(0x7f8b54063230 msgr2=0x7f8b5414fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:21.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 shutdown_connections 2023-10-10T18:57:21.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:21.760+0000 7f8b598f8700 1 -- 172.21.15.175:0/3670159332 wait complete. 2023-10-10T18:57:21.764 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 96 2023-10-10T18:57:21.774 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 96, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 96, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4760": { "addr": "172.21.15.191:6836/3199867402", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 3199867402, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 3199867402, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 4760, "incarnation": 34, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 27 }, "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 31 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:reconnect", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:57:20.611209+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 5332, "mds_2": 5367, "mds_3": 5196, "mds_4": 4760 } } } ], "standbys": [] } 2023-10-10T18:57:23.775 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:23.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.901+0000 7f5236de3700 1 Processor -- start 2023-10-10T18:57:23.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.901+0000 7f5236de3700 1 -- start start 2023-10-10T18:57:23.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.902+0000 7f5236de3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52300b1fc0 0x7f5230155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:23.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.902+0000 7f5236de3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52301574b0 con 0x7f52300aff60 2023-10-10T18:57:23.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.902+0000 7f5236de3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52300afad0 con 0x7f52300b2390 2023-10-10T18:57:23.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.902+0000 7f5236de3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5230150770 con 0x7f52300b1fc0 2023-10-10T18:57:23.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.902+0000 7f5234b7f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f52300b1fc0 0x7f5230155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:23.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.902+0000 7f5235380700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f52300aff60 0x7f5230159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60224/0 (socket says 172.21.15.175:60224) 2023-10-10T18:57:23.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.902+0000 7f5235380700 1 -- 172.21.15.175:0/2293336233 learned_addr learned my addr 172.21.15.175:0/2293336233 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:57:23.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5234b7f700 1 -- 172.21.15.175:0/2293336233 >> v1:172.21.15.191:6789/0 conn(0x7f52300b2390 legacy=0x7f5230068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:23.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5234b7f700 1 -- 172.21.15.175:0/2293336233 >> v1:172.21.15.175:6789/0 conn(0x7f52300aff60 legacy=0x7f5230159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:23.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5234b7f700 1 -- 172.21.15.175:0/2293336233 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52300af4c0 con 0x7f52300b1fc0 2023-10-10T18:57:23.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f52277fe700 1 -- 172.21.15.175:0/2293336233 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758393003 0 0) 0x7f52300afad0 con 0x7f52300b2390 2023-10-10T18:57:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5234b7f700 1 --2- 172.21.15.175:0/2293336233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b1fc0 0x7f5230155fb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f522c009e60 tx=0x7f522c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f4302be5403b0a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f52277fe700 1 -- 172.21.15.175:0/2293336233 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522c00e070 con 0x7f52300b1fc0 2023-10-10T18:57:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f52277fe700 1 -- 172.21.15.175:0/2293336233 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522c00bb30 con 0x7f52300b1fc0 2023-10-10T18:57:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f52277fe700 1 -- 172.21.15.175:0/2293336233 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522c012430 con 0x7f52300b1fc0 2023-10-10T18:57:23.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5236de3700 1 -- 172.21.15.175:0/2293336233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b1fc0 msgr2=0x7f5230155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5236de3700 1 --2- 172.21.15.175:0/2293336233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b1fc0 0x7f5230155fb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f522c009e60 tx=0x7f522c009780 comp rx=0 tx=0).stop 2023-10-10T18:57:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5236de3700 1 -- 172.21.15.175:0/2293336233 shutdown_connections 2023-10-10T18:57:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5236de3700 1 --2- 172.21.15.175:0/2293336233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b1fc0 0x7f5230155fb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.903+0000 7f5236de3700 1 -- 172.21.15.175:0/2293336233 >> 172.21.15.175:0/2293336233 conn(0x7f5230063230 msgr2=0x7f5230063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:23.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.904+0000 7f5236de3700 1 -- 172.21.15.175:0/2293336233 shutdown_connections 2023-10-10T18:57:23.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.904+0000 7f5236de3700 1 -- 172.21.15.175:0/2293336233 wait complete. 2023-10-10T18:57:23.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.905+0000 7f5236de3700 1 Processor -- start 2023-10-10T18:57:23.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.905+0000 7f5236de3700 1 -- start start 2023-10-10T18:57:23.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.905+0000 7f5236de3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52300aff60 0x7f5230163b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:23.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.905+0000 7f5236de3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52300b1fc0 0x7f523015af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:23.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.905+0000 7f5236de3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b2390 0x7f523014d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:23.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.905+0000 7f5236de3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5230169440 con 0x7f52300b2390 2023-10-10T18:57:23.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.905+0000 7f5236de3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5230156720 con 0x7f52300aff60 2023-10-10T18:57:23.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.905+0000 7f5236de3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5230165af0 con 0x7f52300b1fc0 2023-10-10T18:57:23.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b2390 0x7f523014d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:23.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b2390 0x7f523014d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42716/0 (socket says 172.21.15.175:42716) 2023-10-10T18:57:23.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 -- 172.21.15.175:0/4135731926 learned_addr learned my addr 172.21.15.175:0/4135731926 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:23.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5234b7f700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52300aff60 0x7f5230163b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:23.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5227fff700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52300b1fc0 0x7f523015af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:23.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 -- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52300b1fc0 msgr2=0x7f523015af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52300b1fc0 0x7f523015af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 -- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52300aff60 msgr2=0x7f5230163b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52300aff60 0x7f5230163b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 -- 172.21.15.175:0/4135731926 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c009a20 con 0x7f52300b2390 2023-10-10T18:57:23.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5227fff700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52300b1fc0 0x7f523015af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:57:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5235380700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b2390 0x7f523014d5a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f522800c7e0 tx=0x7f5228007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5224ff9700 1 -- 172.21.15.175:0/4135731926 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228005b30 con 0x7f52300b2390 2023-10-10T18:57:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5224ff9700 1 -- 172.21.15.175:0/4135731926 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5228005c90 con 0x7f52300b2390 2023-10-10T18:57:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5230164060 con 0x7f52300b2390 2023-10-10T18:57:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5224ff9700 1 -- 172.21.15.175:0/4135731926 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228007340 con 0x7f52300b2390 2023-10-10T18:57:23.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.906+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5230169160 con 0x7f52300b2390 2023-10-10T18:57:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.907+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5230067570 con 0x7f52300b2390 2023-10-10T18:57:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.908+0000 7f5224ff9700 1 -- 172.21.15.175:0/4135731926 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f522801b400 con 0x7f52300b2390 2023-10-10T18:57:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.908+0000 7f5224ff9700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f521c066f60 0x7f521c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.908+0000 7f5224ff9700 1 -- 172.21.15.175:0/4135731926 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f52280842c0 con 0x7f52300b2390 2023-10-10T18:57:23.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.908+0000 7f5234b7f700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f521c066f60 0x7f521c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:23.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.909+0000 7f5234b7f700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f521c066f60 0x7f521c069410 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f522c0097c0 tx=0x7f522c018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:23.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:23.912+0000 7f5224ff9700 1 -- 172.21.15.175:0/4135731926 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5228054e70 con 0x7f52300b2390 2023-10-10T18:57:24.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.137+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5230158380 con 0x7f52300b2390 2023-10-10T18:57:24.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.137+0000 7f5224ff9700 1 -- 172.21.15.175:0/4135731926 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5443 (secure 0 0 0) 0x7f52280076f0 con 0x7f52300b2390 2023-10-10T18:57:24.138 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:20.611209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f521c066f60 msgr2=0x7f521c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f521c066f60 0x7f521c069410 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f522c0097c0 tx=0x7f522c018000 comp rx=0 tx=0).stop 2023-10-10T18:57:24.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b2390 msgr2=0x7f523014d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:24.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b2390 0x7f523014d5a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f522800c7e0 tx=0x7f5228007ea0 comp rx=0 tx=0).stop 2023-10-10T18:57:24.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 shutdown_connections 2023-10-10T18:57:24.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f521c066f60 0x7f521c069410 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:24.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52300b2390 0x7f523014d5a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:24.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52300aff60 0x7f5230163b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:24.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 --2- 172.21.15.175:0/4135731926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52300b1fc0 0x7f523015af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:24.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 >> 172.21.15.175:0/4135731926 conn(0x7f5230063230 msgr2=0x7f523014fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:24.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 shutdown_connections 2023-10-10T18:57:24.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:24.139+0000 7f5236de3700 1 -- 172.21.15.175:0/4135731926 wait complete. 2023-10-10T18:57:24.143 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 96 2023-10-10T18:57:24.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:57:26.153 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:26.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.277+0000 7f0bfee2a700 1 Processor -- start 2023-10-10T18:57:26.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.277+0000 7f0bfee2a700 1 -- start start 2023-10-10T18:57:26.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.277+0000 7f0bfee2a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0bf80b2390 0x7f0bf8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:26.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.277+0000 7f0bfee2a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0bf80aff60 0x7f0bf80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:26.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.277+0000 7f0bfee2a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf81574b0 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.277+0000 7f0bfee2a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf8150770 con 0x7f0bf80b2390 2023-10-10T18:57:26.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.277+0000 7f0bfee2a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf80afad0 con 0x7f0bf80aff60 2023-10-10T18:57:26.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0bf80b2390 0x7f0bf8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:26.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0bf80b2390 0x7f0bf8155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56720/0 (socket says 172.21.15.175:56720) 2023-10-10T18:57:26.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf7fff700 1 -- 172.21.15.175:0/374092235 learned_addr learned my addr 172.21.15.175:0/374092235 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:26.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bfd3c7700 1 --2- 172.21.15.175:0/374092235 >> v2:172.21.15.191:3300/0 conn(0x7f0bf80aff60 0x7f0bf80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:26.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf7fff700 1 -- 172.21.15.175:0/374092235 >> v2:172.21.15.191:3300/0 conn(0x7f0bf80aff60 msgr2=0x7f0bf80b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:26.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf7fff700 1 --2- 172.21.15.175:0/374092235 >> v2:172.21.15.191:3300/0 conn(0x7f0bf80aff60 0x7f0bf80b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf7fff700 1 -- 172.21.15.175:0/374092235 >> v1:172.21.15.191:6789/0 conn(0x7f0bf80b1fc0 legacy=0x7f0bf8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:26.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf7fff700 1 -- 172.21.15.175:0/374092235 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf80af4c0 con 0x7f0bf80b2390 2023-10-10T18:57:26.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf7fff700 1 --2- 172.21.15.175:0/374092235 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b2390 0x7f0bf8155fb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80641f0 tx=0x7f0bec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7a8b44dcb0edc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf77fe700 1 -- 172.21.15.175:0/374092235 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec00b8b0 con 0x7f0bf80b2390 2023-10-10T18:57:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf77fe700 1 -- 172.21.15.175:0/374092235 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec00ba10 con 0x7f0bf80b2390 2023-10-10T18:57:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.278+0000 7f0bf77fe700 1 -- 172.21.15.175:0/374092235 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec00bdb0 con 0x7f0bf80b2390 2023-10-10T18:57:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.279+0000 7f0bfee2a700 1 -- 172.21.15.175:0/374092235 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b2390 msgr2=0x7f0bf8155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.279+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/374092235 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b2390 0x7f0bf8155fb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80641f0 tx=0x7f0bec009250 comp rx=0 tx=0).stop 2023-10-10T18:57:26.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.279+0000 7f0bfee2a700 1 -- 172.21.15.175:0/374092235 shutdown_connections 2023-10-10T18:57:26.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.279+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/374092235 >> v2:172.21.15.191:3300/0 conn(0x7f0bf80aff60 0x7f0bf80b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.279+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/374092235 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b2390 0x7f0bf8155fb0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.279+0000 7f0bfee2a700 1 -- 172.21.15.175:0/374092235 >> 172.21.15.175:0/374092235 conn(0x7f0bf8063230 msgr2=0x7f0bf8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:26.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.279+0000 7f0bfee2a700 1 -- 172.21.15.175:0/374092235 shutdown_connections 2023-10-10T18:57:26.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.279+0000 7f0bfee2a700 1 -- 172.21.15.175:0/374092235 wait complete. 2023-10-10T18:57:26.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.280+0000 7f0bfee2a700 1 Processor -- start 2023-10-10T18:57:26.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.280+0000 7f0bfee2a700 1 -- start start 2023-10-10T18:57:26.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bfee2a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf80aff60 0x7f0bf81690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bfee2a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b1fc0 0x7f0bf80cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bfee2a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bf80b2390 0x7f0bf80ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bfee2a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf8169d40 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bfee2a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80cc140 con 0x7f0bf80aff60 2023-10-10T18:57:26.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bfee2a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf815bce0 con 0x7f0bf80b2390 2023-10-10T18:57:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b1fc0 0x7f0bf80cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b1fc0 0x7f0bf80cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56722/0 (socket says 172.21.15.175:56722) 2023-10-10T18:57:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 -- 172.21.15.175:0/2967157931 learned_addr learned my addr 172.21.15.175:0/2967157931 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bfcbc6700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf80aff60 0x7f0bf81690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bfd3c7700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bf80b2390 0x7f0bf80ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:26.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 -- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bf80b2390 msgr2=0x7f0bf80ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bf80b2390 0x7f0bf80ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 -- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf80aff60 msgr2=0x7f0bf81690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf80aff60 0x7f0bf81690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 -- 172.21.15.175:0/2967157931 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec0094f0 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf7fff700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b1fc0 0x7f0bf80cc5d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0bec003080 tx=0x7f0bec016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:26.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf4ff9700 1 -- 172.21.15.175:0/2967157931 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec011e20 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf4ff9700 1 -- 172.21.15.175:0/2967157931 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec01c690 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.281+0000 7f0bf4ff9700 1 -- 172.21.15.175:0/2967157931 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec01ca50 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.282+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf80cd540 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.282+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf8169a60 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.282+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf80cc2f0 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.283+0000 7f0bf4ff9700 1 -- 172.21.15.175:0/2967157931 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0bec01a450 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.283+0000 7f0bf4ff9700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0be4066f30 0x7f0be40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:26.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.283+0000 7f0bf4ff9700 1 -- 172.21.15.175:0/2967157931 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f0bec086a40 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.283+0000 7f0bfcbc6700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0be4066f30 0x7f0be40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:26.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.284+0000 7f0bfcbc6700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0be4066f30 0x7f0be40693e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0be8007fd0 tx=0x7f0be8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:26.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.288+0000 7f0bf4ff9700 1 -- 172.21.15.175:0/2967157931 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0bf80cc2f0 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.514+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bf80cc2f0 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.515+0000 7f0bf4ff9700 1 -- 172.21.15.175:0/2967157931 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5440 (secure 0 0 0) 0x7f0bf80cc2f0 con 0x7f0bf80b1fc0 2023-10-10T18:57:26.516 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:26.517 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:25.624607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:26.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0be4066f30 msgr2=0x7f0be40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:26.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0be4066f30 0x7f0be40693e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0be8007fd0 tx=0x7f0be8009040 comp rx=0 tx=0).stop 2023-10-10T18:57:26.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b1fc0 msgr2=0x7f0bf80cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:26.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b1fc0 0x7f0bf80cc5d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0bec003080 tx=0x7f0bec016000 comp rx=0 tx=0).stop 2023-10-10T18:57:26.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 shutdown_connections 2023-10-10T18:57:26.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0be4066f30 0x7f0be40693e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bf80b2390 0x7f0bf80ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bf80aff60 0x7f0bf81690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 --2- 172.21.15.175:0/2967157931 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bf80b1fc0 0x7f0bf80cc5d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:26.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 >> 172.21.15.175:0/2967157931 conn(0x7f0bf8063230 msgr2=0x7f0bf8157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:26.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 shutdown_connections 2023-10-10T18:57:26.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:26.517+0000 7f0bfee2a700 1 -- 172.21.15.175:0/2967157931 wait complete. 2023-10-10T18:57:26.521 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 97 2023-10-10T18:57:26.530 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:57:28.531 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:28.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.659+0000 7f5632eb9700 1 Processor -- start 2023-10-10T18:57:28.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.659+0000 7f5632eb9700 1 -- start start 2023-10-10T18:57:28.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f5632eb9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f562c0b2390 0x7f562c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:28.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f5632eb9700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c1574b0 con 0x7f562c0aff60 2023-10-10T18:57:28.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f5632eb9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c0afad0 con 0x7f562c0b1fc0 2023-10-10T18:57:28.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f5632eb9700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562c150770 con 0x7f562c0b2390 2023-10-10T18:57:28.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f562c0b2390 0x7f562c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:28.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f5631456700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f562c0aff60 0x7f562c159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52346/0 (socket says 172.21.15.175:52346) 2023-10-10T18:57:28.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f5631456700 1 -- 172.21.15.175:0/319616750 learned_addr learned my addr 172.21.15.175:0/319616750 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:57:28.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562bfff700 1 -- 172.21.15.175:0/319616750 >> v1:172.21.15.191:6789/0 conn(0x7f562c0b1fc0 legacy=0x7f562c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:28.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562bfff700 1 -- 172.21.15.175:0/319616750 >> v1:172.21.15.175:6789/0 conn(0x7f562c0aff60 legacy=0x7f562c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:28.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562b7fe700 1 -- 172.21.15.175:0/319616750 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378584183 0 0) 0x7f562c1574b0 con 0x7f562c0aff60 2023-10-10T18:57:28.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562bfff700 1 -- 172.21.15.175:0/319616750 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c0af4c0 con 0x7f562c0b2390 2023-10-10T18:57:28.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562bfff700 1 --2- 172.21.15.175:0/319616750 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c155fb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f561c009e60 tx=0x7f561c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81a6f43be97c649 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:28.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562b7fe700 1 -- 172.21.15.175:0/319616750 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c00e070 con 0x7f562c0b2390 2023-10-10T18:57:28.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562b7fe700 1 -- 172.21.15.175:0/319616750 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c00bcc0 con 0x7f562c0b2390 2023-10-10T18:57:28.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.660+0000 7f562b7fe700 1 -- 172.21.15.175:0/319616750 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c012680 con 0x7f562c0b2390 2023-10-10T18:57:28.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.661+0000 7f5632eb9700 1 -- 172.21.15.175:0/319616750 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 msgr2=0x7f562c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:28.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.661+0000 7f5632eb9700 1 --2- 172.21.15.175:0/319616750 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c155fb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f561c009e60 tx=0x7f561c009780 comp rx=0 tx=0).stop 2023-10-10T18:57:28.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.661+0000 7f5632eb9700 1 -- 172.21.15.175:0/319616750 shutdown_connections 2023-10-10T18:57:28.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.661+0000 7f5632eb9700 1 --2- 172.21.15.175:0/319616750 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c155fb0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:28.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.661+0000 7f5632eb9700 1 -- 172.21.15.175:0/319616750 >> 172.21.15.175:0/319616750 conn(0x7f562c063230 msgr2=0x7f562c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:28.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.661+0000 7f5632eb9700 1 -- 172.21.15.175:0/319616750 shutdown_connections 2023-10-10T18:57:28.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.661+0000 7f5632eb9700 1 -- 172.21.15.175:0/319616750 wait complete. 2023-10-10T18:57:28.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.662+0000 7f5632eb9700 1 Processor -- start 2023-10-10T18:57:28.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.662+0000 7f5632eb9700 1 -- start start 2023-10-10T18:57:28.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5632eb9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f562c0aff60 0x7f562c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5632eb9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f562c0b1fc0 0x7f562c15b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5632eb9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c15b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5632eb9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c166a50 con 0x7f562c0b2390 2023-10-10T18:57:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5632eb9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c159830 con 0x7f562c0aff60 2023-10-10T18:57:28.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5632eb9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c166f20 con 0x7f562c0b1fc0 2023-10-10T18:57:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5631456700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c15b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5631456700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c15b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56754/0 (socket says 172.21.15.175:56754) 2023-10-10T18:57:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5631456700 1 -- 172.21.15.175:0/3718480719 learned_addr learned my addr 172.21.15.175:0/3718480719 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5630c55700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f562c0aff60 0x7f562c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:28.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f562bfff700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f562c0b1fc0 0x7f562c15b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5631456700 1 -- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f562c0b1fc0 msgr2=0x7f562c15b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5631456700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f562c0b1fc0 0x7f562c15b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5631456700 1 -- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f562c0aff60 msgr2=0x7f562c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:28.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5631456700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f562c0aff60 0x7f562c0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5631456700 1 -- 172.21.15.175:0/3718480719 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c009a20 con 0x7f562c0b2390 2023-10-10T18:57:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f5630c55700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f562c0aff60 0x7f562c0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:57:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.663+0000 7f562bfff700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f562c0b1fc0 0x7f562c15b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:57:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.664+0000 7f5631456700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c15b610 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f56140011e0 tx=0x7f5614007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:28.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.664+0000 7f5628ff9700 1 -- 172.21.15.175:0/3718480719 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614005b30 con 0x7f562c0b2390 2023-10-10T18:57:28.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.664+0000 7f5628ff9700 1 -- 172.21.15.175:0/3718480719 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5614005c90 con 0x7f562c0b2390 2023-10-10T18:57:28.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.664+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c158f40 con 0x7f562c0b2390 2023-10-10T18:57:28.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.664+0000 7f5628ff9700 1 -- 172.21.15.175:0/3718480719 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614007300 con 0x7f562c0b2390 2023-10-10T18:57:28.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.664+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c1591e0 con 0x7f562c0b2390 2023-10-10T18:57:28.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.665+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c071de0 con 0x7f562c0b2390 2023-10-10T18:57:28.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.665+0000 7f5628ff9700 1 -- 172.21.15.175:0/3718480719 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5614007670 con 0x7f562c0b2390 2023-10-10T18:57:28.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.665+0000 7f5628ff9700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f560c066f30 0x7f560c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:28.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.666+0000 7f5628ff9700 1 -- 172.21.15.175:0/3718480719 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f5614083da0 con 0x7f562c0b2390 2023-10-10T18:57:28.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.666+0000 7f5630c55700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f560c066f30 0x7f560c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:28.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.666+0000 7f5630c55700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f560c066f30 0x7f560c0693e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5624001860 tx=0x7f5624009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:28.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.670+0000 7f5628ff9700 1 -- 172.21.15.175:0/3718480719 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5614050d00 con 0x7f562c0b2390 2023-10-10T18:57:28.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.894+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f562c067570 con 0x7f562c0b2390 2023-10-10T18:57:28.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.895+0000 7f5628ff9700 1 -- 172.21.15.175:0/3718480719 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5440 (secure 0 0 0) 0x7f5614017020 con 0x7f562c0b2390 2023-10-10T18:57:28.896 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:28.896 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:25.624607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:28.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f560c066f30 msgr2=0x7f560c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:28.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f560c066f30 0x7f560c0693e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5624001860 tx=0x7f5624009040 comp rx=0 tx=0).stop 2023-10-10T18:57:28.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 msgr2=0x7f562c15b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:28.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c15b610 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f56140011e0 tx=0x7f5614007d30 comp rx=0 tx=0).stop 2023-10-10T18:57:28.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 shutdown_connections 2023-10-10T18:57:28.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f560c066f30 0x7f560c0693e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:28.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f562c0b2390 0x7f562c15b610 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:28.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f562c0aff60 0x7f562c0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:28.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 --2- 172.21.15.175:0/3718480719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f562c0b1fc0 0x7f562c15b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:28.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 >> 172.21.15.175:0/3718480719 conn(0x7f562c063230 msgr2=0x7f562c14fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:28.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 shutdown_connections 2023-10-10T18:57:28.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:28.897+0000 7f5632eb9700 1 -- 172.21.15.175:0/3718480719 wait complete. 2023-10-10T18:57:28.901 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 97 2023-10-10T18:57:28.910 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:57:30.911 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:31.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.032+0000 7fdc6ed00700 1 Processor -- start 2023-10-10T18:57:31.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.032+0000 7fdc6ed00700 1 -- start start 2023-10-10T18:57:31.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc6ed00700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdc680b1fc0 0x7fdc68155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:31.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc6ed00700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdc680b2390 0x7fdc68068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:31.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc6ed00700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc681574b0 con 0x7fdc68068bd0 2023-10-10T18:57:31.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc6ed00700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc68150770 con 0x7fdc680b2390 2023-10-10T18:57:31.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc6ed00700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc680afad0 con 0x7fdc680b1fc0 2023-10-10T18:57:31.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc67fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdc680b2390 0x7fdc68068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:31.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc67fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdc680b2390 0x7fdc68068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56766/0 (socket says 172.21.15.175:56766) 2023-10-10T18:57:31.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc67fff700 1 -- 172.21.15.175:0/1832909238 learned_addr learned my addr 172.21.15.175:0/1832909238 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:31.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc6ca9c700 1 --2- 172.21.15.175:0/1832909238 >> v2:172.21.15.191:3300/0 conn(0x7fdc680b1fc0 0x7fdc68155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:31.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc67fff700 1 -- 172.21.15.175:0/1832909238 >> v2:172.21.15.191:3300/0 conn(0x7fdc680b1fc0 msgr2=0x7fdc68155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:31.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc67fff700 1 --2- 172.21.15.175:0/1832909238 >> v2:172.21.15.191:3300/0 conn(0x7fdc680b1fc0 0x7fdc68155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc67fff700 1 -- 172.21.15.175:0/1832909238 >> v1:172.21.15.191:6789/0 conn(0x7fdc68068bd0 legacy=0x7fdc680aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:31.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc67fff700 1 -- 172.21.15.175:0/1832909238 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc680af4c0 con 0x7fdc680b2390 2023-10-10T18:57:31.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc67fff700 1 --2- 172.21.15.175:0/1832909238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b2390 0x7fdc68068690 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdc680b28c0 tx=0x7fdc54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2d99062e7d0066 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:31.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc677fe700 1 -- 172.21.15.175:0/1832909238 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54009e70 con 0x7fdc680b2390 2023-10-10T18:57:31.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc677fe700 1 -- 172.21.15.175:0/1832909238 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5400b8f0 con 0x7fdc680b2390 2023-10-10T18:57:31.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.033+0000 7fdc677fe700 1 -- 172.21.15.175:0/1832909238 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5400bc90 con 0x7fdc680b2390 2023-10-10T18:57:31.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.034+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1832909238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b2390 msgr2=0x7fdc68068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:31.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.034+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1832909238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b2390 0x7fdc68068690 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdc680b28c0 tx=0x7fdc54009250 comp rx=0 tx=0).stop 2023-10-10T18:57:31.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.034+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1832909238 shutdown_connections 2023-10-10T18:57:31.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.034+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1832909238 >> v2:172.21.15.191:3300/0 conn(0x7fdc680b1fc0 0x7fdc68155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.034+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1832909238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b2390 0x7fdc68068690 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.034+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1832909238 >> 172.21.15.175:0/1832909238 conn(0x7fdc68063230 msgr2=0x7fdc68063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:31.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.034+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1832909238 shutdown_connections 2023-10-10T18:57:31.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.034+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1832909238 wait complete. 2023-10-10T18:57:31.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.035+0000 7fdc6ed00700 1 Processor -- start 2023-10-10T18:57:31.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.035+0000 7fdc6ed00700 1 -- start start 2023-10-10T18:57:31.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc6ed00700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68068bd0 0x7fdc680cc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:31.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc6ed00700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b1fc0 0x7fdc681672a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:31.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc6ed00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680b2390 0x7fdc6814d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc6ed00700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680c6b40 con 0x7fdc680b1fc0 2023-10-10T18:57:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc6ed00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680cb980 con 0x7fdc680b2390 2023-10-10T18:57:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc6ed00700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680cce80 con 0x7fdc68068bd0 2023-10-10T18:57:31.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b1fc0 0x7fdc681672a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b1fc0 0x7fdc681672a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56770/0 (socket says 172.21.15.175:56770) 2023-10-10T18:57:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 -- 172.21.15.175:0/1311364318 learned_addr learned my addr 172.21.15.175:0/1311364318 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc6ca9c700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68068bd0 0x7fdc680cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc6d29d700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680b2390 0x7fdc6814d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:31.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 -- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68068bd0 msgr2=0x7fdc680cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68068bd0 0x7fdc680cc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 -- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680b2390 msgr2=0x7fdc6814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680b2390 0x7fdc6814d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 -- 172.21.15.175:0/1311364318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc540094f0 con 0x7fdc680b1fc0 2023-10-10T18:57:31.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc67fff700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b1fc0 0x7fdc681672a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdc6806fd90 tx=0x7fdc54017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc64ff9700 1 -- 172.21.15.175:0/1311364318 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5400bc70 con 0x7fdc680b1fc0 2023-10-10T18:57:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.036+0000 7fdc64ff9700 1 -- 172.21.15.175:0/1311364318 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5401a450 con 0x7fdc680b1fc0 2023-10-10T18:57:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.037+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc680cd160 con 0x7fdc680b1fc0 2023-10-10T18:57:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.037+0000 7fdc64ff9700 1 -- 172.21.15.175:0/1311364318 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5401a810 con 0x7fdc680b1fc0 2023-10-10T18:57:31.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.037+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc680cb330 con 0x7fdc680b1fc0 2023-10-10T18:57:31.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.037+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc68067570 con 0x7fdc680b1fc0 2023-10-10T18:57:31.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.038+0000 7fdc64ff9700 1 -- 172.21.15.175:0/1311364318 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdc5401abc0 con 0x7fdc680b1fc0 2023-10-10T18:57:31.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.038+0000 7fdc64ff9700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc4c066f30 0x7fdc4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:31.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.038+0000 7fdc64ff9700 1 -- 172.21.15.175:0/1311364318 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fdc540862d0 con 0x7fdc680b1fc0 2023-10-10T18:57:31.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.038+0000 7fdc6ca9c700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc4c066f30 0x7fdc4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:31.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.039+0000 7fdc6ca9c700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc4c066f30 0x7fdc4c0693e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009d20 tx=0x7fdc5c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:31.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.042+0000 7fdc64ff9700 1 -- 172.21.15.175:0/1311364318 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc540532c0 con 0x7fdc680b1fc0 2023-10-10T18:57:31.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.265+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc68069060 con 0x7fdc680b1fc0 2023-10-10T18:57:31.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.265+0000 7fdc64ff9700 1 -- 172.21.15.175:0/1311364318 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5440 (secure 0 0 0) 0x7fdc540568e0 con 0x7fdc680b1fc0 2023-10-10T18:57:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:31.267 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:25.624607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:31.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc4c066f30 msgr2=0x7fdc4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:31.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc4c066f30 0x7fdc4c0693e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009d20 tx=0x7fdc5c0091a0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b1fc0 msgr2=0x7fdc681672a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:31.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b1fc0 0x7fdc681672a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdc6806fd90 tx=0x7fdc54017000 comp rx=0 tx=0).stop 2023-10-10T18:57:31.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 shutdown_connections 2023-10-10T18:57:31.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc4c066f30 0x7fdc4c0693e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680b2390 0x7fdc6814d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68068bd0 0x7fdc680cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 --2- 172.21.15.175:0/1311364318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc680b1fc0 0x7fdc681672a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:31.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 >> 172.21.15.175:0/1311364318 conn(0x7fdc68063230 msgr2=0x7fdc681508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:31.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.267+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 shutdown_connections 2023-10-10T18:57:31.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:31.268+0000 7fdc6ed00700 1 -- 172.21.15.175:0/1311364318 wait complete. 2023-10-10T18:57:31.272 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 97 2023-10-10T18:57:31.280 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:57:33.281 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:33.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.402+0000 7fafc89ef700 1 Processor -- start 2023-10-10T18:57:33.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.402+0000 7fafc89ef700 1 -- start start 2023-10-10T18:57:33.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc89ef700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc41577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc89ef700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fafc4157d10 0x7fafc414d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc89ef700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc406d8e0 con 0x7fafc4157400 2023-10-10T18:57:33.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc89ef700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc406e050 con 0x7fafc4157d10 2023-10-10T18:57:33.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc89ef700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc406dbc0 con 0x7fafc40c4640 2023-10-10T18:57:33.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc359e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fafc4157d10 0x7fafc414d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc359e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fafc4157d10 0x7fafc414d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56784/0 (socket says 172.21.15.175:56784) 2023-10-10T18:57:33.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc359e700 1 -- 172.21.15.175:0/594551499 learned_addr learned my addr 172.21.15.175:0/594551499 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:33.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc359e700 1 -- 172.21.15.175:0/594551499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 msgr2=0x7fafc41577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:33.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc259c700 1 --2- 172.21.15.175:0/594551499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc41577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc359e700 1 --2- 172.21.15.175:0/594551499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc41577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc359e700 1 -- 172.21.15.175:0/594551499 >> v1:172.21.15.191:6789/0 conn(0x7fafc4157400 legacy=0x7fafc406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc359e700 1 -- 172.21.15.175:0/594551499 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40c1030 con 0x7fafc4157d10 2023-10-10T18:57:33.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc259c700 1 --2- 172.21.15.175:0/594551499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc41577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:57:33.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc359e700 1 --2- 172.21.15.175:0/594551499 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157d10 0x7fafc414d590 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009f20 tx=0x7fafb4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460617c25bec7280 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:33.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc1d9b700 1 -- 172.21.15.175:0/594551499 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb400e070 con 0x7fafc4157d10 2023-10-10T18:57:33.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.403+0000 7fafc1d9b700 1 -- 172.21.15.175:0/594551499 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb400ba80 con 0x7fafc4157d10 2023-10-10T18:57:33.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc1d9b700 1 -- 172.21.15.175:0/594551499 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40092e0 con 0x7fafc4157d10 2023-10-10T18:57:33.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc89ef700 1 -- 172.21.15.175:0/594551499 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157d10 msgr2=0x7fafc414d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc89ef700 1 --2- 172.21.15.175:0/594551499 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157d10 0x7fafc414d590 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009f20 tx=0x7fafb4009840 comp rx=0 tx=0).stop 2023-10-10T18:57:33.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc89ef700 1 -- 172.21.15.175:0/594551499 shutdown_connections 2023-10-10T18:57:33.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc89ef700 1 --2- 172.21.15.175:0/594551499 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157d10 0x7fafc414d590 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc89ef700 1 --2- 172.21.15.175:0/594551499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc41577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc89ef700 1 -- 172.21.15.175:0/594551499 >> 172.21.15.175:0/594551499 conn(0x7fafc4063cd0 msgr2=0x7fafc406d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:33.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc89ef700 1 -- 172.21.15.175:0/594551499 shutdown_connections 2023-10-10T18:57:33.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.404+0000 7fafc89ef700 1 -- 172.21.15.175:0/594551499 wait complete. 2023-10-10T18:57:33.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.405+0000 7fafc89ef700 1 Processor -- start 2023-10-10T18:57:33.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.405+0000 7fafc89ef700 1 -- start start 2023-10-10T18:57:33.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc89ef700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc40c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc89ef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157400 0x7fafc4156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc89ef700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc4157d10 0x7fafc40c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc89ef700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40c8890 con 0x7fafc4157400 2023-10-10T18:57:33.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc89ef700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40ca420 con 0x7fafc4157d10 2023-10-10T18:57:33.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc89ef700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40c9be0 con 0x7fafc40c4640 2023-10-10T18:57:33.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157400 0x7fafc4156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157400 0x7fafc4156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56798/0 (socket says 172.21.15.175:56798) 2023-10-10T18:57:33.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc259c700 1 -- 172.21.15.175:0/630977337 learned_addr learned my addr 172.21.15.175:0/630977337 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:33.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc2d9d700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc40c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc359e700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc4157d10 0x7fafc40c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc259c700 1 -- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 msgr2=0x7fafc40c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc259c700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc40c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc259c700 1 -- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc4157d10 msgr2=0x7fafc40c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc259c700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc4157d10 0x7fafc40c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.406+0000 7fafc259c700 1 -- 172.21.15.175:0/630977337 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4009ae0 con 0x7fafc4157400 2023-10-10T18:57:33.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.407+0000 7fafc259c700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157400 0x7fafc4156360 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fafb800c830 tx=0x7fafb8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:33.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.407+0000 7fafaaffd700 1 -- 172.21.15.175:0/630977337 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8011600 con 0x7fafc4157400 2023-10-10T18:57:33.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.407+0000 7fafaaffd700 1 -- 172.21.15.175:0/630977337 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb8011760 con 0x7fafc4157400 2023-10-10T18:57:33.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.407+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc40c8560 con 0x7fafc4157400 2023-10-10T18:57:33.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.407+0000 7fafaaffd700 1 -- 172.21.15.175:0/630977337 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8011b20 con 0x7fafc4157400 2023-10-10T18:57:33.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.407+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc40ca140 con 0x7fafc4157400 2023-10-10T18:57:33.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.408+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc40c86f0 con 0x7fafc4157400 2023-10-10T18:57:33.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.408+0000 7fafaaffd700 1 -- 172.21.15.175:0/630977337 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fafb8018360 con 0x7fafc4157400 2023-10-10T18:57:33.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.408+0000 7fafaaffd700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb0066f30 0x7fafb00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.409+0000 7fafaaffd700 1 -- 172.21.15.175:0/630977337 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fafb80847b0 con 0x7fafc4157400 2023-10-10T18:57:33.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.409+0000 7fafc2d9d700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb0066f30 0x7fafb00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.409+0000 7fafc2d9d700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb0066f30 0x7fafb00693e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fafbc003a80 tx=0x7fafbc007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:33.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.413+0000 7fafaaffd700 1 -- 172.21.15.175:0/630977337 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafc40c86f0 con 0x7fafc4157400 2023-10-10T18:57:33.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.636+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafc40c86f0 con 0x7fafc4157400 2023-10-10T18:57:33.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.637+0000 7fafaaffd700 1 -- 172.21.15.175:0/630977337 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5440 (secure 0 0 0) 0x7fafc40c86f0 con 0x7fafc4157400 2023-10-10T18:57:33.638 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:33.638 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:33.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb0066f30 msgr2=0x7fafb00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb0066f30 0x7fafb00693e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fafbc003a80 tx=0x7fafbc007ec0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157400 msgr2=0x7fafc4156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157400 0x7fafc4156360 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fafb800c830 tx=0x7fafb8008040 comp rx=0 tx=0).stop 2023-10-10T18:57:33.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 shutdown_connections 2023-10-10T18:57:33.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb0066f30 0x7fafb00693e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc4157d10 0x7fafc40c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc40c4640 0x7fafc40c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 --2- 172.21.15.175:0/630977337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc4157400 0x7fafc4156360 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 >> 172.21.15.175:0/630977337 conn(0x7fafc4063cd0 msgr2=0x7fafc414b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:33.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 shutdown_connections 2023-10-10T18:57:33.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.639+0000 7fafc89ef700 1 -- 172.21.15.175:0/630977337 wait complete. 2023-10-10T18:57:33.643 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 98 2023-10-10T18:57:33.652 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=4263 2023-10-10T18:57:33.653 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.d 2023-10-10T18:57:33.653 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-10T18:57:33.653 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-10T18:57:33.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-10T18:57:33.656 INFO:tasks.ceph.mds.d:Started 2023-10-10T18:57:33.657 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:33.736 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-10T18:57:33.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.770+0000 7fbb15afc700 1 Processor -- start 2023-10-10T18:57:33.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.770+0000 7fbb15afc700 1 -- start start 2023-10-10T18:57:33.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb15afc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb15afc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb101574b0 con 0x7fbb100aff60 2023-10-10T18:57:33.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb15afc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb100afad0 con 0x7fbb100b2390 2023-10-10T18:57:33.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb15afc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb10150770 con 0x7fbb100b1fc0 2023-10-10T18:57:33.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb0ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbb100aff60 0x7fbb10159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52348/0 (socket says 172.21.15.175:52348) 2023-10-10T18:57:33.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb0ffff700 1 -- 172.21.15.175:0/984356617 learned_addr learned my addr 172.21.15.175:0/984356617 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:57:33.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb0f7fe700 1 --2- 172.21.15.175:0/984356617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb0e7fc700 1 -- 172.21.15.175:0/984356617 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723090795 0 0) 0x7fbb101574b0 con 0x7fbb100aff60 2023-10-10T18:57:33.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb0e7fc700 1 -- 172.21.15.175:0/984356617 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf8003570 con 0x7fbb100aff60 2023-10-10T18:57:33.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb0e7fc700 1 -- 172.21.15.175:0/984356617 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475867645 0 0) 0x7fbb100afad0 con 0x7fbb100b2390 2023-10-10T18:57:33.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.771+0000 7fbb0e7fc700 1 -- 172.21.15.175:0/984356617 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb101574b0 con 0x7fbb100b2390 2023-10-10T18:57:33.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb0f7fe700 1 -- 172.21.15.175:0/984356617 >> v1:172.21.15.191:6789/0 conn(0x7fbb100b2390 legacy=0x7fbb10068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb0f7fe700 1 -- 172.21.15.175:0/984356617 >> v1:172.21.15.175:6789/0 conn(0x7fbb100aff60 legacy=0x7fbb10159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb0f7fe700 1 -- 172.21.15.175:0/984356617 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb100af4c0 con 0x7fbb100b1fc0 2023-10-10T18:57:33.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb0f7fe700 1 --2- 172.21.15.175:0/984356617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10155fb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009ec0 tx=0x7fbb00009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3823a752f8ee93e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:33.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb0e7fc700 1 -- 172.21.15.175:0/984356617 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000e070 con 0x7fbb100b1fc0 2023-10-10T18:57:33.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb0e7fc700 1 -- 172.21.15.175:0/984356617 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0000bbd0 con 0x7fbb100b1fc0 2023-10-10T18:57:33.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb0e7fc700 1 -- 172.21.15.175:0/984356617 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00012430 con 0x7fbb100b1fc0 2023-10-10T18:57:33.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb15afc700 1 -- 172.21.15.175:0/984356617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 msgr2=0x7fbb10155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb15afc700 1 --2- 172.21.15.175:0/984356617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10155fb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009ec0 tx=0x7fbb00009780 comp rx=0 tx=0).stop 2023-10-10T18:57:33.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb15afc700 1 -- 172.21.15.175:0/984356617 shutdown_connections 2023-10-10T18:57:33.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb15afc700 1 --2- 172.21.15.175:0/984356617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10155fb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb15afc700 1 -- 172.21.15.175:0/984356617 >> 172.21.15.175:0/984356617 conn(0x7fbb10063230 msgr2=0x7fbb10063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:33.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb15afc700 1 -- 172.21.15.175:0/984356617 shutdown_connections 2023-10-10T18:57:33.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.772+0000 7fbb15afc700 1 -- 172.21.15.175:0/984356617 wait complete. 2023-10-10T18:57:33.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb15afc700 1 Processor -- start 2023-10-10T18:57:33.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb15afc700 1 -- start start 2023-10-10T18:57:33.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb15afc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb100aff60 0x7fbb10169410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb15afc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10169950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb15afc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb100b2390 0x7fbb10154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb15afc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb100ca950 con 0x7fbb100b2390 2023-10-10T18:57:33.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb15afc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb10156730 con 0x7fbb100aff60 2023-10-10T18:57:33.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb15afc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb10154170 con 0x7fbb100b1fc0 2023-10-10T18:57:33.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb0ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb100b2390 0x7fbb10154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb0f7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb100aff60 0x7fbb10169410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb0ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb100b2390 0x7fbb10154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56804/0 (socket says 172.21.15.175:56804) 2023-10-10T18:57:33.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb0f7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb100aff60 0x7fbb10169410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:46694/0 (socket says 172.21.15.175:46694) 2023-10-10T18:57:33.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb0ffff700 1 -- 172.21.15.175:0/18763444 learned_addr learned my addr 172.21.15.175:0/18763444 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:33.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.774+0000 7fbb0effd700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10169950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb0ffff700 1 -- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 msgr2=0x7fbb10169950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb0ffff700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10169950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb0ffff700 1 -- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb100aff60 msgr2=0x7fbb10169410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:33.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb0ffff700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb100aff60 0x7fbb10169410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:33.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb0ffff700 1 -- 172.21.15.175:0/18763444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb00009a20 con 0x7fbb100b2390 2023-10-10T18:57:33.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb0ffff700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb100b2390 0x7fbb10154600 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800c7c0 tx=0x7fbb08007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:33.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbaf77fe700 1 -- 172.21.15.175:0/18763444 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08005c00 con 0x7fbb100b2390 2023-10-10T18:57:33.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbaf77fe700 1 -- 172.21.15.175:0/18763444 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb08005d60 con 0x7fbb100b2390 2023-10-10T18:57:33.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb10169f70 con 0x7fbb100b2390 2023-10-10T18:57:33.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbaf77fe700 1 -- 172.21.15.175:0/18763444 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08007320 con 0x7fbb100b2390 2023-10-10T18:57:33.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.775+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb10156450 con 0x7fbb100b2390 2023-10-10T18:57:33.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.776+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb10067570 con 0x7fbb100b2390 2023-10-10T18:57:33.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.776+0000 7fbaf77fe700 1 -- 172.21.15.175:0/18763444 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbb0801b3b0 con 0x7fbb100b2390 2023-10-10T18:57:33.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.777+0000 7fbaf77fe700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbafc066f60 0x7fbafc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:33.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.777+0000 7fbaf77fe700 1 -- 172.21.15.175:0/18763444 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fbb08083d40 con 0x7fbb100b2390 2023-10-10T18:57:33.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.777+0000 7fbb0f7fe700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbafc066f60 0x7fbafc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:33.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.777+0000 7fbb0f7fe700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbafc066f60 0x7fbafc069410 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009de0 tx=0x7fbb00019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:33.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:33.781+0000 7fbaf77fe700 1 -- 172.21.15.175:0/18763444 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb08050c90 con 0x7fbb100b2390 2023-10-10T18:57:34.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.005+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb101542b0 con 0x7fbb100b2390 2023-10-10T18:57:34.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.005+0000 7fbaf77fe700 1 -- 172.21.15.175:0/18763444 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5440 (secure 0 0 0) 0x7fbb08017070 con 0x7fbb100b2390 2023-10-10T18:57:34.007 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:34.007 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:34.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.007+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbafc066f60 msgr2=0x7fbafc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:34.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.007+0000 7fbb15afc700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbafc066f60 0x7fbafc069410 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009de0 tx=0x7fbb00019000 comp rx=0 tx=0).stop 2023-10-10T18:57:34.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.007+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb100b2390 msgr2=0x7fbb10154600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:34.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.007+0000 7fbb15afc700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb100b2390 0x7fbb10154600 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800c7c0 tx=0x7fbb08007db0 comp rx=0 tx=0).stop 2023-10-10T18:57:34.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.008+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 shutdown_connections 2023-10-10T18:57:34.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.008+0000 7fbb15afc700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbafc066f60 0x7fbafc069410 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:34.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.008+0000 7fbb15afc700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb100b2390 0x7fbb10154600 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:34.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.008+0000 7fbb15afc700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb100aff60 0x7fbb10169410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:34.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.008+0000 7fbb15afc700 1 --2- 172.21.15.175:0/18763444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb100b1fc0 0x7fbb10169950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:34.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.008+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 >> 172.21.15.175:0/18763444 conn(0x7fbb10063230 msgr2=0x7fbb1014fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:34.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.008+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 shutdown_connections 2023-10-10T18:57:34.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:34.008+0000 7fbb15afc700 1 -- 172.21.15.175:0/18763444 wait complete. 2023-10-10T18:57:34.012 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 98 2023-10-10T18:57:34.020 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-10T18:57:36.022 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:57:36.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.142+0000 7fc1ee36d700 1 Processor -- start 2023-10-10T18:57:36.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.142+0000 7fc1ee36d700 1 -- start start 2023-10-10T18:57:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1ee36d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1e80b1fc0 0x7fc1e8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:36.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1ee36d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 0x7fc1e8068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1ee36d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc1e8068bd0 0x7fc1e80aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1ee36d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e8150770 con 0x7fc1e80b1fc0 2023-10-10T18:57:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1ee36d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e81574b0 con 0x7fc1e8068bd0 2023-10-10T18:57:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1ee36d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80afad0 con 0x7fc1e80b2390 2023-10-10T18:57:36.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1e7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1e80b1fc0 0x7fc1e8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:36.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1e7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1e80b1fc0 0x7fc1e8155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37082/0 (socket says 172.21.15.175:37082) 2023-10-10T18:57:36.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1e7fff700 1 -- 172.21.15.175:0/218224279 learned_addr learned my addr 172.21.15.175:0/218224279 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:36.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1e77fe700 1 --2- 172.21.15.175:0/218224279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 0x7fc1e8068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.143+0000 7fc1ec90a700 1 --2- 172.21.15.175:0/218224279 >> v2:172.21.15.191:3300/0 conn(0x7fc1e8068bd0 0x7fc1e80aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e7fff700 1 -- 172.21.15.175:0/218224279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 msgr2=0x7fc1e8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e7fff700 1 --2- 172.21.15.175:0/218224279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 0x7fc1e8068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e7fff700 1 -- 172.21.15.175:0/218224279 >> v2:172.21.15.191:3300/0 conn(0x7fc1e8068bd0 msgr2=0x7fc1e80aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:57:36.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e7fff700 1 --2- 172.21.15.175:0/218224279 >> v2:172.21.15.191:3300/0 conn(0x7fc1e8068bd0 0x7fc1e80aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e7fff700 1 -- 172.21.15.175:0/218224279 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e80af4c0 con 0x7fc1e80b1fc0 2023-10-10T18:57:36.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e7fff700 1 --2- 172.21.15.175:0/218224279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e80b1fc0 0x7fc1e8155fb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80afa40 tx=0x7fc1dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58461d116525bcb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:36.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/218224279 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc00b6d0 con 0x7fc1e80b1fc0 2023-10-10T18:57:36.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/218224279 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc00b830 con 0x7fc1e80b1fc0 2023-10-10T18:57:36.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/218224279 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc00bbf0 con 0x7fc1e80b1fc0 2023-10-10T18:57:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1ee36d700 1 -- 172.21.15.175:0/218224279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e80b1fc0 msgr2=0x7fc1e8155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/218224279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e80b1fc0 0x7fc1e8155fb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80afa40 tx=0x7fc1dc009250 comp rx=0 tx=0).stop 2023-10-10T18:57:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1ee36d700 1 -- 172.21.15.175:0/218224279 shutdown_connections 2023-10-10T18:57:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/218224279 >> v2:172.21.15.191:3300/0 conn(0x7fc1e8068bd0 0x7fc1e80aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/218224279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e80b1fc0 0x7fc1e8155fb0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/218224279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 0x7fc1e8068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.144+0000 7fc1ee36d700 1 -- 172.21.15.175:0/218224279 >> 172.21.15.175:0/218224279 conn(0x7fc1e8063230 msgr2=0x7fc1e8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.145+0000 7fc1ee36d700 1 -- 172.21.15.175:0/218224279 shutdown_connections 2023-10-10T18:57:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.145+0000 7fc1ee36d700 1 -- 172.21.15.175:0/218224279 wait complete. 2023-10-10T18:57:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.146+0000 7fc1ee36d700 1 Processor -- start 2023-10-10T18:57:36.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.146+0000 7fc1ee36d700 1 -- start start 2023-10-10T18:57:36.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.146+0000 7fc1ee36d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e8068bd0 0x7fc1e80c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:36.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.146+0000 7fc1ee36d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1e80b1fc0 0x7fc1e814f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:36.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.146+0000 7fc1ee36d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 0x7fc1e814fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:36.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.146+0000 7fc1ee36d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80c42a0 con 0x7fc1e8068bd0 2023-10-10T18:57:36.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.146+0000 7fc1ee36d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e8150430 con 0x7fc1e80b1fc0 2023-10-10T18:57:36.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.146+0000 7fc1ee36d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80c6090 con 0x7fc1e80b2390 2023-10-10T18:57:36.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e8068bd0 0x7fc1e80c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:36.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e8068bd0 0x7fc1e80c6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37094/0 (socket says 172.21.15.175:37094) 2023-10-10T18:57:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 -- 172.21.15.175:0/3960230279 learned_addr learned my addr 172.21.15.175:0/3960230279 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:57:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1ec90a700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 0x7fc1e814fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e77fe700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1e80b1fc0 0x7fc1e814f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 -- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 msgr2=0x7fc1e814fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:36.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 0x7fc1e814fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 -- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1e80b1fc0 msgr2=0x7fc1e814f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1e80b1fc0 0x7fc1e814f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 -- 172.21.15.175:0/3960230279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc0094f0 con 0x7fc1e8068bd0 2023-10-10T18:57:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1e7fff700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e8068bd0 0x7fc1e80c6520 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc009f70 tx=0x7fc1dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:36.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1cffff700 1 -- 172.21.15.175:0/3960230279 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc000c90 con 0x7fc1e8068bd0 2023-10-10T18:57:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1cffff700 1 -- 172.21.15.175:0/3960230279 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc00b9a0 con 0x7fc1e8068bd0 2023-10-10T18:57:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e8156960 con 0x7fc1e8068bd0 2023-10-10T18:57:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1cffff700 1 -- 172.21.15.175:0/3960230279 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc026430 con 0x7fc1e8068bd0 2023-10-10T18:57:36.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.147+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e80c6a60 con 0x7fc1e8068bd0 2023-10-10T18:57:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.148+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e8067570 con 0x7fc1e8068bd0 2023-10-10T18:57:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.149+0000 7fc1cffff700 1 -- 172.21.15.175:0/3960230279 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc1dc023030 con 0x7fc1e8068bd0 2023-10-10T18:57:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.149+0000 7fc1cffff700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d4066f30 0x7fc1d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:57:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.149+0000 7fc1cffff700 1 -- 172.21.15.175:0/3960230279 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fc1dc0851a0 con 0x7fc1e8068bd0 2023-10-10T18:57:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.149+0000 7fc1e77fe700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d4066f30 0x7fc1d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:57:36.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.150+0000 7fc1e77fe700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d4066f30 0x7fc1d40693e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80c6c90 tx=0x7fc1d80091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:57:36.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.153+0000 7fc1cffff700 1 -- 172.21.15.175:0/3960230279 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1dc052210 con 0x7fc1e8068bd0 2023-10-10T18:57:36.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.379+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1e8069060 con 0x7fc1e8068bd0 2023-10-10T18:57:36.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.379+0000 7fc1cffff700 1 -- 172.21.15.175:0/3960230279 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6196 (secure 0 0 0) 0x7fc1dc059e60 con 0x7fc1e8068bd0 2023-10-10T18:57:36.381 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:57:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5584,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:57:36.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d4066f30 msgr2=0x7fc1d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:36.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d4066f30 0x7fc1d40693e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80c6c90 tx=0x7fc1d80091a0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e8068bd0 msgr2=0x7fc1e80c6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:57:36.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e8068bd0 0x7fc1e80c6520 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc009f70 tx=0x7fc1dc017000 comp rx=0 tx=0).stop 2023-10-10T18:57:36.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 shutdown_connections 2023-10-10T18:57:36.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d4066f30 0x7fc1d40693e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1e80b2390 0x7fc1e814fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e8068bd0 0x7fc1e80c6520 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 --2- 172.21.15.175:0/3960230279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1e80b1fc0 0x7fc1e814f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:57:36.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 >> 172.21.15.175:0/3960230279 conn(0x7fc1e8063230 msgr2=0x7fc1e8150b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:57:36.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 shutdown_connections 2023-10-10T18:57:36.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:57:36.382+0000 7fc1ee36d700 1 -- 172.21.15.175:0/3960230279 wait complete. 2023-10-10T18:57:36.385 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 99 2023-10-10T18:57:36.395 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-10T18:57:36.395 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-10-10T18:57:39.725 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T18:57:39.725+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-10T18:57:39.725 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T18:57:39.725+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T18:57:39.725 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T18:57:39.725+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-10T18:57:39.725 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T18:57:39.725+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T18:57:39.726 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T18:57:39.725+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-10T18:57:50.062 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:57:50.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:58:20.126 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:58:20.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:58:50.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:58:50.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:58:53.396 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:58:53.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.511+0000 7f892ad4a700 1 Processor -- start 2023-10-10T18:58:53.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.511+0000 7f892ad4a700 1 -- start start 2023-10-10T18:58:53.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f892ad4a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 0x7f89240c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:53.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f892ad4a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8924156520 0x7f89241569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:53.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f892ad4a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8924156f00 con 0x7f89240c5e40 2023-10-10T18:58:53.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f892ad4a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89241560d0 con 0x7f8924156520 2023-10-10T18:58:53.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f892ad4a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89240c6df0 con 0x7f892406bf00 2023-10-10T18:58:53.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f89292e7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8924156520 0x7f89241569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:53.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f89292e7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8924156520 0x7f89241569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37712/0 (socket says 172.21.15.175:37712) 2023-10-10T18:58:53.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f89292e7700 1 -- 172.21.15.175:0/3879534559 learned_addr learned my addr 172.21.15.175:0/3879534559 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:58:53.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f8923fff700 1 --2- 172.21.15.175:0/3879534559 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 0x7f89240c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:53.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f89292e7700 1 -- 172.21.15.175:0/3879534559 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 msgr2=0x7f89240c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:58:53.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f89292e7700 1 --2- 172.21.15.175:0/3879534559 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 0x7f89240c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f89292e7700 1 -- 172.21.15.175:0/3879534559 >> v1:172.21.15.191:6789/0 conn(0x7f89240c5e40 legacy=0x7f89240c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:53.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.512+0000 7f89292e7700 1 -- 172.21.15.175:0/3879534559 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240c6920 con 0x7f8924156520 2023-10-10T18:58:53.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f89292e7700 1 --2- 172.21.15.175:0/3879534559 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8924156520 0x7f89241569c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f891c0099b0 tx=0x7f891c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f54ae8723e0187 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:53.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f89237fe700 1 -- 172.21.15.175:0/3879534559 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c00b8f0 con 0x7f8924156520 2023-10-10T18:58:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f89237fe700 1 -- 172.21.15.175:0/3879534559 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00ba50 con 0x7f8924156520 2023-10-10T18:58:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f89237fe700 1 -- 172.21.15.175:0/3879534559 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c00bdf0 con 0x7f8924156520 2023-10-10T18:58:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f892ad4a700 1 -- 172.21.15.175:0/3879534559 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8924156520 msgr2=0x7f89241569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f892ad4a700 1 --2- 172.21.15.175:0/3879534559 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8924156520 0x7f89241569c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f891c0099b0 tx=0x7f891c009250 comp rx=0 tx=0).stop 2023-10-10T18:58:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f892ad4a700 1 -- 172.21.15.175:0/3879534559 shutdown_connections 2023-10-10T18:58:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f892ad4a700 1 --2- 172.21.15.175:0/3879534559 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8924156520 0x7f89241569c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f892ad4a700 1 --2- 172.21.15.175:0/3879534559 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 0x7f89240c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f892ad4a700 1 -- 172.21.15.175:0/3879534559 >> 172.21.15.175:0/3879534559 conn(0x7f8924068900 msgr2=0x7f8924068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:58:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f892ad4a700 1 -- 172.21.15.175:0/3879534559 shutdown_connections 2023-10-10T18:58:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.513+0000 7f892ad4a700 1 -- 172.21.15.175:0/3879534559 wait complete. 2023-10-10T18:58:53.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.514+0000 7f892ad4a700 1 Processor -- start 2023-10-10T18:58:53.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.514+0000 7f892ad4a700 1 -- start start 2023-10-10T18:58:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f892ad4a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 0x7f89240c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f892ad4a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89240c5e40 0x7f89240c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f892ad4a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8924156520 0x7f892414c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f892ad4a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240c79d0 con 0x7f89240c5e40 2023-10-10T18:58:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f892ad4a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240c77d0 con 0x7f8924156520 2023-10-10T18:58:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f892ad4a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89241696f0 con 0x7f892406bf00 2023-10-10T18:58:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f8923fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89240c5e40 0x7f89240c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f8923fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89240c5e40 0x7f89240c7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37726/0 (socket says 172.21.15.175:37726) 2023-10-10T18:58:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f8923fff700 1 -- 172.21.15.175:0/709262289 learned_addr learned my addr 172.21.15.175:0/709262289 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:58:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f89292e7700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8924156520 0x7f892414c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:53.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.515+0000 7f8928ae6700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 0x7f89240c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:53.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8923fff700 1 -- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 msgr2=0x7f89240c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:53.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8923fff700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 0x7f89240c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8923fff700 1 -- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8924156520 msgr2=0x7f892414c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:53.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8923fff700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8924156520 0x7f892414c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8923fff700 1 -- 172.21.15.175:0/709262289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c0094f0 con 0x7f89240c5e40 2023-10-10T18:58:53.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8923fff700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89240c5e40 0x7f89240c7290 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f891800d860 tx=0x7f8918008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:53.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8920ff9700 1 -- 172.21.15.175:0/709262289 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918011590 con 0x7f89240c5e40 2023-10-10T18:58:53.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8920ff9700 1 -- 172.21.15.175:0/709262289 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89180116f0 con 0x7f89240c5e40 2023-10-10T18:58:53.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89241699d0 con 0x7f89240c5e40 2023-10-10T18:58:53.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f8920ff9700 1 -- 172.21.15.175:0/709262289 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918011a90 con 0x7f89240c5e40 2023-10-10T18:58:53.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.516+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8924169f90 con 0x7f89240c5e40 2023-10-10T18:58:53.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.517+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892416a2c0 con 0x7f89240c5e40 2023-10-10T18:58:53.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.517+0000 7f8920ff9700 1 -- 172.21.15.175:0/709262289 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f891801c460 con 0x7f89240c5e40 2023-10-10T18:58:53.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.517+0000 7f8920ff9700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8910066f30 0x7f89100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:53.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.518+0000 7f8920ff9700 1 -- 172.21.15.175:0/709262289 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f8918084750 con 0x7f89240c5e40 2023-10-10T18:58:53.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.518+0000 7f8928ae6700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8910066f30 0x7f89100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:53.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.518+0000 7f8928ae6700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8910066f30 0x7f89100693e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f891400c580 tx=0x7f8914001b20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:53.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.522+0000 7f8920ff9700 1 -- 172.21.15.175:0/709262289 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89180551d0 con 0x7f89240c5e40 2023-10-10T18:58:53.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.748+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8924067570 con 0x7f89240c5e40 2023-10-10T18:58:53.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.748+0000 7f8920ff9700 1 -- 172.21.15.175:0/709262289 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6196 (secure 0 0 0) 0x7f8918054d60 con 0x7f89240c5e40 2023-10-10T18:58:53.749 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:58:53.749 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5584,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:58:53.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8910066f30 msgr2=0x7f89100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:53.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8910066f30 0x7f89100693e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f891400c580 tx=0x7f8914001b20 comp rx=0 tx=0).stop 2023-10-10T18:58:53.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89240c5e40 msgr2=0x7f89240c7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:53.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89240c5e40 0x7f89240c7290 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f891800d860 tx=0x7f8918008040 comp rx=0 tx=0).stop 2023-10-10T18:58:53.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 shutdown_connections 2023-10-10T18:58:53.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8910066f30 0x7f89100693e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8924156520 0x7f892414c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f892406bf00 0x7f89240c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 --2- 172.21.15.175:0/709262289 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f89240c5e40 0x7f89240c7290 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 >> 172.21.15.175:0/709262289 conn(0x7f8924068900 msgr2=0x7f892414f760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:58:53.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 shutdown_connections 2023-10-10T18:58:53.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:53.750+0000 7f892ad4a700 1 -- 172.21.15.175:0/709262289 wait complete. 2023-10-10T18:58:53.753 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 99 2023-10-10T18:58:53.762 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2023-10-10T18:58:53.763 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-10T18:58:53.763 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T18:58:53.776 INFO:tasks.ceph.mds.b:Stopped 2023-10-10T18:58:53.776 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:58:54.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.030+0000 7fcf85046700 1 Processor -- start 2023-10-10T18:58:54.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.030+0000 7fcf85046700 1 -- start start 2023-10-10T18:58:54.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf85046700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf8006ece0 0x7fcf8015de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:54.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf85046700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf8006df80 con 0x7fcf8014fd10 2023-10-10T18:58:54.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf85046700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf8006e4b0 con 0x7fcf8006e910 2023-10-10T18:58:54.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf85046700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800c4620 con 0x7fcf8006ece0 2023-10-10T18:58:54.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf7e59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcf8014fd10 0x7fcf800c4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37670/0 (socket says 172.21.15.175:37670) 2023-10-10T18:58:54.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf7e59c700 1 -- 172.21.15.175:0/2499549533 learned_addr learned my addr 172.21.15.175:0/2499549533 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:58:54.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680294266 0 0) 0x7fcf8006df80 con 0x7fcf8014fd10 2023-10-10T18:58:54.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf68004680 con 0x7fcf8014fd10 2023-10-10T18:58:54.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 658218528 0 0) 0x7fcf68004680 con 0x7fcf8014fd10 2023-10-10T18:58:54.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf8006ece0 msgr2=0x7fcf8015de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:58:54.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf7dd9b700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf8006ece0 0x7fcf8015de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.031+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 >> v1:172.21.15.191:6789/0 conn(0x7fcf8006e910 legacy=0x7fcf8014f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:58:54.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8006e1d0 con 0x7fcf8014fd10 2023-10-10T18:58:54.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fcf74001470 con 0x7fcf8014fd10 2023-10-10T18:58:54.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 >> v1:172.21.15.175:6789/0 conn(0x7fcf8014fd10 legacy=0x7fcf800c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:54.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf68005010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:54.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf68005920 0x7fcf68005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:54.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf8006ece0 0x7fcf6800cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:54.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800c4620 con 0x7fcf68004c20 2023-10-10T18:58:54.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf680066e0 con 0x7fcf68005920 2023-10-10T18:58:54.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6800d0b0 con 0x7fcf8006ece0 2023-10-10T18:58:54.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7ed9d700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf68005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:54.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7ed9d700 1 -- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf8006ece0 msgr2=0x7fcf6800cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:58:54.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7f59e700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf8006ece0 0x7fcf6800cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:54.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7ed9d700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf8006ece0 0x7fcf6800cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7ed9d700 1 -- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf68005920 msgr2=0x7fcf68005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:58:54.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7ed9d700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf68005920 0x7fcf68005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7ed9d700 1 -- 172.21.15.175:0/2499549533 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8006e1d0 con 0x7fcf68004c20 2023-10-10T18:58:54.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7f59e700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf8006ece0 0x7fcf6800cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:58:54.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7ed9d700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf68005010 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcf70002820 tx=0x7fcf70007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202bb1f628b2f768 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:54.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70006820 con 0x7fcf68004c20 2023-10-10T18:58:54.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf700069a0 con 0x7fcf68004c20 2023-10-10T18:58:54.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.032+0000 7fcf7dd9b700 1 -- 172.21.15.175:0/2499549533 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70006d60 con 0x7fcf68004c20 2023-10-10T18:58:54.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.033+0000 7fcf85046700 1 -- 172.21.15.175:0/2499549533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 msgr2=0x7fcf68005010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:54.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.033+0000 7fcf85046700 1 --2- 172.21.15.175:0/2499549533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf68005010 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcf70002820 tx=0x7fcf70007d30 comp rx=0 tx=0).stop 2023-10-10T18:58:54.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.033+0000 7fcf7f59e700 1 -- 172.21.15.175:0/2499549533 reap_dead start 2023-10-10T18:58:54.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.033+0000 7fcf85046700 1 -- 172.21.15.175:0/2499549533 shutdown_connections 2023-10-10T18:58:54.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.033+0000 7fcf85046700 1 -- 172.21.15.175:0/2499549533 >> 172.21.15.175:0/2499549533 conn(0x7fcf800c5650 msgr2=0x7fcf800c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:58:54.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.033+0000 7fcf85046700 1 -- 172.21.15.175:0/2499549533 shutdown_connections 2023-10-10T18:58:54.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.033+0000 7fcf85046700 1 -- 172.21.15.175:0/2499549533 wait complete. 2023-10-10T18:58:54.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.034+0000 7fcf85046700 1 Processor -- start 2023-10-10T18:58:54.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.034+0000 7fcf85046700 1 -- start start 2023-10-10T18:58:54.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf85046700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf801634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:54.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf85046700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf801639e0 0x7fcf8014a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf85046700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf8014a590 0x7fcf80165700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf85046700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80160ab0 con 0x7fcf68004c20 2023-10-10T18:58:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf85046700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80161910 con 0x7fcf8014a590 2023-10-10T18:58:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf85046700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80161da0 con 0x7fcf801639e0 2023-10-10T18:58:54.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf801634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf801634a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37740/0 (socket says 172.21.15.175:37740) 2023-10-10T18:58:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7ed9d700 1 -- 172.21.15.175:0/4204974572 learned_addr learned my addr 172.21.15.175:0/4204974572 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:58:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7e59c700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf801639e0 0x7fcf8014a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7ed9d700 1 -- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf801639e0 msgr2=0x7fcf8014a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7ed9d700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf801639e0 0x7fcf8014a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7ed9d700 1 -- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf8014a590 msgr2=0x7fcf80165700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7ed9d700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf8014a590 0x7fcf80165700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.035+0000 7fcf7ed9d700 1 -- 172.21.15.175:0/4204974572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf80160d00 con 0x7fcf68004c20 2023-10-10T18:58:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.036+0000 7fcf7ed9d700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf801634a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcf70002e10 tx=0x7fcf70015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.036+0000 7fcf66ffd700 1 -- 172.21.15.175:0/4204974572 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7001e660 con 0x7fcf68004c20 2023-10-10T18:58:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.036+0000 7fcf66ffd700 1 -- 172.21.15.175:0/4204974572 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7001e7c0 con 0x7fcf68004c20 2023-10-10T18:58:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.036+0000 7fcf66ffd700 1 -- 172.21.15.175:0/4204974572 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7001eb80 con 0x7fcf68004c20 2023-10-10T18:58:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.036+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf80160fe0 con 0x7fcf68004c20 2023-10-10T18:58:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.036+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf801615a0 con 0x7fcf68004c20 2023-10-10T18:58:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.037+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf80161760 con 0x7fcf68004c20 2023-10-10T18:58:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.037+0000 7fcf66ffd700 1 -- 172.21.15.175:0/4204974572 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcf7001a410 con 0x7fcf68004c20 2023-10-10T18:58:54.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.038+0000 7fcf66ffd700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf6c066f60 0x7fcf6c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:54.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.038+0000 7fcf66ffd700 1 -- 172.21.15.175:0/4204974572 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fcf70086470 con 0x7fcf68004c20 2023-10-10T18:58:54.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.038+0000 7fcf7e59c700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf6c066f60 0x7fcf6c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:54.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.038+0000 7fcf7e59c700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf6c066f60 0x7fcf6c069410 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcf740010e0 tx=0x7fcf74000d70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:54.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.043+0000 7fcf66ffd700 1 -- 172.21.15.175:0/4204974572 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf70053460 con 0x7fcf68004c20 2023-10-10T18:58:54.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.304+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf80160ab0 con 0x7fcf68004c20 2023-10-10T18:58:54.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.305+0000 7fcf66ffd700 1 -- 172.21.15.175:0/4204974572 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6196 (secure 0 0 0) 0x7fcf70056a80 con 0x7fcf68004c20 2023-10-10T18:58:54.305 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:58:54.305 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5584,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:58:54.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf6c066f60 msgr2=0x7fcf6c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:54.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf6c066f60 0x7fcf6c069410 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcf740010e0 tx=0x7fcf74000d70 comp rx=0 tx=0).stop 2023-10-10T18:58:54.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 msgr2=0x7fcf801634a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:54.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf801634a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcf70002e10 tx=0x7fcf70015000 comp rx=0 tx=0).stop 2023-10-10T18:58:54.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 shutdown_connections 2023-10-10T18:58:54.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf6c066f60 0x7fcf6c069410 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf8014a590 0x7fcf80165700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf68004c20 0x7fcf801634a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 --2- 172.21.15.175:0/4204974572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf801639e0 0x7fcf8014a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:54.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 >> 172.21.15.175:0/4204974572 conn(0x7fcf800c5650 msgr2=0x7fcf800c2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:58:54.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 shutdown_connections 2023-10-10T18:58:54.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:54.307+0000 7fcf85046700 1 -- 172.21.15.175:0/4204974572 wait complete. 2023-10-10T18:58:54.309 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 99 2023-10-10T18:58:54.319 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:58:56.319 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:58:56.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.444+0000 7fbde4197700 1 Processor -- start 2023-10-10T18:58:56.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.444+0000 7fbde4197700 1 -- start start 2023-10-10T18:58:56.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.445+0000 7fbde4197700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbddc0b2390 0x7fbddc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:56.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.445+0000 7fbde4197700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0aff60 0x7fbddc0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:56.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.445+0000 7fbde4197700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc1574b0 con 0x7fbddc0b1fc0 2023-10-10T18:58:56.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.445+0000 7fbde4197700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbddc150770 con 0x7fbddc0b2390 2023-10-10T18:58:56.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.445+0000 7fbde4197700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0afad0 con 0x7fbddc0aff60 2023-10-10T18:58:56.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.445+0000 7fbde1732700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbddc0b2390 0x7fbddc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:56.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.445+0000 7fbde1f33700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbddc0b1fc0 0x7fbddc068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:59608/0 (socket says 172.21.15.175:59608) 2023-10-10T18:58:56.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.445+0000 7fbde1f33700 1 -- 172.21.15.175:0/2967130641 learned_addr learned my addr 172.21.15.175:0/2967130641 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:58:56.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde2734700 1 --2- 172.21.15.175:0/2967130641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0aff60 0x7fbddc0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:56.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde1732700 1 -- 172.21.15.175:0/2967130641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0aff60 msgr2=0x7fbddc0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:58:56.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde1732700 1 --2- 172.21.15.175:0/2967130641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0aff60 0x7fbddc0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde1732700 1 -- 172.21.15.175:0/2967130641 >> v1:172.21.15.175:6789/0 conn(0x7fbddc0b1fc0 legacy=0x7fbddc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:56.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde0f31700 1 -- 172.21.15.175:0/2967130641 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116319365 0 0) 0x7fbddc1574b0 con 0x7fbddc0b1fc0 2023-10-10T18:58:56.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde1732700 1 -- 172.21.15.175:0/2967130641 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc0af4c0 con 0x7fbddc0b2390 2023-10-10T18:58:56.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde1732700 1 --2- 172.21.15.175:0/2967130641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0b2390 0x7fbddc155fb0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0098e0 tx=0x7fbdcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28774a30bd7645d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:56.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde0f31700 1 -- 172.21.15.175:0/2967130641 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc00b6d0 con 0x7fbddc0b2390 2023-10-10T18:58:56.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde0f31700 1 -- 172.21.15.175:0/2967130641 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdcc00b830 con 0x7fbddc0b2390 2023-10-10T18:58:56.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde0f31700 1 -- 172.21.15.175:0/2967130641 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc00bbd0 con 0x7fbddc0b2390 2023-10-10T18:58:56.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde4197700 1 -- 172.21.15.175:0/2967130641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0b2390 msgr2=0x7fbddc155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:56.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.446+0000 7fbde4197700 1 --2- 172.21.15.175:0/2967130641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0b2390 0x7fbddc155fb0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0098e0 tx=0x7fbdcc0091a0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.447+0000 7fbde4197700 1 -- 172.21.15.175:0/2967130641 shutdown_connections 2023-10-10T18:58:56.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.447+0000 7fbde4197700 1 --2- 172.21.15.175:0/2967130641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0aff60 0x7fbddc0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.447+0000 7fbde4197700 1 --2- 172.21.15.175:0/2967130641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0b2390 0x7fbddc155fb0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.447+0000 7fbde4197700 1 -- 172.21.15.175:0/2967130641 >> 172.21.15.175:0/2967130641 conn(0x7fbddc063230 msgr2=0x7fbddc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:58:56.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.447+0000 7fbde4197700 1 -- 172.21.15.175:0/2967130641 shutdown_connections 2023-10-10T18:58:56.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.447+0000 7fbde4197700 1 -- 172.21.15.175:0/2967130641 wait complete. 2023-10-10T18:58:56.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.448+0000 7fbde4197700 1 Processor -- start 2023-10-10T18:58:56.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.448+0000 7fbde4197700 1 -- start start 2023-10-10T18:58:56.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.448+0000 7fbde4197700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0aff60 0x7fbddc159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:56.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.448+0000 7fbde4197700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbddc0b1fc0 0x7fbddc0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:56.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.448+0000 7fbde4197700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0b2390 0x7fbddc0c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:56.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.448+0000 7fbde4197700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc15a180 con 0x7fbddc0aff60 2023-10-10T18:58:56.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.448+0000 7fbde4197700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0aece0 con 0x7fbddc0b1fc0 2023-10-10T18:58:56.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.448+0000 7fbde4197700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0ae720 con 0x7fbddc0b2390 2023-10-10T18:58:56.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0aff60 0x7fbddc159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:56.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0aff60 0x7fbddc159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38884/0 (socket says 172.21.15.175:38884) 2023-10-10T18:58:56.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 -- 172.21.15.175:0/2024471253 learned_addr learned my addr 172.21.15.175:0/2024471253 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:58:56.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde2734700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0b2390 0x7fbddc0c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:56.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1732700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbddc0b1fc0 0x7fbddc0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:56.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 -- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0b2390 msgr2=0x7fbddc0c47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:56.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0b2390 0x7fbddc0c47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 -- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbddc0b1fc0 msgr2=0x7fbddc0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:56.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbddc0b1fc0 0x7fbddc0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 -- 172.21.15.175:0/2024471253 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc009420 con 0x7fbddc0aff60 2023-10-10T18:58:56.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde1f33700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0aff60 0x7fbddc159900 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800c7e0 tx=0x7fbdd8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:56.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbdd1ffb700 1 -- 172.21.15.175:0/2024471253 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8005b30 con 0x7fbddc0aff60 2023-10-10T18:58:56.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbdd1ffb700 1 -- 172.21.15.175:0/2024471253 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd8005c90 con 0x7fbddc0aff60 2023-10-10T18:58:56.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc16a210 con 0x7fbddc0aff60 2023-10-10T18:58:56.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbdd1ffb700 1 -- 172.21.15.175:0/2024471253 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80072e0 con 0x7fbddc0aff60 2023-10-10T18:58:56.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.449+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc0ae970 con 0x7fbddc0aff60 2023-10-10T18:58:56.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.450+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbddc063e90 con 0x7fbddc0aff60 2023-10-10T18:58:56.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.451+0000 7fbdd1ffb700 1 -- 172.21.15.175:0/2024471253 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbdd80076d0 con 0x7fbddc0aff60 2023-10-10T18:58:56.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.451+0000 7fbdd1ffb700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdc0066f30 0x7fbdc00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:56.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.451+0000 7fbdd1ffb700 1 -- 172.21.15.175:0/2024471253 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fbdd8083c30 con 0x7fbddc0aff60 2023-10-10T18:58:56.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.451+0000 7fbde1732700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdc0066f30 0x7fbdc00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:56.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.452+0000 7fbde1732700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdc0066f30 0x7fbdc00693e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0093f0 tx=0x7fbdcc017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:56.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.455+0000 7fbdd1ffb700 1 -- 172.21.15.175:0/2024471253 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbdd804b5a0 con 0x7fbddc0aff60 2023-10-10T18:58:56.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.690+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbddc067570 con 0x7fbddc0aff60 2023-10-10T18:58:56.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.690+0000 7fbdd1ffb700 1 -- 172.21.15.175:0/2024471253 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6196 (secure 0 0 0) 0x7fbdd8017090 con 0x7fbddc0aff60 2023-10-10T18:58:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:58:56.691 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5584,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:58:56.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.692+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdc0066f30 msgr2=0x7fbdc00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:56.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.692+0000 7fbde4197700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdc0066f30 0x7fbdc00693e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0093f0 tx=0x7fbdcc017000 comp rx=0 tx=0).stop 2023-10-10T18:58:56.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.692+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0aff60 msgr2=0x7fbddc159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:56.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0aff60 0x7fbddc159900 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800c7e0 tx=0x7fbdd8007df0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 shutdown_connections 2023-10-10T18:58:56.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdc0066f30 0x7fbdc00693e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbddc0b2390 0x7fbddc0c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbddc0aff60 0x7fbddc159900 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 --2- 172.21.15.175:0/2024471253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbddc0b1fc0 0x7fbddc0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:56.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 >> 172.21.15.175:0/2024471253 conn(0x7fbddc063230 msgr2=0x7fbddc14fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:58:56.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 shutdown_connections 2023-10-10T18:58:56.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:56.693+0000 7fbde4197700 1 -- 172.21.15.175:0/2024471253 wait complete. 2023-10-10T18:58:56.696 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 99 2023-10-10T18:58:56.705 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:58:58.705 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:58:58.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.834+0000 7f707fed3700 1 Processor -- start 2023-10-10T18:58:58.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.834+0000 7f707fed3700 1 -- start start 2023-10-10T18:58:58.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707fed3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f70780b2390 0x7f7078155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:58.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707fed3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f70780aff60 0x7f70780b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:58.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707fed3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70781574b0 con 0x7f70780b1fc0 2023-10-10T18:58:58.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707fed3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7078150770 con 0x7f70780aff60 2023-10-10T18:58:58.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707fed3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70780afad0 con 0x7f70780b2390 2023-10-10T18:58:58.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707e470700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f70780aff60 0x7f70780b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:58.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707dc6f700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f70780b1fc0 0x7f7078068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:59614/0 (socket says 172.21.15.175:59614) 2023-10-10T18:58:58.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707dc6f700 1 -- 172.21.15.175:0/1023874971 learned_addr learned my addr 172.21.15.175:0/1023874971 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:58:58.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.835+0000 7f707d46e700 1 --2- 172.21.15.175:0/1023874971 >> v2:172.21.15.191:3300/0 conn(0x7f70780b2390 0x7f7078155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:58.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707e470700 1 -- 172.21.15.175:0/1023874971 >> v2:172.21.15.191:3300/0 conn(0x7f70780b2390 msgr2=0x7f7078155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:58:58.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707e470700 1 --2- 172.21.15.175:0/1023874971 >> v2:172.21.15.191:3300/0 conn(0x7f70780b2390 0x7f7078155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:58.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707e470700 1 -- 172.21.15.175:0/1023874971 >> v1:172.21.15.175:6789/0 conn(0x7f70780b1fc0 legacy=0x7f7078068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:58.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707e470700 1 -- 172.21.15.175:0/1023874971 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70780af4c0 con 0x7f70780aff60 2023-10-10T18:58:58.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707cc6d700 1 -- 172.21.15.175:0/1023874971 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112648359 0 0) 0x7f70781574b0 con 0x7f70780b1fc0 2023-10-10T18:58:58.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707e470700 1 --2- 172.21.15.175:0/1023874971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780aff60 0x7f70780b03e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f70780c0290 tx=0x7f7070009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77841f892890b4f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:58.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707cc6d700 1 -- 172.21.15.175:0/1023874971 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707000e070 con 0x7f70780aff60 2023-10-10T18:58:58.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707cc6d700 1 -- 172.21.15.175:0/1023874971 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707000bcc0 con 0x7f70780aff60 2023-10-10T18:58:58.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707cc6d700 1 -- 172.21.15.175:0/1023874971 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070012680 con 0x7f70780aff60 2023-10-10T18:58:58.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707fed3700 1 -- 172.21.15.175:0/1023874971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780aff60 msgr2=0x7f70780b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:58.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707fed3700 1 --2- 172.21.15.175:0/1023874971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780aff60 0x7f70780b03e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f70780c0290 tx=0x7f7070009780 comp rx=0 tx=0).stop 2023-10-10T18:58:58.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707fed3700 1 -- 172.21.15.175:0/1023874971 shutdown_connections 2023-10-10T18:58:58.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707fed3700 1 --2- 172.21.15.175:0/1023874971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780aff60 0x7f70780b03e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:58.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707fed3700 1 --2- 172.21.15.175:0/1023874971 >> v2:172.21.15.191:3300/0 conn(0x7f70780b2390 0x7f7078155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:58.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707fed3700 1 -- 172.21.15.175:0/1023874971 >> 172.21.15.175:0/1023874971 conn(0x7f7078063230 msgr2=0x7f7078063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:58:58.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707fed3700 1 -- 172.21.15.175:0/1023874971 shutdown_connections 2023-10-10T18:58:58.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.836+0000 7f707fed3700 1 -- 172.21.15.175:0/1023874971 wait complete. 2023-10-10T18:58:58.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.837+0000 7f707fed3700 1 Processor -- start 2023-10-10T18:58:58.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707fed3700 1 -- start start 2023-10-10T18:58:58.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707fed3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70780aff60 0x7f7078155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:58.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707fed3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1fc0 0x7f707814fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:58.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707fed3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780b2390 0x7f7078150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:58.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707fed3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7078169a20 con 0x7f70780b2390 2023-10-10T18:58:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707fed3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7078168c30 con 0x7f70780aff60 2023-10-10T18:58:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707fed3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7078168e90 con 0x7f70780b1fc0 2023-10-10T18:58:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707e470700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780b2390 0x7f7078150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707e470700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780b2390 0x7f7078150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38896/0 (socket says 172.21.15.175:38896) 2023-10-10T18:58:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707e470700 1 -- 172.21.15.175:0/1228057189 learned_addr learned my addr 172.21.15.175:0/1228057189 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:58:58.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707dc6f700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70780aff60 0x7f7078155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:58.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.838+0000 7f707d46e700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1fc0 0x7f707814fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:58.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f707e470700 1 -- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1fc0 msgr2=0x7f707814fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:58.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f707e470700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1fc0 0x7f707814fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:58.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f707e470700 1 -- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70780aff60 msgr2=0x7f7078155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:58.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f707e470700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70780aff60 0x7f7078155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:58.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f707e470700 1 -- 172.21.15.175:0/1228057189 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070009a20 con 0x7f70780b2390 2023-10-10T18:58:58.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f707e470700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780b2390 0x7f7078150060 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f707806fd90 tx=0x7f7070019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:58.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f706dffb700 1 -- 172.21.15.175:0/1228057189 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070012e10 con 0x7f70780b2390 2023-10-10T18:58:58.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f706dffb700 1 -- 172.21.15.175:0/1228057189 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707001d630 con 0x7f70780b2390 2023-10-10T18:58:58.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7078169130 con 0x7f70780b2390 2023-10-10T18:58:58.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f706dffb700 1 -- 172.21.15.175:0/1228057189 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707001d9f0 con 0x7f70780b2390 2023-10-10T18:58:58.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.839+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70781696b0 con 0x7f70780b2390 2023-10-10T18:58:58.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.840+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7078067570 con 0x7f70780b2390 2023-10-10T18:58:58.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.841+0000 7f706dffb700 1 -- 172.21.15.175:0/1228057189 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7070024070 con 0x7f70780b2390 2023-10-10T18:58:58.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.841+0000 7f706dffb700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f705c066f60 0x7f705c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:58:58.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.841+0000 7f706dffb700 1 -- 172.21.15.175:0/1228057189 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f7070086f80 con 0x7f70780b2390 2023-10-10T18:58:58.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.841+0000 7f707dc6f700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f705c066f60 0x7f705c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:58:58.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.842+0000 7f707dc6f700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f705c066f60 0x7f705c069410 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7074007fd0 tx=0x7f7074007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:58:58.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:58.845+0000 7f706dffb700 1 -- 172.21.15.175:0/1228057189 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7070053f70 con 0x7f70780b2390 2023-10-10T18:58:59.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.070+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7078068da0 con 0x7f70780b2390 2023-10-10T18:58:59.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.071+0000 7f706dffb700 1 -- 172.21.15.175:0/1228057189 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6196 (secure 0 0 0) 0x7f707001c3a0 con 0x7f70780b2390 2023-10-10T18:58:59.071 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:58:59.072 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5584,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:58:59.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f705c066f60 msgr2=0x7f705c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:59.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f705c066f60 0x7f705c069410 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7074007fd0 tx=0x7f7074007d30 comp rx=0 tx=0).stop 2023-10-10T18:58:59.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780b2390 msgr2=0x7f7078150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:58:59.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780b2390 0x7f7078150060 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f707806fd90 tx=0x7f7070019000 comp rx=0 tx=0).stop 2023-10-10T18:58:59.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 shutdown_connections 2023-10-10T18:58:59.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f705c066f60 0x7f705c069410 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:59.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70780b2390 0x7f7078150060 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:59.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70780aff60 0x7f7078155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:59.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 --2- 172.21.15.175:0/1228057189 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70780b1fc0 0x7f707814fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:58:59.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 >> 172.21.15.175:0/1228057189 conn(0x7f7078063230 msgr2=0x7f70781508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:58:59.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 shutdown_connections 2023-10-10T18:58:59.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:58:59.073+0000 7f707fed3700 1 -- 172.21.15.175:0/1228057189 wait complete. 2023-10-10T18:58:59.076 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 99 2023-10-10T18:58:59.085 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:59:01.086 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:01.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.211+0000 7ff9d958d700 1 Processor -- start 2023-10-10T18:59:01.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.211+0000 7ff9d958d700 1 -- start start 2023-10-10T18:59:01.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d958d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff9d40b1fc0 0x7ff9d4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:01.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d958d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 0x7ff9d4068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:01.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d958d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d41574b0 con 0x7ff9d4068bd0 2023-10-10T18:59:01.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d958d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d4150770 con 0x7ff9d40b1fc0 2023-10-10T18:59:01.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d958d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40afad0 con 0x7ff9d40b2390 2023-10-10T18:59:01.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d2ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff9d40b1fc0 0x7ff9d4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:01.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d37fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff9d4068bd0 0x7ff9d40aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:59626/0 (socket says 172.21.15.175:59626) 2023-10-10T18:59:01.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d37fe700 1 -- 172.21.15.175:0/441094054 learned_addr learned my addr 172.21.15.175:0/441094054 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:01.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9ca7fc700 1 --2- 172.21.15.175:0/441094054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 0x7ff9d4068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:01.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d2ffd700 1 -- 172.21.15.175:0/441094054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 msgr2=0x7ff9d4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:01.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d2ffd700 1 --2- 172.21.15.175:0/441094054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 0x7ff9d4068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d2ffd700 1 -- 172.21.15.175:0/441094054 >> v1:172.21.15.175:6789/0 conn(0x7ff9d4068bd0 legacy=0x7ff9d40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:01.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d27fc700 1 -- 172.21.15.175:0/441094054 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469609164 0 0) 0x7ff9d41574b0 con 0x7ff9d4068bd0 2023-10-10T18:59:01.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d2ffd700 1 -- 172.21.15.175:0/441094054 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d40af4c0 con 0x7ff9d40b1fc0 2023-10-10T18:59:01.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.212+0000 7ff9d2ffd700 1 --2- 172.21.15.175:0/441094054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d40b1fc0 0x7ff9d4155fb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40098e0 tx=0x7ff9c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9886941cf4dac85 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:01.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d27fc700 1 -- 172.21.15.175:0/441094054 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4009e80 con 0x7ff9d40b1fc0 2023-10-10T18:59:01.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d27fc700 1 -- 172.21.15.175:0/441094054 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c400b8f0 con 0x7ff9d40b1fc0 2023-10-10T18:59:01.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d27fc700 1 -- 172.21.15.175:0/441094054 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400bc70 con 0x7ff9d40b1fc0 2023-10-10T18:59:01.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d958d700 1 -- 172.21.15.175:0/441094054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d40b1fc0 msgr2=0x7ff9d4155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:01.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d958d700 1 --2- 172.21.15.175:0/441094054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d40b1fc0 0x7ff9d4155fb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40098e0 tx=0x7ff9c40091a0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d958d700 1 -- 172.21.15.175:0/441094054 shutdown_connections 2023-10-10T18:59:01.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d958d700 1 --2- 172.21.15.175:0/441094054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d40b1fc0 0x7ff9d4155fb0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d958d700 1 --2- 172.21.15.175:0/441094054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 0x7ff9d4068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d958d700 1 -- 172.21.15.175:0/441094054 >> 172.21.15.175:0/441094054 conn(0x7ff9d4063230 msgr2=0x7ff9d4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:01.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d958d700 1 -- 172.21.15.175:0/441094054 shutdown_connections 2023-10-10T18:59:01.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.213+0000 7ff9d958d700 1 -- 172.21.15.175:0/441094054 wait complete. 2023-10-10T18:59:01.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.214+0000 7ff9d958d700 1 Processor -- start 2023-10-10T18:59:01.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d958d700 1 -- start start 2023-10-10T18:59:01.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d958d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d4068bd0 0x7ff9d4151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:01.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d958d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d40b1fc0 0x7ff9d40c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:01.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d958d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 0x7ff9d40c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:01.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d958d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d4158b80 con 0x7ff9d4068bd0 2023-10-10T18:59:01.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d958d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40cd210 con 0x7ff9d40b1fc0 2023-10-10T18:59:01.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d958d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40c6280 con 0x7ff9d40b2390 2023-10-10T18:59:01.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d4068bd0 0x7ff9d4151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:01.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d4068bd0 0x7ff9d4151100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38918/0 (socket says 172.21.15.175:38918) 2023-10-10T18:59:01.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9d2ffd700 1 -- 172.21.15.175:0/3570277701 learned_addr learned my addr 172.21.15.175:0/3570277701 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:01.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.215+0000 7ff9ca7fc700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d40b1fc0 0x7ff9d40c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:01.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d2ffd700 1 -- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 msgr2=0x7ff9d40c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:01.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d37fe700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 0x7ff9d40c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:01.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d2ffd700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 0x7ff9d40c4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d2ffd700 1 -- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d40b1fc0 msgr2=0x7ff9d40c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:01.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d2ffd700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d40b1fc0 0x7ff9d40c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d2ffd700 1 -- 172.21.15.175:0/3570277701 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c4009420 con 0x7ff9d4068bd0 2023-10-10T18:59:01.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9ca7fc700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d40b1fc0 0x7ff9d40c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:59:01.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d2ffd700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d4068bd0 0x7ff9d4151100 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400b3c0 tx=0x7ff9c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:01.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9cb7fe700 1 -- 172.21.15.175:0/3570277701 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400bc30 con 0x7ff9d4068bd0 2023-10-10T18:59:01.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9cb7fe700 1 -- 172.21.15.175:0/3570277701 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c4025420 con 0x7ff9d4068bd0 2023-10-10T18:59:01.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d40c6560 con 0x7ff9d4068bd0 2023-10-10T18:59:01.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9cb7fe700 1 -- 172.21.15.175:0/3570277701 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40257c0 con 0x7ff9d4068bd0 2023-10-10T18:59:01.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.216+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d4158810 con 0x7ff9d4068bd0 2023-10-10T18:59:01.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.217+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d4067570 con 0x7ff9d4068bd0 2023-10-10T18:59:01.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.217+0000 7ff9cb7fe700 1 -- 172.21.15.175:0/3570277701 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff9c4023070 con 0x7ff9d4068bd0 2023-10-10T18:59:01.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.218+0000 7ff9cb7fe700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9b4066f30 0x7ff9b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:01.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.218+0000 7ff9cb7fe700 1 -- 172.21.15.175:0/3570277701 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7ff9c4086000 con 0x7ff9d4068bd0 2023-10-10T18:59:01.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.218+0000 7ff9ca7fc700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9b4066f30 0x7ff9b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:01.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.218+0000 7ff9ca7fc700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9b4066f30 0x7ff9b40693e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc006fd0 tx=0x7ff9bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:01.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.222+0000 7ff9cb7fe700 1 -- 172.21.15.175:0/3570277701 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff9c4052ff0 con 0x7ff9d4068bd0 2023-10-10T18:59:01.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.447+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9d4069060 con 0x7ff9d4068bd0 2023-10-10T18:59:01.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.448+0000 7ff9cb7fe700 1 -- 172.21.15.175:0/3570277701 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6196 (secure 0 0 0) 0x7ff9c4056610 con 0x7ff9d4068bd0 2023-10-10T18:59:01.448 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:01.449 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5584,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:01.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9b4066f30 msgr2=0x7ff9b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:01.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9b4066f30 0x7ff9b40693e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc006fd0 tx=0x7ff9bc008040 comp rx=0 tx=0).stop 2023-10-10T18:59:01.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d4068bd0 msgr2=0x7ff9d4151100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:01.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d4068bd0 0x7ff9d4151100 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400b3c0 tx=0x7ff9c4017000 comp rx=0 tx=0).stop 2023-10-10T18:59:01.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 shutdown_connections 2023-10-10T18:59:01.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9b4066f30 0x7ff9b40693e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9d40b2390 0x7ff9d40c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9d4068bd0 0x7ff9d4151100 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 --2- 172.21.15.175:0/3570277701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d40b1fc0 0x7ff9d40c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:01.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 >> 172.21.15.175:0/3570277701 conn(0x7ff9d4063230 msgr2=0x7ff9d414fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:01.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 shutdown_connections 2023-10-10T18:59:01.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:01.450+0000 7ff9d958d700 1 -- 172.21.15.175:0/3570277701 wait complete. 2023-10-10T18:59:01.453 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 99 2023-10-10T18:59:01.462 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:59:03.463 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:03.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.588+0000 7f2ddb59e700 1 Processor -- start 2023-10-10T18:59:03.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.588+0000 7f2ddb59e700 1 -- start start 2023-10-10T18:59:03.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddb59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2ddc14fd90 0x7f2ddc159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:03.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddb59e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2ddc0cd040 0x7f2ddc0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:03.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddb59e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc067fa0 con 0x7f2ddc08a100 2023-10-10T18:59:03.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddb59e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc0c5600 con 0x7f2ddc0cd040 2023-10-10T18:59:03.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddb59e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc156b70 con 0x7f2ddc14fd90 2023-10-10T18:59:03.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddad9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2ddc0cd040 0x7f2ddc0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:03.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2dd1d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2ddc08a100 0x7f2ddc0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:59628/0 (socket says 172.21.15.175:59628) 2023-10-10T18:59:03.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2dd1d9b700 1 -- 172.21.15.175:0/3554488466 learned_addr learned my addr 172.21.15.175:0/3554488466 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:03.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2dda59c700 1 --2- 172.21.15.175:0/3554488466 >> v2:172.21.15.191:3300/0 conn(0x7f2ddc14fd90 0x7f2ddc159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:03.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddad9d700 1 -- 172.21.15.175:0/3554488466 >> v2:172.21.15.191:3300/0 conn(0x7f2ddc14fd90 msgr2=0x7f2ddc159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:03.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddad9d700 1 --2- 172.21.15.175:0/3554488466 >> v2:172.21.15.191:3300/0 conn(0x7f2ddc14fd90 0x7f2ddc159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddad9d700 1 -- 172.21.15.175:0/3554488466 >> v1:172.21.15.175:6789/0 conn(0x7f2ddc08a100 legacy=0x7f2ddc0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:03.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddad9d700 1 -- 172.21.15.175:0/3554488466 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc150160 con 0x7f2ddc0cd040 2023-10-10T18:59:03.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2dda59c700 1 --2- 172.21.15.175:0/3554488466 >> v2:172.21.15.191:3300/0 conn(0x7f2ddc14fd90 0x7f2ddc159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:59:03.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2ddad9d700 1 --2- 172.21.15.175:0/3554488466 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc0cd040 0x7f2ddc0684f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc065080 tx=0x7f2dd4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c648196035d85b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:03.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.589+0000 7f2dd9d9b700 1 -- 172.21.15.175:0/3554488466 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd400e070 con 0x7f2ddc0cd040 2023-10-10T18:59:03.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2dd9d9b700 1 -- 172.21.15.175:0/3554488466 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd400bcc0 con 0x7f2ddc0cd040 2023-10-10T18:59:03.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2dd9d9b700 1 -- 172.21.15.175:0/3554488466 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4012680 con 0x7f2ddc0cd040 2023-10-10T18:59:03.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2ddb59e700 1 -- 172.21.15.175:0/3554488466 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc0cd040 msgr2=0x7f2ddc0684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:03.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/3554488466 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc0cd040 0x7f2ddc0684f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc065080 tx=0x7f2dd4009780 comp rx=0 tx=0).stop 2023-10-10T18:59:03.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2ddb59e700 1 -- 172.21.15.175:0/3554488466 shutdown_connections 2023-10-10T18:59:03.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/3554488466 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc0cd040 0x7f2ddc0684f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/3554488466 >> v2:172.21.15.191:3300/0 conn(0x7f2ddc14fd90 0x7f2ddc159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2ddb59e700 1 -- 172.21.15.175:0/3554488466 >> 172.21.15.175:0/3554488466 conn(0x7f2ddc063d00 msgr2=0x7f2ddc0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:03.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2ddb59e700 1 -- 172.21.15.175:0/3554488466 shutdown_connections 2023-10-10T18:59:03.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.590+0000 7f2ddb59e700 1 -- 172.21.15.175:0/3554488466 wait complete. 2023-10-10T18:59:03.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.591+0000 7f2ddb59e700 1 Processor -- start 2023-10-10T18:59:03.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.591+0000 7f2ddb59e700 1 -- start start 2023-10-10T18:59:03.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddb59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc08a100 0x7f2ddc151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:03.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddb59e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc0cd040 0x7f2ddc0cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:03.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddb59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc14fd90 0x7f2ddc14c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:03.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddb59e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc166970 con 0x7f2ddc14fd90 2023-10-10T18:59:03.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddb59e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc154500 con 0x7f2ddc0cd040 2023-10-10T18:59:03.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddb59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc166e40 con 0x7f2ddc08a100 2023-10-10T18:59:03.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc14fd90 0x7f2ddc14c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:03.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc14fd90 0x7f2ddc14c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38932/0 (socket says 172.21.15.175:38932) 2023-10-10T18:59:03.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddad9d700 1 -- 172.21.15.175:0/2221370373 learned_addr learned my addr 172.21.15.175:0/2221370373 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:03.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2dd1d9b700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc0cd040 0x7f2ddc0cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:03.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2dda59c700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc08a100 0x7f2ddc151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:03.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddad9d700 1 -- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc08a100 msgr2=0x7f2ddc151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:03.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddad9d700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc08a100 0x7f2ddc151950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddad9d700 1 -- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc0cd040 msgr2=0x7f2ddc0cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:03.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddad9d700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc0cd040 0x7f2ddc0cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2ddad9d700 1 -- 172.21.15.175:0/2221370373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4009a20 con 0x7f2ddc14fd90 2023-10-10T18:59:03.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.592+0000 7f2dd1d9b700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc0cd040 0x7f2ddc0cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:59:03.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.593+0000 7f2ddad9d700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc14fd90 0x7f2ddc14c5c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400b6c0 tx=0x7f2dd4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:03.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.593+0000 7f2dd2ffd700 1 -- 172.21.15.175:0/2221370373 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd401e700 con 0x7f2ddc14fd90 2023-10-10T18:59:03.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.593+0000 7f2dd2ffd700 1 -- 172.21.15.175:0/2221370373 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd401e8a0 con 0x7f2ddc14fd90 2023-10-10T18:59:03.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.593+0000 7f2dd2ffd700 1 -- 172.21.15.175:0/2221370373 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd401ec60 con 0x7f2ddc14fd90 2023-10-10T18:59:03.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.593+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc153c10 con 0x7f2ddc14fd90 2023-10-10T18:59:03.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.593+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc153eb0 con 0x7f2ddc14fd90 2023-10-10T18:59:03.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.594+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc151610 con 0x7f2ddc14fd90 2023-10-10T18:59:03.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.594+0000 7f2dd2ffd700 1 -- 172.21.15.175:0/2221370373 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2dd4024070 con 0x7f2ddc14fd90 2023-10-10T18:59:03.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.594+0000 7f2dd2ffd700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2dbc066f60 0x7f2dbc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:03.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.594+0000 7f2dd2ffd700 1 -- 172.21.15.175:0/2221370373 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f2dd40871d0 con 0x7f2ddc14fd90 2023-10-10T18:59:03.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.595+0000 7f2dda59c700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2dbc066f60 0x7f2dbc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:03.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.595+0000 7f2dda59c700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2dbc066f60 0x7f2dbc069410 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc006fd0 tx=0x7f2dcc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:03.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.599+0000 7f2dd2ffd700 1 -- 172.21.15.175:0/2221370373 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2dd40541c0 con 0x7f2ddc14fd90 2023-10-10T18:59:03.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.827+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ddc154640 con 0x7f2ddc14fd90 2023-10-10T18:59:03.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.828+0000 7f2dd2ffd700 1 -- 172.21.15.175:0/2221370373 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6196 (secure 0 0 0) 0x7f2dd40577e0 con 0x7f2ddc14fd90 2023-10-10T18:59:03.828 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:03.829 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5584,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:03.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2dbc066f60 msgr2=0x7f2dbc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:03.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2dbc066f60 0x7f2dbc069410 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc006fd0 tx=0x7f2dcc008040 comp rx=0 tx=0).stop 2023-10-10T18:59:03.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc14fd90 msgr2=0x7f2ddc14c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:03.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc14fd90 0x7f2ddc14c5c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400b6c0 tx=0x7f2dd4019000 comp rx=0 tx=0).stop 2023-10-10T18:59:03.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 shutdown_connections 2023-10-10T18:59:03.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2dbc066f60 0x7f2dbc069410 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ddc14fd90 0x7f2ddc14c5c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc08a100 0x7f2ddc151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 --2- 172.21.15.175:0/2221370373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc0cd040 0x7f2ddc0cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:03.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 >> 172.21.15.175:0/2221370373 conn(0x7f2ddc063d00 msgr2=0x7f2ddc0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:03.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 shutdown_connections 2023-10-10T18:59:03.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:03.830+0000 7f2ddb59e700 1 -- 172.21.15.175:0/2221370373 wait complete. 2023-10-10T18:59:03.832 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 99 2023-10-10T18:59:03.842 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:59:05.843 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:05.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.970+0000 7f300a719700 1 Processor -- start 2023-10-10T18:59:05.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.970+0000 7f300a719700 1 -- start start 2023-10-10T18:59:05.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.971+0000 7f300a719700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f300414c840 0x7f300406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:05.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.971+0000 7f300a719700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004152970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.971+0000 7f300a719700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3004155ca0 con 0x7f300414cc10 2023-10-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.971+0000 7f300a719700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3004156b00 con 0x7f300414c840 2023-10-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.971+0000 7f300a719700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300406d700 con 0x7f30040c52a0 2023-10-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3008cb6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3003fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f300414c840 0x7f300406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3003fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f300414c840 0x7f300406e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:42400/0 (socket says 172.21.15.175:42400) 2023-10-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3008cb6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004152970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:47506/0 (socket says 172.21.15.175:47506) 2023-10-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3003fff700 1 -- 172.21.15.175:0/1476290620 learned_addr learned my addr 172.21.15.175:0/1476290620 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f30037fe700 1 --1- 172.21.15.175:0/1476290620 >> v1:172.21.15.191:6789/0 conn(0x7f300414cc10 0x7f30040c4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:56470/0 (socket says 172.21.15.175:56470) 2023-10-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3008cb6700 1 -- 172.21.15.175:0/1476290620 >> v2:172.21.15.191:3300/0 conn(0x7f300414c840 msgr2=0x7f300406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3008cb6700 1 --2- 172.21.15.175:0/1476290620 >> v2:172.21.15.191:3300/0 conn(0x7f300414c840 0x7f300406e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3008cb6700 1 -- 172.21.15.175:0/1476290620 >> v1:172.21.15.191:6789/0 conn(0x7f300414cc10 legacy=0x7f30040c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3008cb6700 1 -- 172.21.15.175:0/1476290620 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3004155ef0 con 0x7f30040c52a0 2023-10-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3003fff700 1 --2- 172.21.15.175:0/1476290620 >> v2:172.21.15.191:3300/0 conn(0x7f300414c840 0x7f300406e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3008cb6700 1 --2- 172.21.15.175:0/1476290620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004152970 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009e60 tx=0x7f2ffc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d87fae886b3bbb85 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3002ffd700 1 -- 172.21.15.175:0/1476290620 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc00bbf0 con 0x7f30040c52a0 2023-10-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3002ffd700 1 -- 172.21.15.175:0/1476290620 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ffc00bd50 con 0x7f30040c52a0 2023-10-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.972+0000 7f3002ffd700 1 -- 172.21.15.175:0/1476290620 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc011650 con 0x7f30040c52a0 2023-10-10T18:59:05.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.973+0000 7f300a719700 1 -- 172.21.15.175:0/1476290620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 msgr2=0x7f3004152970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:05.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.973+0000 7f300a719700 1 --2- 172.21.15.175:0/1476290620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004152970 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009e60 tx=0x7f2ffc009780 comp rx=0 tx=0).stop 2023-10-10T18:59:05.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.973+0000 7f300a719700 1 -- 172.21.15.175:0/1476290620 shutdown_connections 2023-10-10T18:59:05.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.973+0000 7f300a719700 1 --2- 172.21.15.175:0/1476290620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004152970 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:05.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.973+0000 7f300a719700 1 --2- 172.21.15.175:0/1476290620 >> v2:172.21.15.191:3300/0 conn(0x7f300414c840 0x7f300406e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:05.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.973+0000 7f300a719700 1 -- 172.21.15.175:0/1476290620 >> 172.21.15.175:0/1476290620 conn(0x7f3004068280 msgr2=0x7f300406af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:05.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.973+0000 7f300a719700 1 -- 172.21.15.175:0/1476290620 shutdown_connections 2023-10-10T18:59:05.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.973+0000 7f300a719700 1 -- 172.21.15.175:0/1476290620 wait complete. 2023-10-10T18:59:05.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.974+0000 7f300a719700 1 Processor -- start 2023-10-10T18:59:05.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.974+0000 7f300a719700 1 -- start start 2023-10-10T18:59:05.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f300a719700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004169ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:05.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f300a719700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f300414c840 0x7f30040c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:05.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f300a719700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300414cc10 0x7f3004152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:05.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f300a719700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3004169ff0 con 0x7f300414c840 2023-10-10T18:59:05.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f300a719700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30040c6ef0 con 0x7f300414cc10 2023-10-10T18:59:05.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f300a719700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3004154440 con 0x7f30040c52a0 2023-10-10T18:59:05.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f300414c840 0x7f30040c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:05.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f300414c840 0x7f30040c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:41574/0 (socket says 172.21.15.175:41574) 2023-10-10T18:59:05.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 -- 172.21.15.175:0/144729063 learned_addr learned my addr 172.21.15.175:0/144729063 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:05.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f3008cb6700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300414cc10 0x7f3004152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:05.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f3003fff700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004169ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:05.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 -- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 msgr2=0x7f3004169ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:05.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004169ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:05.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 -- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300414cc10 msgr2=0x7f3004152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:05.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300414cc10 0x7f3004152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:05.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 -- 172.21.15.175:0/144729063 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4009480 con 0x7f300414c840 2023-10-10T18:59:05.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.975+0000 7f30037fe700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f300414c840 0x7f30040c72a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c7c0 tx=0x7f2ff8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:05.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.976+0000 7f2febfff700 1 -- 172.21.15.175:0/144729063 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8005a70 con 0x7f300414c840 2023-10-10T18:59:05.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.976+0000 7f2febfff700 1 -- 172.21.15.175:0/144729063 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff8005bd0 con 0x7f300414c840 2023-10-10T18:59:05.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.976+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc009a20 con 0x7f300414c840 2023-10-10T18:59:05.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.976+0000 7f2febfff700 1 -- 172.21.15.175:0/144729063 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80070d0 con 0x7f300414c840 2023-10-10T18:59:05.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.976+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30041538c0 con 0x7f300414c840 2023-10-10T18:59:05.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.976+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3004067570 con 0x7f300414c840 2023-10-10T18:59:05.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.977+0000 7f2febfff700 1 -- 172.21.15.175:0/144729063 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2ff8007440 con 0x7f300414c840 2023-10-10T18:59:05.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.977+0000 7f2febfff700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ff0066f30 0x7f2ff00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:05.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.977+0000 7f2febfff700 1 -- 172.21.15.175:0/144729063 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f2ff8083c10 con 0x7f300414c840 2023-10-10T18:59:05.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.977+0000 7f3003fff700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ff0066f30 0x7f2ff00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:05.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.978+0000 7f3003fff700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ff0066f30 0x7f2ff00693e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009f30 tx=0x7f2ff4009220 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:05.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:05.981+0000 7f2febfff700 1 -- 172.21.15.175:0/144729063 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ff8050c00 con 0x7f300414c840 2023-10-10T18:59:06.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.206+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30040bf380 con 0x7f300414c840 2023-10-10T18:59:06.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.207+0000 7f2febfff700 1 -- 172.21.15.175:0/144729063 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6196 (secure 0 0 0) 0x7f2ff8017090 con 0x7f300414c840 2023-10-10T18:59:06.207 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:06.207 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5584,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:57:31.640861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":4760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4760":{"gid":4760,"name":"b","rank":4,"incarnation":34,"state":"up:active","state_seq":27,"addr":"172.21.15.191:6836/3199867402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":3199867402},{"type":"v1","addr":"172.21.15.191:6836","nonce":3199867402}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:06.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.208+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ff0066f30 msgr2=0x7f2ff00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:06.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.208+0000 7f300a719700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ff0066f30 0x7f2ff00693e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009f30 tx=0x7f2ff4009220 comp rx=0 tx=0).stop 2023-10-10T18:59:06.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f300414c840 msgr2=0x7f30040c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:06.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f300414c840 0x7f30040c72a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c7c0 tx=0x7f2ff8007f70 comp rx=0 tx=0).stop 2023-10-10T18:59:06.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 shutdown_connections 2023-10-10T18:59:06.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2ff0066f30 0x7f2ff00693e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:06.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f300414cc10 0x7f3004152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:06.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30040c52a0 0x7f3004169ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:06.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 --2- 172.21.15.175:0/144729063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f300414c840 0x7f30040c72a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:06.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 >> 172.21.15.175:0/144729063 conn(0x7f3004068280 msgr2=0x7f3004149ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:06.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 shutdown_connections 2023-10-10T18:59:06.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:06.209+0000 7f300a719700 1 -- 172.21.15.175:0/144729063 wait complete. 2023-10-10T18:59:06.212 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 99 2023-10-10T18:59:06.221 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T18:59:08.221 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:08.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.347+0000 7f4669ce6700 1 Processor -- start 2023-10-10T18:59:08.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.347+0000 7f4669ce6700 1 -- start start 2023-10-10T18:59:08.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4669ce6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46640b1fc0 0x7f4664155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4669ce6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f46640b2390 0x7f4664068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4669ce6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46641574b0 con 0x7f4664068bd0 2023-10-10T18:59:08.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4669ce6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4664150770 con 0x7f46640b2390 2023-10-10T18:59:08.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4669ce6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46640afad0 con 0x7f46640b1fc0 2023-10-10T18:59:08.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4662ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f46640b2390 0x7f4664068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4662ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f46640b2390 0x7f4664068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:41582/0 (socket says 172.21.15.175:41582) 2023-10-10T18:59:08.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4662ffd700 1 -- 172.21.15.175:0/1955710028 learned_addr learned my addr 172.21.15.175:0/1955710028 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:08.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f46637fe700 1 --2- 172.21.15.175:0/1955710028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46640b1fc0 0x7f4664155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4662ffd700 1 -- 172.21.15.175:0/1955710028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46640b1fc0 msgr2=0x7f4664155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:08.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4662ffd700 1 --2- 172.21.15.175:0/1955710028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46640b1fc0 0x7f4664155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4662ffd700 1 -- 172.21.15.175:0/1955710028 >> v1:172.21.15.191:6789/0 conn(0x7f4664068bd0 legacy=0x7f46640aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4662ffd700 1 -- 172.21.15.175:0/1955710028 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46640af4c0 con 0x7f46640b2390 2023-10-10T18:59:08.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f4662ffd700 1 --2- 172.21.15.175:0/1955710028 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b2390 0x7f4664068690 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f46540099b0 tx=0x7f4654009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f489aa0021ccf0f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:08.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f46627fc700 1 -- 172.21.15.175:0/1955710028 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465400b8f0 con 0x7f46640b2390 2023-10-10T18:59:08.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f46627fc700 1 -- 172.21.15.175:0/1955710028 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465400ba50 con 0x7f46640b2390 2023-10-10T18:59:08.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.348+0000 7f46627fc700 1 -- 172.21.15.175:0/1955710028 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465400bdf0 con 0x7f46640b2390 2023-10-10T18:59:08.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.349+0000 7f4669ce6700 1 -- 172.21.15.175:0/1955710028 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b2390 msgr2=0x7f4664068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.349+0000 7f4669ce6700 1 --2- 172.21.15.175:0/1955710028 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b2390 0x7f4664068690 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f46540099b0 tx=0x7f4654009250 comp rx=0 tx=0).stop 2023-10-10T18:59:08.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.349+0000 7f4669ce6700 1 -- 172.21.15.175:0/1955710028 shutdown_connections 2023-10-10T18:59:08.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.349+0000 7f4669ce6700 1 --2- 172.21.15.175:0/1955710028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46640b1fc0 0x7f4664155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.349+0000 7f4669ce6700 1 --2- 172.21.15.175:0/1955710028 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b2390 0x7f4664068690 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.349+0000 7f4669ce6700 1 -- 172.21.15.175:0/1955710028 >> 172.21.15.175:0/1955710028 conn(0x7f4664063230 msgr2=0x7f4664063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:08.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.349+0000 7f4669ce6700 1 -- 172.21.15.175:0/1955710028 shutdown_connections 2023-10-10T18:59:08.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.349+0000 7f4669ce6700 1 -- 172.21.15.175:0/1955710028 wait complete. 2023-10-10T18:59:08.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.350+0000 7f4669ce6700 1 Processor -- start 2023-10-10T18:59:08.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.350+0000 7f4669ce6700 1 -- start start 2023-10-10T18:59:08.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4669ce6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4664068bd0 0x7f46641598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4669ce6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b1fc0 0x7f4664159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4669ce6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46640b2390 0x7f4664165f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4669ce6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46640c4950 con 0x7f46640b1fc0 2023-10-10T18:59:08.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4669ce6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46641550a0 con 0x7f46640b2390 2023-10-10T18:59:08.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4669ce6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466416a110 con 0x7f4664068bd0 2023-10-10T18:59:08.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b1fc0 0x7f4664159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b1fc0 0x7f4664159e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:41596/0 (socket says 172.21.15.175:41596) 2023-10-10T18:59:08.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 -- 172.21.15.175:0/285615174 learned_addr learned my addr 172.21.15.175:0/285615174 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:08.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f46637fe700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4664068bd0 0x7f46641598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4663fff700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46640b2390 0x7f4664165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 -- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4664068bd0 msgr2=0x7f46641598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4664068bd0 0x7f46641598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 -- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46640b2390 msgr2=0x7f4664165f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46640b2390 0x7f4664165f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 -- 172.21.15.175:0/285615174 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46540094f0 con 0x7f46640b1fc0 2023-10-10T18:59:08.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4663fff700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46640b2390 0x7f4664165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:08.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f4662ffd700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b1fc0 0x7f4664159e00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4654009290 tx=0x7f4654017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:08.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f464b7fe700 1 -- 172.21.15.175:0/285615174 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465401a450 con 0x7f46640b1fc0 2023-10-10T18:59:08.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f464b7fe700 1 -- 172.21.15.175:0/285615174 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465401a5d0 con 0x7f46640b1fc0 2023-10-10T18:59:08.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.351+0000 7f464b7fe700 1 -- 172.21.15.175:0/285615174 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465401a990 con 0x7f46640b1fc0 2023-10-10T18:59:08.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.352+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46641552d0 con 0x7f46640b1fc0 2023-10-10T18:59:08.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.352+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46641697e0 con 0x7f46640b1fc0 2023-10-10T18:59:08.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.352+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466416a2c0 con 0x7f46640b1fc0 2023-10-10T18:59:08.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.353+0000 7f464b7fe700 1 -- 172.21.15.175:0/285615174 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4654025440 con 0x7f46640b1fc0 2023-10-10T18:59:08.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.353+0000 7f464b7fe700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4650066f60 0x7f4650069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.353+0000 7f464b7fe700 1 -- 172.21.15.175:0/285615174 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f46540862b0 con 0x7f46640b1fc0 2023-10-10T18:59:08.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.353+0000 7f46637fe700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4650066f60 0x7f4650069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.354+0000 7f46637fe700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4650066f60 0x7f4650069410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4658006fd0 tx=0x7f4658008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:08.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.357+0000 7f464b7fe700 1 -- 172.21.15.175:0/285615174 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f466416a2c0 con 0x7f46640b1fc0 2023-10-10T18:59:08.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.587+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f466416a280 con 0x7f46640b1fc0 2023-10-10T18:59:08.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.588+0000 7f464b7fe700 1 -- 172.21.15.175:0/285615174 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5432 (secure 0 0 0) 0x7f4654053350 con 0x7f46640b1fc0 2023-10-10T18:59:08.588 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:08.588 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:08.550523+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:08.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4650066f60 msgr2=0x7f4650069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4650066f60 0x7f4650069410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4658006fd0 tx=0x7f4658008040 comp rx=0 tx=0).stop 2023-10-10T18:59:08.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b1fc0 msgr2=0x7f4664159e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b1fc0 0x7f4664159e00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4654009290 tx=0x7f4654017000 comp rx=0 tx=0).stop 2023-10-10T18:59:08.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 shutdown_connections 2023-10-10T18:59:08.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4650066f60 0x7f4650069410 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46640b2390 0x7f4664165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4664068bd0 0x7f46641598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 --2- 172.21.15.175:0/285615174 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46640b1fc0 0x7f4664159e00 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 >> 172.21.15.175:0/285615174 conn(0x7f4664063230 msgr2=0x7f466414fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:08.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 shutdown_connections 2023-10-10T18:59:08.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.590+0000 7f4669ce6700 1 -- 172.21.15.175:0/285615174 wait complete. 2023-10-10T18:59:08.593 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 100 2023-10-10T18:59:08.602 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-10T18:59:08.602 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T18:59:08.603 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:08.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.717+0000 7f3110b31700 1 Processor -- start 2023-10-10T18:59:08.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.717+0000 7f3110b31700 1 -- start start 2023-10-10T18:59:08.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f3110b31700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f310c14e050 0x7f310c1547c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f3110b31700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c0c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f3110b31700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0c5280 con 0x7f310c067bd0 2023-10-10T18:59:08.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f3110b31700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310c0c4b60 con 0x7f310c14e050 2023-10-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f3110b31700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c154d00 con 0x7f310c0c2a70 2023-10-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f310a59c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f310c14e050 0x7f310c1547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f310ad9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c0c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f310a59c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f310c14e050 0x7f310c1547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:42430/0 (socket says 172.21.15.175:42430) 2023-10-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.718+0000 7f310a59c700 1 -- 172.21.15.175:0/4074463667 learned_addr learned my addr 172.21.15.175:0/4074463667 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f310ad9d700 1 -- 172.21.15.175:0/4074463667 >> v2:172.21.15.191:3300/0 conn(0x7f310c14e050 msgr2=0x7f310c1547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f310ad9d700 1 --2- 172.21.15.175:0/4074463667 >> v2:172.21.15.191:3300/0 conn(0x7f310c14e050 0x7f310c1547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f310ad9d700 1 -- 172.21.15.175:0/4074463667 >> v1:172.21.15.191:6789/0 conn(0x7f310c067bd0 legacy=0x7f310c068050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f310ad9d700 1 -- 172.21.15.175:0/4074463667 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0c5560 con 0x7f310c0c2a70 2023-10-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f310a59c700 1 --2- 172.21.15.175:0/4074463667 >> v2:172.21.15.191:3300/0 conn(0x7f310c14e050 0x7f310c1547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:59:08.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3109d9b700 1 -- 172.21.15.175:0/4074463667 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713460889 0 0) 0x7f310c0c5280 con 0x7f310c067bd0 2023-10-10T18:59:08.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f310ad9d700 1 --2- 172.21.15.175:0/4074463667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c0c2e90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3104009e60 tx=0x7f3104009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bde0a81858f7f08a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:08.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3109d9b700 1 -- 172.21.15.175:0/4074463667 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400e070 con 0x7f310c0c2a70 2023-10-10T18:59:08.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3109d9b700 1 -- 172.21.15.175:0/4074463667 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310400bb60 con 0x7f310c0c2a70 2023-10-10T18:59:08.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3109d9b700 1 -- 172.21.15.175:0/4074463667 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104012430 con 0x7f310c0c2a70 2023-10-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3110b31700 1 -- 172.21.15.175:0/4074463667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 msgr2=0x7f310c0c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3110b31700 1 --2- 172.21.15.175:0/4074463667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c0c2e90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3104009e60 tx=0x7f3104009780 comp rx=0 tx=0).stop 2023-10-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3110b31700 1 -- 172.21.15.175:0/4074463667 shutdown_connections 2023-10-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3110b31700 1 --2- 172.21.15.175:0/4074463667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c0c2e90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.719+0000 7f3110b31700 1 --2- 172.21.15.175:0/4074463667 >> v2:172.21.15.191:3300/0 conn(0x7f310c14e050 0x7f310c1547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.720+0000 7f3110b31700 1 -- 172.21.15.175:0/4074463667 >> 172.21.15.175:0/4074463667 conn(0x7f310c06c110 msgr2=0x7f310c06c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:08.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.720+0000 7f3110b31700 1 -- 172.21.15.175:0/4074463667 shutdown_connections 2023-10-10T18:59:08.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.720+0000 7f3110b31700 1 -- 172.21.15.175:0/4074463667 wait complete. 2023-10-10T18:59:08.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.721+0000 7f3110b31700 1 Processor -- start 2023-10-10T18:59:08.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.721+0000 7f3110b31700 1 -- start start 2023-10-10T18:59:08.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f3110b31700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f310c067bd0 0x7f310c158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f3110b31700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c13b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f3110b31700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f310c14e050 0x7f310c166a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f3110b31700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c149370 con 0x7f310c14e050 2023-10-10T18:59:08.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f3110b31700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c166f50 con 0x7f310c067bd0 2023-10-10T18:59:08.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f3110b31700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c158a80 con 0x7f310c0c2a70 2023-10-10T18:59:08.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f310ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f310c14e050 0x7f310c166a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f310ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f310c14e050 0x7f310c166a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:41598/0 (socket says 172.21.15.175:41598) 2023-10-10T18:59:08.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f310ad9d700 1 -- 172.21.15.175:0/1744482652 learned_addr learned my addr 172.21.15.175:0/1744482652 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:08.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f310a59c700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f310c067bd0 0x7f310c158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.722+0000 7f3103fff700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c13b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f310ad9d700 1 -- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 msgr2=0x7f310c13b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f310ad9d700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c13b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f310ad9d700 1 -- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f310c067bd0 msgr2=0x7f310c158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f310ad9d700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f310c067bd0 0x7f310c158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f310ad9d700 1 -- 172.21.15.175:0/1744482652 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0094f0 con 0x7f310c14e050 2023-10-10T18:59:08.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f310ad9d700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f310c14e050 0x7f310c166a10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f310400b6c0 tx=0x7f3104019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:08.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f31027fc700 1 -- 172.21.15.175:0/1744482652 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104012e90 con 0x7f310c14e050 2023-10-10T18:59:08.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f31027fc700 1 -- 172.21.15.175:0/1744482652 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104027470 con 0x7f310c14e050 2023-10-10T18:59:08.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3104009a20 con 0x7f310c14e050 2023-10-10T18:59:08.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f31027fc700 1 -- 172.21.15.175:0/1744482652 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104027830 con 0x7f310c14e050 2023-10-10T18:59:08.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.723+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c1581e0 con 0x7f310c14e050 2023-10-10T18:59:08.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.724+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c158c30 con 0x7f310c14e050 2023-10-10T18:59:08.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.724+0000 7f31027fc700 1 -- 172.21.15.175:0/1744482652 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3104024070 con 0x7f310c14e050 2023-10-10T18:59:08.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.724+0000 7f31027fc700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30ec066f30 0x7f30ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:08.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.725+0000 7f31027fc700 1 -- 172.21.15.175:0/1744482652 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f3104086f80 con 0x7f310c14e050 2023-10-10T18:59:08.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.725+0000 7f310a59c700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30ec066f30 0x7f30ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:08.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.725+0000 7f310a59c700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30ec066f30 0x7f30ec0693e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f30fc006d20 tx=0x7f30fc009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:08.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.729+0000 7f31027fc700 1 -- 172.21.15.175:0/1744482652 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31040b85a0 con 0x7f310c14e050 2023-10-10T18:59:08.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.953+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f310c067570 con 0x7f310c14e050 2023-10-10T18:59:08.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.954+0000 7f31027fc700 1 -- 172.21.15.175:0/1744482652 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5432 (secure 0 0 0) 0x7f310402c090 con 0x7f310c14e050 2023-10-10T18:59:08.954 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:08.954 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:08.550523+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:08.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30ec066f30 msgr2=0x7f30ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30ec066f30 0x7f30ec0693e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f30fc006d20 tx=0x7f30fc009250 comp rx=0 tx=0).stop 2023-10-10T18:59:08.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f310c14e050 msgr2=0x7f310c166a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:08.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f310c14e050 0x7f310c166a10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f310400b6c0 tx=0x7f3104019000 comp rx=0 tx=0).stop 2023-10-10T18:59:08.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 shutdown_connections 2023-10-10T18:59:08.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30ec066f30 0x7f30ec0693e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f310c14e050 0x7f310c166a10 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f310c067bd0 0x7f310c158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 --2- 172.21.15.175:0/1744482652 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f310c0c2a70 0x7f310c13b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:08.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 >> 172.21.15.175:0/1744482652 conn(0x7f310c06c110 msgr2=0x7f310c06f070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:08.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 shutdown_connections 2023-10-10T18:59:08.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:08.956+0000 7f3110b31700 1 -- 172.21.15.175:0/1744482652 wait complete. 2023-10-10T18:59:08.959 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 100 2023-10-10T18:59:08.968 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 100, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 31 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:59:08.550523+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 5332, "mds_2": 5367, "mds_3": 5196, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T18:59:10.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:11.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.096+0000 7fb7ad02e700 1 Processor -- start 2023-10-10T18:59:11.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.096+0000 7fb7ad02e700 1 -- start start 2023-10-10T18:59:11.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7ad02e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b2390 0x7fb7a8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:11.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7ad02e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb7a80aff60 0x7fb7a80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:11.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7ad02e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a81574b0 con 0x7fb7a80b1fc0 2023-10-10T18:59:11.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7ad02e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a8150770 con 0x7fb7a80aff60 2023-10-10T18:59:11.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7ad02e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80afad0 con 0x7fb7a80b2390 2023-10-10T18:59:11.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7a6d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb7a80b1fc0 0x7fb7a8068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46872/0 (socket says 172.21.15.175:46872) 2023-10-10T18:59:11.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7a6d9d700 1 -- 172.21.15.175:0/1751260235 learned_addr learned my addr 172.21.15.175:0/1751260235 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:11.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb79e59c700 1 --2- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b2390 0x7fb7a8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:11.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7a759e700 1 --2- 172.21.15.175:0/1751260235 >> v2:172.21.15.191:3300/0 conn(0x7fb7a80aff60 0x7fb7a80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:11.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7a659c700 1 -- 172.21.15.175:0/1751260235 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680382342 0 0) 0x7fb7a81574b0 con 0x7fb7a80b1fc0 2023-10-10T18:59:11.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.097+0000 7fb7a659c700 1 -- 172.21.15.175:0/1751260235 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb794003570 con 0x7fb7a80b1fc0 2023-10-10T18:59:11.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7a759e700 1 -- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b2390 msgr2=0x7fb7a8155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:11.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7a759e700 1 --2- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b2390 0x7fb7a8155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7a759e700 1 -- 172.21.15.175:0/1751260235 >> v1:172.21.15.175:6789/0 conn(0x7fb7a80b1fc0 legacy=0x7fb7a8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:11.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7a759e700 1 -- 172.21.15.175:0/1751260235 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a80af4c0 con 0x7fb7a80aff60 2023-10-10T18:59:11.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb79e59c700 1 --2- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b2390 0x7fb7a8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:59:11.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7a759e700 1 --2- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80aff60 0x7fb7a80b03e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0009e90 tx=0x7fb7a0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8d18f8be90b915c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:11.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7a659c700 1 -- 172.21.15.175:0/1751260235 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a000e070 con 0x7fb7a80aff60 2023-10-10T18:59:11.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7a659c700 1 -- 172.21.15.175:0/1751260235 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a000bbd0 con 0x7fb7a80aff60 2023-10-10T18:59:11.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7a659c700 1 -- 172.21.15.175:0/1751260235 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0012430 con 0x7fb7a80aff60 2023-10-10T18:59:11.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80aff60 msgr2=0x7fb7a80b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:11.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80aff60 0x7fb7a80b03e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0009e90 tx=0x7fb7a0009780 comp rx=0 tx=0).stop 2023-10-10T18:59:11.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1751260235 shutdown_connections 2023-10-10T18:59:11.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80aff60 0x7fb7a80b03e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1751260235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b2390 0x7fb7a8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1751260235 >> 172.21.15.175:0/1751260235 conn(0x7fb7a8063230 msgr2=0x7fb7a8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:11.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1751260235 shutdown_connections 2023-10-10T18:59:11.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.098+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1751260235 wait complete. 2023-10-10T18:59:11.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7ad02e700 1 Processor -- start 2023-10-10T18:59:11.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7ad02e700 1 -- start start 2023-10-10T18:59:11.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7ad02e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a80aff60 0x7fb7a814cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:11.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7ad02e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b1fc0 0x7fb7a80c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:11.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7ad02e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80b2390 0x7fb7a80c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:11.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7ad02e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80c3870 con 0x7fb7a80aff60 2023-10-10T18:59:11.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7ad02e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80c3a80 con 0x7fb7a80b2390 2023-10-10T18:59:11.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7ad02e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80c3d60 con 0x7fb7a80b1fc0 2023-10-10T18:59:11.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.100+0000 7fb7a6d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a80aff60 0x7fb7a814cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:11.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a6d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a80aff60 0x7fb7a814cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:41608/0 (socket says 172.21.15.175:41608) 2023-10-10T18:59:11.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a6d9d700 1 -- 172.21.15.175:0/1151158826 learned_addr learned my addr 172.21.15.175:0/1151158826 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:11.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a759e700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80b2390 0x7fb7a80c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:11.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb79e59c700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b1fc0 0x7fb7a80c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:11.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a6d9d700 1 -- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b1fc0 msgr2=0x7fb7a80c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:11.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a6d9d700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b1fc0 0x7fb7a80c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a6d9d700 1 -- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80b2390 msgr2=0x7fb7a80c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:11.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a6d9d700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80b2390 0x7fb7a80c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a6d9d700 1 -- 172.21.15.175:0/1151158826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb790009420 con 0x7fb7a80aff60 2023-10-10T18:59:11.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a759e700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80b2390 0x7fb7a80c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:11.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb79e59c700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b1fc0 0x7fb7a80c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:11.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7a6d9d700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a80aff60 0x7fb7a814cb30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c7c0 tx=0x7fb798007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:11.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb79f7fe700 1 -- 172.21.15.175:0/1151158826 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798005b30 con 0x7fb7a80aff60 2023-10-10T18:59:11.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb79f7fe700 1 -- 172.21.15.175:0/1151158826 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb798005c90 con 0x7fb7a80aff60 2023-10-10T18:59:11.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb79f7fe700 1 -- 172.21.15.175:0/1151158826 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798007340 con 0x7fb7a80aff60 2023-10-10T18:59:11.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a0009a20 con 0x7fb7a80aff60 2023-10-10T18:59:11.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.101+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a80c4320 con 0x7fb7a80aff60 2023-10-10T18:59:11.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.102+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a8067570 con 0x7fb7a80aff60 2023-10-10T18:59:11.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.103+0000 7fb79f7fe700 1 -- 172.21.15.175:0/1151158826 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb79801b400 con 0x7fb7a80aff60 2023-10-10T18:59:11.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.103+0000 7fb79f7fe700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb788066f60 0x7fb788069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:11.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.103+0000 7fb79f7fe700 1 -- 172.21.15.175:0/1151158826 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fb798084320 con 0x7fb7a80aff60 2023-10-10T18:59:11.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.103+0000 7fb79e59c700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb788066f60 0x7fb788069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:11.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.104+0000 7fb79e59c700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb788066f60 0x7fb788069410 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb7900093f0 tx=0x7fb790008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:11.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.107+0000 7fb79f7fe700 1 -- 172.21.15.175:0/1151158826 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb798051340 con 0x7fb7a80aff60 2023-10-10T18:59:11.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.331+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7a8068da0 con 0x7fb7a80aff60 2023-10-10T18:59:11.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.331+0000 7fb79f7fe700 1 -- 172.21.15.175:0/1151158826 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5432 (secure 0 0 0) 0x7fb798054960 con 0x7fb7a80aff60 2023-10-10T18:59:11.331 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:11.332 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:08.550523+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:11.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb788066f60 msgr2=0x7fb788069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:11.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb788066f60 0x7fb788069410 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb7900093f0 tx=0x7fb790008040 comp rx=0 tx=0).stop 2023-10-10T18:59:11.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a80aff60 msgr2=0x7fb7a814cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:11.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a80aff60 0x7fb7a814cb30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c7c0 tx=0x7fb798007e00 comp rx=0 tx=0).stop 2023-10-10T18:59:11.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 shutdown_connections 2023-10-10T18:59:11.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb788066f60 0x7fb788069410 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a80b2390 0x7fb7a80c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a80aff60 0x7fb7a814cb30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 --2- 172.21.15.175:0/1151158826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a80b1fc0 0x7fb7a80c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:11.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 >> 172.21.15.175:0/1151158826 conn(0x7fb7a8063230 msgr2=0x7fb7a80c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:11.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 shutdown_connections 2023-10-10T18:59:11.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:11.333+0000 7fb7ad02e700 1 -- 172.21.15.175:0/1151158826 wait complete. 2023-10-10T18:59:11.336 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 100 2023-10-10T18:59:11.345 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:13.346 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:13.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.472+0000 7f7eb14b6700 1 Processor -- start 2023-10-10T18:59:13.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.472+0000 7f7eb14b6700 1 -- start start 2023-10-10T18:59:13.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.473+0000 7f7eb14b6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7eac14fd90 0x7f7eac159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:13.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.473+0000 7f7eb14b6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7eac0cd040 0x7f7eac0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:13.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.473+0000 7f7eb14b6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac067fa0 con 0x7f7eac08a100 2023-10-10T18:59:13.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.473+0000 7f7eb14b6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac0c5600 con 0x7f7eac0cd040 2023-10-10T18:59:13.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.473+0000 7f7eb14b6700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac156b70 con 0x7f7eac14fd90 2023-10-10T18:59:13.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.473+0000 7f7eb0cb5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7eac0cd040 0x7f7eac0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:13.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.473+0000 7f7eb0cb5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7eac0cd040 0x7f7eac0684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:41624/0 (socket says 172.21.15.175:41624) 2023-10-10T18:59:13.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.473+0000 7f7eb0cb5700 1 -- 172.21.15.175:0/3161316550 learned_addr learned my addr 172.21.15.175:0/3161316550 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:13.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eabfff700 1 --2- 172.21.15.175:0/3161316550 >> v2:172.21.15.191:3300/0 conn(0x7f7eac14fd90 0x7f7eac159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:13.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eb0cb5700 1 -- 172.21.15.175:0/3161316550 >> v2:172.21.15.191:3300/0 conn(0x7f7eac14fd90 msgr2=0x7f7eac159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:13.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eb0cb5700 1 --2- 172.21.15.175:0/3161316550 >> v2:172.21.15.191:3300/0 conn(0x7f7eac14fd90 0x7f7eac159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eb0cb5700 1 -- 172.21.15.175:0/3161316550 >> v1:172.21.15.191:6789/0 conn(0x7f7eac08a100 legacy=0x7f7eac0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:13.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eb0cb5700 1 -- 172.21.15.175:0/3161316550 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac150160 con 0x7f7eac0cd040 2023-10-10T18:59:13.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eb0cb5700 1 --2- 172.21.15.175:0/3161316550 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac0cd040 0x7f7eac0684f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7eac065080 tx=0x7f7ea4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bf172986194548 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:13.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eab7fe700 1 -- 172.21.15.175:0/3161316550 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea400e070 con 0x7f7eac0cd040 2023-10-10T18:59:13.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eab7fe700 1 -- 172.21.15.175:0/3161316550 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea400bae0 con 0x7f7eac0cd040 2023-10-10T18:59:13.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eab7fe700 1 -- 172.21.15.175:0/3161316550 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea40092e0 con 0x7f7eac0cd040 2023-10-10T18:59:13.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eb14b6700 1 -- 172.21.15.175:0/3161316550 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac0cd040 msgr2=0x7f7eac0684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:13.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.474+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/3161316550 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac0cd040 0x7f7eac0684f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7eac065080 tx=0x7f7ea4009840 comp rx=0 tx=0).stop 2023-10-10T18:59:13.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.475+0000 7f7eb14b6700 1 -- 172.21.15.175:0/3161316550 shutdown_connections 2023-10-10T18:59:13.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.475+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/3161316550 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac0cd040 0x7f7eac0684f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.475+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/3161316550 >> v2:172.21.15.191:3300/0 conn(0x7f7eac14fd90 0x7f7eac159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.475+0000 7f7eb14b6700 1 -- 172.21.15.175:0/3161316550 >> 172.21.15.175:0/3161316550 conn(0x7f7eac063d00 msgr2=0x7f7eac0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:13.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.475+0000 7f7eb14b6700 1 -- 172.21.15.175:0/3161316550 shutdown_connections 2023-10-10T18:59:13.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.475+0000 7f7eb14b6700 1 -- 172.21.15.175:0/3161316550 wait complete. 2023-10-10T18:59:13.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eb14b6700 1 Processor -- start 2023-10-10T18:59:13.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eb14b6700 1 -- start start 2023-10-10T18:59:13.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eb14b6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac08a100 0x7f7eac1671d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:13.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eb14b6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eac0cd040 0x7f7eac152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:13.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eb14b6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7eac14fd90 0x7f7eac169490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:13.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eb14b6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac1526f0 con 0x7f7eac08a100 2023-10-10T18:59:13.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eb14b6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac155c70 con 0x7f7eac0cd040 2023-10-10T18:59:13.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eb14b6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac169e10 con 0x7f7eac14fd90 2023-10-10T18:59:13.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eabfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac08a100 0x7f7eac1671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:13.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.476+0000 7f7eabfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac08a100 0x7f7eac1671d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:41634/0 (socket says 172.21.15.175:41634) 2023-10-10T18:59:13.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eabfff700 1 -- 172.21.15.175:0/2220505862 learned_addr learned my addr 172.21.15.175:0/2220505862 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:13.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7ea3fff700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eac0cd040 0x7f7eac152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:13.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eb0cb5700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7eac14fd90 0x7f7eac169490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:13.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eabfff700 1 -- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7eac14fd90 msgr2=0x7f7eac169490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:13.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eabfff700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7eac14fd90 0x7f7eac169490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eabfff700 1 -- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eac0cd040 msgr2=0x7f7eac152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:13.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eabfff700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eac0cd040 0x7f7eac152b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eabfff700 1 -- 172.21.15.175:0/2220505862 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea4009ae0 con 0x7f7eac08a100 2023-10-10T18:59:13.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eb0cb5700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7eac14fd90 0x7f7eac169490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:13.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eabfff700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac08a100 0x7f7eac1671d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00d890 tx=0x7f7e9c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:13.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7ea8ff9700 1 -- 172.21.15.175:0/2220505862 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c0125f0 con 0x7f7eac08a100 2023-10-10T18:59:13.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7ea8ff9700 1 -- 172.21.15.175:0/2220505862 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9c012750 con 0x7f7eac08a100 2023-10-10T18:59:13.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7ea8ff9700 1 -- 172.21.15.175:0/2220505862 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c012b10 con 0x7f7eac08a100 2023-10-10T18:59:13.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac16a0b0 con 0x7f7eac08a100 2023-10-10T18:59:13.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.477+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac152380 con 0x7f7eac08a100 2023-10-10T18:59:13.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.478+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eac1529d0 con 0x7f7eac08a100 2023-10-10T18:59:13.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.479+0000 7f7ea8ff9700 1 -- 172.21.15.175:0/2220505862 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7e9c01e460 con 0x7f7eac08a100 2023-10-10T18:59:13.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.479+0000 7f7ea8ff9700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e8c066f30 0x7f7e8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:13.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.479+0000 7f7ea8ff9700 1 -- 172.21.15.175:0/2220505862 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f7e9c084da0 con 0x7f7eac08a100 2023-10-10T18:59:13.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.479+0000 7f7ea3fff700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e8c066f30 0x7f7e8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:13.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.480+0000 7f7ea3fff700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e8c066f30 0x7f7e8c0693e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7eac155eb0 tx=0x7f7e94007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:13.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.483+0000 7f7ea8ff9700 1 -- 172.21.15.175:0/2220505862 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e9c0557d0 con 0x7f7eac08a100 2023-10-10T18:59:13.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.720+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7eac067570 con 0x7f7eac08a100 2023-10-10T18:59:13.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.720+0000 7f7ea8ff9700 1 -- 172.21.15.175:0/2220505862 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5432 (secure 0 0 0) 0x7f7e9c055360 con 0x7f7eac08a100 2023-10-10T18:59:13.721 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:13.721 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:08.550523+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:13.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e8c066f30 msgr2=0x7f7e8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:13.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e8c066f30 0x7f7e8c0693e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7eac155eb0 tx=0x7f7e94007ec0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac08a100 msgr2=0x7f7eac1671d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:13.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac08a100 0x7f7eac1671d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00d890 tx=0x7f7e9c008040 comp rx=0 tx=0).stop 2023-10-10T18:59:13.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 shutdown_connections 2023-10-10T18:59:13.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e8c066f30 0x7f7e8c0693e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7eac14fd90 0x7f7eac169490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7eac08a100 0x7f7eac1671d0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 --2- 172.21.15.175:0/2220505862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eac0cd040 0x7f7eac152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:13.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 >> 172.21.15.175:0/2220505862 conn(0x7f7eac063d00 msgr2=0x7f7eac0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:13.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 shutdown_connections 2023-10-10T18:59:13.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:13.723+0000 7f7eb14b6700 1 -- 172.21.15.175:0/2220505862 wait complete. 2023-10-10T18:59:13.726 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 100 2023-10-10T18:59:13.735 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:15.736 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:15.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.862+0000 7f0601292700 1 Processor -- start 2023-10-10T18:59:15.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.862+0000 7f0601292700 1 -- start start 2023-10-10T18:59:15.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f0601292700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05fc0b1fc0 0x7f05fc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:15.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f0601292700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc1574b0 con 0x7f05fc0b2390 2023-10-10T18:59:15.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f0601292700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0afad0 con 0x7f05fc0aff60 2023-10-10T18:59:15.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f0601292700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc150770 con 0x7f05fc0b1fc0 2023-10-10T18:59:15.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05faffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05fc0b1fc0 0x7f05fc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:15.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05f3fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f05fc0b2390 0x7f05fc068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49500/0 (socket says 172.21.15.175:49500) 2023-10-10T18:59:15.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05f3fff700 1 -- 172.21.15.175:0/2571869416 learned_addr learned my addr 172.21.15.175:0/2571869416 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:15.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05faffd700 1 -- 172.21.15.175:0/2571869416 >> v1:172.21.15.191:6789/0 conn(0x7f05fc0aff60 legacy=0x7f05fc159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:15.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05faffd700 1 -- 172.21.15.175:0/2571869416 >> v1:172.21.15.175:6789/0 conn(0x7f05fc0b2390 legacy=0x7f05fc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:15.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05faffd700 1 -- 172.21.15.175:0/2571869416 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc0af4c0 con 0x7f05fc0b1fc0 2023-10-10T18:59:15.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05fa7fc700 1 -- 172.21.15.175:0/2571869416 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516680305 0 0) 0x7f05fc1574b0 con 0x7f05fc0b2390 2023-10-10T18:59:15.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05faffd700 1 --2- 172.21.15.175:0/2571869416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b1fc0 0x7f05fc155fb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0098e0 tx=0x7f05ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f99077b7112cb73 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:15.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05fa7fc700 1 -- 172.21.15.175:0/2571869416 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec009e80 con 0x7f05fc0b1fc0 2023-10-10T18:59:15.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05fa7fc700 1 -- 172.21.15.175:0/2571869416 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec00b8b0 con 0x7f05fc0b1fc0 2023-10-10T18:59:15.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.863+0000 7f05fa7fc700 1 -- 172.21.15.175:0/2571869416 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec00bc30 con 0x7f05fc0b1fc0 2023-10-10T18:59:15.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.864+0000 7f0601292700 1 -- 172.21.15.175:0/2571869416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b1fc0 msgr2=0x7f05fc155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:15.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.864+0000 7f0601292700 1 --2- 172.21.15.175:0/2571869416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b1fc0 0x7f05fc155fb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0098e0 tx=0x7f05ec0091a0 comp rx=0 tx=0).stop 2023-10-10T18:59:15.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.864+0000 7f0601292700 1 -- 172.21.15.175:0/2571869416 shutdown_connections 2023-10-10T18:59:15.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.864+0000 7f0601292700 1 --2- 172.21.15.175:0/2571869416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b1fc0 0x7f05fc155fb0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:15.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.864+0000 7f0601292700 1 -- 172.21.15.175:0/2571869416 >> 172.21.15.175:0/2571869416 conn(0x7f05fc063230 msgr2=0x7f05fc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:15.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.864+0000 7f0601292700 1 -- 172.21.15.175:0/2571869416 shutdown_connections 2023-10-10T18:59:15.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.864+0000 7f0601292700 1 -- 172.21.15.175:0/2571869416 wait complete. 2023-10-10T18:59:15.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.865+0000 7f0601292700 1 Processor -- start 2023-10-10T18:59:15.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.865+0000 7f0601292700 1 -- start start 2023-10-10T18:59:15.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f0601292700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05fc0aff60 0x7f05fc151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:15.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f0601292700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05fc0b1fc0 0x7f05fc0c2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:15.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f0601292700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b2390 0x7f05fc14fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:15.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f0601292700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc0c07e0 con 0x7f05fc0b2390 2023-10-10T18:59:15.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f0601292700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc0c1ee0 con 0x7f05fc0aff60 2023-10-10T18:59:15.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f0601292700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc166cd0 con 0x7f05fc0b1fc0 2023-10-10T18:59:15.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b2390 0x7f05fc14fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:15.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b2390 0x7f05fc14fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54500/0 (socket says 172.21.15.175:54500) 2023-10-10T18:59:15.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 -- 172.21.15.175:0/3024888344 learned_addr learned my addr 172.21.15.175:0/3024888344 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:15.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05faffd700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05fc0aff60 0x7f05fc151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:15.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05f3fff700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05fc0b1fc0 0x7f05fc0c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:15.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 -- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05fc0b1fc0 msgr2=0x7f05fc0c2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:15.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05fc0b1fc0 0x7f05fc0c2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:15.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 -- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05fc0aff60 msgr2=0x7f05fc151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:15.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05fc0aff60 0x7f05fc151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:15.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 -- 172.21.15.175:0/3024888344 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec009420 con 0x7f05fc0b2390 2023-10-10T18:59:15.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05fb7fe700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b2390 0x7f05fc14fb20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c930 tx=0x7f05f40026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:15.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05f2ffd700 1 -- 172.21.15.175:0/3024888344 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4012630 con 0x7f05fc0b2390 2023-10-10T18:59:15.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05f2ffd700 1 -- 172.21.15.175:0/3024888344 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f40127d0 con 0x7f05fc0b2390 2023-10-10T18:59:15.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f05f2ffd700 1 -- 172.21.15.175:0/3024888344 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4012b90 con 0x7f05fc0b2390 2023-10-10T18:59:15.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.866+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc1530e0 con 0x7f05fc0b2390 2023-10-10T18:59:15.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.867+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc169480 con 0x7f05fc0b2390 2023-10-10T18:59:15.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.867+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc0c2090 con 0x7f05fc0b2390 2023-10-10T18:59:15.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.868+0000 7f05f2ffd700 1 -- 172.21.15.175:0/3024888344 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f05f401b400 con 0x7f05fc0b2390 2023-10-10T18:59:15.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.868+0000 7f05f2ffd700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05dc066f30 0x7f05dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:15.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.868+0000 7f05f2ffd700 1 -- 172.21.15.175:0/3024888344 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f05f4084b30 con 0x7f05fc0b2390 2023-10-10T18:59:15.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.869+0000 7f05faffd700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05dc066f30 0x7f05dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:15.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.869+0000 7f05faffd700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05dc066f30 0x7f05dc0693e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0030c0 tx=0x7f05ec016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:15.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:15.873+0000 7f05f2ffd700 1 -- 172.21.15.175:0/3024888344 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05fc0c2090 con 0x7f05fc0b2390 2023-10-10T18:59:16.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.096+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05fc0c2050 con 0x7f05fc0b2390 2023-10-10T18:59:16.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.097+0000 7f05f2ffd700 1 -- 172.21.15.175:0/3024888344 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5434 (secure 0 0 0) 0x7f05f4055610 con 0x7f05fc0b2390 2023-10-10T18:59:16.097 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:16.097 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:15.595170+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:resolve","state_seq":27,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:16.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05dc066f30 msgr2=0x7f05dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:16.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05dc066f30 0x7f05dc0693e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0030c0 tx=0x7f05ec016000 comp rx=0 tx=0).stop 2023-10-10T18:59:16.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b2390 msgr2=0x7f05fc14fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:16.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b2390 0x7f05fc14fb20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c930 tx=0x7f05f40026e0 comp rx=0 tx=0).stop 2023-10-10T18:59:16.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 shutdown_connections 2023-10-10T18:59:16.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f05dc066f30 0x7f05dc0693e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:16.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05fc0b2390 0x7f05fc14fb20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:16.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05fc0aff60 0x7f05fc151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:16.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 --2- 172.21.15.175:0/3024888344 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05fc0b1fc0 0x7f05fc0c2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:16.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 >> 172.21.15.175:0/3024888344 conn(0x7f05fc063230 msgr2=0x7f05fc1527d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:16.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 shutdown_connections 2023-10-10T18:59:16.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:16.099+0000 7f0601292700 1 -- 172.21.15.175:0/3024888344 wait complete. 2023-10-10T18:59:16.102 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 101 2023-10-10T18:59:16.111 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:18.112 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:18.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.239+0000 7f2553012700 1 Processor -- start 2023-10-10T18:59:18.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.239+0000 7f2553012700 1 -- start start 2023-10-10T18:59:18.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f2553012700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f254c0c4580 0x7f254c1504f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:18.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f2553012700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c157030 con 0x7f254c0c7190 2023-10-10T18:59:18.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f2553012700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c156420 con 0x7f254c0c6330 2023-10-10T18:59:18.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f2553012700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c0c5000 con 0x7f254c0c4580 2023-10-10T18:59:18.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f254bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f254c0c7190 0x7f254c14fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49516/0 (socket says 172.21.15.175:49516) 2023-10-10T18:59:18.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f254bfff700 1 -- 172.21.15.175:0/2674185061 learned_addr learned my addr 172.21.15.175:0/2674185061 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:18.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f25515af700 1 --2- 172.21.15.175:0/2674185061 >> v2:172.21.15.191:3300/0 conn(0x7f254c0c4580 0x7f254c1504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:18.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f254b7fe700 1 -- 172.21.15.175:0/2674185061 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132361950 0 0) 0x7f254c157030 con 0x7f254c0c7190 2023-10-10T18:59:18.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.240+0000 7f254b7fe700 1 -- 172.21.15.175:0/2674185061 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2534003570 con 0x7f254c0c7190 2023-10-10T18:59:18.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f25515af700 1 -- 172.21.15.175:0/2674185061 >> v1:172.21.15.191:6789/0 conn(0x7f254c0c6330 legacy=0x7f254c0c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:18.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f25515af700 1 -- 172.21.15.175:0/2674185061 >> v1:172.21.15.175:6789/0 conn(0x7f254c0c7190 legacy=0x7f254c14fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:18.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f25515af700 1 -- 172.21.15.175:0/2674185061 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c156700 con 0x7f254c0c4580 2023-10-10T18:59:18.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f25515af700 1 --2- 172.21.15.175:0/2674185061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c4580 0x7f254c1504f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2544009e60 tx=0x7f2544009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=529ec0741e6058e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:18.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f254b7fe700 1 -- 172.21.15.175:0/2674185061 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254400e070 con 0x7f254c0c4580 2023-10-10T18:59:18.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f254b7fe700 1 -- 172.21.15.175:0/2674185061 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254400bbf0 con 0x7f254c0c4580 2023-10-10T18:59:18.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f254b7fe700 1 -- 172.21.15.175:0/2674185061 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2544012430 con 0x7f254c0c4580 2023-10-10T18:59:18.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f2553012700 1 -- 172.21.15.175:0/2674185061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c4580 msgr2=0x7f254c1504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:18.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f2553012700 1 --2- 172.21.15.175:0/2674185061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c4580 0x7f254c1504f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2544009e60 tx=0x7f2544009780 comp rx=0 tx=0).stop 2023-10-10T18:59:18.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f2553012700 1 -- 172.21.15.175:0/2674185061 shutdown_connections 2023-10-10T18:59:18.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f2553012700 1 --2- 172.21.15.175:0/2674185061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c4580 0x7f254c1504f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:18.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f2553012700 1 -- 172.21.15.175:0/2674185061 >> 172.21.15.175:0/2674185061 conn(0x7f254c06ef20 msgr2=0x7f254c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:18.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f2553012700 1 -- 172.21.15.175:0/2674185061 shutdown_connections 2023-10-10T18:59:18.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.241+0000 7f2553012700 1 -- 172.21.15.175:0/2674185061 wait complete. 2023-10-10T18:59:18.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.242+0000 7f2553012700 1 Processor -- start 2023-10-10T18:59:18.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2553012700 1 -- start start 2023-10-10T18:59:18.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2553012700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f254c0c4580 0x7f254c1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:18.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2553012700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c6330 0x7f254c0c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:18.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2553012700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f254c0c7190 0x7f254c0c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:18.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2553012700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c149e60 con 0x7f254c0c4580 2023-10-10T18:59:18.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2553012700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c154d70 con 0x7f254c0c6330 2023-10-10T18:59:18.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2553012700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0c68b0 con 0x7f254c0c7190 2023-10-10T18:59:18.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2550dae700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f254c0c4580 0x7f254c1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:18.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2550dae700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f254c0c4580 0x7f254c1585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54512/0 (socket says 172.21.15.175:54512) 2023-10-10T18:59:18.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.243+0000 7f2550dae700 1 -- 172.21.15.175:0/723298980 learned_addr learned my addr 172.21.15.175:0/723298980 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:18.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f254bfff700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c6330 0x7f254c0c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:18.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f25515af700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f254c0c7190 0x7f254c0c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:18.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2550dae700 1 -- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f254c0c7190 msgr2=0x7f254c0c2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:18.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2550dae700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f254c0c7190 0x7f254c0c2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:18.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2550dae700 1 -- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c6330 msgr2=0x7f254c0c26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:18.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2550dae700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c6330 0x7f254c0c26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:18.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2550dae700 1 -- 172.21.15.175:0/723298980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544009a20 con 0x7f254c0c4580 2023-10-10T18:59:18.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f25515af700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f254c0c7190 0x7f254c0c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T18:59:18.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2550dae700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f254c0c4580 0x7f254c1585e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f253c00c790 tx=0x7f253c007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:18.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2548ff9700 1 -- 172.21.15.175:0/723298980 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c011590 con 0x7f254c0c4580 2023-10-10T18:59:18.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2548ff9700 1 -- 172.21.15.175:0/723298980 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c0116f0 con 0x7f254c0c4580 2023-10-10T18:59:18.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c0c6b10 con 0x7f254c0c4580 2023-10-10T18:59:18.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2548ff9700 1 -- 172.21.15.175:0/723298980 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c011ab0 con 0x7f254c0c4580 2023-10-10T18:59:18.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.244+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c166850 con 0x7f254c0c4580 2023-10-10T18:59:18.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.245+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c067570 con 0x7f254c0c4580 2023-10-10T18:59:18.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.245+0000 7f2548ff9700 1 -- 172.21.15.175:0/723298980 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f253c01b3b0 con 0x7f254c0c4580 2023-10-10T18:59:18.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.246+0000 7f2548ff9700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2538066f30 0x7f25380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:18.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.246+0000 7f2548ff9700 1 -- 172.21.15.175:0/723298980 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f253c083cc0 con 0x7f254c0c4580 2023-10-10T18:59:18.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.246+0000 7f254bfff700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2538066f30 0x7f25380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:18.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.246+0000 7f254bfff700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2538066f30 0x7f25380693e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f254c153d00 tx=0x7f2540009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:18.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.250+0000 7f2548ff9700 1 -- 172.21.15.175:0/723298980 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f253c050c60 con 0x7f254c0c4580 2023-10-10T18:59:18.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.478+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f254c1569e0 con 0x7f254c0c4580 2023-10-10T18:59:18.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.478+0000 7f2548ff9700 1 -- 172.21.15.175:0/723298980 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5434 (secure 0 0 0) 0x7f253c017070 con 0x7f254c0c4580 2023-10-10T18:59:18.479 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:18.479 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:15.595170+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:resolve","state_seq":27,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:18.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2538066f30 msgr2=0x7f25380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:18.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2538066f30 0x7f25380693e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f254c153d00 tx=0x7f2540009040 comp rx=0 tx=0).stop 2023-10-10T18:59:18.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f254c0c4580 msgr2=0x7f254c1585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:18.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f254c0c4580 0x7f254c1585e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f253c00c790 tx=0x7f253c007f40 comp rx=0 tx=0).stop 2023-10-10T18:59:18.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 shutdown_connections 2023-10-10T18:59:18.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2538066f30 0x7f25380693e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:18.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f254c0c7190 0x7f254c0c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:18.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f254c0c4580 0x7f254c1585e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:18.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 --2- 172.21.15.175:0/723298980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f254c0c6330 0x7f254c0c26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:18.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 >> 172.21.15.175:0/723298980 conn(0x7f254c06ef20 msgr2=0x7f254c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:18.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 shutdown_connections 2023-10-10T18:59:18.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:18.481+0000 7f2553012700 1 -- 172.21.15.175:0/723298980 wait complete. 2023-10-10T18:59:18.483 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 101 2023-10-10T18:59:18.493 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:20.261 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:59:20.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:59:20.494 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:20.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.616+0000 7f3fb359e700 1 Processor -- start 2023-10-10T18:59:20.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.616+0000 7f3fb359e700 1 -- start start 2023-10-10T18:59:20.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb359e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3fb408a100 0x7f3fb4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:20.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb359e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb40cd040 0x7f3fb40684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:20.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb359e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb4067fa0 con 0x7f3fb414fd90 2023-10-10T18:59:20.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb359e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb40c5600 con 0x7f3fb408a100 2023-10-10T18:59:20.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb359e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4156b70 con 0x7f3fb40cd040 2023-10-10T18:59:20.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb1d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3fb408a100 0x7f3fb4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:20.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb1d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3fb408a100 0x7f3fb4159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54526/0 (socket says 172.21.15.175:54526) 2023-10-10T18:59:20.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb1d9b700 1 -- 172.21.15.175:0/2139624879 learned_addr learned my addr 172.21.15.175:0/2139624879 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:20.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb2d9d700 1 --2- 172.21.15.175:0/2139624879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb40cd040 0x7f3fb40684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:20.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb1d9b700 1 -- 172.21.15.175:0/2139624879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb40cd040 msgr2=0x7f3fb40684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:20.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb1d9b700 1 --2- 172.21.15.175:0/2139624879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb40cd040 0x7f3fb40684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb1d9b700 1 -- 172.21.15.175:0/2139624879 >> v1:172.21.15.191:6789/0 conn(0x7f3fb414fd90 legacy=0x7f3fb40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:20.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.617+0000 7f3fb1d9b700 1 -- 172.21.15.175:0/2139624879 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb4150160 con 0x7f3fb408a100 2023-10-10T18:59:20.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb1d9b700 1 --2- 172.21.15.175:0/2139624879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb408a100 0x7f3fb4159c30 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0099b0 tx=0x7f3f9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3651617f586b285f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:20.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb159a700 1 -- 172.21.15.175:0/2139624879 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c009ea0 con 0x7f3fb408a100 2023-10-10T18:59:20.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb159a700 1 -- 172.21.15.175:0/2139624879 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9c00b860 con 0x7f3fb408a100 2023-10-10T18:59:20.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb159a700 1 -- 172.21.15.175:0/2139624879 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c00bc00 con 0x7f3fb408a100 2023-10-10T18:59:20.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb359e700 1 -- 172.21.15.175:0/2139624879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb408a100 msgr2=0x7f3fb4159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:20.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb359e700 1 --2- 172.21.15.175:0/2139624879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb408a100 0x7f3fb4159c30 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0099b0 tx=0x7f3f9c009250 comp rx=0 tx=0).stop 2023-10-10T18:59:20.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb359e700 1 -- 172.21.15.175:0/2139624879 shutdown_connections 2023-10-10T18:59:20.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb359e700 1 --2- 172.21.15.175:0/2139624879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb40cd040 0x7f3fb40684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb359e700 1 --2- 172.21.15.175:0/2139624879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb408a100 0x7f3fb4159c30 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb359e700 1 -- 172.21.15.175:0/2139624879 >> 172.21.15.175:0/2139624879 conn(0x7f3fb4063d00 msgr2=0x7f3fb40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:20.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb359e700 1 -- 172.21.15.175:0/2139624879 shutdown_connections 2023-10-10T18:59:20.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.618+0000 7f3fb359e700 1 -- 172.21.15.175:0/2139624879 wait complete. 2023-10-10T18:59:20.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.619+0000 7f3fb359e700 1 Processor -- start 2023-10-10T18:59:20.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.619+0000 7f3fb359e700 1 -- start start 2023-10-10T18:59:20.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb359e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb408a100 0x7f3fb4169590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:20.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb359e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb40cd040 0x7f3fb4169ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:20.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb359e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb414fd90 0x7f3fb414c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:20.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb359e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4166ca0 con 0x7f3fb40cd040 2023-10-10T18:59:20.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb359e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4166780 con 0x7f3fb408a100 2023-10-10T18:59:20.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb359e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb40a7db0 con 0x7f3fb414fd90 2023-10-10T18:59:20.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb1d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb40cd040 0x7f3fb4169ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:20.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb1d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb40cd040 0x7f3fb4169ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54534/0 (socket says 172.21.15.175:54534) 2023-10-10T18:59:20.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb1d9b700 1 -- 172.21.15.175:0/3783104776 learned_addr learned my addr 172.21.15.175:0/3783104776 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:20.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb259c700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb408a100 0x7f3fb4169590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:20.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb2d9d700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb414fd90 0x7f3fb414c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:20.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb1d9b700 1 -- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb414fd90 msgr2=0x7f3fb414c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:20.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb1d9b700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb414fd90 0x7f3fb414c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb1d9b700 1 -- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb408a100 msgr2=0x7f3fb4169590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:20.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb1d9b700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb408a100 0x7f3fb4169590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.620+0000 7f3fb1d9b700 1 -- 172.21.15.175:0/3783104776 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c0094f0 con 0x7f3fb40cd040 2023-10-10T18:59:20.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.621+0000 7f3fb1d9b700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb40cd040 0x7f3fb4169ad0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4064c20 tx=0x7f3f9c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:20.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.621+0000 7f3faa7fc700 1 -- 172.21.15.175:0/3783104776 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c000d10 con 0x7f3fb40cd040 2023-10-10T18:59:20.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.621+0000 7f3faa7fc700 1 -- 172.21.15.175:0/3783104776 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9c00bbe0 con 0x7f3fb40cd040 2023-10-10T18:59:20.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.621+0000 7f3faa7fc700 1 -- 172.21.15.175:0/3783104776 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c011670 con 0x7f3fb40cd040 2023-10-10T18:59:20.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.621+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb41669c0 con 0x7f3fb40cd040 2023-10-10T18:59:20.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.621+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb40a8330 con 0x7f3fb40cd040 2023-10-10T18:59:20.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.622+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb40a7c10 con 0x7f3fb40cd040 2023-10-10T18:59:20.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.622+0000 7f3faa7fc700 1 -- 172.21.15.175:0/3783104776 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3f9c011a60 con 0x7f3fb40cd040 2023-10-10T18:59:20.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.623+0000 7f3faa7fc700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3f94066f30 0x7f3f940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:20.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.623+0000 7f3faa7fc700 1 -- 172.21.15.175:0/3783104776 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f3f9c085fd0 con 0x7f3fb40cd040 2023-10-10T18:59:20.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.623+0000 7f3fb259c700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3f94066f30 0x7f3f940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:20.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.623+0000 7f3fb259c700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3f94066f30 0x7f3f940693e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4007fd0 tx=0x7f3fa4001650 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:20.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.627+0000 7f3faa7fc700 1 -- 172.21.15.175:0/3783104776 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3fb40a7c10 con 0x7f3fb40cd040 2023-10-10T18:59:20.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.855+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fb41692b0 con 0x7f3fb40cd040 2023-10-10T18:59:20.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.856+0000 7f3faa7fc700 1 -- 172.21.15.175:0/3783104776 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5434 (secure 0 0 0) 0x7f3fb41692b0 con 0x7f3fb40cd040 2023-10-10T18:59:20.856 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:20.857 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:15.595170+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:resolve","state_seq":27,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:20.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3f94066f30 msgr2=0x7f3f940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:20.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3f94066f30 0x7f3f940693e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4007fd0 tx=0x7f3fa4001650 comp rx=0 tx=0).stop 2023-10-10T18:59:20.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb40cd040 msgr2=0x7f3fb4169ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:20.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb40cd040 0x7f3fb4169ad0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4064c20 tx=0x7f3f9c017000 comp rx=0 tx=0).stop 2023-10-10T18:59:20.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 shutdown_connections 2023-10-10T18:59:20.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3f94066f30 0x7f3f940693e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fb414fd90 0x7f3fb414c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb408a100 0x7f3fb4169590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 --2- 172.21.15.175:0/3783104776 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fb40cd040 0x7f3fb4169ad0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:20.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 >> 172.21.15.175:0/3783104776 conn(0x7f3fb4063d00 msgr2=0x7f3fb40cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:20.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 shutdown_connections 2023-10-10T18:59:20.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:20.858+0000 7f3fb359e700 1 -- 172.21.15.175:0/3783104776 wait complete. 2023-10-10T18:59:20.861 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 101 2023-10-10T18:59:20.870 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 31 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:resolve", "state_seq": 27 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:59:15.595170+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 5332, "mds_2": 5367, "mds_3": 5196, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T18:59:22.871 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:22.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.989+0000 7f9b9a1b3700 1 Processor -- start 2023-10-10T18:59:22.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.989+0000 7f9b9a1b3700 1 -- start start 2023-10-10T18:59:22.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b9a1b3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9b940b1fc0 0x7f9b94155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:22.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b9a1b3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b940b2390 0x7f9b94068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:22.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b9a1b3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b941574b0 con 0x7f9b94068bd0 2023-10-10T18:59:22.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b9a1b3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b94150770 con 0x7f9b940b1fc0 2023-10-10T18:59:22.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b9a1b3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940afad0 con 0x7f9b940b2390 2023-10-10T18:59:22.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b937fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9b940b1fc0 0x7f9b94155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:22.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b92ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b940b2390 0x7f9b94068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:22.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b92ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b940b2390 0x7f9b94068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:35612/0 (socket says 172.21.15.175:35612) 2023-10-10T18:59:22.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b937fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9b940b1fc0 0x7f9b94155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:58878/0 (socket says 172.21.15.175:58878) 2023-10-10T18:59:22.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b92ffd700 1 -- 172.21.15.175:0/156541683 learned_addr learned my addr 172.21.15.175:0/156541683 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:22.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.990+0000 7f9b93fff700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f9b94068bd0 0x7f9b940aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:49798/0 (socket says 172.21.15.175:49798) 2023-10-10T18:59:22.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b937fe700 1 -- 172.21.15.175:0/156541683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b940b2390 msgr2=0x7f9b94068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:22.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b937fe700 1 --2- 172.21.15.175:0/156541683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b940b2390 0x7f9b94068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:22.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b937fe700 1 -- 172.21.15.175:0/156541683 >> v1:172.21.15.191:6789/0 conn(0x7f9b94068bd0 legacy=0x7f9b940aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:22.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b937fe700 1 -- 172.21.15.175:0/156541683 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b940af4c0 con 0x7f9b940b1fc0 2023-10-10T18:59:22.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b92ffd700 1 --2- 172.21.15.175:0/156541683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b940b2390 0x7f9b94068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:59:22.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b927fc700 1 -- 172.21.15.175:0/156541683 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450613780 0 0) 0x7f9b941574b0 con 0x7f9b94068bd0 2023-10-10T18:59:22.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b937fe700 1 --2- 172.21.15.175:0/156541683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 0x7f9b94155fb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009e60 tx=0x7f9b84009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21429e82b45f3ea3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:22.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b927fc700 1 -- 172.21.15.175:0/156541683 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400e070 con 0x7f9b940b1fc0 2023-10-10T18:59:22.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b927fc700 1 -- 172.21.15.175:0/156541683 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8400bbf0 con 0x7f9b940b1fc0 2023-10-10T18:59:22.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.991+0000 7f9b927fc700 1 -- 172.21.15.175:0/156541683 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84012430 con 0x7f9b940b1fc0 2023-10-10T18:59:22.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.992+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/156541683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 msgr2=0x7f9b94155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:22.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.992+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/156541683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 0x7f9b94155fb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009e60 tx=0x7f9b84009780 comp rx=0 tx=0).stop 2023-10-10T18:59:22.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.992+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/156541683 shutdown_connections 2023-10-10T18:59:22.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.992+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/156541683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 0x7f9b94155fb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:22.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.992+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/156541683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b940b2390 0x7f9b94068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:22.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.992+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/156541683 >> 172.21.15.175:0/156541683 conn(0x7f9b94063230 msgr2=0x7f9b94063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:22.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.992+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/156541683 shutdown_connections 2023-10-10T18:59:22.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.992+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/156541683 wait complete. 2023-10-10T18:59:23.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.993+0000 7f9b9a1b3700 1 Processor -- start 2023-10-10T18:59:23.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.993+0000 7f9b9a1b3700 1 -- start start 2023-10-10T18:59:23.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b9a1b3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b94068bd0 0x7f9b94159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:23.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b9a1b3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 0x7f9b94166560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:23.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b9a1b3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b940b2390 0x7f9b94150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:23.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b9a1b3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9415a000 con 0x7f9b940b2390 2023-10-10T18:59:23.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b9a1b3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b94166aa0 con 0x7f9b940b1fc0 2023-10-10T18:59:23.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b9a1b3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940cc140 con 0x7f9b94068bd0 2023-10-10T18:59:23.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b940b2390 0x7f9b94150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:23.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b940b2390 0x7f9b94150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54538/0 (socket says 172.21.15.175:54538) 2023-10-10T18:59:23.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 -- 172.21.15.175:0/567181657 learned_addr learned my addr 172.21.15.175:0/567181657 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:23.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b92ffd700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 0x7f9b94166560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:23.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b937fe700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b94068bd0 0x7f9b94159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:23.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 -- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b94068bd0 msgr2=0x7f9b94159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:23.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b94068bd0 0x7f9b94159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:23.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 -- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 msgr2=0x7f9b94166560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:23.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 0x7f9b94166560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:23.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 -- 172.21.15.175:0/567181657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b880094f0 con 0x7f9b940b2390 2023-10-10T18:59:23.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b92ffd700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 0x7f9b94166560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:23.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b93fff700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b940b2390 0x7f9b94150fa0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c007fd0 tx=0x7f9b8c007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:23.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b7b7fe700 1 -- 172.21.15.175:0/567181657 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c005a70 con 0x7f9b940b2390 2023-10-10T18:59:23.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b7b7fe700 1 -- 172.21.15.175:0/567181657 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c005bd0 con 0x7f9b940b2390 2023-10-10T18:59:23.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b7b7fe700 1 -- 172.21.15.175:0/567181657 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0070d0 con 0x7f9b940b2390 2023-10-10T18:59:23.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b84009a20 con 0x7f9b940b2390 2023-10-10T18:59:23.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.994+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b940cccc0 con 0x7f9b940b2390 2023-10-10T18:59:23.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.995+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b940cce50 con 0x7f9b940b2390 2023-10-10T18:59:23.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.996+0000 7f9b7b7fe700 1 -- 172.21.15.175:0/567181657 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9b8c007480 con 0x7f9b940b2390 2023-10-10T18:59:23.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.996+0000 7f9b7b7fe700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b80066f30 0x7f9b800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:23.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.996+0000 7f9b7b7fe700 1 -- 172.21.15.175:0/567181657 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f9b8c083c10 con 0x7f9b940b2390 2023-10-10T18:59:23.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.996+0000 7f9b937fe700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b80066f30 0x7f9b800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:23.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:22.997+0000 7f9b937fe700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b80066f30 0x7f9b800693e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9b840097c0 tx=0x7f9b84019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:23.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.000+0000 7f9b7b7fe700 1 -- 172.21.15.175:0/567181657 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b8c050c60 con 0x7f9b940b2390 2023-10-10T18:59:23.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.226+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b94069060 con 0x7f9b940b2390 2023-10-10T18:59:23.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.227+0000 7f9b7b7fe700 1 -- 172.21.15.175:0/567181657 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5436 (secure 0 0 0) 0x7f9b8c017070 con 0x7f9b940b2390 2023-10-10T18:59:23.228 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:23.228 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:21.606429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:23.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b80066f30 msgr2=0x7f9b800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:23.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b80066f30 0x7f9b800693e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9b840097c0 tx=0x7f9b84019000 comp rx=0 tx=0).stop 2023-10-10T18:59:23.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b940b2390 msgr2=0x7f9b94150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:23.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b940b2390 0x7f9b94150fa0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c007fd0 tx=0x7f9b8c007cf0 comp rx=0 tx=0).stop 2023-10-10T18:59:23.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 shutdown_connections 2023-10-10T18:59:23.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b80066f30 0x7f9b800693e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:23.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b940b2390 0x7f9b94150fa0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:23.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b94068bd0 0x7f9b94159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:23.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 --2- 172.21.15.175:0/567181657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b940b1fc0 0x7f9b94166560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:23.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 >> 172.21.15.175:0/567181657 conn(0x7f9b94063230 msgr2=0x7f9b940c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:23.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 shutdown_connections 2023-10-10T18:59:23.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:23.229+0000 7f9b9a1b3700 1 -- 172.21.15.175:0/567181657 wait complete. 2023-10-10T18:59:23.233 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 102 2023-10-10T18:59:23.242 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:25.243 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:25.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.369+0000 7f5f46e16700 1 Processor -- start 2023-10-10T18:59:25.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.369+0000 7f5f46e16700 1 -- start start 2023-10-10T18:59:25.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f46e16700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5f4014fd90 0x7f5f40159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:25.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f46e16700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5f4008a100 0x7f5f400c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:25.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f46e16700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f40067fa0 con 0x7f5f400cd040 2023-10-10T18:59:25.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f46e16700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f400c5600 con 0x7f5f4014fd90 2023-10-10T18:59:25.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f46e16700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f40156b70 con 0x7f5f4008a100 2023-10-10T18:59:25.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f45e14700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5f4014fd90 0x7f5f40159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:25.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f46615700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5f400cd040 0x7f5f400684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50896/0 (socket says 172.21.15.175:50896) 2023-10-10T18:59:25.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f46615700 1 -- 172.21.15.175:0/4157391854 learned_addr learned my addr 172.21.15.175:0/4157391854 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:25.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f45613700 1 --2- 172.21.15.175:0/4157391854 >> v2:172.21.15.191:3300/0 conn(0x7f5f4008a100 0x7f5f400c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:25.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f45e14700 1 -- 172.21.15.175:0/4157391854 >> v2:172.21.15.191:3300/0 conn(0x7f5f4008a100 msgr2=0x7f5f400c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:25.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.370+0000 7f5f45e14700 1 --2- 172.21.15.175:0/4157391854 >> v2:172.21.15.191:3300/0 conn(0x7f5f4008a100 0x7f5f400c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f45e14700 1 -- 172.21.15.175:0/4157391854 >> v1:172.21.15.175:6789/0 conn(0x7f5f400cd040 legacy=0x7f5f400684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:25.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f44e12700 1 -- 172.21.15.175:0/4157391854 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726416492 0 0) 0x7f5f40067fa0 con 0x7f5f400cd040 2023-10-10T18:59:25.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f45e14700 1 -- 172.21.15.175:0/4157391854 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f40150160 con 0x7f5f4014fd90 2023-10-10T18:59:25.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f45e14700 1 --2- 172.21.15.175:0/4157391854 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40159c30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5f40069990 tx=0x7f5f30009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ad1a2aa26fb5e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:25.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f44e12700 1 -- 172.21.15.175:0/4157391854 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3000e070 con 0x7f5f4014fd90 2023-10-10T18:59:25.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f44e12700 1 -- 172.21.15.175:0/4157391854 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3000bcc0 con 0x7f5f4014fd90 2023-10-10T18:59:25.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f44e12700 1 -- 172.21.15.175:0/4157391854 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30012680 con 0x7f5f4014fd90 2023-10-10T18:59:25.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f46e16700 1 -- 172.21.15.175:0/4157391854 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 msgr2=0x7f5f40159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:25.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f46e16700 1 --2- 172.21.15.175:0/4157391854 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40159c30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5f40069990 tx=0x7f5f30009780 comp rx=0 tx=0).stop 2023-10-10T18:59:25.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f46e16700 1 -- 172.21.15.175:0/4157391854 shutdown_connections 2023-10-10T18:59:25.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f46e16700 1 --2- 172.21.15.175:0/4157391854 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40159c30 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f46e16700 1 --2- 172.21.15.175:0/4157391854 >> v2:172.21.15.191:3300/0 conn(0x7f5f4008a100 0x7f5f400c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f46e16700 1 -- 172.21.15.175:0/4157391854 >> 172.21.15.175:0/4157391854 conn(0x7f5f40063d00 msgr2=0x7f5f400cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:25.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f46e16700 1 -- 172.21.15.175:0/4157391854 shutdown_connections 2023-10-10T18:59:25.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.371+0000 7f5f46e16700 1 -- 172.21.15.175:0/4157391854 wait complete. 2023-10-10T18:59:25.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.372+0000 7f5f46e16700 1 Processor -- start 2023-10-10T18:59:25.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46e16700 1 -- start start 2023-10-10T18:59:25.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46e16700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f4008a100 0x7f5f40156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:25.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46e16700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f400cd040 0x7f5f401695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:25.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46e16700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40165e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:25.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46e16700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40152070 con 0x7f5f4014fd90 2023-10-10T18:59:25.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46e16700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f401572c0 con 0x7f5f400cd040 2023-10-10T18:59:25.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46e16700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40069650 con 0x7f5f4008a100 2023-10-10T18:59:25.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46615700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40165e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:25.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46615700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40165e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59572/0 (socket says 172.21.15.175:59572) 2023-10-10T18:59:25.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f45613700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f400cd040 0x7f5f401695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:25.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f46615700 1 -- 172.21.15.175:0/2240768810 learned_addr learned my addr 172.21.15.175:0/2240768810 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:25.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.373+0000 7f5f45e14700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f4008a100 0x7f5f40156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:25.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f46615700 1 -- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f4008a100 msgr2=0x7f5f40156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:25.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f46615700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f4008a100 0x7f5f40156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f46615700 1 -- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f400cd040 msgr2=0x7f5f401695e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:25.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f46615700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f400cd040 0x7f5f401695e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f46615700 1 -- 172.21.15.175:0/2240768810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f30009a20 con 0x7f5f4014fd90 2023-10-10T18:59:25.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f46615700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40165e10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5f3800c810 tx=0x7f5f38007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:25.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f3dffb700 1 -- 172.21.15.175:0/2240768810 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38005c00 con 0x7f5f4014fd90 2023-10-10T18:59:25.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f3dffb700 1 -- 172.21.15.175:0/2240768810 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f38005d80 con 0x7f5f4014fd90 2023-10-10T18:59:25.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40151780 con 0x7f5f4014fd90 2023-10-10T18:59:25.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f3dffb700 1 -- 172.21.15.175:0/2240768810 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38007320 con 0x7f5f4014fd90 2023-10-10T18:59:25.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.374+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f40151a20 con 0x7f5f4014fd90 2023-10-10T18:59:25.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.375+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f40152220 con 0x7f5f4014fd90 2023-10-10T18:59:25.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.375+0000 7f5f3dffb700 1 -- 172.21.15.175:0/2240768810 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5f3801b3b0 con 0x7f5f4014fd90 2023-10-10T18:59:25.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.376+0000 7f5f3dffb700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f2c066f30 0x7f5f2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:25.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.376+0000 7f5f3dffb700 1 -- 172.21.15.175:0/2240768810 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f5f38083cc0 con 0x7f5f4014fd90 2023-10-10T18:59:25.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.376+0000 7f5f45e14700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f2c066f30 0x7f5f2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:25.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.376+0000 7f5f45e14700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f2c066f30 0x7f5f2c0693e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5f300099f0 tx=0x7f5f30019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:25.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.380+0000 7f5f3dffb700 1 -- 172.21.15.175:0/2240768810 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f40152220 con 0x7f5f4014fd90 2023-10-10T18:59:25.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.607+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f40152220 con 0x7f5f4014fd90 2023-10-10T18:59:25.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.608+0000 7f5f3dffb700 1 -- 172.21.15.175:0/2240768810 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5436 (secure 0 0 0) 0x7f5f40152220 con 0x7f5f4014fd90 2023-10-10T18:59:25.609 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:25.610 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:21.606429+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:25.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f2c066f30 msgr2=0x7f5f2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:25.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f2c066f30 0x7f5f2c0693e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5f300099f0 tx=0x7f5f30019000 comp rx=0 tx=0).stop 2023-10-10T18:59:25.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 msgr2=0x7f5f40165e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:25.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40165e10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5f3800c810 tx=0x7f5f38007d90 comp rx=0 tx=0).stop 2023-10-10T18:59:25.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 shutdown_connections 2023-10-10T18:59:25.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f2c066f30 0x7f5f2c0693e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f4014fd90 0x7f5f40165e10 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5f4008a100 0x7f5f40156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 --2- 172.21.15.175:0/2240768810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f400cd040 0x7f5f401695e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:25.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 >> 172.21.15.175:0/2240768810 conn(0x7f5f40063d00 msgr2=0x7f5f400c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:25.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 shutdown_connections 2023-10-10T18:59:25.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:25.610+0000 7f5f46e16700 1 -- 172.21.15.175:0/2240768810 wait complete. 2023-10-10T18:59:25.614 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 102 2023-10-10T18:59:25.623 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:27.624 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:27.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.741+0000 7fafcff94700 1 Processor -- start 2023-10-10T18:59:27.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.741+0000 7fafcff94700 1 -- start start 2023-10-10T18:59:27.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.742+0000 7fafcff94700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fafc814fbf0 0x7fafc8159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:27.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.742+0000 7fafcff94700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80681e0 0x7fafc80c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:27.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.742+0000 7fafcff94700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc8156800 con 0x7fafc814ffc0 2023-10-10T18:59:27.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.742+0000 7fafcff94700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc80c59e0 con 0x7fafc814fbf0 2023-10-10T18:59:27.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.742+0000 7fafcff94700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc815a2b0 con 0x7fafc80681e0 2023-10-10T18:59:27.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafcdd30700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fafc814fbf0 0x7fafc8159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:27.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafce531700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80681e0 0x7fafc80c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:27.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafcdd30700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fafc814fbf0 0x7fafc8159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:33918/0 (socket says 172.21.15.175:33918) 2023-10-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafcdd30700 1 -- 172.21.15.175:0/1939426915 learned_addr learned my addr 172.21.15.175:0/1939426915 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafce531700 1 --2- 172.21.15.175:0/1939426915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80681e0 0x7fafc80c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:60534/0 (socket says 172.21.15.175:60534) 2023-10-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafcd52f700 1 --1- 172.21.15.175:0/1939426915 >> v1:172.21.15.191:6789/0 conn(0x7fafc814ffc0 0x7fafc8063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:48886/0 (socket says 172.21.15.175:48886) 2023-10-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafce531700 1 -- 172.21.15.175:0/1939426915 >> v2:172.21.15.191:3300/0 conn(0x7fafc814fbf0 msgr2=0x7fafc8159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafce531700 1 --2- 172.21.15.175:0/1939426915 >> v2:172.21.15.191:3300/0 conn(0x7fafc814fbf0 0x7fafc8159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafce531700 1 -- 172.21.15.175:0/1939426915 >> v1:172.21.15.191:6789/0 conn(0x7fafc814ffc0 legacy=0x7fafc8063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafce531700 1 -- 172.21.15.175:0/1939426915 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc8063e30 con 0x7fafc80681e0 2023-10-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafccd2e700 1 -- 172.21.15.175:0/1939426915 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799515082 0 0) 0x7fafc8156800 con 0x7fafc814ffc0 2023-10-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafcdd30700 1 --2- 172.21.15.175:0/1939426915 >> v2:172.21.15.191:3300/0 conn(0x7fafc814fbf0 0x7fafc8159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafce531700 1 --2- 172.21.15.175:0/1939426915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80681e0 0x7fafc80c5fb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fafc00098e0 tx=0x7fafc00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d3116588ea3b5d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:27.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafccd2e700 1 -- 172.21.15.175:0/1939426915 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc000b640 con 0x7fafc80681e0 2023-10-10T18:59:27.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafccd2e700 1 -- 172.21.15.175:0/1939426915 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc000b7a0 con 0x7fafc80681e0 2023-10-10T18:59:27.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.743+0000 7fafccd2e700 1 -- 172.21.15.175:0/1939426915 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc000bb40 con 0x7fafc80681e0 2023-10-10T18:59:27.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.744+0000 7fafcff94700 1 -- 172.21.15.175:0/1939426915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80681e0 msgr2=0x7fafc80c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:27.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.744+0000 7fafcff94700 1 --2- 172.21.15.175:0/1939426915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80681e0 0x7fafc80c5fb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fafc00098e0 tx=0x7fafc00091a0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.744+0000 7fafcff94700 1 -- 172.21.15.175:0/1939426915 shutdown_connections 2023-10-10T18:59:27.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.744+0000 7fafcff94700 1 --2- 172.21.15.175:0/1939426915 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc80681e0 0x7fafc80c5fb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.744+0000 7fafcff94700 1 --2- 172.21.15.175:0/1939426915 >> v2:172.21.15.191:3300/0 conn(0x7fafc814fbf0 0x7fafc8159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.744+0000 7fafcff94700 1 -- 172.21.15.175:0/1939426915 >> 172.21.15.175:0/1939426915 conn(0x7fafc80cc760 msgr2=0x7fafc80ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:27.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.744+0000 7fafcff94700 1 -- 172.21.15.175:0/1939426915 shutdown_connections 2023-10-10T18:59:27.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.744+0000 7fafcff94700 1 -- 172.21.15.175:0/1939426915 wait complete. 2023-10-10T18:59:27.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.745+0000 7fafcff94700 1 Processor -- start 2023-10-10T18:59:27.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.745+0000 7fafcff94700 1 -- start start 2023-10-10T18:59:27.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcff94700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc80681e0 0x7fafc8152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:27.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcff94700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc814fbf0 0x7fafc8158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:27.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcff94700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc814ffc0 0x7fafc8151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:27.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcff94700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8169be0 con 0x7fafc80681e0 2023-10-10T18:59:27.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcff94700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80c9fc0 con 0x7fafc814ffc0 2023-10-10T18:59:27.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcff94700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc816a070 con 0x7fafc814fbf0 2023-10-10T18:59:27.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc80681e0 0x7fafc8152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:27.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc80681e0 0x7fafc8152d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59584/0 (socket says 172.21.15.175:59584) 2023-10-10T18:59:27.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 -- 172.21.15.175:0/1011644500 learned_addr learned my addr 172.21.15.175:0/1011644500 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:27.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafce531700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc814ffc0 0x7fafc8151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:27.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcd52f700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc814fbf0 0x7fafc8158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:27.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 -- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc814fbf0 msgr2=0x7fafc8158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:27.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc814fbf0 0x7fafc8158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 -- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc814ffc0 msgr2=0x7fafc8151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:27.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc814ffc0 0x7fafc8151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 -- 172.21.15.175:0/1011644500 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8009ae0 con 0x7fafc80681e0 2023-10-10T18:59:27.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcd52f700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc814fbf0 0x7fafc8158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:27.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafcdd30700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc80681e0 0x7fafc8152d60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fafb800c170 tx=0x7fafb800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:27.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.746+0000 7fafc5ffb700 1 -- 172.21.15.175:0/1011644500 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb800cb50 con 0x7fafc80681e0 2023-10-10T18:59:27.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.747+0000 7fafc5ffb700 1 -- 172.21.15.175:0/1011644500 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb800f040 con 0x7fafc80681e0 2023-10-10T18:59:27.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.747+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc0009420 con 0x7fafc80681e0 2023-10-10T18:59:27.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.747+0000 7fafc5ffb700 1 -- 172.21.15.175:0/1011644500 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8014480 con 0x7fafc80681e0 2023-10-10T18:59:27.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.747+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc81692b0 con 0x7fafc80681e0 2023-10-10T18:59:27.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.748+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc8067570 con 0x7fafc80681e0 2023-10-10T18:59:27.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.748+0000 7fafc5ffb700 1 -- 172.21.15.175:0/1011644500 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fafb80147f0 con 0x7fafc80681e0 2023-10-10T18:59:27.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.748+0000 7fafc5ffb700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb4066f60 0x7fafb4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:27.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.748+0000 7fafc5ffb700 1 -- 172.21.15.175:0/1011644500 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fafb8085190 con 0x7fafc80681e0 2023-10-10T18:59:27.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.748+0000 7fafcd52f700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb4066f60 0x7fafb4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:27.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.749+0000 7fafcd52f700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb4066f60 0x7fafb4069410 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fafc81527e0 tx=0x7fafbc007da0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:27.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.753+0000 7fafc5ffb700 1 -- 172.21.15.175:0/1011644500 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafb8052130 con 0x7fafc80681e0 2023-10-10T18:59:27.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.977+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafc8063830 con 0x7fafc80681e0 2023-10-10T18:59:27.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.977+0000 7fafc5ffb700 1 -- 172.21.15.175:0/1011644500 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5433 (secure 0 0 0) 0x7fafb8055750 con 0x7fafc80681e0 2023-10-10T18:59:27.979 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:27.979 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:26.625709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:27.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.979+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb4066f60 msgr2=0x7fafb4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.979+0000 7fafcff94700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb4066f60 0x7fafb4069410 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fafc81527e0 tx=0x7fafbc007da0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.979+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc80681e0 msgr2=0x7fafc8152d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.979+0000 7fafcff94700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc80681e0 0x7fafc8152d60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fafb800c170 tx=0x7fafb800b040 comp rx=0 tx=0).stop 2023-10-10T18:59:27.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.979+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 shutdown_connections 2023-10-10T18:59:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.980+0000 7fafcff94700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafb4066f60 0x7fafb4069410 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.980+0000 7fafcff94700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafc814ffc0 0x7fafc8151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.980+0000 7fafcff94700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc80681e0 0x7fafc8152d60 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.980+0000 7fafcff94700 1 --2- 172.21.15.175:0/1011644500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafc814fbf0 0x7fafc8158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:27.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.980+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 >> 172.21.15.175:0/1011644500 conn(0x7fafc80cc760 msgr2=0x7fafc80cad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:27.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.980+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 shutdown_connections 2023-10-10T18:59:27.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:27.980+0000 7fafcff94700 1 -- 172.21.15.175:0/1011644500 wait complete. 2023-10-10T18:59:27.983 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 103 2023-10-10T18:59:27.992 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:29.994 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:30.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.119+0000 7f9da3e38700 1 Processor -- start 2023-10-10T18:59:30.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.119+0000 7f9da3e38700 1 -- start start 2023-10-10T18:59:30.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.119+0000 7f9da3e38700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d9c0b2390 0x7f9d9c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.119+0000 7f9da3e38700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9d9c0aff60 0x7f9d9c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da3e38700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c1574b0 con 0x7f9d9c0b1fc0 2023-10-10T18:59:30.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da3e38700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c150770 con 0x7f9d9c0aff60 2023-10-10T18:59:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da3e38700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0afad0 con 0x7f9d9c0b2390 2023-10-10T18:59:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da1bd4700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9d9c0b1fc0 0x7f9d9c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50910/0 (socket says 172.21.15.175:50910) 2023-10-10T18:59:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da1bd4700 1 -- 172.21.15.175:0/1710746397 learned_addr learned my addr 172.21.15.175:0/1710746397 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da23d5700 1 --2- 172.21.15.175:0/1710746397 >> v2:172.21.15.191:3300/0 conn(0x7f9d9c0aff60 0x7f9d9c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da13d3700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d9c0b2390 0x7f9d9c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698405495 0 0) 0x7f9d9c1574b0 con 0x7f9d9c0b1fc0 2023-10-10T18:59:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d84003570 con 0x7f9d9c0b1fc0 2023-10-10T18:59:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1962663011 0 0) 0x7f9d84003570 con 0x7f9d9c0b1fc0 2023-10-10T18:59:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d9c0b2390 msgr2=0x7f9d9c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d9c0b2390 0x7f9d9c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 >> v2:172.21.15.191:3300/0 conn(0x7f9d9c0aff60 msgr2=0x7f9d9c0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 --2- 172.21.15.175:0/1710746397 >> v2:172.21.15.191:3300/0 conn(0x7f9d9c0aff60 0x7f9d9c0b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0af4c0 con 0x7f9d9c0b1fc0 2023-10-10T18:59:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f9d8c0017f0 con 0x7f9d9c0b1fc0 2023-10-10T18:59:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 >> v1:172.21.15.175:6789/0 conn(0x7f9d9c0b1fc0 legacy=0x7f9d9c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d84003a60 0x7f9d84003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d9c0b2390 0x7f9d84004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d84004d50 0x7f9d840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c150770 con 0x7f9d84003a60 2023-10-10T18:59:30.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0afad0 con 0x7f9d84004d50 2023-10-10T18:59:30.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8400ba60 con 0x7f9d9c0b2390 2023-10-10T18:59:30.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da1bd4700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d84003a60 0x7f9d84003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da23d5700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d84004d50 0x7f9d840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.120+0000 7f9da13d3700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d9c0b2390 0x7f9d84004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da1bd4700 1 -- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d9c0b2390 msgr2=0x7f9d84004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:30.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da1bd4700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d9c0b2390 0x7f9d84004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da1bd4700 1 -- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d84004d50 msgr2=0x7f9d840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:30.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da1bd4700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d84004d50 0x7f9d840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da1bd4700 1 -- 172.21.15.175:0/1710746397 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d84003570 con 0x7f9d84003a60 2023-10-10T18:59:30.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da1bd4700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d84003a60 0x7f9d84003e70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00a900 tx=0x7f9d9c0afa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03780acf6e6cd04 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:30.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0070d0 con 0x7f9d84003a60 2023-10-10T18:59:30.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c007230 con 0x7f9d84003a60 2023-10-10T18:59:30.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.121+0000 7f9da0bd2700 1 -- 172.21.15.175:0/1710746397 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0075f0 con 0x7f9d84003a60 2023-10-10T18:59:30.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.122+0000 7f9da3e38700 1 -- 172.21.15.175:0/1710746397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d84003a60 msgr2=0x7f9d84003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.122+0000 7f9da3e38700 1 --2- 172.21.15.175:0/1710746397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d84003a60 0x7f9d84003e70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00a900 tx=0x7f9d9c0afa40 comp rx=0 tx=0).stop 2023-10-10T18:59:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.122+0000 7f9da23d5700 1 -- 172.21.15.175:0/1710746397 reap_dead start 2023-10-10T18:59:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.122+0000 7f9da3e38700 1 -- 172.21.15.175:0/1710746397 shutdown_connections 2023-10-10T18:59:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.122+0000 7f9da3e38700 1 -- 172.21.15.175:0/1710746397 >> 172.21.15.175:0/1710746397 conn(0x7f9d9c063230 msgr2=0x7f9d9c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.122+0000 7f9da3e38700 1 -- 172.21.15.175:0/1710746397 shutdown_connections 2023-10-10T18:59:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.122+0000 7f9da3e38700 1 -- 172.21.15.175:0/1710746397 wait complete. 2023-10-10T18:59:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.123+0000 7f9da3e38700 1 Processor -- start 2023-10-10T18:59:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.123+0000 7f9da3e38700 1 -- start start 2023-10-10T18:59:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.123+0000 7f9da3e38700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d84003a60 0x7f9d9c162de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.123+0000 7f9da3e38700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d9c159050 0x7f9d9c14f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.123+0000 7f9da3e38700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d9c14e5b0 0x7f9d9c14ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.123+0000 7f9da3e38700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c161960 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.123+0000 7f9da3e38700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c158360 con 0x7f9d9c159050 2023-10-10T18:59:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.123+0000 7f9da3e38700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c158bc0 con 0x7f9d84003a60 2023-10-10T18:59:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d9c14e5b0 0x7f9d9c14ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d9c14e5b0 0x7f9d9c14ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59602/0 (socket says 172.21.15.175:59602) 2023-10-10T18:59:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 -- 172.21.15.175:0/2073492731 learned_addr learned my addr 172.21.15.175:0/2073492731 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da1bd4700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d84003a60 0x7f9d9c162de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da13d3700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d9c159050 0x7f9d9c14f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 -- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d84003a60 msgr2=0x7f9d9c162de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:30.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d84003a60 0x7f9d9c162de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 -- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d9c159050 msgr2=0x7f9d9c14f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:30.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d9c159050 0x7f9d9c14f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 -- 172.21.15.175:0/2073492731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c163320 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da23d5700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d9c14e5b0 0x7f9d9c14ea20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0b1c70 tx=0x7f9d940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9d99ffb700 1 -- 172.21.15.175:0/2073492731 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94013750 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9d99ffb700 1 -- 172.21.15.175:0/2073492731 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d940138b0 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9d99ffb700 1 -- 172.21.15.175:0/2073492731 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94013c50 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c15a270 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.124+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c158570 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.125+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c158d70 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.126+0000 7f9d99ffb700 1 -- 172.21.15.175:0/2073492731 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9d94014650 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.126+0000 7f9d99ffb700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d88066f60 0x7f9d88069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:30.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.126+0000 7f9d99ffb700 1 -- 172.21.15.175:0/2073492731 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f9d94084ce0 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.126+0000 7f9da1bd4700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d88066f60 0x7f9d88069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:30.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.127+0000 7f9da1bd4700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d88066f60 0x7f9d88069410 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00a3f0 tx=0x7f9d8c015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:30.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.130+0000 7f9d99ffb700 1 -- 172.21.15.175:0/2073492731 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d9c158d70 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.352+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d9c158d70 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.352+0000 7f9d99ffb700 1 -- 172.21.15.175:0/2073492731 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5433 (secure 0 0 0) 0x7f9d9c158d70 con 0x7f9d9c14e5b0 2023-10-10T18:59:30.353 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:30.354 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:26.625709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d88066f60 msgr2=0x7f9d88069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d88066f60 0x7f9d88069410 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00a3f0 tx=0x7f9d8c015000 comp rx=0 tx=0).stop 2023-10-10T18:59:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d9c14e5b0 msgr2=0x7f9d9c14ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d9c14e5b0 0x7f9d9c14ea20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0b1c70 tx=0x7f9d940091a0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 shutdown_connections 2023-10-10T18:59:30.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9d88066f60 0x7f9d88069410 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d9c14e5b0 0x7f9d9c14ea20 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d84003a60 0x7f9d9c162de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 --2- 172.21.15.175:0/2073492731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d9c159050 0x7f9d9c14f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:30.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 >> 172.21.15.175:0/2073492731 conn(0x7f9d9c063230 msgr2=0x7f9d9c155020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:30.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.354+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 shutdown_connections 2023-10-10T18:59:30.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:30.355+0000 7f9da3e38700 1 -- 172.21.15.175:0/2073492731 wait complete. 2023-10-10T18:59:30.358 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 103 2023-10-10T18:59:30.367 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:32.368 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:32.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.495+0000 7f3993bab700 1 Processor -- start 2023-10-10T18:59:32.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.495+0000 7f3993bab700 1 -- start start 2023-10-10T18:59:32.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3993bab700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b1fc0 0x7f398c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:32.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3993bab700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f398c0aff60 0x7f398c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:32.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3993bab700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c1574b0 con 0x7f398c0b2390 2023-10-10T18:59:32.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3993bab700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398c150770 con 0x7f398c0aff60 2023-10-10T18:59:32.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3993bab700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398c0afad0 con 0x7f398c0b1fc0 2023-10-10T18:59:32.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3992148700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f398c0aff60 0x7f398c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:32.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3991146700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f398c0b2390 0x7f398c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50914/0 (socket says 172.21.15.175:50914) 2023-10-10T18:59:32.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3991146700 1 -- 172.21.15.175:0/211936096 learned_addr learned my addr 172.21.15.175:0/211936096 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:32.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3991947700 1 --2- 172.21.15.175:0/211936096 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b1fc0 0x7f398c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:32.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3992148700 1 -- 172.21.15.175:0/211936096 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b1fc0 msgr2=0x7f398c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:32.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3992148700 1 --2- 172.21.15.175:0/211936096 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b1fc0 0x7f398c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3992148700 1 -- 172.21.15.175:0/211936096 >> v1:172.21.15.175:6789/0 conn(0x7f398c0b2390 legacy=0x7f398c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:32.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3992148700 1 -- 172.21.15.175:0/211936096 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c0af4c0 con 0x7f398c0aff60 2023-10-10T18:59:32.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3990945700 1 -- 172.21.15.175:0/211936096 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834480087 0 0) 0x7f398c1574b0 con 0x7f398c0b2390 2023-10-10T18:59:32.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3992148700 1 --2- 172.21.15.175:0/211936096 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0aff60 0x7f398c0b03e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3984009e90 tx=0x7f3984009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4298fa3f88481fc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:32.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.496+0000 7f3990945700 1 -- 172.21.15.175:0/211936096 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398400e070 con 0x7f398c0aff60 2023-10-10T18:59:32.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3990945700 1 -- 172.21.15.175:0/211936096 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398400bc50 con 0x7f398c0aff60 2023-10-10T18:59:32.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3990945700 1 -- 172.21.15.175:0/211936096 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3984012430 con 0x7f398c0aff60 2023-10-10T18:59:32.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3993bab700 1 -- 172.21.15.175:0/211936096 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0aff60 msgr2=0x7f398c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:32.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3993bab700 1 --2- 172.21.15.175:0/211936096 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0aff60 0x7f398c0b03e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3984009e90 tx=0x7f3984009780 comp rx=0 tx=0).stop 2023-10-10T18:59:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3993bab700 1 -- 172.21.15.175:0/211936096 shutdown_connections 2023-10-10T18:59:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3993bab700 1 --2- 172.21.15.175:0/211936096 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0aff60 0x7f398c0b03e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3993bab700 1 --2- 172.21.15.175:0/211936096 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b1fc0 0x7f398c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3993bab700 1 -- 172.21.15.175:0/211936096 >> 172.21.15.175:0/211936096 conn(0x7f398c063230 msgr2=0x7f398c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3993bab700 1 -- 172.21.15.175:0/211936096 shutdown_connections 2023-10-10T18:59:32.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.497+0000 7f3993bab700 1 -- 172.21.15.175:0/211936096 wait complete. 2023-10-10T18:59:32.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.498+0000 7f3993bab700 1 Processor -- start 2023-10-10T18:59:32.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.498+0000 7f3993bab700 1 -- start start 2023-10-10T18:59:32.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3993bab700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f398c0aff60 0x7f398c1696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:32.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3993bab700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0b1fc0 0x7f398c150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:32.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3993bab700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b2390 0x7f398c0cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:32.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3993bab700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c0c6390 con 0x7f398c0b1fc0 2023-10-10T18:59:32.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3993bab700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c0ca5a0 con 0x7f398c0aff60 2023-10-10T18:59:32.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3993bab700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398c0ca840 con 0x7f398c0b2390 2023-10-10T18:59:32.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991146700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0b1fc0 0x7f398c150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:32.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991146700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0b1fc0 0x7f398c150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59620/0 (socket says 172.21.15.175:59620) 2023-10-10T18:59:32.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991146700 1 -- 172.21.15.175:0/1222359633 learned_addr learned my addr 172.21.15.175:0/1222359633 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:32.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991947700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f398c0aff60 0x7f398c1696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:32.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3992148700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b2390 0x7f398c0cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:32.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991146700 1 -- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b2390 msgr2=0x7f398c0cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:32.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991146700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b2390 0x7f398c0cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991146700 1 -- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f398c0aff60 msgr2=0x7f398c1696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:32.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991146700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f398c0aff60 0x7f398c1696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.499+0000 7f3991146700 1 -- 172.21.15.175:0/1222359633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3984009a20 con 0x7f398c0b1fc0 2023-10-10T18:59:32.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.500+0000 7f3991146700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0b1fc0 0x7f398c150fa0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f397c00c5b0 tx=0x7f397c001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:32.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.500+0000 7f3981ffb700 1 -- 172.21.15.175:0/1222359633 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c005b30 con 0x7f398c0b1fc0 2023-10-10T18:59:32.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.500+0000 7f3981ffb700 1 -- 172.21.15.175:0/1222359633 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397c005c90 con 0x7f398c0b1fc0 2023-10-10T18:59:32.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.500+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398c0cab00 con 0x7f398c0b1fc0 2023-10-10T18:59:32.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.500+0000 7f3981ffb700 1 -- 172.21.15.175:0/1222359633 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c007320 con 0x7f398c0b1fc0 2023-10-10T18:59:32.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.500+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398c0c68c0 con 0x7f398c0b1fc0 2023-10-10T18:59:32.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.501+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398c0cade0 con 0x7f398c0b1fc0 2023-10-10T18:59:32.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.501+0000 7f3981ffb700 1 -- 172.21.15.175:0/1222359633 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f397c01b3b0 con 0x7f398c0b1fc0 2023-10-10T18:59:32.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.502+0000 7f3981ffb700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3970066f60 0x7f3970069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:32.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.502+0000 7f3981ffb700 1 -- 172.21.15.175:0/1222359633 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f397c083b90 con 0x7f398c0b1fc0 2023-10-10T18:59:32.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.502+0000 7f3991947700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3970066f60 0x7f3970069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:32.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.502+0000 7f3991947700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3970066f60 0x7f3970069410 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3988006fd0 tx=0x7f3988008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:32.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.506+0000 7f3981ffb700 1 -- 172.21.15.175:0/1222359633 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f397c050b30 con 0x7f398c0b1fc0 2023-10-10T18:59:32.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.736+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f398c067570 con 0x7f398c0b1fc0 2023-10-10T18:59:32.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.736+0000 7f3981ffb700 1 -- 172.21.15.175:0/1222359633 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5439 (secure 0 0 0) 0x7f397c017090 con 0x7f398c0b1fc0 2023-10-10T18:59:32.738 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:32.738 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:32.639159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:32.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.738+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3970066f60 msgr2=0x7f3970069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:32.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.738+0000 7f3993bab700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3970066f60 0x7f3970069410 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3988006fd0 tx=0x7f3988008040 comp rx=0 tx=0).stop 2023-10-10T18:59:32.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.738+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0b1fc0 msgr2=0x7f398c150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:32.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.738+0000 7f3993bab700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0b1fc0 0x7f398c150fa0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f397c00c5b0 tx=0x7f397c001070 comp rx=0 tx=0).stop 2023-10-10T18:59:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.739+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 shutdown_connections 2023-10-10T18:59:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.739+0000 7f3993bab700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3970066f60 0x7f3970069410 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.739+0000 7f3993bab700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f398c0b2390 0x7f398c0cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.739+0000 7f3993bab700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f398c0aff60 0x7f398c1696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.739+0000 7f3993bab700 1 --2- 172.21.15.175:0/1222359633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f398c0b1fc0 0x7f398c150fa0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.739+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 >> 172.21.15.175:0/1222359633 conn(0x7f398c063230 msgr2=0x7f398c153dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.739+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 shutdown_connections 2023-10-10T18:59:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:32.739+0000 7f3993bab700 1 -- 172.21.15.175:0/1222359633 wait complete. 2023-10-10T18:59:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 104 2023-10-10T18:59:32.752 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 104, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 104, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5196": { "addr": "172.21.15.175:6834/3874110823", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3874110823, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3874110823, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5196, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 46 }, "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 31 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:clientreplay", "state_seq": 33 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T18:59:32.639159+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 5332, "mds_2": 5367, "mds_3": 5196, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T18:59:34.753 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:34.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.880+0000 7efc0fe92700 1 Processor -- start 2023-10-10T18:59:34.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.880+0000 7efc0fe92700 1 -- start start 2023-10-10T18:59:34.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0fe92700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7efc08155fb0 0x7efc08159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:34.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0fe92700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc081574b0 con 0x7efc080b2390 2023-10-10T18:59:34.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0fe92700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc080afad0 con 0x7efc080b1fc0 2023-10-10T18:59:34.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0fe92700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc08150770 con 0x7efc08155fb0 2023-10-10T18:59:34.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0e42f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7efc08155fb0 0x7efc08159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:34.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0d42d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7efc080b2390 0x7efc080aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35180/0 (socket says 172.21.15.175:35180) 2023-10-10T18:59:34.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0d42d700 1 -- 172.21.15.175:0/577746573 learned_addr learned my addr 172.21.15.175:0/577746573 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:34.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0e42f700 1 -- 172.21.15.175:0/577746573 >> v1:172.21.15.191:6789/0 conn(0x7efc080b1fc0 legacy=0x7efc08068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:34.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0e42f700 1 -- 172.21.15.175:0/577746573 >> v1:172.21.15.175:6789/0 conn(0x7efc080b2390 legacy=0x7efc080aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:34.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.881+0000 7efc0e42f700 1 -- 172.21.15.175:0/577746573 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080af4c0 con 0x7efc08155fb0 2023-10-10T18:59:34.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0e42f700 1 --2- 172.21.15.175:0/577746573 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08159920 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7efc000098e0 tx=0x7efc000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73245b53e34020a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:34.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0cc2c700 1 -- 172.21.15.175:0/577746573 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00009e80 con 0x7efc08155fb0 2023-10-10T18:59:34.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0cc2c700 1 -- 172.21.15.175:0/577746573 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0000b8f0 con 0x7efc08155fb0 2023-10-10T18:59:34.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0cc2c700 1 -- 172.21.15.175:0/577746573 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0000bc70 con 0x7efc08155fb0 2023-10-10T18:59:34.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0fe92700 1 -- 172.21.15.175:0/577746573 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 msgr2=0x7efc08159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:34.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0fe92700 1 --2- 172.21.15.175:0/577746573 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08159920 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7efc000098e0 tx=0x7efc000091a0 comp rx=0 tx=0).stop 2023-10-10T18:59:34.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0fe92700 1 -- 172.21.15.175:0/577746573 shutdown_connections 2023-10-10T18:59:34.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0fe92700 1 --2- 172.21.15.175:0/577746573 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08159920 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:34.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0fe92700 1 -- 172.21.15.175:0/577746573 >> 172.21.15.175:0/577746573 conn(0x7efc08063230 msgr2=0x7efc08063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:34.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0fe92700 1 -- 172.21.15.175:0/577746573 shutdown_connections 2023-10-10T18:59:34.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.882+0000 7efc0fe92700 1 -- 172.21.15.175:0/577746573 wait complete. 2023-10-10T18:59:34.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.883+0000 7efc0fe92700 1 Processor -- start 2023-10-10T18:59:34.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.883+0000 7efc0fe92700 1 -- start start 2023-10-10T18:59:34.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0fe92700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc080b1fc0 0x7efc080ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:34.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0fe92700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc080b2390 0x7efc081518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:34.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0fe92700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08166390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:34.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0fe92700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08165f00 con 0x7efc08155fb0 2023-10-10T18:59:34.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0fe92700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08150a50 con 0x7efc080b2390 2023-10-10T18:59:34.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0fe92700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc08165bb0 con 0x7efc080b1fc0 2023-10-10T18:59:34.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0e42f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08166390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:34.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0e42f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08166390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38158/0 (socket says 172.21.15.175:38158) 2023-10-10T18:59:34.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0e42f700 1 -- 172.21.15.175:0/3566357616 learned_addr learned my addr 172.21.15.175:0/3566357616 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:34.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0d42d700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc080b2390 0x7efc081518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:34.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.884+0000 7efc0dc2e700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc080b1fc0 0x7efc080ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:34.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0e42f700 1 -- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc080b1fc0 msgr2=0x7efc080ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:34.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0e42f700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc080b1fc0 0x7efc080ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:34.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0e42f700 1 -- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc080b2390 msgr2=0x7efc081518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:34.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0e42f700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc080b2390 0x7efc081518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:34.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0e42f700 1 -- 172.21.15.175:0/3566357616 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00009420 con 0x7efc08155fb0 2023-10-10T18:59:34.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0dc2e700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc080b1fc0 0x7efc080ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:34.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0e42f700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08166390 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7efc0806fd90 tx=0x7efc00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:34.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efbfdffb700 1 -- 172.21.15.175:0/3566357616 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0001a630 con 0x7efc08155fb0 2023-10-10T18:59:34.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efbfdffb700 1 -- 172.21.15.175:0/3566357616 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0001a790 con 0x7efc08155fb0 2023-10-10T18:59:34.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc08150cf0 con 0x7efc08155fb0 2023-10-10T18:59:34.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efbfdffb700 1 -- 172.21.15.175:0/3566357616 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0001ab50 con 0x7efc08155fb0 2023-10-10T18:59:34.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.885+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc081512b0 con 0x7efc08155fb0 2023-10-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.886+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc08166070 con 0x7efc08155fb0 2023-10-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.887+0000 7efbfdffb700 1 -- 172.21.15.175:0/3566357616 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efc00023410 con 0x7efc08155fb0 2023-10-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.887+0000 7efbfdffb700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efbec066f30 0x7efbec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.887+0000 7efbfdffb700 1 -- 172.21.15.175:0/3566357616 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7efc00086210 con 0x7efc08155fb0 2023-10-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.887+0000 7efc0dc2e700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efbec066f30 0x7efbec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.888+0000 7efc0dc2e700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efbec066f30 0x7efbec0693e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc04007fd0 tx=0x7efc04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:34.891+0000 7efbfdffb700 1 -- 172.21.15.175:0/3566357616 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc000531b0 con 0x7efc08155fb0 2023-10-10T18:59:35.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.115+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc08067570 con 0x7efc08155fb0 2023-10-10T18:59:35.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.116+0000 7efbfdffb700 1 -- 172.21.15.175:0/3566357616 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5439 (secure 0 0 0) 0x7efc000567d0 con 0x7efc08155fb0 2023-10-10T18:59:35.117 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:35.118 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:32.639159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:35.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efbec066f30 msgr2=0x7efbec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:35.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efbec066f30 0x7efbec0693e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc04007fd0 tx=0x7efc04009040 comp rx=0 tx=0).stop 2023-10-10T18:59:35.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 msgr2=0x7efc08166390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:35.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08166390 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7efc0806fd90 tx=0x7efc00017000 comp rx=0 tx=0).stop 2023-10-10T18:59:35.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 shutdown_connections 2023-10-10T18:59:35.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efbec066f30 0x7efbec0693e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc08155fb0 0x7efc08166390 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc080b1fc0 0x7efc080ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 --2- 172.21.15.175:0/3566357616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc080b2390 0x7efc081518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 >> 172.21.15.175:0/3566357616 conn(0x7efc08063230 msgr2=0x7efc080c8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 shutdown_connections 2023-10-10T18:59:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:35.118+0000 7efc0fe92700 1 -- 172.21.15.175:0/3566357616 wait complete. 2023-10-10T18:59:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 104 2023-10-10T18:59:35.131 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:37.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:37.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.259+0000 7fb79ffff700 1 Processor -- start 2023-10-10T18:59:37.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.259+0000 7fb79ffff700 1 -- start start 2023-10-10T18:59:37.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb7a014fd90 0x7fb7a0159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:37.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:37.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79ffff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a0067fa0 con 0x7fb7a00cd040 2023-10-10T18:59:37.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79ffff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a00c5600 con 0x7fb7a014fd90 2023-10-10T18:59:37.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79ffff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a0156b70 con 0x7fb7a008a100 2023-10-10T18:59:37.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79effd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb7a014fd90 0x7fb7a0159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:37.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79e7fc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:37.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79e7fc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:43506/0 (socket says 172.21.15.175:43506) 2023-10-10T18:59:37.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79effd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb7a014fd90 0x7fb7a0159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:53034/0 (socket says 172.21.15.175:53034) 2023-10-10T18:59:37.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79e7fc700 1 -- 172.21.15.175:0/3716279161 learned_addr learned my addr 172.21.15.175:0/3716279161 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:37.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.260+0000 7fb79f7fe700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb7a00cd040 0x7fb7a00684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:41338/0 (socket says 172.21.15.175:41338) 2023-10-10T18:59:37.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79effd700 1 -- 172.21.15.175:0/3716279161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 msgr2=0x7fb7a00c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:37.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79effd700 1 --2- 172.21.15.175:0/3716279161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79effd700 1 -- 172.21.15.175:0/3716279161 >> v1:172.21.15.191:6789/0 conn(0x7fb7a00cd040 legacy=0x7fb7a00684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:37.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79effd700 1 -- 172.21.15.175:0/3716279161 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a0150160 con 0x7fb7a014fd90 2023-10-10T18:59:37.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79dffb700 1 -- 172.21.15.175:0/3716279161 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053236779 0 0) 0x7fb7a0067fa0 con 0x7fb7a00cd040 2023-10-10T18:59:37.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79e7fc700 1 --2- 172.21.15.175:0/3716279161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T18:59:37.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79effd700 1 --2- 172.21.15.175:0/3716279161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 0x7fb7a0159c30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb790009e60 tx=0x7fb790009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a34fe60c97d399d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:37.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79dffb700 1 -- 172.21.15.175:0/3716279161 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79000e070 con 0x7fb7a014fd90 2023-10-10T18:59:37.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79dffb700 1 -- 172.21.15.175:0/3716279161 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79000bb60 con 0x7fb7a014fd90 2023-10-10T18:59:37.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.261+0000 7fb79dffb700 1 -- 172.21.15.175:0/3716279161 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790012430 con 0x7fb7a014fd90 2023-10-10T18:59:37.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.262+0000 7fb79ffff700 1 -- 172.21.15.175:0/3716279161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 msgr2=0x7fb7a0159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:37.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.262+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3716279161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 0x7fb7a0159c30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb790009e60 tx=0x7fb790009780 comp rx=0 tx=0).stop 2023-10-10T18:59:37.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.262+0000 7fb79ffff700 1 -- 172.21.15.175:0/3716279161 shutdown_connections 2023-10-10T18:59:37.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.262+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3716279161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 0x7fb7a0159c30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.262+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3716279161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.262+0000 7fb79ffff700 1 -- 172.21.15.175:0/3716279161 >> 172.21.15.175:0/3716279161 conn(0x7fb7a0063d00 msgr2=0x7fb7a00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:37.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.262+0000 7fb79ffff700 1 -- 172.21.15.175:0/3716279161 shutdown_connections 2023-10-10T18:59:37.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.262+0000 7fb79ffff700 1 -- 172.21.15.175:0/3716279161 wait complete. 2023-10-10T18:59:37.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.263+0000 7fb79ffff700 1 Processor -- start 2023-10-10T18:59:37.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.263+0000 7fb79ffff700 1 -- start start 2023-10-10T18:59:37.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:37.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a00cd040 0x7fb7a00c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:37.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79ffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 0x7fb7a00cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:37.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79ffff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a0152b00 con 0x7fb7a00cd040 2023-10-10T18:59:37.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79ffff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00c8a10 con 0x7fb7a014fd90 2023-10-10T18:59:37.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79ffff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00c7370 con 0x7fb7a008a100 2023-10-10T18:59:37.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a00cd040 0x7fb7a00c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:37.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a00cd040 0x7fb7a00c7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38170/0 (socket says 172.21.15.175:38170) 2023-10-10T18:59:37.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 -- 172.21.15.175:0/3322751615 learned_addr learned my addr 172.21.15.175:0/3322751615 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:37.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79effd700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:37.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79f7fe700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 0x7fb7a00cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:37.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 -- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 msgr2=0x7fb7a00c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:37.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 -- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 msgr2=0x7fb7a00cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:37.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 0x7fb7a00cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 -- 172.21.15.175:0/3322751615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7940094f0 con 0x7fb7a00cd040 2023-10-10T18:59:37.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79e7fc700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a00cd040 0x7fb7a00c7800 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb79400ec50 tx=0x7fb794009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:37.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb786ffd700 1 -- 172.21.15.175:0/3322751615 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79400ca60 con 0x7fb7a00cd040 2023-10-10T18:59:37.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb786ffd700 1 -- 172.21.15.175:0/3322751615 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79400cbc0 con 0x7fb7a00cd040 2023-10-10T18:59:37.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb790009a20 con 0x7fb7a00cd040 2023-10-10T18:59:37.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb786ffd700 1 -- 172.21.15.175:0/3322751615 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794013470 con 0x7fb7a00cd040 2023-10-10T18:59:37.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.264+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a00c8c60 con 0x7fb7a00cd040 2023-10-10T18:59:37.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.266+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a00c8fa0 con 0x7fb7a00cd040 2023-10-10T18:59:37.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.266+0000 7fb786ffd700 1 -- 172.21.15.175:0/3322751615 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb794013820 con 0x7fb7a00cd040 2023-10-10T18:59:37.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.266+0000 7fb786ffd700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb78c066f30 0x7fb78c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:37.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.266+0000 7fb786ffd700 1 -- 172.21.15.175:0/3322751615 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fb794084c20 con 0x7fb7a00cd040 2023-10-10T18:59:37.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.266+0000 7fb79effd700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb78c066f30 0x7fb78c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:37.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.267+0000 7fb79effd700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb78c066f30 0x7fb78c0693e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb7900097c0 tx=0x7fb790019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:37.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.271+0000 7fb786ffd700 1 -- 172.21.15.175:0/3322751615 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb794051bc0 con 0x7fb7a00cd040 2023-10-10T18:59:37.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.495+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7a0071210 con 0x7fb7a00cd040 2023-10-10T18:59:37.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.496+0000 7fb786ffd700 1 -- 172.21.15.175:0/3322751615 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5439 (secure 0 0 0) 0x7fb7940551e0 con 0x7fb7a00cd040 2023-10-10T18:59:37.497 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:37.498 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:32.639159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb78c066f30 msgr2=0x7fb78c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:37.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb78c066f30 0x7fb78c0693e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb7900097c0 tx=0x7fb790019000 comp rx=0 tx=0).stop 2023-10-10T18:59:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a00cd040 msgr2=0x7fb7a00c7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a00cd040 0x7fb7a00c7800 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb79400ec50 tx=0x7fb794009250 comp rx=0 tx=0).stop 2023-10-10T18:59:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 shutdown_connections 2023-10-10T18:59:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb78c066f30 0x7fb78c0693e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7a014fd90 0x7fb7a00cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7a008a100 0x7fb7a00c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 --2- 172.21.15.175:0/3322751615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7a00cd040 0x7fb7a00c7800 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 >> 172.21.15.175:0/3322751615 conn(0x7fb7a0063d00 msgr2=0x7fb7a00cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 shutdown_connections 2023-10-10T18:59:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:37.498+0000 7fb79ffff700 1 -- 172.21.15.175:0/3322751615 wait complete. 2023-10-10T18:59:37.502 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 104 2023-10-10T18:59:37.511 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T18:59:39.512 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:39.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.639+0000 7f007f574700 1 Processor -- start 2023-10-10T18:59:39.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.639+0000 7f007f574700 1 -- start start 2023-10-10T18:59:39.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007f574700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00780c5850 0x7f00780c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:39.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007f574700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f00780c12d0 0x7f0078156660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:39.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007f574700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 0x7f00780c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:39.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007f574700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0078063ea0 con 0x7f00780c5850 2023-10-10T18:59:39.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007f574700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0078155f30 con 0x7f00780c12d0 2023-10-10T18:59:39.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007f574700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00780c6fb0 con 0x7f0078156ba0 2023-10-10T18:59:39.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007d310700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00780c5850 0x7f00780c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:39.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007d310700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00780c5850 0x7f00780c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38178/0 (socket says 172.21.15.175:38178) 2023-10-10T18:59:39.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.640+0000 7f007d310700 1 -- 172.21.15.175:0/699732569 learned_addr learned my addr 172.21.15.175:0/699732569 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:39.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007db11700 1 --2- 172.21.15.175:0/699732569 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 0x7f00780c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:39.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007cb0f700 1 --2- 172.21.15.175:0/699732569 >> v2:172.21.15.191:3300/0 conn(0x7f00780c12d0 0x7f0078156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:39.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007d310700 1 -- 172.21.15.175:0/699732569 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 msgr2=0x7f00780c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:39.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007d310700 1 --2- 172.21.15.175:0/699732569 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 0x7f00780c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007d310700 1 -- 172.21.15.175:0/699732569 >> v2:172.21.15.191:3300/0 conn(0x7f00780c12d0 msgr2=0x7f0078156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:39.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007d310700 1 --2- 172.21.15.175:0/699732569 >> v2:172.21.15.191:3300/0 conn(0x7f00780c12d0 0x7f0078156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007d310700 1 -- 172.21.15.175:0/699732569 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00780c6160 con 0x7f00780c5850 2023-10-10T18:59:39.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007cb0f700 1 --2- 172.21.15.175:0/699732569 >> v2:172.21.15.191:3300/0 conn(0x7f00780c12d0 0x7f0078156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:39.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007d310700 1 --2- 172.21.15.175:0/699732569 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c5850 0x7f00780c5c20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f007806f1f0 tx=0x7f0068009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eaef7a7c74fe5df server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:39.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f0077fff700 1 -- 172.21.15.175:0/699732569 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006800e070 con 0x7f00780c5850 2023-10-10T18:59:39.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f0077fff700 1 -- 172.21.15.175:0/699732569 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006800ba20 con 0x7f00780c5850 2023-10-10T18:59:39.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f0077fff700 1 -- 172.21.15.175:0/699732569 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00680092e0 con 0x7f00780c5850 2023-10-10T18:59:39.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.641+0000 7f007f574700 1 -- 172.21.15.175:0/699732569 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c5850 msgr2=0x7f00780c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:39.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.642+0000 7f007f574700 1 --2- 172.21.15.175:0/699732569 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c5850 0x7f00780c5c20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f007806f1f0 tx=0x7f0068009840 comp rx=0 tx=0).stop 2023-10-10T18:59:39.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.642+0000 7f007f574700 1 -- 172.21.15.175:0/699732569 shutdown_connections 2023-10-10T18:59:39.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.642+0000 7f007f574700 1 --2- 172.21.15.175:0/699732569 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 0x7f00780c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.642+0000 7f007f574700 1 --2- 172.21.15.175:0/699732569 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c5850 0x7f00780c5c20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.642+0000 7f007f574700 1 --2- 172.21.15.175:0/699732569 >> v2:172.21.15.191:3300/0 conn(0x7f00780c12d0 0x7f0078156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.642+0000 7f007f574700 1 -- 172.21.15.175:0/699732569 >> 172.21.15.175:0/699732569 conn(0x7f007806dc70 msgr2=0x7f0078063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:39.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.642+0000 7f007f574700 1 -- 172.21.15.175:0/699732569 shutdown_connections 2023-10-10T18:59:39.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.642+0000 7f007f574700 1 -- 172.21.15.175:0/699732569 wait complete. 2023-10-10T18:59:39.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.643+0000 7f007f574700 1 Processor -- start 2023-10-10T18:59:39.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.643+0000 7f007f574700 1 -- start start 2023-10-10T18:59:39.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007f574700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c12d0 0x7f0078166680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:39.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007f574700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00780c5850 0x7f0078157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:39.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007f574700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 0x7f00780c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:39.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007f574700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007815a2b0 con 0x7f00780c12d0 2023-10-10T18:59:39.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007f574700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078157890 con 0x7f00780c5850 2023-10-10T18:59:39.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007f574700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0078158260 con 0x7f0078156ba0 2023-10-10T18:59:39.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c12d0 0x7f0078166680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:39.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c12d0 0x7f0078166680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38180/0 (socket says 172.21.15.175:38180) 2023-10-10T18:59:39.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 -- 172.21.15.175:0/3636871150 learned_addr learned my addr 172.21.15.175:0/3636871150 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:39.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007db11700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 0x7f00780c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:39.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007cb0f700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00780c5850 0x7f0078157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:39.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 -- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 msgr2=0x7f00780c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:39.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 0x7f00780c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 -- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00780c5850 msgr2=0x7f0078157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:39.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00780c5850 0x7f0078157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 -- 172.21.15.175:0/3636871150 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0068009ae0 con 0x7f00780c12d0 2023-10-10T18:59:39.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007cb0f700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00780c5850 0x7f0078157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T18:59:39.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007d310700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c12d0 0x7f0078166680 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f006800b610 tx=0x7f0068018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:39.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f00757fa700 1 -- 172.21.15.175:0/3636871150 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068008c30 con 0x7f00780c12d0 2023-10-10T18:59:39.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f00757fa700 1 -- 172.21.15.175:0/3636871150 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0068008d90 con 0x7f00780c12d0 2023-10-10T18:59:39.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00780c8c00 con 0x7f00780c12d0 2023-10-10T18:59:39.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f00757fa700 1 -- 172.21.15.175:0/3636871150 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006801c660 con 0x7f00780c12d0 2023-10-10T18:59:39.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.644+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0078159fd0 con 0x7f00780c12d0 2023-10-10T18:59:39.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.645+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0078067570 con 0x7f00780c12d0 2023-10-10T18:59:39.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.646+0000 7f00757fa700 1 -- 172.21.15.175:0/3636871150 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0068003040 con 0x7f00780c12d0 2023-10-10T18:59:39.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.646+0000 7f00757fa700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0064066f30 0x7f00640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:39.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.646+0000 7f00757fa700 1 -- 172.21.15.175:0/3636871150 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f0068085ff0 con 0x7f00780c12d0 2023-10-10T18:59:39.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.646+0000 7f007cb0f700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0064066f30 0x7f00640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:39.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.647+0000 7f007cb0f700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0064066f30 0x7f00640693e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f006c006f00 tx=0x7f006c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:39.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.650+0000 7f00757fa700 1 -- 172.21.15.175:0/3636871150 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0068052f90 con 0x7f00780c12d0 2023-10-10T18:59:39.731 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T18:59:39.730+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-10T18:59:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.878+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00780c6a80 con 0x7f00780c12d0 2023-10-10T18:59:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.879+0000 7f00757fa700 1 -- 172.21.15.175:0/3636871150 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5433 (secure 0 0 0) 0x7f00680565b0 con 0x7f00780c12d0 2023-10-10T18:59:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:39.881 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:38.659091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0064066f30 msgr2=0x7f00640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0064066f30 0x7f00640693e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f006c006f00 tx=0x7f006c008040 comp rx=0 tx=0).stop 2023-10-10T18:59:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c12d0 msgr2=0x7f0078166680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c12d0 0x7f0078166680 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f006800b610 tx=0x7f0068018000 comp rx=0 tx=0).stop 2023-10-10T18:59:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 shutdown_connections 2023-10-10T18:59:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0064066f30 0x7f00640693e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0078156ba0 0x7f00780c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00780c12d0 0x7f0078166680 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 --2- 172.21.15.175:0/3636871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00780c5850 0x7f0078157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 >> 172.21.15.175:0/3636871150 conn(0x7f007806dc70 msgr2=0x7f0078153210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 shutdown_connections 2023-10-10T18:59:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:39.881+0000 7f007f574700 1 -- 172.21.15.175:0/3636871150 wait complete. 2023-10-10T18:59:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 105 2023-10-10T18:59:39.894 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=4, replacing gid=4760 2023-10-10T18:59:39.894 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2023-10-10T18:59:41.895 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-10T18:59:41.895 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-10T18:59:41.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-10T18:59:41.898 INFO:tasks.ceph.mds.b:Started 2023-10-10T18:59:41.898 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:41.978 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-10T18:59:42.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.020+0000 7f900b67e700 1 Processor -- start 2023-10-10T18:59:42.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.020+0000 7f900b67e700 1 -- start start 2023-10-10T18:59:42.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.021+0000 7f900b67e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9004150480 0x7f9004161940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:42.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.021+0000 7f900b67e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040677f0 con 0x7f90041500b0 2023-10-10T18:59:42.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.021+0000 7f900b67e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004159590 con 0x7f900406d6e0 2023-10-10T18:59:42.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.021+0000 7f900b67e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90040c4250 con 0x7f9004150480 2023-10-10T18:59:42.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.021+0000 7f900941a700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f90041500b0 0x7f90040c46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35192/0 (socket says 172.21.15.175:35192) 2023-10-10T18:59:42.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.021+0000 7f900941a700 1 -- 172.21.15.175:0/2884889046 learned_addr learned my addr 172.21.15.175:0/2884889046 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:42.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.021+0000 7f9009c1b700 1 --2- 172.21.15.175:0/2884889046 >> v2:172.21.15.191:3300/0 conn(0x7f9004150480 0x7f9004161940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:42.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f8ffbfff700 1 -- 172.21.15.175:0/2884889046 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424788613 0 0) 0x7f90040677f0 con 0x7f90041500b0 2023-10-10T18:59:42.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f8ffbfff700 1 -- 172.21.15.175:0/2884889046 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0003570 con 0x7f90041500b0 2023-10-10T18:59:42.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f9009c1b700 1 -- 172.21.15.175:0/2884889046 >> v1:172.21.15.191:6789/0 conn(0x7f900406d6e0 legacy=0x7f900406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:42.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f8ffbfff700 1 -- 172.21.15.175:0/2884889046 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396740781 0 0) 0x7f9004159590 con 0x7f900406d6e0 2023-10-10T18:59:42.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f9009c1b700 1 -- 172.21.15.175:0/2884889046 >> v1:172.21.15.175:6789/0 conn(0x7f90041500b0 legacy=0x7f90040c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:42.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f9009c1b700 1 -- 172.21.15.175:0/2884889046 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9004063760 con 0x7f9004150480 2023-10-10T18:59:42.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f9009c1b700 1 --2- 172.21.15.175:0/2884889046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9004150480 0x7f9004161940 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009e60 tx=0x7f8ffc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1619737ce974b174 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:42.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f8ffbfff700 1 -- 172.21.15.175:0/2884889046 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00e070 con 0x7f9004150480 2023-10-10T18:59:42.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f8ffbfff700 1 -- 172.21.15.175:0/2884889046 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc00bbf0 con 0x7f9004150480 2023-10-10T18:59:42.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.022+0000 7f8ffbfff700 1 -- 172.21.15.175:0/2884889046 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc012430 con 0x7f9004150480 2023-10-10T18:59:42.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.023+0000 7f900b67e700 1 -- 172.21.15.175:0/2884889046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9004150480 msgr2=0x7f9004161940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:42.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.023+0000 7f900b67e700 1 --2- 172.21.15.175:0/2884889046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9004150480 0x7f9004161940 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009e60 tx=0x7f8ffc009780 comp rx=0 tx=0).stop 2023-10-10T18:59:42.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.023+0000 7f900b67e700 1 -- 172.21.15.175:0/2884889046 shutdown_connections 2023-10-10T18:59:42.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.023+0000 7f900b67e700 1 --2- 172.21.15.175:0/2884889046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9004150480 0x7f9004161940 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:42.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.023+0000 7f900b67e700 1 -- 172.21.15.175:0/2884889046 >> 172.21.15.175:0/2884889046 conn(0x7f90040c5650 msgr2=0x7f90040c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:42.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.023+0000 7f900b67e700 1 -- 172.21.15.175:0/2884889046 shutdown_connections 2023-10-10T18:59:42.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.023+0000 7f900b67e700 1 -- 172.21.15.175:0/2884889046 wait complete. 2023-10-10T18:59:42.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.024+0000 7f900b67e700 1 Processor -- start 2023-10-10T18:59:42.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.024+0000 7f900b67e700 1 -- start start 2023-10-10T18:59:42.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900b67e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900406d6e0 0x7f900414ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:42.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900b67e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90041500b0 0x7f900415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:42.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900b67e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9004150480 0x7f900415b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:42.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900b67e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040bc3c0 con 0x7f900406d6e0 2023-10-10T18:59:42.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900b67e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900415bc60 con 0x7f90041500b0 2023-10-10T18:59:42.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900b67e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90040bc190 con 0x7f9004150480 2023-10-10T18:59:42.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900406d6e0 0x7f900414ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:42.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900406d6e0 0x7f900414ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38194/0 (socket says 172.21.15.175:38194) 2023-10-10T18:59:42.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 -- 172.21.15.175:0/3888050892 learned_addr learned my addr 172.21.15.175:0/3888050892 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:42.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f9008c19700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90041500b0 0x7f900415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:42.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f9009c1b700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9004150480 0x7f900415b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:42.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 -- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9004150480 msgr2=0x7f900415b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:42.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9004150480 0x7f900415b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:42.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 -- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90041500b0 msgr2=0x7f900415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:42.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90041500b0 0x7f900415b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:42.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 -- 172.21.15.175:0/3888050892 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc009a20 con 0x7f900406d6e0 2023-10-10T18:59:42.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900941a700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900406d6e0 0x7f900414ced0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f900000c790 tx=0x7f9000007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:42.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f8ff97fa700 1 -- 172.21.15.175:0/3888050892 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000005b30 con 0x7f900406d6e0 2023-10-10T18:59:42.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90040c04b0 con 0x7f900406d6e0 2023-10-10T18:59:42.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f8ff97fa700 1 -- 172.21.15.175:0/3888050892 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9000005c90 con 0x7f900406d6e0 2023-10-10T18:59:42.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f8ff97fa700 1 -- 172.21.15.175:0/3888050892 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000007320 con 0x7f900406d6e0 2023-10-10T18:59:42.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.025+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90040bfef0 con 0x7f900406d6e0 2023-10-10T18:59:42.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.026+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004067570 con 0x7f900406d6e0 2023-10-10T18:59:42.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.027+0000 7f8ff97fa700 1 -- 172.21.15.175:0/3888050892 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9000007690 con 0x7f900406d6e0 2023-10-10T18:59:42.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.027+0000 7f8ff97fa700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f60 0x7f8fe8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:42.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.027+0000 7f8ff97fa700 1 -- 172.21.15.175:0/3888050892 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f90000843c0 con 0x7f900406d6e0 2023-10-10T18:59:42.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.027+0000 7f9008c19700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f60 0x7f8fe8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:42.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.028+0000 7f9008c19700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f60 0x7f8fe8069410 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f90040c4190 tx=0x7f8ff4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:42.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.031+0000 7f8ff97fa700 1 -- 172.21.15.175:0/3888050892 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9000054df0 con 0x7f900406d6e0 2023-10-10T18:59:42.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.254+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9004069c40 con 0x7f900406d6e0 2023-10-10T18:59:42.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.255+0000 7f8ff97fa700 1 -- 172.21.15.175:0/3888050892 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5433 (secure 0 0 0) 0x7f9000054980 con 0x7f900406d6e0 2023-10-10T18:59:42.256 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:42.256 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:38.659091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:42.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f60 msgr2=0x7f8fe8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:42.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f60 0x7f8fe8069410 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f90040c4190 tx=0x7f8ff4009040 comp rx=0 tx=0).stop 2023-10-10T18:59:42.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900406d6e0 msgr2=0x7f900414ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:42.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900406d6e0 0x7f900414ced0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f900000c790 tx=0x7f9000007df0 comp rx=0 tx=0).stop 2023-10-10T18:59:42.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 shutdown_connections 2023-10-10T18:59:42.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f60 0x7f8fe8069410 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:42.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9004150480 0x7f900415b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:42.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900406d6e0 0x7f900414ced0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:42.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 --2- 172.21.15.175:0/3888050892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90041500b0 0x7f900415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:42.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 >> 172.21.15.175:0/3888050892 conn(0x7f90040c5650 msgr2=0x7f90040c69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:42.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 shutdown_connections 2023-10-10T18:59:42.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:42.257+0000 7f900b67e700 1 -- 172.21.15.175:0/3888050892 wait complete. 2023-10-10T18:59:42.261 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 105 2023-10-10T18:59:42.270 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-10T18:59:44.271 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T18:59:44.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.396+0000 7f6923841700 1 Processor -- start 2023-10-10T18:59:44.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.396+0000 7f6923841700 1 -- start start 2023-10-10T18:59:44.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.397+0000 7f6923841700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f691c0c1be0 0x7f691c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.397+0000 7f6923841700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c14ddb0 0x7f691c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.397+0000 7f6923841700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c154e20 con 0x7f691c0c1fb0 2023-10-10T18:59:44.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.397+0000 7f6923841700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c06bca0 con 0x7f691c0c1be0 2023-10-10T18:59:44.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.397+0000 7f6923841700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c154000 con 0x7f691c14ddb0 2023-10-10T18:59:44.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.397+0000 7f6920ddc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f691c0c1fb0 0x7f691c06a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35200/0 (socket says 172.21.15.175:35200) 2023-10-10T18:59:44.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.397+0000 7f6920ddc700 1 -- 172.21.15.175:0/1370043232 learned_addr learned my addr 172.21.15.175:0/1370043232 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T18:59:44.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6921dde700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c14ddb0 0x7f691c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f69215dd700 1 --2- 172.21.15.175:0/1370043232 >> v2:172.21.15.191:3300/0 conn(0x7f691c0c1be0 0x7f691c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315424756 0 0) 0x7f691c154e20 con 0x7f691c0c1fb0 2023-10-10T18:59:44.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6908003570 con 0x7f691c0c1fb0 2023-10-10T18:59:44.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927052254 0 0) 0x7f6908003570 con 0x7f691c0c1fb0 2023-10-10T18:59:44.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c14ddb0 msgr2=0x7f691c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:44.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c14ddb0 0x7f691c14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 >> v2:172.21.15.191:3300/0 conn(0x7f691c0c1be0 msgr2=0x7f691c06b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:44.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 --2- 172.21.15.175:0/1370043232 >> v2:172.21.15.191:3300/0 conn(0x7f691c0c1be0 0x7f691c06b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c14e730 con 0x7f691c0c1fb0 2023-10-10T18:59:44.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f690c001470 con 0x7f691c0c1fb0 2023-10-10T18:59:44.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 >> v1:172.21.15.175:6789/0 conn(0x7f691c0c1fb0 legacy=0x7f691c06a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:44.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c14ddb0 0x7f6908003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6908003fe0 0x7f6908004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908004d50 0x7f69080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c06bca0 con 0x7f6908003fe0 2023-10-10T18:59:44.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c154000 con 0x7f6908004d50 2023-10-10T18:59:44.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6908009a80 con 0x7f691c14ddb0 2023-10-10T18:59:44.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6920ddc700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6908003fe0 0x7f6908004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f69215dd700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c14ddb0 0x7f6908003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6921dde700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908004d50 0x7f69080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6920ddc700 1 -- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c14ddb0 msgr2=0x7f6908003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:44.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6920ddc700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c14ddb0 0x7f6908003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6920ddc700 1 -- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908004d50 msgr2=0x7f69080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T18:59:44.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6920ddc700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908004d50 0x7f69080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.398+0000 7f6920ddc700 1 -- 172.21.15.175:0/1370043232 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6908003570 con 0x7f6908003fe0 2023-10-10T18:59:44.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6920ddc700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6908003fe0 0x7f6908004810 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f690c00c3c0 tx=0x7f690c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4e99047d06dca5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:44.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c00cc30 con 0x7f6908003fe0 2023-10-10T18:59:44.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c00cdb0 con 0x7f6908003fe0 2023-10-10T18:59:44.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6913fff700 1 -- 172.21.15.175:0/1370043232 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c005b70 con 0x7f6908003fe0 2023-10-10T18:59:44.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6923841700 1 -- 172.21.15.175:0/1370043232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6908003fe0 msgr2=0x7f6908004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:44.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6923841700 1 --2- 172.21.15.175:0/1370043232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6908003fe0 0x7f6908004810 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f690c00c3c0 tx=0x7f690c001e90 comp rx=0 tx=0).stop 2023-10-10T18:59:44.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6921dde700 1 -- 172.21.15.175:0/1370043232 reap_dead start 2023-10-10T18:59:44.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6923841700 1 -- 172.21.15.175:0/1370043232 shutdown_connections 2023-10-10T18:59:44.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6923841700 1 -- 172.21.15.175:0/1370043232 >> 172.21.15.175:0/1370043232 conn(0x7f691c068280 msgr2=0x7f691c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:44.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6923841700 1 -- 172.21.15.175:0/1370043232 shutdown_connections 2023-10-10T18:59:44.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.399+0000 7f6923841700 1 -- 172.21.15.175:0/1370043232 wait complete. 2023-10-10T18:59:44.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.400+0000 7f6923841700 1 Processor -- start 2023-10-10T18:59:44.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.400+0000 7f6923841700 1 -- start start 2023-10-10T18:59:44.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6923841700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908003fe0 0x7f691c06b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6923841700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c0c15b0 0x7f691c1409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6923841700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f691c0c4230 0x7f691c140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6923841700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c1439a0 con 0x7f691c0c4230 2023-10-10T18:59:44.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6923841700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c144500 con 0x7f6908003fe0 2023-10-10T18:59:44.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6923841700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c155a20 con 0x7f691c0c15b0 2023-10-10T18:59:44.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6921dde700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f691c0c4230 0x7f691c140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6921dde700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f691c0c4230 0x7f691c140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38208/0 (socket says 172.21.15.175:38208) 2023-10-10T18:59:44.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6921dde700 1 -- 172.21.15.175:0/4205756634 learned_addr learned my addr 172.21.15.175:0/4205756634 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T18:59:44.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f69215dd700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908003fe0 0x7f691c06b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.401+0000 7f6920ddc700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c0c15b0 0x7f691c1409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f6921dde700 1 -- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c0c15b0 msgr2=0x7f691c1409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:44.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f6921dde700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c0c15b0 0x7f691c1409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f6921dde700 1 -- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908003fe0 msgr2=0x7f691c06b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:44.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f6921dde700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908003fe0 0x7f691c06b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f6921dde700 1 -- 172.21.15.175:0/4205756634 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c143bd0 con 0x7f691c0c4230 2023-10-10T18:59:44.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f6921dde700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f691c0c4230 0x7f691c140f20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f691c0c9410 tx=0x7f69140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:44.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f69117fa700 1 -- 172.21.15.175:0/4205756634 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914013750 con 0x7f691c0c4230 2023-10-10T18:59:44.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f69117fa700 1 -- 172.21.15.175:0/4205756634 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69140138b0 con 0x7f691c0c4230 2023-10-10T18:59:44.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c143eb0 con 0x7f691c0c4230 2023-10-10T18:59:44.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f69117fa700 1 -- 172.21.15.175:0/4205756634 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914013c50 con 0x7f691c0c4230 2023-10-10T18:59:44.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.402+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c156070 con 0x7f691c0c4230 2023-10-10T18:59:44.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.403+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c067570 con 0x7f691c0c4230 2023-10-10T18:59:44.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.403+0000 7f69117fa700 1 -- 172.21.15.175:0/4205756634 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6914014650 con 0x7f691c0c4230 2023-10-10T18:59:44.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.404+0000 7f69117fa700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6900066f60 0x7f6900069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T18:59:44.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.404+0000 7f69117fa700 1 -- 172.21.15.175:0/4205756634 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f6914084bf0 con 0x7f691c0c4230 2023-10-10T18:59:44.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.404+0000 7f69215dd700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6900066f60 0x7f6900069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T18:59:44.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.404+0000 7f69215dd700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6900066f60 0x7f6900069410 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f69180094d0 tx=0x7f691800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T18:59:44.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.408+0000 7f69117fa700 1 -- 172.21.15.175:0/4205756634 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6914051c10 con 0x7f691c0c4230 2023-10-10T18:59:44.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.637+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f691c144640 con 0x7f691c0c4230 2023-10-10T18:59:44.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.637+0000 7f69117fa700 1 -- 172.21.15.175:0/4205756634 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+6193 (secure 0 0 0) 0x7f691c144640 con 0x7f691c0c4230 2023-10-10T18:59:44.639 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T18:59:44.639 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T18:59:38.659091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T18:59:44.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.639+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6900066f60 msgr2=0x7f6900069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:44.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.639+0000 7f6923841700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6900066f60 0x7f6900069410 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f69180094d0 tx=0x7f691800b040 comp rx=0 tx=0).stop 2023-10-10T18:59:44.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.639+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f691c0c4230 msgr2=0x7f691c140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T18:59:44.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.639+0000 7f6923841700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f691c0c4230 0x7f691c140f20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f691c0c9410 tx=0x7f69140091a0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.640+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 shutdown_connections 2023-10-10T18:59:44.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.640+0000 7f6923841700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6900066f60 0x7f6900069410 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.640+0000 7f6923841700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f691c0c4230 0x7f691c140f20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.640+0000 7f6923841700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6908003fe0 0x7f691c06b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.640+0000 7f6923841700 1 --2- 172.21.15.175:0/4205756634 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f691c0c15b0 0x7f691c1409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T18:59:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.640+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 >> 172.21.15.175:0/4205756634 conn(0x7f691c068280 msgr2=0x7f691c153430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T18:59:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.640+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 shutdown_connections 2023-10-10T18:59:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T18:59:44.640+0000 7f6923841700 1 -- 172.21.15.175:0/4205756634 wait complete. 2023-10-10T18:59:44.644 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 106 2023-10-10T18:59:44.652 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-10T18:59:44.653 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-10-10T18:59:50.333 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T18:59:50.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:00:01.676 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 0% (66/12048) Updating files: 1% (121/12048) Updating files: 2% (241/12048) Updating files: 3% (362/12048) Updating files: 4% (482/12048) Updating files: 4% (518/12048) Updating files: 5% (603/12048) Updating files: 5% (646/12048) Updating files: 5% (684/12048) Updating files: 6% (723/12048) Updating files: 6% (763/12048) Updating files: 7% (844/12048) Updating files: 8% (964/12048) Updating files: 8% (976/12048) Updating files: 9% (1085/12048) Updating files: 9% (1154/12048) Updating files: 9% (1196/12048) Updating files: 9% (1199/12048) Updating files: 10% (1205/12048) Updating files: 11% (1326/12048) Updating files: 11% (1330/12048) Updating files: 11% (1385/12048) Updating files: 12% (1446/12048) Updating files: 12% (1459/12048) Updating files: 12% (1498/12048) Updating files: 12% (1525/12048) Updating files: 12% (1550/12048) Updating files: 13% (1567/12048) Updating files: 13% (1579/12048) Updating files: 13% (1622/12048) Updating files: 14% (1687/12048) Updating files: 14% (1724/12048) Updating files: 15% (1808/12048) Updating files: 16% (1928/12048) Updating files: 16% (1988/12048) Updating files: 17% (2049/12048) Updating files: 18% (2169/12048) Updating files: 18% (2180/12048) Updating files: 18% (2215/12048) Updating files: 18% (2220/12048) Updating files: 18% (2247/12048) Updating files: 19% (2290/12048) Updating files: 19% (2373/12048) Updating files: 20% (2410/12048) Updating files: 20% (2492/12048) Updating files: 20% (2526/12048) Updating files: 21% (2531/12048) Updating files: 22% (2651/12048) Updating files: 22% (2671/12048) Updating files: 23% (2772/12048) Updating files: 23% (2841/12048) Updating files: 24% (2892/12048) Updating files: 24% (2968/12048) Updating files: 24% (2969/12048) Updating files: 25% (3012/12048) Updating files: 26% (3133/12048) Updating files: 26% (3190/12048) Updating files: 27% (3253/12048) Updating files: 27% (3356/12048) Updating files: 28% (3374/12048) Updating files: 28% (3456/12048) Updating files: 29% (3494/12048) Updating files: 30% (3615/12048) Updating files: 30% (3622/12048) Updating files: 31% (3735/12048) Updating files: 31% (3743/12048) Updating files: 31% (3760/12048) Updating files: 32% (3856/12048) Updating files: 32% (3877/12048) Updating files: 33% (3976/12048) Updating files: 33% (4047/12048) Updating files: 33% (4076/12048) Updating files: 33% (4082/12048) Updating files: 34% (4097/12048) Updating files: 34% (4145/12048) Updating files: 35% (4217/12048) Updating files: 35% (4225/12048) Updating files: 35% (4282/12048) Updating files: 36% (4338/12048) Updating files: 37% (4458/12048) Updating files: 38% (4579/12048) Updating files: 39% (4699/12048) Updating files: 39% (4799/12048) Updating files: 40% (4820/12048) Updating files: 40% (4896/12048) Updating files: 41% (4940/12048) Updating files: 41% (5013/12048) Updating files: 42% (5061/12048) Updating files: 42% (5088/12048) Updating files: 42% (5134/12048) Updating files: 43% (5181/12048) Updating files: 44% (5302/12048) Updating files: 44% (5334/12048) Updating files: 45% (5422/12048) Updating files: 45% (5541/12048) Updating files: 46% (5543/12048) Updating files: 46% (5645/12048) Updating files: 47% (5663/12048) Updating files: 47% (5682/12048) Updating files: 47% (5692/12048) Updating files: 47% (5695/12048) Updating files: 48% (5784/12048) Updating files: 48% (5866/12048) Updating files: 49% (5904/12048) Updating files: 49% (5940/12048) Updating files: 50% (6024/12048) Updating files: 50% (6041/12048) Updating files: 51% (6145/12048) Updating files: 51% (6253/12048) Updating files: 52% (6265/12048) Updating files: 53% (6386/12048) Updating files: 54% (6506/12048) Updating files: 54% (6518/12048) Updating files: 54% (6552/12048) Updating files: 54% (6553/12048) Updating files: 55% (6627/12048) Updating files: 55% (6675/12048) Updating files: 56% (6747/12048) Updating files: 56% (6768/12048) Updating files: 57% (6868/12048) Updating files: 57% (6932/12048) Updating files: 58% (6988/12048) Updating files: 58% (6995/12048) Updating files: 59% (7109/12048) Updating files: 60% (7229/12048) Updating files: 60% (7320/12048) Updating files: 61% (7350/12048) Updating files: 62% (7470/12048) Updating files: 62% (7471/12048) Updating files: 63% (7591/12048) Updating files: 63% (7624/12048) Updating files: 63% (7691/12048) Updating files: 64% (7711/12048) Updating files: 64% (7730/12048) Updating files: 65% (7832/12048) Updating files: 65% (7873/12048) Updating files: 65% (7933/12048) Updating files: 66% (7952/12048) Updating files: 67% (8073/12048) Updating files: 68% (8193/12048) Updating files: 68% (8219/12048) Updating files: 68% (8220/12048) Updating files: 68% (8275/12048) Updating files: 69% (8314/12048) Updating files: 70% (8434/12048) Updating files: 70% (8454/12048) Updating files: 70% (8494/12048) Updating files: 70% (8504/12048) Updating files: 70% (8505/12048) Updating files: 70% (8533/12048) Updating files: 71% (8555/12048) Updating files: 71% (8605/12048) Updating files: 71% (8639/12048) Updating files: 71% (8670/12048) Updating files: 72% (8675/12048) Updating files: 73% (8796/12048) Updating files: 74% (8916/12048) Updating files: 74% (9010/12048) Updating files: 75% (9036/12048) Updating files: 75% (9055/12048) Updating files: 75% (9140/12048) Updating files: 76% (9157/12048) Updating files: 76% (9206/12048) Updating files: 77% (9277/12048) Updating files: 78% (9398/12048) Updating files: 78% (9477/12048) Updating files: 79% (9518/12048) Updating files: 79% (9596/12048) Updating files: 79% (9609/12048) Updating files: 79% (9610/12048) Updating files: 79% (9632/12048) Updating files: 80% (9639/12048) Updating files: 81% (9759/12048) Updating files: 81% (9824/12048) Updating files: 82% (9880/12048) Updating files: 82% (9882/12048) Updating files: 83% (10000/12048) Updating files: 84% (10121/12048) Updating files: 84% (10176/12048) Updating files: 85% (10241/12048) Updating files: 86% (10362/12048) Updating files: 86% (10410/12048) Updating files: 87% (10482/12048) Updating files: 88% (10603/12048) Updating files: 88% (10649/12048) Updating files: 89% (10723/12048) Updating files: 89% (10781/12048) Updating files: 90% (10844/12048) Updating files: 90% (10910/12048) Updating files: 91% (10964/12048) Updating files: 92% (11085/12048) Updating files: 93% (11205/12048) Updating files: 93% (11222/12048) Updating files: 94% (11326/12048) Updating files: 94% (11414/12048) Updating files: 95% (11446/12048) Updating files: 96% (11567/12048) Updating files: 96% (11579/12048) Updating files: 97% (11687/12048) Updating files: 98% (11808/12048) Updating files: 98% (11817/12048) Updating files: 99% (11928/12048) Updating files: 99% (11989/12048) Updating files: 99% (11998/12048) Updating files: 100% (12048/12048) Updating files: 100% (12048/12048), done. 2023-10-10T19:00:02.383 INFO:tasks.workunit.client.0.smithi175.stderr:+ trap - EXIT 2023-10-10T19:00:02.383 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ceph 2023-10-10T19:00:02.384 INFO:tasks.workunit.client.0.smithi175.stderr:++ seq 1 21 2023-10-10T19:00:02.384 INFO:tasks.workunit.client.0.smithi175.stderr:+ versions='1 2023-10-10T19:00:02.385 INFO:tasks.workunit.client.0.smithi175.stderr:2 2023-10-10T19:00:02.385 INFO:tasks.workunit.client.0.smithi175.stderr:3 2023-10-10T19:00:02.385 INFO:tasks.workunit.client.0.smithi175.stderr:4 2023-10-10T19:00:02.385 INFO:tasks.workunit.client.0.smithi175.stderr:5 2023-10-10T19:00:02.385 INFO:tasks.workunit.client.0.smithi175.stderr:6 2023-10-10T19:00:02.386 INFO:tasks.workunit.client.0.smithi175.stderr:7 2023-10-10T19:00:02.386 INFO:tasks.workunit.client.0.smithi175.stderr:8 2023-10-10T19:00:02.386 INFO:tasks.workunit.client.0.smithi175.stderr:9 2023-10-10T19:00:02.386 INFO:tasks.workunit.client.0.smithi175.stderr:10 2023-10-10T19:00:02.386 INFO:tasks.workunit.client.0.smithi175.stderr:11 2023-10-10T19:00:02.387 INFO:tasks.workunit.client.0.smithi175.stderr:12 2023-10-10T19:00:02.387 INFO:tasks.workunit.client.0.smithi175.stderr:13 2023-10-10T19:00:02.387 INFO:tasks.workunit.client.0.smithi175.stderr:14 2023-10-10T19:00:02.387 INFO:tasks.workunit.client.0.smithi175.stderr:15 2023-10-10T19:00:02.387 INFO:tasks.workunit.client.0.smithi175.stderr:16 2023-10-10T19:00:02.388 INFO:tasks.workunit.client.0.smithi175.stderr:17 2023-10-10T19:00:02.388 INFO:tasks.workunit.client.0.smithi175.stderr:18 2023-10-10T19:00:02.388 INFO:tasks.workunit.client.0.smithi175.stderr:19 2023-10-10T19:00:02.388 INFO:tasks.workunit.client.0.smithi175.stderr:20 2023-10-10T19:00:02.388 INFO:tasks.workunit.client.0.smithi175.stderr:21' 2023-10-10T19:00:02.388 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:00:02.389 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.1 2023-10-10T19:00:02.389 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.1 2023-10-10T19:00:02.389 INFO:tasks.workunit.client.0.smithi175.stdout:v0.1 2023-10-10T19:00:02.390 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.1 2023-10-10T19:00:18.654 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:18.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.783+0000 7fdfbcfc3700 1 Processor -- start 2023-10-10T19:00:18.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.783+0000 7fdfbcfc3700 1 -- start start 2023-10-10T19:00:18.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.784+0000 7fdfbcfc3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdfb80c12a0 0x7fdfb81577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:18.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.784+0000 7fdfbcfc3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 0x7fdfb814d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:18.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.784+0000 7fdfbcfc3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb8154aa0 con 0x7fdfb81573d0 2023-10-10T19:00:18.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.784+0000 7fdfbcfc3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb80bfdf0 con 0x7fdfb80c12a0 2023-10-10T19:00:18.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.784+0000 7fdfbcfc3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb8063dc0 con 0x7fdfb80c4f20 2023-10-10T19:00:18.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.784+0000 7fdfb659c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fdfb81573d0 0x7fdfb80bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33366/0 (socket says 172.21.15.175:33366) 2023-10-10T19:00:18.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.784+0000 7fdfb659c700 1 -- 172.21.15.175:0/1216373238 learned_addr learned my addr 172.21.15.175:0/1216373238 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:18.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.784+0000 7fdfb5d9b700 1 --2- 172.21.15.175:0/1216373238 >> v2:172.21.15.175:3300/0 conn(0x7fdfb80c12a0 0x7fdfb81577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:18.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb6d9d700 1 --2- 172.21.15.175:0/1216373238 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 0x7fdfb814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:18.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb559a700 1 -- 172.21.15.175:0/1216373238 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748691841 0 0) 0x7fdfb8154aa0 con 0x7fdfb81573d0 2023-10-10T19:00:18.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb559a700 1 -- 172.21.15.175:0/1216373238 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa0003570 con 0x7fdfb81573d0 2023-10-10T19:00:18.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb5d9b700 1 -- 172.21.15.175:0/1216373238 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 msgr2=0x7fdfb814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:18.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb5d9b700 1 --2- 172.21.15.175:0/1216373238 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 0x7fdfb814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:18.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb5d9b700 1 -- 172.21.15.175:0/1216373238 >> v1:172.21.15.175:6789/0 conn(0x7fdfb81573d0 legacy=0x7fdfb80bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:18.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb5d9b700 1 -- 172.21.15.175:0/1216373238 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb80c5340 con 0x7fdfb80c12a0 2023-10-10T19:00:18.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb5d9b700 1 --2- 172.21.15.175:0/1216373238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb81577a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0098e0 tx=0x7fdfac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ac61fa1bf89baa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:18.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb559a700 1 -- 172.21.15.175:0/1216373238 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac011630 con 0x7fdfb80c12a0 2023-10-10T19:00:18.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb559a700 1 -- 172.21.15.175:0/1216373238 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac011790 con 0x7fdfb80c12a0 2023-10-10T19:00:18.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.785+0000 7fdfb559a700 1 -- 172.21.15.175:0/1216373238 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac011b30 con 0x7fdfb80c12a0 2023-10-10T19:00:18.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.786+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/1216373238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 msgr2=0x7fdfb81577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:18.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.786+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/1216373238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb81577a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0098e0 tx=0x7fdfac0091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:18.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.786+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/1216373238 shutdown_connections 2023-10-10T19:00:18.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.786+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/1216373238 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 0x7fdfb814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:18.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.786+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/1216373238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb81577a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:18.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.786+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/1216373238 >> 172.21.15.175:0/1216373238 conn(0x7fdfb806dc70 msgr2=0x7fdfb8063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:18.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.786+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/1216373238 shutdown_connections 2023-10-10T19:00:18.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.786+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/1216373238 wait complete. 2023-10-10T19:00:18.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.787+0000 7fdfbcfc3700 1 Processor -- start 2023-10-10T19:00:18.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.787+0000 7fdfbcfc3700 1 -- start start 2023-10-10T19:00:18.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfbcfc3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb8163d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:18.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfbcfc3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 0x7fdfb8156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:18.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfbcfc3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb81573d0 0x7fdfb80c3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:18.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfbcfc3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb8164260 con 0x7fdfb80c12a0 2023-10-10T19:00:18.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfbcfc3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb8156ef0 con 0x7fdfb81573d0 2023-10-10T19:00:18.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfbcfc3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb80c73c0 con 0x7fdfb80c4f20 2023-10-10T19:00:18.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb8163d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:18.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb8163d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39836/0 (socket says 172.21.15.175:39836) 2023-10-10T19:00:18.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 -- 172.21.15.175:0/304061938 learned_addr learned my addr 172.21.15.175:0/304061938 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:18.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb5d9b700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 0x7fdfb8156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:18.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb6d9d700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb81573d0 0x7fdfb80c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:18.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 -- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 msgr2=0x7fdfb8156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:18.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 0x7fdfb8156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:18.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 -- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb81573d0 msgr2=0x7fdfb80c3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:18.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb81573d0 0x7fdfb80c3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:18.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 -- 172.21.15.175:0/304061938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac009420 con 0x7fdfb80c12a0 2023-10-10T19:00:18.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfb659c700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb8163d20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000cc40 tx=0x7fdfb0001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:18.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfa67fc700 1 -- 172.21.15.175:0/304061938 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb000a8b0 con 0x7fdfb80c12a0 2023-10-10T19:00:18.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfa67fc700 1 -- 172.21.15.175:0/304061938 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb000aa30 con 0x7fdfb80c12a0 2023-10-10T19:00:18.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfa67fc700 1 -- 172.21.15.175:0/304061938 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb000adf0 con 0x7fdfb80c12a0 2023-10-10T19:00:18.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb80c76a0 con 0x7fdfb80c12a0 2023-10-10T19:00:18.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.788+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb80c5ab0 con 0x7fdfb80c12a0 2023-10-10T19:00:18.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.789+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb8067570 con 0x7fdfb80c12a0 2023-10-10T19:00:18.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.790+0000 7fdfa67fc700 1 -- 172.21.15.175:0/304061938 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdfb0007300 con 0x7fdfb80c12a0 2023-10-10T19:00:18.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.790+0000 7fdfa67fc700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdf98066f30 0x7fdf980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:18.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.790+0000 7fdfa67fc700 1 -- 172.21.15.175:0/304061938 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fdfb0084b30 con 0x7fdfb80c12a0 2023-10-10T19:00:18.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.790+0000 7fdfb5d9b700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdf98066f30 0x7fdf980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:18.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.791+0000 7fdfb5d9b700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdf98066f30 0x7fdf980693e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0091e0 tx=0x7fdfac017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:18.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:18.795+0000 7fdfa67fc700 1 -- 172.21.15.175:0/304061938 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdfb0051b50 con 0x7fdfb80c12a0 2023-10-10T19:00:19.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.020+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfb8157e50 con 0x7fdfb80c12a0 2023-10-10T19:00:19.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.020+0000 7fdfa67fc700 1 -- 172.21.15.175:0/304061938 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6202 (secure 0 0 0) 0x7fdfb0055170 con 0x7fdfb80c12a0 2023-10-10T19:00:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:19.021 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:16.763873+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:19.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdf98066f30 msgr2=0x7fdf980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdf98066f30 0x7fdf980693e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0091e0 tx=0x7fdfac017000 comp rx=0 tx=0).stop 2023-10-10T19:00:19.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 msgr2=0x7fdfb8163d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb8163d20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000cc40 tx=0x7fdfb0001a70 comp rx=0 tx=0).stop 2023-10-10T19:00:19.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 shutdown_connections 2023-10-10T19:00:19.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdf98066f30 0x7fdf980693e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb81573d0 0x7fdfb80c3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfb80c12a0 0x7fdfb8163d20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 --2- 172.21.15.175:0/304061938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb80c4f20 0x7fdfb8156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 >> 172.21.15.175:0/304061938 conn(0x7fdfb806dc70 msgr2=0x7fdfb814d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:19.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 shutdown_connections 2023-10-10T19:00:19.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.022+0000 7fdfbcfc3700 1 -- 172.21.15.175:0/304061938 wait complete. 2023-10-10T19:00:19.025 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 112 2023-10-10T19:00:19.034 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=3) 2023-10-10T19:00:19.035 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-10T19:00:19.035 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:00:19.046 INFO:tasks.ceph.mds.a:Stopped 2023-10-10T19:00:19.047 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.202+0000 7f75465f9700 1 Processor -- start 2023-10-10T19:00:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.202+0000 7f75465f9700 1 -- start start 2023-10-10T19:00:19.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f75465f9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f754014fd90 0x7f7540159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:19.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f75465f9700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7540067fa0 con 0x7f75400cd040 2023-10-10T19:00:19.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f75465f9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7540156b70 con 0x7f754008a100 2023-10-10T19:00:19.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f75465f9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75400c5600 con 0x7f754014fd90 2023-10-10T19:00:19.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f7545df8700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f75400cd040 0x7f75400684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33374/0 (socket says 172.21.15.175:33374) 2023-10-10T19:00:19.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f7545df8700 1 -- 172.21.15.175:0/2891517897 learned_addr learned my addr 172.21.15.175:0/2891517897 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:19.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f75455f7700 1 --2- 172.21.15.175:0/2891517897 >> v2:172.21.15.191:3300/0 conn(0x7f754014fd90 0x7f7540159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:19.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027150491 0 0) 0x7f7540067fa0 con 0x7f75400cd040 2023-10-10T19:00:19.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.203+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c003570 con 0x7f75400cd040 2023-10-10T19:00:19.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067238392 0 0) 0x7f752c003570 con 0x7f75400cd040 2023-10-10T19:00:19.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 >> v2:172.21.15.191:3300/0 conn(0x7f754014fd90 msgr2=0x7f7540159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 --2- 172.21.15.175:0/2891517897 >> v2:172.21.15.191:3300/0 conn(0x7f754014fd90 0x7f7540159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 >> v1:172.21.15.191:6789/0 conn(0x7f754008a100 legacy=0x7f75400c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7540150160 con 0x7f75400cd040 2023-10-10T19:00:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f75380017f0 con 0x7f75400cd040 2023-10-10T19:00:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 >> v1:172.21.15.175:6789/0 conn(0x7f75400cd040 legacy=0x7f75400684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c003a60 0x7f752c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f752c004810 0x7f752c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c0051c0 0x7f752c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75400c5600 con 0x7f752c004810 2023-10-10T19:00:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c00c0d0 con 0x7f752c003a60 2023-10-10T19:00:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752c014aa0 con 0x7f752c0051c0 2023-10-10T19:00:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7544df6700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f752c004810 0x7f752c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f75455f7700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c003a60 0x7f752c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7545df8700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c0051c0 0x7f752c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7544df6700 1 -- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c0051c0 msgr2=0x7f752c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7544df6700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c0051c0 0x7f752c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7544df6700 1 -- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c003a60 msgr2=0x7f752c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7544df6700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c003a60 0x7f752c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7544df6700 1 -- 172.21.15.175:0/2891517897 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c003570 con 0x7f752c004810 2023-10-10T19:00:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7545df8700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c0051c0 0x7f752c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:00:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7545df8700 1 -- 172.21.15.175:0/2891517897 reap_dead start 2023-10-10T19:00:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7544df6700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f752c004810 0x7f752c004c80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f75300018b0 tx=0x7f7530007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c859391192aff1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7530012420 con 0x7f752c004810 2023-10-10T19:00:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75300125a0 con 0x7f752c004810 2023-10-10T19:00:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.204+0000 7f7537fff700 1 -- 172.21.15.175:0/2891517897 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7530012960 con 0x7f752c004810 2023-10-10T19:00:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.205+0000 7f75465f9700 1 -- 172.21.15.175:0/2891517897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f752c004810 msgr2=0x7f752c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.205+0000 7f75465f9700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f752c004810 0x7f752c004c80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f75300018b0 tx=0x7f7530007d30 comp rx=0 tx=0).stop 2023-10-10T19:00:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.205+0000 7f75465f9700 1 -- 172.21.15.175:0/2891517897 shutdown_connections 2023-10-10T19:00:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.205+0000 7f75465f9700 1 --2- 172.21.15.175:0/2891517897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f752c004810 0x7f752c004c80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.205+0000 7f75465f9700 1 -- 172.21.15.175:0/2891517897 >> 172.21.15.175:0/2891517897 conn(0x7f7540063d00 msgr2=0x7f75400cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.205+0000 7f75465f9700 1 -- 172.21.15.175:0/2891517897 shutdown_connections 2023-10-10T19:00:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.205+0000 7f75465f9700 1 -- 172.21.15.175:0/2891517897 wait complete. 2023-10-10T19:00:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.206+0000 7f75465f9700 1 Processor -- start 2023-10-10T19:00:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.206+0000 7f75465f9700 1 -- start start 2023-10-10T19:00:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f75465f9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c004810 0x7f7540152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f75465f9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7540150900 0x7f7540153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f75465f9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7540151720 0x7f7540151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f75465f9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7540152840 con 0x7f7540150900 2023-10-10T19:00:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f75465f9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7540158d60 con 0x7f7540151720 2023-10-10T19:00:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f75465f9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7540162460 con 0x7f752c004810 2023-10-10T19:00:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7540150900 0x7f7540153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:19.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7540150900 0x7f7540153210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39852/0 (socket says 172.21.15.175:39852) 2023-10-10T19:00:19.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 -- 172.21.15.175:0/1849649301 learned_addr learned my addr 172.21.15.175:0/1849649301 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:19.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f75455f7700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c004810 0x7f7540152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:19.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7545df8700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7540151720 0x7f7540151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:19.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 -- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c004810 msgr2=0x7f7540152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c004810 0x7f7540152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 -- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7540151720 msgr2=0x7f7540151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7540151720 0x7f7540151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 -- 172.21.15.175:0/1849649301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7540159000 con 0x7f7540150900 2023-10-10T19:00:19.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f75455f7700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c004810 0x7f7540152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:00:19.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7545df8700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7540151720 0x7f7540151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:00:19.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.207+0000 7f7544df6700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7540150900 0x7f7540153210 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f753000c3f0 tx=0x7f7530015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:19.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.208+0000 7f75357fa700 1 -- 172.21.15.175:0/1849649301 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7530001200 con 0x7f7540150900 2023-10-10T19:00:19.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.208+0000 7f75357fa700 1 -- 172.21.15.175:0/1849649301 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753000cc20 con 0x7f7540150900 2023-10-10T19:00:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.208+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7540162740 con 0x7f7540150900 2023-10-10T19:00:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.208+0000 7f75357fa700 1 -- 172.21.15.175:0/1849649301 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753001f410 con 0x7f7540150900 2023-10-10T19:00:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.208+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75400c6d50 con 0x7f7540150900 2023-10-10T19:00:19.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.209+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7540159240 con 0x7f7540150900 2023-10-10T19:00:19.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.209+0000 7f75357fa700 1 -- 172.21.15.175:0/1849649301 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f753001f7c0 con 0x7f7540150900 2023-10-10T19:00:19.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.210+0000 7f75357fa700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7524066f30 0x7f75240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:19.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.210+0000 7f75357fa700 1 -- 172.21.15.175:0/1849649301 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f75300865b0 con 0x7f7540150900 2023-10-10T19:00:19.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.210+0000 7f75455f7700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7524066f30 0x7f75240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:19.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.210+0000 7f75455f7700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7524066f30 0x7f75240693e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f753c006f00 tx=0x7f753c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:19.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.213+0000 7f75357fa700 1 -- 172.21.15.175:0/1849649301 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f75300535d0 con 0x7f7540150900 2023-10-10T19:00:19.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.439+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7540159240 con 0x7f7540150900 2023-10-10T19:00:19.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.440+0000 7f75357fa700 1 -- 172.21.15.175:0/1849649301 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f7540159240 con 0x7f7540150900 2023-10-10T19:00:19.440 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:19.440 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:16.763873+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:19.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.441+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7524066f30 msgr2=0x7f75240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.441+0000 7f75465f9700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7524066f30 0x7f75240693e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f753c006f00 tx=0x7f753c008040 comp rx=0 tx=0).stop 2023-10-10T19:00:19.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.441+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7540150900 msgr2=0x7f7540153210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:19.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.441+0000 7f75465f9700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7540150900 0x7f7540153210 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f753000c3f0 tx=0x7f7530015000 comp rx=0 tx=0).stop 2023-10-10T19:00:19.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.442+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 shutdown_connections 2023-10-10T19:00:19.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.442+0000 7f75465f9700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7524066f30 0x7f75240693e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.442+0000 7f75465f9700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7540151720 0x7f7540151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.442+0000 7f75465f9700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f752c004810 0x7f7540152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.442+0000 7f75465f9700 1 --2- 172.21.15.175:0/1849649301 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7540150900 0x7f7540153210 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:19.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.442+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 >> 172.21.15.175:0/1849649301 conn(0x7f7540063d00 msgr2=0x7f75400c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:19.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.442+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 shutdown_connections 2023-10-10T19:00:19.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:19.442+0000 7f75465f9700 1 -- 172.21.15.175:0/1849649301 wait complete. 2023-10-10T19:00:19.444 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 112 2023-10-10T19:00:19.454 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:00:20.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:00:20.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:00:21.455 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:21.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.580+0000 7f130efa3700 1 Processor -- start 2023-10-10T19:00:21.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.580+0000 7f130efa3700 1 -- start start 2023-10-10T19:00:21.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.581+0000 7f130efa3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f1308155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:21.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.581+0000 7f130efa3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13081574b0 con 0x7f13080b1fc0 2023-10-10T19:00:21.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.581+0000 7f130efa3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13080afad0 con 0x7f13080aff60 2023-10-10T19:00:21.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.581+0000 7f130efa3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1308150770 con 0x7f13080b2390 2023-10-10T19:00:21.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.581+0000 7f130cd3f700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f13080b1fc0 0x7f1308068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33388/0 (socket says 172.21.15.175:33388) 2023-10-10T19:00:21.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.581+0000 7f130cd3f700 1 -- 172.21.15.175:0/3244235194 learned_addr learned my addr 172.21.15.175:0/3244235194 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:21.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.581+0000 7f1307fff700 1 --2- 172.21.15.175:0/3244235194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f1308155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f13077fe700 1 -- 172.21.15.175:0/3244235194 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929685276 0 0) 0x7f13081574b0 con 0x7f13080b1fc0 2023-10-10T19:00:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f13077fe700 1 -- 172.21.15.175:0/3244235194 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f4003570 con 0x7f13080b1fc0 2023-10-10T19:00:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f1307fff700 1 -- 172.21.15.175:0/3244235194 >> v1:172.21.15.191:6789/0 conn(0x7f13080aff60 legacy=0x7f1308159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f1307fff700 1 -- 172.21.15.175:0/3244235194 >> v1:172.21.15.175:6789/0 conn(0x7f13080b1fc0 legacy=0x7f1308068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f13077fe700 1 -- 172.21.15.175:0/3244235194 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353509599 0 0) 0x7f13080afad0 con 0x7f13080aff60 2023-10-10T19:00:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f1307fff700 1 -- 172.21.15.175:0/3244235194 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13080af4c0 con 0x7f13080b2390 2023-10-10T19:00:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f1307fff700 1 --2- 172.21.15.175:0/3244235194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f1308155fb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009940 tx=0x7f12f00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f4a604cdac8017a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f13077fe700 1 -- 172.21.15.175:0/3244235194 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f000b700 con 0x7f13080b2390 2023-10-10T19:00:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f13077fe700 1 -- 172.21.15.175:0/3244235194 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f000b860 con 0x7f13080b2390 2023-10-10T19:00:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.582+0000 7f13077fe700 1 -- 172.21.15.175:0/3244235194 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f000bc20 con 0x7f13080b2390 2023-10-10T19:00:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.583+0000 7f130efa3700 1 -- 172.21.15.175:0/3244235194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 msgr2=0x7f1308155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.583+0000 7f130efa3700 1 --2- 172.21.15.175:0/3244235194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f1308155fb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009940 tx=0x7f12f00091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.583+0000 7f130efa3700 1 -- 172.21.15.175:0/3244235194 shutdown_connections 2023-10-10T19:00:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.583+0000 7f130efa3700 1 --2- 172.21.15.175:0/3244235194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f1308155fb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.583+0000 7f130efa3700 1 -- 172.21.15.175:0/3244235194 >> 172.21.15.175:0/3244235194 conn(0x7f1308063230 msgr2=0x7f1308063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.583+0000 7f130efa3700 1 -- 172.21.15.175:0/3244235194 shutdown_connections 2023-10-10T19:00:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.583+0000 7f130efa3700 1 -- 172.21.15.175:0/3244235194 wait complete. 2023-10-10T19:00:21.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.584+0000 7f130efa3700 1 Processor -- start 2023-10-10T19:00:21.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.584+0000 7f130efa3700 1 -- start start 2023-10-10T19:00:21.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130efa3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080aff60 0x7f13080c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:21.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130efa3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13080b1fc0 0x7f13080ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:21.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130efa3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f13080c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:21.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130efa3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13081509e0 con 0x7f13080b1fc0 2023-10-10T19:00:21.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130efa3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1308150cc0 con 0x7f13080aff60 2023-10-10T19:00:21.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130efa3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1308150fa0 con 0x7f13080b2390 2023-10-10T19:00:21.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13080b1fc0 0x7f13080ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:21.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13080b1fc0 0x7f13080ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39856/0 (socket says 172.21.15.175:39856) 2023-10-10T19:00:21.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 -- 172.21.15.175:0/4233919205 learned_addr learned my addr 172.21.15.175:0/4233919205 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:21.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130cd3f700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080aff60 0x7f13080c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:21.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130d540700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f13080c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:21.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 -- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 msgr2=0x7f13080c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:21.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f13080c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:21.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 -- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080aff60 msgr2=0x7f13080c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:21.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080aff60 0x7f13080c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:21.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 -- 172.21.15.175:0/4233919205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f0009420 con 0x7f13080b1fc0 2023-10-10T19:00:21.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1307fff700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13080b1fc0 0x7f13080ccd80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f12f00093f0 tx=0x7f12f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:21.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1304ff9700 1 -- 172.21.15.175:0/4233919205 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f001f070 con 0x7f13080b1fc0 2023-10-10T19:00:21.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1304ff9700 1 -- 172.21.15.175:0/4233919205 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f0000c90 con 0x7f13080b1fc0 2023-10-10T19:00:21.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13080c3ef0 con 0x7f13080b1fc0 2023-10-10T19:00:21.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f1304ff9700 1 -- 172.21.15.175:0/4233919205 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f000bc20 con 0x7f13080b1fc0 2023-10-10T19:00:21.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.585+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13080c6390 con 0x7f13080b1fc0 2023-10-10T19:00:21.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.586+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1308067570 con 0x7f13080b1fc0 2023-10-10T19:00:21.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.587+0000 7f1304ff9700 1 -- 172.21.15.175:0/4233919205 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f12f0023070 con 0x7f13080b1fc0 2023-10-10T19:00:21.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.587+0000 7f1304ff9700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12e8066f30 0x7f12e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:21.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.587+0000 7f1304ff9700 1 -- 172.21.15.175:0/4233919205 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f12f0085860 con 0x7f13080b1fc0 2023-10-10T19:00:21.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.587+0000 7f130cd3f700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12e8066f30 0x7f12e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:21.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.588+0000 7f130cd3f700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12e8066f30 0x7f12e80693e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f12f8005c50 tx=0x7f12f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:21.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.592+0000 7f1304ff9700 1 -- 172.21.15.175:0/4233919205 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12f0052800 con 0x7f13080b1fc0 2023-10-10T19:00:21.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.819+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1308068470 con 0x7f13080b1fc0 2023-10-10T19:00:21.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.819+0000 7f1304ff9700 1 -- 172.21.15.175:0/4233919205 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6204 (secure 0 0 0) 0x7f12f0015090 con 0x7f13080b1fc0 2023-10-10T19:00:21.819 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:21.820 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:20.788221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:21.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12e8066f30 msgr2=0x7f12e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:21.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12e8066f30 0x7f12e80693e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f12f8005c50 tx=0x7f12f8009040 comp rx=0 tx=0).stop 2023-10-10T19:00:21.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13080b1fc0 msgr2=0x7f13080ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:21.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13080b1fc0 0x7f13080ccd80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f12f00093f0 tx=0x7f12f0017000 comp rx=0 tx=0).stop 2023-10-10T19:00:21.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 shutdown_connections 2023-10-10T19:00:21.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12e8066f30 0x7f12e80693e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:21.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13080b2390 0x7f13080c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:21.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080aff60 0x7f13080c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:21.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 --2- 172.21.15.175:0/4233919205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13080b1fc0 0x7f13080ccd80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:21.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 >> 172.21.15.175:0/4233919205 conn(0x7f1308063230 msgr2=0x7f130814fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:21.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 shutdown_connections 2023-10-10T19:00:21.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:21.821+0000 7f130efa3700 1 -- 172.21.15.175:0/4233919205 wait complete. 2023-10-10T19:00:21.824 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 113 2023-10-10T19:00:21.833 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:00:23.835 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:23.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.963+0000 7fd35088c700 1 Processor -- start 2023-10-10T19:00:23.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.963+0000 7fd35088c700 1 -- start start 2023-10-10T19:00:23.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd35088c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd34c0b2390 0x7fd34c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd35088c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0aff60 0x7fd34c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd35088c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c1574b0 con 0x7fd34c0b1fc0 2023-10-10T19:00:23.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd35088c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c150770 con 0x7fd34c0b2390 2023-10-10T19:00:23.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd35088c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0afad0 con 0x7fd34c0aff60 2023-10-10T19:00:23.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34a59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd34c0b1fc0 0x7fd34c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33394/0 (socket says 172.21.15.175:33394) 2023-10-10T19:00:23.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34a59c700 1 -- 172.21.15.175:0/699243490 learned_addr learned my addr 172.21.15.175:0/699243490 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:23.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd349d9b700 1 --2- 172.21.15.175:0/699243490 >> v2:172.21.15.191:3300/0 conn(0x7fd34c0b2390 0x7fd34c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34ad9d700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0aff60 0x7fd34c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664805981 0 0) 0x7fd34c1574b0 con 0x7fd34c0b1fc0 2023-10-10T19:00:23.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd334003570 con 0x7fd34c0b1fc0 2023-10-10T19:00:23.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4039362620 0 0) 0x7fd334003570 con 0x7fd34c0b1fc0 2023-10-10T19:00:23.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0aff60 msgr2=0x7fd34c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:23.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0aff60 0x7fd34c0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:23.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 >> v2:172.21.15.191:3300/0 conn(0x7fd34c0b2390 msgr2=0x7fd34c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:23.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 --2- 172.21.15.175:0/699243490 >> v2:172.21.15.191:3300/0 conn(0x7fd34c0b2390 0x7fd34c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:23.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c0af4c0 con 0x7fd34c0b1fc0 2023-10-10T19:00:23.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd349d9b700 1 --2- 172.21.15.175:0/699243490 >> v2:172.21.15.191:3300/0 conn(0x7fd34c0b2390 0x7fd34c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:00:23.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.964+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fd33c001470 con 0x7fd34c0b1fc0 2023-10-10T19:00:23.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 >> v1:172.21.15.175:6789/0 conn(0x7fd34c0b1fc0 legacy=0x7fd34c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:23.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0aff60 0x7fd334003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334003fe0 0x7fd334004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334004d50 0x7fd3340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c150770 con 0x7fd334003fe0 2023-10-10T19:00:23.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0afad0 con 0x7fd334004d50 2023-10-10T19:00:23.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400ba90 con 0x7fd34c0aff60 2023-10-10T19:00:23.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd349d9b700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334003fe0 0x7fd334004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34a59c700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0aff60 0x7fd334003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34ad9d700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334004d50 0x7fd3340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd349d9b700 1 -- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0aff60 msgr2=0x7fd334003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:23.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd349d9b700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0aff60 0x7fd334003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:23.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd349d9b700 1 -- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334004d50 msgr2=0x7fd3340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:23.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd349d9b700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334004d50 0x7fd3340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:23.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd349d9b700 1 -- 172.21.15.175:0/699243490 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334003570 con 0x7fd334003fe0 2023-10-10T19:00:23.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd349d9b700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334003fe0 0x7fd334004810 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd340009df0 tx=0x7fd3400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1c5b9d01153a8a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:23.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34000e8c0 con 0x7fd334003fe0 2023-10-10T19:00:23.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34000ea20 con 0x7fd334003fe0 2023-10-10T19:00:23.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.965+0000 7fd34959a700 1 -- 172.21.15.175:0/699243490 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34000edc0 con 0x7fd334003fe0 2023-10-10T19:00:23.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.966+0000 7fd35088c700 1 -- 172.21.15.175:0/699243490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334003fe0 msgr2=0x7fd334004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:23.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.966+0000 7fd35088c700 1 --2- 172.21.15.175:0/699243490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334003fe0 0x7fd334004810 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd340009df0 tx=0x7fd3400091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:23.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.966+0000 7fd34ad9d700 1 -- 172.21.15.175:0/699243490 reap_dead start 2023-10-10T19:00:23.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.966+0000 7fd35088c700 1 -- 172.21.15.175:0/699243490 shutdown_connections 2023-10-10T19:00:23.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.966+0000 7fd35088c700 1 -- 172.21.15.175:0/699243490 >> 172.21.15.175:0/699243490 conn(0x7fd34c063230 msgr2=0x7fd34c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:23.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.966+0000 7fd35088c700 1 -- 172.21.15.175:0/699243490 shutdown_connections 2023-10-10T19:00:23.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.966+0000 7fd35088c700 1 -- 172.21.15.175:0/699243490 wait complete. 2023-10-10T19:00:23.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.967+0000 7fd35088c700 1 Processor -- start 2023-10-10T19:00:23.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.967+0000 7fd35088c700 1 -- start start 2023-10-10T19:00:23.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd35088c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334003fe0 0x7fd34c0c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd35088c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0c4320 0x7fd34c169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd35088c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34c169850 0x7fd34c14ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd35088c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c15d8b0 con 0x7fd34c169850 2023-10-10T19:00:23.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd35088c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c169cd0 con 0x7fd334003fe0 2023-10-10T19:00:23.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd35088c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c169f30 con 0x7fd34c0c4320 2023-10-10T19:00:23.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34c169850 0x7fd34c14ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34c169850 0x7fd34c14ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39868/0 (socket says 172.21.15.175:39868) 2023-10-10T19:00:23.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 -- 172.21.15.175:0/4267055273 learned_addr learned my addr 172.21.15.175:0/4267055273 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:23.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd349d9b700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0c4320 0x7fd34c169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34a59c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334003fe0 0x7fd34c0c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 -- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0c4320 msgr2=0x7fd34c169310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:23.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0c4320 0x7fd34c169310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:23.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 -- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334003fe0 msgr2=0x7fd34c0c4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:23.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334003fe0 0x7fd34c0c4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:23.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 -- 172.21.15.175:0/4267055273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c15d2f0 con 0x7fd34c169850 2023-10-10T19:00:23.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34a59c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334003fe0 0x7fd34c0c4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:00:23.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd34ad9d700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34c169850 0x7fd34c14ede0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd3440094d0 tx=0x7fd34400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:23.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.968+0000 7fd33a7fc700 1 -- 172.21.15.175:0/4267055273 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34400e040 con 0x7fd34c169850 2023-10-10T19:00:23.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.969+0000 7fd33a7fc700 1 -- 172.21.15.175:0/4267055273 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34400f040 con 0x7fd34c169850 2023-10-10T19:00:23.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.969+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c15d5d0 con 0x7fd34c169850 2023-10-10T19:00:23.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.969+0000 7fd33a7fc700 1 -- 172.21.15.175:0/4267055273 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34400fe50 con 0x7fd34c169850 2023-10-10T19:00:23.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.969+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c1588d0 con 0x7fd34c169850 2023-10-10T19:00:23.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.970+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c158ac0 con 0x7fd34c169850 2023-10-10T19:00:23.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.970+0000 7fd33a7fc700 1 -- 172.21.15.175:0/4267055273 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd344015640 con 0x7fd34c169850 2023-10-10T19:00:23.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.970+0000 7fd33a7fc700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd32c066f60 0x7fd32c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:23.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.971+0000 7fd33a7fc700 1 -- 172.21.15.175:0/4267055273 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fd344084f30 con 0x7fd34c169850 2023-10-10T19:00:23.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.971+0000 7fd34a59c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd32c066f60 0x7fd32c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:23.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.971+0000 7fd34a59c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd32c066f60 0x7fd32c069410 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd33c004830 tx=0x7fd33c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:23.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:23.975+0000 7fd33a7fc700 1 -- 172.21.15.175:0/4267055273 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd344051ed0 con 0x7fd34c169850 2023-10-10T19:00:24.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.203+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd34c067570 con 0x7fd34c169850 2023-10-10T19:00:24.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.203+0000 7fd33a7fc700 1 -- 172.21.15.175:0/4267055273 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fd3440554f0 con 0x7fd34c169850 2023-10-10T19:00:24.204 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:24.204 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:20.788221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:24.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.205+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd32c066f60 msgr2=0x7fd32c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:24.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.205+0000 7fd35088c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd32c066f60 0x7fd32c069410 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd33c004830 tx=0x7fd33c009040 comp rx=0 tx=0).stop 2023-10-10T19:00:24.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.205+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34c169850 msgr2=0x7fd34c14ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:24.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.205+0000 7fd35088c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34c169850 0x7fd34c14ede0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd3440094d0 tx=0x7fd34400b040 comp rx=0 tx=0).stop 2023-10-10T19:00:24.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.206+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 shutdown_connections 2023-10-10T19:00:24.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.206+0000 7fd35088c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd32c066f60 0x7fd32c069410 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:24.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.206+0000 7fd35088c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34c169850 0x7fd34c14ede0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:24.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.206+0000 7fd35088c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334003fe0 0x7fd34c0c4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:24.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.206+0000 7fd35088c700 1 --2- 172.21.15.175:0/4267055273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd34c0c4320 0x7fd34c169310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:24.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.206+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 >> 172.21.15.175:0/4267055273 conn(0x7fd34c063230 msgr2=0x7fd34c152770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:24.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.206+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 shutdown_connections 2023-10-10T19:00:24.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:24.206+0000 7fd35088c700 1 -- 172.21.15.175:0/4267055273 wait complete. 2023-10-10T19:00:24.208 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 113 2023-10-10T19:00:24.217 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:00:26.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:26.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.342+0000 7fa7f7966700 1 Processor -- start 2023-10-10T19:00:26.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.343+0000 7fa7f7966700 1 -- start start 2023-10-10T19:00:26.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.343+0000 7fa7f7966700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f00c4f20 0x7fa7f00c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:26.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.343+0000 7fa7f7966700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f00c6fb0 con 0x7fa7f00c5850 2023-10-10T19:00:26.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.343+0000 7fa7f7966700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f0063940 con 0x7fa7f00c12d0 2023-10-10T19:00:26.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.343+0000 7fa7f7966700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f0063ea0 con 0x7fa7f00c4f20 2023-10-10T19:00:26.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.343+0000 7fa7f5702700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa7f00c5850 0x7fa7f0156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35140/0 (socket says 172.21.15.175:35140) 2023-10-10T19:00:26.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.343+0000 7fa7f5702700 1 -- 172.21.15.175:0/1872160667 learned_addr learned my addr 172.21.15.175:0/1872160667 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:26.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7f5f03700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f00c4f20 0x7fa7f00c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:26.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276586185 0 0) 0x7fa7f00c6fb0 con 0x7fa7f00c5850 2023-10-10T19:00:26.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d8003570 con 0x7fa7f00c5850 2023-10-10T19:00:26.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61678460 0 0) 0x7fa7d8003570 con 0x7fa7f00c5850 2023-10-10T19:00:26.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f00c4f20 msgr2=0x7fa7f00c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:26.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f00c4f20 0x7fa7f00c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 >> v1:172.21.15.191:6789/0 conn(0x7fa7f00c12d0 legacy=0x7fa7f00c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:26.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f0063bc0 con 0x7fa7f00c5850 2023-10-10T19:00:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fa7e0001470 con 0x7fa7f00c5850 2023-10-10T19:00:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 >> v1:172.21.15.175:6789/0 conn(0x7fa7f00c5850 legacy=0x7fa7f0156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8003a90 0x7fa7d8003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7d8004810 0x7fa7d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f00c4f20 0x7fa7d80051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f0063ea0 con 0x7fa7d8004810 2023-10-10T19:00:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d800bb60 con 0x7fa7d8003a90 2023-10-10T19:00:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d800bd20 con 0x7fa7f00c4f20 2023-10-10T19:00:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7f4f01700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7d8004810 0x7fa7d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7f5702700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8003a90 0x7fa7d8003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:26.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.344+0000 7fa7f5f03700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f00c4f20 0x7fa7d80051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:26.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f4f01700 1 -- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f00c4f20 msgr2=0x7fa7d80051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:26.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f4f01700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f00c4f20 0x7fa7d80051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f4f01700 1 -- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8003a90 msgr2=0x7fa7d8003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:26.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f4f01700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8003a90 0x7fa7d8003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f4f01700 1 -- 172.21.15.175:0/1872160667 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8003570 con 0x7fa7d8004810 2023-10-10T19:00:26.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f4f01700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7d8004810 0x7fa7d8004c80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40018b0 tx=0x7fa7e4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9948946c72f0d5b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:26.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4012420 con 0x7fa7d8004810 2023-10-10T19:00:26.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e4012580 con 0x7fa7d8004810 2023-10-10T19:00:26.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7effff700 1 -- 172.21.15.175:0/1872160667 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4012920 con 0x7fa7d8004810 2023-10-10T19:00:26.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f7966700 1 -- 172.21.15.175:0/1872160667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7d8004810 msgr2=0x7fa7d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:26.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f7966700 1 --2- 172.21.15.175:0/1872160667 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7d8004810 0x7fa7d8004c80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40018b0 tx=0x7fa7e4007d30 comp rx=0 tx=0).stop 2023-10-10T19:00:26.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f5f03700 1 -- 172.21.15.175:0/1872160667 reap_dead start 2023-10-10T19:00:26.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f7966700 1 -- 172.21.15.175:0/1872160667 shutdown_connections 2023-10-10T19:00:26.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.345+0000 7fa7f7966700 1 -- 172.21.15.175:0/1872160667 >> 172.21.15.175:0/1872160667 conn(0x7fa7f006dc70 msgr2=0x7fa7f0063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:26.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.346+0000 7fa7f7966700 1 -- 172.21.15.175:0/1872160667 shutdown_connections 2023-10-10T19:00:26.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.346+0000 7fa7f7966700 1 -- 172.21.15.175:0/1872160667 wait complete. 2023-10-10T19:00:26.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f7966700 1 Processor -- start 2023-10-10T19:00:26.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f7966700 1 -- start start 2023-10-10T19:00:26.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f7966700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8004810 0x7fa7f014a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:26.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f7966700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7f015a530 0x7fa7f014ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:26.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f7966700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f015a980 0x7fa7f014f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:26.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f7966700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f0152990 con 0x7fa7f015a530 2023-10-10T19:00:26.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f7966700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f01523b0 con 0x7fa7d8004810 2023-10-10T19:00:26.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f7966700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f014e610 con 0x7fa7f015a980 2023-10-10T19:00:26.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.347+0000 7fa7f4f01700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7f015a530 0x7fa7f014ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:26.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f4f01700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7f015a530 0x7fa7f014ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56630/0 (socket says 172.21.15.175:56630) 2023-10-10T19:00:26.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f4f01700 1 -- 172.21.15.175:0/3554780102 learned_addr learned my addr 172.21.15.175:0/3554780102 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:26.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f5f03700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f015a980 0x7fa7f014f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:26.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f5702700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8004810 0x7fa7f014a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:26.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f4f01700 1 -- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f015a980 msgr2=0x7fa7f014f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:26.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f4f01700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f015a980 0x7fa7f014f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f4f01700 1 -- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8004810 msgr2=0x7fa7f014a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:26.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f4f01700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8004810 0x7fa7f014a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f4f01700 1 -- 172.21.15.175:0/3554780102 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f0152620 con 0x7fa7f015a530 2023-10-10T19:00:26.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f4f01700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7f015a530 0x7fa7f014ec20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400c3f0 tx=0x7fa7e4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:26.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7ed7fa700 1 -- 172.21.15.175:0/3554780102 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4012420 con 0x7fa7f015a530 2023-10-10T19:00:26.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7ed7fa700 1 -- 172.21.15.175:0/3554780102 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e4001200 con 0x7fa7f015a530 2023-10-10T19:00:26.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7ed7fa700 1 -- 172.21.15.175:0/3554780102 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e40126f0 con 0x7fa7f015a530 2023-10-10T19:00:26.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f014b020 con 0x7fa7f015a530 2023-10-10T19:00:26.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.348+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f0152e20 con 0x7fa7f015a530 2023-10-10T19:00:26.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.349+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f01527b0 con 0x7fa7f015a530 2023-10-10T19:00:26.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.349+0000 7fa7ed7fa700 1 -- 172.21.15.175:0/3554780102 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa7e4006a30 con 0x7fa7f015a530 2023-10-10T19:00:26.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.350+0000 7fa7ed7fa700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7dc066f60 0x7fa7dc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:26.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.350+0000 7fa7ed7fa700 1 -- 172.21.15.175:0/3554780102 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fa7e4085cd0 con 0x7fa7f015a530 2023-10-10T19:00:26.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.350+0000 7fa7f5702700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7dc066f60 0x7fa7dc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:26.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.350+0000 7fa7f5702700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7dc066f60 0x7fa7dc069410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0007f70 tx=0x7fa7e0001e50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:26.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.354+0000 7fa7ed7fa700 1 -- 172.21.15.175:0/3554780102 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7e4052c70 con 0x7fa7f015a530 2023-10-10T19:00:26.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.576+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7f006fb90 con 0x7fa7f015a530 2023-10-10T19:00:26.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.577+0000 7fa7ed7fa700 1 -- 172.21.15.175:0/3554780102 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fa7e4017090 con 0x7fa7f015a530 2023-10-10T19:00:26.577 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:26.577 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:20.788221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:26.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7dc066f60 msgr2=0x7fa7dc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:26.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7dc066f60 0x7fa7dc069410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0007f70 tx=0x7fa7e0001e50 comp rx=0 tx=0).stop 2023-10-10T19:00:26.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7f015a530 msgr2=0x7fa7f014ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:26.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7f015a530 0x7fa7f014ec20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400c3f0 tx=0x7fa7e4015000 comp rx=0 tx=0).stop 2023-10-10T19:00:26.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 shutdown_connections 2023-10-10T19:00:26.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7dc066f60 0x7fa7dc069410 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7f015a980 0x7fa7f014f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7d8004810 0x7fa7f014a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 --2- 172.21.15.175:0/3554780102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa7f015a530 0x7fa7f014ec20 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 >> 172.21.15.175:0/3554780102 conn(0x7fa7f006dc70 msgr2=0x7fa7f006b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 shutdown_connections 2023-10-10T19:00:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:26.579+0000 7fa7f7966700 1 -- 172.21.15.175:0/3554780102 wait complete. 2023-10-10T19:00:26.582 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 113 2023-10-10T19:00:26.591 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:00:28.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:28.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.719+0000 7fae9bfff700 1 Processor -- start 2023-10-10T19:00:28.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.719+0000 7fae9bfff700 1 -- start start 2023-10-10T19:00:28.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fae9c08a100 0x7fae9c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:28.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9bfff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fae9c0cd040 0x7fae9c0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:28.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9bfff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c067fa0 con 0x7fae9c14fd90 2023-10-10T19:00:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9bfff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c0c5600 con 0x7fae9c08a100 2023-10-10T19:00:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9bfff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c156b70 con 0x7fae9c0cd040 2023-10-10T19:00:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae927fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fae9c08a100 0x7fae9c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9affd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fae9c14fd90 0x7fae9c0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35156/0 (socket says 172.21.15.175:35156) 2023-10-10T19:00:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9affd700 1 -- 172.21.15.175:0/253262463 learned_addr learned my addr 172.21.15.175:0/253262463 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9b7fe700 1 --2- 172.21.15.175:0/253262463 >> v2:172.21.15.191:3300/0 conn(0x7fae9c0cd040 0x7fae9c0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae927fc700 1 -- 172.21.15.175:0/253262463 >> v2:172.21.15.191:3300/0 conn(0x7fae9c0cd040 msgr2=0x7fae9c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae927fc700 1 --2- 172.21.15.175:0/253262463 >> v2:172.21.15.191:3300/0 conn(0x7fae9c0cd040 0x7fae9c0684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae927fc700 1 -- 172.21.15.175:0/253262463 >> v1:172.21.15.175:6789/0 conn(0x7fae9c14fd90 legacy=0x7fae9c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae9a7fc700 1 -- 172.21.15.175:0/253262463 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181255691 0 0) 0x7fae9c067fa0 con 0x7fae9c14fd90 2023-10-10T19:00:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.720+0000 7fae927fc700 1 -- 172.21.15.175:0/253262463 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c150160 con 0x7fae9c08a100 2023-10-10T19:00:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae927fc700 1 --2- 172.21.15.175:0/253262463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c159c30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fae9c08a6e0 tx=0x7fae840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae5eef50fc234e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9a7fc700 1 -- 172.21.15.175:0/253262463 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400b740 con 0x7fae9c08a100 2023-10-10T19:00:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9a7fc700 1 -- 172.21.15.175:0/253262463 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8400b8a0 con 0x7fae9c08a100 2023-10-10T19:00:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9a7fc700 1 -- 172.21.15.175:0/253262463 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400bc40 con 0x7fae9c08a100 2023-10-10T19:00:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9bfff700 1 -- 172.21.15.175:0/253262463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 msgr2=0x7fae9c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9bfff700 1 --2- 172.21.15.175:0/253262463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c159c30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fae9c08a6e0 tx=0x7fae840091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9bfff700 1 -- 172.21.15.175:0/253262463 shutdown_connections 2023-10-10T19:00:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9bfff700 1 --2- 172.21.15.175:0/253262463 >> v2:172.21.15.191:3300/0 conn(0x7fae9c0cd040 0x7fae9c0684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9bfff700 1 --2- 172.21.15.175:0/253262463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c159c30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9bfff700 1 -- 172.21.15.175:0/253262463 >> 172.21.15.175:0/253262463 conn(0x7fae9c063d00 msgr2=0x7fae9c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:28.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9bfff700 1 -- 172.21.15.175:0/253262463 shutdown_connections 2023-10-10T19:00:28.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.721+0000 7fae9bfff700 1 -- 172.21.15.175:0/253262463 wait complete. 2023-10-10T19:00:28.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9bfff700 1 Processor -- start 2023-10-10T19:00:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9bfff700 1 -- start start 2023-10-10T19:00:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0cd040 0x7fae9c149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9bfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c14fd90 0x7fae9c166190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9bfff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c1574f0 con 0x7fae9c08a100 2023-10-10T19:00:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9bfff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c165780 con 0x7fae9c14fd90 2023-10-10T19:00:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9bfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c149c90 con 0x7fae9c0cd040 2023-10-10T19:00:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56662/0 (socket says 172.21.15.175:56662) 2023-10-10T19:00:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.723+0000 7fae9affd700 1 -- 172.21.15.175:0/2575729167 learned_addr learned my addr 172.21.15.175:0/2575729167 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9b7fe700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c14fd90 0x7fae9c166190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae927fc700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0cd040 0x7fae9c149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9affd700 1 -- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0cd040 msgr2=0x7fae9c149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9affd700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0cd040 0x7fae9c149750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9affd700 1 -- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c14fd90 msgr2=0x7fae9c166190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9affd700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c14fd90 0x7fae9c166190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9affd700 1 -- 172.21.15.175:0/2575729167 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae84009420 con 0x7fae9c08a100 2023-10-10T19:00:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae927fc700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0cd040 0x7fae9c149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9affd700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c150d50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00c5b0 tx=0x7fae8c001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae937fe700 1 -- 172.21.15.175:0/2575729167 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c005b30 con 0x7fae9c08a100 2023-10-10T19:00:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae937fe700 1 -- 172.21.15.175:0/2575729167 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c005c90 con 0x7fae9c08a100 2023-10-10T19:00:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae937fe700 1 -- 172.21.15.175:0/2575729167 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c007300 con 0x7fae9c08a100 2023-10-10T19:00:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c165d00 con 0x7fae9c08a100 2023-10-10T19:00:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.724+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c158de0 con 0x7fae9c08a100 2023-10-10T19:00:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.725+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c165fe0 con 0x7fae9c08a100 2023-10-10T19:00:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.725+0000 7fae937fe700 1 -- 172.21.15.175:0/2575729167 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fae8c01b3b0 con 0x7fae9c08a100 2023-10-10T19:00:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.726+0000 7fae937fe700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae7c066f60 0x7fae7c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.726+0000 7fae937fe700 1 -- 172.21.15.175:0/2575729167 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fae8c083d10 con 0x7fae9c08a100 2023-10-10T19:00:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.726+0000 7fae927fc700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae7c066f60 0x7fae7c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.726+0000 7fae927fc700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae7c066f60 0x7fae7c069410 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fae9c071dd0 tx=0x7fae84017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.730+0000 7fae937fe700 1 -- 172.21.15.175:0/2575729167 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fae8c050cb0 con 0x7fae9c08a100 2023-10-10T19:00:28.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.960+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae9c067570 con 0x7fae9c08a100 2023-10-10T19:00:28.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.961+0000 7fae937fe700 1 -- 172.21.15.175:0/2575729167 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fae8c017090 con 0x7fae9c08a100 2023-10-10T19:00:28.962 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:28.963 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:20.788221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:28.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae7c066f60 msgr2=0x7fae7c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:28.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae7c066f60 0x7fae7c069410 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fae9c071dd0 tx=0x7fae84017000 comp rx=0 tx=0).stop 2023-10-10T19:00:28.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 msgr2=0x7fae9c150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:28.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c150d50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00c5b0 tx=0x7fae8c001070 comp rx=0 tx=0).stop 2023-10-10T19:00:28.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 shutdown_connections 2023-10-10T19:00:28.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae7c066f60 0x7fae7c069410 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c14fd90 0x7fae9c166190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c08a100 0x7fae9c150d50 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 --2- 172.21.15.175:0/2575729167 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0cd040 0x7fae9c149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:28.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 >> 172.21.15.175:0/2575729167 conn(0x7fae9c063d00 msgr2=0x7fae9c0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:28.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 shutdown_connections 2023-10-10T19:00:28.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:28.963+0000 7fae9bfff700 1 -- 172.21.15.175:0/2575729167 wait complete. 2023-10-10T19:00:28.967 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 113 2023-10-10T19:00:28.976 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:00:30.978 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:31.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.104+0000 7fd0f19c5700 1 Processor -- start 2023-10-10T19:00:31.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.104+0000 7fd0f19c5700 1 -- start start 2023-10-10T19:00:31.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.104+0000 7fd0f19c5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd0ec0b2390 0x7fd0ec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:31.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.104+0000 7fd0f19c5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ec1574b0 con 0x7fd0ec0aff60 2023-10-10T19:00:31.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.104+0000 7fd0f19c5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ec0afad0 con 0x7fd0ec0b1fc0 2023-10-10T19:00:31.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.104+0000 7fd0f19c5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec150770 con 0x7fd0ec0b2390 2023-10-10T19:00:31.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.104+0000 7fd0eb7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd0ec0aff60 0x7fd0ec159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35160/0 (socket says 172.21.15.175:35160) 2023-10-10T19:00:31.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0eb7fe700 1 -- 172.21.15.175:0/3370547453 learned_addr learned my addr 172.21.15.175:0/3370547453 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:31.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0ea7fc700 1 --2- 172.21.15.175:0/3370547453 >> v2:172.21.15.191:3300/0 conn(0x7fd0ec0b2390 0x7fd0ec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:31.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656974346 0 0) 0x7fd0ec1574b0 con 0x7fd0ec0aff60 2023-10-10T19:00:31.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d8003570 con 0x7fd0ec0aff60 2023-10-10T19:00:31.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 253536755 0 0) 0x7fd0d8003570 con 0x7fd0ec0aff60 2023-10-10T19:00:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 >> v2:172.21.15.191:3300/0 conn(0x7fd0ec0b2390 msgr2=0x7fd0ec155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 --2- 172.21.15.175:0/3370547453 >> v2:172.21.15.191:3300/0 conn(0x7fd0ec0b2390 0x7fd0ec155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 >> v1:172.21.15.191:6789/0 conn(0x7fd0ec0b1fc0 legacy=0x7fd0ec068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0af4c0 con 0x7fd0ec0aff60 2023-10-10T19:00:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fd0e40017f0 con 0x7fd0ec0aff60 2023-10-10T19:00:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 >> v1:172.21.15.175:6789/0 conn(0x7fd0ec0aff60 legacy=0x7fd0ec159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0d8003a60 0x7fd0d8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0d80051c0 0x7fd0d800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec150770 con 0x7fd0d8004810 2023-10-10T19:00:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800c0d0 con 0x7fd0d8003a60 2023-10-10T19:00:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800c290 con 0x7fd0d80051c0 2023-10-10T19:00:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0ea7fc700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0eaffd700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0d8003a60 0x7fd0d8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0eb7fe700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0d80051c0 0x7fd0d800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:31.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0ea7fc700 1 -- 172.21.15.175:0/3370547453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0d80051c0 msgr2=0x7fd0d800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:31.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0ea7fc700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0d80051c0 0x7fd0d800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0ea7fc700 1 -- 172.21.15.175:0/3370547453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0d8003a60 msgr2=0x7fd0d8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:31.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0ea7fc700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0d8003a60 0x7fd0d8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0ea7fc700 1 -- 172.21.15.175:0/3370547453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8003570 con 0x7fd0d8004810 2023-10-10T19:00:31.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.105+0000 7fd0eb7fe700 1 -- 172.21.15.175:0/3370547453 reap_dead start 2023-10-10T19:00:31.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0ea7fc700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0d8004c80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009c00 tx=0x7fd0d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2832d70fa2f5e16 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:31.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400e710 con 0x7fd0d8004810 2023-10-10T19:00:31.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d400e870 con 0x7fd0d8004810 2023-10-10T19:00:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0e9ffb700 1 -- 172.21.15.175:0/3370547453 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400ec30 con 0x7fd0d8004810 2023-10-10T19:00:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3370547453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 msgr2=0x7fd0d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0f19c5700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0d8004c80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009c00 tx=0x7fd0d40091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3370547453 shutdown_connections 2023-10-10T19:00:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0f19c5700 1 --2- 172.21.15.175:0/3370547453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0d8004c80 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3370547453 >> 172.21.15.175:0/3370547453 conn(0x7fd0ec063230 msgr2=0x7fd0ec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3370547453 shutdown_connections 2023-10-10T19:00:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.106+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3370547453 wait complete. 2023-10-10T19:00:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.107+0000 7fd0f19c5700 1 Processor -- start 2023-10-10T19:00:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0f19c5700 1 -- start start 2023-10-10T19:00:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0f19c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0ec156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0f19c5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0ec0c3380 0x7fd0ec0c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0f19c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0ec155c20 0x7fd0ec169d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0f19c5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec155a70 con 0x7fd0d8004810 2023-10-10T19:00:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0f19c5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0c6820 con 0x7fd0ec0c3380 2023-10-10T19:00:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0f19c5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec169870 con 0x7fd0ec155c20 2023-10-10T19:00:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0eaffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0ec156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0eaffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0ec156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56666/0 (socket says 172.21.15.175:56666) 2023-10-10T19:00:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0eaffd700 1 -- 172.21.15.175:0/3949856762 learned_addr learned my addr 172.21.15.175:0/3949856762 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0eb7fe700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0ec155c20 0x7fd0ec169d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.108+0000 7fd0ea7fc700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0ec0c3380 0x7fd0ec0c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0eaffd700 1 -- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0ec155c20 msgr2=0x7fd0ec169d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0eaffd700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0ec155c20 0x7fd0ec169d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0eaffd700 1 -- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0ec0c3380 msgr2=0x7fd0ec0c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0eaffd700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0ec0c3380 0x7fd0ec0c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0eaffd700 1 -- 172.21.15.175:0/3949856762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec15bf10 con 0x7fd0d8004810 2023-10-10T19:00:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0eaffd700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0ec156b30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc002980 tx=0x7fd0dc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0e2ffd700 1 -- 172.21.15.175:0/3949856762 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc006820 con 0x7fd0d8004810 2023-10-10T19:00:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0e2ffd700 1 -- 172.21.15.175:0/3949856762 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc006980 con 0x7fd0d8004810 2023-10-10T19:00:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec15c1f0 con 0x7fd0d8004810 2023-10-10T19:00:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0e2ffd700 1 -- 172.21.15.175:0/3949856762 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc006d40 con 0x7fd0d8004810 2023-10-10T19:00:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.109+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec155790 con 0x7fd0d8004810 2023-10-10T19:00:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.110+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec169640 con 0x7fd0d8004810 2023-10-10T19:00:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.110+0000 7fd0e2ffd700 1 -- 172.21.15.175:0/3949856762 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd0dc007230 con 0x7fd0d8004810 2023-10-10T19:00:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.111+0000 7fd0e2ffd700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0cc066f60 0x7fd0cc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.111+0000 7fd0e2ffd700 1 -- 172.21.15.175:0/3949856762 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fd0dc0846e0 con 0x7fd0d8004810 2023-10-10T19:00:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.111+0000 7fd0ea7fc700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0cc066f60 0x7fd0cc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.111+0000 7fd0ea7fc700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0cc066f60 0x7fd0cc069410 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40091e0 tx=0x7fd0d401a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.115+0000 7fd0e2ffd700 1 -- 172.21.15.175:0/3949856762 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0ec169640 con 0x7fd0d8004810 2023-10-10T19:00:31.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.339+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0ec169640 con 0x7fd0d8004810 2023-10-10T19:00:31.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.340+0000 7fd0e2ffd700 1 -- 172.21.15.175:0/3949856762 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fd0dc051680 con 0x7fd0d8004810 2023-10-10T19:00:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:31.341 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:20.788221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5196,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":3,"incarnation":57,"state":"up:active","state_seq":46,"addr":"172.21.15.175:6834/3874110823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3874110823},{"type":"v1","addr":"172.21.15.175:6834","nonce":3874110823}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0cc066f60 msgr2=0x7fd0cc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0cc066f60 0x7fd0cc069410 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40091e0 tx=0x7fd0d401a000 comp rx=0 tx=0).stop 2023-10-10T19:00:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 msgr2=0x7fd0ec156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0ec156b30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc002980 tx=0x7fd0dc007d30 comp rx=0 tx=0).stop 2023-10-10T19:00:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 shutdown_connections 2023-10-10T19:00:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0cc066f60 0x7fd0cc069410 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0ec155c20 0x7fd0ec169d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0d8004810 0x7fd0ec156b30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 --2- 172.21.15.175:0/3949856762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0ec0c3380 0x7fd0ec0c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:31.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 >> 172.21.15.175:0/3949856762 conn(0x7fd0ec063230 msgr2=0x7fd0ec14fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:31.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 shutdown_connections 2023-10-10T19:00:31.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:31.342+0000 7fd0f19c5700 1 -- 172.21.15.175:0/3949856762 wait complete. 2023-10-10T19:00:31.346 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 113 2023-10-10T19:00:31.355 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:00:33.356 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:33.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.483+0000 7fb748634700 1 Processor -- start 2023-10-10T19:00:33.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.483+0000 7fb748634700 1 -- start start 2023-10-10T19:00:33.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.484+0000 7fb748634700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb7400b1fc0 0x7fb740155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.484+0000 7fb748634700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7401574b0 con 0x7fb7400aff60 2023-10-10T19:00:33.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.484+0000 7fb748634700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400afad0 con 0x7fb7400b2390 2023-10-10T19:00:33.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.484+0000 7fb748634700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb740150770 con 0x7fb7400b1fc0 2023-10-10T19:00:33.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.484+0000 7fb746bd1700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb7400aff60 0x7fb740159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35174/0 (socket says 172.21.15.175:35174) 2023-10-10T19:00:33.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.484+0000 7fb746bd1700 1 -- 172.21.15.175:0/2976730808 learned_addr learned my addr 172.21.15.175:0/2976730808 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:33.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7463d0700 1 --2- 172.21.15.175:0/2976730808 >> v2:172.21.15.191:3300/0 conn(0x7fb7400b1fc0 0x7fb740155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835115435 0 0) 0x7fb7401574b0 con 0x7fb7400aff60 2023-10-10T19:00:33.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c003570 con 0x7fb7400aff60 2023-10-10T19:00:33.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833685018 0 0) 0x7fb72c003570 con 0x7fb7400aff60 2023-10-10T19:00:33.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 >> v2:172.21.15.191:3300/0 conn(0x7fb7400b1fc0 msgr2=0x7fb740155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:33.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 --2- 172.21.15.175:0/2976730808 >> v2:172.21.15.191:3300/0 conn(0x7fb7400b1fc0 0x7fb740155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 >> v1:172.21.15.191:6789/0 conn(0x7fb7400b2390 legacy=0x7fb740068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400af4c0 con 0x7fb7400aff60 2023-10-10T19:00:33.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fb738001470 con 0x7fb7400aff60 2023-10-10T19:00:33.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 >> v1:172.21.15.175:6789/0 conn(0x7fb7400aff60 legacy=0x7fb740159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c003a60 0x7fb72c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb72c004810 0x7fb72c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb72c0051c0 0x7fb72c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb740150770 con 0x7fb72c0051c0 2023-10-10T19:00:33.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c00c0d0 con 0x7fb72c003a60 2023-10-10T19:00:33.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c00c290 con 0x7fb72c004810 2023-10-10T19:00:33.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb746bd1700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb72c0051c0 0x7fb72c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb7463d0700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c003a60 0x7fb72c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb745bcf700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb72c004810 0x7fb72c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb746bd1700 1 -- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb72c004810 msgr2=0x7fb72c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:33.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb746bd1700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb72c004810 0x7fb72c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb746bd1700 1 -- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c003a60 msgr2=0x7fb72c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:33.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb746bd1700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c003a60 0x7fb72c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb746bd1700 1 -- 172.21.15.175:0/2976730808 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c003570 con 0x7fb72c0051c0 2023-10-10T19:00:33.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.485+0000 7fb745bcf700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb72c004810 0x7fb72c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:00:33.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb746bd1700 1 -- 172.21.15.175:0/2976730808 reap_dead start 2023-10-10T19:00:33.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb746bd1700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb72c0051c0 0x7fb72c00ba60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb73800c390 tx=0x7fb7400af650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6dc4da1376b3002 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:33.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73800cc60 con 0x7fb72c0051c0 2023-10-10T19:00:33.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73800cdc0 con 0x7fb72c0051c0 2023-10-10T19:00:33.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb7453ce700 1 -- 172.21.15.175:0/2976730808 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738005b70 con 0x7fb72c0051c0 2023-10-10T19:00:33.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb748634700 1 -- 172.21.15.175:0/2976730808 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb72c0051c0 msgr2=0x7fb72c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb748634700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb72c0051c0 0x7fb72c00ba60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb73800c390 tx=0x7fb7400af650 comp rx=0 tx=0).stop 2023-10-10T19:00:33.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb748634700 1 -- 172.21.15.175:0/2976730808 shutdown_connections 2023-10-10T19:00:33.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb748634700 1 --2- 172.21.15.175:0/2976730808 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb72c0051c0 0x7fb72c00ba60 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb748634700 1 -- 172.21.15.175:0/2976730808 >> 172.21.15.175:0/2976730808 conn(0x7fb740063230 msgr2=0x7fb740063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:33.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb748634700 1 -- 172.21.15.175:0/2976730808 shutdown_connections 2023-10-10T19:00:33.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.486+0000 7fb748634700 1 -- 172.21.15.175:0/2976730808 wait complete. 2023-10-10T19:00:33.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.487+0000 7fb748634700 1 Processor -- start 2023-10-10T19:00:33.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.487+0000 7fb748634700 1 -- start start 2023-10-10T19:00:33.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb748634700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c0051c0 0x7fb74014def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb748634700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb74015b9a0 0x7fb7400cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb748634700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7400cbab0 0x7fb740157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb748634700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400c6a70 con 0x7fb74015b9a0 2023-10-10T19:00:33.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb748634700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb740159500 con 0x7fb72c0051c0 2023-10-10T19:00:33.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb748634700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400c5e60 con 0x7fb7400cbab0 2023-10-10T19:00:33.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb745bcf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb74015b9a0 0x7fb7400cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb745bcf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb74015b9a0 0x7fb7400cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56684/0 (socket says 172.21.15.175:56684) 2023-10-10T19:00:33.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb7463d0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c0051c0 0x7fb74014def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb745bcf700 1 -- 172.21.15.175:0/476481565 learned_addr learned my addr 172.21.15.175:0/476481565 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:33.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.488+0000 7fb746bd1700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7400cbab0 0x7fb740157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb745bcf700 1 -- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7400cbab0 msgr2=0x7fb740157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb745bcf700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7400cbab0 0x7fb740157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb745bcf700 1 -- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c0051c0 msgr2=0x7fb74014def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb745bcf700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c0051c0 0x7fb74014def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb745bcf700 1 -- 172.21.15.175:0/476481565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400c6140 con 0x7fb74015b9a0 2023-10-10T19:00:33.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb745bcf700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb74015b9a0 0x7fb7400cb570 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb73000e580 tx=0x7fb730009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:33.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb7367fc700 1 -- 172.21.15.175:0/476481565 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730011460 con 0x7fb74015b9a0 2023-10-10T19:00:33.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb7367fc700 1 -- 172.21.15.175:0/476481565 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb730010040 con 0x7fb74015b9a0 2023-10-10T19:00:33.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7400c6420 con 0x7fb74015b9a0 2023-10-10T19:00:33.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb7367fc700 1 -- 172.21.15.175:0/476481565 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7300117f0 con 0x7fb74015b9a0 2023-10-10T19:00:33.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.489+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74014e430 con 0x7fb74015b9a0 2023-10-10T19:00:33.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.490+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7400c68c0 con 0x7fb74015b9a0 2023-10-10T19:00:33.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.490+0000 7fb7367fc700 1 -- 172.21.15.175:0/476481565 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb730011ba0 con 0x7fb74015b9a0 2023-10-10T19:00:33.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.491+0000 7fb7367fc700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb724066fb0 0x7fb724069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.491+0000 7fb7367fc700 1 -- 172.21.15.175:0/476481565 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6423+0+0 (secure 0 0 0) 0x7fb730083a80 con 0x7fb74015b9a0 2023-10-10T19:00:33.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.491+0000 7fb7463d0700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb724066fb0 0x7fb724069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.491+0000 7fb7463d0700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb724066fb0 0x7fb724069460 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fd0 tx=0x7fb73c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:33.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.495+0000 7fb7367fc700 1 -- 172.21.15.175:0/476481565 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb73004d5a0 con 0x7fb74015b9a0 2023-10-10T19:00:33.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.722+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7400cbe80 con 0x7fb74015b9a0 2023-10-10T19:00:33.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.723+0000 7fb7367fc700 1 -- 172.21.15.175:0/476481565 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5443 (secure 0 0 0) 0x7fb730019070 con 0x7fb74015b9a0 2023-10-10T19:00:33.724 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:33.724 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:33.566584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:33.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb724066fb0 msgr2=0x7fb724069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb724066fb0 0x7fb724069460 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fd0 tx=0x7fb73c00b040 comp rx=0 tx=0).stop 2023-10-10T19:00:33.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb74015b9a0 msgr2=0x7fb7400cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb74015b9a0 0x7fb7400cb570 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb73000e580 tx=0x7fb730009040 comp rx=0 tx=0).stop 2023-10-10T19:00:33.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 shutdown_connections 2023-10-10T19:00:33.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb724066fb0 0x7fb724069460 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7400cbab0 0x7fb740157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb72c0051c0 0x7fb74014def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 --2- 172.21.15.175:0/476481565 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb74015b9a0 0x7fb7400cb570 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 >> 172.21.15.175:0/476481565 conn(0x7fb740063230 msgr2=0x7fb740063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:33.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 shutdown_connections 2023-10-10T19:00:33.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.725+0000 7fb748634700 1 -- 172.21.15.175:0/476481565 wait complete. 2023-10-10T19:00:33.728 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 114 2023-10-10T19:00:33.738 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-10T19:00:33.738 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:00:33.739 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:33.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.854+0000 7fe5a9348700 1 Processor -- start 2023-10-10T19:00:33.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.854+0000 7fe5a9348700 1 -- start start 2023-10-10T19:00:33.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a9348700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a9348700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe5a40aff60 0x7fe5a40b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a9348700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a41574b0 con 0x7fe5a40b2390 2023-10-10T19:00:33.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a9348700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a4150770 con 0x7fe5a40aff60 2023-10-10T19:00:33.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a9348700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40afad0 con 0x7fe5a40b1fc0 2023-10-10T19:00:33.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a37fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe5a40aff60 0x7fe5a40b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a27fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe5a40b2390 0x7fe5a4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35188/0 (socket says 172.21.15.175:35188) 2023-10-10T19:00:33.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a27fc700 1 -- 172.21.15.175:0/961948469 learned_addr learned my addr 172.21.15.175:0/961948469 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:33.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.855+0000 7fe5a2ffd700 1 --2- 172.21.15.175:0/961948469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a37fe700 1 -- 172.21.15.175:0/961948469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 msgr2=0x7fe5a4155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a37fe700 1 --2- 172.21.15.175:0/961948469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a37fe700 1 -- 172.21.15.175:0/961948469 >> v1:172.21.15.175:6789/0 conn(0x7fe5a40b2390 legacy=0x7fe5a4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a37fe700 1 -- 172.21.15.175:0/961948469 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40af4c0 con 0x7fe5a40aff60 2023-10-10T19:00:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a1ffb700 1 -- 172.21.15.175:0/961948469 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743894224 0 0) 0x7fe5a41574b0 con 0x7fe5a40b2390 2023-10-10T19:00:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a2ffd700 1 --2- 172.21.15.175:0/961948469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a37fe700 1 --2- 172.21.15.175:0/961948469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40b03e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe59c009910 tx=0x7fe59c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920a174860d735cd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a1ffb700 1 -- 172.21.15.175:0/961948469 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00b8b0 con 0x7fe5a40aff60 2023-10-10T19:00:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a1ffb700 1 -- 172.21.15.175:0/961948469 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c00ba10 con 0x7fe5a40aff60 2023-10-10T19:00:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a1ffb700 1 -- 172.21.15.175:0/961948469 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00bdb0 con 0x7fe5a40aff60 2023-10-10T19:00:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a9348700 1 -- 172.21.15.175:0/961948469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 msgr2=0x7fe5a40b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a9348700 1 --2- 172.21.15.175:0/961948469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40b03e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe59c009910 tx=0x7fe59c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a9348700 1 -- 172.21.15.175:0/961948469 shutdown_connections 2023-10-10T19:00:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a9348700 1 --2- 172.21.15.175:0/961948469 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40b03e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a9348700 1 --2- 172.21.15.175:0/961948469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a9348700 1 -- 172.21.15.175:0/961948469 >> 172.21.15.175:0/961948469 conn(0x7fe5a4063230 msgr2=0x7fe5a4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a9348700 1 -- 172.21.15.175:0/961948469 shutdown_connections 2023-10-10T19:00:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.856+0000 7fe5a9348700 1 -- 172.21.15.175:0/961948469 wait complete. 2023-10-10T19:00:33.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a9348700 1 Processor -- start 2023-10-10T19:00:33.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a9348700 1 -- start start 2023-10-10T19:00:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a9348700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a9348700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a9348700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5a40b2390 0x7fe5a4157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a9348700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4169b30 con 0x7fe5a40aff60 2023-10-10T19:00:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a9348700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40c0070 con 0x7fe5a40b2390 2023-10-10T19:00:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a9348700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41597a0 con 0x7fe5a40b1fc0 2023-10-10T19:00:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56712/0 (socket says 172.21.15.175:56712) 2023-10-10T19:00:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.858+0000 7fe5a2ffd700 1 -- 172.21.15.175:0/687478239 learned_addr learned my addr 172.21.15.175:0/687478239 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a27fc700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a37fe700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5a40b2390 0x7fe5a4157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a2ffd700 1 -- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 msgr2=0x7fe5a4159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a2ffd700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a2ffd700 1 -- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5a40b2390 msgr2=0x7fe5a4157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:33.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a2ffd700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5a40b2390 0x7fe5a4157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:33.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a2ffd700 1 -- 172.21.15.175:0/687478239 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c009420 con 0x7fe5a40aff60 2023-10-10T19:00:33.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a37fe700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5a40b2390 0x7fe5a4157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:33.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a2ffd700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40ca5a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe59400a400 tx=0x7fe594007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:33.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe59affd700 1 -- 172.21.15.175:0/687478239 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59400d650 con 0x7fe5a40aff60 2023-10-10T19:00:33.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe59affd700 1 -- 172.21.15.175:0/687478239 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59400d7b0 con 0x7fe5a40aff60 2023-10-10T19:00:33.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a41594c0 con 0x7fe5a40aff60 2023-10-10T19:00:33.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe59affd700 1 -- 172.21.15.175:0/687478239 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59400db70 con 0x7fe5a40aff60 2023-10-10T19:00:33.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.859+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a40caef0 con 0x7fe5a40aff60 2023-10-10T19:00:33.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.860+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a4159d20 con 0x7fe5a40aff60 2023-10-10T19:00:33.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.860+0000 7fe59affd700 1 -- 172.21.15.175:0/687478239 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe594015440 con 0x7fe5a40aff60 2023-10-10T19:00:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.861+0000 7fe59affd700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe584066f30 0x7fe5840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.861+0000 7fe59affd700 1 -- 172.21.15.175:0/687478239 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fe594085120 con 0x7fe5a40aff60 2023-10-10T19:00:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.861+0000 7fe5a27fc700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe584066f30 0x7fe5840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.861+0000 7fe5a27fc700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe584066f30 0x7fe5840693e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe58c00c870 tx=0x7fe58c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:33.865+0000 7fe59affd700 1 -- 172.21.15.175:0/687478239 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe59404e5a0 con 0x7fe5a40aff60 2023-10-10T19:00:34.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.089+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5a4067570 con 0x7fe5a40aff60 2023-10-10T19:00:34.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.089+0000 7fe59affd700 1 -- 172.21.15.175:0/687478239 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5443 (secure 0 0 0) 0x7fe594052080 con 0x7fe5a40aff60 2023-10-10T19:00:34.091 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:34.091 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:33.566584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:34.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe584066f30 msgr2=0x7fe5840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:34.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe584066f30 0x7fe5840693e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe58c00c870 tx=0x7fe58c009040 comp rx=0 tx=0).stop 2023-10-10T19:00:34.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 msgr2=0x7fe5a40ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:34.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40ca5a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe59400a400 tx=0x7fe594007910 comp rx=0 tx=0).stop 2023-10-10T19:00:34.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 shutdown_connections 2023-10-10T19:00:34.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe584066f30 0x7fe5840693e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:34.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5a40b2390 0x7fe5a4157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:34.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5a40aff60 0x7fe5a40ca5a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:34.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 --2- 172.21.15.175:0/687478239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5a40b1fc0 0x7fe5a4159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:34.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 >> 172.21.15.175:0/687478239 conn(0x7fe5a4063230 msgr2=0x7fe5a4153dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:34.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 shutdown_connections 2023-10-10T19:00:34.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:34.092+0000 7fe5a9348700 1 -- 172.21.15.175:0/687478239 wait complete. 2023-10-10T19:00:34.095 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 114 2023-10-10T19:00:34.106 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 31 }, "gid_5357": { "addr": "172.21.15.191:6836/1945747421", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 1945747421, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1945747421, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5357, "incarnation": 114, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:replay", "state_seq": 1 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:00:33.566584+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 5332, "mds_2": 5367, "mds_3": 5357, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:00:36.107 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:36.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.233+0000 7f808c1f2700 1 Processor -- start 2023-10-10T19:00:36.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.234+0000 7f808c1f2700 1 -- start start 2023-10-10T19:00:36.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.234+0000 7f808c1f2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f80840c52a0 0x7f808406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:36.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.234+0000 7f808c1f2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8084155ca0 con 0x7f808414cc10 2023-10-10T19:00:36.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.234+0000 7f808c1f2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8084156b00 con 0x7f808414c840 2023-10-10T19:00:36.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.234+0000 7f808c1f2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808406d700 con 0x7f80840c52a0 2023-10-10T19:00:36.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.234+0000 7f808a78f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f80840c52a0 0x7f808406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:36.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.234+0000 7f808978d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f808414cc10 0x7f8084152970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56254/0 (socket says 172.21.15.175:56254) 2023-10-10T19:00:36.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.234+0000 7f808978d700 1 -- 172.21.15.175:0/2805608052 learned_addr learned my addr 172.21.15.175:0/2805608052 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:36.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f8088f8c700 1 -- 172.21.15.175:0/2805608052 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775867810 0 0) 0x7f8084155ca0 con 0x7f808414cc10 2023-10-10T19:00:36.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f8088f8c700 1 -- 172.21.15.175:0/2805608052 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8070003570 con 0x7f808414cc10 2023-10-10T19:00:36.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f808a78f700 1 -- 172.21.15.175:0/2805608052 >> v1:172.21.15.191:6789/0 conn(0x7f808414c840 legacy=0x7f80840c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:36.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f808a78f700 1 -- 172.21.15.175:0/2805608052 >> v1:172.21.15.175:6789/0 conn(0x7f808414cc10 legacy=0x7f8084152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:36.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f808a78f700 1 -- 172.21.15.175:0/2805608052 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8084155ef0 con 0x7f80840c52a0 2023-10-10T19:00:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f808a78f700 1 --2- 172.21.15.175:0/2805608052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80840c52a0 0x7f808406e7a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f807c00aa90 tx=0x7f807c00a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6826091191d7906 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f8088f8c700 1 -- 172.21.15.175:0/2805608052 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c00ca00 con 0x7f80840c52a0 2023-10-10T19:00:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f8088f8c700 1 -- 172.21.15.175:0/2805608052 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c00cb60 con 0x7f80840c52a0 2023-10-10T19:00:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.235+0000 7f8088f8c700 1 -- 172.21.15.175:0/2805608052 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c018400 con 0x7f80840c52a0 2023-10-10T19:00:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.236+0000 7f808c1f2700 1 -- 172.21.15.175:0/2805608052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80840c52a0 msgr2=0x7f808406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.236+0000 7f808c1f2700 1 --2- 172.21.15.175:0/2805608052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80840c52a0 0x7f808406e7a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f807c00aa90 tx=0x7f807c00a330 comp rx=0 tx=0).stop 2023-10-10T19:00:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.236+0000 7f808c1f2700 1 -- 172.21.15.175:0/2805608052 shutdown_connections 2023-10-10T19:00:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.236+0000 7f808c1f2700 1 --2- 172.21.15.175:0/2805608052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80840c52a0 0x7f808406e7a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.236+0000 7f808c1f2700 1 -- 172.21.15.175:0/2805608052 >> 172.21.15.175:0/2805608052 conn(0x7f8084068280 msgr2=0x7f808406af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.236+0000 7f808c1f2700 1 -- 172.21.15.175:0/2805608052 shutdown_connections 2023-10-10T19:00:36.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.236+0000 7f808c1f2700 1 -- 172.21.15.175:0/2805608052 wait complete. 2023-10-10T19:00:36.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.237+0000 7f808c1f2700 1 Processor -- start 2023-10-10T19:00:36.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.237+0000 7f808c1f2700 1 -- start start 2023-10-10T19:00:36.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.237+0000 7f808c1f2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80840c52a0 0x7f80840c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:36.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.237+0000 7f808c1f2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f808414c840 0x7f8084153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:36.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808c1f2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808414cc10 0x7f8084154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:36.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808c1f2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084168020 con 0x7f808414c840 2023-10-10T19:00:36.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808c1f2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084154a10 con 0x7f808414cc10 2023-10-10T19:00:36.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808c1f2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80840c3760 con 0x7f80840c52a0 2023-10-10T19:00:36.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f808414c840 0x7f8084153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:36.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f808414c840 0x7f8084153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37270/0 (socket says 172.21.15.175:37270) 2023-10-10T19:00:36.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 -- 172.21.15.175:0/3282184632 learned_addr learned my addr 172.21.15.175:0/3282184632 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:36.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808a78f700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808414cc10 0x7f8084154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:36.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f8089f8e700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80840c52a0 0x7f80840c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:36.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 -- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80840c52a0 msgr2=0x7f80840c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:36.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80840c52a0 0x7f80840c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:36.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 -- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808414cc10 msgr2=0x7f8084154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808414cc10 0x7f8084154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 -- 172.21.15.175:0/3282184632 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c00a5d0 con 0x7f808414c840 2023-10-10T19:00:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f8089f8e700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80840c52a0 0x7f80840c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808a78f700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808414cc10 0x7f8084154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808978d700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f808414c840 0x7f8084153cf0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f807400c7f0 tx=0x7f8074007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f8079ffb700 1 -- 172.21.15.175:0/3282184632 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074005b30 con 0x7f808414c840 2023-10-10T19:00:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f8079ffb700 1 -- 172.21.15.175:0/3282184632 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8074005c90 con 0x7f808414c840 2023-10-10T19:00:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80840c3a40 con 0x7f808414c840 2023-10-10T19:00:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.238+0000 7f8079ffb700 1 -- 172.21.15.175:0/3282184632 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074007340 con 0x7f808414c840 2023-10-10T19:00:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.239+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8084153870 con 0x7f808414c840 2023-10-10T19:00:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.240+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8084154ba0 con 0x7f808414c840 2023-10-10T19:00:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.240+0000 7f8079ffb700 1 -- 172.21.15.175:0/3282184632 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f807401b400 con 0x7f808414c840 2023-10-10T19:00:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.240+0000 7f8079ffb700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8068066fb0 0x7f8068069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.240+0000 7f8079ffb700 1 -- 172.21.15.175:0/3282184632 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f80740843f0 con 0x7f808414c840 2023-10-10T19:00:36.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.240+0000 7f8089f8e700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8068066fb0 0x7f8068069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:36.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.241+0000 7f8089f8e700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8068066fb0 0x7f8068069460 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f808000d220 tx=0x7f8080009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:36.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.245+0000 7f8079ffb700 1 -- 172.21.15.175:0/3282184632 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80740513d0 con 0x7f808414c840 2023-10-10T19:00:36.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.468+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8084067570 con 0x7f808414c840 2023-10-10T19:00:36.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.468+0000 7f8079ffb700 1 -- 172.21.15.175:0/3282184632 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5443 (secure 0 0 0) 0x7f80740076f0 con 0x7f808414c840 2023-10-10T19:00:36.470 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:36.470 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:33.566584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:36.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8068066fb0 msgr2=0x7f8068069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:36.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8068066fb0 0x7f8068069460 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f808000d220 tx=0x7f8080009040 comp rx=0 tx=0).stop 2023-10-10T19:00:36.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f808414c840 msgr2=0x7f8084153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:36.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f808414c840 0x7f8084153cf0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f807400c7f0 tx=0x7f8074007e00 comp rx=0 tx=0).stop 2023-10-10T19:00:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 shutdown_connections 2023-10-10T19:00:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8068066fb0 0x7f8068069460 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808414cc10 0x7f8084154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80840c52a0 0x7f80840c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 --2- 172.21.15.175:0/3282184632 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f808414c840 0x7f8084153cf0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 >> 172.21.15.175:0/3282184632 conn(0x7f8084068280 msgr2=0x7f80840c05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 shutdown_connections 2023-10-10T19:00:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:36.471+0000 7f808c1f2700 1 -- 172.21.15.175:0/3282184632 wait complete. 2023-10-10T19:00:36.475 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 114 2023-10-10T19:00:36.484 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:00:38.485 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:38.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.618+0000 7f0844ab9700 1 Processor -- start 2023-10-10T19:00:38.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.618+0000 7f0844ab9700 1 -- start start 2023-10-10T19:00:38.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.618+0000 7f0844ab9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c6950 0x7f084006c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:38.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.618+0000 7f0844ab9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f084006e8f0 0x7f084006c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:38.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.618+0000 7f0844ab9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084006c710 con 0x7f084006e520 2023-10-10T19:00:38.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.618+0000 7f0844ab9700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08400c5280 con 0x7f084006e8f0 2023-10-10T19:00:38.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.618+0000 7f0844ab9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08400c5540 con 0x7f08400c6950 2023-10-10T19:00:38.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f084006e8f0 0x7f084006c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:38.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f084006e8f0 0x7f084006c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37278/0 (socket says 172.21.15.175:37278) 2023-10-10T19:00:38.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083ffff700 1 -- 172.21.15.175:0/897803490 learned_addr learned my addr 172.21.15.175:0/897803490 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:38.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f0836ffd700 1 --2- 172.21.15.175:0/897803490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c6950 0x7f084006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:38.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083ffff700 1 -- 172.21.15.175:0/897803490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c6950 msgr2=0x7f084006c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:38.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083ffff700 1 --2- 172.21.15.175:0/897803490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c6950 0x7f084006c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083ffff700 1 -- 172.21.15.175:0/897803490 >> v1:172.21.15.191:6789/0 conn(0x7f084006e520 legacy=0x7f08400c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:38.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083ffff700 1 -- 172.21.15.175:0/897803490 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08400c5e90 con 0x7f084006e8f0 2023-10-10T19:00:38.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f0836ffd700 1 --2- 172.21.15.175:0/897803490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c6950 0x7f084006c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:00:38.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083ffff700 1 --2- 172.21.15.175:0/897803490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084006e8f0 0x7f084006c0f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0838009f20 tx=0x7f0838009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27137b91c3ddd7f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083effd700 1 -- 172.21.15.175:0/897803490 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083800e070 con 0x7f084006e8f0 2023-10-10T19:00:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083effd700 1 -- 172.21.15.175:0/897803490 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083800ba80 con 0x7f084006e8f0 2023-10-10T19:00:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.619+0000 7f083effd700 1 -- 172.21.15.175:0/897803490 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08380092e0 con 0x7f084006e8f0 2023-10-10T19:00:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.620+0000 7f0844ab9700 1 -- 172.21.15.175:0/897803490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084006e8f0 msgr2=0x7f084006c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.620+0000 7f0844ab9700 1 --2- 172.21.15.175:0/897803490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084006e8f0 0x7f084006c0f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0838009f20 tx=0x7f0838009840 comp rx=0 tx=0).stop 2023-10-10T19:00:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.620+0000 7f0844ab9700 1 -- 172.21.15.175:0/897803490 shutdown_connections 2023-10-10T19:00:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.620+0000 7f0844ab9700 1 --2- 172.21.15.175:0/897803490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084006e8f0 0x7f084006c0f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.620+0000 7f0844ab9700 1 --2- 172.21.15.175:0/897803490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c6950 0x7f084006c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.620+0000 7f0844ab9700 1 -- 172.21.15.175:0/897803490 >> 172.21.15.175:0/897803490 conn(0x7f0840063230 msgr2=0x7f084006f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.620+0000 7f0844ab9700 1 -- 172.21.15.175:0/897803490 shutdown_connections 2023-10-10T19:00:38.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.620+0000 7f0844ab9700 1 -- 172.21.15.175:0/897803490 wait complete. 2023-10-10T19:00:38.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.621+0000 7f0844ab9700 1 Processor -- start 2023-10-10T19:00:38.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.621+0000 7f0844ab9700 1 -- start start 2023-10-10T19:00:38.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0844ab9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f084006e520 0x7f084014c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:38.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0844ab9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f084006e8f0 0x7f0840149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:38.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0844ab9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08400c6950 0x7f0840147c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:38.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0844ab9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08401663e0 con 0x7f08400c6950 2023-10-10T19:00:38.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0844ab9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400c8e90 con 0x7f084006e8f0 2023-10-10T19:00:38.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0844ab9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084014ee70 con 0x7f084006e520 2023-10-10T19:00:38.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08400c6950 0x7f0840147c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:38.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08400c6950 0x7f0840147c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37282/0 (socket says 172.21.15.175:37282) 2023-10-10T19:00:38.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 -- 172.21.15.175:0/2233464683 learned_addr learned my addr 172.21.15.175:0/2233464683 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:38.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0836ffd700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f084006e8f0 0x7f0840149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:38.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083f7fe700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f084006e520 0x7f084014c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:38.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 -- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f084006e520 msgr2=0x7f084014c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:38.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f084006e520 0x7f084014c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 -- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f084006e8f0 msgr2=0x7f0840149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:38.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f084006e8f0 0x7f0840149280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 -- 172.21.15.175:0/2233464683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838009ae0 con 0x7f08400c6950 2023-10-10T19:00:38.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f083ffff700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08400c6950 0x7f0840147c70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f083800b6c0 tx=0x7f0838018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:38.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0837fff700 1 -- 172.21.15.175:0/2233464683 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08380032f0 con 0x7f08400c6950 2023-10-10T19:00:38.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0837fff700 1 -- 172.21.15.175:0/2233464683 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838008d50 con 0x7f08400c6950 2023-10-10T19:00:38.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0840148b10 con 0x7f08400c6950 2023-10-10T19:00:38.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0837fff700 1 -- 172.21.15.175:0/2233464683 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838026690 con 0x7f08400c6950 2023-10-10T19:00:38.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.622+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08400c9b80 con 0x7f08400c6950 2023-10-10T19:00:38.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.623+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0840067570 con 0x7f08400c6950 2023-10-10T19:00:38.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.624+0000 7f0837fff700 1 -- 172.21.15.175:0/2233464683 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0838003040 con 0x7f08400c6950 2023-10-10T19:00:38.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.624+0000 7f0837fff700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0820066f30 0x7f08200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:38.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.624+0000 7f0837fff700 1 -- 172.21.15.175:0/2233464683 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f08380866b0 con 0x7f08400c6950 2023-10-10T19:00:38.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.624+0000 7f083f7fe700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0820066f30 0x7f08200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:38.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.625+0000 7f083f7fe700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0820066f30 0x7f08200693e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f08300039e0 tx=0x7f0830007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:38.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.628+0000 7f0837fff700 1 -- 172.21.15.175:0/2233464683 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0838053600 con 0x7f08400c6950 2023-10-10T19:00:38.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.858+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08400c31b0 con 0x7f08400c6950 2023-10-10T19:00:38.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.859+0000 7f0837fff700 1 -- 172.21.15.175:0/2233464683 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5443 (secure 0 0 0) 0x7f0838056c20 con 0x7f08400c6950 2023-10-10T19:00:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:38.860 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:33.566584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:38.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0820066f30 msgr2=0x7f08200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:38.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0820066f30 0x7f08200693e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f08300039e0 tx=0x7f0830007ec0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08400c6950 msgr2=0x7f0840147c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:38.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08400c6950 0x7f0840147c70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f083800b6c0 tx=0x7f0838018000 comp rx=0 tx=0).stop 2023-10-10T19:00:38.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 shutdown_connections 2023-10-10T19:00:38.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0820066f30 0x7f08200693e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08400c6950 0x7f0840147c70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f084006e520 0x7f084014c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 --2- 172.21.15.175:0/2233464683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f084006e8f0 0x7f0840149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:38.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 >> 172.21.15.175:0/2233464683 conn(0x7f0840063230 msgr2=0x7f0840154d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:38.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 shutdown_connections 2023-10-10T19:00:38.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:38.861+0000 7f0844ab9700 1 -- 172.21.15.175:0/2233464683 wait complete. 2023-10-10T19:00:38.865 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 114 2023-10-10T19:00:38.874 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:00:40.875 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:41.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.000+0000 7f0549e15700 1 Processor -- start 2023-10-10T19:00:41.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.000+0000 7f0549e15700 1 -- start start 2023-10-10T19:00:41.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f0549e15700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05440b2390 0x7f0544155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:41.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f0549e15700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05441574b0 con 0x7f05440aff60 2023-10-10T19:00:41.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f0549e15700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05440afad0 con 0x7f05440b1fc0 2023-10-10T19:00:41.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f0549e15700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0544150770 con 0x7f05440b2390 2023-10-10T19:00:41.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f053bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05440b2390 0x7f0544155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:41.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f0543fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f05440aff60 0x7f0544159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56262/0 (socket says 172.21.15.175:56262) 2023-10-10T19:00:41.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f0543fff700 1 -- 172.21.15.175:0/1688355702 learned_addr learned my addr 172.21.15.175:0/1688355702 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:41.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f053bfff700 1 -- 172.21.15.175:0/1688355702 >> v1:172.21.15.191:6789/0 conn(0x7f05440b1fc0 legacy=0x7f0544068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:41.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f053bfff700 1 -- 172.21.15.175:0/1688355702 >> v1:172.21.15.175:6789/0 conn(0x7f05440aff60 legacy=0x7f0544159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:41.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f053bfff700 1 -- 172.21.15.175:0/1688355702 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05440af4c0 con 0x7f05440b2390 2023-10-10T19:00:41.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.001+0000 7f0542ffd700 1 -- 172.21.15.175:0/1688355702 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763628606 0 0) 0x7f05441574b0 con 0x7f05440aff60 2023-10-10T19:00:41.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f053bfff700 1 --2- 172.21.15.175:0/1688355702 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440b2390 0x7f0544155fb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f052c0098e0 tx=0x7f052c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6dfedb373605929 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0542ffd700 1 -- 172.21.15.175:0/1688355702 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c009e80 con 0x7f05440b2390 2023-10-10T19:00:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0542ffd700 1 -- 172.21.15.175:0/1688355702 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c00b8f0 con 0x7f05440b2390 2023-10-10T19:00:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0542ffd700 1 -- 172.21.15.175:0/1688355702 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c00bc70 con 0x7f05440b2390 2023-10-10T19:00:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0549e15700 1 -- 172.21.15.175:0/1688355702 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440b2390 msgr2=0x7f0544155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0549e15700 1 --2- 172.21.15.175:0/1688355702 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440b2390 0x7f0544155fb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f052c0098e0 tx=0x7f052c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0549e15700 1 -- 172.21.15.175:0/1688355702 shutdown_connections 2023-10-10T19:00:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0549e15700 1 --2- 172.21.15.175:0/1688355702 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440b2390 0x7f0544155fb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0549e15700 1 -- 172.21.15.175:0/1688355702 >> 172.21.15.175:0/1688355702 conn(0x7f0544063230 msgr2=0x7f0544063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0549e15700 1 -- 172.21.15.175:0/1688355702 shutdown_connections 2023-10-10T19:00:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.002+0000 7f0549e15700 1 -- 172.21.15.175:0/1688355702 wait complete. 2023-10-10T19:00:41.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.003+0000 7f0549e15700 1 Processor -- start 2023-10-10T19:00:41.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f0549e15700 1 -- start start 2023-10-10T19:00:41.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f0549e15700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440aff60 0x7f05440c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:41.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f0549e15700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05440b1fc0 0x7f0544150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:41.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f0549e15700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05440b2390 0x7f0544165d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:41.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f0549e15700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05440cba40 con 0x7f05440aff60 2023-10-10T19:00:41.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f0549e15700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05440cab70 con 0x7f05440b2390 2023-10-10T19:00:41.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f0549e15700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05440cae50 con 0x7f05440b1fc0 2023-10-10T19:00:41.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f05437fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440aff60 0x7f05440c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:41.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f05437fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440aff60 0x7f05440c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37296/0 (socket says 172.21.15.175:37296) 2023-10-10T19:00:41.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f05437fe700 1 -- 172.21.15.175:0/4032701490 learned_addr learned my addr 172.21.15.175:0/4032701490 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:41.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f0543fff700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05440b2390 0x7f0544165d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:41.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.004+0000 7f053bfff700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05440b1fc0 0x7f0544150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:41.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f05437fe700 1 -- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05440b1fc0 msgr2=0x7f0544150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:41.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f05437fe700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05440b1fc0 0x7f0544150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f05437fe700 1 -- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05440b2390 msgr2=0x7f0544165d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:41.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f05437fe700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05440b2390 0x7f0544165d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f05437fe700 1 -- 172.21.15.175:0/4032701490 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c009420 con 0x7f05440aff60 2023-10-10T19:00:41.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f05437fe700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440aff60 0x7f05440c2e90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f053400c590 tx=0x7f053400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:41.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f053b7fe700 1 -- 172.21.15.175:0/4032701490 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534011590 con 0x7f05440aff60 2023-10-10T19:00:41.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f053b7fe700 1 -- 172.21.15.175:0/4032701490 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0534011710 con 0x7f05440aff60 2023-10-10T19:00:41.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f053b7fe700 1 -- 172.21.15.175:0/4032701490 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534011ad0 con 0x7f05440aff60 2023-10-10T19:00:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05440cb3f0 con 0x7f05440aff60 2023-10-10T19:00:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.005+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05440c33d0 con 0x7f05440aff60 2023-10-10T19:00:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.006+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05440cb250 con 0x7f05440aff60 2023-10-10T19:00:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.006+0000 7f053b7fe700 1 -- 172.21.15.175:0/4032701490 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f053401b3b0 con 0x7f05440aff60 2023-10-10T19:00:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.007+0000 7f053b7fe700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0524066f30 0x7f05240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.007+0000 7f053b7fe700 1 -- 172.21.15.175:0/4032701490 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f0534083cc0 con 0x7f05440aff60 2023-10-10T19:00:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.007+0000 7f053bfff700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0524066f30 0x7f05240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.007+0000 7f053bfff700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0524066f30 0x7f05240693e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f052c0091e0 tx=0x7f052c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.011+0000 7f053b7fe700 1 -- 172.21.15.175:0/4032701490 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05440cb250 con 0x7f05440aff60 2023-10-10T19:00:41.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.238+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05440cb250 con 0x7f05440aff60 2023-10-10T19:00:41.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.238+0000 7f053b7fe700 1 -- 172.21.15.175:0/4032701490 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f05440cb250 con 0x7f05440aff60 2023-10-10T19:00:41.240 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:41.240 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:40.837784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:resolve","state_seq":16,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:41.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0524066f30 msgr2=0x7f05240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:41.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0524066f30 0x7f05240693e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f052c0091e0 tx=0x7f052c017000 comp rx=0 tx=0).stop 2023-10-10T19:00:41.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440aff60 msgr2=0x7f05440c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:41.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440aff60 0x7f05440c2e90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f053400c590 tx=0x7f053400c8a0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 shutdown_connections 2023-10-10T19:00:41.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0524066f30 0x7f05240693e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05440b2390 0x7f0544165d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05440aff60 0x7f05440c2e90 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 --2- 172.21.15.175:0/4032701490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05440b1fc0 0x7f0544150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 >> 172.21.15.175:0/4032701490 conn(0x7f0544063230 msgr2=0x7f054414fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 shutdown_connections 2023-10-10T19:00:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:41.241+0000 7f0549e15700 1 -- 172.21.15.175:0/4032701490 wait complete. 2023-10-10T19:00:41.245 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 115 2023-10-10T19:00:41.254 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:00:43.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:43.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.383+0000 7f6f8a8e5700 1 Processor -- start 2023-10-10T19:00:43.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.383+0000 7f6f8a8e5700 1 -- start start 2023-10-10T19:00:43.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.383+0000 7f6f8a8e5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6f840b1fc0 0x7f6f84155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.383+0000 7f6f8a8e5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b2390 0x7f6f84068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.383+0000 7f6f8a8e5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f841574b0 con 0x7f6f84068bd0 2023-10-10T19:00:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.383+0000 7f6f8a8e5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f84150770 con 0x7f6f840b1fc0 2023-10-10T19:00:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.383+0000 7f6f8a8e5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f840afad0 con 0x7f6f840b2390 2023-10-10T19:00:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f83fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6f840b1fc0 0x7f6f84155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f88e82700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6f84068bd0 0x7f6f840aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56264/0 (socket says 172.21.15.175:56264) 2023-10-10T19:00:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f88e82700 1 -- 172.21.15.175:0/1797413174 learned_addr learned my addr 172.21.15.175:0/1797413174 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f837fe700 1 --2- 172.21.15.175:0/1797413174 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b2390 0x7f6f84068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f83fff700 1 -- 172.21.15.175:0/1797413174 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b2390 msgr2=0x7f6f84068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f83fff700 1 --2- 172.21.15.175:0/1797413174 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b2390 0x7f6f84068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f83fff700 1 -- 172.21.15.175:0/1797413174 >> v1:172.21.15.175:6789/0 conn(0x7f6f84068bd0 legacy=0x7f6f840aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f83fff700 1 -- 172.21.15.175:0/1797413174 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f840af4c0 con 0x7f6f840b1fc0 2023-10-10T19:00:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f83fff700 1 --2- 172.21.15.175:0/1797413174 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840b1fc0 0x7f6f84155fb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6f780098e0 tx=0x7f6f780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a35833755b9d60 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f82ffd700 1 -- 172.21.15.175:0/1797413174 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78009e80 con 0x7f6f840b1fc0 2023-10-10T19:00:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f82ffd700 1 -- 172.21.15.175:0/1797413174 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7800b8f0 con 0x7f6f840b1fc0 2023-10-10T19:00:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.384+0000 7f6f82ffd700 1 -- 172.21.15.175:0/1797413174 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7800bc70 con 0x7f6f840b1fc0 2023-10-10T19:00:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.385+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/1797413174 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840b1fc0 msgr2=0x7f6f84155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:43.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.385+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/1797413174 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840b1fc0 0x7f6f84155fb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6f780098e0 tx=0x7f6f780091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.385+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/1797413174 shutdown_connections 2023-10-10T19:00:43.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.385+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/1797413174 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840b1fc0 0x7f6f84155fb0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.385+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/1797413174 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b2390 0x7f6f84068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.385+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/1797413174 >> 172.21.15.175:0/1797413174 conn(0x7f6f84063230 msgr2=0x7f6f84063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:43.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.385+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/1797413174 shutdown_connections 2023-10-10T19:00:43.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.385+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/1797413174 wait complete. 2023-10-10T19:00:43.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.386+0000 7f6f8a8e5700 1 Processor -- start 2023-10-10T19:00:43.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.386+0000 7f6f8a8e5700 1 -- start start 2023-10-10T19:00:43.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f8a8e5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f84068bd0 0x7f6f84169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:43.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f8a8e5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b1fc0 0x7f6f84155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:43.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f8a8e5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840b2390 0x7f6f8414ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:43.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f8a8e5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84166540 con 0x7f6f84068bd0 2023-10-10T19:00:43.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f8a8e5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84169fd0 con 0x7f6f840b2390 2023-10-10T19:00:43.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f8a8e5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84150d20 con 0x7f6f840b1fc0 2023-10-10T19:00:43.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f84068bd0 0x7f6f84169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:43.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f84068bd0 0x7f6f84169a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37304/0 (socket says 172.21.15.175:37304) 2023-10-10T19:00:43.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 -- 172.21.15.175:0/7763589 learned_addr learned my addr 172.21.15.175:0/7763589 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:43.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f837fe700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b1fc0 0x7f6f84155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:43.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f88e82700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840b2390 0x7f6f8414ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:43.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 -- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b1fc0 msgr2=0x7f6f84155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:43.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b1fc0 0x7f6f84155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 -- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840b2390 msgr2=0x7f6f8414ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:43.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840b2390 0x7f6f8414ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 -- 172.21.15.175:0/7763589 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f78009420 con 0x7f6f84068bd0 2023-10-10T19:00:43.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f837fe700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b1fc0 0x7f6f84155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:43.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f88e82700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840b2390 0x7f6f8414ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:43.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f83fff700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f84068bd0 0x7f6f84169a90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009f00 tx=0x7f6f78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:43.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f6bfff700 1 -- 172.21.15.175:0/7763589 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7800bc70 con 0x7f6f84068bd0 2023-10-10T19:00:43.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f6bfff700 1 -- 172.21.15.175:0/7763589 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f78011420 con 0x7f6f84068bd0 2023-10-10T19:00:43.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f6bfff700 1 -- 172.21.15.175:0/7763589 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78011800 con 0x7f6f84068bd0 2023-10-10T19:00:43.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f840cced0 con 0x7f6f84068bd0 2023-10-10T19:00:43.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.387+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f840cd170 con 0x7f6f84068bd0 2023-10-10T19:00:43.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.388+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f84067570 con 0x7f6f84068bd0 2023-10-10T19:00:43.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.389+0000 7f6f6bfff700 1 -- 172.21.15.175:0/7763589 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6f78011b70 con 0x7f6f84068bd0 2023-10-10T19:00:43.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.389+0000 7f6f6bfff700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f70066f30 0x7f6f700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:43.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.389+0000 7f6f6bfff700 1 -- 172.21.15.175:0/7763589 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f6f78086270 con 0x7f6f84068bd0 2023-10-10T19:00:43.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.389+0000 7f6f837fe700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f70066f30 0x7f6f700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:43.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.390+0000 7f6f837fe700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f70066f30 0x7f6f700693e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6f840cce60 tx=0x7f6f74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:43.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.393+0000 7f6f6bfff700 1 -- 172.21.15.175:0/7763589 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f780531c0 con 0x7f6f84068bd0 2023-10-10T19:00:43.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.621+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f84069060 con 0x7f6f84068bd0 2023-10-10T19:00:43.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.621+0000 7f6f6bfff700 1 -- 172.21.15.175:0/7763589 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f6f780567e0 con 0x7f6f84068bd0 2023-10-10T19:00:43.623 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:43.623 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:40.837784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:resolve","state_seq":16,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:43.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f70066f30 msgr2=0x7f6f700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:43.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f70066f30 0x7f6f700693e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6f840cce60 tx=0x7f6f74008040 comp rx=0 tx=0).stop 2023-10-10T19:00:43.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f84068bd0 msgr2=0x7f6f84169a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:43.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f84068bd0 0x7f6f84169a90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009f00 tx=0x7f6f78017000 comp rx=0 tx=0).stop 2023-10-10T19:00:43.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 shutdown_connections 2023-10-10T19:00:43.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f70066f30 0x7f6f700693e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840b2390 0x7f6f8414ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f84068bd0 0x7f6f84169a90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 --2- 172.21.15.175:0/7763589 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f840b1fc0 0x7f6f84155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:43.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 >> 172.21.15.175:0/7763589 conn(0x7f6f84063230 msgr2=0x7f6f8414fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:43.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 shutdown_connections 2023-10-10T19:00:43.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:43.623+0000 7f6f8a8e5700 1 -- 172.21.15.175:0/7763589 wait complete. 2023-10-10T19:00:43.627 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 115 2023-10-10T19:00:43.636 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:00:45.637 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:45.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.763+0000 7f23e5b54700 1 Processor -- start 2023-10-10T19:00:45.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.763+0000 7f23e5b54700 1 -- start start 2023-10-10T19:00:45.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23e5b54700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e0157400 0x7f23e006bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:45.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23e5b54700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e006d8e0 con 0x7f23e006c2e0 2023-10-10T19:00:45.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23e5b54700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e006dbc0 con 0x7f23e00c4640 2023-10-10T19:00:45.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23e5b54700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e006e050 con 0x7f23e0157400 2023-10-10T19:00:45.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23e5353700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f23e006c2e0 0x7f23e014d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38526/0 (socket says 172.21.15.175:38526) 2023-10-10T19:00:45.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23e5353700 1 -- 172.21.15.175:0/425300312 learned_addr learned my addr 172.21.15.175:0/425300312 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:45.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23e4b52700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e0157400 0x7f23e006bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:45.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710260384 0 0) 0x7f23e006d8e0 con 0x7f23e006c2e0 2023-10-10T19:00:45.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.764+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c8003570 con 0x7f23e006c2e0 2023-10-10T19:00:45.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112044676 0 0) 0x7f23e006dbc0 con 0x7f23e00c4640 2023-10-10T19:00:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e006d8e0 con 0x7f23e00c4640 2023-10-10T19:00:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3853759041 0 0) 0x7f23c8003570 con 0x7f23e006c2e0 2023-10-10T19:00:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e0157400 msgr2=0x7f23e006bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e0157400 0x7f23e006bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 >> v1:172.21.15.191:6789/0 conn(0x7f23e00c4640 legacy=0x7f23e01577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:45.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e00c1030 con 0x7f23e006c2e0 2023-10-10T19:00:45.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f23d8001470 con 0x7f23e006c2e0 2023-10-10T19:00:45.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 >> v1:172.21.15.175:6789/0 conn(0x7f23e006c2e0 legacy=0x7f23e014d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:45.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8003c20 0x7f23c8004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:45.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e0157400 0x7f23c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23c8004d50 0x7f23c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e006e050 con 0x7f23c8004d50 2023-10-10T19:00:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c800bbf0 con 0x7f23c8003c20 2023-10-10T19:00:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c800bdb0 con 0x7f23e0157400 2023-10-10T19:00:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23e5353700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23c8004d50 0x7f23c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23e4b52700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8003c20 0x7f23c8004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d7fff700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e0157400 0x7f23c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23e5353700 1 -- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e0157400 msgr2=0x7f23c8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23e5353700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e0157400 0x7f23c8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23e5353700 1 -- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8003c20 msgr2=0x7f23c8004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:45.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23e5353700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8003c20 0x7f23c8004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:45.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23e5353700 1 -- 172.21.15.175:0/425300312 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8003570 con 0x7f23c8004d50 2023-10-10T19:00:45.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23e5353700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23c8004d50 0x7f23c80051c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f23d800c2d0 tx=0x7f23e006ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd60453981444528 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:45.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d800cc00 con 0x7f23c8004d50 2023-10-10T19:00:45.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d800cd60 con 0x7f23c8004d50 2023-10-10T19:00:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.765+0000 7f23d77fe700 1 -- 172.21.15.175:0/425300312 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8005b70 con 0x7f23c8004d50 2023-10-10T19:00:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.766+0000 7f23e5b54700 1 -- 172.21.15.175:0/425300312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23c8004d50 msgr2=0x7f23c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.766+0000 7f23e5b54700 1 --2- 172.21.15.175:0/425300312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23c8004d50 0x7f23c80051c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f23d800c2d0 tx=0x7f23e006ee90 comp rx=0 tx=0).stop 2023-10-10T19:00:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.766+0000 7f23e5353700 1 -- 172.21.15.175:0/425300312 reap_dead start 2023-10-10T19:00:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.766+0000 7f23e5b54700 1 -- 172.21.15.175:0/425300312 shutdown_connections 2023-10-10T19:00:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.766+0000 7f23e5b54700 1 -- 172.21.15.175:0/425300312 >> 172.21.15.175:0/425300312 conn(0x7f23e0063cd0 msgr2=0x7f23e006d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.766+0000 7f23e5b54700 1 -- 172.21.15.175:0/425300312 shutdown_connections 2023-10-10T19:00:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.766+0000 7f23e5b54700 1 -- 172.21.15.175:0/425300312 wait complete. 2023-10-10T19:00:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.767+0000 7f23e5b54700 1 Processor -- start 2023-10-10T19:00:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.767+0000 7f23e5b54700 1 -- start start 2023-10-10T19:00:45.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5b54700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8004d50 0x7f23e00c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:45.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5b54700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e00c6e30 0x7f23e0146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:45.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5b54700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23e0156650 0x7f23e0166e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:45.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5b54700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e00c5740 con 0x7f23e0156650 2023-10-10T19:00:45.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5b54700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e00c7340 con 0x7f23c8004d50 2023-10-10T19:00:45.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5b54700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e0154f00 con 0x7f23e00c6e30 2023-10-10T19:00:45.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5353700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23e0156650 0x7f23e0166e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:45.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5353700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23e0156650 0x7f23e0166e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36762/0 (socket says 172.21.15.175:36762) 2023-10-10T19:00:45.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5353700 1 -- 172.21.15.175:0/3795027766 learned_addr learned my addr 172.21.15.175:0/3795027766 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:45.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23d7fff700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e00c6e30 0x7f23e0146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:45.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e4b52700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8004d50 0x7f23e00c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:45.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5353700 1 -- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e00c6e30 msgr2=0x7f23e0146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:45.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5353700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e00c6e30 0x7f23e0146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:45.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5353700 1 -- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8004d50 msgr2=0x7f23e00c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:45.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5353700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8004d50 0x7f23e00c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:45.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.768+0000 7f23e5353700 1 -- 172.21.15.175:0/3795027766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e00c5950 con 0x7f23e0156650 2023-10-10T19:00:45.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.769+0000 7f23e5353700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23e0156650 0x7f23e0166e50 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f23d8000c00 tx=0x7f23d8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:45.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.769+0000 7f23d4ff9700 1 -- 172.21.15.175:0/3795027766 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8004a40 con 0x7f23e0156650 2023-10-10T19:00:45.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.769+0000 7f23d4ff9700 1 -- 172.21.15.175:0/3795027766 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d8004bc0 con 0x7f23e0156650 2023-10-10T19:00:45.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.769+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e00c5c30 con 0x7f23e0156650 2023-10-10T19:00:45.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.769+0000 7f23d4ff9700 1 -- 172.21.15.175:0/3795027766 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80070d0 con 0x7f23e0156650 2023-10-10T19:00:45.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.769+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e00c6690 con 0x7f23e0156650 2023-10-10T19:00:45.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.770+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e01550c0 con 0x7f23e0156650 2023-10-10T19:00:45.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.770+0000 7f23d4ff9700 1 -- 172.21.15.175:0/3795027766 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f23d8019070 con 0x7f23e0156650 2023-10-10T19:00:45.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.770+0000 7f23d4ff9700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23cc066f30 0x7f23cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:45.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.771+0000 7f23d4ff9700 1 -- 172.21.15.175:0/3795027766 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f23d8085730 con 0x7f23e0156650 2023-10-10T19:00:45.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.771+0000 7f23e4b52700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23cc066f30 0x7f23cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:45.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.771+0000 7f23e4b52700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23cc066f30 0x7f23cc0693e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0094d0 tx=0x7f23dc00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:45.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:45.775+0000 7f23d4ff9700 1 -- 172.21.15.175:0/3795027766 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23d8052680 con 0x7f23e0156650 2023-10-10T19:00:46.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.004+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23e0067570 con 0x7f23e0156650 2023-10-10T19:00:46.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.004+0000 7f23d4ff9700 1 -- 172.21.15.175:0/3795027766 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f23d8055ca0 con 0x7f23e0156650 2023-10-10T19:00:46.005 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:46.006 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:40.837784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:resolve","state_seq":16,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:46.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23cc066f30 msgr2=0x7f23cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:46.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23cc066f30 0x7f23cc0693e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0094d0 tx=0x7f23dc00b040 comp rx=0 tx=0).stop 2023-10-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23e0156650 msgr2=0x7f23e0166e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23e0156650 0x7f23e0166e50 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f23d8000c00 tx=0x7f23d8015000 comp rx=0 tx=0).stop 2023-10-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 shutdown_connections 2023-10-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23cc066f30 0x7f23cc0693e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23e0156650 0x7f23e0166e50 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23c8004d50 0x7f23e00c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 --2- 172.21.15.175:0/3795027766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23e00c6e30 0x7f23e0146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 >> 172.21.15.175:0/3795027766 conn(0x7f23e0063cd0 msgr2=0x7f23e0147eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 shutdown_connections 2023-10-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:46.006+0000 7f23e5b54700 1 -- 172.21.15.175:0/3795027766 wait complete. 2023-10-10T19:00:46.010 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 115 2023-10-10T19:00:46.020 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 115, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 115, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5332": { "addr": "172.21.15.191:6839/1598171176", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1598171176, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1598171176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5332, "incarnation": 82, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 31 }, "gid_5357": { "addr": "172.21.15.191:6836/1945747421", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 1945747421, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1945747421, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5357, "incarnation": 114, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:resolve", "state_seq": 16 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:00:40.837784+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 5332, "mds_2": 5367, "mds_3": 5357, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:00:48.021 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:48.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.146+0000 7fa5f0da3700 1 Processor -- start 2023-10-10T19:00:48.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.146+0000 7fa5f0da3700 1 -- start start 2023-10-10T19:00:48.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.146+0000 7fa5f0da3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa5ec0b2390 0x7fa5ec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:48.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.146+0000 7fa5f0da3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:48.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.146+0000 7fa5f0da3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec1574b0 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.146+0000 7fa5f0da3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec150770 con 0x7fa5ec0b2390 2023-10-10T19:00:48.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.146+0000 7fa5f0da3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0afad0 con 0x7fa5ec0aff60 2023-10-10T19:00:48.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e1d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa5ec0b2390 0x7fa5ec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:48.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5ea59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa5ec0b1fc0 0x7fa5ec068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38540/0 (socket says 172.21.15.175:38540) 2023-10-10T19:00:48.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5ea59c700 1 -- 172.21.15.175:0/1927232322 learned_addr learned my addr 172.21.15.175:0/1927232322 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:48.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e1d9b700 1 -- 172.21.15.175:0/1927232322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 msgr2=0x7fa5ec0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:48.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5ead9d700 1 --2- 172.21.15.175:0/1927232322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:48.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e1d9b700 1 --2- 172.21.15.175:0/1927232322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e9d9b700 1 -- 172.21.15.175:0/1927232322 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642635832 0 0) 0x7fa5ec1574b0 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e1d9b700 1 -- 172.21.15.175:0/1927232322 >> v1:172.21.15.175:6789/0 conn(0x7fa5ec0b1fc0 legacy=0x7fa5ec068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:48.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e1d9b700 1 -- 172.21.15.175:0/1927232322 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec0af4c0 con 0x7fa5ec0b2390 2023-10-10T19:00:48.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5ead9d700 1 --2- 172.21.15.175:0/1927232322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:48.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e1d9b700 1 --2- 172.21.15.175:0/1927232322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b2390 0x7fa5ec155fb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40098e0 tx=0x7fa5d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea507c90eb43416 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:48.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e9d9b700 1 -- 172.21.15.175:0/1927232322 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4009e80 con 0x7fa5ec0b2390 2023-10-10T19:00:48.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e9d9b700 1 -- 172.21.15.175:0/1927232322 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d400b8f0 con 0x7fa5ec0b2390 2023-10-10T19:00:48.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.147+0000 7fa5e9d9b700 1 -- 172.21.15.175:0/1927232322 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d400bc70 con 0x7fa5ec0b2390 2023-10-10T19:00:48.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.148+0000 7fa5f0da3700 1 -- 172.21.15.175:0/1927232322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b2390 msgr2=0x7fa5ec155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:48.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.148+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/1927232322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b2390 0x7fa5ec155fb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40098e0 tx=0x7fa5d40091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.148+0000 7fa5f0da3700 1 -- 172.21.15.175:0/1927232322 shutdown_connections 2023-10-10T19:00:48.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.148+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/1927232322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.148+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/1927232322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b2390 0x7fa5ec155fb0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.148+0000 7fa5f0da3700 1 -- 172.21.15.175:0/1927232322 >> 172.21.15.175:0/1927232322 conn(0x7fa5ec063230 msgr2=0x7fa5ec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:48.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.148+0000 7fa5f0da3700 1 -- 172.21.15.175:0/1927232322 shutdown_connections 2023-10-10T19:00:48.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.148+0000 7fa5f0da3700 1 -- 172.21.15.175:0/1927232322 wait complete. 2023-10-10T19:00:48.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.149+0000 7fa5f0da3700 1 Processor -- start 2023-10-10T19:00:48.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.149+0000 7fa5f0da3700 1 -- start start 2023-10-10T19:00:48.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5f0da3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec1658a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:48.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5f0da3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b1fc0 0x7fa5ec0c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:48.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5f0da3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5ec0b2390 0x7fa5ec14fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:48.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5f0da3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0c4000 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5f0da3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0cbf30 con 0x7fa5ec0b2390 2023-10-10T19:00:48.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5f0da3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec166550 con 0x7fa5ec0aff60 2023-10-10T19:00:48.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b1fc0 0x7fa5ec0c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:48.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b1fc0 0x7fa5ec0c4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36772/0 (socket says 172.21.15.175:36772) 2023-10-10T19:00:48.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 -- 172.21.15.175:0/2214093258 learned_addr learned my addr 172.21.15.175:0/2214093258 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:48.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5ea59c700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec1658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:48.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5ead9d700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5ec0b2390 0x7fa5ec14fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:48.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 -- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 msgr2=0x7fa5ec1658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:48.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec1658a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 -- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5ec0b2390 msgr2=0x7fa5ec14fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:48.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5ec0b2390 0x7fa5ec14fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 -- 172.21.15.175:0/2214093258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4009420 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.150+0000 7fa5e1d9b700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b1fc0 0x7fa5ec0c4e70 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4009fa0 tx=0x7fa5d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:48.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.151+0000 7fa5e2ffd700 1 -- 172.21.15.175:0/2214093258 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d400bc70 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.151+0000 7fa5e2ffd700 1 -- 172.21.15.175:0/2214093258 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d4011420 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.151+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec0cca10 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.151+0000 7fa5e2ffd700 1 -- 172.21.15.175:0/2214093258 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d40117c0 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.151+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec0ccea0 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.152+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec067570 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.152+0000 7fa5e2ffd700 1 -- 172.21.15.175:0/2214093258 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa5d4011bf0 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.152+0000 7fa5e2ffd700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa5cc066f30 0x7fa5cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:48.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.153+0000 7fa5e2ffd700 1 -- 172.21.15.175:0/2214093258 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fa5d4086190 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.153+0000 7fa5ea59c700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa5cc066f30 0x7fa5cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:48.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.153+0000 7fa5ea59c700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa5cc066f30 0x7fa5cc0693e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc007fd0 tx=0x7fa5dc007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:48.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.157+0000 7fa5e2ffd700 1 -- 172.21.15.175:0/2214093258 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5d40530e0 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.379+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5ec068da0 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.379+0000 7fa5e2ffd700 1 -- 172.21.15.175:0/2214093258 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5447 (secure 0 0 0) 0x7fa5d4056700 con 0x7fa5ec0b1fc0 2023-10-10T19:00:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:48.383 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:46.572587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:48.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa5cc066f30 msgr2=0x7fa5cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:48.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa5cc066f30 0x7fa5cc0693e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc007fd0 tx=0x7fa5dc007d30 comp rx=0 tx=0).stop 2023-10-10T19:00:48.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b1fc0 msgr2=0x7fa5ec0c4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:48.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b1fc0 0x7fa5ec0c4e70 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4009fa0 tx=0x7fa5d4017000 comp rx=0 tx=0).stop 2023-10-10T19:00:48.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 shutdown_connections 2023-10-10T19:00:48.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa5cc066f30 0x7fa5cc0693e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5ec0b2390 0x7fa5ec14fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5ec0aff60 0x7fa5ec1658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 --2- 172.21.15.175:0/2214093258 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5ec0b1fc0 0x7fa5ec0c4e70 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:48.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 >> 172.21.15.175:0/2214093258 conn(0x7fa5ec063230 msgr2=0x7fa5ec15a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:48.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 shutdown_connections 2023-10-10T19:00:48.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:48.381+0000 7fa5f0da3700 1 -- 172.21.15.175:0/2214093258 wait complete. 2023-10-10T19:00:48.391 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 116 2023-10-10T19:00:48.395 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:00:50.397 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:50.517 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:00:50.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:00:50.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.526+0000 7f1d4279d700 1 Processor -- start 2023-10-10T19:00:50.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.526+0000 7f1d4279d700 1 -- start start 2023-10-10T19:00:50.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d4279d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c08a100 0x7f1d3c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:50.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d4279d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c067fa0 con 0x7f1d3c0cd040 2023-10-10T19:00:50.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d4279d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c156b70 con 0x7f1d3c14fd90 2023-10-10T19:00:50.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d4279d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0c5600 con 0x7f1d3c08a100 2023-10-10T19:00:50.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d41f9c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1d3c0cd040 0x7f1d3c0684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38554/0 (socket says 172.21.15.175:38554) 2023-10-10T19:00:50.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d41f9c700 1 -- 172.21.15.175:0/964042664 learned_addr learned my addr 172.21.15.175:0/964042664 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:50.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d40f9a700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c08a100 0x7f1d3c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:50.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310632278 0 0) 0x7f1d3c067fa0 con 0x7f1d3c0cd040 2023-10-10T19:00:50.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d28003570 con 0x7f1d3c0cd040 2023-10-10T19:00:50.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923900797 0 0) 0x7f1d3c156b70 con 0x7f1d3c14fd90 2023-10-10T19:00:50.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d3c067fa0 con 0x7f1d3c14fd90 2023-10-10T19:00:50.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1108178385 0 0) 0x7f1d28003570 con 0x7f1d3c0cd040 2023-10-10T19:00:50.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c08a100 msgr2=0x7f1d3c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:50.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c08a100 0x7f1d3c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 >> v1:172.21.15.191:6789/0 conn(0x7f1d3c14fd90 legacy=0x7f1d3c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:50.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c150160 con 0x7f1d3c0cd040 2023-10-10T19:00:50.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f1d340017f0 con 0x7f1d3c0cd040 2023-10-10T19:00:50.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 >> v1:172.21.15.175:6789/0 conn(0x7f1d3c0cd040 legacy=0x7f1d3c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:50.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d28004c80 0x7f1d28005070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:50.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d280037b0 0x7f1d28003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:50.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c08a100 0x7f1d28004160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:50.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0c5600 con 0x7f1d28004c80 2023-10-10T19:00:50.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2800bb40 con 0x7f1d280037b0 2023-10-10T19:00:50.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.527+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2800bd00 con 0x7f1d3c08a100 2023-10-10T19:00:50.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d4179b700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d28004c80 0x7f1d28005070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:50.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d40f9a700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d280037b0 0x7f1d28003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:50.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d41f9c700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c08a100 0x7f1d28004160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:50.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d4179b700 1 -- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c08a100 msgr2=0x7f1d28004160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:50.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d4179b700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c08a100 0x7f1d28004160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d4179b700 1 -- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d280037b0 msgr2=0x7f1d28003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:50.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d4179b700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d280037b0 0x7f1d28003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d4179b700 1 -- 172.21.15.175:0/964042664 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d28003570 con 0x7f1d28004c80 2023-10-10T19:00:50.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d4179b700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d28004c80 0x7f1d28005070 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1d38001a10 tx=0x7f1d38007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb6abdc66c13aff server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:50.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800c5d0 con 0x7f1d28004c80 2023-10-10T19:00:50.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3800c730 con 0x7f1d28004c80 2023-10-10T19:00:50.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.528+0000 7f1d33fff700 1 -- 172.21.15.175:0/964042664 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800cad0 con 0x7f1d28004c80 2023-10-10T19:00:50.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.529+0000 7f1d4279d700 1 -- 172.21.15.175:0/964042664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d28004c80 msgr2=0x7f1d28005070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:50.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.529+0000 7f1d4279d700 1 --2- 172.21.15.175:0/964042664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d28004c80 0x7f1d28005070 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1d38001a10 tx=0x7f1d38007d30 comp rx=0 tx=0).stop 2023-10-10T19:00:50.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.529+0000 7f1d41f9c700 1 -- 172.21.15.175:0/964042664 reap_dead start 2023-10-10T19:00:50.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.529+0000 7f1d4279d700 1 -- 172.21.15.175:0/964042664 shutdown_connections 2023-10-10T19:00:50.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.529+0000 7f1d4279d700 1 -- 172.21.15.175:0/964042664 >> 172.21.15.175:0/964042664 conn(0x7f1d3c063d00 msgr2=0x7f1d3c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:50.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.529+0000 7f1d4279d700 1 -- 172.21.15.175:0/964042664 shutdown_connections 2023-10-10T19:00:50.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.529+0000 7f1d4279d700 1 -- 172.21.15.175:0/964042664 wait complete. 2023-10-10T19:00:50.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.530+0000 7f1d4279d700 1 Processor -- start 2023-10-10T19:00:50.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.530+0000 7f1d4279d700 1 -- start start 2023-10-10T19:00:50.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d4279d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d28004c80 0x7f1d3c149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:50.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d4279d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d3c0cbd10 0x7f1d3c14e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:50.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d4279d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d3c14ea60 0x7f1d3c152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:50.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d4279d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0cba30 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d4279d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0cb750 con 0x7f1d3c14ea60 2023-10-10T19:00:50.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d4279d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c150640 con 0x7f1d28004c80 2023-10-10T19:00:50.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d40f9a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d3c0cbd10 0x7f1d3c14e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:50.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d40f9a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d3c0cbd10 0x7f1d3c14e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36798/0 (socket says 172.21.15.175:36798) 2023-10-10T19:00:50.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d40f9a700 1 -- 172.21.15.175:0/674268743 learned_addr learned my addr 172.21.15.175:0/674268743 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:50.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d41f9c700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d3c14ea60 0x7f1d3c152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:50.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d4179b700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d28004c80 0x7f1d3c149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:50.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d40f9a700 1 -- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d28004c80 msgr2=0x7f1d3c149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:50.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d40f9a700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d28004c80 0x7f1d3c149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d40f9a700 1 -- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d3c14ea60 msgr2=0x7f1d3c152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:50.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d40f9a700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d3c14ea60 0x7f1d3c152d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.531+0000 7f1d40f9a700 1 -- 172.21.15.175:0/674268743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c151fe0 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.532+0000 7f1d40f9a700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d3c0cbd10 0x7f1d3c14e520 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c011d40 tx=0x7f1d2c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:50.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.532+0000 7f1d317fa700 1 -- 172.21.15.175:0/674268743 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00fa10 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.532+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c1522c0 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.532+0000 7f1d317fa700 1 -- 172.21.15.175:0/674268743 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c00fb70 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.532+0000 7f1d317fa700 1 -- 172.21.15.175:0/674268743 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c014440 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.532+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c14f3e0 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.533+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c151e40 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.533+0000 7f1d317fa700 1 -- 172.21.15.175:0/674268743 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1d2c014830 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.534+0000 7f1d317fa700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d20066f60 0x7f1d20069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:50.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.534+0000 7f1d317fa700 1 -- 172.21.15.175:0/674268743 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f1d2c018030 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.534+0000 7f1d4179b700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d20066f60 0x7f1d20069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:50.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.534+0000 7f1d4179b700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d20066f60 0x7f1d20069410 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1d38000c00 tx=0x7f1d38015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:50.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.538+0000 7f1d317fa700 1 -- 172.21.15.175:0/674268743 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d3c151e40 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.768+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d3c151e40 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.768+0000 7f1d317fa700 1 -- 172.21.15.175:0/674268743 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5447 (secure 0 0 0) 0x7f1d3c151e40 con 0x7f1d3c0cbd10 2023-10-10T19:00:50.770 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:50.770 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:46.572587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:50.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d20066f60 msgr2=0x7f1d20069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:50.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d20066f60 0x7f1d20069410 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1d38000c00 tx=0x7f1d38015000 comp rx=0 tx=0).stop 2023-10-10T19:00:50.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d3c0cbd10 msgr2=0x7f1d3c14e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:50.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d3c0cbd10 0x7f1d3c14e520 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c011d40 tx=0x7f1d2c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 shutdown_connections 2023-10-10T19:00:50.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d20066f60 0x7f1d20069410 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d3c14ea60 0x7f1d3c152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d28004c80 0x7f1d3c149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 --2- 172.21.15.175:0/674268743 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d3c0cbd10 0x7f1d3c14e520 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:50.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 >> 172.21.15.175:0/674268743 conn(0x7f1d3c063d00 msgr2=0x7f1d3c0c6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:50.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.770+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 shutdown_connections 2023-10-10T19:00:50.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:50.771+0000 7f1d4279d700 1 -- 172.21.15.175:0/674268743 wait complete. 2023-10-10T19:00:50.774 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 116 2023-10-10T19:00:50.784 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:00:52.786 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:52.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.913+0000 7f4a42f94700 1 Processor -- start 2023-10-10T19:00:52.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.913+0000 7f4a42f94700 1 -- start start 2023-10-10T19:00:52.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a42f94700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4a3c06b640 0x7f4a3c14cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:52.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a42f94700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c14d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:52.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a42f94700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3c1533e0 con 0x7f4a3c0c3330 2023-10-10T19:00:52.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a42f94700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3c14d820 con 0x7f4a3c06b640 2023-10-10T19:00:52.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a42f94700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c06e130 con 0x7f4a3c0c3700 2023-10-10T19:00:52.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a33fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4a3c06b640 0x7f4a3c14cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:52.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a41531700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c14d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:52.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a40d30700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4a3c0c3330 0x7f4a3c0c3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:49956/0 (socket says 172.21.15.175:49956) 2023-10-10T19:00:52.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a33fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4a3c06b640 0x7f4a3c14cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:49214/0 (socket says 172.21.15.175:49214) 2023-10-10T19:00:52.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a41531700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c14d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:51846/0 (socket says 172.21.15.175:51846) 2023-10-10T19:00:52.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.914+0000 7f4a40d30700 1 -- 172.21.15.175:0/362741339 learned_addr learned my addr 172.21.15.175:0/362741339 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:52.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a33fff700 1 -- 172.21.15.175:0/362741339 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 msgr2=0x7f4a3c14d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:52.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a33fff700 1 --2- 172.21.15.175:0/362741339 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c14d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:52.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a33fff700 1 -- 172.21.15.175:0/362741339 >> v1:172.21.15.191:6789/0 conn(0x7f4a3c0c3330 legacy=0x7f4a3c0c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:52.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a337fe700 1 -- 172.21.15.175:0/362741339 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154735157 0 0) 0x7f4a3c1533e0 con 0x7f4a3c0c3330 2023-10-10T19:00:52.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a33fff700 1 -- 172.21.15.175:0/362741339 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c06a970 con 0x7f4a3c06b640 2023-10-10T19:00:52.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a41531700 1 --2- 172.21.15.175:0/362741339 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c14d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:00:52.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a33fff700 1 --2- 172.21.15.175:0/362741339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c06b640 0x7f4a3c14cda0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0098e0 tx=0x7f4a2c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17a84b5275315744 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:52.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a337fe700 1 -- 172.21.15.175:0/362741339 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00b640 con 0x7f4a3c06b640 2023-10-10T19:00:52.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a337fe700 1 -- 172.21.15.175:0/362741339 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c00b7a0 con 0x7f4a3c06b640 2023-10-10T19:00:52.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.915+0000 7f4a337fe700 1 -- 172.21.15.175:0/362741339 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00bb40 con 0x7f4a3c06b640 2023-10-10T19:00:52.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.916+0000 7f4a42f94700 1 -- 172.21.15.175:0/362741339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c06b640 msgr2=0x7f4a3c14cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:52.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.916+0000 7f4a42f94700 1 --2- 172.21.15.175:0/362741339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c06b640 0x7f4a3c14cda0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0098e0 tx=0x7f4a2c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:00:52.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.916+0000 7f4a42f94700 1 -- 172.21.15.175:0/362741339 shutdown_connections 2023-10-10T19:00:52.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.916+0000 7f4a42f94700 1 --2- 172.21.15.175:0/362741339 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c14d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:52.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.916+0000 7f4a42f94700 1 --2- 172.21.15.175:0/362741339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c06b640 0x7f4a3c14cda0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:52.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.916+0000 7f4a42f94700 1 -- 172.21.15.175:0/362741339 >> 172.21.15.175:0/362741339 conn(0x7f4a3c068280 msgr2=0x7f4a3c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:52.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.916+0000 7f4a42f94700 1 -- 172.21.15.175:0/362741339 shutdown_connections 2023-10-10T19:00:52.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.916+0000 7f4a42f94700 1 -- 172.21.15.175:0/362741339 wait complete. 2023-10-10T19:00:52.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.917+0000 7f4a42f94700 1 Processor -- start 2023-10-10T19:00:52.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.917+0000 7f4a42f94700 1 -- start start 2023-10-10T19:00:52.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a42f94700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a3c06b640 0x7f4a3c143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:52.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a42f94700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c0c3330 0x7f4a3c14c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:52.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a42f94700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c1431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:52.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a42f94700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c1450b0 con 0x7f4a3c06b640 2023-10-10T19:00:52.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a42f94700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c1523f0 con 0x7f4a3c0c3330 2023-10-10T19:00:52.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a42f94700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c166b90 con 0x7f4a3c0c3700 2023-10-10T19:00:52.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a3c06b640 0x7f4a3c143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:52.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a3c06b640 0x7f4a3c143ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36806/0 (socket says 172.21.15.175:36806) 2023-10-10T19:00:52.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 -- 172.21.15.175:0/3806413801 learned_addr learned my addr 172.21.15.175:0/3806413801 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:52.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a33fff700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c0c3330 0x7f4a3c14c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:52.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a41531700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c1431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:52.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 -- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 msgr2=0x7f4a3c1431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:52.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c1431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:52.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 -- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c0c3330 msgr2=0x7f4a3c14c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:52.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c0c3330 0x7f4a3c14c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:52.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 -- 172.21.15.175:0/3806413801 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a34009a20 con 0x7f4a3c06b640 2023-10-10T19:00:52.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a40d30700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a3c06b640 0x7f4a3c143ed0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800c5e0 tx=0x7f4a38001030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:52.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.918+0000 7f4a30ff9700 1 -- 172.21.15.175:0/3806413801 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38005b80 con 0x7f4a3c06b640 2023-10-10T19:00:52.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.919+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c009420 con 0x7f4a3c06b640 2023-10-10T19:00:52.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.919+0000 7f4a30ff9700 1 -- 172.21.15.175:0/3806413801 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38005ce0 con 0x7f4a3c06b640 2023-10-10T19:00:52.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.919+0000 7f4a30ff9700 1 -- 172.21.15.175:0/3806413801 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380072e0 con 0x7f4a3c06b640 2023-10-10T19:00:52.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.919+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3c144410 con 0x7f4a3c06b640 2023-10-10T19:00:52.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.920+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3c144a00 con 0x7f4a3c06b640 2023-10-10T19:00:52.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.920+0000 7f4a30ff9700 1 -- 172.21.15.175:0/3806413801 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4a38007690 con 0x7f4a3c06b640 2023-10-10T19:00:52.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.920+0000 7f4a30ff9700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a28066f60 0x7f4a28069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:52.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.920+0000 7f4a30ff9700 1 -- 172.21.15.175:0/3806413801 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f4a38083b30 con 0x7f4a3c06b640 2023-10-10T19:00:52.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.920+0000 7f4a33fff700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a28066f60 0x7f4a28069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:52.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.921+0000 7f4a33fff700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a28066f60 0x7f4a28069410 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0bf3b0 tx=0x7f4a2c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:52.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:52.925+0000 7f4a30ff9700 1 -- 172.21.15.175:0/3806413801 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4a38050bf0 con 0x7f4a3c06b640 2023-10-10T19:00:53.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.146+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a3c067570 con 0x7f4a3c06b640 2023-10-10T19:00:53.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.147+0000 7f4a30ff9700 1 -- 172.21.15.175:0/3806413801 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5444 (secure 0 0 0) 0x7f4a38017020 con 0x7f4a3c06b640 2023-10-10T19:00:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:53.148 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:51.592137+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:53.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a28066f60 msgr2=0x7f4a28069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:53.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a28066f60 0x7f4a28069410 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0bf3b0 tx=0x7f4a2c017000 comp rx=0 tx=0).stop 2023-10-10T19:00:53.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a3c06b640 msgr2=0x7f4a3c143ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:53.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a3c06b640 0x7f4a3c143ed0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800c5e0 tx=0x7f4a38001030 comp rx=0 tx=0).stop 2023-10-10T19:00:53.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 shutdown_connections 2023-10-10T19:00:53.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a28066f60 0x7f4a28069410 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:53.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a3c0c3700 0x7f4a3c1431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:53.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a3c06b640 0x7f4a3c143ed0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:53.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 --2- 172.21.15.175:0/3806413801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a3c0c3330 0x7f4a3c14c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:53.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 >> 172.21.15.175:0/3806413801 conn(0x7f4a3c068280 msgr2=0x7f4a3c0c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:53.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 shutdown_connections 2023-10-10T19:00:53.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:53.150+0000 7f4a42f94700 1 -- 172.21.15.175:0/3806413801 wait complete. 2023-10-10T19:00:53.153 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 117 2023-10-10T19:00:53.162 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:00:55.163 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:55.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.291+0000 7f3815f09700 1 Processor -- start 2023-10-10T19:00:55.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.291+0000 7f3815f09700 1 -- start start 2023-10-10T19:00:55.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.291+0000 7f3815f09700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b2390 0x7f3810155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:55.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.291+0000 7f3815f09700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f38100aff60 0x7f38100b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:55.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.291+0000 7f3815f09700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38101574b0 con 0x7f38100b1fc0 2023-10-10T19:00:55.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.291+0000 7f3815f09700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3810150770 con 0x7f38100aff60 2023-10-10T19:00:55.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f3815f09700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38100afad0 con 0x7f38100b2390 2023-10-10T19:00:55.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f38100aff60 0x7f38100b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:55.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f38100aff60 0x7f38100b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39880/0 (socket says 172.21.15.175:39880) 2023-10-10T19:00:55.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380f7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f38100b1fc0 0x7f3810068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35042/0 (socket says 172.21.15.175:35042) 2023-10-10T19:00:55.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380ffff700 1 -- 172.21.15.175:0/101022625 learned_addr learned my addr 172.21.15.175:0/101022625 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:55.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380effd700 1 --2- 172.21.15.175:0/101022625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b2390 0x7f3810155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:55.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380ffff700 1 -- 172.21.15.175:0/101022625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b2390 msgr2=0x7f3810155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:00:55.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380ffff700 1 --2- 172.21.15.175:0/101022625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b2390 0x7f3810155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380ffff700 1 -- 172.21.15.175:0/101022625 >> v1:172.21.15.175:6789/0 conn(0x7f38100b1fc0 legacy=0x7f3810068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:55.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380e7fc700 1 -- 172.21.15.175:0/101022625 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489875090 0 0) 0x7f38101574b0 con 0x7f38100b1fc0 2023-10-10T19:00:55.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380ffff700 1 -- 172.21.15.175:0/101022625 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38100af4c0 con 0x7f38100aff60 2023-10-10T19:00:55.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380effd700 1 --2- 172.21.15.175:0/101022625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b2390 0x7f3810155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:00:55.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380ffff700 1 --2- 172.21.15.175:0/101022625 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100aff60 0x7f38100b03e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f38000099e0 tx=0x7f3800009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ba0a7f1d296e5a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:55.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380e7fc700 1 -- 172.21.15.175:0/101022625 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380000b8b0 con 0x7f38100aff60 2023-10-10T19:00:55.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380e7fc700 1 -- 172.21.15.175:0/101022625 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380000ba10 con 0x7f38100aff60 2023-10-10T19:00:55.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.292+0000 7f380e7fc700 1 -- 172.21.15.175:0/101022625 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380000bdb0 con 0x7f38100aff60 2023-10-10T19:00:55.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.293+0000 7f3815f09700 1 -- 172.21.15.175:0/101022625 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100aff60 msgr2=0x7f38100b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:55.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.293+0000 7f3815f09700 1 --2- 172.21.15.175:0/101022625 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100aff60 0x7f38100b03e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f38000099e0 tx=0x7f3800009250 comp rx=0 tx=0).stop 2023-10-10T19:00:55.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.293+0000 7f3815f09700 1 -- 172.21.15.175:0/101022625 shutdown_connections 2023-10-10T19:00:55.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.293+0000 7f3815f09700 1 --2- 172.21.15.175:0/101022625 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100aff60 0x7f38100b03e0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.293+0000 7f3815f09700 1 --2- 172.21.15.175:0/101022625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b2390 0x7f3810155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.293+0000 7f3815f09700 1 -- 172.21.15.175:0/101022625 >> 172.21.15.175:0/101022625 conn(0x7f3810063230 msgr2=0x7f3810063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:55.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.293+0000 7f3815f09700 1 -- 172.21.15.175:0/101022625 shutdown_connections 2023-10-10T19:00:55.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.293+0000 7f3815f09700 1 -- 172.21.15.175:0/101022625 wait complete. 2023-10-10T19:00:55.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.294+0000 7f3815f09700 1 Processor -- start 2023-10-10T19:00:55.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.294+0000 7f3815f09700 1 -- start start 2023-10-10T19:00:55.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f3815f09700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38100aff60 0x7f38100c4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:55.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f3815f09700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b1fc0 0x7f381014f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:55.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f3815f09700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100b2390 0x7f3810168d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:55.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f3815f09700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3810169c50 con 0x7f38100b2390 2023-10-10T19:00:55.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f3815f09700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38100caa20 con 0x7f38100aff60 2023-10-10T19:00:55.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f3815f09700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38100c5420 con 0x7f38100b1fc0 2023-10-10T19:00:55.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100b2390 0x7f3810168d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:55.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100b2390 0x7f3810168d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39894/0 (socket says 172.21.15.175:39894) 2023-10-10T19:00:55.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 -- 172.21.15.175:0/3437911672 learned_addr learned my addr 172.21.15.175:0/3437911672 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:55.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380f7fe700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38100aff60 0x7f38100c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:55.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380effd700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b1fc0 0x7f381014f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:55.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 -- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b1fc0 msgr2=0x7f381014f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:55.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b1fc0 0x7f381014f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 -- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38100aff60 msgr2=0x7f38100c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:55.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38100aff60 0x7f38100c4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 -- 172.21.15.175:0/3437911672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000094f0 con 0x7f38100b2390 2023-10-10T19:00:55.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f380ffff700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100b2390 0x7f3810168d30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f381006fd90 tx=0x7f3800017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:55.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.295+0000 7f38077fe700 1 -- 172.21.15.175:0/3437911672 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380001a630 con 0x7f38100b2390 2023-10-10T19:00:55.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.296+0000 7f38077fe700 1 -- 172.21.15.175:0/3437911672 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380001a790 con 0x7f38100b2390 2023-10-10T19:00:55.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.296+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3810153710 con 0x7f38100b2390 2023-10-10T19:00:55.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.296+0000 7f38077fe700 1 -- 172.21.15.175:0/3437911672 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380001ab30 con 0x7f38100b2390 2023-10-10T19:00:55.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.296+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3810169970 con 0x7f38100b2390 2023-10-10T19:00:55.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.297+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38100c55d0 con 0x7f38100b2390 2023-10-10T19:00:55.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.297+0000 7f38077fe700 1 -- 172.21.15.175:0/3437911672 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f38000253f0 con 0x7f38100b2390 2023-10-10T19:00:55.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.297+0000 7f38077fe700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f0066f60 0x7f37f0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:55.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.297+0000 7f38077fe700 1 -- 172.21.15.175:0/3437911672 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f38000865f0 con 0x7f38100b2390 2023-10-10T19:00:55.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.297+0000 7f380f7fe700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f0066f60 0x7f37f0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:55.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.298+0000 7f380f7fe700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f0066f60 0x7f37f0069410 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3808001560 tx=0x7f3808005d10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:55.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.302+0000 7f38077fe700 1 -- 172.21.15.175:0/3437911672 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f38100c55d0 con 0x7f38100b2390 2023-10-10T19:00:55.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.526+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38100c5590 con 0x7f38100b2390 2023-10-10T19:00:55.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.527+0000 7f38077fe700 1 -- 172.21.15.175:0/3437911672 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5444 (secure 0 0 0) 0x7f3800053540 con 0x7f38100b2390 2023-10-10T19:00:55.527 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:55.527 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:51.592137+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:55.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f0066f60 msgr2=0x7f37f0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:55.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f0066f60 0x7f37f0069410 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3808001560 tx=0x7f3808005d10 comp rx=0 tx=0).stop 2023-10-10T19:00:55.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100b2390 msgr2=0x7f3810168d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:55.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100b2390 0x7f3810168d30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f381006fd90 tx=0x7f3800017000 comp rx=0 tx=0).stop 2023-10-10T19:00:55.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 shutdown_connections 2023-10-10T19:00:55.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f37f0066f60 0x7f37f0069410 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38100b2390 0x7f3810168d30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38100aff60 0x7f38100c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 --2- 172.21.15.175:0/3437911672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38100b1fc0 0x7f381014f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:55.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 >> 172.21.15.175:0/3437911672 conn(0x7f3810063230 msgr2=0x7f38100c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:55.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 shutdown_connections 2023-10-10T19:00:55.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:55.529+0000 7f3815f09700 1 -- 172.21.15.175:0/3437911672 wait complete. 2023-10-10T19:00:55.532 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 117 2023-10-10T19:00:55.541 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:00:57.542 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:00:57.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.665+0000 7f908161f700 1 Processor -- start 2023-10-10T19:00:57.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.665+0000 7f908161f700 1 -- start start 2023-10-10T19:00:57.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.665+0000 7f908161f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c3210 0x7f907c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:57.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.665+0000 7f908161f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c0c5e40 con 0x7f907c0c9390 2023-10-10T19:00:57.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.665+0000 7f908161f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c0c9880 con 0x7f907c0c35e0 2023-10-10T19:00:57.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.665+0000 7f908161f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c0c4c50 con 0x7f907c0c3210 2023-10-10T19:00:57.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f907b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f907c0c9390 0x7f907c0c62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35054/0 (socket says 172.21.15.175:35054) 2023-10-10T19:00:57.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f907b7fe700 1 -- 172.21.15.175:0/2128874259 learned_addr learned my addr 172.21.15.175:0/2128874259 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:00:57.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f907affd700 1 --2- 172.21.15.175:0/2128874259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c3210 0x7f907c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:57.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f9079ffb700 1 -- 172.21.15.175:0/2128874259 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655144472 0 0) 0x7f907c0c5e40 con 0x7f907c0c9390 2023-10-10T19:00:57.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f9079ffb700 1 -- 172.21.15.175:0/2128874259 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9064003570 con 0x7f907c0c9390 2023-10-10T19:00:57.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f9079ffb700 1 -- 172.21.15.175:0/2128874259 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690305065 0 0) 0x7f907c0c9880 con 0x7f907c0c35e0 2023-10-10T19:00:57.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f9079ffb700 1 -- 172.21.15.175:0/2128874259 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f907c0c5e40 con 0x7f907c0c35e0 2023-10-10T19:00:57.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f907affd700 1 -- 172.21.15.175:0/2128874259 >> v1:172.21.15.191:6789/0 conn(0x7f907c0c35e0 legacy=0x7f907c0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:57.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f907affd700 1 -- 172.21.15.175:0/2128874259 >> v1:172.21.15.175:6789/0 conn(0x7f907c0c9390 legacy=0x7f907c0c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:57.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f907affd700 1 -- 172.21.15.175:0/2128874259 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c0c9b60 con 0x7f907c0c3210 2023-10-10T19:00:57.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f907affd700 1 --2- 172.21.15.175:0/2128874259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c3210 0x7f907c159fa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9074009e90 tx=0x7f9074009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500ef1c64a0bcf99 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:57.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f9079ffb700 1 -- 172.21.15.175:0/2128874259 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907400e070 con 0x7f907c0c3210 2023-10-10T19:00:57.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f9079ffb700 1 -- 172.21.15.175:0/2128874259 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907400bb80 con 0x7f907c0c3210 2023-10-10T19:00:57.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.666+0000 7f9079ffb700 1 -- 172.21.15.175:0/2128874259 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9074012430 con 0x7f907c0c3210 2023-10-10T19:00:57.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.667+0000 7f908161f700 1 -- 172.21.15.175:0/2128874259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c3210 msgr2=0x7f907c159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:57.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.667+0000 7f908161f700 1 --2- 172.21.15.175:0/2128874259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c3210 0x7f907c159fa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9074009e90 tx=0x7f9074009780 comp rx=0 tx=0).stop 2023-10-10T19:00:57.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.667+0000 7f908161f700 1 -- 172.21.15.175:0/2128874259 shutdown_connections 2023-10-10T19:00:57.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.667+0000 7f908161f700 1 --2- 172.21.15.175:0/2128874259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c3210 0x7f907c159fa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:57.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.667+0000 7f908161f700 1 -- 172.21.15.175:0/2128874259 >> 172.21.15.175:0/2128874259 conn(0x7f907c06c110 msgr2=0x7f907c06c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:57.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.667+0000 7f908161f700 1 -- 172.21.15.175:0/2128874259 shutdown_connections 2023-10-10T19:00:57.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.667+0000 7f908161f700 1 -- 172.21.15.175:0/2128874259 wait complete. 2023-10-10T19:00:57.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.668+0000 7f908161f700 1 Processor -- start 2023-10-10T19:00:57.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.668+0000 7f908161f700 1 -- start start 2023-10-10T19:00:57.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f908161f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907c0c3210 0x7f907c13e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:57.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f908161f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c35e0 0x7f907c150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:57.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f908161f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f907c0c9390 0x7f907c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:57.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f908161f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c0ca400 con 0x7f907c0c9390 2023-10-10T19:00:57.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f908161f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c0c5840 con 0x7f907c0c3210 2023-10-10T19:00:57.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f908161f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c154890 con 0x7f907c0c35e0 2023-10-10T19:00:57.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f907c0c9390 0x7f907c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:57.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f907c0c9390 0x7f907c1564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39902/0 (socket says 172.21.15.175:39902) 2023-10-10T19:00:57.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 -- 172.21.15.175:0/4022659864 learned_addr learned my addr 172.21.15.175:0/4022659864 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:00:57.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907a7fc700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c35e0 0x7f907c150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:57.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907affd700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907c0c3210 0x7f907c13e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:57.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 -- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c35e0 msgr2=0x7f907c150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:57.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c35e0 0x7f907c150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:57.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 -- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907c0c3210 msgr2=0x7f907c13e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:57.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907c0c3210 0x7f907c13e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:57.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 -- 172.21.15.175:0/4022659864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074009a20 con 0x7f907c0c9390 2023-10-10T19:00:57.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907a7fc700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c35e0 0x7f907c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:00:57.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.669+0000 7f907b7fe700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f907c0c9390 0x7f907c1564b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f906c00c590 tx=0x7f906c007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:57.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.670+0000 7f9062ffd700 1 -- 172.21.15.175:0/4022659864 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c005b30 con 0x7f907c0c9390 2023-10-10T19:00:57.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.670+0000 7f9062ffd700 1 -- 172.21.15.175:0/4022659864 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c005c90 con 0x7f907c0c9390 2023-10-10T19:00:57.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.670+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907c167090 con 0x7f907c0c9390 2023-10-10T19:00:57.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.670+0000 7f9062ffd700 1 -- 172.21.15.175:0/4022659864 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c007320 con 0x7f907c0c9390 2023-10-10T19:00:57.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.670+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c1667c0 con 0x7f907c0c9390 2023-10-10T19:00:57.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.671+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907c067570 con 0x7f907c0c9390 2023-10-10T19:00:57.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.671+0000 7f9062ffd700 1 -- 172.21.15.175:0/4022659864 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f906c01b3b0 con 0x7f907c0c9390 2023-10-10T19:00:57.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.671+0000 7f9062ffd700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9068066f60 0x7f9068069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:00:57.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.672+0000 7f9062ffd700 1 -- 172.21.15.175:0/4022659864 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f906c083c10 con 0x7f907c0c9390 2023-10-10T19:00:57.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.672+0000 7f907affd700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9068066f60 0x7f9068069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:00:57.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.672+0000 7f907affd700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9068066f60 0x7f9068069410 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f90740099f0 tx=0x7f9074019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:00:57.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.676+0000 7f9062ffd700 1 -- 172.21.15.175:0/4022659864 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f906c050b60 con 0x7f907c0c9390 2023-10-10T19:00:57.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.923+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f907c067d00 con 0x7f907c0c9390 2023-10-10T19:00:57.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.924+0000 7f9062ffd700 1 -- 172.21.15.175:0/4022659864 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5444 (secure 0 0 0) 0x7f906c017020 con 0x7f907c0c9390 2023-10-10T19:00:57.924 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:00:57.924 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:00:57.616867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:00:57.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9068066f60 msgr2=0x7f9068069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:57.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9068066f60 0x7f9068069410 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f90740099f0 tx=0x7f9074019000 comp rx=0 tx=0).stop 2023-10-10T19:00:57.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f907c0c9390 msgr2=0x7f907c1564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:00:57.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f907c0c9390 0x7f907c1564b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f906c00c590 tx=0x7f906c007d50 comp rx=0 tx=0).stop 2023-10-10T19:00:57.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 shutdown_connections 2023-10-10T19:00:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9068066f60 0x7f9068069410 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f907c0c9390 0x7f907c1564b0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907c0c3210 0x7f907c13e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 --2- 172.21.15.175:0/4022659864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907c0c35e0 0x7f907c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:00:57.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 >> 172.21.15.175:0/4022659864 conn(0x7f907c06c110 msgr2=0x7f907c14fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:00:57.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 shutdown_connections 2023-10-10T19:00:57.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:00:57.927+0000 7f908161f700 1 -- 172.21.15.175:0/4022659864 wait complete. 2023-10-10T19:00:57.930 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 118 2023-10-10T19:00:57.942 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=3, replacing gid=5196 2023-10-10T19:00:57.943 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2023-10-10T19:00:59.736 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:00:59.736+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T19:01:03.944 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-10T19:01:03.944 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-10T19:01:03.945 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-10T19:01:03.948 INFO:tasks.ceph.mds.a:Started 2023-10-10T19:01:03.948 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:01:04.022 INFO:tasks.ceph.mds.a.smithi175.stdout:starting mds.a at 2023-10-10T19:01:04.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.065+0000 7fad96c22700 1 Processor -- start 2023-10-10T19:01:04.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.065+0000 7fad96c22700 1 -- start start 2023-10-10T19:01:04.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad96c22700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fad900b1fc0 0x7fad90155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad96c22700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad901574b0 con 0x7fad900b2390 2023-10-10T19:01:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad96c22700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad900afad0 con 0x7fad900aff60 2023-10-10T19:01:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad96c22700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad90150770 con 0x7fad900b1fc0 2023-10-10T19:01:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad87fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fad900b2390 0x7fad90068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35082/0 (socket says 172.21.15.175:35082) 2023-10-10T19:01:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad87fff700 1 -- 172.21.15.175:0/2966220940 learned_addr learned my addr 172.21.15.175:0/2966220940 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:01:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad949be700 1 --2- 172.21.15.175:0/2966220940 >> v2:172.21.15.191:3300/0 conn(0x7fad900b1fc0 0x7fad90155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480114069 0 0) 0x7fad901574b0 con 0x7fad900b2390 2023-10-10T19:01:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.066+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad78003570 con 0x7fad900b2390 2023-10-10T19:01:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2821409363 0 0) 0x7fad78003570 con 0x7fad900b2390 2023-10-10T19:01:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 >> v2:172.21.15.191:3300/0 conn(0x7fad900b1fc0 msgr2=0x7fad90155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:01:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 --2- 172.21.15.175:0/2966220940 >> v2:172.21.15.191:3300/0 conn(0x7fad900b1fc0 0x7fad90155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 >> v1:172.21.15.191:6789/0 conn(0x7fad900aff60 legacy=0x7fad90159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad900af4c0 con 0x7fad900b2390 2023-10-10T19:01:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fad80001470 con 0x7fad900b2390 2023-10-10T19:01:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 >> v1:172.21.15.175:6789/0 conn(0x7fad900b2390 legacy=0x7fad90068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad78003a60 0x7fad78003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad78004810 0x7fad78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad7800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad90150770 con 0x7fad780051c0 2023-10-10T19:01:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7800c0d0 con 0x7fad78004810 2023-10-10T19:01:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7800c290 con 0x7fad78003a60 2023-10-10T19:01:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad951bf700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad7800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad949be700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad78003a60 0x7fad78003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad87fff700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad78004810 0x7fad78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:04.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad951bf700 1 -- 172.21.15.175:0/2966220940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad78003a60 msgr2=0x7fad78003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:01:04.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad951bf700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad78003a60 0x7fad78003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad951bf700 1 -- 172.21.15.175:0/2966220940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad78004810 msgr2=0x7fad78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:01:04.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad951bf700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad78004810 0x7fad78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad951bf700 1 -- 172.21.15.175:0/2966220940 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78003570 con 0x7fad780051c0 2023-10-10T19:01:04.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad951bf700 1 -- 172.21.15.175:0/2966220940 reap_dead start 2023-10-10T19:01:04.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad951bf700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad7800ba60 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fad880018b0 tx=0x7fad88007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5caa95c614263f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:01:04.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88012420 con 0x7fad780051c0 2023-10-10T19:01:04.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad880125a0 con 0x7fad780051c0 2023-10-10T19:01:04.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.067+0000 7fad877fe700 1 -- 172.21.15.175:0/2966220940 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88012960 con 0x7fad780051c0 2023-10-10T19:01:04.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.068+0000 7fad96c22700 1 -- 172.21.15.175:0/2966220940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 msgr2=0x7fad7800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:04.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.068+0000 7fad96c22700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad7800ba60 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fad880018b0 tx=0x7fad88007d30 comp rx=0 tx=0).stop 2023-10-10T19:01:04.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.068+0000 7fad96c22700 1 -- 172.21.15.175:0/2966220940 shutdown_connections 2023-10-10T19:01:04.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.068+0000 7fad96c22700 1 --2- 172.21.15.175:0/2966220940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad7800ba60 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.068+0000 7fad96c22700 1 -- 172.21.15.175:0/2966220940 >> 172.21.15.175:0/2966220940 conn(0x7fad90063230 msgr2=0x7fad90063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:01:04.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.068+0000 7fad96c22700 1 -- 172.21.15.175:0/2966220940 shutdown_connections 2023-10-10T19:01:04.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.068+0000 7fad96c22700 1 -- 172.21.15.175:0/2966220940 wait complete. 2023-10-10T19:01:04.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.069+0000 7fad96c22700 1 Processor -- start 2023-10-10T19:01:04.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.069+0000 7fad96c22700 1 -- start start 2023-10-10T19:01:04.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad96c22700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad90158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:04.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad96c22700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad900c6bb0 0x7fad90158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad96c22700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad900c63f0 0x7fad9014ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad96c22700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad900cbd70 con 0x7fad780051c0 2023-10-10T19:01:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad96c22700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad900c5da0 con 0x7fad900c63f0 2023-10-10T19:01:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad96c22700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad900c6080 con 0x7fad900c6bb0 2023-10-10T19:01:04.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad90158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:04.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad90158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39948/0 (socket says 172.21.15.175:39948) 2023-10-10T19:01:04.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 -- 172.21.15.175:0/1629286784 learned_addr learned my addr 172.21.15.175:0/1629286784 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:01:04.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad87fff700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad900c6bb0 0x7fad90158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:04.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad951bf700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad900c63f0 0x7fad9014ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:04.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 -- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad900c6bb0 msgr2=0x7fad90158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:04.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad900c6bb0 0x7fad90158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 -- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad900c63f0 msgr2=0x7fad9014ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:04.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad900c63f0 0x7fad9014ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 -- 172.21.15.175:0/1629286784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad90157c30 con 0x7fad780051c0 2023-10-10T19:01:04.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad951bf700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad900c63f0 0x7fad9014ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:01:04.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.070+0000 7fad949be700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad90158810 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009fd0 tx=0x7fad8c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:01:04.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.071+0000 7fad84ff9700 1 -- 172.21.15.175:0/1629286784 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00e040 con 0x7fad780051c0 2023-10-10T19:01:04.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.071+0000 7fad84ff9700 1 -- 172.21.15.175:0/1629286784 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c00f040 con 0x7fad780051c0 2023-10-10T19:01:04.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.071+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad90157f10 con 0x7fad780051c0 2023-10-10T19:01:04.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.071+0000 7fad84ff9700 1 -- 172.21.15.175:0/1629286784 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c015430 con 0x7fad780051c0 2023-10-10T19:01:04.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.071+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9014f200 con 0x7fad780051c0 2023-10-10T19:01:04.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.072+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad90067570 con 0x7fad780051c0 2023-10-10T19:01:04.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.072+0000 7fad84ff9700 1 -- 172.21.15.175:0/1629286784 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fad8c015860 con 0x7fad780051c0 2023-10-10T19:01:04.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.072+0000 7fad84ff9700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad7c066f60 0x7fad7c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:04.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.073+0000 7fad84ff9700 1 -- 172.21.15.175:0/1629286784 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fad8c085520 con 0x7fad780051c0 2023-10-10T19:01:04.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.073+0000 7fad87fff700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad7c066f60 0x7fad7c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:04.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.073+0000 7fad87fff700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad7c066f60 0x7fad7c069410 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fad80001c00 tx=0x7fad80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:01:04.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.077+0000 7fad84ff9700 1 -- 172.21.15.175:0/1629286784 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fad8c052470 con 0x7fad780051c0 2023-10-10T19:01:04.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.305+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad90158430 con 0x7fad780051c0 2023-10-10T19:01:04.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.305+0000 7fad84ff9700 1 -- 172.21.15.175:0/1629286784 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5446 (secure 0 0 0) 0x7fad8c055a90 con 0x7fad780051c0 2023-10-10T19:01:04.306 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:01:04.306 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:01:02.622495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:01:04.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad7c066f60 msgr2=0x7fad7c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:04.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad7c066f60 0x7fad7c069410 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fad80001c00 tx=0x7fad80009040 comp rx=0 tx=0).stop 2023-10-10T19:01:04.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 msgr2=0x7fad90158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:04.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad90158810 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009fd0 tx=0x7fad8c00b040 comp rx=0 tx=0).stop 2023-10-10T19:01:04.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 shutdown_connections 2023-10-10T19:01:04.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad7c066f60 0x7fad7c069410 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad900c63f0 0x7fad9014ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad780051c0 0x7fad90158810 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 --2- 172.21.15.175:0/1629286784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad900c6bb0 0x7fad90158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:04.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 >> 172.21.15.175:0/1629286784 conn(0x7fad90063230 msgr2=0x7fad90152430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:01:04.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 shutdown_connections 2023-10-10T19:01:04.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:04.308+0000 7fad96c22700 1 -- 172.21.15.175:0/1629286784 wait complete. 2023-10-10T19:01:04.311 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 119 2023-10-10T19:01:04.320 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-10T19:01:06.321 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:01:06.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.450+0000 7f19b29f2700 1 Processor -- start 2023-10-10T19:01:06.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.450+0000 7f19b29f2700 1 -- start start 2023-10-10T19:01:06.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.450+0000 7f19b29f2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f19ac0c12a0 0x7f19ac1577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:06.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.450+0000 7f19b29f2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f19ac0c4f20 0x7f19ac14d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:06.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.450+0000 7f19b29f2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac154aa0 con 0x7f19ac1573d0 2023-10-10T19:01:06.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.450+0000 7f19b29f2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ac0bfdf0 con 0x7f19ac0c12a0 2023-10-10T19:01:06.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.450+0000 7f19b29f2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ac063dc0 con 0x7f19ac0c4f20 2023-10-10T19:01:06.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19ab7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f19ac0c12a0 0x7f19ac1577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:06.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19ab7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f19ac0c12a0 0x7f19ac1577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50428/0 (socket says 172.21.15.175:50428) 2023-10-10T19:01:06.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19ab7fe700 1 -- 172.21.15.175:0/62399291 learned_addr learned my addr 172.21.15.175:0/62399291 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:01:06.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19b0f8f700 1 --2- 172.21.15.175:0/62399291 >> v2:172.21.15.191:3300/0 conn(0x7f19ac0c4f20 0x7f19ac14d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:06.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19ab7fe700 1 -- 172.21.15.175:0/62399291 >> v2:172.21.15.191:3300/0 conn(0x7f19ac0c4f20 msgr2=0x7f19ac14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:01:06.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19ab7fe700 1 --2- 172.21.15.175:0/62399291 >> v2:172.21.15.191:3300/0 conn(0x7f19ac0c4f20 0x7f19ac14d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19ab7fe700 1 -- 172.21.15.175:0/62399291 >> v1:172.21.15.191:6789/0 conn(0x7f19ac1573d0 legacy=0x7f19ac0bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:01:06.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19ab7fe700 1 -- 172.21.15.175:0/62399291 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac0c5320 con 0x7f19ac0c12a0 2023-10-10T19:01:06.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19ab7fe700 1 --2- 172.21.15.175:0/62399291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c12a0 0x7f19ac1577a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f19ac06e550 tx=0x7f19a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38d0dce54da12ae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:01:06.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19aaffd700 1 -- 172.21.15.175:0/62399291 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a000b860 con 0x7f19ac0c12a0 2023-10-10T19:01:06.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19aaffd700 1 -- 172.21.15.175:0/62399291 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a000b9c0 con 0x7f19ac0c12a0 2023-10-10T19:01:06.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.451+0000 7f19aaffd700 1 -- 172.21.15.175:0/62399291 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a000bd60 con 0x7f19ac0c12a0 2023-10-10T19:01:06.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.452+0000 7f19b29f2700 1 -- 172.21.15.175:0/62399291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c12a0 msgr2=0x7f19ac1577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:06.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.452+0000 7f19b29f2700 1 --2- 172.21.15.175:0/62399291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c12a0 0x7f19ac1577a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f19ac06e550 tx=0x7f19a0009250 comp rx=0 tx=0).stop 2023-10-10T19:01:06.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.452+0000 7f19b29f2700 1 -- 172.21.15.175:0/62399291 shutdown_connections 2023-10-10T19:01:06.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.452+0000 7f19b29f2700 1 --2- 172.21.15.175:0/62399291 >> v2:172.21.15.191:3300/0 conn(0x7f19ac0c4f20 0x7f19ac14d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.452+0000 7f19b29f2700 1 --2- 172.21.15.175:0/62399291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c12a0 0x7f19ac1577a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.452+0000 7f19b29f2700 1 -- 172.21.15.175:0/62399291 >> 172.21.15.175:0/62399291 conn(0x7f19ac06dc70 msgr2=0x7f19ac063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:01:06.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.452+0000 7f19b29f2700 1 -- 172.21.15.175:0/62399291 shutdown_connections 2023-10-10T19:01:06.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.452+0000 7f19b29f2700 1 -- 172.21.15.175:0/62399291 wait complete. 2023-10-10T19:01:06.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.453+0000 7f19b29f2700 1 Processor -- start 2023-10-10T19:01:06.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.453+0000 7f19b29f2700 1 -- start start 2023-10-10T19:01:06.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19b29f2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19ac0c12a0 0x7f19ac153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:06.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19b29f2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c4f20 0x7f19ac159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:06.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19b29f2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19ac1573d0 0x7f19ac147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:06.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19b29f2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac14b610 con 0x7f19ac0c4f20 2023-10-10T19:01:06.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19b29f2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac15b570 con 0x7f19ac1573d0 2023-10-10T19:01:06.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19b29f2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac153b70 con 0x7f19ac0c12a0 2023-10-10T19:01:06.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c4f20 0x7f19ac159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:06.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c4f20 0x7f19ac159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50442/0 (socket says 172.21.15.175:50442) 2023-10-10T19:01:06.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 -- 172.21.15.175:0/1867421702 learned_addr learned my addr 172.21.15.175:0/1867421702 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:01:06.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19b0f8f700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19ac1573d0 0x7f19ac147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:06.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19abfff700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19ac0c12a0 0x7f19ac153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:06.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 -- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19ac0c12a0 msgr2=0x7f19ac153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:06.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19ac0c12a0 0x7f19ac153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 -- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19ac1573d0 msgr2=0x7f19ac147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:06.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19ac1573d0 0x7f19ac147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 -- 172.21.15.175:0/1867421702 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00094f0 con 0x7f19ac0c4f20 2023-10-10T19:01:06.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.454+0000 7f19ab7fe700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c4f20 0x7f19ac159680 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f19ac06e6e0 tx=0x7f19a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:01:06.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.455+0000 7f1993fff700 1 -- 172.21.15.175:0/1867421702 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0024440 con 0x7f19ac0c4f20 2023-10-10T19:01:06.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.455+0000 7f1993fff700 1 -- 172.21.15.175:0/1867421702 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a00245a0 con 0x7f19ac0c4f20 2023-10-10T19:01:06.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.455+0000 7f1993fff700 1 -- 172.21.15.175:0/1867421702 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0024960 con 0x7f19ac0c4f20 2023-10-10T19:01:06.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.455+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac151cc0 con 0x7f19ac0c4f20 2023-10-10T19:01:06.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.455+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac152280 con 0x7f19ac0c4f20 2023-10-10T19:01:06.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.456+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ac147c10 con 0x7f19ac0c4f20 2023-10-10T19:01:06.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.456+0000 7f1993fff700 1 -- 172.21.15.175:0/1867421702 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f19a001a420 con 0x7f19ac0c4f20 2023-10-10T19:01:06.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.456+0000 7f1993fff700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1998066f00 0x7f19980693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:01:06.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.457+0000 7f1993fff700 1 -- 172.21.15.175:0/1867421702 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f19a0086380 con 0x7f19ac0c4f20 2023-10-10T19:01:06.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.457+0000 7f19abfff700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1998066f00 0x7f19980693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:01:06.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.457+0000 7f19abfff700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1998066f00 0x7f19980693b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f199c0039e0 tx=0x7f199c007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:01:06.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.461+0000 7f1993fff700 1 -- 172.21.15.175:0/1867421702 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f19a00532d0 con 0x7f19ac0c4f20 2023-10-10T19:01:06.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.685+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19ac0675a0 con 0x7f19ac0c4f20 2023-10-10T19:01:06.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.685+0000 7f1993fff700 1 -- 172.21.15.175:0/1867421702 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+6206 (secure 0 0 0) 0x7f19a00568f0 con 0x7f19ac0c4f20 2023-10-10T19:01:06.686 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:01:06.686 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:01:02.622495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:01:06.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.687+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1998066f00 msgr2=0x7f19980693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:06.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.687+0000 7f19b29f2700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1998066f00 0x7f19980693b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f199c0039e0 tx=0x7f199c007ec0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c4f20 msgr2=0x7f19ac159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:01:06.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c4f20 0x7f19ac159680 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f19ac06e6e0 tx=0x7f19a0017000 comp rx=0 tx=0).stop 2023-10-10T19:01:06.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 shutdown_connections 2023-10-10T19:01:06.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1998066f00 0x7f19980693b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19ac1573d0 0x7f19ac147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19ac0c12a0 0x7f19ac153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 --2- 172.21.15.175:0/1867421702 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19ac0c4f20 0x7f19ac159680 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:01:06.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 >> 172.21.15.175:0/1867421702 conn(0x7f19ac06dc70 msgr2=0x7f19ac14d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:01:06.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 shutdown_connections 2023-10-10T19:01:06.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:01:06.688+0000 7f19b29f2700 1 -- 172.21.15.175:0/1867421702 wait complete. 2023-10-10T19:01:06.691 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 120 2023-10-10T19:01:06.699 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-10T19:01:06.700 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2023-10-10T19:01:20.606 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:01:20.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:01:50.685 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:01:50.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:02:20.757 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:02:20.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:02:50.853 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:02:50.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:02:55.701 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:02:55.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.816+0000 7f01d745d700 1 Processor -- start 2023-10-10T19:02:55.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.816+0000 7f01d745d700 1 -- start start 2023-10-10T19:02:55.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d745d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f01d00adda0 0x7f01d00677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:55.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d745d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f01d00bd8a0 0x7f01d0067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:55.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d745d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d0152c70 con 0x7f01d00c4120 2023-10-10T19:02:55.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d745d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d01563a0 con 0x7f01d00bd8a0 2023-10-10T19:02:55.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d745d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d0068900 con 0x7f01d00adda0 2023-10-10T19:02:55.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d6c5c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f01d00bd8a0 0x7f01d0067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:55.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d6c5c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f01d00bd8a0 0x7f01d0067d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40808/0 (socket says 172.21.15.175:40808) 2023-10-10T19:02:55.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d6c5c700 1 -- 172.21.15.175:0/2411803789 learned_addr learned my addr 172.21.15.175:0/2411803789 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:02:55.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d645b700 1 --2- 172.21.15.175:0/2411803789 >> v2:172.21.15.191:3300/0 conn(0x7f01d00adda0 0x7f01d00677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:55.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d6c5c700 1 -- 172.21.15.175:0/2411803789 >> v2:172.21.15.191:3300/0 conn(0x7f01d00adda0 msgr2=0x7f01d00677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:02:55.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d6c5c700 1 --2- 172.21.15.175:0/2411803789 >> v2:172.21.15.191:3300/0 conn(0x7f01d00adda0 0x7f01d00677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:55.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d6c5c700 1 -- 172.21.15.175:0/2411803789 >> v1:172.21.15.191:6789/0 conn(0x7f01d00c4120 legacy=0x7f01d00bd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:55.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.817+0000 7f01d6c5c700 1 -- 172.21.15.175:0/2411803789 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0152f30 con 0x7f01d00bd8a0 2023-10-10T19:02:55.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d6c5c700 1 --2- 172.21.15.175:0/2411803789 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00bd8a0 0x7f01d0067d30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f01d0066f20 tx=0x7f01c800a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d9525443f626ca server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:55.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d5459700 1 -- 172.21.15.175:0/2411803789 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800a0d0 con 0x7f01d00bd8a0 2023-10-10T19:02:55.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d5459700 1 -- 172.21.15.175:0/2411803789 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c800a230 con 0x7f01d00bd8a0 2023-10-10T19:02:55.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d5459700 1 -- 172.21.15.175:0/2411803789 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800cd80 con 0x7f01d00bd8a0 2023-10-10T19:02:55.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d745d700 1 -- 172.21.15.175:0/2411803789 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00bd8a0 msgr2=0x7f01d0067d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:55.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d745d700 1 --2- 172.21.15.175:0/2411803789 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00bd8a0 0x7f01d0067d30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f01d0066f20 tx=0x7f01c800a950 comp rx=0 tx=0).stop 2023-10-10T19:02:55.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d745d700 1 -- 172.21.15.175:0/2411803789 shutdown_connections 2023-10-10T19:02:55.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d745d700 1 --2- 172.21.15.175:0/2411803789 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00bd8a0 0x7f01d0067d30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:55.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d745d700 1 --2- 172.21.15.175:0/2411803789 >> v2:172.21.15.191:3300/0 conn(0x7f01d00adda0 0x7f01d00677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:55.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d745d700 1 -- 172.21.15.175:0/2411803789 >> 172.21.15.175:0/2411803789 conn(0x7f01d00c8d00 msgr2=0x7f01d00c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:02:55.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d745d700 1 -- 172.21.15.175:0/2411803789 shutdown_connections 2023-10-10T19:02:55.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.818+0000 7f01d745d700 1 -- 172.21.15.175:0/2411803789 wait complete. 2023-10-10T19:02:55.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.819+0000 7f01d745d700 1 Processor -- start 2023-10-10T19:02:55.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.819+0000 7f01d745d700 1 -- start start 2023-10-10T19:02:55.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d745d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d00adda0 0x7f01d0146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:55.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d745d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d00bd8a0 0x7f01d0161860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:55.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d745d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00c4120 0x7f01d0161da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:55.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d745d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0165840 con 0x7f01d00c4120 2023-10-10T19:02:55.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d745d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d01449f0 con 0x7f01d00bd8a0 2023-10-10T19:02:55.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d745d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0144710 con 0x7f01d00adda0 2023-10-10T19:02:55.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d6c5c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00c4120 0x7f01d0161da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:55.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d6c5c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00c4120 0x7f01d0161da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40822/0 (socket says 172.21.15.175:40822) 2023-10-10T19:02:55.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d6c5c700 1 -- 172.21.15.175:0/1652079966 learned_addr learned my addr 172.21.15.175:0/1652079966 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:02:55.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d5c5a700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d00bd8a0 0x7f01d0161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:55.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d645b700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d00adda0 0x7f01d0146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:55.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d6c5c700 1 -- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d00adda0 msgr2=0x7f01d0146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:55.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d6c5c700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d00adda0 0x7f01d0146520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:55.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d6c5c700 1 -- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d00bd8a0 msgr2=0x7f01d0161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:55.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d6c5c700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d00bd8a0 0x7f01d0161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:55.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d6c5c700 1 -- 172.21.15.175:0/1652079966 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c800abf0 con 0x7f01d00c4120 2023-10-10T19:02:55.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.820+0000 7f01d645b700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d00adda0 0x7f01d0146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:02:55.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.821+0000 7f01d6c5c700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00c4120 0x7f01d0161da0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f01d006aa90 tx=0x7f01c8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:55.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.821+0000 7f01c67fc700 1 -- 172.21.15.175:0/1652079966 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c80032b0 con 0x7f01d00c4120 2023-10-10T19:02:55.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.821+0000 7f01c67fc700 1 -- 172.21.15.175:0/1652079966 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c800a3a0 con 0x7f01d00c4120 2023-10-10T19:02:55.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.821+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d0164f10 con 0x7f01d00c4120 2023-10-10T19:02:55.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.821+0000 7f01c67fc700 1 -- 172.21.15.175:0/1652079966 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8013860 con 0x7f01d00c4120 2023-10-10T19:02:55.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.821+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d01654d0 con 0x7f01d00c4120 2023-10-10T19:02:55.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.822+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d01659f0 con 0x7f01d00c4120 2023-10-10T19:02:55.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.822+0000 7f01c67fc700 1 -- 172.21.15.175:0/1652079966 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f01c801a070 con 0x7f01d00c4120 2023-10-10T19:02:55.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.822+0000 7f01c67fc700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01b4066f30 0x7f01b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:55.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.822+0000 7f01c67fc700 1 -- 172.21.15.175:0/1652079966 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f01c8086060 con 0x7f01d00c4120 2023-10-10T19:02:55.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.823+0000 7f01d645b700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01b4066f30 0x7f01b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:55.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.823+0000 7f01d645b700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01b4066f30 0x7f01b40693e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00bf90 tx=0x7f01cc00b430 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:55.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:55.827+0000 7f01c67fc700 1 -- 172.21.15.175:0/1652079966 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01c8052fb0 con 0x7f01d00c4120 2023-10-10T19:02:56.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.099+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01d0067570 con 0x7f01d00c4120 2023-10-10T19:02:56.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.100+0000 7f01c67fc700 1 -- 172.21.15.175:0/1652079966 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7f01c8008370 con 0x7f01d00c4120 2023-10-10T19:02:56.101 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:02:56.102 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:02:56.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01b4066f30 msgr2=0x7f01b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01b4066f30 0x7f01b40693e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00bf90 tx=0x7f01cc00b430 comp rx=0 tx=0).stop 2023-10-10T19:02:56.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00c4120 msgr2=0x7f01d0161da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00c4120 0x7f01d0161da0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f01d006aa90 tx=0x7f01c8018000 comp rx=0 tx=0).stop 2023-10-10T19:02:56.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 shutdown_connections 2023-10-10T19:02:56.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f01b4066f30 0x7f01b40693e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01d00c4120 0x7f01d0161da0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d00adda0 0x7f01d0146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 --2- 172.21.15.175:0/1652079966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d00bd8a0 0x7f01d0161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 >> 172.21.15.175:0/1652079966 conn(0x7f01d00c8d00 msgr2=0x7f01d00bd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:02:56.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 shutdown_connections 2023-10-10T19:02:56.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.102+0000 7f01d745d700 1 -- 172.21.15.175:0/1652079966 wait complete. 2023-10-10T19:02:56.106 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:02:56.115 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2023-10-10T19:02:56.116 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-10T19:02:56.116 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:02:56.126 INFO:tasks.ceph.mds.f:Stopped 2023-10-10T19:02:56.127 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:02:56.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.239+0000 7fbf56de4700 1 Processor -- start 2023-10-10T19:02:56.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.239+0000 7fbf56de4700 1 -- start start 2023-10-10T19:02:56.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.240+0000 7fbf56de4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbf500b1fc0 0x7fbf50155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:56.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.240+0000 7fbf56de4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf501574b0 con 0x7fbf500aff60 2023-10-10T19:02:56.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.240+0000 7fbf56de4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf500afad0 con 0x7fbf500b2390 2023-10-10T19:02:56.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.240+0000 7fbf56de4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf50150770 con 0x7fbf500b1fc0 2023-10-10T19:02:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.240+0000 7fbf55381700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbf500aff60 0x7fbf50159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:34270/0 (socket says 172.21.15.175:34270) 2023-10-10T19:02:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.240+0000 7fbf55381700 1 -- 172.21.15.175:0/106407560 learned_addr learned my addr 172.21.15.175:0/106407560 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:02:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf54b80700 1 --2- 172.21.15.175:0/106407560 >> v2:172.21.15.191:3300/0 conn(0x7fbf500b1fc0 0x7fbf50155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf4f7fe700 1 -- 172.21.15.175:0/106407560 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276850488 0 0) 0x7fbf501574b0 con 0x7fbf500aff60 2023-10-10T19:02:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf4f7fe700 1 -- 172.21.15.175:0/106407560 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf3c003570 con 0x7fbf500aff60 2023-10-10T19:02:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf4f7fe700 1 -- 172.21.15.175:0/106407560 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742681016 0 0) 0x7fbf500afad0 con 0x7fbf500b2390 2023-10-10T19:02:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf4f7fe700 1 -- 172.21.15.175:0/106407560 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf501574b0 con 0x7fbf500b2390 2023-10-10T19:02:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf4f7fe700 1 -- 172.21.15.175:0/106407560 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094101758 0 0) 0x7fbf3c003570 con 0x7fbf500aff60 2023-10-10T19:02:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf54b80700 1 -- 172.21.15.175:0/106407560 >> v1:172.21.15.191:6789/0 conn(0x7fbf500b2390 legacy=0x7fbf50068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf54b80700 1 -- 172.21.15.175:0/106407560 >> v1:172.21.15.175:6789/0 conn(0x7fbf500aff60 legacy=0x7fbf50159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf54b80700 1 -- 172.21.15.175:0/106407560 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf500af4c0 con 0x7fbf500b1fc0 2023-10-10T19:02:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf54b80700 1 --2- 172.21.15.175:0/106407560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b1fc0 0x7fbf50155fb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009e60 tx=0x7fbf40009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1022b8c7ee981404 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf4f7fe700 1 -- 172.21.15.175:0/106407560 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4000e070 con 0x7fbf500b1fc0 2023-10-10T19:02:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf4f7fe700 1 -- 172.21.15.175:0/106407560 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4000b830 con 0x7fbf500b1fc0 2023-10-10T19:02:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.241+0000 7fbf4f7fe700 1 -- 172.21.15.175:0/106407560 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48001470 con 0x7fbf500b1fc0 2023-10-10T19:02:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.242+0000 7fbf56de4700 1 -- 172.21.15.175:0/106407560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b1fc0 msgr2=0x7fbf50155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.242+0000 7fbf56de4700 1 --2- 172.21.15.175:0/106407560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b1fc0 0x7fbf50155fb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009e60 tx=0x7fbf40009780 comp rx=0 tx=0).stop 2023-10-10T19:02:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.242+0000 7fbf56de4700 1 -- 172.21.15.175:0/106407560 shutdown_connections 2023-10-10T19:02:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.242+0000 7fbf56de4700 1 --2- 172.21.15.175:0/106407560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b1fc0 0x7fbf50155fb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.242+0000 7fbf56de4700 1 -- 172.21.15.175:0/106407560 >> 172.21.15.175:0/106407560 conn(0x7fbf50063230 msgr2=0x7fbf50063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:02:56.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.242+0000 7fbf56de4700 1 -- 172.21.15.175:0/106407560 shutdown_connections 2023-10-10T19:02:56.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.242+0000 7fbf56de4700 1 -- 172.21.15.175:0/106407560 wait complete. 2023-10-10T19:02:56.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.243+0000 7fbf56de4700 1 Processor -- start 2023-10-10T19:02:56.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.243+0000 7fbf56de4700 1 -- start start 2023-10-10T19:02:56.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf56de4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf500aff60 0x7fbf500c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:56.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf56de4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf500b1fc0 0x7fbf50169140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:56.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf56de4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b2390 0x7fbf500c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf56de4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50169a30 con 0x7fbf500aff60 2023-10-10T19:02:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf56de4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50154170 con 0x7fbf500b2390 2023-10-10T19:02:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf56de4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50169cd0 con 0x7fbf500b1fc0 2023-10-10T19:02:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf500aff60 0x7fbf500c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf500aff60 0x7fbf500c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40828/0 (socket says 172.21.15.175:40828) 2023-10-10T19:02:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 -- 172.21.15.175:0/3940456706 learned_addr learned my addr 172.21.15.175:0/3940456706 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:02:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf55381700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b2390 0x7fbf500c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf4ffff700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf500b1fc0 0x7fbf50169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 -- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf500b1fc0 msgr2=0x7fbf50169140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf500b1fc0 0x7fbf50169140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 -- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b2390 msgr2=0x7fbf500c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b2390 0x7fbf500c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 -- 172.21.15.175:0/3940456706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40009a20 con 0x7fbf500aff60 2023-10-10T19:02:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf4ffff700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf500b1fc0 0x7fbf50169140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:02:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf54b80700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf500aff60 0x7fbf500c3fa0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbf5006fd90 tx=0x7fbf40017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf4cff9700 1 -- 172.21.15.175:0/3940456706 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40012be0 con 0x7fbf500aff60 2023-10-10T19:02:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf4cff9700 1 -- 172.21.15.175:0/3940456706 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf40012d60 con 0x7fbf500aff60 2023-10-10T19:02:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf500c4ea0 con 0x7fbf500aff60 2023-10-10T19:02:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf4cff9700 1 -- 172.21.15.175:0/3940456706 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4001e6a0 con 0x7fbf500aff60 2023-10-10T19:02:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.244+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf50153e20 con 0x7fbf500aff60 2023-10-10T19:02:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.245+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf500713a0 con 0x7fbf500aff60 2023-10-10T19:02:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.246+0000 7fbf4cff9700 1 -- 172.21.15.175:0/3940456706 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbf4001ea50 con 0x7fbf500aff60 2023-10-10T19:02:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.246+0000 7fbf4cff9700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f60 0x7fbf30069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.246+0000 7fbf4cff9700 1 -- 172.21.15.175:0/3940456706 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fbf400868e0 con 0x7fbf500aff60 2023-10-10T19:02:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.246+0000 7fbf4ffff700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f60 0x7fbf30069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:56.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.247+0000 7fbf4ffff700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f60 0x7fbf30069410 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbf50157130 tx=0x7fbf38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:56.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.250+0000 7fbf4cff9700 1 -- 172.21.15.175:0/3940456706 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf500713a0 con 0x7fbf500aff60 2023-10-10T19:02:56.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.472+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf500713a0 con 0x7fbf500aff60 2023-10-10T19:02:56.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.473+0000 7fbf4cff9700 1 -- 172.21.15.175:0/3940456706 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7fbf500713a0 con 0x7fbf500aff60 2023-10-10T19:02:56.475 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:02:56.475 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:02:56.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f60 msgr2=0x7fbf30069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f60 0x7fbf30069410 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbf50157130 tx=0x7fbf38009040 comp rx=0 tx=0).stop 2023-10-10T19:02:56.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf500aff60 msgr2=0x7fbf500c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:56.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf500aff60 0x7fbf500c3fa0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbf5006fd90 tx=0x7fbf40017000 comp rx=0 tx=0).stop 2023-10-10T19:02:56.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 shutdown_connections 2023-10-10T19:02:56.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbf30066f60 0x7fbf30069410 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf500b2390 0x7fbf500c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf500aff60 0x7fbf500c3fa0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 --2- 172.21.15.175:0/3940456706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf500b1fc0 0x7fbf50169140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:56.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 >> 172.21.15.175:0/3940456706 conn(0x7fbf50063230 msgr2=0x7fbf50159080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:02:56.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 shutdown_connections 2023-10-10T19:02:56.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:56.475+0000 7fbf56de4700 1 -- 172.21.15.175:0/3940456706 wait complete. 2023-10-10T19:02:56.479 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:02:56.488 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:02:58.489 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:02:58.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.605+0000 7f3114886700 1 Processor -- start 2023-10-10T19:02:58.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.605+0000 7f3114886700 1 -- start start 2023-10-10T19:02:58.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f3114886700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f311006ca50 0x7f31101501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:58.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f3114886700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311006de80 con 0x7f311006c190 2023-10-10T19:02:58.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f3114886700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3110067ee0 con 0x7f3110069dc0 2023-10-10T19:02:58.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f3114886700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3110159e90 con 0x7f311006ca50 2023-10-10T19:02:58.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310ed9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f311006c190 0x7f311014fad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:34272/0 (socket says 172.21.15.175:34272) 2023-10-10T19:02:58.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310ed9d700 1 -- 172.21.15.175:0/1223326579 learned_addr learned my addr 172.21.15.175:0/1223326579 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:02:58.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127834854 0 0) 0x7f311006de80 con 0x7f311006c190 2023-10-10T19:02:58.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f8003570 con 0x7f311006c190 2023-10-10T19:02:58.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310f59e700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f311006ca50 0x7f31101501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:58.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1387410207 0 0) 0x7f30f8003570 con 0x7f311006c190 2023-10-10T19:02:58.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f311006ca50 msgr2=0x7f31101501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:02:58.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f311006ca50 0x7f31101501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 >> v1:172.21.15.191:6789/0 conn(0x7f3110069dc0 legacy=0x7f311015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:58.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311015a170 con 0x7f311006c190 2023-10-10T19:02:58.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f3100001470 con 0x7f311006c190 2023-10-10T19:02:58.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 >> v1:172.21.15.175:6789/0 conn(0x7f311006c190 legacy=0x7f311014fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:58.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30f8003a90 0x7f30f8003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:58.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30f8004810 0x7f30f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:58.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f311006ca50 0x7f30f8005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:58.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110159e90 con 0x7f30f8004810 2023-10-10T19:02:58.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f800bbf0 con 0x7f30f8003a90 2023-10-10T19:02:58.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f800bd80 con 0x7f311006ca50 2023-10-10T19:02:58.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.606+0000 7f310e59c700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30f8004810 0x7f30f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:58.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310f59e700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f311006ca50 0x7f30f8005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:58.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310ed9d700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30f8003a90 0x7f30f8003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:58.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310e59c700 1 -- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f311006ca50 msgr2=0x7f30f8005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:02:58.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310e59c700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f311006ca50 0x7f30f8005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310e59c700 1 -- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30f8003a90 msgr2=0x7f30f8003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:02:58.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310e59c700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30f8003a90 0x7f30f8003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310e59c700 1 -- 172.21.15.175:0/1223326579 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f8003570 con 0x7f30f8004810 2023-10-10T19:02:58.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310e59c700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30f8004810 0x7f30f8004c80 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f31040018b0 tx=0x7f3104007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91269e8fb84fa469 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:58.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104012420 con 0x7f30f8004810 2023-10-10T19:02:58.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104012580 con 0x7f30f8004810 2023-10-10T19:02:58.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310dd9b700 1 -- 172.21.15.175:0/1223326579 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104012920 con 0x7f30f8004810 2023-10-10T19:02:58.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f3114886700 1 -- 172.21.15.175:0/1223326579 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30f8004810 msgr2=0x7f30f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f3114886700 1 --2- 172.21.15.175:0/1223326579 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30f8004810 0x7f30f8004c80 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f31040018b0 tx=0x7f3104007d30 comp rx=0 tx=0).stop 2023-10-10T19:02:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f310f59e700 1 -- 172.21.15.175:0/1223326579 reap_dead start 2023-10-10T19:02:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f3114886700 1 -- 172.21.15.175:0/1223326579 shutdown_connections 2023-10-10T19:02:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f3114886700 1 -- 172.21.15.175:0/1223326579 >> 172.21.15.175:0/1223326579 conn(0x7f311006a3c0 msgr2=0x7f311006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:02:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.607+0000 7f3114886700 1 -- 172.21.15.175:0/1223326579 shutdown_connections 2023-10-10T19:02:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.608+0000 7f3114886700 1 -- 172.21.15.175:0/1223326579 wait complete. 2023-10-10T19:02:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f3114886700 1 Processor -- start 2023-10-10T19:02:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f3114886700 1 -- start start 2023-10-10T19:02:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f3114886700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30f8004810 0x7f31100c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f3114886700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f311014a100 0x7f311014a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f3114886700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3110141b00 0x7f3110141f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f3114886700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110144260 con 0x7f311014a100 2023-10-10T19:02:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f3114886700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31101534a0 con 0x7f3110141b00 2023-10-10T19:02:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f3114886700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3110152ec0 con 0x7f30f8004810 2023-10-10T19:02:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.609+0000 7f310e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f311014a100 0x7f311014a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f311014a100 0x7f311014a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40836/0 (socket says 172.21.15.175:40836) 2023-10-10T19:02:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310e59c700 1 -- 172.21.15.175:0/3453640760 learned_addr learned my addr 172.21.15.175:0/3453640760 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:02:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310ed9d700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30f8004810 0x7f31100c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310f59e700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3110141b00 0x7f3110141f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310e59c700 1 -- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30f8004810 msgr2=0x7f31100c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:58.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310e59c700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30f8004810 0x7f31100c83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310e59c700 1 -- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3110141b00 msgr2=0x7f3110141f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:58.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310e59c700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3110141b00 0x7f3110141f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310e59c700 1 -- 172.21.15.175:0/3453640760 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311014aab0 con 0x7f311014a100 2023-10-10T19:02:58.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f310e59c700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f311014a100 0x7f311014a570 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f310400cee0 tx=0x7f3104015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:58.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f30f6ffd700 1 -- 172.21.15.175:0/3453640760 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104012420 con 0x7f311014a100 2023-10-10T19:02:58.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f30f6ffd700 1 -- 172.21.15.175:0/3453640760 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104001200 con 0x7f311014a100 2023-10-10T19:02:58.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3110144490 con 0x7f311014a100 2023-10-10T19:02:58.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f30f6ffd700 1 -- 172.21.15.175:0/3453640760 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040126f0 con 0x7f311014a100 2023-10-10T19:02:58.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.610+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3110166420 con 0x7f311014a100 2023-10-10T19:02:58.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.611+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3110149e20 con 0x7f311014a100 2023-10-10T19:02:58.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.611+0000 7f30f6ffd700 1 -- 172.21.15.175:0/3453640760 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3104006820 con 0x7f311014a100 2023-10-10T19:02:58.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.612+0000 7f30f6ffd700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30fc066f60 0x7f30fc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:02:58.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.612+0000 7f30f6ffd700 1 -- 172.21.15.175:0/3453640760 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f3104085c80 con 0x7f311014a100 2023-10-10T19:02:58.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.612+0000 7f310ed9d700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30fc066f60 0x7f30fc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:02:58.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.612+0000 7f310ed9d700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30fc066f60 0x7f30fc069410 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3100001050 tx=0x7f3100001d80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:02:58.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.616+0000 7f30f6ffd700 1 -- 172.21.15.175:0/3453640760 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3110149e20 con 0x7f311014a100 2023-10-10T19:02:58.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.843+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3110149e20 con 0x7f311014a100 2023-10-10T19:02:58.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.843+0000 7f30f6ffd700 1 -- 172.21.15.175:0/3453640760 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7f3110149e20 con 0x7f311014a100 2023-10-10T19:02:58.845 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:02:58.845 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:02:58.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30fc066f60 msgr2=0x7f30fc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:58.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30fc066f60 0x7f30fc069410 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3100001050 tx=0x7f3100001d80 comp rx=0 tx=0).stop 2023-10-10T19:02:58.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f311014a100 msgr2=0x7f311014a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:02:58.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f311014a100 0x7f311014a570 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f310400cee0 tx=0x7f3104015000 comp rx=0 tx=0).stop 2023-10-10T19:02:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 shutdown_connections 2023-10-10T19:02:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30fc066f60 0x7f30fc069410 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3110141b00 0x7f3110141f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30f8004810 0x7f31100c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 --2- 172.21.15.175:0/3453640760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f311014a100 0x7f311014a570 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:02:58.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 >> 172.21.15.175:0/3453640760 conn(0x7f311006a3c0 msgr2=0x7f31100c4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:02:58.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 shutdown_connections 2023-10-10T19:02:58.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:02:58.845+0000 7f3114886700 1 -- 172.21.15.175:0/3453640760 wait complete. 2023-10-10T19:02:58.850 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:02:58.858 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:03:00.859 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:00.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.982+0000 7f33500bc700 1 Processor -- start 2023-10-10T19:03:00.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.982+0000 7f33500bc700 1 -- start start 2023-10-10T19:03:00.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f33500bc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f33480b1fc0 0x7f3348155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:00.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f33500bc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f33480b2390 0x7f3348068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:00.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f33500bc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33481574b0 con 0x7f3348068bd0 2023-10-10T19:03:00.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f33500bc700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3348150770 con 0x7f33480b2390 2023-10-10T19:03:00.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f33500bc700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33480afad0 con 0x7f33480b1fc0 2023-10-10T19:03:00.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f334d657700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f33480b2390 0x7f3348068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:00.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f334d657700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f33480b2390 0x7f3348068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40850/0 (socket says 172.21.15.175:40850) 2023-10-10T19:03:00.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f334d657700 1 -- 172.21.15.175:0/217289128 learned_addr learned my addr 172.21.15.175:0/217289128 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:00.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.983+0000 7f334de58700 1 --2- 172.21.15.175:0/217289128 >> v2:172.21.15.191:3300/0 conn(0x7f33480b1fc0 0x7f3348155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:00.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334d657700 1 -- 172.21.15.175:0/217289128 >> v2:172.21.15.191:3300/0 conn(0x7f33480b1fc0 msgr2=0x7f3348155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:00.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334d657700 1 --2- 172.21.15.175:0/217289128 >> v2:172.21.15.191:3300/0 conn(0x7f33480b1fc0 0x7f3348155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:00.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334d657700 1 -- 172.21.15.175:0/217289128 >> v1:172.21.15.191:6789/0 conn(0x7f3348068bd0 legacy=0x7f33480aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:00.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334d657700 1 -- 172.21.15.175:0/217289128 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33480af4c0 con 0x7f33480b2390 2023-10-10T19:03:00.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334de58700 1 --2- 172.21.15.175:0/217289128 >> v2:172.21.15.191:3300/0 conn(0x7f33480b1fc0 0x7f3348155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:00.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334d657700 1 --2- 172.21.15.175:0/217289128 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33480b2390 0x7f3348068690 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f33480b28c0 tx=0x7f3338009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e43a6831dc42fae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:00.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334ce56700 1 -- 172.21.15.175:0/217289128 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800b8f0 con 0x7f33480b2390 2023-10-10T19:03:00.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334ce56700 1 -- 172.21.15.175:0/217289128 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333800ba50 con 0x7f33480b2390 2023-10-10T19:03:00.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f334ce56700 1 -- 172.21.15.175:0/217289128 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800bdf0 con 0x7f33480b2390 2023-10-10T19:03:00.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f33500bc700 1 -- 172.21.15.175:0/217289128 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33480b2390 msgr2=0x7f3348068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:00.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f33500bc700 1 --2- 172.21.15.175:0/217289128 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33480b2390 0x7f3348068690 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f33480b28c0 tx=0x7f3338009250 comp rx=0 tx=0).stop 2023-10-10T19:03:00.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f33500bc700 1 -- 172.21.15.175:0/217289128 shutdown_connections 2023-10-10T19:03:00.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f33500bc700 1 --2- 172.21.15.175:0/217289128 >> v2:172.21.15.191:3300/0 conn(0x7f33480b1fc0 0x7f3348155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:00.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f33500bc700 1 --2- 172.21.15.175:0/217289128 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33480b2390 0x7f3348068690 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:00.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f33500bc700 1 -- 172.21.15.175:0/217289128 >> 172.21.15.175:0/217289128 conn(0x7f3348063230 msgr2=0x7f3348063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:00.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f33500bc700 1 -- 172.21.15.175:0/217289128 shutdown_connections 2023-10-10T19:03:00.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.984+0000 7f33500bc700 1 -- 172.21.15.175:0/217289128 wait complete. 2023-10-10T19:03:00.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.985+0000 7f33500bc700 1 Processor -- start 2023-10-10T19:03:00.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f33500bc700 1 -- start start 2023-10-10T19:03:00.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f33500bc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3348068bd0 0x7f3348148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:00.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f33500bc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33480b1fc0 0x7f3348156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:00.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f33500bc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33480b2390 0x7f3348156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:00.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f33500bc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334814a250 con 0x7f3348068bd0 2023-10-10T19:03:00.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f33500bc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3348156fd0 con 0x7f33480b2390 2023-10-10T19:03:00.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f33500bc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334814ace0 con 0x7f33480b1fc0 2023-10-10T19:03:00.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f334de58700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3348068bd0 0x7f3348148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:00.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f334de58700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3348068bd0 0x7f3348148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40858/0 (socket says 172.21.15.175:40858) 2023-10-10T19:03:00.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f334de58700 1 -- 172.21.15.175:0/2706386191 learned_addr learned my addr 172.21.15.175:0/2706386191 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:00.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f334e659700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33480b2390 0x7f3348156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:00.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.986+0000 7f334d657700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33480b1fc0 0x7f3348156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:00.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f334de58700 1 -- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33480b1fc0 msgr2=0x7f3348156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:00.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f334de58700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33480b1fc0 0x7f3348156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:00.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f334de58700 1 -- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33480b2390 msgr2=0x7f3348156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:00.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f334de58700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33480b2390 0x7f3348156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f334de58700 1 -- 172.21.15.175:0/2706386191 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33380094f0 con 0x7f3348068bd0 2023-10-10T19:03:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f334de58700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3348068bd0 0x7f3348148dd0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f334400e8d0 tx=0x7f3344008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f333dffb700 1 -- 172.21.15.175:0/2706386191 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33440093b0 con 0x7f3348068bd0 2023-10-10T19:03:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f333dffb700 1 -- 172.21.15.175:0/2706386191 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3344013410 con 0x7f3348068bd0 2023-10-10T19:03:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3348165cb0 con 0x7f3348068bd0 2023-10-10T19:03:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f333dffb700 1 -- 172.21.15.175:0/2706386191 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33440137b0 con 0x7f3348068bd0 2023-10-10T19:03:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.987+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3348165f70 con 0x7f3348068bd0 2023-10-10T19:03:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.988+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334814c370 con 0x7f3348068bd0 2023-10-10T19:03:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.988+0000 7f333dffb700 1 -- 172.21.15.175:0/2706386191 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3344013be0 con 0x7f3348068bd0 2023-10-10T19:03:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.989+0000 7f333dffb700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f332c066f60 0x7f332c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:01.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.989+0000 7f333dffb700 1 -- 172.21.15.175:0/2706386191 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f3344084450 con 0x7f3348068bd0 2023-10-10T19:03:01.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.989+0000 7f334d657700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f332c066f60 0x7f332c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:01.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.989+0000 7f334d657700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f332c066f60 0x7f332c069410 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f33480b28c0 tx=0x7f3338017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:01.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:00.993+0000 7f333dffb700 1 -- 172.21.15.175:0/2706386191 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33440513a0 con 0x7f3348068bd0 2023-10-10T19:03:01.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.239+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3348067570 con 0x7f3348068bd0 2023-10-10T19:03:01.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.239+0000 7f333dffb700 1 -- 172.21.15.175:0/2706386191 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7f33440549c0 con 0x7f3348068bd0 2023-10-10T19:03:01.241 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:01.241 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:01.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f332c066f60 msgr2=0x7f332c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:01.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f332c066f60 0x7f332c069410 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f33480b28c0 tx=0x7f3338017000 comp rx=0 tx=0).stop 2023-10-10T19:03:01.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3348068bd0 msgr2=0x7f3348148dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:01.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3348068bd0 0x7f3348148dd0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f334400e8d0 tx=0x7f3344008040 comp rx=0 tx=0).stop 2023-10-10T19:03:01.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 shutdown_connections 2023-10-10T19:03:01.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f332c066f60 0x7f332c069410 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:01.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33480b2390 0x7f3348156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:01.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3348068bd0 0x7f3348148dd0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:01.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 --2- 172.21.15.175:0/2706386191 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33480b1fc0 0x7f3348156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:01.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 >> 172.21.15.175:0/2706386191 conn(0x7f3348063230 msgr2=0x7f334814fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:01.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 shutdown_connections 2023-10-10T19:03:01.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:01.242+0000 7f33500bc700 1 -- 172.21.15.175:0/2706386191 wait complete. 2023-10-10T19:03:01.247 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:03:01.255 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:03:03.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:03.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.382+0000 7f7c3d931700 1 Processor -- start 2023-10-10T19:03:03.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.382+0000 7f7c3d931700 1 -- start start 2023-10-10T19:03:03.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.382+0000 7f7c3d931700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159760 0x7f7c3806d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.382+0000 7f7c3d931700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c3806a540 0x7f7c38156020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.382+0000 7f7c3d931700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c38068900 con 0x7f7c38159b30 2023-10-10T19:03:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.382+0000 7f7c3d931700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3806bc80 con 0x7f7c3806a540 2023-10-10T19:03:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.382+0000 7f7c3d931700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c380c4f00 con 0x7f7c38159760 2023-10-10T19:03:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.382+0000 7f7c377fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c3806a540 0x7f7c38156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c377fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c3806a540 0x7f7c38156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40872/0 (socket says 172.21.15.175:40872) 2023-10-10T19:03:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c377fe700 1 -- 172.21.15.175:0/1254934416 learned_addr learned my addr 172.21.15.175:0/1254934416 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c36ffd700 1 --2- 172.21.15.175:0/1254934416 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159760 0x7f7c3806d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c377fe700 1 -- 172.21.15.175:0/1254934416 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159760 msgr2=0x7f7c3806d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c377fe700 1 --2- 172.21.15.175:0/1254934416 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159760 0x7f7c3806d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c377fe700 1 -- 172.21.15.175:0/1254934416 >> v1:172.21.15.191:6789/0 conn(0x7f7c38159b30 legacy=0x7f7c3806c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c377fe700 1 -- 172.21.15.175:0/1254934416 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380c51e0 con 0x7f7c3806a540 2023-10-10T19:03:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c377fe700 1 --2- 172.21.15.175:0/1254934416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c3806a540 0x7f7c38156020 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009f20 tx=0x7f7c30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889ac7f2b1113b43 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c367fc700 1 -- 172.21.15.175:0/1254934416 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3000e070 con 0x7f7c3806a540 2023-10-10T19:03:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c367fc700 1 -- 172.21.15.175:0/1254934416 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3000b9d0 con 0x7f7c3806a540 2023-10-10T19:03:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.383+0000 7f7c367fc700 1 -- 172.21.15.175:0/1254934416 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c300092e0 con 0x7f7c3806a540 2023-10-10T19:03:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.384+0000 7f7c3d931700 1 -- 172.21.15.175:0/1254934416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c3806a540 msgr2=0x7f7c38156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.384+0000 7f7c3d931700 1 --2- 172.21.15.175:0/1254934416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c3806a540 0x7f7c38156020 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009f20 tx=0x7f7c30009840 comp rx=0 tx=0).stop 2023-10-10T19:03:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.384+0000 7f7c3d931700 1 -- 172.21.15.175:0/1254934416 shutdown_connections 2023-10-10T19:03:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.384+0000 7f7c3d931700 1 --2- 172.21.15.175:0/1254934416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c3806a540 0x7f7c38156020 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.384+0000 7f7c3d931700 1 --2- 172.21.15.175:0/1254934416 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159760 0x7f7c3806d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.384+0000 7f7c3d931700 1 -- 172.21.15.175:0/1254934416 >> 172.21.15.175:0/1254934416 conn(0x7f7c38063cc0 msgr2=0x7f7c3806cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.384+0000 7f7c3d931700 1 -- 172.21.15.175:0/1254934416 shutdown_connections 2023-10-10T19:03:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.384+0000 7f7c3d931700 1 -- 172.21.15.175:0/1254934416 wait complete. 2023-10-10T19:03:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.385+0000 7f7c3d931700 1 Processor -- start 2023-10-10T19:03:03.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.385+0000 7f7c3d931700 1 -- start start 2023-10-10T19:03:03.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c3d931700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c3806a540 0x7f7c380c2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:03.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c3d931700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c38159760 0x7f7c381660d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:03.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c3d931700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159b30 0x7f7c38156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:03.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c3d931700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c380c3950 con 0x7f7c38159760 2023-10-10T19:03:03.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c3d931700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38153b00 con 0x7f7c3806a540 2023-10-10T19:03:03.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c3d931700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38166610 con 0x7f7c38159b30 2023-10-10T19:03:03.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c38159760 0x7f7c381660d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:03.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c38159760 0x7f7c381660d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40878/0 (socket says 172.21.15.175:40878) 2023-10-10T19:03:03.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 -- 172.21.15.175:0/384963592 learned_addr learned my addr 172.21.15.175:0/384963592 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:03.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c377fe700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159b30 0x7f7c38156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:03.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c36ffd700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c3806a540 0x7f7c380c2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:03.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 -- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159b30 msgr2=0x7f7c38156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:03.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159b30 0x7f7c38156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 -- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c3806a540 msgr2=0x7f7c380c2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:03.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c3806a540 0x7f7c380c2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 -- 172.21.15.175:0/384963592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c30009ae0 con 0x7f7c38159760 2023-10-10T19:03:03.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2e7fc700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c38159760 0x7f7c381660d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000c5f0 tx=0x7f7c2000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:03.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2f7fe700 1 -- 172.21.15.175:0/384963592 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20011590 con 0x7f7c38159760 2023-10-10T19:03:03.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2f7fe700 1 -- 172.21.15.175:0/384963592 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c20011710 con 0x7f7c38159760 2023-10-10T19:03:03.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c380c1c10 con 0x7f7c38159760 2023-10-10T19:03:03.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c2f7fe700 1 -- 172.21.15.175:0/384963592 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20011ad0 con 0x7f7c38159760 2023-10-10T19:03:03.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.386+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c38152fe0 con 0x7f7c38159760 2023-10-10T19:03:03.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.387+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c38067570 con 0x7f7c38159760 2023-10-10T19:03:03.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.388+0000 7f7c2f7fe700 1 -- 172.21.15.175:0/384963592 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7c2001b3b0 con 0x7f7c38159760 2023-10-10T19:03:03.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.388+0000 7f7c2f7fe700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c18066f30 0x7f7c180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:03.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.388+0000 7f7c2f7fe700 1 -- 172.21.15.175:0/384963592 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f7c20083d30 con 0x7f7c38159760 2023-10-10T19:03:03.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.388+0000 7f7c36ffd700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c18066f30 0x7f7c180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:03.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.389+0000 7f7c36ffd700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c18066f30 0x7f7c180693e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009db0 tx=0x7f7c280091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:03.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.393+0000 7f7c2f7fe700 1 -- 172.21.15.175:0/384963592 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c20050c80 con 0x7f7c38159760 2023-10-10T19:03:03.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.622+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c38068b50 con 0x7f7c38159760 2023-10-10T19:03:03.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.622+0000 7f7c2f7fe700 1 -- 172.21.15.175:0/384963592 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7f7c20017070 con 0x7f7c38159760 2023-10-10T19:03:03.624 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:03.624 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:03.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.624+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c18066f30 msgr2=0x7f7c180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:03.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.624+0000 7f7c3d931700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c18066f30 0x7f7c180693e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009db0 tx=0x7f7c280091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.624+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c38159760 msgr2=0x7f7c381660d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:03.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.624+0000 7f7c3d931700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c38159760 0x7f7c381660d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000c5f0 tx=0x7f7c2000c900 comp rx=0 tx=0).stop 2023-10-10T19:03:03.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.625+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 shutdown_connections 2023-10-10T19:03:03.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.625+0000 7f7c3d931700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c18066f30 0x7f7c180693e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.625+0000 7f7c3d931700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c38159b30 0x7f7c38156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.625+0000 7f7c3d931700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c3806a540 0x7f7c380c2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.625+0000 7f7c3d931700 1 --2- 172.21.15.175:0/384963592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c38159760 0x7f7c381660d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:03.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.625+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 >> 172.21.15.175:0/384963592 conn(0x7f7c38063cc0 msgr2=0x7f7c38158980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:03.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.625+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 shutdown_connections 2023-10-10T19:03:03.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:03.625+0000 7f7c3d931700 1 -- 172.21.15.175:0/384963592 wait complete. 2023-10-10T19:03:03.631 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:03:03.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:03:05.638 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:05.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.754+0000 7f56bfed2700 1 Processor -- start 2023-10-10T19:03:05.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.754+0000 7f56bfed2700 1 -- start start 2023-10-10T19:03:05.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bfed2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80b1fc0 0x7f56b8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:05.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bfed2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56b80aff60 0x7f56b80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:05.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bfed2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b81574b0 con 0x7f56b80b2390 2023-10-10T19:03:05.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bfed2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b8150770 con 0x7f56b80aff60 2023-10-10T19:03:05.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bfed2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80afad0 con 0x7f56b80b1fc0 2023-10-10T19:03:05.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bd46d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f56b80b2390 0x7f56b8068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53182/0 (socket says 172.21.15.175:53182) 2023-10-10T19:03:05.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bd46d700 1 -- 172.21.15.175:0/2696834134 learned_addr learned my addr 172.21.15.175:0/2696834134 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:05.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bdc6e700 1 --2- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80b1fc0 0x7f56b8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:05.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56be46f700 1 --2- 172.21.15.175:0/2696834134 >> v2:172.21.15.191:3300/0 conn(0x7f56b80aff60 0x7f56b80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:05.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bcc6c700 1 -- 172.21.15.175:0/2696834134 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101680358 0 0) 0x7f56b81574b0 con 0x7f56b80b2390 2023-10-10T19:03:05.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.755+0000 7f56bcc6c700 1 -- 172.21.15.175:0/2696834134 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a4003570 con 0x7f56b80b2390 2023-10-10T19:03:05.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56be46f700 1 -- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80b1fc0 msgr2=0x7f56b8155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:05.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56be46f700 1 --2- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80b1fc0 0x7f56b8155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56be46f700 1 -- 172.21.15.175:0/2696834134 >> v1:172.21.15.175:6789/0 conn(0x7f56b80b2390 legacy=0x7f56b8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:05.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56be46f700 1 -- 172.21.15.175:0/2696834134 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80af4c0 con 0x7f56b80aff60 2023-10-10T19:03:05.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bdc6e700 1 --2- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80b1fc0 0x7f56b8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:03:05.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56be46f700 1 --2- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80aff60 0x7f56b80b03e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009e90 tx=0x7f56b0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8848cacca504d88 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:05.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bcc6c700 1 -- 172.21.15.175:0/2696834134 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000e070 con 0x7f56b80aff60 2023-10-10T19:03:05.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bcc6c700 1 -- 172.21.15.175:0/2696834134 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b000bbd0 con 0x7f56b80aff60 2023-10-10T19:03:05.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bcc6c700 1 -- 172.21.15.175:0/2696834134 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0012430 con 0x7f56b80aff60 2023-10-10T19:03:05.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bfed2700 1 -- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80aff60 msgr2=0x7f56b80b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:05.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bfed2700 1 --2- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80aff60 0x7f56b80b03e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009e90 tx=0x7f56b0009780 comp rx=0 tx=0).stop 2023-10-10T19:03:05.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bfed2700 1 -- 172.21.15.175:0/2696834134 shutdown_connections 2023-10-10T19:03:05.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bfed2700 1 --2- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80aff60 0x7f56b80b03e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bfed2700 1 --2- 172.21.15.175:0/2696834134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80b1fc0 0x7f56b8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.756+0000 7f56bfed2700 1 -- 172.21.15.175:0/2696834134 >> 172.21.15.175:0/2696834134 conn(0x7f56b8063230 msgr2=0x7f56b8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:05.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.757+0000 7f56bfed2700 1 -- 172.21.15.175:0/2696834134 shutdown_connections 2023-10-10T19:03:05.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.757+0000 7f56bfed2700 1 -- 172.21.15.175:0/2696834134 wait complete. 2023-10-10T19:03:05.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bfed2700 1 Processor -- start 2023-10-10T19:03:05.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bfed2700 1 -- start start 2023-10-10T19:03:05.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bfed2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80aff60 0x7f56b81666b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:05.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bfed2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80b1fc0 0x7f56b80cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:05.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bfed2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b80b2390 0x7f56b8150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:05.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bfed2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8166220 con 0x7f56b80b2390 2023-10-10T19:03:05.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bfed2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8159a90 con 0x7f56b80b1fc0 2023-10-10T19:03:05.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bfed2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8159d70 con 0x7f56b80aff60 2023-10-10T19:03:05.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56be46f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b80b2390 0x7f56b8150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:05.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56be46f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b80b2390 0x7f56b8150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38378/0 (socket says 172.21.15.175:38378) 2023-10-10T19:03:05.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56be46f700 1 -- 172.21.15.175:0/1369983464 learned_addr learned my addr 172.21.15.175:0/1369983464 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:05.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bdc6e700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80aff60 0x7f56b81666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:05.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.758+0000 7f56bd46d700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80b1fc0 0x7f56b80cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56be46f700 1 -- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80aff60 msgr2=0x7f56b81666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56be46f700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80aff60 0x7f56b81666b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56be46f700 1 -- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80b1fc0 msgr2=0x7f56b80cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56be46f700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80b1fc0 0x7f56b80cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56be46f700 1 -- 172.21.15.175:0/1369983464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4009420 con 0x7f56b80b2390 2023-10-10T19:03:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56bd46d700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80b1fc0 0x7f56b80cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56be46f700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b80b2390 0x7f56b8150fe0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f56b806fd90 tx=0x7f56b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56adffb700 1 -- 172.21.15.175:0/1369983464 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0009420 con 0x7f56b80b2390 2023-10-10T19:03:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56adffb700 1 -- 172.21.15.175:0/1369983464 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b0027470 con 0x7f56b80b2390 2023-10-10T19:03:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b0009a20 con 0x7f56b80b2390 2023-10-10T19:03:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56adffb700 1 -- 172.21.15.175:0/1369983464 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0027810 con 0x7f56b80b2390 2023-10-10T19:03:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.759+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b815a330 con 0x7f56b80b2390 2023-10-10T19:03:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.760+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b8067570 con 0x7f56b80b2390 2023-10-10T19:03:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.760+0000 7f56adffb700 1 -- 172.21.15.175:0/1369983464 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f56b0024070 con 0x7f56b80b2390 2023-10-10T19:03:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.761+0000 7f56adffb700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f569c066f30 0x7f569c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.761+0000 7f56adffb700 1 -- 172.21.15.175:0/1369983464 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f56b0086fd0 con 0x7f56b80b2390 2023-10-10T19:03:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.761+0000 7f56bdc6e700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f569c066f30 0x7f569c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.761+0000 7f56bdc6e700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f569c066f30 0x7f569c0693e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009f40 tx=0x7f56b40091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.765+0000 7f56adffb700 1 -- 172.21.15.175:0/1369983464 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56b0053f20 con 0x7f56b80b2390 2023-10-10T19:03:05.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.992+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56b8166360 con 0x7f56b80b2390 2023-10-10T19:03:05.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.993+0000 7f56adffb700 1 -- 172.21.15.175:0/1369983464 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7f56b0057540 con 0x7f56b80b2390 2023-10-10T19:03:05.994 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:05.994 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:05.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f569c066f30 msgr2=0x7f569c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:05.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f569c066f30 0x7f569c0693e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009f40 tx=0x7f56b40091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b80b2390 msgr2=0x7f56b8150fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:05.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b80b2390 0x7f56b8150fe0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f56b806fd90 tx=0x7f56b0019000 comp rx=0 tx=0).stop 2023-10-10T19:03:05.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 shutdown_connections 2023-10-10T19:03:05.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f569c066f30 0x7f569c0693e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56b80b2390 0x7f56b8150fe0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b80aff60 0x7f56b81666b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 --2- 172.21.15.175:0/1369983464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b80b1fc0 0x7f56b80cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:05.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 >> 172.21.15.175:0/1369983464 conn(0x7f56b8063230 msgr2=0x7f56b80c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:05.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 shutdown_connections 2023-10-10T19:03:05.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:05.995+0000 7f56bfed2700 1 -- 172.21.15.175:0/1369983464 wait complete. 2023-10-10T19:03:05.998 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:03:06.007 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:03:08.009 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:08.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.136+0000 7f0ab251c700 1 Processor -- start 2023-10-10T19:03:08.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.136+0000 7f0ab251c700 1 -- start start 2023-10-10T19:03:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.136+0000 7f0ab251c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0aac0b1fc0 0x7f0aac155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.136+0000 7f0ab251c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0aac0aff60 0x7f0aac0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.136+0000 7f0ab251c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac1574b0 con 0x7f0aac0b2390 2023-10-10T19:03:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.136+0000 7f0ab251c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac150770 con 0x7f0aac0aff60 2023-10-10T19:03:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.136+0000 7f0ab251c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac0afad0 con 0x7f0aac0b1fc0 2023-10-10T19:03:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0ab0ab9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0aac0aff60 0x7f0aac0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0ab0ab9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0aac0aff60 0x7f0aac0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38380/0 (socket says 172.21.15.175:38380) 2023-10-10T19:03:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0ab0ab9700 1 -- 172.21.15.175:0/2657093137 learned_addr learned my addr 172.21.15.175:0/2657093137 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0aabfff700 1 --2- 172.21.15.175:0/2657093137 >> v2:172.21.15.191:3300/0 conn(0x7f0aac0b1fc0 0x7f0aac155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:08.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0ab0ab9700 1 -- 172.21.15.175:0/2657093137 >> v2:172.21.15.191:3300/0 conn(0x7f0aac0b1fc0 msgr2=0x7f0aac155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0ab0ab9700 1 --2- 172.21.15.175:0/2657093137 >> v2:172.21.15.191:3300/0 conn(0x7f0aac0b1fc0 0x7f0aac155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0ab0ab9700 1 -- 172.21.15.175:0/2657093137 >> v1:172.21.15.191:6789/0 conn(0x7f0aac0b2390 legacy=0x7f0aac068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0ab0ab9700 1 -- 172.21.15.175:0/2657093137 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac0af4c0 con 0x7f0aac0aff60 2023-10-10T19:03:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0ab0ab9700 1 --2- 172.21.15.175:0/2657093137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0b03e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0b28c0 tx=0x7f0aa4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc76999c4ae69d4f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:08.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0aaaffd700 1 -- 172.21.15.175:0/2657093137 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400e070 con 0x7f0aac0aff60 2023-10-10T19:03:08.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0aaaffd700 1 -- 172.21.15.175:0/2657093137 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa400bae0 con 0x7f0aac0aff60 2023-10-10T19:03:08.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.137+0000 7f0aaaffd700 1 -- 172.21.15.175:0/2657093137 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa40092e0 con 0x7f0aac0aff60 2023-10-10T19:03:08.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.138+0000 7f0ab251c700 1 -- 172.21.15.175:0/2657093137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 msgr2=0x7f0aac0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:08.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.138+0000 7f0ab251c700 1 --2- 172.21.15.175:0/2657093137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0b03e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0b28c0 tx=0x7f0aa4009840 comp rx=0 tx=0).stop 2023-10-10T19:03:08.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.138+0000 7f0ab251c700 1 -- 172.21.15.175:0/2657093137 shutdown_connections 2023-10-10T19:03:08.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.138+0000 7f0ab251c700 1 --2- 172.21.15.175:0/2657093137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0b03e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.138+0000 7f0ab251c700 1 --2- 172.21.15.175:0/2657093137 >> v2:172.21.15.191:3300/0 conn(0x7f0aac0b1fc0 0x7f0aac155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.138+0000 7f0ab251c700 1 -- 172.21.15.175:0/2657093137 >> 172.21.15.175:0/2657093137 conn(0x7f0aac063230 msgr2=0x7f0aac063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:08.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.138+0000 7f0ab251c700 1 -- 172.21.15.175:0/2657093137 shutdown_connections 2023-10-10T19:03:08.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.138+0000 7f0ab251c700 1 -- 172.21.15.175:0/2657093137 wait complete. 2023-10-10T19:03:08.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.139+0000 7f0ab251c700 1 Processor -- start 2023-10-10T19:03:08.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.139+0000 7f0ab251c700 1 -- start start 2023-10-10T19:03:08.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0ab251c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:08.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0ab251c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aac0b1fc0 0x7f0aac156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:08.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0ab251c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aac0b2390 0x7f0aac0cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:08.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0ab251c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0ca9a0 con 0x7f0aac0aff60 2023-10-10T19:03:08.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0ab251c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0c39f0 con 0x7f0aac0b2390 2023-10-10T19:03:08.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0ab251c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0cabf0 con 0x7f0aac0b1fc0 2023-10-10T19:03:08.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0c4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38386/0 (socket says 172.21.15.175:38386) 2023-10-10T19:03:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 -- 172.21.15.175:0/3475231130 learned_addr learned my addr 172.21.15.175:0/3475231130 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0ab0ab9700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aac0b2390 0x7f0aac0cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aab7fe700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aac0b1fc0 0x7f0aac156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 -- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aac0b1fc0 msgr2=0x7f0aac156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aac0b1fc0 0x7f0aac156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 -- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aac0b2390 msgr2=0x7f0aac0cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aac0b2390 0x7f0aac0cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 -- 172.21.15.175:0/3475231130 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4009ae0 con 0x7f0aac0aff60 2023-10-10T19:03:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0ab0ab9700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aac0b2390 0x7f0aac0cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:03:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.140+0000 7f0aabfff700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0c4920 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000d890 tx=0x7f0aa0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.141+0000 7f0a93fff700 1 -- 172.21.15.175:0/3475231130 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa000b600 con 0x7f0aac0aff60 2023-10-10T19:03:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.141+0000 7f0a93fff700 1 -- 172.21.15.175:0/3475231130 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa000b760 con 0x7f0aac0aff60 2023-10-10T19:03:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.141+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aac0caed0 con 0x7f0aac0aff60 2023-10-10T19:03:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.141+0000 7f0a93fff700 1 -- 172.21.15.175:0/3475231130 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa000bae0 con 0x7f0aac0aff60 2023-10-10T19:03:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.141+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac0cb490 con 0x7f0aac0aff60 2023-10-10T19:03:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.141+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aac067570 con 0x7f0aac0aff60 2023-10-10T19:03:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.142+0000 7f0a93fff700 1 -- 172.21.15.175:0/3475231130 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0aa001e460 con 0x7f0aac0aff60 2023-10-10T19:03:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.143+0000 7f0a93fff700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a98066f60 0x7f0a98069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.143+0000 7f0a93fff700 1 -- 172.21.15.175:0/3475231130 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f0aa00859a0 con 0x7f0aac0aff60 2023-10-10T19:03:08.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.143+0000 7f0aab7fe700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a98066f60 0x7f0a98069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:08.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.143+0000 7f0aab7fe700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a98066f60 0x7f0a98069410 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0aac15a350 tx=0x7f0a9c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:08.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.146+0000 7f0a93fff700 1 -- 172.21.15.175:0/3475231130 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0aa0052970 con 0x7f0aac0aff60 2023-10-10T19:03:08.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.372+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aac068da0 con 0x7f0aac0aff60 2023-10-10T19:03:08.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.372+0000 7f0a93fff700 1 -- 172.21.15.175:0/3475231130 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7f0aa0055f90 con 0x7f0aac0aff60 2023-10-10T19:03:08.373 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:08.373 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:08.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a98066f60 msgr2=0x7f0a98069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:08.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a98066f60 0x7f0a98069410 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0aac15a350 tx=0x7f0a9c009040 comp rx=0 tx=0).stop 2023-10-10T19:03:08.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 msgr2=0x7f0aac0c4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:08.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0c4920 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000d890 tx=0x7f0aa0008040 comp rx=0 tx=0).stop 2023-10-10T19:03:08.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 shutdown_connections 2023-10-10T19:03:08.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a98066f60 0x7f0a98069410 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0aac0b2390 0x7f0aac0cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0aac0aff60 0x7f0aac0c4920 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 --2- 172.21.15.175:0/3475231130 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0aac0b1fc0 0x7f0aac156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:08.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.374+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 >> 172.21.15.175:0/3475231130 conn(0x7f0aac063230 msgr2=0x7f0aac150c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:08.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.375+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 shutdown_connections 2023-10-10T19:03:08.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:08.375+0000 7f0ab251c700 1 -- 172.21.15.175:0/3475231130 wait complete. 2023-10-10T19:03:08.377 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:03:08.386 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:03:10.388 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:10.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.516+0000 7fba4aede700 1 Processor -- start 2023-10-10T19:03:10.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.516+0000 7fba4aede700 1 -- start start 2023-10-10T19:03:10.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba4aede700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fba4414cdc0 0x7fba440c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:10.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba4aede700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba4406f620 0x7fba440c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:10.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba4aede700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4406e940 con 0x7fba4414d190 2023-10-10T19:03:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba4aede700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba440c1c90 con 0x7fba4406f620 2023-10-10T19:03:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba4aede700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba440c6e20 con 0x7fba4414cdc0 2023-10-10T19:03:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba4947b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba4406f620 0x7fba440c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:10.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba3bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fba4414d190 0x7fba44156300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53188/0 (socket says 172.21.15.175:53188) 2023-10-10T19:03:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba4947b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba4406f620 0x7fba440c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38392/0 (socket says 172.21.15.175:38392) 2023-10-10T19:03:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.517+0000 7fba3bfff700 1 -- 172.21.15.175:0/3014098629 learned_addr learned my addr 172.21.15.175:0/3014098629 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba48c7a700 1 --2- 172.21.15.175:0/3014098629 >> v2:172.21.15.191:3300/0 conn(0x7fba4414cdc0 0x7fba440c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:10.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba4947b700 1 -- 172.21.15.175:0/3014098629 >> v2:172.21.15.191:3300/0 conn(0x7fba4414cdc0 msgr2=0x7fba440c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba4947b700 1 --2- 172.21.15.175:0/3014098629 >> v2:172.21.15.191:3300/0 conn(0x7fba4414cdc0 0x7fba440c26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba4947b700 1 -- 172.21.15.175:0/3014098629 >> v1:172.21.15.175:6789/0 conn(0x7fba4414d190 legacy=0x7fba44156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba4947b700 1 -- 172.21.15.175:0/3014098629 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4406ca40 con 0x7fba4406f620 2023-10-10T19:03:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba4947b700 1 --2- 172.21.15.175:0/3014098629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c2c20 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0098e0 tx=0x7fba3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cdd451b9255ba8b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:10.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba3b7fe700 1 -- 172.21.15.175:0/3014098629 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c009e80 con 0x7fba4406f620 2023-10-10T19:03:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba3b7fe700 1 -- 172.21.15.175:0/3014098629 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c00b820 con 0x7fba4406f620 2023-10-10T19:03:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.518+0000 7fba3b7fe700 1 -- 172.21.15.175:0/3014098629 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00bba0 con 0x7fba4406f620 2023-10-10T19:03:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.519+0000 7fba4aede700 1 -- 172.21.15.175:0/3014098629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 msgr2=0x7fba440c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.519+0000 7fba4aede700 1 --2- 172.21.15.175:0/3014098629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c2c20 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0098e0 tx=0x7fba3c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.519+0000 7fba4aede700 1 -- 172.21.15.175:0/3014098629 shutdown_connections 2023-10-10T19:03:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.519+0000 7fba4aede700 1 --2- 172.21.15.175:0/3014098629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c2c20 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.519+0000 7fba4aede700 1 --2- 172.21.15.175:0/3014098629 >> v2:172.21.15.191:3300/0 conn(0x7fba4414cdc0 0x7fba440c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.519+0000 7fba4aede700 1 -- 172.21.15.175:0/3014098629 >> 172.21.15.175:0/3014098629 conn(0x7fba44068900 msgr2=0x7fba44068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.519+0000 7fba4aede700 1 -- 172.21.15.175:0/3014098629 shutdown_connections 2023-10-10T19:03:10.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.519+0000 7fba4aede700 1 -- 172.21.15.175:0/3014098629 wait complete. 2023-10-10T19:03:10.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.520+0000 7fba4aede700 1 Processor -- start 2023-10-10T19:03:10.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.520+0000 7fba4aede700 1 -- start start 2023-10-10T19:03:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba4aede700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba4aede700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba4414cdc0 0x7fba44149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba4aede700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba4414d190 0x7fba4414a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba4aede700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba44151370 con 0x7fba4406f620 2023-10-10T19:03:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba4aede700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba44150770 con 0x7fba4414cdc0 2023-10-10T19:03:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba4aede700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba44150190 con 0x7fba4414d190 2023-10-10T19:03:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38396/0 (socket says 172.21.15.175:38396) 2023-10-10T19:03:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 -- 172.21.15.175:0/2441903022 learned_addr learned my addr 172.21.15.175:0/2441903022 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba3bfff700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba4414cdc0 0x7fba44149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba4947b700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba4414d190 0x7fba4414a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 -- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba4414d190 msgr2=0x7fba4414a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba4414d190 0x7fba4414a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 -- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba4414cdc0 msgr2=0x7fba44149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba4414cdc0 0x7fba44149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 -- 172.21.15.175:0/2441903022 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c009420 con 0x7fba4406f620 2023-10-10T19:03:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba48c7a700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c8ae0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fba4000e1b0 tx=0x7fba40007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba38ff9700 1 -- 172.21.15.175:0/2441903022 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40011070 con 0x7fba4406f620 2023-10-10T19:03:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba38ff9700 1 -- 172.21.15.175:0/2441903022 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4000c2e0 con 0x7fba4406f620 2023-10-10T19:03:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba440ca390 con 0x7fba4406f620 2023-10-10T19:03:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.521+0000 7fba38ff9700 1 -- 172.21.15.175:0/2441903022 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4000ed30 con 0x7fba4406f620 2023-10-10T19:03:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.522+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba441518a0 con 0x7fba4406f620 2023-10-10T19:03:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.522+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba44067570 con 0x7fba4406f620 2023-10-10T19:03:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.523+0000 7fba38ff9700 1 -- 172.21.15.175:0/2441903022 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fba40003020 con 0x7fba4406f620 2023-10-10T19:03:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.523+0000 7fba38ff9700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba30066f60 0x7fba30069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.523+0000 7fba38ff9700 1 -- 172.21.15.175:0/2441903022 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fba400862f0 con 0x7fba4406f620 2023-10-10T19:03:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.523+0000 7fba3bfff700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba30066f60 0x7fba30069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.524+0000 7fba3bfff700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba30066f60 0x7fba30069410 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fba440c5490 tx=0x7fba34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.527+0000 7fba38ff9700 1 -- 172.21.15.175:0/2441903022 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba4004e5a0 con 0x7fba4406f620 2023-10-10T19:03:10.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.756+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba4406eb50 con 0x7fba4406f620 2023-10-10T19:03:10.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.756+0000 7fba38ff9700 1 -- 172.21.15.175:0/2441903022 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7fba40053240 con 0x7fba4406f620 2023-10-10T19:03:10.757 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:10.757 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:10.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.758+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba30066f60 msgr2=0x7fba30069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:10.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.758+0000 7fba4aede700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba30066f60 0x7fba30069410 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fba440c5490 tx=0x7fba34009040 comp rx=0 tx=0).stop 2023-10-10T19:03:10.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.758+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 msgr2=0x7fba440c8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:10.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.758+0000 7fba4aede700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c8ae0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fba4000e1b0 tx=0x7fba40007900 comp rx=0 tx=0).stop 2023-10-10T19:03:10.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.758+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 shutdown_connections 2023-10-10T19:03:10.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.758+0000 7fba4aede700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba30066f60 0x7fba30069410 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.758+0000 7fba4aede700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba4414d190 0x7fba4414a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.758+0000 7fba4aede700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba4406f620 0x7fba440c8ae0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.759+0000 7fba4aede700 1 --2- 172.21.15.175:0/2441903022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba4414cdc0 0x7fba44149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:10.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.759+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 >> 172.21.15.175:0/2441903022 conn(0x7fba44068900 msgr2=0x7fba4414bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:10.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.759+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 shutdown_connections 2023-10-10T19:03:10.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:10.759+0000 7fba4aede700 1 -- 172.21.15.175:0/2441903022 wait complete. 2023-10-10T19:03:10.761 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:03:10.770 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:03:12.771 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:12.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.898+0000 7f4b4cb08700 1 Processor -- start 2023-10-10T19:03:12.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.898+0000 7f4b4cb08700 1 -- start start 2023-10-10T19:03:12.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b4cb08700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480b2390 0x7f4b48155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:12.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b4cb08700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4b480aff60 0x7f4b480b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:12.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b4cb08700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b481574b0 con 0x7f4b480b1fc0 2023-10-10T19:03:12.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b4cb08700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b48150770 con 0x7f4b480aff60 2023-10-10T19:03:12.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b4cb08700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480afad0 con 0x7f4b480b2390 2023-10-10T19:03:12.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b46d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4b480aff60 0x7f4b480b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:12.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b46d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4b480aff60 0x7f4b480b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38400/0 (socket says 172.21.15.175:38400) 2023-10-10T19:03:12.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b46d9d700 1 -- 172.21.15.175:0/570524847 learned_addr learned my addr 172.21.15.175:0/570524847 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:12.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.899+0000 7f4b45d9b700 1 --2- 172.21.15.175:0/570524847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480b2390 0x7f4b48155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:12.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b46d9d700 1 -- 172.21.15.175:0/570524847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480b2390 msgr2=0x7f4b48155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:12.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b46d9d700 1 --2- 172.21.15.175:0/570524847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480b2390 0x7f4b48155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:12.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b46d9d700 1 -- 172.21.15.175:0/570524847 >> v1:172.21.15.191:6789/0 conn(0x7f4b480b1fc0 legacy=0x7f4b48068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:12.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b46d9d700 1 -- 172.21.15.175:0/570524847 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b480af4c0 con 0x7f4b480aff60 2023-10-10T19:03:12.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b46d9d700 1 --2- 172.21.15.175:0/570524847 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480aff60 0x7f4b480b03e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4b400099e0 tx=0x7f4b40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5c3151cbe5a195 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:12.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4559a700 1 -- 172.21.15.175:0/570524847 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4000b8b0 con 0x7f4b480aff60 2023-10-10T19:03:12.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4559a700 1 -- 172.21.15.175:0/570524847 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4000ba10 con 0x7f4b480aff60 2023-10-10T19:03:12.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4559a700 1 -- 172.21.15.175:0/570524847 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4000bdb0 con 0x7f4b480aff60 2023-10-10T19:03:12.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4cb08700 1 -- 172.21.15.175:0/570524847 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480aff60 msgr2=0x7f4b480b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:12.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/570524847 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480aff60 0x7f4b480b03e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4b400099e0 tx=0x7f4b40009250 comp rx=0 tx=0).stop 2023-10-10T19:03:12.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4cb08700 1 -- 172.21.15.175:0/570524847 shutdown_connections 2023-10-10T19:03:12.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/570524847 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480aff60 0x7f4b480b03e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:12.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/570524847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480b2390 0x7f4b48155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:12.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.900+0000 7f4b4cb08700 1 -- 172.21.15.175:0/570524847 >> 172.21.15.175:0/570524847 conn(0x7f4b48063230 msgr2=0x7f4b48063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:12.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.901+0000 7f4b4cb08700 1 -- 172.21.15.175:0/570524847 shutdown_connections 2023-10-10T19:03:12.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.901+0000 7f4b4cb08700 1 -- 172.21.15.175:0/570524847 wait complete. 2023-10-10T19:03:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b4cb08700 1 Processor -- start 2023-10-10T19:03:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b4cb08700 1 -- start start 2023-10-10T19:03:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b4cb08700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480aff60 0x7f4b480c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b4cb08700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480b1fc0 0x7f4b480c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b4cb08700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b480b2390 0x7f4b48169550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b4cb08700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c4500 con 0x7f4b480b1fc0 2023-10-10T19:03:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b4cb08700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48158290 con 0x7f4b480b2390 2023-10-10T19:03:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b4cb08700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c3910 con 0x7f4b480aff60 2023-10-10T19:03:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b45d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480b1fc0 0x7f4b480c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b45d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480b1fc0 0x7f4b480c5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38414/0 (socket says 172.21.15.175:38414) 2023-10-10T19:03:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.902+0000 7f4b45d9b700 1 -- 172.21.15.175:0/2848629319 learned_addr learned my addr 172.21.15.175:0/2848629319 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b4659c700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480aff60 0x7f4b480c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b46d9d700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b480b2390 0x7f4b48169550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b45d9b700 1 -- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480aff60 msgr2=0x7f4b480c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b45d9b700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480aff60 0x7f4b480c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b45d9b700 1 -- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b480b2390 msgr2=0x7f4b48169550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b45d9b700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b480b2390 0x7f4b48169550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b45d9b700 1 -- 172.21.15.175:0/2848629319 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b400094f0 con 0x7f4b480b1fc0 2023-10-10T19:03:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b45d9b700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480b1fc0 0x7f4b480c5df0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00e930 tx=0x7f4b3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b367fc700 1 -- 172.21.15.175:0/2848629319 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c00eed0 con 0x7f4b480b1fc0 2023-10-10T19:03:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b367fc700 1 -- 172.21.15.175:0/2848629319 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c00c850 con 0x7f4b480b1fc0 2023-10-10T19:03:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b367fc700 1 -- 172.21.15.175:0/2848629319 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c00cbf0 con 0x7f4b480b1fc0 2023-10-10T19:03:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b480c3bd0 con 0x7f4b480b1fc0 2023-10-10T19:03:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.903+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b480c4190 con 0x7f4b480b1fc0 2023-10-10T19:03:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.904+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b480c4670 con 0x7f4b480b1fc0 2023-10-10T19:03:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.904+0000 7f4b367fc700 1 -- 172.21.15.175:0/2848629319 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4b3c013470 con 0x7f4b480b1fc0 2023-10-10T19:03:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.905+0000 7f4b367fc700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b28066f30 0x7f4b280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.905+0000 7f4b367fc700 1 -- 172.21.15.175:0/2848629319 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f4b3c084a40 con 0x7f4b480b1fc0 2023-10-10T19:03:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.905+0000 7f4b4659c700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b28066f30 0x7f4b280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:12.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.905+0000 7f4b4659c700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b28066f30 0x7f4b280693e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4b38007fd0 tx=0x7f4b38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:12.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:12.909+0000 7f4b367fc700 1 -- 172.21.15.175:0/2848629319 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b3c086e60 con 0x7f4b480b1fc0 2023-10-10T19:03:13.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.135+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b48067570 con 0x7f4b480b1fc0 2023-10-10T19:03:13.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.136+0000 7f4b367fc700 1 -- 172.21.15.175:0/2848629319 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6217 (secure 0 0 0) 0x7f4b3c051990 con 0x7f4b480b1fc0 2023-10-10T19:03:13.136 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:13.136 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:02:52.672621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":5332,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5332":{"gid":5332,"name":"f","rank":1,"incarnation":82,"state":"up:active","state_seq":31,"addr":"172.21.15.191:6839/1598171176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1598171176},{"type":"v1","addr":"172.21.15.191:6839","nonce":1598171176}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:13.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b28066f30 msgr2=0x7f4b280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:13.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b28066f30 0x7f4b280693e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4b38007fd0 tx=0x7f4b38009040 comp rx=0 tx=0).stop 2023-10-10T19:03:13.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480b1fc0 msgr2=0x7f4b480c5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:13.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480b1fc0 0x7f4b480c5df0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00e930 tx=0x7f4b3c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:13.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 shutdown_connections 2023-10-10T19:03:13.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b28066f30 0x7f4b280693e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:13.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b480b2390 0x7f4b48169550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:13.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b480aff60 0x7f4b480c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:13.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 --2- 172.21.15.175:0/2848629319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b480b1fc0 0x7f4b480c5df0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:13.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 >> 172.21.15.175:0/2848629319 conn(0x7f4b48063230 msgr2=0x7f4b4814fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:13.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 shutdown_connections 2023-10-10T19:03:13.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:13.138+0000 7f4b4cb08700 1 -- 172.21.15.175:0/2848629319 wait complete. 2023-10-10T19:03:13.141 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 133 2023-10-10T19:03:13.150 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:03:15.151 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:15.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f142aada700 1 Processor -- start 2023-10-10T19:03:15.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f142aada700 1 -- start start 2023-10-10T19:03:15.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f142aada700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 0x7f1424155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f142aada700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f14240aff60 0x7f14240b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f142aada700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14241574b0 con 0x7f14240b1fc0 2023-10-10T19:03:15.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f142aada700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1424150770 con 0x7f14240aff60 2023-10-10T19:03:15.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f142aada700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14240afad0 con 0x7f14240b2390 2023-10-10T19:03:15.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f1428876700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f14240b1fc0 0x7f1424068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:57718/0 (socket says 172.21.15.175:57718) 2023-10-10T19:03:15.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.278+0000 7f1428876700 1 -- 172.21.15.175:0/4291628718 learned_addr learned my addr 172.21.15.175:0/4291628718 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:15.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141bfff700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 0x7f1424155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f1429077700 1 --2- 172.21.15.175:0/4291628718 >> v2:172.21.15.191:3300/0 conn(0x7f14240aff60 0x7f14240b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283075719 0 0) 0x7f14241574b0 con 0x7f14240b1fc0 2023-10-10T19:03:15.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f140c003570 con 0x7f14240b1fc0 2023-10-10T19:03:15.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1376682553 0 0) 0x7f140c003570 con 0x7f14240b1fc0 2023-10-10T19:03:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 msgr2=0x7f1424155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 0x7f1424155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 >> v2:172.21.15.191:3300/0 conn(0x7f14240aff60 msgr2=0x7f14240b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 --2- 172.21.15.175:0/4291628718 >> v2:172.21.15.191:3300/0 conn(0x7f14240aff60 0x7f14240b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14240af4c0 con 0x7f14240b1fc0 2023-10-10T19:03:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f1414001470 con 0x7f14240b1fc0 2023-10-10T19:03:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 >> v1:172.21.15.175:6789/0 conn(0x7f14240b1fc0 legacy=0x7f1424068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f140c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 0x7f140c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f140c004d50 0x7f140c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1424150770 con 0x7f140c003a60 2023-10-10T19:03:15.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14240afad0 con 0x7f140c004d50 2023-10-10T19:03:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140c00ba60 con 0x7f14240b2390 2023-10-10T19:03:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f1428876700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f140c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f1429077700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f140c004d50 0x7f140c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141bfff700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 0x7f140c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f1428876700 1 -- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 msgr2=0x7f140c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f1428876700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 0x7f140c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f1428876700 1 -- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f140c004d50 msgr2=0x7f140c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f1428876700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f140c004d50 0x7f140c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f1428876700 1 -- 172.21.15.175:0/4291628718 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c003570 con 0x7f140c003a60 2023-10-10T19:03:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.279+0000 7f141bfff700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14240b2390 0x7f140c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f1428876700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f140c003e70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f141400c390 tx=0x7f14240afa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d7f6ad55aa26c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400cc00 con 0x7f140c003a60 2023-10-10T19:03:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141400cd60 con 0x7f140c003a60 2023-10-10T19:03:15.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f141b7fe700 1 -- 172.21.15.175:0/4291628718 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414005b70 con 0x7f140c003a60 2023-10-10T19:03:15.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f142aada700 1 -- 172.21.15.175:0/4291628718 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 msgr2=0x7f140c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f142aada700 1 --2- 172.21.15.175:0/4291628718 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f140c003e70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f141400c390 tx=0x7f14240afa40 comp rx=0 tx=0).stop 2023-10-10T19:03:15.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f1429077700 1 -- 172.21.15.175:0/4291628718 reap_dead start 2023-10-10T19:03:15.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f142aada700 1 -- 172.21.15.175:0/4291628718 shutdown_connections 2023-10-10T19:03:15.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f142aada700 1 -- 172.21.15.175:0/4291628718 >> 172.21.15.175:0/4291628718 conn(0x7f1424063230 msgr2=0x7f1424063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:15.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f142aada700 1 -- 172.21.15.175:0/4291628718 shutdown_connections 2023-10-10T19:03:15.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.280+0000 7f142aada700 1 -- 172.21.15.175:0/4291628718 wait complete. 2023-10-10T19:03:15.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.281+0000 7f142aada700 1 Processor -- start 2023-10-10T19:03:15.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f142aada700 1 -- start start 2023-10-10T19:03:15.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f142aada700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f1424157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f142aada700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1424166570 0x7f14241576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f142aada700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14240cc8b0 0x7f14240ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f142aada700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14240c3d10 con 0x7f140c003a60 2023-10-10T19:03:15.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f142aada700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142415a3d0 con 0x7f14240cc8b0 2023-10-10T19:03:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f142aada700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1424159e10 con 0x7f1424166570 2023-10-10T19:03:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f1428876700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f1424157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f1428876700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f1424157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53220/0 (socket says 172.21.15.175:53220) 2023-10-10T19:03:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f1428876700 1 -- 172.21.15.175:0/393904565 learned_addr learned my addr 172.21.15.175:0/393904565 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f141bfff700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1424166570 0x7f14241576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.282+0000 7f1429077700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14240cc8b0 0x7f14240ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1428876700 1 -- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1424166570 msgr2=0x7f14241576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1428876700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1424166570 0x7f14241576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1428876700 1 -- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14240cc8b0 msgr2=0x7f14240ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1428876700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14240cc8b0 0x7f14240ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1428876700 1 -- 172.21.15.175:0/393904565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142415a0f0 con 0x7f140c003a60 2023-10-10T19:03:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1428876700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f1424157180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f140c004760 tx=0x7f1414015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1418ff9700 1 -- 172.21.15.175:0/393904565 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400cc00 con 0x7f140c003a60 2023-10-10T19:03:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1418ff9700 1 -- 172.21.15.175:0/393904565 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141400ced0 con 0x7f140c003a60 2023-10-10T19:03:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f1418ff9700 1 -- 172.21.15.175:0/393904565 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414007530 con 0x7f140c003a60 2023-10-10T19:03:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14240c39a0 con 0x7f140c003a60 2023-10-10T19:03:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.283+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142415bc30 con 0x7f140c003a60 2023-10-10T19:03:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.284+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1424067570 con 0x7f140c003a60 2023-10-10T19:03:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.284+0000 7f1418ff9700 1 -- 172.21.15.175:0/393904565 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1414019030 con 0x7f140c003a60 2023-10-10T19:03:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.285+0000 7f1418ff9700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1410066f60 0x7f1410069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.285+0000 7f1418ff9700 1 -- 172.21.15.175:0/393904565 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f1414085aa0 con 0x7f140c003a60 2023-10-10T19:03:15.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.285+0000 7f141bfff700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1410066f60 0x7f1410069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.285+0000 7f141bfff700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1410066f60 0x7f1410069410 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1424151340 tx=0x7f1420008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:15.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.289+0000 7f1418ff9700 1 -- 172.21.15.175:0/393904565 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1414052990 con 0x7f140c003a60 2023-10-10T19:03:15.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.514+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1424068da0 con 0x7f140c003a60 2023-10-10T19:03:15.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.514+0000 7f1418ff9700 1 -- 172.21.15.175:0/393904565 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5454 (secure 0 0 0) 0x7f1414017090 con 0x7f140c003a60 2023-10-10T19:03:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:15.515 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:13.596536+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:15.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.516+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1410066f60 msgr2=0x7f1410069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.516+0000 7f142aada700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1410066f60 0x7f1410069410 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1424151340 tx=0x7f1420008040 comp rx=0 tx=0).stop 2023-10-10T19:03:15.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.516+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 msgr2=0x7f1424157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.516+0000 7f142aada700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f1424157180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f140c004760 tx=0x7f1414015000 comp rx=0 tx=0).stop 2023-10-10T19:03:15.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.516+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 shutdown_connections 2023-10-10T19:03:15.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.517+0000 7f142aada700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1410066f60 0x7f1410069410 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.517+0000 7f142aada700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14240cc8b0 0x7f14240ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.517+0000 7f142aada700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f140c003a60 0x7f1424157180 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.517+0000 7f142aada700 1 --2- 172.21.15.175:0/393904565 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1424166570 0x7f14241576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.517+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 >> 172.21.15.175:0/393904565 conn(0x7f1424063230 msgr2=0x7f1424154130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:15.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.517+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 shutdown_connections 2023-10-10T19:03:15.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.517+0000 7f142aada700 1 -- 172.21.15.175:0/393904565 wait complete. 2023-10-10T19:03:15.520 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 134 2023-10-10T19:03:15.529 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-10T19:03:15.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:03:15.529 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:15.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.646+0000 7fab51e6a700 1 Processor -- start 2023-10-10T19:03:15.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.646+0000 7fab51e6a700 1 -- start start 2023-10-10T19:03:15.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab51e6a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fab4c0b1fc0 0x7fab4c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab51e6a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b2390 0x7fab4c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab51e6a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fab4c068bd0 0x7fab4c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab51e6a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4c150770 con 0x7fab4c068bd0 2023-10-10T19:03:15.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab51e6a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4c1574b0 con 0x7fab4c0b1fc0 2023-10-10T19:03:15.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab51e6a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0afad0 con 0x7fab4c0b2390 2023-10-10T19:03:15.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fab4c068bd0 0x7fab4c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fab4c068bd0 0x7fab4c0aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53226/0 (socket says 172.21.15.175:53226) 2023-10-10T19:03:15.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4bfff700 1 -- 172.21.15.175:0/3154796963 learned_addr learned my addr 172.21.15.175:0/3154796963 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:15.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4b7fe700 1 --2- 172.21.15.175:0/3154796963 >> v2:172.21.15.191:3300/0 conn(0x7fab4c0b1fc0 0x7fab4c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4affd700 1 --2- 172.21.15.175:0/3154796963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b2390 0x7fab4c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4bfff700 1 -- 172.21.15.175:0/3154796963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b2390 msgr2=0x7fab4c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:15.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4bfff700 1 --2- 172.21.15.175:0/3154796963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b2390 0x7fab4c068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4bfff700 1 -- 172.21.15.175:0/3154796963 >> v2:172.21.15.191:3300/0 conn(0x7fab4c0b1fc0 msgr2=0x7fab4c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:15.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4bfff700 1 --2- 172.21.15.175:0/3154796963 >> v2:172.21.15.191:3300/0 conn(0x7fab4c0b1fc0 0x7fab4c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4bfff700 1 -- 172.21.15.175:0/3154796963 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c0af4c0 con 0x7fab4c068bd0 2023-10-10T19:03:15.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.647+0000 7fab4affd700 1 --2- 172.21.15.175:0/3154796963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b2390 0x7fab4c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:15.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab4bfff700 1 --2- 172.21.15.175:0/3154796963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c068bd0 0x7fab4c0aff60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fab4c0b28c0 tx=0x7fab44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ec144622a068b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:15.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab4a7fc700 1 -- 172.21.15.175:0/3154796963 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44009e70 con 0x7fab4c068bd0 2023-10-10T19:03:15.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab4a7fc700 1 -- 172.21.15.175:0/3154796963 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4400b8f0 con 0x7fab4c068bd0 2023-10-10T19:03:15.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab4a7fc700 1 -- 172.21.15.175:0/3154796963 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4400bc90 con 0x7fab4c068bd0 2023-10-10T19:03:15.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 -- 172.21.15.175:0/3154796963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c068bd0 msgr2=0x7fab4c0aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 --2- 172.21.15.175:0/3154796963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c068bd0 0x7fab4c0aff60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fab4c0b28c0 tx=0x7fab44009250 comp rx=0 tx=0).stop 2023-10-10T19:03:15.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 -- 172.21.15.175:0/3154796963 shutdown_connections 2023-10-10T19:03:15.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 --2- 172.21.15.175:0/3154796963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c068bd0 0x7fab4c0aff60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 --2- 172.21.15.175:0/3154796963 >> v2:172.21.15.191:3300/0 conn(0x7fab4c0b1fc0 0x7fab4c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 --2- 172.21.15.175:0/3154796963 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b2390 0x7fab4c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 -- 172.21.15.175:0/3154796963 >> 172.21.15.175:0/3154796963 conn(0x7fab4c063230 msgr2=0x7fab4c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:15.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 -- 172.21.15.175:0/3154796963 shutdown_connections 2023-10-10T19:03:15.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.648+0000 7fab51e6a700 1 -- 172.21.15.175:0/3154796963 wait complete. 2023-10-10T19:03:15.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.649+0000 7fab51e6a700 1 Processor -- start 2023-10-10T19:03:15.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab51e6a700 1 -- start start 2023-10-10T19:03:15.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab51e6a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab4c068bd0 0x7fab4c0c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab51e6a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b1fc0 0x7fab4c169090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab51e6a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c0b2390 0x7fab4c0c2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab51e6a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0c1bf0 con 0x7fab4c0b2390 2023-10-10T19:03:15.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab51e6a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0c3f90 con 0x7fab4c068bd0 2023-10-10T19:03:15.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab51e6a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0c3870 con 0x7fab4c0b1fc0 2023-10-10T19:03:15.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab4bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c0b2390 0x7fab4c0c2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab4bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c0b2390 0x7fab4c0c2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53234/0 (socket says 172.21.15.175:53234) 2023-10-10T19:03:15.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab4bfff700 1 -- 172.21.15.175:0/1301906656 learned_addr learned my addr 172.21.15.175:0/1301906656 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:15.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab4b7fe700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab4c068bd0 0x7fab4c0c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.650+0000 7fab4affd700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b1fc0 0x7fab4c169090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab4bfff700 1 -- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b1fc0 msgr2=0x7fab4c169090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab4bfff700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b1fc0 0x7fab4c169090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab4bfff700 1 -- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab4c068bd0 msgr2=0x7fab4c0c0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab4bfff700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab4c068bd0 0x7fab4c0c0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab4bfff700 1 -- 172.21.15.175:0/1301906656 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab440094f0 con 0x7fab4c0b2390 2023-10-10T19:03:15.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab4bfff700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c0b2390 0x7fab4c0c2080 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fab440094c0 tx=0x7fab44017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:15.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab337fe700 1 -- 172.21.15.175:0/1301906656 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44026480 con 0x7fab4c0b2390 2023-10-10T19:03:15.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab337fe700 1 -- 172.21.15.175:0/1301906656 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4401f070 con 0x7fab4c0b2390 2023-10-10T19:03:15.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c0c3b30 con 0x7fab4c0b2390 2023-10-10T19:03:15.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab337fe700 1 -- 172.21.15.175:0/1301906656 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44026830 con 0x7fab4c0b2390 2023-10-10T19:03:15.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.651+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c0c4230 con 0x7fab4c0b2390 2023-10-10T19:03:15.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.652+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c0c1da0 con 0x7fab4c0b2390 2023-10-10T19:03:15.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.652+0000 7fab337fe700 1 -- 172.21.15.175:0/1301906656 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fab44026ba0 con 0x7fab4c0b2390 2023-10-10T19:03:15.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.652+0000 7fab337fe700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab38066f30 0x7fab380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:15.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.653+0000 7fab337fe700 1 -- 172.21.15.175:0/1301906656 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7fab44021070 con 0x7fab4c0b2390 2023-10-10T19:03:15.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.653+0000 7fab4b7fe700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab38066f30 0x7fab380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:15.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.653+0000 7fab4b7fe700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab38066f30 0x7fab380693e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00f570 tx=0x7fab3c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:15.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.657+0000 7fab337fe700 1 -- 172.21.15.175:0/1301906656 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab4c0c1da0 con 0x7fab4c0b2390 2023-10-10T19:03:15.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.886+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab4c0c1da0 con 0x7fab4c0b2390 2023-10-10T19:03:15.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.886+0000 7fab337fe700 1 -- 172.21.15.175:0/1301906656 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5454 (secure 0 0 0) 0x7fab4c0c1da0 con 0x7fab4c0b2390 2023-10-10T19:03:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:15.887 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:13.596536+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:15.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab38066f30 msgr2=0x7fab380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab38066f30 0x7fab380693e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00f570 tx=0x7fab3c00b040 comp rx=0 tx=0).stop 2023-10-10T19:03:15.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c0b2390 msgr2=0x7fab4c0c2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:15.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c0b2390 0x7fab4c0c2080 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fab440094c0 tx=0x7fab44017000 comp rx=0 tx=0).stop 2023-10-10T19:03:15.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 shutdown_connections 2023-10-10T19:03:15.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab38066f30 0x7fab380693e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab4c0b2390 0x7fab4c0c2080 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab4c068bd0 0x7fab4c0c0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 --2- 172.21.15.175:0/1301906656 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab4c0b1fc0 0x7fab4c169090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 >> 172.21.15.175:0/1301906656 conn(0x7fab4c063230 msgr2=0x7fab4c152440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 shutdown_connections 2023-10-10T19:03:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:15.889+0000 7fab51e6a700 1 -- 172.21.15.175:0/1301906656 wait complete. 2023-10-10T19:03:15.892 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 134 2023-10-10T19:03:15.901 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 134, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 134, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5357": { "addr": "172.21.15.191:6836/1945747421", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 1945747421, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1945747421, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5357, "incarnation": 114, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 22 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:03:13.596536+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 5367, "mds_3": 5357, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:03:17.903 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:18.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.030+0000 7f2ac314b700 1 Processor -- start 2023-10-10T19:03:18.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.030+0000 7f2ac314b700 1 -- start start 2023-10-10T19:03:18.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.031+0000 7f2ac314b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2abc155fb0 0x7f2abc159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:18.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.031+0000 7f2ac314b700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc1574b0 con 0x7f2abc0b2390 2023-10-10T19:03:18.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.031+0000 7f2ac314b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc0afad0 con 0x7f2abc0b1fc0 2023-10-10T19:03:18.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.031+0000 7f2ac314b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2abc150770 con 0x7f2abc155fb0 2023-10-10T19:03:18.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.031+0000 7f2ab3fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2abc0b2390 0x7f2abc0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:57734/0 (socket says 172.21.15.175:57734) 2023-10-10T19:03:18.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.031+0000 7f2ab3fff700 1 -- 172.21.15.175:0/3858028008 learned_addr learned my addr 172.21.15.175:0/3858028008 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:18.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.031+0000 7f2ac16e8700 1 --2- 172.21.15.175:0/3858028008 >> v2:172.21.15.191:3300/0 conn(0x7f2abc155fb0 0x7f2abc159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:18.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.031+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616872447 0 0) 0x7f2abc1574b0 con 0x7f2abc0b2390 2023-10-10T19:03:18.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa4003570 con 0x7f2abc0b2390 2023-10-10T19:03:18.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 87777206 0 0) 0x7f2aa4003570 con 0x7f2abc0b2390 2023-10-10T19:03:18.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 >> v2:172.21.15.191:3300/0 conn(0x7f2abc155fb0 msgr2=0x7f2abc159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:18.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 --2- 172.21.15.175:0/3858028008 >> v2:172.21.15.191:3300/0 conn(0x7f2abc155fb0 0x7f2abc159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 >> v1:172.21.15.191:6789/0 conn(0x7f2abc0b1fc0 legacy=0x7f2abc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:18.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc0af4c0 con 0x7f2abc0b2390 2023-10-10T19:03:18.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f2aac001470 con 0x7f2abc0b2390 2023-10-10T19:03:18.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 >> v1:172.21.15.175:6789/0 conn(0x7f2abc0b2390 legacy=0x7f2abc0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:18.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2aa4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:18.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2aa4004810 0x7f2aa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:18.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa40051c0 0x7f2aa400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:18.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc150770 con 0x7f2aa4003a60 2023-10-10T19:03:18.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa400c0d0 con 0x7f2aa40051c0 2023-10-10T19:03:18.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa400c290 con 0x7f2aa4004810 2023-10-10T19:03:18.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac0ee7700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2aa4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:18.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ab3fff700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2aa4004810 0x7f2aa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:18.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac16e8700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa40051c0 0x7f2aa400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:18.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac0ee7700 1 -- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2aa4004810 msgr2=0x7f2aa4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:18.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac0ee7700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2aa4004810 0x7f2aa4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac0ee7700 1 -- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa40051c0 msgr2=0x7f2aa400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:18.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac0ee7700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa40051c0 0x7f2aa400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac0ee7700 1 -- 172.21.15.175:0/3858028008 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa4003570 con 0x7f2aa4003a60 2023-10-10T19:03:18.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac16e8700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2aa40051c0 0x7f2aa400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:18.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac16e8700 1 -- 172.21.15.175:0/3858028008 reap_dead start 2023-10-10T19:03:18.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.032+0000 7f2ac0ee7700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2aa4003e70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8001fc0 tx=0x7f2ab8001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552d522dd028a944 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:18.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8001950 con 0x7f2aa4003a60 2023-10-10T19:03:18.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab800c8b0 con 0x7f2aa4003a60 2023-10-10T19:03:18.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ab37fe700 1 -- 172.21.15.175:0/3858028008 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800cc50 con 0x7f2aa4003a60 2023-10-10T19:03:18.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ac314b700 1 -- 172.21.15.175:0/3858028008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 msgr2=0x7f2aa4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:18.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ac314b700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2aa4003e70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8001fc0 tx=0x7f2ab8001640 comp rx=0 tx=0).stop 2023-10-10T19:03:18.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ac314b700 1 -- 172.21.15.175:0/3858028008 shutdown_connections 2023-10-10T19:03:18.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ac314b700 1 --2- 172.21.15.175:0/3858028008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2aa4003e70 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ac314b700 1 -- 172.21.15.175:0/3858028008 >> 172.21.15.175:0/3858028008 conn(0x7f2abc063230 msgr2=0x7f2abc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:18.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ac314b700 1 -- 172.21.15.175:0/3858028008 shutdown_connections 2023-10-10T19:03:18.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.033+0000 7f2ac314b700 1 -- 172.21.15.175:0/3858028008 wait complete. 2023-10-10T19:03:18.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.034+0000 7f2ac314b700 1 Processor -- start 2023-10-10T19:03:18.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.034+0000 7f2ac314b700 1 -- start start 2023-10-10T19:03:18.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac314b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2abc0c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:18.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac314b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc166120 0x7f2abc166570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:18.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac314b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc169070 0x7f2abc1694c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:18.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac314b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc15c450 con 0x7f2aa4003a60 2023-10-10T19:03:18.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac314b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc165c90 con 0x7f2abc169070 2023-10-10T19:03:18.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac314b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc153460 con 0x7f2abc166120 2023-10-10T19:03:18.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac0ee7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2abc0c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:18.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac0ee7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2abc0c7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53254/0 (socket says 172.21.15.175:53254) 2023-10-10T19:03:18.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac0ee7700 1 -- 172.21.15.175:0/3769878057 learned_addr learned my addr 172.21.15.175:0/3769878057 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:18.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac16e8700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc169070 0x7f2abc1694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:18.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ab3fff700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc166120 0x7f2abc166570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:18.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac0ee7700 1 -- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc166120 msgr2=0x7f2abc166570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:18.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac0ee7700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc166120 0x7f2abc166570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac0ee7700 1 -- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc169070 msgr2=0x7f2abc1694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:18.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac0ee7700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc169070 0x7f2abc1694c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac0ee7700 1 -- 172.21.15.175:0/3769878057 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc1539b0 con 0x7f2aa4003a60 2023-10-10T19:03:18.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.035+0000 7f2ac16e8700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc169070 0x7f2abc1694c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:18.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.036+0000 7f2ac0ee7700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2abc0c7de0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8012990 tx=0x7f2ab8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:18.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.036+0000 7f2ab0ff9700 1 -- 172.21.15.175:0/3769878057 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8001950 con 0x7f2aa4003a60 2023-10-10T19:03:18.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.036+0000 7f2ab0ff9700 1 -- 172.21.15.175:0/3769878057 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab800c6d0 con 0x7f2aa4003a60 2023-10-10T19:03:18.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.036+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2abc15a960 con 0x7f2aa4003a60 2023-10-10T19:03:18.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.036+0000 7f2ab0ff9700 1 -- 172.21.15.175:0/3769878057 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8026630 con 0x7f2aa4003a60 2023-10-10T19:03:18.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.036+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2abc152620 con 0x7f2aa4003a60 2023-10-10T19:03:18.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.037+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2abc067570 con 0x7f2aa4003a60 2023-10-10T19:03:18.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.037+0000 7f2ab0ff9700 1 -- 172.21.15.175:0/3769878057 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2ab8019030 con 0x7f2aa4003a60 2023-10-10T19:03:18.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.037+0000 7f2ab0ff9700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2aa8066f60 0x7f2aa8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:18.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.037+0000 7f2ab0ff9700 1 -- 172.21.15.175:0/3769878057 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f2ab8086460 con 0x7f2aa4003a60 2023-10-10T19:03:18.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.038+0000 7f2ab3fff700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2aa8066f60 0x7f2aa8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:18.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.038+0000 7f2ab3fff700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2aa8066f60 0x7f2aa8069410 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0015a0 tx=0x7f2aac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:18.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.042+0000 7f2ab0ff9700 1 -- 172.21.15.175:0/3769878057 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ab8053350 con 0x7f2aa4003a60 2023-10-10T19:03:18.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.267+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2abc15c450 con 0x7f2aa4003a60 2023-10-10T19:03:18.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.267+0000 7f2ab0ff9700 1 -- 172.21.15.175:0/3769878057 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5454 (secure 0 0 0) 0x7f2ab801ae60 con 0x7f2aa4003a60 2023-10-10T19:03:18.268 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:18.268 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:13.596536+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2aa8066f60 msgr2=0x7f2aa8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2aa8066f60 0x7f2aa8069410 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0015a0 tx=0x7f2aac009040 comp rx=0 tx=0).stop 2023-10-10T19:03:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 msgr2=0x7f2abc0c7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2abc0c7de0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8012990 tx=0x7f2ab8015000 comp rx=0 tx=0).stop 2023-10-10T19:03:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 shutdown_connections 2023-10-10T19:03:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2aa8066f60 0x7f2aa8069410 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2abc169070 0x7f2abc1694c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2aa4003a60 0x7f2abc0c7de0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 --2- 172.21.15.175:0/3769878057 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2abc166120 0x7f2abc166570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.269+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 >> 172.21.15.175:0/3769878057 conn(0x7f2abc063230 msgr2=0x7f2abc0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.270+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 shutdown_connections 2023-10-10T19:03:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:18.270+0000 7f2ac314b700 1 -- 172.21.15.175:0/3769878057 wait complete. 2023-10-10T19:03:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 134 2023-10-10T19:03:18.281 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:03:20.282 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:20.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.403+0000 7f94b355b700 1 Processor -- start 2023-10-10T19:03:20.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.403+0000 7f94b355b700 1 -- start start 2023-10-10T19:03:20.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b355b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94ac068c00 0x7f94ac159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:20.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b355b700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac067fa0 con 0x7f94ac08a100 2023-10-10T19:03:20.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b355b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac156b70 con 0x7f94ac14fd90 2023-10-10T19:03:20.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b355b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac0c5600 con 0x7f94ac068c00 2023-10-10T19:03:20.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b1d58700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f94ac08a100 0x7f94ac0684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:57740/0 (socket says 172.21.15.175:57740) 2023-10-10T19:03:20.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b1d58700 1 -- 172.21.15.175:0/2881629464 learned_addr learned my addr 172.21.15.175:0/2881629464 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:20.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b2d5a700 1 --2- 172.21.15.175:0/2881629464 >> v2:172.21.15.175:3300/0 conn(0x7f94ac068c00 0x7f94ac159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:20.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b1557700 1 -- 172.21.15.175:0/2881629464 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873687517 0 0) 0x7f94ac067fa0 con 0x7f94ac08a100 2023-10-10T19:03:20.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b1557700 1 -- 172.21.15.175:0/2881629464 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9494003570 con 0x7f94ac08a100 2023-10-10T19:03:20.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b2d5a700 1 -- 172.21.15.175:0/2881629464 >> v1:172.21.15.191:6789/0 conn(0x7f94ac14fd90 legacy=0x7f94ac0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:20.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b2d5a700 1 -- 172.21.15.175:0/2881629464 >> v1:172.21.15.175:6789/0 conn(0x7f94ac08a100 legacy=0x7f94ac0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:20.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.404+0000 7f94b2d5a700 1 -- 172.21.15.175:0/2881629464 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac150160 con 0x7f94ac068c00 2023-10-10T19:03:20.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b2d5a700 1 --2- 172.21.15.175:0/2881629464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac159c30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f94a40098e0 tx=0x7f94a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3fdcf6788d92d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:20.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b1557700 1 -- 172.21.15.175:0/2881629464 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a400b640 con 0x7f94ac068c00 2023-10-10T19:03:20.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b1557700 1 -- 172.21.15.175:0/2881629464 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a400b7a0 con 0x7f94ac068c00 2023-10-10T19:03:20.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b1557700 1 -- 172.21.15.175:0/2881629464 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a400bb40 con 0x7f94ac068c00 2023-10-10T19:03:20.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b355b700 1 -- 172.21.15.175:0/2881629464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 msgr2=0x7f94ac159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:20.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b355b700 1 --2- 172.21.15.175:0/2881629464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac159c30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f94a40098e0 tx=0x7f94a40091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:20.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b355b700 1 -- 172.21.15.175:0/2881629464 shutdown_connections 2023-10-10T19:03:20.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b355b700 1 --2- 172.21.15.175:0/2881629464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac159c30 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:20.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b355b700 1 -- 172.21.15.175:0/2881629464 >> 172.21.15.175:0/2881629464 conn(0x7f94ac063d00 msgr2=0x7f94ac0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:20.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b355b700 1 -- 172.21.15.175:0/2881629464 shutdown_connections 2023-10-10T19:03:20.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.405+0000 7f94b355b700 1 -- 172.21.15.175:0/2881629464 wait complete. 2023-10-10T19:03:20.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.406+0000 7f94b355b700 1 Processor -- start 2023-10-10T19:03:20.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b355b700 1 -- start start 2023-10-10T19:03:20.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b355b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:20.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b355b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94ac08a100 0x7f94ac157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:20.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b355b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94ac14fd90 0x7f94ac152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:20.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b355b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac155990 con 0x7f94ac068c00 2023-10-10T19:03:20.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b355b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1544a0 con 0x7f94ac14fd90 2023-10-10T19:03:20.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b355b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac155460 con 0x7f94ac08a100 2023-10-10T19:03:20.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b2559700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:20.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b2559700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac150d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53274/0 (socket says 172.21.15.175:53274) 2023-10-10T19:03:20.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b2559700 1 -- 172.21.15.175:0/2846714932 learned_addr learned my addr 172.21.15.175:0/2846714932 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:20.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b2d5a700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94ac14fd90 0x7f94ac152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:20.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.407+0000 7f94b1d58700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94ac08a100 0x7f94ac157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:20.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b2559700 1 -- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94ac08a100 msgr2=0x7f94ac157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:20.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b2559700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94ac08a100 0x7f94ac157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:20.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b2559700 1 -- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94ac14fd90 msgr2=0x7f94ac152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:20.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b2559700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94ac14fd90 0x7f94ac152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:20.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b2559700 1 -- 172.21.15.175:0/2846714932 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4009420 con 0x7f94ac068c00 2023-10-10T19:03:20.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b1d58700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94ac08a100 0x7f94ac157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:20.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b2559700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac150d10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f949c00c8e0 tx=0x7f949c001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:20.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94aa7fc700 1 -- 172.21.15.175:0/2846714932 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c006ba0 con 0x7f94ac068c00 2023-10-10T19:03:20.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94aa7fc700 1 -- 172.21.15.175:0/2846714932 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c006d00 con 0x7f94ac068c00 2023-10-10T19:03:20.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac0ca9f0 con 0x7f94ac068c00 2023-10-10T19:03:20.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94aa7fc700 1 -- 172.21.15.175:0/2846714932 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c012880 con 0x7f94ac068c00 2023-10-10T19:03:20.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.408+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac154690 con 0x7f94ac068c00 2023-10-10T19:03:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.409+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac154a30 con 0x7f94ac068c00 2023-10-10T19:03:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.409+0000 7f94aa7fc700 1 -- 172.21.15.175:0/2846714932 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f949c012cb0 con 0x7f94ac068c00 2023-10-10T19:03:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.410+0000 7f94aa7fc700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9498066f30 0x7f94980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.410+0000 7f94aa7fc700 1 -- 172.21.15.175:0/2846714932 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f949c0846b0 con 0x7f94ac068c00 2023-10-10T19:03:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.410+0000 7f94b1d58700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9498066f30 0x7f94980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.410+0000 7f94b1d58700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9498066f30 0x7f94980693e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f94ac14e140 tx=0x7f94a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:20.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.414+0000 7f94aa7fc700 1 -- 172.21.15.175:0/2846714932 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f94ac154a30 con 0x7f94ac068c00 2023-10-10T19:03:20.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.639+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94ac154a30 con 0x7f94ac068c00 2023-10-10T19:03:20.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.640+0000 7f94aa7fc700 1 -- 172.21.15.175:0/2846714932 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5456 (secure 0 0 0) 0x7f94ac154a30 con 0x7f94ac068c00 2023-10-10T19:03:20.640 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:20.641 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:20.626884+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:resolve","state_seq":35,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:20.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9498066f30 msgr2=0x7f94980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:20.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9498066f30 0x7f94980693e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f94ac14e140 tx=0x7f94a0009040 comp rx=0 tx=0).stop 2023-10-10T19:03:20.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 msgr2=0x7f94ac150d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:20.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac150d10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f949c00c8e0 tx=0x7f949c001710 comp rx=0 tx=0).stop 2023-10-10T19:03:20.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 shutdown_connections 2023-10-10T19:03:20.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9498066f30 0x7f94980693e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:20.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94ac14fd90 0x7f94ac152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:20.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94ac068c00 0x7f94ac150d10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:20.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 --2- 172.21.15.175:0/2846714932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94ac08a100 0x7f94ac157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:20.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 >> 172.21.15.175:0/2846714932 conn(0x7f94ac063d00 msgr2=0x7f94ac0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:20.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 shutdown_connections 2023-10-10T19:03:20.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:20.642+0000 7f94b355b700 1 -- 172.21.15.175:0/2846714932 wait complete. 2023-10-10T19:03:20.645 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 135 2023-10-10T19:03:20.654 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:03:20.907 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:03:20.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:03:22.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:22.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.779+0000 7f74fb52d700 1 Processor -- start 2023-10-10T19:03:22.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.779+0000 7f74fb52d700 1 -- start start 2023-10-10T19:03:22.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.780+0000 7f74fb52d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f40c35e0 0x7f74f4159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:22.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.780+0000 7f74fb52d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f40c5e40 con 0x7f74f40c3210 2023-10-10T19:03:22.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.780+0000 7f74fb52d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f40c9880 con 0x7f74f40c9390 2023-10-10T19:03:22.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.780+0000 7f74fb52d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40c4c50 con 0x7f74f40c35e0 2023-10-10T19:03:22.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.780+0000 7f74f92c9700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f74f40c3210 0x7f74f40c3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:57756/0 (socket says 172.21.15.175:57756) 2023-10-10T19:03:22.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.780+0000 7f74f92c9700 1 -- 172.21.15.175:0/2380449463 learned_addr learned my addr 172.21.15.175:0/2380449463 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:22.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f40c35e0 0x7f74f4159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:22.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256157106 0 0) 0x7f74f40c5e40 con 0x7f74f40c3210 2023-10-10T19:03:22.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e0003570 con 0x7f74f40c3210 2023-10-10T19:03:22.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731362819 0 0) 0x7f74f40c9880 con 0x7f74f40c9390 2023-10-10T19:03:22.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f40c5e40 con 0x7f74f40c9390 2023-10-10T19:03:22.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1038061163 0 0) 0x7f74e0003570 con 0x7f74f40c3210 2023-10-10T19:03:22.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f40c35e0 msgr2=0x7f74f4159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:22.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f40c35e0 0x7f74f4159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:22.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 >> v1:172.21.15.191:6789/0 conn(0x7f74f40c9390 legacy=0x7f74f40c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:22.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f40c9b60 con 0x7f74f40c3210 2023-10-10T19:03:22.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f74f0001470 con 0x7f74f40c3210 2023-10-10T19:03:22.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 >> v1:172.21.15.175:6789/0 conn(0x7f74f40c3210 legacy=0x7f74f40c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:22.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74e0003c20 0x7f74e0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:22.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:22.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74e00051c0 0x7f74e000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:22.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40c4c50 con 0x7f74e0004810 2023-10-10T19:03:22.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e000c020 con 0x7f74e00051c0 2023-10-10T19:03:22.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e000c1e0 con 0x7f74e0003c20 2023-10-10T19:03:22.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f40c35e0 0x7f74f4159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:03:22.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f92c9700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74e0003c20 0x7f74e0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:22.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:22.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f9aca700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74e00051c0 0x7f74e000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:22.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f8ac8700 1 -- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74e0003c20 msgr2=0x7f74e0004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:22.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74e0003c20 0x7f74e0004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:22.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f8ac8700 1 -- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74e00051c0 msgr2=0x7f74e000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:22.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74e00051c0 0x7f74e000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:22.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.781+0000 7f74f8ac8700 1 -- 172.21.15.175:0/2380449463 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e0003570 con 0x7f74e0004810 2023-10-10T19:03:22.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74e0004c80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74f406e320 tx=0x7f74e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02ae76b40d55769 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:22.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e400e6e0 con 0x7f74e0004810 2023-10-10T19:03:22.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e400e840 con 0x7f74e0004810 2023-10-10T19:03:22.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74ebfff700 1 -- 172.21.15.175:0/2380449463 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e400ebe0 con 0x7f74e0004810 2023-10-10T19:03:22.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74fb52d700 1 -- 172.21.15.175:0/2380449463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 msgr2=0x7f74e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:22.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74fb52d700 1 --2- 172.21.15.175:0/2380449463 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74e0004c80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74f406e320 tx=0x7f74e40091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:22.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74f9aca700 1 -- 172.21.15.175:0/2380449463 reap_dead start 2023-10-10T19:03:22.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74fb52d700 1 -- 172.21.15.175:0/2380449463 shutdown_connections 2023-10-10T19:03:22.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74fb52d700 1 -- 172.21.15.175:0/2380449463 >> 172.21.15.175:0/2380449463 conn(0x7f74f406c110 msgr2=0x7f74f406c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:22.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74fb52d700 1 -- 172.21.15.175:0/2380449463 shutdown_connections 2023-10-10T19:03:22.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.782+0000 7f74fb52d700 1 -- 172.21.15.175:0/2380449463 wait complete. 2023-10-10T19:03:22.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.783+0000 7f74fb52d700 1 Processor -- start 2023-10-10T19:03:22.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.783+0000 7f74fb52d700 1 -- start start 2023-10-10T19:03:22.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74fb52d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74f4150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:22.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74fb52d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f4153eb0 0x7f74f4166830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:22.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74fb52d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f40c7d90 0x7f74f413ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:22.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74fb52d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40ca400 con 0x7f74e0004810 2023-10-10T19:03:22.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74fb52d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40c5840 con 0x7f74f40c7d90 2023-10-10T19:03:22.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74fb52d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f416a420 con 0x7f74f4153eb0 2023-10-10T19:03:22.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f92c9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74f4150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:22.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f92c9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74f4150790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53294/0 (socket says 172.21.15.175:53294) 2023-10-10T19:03:22.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f92c9700 1 -- 172.21.15.175:0/316647084 learned_addr learned my addr 172.21.15.175:0/316647084 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:22.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f4153eb0 0x7f74f4166830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:22.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f9aca700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f40c7d90 0x7f74f413ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:22.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f92c9700 1 -- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f4153eb0 msgr2=0x7f74f4166830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:22.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f92c9700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f4153eb0 0x7f74f4166830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:22.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f92c9700 1 -- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f40c7d90 msgr2=0x7f74f413ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:22.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.784+0000 7f74f92c9700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f40c7d90 0x7f74f413ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:22.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.785+0000 7f74f92c9700 1 -- 172.21.15.175:0/316647084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f41663a0 con 0x7f74e0004810 2023-10-10T19:03:22.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.785+0000 7f74f9aca700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f40c7d90 0x7f74f413ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:22.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.785+0000 7f74f92c9700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74f4150790 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f74f0000e10 tx=0x7f74f4071740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:22.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.785+0000 7f74e97fa700 1 -- 172.21.15.175:0/316647084 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0018030 con 0x7f74e0004810 2023-10-10T19:03:22.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.785+0000 7f74e97fa700 1 -- 172.21.15.175:0/316647084 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f00077f0 con 0x7f74e0004810 2023-10-10T19:03:22.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.785+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f4154320 con 0x7f74e0004810 2023-10-10T19:03:22.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.785+0000 7f74e97fa700 1 -- 172.21.15.175:0/316647084 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f000fb20 con 0x7f74e0004810 2023-10-10T19:03:22.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.785+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f416a160 con 0x7f74e0004810 2023-10-10T19:03:22.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.786+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f4067570 con 0x7f74e0004810 2023-10-10T19:03:22.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.786+0000 7f74e97fa700 1 -- 172.21.15.175:0/316647084 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f74f001b070 con 0x7f74e0004810 2023-10-10T19:03:22.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.787+0000 7f74e97fa700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74d8066f30 0x7f74d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:22.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.787+0000 7f74e97fa700 1 -- 172.21.15.175:0/316647084 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f74f0019070 con 0x7f74e0004810 2023-10-10T19:03:22.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.787+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74d8066f30 0x7f74d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:22.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.787+0000 7f74f8ac8700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74d8066f30 0x7f74d80693e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f74e40091e0 tx=0x7f74e4019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:22.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:22.791+0000 7f74e97fa700 1 -- 172.21.15.175:0/316647084 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74f00516f0 con 0x7f74e0004810 2023-10-10T19:03:23.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.018+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74f40c9e40 con 0x7f74e0004810 2023-10-10T19:03:23.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.018+0000 7f74e97fa700 1 -- 172.21.15.175:0/316647084 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5456 (secure 0 0 0) 0x7f74f0054d10 con 0x7f74e0004810 2023-10-10T19:03:23.019 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:23.019 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:20.626884+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:resolve","state_seq":35,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:23.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74d8066f30 msgr2=0x7f74d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:23.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74d8066f30 0x7f74d80693e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f74e40091e0 tx=0x7f74e4019000 comp rx=0 tx=0).stop 2023-10-10T19:03:23.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 msgr2=0x7f74f4150790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:23.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74f4150790 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f74f0000e10 tx=0x7f74f4071740 comp rx=0 tx=0).stop 2023-10-10T19:03:23.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 shutdown_connections 2023-10-10T19:03:23.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f74d8066f30 0x7f74d80693e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:23.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f40c7d90 0x7f74f413ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:23.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e0004810 0x7f74f4150790 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:23.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 --2- 172.21.15.175:0/316647084 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74f4153eb0 0x7f74f4166830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:23.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.020+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 >> 172.21.15.175:0/316647084 conn(0x7f74f406c110 msgr2=0x7f74f40c0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:23.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.021+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 shutdown_connections 2023-10-10T19:03:23.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:23.021+0000 7f74fb52d700 1 -- 172.21.15.175:0/316647084 wait complete. 2023-10-10T19:03:23.024 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 135 2023-10-10T19:03:23.033 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:03:25.033 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:25.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.160+0000 7f4d62d8d700 1 Processor -- start 2023-10-10T19:03:25.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.160+0000 7f4d62d8d700 1 -- start start 2023-10-10T19:03:25.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.161+0000 7f4d62d8d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4d5c0b2390 0x7f4d5c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:25.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.161+0000 7f4d62d8d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0aff60 0x7f4d5c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:25.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.161+0000 7f4d62d8d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5c1574b0 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.161+0000 7f4d62d8d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d5c150770 con 0x7f4d5c0b2390 2023-10-10T19:03:25.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.161+0000 7f4d62d8d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c0afad0 con 0x7f4d5c0aff60 2023-10-10T19:03:25.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.161+0000 7f4d5bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4d5c0b2390 0x7f4d5c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:25.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.161+0000 7f4d5bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4d5c0b2390 0x7f4d5c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42148/0 (socket says 172.21.15.175:42148) 2023-10-10T19:03:25.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.161+0000 7f4d5bfff700 1 -- 172.21.15.175:0/1671211415 learned_addr learned my addr 172.21.15.175:0/1671211415 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:25.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d6132a700 1 --2- 172.21.15.175:0/1671211415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0aff60 0x7f4d5c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:25.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d5bfff700 1 -- 172.21.15.175:0/1671211415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0aff60 msgr2=0x7f4d5c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:25.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d5bfff700 1 --2- 172.21.15.175:0/1671211415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0aff60 0x7f4d5c0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d5bfff700 1 -- 172.21.15.175:0/1671211415 >> v1:172.21.15.191:6789/0 conn(0x7f4d5c0b1fc0 legacy=0x7f4d5c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:25.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d5bfff700 1 -- 172.21.15.175:0/1671211415 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c0af4c0 con 0x7f4d5c0b2390 2023-10-10T19:03:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d5bfff700 1 --2- 172.21.15.175:0/1671211415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b2390 0x7f4d5c155fb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4d500099b0 tx=0x7f4d50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244a8cbf239314bc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d5b7fe700 1 -- 172.21.15.175:0/1671211415 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50009ea0 con 0x7f4d5c0b2390 2023-10-10T19:03:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d5b7fe700 1 -- 172.21.15.175:0/1671211415 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5000b8f0 con 0x7f4d5c0b2390 2023-10-10T19:03:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.162+0000 7f4d5b7fe700 1 -- 172.21.15.175:0/1671211415 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5000bc90 con 0x7f4d5c0b2390 2023-10-10T19:03:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.163+0000 7f4d62d8d700 1 -- 172.21.15.175:0/1671211415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b2390 msgr2=0x7f4d5c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.163+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/1671211415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b2390 0x7f4d5c155fb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4d500099b0 tx=0x7f4d50009250 comp rx=0 tx=0).stop 2023-10-10T19:03:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.163+0000 7f4d62d8d700 1 -- 172.21.15.175:0/1671211415 shutdown_connections 2023-10-10T19:03:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.163+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/1671211415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0aff60 0x7f4d5c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.163+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/1671211415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b2390 0x7f4d5c155fb0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.163+0000 7f4d62d8d700 1 -- 172.21.15.175:0/1671211415 >> 172.21.15.175:0/1671211415 conn(0x7f4d5c063230 msgr2=0x7f4d5c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:25.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.163+0000 7f4d62d8d700 1 -- 172.21.15.175:0/1671211415 shutdown_connections 2023-10-10T19:03:25.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.163+0000 7f4d62d8d700 1 -- 172.21.15.175:0/1671211415 wait complete. 2023-10-10T19:03:25.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.164+0000 7f4d62d8d700 1 Processor -- start 2023-10-10T19:03:25.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.164+0000 7f4d62d8d700 1 -- start start 2023-10-10T19:03:25.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d62d8d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c0aff60 0x7f4d5c15bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:25.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d62d8d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b1fc0 0x7f4d5c159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:25.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d62d8d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0b2390 0x7f4d5c159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:25.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d62d8d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c0cd4b0 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d62d8d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c166970 con 0x7f4d5c0aff60 2023-10-10T19:03:25.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d62d8d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c166c50 con 0x7f4d5c0b2390 2023-10-10T19:03:25.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b1fc0 0x7f4d5c159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:25.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b1fc0 0x7f4d5c159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42154/0 (socket says 172.21.15.175:42154) 2023-10-10T19:03:25.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 -- 172.21.15.175:0/233255119 learned_addr learned my addr 172.21.15.175:0/233255119 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:25.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d6132a700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0b2390 0x7f4d5c159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:25.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d60b29700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c0aff60 0x7f4d5c15bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:25.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 -- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0b2390 msgr2=0x7f4d5c159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:25.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0b2390 0x7f4d5c159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 -- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c0aff60 msgr2=0x7f4d5c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:25.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c0aff60 0x7f4d5c15bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 -- 172.21.15.175:0/233255119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500094f0 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d5bfff700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b1fc0 0x7f4d5c159420 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c06fd90 tx=0x7f4d50017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d58ff9700 1 -- 172.21.15.175:0/233255119 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5000bc70 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d58ff9700 1 -- 172.21.15.175:0/233255119 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5001a450 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5c14ce50 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d58ff9700 1 -- 172.21.15.175:0/233255119 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5001a810 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.165+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5c158f00 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.166+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5c067570 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.167+0000 7f4d58ff9700 1 -- 172.21.15.175:0/233255119 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4d5001ab80 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.167+0000 7f4d58ff9700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d48066f30 0x7f4d480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.167+0000 7f4d58ff9700 1 -- 172.21.15.175:0/233255119 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f4d50086360 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.167+0000 7f4d60b29700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d48066f30 0x7f4d480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:25.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.168+0000 7f4d60b29700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d48066f30 0x7f4d480693e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c003860 tx=0x7f4d4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:25.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.171+0000 7f4d58ff9700 1 -- 172.21.15.175:0/233255119 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d500532d0 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.397+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d5c068da0 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.397+0000 7f4d58ff9700 1 -- 172.21.15.175:0/233255119 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5456 (secure 0 0 0) 0x7f4d500568f0 con 0x7f4d5c0b1fc0 2023-10-10T19:03:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:25.398 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:20.626884+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:resolve","state_seq":35,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:25.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d48066f30 msgr2=0x7f4d480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:25.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d48066f30 0x7f4d480693e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c003860 tx=0x7f4d4c009040 comp rx=0 tx=0).stop 2023-10-10T19:03:25.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b1fc0 msgr2=0x7f4d5c159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:25.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b1fc0 0x7f4d5c159420 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c06fd90 tx=0x7f4d50017000 comp rx=0 tx=0).stop 2023-10-10T19:03:25.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 shutdown_connections 2023-10-10T19:03:25.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d48066f30 0x7f4d480693e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0b2390 0x7f4d5c159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c0aff60 0x7f4d5c15bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 --2- 172.21.15.175:0/233255119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d5c0b1fc0 0x7f4d5c159420 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:25.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 >> 172.21.15.175:0/233255119 conn(0x7f4d5c063230 msgr2=0x7f4d5c157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:25.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 shutdown_connections 2023-10-10T19:03:25.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:25.400+0000 7f4d62d8d700 1 -- 172.21.15.175:0/233255119 wait complete. 2023-10-10T19:03:25.403 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 135 2023-10-10T19:03:25.411 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:03:27.412 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:27.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.541+0000 7fda2b355700 1 Processor -- start 2023-10-10T19:03:27.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.541+0000 7fda2b355700 1 -- start start 2023-10-10T19:03:27.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda2b355700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fda240b2390 0x7fda24155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:27.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda2b355700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda240aff60 0x7fda240b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda2b355700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda241574b0 con 0x7fda240b1fc0 2023-10-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda2b355700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda24150770 con 0x7fda240b2390 2023-10-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda2b355700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda240afad0 con 0x7fda240aff60 2023-10-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda288f0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fda240b2390 0x7fda24155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda288f0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fda240b2390 0x7fda24155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42158/0 (socket says 172.21.15.175:42158) 2023-10-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda288f0700 1 -- 172.21.15.175:0/1484147699 learned_addr learned my addr 172.21.15.175:0/1484147699 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda298f2700 1 --2- 172.21.15.175:0/1484147699 >> v2:172.21.15.191:3300/0 conn(0x7fda240aff60 0x7fda240b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda288f0700 1 -- 172.21.15.175:0/1484147699 >> v2:172.21.15.191:3300/0 conn(0x7fda240aff60 msgr2=0x7fda240b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda288f0700 1 --2- 172.21.15.175:0/1484147699 >> v2:172.21.15.191:3300/0 conn(0x7fda240aff60 0x7fda240b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda288f0700 1 -- 172.21.15.175:0/1484147699 >> v1:172.21.15.191:6789/0 conn(0x7fda240b1fc0 legacy=0x7fda24068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda288f0700 1 -- 172.21.15.175:0/1484147699 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda240af4c0 con 0x7fda240b2390 2023-10-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.542+0000 7fda288f0700 1 --2- 172.21.15.175:0/1484147699 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda24155fb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fda240641f0 tx=0x7fda14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef3d7f019fd92f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda1bfff700 1 -- 172.21.15.175:0/1484147699 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1400b8f0 con 0x7fda240b2390 2023-10-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda1bfff700 1 -- 172.21.15.175:0/1484147699 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1400ba50 con 0x7fda240b2390 2023-10-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda1bfff700 1 -- 172.21.15.175:0/1484147699 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1400bdf0 con 0x7fda240b2390 2023-10-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda2b355700 1 -- 172.21.15.175:0/1484147699 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 msgr2=0x7fda24155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda2b355700 1 --2- 172.21.15.175:0/1484147699 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda24155fb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fda240641f0 tx=0x7fda14009250 comp rx=0 tx=0).stop 2023-10-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda2b355700 1 -- 172.21.15.175:0/1484147699 shutdown_connections 2023-10-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda2b355700 1 --2- 172.21.15.175:0/1484147699 >> v2:172.21.15.191:3300/0 conn(0x7fda240aff60 0x7fda240b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda2b355700 1 --2- 172.21.15.175:0/1484147699 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda24155fb0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda2b355700 1 -- 172.21.15.175:0/1484147699 >> 172.21.15.175:0/1484147699 conn(0x7fda24063230 msgr2=0x7fda24063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:27.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda2b355700 1 -- 172.21.15.175:0/1484147699 shutdown_connections 2023-10-10T19:03:27.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.543+0000 7fda2b355700 1 -- 172.21.15.175:0/1484147699 wait complete. 2023-10-10T19:03:27.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.544+0000 7fda2b355700 1 Processor -- start 2023-10-10T19:03:27.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.544+0000 7fda2b355700 1 -- start start 2023-10-10T19:03:27.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda2b355700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda240aff60 0x7fda241550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:27.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda2b355700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda240b1fc0 0x7fda24155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:27.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda2b355700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda241562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:27.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda2b355700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda240cbee0 con 0x7fda240b2390 2023-10-10T19:03:27.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda2b355700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda24169640 con 0x7fda240b1fc0 2023-10-10T19:03:27.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda2b355700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda240cd1b0 con 0x7fda240aff60 2023-10-10T19:03:27.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda298f2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda241562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:27.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda298f2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda241562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42174/0 (socket says 172.21.15.175:42174) 2023-10-10T19:03:27.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda298f2700 1 -- 172.21.15.175:0/4243972406 learned_addr learned my addr 172.21.15.175:0/4243972406 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:27.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda288f0700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda240b1fc0 0x7fda24155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:27.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.545+0000 7fda290f1700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda240aff60 0x7fda241550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:27.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda298f2700 1 -- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda240aff60 msgr2=0x7fda241550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:27.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda298f2700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda240aff60 0x7fda241550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda298f2700 1 -- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda240b1fc0 msgr2=0x7fda24155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:27.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda298f2700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda240b1fc0 0x7fda24155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda298f2700 1 -- 172.21.15.175:0/4243972406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda140094f0 con 0x7fda240b2390 2023-10-10T19:03:27.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda298f2700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda241562b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fda2000d890 tx=0x7fda20008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:27.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda197fa700 1 -- 172.21.15.175:0/4243972406 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda200125f0 con 0x7fda240b2390 2023-10-10T19:03:27.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda197fa700 1 -- 172.21.15.175:0/4243972406 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda20012770 con 0x7fda240b2390 2023-10-10T19:03:27.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda197fa700 1 -- 172.21.15.175:0/4243972406 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20012b30 con 0x7fda240b2390 2023-10-10T19:03:27.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda24169890 con 0x7fda240b2390 2023-10-10T19:03:27.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.546+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda24169e50 con 0x7fda240b2390 2023-10-10T19:03:27.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.547+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda24067570 con 0x7fda240b2390 2023-10-10T19:03:27.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.548+0000 7fda197fa700 1 -- 172.21.15.175:0/4243972406 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fda2001e460 con 0x7fda240b2390 2023-10-10T19:03:27.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.548+0000 7fda197fa700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda08066f30 0x7fda080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:27.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.548+0000 7fda197fa700 1 -- 172.21.15.175:0/4243972406 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7fda20084e00 con 0x7fda240b2390 2023-10-10T19:03:27.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.548+0000 7fda290f1700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda08066f30 0x7fda080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:27.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.549+0000 7fda290f1700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda08066f30 0x7fda080693e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fda1c007fd0 tx=0x7fda1c0016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:27.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.552+0000 7fda197fa700 1 -- 172.21.15.175:0/4243972406 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda20051cf0 con 0x7fda240b2390 2023-10-10T19:03:27.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.777+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda24156fd0 con 0x7fda240b2390 2023-10-10T19:03:27.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.778+0000 7fda197fa700 1 -- 172.21.15.175:0/4243972406 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5456 (secure 0 0 0) 0x7fda20055310 con 0x7fda240b2390 2023-10-10T19:03:27.779 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:27.779 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:27.660970+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:27.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda08066f30 msgr2=0x7fda080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:27.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda08066f30 0x7fda080693e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fda1c007fd0 tx=0x7fda1c0016f0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 msgr2=0x7fda241562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:27.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda241562b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fda2000d890 tx=0x7fda20008040 comp rx=0 tx=0).stop 2023-10-10T19:03:27.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 shutdown_connections 2023-10-10T19:03:27.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda08066f30 0x7fda080693e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda240b2390 0x7fda241562b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda240aff60 0x7fda241550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 --2- 172.21.15.175:0/4243972406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda240b1fc0 0x7fda24155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:27.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 >> 172.21.15.175:0/4243972406 conn(0x7fda24063230 msgr2=0x7fda24068690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:27.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 shutdown_connections 2023-10-10T19:03:27.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:27.780+0000 7fda2b355700 1 -- 172.21.15.175:0/4243972406 wait complete. 2023-10-10T19:03:27.783 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 137 2023-10-10T19:03:27.792 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 137, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 137, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5357": { "addr": "172.21.15.191:6836/1945747421", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 1945747421, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 1945747421, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5357, "incarnation": 114, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 22 }, "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:03:27.660970+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 5367, "mds_3": 5357, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:03:29.794 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:29.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.922+0000 7f10f83aa700 1 Processor -- start 2023-10-10T19:03:29.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.922+0000 7f10f83aa700 1 -- start start 2023-10-10T19:03:29.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f83aa700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f10f0159d70 0x7f10f00c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:29.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f83aa700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f0156800 con 0x7f10f014fbf0 2023-10-10T19:03:29.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f83aa700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f0063e30 con 0x7f10f014ffc0 2023-10-10T19:03:29.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f83aa700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f00c59e0 con 0x7f10f0159d70 2023-10-10T19:03:29.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f6146700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f10f014fbf0 0x7f10f0063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39556/0 (socket says 172.21.15.175:39556) 2023-10-10T19:03:29.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f6146700 1 -- 172.21.15.175:0/616803387 learned_addr learned my addr 172.21.15.175:0/616803387 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:29.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419684371 0 0) 0x7f10f0156800 con 0x7f10f014fbf0 2023-10-10T19:03:29.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f6947700 1 --2- 172.21.15.175:0/616803387 >> v2:172.21.15.191:3300/0 conn(0x7f10f0159d70 0x7f10f00c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:29.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d8003570 con 0x7f10f014fbf0 2023-10-10T19:03:29.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 580497288 0 0) 0x7f10d8003570 con 0x7f10f014fbf0 2023-10-10T19:03:29.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 >> v2:172.21.15.191:3300/0 conn(0x7f10f0159d70 msgr2=0x7f10f00c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:29.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 --2- 172.21.15.175:0/616803387 >> v2:172.21.15.191:3300/0 conn(0x7f10f0159d70 0x7f10f00c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:29.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 >> v1:172.21.15.191:6789/0 conn(0x7f10f014ffc0 legacy=0x7f10f00681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:29.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f0156c90 con 0x7f10f014fbf0 2023-10-10T19:03:29.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f10e8001970 con 0x7f10f014fbf0 2023-10-10T19:03:29.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 >> v1:172.21.15.175:6789/0 conn(0x7f10f014fbf0 legacy=0x7f10f0063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:29.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10d8003a60 0x7f10d8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:29.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10d8004810 0x7f10d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:29.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d80051c0 0x7f10d800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:29.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f00c59e0 con 0x7f10d8003a60 2023-10-10T19:03:29.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d800c0d0 con 0x7f10d80051c0 2023-10-10T19:03:29.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.923+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d800c290 con 0x7f10d8004810 2023-10-10T19:03:29.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6146700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10d8003a60 0x7f10d8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:29.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f5945700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10d8004810 0x7f10d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:29.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6947700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d80051c0 0x7f10d800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:29.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6146700 1 -- 172.21.15.175:0/616803387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10d8004810 msgr2=0x7f10d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:29.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6146700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10d8004810 0x7f10d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:29.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6146700 1 -- 172.21.15.175:0/616803387 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d80051c0 msgr2=0x7f10d800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:29.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6146700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d80051c0 0x7f10d800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:29.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6947700 1 -- 172.21.15.175:0/616803387 reap_dead start 2023-10-10T19:03:29.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6146700 1 -- 172.21.15.175:0/616803387 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d8003570 con 0x7f10d8003a60 2023-10-10T19:03:29.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f6146700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10d8003a60 0x7f10d8003e70 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f10e800c670 tx=0x7f10e8007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e70ed6256267ad server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:29.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e8005b70 con 0x7f10d8003a60 2023-10-10T19:03:29.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e8005cf0 con 0x7f10d8003a60 2023-10-10T19:03:29.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.924+0000 7f10f5144700 1 -- 172.21.15.175:0/616803387 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e8007360 con 0x7f10d8003a60 2023-10-10T19:03:29.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.925+0000 7f10f83aa700 1 -- 172.21.15.175:0/616803387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10d8003a60 msgr2=0x7f10d8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:29.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.925+0000 7f10f83aa700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10d8003a60 0x7f10d8003e70 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f10e800c670 tx=0x7f10e8007ca0 comp rx=0 tx=0).stop 2023-10-10T19:03:29.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.925+0000 7f10f83aa700 1 -- 172.21.15.175:0/616803387 shutdown_connections 2023-10-10T19:03:29.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.925+0000 7f10f83aa700 1 --2- 172.21.15.175:0/616803387 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10d8003a60 0x7f10d8003e70 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:29.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.925+0000 7f10f83aa700 1 -- 172.21.15.175:0/616803387 >> 172.21.15.175:0/616803387 conn(0x7f10f00cc760 msgr2=0x7f10f00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:29.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.925+0000 7f10f83aa700 1 -- 172.21.15.175:0/616803387 shutdown_connections 2023-10-10T19:03:29.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.925+0000 7f10f83aa700 1 -- 172.21.15.175:0/616803387 wait complete. 2023-10-10T19:03:29.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.926+0000 7f10f83aa700 1 Processor -- start 2023-10-10T19:03:29.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.926+0000 7f10f83aa700 1 -- start start 2023-10-10T19:03:29.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f83aa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d8003a60 0x7f10f0167290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:29.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f83aa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10f00cc010 0x7f10f01585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:29.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f83aa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10f014a9e0 0x7f10f014ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:29.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f83aa700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f014b390 con 0x7f10f00cc010 2023-10-10T19:03:29.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f83aa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0152020 con 0x7f10d8003a60 2023-10-10T19:03:29.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f83aa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0166990 con 0x7f10f014a9e0 2023-10-10T19:03:29.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10f00cc010 0x7f10f01585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:29.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10f00cc010 0x7f10f01585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42200/0 (socket says 172.21.15.175:42200) 2023-10-10T19:03:29.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 -- 172.21.15.175:0/2977804467 learned_addr learned my addr 172.21.15.175:0/2977804467 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:29.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f6947700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10f014a9e0 0x7f10f014ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:29.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f6146700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d8003a60 0x7f10f0167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:29.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 -- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10f014a9e0 msgr2=0x7f10f014ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:29.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10f014a9e0 0x7f10f014ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:29.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 -- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d8003a60 msgr2=0x7f10f0167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d8003a60 0x7f10f0167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 -- 172.21.15.175:0/2977804467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f0166c50 con 0x7f10f00cc010 2023-10-10T19:03:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f5945700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10f00cc010 0x7f10f01585b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f10e4002950 tx=0x7f10e4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10ee7fc700 1 -- 172.21.15.175:0/2977804467 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e400f670 con 0x7f10f00cc010 2023-10-10T19:03:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10ee7fc700 1 -- 172.21.15.175:0/2977804467 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e400f7d0 con 0x7f10f00cc010 2023-10-10T19:03:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10ee7fc700 1 -- 172.21.15.175:0/2977804467 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e400fb70 con 0x7f10f00cc010 2023-10-10T19:03:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f01516f0 con 0x7f10f00cc010 2023-10-10T19:03:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.927+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f0151cb0 con 0x7f10f00cc010 2023-10-10T19:03:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.928+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f01521d0 con 0x7f10f00cc010 2023-10-10T19:03:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.929+0000 7f10ee7fc700 1 -- 172.21.15.175:0/2977804467 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f10e401c400 con 0x7f10f00cc010 2023-10-10T19:03:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.929+0000 7f10ee7fc700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f10dc066f30 0x7f10dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.929+0000 7f10ee7fc700 1 -- 172.21.15.175:0/2977804467 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f10e4084d70 con 0x7f10f00cc010 2023-10-10T19:03:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.929+0000 7f10f6146700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f10dc066f30 0x7f10dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.930+0000 7f10f6146700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f10dc066f30 0x7f10dc0693e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f10e800c670 tx=0x7f10e8015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:29.933+0000 7f10ee7fc700 1 -- 172.21.15.175:0/2977804467 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10f01521d0 con 0x7f10f00cc010 2023-10-10T19:03:30.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.160+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10f00c48b0 con 0x7f10f00cc010 2023-10-10T19:03:30.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.160+0000 7f10ee7fc700 1 -- 172.21.15.175:0/2977804467 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5456 (secure 0 0 0) 0x7f10e4051d10 con 0x7f10f00cc010 2023-10-10T19:03:30.161 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:30.161 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:27.660970+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:30.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f10dc066f30 msgr2=0x7f10dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:30.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f10dc066f30 0x7f10dc0693e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f10e800c670 tx=0x7f10e8015000 comp rx=0 tx=0).stop 2023-10-10T19:03:30.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10f00cc010 msgr2=0x7f10f01585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:30.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10f00cc010 0x7f10f01585b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f10e4002950 tx=0x7f10e4007d30 comp rx=0 tx=0).stop 2023-10-10T19:03:30.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 shutdown_connections 2023-10-10T19:03:30.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f10dc066f30 0x7f10dc0693e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:30.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10f014a9e0 0x7f10f014ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:30.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10d8003a60 0x7f10f0167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:30.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 --2- 172.21.15.175:0/2977804467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10f00cc010 0x7f10f01585b0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:30.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 >> 172.21.15.175:0/2977804467 conn(0x7f10f00cc760 msgr2=0x7f10f0154b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:30.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 shutdown_connections 2023-10-10T19:03:30.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:30.162+0000 7f10f83aa700 1 -- 172.21.15.175:0/2977804467 wait complete. 2023-10-10T19:03:30.165 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 137 2023-10-10T19:03:30.174 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:03:32.175 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:32.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.303+0000 7f450d768700 1 Processor -- start 2023-10-10T19:03:32.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.303+0000 7f450d768700 1 -- start start 2023-10-10T19:03:32.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f450d768700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f450808a100 0x7f4508159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:32.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f450d768700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45080cd040 0x7f45080684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:32.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f450d768700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4508067fa0 con 0x7f450814fd90 2023-10-10T19:03:32.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f450d768700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45080c5600 con 0x7f450808a100 2023-10-10T19:03:32.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f450d768700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4508156b70 con 0x7f45080cd040 2023-10-10T19:03:32.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f45077fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f450808a100 0x7f4508159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:32.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f4507fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f450814fd90 0x7f45080c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39566/0 (socket says 172.21.15.175:39566) 2023-10-10T19:03:32.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f4507fff700 1 -- 172.21.15.175:0/1763179228 learned_addr learned my addr 172.21.15.175:0/1763179228 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:32.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f450cf67700 1 --2- 172.21.15.175:0/1763179228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45080cd040 0x7f45080684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:32.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f45077fe700 1 -- 172.21.15.175:0/1763179228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45080cd040 msgr2=0x7f45080684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:32.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f45077fe700 1 --2- 172.21.15.175:0/1763179228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45080cd040 0x7f45080684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f45077fe700 1 -- 172.21.15.175:0/1763179228 >> v1:172.21.15.175:6789/0 conn(0x7f450814fd90 legacy=0x7f45080c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:32.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f45077fe700 1 -- 172.21.15.175:0/1763179228 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508150160 con 0x7f450808a100 2023-10-10T19:03:32.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.304+0000 7f4506ffd700 1 -- 172.21.15.175:0/1763179228 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293781489 0 0) 0x7f4508067fa0 con 0x7f450814fd90 2023-10-10T19:03:32.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f45077fe700 1 --2- 172.21.15.175:0/1763179228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450808a100 0x7f4508159c30 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0098e0 tx=0x7f44fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d06a4f5ea6d98a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f4506ffd700 1 -- 172.21.15.175:0/1763179228 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc009e80 con 0x7f450808a100 2023-10-10T19:03:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f4506ffd700 1 -- 172.21.15.175:0/1763179228 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc00b860 con 0x7f450808a100 2023-10-10T19:03:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f4506ffd700 1 -- 172.21.15.175:0/1763179228 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc00bbe0 con 0x7f450808a100 2023-10-10T19:03:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f450d768700 1 -- 172.21.15.175:0/1763179228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450808a100 msgr2=0x7f4508159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f450d768700 1 --2- 172.21.15.175:0/1763179228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450808a100 0x7f4508159c30 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0098e0 tx=0x7f44fc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f450d768700 1 -- 172.21.15.175:0/1763179228 shutdown_connections 2023-10-10T19:03:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f450d768700 1 --2- 172.21.15.175:0/1763179228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45080cd040 0x7f45080684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f450d768700 1 --2- 172.21.15.175:0/1763179228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450808a100 0x7f4508159c30 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f450d768700 1 -- 172.21.15.175:0/1763179228 >> 172.21.15.175:0/1763179228 conn(0x7f4508063d00 msgr2=0x7f45080cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f450d768700 1 -- 172.21.15.175:0/1763179228 shutdown_connections 2023-10-10T19:03:32.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.305+0000 7f450d768700 1 -- 172.21.15.175:0/1763179228 wait complete. 2023-10-10T19:03:32.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.306+0000 7f450d768700 1 Processor -- start 2023-10-10T19:03:32.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.307+0000 7f450d768700 1 -- start start 2023-10-10T19:03:32.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.307+0000 7f450d768700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450808a100 0x7f45081505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:32.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.307+0000 7f450d768700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45080cd040 0x7f4508166610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:32.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.307+0000 7f450d768700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450814fd90 0x7f45081523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:32.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.307+0000 7f450d768700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4508166180 con 0x7f450814fd90 2023-10-10T19:03:32.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.307+0000 7f450d768700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45080a81c0 con 0x7f45080cd040 2023-10-10T19:03:32.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.307+0000 7f450d768700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4508150ea0 con 0x7f450808a100 2023-10-10T19:03:32.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450814fd90 0x7f45081523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:32.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450814fd90 0x7f45081523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42216/0 (socket says 172.21.15.175:42216) 2023-10-10T19:03:32.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 -- 172.21.15.175:0/2848529951 learned_addr learned my addr 172.21.15.175:0/2848529951 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:32.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f4507fff700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450808a100 0x7f45081505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:32.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f45077fe700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45080cd040 0x7f4508166610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:32.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 -- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450808a100 msgr2=0x7f45081505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:32.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450808a100 0x7f45081505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 -- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45080cd040 msgr2=0x7f4508166610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:32.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45080cd040 0x7f4508166610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 -- 172.21.15.175:0/2848529951 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc009420 con 0x7f450814fd90 2023-10-10T19:03:32.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450cf67700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450814fd90 0x7f45081523e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4500009fa0 tx=0x7f4500009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:32.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f44effff700 1 -- 172.21.15.175:0/2848529951 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450000ebc0 con 0x7f450814fd90 2023-10-10T19:03:32.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f44effff700 1 -- 172.21.15.175:0/2848529951 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450000ed20 con 0x7f450814fd90 2023-10-10T19:03:32.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f44effff700 1 -- 172.21.15.175:0/2848529951 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500014670 con 0x7f450814fd90 2023-10-10T19:03:32.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4508151180 con 0x7f450814fd90 2023-10-10T19:03:32.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.308+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4508150b10 con 0x7f450814fd90 2023-10-10T19:03:32.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.309+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45081662f0 con 0x7f450814fd90 2023-10-10T19:03:32.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.310+0000 7f44effff700 1 -- 172.21.15.175:0/2848529951 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f45000149e0 con 0x7f450814fd90 2023-10-10T19:03:32.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.310+0000 7f44effff700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f44f4066f60 0x7f44f4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:32.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.310+0000 7f44effff700 1 -- 172.21.15.175:0/2848529951 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f4500084f00 con 0x7f450814fd90 2023-10-10T19:03:32.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.310+0000 7f4507fff700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f44f4066f60 0x7f44f4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:32.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.311+0000 7f4507fff700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f44f4066f60 0x7f44f4069410 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f44f8007160 tx=0x7f44f8007f50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:32.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.314+0000 7f44effff700 1 -- 172.21.15.175:0/2848529951 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4500051df0 con 0x7f450814fd90 2023-10-10T19:03:32.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.537+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4508067570 con 0x7f450814fd90 2023-10-10T19:03:32.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.538+0000 7f44effff700 1 -- 172.21.15.175:0/2848529951 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5453 (secure 0 0 0) 0x7f4500055410 con 0x7f450814fd90 2023-10-10T19:03:32.538 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:32.538 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:31.670239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:32.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f44f4066f60 msgr2=0x7f44f4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:32.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f44f4066f60 0x7f44f4069410 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f44f8007160 tx=0x7f44f8007f50 comp rx=0 tx=0).stop 2023-10-10T19:03:32.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450814fd90 msgr2=0x7f45081523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:32.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450814fd90 0x7f45081523e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4500009fa0 tx=0x7f4500009900 comp rx=0 tx=0).stop 2023-10-10T19:03:32.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 shutdown_connections 2023-10-10T19:03:32.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f44f4066f60 0x7f44f4069410 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f450814fd90 0x7f45081523e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f450808a100 0x7f45081505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 --2- 172.21.15.175:0/2848529951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45080cd040 0x7f4508166610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:32.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 >> 172.21.15.175:0/2848529951 conn(0x7f4508063d00 msgr2=0x7f45080c7140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:32.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 shutdown_connections 2023-10-10T19:03:32.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:32.540+0000 7f450d768700 1 -- 172.21.15.175:0/2848529951 wait complete. 2023-10-10T19:03:32.543 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 138 2023-10-10T19:03:32.552 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:03:34.553 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:34.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.681+0000 7f229057e700 1 Processor -- start 2023-10-10T19:03:34.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.681+0000 7f229057e700 1 -- start start 2023-10-10T19:03:34.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f229057e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f22880b1fc0 0x7f2288155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f229057e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b2390 0x7f2288068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f229057e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22881574b0 con 0x7f2288068bd0 2023-10-10T19:03:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f229057e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2288150770 con 0x7f22880b1fc0 2023-10-10T19:03:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f229057e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22880afad0 con 0x7f22880b2390 2023-10-10T19:03:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f228e31a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f22880b1fc0 0x7f2288155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f228e31a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f22880b1fc0 0x7f2288155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32796/0 (socket says 172.21.15.175:32796) 2023-10-10T19:03:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f228e31a700 1 -- 172.21.15.175:0/2247826596 learned_addr learned my addr 172.21.15.175:0/2247826596 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f228db19700 1 --2- 172.21.15.175:0/2247826596 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b2390 0x7f2288068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f228e31a700 1 -- 172.21.15.175:0/2247826596 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b2390 msgr2=0x7f2288068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f228e31a700 1 --2- 172.21.15.175:0/2247826596 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b2390 0x7f2288068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f228e31a700 1 -- 172.21.15.175:0/2247826596 >> v1:172.21.15.191:6789/0 conn(0x7f2288068bd0 legacy=0x7f22880aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.682+0000 7f228e31a700 1 -- 172.21.15.175:0/2247826596 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880af4c0 con 0x7f22880b1fc0 2023-10-10T19:03:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f228e31a700 1 --2- 172.21.15.175:0/2247826596 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22880b1fc0 0x7f2288155fb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2284009f20 tx=0x7f2284009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6536d7bad2407b79 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f228d318700 1 -- 172.21.15.175:0/2247826596 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228400e070 con 0x7f22880b1fc0 2023-10-10T19:03:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f228d318700 1 -- 172.21.15.175:0/2247826596 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228400ba60 con 0x7f22880b1fc0 2023-10-10T19:03:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f228d318700 1 -- 172.21.15.175:0/2247826596 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22840092e0 con 0x7f22880b1fc0 2023-10-10T19:03:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f229057e700 1 -- 172.21.15.175:0/2247826596 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22880b1fc0 msgr2=0x7f2288155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f229057e700 1 --2- 172.21.15.175:0/2247826596 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22880b1fc0 0x7f2288155fb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2284009f20 tx=0x7f2284009840 comp rx=0 tx=0).stop 2023-10-10T19:03:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f229057e700 1 -- 172.21.15.175:0/2247826596 shutdown_connections 2023-10-10T19:03:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f229057e700 1 --2- 172.21.15.175:0/2247826596 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22880b1fc0 0x7f2288155fb0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f229057e700 1 --2- 172.21.15.175:0/2247826596 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b2390 0x7f2288068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f229057e700 1 -- 172.21.15.175:0/2247826596 >> 172.21.15.175:0/2247826596 conn(0x7f2288063230 msgr2=0x7f2288063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f229057e700 1 -- 172.21.15.175:0/2247826596 shutdown_connections 2023-10-10T19:03:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.683+0000 7f229057e700 1 -- 172.21.15.175:0/2247826596 wait complete. 2023-10-10T19:03:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.684+0000 7f229057e700 1 Processor -- start 2023-10-10T19:03:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.684+0000 7f229057e700 1 -- start start 2023-10-10T19:03:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f229057e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2288068bd0 0x7f2288169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f229057e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b1fc0 0x7f2288155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f229057e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22880b2390 0x7f228814ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f229057e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2288166540 con 0x7f2288068bd0 2023-10-10T19:03:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f229057e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2288169fd0 con 0x7f22880b2390 2023-10-10T19:03:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f229057e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2288150d20 con 0x7f22880b1fc0 2023-10-10T19:03:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f228e31a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2288068bd0 0x7f2288169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f228e31a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2288068bd0 0x7f2288169a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32802/0 (socket says 172.21.15.175:32802) 2023-10-10T19:03:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f228e31a700 1 -- 172.21.15.175:0/1617251835 learned_addr learned my addr 172.21.15.175:0/1617251835 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f228db19700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b1fc0 0x7f2288155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.685+0000 7f228eb1b700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22880b2390 0x7f228814ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f228e31a700 1 -- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b1fc0 msgr2=0x7f2288155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f228e31a700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b1fc0 0x7f2288155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f228e31a700 1 -- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22880b2390 msgr2=0x7f228814ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f228e31a700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22880b2390 0x7f228814ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f228e31a700 1 -- 172.21.15.175:0/1617251835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2284009ae0 con 0x7f2288068bd0 2023-10-10T19:03:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f228e31a700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2288068bd0 0x7f2288169a90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f228806fd90 tx=0x7f2284018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f227e7fc700 1 -- 172.21.15.175:0/1617251835 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284007170 con 0x7f2288068bd0 2023-10-10T19:03:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f227e7fc700 1 -- 172.21.15.175:0/1617251835 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2284008980 con 0x7f2288068bd0 2023-10-10T19:03:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22880cced0 con 0x7f2288068bd0 2023-10-10T19:03:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f227e7fc700 1 -- 172.21.15.175:0/1617251835 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284008d40 con 0x7f2288068bd0 2023-10-10T19:03:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.686+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22880cd170 con 0x7f2288068bd0 2023-10-10T19:03:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.687+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2288067570 con 0x7f2288068bd0 2023-10-10T19:03:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.687+0000 7f227e7fc700 1 -- 172.21.15.175:0/1617251835 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2284003040 con 0x7f2288068bd0 2023-10-10T19:03:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.687+0000 7f227e7fc700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f226c066f30 0x7f226c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.688+0000 7f227e7fc700 1 -- 172.21.15.175:0/1617251835 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f2284085fd0 con 0x7f2288068bd0 2023-10-10T19:03:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.688+0000 7f228db19700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f226c066f30 0x7f226c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.688+0000 7f228db19700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f226c066f30 0x7f226c0693e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f22880cce60 tx=0x7f22780091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.692+0000 7f227e7fc700 1 -- 172.21.15.175:0/1617251835 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2284052ec0 con 0x7f2288068bd0 2023-10-10T19:03:34.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.919+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2288069060 con 0x7f2288068bd0 2023-10-10T19:03:34.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.919+0000 7f227e7fc700 1 -- 172.21.15.175:0/1617251835 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5453 (secure 0 0 0) 0x7f22840564e0 con 0x7f2288068bd0 2023-10-10T19:03:34.920 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:34.920 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:31.670239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:34.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f226c066f30 msgr2=0x7f226c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:34.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f226c066f30 0x7f226c0693e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f22880cce60 tx=0x7f22780091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2288068bd0 msgr2=0x7f2288169a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2288068bd0 0x7f2288169a90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f228806fd90 tx=0x7f2284018000 comp rx=0 tx=0).stop 2023-10-10T19:03:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 shutdown_connections 2023-10-10T19:03:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f226c066f30 0x7f226c0693e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22880b2390 0x7f228814ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2288068bd0 0x7f2288169a90 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 --2- 172.21.15.175:0/1617251835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22880b1fc0 0x7f2288155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 >> 172.21.15.175:0/1617251835 conn(0x7f2288063230 msgr2=0x7f228814fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 shutdown_connections 2023-10-10T19:03:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:34.922+0000 7f229057e700 1 -- 172.21.15.175:0/1617251835 wait complete. 2023-10-10T19:03:34.925 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 138 2023-10-10T19:03:34.933 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:03:36.934 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:37.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.059+0000 7f3c30008700 1 Processor -- start 2023-10-10T19:03:37.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.060+0000 7f3c30008700 1 -- start start 2023-10-10T19:03:37.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.060+0000 7f3c30008700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3c280b1fc0 0x7f3c28155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:37.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.060+0000 7f3c30008700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3c280b2390 0x7f3c28068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:37.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.060+0000 7f3c30008700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c28068bd0 0x7f3c280aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:37.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.060+0000 7f3c30008700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c28150770 con 0x7f3c280b1fc0 2023-10-10T19:03:37.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.060+0000 7f3c30008700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c281574b0 con 0x7f3c280b2390 2023-10-10T19:03:37.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.060+0000 7f3c30008700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280afad0 con 0x7f3c28068bd0 2023-10-10T19:03:37.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.060+0000 7f3c2dda4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3c280b1fc0 0x7f3c28155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:37.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2dda4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3c280b1fc0 0x7f3c28155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32810/0 (socket says 172.21.15.175:32810) 2023-10-10T19:03:37.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2dda4700 1 -- 172.21.15.175:0/2095994450 learned_addr learned my addr 172.21.15.175:0/2095994450 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:37.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2e5a5700 1 --2- 172.21.15.175:0/2095994450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c28068bd0 0x7f3c280aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:37.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2d5a3700 1 --2- 172.21.15.175:0/2095994450 >> v2:172.21.15.191:3300/0 conn(0x7f3c280b2390 0x7f3c28068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:37.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2dda4700 1 -- 172.21.15.175:0/2095994450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c28068bd0 msgr2=0x7f3c280aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:37.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2dda4700 1 --2- 172.21.15.175:0/2095994450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c28068bd0 0x7f3c280aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2dda4700 1 -- 172.21.15.175:0/2095994450 >> v2:172.21.15.191:3300/0 conn(0x7f3c280b2390 msgr2=0x7f3c28068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:37.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2dda4700 1 --2- 172.21.15.175:0/2095994450 >> v2:172.21.15.191:3300/0 conn(0x7f3c280b2390 0x7f3c28068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2dda4700 1 -- 172.21.15.175:0/2095994450 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c280af4c0 con 0x7f3c280b1fc0 2023-10-10T19:03:37.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2e5a5700 1 --2- 172.21.15.175:0/2095994450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c28068bd0 0x7f3c280aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:03:37.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2d5a3700 1 --2- 172.21.15.175:0/2095994450 >> v2:172.21.15.191:3300/0 conn(0x7f3c280b2390 0x7f3c28068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:03:37.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2dda4700 1 --2- 172.21.15.175:0/2095994450 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b1fc0 0x7f3c28155fb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3c280c0290 tx=0x7f3c24009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5246fdca1cde20 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:37.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2cda2700 1 -- 172.21.15.175:0/2095994450 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400e070 con 0x7f3c280b1fc0 2023-10-10T19:03:37.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2cda2700 1 -- 172.21.15.175:0/2095994450 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2400bae0 con 0x7f3c280b1fc0 2023-10-10T19:03:37.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.061+0000 7f3c2cda2700 1 -- 172.21.15.175:0/2095994450 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240092e0 con 0x7f3c280b1fc0 2023-10-10T19:03:37.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 -- 172.21.15.175:0/2095994450 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b1fc0 msgr2=0x7f3c28155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:37.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 --2- 172.21.15.175:0/2095994450 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b1fc0 0x7f3c28155fb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3c280c0290 tx=0x7f3c24009840 comp rx=0 tx=0).stop 2023-10-10T19:03:37.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 -- 172.21.15.175:0/2095994450 shutdown_connections 2023-10-10T19:03:37.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 --2- 172.21.15.175:0/2095994450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c28068bd0 0x7f3c280aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 --2- 172.21.15.175:0/2095994450 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b1fc0 0x7f3c28155fb0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 --2- 172.21.15.175:0/2095994450 >> v2:172.21.15.191:3300/0 conn(0x7f3c280b2390 0x7f3c28068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 -- 172.21.15.175:0/2095994450 >> 172.21.15.175:0/2095994450 conn(0x7f3c28063230 msgr2=0x7f3c28063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:37.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 -- 172.21.15.175:0/2095994450 shutdown_connections 2023-10-10T19:03:37.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.062+0000 7f3c30008700 1 -- 172.21.15.175:0/2095994450 wait complete. 2023-10-10T19:03:37.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.063+0000 7f3c30008700 1 Processor -- start 2023-10-10T19:03:37.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.063+0000 7f3c30008700 1 -- start start 2023-10-10T19:03:37.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.063+0000 7f3c30008700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c28068bd0 0x7f3c280c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:37.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.063+0000 7f3c30008700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c280b1fc0 0x7f3c28159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:37.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.063+0000 7f3c30008700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b2390 0x7f3c280c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:37.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.063+0000 7f3c30008700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280c53d0 con 0x7f3c280b2390 2023-10-10T19:03:37.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.063+0000 7f3c30008700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280c3db0 con 0x7f3c28068bd0 2023-10-10T19:03:37.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.063+0000 7f3c30008700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28169110 con 0x7f3c280b1fc0 2023-10-10T19:03:37.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b2390 0x7f3c280c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:37.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b2390 0x7f3c280c5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32824/0 (socket says 172.21.15.175:32824) 2023-10-10T19:03:37.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 -- 172.21.15.175:0/1645467444 learned_addr learned my addr 172.21.15.175:0/1645467444 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:37.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2d5a3700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c280b1fc0 0x7f3c28159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:37.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2dda4700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c28068bd0 0x7f3c280c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:37.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 -- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c280b1fc0 msgr2=0x7f3c28159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:37.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c280b1fc0 0x7f3c28159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 -- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c28068bd0 msgr2=0x7f3c280c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:37.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c28068bd0 0x7f3c280c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 -- 172.21.15.175:0/1645467444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c24009ae0 con 0x7f3c280b2390 2023-10-10T19:03:37.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2dda4700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c28068bd0 0x7f3c280c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:37.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2d5a3700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c280b1fc0 0x7f3c28159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:03:37.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c2e5a5700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b2390 0x7f3c280c5880 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000c8b0 tx=0x7f3c20008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:37.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c1dffb700 1 -- 172.21.15.175:0/1645467444 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20011600 con 0x7f3c280b2390 2023-10-10T19:03:37.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c1dffb700 1 -- 172.21.15.175:0/1645467444 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c20011780 con 0x7f3c280b2390 2023-10-10T19:03:37.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c1dffb700 1 -- 172.21.15.175:0/1645467444 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20011b40 con 0x7f3c280b2390 2023-10-10T19:03:37.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c281693f0 con 0x7f3c280b2390 2023-10-10T19:03:37.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.064+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c281699b0 con 0x7f3c280b2390 2023-10-10T19:03:37.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.065+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c28071350 con 0x7f3c280b2390 2023-10-10T19:03:37.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.066+0000 7f3c1dffb700 1 -- 172.21.15.175:0/1645467444 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3c20018360 con 0x7f3c280b2390 2023-10-10T19:03:37.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.066+0000 7f3c1dffb700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c0c066f30 0x7f3c0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:37.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.066+0000 7f3c1dffb700 1 -- 172.21.15.175:0/1645467444 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f3c200845d0 con 0x7f3c280b2390 2023-10-10T19:03:37.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.066+0000 7f3c2dda4700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c0c066f30 0x7f3c0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:37.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.067+0000 7f3c2dda4700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c0c066f30 0x7f3c0c0693e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009880 tx=0x7f3c24018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:37.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.070+0000 7f3c1dffb700 1 -- 172.21.15.175:0/1645467444 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c28071350 con 0x7f3c280b2390 2023-10-10T19:03:37.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.296+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c28071350 con 0x7f3c280b2390 2023-10-10T19:03:37.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.296+0000 7f3c1dffb700 1 -- 172.21.15.175:0/1645467444 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5453 (secure 0 0 0) 0x7f3c28071350 con 0x7f3c280b2390 2023-10-10T19:03:37.297 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:37.297 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:31.670239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:37.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c0c066f30 msgr2=0x7f3c0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:37.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c0c066f30 0x7f3c0c0693e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009880 tx=0x7f3c24018000 comp rx=0 tx=0).stop 2023-10-10T19:03:37.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b2390 msgr2=0x7f3c280c5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:37.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b2390 0x7f3c280c5880 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000c8b0 tx=0x7f3c20008040 comp rx=0 tx=0).stop 2023-10-10T19:03:37.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 shutdown_connections 2023-10-10T19:03:37.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c0c066f30 0x7f3c0c0693e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c280b2390 0x7f3c280c5880 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c28068bd0 0x7f3c280c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 --2- 172.21.15.175:0/1645467444 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c280b1fc0 0x7f3c28159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:37.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 >> 172.21.15.175:0/1645467444 conn(0x7f3c28063230 msgr2=0x7f3c28150a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:37.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 shutdown_connections 2023-10-10T19:03:37.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:37.298+0000 7f3c30008700 1 -- 172.21.15.175:0/1645467444 wait complete. 2023-10-10T19:03:37.301 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 138 2023-10-10T19:03:37.310 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:03:39.312 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:39.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.435+0000 7f5a43bf1700 1 Processor -- start 2023-10-10T19:03:39.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.435+0000 7f5a43bf1700 1 -- start start 2023-10-10T19:03:39.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a43bf1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a3c1573d0 0x7f5a3c0c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:39.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a43bf1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c154aa0 con 0x7f5a3c0c4f20 2023-10-10T19:03:39.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a43bf1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c157c20 con 0x7f5a3c1577a0 2023-10-10T19:03:39.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a43bf1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3c0bfdf0 con 0x7f5a3c1573d0 2023-10-10T19:03:39.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4198d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a3c1573d0 0x7f5a3c0c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:39.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4218e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5a3c0c4f20 0x7f5a3c14d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60708/0 (socket says 172.21.15.175:60708) 2023-10-10T19:03:39.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4218e700 1 -- 172.21.15.175:0/3820982577 learned_addr learned my addr 172.21.15.175:0/3820982577 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:39.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4198d700 1 -- 172.21.15.175:0/3820982577 >> v1:172.21.15.191:6789/0 conn(0x7f5a3c1577a0 legacy=0x7f5a3c0bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:39.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4198d700 1 -- 172.21.15.175:0/3820982577 >> v1:172.21.15.175:6789/0 conn(0x7f5a3c0c4f20 legacy=0x7f5a3c14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:39.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4198d700 1 -- 172.21.15.175:0/3820982577 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c063dc0 con 0x7f5a3c1573d0 2023-10-10T19:03:39.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4098b700 1 -- 172.21.15.175:0/3820982577 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403914756 0 0) 0x7f5a3c154aa0 con 0x7f5a3c0c4f20 2023-10-10T19:03:39.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4198d700 1 --2- 172.21.15.175:0/3820982577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1573d0 0x7f5a3c0c12a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5a340098e0 tx=0x7f5a340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a4da983186a745 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:39.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4098b700 1 -- 172.21.15.175:0/3820982577 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a34009e80 con 0x7f5a3c1573d0 2023-10-10T19:03:39.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4098b700 1 -- 172.21.15.175:0/3820982577 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3400b820 con 0x7f5a3c1573d0 2023-10-10T19:03:39.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.436+0000 7f5a4098b700 1 -- 172.21.15.175:0/3820982577 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400bba0 con 0x7f5a3c1573d0 2023-10-10T19:03:39.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.437+0000 7f5a43bf1700 1 -- 172.21.15.175:0/3820982577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1573d0 msgr2=0x7f5a3c0c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:39.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.437+0000 7f5a43bf1700 1 --2- 172.21.15.175:0/3820982577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1573d0 0x7f5a3c0c12a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5a340098e0 tx=0x7f5a340091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:39.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.437+0000 7f5a43bf1700 1 -- 172.21.15.175:0/3820982577 shutdown_connections 2023-10-10T19:03:39.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.437+0000 7f5a43bf1700 1 --2- 172.21.15.175:0/3820982577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1573d0 0x7f5a3c0c12a0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:39.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.437+0000 7f5a43bf1700 1 -- 172.21.15.175:0/3820982577 >> 172.21.15.175:0/3820982577 conn(0x7f5a3c06dc70 msgr2=0x7f5a3c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:39.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.437+0000 7f5a43bf1700 1 -- 172.21.15.175:0/3820982577 shutdown_connections 2023-10-10T19:03:39.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.437+0000 7f5a43bf1700 1 -- 172.21.15.175:0/3820982577 wait complete. 2023-10-10T19:03:39.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.438+0000 7f5a43bf1700 1 Processor -- start 2023-10-10T19:03:39.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.438+0000 7f5a43bf1700 1 -- start start 2023-10-10T19:03:39.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a43bf1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a3c0c4f20 0x7f5a3c164340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:39.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a43bf1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a3c1573d0 0x7f5a3c148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:39.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a43bf1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1577a0 0x7f5a3c152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:39.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a43bf1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c156f40 con 0x7f5a3c1577a0 2023-10-10T19:03:39.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a43bf1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c151a90 con 0x7f5a3c0c4f20 2023-10-10T19:03:39.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a43bf1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c152010 con 0x7f5a3c1573d0 2023-10-10T19:03:39.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1577a0 0x7f5a3c152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:39.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1577a0 0x7f5a3c152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32840/0 (socket says 172.21.15.175:32840) 2023-10-10T19:03:39.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 -- 172.21.15.175:0/1850300964 learned_addr learned my addr 172.21.15.175:0/1850300964 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:39.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4198d700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a3c0c4f20 0x7f5a3c164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:39.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4118c700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a3c1573d0 0x7f5a3c148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:39.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 -- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a3c1573d0 msgr2=0x7f5a3c148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:39.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a3c1573d0 0x7f5a3c148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:39.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 -- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a3c0c4f20 msgr2=0x7f5a3c164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:39.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a3c0c4f20 0x7f5a3c164340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:39.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 -- 172.21.15.175:0/1850300964 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a34009420 con 0x7f5a3c1577a0 2023-10-10T19:03:39.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4118c700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a3c1573d0 0x7f5a3c148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:03:39.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4198d700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a3c0c4f20 0x7f5a3c164340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:03:39.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a4218e700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1577a0 0x7f5a3c152ac0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00cc10 tx=0x7f5a2c001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:39.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a39ffb700 1 -- 172.21.15.175:0/1850300964 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00a9c0 con 0x7f5a3c1577a0 2023-10-10T19:03:39.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a39ffb700 1 -- 172.21.15.175:0/1850300964 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c00ab40 con 0x7f5a3c1577a0 2023-10-10T19:03:39.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.439+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3c151d30 con 0x7f5a3c1577a0 2023-10-10T19:03:39.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.440+0000 7f5a39ffb700 1 -- 172.21.15.175:0/1850300964 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0070f0 con 0x7f5a3c1577a0 2023-10-10T19:03:39.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.440+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3c156bd0 con 0x7f5a3c1577a0 2023-10-10T19:03:39.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.441+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3c1527e0 con 0x7f5a3c1577a0 2023-10-10T19:03:39.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.441+0000 7f5a39ffb700 1 -- 172.21.15.175:0/1850300964 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5a2c0074a0 con 0x7f5a3c1577a0 2023-10-10T19:03:39.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.441+0000 7f5a39ffb700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a28066f30 0x7f5a280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:39.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.442+0000 7f5a39ffb700 1 -- 172.21.15.175:0/1850300964 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f5a2c084d70 con 0x7f5a3c1577a0 2023-10-10T19:03:39.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.442+0000 7f5a4198d700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a28066f30 0x7f5a280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:39.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.442+0000 7f5a4198d700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a28066f30 0x7f5a280693e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5a340093a0 tx=0x7f5a34016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:39.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.446+0000 7f5a39ffb700 1 -- 172.21.15.175:0/1850300964 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a3c1527e0 con 0x7f5a3c1577a0 2023-10-10T19:03:39.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.668+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a3c1527e0 con 0x7f5a3c1577a0 2023-10-10T19:03:39.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.668+0000 7f5a39ffb700 1 -- 172.21.15.175:0/1850300964 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5453 (secure 0 0 0) 0x7f5a3c1527e0 con 0x7f5a3c1577a0 2023-10-10T19:03:39.669 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:39.670 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:37.685102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:39.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.670+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a28066f30 msgr2=0x7f5a280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:39.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.670+0000 7f5a43bf1700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a28066f30 0x7f5a280693e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5a340093a0 tx=0x7f5a34016000 comp rx=0 tx=0).stop 2023-10-10T19:03:39.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1577a0 msgr2=0x7f5a3c152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:39.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1577a0 0x7f5a3c152ac0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00cc10 tx=0x7f5a2c001970 comp rx=0 tx=0).stop 2023-10-10T19:03:39.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 shutdown_connections 2023-10-10T19:03:39.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a28066f30 0x7f5a280693e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:39.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a3c1577a0 0x7f5a3c152ac0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:39.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a3c0c4f20 0x7f5a3c164340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:39.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 --2- 172.21.15.175:0/1850300964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a3c1573d0 0x7f5a3c148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:39.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 >> 172.21.15.175:0/1850300964 conn(0x7f5a3c06dc70 msgr2=0x7f5a3c145890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:39.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 shutdown_connections 2023-10-10T19:03:39.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:39.671+0000 7f5a43bf1700 1 -- 172.21.15.175:0/1850300964 wait complete. 2023-10-10T19:03:39.676 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 139 2023-10-10T19:03:39.683 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=5332 2023-10-10T19:03:39.683 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.f 2023-10-10T19:03:39.745 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:03:39.745+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:03:43.684 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-10T19:03:43.684 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-10T19:03:43.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-10T19:03:43.688 INFO:tasks.ceph.mds.f:Started 2023-10-10T19:03:43.688 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:43.771 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-10T19:03:43.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.808+0000 7fc42d2d4700 1 Processor -- start 2023-10-10T19:03:43.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.808+0000 7fc42d2d4700 1 -- start start 2023-10-10T19:03:43.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc42d2d4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b1fc0 0x7fc428155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:43.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc42d2d4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4281574b0 con 0x7fc4280aff60 2023-10-10T19:03:43.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc42d2d4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280afad0 con 0x7fc4280b2390 2023-10-10T19:03:43.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc42d2d4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc428150770 con 0x7fc4280b1fc0 2023-10-10T19:03:43.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc4277fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc4280aff60 0x7fc428159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60712/0 (socket says 172.21.15.175:60712) 2023-10-10T19:03:43.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc4277fe700 1 -- 172.21.15.175:0/1016074571 learned_addr learned my addr 172.21.15.175:0/1016074571 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:43.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc426ffd700 1 --2- 172.21.15.175:0/1016074571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b1fc0 0x7fc428155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:43.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc425ffb700 1 -- 172.21.15.175:0/1016074571 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918637120 0 0) 0x7fc4281574b0 con 0x7fc4280aff60 2023-10-10T19:03:43.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc425ffb700 1 -- 172.21.15.175:0/1016074571 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc410003570 con 0x7fc4280aff60 2023-10-10T19:03:43.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc425ffb700 1 -- 172.21.15.175:0/1016074571 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957649981 0 0) 0x7fc4280afad0 con 0x7fc4280b2390 2023-10-10T19:03:43.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc425ffb700 1 -- 172.21.15.175:0/1016074571 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4281574b0 con 0x7fc4280b2390 2023-10-10T19:03:43.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc426ffd700 1 -- 172.21.15.175:0/1016074571 >> v1:172.21.15.191:6789/0 conn(0x7fc4280b2390 legacy=0x7fc428068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:43.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc426ffd700 1 -- 172.21.15.175:0/1016074571 >> v1:172.21.15.175:6789/0 conn(0x7fc4280aff60 legacy=0x7fc428159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:43.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc426ffd700 1 -- 172.21.15.175:0/1016074571 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4280af4c0 con 0x7fc4280b1fc0 2023-10-10T19:03:43.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc426ffd700 1 --2- 172.21.15.175:0/1016074571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b1fc0 0x7fc428155fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc418009940 tx=0x7fc4180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82fc3ac4eefddc33 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:43.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.809+0000 7fc425ffb700 1 -- 172.21.15.175:0/1016074571 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418011630 con 0x7fc4280b1fc0 2023-10-10T19:03:43.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc425ffb700 1 -- 172.21.15.175:0/1016074571 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418011790 con 0x7fc4280b1fc0 2023-10-10T19:03:43.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc425ffb700 1 -- 172.21.15.175:0/1016074571 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418011b30 con 0x7fc4280b1fc0 2023-10-10T19:03:43.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc42d2d4700 1 -- 172.21.15.175:0/1016074571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b1fc0 msgr2=0x7fc428155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:43.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc42d2d4700 1 --2- 172.21.15.175:0/1016074571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b1fc0 0x7fc428155fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc418009940 tx=0x7fc4180091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:43.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc42d2d4700 1 -- 172.21.15.175:0/1016074571 shutdown_connections 2023-10-10T19:03:43.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc42d2d4700 1 --2- 172.21.15.175:0/1016074571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b1fc0 0x7fc428155fb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:43.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc42d2d4700 1 -- 172.21.15.175:0/1016074571 >> 172.21.15.175:0/1016074571 conn(0x7fc428063230 msgr2=0x7fc428063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:43.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc42d2d4700 1 -- 172.21.15.175:0/1016074571 shutdown_connections 2023-10-10T19:03:43.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.810+0000 7fc42d2d4700 1 -- 172.21.15.175:0/1016074571 wait complete. 2023-10-10T19:03:43.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.811+0000 7fc42d2d4700 1 Processor -- start 2023-10-10T19:03:43.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.811+0000 7fc42d2d4700 1 -- start start 2023-10-10T19:03:43.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc42d2d4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4280aff60 0x7fc4280c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:43.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc42d2d4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4280b1fc0 0x7fc4280c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:43.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc42d2d4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b2390 0x7fc428157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:43.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc42d2d4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280c66c0 con 0x7fc4280aff60 2023-10-10T19:03:43.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc42d2d4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280c5190 con 0x7fc4280b1fc0 2023-10-10T19:03:43.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc42d2d4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280c59f0 con 0x7fc4280b2390 2023-10-10T19:03:43.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc426ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4280aff60 0x7fc4280c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:43.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc426ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4280aff60 0x7fc4280c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32846/0 (socket says 172.21.15.175:32846) 2023-10-10T19:03:43.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc426ffd700 1 -- 172.21.15.175:0/2182188398 learned_addr learned my addr 172.21.15.175:0/2182188398 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:43.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc4277fe700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b2390 0x7fc428157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:43.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc4267fc700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4280b1fc0 0x7fc4280c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:43.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc426ffd700 1 -- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b2390 msgr2=0x7fc428157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:43.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc426ffd700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b2390 0x7fc428157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:43.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc426ffd700 1 -- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4280b1fc0 msgr2=0x7fc4280c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:43.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc426ffd700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4280b1fc0 0x7fc4280c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:43.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.812+0000 7fc426ffd700 1 -- 172.21.15.175:0/2182188398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc418009420 con 0x7fc4280aff60 2023-10-10T19:03:43.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.813+0000 7fc426ffd700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4280aff60 0x7fc4280c5e80 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc4180093f0 tx=0x7fc418017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:43.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.813+0000 7fc40effd700 1 -- 172.21.15.175:0/2182188398 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418011b10 con 0x7fc4280aff60 2023-10-10T19:03:43.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.813+0000 7fc40effd700 1 -- 172.21.15.175:0/2182188398 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418000c90 con 0x7fc4280aff60 2023-10-10T19:03:43.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.813+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc428166690 con 0x7fc4280aff60 2023-10-10T19:03:43.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.813+0000 7fc40effd700 1 -- 172.21.15.175:0/2182188398 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41801a660 con 0x7fc4280aff60 2023-10-10T19:03:43.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.813+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4280c5680 con 0x7fc4280aff60 2023-10-10T19:03:43.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.814+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc428067570 con 0x7fc4280aff60 2023-10-10T19:03:43.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.814+0000 7fc40effd700 1 -- 172.21.15.175:0/2182188398 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc41801a9d0 con 0x7fc4280aff60 2023-10-10T19:03:43.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.814+0000 7fc40effd700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc414066f30 0x7fc4140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:43.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.815+0000 7fc40effd700 1 -- 172.21.15.175:0/2182188398 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7fc418086530 con 0x7fc4280aff60 2023-10-10T19:03:43.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.815+0000 7fc4267fc700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc414066f30 0x7fc4140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:43.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.815+0000 7fc4267fc700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc414066f30 0x7fc4140693e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc4281512c0 tx=0x7fc41c007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:43.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:43.819+0000 7fc40effd700 1 -- 172.21.15.175:0/2182188398 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc418053420 con 0x7fc4280aff60 2023-10-10T19:03:44.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.045+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc428068da0 con 0x7fc4280aff60 2023-10-10T19:03:44.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.046+0000 7fc40effd700 1 -- 172.21.15.175:0/2182188398 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5453 (secure 0 0 0) 0x7fc418056a40 con 0x7fc4280aff60 2023-10-10T19:03:44.047 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:44.048 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:37.685102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:44.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc414066f30 msgr2=0x7fc4140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:44.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc414066f30 0x7fc4140693e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc4281512c0 tx=0x7fc41c007d30 comp rx=0 tx=0).stop 2023-10-10T19:03:44.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4280aff60 msgr2=0x7fc4280c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:44.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4280aff60 0x7fc4280c5e80 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc4180093f0 tx=0x7fc418017000 comp rx=0 tx=0).stop 2023-10-10T19:03:44.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 shutdown_connections 2023-10-10T19:03:44.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc414066f30 0x7fc4140693e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:44.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4280b2390 0x7fc428157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:44.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4280aff60 0x7fc4280c5e80 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:44.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 --2- 172.21.15.175:0/2182188398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4280b1fc0 0x7fc4280c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:44.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 >> 172.21.15.175:0/2182188398 conn(0x7fc428063230 msgr2=0x7fc42814fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:44.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 shutdown_connections 2023-10-10T19:03:44.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:44.048+0000 7fc42d2d4700 1 -- 172.21.15.175:0/2182188398 wait complete. 2023-10-10T19:03:44.052 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 139 2023-10-10T19:03:44.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-10T19:03:46.062 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:03:46.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.176+0000 7fca58427700 1 Processor -- start 2023-10-10T19:03:46.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.176+0000 7fca58427700 1 -- start start 2023-10-10T19:03:46.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.177+0000 7fca58427700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca5014e140 0x7fca50155270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:46.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.177+0000 7fca58427700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fca5006edb0 0x7fca5015e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:46.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.177+0000 7fca58427700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5006b740 con 0x7fca500c1210 2023-10-10T19:03:46.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.177+0000 7fca58427700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca500c0960 con 0x7fca5006edb0 2023-10-10T19:03:46.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.177+0000 7fca58427700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5006ca30 con 0x7fca5014e140 2023-10-10T19:03:46.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.177+0000 7fca559c2700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fca500c1210 0x7fca500bf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44422/0 (socket says 172.21.15.175:44422) 2023-10-10T19:03:46.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca559c2700 1 -- 172.21.15.175:0/4149892885 learned_addr learned my addr 172.21.15.175:0/4149892885 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:03:46.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca561c3700 1 --2- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca5014e140 0x7fca50155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:46.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca569c4700 1 --2- 172.21.15.175:0/4149892885 >> v2:172.21.15.191:3300/0 conn(0x7fca5006edb0 0x7fca5015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:46.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca551c1700 1 -- 172.21.15.175:0/4149892885 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550376881 0 0) 0x7fca5006b740 con 0x7fca500c1210 2023-10-10T19:03:46.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca551c1700 1 -- 172.21.15.175:0/4149892885 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca38003570 con 0x7fca500c1210 2023-10-10T19:03:46.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca551c1700 1 -- 172.21.15.175:0/4149892885 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2915446086 0 0) 0x7fca38003570 con 0x7fca500c1210 2023-10-10T19:03:46.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca569c4700 1 -- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca5014e140 msgr2=0x7fca50155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:03:46.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca569c4700 1 --2- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca5014e140 0x7fca50155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca569c4700 1 -- 172.21.15.175:0/4149892885 >> v1:172.21.15.175:6789/0 conn(0x7fca500c1210 legacy=0x7fca500bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:46.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca569c4700 1 -- 172.21.15.175:0/4149892885 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5006c5a0 con 0x7fca5006edb0 2023-10-10T19:03:46.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca561c3700 1 --2- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca5014e140 0x7fca50155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:03:46.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca569c4700 1 --2- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5006edb0 0x7fca5015e710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fca48009910 tx=0x7fca480091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32a28adb500b2d30 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:46.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca551c1700 1 -- 172.21.15.175:0/4149892885 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44001470 con 0x7fca5006edb0 2023-10-10T19:03:46.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca551c1700 1 -- 172.21.15.175:0/4149892885 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca48011420 con 0x7fca5006edb0 2023-10-10T19:03:46.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.178+0000 7fca551c1700 1 -- 172.21.15.175:0/4149892885 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca480117c0 con 0x7fca5006edb0 2023-10-10T19:03:46.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.179+0000 7fca58427700 1 -- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5006edb0 msgr2=0x7fca5015e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:46.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.179+0000 7fca58427700 1 --2- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5006edb0 0x7fca5015e710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fca48009910 tx=0x7fca480091a0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.179+0000 7fca58427700 1 -- 172.21.15.175:0/4149892885 shutdown_connections 2023-10-10T19:03:46.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.179+0000 7fca58427700 1 --2- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5006edb0 0x7fca5015e710 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.179+0000 7fca58427700 1 --2- 172.21.15.175:0/4149892885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca5014e140 0x7fca50155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.179+0000 7fca58427700 1 -- 172.21.15.175:0/4149892885 >> 172.21.15.175:0/4149892885 conn(0x7fca50063cb0 msgr2=0x7fca5006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:46.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.179+0000 7fca58427700 1 -- 172.21.15.175:0/4149892885 shutdown_connections 2023-10-10T19:03:46.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.179+0000 7fca58427700 1 -- 172.21.15.175:0/4149892885 wait complete. 2023-10-10T19:03:46.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.180+0000 7fca58427700 1 Processor -- start 2023-10-10T19:03:46.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.180+0000 7fca58427700 1 -- start start 2023-10-10T19:03:46.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca58427700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca5006edb0 0x7fca5006d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:46.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca58427700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5014e140 0x7fca501640f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:46.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca58427700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca501513a0 0x7fca50164630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:46.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca58427700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5014f360 con 0x7fca5006edb0 2023-10-10T19:03:46.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca58427700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca500c1870 con 0x7fca5014e140 2023-10-10T19:03:46.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca58427700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca500c1b10 con 0x7fca501513a0 2023-10-10T19:03:46.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca5006edb0 0x7fca5006d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:46.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca5006edb0 0x7fca5006d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54898/0 (socket says 172.21.15.175:54898) 2023-10-10T19:03:46.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 -- 172.21.15.175:0/286994934 learned_addr learned my addr 172.21.15.175:0/286994934 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:03:46.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca569c4700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca501513a0 0x7fca50164630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:46.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca559c2700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5014e140 0x7fca501640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:46.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 -- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca501513a0 msgr2=0x7fca50164630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:46.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca501513a0 0x7fca50164630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 -- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5014e140 msgr2=0x7fca501640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:46.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5014e140 0x7fca501640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 -- 172.21.15.175:0/286994934 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca40009a20 con 0x7fca5006edb0 2023-10-10T19:03:46.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca561c3700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca5006edb0 0x7fca5006d230 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c0c0 tx=0x7fca4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:46.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.181+0000 7fca4e7fc700 1 -- 172.21.15.175:0/286994934 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4000cc90 con 0x7fca5006edb0 2023-10-10T19:03:46.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.182+0000 7fca4e7fc700 1 -- 172.21.15.175:0/286994934 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4000f040 con 0x7fca5006edb0 2023-10-10T19:03:46.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.182+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca48009420 con 0x7fca5006edb0 2023-10-10T19:03:46.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.182+0000 7fca4e7fc700 1 -- 172.21.15.175:0/286994934 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40014480 con 0x7fca5006edb0 2023-10-10T19:03:46.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.182+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca500c1dd0 con 0x7fca5006edb0 2023-10-10T19:03:46.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.182+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca50067570 con 0x7fca5006edb0 2023-10-10T19:03:46.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.183+0000 7fca4e7fc700 1 -- 172.21.15.175:0/286994934 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fca400147f0 con 0x7fca5006edb0 2023-10-10T19:03:46.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.183+0000 7fca4e7fc700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca3c066f30 0x7fca3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:03:46.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.183+0000 7fca4e7fc700 1 -- 172.21.15.175:0/286994934 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7fca40085630 con 0x7fca5006edb0 2023-10-10T19:03:46.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.183+0000 7fca559c2700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca3c066f30 0x7fca3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:03:46.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.184+0000 7fca559c2700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca3c066f30 0x7fca3c0693e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fca500c6590 tx=0x7fca44000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:03:46.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.187+0000 7fca4e7fc700 1 -- 172.21.15.175:0/286994934 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca40052520 con 0x7fca5006edb0 2023-10-10T19:03:46.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.417+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca5014e510 con 0x7fca5006edb0 2023-10-10T19:03:46.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.418+0000 7fca4e7fc700 1 -- 172.21.15.175:0/286994934 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+6213 (secure 0 0 0) 0x7fca4005b030 con 0x7fca5006edb0 2023-10-10T19:03:46.419 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:03:46.419 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:03:37.685102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:03:46.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca3c066f30 msgr2=0x7fca3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:46.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca3c066f30 0x7fca3c0693e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fca500c6590 tx=0x7fca44000bc0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca5006edb0 msgr2=0x7fca5006d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:03:46.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca5006edb0 0x7fca5006d230 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c0c0 tx=0x7fca4000b040 comp rx=0 tx=0).stop 2023-10-10T19:03:46.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 shutdown_connections 2023-10-10T19:03:46.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca3c066f30 0x7fca3c0693e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca501513a0 0x7fca50164630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca5006edb0 0x7fca5006d230 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 --2- 172.21.15.175:0/286994934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca5014e140 0x7fca501640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:03:46.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 >> 172.21.15.175:0/286994934 conn(0x7fca50063cb0 msgr2=0x7fca50155170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:03:46.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 shutdown_connections 2023-10-10T19:03:46.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:03:46.420+0000 7fca58427700 1 -- 172.21.15.175:0/286994934 wait complete. 2023-10-10T19:03:46.424 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 140 2023-10-10T19:03:46.433 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-10T19:03:46.433 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-10-10T19:03:50.971 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:03:50.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:04:21.271 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:04:21.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:04:51.342 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:04:51.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:05:10.434 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:10.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.549+0000 7ff7da0b9700 1 Processor -- start 2023-10-10T19:05:10.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.549+0000 7ff7da0b9700 1 -- start start 2023-10-10T19:05:10.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.549+0000 7ff7da0b9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff7d414c840 0x7ff7d406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.549+0000 7ff7da0b9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff7d40c52a0 0x7ff7d4152970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.549+0000 7ff7da0b9700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d4155ca0 con 0x7ff7d414cc10 2023-10-10T19:05:10.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7da0b9700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d4156b00 con 0x7ff7d40c52a0 2023-10-10T19:05:10.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7da0b9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d406d700 con 0x7ff7d414c840 2023-10-10T19:05:10.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d3fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff7d40c52a0 0x7ff7d4152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d2ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff7d414cc10 0x7ff7d40c4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50332/0 (socket says 172.21.15.175:50332) 2023-10-10T19:05:10.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d2ffd700 1 -- 172.21.15.175:0/381605066 learned_addr learned my addr 172.21.15.175:0/381605066 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:10.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d37fe700 1 --2- 172.21.15.175:0/381605066 >> v2:172.21.15.191:3300/0 conn(0x7ff7d414c840 0x7ff7d406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d3fff700 1 -- 172.21.15.175:0/381605066 >> v2:172.21.15.191:3300/0 conn(0x7ff7d414c840 msgr2=0x7ff7d406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:10.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d3fff700 1 --2- 172.21.15.175:0/381605066 >> v2:172.21.15.191:3300/0 conn(0x7ff7d414c840 0x7ff7d406e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d3fff700 1 -- 172.21.15.175:0/381605066 >> v1:172.21.15.175:6789/0 conn(0x7ff7d414cc10 legacy=0x7ff7d40c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d27fc700 1 -- 172.21.15.175:0/381605066 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632521361 0 0) 0x7ff7d4155ca0 con 0x7ff7d414cc10 2023-10-10T19:05:10.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d3fff700 1 -- 172.21.15.175:0/381605066 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d4155ef0 con 0x7ff7d40c52a0 2023-10-10T19:05:10.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d3fff700 1 --2- 172.21.15.175:0/381605066 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d4152970 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff7d406cb20 tx=0x7ff7cc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3194c9857822473f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:10.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d27fc700 1 -- 172.21.15.175:0/381605066 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc00e070 con 0x7ff7d40c52a0 2023-10-10T19:05:10.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d27fc700 1 -- 172.21.15.175:0/381605066 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc00bbf0 con 0x7ff7d40c52a0 2023-10-10T19:05:10.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.550+0000 7ff7d27fc700 1 -- 172.21.15.175:0/381605066 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc012430 con 0x7ff7d40c52a0 2023-10-10T19:05:10.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.551+0000 7ff7da0b9700 1 -- 172.21.15.175:0/381605066 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 msgr2=0x7ff7d4152970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.551+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/381605066 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d4152970 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff7d406cb20 tx=0x7ff7cc009780 comp rx=0 tx=0).stop 2023-10-10T19:05:10.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.551+0000 7ff7da0b9700 1 -- 172.21.15.175:0/381605066 shutdown_connections 2023-10-10T19:05:10.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.551+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/381605066 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d4152970 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.551+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/381605066 >> v2:172.21.15.191:3300/0 conn(0x7ff7d414c840 0x7ff7d406e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.551+0000 7ff7da0b9700 1 -- 172.21.15.175:0/381605066 >> 172.21.15.175:0/381605066 conn(0x7ff7d4068280 msgr2=0x7ff7d406af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:10.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.551+0000 7ff7da0b9700 1 -- 172.21.15.175:0/381605066 shutdown_connections 2023-10-10T19:05:10.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.551+0000 7ff7da0b9700 1 -- 172.21.15.175:0/381605066 wait complete. 2023-10-10T19:05:10.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.552+0000 7ff7da0b9700 1 Processor -- start 2023-10-10T19:05:10.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.552+0000 7ff7da0b9700 1 -- start start 2023-10-10T19:05:10.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7da0b9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d40c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7da0b9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7d414c840 0x7ff7d4142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7da0b9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7d414cc10 0x7ff7d4142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7da0b9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40c6180 con 0x7ff7d40c52a0 2023-10-10T19:05:10.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7da0b9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40c69e0 con 0x7ff7d414c840 2023-10-10T19:05:10.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7da0b9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d4155860 con 0x7ff7d414cc10 2023-10-10T19:05:10.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d40c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d40c84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42910/0 (socket says 172.21.15.175:42910) 2023-10-10T19:05:10.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d37fe700 1 -- 172.21.15.175:0/671206736 learned_addr learned my addr 172.21.15.175:0/671206736 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:10.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d2ffd700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7d414c840 0x7ff7d4142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d3fff700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7d414cc10 0x7ff7d4142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d37fe700 1 -- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7d414cc10 msgr2=0x7ff7d4142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d37fe700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7d414cc10 0x7ff7d4142d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d37fe700 1 -- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7d414c840 msgr2=0x7ff7d4142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d37fe700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7d414c840 0x7ff7d4142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.553+0000 7ff7d37fe700 1 -- 172.21.15.175:0/671206736 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc009a20 con 0x7ff7d40c52a0 2023-10-10T19:05:10.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.554+0000 7ff7d37fe700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d40c84c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400ec80 tx=0x7ff7c4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:10.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.554+0000 7ff7bb7fe700 1 -- 172.21.15.175:0/671206736 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400c850 con 0x7ff7d40c52a0 2023-10-10T19:05:10.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.554+0000 7ff7bb7fe700 1 -- 172.21.15.175:0/671206736 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c400c9b0 con 0x7ff7d40c52a0 2023-10-10T19:05:10.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.554+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d4155b40 con 0x7ff7d40c52a0 2023-10-10T19:05:10.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.554+0000 7ff7bb7fe700 1 -- 172.21.15.175:0/671206736 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400cd50 con 0x7ff7d40c52a0 2023-10-10T19:05:10.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.554+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d40c6670 con 0x7ff7d40c52a0 2023-10-10T19:05:10.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.555+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d40c6cc0 con 0x7ff7d40c52a0 2023-10-10T19:05:10.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.555+0000 7ff7bb7fe700 1 -- 172.21.15.175:0/671206736 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff7c4013680 con 0x7ff7d40c52a0 2023-10-10T19:05:10.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.555+0000 7ff7bb7fe700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7c0066f30 0x7ff7c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.556+0000 7ff7bb7fe700 1 -- 172.21.15.175:0/671206736 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7ff7c4084e60 con 0x7ff7d40c52a0 2023-10-10T19:05:10.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.556+0000 7ff7d2ffd700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7c0066f30 0x7ff7c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.556+0000 7ff7d2ffd700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7c0066f30 0x7ff7c00693e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40c8de0 tx=0x7ff7c8007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:10.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.560+0000 7ff7bb7fe700 1 -- 172.21.15.175:0/671206736 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7c4051d50 con 0x7ff7d40c52a0 2023-10-10T19:05:10.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.796+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7d40c56e0 con 0x7ff7d40c52a0 2023-10-10T19:05:10.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.796+0000 7ff7bb7fe700 1 -- 172.21.15.175:0/671206736 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7ff7c4021090 con 0x7ff7d40c52a0 2023-10-10T19:05:10.798 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:10.798 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:10.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7c0066f30 msgr2=0x7ff7c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7c0066f30 0x7ff7c00693e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40c8de0 tx=0x7ff7c8007d30 comp rx=0 tx=0).stop 2023-10-10T19:05:10.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 msgr2=0x7ff7d40c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d40c84c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400ec80 tx=0x7ff7c4009120 comp rx=0 tx=0).stop 2023-10-10T19:05:10.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 shutdown_connections 2023-10-10T19:05:10.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7c0066f30 0x7ff7c00693e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7d414cc10 0x7ff7d4142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7d40c52a0 0x7ff7d40c84c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 --2- 172.21.15.175:0/671206736 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7d414c840 0x7ff7d4142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 >> 172.21.15.175:0/671206736 conn(0x7ff7d4068280 msgr2=0x7ff7d40c4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:10.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 shutdown_connections 2023-10-10T19:05:10.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.798+0000 7ff7da0b9700 1 -- 172.21.15.175:0/671206736 wait complete. 2023-10-10T19:05:10.802 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:10.811 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=3) 2023-10-10T19:05:10.811 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-10T19:05:10.812 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:05:10.821 INFO:tasks.ceph.mds.b:Stopped 2023-10-10T19:05:10.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:10.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.947+0000 7f373ab08700 1 Processor -- start 2023-10-10T19:05:10.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.947+0000 7f373ab08700 1 -- start start 2023-10-10T19:05:10.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.948+0000 7f373ab08700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37340c9d20 0x7f37341569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.948+0000 7f373ab08700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3734152b70 con 0x7f373406e8f0 2023-10-10T19:05:10.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.948+0000 7f373ab08700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3734152e50 con 0x7f373406e520 2023-10-10T19:05:10.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.948+0000 7f373ab08700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37340c33c0 con 0x7f37340c9d20 2023-10-10T19:05:10.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.948+0000 7f373a307700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37340c9d20 0x7f37341569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.948+0000 7f3739305700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f373406e8f0 0x7f373414c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50344/0 (socket says 172.21.15.175:50344) 2023-10-10T19:05:10.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.948+0000 7f3739305700 1 -- 172.21.15.175:0/3350103725 learned_addr learned my addr 172.21.15.175:0/3350103725 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:10.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373a307700 1 -- 172.21.15.175:0/3350103725 >> v1:172.21.15.191:6789/0 conn(0x7f373406e520 legacy=0x7f37340c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373a307700 1 -- 172.21.15.175:0/3350103725 >> v1:172.21.15.175:6789/0 conn(0x7f373406e8f0 legacy=0x7f373414c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f3738b04700 1 -- 172.21.15.175:0/3350103725 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572002837 0 0) 0x7f3734152e50 con 0x7f373406e520 2023-10-10T19:05:10.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373a307700 1 -- 172.21.15.175:0/3350103725 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3734153130 con 0x7f37340c9d20 2023-10-10T19:05:10.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373a307700 1 --2- 172.21.15.175:0/3350103725 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f37341569f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f372c0098e0 tx=0x7f372c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa24a1228d0f0e31 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:10.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f3738b04700 1 -- 172.21.15.175:0/3350103725 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c009e80 con 0x7f37340c9d20 2023-10-10T19:05:10.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f3738b04700 1 -- 172.21.15.175:0/3350103725 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372c00b760 con 0x7f37340c9d20 2023-10-10T19:05:10.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f3738b04700 1 -- 172.21.15.175:0/3350103725 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c00bae0 con 0x7f37340c9d20 2023-10-10T19:05:10.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373ab08700 1 -- 172.21.15.175:0/3350103725 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 msgr2=0x7f37341569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373ab08700 1 --2- 172.21.15.175:0/3350103725 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f37341569f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f372c0098e0 tx=0x7f372c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373ab08700 1 -- 172.21.15.175:0/3350103725 shutdown_connections 2023-10-10T19:05:10.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373ab08700 1 --2- 172.21.15.175:0/3350103725 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f37341569f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.949+0000 7f373ab08700 1 -- 172.21.15.175:0/3350103725 >> 172.21.15.175:0/3350103725 conn(0x7f3734063230 msgr2=0x7f373406f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:10.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.950+0000 7f373ab08700 1 -- 172.21.15.175:0/3350103725 shutdown_connections 2023-10-10T19:05:10.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.950+0000 7f373ab08700 1 -- 172.21.15.175:0/3350103725 wait complete. 2023-10-10T19:05:10.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373ab08700 1 Processor -- start 2023-10-10T19:05:10.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373ab08700 1 -- start start 2023-10-10T19:05:10.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373ab08700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f373406e520 0x7f3734166f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373ab08700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f373406e8f0 0x7f3734069060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373ab08700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f3734153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373ab08700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37341550b0 con 0x7f37340c9d20 2023-10-10T19:05:10.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373ab08700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3734167490 con 0x7f373406e520 2023-10-10T19:05:10.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373ab08700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3734163ef0 con 0x7f373406e8f0 2023-10-10T19:05:10.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.951+0000 7f373a307700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f3734153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f3739305700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f373406e8f0 0x7f3734069060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373a307700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f3734153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42918/0 (socket says 172.21.15.175:42918) 2023-10-10T19:05:10.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373a307700 1 -- 172.21.15.175:0/2926420521 learned_addr learned my addr 172.21.15.175:0/2926420521 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:10.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f3739b06700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f373406e520 0x7f3734166f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373a307700 1 -- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f373406e8f0 msgr2=0x7f3734069060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373a307700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f373406e8f0 0x7f3734069060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373a307700 1 -- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f373406e520 msgr2=0x7f3734166f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:10.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373a307700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f373406e520 0x7f3734166f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:10.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373a307700 1 -- 172.21.15.175:0/2926420521 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c009420 con 0x7f37340c9d20 2023-10-10T19:05:10.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373a307700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f3734153bd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f372c0093f0 tx=0x7f372c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:10.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f3729ffb700 1 -- 172.21.15.175:0/2926420521 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c00baa0 con 0x7f37340c9d20 2023-10-10T19:05:10.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f3729ffb700 1 -- 172.21.15.175:0/2926420521 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372c01c670 con 0x7f37340c9d20 2023-10-10T19:05:10.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f3729ffb700 1 -- 172.21.15.175:0/2926420521 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c01ca30 con 0x7f37340c9d20 2023-10-10T19:05:10.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3734157040 con 0x7f37340c9d20 2023-10-10T19:05:10.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.952+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3734157550 con 0x7f37340c9d20 2023-10-10T19:05:10.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.953+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3734067570 con 0x7f37340c9d20 2023-10-10T19:05:10.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.954+0000 7f3729ffb700 1 -- 172.21.15.175:0/2926420521 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f372c01a450 con 0x7f37340c9d20 2023-10-10T19:05:10.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.954+0000 7f3729ffb700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3718066f30 0x7f37180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:10.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.954+0000 7f3729ffb700 1 -- 172.21.15.175:0/2926420521 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f372c086920 con 0x7f37340c9d20 2023-10-10T19:05:10.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.954+0000 7f3739b06700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3718066f30 0x7f37180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:10.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.954+0000 7f3739b06700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3718066f30 0x7f37180693e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3730001560 tx=0x7f37300016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:10.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:10.958+0000 7f3729ffb700 1 -- 172.21.15.175:0/2926420521 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f372c0538d0 con 0x7f37340c9d20 2023-10-10T19:05:11.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.194+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f373414ca90 con 0x7f37340c9d20 2023-10-10T19:05:11.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.195+0000 7f3729ffb700 1 -- 172.21.15.175:0/2926420521 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7f372c056ef0 con 0x7f37340c9d20 2023-10-10T19:05:11.196 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:11.197 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:11.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3718066f30 msgr2=0x7f37180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:11.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3718066f30 0x7f37180693e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3730001560 tx=0x7f37300016f0 comp rx=0 tx=0).stop 2023-10-10T19:05:11.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 msgr2=0x7f3734153bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:11.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f3734153bd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f372c0093f0 tx=0x7f372c016000 comp rx=0 tx=0).stop 2023-10-10T19:05:11.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 shutdown_connections 2023-10-10T19:05:11.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3718066f30 0x7f37180693e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:11.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37340c9d20 0x7f3734153bd0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:11.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f373406e520 0x7f3734166f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:11.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 --2- 172.21.15.175:0/2926420521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f373406e8f0 0x7f3734069060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:11.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 >> 172.21.15.175:0/2926420521 conn(0x7f3734063230 msgr2=0x7f37340c5160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:11.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 shutdown_connections 2023-10-10T19:05:11.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:11.197+0000 7f373ab08700 1 -- 172.21.15.175:0/2926420521 wait complete. 2023-10-10T19:05:11.201 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:11.210 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:05:13.211 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:13.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.337+0000 7f23fffff700 1 Processor -- start 2023-10-10T19:05:13.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.337+0000 7f23fffff700 1 -- start start 2023-10-10T19:05:13.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23fffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f240014fd90 0x7f2400159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:13.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23fffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f240008a100 0x7f24000c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:13.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23fffff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2400067fa0 con 0x7f24000cd040 2023-10-10T19:05:13.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23fffff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24000c5600 con 0x7f240014fd90 2023-10-10T19:05:13.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23fffff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2400156b70 con 0x7f240008a100 2023-10-10T19:05:13.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23feffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f240014fd90 0x7f2400159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:13.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23ff7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f24000cd040 0x7f24000684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50348/0 (socket says 172.21.15.175:50348) 2023-10-10T19:05:13.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23ff7fe700 1 -- 172.21.15.175:0/3691309470 learned_addr learned my addr 172.21.15.175:0/3691309470 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:13.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23fe7fc700 1 --2- 172.21.15.175:0/3691309470 >> v2:172.21.15.191:3300/0 conn(0x7f240008a100 0x7f24000c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:13.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23feffd700 1 -- 172.21.15.175:0/3691309470 >> v2:172.21.15.191:3300/0 conn(0x7f240008a100 msgr2=0x7f24000c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:13.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23feffd700 1 --2- 172.21.15.175:0/3691309470 >> v2:172.21.15.191:3300/0 conn(0x7f240008a100 0x7f24000c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23feffd700 1 -- 172.21.15.175:0/3691309470 >> v1:172.21.15.175:6789/0 conn(0x7f24000cd040 legacy=0x7f24000684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:13.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23feffd700 1 -- 172.21.15.175:0/3691309470 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2400150160 con 0x7f240014fd90 2023-10-10T19:05:13.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.338+0000 7f23fdffb700 1 -- 172.21.15.175:0/3691309470 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935023256 0 0) 0x7f2400067fa0 con 0x7f24000cd040 2023-10-10T19:05:13.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23feffd700 1 --2- 172.21.15.175:0/3691309470 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400159c30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2400069990 tx=0x7f23f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13d2240e4359ab9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:13.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fdffb700 1 -- 172.21.15.175:0/3691309470 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f800e070 con 0x7f240014fd90 2023-10-10T19:05:13.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fdffb700 1 -- 172.21.15.175:0/3691309470 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f800bcc0 con 0x7f240014fd90 2023-10-10T19:05:13.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fdffb700 1 -- 172.21.15.175:0/3691309470 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8012680 con 0x7f240014fd90 2023-10-10T19:05:13.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fffff700 1 -- 172.21.15.175:0/3691309470 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 msgr2=0x7f2400159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:13.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fffff700 1 --2- 172.21.15.175:0/3691309470 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400159c30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2400069990 tx=0x7f23f8009780 comp rx=0 tx=0).stop 2023-10-10T19:05:13.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fffff700 1 -- 172.21.15.175:0/3691309470 shutdown_connections 2023-10-10T19:05:13.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fffff700 1 --2- 172.21.15.175:0/3691309470 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400159c30 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fffff700 1 --2- 172.21.15.175:0/3691309470 >> v2:172.21.15.191:3300/0 conn(0x7f240008a100 0x7f24000c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fffff700 1 -- 172.21.15.175:0/3691309470 >> 172.21.15.175:0/3691309470 conn(0x7f2400063d00 msgr2=0x7f24000cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:13.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fffff700 1 -- 172.21.15.175:0/3691309470 shutdown_connections 2023-10-10T19:05:13.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.339+0000 7f23fffff700 1 -- 172.21.15.175:0/3691309470 wait complete. 2023-10-10T19:05:13.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23fffff700 1 Processor -- start 2023-10-10T19:05:13.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23fffff700 1 -- start start 2023-10-10T19:05:13.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23fffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f240008a100 0x7f24001672b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:13.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23fffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24000cd040 0x7f2400169600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:13.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23fffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400169b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:13.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23fffff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000cb9b0 con 0x7f240014fd90 2023-10-10T19:05:13.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23fffff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000cadd0 con 0x7f24000cd040 2023-10-10T19:05:13.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23fffff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000cc9b0 con 0x7f240008a100 2023-10-10T19:05:13.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23ff7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400169b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:13.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23ff7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400169b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42948/0 (socket says 172.21.15.175:42948) 2023-10-10T19:05:13.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23ff7fe700 1 -- 172.21.15.175:0/77111588 learned_addr learned my addr 172.21.15.175:0/77111588 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:13.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.341+0000 7f23feffd700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f240008a100 0x7f24001672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:13.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23fe7fc700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24000cd040 0x7f2400169600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:13.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23ff7fe700 1 -- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f240008a100 msgr2=0x7f24001672b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:13.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23ff7fe700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f240008a100 0x7f24001672b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23ff7fe700 1 -- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24000cd040 msgr2=0x7f2400169600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:13.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23ff7fe700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24000cd040 0x7f2400169600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23ff7fe700 1 -- 172.21.15.175:0/77111588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8009a20 con 0x7f240014fd90 2023-10-10T19:05:13.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23ff7fe700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400169b40 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f23f000c7e0 tx=0x7f23f0007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:13.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23f6ffd700 1 -- 172.21.15.175:0/77111588 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0005c00 con 0x7f240014fd90 2023-10-10T19:05:13.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23f6ffd700 1 -- 172.21.15.175:0/77111588 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f0005d80 con 0x7f240014fd90 2023-10-10T19:05:13.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24000cb080 con 0x7f240014fd90 2023-10-10T19:05:13.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23f6ffd700 1 -- 172.21.15.175:0/77111588 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0007320 con 0x7f240014fd90 2023-10-10T19:05:13.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.342+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24000cb640 con 0x7f240014fd90 2023-10-10T19:05:13.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.343+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24000cbb60 con 0x7f240014fd90 2023-10-10T19:05:13.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.343+0000 7f23f6ffd700 1 -- 172.21.15.175:0/77111588 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f23f001b3b0 con 0x7f240014fd90 2023-10-10T19:05:13.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.344+0000 7f23f6ffd700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23e0066f30 0x7f23e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:13.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.344+0000 7f23f6ffd700 1 -- 172.21.15.175:0/77111588 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f23f0083cf0 con 0x7f240014fd90 2023-10-10T19:05:13.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.344+0000 7f23feffd700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23e0066f30 0x7f23e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:13.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.344+0000 7f23feffd700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23e0066f30 0x7f23e00693e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f23f80099f0 tx=0x7f23f8019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:13.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.349+0000 7f23f6ffd700 1 -- 172.21.15.175:0/77111588 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24000cbb60 con 0x7f240014fd90 2023-10-10T19:05:13.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.572+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24000cbb60 con 0x7f240014fd90 2023-10-10T19:05:13.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.572+0000 7f23f6ffd700 1 -- 172.21.15.175:0/77111588 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7f24000cbb60 con 0x7f240014fd90 2023-10-10T19:05:13.574 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:13.574 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:13.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23e0066f30 msgr2=0x7f23e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:13.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23e0066f30 0x7f23e00693e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f23f80099f0 tx=0x7f23f8019000 comp rx=0 tx=0).stop 2023-10-10T19:05:13.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 msgr2=0x7f2400169b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:13.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400169b40 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f23f000c7e0 tx=0x7f23f0007d90 comp rx=0 tx=0).stop 2023-10-10T19:05:13.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 shutdown_connections 2023-10-10T19:05:13.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f23e0066f30 0x7f23e00693e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f240014fd90 0x7f2400169b40 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f240008a100 0x7f24001672b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 --2- 172.21.15.175:0/77111588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24000cd040 0x7f2400169600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:13.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 >> 172.21.15.175:0/77111588 conn(0x7f2400063d00 msgr2=0x7f240014ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:13.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 shutdown_connections 2023-10-10T19:05:13.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:13.575+0000 7f23fffff700 1 -- 172.21.15.175:0/77111588 wait complete. 2023-10-10T19:05:13.579 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:13.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:05:15.588 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:15.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.712+0000 7fb44e416700 1 Processor -- start 2023-10-10T19:05:15.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.712+0000 7fb44e416700 1 -- start start 2023-10-10T19:05:15.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44e416700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb44814fd90 0x7fb448159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:15.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44e416700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb44808a100 0x7fb4480c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:15.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44e416700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb4480cd040 0x7fb4480c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:15.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44e416700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4480c5600 con 0x7fb44808a100 2023-10-10T19:05:15.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44e416700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb448067fa0 con 0x7fb4480cd040 2023-10-10T19:05:15.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44e416700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb448156b70 con 0x7fb44814fd90 2023-10-10T19:05:15.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44cc13700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb44808a100 0x7fb4480c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:15.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44cc13700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb44808a100 0x7fb4480c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54832/0 (socket says 172.21.15.175:54832) 2023-10-10T19:05:15.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44cc13700 1 -- 172.21.15.175:0/2073071327 learned_addr learned my addr 172.21.15.175:0/2073071327 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:15.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44d414700 1 --2- 172.21.15.175:0/2073071327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb44814fd90 0x7fb448159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:15.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44dc15700 1 --2- 172.21.15.175:0/2073071327 >> v2:172.21.15.191:3300/0 conn(0x7fb4480cd040 0x7fb4480c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:15.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44cc13700 1 -- 172.21.15.175:0/2073071327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb44814fd90 msgr2=0x7fb448159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:15.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44cc13700 1 --2- 172.21.15.175:0/2073071327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb44814fd90 0x7fb448159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44cc13700 1 -- 172.21.15.175:0/2073071327 >> v2:172.21.15.191:3300/0 conn(0x7fb4480cd040 msgr2=0x7fb4480c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:15.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44cc13700 1 --2- 172.21.15.175:0/2073071327 >> v2:172.21.15.191:3300/0 conn(0x7fb4480cd040 0x7fb4480c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.713+0000 7fb44cc13700 1 -- 172.21.15.175:0/2073071327 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb448150160 con 0x7fb44808a100 2023-10-10T19:05:15.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44cc13700 1 --2- 172.21.15.175:0/2073071327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44808a100 0x7fb4480c5eb0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb448069e40 tx=0x7fb438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf1e12cd027eaf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:15.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb43ffff700 1 -- 172.21.15.175:0/2073071327 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43800b640 con 0x7fb44808a100 2023-10-10T19:05:15.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb43ffff700 1 -- 172.21.15.175:0/2073071327 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43800b7a0 con 0x7fb44808a100 2023-10-10T19:05:15.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb43ffff700 1 -- 172.21.15.175:0/2073071327 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43800bb60 con 0x7fb44808a100 2023-10-10T19:05:15.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 -- 172.21.15.175:0/2073071327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44808a100 msgr2=0x7fb4480c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:15.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 --2- 172.21.15.175:0/2073071327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44808a100 0x7fb4480c5eb0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb448069e40 tx=0x7fb438009250 comp rx=0 tx=0).stop 2023-10-10T19:05:15.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 -- 172.21.15.175:0/2073071327 shutdown_connections 2023-10-10T19:05:15.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 --2- 172.21.15.175:0/2073071327 >> v2:172.21.15.191:3300/0 conn(0x7fb4480cd040 0x7fb4480c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 --2- 172.21.15.175:0/2073071327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb44814fd90 0x7fb448159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 --2- 172.21.15.175:0/2073071327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44808a100 0x7fb4480c5eb0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 -- 172.21.15.175:0/2073071327 >> 172.21.15.175:0/2073071327 conn(0x7fb448063d00 msgr2=0x7fb4480cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:15.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 -- 172.21.15.175:0/2073071327 shutdown_connections 2023-10-10T19:05:15.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.714+0000 7fb44e416700 1 -- 172.21.15.175:0/2073071327 wait complete. 2023-10-10T19:05:15.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.715+0000 7fb44e416700 1 Processor -- start 2023-10-10T19:05:15.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.715+0000 7fb44e416700 1 -- start start 2023-10-10T19:05:15.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44e416700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb44808a100 0x7fb4481694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:15.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44e416700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4480cd040 0x7fb4480c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:15.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44e416700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44814fd90 0x7fb44814c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:15.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44e416700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b1e60 con 0x7fb44814fd90 2023-10-10T19:05:15.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44e416700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44816a1a0 con 0x7fb44808a100 2023-10-10T19:05:15.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44e416700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480c74e0 con 0x7fb4480cd040 2023-10-10T19:05:15.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44dc15700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44814fd90 0x7fb44814c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:15.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44dc15700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44814fd90 0x7fb44814c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54842/0 (socket says 172.21.15.175:54842) 2023-10-10T19:05:15.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44dc15700 1 -- 172.21.15.175:0/939498666 learned_addr learned my addr 172.21.15.175:0/939498666 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:15.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44cc13700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4480cd040 0x7fb4480c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:15.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.716+0000 7fb44d414700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb44808a100 0x7fb4481694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:15.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb44dc15700 1 -- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4480cd040 msgr2=0x7fb4480c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:15.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb44dc15700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4480cd040 0x7fb4480c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb44dc15700 1 -- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb44808a100 msgr2=0x7fb4481694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:15.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb44dc15700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb44808a100 0x7fb4481694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb44dc15700 1 -- 172.21.15.175:0/939498666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4380094f0 con 0x7fb44814fd90 2023-10-10T19:05:15.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb44dc15700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44814fd90 0x7fb44814c8e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb440009ec0 tx=0x7fb44000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:15.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb43d7fa700 1 -- 172.21.15.175:0/939498666 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000cb70 con 0x7fb44814fd90 2023-10-10T19:05:15.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb43d7fa700 1 -- 172.21.15.175:0/939498666 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44000f040 con 0x7fb44814fd90 2023-10-10T19:05:15.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb448158c30 con 0x7fb44814fd90 2023-10-10T19:05:15.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb43d7fa700 1 -- 172.21.15.175:0/939498666 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440014480 con 0x7fb44814fd90 2023-10-10T19:05:15.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.717+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb448169ec0 con 0x7fb44814fd90 2023-10-10T19:05:15.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.718+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4480697d0 con 0x7fb44814fd90 2023-10-10T19:05:15.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.718+0000 7fb43d7fa700 1 -- 172.21.15.175:0/939498666 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb440014830 con 0x7fb44814fd90 2023-10-10T19:05:15.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.719+0000 7fb43d7fa700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb42c066f30 0x7fb42c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:15.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.719+0000 7fb43d7fa700 1 -- 172.21.15.175:0/939498666 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7fb440085150 con 0x7fb44814fd90 2023-10-10T19:05:15.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.719+0000 7fb44d414700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb42c066f30 0x7fb42c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:15.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.719+0000 7fb44d414700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb42c066f30 0x7fb42c0693e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb444009db0 tx=0x7fb4440091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:15.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.723+0000 7fb43d7fa700 1 -- 172.21.15.175:0/939498666 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb440052040 con 0x7fb44814fd90 2023-10-10T19:05:15.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.946+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb448067570 con 0x7fb44814fd90 2023-10-10T19:05:15.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.947+0000 7fb43d7fa700 1 -- 172.21.15.175:0/939498666 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7fb440055660 con 0x7fb44814fd90 2023-10-10T19:05:15.947 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:15.948 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:15.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb42c066f30 msgr2=0x7fb42c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:15.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb42c066f30 0x7fb42c0693e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb444009db0 tx=0x7fb4440091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44814fd90 msgr2=0x7fb44814c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:15.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44814fd90 0x7fb44814c8e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb440009ec0 tx=0x7fb44000b040 comp rx=0 tx=0).stop 2023-10-10T19:05:15.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 shutdown_connections 2023-10-10T19:05:15.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb42c066f30 0x7fb42c0693e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb44814fd90 0x7fb44814c8e0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb44808a100 0x7fb4481694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 --2- 172.21.15.175:0/939498666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4480cd040 0x7fb4480c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:15.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 >> 172.21.15.175:0/939498666 conn(0x7fb448063d00 msgr2=0x7fb4480c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:15.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 shutdown_connections 2023-10-10T19:05:15.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:15.949+0000 7fb44e416700 1 -- 172.21.15.175:0/939498666 wait complete. 2023-10-10T19:05:15.952 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:15.961 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:05:17.962 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:18.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.088+0000 7f70a456c700 1 Processor -- start 2023-10-10T19:05:18.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.088+0000 7f70a456c700 1 -- start start 2023-10-10T19:05:18.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.089+0000 7f70a456c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:18.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.089+0000 7f70a456c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f709c0b2390 0x7f709c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:18.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.089+0000 7f70a456c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709c1574b0 con 0x7f709c068bd0 2023-10-10T19:05:18.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.089+0000 7f70a456c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709c150770 con 0x7f709c0b2390 2023-10-10T19:05:18.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.089+0000 7f70a456c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709c0afad0 con 0x7f709c0b1fc0 2023-10-10T19:05:18.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.089+0000 7f70a1b07700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f709c0b2390 0x7f709c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:18.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.089+0000 7f70a1b07700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f709c0b2390 0x7f709c068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54850/0 (socket says 172.21.15.175:54850) 2023-10-10T19:05:18.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.089+0000 7f70a1b07700 1 -- 172.21.15.175:0/265426481 learned_addr learned my addr 172.21.15.175:0/265426481 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:18.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a2308700 1 --2- 172.21.15.175:0/265426481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:18.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a1b07700 1 -- 172.21.15.175:0/265426481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 msgr2=0x7f709c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:18.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a1b07700 1 --2- 172.21.15.175:0/265426481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a1b07700 1 -- 172.21.15.175:0/265426481 >> v1:172.21.15.191:6789/0 conn(0x7f709c068bd0 legacy=0x7f709c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:18.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a1b07700 1 -- 172.21.15.175:0/265426481 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c0af4c0 con 0x7f709c0b2390 2023-10-10T19:05:18.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a2308700 1 --2- 172.21.15.175:0/265426481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:05:18.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a1b07700 1 --2- 172.21.15.175:0/265426481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c0b2390 0x7f709c068690 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f708c0099b0 tx=0x7f708c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20469fc0f82a538c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:18.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a1306700 1 -- 172.21.15.175:0/265426481 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c00b8b0 con 0x7f709c0b2390 2023-10-10T19:05:18.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a1306700 1 -- 172.21.15.175:0/265426481 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708c00ba10 con 0x7f709c0b2390 2023-10-10T19:05:18.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a1306700 1 -- 172.21.15.175:0/265426481 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c00bdb0 con 0x7f709c0b2390 2023-10-10T19:05:18.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a456c700 1 -- 172.21.15.175:0/265426481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c0b2390 msgr2=0x7f709c068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:18.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a456c700 1 --2- 172.21.15.175:0/265426481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c0b2390 0x7f709c068690 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f708c0099b0 tx=0x7f708c009250 comp rx=0 tx=0).stop 2023-10-10T19:05:18.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a456c700 1 -- 172.21.15.175:0/265426481 shutdown_connections 2023-10-10T19:05:18.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a456c700 1 --2- 172.21.15.175:0/265426481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.090+0000 7f70a456c700 1 --2- 172.21.15.175:0/265426481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c0b2390 0x7f709c068690 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.091+0000 7f70a456c700 1 -- 172.21.15.175:0/265426481 >> 172.21.15.175:0/265426481 conn(0x7f709c063230 msgr2=0x7f709c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:18.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.091+0000 7f70a456c700 1 -- 172.21.15.175:0/265426481 shutdown_connections 2023-10-10T19:05:18.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.091+0000 7f70a456c700 1 -- 172.21.15.175:0/265426481 wait complete. 2023-10-10T19:05:18.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a456c700 1 Processor -- start 2023-10-10T19:05:18.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a456c700 1 -- start start 2023-10-10T19:05:18.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a456c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c068bd0 0x7f709c1665f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:18.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a456c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c0cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:18.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a456c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f709c0b2390 0x7f709c0c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:18.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a456c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709c166b30 con 0x7f709c068bd0 2023-10-10T19:05:18.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a456c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709c0c1b80 con 0x7f709c0b2390 2023-10-10T19:05:18.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a456c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709c0c1290 con 0x7f709c0b1fc0 2023-10-10T19:05:18.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a2308700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c068bd0 0x7f709c1665f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:18.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a2308700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c068bd0 0x7f709c1665f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54856/0 (socket says 172.21.15.175:54856) 2023-10-10T19:05:18.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a2308700 1 -- 172.21.15.175:0/1969397672 learned_addr learned my addr 172.21.15.175:0/1969397672 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:18.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a1b07700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c0cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:18.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.092+0000 7f70a2b09700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f709c0b2390 0x7f709c0c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:18.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70a2308700 1 -- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 msgr2=0x7f709c0cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:18.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70a2308700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c0cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70a2308700 1 -- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f709c0b2390 msgr2=0x7f709c0c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:18.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70a2308700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f709c0b2390 0x7f709c0c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70a2308700 1 -- 172.21.15.175:0/1969397672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c0094f0 con 0x7f709c068bd0 2023-10-10T19:05:18.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70a2308700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c068bd0 0x7f709c1665f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f709800a430 tx=0x7f7098007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:18.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70927fc700 1 -- 172.21.15.175:0/1969397672 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098015440 con 0x7f709c068bd0 2023-10-10T19:05:18.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70927fc700 1 -- 172.21.15.175:0/1969397672 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70980155a0 con 0x7f709c068bd0 2023-10-10T19:05:18.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70927fc700 1 -- 172.21.15.175:0/1969397672 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098015940 con 0x7f709c068bd0 2023-10-10T19:05:18.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f709c0c2d90 con 0x7f709c068bd0 2023-10-10T19:05:18.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.093+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f709c0c1530 con 0x7f709c068bd0 2023-10-10T19:05:18.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.094+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709c067570 con 0x7f709c068bd0 2023-10-10T19:05:18.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.095+0000 7f70927fc700 1 -- 172.21.15.175:0/1969397672 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7098015cb0 con 0x7f709c068bd0 2023-10-10T19:05:18.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.095+0000 7f70927fc700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7080066f30 0x7f70800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:18.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.095+0000 7f70927fc700 1 -- 172.21.15.175:0/1969397672 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f7098085030 con 0x7f709c068bd0 2023-10-10T19:05:18.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.095+0000 7f70a1b07700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7080066f30 0x7f70800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:18.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.096+0000 7f70a1b07700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7080066f30 0x7f70800693e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f708c0099b0 tx=0x7f708c016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:18.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.099+0000 7f70927fc700 1 -- 172.21.15.175:0/1969397672 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f709804e5a0 con 0x7f709c068bd0 2023-10-10T19:05:18.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.322+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f709c069060 con 0x7f709c068bd0 2023-10-10T19:05:18.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.323+0000 7f70927fc700 1 -- 172.21.15.175:0/1969397672 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7f7098051fd0 con 0x7f709c068bd0 2023-10-10T19:05:18.324 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:18.324 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:18.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7080066f30 msgr2=0x7f70800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:18.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7080066f30 0x7f70800693e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f708c0099b0 tx=0x7f708c016000 comp rx=0 tx=0).stop 2023-10-10T19:05:18.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c068bd0 msgr2=0x7f709c1665f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:18.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c068bd0 0x7f709c1665f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f709800a430 tx=0x7f7098007910 comp rx=0 tx=0).stop 2023-10-10T19:05:18.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 shutdown_connections 2023-10-10T19:05:18.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7080066f30 0x7f70800693e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f709c0b2390 0x7f709c0c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f709c068bd0 0x7f709c1665f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 --2- 172.21.15.175:0/1969397672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f709c0b1fc0 0x7f709c0cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:18.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 >> 172.21.15.175:0/1969397672 conn(0x7f709c063230 msgr2=0x7f709c150db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:18.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 shutdown_connections 2023-10-10T19:05:18.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:18.325+0000 7f70a456c700 1 -- 172.21.15.175:0/1969397672 wait complete. 2023-10-10T19:05:18.328 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:18.337 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:05:20.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:20.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.466+0000 7fd5b4348700 1 Processor -- start 2023-10-10T19:05:20.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.466+0000 7fd5b4348700 1 -- start start 2023-10-10T19:05:20.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b4348700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5ac0c2bd0 0x7fd5ac06f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:20.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b4348700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2fa0 0x7fd5ac0c33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:20.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b4348700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5ac157970 0x7fd5ac157da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:20.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b4348700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ac06f830 con 0x7fd5ac157970 2023-10-10T19:05:20.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b4348700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ac0c65a0 con 0x7fd5ac0c2bd0 2023-10-10T19:05:20.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b4348700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac06a3e0 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5ac157970 0x7fd5ac157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:20.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5ac157970 0x7fd5ac157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54860/0 (socket says 172.21.15.175:54860) 2023-10-10T19:05:20.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 -- 172.21.15.175:0/367487900 learned_addr learned my addr 172.21.15.175:0/367487900 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:20.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b20e4700 1 --2- 172.21.15.175:0/367487900 >> v2:172.21.15.191:3300/0 conn(0x7fd5ac0c2bd0 0x7fd5ac06f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:20.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b18e3700 1 --2- 172.21.15.175:0/367487900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2fa0 0x7fd5ac0c33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:20.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 -- 172.21.15.175:0/367487900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2fa0 msgr2=0x7fd5ac0c33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:20.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 --2- 172.21.15.175:0/367487900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2fa0 0x7fd5ac0c33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 -- 172.21.15.175:0/367487900 >> v2:172.21.15.191:3300/0 conn(0x7fd5ac0c2bd0 msgr2=0x7fd5ac06f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:20.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 --2- 172.21.15.175:0/367487900 >> v2:172.21.15.191:3300/0 conn(0x7fd5ac0c2bd0 0x7fd5ac06f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 -- 172.21.15.175:0/367487900 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac1582e0 con 0x7fd5ac157970 2023-10-10T19:05:20.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.467+0000 7fd5b28e5700 1 --2- 172.21.15.175:0/367487900 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac157970 0x7fd5ac157da0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac06faf0 tx=0x7fd5a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669eaab25b381ef5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:20.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b10e2700 1 -- 172.21.15.175:0/367487900 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a800b860 con 0x7fd5ac157970 2023-10-10T19:05:20.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b10e2700 1 -- 172.21.15.175:0/367487900 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a800b9c0 con 0x7fd5ac157970 2023-10-10T19:05:20.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b10e2700 1 -- 172.21.15.175:0/367487900 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a800bd60 con 0x7fd5ac157970 2023-10-10T19:05:20.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 -- 172.21.15.175:0/367487900 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac157970 msgr2=0x7fd5ac157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:20.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 --2- 172.21.15.175:0/367487900 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac157970 0x7fd5ac157da0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac06faf0 tx=0x7fd5a8009250 comp rx=0 tx=0).stop 2023-10-10T19:05:20.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 -- 172.21.15.175:0/367487900 shutdown_connections 2023-10-10T19:05:20.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 --2- 172.21.15.175:0/367487900 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac157970 0x7fd5ac157da0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 --2- 172.21.15.175:0/367487900 >> v2:172.21.15.191:3300/0 conn(0x7fd5ac0c2bd0 0x7fd5ac06f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 --2- 172.21.15.175:0/367487900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2fa0 0x7fd5ac0c33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 -- 172.21.15.175:0/367487900 >> 172.21.15.175:0/367487900 conn(0x7fd5ac0c9eb0 msgr2=0x7fd5ac0c5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:20.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 -- 172.21.15.175:0/367487900 shutdown_connections 2023-10-10T19:05:20.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.468+0000 7fd5b4348700 1 -- 172.21.15.175:0/367487900 wait complete. 2023-10-10T19:05:20.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.469+0000 7fd5b4348700 1 Processor -- start 2023-10-10T19:05:20.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.469+0000 7fd5b4348700 1 -- start start 2023-10-10T19:05:20.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b4348700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2bd0 0x7fd5ac0c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:20.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b4348700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac0c2fa0 0x7fd5ac165740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:20.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b4348700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5ac157970 0x7fd5ac165c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:20.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b4348700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac0c8830 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b4348700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac1652b0 con 0x7fd5ac157970 2023-10-10T19:05:20.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b4348700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac1645f0 con 0x7fd5ac0c2bd0 2023-10-10T19:05:20.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b18e3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac0c2fa0 0x7fd5ac165740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:20.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b18e3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac0c2fa0 0x7fd5ac165740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54876/0 (socket says 172.21.15.175:54876) 2023-10-10T19:05:20.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b18e3700 1 -- 172.21.15.175:0/1226353847 learned_addr learned my addr 172.21.15.175:0/1226353847 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:20.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b20e4700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2bd0 0x7fd5ac0c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:20.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b28e5700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5ac157970 0x7fd5ac165c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:20.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b18e3700 1 -- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2bd0 msgr2=0x7fd5ac0c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:20.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b18e3700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2bd0 0x7fd5ac0c7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b18e3700 1 -- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5ac157970 msgr2=0x7fd5ac165c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:20.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b18e3700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5ac157970 0x7fd5ac165c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.470+0000 7fd5b18e3700 1 -- 172.21.15.175:0/1226353847 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a80094f0 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.471+0000 7fd5b18e3700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac0c2fa0 0x7fd5ac165740 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00d8c0 tx=0x7fd59c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:20.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.471+0000 7fd5a27fc700 1 -- 172.21.15.175:0/1226353847 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0125f0 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.471+0000 7fd5a27fc700 1 -- 172.21.15.175:0/1226353847 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c012750 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.471+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ac164ca0 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.471+0000 7fd5a27fc700 1 -- 172.21.15.175:0/1226353847 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c012af0 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.471+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ac164f40 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.472+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ac165460 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.472+0000 7fd5a27fc700 1 -- 172.21.15.175:0/1226353847 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd59c01e460 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.472+0000 7fd5a27fc700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd590066f30 0x7fd5900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:20.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.473+0000 7fd5a27fc700 1 -- 172.21.15.175:0/1226353847 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7fd59c084e30 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.473+0000 7fd5b20e4700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd590066f30 0x7fd5900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:20.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.473+0000 7fd5b20e4700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd590066f30 0x7fd5900693e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4009fd0 tx=0x7fd5a4009880 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:20.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.477+0000 7fd5a27fc700 1 -- 172.21.15.175:0/1226353847 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd5ac165460 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.701+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5ac165460 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.701+0000 7fd5a27fc700 1 -- 172.21.15.175:0/1226353847 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7fd5ac165460 con 0x7fd5ac0c2fa0 2023-10-10T19:05:20.702 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:20.702 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:20.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.703+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd590066f30 msgr2=0x7fd5900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:20.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd590066f30 0x7fd5900693e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4009fd0 tx=0x7fd5a4009880 comp rx=0 tx=0).stop 2023-10-10T19:05:20.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac0c2fa0 msgr2=0x7fd5ac165740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:20.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac0c2fa0 0x7fd5ac165740 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00d8c0 tx=0x7fd59c008040 comp rx=0 tx=0).stop 2023-10-10T19:05:20.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 shutdown_connections 2023-10-10T19:05:20.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd590066f30 0x7fd5900693e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5ac157970 0x7fd5ac165c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5ac0c2bd0 0x7fd5ac0c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 --2- 172.21.15.175:0/1226353847 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5ac0c2fa0 0x7fd5ac165740 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:20.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 >> 172.21.15.175:0/1226353847 conn(0x7fd5ac0c9eb0 msgr2=0x7fd5ac149100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:20.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 shutdown_connections 2023-10-10T19:05:20.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:20.704+0000 7fd5b4348700 1 -- 172.21.15.175:0/1226353847 wait complete. 2023-10-10T19:05:20.707 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:20.716 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:05:21.405 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:05:21.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:05:22.718 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:22.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.844+0000 7ff0742a4700 1 Processor -- start 2023-10-10T19:05:22.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.844+0000 7ff0742a4700 1 -- start start 2023-10-10T19:05:22.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.845+0000 7ff0742a4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff06c1577a0 0x7ff06c0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:22.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.845+0000 7ff0742a4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06c154aa0 con 0x7ff06c1573d0 2023-10-10T19:05:22.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.845+0000 7ff0742a4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06c063dc0 con 0x7ff06c0c12a0 2023-10-10T19:05:22.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.845+0000 7ff0742a4700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06c0bfdf0 con 0x7ff06c1577a0 2023-10-10T19:05:22.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff072040700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff06c1573d0 0x7ff06c0bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46578/0 (socket says 172.21.15.175:46578) 2023-10-10T19:05:22.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff072040700 1 -- 172.21.15.175:0/1522969095 learned_addr learned my addr 172.21.15.175:0/1522969095 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:22.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff072841700 1 --2- 172.21.15.175:0/1522969095 >> v2:172.21.15.175:3300/0 conn(0x7ff06c1577a0 0x7ff06c0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:22.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff07103e700 1 -- 172.21.15.175:0/1522969095 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968852261 0 0) 0x7ff06c154aa0 con 0x7ff06c1573d0 2023-10-10T19:05:22.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff07103e700 1 -- 172.21.15.175:0/1522969095 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff058003570 con 0x7ff06c1573d0 2023-10-10T19:05:22.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff072841700 1 -- 172.21.15.175:0/1522969095 >> v1:172.21.15.191:6789/0 conn(0x7ff06c0c12a0 legacy=0x7ff06c14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:22.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff072841700 1 -- 172.21.15.175:0/1522969095 >> v1:172.21.15.175:6789/0 conn(0x7ff06c1573d0 legacy=0x7ff06c0bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:22.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff072841700 1 -- 172.21.15.175:0/1522969095 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c157ba0 con 0x7ff06c1577a0 2023-10-10T19:05:22.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff072841700 1 --2- 172.21.15.175:0/1522969095 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c4f20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff0640098e0 tx=0x7ff0640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf110f7215d931d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:22.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff07103e700 1 -- 172.21.15.175:0/1522969095 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06400b640 con 0x7ff06c1577a0 2023-10-10T19:05:22.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff07103e700 1 -- 172.21.15.175:0/1522969095 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06400b7a0 con 0x7ff06c1577a0 2023-10-10T19:05:22.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.846+0000 7ff07103e700 1 -- 172.21.15.175:0/1522969095 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06400bb40 con 0x7ff06c1577a0 2023-10-10T19:05:22.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.847+0000 7ff0742a4700 1 -- 172.21.15.175:0/1522969095 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 msgr2=0x7ff06c0c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:22.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.847+0000 7ff0742a4700 1 --2- 172.21.15.175:0/1522969095 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c4f20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff0640098e0 tx=0x7ff0640091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:22.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.847+0000 7ff0742a4700 1 -- 172.21.15.175:0/1522969095 shutdown_connections 2023-10-10T19:05:22.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.847+0000 7ff0742a4700 1 --2- 172.21.15.175:0/1522969095 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c4f20 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:22.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.847+0000 7ff0742a4700 1 -- 172.21.15.175:0/1522969095 >> 172.21.15.175:0/1522969095 conn(0x7ff06c06dc70 msgr2=0x7ff06c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:22.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.847+0000 7ff0742a4700 1 -- 172.21.15.175:0/1522969095 shutdown_connections 2023-10-10T19:05:22.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.847+0000 7ff0742a4700 1 -- 172.21.15.175:0/1522969095 wait complete. 2023-10-10T19:05:22.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.848+0000 7ff0742a4700 1 Processor -- start 2023-10-10T19:05:22.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.848+0000 7ff0742a4700 1 -- start start 2023-10-10T19:05:22.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0742a4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff06c0c12a0 0x7ff06c1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:22.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0742a4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff06c1573d0 0x7ff06c0c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:22.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0742a4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:22.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0742a4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c155a00 con 0x7ff06c1577a0 2023-10-10T19:05:22.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0742a4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c155ce0 con 0x7ff06c1573d0 2023-10-10T19:05:22.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0742a4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c155fc0 con 0x7ff06c0c12a0 2023-10-10T19:05:22.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:22.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54890/0 (socket says 172.21.15.175:54890) 2023-10-10T19:05:22.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 -- 172.21.15.175:0/974393225 learned_addr learned my addr 172.21.15.175:0/974393225 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:22.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072040700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff06c0c12a0 0x7ff06c1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:22.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff07183f700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff06c1573d0 0x7ff06c0c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:22.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 -- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff06c0c12a0 msgr2=0x7ff06c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:22.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff06c0c12a0 0x7ff06c1533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:22.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 -- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff06c1573d0 msgr2=0x7ff06c0c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:22.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff06c1573d0 0x7ff06c0c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:22.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 -- 172.21.15.175:0/974393225 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff064009420 con 0x7ff06c1577a0 2023-10-10T19:05:22.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff072841700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c5d00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff06c06e6e0 tx=0x7ff064017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:22.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0627fc700 1 -- 172.21.15.175:0/974393225 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06400bae0 con 0x7ff06c1577a0 2023-10-10T19:05:22.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0627fc700 1 -- 172.21.15.175:0/974393225 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff064011630 con 0x7ff06c1577a0 2023-10-10T19:05:22.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06c1562a0 con 0x7ff06c1577a0 2023-10-10T19:05:22.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0627fc700 1 -- 172.21.15.175:0/974393225 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0640119d0 con 0x7ff06c1577a0 2023-10-10T19:05:22.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.849+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06c156f30 con 0x7ff06c1577a0 2023-10-10T19:05:22.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.850+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06c1567c0 con 0x7ff06c1577a0 2023-10-10T19:05:22.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.851+0000 7ff0627fc700 1 -- 172.21.15.175:0/974393225 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff064011d40 con 0x7ff06c1577a0 2023-10-10T19:05:22.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.851+0000 7ff0627fc700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff050066f30 0x7ff0500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:22.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.851+0000 7ff0627fc700 1 -- 172.21.15.175:0/974393225 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7ff064085e70 con 0x7ff06c1577a0 2023-10-10T19:05:22.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.851+0000 7ff072040700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff050066f30 0x7ff0500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:22.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.852+0000 7ff072040700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff050066f30 0x7ff0500693e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff068005c50 tx=0x7ff068009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:22.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:22.855+0000 7ff0627fc700 1 -- 172.21.15.175:0/974393225 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff06c1567c0 con 0x7ff06c1577a0 2023-10-10T19:05:23.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.083+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff06c1567c0 con 0x7ff06c1577a0 2023-10-10T19:05:23.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.083+0000 7ff0627fc700 1 -- 172.21.15.175:0/974393225 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7ff06c1567c0 con 0x7ff06c1577a0 2023-10-10T19:05:23.084 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:23.084 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:23.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff050066f30 msgr2=0x7ff0500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:23.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff050066f30 0x7ff0500693e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff068005c50 tx=0x7ff068009040 comp rx=0 tx=0).stop 2023-10-10T19:05:23.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 msgr2=0x7ff06c0c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:23.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c5d00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff06c06e6e0 tx=0x7ff064017000 comp rx=0 tx=0).stop 2023-10-10T19:05:23.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 shutdown_connections 2023-10-10T19:05:23.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff050066f30 0x7ff0500693e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:23.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff06c1577a0 0x7ff06c0c5d00 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:23.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff06c0c12a0 0x7ff06c1533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:23.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 --2- 172.21.15.175:0/974393225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff06c1573d0 0x7ff06c0c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:23.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 >> 172.21.15.175:0/974393225 conn(0x7ff06c06dc70 msgr2=0x7ff06c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:23.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 shutdown_connections 2023-10-10T19:05:23.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:23.086+0000 7ff0742a4700 1 -- 172.21.15.175:0/974393225 wait complete. 2023-10-10T19:05:23.089 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:23.098 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:05:25.098 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:25.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.225+0000 7ff463648700 1 Processor -- start 2023-10-10T19:05:25.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.225+0000 7ff463648700 1 -- start start 2023-10-10T19:05:25.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.225+0000 7ff463648700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 0x7ff45c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:25.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.225+0000 7ff463648700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff45c0aff60 0x7ff45c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff463648700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c1574b0 con 0x7ff45c0b1fc0 2023-10-10T19:05:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff463648700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c150770 con 0x7ff45c0aff60 2023-10-10T19:05:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff463648700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0afad0 con 0x7ff45c0b2390 2023-10-10T19:05:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff461be5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff45c0aff60 0x7ff45c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff461be5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff45c0aff60 0x7ff45c0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39916/0 (socket says 172.21.15.175:39916) 2023-10-10T19:05:25.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff461be5700 1 -- 172.21.15.175:0/3708001358 learned_addr learned my addr 172.21.15.175:0/3708001358 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:25.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff460be3700 1 --2- 172.21.15.175:0/3708001358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 0x7ff45c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:25.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff461be5700 1 -- 172.21.15.175:0/3708001358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 msgr2=0x7ff45c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:25.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff461be5700 1 --2- 172.21.15.175:0/3708001358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 0x7ff45c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff461be5700 1 -- 172.21.15.175:0/3708001358 >> v1:172.21.15.191:6789/0 conn(0x7ff45c0b1fc0 legacy=0x7ff45c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff461be5700 1 -- 172.21.15.175:0/3708001358 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c0af4c0 con 0x7ff45c0aff60 2023-10-10T19:05:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff461be5700 1 --2- 172.21.15.175:0/3708001358 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0b03e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff4540099e0 tx=0x7ff454009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd54c7463c7dcf57 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff453fff700 1 -- 172.21.15.175:0/3708001358 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45400b8b0 con 0x7ff45c0aff60 2023-10-10T19:05:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff453fff700 1 -- 172.21.15.175:0/3708001358 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45400ba10 con 0x7ff45c0aff60 2023-10-10T19:05:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.226+0000 7ff453fff700 1 -- 172.21.15.175:0/3708001358 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45400bdb0 con 0x7ff45c0aff60 2023-10-10T19:05:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.227+0000 7ff463648700 1 -- 172.21.15.175:0/3708001358 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 msgr2=0x7ff45c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.227+0000 7ff463648700 1 --2- 172.21.15.175:0/3708001358 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0b03e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff4540099e0 tx=0x7ff454009250 comp rx=0 tx=0).stop 2023-10-10T19:05:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.227+0000 7ff463648700 1 -- 172.21.15.175:0/3708001358 shutdown_connections 2023-10-10T19:05:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.227+0000 7ff463648700 1 --2- 172.21.15.175:0/3708001358 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0b03e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.227+0000 7ff463648700 1 --2- 172.21.15.175:0/3708001358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 0x7ff45c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.227+0000 7ff463648700 1 -- 172.21.15.175:0/3708001358 >> 172.21.15.175:0/3708001358 conn(0x7ff45c063230 msgr2=0x7ff45c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.227+0000 7ff463648700 1 -- 172.21.15.175:0/3708001358 shutdown_connections 2023-10-10T19:05:25.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.227+0000 7ff463648700 1 -- 172.21.15.175:0/3708001358 wait complete. 2023-10-10T19:05:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.228+0000 7ff463648700 1 Processor -- start 2023-10-10T19:05:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.228+0000 7ff463648700 1 -- start start 2023-10-10T19:05:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff463648700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:25.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff463648700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c0b1fc0 0x7ff45c169120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:25.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff463648700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 0x7ff45c14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:25.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff463648700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c154740 con 0x7ff45c0aff60 2023-10-10T19:05:25.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff463648700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c156a90 con 0x7ff45c0b1fc0 2023-10-10T19:05:25.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff463648700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c169c30 con 0x7ff45c0b2390 2023-10-10T19:05:25.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff4613e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:25.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff4613e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0c3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39928/0 (socket says 172.21.15.175:39928) 2023-10-10T19:05:25.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff4613e4700 1 -- 172.21.15.175:0/2743579626 learned_addr learned my addr 172.21.15.175:0/2743579626 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:25.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff460be3700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c0b1fc0 0x7ff45c169120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:25.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff461be5700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 0x7ff45c14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:25.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff4613e4700 1 -- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 msgr2=0x7ff45c14f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:25.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff4613e4700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 0x7ff45c14f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff4613e4700 1 -- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c0b1fc0 msgr2=0x7ff45c169120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:25.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff4613e4700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c0b1fc0 0x7ff45c169120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.229+0000 7ff4613e4700 1 -- 172.21.15.175:0/2743579626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4540094f0 con 0x7ff45c0aff60 2023-10-10T19:05:25.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.230+0000 7ff4613e4700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0c3870 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff45800cb40 tx=0x7ff458001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:25.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.230+0000 7ff4517fa700 1 -- 172.21.15.175:0/2743579626 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458002060 con 0x7ff45c0aff60 2023-10-10T19:05:25.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.230+0000 7ff4517fa700 1 -- 172.21.15.175:0/2743579626 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45800a7e0 con 0x7ff45c0aff60 2023-10-10T19:05:25.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.230+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c156f20 con 0x7ff45c0aff60 2023-10-10T19:05:25.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.230+0000 7ff4517fa700 1 -- 172.21.15.175:0/2743579626 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45800aba0 con 0x7ff45c0aff60 2023-10-10T19:05:25.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.230+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c154460 con 0x7ff45c0aff60 2023-10-10T19:05:25.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.231+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c156d70 con 0x7ff45c0aff60 2023-10-10T19:05:25.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.231+0000 7ff4517fa700 1 -- 172.21.15.175:0/2743579626 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff458003030 con 0x7ff45c0aff60 2023-10-10T19:05:25.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.231+0000 7ff4517fa700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff440066f30 0x7ff4400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:25.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.232+0000 7ff4517fa700 1 -- 172.21.15.175:0/2743579626 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7ff458084810 con 0x7ff45c0aff60 2023-10-10T19:05:25.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.232+0000 7ff460be3700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff440066f30 0x7ff4400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:25.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.232+0000 7ff460be3700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff440066f30 0x7ff4400693e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff45c154070 tx=0x7ff44c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:25.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.236+0000 7ff4517fa700 1 -- 172.21.15.175:0/2743579626 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff458051700 con 0x7ff45c0aff60 2023-10-10T19:05:25.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.459+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff45c1568c0 con 0x7ff45c0aff60 2023-10-10T19:05:25.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.460+0000 7ff4517fa700 1 -- 172.21.15.175:0/2743579626 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7ff458054d20 con 0x7ff45c0aff60 2023-10-10T19:05:25.460 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:25.461 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:25.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff440066f30 msgr2=0x7ff4400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:25.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff440066f30 0x7ff4400693e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff45c154070 tx=0x7ff44c008040 comp rx=0 tx=0).stop 2023-10-10T19:05:25.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 msgr2=0x7ff45c0c3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:25.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0c3870 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff45800cb40 tx=0x7ff458001770 comp rx=0 tx=0).stop 2023-10-10T19:05:25.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 shutdown_connections 2023-10-10T19:05:25.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff440066f30 0x7ff4400693e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff45c0b2390 0x7ff45c14f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff45c0aff60 0x7ff45c0c3870 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 --2- 172.21.15.175:0/2743579626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c0b1fc0 0x7ff45c169120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:25.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 >> 172.21.15.175:0/2743579626 conn(0x7ff45c063230 msgr2=0x7ff45c0c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:25.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 shutdown_connections 2023-10-10T19:05:25.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:25.462+0000 7ff463648700 1 -- 172.21.15.175:0/2743579626 wait complete. 2023-10-10T19:05:25.465 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:25.474 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:05:27.477 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:27.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.605+0000 7f6495b12700 1 Processor -- start 2023-10-10T19:05:27.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.605+0000 7f6495b12700 1 -- start start 2023-10-10T19:05:27.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f6495b12700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64900b1fc0 0x7f6490155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:27.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f6495b12700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f64900b2390 0x7f6490068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:27.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f6495b12700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6490068bd0 0x7f64900aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:27.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f6495b12700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6490150770 con 0x7f6490068bd0 2023-10-10T19:05:27.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f6495b12700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64901574b0 con 0x7f64900b2390 2023-10-10T19:05:27.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f6495b12700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64900afad0 con 0x7f64900b1fc0 2023-10-10T19:05:27.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f648ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6490068bd0 0x7f64900aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:27.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f648ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6490068bd0 0x7f64900aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39938/0 (socket says 172.21.15.175:39938) 2023-10-10T19:05:27.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f648ffff700 1 -- 172.21.15.175:0/2781844327 learned_addr learned my addr 172.21.15.175:0/2781844327 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:27.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f648effd700 1 --2- 172.21.15.175:0/2781844327 >> v2:172.21.15.191:3300/0 conn(0x7f64900b2390 0x7f6490068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:27.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.606+0000 7f648f7fe700 1 --2- 172.21.15.175:0/2781844327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64900b1fc0 0x7f6490155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:27.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648ffff700 1 -- 172.21.15.175:0/2781844327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64900b1fc0 msgr2=0x7f6490155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:27.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648ffff700 1 --2- 172.21.15.175:0/2781844327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64900b1fc0 0x7f6490155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648ffff700 1 -- 172.21.15.175:0/2781844327 >> v2:172.21.15.191:3300/0 conn(0x7f64900b2390 msgr2=0x7f6490068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:27.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648ffff700 1 --2- 172.21.15.175:0/2781844327 >> v2:172.21.15.191:3300/0 conn(0x7f64900b2390 0x7f6490068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648ffff700 1 -- 172.21.15.175:0/2781844327 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64900af4c0 con 0x7f6490068bd0 2023-10-10T19:05:27.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648ffff700 1 --2- 172.21.15.175:0/2781844327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6490068bd0 0x7f64900aff60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6490071640 tx=0x7f6488009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61155061f84ed9c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:27.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648e7fc700 1 -- 172.21.15.175:0/2781844327 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648800b6d0 con 0x7f6490068bd0 2023-10-10T19:05:27.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648e7fc700 1 -- 172.21.15.175:0/2781844327 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648800b830 con 0x7f6490068bd0 2023-10-10T19:05:27.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f648e7fc700 1 -- 172.21.15.175:0/2781844327 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648800bbf0 con 0x7f6490068bd0 2023-10-10T19:05:27.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f6495b12700 1 -- 172.21.15.175:0/2781844327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6490068bd0 msgr2=0x7f64900aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:27.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.607+0000 7f6495b12700 1 --2- 172.21.15.175:0/2781844327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6490068bd0 0x7f64900aff60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6490071640 tx=0x7f6488009250 comp rx=0 tx=0).stop 2023-10-10T19:05:27.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.608+0000 7f6495b12700 1 -- 172.21.15.175:0/2781844327 shutdown_connections 2023-10-10T19:05:27.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.608+0000 7f6495b12700 1 --2- 172.21.15.175:0/2781844327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6490068bd0 0x7f64900aff60 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.608+0000 7f6495b12700 1 --2- 172.21.15.175:0/2781844327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64900b1fc0 0x7f6490155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.608+0000 7f6495b12700 1 --2- 172.21.15.175:0/2781844327 >> v2:172.21.15.191:3300/0 conn(0x7f64900b2390 0x7f6490068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.608+0000 7f6495b12700 1 -- 172.21.15.175:0/2781844327 >> 172.21.15.175:0/2781844327 conn(0x7f6490063230 msgr2=0x7f6490063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:27.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.608+0000 7f6495b12700 1 -- 172.21.15.175:0/2781844327 shutdown_connections 2023-10-10T19:05:27.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.608+0000 7f6495b12700 1 -- 172.21.15.175:0/2781844327 wait complete. 2023-10-10T19:05:27.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.609+0000 7f6495b12700 1 Processor -- start 2023-10-10T19:05:27.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.609+0000 7f6495b12700 1 -- start start 2023-10-10T19:05:27.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f6495b12700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6490068bd0 0x7f64901692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:27.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f6495b12700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64900b1fc0 0x7f6490165b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:27.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f6495b12700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64900b2390 0x7f649015cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:27.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f6495b12700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6490165440 con 0x7f64900b2390 2023-10-10T19:05:27.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f6495b12700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64900c4c20 con 0x7f64900b1fc0 2023-10-10T19:05:27.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f6495b12700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64900c39f0 con 0x7f6490068bd0 2023-10-10T19:05:27.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64900b2390 0x7f649015cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:27.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64900b2390 0x7f649015cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39942/0 (socket says 172.21.15.175:39942) 2023-10-10T19:05:27.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 -- 172.21.15.175:0/3624097340 learned_addr learned my addr 172.21.15.175:0/3624097340 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:27.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648effd700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64900b1fc0 0x7f6490165b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:27.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648f7fe700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6490068bd0 0x7f64901692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:27.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 -- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6490068bd0 msgr2=0x7f64901692b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:27.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6490068bd0 0x7f64901692b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 -- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64900b1fc0 msgr2=0x7f6490165b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:27.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64900b1fc0 0x7f6490165b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 -- 172.21.15.175:0/3624097340 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64880094f0 con 0x7f64900b2390 2023-10-10T19:05:27.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648effd700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64900b1fc0 0x7f6490165b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:05:27.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648f7fe700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6490068bd0 0x7f64901692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:05:27.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f648ffff700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64900b2390 0x7f649015cc50 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f649006fd90 tx=0x7f6488017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:27.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f64777fe700 1 -- 172.21.15.175:0/3624097340 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648800bb70 con 0x7f64900b2390 2023-10-10T19:05:27.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f64777fe700 1 -- 172.21.15.175:0/3624097340 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6488011630 con 0x7f64900b2390 2023-10-10T19:05:27.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64901656e0 con 0x7f64900b2390 2023-10-10T19:05:27.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f64777fe700 1 -- 172.21.15.175:0/3624097340 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64880119f0 con 0x7f64900b2390 2023-10-10T19:05:27.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.610+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64900cb110 con 0x7f64900b2390 2023-10-10T19:05:27.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.611+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6490165870 con 0x7f64900b2390 2023-10-10T19:05:27.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.612+0000 7f64777fe700 1 -- 172.21.15.175:0/3624097340 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f64880253f0 con 0x7f64900b2390 2023-10-10T19:05:27.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.612+0000 7f64777fe700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f647c066f30 0x7f647c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:27.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.612+0000 7f64777fe700 1 -- 172.21.15.175:0/3624097340 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f64880863e0 con 0x7f64900b2390 2023-10-10T19:05:27.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.612+0000 7f648f7fe700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f647c066f30 0x7f647c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:27.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.613+0000 7f648f7fe700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f647c066f30 0x7f647c0693e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f64800076c0 tx=0x7f648000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:27.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.617+0000 7f64777fe700 1 -- 172.21.15.175:0/3624097340 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6488053350 con 0x7f64900b2390 2023-10-10T19:05:27.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.844+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6490067570 con 0x7f64900b2390 2023-10-10T19:05:27.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.845+0000 7f64777fe700 1 -- 172.21.15.175:0/3624097340 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6211 (secure 0 0 0) 0x7f6488056970 con 0x7f64900b2390 2023-10-10T19:05:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:27.845 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:07.629134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5357,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5357":{"gid":5357,"name":"b","rank":3,"incarnation":114,"state":"up:active","state_seq":22,"addr":"172.21.15.191:6836/1945747421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1945747421},{"type":"v1","addr":"172.21.15.191:6836","nonce":1945747421}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:27.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f647c066f30 msgr2=0x7f647c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:27.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f647c066f30 0x7f647c0693e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f64800076c0 tx=0x7f648000b040 comp rx=0 tx=0).stop 2023-10-10T19:05:27.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64900b2390 msgr2=0x7f649015cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:27.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64900b2390 0x7f649015cc50 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f649006fd90 tx=0x7f6488017000 comp rx=0 tx=0).stop 2023-10-10T19:05:27.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 shutdown_connections 2023-10-10T19:05:27.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f647c066f30 0x7f647c0693e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f64900b2390 0x7f649015cc50 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6490068bd0 0x7f64901692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 --2- 172.21.15.175:0/3624097340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64900b1fc0 0x7f6490165b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:27.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 >> 172.21.15.175:0/3624097340 conn(0x7f6490063230 msgr2=0x7f6490151f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:27.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 shutdown_connections 2023-10-10T19:05:27.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:27.847+0000 7f6495b12700 1 -- 172.21.15.175:0/3624097340 wait complete. 2023-10-10T19:05:27.850 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 153 2023-10-10T19:05:27.859 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:05:29.860 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:29.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.988+0000 7f38505cc700 1 Processor -- start 2023-10-10T19:05:29.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.988+0000 7f38505cc700 1 -- start start 2023-10-10T19:05:29.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f38505cc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f384814ffc0 0x7f3848159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:29.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f38505cc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38480681e0 0x7f38480c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:29.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f38505cc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3848156800 con 0x7f384814fbf0 2023-10-10T19:05:29.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f38505cc700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38480c59e0 con 0x7f384814ffc0 2023-10-10T19:05:29.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f38505cc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384815a2b0 con 0x7f38480681e0 2023-10-10T19:05:29.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384e368700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f384814fbf0 0x7f3848063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48864/0 (socket says 172.21.15.175:48864) 2023-10-10T19:05:29.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384e368700 1 -- 172.21.15.175:0/3799848244 learned_addr learned my addr 172.21.15.175:0/3799848244 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:29.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384db67700 1 --2- 172.21.15.175:0/3799848244 >> v2:172.21.15.191:3300/0 conn(0x7f384814ffc0 0x7f3848159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:29.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384eb69700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38480681e0 0x7f38480c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:29.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472859119 0 0) 0x7f3848156800 con 0x7f384814fbf0 2023-10-10T19:05:29.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3834003570 con 0x7f384814fbf0 2023-10-10T19:05:29.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3143117659 0 0) 0x7f3834003570 con 0x7f384814fbf0 2023-10-10T19:05:29.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38480681e0 msgr2=0x7f38480c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:29.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38480681e0 0x7f38480c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:29.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 >> v2:172.21.15.191:3300/0 conn(0x7f384814ffc0 msgr2=0x7f3848159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:29.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 --2- 172.21.15.175:0/3799848244 >> v2:172.21.15.191:3300/0 conn(0x7f384814ffc0 0x7f3848159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:29.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3848063e30 con 0x7f384814fbf0 2023-10-10T19:05:29.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f38440015f0 con 0x7f384814fbf0 2023-10-10T19:05:29.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 >> v1:172.21.15.175:6789/0 conn(0x7f384814fbf0 legacy=0x7f3848063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38480681e0 0x7f3834003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f3834004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3834004d50 0x7f38340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38480c59e0 con 0x7f3834003fe0 2023-10-10T19:05:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383400ba60 con 0x7f3834004d50 2023-10-10T19:05:30.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383400bbd0 con 0x7f38480681e0 2023-10-10T19:05:30.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.989+0000 7f384db67700 1 --2- 172.21.15.175:0/3799848244 >> v2:172.21.15.191:3300/0 conn(0x7f384814ffc0 0x7f3848159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:05:30.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384db67700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f3834004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384e368700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38480681e0 0x7f3834003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384eb69700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3834004d50 0x7f38340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384db67700 1 -- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38480681e0 msgr2=0x7f3834003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:30.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384db67700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38480681e0 0x7f3834003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384db67700 1 -- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3834004d50 msgr2=0x7f38340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:30.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384db67700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3834004d50 0x7f38340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384db67700 1 -- 172.21.15.175:0/3799848244 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3834003570 con 0x7f3834003fe0 2023-10-10T19:05:30.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384db67700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f3834004810 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3838009db0 tx=0x7f38380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03a6ec3b794c08e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:30.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838009740 con 0x7f3834003fe0 2023-10-10T19:05:30.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383800e8c0 con 0x7f3834003fe0 2023-10-10T19:05:30.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.990+0000 7f384d366700 1 -- 172.21.15.175:0/3799848244 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383800ec40 con 0x7f3834003fe0 2023-10-10T19:05:30.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.991+0000 7f38505cc700 1 -- 172.21.15.175:0/3799848244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 msgr2=0x7f3834004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.991+0000 7f38505cc700 1 --2- 172.21.15.175:0/3799848244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f3834004810 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3838009db0 tx=0x7f38380091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.991+0000 7f384eb69700 1 -- 172.21.15.175:0/3799848244 reap_dead start 2023-10-10T19:05:30.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.991+0000 7f38505cc700 1 -- 172.21.15.175:0/3799848244 shutdown_connections 2023-10-10T19:05:30.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.991+0000 7f38505cc700 1 -- 172.21.15.175:0/3799848244 >> 172.21.15.175:0/3799848244 conn(0x7f38480cc760 msgr2=0x7f38480ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:30.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.991+0000 7f38505cc700 1 -- 172.21.15.175:0/3799848244 shutdown_connections 2023-10-10T19:05:30.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.991+0000 7f38505cc700 1 -- 172.21.15.175:0/3799848244 wait complete. 2023-10-10T19:05:30.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.992+0000 7f38505cc700 1 Processor -- start 2023-10-10T19:05:30.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.992+0000 7f38505cc700 1 -- start start 2023-10-10T19:05:30.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.992+0000 7f38505cc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f38481531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.992+0000 7f38505cc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38480c2a00 0x7f38480c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.992+0000 7f38505cc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3848167bc0 0x7f3848168030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.992+0000 7f38505cc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3848167730 con 0x7f3834003fe0 2023-10-10T19:05:30.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.992+0000 7f38505cc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384814eb00 con 0x7f38480c2a00 2023-10-10T19:05:30.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.992+0000 7f38505cc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3848168570 con 0x7f3848167bc0 2023-10-10T19:05:30.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f38481531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f38481531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39962/0 (socket says 172.21.15.175:39962) 2023-10-10T19:05:30.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 -- 172.21.15.175:0/3349119240 learned_addr learned my addr 172.21.15.175:0/3349119240 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:30.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384db67700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38480c2a00 0x7f38480c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384eb69700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3848167bc0 0x7f3848168030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 -- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3848167bc0 msgr2=0x7f3848168030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3848167bc0 0x7f3848168030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 -- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38480c2a00 msgr2=0x7f38480c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38480c2a00 0x7f38480c2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 -- 172.21.15.175:0/3349119240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3848166e40 con 0x7f3834003fe0 2023-10-10T19:05:30.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384db67700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38480c2a00 0x7f38480c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:05:30.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f384e368700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f38481531b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f384400fb80 tx=0x7f3844009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:30.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f383e7fc700 1 -- 172.21.15.175:0/3349119240 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38440072e0 con 0x7f3834003fe0 2023-10-10T19:05:30.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f383e7fc700 1 -- 172.21.15.175:0/3349119240 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3844007460 con 0x7f3834003fe0 2023-10-10T19:05:30.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f383e7fc700 1 -- 172.21.15.175:0/3349119240 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38440057f0 con 0x7f3834003fe0 2023-10-10T19:05:30.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3848152ad0 con 0x7f3834003fe0 2023-10-10T19:05:30.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.993+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38481670e0 con 0x7f3834003fe0 2023-10-10T19:05:30.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.994+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3848067570 con 0x7f3834003fe0 2023-10-10T19:05:30.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.995+0000 7f383e7fc700 1 -- 172.21.15.175:0/3349119240 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3844005b60 con 0x7f3834003fe0 2023-10-10T19:05:30.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.995+0000 7f383e7fc700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f382c066fe0 0x7f382c069490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.995+0000 7f383e7fc700 1 -- 172.21.15.175:0/3349119240 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f3844084ca0 con 0x7f3834003fe0 2023-10-10T19:05:30.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.995+0000 7f384db67700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f382c066fe0 0x7f382c069490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.996+0000 7f384db67700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f382c066fe0 0x7f382c069490 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f38380091e0 tx=0x7f3838019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:30.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:29.999+0000 7f383e7fc700 1 -- 172.21.15.175:0/3349119240 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3844051aa0 con 0x7f3834003fe0 2023-10-10T19:05:30.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.226+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38480c3390 con 0x7f3834003fe0 2023-10-10T19:05:30.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.227+0000 7f383e7fc700 1 -- 172.21.15.175:0/3349119240 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5449 (secure 0 0 0) 0x7f38440550c0 con 0x7f3834003fe0 2023-10-10T19:05:30.227 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:30.228 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:28.622872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:30.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f382c066fe0 msgr2=0x7f382c069490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f382c066fe0 0x7f382c069490 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f38380091e0 tx=0x7f3838019000 comp rx=0 tx=0).stop 2023-10-10T19:05:30.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 msgr2=0x7f38481531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f38481531b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f384400fb80 tx=0x7f3844009040 comp rx=0 tx=0).stop 2023-10-10T19:05:30.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 shutdown_connections 2023-10-10T19:05:30.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f382c066fe0 0x7f382c069490 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3848167bc0 0x7f3848168030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3834003fe0 0x7f38481531b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 --2- 172.21.15.175:0/3349119240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38480c2a00 0x7f38480c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 >> 172.21.15.175:0/3349119240 conn(0x7f38480cc760 msgr2=0x7f38480c3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:30.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 shutdown_connections 2023-10-10T19:05:30.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.229+0000 7f38505cc700 1 -- 172.21.15.175:0/3349119240 wait complete. 2023-10-10T19:05:30.232 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 154 2023-10-10T19:05:30.241 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-10T19:05:30.241 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:05:30.241 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:30.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.355+0000 7fd0f4a25700 1 Processor -- start 2023-10-10T19:05:30.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.355+0000 7fd0f4a25700 1 -- start start 2023-10-10T19:05:30.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0f4a25700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f014fd90 0x7fd0f0159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0f4a25700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f0067fa0 con 0x7fd0f008a100 2023-10-10T19:05:30.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0f4a25700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f0156b70 con 0x7fd0f00cd040 2023-10-10T19:05:30.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0f4a25700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00c5600 con 0x7fd0f014fd90 2023-10-10T19:05:30.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0eeffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd0f008a100 0x7fd0f00c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48868/0 (socket says 172.21.15.175:48868) 2023-10-10T19:05:30.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0eeffd700 1 -- 172.21.15.175:0/1496340974 learned_addr learned my addr 172.21.15.175:0/1496340974 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0ef7fe700 1 --2- 172.21.15.175:0/1496340974 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f014fd90 0x7fd0f0159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0ee7fc700 1 -- 172.21.15.175:0/1496340974 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339831177 0 0) 0x7fd0f0067fa0 con 0x7fd0f008a100 2023-10-10T19:05:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0ee7fc700 1 -- 172.21.15.175:0/1496340974 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d8003570 con 0x7fd0f008a100 2023-10-10T19:05:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0ee7fc700 1 -- 172.21.15.175:0/1496340974 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951937450 0 0) 0x7fd0f0156b70 con 0x7fd0f00cd040 2023-10-10T19:05:30.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.356+0000 7fd0ee7fc700 1 -- 172.21.15.175:0/1496340974 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0f0067fa0 con 0x7fd0f00cd040 2023-10-10T19:05:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0ee7fc700 1 -- 172.21.15.175:0/1496340974 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2262084205 0 0) 0x7fd0d8003570 con 0x7fd0f008a100 2023-10-10T19:05:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0ef7fe700 1 -- 172.21.15.175:0/1496340974 >> v1:172.21.15.191:6789/0 conn(0x7fd0f00cd040 legacy=0x7fd0f00684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0ef7fe700 1 -- 172.21.15.175:0/1496340974 >> v1:172.21.15.175:6789/0 conn(0x7fd0f008a100 legacy=0x7fd0f00c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0ef7fe700 1 -- 172.21.15.175:0/1496340974 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f0150160 con 0x7fd0f014fd90 2023-10-10T19:05:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0ef7fe700 1 --2- 172.21.15.175:0/1496340974 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f014fd90 0x7fd0f0159c30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009e90 tx=0x7fd0e0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b1e6a8767648cf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0ee7fc700 1 -- 172.21.15.175:0/1496340974 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e000e070 con 0x7fd0f014fd90 2023-10-10T19:05:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0ee7fc700 1 -- 172.21.15.175:0/1496340974 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e000b870 con 0x7fd0f014fd90 2023-10-10T19:05:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0ee7fc700 1 -- 172.21.15.175:0/1496340974 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4001470 con 0x7fd0f014fd90 2023-10-10T19:05:30.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0f4a25700 1 -- 172.21.15.175:0/1496340974 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f014fd90 msgr2=0x7fd0f0159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0f4a25700 1 --2- 172.21.15.175:0/1496340974 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f014fd90 0x7fd0f0159c30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009e90 tx=0x7fd0e0009780 comp rx=0 tx=0).stop 2023-10-10T19:05:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0f4a25700 1 -- 172.21.15.175:0/1496340974 shutdown_connections 2023-10-10T19:05:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0f4a25700 1 --2- 172.21.15.175:0/1496340974 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f014fd90 0x7fd0f0159c30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0f4a25700 1 -- 172.21.15.175:0/1496340974 >> 172.21.15.175:0/1496340974 conn(0x7fd0f0063d00 msgr2=0x7fd0f00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:30.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0f4a25700 1 -- 172.21.15.175:0/1496340974 shutdown_connections 2023-10-10T19:05:30.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.357+0000 7fd0f4a25700 1 -- 172.21.15.175:0/1496340974 wait complete. 2023-10-10T19:05:30.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0f4a25700 1 Processor -- start 2023-10-10T19:05:30.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0f4a25700 1 -- start start 2023-10-10T19:05:30.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0f4a25700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0f008a100 0x7fd0f01524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0f4a25700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f00cd040 0x7fd0f00ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0f4a25700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0f014fd90 0x7fd0f00ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0f4a25700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f01511b0 con 0x7fd0f014fd90 2023-10-10T19:05:30.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0f4a25700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f0165b40 con 0x7fd0f008a100 2023-10-10T19:05:30.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0f4a25700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f0159490 con 0x7fd0f00cd040 2023-10-10T19:05:30.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0ef7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0f008a100 0x7fd0f01524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.359+0000 7fd0effff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0f014fd90 0x7fd0f00ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0ef7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0f008a100 0x7fd0f01524f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:51748/0 (socket says 172.21.15.175:51748) 2023-10-10T19:05:30.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0eeffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f00cd040 0x7fd0f00ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0ef7fe700 1 -- 172.21.15.175:0/942358784 learned_addr learned my addr 172.21.15.175:0/942358784 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:30.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0effff700 1 -- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f00cd040 msgr2=0x7fd0f00ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0effff700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f00cd040 0x7fd0f00ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0effff700 1 -- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0f008a100 msgr2=0x7fd0f01524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0effff700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0f008a100 0x7fd0f01524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0effff700 1 -- 172.21.15.175:0/942358784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e0009a20 con 0x7fd0f014fd90 2023-10-10T19:05:30.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0ef7fe700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0f008a100 0x7fd0f01524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:05:30.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0effff700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0f014fd90 0x7fd0f00ca960 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800c560 tx=0x7fd0e8007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:30.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0d77fe700 1 -- 172.21.15.175:0/942358784 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8011590 con 0x7fd0f014fd90 2023-10-10T19:05:30.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0d77fe700 1 -- 172.21.15.175:0/942358784 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e8011710 con 0x7fd0f014fd90 2023-10-10T19:05:30.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0d77fe700 1 -- 172.21.15.175:0/942358784 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8011ad0 con 0x7fd0f014fd90 2023-10-10T19:05:30.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f0165dc0 con 0x7fd0f014fd90 2023-10-10T19:05:30.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.360+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f0166380 con 0x7fd0f014fd90 2023-10-10T19:05:30.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.361+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f0067570 con 0x7fd0f014fd90 2023-10-10T19:05:30.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.361+0000 7fd0d77fe700 1 -- 172.21.15.175:0/942358784 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd0e8012420 con 0x7fd0f014fd90 2023-10-10T19:05:30.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.362+0000 7fd0d77fe700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0dc066f30 0x7fd0dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:30.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.362+0000 7fd0d77fe700 1 -- 172.21.15.175:0/942358784 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fd0e8084260 con 0x7fd0f014fd90 2023-10-10T19:05:30.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.362+0000 7fd0ef7fe700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0dc066f30 0x7fd0dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:30.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.362+0000 7fd0ef7fe700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0dc066f30 0x7fd0dc0693e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000b840 tx=0x7fd0e0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:30.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.366+0000 7fd0d77fe700 1 -- 172.21.15.175:0/942358784 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0e8051100 con 0x7fd0f014fd90 2023-10-10T19:05:30.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.594+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0f0155f80 con 0x7fd0f014fd90 2023-10-10T19:05:30.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.594+0000 7fd0d77fe700 1 -- 172.21.15.175:0/942358784 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5449 (secure 0 0 0) 0x7fd0e8054720 con 0x7fd0f014fd90 2023-10-10T19:05:30.595 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:30.595 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:28.622872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:30.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0dc066f30 msgr2=0x7fd0dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0dc066f30 0x7fd0dc0693e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000b840 tx=0x7fd0e0017000 comp rx=0 tx=0).stop 2023-10-10T19:05:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0f014fd90 msgr2=0x7fd0f00ca960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0f014fd90 0x7fd0f00ca960 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800c560 tx=0x7fd0e8007cf0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 shutdown_connections 2023-10-10T19:05:30.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd0dc066f30 0x7fd0dc0693e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0f014fd90 0x7fd0f00ca960 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0f008a100 0x7fd0f01524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 --2- 172.21.15.175:0/942358784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0f00cd040 0x7fd0f00ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:30.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 >> 172.21.15.175:0/942358784 conn(0x7fd0f0063d00 msgr2=0x7fd0f00c6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:30.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 shutdown_connections 2023-10-10T19:05:30.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:30.596+0000 7fd0f4a25700 1 -- 172.21.15.175:0/942358784 wait complete. 2023-10-10T19:05:30.599 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 154 2023-10-10T19:05:30.610 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5552": { "addr": "172.21.15.191:6839/1910071809", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1910071809, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1910071809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5552, "incarnation": 154, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:replay", "state_seq": 1 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:05:28.622872+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 5367, "mds_3": 5552, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:05:32.611 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:32.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.739+0000 7fdd91b99700 1 Processor -- start 2023-10-10T19:05:32.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.739+0000 7fdd91b99700 1 -- start start 2023-10-10T19:05:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd91b99700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdd8c155fb0 0x7fdd8c159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd91b99700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8c1574b0 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd91b99700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8c0afad0 con 0x7fdd8c0b2390 2023-10-10T19:05:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd91b99700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8c150770 con 0x7fdd8c155fb0 2023-10-10T19:05:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd8b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fdd8c0b1fc0 0x7fdd8c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48882/0 (socket says 172.21.15.175:48882) 2023-10-10T19:05:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd8b7fe700 1 -- 172.21.15.175:0/1439274431 learned_addr learned my addr 172.21.15.175:0/1439274431 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd8bfff700 1 --2- 172.21.15.175:0/1439274431 >> v2:172.21.15.175:3300/0 conn(0x7fdd8c155fb0 0x7fdd8c159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:32.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd8a7fc700 1 -- 172.21.15.175:0/1439274431 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839041450 0 0) 0x7fdd8c1574b0 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd8a7fc700 1 -- 172.21.15.175:0/1439274431 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd74003570 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd8bfff700 1 -- 172.21.15.175:0/1439274431 >> v1:172.21.15.191:6789/0 conn(0x7fdd8c0b2390 legacy=0x7fdd8c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd8bfff700 1 -- 172.21.15.175:0/1439274431 >> v1:172.21.15.175:6789/0 conn(0x7fdd8c0b1fc0 legacy=0x7fdd8c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.740+0000 7fdd8bfff700 1 -- 172.21.15.175:0/1439274431 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8c0af4c0 con 0x7fdd8c155fb0 2023-10-10T19:05:32.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd8bfff700 1 --2- 172.21.15.175:0/1439274431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c155fb0 0x7fdd8c159920 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009e60 tx=0x7fdd84009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4d2d9c8ca7999a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd8a7fc700 1 -- 172.21.15.175:0/1439274431 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8400e070 con 0x7fdd8c155fb0 2023-10-10T19:05:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd8a7fc700 1 -- 172.21.15.175:0/1439274431 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8400bbf0 con 0x7fdd8c155fb0 2023-10-10T19:05:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd8a7fc700 1 -- 172.21.15.175:0/1439274431 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd84012430 con 0x7fdd8c155fb0 2023-10-10T19:05:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd91b99700 1 -- 172.21.15.175:0/1439274431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c155fb0 msgr2=0x7fdd8c159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:32.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd91b99700 1 --2- 172.21.15.175:0/1439274431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c155fb0 0x7fdd8c159920 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009e60 tx=0x7fdd84009780 comp rx=0 tx=0).stop 2023-10-10T19:05:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd91b99700 1 -- 172.21.15.175:0/1439274431 shutdown_connections 2023-10-10T19:05:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd91b99700 1 --2- 172.21.15.175:0/1439274431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c155fb0 0x7fdd8c159920 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd91b99700 1 -- 172.21.15.175:0/1439274431 >> 172.21.15.175:0/1439274431 conn(0x7fdd8c063230 msgr2=0x7fdd8c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:32.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd91b99700 1 -- 172.21.15.175:0/1439274431 shutdown_connections 2023-10-10T19:05:32.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.741+0000 7fdd91b99700 1 -- 172.21.15.175:0/1439274431 wait complete. 2023-10-10T19:05:32.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.742+0000 7fdd91b99700 1 Processor -- start 2023-10-10T19:05:32.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd91b99700 1 -- start start 2023-10-10T19:05:32.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd91b99700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c0b1fc0 0x7fdd8c1696f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:32.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd91b99700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd8c0b2390 0x7fdd8c15c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:32.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd91b99700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd8c155fb0 0x7fdd8c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:32.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd91b99700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c0cd650 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd91b99700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c157cb0 con 0x7fdd8c0b2390 2023-10-10T19:05:32.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd91b99700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c15d430 con 0x7fdd8c155fb0 2023-10-10T19:05:32.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c0b1fc0 0x7fdd8c1696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:32.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c0b1fc0 0x7fdd8c1696f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39982/0 (socket says 172.21.15.175:39982) 2023-10-10T19:05:32.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd8b7fe700 1 -- 172.21.15.175:0/3927207055 learned_addr learned my addr 172.21.15.175:0/3927207055 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:32.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd8affd700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd8c0b2390 0x7fdd8c15c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:32.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.743+0000 7fdd8bfff700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd8c155fb0 0x7fdd8c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:32.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd8b7fe700 1 -- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd8c155fb0 msgr2=0x7fdd8c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:32.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd8b7fe700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd8c155fb0 0x7fdd8c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:32.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd8b7fe700 1 -- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd8c0b2390 msgr2=0x7fdd8c15c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:32.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd8b7fe700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd8c0b2390 0x7fdd8c15c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:32.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd8b7fe700 1 -- 172.21.15.175:0/3927207055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84009a20 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd8bfff700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd8c155fb0 0x7fdd8c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:05:32.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd8b7fe700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c0b1fc0 0x7fdd8c1696f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c00c7c0 tx=0x7fdd7c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:32.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd737fe700 1 -- 172.21.15.175:0/3927207055 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c005b30 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd737fe700 1 -- 172.21.15.175:0/3927207055 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c005cb0 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd737fe700 1 -- 172.21.15.175:0/3927207055 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c007320 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c159f10 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.744+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c0cd370 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.745+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c067570 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.745+0000 7fdd737fe700 1 -- 172.21.15.175:0/3927207055 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdd7c01b3b0 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.746+0000 7fdd737fe700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdd78066f30 0x7fdd780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:32.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.746+0000 7fdd737fe700 1 -- 172.21.15.175:0/3927207055 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fdd7c083f70 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.746+0000 7fdd8affd700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdd78066f30 0x7fdd780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:32.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.747+0000 7fdd8affd700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdd78066f30 0x7fdd780693e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c1515c0 tx=0x7fdd80007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:32.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.750+0000 7fdd737fe700 1 -- 172.21.15.175:0/3927207055 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd7c050d80 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.976+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd8c068da0 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.976+0000 7fdd737fe700 1 -- 172.21.15.175:0/3927207055 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5449 (secure 0 0 0) 0x7fdd7c017020 con 0x7fdd8c0b1fc0 2023-10-10T19:05:32.977 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:32.977 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:28.622872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:32.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.978+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdd78066f30 msgr2=0x7fdd780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:32.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.978+0000 7fdd91b99700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdd78066f30 0x7fdd780693e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c1515c0 tx=0x7fdd80007ec0 comp rx=0 tx=0).stop 2023-10-10T19:05:32.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c0b1fc0 msgr2=0x7fdd8c1696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:32.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c0b1fc0 0x7fdd8c1696f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c00c7c0 tx=0x7fdd7c007e00 comp rx=0 tx=0).stop 2023-10-10T19:05:32.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 shutdown_connections 2023-10-10T19:05:32.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdd78066f30 0x7fdd780693e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:32.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd8c155fb0 0x7fdd8c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:32.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd8c0b1fc0 0x7fdd8c1696f0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:32.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 --2- 172.21.15.175:0/3927207055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd8c0b2390 0x7fdd8c15c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:32.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 >> 172.21.15.175:0/3927207055 conn(0x7fdd8c063230 msgr2=0x7fdd8c1508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:32.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 shutdown_connections 2023-10-10T19:05:32.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:32.979+0000 7fdd91b99700 1 -- 172.21.15.175:0/3927207055 wait complete. 2023-10-10T19:05:32.982 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 154 2023-10-10T19:05:32.991 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:05:34.992 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:35.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.118+0000 7f6331571700 1 Processor -- start 2023-10-10T19:05:35.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.118+0000 7f6331571700 1 -- start start 2023-10-10T19:05:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.119+0000 7f6331571700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:35.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.119+0000 7f6331571700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c154aa0 con 0x7f632c0c12a0 2023-10-10T19:05:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.119+0000 7f6331571700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c063dc0 con 0x7f632c1573d0 2023-10-10T19:05:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.119+0000 7f6331571700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0bfdf0 con 0x7f632c1577a0 2023-10-10T19:05:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.119+0000 7f632a7fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f632c0c12a0 0x7f632c14d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55282/0 (socket says 172.21.15.175:55282) 2023-10-10T19:05:35.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.119+0000 7f632a7fc700 1 -- 172.21.15.175:0/2891358171 learned_addr learned my addr 172.21.15.175:0/2891358171 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:35.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.119+0000 7f632b7fe700 1 --2- 172.21.15.175:0/2891358171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:35.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f6329ffb700 1 -- 172.21.15.175:0/2891358171 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226446747 0 0) 0x7f632c154aa0 con 0x7f632c0c12a0 2023-10-10T19:05:35.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f6329ffb700 1 -- 172.21.15.175:0/2891358171 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6314003570 con 0x7f632c0c12a0 2023-10-10T19:05:35.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f6329ffb700 1 -- 172.21.15.175:0/2891358171 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054267170 0 0) 0x7f632c063dc0 con 0x7f632c1573d0 2023-10-10T19:05:35.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f6329ffb700 1 -- 172.21.15.175:0/2891358171 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c154aa0 con 0x7f632c1573d0 2023-10-10T19:05:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f632b7fe700 1 -- 172.21.15.175:0/2891358171 >> v1:172.21.15.191:6789/0 conn(0x7f632c1573d0 legacy=0x7f632c0bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f632b7fe700 1 -- 172.21.15.175:0/2891358171 >> v1:172.21.15.175:6789/0 conn(0x7f632c0c12a0 legacy=0x7f632c14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f632b7fe700 1 -- 172.21.15.175:0/2891358171 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c157bc0 con 0x7f632c1577a0 2023-10-10T19:05:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f632b7fe700 1 --2- 172.21.15.175:0/2891358171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c4f20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6324009910 tx=0x7f63240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f23b3f5c56b55e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:35.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f6329ffb700 1 -- 172.21.15.175:0/2891358171 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324011630 con 0x7f632c1577a0 2023-10-10T19:05:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f6329ffb700 1 -- 172.21.15.175:0/2891358171 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6324011790 con 0x7f632c1577a0 2023-10-10T19:05:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.120+0000 7f6329ffb700 1 -- 172.21.15.175:0/2891358171 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324011b30 con 0x7f632c1577a0 2023-10-10T19:05:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.121+0000 7f6331571700 1 -- 172.21.15.175:0/2891358171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 msgr2=0x7f632c0c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.121+0000 7f6331571700 1 --2- 172.21.15.175:0/2891358171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c4f20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6324009910 tx=0x7f63240091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:35.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.121+0000 7f6331571700 1 -- 172.21.15.175:0/2891358171 shutdown_connections 2023-10-10T19:05:35.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.121+0000 7f6331571700 1 --2- 172.21.15.175:0/2891358171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c4f20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:35.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.121+0000 7f6331571700 1 -- 172.21.15.175:0/2891358171 >> 172.21.15.175:0/2891358171 conn(0x7f632c06dc70 msgr2=0x7f632c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:35.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.121+0000 7f6331571700 1 -- 172.21.15.175:0/2891358171 shutdown_connections 2023-10-10T19:05:35.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.121+0000 7f6331571700 1 -- 172.21.15.175:0/2891358171 wait complete. 2023-10-10T19:05:35.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.122+0000 7f6331571700 1 Processor -- start 2023-10-10T19:05:35.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.122+0000 7f6331571700 1 -- start start 2023-10-10T19:05:35.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.122+0000 7f6331571700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c0c12a0 0x7f632c163240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:35.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6331571700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f632c1573d0 0x7f632c153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:35.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6331571700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:35.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6331571700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c152a60 con 0x7f632c1573d0 2023-10-10T19:05:35.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6331571700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c153880 con 0x7f632c0c12a0 2023-10-10T19:05:35.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6331571700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c152c90 con 0x7f632c1577a0 2023-10-10T19:05:35.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f632c1573d0 0x7f632c153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:35.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f632c1573d0 0x7f632c153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32836/0 (socket says 172.21.15.175:32836) 2023-10-10T19:05:35.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 -- 172.21.15.175:0/2293595076 learned_addr learned my addr 172.21.15.175:0/2293595076 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:35.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632b7fe700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:35.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632affd700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c0c12a0 0x7f632c163240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:35.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 -- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 msgr2=0x7f632c0c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:35.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:35.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 -- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c0c12a0 msgr2=0x7f632c163240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:35.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c0c12a0 0x7f632c163240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:35.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 -- 172.21.15.175:0/2293595076 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324009420 con 0x7f632c1573d0 2023-10-10T19:05:35.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f632a7fc700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f632c1573d0 0x7f632c153d10 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f632000c790 tx=0x7f6320007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:35.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6312ffd700 1 -- 172.21.15.175:0/2293595076 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320005b30 con 0x7f632c1573d0 2023-10-10T19:05:35.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6312ffd700 1 -- 172.21.15.175:0/2293595076 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6320005cd0 con 0x7f632c1573d0 2023-10-10T19:05:35.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c152f50 con 0x7f632c1573d0 2023-10-10T19:05:35.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6312ffd700 1 -- 172.21.15.175:0/2293595076 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320007320 con 0x7f632c1573d0 2023-10-10T19:05:35.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.123+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c153510 con 0x7f632c1573d0 2023-10-10T19:05:35.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.124+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c067570 con 0x7f632c1573d0 2023-10-10T19:05:35.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.125+0000 7f6312ffd700 1 -- 172.21.15.175:0/2293595076 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6320012400 con 0x7f632c1573d0 2023-10-10T19:05:35.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.125+0000 7f6312ffd700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6318066f30 0x7f63180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:35.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.125+0000 7f6312ffd700 1 -- 172.21.15.175:0/2293595076 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f6320083d80 con 0x7f632c1573d0 2023-10-10T19:05:35.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.125+0000 7f632affd700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6318066f30 0x7f63180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:35.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.126+0000 7f632affd700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6318066f30 0x7f63180693e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f631c004140 tx=0x7f631c007f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:35.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.129+0000 7f6312ffd700 1 -- 172.21.15.175:0/2293595076 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f632004b5a0 con 0x7f632c1573d0 2023-10-10T19:05:35.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.358+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f632c157e50 con 0x7f632c1573d0 2023-10-10T19:05:35.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.359+0000 7f6312ffd700 1 -- 172.21.15.175:0/2293595076 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5449 (secure 0 0 0) 0x7f6320017020 con 0x7f632c1573d0 2023-10-10T19:05:35.360 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:35.360 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:28.622872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:35.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6318066f30 msgr2=0x7f63180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:35.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6318066f30 0x7f63180693e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f631c004140 tx=0x7f631c007f90 comp rx=0 tx=0).stop 2023-10-10T19:05:35.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f632c1573d0 msgr2=0x7f632c153d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:35.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f632c1573d0 0x7f632c153d10 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f632000c790 tx=0x7f6320007e80 comp rx=0 tx=0).stop 2023-10-10T19:05:35.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 shutdown_connections 2023-10-10T19:05:35.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6318066f30 0x7f63180693e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:35.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f632c1577a0 0x7f632c0c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:35.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f632c0c12a0 0x7f632c163240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:35.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 --2- 172.21.15.175:0/2293595076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f632c1573d0 0x7f632c153d10 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:35.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 >> 172.21.15.175:0/2293595076 conn(0x7f632c06dc70 msgr2=0x7f632c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:35.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 shutdown_connections 2023-10-10T19:05:35.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:35.361+0000 7f6331571700 1 -- 172.21.15.175:0/2293595076 wait complete. 2023-10-10T19:05:35.364 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 154 2023-10-10T19:05:35.373 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:05:37.374 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:37.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.502+0000 7ffbc5a7d700 1 Processor -- start 2023-10-10T19:05:37.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.502+0000 7ffbc5a7d700 1 -- start start 2023-10-10T19:05:37.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.502+0000 7ffbc5a7d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ffbc00b2390 0x7ffbc0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:37.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.502+0000 7ffbc5a7d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00aff60 0x7ffbc00b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:37.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.502+0000 7ffbc5a7d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc01574b0 con 0x7ffbc00b1fc0 2023-10-10T19:05:37.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.502+0000 7ffbc5a7d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc0150770 con 0x7ffbc00b2390 2023-10-10T19:05:37.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.502+0000 7ffbc5a7d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc00afad0 con 0x7ffbc00aff60 2023-10-10T19:05:37.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbeffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ffbc00b2390 0x7ffbc0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:37.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbf7fe700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ffbc00b1fc0 0x7ffbc0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:58672/0 (socket says 172.21.15.175:58672) 2023-10-10T19:05:37.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00aff60 0x7ffbc00b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:37.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbf7fe700 1 -- 172.21.15.175:0/864804767 learned_addr learned my addr 172.21.15.175:0/864804767 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:37.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbe7fc700 1 -- 172.21.15.175:0/864804767 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340630255 0 0) 0x7ffbc01574b0 con 0x7ffbc00b1fc0 2023-10-10T19:05:37.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbe7fc700 1 -- 172.21.15.175:0/864804767 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba8003570 con 0x7ffbc00b1fc0 2023-10-10T19:05:37.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbeffd700 1 -- 172.21.15.175:0/864804767 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00aff60 msgr2=0x7ffbc00b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:37.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbeffd700 1 --2- 172.21.15.175:0/864804767 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00aff60 0x7ffbc00b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbeffd700 1 -- 172.21.15.175:0/864804767 >> v1:172.21.15.191:6789/0 conn(0x7ffbc00b1fc0 legacy=0x7ffbc0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:37.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbeffd700 1 -- 172.21.15.175:0/864804767 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc00af4c0 con 0x7ffbc00b2390 2023-10-10T19:05:37.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbffff700 1 --2- 172.21.15.175:0/864804767 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00aff60 0x7ffbc00b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:05:37.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.503+0000 7ffbbeffd700 1 --2- 172.21.15.175:0/864804767 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00b2390 0x7ffbc0155fb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40098e0 tx=0x7ffbb40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ce25c73eebfdde0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:37.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbbe7fc700 1 -- 172.21.15.175:0/864804767 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4011630 con 0x7ffbc00b2390 2023-10-10T19:05:37.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbbe7fc700 1 -- 172.21.15.175:0/864804767 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb4011790 con 0x7ffbc00b2390 2023-10-10T19:05:37.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbbe7fc700 1 -- 172.21.15.175:0/864804767 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4011b30 con 0x7ffbc00b2390 2023-10-10T19:05:37.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/864804767 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00b2390 msgr2=0x7ffbc0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:37.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/864804767 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00b2390 0x7ffbc0155fb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40098e0 tx=0x7ffbb40091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/864804767 shutdown_connections 2023-10-10T19:05:37.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/864804767 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00aff60 0x7ffbc00b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/864804767 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00b2390 0x7ffbc0155fb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/864804767 >> 172.21.15.175:0/864804767 conn(0x7ffbc0063230 msgr2=0x7ffbc0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:37.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/864804767 shutdown_connections 2023-10-10T19:05:37.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.504+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/864804767 wait complete. 2023-10-10T19:05:37.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.505+0000 7ffbc5a7d700 1 Processor -- start 2023-10-10T19:05:37.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.505+0000 7ffbc5a7d700 1 -- start start 2023-10-10T19:05:37.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbc5a7d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00aff60 0x7ffbc014e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:37.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbc5a7d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00b1fc0 0x7ffbc0158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:37.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbc5a7d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffbc00b2390 0x7ffbc0169220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:37.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbc5a7d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0156f70 con 0x7ffbc00b2390 2023-10-10T19:05:37.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbc5a7d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc01550f0 con 0x7ffbc00aff60 2023-10-10T19:05:37.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbc5a7d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0155e70 con 0x7ffbc00b1fc0 2023-10-10T19:05:37.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffbc00b2390 0x7ffbc0169220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:37.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffbc00b2390 0x7ffbc0169220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32852/0 (socket says 172.21.15.175:32852) 2023-10-10T19:05:37.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbffff700 1 -- 172.21.15.175:0/2728927323 learned_addr learned my addr 172.21.15.175:0/2728927323 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:37.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbf7fe700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00aff60 0x7ffbc014e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:37.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbeffd700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00b1fc0 0x7ffbc0158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:37.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbffff700 1 -- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00b1fc0 msgr2=0x7ffbc0158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:37.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbffff700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00b1fc0 0x7ffbc0158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbffff700 1 -- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00aff60 msgr2=0x7ffbc014e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:37.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.506+0000 7ffbbffff700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00aff60 0x7ffbc014e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffbbffff700 1 -- 172.21.15.175:0/2728927323 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb8009a20 con 0x7ffbc00b2390 2023-10-10T19:05:37.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffbbf7fe700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00aff60 0x7ffbc014e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:05:37.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffbbeffd700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00b1fc0 0x7ffbc0158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:05:37.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffbbffff700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffbc00b2390 0x7ffbc0169220 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009fc0 tx=0x7ffbb800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:37.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffba77fe700 1 -- 172.21.15.175:0/2728927323 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb800cd30 con 0x7ffbc00b2390 2023-10-10T19:05:37.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffba77fe700 1 -- 172.21.15.175:0/2728927323 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb800f040 con 0x7ffbc00b2390 2023-10-10T19:05:37.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb4009420 con 0x7ffbc00b2390 2023-10-10T19:05:37.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffba77fe700 1 -- 172.21.15.175:0/2728927323 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80146d0 con 0x7ffbc00b2390 2023-10-10T19:05:37.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.507+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc0155390 con 0x7ffbc00b2390 2023-10-10T19:05:37.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.508+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc0067570 con 0x7ffbc00b2390 2023-10-10T19:05:37.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.508+0000 7ffba77fe700 1 -- 172.21.15.175:0/2728927323 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffbb8003020 con 0x7ffbc00b2390 2023-10-10T19:05:37.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.509+0000 7ffba77fe700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffbac066f30 0x7ffbac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:37.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.509+0000 7ffba77fe700 1 -- 172.21.15.175:0/2728927323 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7ffbb8085430 con 0x7ffbc00b2390 2023-10-10T19:05:37.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.509+0000 7ffbbf7fe700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffbac066f30 0x7ffbac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:37.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.509+0000 7ffbbf7fe700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffbac066f30 0x7ffbac0693e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00058d0 tx=0x7ffbb0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:37.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.513+0000 7ffba77fe700 1 -- 172.21.15.175:0/2728927323 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffbb80522d0 con 0x7ffbc00b2390 2023-10-10T19:05:37.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.739+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbc0068da0 con 0x7ffbc00b2390 2023-10-10T19:05:37.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.739+0000 7ffba77fe700 1 -- 172.21.15.175:0/2728927323 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5451 (secure 0 0 0) 0x7ffbb80558f0 con 0x7ffbc00b2390 2023-10-10T19:05:37.740 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:37.740 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:36.698112+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:resolve","state_seq":30,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:37.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffbac066f30 msgr2=0x7ffbac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:37.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffbac066f30 0x7ffbac0693e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00058d0 tx=0x7ffbb0009040 comp rx=0 tx=0).stop 2023-10-10T19:05:37.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffbc00b2390 msgr2=0x7ffbc0169220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:37.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffbc00b2390 0x7ffbc0169220 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009fc0 tx=0x7ffbb800b040 comp rx=0 tx=0).stop 2023-10-10T19:05:37.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 shutdown_connections 2023-10-10T19:05:37.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffbac066f30 0x7ffbac0693e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffbc00b2390 0x7ffbc0169220 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc00aff60 0x7ffbc014e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 --2- 172.21.15.175:0/2728927323 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc00b1fc0 0x7ffbc0158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:37.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 >> 172.21.15.175:0/2728927323 conn(0x7ffbc0063230 msgr2=0x7ffbc0152790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:37.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 shutdown_connections 2023-10-10T19:05:37.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:37.741+0000 7ffbc5a7d700 1 -- 172.21.15.175:0/2728927323 wait complete. 2023-10-10T19:05:37.744 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 155 2023-10-10T19:05:37.753 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:05:39.755 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.881+0000 7fd537f1a700 1 Processor -- start 2023-10-10T19:05:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.881+0000 7fd537f1a700 1 -- start start 2023-10-10T19:05:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.881+0000 7fd537f1a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5300c6330 0x7fd53015a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.881+0000 7fd537f1a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd530157030 con 0x7fd5300c4580 2023-10-10T19:05:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.881+0000 7fd537f1a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd530156420 con 0x7fd5300c7190 2023-10-10T19:05:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd537f1a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5300c5000 con 0x7fd5300c6330 2023-10-10T19:05:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd535cb6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5300c6330 0x7fd53015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd5364b7700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd5300c4580 0x7fd53014fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55286/0 (socket says 172.21.15.175:55286) 2023-10-10T19:05:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd5364b7700 1 -- 172.21.15.175:0/380948982 learned_addr learned my addr 172.21.15.175:0/380948982 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd535cb6700 1 -- 172.21.15.175:0/380948982 >> v1:172.21.15.191:6789/0 conn(0x7fd5300c7190 legacy=0x7fd5300c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd535cb6700 1 -- 172.21.15.175:0/380948982 >> v1:172.21.15.175:6789/0 conn(0x7fd5300c4580 legacy=0x7fd53014fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd535cb6700 1 -- 172.21.15.175:0/380948982 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530156700 con 0x7fd5300c6330 2023-10-10T19:05:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd535cb6700 1 --2- 172.21.15.175:0/380948982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd53015a030 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0098e0 tx=0x7fd52c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f00433d187d5053 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd534cb4700 1 -- 172.21.15.175:0/380948982 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c009e80 con 0x7fd5300c6330 2023-10-10T19:05:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd534cb4700 1 -- 172.21.15.175:0/380948982 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c00b8b0 con 0x7fd5300c6330 2023-10-10T19:05:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.882+0000 7fd534cb4700 1 -- 172.21.15.175:0/380948982 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c00bc30 con 0x7fd5300c6330 2023-10-10T19:05:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.883+0000 7fd537f1a700 1 -- 172.21.15.175:0/380948982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 msgr2=0x7fd53015a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.883+0000 7fd537f1a700 1 --2- 172.21.15.175:0/380948982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd53015a030 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0098e0 tx=0x7fd52c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.883+0000 7fd537f1a700 1 -- 172.21.15.175:0/380948982 shutdown_connections 2023-10-10T19:05:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.883+0000 7fd537f1a700 1 --2- 172.21.15.175:0/380948982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd53015a030 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.883+0000 7fd537f1a700 1 -- 172.21.15.175:0/380948982 >> 172.21.15.175:0/380948982 conn(0x7fd53006ef20 msgr2=0x7fd530063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.883+0000 7fd537f1a700 1 -- 172.21.15.175:0/380948982 shutdown_connections 2023-10-10T19:05:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.883+0000 7fd537f1a700 1 -- 172.21.15.175:0/380948982 wait complete. 2023-10-10T19:05:39.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.884+0000 7fd537f1a700 1 Processor -- start 2023-10-10T19:05:39.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.884+0000 7fd537f1a700 1 -- start start 2023-10-10T19:05:39.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd537f1a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5300c4580 0x7fd5300c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:39.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd537f1a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd5300c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:39.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd537f1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5300c7190 0x7fd5301664b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:39.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd537f1a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300c11c0 con 0x7fd5300c6330 2023-10-10T19:05:39.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd537f1a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd530166020 con 0x7fd5300c4580 2023-10-10T19:05:39.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd537f1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd530068890 con 0x7fd5300c7190 2023-10-10T19:05:39.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd5300c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:39.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd5300c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32882/0 (socket says 172.21.15.175:32882) 2023-10-10T19:05:39.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 -- 172.21.15.175:0/1515061326 learned_addr learned my addr 172.21.15.175:0/1515061326 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:39.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd535cb6700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5300c4580 0x7fd5300c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:39.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5364b7700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5300c7190 0x7fd5301664b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:39.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 -- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5300c7190 msgr2=0x7fd5301664b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:39.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5300c7190 0x7fd5301664b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:39.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 -- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5300c4580 msgr2=0x7fd5300c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:39.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5300c4580 0x7fd5300c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:39.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 -- 172.21.15.175:0/1515061326 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c009420 con 0x7fd5300c6330 2023-10-10T19:05:39.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd535cb6700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5300c4580 0x7fd5300c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:05:39.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd5354b5700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd5300c8d30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd52000b5e0 tx=0x7fd520009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:39.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd525ffb700 1 -- 172.21.15.175:0/1515061326 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52000f510 con 0x7fd5300c6330 2023-10-10T19:05:39.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd525ffb700 1 -- 172.21.15.175:0/1515061326 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52000f670 con 0x7fd5300c6330 2023-10-10T19:05:39.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5300c00d0 con 0x7fd5300c6330 2023-10-10T19:05:39.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.885+0000 7fd525ffb700 1 -- 172.21.15.175:0/1515061326 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52000fa30 con 0x7fd5300c6330 2023-10-10T19:05:39.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.886+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5300c0670 con 0x7fd5300c6330 2023-10-10T19:05:39.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.886+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5301661b0 con 0x7fd5300c6330 2023-10-10T19:05:39.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.887+0000 7fd525ffb700 1 -- 172.21.15.175:0/1515061326 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd52001a400 con 0x7fd5300c6330 2023-10-10T19:05:39.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.887+0000 7fd525ffb700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd514066f30 0x7fd5140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:39.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.887+0000 7fd525ffb700 1 -- 172.21.15.175:0/1515061326 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fd5200843f0 con 0x7fd5300c6330 2023-10-10T19:05:39.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.888+0000 7fd535cb6700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd514066f30 0x7fd5140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:39.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.888+0000 7fd535cb6700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd514066f30 0x7fd5140693e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0097b0 tx=0x7fd52c016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:39.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:39.892+0000 7fd525ffb700 1 -- 172.21.15.175:0/1515061326 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd520051310 con 0x7fd5300c6330 2023-10-10T19:05:40.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.116+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd530067570 con 0x7fd5300c6330 2023-10-10T19:05:40.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.117+0000 7fd525ffb700 1 -- 172.21.15.175:0/1515061326 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5451 (secure 0 0 0) 0x7fd520054930 con 0x7fd5300c6330 2023-10-10T19:05:40.118 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:40.118 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:36.698112+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:resolve","state_seq":30,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:40.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd514066f30 msgr2=0x7fd5140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:40.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd514066f30 0x7fd5140693e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0097b0 tx=0x7fd52c016000 comp rx=0 tx=0).stop 2023-10-10T19:05:40.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 msgr2=0x7fd5300c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:40.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd5300c8d30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd52000b5e0 tx=0x7fd520009120 comp rx=0 tx=0).stop 2023-10-10T19:05:40.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 shutdown_connections 2023-10-10T19:05:40.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd514066f30 0x7fd5140693e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:40.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5300c7190 0x7fd5301664b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:40.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5300c4580 0x7fd5300c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:40.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 --2- 172.21.15.175:0/1515061326 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5300c6330 0x7fd5300c8d30 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:40.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 >> 172.21.15.175:0/1515061326 conn(0x7fd53006ef20 msgr2=0x7fd530068e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:40.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 shutdown_connections 2023-10-10T19:05:40.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:40.119+0000 7fd537f1a700 1 -- 172.21.15.175:0/1515061326 wait complete. 2023-10-10T19:05:40.122 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 155 2023-10-10T19:05:40.131 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:05:42.133 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:42.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.259+0000 7fc11f885700 1 Processor -- start 2023-10-10T19:05:42.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.260+0000 7fc11f885700 1 -- start start 2023-10-10T19:05:42.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.260+0000 7fc11f885700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1180c12a0 0x7fc1181577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:42.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11f885700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc118154aa0 con 0x7fc1180c4f20 2023-10-10T19:05:42.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11f885700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc118063dc0 con 0x7fc1181573d0 2023-10-10T19:05:42.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11f885700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1180bfdf0 con 0x7fc1180c12a0 2023-10-10T19:05:42.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11ce20700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1180c12a0 0x7fc1181577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:42.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11de22700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc1180c4f20 0x7fc11814d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55298/0 (socket says 172.21.15.175:55298) 2023-10-10T19:05:42.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11de22700 1 -- 172.21.15.175:0/708602354 learned_addr learned my addr 172.21.15.175:0/708602354 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:42.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11ce20700 1 -- 172.21.15.175:0/708602354 >> v1:172.21.15.191:6789/0 conn(0x7fc1181573d0 legacy=0x7fc1180bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:42.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11ce20700 1 -- 172.21.15.175:0/708602354 >> v1:172.21.15.175:6789/0 conn(0x7fc1180c4f20 legacy=0x7fc11814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:42.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11ce20700 1 -- 172.21.15.175:0/708602354 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180c5320 con 0x7fc1180c12a0 2023-10-10T19:05:42.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc117fff700 1 -- 172.21.15.175:0/708602354 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726201734 0 0) 0x7fc118154aa0 con 0x7fc1180c4f20 2023-10-10T19:05:42.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc11ce20700 1 --2- 172.21.15.175:0/708602354 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1180c12a0 0x7fc1181577a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0098e0 tx=0x7fc10c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1104bb55d73b7bf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:42.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc117fff700 1 -- 172.21.15.175:0/708602354 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c009e80 con 0x7fc1180c12a0 2023-10-10T19:05:42.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc117fff700 1 -- 172.21.15.175:0/708602354 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c00b860 con 0x7fc1180c12a0 2023-10-10T19:05:42.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.261+0000 7fc117fff700 1 -- 172.21.15.175:0/708602354 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c00bbe0 con 0x7fc1180c12a0 2023-10-10T19:05:42.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.262+0000 7fc11f885700 1 -- 172.21.15.175:0/708602354 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1180c12a0 msgr2=0x7fc1181577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:42.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.262+0000 7fc11f885700 1 --2- 172.21.15.175:0/708602354 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1180c12a0 0x7fc1181577a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0098e0 tx=0x7fc10c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:42.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.262+0000 7fc11f885700 1 -- 172.21.15.175:0/708602354 shutdown_connections 2023-10-10T19:05:42.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.262+0000 7fc11f885700 1 --2- 172.21.15.175:0/708602354 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1180c12a0 0x7fc1181577a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:42.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.262+0000 7fc11f885700 1 -- 172.21.15.175:0/708602354 >> 172.21.15.175:0/708602354 conn(0x7fc11806dc70 msgr2=0x7fc118063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:42.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.262+0000 7fc11f885700 1 -- 172.21.15.175:0/708602354 shutdown_connections 2023-10-10T19:05:42.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.262+0000 7fc11f885700 1 -- 172.21.15.175:0/708602354 wait complete. 2023-10-10T19:05:42.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.263+0000 7fc11f885700 1 Processor -- start 2023-10-10T19:05:42.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.263+0000 7fc11f885700 1 -- start start 2023-10-10T19:05:42.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11f885700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1180c12a0 0x7fc118156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:42.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11f885700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1180c4f20 0x7fc118147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:42.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11f885700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1181573d0 0x7fc118148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:42.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11f885700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc118164460 con 0x7fc1181573d0 2023-10-10T19:05:42.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11f885700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc118163f30 con 0x7fc1180c12a0 2023-10-10T19:05:42.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11f885700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc118149220 con 0x7fc1180c4f20 2023-10-10T19:05:42.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11de22700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1181573d0 0x7fc118148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:42.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11de22700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1181573d0 0x7fc118148350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:32896/0 (socket says 172.21.15.175:32896) 2023-10-10T19:05:42.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11de22700 1 -- 172.21.15.175:0/3105853572 learned_addr learned my addr 172.21.15.175:0/3105853572 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:42.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11d621700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1180c12a0 0x7fc118156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:42.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11de22700 1 -- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1180c4f20 msgr2=0x7fc118147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:42.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11ce20700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1180c4f20 0x7fc118147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:42.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11de22700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1180c4f20 0x7fc118147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:42.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11de22700 1 -- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1180c12a0 msgr2=0x7fc118156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:42.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11de22700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1180c12a0 0x7fc118156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:42.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11de22700 1 -- 172.21.15.175:0/3105853572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c009420 con 0x7fc1181573d0 2023-10-10T19:05:42.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.264+0000 7fc11d621700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1180c12a0 0x7fc118156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:05:42.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.265+0000 7fc11de22700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1181573d0 0x7fc118148350 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc11000cc90 tx=0x7fc110001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:42.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.265+0000 7fc1157fa700 1 -- 172.21.15.175:0/3105853572 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11000cea0 con 0x7fc1181573d0 2023-10-10T19:05:42.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.265+0000 7fc1157fa700 1 -- 172.21.15.175:0/3105853572 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11000ac00 con 0x7fc1181573d0 2023-10-10T19:05:42.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.265+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc118164180 con 0x7fc1181573d0 2023-10-10T19:05:42.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.265+0000 7fc1157fa700 1 -- 172.21.15.175:0/3105853572 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1100070f0 con 0x7fc1181573d0 2023-10-10T19:05:42.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.265+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1181497a0 con 0x7fc1181573d0 2023-10-10T19:05:42.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.266+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc118147c10 con 0x7fc1181573d0 2023-10-10T19:05:42.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.266+0000 7fc1157fa700 1 -- 172.21.15.175:0/3105853572 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc110007460 con 0x7fc1181573d0 2023-10-10T19:05:42.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.266+0000 7fc1157fa700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc104066f30 0x7fc1040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:42.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.267+0000 7fc1157fa700 1 -- 172.21.15.175:0/3105853572 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fc110084e00 con 0x7fc1181573d0 2023-10-10T19:05:42.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.267+0000 7fc11d621700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc104066f30 0x7fc1040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:42.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.267+0000 7fc11d621700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc104066f30 0x7fc1040693e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc1080017a0 tx=0x7fc108009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:42.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.271+0000 7fc1157fa700 1 -- 172.21.15.175:0/3105853572 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc110051d20 con 0x7fc1181573d0 2023-10-10T19:05:42.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.497+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc118067570 con 0x7fc1181573d0 2023-10-10T19:05:42.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.497+0000 7fc1157fa700 1 -- 172.21.15.175:0/3105853572 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5451 (secure 0 0 0) 0x7fc110055340 con 0x7fc1181573d0 2023-10-10T19:05:42.498 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:42.498 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:36.698112+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:resolve","state_seq":30,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:42.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc104066f30 msgr2=0x7fc1040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:42.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc104066f30 0x7fc1040693e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc1080017a0 tx=0x7fc108009040 comp rx=0 tx=0).stop 2023-10-10T19:05:42.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1181573d0 msgr2=0x7fc118148350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:42.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1181573d0 0x7fc118148350 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc11000cc90 tx=0x7fc110001a80 comp rx=0 tx=0).stop 2023-10-10T19:05:42.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 shutdown_connections 2023-10-10T19:05:42.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc104066f30 0x7fc1040693e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:42.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1181573d0 0x7fc118148350 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:42.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1180c12a0 0x7fc118156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:42.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 --2- 172.21.15.175:0/3105853572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1180c4f20 0x7fc118147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:42.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 >> 172.21.15.175:0/3105853572 conn(0x7fc11806dc70 msgr2=0x7fc11814d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:42.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 shutdown_connections 2023-10-10T19:05:42.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:42.499+0000 7fc11f885700 1 -- 172.21.15.175:0/3105853572 wait complete. 2023-10-10T19:05:42.502 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 155 2023-10-10T19:05:42.512 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 155, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 155, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5367": { "addr": "172.21.15.175:6836/2375829435", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2375829435, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2375829435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5367, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 17 }, "gid_5552": { "addr": "172.21.15.191:6839/1910071809", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1910071809, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1910071809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5552, "incarnation": 154, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:resolve", "state_seq": 30 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:05:36.698112+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 5367, "mds_3": 5552, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:05:44.513 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:44.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.639+0000 7f3b3c6a3700 1 Processor -- start 2023-10-10T19:05:44.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.639+0000 7f3b3c6a3700 1 -- start start 2023-10-10T19:05:44.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.640+0000 7f3b3c6a3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3b34150480 0x7f3b34161940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:44.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.640+0000 7f3b3c6a3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b340677f0 con 0x7f3b3406d6e0 2023-10-10T19:05:44.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.640+0000 7f3b3c6a3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b34159590 con 0x7f3b341500b0 2023-10-10T19:05:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.640+0000 7f3b3c6a3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b340c4250 con 0x7f3b34150480 2023-10-10T19:05:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.640+0000 7f3b3ac40700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3b34150480 0x7f3b34161940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.640+0000 7f3b39c3e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f3b3406d6e0 0x7f3b3406dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39092/0 (socket says 172.21.15.175:39092) 2023-10-10T19:05:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.640+0000 7f3b39c3e700 1 -- 172.21.15.175:0/3029145680 learned_addr learned my addr 172.21.15.175:0/3029145680 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:44.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3943d700 1 -- 172.21.15.175:0/3029145680 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258762923 0 0) 0x7f3b340677f0 con 0x7f3b3406d6e0 2023-10-10T19:05:44.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3943d700 1 -- 172.21.15.175:0/3029145680 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b20003570 con 0x7f3b3406d6e0 2023-10-10T19:05:44.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3ac40700 1 -- 172.21.15.175:0/3029145680 >> v1:172.21.15.191:6789/0 conn(0x7f3b341500b0 legacy=0x7f3b340c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:44.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3ac40700 1 -- 172.21.15.175:0/3029145680 >> v1:172.21.15.175:6789/0 conn(0x7f3b3406d6e0 legacy=0x7f3b3406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:44.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3ac40700 1 -- 172.21.15.175:0/3029145680 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34063760 con 0x7f3b34150480 2023-10-10T19:05:44.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3ac40700 1 --2- 172.21.15.175:0/3029145680 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b34150480 0x7f3b34161940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0098e0 tx=0x7f3b2c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672a2d5067658d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:44.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3943d700 1 -- 172.21.15.175:0/3029145680 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c00b6d0 con 0x7f3b34150480 2023-10-10T19:05:44.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3943d700 1 -- 172.21.15.175:0/3029145680 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c00b830 con 0x7f3b34150480 2023-10-10T19:05:44.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.641+0000 7f3b3943d700 1 -- 172.21.15.175:0/3029145680 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c00bbb0 con 0x7f3b34150480 2023-10-10T19:05:44.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.642+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/3029145680 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b34150480 msgr2=0x7f3b34161940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:44.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.642+0000 7f3b3c6a3700 1 --2- 172.21.15.175:0/3029145680 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b34150480 0x7f3b34161940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0098e0 tx=0x7f3b2c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.642+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/3029145680 shutdown_connections 2023-10-10T19:05:44.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.642+0000 7f3b3c6a3700 1 --2- 172.21.15.175:0/3029145680 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b34150480 0x7f3b34161940 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.642+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/3029145680 >> 172.21.15.175:0/3029145680 conn(0x7f3b340c5650 msgr2=0x7f3b340c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:44.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.642+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/3029145680 shutdown_connections 2023-10-10T19:05:44.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.642+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/3029145680 wait complete. 2023-10-10T19:05:44.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.643+0000 7f3b3c6a3700 1 Processor -- start 2023-10-10T19:05:44.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.643+0000 7f3b3c6a3700 1 -- start start 2023-10-10T19:05:44.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3c6a3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b3406d6e0 0x7f3b3414cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:44.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3c6a3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b341500b0 0x7f3b3414edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:44.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3c6a3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b34150480 0x7f3b3414f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:44.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3c6a3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34157230 con 0x7f3b3406d6e0 2023-10-10T19:05:44.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3c6a3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340c0780 con 0x7f3b341500b0 2023-10-10T19:05:44.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3c6a3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340b94a0 con 0x7f3b34150480 2023-10-10T19:05:44.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b3406d6e0 0x7f3b3414cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:44.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b3406d6e0 0x7f3b3414cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42426/0 (socket says 172.21.15.175:42426) 2023-10-10T19:05:44.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 -- 172.21.15.175:0/955263342 learned_addr learned my addr 172.21.15.175:0/955263342 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:44.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 -- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b34150480 msgr2=0x7f3b3414f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:44.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b34150480 0x7f3b3414f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 -- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b341500b0 msgr2=0x7f3b3414edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:44.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b341500b0 0x7f3b3414edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 -- 172.21.15.175:0/955263342 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c009420 con 0x7f3b3406d6e0 2023-10-10T19:05:44.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3a43f700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b3406d6e0 0x7f3b3414cc30 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000cb40 tx=0x7f3b30001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:44.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b2a7fc700 1 -- 172.21.15.175:0/955263342 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000a7e0 con 0x7f3b3406d6e0 2023-10-10T19:05:44.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b2a7fc700 1 -- 172.21.15.175:0/955263342 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3000a940 con 0x7f3b3406d6e0 2023-10-10T19:05:44.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b2a7fc700 1 -- 172.21.15.175:0/955263342 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000ad00 con 0x7f3b3406d6e0 2023-10-10T19:05:44.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b340c7bd0 con 0x7f3b3406d6e0 2023-10-10T19:05:44.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.644+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b340bfe50 con 0x7f3b3406d6e0 2023-10-10T19:05:44.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.645+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b34067570 con 0x7f3b3406d6e0 2023-10-10T19:05:44.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.646+0000 7f3b2a7fc700 1 -- 172.21.15.175:0/955263342 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3b30003030 con 0x7f3b3406d6e0 2023-10-10T19:05:44.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.646+0000 7f3b2a7fc700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b18066f30 0x7f3b180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:44.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.646+0000 7f3b2a7fc700 1 -- 172.21.15.175:0/955263342 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f3b300849b0 con 0x7f3b3406d6e0 2023-10-10T19:05:44.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.646+0000 7f3b39c3e700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b18066f30 0x7f3b180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:44.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.647+0000 7f3b39c3e700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b18066f30 0x7f3b180693e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3b240058d0 tx=0x7f3b240079d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:44.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.650+0000 7f3b2a7fc700 1 -- 172.21.15.175:0/955263342 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3b30051850 con 0x7f3b3406d6e0 2023-10-10T19:05:44.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.876+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b3414e960 con 0x7f3b3406d6e0 2023-10-10T19:05:44.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.876+0000 7f3b2a7fc700 1 -- 172.21.15.175:0/955263342 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+5451 (secure 0 0 0) 0x7f3b3000ae60 con 0x7f3b3406d6e0 2023-10-10T19:05:44.877 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:44.877 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:44.627158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:44.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.878+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b18066f30 msgr2=0x7f3b180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:44.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b18066f30 0x7f3b180693e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3b240058d0 tx=0x7f3b240079d0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b3406d6e0 msgr2=0x7f3b3414cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:44.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b3406d6e0 0x7f3b3414cc30 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000cb40 tx=0x7f3b30001770 comp rx=0 tx=0).stop 2023-10-10T19:05:44.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 shutdown_connections 2023-10-10T19:05:44.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b18066f30 0x7f3b180693e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b34150480 0x7f3b3414f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b3406d6e0 0x7f3b3414cc30 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 --2- 172.21.15.175:0/955263342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b341500b0 0x7f3b3414edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:44.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 >> 172.21.15.175:0/955263342 conn(0x7f3b340c5650 msgr2=0x7f3b340c71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:44.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 shutdown_connections 2023-10-10T19:05:44.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:44.879+0000 7f3b3c6a3700 1 -- 172.21.15.175:0/955263342 wait complete. 2023-10-10T19:05:44.883 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 157 2023-10-10T19:05:44.891 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:05:46.892 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:47.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.020+0000 7fa9e5224700 1 Processor -- start 2023-10-10T19:05:47.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.020+0000 7fa9e5224700 1 -- start start 2023-10-10T19:05:47.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9e5224700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa9e00b2390 0x7fa9e0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:47.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9e5224700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00aff60 0x7fa9e00b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:47.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9e5224700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e01574b0 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9e5224700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e0150770 con 0x7fa9e00b2390 2023-10-10T19:05:47.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9e5224700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00afad0 con 0x7fa9e00aff60 2023-10-10T19:05:47.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9d659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa9e00b2390 0x7fa9e0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:47.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9ded9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa9e00b1fc0 0x7fa9e0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39094/0 (socket says 172.21.15.175:39094) 2023-10-10T19:05:47.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9ded9d700 1 -- 172.21.15.175:0/46106249 learned_addr learned my addr 172.21.15.175:0/46106249 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:47.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9df59e700 1 --2- 172.21.15.175:0/46106249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00aff60 0x7fa9e00b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:47.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9d659c700 1 -- 172.21.15.175:0/46106249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00aff60 msgr2=0x7fa9e00b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:47.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9d659c700 1 --2- 172.21.15.175:0/46106249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00aff60 0x7fa9e00b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9d659c700 1 -- 172.21.15.175:0/46106249 >> v1:172.21.15.175:6789/0 conn(0x7fa9e00b1fc0 legacy=0x7fa9e0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:47.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9d659c700 1 -- 172.21.15.175:0/46106249 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e00af4c0 con 0x7fa9e00b2390 2023-10-10T19:05:47.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9de59c700 1 -- 172.21.15.175:0/46106249 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531184407 0 0) 0x7fa9e01574b0 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9df59e700 1 --2- 172.21.15.175:0/46106249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00aff60 0x7fa9e00b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:05:47.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.021+0000 7fa9d659c700 1 --2- 172.21.15.175:0/46106249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b2390 0x7fa9e0155fb0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80098e0 tx=0x7fa9c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668a0a9ac755d195 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:47.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9de59c700 1 -- 172.21.15.175:0/46106249 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8009e80 con 0x7fa9e00b2390 2023-10-10T19:05:47.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9de59c700 1 -- 172.21.15.175:0/46106249 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c800b8f0 con 0x7fa9e00b2390 2023-10-10T19:05:47.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9de59c700 1 -- 172.21.15.175:0/46106249 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800bc70 con 0x7fa9e00b2390 2023-10-10T19:05:47.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9e5224700 1 -- 172.21.15.175:0/46106249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b2390 msgr2=0x7fa9e0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:47.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9e5224700 1 --2- 172.21.15.175:0/46106249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b2390 0x7fa9e0155fb0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80098e0 tx=0x7fa9c80091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9e5224700 1 -- 172.21.15.175:0/46106249 shutdown_connections 2023-10-10T19:05:47.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9e5224700 1 --2- 172.21.15.175:0/46106249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00aff60 0x7fa9e00b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9e5224700 1 --2- 172.21.15.175:0/46106249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b2390 0x7fa9e0155fb0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9e5224700 1 -- 172.21.15.175:0/46106249 >> 172.21.15.175:0/46106249 conn(0x7fa9e0063230 msgr2=0x7fa9e0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:47.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9e5224700 1 -- 172.21.15.175:0/46106249 shutdown_connections 2023-10-10T19:05:47.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.022+0000 7fa9e5224700 1 -- 172.21.15.175:0/46106249 wait complete. 2023-10-10T19:05:47.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.023+0000 7fa9e5224700 1 Processor -- start 2023-10-10T19:05:47.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.023+0000 7fa9e5224700 1 -- start start 2023-10-10T19:05:47.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9e5224700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9e00aff60 0x7fa9e0163ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:47.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9e5224700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b1fc0 0x7fa9e01599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:47.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9e5224700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00b2390 0x7fa9e0165f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:47.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9e5224700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00ccd60 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9e5224700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0158d20 con 0x7fa9e00aff60 2023-10-10T19:05:47.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9e5224700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e015aab0 con 0x7fa9e00b2390 2023-10-10T19:05:47.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9d659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b1fc0 0x7fa9e01599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:47.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9d659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b1fc0 0x7fa9e01599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42456/0 (socket says 172.21.15.175:42456) 2023-10-10T19:05:47.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9d659c700 1 -- 172.21.15.175:0/2670033869 learned_addr learned my addr 172.21.15.175:0/2670033869 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:47.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9df59e700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00b2390 0x7fa9e0165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:47.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.024+0000 7fa9ded9d700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9e00aff60 0x7fa9e0163ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:47.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d659c700 1 -- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00b2390 msgr2=0x7fa9e0165f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:47.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d659c700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00b2390 0x7fa9e0165f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d659c700 1 -- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9e00aff60 msgr2=0x7fa9e0163ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:47.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d659c700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9e00aff60 0x7fa9e0163ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d659c700 1 -- 172.21.15.175:0/2670033869 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8009420 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9df59e700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00b2390 0x7fa9e0165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:05:47.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d659c700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b1fc0 0x7fa9e01599f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009fa0 tx=0x7fa9c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:47.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d77fe700 1 -- 172.21.15.175:0/2670033869 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8011420 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d77fe700 1 -- 172.21.15.175:0/2670033869 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c8011580 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9d77fe700 1 -- 172.21.15.175:0/2670033869 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8011940 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e0159560 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.025+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e0166900 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.026+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e0063e90 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.026+0000 7fa9d77fe700 1 -- 172.21.15.175:0/2670033869 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa9c80253f0 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.027+0000 7fa9d77fe700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa9c0066f00 0x7fa9c00693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:47.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.027+0000 7fa9d77fe700 1 -- 172.21.15.175:0/2670033869 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fa9c80864a0 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.027+0000 7fa9ded9d700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa9c0066f00 0x7fa9c00693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:47.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.027+0000 7fa9ded9d700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa9c0066f00 0x7fa9c00693b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00011e0 tx=0x7fa9d0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:47.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.031+0000 7fa9d77fe700 1 -- 172.21.15.175:0/2670033869 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9c8053340 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.258+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9e0067570 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.258+0000 7fa9d77fe700 1 -- 172.21.15.175:0/2670033869 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+5451 (secure 0 0 0) 0x7fa9c8056960 con 0x7fa9e00b1fc0 2023-10-10T19:05:47.259 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:47.259 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:44.627158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:47.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa9c0066f00 msgr2=0x7fa9c00693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:47.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa9c0066f00 0x7fa9c00693b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00011e0 tx=0x7fa9d0009040 comp rx=0 tx=0).stop 2023-10-10T19:05:47.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b1fc0 msgr2=0x7fa9e01599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:47.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b1fc0 0x7fa9e01599f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009fa0 tx=0x7fa9c8017000 comp rx=0 tx=0).stop 2023-10-10T19:05:47.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 shutdown_connections 2023-10-10T19:05:47.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa9c0066f00 0x7fa9c00693b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9e00b2390 0x7fa9e0165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9e00aff60 0x7fa9e0163ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 --2- 172.21.15.175:0/2670033869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9e00b1fc0 0x7fa9e01599f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:47.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 >> 172.21.15.175:0/2670033869 conn(0x7fa9e0063230 msgr2=0x7fa9e01525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:47.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 shutdown_connections 2023-10-10T19:05:47.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:47.260+0000 7fa9e5224700 1 -- 172.21.15.175:0/2670033869 wait complete. 2023-10-10T19:05:47.264 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 157 2023-10-10T19:05:47.272 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:05:49.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:49.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.399+0000 7fc8d465b700 1 Processor -- start 2023-10-10T19:05:49.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.399+0000 7fc8d465b700 1 -- start start 2023-10-10T19:05:49.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.400+0000 7fc8d465b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc8cc0b1fc0 0x7fc8cc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:49.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.400+0000 7fc8d465b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc8cc0b2390 0x7fc8cc068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:49.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.400+0000 7fc8d465b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc1574b0 con 0x7fc8cc068bd0 2023-10-10T19:05:49.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.400+0000 7fc8d465b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc150770 con 0x7fc8cc0b1fc0 2023-10-10T19:05:49.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.400+0000 7fc8d465b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc0afad0 con 0x7fc8cc0b2390 2023-10-10T19:05:49.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d23f7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc8cc0b1fc0 0x7fc8cc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:49.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d23f7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc8cc0b1fc0 0x7fc8cc155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42468/0 (socket says 172.21.15.175:42468) 2023-10-10T19:05:49.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d23f7700 1 -- 172.21.15.175:0/479802563 learned_addr learned my addr 172.21.15.175:0/479802563 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:49.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d1bf6700 1 --2- 172.21.15.175:0/479802563 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc0b2390 0x7fc8cc068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:49.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d23f7700 1 -- 172.21.15.175:0/479802563 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc0b2390 msgr2=0x7fc8cc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:49.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d23f7700 1 --2- 172.21.15.175:0/479802563 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc0b2390 0x7fc8cc068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d23f7700 1 -- 172.21.15.175:0/479802563 >> v1:172.21.15.191:6789/0 conn(0x7fc8cc068bd0 legacy=0x7fc8cc0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:49.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d23f7700 1 -- 172.21.15.175:0/479802563 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0af4c0 con 0x7fc8cc0b1fc0 2023-10-10T19:05:49.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d23f7700 1 --2- 172.21.15.175:0/479802563 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0b1fc0 0x7fc8cc155fb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc0c0290 tx=0x7fc8bc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530149c0271ded3d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:49.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d13f5700 1 -- 172.21.15.175:0/479802563 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc00e070 con 0x7fc8cc0b1fc0 2023-10-10T19:05:49.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d13f5700 1 -- 172.21.15.175:0/479802563 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc00bae0 con 0x7fc8cc0b1fc0 2023-10-10T19:05:49.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.401+0000 7fc8d13f5700 1 -- 172.21.15.175:0/479802563 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc0092e0 con 0x7fc8cc0b1fc0 2023-10-10T19:05:49.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.402+0000 7fc8d465b700 1 -- 172.21.15.175:0/479802563 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0b1fc0 msgr2=0x7fc8cc155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:49.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.402+0000 7fc8d465b700 1 --2- 172.21.15.175:0/479802563 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0b1fc0 0x7fc8cc155fb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc0c0290 tx=0x7fc8bc009840 comp rx=0 tx=0).stop 2023-10-10T19:05:49.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.402+0000 7fc8d465b700 1 -- 172.21.15.175:0/479802563 shutdown_connections 2023-10-10T19:05:49.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.402+0000 7fc8d465b700 1 --2- 172.21.15.175:0/479802563 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc0b1fc0 0x7fc8cc155fb0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.402+0000 7fc8d465b700 1 --2- 172.21.15.175:0/479802563 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc0b2390 0x7fc8cc068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.402+0000 7fc8d465b700 1 -- 172.21.15.175:0/479802563 >> 172.21.15.175:0/479802563 conn(0x7fc8cc063230 msgr2=0x7fc8cc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:49.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.402+0000 7fc8d465b700 1 -- 172.21.15.175:0/479802563 shutdown_connections 2023-10-10T19:05:49.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.402+0000 7fc8d465b700 1 -- 172.21.15.175:0/479802563 wait complete. 2023-10-10T19:05:49.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.403+0000 7fc8d465b700 1 Processor -- start 2023-10-10T19:05:49.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.403+0000 7fc8d465b700 1 -- start start 2023-10-10T19:05:49.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d465b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc068bd0 0x7fc8cc0cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:49.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d465b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc0b1fc0 0x7fc8cc0cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:49.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d465b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0b2390 0x7fc8cc1550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:49.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d465b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0c41e0 con 0x7fc8cc068bd0 2023-10-10T19:05:49.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d465b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0cb570 con 0x7fc8cc0b1fc0 2023-10-10T19:05:49.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d465b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0cb850 con 0x7fc8cc0b2390 2023-10-10T19:05:49.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc068bd0 0x7fc8cc0cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:49.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc068bd0 0x7fc8cc0cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42482/0 (socket says 172.21.15.175:42482) 2023-10-10T19:05:49.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 -- 172.21.15.175:0/1346779818 learned_addr learned my addr 172.21.15.175:0/1346779818 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:49.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 -- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0b2390 msgr2=0x7fc8cc1550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:49.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d2bf8700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0b2390 0x7fc8cc1550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:49.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0b2390 0x7fc8cc1550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 -- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc0b1fc0 msgr2=0x7fc8cc0cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:49.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d1bf6700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc0b1fc0 0x7fc8cc0cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:49.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc0b1fc0 0x7fc8cc0cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 -- 172.21.15.175:0/1346779818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc009ae0 con 0x7fc8cc068bd0 2023-10-10T19:05:49.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d2bf8700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0b2390 0x7fc8cc1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:05:49.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d23f7700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc068bd0 0x7fc8cc0cbce0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009fd0 tx=0x7fc8bc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:49.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1346779818 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc008b90 con 0x7fc8cc068bd0 2023-10-10T19:05:49.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1346779818 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc008d10 con 0x7fc8cc068bd0 2023-10-10T19:05:49.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc156cf0 con 0x7fc8cc068bd0 2023-10-10T19:05:49.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1346779818 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc0266b0 con 0x7fc8cc068bd0 2023-10-10T19:05:49.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.404+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc15c130 con 0x7fc8cc068bd0 2023-10-10T19:05:49.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.405+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc067570 con 0x7fc8cc068bd0 2023-10-10T19:05:49.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.406+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1346779818 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc8bc003040 con 0x7fc8cc068bd0 2023-10-10T19:05:49.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.406+0000 7fc8ca7fc700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f60 0x7fc8b8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:49.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.406+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1346779818 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fc8bc0861e0 con 0x7fc8cc068bd0 2023-10-10T19:05:49.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.407+0000 7fc8d1bf6700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f60 0x7fc8b8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:49.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.407+0000 7fc8d1bf6700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f60 0x7fc8b8069410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc1513c0 tx=0x7fc8c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:49.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.411+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1346779818 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8bc053130 con 0x7fc8cc068bd0 2023-10-10T19:05:49.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.636+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8cc069060 con 0x7fc8cc068bd0 2023-10-10T19:05:49.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.636+0000 7fc8ca7fc700 1 -- 172.21.15.175:0/1346779818 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+5448 (secure 0 0 0) 0x7fc8bc056750 con 0x7fc8cc068bd0 2023-10-10T19:05:49.637 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:49.637 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:47.737948+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:49.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f60 msgr2=0x7fc8b8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:49.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f60 0x7fc8b8069410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc1513c0 tx=0x7fc8c0008040 comp rx=0 tx=0).stop 2023-10-10T19:05:49.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc068bd0 msgr2=0x7fc8cc0cbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:49.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc068bd0 0x7fc8cc0cbce0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009fd0 tx=0x7fc8bc018000 comp rx=0 tx=0).stop 2023-10-10T19:05:49.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 shutdown_connections 2023-10-10T19:05:49.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f60 0x7fc8b8069410 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0b2390 0x7fc8cc1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc068bd0 0x7fc8cc0cbce0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 --2- 172.21.15.175:0/1346779818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc0b1fc0 0x7fc8cc0cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:49.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.638+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 >> 172.21.15.175:0/1346779818 conn(0x7fc8cc063230 msgr2=0x7fc8cc14fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:49.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.639+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 shutdown_connections 2023-10-10T19:05:49.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:49.639+0000 7fc8d465b700 1 -- 172.21.15.175:0/1346779818 wait complete. 2023-10-10T19:05:49.642 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 158 2023-10-10T19:05:49.651 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:05:51.531 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:05:51.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:05:51.652 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:51.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.777+0000 7fb64497d700 1 Processor -- start 2023-10-10T19:05:51.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.777+0000 7fb64497d700 1 -- start start 2023-10-10T19:05:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.777+0000 7fb64497d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb6400b2390 0x7fb640155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.777+0000 7fb64497d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6401574b0 con 0x7fb6400b1fc0 2023-10-10T19:05:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.777+0000 7fb64497d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400afad0 con 0x7fb6400aff60 2023-10-10T19:05:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.777+0000 7fb64497d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb640150770 con 0x7fb6400b2390 2023-10-10T19:05:51.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63e59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb6400b1fc0 0x7fb640068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39102/0 (socket says 172.21.15.175:39102) 2023-10-10T19:05:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63e59c700 1 -- 172.21.15.175:0/4120763961 learned_addr learned my addr 172.21.15.175:0/4120763961 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb635d9b700 1 --2- 172.21.15.175:0/4120763961 >> v2:172.21.15.191:3300/0 conn(0x7fb6400b2390 0x7fb640155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16231679 0 0) 0x7fb6401574b0 con 0x7fb6400b1fc0 2023-10-10T19:05:51.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c003570 con 0x7fb6400b1fc0 2023-10-10T19:05:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714921535 0 0) 0x7fb62c003570 con 0x7fb6400b1fc0 2023-10-10T19:05:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 >> v2:172.21.15.191:3300/0 conn(0x7fb6400b2390 msgr2=0x7fb640155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 --2- 172.21.15.175:0/4120763961 >> v2:172.21.15.191:3300/0 conn(0x7fb6400b2390 0x7fb640155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 >> v1:172.21.15.191:6789/0 conn(0x7fb6400aff60 legacy=0x7fb640159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:51.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6400af4c0 con 0x7fb6400b1fc0 2023-10-10T19:05:51.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fb630001470 con 0x7fb6400b1fc0 2023-10-10T19:05:51.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 >> v1:172.21.15.175:6789/0 conn(0x7fb6400b1fc0 legacy=0x7fb640068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:51.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb62c003a60 0x7fb62c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:51.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c004810 0x7fb62c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:51.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb62c0051c0 0x7fb62c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:51.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb640150770 con 0x7fb62c0051c0 2023-10-10T19:05:51.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c00c0d0 con 0x7fb62c003a60 2023-10-10T19:05:51.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c00c290 con 0x7fb62c004810 2023-10-10T19:05:51.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63e59c700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb62c003a60 0x7fb62c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:51.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb63ed9d700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb62c0051c0 0x7fb62c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:51.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.778+0000 7fb635d9b700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c004810 0x7fb62c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:51.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63e59c700 1 -- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c004810 msgr2=0x7fb62c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:51.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63e59c700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c004810 0x7fb62c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:51.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63e59c700 1 -- 172.21.15.175:0/4120763961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb62c0051c0 msgr2=0x7fb62c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:51.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63e59c700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb62c0051c0 0x7fb62c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:51.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63e59c700 1 -- 172.21.15.175:0/4120763961 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c003570 con 0x7fb62c003a60 2023-10-10T19:05:51.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63ed9d700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb62c0051c0 0x7fb62c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:05:51.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63ed9d700 1 -- 172.21.15.175:0/4120763961 reap_dead start 2023-10-10T19:05:51.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63e59c700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb62c003a60 0x7fb62c003e70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb63000c390 tx=0x7fb6400af650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64b5554d6712becd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:51.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63000cc60 con 0x7fb62c003a60 2023-10-10T19:05:51.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63000cdc0 con 0x7fb62c003a60 2023-10-10T19:05:51.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb63dd9b700 1 -- 172.21.15.175:0/4120763961 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630005b70 con 0x7fb62c003a60 2023-10-10T19:05:51.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb64497d700 1 -- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb62c003a60 msgr2=0x7fb62c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:51.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb64497d700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb62c003a60 0x7fb62c003e70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb63000c390 tx=0x7fb6400af650 comp rx=0 tx=0).stop 2023-10-10T19:05:51.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb64497d700 1 -- 172.21.15.175:0/4120763961 shutdown_connections 2023-10-10T19:05:51.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.779+0000 7fb64497d700 1 --2- 172.21.15.175:0/4120763961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb62c003a60 0x7fb62c003e70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:51.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.780+0000 7fb64497d700 1 -- 172.21.15.175:0/4120763961 >> 172.21.15.175:0/4120763961 conn(0x7fb640063230 msgr2=0x7fb640063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:51.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.780+0000 7fb64497d700 1 -- 172.21.15.175:0/4120763961 shutdown_connections 2023-10-10T19:05:51.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.780+0000 7fb64497d700 1 -- 172.21.15.175:0/4120763961 wait complete. 2023-10-10T19:05:51.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb64497d700 1 Processor -- start 2023-10-10T19:05:51.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb64497d700 1 -- start start 2023-10-10T19:05:51.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb64497d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c003a60 0x7fb6400cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:51.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb64497d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6401568b0 0x7fb640165bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:51.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb64497d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb640166100 0x7fb640158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:51.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb64497d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb640154360 con 0x7fb640166100 2023-10-10T19:05:51.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb64497d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb640169d50 con 0x7fb6401568b0 2023-10-10T19:05:51.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb64497d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb640156e80 con 0x7fb62c003a60 2023-10-10T19:05:51.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.781+0000 7fb63ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb640166100 0x7fb640158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:51.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb640166100 0x7fb640158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42492/0 (socket says 172.21.15.175:42492) 2023-10-10T19:05:51.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63ed9d700 1 -- 172.21.15.175:0/754628299 learned_addr learned my addr 172.21.15.175:0/754628299 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:51.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb635d9b700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6401568b0 0x7fb640165bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:51.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63e59c700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c003a60 0x7fb6400cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:51.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63ed9d700 1 -- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c003a60 msgr2=0x7fb6400cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:51.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63ed9d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c003a60 0x7fb6400cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:51.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63ed9d700 1 -- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6401568b0 msgr2=0x7fb640165bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:51.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63ed9d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6401568b0 0x7fb640165bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:51.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63ed9d700 1 -- 172.21.15.175:0/754628299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb640169140 con 0x7fb640166100 2023-10-10T19:05:51.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb63ed9d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb640166100 0x7fb640158360 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb6400b2760 tx=0x7fb638009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:51.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb636ffd700 1 -- 172.21.15.175:0/754628299 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63800d8f0 con 0x7fb640166100 2023-10-10T19:05:51.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb636ffd700 1 -- 172.21.15.175:0/754628299 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63800da50 con 0x7fb640166100 2023-10-10T19:05:51.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb640169420 con 0x7fb640166100 2023-10-10T19:05:51.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb636ffd700 1 -- 172.21.15.175:0/754628299 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63800de10 con 0x7fb640166100 2023-10-10T19:05:51.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.782+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6401699e0 con 0x7fb640166100 2023-10-10T19:05:51.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.783+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb640169f00 con 0x7fb640166100 2023-10-10T19:05:51.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.783+0000 7fb636ffd700 1 -- 172.21.15.175:0/754628299 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb638003030 con 0x7fb640166100 2023-10-10T19:05:51.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.784+0000 7fb636ffd700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb620066f60 0x7fb620069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:51.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.784+0000 7fb636ffd700 1 -- 172.21.15.175:0/754628299 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fb638084a20 con 0x7fb640166100 2023-10-10T19:05:51.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.784+0000 7fb63e59c700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb620066f60 0x7fb620069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:51.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.784+0000 7fb63e59c700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb620066f60 0x7fb620069410 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb630007cd0 tx=0x7fb630015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:51.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:51.788+0000 7fb636ffd700 1 -- 172.21.15.175:0/754628299 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb640169f00 con 0x7fb640166100 2023-10-10T19:05:52.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.014+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb640169f00 con 0x7fb640166100 2023-10-10T19:05:52.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.015+0000 7fb636ffd700 1 -- 172.21.15.175:0/754628299 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+5448 (secure 0 0 0) 0x7fb640169f00 con 0x7fb640166100 2023-10-10T19:05:52.015 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:52.016 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:47.737948+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:52.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb620066f60 msgr2=0x7fb620069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:52.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb620066f60 0x7fb620069410 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb630007cd0 tx=0x7fb630015000 comp rx=0 tx=0).stop 2023-10-10T19:05:52.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb640166100 msgr2=0x7fb640158360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:52.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb640166100 0x7fb640158360 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb6400b2760 tx=0x7fb638009040 comp rx=0 tx=0).stop 2023-10-10T19:05:52.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 shutdown_connections 2023-10-10T19:05:52.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb620066f60 0x7fb620069410 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:52.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb640166100 0x7fb640158360 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:52.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb62c003a60 0x7fb6400cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:52.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 --2- 172.21.15.175:0/754628299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6401568b0 0x7fb640165bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:52.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 >> 172.21.15.175:0/754628299 conn(0x7fb640063230 msgr2=0x7fb64014fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:52.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 shutdown_connections 2023-10-10T19:05:52.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:52.017+0000 7fb64497d700 1 -- 172.21.15.175:0/754628299 wait complete. 2023-10-10T19:05:52.020 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 158 2023-10-10T19:05:52.029 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:05:54.030 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:54.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.155+0000 7fc520618700 1 Processor -- start 2023-10-10T19:05:54.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.155+0000 7fc520618700 1 -- start start 2023-10-10T19:05:54.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc520618700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc5180b2390 0x7fc518155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:54.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc520618700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5181574b0 con 0x7fc5180aff60 2023-10-10T19:05:54.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc520618700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5180afad0 con 0x7fc5180b1fc0 2023-10-10T19:05:54.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc520618700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc518150770 con 0x7fc5180b2390 2023-10-10T19:05:54.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51dbb3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc5180b2390 0x7fc518155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:54.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51ebb5700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc5180aff60 0x7fc518159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39106/0 (socket says 172.21.15.175:39106) 2023-10-10T19:05:54.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51ebb5700 1 -- 172.21.15.175:0/3759275065 learned_addr learned my addr 172.21.15.175:0/3759275065 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:54.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51dbb3700 1 -- 172.21.15.175:0/3759275065 >> v1:172.21.15.191:6789/0 conn(0x7fc5180b1fc0 legacy=0x7fc518068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:54.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51dbb3700 1 -- 172.21.15.175:0/3759275065 >> v1:172.21.15.175:6789/0 conn(0x7fc5180aff60 legacy=0x7fc518159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:54.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51dbb3700 1 -- 172.21.15.175:0/3759275065 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5180af4c0 con 0x7fc5180b2390 2023-10-10T19:05:54.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51d3b2700 1 -- 172.21.15.175:0/3759275065 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661520261 0 0) 0x7fc5181574b0 con 0x7fc5180aff60 2023-10-10T19:05:54.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51dbb3700 1 --2- 172.21.15.175:0/3759275065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b2390 0x7fc518155fb0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc5080098e0 tx=0x7fc5080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b652f31f266faad server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:54.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.156+0000 7fc51d3b2700 1 -- 172.21.15.175:0/3759275065 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508009e80 con 0x7fc5180b2390 2023-10-10T19:05:54.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc51d3b2700 1 -- 172.21.15.175:0/3759275065 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50800b8b0 con 0x7fc5180b2390 2023-10-10T19:05:54.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc51d3b2700 1 -- 172.21.15.175:0/3759275065 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50800bc30 con 0x7fc5180b2390 2023-10-10T19:05:54.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc520618700 1 -- 172.21.15.175:0/3759275065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b2390 msgr2=0x7fc518155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:54.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc520618700 1 --2- 172.21.15.175:0/3759275065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b2390 0x7fc518155fb0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc5080098e0 tx=0x7fc5080091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:54.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc520618700 1 -- 172.21.15.175:0/3759275065 shutdown_connections 2023-10-10T19:05:54.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc520618700 1 --2- 172.21.15.175:0/3759275065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b2390 0x7fc518155fb0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:54.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc520618700 1 -- 172.21.15.175:0/3759275065 >> 172.21.15.175:0/3759275065 conn(0x7fc518063230 msgr2=0x7fc518063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:54.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc520618700 1 -- 172.21.15.175:0/3759275065 shutdown_connections 2023-10-10T19:05:54.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.157+0000 7fc520618700 1 -- 172.21.15.175:0/3759275065 wait complete. 2023-10-10T19:05:54.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.158+0000 7fc520618700 1 Processor -- start 2023-10-10T19:05:54.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.158+0000 7fc520618700 1 -- start start 2023-10-10T19:05:54.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc520618700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5180aff60 0x7fc518158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:54.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc520618700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b1fc0 0x7fc518150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:54.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc520618700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5180b2390 0x7fc51814d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:54.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc520618700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518158970 con 0x7fc5180b1fc0 2023-10-10T19:05:54.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc520618700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518159340 con 0x7fc5180aff60 2023-10-10T19:05:54.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc520618700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc518158360 con 0x7fc5180b2390 2023-10-10T19:05:54.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b1fc0 0x7fc518150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:54.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b1fc0 0x7fc518150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42512/0 (socket says 172.21.15.175:42512) 2023-10-10T19:05:54.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 -- 172.21.15.175:0/949753669 learned_addr learned my addr 172.21.15.175:0/949753669 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:54.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51e3b4700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5180aff60 0x7fc518158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:54.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51ebb5700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5180b2390 0x7fc51814d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:54.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 -- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5180b2390 msgr2=0x7fc51814d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:54.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5180b2390 0x7fc51814d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:54.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 -- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5180aff60 msgr2=0x7fc518158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:54.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5180aff60 0x7fc518158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:54.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 -- 172.21.15.175:0/949753669 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc508009420 con 0x7fc5180b1fc0 2023-10-10T19:05:54.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51e3b4700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5180aff60 0x7fc518158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:05:54.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.159+0000 7fc51dbb3700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b1fc0 0x7fc518150fa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc5080030c0 tx=0x7fc508016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:54.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.160+0000 7fc50e7fc700 1 -- 172.21.15.175:0/949753669 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50801c680 con 0x7fc5180b1fc0 2023-10-10T19:05:54.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.160+0000 7fc50e7fc700 1 -- 172.21.15.175:0/949753669 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50801c800 con 0x7fc5180b1fc0 2023-10-10T19:05:54.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.160+0000 7fc50e7fc700 1 -- 172.21.15.175:0/949753669 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50801cbc0 con 0x7fc5180b1fc0 2023-10-10T19:05:54.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.160+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc518159fd0 con 0x7fc5180b1fc0 2023-10-10T19:05:54.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.160+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc518158600 con 0x7fc5180b1fc0 2023-10-10T19:05:54.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.160+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc518158ae0 con 0x7fc5180b1fc0 2023-10-10T19:05:54.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.161+0000 7fc50e7fc700 1 -- 172.21.15.175:0/949753669 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc50801a450 con 0x7fc5180b1fc0 2023-10-10T19:05:54.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.161+0000 7fc50e7fc700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4fc066f60 0x7fc4fc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:54.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.162+0000 7fc50e7fc700 1 -- 172.21.15.175:0/949753669 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fc5080865c0 con 0x7fc5180b1fc0 2023-10-10T19:05:54.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.162+0000 7fc51e3b4700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4fc066f60 0x7fc4fc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:54.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.162+0000 7fc51e3b4700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4fc066f60 0x7fc4fc069410 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc514007fd0 tx=0x7fc514009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:54.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.166+0000 7fc50e7fc700 1 -- 172.21.15.175:0/949753669 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc508053460 con 0x7fc5180b1fc0 2023-10-10T19:05:54.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.388+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc518067570 con 0x7fc5180b1fc0 2023-10-10T19:05:54.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.389+0000 7fc50e7fc700 1 -- 172.21.15.175:0/949753669 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5448 (secure 0 0 0) 0x7fc508056a80 con 0x7fc5180b1fc0 2023-10-10T19:05:54.390 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:54.391 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:53.752338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:54.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4fc066f60 msgr2=0x7fc4fc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:54.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4fc066f60 0x7fc4fc069410 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc514007fd0 tx=0x7fc514009040 comp rx=0 tx=0).stop 2023-10-10T19:05:54.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b1fc0 msgr2=0x7fc518150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:54.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b1fc0 0x7fc518150fa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc5080030c0 tx=0x7fc508016000 comp rx=0 tx=0).stop 2023-10-10T19:05:54.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 shutdown_connections 2023-10-10T19:05:54.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4fc066f60 0x7fc4fc069410 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:54.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5180b2390 0x7fc51814d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:54.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5180aff60 0x7fc518158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:54.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 --2- 172.21.15.175:0/949753669 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5180b1fc0 0x7fc518150fa0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:54.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 >> 172.21.15.175:0/949753669 conn(0x7fc518063230 msgr2=0x7fc51814fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:54.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 shutdown_connections 2023-10-10T19:05:54.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:54.391+0000 7fc520618700 1 -- 172.21.15.175:0/949753669 wait complete. 2023-10-10T19:05:54.395 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 159 2023-10-10T19:05:54.404 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=3, replacing gid=5357 2023-10-10T19:05:54.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2023-10-10T19:05:54.752 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:05:54.751+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T19:05:59.405 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-10T19:05:59.406 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-10T19:05:59.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-10T19:05:59.409 INFO:tasks.ceph.mds.b:Started 2023-10-10T19:05:59.409 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:05:59.495 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-10T19:05:59.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.545+0000 7f34b6162700 1 Processor -- start 2023-10-10T19:05:59.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.545+0000 7f34b6162700 1 -- start start 2023-10-10T19:05:59.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34b6162700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34b00c3210 0x7f34b0159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:59.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34b6162700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c35e0 0x7f34b00c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:59.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34b6162700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b00c5e40 con 0x7f34b00c9390 2023-10-10T19:05:59.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34b6162700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b00c4c50 con 0x7f34b00c3210 2023-10-10T19:05:59.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34b6162700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b00ca400 con 0x7f34b00c35e0 2023-10-10T19:05:59.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34af7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34b00c3210 0x7f34b0159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:59.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34affff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f34b00c9390 0x7f34b00c62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35286/0 (socket says 172.21.15.175:35286) 2023-10-10T19:05:59.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34affff700 1 -- 172.21.15.175:0/4135812637 learned_addr learned my addr 172.21.15.175:0/4135812637 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:05:59.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34aeffd700 1 --2- 172.21.15.175:0/4135812637 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c35e0 0x7f34b00c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:59.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34af7fe700 1 -- 172.21.15.175:0/4135812637 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c35e0 msgr2=0x7f34b00c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:05:59.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34af7fe700 1 --2- 172.21.15.175:0/4135812637 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c35e0 0x7f34b00c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34af7fe700 1 -- 172.21.15.175:0/4135812637 >> v1:172.21.15.175:6789/0 conn(0x7f34b00c9390 legacy=0x7f34b00c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:59.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34af7fe700 1 -- 172.21.15.175:0/4135812637 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b00c9880 con 0x7f34b00c3210 2023-10-10T19:05:59.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34ae7fc700 1 -- 172.21.15.175:0/4135812637 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952649591 0 0) 0x7f34b00c5e40 con 0x7f34b00c9390 2023-10-10T19:05:59.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34af7fe700 1 --2- 172.21.15.175:0/4135812637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0159fa0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f34a00098e0 tx=0x7f34a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cba1dbd6b1cb95 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:59.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.546+0000 7f34ae7fc700 1 -- 172.21.15.175:0/4135812637 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a000b640 con 0x7f34b00c3210 2023-10-10T19:05:59.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34ae7fc700 1 -- 172.21.15.175:0/4135812637 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a000b7a0 con 0x7f34b00c3210 2023-10-10T19:05:59.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34ae7fc700 1 -- 172.21.15.175:0/4135812637 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a000bb40 con 0x7f34b00c3210 2023-10-10T19:05:59.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34b6162700 1 -- 172.21.15.175:0/4135812637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 msgr2=0x7f34b0159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:59.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34b6162700 1 --2- 172.21.15.175:0/4135812637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0159fa0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f34a00098e0 tx=0x7f34a00091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34b6162700 1 -- 172.21.15.175:0/4135812637 shutdown_connections 2023-10-10T19:05:59.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34b6162700 1 --2- 172.21.15.175:0/4135812637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0159fa0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34b6162700 1 --2- 172.21.15.175:0/4135812637 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c35e0 0x7f34b00c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34b6162700 1 -- 172.21.15.175:0/4135812637 >> 172.21.15.175:0/4135812637 conn(0x7f34b006c110 msgr2=0x7f34b006c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:59.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34b6162700 1 -- 172.21.15.175:0/4135812637 shutdown_connections 2023-10-10T19:05:59.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.547+0000 7f34b6162700 1 -- 172.21.15.175:0/4135812637 wait complete. 2023-10-10T19:05:59.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.548+0000 7f34b6162700 1 Processor -- start 2023-10-10T19:05:59.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.548+0000 7f34b6162700 1 -- start start 2023-10-10T19:05:59.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34b6162700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:59.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34b6162700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b00c35e0 0x7f34b00c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:59.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34b6162700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c9390 0x7f34b0165e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:59.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34b6162700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b00c5840 con 0x7f34b00c3210 2023-10-10T19:05:59.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34b6162700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0169d70 con 0x7f34b00c35e0 2023-10-10T19:05:59.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34b6162700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b016a200 con 0x7f34b00c9390 2023-10-10T19:05:59.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34af7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:59.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34af7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47394/0 (socket says 172.21.15.175:47394) 2023-10-10T19:05:59.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34af7fe700 1 -- 172.21.15.175:0/2306037568 learned_addr learned my addr 172.21.15.175:0/2306037568 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:05:59.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34aeffd700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b00c35e0 0x7f34b00c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:59.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34affff700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c9390 0x7f34b0165e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:59.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34af7fe700 1 -- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c9390 msgr2=0x7f34b0165e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:59.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.549+0000 7f34af7fe700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c9390 0x7f34b0165e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34af7fe700 1 -- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b00c35e0 msgr2=0x7f34b00c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:59.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34af7fe700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b00c35e0 0x7f34b00c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34af7fe700 1 -- 172.21.15.175:0/2306037568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a0009420 con 0x7f34b00c3210 2023-10-10T19:05:59.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34aeffd700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b00c35e0 0x7f34b00c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:05:59.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34af7fe700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0153eb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f34a00093f0 tx=0x7f34a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:59.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34977fe700 1 -- 172.21.15.175:0/2306037568 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a000bb20 con 0x7f34b00c3210 2023-10-10T19:05:59.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34977fe700 1 -- 172.21.15.175:0/2306037568 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a0000c90 con 0x7f34b00c3210 2023-10-10T19:05:59.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b0153700 con 0x7f34b00c3210 2023-10-10T19:05:59.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34977fe700 1 -- 172.21.15.175:0/2306037568 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0011670 con 0x7f34b00c3210 2023-10-10T19:05:59.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.550+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b0169440 con 0x7f34b00c3210 2023-10-10T19:05:59.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.551+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b016a050 con 0x7f34b00c3210 2023-10-10T19:05:59.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.551+0000 7f34977fe700 1 -- 172.21.15.175:0/2306037568 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f34a0011a60 con 0x7f34b00c3210 2023-10-10T19:05:59.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.552+0000 7f34977fe700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f349c066f60 0x7f349c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:05:59.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.552+0000 7f34977fe700 1 -- 172.21.15.175:0/2306037568 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f34a0086080 con 0x7f34b00c3210 2023-10-10T19:05:59.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.552+0000 7f34aeffd700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f349c066f60 0x7f349c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:05:59.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.552+0000 7f34aeffd700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f349c066f60 0x7f349c069410 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f34b0153090 tx=0x7f34a40091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:05:59.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.556+0000 7f34977fe700 1 -- 172.21.15.175:0/2306037568 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34a0052f20 con 0x7f34b00c3210 2023-10-10T19:05:59.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.782+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b0067570 con 0x7f34b00c3210 2023-10-10T19:05:59.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.782+0000 7f34977fe700 1 -- 172.21.15.175:0/2306037568 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f34a0056540 con 0x7f34b00c3210 2023-10-10T19:05:59.784 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:05:59.784 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:05:53.752338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:05:59.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f349c066f60 msgr2=0x7f349c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:59.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f349c066f60 0x7f349c069410 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f34b0153090 tx=0x7f34a40091a0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 msgr2=0x7f34b0153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:05:59.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0153eb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f34a00093f0 tx=0x7f34a0017000 comp rx=0 tx=0).stop 2023-10-10T19:05:59.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 shutdown_connections 2023-10-10T19:05:59.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f349c066f60 0x7f349c069410 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b00c9390 0x7f34b0165e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34b00c3210 0x7f34b0153eb0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 --2- 172.21.15.175:0/2306037568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b00c35e0 0x7f34b00c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:05:59.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 >> 172.21.15.175:0/2306037568 conn(0x7f34b006c110 msgr2=0x7f34b00c62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:05:59.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 shutdown_connections 2023-10-10T19:05:59.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:05:59.785+0000 7f34b6162700 1 -- 172.21.15.175:0/2306037568 wait complete. 2023-10-10T19:05:59.789 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 160 2023-10-10T19:05:59.798 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-10T19:05:59.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-10-10T19:06:02.787 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 0% (24/12838) Updating files: 1% (129/12838) Updating files: 1% (138/12838) Updating files: 1% (232/12838) Updating files: 1% (246/12838) Updating files: 2% (257/12838) Updating files: 2% (369/12838) Updating files: 3% (386/12838) Updating files: 3% (428/12838) Updating files: 4% (514/12838) Updating files: 4% (528/12838) Updating files: 4% (615/12838) Updating files: 5% (642/12838) Updating files: 5% (769/12838) Updating files: 5% (770/12838) Updating files: 6% (771/12838) Updating files: 6% (804/12838) Updating files: 6% (826/12838) Updating files: 6% (827/12838) Updating files: 6% (849/12838) Updating files: 7% (899/12838) Updating files: 7% (939/12838) Updating files: 7% (964/12838) Updating files: 7% (968/12838) Updating files: 7% (986/12838) Updating files: 8% (1028/12838) Updating files: 8% (1124/12838) Updating files: 8% (1141/12838) Updating files: 8% (1148/12838) Updating files: 8% (1151/12838) Updating files: 9% (1156/12838) Updating files: 10% (1284/12838) Updating files: 10% (1310/12838) Updating files: 10% (1391/12838) Updating files: 11% (1413/12838) Updating files: 11% (1478/12838) Updating files: 11% (1518/12838) Updating files: 11% (1530/12838) Updating files: 12% (1541/12838) Updating files: 12% (1597/12838) Updating files: 12% (1610/12838) Updating files: 12% (1663/12838) Updating files: 13% (1669/12838) Updating files: 13% (1700/12838) Updating files: 13% (1781/12838) Updating files: 14% (1798/12838) Updating files: 14% (1883/12838) Updating files: 14% (1905/12838) Updating files: 14% (1924/12838) Updating files: 15% (1926/12838) Updating files: 15% (1991/12838) Updating files: 16% (2055/12838) Updating files: 16% (2076/12838) Updating files: 16% (2174/12838) Updating files: 17% (2183/12838) Updating files: 17% (2278/12838) Updating files: 18% (2311/12838) Updating files: 18% (2372/12838) Updating files: 19% (2440/12838) Updating files: 19% (2472/12838) Updating files: 19% (2533/12838) Updating files: 20% (2568/12838) Updating files: 20% (2585/12838) Updating files: 20% (2612/12838) Updating files: 20% (2653/12838) Updating files: 20% (2686/12838) Updating files: 21% (2696/12838) Updating files: 22% (2825/12838) Updating files: 22% (2860/12838) Updating files: 22% (2877/12838) Updating files: 22% (2906/12838) Updating files: 22% (2907/12838) Updating files: 23% (2953/12838) Updating files: 23% (2987/12838) Updating files: 23% (3009/12838) Updating files: 24% (3082/12838) Updating files: 24% (3175/12838) Updating files: 25% (3210/12838) Updating files: 25% (3243/12838) Updating files: 26% (3338/12838) Updating files: 26% (3340/12838) Updating files: 27% (3467/12838) Updating files: 27% (3499/12838) Updating files: 28% (3595/12838) Updating files: 28% (3614/12838) Updating files: 28% (3640/12838) Updating files: 29% (3724/12838) Updating files: 29% (3778/12838) Updating files: 29% (3780/12838) Updating files: 29% (3831/12838) Updating files: 30% (3852/12838) Updating files: 30% (3868/12838) Updating files: 30% (3946/12838) Updating files: 31% (3980/12838) Updating files: 31% (4072/12838) Updating files: 32% (4109/12838) Updating files: 32% (4112/12838) Updating files: 32% (4220/12838) Updating files: 32% (4224/12838) Updating files: 33% (4237/12838) Updating files: 33% (4311/12838) Updating files: 33% (4324/12838) Updating files: 34% (4365/12838) Updating files: 34% (4403/12838) Updating files: 35% (4494/12838) Updating files: 36% (4622/12838) Updating files: 36% (4638/12838) Updating files: 36% (4707/12838) Updating files: 36% (4736/12838) Updating files: 37% (4751/12838) Updating files: 37% (4758/12838) Updating files: 37% (4812/12838) Updating files: 37% (4845/12838) Updating files: 38% (4879/12838) Updating files: 38% (4914/12838) Updating files: 39% (5007/12838) Updating files: 39% (5083/12838) Updating files: 39% (5118/12838) Updating files: 40% (5136/12838) Updating files: 40% (5154/12838) Updating files: 40% (5248/12838) Updating files: 41% (5264/12838) Updating files: 41% (5310/12838) Updating files: 41% (5382/12838) Updating files: 42% (5392/12838) Updating files: 42% (5397/12838) Updating files: 43% (5521/12838) Updating files: 43% (5552/12838) Updating files: 43% (5567/12838) Updating files: 43% (5628/12838) Updating files: 44% (5649/12838) Updating files: 44% (5651/12838) Updating files: 44% (5761/12838) Updating files: 45% (5778/12838) Updating files: 45% (5792/12838) Updating files: 45% (5793/12838) Updating files: 45% (5813/12838) Updating files: 46% (5906/12838) Updating files: 46% (5930/12838) Updating files: 47% (6034/12838) Updating files: 47% (6121/12838) Updating files: 47% (6144/12838) Updating files: 48% (6163/12838) Updating files: 48% (6208/12838) Updating files: 48% (6269/12838) Updating files: 48% (6290/12838) Updating files: 49% (6291/12838) Updating files: 49% (6352/12838) Updating files: 49% (6361/12838) Updating files: 49% (6382/12838) Updating files: 50% (6419/12838) Updating files: 50% (6430/12838) Updating files: 50% (6461/12838) Updating files: 51% (6548/12838) Updating files: 51% (6568/12838) Updating files: 51% (6616/12838) Updating files: 52% (6676/12838) Updating files: 52% (6683/12838) Updating files: 52% (6738/12838) Updating files: 53% (6805/12838) Updating files: 53% (6878/12838) Updating files: 54% (6933/12838) Updating files: 54% (6954/12838) Updating files: 54% (6998/12838) Updating files: 54% (7014/12838) Updating files: 54% (7026/12838) Updating files: 55% (7061/12838) Updating files: 55% (7091/12838) Updating files: 55% (7112/12838) Updating files: 56% (7190/12838) Updating files: 56% (7239/12838) Updating files: 56% (7291/12838) Updating files: 57% (7318/12838) Updating files: 58% (7447/12838) Updating files: 58% (7455/12838) Updating files: 59% (7575/12838) Updating files: 59% (7585/12838) Updating files: 59% (7685/12838) Updating files: 60% (7703/12838) Updating files: 60% (7719/12838) Updating files: 60% (7722/12838) Updating files: 60% (7738/12838) Updating files: 60% (7761/12838) Updating files: 61% (7832/12838) Updating files: 61% (7851/12838) Updating files: 61% (7891/12838) Updating files: 61% (7910/12838) Updating files: 61% (7917/12838) Updating files: 62% (7960/12838) Updating files: 62% (7965/12838) Updating files: 62% (8024/12838) Updating files: 63% (8088/12838) Updating files: 63% (8158/12838) Updating files: 64% (8217/12838) Updating files: 64% (8227/12838) Updating files: 64% (8263/12838) Updating files: 64% (8328/12838) Updating files: 65% (8345/12838) Updating files: 65% (8389/12838) Updating files: 65% (8395/12838) Updating files: 65% (8429/12838) Updating files: 65% (8457/12838) Updating files: 66% (8474/12838) Updating files: 66% (8517/12838) Updating files: 66% (8570/12838) Updating files: 67% (8602/12838) Updating files: 67% (8727/12838) Updating files: 68% (8730/12838) Updating files: 68% (8776/12838) Updating files: 68% (8821/12838) Updating files: 68% (8825/12838) Updating files: 69% (8859/12838) Updating files: 69% (8949/12838) Updating files: 70% (8987/12838) Updating files: 70% (9104/12838) Updating files: 71% (9115/12838) Updating files: 71% (9231/12838) Updating files: 72% (9244/12838) Updating files: 72% (9262/12838) Updating files: 72% (9313/12838) Updating files: 72% (9322/12838) Updating files: 73% (9372/12838) Updating files: 73% (9401/12838) Updating files: 73% (9417/12838) Updating files: 73% (9467/12838) Updating files: 73% (9486/12838) Updating files: 74% (9501/12838) Updating files: 74% (9526/12838) Updating files: 74% (9595/12838) Updating files: 75% (9629/12838) Updating files: 75% (9707/12838) Updating files: 76% (9757/12838) Updating files: 76% (9821/12838) Updating files: 76% (9842/12838) Updating files: 77% (9886/12838) Updating files: 78% (10014/12838) Updating files: 78% (10034/12838) Updating files: 79% (10143/12838) Updating files: 79% (10156/12838) Updating files: 79% (10254/12838) Updating files: 80% (10271/12838) Updating files: 80% (10282/12838) Updating files: 81% (10399/12838) Updating files: 81% (10430/12838) Updating files: 81% (10466/12838) Updating files: 82% (10528/12838) Updating files: 82% (10555/12838) Updating files: 83% (10656/12838) Updating files: 83% (10695/12838) Updating files: 83% (10730/12838) Updating files: 83% (10740/12838) Updating files: 84% (10784/12838) Updating files: 84% (10903/12838) Updating files: 85% (10913/12838) Updating files: 85% (10919/12838) Updating files: 85% (10966/12838) Updating files: 85% (10985/12838) Updating files: 85% (11040/12838) Updating files: 86% (11041/12838) Updating files: 86% (11168/12838) Updating files: 87% (11170/12838) Updating files: 87% (11176/12838) Updating files: 87% (11275/12838) Updating files: 88% (11298/12838) Updating files: 88% (11418/12838) Updating files: 89% (11426/12838) Updating files: 89% (11435/12838) Updating files: 89% (11542/12838) Updating files: 90% (11555/12838) Updating files: 90% (11613/12838) Updating files: 91% (11683/12838) Updating files: 91% (11792/12838) Updating files: 92% (11811/12838) Updating files: 92% (11876/12838) Updating files: 93% (11940/12838) Updating files: 94% (12068/12838) Updating files: 95% (12197/12838) Updating files: 95% (12263/12838) Updating files: 96% (12325/12838) Updating files: 96% (12355/12838) Updating files: 96% (12360/12838) Updating files: 97% (12453/12838) Updating files: 97% (12487/12838) Updating files: 97% (12501/12838) Updating files: 98% (12582/12838) Updating files: 98% (12633/12838) Updating files: 99% (12710/12838) Updating files: 99% (12810/12838) Updating files: 100% (12838/12838) Updating files: 100% (12838/12838), done. 2023-10-10T19:06:03.709 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2023-10-10T19:06:03.711 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.1 2023-10-10T19:06:03.758 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:03.758 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.2 2023-10-10T19:06:03.759 INFO:tasks.workunit.client.0.smithi175.stdout:v0.2 2023-10-10T19:06:03.759 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.2 2023-10-10T19:06:03.759 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.2 2023-10-10T19:06:09.728 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2023-10-10T19:06:09.744 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 2b5f76f48e9 readme 2023-10-10T19:06:09.746 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.2 2023-10-10T19:06:09.829 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:09.830 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.3 2023-10-10T19:06:09.830 INFO:tasks.workunit.client.0.smithi175.stdout:v0.3 2023-10-10T19:06:09.831 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.3 2023-10-10T19:06:09.831 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.3 2023-10-10T19:06:19.140 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 39% (117/295) Updating files: 40% (118/295) Updating files: 41% (121/295) Updating files: 42% (124/295) Updating files: 43% (127/295) Updating files: 43% (129/295) Updating files: 44% (130/295) Updating files: 45% (133/295) Updating files: 46% (136/295) Updating files: 47% (139/295) Updating files: 48% (142/295) Updating files: 49% (145/295) Updating files: 50% (148/295) Updating files: 51% (151/295) Updating files: 52% (154/295) Updating files: 53% (157/295) Updating files: 54% (160/295) Updating files: 55% (163/295) Updating files: 56% (166/295) Updating files: 57% (169/295) Updating files: 58% (172/295) Updating files: 59% (175/295) Updating files: 60% (177/295) Updating files: 61% (180/295) Updating files: 62% (183/295) Updating files: 63% (186/295) Updating files: 64% (189/295) Updating files: 65% (192/295) Updating files: 66% (195/295) Updating files: 67% (198/295) Updating files: 68% (201/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 91% (270/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2023-10-10T19:06:19.164 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2023-10-10T19:06:19.167 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.3 2023-10-10T19:06:19.209 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:19.210 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.4 2023-10-10T19:06:19.210 INFO:tasks.workunit.client.0.smithi175.stdout:v0.4 2023-10-10T19:06:19.210 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.4 2023-10-10T19:06:19.211 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.4 2023-10-10T19:06:21.609 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:06:21.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:06:28.054 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 6% (16/247) Updating files: 7% (18/247) Updating files: 8% (20/247) Updating files: 9% (23/247) Updating files: 10% (25/247) Updating files: 11% (28/247) Updating files: 12% (30/247) Updating files: 13% (33/247) Updating files: 14% (35/247) Updating files: 15% (38/247) Updating files: 15% (39/247) Updating files: 16% (40/247) Updating files: 17% (42/247) Updating files: 18% (45/247) Updating files: 19% (47/247) Updating files: 20% (50/247) Updating files: 21% (52/247) Updating files: 22% (55/247) Updating files: 23% (57/247) Updating files: 24% (60/247) Updating files: 25% (62/247) Updating files: 26% (65/247) Updating files: 27% (67/247) Updating files: 28% (70/247) Updating files: 29% (72/247) Updating files: 30% (75/247) Updating files: 31% (77/247) Updating files: 32% (80/247) Updating files: 33% (82/247) Updating files: 34% (84/247) Updating files: 35% (87/247) Updating files: 36% (89/247) Updating files: 36% (90/247) Updating files: 37% (92/247) Updating files: 38% (94/247) Updating files: 39% (97/247) Updating files: 40% (99/247) Updating files: 41% (102/247) Updating files: 42% (104/247) Updating files: 43% (107/247) Updating files: 44% (109/247) Updating files: 45% (112/247) Updating files: 46% (114/247) Updating files: 47% (117/247) Updating files: 48% (119/247) Updating files: 49% (122/247) Updating files: 50% (124/247) Updating files: 51% (126/247) Updating files: 52% (129/247) Updating files: 53% (131/247) Updating files: 54% (134/247) Updating files: 55% (136/247) Updating files: 56% (139/247) Updating files: 57% (141/247) Updating files: 58% (144/247) Updating files: 59% (146/247) Updating files: 60% (149/247) Updating files: 61% (151/247) Updating files: 62% (154/247) Updating files: 63% (156/247) Updating files: 64% (159/247) Updating files: 64% (160/247) Updating files: 65% (161/247) Updating files: 66% (164/247) Updating files: 67% (166/247) Updating files: 68% (168/247) Updating files: 69% (171/247) Updating files: 70% (173/247) Updating files: 71% (176/247) Updating files: 72% (178/247) Updating files: 73% (181/247) Updating files: 74% (183/247) Updating files: 75% (186/247) Updating files: 76% (188/247) Updating files: 77% (191/247) Updating files: 78% (193/247) Updating files: 79% (196/247) Updating files: 80% (198/247) Updating files: 81% (201/247) Updating files: 82% (203/247) Updating files: 83% (206/247) Updating files: 84% (208/247) Updating files: 85% (210/247) Updating files: 86% (213/247) Updating files: 87% (215/247) Updating files: 88% (218/247) Updating files: 89% (220/247) Updating files: 90% (223/247) Updating files: 91% (225/247) Updating files: 92% (228/247) Updating files: 93% (230/247) Updating files: 94% (233/247) Updating files: 95% (235/247) Updating files: 96% (238/247) Updating files: 97% (240/247) Updating files: 98% (243/247) Updating files: 99% (245/247) Updating files: 100% (247/247) Updating files: 100% (247/247), done. 2023-10-10T19:06:28.078 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2023-10-10T19:06:28.080 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.4 2023-10-10T19:06:28.154 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:28.154 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.5 2023-10-10T19:06:28.155 INFO:tasks.workunit.client.0.smithi175.stdout:v0.5 2023-10-10T19:06:28.155 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.5 2023-10-10T19:06:28.155 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.5 2023-10-10T19:06:30.528 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 9% (23/245) Updating files: 10% (25/245) Updating files: 11% (27/245) Updating files: 12% (30/245) Updating files: 13% (32/245) Updating files: 14% (35/245) Updating files: 15% (37/245) Updating files: 16% (40/245) Updating files: 17% (42/245) Updating files: 18% (45/245) Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 40% (100/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-10T19:06:30.546 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2023-10-10T19:06:30.548 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.5 2023-10-10T19:06:30.625 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:30.626 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.6 2023-10-10T19:06:30.627 INFO:tasks.workunit.client.0.smithi175.stdout:v0.6 2023-10-10T19:06:30.627 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.6 2023-10-10T19:06:30.627 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.6 2023-10-10T19:06:37.842 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 1% (3/227) Updating files: 2% (5/227) Updating files: 3% (7/227) Updating files: 4% (10/227) Updating files: 5% (12/227) Updating files: 6% (14/227) Updating files: 7% (16/227) Updating files: 8% (19/227) Updating files: 9% (21/227) Updating files: 10% (23/227) Updating files: 11% (25/227) Updating files: 12% (28/227) Updating files: 13% (30/227) Updating files: 14% (32/227) Updating files: 15% (35/227) Updating files: 16% (37/227) Updating files: 17% (39/227) Updating files: 18% (41/227) Updating files: 19% (44/227) Updating files: 20% (46/227) Updating files: 21% (48/227) Updating files: 22% (50/227) Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 27% (63/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 37% (86/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 70% (160/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-10T19:06:37.857 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at a4c752df713 v0.6 2023-10-10T19:06:37.859 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.6 2023-10-10T19:06:38.073 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:38.074 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.7 2023-10-10T19:06:38.074 INFO:tasks.workunit.client.0.smithi175.stdout:v0.7 2023-10-10T19:06:38.075 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.7 2023-10-10T19:06:38.075 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.7 2023-10-10T19:06:40.300 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 8% (18/201) Updating files: 9% (19/201) Updating files: 10% (21/201) Updating files: 11% (23/201) Updating files: 12% (25/201) Updating files: 13% (27/201) Updating files: 14% (29/201) Updating files: 15% (31/201) Updating files: 16% (33/201) Updating files: 17% (35/201) Updating files: 18% (37/201) Updating files: 19% (39/201) Updating files: 20% (41/201) Updating files: 21% (43/201) Updating files: 22% (45/201) Updating files: 23% (47/201) Updating files: 24% (49/201) Updating files: 25% (51/201) Updating files: 26% (53/201) Updating files: 27% (55/201) Updating files: 28% (57/201) Updating files: 29% (59/201) Updating files: 30% (61/201) Updating files: 31% (63/201) Updating files: 32% (65/201) Updating files: 33% (67/201) Updating files: 34% (69/201) Updating files: 35% (71/201) Updating files: 36% (73/201) Updating files: 37% (75/201) Updating files: 38% (77/201) Updating files: 39% (79/201) Updating files: 40% (81/201) Updating files: 41% (83/201) Updating files: 42% (85/201) Updating files: 43% (87/201) Updating files: 44% (89/201) Updating files: 45% (91/201) Updating files: 46% (93/201) Updating files: 47% (95/201) Updating files: 48% (97/201) Updating files: 49% (99/201) Updating files: 50% (101/201) Updating files: 51% (103/201) Updating files: 52% (105/201) Updating files: 53% (107/201) Updating files: 54% (109/201) Updating files: 55% (111/201) Updating files: 56% (113/201) Updating files: 57% (115/201) Updating files: 58% (117/201) Updating files: 59% (119/201) Updating files: 60% (121/201) Updating files: 61% (123/201) Updating files: 62% (125/201) Updating files: 63% (127/201) Updating files: 64% (129/201) Updating files: 65% (131/201) Updating files: 66% (133/201) Updating files: 67% (135/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 89% (180/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2023-10-10T19:06:40.320 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2023-10-10T19:06:40.322 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.7 2023-10-10T19:06:40.760 INFO:tasks.workunit.client.0.smithi175.stdout:v0.8 2023-10-10T19:06:40.761 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:40.761 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.8 2023-10-10T19:06:40.761 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.8 2023-10-10T19:06:40.761 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.8 2023-10-10T19:06:51.699 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:06:51.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:06:53.580 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 48% (110/225) Updating files: 49% (111/225) Updating files: 50% (113/225) Updating files: 51% (115/225) Updating files: 52% (117/225) Updating files: 53% (120/225) Updating files: 54% (122/225) Updating files: 55% (124/225) Updating files: 56% (126/225) Updating files: 57% (129/225) Updating files: 58% (131/225) Updating files: 59% (133/225) Updating files: 60% (135/225) Updating files: 60% (136/225) Updating files: 61% (138/225) Updating files: 62% (140/225) Updating files: 63% (142/225) Updating files: 64% (144/225) Updating files: 65% (147/225) Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 77% (174/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2023-10-10T19:06:53.610 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2023-10-10T19:06:53.612 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.8 2023-10-10T19:06:53.668 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:53.668 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.9 2023-10-10T19:06:53.669 INFO:tasks.workunit.client.0.smithi175.stdout:v0.9 2023-10-10T19:06:53.669 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.9 2023-10-10T19:06:53.669 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.9 2023-10-10T19:06:57.339 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 5% (10/193) Updating files: 6% (12/193) Updating files: 7% (14/193) Updating files: 8% (16/193) Updating files: 9% (18/193) Updating files: 10% (20/193) Updating files: 11% (22/193) Updating files: 12% (24/193) Updating files: 13% (26/193) Updating files: 14% (28/193) Updating files: 15% (29/193) Updating files: 16% (31/193) Updating files: 17% (33/193) Updating files: 18% (35/193) Updating files: 19% (37/193) Updating files: 20% (39/193) Updating files: 21% (41/193) Updating files: 22% (43/193) Updating files: 23% (45/193) Updating files: 24% (47/193) Updating files: 25% (49/193) Updating files: 26% (51/193) Updating files: 27% (53/193) Updating files: 28% (55/193) Updating files: 29% (56/193) Updating files: 30% (58/193) Updating files: 31% (60/193) Updating files: 31% (61/193) Updating files: 32% (62/193) Updating files: 33% (64/193) Updating files: 34% (66/193) Updating files: 35% (68/193) Updating files: 36% (70/193) Updating files: 37% (72/193) Updating files: 38% (74/193) Updating files: 39% (76/193) Updating files: 40% (78/193) Updating files: 41% (80/193) Updating files: 42% (82/193) Updating files: 43% (83/193) Updating files: 44% (85/193) Updating files: 45% (87/193) Updating files: 46% (89/193) Updating files: 47% (91/193) Updating files: 48% (93/193) Updating files: 49% (95/193) Updating files: 50% (97/193) Updating files: 51% (99/193) Updating files: 52% (101/193) Updating files: 53% (103/193) Updating files: 54% (105/193) Updating files: 55% (107/193) Updating files: 56% (109/193) Updating files: 57% (111/193) Updating files: 58% (112/193) Updating files: 59% (114/193) Updating files: 60% (116/193) Updating files: 61% (118/193) Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 89% (173/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2023-10-10T19:06:57.361 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2023-10-10T19:06:57.363 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.9 2023-10-10T19:06:57.409 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:06:57.409 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.10 2023-10-10T19:06:57.410 INFO:tasks.workunit.client.0.smithi175.stdout:v0.10 2023-10-10T19:06:57.410 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.10 2023-10-10T19:06:57.410 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.10 2023-10-10T19:07:00.092 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2023-10-10T19:07:00.114 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 007313fe844 v0.10 2023-10-10T19:07:00.116 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.10 2023-10-10T19:07:00.175 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:07:00.175 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.11 2023-10-10T19:07:00.176 INFO:tasks.workunit.client.0.smithi175.stdout:v0.11 2023-10-10T19:07:00.176 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.11 2023-10-10T19:07:00.176 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.11 2023-10-10T19:07:04.750 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 8% (6/69) Updating files: 10% (7/69) Updating files: 11% (8/69) Updating files: 13% (9/69) Updating files: 14% (10/69) Updating files: 15% (11/69) Updating files: 17% (12/69) Updating files: 18% (13/69) Updating files: 20% (14/69) Updating files: 21% (15/69) Updating files: 23% (16/69) Updating files: 24% (17/69) Updating files: 26% (18/69) Updating files: 27% (19/69) Updating files: 28% (20/69) Updating files: 30% (21/69) Updating files: 31% (22/69) Updating files: 33% (23/69) Updating files: 34% (24/69) Updating files: 36% (25/69) Updating files: 37% (26/69) Updating files: 39% (27/69) Updating files: 40% (28/69) Updating files: 42% (29/69) Updating files: 43% (30/69) Updating files: 44% (31/69) Updating files: 46% (32/69) Updating files: 47% (33/69) Updating files: 49% (34/69) Updating files: 50% (35/69) Updating files: 52% (36/69) Updating files: 53% (37/69) Updating files: 55% (38/69) Updating files: 56% (39/69) Updating files: 57% (40/69) Updating files: 59% (41/69) Updating files: 60% (42/69) Updating files: 62% (43/69) Updating files: 63% (44/69) Updating files: 65% (45/69) Updating files: 66% (46/69) Updating files: 68% (47/69) Updating files: 69% (48/69) Updating files: 71% (49/69) Updating files: 72% (50/69) Updating files: 73% (51/69) Updating files: 75% (52/69) Updating files: 76% (53/69) Updating files: 78% (54/69) Updating files: 79% (55/69) Updating files: 81% (56/69) Updating files: 82% (57/69) Updating files: 84% (58/69) Updating files: 85% (59/69) Updating files: 86% (60/69) Updating files: 88% (61/69) Updating files: 89% (62/69) Updating files: 91% (63/69) Updating files: 92% (64/69) Updating files: 94% (65/69) Updating files: 95% (66/69) Updating files: 97% (67/69) Updating files: 98% (68/69) Updating files: 100% (69/69) Updating files: 100% (69/69), done. 2023-10-10T19:07:04.769 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2023-10-10T19:07:04.771 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.11 2023-10-10T19:07:05.698 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:07:05.698 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.12 2023-10-10T19:07:05.699 INFO:tasks.workunit.client.0.smithi175.stdout:v0.12 2023-10-10T19:07:05.699 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.12 2023-10-10T19:07:05.699 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.12 2023-10-10T19:07:05.799 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:05.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.915+0000 7fb29f648700 1 Processor -- start 2023-10-10T19:07:05.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.915+0000 7fb29f648700 1 -- start start 2023-10-10T19:07:05.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.916+0000 7fb29f648700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb2980b1fc0 0x7fb298155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:05.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.916+0000 7fb29f648700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb2980aff60 0x7fb2980b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:05.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.916+0000 7fb29f648700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2981574b0 con 0x7fb2980b2390 2023-10-10T19:07:05.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.916+0000 7fb29f648700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb298150770 con 0x7fb2980b1fc0 2023-10-10T19:07:05.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.916+0000 7fb29f648700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2980afad0 con 0x7fb2980aff60 2023-10-10T19:07:05.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.916+0000 7fb29d3e4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb2980b1fc0 0x7fb298155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:05.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.916+0000 7fb29d3e4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb2980b1fc0 0x7fb298155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34600/0 (socket says 172.21.15.175:34600) 2023-10-10T19:07:05.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.916+0000 7fb29d3e4700 1 -- 172.21.15.175:0/3630759236 learned_addr learned my addr 172.21.15.175:0/3630759236 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:05.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb29dbe5700 1 --2- 172.21.15.175:0/3630759236 >> v2:172.21.15.191:3300/0 conn(0x7fb2980aff60 0x7fb2980b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:05.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb29d3e4700 1 -- 172.21.15.175:0/3630759236 >> v2:172.21.15.191:3300/0 conn(0x7fb2980aff60 msgr2=0x7fb2980b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:05.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb29d3e4700 1 --2- 172.21.15.175:0/3630759236 >> v2:172.21.15.191:3300/0 conn(0x7fb2980aff60 0x7fb2980b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:05.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb29d3e4700 1 -- 172.21.15.175:0/3630759236 >> v1:172.21.15.191:6789/0 conn(0x7fb2980b2390 legacy=0x7fb298068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:05.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb29d3e4700 1 -- 172.21.15.175:0/3630759236 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2980af4c0 con 0x7fb2980b1fc0 2023-10-10T19:07:05.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb28ffff700 1 -- 172.21.15.175:0/3630759236 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416201823 0 0) 0x7fb2981574b0 con 0x7fb2980b2390 2023-10-10T19:07:05.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb29d3e4700 1 --2- 172.21.15.175:0/3630759236 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980b1fc0 0x7fb298155fb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb2980641f0 tx=0x7fb294009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d6ec48430ad16a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:05.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb28ffff700 1 -- 172.21.15.175:0/3630759236 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29400e070 con 0x7fb2980b1fc0 2023-10-10T19:07:05.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb28ffff700 1 -- 172.21.15.175:0/3630759236 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29400bab0 con 0x7fb2980b1fc0 2023-10-10T19:07:05.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb28ffff700 1 -- 172.21.15.175:0/3630759236 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2940092e0 con 0x7fb2980b1fc0 2023-10-10T19:07:05.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.917+0000 7fb29f648700 1 -- 172.21.15.175:0/3630759236 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980b1fc0 msgr2=0x7fb298155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:05.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.918+0000 7fb29f648700 1 --2- 172.21.15.175:0/3630759236 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980b1fc0 0x7fb298155fb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb2980641f0 tx=0x7fb294009840 comp rx=0 tx=0).stop 2023-10-10T19:07:05.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.918+0000 7fb29f648700 1 -- 172.21.15.175:0/3630759236 shutdown_connections 2023-10-10T19:07:05.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.918+0000 7fb29f648700 1 --2- 172.21.15.175:0/3630759236 >> v2:172.21.15.191:3300/0 conn(0x7fb2980aff60 0x7fb2980b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:05.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.918+0000 7fb29f648700 1 --2- 172.21.15.175:0/3630759236 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980b1fc0 0x7fb298155fb0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:05.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.918+0000 7fb29f648700 1 -- 172.21.15.175:0/3630759236 >> 172.21.15.175:0/3630759236 conn(0x7fb298063230 msgr2=0x7fb298063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:05.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.918+0000 7fb29f648700 1 -- 172.21.15.175:0/3630759236 shutdown_connections 2023-10-10T19:07:05.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.918+0000 7fb29f648700 1 -- 172.21.15.175:0/3630759236 wait complete. 2023-10-10T19:07:05.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.919+0000 7fb29f648700 1 Processor -- start 2023-10-10T19:07:05.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.919+0000 7fb29f648700 1 -- start start 2023-10-10T19:07:05.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.919+0000 7fb29f648700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980aff60 0x7fb2980ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:05.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29f648700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2980b1fc0 0x7fb298157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:05.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29f648700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2980b2390 0x7fb2981697a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:05.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29f648700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2980c3d60 con 0x7fb2980aff60 2023-10-10T19:07:05.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29f648700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29815c140 con 0x7fb2980b1fc0 2023-10-10T19:07:05.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29f648700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb298158c90 con 0x7fb2980b2390 2023-10-10T19:07:05.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980aff60 0x7fb2980ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:05.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980aff60 0x7fb2980ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34604/0 (socket says 172.21.15.175:34604) 2023-10-10T19:07:05.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 -- 172.21.15.175:0/1662838389 learned_addr learned my addr 172.21.15.175:0/1662838389 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:05.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29cbe3700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2980b1fc0 0x7fb298157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:05.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29dbe5700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2980b2390 0x7fb2981697a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:05.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 -- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2980b2390 msgr2=0x7fb2981697a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:05.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2980b2390 0x7fb2981697a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:05.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 -- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2980b1fc0 msgr2=0x7fb298157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:05.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2980b1fc0 0x7fb298157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:05.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 -- 172.21.15.175:0/1662838389 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb294009ae0 con 0x7fb2980aff60 2023-10-10T19:07:05.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29cbe3700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2980b1fc0 0x7fb298157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:05.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29dbe5700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2980b2390 0x7fb2981697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:05.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29d3e4700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980aff60 0x7fb2980ccda0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb29400b6c0 tx=0x7fb294018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:05.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb28d7fa700 1 -- 172.21.15.175:0/1662838389 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294008b30 con 0x7fb2980aff60 2023-10-10T19:07:05.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb28d7fa700 1 -- 172.21.15.175:0/1662838389 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb294008c90 con 0x7fb2980aff60 2023-10-10T19:07:05.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb28d7fa700 1 -- 172.21.15.175:0/1662838389 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29401d670 con 0x7fb2980aff60 2023-10-10T19:07:05.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2980cd430 con 0x7fb2980aff60 2023-10-10T19:07:05.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.920+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb298158640 con 0x7fb2980aff60 2023-10-10T19:07:05.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.921+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb298067570 con 0x7fb2980aff60 2023-10-10T19:07:05.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.922+0000 7fb28d7fa700 1 -- 172.21.15.175:0/1662838389 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb294003040 con 0x7fb2980aff60 2023-10-10T19:07:05.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.922+0000 7fb28d7fa700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb27c066f30 0x7fb27c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:05.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.922+0000 7fb28d7fa700 1 -- 172.21.15.175:0/1662838389 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fb2940863c0 con 0x7fb2980aff60 2023-10-10T19:07:05.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.922+0000 7fb29cbe3700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb27c066f30 0x7fb27c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:05.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.922+0000 7fb29cbe3700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb27c066f30 0x7fb27c0693e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb288007fd0 tx=0x7fb288009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:05.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:05.926+0000 7fb28d7fa700 1 -- 172.21.15.175:0/1662838389 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb294053260 con 0x7fb2980aff60 2023-10-10T19:07:06.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.169+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb298068da0 con 0x7fb2980aff60 2023-10-10T19:07:06.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.170+0000 7fb28d7fa700 1 -- 172.21.15.175:0/1662838389 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6209 (secure 0 0 0) 0x7fb294056880 con 0x7fb2980aff60 2023-10-10T19:07:06.171 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:06.172 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:06:59.641165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:06.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb27c066f30 msgr2=0x7fb27c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:06.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb27c066f30 0x7fb27c0693e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb288007fd0 tx=0x7fb288009040 comp rx=0 tx=0).stop 2023-10-10T19:07:06.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980aff60 msgr2=0x7fb2980ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:06.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980aff60 0x7fb2980ccda0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb29400b6c0 tx=0x7fb294018000 comp rx=0 tx=0).stop 2023-10-10T19:07:06.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 shutdown_connections 2023-10-10T19:07:06.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb27c066f30 0x7fb27c0693e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2980b2390 0x7fb2981697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2980aff60 0x7fb2980ccda0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 --2- 172.21.15.175:0/1662838389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2980b1fc0 0x7fb298157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 >> 172.21.15.175:0/1662838389 conn(0x7fb298063230 msgr2=0x7fb298152640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:06.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 shutdown_connections 2023-10-10T19:07:06.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.172+0000 7fb29f648700 1 -- 172.21.15.175:0/1662838389 wait complete. 2023-10-10T19:07:06.176 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 165 2023-10-10T19:07:06.185 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2023-10-10T19:07:06.186 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-10T19:07:06.186 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:07:06.196 INFO:tasks.ceph.mds.c:Stopped 2023-10-10T19:07:06.196 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:06.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.350+0000 7f94a8dbd700 1 Processor -- start 2023-10-10T19:07:06.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.350+0000 7f94a8dbd700 1 -- start start 2023-10-10T19:07:06.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a8dbd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e8f0 0x7f94a41569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:06.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a8dbd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94a40c9d20 0x7f94a406c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:06.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a8dbd700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a406cca0 con 0x7f94a406e520 2023-10-10T19:07:06.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a8dbd700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a40c33c0 con 0x7f94a40c9d20 2023-10-10T19:07:06.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a8dbd700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40c07f0 con 0x7f94a406e8f0 2023-10-10T19:07:06.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a3fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94a40c9d20 0x7f94a406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:06.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a37fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f94a406e520 0x7f94a40c9610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44948/0 (socket says 172.21.15.175:44948) 2023-10-10T19:07:06.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a37fe700 1 -- 172.21.15.175:0/2044647633 learned_addr learned my addr 172.21.15.175:0/2044647633 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:06.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a2ffd700 1 --2- 172.21.15.175:0/2044647633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e8f0 0x7f94a41569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:06.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a3fff700 1 -- 172.21.15.175:0/2044647633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e8f0 msgr2=0x7f94a41569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:06.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a3fff700 1 --2- 172.21.15.175:0/2044647633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e8f0 0x7f94a41569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a3fff700 1 -- 172.21.15.175:0/2044647633 >> v1:172.21.15.175:6789/0 conn(0x7f94a406e520 legacy=0x7f94a40c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:06.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a27fc700 1 -- 172.21.15.175:0/2044647633 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93025062 0 0) 0x7f94a406cca0 con 0x7f94a406e520 2023-10-10T19:07:06.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a3fff700 1 -- 172.21.15.175:0/2044647633 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4152b70 con 0x7f94a40c9d20 2023-10-10T19:07:06.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.351+0000 7f94a3fff700 1 --2- 172.21.15.175:0/2044647633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a406c760 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f949c0098e0 tx=0x7f949c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8074226c0f318686 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:06.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a27fc700 1 -- 172.21.15.175:0/2044647633 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c009e80 con 0x7f94a40c9d20 2023-10-10T19:07:06.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a27fc700 1 -- 172.21.15.175:0/2044647633 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c00b8f0 con 0x7f94a40c9d20 2023-10-10T19:07:06.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a27fc700 1 -- 172.21.15.175:0/2044647633 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c00bc70 con 0x7f94a40c9d20 2023-10-10T19:07:06.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a8dbd700 1 -- 172.21.15.175:0/2044647633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 msgr2=0x7f94a406c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:06.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/2044647633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a406c760 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f949c0098e0 tx=0x7f949c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a8dbd700 1 -- 172.21.15.175:0/2044647633 shutdown_connections 2023-10-10T19:07:06.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/2044647633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a406c760 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/2044647633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e8f0 0x7f94a41569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a8dbd700 1 -- 172.21.15.175:0/2044647633 >> 172.21.15.175:0/2044647633 conn(0x7f94a4063230 msgr2=0x7f94a406f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:06.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a8dbd700 1 -- 172.21.15.175:0/2044647633 shutdown_connections 2023-10-10T19:07:06.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.352+0000 7f94a8dbd700 1 -- 172.21.15.175:0/2044647633 wait complete. 2023-10-10T19:07:06.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.353+0000 7f94a8dbd700 1 Processor -- start 2023-10-10T19:07:06.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a8dbd700 1 -- start start 2023-10-10T19:07:06.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a8dbd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e520 0x7f94a4165770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:06.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a8dbd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94a406e8f0 0x7f94a414c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:06.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a8dbd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a40c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:06.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a8dbd700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a4145260 con 0x7f94a40c9d20 2023-10-10T19:07:06.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a8dbd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a4162b00 con 0x7f94a406e8f0 2023-10-10T19:07:06.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a8dbd700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a41621f0 con 0x7f94a406e520 2023-10-10T19:07:06.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a40c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:06.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a40c48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34622/0 (socket says 172.21.15.175:34622) 2023-10-10T19:07:06.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a3fff700 1 -- 172.21.15.175:0/4144219404 learned_addr learned my addr 172.21.15.175:0/4144219404 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:06.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a2ffd700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94a406e8f0 0x7f94a414c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:06.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.354+0000 7f94a37fe700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e520 0x7f94a4165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:06.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a3fff700 1 -- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e520 msgr2=0x7f94a4165770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:06.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a3fff700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e520 0x7f94a4165770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a3fff700 1 -- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94a406e8f0 msgr2=0x7f94a414c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:06.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a3fff700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94a406e8f0 0x7f94a414c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a3fff700 1 -- 172.21.15.175:0/4144219404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c009420 con 0x7f94a40c9d20 2023-10-10T19:07:06.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a2ffd700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94a406e8f0 0x7f94a414c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:06.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a3fff700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a40c48a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f94a406ee40 tx=0x7f949c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:06.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f948b7fe700 1 -- 172.21.15.175:0/4144219404 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0114f0 con 0x7f94a40c9d20 2023-10-10T19:07:06.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f948b7fe700 1 -- 172.21.15.175:0/4144219404 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c011650 con 0x7f94a40c9d20 2023-10-10T19:07:06.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a4145490 con 0x7f94a40c9d20 2023-10-10T19:07:06.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f948b7fe700 1 -- 172.21.15.175:0/4144219404 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c011a10 con 0x7f94a40c9d20 2023-10-10T19:07:06.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.355+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a41624b0 con 0x7f94a40c9d20 2023-10-10T19:07:06.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.356+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a4156c90 con 0x7f94a40c9d20 2023-10-10T19:07:06.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.356+0000 7f948b7fe700 1 -- 172.21.15.175:0/4144219404 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f949c023070 con 0x7f94a40c9d20 2023-10-10T19:07:06.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.357+0000 7f948b7fe700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9490066f30 0x7f94900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:06.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.357+0000 7f948b7fe700 1 -- 172.21.15.175:0/4144219404 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f949c086280 con 0x7f94a40c9d20 2023-10-10T19:07:06.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.357+0000 7f94a37fe700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9490066f30 0x7f94900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:06.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.357+0000 7f94a37fe700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9490066f30 0x7f94900693e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9494001560 tx=0x7f9494009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:06.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.361+0000 7f948b7fe700 1 -- 172.21.15.175:0/4144219404 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f949c053120 con 0x7f94a40c9d20 2023-10-10T19:07:06.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.586+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94a40c36c0 con 0x7f94a40c9d20 2023-10-10T19:07:06.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.586+0000 7f948b7fe700 1 -- 172.21.15.175:0/4144219404 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6209 (secure 0 0 0) 0x7f949c056740 con 0x7f94a40c9d20 2023-10-10T19:07:06.588 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:06.588 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:06:59.641165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:06.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.588+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9490066f30 msgr2=0x7f94900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:06.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.588+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9490066f30 0x7f94900693e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9494001560 tx=0x7f9494009040 comp rx=0 tx=0).stop 2023-10-10T19:07:06.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 msgr2=0x7f94a40c48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:06.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a40c48a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f94a406ee40 tx=0x7f949c017000 comp rx=0 tx=0).stop 2023-10-10T19:07:06.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 shutdown_connections 2023-10-10T19:07:06.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9490066f30 0x7f94900693e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94a40c9d20 0x7f94a40c48a0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94a406e520 0x7f94a4165770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 --2- 172.21.15.175:0/4144219404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94a406e8f0 0x7f94a414c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:06.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 >> 172.21.15.175:0/4144219404 conn(0x7f94a4063230 msgr2=0x7f94a4152110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:06.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 shutdown_connections 2023-10-10T19:07:06.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:06.589+0000 7f94a8dbd700 1 -- 172.21.15.175:0/4144219404 wait complete. 2023-10-10T19:07:06.593 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 165 2023-10-10T19:07:06.602 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:07:08.603 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:08.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.733+0000 7fd613c59700 1 Processor -- start 2023-10-10T19:07:08.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.733+0000 7fd613c59700 1 -- start start 2023-10-10T19:07:08.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.734+0000 7fd613c59700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0b2390 0x7fd60c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.734+0000 7fd613c59700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c1574b0 con 0x7fd60c0aff60 2023-10-10T19:07:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.734+0000 7fd613c59700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c0afad0 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.734+0000 7fd613c59700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c150770 con 0x7fd60c0b2390 2023-10-10T19:07:08.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.734+0000 7fd6121f6700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd60c0aff60 0x7fd60c159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44958/0 (socket says 172.21.15.175:44958) 2023-10-10T19:07:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.734+0000 7fd6121f6700 1 -- 172.21.15.175:0/2941410449 learned_addr learned my addr 172.21.15.175:0/2941410449 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.734+0000 7fd6111f4700 1 --2- 172.21.15.175:0/2941410449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0b2390 0x7fd60c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6109f3700 1 -- 172.21.15.175:0/2941410449 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876157657 0 0) 0x7fd60c1574b0 con 0x7fd60c0aff60 2023-10-10T19:07:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6109f3700 1 -- 172.21.15.175:0/2941410449 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f8003570 con 0x7fd60c0aff60 2023-10-10T19:07:08.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6109f3700 1 -- 172.21.15.175:0/2941410449 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297346374 0 0) 0x7fd60c0afad0 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6109f3700 1 -- 172.21.15.175:0/2941410449 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd60c1574b0 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6109f3700 1 -- 172.21.15.175:0/2941410449 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3130492209 0 0) 0x7fd5f8003570 con 0x7fd60c0aff60 2023-10-10T19:07:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6111f4700 1 -- 172.21.15.175:0/2941410449 >> v1:172.21.15.191:6789/0 conn(0x7fd60c0b1fc0 legacy=0x7fd60c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6111f4700 1 -- 172.21.15.175:0/2941410449 >> v1:172.21.15.175:6789/0 conn(0x7fd60c0aff60 legacy=0x7fd60c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:08.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6111f4700 1 -- 172.21.15.175:0/2941410449 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0af4c0 con 0x7fd60c0b2390 2023-10-10T19:07:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6111f4700 1 --2- 172.21.15.175:0/2941410449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0b2390 0x7fd60c155fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009940 tx=0x7fd5fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31d94ea79c112959 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6109f3700 1 -- 172.21.15.175:0/2941410449 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6040017f0 con 0x7fd60c0b2390 2023-10-10T19:07:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6109f3700 1 -- 172.21.15.175:0/2941410449 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc011420 con 0x7fd60c0b2390 2023-10-10T19:07:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.735+0000 7fd6109f3700 1 -- 172.21.15.175:0/2941410449 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0117c0 con 0x7fd60c0b2390 2023-10-10T19:07:08.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.736+0000 7fd613c59700 1 -- 172.21.15.175:0/2941410449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0b2390 msgr2=0x7fd60c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:08.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.736+0000 7fd613c59700 1 --2- 172.21.15.175:0/2941410449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0b2390 0x7fd60c155fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009940 tx=0x7fd5fc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:08.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.736+0000 7fd613c59700 1 -- 172.21.15.175:0/2941410449 shutdown_connections 2023-10-10T19:07:08.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.736+0000 7fd613c59700 1 --2- 172.21.15.175:0/2941410449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0b2390 0x7fd60c155fb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:08.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.736+0000 7fd613c59700 1 -- 172.21.15.175:0/2941410449 >> 172.21.15.175:0/2941410449 conn(0x7fd60c063230 msgr2=0x7fd60c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:08.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.736+0000 7fd613c59700 1 -- 172.21.15.175:0/2941410449 shutdown_connections 2023-10-10T19:07:08.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.736+0000 7fd613c59700 1 -- 172.21.15.175:0/2941410449 wait complete. 2023-10-10T19:07:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.737+0000 7fd613c59700 1 Processor -- start 2023-10-10T19:07:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.737+0000 7fd613c59700 1 -- start start 2023-10-10T19:07:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd613c59700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0aff60 0x7fd60c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd613c59700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd60c0b1fc0 0x7fd60c0cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd613c59700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd60c0b2390 0x7fd60c0cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd613c59700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0cbf70 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd613c59700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c165cb0 con 0x7fd60c0b2390 2023-10-10T19:07:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd613c59700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c165f70 con 0x7fd60c0aff60 2023-10-10T19:07:08.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd60c0b1fc0 0x7fd60c0cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd60c0b1fc0 0x7fd60c0cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34626/0 (socket says 172.21.15.175:34626) 2023-10-10T19:07:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 -- 172.21.15.175:0/1322444947 learned_addr learned my addr 172.21.15.175:0/1322444947 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6121f6700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd60c0b2390 0x7fd60c0cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:08.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6119f5700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0aff60 0x7fd60c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 -- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0aff60 msgr2=0x7fd60c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0aff60 0x7fd60c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 -- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd60c0b2390 msgr2=0x7fd60c0cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd60c0b2390 0x7fd60c0cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:08.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 -- 172.21.15.175:0/1322444947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc009420 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6119f5700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0aff60 0x7fd60c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:07:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd6111f4700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd60c0b1fc0 0x7fd60c0cc460 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009860 tx=0x7fd5fc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd601ffb700 1 -- 172.21.15.175:0/1322444947 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc01bc80 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd601ffb700 1 -- 172.21.15.175:0/1322444947 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc01bde0 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c166250 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.738+0000 7fd601ffb700 1 -- 172.21.15.175:0/1322444947 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc01c680 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.739+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c166810 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.739+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c067570 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.740+0000 7fd601ffb700 1 -- 172.21.15.175:0/1322444947 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd5fc01ca30 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.741+0000 7fd601ffb700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5f0066f30 0x7fd5f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:08.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.741+0000 7fd601ffb700 1 -- 172.21.15.175:0/1322444947 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fd5fc086460 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.741+0000 7fd6119f5700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5f0066f30 0x7fd5f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:08.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.741+0000 7fd6119f5700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5f0066f30 0x7fd5f00693e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd608007fd0 tx=0x7fd608009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:08.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.744+0000 7fd601ffb700 1 -- 172.21.15.175:0/1322444947 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd5fc0532e0 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.979+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd60c068da0 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.980+0000 7fd601ffb700 1 -- 172.21.15.175:0/1322444947 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6209 (secure 0 0 0) 0x7fd5fc056900 con 0x7fd60c0b1fc0 2023-10-10T19:07:08.981 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:08.982 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:06:59.641165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:08.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5f0066f30 msgr2=0x7fd5f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:08.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5f0066f30 0x7fd5f00693e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd608007fd0 tx=0x7fd608009040 comp rx=0 tx=0).stop 2023-10-10T19:07:08.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd60c0b1fc0 msgr2=0x7fd60c0cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:08.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd60c0b1fc0 0x7fd60c0cc460 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009860 tx=0x7fd5fc016000 comp rx=0 tx=0).stop 2023-10-10T19:07:08.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 shutdown_connections 2023-10-10T19:07:08.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5f0066f30 0x7fd5f00693e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:08.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd60c0b2390 0x7fd60c0cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:08.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd60c0aff60 0x7fd60c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:08.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 --2- 172.21.15.175:0/1322444947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd60c0b1fc0 0x7fd60c0cc460 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:08.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 >> 172.21.15.175:0/1322444947 conn(0x7fd60c063230 msgr2=0x7fd60c068690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:08.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 shutdown_connections 2023-10-10T19:07:08.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:08.982+0000 7fd613c59700 1 -- 172.21.15.175:0/1322444947 wait complete. 2023-10-10T19:07:08.986 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 165 2023-10-10T19:07:08.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:07:10.998 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:11.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.123+0000 7fcb87462700 1 Processor -- start 2023-10-10T19:07:11.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.123+0000 7fcb87462700 1 -- start start 2023-10-10T19:07:11.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb87462700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcb8014fd90 0x7fcb80159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:11.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb87462700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb80067fa0 con 0x7fcb800cd040 2023-10-10T19:07:11.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb87462700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb80156b70 con 0x7fcb8008a100 2023-10-10T19:07:11.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb87462700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb800c5600 con 0x7fcb8014fd90 2023-10-10T19:07:11.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb86c61700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcb800cd040 0x7fcb800684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44972/0 (socket says 172.21.15.175:44972) 2023-10-10T19:07:11.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb86c61700 1 -- 172.21.15.175:0/3372573223 learned_addr learned my addr 172.21.15.175:0/3372573223 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:11.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb86460700 1 --2- 172.21.15.175:0/3372573223 >> v2:172.21.15.191:3300/0 conn(0x7fcb8014fd90 0x7fcb80159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:11.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198310199 0 0) 0x7fcb80067fa0 con 0x7fcb800cd040 2023-10-10T19:07:11.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb6c003570 con 0x7fcb800cd040 2023-10-10T19:07:11.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 744464130 0 0) 0x7fcb6c003570 con 0x7fcb800cd040 2023-10-10T19:07:11.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 >> v2:172.21.15.191:3300/0 conn(0x7fcb8014fd90 msgr2=0x7fcb80159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:11.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 --2- 172.21.15.175:0/3372573223 >> v2:172.21.15.191:3300/0 conn(0x7fcb8014fd90 0x7fcb80159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 >> v1:172.21.15.191:6789/0 conn(0x7fcb8008a100 legacy=0x7fcb800c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:11.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb80150160 con 0x7fcb800cd040 2023-10-10T19:07:11.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fcb7c0017f0 con 0x7fcb800cd040 2023-10-10T19:07:11.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 >> v1:172.21.15.175:6789/0 conn(0x7fcb800cd040 legacy=0x7fcb800684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:11.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb6c003a60 0x7fcb6c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:11.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb6c004810 0x7fcb6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:11.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.124+0000 7fcb8545e700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c0051c0 0x7fcb6c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:11.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800c5600 con 0x7fcb6c004810 2023-10-10T19:07:11.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c00c0d0 con 0x7fcb6c0051c0 2023-10-10T19:07:11.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c00c290 con 0x7fcb6c003a60 2023-10-10T19:07:11.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb85c5f700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb6c004810 0x7fcb6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:11.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb86c61700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c0051c0 0x7fcb6c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:11.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb86460700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb6c003a60 0x7fcb6c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:11.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb85c5f700 1 -- 172.21.15.175:0/3372573223 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb6c003a60 msgr2=0x7fcb6c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:11.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb85c5f700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb6c003a60 0x7fcb6c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb85c5f700 1 -- 172.21.15.175:0/3372573223 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c0051c0 msgr2=0x7fcb6c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:11.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb85c5f700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c0051c0 0x7fcb6c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb86c61700 1 -- 172.21.15.175:0/3372573223 reap_dead start 2023-10-10T19:07:11.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb85c5f700 1 -- 172.21.15.175:0/3372573223 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c003570 con 0x7fcb6c004810 2023-10-10T19:07:11.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb85c5f700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb6c004810 0x7fcb6c004c80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fcb700018b0 tx=0x7fcb70007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34948dd537b2d9f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:11.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70012420 con 0x7fcb6c004810 2023-10-10T19:07:11.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb700125a0 con 0x7fcb6c004810 2023-10-10T19:07:11.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.125+0000 7fcb8545e700 1 -- 172.21.15.175:0/3372573223 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70012960 con 0x7fcb6c004810 2023-10-10T19:07:11.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.126+0000 7fcb87462700 1 -- 172.21.15.175:0/3372573223 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb6c004810 msgr2=0x7fcb6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:11.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.126+0000 7fcb87462700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb6c004810 0x7fcb6c004c80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fcb700018b0 tx=0x7fcb70007d30 comp rx=0 tx=0).stop 2023-10-10T19:07:11.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.126+0000 7fcb87462700 1 -- 172.21.15.175:0/3372573223 shutdown_connections 2023-10-10T19:07:11.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.126+0000 7fcb87462700 1 --2- 172.21.15.175:0/3372573223 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb6c004810 0x7fcb6c004c80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.126+0000 7fcb87462700 1 -- 172.21.15.175:0/3372573223 >> 172.21.15.175:0/3372573223 conn(0x7fcb80063d00 msgr2=0x7fcb800cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:11.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.126+0000 7fcb87462700 1 -- 172.21.15.175:0/3372573223 shutdown_connections 2023-10-10T19:07:11.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.126+0000 7fcb87462700 1 -- 172.21.15.175:0/3372573223 wait complete. 2023-10-10T19:07:11.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.127+0000 7fcb87462700 1 Processor -- start 2023-10-10T19:07:11.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.127+0000 7fcb87462700 1 -- start start 2023-10-10T19:07:11.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.127+0000 7fcb87462700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c004810 0x7fcb80154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:11.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb87462700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb80154df0 0x7fcb80157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:11.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb87462700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb8014c640 0x7fcb8014bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:11.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb87462700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb80150ae0 con 0x7fcb80154df0 2023-10-10T19:07:11.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb87462700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb80154bf0 con 0x7fcb6c004810 2023-10-10T19:07:11.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb87462700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb80153ec0 con 0x7fcb8014c640 2023-10-10T19:07:11.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb80154df0 0x7fcb80157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:11.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb80154df0 0x7fcb80157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34644/0 (socket says 172.21.15.175:34644) 2023-10-10T19:07:11.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 -- 172.21.15.175:0/553244866 learned_addr learned my addr 172.21.15.175:0/553244866 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:11.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb86460700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c004810 0x7fcb80154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:11.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb86c61700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb8014c640 0x7fcb8014bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:11.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 -- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb8014c640 msgr2=0x7fcb8014bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:11.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb8014c640 0x7fcb8014bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 -- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c004810 msgr2=0x7fcb80154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:11.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c004810 0x7fcb80154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 -- 172.21.15.175:0/553244866 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb80155730 con 0x7fcb80154df0 2023-10-10T19:07:11.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb86c61700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb8014c640 0x7fcb8014bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:07:11.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb85c5f700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb80154df0 0x7fcb80157fc0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c0a0 tx=0x7fcb70015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:11.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb767fc700 1 -- 172.21.15.175:0/553244866 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7000c800 con 0x7fcb80154df0 2023-10-10T19:07:11.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb767fc700 1 -- 172.21.15.175:0/553244866 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7000c960 con 0x7fcb80154df0 2023-10-10T19:07:11.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb80166760 con 0x7fcb80154df0 2023-10-10T19:07:11.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb767fc700 1 -- 172.21.15.175:0/553244866 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7000cd00 con 0x7fcb80154df0 2023-10-10T19:07:11.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.128+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb80166a00 con 0x7fcb80154df0 2023-10-10T19:07:11.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.129+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb80067570 con 0x7fcb80154df0 2023-10-10T19:07:11.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.130+0000 7fcb767fc700 1 -- 172.21.15.175:0/553244866 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcb7001f620 con 0x7fcb80154df0 2023-10-10T19:07:11.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.130+0000 7fcb767fc700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb64066f30 0x7fcb640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:11.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.130+0000 7fcb767fc700 1 -- 172.21.15.175:0/553244866 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fcb70086170 con 0x7fcb80154df0 2023-10-10T19:07:11.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.130+0000 7fcb86460700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb64066f30 0x7fcb640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:11.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.131+0000 7fcb86460700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb64066f30 0x7fcb640693e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcb78006f00 tx=0x7fcb78008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:11.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.134+0000 7fcb767fc700 1 -- 172.21.15.175:0/553244866 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb70053010 con 0x7fcb80154df0 2023-10-10T19:07:11.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.362+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb80154000 con 0x7fcb80154df0 2023-10-10T19:07:11.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.362+0000 7fcb767fc700 1 -- 172.21.15.175:0/553244866 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6209 (secure 0 0 0) 0x7fcb70056630 con 0x7fcb80154df0 2023-10-10T19:07:11.364 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:11.364 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:06:59.641165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:11.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb64066f30 msgr2=0x7fcb640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:11.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb64066f30 0x7fcb640693e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcb78006f00 tx=0x7fcb78008040 comp rx=0 tx=0).stop 2023-10-10T19:07:11.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb80154df0 msgr2=0x7fcb80157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:11.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb80154df0 0x7fcb80157fc0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c0a0 tx=0x7fcb70015000 comp rx=0 tx=0).stop 2023-10-10T19:07:11.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 shutdown_connections 2023-10-10T19:07:11.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb64066f30 0x7fcb640693e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb8014c640 0x7fcb8014bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb6c004810 0x7fcb80154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 --2- 172.21.15.175:0/553244866 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb80154df0 0x7fcb80157fc0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:11.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 >> 172.21.15.175:0/553244866 conn(0x7fcb80063d00 msgr2=0x7fcb800c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:11.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 shutdown_connections 2023-10-10T19:07:11.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:11.365+0000 7fcb87462700 1 -- 172.21.15.175:0/553244866 wait complete. 2023-10-10T19:07:11.369 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 165 2023-10-10T19:07:11.378 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:07:13.379 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:13.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.504+0000 7f1340f66700 1 Processor -- start 2023-10-10T19:07:13.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.504+0000 7f1340f66700 1 -- start start 2023-10-10T19:07:13.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f1340f66700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c5850 0x7f133c0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:13.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f1340f66700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f133c0c4f20 0x7f133c0c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:13.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f1340f66700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0c6fb0 con 0x7f133c0c12d0 2023-10-10T19:07:13.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f1340f66700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133c063ea0 con 0x7f133c0c4f20 2023-10-10T19:07:13.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f1340f66700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133c155f30 con 0x7f133c0c5850 2023-10-10T19:07:13.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f133ad9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f133c0c4f20 0x7f133c0c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:13.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f1339d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f133c0c12d0 0x7f133c156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44976/0 (socket says 172.21.15.175:44976) 2023-10-10T19:07:13.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f1339d9b700 1 -- 172.21.15.175:0/1643562657 learned_addr learned my addr 172.21.15.175:0/1643562657 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:13.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.505+0000 7f133a59c700 1 --2- 172.21.15.175:0/1643562657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c5850 0x7f133c0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:13.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133959a700 1 -- 172.21.15.175:0/1643562657 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315723855 0 0) 0x7f133c0c6fb0 con 0x7f133c0c12d0 2023-10-10T19:07:13.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133959a700 1 -- 172.21.15.175:0/1643562657 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1324003570 con 0x7f133c0c12d0 2023-10-10T19:07:13.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133ad9d700 1 -- 172.21.15.175:0/1643562657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c5850 msgr2=0x7f133c0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:13.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133ad9d700 1 --2- 172.21.15.175:0/1643562657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c5850 0x7f133c0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133ad9d700 1 -- 172.21.15.175:0/1643562657 >> v1:172.21.15.175:6789/0 conn(0x7f133c0c12d0 legacy=0x7f133c156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:13.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133ad9d700 1 -- 172.21.15.175:0/1643562657 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0c6160 con 0x7f133c0c4f20 2023-10-10T19:07:13.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133ad9d700 1 --2- 172.21.15.175:0/1643562657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c4f20 0x7f133c0c7370 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f13340098e0 tx=0x7f13340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8b4dbb750cce2c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:13.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133959a700 1 -- 172.21.15.175:0/1643562657 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334011630 con 0x7f133c0c4f20 2023-10-10T19:07:13.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133959a700 1 -- 172.21.15.175:0/1643562657 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1334011790 con 0x7f133c0c4f20 2023-10-10T19:07:13.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.506+0000 7f133959a700 1 -- 172.21.15.175:0/1643562657 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334011b30 con 0x7f133c0c4f20 2023-10-10T19:07:13.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.507+0000 7f1340f66700 1 -- 172.21.15.175:0/1643562657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c4f20 msgr2=0x7f133c0c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:13.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.507+0000 7f1340f66700 1 --2- 172.21.15.175:0/1643562657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c4f20 0x7f133c0c7370 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f13340098e0 tx=0x7f13340091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.507+0000 7f1340f66700 1 -- 172.21.15.175:0/1643562657 shutdown_connections 2023-10-10T19:07:13.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.507+0000 7f1340f66700 1 --2- 172.21.15.175:0/1643562657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c4f20 0x7f133c0c7370 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.507+0000 7f1340f66700 1 --2- 172.21.15.175:0/1643562657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c5850 0x7f133c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.507+0000 7f1340f66700 1 -- 172.21.15.175:0/1643562657 >> 172.21.15.175:0/1643562657 conn(0x7f133c06dc70 msgr2=0x7f133c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:13.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.507+0000 7f1340f66700 1 -- 172.21.15.175:0/1643562657 shutdown_connections 2023-10-10T19:07:13.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.507+0000 7f1340f66700 1 -- 172.21.15.175:0/1643562657 wait complete. 2023-10-10T19:07:13.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.508+0000 7f1340f66700 1 Processor -- start 2023-10-10T19:07:13.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.508+0000 7f1340f66700 1 -- start start 2023-10-10T19:07:13.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.508+0000 7f1340f66700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c12d0 0x7f133c155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:13.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.508+0000 7f1340f66700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c4f20 0x7f133c0c2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:13.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.508+0000 7f1340f66700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f133c0c5850 0x7f133c0c3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:13.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.508+0000 7f1340f66700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c0c1c20 con 0x7f133c0c12d0 2023-10-10T19:07:13.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.508+0000 7f1340f66700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c169570 con 0x7f133c0c5850 2023-10-10T19:07:13.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.508+0000 7f1340f66700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133c0c2a90 con 0x7f133c0c4f20 2023-10-10T19:07:13.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c12d0 0x7f133c155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:13.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c12d0 0x7f133c155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34662/0 (socket says 172.21.15.175:34662) 2023-10-10T19:07:13.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 -- 172.21.15.175:0/2915682720 learned_addr learned my addr 172.21.15.175:0/2915682720 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:13.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133ad9d700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f133c0c5850 0x7f133c0c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:13.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f1339d9b700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c4f20 0x7f133c0c2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:13.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 -- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c4f20 msgr2=0x7f133c0c2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:13.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c4f20 0x7f133c0c2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 -- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f133c0c5850 msgr2=0x7f133c0c3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:13.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f133c0c5850 0x7f133c0c3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 -- 172.21.15.175:0/2915682720 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1334009420 con 0x7f133c0c12d0 2023-10-10T19:07:13.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f1339d9b700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c4f20 0x7f133c0c2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:07:13.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f133a59c700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c12d0 0x7f133c155510 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f132c009ef0 tx=0x7f132c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:13.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f132a7fc700 1 -- 172.21.15.175:0/2915682720 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c00cce0 con 0x7f133c0c12d0 2023-10-10T19:07:13.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f132a7fc700 1 -- 172.21.15.175:0/2915682720 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c00f040 con 0x7f133c0c12d0 2023-10-10T19:07:13.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c168ff0 con 0x7f133c0c12d0 2023-10-10T19:07:13.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f132a7fc700 1 -- 172.21.15.175:0/2915682720 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c0146b0 con 0x7f133c0c12d0 2023-10-10T19:07:13.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.509+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c169290 con 0x7f133c0c12d0 2023-10-10T19:07:13.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.510+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c0c79d0 con 0x7f133c0c12d0 2023-10-10T19:07:13.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.511+0000 7f132a7fc700 1 -- 172.21.15.175:0/2915682720 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f132c014a20 con 0x7f133c0c12d0 2023-10-10T19:07:13.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.511+0000 7f132a7fc700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f131c066f30 0x7f131c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:13.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.511+0000 7f132a7fc700 1 -- 172.21.15.175:0/2915682720 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f132c085690 con 0x7f133c0c12d0 2023-10-10T19:07:13.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.511+0000 7f1339d9b700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f131c066f30 0x7f131c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:13.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.511+0000 7f1339d9b700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f131c066f30 0x7f131c0693e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f13300058d0 tx=0x7f1330009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:13.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.515+0000 7f132a7fc700 1 -- 172.21.15.175:0/2915682720 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f133c0c79d0 con 0x7f133c0c12d0 2023-10-10T19:07:13.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.741+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f133c0c79d0 con 0x7f133c0c12d0 2023-10-10T19:07:13.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.742+0000 7f132a7fc700 1 -- 172.21.15.175:0/2915682720 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6209 (secure 0 0 0) 0x7f133c0c79d0 con 0x7f133c0c12d0 2023-10-10T19:07:13.744 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:13.745 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:06:59.641165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:13.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f131c066f30 msgr2=0x7f131c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:13.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f131c066f30 0x7f131c0693e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f13300058d0 tx=0x7f1330009040 comp rx=0 tx=0).stop 2023-10-10T19:07:13.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c12d0 msgr2=0x7f133c155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:13.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c12d0 0x7f133c155510 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f132c009ef0 tx=0x7f132c00b040 comp rx=0 tx=0).stop 2023-10-10T19:07:13.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 shutdown_connections 2023-10-10T19:07:13.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f131c066f30 0x7f131c0693e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f133c0c5850 0x7f133c0c3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f133c0c12d0 0x7f133c155510 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 --2- 172.21.15.175:0/2915682720 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f133c0c4f20 0x7f133c0c2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:13.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 >> 172.21.15.175:0/2915682720 conn(0x7f133c06dc70 msgr2=0x7f133c14eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:13.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 shutdown_connections 2023-10-10T19:07:13.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:13.744+0000 7f1340f66700 1 -- 172.21.15.175:0/2915682720 wait complete. 2023-10-10T19:07:13.749 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 165 2023-10-10T19:07:13.758 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:07:15.759 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:15.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.886+0000 7f4a3bac3700 1 Processor -- start 2023-10-10T19:07:15.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.886+0000 7f4a3bac3700 1 -- start start 2023-10-10T19:07:15.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3bac3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4a340b1fc0 0x7f4a34155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:15.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3bac3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4a340b2390 0x7f4a34068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:15.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3bac3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a341574b0 con 0x7f4a34068bd0 2023-10-10T19:07:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3bac3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a34150770 con 0x7f4a340b2390 2023-10-10T19:07:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3bac3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a340afad0 con 0x7f4a340b1fc0 2023-10-10T19:07:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3905e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4a340b2390 0x7f4a34068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3a060700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f4a34068bd0 0x7f4a340aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51808/0 (socket says 172.21.15.175:51808) 2023-10-10T19:07:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3a060700 1 -- 172.21.15.175:0/2312792850 learned_addr learned my addr 172.21.15.175:0/2312792850 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:15.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3985f700 1 --2- 172.21.15.175:0/2312792850 >> v2:172.21.15.191:3300/0 conn(0x7f4a340b1fc0 0x7f4a34155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:15.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3905e700 1 -- 172.21.15.175:0/2312792850 >> v2:172.21.15.191:3300/0 conn(0x7f4a340b1fc0 msgr2=0x7f4a34155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:15.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3905e700 1 --2- 172.21.15.175:0/2312792850 >> v2:172.21.15.191:3300/0 conn(0x7f4a340b1fc0 0x7f4a34155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:15.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3905e700 1 -- 172.21.15.175:0/2312792850 >> v1:172.21.15.175:6789/0 conn(0x7f4a34068bd0 legacy=0x7f4a340aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:15.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3905e700 1 -- 172.21.15.175:0/2312792850 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a340af4c0 con 0x7f4a340b2390 2023-10-10T19:07:15.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3885d700 1 -- 172.21.15.175:0/2312792850 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168631983 0 0) 0x7f4a341574b0 con 0x7f4a34068bd0 2023-10-10T19:07:15.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.887+0000 7f4a3985f700 1 --2- 172.21.15.175:0/2312792850 >> v2:172.21.15.191:3300/0 conn(0x7f4a340b1fc0 0x7f4a34155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:15.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3905e700 1 --2- 172.21.15.175:0/2312792850 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a34068690 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4a340b28c0 tx=0x7f4a240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196376233c34ac75 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:15.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3885d700 1 -- 172.21.15.175:0/2312792850 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2400b740 con 0x7f4a340b2390 2023-10-10T19:07:15.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3885d700 1 -- 172.21.15.175:0/2312792850 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2400b8a0 con 0x7f4a340b2390 2023-10-10T19:07:15.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3885d700 1 -- 172.21.15.175:0/2312792850 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2400bc40 con 0x7f4a340b2390 2023-10-10T19:07:15.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3bac3700 1 -- 172.21.15.175:0/2312792850 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 msgr2=0x7f4a34068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:15.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/2312792850 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a34068690 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4a340b28c0 tx=0x7f4a240091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:15.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3bac3700 1 -- 172.21.15.175:0/2312792850 shutdown_connections 2023-10-10T19:07:15.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/2312792850 >> v2:172.21.15.191:3300/0 conn(0x7f4a340b1fc0 0x7f4a34155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:15.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/2312792850 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a34068690 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:15.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3bac3700 1 -- 172.21.15.175:0/2312792850 >> 172.21.15.175:0/2312792850 conn(0x7f4a34063230 msgr2=0x7f4a34063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:15.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3bac3700 1 -- 172.21.15.175:0/2312792850 shutdown_connections 2023-10-10T19:07:15.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.888+0000 7f4a3bac3700 1 -- 172.21.15.175:0/2312792850 wait complete. 2023-10-10T19:07:15.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.889+0000 7f4a3bac3700 1 Processor -- start 2023-10-10T19:07:15.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3bac3700 1 -- start start 2023-10-10T19:07:15.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3bac3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a34068bd0 0x7f4a340ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:15.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3bac3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a340b1fc0 0x7f4a340cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:15.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3bac3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a3414df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:15.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3bac3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a340cd1d0 con 0x7f4a340b2390 2023-10-10T19:07:15.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3bac3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3414d540 con 0x7f4a34068bd0 2023-10-10T19:07:15.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3bac3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3414db00 con 0x7f4a340b1fc0 2023-10-10T19:07:15.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3a060700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a3414df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:15.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3a060700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a3414df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39382/0 (socket says 172.21.15.175:39382) 2023-10-10T19:07:15.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3a060700 1 -- 172.21.15.175:0/130969072 learned_addr learned my addr 172.21.15.175:0/130969072 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:15.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.890+0000 7f4a3985f700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a34068bd0 0x7f4a340ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:15.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3905e700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a340b1fc0 0x7f4a340cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:15.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3a060700 1 -- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a340b1fc0 msgr2=0x7f4a340cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:15.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3a060700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a340b1fc0 0x7f4a340cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:15.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3a060700 1 -- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a34068bd0 msgr2=0x7f4a340ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:15.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3a060700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a34068bd0 0x7f4a340ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:15.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3a060700 1 -- 172.21.15.175:0/130969072 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a24009420 con 0x7f4a340b2390 2023-10-10T19:07:15.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3a060700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a3414df90 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00ccc0 tx=0x7f4a2c001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:15.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a29ffb700 1 -- 172.21.15.175:0/130969072 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00ab30 con 0x7f4a340b2390 2023-10-10T19:07:15.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a29ffb700 1 -- 172.21.15.175:0/130969072 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c00ac90 con 0x7f4a340b2390 2023-10-10T19:07:15.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a29ffb700 1 -- 172.21.15.175:0/130969072 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c007340 con 0x7f4a340b2390 2023-10-10T19:07:15.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a34156420 con 0x7f4a340b2390 2023-10-10T19:07:15.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.891+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a341569e0 con 0x7f4a340b2390 2023-10-10T19:07:15.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.892+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a34067570 con 0x7f4a340b2390 2023-10-10T19:07:15.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.892+0000 7f4a29ffb700 1 -- 172.21.15.175:0/130969072 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4a2c0076f0 con 0x7f4a340b2390 2023-10-10T19:07:15.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.893+0000 7f4a29ffb700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a18066f30 0x7f4a180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:15.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.893+0000 7f4a29ffb700 1 -- 172.21.15.175:0/130969072 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f4a2c084df0 con 0x7f4a340b2390 2023-10-10T19:07:15.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.893+0000 7f4a3985f700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a18066f30 0x7f4a180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:15.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.893+0000 7f4a3985f700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a18066f30 0x7f4a180693e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4a30006f00 tx=0x7f4a30008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:15.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:15.897+0000 7f4a29ffb700 1 -- 172.21.15.175:0/130969072 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4a2c051d10 con 0x7f4a340b2390 2023-10-10T19:07:16.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.120+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a3414dc40 con 0x7f4a340b2390 2023-10-10T19:07:16.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.121+0000 7f4a29ffb700 1 -- 172.21.15.175:0/130969072 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6209 (secure 0 0 0) 0x7f4a2c055330 con 0x7f4a340b2390 2023-10-10T19:07:16.123 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:16.123 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:06:59.641165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:16.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a18066f30 msgr2=0x7f4a180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:16.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a18066f30 0x7f4a180693e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4a30006f00 tx=0x7f4a30008040 comp rx=0 tx=0).stop 2023-10-10T19:07:16.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 msgr2=0x7f4a3414df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:16.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a3414df90 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00ccc0 tx=0x7f4a2c001a80 comp rx=0 tx=0).stop 2023-10-10T19:07:16.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 shutdown_connections 2023-10-10T19:07:16.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4a18066f30 0x7f4a180693e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:16.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a340b2390 0x7f4a3414df90 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:16.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a34068bd0 0x7f4a340ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:16.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 --2- 172.21.15.175:0/130969072 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a340b1fc0 0x7f4a340cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:16.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 >> 172.21.15.175:0/130969072 conn(0x7f4a34063230 msgr2=0x7f4a341508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:16.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 shutdown_connections 2023-10-10T19:07:16.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:16.123+0000 7f4a3bac3700 1 -- 172.21.15.175:0/130969072 wait complete. 2023-10-10T19:07:16.127 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 165 2023-10-10T19:07:16.136 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:07:18.137 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:18.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.263+0000 7f4233b1a700 1 Processor -- start 2023-10-10T19:07:18.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.264+0000 7f4233b1a700 1 -- start start 2023-10-10T19:07:18.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.264+0000 7f4233b1a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f422c1573d0 0x7f422c0c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:18.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.264+0000 7f4233b1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c1577a0 0x7f422c0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:18.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f4233b1a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c154aa0 con 0x7f422c0bf6e0 2023-10-10T19:07:18.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f4233b1a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c063dc0 con 0x7f422c1573d0 2023-10-10T19:07:18.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f4233b1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c157bf0 con 0x7f422c1577a0 2023-10-10T19:07:18.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42318b6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f422c1573d0 0x7f422c0c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:18.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42320b7700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f422c0bf6e0 0x7f422c0c6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51810/0 (socket says 172.21.15.175:51810) 2023-10-10T19:07:18.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42320b7700 1 -- 172.21.15.175:0/4016829176 learned_addr learned my addr 172.21.15.175:0/4016829176 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:18.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42310b5700 1 --2- 172.21.15.175:0/4016829176 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c1577a0 0x7f422c0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:18.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42318b6700 1 -- 172.21.15.175:0/4016829176 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c1577a0 msgr2=0x7f422c0c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42318b6700 1 --2- 172.21.15.175:0/4016829176 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c1577a0 0x7f422c0c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42318b6700 1 -- 172.21.15.175:0/4016829176 >> v1:172.21.15.175:6789/0 conn(0x7f422c0bf6e0 legacy=0x7f422c0c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42308b4700 1 -- 172.21.15.175:0/4016829176 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337823767 0 0) 0x7f422c154aa0 con 0x7f422c0bf6e0 2023-10-10T19:07:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42318b6700 1 -- 172.21.15.175:0/4016829176 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c157e50 con 0x7f422c1573d0 2023-10-10T19:07:18.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42318b6700 1 --2- 172.21.15.175:0/4016829176 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c0c12a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f42280098e0 tx=0x7f42280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d12e93914e3b3f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42308b4700 1 -- 172.21.15.175:0/4016829176 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228009e80 con 0x7f422c1573d0 2023-10-10T19:07:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42308b4700 1 -- 172.21.15.175:0/4016829176 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422800b860 con 0x7f422c1573d0 2023-10-10T19:07:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.265+0000 7f42308b4700 1 -- 172.21.15.175:0/4016829176 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800bbe0 con 0x7f422c1573d0 2023-10-10T19:07:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.266+0000 7f4233b1a700 1 -- 172.21.15.175:0/4016829176 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 msgr2=0x7f422c0c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:18.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.266+0000 7f4233b1a700 1 --2- 172.21.15.175:0/4016829176 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c0c12a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f42280098e0 tx=0x7f42280091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.266+0000 7f4233b1a700 1 -- 172.21.15.175:0/4016829176 shutdown_connections 2023-10-10T19:07:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.266+0000 7f4233b1a700 1 --2- 172.21.15.175:0/4016829176 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c0c12a0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.266+0000 7f4233b1a700 1 --2- 172.21.15.175:0/4016829176 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c1577a0 0x7f422c0c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.266+0000 7f4233b1a700 1 -- 172.21.15.175:0/4016829176 >> 172.21.15.175:0/4016829176 conn(0x7f422c06dc70 msgr2=0x7f422c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:18.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.266+0000 7f4233b1a700 1 -- 172.21.15.175:0/4016829176 shutdown_connections 2023-10-10T19:07:18.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.266+0000 7f4233b1a700 1 -- 172.21.15.175:0/4016829176 wait complete. 2023-10-10T19:07:18.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.267+0000 7f4233b1a700 1 Processor -- start 2023-10-10T19:07:18.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.267+0000 7f4233b1a700 1 -- start start 2023-10-10T19:07:18.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f4233b1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c0bf6e0 0x7f422c147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:18.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f4233b1a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:18.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f4233b1a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f422c1577a0 0x7f422c158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:18.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f4233b1a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c1639a0 con 0x7f422c1573d0 2023-10-10T19:07:18.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f4233b1a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c1584f0 con 0x7f422c1577a0 2023-10-10T19:07:18.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f4233b1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c1480d0 con 0x7f422c0bf6e0 2023-10-10T19:07:18.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:18.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c15ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39402/0 (socket says 172.21.15.175:39402) 2023-10-10T19:07:18.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42318b6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c0bf6e0 0x7f422c147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:18.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 -- 172.21.15.175:0/1003033495 learned_addr learned my addr 172.21.15.175:0/1003033495 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:18.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42320b7700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f422c1577a0 0x7f422c158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:18.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 -- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c0bf6e0 msgr2=0x7f422c147b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:18.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c0bf6e0 0x7f422c147b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 -- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f422c1577a0 msgr2=0x7f422c158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:18.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f422c1577a0 0x7f422c158980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 -- 172.21.15.175:0/1003033495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4228009420 con 0x7f422c1573d0 2023-10-10T19:07:18.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.268+0000 7f42310b5700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c15ac40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d8c0 tx=0x7f421c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:18.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.269+0000 7f4221ffb700 1 -- 172.21.15.175:0/1003033495 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c012630 con 0x7f422c1573d0 2023-10-10T19:07:18.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.269+0000 7f4221ffb700 1 -- 172.21.15.175:0/1003033495 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c012790 con 0x7f422c1573d0 2023-10-10T19:07:18.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.269+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c163b90 con 0x7f422c1573d0 2023-10-10T19:07:18.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.269+0000 7f4221ffb700 1 -- 172.21.15.175:0/1003033495 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c012b30 con 0x7f422c1573d0 2023-10-10T19:07:18.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.269+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c164150 con 0x7f422c1573d0 2023-10-10T19:07:18.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.270+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c067570 con 0x7f422c1573d0 2023-10-10T19:07:18.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.270+0000 7f4221ffb700 1 -- 172.21.15.175:0/1003033495 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f421c01d400 con 0x7f422c1573d0 2023-10-10T19:07:18.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.270+0000 7f4221ffb700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4210066f30 0x7f42100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:18.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.270+0000 7f4221ffb700 1 -- 172.21.15.175:0/1003033495 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f421c084be0 con 0x7f422c1573d0 2023-10-10T19:07:18.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.271+0000 7f42318b6700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4210066f30 0x7f42100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:18.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.271+0000 7f42318b6700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4210066f30 0x7f42100693e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f42280091e0 tx=0x7f4228017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:18.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.275+0000 7f4221ffb700 1 -- 172.21.15.175:0/1003033495 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f421c051a80 con 0x7f422c1573d0 2023-10-10T19:07:18.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.500+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f422c142810 con 0x7f422c1573d0 2023-10-10T19:07:18.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.500+0000 7f4221ffb700 1 -- 172.21.15.175:0/1003033495 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6207 (secure 0 0 0) 0x7f421c0550a0 con 0x7f422c1573d0 2023-10-10T19:07:18.502 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:18.502 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:17.678034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:18.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4210066f30 msgr2=0x7f42100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:18.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4210066f30 0x7f42100693e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f42280091e0 tx=0x7f4228017000 comp rx=0 tx=0).stop 2023-10-10T19:07:18.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 msgr2=0x7f422c15ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:18.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c15ac40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d8c0 tx=0x7f421c008040 comp rx=0 tx=0).stop 2023-10-10T19:07:18.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 shutdown_connections 2023-10-10T19:07:18.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4210066f30 0x7f42100693e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f422c1577a0 0x7f422c158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f422c0bf6e0 0x7f422c147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 --2- 172.21.15.175:0/1003033495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f422c1573d0 0x7f422c15ac40 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:18.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.502+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 >> 172.21.15.175:0/1003033495 conn(0x7f422c06dc70 msgr2=0x7f422c146520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:18.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.503+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 shutdown_connections 2023-10-10T19:07:18.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:18.503+0000 7f4233b1a700 1 -- 172.21.15.175:0/1003033495 wait complete. 2023-10-10T19:07:18.507 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 166 2023-10-10T19:07:18.516 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:07:20.516 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:20.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.643+0000 7fd01d145700 1 Processor -- start 2023-10-10T19:07:20.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.643+0000 7fd01d145700 1 -- start start 2023-10-10T19:07:20.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd01d145700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b1fc0 0x7fd018155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:20.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd01d145700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd0180b2390 0x7fd018068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:20.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd01d145700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0181574b0 con 0x7fd018068bd0 2023-10-10T19:07:20.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd01d145700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd018150770 con 0x7fd0180b2390 2023-10-10T19:07:20.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd01d145700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0180afad0 con 0x7fd0180b1fc0 2023-10-10T19:07:20.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd01659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd0180b2390 0x7fd018068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:20.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd01659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd0180b2390 0x7fd018068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39410/0 (socket says 172.21.15.175:39410) 2023-10-10T19:07:20.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd01659c700 1 -- 172.21.15.175:0/1208994811 learned_addr learned my addr 172.21.15.175:0/1208994811 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:20.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.644+0000 7fd016d9d700 1 --2- 172.21.15.175:0/1208994811 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b1fc0 0x7fd018155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:20.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01659c700 1 -- 172.21.15.175:0/1208994811 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b1fc0 msgr2=0x7fd018155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:20.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01659c700 1 --2- 172.21.15.175:0/1208994811 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b1fc0 0x7fd018155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01659c700 1 -- 172.21.15.175:0/1208994811 >> v1:172.21.15.191:6789/0 conn(0x7fd018068bd0 legacy=0x7fd0180aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:20.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01659c700 1 -- 172.21.15.175:0/1208994811 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0180af4c0 con 0x7fd0180b2390 2023-10-10T19:07:20.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01659c700 1 --2- 172.21.15.175:0/1208994811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180b2390 0x7fd018068690 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0099b0 tx=0x7fd00c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4682b935c69138e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:20.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd015d9b700 1 -- 172.21.15.175:0/1208994811 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c009ea0 con 0x7fd0180b2390 2023-10-10T19:07:20.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd015d9b700 1 -- 172.21.15.175:0/1208994811 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c00b8f0 con 0x7fd0180b2390 2023-10-10T19:07:20.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd015d9b700 1 -- 172.21.15.175:0/1208994811 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c00bc90 con 0x7fd0180b2390 2023-10-10T19:07:20.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01d145700 1 -- 172.21.15.175:0/1208994811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180b2390 msgr2=0x7fd018068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:20.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01d145700 1 --2- 172.21.15.175:0/1208994811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180b2390 0x7fd018068690 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0099b0 tx=0x7fd00c009250 comp rx=0 tx=0).stop 2023-10-10T19:07:20.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01d145700 1 -- 172.21.15.175:0/1208994811 shutdown_connections 2023-10-10T19:07:20.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01d145700 1 --2- 172.21.15.175:0/1208994811 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b1fc0 0x7fd018155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01d145700 1 --2- 172.21.15.175:0/1208994811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180b2390 0x7fd018068690 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01d145700 1 -- 172.21.15.175:0/1208994811 >> 172.21.15.175:0/1208994811 conn(0x7fd018063230 msgr2=0x7fd018063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:20.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01d145700 1 -- 172.21.15.175:0/1208994811 shutdown_connections 2023-10-10T19:07:20.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.645+0000 7fd01d145700 1 -- 172.21.15.175:0/1208994811 wait complete. 2023-10-10T19:07:20.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd01d145700 1 Processor -- start 2023-10-10T19:07:20.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd01d145700 1 -- start start 2023-10-10T19:07:20.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd01d145700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd018068bd0 0x7fd0181598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:20.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd01d145700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0180b1fc0 0x7fd018159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:20.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd01d145700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b2390 0x7fd018165f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:20.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd01d145700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0180c4950 con 0x7fd018068bd0 2023-10-10T19:07:20.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd01d145700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0181550a0 con 0x7fd0180b1fc0 2023-10-10T19:07:20.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd01d145700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01816a110 con 0x7fd0180b2390 2023-10-10T19:07:20.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd016d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd018068bd0 0x7fd0181598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:20.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd016d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd018068bd0 0x7fd0181598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39426/0 (socket says 172.21.15.175:39426) 2023-10-10T19:07:20.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.647+0000 7fd016d9d700 1 -- 172.21.15.175:0/400651002 learned_addr learned my addr 172.21.15.175:0/400651002 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:20.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd01659c700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0180b1fc0 0x7fd018159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:20.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd01759e700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b2390 0x7fd018165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:20.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd016d9d700 1 -- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b2390 msgr2=0x7fd018165f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:20.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd016d9d700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b2390 0x7fd018165f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd016d9d700 1 -- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0180b1fc0 msgr2=0x7fd018159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:20.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd016d9d700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0180b1fc0 0x7fd018159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd016d9d700 1 -- 172.21.15.175:0/400651002 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c0094f0 con 0x7fd018068bd0 2023-10-10T19:07:20.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd01759e700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b2390 0x7fd018165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:07:20.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd016d9d700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd018068bd0 0x7fd0181598c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd008009ef0 tx=0x7fd00800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:20.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fcffeffd700 1 -- 172.21.15.175:0/400651002 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00800cc50 con 0x7fd018068bd0 2023-10-10T19:07:20.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fcffeffd700 1 -- 172.21.15.175:0/400651002 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00800f040 con 0x7fd018068bd0 2023-10-10T19:07:20.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0181552d0 con 0x7fd018068bd0 2023-10-10T19:07:20.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fcffeffd700 1 -- 172.21.15.175:0/400651002 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd008014480 con 0x7fd018068bd0 2023-10-10T19:07:20.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.648+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0181697e0 con 0x7fd018068bd0 2023-10-10T19:07:20.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.649+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd018151530 con 0x7fd018068bd0 2023-10-10T19:07:20.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.649+0000 7fcffeffd700 1 -- 172.21.15.175:0/400651002 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd0080147f0 con 0x7fd018068bd0 2023-10-10T19:07:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.650+0000 7fcffeffd700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd004066f60 0x7fd004069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.650+0000 7fcffeffd700 1 -- 172.21.15.175:0/400651002 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fd008085370 con 0x7fd018068bd0 2023-10-10T19:07:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.650+0000 7fd01659c700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd004066f60 0x7fd004069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.650+0000 7fd01659c700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd004066f60 0x7fd004069410 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0099b0 tx=0x7fd00c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.654+0000 7fcffeffd700 1 -- 172.21.15.175:0/400651002 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd008052210 con 0x7fd018068bd0 2023-10-10T19:07:20.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.882+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd018067570 con 0x7fd018068bd0 2023-10-10T19:07:20.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.882+0000 7fcffeffd700 1 -- 172.21.15.175:0/400651002 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6207 (secure 0 0 0) 0x7fd008059e60 con 0x7fd018068bd0 2023-10-10T19:07:20.884 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:20.884 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:17.678034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:20.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd004066f60 msgr2=0x7fd004069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:20.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd004066f60 0x7fd004069410 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0099b0 tx=0x7fd00c017000 comp rx=0 tx=0).stop 2023-10-10T19:07:20.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd018068bd0 msgr2=0x7fd0181598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:20.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd018068bd0 0x7fd0181598c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd008009ef0 tx=0x7fd00800b040 comp rx=0 tx=0).stop 2023-10-10T19:07:20.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 shutdown_connections 2023-10-10T19:07:20.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd004066f60 0x7fd004069410 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0180b2390 0x7fd018165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd018068bd0 0x7fd0181598c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 --2- 172.21.15.175:0/400651002 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0180b1fc0 0x7fd018159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:20.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 >> 172.21.15.175:0/400651002 conn(0x7fd018063230 msgr2=0x7fd01814fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:20.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 shutdown_connections 2023-10-10T19:07:20.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:20.884+0000 7fd01d145700 1 -- 172.21.15.175:0/400651002 wait complete. 2023-10-10T19:07:20.888 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 166 2023-10-10T19:07:20.897 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:07:21.784 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:07:21.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:07:22.898 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:23.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35f10f5700 1 Processor -- start 2023-10-10T19:07:23.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35f10f5700 1 -- start start 2023-10-10T19:07:23.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35f10f5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec0b2390 0x7f35ec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:23.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35f10f5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec1574b0 con 0x7f35ec0b1fc0 2023-10-10T19:07:23.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35f10f5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec0afad0 con 0x7f35ec0aff60 2023-10-10T19:07:23.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35f10f5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec150770 con 0x7f35ec0b2390 2023-10-10T19:07:23.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35ead9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f35ec0b1fc0 0x7f35ec068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51824/0 (socket says 172.21.15.175:51824) 2023-10-10T19:07:23.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35ead9d700 1 -- 172.21.15.175:0/3649434341 learned_addr learned my addr 172.21.15.175:0/3649434341 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:23.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35ea59c700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec0b2390 0x7f35ec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:23.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602736682 0 0) 0x7f35ec1574b0 con 0x7f35ec0b1fc0 2023-10-10T19:07:23.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d8003570 con 0x7f35ec0b1fc0 2023-10-10T19:07:23.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098723565 0 0) 0x7f35ec0afad0 con 0x7f35ec0aff60 2023-10-10T19:07:23.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ec1574b0 con 0x7f35ec0aff60 2023-10-10T19:07:23.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2481911577 0 0) 0x7f35d8003570 con 0x7f35ec0b1fc0 2023-10-10T19:07:23.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec0b2390 msgr2=0x7f35ec155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:23.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec0b2390 0x7f35ec155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 >> v1:172.21.15.191:6789/0 conn(0x7f35ec0aff60 legacy=0x7f35ec159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:23.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec0af4c0 con 0x7f35ec0b1fc0 2023-10-10T19:07:23.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f35e40017f0 con 0x7f35ec0b1fc0 2023-10-10T19:07:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 >> v1:172.21.15.175:6789/0 conn(0x7f35ec0b1fc0 legacy=0x7f35ec068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d8003c20 0x7f35d8004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d8004810 0x7f35d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8005210 0x7f35d800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec150770 con 0x7f35d8004810 2023-10-10T19:07:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d800c0b0 con 0x7f35d8003c20 2023-10-10T19:07:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d800c270 con 0x7f35d8005210 2023-10-10T19:07:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35ea59c700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec0b2390 0x7f35ec155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:07:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35ead9d700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d8003c20 0x7f35d8004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.028+0000 7f35ea59c700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d8004810 0x7f35d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:23.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35eb59e700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8005210 0x7f35d800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:23.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35ea59c700 1 -- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8005210 msgr2=0x7f35d800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:23.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35ea59c700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8005210 0x7f35d800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35ea59c700 1 -- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d8003c20 msgr2=0x7f35d8004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:23.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35ea59c700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d8003c20 0x7f35d8004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35ea59c700 1 -- 172.21.15.175:0/3649434341 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8003570 con 0x7f35d8004810 2023-10-10T19:07:23.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35ea59c700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d8004810 0x7f35d8004c80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f35ec064470 tx=0x7f35d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a7adf09a7c1af9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:23.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d400e710 con 0x7f35d8004810 2023-10-10T19:07:23.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d400e870 con 0x7f35d8004810 2023-10-10T19:07:23.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.029+0000 7f35e9d9b700 1 -- 172.21.15.175:0/3649434341 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d400ec10 con 0x7f35d8004810 2023-10-10T19:07:23.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.030+0000 7f35f10f5700 1 -- 172.21.15.175:0/3649434341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d8004810 msgr2=0x7f35d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:23.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.030+0000 7f35f10f5700 1 --2- 172.21.15.175:0/3649434341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d8004810 0x7f35d8004c80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f35ec064470 tx=0x7f35d40091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.030+0000 7f35eb59e700 1 -- 172.21.15.175:0/3649434341 reap_dead start 2023-10-10T19:07:23.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.030+0000 7f35f10f5700 1 -- 172.21.15.175:0/3649434341 shutdown_connections 2023-10-10T19:07:23.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.030+0000 7f35f10f5700 1 -- 172.21.15.175:0/3649434341 >> 172.21.15.175:0/3649434341 conn(0x7f35ec063230 msgr2=0x7f35ec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:23.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.030+0000 7f35f10f5700 1 -- 172.21.15.175:0/3649434341 shutdown_connections 2023-10-10T19:07:23.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.030+0000 7f35f10f5700 1 -- 172.21.15.175:0/3649434341 wait complete. 2023-10-10T19:07:23.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.031+0000 7f35f10f5700 1 Processor -- start 2023-10-10T19:07:23.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.031+0000 7f35f10f5700 1 -- start start 2023-10-10T19:07:23.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35f10f5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8004810 0x7f35ec0c0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:23.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35f10f5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec156400 0x7f35ec156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:23.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35f10f5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35ec14ede0 0x7f35ec14f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:23.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35f10f5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec0bfcb0 con 0x7f35ec14ede0 2023-10-10T19:07:23.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35f10f5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec0c2060 con 0x7f35ec156400 2023-10-10T19:07:23.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35f10f5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec156db0 con 0x7f35d8004810 2023-10-10T19:07:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35ec14ede0 0x7f35ec14f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35ec14ede0 0x7f35ec14f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39440/0 (socket says 172.21.15.175:39440) 2023-10-10T19:07:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 -- 172.21.15.175:0/2182017731 learned_addr learned my addr 172.21.15.175:0/2182017731 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35ea59c700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec156400 0x7f35ec156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35ead9d700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8004810 0x7f35ec0c0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 -- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8004810 msgr2=0x7f35ec0c0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8004810 0x7f35ec0c0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 -- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec156400 msgr2=0x7f35ec156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec156400 0x7f35ec156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 -- 172.21.15.175:0/2182017731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec14f770 con 0x7f35ec14ede0 2023-10-10T19:07:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35ead9d700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8004810 0x7f35ec0c0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35eb59e700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35ec14ede0 0x7f35ec14f230 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f35ec08a030 tx=0x7f35dc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35e2ffd700 1 -- 172.21.15.175:0/2182017731 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc006a30 con 0x7f35ec14ede0 2023-10-10T19:07:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35e2ffd700 1 -- 172.21.15.175:0/2182017731 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc006b90 con 0x7f35ec14ede0 2023-10-10T19:07:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec0c34e0 con 0x7f35ec14ede0 2023-10-10T19:07:23.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35e2ffd700 1 -- 172.21.15.175:0/2182017731 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc007030 con 0x7f35ec14ede0 2023-10-10T19:07:23.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.032+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec155c00 con 0x7f35ec14ede0 2023-10-10T19:07:23.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.033+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec067570 con 0x7f35ec14ede0 2023-10-10T19:07:23.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.034+0000 7f35e2ffd700 1 -- 172.21.15.175:0/2182017731 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f35dc0073e0 con 0x7f35ec14ede0 2023-10-10T19:07:23.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.034+0000 7f35e2ffd700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35cc066f30 0x7f35cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:23.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.034+0000 7f35e2ffd700 1 -- 172.21.15.175:0/2182017731 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f35dc0844b0 con 0x7f35ec14ede0 2023-10-10T19:07:23.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.034+0000 7f35ead9d700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35cc066f30 0x7f35cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:23.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.035+0000 7f35ead9d700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35cc066f30 0x7f35cc0693e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001190 tx=0x7f35e4006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:23.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.038+0000 7f35e2ffd700 1 -- 172.21.15.175:0/2182017731 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35dc051350 con 0x7f35ec14ede0 2023-10-10T19:07:23.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.265+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35ec068da0 con 0x7f35ec14ede0 2023-10-10T19:07:23.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.266+0000 7f35e2ffd700 1 -- 172.21.15.175:0/2182017731 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6207 (secure 0 0 0) 0x7f35dc054970 con 0x7f35ec14ede0 2023-10-10T19:07:23.267 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:23.268 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:17.678034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":5367,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5367":{"gid":5367,"name":"c","rank":2,"incarnation":70,"state":"up:active","state_seq":17,"addr":"172.21.15.175:6836/2375829435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2375829435},{"type":"v1","addr":"172.21.15.175:6836","nonce":2375829435}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:23.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35cc066f30 msgr2=0x7f35cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:23.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35cc066f30 0x7f35cc0693e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001190 tx=0x7f35e4006040 comp rx=0 tx=0).stop 2023-10-10T19:07:23.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35ec14ede0 msgr2=0x7f35ec14f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:23.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35ec14ede0 0x7f35ec14f230 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f35ec08a030 tx=0x7f35dc007d30 comp rx=0 tx=0).stop 2023-10-10T19:07:23.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 shutdown_connections 2023-10-10T19:07:23.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35cc066f30 0x7f35cc0693e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35ec14ede0 0x7f35ec14f230 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d8004810 0x7f35ec0c0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 --2- 172.21.15.175:0/2182017731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec156400 0x7f35ec156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:23.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 >> 172.21.15.175:0/2182017731 conn(0x7f35ec063230 msgr2=0x7f35ec158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:23.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 shutdown_connections 2023-10-10T19:07:23.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:23.268+0000 7f35f10f5700 1 -- 172.21.15.175:0/2182017731 wait complete. 2023-10-10T19:07:23.272 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 166 2023-10-10T19:07:23.281 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:07:25.282 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:25.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.409+0000 7f677ca4c700 1 Processor -- start 2023-10-10T19:07:25.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.409+0000 7f677ca4c700 1 -- start start 2023-10-10T19:07:25.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f677ca4c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6778155fb0 0x7f6778159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f677ca4c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67781574b0 con 0x7f67780b2390 2023-10-10T19:07:25.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f677ca4c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67780afad0 con 0x7f67780b1fc0 2023-10-10T19:07:25.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f677ca4c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6778150770 con 0x7f6778155fb0 2023-10-10T19:07:25.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f6776d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6778155fb0 0x7f6778159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f6775d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f67780b2390 0x7f67780aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:57886/0 (socket says 172.21.15.175:57886) 2023-10-10T19:07:25.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f6775d9b700 1 -- 172.21.15.175:0/3668336778 learned_addr learned my addr 172.21.15.175:0/3668336778 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:25.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f6776d9d700 1 -- 172.21.15.175:0/3668336778 >> v1:172.21.15.191:6789/0 conn(0x7f67780b1fc0 legacy=0x7f6778068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f6776d9d700 1 -- 172.21.15.175:0/3668336778 >> v1:172.21.15.175:6789/0 conn(0x7f67780b2390 legacy=0x7f67780aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.410+0000 7f6776d9d700 1 -- 172.21.15.175:0/3668336778 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67780af4c0 con 0x7f6778155fb0 2023-10-10T19:07:25.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f6776d9d700 1 --2- 172.21.15.175:0/3668336778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6778155fb0 0x7f6778159920 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6770009e60 tx=0x7f6770009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b404194c25078c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:25.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677559a700 1 -- 172.21.15.175:0/3668336778 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677000e070 con 0x7f6778155fb0 2023-10-10T19:07:25.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677559a700 1 -- 172.21.15.175:0/3668336778 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677000bb30 con 0x7f6778155fb0 2023-10-10T19:07:25.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677559a700 1 -- 172.21.15.175:0/3668336778 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770012430 con 0x7f6778155fb0 2023-10-10T19:07:25.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677ca4c700 1 -- 172.21.15.175:0/3668336778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6778155fb0 msgr2=0x7f6778159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677ca4c700 1 --2- 172.21.15.175:0/3668336778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6778155fb0 0x7f6778159920 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6770009e60 tx=0x7f6770009780 comp rx=0 tx=0).stop 2023-10-10T19:07:25.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677ca4c700 1 -- 172.21.15.175:0/3668336778 shutdown_connections 2023-10-10T19:07:25.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677ca4c700 1 --2- 172.21.15.175:0/3668336778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6778155fb0 0x7f6778159920 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677ca4c700 1 -- 172.21.15.175:0/3668336778 >> 172.21.15.175:0/3668336778 conn(0x7f6778063230 msgr2=0x7f6778063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:25.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677ca4c700 1 -- 172.21.15.175:0/3668336778 shutdown_connections 2023-10-10T19:07:25.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.411+0000 7f677ca4c700 1 -- 172.21.15.175:0/3668336778 wait complete. 2023-10-10T19:07:25.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.412+0000 7f677ca4c700 1 Processor -- start 2023-10-10T19:07:25.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677ca4c700 1 -- start start 2023-10-10T19:07:25.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677ca4c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f67780b1fc0 0x7f67780c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677ca4c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67780b2390 0x7f6778150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677ca4c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778155fb0 0x7f6778152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677ca4c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778166380 con 0x7f67780b1fc0 2023-10-10T19:07:25.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677ca4c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778166630 con 0x7f67780b2390 2023-10-10T19:07:25.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677ca4c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6778151590 con 0x7f6778155fb0 2023-10-10T19:07:25.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f67780b1fc0 0x7f67780c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f67780b1fc0 0x7f67780c3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36826/0 (socket says 172.21.15.175:36826) 2023-10-10T19:07:25.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f677659c700 1 -- 172.21.15.175:0/2084980968 learned_addr learned my addr 172.21.15.175:0/2084980968 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:25.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f6775d9b700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67780b2390 0x7f6778150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.413+0000 7f6776d9d700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778155fb0 0x7f6778152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f677659c700 1 -- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778155fb0 msgr2=0x7f6778152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f677659c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778155fb0 0x7f6778152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f677659c700 1 -- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67780b2390 msgr2=0x7f6778150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f677659c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67780b2390 0x7f6778150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f677659c700 1 -- 172.21.15.175:0/2084980968 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6770009a20 con 0x7f67780b1fc0 2023-10-10T19:07:25.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f6776d9d700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778155fb0 0x7f6778152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:25.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f677659c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f67780b1fc0 0x7f67780c3d00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f676800c590 tx=0x7f676800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:25.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f67667fc700 1 -- 172.21.15.175:0/2084980968 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768011590 con 0x7f67780b1fc0 2023-10-10T19:07:25.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f67667fc700 1 -- 172.21.15.175:0/2084980968 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6768011710 con 0x7f67780b1fc0 2023-10-10T19:07:25.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6778151870 con 0x7f67780b1fc0 2023-10-10T19:07:25.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f67667fc700 1 -- 172.21.15.175:0/2084980968 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768011ad0 con 0x7f67780b1fc0 2023-10-10T19:07:25.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.414+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6778151060 con 0x7f67780b1fc0 2023-10-10T19:07:25.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.415+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67781513f0 con 0x7f67780b1fc0 2023-10-10T19:07:25.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.415+0000 7f67667fc700 1 -- 172.21.15.175:0/2084980968 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f676801b3b0 con 0x7f67780b1fc0 2023-10-10T19:07:25.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.416+0000 7f67667fc700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6758066f30 0x7f67580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.416+0000 7f67667fc700 1 -- 172.21.15.175:0/2084980968 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f6768083ee0 con 0x7f67780b1fc0 2023-10-10T19:07:25.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.416+0000 7f6775d9b700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6758066f30 0x7f67580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.416+0000 7f6775d9b700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6758066f30 0x7f67580693e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f67780b0b00 tx=0x7f676c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:25.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.420+0000 7f67667fc700 1 -- 172.21.15.175:0/2084980968 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f67781513f0 con 0x7f67780b1fc0 2023-10-10T19:07:25.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.643+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67781513f0 con 0x7f67780b1fc0 2023-10-10T19:07:25.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.644+0000 7f67667fc700 1 -- 172.21.15.175:0/2084980968 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f67781513f0 con 0x7f67780b1fc0 2023-10-10T19:07:25.644 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:25.644 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:23.644027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:25.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6758066f30 msgr2=0x7f67580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6758066f30 0x7f67580693e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f67780b0b00 tx=0x7f676c009040 comp rx=0 tx=0).stop 2023-10-10T19:07:25.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f67780b1fc0 msgr2=0x7f67780c3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f67780b1fc0 0x7f67780c3d00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f676800c590 tx=0x7f676800c8a0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 shutdown_connections 2023-10-10T19:07:25.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6758066f30 0x7f67580693e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778155fb0 0x7f6778152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f67780b1fc0 0x7f67780c3d00 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 --2- 172.21.15.175:0/2084980968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67780b2390 0x7f6778150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 >> 172.21.15.175:0/2084980968 conn(0x7f6778063230 msgr2=0x7f677814fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:25.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 shutdown_connections 2023-10-10T19:07:25.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.646+0000 7f677ca4c700 1 -- 172.21.15.175:0/2084980968 wait complete. 2023-10-10T19:07:25.649 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 167 2023-10-10T19:07:25.658 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-10T19:07:25.658 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:07:25.658 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:25.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.774+0000 7f9ba9240700 1 Processor -- start 2023-10-10T19:07:25.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.774+0000 7f9ba9240700 1 -- start start 2023-10-10T19:07:25.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba9240700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b2390 0x7f9ba4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba9240700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9ba40aff60 0x7f9ba40b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba9240700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba41574b0 con 0x7f9ba40b1fc0 2023-10-10T19:07:25.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba9240700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba4150770 con 0x7f9ba40aff60 2023-10-10T19:07:25.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba9240700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba40afad0 con 0x7f9ba40b2390 2023-10-10T19:07:25.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9b9bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b2390 0x7f9ba4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba359e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9ba40aff60 0x7f9ba40b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba2d9d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f9ba40b1fc0 0x7f9ba4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:46960/0 (socket says 172.21.15.175:46960) 2023-10-10T19:07:25.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba2d9d700 1 -- 172.21.15.175:0/1560994562 learned_addr learned my addr 172.21.15.175:0/1560994562 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:25.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba359e700 1 -- 172.21.15.175:0/1560994562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b2390 msgr2=0x7f9ba4155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:25.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba359e700 1 --2- 172.21.15.175:0/1560994562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b2390 0x7f9ba4155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba359e700 1 -- 172.21.15.175:0/1560994562 >> v1:172.21.15.191:6789/0 conn(0x7f9ba40b1fc0 legacy=0x7f9ba4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba359e700 1 -- 172.21.15.175:0/1560994562 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba40af4c0 con 0x7f9ba40aff60 2023-10-10T19:07:25.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9b9bfff700 1 --2- 172.21.15.175:0/1560994562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b2390 0x7f9ba4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:07:25.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.775+0000 7f9ba359e700 1 --2- 172.21.15.175:0/1560994562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40aff60 0x7f9ba40b03e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009e90 tx=0x7f9b9c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=723a8009f46fde4e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:25.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba259c700 1 -- 172.21.15.175:0/1560994562 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c00e070 con 0x7f9ba40aff60 2023-10-10T19:07:25.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba259c700 1 -- 172.21.15.175:0/1560994562 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c00bc80 con 0x7f9ba40aff60 2023-10-10T19:07:25.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba259c700 1 -- 172.21.15.175:0/1560994562 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c012430 con 0x7f9ba40aff60 2023-10-10T19:07:25.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba9240700 1 -- 172.21.15.175:0/1560994562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40aff60 msgr2=0x7f9ba40b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1560994562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40aff60 0x7f9ba40b03e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009e90 tx=0x7f9b9c009780 comp rx=0 tx=0).stop 2023-10-10T19:07:25.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba9240700 1 -- 172.21.15.175:0/1560994562 shutdown_connections 2023-10-10T19:07:25.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1560994562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40aff60 0x7f9ba40b03e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1560994562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b2390 0x7f9ba4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba9240700 1 -- 172.21.15.175:0/1560994562 >> 172.21.15.175:0/1560994562 conn(0x7f9ba4063230 msgr2=0x7f9ba4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:25.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba9240700 1 -- 172.21.15.175:0/1560994562 shutdown_connections 2023-10-10T19:07:25.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.776+0000 7f9ba9240700 1 -- 172.21.15.175:0/1560994562 wait complete. 2023-10-10T19:07:25.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.777+0000 7f9ba9240700 1 Processor -- start 2023-10-10T19:07:25.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.777+0000 7f9ba9240700 1 -- start start 2023-10-10T19:07:25.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba9240700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40aff60 0x7f9ba4165f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba9240700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b1fc0 0x7f9ba40c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba9240700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40b2390 0x7f9ba414fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba9240700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba41664d0 con 0x7f9ba40aff60 2023-10-10T19:07:25.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba9240700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba40c1290 con 0x7f9ba40b2390 2023-10-10T19:07:25.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba9240700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba40c2bc0 con 0x7f9ba40b1fc0 2023-10-10T19:07:25.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba2d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40aff60 0x7f9ba4165f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba2d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40aff60 0x7f9ba4165f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36828/0 (socket says 172.21.15.175:36828) 2023-10-10T19:07:25.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba2d9d700 1 -- 172.21.15.175:0/1450555446 learned_addr learned my addr 172.21.15.175:0/1450555446 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:25.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9b9bfff700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b1fc0 0x7f9ba40c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.778+0000 7f9ba359e700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40b2390 0x7f9ba414fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9ba2d9d700 1 -- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b1fc0 msgr2=0x7f9ba40c1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9ba2d9d700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b1fc0 0x7f9ba40c1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9ba2d9d700 1 -- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40b2390 msgr2=0x7f9ba414fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:25.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9ba2d9d700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40b2390 0x7f9ba414fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:25.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9ba2d9d700 1 -- 172.21.15.175:0/1450555446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c009420 con 0x7f9ba40aff60 2023-10-10T19:07:25.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9ba2d9d700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40aff60 0x7f9ba4165f90 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c7c0 tx=0x7f9b94007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:25.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9b9affd700 1 -- 172.21.15.175:0/1450555446 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94005b30 con 0x7f9ba40aff60 2023-10-10T19:07:25.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9b9affd700 1 -- 172.21.15.175:0/1450555446 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94005c90 con 0x7f9ba40aff60 2023-10-10T19:07:25.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c009a20 con 0x7f9ba40aff60 2023-10-10T19:07:25.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9b9affd700 1 -- 172.21.15.175:0/1450555446 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94004a80 con 0x7f9ba40aff60 2023-10-10T19:07:25.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.779+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba40c3140 con 0x7f9ba40aff60 2023-10-10T19:07:25.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.780+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba4067570 con 0x7f9ba40aff60 2023-10-10T19:07:25.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.780+0000 7f9b9affd700 1 -- 172.21.15.175:0/1450555446 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9b940070d0 con 0x7f9ba40aff60 2023-10-10T19:07:25.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.780+0000 7f9b9affd700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b84066f60 0x7f9b84069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:25.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.781+0000 7f9b9affd700 1 -- 172.21.15.175:0/1450555446 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f9b94083ce0 con 0x7f9ba40aff60 2023-10-10T19:07:25.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.781+0000 7f9b9bfff700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b84066f60 0x7f9b84069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:25.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.781+0000 7f9b9bfff700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b84066f60 0x7f9b84069410 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9ba41505f0 tx=0x7f9b8c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:25.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:25.785+0000 7f9b9affd700 1 -- 172.21.15.175:0/1450555446 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b9404b5a0 con 0x7f9ba40aff60 2023-10-10T19:07:26.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.012+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ba4068da0 con 0x7f9ba40aff60 2023-10-10T19:07:26.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.013+0000 7f9b9affd700 1 -- 172.21.15.175:0/1450555446 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f9b94050a90 con 0x7f9ba40aff60 2023-10-10T19:07:26.013 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:26.013 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:23.644027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:26.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b84066f60 msgr2=0x7f9b84069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:26.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b84066f60 0x7f9b84069410 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9ba41505f0 tx=0x7f9b8c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:26.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40aff60 msgr2=0x7f9ba4165f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:26.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40aff60 0x7f9ba4165f90 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c7c0 tx=0x7f9b94007f70 comp rx=0 tx=0).stop 2023-10-10T19:07:26.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 shutdown_connections 2023-10-10T19:07:26.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b84066f60 0x7f9b84069410 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:26.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ba40b2390 0x7f9ba414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:26.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ba40aff60 0x7f9ba4165f90 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:26.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 --2- 172.21.15.175:0/1450555446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9ba40b1fc0 0x7f9ba40c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:26.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 >> 172.21.15.175:0/1450555446 conn(0x7f9ba4063230 msgr2=0x7f9ba41508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:26.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 shutdown_connections 2023-10-10T19:07:26.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:26.015+0000 7f9ba9240700 1 -- 172.21.15.175:0/1450555446 wait complete. 2023-10-10T19:07:26.019 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 167 2023-10-10T19:07:26.028 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 167, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 167, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5552": { "addr": "172.21.15.191:6839/1910071809", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1910071809, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1910071809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5552, "incarnation": 154, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 36 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:07:23.644027+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 6106, "mds_3": 5552, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:07:28.029 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:28.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.154+0000 7fcc5a9be700 1 Processor -- start 2023-10-10T19:07:28.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.154+0000 7fcc5a9be700 1 -- start start 2023-10-10T19:07:28.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.154+0000 7fcc5a9be700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcc540c5a10 0x7fcc540c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:28.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.154+0000 7fcc5a9be700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc54152b20 0x7fcc54152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:28.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.154+0000 7fcc5a9be700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc540c1240 con 0x7fcc540c4f20 2023-10-10T19:07:28.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.154+0000 7fcc5a9be700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5406fd00 con 0x7fcc540c5a10 2023-10-10T19:07:28.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.154+0000 7fcc5a9be700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540c6ce0 con 0x7fcc54152b20 2023-10-10T19:07:28.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc53fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcc540c5a10 0x7fcc540c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:28.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc53fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcc540c5a10 0x7fcc540c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36838/0 (socket says 172.21.15.175:36838) 2023-10-10T19:07:28.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc53fff700 1 -- 172.21.15.175:0/807586871 learned_addr learned my addr 172.21.15.175:0/807586871 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:28.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc58f5b700 1 --2- 172.21.15.175:0/807586871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc54152b20 0x7fcc54152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:28.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc53fff700 1 -- 172.21.15.175:0/807586871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc54152b20 msgr2=0x7fcc54152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:28.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc53fff700 1 --2- 172.21.15.175:0/807586871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc54152b20 0x7fcc54152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc53fff700 1 -- 172.21.15.175:0/807586871 >> v1:172.21.15.191:6789/0 conn(0x7fcc540c4f20 legacy=0x7fcc54156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:28.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc53fff700 1 -- 172.21.15.175:0/807586871 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc540c1490 con 0x7fcc540c5a10 2023-10-10T19:07:28.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc53fff700 1 --2- 172.21.15.175:0/807586871 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc540c5de0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fcc440099b0 tx=0x7fcc44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6483853c5b34abb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:28.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc52ffd700 1 -- 172.21.15.175:0/807586871 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44009ea0 con 0x7fcc540c5a10 2023-10-10T19:07:28.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc52ffd700 1 -- 172.21.15.175:0/807586871 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4400b860 con 0x7fcc540c5a10 2023-10-10T19:07:28.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.155+0000 7fcc52ffd700 1 -- 172.21.15.175:0/807586871 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4400bc00 con 0x7fcc540c5a10 2023-10-10T19:07:28.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.156+0000 7fcc5a9be700 1 -- 172.21.15.175:0/807586871 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 msgr2=0x7fcc540c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:28.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.156+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/807586871 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc540c5de0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fcc440099b0 tx=0x7fcc44009250 comp rx=0 tx=0).stop 2023-10-10T19:07:28.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.156+0000 7fcc5a9be700 1 -- 172.21.15.175:0/807586871 shutdown_connections 2023-10-10T19:07:28.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.156+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/807586871 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc54152b20 0x7fcc54152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.156+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/807586871 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc540c5de0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.156+0000 7fcc5a9be700 1 -- 172.21.15.175:0/807586871 >> 172.21.15.175:0/807586871 conn(0x7fcc5406dc70 msgr2=0x7fcc54063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:28.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.156+0000 7fcc5a9be700 1 -- 172.21.15.175:0/807586871 shutdown_connections 2023-10-10T19:07:28.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.156+0000 7fcc5a9be700 1 -- 172.21.15.175:0/807586871 wait complete. 2023-10-10T19:07:28.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.157+0000 7fcc5a9be700 1 Processor -- start 2023-10-10T19:07:28.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.157+0000 7fcc5a9be700 1 -- start start 2023-10-10T19:07:28.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc5a9be700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc540c4f20 0x7fcc540c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:28.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc5a9be700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc5414ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:28.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc5a9be700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc54152b20 0x7fcc5414efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:28.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc5a9be700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54166e20 con 0x7fcc540c5a10 2023-10-10T19:07:28.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc5a9be700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5414e5d0 con 0x7fcc54152b20 2023-10-10T19:07:28.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc5a9be700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc541662a0 con 0x7fcc540c4f20 2023-10-10T19:07:28.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc5414ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:28.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc5414ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36846/0 (socket says 172.21.15.175:36846) 2023-10-10T19:07:28.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 -- 172.21.15.175:0/1662724766 learned_addr learned my addr 172.21.15.175:0/1662724766 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:28.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc53fff700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc540c4f20 0x7fcc540c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:28.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc58f5b700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc54152b20 0x7fcc5414efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:28.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 -- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc540c4f20 msgr2=0x7fcc540c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:28.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc540c4f20 0x7fcc540c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 -- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc54152b20 msgr2=0x7fcc5414efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:28.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc54152b20 0x7fcc5414efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 -- 172.21.15.175:0/1662724766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc440094f0 con 0x7fcc540c5a10 2023-10-10T19:07:28.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc537fe700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc5414ea60 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800c4e0 tx=0x7fcc4800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:28.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc3bfff700 1 -- 172.21.15.175:0/1662724766 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48011590 con 0x7fcc540c5a10 2023-10-10T19:07:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc3bfff700 1 -- 172.21.15.175:0/1662724766 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc48011710 con 0x7fcc540c5a10 2023-10-10T19:07:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc54166580 con 0x7fcc540c5a10 2023-10-10T19:07:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.158+0000 7fcc3bfff700 1 -- 172.21.15.175:0/1662724766 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48011ad0 con 0x7fcc540c5a10 2023-10-10T19:07:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.159+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc54166b40 con 0x7fcc540c5a10 2023-10-10T19:07:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.159+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5414e780 con 0x7fcc540c5a10 2023-10-10T19:07:28.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.160+0000 7fcc3bfff700 1 -- 172.21.15.175:0/1662724766 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcc4801c440 con 0x7fcc540c5a10 2023-10-10T19:07:28.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.160+0000 7fcc3bfff700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc40066f30 0x7fcc400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:28.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.160+0000 7fcc3bfff700 1 -- 172.21.15.175:0/1662724766 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7fcc48083c90 con 0x7fcc540c5a10 2023-10-10T19:07:28.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.160+0000 7fcc53fff700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc40066f30 0x7fcc400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:28.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.161+0000 7fcc53fff700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc40066f30 0x7fcc400693e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcc440099b0 tx=0x7fcc44017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:28.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.164+0000 7fcc3bfff700 1 -- 172.21.15.175:0/1662724766 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc5414e780 con 0x7fcc540c5a10 2023-10-10T19:07:28.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.387+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc5414e780 con 0x7fcc540c5a10 2023-10-10T19:07:28.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.388+0000 7fcc3bfff700 1 -- 172.21.15.175:0/1662724766 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5445 (secure 0 0 0) 0x7fcc5414e780 con 0x7fcc540c5a10 2023-10-10T19:07:28.389 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:28.389 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:23.644027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:28.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc40066f30 msgr2=0x7fcc400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:28.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc40066f30 0x7fcc400693e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcc440099b0 tx=0x7fcc44017000 comp rx=0 tx=0).stop 2023-10-10T19:07:28.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 msgr2=0x7fcc5414ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:28.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc5414ea60 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800c4e0 tx=0x7fcc4800c7f0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 shutdown_connections 2023-10-10T19:07:28.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcc40066f30 0x7fcc400693e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc54152b20 0x7fcc5414efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcc540c4f20 0x7fcc540c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 --2- 172.21.15.175:0/1662724766 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc540c5a10 0x7fcc5414ea60 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:28.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 >> 172.21.15.175:0/1662724766 conn(0x7fcc5406dc70 msgr2=0x7fcc54063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:28.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 shutdown_connections 2023-10-10T19:07:28.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:28.390+0000 7fcc5a9be700 1 -- 172.21.15.175:0/1662724766 wait complete. 2023-10-10T19:07:28.393 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 167 2023-10-10T19:07:28.403 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:30.404 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:30.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.532+0000 7f467a99a700 1 Processor -- start 2023-10-10T19:07:30.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.532+0000 7f467a99a700 1 -- start start 2023-10-10T19:07:30.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.533+0000 7f467a99a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4674068c00 0x7f4674159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:30.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.533+0000 7f467a99a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674067fa0 con 0x7f467408a100 2023-10-10T19:07:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.533+0000 7f467a99a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674156b70 con 0x7f467414fd90 2023-10-10T19:07:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.533+0000 7f467a99a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46740c5600 con 0x7f4674068c00 2023-10-10T19:07:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.533+0000 7f4679197700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f467408a100 0x7f46740684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:57890/0 (socket says 172.21.15.175:57890) 2023-10-10T19:07:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.533+0000 7f4679197700 1 -- 172.21.15.175:0/824841867 learned_addr learned my addr 172.21.15.175:0/824841867 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.533+0000 7f467a199700 1 --2- 172.21.15.175:0/824841867 >> v2:172.21.15.191:3300/0 conn(0x7f4674068c00 0x7f4674159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.533+0000 7f4678996700 1 -- 172.21.15.175:0/824841867 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074666192 0 0) 0x7f4674067fa0 con 0x7f467408a100 2023-10-10T19:07:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f4678996700 1 -- 172.21.15.175:0/824841867 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660003570 con 0x7f467408a100 2023-10-10T19:07:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f4678996700 1 -- 172.21.15.175:0/824841867 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700711425 0 0) 0x7f4674156b70 con 0x7f467414fd90 2023-10-10T19:07:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f4678996700 1 -- 172.21.15.175:0/824841867 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4674067fa0 con 0x7f467414fd90 2023-10-10T19:07:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f4678996700 1 -- 172.21.15.175:0/824841867 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 518804960 0 0) 0x7f4660003570 con 0x7f467408a100 2023-10-10T19:07:30.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f467a199700 1 -- 172.21.15.175:0/824841867 >> v1:172.21.15.191:6789/0 conn(0x7f467414fd90 legacy=0x7f46740c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:30.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f467a199700 1 -- 172.21.15.175:0/824841867 >> v1:172.21.15.175:6789/0 conn(0x7f467408a100 legacy=0x7f46740684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:30.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f467a199700 1 -- 172.21.15.175:0/824841867 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4674150160 con 0x7f4674068c00 2023-10-10T19:07:30.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f467a199700 1 --2- 172.21.15.175:0/824841867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4674068c00 0x7f4674159c30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f466c009e60 tx=0x7f466c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49dd1fa4011d63c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:30.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f4678996700 1 -- 172.21.15.175:0/824841867 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c00e070 con 0x7f4674068c00 2023-10-10T19:07:30.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f4678996700 1 -- 172.21.15.175:0/824841867 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c00b7a0 con 0x7f4674068c00 2023-10-10T19:07:30.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.534+0000 7f4678996700 1 -- 172.21.15.175:0/824841867 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664001470 con 0x7f4674068c00 2023-10-10T19:07:30.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.535+0000 7f467a99a700 1 -- 172.21.15.175:0/824841867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4674068c00 msgr2=0x7f4674159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:30.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.535+0000 7f467a99a700 1 --2- 172.21.15.175:0/824841867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4674068c00 0x7f4674159c30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f466c009e60 tx=0x7f466c009780 comp rx=0 tx=0).stop 2023-10-10T19:07:30.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.535+0000 7f467a99a700 1 -- 172.21.15.175:0/824841867 shutdown_connections 2023-10-10T19:07:30.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.535+0000 7f467a99a700 1 --2- 172.21.15.175:0/824841867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4674068c00 0x7f4674159c30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:30.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.535+0000 7f467a99a700 1 -- 172.21.15.175:0/824841867 >> 172.21.15.175:0/824841867 conn(0x7f4674063d00 msgr2=0x7f46740cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:30.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.535+0000 7f467a99a700 1 -- 172.21.15.175:0/824841867 shutdown_connections 2023-10-10T19:07:30.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.535+0000 7f467a99a700 1 -- 172.21.15.175:0/824841867 wait complete. 2023-10-10T19:07:30.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.536+0000 7f467a99a700 1 Processor -- start 2023-10-10T19:07:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.536+0000 7f467a99a700 1 -- start start 2023-10-10T19:07:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.536+0000 7f467a99a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4674068c00 0x7f4674169d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.536+0000 7f467a99a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f467408a100 0x7f46740cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.536+0000 7f467a99a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f467414fd90 0x7f46740cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.536+0000 7f467a99a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46741508f0 con 0x7f4674068c00 2023-10-10T19:07:30.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.536+0000 7f467a99a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4674155650 con 0x7f467414fd90 2023-10-10T19:07:30.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.536+0000 7f467a99a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46740cadb0 con 0x7f467408a100 2023-10-10T19:07:30.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4674068c00 0x7f4674169d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:30.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4674068c00 0x7f4674169d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36850/0 (socket says 172.21.15.175:36850) 2023-10-10T19:07:30.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f467a199700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f467414fd90 0x7f46740cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:30.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 -- 172.21.15.175:0/302485558 learned_addr learned my addr 172.21.15.175:0/302485558 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:30.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679197700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f467408a100 0x7f46740cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:30.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 -- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f467408a100 msgr2=0x7f46740cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:30.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f467408a100 0x7f46740cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:30.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 -- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f467414fd90 msgr2=0x7f46740cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:30.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f467414fd90 0x7f46740cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:30.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 -- 172.21.15.175:0/302485558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c009a20 con 0x7f4674068c00 2023-10-10T19:07:30.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f467a199700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f467414fd90 0x7f46740cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:07:30.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4679998700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4674068c00 0x7f4674169d60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f467000c580 tx=0x7f4670007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4669ffb700 1 -- 172.21.15.175:0/302485558 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4670010510 con 0x7f4674068c00 2023-10-10T19:07:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4669ffb700 1 -- 172.21.15.175:0/302485558 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4670010670 con 0x7f4674068c00 2023-10-10T19:07:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f4669ffb700 1 -- 172.21.15.175:0/302485558 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4670010a30 con 0x7f4674068c00 2023-10-10T19:07:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.537+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46740cb760 con 0x7f4674068c00 2023-10-10T19:07:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.538+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46740caad0 con 0x7f4674068c00 2023-10-10T19:07:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.539+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4674067570 con 0x7f4674068c00 2023-10-10T19:07:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.539+0000 7f4669ffb700 1 -- 172.21.15.175:0/302485558 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f467001b400 con 0x7f4674068c00 2023-10-10T19:07:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.539+0000 7f4669ffb700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4658066f30 0x7f46580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.539+0000 7f4669ffb700 1 -- 172.21.15.175:0/302485558 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f4670084490 con 0x7f4674068c00 2023-10-10T19:07:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.539+0000 7f4679197700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4658066f30 0x7f46580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:30.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.540+0000 7f4679197700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4658066f30 0x7f46580693e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f467414e590 tx=0x7f4664000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:30.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.543+0000 7f4669ffb700 1 -- 172.21.15.175:0/302485558 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46700512e0 con 0x7f4674068c00 2023-10-10T19:07:30.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.770+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4674155f80 con 0x7f4674068c00 2023-10-10T19:07:30.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.770+0000 7f4669ffb700 1 -- 172.21.15.175:0/302485558 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f4670054900 con 0x7f4674068c00 2023-10-10T19:07:30.771 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:30.771 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:30.714005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:resolve","state_seq":24,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:30.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4658066f30 msgr2=0x7f46580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:30.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4658066f30 0x7f46580693e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f467414e590 tx=0x7f4664000bc0 comp rx=0 tx=0).stop 2023-10-10T19:07:30.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4674068c00 msgr2=0x7f4674169d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:30.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4674068c00 0x7f4674169d60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f467000c580 tx=0x7f4670007e10 comp rx=0 tx=0).stop 2023-10-10T19:07:30.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 shutdown_connections 2023-10-10T19:07:30.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4658066f30 0x7f46580693e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:30.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f467414fd90 0x7f46740cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:30.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4674068c00 0x7f4674169d60 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:30.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 --2- 172.21.15.175:0/302485558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f467408a100 0x7f46740cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:30.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 >> 172.21.15.175:0/302485558 conn(0x7f4674063d00 msgr2=0x7f467414e800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:30.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 shutdown_connections 2023-10-10T19:07:30.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:30.773+0000 7f467a99a700 1 -- 172.21.15.175:0/302485558 wait complete. 2023-10-10T19:07:30.776 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 169 2023-10-10T19:07:30.785 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:32.786 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:32.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.914+0000 7efcda396700 1 Processor -- start 2023-10-10T19:07:32.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.914+0000 7efcda396700 1 -- start start 2023-10-10T19:07:32.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.914+0000 7efcda396700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 0x7efcd4159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.914+0000 7efcda396700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efcd40681e0 0x7efcd40c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.914+0000 7efcda396700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd4156800 con 0x7efcd414ffc0 2023-10-10T19:07:32.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.914+0000 7efcda396700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd40c59e0 con 0x7efcd40681e0 2023-10-10T19:07:32.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.914+0000 7efcda396700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415a2b0 con 0x7efcd414fbf0 2023-10-10T19:07:32.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd37fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7efcd414ffc0 0x7efcd4063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:57900/0 (socket says 172.21.15.175:57900) 2023-10-10T19:07:32.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd37fe700 1 -- 172.21.15.175:0/193245022 learned_addr learned my addr 172.21.15.175:0/193245022 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:32.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd3fff700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 0x7efcd4159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:32.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd8933700 1 --2- 172.21.15.175:0/193245022 >> v2:172.21.15.191:3300/0 conn(0x7efcd40681e0 0x7efcd40c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:32.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124819098 0 0) 0x7efcd4156800 con 0x7efcd414ffc0 2023-10-10T19:07:32.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcbc003570 con 0x7efcd414ffc0 2023-10-10T19:07:32.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 791354760 0 0) 0x7efcbc003570 con 0x7efcd414ffc0 2023-10-10T19:07:32.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 msgr2=0x7efcd4159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:32.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 0x7efcd4159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:32.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 >> v2:172.21.15.191:3300/0 conn(0x7efcd40681e0 msgr2=0x7efcd40c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:32.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 --2- 172.21.15.175:0/193245022 >> v2:172.21.15.191:3300/0 conn(0x7efcd40681e0 0x7efcd40c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:32.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd4063e30 con 0x7efcd414ffc0 2023-10-10T19:07:32.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7efcc8001470 con 0x7efcd414ffc0 2023-10-10T19:07:32.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 >> v1:172.21.15.175:6789/0 conn(0x7efcd414ffc0 legacy=0x7efcd4063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:32.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 0x7efcbc003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcbc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcbc004d50 0x7efcbc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40c59e0 con 0x7efcbc003fe0 2023-10-10T19:07:32.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc00ba60 con 0x7efcbc004d50 2023-10-10T19:07:32.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc00bbd0 con 0x7efcd414fbf0 2023-10-10T19:07:32.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd37fe700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcbc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:32.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd3fff700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 0x7efcbc003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:32.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.915+0000 7efcd8933700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcbc004d50 0x7efcbc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:32.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd37fe700 1 -- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 msgr2=0x7efcbc003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:32.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd37fe700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 0x7efcbc003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:32.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd37fe700 1 -- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcbc004d50 msgr2=0x7efcbc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:32.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd37fe700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcbc004d50 0x7efcbc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:32.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd37fe700 1 -- 172.21.15.175:0/193245022 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc003570 con 0x7efcbc003fe0 2023-10-10T19:07:32.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd3fff700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd414fbf0 0x7efcbc003a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:07:32.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd37fe700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcbc004810 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7efcc800c3c0 tx=0x7efcc8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa3ed1dc7bbb670 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:32.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc800cc30 con 0x7efcbc003fe0 2023-10-10T19:07:32.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc800cd90 con 0x7efcbc003fe0 2023-10-10T19:07:32.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd2ffd700 1 -- 172.21.15.175:0/193245022 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8005b50 con 0x7efcbc003fe0 2023-10-10T19:07:32.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcda396700 1 -- 172.21.15.175:0/193245022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 msgr2=0x7efcbc004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:32.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcda396700 1 --2- 172.21.15.175:0/193245022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcbc004810 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7efcc800c3c0 tx=0x7efcc8001e50 comp rx=0 tx=0).stop 2023-10-10T19:07:32.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.916+0000 7efcd8933700 1 -- 172.21.15.175:0/193245022 reap_dead start 2023-10-10T19:07:32.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.917+0000 7efcda396700 1 -- 172.21.15.175:0/193245022 shutdown_connections 2023-10-10T19:07:32.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.917+0000 7efcda396700 1 -- 172.21.15.175:0/193245022 >> 172.21.15.175:0/193245022 conn(0x7efcd40cc760 msgr2=0x7efcd40ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:32.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.917+0000 7efcda396700 1 -- 172.21.15.175:0/193245022 shutdown_connections 2023-10-10T19:07:32.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.917+0000 7efcda396700 1 -- 172.21.15.175:0/193245022 wait complete. 2023-10-10T19:07:32.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.918+0000 7efcda396700 1 Processor -- start 2023-10-10T19:07:32.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.918+0000 7efcda396700 1 -- start start 2023-10-10T19:07:32.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcda396700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcd40cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcda396700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd4159a60 0x7efcd4165c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcda396700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcd40c2f60 0x7efcd4166170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcda396700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd4153620 con 0x7efcbc003fe0 2023-10-10T19:07:32.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcda396700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd41551a0 con 0x7efcd40c2f60 2023-10-10T19:07:32.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcda396700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd4152a30 con 0x7efcd4159a60 2023-10-10T19:07:32.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcd40cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:32.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcd40cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36872/0 (socket says 172.21.15.175:36872) 2023-10-10T19:07:32.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 -- 172.21.15.175:0/2814166487 learned_addr learned my addr 172.21.15.175:0/2814166487 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:32.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 -- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd4159a60 msgr2=0x7efcd4165c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:32.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd8933700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcd40c2f60 0x7efcd4166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:32.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd4159a60 0x7efcd4165c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:32.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 -- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcd40c2f60 msgr2=0x7efcd4166170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:32.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcd40c2f60 0x7efcd4166170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:32.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 -- 172.21.15.175:0/2814166487 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40c3450 con 0x7efcbc003fe0 2023-10-10T19:07:32.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcd3fff700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcd40cb880 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7efcc4006c20 tx=0x7efcc4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:32.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcbbfff700 1 -- 172.21.15.175:0/2814166487 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc400fce0 con 0x7efcbc003fe0 2023-10-10T19:07:32.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcbbfff700 1 -- 172.21.15.175:0/2814166487 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc400fe40 con 0x7efcbc003fe0 2023-10-10T19:07:32.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd4152cf0 con 0x7efcbc003fe0 2023-10-10T19:07:32.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcbbfff700 1 -- 172.21.15.175:0/2814166487 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4014690 con 0x7efcbc003fe0 2023-10-10T19:07:32.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.919+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd41532b0 con 0x7efcbc003fe0 2023-10-10T19:07:32.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.921+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd4067570 con 0x7efcbc003fe0 2023-10-10T19:07:32.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.921+0000 7efcbbfff700 1 -- 172.21.15.175:0/2814166487 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efcc4014a00 con 0x7efcbc003fe0 2023-10-10T19:07:32.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.921+0000 7efcbbfff700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcc0066f30 0x7efcc00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:32.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.921+0000 7efcbbfff700 1 -- 172.21.15.175:0/2814166487 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7efcc4018070 con 0x7efcbc003fe0 2023-10-10T19:07:32.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.921+0000 7efcd37fe700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcc0066f30 0x7efcc00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:32.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.922+0000 7efcd37fe700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcc0066f30 0x7efcc00693e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efcd4158c30 tx=0x7efcc8015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:32.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:32.925+0000 7efcbbfff700 1 -- 172.21.15.175:0/2814166487 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efcc4052480 con 0x7efcbc003fe0 2023-10-10T19:07:33.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.148+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcd4069780 con 0x7efcbc003fe0 2023-10-10T19:07:33.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.148+0000 7efcbbfff700 1 -- 172.21.15.175:0/2814166487 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5445 (secure 0 0 0) 0x7efcc4055aa0 con 0x7efcbc003fe0 2023-10-10T19:07:33.149 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:33.149 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:30.714005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:resolve","state_seq":24,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:33.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.150+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcc0066f30 msgr2=0x7efcc00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:33.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.150+0000 7efcda396700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcc0066f30 0x7efcc00693e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efcd4158c30 tx=0x7efcc8015000 comp rx=0 tx=0).stop 2023-10-10T19:07:33.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.150+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 msgr2=0x7efcd40cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:33.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.150+0000 7efcda396700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcd40cb880 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7efcc4006c20 tx=0x7efcc4009120 comp rx=0 tx=0).stop 2023-10-10T19:07:33.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.151+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 shutdown_connections 2023-10-10T19:07:33.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.151+0000 7efcda396700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efcc0066f30 0x7efcc00693e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:33.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.151+0000 7efcda396700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcd40c2f60 0x7efcd4166170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:33.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.151+0000 7efcda396700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcbc003fe0 0x7efcd40cb880 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:33.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.151+0000 7efcda396700 1 --2- 172.21.15.175:0/2814166487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efcd4159a60 0x7efcd4165c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:33.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.151+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 >> 172.21.15.175:0/2814166487 conn(0x7efcd40cc760 msgr2=0x7efcd414e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:33.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.151+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 shutdown_connections 2023-10-10T19:07:33.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:33.151+0000 7efcda396700 1 -- 172.21.15.175:0/2814166487 wait complete. 2023-10-10T19:07:33.154 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 169 2023-10-10T19:07:33.163 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:35.164 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:35.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.293+0000 7f2ca74e1700 1 Processor -- start 2023-10-10T19:07:35.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.293+0000 7f2ca74e1700 1 -- start start 2023-10-10T19:07:35.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.293+0000 7f2ca74e1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 0x7f2ca0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:35.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca74e1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2ca00b2390 0x7f2ca0068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:35.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca74e1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2ca0068bd0 0x7f2ca00aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:35.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca74e1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca0150770 con 0x7f2ca00b2390 2023-10-10T19:07:35.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca74e1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca01574b0 con 0x7f2ca0068bd0 2023-10-10T19:07:35.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca74e1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00afad0 con 0x7f2ca00b1fc0 2023-10-10T19:07:35.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2ca00b2390 0x7f2ca0068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:35.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2ca00b2390 0x7f2ca0068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57570/0 (socket says 172.21.15.175:57570) 2023-10-10T19:07:35.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 -- 172.21.15.175:0/2900329125 learned_addr learned my addr 172.21.15.175:0/2900329125 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:35.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca5a7e700 1 --2- 172.21.15.175:0/2900329125 >> v2:172.21.15.191:3300/0 conn(0x7f2ca0068bd0 0x7f2ca00aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:35.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 -- 172.21.15.175:0/2900329125 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 msgr2=0x7f2ca0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:35.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 --2- 172.21.15.175:0/2900329125 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 0x7f2ca0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 -- 172.21.15.175:0/2900329125 >> v2:172.21.15.191:3300/0 conn(0x7f2ca0068bd0 msgr2=0x7f2ca00aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:35.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 --2- 172.21.15.175:0/2900329125 >> v2:172.21.15.191:3300/0 conn(0x7f2ca0068bd0 0x7f2ca00aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 -- 172.21.15.175:0/2900329125 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00af4c0 con 0x7f2ca00b2390 2023-10-10T19:07:35.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca5a7e700 1 --2- 172.21.15.175:0/2900329125 >> v2:172.21.15.191:3300/0 conn(0x7f2ca0068bd0 0x7f2ca00aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:07:35.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2ca4a7c700 1 --2- 172.21.15.175:0/2900329125 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca00b2390 0x7f2ca0068690 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00afa40 tx=0x7f2c94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e19f95f3e360e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:35.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2c9ffff700 1 -- 172.21.15.175:0/2900329125 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9400b8f0 con 0x7f2ca00b2390 2023-10-10T19:07:35.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2c9ffff700 1 -- 172.21.15.175:0/2900329125 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9400ba50 con 0x7f2ca00b2390 2023-10-10T19:07:35.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.294+0000 7f2c9ffff700 1 -- 172.21.15.175:0/2900329125 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9400bdf0 con 0x7f2ca00b2390 2023-10-10T19:07:35.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 -- 172.21.15.175:0/2900329125 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca00b2390 msgr2=0x7f2ca0068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:35.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/2900329125 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca00b2390 0x7f2ca0068690 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00afa40 tx=0x7f2c94009250 comp rx=0 tx=0).stop 2023-10-10T19:07:35.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 -- 172.21.15.175:0/2900329125 shutdown_connections 2023-10-10T19:07:35.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/2900329125 >> v2:172.21.15.191:3300/0 conn(0x7f2ca0068bd0 0x7f2ca00aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/2900329125 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 0x7f2ca0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/2900329125 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca00b2390 0x7f2ca0068690 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 -- 172.21.15.175:0/2900329125 >> 172.21.15.175:0/2900329125 conn(0x7f2ca0063230 msgr2=0x7f2ca0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:35.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 -- 172.21.15.175:0/2900329125 shutdown_connections 2023-10-10T19:07:35.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.295+0000 7f2ca74e1700 1 -- 172.21.15.175:0/2900329125 wait complete. 2023-10-10T19:07:35.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.296+0000 7f2ca74e1700 1 Processor -- start 2023-10-10T19:07:35.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.296+0000 7f2ca74e1700 1 -- start start 2023-10-10T19:07:35.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca74e1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca0068bd0 0x7f2ca0169730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:35.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca74e1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 0x7f2ca00c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:35.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca74e1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ca00b2390 0x7f2ca0155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:35.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca74e1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00cd5f0 con 0x7f2ca0068bd0 2023-10-10T19:07:35.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca74e1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00cd350 con 0x7f2ca00b2390 2023-10-10T19:07:35.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca74e1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00c39a0 con 0x7f2ca00b1fc0 2023-10-10T19:07:35.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca527d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca0068bd0 0x7f2ca0169730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:35.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca527d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca0068bd0 0x7f2ca0169730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57572/0 (socket says 172.21.15.175:57572) 2023-10-10T19:07:35.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca527d700 1 -- 172.21.15.175:0/4032069437 learned_addr learned my addr 172.21.15.175:0/4032069437 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:35.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca5a7e700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ca00b2390 0x7f2ca0155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:35.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca4a7c700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 0x7f2ca00c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:35.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca527d700 1 -- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 msgr2=0x7f2ca00c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:35.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca527d700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 0x7f2ca00c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca527d700 1 -- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ca00b2390 msgr2=0x7f2ca0155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:35.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca527d700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ca00b2390 0x7f2ca0155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca527d700 1 -- 172.21.15.175:0/4032069437 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c940094f0 con 0x7f2ca0068bd0 2023-10-10T19:07:35.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.297+0000 7f2ca4a7c700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 0x7f2ca00c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:35.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.298+0000 7f2ca527d700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca0068bd0 0x7f2ca0169730 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000b810 tx=0x7f2c90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:35.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.298+0000 7f2c9d7fa700 1 -- 172.21.15.175:0/4032069437 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9000f510 con 0x7f2ca0068bd0 2023-10-10T19:07:35.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.298+0000 7f2c9d7fa700 1 -- 172.21.15.175:0/4032069437 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9000f690 con 0x7f2ca0068bd0 2023-10-10T19:07:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.298+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca00c3c40 con 0x7f2ca0068bd0 2023-10-10T19:07:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.298+0000 7f2c9d7fa700 1 -- 172.21.15.175:0/4032069437 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9000fa50 con 0x7f2ca0068bd0 2023-10-10T19:07:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.298+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca0168f30 con 0x7f2ca0068bd0 2023-10-10T19:07:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.299+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca0067570 con 0x7f2ca0068bd0 2023-10-10T19:07:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.299+0000 7f2c9d7fa700 1 -- 172.21.15.175:0/4032069437 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2c9001a400 con 0x7f2ca0068bd0 2023-10-10T19:07:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.299+0000 7f2c9d7fa700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c8c066f30 0x7f2c8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.300+0000 7f2c9d7fa700 1 -- 172.21.15.175:0/4032069437 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f2c90084550 con 0x7f2ca0068bd0 2023-10-10T19:07:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.300+0000 7f2ca4a7c700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c8c066f30 0x7f2c8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.300+0000 7f2ca4a7c700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c8c066f30 0x7f2c8c0693e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00afa40 tx=0x7f2c94017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.304+0000 7f2c9d7fa700 1 -- 172.21.15.175:0/4032069437 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c900513a0 con 0x7f2ca0068bd0 2023-10-10T19:07:35.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.531+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ca0069060 con 0x7f2ca0068bd0 2023-10-10T19:07:35.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.531+0000 7f2c9d7fa700 1 -- 172.21.15.175:0/4032069437 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f2c900549c0 con 0x7f2ca0068bd0 2023-10-10T19:07:35.532 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:35.532 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:30.714005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:resolve","state_seq":24,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:35.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.533+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c8c066f30 msgr2=0x7f2c8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:35.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.533+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c8c066f30 0x7f2c8c0693e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00afa40 tx=0x7f2c94017000 comp rx=0 tx=0).stop 2023-10-10T19:07:35.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.533+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca0068bd0 msgr2=0x7f2ca0169730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:35.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.533+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca0068bd0 0x7f2ca0169730 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000b810 tx=0x7f2c90009250 comp rx=0 tx=0).stop 2023-10-10T19:07:35.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.534+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 shutdown_connections 2023-10-10T19:07:35.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.534+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c8c066f30 0x7f2c8c0693e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.534+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ca00b2390 0x7f2ca0155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.534+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ca0068bd0 0x7f2ca0169730 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.534+0000 7f2ca74e1700 1 --2- 172.21.15.175:0/4032069437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ca00b1fc0 0x7f2ca00c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:35.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.534+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 >> 172.21.15.175:0/4032069437 conn(0x7f2ca0063230 msgr2=0x7f2ca01575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:35.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.534+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 shutdown_connections 2023-10-10T19:07:35.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:35.534+0000 7f2ca74e1700 1 -- 172.21.15.175:0/4032069437 wait complete. 2023-10-10T19:07:35.537 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 169 2023-10-10T19:07:35.546 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:37.547 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:37.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.673+0000 7fcfd57d6700 1 Processor -- start 2023-10-10T19:07:37.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.673+0000 7fcfd57d6700 1 -- start start 2023-10-10T19:07:37.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.674+0000 7fcfd57d6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcfd0159d70 0x7fcfd00c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:37.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.674+0000 7fcfd57d6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd0156800 con 0x7fcfd014ffc0 2023-10-10T19:07:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.674+0000 7fcfd57d6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd0063e30 con 0x7fcfd014fbf0 2023-10-10T19:07:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.674+0000 7fcfd57d6700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd00c59e0 con 0x7fcfd0159d70 2023-10-10T19:07:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.674+0000 7fcfc67fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcfd014ffc0 0x7fcfd00681e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42724/0 (socket says 172.21.15.175:42724) 2023-10-10T19:07:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.674+0000 7fcfc67fc700 1 -- 172.21.15.175:0/1728938891 learned_addr learned my addr 172.21.15.175:0/1728938891 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:37.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.674+0000 7fcfcf7fe700 1 --2- 172.21.15.175:0/1728938891 >> v2:172.21.15.191:3300/0 conn(0x7fcfd0159d70 0x7fcfd00c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:37.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442045990 0 0) 0x7fcfd0156800 con 0x7fcfd014ffc0 2023-10-10T19:07:37.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfbc003570 con 0x7fcfd014ffc0 2023-10-10T19:07:37.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472928904 0 0) 0x7fcfd0063e30 con 0x7fcfd014fbf0 2023-10-10T19:07:37.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd0156800 con 0x7fcfd014fbf0 2023-10-10T19:07:37.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591492110 0 0) 0x7fcfbc003570 con 0x7fcfd014ffc0 2023-10-10T19:07:37.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 >> v2:172.21.15.191:3300/0 conn(0x7fcfd0159d70 msgr2=0x7fcfd00c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:37.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 --2- 172.21.15.175:0/1728938891 >> v2:172.21.15.191:3300/0 conn(0x7fcfd0159d70 0x7fcfd00c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 >> v1:172.21.15.191:6789/0 conn(0x7fcfd014fbf0 legacy=0x7fcfd0063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:37.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd0156c90 con 0x7fcfd014ffc0 2023-10-10T19:07:37.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fcfb8001470 con 0x7fcfd014ffc0 2023-10-10T19:07:37.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfcf7fe700 1 --2- 172.21.15.175:0/1728938891 >> v2:172.21.15.191:3300/0 conn(0x7fcfd0159d70 0x7fcfd00c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:07:37.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 >> v1:172.21.15.175:6789/0 conn(0x7fcfd014ffc0 legacy=0x7fcfd00681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:37.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfbc003bf0 0x7fcfbc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:37.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfbc004810 0x7fcfbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc0051c0 0x7fcfbc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00c59e0 con 0x7fcfbc004810 2023-10-10T19:07:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc00c0b0 con 0x7fcfbc0051c0 2023-10-10T19:07:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc00c270 con 0x7fcfbc003bf0 2023-10-10T19:07:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfc67fc700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfbc004810 0x7fcfbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:37.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfceffd700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfbc003bf0 0x7fcfbc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:37.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfcf7fe700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc0051c0 0x7fcfbc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:37.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfc67fc700 1 -- 172.21.15.175:0/1728938891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfbc003bf0 msgr2=0x7fcfbc003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:37.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfc67fc700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfbc003bf0 0x7fcfbc003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfc67fc700 1 -- 172.21.15.175:0/1728938891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc0051c0 msgr2=0x7fcfbc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:37.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfc67fc700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc0051c0 0x7fcfbc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfcf7fe700 1 -- 172.21.15.175:0/1728938891 reap_dead start 2023-10-10T19:07:37.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.675+0000 7fcfc67fc700 1 -- 172.21.15.175:0/1728938891 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc003570 con 0x7fcfbc004810 2023-10-10T19:07:37.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfc67fc700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfbc004810 0x7fcfbc004c80 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800c2d0 tx=0x7fcfb8001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ff56618f06ac2d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:37.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb800cb60 con 0x7fcfbc004810 2023-10-10T19:07:37.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb800cce0 con 0x7fcfbc004810 2023-10-10T19:07:37.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfce7fc700 1 -- 172.21.15.175:0/1728938891 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8005b70 con 0x7fcfbc004810 2023-10-10T19:07:37.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfd57d6700 1 -- 172.21.15.175:0/1728938891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfbc004810 msgr2=0x7fcfbc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:37.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfd57d6700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfbc004810 0x7fcfbc004c80 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800c2d0 tx=0x7fcfb8001e10 comp rx=0 tx=0).stop 2023-10-10T19:07:37.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfd57d6700 1 -- 172.21.15.175:0/1728938891 shutdown_connections 2023-10-10T19:07:37.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfd57d6700 1 --2- 172.21.15.175:0/1728938891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfbc004810 0x7fcfbc004c80 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfd57d6700 1 -- 172.21.15.175:0/1728938891 >> 172.21.15.175:0/1728938891 conn(0x7fcfd00cc760 msgr2=0x7fcfd00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:37.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfd57d6700 1 -- 172.21.15.175:0/1728938891 shutdown_connections 2023-10-10T19:07:37.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.676+0000 7fcfd57d6700 1 -- 172.21.15.175:0/1728938891 wait complete. 2023-10-10T19:07:37.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfd57d6700 1 Processor -- start 2023-10-10T19:07:37.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfd57d6700 1 -- start start 2023-10-10T19:07:37.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfd57d6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc004810 0x7fcfd0158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:37.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfd57d6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd00c3190 0x7fcfd01586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:37.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfd57d6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd00c7360 0x7fcfd00c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:37.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfd57d6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00c6ed0 con 0x7fcfd00c7360 2023-10-10T19:07:37.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfd57d6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00c7d10 con 0x7fcfbc004810 2023-10-10T19:07:37.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfd57d6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd0165e40 con 0x7fcfd00c3190 2023-10-10T19:07:37.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfcf7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd00c7360 0x7fcfd00c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:37.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfcf7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd00c7360 0x7fcfd00c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57590/0 (socket says 172.21.15.175:57590) 2023-10-10T19:07:37.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.678+0000 7fcfcf7fe700 1 -- 172.21.15.175:0/2600825880 learned_addr learned my addr 172.21.15.175:0/2600825880 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:37.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfceffd700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc004810 0x7fcfd0158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:37.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfc67fc700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd00c3190 0x7fcfd01586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:37.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfcf7fe700 1 -- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd00c3190 msgr2=0x7fcfd01586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:37.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfcf7fe700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd00c3190 0x7fcfd01586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfcf7fe700 1 -- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc004810 msgr2=0x7fcfd0158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:37.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfcf7fe700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc004810 0x7fcfd0158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfcf7fe700 1 -- 172.21.15.175:0/2600825880 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd0166120 con 0x7fcfd00c7360 2023-10-10T19:07:37.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfceffd700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc004810 0x7fcfd0158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:37.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfcf7fe700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd00c7360 0x7fcfd00c77d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0153930 tx=0x7fcfc80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:37.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfc77fe700 1 -- 172.21.15.175:0/2600825880 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc800f9e0 con 0x7fcfd00c7360 2023-10-10T19:07:37.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfc77fe700 1 -- 172.21.15.175:0/2600825880 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc800fb60 con 0x7fcfd00c7360 2023-10-10T19:07:37.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd0166400 con 0x7fcfd00c7360 2023-10-10T19:07:37.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfc77fe700 1 -- 172.21.15.175:0/2600825880 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8014440 con 0x7fcfd00c7360 2023-10-10T19:07:37.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.679+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd01669c0 con 0x7fcfd00c7360 2023-10-10T19:07:37.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.680+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd00c7040 con 0x7fcfd00c7360 2023-10-10T19:07:37.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.681+0000 7fcfc77fe700 1 -- 172.21.15.175:0/2600825880 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcfc80147f0 con 0x7fcfd00c7360 2023-10-10T19:07:37.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.681+0000 7fcfc77fe700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb0066f30 0x7fcfb00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:37.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.681+0000 7fcfc77fe700 1 -- 172.21.15.175:0/2600825880 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7fcfc8085170 con 0x7fcfd00c7360 2023-10-10T19:07:37.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.681+0000 7fcfceffd700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb0066f30 0x7fcfb00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:37.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.682+0000 7fcfceffd700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb0066f30 0x7fcfb00693e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0001920 tx=0x7fcfc0001bd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:37.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.685+0000 7fcfc77fe700 1 -- 172.21.15.175:0/2600825880 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcfc8052040 con 0x7fcfd00c7360 2023-10-10T19:07:37.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.915+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfd0067570 con 0x7fcfd00c7360 2023-10-10T19:07:37.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.916+0000 7fcfc77fe700 1 -- 172.21.15.175:0/2600825880 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5445 (secure 0 0 0) 0x7fcfc8055660 con 0x7fcfd00c7360 2023-10-10T19:07:37.916 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:37.917 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:37.738939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:37.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb0066f30 msgr2=0x7fcfb00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:37.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb0066f30 0x7fcfb00693e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0001920 tx=0x7fcfc0001bd0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd00c7360 msgr2=0x7fcfd00c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:37.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd00c7360 0x7fcfd00c77d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0153930 tx=0x7fcfc80091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 shutdown_connections 2023-10-10T19:07:37.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcfb0066f30 0x7fcfb00693e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcfd00c7360 0x7fcfd00c77d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfbc004810 0x7fcfd0158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 --2- 172.21.15.175:0/2600825880 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcfd00c3190 0x7fcfd01586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:37.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 >> 172.21.15.175:0/2600825880 conn(0x7fcfd00cc760 msgr2=0x7fcfd01594c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:37.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 shutdown_connections 2023-10-10T19:07:37.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:37.918+0000 7fcfd57d6700 1 -- 172.21.15.175:0/2600825880 wait complete. 2023-10-10T19:07:37.924 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 171 2023-10-10T19:07:37.931 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 171, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 171, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5552": { "addr": "172.21.15.191:6839/1910071809", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1910071809, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1910071809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5552, "incarnation": 154, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 36 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:reconnect", "state_seq": 26 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:07:37.738939+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 6106, "mds_3": 5552, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:07:39.933 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:40.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.059+0000 7f35a35e0700 1 Processor -- start 2023-10-10T19:07:40.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.060+0000 7f35a35e0700 1 -- start start 2023-10-10T19:07:40.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.060+0000 7f35a35e0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f359c14ffc0 0x7f359c159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:40.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.060+0000 7f35a35e0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c0681e0 0x7f359c0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:40.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.060+0000 7f35a35e0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c156800 con 0x7f359c14fbf0 2023-10-10T19:07:40.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.060+0000 7f35a35e0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359c0c59e0 con 0x7f359c14ffc0 2023-10-10T19:07:40.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a35e0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c15a2b0 con 0x7f359c0681e0 2023-10-10T19:07:40.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a0b7b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f359c14ffc0 0x7f359c159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:40.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a0b7b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f359c14ffc0 0x7f359c159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57592/0 (socket says 172.21.15.175:57592) 2023-10-10T19:07:40.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a0b7b700 1 -- 172.21.15.175:0/215851402 learned_addr learned my addr 172.21.15.175:0/215851402 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:40.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a1b7d700 1 --2- 172.21.15.175:0/215851402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c0681e0 0x7f359c0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:40.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a0b7b700 1 -- 172.21.15.175:0/215851402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c0681e0 msgr2=0x7f359c0c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:40.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a0b7b700 1 --2- 172.21.15.175:0/215851402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c0681e0 0x7f359c0c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a0b7b700 1 -- 172.21.15.175:0/215851402 >> v1:172.21.15.191:6789/0 conn(0x7f359c14fbf0 legacy=0x7f359c063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:40.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a0b7b700 1 -- 172.21.15.175:0/215851402 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c063e30 con 0x7f359c14ffc0 2023-10-10T19:07:40.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f35a0b7b700 1 --2- 172.21.15.175:0/215851402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14ffc0 0x7f359c159d70 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f358c0099b0 tx=0x7f358c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0eb478ee9a45389 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f3593fff700 1 -- 172.21.15.175:0/215851402 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c009ea0 con 0x7f359c14ffc0 2023-10-10T19:07:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f3593fff700 1 -- 172.21.15.175:0/215851402 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c00b860 con 0x7f359c14ffc0 2023-10-10T19:07:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.061+0000 7f3593fff700 1 -- 172.21.15.175:0/215851402 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c00bc00 con 0x7f359c14ffc0 2023-10-10T19:07:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.062+0000 7f35a35e0700 1 -- 172.21.15.175:0/215851402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14ffc0 msgr2=0x7f359c159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.062+0000 7f35a35e0700 1 --2- 172.21.15.175:0/215851402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14ffc0 0x7f359c159d70 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f358c0099b0 tx=0x7f358c009250 comp rx=0 tx=0).stop 2023-10-10T19:07:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.062+0000 7f35a35e0700 1 -- 172.21.15.175:0/215851402 shutdown_connections 2023-10-10T19:07:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.062+0000 7f35a35e0700 1 --2- 172.21.15.175:0/215851402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c0681e0 0x7f359c0c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.062+0000 7f35a35e0700 1 --2- 172.21.15.175:0/215851402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14ffc0 0x7f359c159d70 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.062+0000 7f35a35e0700 1 -- 172.21.15.175:0/215851402 >> 172.21.15.175:0/215851402 conn(0x7f359c0cc760 msgr2=0x7f359c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.062+0000 7f35a35e0700 1 -- 172.21.15.175:0/215851402 shutdown_connections 2023-10-10T19:07:40.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.062+0000 7f35a35e0700 1 -- 172.21.15.175:0/215851402 wait complete. 2023-10-10T19:07:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.063+0000 7f35a35e0700 1 Processor -- start 2023-10-10T19:07:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.063+0000 7f35a35e0700 1 -- start start 2023-10-10T19:07:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a35e0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359c0681e0 0x7f359c16a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a35e0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14fbf0 0x7f359c166290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a35e0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c14ffc0 0x7f359c152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a35e0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c151f40 con 0x7f359c14fbf0 2023-10-10T19:07:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a35e0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c153b70 con 0x7f359c0681e0 2023-10-10T19:07:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a35e0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c169ec0 con 0x7f359c14ffc0 2023-10-10T19:07:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14fbf0 0x7f359c166290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14fbf0 0x7f359c166290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57596/0 (socket says 172.21.15.175:57596) 2023-10-10T19:07:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 -- 172.21.15.175:0/3528850357 learned_addr learned my addr 172.21.15.175:0/3528850357 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a1b7d700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c14ffc0 0x7f359c152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a137c700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359c0681e0 0x7f359c16a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 -- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c14ffc0 msgr2=0x7f359c152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c14ffc0 0x7f359c152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 -- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359c0681e0 msgr2=0x7f359c16a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359c0681e0 0x7f359c16a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 -- 172.21.15.175:0/3528850357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0094f0 con 0x7f359c14fbf0 2023-10-10T19:07:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a137c700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359c0681e0 0x7f359c16a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:07:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.064+0000 7f35a0b7b700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14fbf0 0x7f359c166290 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f358c006fd0 tx=0x7f358c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.065+0000 7f35917fa700 1 -- 172.21.15.175:0/3528850357 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c00bbe0 con 0x7f359c14fbf0 2023-10-10T19:07:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.065+0000 7f35917fa700 1 -- 172.21.15.175:0/3528850357 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c01a450 con 0x7f359c14fbf0 2023-10-10T19:07:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.065+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c169890 con 0x7f359c14fbf0 2023-10-10T19:07:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.065+0000 7f35917fa700 1 -- 172.21.15.175:0/3528850357 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c01a7f0 con 0x7f359c14fbf0 2023-10-10T19:07:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.065+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c1540a0 con 0x7f359c14fbf0 2023-10-10T19:07:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.066+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c0c5ed0 con 0x7f359c14fbf0 2023-10-10T19:07:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.066+0000 7f35917fa700 1 -- 172.21.15.175:0/3528850357 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f358c01aba0 con 0x7f359c14fbf0 2023-10-10T19:07:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.066+0000 7f35917fa700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3580066f30 0x7f35800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.066+0000 7f35917fa700 1 -- 172.21.15.175:0/3528850357 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f358c0864a0 con 0x7f359c14fbf0 2023-10-10T19:07:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.067+0000 7f35a137c700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3580066f30 0x7f35800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.067+0000 7f35a137c700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3580066f30 0x7f35800693e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f359800d220 tx=0x7f3598009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:40.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.071+0000 7f35917fa700 1 -- 172.21.15.175:0/3528850357 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f358c053250 con 0x7f359c14fbf0 2023-10-10T19:07:40.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.294+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f359c069580 con 0x7f359c14fbf0 2023-10-10T19:07:40.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.295+0000 7f35917fa700 1 -- 172.21.15.175:0/3528850357 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f358c056870 con 0x7f359c14fbf0 2023-10-10T19:07:40.295 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:40.296 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:37.738939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:40.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3580066f30 msgr2=0x7f35800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:40.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3580066f30 0x7f35800693e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f359800d220 tx=0x7f3598009040 comp rx=0 tx=0).stop 2023-10-10T19:07:40.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14fbf0 msgr2=0x7f359c166290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:40.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14fbf0 0x7f359c166290 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f358c006fd0 tx=0x7f358c017000 comp rx=0 tx=0).stop 2023-10-10T19:07:40.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 shutdown_connections 2023-10-10T19:07:40.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3580066f30 0x7f35800693e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359c14ffc0 0x7f359c152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359c0681e0 0x7f359c16a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 --2- 172.21.15.175:0/3528850357 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359c14fbf0 0x7f359c166290 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:40.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 >> 172.21.15.175:0/3528850357 conn(0x7f359c0cc760 msgr2=0x7f359c15bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:40.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 shutdown_connections 2023-10-10T19:07:40.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:40.297+0000 7f35a35e0700 1 -- 172.21.15.175:0/3528850357 wait complete. 2023-10-10T19:07:40.300 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 171 2023-10-10T19:07:40.310 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:42.312 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:42.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.439+0000 7fe5cca14700 1 Processor -- start 2023-10-10T19:07:42.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.439+0000 7fe5cca14700 1 -- start start 2023-10-10T19:07:42.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5cca14700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c814fd90 0x7fe5c8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:42.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5cca14700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe5c80cd040 0x7fe5c80684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:42.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5cca14700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c8067fa0 con 0x7fe5c808a100 2023-10-10T19:07:42.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5cca14700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c80c5600 con 0x7fe5c80cd040 2023-10-10T19:07:42.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5cca14700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c8156b70 con 0x7fe5c814fd90 2023-10-10T19:07:42.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c759e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe5c80cd040 0x7fe5c80684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:42.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c759e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe5c80cd040 0x7fe5c80684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57610/0 (socket says 172.21.15.175:57610) 2023-10-10T19:07:42.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c759e700 1 -- 172.21.15.175:0/2769357322 learned_addr learned my addr 172.21.15.175:0/2769357322 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:42.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c6d9d700 1 --2- 172.21.15.175:0/2769357322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c814fd90 0x7fe5c8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:42.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c759e700 1 -- 172.21.15.175:0/2769357322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c814fd90 msgr2=0x7fe5c8159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:42.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c759e700 1 --2- 172.21.15.175:0/2769357322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c814fd90 0x7fe5c8159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c759e700 1 -- 172.21.15.175:0/2769357322 >> v1:172.21.15.191:6789/0 conn(0x7fe5c808a100 legacy=0x7fe5c80c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c759e700 1 -- 172.21.15.175:0/2769357322 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c8150160 con 0x7fe5c80cd040 2023-10-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.440+0000 7fe5c759e700 1 --2- 172.21.15.175:0/2769357322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c80cd040 0x7fe5c80684f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009f20 tx=0x7fe5c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50f01b6c43a8eef server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5c659c700 1 -- 172.21.15.175:0/2769357322 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c000e070 con 0x7fe5c80cd040 2023-10-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5c659c700 1 -- 172.21.15.175:0/2769357322 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c000b9d0 con 0x7fe5c80cd040 2023-10-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5c659c700 1 -- 172.21.15.175:0/2769357322 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00092e0 con 0x7fe5c80cd040 2023-10-10T19:07:42.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5cca14700 1 -- 172.21.15.175:0/2769357322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c80cd040 msgr2=0x7fe5c80684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:42.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2769357322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c80cd040 0x7fe5c80684f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009f20 tx=0x7fe5c0009840 comp rx=0 tx=0).stop 2023-10-10T19:07:42.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5cca14700 1 -- 172.21.15.175:0/2769357322 shutdown_connections 2023-10-10T19:07:42.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2769357322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c80cd040 0x7fe5c80684f0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2769357322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c814fd90 0x7fe5c8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5cca14700 1 -- 172.21.15.175:0/2769357322 >> 172.21.15.175:0/2769357322 conn(0x7fe5c8063d00 msgr2=0x7fe5c80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:42.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5cca14700 1 -- 172.21.15.175:0/2769357322 shutdown_connections 2023-10-10T19:07:42.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.441+0000 7fe5cca14700 1 -- 172.21.15.175:0/2769357322 wait complete. 2023-10-10T19:07:42.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.442+0000 7fe5cca14700 1 Processor -- start 2023-10-10T19:07:42.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.442+0000 7fe5cca14700 1 -- start start 2023-10-10T19:07:42.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5cca14700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c808a100 0x7fe5c8152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:42.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5cca14700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c80cd040 0x7fe5c80c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:42.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5cca14700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c814fd90 0x7fe5c80c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:42.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5cca14700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c8166b70 con 0x7fe5c814fd90 2023-10-10T19:07:42.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5cca14700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c80b1d30 con 0x7fe5c808a100 2023-10-10T19:07:42.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5cca14700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c8152370 con 0x7fe5c80cd040 2023-10-10T19:07:42.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5c759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c814fd90 0x7fe5c80c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:42.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5c759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c814fd90 0x7fe5c80c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57624/0 (socket says 172.21.15.175:57624) 2023-10-10T19:07:42.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5c759e700 1 -- 172.21.15.175:0/2210517548 learned_addr learned my addr 172.21.15.175:0/2210517548 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:42.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5c6d9d700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c808a100 0x7fe5c8152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:42.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5be59c700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c80cd040 0x7fe5c80c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:42.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5c759e700 1 -- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c80cd040 msgr2=0x7fe5c80c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:42.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5c759e700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c80cd040 0x7fe5c80c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5c759e700 1 -- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c808a100 msgr2=0x7fe5c8152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:42.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.443+0000 7fe5c759e700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c808a100 0x7fe5c8152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.444+0000 7fe5c759e700 1 -- 172.21.15.175:0/2210517548 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c0009ae0 con 0x7fe5c814fd90 2023-10-10T19:07:42.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.444+0000 7fe5be59c700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c80cd040 0x7fe5c80c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:07:42.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.444+0000 7fe5c759e700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c814fd90 0x7fe5c80c7330 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8064c20 tx=0x7fe5c0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:42.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.444+0000 7fe5bf7fe700 1 -- 172.21.15.175:0/2210517548 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0008c70 con 0x7fe5c814fd90 2023-10-10T19:07:42.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.444+0000 7fe5bf7fe700 1 -- 172.21.15.175:0/2210517548 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c0008e10 con 0x7fe5c814fd90 2023-10-10T19:07:42.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.444+0000 7fe5bf7fe700 1 -- 172.21.15.175:0/2210517548 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00266b0 con 0x7fe5c814fd90 2023-10-10T19:07:42.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.444+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c81665f0 con 0x7fe5c814fd90 2023-10-10T19:07:42.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.444+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c8166890 con 0x7fe5c814fd90 2023-10-10T19:07:42.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.446+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c80b1e70 con 0x7fe5c814fd90 2023-10-10T19:07:42.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.446+0000 7fe5bf7fe700 1 -- 172.21.15.175:0/2210517548 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe5c0003040 con 0x7fe5c814fd90 2023-10-10T19:07:42.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.446+0000 7fe5bf7fe700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe5a8066f30 0x7fe5a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:42.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.447+0000 7fe5bf7fe700 1 -- 172.21.15.175:0/2210517548 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7fe5c00869d0 con 0x7fe5c814fd90 2023-10-10T19:07:42.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.447+0000 7fe5c6d9d700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe5a8066f30 0x7fe5a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:42.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.447+0000 7fe5c6d9d700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe5a8066f30 0x7fe5a80693e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009ca0 tx=0x7fe5b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:42.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.451+0000 7fe5bf7fe700 1 -- 172.21.15.175:0/2210517548 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe5c80b1e70 con 0x7fe5c814fd90 2023-10-10T19:07:42.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.674+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5c80b1e70 con 0x7fe5c814fd90 2023-10-10T19:07:42.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.675+0000 7fe5bf7fe700 1 -- 172.21.15.175:0/2210517548 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5442 (secure 0 0 0) 0x7fe5c80b1e70 con 0x7fe5c814fd90 2023-10-10T19:07:42.676 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:42.676 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:41.747975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:42.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe5a8066f30 msgr2=0x7fe5a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:42.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe5a8066f30 0x7fe5a80693e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009ca0 tx=0x7fe5b8008040 comp rx=0 tx=0).stop 2023-10-10T19:07:42.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c814fd90 msgr2=0x7fe5c80c7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:42.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c814fd90 0x7fe5c80c7330 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8064c20 tx=0x7fe5c0018000 comp rx=0 tx=0).stop 2023-10-10T19:07:42.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 shutdown_connections 2023-10-10T19:07:42.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe5a8066f30 0x7fe5a80693e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5c814fd90 0x7fe5c80c7330 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe5c808a100 0x7fe5c8152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 --2- 172.21.15.175:0/2210517548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5c80cd040 0x7fe5c80c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:42.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 >> 172.21.15.175:0/2210517548 conn(0x7fe5c8063d00 msgr2=0x7fe5c815ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:42.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 shutdown_connections 2023-10-10T19:07:42.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:42.677+0000 7fe5cca14700 1 -- 172.21.15.175:0/2210517548 wait complete. 2023-10-10T19:07:42.681 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 172 2023-10-10T19:07:42.689 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:44.691 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:44.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.820+0000 7f5fe1f1a700 1 Processor -- start 2023-10-10T19:07:44.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.820+0000 7f5fe1f1a700 1 -- start start 2023-10-10T19:07:44.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.820+0000 7f5fe1f1a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5fdc14fbf0 0x7f5fdc159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:44.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.820+0000 7f5fe1f1a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5fdc14ffc0 0x7f5fdc063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:44.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.820+0000 7f5fe1f1a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc156800 con 0x7f5fdc0681e0 2023-10-10T19:07:44.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fe1f1a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc0c59e0 con 0x7f5fdc14ffc0 2023-10-10T19:07:44.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fe1f1a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc15a2b0 con 0x7f5fdc14fbf0 2023-10-10T19:07:44.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdaffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5fdc14ffc0 0x7f5fdc063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:44.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdaffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5fdc14ffc0 0x7f5fdc063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49876/0 (socket says 172.21.15.175:49876) 2023-10-10T19:07:44.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdaffd700 1 -- 172.21.15.175:0/1760593650 learned_addr learned my addr 172.21.15.175:0/1760593650 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:44.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdb7fe700 1 --2- 172.21.15.175:0/1760593650 >> v2:172.21.15.191:3300/0 conn(0x7f5fdc14fbf0 0x7f5fdc159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:44.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdaffd700 1 -- 172.21.15.175:0/1760593650 >> v2:172.21.15.191:3300/0 conn(0x7f5fdc14fbf0 msgr2=0x7f5fdc159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:44.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdaffd700 1 --2- 172.21.15.175:0/1760593650 >> v2:172.21.15.191:3300/0 conn(0x7f5fdc14fbf0 0x7f5fdc159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:44.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdaffd700 1 -- 172.21.15.175:0/1760593650 >> v1:172.21.15.191:6789/0 conn(0x7f5fdc0681e0 legacy=0x7f5fdc0c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:44.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdaffd700 1 -- 172.21.15.175:0/1760593650 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc063e30 con 0x7f5fdc14ffc0 2023-10-10T19:07:44.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fdaffd700 1 --2- 172.21.15.175:0/1760593650 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc14ffc0 0x7f5fdc063230 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0cc6c0 tx=0x7f5fcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b70631bf13457e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:44.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fda7fc700 1 -- 172.21.15.175:0/1760593650 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00b820 con 0x7f5fdc14ffc0 2023-10-10T19:07:44.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fda7fc700 1 -- 172.21.15.175:0/1760593650 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc00b980 con 0x7f5fdc14ffc0 2023-10-10T19:07:44.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.821+0000 7f5fda7fc700 1 -- 172.21.15.175:0/1760593650 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00bd20 con 0x7f5fdc14ffc0 2023-10-10T19:07:44.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.822+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/1760593650 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc14ffc0 msgr2=0x7f5fdc063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:44.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.822+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/1760593650 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc14ffc0 0x7f5fdc063230 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0cc6c0 tx=0x7f5fcc009250 comp rx=0 tx=0).stop 2023-10-10T19:07:44.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.822+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/1760593650 shutdown_connections 2023-10-10T19:07:44.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.822+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/1760593650 >> v2:172.21.15.191:3300/0 conn(0x7f5fdc14fbf0 0x7f5fdc159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:44.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.822+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/1760593650 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc14ffc0 0x7f5fdc063230 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:44.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.822+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/1760593650 >> 172.21.15.175:0/1760593650 conn(0x7f5fdc0cc760 msgr2=0x7f5fdc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:44.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.822+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/1760593650 shutdown_connections 2023-10-10T19:07:44.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.822+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/1760593650 wait complete. 2023-10-10T19:07:44.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.823+0000 7f5fe1f1a700 1 Processor -- start 2023-10-10T19:07:44.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.823+0000 7f5fe1f1a700 1 -- start start 2023-10-10T19:07:44.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fe1f1a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc0681e0 0x7f5fdc169f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:44.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fe1f1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fdc14fbf0 0x7f5fdc0cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:44.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fe1f1a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fdc14ffc0 0x7f5fdc1518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:44.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fe1f1a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0c34a0 con 0x7f5fdc0681e0 2023-10-10T19:07:44.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fe1f1a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc1533e0 con 0x7f5fdc14ffc0 2023-10-10T19:07:44.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fe1f1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc1536c0 con 0x7f5fdc14fbf0 2023-10-10T19:07:44.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc0681e0 0x7f5fdc169f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:44.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc0681e0 0x7f5fdc169f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49882/0 (socket says 172.21.15.175:49882) 2023-10-10T19:07:44.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 -- 172.21.15.175:0/3840205827 learned_addr learned my addr 172.21.15.175:0/3840205827 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:44.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdaffd700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fdc14fbf0 0x7f5fdc0cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:44.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdbfff700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fdc14ffc0 0x7f5fdc1518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:44.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 -- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fdc14fbf0 msgr2=0x7f5fdc0cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:44.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fdc14fbf0 0x7f5fdc0cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:44.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 -- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fdc14ffc0 msgr2=0x7f5fdc1518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:44.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fdc14ffc0 0x7f5fdc1518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:44.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 -- 172.21.15.175:0/3840205827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0094f0 con 0x7f5fdc0681e0 2023-10-10T19:07:44.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdbfff700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fdc14ffc0 0x7f5fdc1518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:07:44.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fdb7fe700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc0681e0 0x7f5fdc169f90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400d810 tx=0x7f5fc4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:44.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.824+0000 7f5fd37fe700 1 -- 172.21.15.175:0/3840205827 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc40125f0 con 0x7f5fdc0681e0 2023-10-10T19:07:44.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.825+0000 7f5fd37fe700 1 -- 172.21.15.175:0/3840205827 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4012750 con 0x7f5fdc0681e0 2023-10-10T19:07:44.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.825+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc0c2e50 con 0x7f5fdc0681e0 2023-10-10T19:07:44.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.825+0000 7f5fd37fe700 1 -- 172.21.15.175:0/3840205827 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4012b10 con 0x7f5fdc0681e0 2023-10-10T19:07:44.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.825+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc0a8aa0 con 0x7f5fdc0681e0 2023-10-10T19:07:44.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.826+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc151700 con 0x7f5fdc0681e0 2023-10-10T19:07:44.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.826+0000 7f5fd37fe700 1 -- 172.21.15.175:0/3840205827 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5fc401e460 con 0x7f5fdc0681e0 2023-10-10T19:07:44.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.826+0000 7f5fd37fe700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5fbc066f60 0x7f5fbc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:44.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.826+0000 7f5fd37fe700 1 -- 172.21.15.175:0/3840205827 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f5fc4084f70 con 0x7f5fdc0681e0 2023-10-10T19:07:44.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.827+0000 7f5fdaffd700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5fbc066f60 0x7f5fbc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:44.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.827+0000 7f5fdaffd700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5fbc066f60 0x7f5fbc069410 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0cc6c0 tx=0x7f5fcc017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:44.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:44.831+0000 7f5fd37fe700 1 -- 172.21.15.175:0/3840205827 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5fc4051dc0 con 0x7f5fdc0681e0 2023-10-10T19:07:45.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.058+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fdc067570 con 0x7f5fdc0681e0 2023-10-10T19:07:45.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.058+0000 7f5fd37fe700 1 -- 172.21.15.175:0/3840205827 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5442 (secure 0 0 0) 0x7f5fc40553e0 con 0x7f5fdc0681e0 2023-10-10T19:07:45.059 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:45.059 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:41.747975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:45.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5fbc066f60 msgr2=0x7f5fbc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:45.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5fbc066f60 0x7f5fbc069410 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0cc6c0 tx=0x7f5fcc017000 comp rx=0 tx=0).stop 2023-10-10T19:07:45.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc0681e0 msgr2=0x7f5fdc169f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:45.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc0681e0 0x7f5fdc169f90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400d810 tx=0x7f5fc4008040 comp rx=0 tx=0).stop 2023-10-10T19:07:45.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 shutdown_connections 2023-10-10T19:07:45.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5fbc066f60 0x7f5fbc069410 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:45.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fdc14ffc0 0x7f5fdc1518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:45.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fdc0681e0 0x7f5fdc169f90 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:45.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 --2- 172.21.15.175:0/3840205827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fdc14fbf0 0x7f5fdc0cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:45.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 >> 172.21.15.175:0/3840205827 conn(0x7f5fdc0cc760 msgr2=0x7f5fdc154160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:45.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 shutdown_connections 2023-10-10T19:07:45.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:45.060+0000 7f5fe1f1a700 1 -- 172.21.15.175:0/3840205827 wait complete. 2023-10-10T19:07:45.063 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 172 2023-10-10T19:07:45.072 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:47.074 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:47.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.199+0000 7ff733d69700 1 Processor -- start 2023-10-10T19:07:47.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.200+0000 7ff733d69700 1 -- start start 2023-10-10T19:07:47.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.200+0000 7ff733d69700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14c840 0x7ff72c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:47.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.200+0000 7ff733d69700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff72c14cc10 0x7ff72c0c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:47.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.200+0000 7ff733d69700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c155ca0 con 0x7ff72c0c59d0 2023-10-10T19:07:47.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.200+0000 7ff733d69700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72c156b00 con 0x7ff72c14cc10 2023-10-10T19:07:47.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.200+0000 7ff733d69700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c06d700 con 0x7ff72c14c840 2023-10-10T19:07:47.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731b05700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14c840 0x7ff72c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:47.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731304700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff72c14cc10 0x7ff72c0c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:47.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731304700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff72c14cc10 0x7ff72c0c4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:58448/0 (socket says 172.21.15.175:58448) 2023-10-10T19:07:47.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731304700 1 -- 172.21.15.175:0/2733852798 learned_addr learned my addr 172.21.15.175:0/2733852798 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:47.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff732306700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff72c0c59d0 0x7ff72c152970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:44540/0 (socket says 172.21.15.175:44540) 2023-10-10T19:07:47.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731b05700 1 -- 172.21.15.175:0/2733852798 >> v2:172.21.15.191:3300/0 conn(0x7ff72c14cc10 msgr2=0x7ff72c0c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:47.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731b05700 1 --2- 172.21.15.175:0/2733852798 >> v2:172.21.15.191:3300/0 conn(0x7ff72c14cc10 0x7ff72c0c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731b05700 1 -- 172.21.15.175:0/2733852798 >> v1:172.21.15.191:6789/0 conn(0x7ff72c0c59d0 legacy=0x7ff72c152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:47.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731b05700 1 -- 172.21.15.175:0/2733852798 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c155ef0 con 0x7ff72c14c840 2023-10-10T19:07:47.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731304700 1 --2- 172.21.15.175:0/2733852798 >> v2:172.21.15.191:3300/0 conn(0x7ff72c14cc10 0x7ff72c0c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:07:47.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.201+0000 7ff731b05700 1 --2- 172.21.15.175:0/2733852798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14c840 0x7ff72c06e7a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff728009eb0 tx=0x7ff728009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd75a86e99369eae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:47.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff730b03700 1 -- 172.21.15.175:0/2733852798 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72800e070 con 0x7ff72c14c840 2023-10-10T19:07:47.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff730b03700 1 -- 172.21.15.175:0/2733852798 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72800bcc0 con 0x7ff72c14c840 2023-10-10T19:07:47.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff730b03700 1 -- 172.21.15.175:0/2733852798 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728012680 con 0x7ff72c14c840 2023-10-10T19:07:47.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff733d69700 1 -- 172.21.15.175:0/2733852798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14c840 msgr2=0x7ff72c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:47.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff733d69700 1 --2- 172.21.15.175:0/2733852798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14c840 0x7ff72c06e7a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff728009eb0 tx=0x7ff728009780 comp rx=0 tx=0).stop 2023-10-10T19:07:47.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff733d69700 1 -- 172.21.15.175:0/2733852798 shutdown_connections 2023-10-10T19:07:47.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff733d69700 1 --2- 172.21.15.175:0/2733852798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14c840 0x7ff72c06e7a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff733d69700 1 --2- 172.21.15.175:0/2733852798 >> v2:172.21.15.191:3300/0 conn(0x7ff72c14cc10 0x7ff72c0c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff733d69700 1 -- 172.21.15.175:0/2733852798 >> 172.21.15.175:0/2733852798 conn(0x7ff72c068280 msgr2=0x7ff72c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:47.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff733d69700 1 -- 172.21.15.175:0/2733852798 shutdown_connections 2023-10-10T19:07:47.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.202+0000 7ff733d69700 1 -- 172.21.15.175:0/2733852798 wait complete. 2023-10-10T19:07:47.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.203+0000 7ff733d69700 1 Processor -- start 2023-10-10T19:07:47.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.203+0000 7ff733d69700 1 -- start start 2023-10-10T19:07:47.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff733d69700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff72c0c59d0 0x7ff72c151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:47.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff733d69700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff72c14c840 0x7ff72c152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:47.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff733d69700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14cc10 0x7ff72c154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:47.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff733d69700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0c77c0 con 0x7ff72c0c59d0 2023-10-10T19:07:47.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff733d69700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c1544f0 con 0x7ff72c14c840 2023-10-10T19:07:47.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff733d69700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0c4360 con 0x7ff72c14cc10 2023-10-10T19:07:47.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff731b05700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff72c0c59d0 0x7ff72c151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:47.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff731b05700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff72c0c59d0 0x7ff72c151b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49896/0 (socket says 172.21.15.175:49896) 2023-10-10T19:07:47.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff731b05700 1 -- 172.21.15.175:0/729063724 learned_addr learned my addr 172.21.15.175:0/729063724 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:47.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff731304700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff72c14c840 0x7ff72c152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:47.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.204+0000 7ff732306700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14cc10 0x7ff72c154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:47.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff731b05700 1 -- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14cc10 msgr2=0x7ff72c154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:47.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff731b05700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14cc10 0x7ff72c154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff731b05700 1 -- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff72c14c840 msgr2=0x7ff72c152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:47.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff731b05700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff72c14c840 0x7ff72c152070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff731b05700 1 -- 172.21.15.175:0/729063724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c0094f0 con 0x7ff72c0c59d0 2023-10-10T19:07:47.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff731304700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff72c14c840 0x7ff72c152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:07:47.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff731b05700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff72c0c59d0 0x7ff72c151b30 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff72800b6c0 tx=0x7ff728019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:47.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff721ffb700 1 -- 172.21.15.175:0/729063724 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728012d00 con 0x7ff72c0c59d0 2023-10-10T19:07:47.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff721ffb700 1 -- 172.21.15.175:0/729063724 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff728012e60 con 0x7ff72c0c59d0 2023-10-10T19:07:47.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff721ffb700 1 -- 172.21.15.175:0/729063724 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7280276c0 con 0x7ff72c0c59d0 2023-10-10T19:07:47.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff728009a20 con 0x7ff72c0c59d0 2023-10-10T19:07:47.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.205+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c0c6f10 con 0x7ff72c0c59d0 2023-10-10T19:07:47.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.206+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c067570 con 0x7ff72c0c59d0 2023-10-10T19:07:47.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.206+0000 7ff721ffb700 1 -- 172.21.15.175:0/729063724 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff728024070 con 0x7ff72c0c59d0 2023-10-10T19:07:47.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.207+0000 7ff721ffb700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff710066f30 0x7ff7100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:47.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.207+0000 7ff721ffb700 1 -- 172.21.15.175:0/729063724 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7ff728086f00 con 0x7ff72c0c59d0 2023-10-10T19:07:47.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.207+0000 7ff731304700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff710066f30 0x7ff7100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:47.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.207+0000 7ff731304700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff710066f30 0x7ff7100693e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff72c156820 tx=0x7ff71c009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:47.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.211+0000 7ff721ffb700 1 -- 172.21.15.175:0/729063724 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff728053d50 con 0x7ff72c0c59d0 2023-10-10T19:07:47.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.437+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff72c06da40 con 0x7ff72c0c59d0 2023-10-10T19:07:47.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.438+0000 7ff721ffb700 1 -- 172.21.15.175:0/729063724 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5442 (secure 0 0 0) 0x7ff72802c070 con 0x7ff72c0c59d0 2023-10-10T19:07:47.438 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:47.438 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:41.747975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:47.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff710066f30 msgr2=0x7ff7100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:47.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff710066f30 0x7ff7100693e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff72c156820 tx=0x7ff71c009250 comp rx=0 tx=0).stop 2023-10-10T19:07:47.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff72c0c59d0 msgr2=0x7ff72c151b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:47.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff72c0c59d0 0x7ff72c151b30 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff72800b6c0 tx=0x7ff728019000 comp rx=0 tx=0).stop 2023-10-10T19:07:47.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 shutdown_connections 2023-10-10T19:07:47.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff710066f30 0x7ff7100693e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff72c14cc10 0x7ff72c154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff72c0c59d0 0x7ff72c151b30 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 --2- 172.21.15.175:0/729063724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff72c14c840 0x7ff72c152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:47.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 >> 172.21.15.175:0/729063724 conn(0x7ff72c068280 msgr2=0x7ff72c0c0300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:47.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 shutdown_connections 2023-10-10T19:07:47.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:47.440+0000 7ff733d69700 1 -- 172.21.15.175:0/729063724 wait complete. 2023-10-10T19:07:47.443 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 172 2023-10-10T19:07:47.452 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:49.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:49.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.580+0000 7fd5d24da700 1 Processor -- start 2023-10-10T19:07:49.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.580+0000 7fd5d24da700 1 -- start start 2023-10-10T19:07:49.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.581+0000 7fd5d24da700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5cc0c5e40 0x7fd5cc0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:49.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.581+0000 7fd5d24da700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5cc156520 0x7fd5cc1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:49.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.581+0000 7fd5d24da700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc156ee0 con 0x7fd5cc06bf00 2023-10-10T19:07:49.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.581+0000 7fd5d24da700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5cc1560d0 con 0x7fd5cc156520 2023-10-10T19:07:49.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.581+0000 7fd5d24da700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5cc0c6df0 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.581+0000 7fd5d0a77700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5cc156520 0x7fd5cc1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:49.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.581+0000 7fd5d0a77700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5cc156520 0x7fd5cc1569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49898/0 (socket says 172.21.15.175:49898) 2023-10-10T19:07:49.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5d0a77700 1 -- 172.21.15.175:0/2565219954 learned_addr learned my addr 172.21.15.175:0/2565219954 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:49.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5cbfff700 1 --2- 172.21.15.175:0/2565219954 >> v2:172.21.15.191:3300/0 conn(0x7fd5cc0c5e40 0x7fd5cc0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:49.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5d0a77700 1 -- 172.21.15.175:0/2565219954 >> v2:172.21.15.191:3300/0 conn(0x7fd5cc0c5e40 msgr2=0x7fd5cc0c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:49.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5d0a77700 1 --2- 172.21.15.175:0/2565219954 >> v2:172.21.15.191:3300/0 conn(0x7fd5cc0c5e40 0x7fd5cc0c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5d0a77700 1 -- 172.21.15.175:0/2565219954 >> v1:172.21.15.191:6789/0 conn(0x7fd5cc06bf00 legacy=0x7fd5cc0c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:49.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5d0a77700 1 -- 172.21.15.175:0/2565219954 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc0c6920 con 0x7fd5cc156520 2023-10-10T19:07:49.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5d0a77700 1 --2- 172.21.15.175:0/2565219954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc156520 0x7fd5cc1569a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc065240 tx=0x7fd5c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5262ef3f710801f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:49.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5caffd700 1 -- 172.21.15.175:0/2565219954 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c400e070 con 0x7fd5cc156520 2023-10-10T19:07:49.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5caffd700 1 -- 172.21.15.175:0/2565219954 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c400bab0 con 0x7fd5cc156520 2023-10-10T19:07:49.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.582+0000 7fd5caffd700 1 -- 172.21.15.175:0/2565219954 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c40092e0 con 0x7fd5cc156520 2023-10-10T19:07:49.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.583+0000 7fd5d24da700 1 -- 172.21.15.175:0/2565219954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc156520 msgr2=0x7fd5cc1569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:49.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.583+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2565219954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc156520 0x7fd5cc1569a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc065240 tx=0x7fd5c4009840 comp rx=0 tx=0).stop 2023-10-10T19:07:49.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.583+0000 7fd5d24da700 1 -- 172.21.15.175:0/2565219954 shutdown_connections 2023-10-10T19:07:49.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.583+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2565219954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc156520 0x7fd5cc1569a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.583+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2565219954 >> v2:172.21.15.191:3300/0 conn(0x7fd5cc0c5e40 0x7fd5cc0c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.583+0000 7fd5d24da700 1 -- 172.21.15.175:0/2565219954 >> 172.21.15.175:0/2565219954 conn(0x7fd5cc068900 msgr2=0x7fd5cc068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:49.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.583+0000 7fd5d24da700 1 -- 172.21.15.175:0/2565219954 shutdown_connections 2023-10-10T19:07:49.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.583+0000 7fd5d24da700 1 -- 172.21.15.175:0/2565219954 wait complete. 2023-10-10T19:07:49.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.584+0000 7fd5d24da700 1 Processor -- start 2023-10-10T19:07:49.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.584+0000 7fd5d24da700 1 -- start start 2023-10-10T19:07:49.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5d24da700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5cc06bf00 0x7fd5cc0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:49.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5d24da700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc0c5e40 0x7fd5cc135c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:49.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5d24da700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5cc156520 0x7fd5cc1661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:49.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5d24da700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc0c8590 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5d24da700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc158550 con 0x7fd5cc156520 2023-10-10T19:07:49.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5d24da700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc0c87e0 con 0x7fd5cc06bf00 2023-10-10T19:07:49.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc0c5e40 0x7fd5cc135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:49.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc0c5e40 0x7fd5cc135c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49900/0 (socket says 172.21.15.175:49900) 2023-10-10T19:07:49.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 -- 172.21.15.175:0/2860636491 learned_addr learned my addr 172.21.15.175:0/2860636491 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:49.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5d0a77700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5cc156520 0x7fd5cc1661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:49.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cbfff700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5cc06bf00 0x7fd5cc0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:49.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 -- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5cc06bf00 msgr2=0x7fd5cc0c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:49.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5cc06bf00 0x7fd5cc0c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 -- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5cc156520 msgr2=0x7fd5cc1661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:49.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5cc156520 0x7fd5cc1661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 -- 172.21.15.175:0/2860636491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c4009ae0 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5cb7fe700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc0c5e40 0x7fd5cc135c50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c590 tx=0x7fd5c0007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:49.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.585+0000 7fd5b3fff700 1 -- 172.21.15.175:0/2860636491 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0011590 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.586+0000 7fd5b3fff700 1 -- 172.21.15.175:0/2860636491 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c0011710 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.586+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc169900 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.586+0000 7fd5b3fff700 1 -- 172.21.15.175:0/2860636491 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0011ad0 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.586+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc169ec0 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.586+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc169760 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.587+0000 7fd5b3fff700 1 -- 172.21.15.175:0/2860636491 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd5c001b3b0 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.587+0000 7fd5b3fff700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5b8066f30 0x7fd5b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:49.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.587+0000 7fd5b3fff700 1 -- 172.21.15.175:0/2860636491 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7fd5c0083c60 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.587+0000 7fd5cbfff700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5b8066f30 0x7fd5b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:49.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.588+0000 7fd5cbfff700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5b8066f30 0x7fd5b80693e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009da0 tx=0x7fd5bc009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:49.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.591+0000 7fd5b3fff700 1 -- 172.21.15.175:0/2860636491 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd5cc169760 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.816+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5cc169760 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.817+0000 7fd5b3fff700 1 -- 172.21.15.175:0/2860636491 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+5448 (secure 0 0 0) 0x7fd5cc169760 con 0x7fd5cc0c5e40 2023-10-10T19:07:49.817 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:49.817 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:47.767439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:clientreplay","state_seq":30,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:49.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5b8066f30 msgr2=0x7fd5b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:49.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5b8066f30 0x7fd5b80693e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009da0 tx=0x7fd5bc009120 comp rx=0 tx=0).stop 2023-10-10T19:07:49.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc0c5e40 msgr2=0x7fd5cc135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:49.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc0c5e40 0x7fd5cc135c50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c590 tx=0x7fd5c0007f50 comp rx=0 tx=0).stop 2023-10-10T19:07:49.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 shutdown_connections 2023-10-10T19:07:49.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd5b8066f30 0x7fd5b80693e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5cc156520 0x7fd5cc1661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5cc06bf00 0x7fd5cc0c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 --2- 172.21.15.175:0/2860636491 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5cc0c5e40 0x7fd5cc135c50 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 >> 172.21.15.175:0/2860636491 conn(0x7fd5cc068900 msgr2=0x7fd5cc136f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:49.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 shutdown_connections 2023-10-10T19:07:49.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:49.819+0000 7fd5d24da700 1 -- 172.21.15.175:0/2860636491 wait complete. 2023-10-10T19:07:49.822 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 173 2023-10-10T19:07:49.832 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 173, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 173, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5552": { "addr": "172.21.15.191:6839/1910071809", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 1910071809, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 1910071809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5552, "incarnation": 154, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 36 }, "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:clientreplay", "state_seq": 30 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:07:47.767439+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 6106, "mds_3": 5552, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:07:51.833 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:51.842 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:07:51.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:07:51.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.964+0000 7f1d659f3700 1 Processor -- start 2023-10-10T19:07:51.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.964+0000 7f1d659f3700 1 -- start start 2023-10-10T19:07:51.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d659f3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 0x7f1d60155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:51.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d659f3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1d600b2390 0x7f1d60068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:51.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d659f3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d601574b0 con 0x7f1d60068bd0 2023-10-10T19:07:51.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d659f3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d60150770 con 0x7f1d600b2390 2023-10-10T19:07:51.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d659f3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600afad0 con 0x7f1d600b1fc0 2023-10-10T19:07:51.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d567fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1d600b2390 0x7f1d60068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:51.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d5f7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1d60068bd0 0x7f1d600aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48508/0 (socket says 172.21.15.175:48508) 2023-10-10T19:07:51.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d5f7fe700 1 -- 172.21.15.175:0/989763937 learned_addr learned my addr 172.21.15.175:0/989763937 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:51.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d5effd700 1 --2- 172.21.15.175:0/989763937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 0x7f1d60155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:51.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d567fc700 1 -- 172.21.15.175:0/989763937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 msgr2=0x7f1d60155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:51.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d567fc700 1 --2- 172.21.15.175:0/989763937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 0x7f1d60155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:51.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d567fc700 1 -- 172.21.15.175:0/989763937 >> v1:172.21.15.175:6789/0 conn(0x7f1d60068bd0 legacy=0x7f1d600aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:51.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d567fc700 1 -- 172.21.15.175:0/989763937 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d600af4c0 con 0x7f1d600b2390 2023-10-10T19:07:51.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.965+0000 7f1d5e7fc700 1 -- 172.21.15.175:0/989763937 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713110603 0 0) 0x7f1d601574b0 con 0x7f1d60068bd0 2023-10-10T19:07:51.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d567fc700 1 --2- 172.21.15.175:0/989763937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d600b2390 0x7f1d60068690 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1d480098e0 tx=0x7f1d480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13f20a769f89e74 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:51.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d5e7fc700 1 -- 172.21.15.175:0/989763937 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48009e80 con 0x7f1d600b2390 2023-10-10T19:07:51.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d5e7fc700 1 -- 172.21.15.175:0/989763937 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4800b8f0 con 0x7f1d600b2390 2023-10-10T19:07:51.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d5e7fc700 1 -- 172.21.15.175:0/989763937 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4800bc70 con 0x7f1d600b2390 2023-10-10T19:07:51.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d659f3700 1 -- 172.21.15.175:0/989763937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d600b2390 msgr2=0x7f1d60068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:51.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d659f3700 1 --2- 172.21.15.175:0/989763937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d600b2390 0x7f1d60068690 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1d480098e0 tx=0x7f1d480091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:51.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d659f3700 1 -- 172.21.15.175:0/989763937 shutdown_connections 2023-10-10T19:07:51.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d659f3700 1 --2- 172.21.15.175:0/989763937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 0x7f1d60155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:51.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d659f3700 1 --2- 172.21.15.175:0/989763937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d600b2390 0x7f1d60068690 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:51.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d659f3700 1 -- 172.21.15.175:0/989763937 >> 172.21.15.175:0/989763937 conn(0x7f1d60063230 msgr2=0x7f1d60063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:51.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d659f3700 1 -- 172.21.15.175:0/989763937 shutdown_connections 2023-10-10T19:07:51.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.966+0000 7f1d659f3700 1 -- 172.21.15.175:0/989763937 wait complete. 2023-10-10T19:07:51.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d659f3700 1 Processor -- start 2023-10-10T19:07:51.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d659f3700 1 -- start start 2023-10-10T19:07:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d659f3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d60068bd0 0x7f1d600cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d659f3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 0x7f1d60165d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d659f3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d600b2390 0x7f1d60166270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d659f3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600c4570 con 0x7f1d60068bd0 2023-10-10T19:07:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d659f3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600cc570 con 0x7f1d600b2390 2023-10-10T19:07:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d659f3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600cd660 con 0x7f1d600b1fc0 2023-10-10T19:07:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d5effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d60068bd0 0x7f1d600cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d5effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d60068bd0 0x7f1d600cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49914/0 (socket says 172.21.15.175:49914) 2023-10-10T19:07:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.968+0000 7f1d5effd700 1 -- 172.21.15.175:0/3740586044 learned_addr learned my addr 172.21.15.175:0/3740586044 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:51.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d567fc700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 0x7f1d60165d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:51.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d5f7fe700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d600b2390 0x7f1d60166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:51.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d5effd700 1 -- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 msgr2=0x7f1d60165d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:51.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d5effd700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 0x7f1d60165d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:51.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d5effd700 1 -- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d600b2390 msgr2=0x7f1d60166270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:51.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d5effd700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d600b2390 0x7f1d60166270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:51.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d5effd700 1 -- 172.21.15.175:0/3740586044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d48009420 con 0x7f1d60068bd0 2023-10-10T19:07:51.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d5effd700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d60068bd0 0x7f1d600cb300 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000d890 tx=0x7f1d50008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:51.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d577fe700 1 -- 172.21.15.175:0/3740586044 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5000b600 con 0x7f1d60068bd0 2023-10-10T19:07:51.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d577fe700 1 -- 172.21.15.175:0/3740586044 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5000b760 con 0x7f1d60068bd0 2023-10-10T19:07:51.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d600cbc80 con 0x7f1d60068bd0 2023-10-10T19:07:51.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d577fe700 1 -- 172.21.15.175:0/3740586044 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5000bae0 con 0x7f1d60068bd0 2023-10-10T19:07:51.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.969+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d600cbf20 con 0x7f1d60068bd0 2023-10-10T19:07:51.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.970+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d60067570 con 0x7f1d60068bd0 2023-10-10T19:07:51.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.970+0000 7f1d577fe700 1 -- 172.21.15.175:0/3740586044 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1d5001e460 con 0x7f1d60068bd0 2023-10-10T19:07:51.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.971+0000 7f1d577fe700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d40066f30 0x7f1d400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:51.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.971+0000 7f1d577fe700 1 -- 172.21.15.175:0/3740586044 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f1d50085000 con 0x7f1d60068bd0 2023-10-10T19:07:51.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.971+0000 7f1d567fc700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d40066f30 0x7f1d400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:51.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.971+0000 7f1d567fc700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d40066f30 0x7f1d400693e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1d60068ba0 tx=0x7f1d48017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:51.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:51.975+0000 7f1d577fe700 1 -- 172.21.15.175:0/3740586044 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d50051e50 con 0x7f1d60068bd0 2023-10-10T19:07:52.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.202+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d600690b0 con 0x7f1d60068bd0 2023-10-10T19:07:52.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.202+0000 7f1d577fe700 1 -- 172.21.15.175:0/3740586044 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f1d50055470 con 0x7f1d60068bd0 2023-10-10T19:07:52.203 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:52.203 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:47.767439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:clientreplay","state_seq":30,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:52.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.204+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d40066f30 msgr2=0x7f1d400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:52.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.204+0000 7f1d659f3700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d40066f30 0x7f1d400693e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1d60068ba0 tx=0x7f1d48017000 comp rx=0 tx=0).stop 2023-10-10T19:07:52.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.204+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d60068bd0 msgr2=0x7f1d600cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:52.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d60068bd0 0x7f1d600cb300 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000d890 tx=0x7f1d50008040 comp rx=0 tx=0).stop 2023-10-10T19:07:52.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 shutdown_connections 2023-10-10T19:07:52.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1d40066f30 0x7f1d400693e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:52.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d600b2390 0x7f1d60166270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:52.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d60068bd0 0x7f1d600cb300 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:52.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 --2- 172.21.15.175:0/3740586044 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d600b1fc0 0x7f1d60165d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:52.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 >> 172.21.15.175:0/3740586044 conn(0x7f1d60063230 msgr2=0x7f1d60152540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:52.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 shutdown_connections 2023-10-10T19:07:52.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:52.205+0000 7f1d659f3700 1 -- 172.21.15.175:0/3740586044 wait complete. 2023-10-10T19:07:52.208 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 173 2023-10-10T19:07:52.217 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:07:54.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:54.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.343+0000 7f71c704f700 1 Processor -- start 2023-10-10T19:07:54.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.343+0000 7f71c704f700 1 -- start start 2023-10-10T19:07:54.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.343+0000 7f71c704f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c0068a40 0x7f71c0063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:54.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.343+0000 7f71c704f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f71c0068e10 0x7f71c00af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:54.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.343+0000 7f71c704f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c00c4ee0 con 0x7f71c00cac80 2023-10-10T19:07:54.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.343+0000 7f71c704f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c0064180 con 0x7f71c0068e10 2023-10-10T19:07:54.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.343+0000 7f71c704f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c00c5780 con 0x7f71c0068a40 2023-10-10T19:07:54.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71c55ec700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f71c0068e10 0x7f71c00af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:54.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71b7fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f71c00cac80 0x7f71c006b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48510/0 (socket says 172.21.15.175:48510) 2023-10-10T19:07:54.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71b7fff700 1 -- 172.21.15.175:0/431609759 learned_addr learned my addr 172.21.15.175:0/431609759 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:07:54.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71c4deb700 1 --2- 172.21.15.175:0/431609759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c0068a40 0x7f71c0063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:54.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71c55ec700 1 -- 172.21.15.175:0/431609759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c0068a40 msgr2=0x7f71c0063c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:54.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71c55ec700 1 --2- 172.21.15.175:0/431609759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c0068a40 0x7f71c0063c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71c55ec700 1 -- 172.21.15.175:0/431609759 >> v1:172.21.15.175:6789/0 conn(0x7f71c00cac80 legacy=0x7f71c006b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:54.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71b77fe700 1 -- 172.21.15.175:0/431609759 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698007644 0 0) 0x7f71c00c4ee0 con 0x7f71c00cac80 2023-10-10T19:07:54.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71c55ec700 1 -- 172.21.15.175:0/431609759 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c00c6710 con 0x7f71c0068e10 2023-10-10T19:07:54.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71c55ec700 1 --2- 172.21.15.175:0/431609759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c00af910 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f71b80098e0 tx=0x7f71b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276b935e7182a08d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:54.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71b77fe700 1 -- 172.21.15.175:0/431609759 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8009e80 con 0x7f71c0068e10 2023-10-10T19:07:54.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71b77fe700 1 -- 172.21.15.175:0/431609759 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b800b860 con 0x7f71c0068e10 2023-10-10T19:07:54.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.344+0000 7f71b77fe700 1 -- 172.21.15.175:0/431609759 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b800bbe0 con 0x7f71c0068e10 2023-10-10T19:07:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.345+0000 7f71c704f700 1 -- 172.21.15.175:0/431609759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 msgr2=0x7f71c00af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.345+0000 7f71c704f700 1 --2- 172.21.15.175:0/431609759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c00af910 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f71b80098e0 tx=0x7f71b80091a0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.345+0000 7f71c704f700 1 -- 172.21.15.175:0/431609759 shutdown_connections 2023-10-10T19:07:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.345+0000 7f71c704f700 1 --2- 172.21.15.175:0/431609759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c00af910 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.345+0000 7f71c704f700 1 --2- 172.21.15.175:0/431609759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c0068a40 0x7f71c0063c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.345+0000 7f71c704f700 1 -- 172.21.15.175:0/431609759 >> 172.21.15.175:0/431609759 conn(0x7f71c006dbe0 msgr2=0x7f71c00ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.345+0000 7f71c704f700 1 -- 172.21.15.175:0/431609759 shutdown_connections 2023-10-10T19:07:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.345+0000 7f71c704f700 1 -- 172.21.15.175:0/431609759 wait complete. 2023-10-10T19:07:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.346+0000 7f71c704f700 1 Processor -- start 2023-10-10T19:07:54.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.346+0000 7f71c704f700 1 -- start start 2023-10-10T19:07:54.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71c704f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71c0068a40 0x7f71c01690f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:54.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71c704f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c01676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:54.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71c704f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c00cac80 0x7f71c0167bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:54.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71c704f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c014e750 con 0x7f71c0068e10 2023-10-10T19:07:54.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71c704f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c0169d50 con 0x7f71c0068a40 2023-10-10T19:07:54.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71c704f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c00b0120 con 0x7f71c00cac80 2023-10-10T19:07:54.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c01676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:54.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c01676b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49934/0 (socket says 172.21.15.175:49934) 2023-10-10T19:07:54.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 -- 172.21.15.175:0/978527601 learned_addr learned my addr 172.21.15.175:0/978527601 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:54.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71c55ec700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c00cac80 0x7f71c0167bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:54.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71c4deb700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71c0068a40 0x7f71c01690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:54.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 -- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c00cac80 msgr2=0x7f71c0167bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:54.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c00cac80 0x7f71c0167bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 -- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71c0068a40 msgr2=0x7f71c01690f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:54.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71c0068a40 0x7f71c01690f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 -- 172.21.15.175:0/978527601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b8009420 con 0x7f71c0068e10 2023-10-10T19:07:54.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.347+0000 7f71b7fff700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c01676b0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f71b000c5e0 tx=0x7f71b0007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:54.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.348+0000 7f71b4ff9700 1 -- 172.21.15.175:0/978527601 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b00072e0 con 0x7f71c0068e10 2023-10-10T19:07:54.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.348+0000 7f71b4ff9700 1 -- 172.21.15.175:0/978527601 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b0007440 con 0x7f71c0068e10 2023-10-10T19:07:54.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.348+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c0169a70 con 0x7f71c0068e10 2023-10-10T19:07:54.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.348+0000 7f71b4ff9700 1 -- 172.21.15.175:0/978527601 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b00057f0 con 0x7f71c0068e10 2023-10-10T19:07:54.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.348+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c00c1260 con 0x7f71c0068e10 2023-10-10T19:07:54.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.349+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c0067570 con 0x7f71c0068e10 2023-10-10T19:07:54.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.349+0000 7f71b4ff9700 1 -- 172.21.15.175:0/978527601 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f71b0005c00 con 0x7f71c0068e10 2023-10-10T19:07:54.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.349+0000 7f71b4ff9700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f71ac066f30 0x7f71ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:54.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.350+0000 7f71b4ff9700 1 -- 172.21.15.175:0/978527601 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f71b0083ee0 con 0x7f71c0068e10 2023-10-10T19:07:54.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.350+0000 7f71c4deb700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f71ac066f30 0x7f71ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:54.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.350+0000 7f71c4deb700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f71ac066f30 0x7f71ac0693e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009fd0 tx=0x7f71bc00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:54.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.354+0000 7f71b4ff9700 1 -- 172.21.15.175:0/978527601 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71b0050d30 con 0x7f71c0068e10 2023-10-10T19:07:54.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.579+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71c00c6990 con 0x7f71c0068e10 2023-10-10T19:07:54.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.579+0000 7f71b4ff9700 1 -- 172.21.15.175:0/978527601 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+5442 (secure 0 0 0) 0x7f71b0017070 con 0x7f71c0068e10 2023-10-10T19:07:54.580 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:54.580 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:53.780650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:54.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f71ac066f30 msgr2=0x7f71ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:54.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f71ac066f30 0x7f71ac0693e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009fd0 tx=0x7f71bc00b040 comp rx=0 tx=0).stop 2023-10-10T19:07:54.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 msgr2=0x7f71c01676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:54.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c01676b0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f71b000c5e0 tx=0x7f71b0007e80 comp rx=0 tx=0).stop 2023-10-10T19:07:54.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 shutdown_connections 2023-10-10T19:07:54.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f71ac066f30 0x7f71ac0693e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71c00cac80 0x7f71c0167bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71c0068a40 0x7f71c01690f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 --2- 172.21.15.175:0/978527601 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71c0068e10 0x7f71c01676b0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:54.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 >> 172.21.15.175:0/978527601 conn(0x7f71c006dbe0 msgr2=0x7f71c00c45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:54.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 shutdown_connections 2023-10-10T19:07:54.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:54.581+0000 7f71c704f700 1 -- 172.21.15.175:0/978527601 wait complete. 2023-10-10T19:07:54.585 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 174 2023-10-10T19:07:54.594 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=2, replacing gid=5367 2023-10-10T19:07:54.594 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.c 2023-10-10T19:07:54.759 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:07:54.759+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-10T19:07:56.595 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-10T19:07:56.596 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-10T19:07:56.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-10T19:07:56.599 INFO:tasks.ceph.mds.c:Started 2023-10-10T19:07:56.599 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:07:56.681 INFO:tasks.ceph.mds.c.smithi175.stdout:starting mds.c at 2023-10-10T19:07:56.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.728+0000 7f1190c49700 1 Processor -- start 2023-10-10T19:07:56.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.728+0000 7f1190c49700 1 -- start start 2023-10-10T19:07:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f1190c49700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f118c14e090 0x7f118c153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f1190c49700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c15e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f1190c49700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c157b10 con 0x7f118c063640 2023-10-10T19:07:56.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f1190c49700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118c0c5200 con 0x7f118c14e090 2023-10-10T19:07:56.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f1190c49700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c068ab0 con 0x7f118c0c2d20 2023-10-10T19:07:56.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f118a59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f118c14e090 0x7f118c153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:56.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f118a59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f118c14e090 0x7f118c153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34838/0 (socket says 172.21.15.175:34838) 2023-10-10T19:07:56.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f118a59c700 1 -- 172.21.15.175:0/4220090120 learned_addr learned my addr 172.21.15.175:0/4220090120 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:56.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.729+0000 7f118ad9d700 1 --2- 172.21.15.175:0/4220090120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c15e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:56.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f118a59c700 1 -- 172.21.15.175:0/4220090120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 msgr2=0x7f118c15e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:07:56.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f118a59c700 1 --2- 172.21.15.175:0/4220090120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c15e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f118a59c700 1 -- 172.21.15.175:0/4220090120 >> v1:172.21.15.191:6789/0 conn(0x7f118c063640 legacy=0x7f118c0c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:56.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f118a59c700 1 -- 172.21.15.175:0/4220090120 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c068d70 con 0x7f118c14e090 2023-10-10T19:07:56.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f118a59c700 1 --2- 172.21.15.175:0/4220090120 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c14e090 0x7f118c153ea0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f117c0099b0 tx=0x7f117c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1590f24f67e51141 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:56.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f118959a700 1 -- 172.21.15.175:0/4220090120 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c00b860 con 0x7f118c14e090 2023-10-10T19:07:56.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f118959a700 1 -- 172.21.15.175:0/4220090120 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c00b9c0 con 0x7f118c14e090 2023-10-10T19:07:56.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f118959a700 1 -- 172.21.15.175:0/4220090120 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c00bd60 con 0x7f118c14e090 2023-10-10T19:07:56.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f1190c49700 1 -- 172.21.15.175:0/4220090120 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c14e090 msgr2=0x7f118c153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:56.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f1190c49700 1 --2- 172.21.15.175:0/4220090120 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c14e090 0x7f118c153ea0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f117c0099b0 tx=0x7f117c009250 comp rx=0 tx=0).stop 2023-10-10T19:07:56.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f1190c49700 1 -- 172.21.15.175:0/4220090120 shutdown_connections 2023-10-10T19:07:56.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f1190c49700 1 --2- 172.21.15.175:0/4220090120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c15e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.730+0000 7f1190c49700 1 --2- 172.21.15.175:0/4220090120 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c14e090 0x7f118c153ea0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.731+0000 7f1190c49700 1 -- 172.21.15.175:0/4220090120 >> 172.21.15.175:0/4220090120 conn(0x7f118c0c5650 msgr2=0x7f118c0c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:56.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.731+0000 7f1190c49700 1 -- 172.21.15.175:0/4220090120 shutdown_connections 2023-10-10T19:07:56.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.731+0000 7f1190c49700 1 -- 172.21.15.175:0/4220090120 wait complete. 2023-10-10T19:07:56.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f1190c49700 1 Processor -- start 2023-10-10T19:07:56.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f1190c49700 1 -- start start 2023-10-10T19:07:56.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f1190c49700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c063640 0x7f118c148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:56.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f1190c49700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:56.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f1190c49700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c14e090 0x7f118c167710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:56.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f1190c49700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c15bd50 con 0x7f118c063640 2023-10-10T19:07:56.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f1190c49700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c1482e0 con 0x7f118c14e090 2023-10-10T19:07:56.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f1190c49700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c147d10 con 0x7f118c0c2d20 2023-10-10T19:07:56.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f118a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c063640 0x7f118c148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:56.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f118a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c063640 0x7f118c148770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34844/0 (socket says 172.21.15.175:34844) 2023-10-10T19:07:56.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f118a59c700 1 -- 172.21.15.175:0/1128372055 learned_addr learned my addr 172.21.15.175:0/1128372055 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:07:56.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.732+0000 7f118ad9d700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c14e090 0x7f118c167710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:56.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f1189d9b700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:56.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f118a59c700 1 -- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 msgr2=0x7f118c149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:56.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f118a59c700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c149c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f118a59c700 1 -- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c14e090 msgr2=0x7f118c167710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:56.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f118a59c700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c14e090 0x7f118c167710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f118a59c700 1 -- 172.21.15.175:0/1128372055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0094f0 con 0x7f118c063640 2023-10-10T19:07:56.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f1189d9b700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:07:56.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f118a59c700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c063640 0x7f118c148770 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f117c006fd0 tx=0x7f117c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:56.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f117a7fc700 1 -- 172.21.15.175:0/1128372055 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c024440 con 0x7f118c063640 2023-10-10T19:07:56.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f117a7fc700 1 -- 172.21.15.175:0/1128372055 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c0245c0 con 0x7f118c063640 2023-10-10T19:07:56.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c149330 con 0x7f118c063640 2023-10-10T19:07:56.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f117a7fc700 1 -- 172.21.15.175:0/1128372055 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c024980 con 0x7f118c063640 2023-10-10T19:07:56.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.733+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c147f70 con 0x7f118c063640 2023-10-10T19:07:56.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.734+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c067570 con 0x7f118c063640 2023-10-10T19:07:56.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.734+0000 7f117a7fc700 1 -- 172.21.15.175:0/1128372055 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f117c01a420 con 0x7f118c063640 2023-10-10T19:07:56.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.735+0000 7f117a7fc700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f116c066f30 0x7f116c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:07:56.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.735+0000 7f117a7fc700 1 -- 172.21.15.175:0/1128372055 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f117c085960 con 0x7f118c063640 2023-10-10T19:07:56.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.735+0000 7f1189d9b700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f116c066f30 0x7f116c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:07:56.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.735+0000 7f1189d9b700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f116c066f30 0x7f116c0693e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1180007fd0 tx=0x7f1180009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:07:56.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.739+0000 7f117a7fc700 1 -- 172.21.15.175:0/1128372055 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f117c0527b0 con 0x7f118c063640 2023-10-10T19:07:56.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.967+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f118c1574c0 con 0x7f118c063640 2023-10-10T19:07:56.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.967+0000 7f117a7fc700 1 -- 172.21.15.175:0/1128372055 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+6199 (secure 0 0 0) 0x7f117c055dd0 con 0x7f118c063640 2023-10-10T19:07:56.968 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:07:56.968 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:07:53.780650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:07:56.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f116c066f30 msgr2=0x7f116c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:56.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f116c066f30 0x7f116c0693e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1180007fd0 tx=0x7f1180009040 comp rx=0 tx=0).stop 2023-10-10T19:07:56.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c063640 msgr2=0x7f118c148770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:07:56.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c063640 0x7f118c148770 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f117c006fd0 tx=0x7f117c017000 comp rx=0 tx=0).stop 2023-10-10T19:07:56.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 shutdown_connections 2023-10-10T19:07:56.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f116c066f30 0x7f116c0693e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f118c14e090 0x7f118c167710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f118c063640 0x7f118c148770 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 --2- 172.21.15.175:0/1128372055 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f118c0c2d20 0x7f118c149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:07:56.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 >> 172.21.15.175:0/1128372055 conn(0x7f118c0c5650 msgr2=0x7f118c154470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:07:56.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 shutdown_connections 2023-10-10T19:07:56.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:07:56.969+0000 7f1190c49700 1 -- 172.21.15.175:0/1128372055 wait complete. 2023-10-10T19:07:56.972 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 175 2023-10-10T19:07:56.981 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-10T19:07:56.982 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 18 secs before thrashing 2023-10-10T19:08:00.223 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at f524150822a v0.12 2023-10-10T19:08:00.225 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.12 2023-10-10T19:08:00.618 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:08:00.618 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.13 2023-10-10T19:08:00.619 INFO:tasks.workunit.client.0.smithi175.stdout:v0.13 2023-10-10T19:08:00.620 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.13 2023-10-10T19:08:00.620 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.13 2023-10-10T19:08:07.117 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 2b32297267d v0.13 2023-10-10T19:08:07.118 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.13 2023-10-10T19:08:07.185 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:08:07.185 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.14 2023-10-10T19:08:07.185 INFO:tasks.workunit.client.0.smithi175.stdout:v0.14 2023-10-10T19:08:07.186 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.14 2023-10-10T19:08:07.186 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.14 2023-10-10T19:08:09.456 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2023-10-10T19:08:09.480 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2023-10-10T19:08:09.481 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.14 2023-10-10T19:08:09.566 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:08:09.566 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.15 2023-10-10T19:08:09.566 INFO:tasks.workunit.client.0.smithi175.stdout:v0.15 2023-10-10T19:08:09.567 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.15 2023-10-10T19:08:09.567 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.15 2023-10-10T19:08:13.230 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 34% (39/112) Updating files: 35% (40/112) Updating files: 36% (41/112) Updating files: 37% (42/112) Updating files: 38% (43/112) Updating files: 39% (44/112) Updating files: 40% (45/112) Updating files: 41% (46/112) Updating files: 42% (48/112) Updating files: 43% (49/112) Updating files: 44% (50/112) Updating files: 45% (51/112) Updating files: 46% (52/112) Updating files: 47% (53/112) Updating files: 48% (54/112) Updating files: 49% (55/112) Updating files: 50% (56/112) Updating files: 51% (58/112) Updating files: 52% (59/112) Updating files: 53% (60/112) Updating files: 54% (61/112) Updating files: 55% (62/112) Updating files: 56% (63/112) Updating files: 57% (64/112) Updating files: 58% (65/112) Updating files: 59% (67/112) Updating files: 60% (68/112) Updating files: 61% (69/112) Updating files: 62% (70/112) Updating files: 63% (71/112) Updating files: 64% (72/112) Updating files: 65% (73/112) Updating files: 66% (74/112) Updating files: 67% (76/112) Updating files: 68% (77/112) Updating files: 69% (78/112) Updating files: 70% (79/112) Updating files: 71% (80/112) Updating files: 72% (81/112) Updating files: 73% (82/112) Updating files: 74% (83/112) Updating files: 75% (84/112) Updating files: 76% (86/112) Updating files: 77% (87/112) Updating files: 78% (88/112) Updating files: 79% (89/112) Updating files: 80% (90/112) Updating files: 81% (91/112) Updating files: 82% (92/112) Updating files: 83% (93/112) Updating files: 84% (95/112) Updating files: 85% (96/112) Updating files: 86% (97/112) Updating files: 87% (98/112) Updating files: 88% (99/112) Updating files: 89% (100/112) Updating files: 90% (101/112) Updating files: 91% (102/112) Updating files: 92% (104/112) Updating files: 93% (105/112) Updating files: 94% (106/112) Updating files: 95% (107/112) Updating files: 96% (108/112) Updating files: 97% (109/112) Updating files: 98% (110/112) Updating files: 99% (111/112) Updating files: 100% (112/112) Updating files: 100% (112/112), done. 2023-10-10T19:08:13.255 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at ab3c421eb95 v0.15 2023-10-10T19:08:13.256 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.15 2023-10-10T19:08:13.294 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:08:13.294 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.16 2023-10-10T19:08:13.295 INFO:tasks.workunit.client.0.smithi175.stdout:v0.16 2023-10-10T19:08:13.295 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.16 2023-10-10T19:08:13.295 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.16 2023-10-10T19:08:14.982 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:15.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.106+0000 7f6dfa759700 1 Processor -- start 2023-10-10T19:08:15.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.106+0000 7f6dfa759700 1 -- start start 2023-10-10T19:08:15.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6dfa759700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df408a100 0x7f6df4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6dfa759700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6df40cd040 0x7f6df40684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6dfa759700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df4067fa0 con 0x7f6df414fd90 2023-10-10T19:08:15.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6dfa759700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df40c5600 con 0x7f6df40cd040 2023-10-10T19:08:15.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6dfa759700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df4156b70 con 0x7f6df408a100 2023-10-10T19:08:15.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df9f58700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6df40cd040 0x7f6df40684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df9f58700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6df40cd040 0x7f6df40684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43794/0 (socket says 172.21.15.175:43794) 2023-10-10T19:08:15.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df9f58700 1 -- 172.21.15.175:0/1270947839 learned_addr learned my addr 172.21.15.175:0/1270947839 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:15.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df8f56700 1 --2- 172.21.15.175:0/1270947839 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df408a100 0x7f6df4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df9f58700 1 -- 172.21.15.175:0/1270947839 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df408a100 msgr2=0x7f6df4159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:15.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df9f58700 1 --2- 172.21.15.175:0/1270947839 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df408a100 0x7f6df4159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df9f58700 1 -- 172.21.15.175:0/1270947839 >> v1:172.21.15.191:6789/0 conn(0x7f6df414fd90 legacy=0x7f6df40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:15.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df9f58700 1 -- 172.21.15.175:0/1270947839 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df4150160 con 0x7f6df40cd040 2023-10-10T19:08:15.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6df9f58700 1 --2- 172.21.15.175:0/1270947839 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df40684f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009f20 tx=0x7f6dec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a060ba53dc6eb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:15.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.107+0000 7f6debfff700 1 -- 172.21.15.175:0/1270947839 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00e070 con 0x7f6df40cd040 2023-10-10T19:08:15.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6debfff700 1 -- 172.21.15.175:0/1270947839 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dec00ba80 con 0x7f6df40cd040 2023-10-10T19:08:15.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6debfff700 1 -- 172.21.15.175:0/1270947839 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec0092e0 con 0x7f6df40cd040 2023-10-10T19:08:15.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6dfa759700 1 -- 172.21.15.175:0/1270947839 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 msgr2=0x7f6df40684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6dfa759700 1 --2- 172.21.15.175:0/1270947839 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df40684f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009f20 tx=0x7f6dec009840 comp rx=0 tx=0).stop 2023-10-10T19:08:15.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6dfa759700 1 -- 172.21.15.175:0/1270947839 shutdown_connections 2023-10-10T19:08:15.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6dfa759700 1 --2- 172.21.15.175:0/1270947839 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df40684f0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6dfa759700 1 --2- 172.21.15.175:0/1270947839 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df408a100 0x7f6df4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6dfa759700 1 -- 172.21.15.175:0/1270947839 >> 172.21.15.175:0/1270947839 conn(0x7f6df4063d00 msgr2=0x7f6df40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:15.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6dfa759700 1 -- 172.21.15.175:0/1270947839 shutdown_connections 2023-10-10T19:08:15.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.108+0000 7f6dfa759700 1 -- 172.21.15.175:0/1270947839 wait complete. 2023-10-10T19:08:15.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.109+0000 7f6dfa759700 1 Processor -- start 2023-10-10T19:08:15.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.109+0000 7f6dfa759700 1 -- start start 2023-10-10T19:08:15.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6dfa759700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df408a100 0x7f6df40c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6dfa759700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df415ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6dfa759700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df414fd90 0x7f6df41697b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6dfa759700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df41536f0 con 0x7f6df40cd040 2023-10-10T19:08:15.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6dfa759700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40c6c80 con 0x7f6df408a100 2023-10-10T19:08:15.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6dfa759700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df4169320 con 0x7f6df414fd90 2023-10-10T19:08:15.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df8f56700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df415ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df8f56700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df415ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43798/0 (socket says 172.21.15.175:43798) 2023-10-10T19:08:15.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df8f56700 1 -- 172.21.15.175:0/3760939772 learned_addr learned my addr 172.21.15.175:0/3760939772 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:15.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df9757700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df408a100 0x7f6df40c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df9f58700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df414fd90 0x7f6df41697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df8f56700 1 -- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df414fd90 msgr2=0x7f6df41697b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df8f56700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df414fd90 0x7f6df41697b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df8f56700 1 -- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df408a100 msgr2=0x7f6df40c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df8f56700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df408a100 0x7f6df40c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.110+0000 7f6df8f56700 1 -- 172.21.15.175:0/3760939772 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec009ae0 con 0x7f6df40cd040 2023-10-10T19:08:15.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.111+0000 7f6df8f56700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df415ad30 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6de400d890 tx=0x7f6de4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:15.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.111+0000 7f6de97fa700 1 -- 172.21.15.175:0/3760939772 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de40125f0 con 0x7f6df40cd040 2023-10-10T19:08:15.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.111+0000 7f6de97fa700 1 -- 172.21.15.175:0/3760939772 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de4012750 con 0x7f6df40cd040 2023-10-10T19:08:15.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.111+0000 7f6de97fa700 1 -- 172.21.15.175:0/3760939772 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de4012af0 con 0x7f6df40cd040 2023-10-10T19:08:15.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.111+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df4152e50 con 0x7f6df40cd040 2023-10-10T19:08:15.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.111+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df4153410 con 0x7f6df40cd040 2023-10-10T19:08:15.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.112+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df41690f0 con 0x7f6df40cd040 2023-10-10T19:08:15.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.112+0000 7f6de97fa700 1 -- 172.21.15.175:0/3760939772 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6de401e460 con 0x7f6df40cd040 2023-10-10T19:08:15.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.112+0000 7f6de97fa700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dd8066f30 0x7f6dd80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.113+0000 7f6de97fa700 1 -- 172.21.15.175:0/3760939772 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f6de4084ef0 con 0x7f6df40cd040 2023-10-10T19:08:15.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.113+0000 7f6df9757700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dd8066f30 0x7f6dd80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.113+0000 7f6df9757700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dd8066f30 0x7f6dd80693e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6df00039e0 tx=0x7f6df0007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:15.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.117+0000 7f6de97fa700 1 -- 172.21.15.175:0/3760939772 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6de40557d0 con 0x7f6df40cd040 2023-10-10T19:08:15.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.348+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6df4169490 con 0x7f6df40cd040 2023-10-10T19:08:15.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.349+0000 7f6de97fa700 1 -- 172.21.15.175:0/3760939772 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f6de4055360 con 0x7f6df40cd040 2023-10-10T19:08:15.351 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:15.351 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:04.652754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:15.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dd8066f30 msgr2=0x7f6dd80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dd8066f30 0x7f6dd80693e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6df00039e0 tx=0x7f6df0007ec0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 msgr2=0x7f6df415ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df415ad30 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6de400d890 tx=0x7f6de4008040 comp rx=0 tx=0).stop 2023-10-10T19:08:15.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 shutdown_connections 2023-10-10T19:08:15.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6dd8066f30 0x7f6dd80693e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6df414fd90 0x7f6df41697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6df408a100 0x7f6df40c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 --2- 172.21.15.175:0/3760939772 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6df40cd040 0x7f6df415ad30 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 >> 172.21.15.175:0/3760939772 conn(0x7f6df4063d00 msgr2=0x7f6df40c2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:15.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 shutdown_connections 2023-10-10T19:08:15.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.351+0000 7f6dfa759700 1 -- 172.21.15.175:0/3760939772 wait complete. 2023-10-10T19:08:15.355 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 176 2023-10-10T19:08:15.364 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=3) 2023-10-10T19:08:15.365 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-10T19:08:15.365 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:08:15.376 INFO:tasks.ceph.mds.f:Stopped 2023-10-10T19:08:15.376 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:15.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.491+0000 7fc8bc444700 1 Processor -- start 2023-10-10T19:08:15.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.491+0000 7fc8bc444700 1 -- start start 2023-10-10T19:08:15.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8bc444700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b4155fb0 0x7fc8b4159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8bc444700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b41574b0 con 0x7fc8b40b1fc0 2023-10-10T19:08:15.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8bc444700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40afad0 con 0x7fc8b40b2390 2023-10-10T19:08:15.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8bc444700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4150770 con 0x7fc8b4155fb0 2023-10-10T19:08:15.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8ba1e0700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc8b40b1fc0 0x7fc8b4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48644/0 (socket says 172.21.15.175:48644) 2023-10-10T19:08:15.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8ba1e0700 1 -- 172.21.15.175:0/3569069958 learned_addr learned my addr 172.21.15.175:0/3569069958 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:15.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286791714 0 0) 0x7fc8b41574b0 con 0x7fc8b40b1fc0 2023-10-10T19:08:15.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8ba9e1700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b4155fb0 0x7fc8b4159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0003570 con 0x7fc8b40b1fc0 2023-10-10T19:08:15.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2098760619 0 0) 0x7fc8a0003570 con 0x7fc8b40b1fc0 2023-10-10T19:08:15.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b4155fb0 msgr2=0x7fc8b4159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:15.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b4155fb0 0x7fc8b4159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 >> v1:172.21.15.191:6789/0 conn(0x7fc8b40b2390 legacy=0x7fc8b40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40af4c0 con 0x7fc8b40b1fc0 2023-10-10T19:08:15.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fc8b0001470 con 0x7fc8b40b1fc0 2023-10-10T19:08:15.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 >> v1:172.21.15.175:6789/0 conn(0x7fc8b40b1fc0 legacy=0x7fc8b4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b4155fb0 0x7fc8a0003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8a0004110 0x7fc8a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8a00052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4150770 con 0x7fc8a0004e70 2023-10-10T19:08:15.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a000bcc0 con 0x7fc8a0004110 2023-10-10T19:08:15.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a000be80 con 0x7fc8b4155fb0 2023-10-10T19:08:15.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8ba9e1700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8a00052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8ba1e0700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b4155fb0 0x7fc8a0003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.492+0000 7fc8b99df700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8a0004110 0x7fc8a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8ba9e1700 1 -- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b4155fb0 msgr2=0x7fc8a0003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:15.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8ba9e1700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b4155fb0 0x7fc8a0003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8ba9e1700 1 -- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8a0004110 msgr2=0x7fc8a0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:15.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8ba9e1700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8a0004110 0x7fc8a0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8ba9e1700 1 -- 172.21.15.175:0/3569069958 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a0003570 con 0x7fc8a0004e70 2023-10-10T19:08:15.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8ba9e1700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8a00052e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac007440 tx=0x7fc8ac009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7887f3686945be server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:15.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac009a40 con 0x7fc8a0004e70 2023-10-10T19:08:15.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac007cf0 con 0x7fc8a0004e70 2023-10-10T19:08:15.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8b91de700 1 -- 172.21.15.175:0/3569069958 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac014630 con 0x7fc8a0004e70 2023-10-10T19:08:15.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8bc444700 1 -- 172.21.15.175:0/3569069958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 msgr2=0x7fc8a00052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8bc444700 1 --2- 172.21.15.175:0/3569069958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8a00052e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac007440 tx=0x7fc8ac009710 comp rx=0 tx=0).stop 2023-10-10T19:08:15.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.493+0000 7fc8ba9e1700 1 -- 172.21.15.175:0/3569069958 reap_dead start 2023-10-10T19:08:15.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.494+0000 7fc8bc444700 1 -- 172.21.15.175:0/3569069958 shutdown_connections 2023-10-10T19:08:15.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.494+0000 7fc8bc444700 1 -- 172.21.15.175:0/3569069958 >> 172.21.15.175:0/3569069958 conn(0x7fc8b4063230 msgr2=0x7fc8b4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:15.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.494+0000 7fc8bc444700 1 -- 172.21.15.175:0/3569069958 shutdown_connections 2023-10-10T19:08:15.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.494+0000 7fc8bc444700 1 -- 172.21.15.175:0/3569069958 wait complete. 2023-10-10T19:08:15.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8bc444700 1 Processor -- start 2023-10-10T19:08:15.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8bc444700 1 -- start start 2023-10-10T19:08:15.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8bc444700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8b4158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8bc444700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b41668b0 0x7fc8b41564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8bc444700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8b40c3090 0x7fc8b4156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8bc444700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4158400 con 0x7fc8a0004e70 2023-10-10T19:08:15.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8bc444700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4166d20 con 0x7fc8b40c3090 2023-10-10T19:08:15.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8bc444700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4159cf0 con 0x7fc8b41668b0 2023-10-10T19:08:15.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.495+0000 7fc8ba1e0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8b4158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba1e0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8b4158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43812/0 (socket says 172.21.15.175:43812) 2023-10-10T19:08:15.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba1e0700 1 -- 172.21.15.175:0/2137069406 learned_addr learned my addr 172.21.15.175:0/2137069406 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:15.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba9e1700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8b40c3090 0x7fc8b4156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8b99df700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b41668b0 0x7fc8b41564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba1e0700 1 -- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b41668b0 msgr2=0x7fc8b41564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba1e0700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b41668b0 0x7fc8b41564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba1e0700 1 -- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8b40c3090 msgr2=0x7fc8b4156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba1e0700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8b40c3090 0x7fc8b4156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba1e0700 1 -- 172.21.15.175:0/2137069406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4159fd0 con 0x7fc8a0004e70 2023-10-10T19:08:15.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8ba1e0700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8b4158bc0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0007390 tx=0x7fc8b40712f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:15.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8aa7fc700 1 -- 172.21.15.175:0/2137069406 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b000f6c0 con 0x7fc8a0004e70 2023-10-10T19:08:15.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8aa7fc700 1 -- 172.21.15.175:0/2137069406 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b000f840 con 0x7fc8a0004e70 2023-10-10T19:08:15.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8aa7fc700 1 -- 172.21.15.175:0/2137069406 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b000fc00 con 0x7fc8a0004e70 2023-10-10T19:08:15.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b41586a0 con 0x7fc8a0004e70 2023-10-10T19:08:15.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.496+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b40c0e20 con 0x7fc8a0004e70 2023-10-10T19:08:15.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.497+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b4158890 con 0x7fc8a0004e70 2023-10-10T19:08:15.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.497+0000 7fc8aa7fc700 1 -- 172.21.15.175:0/2137069406 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc8b001d460 con 0x7fc8a0004e70 2023-10-10T19:08:15.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.498+0000 7fc8aa7fc700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc898066f30 0x7fc8980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:15.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.498+0000 7fc8aa7fc700 1 -- 172.21.15.175:0/2137069406 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7fc8b0084a70 con 0x7fc8a0004e70 2023-10-10T19:08:15.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.498+0000 7fc8b99df700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc898066f30 0x7fc8980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:15.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.498+0000 7fc8b99df700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc898066f30 0x7fc8980693e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4159970 tx=0x7fc8a4007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:15.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.502+0000 7fc8aa7fc700 1 -- 172.21.15.175:0/2137069406 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8b00518c0 con 0x7fc8a0004e70 2023-10-10T19:08:15.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.726+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8b4067570 con 0x7fc8a0004e70 2023-10-10T19:08:15.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.727+0000 7fc8aa7fc700 1 -- 172.21.15.175:0/2137069406 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6202 (secure 0 0 0) 0x7fc8b0054ee0 con 0x7fc8a0004e70 2023-10-10T19:08:15.728 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:15.729 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:04.652754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:15.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc898066f30 msgr2=0x7fc8980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc898066f30 0x7fc8980693e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4159970 tx=0x7fc8a4007d30 comp rx=0 tx=0).stop 2023-10-10T19:08:15.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 msgr2=0x7fc8b4158bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:15.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8b4158bc0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0007390 tx=0x7fc8b40712f0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 shutdown_connections 2023-10-10T19:08:15.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc898066f30 0x7fc8980693e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8b40c3090 0x7fc8b4156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8a0004e70 0x7fc8b4158bc0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 --2- 172.21.15.175:0/2137069406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8b41668b0 0x7fc8b41564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:15.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 >> 172.21.15.175:0/2137069406 conn(0x7fc8b4063230 msgr2=0x7fc8b40cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:15.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 shutdown_connections 2023-10-10T19:08:15.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:15.729+0000 7fc8bc444700 1 -- 172.21.15.175:0/2137069406 wait complete. 2023-10-10T19:08:15.734 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 176 2023-10-10T19:08:15.742 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:08:17.743 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:17.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.870+0000 7f4854fa9700 1 Processor -- start 2023-10-10T19:08:17.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.870+0000 7f4854fa9700 1 -- start start 2023-10-10T19:08:17.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f4854fa9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f48500b1fc0 0x7f4850155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:17.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f4854fa9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48500b2390 0x7f4850068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:17.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f4854fa9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48501574b0 con 0x7f4850068bd0 2023-10-10T19:08:17.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f4854fa9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4850150770 con 0x7f48500b1fc0 2023-10-10T19:08:17.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f4854fa9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48500afad0 con 0x7f48500b2390 2023-10-10T19:08:17.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f484dd9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48500b2390 0x7f4850068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:17.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f484e59c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f48500b1fc0 0x7f4850155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:17.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f484dd9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48500b2390 0x7f4850068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:55180/0 (socket says 172.21.15.175:55180) 2023-10-10T19:08:17.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f484ed9d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4850068bd0 0x7f48500aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:47106/0 (socket says 172.21.15.175:47106) 2023-10-10T19:08:17.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.871+0000 7f484dd9b700 1 -- 172.21.15.175:0/3717701296 learned_addr learned my addr 172.21.15.175:0/3717701296 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:17.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484e59c700 1 -- 172.21.15.175:0/3717701296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48500b2390 msgr2=0x7f4850068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:17.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484e59c700 1 --2- 172.21.15.175:0/3717701296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48500b2390 0x7f4850068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:17.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484e59c700 1 -- 172.21.15.175:0/3717701296 >> v1:172.21.15.191:6789/0 conn(0x7f4850068bd0 legacy=0x7f48500aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:17.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484e59c700 1 -- 172.21.15.175:0/3717701296 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48500af4c0 con 0x7f48500b1fc0 2023-10-10T19:08:17.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484d59a700 1 -- 172.21.15.175:0/3717701296 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036183315 0 0) 0x7f48501574b0 con 0x7f4850068bd0 2023-10-10T19:08:17.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484dd9b700 1 --2- 172.21.15.175:0/3717701296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48500b2390 0x7f4850068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:08:17.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484e59c700 1 --2- 172.21.15.175:0/3717701296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b1fc0 0x7f4850155fb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f48400098e0 tx=0x7f48400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90f2eb6534af1a6b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:17.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484d59a700 1 -- 172.21.15.175:0/3717701296 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000b6d0 con 0x7f48500b1fc0 2023-10-10T19:08:17.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484d59a700 1 -- 172.21.15.175:0/3717701296 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484000b830 con 0x7f48500b1fc0 2023-10-10T19:08:17.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.872+0000 7f484d59a700 1 -- 172.21.15.175:0/3717701296 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000bbd0 con 0x7f48500b1fc0 2023-10-10T19:08:17.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.873+0000 7f4854fa9700 1 -- 172.21.15.175:0/3717701296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b1fc0 msgr2=0x7f4850155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:17.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.873+0000 7f4854fa9700 1 --2- 172.21.15.175:0/3717701296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b1fc0 0x7f4850155fb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f48400098e0 tx=0x7f48400091a0 comp rx=0 tx=0).stop 2023-10-10T19:08:17.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.873+0000 7f4854fa9700 1 -- 172.21.15.175:0/3717701296 shutdown_connections 2023-10-10T19:08:17.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.873+0000 7f4854fa9700 1 --2- 172.21.15.175:0/3717701296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b1fc0 0x7f4850155fb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:17.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.873+0000 7f4854fa9700 1 --2- 172.21.15.175:0/3717701296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48500b2390 0x7f4850068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:17.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.873+0000 7f4854fa9700 1 -- 172.21.15.175:0/3717701296 >> 172.21.15.175:0/3717701296 conn(0x7f4850063230 msgr2=0x7f4850063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:17.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.873+0000 7f4854fa9700 1 -- 172.21.15.175:0/3717701296 shutdown_connections 2023-10-10T19:08:17.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.873+0000 7f4854fa9700 1 -- 172.21.15.175:0/3717701296 wait complete. 2023-10-10T19:08:17.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.874+0000 7f4854fa9700 1 Processor -- start 2023-10-10T19:08:17.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.874+0000 7f4854fa9700 1 -- start start 2023-10-10T19:08:17.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f4854fa9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4850068bd0 0x7f4850158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:17.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f4854fa9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48500b1fc0 0x7f4850157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:17.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f4854fa9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b2390 0x7f48501584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:17.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f4854fa9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48501666b0 con 0x7f48500b1fc0 2023-10-10T19:08:17.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f4854fa9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485015c0b0 con 0x7f48500b2390 2023-10-10T19:08:17.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f4854fa9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485015c390 con 0x7f4850068bd0 2023-10-10T19:08:17.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48500b1fc0 0x7f4850157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:17.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48500b1fc0 0x7f4850157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43828/0 (socket says 172.21.15.175:43828) 2023-10-10T19:08:17.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 -- 172.21.15.175:0/2743011044 learned_addr learned my addr 172.21.15.175:0/2743011044 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:17.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484ed9d700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b2390 0x7f48501584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:17.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484e59c700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4850068bd0 0x7f4850158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:17.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 -- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4850068bd0 msgr2=0x7f4850158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:17.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4850068bd0 0x7f4850158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:17.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 -- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b2390 msgr2=0x7f48501584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:17.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b2390 0x7f48501584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:17.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 -- 172.21.15.175:0/2743011044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48440094f0 con 0x7f48500b1fc0 2023-10-10T19:08:17.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f484dd9b700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48500b1fc0 0x7f4850157f60 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f484400ec50 tx=0x7f4844009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:17.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f483e7fc700 1 -- 172.21.15.175:0/2743011044 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484400c850 con 0x7f48500b1fc0 2023-10-10T19:08:17.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f483e7fc700 1 -- 172.21.15.175:0/2743011044 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484400c9b0 con 0x7f48500b1fc0 2023-10-10T19:08:17.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4840009420 con 0x7f48500b1fc0 2023-10-10T19:08:17.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f483e7fc700 1 -- 172.21.15.175:0/2743011044 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484400cd30 con 0x7f48500b1fc0 2023-10-10T19:08:17.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.875+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4850166340 con 0x7f48500b1fc0 2023-10-10T19:08:17.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.876+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4850166860 con 0x7f48500b1fc0 2023-10-10T19:08:17.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.877+0000 7f483e7fc700 1 -- 172.21.15.175:0/2743011044 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4844013680 con 0x7f48500b1fc0 2023-10-10T19:08:17.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.877+0000 7f483e7fc700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4830066f60 0x7f4830069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:17.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.877+0000 7f483e7fc700 1 -- 172.21.15.175:0/2743011044 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f4844084b40 con 0x7f48500b1fc0 2023-10-10T19:08:17.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.877+0000 7f484e59c700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4830066f60 0x7f4830069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:17.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.878+0000 7f484e59c700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4830066f60 0x7f4830069410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f48400091e0 tx=0x7f4840017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:17.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:17.881+0000 7f483e7fc700 1 -- 172.21.15.175:0/2743011044 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4850166860 con 0x7f48500b1fc0 2023-10-10T19:08:18.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.106+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4850067570 con 0x7f48500b1fc0 2023-10-10T19:08:18.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.106+0000 7f483e7fc700 1 -- 172.21.15.175:0/2743011044 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f4844051990 con 0x7f48500b1fc0 2023-10-10T19:08:18.108 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:18.108 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:04.652754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:18.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4830066f60 msgr2=0x7f4830069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:18.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4830066f60 0x7f4830069410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f48400091e0 tx=0x7f4840017000 comp rx=0 tx=0).stop 2023-10-10T19:08:18.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48500b1fc0 msgr2=0x7f4850157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:18.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48500b1fc0 0x7f4850157f60 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f484400ec50 tx=0x7f4844009250 comp rx=0 tx=0).stop 2023-10-10T19:08:18.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 shutdown_connections 2023-10-10T19:08:18.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4830066f60 0x7f4830069410 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:18.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48500b2390 0x7f48501584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:18.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4850068bd0 0x7f4850158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:18.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 --2- 172.21.15.175:0/2743011044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48500b1fc0 0x7f4850157f60 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:18.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 >> 172.21.15.175:0/2743011044 conn(0x7f4850063230 msgr2=0x7f48501526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:18.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 shutdown_connections 2023-10-10T19:08:18.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:18.108+0000 7f4854fa9700 1 -- 172.21.15.175:0/2743011044 wait complete. 2023-10-10T19:08:18.113 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 176 2023-10-10T19:08:18.121 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:08:20.122 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:20.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.249+0000 7f455e75c700 1 Processor -- start 2023-10-10T19:08:20.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.249+0000 7f455e75c700 1 -- start start 2023-10-10T19:08:20.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f455e75c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580b0240 0x7f455806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:20.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f455e75c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455806a8c0 con 0x7f455806bda0 2023-10-10T19:08:20.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f455e75c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45580c5ab0 con 0x7f45581595a0 2023-10-10T19:08:20.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f455e75c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4558159fb0 con 0x7f45580b0240 2023-10-10T19:08:20.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f455df5b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f455806bda0 0x7f4558160780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48648/0 (socket says 172.21.15.175:48648) 2023-10-10T19:08:20.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f455df5b700 1 -- 172.21.15.175:0/3692283273 learned_addr learned my addr 172.21.15.175:0/3692283273 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:20.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f455cf59700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580b0240 0x7f455806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:20.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227346865 0 0) 0x7f455806a8c0 con 0x7f455806bda0 2023-10-10T19:08:20.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4544003570 con 0x7f455806bda0 2023-10-10T19:08:20.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842174887 0 0) 0x7f45580c5ab0 con 0x7f45581595a0 2023-10-10T19:08:20.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.250+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455806a8c0 con 0x7f45581595a0 2023-10-10T19:08:20.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 644553400 0 0) 0x7f4544003570 con 0x7f455806bda0 2023-10-10T19:08:20.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580b0240 msgr2=0x7f455806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:20.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580b0240 0x7f455806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 >> v1:172.21.15.191:6789/0 conn(0x7f45581595a0 legacy=0x7f45580afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:20.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45580c57d0 con 0x7f455806bda0 2023-10-10T19:08:20.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f4554001470 con 0x7f455806bda0 2023-10-10T19:08:20.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 >> v1:172.21.15.175:6789/0 conn(0x7f455806bda0 legacy=0x7f4558160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:20.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544003c20 0x7f4544004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:20.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4544004810 0x7f4544004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:20.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580b0240 0x7f45440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:20.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4558159fb0 con 0x7f4544004810 2023-10-10T19:08:20.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454400bb40 con 0x7f4544003c20 2023-10-10T19:08:20.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454400bd00 con 0x7f45580b0240 2023-10-10T19:08:20.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455cf59700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4544004810 0x7f4544004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:20.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455d75a700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544003c20 0x7f4544004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:20.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455df5b700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580b0240 0x7f45440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:20.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455cf59700 1 -- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580b0240 msgr2=0x7f45440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:20.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455cf59700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580b0240 0x7f45440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455cf59700 1 -- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544003c20 msgr2=0x7f4544004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:20.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455cf59700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544003c20 0x7f4544004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455cf59700 1 -- 172.21.15.175:0/3692283273 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544003570 con 0x7f4544004810 2023-10-10T19:08:20.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f455cf59700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4544004810 0x7f4544004c80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f45580b0650 tx=0x7f45480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0ad7e6f1a110ff server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:20.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454800e710 con 0x7f4544004810 2023-10-10T19:08:20.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454800e870 con 0x7f4544004810 2023-10-10T19:08:20.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.251+0000 7f454ffff700 1 -- 172.21.15.175:0/3692283273 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454800ec30 con 0x7f4544004810 2023-10-10T19:08:20.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.252+0000 7f455e75c700 1 -- 172.21.15.175:0/3692283273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4544004810 msgr2=0x7f4544004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:20.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.252+0000 7f455e75c700 1 --2- 172.21.15.175:0/3692283273 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4544004810 0x7f4544004c80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f45580b0650 tx=0x7f45480091a0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.252+0000 7f455df5b700 1 -- 172.21.15.175:0/3692283273 reap_dead start 2023-10-10T19:08:20.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.252+0000 7f455e75c700 1 -- 172.21.15.175:0/3692283273 shutdown_connections 2023-10-10T19:08:20.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.252+0000 7f455e75c700 1 -- 172.21.15.175:0/3692283273 >> 172.21.15.175:0/3692283273 conn(0x7f4558063cd0 msgr2=0x7f455806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:20.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.252+0000 7f455e75c700 1 -- 172.21.15.175:0/3692283273 shutdown_connections 2023-10-10T19:08:20.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.252+0000 7f455e75c700 1 -- 172.21.15.175:0/3692283273 wait complete. 2023-10-10T19:08:20.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.253+0000 7f455e75c700 1 Processor -- start 2023-10-10T19:08:20.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.253+0000 7f455e75c700 1 -- start start 2023-10-10T19:08:20.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455e75c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544004810 0x7f45581696e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:20.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455e75c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580c9ff0 0x7f4558157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:20.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455e75c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f45581581e0 0x7f4558152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:20.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455e75c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45580aae90 con 0x7f45581581e0 2023-10-10T19:08:20.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455e75c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45580c1d60 con 0x7f4544004810 2023-10-10T19:08:20.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455e75c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4558169250 con 0x7f45580c9ff0 2023-10-10T19:08:20.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455df5b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f45581581e0 0x7f4558152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:20.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455df5b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f45581581e0 0x7f4558152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43856/0 (socket says 172.21.15.175:43856) 2023-10-10T19:08:20.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455df5b700 1 -- 172.21.15.175:0/1516478597 learned_addr learned my addr 172.21.15.175:0/1516478597 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:20.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455d75a700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544004810 0x7f45581696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:20.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455cf59700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580c9ff0 0x7f4558157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:20.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455df5b700 1 -- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580c9ff0 msgr2=0x7f4558157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:20.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455df5b700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580c9ff0 0x7f4558157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455df5b700 1 -- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544004810 msgr2=0x7f45581696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:20.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455df5b700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544004810 0x7f45581696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.254+0000 7f455df5b700 1 -- 172.21.15.175:0/1516478597 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45580c2320 con 0x7f45581581e0 2023-10-10T19:08:20.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.255+0000 7f455df5b700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f45581581e0 0x7f4558152b30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f45580cb760 tx=0x7f455806ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:20.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.255+0000 7f454d7fa700 1 -- 172.21.15.175:0/1516478597 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455400f670 con 0x7f45581581e0 2023-10-10T19:08:20.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.255+0000 7f454d7fa700 1 -- 172.21.15.175:0/1516478597 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455400f810 con 0x7f45581581e0 2023-10-10T19:08:20.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.255+0000 7f454d7fa700 1 -- 172.21.15.175:0/1516478597 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455400fbd0 con 0x7f45581581e0 2023-10-10T19:08:20.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.255+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45580c1a80 con 0x7f45581581e0 2023-10-10T19:08:20.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.255+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4558166720 con 0x7f45581581e0 2023-10-10T19:08:20.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.256+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4558067570 con 0x7f45581581e0 2023-10-10T19:08:20.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.256+0000 7f454d7fa700 1 -- 172.21.15.175:0/1516478597 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f455401e3b0 con 0x7f45581581e0 2023-10-10T19:08:20.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.256+0000 7f454d7fa700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f453c066f60 0x7f453c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:20.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.257+0000 7f454d7fa700 1 -- 172.21.15.175:0/1516478597 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f4554084420 con 0x7f45581581e0 2023-10-10T19:08:20.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.257+0000 7f455d75a700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f453c066f60 0x7f453c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:20.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.257+0000 7f455d75a700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f453c066f60 0x7f453c069410 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4550011250 tx=0x7f45500017a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:20.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.261+0000 7f454d7fa700 1 -- 172.21.15.175:0/1516478597 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f455404e5a0 con 0x7f45581581e0 2023-10-10T19:08:20.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.485+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4558169390 con 0x7f45581581e0 2023-10-10T19:08:20.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.485+0000 7f454d7fa700 1 -- 172.21.15.175:0/1516478597 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f4554051280 con 0x7f45581581e0 2023-10-10T19:08:20.487 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:20.487 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:04.652754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.487+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f453c066f60 msgr2=0x7f453c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.487+0000 7f455e75c700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f453c066f60 0x7f453c069410 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4550011250 tx=0x7f45500017a0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.487+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f45581581e0 msgr2=0x7f4558152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.487+0000 7f455e75c700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f45581581e0 0x7f4558152b30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f45580cb760 tx=0x7f455806ee90 comp rx=0 tx=0).stop 2023-10-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.488+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 shutdown_connections 2023-10-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.488+0000 7f455e75c700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f453c066f60 0x7f453c069410 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.488+0000 7f455e75c700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f45581581e0 0x7f4558152b30 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.488+0000 7f455e75c700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544004810 0x7f45581696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.488+0000 7f455e75c700 1 --2- 172.21.15.175:0/1516478597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45580c9ff0 0x7f4558157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.488+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 >> 172.21.15.175:0/1516478597 conn(0x7f4558063cd0 msgr2=0x7f455806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.488+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 shutdown_connections 2023-10-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:20.488+0000 7f455e75c700 1 -- 172.21.15.175:0/1516478597 wait complete. 2023-10-10T19:08:20.492 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 176 2023-10-10T19:08:20.501 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:08:21.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:08:21.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:08:22.502 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:22.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.628+0000 7f1097680700 1 Processor -- start 2023-10-10T19:08:22.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.628+0000 7f1097680700 1 -- start start 2023-10-10T19:08:22.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f1097680700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f109006e910 0x7f109014f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:22.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f1097680700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f109006ece0 0x7f109014fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:22.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f1097680700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f109006df80 con 0x7f10900c6210 2023-10-10T19:08:22.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f1097680700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f109006e4b0 con 0x7f109006e910 2023-10-10T19:08:22.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f1097680700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10900c4620 con 0x7f109006ece0 2023-10-10T19:08:22.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f109541c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f109006e910 0x7f109014f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:22.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f109541c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f109006e910 0x7f109014f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43864/0 (socket says 172.21.15.175:43864) 2023-10-10T19:08:22.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f109541c700 1 -- 172.21.15.175:0/2388720582 learned_addr learned my addr 172.21.15.175:0/2388720582 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:22.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f1094c1b700 1 --2- 172.21.15.175:0/2388720582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f109006ece0 0x7f109014fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:22.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f109541c700 1 -- 172.21.15.175:0/2388720582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f109006ece0 msgr2=0x7f109014fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:22.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f109541c700 1 --2- 172.21.15.175:0/2388720582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f109006ece0 0x7f109014fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f109541c700 1 -- 172.21.15.175:0/2388720582 >> v1:172.21.15.191:6789/0 conn(0x7f10900c6210 legacy=0x7f10900c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:22.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f109541c700 1 -- 172.21.15.175:0/2388720582 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109006e1d0 con 0x7f109006e910 2023-10-10T19:08:22.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.629+0000 7f109541c700 1 --2- 172.21.15.175:0/2388720582 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006e910 0x7f109014f600 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f108c0099b0 tx=0x7f108c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8d241aba4a15ff server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:22.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1087fff700 1 -- 172.21.15.175:0/2388720582 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c00b820 con 0x7f109006e910 2023-10-10T19:08:22.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1087fff700 1 -- 172.21.15.175:0/2388720582 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108c00b980 con 0x7f109006e910 2023-10-10T19:08:22.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1087fff700 1 -- 172.21.15.175:0/2388720582 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c00bd20 con 0x7f109006e910 2023-10-10T19:08:22.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1097680700 1 -- 172.21.15.175:0/2388720582 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006e910 msgr2=0x7f109014f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:22.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1097680700 1 --2- 172.21.15.175:0/2388720582 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006e910 0x7f109014f600 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f108c0099b0 tx=0x7f108c009250 comp rx=0 tx=0).stop 2023-10-10T19:08:22.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1097680700 1 -- 172.21.15.175:0/2388720582 shutdown_connections 2023-10-10T19:08:22.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1097680700 1 --2- 172.21.15.175:0/2388720582 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006e910 0x7f109014f600 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1097680700 1 --2- 172.21.15.175:0/2388720582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f109006ece0 0x7f109014fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1097680700 1 -- 172.21.15.175:0/2388720582 >> 172.21.15.175:0/2388720582 conn(0x7f10900c5650 msgr2=0x7f10900c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:22.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1097680700 1 -- 172.21.15.175:0/2388720582 shutdown_connections 2023-10-10T19:08:22.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.630+0000 7f1097680700 1 -- 172.21.15.175:0/2388720582 wait complete. 2023-10-10T19:08:22.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.631+0000 7f1097680700 1 Processor -- start 2023-10-10T19:08:22.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.631+0000 7f1097680700 1 -- start start 2023-10-10T19:08:22.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1097680700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f109006e910 0x7f1090167150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:22.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1097680700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006ece0 0x7f1090140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:22.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1097680700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10900c6210 0x7f1090140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:22.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1097680700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900bc6c0 con 0x7f109006ece0 2023-10-10T19:08:22.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1097680700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1090151cc0 con 0x7f109006e910 2023-10-10T19:08:22.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1097680700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f109006c3e0 con 0x7f10900c6210 2023-10-10T19:08:22.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1094c1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006ece0 0x7f1090140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:22.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1094c1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006ece0 0x7f1090140510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43878/0 (socket says 172.21.15.175:43878) 2023-10-10T19:08:22.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1094c1b700 1 -- 172.21.15.175:0/1469649221 learned_addr learned my addr 172.21.15.175:0/1469649221 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:22.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f109541c700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f109006e910 0x7f1090167150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:22.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1095c1d700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10900c6210 0x7f1090140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:22.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.632+0000 7f1094c1b700 1 -- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10900c6210 msgr2=0x7f1090140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:22.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f1094c1b700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10900c6210 0x7f1090140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f1094c1b700 1 -- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f109006e910 msgr2=0x7f1090167150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:22.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f1094c1b700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f109006e910 0x7f1090167150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f1094c1b700 1 -- 172.21.15.175:0/1469649221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c0094f0 con 0x7f109006ece0 2023-10-10T19:08:22.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f1095c1d700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10900c6210 0x7f1090140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:08:22.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f1094c1b700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006ece0 0x7f1090140510 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f108000c800 tx=0x7f1080008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:22.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f10857fa700 1 -- 172.21.15.175:0/1469649221 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1080011600 con 0x7f109006ece0 2023-10-10T19:08:22.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f10857fa700 1 -- 172.21.15.175:0/1469649221 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1080011760 con 0x7f109006ece0 2023-10-10T19:08:22.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10900bc8d0 con 0x7f109006ece0 2023-10-10T19:08:22.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f10857fa700 1 -- 172.21.15.175:0/1469649221 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1080011b20 con 0x7f109006ece0 2023-10-10T19:08:22.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.633+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1090151670 con 0x7f109006ece0 2023-10-10T19:08:22.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.634+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1090067570 con 0x7f109006ece0 2023-10-10T19:08:22.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.634+0000 7f10857fa700 1 -- 172.21.15.175:0/1469649221 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f10800193f0 con 0x7f109006ece0 2023-10-10T19:08:22.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.634+0000 7f10857fa700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1074066f30 0x7f10740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:22.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.635+0000 7f10857fa700 1 -- 172.21.15.175:0/1469649221 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f1080084f10 con 0x7f109006ece0 2023-10-10T19:08:22.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.635+0000 7f109541c700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1074066f30 0x7f10740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:22.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.635+0000 7f109541c700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1074066f30 0x7f10740693e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f108c0099b0 tx=0x7f108c016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:22.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.639+0000 7f10857fa700 1 -- 172.21.15.175:0/1469649221 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1080051d60 con 0x7f109006ece0 2023-10-10T19:08:22.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.864+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1090151e00 con 0x7f109006ece0 2023-10-10T19:08:22.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.865+0000 7f10857fa700 1 -- 172.21.15.175:0/1469649221 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f1080055380 con 0x7f109006ece0 2023-10-10T19:08:22.867 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:22.867 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:04.652754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:22.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1074066f30 msgr2=0x7f10740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:22.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1074066f30 0x7f10740693e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f108c0099b0 tx=0x7f108c016000 comp rx=0 tx=0).stop 2023-10-10T19:08:22.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006ece0 msgr2=0x7f1090140510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:22.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006ece0 0x7f1090140510 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f108000c800 tx=0x7f1080008040 comp rx=0 tx=0).stop 2023-10-10T19:08:22.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 shutdown_connections 2023-10-10T19:08:22.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1074066f30 0x7f10740693e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10900c6210 0x7f1090140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f109006e910 0x7f1090167150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 --2- 172.21.15.175:0/1469649221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f109006ece0 0x7f1090140510 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:22.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 >> 172.21.15.175:0/1469649221 conn(0x7f10900c5650 msgr2=0x7f1090146ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:22.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 shutdown_connections 2023-10-10T19:08:22.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:22.867+0000 7f1097680700 1 -- 172.21.15.175:0/1469649221 wait complete. 2023-10-10T19:08:22.871 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 176 2023-10-10T19:08:22.880 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:08:24.881 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:25.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.007+0000 7f40ef0f1700 1 Processor -- start 2023-10-10T19:08:25.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.007+0000 7f40ef0f1700 1 -- start start 2023-10-10T19:08:25.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.008+0000 7f40ef0f1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e80c3330 0x7f40e80c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.008+0000 7f40ef0f1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f40e806b640 0x7f40e815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.008+0000 7f40ef0f1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e81533e0 con 0x7f40e80c41e0 2023-10-10T19:08:25.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.008+0000 7f40ef0f1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e806e130 con 0x7f40e806b640 2023-10-10T19:08:25.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.008+0000 7f40ef0f1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e806a970 con 0x7f40e80c3330 2023-10-10T19:08:25.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.008+0000 7f40dffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f40e80c41e0 0x7f40e814cda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:43520/0 (socket says 172.21.15.175:43520) 2023-10-10T19:08:25.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.008+0000 7f40dffff700 1 -- 172.21.15.175:0/3788898967 learned_addr learned my addr 172.21.15.175:0/3788898967 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:25.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ed68e700 1 --2- 172.21.15.175:0/3788898967 >> v2:172.21.15.191:3300/0 conn(0x7f40e806b640 0x7f40e815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ece8d700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e80c3330 0x7f40e80c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601235827 0 0) 0x7f40e81533e0 con 0x7f40e80c41e0 2023-10-10T19:08:25.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d0003570 con 0x7f40e80c41e0 2023-10-10T19:08:25.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1363155616 0 0) 0x7f40d0003570 con 0x7f40e80c41e0 2023-10-10T19:08:25.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e80c3330 msgr2=0x7f40e80c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:25.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e80c3330 0x7f40e80c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 >> v2:172.21.15.191:3300/0 conn(0x7f40e806b640 msgr2=0x7f40e815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:25.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 --2- 172.21.15.175:0/3788898967 >> v2:172.21.15.191:3300/0 conn(0x7f40e806b640 0x7f40e815d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e806d520 con 0x7f40e80c41e0 2023-10-10T19:08:25.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f40d8001470 con 0x7f40e80c41e0 2023-10-10T19:08:25.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ed68e700 1 --2- 172.21.15.175:0/3788898967 >> v2:172.21.15.191:3300/0 conn(0x7f40e806b640 0x7f40e815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:08:25.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 >> v1:172.21.15.175:6789/0 conn(0x7f40e80c41e0 legacy=0x7f40e814cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:25.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 0x7f40d0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e80c3330 0x7f40d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40d0004d50 0x7f40d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e806e130 con 0x7f40d0004d50 2023-10-10T19:08:25.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e806a970 con 0x7f40d0003a60 2023-10-10T19:08:25.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d000ba60 con 0x7f40e80c3330 2023-10-10T19:08:25.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ece8d700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 0x7f40d0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ed68e700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40d0004d50 0x7f40d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40dffff700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e80c3330 0x7f40d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ece8d700 1 -- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e80c3330 msgr2=0x7f40d0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:25.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ece8d700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e80c3330 0x7f40d0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ece8d700 1 -- 172.21.15.175:0/3788898967 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40d0004d50 msgr2=0x7f40d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:25.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ece8d700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40d0004d50 0x7f40d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ece8d700 1 -- 172.21.15.175:0/3788898967 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d0003570 con 0x7f40d0003a60 2023-10-10T19:08:25.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.009+0000 7f40ed68e700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40d0004d50 0x7f40d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:08:25.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40ece8d700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 0x7f40d0003e70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f40e400e2d0 tx=0x7f40e400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=812a3afef2679919 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:25.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e400eb20 con 0x7f40d0003a60 2023-10-10T19:08:25.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e400f040 con 0x7f40d0003a60 2023-10-10T19:08:25.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40df7fe700 1 -- 172.21.15.175:0/3788898967 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e400eed0 con 0x7f40d0003a60 2023-10-10T19:08:25.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40ef0f1700 1 -- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 msgr2=0x7f40d0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:25.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40ef0f1700 1 --2- 172.21.15.175:0/3788898967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 0x7f40d0003e70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f40e400e2d0 tx=0x7f40e400b040 comp rx=0 tx=0).stop 2023-10-10T19:08:25.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40ed68e700 1 -- 172.21.15.175:0/3788898967 reap_dead start 2023-10-10T19:08:25.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40ef0f1700 1 -- 172.21.15.175:0/3788898967 shutdown_connections 2023-10-10T19:08:25.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40ef0f1700 1 -- 172.21.15.175:0/3788898967 >> 172.21.15.175:0/3788898967 conn(0x7f40e8068280 msgr2=0x7f40e806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:25.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40ef0f1700 1 -- 172.21.15.175:0/3788898967 shutdown_connections 2023-10-10T19:08:25.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.010+0000 7f40ef0f1700 1 -- 172.21.15.175:0/3788898967 wait complete. 2023-10-10T19:08:25.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.011+0000 7f40ef0f1700 1 Processor -- start 2023-10-10T19:08:25.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.012+0000 7f40ef0f1700 1 -- start start 2023-10-10T19:08:25.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.012+0000 7f40ef0f1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 0x7f40e8069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.012+0000 7f40ef0f1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e8153aa0 0x7f40e81643a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.012+0000 7f40ef0f1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e81648e0 0x7f40e8146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.012+0000 7f40ef0f1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e81530d0 con 0x7f40e8153aa0 2023-10-10T19:08:25.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.012+0000 7f40ef0f1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e8167f90 con 0x7f40d0003a60 2023-10-10T19:08:25.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.012+0000 7f40ef0f1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e81528c0 con 0x7f40e81648e0 2023-10-10T19:08:25.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e8153aa0 0x7f40e81643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e8153aa0 0x7f40e81643a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47908/0 (socket says 172.21.15.175:47908) 2023-10-10T19:08:25.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 -- 172.21.15.175:0/2479783855 learned_addr learned my addr 172.21.15.175:0/2479783855 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:25.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40ece8d700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 0x7f40e8069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40ed68e700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e81648e0 0x7f40e8146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 -- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e81648e0 msgr2=0x7f40e8146d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:25.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e81648e0 0x7f40e8146d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 -- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 msgr2=0x7f40e8069740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:25.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 0x7f40e8069740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 -- 172.21.15.175:0/2479783855 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e8168460 con 0x7f40e8153aa0 2023-10-10T19:08:25.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dffff700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e8153aa0 0x7f40e81643a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f40d80073f0 tx=0x7f40d8001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:25.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dcff9700 1 -- 172.21.15.175:0/2479783855 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d800fa70 con 0x7f40e8153aa0 2023-10-10T19:08:25.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dcff9700 1 -- 172.21.15.175:0/2479783855 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d800fbd0 con 0x7f40e8153aa0 2023-10-10T19:08:25.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e8152340 con 0x7f40e8153aa0 2023-10-10T19:08:25.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40dcff9700 1 -- 172.21.15.175:0/2479783855 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d80133b0 con 0x7f40e8153aa0 2023-10-10T19:08:25.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.013+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e8168180 con 0x7f40e8153aa0 2023-10-10T19:08:25.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.014+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e8068a40 con 0x7f40e8153aa0 2023-10-10T19:08:25.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.014+0000 7f40dcff9700 1 -- 172.21.15.175:0/2479783855 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f40d80137a0 con 0x7f40e8153aa0 2023-10-10T19:08:25.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.015+0000 7f40dcff9700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40d4066f60 0x7f40d4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:25.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.015+0000 7f40dcff9700 1 -- 172.21.15.175:0/2479783855 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f40d80848b0 con 0x7f40e8153aa0 2023-10-10T19:08:25.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.015+0000 7f40ece8d700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40d4066f60 0x7f40d4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:25.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.015+0000 7f40ece8d700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40d4066f60 0x7f40d4069410 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40e40094a0 tx=0x7f40e401a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:25.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.019+0000 7f40dcff9700 1 -- 172.21.15.175:0/2479783855 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f40d8051700 con 0x7f40e8153aa0 2023-10-10T19:08:25.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.241+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40e81640a0 con 0x7f40e8153aa0 2023-10-10T19:08:25.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.242+0000 7f40dcff9700 1 -- 172.21.15.175:0/2479783855 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f40d8054d20 con 0x7f40e8153aa0 2023-10-10T19:08:25.243 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:25.243 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:04.652754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:25.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40d4066f60 msgr2=0x7f40d4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:25.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40d4066f60 0x7f40d4069410 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40e40094a0 tx=0x7f40e401a000 comp rx=0 tx=0).stop 2023-10-10T19:08:25.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e8153aa0 msgr2=0x7f40e81643a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:25.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e8153aa0 0x7f40e81643a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f40d80073f0 tx=0x7f40d8001e90 comp rx=0 tx=0).stop 2023-10-10T19:08:25.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 shutdown_connections 2023-10-10T19:08:25.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40d4066f60 0x7f40d4069410 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40e81648e0 0x7f40e8146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40d0003a60 0x7f40e8069740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 --2- 172.21.15.175:0/2479783855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40e8153aa0 0x7f40e81643a0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:25.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 >> 172.21.15.175:0/2479783855 conn(0x7f40e8068280 msgr2=0x7f40e814a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:25.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 shutdown_connections 2023-10-10T19:08:25.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:25.244+0000 7f40ef0f1700 1 -- 172.21.15.175:0/2479783855 wait complete. 2023-10-10T19:08:25.248 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 176 2023-10-10T19:08:25.257 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:08:27.258 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:27.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.384+0000 7f7e87fff700 1 Processor -- start 2023-10-10T19:08:27.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.384+0000 7f7e87fff700 1 -- start start 2023-10-10T19:08:27.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e87fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7e880c7950 0x7f7e88159b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:27.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e87fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880aa8b0 0x7f7e8814f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:27.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e87fff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8806d370 con 0x7f7e8806b5b0 2023-10-10T19:08:27.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e87fff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8806a290 con 0x7f7e880c7950 2023-10-10T19:08:27.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e87fff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8806a550 con 0x7f7e880aa8b0 2023-10-10T19:08:27.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e86ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7e8806b5b0 0x7f7e8806d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:43532/0 (socket says 172.21.15.175:43532) 2023-10-10T19:08:27.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e7e7fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7e880c7950 0x7f7e88159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:27.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e86ffd700 1 -- 172.21.15.175:0/3353505198 learned_addr learned my addr 172.21.15.175:0/3353505198 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:27.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.385+0000 7f7e877fe700 1 --2- 172.21.15.175:0/3353505198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880aa8b0 0x7f7e8814f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:27.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e7e7fc700 1 -- 172.21.15.175:0/3353505198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880aa8b0 msgr2=0x7f7e8814f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:27.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e7e7fc700 1 --2- 172.21.15.175:0/3353505198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880aa8b0 0x7f7e8814f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e7e7fc700 1 -- 172.21.15.175:0/3353505198 >> v1:172.21.15.175:6789/0 conn(0x7f7e8806b5b0 legacy=0x7f7e8806d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:27.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e867fc700 1 -- 172.21.15.175:0/3353505198 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657113866 0 0) 0x7f7e8806d370 con 0x7f7e8806b5b0 2023-10-10T19:08:27.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e7e7fc700 1 -- 172.21.15.175:0/3353505198 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8806a830 con 0x7f7e880c7950 2023-10-10T19:08:27.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e7e7fc700 1 --2- 172.21.15.175:0/3353505198 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880c7950 0x7f7e88159b80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7e700098e0 tx=0x7f7e700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a39dbb1b050026 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:27.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e867fc700 1 -- 172.21.15.175:0/3353505198 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7000b640 con 0x7f7e880c7950 2023-10-10T19:08:27.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e867fc700 1 -- 172.21.15.175:0/3353505198 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7000b7a0 con 0x7f7e880c7950 2023-10-10T19:08:27.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e867fc700 1 -- 172.21.15.175:0/3353505198 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7000bb40 con 0x7f7e880c7950 2023-10-10T19:08:27.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e87fff700 1 -- 172.21.15.175:0/3353505198 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880c7950 msgr2=0x7f7e88159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:27.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3353505198 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880c7950 0x7f7e88159b80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7e700098e0 tx=0x7f7e700091a0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e87fff700 1 -- 172.21.15.175:0/3353505198 shutdown_connections 2023-10-10T19:08:27.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3353505198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880aa8b0 0x7f7e8814f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3353505198 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880c7950 0x7f7e88159b80 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.386+0000 7f7e87fff700 1 -- 172.21.15.175:0/3353505198 >> 172.21.15.175:0/3353505198 conn(0x7f7e880677f0 msgr2=0x7f7e88067bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:27.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.387+0000 7f7e87fff700 1 -- 172.21.15.175:0/3353505198 shutdown_connections 2023-10-10T19:08:27.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.387+0000 7f7e87fff700 1 -- 172.21.15.175:0/3353505198 wait complete. 2023-10-10T19:08:27.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.388+0000 7f7e87fff700 1 Processor -- start 2023-10-10T19:08:27.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.388+0000 7f7e87fff700 1 -- start start 2023-10-10T19:08:27.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.388+0000 7f7e87fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e8806b5b0 0x7f7e88151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:27.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.388+0000 7f7e87fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880aa8b0 0x7f7e88157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:27.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.388+0000 7f7e87fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880c7950 0x7f7e8815a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:27.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.388+0000 7f7e87fff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88151a60 con 0x7f7e880aa8b0 2023-10-10T19:08:27.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.388+0000 7f7e87fff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88150be0 con 0x7f7e8806b5b0 2023-10-10T19:08:27.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.388+0000 7f7e87fff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88152fb0 con 0x7f7e880c7950 2023-10-10T19:08:27.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880aa8b0 0x7f7e88157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:27.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880aa8b0 0x7f7e88157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47932/0 (socket says 172.21.15.175:47932) 2023-10-10T19:08:27.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 -- 172.21.15.175:0/3280976083 learned_addr learned my addr 172.21.15.175:0/3280976083 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:27.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e877fe700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880c7950 0x7f7e8815a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:27.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e86ffd700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e8806b5b0 0x7f7e88151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:27.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 -- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880c7950 msgr2=0x7f7e8815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:27.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880c7950 0x7f7e8815a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 -- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e8806b5b0 msgr2=0x7f7e88151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:27.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e8806b5b0 0x7f7e88151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 -- 172.21.15.175:0/3280976083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e70009420 con 0x7f7e880aa8b0 2023-10-10T19:08:27.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7e7fc700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880aa8b0 0x7f7e88157ae0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009f80 tx=0x7f7e70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:27.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7f7fe700 1 -- 172.21.15.175:0/3280976083 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7000bb20 con 0x7f7e880aa8b0 2023-10-10T19:08:27.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7f7fe700 1 -- 172.21.15.175:0/3280976083 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e70011420 con 0x7f7e880aa8b0 2023-10-10T19:08:27.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e88150e50 con 0x7f7e880aa8b0 2023-10-10T19:08:27.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e7f7fe700 1 -- 172.21.15.175:0/3280976083 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e700117a0 con 0x7f7e880aa8b0 2023-10-10T19:08:27.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.389+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e88151410 con 0x7f7e880aa8b0 2023-10-10T19:08:27.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.390+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e88151c10 con 0x7f7e880aa8b0 2023-10-10T19:08:27.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.390+0000 7f7e7f7fe700 1 -- 172.21.15.175:0/3280976083 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7e70011b50 con 0x7f7e880aa8b0 2023-10-10T19:08:27.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.391+0000 7f7e7f7fe700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e68066f30 0x7f7e680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:27.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.391+0000 7f7e7f7fe700 1 -- 172.21.15.175:0/3280976083 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6767+0+0 (secure 0 0 0) 0x7f7e70085f30 con 0x7f7e880aa8b0 2023-10-10T19:08:27.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.391+0000 7f7e86ffd700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e68066f30 0x7f7e680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:27.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.391+0000 7f7e86ffd700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e68066f30 0x7f7e680693e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7e780011e0 tx=0x7f7e78009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:27.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.396+0000 7f7e7f7fe700 1 -- 172.21.15.175:0/3280976083 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e88151c10 con 0x7f7e880aa8b0 2023-10-10T19:08:27.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.621+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e88151c10 con 0x7f7e880aa8b0 2023-10-10T19:08:27.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.621+0000 7f7e7f7fe700 1 -- 172.21.15.175:0/3280976083 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f7e88151c10 con 0x7f7e880aa8b0 2023-10-10T19:08:27.623 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:27.623 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:04.652754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":5552,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5552":{"gid":5552,"name":"f","rank":3,"incarnation":154,"state":"up:active","state_seq":36,"addr":"172.21.15.191:6839/1910071809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1910071809},{"type":"v1","addr":"172.21.15.191:6839","nonce":1910071809}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:27.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e68066f30 msgr2=0x7f7e680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:27.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e68066f30 0x7f7e680693e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7e780011e0 tx=0x7f7e78009040 comp rx=0 tx=0).stop 2023-10-10T19:08:27.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880aa8b0 msgr2=0x7f7e88157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:27.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880aa8b0 0x7f7e88157ae0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009f80 tx=0x7f7e70017000 comp rx=0 tx=0).stop 2023-10-10T19:08:27.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 shutdown_connections 2023-10-10T19:08:27.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7e68066f30 0x7f7e680693e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e880c7950 0x7f7e8815a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e8806b5b0 0x7f7e88151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 --2- 172.21.15.175:0/3280976083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e880aa8b0 0x7f7e88157ae0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:27.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 >> 172.21.15.175:0/3280976083 conn(0x7f7e880677f0 msgr2=0x7f7e880c0170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:27.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 shutdown_connections 2023-10-10T19:08:27.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:27.624+0000 7f7e87fff700 1 -- 172.21.15.175:0/3280976083 wait complete. 2023-10-10T19:08:27.628 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 176 2023-10-10T19:08:27.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:08:29.638 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:29.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.758+0000 7f7decbdf700 1 Processor -- start 2023-10-10T19:08:29.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.758+0000 7f7decbdf700 1 -- start start 2023-10-10T19:08:29.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7decbdf700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7de808a100 0x7f7de8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:29.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7decbdf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7de80cd040 0x7f7de80684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:29.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7decbdf700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de8067fa0 con 0x7f7de814fd90 2023-10-10T19:08:29.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7decbdf700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de80c5600 con 0x7f7de80cd040 2023-10-10T19:08:29.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7decbdf700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de8156b70 con 0x7f7de808a100 2023-10-10T19:08:29.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7de7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7de80cd040 0x7f7de80684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:29.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7de7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7de80cd040 0x7f7de80684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47940/0 (socket says 172.21.15.175:47940) 2023-10-10T19:08:29.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7de7fff700 1 -- 172.21.15.175:0/3841671372 learned_addr learned my addr 172.21.15.175:0/3841671372 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:29.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.759+0000 7f7ddeffd700 1 --2- 172.21.15.175:0/3841671372 >> v2:172.21.15.191:3300/0 conn(0x7f7de808a100 0x7f7de8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:29.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7de7fff700 1 -- 172.21.15.175:0/3841671372 >> v2:172.21.15.191:3300/0 conn(0x7f7de808a100 msgr2=0x7f7de8159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:29.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7de7fff700 1 --2- 172.21.15.175:0/3841671372 >> v2:172.21.15.191:3300/0 conn(0x7f7de808a100 0x7f7de8159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7de7fff700 1 -- 172.21.15.175:0/3841671372 >> v1:172.21.15.191:6789/0 conn(0x7f7de814fd90 legacy=0x7f7de80c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:29.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7de7fff700 1 -- 172.21.15.175:0/3841671372 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de8150160 con 0x7f7de80cd040 2023-10-10T19:08:29.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7de7fff700 1 --2- 172.21.15.175:0/3841671372 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de80cd040 0x7f7de80684f0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7de8069990 tx=0x7f7de0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90cc38bfdc150792 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:29.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7de6ffd700 1 -- 172.21.15.175:0/3841671372 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de000e070 con 0x7f7de80cd040 2023-10-10T19:08:29.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7de6ffd700 1 -- 172.21.15.175:0/3841671372 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de000bae0 con 0x7f7de80cd040 2023-10-10T19:08:29.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7de6ffd700 1 -- 172.21.15.175:0/3841671372 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de00092e0 con 0x7f7de80cd040 2023-10-10T19:08:29.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7decbdf700 1 -- 172.21.15.175:0/3841671372 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de80cd040 msgr2=0x7f7de80684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:29.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7decbdf700 1 --2- 172.21.15.175:0/3841671372 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de80cd040 0x7f7de80684f0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7de8069990 tx=0x7f7de0009840 comp rx=0 tx=0).stop 2023-10-10T19:08:29.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7decbdf700 1 -- 172.21.15.175:0/3841671372 shutdown_connections 2023-10-10T19:08:29.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7decbdf700 1 --2- 172.21.15.175:0/3841671372 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de80cd040 0x7f7de80684f0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.760+0000 7f7decbdf700 1 --2- 172.21.15.175:0/3841671372 >> v2:172.21.15.191:3300/0 conn(0x7f7de808a100 0x7f7de8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.761+0000 7f7decbdf700 1 -- 172.21.15.175:0/3841671372 >> 172.21.15.175:0/3841671372 conn(0x7f7de8063d00 msgr2=0x7f7de80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:29.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.761+0000 7f7decbdf700 1 -- 172.21.15.175:0/3841671372 shutdown_connections 2023-10-10T19:08:29.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.761+0000 7f7decbdf700 1 -- 172.21.15.175:0/3841671372 wait complete. 2023-10-10T19:08:29.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.762+0000 7f7decbdf700 1 Processor -- start 2023-10-10T19:08:29.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.762+0000 7f7decbdf700 1 -- start start 2023-10-10T19:08:29.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.762+0000 7f7decbdf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7de808a100 0x7f7de814b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:29.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.762+0000 7f7decbdf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de80cd040 0x7f7de814b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:29.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.762+0000 7f7decbdf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de814fd90 0x7f7de8168cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:29.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.762+0000 7f7decbdf700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de8152af0 con 0x7f7de814fd90 2023-10-10T19:08:29.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.762+0000 7f7decbdf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de8169b60 con 0x7f7de80cd040 2023-10-10T19:08:29.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.762+0000 7f7decbdf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de814d2c0 con 0x7f7de808a100 2023-10-10T19:08:29.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de814fd90 0x7f7de8168cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:29.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de814fd90 0x7f7de8168cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47952/0 (socket says 172.21.15.175:47952) 2023-10-10T19:08:29.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 -- 172.21.15.175:0/2807955458 learned_addr learned my addr 172.21.15.175:0/2807955458 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:29.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7ddeffd700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de80cd040 0x7f7de814b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:29.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de77fe700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7de808a100 0x7f7de814b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:29.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 -- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7de808a100 msgr2=0x7f7de814b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:29.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7de808a100 0x7f7de814b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 -- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de80cd040 msgr2=0x7f7de814b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:29.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de80cd040 0x7f7de814b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 -- 172.21.15.175:0/2807955458 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de0009ae0 con 0x7f7de814fd90 2023-10-10T19:08:29.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7de7fff700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de814fd90 0x7f7de8168cf0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009fd0 tx=0x7f7de0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:29.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7ddffff700 1 -- 172.21.15.175:0/2807955458 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de0008e70 con 0x7f7de814fd90 2023-10-10T19:08:29.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7ddffff700 1 -- 172.21.15.175:0/2807955458 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de0026460 con 0x7f7de814fd90 2023-10-10T19:08:29.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7ddffff700 1 -- 172.21.15.175:0/2807955458 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de0026820 con 0x7f7de814fd90 2023-10-10T19:08:29.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de80cc970 con 0x7f7de814fd90 2023-10-10T19:08:29.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.763+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de81698c0 con 0x7f7de814fd90 2023-10-10T19:08:29.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.764+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de8157790 con 0x7f7de814fd90 2023-10-10T19:08:29.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.764+0000 7f7ddffff700 1 -- 172.21.15.175:0/2807955458 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7de0003040 con 0x7f7de814fd90 2023-10-10T19:08:29.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.765+0000 7f7ddffff700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7dc8066f30 0x7f7dc80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:29.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.765+0000 7f7ddffff700 1 -- 172.21.15.175:0/2807955458 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f7de00864c0 con 0x7f7de814fd90 2023-10-10T19:08:29.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.765+0000 7f7de77fe700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7dc8066f30 0x7f7dc80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:29.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.765+0000 7f7de77fe700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7dc8066f30 0x7f7dc80693e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7dd80037c0 tx=0x7f7dd8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:29.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.769+0000 7f7ddffff700 1 -- 172.21.15.175:0/2807955458 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7de8157790 con 0x7f7de814fd90 2023-10-10T19:08:29.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.994+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7de8157790 con 0x7f7de814fd90 2023-10-10T19:08:29.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.995+0000 7f7ddffff700 1 -- 172.21.15.175:0/2807955458 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f7de8157790 con 0x7f7de814fd90 2023-10-10T19:08:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:29.996 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:28.655608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:29.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7dc8066f30 msgr2=0x7f7dc80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:29.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7dc8066f30 0x7f7dc80693e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7dd80037c0 tx=0x7f7dd8009040 comp rx=0 tx=0).stop 2023-10-10T19:08:29.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de814fd90 msgr2=0x7f7de8168cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:29.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de814fd90 0x7f7de8168cf0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009fd0 tx=0x7f7de0018000 comp rx=0 tx=0).stop 2023-10-10T19:08:29.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 shutdown_connections 2023-10-10T19:08:29.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7dc8066f30 0x7f7dc80693e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7de814fd90 0x7f7de8168cf0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7de808a100 0x7f7de814b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 --2- 172.21.15.175:0/2807955458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de80cd040 0x7f7de814b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 >> 172.21.15.175:0/2807955458 conn(0x7f7de8063d00 msgr2=0x7f7de80c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:29.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 shutdown_connections 2023-10-10T19:08:30.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:29.997+0000 7f7decbdf700 1 -- 172.21.15.175:0/2807955458 wait complete. 2023-10-10T19:08:30.000 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 177 2023-10-10T19:08:30.009 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-10T19:08:30.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:08:30.009 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:30.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.123+0000 7f87abc42700 1 Processor -- start 2023-10-10T19:08:30.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.123+0000 7f87abc42700 1 -- start start 2023-10-10T19:08:30.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.124+0000 7f87abc42700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a40c2b80 0x7f87a4156580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:30.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.124+0000 7f87abc42700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f87a4156ac0 0x7f87a40c7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.124+0000 7f87abc42700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a415a6c0 con 0x7f87a40c35a0 2023-10-10T19:08:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.124+0000 7f87abc42700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a414d4a0 con 0x7f87a4156ac0 2023-10-10T19:08:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.124+0000 7f87abc42700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a4159830 con 0x7f87a40c2b80 2023-10-10T19:08:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.124+0000 7f87aa1df700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f87a4156ac0 0x7f87a40c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:30.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.124+0000 7f87a91dd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f87a40c35a0 0x7f87a40c6a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:43540/0 (socket says 172.21.15.175:43540) 2023-10-10T19:08:30.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.124+0000 7f87a91dd700 1 -- 172.21.15.175:0/4236677563 learned_addr learned my addr 172.21.15.175:0/4236677563 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:30.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87a99de700 1 --2- 172.21.15.175:0/4236677563 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a40c2b80 0x7f87a4156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:30.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87aa1df700 1 -- 172.21.15.175:0/4236677563 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a40c2b80 msgr2=0x7f87a4156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:30.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87aa1df700 1 --2- 172.21.15.175:0/4236677563 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a40c2b80 0x7f87a4156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87aa1df700 1 -- 172.21.15.175:0/4236677563 >> v1:172.21.15.175:6789/0 conn(0x7f87a40c35a0 legacy=0x7f87a40c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:30.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87aa1df700 1 -- 172.21.15.175:0/4236677563 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a4156f40 con 0x7f87a4156ac0 2023-10-10T19:08:30.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87aa1df700 1 --2- 172.21.15.175:0/4236677563 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a4156ac0 0x7f87a40c7140 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f879c0098e0 tx=0x7f879c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22819082c66733ea server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:30.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87a89dc700 1 -- 172.21.15.175:0/4236677563 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c009e80 con 0x7f87a4156ac0 2023-10-10T19:08:30.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87a89dc700 1 -- 172.21.15.175:0/4236677563 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879c00b8f0 con 0x7f87a4156ac0 2023-10-10T19:08:30.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87a89dc700 1 -- 172.21.15.175:0/4236677563 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c00bc70 con 0x7f87a4156ac0 2023-10-10T19:08:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87abc42700 1 -- 172.21.15.175:0/4236677563 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a4156ac0 msgr2=0x7f87a40c7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87abc42700 1 --2- 172.21.15.175:0/4236677563 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a4156ac0 0x7f87a40c7140 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f879c0098e0 tx=0x7f879c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87abc42700 1 -- 172.21.15.175:0/4236677563 shutdown_connections 2023-10-10T19:08:30.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87abc42700 1 --2- 172.21.15.175:0/4236677563 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a4156ac0 0x7f87a40c7140 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87abc42700 1 --2- 172.21.15.175:0/4236677563 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a40c2b80 0x7f87a4156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.125+0000 7f87abc42700 1 -- 172.21.15.175:0/4236677563 >> 172.21.15.175:0/4236677563 conn(0x7f87a40c56a0 msgr2=0x7f87a40c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.126+0000 7f87abc42700 1 -- 172.21.15.175:0/4236677563 shutdown_connections 2023-10-10T19:08:30.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.126+0000 7f87abc42700 1 -- 172.21.15.175:0/4236677563 wait complete. 2023-10-10T19:08:30.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87abc42700 1 Processor -- start 2023-10-10T19:08:30.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87abc42700 1 -- start start 2023-10-10T19:08:30.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87abc42700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a40c2b80 0x7f87a41553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:30.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87abc42700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a40c35a0 0x7f87a40c0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87abc42700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a4156ac0 0x7f87a4144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87abc42700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a4155920 con 0x7f87a40c2b80 2023-10-10T19:08:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87abc42700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a4154f50 con 0x7f87a40c35a0 2023-10-10T19:08:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87abc42700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a414cee0 con 0x7f87a4156ac0 2023-10-10T19:08:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87a99de700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a40c2b80 0x7f87a41553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87a99de700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a40c2b80 0x7f87a41553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47962/0 (socket says 172.21.15.175:47962) 2023-10-10T19:08:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87a99de700 1 -- 172.21.15.175:0/1536362729 learned_addr learned my addr 172.21.15.175:0/1536362729 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87a91dd700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a40c35a0 0x7f87a40c0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.127+0000 7f87aa1df700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a4156ac0 0x7f87a4144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:30.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f87a99de700 1 -- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a4156ac0 msgr2=0x7f87a4144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f87a99de700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a4156ac0 0x7f87a4144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f87a99de700 1 -- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a40c35a0 msgr2=0x7f87a40c0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f87a99de700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a40c35a0 0x7f87a40c0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f87a99de700 1 -- 172.21.15.175:0/1536362729 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c009420 con 0x7f87a40c2b80 2023-10-10T19:08:30.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f87a99de700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a40c2b80 0x7f87a41553e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f87a000e190 tx=0x7f87a0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f8799ffb700 1 -- 172.21.15.175:0/1536362729 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a000ed60 con 0x7f87a40c2b80 2023-10-10T19:08:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f8799ffb700 1 -- 172.21.15.175:0/1536362729 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a0014420 con 0x7f87a40c2b80 2023-10-10T19:08:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a4153f50 con 0x7f87a40c2b80 2023-10-10T19:08:30.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f8799ffb700 1 -- 172.21.15.175:0/1536362729 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a00147e0 con 0x7f87a40c2b80 2023-10-10T19:08:30.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.128+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a4154510 con 0x7f87a40c2b80 2023-10-10T19:08:30.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.129+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a406a140 con 0x7f87a40c2b80 2023-10-10T19:08:30.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.129+0000 7f8799ffb700 1 -- 172.21.15.175:0/1536362729 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f87a0014b50 con 0x7f87a40c2b80 2023-10-10T19:08:30.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.130+0000 7f8799ffb700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8788066f30 0x7f87880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.130+0000 7f8799ffb700 1 -- 172.21.15.175:0/1536362729 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f87a0085150 con 0x7f87a40c2b80 2023-10-10T19:08:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.130+0000 7f87a91dd700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8788066f30 0x7f87880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.130+0000 7f87a91dd700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8788066f30 0x7f87880693e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f87a406d280 tx=0x7f8794007eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:30.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.134+0000 7f8799ffb700 1 -- 172.21.15.175:0/1536362729 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87a0051ea0 con 0x7f87a40c2b80 2023-10-10T19:08:30.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.401+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87a4067570 con 0x7f87a40c2b80 2023-10-10T19:08:30.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.402+0000 7f8799ffb700 1 -- 172.21.15.175:0/1536362729 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f87a0009a60 con 0x7f87a40c2b80 2023-10-10T19:08:30.403 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:30.403 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:28.655608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:30.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8788066f30 msgr2=0x7f87880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:30.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8788066f30 0x7f87880693e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f87a406d280 tx=0x7f8794007eb0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a40c2b80 msgr2=0x7f87a41553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:30.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a40c2b80 0x7f87a41553e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f87a000e190 tx=0x7f87a0009920 comp rx=0 tx=0).stop 2023-10-10T19:08:30.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 shutdown_connections 2023-10-10T19:08:30.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8788066f30 0x7f87880693e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87a4156ac0 0x7f87a4144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87a40c2b80 0x7f87a41553e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 --2- 172.21.15.175:0/1536362729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a40c35a0 0x7f87a40c0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:30.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 >> 172.21.15.175:0/1536362729 conn(0x7f87a40c56a0 msgr2=0x7f87a40c5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:30.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 shutdown_connections 2023-10-10T19:08:30.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:30.404+0000 7f87abc42700 1 -- 172.21.15.175:0/1536362729 wait complete. 2023-10-10T19:08:30.408 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 177 2023-10-10T19:08:30.417 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 177, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 177, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:08:28.655608+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 6106, "mds_3": 6618, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:08:32.418 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:32.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.546+0000 7f945eaae700 1 Processor -- start 2023-10-10T19:08:32.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.546+0000 7f945eaae700 1 -- start start 2023-10-10T19:08:32.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.546+0000 7f945eaae700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9458063e20 0x7f94580cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:32.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.546+0000 7f945eaae700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94580c4170 con 0x7f94580c6780 2023-10-10T19:08:32.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.546+0000 7f945eaae700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9458154be0 con 0x7f94580c6b50 2023-10-10T19:08:32.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.546+0000 7f945eaae700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94580c2c30 con 0x7f9458063e20 2023-10-10T19:08:32.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f945c84a700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f94580c6780 0x7f945806b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:43548/0 (socket says 172.21.15.175:43548) 2023-10-10T19:08:32.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9458063e20 0x7f94580cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:32.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f945c84a700 1 -- 172.21.15.175:0/3684603375 learned_addr learned my addr 172.21.15.175:0/3684603375 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:32.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944ffff700 1 -- 172.21.15.175:0/3684603375 >> v1:172.21.15.191:6789/0 conn(0x7f94580c6b50 legacy=0x7f9458063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:32.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944ffff700 1 -- 172.21.15.175:0/3684603375 >> v1:172.21.15.175:6789/0 conn(0x7f94580c6780 legacy=0x7f945806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:32.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944f7fe700 1 -- 172.21.15.175:0/3684603375 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888570324 0 0) 0x7f94580c4170 con 0x7f94580c6780 2023-10-10T19:08:32.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944ffff700 1 -- 172.21.15.175:0/3684603375 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580c3e90 con 0x7f9458063e20 2023-10-10T19:08:32.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944ffff700 1 --2- 172.21.15.175:0/3684603375 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9458063e20 0x7f94580cac80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f94480098d0 tx=0x7f9448009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64103a7e3aa1114e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:32.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944f7fe700 1 -- 172.21.15.175:0/3684603375 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448009e70 con 0x7f9458063e20 2023-10-10T19:08:32.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944f7fe700 1 -- 172.21.15.175:0/3684603375 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9448005c10 con 0x7f9458063e20 2023-10-10T19:08:32.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.547+0000 7f944f7fe700 1 -- 172.21.15.175:0/3684603375 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944800b650 con 0x7f9458063e20 2023-10-10T19:08:32.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.548+0000 7f945eaae700 1 -- 172.21.15.175:0/3684603375 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9458063e20 msgr2=0x7f94580cac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:32.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.548+0000 7f945eaae700 1 --2- 172.21.15.175:0/3684603375 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9458063e20 0x7f94580cac80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f94480098d0 tx=0x7f9448009120 comp rx=0 tx=0).stop 2023-10-10T19:08:32.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.548+0000 7f945eaae700 1 -- 172.21.15.175:0/3684603375 shutdown_connections 2023-10-10T19:08:32.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.548+0000 7f945eaae700 1 --2- 172.21.15.175:0/3684603375 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9458063e20 0x7f94580cac80 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:32.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.548+0000 7f945eaae700 1 -- 172.21.15.175:0/3684603375 >> 172.21.15.175:0/3684603375 conn(0x7f945806dbe0 msgr2=0x7f94580ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:32.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.548+0000 7f945eaae700 1 -- 172.21.15.175:0/3684603375 shutdown_connections 2023-10-10T19:08:32.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.548+0000 7f945eaae700 1 -- 172.21.15.175:0/3684603375 wait complete. 2023-10-10T19:08:32.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.549+0000 7f945eaae700 1 Processor -- start 2023-10-10T19:08:32.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.549+0000 7f945eaae700 1 -- start start 2023-10-10T19:08:32.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945eaae700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9458063e20 0x7f945814f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:32.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945eaae700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94580c6780 0x7f945814fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:32.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945eaae700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94580c6b50 0x7f945814b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:32.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945eaae700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580c0b60 con 0x7f94580c6b50 2023-10-10T19:08:32.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945eaae700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94581519d0 con 0x7f94580c6780 2023-10-10T19:08:32.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945eaae700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9458150d20 con 0x7f9458063e20 2023-10-10T19:08:32.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94580c6b50 0x7f945814b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:32.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94580c6b50 0x7f945814b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47986/0 (socket says 172.21.15.175:47986) 2023-10-10T19:08:32.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 -- 172.21.15.175:0/607607446 learned_addr learned my addr 172.21.15.175:0/607607446 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:32.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f944ffff700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94580c6780 0x7f945814fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:32.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 -- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9458063e20 msgr2=0x7f945814f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:32.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9458063e20 0x7f945814f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:32.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 -- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94580c6780 msgr2=0x7f945814fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:32.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94580c6780 0x7f945814fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:32.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 -- 172.21.15.175:0/607607446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94480093c0 con 0x7f94580c6b50 2023-10-10T19:08:32.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f944ffff700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94580c6780 0x7f945814fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:08:32.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945d04b700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94580c6b50 0x7f945814b720 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f945000c810 tx=0x7f9450001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:32.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f944cff9700 1 -- 172.21.15.175:0/607607446 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450012630 con 0x7f94580c6b50 2023-10-10T19:08:32.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f944cff9700 1 -- 172.21.15.175:0/607607446 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94500127b0 con 0x7f94580c6b50 2023-10-10T19:08:32.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94580afe20 con 0x7f94580c6b50 2023-10-10T19:08:32.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f944cff9700 1 -- 172.21.15.175:0/607607446 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450012b70 con 0x7f94580c6b50 2023-10-10T19:08:32.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.550+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94581512c0 con 0x7f94580c6b50 2023-10-10T19:08:32.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.551+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9458067570 con 0x7f94580c6b50 2023-10-10T19:08:32.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.552+0000 7f944cff9700 1 -- 172.21.15.175:0/607607446 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f945000a3b0 con 0x7f94580c6b50 2023-10-10T19:08:32.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.552+0000 7f944cff9700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9444066f30 0x7f94440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:32.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.552+0000 7f944cff9700 1 -- 172.21.15.175:0/607607446 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f94500845c0 con 0x7f94580c6b50 2023-10-10T19:08:32.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.552+0000 7f945c84a700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9444066f30 0x7f94440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:32.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.553+0000 7f945c84a700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9444066f30 0x7f94440693e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9454000c00 tx=0x7f9454007810 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:32.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.556+0000 7f944cff9700 1 -- 172.21.15.175:0/607607446 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f94500513b0 con 0x7f94580c6b50 2023-10-10T19:08:32.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.787+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9458151b10 con 0x7f94580c6b50 2023-10-10T19:08:32.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.788+0000 7f944cff9700 1 -- 172.21.15.175:0/607607446 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f94500549d0 con 0x7f94580c6b50 2023-10-10T19:08:32.788 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:32.788 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:28.655608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:32.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.789+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9444066f30 msgr2=0x7f94440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:32.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.789+0000 7f945eaae700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9444066f30 0x7f94440693e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9454000c00 tx=0x7f9454007810 comp rx=0 tx=0).stop 2023-10-10T19:08:32.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.789+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94580c6b50 msgr2=0x7f945814b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:32.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.789+0000 7f945eaae700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94580c6b50 0x7f945814b720 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f945000c810 tx=0x7f9450001650 comp rx=0 tx=0).stop 2023-10-10T19:08:32.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.790+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 shutdown_connections 2023-10-10T19:08:32.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.790+0000 7f945eaae700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9444066f30 0x7f94440693e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:32.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.790+0000 7f945eaae700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94580c6b50 0x7f945814b720 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:32.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.790+0000 7f945eaae700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9458063e20 0x7f945814f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:32.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.790+0000 7f945eaae700 1 --2- 172.21.15.175:0/607607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94580c6780 0x7f945814fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:32.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.790+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 >> 172.21.15.175:0/607607446 conn(0x7f945806dbe0 msgr2=0x7f94580c36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:32.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.790+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 shutdown_connections 2023-10-10T19:08:32.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:32.790+0000 7f945eaae700 1 -- 172.21.15.175:0/607607446 wait complete. 2023-10-10T19:08:32.793 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 177 2023-10-10T19:08:32.802 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:34.803 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:34.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.918+0000 7f123291a700 1 Processor -- start 2023-10-10T19:08:34.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.918+0000 7f123291a700 1 -- start start 2023-10-10T19:08:34.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.918+0000 7f123291a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0b1fc0 0x7f122c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:34.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.918+0000 7f123291a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c1574b0 con 0x7f122c0aff60 2023-10-10T19:08:34.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.918+0000 7f123291a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c0afad0 con 0x7f122c0b2390 2023-10-10T19:08:34.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.918+0000 7f123291a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122c150770 con 0x7f122c0b1fc0 2023-10-10T19:08:34.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.918+0000 7f1230eb7700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f122c0aff60 0x7f122c159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46936/0 (socket says 172.21.15.175:46936) 2023-10-10T19:08:34.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.918+0000 7f1230eb7700 1 -- 172.21.15.175:0/2817159952 learned_addr learned my addr 172.21.15.175:0/2817159952 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:34.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122bfff700 1 --2- 172.21.15.175:0/2817159952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0b1fc0 0x7f122c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:34.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122affd700 1 -- 172.21.15.175:0/2817159952 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303537234 0 0) 0x7f122c1574b0 con 0x7f122c0aff60 2023-10-10T19:08:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122affd700 1 -- 172.21.15.175:0/2817159952 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1214003570 con 0x7f122c0aff60 2023-10-10T19:08:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122affd700 1 -- 172.21.15.175:0/2817159952 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413125498 0 0) 0x7f122c0afad0 con 0x7f122c0b2390 2023-10-10T19:08:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122affd700 1 -- 172.21.15.175:0/2817159952 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c1574b0 con 0x7f122c0b2390 2023-10-10T19:08:34.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122bfff700 1 -- 172.21.15.175:0/2817159952 >> v1:172.21.15.191:6789/0 conn(0x7f122c0b2390 legacy=0x7f122c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122bfff700 1 -- 172.21.15.175:0/2817159952 >> v1:172.21.15.175:6789/0 conn(0x7f122c0aff60 legacy=0x7f122c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122affd700 1 -- 172.21.15.175:0/2817159952 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1058785017 0 0) 0x7f1214003570 con 0x7f122c0aff60 2023-10-10T19:08:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122bfff700 1 -- 172.21.15.175:0/2817159952 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c0af4c0 con 0x7f122c0b1fc0 2023-10-10T19:08:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122bfff700 1 --2- 172.21.15.175:0/2817159952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0b1fc0 0x7f122c155fb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1220009940 tx=0x7f12200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97513503cb9cc5ea server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:34.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122affd700 1 -- 172.21.15.175:0/2817159952 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12240017f0 con 0x7f122c0b1fc0 2023-10-10T19:08:34.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122affd700 1 -- 172.21.15.175:0/2817159952 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1220011420 con 0x7f122c0b1fc0 2023-10-10T19:08:34.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.919+0000 7f122affd700 1 -- 172.21.15.175:0/2817159952 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12200117a0 con 0x7f122c0b1fc0 2023-10-10T19:08:34.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.920+0000 7f123291a700 1 -- 172.21.15.175:0/2817159952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0b1fc0 msgr2=0x7f122c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:34.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.920+0000 7f123291a700 1 --2- 172.21.15.175:0/2817159952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0b1fc0 0x7f122c155fb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1220009940 tx=0x7f12200091a0 comp rx=0 tx=0).stop 2023-10-10T19:08:34.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.920+0000 7f123291a700 1 -- 172.21.15.175:0/2817159952 shutdown_connections 2023-10-10T19:08:34.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.920+0000 7f123291a700 1 --2- 172.21.15.175:0/2817159952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0b1fc0 0x7f122c155fb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:34.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.920+0000 7f123291a700 1 -- 172.21.15.175:0/2817159952 >> 172.21.15.175:0/2817159952 conn(0x7f122c063230 msgr2=0x7f122c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:34.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.920+0000 7f123291a700 1 -- 172.21.15.175:0/2817159952 shutdown_connections 2023-10-10T19:08:34.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.920+0000 7f123291a700 1 -- 172.21.15.175:0/2817159952 wait complete. 2023-10-10T19:08:34.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.921+0000 7f123291a700 1 Processor -- start 2023-10-10T19:08:34.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.921+0000 7f123291a700 1 -- start start 2023-10-10T19:08:34.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f123291a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0aff60 0x7f122c0c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:34.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f123291a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f122c0b1fc0 0x7f122c0c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:34.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f123291a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f122c0b2390 0x7f122c0cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:34.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f123291a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c150180 con 0x7f122c0b2390 2023-10-10T19:08:34.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f123291a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c14e530 con 0x7f122c0b1fc0 2023-10-10T19:08:34.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f123291a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122c0c53d0 con 0x7f122c0aff60 2023-10-10T19:08:34.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f1230eb7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f122c0b2390 0x7f122c0cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:34.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f1230eb7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f122c0b2390 0x7f122c0cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55014/0 (socket says 172.21.15.175:55014) 2023-10-10T19:08:34.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f1230eb7700 1 -- 172.21.15.175:0/2397062462 learned_addr learned my addr 172.21.15.175:0/2397062462 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:34.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f122bfff700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0aff60 0x7f122c0c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:34.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f122b7fe700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f122c0b1fc0 0x7f122c0c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:34.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f1230eb7700 1 -- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0aff60 msgr2=0x7f122c0c3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:34.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f1230eb7700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0aff60 0x7f122c0c3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:34.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f1230eb7700 1 -- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f122c0b1fc0 msgr2=0x7f122c0c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:34.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f1230eb7700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f122c0b1fc0 0x7f122c0c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:34.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f1230eb7700 1 -- 172.21.15.175:0/2397062462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1220009420 con 0x7f122c0b2390 2023-10-10T19:08:34.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.922+0000 7f122b7fe700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f122c0b1fc0 0x7f122c0c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:08:34.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.923+0000 7f1230eb7700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f122c0b2390 0x7f122c0cba00 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f122400ccd0 tx=0x7f12240018e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:34.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.923+0000 7f1213fff700 1 -- 172.21.15.175:0/2397062462 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224007e70 con 0x7f122c0b2390 2023-10-10T19:08:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.923+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122c0c60b0 con 0x7f122c0b2390 2023-10-10T19:08:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.923+0000 7f1213fff700 1 -- 172.21.15.175:0/2397062462 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1224004dc0 con 0x7f122c0b2390 2023-10-10T19:08:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.923+0000 7f1213fff700 1 -- 172.21.15.175:0/2397062462 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122400ab90 con 0x7f122c0b2390 2023-10-10T19:08:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.923+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122c0bff40 con 0x7f122c0b2390 2023-10-10T19:08:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.924+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122c0cb720 con 0x7f122c0b2390 2023-10-10T19:08:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.924+0000 7f1213fff700 1 -- 172.21.15.175:0/2397062462 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1224007150 con 0x7f122c0b2390 2023-10-10T19:08:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.924+0000 7f1213fff700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1218066f60 0x7f1218069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.925+0000 7f1213fff700 1 -- 172.21.15.175:0/2397062462 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f1224084100 con 0x7f122c0b2390 2023-10-10T19:08:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.925+0000 7f122bfff700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1218066f60 0x7f1218069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.925+0000 7f122bfff700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1218066f60 0x7f1218069410 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1220009860 tx=0x7f1220016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:34.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:34.929+0000 7f1213fff700 1 -- 172.21.15.175:0/2397062462 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f122c0cb720 con 0x7f122c0b2390 2023-10-10T19:08:35.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.163+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f122c0cb720 con 0x7f122c0b2390 2023-10-10T19:08:35.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.164+0000 7f1213fff700 1 -- 172.21.15.175:0/2397062462 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f122c0cb720 con 0x7f122c0b2390 2023-10-10T19:08:35.165 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:35.165 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:28.655608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:35.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1218066f60 msgr2=0x7f1218069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:35.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1218066f60 0x7f1218069410 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1220009860 tx=0x7f1220016000 comp rx=0 tx=0).stop 2023-10-10T19:08:35.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f122c0b2390 msgr2=0x7f122c0cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:35.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f122c0b2390 0x7f122c0cba00 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f122400ccd0 tx=0x7f12240018e0 comp rx=0 tx=0).stop 2023-10-10T19:08:35.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 shutdown_connections 2023-10-10T19:08:35.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1218066f60 0x7f1218069410 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:35.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f122c0b2390 0x7f122c0cba00 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:35.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f122c0aff60 0x7f122c0c3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:35.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 --2- 172.21.15.175:0/2397062462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f122c0b1fc0 0x7f122c0c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:35.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 >> 172.21.15.175:0/2397062462 conn(0x7f122c063230 msgr2=0x7f122c14fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:35.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 shutdown_connections 2023-10-10T19:08:35.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:35.167+0000 7f123291a700 1 -- 172.21.15.175:0/2397062462 wait complete. 2023-10-10T19:08:35.170 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 177 2023-10-10T19:08:35.180 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:37.181 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:37.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.307+0000 7f5d716ad700 1 Processor -- start 2023-10-10T19:08:37.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.307+0000 7f5d716ad700 1 -- start start 2023-10-10T19:08:37.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.307+0000 7f5d716ad700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5d6c06bf00 0x7f5d6c0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:37.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.307+0000 7f5d716ad700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0c6df0 con 0x7f5d6c156520 2023-10-10T19:08:37.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.307+0000 7f5d716ad700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0c6920 con 0x7f5d6c0c5e40 2023-10-10T19:08:37.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.307+0000 7f5d716ad700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d6c1560d0 con 0x7f5d6c06bf00 2023-10-10T19:08:37.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d6a7fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5d6c06bf00 0x7f5d6c0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:37.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d6b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5d6c156520 0x7f5d6c1569a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46940/0 (socket says 172.21.15.175:46940) 2023-10-10T19:08:37.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d6b7fe700 1 -- 172.21.15.175:0/196949045 learned_addr learned my addr 172.21.15.175:0/196949045 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:37.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d6a7fc700 1 -- 172.21.15.175:0/196949045 >> v1:172.21.15.191:6789/0 conn(0x7f5d6c0c5e40 legacy=0x7f5d6c0c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:37.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d6a7fc700 1 -- 172.21.15.175:0/196949045 >> v1:172.21.15.175:6789/0 conn(0x7f5d6c156520 legacy=0x7f5d6c1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:37.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d6a7fc700 1 -- 172.21.15.175:0/196949045 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c1597e0 con 0x7f5d6c06bf00 2023-10-10T19:08:37.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d69ffb700 1 -- 172.21.15.175:0/196949045 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814252161 0 0) 0x7f5d6c0c6df0 con 0x7f5d6c156520 2023-10-10T19:08:37.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d6a7fc700 1 --2- 172.21.15.175:0/196949045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c0c47d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5d600098e0 tx=0x7f5d600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf7bdbe25b6ba87 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:37.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d69ffb700 1 -- 172.21.15.175:0/196949045 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60009e80 con 0x7f5d6c06bf00 2023-10-10T19:08:37.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d69ffb700 1 -- 172.21.15.175:0/196949045 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6000b8f0 con 0x7f5d6c06bf00 2023-10-10T19:08:37.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.308+0000 7f5d69ffb700 1 -- 172.21.15.175:0/196949045 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6000bc70 con 0x7f5d6c06bf00 2023-10-10T19:08:37.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.309+0000 7f5d716ad700 1 -- 172.21.15.175:0/196949045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 msgr2=0x7f5d6c0c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:37.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.309+0000 7f5d716ad700 1 --2- 172.21.15.175:0/196949045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c0c47d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5d600098e0 tx=0x7f5d600091a0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.309+0000 7f5d716ad700 1 -- 172.21.15.175:0/196949045 shutdown_connections 2023-10-10T19:08:37.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.309+0000 7f5d716ad700 1 --2- 172.21.15.175:0/196949045 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c0c47d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.309+0000 7f5d716ad700 1 -- 172.21.15.175:0/196949045 >> 172.21.15.175:0/196949045 conn(0x7f5d6c068900 msgr2=0x7f5d6c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:37.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.309+0000 7f5d716ad700 1 -- 172.21.15.175:0/196949045 shutdown_connections 2023-10-10T19:08:37.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.309+0000 7f5d716ad700 1 -- 172.21.15.175:0/196949045 wait complete. 2023-10-10T19:08:37.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.310+0000 7f5d716ad700 1 Processor -- start 2023-10-10T19:08:37.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.310+0000 7f5d716ad700 1 -- start start 2023-10-10T19:08:37.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d716ad700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:37.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d716ad700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0c5e40 0x7f5d6c0c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:37.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d716ad700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5d6c156520 0x7f5d6c0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:37.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d716ad700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c166450 con 0x7f5d6c06bf00 2023-10-10T19:08:37.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d716ad700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c06f5f0 con 0x7f5d6c0c5e40 2023-10-10T19:08:37.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d716ad700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c167270 con 0x7f5d6c156520 2023-10-10T19:08:37.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:37.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55040/0 (socket says 172.21.15.175:55040) 2023-10-10T19:08:37.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 -- 172.21.15.175:0/1772108860 learned_addr learned my addr 172.21.15.175:0/1772108860 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:37.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6a7fc700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0c5e40 0x7f5d6c0c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:37.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 -- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5d6c156520 msgr2=0x7f5d6c0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:37.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5d6c156520 0x7f5d6c0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 -- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0c5e40 msgr2=0x7f5d6c0c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:37.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0c5e40 0x7f5d6c0c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 -- 172.21.15.175:0/1772108860 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d60009420 con 0x7f5d6c06bf00 2023-10-10T19:08:37.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d6affd700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c158e30 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00cde0 tx=0x7f5d5c001be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:37.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.311+0000 7f5d52ffd700 1 -- 172.21.15.175:0/1772108860 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00ac90 con 0x7f5d6c06bf00 2023-10-10T19:08:37.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.312+0000 7f5d52ffd700 1 -- 172.21.15.175:0/1772108860 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c00ae10 con 0x7f5d6c06bf00 2023-10-10T19:08:37.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.312+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6c1576c0 con 0x7f5d6c06bf00 2023-10-10T19:08:37.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.312+0000 7f5d52ffd700 1 -- 172.21.15.175:0/1772108860 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c012670 con 0x7f5d6c06bf00 2023-10-10T19:08:37.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.312+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6c166940 con 0x7f5d6c06bf00 2023-10-10T19:08:37.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.313+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6c067570 con 0x7f5d6c06bf00 2023-10-10T19:08:37.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.313+0000 7f5d52ffd700 1 -- 172.21.15.175:0/1772108860 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5d5c012a60 con 0x7f5d6c06bf00 2023-10-10T19:08:37.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.313+0000 7f5d52ffd700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5d58066f60 0x7f5d58069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:37.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.313+0000 7f5d52ffd700 1 -- 172.21.15.175:0/1772108860 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f5d5c085360 con 0x7f5d6c06bf00 2023-10-10T19:08:37.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.313+0000 7f5d6a7fc700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5d58066f60 0x7f5d58069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:37.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.314+0000 7f5d6a7fc700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5d58066f60 0x7f5d58069410 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5d600098e0 tx=0x7f5d60017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:37.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.317+0000 7f5d52ffd700 1 -- 172.21.15.175:0/1772108860 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5d5c052150 con 0x7f5d6c06bf00 2023-10-10T19:08:37.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.540+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d6c06c690 con 0x7f5d6c06bf00 2023-10-10T19:08:37.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.541+0000 7f5d52ffd700 1 -- 172.21.15.175:0/1772108860 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+5442 (secure 0 0 0) 0x7f5d5c055770 con 0x7f5d6c06bf00 2023-10-10T19:08:37.542 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:37.542 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:36.875246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:37.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5d58066f60 msgr2=0x7f5d58069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:37.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5d58066f60 0x7f5d58069410 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5d600098e0 tx=0x7f5d60017000 comp rx=0 tx=0).stop 2023-10-10T19:08:37.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 msgr2=0x7f5d6c158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:37.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c158e30 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00cde0 tx=0x7f5d5c001be0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 shutdown_connections 2023-10-10T19:08:37.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5d58066f60 0x7f5d58069410 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5d6c156520 0x7f5d6c0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d6c06bf00 0x7f5d6c158e30 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 --2- 172.21.15.175:0/1772108860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0c5e40 0x7f5d6c0c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:37.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 >> 172.21.15.175:0/1772108860 conn(0x7f5d6c068900 msgr2=0x7f5d6c0c3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:37.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 shutdown_connections 2023-10-10T19:08:37.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:37.543+0000 7f5d716ad700 1 -- 172.21.15.175:0/1772108860 wait complete. 2023-10-10T19:08:37.546 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 178 2023-10-10T19:08:37.555 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:39.556 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:39.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.685+0000 7fe9bd298700 1 Processor -- start 2023-10-10T19:08:39.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.685+0000 7fe9bd298700 1 -- start start 2023-10-10T19:08:39.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9bd298700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9b808a100 0x7fe9b8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:39.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9bd298700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b80684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:39.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9bd298700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b8067fa0 con 0x7fe9b814fd90 2023-10-10T19:08:39.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9bd298700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b80c5600 con 0x7fe9b808a100 2023-10-10T19:08:39.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9bd298700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b8156b70 con 0x7fe9b80cd040 2023-10-10T19:08:39.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9b7fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe9b814fd90 0x7fe9b80c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46954/0 (socket says 172.21.15.175:46954) 2023-10-10T19:08:39.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9b7fff700 1 -- 172.21.15.175:0/4128807586 learned_addr learned my addr 172.21.15.175:0/4128807586 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:39.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9b77fe700 1 --2- 172.21.15.175:0/4128807586 >> v2:172.21.15.191:3300/0 conn(0x7fe9b808a100 0x7fe9b8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:39.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9bca97700 1 --2- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b80684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:39.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9b6ffd700 1 -- 172.21.15.175:0/4128807586 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062962067 0 0) 0x7fe9b8067fa0 con 0x7fe9b814fd90 2023-10-10T19:08:39.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.686+0000 7fe9b6ffd700 1 -- 172.21.15.175:0/4128807586 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a0003570 con 0x7fe9b814fd90 2023-10-10T19:08:39.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b77fe700 1 -- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 msgr2=0x7fe9b80684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:39.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b6ffd700 1 -- 172.21.15.175:0/4128807586 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 88289238 0 0) 0x7fe9a0003570 con 0x7fe9b814fd90 2023-10-10T19:08:39.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b77fe700 1 --2- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b80684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b77fe700 1 -- 172.21.15.175:0/4128807586 >> v1:172.21.15.175:6789/0 conn(0x7fe9b814fd90 legacy=0x7fe9b80c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:39.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b77fe700 1 -- 172.21.15.175:0/4128807586 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b8150160 con 0x7fe9b808a100 2023-10-10T19:08:39.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9bca97700 1 --2- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b80684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:08:39.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b77fe700 1 --2- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b808a100 0x7fe9b8159c30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0098e0 tx=0x7fe9ac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3acc67a91822731c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:39.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b6ffd700 1 -- 172.21.15.175:0/4128807586 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8001470 con 0x7fe9b808a100 2023-10-10T19:08:39.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b6ffd700 1 -- 172.21.15.175:0/4128807586 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac011420 con 0x7fe9b808a100 2023-10-10T19:08:39.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9b6ffd700 1 -- 172.21.15.175:0/4128807586 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac0117a0 con 0x7fe9b808a100 2023-10-10T19:08:39.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9bd298700 1 -- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b808a100 msgr2=0x7fe9b8159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:39.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9bd298700 1 --2- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b808a100 0x7fe9b8159c30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0098e0 tx=0x7fe9ac0091a0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.687+0000 7fe9bd298700 1 -- 172.21.15.175:0/4128807586 shutdown_connections 2023-10-10T19:08:39.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.688+0000 7fe9bd298700 1 --2- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b80684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.688+0000 7fe9bd298700 1 --2- 172.21.15.175:0/4128807586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b808a100 0x7fe9b8159c30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.688+0000 7fe9bd298700 1 -- 172.21.15.175:0/4128807586 >> 172.21.15.175:0/4128807586 conn(0x7fe9b8063d00 msgr2=0x7fe9b80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:39.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.688+0000 7fe9bd298700 1 -- 172.21.15.175:0/4128807586 shutdown_connections 2023-10-10T19:08:39.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.688+0000 7fe9bd298700 1 -- 172.21.15.175:0/4128807586 wait complete. 2023-10-10T19:08:39.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.689+0000 7fe9bd298700 1 Processor -- start 2023-10-10T19:08:39.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.689+0000 7fe9bd298700 1 -- start start 2023-10-10T19:08:39.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.689+0000 7fe9bd298700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9b808a100 0x7fe9b814bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:39.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.689+0000 7fe9bd298700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b8154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:39.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.689+0000 7fe9bd298700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b814fd90 0x7fe9b8169640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:39.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.689+0000 7fe9bd298700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b8158ab0 con 0x7fe9b808a100 2023-10-10T19:08:39.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.689+0000 7fe9bd298700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b8168ed0 con 0x7fe9b814fd90 2023-10-10T19:08:39.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.689+0000 7fe9bd298700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b81691b0 con 0x7fe9b80cd040 2023-10-10T19:08:39.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9b808a100 0x7fe9b814bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:39.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9b808a100 0x7fe9b814bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55050/0 (socket says 172.21.15.175:55050) 2023-10-10T19:08:39.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 -- 172.21.15.175:0/453424442 learned_addr learned my addr 172.21.15.175:0/453424442 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:39.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b77fe700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b8154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:39.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9bca97700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b814fd90 0x7fe9b8169640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:39.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 -- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 msgr2=0x7fe9b8154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:39.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b8154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 -- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b814fd90 msgr2=0x7fe9b8169640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:39.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b814fd90 0x7fe9b8169640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 -- 172.21.15.175:0/453424442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b0009a20 con 0x7fe9b808a100 2023-10-10T19:08:39.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b77fe700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b8154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:08:39.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9bca97700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b814fd90 0x7fe9b8169640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:08:39.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9b7fff700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9b808a100 0x7fe9b814bc90 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800cbc0 tx=0x7fe9a8001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:39.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe99ffff700 1 -- 172.21.15.175:0/453424442 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8007360 con 0x7fe9b808a100 2023-10-10T19:08:39.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe99ffff700 1 -- 172.21.15.175:0/453424442 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a80074e0 con 0x7fe9b808a100 2023-10-10T19:08:39.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac009420 con 0x7fe9b808a100 2023-10-10T19:08:39.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe99ffff700 1 -- 172.21.15.175:0/453424442 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8004830 con 0x7fe9b808a100 2023-10-10T19:08:39.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.690+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b8158e60 con 0x7fe9b808a100 2023-10-10T19:08:39.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.691+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b8067570 con 0x7fe9b808a100 2023-10-10T19:08:39.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.692+0000 7fe99ffff700 1 -- 172.21.15.175:0/453424442 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe9a8004c20 con 0x7fe9b808a100 2023-10-10T19:08:39.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.692+0000 7fe99ffff700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9a4066f30 0x7fe9a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:39.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.692+0000 7fe99ffff700 1 -- 172.21.15.175:0/453424442 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fe9a8084190 con 0x7fe9b808a100 2023-10-10T19:08:39.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.692+0000 7fe9b77fe700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9a4066f30 0x7fe9a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:39.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.693+0000 7fe9b77fe700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9a4066f30 0x7fe9a40693e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0091e0 tx=0x7fe9ac016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:39.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.696+0000 7fe99ffff700 1 -- 172.21.15.175:0/453424442 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe9a8051000 con 0x7fe9b808a100 2023-10-10T19:08:39.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.923+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9b8155f80 con 0x7fe9b808a100 2023-10-10T19:08:39.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.924+0000 7fe99ffff700 1 -- 172.21.15.175:0/453424442 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5440 (secure 0 0 0) 0x7fe9a8054620 con 0x7fe9b808a100 2023-10-10T19:08:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:39.925 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:39.659081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:39.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9a4066f30 msgr2=0x7fe9a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9a4066f30 0x7fe9a40693e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0091e0 tx=0x7fe9ac016000 comp rx=0 tx=0).stop 2023-10-10T19:08:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9b808a100 msgr2=0x7fe9b814bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9b808a100 0x7fe9b814bc90 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800cbc0 tx=0x7fe9a8001a30 comp rx=0 tx=0).stop 2023-10-10T19:08:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 shutdown_connections 2023-10-10T19:08:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9a4066f30 0x7fe9a40693e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9b814fd90 0x7fe9b8169640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9b808a100 0x7fe9b814bc90 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 --2- 172.21.15.175:0/453424442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9b80cd040 0x7fe9b8154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:39.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 >> 172.21.15.175:0/453424442 conn(0x7fe9b8063d00 msgr2=0x7fe9b814ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:39.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 shutdown_connections 2023-10-10T19:08:39.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:39.926+0000 7fe9bd298700 1 -- 172.21.15.175:0/453424442 wait complete. 2023-10-10T19:08:39.929 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 179 2023-10-10T19:08:39.938 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:41.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:42.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.061+0000 7fabb44cb700 1 Processor -- start 2023-10-10T19:08:42.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.061+0000 7fabb44cb700 1 -- start start 2023-10-10T19:08:42.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.062+0000 7fabb44cb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b1fc0 0x7fabac155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:42.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.062+0000 7fabb44cb700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fabac0b2390 0x7fabac068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:42.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.062+0000 7fabb44cb700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabac1574b0 con 0x7fabac068bd0 2023-10-10T19:08:42.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.062+0000 7fabb44cb700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabac150770 con 0x7fabac0b2390 2023-10-10T19:08:42.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.062+0000 7fabb44cb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0afad0 con 0x7fabac0b1fc0 2023-10-10T19:08:42.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.062+0000 7fabb1a66700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fabac0b2390 0x7fabac068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:42.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.062+0000 7fabb1a66700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fabac0b2390 0x7fabac068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55058/0 (socket says 172.21.15.175:55058) 2023-10-10T19:08:42.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.062+0000 7fabb1a66700 1 -- 172.21.15.175:0/2731061311 learned_addr learned my addr 172.21.15.175:0/2731061311 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:42.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb2267700 1 --2- 172.21.15.175:0/2731061311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b1fc0 0x7fabac155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:42.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb1a66700 1 -- 172.21.15.175:0/2731061311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b1fc0 msgr2=0x7fabac155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:42.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb1a66700 1 --2- 172.21.15.175:0/2731061311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b1fc0 0x7fabac155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb1a66700 1 -- 172.21.15.175:0/2731061311 >> v1:172.21.15.191:6789/0 conn(0x7fabac068bd0 legacy=0x7fabac0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:42.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb1a66700 1 -- 172.21.15.175:0/2731061311 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac0af4c0 con 0x7fabac0b2390 2023-10-10T19:08:42.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb1a66700 1 --2- 172.21.15.175:0/2731061311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac0b2390 0x7fabac068690 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0099b0 tx=0x7fab9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffc975c4078db2c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:42.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb1265700 1 -- 172.21.15.175:0/2731061311 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00b8f0 con 0x7fabac0b2390 2023-10-10T19:08:42.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb1265700 1 -- 172.21.15.175:0/2731061311 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c00ba50 con 0x7fabac0b2390 2023-10-10T19:08:42.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb1265700 1 -- 172.21.15.175:0/2731061311 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00bdf0 con 0x7fabac0b2390 2023-10-10T19:08:42.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb44cb700 1 -- 172.21.15.175:0/2731061311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac0b2390 msgr2=0x7fabac068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:42.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.063+0000 7fabb44cb700 1 --2- 172.21.15.175:0/2731061311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac0b2390 0x7fabac068690 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0099b0 tx=0x7fab9c009250 comp rx=0 tx=0).stop 2023-10-10T19:08:42.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.064+0000 7fabb44cb700 1 -- 172.21.15.175:0/2731061311 shutdown_connections 2023-10-10T19:08:42.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.064+0000 7fabb44cb700 1 --2- 172.21.15.175:0/2731061311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b1fc0 0x7fabac155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.064+0000 7fabb44cb700 1 --2- 172.21.15.175:0/2731061311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac0b2390 0x7fabac068690 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.064+0000 7fabb44cb700 1 -- 172.21.15.175:0/2731061311 >> 172.21.15.175:0/2731061311 conn(0x7fabac063230 msgr2=0x7fabac063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:42.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.064+0000 7fabb44cb700 1 -- 172.21.15.175:0/2731061311 shutdown_connections 2023-10-10T19:08:42.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.064+0000 7fabb44cb700 1 -- 172.21.15.175:0/2731061311 wait complete. 2023-10-10T19:08:42.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb44cb700 1 Processor -- start 2023-10-10T19:08:42.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb44cb700 1 -- start start 2023-10-10T19:08:42.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb44cb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac068bd0 0x7fabac169fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:42.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb44cb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabac0b1fc0 0x7fabac155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:42.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb44cb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b2390 0x7fabac14edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:42.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb44cb700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0ccd80 con 0x7fabac068bd0 2023-10-10T19:08:42.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb44cb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac166960 con 0x7fabac0b1fc0 2023-10-10T19:08:42.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb44cb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac166090 con 0x7fabac0b2390 2023-10-10T19:08:42.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb2267700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac068bd0 0x7fabac169fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:42.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb2267700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac068bd0 0x7fabac169fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55060/0 (socket says 172.21.15.175:55060) 2023-10-10T19:08:42.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb2267700 1 -- 172.21.15.175:0/1070445464 learned_addr learned my addr 172.21.15.175:0/1070445464 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:42.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb1a66700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabac0b1fc0 0x7fabac155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:42.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.065+0000 7fabb2a68700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b2390 0x7fabac14edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:42.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7fabb2267700 1 -- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b2390 msgr2=0x7fabac14edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:42.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7fabb2267700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b2390 0x7fabac14edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7fabb2267700 1 -- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabac0b1fc0 msgr2=0x7fabac155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:42.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7fabb2267700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabac0b1fc0 0x7fabac155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7fabb2267700 1 -- 172.21.15.175:0/1070445464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c0094f0 con 0x7fabac068bd0 2023-10-10T19:08:42.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7fabb2267700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac068bd0 0x7fabac169fd0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7faba800d8c0 tx=0x7faba8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:42.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7faba27fc700 1 -- 172.21.15.175:0/1070445464 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba80125f0 con 0x7fabac068bd0 2023-10-10T19:08:42.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7faba27fc700 1 -- 172.21.15.175:0/1070445464 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba8012750 con 0x7fabac068bd0 2023-10-10T19:08:42.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabac0ccfb0 con 0x7fabac068bd0 2023-10-10T19:08:42.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7faba27fc700 1 -- 172.21.15.175:0/1070445464 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba8012af0 con 0x7fabac068bd0 2023-10-10T19:08:42.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.066+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabac166310 con 0x7fabac068bd0 2023-10-10T19:08:42.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.067+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabac071340 con 0x7fabac068bd0 2023-10-10T19:08:42.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.067+0000 7faba27fc700 1 -- 172.21.15.175:0/1070445464 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faba801e460 con 0x7fabac068bd0 2023-10-10T19:08:42.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.068+0000 7faba27fc700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab90066f30 0x7fab900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:42.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.068+0000 7faba27fc700 1 -- 172.21.15.175:0/1070445464 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7faba8084ee0 con 0x7fabac068bd0 2023-10-10T19:08:42.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.068+0000 7fabb1a66700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab90066f30 0x7fab900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:42.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.068+0000 7fabb1a66700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab90066f30 0x7fab900693e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0099b0 tx=0x7fab9c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:42.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.072+0000 7faba27fc700 1 -- 172.21.15.175:0/1070445464 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faba8051cd0 con 0x7fabac068bd0 2023-10-10T19:08:42.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.293+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabac067570 con 0x7fabac068bd0 2023-10-10T19:08:42.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.293+0000 7faba27fc700 1 -- 172.21.15.175:0/1070445464 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5440 (secure 0 0 0) 0x7faba80552f0 con 0x7fabac068bd0 2023-10-10T19:08:42.294 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:42.294 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:39.659081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:42.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab90066f30 msgr2=0x7fab900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:42.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab90066f30 0x7fab900693e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0099b0 tx=0x7fab9c017000 comp rx=0 tx=0).stop 2023-10-10T19:08:42.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac068bd0 msgr2=0x7fabac169fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:42.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac068bd0 0x7fabac169fd0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7faba800d8c0 tx=0x7faba8008040 comp rx=0 tx=0).stop 2023-10-10T19:08:42.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 shutdown_connections 2023-10-10T19:08:42.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab90066f30 0x7fab900693e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fabac0b2390 0x7fabac14edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fabac068bd0 0x7fabac169fd0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 --2- 172.21.15.175:0/1070445464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fabac0b1fc0 0x7fabac155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:42.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 >> 172.21.15.175:0/1070445464 conn(0x7fabac063230 msgr2=0x7fabac14fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:42.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 shutdown_connections 2023-10-10T19:08:42.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:42.295+0000 7fabb44cb700 1 -- 172.21.15.175:0/1070445464 wait complete. 2023-10-10T19:08:42.298 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 179 2023-10-10T19:08:42.307 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 179, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 179, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:resolve", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:08:39.659081+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 6106, "mds_3": 6618, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:08:44.309 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:44.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.436+0000 7fda919e5700 1 Processor -- start 2023-10-10T19:08:44.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.436+0000 7fda919e5700 1 -- start start 2023-10-10T19:08:44.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda919e5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c08a100 0x7fda8c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:44.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda919e5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fda8c0cd040 0x7fda8c0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:44.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda919e5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8c067fa0 con 0x7fda8c14fd90 2023-10-10T19:08:44.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda919e5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c0c5600 con 0x7fda8c0cd040 2023-10-10T19:08:44.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda919e5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c156b70 con 0x7fda8c08a100 2023-10-10T19:08:44.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda911e4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fda8c0cd040 0x7fda8c0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:44.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda909e3700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fda8c14fd90 0x7fda8c0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46970/0 (socket says 172.21.15.175:46970) 2023-10-10T19:08:44.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda911e4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fda8c0cd040 0x7fda8c0684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55064/0 (socket says 172.21.15.175:55064) 2023-10-10T19:08:44.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda909e3700 1 -- 172.21.15.175:0/3525800133 learned_addr learned my addr 172.21.15.175:0/3525800133 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:44.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda83fff700 1 --2- 172.21.15.175:0/3525800133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c08a100 0x7fda8c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:44.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.437+0000 7fda911e4700 1 -- 172.21.15.175:0/3525800133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c08a100 msgr2=0x7fda8c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:44.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda911e4700 1 --2- 172.21.15.175:0/3525800133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c08a100 0x7fda8c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda911e4700 1 -- 172.21.15.175:0/3525800133 >> v1:172.21.15.175:6789/0 conn(0x7fda8c14fd90 legacy=0x7fda8c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:44.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda837fe700 1 -- 172.21.15.175:0/3525800133 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176120759 0 0) 0x7fda8c067fa0 con 0x7fda8c14fd90 2023-10-10T19:08:44.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda911e4700 1 -- 172.21.15.175:0/3525800133 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c150160 con 0x7fda8c0cd040 2023-10-10T19:08:44.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda83fff700 1 --2- 172.21.15.175:0/3525800133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c08a100 0x7fda8c159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:08:44.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda911e4700 1 --2- 172.21.15.175:0/3525800133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c0cd040 0x7fda8c0684f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fda840098e0 tx=0x7fda840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9d9d701921da0a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:44.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda837fe700 1 -- 172.21.15.175:0/3525800133 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84009e80 con 0x7fda8c0cd040 2023-10-10T19:08:44.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda837fe700 1 -- 172.21.15.175:0/3525800133 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8400b860 con 0x7fda8c0cd040 2023-10-10T19:08:44.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda837fe700 1 -- 172.21.15.175:0/3525800133 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8400bbe0 con 0x7fda8c0cd040 2023-10-10T19:08:44.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda919e5700 1 -- 172.21.15.175:0/3525800133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c0cd040 msgr2=0x7fda8c0684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:44.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda919e5700 1 --2- 172.21.15.175:0/3525800133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c0cd040 0x7fda8c0684f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fda840098e0 tx=0x7fda840091a0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda919e5700 1 -- 172.21.15.175:0/3525800133 shutdown_connections 2023-10-10T19:08:44.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda919e5700 1 --2- 172.21.15.175:0/3525800133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c0cd040 0x7fda8c0684f0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda919e5700 1 --2- 172.21.15.175:0/3525800133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c08a100 0x7fda8c159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda919e5700 1 -- 172.21.15.175:0/3525800133 >> 172.21.15.175:0/3525800133 conn(0x7fda8c063d00 msgr2=0x7fda8c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:44.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda919e5700 1 -- 172.21.15.175:0/3525800133 shutdown_connections 2023-10-10T19:08:44.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.438+0000 7fda919e5700 1 -- 172.21.15.175:0/3525800133 wait complete. 2023-10-10T19:08:44.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.440+0000 7fda919e5700 1 Processor -- start 2023-10-10T19:08:44.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.440+0000 7fda919e5700 1 -- start start 2023-10-10T19:08:44.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.440+0000 7fda919e5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda8c08a100 0x7fda8c1519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:44.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.440+0000 7fda919e5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c0cd040 0x7fda8c0cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:44.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.440+0000 7fda919e5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c14fd90 0x7fda8c14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:44.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.440+0000 7fda919e5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c152520 con 0x7fda8c14fd90 2023-10-10T19:08:44.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.440+0000 7fda919e5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c151540 con 0x7fda8c08a100 2023-10-10T19:08:44.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda919e5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c151f10 con 0x7fda8c0cd040 2023-10-10T19:08:44.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c14fd90 0x7fda8c14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:44.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c14fd90 0x7fda8c14bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55076/0 (socket says 172.21.15.175:55076) 2023-10-10T19:08:44.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 -- 172.21.15.175:0/2483943170 learned_addr learned my addr 172.21.15.175:0/2483943170 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:44.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda909e3700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda8c08a100 0x7fda8c1519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:44.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda83fff700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c0cd040 0x7fda8c0cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:44.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 -- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c0cd040 msgr2=0x7fda8c0cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:44.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c0cd040 0x7fda8c0cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 -- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda8c08a100 msgr2=0x7fda8c1519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:44.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda8c08a100 0x7fda8c1519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 -- 172.21.15.175:0/2483943170 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84009420 con 0x7fda8c14fd90 2023-10-10T19:08:44.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda83fff700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c0cd040 0x7fda8c0cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:08:44.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda911e4700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c14fd90 0x7fda8c14bc30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fda8400b3c0 tx=0x7fda84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:44.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda80ff9700 1 -- 172.21.15.175:0/2483943170 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84011630 con 0x7fda8c14fd90 2023-10-10T19:08:44.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda80ff9700 1 -- 172.21.15.175:0/2483943170 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda84011790 con 0x7fda8c14fd90 2023-10-10T19:08:44.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c152240 con 0x7fda8c14fd90 2023-10-10T19:08:44.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda80ff9700 1 -- 172.21.15.175:0/2483943170 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84011b50 con 0x7fda8c14fd90 2023-10-10T19:08:44.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.441+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c150ef0 con 0x7fda8c14fd90 2023-10-10T19:08:44.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.442+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c071210 con 0x7fda8c14fd90 2023-10-10T19:08:44.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.443+0000 7fda80ff9700 1 -- 172.21.15.175:0/2483943170 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fda840253f0 con 0x7fda8c14fd90 2023-10-10T19:08:44.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.443+0000 7fda80ff9700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda78066f30 0x7fda780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:44.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.443+0000 7fda80ff9700 1 -- 172.21.15.175:0/2483943170 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fda84086620 con 0x7fda8c14fd90 2023-10-10T19:08:44.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.443+0000 7fda909e3700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda78066f30 0x7fda780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:44.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.444+0000 7fda909e3700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda78066f30 0x7fda780693e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fda88002250 tx=0x7fda88001a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:44.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.447+0000 7fda80ff9700 1 -- 172.21.15.175:0/2483943170 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda8c071210 con 0x7fda8c14fd90 2023-10-10T19:08:44.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.675+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda8c071210 con 0x7fda8c14fd90 2023-10-10T19:08:44.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.676+0000 7fda80ff9700 1 -- 172.21.15.175:0/2483943170 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5442 (secure 0 0 0) 0x7fda8c071210 con 0x7fda8c14fd90 2023-10-10T19:08:44.677 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:44.677 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:42.663052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:44.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda78066f30 msgr2=0x7fda780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:44.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda78066f30 0x7fda780693e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fda88002250 tx=0x7fda88001a20 comp rx=0 tx=0).stop 2023-10-10T19:08:44.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c14fd90 msgr2=0x7fda8c14bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:44.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c14fd90 0x7fda8c14bc30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fda8400b3c0 tx=0x7fda84017000 comp rx=0 tx=0).stop 2023-10-10T19:08:44.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 shutdown_connections 2023-10-10T19:08:44.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fda78066f30 0x7fda780693e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda8c14fd90 0x7fda8c14bc30 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda8c08a100 0x7fda8c1519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 --2- 172.21.15.175:0/2483943170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda8c0cd040 0x7fda8c0cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:44.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 >> 172.21.15.175:0/2483943170 conn(0x7fda8c063d00 msgr2=0x7fda8c0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:44.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 shutdown_connections 2023-10-10T19:08:44.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:44.678+0000 7fda919e5700 1 -- 172.21.15.175:0/2483943170 wait complete. 2023-10-10T19:08:44.682 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 180 2023-10-10T19:08:44.692 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:46.693 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:46.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.821+0000 7fe6f52fe700 1 Processor -- start 2023-10-10T19:08:46.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.821+0000 7fe6f52fe700 1 -- start start 2023-10-10T19:08:46.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.822+0000 7fe6f52fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe6f006cf90 0x7fe6f006d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.822+0000 7fe6f52fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6f00c0f70 0x7fe6f0156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.822+0000 7fe6f52fe700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f006d8a0 con 0x7fe6f00bf2b0 2023-10-10T19:08:46.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.822+0000 7fe6f52fe700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f00c0990 con 0x7fe6f006cf90 2023-10-10T19:08:46.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.822+0000 7fe6f52fe700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f01595e0 con 0x7fe6f00c0f70 2023-10-10T19:08:46.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.822+0000 7fe6f4afd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe6f00bf2b0 0x7fe6f00ca3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44222/0 (socket says 172.21.15.175:44222) 2023-10-10T19:08:46.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.822+0000 7fe6f4afd700 1 -- 172.21.15.175:0/2575786754 learned_addr learned my addr 172.21.15.175:0/2575786754 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:46.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6effff700 1 --2- 172.21.15.175:0/2575786754 >> v2:172.21.15.191:3300/0 conn(0x7fe6f006cf90 0x7fe6f006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49617318 0 0) 0x7fe6f006d8a0 con 0x7fe6f00bf2b0 2023-10-10T19:08:46.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6ef7fe700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6f00c0f70 0x7fe6f0156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d8003570 con 0x7fe6f00bf2b0 2023-10-10T19:08:46.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2863603914 0 0) 0x7fe6d8003570 con 0x7fe6f00bf2b0 2023-10-10T19:08:46.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6f00c0f70 msgr2=0x7fe6f0156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:46.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6f00c0f70 0x7fe6f0156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:46.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 >> v2:172.21.15.191:3300/0 conn(0x7fe6f006cf90 msgr2=0x7fe6f006d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:46.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 --2- 172.21.15.175:0/2575786754 >> v2:172.21.15.191:3300/0 conn(0x7fe6f006cf90 0x7fe6f006d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:46.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f015a440 con 0x7fe6f00bf2b0 2023-10-10T19:08:46.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fe6e80017f0 con 0x7fe6f00bf2b0 2023-10-10T19:08:46.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 >> v1:172.21.15.175:6789/0 conn(0x7fe6f00bf2b0 legacy=0x7fe6f00ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:46.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6d8003a60 0x7fe6d8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6d8004810 0x7fe6d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6f00c0f70 0x7fe6d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f01595e0 con 0x7fe6d8004810 2023-10-10T19:08:46.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d800ba60 con 0x7fe6d8003a60 2023-10-10T19:08:46.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d800bba0 con 0x7fe6f00c0f70 2023-10-10T19:08:46.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6ef7fe700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6d8004810 0x7fe6d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6effff700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6d8003a60 0x7fe6d8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6f4afd700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6f00c0f70 0x7fe6d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6ef7fe700 1 -- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6f00c0f70 msgr2=0x7fe6d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:46.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6ef7fe700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6f00c0f70 0x7fe6d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:46.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6ef7fe700 1 -- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6d8003a60 msgr2=0x7fe6d8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:46.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6ef7fe700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6d8003a60 0x7fe6d8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:46.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.823+0000 7fe6ef7fe700 1 -- 172.21.15.175:0/2575786754 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d8003570 con 0x7fe6d8004810 2023-10-10T19:08:46.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6ef7fe700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6d8004810 0x7fe6d8004c80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009c20 tx=0x7fe6e4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef75a7e86d08db9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:46.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4005a00 con 0x7fe6d8004810 2023-10-10T19:08:46.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e4005b60 con 0x7fe6d8004810 2023-10-10T19:08:46.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6eeffd700 1 -- 172.21.15.175:0/2575786754 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e400e710 con 0x7fe6d8004810 2023-10-10T19:08:46.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6f52fe700 1 -- 172.21.15.175:0/2575786754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6d8004810 msgr2=0x7fe6d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:46.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6f52fe700 1 --2- 172.21.15.175:0/2575786754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6d8004810 0x7fe6d8004c80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009c20 tx=0x7fe6e4009120 comp rx=0 tx=0).stop 2023-10-10T19:08:46.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6f4afd700 1 -- 172.21.15.175:0/2575786754 reap_dead start 2023-10-10T19:08:46.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6f52fe700 1 -- 172.21.15.175:0/2575786754 shutdown_connections 2023-10-10T19:08:46.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6f52fe700 1 -- 172.21.15.175:0/2575786754 >> 172.21.15.175:0/2575786754 conn(0x7fe6f006b310 msgr2=0x7fe6f006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:46.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6f52fe700 1 -- 172.21.15.175:0/2575786754 shutdown_connections 2023-10-10T19:08:46.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.824+0000 7fe6f52fe700 1 -- 172.21.15.175:0/2575786754 wait complete. 2023-10-10T19:08:46.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.825+0000 7fe6f52fe700 1 Processor -- start 2023-10-10T19:08:46.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f52fe700 1 -- start start 2023-10-10T19:08:46.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f52fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6d8004810 0x7fe6f01668b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f52fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6f01539c0 0x7fe6f014b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f52fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6f014ba80 0x7fe6f0148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f52fe700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f0159090 con 0x7fe6f014ba80 2023-10-10T19:08:46.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f52fe700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00c4650 con 0x7fe6f01539c0 2023-10-10T19:08:46.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f52fe700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f014b0b0 con 0x7fe6d8004810 2023-10-10T19:08:46.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f4afd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6f014ba80 0x7fe6f0148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f4afd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6f014ba80 0x7fe6f0148ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50462/0 (socket says 172.21.15.175:50462) 2023-10-10T19:08:46.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.826+0000 7fe6f4afd700 1 -- 172.21.15.175:0/4021445604 learned_addr learned my addr 172.21.15.175:0/4021445604 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:46.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6effff700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6d8004810 0x7fe6f01668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6ef7fe700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6f01539c0 0x7fe6f014b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6f4afd700 1 -- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6d8004810 msgr2=0x7fe6f01668b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:46.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6f4afd700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6d8004810 0x7fe6f01668b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:46.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6f4afd700 1 -- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6f01539c0 msgr2=0x7fe6f014b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:46.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6f4afd700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6f01539c0 0x7fe6f014b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:46.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6f4afd700 1 -- 172.21.15.175:0/4021445604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f014bed0 con 0x7fe6f014ba80 2023-10-10T19:08:46.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6ef7fe700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6f01539c0 0x7fe6f014b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:08:46.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6f4afd700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6f014ba80 0x7fe6f0148ad0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0065040 tx=0x7fe6e8002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:46.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6d7fff700 1 -- 172.21.15.175:0/4021445604 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8007b40 con 0x7fe6f014ba80 2023-10-10T19:08:46.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6d7fff700 1 -- 172.21.15.175:0/4021445604 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e800ad80 con 0x7fe6f014ba80 2023-10-10T19:08:46.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f0158af0 con 0x7fe6f014ba80 2023-10-10T19:08:46.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6d7fff700 1 -- 172.21.15.175:0/4021445604 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8013620 con 0x7fe6f014ba80 2023-10-10T19:08:46.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.827+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f014aa60 con 0x7fe6f014ba80 2023-10-10T19:08:46.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.828+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f014a8c0 con 0x7fe6f014ba80 2023-10-10T19:08:46.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.828+0000 7fe6d7fff700 1 -- 172.21.15.175:0/4021445604 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe6e8013990 con 0x7fe6f014ba80 2023-10-10T19:08:46.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.829+0000 7fe6d7fff700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe6dc066f60 0x7fe6dc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:46.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.829+0000 7fe6d7fff700 1 -- 172.21.15.175:0/4021445604 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fe6e80852f0 con 0x7fe6f014ba80 2023-10-10T19:08:46.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.829+0000 7fe6effff700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe6dc066f60 0x7fe6dc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:46.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.829+0000 7fe6effff700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe6dc066f60 0x7fe6dc069410 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0006ea0 tx=0x7fe6e0009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:46.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:46.833+0000 7fe6d7fff700 1 -- 172.21.15.175:0/4021445604 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6f014a8c0 con 0x7fe6f014ba80 2023-10-10T19:08:47.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.058+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6f014a8c0 con 0x7fe6f014ba80 2023-10-10T19:08:47.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.058+0000 7fe6d7fff700 1 -- 172.21.15.175:0/4021445604 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5440 (secure 0 0 0) 0x7fe6f014a8c0 con 0x7fe6f014ba80 2023-10-10T19:08:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:47.059 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:44.682405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:47.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.060+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe6dc066f60 msgr2=0x7fe6dc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:47.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.060+0000 7fe6f52fe700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe6dc066f60 0x7fe6dc069410 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0006ea0 tx=0x7fe6e0009120 comp rx=0 tx=0).stop 2023-10-10T19:08:47.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.060+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6f014ba80 msgr2=0x7fe6f0148ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:47.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.060+0000 7fe6f52fe700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6f014ba80 0x7fe6f0148ad0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0065040 tx=0x7fe6e8002210 comp rx=0 tx=0).stop 2023-10-10T19:08:47.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.060+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 shutdown_connections 2023-10-10T19:08:47.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.060+0000 7fe6f52fe700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe6dc066f60 0x7fe6dc069410 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:47.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.061+0000 7fe6f52fe700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6f014ba80 0x7fe6f0148ad0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:47.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.061+0000 7fe6f52fe700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6d8004810 0x7fe6f01668b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.061+0000 7fe6f52fe700 1 --2- 172.21.15.175:0/4021445604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6f01539c0 0x7fe6f014b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.061+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 >> 172.21.15.175:0/4021445604 conn(0x7fe6f006b310 msgr2=0x7fe6f00c6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.061+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 shutdown_connections 2023-10-10T19:08:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:47.061+0000 7fe6f52fe700 1 -- 172.21.15.175:0/4021445604 wait complete. 2023-10-10T19:08:47.064 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 181 2023-10-10T19:08:47.073 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:49.074 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:49.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.200+0000 7f4290bab700 1 Processor -- start 2023-10-10T19:08:49.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.200+0000 7f4290bab700 1 -- start start 2023-10-10T19:08:49.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f4290bab700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f428c08a100 0x7f428c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:49.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f4290bab700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c0cd040 0x7f428c0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:49.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f4290bab700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428c067fa0 con 0x7f428c14fd90 2023-10-10T19:08:49.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f4290bab700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428c0c5600 con 0x7f428c08a100 2023-10-10T19:08:49.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f4290bab700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428c156b70 con 0x7f428c0cd040 2023-10-10T19:08:49.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f428affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f428c08a100 0x7f428c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:49.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f428affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f428c08a100 0x7f428c159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50464/0 (socket says 172.21.15.175:50464) 2023-10-10T19:08:49.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f428affd700 1 -- 172.21.15.175:0/679923986 learned_addr learned my addr 172.21.15.175:0/679923986 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:49.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f428bfff700 1 --2- 172.21.15.175:0/679923986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c0cd040 0x7f428c0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:49.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f428affd700 1 -- 172.21.15.175:0/679923986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c0cd040 msgr2=0x7f428c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:49.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f428affd700 1 --2- 172.21.15.175:0/679923986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c0cd040 0x7f428c0684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.201+0000 7f428affd700 1 -- 172.21.15.175:0/679923986 >> v1:172.21.15.191:6789/0 conn(0x7f428c14fd90 legacy=0x7f428c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:49.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f428affd700 1 -- 172.21.15.175:0/679923986 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c150160 con 0x7f428c08a100 2023-10-10T19:08:49.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f428affd700 1 --2- 172.21.15.175:0/679923986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c159c30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f42800099b0 tx=0x7f4280009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245ce65d0cbcd67b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:49.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f428a7fc700 1 -- 172.21.15.175:0/679923986 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428000b860 con 0x7f428c08a100 2023-10-10T19:08:49.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f428a7fc700 1 -- 172.21.15.175:0/679923986 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428000b9c0 con 0x7f428c08a100 2023-10-10T19:08:49.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f428a7fc700 1 -- 172.21.15.175:0/679923986 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428000bd60 con 0x7f428c08a100 2023-10-10T19:08:49.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f4290bab700 1 -- 172.21.15.175:0/679923986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 msgr2=0x7f428c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:49.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f4290bab700 1 --2- 172.21.15.175:0/679923986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c159c30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f42800099b0 tx=0x7f4280009250 comp rx=0 tx=0).stop 2023-10-10T19:08:49.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f4290bab700 1 -- 172.21.15.175:0/679923986 shutdown_connections 2023-10-10T19:08:49.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f4290bab700 1 --2- 172.21.15.175:0/679923986 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c0cd040 0x7f428c0684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f4290bab700 1 --2- 172.21.15.175:0/679923986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c159c30 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f4290bab700 1 -- 172.21.15.175:0/679923986 >> 172.21.15.175:0/679923986 conn(0x7f428c063d00 msgr2=0x7f428c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:49.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f4290bab700 1 -- 172.21.15.175:0/679923986 shutdown_connections 2023-10-10T19:08:49.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.202+0000 7f4290bab700 1 -- 172.21.15.175:0/679923986 wait complete. 2023-10-10T19:08:49.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.204+0000 7f4290bab700 1 Processor -- start 2023-10-10T19:08:49.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.204+0000 7f4290bab700 1 -- start start 2023-10-10T19:08:49.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.204+0000 7f4290bab700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c169590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:49.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.204+0000 7f4290bab700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c0cd040 0x7f428c169ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:49.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.204+0000 7f4290bab700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c14fd90 0x7f428c14c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:49.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.204+0000 7f4290bab700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c166ca0 con 0x7f428c08a100 2023-10-10T19:08:49.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.204+0000 7f4290bab700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c166780 con 0x7f428c0cd040 2023-10-10T19:08:49.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.204+0000 7f4290bab700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428c0a7db0 con 0x7f428c14fd90 2023-10-10T19:08:49.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c169590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:49.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c169590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50468/0 (socket says 172.21.15.175:50468) 2023-10-10T19:08:49.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 -- 172.21.15.175:0/425207608 learned_addr learned my addr 172.21.15.175:0/425207608 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:49.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428affd700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c0cd040 0x7f428c169ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:49.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428bfff700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c14fd90 0x7f428c14c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:49.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 -- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c14fd90 msgr2=0x7f428c14c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:49.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c14fd90 0x7f428c14c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 -- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c0cd040 msgr2=0x7f428c169ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:49.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c0cd040 0x7f428c169ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 -- 172.21.15.175:0/425207608 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800094f0 con 0x7f428c08a100 2023-10-10T19:08:49.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f428b7fe700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c169590 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c4e0 tx=0x7f427c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:49.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f42737fe700 1 -- 172.21.15.175:0/425207608 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c00cea0 con 0x7f428c08a100 2023-10-10T19:08:49.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f42737fe700 1 -- 172.21.15.175:0/425207608 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c011590 con 0x7f428c08a100 2023-10-10T19:08:49.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c1669c0 con 0x7f428c08a100 2023-10-10T19:08:49.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f42737fe700 1 -- 172.21.15.175:0/425207608 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c011930 con 0x7f428c08a100 2023-10-10T19:08:49.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.205+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c0a8330 con 0x7f428c08a100 2023-10-10T19:08:49.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.206+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c1692b0 con 0x7f428c08a100 2023-10-10T19:08:49.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.206+0000 7f42737fe700 1 -- 172.21.15.175:0/425207608 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f427c011ce0 con 0x7f428c08a100 2023-10-10T19:08:49.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.207+0000 7f42737fe700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4278066f30 0x7f42780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:49.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.207+0000 7f42737fe700 1 -- 172.21.15.175:0/425207608 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f427c083ca0 con 0x7f428c08a100 2023-10-10T19:08:49.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.207+0000 7f428affd700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4278066f30 0x7f42780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:49.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.207+0000 7f428affd700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4278066f30 0x7f42780693e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f42800099b0 tx=0x7f4280017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:49.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.211+0000 7f42737fe700 1 -- 172.21.15.175:0/425207608 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f428c1692b0 con 0x7f428c08a100 2023-10-10T19:08:49.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.434+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f428c1692b0 con 0x7f428c08a100 2023-10-10T19:08:49.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.434+0000 7f42737fe700 1 -- 172.21.15.175:0/425207608 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5437 (secure 0 0 0) 0x7f427c017070 con 0x7f428c08a100 2023-10-10T19:08:49.435 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:49.435 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:47.697885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4278066f30 msgr2=0x7f42780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4278066f30 0x7f42780693e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f42800099b0 tx=0x7f4280017000 comp rx=0 tx=0).stop 2023-10-10T19:08:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 msgr2=0x7f428c169590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c169590 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c4e0 tx=0x7f427c00c7f0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 shutdown_connections 2023-10-10T19:08:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4278066f30 0x7f42780693e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f428c14fd90 0x7f428c14c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428c08a100 0x7f428c169590 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 --2- 172.21.15.175:0/425207608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c0cd040 0x7f428c169ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 >> 172.21.15.175:0/425207608 conn(0x7f428c063d00 msgr2=0x7f428c0cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 shutdown_connections 2023-10-10T19:08:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:49.436+0000 7f4290bab700 1 -- 172.21.15.175:0/425207608 wait complete. 2023-10-10T19:08:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 182 2023-10-10T19:08:49.449 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:51.450 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:51.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.577+0000 7f16486ce700 1 Processor -- start 2023-10-10T19:08:51.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.578+0000 7f16486ce700 1 -- start start 2023-10-10T19:08:51.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.578+0000 7f16486ce700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1640159d70 0x7f16400c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:51.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.578+0000 7f16486ce700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1640156800 con 0x7f164014fbf0 2023-10-10T19:08:51.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.578+0000 7f16486ce700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1640063e30 con 0x7f164014ffc0 2023-10-10T19:08:51.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.578+0000 7f16486ce700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16400c59e0 con 0x7f1640159d70 2023-10-10T19:08:51.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1646c6b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1640159d70 0x7f16400c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:51.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f164646a700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f164014fbf0 0x7f1640063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44226/0 (socket says 172.21.15.175:44226) 2023-10-10T19:08:51.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1646c6b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1640159d70 0x7f16400c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50484/0 (socket says 172.21.15.175:50484) 2023-10-10T19:08:51.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f164646a700 1 -- 172.21.15.175:0/2446079835 learned_addr learned my addr 172.21.15.175:0/2446079835 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:08:51.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1646c6b700 1 -- 172.21.15.175:0/2446079835 >> v1:172.21.15.191:6789/0 conn(0x7f164014ffc0 legacy=0x7f16400681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:51.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1646c6b700 1 -- 172.21.15.175:0/2446079835 >> v1:172.21.15.175:6789/0 conn(0x7f164014fbf0 legacy=0x7f1640063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:51.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1646c6b700 1 -- 172.21.15.175:0/2446079835 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640156c90 con 0x7f1640159d70 2023-10-10T19:08:51.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1645468700 1 -- 172.21.15.175:0/2446079835 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544288914 0 0) 0x7f1640156800 con 0x7f164014fbf0 2023-10-10T19:08:51.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1646c6b700 1 --2- 172.21.15.175:0/2446079835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f16400c5fb0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1638009e60 tx=0x7f1638009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80df6b4b94a673f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:51.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1645468700 1 -- 172.21.15.175:0/2446079835 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163800e070 con 0x7f1640159d70 2023-10-10T19:08:51.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1645468700 1 -- 172.21.15.175:0/2446079835 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163800bc30 con 0x7f1640159d70 2023-10-10T19:08:51.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.579+0000 7f1645468700 1 -- 172.21.15.175:0/2446079835 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638012430 con 0x7f1640159d70 2023-10-10T19:08:51.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.580+0000 7f16486ce700 1 -- 172.21.15.175:0/2446079835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 msgr2=0x7f16400c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:51.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.580+0000 7f16486ce700 1 --2- 172.21.15.175:0/2446079835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f16400c5fb0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1638009e60 tx=0x7f1638009780 comp rx=0 tx=0).stop 2023-10-10T19:08:51.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.580+0000 7f16486ce700 1 -- 172.21.15.175:0/2446079835 shutdown_connections 2023-10-10T19:08:51.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.580+0000 7f16486ce700 1 --2- 172.21.15.175:0/2446079835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f16400c5fb0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:51.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.580+0000 7f16486ce700 1 -- 172.21.15.175:0/2446079835 >> 172.21.15.175:0/2446079835 conn(0x7f16400cc760 msgr2=0x7f16400ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:51.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.580+0000 7f16486ce700 1 -- 172.21.15.175:0/2446079835 shutdown_connections 2023-10-10T19:08:51.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.580+0000 7f16486ce700 1 -- 172.21.15.175:0/2446079835 wait complete. 2023-10-10T19:08:51.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.581+0000 7f16486ce700 1 Processor -- start 2023-10-10T19:08:51.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.581+0000 7f16486ce700 1 -- start start 2023-10-10T19:08:51.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16486ce700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f164014fbf0 0x7f1640167790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:51.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16486ce700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f164014ffc0 0x7f1640167cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:51.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16486ce700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f164014c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:51.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16486ce700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1640167370 con 0x7f1640159d70 2023-10-10T19:08:51.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16486ce700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1640151300 con 0x7f164014fbf0 2023-10-10T19:08:51.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16486ce700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16401515e0 con 0x7f164014ffc0 2023-10-10T19:08:51.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f164014c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:51.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f164014c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50488/0 (socket says 172.21.15.175:50488) 2023-10-10T19:08:51.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 -- 172.21.15.175:0/1134494504 learned_addr learned my addr 172.21.15.175:0/1134494504 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:51.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f164646a700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f164014fbf0 0x7f1640167790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:51.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1645c69700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f164014ffc0 0x7f1640167cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:51.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 -- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f164014ffc0 msgr2=0x7f1640167cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:51.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f164014ffc0 0x7f1640167cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:51.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 -- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f164014fbf0 msgr2=0x7f1640167790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:51.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f164014fbf0 0x7f1640167790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:51.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 -- 172.21.15.175:0/1134494504 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1638009a20 con 0x7f1640159d70 2023-10-10T19:08:51.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f1646c6b700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f164014c920 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1638013000 tx=0x7f1638019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:51.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16367fc700 1 -- 172.21.15.175:0/1134494504 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163801b420 con 0x7f1640159d70 2023-10-10T19:08:51.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16367fc700 1 -- 172.21.15.175:0/1134494504 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163801b580 con 0x7f1640159d70 2023-10-10T19:08:51.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.582+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16401518c0 con 0x7f1640159d70 2023-10-10T19:08:51.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.583+0000 7f16367fc700 1 -- 172.21.15.175:0/1134494504 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163801b900 con 0x7f1640159d70 2023-10-10T19:08:51.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.583+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1640151e80 con 0x7f1640159d70 2023-10-10T19:08:51.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.583+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16401674b0 con 0x7f1640159d70 2023-10-10T19:08:51.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.584+0000 7f16367fc700 1 -- 172.21.15.175:0/1134494504 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f163801bc70 con 0x7f1640159d70 2023-10-10T19:08:51.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.584+0000 7f16367fc700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1624066f30 0x7f16240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:51.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.584+0000 7f16367fc700 1 -- 172.21.15.175:0/1134494504 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f1638086750 con 0x7f1640159d70 2023-10-10T19:08:51.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.585+0000 7f164646a700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1624066f30 0x7f16240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:51.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.585+0000 7f164646a700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1624066f30 0x7f16240693e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f163c001360 tx=0x7f163c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:51.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.589+0000 7f16367fc700 1 -- 172.21.15.175:0/1134494504 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16401674b0 con 0x7f1640159d70 2023-10-10T19:08:51.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.818+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16401674b0 con 0x7f1640159d70 2023-10-10T19:08:51.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.819+0000 7f16367fc700 1 -- 172.21.15.175:0/1134494504 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5437 (secure 0 0 0) 0x7f16401674b0 con 0x7f1640159d70 2023-10-10T19:08:51.820 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:51.820 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:47.697885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:51.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1624066f30 msgr2=0x7f16240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:51.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1624066f30 0x7f16240693e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f163c001360 tx=0x7f163c009040 comp rx=0 tx=0).stop 2023-10-10T19:08:51.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 msgr2=0x7f164014c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:51.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f164014c920 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1638013000 tx=0x7f1638019000 comp rx=0 tx=0).stop 2023-10-10T19:08:51.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 shutdown_connections 2023-10-10T19:08:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1624066f30 0x7f16240693e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1640159d70 0x7f164014c920 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f164014fbf0 0x7f1640167790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 --2- 172.21.15.175:0/1134494504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f164014ffc0 0x7f1640167cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 >> 172.21.15.175:0/1134494504 conn(0x7f16400cc760 msgr2=0x7f16400ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:51.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 shutdown_connections 2023-10-10T19:08:51.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:51.821+0000 7f16486ce700 1 -- 172.21.15.175:0/1134494504 wait complete. 2023-10-10T19:08:51.824 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 182 2023-10-10T19:08:51.833 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:52.037 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:08:52.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:08:53.834 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:53.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.956+0000 7f5a47eb2700 1 Processor -- start 2023-10-10T19:08:53.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.956+0000 7f5a47eb2700 1 -- start start 2023-10-10T19:08:53.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.957+0000 7f5a47eb2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a400b1fc0 0x7f5a40155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:53.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.957+0000 7f5a47eb2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a400b2390 0x7f5a40068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:53.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.957+0000 7f5a47eb2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a401574b0 con 0x7f5a40068bd0 2023-10-10T19:08:53.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.957+0000 7f5a47eb2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a40150770 con 0x7f5a400b2390 2023-10-10T19:08:53.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.957+0000 7f5a47eb2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400afad0 con 0x7f5a400b1fc0 2023-10-10T19:08:53.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.957+0000 7f5a4544d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a400b2390 0x7f5a40068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:53.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.957+0000 7f5a4544d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a400b2390 0x7f5a40068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50502/0 (socket says 172.21.15.175:50502) 2023-10-10T19:08:53.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.957+0000 7f5a4544d700 1 -- 172.21.15.175:0/1508556656 learned_addr learned my addr 172.21.15.175:0/1508556656 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:53.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a45c4e700 1 --2- 172.21.15.175:0/1508556656 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a400b1fc0 0x7f5a40155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:53.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a4544d700 1 -- 172.21.15.175:0/1508556656 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a400b1fc0 msgr2=0x7f5a40155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:53.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a4544d700 1 --2- 172.21.15.175:0/1508556656 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a400b1fc0 0x7f5a40155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:53.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a4544d700 1 -- 172.21.15.175:0/1508556656 >> v1:172.21.15.191:6789/0 conn(0x7f5a40068bd0 legacy=0x7f5a400aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:53.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a4544d700 1 -- 172.21.15.175:0/1508556656 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400af4c0 con 0x7f5a400b2390 2023-10-10T19:08:53.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a4544d700 1 --2- 172.21.15.175:0/1508556656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40068690 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5a300099b0 tx=0x7f5a30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82aa630775efc83e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:53.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a44c4c700 1 -- 172.21.15.175:0/1508556656 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3000b8f0 con 0x7f5a400b2390 2023-10-10T19:08:53.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a44c4c700 1 -- 172.21.15.175:0/1508556656 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3000ba50 con 0x7f5a400b2390 2023-10-10T19:08:53.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a44c4c700 1 -- 172.21.15.175:0/1508556656 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3000bdf0 con 0x7f5a400b2390 2023-10-10T19:08:53.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1508556656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 msgr2=0x7f5a40068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:53.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1508556656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40068690 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5a300099b0 tx=0x7f5a30009250 comp rx=0 tx=0).stop 2023-10-10T19:08:53.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1508556656 shutdown_connections 2023-10-10T19:08:53.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1508556656 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a400b1fc0 0x7f5a40155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:53.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1508556656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40068690 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:53.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.958+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1508556656 >> 172.21.15.175:0/1508556656 conn(0x7f5a40063230 msgr2=0x7f5a40063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:53.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.959+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1508556656 shutdown_connections 2023-10-10T19:08:53.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.959+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1508556656 wait complete. 2023-10-10T19:08:53.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a47eb2700 1 Processor -- start 2023-10-10T19:08:53.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a47eb2700 1 -- start start 2023-10-10T19:08:53.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a47eb2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a40068bd0 0x7f5a400c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:53.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a47eb2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400b1fc0 0x7f5a401699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:53.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a47eb2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40169f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:53.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a47eb2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400cb330 con 0x7f5a400b2390 2023-10-10T19:08:53.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a47eb2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400ca720 con 0x7f5a400b1fc0 2023-10-10T19:08:53.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a47eb2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400caa00 con 0x7f5a40068bd0 2023-10-10T19:08:53.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a4644f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40169f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:53.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a4644f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40169f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50506/0 (socket says 172.21.15.175:50506) 2023-10-10T19:08:53.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a4644f700 1 -- 172.21.15.175:0/1133586603 learned_addr learned my addr 172.21.15.175:0/1133586603 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:53.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a45c4e700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a40068bd0 0x7f5a400c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:53.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.960+0000 7f5a4544d700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400b1fc0 0x7f5a401699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:53.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a4644f700 1 -- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a40068bd0 msgr2=0x7f5a400c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:53.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a4644f700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a40068bd0 0x7f5a400c85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:53.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a4644f700 1 -- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400b1fc0 msgr2=0x7f5a401699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:53.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a4644f700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400b1fc0 0x7f5a401699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:53.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a4644f700 1 -- 172.21.15.175:0/1133586603 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300094f0 con 0x7f5a400b2390 2023-10-10T19:08:53.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a4644f700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40169f00 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800c910 tx=0x7f5a380016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:53.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a35ffb700 1 -- 172.21.15.175:0/1133586603 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38012630 con 0x7f5a400b2390 2023-10-10T19:08:53.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a35ffb700 1 -- 172.21.15.175:0/1133586603 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a380127b0 con 0x7f5a400b2390 2023-10-10T19:08:53.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a400cace0 con 0x7f5a400b2390 2023-10-10T19:08:53.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a35ffb700 1 -- 172.21.15.175:0/1133586603 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38012b70 con 0x7f5a400b2390 2023-10-10T19:08:53.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.961+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a400c6680 con 0x7f5a400b2390 2023-10-10T19:08:53.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.962+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40154c80 con 0x7f5a400b2390 2023-10-10T19:08:53.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.962+0000 7f5a35ffb700 1 -- 172.21.15.175:0/1133586603 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5a38003030 con 0x7f5a400b2390 2023-10-10T19:08:53.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.963+0000 7f5a35ffb700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a24066f60 0x7f5a24069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:53.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.963+0000 7f5a35ffb700 1 -- 172.21.15.175:0/1133586603 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f5a38084800 con 0x7f5a400b2390 2023-10-10T19:08:53.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.963+0000 7f5a45c4e700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a24066f60 0x7f5a24069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:53.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.964+0000 7f5a45c4e700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a24066f60 0x7f5a24069410 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00acb0 tx=0x7f5a3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:53.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:53.967+0000 7f5a35ffb700 1 -- 172.21.15.175:0/1133586603 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a380515f0 con 0x7f5a400b2390 2023-10-10T19:08:54.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.198+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a400cb180 con 0x7f5a400b2390 2023-10-10T19:08:54.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.199+0000 7f5a35ffb700 1 -- 172.21.15.175:0/1133586603 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5435 (secure 0 0 0) 0x7f5a38054c10 con 0x7f5a400b2390 2023-10-10T19:08:54.199 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:54.200 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:52.714166+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a24066f60 msgr2=0x7f5a24069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a24066f60 0x7f5a24069410 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00acb0 tx=0x7f5a3c008040 comp rx=0 tx=0).stop 2023-10-10T19:08:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 msgr2=0x7f5a40169f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40169f00 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800c910 tx=0x7f5a380016f0 comp rx=0 tx=0).stop 2023-10-10T19:08:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 shutdown_connections 2023-10-10T19:08:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a24066f60 0x7f5a24069410 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a400b2390 0x7f5a40169f00 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a40068bd0 0x7f5a400c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 --2- 172.21.15.175:0/1133586603 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a400b1fc0 0x7f5a401699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 >> 172.21.15.175:0/1133586603 conn(0x7f5a40063230 msgr2=0x7f5a4014fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 shutdown_connections 2023-10-10T19:08:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:54.201+0000 7f5a47eb2700 1 -- 172.21.15.175:0/1133586603 wait complete. 2023-10-10T19:08:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 183 2023-10-10T19:08:54.214 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 183, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 183, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5584": { "addr": "172.21.15.191:6837/824552134", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 824552134, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 824552134, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 5584, "incarnation": 100, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 35 }, "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:rejoin", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:08:52.714166+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 6106, "mds_3": 6618, "mds_4": 5584 } } } ], "standbys": [] } 2023-10-10T19:08:56.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:56.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.342+0000 7ffa1bb78700 1 Processor -- start 2023-10-10T19:08:56.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.342+0000 7ffa1bb78700 1 -- start start 2023-10-10T19:08:56.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa1bb78700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 0x7ffa14159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:56.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa1bb78700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffa1414ffc0 0x7ffa14063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:56.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa1bb78700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa14156800 con 0x7ffa140681e0 2023-10-10T19:08:56.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa1bb78700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa140c59e0 con 0x7ffa1414ffc0 2023-10-10T19:08:56.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa1bb78700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1415a2b0 con 0x7ffa1414fbf0 2023-10-10T19:08:56.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19113700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffa1414ffc0 0x7ffa14063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:56.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19113700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffa1414ffc0 0x7ffa14063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50640/0 (socket says 172.21.15.175:50640) 2023-10-10T19:08:56.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19113700 1 -- 172.21.15.175:0/3986317358 learned_addr learned my addr 172.21.15.175:0/3986317358 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:56.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19914700 1 --2- 172.21.15.175:0/3986317358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 0x7ffa14159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:56.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19113700 1 -- 172.21.15.175:0/3986317358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 msgr2=0x7ffa14159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:56.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19113700 1 --2- 172.21.15.175:0/3986317358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 0x7ffa14159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19113700 1 -- 172.21.15.175:0/3986317358 >> v1:172.21.15.191:6789/0 conn(0x7ffa140681e0 legacy=0x7ffa140c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:56.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19113700 1 -- 172.21.15.175:0/3986317358 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa14063e30 con 0x7ffa1414ffc0 2023-10-10T19:08:56.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa19113700 1 --2- 172.21.15.175:0/3986317358 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14063230 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ffa040099e0 tx=0x7ffa04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c34f92f22d56ff2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:56.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.343+0000 7ffa18912700 1 -- 172.21.15.175:0/3986317358 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0400b820 con 0x7ffa1414ffc0 2023-10-10T19:08:56.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa18912700 1 -- 172.21.15.175:0/3986317358 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0400b980 con 0x7ffa1414ffc0 2023-10-10T19:08:56.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa18912700 1 -- 172.21.15.175:0/3986317358 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0400bd20 con 0x7ffa1414ffc0 2023-10-10T19:08:56.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa1bb78700 1 -- 172.21.15.175:0/3986317358 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 msgr2=0x7ffa14063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:56.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/3986317358 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14063230 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ffa040099e0 tx=0x7ffa04009250 comp rx=0 tx=0).stop 2023-10-10T19:08:56.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa1bb78700 1 -- 172.21.15.175:0/3986317358 shutdown_connections 2023-10-10T19:08:56.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/3986317358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 0x7ffa14159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/3986317358 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14063230 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa1bb78700 1 -- 172.21.15.175:0/3986317358 >> 172.21.15.175:0/3986317358 conn(0x7ffa140cc760 msgr2=0x7ffa140ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:56.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa1bb78700 1 -- 172.21.15.175:0/3986317358 shutdown_connections 2023-10-10T19:08:56.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.344+0000 7ffa1bb78700 1 -- 172.21.15.175:0/3986317358 wait complete. 2023-10-10T19:08:56.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.345+0000 7ffa1bb78700 1 Processor -- start 2023-10-10T19:08:56.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.345+0000 7ffa1bb78700 1 -- start start 2023-10-10T19:08:56.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1bb78700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa140681e0 0x7ffa14167210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:56.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1bb78700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 0x7ffa140c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:56.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1bb78700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:56.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1bb78700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14151620 con 0x7ffa1414ffc0 2023-10-10T19:08:56.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1bb78700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14151ec0 con 0x7ffa140681e0 2023-10-10T19:08:56.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1bb78700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa141668b0 con 0x7ffa1414fbf0 2023-10-10T19:08:56.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1a115700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:56.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1a115700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14169310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50654/0 (socket says 172.21.15.175:50654) 2023-10-10T19:08:56.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1a115700 1 -- 172.21.15.175:0/4030032918 learned_addr learned my addr 172.21.15.175:0/4030032918 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:56.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa19914700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa140681e0 0x7ffa14167210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:56.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa19113700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 0x7ffa140c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:56.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1a115700 1 -- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 msgr2=0x7ffa140c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:56.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1a115700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 0x7ffa140c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1a115700 1 -- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa140681e0 msgr2=0x7ffa14167210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:56.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1a115700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa140681e0 0x7ffa14167210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.346+0000 7ffa1a115700 1 -- 172.21.15.175:0/4030032918 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa040094f0 con 0x7ffa1414ffc0 2023-10-10T19:08:56.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.347+0000 7ffa1a115700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14169310 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00c8e0 tx=0x7ffa0c001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:56.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.347+0000 7ffa09ffb700 1 -- 172.21.15.175:0/4030032918 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c012630 con 0x7ffa1414ffc0 2023-10-10T19:08:56.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.347+0000 7ffa09ffb700 1 -- 172.21.15.175:0/4030032918 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c012790 con 0x7ffa1414ffc0 2023-10-10T19:08:56.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.347+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa14166b50 con 0x7ffa1414ffc0 2023-10-10T19:08:56.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.347+0000 7ffa09ffb700 1 -- 172.21.15.175:0/4030032918 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c012b50 con 0x7ffa1414ffc0 2023-10-10T19:08:56.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.347+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa14151b50 con 0x7ffa1414ffc0 2023-10-10T19:08:56.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.348+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa14152030 con 0x7ffa1414ffc0 2023-10-10T19:08:56.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.348+0000 7ffa09ffb700 1 -- 172.21.15.175:0/4030032918 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffa0c01a3a0 con 0x7ffa1414ffc0 2023-10-10T19:08:56.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.348+0000 7ffa09ffb700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9f8066f60 0x7ff9f8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:56.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.348+0000 7ffa09ffb700 1 -- 172.21.15.175:0/4030032918 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7ffa0c084b20 con 0x7ffa1414ffc0 2023-10-10T19:08:56.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.349+0000 7ffa19914700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9f8066f60 0x7ff9f8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:56.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.349+0000 7ffa19914700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9f8066f60 0x7ff9f8069410 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffa10006aa0 tx=0x7ffa10008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:56.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.352+0000 7ffa09ffb700 1 -- 172.21.15.175:0/4030032918 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffa0c051910 con 0x7ffa1414ffc0 2023-10-10T19:08:56.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.574+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa14067570 con 0x7ffa1414ffc0 2023-10-10T19:08:56.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.575+0000 7ffa09ffb700 1 -- 172.21.15.175:0/4030032918 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+5434 (secure 0 0 0) 0x7ffa0c054f30 con 0x7ffa1414ffc0 2023-10-10T19:08:56.575 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:56.576 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:54.661699+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:56.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9f8066f60 msgr2=0x7ff9f8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:56.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9f8066f60 0x7ff9f8069410 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffa10006aa0 tx=0x7ffa10008040 comp rx=0 tx=0).stop 2023-10-10T19:08:56.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 msgr2=0x7ffa14169310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:56.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14169310 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00c8e0 tx=0x7ffa0c001710 comp rx=0 tx=0).stop 2023-10-10T19:08:56.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 shutdown_connections 2023-10-10T19:08:56.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff9f8066f60 0x7ff9f8069410 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa1414ffc0 0x7ffa14169310 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa140681e0 0x7ffa14167210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 --2- 172.21.15.175:0/4030032918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa1414fbf0 0x7ffa140c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 >> 172.21.15.175:0/4030032918 conn(0x7ffa140cc760 msgr2=0x7ffa14155640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 shutdown_connections 2023-10-10T19:08:56.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:56.577+0000 7ffa1bb78700 1 -- 172.21.15.175:0/4030032918 wait complete. 2023-10-10T19:08:56.580 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 184 2023-10-10T19:08:56.589 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:08:58.590 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:08:58.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.717+0000 7f3a2a7f3700 1 Processor -- start 2023-10-10T19:08:58.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.717+0000 7f3a2a7f3700 1 -- start start 2023-10-10T19:08:58.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a2a7f3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3a240b1fc0 0x7f3a24155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:58.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a2a7f3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3a240b2390 0x7f3a24068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:58.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a2a7f3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a24068bd0 0x7f3a240aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:58.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a2a7f3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a24150770 con 0x7f3a240b1fc0 2023-10-10T19:08:58.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a2a7f3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a241574b0 con 0x7f3a240b2390 2023-10-10T19:08:58.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a2a7f3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240afad0 con 0x7f3a24068bd0 2023-10-10T19:08:58.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a23fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3a240b1fc0 0x7f3a24155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:58.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a23fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3a240b1fc0 0x7f3a24155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50658/0 (socket says 172.21.15.175:50658) 2023-10-10T19:08:58.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a23fff700 1 -- 172.21.15.175:0/3370238809 learned_addr learned my addr 172.21.15.175:0/3370238809 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:58.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a237fe700 1 --2- 172.21.15.175:0/3370238809 >> v2:172.21.15.191:3300/0 conn(0x7f3a240b2390 0x7f3a24068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:58.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.718+0000 7f3a28d90700 1 --2- 172.21.15.175:0/3370238809 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a24068bd0 0x7f3a240aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:58.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a23fff700 1 -- 172.21.15.175:0/3370238809 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a24068bd0 msgr2=0x7f3a240aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:58.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a23fff700 1 --2- 172.21.15.175:0/3370238809 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a24068bd0 0x7f3a240aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a23fff700 1 -- 172.21.15.175:0/3370238809 >> v2:172.21.15.191:3300/0 conn(0x7f3a240b2390 msgr2=0x7f3a24068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:08:58.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a23fff700 1 --2- 172.21.15.175:0/3370238809 >> v2:172.21.15.191:3300/0 conn(0x7f3a240b2390 0x7f3a24068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a23fff700 1 -- 172.21.15.175:0/3370238809 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a240af4c0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a237fe700 1 --2- 172.21.15.175:0/3370238809 >> v2:172.21.15.191:3300/0 conn(0x7f3a240b2390 0x7f3a24068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:08:58.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a23fff700 1 --2- 172.21.15.175:0/3370238809 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24155fb0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3a240c0290 tx=0x7f3a14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940f8fb0c3097f95 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:58.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a22ffd700 1 -- 172.21.15.175:0/3370238809 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1400b8f0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a22ffd700 1 -- 172.21.15.175:0/3370238809 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1400ba50 con 0x7f3a240b1fc0 2023-10-10T19:08:58.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a22ffd700 1 -- 172.21.15.175:0/3370238809 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1400bdf0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/3370238809 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 msgr2=0x7f3a24155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:58.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/3370238809 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24155fb0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3a240c0290 tx=0x7f3a14009250 comp rx=0 tx=0).stop 2023-10-10T19:08:58.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/3370238809 shutdown_connections 2023-10-10T19:08:58.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/3370238809 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a24068bd0 0x7f3a240aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/3370238809 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24155fb0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/3370238809 >> v2:172.21.15.191:3300/0 conn(0x7f3a240b2390 0x7f3a24068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/3370238809 >> 172.21.15.175:0/3370238809 conn(0x7f3a24063230 msgr2=0x7f3a24063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:58.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/3370238809 shutdown_connections 2023-10-10T19:08:58.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.719+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/3370238809 wait complete. 2023-10-10T19:08:58.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.720+0000 7f3a2a7f3700 1 Processor -- start 2023-10-10T19:08:58.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a2a7f3700 1 -- start start 2023-10-10T19:08:58.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a2a7f3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a24068bd0 0x7f3a240c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:58.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a2a7f3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:58.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a2a7f3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240b2390 0x7f3a24154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:58.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a2a7f3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a24169dd0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a2a7f3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a24165da0 con 0x7f3a24068bd0 2023-10-10T19:08:58.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a2a7f3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240af010 con 0x7f3a240b2390 2023-10-10T19:08:58.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a237fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:58.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a23fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a24068bd0 0x7f3a240c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:58.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a237fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50672/0 (socket says 172.21.15.175:50672) 2023-10-10T19:08:58.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a237fe700 1 -- 172.21.15.175:0/1368890705 learned_addr learned my addr 172.21.15.175:0/1368890705 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:08:58.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.721+0000 7f3a28d90700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240b2390 0x7f3a24154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:58.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a237fe700 1 -- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240b2390 msgr2=0x7f3a24154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:58.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a237fe700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240b2390 0x7f3a24154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a237fe700 1 -- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a24068bd0 msgr2=0x7f3a240c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:58.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a237fe700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a24068bd0 0x7f3a240c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a237fe700 1 -- 172.21.15.175:0/1368890705 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a140094f0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a237fe700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24157d80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800d860 tx=0x7f3a18008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:58.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a0bfff700 1 -- 172.21.15.175:0/1368890705 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a180125f0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a0bfff700 1 -- 172.21.15.175:0/1368890705 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a18012750 con 0x7f3a240b1fc0 2023-10-10T19:08:58.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a0bfff700 1 -- 172.21.15.175:0/1368890705 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18012af0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2416a040 con 0x7f3a240b1fc0 2023-10-10T19:08:58.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.722+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a24166620 con 0x7f3a240b1fc0 2023-10-10T19:08:58.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.723+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a24067570 con 0x7f3a240b1fc0 2023-10-10T19:08:58.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.724+0000 7f3a0bfff700 1 -- 172.21.15.175:0/1368890705 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3a18013480 con 0x7f3a240b1fc0 2023-10-10T19:08:58.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.724+0000 7f3a0bfff700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:08:58.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.724+0000 7f3a0bfff700 1 -- 172.21.15.175:0/1368890705 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f3a18084ab0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.724+0000 7f3a23fff700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:08:58.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.725+0000 7f3a23fff700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3a240c0290 tx=0x7f3a14017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:08:58.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.728+0000 7f3a0bfff700 1 -- 172.21.15.175:0/1368890705 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a180518a0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.957+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a24068470 con 0x7f3a240b1fc0 2023-10-10T19:08:58.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.957+0000 7f3a0bfff700 1 -- 172.21.15.175:0/1368890705 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f3a18054ec0 con 0x7f3a240b1fc0 2023-10-10T19:08:58.958 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:08:58.958 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:57.727835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:clientreplay","state_seq":18,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:08:58.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 msgr2=0x7f3a100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:58.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3a240c0290 tx=0x7f3a14017000 comp rx=0 tx=0).stop 2023-10-10T19:08:58.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 msgr2=0x7f3a24157d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:08:58.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24157d80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800d860 tx=0x7f3a18008040 comp rx=0 tx=0).stop 2023-10-10T19:08:58.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 shutdown_connections 2023-10-10T19:08:58.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240b2390 0x7f3a24154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a24068bd0 0x7f3a240c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 --2- 172.21.15.175:0/1368890705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a240b1fc0 0x7f3a24157d80 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:08:58.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.959+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 >> 172.21.15.175:0/1368890705 conn(0x7f3a24063230 msgr2=0x7f3a2414fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:08:58.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.960+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 shutdown_connections 2023-10-10T19:08:58.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:08:58.960+0000 7f3a2a7f3700 1 -- 172.21.15.175:0/1368890705 wait complete. 2023-10-10T19:08:58.963 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 185 2023-10-10T19:08:58.972 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:09:00.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:01.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.099+0000 7f76debca700 1 Processor -- start 2023-10-10T19:09:01.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.099+0000 7f76debca700 1 -- start start 2023-10-10T19:09:01.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.099+0000 7f76debca700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f76d80b1fc0 0x7f76d8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:01.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.099+0000 7f76debca700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80aff60 0x7f76d80b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:01.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.099+0000 7f76debca700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d81574b0 con 0x7f76d80b2390 2023-10-10T19:09:01.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.099+0000 7f76debca700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d8150770 con 0x7f76d80b1fc0 2023-10-10T19:09:01.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.099+0000 7f76debca700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80afad0 con 0x7f76d80aff60 2023-10-10T19:09:01.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76dc966700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f76d80b1fc0 0x7f76d8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:01.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76cffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f76d80b2390 0x7f76d8068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58244/0 (socket says 172.21.15.175:58244) 2023-10-10T19:09:01.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76cffff700 1 -- 172.21.15.175:0/3150033181 learned_addr learned my addr 172.21.15.175:0/3150033181 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:09:01.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76dd167700 1 --2- 172.21.15.175:0/3150033181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80aff60 0x7f76d80b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:01.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76dc966700 1 -- 172.21.15.175:0/3150033181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80aff60 msgr2=0x7f76d80b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:01.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76dc966700 1 --2- 172.21.15.175:0/3150033181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80aff60 0x7f76d80b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76dc966700 1 -- 172.21.15.175:0/3150033181 >> v1:172.21.15.175:6789/0 conn(0x7f76d80b2390 legacy=0x7f76d8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:01.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76dc966700 1 -- 172.21.15.175:0/3150033181 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d80af4c0 con 0x7f76d80b1fc0 2023-10-10T19:09:01.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76cf7fe700 1 -- 172.21.15.175:0/3150033181 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986634972 0 0) 0x7f76d81574b0 con 0x7f76d80b2390 2023-10-10T19:09:01.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76dd167700 1 --2- 172.21.15.175:0/3150033181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80aff60 0x7f76d80b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:09:01.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76dc966700 1 --2- 172.21.15.175:0/3150033181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d8155fb0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009e60 tx=0x7f76d4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bca4f214efacb68 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:01.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76cf7fe700 1 -- 172.21.15.175:0/3150033181 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d400e070 con 0x7f76d80b1fc0 2023-10-10T19:09:01.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76cf7fe700 1 -- 172.21.15.175:0/3150033181 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d400bcc0 con 0x7f76d80b1fc0 2023-10-10T19:09:01.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.100+0000 7f76cf7fe700 1 -- 172.21.15.175:0/3150033181 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4012680 con 0x7f76d80b1fc0 2023-10-10T19:09:01.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.101+0000 7f76debca700 1 -- 172.21.15.175:0/3150033181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 msgr2=0x7f76d8155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:01.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.101+0000 7f76debca700 1 --2- 172.21.15.175:0/3150033181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d8155fb0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009e60 tx=0x7f76d4009780 comp rx=0 tx=0).stop 2023-10-10T19:09:01.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.101+0000 7f76debca700 1 -- 172.21.15.175:0/3150033181 shutdown_connections 2023-10-10T19:09:01.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.101+0000 7f76debca700 1 --2- 172.21.15.175:0/3150033181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80aff60 0x7f76d80b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.101+0000 7f76debca700 1 --2- 172.21.15.175:0/3150033181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d8155fb0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.101+0000 7f76debca700 1 -- 172.21.15.175:0/3150033181 >> 172.21.15.175:0/3150033181 conn(0x7f76d8063230 msgr2=0x7f76d8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:01.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.101+0000 7f76debca700 1 -- 172.21.15.175:0/3150033181 shutdown_connections 2023-10-10T19:09:01.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.101+0000 7f76debca700 1 -- 172.21.15.175:0/3150033181 wait complete. 2023-10-10T19:09:01.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.102+0000 7f76debca700 1 Processor -- start 2023-10-10T19:09:01.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.102+0000 7f76debca700 1 -- start start 2023-10-10T19:09:01.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76debca700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76d80aff60 0x7f76d8165c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:01.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76debca700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d81598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:01.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76debca700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80b2390 0x7f76d814db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:01.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76debca700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80ccf40 con 0x7f76d80b1fc0 2023-10-10T19:09:01.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76debca700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80ca8e0 con 0x7f76d80aff60 2023-10-10T19:09:01.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76debca700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80cbe80 con 0x7f76d80b2390 2023-10-10T19:09:01.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d81598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:01.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d81598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50690/0 (socket says 172.21.15.175:50690) 2023-10-10T19:09:01.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 -- 172.21.15.175:0/1583979671 learned_addr learned my addr 172.21.15.175:0/1583979671 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:01.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76dc966700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76d80aff60 0x7f76d8165c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:01.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76dd167700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80b2390 0x7f76d814db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:01.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 -- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80b2390 msgr2=0x7f76d814db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:01.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80b2390 0x7f76d814db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 -- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76d80aff60 msgr2=0x7f76d8165c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:01.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76d80aff60 0x7f76d8165c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 -- 172.21.15.175:0/1583979671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4009a20 con 0x7f76d80b1fc0 2023-10-10T19:09:01.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76dd167700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80b2390 0x7f76d814db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:09:01.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76dc966700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76d80aff60 0x7f76d8165c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:09:01.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76cffff700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d81598b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c560 tx=0x7f76c8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:01.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76ccff9700 1 -- 172.21.15.175:0/1583979671 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8005b30 con 0x7f76d80b1fc0 2023-10-10T19:09:01.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.103+0000 7f76ccff9700 1 -- 172.21.15.175:0/1583979671 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c8005cd0 con 0x7f76d80b1fc0 2023-10-10T19:09:01.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.104+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d80cc140 con 0x7f76d80b1fc0 2023-10-10T19:09:01.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.104+0000 7f76ccff9700 1 -- 172.21.15.175:0/1583979671 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8007320 con 0x7f76d80b1fc0 2023-10-10T19:09:01.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.104+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d80cc700 con 0x7f76d80b1fc0 2023-10-10T19:09:01.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.104+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d80caa50 con 0x7f76d80b1fc0 2023-10-10T19:09:01.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.105+0000 7f76ccff9700 1 -- 172.21.15.175:0/1583979671 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f76c8012400 con 0x7f76d80b1fc0 2023-10-10T19:09:01.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.105+0000 7f76ccff9700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f76c4066f30 0x7f76c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:01.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.105+0000 7f76ccff9700 1 -- 172.21.15.175:0/1583979671 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f76c8083ef0 con 0x7f76d80b1fc0 2023-10-10T19:09:01.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.105+0000 7f76dc966700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f76c4066f30 0x7f76c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:01.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.106+0000 7f76dc966700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f76c4066f30 0x7f76c40693e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f76d40097c0 tx=0x7f76d4019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:01.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.109+0000 7f76ccff9700 1 -- 172.21.15.175:0/1583979671 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f76c804b5a0 con 0x7f76d80b1fc0 2023-10-10T19:09:01.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.335+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d8067570 con 0x7f76d80b1fc0 2023-10-10T19:09:01.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.335+0000 7f76ccff9700 1 -- 172.21.15.175:0/1583979671 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f76c8017090 con 0x7f76d80b1fc0 2023-10-10T19:09:01.336 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:01.336 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:57.727835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:clientreplay","state_seq":18,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:01.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f76c4066f30 msgr2=0x7f76c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:01.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f76c4066f30 0x7f76c40693e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f76d40097c0 tx=0x7f76d4019000 comp rx=0 tx=0).stop 2023-10-10T19:09:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 msgr2=0x7f76d81598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d81598b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c560 tx=0x7f76c8000fe0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 shutdown_connections 2023-10-10T19:09:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f76c4066f30 0x7f76c40693e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76d80b2390 0x7f76d814db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76d80aff60 0x7f76d8165c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 --2- 172.21.15.175:0/1583979671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f76d80b1fc0 0x7f76d81598b0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:01.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 >> 172.21.15.175:0/1583979671 conn(0x7f76d8063230 msgr2=0x7f76d8154430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:01.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 shutdown_connections 2023-10-10T19:09:01.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:01.338+0000 7f76debca700 1 -- 172.21.15.175:0/1583979671 wait complete. 2023-10-10T19:09:01.341 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 185 2023-10-10T19:09:01.350 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:09:03.351 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:03.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.467+0000 7fd42ee5f700 1 Processor -- start 2023-10-10T19:09:03.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.468+0000 7fd42ee5f700 1 -- start start 2023-10-10T19:09:03.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.468+0000 7fd42ee5f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd4280c0f70 0x7fd42806d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:03.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.468+0000 7fd42ee5f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42806d8a0 con 0x7fd42806cf90 2023-10-10T19:09:03.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.468+0000 7fd42ee5f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4281595e0 con 0x7fd4281569a0 2023-10-10T19:09:03.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.468+0000 7fd42ee5f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4280c0990 con 0x7fd4280c0f70 2023-10-10T19:09:03.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.468+0000 7fd42de5d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd42806cf90 0x7fd428156290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58254/0 (socket says 172.21.15.175:58254) 2023-10-10T19:09:03.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42de5d700 1 -- 172.21.15.175:0/1301415801 learned_addr learned my addr 172.21.15.175:0/1301415801 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:09:03.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42e65e700 1 --2- 172.21.15.175:0/1301415801 >> v2:172.21.15.191:3300/0 conn(0x7fd4280c0f70 0x7fd42806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:03.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798240228 0 0) 0x7fd42806d8a0 con 0x7fd42806cf90 2023-10-10T19:09:03.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd414003570 con 0x7fd42806cf90 2023-10-10T19:09:03.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3480692945 0 0) 0x7fd414003570 con 0x7fd42806cf90 2023-10-10T19:09:03.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 >> v2:172.21.15.191:3300/0 conn(0x7fd4280c0f70 msgr2=0x7fd42806d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:03.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 --2- 172.21.15.175:0/1301415801 >> v2:172.21.15.191:3300/0 conn(0x7fd4280c0f70 0x7fd42806d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 >> v1:172.21.15.191:6789/0 conn(0x7fd4281569a0 legacy=0x7fd42815e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:03.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42815a440 con 0x7fd42806cf90 2023-10-10T19:09:03.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fd4240015f0 con 0x7fd42806cf90 2023-10-10T19:09:03.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 >> v1:172.21.15.175:6789/0 conn(0x7fd42806cf90 legacy=0x7fd428156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:03.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd414003a60 0x7fd414003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:03.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414004810 0x7fd414004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:03.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4140051c0 0x7fd41400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:03.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41400c0a0 con 0x7fd414003a60 2023-10-10T19:09:03.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41400c210 con 0x7fd4140051c0 2023-10-10T19:09:03.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41400c3d0 con 0x7fd414004810 2023-10-10T19:09:03.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42de5d700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd414003a60 0x7fd414003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:03.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42e65e700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4140051c0 0x7fd41400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:03.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.469+0000 7fd42d65c700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414004810 0x7fd414004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:03.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42de5d700 1 -- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414004810 msgr2=0x7fd414004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:03.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42de5d700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414004810 0x7fd414004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42de5d700 1 -- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4140051c0 msgr2=0x7fd41400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:03.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42de5d700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4140051c0 0x7fd41400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42de5d700 1 -- 172.21.15.175:0/1301415801 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd414003570 con 0x7fd414003a60 2023-10-10T19:09:03.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42d65c700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414004810 0x7fd414004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:09:03.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42e65e700 1 -- 172.21.15.175:0/1301415801 reap_dead start 2023-10-10T19:09:03.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42de5d700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd414003a60 0x7fd414003e70 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd42400c3c0 tx=0x7fd424007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0517b6309b50bd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:03.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42400cc90 con 0x7fd414003a60 2023-10-10T19:09:03.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42400cdf0 con 0x7fd414003a60 2023-10-10T19:09:03.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ce5b700 1 -- 172.21.15.175:0/1301415801 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424007320 con 0x7fd414003a60 2023-10-10T19:09:03.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ee5f700 1 -- 172.21.15.175:0/1301415801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd414003a60 msgr2=0x7fd414003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:03.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ee5f700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd414003a60 0x7fd414003e70 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd42400c3c0 tx=0x7fd424007920 comp rx=0 tx=0).stop 2023-10-10T19:09:03.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ee5f700 1 -- 172.21.15.175:0/1301415801 shutdown_connections 2023-10-10T19:09:03.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ee5f700 1 --2- 172.21.15.175:0/1301415801 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd414003a60 0x7fd414003e70 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ee5f700 1 -- 172.21.15.175:0/1301415801 >> 172.21.15.175:0/1301415801 conn(0x7fd42806b310 msgr2=0x7fd42806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:03.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.470+0000 7fd42ee5f700 1 -- 172.21.15.175:0/1301415801 shutdown_connections 2023-10-10T19:09:03.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.471+0000 7fd42ee5f700 1 -- 172.21.15.175:0/1301415801 wait complete. 2023-10-10T19:09:03.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.471+0000 7fd42ee5f700 1 Processor -- start 2023-10-10T19:09:03.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42ee5f700 1 -- start start 2023-10-10T19:09:03.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42ee5f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414003a60 0x7fd428157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:03.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42ee5f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd428158410 0x7fd42814ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:03.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42ee5f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4281695b0 0x7fd428169a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:03.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42ee5f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4280699d0 con 0x7fd4281695b0 2023-10-10T19:09:03.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42ee5f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd428069360 con 0x7fd428158410 2023-10-10T19:09:03.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42ee5f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd428068ed0 con 0x7fd414003a60 2023-10-10T19:09:03.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42e65e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4281695b0 0x7fd428169a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:03.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42e65e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4281695b0 0x7fd428169a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50698/0 (socket says 172.21.15.175:50698) 2023-10-10T19:09:03.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42e65e700 1 -- 172.21.15.175:0/2867051716 learned_addr learned my addr 172.21.15.175:0/2867051716 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:03.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42de5d700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414003a60 0x7fd428157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:03.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.472+0000 7fd42d65c700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd428158410 0x7fd42814ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:03.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd42e65e700 1 -- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414003a60 msgr2=0x7fd428157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:03.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd42e65e700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414003a60 0x7fd428157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd42e65e700 1 -- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd428158410 msgr2=0x7fd42814ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:03.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd42e65e700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd428158410 0x7fd42814ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd42e65e700 1 -- 172.21.15.175:0/2867051716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4280688f0 con 0x7fd4281695b0 2023-10-10T19:09:03.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd42e65e700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4281695b0 0x7fd428169a20 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd4200094d0 tx=0x7fd42000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:03.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd41dffb700 1 -- 172.21.15.175:0/2867051716 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42000e040 con 0x7fd4281695b0 2023-10-10T19:09:03.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd41dffb700 1 -- 172.21.15.175:0/2867051716 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42000f040 con 0x7fd4281695b0 2023-10-10T19:09:03.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4281555b0 con 0x7fd4281695b0 2023-10-10T19:09:03.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd41dffb700 1 -- 172.21.15.175:0/2867051716 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42000fda0 con 0x7fd4281695b0 2023-10-10T19:09:03.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.473+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd428155b30 con 0x7fd4281695b0 2023-10-10T19:09:03.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.474+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd428069080 con 0x7fd4281695b0 2023-10-10T19:09:03.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.474+0000 7fd41dffb700 1 -- 172.21.15.175:0/2867051716 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd420015640 con 0x7fd4281695b0 2023-10-10T19:09:03.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.475+0000 7fd41dffb700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd40c066f60 0x7fd40c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:03.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.475+0000 7fd41dffb700 1 -- 172.21.15.175:0/2867051716 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fd420085090 con 0x7fd4281695b0 2023-10-10T19:09:03.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.475+0000 7fd42de5d700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd40c066f60 0x7fd40c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:03.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.475+0000 7fd42de5d700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd40c066f60 0x7fd40c069410 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd424000ef0 tx=0x7fd424015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:03.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.479+0000 7fd41dffb700 1 -- 172.21.15.175:0/2867051716 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd428069080 con 0x7fd4281695b0 2023-10-10T19:09:03.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.716+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd428069080 con 0x7fd4281695b0 2023-10-10T19:09:03.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.717+0000 7fd41dffb700 1 -- 172.21.15.175:0/2867051716 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5440 (secure 0 0 0) 0x7fd428069080 con 0x7fd4281695b0 2023-10-10T19:09:03.718 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:03.718 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:08:57.727835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:clientreplay","state_seq":18,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:03.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd40c066f60 msgr2=0x7fd40c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:03.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd40c066f60 0x7fd40c069410 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd424000ef0 tx=0x7fd424015000 comp rx=0 tx=0).stop 2023-10-10T19:09:03.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4281695b0 msgr2=0x7fd428169a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4281695b0 0x7fd428169a20 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd4200094d0 tx=0x7fd42000b040 comp rx=0 tx=0).stop 2023-10-10T19:09:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 shutdown_connections 2023-10-10T19:09:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd40c066f60 0x7fd40c069410 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4281695b0 0x7fd428169a20 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd414003a60 0x7fd428157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 --2- 172.21.15.175:0/2867051716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd428158410 0x7fd42814ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:03.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 >> 172.21.15.175:0/2867051716 conn(0x7fd42806b310 msgr2=0x7fd42806b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:03.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 shutdown_connections 2023-10-10T19:09:03.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:03.719+0000 7fd42ee5f700 1 -- 172.21.15.175:0/2867051716 wait complete. 2023-10-10T19:09:03.723 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 185 2023-10-10T19:09:03.734 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:09:04.763 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:09:04.763+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T19:09:05.734 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:05.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.864+0000 7f05ae918700 1 Processor -- start 2023-10-10T19:09:05.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.864+0000 7f05ae918700 1 -- start start 2023-10-10T19:09:05.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.864+0000 7f05ae918700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 0x7f05a8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:05.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.864+0000 7f05ae918700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05a80b2390 0x7f05a8068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:05.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.864+0000 7f05ae918700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a81574b0 con 0x7f05a8068bd0 2023-10-10T19:09:05.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.864+0000 7f05ae918700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a8150770 con 0x7f05a80b2390 2023-10-10T19:09:05.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.864+0000 7f05ae918700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80afad0 con 0x7f05a80b1fc0 2023-10-10T19:09:05.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.864+0000 7f05a77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05a80b2390 0x7f05a8068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:05.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f05a80b2390 0x7f05a8068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39842/0 (socket says 172.21.15.175:39842) 2023-10-10T19:09:05.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05aceb5700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f05a8068bd0 0x7f05a80aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39944/0 (socket says 172.21.15.175:39944) 2023-10-10T19:09:05.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a77fe700 1 -- 172.21.15.175:0/3596530180 learned_addr learned my addr 172.21.15.175:0/3596530180 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:05.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a7fff700 1 --2- 172.21.15.175:0/3596530180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 0x7f05a8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:05.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a77fe700 1 -- 172.21.15.175:0/3596530180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 msgr2=0x7f05a8155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:05.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a77fe700 1 --2- 172.21.15.175:0/3596530180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 0x7f05a8155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:05.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a77fe700 1 -- 172.21.15.175:0/3596530180 >> v1:172.21.15.175:6789/0 conn(0x7f05a8068bd0 legacy=0x7f05a80aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:05.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a6ffd700 1 -- 172.21.15.175:0/3596530180 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776731001 0 0) 0x7f05a81574b0 con 0x7f05a8068bd0 2023-10-10T19:09:05.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a77fe700 1 -- 172.21.15.175:0/3596530180 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a80af4c0 con 0x7f05a80b2390 2023-10-10T19:09:05.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a77fe700 1 --2- 172.21.15.175:0/3596530180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a80b2390 0x7f05a8068690 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f059c0099b0 tx=0x7f059c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c47bcaaa266cc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:05.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a6ffd700 1 -- 172.21.15.175:0/3596530180 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c00b8f0 con 0x7f05a80b2390 2023-10-10T19:09:05.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a6ffd700 1 -- 172.21.15.175:0/3596530180 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c00ba50 con 0x7f05a80b2390 2023-10-10T19:09:05.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.865+0000 7f05a6ffd700 1 -- 172.21.15.175:0/3596530180 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c00bdf0 con 0x7f05a80b2390 2023-10-10T19:09:05.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.866+0000 7f05ae918700 1 -- 172.21.15.175:0/3596530180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a80b2390 msgr2=0x7f05a8068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:05.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.866+0000 7f05ae918700 1 --2- 172.21.15.175:0/3596530180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a80b2390 0x7f05a8068690 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f059c0099b0 tx=0x7f059c009250 comp rx=0 tx=0).stop 2023-10-10T19:09:05.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.866+0000 7f05ae918700 1 -- 172.21.15.175:0/3596530180 shutdown_connections 2023-10-10T19:09:05.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.866+0000 7f05ae918700 1 --2- 172.21.15.175:0/3596530180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 0x7f05a8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:05.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.866+0000 7f05ae918700 1 --2- 172.21.15.175:0/3596530180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a80b2390 0x7f05a8068690 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:05.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.866+0000 7f05ae918700 1 -- 172.21.15.175:0/3596530180 >> 172.21.15.175:0/3596530180 conn(0x7f05a8063230 msgr2=0x7f05a8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:05.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.866+0000 7f05ae918700 1 -- 172.21.15.175:0/3596530180 shutdown_connections 2023-10-10T19:09:05.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.866+0000 7f05ae918700 1 -- 172.21.15.175:0/3596530180 wait complete. 2023-10-10T19:09:05.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.867+0000 7f05ae918700 1 Processor -- start 2023-10-10T19:09:05.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.867+0000 7f05ae918700 1 -- start start 2023-10-10T19:09:05.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05ae918700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a8068bd0 0x7f05a80cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:05.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05ae918700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 0x7f05a8165d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:05.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05ae918700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a80b2390 0x7f05a8166270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:05.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05ae918700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80c4570 con 0x7f05a8068bd0 2023-10-10T19:09:05.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05ae918700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80cc570 con 0x7f05a80b2390 2023-10-10T19:09:05.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05ae918700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80cd660 con 0x7f05a80b1fc0 2023-10-10T19:09:05.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a8068bd0 0x7f05a80cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:05.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a8068bd0 0x7f05a80cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39856/0 (socket says 172.21.15.175:39856) 2023-10-10T19:09:05.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 -- 172.21.15.175:0/807017886 learned_addr learned my addr 172.21.15.175:0/807017886 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:05.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 -- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 msgr2=0x7f05a8165d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:05.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a77fe700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 0x7f05a8165d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:05.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05aceb5700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a80b2390 0x7f05a8166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:05.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 0x7f05a8165d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:05.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 -- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a80b2390 msgr2=0x7f05a8166270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:05.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a80b2390 0x7f05a8166270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:05.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 -- 172.21.15.175:0/807017886 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0094f0 con 0x7f05a8068bd0 2023-10-10T19:09:05.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05a7fff700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a8068bd0 0x7f05a80cb300 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f05a000e190 tx=0x7f05a0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:05.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f058ffff700 1 -- 172.21.15.175:0/807017886 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a000ee30 con 0x7f05a8068bd0 2023-10-10T19:09:05.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f058ffff700 1 -- 172.21.15.175:0/807017886 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a0014420 con 0x7f05a8068bd0 2023-10-10T19:09:05.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a80cbc80 con 0x7f05a8068bd0 2023-10-10T19:09:05.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.868+0000 7f058ffff700 1 -- 172.21.15.175:0/807017886 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a00147e0 con 0x7f05a8068bd0 2023-10-10T19:09:05.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.869+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a80cbf20 con 0x7f05a8068bd0 2023-10-10T19:09:05.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.869+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a8067570 con 0x7f05a8068bd0 2023-10-10T19:09:05.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.870+0000 7f058ffff700 1 -- 172.21.15.175:0/807017886 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f05a0014b90 con 0x7f05a8068bd0 2023-10-10T19:09:05.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.870+0000 7f058ffff700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0594066f30 0x7f05940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:05.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.870+0000 7f058ffff700 1 -- 172.21.15.175:0/807017886 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f05a0085270 con 0x7f05a8068bd0 2023-10-10T19:09:05.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.870+0000 7f05a77fe700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0594066f30 0x7f05940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:05.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.871+0000 7f05a77fe700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0594066f30 0x7f05940693e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f059c0099b0 tx=0x7f059c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:05.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:05.874+0000 7f058ffff700 1 -- 172.21.15.175:0/807017886 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05a0052060 con 0x7f05a8068bd0 2023-10-10T19:09:06.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.104+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05a8069060 con 0x7f05a8068bd0 2023-10-10T19:09:06.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.104+0000 7f058ffff700 1 -- 172.21.15.175:0/807017886 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+5434 (secure 0 0 0) 0x7f05a0009a60 con 0x7f05a8068bd0 2023-10-10T19:09:06.105 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:06.105 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:04.664614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:06.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0594066f30 msgr2=0x7f05940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:06.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0594066f30 0x7f05940693e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f059c0099b0 tx=0x7f059c017000 comp rx=0 tx=0).stop 2023-10-10T19:09:06.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a8068bd0 msgr2=0x7f05a80cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:06.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a8068bd0 0x7f05a80cb300 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f05a000e190 tx=0x7f05a0009920 comp rx=0 tx=0).stop 2023-10-10T19:09:06.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 shutdown_connections 2023-10-10T19:09:06.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0594066f30 0x7f05940693e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:06.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a80b2390 0x7f05a8166270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:06.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05a8068bd0 0x7f05a80cb300 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:06.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 --2- 172.21.15.175:0/807017886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05a80b1fc0 0x7f05a8165d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:06.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 >> 172.21.15.175:0/807017886 conn(0x7f05a8063230 msgr2=0x7f05a8152540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:06.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 shutdown_connections 2023-10-10T19:09:06.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:06.107+0000 7f05ae918700 1 -- 172.21.15.175:0/807017886 wait complete. 2023-10-10T19:09:06.109 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 186 2023-10-10T19:09:06.119 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=3, replacing gid=5552 2023-10-10T19:09:06.119 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.f 2023-10-10T19:09:09.793 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 4% (5/107) Updating files: 5% (6/107) Updating files: 6% (7/107) Updating files: 7% (8/107) Updating files: 8% (9/107) Updating files: 9% (10/107) Updating files: 10% (11/107) Updating files: 11% (12/107) Updating files: 12% (13/107) Updating files: 13% (14/107) Updating files: 14% (15/107) Updating files: 15% (17/107) Updating files: 16% (18/107) Updating files: 17% (19/107) Updating files: 18% (20/107) Updating files: 19% (21/107) Updating files: 20% (22/107) Updating files: 21% (23/107) Updating files: 22% (24/107) Updating files: 23% (25/107) Updating files: 24% (26/107) Updating files: 25% (27/107) Updating files: 26% (28/107) Updating files: 27% (29/107) Updating files: 28% (30/107) Updating files: 29% (32/107) Updating files: 30% (33/107) Updating files: 31% (34/107) Updating files: 32% (35/107) Updating files: 33% (36/107) Updating files: 34% (37/107) Updating files: 35% (38/107) Updating files: 36% (39/107) Updating files: 37% (40/107) Updating files: 38% (41/107) Updating files: 39% (42/107) Updating files: 40% (43/107) Updating files: 41% (44/107) Updating files: 42% (45/107) Updating files: 43% (47/107) Updating files: 44% (48/107) Updating files: 45% (49/107) Updating files: 46% (50/107) Updating files: 47% (51/107) Updating files: 48% (52/107) Updating files: 49% (53/107) Updating files: 50% (54/107) Updating files: 51% (55/107) Updating files: 52% (56/107) Updating files: 53% (57/107) Updating files: 54% (58/107) Updating files: 55% (59/107) Updating files: 56% (60/107) Updating files: 57% (61/107) Updating files: 58% (63/107) Updating files: 59% (64/107) Updating files: 60% (65/107) Updating files: 61% (66/107) Updating files: 62% (67/107) Updating files: 63% (68/107) Updating files: 64% (69/107) Updating files: 65% (70/107) Updating files: 66% (71/107) Updating files: 67% (72/107) Updating files: 68% (73/107) Updating files: 69% (74/107) Updating files: 70% (75/107) Updating files: 71% (76/107) Updating files: 72% (78/107) Updating files: 73% (79/107) Updating files: 74% (80/107) Updating files: 75% (81/107) Updating files: 76% (82/107) Updating files: 77% (83/107) Updating files: 78% (84/107) Updating files: 79% (85/107) Updating files: 80% (86/107) Updating files: 81% (87/107) Updating files: 82% (88/107) Updating files: 83% (89/107) Updating files: 84% (90/107) Updating files: 85% (91/107) Updating files: 86% (93/107) Updating files: 87% (94/107) Updating files: 88% (95/107) Updating files: 89% (96/107) Updating files: 90% (97/107) Updating files: 91% (98/107) Updating files: 92% (99/107) Updating files: 93% (100/107) Updating files: 94% (101/107) Updating files: 95% (102/107) Updating files: 96% (103/107) Updating files: 97% (104/107) Updating files: 98% (105/107) Updating files: 99% (106/107) Updating files: 100% (107/107) Updating files: 100% (107/107), done. 2023-10-10T19:09:09.858 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 38dbaa5175c v0.16 2023-10-10T19:09:09.859 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.16 2023-10-10T19:09:09.903 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:09:09.903 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.17 2023-10-10T19:09:09.903 INFO:tasks.workunit.client.0.smithi175.stdout:v0.17 2023-10-10T19:09:09.904 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.17 2023-10-10T19:09:09.904 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.17 2023-10-10T19:09:14.120 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-10T19:09:14.121 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-10T19:09:14.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-10T19:09:14.124 INFO:tasks.ceph.mds.f:Started 2023-10-10T19:09:14.124 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:14.202 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-10T19:09:14.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.248+0000 7fb53e3e8700 1 Processor -- start 2023-10-10T19:09:14.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.248+0000 7fb53e3e8700 1 -- start start 2023-10-10T19:09:14.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53e3e8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb53814fd90 0x7fb538159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:14.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53e3e8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb53808a100 0x7fb5380c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:14.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53e3e8700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb538067fa0 con 0x7fb5380cd040 2023-10-10T19:09:14.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53e3e8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380c5600 con 0x7fb53808a100 2023-10-10T19:09:14.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53e3e8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb538156b70 con 0x7fb53814fd90 2023-10-10T19:09:14.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53cbe5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb53808a100 0x7fb5380c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:14.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53dbe7700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb5380cd040 0x7fb5380684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39956/0 (socket says 172.21.15.175:39956) 2023-10-10T19:09:14.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53dbe7700 1 -- 172.21.15.175:0/1095146158 learned_addr learned my addr 172.21.15.175:0/1095146158 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:09:14.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53d3e6700 1 --2- 172.21.15.175:0/1095146158 >> v2:172.21.15.191:3300/0 conn(0x7fb53814fd90 0x7fb538159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:14.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53cbe5700 1 -- 172.21.15.175:0/1095146158 >> v2:172.21.15.191:3300/0 conn(0x7fb53814fd90 msgr2=0x7fb538159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:14.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53cbe5700 1 --2- 172.21.15.175:0/1095146158 >> v2:172.21.15.191:3300/0 conn(0x7fb53814fd90 0x7fb538159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53cbe5700 1 -- 172.21.15.175:0/1095146158 >> v1:172.21.15.175:6789/0 conn(0x7fb5380cd040 legacy=0x7fb5380684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:14.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53cbe5700 1 -- 172.21.15.175:0/1095146158 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538150160 con 0x7fb53808a100 2023-10-10T19:09:14.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb537fff700 1 -- 172.21.15.175:0/1095146158 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881121847 0 0) 0x7fb538067fa0 con 0x7fb5380cd040 2023-10-10T19:09:14.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.249+0000 7fb53cbe5700 1 --2- 172.21.15.175:0/1095146158 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb53808a100 0x7fb5380c5eb0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb538065080 tx=0x7fb52c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d2f61b5aecf373 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:14.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb537fff700 1 -- 172.21.15.175:0/1095146158 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c00b690 con 0x7fb53808a100 2023-10-10T19:09:14.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb537fff700 1 -- 172.21.15.175:0/1095146158 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c00b7f0 con 0x7fb53808a100 2023-10-10T19:09:14.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb537fff700 1 -- 172.21.15.175:0/1095146158 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c00bb90 con 0x7fb53808a100 2023-10-10T19:09:14.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb53e3e8700 1 -- 172.21.15.175:0/1095146158 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb53808a100 msgr2=0x7fb5380c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:14.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/1095146158 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb53808a100 0x7fb5380c5eb0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb538065080 tx=0x7fb52c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb53e3e8700 1 -- 172.21.15.175:0/1095146158 shutdown_connections 2023-10-10T19:09:14.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/1095146158 >> v2:172.21.15.191:3300/0 conn(0x7fb53814fd90 0x7fb538159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/1095146158 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb53808a100 0x7fb5380c5eb0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb53e3e8700 1 -- 172.21.15.175:0/1095146158 >> 172.21.15.175:0/1095146158 conn(0x7fb538063d00 msgr2=0x7fb5380cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:14.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb53e3e8700 1 -- 172.21.15.175:0/1095146158 shutdown_connections 2023-10-10T19:09:14.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.250+0000 7fb53e3e8700 1 -- 172.21.15.175:0/1095146158 wait complete. 2023-10-10T19:09:14.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.251+0000 7fb53e3e8700 1 Processor -- start 2023-10-10T19:09:14.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.251+0000 7fb53e3e8700 1 -- start start 2023-10-10T19:09:14.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53e3e8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb53808a100 0x7fb5380cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:14.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53e3e8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5380cd040 0x7fb5381520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:14.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53e3e8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb53814fd90 0x7fb5381498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:14.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53e3e8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380cc0f0 con 0x7fb5380cd040 2023-10-10T19:09:14.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53e3e8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380cbdd0 con 0x7fb53814fd90 2023-10-10T19:09:14.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53e3e8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380cc340 con 0x7fb53808a100 2023-10-10T19:09:14.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53d3e6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb53808a100 0x7fb5380cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:14.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53cbe5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5380cd040 0x7fb5381520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:14.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53d3e6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb53808a100 0x7fb5380cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:51166/0 (socket says 172.21.15.175:51166) 2023-10-10T19:09:14.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53d3e6700 1 -- 172.21.15.175:0/2069331487 learned_addr learned my addr 172.21.15.175:0/2069331487 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:14.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.252+0000 7fb53dbe7700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb53814fd90 0x7fb5381498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:14.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53d3e6700 1 -- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb53814fd90 msgr2=0x7fb5381498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:14.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53d3e6700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb53814fd90 0x7fb5381498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53d3e6700 1 -- 172.21.15.175:0/2069331487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5380cd040 msgr2=0x7fb5381520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:14.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53d3e6700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5380cd040 0x7fb5381520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53d3e6700 1 -- 172.21.15.175:0/2069331487 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c009420 con 0x7fb53808a100 2023-10-10T19:09:14.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53dbe7700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb53814fd90 0x7fb5381498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:09:14.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53cbe5700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5380cd040 0x7fb5381520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:09:14.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53d3e6700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb53808a100 0x7fb5380cb890 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb53000e930 tx=0x7fb5300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:14.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb5357fa700 1 -- 172.21.15.175:0/2069331487 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530013680 con 0x7fb53808a100 2023-10-10T19:09:14.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb5357fa700 1 -- 172.21.15.175:0/2069331487 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5300137e0 con 0x7fb53808a100 2023-10-10T19:09:14.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5380cc620 con 0x7fb53808a100 2023-10-10T19:09:14.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb5357fa700 1 -- 172.21.15.175:0/2069331487 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530013b80 con 0x7fb53808a100 2023-10-10T19:09:14.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.253+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5380cb090 con 0x7fb53808a100 2023-10-10T19:09:14.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.254+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb538067570 con 0x7fb53808a100 2023-10-10T19:09:14.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.254+0000 7fb5357fa700 1 -- 172.21.15.175:0/2069331487 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb53000c3e0 con 0x7fb53808a100 2023-10-10T19:09:14.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.255+0000 7fb5357fa700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb524066f30 0x7fb5240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:14.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.255+0000 7fb5357fa700 1 -- 172.21.15.175:0/2069331487 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fb530084a60 con 0x7fb53808a100 2023-10-10T19:09:14.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.255+0000 7fb53cbe5700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb524066f30 0x7fb5240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:14.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.255+0000 7fb53cbe5700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb524066f30 0x7fb5240693e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0093f0 tx=0x7fb52c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:14.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.260+0000 7fb5357fa700 1 -- 172.21.15.175:0/2069331487 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb530051850 con 0x7fb53808a100 2023-10-10T19:09:14.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.488+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb538155f80 con 0x7fb53808a100 2023-10-10T19:09:14.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.489+0000 7fb5357fa700 1 -- 172.21.15.175:0/2069331487 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5430 (secure 0 0 0) 0x7fb530054e70 con 0x7fb53808a100 2023-10-10T19:09:14.490 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:14.490 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:13.686478+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:14.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb524066f30 msgr2=0x7fb5240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:14.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb524066f30 0x7fb5240693e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0093f0 tx=0x7fb52c017000 comp rx=0 tx=0).stop 2023-10-10T19:09:14.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb53808a100 msgr2=0x7fb5380cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:14.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb53808a100 0x7fb5380cb890 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb53000e930 tx=0x7fb5300091a0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 shutdown_connections 2023-10-10T19:09:14.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb524066f30 0x7fb5240693e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb53814fd90 0x7fb5381498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb53808a100 0x7fb5380cb890 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 --2- 172.21.15.175:0/2069331487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5380cd040 0x7fb5381520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:14.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 >> 172.21.15.175:0/2069331487 conn(0x7fb538063d00 msgr2=0x7fb5381592b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:14.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 shutdown_connections 2023-10-10T19:09:14.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:14.491+0000 7fb53e3e8700 1 -- 172.21.15.175:0/2069331487 wait complete. 2023-10-10T19:09:14.496 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 188 2023-10-10T19:09:14.504 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-10T19:09:16.505 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:16.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.620+0000 7fb1d112c700 1 Processor -- start 2023-10-10T19:09:16.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.620+0000 7fb1d112c700 1 -- start start 2023-10-10T19:09:16.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.620+0000 7fb1d112c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb1cc14fbf0 0x7fb1cc159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:16.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.620+0000 7fb1d112c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb1cc14ffc0 0x7fb1cc063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:16.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.620+0000 7fb1d112c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1cc156800 con 0x7fb1cc0681e0 2023-10-10T19:09:16.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.620+0000 7fb1d112c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1cc0c59e0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.620+0000 7fb1d112c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1cc15a2b0 con 0x7fb1cc14fbf0 2023-10-10T19:09:16.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1ca59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb1cc14ffc0 0x7fb1cc063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:16.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1ca59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb1cc14ffc0 0x7fb1cc063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46470/0 (socket says 172.21.15.175:46470) 2023-10-10T19:09:16.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1ca59c700 1 -- 172.21.15.175:0/1582870875 learned_addr learned my addr 172.21.15.175:0/1582870875 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:16.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1cad9d700 1 --2- 172.21.15.175:0/1582870875 >> v2:172.21.15.191:3300/0 conn(0x7fb1cc14fbf0 0x7fb1cc159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:16.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1ca59c700 1 -- 172.21.15.175:0/1582870875 >> v2:172.21.15.191:3300/0 conn(0x7fb1cc14fbf0 msgr2=0x7fb1cc159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:16.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1ca59c700 1 --2- 172.21.15.175:0/1582870875 >> v2:172.21.15.191:3300/0 conn(0x7fb1cc14fbf0 0x7fb1cc159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1ca59c700 1 -- 172.21.15.175:0/1582870875 >> v1:172.21.15.191:6789/0 conn(0x7fb1cc0681e0 legacy=0x7fb1cc0c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:16.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1ca59c700 1 -- 172.21.15.175:0/1582870875 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc063e30 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1ca59c700 1 --2- 172.21.15.175:0/1582870875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc063230 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0cc6c0 tx=0x7fb1c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d228833b7bb7f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:16.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1c9d9b700 1 -- 172.21.15.175:0/1582870875 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c000b7e0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1c9d9b700 1 -- 172.21.15.175:0/1582870875 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c000b940 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.621+0000 7fb1c9d9b700 1 -- 172.21.15.175:0/1582870875 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c000bce0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.622+0000 7fb1d112c700 1 -- 172.21.15.175:0/1582870875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 msgr2=0x7fb1cc063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:16.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.622+0000 7fb1d112c700 1 --2- 172.21.15.175:0/1582870875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc063230 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0cc6c0 tx=0x7fb1c0009250 comp rx=0 tx=0).stop 2023-10-10T19:09:16.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.622+0000 7fb1d112c700 1 -- 172.21.15.175:0/1582870875 shutdown_connections 2023-10-10T19:09:16.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.622+0000 7fb1d112c700 1 --2- 172.21.15.175:0/1582870875 >> v2:172.21.15.191:3300/0 conn(0x7fb1cc14fbf0 0x7fb1cc159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.622+0000 7fb1d112c700 1 --2- 172.21.15.175:0/1582870875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc063230 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.622+0000 7fb1d112c700 1 -- 172.21.15.175:0/1582870875 >> 172.21.15.175:0/1582870875 conn(0x7fb1cc0cc760 msgr2=0x7fb1cc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:16.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.622+0000 7fb1d112c700 1 -- 172.21.15.175:0/1582870875 shutdown_connections 2023-10-10T19:09:16.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.622+0000 7fb1d112c700 1 -- 172.21.15.175:0/1582870875 wait complete. 2023-10-10T19:09:16.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.623+0000 7fb1d112c700 1 Processor -- start 2023-10-10T19:09:16.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.623+0000 7fb1d112c700 1 -- start start 2023-10-10T19:09:16.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1d112c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1cc0681e0 0x7fb1cc166ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:16.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1d112c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1cc14fbf0 0x7fb1cc1593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:16.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1d112c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc167530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:16.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1d112c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc1534d0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1d112c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc166ff0 con 0x7fb1cc0681e0 2023-10-10T19:09:16.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1d112c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc152140 con 0x7fb1cc14fbf0 2023-10-10T19:09:16.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc167530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:16.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc167530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46474/0 (socket says 172.21.15.175:46474) 2023-10-10T19:09:16.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 -- 172.21.15.175:0/2118059959 learned_addr learned my addr 172.21.15.175:0/2118059959 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:16.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cad9d700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1cc0681e0 0x7fb1cc166ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:16.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1ca59c700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1cc14fbf0 0x7fb1cc1593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:16.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 -- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1cc14fbf0 msgr2=0x7fb1cc1593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:16.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1cc14fbf0 0x7fb1cc1593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 -- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1cc0681e0 msgr2=0x7fb1cc166ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:16.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1cc0681e0 0x7fb1cc166ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 -- 172.21.15.175:0/2118059959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c00094f0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1cb59e700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc167530 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c5b0 tx=0x7fb1bc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:16.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1b2ffd700 1 -- 172.21.15.175:0/2118059959 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc011590 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1b2ffd700 1 -- 172.21.15.175:0/2118059959 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc011710 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1cc1531f0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1b2ffd700 1 -- 172.21.15.175:0/2118059959 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc011ad0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.624+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1cc151af0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.625+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1cc1522d0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.626+0000 7fb1b2ffd700 1 -- 172.21.15.175:0/2118059959 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb1bc01c460 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.626+0000 7fb1b2ffd700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1b8066f30 0x7fb1b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:16.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.626+0000 7fb1b2ffd700 1 -- 172.21.15.175:0/2118059959 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fb1bc084600 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.626+0000 7fb1cad9d700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1b8066f30 0x7fb1b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:16.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.627+0000 7fb1cad9d700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1b8066f30 0x7fb1b80693e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4007680 tx=0x7fb1c40098d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:16.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.630+0000 7fb1b2ffd700 1 -- 172.21.15.175:0/2118059959 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1bc051350 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.859+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1cc1524d0 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.860+0000 7fb1b2ffd700 1 -- 172.21.15.175:0/2118059959 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+6193 (secure 0 0 0) 0x7fb1bc054970 con 0x7fb1cc14ffc0 2023-10-10T19:09:16.862 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:16.862 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6373,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:14.689894+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:16.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1b8066f30 msgr2=0x7fb1b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:16.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1b8066f30 0x7fb1b80693e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4007680 tx=0x7fb1c40098d0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 msgr2=0x7fb1cc167530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:16.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc167530 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c5b0 tx=0x7fb1bc007f60 comp rx=0 tx=0).stop 2023-10-10T19:09:16.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 shutdown_connections 2023-10-10T19:09:16.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1b8066f30 0x7fb1b80693e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1cc14ffc0 0x7fb1cc167530 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1cc0681e0 0x7fb1cc166ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 --2- 172.21.15.175:0/2118059959 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1cc14fbf0 0x7fb1cc1593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:16.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 >> 172.21.15.175:0/2118059959 conn(0x7fb1cc0cc760 msgr2=0x7fb1cc0ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:16.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 shutdown_connections 2023-10-10T19:09:16.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:16.862+0000 7fb1d112c700 1 -- 172.21.15.175:0/2118059959 wait complete. 2023-10-10T19:09:16.866 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 189 2023-10-10T19:09:16.875 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-10T19:09:16.876 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-10-10T19:09:22.103 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:09:22.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:09:24.617 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 15% (33/217) Updating files: 16% (35/217) Updating files: 17% (37/217) Updating files: 18% (40/217) Updating files: 19% (42/217) Updating files: 20% (44/217) Updating files: 21% (46/217) Updating files: 22% (48/217) Updating files: 23% (50/217) Updating files: 24% (53/217) Updating files: 25% (55/217) Updating files: 26% (57/217) Updating files: 27% (59/217) Updating files: 28% (61/217) Updating files: 29% (63/217) Updating files: 30% (66/217) Updating files: 31% (68/217) Updating files: 32% (70/217) Updating files: 33% (72/217) Updating files: 34% (74/217) Updating files: 35% (76/217) Updating files: 36% (79/217) Updating files: 37% (81/217) Updating files: 38% (83/217) Updating files: 39% (85/217) Updating files: 40% (87/217) Updating files: 41% (89/217) Updating files: 42% (92/217) Updating files: 43% (94/217) Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 64% (140/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 85% (186/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-10T19:09:24.639 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at dac3a8c35fb v0.17 2023-10-10T19:09:24.640 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.17 2023-10-10T19:09:24.701 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:09:24.702 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.18 2023-10-10T19:09:24.702 INFO:tasks.workunit.client.0.smithi175.stdout:v0.18 2023-10-10T19:09:24.703 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.18 2023-10-10T19:09:24.703 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.18 2023-10-10T19:09:37.671 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 8% (65/779) Updating files: 9% (71/779) Updating files: 10% (78/779) Updating files: 11% (86/779) Updating files: 12% (94/779) Updating files: 13% (102/779) Updating files: 14% (110/779) Updating files: 15% (117/779) Updating files: 16% (125/779) Updating files: 16% (129/779) Updating files: 17% (133/779) Updating files: 18% (141/779) Updating files: 19% (149/779) Updating files: 20% (156/779) Updating files: 21% (164/779) Updating files: 22% (172/779) Updating files: 23% (180/779) Updating files: 24% (187/779) Updating files: 25% (195/779) Updating files: 26% (203/779) Updating files: 27% (211/779) Updating files: 28% (219/779) Updating files: 29% (226/779) Updating files: 30% (234/779) Updating files: 31% (242/779) Updating files: 32% (250/779) Updating files: 33% (258/779) Updating files: 34% (265/779) Updating files: 35% (273/779) Updating files: 36% (281/779) Updating files: 37% (289/779) Updating files: 38% (297/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 42% (332/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 49% (383/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 51% (403/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 59% (464/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 75% (585/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 80% (626/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 90% (706/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2023-10-10T19:09:37.689 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 2f10137986a v0.18 2023-10-10T19:09:37.690 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.18 2023-10-10T19:09:37.799 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:09:37.800 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.19 2023-10-10T19:09:37.800 INFO:tasks.workunit.client.0.smithi175.stdout:v0.19 2023-10-10T19:09:37.801 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.19 2023-10-10T19:09:37.801 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.19 2023-10-10T19:09:39.877 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:39.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.992+0000 7f677198a700 1 Processor -- start 2023-10-10T19:09:39.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.992+0000 7f677198a700 1 -- start start 2023-10-10T19:09:39.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.992+0000 7f677198a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c0b1fc0 0x7f676c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:39.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.992+0000 7f677198a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f676c0b2390 0x7f676c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:39.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.992+0000 7f677198a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f676c1574b0 con 0x7f676c068bd0 2023-10-10T19:09:39.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.992+0000 7f677198a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c150770 con 0x7f676c0b2390 2023-10-10T19:09:39.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.992+0000 7f677198a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0afad0 con 0x7f676c0b1fc0 2023-10-10T19:09:39.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.992+0000 7f67627fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f676c0b2390 0x7f676c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:39.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f67627fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f676c0b2390 0x7f676c068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60322/0 (socket says 172.21.15.175:60322) 2023-10-10T19:09:39.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f67627fc700 1 -- 172.21.15.175:0/785634079 learned_addr learned my addr 172.21.15.175:0/785634079 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:39.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f676affd700 1 --2- 172.21.15.175:0/785634079 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c0b1fc0 0x7f676c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:39.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f67627fc700 1 -- 172.21.15.175:0/785634079 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c0b1fc0 msgr2=0x7f676c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f67627fc700 1 --2- 172.21.15.175:0/785634079 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c0b1fc0 0x7f676c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f67627fc700 1 -- 172.21.15.175:0/785634079 >> v1:172.21.15.191:6789/0 conn(0x7f676c068bd0 legacy=0x7f676c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f67627fc700 1 -- 172.21.15.175:0/785634079 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c0af4c0 con 0x7f676c0b2390 2023-10-10T19:09:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f67627fc700 1 --2- 172.21.15.175:0/785634079 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c068690 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f67540099b0 tx=0x7f6754009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbb6d841539de99 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f676a7fc700 1 -- 172.21.15.175:0/785634079 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754009ea0 con 0x7f676c0b2390 2023-10-10T19:09:39.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f676a7fc700 1 -- 172.21.15.175:0/785634079 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675400b8f0 con 0x7f676c0b2390 2023-10-10T19:09:39.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.993+0000 7f676a7fc700 1 -- 172.21.15.175:0/785634079 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675400bc90 con 0x7f676c0b2390 2023-10-10T19:09:39.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.994+0000 7f677198a700 1 -- 172.21.15.175:0/785634079 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 msgr2=0x7f676c068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:39.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.994+0000 7f677198a700 1 --2- 172.21.15.175:0/785634079 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c068690 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f67540099b0 tx=0x7f6754009250 comp rx=0 tx=0).stop 2023-10-10T19:09:39.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.994+0000 7f677198a700 1 -- 172.21.15.175:0/785634079 shutdown_connections 2023-10-10T19:09:40.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.994+0000 7f677198a700 1 --2- 172.21.15.175:0/785634079 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c0b1fc0 0x7f676c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.994+0000 7f677198a700 1 --2- 172.21.15.175:0/785634079 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c068690 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.994+0000 7f677198a700 1 -- 172.21.15.175:0/785634079 >> 172.21.15.175:0/785634079 conn(0x7f676c063230 msgr2=0x7f676c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:40.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.994+0000 7f677198a700 1 -- 172.21.15.175:0/785634079 shutdown_connections 2023-10-10T19:09:40.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.994+0000 7f677198a700 1 -- 172.21.15.175:0/785634079 wait complete. 2023-10-10T19:09:40.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.995+0000 7f677198a700 1 Processor -- start 2023-10-10T19:09:40.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.995+0000 7f677198a700 1 -- start start 2023-10-10T19:09:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f677198a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c068bd0 0x7f676c1598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f677198a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f676c0b1fc0 0x7f676c159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f677198a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c165f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f677198a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0c4950 con 0x7f676c0b2390 2023-10-10T19:09:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f677198a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c1550a0 con 0x7f676c0b1fc0 2023-10-10T19:09:40.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f677198a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c16a110 con 0x7f676c068bd0 2023-10-10T19:09:40.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c165f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60328/0 (socket says 172.21.15.175:60328) 2023-10-10T19:09:40.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 -- 172.21.15.175:0/3890636240 learned_addr learned my addr 172.21.15.175:0/3890636240 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:40.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f67627fc700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f676c0b1fc0 0x7f676c159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676affd700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c068bd0 0x7f676c1598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 -- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c068bd0 msgr2=0x7f676c1598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c068bd0 0x7f676c1598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 -- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f676c0b1fc0 msgr2=0x7f676c159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f676c0b1fc0 0x7f676c159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 -- 172.21.15.175:0/3890636240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67540094f0 con 0x7f676c0b2390 2023-10-10T19:09:40.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f676b7fe700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c165f80 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f676400c910 tx=0x7f67640016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:40.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f67637fe700 1 -- 172.21.15.175:0/3890636240 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764012630 con 0x7f676c0b2390 2023-10-10T19:09:40.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f67637fe700 1 -- 172.21.15.175:0/3890636240 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6764012790 con 0x7f676c0b2390 2023-10-10T19:09:40.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f67637fe700 1 -- 172.21.15.175:0/3890636240 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764012b50 con 0x7f676c0b2390 2023-10-10T19:09:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c1552d0 con 0x7f676c0b2390 2023-10-10T19:09:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.996+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c1697e0 con 0x7f676c0b2390 2023-10-10T19:09:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.997+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c151530 con 0x7f676c0b2390 2023-10-10T19:09:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.998+0000 7f67637fe700 1 -- 172.21.15.175:0/3890636240 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f676401b450 con 0x7f676c0b2390 2023-10-10T19:09:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.998+0000 7f67637fe700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f674c066f60 0x7f674c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.998+0000 7f67637fe700 1 -- 172.21.15.175:0/3890636240 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f6764084b00 con 0x7f676c0b2390 2023-10-10T19:09:40.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.998+0000 7f676affd700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f674c066f60 0x7f674c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:39.999+0000 7f676affd700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f674c066f60 0x7f674c069410 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f675c009db0 tx=0x7f675c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:40.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.002+0000 7f67637fe700 1 -- 172.21.15.175:0/3890636240 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6764055380 con 0x7f676c0b2390 2023-10-10T19:09:40.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.241+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f676c067570 con 0x7f676c0b2390 2023-10-10T19:09:40.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.241+0000 7f67637fe700 1 -- 172.21.15.175:0/3890636240 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f6764054f10 con 0x7f676c0b2390 2023-10-10T19:09:40.242 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:40.242 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6373,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:39.671104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:40.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f674c066f60 msgr2=0x7f674c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f674c066f60 0x7f674c069410 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f675c009db0 tx=0x7f675c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 msgr2=0x7f676c165f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c165f80 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f676400c910 tx=0x7f67640016f0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 shutdown_connections 2023-10-10T19:09:40.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f674c066f60 0x7f674c069410 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f676c0b2390 0x7f676c165f80 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f676c068bd0 0x7f676c1598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 --2- 172.21.15.175:0/3890636240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f676c0b1fc0 0x7f676c159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 >> 172.21.15.175:0/3890636240 conn(0x7f676c063230 msgr2=0x7f676c14fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:40.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 shutdown_connections 2023-10-10T19:09:40.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.243+0000 7f677198a700 1 -- 172.21.15.175:0/3890636240 wait complete. 2023-10-10T19:09:40.246 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 194 2023-10-10T19:09:40.255 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=4) 2023-10-10T19:09:40.257 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-10T19:09:40.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:09:40.268 INFO:tasks.ceph.mds.d:Stopped 2023-10-10T19:09:40.269 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:40.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.391+0000 7f035157e700 1 Processor -- start 2023-10-10T19:09:40.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.391+0000 7f035157e700 1 -- start start 2023-10-10T19:09:40.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f035157e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f034c06e910 0x7f034c14f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f035157e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f034c06ece0 0x7f034c14fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f035157e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c06df80 con 0x7f034c0c6210 2023-10-10T19:09:40.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f035157e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034c06e4b0 con 0x7f034c06e910 2023-10-10T19:09:40.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f035157e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034c0c4620 con 0x7f034c06ece0 2023-10-10T19:09:40.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f034affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f034c06e910 0x7f034c14f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f034affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f034c06e910 0x7f034c14f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60336/0 (socket says 172.21.15.175:60336) 2023-10-10T19:09:40.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f034affd700 1 -- 172.21.15.175:0/3166913530 learned_addr learned my addr 172.21.15.175:0/3166913530 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:40.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.392+0000 7f034a7fc700 1 --2- 172.21.15.175:0/3166913530 >> v2:172.21.15.191:3300/0 conn(0x7f034c06ece0 0x7f034c14fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.393+0000 7f034affd700 1 -- 172.21.15.175:0/3166913530 >> v2:172.21.15.191:3300/0 conn(0x7f034c06ece0 msgr2=0x7f034c14fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:40.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.393+0000 7f034affd700 1 --2- 172.21.15.175:0/3166913530 >> v2:172.21.15.191:3300/0 conn(0x7f034c06ece0 0x7f034c14fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.393+0000 7f034affd700 1 -- 172.21.15.175:0/3166913530 >> v1:172.21.15.191:6789/0 conn(0x7f034c0c6210 legacy=0x7f034c0c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.393+0000 7f034affd700 1 -- 172.21.15.175:0/3166913530 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c06e1d0 con 0x7f034c06e910 2023-10-10T19:09:40.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.393+0000 7f034affd700 1 --2- 172.21.15.175:0/3166913530 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c06e910 0x7f034c14f600 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f034c071760 tx=0x7f033c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1626b35ab4f7c1ae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:40.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.393+0000 7f0349ffb700 1 -- 172.21.15.175:0/3166913530 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c009e70 con 0x7f034c06e910 2023-10-10T19:09:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.393+0000 7f0349ffb700 1 -- 172.21.15.175:0/3166913530 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033c00b860 con 0x7f034c06e910 2023-10-10T19:09:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.393+0000 7f0349ffb700 1 -- 172.21.15.175:0/3166913530 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c00bc00 con 0x7f034c06e910 2023-10-10T19:09:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.394+0000 7f035157e700 1 -- 172.21.15.175:0/3166913530 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c06e910 msgr2=0x7f034c14f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.394+0000 7f035157e700 1 --2- 172.21.15.175:0/3166913530 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c06e910 0x7f034c14f600 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f034c071760 tx=0x7f033c009250 comp rx=0 tx=0).stop 2023-10-10T19:09:40.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.394+0000 7f035157e700 1 -- 172.21.15.175:0/3166913530 shutdown_connections 2023-10-10T19:09:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.394+0000 7f035157e700 1 --2- 172.21.15.175:0/3166913530 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c06e910 0x7f034c14f600 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.394+0000 7f035157e700 1 --2- 172.21.15.175:0/3166913530 >> v2:172.21.15.191:3300/0 conn(0x7f034c06ece0 0x7f034c14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.394+0000 7f035157e700 1 -- 172.21.15.175:0/3166913530 >> 172.21.15.175:0/3166913530 conn(0x7f034c0c5650 msgr2=0x7f034c0c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.394+0000 7f035157e700 1 -- 172.21.15.175:0/3166913530 shutdown_connections 2023-10-10T19:09:40.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.394+0000 7f035157e700 1 -- 172.21.15.175:0/3166913530 wait complete. 2023-10-10T19:09:40.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.395+0000 7f035157e700 1 Processor -- start 2023-10-10T19:09:40.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.395+0000 7f035157e700 1 -- start start 2023-10-10T19:09:40.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f035157e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c06e910 0x7f034c0bc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f035157e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c06ece0 0x7f034c14d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f035157e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c0c6210 0x7f034c0be6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f035157e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c1646d0 con 0x7f034c0c6210 2023-10-10T19:09:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f035157e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c1649b0 con 0x7f034c06e910 2023-10-10T19:09:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f035157e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c164c90 con 0x7f034c06ece0 2023-10-10T19:09:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c0c6210 0x7f034c0be6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c0c6210 0x7f034c0be6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60352/0 (socket says 172.21.15.175:60352) 2023-10-10T19:09:40.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 -- 172.21.15.175:0/2431372741 learned_addr learned my addr 172.21.15.175:0/2431372741 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:40.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034affd700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c06e910 0x7f034c0bc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034a7fc700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c06ece0 0x7f034c14d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 -- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c06ece0 msgr2=0x7f034c14d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c06ece0 0x7f034c14d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 -- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c06e910 msgr2=0x7f034c0bc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c06e910 0x7f034c0bc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 -- 172.21.15.175:0/2431372741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c0094f0 con 0x7f034c0c6210 2023-10-10T19:09:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f034b7fe700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c0c6210 0x7f034c0be6c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f034400c8e0 tx=0x7f0344001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f0332ffd700 1 -- 172.21.15.175:0/2431372741 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034400ce80 con 0x7f034c0c6210 2023-10-10T19:09:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f0332ffd700 1 -- 172.21.15.175:0/2431372741 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034400a870 con 0x7f034c0c6210 2023-10-10T19:09:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f0332ffd700 1 -- 172.21.15.175:0/2431372741 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034400ac10 con 0x7f034c0c6210 2023-10-10T19:09:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c164f70 con 0x7f034c0c6210 2023-10-10T19:09:40.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.396+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c165250 con 0x7f034c0c6210 2023-10-10T19:09:40.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.397+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c067570 con 0x7f034c0c6210 2023-10-10T19:09:40.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.398+0000 7f0332ffd700 1 -- 172.21.15.175:0/2431372741 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0344012420 con 0x7f034c0c6210 2023-10-10T19:09:40.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.398+0000 7f0332ffd700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0338066f00 0x7f03380693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:40.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.398+0000 7f0332ffd700 1 -- 172.21.15.175:0/2431372741 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f0344084d30 con 0x7f034c0c6210 2023-10-10T19:09:40.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.398+0000 7f034affd700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0338066f00 0x7f03380693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:40.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.399+0000 7f034affd700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0338066f00 0x7f03380693b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f034c071760 tx=0x7f033c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:40.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.402+0000 7f0332ffd700 1 -- 172.21.15.175:0/2431372741 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f03440555b0 con 0x7f034c0c6210 2023-10-10T19:09:40.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.630+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f034c0c3fd0 con 0x7f034c0c6210 2023-10-10T19:09:40.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.630+0000 7f0332ffd700 1 -- 172.21.15.175:0/2431372741 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f0344055140 con 0x7f034c0c6210 2023-10-10T19:09:40.631 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:40.631 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6373,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:39.671104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:40.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0338066f00 msgr2=0x7f03380693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0338066f00 0x7f03380693b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f034c071760 tx=0x7f033c017000 comp rx=0 tx=0).stop 2023-10-10T19:09:40.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c0c6210 msgr2=0x7f034c0be6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:40.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c0c6210 0x7f034c0be6c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f034400c8e0 tx=0x7f0344001710 comp rx=0 tx=0).stop 2023-10-10T19:09:40.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 shutdown_connections 2023-10-10T19:09:40.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0338066f00 0x7f03380693b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f034c0c6210 0x7f034c0be6c0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f034c06e910 0x7f034c0bc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 --2- 172.21.15.175:0/2431372741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f034c06ece0 0x7f034c14d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:40.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 >> 172.21.15.175:0/2431372741 conn(0x7f034c0c5650 msgr2=0x7f034c159a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:40.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 shutdown_connections 2023-10-10T19:09:40.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:40.632+0000 7f035157e700 1 -- 172.21.15.175:0/2431372741 wait complete. 2023-10-10T19:09:40.635 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 194 2023-10-10T19:09:40.645 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:09:42.646 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:42.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.763+0000 7f6396c67700 1 Processor -- start 2023-10-10T19:09:42.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.763+0000 7f6396c67700 1 -- start start 2023-10-10T19:09:42.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6396c67700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b1fc0 0x7f6390155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:42.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6396c67700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63900b2390 0x7f6390068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:42.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6396c67700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6390068bd0 0x7f63900aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:42.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6396c67700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6390150770 con 0x7f63900b2390 2023-10-10T19:09:42.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6396c67700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63901574b0 con 0x7f6390068bd0 2023-10-10T19:09:42.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6396c67700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63900afad0 con 0x7f63900b1fc0 2023-10-10T19:09:42.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6387fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63900b2390 0x7f6390068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:42.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6387fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63900b2390 0x7f6390068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60354/0 (socket says 172.21.15.175:60354) 2023-10-10T19:09:42.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6387fff700 1 -- 172.21.15.175:0/1481569715 learned_addr learned my addr 172.21.15.175:0/1481569715 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:42.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6395204700 1 --2- 172.21.15.175:0/1481569715 >> v2:172.21.15.191:3300/0 conn(0x7f6390068bd0 0x7f63900aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:42.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.764+0000 7f6394a03700 1 --2- 172.21.15.175:0/1481569715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b1fc0 0x7f6390155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:42.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6387fff700 1 -- 172.21.15.175:0/1481569715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b1fc0 msgr2=0x7f6390155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:42.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6387fff700 1 --2- 172.21.15.175:0/1481569715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b1fc0 0x7f6390155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:42.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6387fff700 1 -- 172.21.15.175:0/1481569715 >> v2:172.21.15.191:3300/0 conn(0x7f6390068bd0 msgr2=0x7f63900aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:42.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6387fff700 1 --2- 172.21.15.175:0/1481569715 >> v2:172.21.15.191:3300/0 conn(0x7f6390068bd0 0x7f63900aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:42.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6387fff700 1 -- 172.21.15.175:0/1481569715 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63900af4c0 con 0x7f63900b2390 2023-10-10T19:09:42.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6395204700 1 --2- 172.21.15.175:0/1481569715 >> v2:172.21.15.191:3300/0 conn(0x7f6390068bd0 0x7f63900aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:09:42.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6387fff700 1 --2- 172.21.15.175:0/1481569715 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b2390 0x7f6390068690 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f63900afa40 tx=0x7f6380009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3892f0520b742b08 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:42.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f63877fe700 1 -- 172.21.15.175:0/1481569715 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000b8f0 con 0x7f63900b2390 2023-10-10T19:09:42.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f63877fe700 1 -- 172.21.15.175:0/1481569715 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638000ba50 con 0x7f63900b2390 2023-10-10T19:09:42.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f63877fe700 1 -- 172.21.15.175:0/1481569715 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000bdf0 con 0x7f63900b2390 2023-10-10T19:09:42.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6396c67700 1 -- 172.21.15.175:0/1481569715 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b2390 msgr2=0x7f6390068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:42.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6396c67700 1 --2- 172.21.15.175:0/1481569715 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b2390 0x7f6390068690 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f63900afa40 tx=0x7f6380009250 comp rx=0 tx=0).stop 2023-10-10T19:09:42.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6396c67700 1 -- 172.21.15.175:0/1481569715 shutdown_connections 2023-10-10T19:09:42.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6396c67700 1 --2- 172.21.15.175:0/1481569715 >> v2:172.21.15.191:3300/0 conn(0x7f6390068bd0 0x7f63900aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:42.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6396c67700 1 --2- 172.21.15.175:0/1481569715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b1fc0 0x7f6390155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:42.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6396c67700 1 --2- 172.21.15.175:0/1481569715 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b2390 0x7f6390068690 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:42.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.765+0000 7f6396c67700 1 -- 172.21.15.175:0/1481569715 >> 172.21.15.175:0/1481569715 conn(0x7f6390063230 msgr2=0x7f6390063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:42.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.766+0000 7f6396c67700 1 -- 172.21.15.175:0/1481569715 shutdown_connections 2023-10-10T19:09:42.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.766+0000 7f6396c67700 1 -- 172.21.15.175:0/1481569715 wait complete. 2023-10-10T19:09:42.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.767+0000 7f6396c67700 1 Processor -- start 2023-10-10T19:09:42.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.767+0000 7f6396c67700 1 -- start start 2023-10-10T19:09:42.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.767+0000 7f6396c67700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6390068bd0 0x7f6390167030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:42.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.767+0000 7f6396c67700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b1fc0 0x7f639014da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:42.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.767+0000 7f6396c67700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b2390 0x7f639014dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:42.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.767+0000 7f6396c67700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63900c6700 con 0x7f63900b1fc0 2023-10-10T19:09:42.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.767+0000 7f6396c67700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63900c6250 con 0x7f6390068bd0 2023-10-10T19:09:42.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.767+0000 7f6396c67700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63900c45f0 con 0x7f63900b2390 2023-10-10T19:09:42.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b1fc0 0x7f639014da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:42.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b1fc0 0x7f639014da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60370/0 (socket says 172.21.15.175:60370) 2023-10-10T19:09:42.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 -- 172.21.15.175:0/1171065807 learned_addr learned my addr 172.21.15.175:0/1171065807 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:42.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6394a03700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6390068bd0 0x7f6390167030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:42.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6395204700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b2390 0x7f639014dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:42.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 -- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b2390 msgr2=0x7f639014dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:42.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b2390 0x7f639014dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:42.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 -- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6390068bd0 msgr2=0x7f6390167030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:42.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6390068bd0 0x7f6390167030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:42.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 -- 172.21.15.175:0/1171065807 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63800094f0 con 0x7f63900b1fc0 2023-10-10T19:09:42.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6395204700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b2390 0x7f639014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:09:42.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6387fff700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b1fc0 0x7f639014da70 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f6380006fd0 tx=0x7f6380017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:42.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6384ff9700 1 -- 172.21.15.175:0/1171065807 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380011e20 con 0x7f63900b1fc0 2023-10-10T19:09:42.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6384ff9700 1 -- 172.21.15.175:0/1171065807 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638001a660 con 0x7f63900b1fc0 2023-10-10T19:09:42.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639014e4f0 con 0x7f63900b1fc0 2023-10-10T19:09:42.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6384ff9700 1 -- 172.21.15.175:0/1171065807 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638001aa20 con 0x7f63900b1fc0 2023-10-10T19:09:42.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.768+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63900c6000 con 0x7f63900b1fc0 2023-10-10T19:09:42.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.769+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6390067570 con 0x7f63900b1fc0 2023-10-10T19:09:42.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.769+0000 7f6384ff9700 1 -- 172.21.15.175:0/1171065807 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6380025440 con 0x7f63900b1fc0 2023-10-10T19:09:42.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.770+0000 7f6384ff9700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f637c066f60 0x7f637c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:42.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.770+0000 7f6384ff9700 1 -- 172.21.15.175:0/1171065807 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f63800866d0 con 0x7f63900b1fc0 2023-10-10T19:09:42.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.770+0000 7f6394a03700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f637c066f60 0x7f637c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:42.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.770+0000 7f6394a03700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f637c066f60 0x7f637c069410 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f638c00e4d0 tx=0x7f638c007680 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:42.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:42.774+0000 7f6384ff9700 1 -- 172.21.15.175:0/1171065807 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63800534c0 con 0x7f63900b1fc0 2023-10-10T19:09:43.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.001+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6390069060 con 0x7f63900b1fc0 2023-10-10T19:09:43.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.002+0000 7f6384ff9700 1 -- 172.21.15.175:0/1171065807 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f6380056ae0 con 0x7f63900b1fc0 2023-10-10T19:09:43.002 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:43.002 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6373,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:39.671104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:43.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f637c066f60 msgr2=0x7f637c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:43.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f637c066f60 0x7f637c069410 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f638c00e4d0 tx=0x7f638c007680 comp rx=0 tx=0).stop 2023-10-10T19:09:43.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b1fc0 msgr2=0x7f639014da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:43.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b1fc0 0x7f639014da70 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f6380006fd0 tx=0x7f6380017000 comp rx=0 tx=0).stop 2023-10-10T19:09:43.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 shutdown_connections 2023-10-10T19:09:43.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f637c066f60 0x7f637c069410 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:43.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63900b2390 0x7f639014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:43.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6390068bd0 0x7f6390167030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:43.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 --2- 172.21.15.175:0/1171065807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63900b1fc0 0x7f639014da70 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:43.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 >> 172.21.15.175:0/1171065807 conn(0x7f6390063230 msgr2=0x7f639014fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:43.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 shutdown_connections 2023-10-10T19:09:43.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:43.004+0000 7f6396c67700 1 -- 172.21.15.175:0/1171065807 wait complete. 2023-10-10T19:09:43.007 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 194 2023-10-10T19:09:43.016 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:09:45.017 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:45.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.132+0000 7fdfd7a47700 1 Processor -- start 2023-10-10T19:09:45.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.132+0000 7fdfd7a47700 1 -- start start 2023-10-10T19:09:45.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd7a47700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdfd00b1fc0 0x7fdfd0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:45.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd7a47700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdfd00aff60 0x7fdfd00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:45.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd7a47700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd01574b0 con 0x7fdfd00b2390 2023-10-10T19:09:45.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd7a47700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd0150770 con 0x7fdfd00aff60 2023-10-10T19:09:45.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd7a47700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd00afad0 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd5fe4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdfd00aff60 0x7fdfd00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:45.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd5fe4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdfd00aff60 0x7fdfd00b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52992/0 (socket says 172.21.15.175:52992) 2023-10-10T19:09:45.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd5fe4700 1 -- 172.21.15.175:0/752526553 learned_addr learned my addr 172.21.15.175:0/752526553 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:45.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd57e3700 1 --2- 172.21.15.175:0/752526553 >> v2:172.21.15.191:3300/0 conn(0x7fdfd00b1fc0 0x7fdfd0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:45.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd5fe4700 1 -- 172.21.15.175:0/752526553 >> v2:172.21.15.191:3300/0 conn(0x7fdfd00b1fc0 msgr2=0x7fdfd0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:45.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd5fe4700 1 --2- 172.21.15.175:0/752526553 >> v2:172.21.15.191:3300/0 conn(0x7fdfd00b1fc0 0x7fdfd0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.133+0000 7fdfd5fe4700 1 -- 172.21.15.175:0/752526553 >> v1:172.21.15.191:6789/0 conn(0x7fdfd00b2390 legacy=0x7fdfd0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:45.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfd5fe4700 1 -- 172.21.15.175:0/752526553 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd00af4c0 con 0x7fdfd00aff60 2023-10-10T19:09:45.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfd5fe4700 1 --2- 172.21.15.175:0/752526553 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00aff60 0x7fdfd00b03e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00b28c0 tx=0x7fdfc8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd9e65d57e0dbf6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:45.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfc7fff700 1 -- 172.21.15.175:0/752526553 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc800e070 con 0x7fdfd00aff60 2023-10-10T19:09:45.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfc7fff700 1 -- 172.21.15.175:0/752526553 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc800baa0 con 0x7fdfd00aff60 2023-10-10T19:09:45.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfc7fff700 1 -- 172.21.15.175:0/752526553 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80092e0 con 0x7fdfd00aff60 2023-10-10T19:09:45.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfd7a47700 1 -- 172.21.15.175:0/752526553 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00aff60 msgr2=0x7fdfd00b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:45.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/752526553 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00aff60 0x7fdfd00b03e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00b28c0 tx=0x7fdfc8009840 comp rx=0 tx=0).stop 2023-10-10T19:09:45.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfd7a47700 1 -- 172.21.15.175:0/752526553 shutdown_connections 2023-10-10T19:09:45.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/752526553 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00aff60 0x7fdfd00b03e0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/752526553 >> v2:172.21.15.191:3300/0 conn(0x7fdfd00b1fc0 0x7fdfd0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.134+0000 7fdfd7a47700 1 -- 172.21.15.175:0/752526553 >> 172.21.15.175:0/752526553 conn(0x7fdfd0063230 msgr2=0x7fdfd0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:45.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.135+0000 7fdfd7a47700 1 -- 172.21.15.175:0/752526553 shutdown_connections 2023-10-10T19:09:45.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.135+0000 7fdfd7a47700 1 -- 172.21.15.175:0/752526553 wait complete. 2023-10-10T19:09:45.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd7a47700 1 Processor -- start 2023-10-10T19:09:45.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd7a47700 1 -- start start 2023-10-10T19:09:45.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd7a47700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfd00aff60 0x7fdfd00c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:45.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd7a47700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00b1fc0 0x7fdfd01555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:45.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd7a47700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfd00b2390 0x7fdfd0151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:45.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd7a47700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd0155110 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd7a47700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd00c3f20 con 0x7fdfd00aff60 2023-10-10T19:09:45.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd7a47700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd00cd470 con 0x7fdfd00b2390 2023-10-10T19:09:45.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.136+0000 7fdfd4fe2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00b1fc0 0x7fdfd01555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:45.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd4fe2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00b1fc0 0x7fdfd01555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53004/0 (socket says 172.21.15.175:53004) 2023-10-10T19:09:45.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd4fe2700 1 -- 172.21.15.175:0/2064608474 learned_addr learned my addr 172.21.15.175:0/2064608474 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:45.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd57e3700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfd00aff60 0x7fdfd00c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:45.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd5fe4700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfd00b2390 0x7fdfd0151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:45.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd4fe2700 1 -- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfd00b2390 msgr2=0x7fdfd0151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:45.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd4fe2700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfd00b2390 0x7fdfd0151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd4fe2700 1 -- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfd00aff60 msgr2=0x7fdfd00c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:45.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd4fe2700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfd00aff60 0x7fdfd00c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd4fe2700 1 -- 172.21.15.175:0/2064608474 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc8009ae0 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd4fe2700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00b1fc0 0x7fdfd01555a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000b5e0 tx=0x7fdfc0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:45.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfc57fa700 1 -- 172.21.15.175:0/2064608474 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc000f510 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfc57fa700 1 -- 172.21.15.175:0/2064608474 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc000f670 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd00cd230 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfc57fa700 1 -- 172.21.15.175:0/2064608474 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc000fa30 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.137+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd00ccc70 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.138+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd00712e0 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.139+0000 7fdfc57fa700 1 -- 172.21.15.175:0/2064608474 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdfc001a400 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.139+0000 7fdfc57fa700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfb4066f30 0x7fdfb40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:45.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.139+0000 7fdfc57fa700 1 -- 172.21.15.175:0/2064608474 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7fdfc0084400 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.139+0000 7fdfd57e3700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfb4066f30 0x7fdfb40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:45.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.140+0000 7fdfd57e3700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfb4066f30 0x7fdfb40693e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006fd0 tx=0x7fdfcc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:45.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.143+0000 7fdfc57fa700 1 -- 172.21.15.175:0/2064608474 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdfc0051270 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.368+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfd0068da0 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.368+0000 7fdfc57fa700 1 -- 172.21.15.175:0/2064608474 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6202 (secure 0 0 0) 0x7fdfc0054890 con 0x7fdfd00b1fc0 2023-10-10T19:09:45.369 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:45.369 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6373,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:39.671104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:45.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.370+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfb4066f30 msgr2=0x7fdfb40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:45.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfb4066f30 0x7fdfb40693e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006fd0 tx=0x7fdfcc008040 comp rx=0 tx=0).stop 2023-10-10T19:09:45.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00b1fc0 msgr2=0x7fdfd01555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:45.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00b1fc0 0x7fdfd01555a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000b5e0 tx=0x7fdfc0009120 comp rx=0 tx=0).stop 2023-10-10T19:09:45.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 shutdown_connections 2023-10-10T19:09:45.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdfb4066f30 0x7fdfb40693e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfd00b2390 0x7fdfd0151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfd00aff60 0x7fdfd00c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 --2- 172.21.15.175:0/2064608474 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfd00b1fc0 0x7fdfd01555a0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:45.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 >> 172.21.15.175:0/2064608474 conn(0x7fdfd0063230 msgr2=0x7fdfd0158fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:45.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 shutdown_connections 2023-10-10T19:09:45.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:45.371+0000 7fdfd7a47700 1 -- 172.21.15.175:0/2064608474 wait complete. 2023-10-10T19:09:45.374 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 194 2023-10-10T19:09:45.383 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:09:47.384 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:47.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.511+0000 7f99a7816700 1 Processor -- start 2023-10-10T19:09:47.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.511+0000 7f99a7816700 1 -- start start 2023-10-10T19:09:47.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a7816700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f99a00b1fc0 0x7f99a0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:47.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a7816700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a01574b0 con 0x7f99a00aff60 2023-10-10T19:09:47.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a7816700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a00afad0 con 0x7f99a00b2390 2023-10-10T19:09:47.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a7816700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a0150770 con 0x7f99a00b1fc0 2023-10-10T19:09:47.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a5db3700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f99a00aff60 0x7f99a0159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44608/0 (socket says 172.21.15.175:44608) 2023-10-10T19:09:47.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a5db3700 1 -- 172.21.15.175:0/2779590026 learned_addr learned my addr 172.21.15.175:0/2779590026 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:09:47.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a55b2700 1 --2- 172.21.15.175:0/2779590026 >> v2:172.21.15.191:3300/0 conn(0x7f99a00b1fc0 0x7f99a0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:47.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f9997fff700 1 -- 172.21.15.175:0/2779590026 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652565392 0 0) 0x7f99a01574b0 con 0x7f99a00aff60 2023-10-10T19:09:47.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f9997fff700 1 -- 172.21.15.175:0/2779590026 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f998c003570 con 0x7f99a00aff60 2023-10-10T19:09:47.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f9997fff700 1 -- 172.21.15.175:0/2779590026 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232312408 0 0) 0x7f99a00afad0 con 0x7f99a00b2390 2023-10-10T19:09:47.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f9997fff700 1 -- 172.21.15.175:0/2779590026 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a01574b0 con 0x7f99a00b2390 2023-10-10T19:09:47.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a55b2700 1 -- 172.21.15.175:0/2779590026 >> v1:172.21.15.191:6789/0 conn(0x7f99a00b2390 legacy=0x7f99a0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:47.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a55b2700 1 -- 172.21.15.175:0/2779590026 >> v1:172.21.15.175:6789/0 conn(0x7f99a00aff60 legacy=0x7f99a0159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:47.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a55b2700 1 -- 172.21.15.175:0/2779590026 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a00af4c0 con 0x7f99a00b1fc0 2023-10-10T19:09:47.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.512+0000 7f99a55b2700 1 --2- 172.21.15.175:0/2779590026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00b1fc0 0x7f99a0155fb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f999c0098e0 tx=0x7f999c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8b4c47ec4d148e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:47.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f9997fff700 1 -- 172.21.15.175:0/2779590026 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c011630 con 0x7f99a00b1fc0 2023-10-10T19:09:47.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f9997fff700 1 -- 172.21.15.175:0/2779590026 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c011790 con 0x7f99a00b1fc0 2023-10-10T19:09:47.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f9997fff700 1 -- 172.21.15.175:0/2779590026 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c011b10 con 0x7f99a00b1fc0 2023-10-10T19:09:47.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f99a7816700 1 -- 172.21.15.175:0/2779590026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00b1fc0 msgr2=0x7f99a0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:47.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f99a7816700 1 --2- 172.21.15.175:0/2779590026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00b1fc0 0x7f99a0155fb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f999c0098e0 tx=0x7f999c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f99a7816700 1 -- 172.21.15.175:0/2779590026 shutdown_connections 2023-10-10T19:09:47.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f99a7816700 1 --2- 172.21.15.175:0/2779590026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00b1fc0 0x7f99a0155fb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f99a7816700 1 -- 172.21.15.175:0/2779590026 >> 172.21.15.175:0/2779590026 conn(0x7f99a0063230 msgr2=0x7f99a0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:47.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f99a7816700 1 -- 172.21.15.175:0/2779590026 shutdown_connections 2023-10-10T19:09:47.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.513+0000 7f99a7816700 1 -- 172.21.15.175:0/2779590026 wait complete. 2023-10-10T19:09:47.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.514+0000 7f99a7816700 1 Processor -- start 2023-10-10T19:09:47.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.514+0000 7f99a7816700 1 -- start start 2023-10-10T19:09:47.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a7816700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00aff60 0x7f99a0167390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:47.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a7816700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99a00b1fc0 0x7f99a00cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:47.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a7816700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f99a00b2390 0x7f99a00ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:47.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a7816700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00cd4c0 con 0x7f99a00b2390 2023-10-10T19:09:47.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a7816700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00ca6d0 con 0x7f99a00aff60 2023-10-10T19:09:47.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a7816700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00ca970 con 0x7f99a00b1fc0 2023-10-10T19:09:47.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a5db3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f99a00b2390 0x7f99a00ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:47.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a5db3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f99a00b2390 0x7f99a00ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53010/0 (socket says 172.21.15.175:53010) 2023-10-10T19:09:47.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a5db3700 1 -- 172.21.15.175:0/3933655218 learned_addr learned my addr 172.21.15.175:0/3933655218 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:47.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a55b2700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00aff60 0x7f99a0167390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:47.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.515+0000 7f99a4db1700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99a00b1fc0 0x7f99a00cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:47.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99a5db3700 1 -- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99a00b1fc0 msgr2=0x7f99a00cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:47.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99a5db3700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99a00b1fc0 0x7f99a00cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99a5db3700 1 -- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00aff60 msgr2=0x7f99a0167390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:47.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99a5db3700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00aff60 0x7f99a0167390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99a5db3700 1 -- 172.21.15.175:0/3933655218 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c009420 con 0x7f99a00b2390 2023-10-10T19:09:47.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99a5db3700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f99a00b2390 0x7f99a00ccf80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f999800cd00 tx=0x7f9998001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:47.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99957fa700 1 -- 172.21.15.175:0/3933655218 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999800ce00 con 0x7f99a00b2390 2023-10-10T19:09:47.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99957fa700 1 -- 172.21.15.175:0/3933655218 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999800aac0 con 0x7f99a00b2390 2023-10-10T19:09:47.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a00cac50 con 0x7f99a00b2390 2023-10-10T19:09:47.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99957fa700 1 -- 172.21.15.175:0/3933655218 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999800ae80 con 0x7f99a00b2390 2023-10-10T19:09:47.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.516+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a00cb210 con 0x7f99a00b2390 2023-10-10T19:09:47.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.517+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a0151150 con 0x7f99a00b2390 2023-10-10T19:09:47.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.517+0000 7f99957fa700 1 -- 172.21.15.175:0/3933655218 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9998007300 con 0x7f99a00b2390 2023-10-10T19:09:47.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.518+0000 7f99957fa700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:47.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.518+0000 7f99957fa700 1 -- 172.21.15.175:0/3933655218 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f9998084dd0 con 0x7f99a00b2390 2023-10-10T19:09:47.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.518+0000 7f99a55b2700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:47.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.518+0000 7f99a55b2700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9984066f30 0x7f99840693e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f999c0093f0 tx=0x7f999c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:47.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.522+0000 7f99957fa700 1 -- 172.21.15.175:0/3933655218 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9998051bc0 con 0x7f99a00b2390 2023-10-10T19:09:47.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.755+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99a0067570 con 0x7f99a00b2390 2023-10-10T19:09:47.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.756+0000 7f99957fa700 1 -- 172.21.15.175:0/3933655218 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f99980551e0 con 0x7f99a00b2390 2023-10-10T19:09:47.756 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:47.757 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6373,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:39.671104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:47.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9984066f30 msgr2=0x7f99840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:47.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9984066f30 0x7f99840693e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f999c0093f0 tx=0x7f999c017000 comp rx=0 tx=0).stop 2023-10-10T19:09:47.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f99a00b2390 msgr2=0x7f99a00ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:47.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f99a00b2390 0x7f99a00ccf80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f999800cd00 tx=0x7f9998001ae0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 shutdown_connections 2023-10-10T19:09:47.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f99a00b2390 0x7f99a00ccf80 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99a00aff60 0x7f99a0167390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 --2- 172.21.15.175:0/3933655218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99a00b1fc0 0x7f99a00cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:47.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 >> 172.21.15.175:0/3933655218 conn(0x7f99a0063230 msgr2=0x7f99a014fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:47.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 shutdown_connections 2023-10-10T19:09:47.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:47.758+0000 7f99a7816700 1 -- 172.21.15.175:0/3933655218 wait complete. 2023-10-10T19:09:47.762 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 194 2023-10-10T19:09:47.770 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:09:49.771 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:49.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.898+0000 7f3f1fc3b700 1 Processor -- start 2023-10-10T19:09:49.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.898+0000 7f3f1fc3b700 1 -- start start 2023-10-10T19:09:49.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1fc3b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3f18155fb0 0x7f3f18159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:49.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1fc3b700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f181574b0 con 0x7f3f180b1fc0 2023-10-10T19:09:49.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1fc3b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f180afad0 con 0x7f3f180b2390 2023-10-10T19:09:49.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1fc3b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f18150770 con 0x7f3f18155fb0 2023-10-10T19:09:49.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1e1d8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3f18155fb0 0x7f3f18159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:49.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1e1d8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3f18155fb0 0x7f3f18159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53016/0 (socket says 172.21.15.175:53016) 2023-10-10T19:09:49.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1d9d7700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f3f180b1fc0 0x7f3f18068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44614/0 (socket says 172.21.15.175:44614) 2023-10-10T19:09:49.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1e1d8700 1 -- 172.21.15.175:0/3370046334 learned_addr learned my addr 172.21.15.175:0/3370046334 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:49.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1e1d8700 1 -- 172.21.15.175:0/3370046334 >> v1:172.21.15.191:6789/0 conn(0x7f3f180b2390 legacy=0x7f3f180aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:49.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1e1d8700 1 -- 172.21.15.175:0/3370046334 >> v1:172.21.15.175:6789/0 conn(0x7f3f180b1fc0 legacy=0x7f3f18068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:49.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1e1d8700 1 -- 172.21.15.175:0/3370046334 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180af4c0 con 0x7f3f18155fb0 2023-10-10T19:09:49.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1c9d5700 1 -- 172.21.15.175:0/3370046334 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214257376 0 0) 0x7f3f181574b0 con 0x7f3f180b1fc0 2023-10-10T19:09:49.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.899+0000 7f3f1e1d8700 1 --2- 172.21.15.175:0/3370046334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f18155fb0 0x7f3f18159920 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009f20 tx=0x7f3f10009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70eca83cbe8194b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:49.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1c9d5700 1 -- 172.21.15.175:0/3370046334 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1000e070 con 0x7f3f18155fb0 2023-10-10T19:09:49.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1c9d5700 1 -- 172.21.15.175:0/3370046334 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1000bb10 con 0x7f3f18155fb0 2023-10-10T19:09:49.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1c9d5700 1 -- 172.21.15.175:0/3370046334 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100092e0 con 0x7f3f18155fb0 2023-10-10T19:09:49.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3370046334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f18155fb0 msgr2=0x7f3f18159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:49.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1fc3b700 1 --2- 172.21.15.175:0/3370046334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f18155fb0 0x7f3f18159920 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009f20 tx=0x7f3f10009840 comp rx=0 tx=0).stop 2023-10-10T19:09:49.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3370046334 shutdown_connections 2023-10-10T19:09:49.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1fc3b700 1 --2- 172.21.15.175:0/3370046334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f18155fb0 0x7f3f18159920 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:49.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3370046334 >> 172.21.15.175:0/3370046334 conn(0x7f3f18063230 msgr2=0x7f3f18063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:49.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3370046334 shutdown_connections 2023-10-10T19:09:49.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.900+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3370046334 wait complete. 2023-10-10T19:09:49.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.901+0000 7f3f1fc3b700 1 Processor -- start 2023-10-10T19:09:49.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.901+0000 7f3f1fc3b700 1 -- start start 2023-10-10T19:09:49.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1fc3b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f180b1fc0 0x7f3f180cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:49.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1fc3b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f180b2390 0x7f3f180cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:49.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1fc3b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f18155fb0 0x7f3f18169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:49.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1fc3b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180688a0 con 0x7f3f180b2390 2023-10-10T19:09:49.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1fc3b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180c3870 con 0x7f3f18155fb0 2023-10-10T19:09:49.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1fc3b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f18169e00 con 0x7f3f180b1fc0 2023-10-10T19:09:49.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d1d6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f180b2390 0x7f3f180cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:49.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d1d6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f180b2390 0x7f3f180cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53022/0 (socket says 172.21.15.175:53022) 2023-10-10T19:09:49.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d1d6700 1 -- 172.21.15.175:0/3656309998 learned_addr learned my addr 172.21.15.175:0/3656309998 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:49.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1e1d8700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f18155fb0 0x7f3f18169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:49.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d9d7700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f180b1fc0 0x7f3f180cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:49.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d1d6700 1 -- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f180b1fc0 msgr2=0x7f3f180cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:49.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d1d6700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f180b1fc0 0x7f3f180cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:49.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d1d6700 1 -- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f18155fb0 msgr2=0x7f3f18169310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:49.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d1d6700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f18155fb0 0x7f3f18169310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:49.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.902+0000 7f3f1d1d6700 1 -- 172.21.15.175:0/3656309998 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f10009ae0 con 0x7f3f180b2390 2023-10-10T19:09:49.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.903+0000 7f3f1d1d6700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f180b2390 0x7f3f180cad10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800c590 tx=0x7f3f0800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:49.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.903+0000 7f3f0dffb700 1 -- 172.21.15.175:0/3656309998 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f08011590 con 0x7f3f180b2390 2023-10-10T19:09:49.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.903+0000 7f3f0dffb700 1 -- 172.21.15.175:0/3656309998 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f08011710 con 0x7f3f180b2390 2023-10-10T19:09:49.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.903+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f180c3b30 con 0x7f3f180b2390 2023-10-10T19:09:49.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.903+0000 7f3f0dffb700 1 -- 172.21.15.175:0/3656309998 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f08011ad0 con 0x7f3f180b2390 2023-10-10T19:09:49.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.903+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f180cbb50 con 0x7f3f180b2390 2023-10-10T19:09:49.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.904+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f180cbd70 con 0x7f3f180b2390 2023-10-10T19:09:49.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.904+0000 7f3f0dffb700 1 -- 172.21.15.175:0/3656309998 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3f0801b3b0 con 0x7f3f180b2390 2023-10-10T19:09:49.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.904+0000 7f3f0dffb700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3efc066f60 0x7f3efc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:49.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.905+0000 7f3f0dffb700 1 -- 172.21.15.175:0/3656309998 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f3f08083de0 con 0x7f3f180b2390 2023-10-10T19:09:49.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.905+0000 7f3f1d9d7700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3efc066f60 0x7f3efc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:49.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.905+0000 7f3f1d9d7700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3efc066f60 0x7f3efc069410 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3f14007fd0 tx=0x7f3f14007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:49.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:49.909+0000 7f3f0dffb700 1 -- 172.21.15.175:0/3656309998 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3f08050bd0 con 0x7f3f180b2390 2023-10-10T19:09:50.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.137+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f18067570 con 0x7f3f180b2390 2023-10-10T19:09:50.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.137+0000 7f3f0dffb700 1 -- 172.21.15.175:0/3656309998 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f3f08017070 con 0x7f3f180b2390 2023-10-10T19:09:50.138 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:50.138 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6373,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:39.671104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:50.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3efc066f60 msgr2=0x7f3efc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:50.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3efc066f60 0x7f3efc069410 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3f14007fd0 tx=0x7f3f14007d30 comp rx=0 tx=0).stop 2023-10-10T19:09:50.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f180b2390 msgr2=0x7f3f180cad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:50.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f180b2390 0x7f3f180cad10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800c590 tx=0x7f3f0800c8a0 comp rx=0 tx=0).stop 2023-10-10T19:09:50.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 shutdown_connections 2023-10-10T19:09:50.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3efc066f60 0x7f3efc069410 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:50.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f18155fb0 0x7f3f18169310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:50.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f180b1fc0 0x7f3f180cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:50.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 --2- 172.21.15.175:0/3656309998 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3f180b2390 0x7f3f180cad10 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:50.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 >> 172.21.15.175:0/3656309998 conn(0x7f3f18063230 msgr2=0x7f3f18151700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:50.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.139+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 shutdown_connections 2023-10-10T19:09:50.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:50.140+0000 7f3f1fc3b700 1 -- 172.21.15.175:0/3656309998 wait complete. 2023-10-10T19:09:50.143 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 194 2023-10-10T19:09:50.151 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:09:52.152 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:52.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:09:52.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:09:52.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.283+0000 7ff28a834700 1 Processor -- start 2023-10-10T19:09:52.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.283+0000 7ff28a834700 1 -- start start 2023-10-10T19:09:52.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.284+0000 7ff28a834700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff2840c1210 0x7ff2840bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.284+0000 7ff28a834700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 0x7ff2840bf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.284+0000 7ff28a834700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28406b740 con 0x7ff28414e140 2023-10-10T19:09:52.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.284+0000 7ff28a834700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2840c0960 con 0x7ff2840c1210 2023-10-10T19:09:52.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.284+0000 7ff28a834700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28406ca30 con 0x7ff28406edb0 2023-10-10T19:09:52.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.284+0000 7ff283fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff28414e140 0x7ff284155270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44616/0 (socket says 172.21.15.175:44616) 2023-10-10T19:09:52.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.284+0000 7ff283fff700 1 -- 172.21.15.175:0/1559802899 learned_addr learned my addr 172.21.15.175:0/1559802899 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:09:52.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff288dd1700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 0x7ff2840bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291889086 0 0) 0x7ff28406b740 con 0x7ff28414e140 2023-10-10T19:09:52.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff2837fe700 1 --2- 172.21.15.175:0/1559802899 >> v2:172.21.15.191:3300/0 conn(0x7ff2840c1210 0x7ff2840bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c003570 con 0x7ff28414e140 2023-10-10T19:09:52.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2135400473 0 0) 0x7ff26c003570 con 0x7ff28414e140 2023-10-10T19:09:52.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 msgr2=0x7ff2840bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:52.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 0x7ff2840bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 >> v2:172.21.15.191:3300/0 conn(0x7ff2840c1210 msgr2=0x7ff2840bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:52.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 --2- 172.21.15.175:0/1559802899 >> v2:172.21.15.191:3300/0 conn(0x7ff2840c1210 0x7ff2840bf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28406c5a0 con 0x7ff28414e140 2023-10-10T19:09:52.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7ff274001470 con 0x7ff28414e140 2023-10-10T19:09:52.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 >> v1:172.21.15.175:6789/0 conn(0x7ff28414e140 legacy=0x7ff284155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:52.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff26c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff26c004810 0x7ff26c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 0x7ff26c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2840c0960 con 0x7ff26c003a90 2023-10-10T19:09:52.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28406ca30 con 0x7ff26c004810 2023-10-10T19:09:52.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c00ba60 con 0x7ff28406edb0 2023-10-10T19:09:52.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff283fff700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff26c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff2837fe700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff26c004810 0x7ff26c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff288dd1700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 0x7ff26c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff283fff700 1 -- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 msgr2=0x7ff26c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:52.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff283fff700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 0x7ff26c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff283fff700 1 -- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff26c004810 msgr2=0x7ff26c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:52.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff283fff700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff26c004810 0x7ff26c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff283fff700 1 -- 172.21.15.175:0/1559802899 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c003570 con 0x7ff26c003a90 2023-10-10T19:09:52.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.285+0000 7ff288dd1700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff28406edb0 0x7ff26c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:09:52.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff283fff700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff26c003ea0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff27400c3f0 tx=0x7ff274001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f1931fe21e3d5c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:52.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27400cc80 con 0x7ff26c003a90 2023-10-10T19:09:52.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27400ce00 con 0x7ff26c003a90 2023-10-10T19:09:52.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff282ffd700 1 -- 172.21.15.175:0/1559802899 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274005b70 con 0x7ff26c003a90 2023-10-10T19:09:52.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff28a834700 1 -- 172.21.15.175:0/1559802899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 msgr2=0x7ff26c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:52.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff28a834700 1 --2- 172.21.15.175:0/1559802899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff26c003ea0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff27400c3f0 tx=0x7ff274001e50 comp rx=0 tx=0).stop 2023-10-10T19:09:52.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff288dd1700 1 -- 172.21.15.175:0/1559802899 reap_dead start 2023-10-10T19:09:52.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff28a834700 1 -- 172.21.15.175:0/1559802899 shutdown_connections 2023-10-10T19:09:52.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff28a834700 1 -- 172.21.15.175:0/1559802899 >> 172.21.15.175:0/1559802899 conn(0x7ff284063cb0 msgr2=0x7ff28406d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:52.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff28a834700 1 -- 172.21.15.175:0/1559802899 shutdown_connections 2023-10-10T19:09:52.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.286+0000 7ff28a834700 1 -- 172.21.15.175:0/1559802899 wait complete. 2023-10-10T19:09:52.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.287+0000 7ff28a834700 1 Processor -- start 2023-10-10T19:09:52.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff28a834700 1 -- start start 2023-10-10T19:09:52.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff28a834700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff28414f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff28a834700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2840c39b0 0x7ff28414fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff28a834700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff284147040 0x7ff2841474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff28a834700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff284164730 con 0x7ff26c003a90 2023-10-10T19:09:52.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff28a834700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28414f330 con 0x7ff2840c39b0 2023-10-10T19:09:52.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff28a834700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2841479f0 con 0x7ff284147040 2023-10-10T19:09:52.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff283fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff28414f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff283fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff28414f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53032/0 (socket says 172.21.15.175:53032) 2023-10-10T19:09:52.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff283fff700 1 -- 172.21.15.175:0/1702580679 learned_addr learned my addr 172.21.15.175:0/1702580679 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:52.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff288dd1700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff284147040 0x7ff2841474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.288+0000 7ff2837fe700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2840c39b0 0x7ff28414fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff283fff700 1 -- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff284147040 msgr2=0x7ff2841474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:52.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff283fff700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff284147040 0x7ff2841474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff283fff700 1 -- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2840c39b0 msgr2=0x7ff28414fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:52.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff283fff700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2840c39b0 0x7ff28414fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff283fff700 1 -- 172.21.15.175:0/1702580679 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284164470 con 0x7ff26c003a90 2023-10-10T19:09:52.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff283fff700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff28414f7c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff26c004700 tx=0x7ff274015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:52.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff26bfff700 1 -- 172.21.15.175:0/1702580679 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27400cc80 con 0x7ff26c003a90 2023-10-10T19:09:52.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff26bfff700 1 -- 172.21.15.175:0/1702580679 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2740072e0 con 0x7ff26c003a90 2023-10-10T19:09:52.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff26bfff700 1 -- 172.21.15.175:0/1702580679 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274007680 con 0x7ff26c003a90 2023-10-10T19:09:52.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28414ea00 con 0x7ff26c003a90 2023-10-10T19:09:52.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.289+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28414efc0 con 0x7ff26c003a90 2023-10-10T19:09:52.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.290+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff284067570 con 0x7ff26c003a90 2023-10-10T19:09:52.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.290+0000 7ff26bfff700 1 -- 172.21.15.175:0/1702580679 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff274019070 con 0x7ff26c003a90 2023-10-10T19:09:52.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.291+0000 7ff26bfff700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff270066f60 0x7ff270069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:52.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.291+0000 7ff26bfff700 1 -- 172.21.15.175:0/1702580679 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6853+0+0 (secure 0 0 0) 0x7ff274085cd0 con 0x7ff26c003a90 2023-10-10T19:09:52.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.291+0000 7ff2837fe700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff270066f60 0x7ff270069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:52.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.291+0000 7ff2837fe700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff270066f60 0x7ff270069410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff284165420 tx=0x7ff2780091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:52.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.295+0000 7ff26bfff700 1 -- 172.21.15.175:0/1702580679 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff274052ac0 con 0x7ff26c003a90 2023-10-10T19:09:52.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.521+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff284164730 con 0x7ff26c003a90 2023-10-10T19:09:52.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.522+0000 7ff26bfff700 1 -- 172.21.15.175:0/1702580679 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6202 (secure 0 0 0) 0x7ff274022090 con 0x7ff26c003a90 2023-10-10T19:09:52.522 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:52.523 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6373,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:39.671104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"d","rank":4,"incarnation":100,"state":"up:active","state_seq":35,"addr":"172.21.15.191:6837/824552134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":824552134},{"type":"v1","addr":"172.21.15.191:6837","nonce":824552134}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:52.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff270066f60 msgr2=0x7ff270069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:52.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff270066f60 0x7ff270069410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff284165420 tx=0x7ff2780091a0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 msgr2=0x7ff28414f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:52.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff28414f7c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff26c004700 tx=0x7ff274015000 comp rx=0 tx=0).stop 2023-10-10T19:09:52.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 shutdown_connections 2023-10-10T19:09:52.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff270066f60 0x7ff270069410 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff284147040 0x7ff2841474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff26c003a90 0x7ff28414f7c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 --2- 172.21.15.175:0/1702580679 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2840c39b0 0x7ff28414fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:52.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 >> 172.21.15.175:0/1702580679 conn(0x7ff284063cb0 msgr2=0x7ff2840c4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:52.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 shutdown_connections 2023-10-10T19:09:52.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:52.524+0000 7ff28a834700 1 -- 172.21.15.175:0/1702580679 wait complete. 2023-10-10T19:09:52.527 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 194 2023-10-10T19:09:52.536 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:09:54.537 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:54.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.662+0000 7f3070230700 1 Processor -- start 2023-10-10T19:09:54.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.662+0000 7f3070230700 1 -- start start 2023-10-10T19:09:54.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.663+0000 7f3070230700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f30681573d0 0x7f30680c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:54.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.663+0000 7f3070230700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30680c4f20 0x7f306814d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:54.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.663+0000 7f3070230700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3068154aa0 con 0x7f30681577a0 2023-10-10T19:09:54.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.663+0000 7f3070230700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30680bfdf0 con 0x7f30681573d0 2023-10-10T19:09:54.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.663+0000 7f3070230700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3068063dc0 con 0x7f30680c4f20 2023-10-10T19:09:54.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.663+0000 7f306dfcc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f30681573d0 0x7f30680c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:54.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.663+0000 7f306dfcc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f30681573d0 0x7f30680c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39222/0 (socket says 172.21.15.175:39222) 2023-10-10T19:09:54.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.663+0000 7f306dfcc700 1 -- 172.21.15.175:0/2126089118 learned_addr learned my addr 172.21.15.175:0/2126089118 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:54.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306e7cd700 1 --2- 172.21.15.175:0/2126089118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30680c4f20 0x7f306814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:54.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306dfcc700 1 -- 172.21.15.175:0/2126089118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30680c4f20 msgr2=0x7f306814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:54.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306dfcc700 1 --2- 172.21.15.175:0/2126089118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30680c4f20 0x7f306814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306dfcc700 1 -- 172.21.15.175:0/2126089118 >> v1:172.21.15.191:6789/0 conn(0x7f30681577a0 legacy=0x7f30680bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:54.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306dfcc700 1 -- 172.21.15.175:0/2126089118 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068157c20 con 0x7f30681573d0 2023-10-10T19:09:54.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306e7cd700 1 --2- 172.21.15.175:0/2126089118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30680c4f20 0x7f306814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:09:54.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306dfcc700 1 --2- 172.21.15.175:0/2126089118 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f30680c12a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3064009f20 tx=0x7f3064009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc528b10ad09badf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:54.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306cfca700 1 -- 172.21.15.175:0/2126089118 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306400e070 con 0x7f30681573d0 2023-10-10T19:09:54.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306cfca700 1 -- 172.21.15.175:0/2126089118 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306400ba40 con 0x7f30681573d0 2023-10-10T19:09:54.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.664+0000 7f306cfca700 1 -- 172.21.15.175:0/2126089118 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30640092e0 con 0x7f30681573d0 2023-10-10T19:09:54.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.665+0000 7f3070230700 1 -- 172.21.15.175:0/2126089118 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 msgr2=0x7f30680c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:54.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.665+0000 7f3070230700 1 --2- 172.21.15.175:0/2126089118 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f30680c12a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3064009f20 tx=0x7f3064009840 comp rx=0 tx=0).stop 2023-10-10T19:09:54.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.665+0000 7f3070230700 1 -- 172.21.15.175:0/2126089118 shutdown_connections 2023-10-10T19:09:54.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.665+0000 7f3070230700 1 --2- 172.21.15.175:0/2126089118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30680c4f20 0x7f306814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.665+0000 7f3070230700 1 --2- 172.21.15.175:0/2126089118 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f30680c12a0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.665+0000 7f3070230700 1 -- 172.21.15.175:0/2126089118 >> 172.21.15.175:0/2126089118 conn(0x7f306806dc70 msgr2=0x7f3068063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:54.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.665+0000 7f3070230700 1 -- 172.21.15.175:0/2126089118 shutdown_connections 2023-10-10T19:09:54.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.665+0000 7f3070230700 1 -- 172.21.15.175:0/2126089118 wait complete. 2023-10-10T19:09:54.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.666+0000 7f3070230700 1 Processor -- start 2023-10-10T19:09:54.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.666+0000 7f3070230700 1 -- start start 2023-10-10T19:09:54.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f3070230700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30680c4f20 0x7f30680c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:54.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f3070230700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f3068151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:54.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f3070230700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30681577a0 0x7f30681532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:54.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f3070230700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680ca890 con 0x7f30681573d0 2023-10-10T19:09:54.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f3070230700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680cb360 con 0x7f30680c4f20 2023-10-10T19:09:54.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f3070230700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3068152e60 con 0x7f30681577a0 2023-10-10T19:09:54.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f3068151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:54.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f3068151c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39234/0 (socket says 172.21.15.175:39234) 2023-10-10T19:09:54.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 -- 172.21.15.175:0/1621195660 learned_addr learned my addr 172.21.15.175:0/1621195660 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:54.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306dfcc700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30680c4f20 0x7f30680c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:54.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306e7cd700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30681577a0 0x7f30681532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:54.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 -- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30681577a0 msgr2=0x7f30681532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:54.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30681577a0 0x7f30681532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 -- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30680c4f20 msgr2=0x7f30680c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:54.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30680c4f20 0x7f30680c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 -- 172.21.15.175:0/1621195660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3064009ae0 con 0x7f30681573d0 2023-10-10T19:09:54.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f306d7cb700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f3068151c40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f305800c600 tx=0x7f3058007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:54.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f305dffb700 1 -- 172.21.15.175:0/1621195660 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058011590 con 0x7f30681573d0 2023-10-10T19:09:54.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f305dffb700 1 -- 172.21.15.175:0/1621195660 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3058011710 con 0x7f30681573d0 2023-10-10T19:09:54.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30680caae0 con 0x7f30681573d0 2023-10-10T19:09:54.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.667+0000 7f305dffb700 1 -- 172.21.15.175:0/1621195660 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058011ad0 con 0x7f30681573d0 2023-10-10T19:09:54.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.668+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30680cb080 con 0x7f30681573d0 2023-10-10T19:09:54.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.669+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3068067570 con 0x7f30681573d0 2023-10-10T19:09:54.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.669+0000 7f305dffb700 1 -- 172.21.15.175:0/1621195660 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f305801c460 con 0x7f30681573d0 2023-10-10T19:09:54.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.669+0000 7f305dffb700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f304c066f60 0x7f304c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:54.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.669+0000 7f305dffb700 1 -- 172.21.15.175:0/1621195660 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f30580847e0 con 0x7f30681573d0 2023-10-10T19:09:54.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.669+0000 7f306dfcc700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f304c066f60 0x7f304c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:54.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.670+0000 7f306dfcc700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f304c066f60 0x7f304c069410 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3064009880 tx=0x7f3064017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:54.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.674+0000 7f305dffb700 1 -- 172.21.15.175:0/1621195660 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3058051580 con 0x7f30681573d0 2023-10-10T19:09:54.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.899+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3068152fa0 con 0x7f30681573d0 2023-10-10T19:09:54.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.900+0000 7f305dffb700 1 -- 172.21.15.175:0/1621195660 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5439 (secure 0 0 0) 0x7f3058054ba0 con 0x7f30681573d0 2023-10-10T19:09:54.900 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:54.901 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:53.672616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:54.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f304c066f60 msgr2=0x7f304c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:54.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f304c066f60 0x7f304c069410 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3064009880 tx=0x7f3064017000 comp rx=0 tx=0).stop 2023-10-10T19:09:54.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 msgr2=0x7f3068151c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:54.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f3068151c40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f305800c600 tx=0x7f3058007f60 comp rx=0 tx=0).stop 2023-10-10T19:09:54.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 shutdown_connections 2023-10-10T19:09:54.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f304c066f60 0x7f304c069410 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30681577a0 0x7f30681532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30680c4f20 0x7f30680c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 --2- 172.21.15.175:0/1621195660 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30681573d0 0x7f3068151c40 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:54.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 >> 172.21.15.175:0/1621195660 conn(0x7f306806dc70 msgr2=0x7f30680c6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:54.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 shutdown_connections 2023-10-10T19:09:54.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:54.902+0000 7f3070230700 1 -- 172.21.15.175:0/1621195660 wait complete. 2023-10-10T19:09:54.905 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 195 2023-10-10T19:09:54.914 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-10T19:09:54.914 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:09:54.915 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:55.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.030+0000 7f376359e700 1 Processor -- start 2023-10-10T19:09:55.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.030+0000 7f376359e700 1 -- start start 2023-10-10T19:09:55.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f376359e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f376408a100 0x7f3764159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:55.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f376359e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37640cd040 0x7f37640684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:55.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f376359e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3764067fa0 con 0x7f376414fd90 2023-10-10T19:09:55.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f376359e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640c5600 con 0x7f37640cd040 2023-10-10T19:09:55.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f376359e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3764156b70 con 0x7f376408a100 2023-10-10T19:09:55.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3762d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37640cd040 0x7f37640684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:55.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3762d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37640cd040 0x7f37640684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39236/0 (socket says 172.21.15.175:39236) 2023-10-10T19:09:55.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3762d9d700 1 -- 172.21.15.175:0/199107722 learned_addr learned my addr 172.21.15.175:0/199107722 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:55.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3761d9b700 1 --2- 172.21.15.175:0/199107722 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f376408a100 0x7f3764159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:55.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3762d9d700 1 -- 172.21.15.175:0/199107722 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f376408a100 msgr2=0x7f3764159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:55.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3762d9d700 1 --2- 172.21.15.175:0/199107722 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f376408a100 0x7f3764159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3762d9d700 1 -- 172.21.15.175:0/199107722 >> v1:172.21.15.191:6789/0 conn(0x7f376414fd90 legacy=0x7f37640c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:55.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3762d9d700 1 -- 172.21.15.175:0/199107722 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3764150160 con 0x7f37640cd040 2023-10-10T19:09:55.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3761d9b700 1 --2- 172.21.15.175:0/199107722 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f376408a100 0x7f3764159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:09:55.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f3762d9d700 1 --2- 172.21.15.175:0/199107722 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37640cd040 0x7f37640684f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f375c009f20 tx=0x7f375c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6c4a865770bc29 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:55.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f376159a700 1 -- 172.21.15.175:0/199107722 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c00e070 con 0x7f37640cd040 2023-10-10T19:09:55.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f376159a700 1 -- 172.21.15.175:0/199107722 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c00ba80 con 0x7f37640cd040 2023-10-10T19:09:55.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.031+0000 7f376159a700 1 -- 172.21.15.175:0/199107722 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c0092e0 con 0x7f37640cd040 2023-10-10T19:09:55.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.032+0000 7f376359e700 1 -- 172.21.15.175:0/199107722 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37640cd040 msgr2=0x7f37640684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:55.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.032+0000 7f376359e700 1 --2- 172.21.15.175:0/199107722 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37640cd040 0x7f37640684f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f375c009f20 tx=0x7f375c009840 comp rx=0 tx=0).stop 2023-10-10T19:09:55.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.032+0000 7f376359e700 1 -- 172.21.15.175:0/199107722 shutdown_connections 2023-10-10T19:09:55.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.032+0000 7f376359e700 1 --2- 172.21.15.175:0/199107722 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37640cd040 0x7f37640684f0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.032+0000 7f376359e700 1 --2- 172.21.15.175:0/199107722 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f376408a100 0x7f3764159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.032+0000 7f376359e700 1 -- 172.21.15.175:0/199107722 >> 172.21.15.175:0/199107722 conn(0x7f3764063d00 msgr2=0x7f37640cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:55.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.032+0000 7f376359e700 1 -- 172.21.15.175:0/199107722 shutdown_connections 2023-10-10T19:09:55.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.032+0000 7f376359e700 1 -- 172.21.15.175:0/199107722 wait complete. 2023-10-10T19:09:55.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.033+0000 7f376359e700 1 Processor -- start 2023-10-10T19:09:55.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.033+0000 7f376359e700 1 -- start start 2023-10-10T19:09:55.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f376359e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f376408a100 0x7f37640a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:55.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f376359e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37640cd040 0x7f3764165550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:55.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f376359e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f376414fd90 0x7f3764153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:55.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f376359e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3764165a90 con 0x7f376414fd90 2023-10-10T19:09:55.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f376359e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640c7cf0 con 0x7f376408a100 2023-10-10T19:09:55.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f376359e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640c7f90 con 0x7f37640cd040 2023-10-10T19:09:55.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f376414fd90 0x7f3764153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:55.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f376414fd90 0x7f3764153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39240/0 (socket says 172.21.15.175:39240) 2023-10-10T19:09:55.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 -- 172.21.15.175:0/1980234330 learned_addr learned my addr 172.21.15.175:0/1980234330 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:55.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f376259c700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f376408a100 0x7f37640a7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:55.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3761d9b700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37640cd040 0x7f3764165550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:55.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 -- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37640cd040 msgr2=0x7f3764165550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:55.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37640cd040 0x7f3764165550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 -- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f376408a100 msgr2=0x7f37640a7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f376408a100 0x7f37640a7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 -- 172.21.15.175:0/1980234330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c009ae0 con 0x7f376414fd90 2023-10-10T19:09:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f376259c700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f376408a100 0x7f37640a7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:09:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3761d9b700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37640cd040 0x7f3764165550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:09:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.034+0000 7f3762d9d700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f376414fd90 0x7f3764153ca0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f3764064c20 tx=0x7f375c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.035+0000 7f37527fc700 1 -- 172.21.15.175:0/1980234330 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c0032f0 con 0x7f376414fd90 2023-10-10T19:09:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.035+0000 7f37527fc700 1 -- 172.21.15.175:0/1980234330 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c008e80 con 0x7f376414fd90 2023-10-10T19:09:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.035+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37640c8500 con 0x7f376414fd90 2023-10-10T19:09:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.035+0000 7f37527fc700 1 -- 172.21.15.175:0/1980234330 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c0266b0 con 0x7f376414fd90 2023-10-10T19:09:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.035+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37640c92e0 con 0x7f376414fd90 2023-10-10T19:09:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.036+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3764067570 con 0x7f376414fd90 2023-10-10T19:09:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.036+0000 7f37527fc700 1 -- 172.21.15.175:0/1980234330 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f375c003040 con 0x7f376414fd90 2023-10-10T19:09:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.036+0000 7f37527fc700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3744066f30 0x7f37440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.036+0000 7f37527fc700 1 -- 172.21.15.175:0/1980234330 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f375c086760 con 0x7f376414fd90 2023-10-10T19:09:55.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.037+0000 7f376259c700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3744066f30 0x7f37440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:55.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.037+0000 7f376259c700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3744066f30 0x7f37440693e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3754007fd0 tx=0x7f3754009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:55.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.040+0000 7f37527fc700 1 -- 172.21.15.175:0/1980234330 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f375c0535b0 con 0x7f376414fd90 2023-10-10T19:09:55.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.268+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37640cce20 con 0x7f376414fd90 2023-10-10T19:09:55.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.269+0000 7f37527fc700 1 -- 172.21.15.175:0/1980234330 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5439 (secure 0 0 0) 0x7f375c056bd0 con 0x7f376414fd90 2023-10-10T19:09:55.269 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:55.270 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:53.672616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:55.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3744066f30 msgr2=0x7f37440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:55.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3744066f30 0x7f37440693e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3754007fd0 tx=0x7f3754009040 comp rx=0 tx=0).stop 2023-10-10T19:09:55.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f376414fd90 msgr2=0x7f3764153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:55.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f376414fd90 0x7f3764153ca0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f3764064c20 tx=0x7f375c018000 comp rx=0 tx=0).stop 2023-10-10T19:09:55.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 shutdown_connections 2023-10-10T19:09:55.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3744066f30 0x7f37440693e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f376414fd90 0x7f3764153ca0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f376408a100 0x7f37640a7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 --2- 172.21.15.175:0/1980234330 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37640cd040 0x7f3764165550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:55.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 >> 172.21.15.175:0/1980234330 conn(0x7f3764063d00 msgr2=0x7f37640cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:55.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 shutdown_connections 2023-10-10T19:09:55.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:55.271+0000 7f376359e700 1 -- 172.21.15.175:0/1980234330 wait complete. 2023-10-10T19:09:55.274 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 195 2023-10-10T19:09:55.283 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 195, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 195, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:09:53.672616+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 6106, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:09:57.285 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:57.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.412+0000 7f34dc21f700 1 Processor -- start 2023-10-10T19:09:57.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.412+0000 7f34dc21f700 1 -- start start 2023-10-10T19:09:57.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34dc21f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40b2390 0x7f34d4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:57.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34dc21f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34d40aff60 0x7f34d40b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:57.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34dc21f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d41574b0 con 0x7f34d40b1fc0 2023-10-10T19:09:57.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34dc21f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d4150770 con 0x7f34d40aff60 2023-10-10T19:09:57.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34dc21f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40afad0 con 0x7f34d40b2390 2023-10-10T19:09:57.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34da7bc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34d40aff60 0x7f34d40b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:57.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34da7bc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34d40aff60 0x7f34d40b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39252/0 (socket says 172.21.15.175:39252) 2023-10-10T19:09:57.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34da7bc700 1 -- 172.21.15.175:0/4047495341 learned_addr learned my addr 172.21.15.175:0/4047495341 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:57.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34d97ba700 1 --2- 172.21.15.175:0/4047495341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40b2390 0x7f34d4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:57.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34da7bc700 1 -- 172.21.15.175:0/4047495341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40b2390 msgr2=0x7f34d4155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:57.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34da7bc700 1 --2- 172.21.15.175:0/4047495341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40b2390 0x7f34d4155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34da7bc700 1 -- 172.21.15.175:0/4047495341 >> v1:172.21.15.191:6789/0 conn(0x7f34d40b1fc0 legacy=0x7f34d4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:57.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34da7bc700 1 -- 172.21.15.175:0/4047495341 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d40af4c0 con 0x7f34d40aff60 2023-10-10T19:09:57.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34da7bc700 1 --2- 172.21.15.175:0/4047495341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40aff60 0x7f34d40b03e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009f50 tx=0x7f34cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4423e1868ec850df server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:57.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.413+0000 7f34d8fb9700 1 -- 172.21.15.175:0/4047495341 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00e070 con 0x7f34d40aff60 2023-10-10T19:09:57.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34d8fb9700 1 -- 172.21.15.175:0/4047495341 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc00ba50 con 0x7f34d40aff60 2023-10-10T19:09:57.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34d8fb9700 1 -- 172.21.15.175:0/4047495341 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc0092e0 con 0x7f34d40aff60 2023-10-10T19:09:57.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34dc21f700 1 -- 172.21.15.175:0/4047495341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40aff60 msgr2=0x7f34d40b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:57.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34dc21f700 1 --2- 172.21.15.175:0/4047495341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40aff60 0x7f34d40b03e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009f50 tx=0x7f34cc009840 comp rx=0 tx=0).stop 2023-10-10T19:09:57.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34dc21f700 1 -- 172.21.15.175:0/4047495341 shutdown_connections 2023-10-10T19:09:57.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34dc21f700 1 --2- 172.21.15.175:0/4047495341 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40aff60 0x7f34d40b03e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34dc21f700 1 --2- 172.21.15.175:0/4047495341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40b2390 0x7f34d4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34dc21f700 1 -- 172.21.15.175:0/4047495341 >> 172.21.15.175:0/4047495341 conn(0x7f34d4063230 msgr2=0x7f34d4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:57.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34dc21f700 1 -- 172.21.15.175:0/4047495341 shutdown_connections 2023-10-10T19:09:57.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.414+0000 7f34dc21f700 1 -- 172.21.15.175:0/4047495341 wait complete. 2023-10-10T19:09:57.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.415+0000 7f34dc21f700 1 Processor -- start 2023-10-10T19:09:57.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.415+0000 7f34dc21f700 1 -- start start 2023-10-10T19:09:57.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34dc21f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40aff60 0x7f34d40c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:57.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34dc21f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d40b1fc0 0x7f34d40c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:57.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34dc21f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40b2390 0x7f34d4169550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:57.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34dc21f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40c4500 con 0x7f34d40b2390 2023-10-10T19:09:57.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34dc21f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4158290 con 0x7f34d40b1fc0 2023-10-10T19:09:57.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34dc21f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40c3910 con 0x7f34d40aff60 2023-10-10T19:09:57.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34da7bc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40b2390 0x7f34d4169550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:57.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34d97ba700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d40b1fc0 0x7f34d40c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:57.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34da7bc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40b2390 0x7f34d4169550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39264/0 (socket says 172.21.15.175:39264) 2023-10-10T19:09:57.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34da7bc700 1 -- 172.21.15.175:0/1424098754 learned_addr learned my addr 172.21.15.175:0/1424098754 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:57.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34d9fbb700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40aff60 0x7f34d40c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:57.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34da7bc700 1 -- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40aff60 msgr2=0x7f34d40c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:57.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34da7bc700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40aff60 0x7f34d40c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34da7bc700 1 -- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d40b1fc0 msgr2=0x7f34d40c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:57.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34da7bc700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d40b1fc0 0x7f34d40c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.416+0000 7f34da7bc700 1 -- 172.21.15.175:0/1424098754 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc009ae0 con 0x7f34d40b2390 2023-10-10T19:09:57.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.417+0000 7f34da7bc700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40b2390 0x7f34d4169550 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f34d406fd90 tx=0x7f34cc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:57.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.417+0000 7f34c9ffb700 1 -- 172.21.15.175:0/1424098754 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc008ce0 con 0x7f34d40b2390 2023-10-10T19:09:57.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.417+0000 7f34c9ffb700 1 -- 172.21.15.175:0/1424098754 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc008e80 con 0x7f34d40b2390 2023-10-10T19:09:57.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.417+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d40c3bd0 con 0x7f34d40b2390 2023-10-10T19:09:57.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.417+0000 7f34c9ffb700 1 -- 172.21.15.175:0/1424098754 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc01d670 con 0x7f34d40b2390 2023-10-10T19:09:57.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.417+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d40c4190 con 0x7f34d40b2390 2023-10-10T19:09:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.418+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d40c4670 con 0x7f34d40b2390 2023-10-10T19:09:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.418+0000 7f34c9ffb700 1 -- 172.21.15.175:0/1424098754 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f34cc003040 con 0x7f34d40b2390 2023-10-10T19:09:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.419+0000 7f34c9ffb700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34b8066f30 0x7f34b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.419+0000 7f34c9ffb700 1 -- 172.21.15.175:0/1424098754 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f34cc0865b0 con 0x7f34d40b2390 2023-10-10T19:09:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.419+0000 7f34d9fbb700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34b8066f30 0x7f34b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:57.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.419+0000 7f34d9fbb700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34b8066f30 0x7f34b80693e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f34d0003a80 tx=0x7f34d0007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:57.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.423+0000 7f34c9ffb700 1 -- 172.21.15.175:0/1424098754 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34cc053350 con 0x7f34d40b2390 2023-10-10T19:09:57.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.649+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34d4067570 con 0x7f34d40b2390 2023-10-10T19:09:57.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.649+0000 7f34c9ffb700 1 -- 172.21.15.175:0/1424098754 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5439 (secure 0 0 0) 0x7f34cc056970 con 0x7f34d40b2390 2023-10-10T19:09:57.650 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:09:57.650 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:53.672616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:09:57.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34b8066f30 msgr2=0x7f34b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:57.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34b8066f30 0x7f34b80693e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f34d0003a80 tx=0x7f34d0007ec0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40b2390 msgr2=0x7f34d4169550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:57.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40b2390 0x7f34d4169550 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f34d406fd90 tx=0x7f34cc018000 comp rx=0 tx=0).stop 2023-10-10T19:09:57.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 shutdown_connections 2023-10-10T19:09:57.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34b8066f30 0x7f34b80693e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d40b2390 0x7f34d4169550 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d40aff60 0x7f34d40c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 --2- 172.21.15.175:0/1424098754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d40b1fc0 0x7f34d40c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:57.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 >> 172.21.15.175:0/1424098754 conn(0x7f34d4063230 msgr2=0x7f34d414fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:57.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 shutdown_connections 2023-10-10T19:09:57.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:57.652+0000 7f34dc21f700 1 -- 172.21.15.175:0/1424098754 wait complete. 2023-10-10T19:09:57.655 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 195 2023-10-10T19:09:57.664 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:09:59.665 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:09:59.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.788+0000 7f91f1a22700 1 Processor -- start 2023-10-10T19:09:59.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91f1a22700 1 -- start start 2023-10-10T19:09:59.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91f1a22700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f91ec14fbf0 0x7f91ec159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:59.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91f1a22700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f91ec0681e0 0x7f91ec0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:59.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91f1a22700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec156800 con 0x7f91ec14ffc0 2023-10-10T19:09:59.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91f1a22700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec0c59e0 con 0x7f91ec14fbf0 2023-10-10T19:09:59.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91f1a22700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec15a2b0 con 0x7f91ec0681e0 2023-10-10T19:09:59.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91eaffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f91ec14fbf0 0x7f91ec159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:59.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91eaffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f91ec14fbf0 0x7f91ec159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39266/0 (socket says 172.21.15.175:39266) 2023-10-10T19:09:59.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91eaffd700 1 -- 172.21.15.175:0/2266037369 learned_addr learned my addr 172.21.15.175:0/2266037369 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:59.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.789+0000 7f91eb7fe700 1 --2- 172.21.15.175:0/2266037369 >> v2:172.21.15.191:3300/0 conn(0x7f91ec0681e0 0x7f91ec0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:59.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91eaffd700 1 -- 172.21.15.175:0/2266037369 >> v2:172.21.15.191:3300/0 conn(0x7f91ec0681e0 msgr2=0x7f91ec0c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:09:59.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91eaffd700 1 --2- 172.21.15.175:0/2266037369 >> v2:172.21.15.191:3300/0 conn(0x7f91ec0681e0 0x7f91ec0c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:59.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91eaffd700 1 -- 172.21.15.175:0/2266037369 >> v1:172.21.15.191:6789/0 conn(0x7f91ec14ffc0 legacy=0x7f91ec063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:59.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91ea7fc700 1 -- 172.21.15.175:0/2266037369 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437564811 0 0) 0x7f91ec156800 con 0x7f91ec14ffc0 2023-10-10T19:09:59.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91eaffd700 1 -- 172.21.15.175:0/2266037369 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec063e30 con 0x7f91ec14fbf0 2023-10-10T19:09:59.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91eaffd700 1 --2- 172.21.15.175:0/2266037369 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec159d70 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0cd790 tx=0x7f91dc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6c12d9f0e6718d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:59.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91ea7fc700 1 -- 172.21.15.175:0/2266037369 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc009e80 con 0x7f91ec14fbf0 2023-10-10T19:09:59.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91ea7fc700 1 -- 172.21.15.175:0/2266037369 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc005c10 con 0x7f91ec14fbf0 2023-10-10T19:09:59.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.790+0000 7f91ea7fc700 1 -- 172.21.15.175:0/2266037369 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00b610 con 0x7f91ec14fbf0 2023-10-10T19:09:59.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.791+0000 7f91f1a22700 1 -- 172.21.15.175:0/2266037369 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 msgr2=0x7f91ec159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:59.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.791+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2266037369 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec159d70 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0cd790 tx=0x7f91dc009220 comp rx=0 tx=0).stop 2023-10-10T19:09:59.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.791+0000 7f91f1a22700 1 -- 172.21.15.175:0/2266037369 shutdown_connections 2023-10-10T19:09:59.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.791+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2266037369 >> v2:172.21.15.191:3300/0 conn(0x7f91ec0681e0 0x7f91ec0c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:59.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.791+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2266037369 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec159d70 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:59.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.791+0000 7f91f1a22700 1 -- 172.21.15.175:0/2266037369 >> 172.21.15.175:0/2266037369 conn(0x7f91ec0cc760 msgr2=0x7f91ec0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:09:59.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.791+0000 7f91f1a22700 1 -- 172.21.15.175:0/2266037369 shutdown_connections 2023-10-10T19:09:59.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.791+0000 7f91f1a22700 1 -- 172.21.15.175:0/2266037369 wait complete. 2023-10-10T19:09:59.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.792+0000 7f91f1a22700 1 Processor -- start 2023-10-10T19:09:59.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.792+0000 7f91f1a22700 1 -- start start 2023-10-10T19:09:59.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91f1a22700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91ec0681e0 0x7f91ec166fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:59.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91f1a22700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec0c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:59.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91f1a22700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91ec14ffc0 0x7f91ec169260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:59.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91f1a22700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0c1d20 con 0x7f91ec14fbf0 2023-10-10T19:09:59.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91f1a22700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0c1350 con 0x7f91ec14ffc0 2023-10-10T19:09:59.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91f1a22700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0c0780 con 0x7f91ec0681e0 2023-10-10T19:09:59.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec0c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:59.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec0c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39280/0 (socket says 172.21.15.175:39280) 2023-10-10T19:09:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 -- 172.21.15.175:0/2142269247 learned_addr learned my addr 172.21.15.175:0/2142269247 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:09:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91eb7fe700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91ec14ffc0 0x7f91ec169260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91eaffd700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91ec0681e0 0x7f91ec166fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 -- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91ec0681e0 msgr2=0x7f91ec166fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91ec0681e0 0x7f91ec166fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 -- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91ec14ffc0 msgr2=0x7f91ec169260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:09:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91ec14ffc0 0x7f91ec169260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:09:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 -- 172.21.15.175:0/2142269247 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc009480 con 0x7f91ec14fbf0 2023-10-10T19:09:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e27fc700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec0c17e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f91d400c530 tx=0x7f91d400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e37fe700 1 -- 172.21.15.175:0/2142269247 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4011590 con 0x7f91ec14fbf0 2023-10-10T19:09:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e37fe700 1 -- 172.21.15.175:0/2142269247 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d4011710 con 0x7f91ec14fbf0 2023-10-10T19:09:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec169e80 con 0x7f91ec14fbf0 2023-10-10T19:09:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91e37fe700 1 -- 172.21.15.175:0/2142269247 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4011ad0 con 0x7f91ec14fbf0 2023-10-10T19:09:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.793+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec0c0d00 con 0x7f91ec14fbf0 2023-10-10T19:09:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.794+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ec0c1500 con 0x7f91ec14fbf0 2023-10-10T19:09:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.795+0000 7f91e37fe700 1 -- 172.21.15.175:0/2142269247 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f91d4001010 con 0x7f91ec14fbf0 2023-10-10T19:09:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.795+0000 7f91e37fe700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f91cc066f30 0x7f91cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:09:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.795+0000 7f91e37fe700 1 -- 172.21.15.175:0/2142269247 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f91d4083d30 con 0x7f91ec14fbf0 2023-10-10T19:09:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.795+0000 7f91eaffd700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f91cc066f30 0x7f91cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:09:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.795+0000 7f91eaffd700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f91cc066f30 0x7f91cc0693e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009260 tx=0x7f91dc016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:09:59.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:09:59.799+0000 7f91e37fe700 1 -- 172.21.15.175:0/2142269247 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91ec0c1500 con 0x7f91ec14fbf0 2023-10-10T19:10:00.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.026+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91ec0c14c0 con 0x7f91ec14fbf0 2023-10-10T19:10:00.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.027+0000 7f91e37fe700 1 -- 172.21.15.175:0/2142269247 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5439 (secure 0 0 0) 0x7f91d4017070 con 0x7f91ec14fbf0 2023-10-10T19:10:00.028 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:00.028 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:09:53.672616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:00.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f91cc066f30 msgr2=0x7f91cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:00.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f91cc066f30 0x7f91cc0693e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009260 tx=0x7f91dc016000 comp rx=0 tx=0).stop 2023-10-10T19:10:00.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 msgr2=0x7f91ec0c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:00.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec0c17e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f91d400c530 tx=0x7f91d400c840 comp rx=0 tx=0).stop 2023-10-10T19:10:00.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 shutdown_connections 2023-10-10T19:10:00.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f91cc066f30 0x7f91cc0693e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:00.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91ec14ffc0 0x7f91ec169260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:00.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91ec0681e0 0x7f91ec166fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:00.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 --2- 172.21.15.175:0/2142269247 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f91ec14fbf0 0x7f91ec0c17e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:00.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 >> 172.21.15.175:0/2142269247 conn(0x7f91ec0cc760 msgr2=0x7f91ec159520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:00.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 shutdown_connections 2023-10-10T19:10:00.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:00.029+0000 7f91f1a22700 1 -- 172.21.15.175:0/2142269247 wait complete. 2023-10-10T19:10:00.035 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 195 2023-10-10T19:10:00.041 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:10:02.042 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:02.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.171+0000 7f0fa2f2c700 1 Processor -- start 2023-10-10T19:10:02.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.171+0000 7f0fa2f2c700 1 -- start start 2023-10-10T19:10:02.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0fa2f2c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0f9c0b1fc0 0x7f0f9c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:02.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0fa2f2c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0f9c0b2390 0x7f0f9c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:02.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0fa2f2c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c1574b0 con 0x7f0f9c068bd0 2023-10-10T19:10:02.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0fa2f2c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c150770 con 0x7f0f9c0b2390 2023-10-10T19:10:02.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0fa2f2c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c0afad0 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0f9bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0f9c0b2390 0x7f0f9c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:02.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0f9bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0f9c0b2390 0x7f0f9c068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39292/0 (socket says 172.21.15.175:39292) 2023-10-10T19:10:02.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0f9bfff700 1 -- 172.21.15.175:0/3456536629 learned_addr learned my addr 172.21.15.175:0/3456536629 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:02.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0fa14c9700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f0f9c068bd0 0x7f0f9c0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47442/0 (socket says 172.21.15.175:47442) 2023-10-10T19:10:02.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.172+0000 7f0fa0cc8700 1 --2- 172.21.15.175:0/3456536629 >> v2:172.21.15.191:3300/0 conn(0x7f0f9c0b1fc0 0x7f0f9c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:02.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9bfff700 1 -- 172.21.15.175:0/3456536629 >> v2:172.21.15.191:3300/0 conn(0x7f0f9c0b1fc0 msgr2=0x7f0f9c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:02.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9bfff700 1 --2- 172.21.15.175:0/3456536629 >> v2:172.21.15.191:3300/0 conn(0x7f0f9c0b1fc0 0x7f0f9c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9bfff700 1 -- 172.21.15.175:0/3456536629 >> v1:172.21.15.175:6789/0 conn(0x7f0f9c068bd0 legacy=0x7f0f9c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:02.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9bfff700 1 -- 172.21.15.175:0/3456536629 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c0af4c0 con 0x7f0f9c0b2390 2023-10-10T19:10:02.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9b7fe700 1 -- 172.21.15.175:0/3456536629 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159103344 0 0) 0x7f0f9c1574b0 con 0x7f0f9c068bd0 2023-10-10T19:10:02.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9bfff700 1 --2- 172.21.15.175:0/3456536629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b2390 0x7f0f9c068690 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c0b28c0 tx=0x7f0f88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6952ecf7e177523 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:02.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9b7fe700 1 -- 172.21.15.175:0/3456536629 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88009e70 con 0x7f0f9c0b2390 2023-10-10T19:10:02.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9b7fe700 1 -- 172.21.15.175:0/3456536629 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8800b8f0 con 0x7f0f9c0b2390 2023-10-10T19:10:02.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0f9b7fe700 1 -- 172.21.15.175:0/3456536629 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800bc90 con 0x7f0f9c0b2390 2023-10-10T19:10:02.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/3456536629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b2390 msgr2=0x7f0f9c068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:02.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/3456536629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b2390 0x7f0f9c068690 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c0b28c0 tx=0x7f0f88009250 comp rx=0 tx=0).stop 2023-10-10T19:10:02.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/3456536629 shutdown_connections 2023-10-10T19:10:02.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/3456536629 >> v2:172.21.15.191:3300/0 conn(0x7f0f9c0b1fc0 0x7f0f9c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.173+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/3456536629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b2390 0x7f0f9c068690 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.174+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/3456536629 >> 172.21.15.175:0/3456536629 conn(0x7f0f9c063230 msgr2=0x7f0f9c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:02.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.174+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/3456536629 shutdown_connections 2023-10-10T19:10:02.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.174+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/3456536629 wait complete. 2023-10-10T19:10:02.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.175+0000 7f0fa2f2c700 1 Processor -- start 2023-10-10T19:10:02.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.175+0000 7f0fa2f2c700 1 -- start start 2023-10-10T19:10:02.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa2f2c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f9c068bd0 0x7f0f9c169bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:02.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa2f2c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b1fc0 0x7f0f9c14d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:02.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa2f2c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0f9c0b2390 0x7f0f9c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:02.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa2f2c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c157830 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa2f2c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c169730 con 0x7f0f9c068bd0 2023-10-10T19:10:02.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa2f2c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c166ca0 con 0x7f0f9c0b2390 2023-10-10T19:10:02.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b1fc0 0x7f0f9c14d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:02.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b1fc0 0x7f0f9c14d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39296/0 (socket says 172.21.15.175:39296) 2023-10-10T19:10:02.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 -- 172.21.15.175:0/1981741916 learned_addr learned my addr 172.21.15.175:0/1981741916 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:02.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa14c9700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0f9c0b2390 0x7f0f9c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:02.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa0cc8700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f9c068bd0 0x7f0f9c169bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:02.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 -- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0f9c0b2390 msgr2=0x7f0f9c14d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:02.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0f9c0b2390 0x7f0f9c14d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 -- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f9c068bd0 msgr2=0x7f0f9c169bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:02.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f9c068bd0 0x7f0f9c169bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 -- 172.21.15.175:0/1981741916 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f880094f0 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa14c9700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0f9c0b2390 0x7f0f9c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:10:02.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f9bfff700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b1fc0 0x7f0f9c14d460 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c06fd90 tx=0x7f0f88017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:02.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f98ff9700 1 -- 172.21.15.175:0/1981741916 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800bc70 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f98ff9700 1 -- 172.21.15.175:0/1981741916 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8801a450 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c1572b0 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0f98ff9700 1 -- 172.21.15.175:0/1981741916 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8801a7f0 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.176+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c1693c0 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.177+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c0c9b00 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.178+0000 7f0f98ff9700 1 -- 172.21.15.175:0/1981741916 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0f8801ab60 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.178+0000 7f0f98ff9700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0f80066f30 0x7f0f800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:02.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.178+0000 7f0f98ff9700 1 -- 172.21.15.175:0/1981741916 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f0f880865c0 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.178+0000 7f0fa0cc8700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0f80066f30 0x7f0f800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:02.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.179+0000 7f0fa0cc8700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0f80066f30 0x7f0f800693e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009d40 tx=0x7f0f90008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:02.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.182+0000 7f0f98ff9700 1 -- 172.21.15.175:0/1981741916 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f88053360 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.406+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f9c067570 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.407+0000 7f0f98ff9700 1 -- 172.21.15.175:0/1981741916 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5441 (secure 0 0 0) 0x7f0f88056980 con 0x7f0f9c0b1fc0 2023-10-10T19:10:02.407 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:02.408 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:01.721661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:resolve","state_seq":13,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:02.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0f80066f30 msgr2=0x7f0f800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:02.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0f80066f30 0x7f0f800693e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009d40 tx=0x7f0f90008040 comp rx=0 tx=0).stop 2023-10-10T19:10:02.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b1fc0 msgr2=0x7f0f9c14d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:02.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b1fc0 0x7f0f9c14d460 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c06fd90 tx=0x7f0f88017000 comp rx=0 tx=0).stop 2023-10-10T19:10:02.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 shutdown_connections 2023-10-10T19:10:02.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0f80066f30 0x7f0f800693e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0f9c0b2390 0x7f0f9c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f9c068bd0 0x7f0f9c169bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 --2- 172.21.15.175:0/1981741916 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f9c0b1fc0 0x7f0f9c14d460 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:02.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.408+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 >> 172.21.15.175:0/1981741916 conn(0x7f0f9c063230 msgr2=0x7f0f9c0c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:02.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.409+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 shutdown_connections 2023-10-10T19:10:02.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:02.409+0000 7f0fa2f2c700 1 -- 172.21.15.175:0/1981741916 wait complete. 2023-10-10T19:10:02.412 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 196 2023-10-10T19:10:02.421 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:10:04.421 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:04.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.549+0000 7f62a3ae5700 1 Processor -- start 2023-10-10T19:10:04.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.549+0000 7f62a3ae5700 1 -- start start 2023-10-10T19:10:04.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.550+0000 7f62a3ae5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f629c0c5850 0x7f629c0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:04.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.550+0000 7f62a3ae5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f629c0c4f20 0x7f629c0c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:04.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.550+0000 7f62a3ae5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629c0c6fb0 con 0x7f629c0c12d0 2023-10-10T19:10:04.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.550+0000 7f62a3ae5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629c063ea0 con 0x7f629c0c4f20 2023-10-10T19:10:04.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.550+0000 7f62a3ae5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629c155f30 con 0x7f629c0c5850 2023-10-10T19:10:04.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.550+0000 7f62a2082700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f629c0c4f20 0x7f629c0c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:04.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.550+0000 7f62a1080700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f629c0c12d0 0x7f629c156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47450/0 (socket says 172.21.15.175:47450) 2023-10-10T19:10:04.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.550+0000 7f62a1080700 1 -- 172.21.15.175:0/1326423167 learned_addr learned my addr 172.21.15.175:0/1326423167 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:10:04.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a1881700 1 --2- 172.21.15.175:0/1326423167 >> v2:172.21.15.191:3300/0 conn(0x7f629c0c5850 0x7f629c0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:04.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a2082700 1 -- 172.21.15.175:0/1326423167 >> v2:172.21.15.191:3300/0 conn(0x7f629c0c5850 msgr2=0x7f629c0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:04.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a2082700 1 --2- 172.21.15.175:0/1326423167 >> v2:172.21.15.191:3300/0 conn(0x7f629c0c5850 0x7f629c0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a2082700 1 -- 172.21.15.175:0/1326423167 >> v1:172.21.15.175:6789/0 conn(0x7f629c0c12d0 legacy=0x7f629c156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:04.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a2082700 1 -- 172.21.15.175:0/1326423167 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629c0c6160 con 0x7f629c0c4f20 2023-10-10T19:10:04.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a087f700 1 -- 172.21.15.175:0/1326423167 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773327191 0 0) 0x7f629c0c6fb0 con 0x7f629c0c12d0 2023-10-10T19:10:04.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a1881700 1 --2- 172.21.15.175:0/1326423167 >> v2:172.21.15.191:3300/0 conn(0x7f629c0c5850 0x7f629c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:10:04.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a2082700 1 --2- 172.21.15.175:0/1326423167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c4f20 0x7f629c0c7370 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f629c0cb5c0 tx=0x7f62940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1266e4cef8d4655d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:04.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a087f700 1 -- 172.21.15.175:0/1326423167 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629400b740 con 0x7f629c0c4f20 2023-10-10T19:10:04.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a087f700 1 -- 172.21.15.175:0/1326423167 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629400b8a0 con 0x7f629c0c4f20 2023-10-10T19:10:04.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.551+0000 7f62a087f700 1 -- 172.21.15.175:0/1326423167 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629400bc40 con 0x7f629c0c4f20 2023-10-10T19:10:04.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.552+0000 7f62a3ae5700 1 -- 172.21.15.175:0/1326423167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c4f20 msgr2=0x7f629c0c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:04.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.552+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/1326423167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c4f20 0x7f629c0c7370 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f629c0cb5c0 tx=0x7f62940091a0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.552+0000 7f62a3ae5700 1 -- 172.21.15.175:0/1326423167 shutdown_connections 2023-10-10T19:10:04.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.552+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/1326423167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c4f20 0x7f629c0c7370 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.552+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/1326423167 >> v2:172.21.15.191:3300/0 conn(0x7f629c0c5850 0x7f629c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.552+0000 7f62a3ae5700 1 -- 172.21.15.175:0/1326423167 >> 172.21.15.175:0/1326423167 conn(0x7f629c06dc70 msgr2=0x7f629c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:04.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.552+0000 7f62a3ae5700 1 -- 172.21.15.175:0/1326423167 shutdown_connections 2023-10-10T19:10:04.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.552+0000 7f62a3ae5700 1 -- 172.21.15.175:0/1326423167 wait complete. 2023-10-10T19:10:04.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.553+0000 7f62a3ae5700 1 Processor -- start 2023-10-10T19:10:04.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.553+0000 7f62a3ae5700 1 -- start start 2023-10-10T19:10:04.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.553+0000 7f62a3ae5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f629c0c12d0 0x7f629c15a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.553+0000 7f62a3ae5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f629c0c4f20 0x7f629c06ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.553+0000 7f62a3ae5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c5850 0x7f629c06b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.553+0000 7f62a3ae5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c0c20f0 con 0x7f629c0c5850 2023-10-10T19:10:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.553+0000 7f62a3ae5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c14ffc0 con 0x7f629c0c4f20 2023-10-10T19:10:04.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a3ae5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c150580 con 0x7f629c0c12d0 2023-10-10T19:10:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c5850 0x7f629c06b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c5850 0x7f629c06b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39308/0 (socket says 172.21.15.175:39308) 2023-10-10T19:10:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 -- 172.21.15.175:0/3629171683 learned_addr learned my addr 172.21.15.175:0/3629171683 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:04.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a1881700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f629c0c12d0 0x7f629c15a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a1080700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f629c0c4f20 0x7f629c06ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 -- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f629c0c12d0 msgr2=0x7f629c15a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f629c0c12d0 0x7f629c15a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 -- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f629c0c4f20 msgr2=0x7f629c06ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:04.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f629c0c4f20 0x7f629c06ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 -- 172.21.15.175:0/3629171683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6294009420 con 0x7f629c0c5850 2023-10-10T19:10:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a1080700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f629c0c4f20 0x7f629c06ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:10:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a2082700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c5850 0x7f629c06b3c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f629c06e6e0 tx=0x7f6294017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:04.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f6299ffb700 1 -- 172.21.15.175:0/3629171683 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6294025420 con 0x7f629c0c5850 2023-10-10T19:10:04.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f6299ffb700 1 -- 172.21.15.175:0/3629171683 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6294025580 con 0x7f629c0c5850 2023-10-10T19:10:04.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629c15beb0 con 0x7f629c0c5850 2023-10-10T19:10:04.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f6299ffb700 1 -- 172.21.15.175:0/3629171683 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6294025940 con 0x7f629c0c5850 2023-10-10T19:10:04.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.554+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629c15c110 con 0x7f629c0c5850 2023-10-10T19:10:04.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.555+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629c067570 con 0x7f629c0c5850 2023-10-10T19:10:04.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.556+0000 7f6299ffb700 1 -- 172.21.15.175:0/3629171683 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6294023070 con 0x7f629c0c5850 2023-10-10T19:10:04.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.556+0000 7f6299ffb700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6288066f60 0x7f6288069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:04.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.556+0000 7f6299ffb700 1 -- 172.21.15.175:0/3629171683 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f62940866a0 con 0x7f629c0c5850 2023-10-10T19:10:04.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.556+0000 7f62a1881700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6288066f60 0x7f6288069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:04.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.557+0000 7f62a1881700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6288066f60 0x7f6288069410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f628c00d560 tx=0x7f628c0075a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:04.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.560+0000 7f6299ffb700 1 -- 172.21.15.175:0/3629171683 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6294053440 con 0x7f629c0c5850 2023-10-10T19:10:04.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.787+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f629c1506c0 con 0x7f629c0c5850 2023-10-10T19:10:04.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.788+0000 7f6299ffb700 1 -- 172.21.15.175:0/3629171683 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5441 (secure 0 0 0) 0x7f6294056a60 con 0x7f629c0c5850 2023-10-10T19:10:04.789 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:04.789 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:01.721661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:resolve","state_seq":13,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:04.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6288066f60 msgr2=0x7f6288069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:04.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6288066f60 0x7f6288069410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f628c00d560 tx=0x7f628c0075a0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c5850 msgr2=0x7f629c06b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:04.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c5850 0x7f629c06b3c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f629c06e6e0 tx=0x7f6294017000 comp rx=0 tx=0).stop 2023-10-10T19:10:04.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 shutdown_connections 2023-10-10T19:10:04.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6288066f60 0x7f6288069410 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f629c0c5850 0x7f629c06b3c0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f629c0c12d0 0x7f629c15a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 --2- 172.21.15.175:0/3629171683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f629c0c4f20 0x7f629c06ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:04.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 >> 172.21.15.175:0/3629171683 conn(0x7f629c06dc70 msgr2=0x7f629c152e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:04.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 shutdown_connections 2023-10-10T19:10:04.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:04.790+0000 7f62a3ae5700 1 -- 172.21.15.175:0/3629171683 wait complete. 2023-10-10T19:10:04.793 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 196 2023-10-10T19:10:04.802 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:10:06.803 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:06.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.930+0000 7fca7b777700 1 Processor -- start 2023-10-10T19:10:06.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.931+0000 7fca7b777700 1 -- start start 2023-10-10T19:10:06.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.931+0000 7fca7b777700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fca74155fb0 0x7fca74159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:06.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.931+0000 7fca7b777700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca741574b0 con 0x7fca740b1fc0 2023-10-10T19:10:06.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.931+0000 7fca7b777700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740afad0 con 0x7fca740b2390 2023-10-10T19:10:06.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.931+0000 7fca7b777700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca74150770 con 0x7fca74155fb0 2023-10-10T19:10:06.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.931+0000 7fca79513700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fca740b1fc0 0x7fca74068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39818/0 (socket says 172.21.15.175:39818) 2023-10-10T19:10:06.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.931+0000 7fca79513700 1 -- 172.21.15.175:0/2844220537 learned_addr learned my addr 172.21.15.175:0/2844220537 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:10:06.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79d14700 1 --2- 172.21.15.175:0/2844220537 >> v2:172.21.15.191:3300/0 conn(0x7fca74155fb0 0x7fca74159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:06.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766904925 0 0) 0x7fca741574b0 con 0x7fca740b1fc0 2023-10-10T19:10:06.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca60003570 con 0x7fca740b1fc0 2023-10-10T19:10:06.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254358401 0 0) 0x7fca740afad0 con 0x7fca740b2390 2023-10-10T19:10:06.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca741574b0 con 0x7fca740b2390 2023-10-10T19:10:06.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1496387341 0 0) 0x7fca60003570 con 0x7fca740b1fc0 2023-10-10T19:10:06.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 >> v2:172.21.15.191:3300/0 conn(0x7fca74155fb0 msgr2=0x7fca74159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:06.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 --2- 172.21.15.175:0/2844220537 >> v2:172.21.15.191:3300/0 conn(0x7fca74155fb0 0x7fca74159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:06.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 >> v1:172.21.15.191:6789/0 conn(0x7fca740b2390 legacy=0x7fca740aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:06.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740af4c0 con 0x7fca740b1fc0 2023-10-10T19:10:06.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fca70001470 con 0x7fca740b1fc0 2023-10-10T19:10:06.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 >> v1:172.21.15.175:6789/0 conn(0x7fca740b1fc0 legacy=0x7fca74068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:06.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca60003bf0 0x7fca60003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:06.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca60004810 0x7fca60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:06.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca600051c0 0x7fca6000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:06.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca74150770 con 0x7fca60003bf0 2023-10-10T19:10:06.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6000c0b0 con 0x7fca60004810 2023-10-10T19:10:06.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6000c270 con 0x7fca600051c0 2023-10-10T19:10:06.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79513700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca60003bf0 0x7fca60003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:06.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca78d12700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca60004810 0x7fca60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:06.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79d14700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca600051c0 0x7fca6000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:06.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79513700 1 -- 172.21.15.175:0/2844220537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca600051c0 msgr2=0x7fca6000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:06.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79513700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca600051c0 0x7fca6000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:06.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79513700 1 -- 172.21.15.175:0/2844220537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca60004810 msgr2=0x7fca60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:06.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79513700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca60004810 0x7fca60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:06.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79513700 1 -- 172.21.15.175:0/2844220537 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca60003570 con 0x7fca60003bf0 2023-10-10T19:10:06.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.932+0000 7fca79d14700 1 -- 172.21.15.175:0/2844220537 reap_dead start 2023-10-10T19:10:06.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca79513700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca60003bf0 0x7fca60003fe0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fca7000c2d0 tx=0x7fca740af650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d14f466a1569b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:06.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7000cc00 con 0x7fca60003bf0 2023-10-10T19:10:06.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7000cda0 con 0x7fca60003bf0 2023-10-10T19:10:06.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca6bfff700 1 -- 172.21.15.175:0/2844220537 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70005b70 con 0x7fca60003bf0 2023-10-10T19:10:06.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca7b777700 1 -- 172.21.15.175:0/2844220537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca60003bf0 msgr2=0x7fca60003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:06.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca7b777700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca60003bf0 0x7fca60003fe0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fca7000c2d0 tx=0x7fca740af650 comp rx=0 tx=0).stop 2023-10-10T19:10:06.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca7b777700 1 -- 172.21.15.175:0/2844220537 shutdown_connections 2023-10-10T19:10:06.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca7b777700 1 --2- 172.21.15.175:0/2844220537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca60003bf0 0x7fca60003fe0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:06.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca7b777700 1 -- 172.21.15.175:0/2844220537 >> 172.21.15.175:0/2844220537 conn(0x7fca74063230 msgr2=0x7fca74063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:06.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca7b777700 1 -- 172.21.15.175:0/2844220537 shutdown_connections 2023-10-10T19:10:06.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.933+0000 7fca7b777700 1 -- 172.21.15.175:0/2844220537 wait complete. 2023-10-10T19:10:06.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.934+0000 7fca7b777700 1 Processor -- start 2023-10-10T19:10:06.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.934+0000 7fca7b777700 1 -- start start 2023-10-10T19:10:06.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca7b777700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca60003bf0 0x7fca7414b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:06.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca7b777700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca74167660 0x7fca74148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:06.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca7b777700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca74167ab0 0x7fca74148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:06.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca7b777700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7414bd90 con 0x7fca74167660 2023-10-10T19:10:06.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca7b777700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7414e600 con 0x7fca74167ab0 2023-10-10T19:10:06.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca7b777700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7414d310 con 0x7fca60003bf0 2023-10-10T19:10:06.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca78d12700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca74167660 0x7fca74148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:06.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca78d12700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca74167660 0x7fca74148ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53676/0 (socket says 172.21.15.175:53676) 2023-10-10T19:10:06.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca78d12700 1 -- 172.21.15.175:0/2022837010 learned_addr learned my addr 172.21.15.175:0/2022837010 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:06.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca79513700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca60003bf0 0x7fca7414b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:06.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.935+0000 7fca79d14700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca74167ab0 0x7fca74148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:06.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca78d12700 1 -- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca60003bf0 msgr2=0x7fca7414b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:06.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca78d12700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca60003bf0 0x7fca7414b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:06.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca78d12700 1 -- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca74167ab0 msgr2=0x7fca74148ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:06.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca78d12700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca74167ab0 0x7fca74148ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:06.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca78d12700 1 -- 172.21.15.175:0/2022837010 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7414bf90 con 0x7fca74167660 2023-10-10T19:10:06.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca78d12700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca74167660 0x7fca74148ab0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fca64002c00 tx=0x7fca64007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca697fa700 1 -- 172.21.15.175:0/2022837010 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6400f9b0 con 0x7fca74167660 2023-10-10T19:10:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca697fa700 1 -- 172.21.15.175:0/2022837010 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6400fb10 con 0x7fca74167660 2023-10-10T19:10:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7414c270 con 0x7fca74167660 2023-10-10T19:10:06.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca697fa700 1 -- 172.21.15.175:0/2022837010 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6400fed0 con 0x7fca74167660 2023-10-10T19:10:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.936+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7414c830 con 0x7fca74167660 2023-10-10T19:10:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.937+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca74067570 con 0x7fca74167660 2023-10-10T19:10:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.937+0000 7fca697fa700 1 -- 172.21.15.175:0/2022837010 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fca64013600 con 0x7fca74167660 2023-10-10T19:10:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.938+0000 7fca697fa700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca58066f60 0x7fca58069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:06.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.938+0000 7fca697fa700 1 -- 172.21.15.175:0/2022837010 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fca64085070 con 0x7fca74167660 2023-10-10T19:10:06.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.938+0000 7fca79513700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca58066f60 0x7fca58069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:06.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.939+0000 7fca79513700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca58066f60 0x7fca58069410 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fca70007c90 tx=0x7fca70015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:06.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:06.942+0000 7fca697fa700 1 -- 172.21.15.175:0/2022837010 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca64051e10 con 0x7fca74167660 2023-10-10T19:10:07.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.164+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca74069060 con 0x7fca74167660 2023-10-10T19:10:07.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.164+0000 7fca697fa700 1 -- 172.21.15.175:0/2022837010 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5441 (secure 0 0 0) 0x7fca64055430 con 0x7fca74167660 2023-10-10T19:10:07.165 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:07.165 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:01.721661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:resolve","state_seq":13,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:07.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca58066f60 msgr2=0x7fca58069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:07.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca58066f60 0x7fca58069410 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fca70007c90 tx=0x7fca70015000 comp rx=0 tx=0).stop 2023-10-10T19:10:07.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca74167660 msgr2=0x7fca74148ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:07.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca74167660 0x7fca74148ab0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fca64002c00 tx=0x7fca64007d30 comp rx=0 tx=0).stop 2023-10-10T19:10:07.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 shutdown_connections 2023-10-10T19:10:07.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fca58066f60 0x7fca58069410 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:07.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca74167ab0 0x7fca74148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:07.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fca60003bf0 0x7fca7414b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:07.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 --2- 172.21.15.175:0/2022837010 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fca74167660 0x7fca74148ab0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:07.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 >> 172.21.15.175:0/2022837010 conn(0x7fca74063230 msgr2=0x7fca741564c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:07.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 shutdown_connections 2023-10-10T19:10:07.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:07.167+0000 7fca7b777700 1 -- 172.21.15.175:0/2022837010 wait complete. 2023-10-10T19:10:07.170 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 196 2023-10-10T19:10:07.180 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 196, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 196, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5634": { "addr": "172.21.15.175:6837/1881948738", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 1881948738, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 1881948738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 5634, "incarnation": 94, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 25 }, "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:resolve", "state_seq": 13 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:10:01.721661+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5634, "mds_1": 6117, "mds_2": 6106, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:10:09.181 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:09.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.308+0000 7f74681c4700 1 Processor -- start 2023-10-10T19:10:09.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.308+0000 7f74681c4700 1 -- start start 2023-10-10T19:10:09.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f74681c4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600b1fc0 0x7f7460155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:09.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f74681c4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f74600aff60 0x7f74600b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:09.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f74681c4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74601574b0 con 0x7f74600b2390 2023-10-10T19:10:09.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f74681c4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7460150770 con 0x7f74600aff60 2023-10-10T19:10:09.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f74681c4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74600afad0 con 0x7f74600b1fc0 2023-10-10T19:10:09.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f7465f60700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600b1fc0 0x7f7460155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:09.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f7466761700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f74600aff60 0x7f74600b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:09.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f7465f60700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600b1fc0 0x7f7460155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:58712/0 (socket says 172.21.15.175:58712) 2023-10-10T19:10:09.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f7466761700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f74600aff60 0x7f74600b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:60628/0 (socket says 172.21.15.175:60628) 2023-10-10T19:10:09.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.309+0000 7f7465f60700 1 -- 172.21.15.175:0/1019739833 learned_addr learned my addr 172.21.15.175:0/1019739833 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:09.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.310+0000 7f7466761700 1 -- 172.21.15.175:0/1019739833 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600b1fc0 msgr2=0x7f7460155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:09.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.310+0000 7f7466761700 1 --2- 172.21.15.175:0/1019739833 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600b1fc0 0x7f7460155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.310+0000 7f7466761700 1 -- 172.21.15.175:0/1019739833 >> v1:172.21.15.191:6789/0 conn(0x7f74600b2390 legacy=0x7f7460068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:09.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.310+0000 7f7466761700 1 -- 172.21.15.175:0/1019739833 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74600af4c0 con 0x7f74600aff60 2023-10-10T19:10:09.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.310+0000 7f7466761700 1 --2- 172.21.15.175:0/1019739833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600aff60 0x7f74600b03e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7458009e90 tx=0x7f7458009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1017b5004879e5ce server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:09.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.310+0000 7f7464f5e700 1 -- 172.21.15.175:0/1019739833 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745800e070 con 0x7f74600aff60 2023-10-10T19:10:09.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.310+0000 7f7464f5e700 1 -- 172.21.15.175:0/1019739833 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745800bc80 con 0x7f74600aff60 2023-10-10T19:10:09.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.310+0000 7f7464f5e700 1 -- 172.21.15.175:0/1019739833 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458012430 con 0x7f74600aff60 2023-10-10T19:10:09.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.311+0000 7f74681c4700 1 -- 172.21.15.175:0/1019739833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600aff60 msgr2=0x7f74600b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:09.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.311+0000 7f74681c4700 1 --2- 172.21.15.175:0/1019739833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600aff60 0x7f74600b03e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7458009e90 tx=0x7f7458009780 comp rx=0 tx=0).stop 2023-10-10T19:10:09.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.311+0000 7f74681c4700 1 -- 172.21.15.175:0/1019739833 shutdown_connections 2023-10-10T19:10:09.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.311+0000 7f74681c4700 1 --2- 172.21.15.175:0/1019739833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600aff60 0x7f74600b03e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.311+0000 7f74681c4700 1 --2- 172.21.15.175:0/1019739833 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600b1fc0 0x7f7460155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.311+0000 7f74681c4700 1 -- 172.21.15.175:0/1019739833 >> 172.21.15.175:0/1019739833 conn(0x7f7460063230 msgr2=0x7f7460063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:09.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.311+0000 7f74681c4700 1 -- 172.21.15.175:0/1019739833 shutdown_connections 2023-10-10T19:10:09.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.311+0000 7f74681c4700 1 -- 172.21.15.175:0/1019739833 wait complete. 2023-10-10T19:10:09.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.312+0000 7f74681c4700 1 Processor -- start 2023-10-10T19:10:09.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.312+0000 7f74681c4700 1 -- start start 2023-10-10T19:10:09.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.312+0000 7f74681c4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600aff60 0x7f7460154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:09.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.312+0000 7f74681c4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74600b1fc0 0x7f7460169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:09.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.312+0000 7f74681c4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600b2390 0x7f746015cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:09.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.312+0000 7f74681c4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74601694d0 con 0x7f74600b1fc0 2023-10-10T19:10:09.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.312+0000 7f74681c4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74600c4550 con 0x7f74600b2390 2023-10-10T19:10:09.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.312+0000 7f74681c4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7460169ea0 con 0x7f74600aff60 2023-10-10T19:10:09.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74600b1fc0 0x7f7460169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:09.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74600b1fc0 0x7f7460169960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53690/0 (socket says 172.21.15.175:53690) 2023-10-10T19:10:09.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 -- 172.21.15.175:0/1053555196 learned_addr learned my addr 172.21.15.175:0/1053555196 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:09.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f7466761700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600b2390 0x7f746015cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:09.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f7465f60700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600aff60 0x7f7460154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:09.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 -- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600aff60 msgr2=0x7f7460154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:09.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600aff60 0x7f7460154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 -- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600b2390 msgr2=0x7f746015cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:09.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600b2390 0x7f746015cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 -- 172.21.15.175:0/1053555196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7458009a20 con 0x7f74600b1fc0 2023-10-10T19:10:09.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f746575f700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74600b1fc0 0x7f7460169960 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f745000c4e0 tx=0x7f745000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:09.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f7455ffb700 1 -- 172.21.15.175:0/1053555196 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450011590 con 0x7f74600b1fc0 2023-10-10T19:10:09.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74600c3cb0 con 0x7f74600b1fc0 2023-10-10T19:10:09.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f7455ffb700 1 -- 172.21.15.175:0/1053555196 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7450011710 con 0x7f74600b1fc0 2023-10-10T19:10:09.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f7455ffb700 1 -- 172.21.15.175:0/1053555196 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450011ad0 con 0x7f74600b1fc0 2023-10-10T19:10:09.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.313+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74600c4270 con 0x7f74600b1fc0 2023-10-10T19:10:09.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.314+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7460169680 con 0x7f74600b1fc0 2023-10-10T19:10:09.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.315+0000 7f7455ffb700 1 -- 172.21.15.175:0/1053555196 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f745001b3b0 con 0x7f74600b1fc0 2023-10-10T19:10:09.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.315+0000 7f7455ffb700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7444066f30 0x7f74440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:09.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.315+0000 7f7455ffb700 1 -- 172.21.15.175:0/1053555196 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f7450083fd0 con 0x7f74600b1fc0 2023-10-10T19:10:09.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.315+0000 7f7465f60700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7444066f30 0x7f74440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:09.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.316+0000 7f7465f60700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7444066f30 0x7f74440693e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f745c006fd0 tx=0x7f745c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:09.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.319+0000 7f7455ffb700 1 -- 172.21.15.175:0/1053555196 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7460169680 con 0x7f74600b1fc0 2023-10-10T19:10:09.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.543+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7460169680 con 0x7f74600b1fc0 2023-10-10T19:10:09.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.543+0000 7f7455ffb700 1 -- 172.21.15.175:0/1053555196 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5443 (secure 0 0 0) 0x7f7460169680 con 0x7f74600b1fc0 2023-10-10T19:10:09.626 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:09.626 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:07.737766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:09.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7444066f30 msgr2=0x7f74440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:09.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7444066f30 0x7f74440693e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f745c006fd0 tx=0x7f745c008040 comp rx=0 tx=0).stop 2023-10-10T19:10:09.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74600b1fc0 msgr2=0x7f7460169960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:09.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74600b1fc0 0x7f7460169960 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f745000c4e0 tx=0x7f745000c8a0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 shutdown_connections 2023-10-10T19:10:09.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7444066f30 0x7f74440693e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74600b2390 0x7f746015cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74600aff60 0x7f7460154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 --2- 172.21.15.175:0/1053555196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74600b1fc0 0x7f7460169960 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:09.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 >> 172.21.15.175:0/1053555196 conn(0x7f7460063230 msgr2=0x7f74601523f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:09.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 shutdown_connections 2023-10-10T19:10:09.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:09.545+0000 7f74681c4700 1 -- 172.21.15.175:0/1053555196 wait complete. 2023-10-10T19:10:09.631 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 197 2023-10-10T19:10:09.631 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:10:11.633 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:11.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.770+0000 7f878ca14700 1 Processor -- start 2023-10-10T19:10:11.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.770+0000 7f878ca14700 1 -- start start 2023-10-10T19:10:11.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f878ca14700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f87880c4580 0x7f87881504f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:11.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f878ca14700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8788157030 con 0x7f87880c7190 2023-10-10T19:10:11.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f878ca14700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8788156420 con 0x7f87880c6330 2023-10-10T19:10:11.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f878ca14700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87880c5000 con 0x7f87880c4580 2023-10-10T19:10:11.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f8786d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f87880c4580 0x7f87881504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:11.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f8785d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f87880c7190 0x7f878814fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39822/0 (socket says 172.21.15.175:39822) 2023-10-10T19:10:11.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f8785d9b700 1 -- 172.21.15.175:0/3287948873 learned_addr learned my addr 172.21.15.175:0/3287948873 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:10:11.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f8786d9d700 1 -- 172.21.15.175:0/3287948873 >> v1:172.21.15.191:6789/0 conn(0x7f87880c6330 legacy=0x7f87880c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:11.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f8786d9d700 1 -- 172.21.15.175:0/3287948873 >> v1:172.21.15.175:6789/0 conn(0x7f87880c7190 legacy=0x7f878814fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:11.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f8786d9d700 1 -- 172.21.15.175:0/3287948873 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788156700 con 0x7f87880c4580 2023-10-10T19:10:11.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f878559a700 1 -- 172.21.15.175:0/3287948873 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637902761 0 0) 0x7f8788157030 con 0x7f87880c7190 2023-10-10T19:10:11.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f8786d9d700 1 --2- 172.21.15.175:0/3287948873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f87881504f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8778009e60 tx=0x7f8778009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfbd5e0d53f4a94 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:11.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.771+0000 7f878559a700 1 -- 172.21.15.175:0/3287948873 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877800e070 con 0x7f87880c4580 2023-10-10T19:10:11.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878559a700 1 -- 172.21.15.175:0/3287948873 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877800bc80 con 0x7f87880c4580 2023-10-10T19:10:11.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878559a700 1 -- 172.21.15.175:0/3287948873 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778012430 con 0x7f87880c4580 2023-10-10T19:10:11.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878ca14700 1 -- 172.21.15.175:0/3287948873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 msgr2=0x7f87881504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:11.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878ca14700 1 --2- 172.21.15.175:0/3287948873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f87881504f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8778009e60 tx=0x7f8778009780 comp rx=0 tx=0).stop 2023-10-10T19:10:11.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878ca14700 1 -- 172.21.15.175:0/3287948873 shutdown_connections 2023-10-10T19:10:11.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878ca14700 1 --2- 172.21.15.175:0/3287948873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f87881504f0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:11.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878ca14700 1 -- 172.21.15.175:0/3287948873 >> 172.21.15.175:0/3287948873 conn(0x7f878806ef20 msgr2=0x7f8788063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:11.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878ca14700 1 -- 172.21.15.175:0/3287948873 shutdown_connections 2023-10-10T19:10:11.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.772+0000 7f878ca14700 1 -- 172.21.15.175:0/3287948873 wait complete. 2023-10-10T19:10:11.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.773+0000 7f878ca14700 1 Processor -- start 2023-10-10T19:10:11.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.773+0000 7f878ca14700 1 -- start start 2023-10-10T19:10:11.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878ca14700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f8788159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:11.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878ca14700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87880c6330 0x7f8788158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:11.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878ca14700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87880c7190 0x7f8788149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:11.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878ca14700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87881589f0 con 0x7f87880c4580 2023-10-10T19:10:11.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878ca14700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87880c8090 con 0x7f87880c6330 2023-10-10T19:10:11.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878ca14700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87880cad10 con 0x7f87880c7190 2023-10-10T19:10:11.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f8788159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:11.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f8788159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53706/0 (socket says 172.21.15.175:53706) 2023-10-10T19:10:11.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878659c700 1 -- 172.21.15.175:0/3171183786 learned_addr learned my addr 172.21.15.175:0/3171183786 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:11.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f8785d9b700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87880c6330 0x7f8788158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:11.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f8786d9d700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87880c7190 0x7f8788149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:11.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878659c700 1 -- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87880c7190 msgr2=0x7f8788149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:11.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878659c700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87880c7190 0x7f8788149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:11.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878659c700 1 -- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87880c6330 msgr2=0x7f8788158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:11.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878659c700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87880c6330 0x7f8788158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:11.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.774+0000 7f878659c700 1 -- 172.21.15.175:0/3171183786 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8778009a20 con 0x7f87880c4580 2023-10-10T19:10:11.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.775+0000 7f878659c700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f8788159230 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f878000c600 tx=0x7f8780007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:11.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.775+0000 7f877e7fc700 1 -- 172.21.15.175:0/3171183786 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780011590 con 0x7f87880c4580 2023-10-10T19:10:11.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.775+0000 7f877e7fc700 1 -- 172.21.15.175:0/3171183786 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8780011710 con 0x7f87880c4580 2023-10-10T19:10:11.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.775+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8788158710 con 0x7f87880c4580 2023-10-10T19:10:11.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.775+0000 7f877e7fc700 1 -- 172.21.15.175:0/3171183786 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780011ad0 con 0x7f87880c4580 2023-10-10T19:10:11.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.775+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87880c7a40 con 0x7f87880c4580 2023-10-10T19:10:11.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.776+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8788067570 con 0x7f87880c4580 2023-10-10T19:10:11.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.776+0000 7f877e7fc700 1 -- 172.21.15.175:0/3171183786 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f878001c460 con 0x7f87880c4580 2023-10-10T19:10:11.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.777+0000 7f877e7fc700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8768066f30 0x7f87680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:11.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.777+0000 7f877e7fc700 1 -- 172.21.15.175:0/3171183786 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f87800847b0 con 0x7f87880c4580 2023-10-10T19:10:11.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.777+0000 7f8785d9b700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8768066f30 0x7f87680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:11.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.777+0000 7f8785d9b700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8768066f30 0x7f87680693e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8788159770 tx=0x7f8770009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:11.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:11.781+0000 7f877e7fc700 1 -- 172.21.15.175:0/3171183786 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8780051550 con 0x7f87880c4580 2023-10-10T19:10:12.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.005+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87881569e0 con 0x7f87880c4580 2023-10-10T19:10:12.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.006+0000 7f877e7fc700 1 -- 172.21.15.175:0/3171183786 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5443 (secure 0 0 0) 0x7f8780054b70 con 0x7f87880c4580 2023-10-10T19:10:12.007 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:12.007 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:07.737766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:12.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8768066f30 msgr2=0x7f87680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:12.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8768066f30 0x7f87680693e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8788159770 tx=0x7f8770009040 comp rx=0 tx=0).stop 2023-10-10T19:10:12.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 msgr2=0x7f8788159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:12.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f8788159230 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f878000c600 tx=0x7f8780007f60 comp rx=0 tx=0).stop 2023-10-10T19:10:12.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 shutdown_connections 2023-10-10T19:10:12.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8768066f30 0x7f87680693e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:12.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87880c7190 0x7f8788149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:12.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87880c4580 0x7f8788159230 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:12.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 --2- 172.21.15.175:0/3171183786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87880c6330 0x7f8788158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:12.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 >> 172.21.15.175:0/3171183786 conn(0x7f878806ef20 msgr2=0x7f8788153f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:12.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 shutdown_connections 2023-10-10T19:10:12.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:12.008+0000 7f878ca14700 1 -- 172.21.15.175:0/3171183786 wait complete. 2023-10-10T19:10:12.012 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 197 2023-10-10T19:10:12.021 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:10:14.022 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:14.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.142+0000 7f6f9b977700 1 Processor -- start 2023-10-10T19:10:14.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.142+0000 7f6f9b977700 1 -- start start 2023-10-10T19:10:14.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.143+0000 7f6f9b977700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6f940b1fc0 0x7f6f94155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:14.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.143+0000 7f6f9b977700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f941574b0 con 0x7f6f940aff60 2023-10-10T19:10:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.143+0000 7f6f9b977700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f940afad0 con 0x7f6f940b2390 2023-10-10T19:10:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.143+0000 7f6f9b977700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f94150770 con 0x7f6f940b1fc0 2023-10-10T19:10:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.143+0000 7f6f99f14700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6f940aff60 0x7f6f94159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39828/0 (socket says 172.21.15.175:39828) 2023-10-10T19:10:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.143+0000 7f6f99f14700 1 -- 172.21.15.175:0/3652589450 learned_addr learned my addr 172.21.15.175:0/3652589450 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:10:14.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.143+0000 7f6f99713700 1 --2- 172.21.15.175:0/3652589450 >> v2:172.21.15.191:3300/0 conn(0x7f6f940b1fc0 0x7f6f94155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f8bfff700 1 -- 172.21.15.175:0/3652589450 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498413258 0 0) 0x7f6f941574b0 con 0x7f6f940aff60 2023-10-10T19:10:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f8bfff700 1 -- 172.21.15.175:0/3652589450 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f80003570 con 0x7f6f940aff60 2023-10-10T19:10:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f99713700 1 -- 172.21.15.175:0/3652589450 >> v1:172.21.15.191:6789/0 conn(0x7f6f940b2390 legacy=0x7f6f94068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f8bfff700 1 -- 172.21.15.175:0/3652589450 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4040328826 0 0) 0x7f6f80003570 con 0x7f6f940aff60 2023-10-10T19:10:14.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f99713700 1 -- 172.21.15.175:0/3652589450 >> v1:172.21.15.175:6789/0 conn(0x7f6f940aff60 legacy=0x7f6f94159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:14.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f99713700 1 -- 172.21.15.175:0/3652589450 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f940af4c0 con 0x7f6f940b1fc0 2023-10-10T19:10:14.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f99713700 1 --2- 172.21.15.175:0/3652589450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940b1fc0 0x7f6f94155fb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6f900098e0 tx=0x7f6f900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5a84a4760ab7ad1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:14.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f8bfff700 1 -- 172.21.15.175:0/3652589450 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0017f0 con 0x7f6f940b1fc0 2023-10-10T19:10:14.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f8bfff700 1 -- 172.21.15.175:0/3652589450 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f90011420 con 0x7f6f940b1fc0 2023-10-10T19:10:14.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.144+0000 7f6f8bfff700 1 -- 172.21.15.175:0/3652589450 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f900117c0 con 0x7f6f940b1fc0 2023-10-10T19:10:14.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.145+0000 7f6f9b977700 1 -- 172.21.15.175:0/3652589450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940b1fc0 msgr2=0x7f6f94155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:14.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.145+0000 7f6f9b977700 1 --2- 172.21.15.175:0/3652589450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940b1fc0 0x7f6f94155fb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6f900098e0 tx=0x7f6f900091a0 comp rx=0 tx=0).stop 2023-10-10T19:10:14.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.145+0000 7f6f9b977700 1 -- 172.21.15.175:0/3652589450 shutdown_connections 2023-10-10T19:10:14.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.145+0000 7f6f9b977700 1 --2- 172.21.15.175:0/3652589450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940b1fc0 0x7f6f94155fb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:14.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.145+0000 7f6f9b977700 1 -- 172.21.15.175:0/3652589450 >> 172.21.15.175:0/3652589450 conn(0x7f6f94063230 msgr2=0x7f6f94063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:14.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.145+0000 7f6f9b977700 1 -- 172.21.15.175:0/3652589450 shutdown_connections 2023-10-10T19:10:14.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.145+0000 7f6f9b977700 1 -- 172.21.15.175:0/3652589450 wait complete. 2023-10-10T19:10:14.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.146+0000 7f6f9b977700 1 Processor -- start 2023-10-10T19:10:14.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.146+0000 7f6f9b977700 1 -- start start 2023-10-10T19:10:14.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f9b977700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940aff60 0x7f6f94167630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:14.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f9b977700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f940b1fc0 0x7f6f941663c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:14.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f9b977700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f940b2390 0x7f6f94166900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:14.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f9b977700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94159e50 con 0x7f6f940b1fc0 2023-10-10T19:10:14.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f9b977700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94157c30 con 0x7f6f940aff60 2023-10-10T19:10:14.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f9b977700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94157f10 con 0x7f6f940b2390 2023-10-10T19:10:14.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f940b1fc0 0x7f6f941663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:14.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f940b1fc0 0x7f6f941663c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53716/0 (socket says 172.21.15.175:53716) 2023-10-10T19:10:14.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 -- 172.21.15.175:0/1888388418 learned_addr learned my addr 172.21.15.175:0/1888388418 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:14.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f99713700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940aff60 0x7f6f94167630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:14.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f99f14700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f940b2390 0x7f6f94166900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:14.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 -- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f940b2390 msgr2=0x7f6f94166900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:14.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f940b2390 0x7f6f94166900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:14.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 -- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940aff60 msgr2=0x7f6f94167630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:14.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940aff60 0x7f6f94167630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:14.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 -- 172.21.15.175:0/1888388418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90009420 con 0x7f6f940b1fc0 2023-10-10T19:10:14.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f98f12700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f940b1fc0 0x7f6f941663c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400c5c0 tx=0x7f6f8400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:14.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f897fa700 1 -- 172.21.15.175:0/1888388418 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f84012630 con 0x7f6f940b1fc0 2023-10-10T19:10:14.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f897fa700 1 -- 172.21.15.175:0/1888388418 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f840127b0 con 0x7f6f940b1fc0 2023-10-10T19:10:14.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f941581f0 con 0x7f6f940b1fc0 2023-10-10T19:10:14.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f897fa700 1 -- 172.21.15.175:0/1888388418 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f84012b70 con 0x7f6f940b1fc0 2023-10-10T19:10:14.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.147+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f941587b0 con 0x7f6f940b1fc0 2023-10-10T19:10:14.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.148+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f941589a0 con 0x7f6f940b1fc0 2023-10-10T19:10:14.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.149+0000 7f6f897fa700 1 -- 172.21.15.175:0/1888388418 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6f8401b400 con 0x7f6f940b1fc0 2023-10-10T19:10:14.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.149+0000 7f6f897fa700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f78066f30 0x7f6f780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:14.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.149+0000 7f6f897fa700 1 -- 172.21.15.175:0/1888388418 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f6f84084570 con 0x7f6f940b1fc0 2023-10-10T19:10:14.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.149+0000 7f6f99713700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f78066f30 0x7f6f780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:14.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.150+0000 7f6f99713700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f78066f30 0x7f6f780693e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6f900091e0 tx=0x7f6f90017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:14.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.153+0000 7f6f897fa700 1 -- 172.21.15.175:0/1888388418 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f84051310 con 0x7f6f940b1fc0 2023-10-10T19:10:14.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.380+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f94067570 con 0x7f6f940b1fc0 2023-10-10T19:10:14.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.380+0000 7f6f897fa700 1 -- 172.21.15.175:0/1888388418 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f6f84054930 con 0x7f6f940b1fc0 2023-10-10T19:10:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:14.383 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:12.753026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:14.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f78066f30 msgr2=0x7f6f780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:14.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f78066f30 0x7f6f780693e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6f900091e0 tx=0x7f6f90017000 comp rx=0 tx=0).stop 2023-10-10T19:10:14.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f940b1fc0 msgr2=0x7f6f941663c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:14.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f940b1fc0 0x7f6f941663c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400c5c0 tx=0x7f6f8400c980 comp rx=0 tx=0).stop 2023-10-10T19:10:14.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 shutdown_connections 2023-10-10T19:10:14.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f78066f30 0x7f6f780693e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:14.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f940b2390 0x7f6f94166900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:14.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f940aff60 0x7f6f94167630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:14.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 --2- 172.21.15.175:0/1888388418 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f940b1fc0 0x7f6f941663c0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:14.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 >> 172.21.15.175:0/1888388418 conn(0x7f6f94063230 msgr2=0x7f6f940b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:14.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.382+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 shutdown_connections 2023-10-10T19:10:14.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:14.383+0000 7f6f9b977700 1 -- 172.21.15.175:0/1888388418 wait complete. 2023-10-10T19:10:14.387 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 198 2023-10-10T19:10:14.395 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:10:16.397 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:16.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.521+0000 7fba2887e700 1 Processor -- start 2023-10-10T19:10:16.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.521+0000 7fba2887e700 1 -- start start 2023-10-10T19:10:16.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba2887e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fba2408a100 0x7fba24159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:16.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba2887e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba240cd040 0x7fba240684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:16.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba2887e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba24067fa0 con 0x7fba2414fd90 2023-10-10T19:10:16.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba2887e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba240c5600 con 0x7fba240cd040 2023-10-10T19:10:16.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba2887e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba24156b70 con 0x7fba2408a100 2023-10-10T19:10:16.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba23fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba240cd040 0x7fba240684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:16.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba23fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba240cd040 0x7fba240684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34150/0 (socket says 172.21.15.175:34150) 2023-10-10T19:10:16.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba237fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fba2414fd90 0x7fba240c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38150/0 (socket says 172.21.15.175:38150) 2023-10-10T19:10:16.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba23fff700 1 -- 172.21.15.175:0/3919646333 learned_addr learned my addr 172.21.15.175:0/3919646333 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:16.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba22ffd700 1 --2- 172.21.15.175:0/3919646333 >> v2:172.21.15.191:3300/0 conn(0x7fba2408a100 0x7fba24159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:16.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba23fff700 1 -- 172.21.15.175:0/3919646333 >> v2:172.21.15.191:3300/0 conn(0x7fba2408a100 msgr2=0x7fba24159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba23fff700 1 --2- 172.21.15.175:0/3919646333 >> v2:172.21.15.191:3300/0 conn(0x7fba2408a100 0x7fba24159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba23fff700 1 -- 172.21.15.175:0/3919646333 >> v1:172.21.15.175:6789/0 conn(0x7fba2414fd90 legacy=0x7fba240c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba227fc700 1 -- 172.21.15.175:0/3919646333 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191244915 0 0) 0x7fba24067fa0 con 0x7fba2414fd90 2023-10-10T19:10:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba23fff700 1 -- 172.21.15.175:0/3919646333 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba24150160 con 0x7fba240cd040 2023-10-10T19:10:16.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.522+0000 7fba23fff700 1 --2- 172.21.15.175:0/3919646333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba240684f0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fba24069990 tx=0x7fba1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6eee9e062fa9c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba227fc700 1 -- 172.21.15.175:0/3919646333 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c00b8f0 con 0x7fba240cd040 2023-10-10T19:10:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba227fc700 1 -- 172.21.15.175:0/3919646333 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1c00ba50 con 0x7fba240cd040 2023-10-10T19:10:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba227fc700 1 -- 172.21.15.175:0/3919646333 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c00bdf0 con 0x7fba240cd040 2023-10-10T19:10:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba2887e700 1 -- 172.21.15.175:0/3919646333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 msgr2=0x7fba240684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:16.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba2887e700 1 --2- 172.21.15.175:0/3919646333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba240684f0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fba24069990 tx=0x7fba1c009250 comp rx=0 tx=0).stop 2023-10-10T19:10:16.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba2887e700 1 -- 172.21.15.175:0/3919646333 shutdown_connections 2023-10-10T19:10:16.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba2887e700 1 --2- 172.21.15.175:0/3919646333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba240684f0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba2887e700 1 --2- 172.21.15.175:0/3919646333 >> v2:172.21.15.191:3300/0 conn(0x7fba2408a100 0x7fba24159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba2887e700 1 -- 172.21.15.175:0/3919646333 >> 172.21.15.175:0/3919646333 conn(0x7fba24063d00 msgr2=0x7fba240cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:16.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba2887e700 1 -- 172.21.15.175:0/3919646333 shutdown_connections 2023-10-10T19:10:16.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.523+0000 7fba2887e700 1 -- 172.21.15.175:0/3919646333 wait complete. 2023-10-10T19:10:16.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.524+0000 7fba2887e700 1 Processor -- start 2023-10-10T19:10:16.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba2887e700 1 -- start start 2023-10-10T19:10:16.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba2887e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba2408a100 0x7fba241510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:16.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba2887e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba241531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:16.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba2887e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba2414fd90 0x7fba24154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:16.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba2887e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba24150a60 con 0x7fba240cd040 2023-10-10T19:10:16.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba2887e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba240cade0 con 0x7fba2408a100 2023-10-10T19:10:16.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba2887e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2414eed0 con 0x7fba2414fd90 2023-10-10T19:10:16.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba22ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba241531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:16.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba22ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba241531e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34152/0 (socket says 172.21.15.175:34152) 2023-10-10T19:10:16.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba22ffd700 1 -- 172.21.15.175:0/675650361 learned_addr learned my addr 172.21.15.175:0/675650361 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:16.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba237fe700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba2408a100 0x7fba241510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:16.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.525+0000 7fba23fff700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba2414fd90 0x7fba24154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:16.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba22ffd700 1 -- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba2414fd90 msgr2=0x7fba24154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:16.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba22ffd700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba2414fd90 0x7fba24154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba22ffd700 1 -- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba2408a100 msgr2=0x7fba241510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:16.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba22ffd700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba2408a100 0x7fba241510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba22ffd700 1 -- 172.21.15.175:0/675650361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0094f0 con 0x7fba240cd040 2023-10-10T19:10:16.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba22ffd700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba241531e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fba1800d8c0 tx=0x7fba18008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:16.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba0b7fe700 1 -- 172.21.15.175:0/675650361 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180125f0 con 0x7fba240cd040 2023-10-10T19:10:16.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba0b7fe700 1 -- 172.21.15.175:0/675650361 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18012750 con 0x7fba240cd040 2023-10-10T19:10:16.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2414f1b0 con 0x7fba240cd040 2023-10-10T19:10:16.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba0b7fe700 1 -- 172.21.15.175:0/675650361 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18012af0 con 0x7fba240cd040 2023-10-10T19:10:16.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.526+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba240cb2b0 con 0x7fba240cd040 2023-10-10T19:10:16.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.527+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba24152f00 con 0x7fba240cd040 2023-10-10T19:10:16.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.527+0000 7fba0b7fe700 1 -- 172.21.15.175:0/675650361 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fba1801e460 con 0x7fba240cd040 2023-10-10T19:10:16.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.528+0000 7fba0b7fe700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:16.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.528+0000 7fba0b7fe700 1 -- 172.21.15.175:0/675650361 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fba18084f00 con 0x7fba240cd040 2023-10-10T19:10:16.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.528+0000 7fba237fe700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:16.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.528+0000 7fba237fe700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fba14002090 tx=0x7fba14001a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:16.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.532+0000 7fba0b7fe700 1 -- 172.21.15.175:0/675650361 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba24152f00 con 0x7fba240cd040 2023-10-10T19:10:16.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.756+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba24152f00 con 0x7fba240cd040 2023-10-10T19:10:16.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.757+0000 7fba0b7fe700 1 -- 172.21.15.175:0/675650361 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5440 (secure 0 0 0) 0x7fba24152f00 con 0x7fba240cd040 2023-10-10T19:10:16.758 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:16.759 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:12.753026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:16.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 msgr2=0x7fba10069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:16.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fba14002090 tx=0x7fba14001a30 comp rx=0 tx=0).stop 2023-10-10T19:10:16.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 msgr2=0x7fba241531e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:16.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba241531e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fba1800d8c0 tx=0x7fba18008040 comp rx=0 tx=0).stop 2023-10-10T19:10:16.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 shutdown_connections 2023-10-10T19:10:16.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba2414fd90 0x7fba24154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba2408a100 0x7fba241510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 --2- 172.21.15.175:0/675650361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba240cd040 0x7fba241531e0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:16.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 >> 172.21.15.175:0/675650361 conn(0x7fba24063d00 msgr2=0x7fba240c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:16.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 shutdown_connections 2023-10-10T19:10:16.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:16.759+0000 7fba2887e700 1 -- 172.21.15.175:0/675650361 wait complete. 2023-10-10T19:10:16.763 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 198 2023-10-10T19:10:16.772 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:10:18.773 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:18.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.899+0000 7f2b4b893700 1 Processor -- start 2023-10-10T19:10:18.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4b893700 1 -- start start 2023-10-10T19:10:18.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4b893700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2b441573d0 0x7f2b440c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:18.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4b893700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2b441577a0 0x7f2b440c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:18.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4b893700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b44154aa0 con 0x7f2b440bf6e0 2023-10-10T19:10:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4b893700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b44063dc0 con 0x7f2b441573d0 2023-10-10T19:10:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4b893700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b44157e50 con 0x7f2b441577a0 2023-10-10T19:10:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4962f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2b441573d0 0x7f2b440c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4962f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2b441573d0 0x7f2b440c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34164/0 (socket says 172.21.15.175:34164) 2023-10-10T19:10:18.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.900+0000 7f2b4962f700 1 -- 172.21.15.175:0/1250537984 learned_addr learned my addr 172.21.15.175:0/1250537984 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:18.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b48e2e700 1 --2- 172.21.15.175:0/1250537984 >> v2:172.21.15.191:3300/0 conn(0x7f2b441577a0 0x7f2b440c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:18.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b4962f700 1 -- 172.21.15.175:0/1250537984 >> v2:172.21.15.191:3300/0 conn(0x7f2b441577a0 msgr2=0x7f2b440c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:18.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b4962f700 1 --2- 172.21.15.175:0/1250537984 >> v2:172.21.15.191:3300/0 conn(0x7f2b441577a0 0x7f2b440c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:18.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b4962f700 1 -- 172.21.15.175:0/1250537984 >> v1:172.21.15.191:6789/0 conn(0x7f2b440bf6e0 legacy=0x7f2b440c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:18.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b4962f700 1 -- 172.21.15.175:0/1250537984 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b44157c20 con 0x7f2b441573d0 2023-10-10T19:10:18.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b4962f700 1 --2- 172.21.15.175:0/1250537984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b441573d0 0x7f2b440c12a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c17e0 tx=0x7f2b40009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c3b9ce9ea91996 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:18.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b3bfff700 1 -- 172.21.15.175:0/1250537984 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4000e070 con 0x7f2b441573d0 2023-10-10T19:10:18.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b3bfff700 1 -- 172.21.15.175:0/1250537984 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4000ba50 con 0x7f2b441573d0 2023-10-10T19:10:18.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.901+0000 7f2b3bfff700 1 -- 172.21.15.175:0/1250537984 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b400092e0 con 0x7f2b441573d0 2023-10-10T19:10:18.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.902+0000 7f2b4b893700 1 -- 172.21.15.175:0/1250537984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b441573d0 msgr2=0x7f2b440c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:18.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.902+0000 7f2b4b893700 1 --2- 172.21.15.175:0/1250537984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b441573d0 0x7f2b440c12a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c17e0 tx=0x7f2b40009840 comp rx=0 tx=0).stop 2023-10-10T19:10:18.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.902+0000 7f2b4b893700 1 -- 172.21.15.175:0/1250537984 shutdown_connections 2023-10-10T19:10:18.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.902+0000 7f2b4b893700 1 --2- 172.21.15.175:0/1250537984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b441573d0 0x7f2b440c12a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:18.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.902+0000 7f2b4b893700 1 --2- 172.21.15.175:0/1250537984 >> v2:172.21.15.191:3300/0 conn(0x7f2b441577a0 0x7f2b440c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:18.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.902+0000 7f2b4b893700 1 -- 172.21.15.175:0/1250537984 >> 172.21.15.175:0/1250537984 conn(0x7f2b4406dc70 msgr2=0x7f2b44063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:18.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.902+0000 7f2b4b893700 1 -- 172.21.15.175:0/1250537984 shutdown_connections 2023-10-10T19:10:18.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.902+0000 7f2b4b893700 1 -- 172.21.15.175:0/1250537984 wait complete. 2023-10-10T19:10:18.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.903+0000 7f2b4b893700 1 Processor -- start 2023-10-10T19:10:18.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.903+0000 7f2b4b893700 1 -- start start 2023-10-10T19:10:18.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4b893700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b440bf6e0 0x7f2b44146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:18.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4b893700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2b441573d0 0x7f2b44143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:18.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4b893700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2b441577a0 0x7f2b44143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:18.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4b893700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b440c6840 con 0x7f2b440bf6e0 2023-10-10T19:10:18.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4b893700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b44142e80 con 0x7f2b441577a0 2023-10-10T19:10:18.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4b893700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b440b04c0 con 0x7f2b441573d0 2023-10-10T19:10:18.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b440bf6e0 0x7f2b44146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:18.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b440bf6e0 0x7f2b44146a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34168/0 (socket says 172.21.15.175:34168) 2023-10-10T19:10:18.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 -- 172.21.15.175:0/2111765575 learned_addr learned my addr 172.21.15.175:0/2111765575 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:18.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b48e2e700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2b441573d0 0x7f2b44143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:18.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b49e30700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2b441577a0 0x7f2b44143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:18.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 -- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2b441573d0 msgr2=0x7f2b44143520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:18.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2b441573d0 0x7f2b44143520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:18.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 -- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2b441577a0 msgr2=0x7f2b44143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:18.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2b441577a0 0x7f2b44143a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:18.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 -- 172.21.15.175:0/2111765575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b40009ae0 con 0x7f2b440bf6e0 2023-10-10T19:10:18.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b49e30700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2b441577a0 0x7f2b44143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:10:18.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4962f700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b440bf6e0 0x7f2b44146a00 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2b4406e6e0 tx=0x7f2b40018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:18.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b397fa700 1 -- 172.21.15.175:0/2111765575 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b40008a80 con 0x7f2b440bf6e0 2023-10-10T19:10:18.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b397fa700 1 -- 172.21.15.175:0/2111765575 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b40008c00 con 0x7f2b440bf6e0 2023-10-10T19:10:18.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b44142c20 con 0x7f2b440bf6e0 2023-10-10T19:10:18.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b397fa700 1 -- 172.21.15.175:0/2111765575 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4001d420 con 0x7f2b440bf6e0 2023-10-10T19:10:18.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.904+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b440afe70 con 0x7f2b440bf6e0 2023-10-10T19:10:18.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.905+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b44067570 con 0x7f2b440bf6e0 2023-10-10T19:10:18.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.906+0000 7f2b397fa700 1 -- 172.21.15.175:0/2111765575 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2b40003040 con 0x7f2b440bf6e0 2023-10-10T19:10:18.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.906+0000 7f2b397fa700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2b28066f30 0x7f2b280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:18.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.906+0000 7f2b397fa700 1 -- 172.21.15.175:0/2111765575 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f2b40086760 con 0x7f2b440bf6e0 2023-10-10T19:10:18.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.906+0000 7f2b48e2e700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2b28066f30 0x7f2b280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:18.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.907+0000 7f2b48e2e700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2b28066f30 0x7f2b280693e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c6410 tx=0x7f2b34008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:18.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:18.910+0000 7f2b397fa700 1 -- 172.21.15.175:0/2111765575 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b40053500 con 0x7f2b440bf6e0 2023-10-10T19:10:19.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.140+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b44063ae0 con 0x7f2b440bf6e0 2023-10-10T19:10:19.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.141+0000 7f2b397fa700 1 -- 172.21.15.175:0/2111765575 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f2b40056b20 con 0x7f2b440bf6e0 2023-10-10T19:10:19.142 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:19.143 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:18.772686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:19.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2b28066f30 msgr2=0x7f2b280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:19.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2b28066f30 0x7f2b280693e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2b440c6410 tx=0x7f2b34008040 comp rx=0 tx=0).stop 2023-10-10T19:10:19.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b440bf6e0 msgr2=0x7f2b44146a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:19.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b440bf6e0 0x7f2b44146a00 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2b4406e6e0 tx=0x7f2b40018000 comp rx=0 tx=0).stop 2023-10-10T19:10:19.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 shutdown_connections 2023-10-10T19:10:19.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2b28066f30 0x7f2b280693e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:19.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2b441577a0 0x7f2b44143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:19.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b440bf6e0 0x7f2b44146a00 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:19.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 --2- 172.21.15.175:0/2111765575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2b441573d0 0x7f2b44143520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:19.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 >> 172.21.15.175:0/2111765575 conn(0x7f2b4406dc70 msgr2=0x7f2b440c78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:19.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 shutdown_connections 2023-10-10T19:10:19.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:19.143+0000 7f2b4b893700 1 -- 172.21.15.175:0/2111765575 wait complete. 2023-10-10T19:10:19.148 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 199 2023-10-10T19:10:19.156 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=4, replacing gid=5584 2023-10-10T19:10:19.157 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.d 2023-10-10T19:10:19.766 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:10:19.765+0000 7f5de131e700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-10T19:10:22.258 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:10:22.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:10:24.147 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 38% (83/217) Updating files: 39% (85/217) Updating files: 40% (87/217) Updating files: 41% (89/217) Updating files: 42% (92/217) Updating files: 43% (94/217) Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 91% (199/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-10T19:10:24.158 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-10T19:10:24.159 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-10T19:10:24.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-10T19:10:24.161 INFO:tasks.ceph.mds.d:Started 2023-10-10T19:10:24.162 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:24.232 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-10T19:10:24.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.277+0000 7fc849070700 1 Processor -- start 2023-10-10T19:10:24.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.277+0000 7fc849070700 1 -- start start 2023-10-10T19:10:24.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc849070700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc84414fd90 0x7fc844159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:24.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc849070700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84408a100 0x7fc8440c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:24.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc849070700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc844067fa0 con 0x7fc8440cd040 2023-10-10T19:10:24.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc849070700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8440c5600 con 0x7fc84414fd90 2023-10-10T19:10:24.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc849070700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc844156b70 con 0x7fc84408a100 2023-10-10T19:10:24.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc84886f700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc8440cd040 0x7fc8440684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38160/0 (socket says 172.21.15.175:38160) 2023-10-10T19:10:24.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc84886f700 1 -- 172.21.15.175:0/80422808 learned_addr learned my addr 172.21.15.175:0/80422808 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:10:24.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc8437fe700 1 --2- 172.21.15.175:0/80422808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84408a100 0x7fc8440c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:24.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.278+0000 7fc843fff700 1 --2- 172.21.15.175:0/80422808 >> v2:172.21.15.191:3300/0 conn(0x7fc84414fd90 0x7fc844159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:24.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc842ffd700 1 -- 172.21.15.175:0/80422808 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696838103 0 0) 0x7fc844067fa0 con 0x7fc8440cd040 2023-10-10T19:10:24.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc842ffd700 1 -- 172.21.15.175:0/80422808 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82c003570 con 0x7fc8440cd040 2023-10-10T19:10:24.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc8437fe700 1 -- 172.21.15.175:0/80422808 >> v2:172.21.15.191:3300/0 conn(0x7fc84414fd90 msgr2=0x7fc844159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:24.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc8437fe700 1 --2- 172.21.15.175:0/80422808 >> v2:172.21.15.191:3300/0 conn(0x7fc84414fd90 0x7fc844159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc8437fe700 1 -- 172.21.15.175:0/80422808 >> v1:172.21.15.175:6789/0 conn(0x7fc8440cd040 legacy=0x7fc8440684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:24.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc8437fe700 1 -- 172.21.15.175:0/80422808 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844150160 con 0x7fc84408a100 2023-10-10T19:10:24.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc843fff700 1 --2- 172.21.15.175:0/80422808 >> v2:172.21.15.191:3300/0 conn(0x7fc84414fd90 0x7fc844159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:10:24.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc8437fe700 1 --2- 172.21.15.175:0/80422808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84408a100 0x7fc8440c5eb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc8380098e0 tx=0x7fc8380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83990aca5ac4ceeb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:24.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc842ffd700 1 -- 172.21.15.175:0/80422808 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838011630 con 0x7fc84408a100 2023-10-10T19:10:24.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc842ffd700 1 -- 172.21.15.175:0/80422808 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc838011790 con 0x7fc84408a100 2023-10-10T19:10:24.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.279+0000 7fc842ffd700 1 -- 172.21.15.175:0/80422808 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838011b30 con 0x7fc84408a100 2023-10-10T19:10:24.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.280+0000 7fc849070700 1 -- 172.21.15.175:0/80422808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84408a100 msgr2=0x7fc8440c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:24.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.280+0000 7fc849070700 1 --2- 172.21.15.175:0/80422808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84408a100 0x7fc8440c5eb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc8380098e0 tx=0x7fc8380091a0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.280+0000 7fc849070700 1 -- 172.21.15.175:0/80422808 shutdown_connections 2023-10-10T19:10:24.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.280+0000 7fc849070700 1 --2- 172.21.15.175:0/80422808 >> v2:172.21.15.191:3300/0 conn(0x7fc84414fd90 0x7fc844159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.280+0000 7fc849070700 1 --2- 172.21.15.175:0/80422808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84408a100 0x7fc8440c5eb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.280+0000 7fc849070700 1 -- 172.21.15.175:0/80422808 >> 172.21.15.175:0/80422808 conn(0x7fc844063d00 msgr2=0x7fc8440cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:24.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.280+0000 7fc849070700 1 -- 172.21.15.175:0/80422808 shutdown_connections 2023-10-10T19:10:24.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.280+0000 7fc849070700 1 -- 172.21.15.175:0/80422808 wait complete. 2023-10-10T19:10:24.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.281+0000 7fc849070700 1 Processor -- start 2023-10-10T19:10:24.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.281+0000 7fc849070700 1 -- start start 2023-10-10T19:10:24.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.281+0000 7fc849070700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc84408a100 0x7fc8440c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:24.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc849070700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8440cd040 0x7fc844152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:24.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc849070700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84414fd90 0x7fc84414b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:24.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc849070700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc844166b80 con 0x7fc8440cd040 2023-10-10T19:10:24.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc849070700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc844151ab0 con 0x7fc84408a100 2023-10-10T19:10:24.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc849070700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440c71c0 con 0x7fc84414fd90 2023-10-10T19:10:24.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8440cd040 0x7fc844152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:24.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8440cd040 0x7fc844152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34170/0 (socket says 172.21.15.175:34170) 2023-10-10T19:10:24.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 -- 172.21.15.175:0/4284367261 learned_addr learned my addr 172.21.15.175:0/4284367261 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:24.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc84886f700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84414fd90 0x7fc84414b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:24.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc843fff700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc84408a100 0x7fc8440c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:24.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 -- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84414fd90 msgr2=0x7fc84414b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:24.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84414fd90 0x7fc84414b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 -- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc84408a100 msgr2=0x7fc8440c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:24.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc84408a100 0x7fc8440c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 -- 172.21.15.175:0/4284367261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc834009420 con 0x7fc8440cd040 2023-10-10T19:10:24.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc8437fe700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8440cd040 0x7fc844152dc0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc844156cb0 tx=0x7fc838017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:24.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc82bfff700 1 -- 172.21.15.175:0/4284367261 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838011b10 con 0x7fc8440cd040 2023-10-10T19:10:24.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc82bfff700 1 -- 172.21.15.175:0/4284367261 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8380233f0 con 0x7fc8440cd040 2023-10-10T19:10:24.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc838009420 con 0x7fc8440cd040 2023-10-10T19:10:24.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc82bfff700 1 -- 172.21.15.175:0/4284367261 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838023770 con 0x7fc8440cd040 2023-10-10T19:10:24.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.282+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8441662e0 con 0x7fc8440cd040 2023-10-10T19:10:24.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.283+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc844151880 con 0x7fc8440cd040 2023-10-10T19:10:24.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.284+0000 7fc82bfff700 1 -- 172.21.15.175:0/4284367261 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc83802c3b0 con 0x7fc8440cd040 2023-10-10T19:10:24.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.284+0000 7fc82bfff700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc830066f30 0x7fc8300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:24.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.284+0000 7fc82bfff700 1 -- 172.21.15.175:0/4284367261 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fc838085e10 con 0x7fc8440cd040 2023-10-10T19:10:24.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.284+0000 7fc843fff700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc830066f30 0x7fc8300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:24.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.285+0000 7fc843fff700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc830066f30 0x7fc8300693e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc8340093f0 tx=0x7fc834008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:24.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.289+0000 7fc82bfff700 1 -- 172.21.15.175:0/4284367261 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc838052c30 con 0x7fc8440cd040 2023-10-10T19:10:24.490 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 98f5be53f85 v0.19 2023-10-10T19:10:24.491 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.19 2023-10-10T19:10:24.521 INFO:tasks.workunit.client.0.smithi175.stdout:v0.20 2023-10-10T19:10:24.522 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:10:24.522 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.20 2023-10-10T19:10:24.522 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.20 2023-10-10T19:10:24.523 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.20 2023-10-10T19:10:24.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.526+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc844067570 con 0x7fc8440cd040 2023-10-10T19:10:24.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.526+0000 7fc82bfff700 1 -- 172.21.15.175:0/4284367261 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5440 (secure 0 0 0) 0x7fc838028090 con 0x7fc8440cd040 2023-10-10T19:10:24.528 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:24.528 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:18.772686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:24.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc830066f30 msgr2=0x7fc8300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:24.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc830066f30 0x7fc8300693e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc8340093f0 tx=0x7fc834008040 comp rx=0 tx=0).stop 2023-10-10T19:10:24.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8440cd040 msgr2=0x7fc844152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:24.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8440cd040 0x7fc844152dc0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc844156cb0 tx=0x7fc838017000 comp rx=0 tx=0).stop 2023-10-10T19:10:24.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 shutdown_connections 2023-10-10T19:10:24.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc830066f30 0x7fc8300693e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc84414fd90 0x7fc84414b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc84408a100 0x7fc8440c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 --2- 172.21.15.175:0/4284367261 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8440cd040 0x7fc844152dc0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:24.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.528+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 >> 172.21.15.175:0/4284367261 conn(0x7fc844063d00 msgr2=0x7fc8440c2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:24.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.529+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 shutdown_connections 2023-10-10T19:10:24.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:24.529+0000 7fc849070700 1 -- 172.21.15.175:0/4284367261 wait complete. 2023-10-10T19:10:24.533 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 199 2023-10-10T19:10:24.543 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-10T19:10:26.544 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:26.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.658+0000 7f5eed9f8700 1 Processor -- start 2023-10-10T19:10:26.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.658+0000 7f5eed9f8700 1 -- start start 2023-10-10T19:10:26.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5eed9f8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c2cf0 0x7f5ee806e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:26.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5eed9f8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5ee80c30c0 0x7f5ee814db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:26.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5eed9f8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5ee80c0270 0x7f5ee814e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:26.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5eed9f8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee814e600 con 0x7f5ee80c0270 2023-10-10T19:10:26.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5eed9f8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee8153f70 con 0x7f5ee80c30c0 2023-10-10T19:10:26.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5eed9f8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee8067bd0 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5ee77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5ee80c0270 0x7f5ee814e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:26.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5ee77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5ee80c0270 0x7f5ee814e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46488/0 (socket says 172.21.15.175:46488) 2023-10-10T19:10:26.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.659+0000 7f5ee77fe700 1 -- 172.21.15.175:0/172853574 learned_addr learned my addr 172.21.15.175:0/172853574 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:26.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee6ffd700 1 --2- 172.21.15.175:0/172853574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c2cf0 0x7f5ee806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:26.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee67fc700 1 --2- 172.21.15.175:0/172853574 >> v2:172.21.15.191:3300/0 conn(0x7f5ee80c30c0 0x7f5ee814db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:26.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee77fe700 1 -- 172.21.15.175:0/172853574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c2cf0 msgr2=0x7f5ee806e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:26.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee77fe700 1 --2- 172.21.15.175:0/172853574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c2cf0 0x7f5ee806e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee77fe700 1 -- 172.21.15.175:0/172853574 >> v2:172.21.15.191:3300/0 conn(0x7f5ee80c30c0 msgr2=0x7f5ee814db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:26.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee77fe700 1 --2- 172.21.15.175:0/172853574 >> v2:172.21.15.191:3300/0 conn(0x7f5ee80c30c0 0x7f5ee814db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee77fe700 1 -- 172.21.15.175:0/172853574 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8154210 con 0x7f5ee80c0270 2023-10-10T19:10:26.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee77fe700 1 --2- 172.21.15.175:0/172853574 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c0270 0x7f5ee814e0c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80c6f90 tx=0x7f5ee0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a748483dd7df130e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:26.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee5ffb700 1 -- 172.21.15.175:0/172853574 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee000b8f0 con 0x7f5ee80c0270 2023-10-10T19:10:26.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee5ffb700 1 -- 172.21.15.175:0/172853574 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee000ba50 con 0x7f5ee80c0270 2023-10-10T19:10:26.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5ee5ffb700 1 -- 172.21.15.175:0/172853574 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee000bdf0 con 0x7f5ee80c0270 2023-10-10T19:10:26.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5eed9f8700 1 -- 172.21.15.175:0/172853574 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c0270 msgr2=0x7f5ee814e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:26.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.660+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/172853574 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c0270 0x7f5ee814e0c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80c6f90 tx=0x7f5ee0009250 comp rx=0 tx=0).stop 2023-10-10T19:10:26.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.661+0000 7f5eed9f8700 1 -- 172.21.15.175:0/172853574 shutdown_connections 2023-10-10T19:10:26.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.661+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/172853574 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c0270 0x7f5ee814e0c0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.661+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/172853574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c2cf0 0x7f5ee806e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.661+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/172853574 >> v2:172.21.15.191:3300/0 conn(0x7f5ee80c30c0 0x7f5ee814db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.661+0000 7f5eed9f8700 1 -- 172.21.15.175:0/172853574 >> 172.21.15.175:0/172853574 conn(0x7f5ee8068900 msgr2=0x7f5ee8068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:26.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.661+0000 7f5eed9f8700 1 -- 172.21.15.175:0/172853574 shutdown_connections 2023-10-10T19:10:26.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.661+0000 7f5eed9f8700 1 -- 172.21.15.175:0/172853574 wait complete. 2023-10-10T19:10:26.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.662+0000 7f5eed9f8700 1 Processor -- start 2023-10-10T19:10:26.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.662+0000 7f5eed9f8700 1 -- start start 2023-10-10T19:10:26.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eed9f8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ee80c0270 0x7f5ee8149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:26.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eed9f8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c2cf0 0x7f5ee8145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:26.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eed9f8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c30c0 0x7f5ee8164e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:26.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eed9f8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee81672d0 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eed9f8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee81639d0 con 0x7f5ee80c0270 2023-10-10T19:10:26.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eed9f8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee80ca330 con 0x7f5ee80c30c0 2023-10-10T19:10:26.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c2cf0 0x7f5ee8145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:26.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c2cf0 0x7f5ee8145050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46490/0 (socket says 172.21.15.175:46490) 2023-10-10T19:10:26.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 -- 172.21.15.175:0/1651688921 learned_addr learned my addr 172.21.15.175:0/1651688921 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:26.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee77fe700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c30c0 0x7f5ee8164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:26.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee6ffd700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ee80c0270 0x7f5ee8149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:26.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 -- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c30c0 msgr2=0x7f5ee8164e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:26.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c30c0 0x7f5ee8164e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 -- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ee80c0270 msgr2=0x7f5ee8149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:26.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ee80c0270 0x7f5ee8149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 -- 172.21.15.175:0/1651688921 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee00094f0 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5ee67fc700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c2cf0 0x7f5ee8145050 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00d890 tx=0x7f5edc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:26.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eceffd700 1 -- 172.21.15.175:0/1651688921 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5edc0125f0 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eceffd700 1 -- 172.21.15.175:0/1651688921 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5edc012750 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eceffd700 1 -- 172.21.15.175:0/1651688921 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5edc012af0 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee8164510 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.663+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee8163be0 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.664+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee81646a0 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.665+0000 7f5eceffd700 1 -- 172.21.15.175:0/1651688921 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5edc01e460 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.665+0000 7f5eceffd700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:26.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.665+0000 7f5eceffd700 1 -- 172.21.15.175:0/1651688921 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f5edc084f70 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.665+0000 7f5ee6ffd700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:26.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.666+0000 7f5ee6ffd700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8007680 tx=0x7f5ed80098d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:26.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.669+0000 7f5eceffd700 1 -- 172.21.15.175:0/1651688921 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5edc051d10 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.901+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ee8067570 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.901+0000 7f5eceffd700 1 -- 172.21.15.175:0/1651688921 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+6200 (secure 0 0 0) 0x7f5edc055330 con 0x7f5ee80c2cf0 2023-10-10T19:10:26.903 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:26.903 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:18.772686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:26.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 msgr2=0x7f5ed40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:26.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8007680 tx=0x7f5ed80098d0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c2cf0 msgr2=0x7f5ee8145050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:26.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c2cf0 0x7f5ee8145050 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00d890 tx=0x7f5edc008040 comp rx=0 tx=0).stop 2023-10-10T19:10:26.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 shutdown_connections 2023-10-10T19:10:26.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ee80c30c0 0x7f5ee8164e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ee80c0270 0x7f5ee8149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 --2- 172.21.15.175:0/1651688921 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ee80c2cf0 0x7f5ee8145050 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:26.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 >> 172.21.15.175:0/1651688921 conn(0x7f5ee8068900 msgr2=0x7f5ee814cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:26.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 shutdown_connections 2023-10-10T19:10:26.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:26.903+0000 7f5eed9f8700 1 -- 172.21.15.175:0/1651688921 wait complete. 2023-10-10T19:10:26.907 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 200 2023-10-10T19:10:26.916 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-10T19:10:26.916 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 22 secs before thrashing 2023-10-10T19:10:34.060 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 6% (15/245) Updating files: 7% (18/245) Updating files: 8% (20/245) Updating files: 9% (23/245) Updating files: 10% (25/245) Updating files: 11% (27/245) Updating files: 12% (30/245) Updating files: 13% (32/245) Updating files: 14% (35/245) Updating files: 15% (37/245) Updating files: 16% (40/245) Updating files: 17% (42/245) Updating files: 18% (45/245) Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 91% (225/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-10T19:10:34.080 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2023-10-10T19:10:34.081 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.20 2023-10-10T19:10:34.136 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:10:34.137 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.21 2023-10-10T19:10:34.137 INFO:tasks.workunit.client.0.smithi175.stdout:v0.21 2023-10-10T19:10:34.138 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo v0.21 2023-10-10T19:10:34.138 INFO:tasks.workunit.client.0.smithi175.stderr:+ git reset --hard v0.21 2023-10-10T19:10:39.512 INFO:tasks.workunit.client.0.smithi175.stderr:Updating files: 16% (91/537) Updating files: 17% (92/537) Updating files: 18% (97/537) Updating files: 19% (103/537) Updating files: 20% (108/537) Updating files: 21% (113/537) Updating files: 22% (119/537) Updating files: 23% (124/537) Updating files: 24% (129/537) Updating files: 25% (135/537) Updating files: 26% (140/537) Updating files: 27% (145/537) Updating files: 28% (151/537) Updating files: 29% (156/537) Updating files: 30% (162/537) Updating files: 31% (167/537) Updating files: 32% (172/537) Updating files: 33% (178/537) Updating files: 34% (183/537) Updating files: 35% (188/537) Updating files: 36% (194/537) Updating files: 37% (199/537) Updating files: 38% (205/537) Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 40% (219/537) Updating files: 41% (221/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 46% (248/537) Updating files: 47% (253/537) Updating files: 48% (258/537) Updating files: 49% (264/537) Updating files: 50% (269/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 66% (359/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 83% (447/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2023-10-10T19:10:39.526 INFO:tasks.workunit.client.0.smithi175.stdout:HEAD is now at 090436f56ac v0.21 2023-10-10T19:10:39.528 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/v0.21 2023-10-10T19:10:39.600 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:10:39.600 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.1 2023-10-10T19:10:39.600 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.1 2023-10-10T19:10:39.601 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.1 2023-10-10T19:10:39.601 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.1 2023-10-10T19:10:39.601 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:10:45.948 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:10:45.949 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:10:45.949 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.2 2023-10-10T19:10:45.949 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.2 2023-10-10T19:10:45.950 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.2 2023-10-10T19:10:45.951 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.2 2023-10-10T19:10:45.951 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:10:48.918 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:49.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.040+0000 7f6e5eb49700 1 Processor -- start 2023-10-10T19:10:49.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.040+0000 7f6e5eb49700 1 -- start start 2023-10-10T19:10:49.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5eb49700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6e5808a100 0x7f6e58159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5eb49700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6e580cd040 0x7f6e580684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5eb49700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e58067fa0 con 0x7f6e5814fd90 2023-10-10T19:10:49.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5eb49700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e580c5600 con 0x7f6e5808a100 2023-10-10T19:10:49.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5eb49700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e58156b70 con 0x7f6e580cd040 2023-10-10T19:10:49.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5d346700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6e5808a100 0x7f6e58159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5d346700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6e5808a100 0x7f6e58159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48848/0 (socket says 172.21.15.175:48848) 2023-10-10T19:10:49.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5d346700 1 -- 172.21.15.175:0/1639738047 learned_addr learned my addr 172.21.15.175:0/1639738047 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:49.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5e348700 1 --2- 172.21.15.175:0/1639738047 >> v2:172.21.15.191:3300/0 conn(0x7f6e580cd040 0x7f6e580684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5d346700 1 -- 172.21.15.175:0/1639738047 >> v2:172.21.15.191:3300/0 conn(0x7f6e580cd040 msgr2=0x7f6e580684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:49.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5d346700 1 --2- 172.21.15.175:0/1639738047 >> v2:172.21.15.191:3300/0 conn(0x7f6e580cd040 0x7f6e580684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5d346700 1 -- 172.21.15.175:0/1639738047 >> v1:172.21.15.191:6789/0 conn(0x7f6e5814fd90 legacy=0x7f6e580c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.041+0000 7f6e5d346700 1 -- 172.21.15.175:0/1639738047 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e58150160 con 0x7f6e5808a100 2023-10-10T19:10:49.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5d346700 1 --2- 172.21.15.175:0/1639738047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58159c30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6e5808a6e0 tx=0x7f6e48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00816a3a5ef8b3a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:49.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5cb45700 1 -- 172.21.15.175:0/1639738047 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48009e70 con 0x7f6e5808a100 2023-10-10T19:10:49.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5cb45700 1 -- 172.21.15.175:0/1639738047 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4800b8f0 con 0x7f6e5808a100 2023-10-10T19:10:49.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5cb45700 1 -- 172.21.15.175:0/1639738047 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4800bc90 con 0x7f6e5808a100 2023-10-10T19:10:49.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5eb49700 1 -- 172.21.15.175:0/1639738047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 msgr2=0x7f6e58159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/1639738047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58159c30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6e5808a6e0 tx=0x7f6e48009250 comp rx=0 tx=0).stop 2023-10-10T19:10:49.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5eb49700 1 -- 172.21.15.175:0/1639738047 shutdown_connections 2023-10-10T19:10:49.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/1639738047 >> v2:172.21.15.191:3300/0 conn(0x7f6e580cd040 0x7f6e580684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/1639738047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58159c30 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5eb49700 1 -- 172.21.15.175:0/1639738047 >> 172.21.15.175:0/1639738047 conn(0x7f6e58063d00 msgr2=0x7f6e580cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:49.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5eb49700 1 -- 172.21.15.175:0/1639738047 shutdown_connections 2023-10-10T19:10:49.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.042+0000 7f6e5eb49700 1 -- 172.21.15.175:0/1639738047 wait complete. 2023-10-10T19:10:49.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.043+0000 7f6e5eb49700 1 Processor -- start 2023-10-10T19:10:49.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.043+0000 7f6e5eb49700 1 -- start start 2023-10-10T19:10:49.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5eb49700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58165ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5eb49700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e580cd040 0x7f6e58152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5eb49700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e5814fd90 0x7f6e58169050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5eb49700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e581535c0 con 0x7f6e5808a100 2023-10-10T19:10:49.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5eb49700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e580a7d40 con 0x7f6e580cd040 2023-10-10T19:10:49.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5eb49700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e58153300 con 0x7f6e5814fd90 2023-10-10T19:10:49.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5db47700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58165ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5db47700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58165ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48858/0 (socket says 172.21.15.175:48858) 2023-10-10T19:10:49.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5db47700 1 -- 172.21.15.175:0/15528448 learned_addr learned my addr 172.21.15.175:0/15528448 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:49.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5e348700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e5814fd90 0x7f6e58169050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.044+0000 7f6e5d346700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e580cd040 0x7f6e58152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e5db47700 1 -- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e5814fd90 msgr2=0x7f6e58169050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e5db47700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e5814fd90 0x7f6e58169050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e5db47700 1 -- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e580cd040 msgr2=0x7f6e58152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e5db47700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e580cd040 0x7f6e58152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e5db47700 1 -- 172.21.15.175:0/15528448 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480094f0 con 0x7f6e5808a100 2023-10-10T19:10:49.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e5db47700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58165ad0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c4e0 tx=0x7f6e5400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:49.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e4dffb700 1 -- 172.21.15.175:0/15528448 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54011590 con 0x7f6e5808a100 2023-10-10T19:10:49.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e4dffb700 1 -- 172.21.15.175:0/15528448 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54011710 con 0x7f6e5808a100 2023-10-10T19:10:49.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e580c86f0 con 0x7f6e5808a100 2023-10-10T19:10:49.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e4dffb700 1 -- 172.21.15.175:0/15528448 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54011ad0 con 0x7f6e5808a100 2023-10-10T19:10:49.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.045+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e580c9b30 con 0x7f6e5808a100 2023-10-10T19:10:49.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.046+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e58067570 con 0x7f6e5808a100 2023-10-10T19:10:49.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.046+0000 7f6e4dffb700 1 -- 172.21.15.175:0/15528448 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6e54000f90 con 0x7f6e5808a100 2023-10-10T19:10:49.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.047+0000 7f6e4dffb700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e3c066f30 0x7f6e3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.047+0000 7f6e4dffb700 1 -- 172.21.15.175:0/15528448 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f6e54083cd0 con 0x7f6e5808a100 2023-10-10T19:10:49.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.047+0000 7f6e5d346700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e3c066f30 0x7f6e3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.048+0000 7f6e5d346700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e3c066f30 0x7f6e3c0693e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6e5808a6e0 tx=0x7f6e48017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:49.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.051+0000 7f6e4dffb700 1 -- 172.21.15.175:0/15528448 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e54050a70 con 0x7f6e5808a100 2023-10-10T19:10:49.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.290+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e58155f80 con 0x7f6e5808a100 2023-10-10T19:10:49.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.290+0000 7f6e4dffb700 1 -- 172.21.15.175:0/15528448 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+6211 (secure 0 0 0) 0x7f6e54017070 con 0x7f6e5808a100 2023-10-10T19:10:49.291 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:49.291 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:44.692939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:49.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e3c066f30 msgr2=0x7f6e3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e3c066f30 0x7f6e3c0693e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6e5808a6e0 tx=0x7f6e48017000 comp rx=0 tx=0).stop 2023-10-10T19:10:49.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 msgr2=0x7f6e58165ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58165ad0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c4e0 tx=0x7f6e5400c7f0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 shutdown_connections 2023-10-10T19:10:49.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6e3c066f30 0x7f6e3c0693e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e5814fd90 0x7f6e58169050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e5808a100 0x7f6e58165ad0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 --2- 172.21.15.175:0/15528448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e580cd040 0x7f6e58152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 >> 172.21.15.175:0/15528448 conn(0x7f6e58063d00 msgr2=0x7f6e580cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:49.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 shutdown_connections 2023-10-10T19:10:49.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.292+0000 7f6e5eb49700 1 -- 172.21.15.175:0/15528448 wait complete. 2023-10-10T19:10:49.295 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 203 2023-10-10T19:10:49.305 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-10T19:10:49.306 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-10T19:10:49.306 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:10:49.317 INFO:tasks.ceph.mds.e:Stopped 2023-10-10T19:10:49.318 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:49.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.473+0000 7fb29a3d4700 1 Processor -- start 2023-10-10T19:10:49.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.473+0000 7fb29a3d4700 1 -- start start 2023-10-10T19:10:49.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.474+0000 7fb29a3d4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.474+0000 7fb29a3d4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb2940c4f20 0x7fb2940c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.474+0000 7fb29a3d4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2940c6fb0 con 0x7fb2940c12d0 2023-10-10T19:10:49.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.474+0000 7fb29a3d4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb294063ea0 con 0x7fb2940c4f20 2023-10-10T19:10:49.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.474+0000 7fb29a3d4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb294155f30 con 0x7fb2940c5850 2023-10-10T19:10:49.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb293fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb298971700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb2940c4f20 0x7fb2940c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb293fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:41198/0 (socket says 172.21.15.175:41198) 2023-10-10T19:10:49.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb293fff700 1 -- 172.21.15.175:0/1938824375 learned_addr learned my addr 172.21.15.175:0/1938824375 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:49.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb298971700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb2940c4f20 0x7fb2940c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:52820/0 (socket says 172.21.15.175:52820) 2023-10-10T19:10:49.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb298971700 1 -- 172.21.15.175:0/1938824375 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 msgr2=0x7fb2940c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:49.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb298971700 1 --2- 172.21.15.175:0/1938824375 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb298971700 1 -- 172.21.15.175:0/1938824375 >> v1:172.21.15.191:6789/0 conn(0x7fb2940c12d0 legacy=0x7fb294156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb298971700 1 -- 172.21.15.175:0/1938824375 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2940c6160 con 0x7fb2940c4f20 2023-10-10T19:10:49.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb292ffd700 1 -- 172.21.15.175:0/1938824375 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260588502 0 0) 0x7fb2940c6fb0 con 0x7fb2940c12d0 2023-10-10T19:10:49.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb298971700 1 --2- 172.21.15.175:0/1938824375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c4f20 0x7fb2940c7370 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009e60 tx=0x7fb28c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90c12dcb32487db4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:49.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb292ffd700 1 -- 172.21.15.175:0/1938824375 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c00e070 con 0x7fb2940c4f20 2023-10-10T19:10:49.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb292ffd700 1 -- 172.21.15.175:0/1938824375 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28c00bbf0 con 0x7fb2940c4f20 2023-10-10T19:10:49.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.475+0000 7fb292ffd700 1 -- 172.21.15.175:0/1938824375 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c012430 con 0x7fb2940c4f20 2023-10-10T19:10:49.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.476+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1938824375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c4f20 msgr2=0x7fb2940c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.476+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1938824375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c4f20 0x7fb2940c7370 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009e60 tx=0x7fb28c009780 comp rx=0 tx=0).stop 2023-10-10T19:10:49.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.476+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1938824375 shutdown_connections 2023-10-10T19:10:49.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.476+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1938824375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c4f20 0x7fb2940c7370 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.476+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1938824375 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.476+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1938824375 >> 172.21.15.175:0/1938824375 conn(0x7fb29406dc70 msgr2=0x7fb294063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:49.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.476+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1938824375 shutdown_connections 2023-10-10T19:10:49.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.476+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1938824375 wait complete. 2023-10-10T19:10:49.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.477+0000 7fb29a3d4700 1 Processor -- start 2023-10-10T19:10:49.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.477+0000 7fb29a3d4700 1 -- start start 2023-10-10T19:10:49.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb29a3d4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c12d0 0x7fb294157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb29a3d4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2940c4f20 0x7fb2940c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb29a3d4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb29a3d4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2941500e0 con 0x7fb2940c4f20 2023-10-10T19:10:49.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb29a3d4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294158b80 con 0x7fb2940c12d0 2023-10-10T19:10:49.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb29a3d4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2940cae40 con 0x7fb2940c5850 2023-10-10T19:10:49.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2940c4f20 0x7fb2940c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2940c4f20 0x7fb2940c8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48860/0 (socket says 172.21.15.175:48860) 2023-10-10T19:10:49.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 -- 172.21.15.175:0/1368772338 learned_addr learned my addr 172.21.15.175:0/1368772338 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:49.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb293fff700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c12d0 0x7fb294157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb298971700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 -- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 msgr2=0x7fb2940c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 -- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c12d0 msgr2=0x7fb294157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c12d0 0x7fb294157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 -- 172.21.15.175:0/1368772338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c009a20 con 0x7fb2940c4f20 2023-10-10T19:10:49.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb293fff700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c12d0 0x7fb294157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:10:49.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.478+0000 7fb2937fe700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2940c4f20 0x7fb2940c8970 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fb28800c560 tx=0x7fb28800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:49.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.479+0000 7fb27bfff700 1 -- 172.21.15.175:0/1368772338 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb288011590 con 0x7fb2940c4f20 2023-10-10T19:10:49.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.479+0000 7fb27bfff700 1 -- 172.21.15.175:0/1368772338 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb288011710 con 0x7fb2940c4f20 2023-10-10T19:10:49.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.479+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29414fe00 con 0x7fb2940c4f20 2023-10-10T19:10:49.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.479+0000 7fb27bfff700 1 -- 172.21.15.175:0/1368772338 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb288011ad0 con 0x7fb2940c4f20 2023-10-10T19:10:49.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.479+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb294157ba0 con 0x7fb2940c4f20 2023-10-10T19:10:49.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.479+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2940cb2a0 con 0x7fb2940c4f20 2023-10-10T19:10:49.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.480+0000 7fb27bfff700 1 -- 172.21.15.175:0/1368772338 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb288001010 con 0x7fb2940c4f20 2023-10-10T19:10:49.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.480+0000 7fb27bfff700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb280066f30 0x7fb2800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:49.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.480+0000 7fb27bfff700 1 -- 172.21.15.175:0/1368772338 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fb288083d60 con 0x7fb2940c4f20 2023-10-10T19:10:49.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.481+0000 7fb293fff700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb280066f30 0x7fb2800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:49.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.481+0000 7fb293fff700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb280066f30 0x7fb2800693e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb284006d40 tx=0x7fb284008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:49.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.485+0000 7fb27bfff700 1 -- 172.21.15.175:0/1368772338 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb28804b5a0 con 0x7fb2940c4f20 2023-10-10T19:10:49.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.715+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb294158d30 con 0x7fb2940c4f20 2023-10-10T19:10:49.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.716+0000 7fb27bfff700 1 -- 172.21.15.175:0/1368772338 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+6211 (secure 0 0 0) 0x7fb294158d30 con 0x7fb2940c4f20 2023-10-10T19:10:49.716 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:49.717 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:44.692939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:49.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb280066f30 msgr2=0x7fb2800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb280066f30 0x7fb2800693e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb284006d40 tx=0x7fb284008040 comp rx=0 tx=0).stop 2023-10-10T19:10:49.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2940c4f20 msgr2=0x7fb2940c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:49.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2940c4f20 0x7fb2940c8970 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fb28800c560 tx=0x7fb28800c920 comp rx=0 tx=0).stop 2023-10-10T19:10:49.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 shutdown_connections 2023-10-10T19:10:49.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb280066f30 0x7fb2800693e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2940c5850 0x7fb2940c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2940c12d0 0x7fb294157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 --2- 172.21.15.175:0/1368772338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2940c4f20 0x7fb2940c8970 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:49.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 >> 172.21.15.175:0/1368772338 conn(0x7fb29406dc70 msgr2=0x7fb2940c6730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:49.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 shutdown_connections 2023-10-10T19:10:49.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:49.718+0000 7fb29a3d4700 1 -- 172.21.15.175:0/1368772338 wait complete. 2023-10-10T19:10:49.722 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 203 2023-10-10T19:10:49.730 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:10:51.732 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:51.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.855+0000 7faea3395700 1 Processor -- start 2023-10-10T19:10:51.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.856+0000 7faea3395700 1 -- start start 2023-10-10T19:10:51.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.856+0000 7faea3395700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 0x7fae9c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:51.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.856+0000 7faea3395700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fae9c0b2390 0x7fae9c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:51.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.856+0000 7faea3395700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c1574b0 con 0x7fae9c068bd0 2023-10-10T19:10:51.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.856+0000 7faea3395700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c150770 con 0x7fae9c0b2390 2023-10-10T19:10:51.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.856+0000 7faea3395700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0afad0 con 0x7fae9c0b1fc0 2023-10-10T19:10:51.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea0930700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fae9c0b2390 0x7fae9c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:51.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea0930700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fae9c0b2390 0x7fae9c068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48872/0 (socket says 172.21.15.175:48872) 2023-10-10T19:10:51.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea0930700 1 -- 172.21.15.175:0/496902633 learned_addr learned my addr 172.21.15.175:0/496902633 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:51.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea0930700 1 -- 172.21.15.175:0/496902633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 msgr2=0x7fae9c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:51.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea1131700 1 --2- 172.21.15.175:0/496902633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 0x7fae9c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:51.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea0930700 1 --2- 172.21.15.175:0/496902633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 0x7fae9c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:51.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea0930700 1 -- 172.21.15.175:0/496902633 >> v1:172.21.15.191:6789/0 conn(0x7fae9c068bd0 legacy=0x7fae9c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:51.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea0930700 1 -- 172.21.15.175:0/496902633 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c0af4c0 con 0x7fae9c0b2390 2023-10-10T19:10:51.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7faea0930700 1 --2- 172.21.15.175:0/496902633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c068690 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0099b0 tx=0x7fae8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beda0ab0be827b1f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:51.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7fae93fff700 1 -- 172.21.15.175:0/496902633 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c00b8b0 con 0x7fae9c0b2390 2023-10-10T19:10:51.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7fae93fff700 1 -- 172.21.15.175:0/496902633 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c00ba10 con 0x7fae9c0b2390 2023-10-10T19:10:51.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.857+0000 7fae93fff700 1 -- 172.21.15.175:0/496902633 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c00bdb0 con 0x7fae9c0b2390 2023-10-10T19:10:51.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.858+0000 7faea3395700 1 -- 172.21.15.175:0/496902633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 msgr2=0x7fae9c068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:51.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.858+0000 7faea3395700 1 --2- 172.21.15.175:0/496902633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c068690 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0099b0 tx=0x7fae8c009250 comp rx=0 tx=0).stop 2023-10-10T19:10:51.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.858+0000 7faea3395700 1 -- 172.21.15.175:0/496902633 shutdown_connections 2023-10-10T19:10:51.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.858+0000 7faea3395700 1 --2- 172.21.15.175:0/496902633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 0x7fae9c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:51.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.858+0000 7faea3395700 1 --2- 172.21.15.175:0/496902633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c068690 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:51.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.858+0000 7faea3395700 1 -- 172.21.15.175:0/496902633 >> 172.21.15.175:0/496902633 conn(0x7fae9c063230 msgr2=0x7fae9c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:51.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.858+0000 7faea3395700 1 -- 172.21.15.175:0/496902633 shutdown_connections 2023-10-10T19:10:51.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.858+0000 7faea3395700 1 -- 172.21.15.175:0/496902633 wait complete. 2023-10-10T19:10:51.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.859+0000 7faea3395700 1 Processor -- start 2023-10-10T19:10:51.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.859+0000 7faea3395700 1 -- start start 2023-10-10T19:10:51.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea3395700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c068bd0 0x7fae9c155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:51.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea3395700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 0x7fae9c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:51.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea3395700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:51.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea3395700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c166e30 con 0x7fae9c0b2390 2023-10-10T19:10:51.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea3395700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0cbee0 con 0x7fae9c068bd0 2023-10-10T19:10:51.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea3395700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0cd330 con 0x7fae9c0b1fc0 2023-10-10T19:10:51.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:51.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c0ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48880/0 (socket says 172.21.15.175:48880) 2023-10-10T19:10:51.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 -- 172.21.15.175:0/707593159 learned_addr learned my addr 172.21.15.175:0/707593159 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:51.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1131700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c068bd0 0x7fae9c155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:51.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea0930700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 0x7fae9c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:51.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 -- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 msgr2=0x7fae9c155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:51.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 0x7fae9c155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:51.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 -- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c068bd0 msgr2=0x7fae9c155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:51.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c068bd0 0x7fae9c155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:51.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 -- 172.21.15.175:0/707593159 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0094f0 con 0x7fae9c0b2390 2023-10-10T19:10:51.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea1932700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c0ca910 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fae9400c510 tx=0x7fae94007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:51.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7fae917fa700 1 -- 172.21.15.175:0/707593159 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94011590 con 0x7fae9c0b2390 2023-10-10T19:10:51.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7fae917fa700 1 -- 172.21.15.175:0/707593159 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae940116f0 con 0x7fae9c0b2390 2023-10-10T19:10:51.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c0cd610 con 0x7fae9c0b2390 2023-10-10T19:10:51.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7fae917fa700 1 -- 172.21.15.175:0/707593159 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94011ab0 con 0x7fae9c0b2390 2023-10-10T19:10:51.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.860+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c156930 con 0x7fae9c0b2390 2023-10-10T19:10:51.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.861+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c0ca630 con 0x7fae9c0b2390 2023-10-10T19:10:51.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.862+0000 7fae917fa700 1 -- 172.21.15.175:0/707593159 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fae9401c460 con 0x7fae9c0b2390 2023-10-10T19:10:51.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.862+0000 7fae917fa700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae80066f60 0x7fae80069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:51.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.862+0000 7fae917fa700 1 -- 172.21.15.175:0/707593159 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fae940845c0 con 0x7fae9c0b2390 2023-10-10T19:10:51.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.862+0000 7faea1131700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae80066f60 0x7fae80069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:51.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.863+0000 7faea1131700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae80066f60 0x7fae80069410 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fae9800f610 tx=0x7fae980077f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:51.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:51.866+0000 7fae917fa700 1 -- 172.21.15.175:0/707593159 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fae9c0ca630 con 0x7fae9c0b2390 2023-10-10T19:10:52.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.093+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae9c0ca630 con 0x7fae9c0b2390 2023-10-10T19:10:52.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.093+0000 7fae917fa700 1 -- 172.21.15.175:0/707593159 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+6211 (secure 0 0 0) 0x7fae9c0ca630 con 0x7fae9c0b2390 2023-10-10T19:10:52.094 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:52.094 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:44.692939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:52.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae80066f60 msgr2=0x7fae80069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:52.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae80066f60 0x7fae80069410 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fae9800f610 tx=0x7fae980077f0 comp rx=0 tx=0).stop 2023-10-10T19:10:52.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 msgr2=0x7fae9c0ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c0ca910 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fae9400c510 tx=0x7fae94007f30 comp rx=0 tx=0).stop 2023-10-10T19:10:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 shutdown_connections 2023-10-10T19:10:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fae80066f60 0x7fae80069410 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9c0b2390 0x7fae9c0ca910 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:52.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae9c068bd0 0x7fae9c155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:52.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 --2- 172.21.15.175:0/707593159 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae9c0b1fc0 0x7fae9c155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:52.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.095+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 >> 172.21.15.175:0/707593159 conn(0x7fae9c063230 msgr2=0x7fae9c14fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:52.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.096+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 shutdown_connections 2023-10-10T19:10:52.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:52.096+0000 7faea3395700 1 -- 172.21.15.175:0/707593159 wait complete. 2023-10-10T19:10:52.099 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 203 2023-10-10T19:10:52.108 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:10:52.322 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:10:52.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:10:54.109 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:54.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.227+0000 7fa26b786700 1 Processor -- start 2023-10-10T19:10:54.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.227+0000 7fa26b786700 1 -- start start 2023-10-10T19:10:54.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa26b786700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa26414ffc0 0x7fa264159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:54.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa26b786700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa264156800 con 0x7fa26414fbf0 2023-10-10T19:10:54.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa26b786700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa264063e30 con 0x7fa2640681e0 2023-10-10T19:10:54.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa26b786700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2640c59e0 con 0x7fa26414ffc0 2023-10-10T19:10:54.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa268d21700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa26414ffc0 0x7fa264159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:54.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa269522700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa26414fbf0 0x7fa264063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47474/0 (socket says 172.21.15.175:47474) 2023-10-10T19:10:54.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa269522700 1 -- 172.21.15.175:0/2500922884 learned_addr learned my addr 172.21.15.175:0/2500922884 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:10:54.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa268d21700 1 -- 172.21.15.175:0/2500922884 >> v1:172.21.15.191:6789/0 conn(0x7fa2640681e0 legacy=0x7fa2640c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:54.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa268d21700 1 -- 172.21.15.175:0/2500922884 >> v1:172.21.15.175:6789/0 conn(0x7fa26414fbf0 legacy=0x7fa264063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:54.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa268d21700 1 -- 172.21.15.175:0/2500922884 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa264156c90 con 0x7fa26414ffc0 2023-10-10T19:10:54.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa25bfff700 1 -- 172.21.15.175:0/2500922884 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638551207 0 0) 0x7fa264156800 con 0x7fa26414fbf0 2023-10-10T19:10:54.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.228+0000 7fa268d21700 1 --2- 172.21.15.175:0/2500922884 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa26414ffc0 0x7fa264159d70 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fa2540098e0 tx=0x7fa2540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5472802600adda34 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:54.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa25bfff700 1 -- 172.21.15.175:0/2500922884 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254009e80 con 0x7fa26414ffc0 2023-10-10T19:10:54.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa25bfff700 1 -- 172.21.15.175:0/2500922884 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25400b860 con 0x7fa26414ffc0 2023-10-10T19:10:54.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa25bfff700 1 -- 172.21.15.175:0/2500922884 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25400bbe0 con 0x7fa26414ffc0 2023-10-10T19:10:54.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa26b786700 1 -- 172.21.15.175:0/2500922884 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa26414ffc0 msgr2=0x7fa264159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:54.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa26b786700 1 --2- 172.21.15.175:0/2500922884 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa26414ffc0 0x7fa264159d70 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fa2540098e0 tx=0x7fa2540091a0 comp rx=0 tx=0).stop 2023-10-10T19:10:54.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa26b786700 1 -- 172.21.15.175:0/2500922884 shutdown_connections 2023-10-10T19:10:54.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa26b786700 1 --2- 172.21.15.175:0/2500922884 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa26414ffc0 0x7fa264159d70 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa26b786700 1 -- 172.21.15.175:0/2500922884 >> 172.21.15.175:0/2500922884 conn(0x7fa2640cc760 msgr2=0x7fa2640ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa26b786700 1 -- 172.21.15.175:0/2500922884 shutdown_connections 2023-10-10T19:10:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.229+0000 7fa26b786700 1 -- 172.21.15.175:0/2500922884 wait complete. 2023-10-10T19:10:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.230+0000 7fa26b786700 1 Processor -- start 2023-10-10T19:10:54.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.230+0000 7fa26b786700 1 -- start start 2023-10-10T19:10:54.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa26b786700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2640681e0 0x7fa264167680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:54.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa26b786700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa26414fbf0 0x7fa264166430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:54.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa26b786700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa26414ffc0 0x7fa264166970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:54.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa26b786700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2640c6da0 con 0x7fa2640681e0 2023-10-10T19:10:54.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa26b786700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa264152c80 con 0x7fa26414fbf0 2023-10-10T19:10:54.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa26b786700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa264166eb0 con 0x7fa26414ffc0 2023-10-10T19:10:54.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269522700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2640681e0 0x7fa264167680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:54.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269522700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2640681e0 0x7fa264167680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48898/0 (socket says 172.21.15.175:48898) 2023-10-10T19:10:54.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269522700 1 -- 172.21.15.175:0/3354117279 learned_addr learned my addr 172.21.15.175:0/3354117279 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:54.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa268d21700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa26414fbf0 0x7fa264166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:54.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269d23700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa26414ffc0 0x7fa264166970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:54.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269522700 1 -- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa26414ffc0 msgr2=0x7fa264166970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:54.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269522700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa26414ffc0 0x7fa264166970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:54.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269522700 1 -- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa26414fbf0 msgr2=0x7fa264166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:54.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269522700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa26414fbf0 0x7fa264166430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:54.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269522700 1 -- 172.21.15.175:0/3354117279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa254009420 con 0x7fa2640681e0 2023-10-10T19:10:54.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.231+0000 7fa269d23700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa26414ffc0 0x7fa264166970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:10:54.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.232+0000 7fa269522700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2640681e0 0x7fa264167680 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fa26000c840 tx=0x7fa260007c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:54.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.232+0000 7fa2597fa700 1 -- 172.21.15.175:0/3354117279 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600072e0 con 0x7fa2640681e0 2023-10-10T19:10:54.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.232+0000 7fa2597fa700 1 -- 172.21.15.175:0/3354117279 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa260007480 con 0x7fa2640681e0 2023-10-10T19:10:54.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.232+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2640a8ca0 con 0x7fa2640681e0 2023-10-10T19:10:54.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.232+0000 7fa2597fa700 1 -- 172.21.15.175:0/3354117279 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600057f0 con 0x7fa2640681e0 2023-10-10T19:10:54.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.232+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2640c6ff0 con 0x7fa2640681e0 2023-10-10T19:10:54.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.233+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa264068e80 con 0x7fa2640681e0 2023-10-10T19:10:54.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.233+0000 7fa2597fa700 1 -- 172.21.15.175:0/3354117279 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa260005c00 con 0x7fa2640681e0 2023-10-10T19:10:54.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.233+0000 7fa2597fa700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa248066f60 0x7fa248069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:54.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.234+0000 7fa2597fa700 1 -- 172.21.15.175:0/3354117279 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fa260083e80 con 0x7fa2640681e0 2023-10-10T19:10:54.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.234+0000 7fa268d21700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa248066f60 0x7fa248069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:54.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.234+0000 7fa268d21700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa248066f60 0x7fa248069410 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa2540093f0 tx=0x7fa254017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:54.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.238+0000 7fa2597fa700 1 -- 172.21.15.175:0/3354117279 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa26004b5a0 con 0x7fa2640681e0 2023-10-10T19:10:54.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.466+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa264067570 con 0x7fa2640681e0 2023-10-10T19:10:54.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.467+0000 7fa2597fa700 1 -- 172.21.15.175:0/3354117279 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+6211 (secure 0 0 0) 0x7fa260017020 con 0x7fa2640681e0 2023-10-10T19:10:54.468 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:54.468 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:44.692939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:54.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa248066f60 msgr2=0x7fa248069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:54.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa248066f60 0x7fa248069410 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa2540093f0 tx=0x7fa254017000 comp rx=0 tx=0).stop 2023-10-10T19:10:54.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2640681e0 msgr2=0x7fa264167680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:54.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2640681e0 0x7fa264167680 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fa26000c840 tx=0x7fa260007c00 comp rx=0 tx=0).stop 2023-10-10T19:10:54.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 shutdown_connections 2023-10-10T19:10:54.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa248066f60 0x7fa248069410 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:54.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa26414ffc0 0x7fa264166970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:54.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2640681e0 0x7fa264167680 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:54.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 --2- 172.21.15.175:0/3354117279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa26414fbf0 0x7fa264166430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:54.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 >> 172.21.15.175:0/3354117279 conn(0x7fa2640cc760 msgr2=0x7fa2640ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:54.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 shutdown_connections 2023-10-10T19:10:54.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:54.469+0000 7fa26b786700 1 -- 172.21.15.175:0/3354117279 wait complete. 2023-10-10T19:10:54.475 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 203 2023-10-10T19:10:54.482 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:10:56.483 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:56.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.609+0000 7f7149334700 1 Processor -- start 2023-10-10T19:10:56.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.609+0000 7f7149334700 1 -- start start 2023-10-10T19:10:56.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.609+0000 7f7149334700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f71440b1fc0 0x7f7144155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:56.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.609+0000 7f7149334700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71441574b0 con 0x7f71440aff60 2023-10-10T19:10:56.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.609+0000 7f7149334700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71440afad0 con 0x7f71440b2390 2023-10-10T19:10:56.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.609+0000 7f7149334700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7144150770 con 0x7f71440b1fc0 2023-10-10T19:10:56.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.609+0000 7f7142ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f71440b1fc0 0x7f7144155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:56.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f71437fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f71440aff60 0x7f7144159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44502/0 (socket says 172.21.15.175:44502) 2023-10-10T19:10:56.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f71437fe700 1 -- 172.21.15.175:0/2172861311 learned_addr learned my addr 172.21.15.175:0/2172861311 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:10:56.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f7142ffd700 1 -- 172.21.15.175:0/2172861311 >> v1:172.21.15.191:6789/0 conn(0x7f71440b2390 legacy=0x7f7144068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:56.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f7142ffd700 1 -- 172.21.15.175:0/2172861311 >> v1:172.21.15.175:6789/0 conn(0x7f71440aff60 legacy=0x7f7144159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:56.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f7142ffd700 1 -- 172.21.15.175:0/2172861311 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71440af4c0 con 0x7f71440b1fc0 2023-10-10T19:10:56.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f7141ffb700 1 -- 172.21.15.175:0/2172861311 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223319267 0 0) 0x7f71441574b0 con 0x7f71440aff60 2023-10-10T19:10:56.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f7142ffd700 1 --2- 172.21.15.175:0/2172861311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144155fb0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7134009e60 tx=0x7f7134009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab14ecffa955fd08 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:56.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f7141ffb700 1 -- 172.21.15.175:0/2172861311 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713400e070 con 0x7f71440b1fc0 2023-10-10T19:10:56.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f7141ffb700 1 -- 172.21.15.175:0/2172861311 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713400bc80 con 0x7f71440b1fc0 2023-10-10T19:10:56.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.610+0000 7f7141ffb700 1 -- 172.21.15.175:0/2172861311 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134012430 con 0x7f71440b1fc0 2023-10-10T19:10:56.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.611+0000 7f7149334700 1 -- 172.21.15.175:0/2172861311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 msgr2=0x7f7144155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:56.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.611+0000 7f7149334700 1 --2- 172.21.15.175:0/2172861311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144155fb0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7134009e60 tx=0x7f7134009780 comp rx=0 tx=0).stop 2023-10-10T19:10:56.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.611+0000 7f7149334700 1 -- 172.21.15.175:0/2172861311 shutdown_connections 2023-10-10T19:10:56.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.611+0000 7f7149334700 1 --2- 172.21.15.175:0/2172861311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144155fb0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:56.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.611+0000 7f7149334700 1 -- 172.21.15.175:0/2172861311 >> 172.21.15.175:0/2172861311 conn(0x7f7144063230 msgr2=0x7f7144063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:56.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.611+0000 7f7149334700 1 -- 172.21.15.175:0/2172861311 shutdown_connections 2023-10-10T19:10:56.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.611+0000 7f7149334700 1 -- 172.21.15.175:0/2172861311 wait complete. 2023-10-10T19:10:56.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.612+0000 7f7149334700 1 Processor -- start 2023-10-10T19:10:56.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.612+0000 7f7149334700 1 -- start start 2023-10-10T19:10:56.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7149334700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71440aff60 0x7f71440c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:56.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7149334700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144169a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:56.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7149334700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71440b2390 0x7f7144169f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:56.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7149334700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71441514c0 con 0x7f71440b1fc0 2023-10-10T19:10:56.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7149334700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7144159840 con 0x7f71440aff60 2023-10-10T19:10:56.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7149334700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71441516f0 con 0x7f71440b2390 2023-10-10T19:10:56.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144169a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:56.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144169a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33630/0 (socket says 172.21.15.175:33630) 2023-10-10T19:10:56.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 -- 172.21.15.175:0/1442290296 learned_addr learned my addr 172.21.15.175:0/1442290296 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:56.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71437fe700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71440b2390 0x7f7144169f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:56.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 -- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71440b2390 msgr2=0x7f7144169f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:56.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7142ffd700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71440aff60 0x7f71440c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:56.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71440b2390 0x7f7144169f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:56.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 -- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71440aff60 msgr2=0x7f71440c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:56.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71440aff60 0x7f71440c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:56.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 -- 172.21.15.175:0/1442290296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7134009a20 con 0x7f71440b1fc0 2023-10-10T19:10:56.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f71427fc700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144169a50 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f713800b5e0 tx=0x7f7138009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:56.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f712affd700 1 -- 172.21.15.175:0/1442290296 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713800f510 con 0x7f71440b1fc0 2023-10-10T19:10:56.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f712affd700 1 -- 172.21.15.175:0/1442290296 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713800f670 con 0x7f71440b1fc0 2023-10-10T19:10:56.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f712affd700 1 -- 172.21.15.175:0/1442290296 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713800fa30 con 0x7f71440b1fc0 2023-10-10T19:10:56.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7144158f10 con 0x7f71440b1fc0 2023-10-10T19:10:56.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.613+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71441594d0 con 0x7f71440b1fc0 2023-10-10T19:10:56.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.614+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71441599f0 con 0x7f71440b1fc0 2023-10-10T19:10:56.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.615+0000 7f712affd700 1 -- 172.21.15.175:0/1442290296 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f713801a400 con 0x7f71440b1fc0 2023-10-10T19:10:56.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.615+0000 7f712affd700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7130066f30 0x7f71300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:56.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.615+0000 7f712affd700 1 -- 172.21.15.175:0/1442290296 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f7138084680 con 0x7f71440b1fc0 2023-10-10T19:10:56.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.615+0000 7f7142ffd700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7130066f30 0x7f71300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:56.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.616+0000 7f7142ffd700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7130066f30 0x7f71300693e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f71340099f0 tx=0x7f7134018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:56.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.620+0000 7f712affd700 1 -- 172.21.15.175:0/1442290296 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71441599f0 con 0x7f71440b1fc0 2023-10-10T19:10:56.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.845+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71441599b0 con 0x7f71440b1fc0 2023-10-10T19:10:56.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.846+0000 7f712affd700 1 -- 172.21.15.175:0/1442290296 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+6211 (secure 0 0 0) 0x7f7138051410 con 0x7f71440b1fc0 2023-10-10T19:10:56.846 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:56.847 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:44.692939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:56.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7130066f30 msgr2=0x7f71300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:56.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7130066f30 0x7f71300693e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f71340099f0 tx=0x7f7134018000 comp rx=0 tx=0).stop 2023-10-10T19:10:56.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 msgr2=0x7f7144169a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:56.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144169a50 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f713800b5e0 tx=0x7f7138009120 comp rx=0 tx=0).stop 2023-10-10T19:10:56.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 shutdown_connections 2023-10-10T19:10:56.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7130066f30 0x7f71300693e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:56.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71440b2390 0x7f7144169f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:56.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71440aff60 0x7f71440c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:56.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 --2- 172.21.15.175:0/1442290296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f71440b1fc0 0x7f7144169a50 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:56.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 >> 172.21.15.175:0/1442290296 conn(0x7f7144063230 msgr2=0x7f714414fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:56.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 shutdown_connections 2023-10-10T19:10:56.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:56.848+0000 7f7149334700 1 -- 172.21.15.175:0/1442290296 wait complete. 2023-10-10T19:10:56.851 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 203 2023-10-10T19:10:56.860 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:10:58.861 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:10:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.989+0000 7f7b9e227700 1 Processor -- start 2023-10-10T19:10:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.989+0000 7f7b9e227700 1 -- start start 2023-10-10T19:10:58.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b9e227700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7b980b1fc0 0x7f7b98155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:58.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b9e227700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b980b2390 0x7f7b98068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:58.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b9e227700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b981574b0 con 0x7f7b98068bd0 2023-10-10T19:10:58.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b9e227700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b98150770 con 0x7f7b980b1fc0 2023-10-10T19:10:58.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b9e227700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980afad0 con 0x7f7b980b2390 2023-10-10T19:10:58.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b97fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7b98068bd0 0x7f7b980aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44518/0 (socket says 172.21.15.175:44518) 2023-10-10T19:10:58.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b97fff700 1 -- 172.21.15.175:0/460228911 learned_addr learned my addr 172.21.15.175:0/460228911 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:10:58.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b96ffd700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b980b2390 0x7f7b98068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:58.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b977fe700 1 --2- 172.21.15.175:0/460228911 >> v2:172.21.15.191:3300/0 conn(0x7f7b980b1fc0 0x7f7b98155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:58.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439705781 0 0) 0x7f7b981574b0 con 0x7f7b98068bd0 2023-10-10T19:10:58.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b80003570 con 0x7f7b98068bd0 2023-10-10T19:10:58.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 34243122 0 0) 0x7f7b80003570 con 0x7f7b98068bd0 2023-10-10T19:10:58.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b980b2390 msgr2=0x7f7b98068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:58.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b980b2390 0x7f7b98068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:58.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 >> v2:172.21.15.191:3300/0 conn(0x7f7b980b1fc0 msgr2=0x7f7b98155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:58.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 --2- 172.21.15.175:0/460228911 >> v2:172.21.15.191:3300/0 conn(0x7f7b980b1fc0 0x7f7b98155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:58.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980af4c0 con 0x7f7b98068bd0 2023-10-10T19:10:58.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.990+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f7b90001470 con 0x7f7b98068bd0 2023-10-10T19:10:58.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 >> v1:172.21.15.175:6789/0 conn(0x7f7b98068bd0 legacy=0x7f7b980aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:58.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b80003a60 0x7f7b80003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b80004810 0x7f7b80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b980b2390 0x7f7b800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98150770 con 0x7f7b80004810 2023-10-10T19:10:59.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980afad0 con 0x7f7b80003a60 2023-10-10T19:10:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8000ba60 con 0x7f7b980b2390 2023-10-10T19:10:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b96ffd700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b80004810 0x7f7b80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b977fe700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b80003a60 0x7f7b80003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b97fff700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b980b2390 0x7f7b800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b96ffd700 1 -- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b980b2390 msgr2=0x7f7b800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b96ffd700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b980b2390 0x7f7b800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b96ffd700 1 -- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b80003a60 msgr2=0x7f7b80003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:10:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b96ffd700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b80003a60 0x7f7b80003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b96ffd700 1 -- 172.21.15.175:0/460228911 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80003570 con 0x7f7b80004810 2023-10-10T19:10:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b96ffd700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b80004810 0x7f7b80004c80 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c009c30 tx=0x7f7b8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874b051a7f4e5f7b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c00e710 con 0x7f7b80004810 2023-10-10T19:10:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8c00e870 con 0x7f7b80004810 2023-10-10T19:10:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.991+0000 7f7b967fc700 1 -- 172.21.15.175:0/460228911 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c00ec30 con 0x7f7b80004810 2023-10-10T19:10:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.992+0000 7f7b9e227700 1 -- 172.21.15.175:0/460228911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b80004810 msgr2=0x7f7b80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.992+0000 7f7b9e227700 1 --2- 172.21.15.175:0/460228911 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b80004810 0x7f7b80004c80 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c009c30 tx=0x7f7b8c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.992+0000 7f7b97fff700 1 -- 172.21.15.175:0/460228911 reap_dead start 2023-10-10T19:10:59.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.992+0000 7f7b9e227700 1 -- 172.21.15.175:0/460228911 shutdown_connections 2023-10-10T19:10:59.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.992+0000 7f7b9e227700 1 -- 172.21.15.175:0/460228911 >> 172.21.15.175:0/460228911 conn(0x7f7b98063230 msgr2=0x7f7b98063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:59.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.992+0000 7f7b9e227700 1 -- 172.21.15.175:0/460228911 shutdown_connections 2023-10-10T19:10:59.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.992+0000 7f7b9e227700 1 -- 172.21.15.175:0/460228911 wait complete. 2023-10-10T19:10:59.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.993+0000 7f7b9e227700 1 Processor -- start 2023-10-10T19:10:59.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.993+0000 7f7b9e227700 1 -- start start 2023-10-10T19:10:59.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b9e227700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b80004810 0x7f7b98166d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:59.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b9e227700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b98151090 0x7f7b9814e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:59.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b9e227700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b98156450 0x7f7b9814eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:59.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b9e227700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980c6c90 con 0x7f7b98151090 2023-10-10T19:10:59.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b9e227700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98169b30 con 0x7f7b98156450 2023-10-10T19:10:59.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b9e227700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98159110 con 0x7f7b80004810 2023-10-10T19:10:59.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b98151090 0x7f7b9814e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:59.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b98151090 0x7f7b9814e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33648/0 (socket says 172.21.15.175:33648) 2023-10-10T19:10:59.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 -- 172.21.15.175:0/1673268523 learned_addr learned my addr 172.21.15.175:0/1673268523 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:10:59.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b977fe700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b80004810 0x7f7b98166d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:59.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b97fff700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b98156450 0x7f7b9814eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:59.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 -- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b80004810 msgr2=0x7f7b98166d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:59.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b80004810 0x7f7b98166d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 -- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b98156450 msgr2=0x7f7b9814eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:59.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b98156450 0x7f7b9814eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 -- 172.21.15.175:0/1673268523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b981593f0 con 0x7f7b98151090 2023-10-10T19:10:59.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.994+0000 7f7b96ffd700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b98151090 0x7f7b9814e5b0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00e4b0 tx=0x7f7b8c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:59.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.995+0000 7f7b7f7fe700 1 -- 172.21.15.175:0/1673268523 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c023070 con 0x7f7b98151090 2023-10-10T19:10:59.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.995+0000 7f7b7f7fe700 1 -- 172.21.15.175:0/1673268523 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8c00e9e0 con 0x7f7b98151090 2023-10-10T19:10:59.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.995+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b981596d0 con 0x7f7b98151090 2023-10-10T19:10:59.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.995+0000 7f7b7f7fe700 1 -- 172.21.15.175:0/1673268523 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c01e030 con 0x7f7b98151090 2023-10-10T19:10:59.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.995+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b98159c90 con 0x7f7b98151090 2023-10-10T19:10:59.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.995+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b98159e80 con 0x7f7b98151090 2023-10-10T19:10:59.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.996+0000 7f7b7f7fe700 1 -- 172.21.15.175:0/1673268523 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7b8c013470 con 0x7f7b98151090 2023-10-10T19:10:59.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.996+0000 7f7b7f7fe700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b84066f30 0x7f7b840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:10:59.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.996+0000 7f7b7f7fe700 1 -- 172.21.15.175:0/1673268523 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f7b8c086120 con 0x7f7b98151090 2023-10-10T19:10:59.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.996+0000 7f7b977fe700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b84066f30 0x7f7b840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:10:59.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:58.997+0000 7f7b977fe700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b84066f30 0x7f7b840693e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7b880094d0 tx=0x7f7b8800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:10:59.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.001+0000 7f7b7f7fe700 1 -- 172.21.15.175:0/1673268523 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b8c052ec0 con 0x7f7b98151090 2023-10-10T19:10:59.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.225+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b980c6c90 con 0x7f7b98151090 2023-10-10T19:10:59.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.225+0000 7f7b7f7fe700 1 -- 172.21.15.175:0/1673268523 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+6211 (secure 0 0 0) 0x7f7b8c0564e0 con 0x7f7b98151090 2023-10-10T19:10:59.226 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:10:59.227 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:44.692939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:10:59.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b84066f30 msgr2=0x7f7b840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:59.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b84066f30 0x7f7b840693e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7b880094d0 tx=0x7f7b8800b040 comp rx=0 tx=0).stop 2023-10-10T19:10:59.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b98151090 msgr2=0x7f7b9814e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:10:59.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b98151090 0x7f7b9814e5b0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00e4b0 tx=0x7f7b8c01a000 comp rx=0 tx=0).stop 2023-10-10T19:10:59.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 shutdown_connections 2023-10-10T19:10:59.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7b84066f30 0x7f7b840693e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b98156450 0x7f7b9814eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b80004810 0x7f7b98166d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 --2- 172.21.15.175:0/1673268523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b98151090 0x7f7b9814e5b0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:10:59.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 >> 172.21.15.175:0/1673268523 conn(0x7f7b98063230 msgr2=0x7f7b981508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:10:59.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 shutdown_connections 2023-10-10T19:10:59.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:10:59.228+0000 7f7b9e227700 1 -- 172.21.15.175:0/1673268523 wait complete. 2023-10-10T19:10:59.233 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 203 2023-10-10T19:10:59.241 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:11:01.241 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:01.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.369+0000 7f3d1e460700 1 Processor -- start 2023-10-10T19:11:01.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.369+0000 7f3d1e460700 1 -- start start 2023-10-10T19:11:01.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1e460700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3d180b2390 0x7f3d18155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:01.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1e460700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180aff60 0x7f3d180b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:01.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1e460700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d181574b0 con 0x7f3d180b1fc0 2023-10-10T19:11:01.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1e460700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d18150770 con 0x7f3d180b2390 2023-10-10T19:11:01.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1e460700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180afad0 con 0x7f3d180aff60 2023-10-10T19:11:01.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d177fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3d180b2390 0x7f3d18155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:01.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1c9fd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180aff60 0x7f3d180b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:01.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d177fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3d180b2390 0x7f3d18155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:57352/0 (socket says 172.21.15.175:57352) 2023-10-10T19:11:01.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d177fe700 1 -- 172.21.15.175:0/529681168 learned_addr learned my addr 172.21.15.175:0/529681168 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:01.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d17fff700 1 --1- 172.21.15.175:0/529681168 >> v1:172.21.15.191:6789/0 conn(0x7f3d180b1fc0 0x7f3d18068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:47066/0 (socket says 172.21.15.175:47066) 2023-10-10T19:11:01.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1c9fd700 1 -- 172.21.15.175:0/529681168 >> v2:172.21.15.191:3300/0 conn(0x7f3d180b2390 msgr2=0x7f3d18155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:01.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1c9fd700 1 --2- 172.21.15.175:0/529681168 >> v2:172.21.15.191:3300/0 conn(0x7f3d180b2390 0x7f3d18155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1c9fd700 1 -- 172.21.15.175:0/529681168 >> v1:172.21.15.191:6789/0 conn(0x7f3d180b1fc0 legacy=0x7f3d18068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:01.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d16ffd700 1 -- 172.21.15.175:0/529681168 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096246723 0 0) 0x7f3d181574b0 con 0x7f3d180b1fc0 2023-10-10T19:11:01.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d1c9fd700 1 -- 172.21.15.175:0/529681168 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d180af4c0 con 0x7f3d180aff60 2023-10-10T19:11:01.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.370+0000 7f3d177fe700 1 --2- 172.21.15.175:0/529681168 >> v2:172.21.15.191:3300/0 conn(0x7f3d180b2390 0x7f3d18155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:11:01.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1c9fd700 1 --2- 172.21.15.175:0/529681168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180aff60 0x7f3d180b0400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009e60 tx=0x7f3d10009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f27203146c966e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:01.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d16ffd700 1 -- 172.21.15.175:0/529681168 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000e070 con 0x7f3d180aff60 2023-10-10T19:11:01.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d16ffd700 1 -- 172.21.15.175:0/529681168 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1000bbf0 con 0x7f3d180aff60 2023-10-10T19:11:01.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d16ffd700 1 -- 172.21.15.175:0/529681168 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10012430 con 0x7f3d180aff60 2023-10-10T19:11:01.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1e460700 1 -- 172.21.15.175:0/529681168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180aff60 msgr2=0x7f3d180b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:01.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1e460700 1 --2- 172.21.15.175:0/529681168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180aff60 0x7f3d180b0400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009e60 tx=0x7f3d10009780 comp rx=0 tx=0).stop 2023-10-10T19:11:01.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1e460700 1 -- 172.21.15.175:0/529681168 shutdown_connections 2023-10-10T19:11:01.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1e460700 1 --2- 172.21.15.175:0/529681168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180aff60 0x7f3d180b0400 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1e460700 1 --2- 172.21.15.175:0/529681168 >> v2:172.21.15.191:3300/0 conn(0x7f3d180b2390 0x7f3d18155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1e460700 1 -- 172.21.15.175:0/529681168 >> 172.21.15.175:0/529681168 conn(0x7f3d18063230 msgr2=0x7f3d18063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:01.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1e460700 1 -- 172.21.15.175:0/529681168 shutdown_connections 2023-10-10T19:11:01.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.371+0000 7f3d1e460700 1 -- 172.21.15.175:0/529681168 wait complete. 2023-10-10T19:11:01.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.372+0000 7f3d1e460700 1 Processor -- start 2023-10-10T19:11:01.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.373+0000 7f3d1e460700 1 -- start start 2023-10-10T19:11:01.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.373+0000 7f3d1e460700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d180aff60 0x7f3d1815a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:01.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.373+0000 7f3d1e460700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d180b1fc0 0x7f3d1815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:01.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.373+0000 7f3d1e460700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180b2390 0x7f3d181573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:01.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.373+0000 7f3d1e460700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18158b50 con 0x7f3d180aff60 2023-10-10T19:11:01.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.373+0000 7f3d1e460700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1814a130 con 0x7f3d180b1fc0 2023-10-10T19:11:01.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.373+0000 7f3d1e460700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18158da0 con 0x7f3d180b2390 2023-10-10T19:11:01.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d180aff60 0x7f3d1815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:01.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d180aff60 0x7f3d1815a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33652/0 (socket says 172.21.15.175:33652) 2023-10-10T19:11:01.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 -- 172.21.15.175:0/3880145663 learned_addr learned my addr 172.21.15.175:0/3880145663 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:01.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d177fe700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d180b1fc0 0x7f3d1815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:01.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d1c9fd700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180b2390 0x7f3d181573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:01.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 -- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180b2390 msgr2=0x7f3d181573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:01.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180b2390 0x7f3d181573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 -- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d180b1fc0 msgr2=0x7f3d1815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:01.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d180b1fc0 0x7f3d1815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 -- 172.21.15.175:0/3880145663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0094f0 con 0x7f3d180aff60 2023-10-10T19:11:01.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d17fff700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d180aff60 0x7f3d1815a820 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3d080011e0 tx=0x7f3d08007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:01.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3cfffff700 1 -- 172.21.15.175:0/3880145663 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08007300 con 0x7f3d180aff60 2023-10-10T19:11:01.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3cfffff700 1 -- 172.21.15.175:0/3880145663 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08007460 con 0x7f3d180aff60 2023-10-10T19:11:01.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3cfffff700 1 -- 172.21.15.175:0/3880145663 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08004830 con 0x7f3d180aff60 2023-10-10T19:11:01.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d10009a20 con 0x7f3d180aff60 2023-10-10T19:11:01.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.374+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d18159360 con 0x7f3d180aff60 2023-10-10T19:11:01.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.375+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1815a670 con 0x7f3d180aff60 2023-10-10T19:11:01.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.376+0000 7f3cfffff700 1 -- 172.21.15.175:0/3880145663 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3d08004be0 con 0x7f3d180aff60 2023-10-10T19:11:01.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.376+0000 7f3cfffff700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d04066f30 0x7f3d040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:01.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.376+0000 7f3cfffff700 1 -- 172.21.15.175:0/3880145663 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f3d08083fa0 con 0x7f3d180aff60 2023-10-10T19:11:01.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.376+0000 7f3d177fe700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d04066f30 0x7f3d040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:01.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.377+0000 7f3d177fe700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d04066f30 0x7f3d040693e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c006d40 tx=0x7f3d0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:01.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.380+0000 7f3cfffff700 1 -- 172.21.15.175:0/3880145663 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d08050d40 con 0x7f3d180aff60 2023-10-10T19:11:01.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.605+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d18067570 con 0x7f3d180aff60 2023-10-10T19:11:01.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.606+0000 7f3cfffff700 1 -- 172.21.15.175:0/3880145663 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+6211 (secure 0 0 0) 0x7f3d08018070 con 0x7f3d180aff60 2023-10-10T19:11:01.606 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:01.607 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:10:44.692939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5634,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"e","rank":0,"incarnation":94,"state":"up:active","state_seq":25,"addr":"172.21.15.175:6837/1881948738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":1881948738},{"type":"v1","addr":"172.21.15.175:6837","nonce":1881948738}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:01.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d04066f30 msgr2=0x7f3d040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:01.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d04066f30 0x7f3d040693e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c006d40 tx=0x7f3d0c008040 comp rx=0 tx=0).stop 2023-10-10T19:11:01.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d180aff60 msgr2=0x7f3d1815a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:01.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d180aff60 0x7f3d1815a820 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3d080011e0 tx=0x7f3d08007e10 comp rx=0 tx=0).stop 2023-10-10T19:11:01.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 shutdown_connections 2023-10-10T19:11:01.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3d04066f30 0x7f3d040693e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d180b2390 0x7f3d181573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d180aff60 0x7f3d1815a820 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 --2- 172.21.15.175:0/3880145663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d180b1fc0 0x7f3d1815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:01.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 >> 172.21.15.175:0/3880145663 conn(0x7f3d18063230 msgr2=0x7f3d180c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:01.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 shutdown_connections 2023-10-10T19:11:01.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:01.608+0000 7f3d1e460700 1 -- 172.21.15.175:0/3880145663 wait complete. 2023-10-10T19:11:01.613 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 203 2023-10-10T19:11:01.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:11:03.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:03.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.753+0000 7f186940b700 1 Processor -- start 2023-10-10T19:11:03.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.753+0000 7f186940b700 1 -- start start 2023-10-10T19:11:03.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.753+0000 7f186940b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c7190 0x7f186415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:03.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.753+0000 7f186940b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18640c4580 0x7f1864063c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:03.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.753+0000 7f186940b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1864157030 con 0x7f18640c6330 2023-10-10T19:11:03.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.753+0000 7f186940b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18640c5000 con 0x7f18640c4580 2023-10-10T19:11:03.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.753+0000 7f186940b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1864156420 con 0x7f18640c7190 2023-10-10T19:11:03.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.753+0000 7f18637fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18640c4580 0x7f1864063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:03.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18637fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18640c4580 0x7f1864063c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33658/0 (socket says 172.21.15.175:33658) 2023-10-10T19:11:03.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18637fe700 1 -- 172.21.15.175:0/1889374109 learned_addr learned my addr 172.21.15.175:0/1889374109 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:03.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f185a7fc700 1 --2- 172.21.15.175:0/1889374109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c7190 0x7f186415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:03.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18637fe700 1 -- 172.21.15.175:0/1889374109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c7190 msgr2=0x7f186415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:03.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18637fe700 1 --2- 172.21.15.175:0/1889374109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c7190 0x7f186415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18637fe700 1 -- 172.21.15.175:0/1889374109 >> v1:172.21.15.191:6789/0 conn(0x7f18640c6330 legacy=0x7f18640c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:03.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18637fe700 1 -- 172.21.15.175:0/1889374109 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1864156700 con 0x7f18640c4580 2023-10-10T19:11:03.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18637fe700 1 --2- 172.21.15.175:0/1889374109 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f1864063c20 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f185c009f20 tx=0x7f185c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e226cc233ae839 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:03.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18627fc700 1 -- 172.21.15.175:0/1889374109 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c00e070 con 0x7f18640c4580 2023-10-10T19:11:03.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18627fc700 1 -- 172.21.15.175:0/1889374109 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c00bb10 con 0x7f18640c4580 2023-10-10T19:11:03.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.754+0000 7f18627fc700 1 -- 172.21.15.175:0/1889374109 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0092e0 con 0x7f18640c4580 2023-10-10T19:11:03.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.755+0000 7f186940b700 1 -- 172.21.15.175:0/1889374109 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 msgr2=0x7f1864063c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:03.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.755+0000 7f186940b700 1 --2- 172.21.15.175:0/1889374109 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f1864063c20 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f185c009f20 tx=0x7f185c009840 comp rx=0 tx=0).stop 2023-10-10T19:11:03.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.755+0000 7f186940b700 1 -- 172.21.15.175:0/1889374109 shutdown_connections 2023-10-10T19:11:03.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.755+0000 7f186940b700 1 --2- 172.21.15.175:0/1889374109 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f1864063c20 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.755+0000 7f186940b700 1 --2- 172.21.15.175:0/1889374109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c7190 0x7f186415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.755+0000 7f186940b700 1 -- 172.21.15.175:0/1889374109 >> 172.21.15.175:0/1889374109 conn(0x7f186406ef20 msgr2=0x7f1864063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:03.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.755+0000 7f186940b700 1 -- 172.21.15.175:0/1889374109 shutdown_connections 2023-10-10T19:11:03.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.755+0000 7f186940b700 1 -- 172.21.15.175:0/1889374109 wait complete. 2023-10-10T19:11:03.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.756+0000 7f186940b700 1 Processor -- start 2023-10-10T19:11:03.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.756+0000 7f186940b700 1 -- start start 2023-10-10T19:11:03.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f186940b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f18640cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:03.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f186940b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c6330 0x7f18640c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:03.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f186940b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18640c7190 0x7f1864168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:03.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f186940b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1864150da0 con 0x7f18640c4580 2023-10-10T19:11:03.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f186940b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640c15d0 con 0x7f18640c7190 2023-10-10T19:11:03.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f186940b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18640c31a0 con 0x7f18640c6330 2023-10-10T19:11:03.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f18640cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:03.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f18640cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33660/0 (socket says 172.21.15.175:33660) 2023-10-10T19:11:03.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 -- 172.21.15.175:0/911551443 learned_addr learned my addr 172.21.15.175:0/911551443 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:03.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f18637fe700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18640c7190 0x7f1864168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:03.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 -- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c6330 msgr2=0x7f18640c1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:03.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c6330 0x7f18640c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 -- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18640c7190 msgr2=0x7f1864168fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:03.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18640c7190 0x7f1864168fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 -- 172.21.15.175:0/911551443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c009ae0 con 0x7f18640c4580 2023-10-10T19:11:03.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f1862ffd700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f18640cce80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f185400c590 tx=0x7f185400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:03.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f185b7fe700 1 -- 172.21.15.175:0/911551443 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854011590 con 0x7f18640c4580 2023-10-10T19:11:03.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f185b7fe700 1 -- 172.21.15.175:0/911551443 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18540116f0 con 0x7f18640c4580 2023-10-10T19:11:03.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18640c3480 con 0x7f18640c4580 2023-10-10T19:11:03.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.757+0000 7f185b7fe700 1 -- 172.21.15.175:0/911551443 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854011ab0 con 0x7f18640c4580 2023-10-10T19:11:03.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.758+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1864165e30 con 0x7f18640c4580 2023-10-10T19:11:03.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.758+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1864067570 con 0x7f18640c4580 2023-10-10T19:11:03.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.759+0000 7f185b7fe700 1 -- 172.21.15.175:0/911551443 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f185401b3b0 con 0x7f18640c4580 2023-10-10T19:11:03.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.759+0000 7f185b7fe700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1844066f30 0x7f18440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:03.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.759+0000 7f185b7fe700 1 -- 172.21.15.175:0/911551443 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f1854084090 con 0x7f18640c4580 2023-10-10T19:11:03.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.759+0000 7f185a7fc700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1844066f30 0x7f18440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:03.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.760+0000 7f185a7fc700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1844066f30 0x7f18440693e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f18640c43e0 tx=0x7f184c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:03.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.763+0000 7f185b7fe700 1 -- 172.21.15.175:0/911551443 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1854050d30 con 0x7f18640c4580 2023-10-10T19:11:03.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.990+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18641569e0 con 0x7f18640c4580 2023-10-10T19:11:03.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.990+0000 7f185b7fe700 1 -- 172.21.15.175:0/911551443 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f1854017020 con 0x7f18640c4580 2023-10-10T19:11:03.991 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:03.991 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:03.690976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:03.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1844066f30 msgr2=0x7f18440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:03.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1844066f30 0x7f18440693e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f18640c43e0 tx=0x7f184c008040 comp rx=0 tx=0).stop 2023-10-10T19:11:03.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 msgr2=0x7f18640cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:03.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f18640cce80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f185400c590 tx=0x7f185400c950 comp rx=0 tx=0).stop 2023-10-10T19:11:03.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 shutdown_connections 2023-10-10T19:11:03.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1844066f30 0x7f18440693e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18640c7190 0x7f1864168fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18640c4580 0x7f18640cce80 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 --2- 172.21.15.175:0/911551443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18640c6330 0x7f18640c1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:03.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 >> 172.21.15.175:0/911551443 conn(0x7f186406ef20 msgr2=0x7f1864157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:03.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 shutdown_connections 2023-10-10T19:11:03.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:03.992+0000 7f186940b700 1 -- 172.21.15.175:0/911551443 wait complete. 2023-10-10T19:11:03.996 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 204 2023-10-10T19:11:04.005 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-10T19:11:04.005 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:11:04.006 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:04.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.119+0000 7f7efb04c700 1 Processor -- start 2023-10-10T19:11:04.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.119+0000 7f7efb04c700 1 -- start start 2023-10-10T19:11:04.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.120+0000 7f7efb04c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7ef40c2cf0 0x7f7ef406e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:04.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.120+0000 7f7efb04c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef4153f70 con 0x7f7ef40c30c0 2023-10-10T19:11:04.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.120+0000 7f7efb04c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef4067bd0 con 0x7f7ef414e290 2023-10-10T19:11:04.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.120+0000 7f7efb04c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef4154210 con 0x7f7ef40c2cf0 2023-10-10T19:11:04.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef8de8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7ef40c2cf0 0x7f7ef406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:04.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef3fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7ef40c30c0 0x7f7ef414db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:44534/0 (socket says 172.21.15.175:44534) 2023-10-10T19:11:04.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef3fff700 1 -- 172.21.15.175:0/4220648930 learned_addr learned my addr 172.21.15.175:0/4220648930 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:04.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef8de8700 1 -- 172.21.15.175:0/4220648930 >> v1:172.21.15.191:6789/0 conn(0x7f7ef414e290 legacy=0x7f7ef415e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:04.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef8de8700 1 -- 172.21.15.175:0/4220648930 >> v1:172.21.15.175:6789/0 conn(0x7f7ef40c30c0 legacy=0x7f7ef414db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:04.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef8de8700 1 -- 172.21.15.175:0/4220648930 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef41544f0 con 0x7f7ef40c2cf0 2023-10-10T19:11:04.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef37fe700 1 -- 172.21.15.175:0/4220648930 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637580546 0 0) 0x7f7ef4153f70 con 0x7f7ef40c30c0 2023-10-10T19:11:04.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef8de8700 1 --2- 172.21.15.175:0/4220648930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef40c2cf0 0x7f7ef406e500 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009eb0 tx=0x7f7ee4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caed34640a78b8c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:04.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef37fe700 1 -- 172.21.15.175:0/4220648930 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee400e070 con 0x7f7ef40c2cf0 2023-10-10T19:11:04.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef37fe700 1 -- 172.21.15.175:0/4220648930 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee400bcc0 con 0x7f7ef40c2cf0 2023-10-10T19:11:04.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.121+0000 7f7ef37fe700 1 -- 172.21.15.175:0/4220648930 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4012680 con 0x7f7ef40c2cf0 2023-10-10T19:11:04.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.122+0000 7f7efb04c700 1 -- 172.21.15.175:0/4220648930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef40c2cf0 msgr2=0x7f7ef406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:04.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.122+0000 7f7efb04c700 1 --2- 172.21.15.175:0/4220648930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef40c2cf0 0x7f7ef406e500 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009eb0 tx=0x7f7ee4009780 comp rx=0 tx=0).stop 2023-10-10T19:11:04.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.122+0000 7f7efb04c700 1 -- 172.21.15.175:0/4220648930 shutdown_connections 2023-10-10T19:11:04.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.122+0000 7f7efb04c700 1 --2- 172.21.15.175:0/4220648930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef40c2cf0 0x7f7ef406e500 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:04.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.122+0000 7f7efb04c700 1 -- 172.21.15.175:0/4220648930 >> 172.21.15.175:0/4220648930 conn(0x7f7ef4068900 msgr2=0x7f7ef4068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:04.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.122+0000 7f7efb04c700 1 -- 172.21.15.175:0/4220648930 shutdown_connections 2023-10-10T19:11:04.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.122+0000 7f7efb04c700 1 -- 172.21.15.175:0/4220648930 wait complete. 2023-10-10T19:11:04.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.123+0000 7f7efb04c700 1 Processor -- start 2023-10-10T19:11:04.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.123+0000 7f7efb04c700 1 -- start start 2023-10-10T19:11:04.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7efb04c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ef40c2cf0 0x7f7ef4167050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:04.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7efb04c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ef40c30c0 0x7f7ef4144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:04.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7efb04c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef414e290 0x7f7ef41454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:04.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7efb04c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef40c7b10 con 0x7f7ef414e290 2023-10-10T19:11:04.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7efb04c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef4145b40 con 0x7f7ef40c2cf0 2023-10-10T19:11:04.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7efb04c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef4169800 con 0x7f7ef40c30c0 2023-10-10T19:11:04.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef414e290 0x7f7ef41454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:04.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef414e290 0x7f7ef41454c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33678/0 (socket says 172.21.15.175:33678) 2023-10-10T19:11:04.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 -- 172.21.15.175:0/1197240133 learned_addr learned my addr 172.21.15.175:0/1197240133 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:04.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef8de8700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ef40c2cf0 0x7f7ef4167050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:04.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef3fff700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ef40c30c0 0x7f7ef4144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:04.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 -- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ef40c30c0 msgr2=0x7f7ef4144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:04.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ef40c30c0 0x7f7ef4144f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:04.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 -- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ef40c2cf0 msgr2=0x7f7ef4167050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:04.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ef40c2cf0 0x7f7ef4167050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:04.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 -- 172.21.15.175:0/1197240133 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee4009a20 con 0x7f7ef414e290 2023-10-10T19:11:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef3fff700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ef40c30c0 0x7f7ef4144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:11:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.124+0000 7f7ef95e9700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef414e290 0x7f7ef41454c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00c560 tx=0x7f7eec007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.125+0000 7f7ef0ff9700 1 -- 172.21.15.175:0/1197240133 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec011590 con 0x7f7ef414e290 2023-10-10T19:11:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.125+0000 7f7ef0ff9700 1 -- 172.21.15.175:0/1197240133 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eec0116f0 con 0x7f7ef414e290 2023-10-10T19:11:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.125+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef41691f0 con 0x7f7ef414e290 2023-10-10T19:11:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.125+0000 7f7ef0ff9700 1 -- 172.21.15.175:0/1197240133 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec011ab0 con 0x7f7ef414e290 2023-10-10T19:11:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.125+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef4169490 con 0x7f7ef414e290 2023-10-10T19:11:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.125+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef4169040 con 0x7f7ef414e290 2023-10-10T19:11:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.126+0000 7f7ef0ff9700 1 -- 172.21.15.175:0/1197240133 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7eec01b3b0 con 0x7f7ef414e290 2023-10-10T19:11:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.126+0000 7f7ef0ff9700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ed4066f00 0x7f7ed40693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:04.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.126+0000 7f7ef0ff9700 1 -- 172.21.15.175:0/1197240133 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f7eec083fe0 con 0x7f7ef414e290 2023-10-10T19:11:04.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.126+0000 7f7ef8de8700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ed4066f00 0x7f7ed40693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:04.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.127+0000 7f7ef8de8700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ed4066f00 0x7f7ed40693b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40097c0 tx=0x7f7ee4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:04.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.130+0000 7f7ef0ff9700 1 -- 172.21.15.175:0/1197240133 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7eec050c80 con 0x7f7ef414e290 2023-10-10T19:11:04.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.355+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ef4169970 con 0x7f7ef414e290 2023-10-10T19:11:04.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.356+0000 7f7ef0ff9700 1 -- 172.21.15.175:0/1197240133 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f7eec017070 con 0x7f7ef414e290 2023-10-10T19:11:04.357 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:04.357 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:03.690976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:04.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ed4066f00 msgr2=0x7f7ed40693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:04.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ed4066f00 0x7f7ed40693b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40097c0 tx=0x7f7ee4017000 comp rx=0 tx=0).stop 2023-10-10T19:11:04.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef414e290 msgr2=0x7f7ef41454c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:04.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef414e290 0x7f7ef41454c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00c560 tx=0x7f7eec007ef0 comp rx=0 tx=0).stop 2023-10-10T19:11:04.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 shutdown_connections 2023-10-10T19:11:04.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ed4066f00 0x7f7ed40693b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:04.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7ef414e290 0x7f7ef41454c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:04.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ef40c2cf0 0x7f7ef4167050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:04.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 --2- 172.21.15.175:0/1197240133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7ef40c30c0 0x7f7ef4144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:04.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 >> 172.21.15.175:0/1197240133 conn(0x7f7ef4068900 msgr2=0x7f7ef406e500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:04.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 shutdown_connections 2023-10-10T19:11:04.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:04.358+0000 7f7efb04c700 1 -- 172.21.15.175:0/1197240133 wait complete. 2023-10-10T19:11:04.361 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 204 2023-10-10T19:11:04.371 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 204, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 204, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:11:03.690976+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 6117, "mds_2": 6106, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:11:06.372 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:06.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.497+0000 7f8f64b2f700 1 Processor -- start 2023-10-10T19:11:06.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.497+0000 7f8f64b2f700 1 -- start start 2023-10-10T19:11:06.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f64b2f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8f600c3210 0x7f8f60159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:06.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f64b2f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8f600c35e0 0x7f8f600c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:06.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f64b2f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f600c5e40 con 0x7f8f600c9390 2023-10-10T19:11:06.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f64b2f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f600c4c50 con 0x7f8f600c35e0 2023-10-10T19:11:06.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f64b2f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f600ca400 con 0x7f8f600c3210 2023-10-10T19:11:06.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f55d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8f600c35e0 0x7f8f600c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:06.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f5ed9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8f600c9390 0x7f8f600c62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45328/0 (socket says 172.21.15.175:45328) 2023-10-10T19:11:06.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f5ed9d700 1 -- 172.21.15.175:0/459297720 learned_addr learned my addr 172.21.15.175:0/459297720 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:06.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.498+0000 7f8f5e59c700 1 --2- 172.21.15.175:0/459297720 >> v2:172.21.15.191:3300/0 conn(0x7f8f600c3210 0x7f8f60159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:06.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f55d9b700 1 -- 172.21.15.175:0/459297720 >> v2:172.21.15.191:3300/0 conn(0x7f8f600c3210 msgr2=0x7f8f60159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:06.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f55d9b700 1 --2- 172.21.15.175:0/459297720 >> v2:172.21.15.191:3300/0 conn(0x7f8f600c3210 0x7f8f60159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f55d9b700 1 -- 172.21.15.175:0/459297720 >> v1:172.21.15.175:6789/0 conn(0x7f8f600c9390 legacy=0x7f8f600c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:06.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f55d9b700 1 -- 172.21.15.175:0/459297720 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f600c9880 con 0x7f8f600c35e0 2023-10-10T19:11:06.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f5dd9b700 1 -- 172.21.15.175:0/459297720 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863341374 0 0) 0x7f8f600c5e40 con 0x7f8f600c9390 2023-10-10T19:11:06.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f5e59c700 1 --2- 172.21.15.175:0/459297720 >> v2:172.21.15.191:3300/0 conn(0x7f8f600c3210 0x7f8f60159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:11:06.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f55d9b700 1 --2- 172.21.15.175:0/459297720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c35e0 0x7f8f600c3df0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8f60067f90 tx=0x7f8f480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34cd66b6def00f5c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:06.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f5dd9b700 1 -- 172.21.15.175:0/459297720 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f48009e80 con 0x7f8f600c35e0 2023-10-10T19:11:06.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f5dd9b700 1 -- 172.21.15.175:0/459297720 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4800b820 con 0x7f8f600c35e0 2023-10-10T19:11:06.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f5dd9b700 1 -- 172.21.15.175:0/459297720 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800bba0 con 0x7f8f600c35e0 2023-10-10T19:11:06.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f64b2f700 1 -- 172.21.15.175:0/459297720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c35e0 msgr2=0x7f8f600c3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:06.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/459297720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c35e0 0x7f8f600c3df0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8f60067f90 tx=0x7f8f480091a0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f64b2f700 1 -- 172.21.15.175:0/459297720 shutdown_connections 2023-10-10T19:11:06.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/459297720 >> v2:172.21.15.191:3300/0 conn(0x7f8f600c3210 0x7f8f60159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/459297720 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c35e0 0x7f8f600c3df0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f64b2f700 1 -- 172.21.15.175:0/459297720 >> 172.21.15.175:0/459297720 conn(0x7f8f6006c110 msgr2=0x7f8f6006c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:06.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f64b2f700 1 -- 172.21.15.175:0/459297720 shutdown_connections 2023-10-10T19:11:06.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.499+0000 7f8f64b2f700 1 -- 172.21.15.175:0/459297720 wait complete. 2023-10-10T19:11:06.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f64b2f700 1 Processor -- start 2023-10-10T19:11:06.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f64b2f700 1 -- start start 2023-10-10T19:11:06.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f64b2f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f600c3210 0x7f8f60156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:06.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f64b2f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f600c35e0 0x7f8f60156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:06.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f64b2f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c9390 0x7f8f60166140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:06.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f64b2f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600c5840 con 0x7f8f600c9390 2023-10-10T19:11:06.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f64b2f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600b9320 con 0x7f8f600c3210 2023-10-10T19:11:06.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f64b2f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600c27d0 con 0x7f8f600c35e0 2023-10-10T19:11:06.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f5ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c9390 0x7f8f60166140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:06.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f5ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c9390 0x7f8f60166140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38262/0 (socket says 172.21.15.175:38262) 2023-10-10T19:11:06.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.501+0000 7f8f5ed9d700 1 -- 172.21.15.175:0/3917986551 learned_addr learned my addr 172.21.15.175:0/3917986551 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:06.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f5e59c700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f600c3210 0x7f8f60156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:06.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f55d9b700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f600c35e0 0x7f8f60156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:06.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f5ed9d700 1 -- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f600c35e0 msgr2=0x7f8f60156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:06.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f5ed9d700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f600c35e0 0x7f8f60156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f5ed9d700 1 -- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f600c3210 msgr2=0x7f8f60156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:06.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f5ed9d700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f600c3210 0x7f8f60156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f5ed9d700 1 -- 172.21.15.175:0/3917986551 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48009420 con 0x7f8f600c9390 2023-10-10T19:11:06.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f5e59c700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f600c3210 0x7f8f60156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:11:06.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f55d9b700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f600c35e0 0x7f8f60156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:11:06.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f5ed9d700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c9390 0x7f8f60166140 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800cbc0 tx=0x7f8f58001940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:06.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f56ffd700 1 -- 172.21.15.175:0/3917986551 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5800a900 con 0x7f8f600c9390 2023-10-10T19:11:06.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f56ffd700 1 -- 172.21.15.175:0/3917986551 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5800aa80 con 0x7f8f600c9390 2023-10-10T19:11:06.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f60169ff0 con 0x7f8f600c9390 2023-10-10T19:11:06.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f56ffd700 1 -- 172.21.15.175:0/3917986551 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5800ae40 con 0x7f8f600c9390 2023-10-10T19:11:06.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.502+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f600b9830 con 0x7f8f600c9390 2023-10-10T19:11:06.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.503+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f60067570 con 0x7f8f600c9390 2023-10-10T19:11:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.503+0000 7f8f56ffd700 1 -- 172.21.15.175:0/3917986551 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8f58007300 con 0x7f8f600c9390 2023-10-10T19:11:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.504+0000 7f8f56ffd700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f40066f30 0x7f8f400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.504+0000 7f8f56ffd700 1 -- 172.21.15.175:0/3917986551 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f8f58084eb0 con 0x7f8f600c9390 2023-10-10T19:11:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.504+0000 7f8f5e59c700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f40066f30 0x7f8f400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.504+0000 7f8f5e59c700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f40066f30 0x7f8f400693e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000ac10 tx=0x7f8f50006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:06.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.508+0000 7f8f56ffd700 1 -- 172.21.15.175:0/3917986551 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f58051be0 con 0x7f8f600c9390 2023-10-10T19:11:06.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.734+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f60067d00 con 0x7f8f600c9390 2023-10-10T19:11:06.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.735+0000 7f8f56ffd700 1 -- 172.21.15.175:0/3917986551 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f8f58055200 con 0x7f8f600c9390 2023-10-10T19:11:06.736 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:06.736 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:03.690976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:06.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f40066f30 msgr2=0x7f8f400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:06.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f40066f30 0x7f8f400693e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000ac10 tx=0x7f8f50006f90 comp rx=0 tx=0).stop 2023-10-10T19:11:06.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c9390 msgr2=0x7f8f60166140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:06.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c9390 0x7f8f60166140 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800cbc0 tx=0x7f8f58001940 comp rx=0 tx=0).stop 2023-10-10T19:11:06.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 shutdown_connections 2023-10-10T19:11:06.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8f40066f30 0x7f8f400693e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8f600c9390 0x7f8f60166140 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f600c3210 0x7f8f60156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 --2- 172.21.15.175:0/3917986551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f600c35e0 0x7f8f60156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:06.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 >> 172.21.15.175:0/3917986551 conn(0x7f8f6006c110 msgr2=0x7f8f6006e520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:06.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 shutdown_connections 2023-10-10T19:11:06.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:06.737+0000 7f8f64b2f700 1 -- 172.21.15.175:0/3917986551 wait complete. 2023-10-10T19:11:06.741 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 204 2023-10-10T19:11:06.749 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:11:08.750 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:08.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.874+0000 7f5a24ebc700 1 Processor -- start 2023-10-10T19:11:08.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.874+0000 7f5a24ebc700 1 -- start start 2023-10-10T19:11:08.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.874+0000 7f5a24ebc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5a20155fb0 0x7f5a20159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:08.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.874+0000 7f5a24ebc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a201574b0 con 0x7f5a200b2390 2023-10-10T19:11:08.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.874+0000 7f5a24ebc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a200afad0 con 0x7f5a200b1fc0 2023-10-10T19:11:08.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.874+0000 7f5a24ebc700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a20150770 con 0x7f5a20155fb0 2023-10-10T19:11:08.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.874+0000 7f5a15d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5a200b2390 0x7f5a200aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45330/0 (socket says 172.21.15.175:45330) 2023-10-10T19:11:08.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a15d9b700 1 -- 172.21.15.175:0/2633111506 learned_addr learned my addr 172.21.15.175:0/2633111506 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:08.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1ed9d700 1 --2- 172.21.15.175:0/2633111506 >> v2:172.21.15.191:3300/0 conn(0x7f5a20155fb0 0x7f5a20159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:08.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1dd9b700 1 -- 172.21.15.175:0/2633111506 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508973252 0 0) 0x7f5a201574b0 con 0x7f5a200b2390 2023-10-10T19:11:08.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1dd9b700 1 -- 172.21.15.175:0/2633111506 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a0c003570 con 0x7f5a200b2390 2023-10-10T19:11:08.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1ed9d700 1 -- 172.21.15.175:0/2633111506 >> v1:172.21.15.191:6789/0 conn(0x7f5a200b1fc0 legacy=0x7f5a20068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:08.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1ed9d700 1 -- 172.21.15.175:0/2633111506 >> v1:172.21.15.175:6789/0 conn(0x7f5a200b2390 legacy=0x7f5a200aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:08.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1ed9d700 1 -- 172.21.15.175:0/2633111506 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a200af4c0 con 0x7f5a20155fb0 2023-10-10T19:11:08.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1ed9d700 1 --2- 172.21.15.175:0/2633111506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a20155fb0 0x7f5a20159920 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009e60 tx=0x7f5a18009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f0a09e75bfa1df9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:08.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1dd9b700 1 -- 172.21.15.175:0/2633111506 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1800e070 con 0x7f5a20155fb0 2023-10-10T19:11:08.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1dd9b700 1 -- 172.21.15.175:0/2633111506 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1800bbf0 con 0x7f5a20155fb0 2023-10-10T19:11:08.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.875+0000 7f5a1dd9b700 1 -- 172.21.15.175:0/2633111506 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18012430 con 0x7f5a20155fb0 2023-10-10T19:11:08.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.876+0000 7f5a24ebc700 1 -- 172.21.15.175:0/2633111506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a20155fb0 msgr2=0x7f5a20159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:08.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.876+0000 7f5a24ebc700 1 --2- 172.21.15.175:0/2633111506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a20155fb0 0x7f5a20159920 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009e60 tx=0x7f5a18009780 comp rx=0 tx=0).stop 2023-10-10T19:11:08.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.876+0000 7f5a24ebc700 1 -- 172.21.15.175:0/2633111506 shutdown_connections 2023-10-10T19:11:08.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.876+0000 7f5a24ebc700 1 --2- 172.21.15.175:0/2633111506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a20155fb0 0x7f5a20159920 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:08.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.876+0000 7f5a24ebc700 1 -- 172.21.15.175:0/2633111506 >> 172.21.15.175:0/2633111506 conn(0x7f5a20063230 msgr2=0x7f5a20063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:08.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.876+0000 7f5a24ebc700 1 -- 172.21.15.175:0/2633111506 shutdown_connections 2023-10-10T19:11:08.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.876+0000 7f5a24ebc700 1 -- 172.21.15.175:0/2633111506 wait complete. 2023-10-10T19:11:08.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.877+0000 7f5a24ebc700 1 Processor -- start 2023-10-10T19:11:08.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.877+0000 7f5a24ebc700 1 -- start start 2023-10-10T19:11:08.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a24ebc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a200b1fc0 0x7f5a200cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:08.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a24ebc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a200b2390 0x7f5a200c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:08.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a24ebc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a20155fb0 0x7f5a200c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:08.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a24ebc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a200c3c90 con 0x7f5a20155fb0 2023-10-10T19:11:08.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a24ebc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a200c39f0 con 0x7f5a200b1fc0 2023-10-10T19:11:08.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a24ebc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a200c53c0 con 0x7f5a200b2390 2023-10-10T19:11:08.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a20155fb0 0x7f5a200c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:08.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a20155fb0 0x7f5a200c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38270/0 (socket says 172.21.15.175:38270) 2023-10-10T19:11:08.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 -- 172.21.15.175:0/666849397 learned_addr learned my addr 172.21.15.175:0/666849397 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:08.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1e59c700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a200b1fc0 0x7f5a200cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:08.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a15d9b700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a200b2390 0x7f5a200c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:08.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 -- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a200b2390 msgr2=0x7f5a200c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:08.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a200b2390 0x7f5a200c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:08.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 -- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a200b1fc0 msgr2=0x7f5a200cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:08.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a200b1fc0 0x7f5a200cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:08.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 -- 172.21.15.175:0/666849397 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18009a20 con 0x7f5a20155fb0 2023-10-10T19:11:08.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a1ed9d700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a20155fb0 0x7f5a200c5de0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f5a2006fd90 tx=0x7f5a18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:08.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a16ffd700 1 -- 172.21.15.175:0/666849397 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18027470 con 0x7f5a20155fb0 2023-10-10T19:11:08.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a16ffd700 1 -- 172.21.15.175:0/666849397 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a180275f0 con 0x7f5a20155fb0 2023-10-10T19:11:08.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a16ffd700 1 -- 172.21.15.175:0/666849397 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180279b0 con 0x7f5a20155fb0 2023-10-10T19:11:08.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a200cbd60 con 0x7f5a20155fb0 2023-10-10T19:11:08.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.878+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a200cba80 con 0x7f5a20155fb0 2023-10-10T19:11:08.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.879+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a200c5580 con 0x7f5a20155fb0 2023-10-10T19:11:08.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.880+0000 7f5a16ffd700 1 -- 172.21.15.175:0/666849397 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5a18024070 con 0x7f5a20155fb0 2023-10-10T19:11:08.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.880+0000 7f5a16ffd700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a00066f00 0x7f5a000693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:08.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.880+0000 7f5a16ffd700 1 -- 172.21.15.175:0/666849397 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f5a18087570 con 0x7f5a20155fb0 2023-10-10T19:11:08.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.880+0000 7f5a1e59c700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a00066f00 0x7f5a000693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:08.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.881+0000 7f5a1e59c700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a00066f00 0x7f5a000693b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5a10003aa0 tx=0x7f5a10000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:08.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:08.885+0000 7f5a16ffd700 1 -- 172.21.15.175:0/666849397 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a180542b0 con 0x7f5a20155fb0 2023-10-10T19:11:09.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.111+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a20067570 con 0x7f5a20155fb0 2023-10-10T19:11:09.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.112+0000 7f5a16ffd700 1 -- 172.21.15.175:0/666849397 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f5a180578d0 con 0x7f5a20155fb0 2023-10-10T19:11:09.113 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:09.113 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:03.690976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:09.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a00066f00 msgr2=0x7f5a000693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:09.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a00066f00 0x7f5a000693b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5a10003aa0 tx=0x7f5a10000bc0 comp rx=0 tx=0).stop 2023-10-10T19:11:09.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a20155fb0 msgr2=0x7f5a200c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:09.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a20155fb0 0x7f5a200c5de0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f5a2006fd90 tx=0x7f5a18019000 comp rx=0 tx=0).stop 2023-10-10T19:11:09.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 shutdown_connections 2023-10-10T19:11:09.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a00066f00 0x7f5a000693b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:09.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a20155fb0 0x7f5a200c5de0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:09.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a200b1fc0 0x7f5a200cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:09.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 --2- 172.21.15.175:0/666849397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a200b2390 0x7f5a200c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:09.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 >> 172.21.15.175:0/666849397 conn(0x7f5a20063230 msgr2=0x7f5a200c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:09.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 shutdown_connections 2023-10-10T19:11:09.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:09.114+0000 7f5a24ebc700 1 -- 172.21.15.175:0/666849397 wait complete. 2023-10-10T19:11:09.118 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 204 2023-10-10T19:11:09.127 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:11:11.128 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:11.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.255+0000 7f0485ad6700 1 Processor -- start 2023-10-10T19:11:11.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.255+0000 7f0485ad6700 1 -- start start 2023-10-10T19:11:11.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f0485ad6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f048006bda0 0x7f048006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:11.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f0485ad6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048006a8c0 con 0x7f04801595a0 2023-10-10T19:11:11.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f0485ad6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800c5ab0 con 0x7f04800b0240 2023-10-10T19:11:11.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f0485ad6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0480159fb0 con 0x7f048006bda0 2023-10-10T19:11:11.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f0484ad4700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f04801595a0 0x7f04800afb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45336/0 (socket says 172.21.15.175:45336) 2023-10-10T19:11:11.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f0484ad4700 1 -- 172.21.15.175:0/3388657942 learned_addr learned my addr 172.21.15.175:0/3388657942 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:11.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f04852d5700 1 --2- 172.21.15.175:0/3388657942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f048006bda0 0x7f048006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:11.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f047f7fe700 1 -- 172.21.15.175:0/3388657942 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222617336 0 0) 0x7f048006a8c0 con 0x7f04801595a0 2023-10-10T19:11:11.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f047f7fe700 1 -- 172.21.15.175:0/3388657942 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0468003570 con 0x7f04801595a0 2023-10-10T19:11:11.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.256+0000 7f047f7fe700 1 -- 172.21.15.175:0/3388657942 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733554187 0 0) 0x7f04800c5ab0 con 0x7f04800b0240 2023-10-10T19:11:11.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f047f7fe700 1 -- 172.21.15.175:0/3388657942 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048006a8c0 con 0x7f04800b0240 2023-10-10T19:11:11.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f04852d5700 1 -- 172.21.15.175:0/3388657942 >> v1:172.21.15.191:6789/0 conn(0x7f04800b0240 legacy=0x7f048015e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:11.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f04852d5700 1 -- 172.21.15.175:0/3388657942 >> v1:172.21.15.175:6789/0 conn(0x7f04801595a0 legacy=0x7f04800afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:11.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f04852d5700 1 -- 172.21.15.175:0/3388657942 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04800c57d0 con 0x7f048006bda0 2023-10-10T19:11:11.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f04852d5700 1 --2- 172.21.15.175:0/3388657942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f048006bda0 0x7f048006dc90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0478009910 tx=0x7f04780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32d7cc3b1fe5fec server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:11.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f047f7fe700 1 -- 172.21.15.175:0/3388657942 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047800b690 con 0x7f048006bda0 2023-10-10T19:11:11.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f047f7fe700 1 -- 172.21.15.175:0/3388657942 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047800b7f0 con 0x7f048006bda0 2023-10-10T19:11:11.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f047f7fe700 1 -- 172.21.15.175:0/3388657942 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047800bb90 con 0x7f048006bda0 2023-10-10T19:11:11.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f0485ad6700 1 -- 172.21.15.175:0/3388657942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f048006bda0 msgr2=0x7f048006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:11.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f0485ad6700 1 --2- 172.21.15.175:0/3388657942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f048006bda0 0x7f048006dc90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0478009910 tx=0x7f04780091a0 comp rx=0 tx=0).stop 2023-10-10T19:11:11.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f0485ad6700 1 -- 172.21.15.175:0/3388657942 shutdown_connections 2023-10-10T19:11:11.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f0485ad6700 1 --2- 172.21.15.175:0/3388657942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f048006bda0 0x7f048006dc90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:11.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f0485ad6700 1 -- 172.21.15.175:0/3388657942 >> 172.21.15.175:0/3388657942 conn(0x7f0480063cd0 msgr2=0x7f048006d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:11.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.257+0000 7f0485ad6700 1 -- 172.21.15.175:0/3388657942 shutdown_connections 2023-10-10T19:11:11.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.258+0000 7f0485ad6700 1 -- 172.21.15.175:0/3388657942 wait complete. 2023-10-10T19:11:11.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.258+0000 7f0485ad6700 1 Processor -- start 2023-10-10T19:11:11.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0485ad6700 1 -- start start 2023-10-10T19:11:11.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0485ad6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f048006bda0 0x7f0480166200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:11.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0485ad6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04800b0240 0x7f04800c2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:11.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0485ad6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04801595a0 0x7f04800c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:11.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0485ad6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0480166740 con 0x7f048006bda0 2023-10-10T19:11:11.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0485ad6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04800c8ff0 con 0x7f04801595a0 2023-10-10T19:11:11.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0485ad6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04800c37d0 con 0x7f04800b0240 2023-10-10T19:11:11.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0484ad4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f048006bda0 0x7f0480166200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:11.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0484ad4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f048006bda0 0x7f0480166200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38280/0 (socket says 172.21.15.175:38280) 2023-10-10T19:11:11.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.259+0000 7f0484ad4700 1 -- 172.21.15.175:0/1111569750 learned_addr learned my addr 172.21.15.175:0/1111569750 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:11.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f04852d5700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04801595a0 0x7f04800c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:11.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f047ffff700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04800b0240 0x7f04800c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:11.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f0484ad4700 1 -- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04800b0240 msgr2=0x7f04800c2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:11.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f0484ad4700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04800b0240 0x7f04800c2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:11.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f0484ad4700 1 -- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04801595a0 msgr2=0x7f04800c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:11.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f0484ad4700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04801595a0 0x7f04800c3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:11.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f0484ad4700 1 -- 172.21.15.175:0/1111569750 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0478009420 con 0x7f048006bda0 2023-10-10T19:11:11.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f047ffff700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04800b0240 0x7f04800c2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:11:11.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f0484ad4700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f048006bda0 0x7f0480166200 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f047000cd30 tx=0x7f0470001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:11.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f047cff9700 1 -- 172.21.15.175:0/1111569750 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047000aaf0 con 0x7f048006bda0 2023-10-10T19:11:11.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f047cff9700 1 -- 172.21.15.175:0/1111569750 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047000ac50 con 0x7f048006bda0 2023-10-10T19:11:11.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04800c2c00 con 0x7f048006bda0 2023-10-10T19:11:11.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f047cff9700 1 -- 172.21.15.175:0/1111569750 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04700070f0 con 0x7f048006bda0 2023-10-10T19:11:11.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.260+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04800c2ea0 con 0x7f048006bda0 2023-10-10T19:11:11.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.261+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048006a620 con 0x7f048006bda0 2023-10-10T19:11:11.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.261+0000 7f047cff9700 1 -- 172.21.15.175:0/1111569750 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0470007460 con 0x7f048006bda0 2023-10-10T19:11:11.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.262+0000 7f047cff9700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f046c066f60 0x7f046c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:11.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.262+0000 7f047cff9700 1 -- 172.21.15.175:0/1111569750 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f0470084f50 con 0x7f048006bda0 2023-10-10T19:11:11.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.262+0000 7f047ffff700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f046c066f60 0x7f046c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:11.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.262+0000 7f047ffff700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f046c066f60 0x7f046c069410 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0474007fd0 tx=0x7f0474009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:11.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.266+0000 7f047cff9700 1 -- 172.21.15.175:0/1111569750 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0470051c90 con 0x7f048006bda0 2023-10-10T19:11:11.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.492+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0480067570 con 0x7f048006bda0 2023-10-10T19:11:11.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.493+0000 7f047cff9700 1 -- 172.21.15.175:0/1111569750 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5446 (secure 0 0 0) 0x7f04700552b0 con 0x7f048006bda0 2023-10-10T19:11:11.494 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:11.495 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:09.752559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:11.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f046c066f60 msgr2=0x7f046c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:11.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f046c066f60 0x7f046c069410 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0474007fd0 tx=0x7f0474009040 comp rx=0 tx=0).stop 2023-10-10T19:11:11.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f048006bda0 msgr2=0x7f0480166200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:11.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f048006bda0 0x7f0480166200 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f047000cd30 tx=0x7f0470001b10 comp rx=0 tx=0).stop 2023-10-10T19:11:11.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 shutdown_connections 2023-10-10T19:11:11.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f046c066f60 0x7f046c069410 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:11.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04801595a0 0x7f04800c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:11.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f048006bda0 0x7f0480166200 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:11.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 --2- 172.21.15.175:0/1111569750 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04800b0240 0x7f04800c2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:11.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 >> 172.21.15.175:0/1111569750 conn(0x7f0480063cd0 msgr2=0x7f0480157250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:11.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 shutdown_connections 2023-10-10T19:11:11.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:11.495+0000 7f0485ad6700 1 -- 172.21.15.175:0/1111569750 wait complete. 2023-10-10T19:11:11.501 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 205 2023-10-10T19:11:11.508 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:11:13.510 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:13.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.638+0000 7f289bfff700 1 Processor -- start 2023-10-10T19:11:13.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.638+0000 7f289bfff700 1 -- start start 2023-10-10T19:11:13.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f289c14fd90 0x7f289c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:13.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289bfff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c067fa0 con 0x7f289c0cd040 2023-10-10T19:11:13.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289bfff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c156b70 con 0x7f289c08a100 2023-10-10T19:11:13.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289bfff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f289c0c5600 con 0x7f289c14fd90 2023-10-10T19:11:13.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f289c14fd90 0x7f289c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:13.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f289c0cd040 0x7f289c0684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45348/0 (socket says 172.21.15.175:45348) 2023-10-10T19:11:13.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289b7fe700 1 -- 172.21.15.175:0/50230510 learned_addr learned my addr 172.21.15.175:0/50230510 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:13.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289affd700 1 -- 172.21.15.175:0/50230510 >> v1:172.21.15.191:6789/0 conn(0x7f289c08a100 legacy=0x7f289c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:13.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289affd700 1 -- 172.21.15.175:0/50230510 >> v1:172.21.15.175:6789/0 conn(0x7f289c0cd040 legacy=0x7f289c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:13.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289affd700 1 -- 172.21.15.175:0/50230510 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c150160 con 0x7f289c14fd90 2023-10-10T19:11:13.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f2899ffb700 1 -- 172.21.15.175:0/50230510 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151985599 0 0) 0x7f289c067fa0 con 0x7f289c0cd040 2023-10-10T19:11:13.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.639+0000 7f289affd700 1 --2- 172.21.15.175:0/50230510 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c14fd90 0x7f289c159c30 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f288c0098e0 tx=0x7f288c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87a2bcbb7bd8bec server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:13.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f2899ffb700 1 -- 172.21.15.175:0/50230510 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c009e80 con 0x7f289c14fd90 2023-10-10T19:11:13.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f2899ffb700 1 -- 172.21.15.175:0/50230510 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288c00b8f0 con 0x7f289c14fd90 2023-10-10T19:11:13.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f2899ffb700 1 -- 172.21.15.175:0/50230510 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c00bc70 con 0x7f289c14fd90 2023-10-10T19:11:13.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f289bfff700 1 -- 172.21.15.175:0/50230510 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c14fd90 msgr2=0x7f289c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:13.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f289bfff700 1 --2- 172.21.15.175:0/50230510 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c14fd90 0x7f289c159c30 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f288c0098e0 tx=0x7f288c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:11:13.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f289bfff700 1 -- 172.21.15.175:0/50230510 shutdown_connections 2023-10-10T19:11:13.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f289bfff700 1 --2- 172.21.15.175:0/50230510 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c14fd90 0x7f289c159c30 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:13.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f289bfff700 1 -- 172.21.15.175:0/50230510 >> 172.21.15.175:0/50230510 conn(0x7f289c063d00 msgr2=0x7f289c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:13.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f289bfff700 1 -- 172.21.15.175:0/50230510 shutdown_connections 2023-10-10T19:11:13.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.640+0000 7f289bfff700 1 -- 172.21.15.175:0/50230510 wait complete. 2023-10-10T19:11:13.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.641+0000 7f289bfff700 1 Processor -- start 2023-10-10T19:11:13.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.641+0000 7f289bfff700 1 -- start start 2023-10-10T19:11:13.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c08a100 0x7f289c14ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:13.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f289c0cd040 0x7f289c1662f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:13.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289bfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f289c14fd90 0x7f289c166830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:13.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289bfff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c16a0b0 con 0x7f289c08a100 2023-10-10T19:11:13.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289bfff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c165e60 con 0x7f289c14fd90 2023-10-10T19:11:13.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289bfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f289c156df0 con 0x7f289c0cd040 2023-10-10T19:11:13.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c08a100 0x7f289c14ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:13.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c08a100 0x7f289c14ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38312/0 (socket says 172.21.15.175:38312) 2023-10-10T19:11:13.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289affd700 1 -- 172.21.15.175:0/389468064 learned_addr learned my addr 172.21.15.175:0/389468064 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:13.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289b7fe700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f289c14fd90 0x7f289c166830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:13.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289a7fc700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f289c0cd040 0x7f289c1662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:13.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289affd700 1 -- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f289c0cd040 msgr2=0x7f289c1662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:13.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289affd700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f289c0cd040 0x7f289c1662f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:13.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289affd700 1 -- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f289c14fd90 msgr2=0x7f289c166830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:13.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.642+0000 7f289affd700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f289c14fd90 0x7f289c166830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:13.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.643+0000 7f289affd700 1 -- 172.21.15.175:0/389468064 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c009420 con 0x7f289c08a100 2023-10-10T19:11:13.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.643+0000 7f289affd700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c08a100 0x7f289c14ef30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f288c00b3c0 tx=0x7f288c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:13.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.643+0000 7f2882ffd700 1 -- 172.21.15.175:0/389468064 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c00bc70 con 0x7f289c08a100 2023-10-10T19:11:13.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.643+0000 7f2882ffd700 1 -- 172.21.15.175:0/389468064 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288c024440 con 0x7f289c08a100 2023-10-10T19:11:13.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.643+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289c169dd0 con 0x7f289c08a100 2023-10-10T19:11:13.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.643+0000 7f2882ffd700 1 -- 172.21.15.175:0/389468064 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c024800 con 0x7f289c08a100 2023-10-10T19:11:13.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.643+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289c150960 con 0x7f289c08a100 2023-10-10T19:11:13.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.644+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289c166140 con 0x7f289c08a100 2023-10-10T19:11:13.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.644+0000 7f2882ffd700 1 -- 172.21.15.175:0/389468064 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f288c01a420 con 0x7f289c08a100 2023-10-10T19:11:13.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.645+0000 7f2882ffd700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2888066f30 0x7f28880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:13.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.645+0000 7f2882ffd700 1 -- 172.21.15.175:0/389468064 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f288c0864c0 con 0x7f289c08a100 2023-10-10T19:11:13.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.645+0000 7f289a7fc700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2888066f30 0x7f28880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:13.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.645+0000 7f289a7fc700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2888066f30 0x7f28880693e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f289c155b80 tx=0x7f2890009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:13.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.649+0000 7f2882ffd700 1 -- 172.21.15.175:0/389468064 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f288c053200 con 0x7f289c08a100 2023-10-10T19:11:13.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.879+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f289c067570 con 0x7f289c08a100 2023-10-10T19:11:13.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.879+0000 7f2882ffd700 1 -- 172.21.15.175:0/389468064 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f288c056820 con 0x7f289c08a100 2023-10-10T19:11:13.880 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:13.881 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:11.761697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:resolve","state_seq":13,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:13.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2888066f30 msgr2=0x7f28880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:13.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2888066f30 0x7f28880693e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f289c155b80 tx=0x7f2890009040 comp rx=0 tx=0).stop 2023-10-10T19:11:13.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c08a100 msgr2=0x7f289c14ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:13.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c08a100 0x7f289c14ef30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f288c00b3c0 tx=0x7f288c017000 comp rx=0 tx=0).stop 2023-10-10T19:11:13.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 shutdown_connections 2023-10-10T19:11:13.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2888066f30 0x7f28880693e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:13.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f289c14fd90 0x7f289c166830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:13.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f289c08a100 0x7f289c14ef30 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:13.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 --2- 172.21.15.175:0/389468064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f289c0cd040 0x7f289c1662f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:13.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 >> 172.21.15.175:0/389468064 conn(0x7f289c063d00 msgr2=0x7f289c0684f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:13.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 shutdown_connections 2023-10-10T19:11:13.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:13.882+0000 7f289bfff700 1 -- 172.21.15.175:0/389468064 wait complete. 2023-10-10T19:11:13.885 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 206 2023-10-10T19:11:13.894 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:11:15.896 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:16.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.020+0000 7fb799ec3700 1 Processor -- start 2023-10-10T19:11:16.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.020+0000 7fb799ec3700 1 -- start start 2023-10-10T19:11:16.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.021+0000 7fb799ec3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb7940b1fc0 0x7fb794155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:16.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.021+0000 7fb799ec3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7941574b0 con 0x7fb7940aff60 2023-10-10T19:11:16.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.021+0000 7fb799ec3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7940afad0 con 0x7fb7940b2390 2023-10-10T19:11:16.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.021+0000 7fb799ec3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb794150770 con 0x7fb7940b1fc0 2023-10-10T19:11:16.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.021+0000 7fb7937fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb7940b1fc0 0x7fb794155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:16.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.021+0000 7fb793fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb7940aff60 0x7fb794159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53406/0 (socket says 172.21.15.175:53406) 2023-10-10T19:11:16.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.021+0000 7fb793fff700 1 -- 172.21.15.175:0/721462462 learned_addr learned my addr 172.21.15.175:0/721462462 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:16.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.022+0000 7fb7937fe700 1 -- 172.21.15.175:0/721462462 >> v1:172.21.15.191:6789/0 conn(0x7fb7940b2390 legacy=0x7fb794068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:16.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.022+0000 7fb7937fe700 1 -- 172.21.15.175:0/721462462 >> v1:172.21.15.175:6789/0 conn(0x7fb7940aff60 legacy=0x7fb794159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:16.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.022+0000 7fb7927fc700 1 -- 172.21.15.175:0/721462462 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112325421 0 0) 0x7fb7941574b0 con 0x7fb7940aff60 2023-10-10T19:11:16.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.022+0000 7fb7937fe700 1 -- 172.21.15.175:0/721462462 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7940af4c0 con 0x7fb7940b1fc0 2023-10-10T19:11:16.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.022+0000 7fb7937fe700 1 --2- 172.21.15.175:0/721462462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794155fb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb7840098e0 tx=0x7fb7840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95364587e77ba7e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:16.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.022+0000 7fb7927fc700 1 -- 172.21.15.175:0/721462462 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784009e80 con 0x7fb7940b1fc0 2023-10-10T19:11:16.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.022+0000 7fb7927fc700 1 -- 172.21.15.175:0/721462462 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78400b8f0 con 0x7fb7940b1fc0 2023-10-10T19:11:16.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.022+0000 7fb7927fc700 1 -- 172.21.15.175:0/721462462 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78400bc70 con 0x7fb7940b1fc0 2023-10-10T19:11:16.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.023+0000 7fb799ec3700 1 -- 172.21.15.175:0/721462462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 msgr2=0x7fb794155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:16.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.023+0000 7fb799ec3700 1 --2- 172.21.15.175:0/721462462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794155fb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb7840098e0 tx=0x7fb7840091a0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.023+0000 7fb799ec3700 1 -- 172.21.15.175:0/721462462 shutdown_connections 2023-10-10T19:11:16.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.023+0000 7fb799ec3700 1 --2- 172.21.15.175:0/721462462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794155fb0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.023+0000 7fb799ec3700 1 -- 172.21.15.175:0/721462462 >> 172.21.15.175:0/721462462 conn(0x7fb794063230 msgr2=0x7fb794063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:16.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.023+0000 7fb799ec3700 1 -- 172.21.15.175:0/721462462 shutdown_connections 2023-10-10T19:11:16.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.023+0000 7fb799ec3700 1 -- 172.21.15.175:0/721462462 wait complete. 2023-10-10T19:11:16.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.024+0000 7fb799ec3700 1 Processor -- start 2023-10-10T19:11:16.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.024+0000 7fb799ec3700 1 -- start start 2023-10-10T19:11:16.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.024+0000 7fb799ec3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7940aff60 0x7fb794166360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:16.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb799ec3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:16.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb799ec3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7940b2390 0x7fb794158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:16.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb799ec3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7940cd5e0 con 0x7fb7940b1fc0 2023-10-10T19:11:16.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb799ec3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79414f2d0 con 0x7fb7940aff60 2023-10-10T19:11:16.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb799ec3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79414e7e0 con 0x7fb7940b2390 2023-10-10T19:11:16.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:16.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43404/0 (socket says 172.21.15.175:43404) 2023-10-10T19:11:16.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 -- 172.21.15.175:0/634638538 learned_addr learned my addr 172.21.15.175:0/634638538 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:16.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb7937fe700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7940aff60 0x7fb794166360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:16.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb793fff700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7940b2390 0x7fb794158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:16.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 -- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7940b2390 msgr2=0x7fb794158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7940b2390 0x7fb794158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 -- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7940aff60 msgr2=0x7fb794166360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7940aff60 0x7fb794166360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 -- 172.21.15.175:0/634638538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb784009420 con 0x7fb7940b1fc0 2023-10-10T19:11:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb792ffd700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794159bf0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fb78800c510 tx=0x7fb78800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb77b7fe700 1 -- 172.21.15.175:0/634638538 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788011590 con 0x7fb7940b1fc0 2023-10-10T19:11:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb77b7fe700 1 -- 172.21.15.175:0/634638538 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb788011710 con 0x7fb7940b1fc0 2023-10-10T19:11:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb794158690 con 0x7fb7940b1fc0 2023-10-10T19:11:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb77b7fe700 1 -- 172.21.15.175:0/634638538 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788011ad0 con 0x7fb7940b1fc0 2023-10-10T19:11:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.025+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79414ed10 con 0x7fb7940b1fc0 2023-10-10T19:11:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.026+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7941588b0 con 0x7fb7940b1fc0 2023-10-10T19:11:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.027+0000 7fb77b7fe700 1 -- 172.21.15.175:0/634638538 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb78801b3b0 con 0x7fb7940b1fc0 2023-10-10T19:11:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.027+0000 7fb77b7fe700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb780066f60 0x7fb780069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.027+0000 7fb77b7fe700 1 -- 172.21.15.175:0/634638538 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7fb788083f50 con 0x7fb7940b1fc0 2023-10-10T19:11:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.027+0000 7fb7937fe700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb780066f60 0x7fb780069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:16.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.028+0000 7fb7937fe700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb780066f60 0x7fb780069410 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb7840098e0 tx=0x7fb784017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:16.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.031+0000 7fb77b7fe700 1 -- 172.21.15.175:0/634638538 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7941588b0 con 0x7fb7940b1fc0 2023-10-10T19:11:16.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.257+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb794158880 con 0x7fb7940b1fc0 2023-10-10T19:11:16.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.257+0000 7fb77b7fe700 1 -- 172.21.15.175:0/634638538 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+5448 (secure 0 0 0) 0x7fb788017070 con 0x7fb7940b1fc0 2023-10-10T19:11:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:16.258 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:11.761697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:resolve","state_seq":13,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:16.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb780066f60 msgr2=0x7fb780069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:16.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb780066f60 0x7fb780069410 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb7840098e0 tx=0x7fb784017000 comp rx=0 tx=0).stop 2023-10-10T19:11:16.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 msgr2=0x7fb794159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:16.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794159bf0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fb78800c510 tx=0x7fb78800c8d0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 shutdown_connections 2023-10-10T19:11:16.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb780066f60 0x7fb780069410 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7940b2390 0x7fb794158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7940aff60 0x7fb794166360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 --2- 172.21.15.175:0/634638538 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7940b1fc0 0x7fb794159bf0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:16.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 >> 172.21.15.175:0/634638538 conn(0x7fb794063230 msgr2=0x7fb79414fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:16.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 shutdown_connections 2023-10-10T19:11:16.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:16.260+0000 7fb799ec3700 1 -- 172.21.15.175:0/634638538 wait complete. 2023-10-10T19:11:16.263 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 206 2023-10-10T19:11:16.272 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 206, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 206, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:resolve", "state_seq": 13 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:11:11.761697+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 6117, "mds_2": 6106, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:11:18.283 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:18.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.403+0000 7fc86a088700 1 Processor -- start 2023-10-10T19:11:18.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.403+0000 7fc86a088700 1 -- start start 2023-10-10T19:11:18.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc86a088700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc8640c6330 0x7fc86415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:18.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc86a088700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc8640c7190 0x7fc864063c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:18.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc86a088700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc864157030 con 0x7fc8640c6700 2023-10-10T19:11:18.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc86a088700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640c5000 con 0x7fc8640c7190 2023-10-10T19:11:18.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc86a088700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc864156420 con 0x7fc8640c6330 2023-10-10T19:11:18.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc862ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc8640c7190 0x7fc864063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:18.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc863fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc8640c6700 0x7fc8640c3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53418/0 (socket says 172.21.15.175:53418) 2023-10-10T19:11:18.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc863fff700 1 -- 172.21.15.175:0/129997873 learned_addr learned my addr 172.21.15.175:0/129997873 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:18.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc8637fe700 1 --2- 172.21.15.175:0/129997873 >> v2:172.21.15.191:3300/0 conn(0x7fc8640c6330 0x7fc86415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:18.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc862ffd700 1 -- 172.21.15.175:0/129997873 >> v2:172.21.15.191:3300/0 conn(0x7fc8640c6330 msgr2=0x7fc86415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:18.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.404+0000 7fc862ffd700 1 --2- 172.21.15.175:0/129997873 >> v2:172.21.15.191:3300/0 conn(0x7fc8640c6330 0x7fc86415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc862ffd700 1 -- 172.21.15.175:0/129997873 >> v1:172.21.15.175:6789/0 conn(0x7fc8640c6700 legacy=0x7fc8640c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:18.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc862ffd700 1 -- 172.21.15.175:0/129997873 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc864156700 con 0x7fc8640c7190 2023-10-10T19:11:18.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc8637fe700 1 --2- 172.21.15.175:0/129997873 >> v2:172.21.15.191:3300/0 conn(0x7fc8640c6330 0x7fc86415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:11:18.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc862ffd700 1 --2- 172.21.15.175:0/129997873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c7190 0x7fc864063c20 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc8640c01e0 tx=0x7fc8580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa48cf819b3f5472 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:18.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc8627fc700 1 -- 172.21.15.175:0/129997873 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85800b740 con 0x7fc8640c7190 2023-10-10T19:11:18.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc8627fc700 1 -- 172.21.15.175:0/129997873 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85800b8a0 con 0x7fc8640c7190 2023-10-10T19:11:18.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc8627fc700 1 -- 172.21.15.175:0/129997873 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85800bc40 con 0x7fc8640c7190 2023-10-10T19:11:18.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc86a088700 1 -- 172.21.15.175:0/129997873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c7190 msgr2=0x7fc864063c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:18.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc86a088700 1 --2- 172.21.15.175:0/129997873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c7190 0x7fc864063c20 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc8640c01e0 tx=0x7fc8580091a0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc86a088700 1 -- 172.21.15.175:0/129997873 shutdown_connections 2023-10-10T19:11:18.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc86a088700 1 --2- 172.21.15.175:0/129997873 >> v2:172.21.15.191:3300/0 conn(0x7fc8640c6330 0x7fc86415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc86a088700 1 --2- 172.21.15.175:0/129997873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c7190 0x7fc864063c20 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc86a088700 1 -- 172.21.15.175:0/129997873 >> 172.21.15.175:0/129997873 conn(0x7fc86406ef20 msgr2=0x7fc864063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:18.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc86a088700 1 -- 172.21.15.175:0/129997873 shutdown_connections 2023-10-10T19:11:18.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.405+0000 7fc86a088700 1 -- 172.21.15.175:0/129997873 wait complete. 2023-10-10T19:11:18.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc86a088700 1 Processor -- start 2023-10-10T19:11:18.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc86a088700 1 -- start start 2023-10-10T19:11:18.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc86a088700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8640c6330 0x7fc86414bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:18.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc86a088700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c6700 0x7fc864167490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:18.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc86a088700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8640c7190 0x7fc8641679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:18.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc86a088700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc864166340 con 0x7fc8640c6700 2023-10-10T19:11:18.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc86a088700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640cbf90 con 0x7fc8640c6330 2023-10-10T19:11:18.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc86a088700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640cc270 con 0x7fc8640c7190 2023-10-10T19:11:18.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc862ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c6700 0x7fc864167490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:18.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc862ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c6700 0x7fc864167490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43432/0 (socket says 172.21.15.175:43432) 2023-10-10T19:11:18.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc862ffd700 1 -- 172.21.15.175:0/4058668552 learned_addr learned my addr 172.21.15.175:0/4058668552 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:18.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc8637fe700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8640c6330 0x7fc86414bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:18.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.407+0000 7fc863fff700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8640c7190 0x7fc8641679d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:18.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc862ffd700 1 -- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8640c7190 msgr2=0x7fc8641679d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:18.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc862ffd700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8640c7190 0x7fc8641679d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc862ffd700 1 -- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8640c6330 msgr2=0x7fc86414bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:18.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc862ffd700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8640c6330 0x7fc86414bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc862ffd700 1 -- 172.21.15.175:0/4058668552 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc858009420 con 0x7fc8640c6700 2023-10-10T19:11:18.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc863fff700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8640c7190 0x7fc8641679d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:11:18.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc8637fe700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8640c6330 0x7fc86414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:11:18.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc862ffd700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c6700 0x7fc864167490 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc8580097b0 tx=0x7fc858017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:18.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc84b7fe700 1 -- 172.21.15.175:0/4058668552 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85800bbe0 con 0x7fc8640c6700 2023-10-10T19:11:18.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc84b7fe700 1 -- 172.21.15.175:0/4058668552 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc858026470 con 0x7fc8640c6700 2023-10-10T19:11:18.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8640cc550 con 0x7fc8640c6700 2023-10-10T19:11:18.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc84b7fe700 1 -- 172.21.15.175:0/4058668552 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc858026830 con 0x7fc8640c6700 2023-10-10T19:11:18.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.408+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8640ccb10 con 0x7fc8640c6700 2023-10-10T19:11:18.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.409+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc864067570 con 0x7fc8640c6700 2023-10-10T19:11:18.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.409+0000 7fc84b7fe700 1 -- 172.21.15.175:0/4058668552 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc858023070 con 0x7fc8640c6700 2023-10-10T19:11:18.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.410+0000 7fc84b7fe700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc850066f30 0x7fc8500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:18.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.410+0000 7fc84b7fe700 1 -- 172.21.15.175:0/4058668552 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7fc858086300 con 0x7fc8640c6700 2023-10-10T19:11:18.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.410+0000 7fc8637fe700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc850066f30 0x7fc8500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:18.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.410+0000 7fc8637fe700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc850066f30 0x7fc8500693e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc854007fd0 tx=0x7fc8540077a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:18.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.414+0000 7fc84b7fe700 1 -- 172.21.15.175:0/4058668552 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8580530c0 con 0x7fc8640c6700 2023-10-10T19:11:18.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.640+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8640b21b0 con 0x7fc8640c6700 2023-10-10T19:11:18.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.640+0000 7fc84b7fe700 1 -- 172.21.15.175:0/4058668552 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5450 (secure 0 0 0) 0x7fc8580566e0 con 0x7fc8640c6700 2023-10-10T19:11:18.642 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:18.642 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:17.773280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:18.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc850066f30 msgr2=0x7fc8500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:18.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc850066f30 0x7fc8500693e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc854007fd0 tx=0x7fc8540077a0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c6700 msgr2=0x7fc864167490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:18.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c6700 0x7fc864167490 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc8580097b0 tx=0x7fc858017000 comp rx=0 tx=0).stop 2023-10-10T19:11:18.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 shutdown_connections 2023-10-10T19:11:18.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc850066f30 0x7fc8500693e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8640c7190 0x7fc8641679d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8640c6330 0x7fc86414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 --2- 172.21.15.175:0/4058668552 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c6700 0x7fc864167490 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:18.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.642+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 >> 172.21.15.175:0/4058668552 conn(0x7fc86406ef20 msgr2=0x7fc86414cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:18.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.643+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 shutdown_connections 2023-10-10T19:11:18.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:18.643+0000 7fc86a088700 1 -- 172.21.15.175:0/4058668552 wait complete. 2023-10-10T19:11:18.647 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 207 2023-10-10T19:11:18.656 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:11:20.657 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:20.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.783+0000 7f26bd66a700 1 Processor -- start 2023-10-10T19:11:20.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.783+0000 7f26bd66a700 1 -- start start 2023-10-10T19:11:20.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26bd66a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f26b814fd90 0x7f26b8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:20.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26bd66a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f26b808a100 0x7f26b80c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:20.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26bd66a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 0x7f26b80c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:20.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26bd66a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b80c5600 con 0x7f26b808a100 2023-10-10T19:11:20.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26bd66a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b8067fa0 con 0x7f26b814fd90 2023-10-10T19:11:20.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26bd66a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8156b70 con 0x7f26b80cd040 2023-10-10T19:11:20.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26b77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f26b808a100 0x7f26b80c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:20.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26b7fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f26b814fd90 0x7f26b8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:20.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26bce69700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 0x7f26b80c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:20.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26b77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f26b808a100 0x7f26b80c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43440/0 (socket says 172.21.15.175:43440) 2023-10-10T19:11:20.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.784+0000 7f26b77fe700 1 -- 172.21.15.175:0/3871753493 learned_addr learned my addr 172.21.15.175:0/3871753493 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:20.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b77fe700 1 -- 172.21.15.175:0/3871753493 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 msgr2=0x7f26b80c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:20.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b77fe700 1 --2- 172.21.15.175:0/3871753493 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 0x7f26b80c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:20.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b77fe700 1 -- 172.21.15.175:0/3871753493 >> v2:172.21.15.191:3300/0 conn(0x7f26b814fd90 msgr2=0x7f26b8159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:20.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b77fe700 1 --2- 172.21.15.175:0/3871753493 >> v2:172.21.15.191:3300/0 conn(0x7f26b814fd90 0x7f26b8159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:20.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b77fe700 1 -- 172.21.15.175:0/3871753493 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b8150160 con 0x7f26b808a100 2023-10-10T19:11:20.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b77fe700 1 --2- 172.21.15.175:0/3871753493 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b80c5eb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f26b8065080 tx=0x7f26ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6cab01d0ef7e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:20.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b6ffd700 1 -- 172.21.15.175:0/3871753493 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac00b640 con 0x7f26b808a100 2023-10-10T19:11:20.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b6ffd700 1 -- 172.21.15.175:0/3871753493 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac00b7a0 con 0x7f26b808a100 2023-10-10T19:11:20.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26b6ffd700 1 -- 172.21.15.175:0/3871753493 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac00bb60 con 0x7f26b808a100 2023-10-10T19:11:20.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26bd66a700 1 -- 172.21.15.175:0/3871753493 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 msgr2=0x7f26b80c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:20.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26bd66a700 1 --2- 172.21.15.175:0/3871753493 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b80c5eb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f26b8065080 tx=0x7f26ac009250 comp rx=0 tx=0).stop 2023-10-10T19:11:20.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26bd66a700 1 -- 172.21.15.175:0/3871753493 shutdown_connections 2023-10-10T19:11:20.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26bd66a700 1 --2- 172.21.15.175:0/3871753493 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 0x7f26b80c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:20.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26bd66a700 1 --2- 172.21.15.175:0/3871753493 >> v2:172.21.15.191:3300/0 conn(0x7f26b814fd90 0x7f26b8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:20.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26bd66a700 1 --2- 172.21.15.175:0/3871753493 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b80c5eb0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:20.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.785+0000 7f26bd66a700 1 -- 172.21.15.175:0/3871753493 >> 172.21.15.175:0/3871753493 conn(0x7f26b8063d00 msgr2=0x7f26b80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:20.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.786+0000 7f26bd66a700 1 -- 172.21.15.175:0/3871753493 shutdown_connections 2023-10-10T19:11:20.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.786+0000 7f26bd66a700 1 -- 172.21.15.175:0/3871753493 wait complete. 2023-10-10T19:11:20.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.787+0000 7f26bd66a700 1 Processor -- start 2023-10-10T19:11:20.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.787+0000 7f26bd66a700 1 -- start start 2023-10-10T19:11:20.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.787+0000 7f26bd66a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b8168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:20.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.787+0000 7f26bd66a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 0x7f26b8163e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:20.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.787+0000 7f26bd66a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26b814fd90 0x7f26b80b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:20.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.787+0000 7f26bd66a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8168a10 con 0x7f26b808a100 2023-10-10T19:11:20.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.787+0000 7f26bd66a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8150c70 con 0x7f26b814fd90 2023-10-10T19:11:20.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.787+0000 7f26bd66a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b81680e0 con 0x7f26b80cd040 2023-10-10T19:11:20.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b8168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:20.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b8168ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43450/0 (socket says 172.21.15.175:43450) 2023-10-10T19:11:20.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 -- 172.21.15.175:0/454206297 learned_addr learned my addr 172.21.15.175:0/454206297 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:20.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b77fe700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 0x7f26b8163e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:20.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26bce69700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26b814fd90 0x7f26b80b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:20.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 -- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 msgr2=0x7f26b8163e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:20.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 0x7f26b8163e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:20.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 -- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26b814fd90 msgr2=0x7f26b80b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:20.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26b814fd90 0x7f26b80b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:20.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 -- 172.21.15.175:0/454206297 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac0094f0 con 0x7f26b808a100 2023-10-10T19:11:20.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26b7fff700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b8168ea0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f26a800e930 tx=0x7f26a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:20.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f269ffff700 1 -- 172.21.15.175:0/454206297 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8013680 con 0x7f26b808a100 2023-10-10T19:11:20.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f269ffff700 1 -- 172.21.15.175:0/454206297 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a80137e0 con 0x7f26b808a100 2023-10-10T19:11:20.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b81683c0 con 0x7f26b808a100 2023-10-10T19:11:20.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f269ffff700 1 -- 172.21.15.175:0/454206297 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8013b80 con 0x7f26b808a100 2023-10-10T19:11:20.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.788+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b80b2190 con 0x7f26b808a100 2023-10-10T19:11:20.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.789+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b8168cf0 con 0x7f26b808a100 2023-10-10T19:11:20.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.790+0000 7f269ffff700 1 -- 172.21.15.175:0/454206297 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f26a800c470 con 0x7f26b808a100 2023-10-10T19:11:20.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.790+0000 7f269ffff700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a4066f30 0x7f26a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:20.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.790+0000 7f269ffff700 1 -- 172.21.15.175:0/454206297 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f26a8084bb0 con 0x7f26b808a100 2023-10-10T19:11:20.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.790+0000 7f26b77fe700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a4066f30 0x7f26a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:20.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.791+0000 7f26b77fe700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a4066f30 0x7f26a40693e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f26b8065080 tx=0x7f26ac017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:20.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:20.794+0000 7f269ffff700 1 -- 172.21.15.175:0/454206297 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26a80518f0 con 0x7f26b808a100 2023-10-10T19:11:21.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.020+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26b8067570 con 0x7f26b808a100 2023-10-10T19:11:21.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.020+0000 7f269ffff700 1 -- 172.21.15.175:0/454206297 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5450 (secure 0 0 0) 0x7f26a8054f10 con 0x7f26b808a100 2023-10-10T19:11:21.022 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:21.022 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:17.773280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:21.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a4066f30 msgr2=0x7f26a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:21.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a4066f30 0x7f26a40693e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f26b8065080 tx=0x7f26ac017000 comp rx=0 tx=0).stop 2023-10-10T19:11:21.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 msgr2=0x7f26b8168ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:21.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b8168ea0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f26a800e930 tx=0x7f26a80091a0 comp rx=0 tx=0).stop 2023-10-10T19:11:21.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 shutdown_connections 2023-10-10T19:11:21.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a4066f30 0x7f26a40693e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:21.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26b814fd90 0x7f26b80b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:21.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26b808a100 0x7f26b8168ea0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:21.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 --2- 172.21.15.175:0/454206297 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26b80cd040 0x7f26b8163e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:21.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.022+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 >> 172.21.15.175:0/454206297 conn(0x7f26b8063d00 msgr2=0x7f26b814ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:21.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.023+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 shutdown_connections 2023-10-10T19:11:21.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:21.023+0000 7f26bd66a700 1 -- 172.21.15.175:0/454206297 wait complete. 2023-10-10T19:11:21.027 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 207 2023-10-10T19:11:21.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:11:22.860 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:11:22.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:11:23.038 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:23.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.161+0000 7f4b6f348700 1 Processor -- start 2023-10-10T19:11:23.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.161+0000 7f4b6f348700 1 -- start start 2023-10-10T19:11:23.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.161+0000 7f4b6f348700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b680b2390 0x7f4b68155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:23.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.161+0000 7f4b6f348700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b681574b0 con 0x7f4b680b1fc0 2023-10-10T19:11:23.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.161+0000 7f4b6f348700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b680afad0 con 0x7f4b680aff60 2023-10-10T19:11:23.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b6f348700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b68150770 con 0x7f4b680b2390 2023-10-10T19:11:23.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b6d0e4700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f4b680b1fc0 0x7f4b68068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53434/0 (socket says 172.21.15.175:53434) 2023-10-10T19:11:23.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b6d0e4700 1 -- 172.21.15.175:0/3247230181 learned_addr learned my addr 172.21.15.175:0/3247230181 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:23.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b6c8e3700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b680b2390 0x7f4b68155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:23.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294370118 0 0) 0x7f4b681574b0 con 0x7f4b680b1fc0 2023-10-10T19:11:23.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b54003570 con 0x7f4b680b1fc0 2023-10-10T19:11:23.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1798665497 0 0) 0x7f4b54003570 con 0x7f4b680b1fc0 2023-10-10T19:11:23.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b680b2390 msgr2=0x7f4b68155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:23.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b680b2390 0x7f4b68155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 >> v1:172.21.15.191:6789/0 conn(0x7f4b680aff60 legacy=0x7f4b68159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:23.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b680af4c0 con 0x7f4b680b1fc0 2023-10-10T19:11:23.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f4b640017f0 con 0x7f4b680b1fc0 2023-10-10T19:11:23.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 >> v1:172.21.15.175:6789/0 conn(0x7f4b680b1fc0 legacy=0x7f4b68068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:23.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b54003a90 0x7f4b54003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:23.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b680b2390 0x7f4b54004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:23.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:23.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b68150770 con 0x7f4b54004d50 2023-10-10T19:11:23.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5400bb40 con 0x7f4b54003a90 2023-10-10T19:11:23.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5400bd00 con 0x7f4b680b2390 2023-10-10T19:11:23.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b6d8e5700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:23.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b6d0e4700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b54003a90 0x7f4b54003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:23.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.162+0000 7f4b6c8e3700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b680b2390 0x7f4b54004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:23.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6d8e5700 1 -- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b680b2390 msgr2=0x7f4b54004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:23.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6d8e5700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b680b2390 0x7f4b54004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6d8e5700 1 -- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b54003a90 msgr2=0x7f4b54003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:23.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6d8e5700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b54003a90 0x7f4b54003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6d8e5700 1 -- 172.21.15.175:0/3247230181 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b54003570 con 0x7f4b54004d50 2023-10-10T19:11:23.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6d8e5700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b540051c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4b60001a60 tx=0x7f4b60007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd52fa78502b736f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:23.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60001ea0 con 0x7f4b54004d50 2023-10-10T19:11:23.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6000c590 con 0x7f4b54004d50 2023-10-10T19:11:23.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b5ffff700 1 -- 172.21.15.175:0/3247230181 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000c930 con 0x7f4b54004d50 2023-10-10T19:11:23.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6f348700 1 -- 172.21.15.175:0/3247230181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 msgr2=0x7f4b540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:23.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6f348700 1 --2- 172.21.15.175:0/3247230181 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b540051c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4b60001a60 tx=0x7f4b60007d30 comp rx=0 tx=0).stop 2023-10-10T19:11:23.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.163+0000 7f4b6d8e5700 1 -- 172.21.15.175:0/3247230181 reap_dead start 2023-10-10T19:11:23.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.164+0000 7f4b6f348700 1 -- 172.21.15.175:0/3247230181 shutdown_connections 2023-10-10T19:11:23.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.164+0000 7f4b6f348700 1 -- 172.21.15.175:0/3247230181 >> 172.21.15.175:0/3247230181 conn(0x7f4b68063230 msgr2=0x7f4b68063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:23.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.164+0000 7f4b6f348700 1 -- 172.21.15.175:0/3247230181 shutdown_connections 2023-10-10T19:11:23.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.164+0000 7f4b6f348700 1 -- 172.21.15.175:0/3247230181 wait complete. 2023-10-10T19:11:23.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.165+0000 7f4b6f348700 1 Processor -- start 2023-10-10T19:11:23.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.165+0000 7f4b6f348700 1 -- start start 2023-10-10T19:11:23.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.165+0000 7f4b6f348700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b6814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:23.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.165+0000 7f4b6f348700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6814ebb0 0x7f4b6814f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:23.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.165+0000 7f4b6f348700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6814bee0 0x7f4b68148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:23.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.165+0000 7f4b6f348700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6814e340 con 0x7f4b54004d50 2023-10-10T19:11:23.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.165+0000 7f4b6f348700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b681695a0 con 0x7f4b6814bee0 2023-10-10T19:11:23.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.165+0000 7f4b6f348700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6814e590 con 0x7f4b6814ebb0 2023-10-10T19:11:23.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b6814d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:23.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b6814d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43462/0 (socket says 172.21.15.175:43462) 2023-10-10T19:11:23.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 -- 172.21.15.175:0/238582196 learned_addr learned my addr 172.21.15.175:0/238582196 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:23.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6c8e3700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6814ebb0 0x7f4b6814f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:23.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d8e5700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6814bee0 0x7f4b68148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:23.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 -- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6814ebb0 msgr2=0x7f4b6814f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:23.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6814ebb0 0x7f4b6814f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 -- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6814bee0 msgr2=0x7f4b68148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:23.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6814bee0 0x7f4b68148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 -- 172.21.15.175:0/238582196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6814e870 con 0x7f4b54004d50 2023-10-10T19:11:23.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6d0e4700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b6814d0c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4b64007a90 tx=0x7f4b680afa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:23.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b5d7fa700 1 -- 172.21.15.175:0/238582196 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6400ad00 con 0x7f4b54004d50 2023-10-10T19:11:23.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b5d7fa700 1 -- 172.21.15.175:0/238582196 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b640133f0 con 0x7f4b54004d50 2023-10-10T19:11:23.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b5d7fa700 1 -- 172.21.15.175:0/238582196 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6400aea0 con 0x7f4b54004d50 2023-10-10T19:11:23.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b68168f50 con 0x7f4b54004d50 2023-10-10T19:11:23.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.166+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b680c3d60 con 0x7f4b54004d50 2023-10-10T19:11:23.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.167+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b681693f0 con 0x7f4b54004d50 2023-10-10T19:11:23.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.168+0000 7f4b5d7fa700 1 -- 172.21.15.175:0/238582196 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4b64013a40 con 0x7f4b54004d50 2023-10-10T19:11:23.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.168+0000 7f4b5d7fa700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b4c066f30 0x7f4b4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:23.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.168+0000 7f4b5d7fa700 1 -- 172.21.15.175:0/238582196 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f4b640857e0 con 0x7f4b54004d50 2023-10-10T19:11:23.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.168+0000 7f4b6c8e3700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b4c066f30 0x7f4b4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:23.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.169+0000 7f4b6c8e3700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b4c066f30 0x7f4b4c0693e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4b680b0a70 tx=0x7f4b58008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:23.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.172+0000 7f4b5d7fa700 1 -- 172.21.15.175:0/238582196 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b64052520 con 0x7f4b54004d50 2023-10-10T19:11:23.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.396+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b68067570 con 0x7f4b54004d50 2023-10-10T19:11:23.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.397+0000 7f4b5d7fa700 1 -- 172.21.15.175:0/238582196 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+5447 (secure 0 0 0) 0x7f4b6405b030 con 0x7f4b54004d50 2023-10-10T19:11:23.399 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:23.399 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:22.786510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:23.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b4c066f30 msgr2=0x7f4b4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:23.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b4c066f30 0x7f4b4c0693e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4b680b0a70 tx=0x7f4b58008040 comp rx=0 tx=0).stop 2023-10-10T19:11:23.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 msgr2=0x7f4b6814d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:23.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b6814d0c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4b64007a90 tx=0x7f4b680afa40 comp rx=0 tx=0).stop 2023-10-10T19:11:23.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 shutdown_connections 2023-10-10T19:11:23.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b4c066f30 0x7f4b4c0693e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b6814bee0 0x7f4b68148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b54004d50 0x7f4b6814d0c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 --2- 172.21.15.175:0/238582196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b6814ebb0 0x7f4b6814f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:23.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 >> 172.21.15.175:0/238582196 conn(0x7f4b68063230 msgr2=0x7f4b680c7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:23.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 shutdown_connections 2023-10-10T19:11:23.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:23.399+0000 7f4b6f348700 1 -- 172.21.15.175:0/238582196 wait complete. 2023-10-10T19:11:23.404 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 208 2023-10-10T19:11:23.413 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:11:25.414 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:25.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.539+0000 7f26cd9c8700 1 Processor -- start 2023-10-10T19:11:25.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.540+0000 7f26cd9c8700 1 -- start start 2023-10-10T19:11:25.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.540+0000 7f26cd9c8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c806dd70 0x7f26c8068710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:25.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.540+0000 7f26cd9c8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f26c8063230 0x7f26c8068c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:25.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.540+0000 7f26cd9c8700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c8068280 con 0x7f26c80c5c50 2023-10-10T19:11:25.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.540+0000 7f26cd9c8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c815a0d0 con 0x7f26c8063230 2023-10-10T19:11:25.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.540+0000 7f26cd9c8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c8159b70 con 0x7f26c806dd70 2023-10-10T19:11:25.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.540+0000 7f26c77fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f26c8063230 0x7f26c8068c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:25.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26be7fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f26c80c5c50 0x7f26c814f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53446/0 (socket says 172.21.15.175:53446) 2023-10-10T19:11:25.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26be7fc700 1 -- 172.21.15.175:0/2784265758 learned_addr learned my addr 172.21.15.175:0/2784265758 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:25.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c6ffd700 1 --2- 172.21.15.175:0/2784265758 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c806dd70 0x7f26c8068710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:25.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c77fe700 1 -- 172.21.15.175:0/2784265758 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c806dd70 msgr2=0x7f26c8068710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:25.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c77fe700 1 --2- 172.21.15.175:0/2784265758 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c806dd70 0x7f26c8068710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c77fe700 1 -- 172.21.15.175:0/2784265758 >> v1:172.21.15.175:6789/0 conn(0x7f26c80c5c50 legacy=0x7f26c814f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:25.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c77fe700 1 -- 172.21.15.175:0/2784265758 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c8067990 con 0x7f26c8063230 2023-10-10T19:11:25.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c67fc700 1 -- 172.21.15.175:0/2784265758 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594079947 0 0) 0x7f26c8068280 con 0x7f26c80c5c50 2023-10-10T19:11:25.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c77fe700 1 --2- 172.21.15.175:0/2784265758 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c8063230 0x7f26c8068c50 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f26c0009e60 tx=0x7f26c0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b7cd55353734de server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:25.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c67fc700 1 -- 172.21.15.175:0/2784265758 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c000e070 con 0x7f26c8063230 2023-10-10T19:11:25.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c67fc700 1 -- 172.21.15.175:0/2784265758 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c000bcc0 con 0x7f26c8063230 2023-10-10T19:11:25.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.541+0000 7f26c67fc700 1 -- 172.21.15.175:0/2784265758 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0012680 con 0x7f26c8063230 2023-10-10T19:11:25.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.542+0000 7f26cd9c8700 1 -- 172.21.15.175:0/2784265758 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c8063230 msgr2=0x7f26c8068c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:25.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.542+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/2784265758 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c8063230 0x7f26c8068c50 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f26c0009e60 tx=0x7f26c0009780 comp rx=0 tx=0).stop 2023-10-10T19:11:25.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.542+0000 7f26cd9c8700 1 -- 172.21.15.175:0/2784265758 shutdown_connections 2023-10-10T19:11:25.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.542+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/2784265758 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c8063230 0x7f26c8068c50 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.542+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/2784265758 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c806dd70 0x7f26c8068710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.542+0000 7f26cd9c8700 1 -- 172.21.15.175:0/2784265758 >> 172.21.15.175:0/2784265758 conn(0x7f26c80c95a0 msgr2=0x7f26c80c5540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:25.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.542+0000 7f26cd9c8700 1 -- 172.21.15.175:0/2784265758 shutdown_connections 2023-10-10T19:11:25.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.542+0000 7f26cd9c8700 1 -- 172.21.15.175:0/2784265758 wait complete. 2023-10-10T19:11:25.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.543+0000 7f26cd9c8700 1 Processor -- start 2023-10-10T19:11:25.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.543+0000 7f26cd9c8700 1 -- start start 2023-10-10T19:11:25.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26cd9c8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c8063230 0x7f26c80c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:25.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26cd9c8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26c806dd70 0x7f26c8147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:25.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26cd9c8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c80c5c50 0x7f26c81691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:25.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26cd9c8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c8068c90 con 0x7f26c80c5c50 2023-10-10T19:11:25.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26cd9c8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80c03f0 con 0x7f26c806dd70 2023-10-10T19:11:25.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26cd9c8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c81655d0 con 0x7f26c8063230 2023-10-10T19:11:25.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c80c5c50 0x7f26c81691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:25.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c80c5c50 0x7f26c81691a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44736/0 (socket says 172.21.15.175:44736) 2023-10-10T19:11:25.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 -- 172.21.15.175:0/3519677961 learned_addr learned my addr 172.21.15.175:0/3519677961 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:25.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c6ffd700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c8063230 0x7f26c80c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:25.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26be7fc700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26c806dd70 0x7f26c8147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:25.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 -- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c8063230 msgr2=0x7f26c80c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:25.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c8063230 0x7f26c80c28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 -- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26c806dd70 msgr2=0x7f26c8147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:25.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26c806dd70 0x7f26c8147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 -- 172.21.15.175:0/3519677961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c0009a20 con 0x7f26c80c5c50 2023-10-10T19:11:25.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26c77fe700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c80c5c50 0x7f26c81691a0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f26c806b270 tx=0x7f26c0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:25.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26bf7fe700 1 -- 172.21.15.175:0/3519677961 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c000beb0 con 0x7f26c80c5c50 2023-10-10T19:11:25.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26bf7fe700 1 -- 172.21.15.175:0/3519677961 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c001e630 con 0x7f26c80c5c50 2023-10-10T19:11:25.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c80c0180 con 0x7f26c80c5c50 2023-10-10T19:11:25.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26bf7fe700 1 -- 172.21.15.175:0/3519677961 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c001ea30 con 0x7f26c80c5c50 2023-10-10T19:11:25.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.544+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c8168c80 con 0x7f26c80c5c50 2023-10-10T19:11:25.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.546+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c814d0c0 con 0x7f26c80c5c50 2023-10-10T19:11:25.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.546+0000 7f26bf7fe700 1 -- 172.21.15.175:0/3519677961 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f26c0024070 con 0x7f26c80c5c50 2023-10-10T19:11:25.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.546+0000 7f26bf7fe700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a8066fb0 0x7f26a8069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:25.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.546+0000 7f26bf7fe700 1 -- 172.21.15.175:0/3519677961 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f26c0087400 con 0x7f26c80c5c50 2023-10-10T19:11:25.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.546+0000 7f26c6ffd700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a8066fb0 0x7f26a8069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:25.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.547+0000 7f26c6ffd700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a8066fb0 0x7f26a8069460 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f26b8006fd0 tx=0x7f26b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:25.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.550+0000 7f26bf7fe700 1 -- 172.21.15.175:0/3519677961 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26c00540a0 con 0x7f26c80c5c50 2023-10-10T19:11:25.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.773+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26c806f680 con 0x7f26c80c5c50 2023-10-10T19:11:25.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.774+0000 7f26bf7fe700 1 -- 172.21.15.175:0/3519677961 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f26c00576c0 con 0x7f26c80c5c50 2023-10-10T19:11:25.775 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:25.776 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:24.802757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:25.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a8066fb0 msgr2=0x7f26a8069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:25.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a8066fb0 0x7f26a8069460 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f26b8006fd0 tx=0x7f26b8008040 comp rx=0 tx=0).stop 2023-10-10T19:11:25.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c80c5c50 msgr2=0x7f26c81691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:25.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c80c5c50 0x7f26c81691a0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f26c806b270 tx=0x7f26c0019000 comp rx=0 tx=0).stop 2023-10-10T19:11:25.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 shutdown_connections 2023-10-10T19:11:25.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26a8066fb0 0x7f26a8069460 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26c80c5c50 0x7f26c81691a0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26c8063230 0x7f26c80c28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 --2- 172.21.15.175:0/3519677961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26c806dd70 0x7f26c8147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:25.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 >> 172.21.15.175:0/3519677961 conn(0x7f26c80c95a0 msgr2=0x7f26c81591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:25.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 shutdown_connections 2023-10-10T19:11:25.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:25.776+0000 7f26cd9c8700 1 -- 172.21.15.175:0/3519677961 wait complete. 2023-10-10T19:11:25.780 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 209 2023-10-10T19:11:25.790 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:11:27.791 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:27.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.914+0000 7f255a630700 1 Processor -- start 2023-10-10T19:11:27.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f255a630700 1 -- start start 2023-10-10T19:11:27.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f255a630700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f255414ffc0 0x7f2554159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:27.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f255a630700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f25540681e0 0x7f25540c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:27.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f255a630700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2554156800 con 0x7f255414fbf0 2023-10-10T19:11:27.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f255a630700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25540c59e0 con 0x7f255414ffc0 2023-10-10T19:11:27.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f255a630700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255415a2b0 con 0x7f25540681e0 2023-10-10T19:11:27.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f2553fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f255414fbf0 0x7f2554063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53448/0 (socket says 172.21.15.175:53448) 2023-10-10T19:11:27.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f25537fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f255414ffc0 0x7f2554159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:27.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.915+0000 7f2553fff700 1 -- 172.21.15.175:0/93916070 learned_addr learned my addr 172.21.15.175:0/93916070 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:27.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f2558bcd700 1 --2- 172.21.15.175:0/93916070 >> v2:172.21.15.191:3300/0 conn(0x7f25540681e0 0x7f25540c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:27.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f25537fe700 1 -- 172.21.15.175:0/93916070 >> v2:172.21.15.191:3300/0 conn(0x7f25540681e0 msgr2=0x7f25540c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:27.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f25537fe700 1 --2- 172.21.15.175:0/93916070 >> v2:172.21.15.191:3300/0 conn(0x7f25540681e0 0x7f25540c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:27.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f25537fe700 1 -- 172.21.15.175:0/93916070 >> v1:172.21.15.175:6789/0 conn(0x7f255414fbf0 legacy=0x7f2554063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:27.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f25537fe700 1 -- 172.21.15.175:0/93916070 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2554063e30 con 0x7f255414ffc0 2023-10-10T19:11:27.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f2552ffd700 1 -- 172.21.15.175:0/93916070 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268787986 0 0) 0x7f2554156800 con 0x7f255414fbf0 2023-10-10T19:11:27.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f25537fe700 1 --2- 172.21.15.175:0/93916070 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414ffc0 0x7f2554159d70 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f25540cd790 tx=0x7f25480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8e230fd30a0fca server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:27.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f2552ffd700 1 -- 172.21.15.175:0/93916070 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548011630 con 0x7f255414ffc0 2023-10-10T19:11:27.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f2552ffd700 1 -- 172.21.15.175:0/93916070 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2548011790 con 0x7f255414ffc0 2023-10-10T19:11:27.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.916+0000 7f2552ffd700 1 -- 172.21.15.175:0/93916070 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548011b30 con 0x7f255414ffc0 2023-10-10T19:11:27.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.917+0000 7f255a630700 1 -- 172.21.15.175:0/93916070 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414ffc0 msgr2=0x7f2554159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:27.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.917+0000 7f255a630700 1 --2- 172.21.15.175:0/93916070 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414ffc0 0x7f2554159d70 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f25540cd790 tx=0x7f25480091a0 comp rx=0 tx=0).stop 2023-10-10T19:11:27.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.917+0000 7f255a630700 1 -- 172.21.15.175:0/93916070 shutdown_connections 2023-10-10T19:11:27.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.917+0000 7f255a630700 1 --2- 172.21.15.175:0/93916070 >> v2:172.21.15.191:3300/0 conn(0x7f25540681e0 0x7f25540c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:27.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.917+0000 7f255a630700 1 --2- 172.21.15.175:0/93916070 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414ffc0 0x7f2554159d70 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:27.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.917+0000 7f255a630700 1 -- 172.21.15.175:0/93916070 >> 172.21.15.175:0/93916070 conn(0x7f25540cc760 msgr2=0x7f25540ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:27.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.917+0000 7f255a630700 1 -- 172.21.15.175:0/93916070 shutdown_connections 2023-10-10T19:11:27.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.917+0000 7f255a630700 1 -- 172.21.15.175:0/93916070 wait complete. 2023-10-10T19:11:27.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.918+0000 7f255a630700 1 Processor -- start 2023-10-10T19:11:27.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.918+0000 7f255a630700 1 -- start start 2023-10-10T19:11:27.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f255a630700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25540681e0 0x7f25541678f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:27.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f255a630700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414fbf0 0x7f25540c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:27.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f255a630700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255414ffc0 0x7f2554169280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:27.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f255a630700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2554069500 con 0x7f255414fbf0 2023-10-10T19:11:27.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f255a630700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2554153760 con 0x7f25540681e0 2023-10-10T19:11:27.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f255a630700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25540635d0 con 0x7f255414ffc0 2023-10-10T19:11:27.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414fbf0 0x7f25540c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414fbf0 0x7f25540c9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44746/0 (socket says 172.21.15.175:44746) 2023-10-10T19:11:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 -- 172.21.15.175:0/1140729141 learned_addr learned my addr 172.21.15.175:0/1140729141 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f2553fff700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25540681e0 0x7f25541678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f2558bcd700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255414ffc0 0x7f2554169280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 -- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255414ffc0 msgr2=0x7f2554169280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:27.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255414ffc0 0x7f2554169280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:27.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 -- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25540681e0 msgr2=0x7f25541678f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:27.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25540681e0 0x7f25541678f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:27.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 -- 172.21.15.175:0/1140729141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2548009420 con 0x7f255414fbf0 2023-10-10T19:11:27.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f2553fff700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25540681e0 0x7f25541678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:11:27.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f25537fe700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414fbf0 0x7f25540c9b40 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f25480091e0 tx=0x7f2548017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:27.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f253bfff700 1 -- 172.21.15.175:0/1140729141 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548011900 con 0x7f255414fbf0 2023-10-10T19:11:27.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f253bfff700 1 -- 172.21.15.175:0/1140729141 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2548011a80 con 0x7f255414fbf0 2023-10-10T19:11:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f253bfff700 1 -- 172.21.15.175:0/1140729141 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548025670 con 0x7f255414fbf0 2023-10-10T19:11:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25540c8300 con 0x7f255414fbf0 2023-10-10T19:11:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.919+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25541533f0 con 0x7f255414fbf0 2023-10-10T19:11:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.920+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2554067570 con 0x7f255414fbf0 2023-10-10T19:11:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.921+0000 7f253bfff700 1 -- 172.21.15.175:0/1140729141 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2548023070 con 0x7f255414fbf0 2023-10-10T19:11:27.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.921+0000 7f253bfff700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2540066f60 0x7f2540069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:27.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.921+0000 7f253bfff700 1 -- 172.21.15.175:0/1140729141 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f2548085f30 con 0x7f255414fbf0 2023-10-10T19:11:27.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.921+0000 7f2553fff700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2540066f60 0x7f2540069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:27.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.922+0000 7f2553fff700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2540066f60 0x7f2540069410 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2544001360 tx=0x7f2544009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:27.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:27.925+0000 7f253bfff700 1 -- 172.21.15.175:0/1140729141 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2548052c70 con 0x7f255414fbf0 2023-10-10T19:11:28.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.148+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2554069780 con 0x7f255414fbf0 2023-10-10T19:11:28.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.149+0000 7f253bfff700 1 -- 172.21.15.175:0/1140729141 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5445 (secure 0 0 0) 0x7f2548015090 con 0x7f255414fbf0 2023-10-10T19:11:28.151 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:28.151 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:24.802757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:28.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2540066f60 msgr2=0x7f2540069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:28.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2540066f60 0x7f2540069410 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2544001360 tx=0x7f2544009040 comp rx=0 tx=0).stop 2023-10-10T19:11:28.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414fbf0 msgr2=0x7f25540c9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:28.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414fbf0 0x7f25540c9b40 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f25480091e0 tx=0x7f2548017000 comp rx=0 tx=0).stop 2023-10-10T19:11:28.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 shutdown_connections 2023-10-10T19:11:28.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2540066f60 0x7f2540069410 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:28.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255414ffc0 0x7f2554169280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:28.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25540681e0 0x7f25541678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:28.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 --2- 172.21.15.175:0/1140729141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f255414fbf0 0x7f25540c9b40 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:28.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 >> 172.21.15.175:0/1140729141 conn(0x7f25540cc760 msgr2=0x7f255414ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:28.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 shutdown_connections 2023-10-10T19:11:28.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:28.151+0000 7f255a630700 1 -- 172.21.15.175:0/1140729141 wait complete. 2023-10-10T19:11:28.156 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 209 2023-10-10T19:11:28.165 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 209, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 209, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6106": { "addr": "172.21.15.191:6836/62572703", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 62572703, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 62572703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6106, "incarnation": 167, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 17 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:11:24.802757+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 6117, "mds_2": 6106, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:11:30.168 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:30.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.294+0000 7fbe0be1c700 1 Processor -- start 2023-10-10T19:11:30.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.294+0000 7fbe0be1c700 1 -- start start 2023-10-10T19:11:30.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe0be1c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe04155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:30.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe0be1c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbe040b2390 0x7fbe04068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:30.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe0be1c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe041574b0 con 0x7fbe04068bd0 2023-10-10T19:11:30.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe0be1c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe04150770 con 0x7fbe040b2390 2023-10-10T19:11:30.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe0be1c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040afad0 con 0x7fbe040b1fc0 2023-10-10T19:11:30.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe093b7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbe040b2390 0x7fbe04068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:30.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe0a3b9700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbe04068bd0 0x7fbe040aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53460/0 (socket says 172.21.15.175:53460) 2023-10-10T19:11:30.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe0a3b9700 1 -- 172.21.15.175:0/151284929 learned_addr learned my addr 172.21.15.175:0/151284929 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:11:30.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe09bb8700 1 --2- 172.21.15.175:0/151284929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe04155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:30.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe093b7700 1 -- 172.21.15.175:0/151284929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 msgr2=0x7fbe04155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:30.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe093b7700 1 --2- 172.21.15.175:0/151284929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe04155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe093b7700 1 -- 172.21.15.175:0/151284929 >> v1:172.21.15.175:6789/0 conn(0x7fbe04068bd0 legacy=0x7fbe040aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:30.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe08bb6700 1 -- 172.21.15.175:0/151284929 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274229032 0 0) 0x7fbe041574b0 con 0x7fbe04068bd0 2023-10-10T19:11:30.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe093b7700 1 -- 172.21.15.175:0/151284929 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe040af4c0 con 0x7fbe040b2390 2023-10-10T19:11:30.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe093b7700 1 --2- 172.21.15.175:0/151284929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe04068690 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbdf80098e0 tx=0x7fbdf80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5963331e19d682f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:30.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe08bb6700 1 -- 172.21.15.175:0/151284929 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8009e80 con 0x7fbe040b2390 2023-10-10T19:11:30.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe08bb6700 1 -- 172.21.15.175:0/151284929 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf800b8f0 con 0x7fbe040b2390 2023-10-10T19:11:30.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.295+0000 7fbe08bb6700 1 -- 172.21.15.175:0/151284929 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf800bc70 con 0x7fbe040b2390 2023-10-10T19:11:30.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.296+0000 7fbe0be1c700 1 -- 172.21.15.175:0/151284929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 msgr2=0x7fbe04068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:30.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.296+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/151284929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe04068690 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbdf80098e0 tx=0x7fbdf80091a0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.296+0000 7fbe0be1c700 1 -- 172.21.15.175:0/151284929 shutdown_connections 2023-10-10T19:11:30.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.296+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/151284929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe04155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.296+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/151284929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe04068690 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.296+0000 7fbe0be1c700 1 -- 172.21.15.175:0/151284929 >> 172.21.15.175:0/151284929 conn(0x7fbe04063230 msgr2=0x7fbe04063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:30.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.296+0000 7fbe0be1c700 1 -- 172.21.15.175:0/151284929 shutdown_connections 2023-10-10T19:11:30.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.296+0000 7fbe0be1c700 1 -- 172.21.15.175:0/151284929 wait complete. 2023-10-10T19:11:30.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.297+0000 7fbe0be1c700 1 Processor -- start 2023-10-10T19:11:30.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.297+0000 7fbe0be1c700 1 -- start start 2023-10-10T19:11:30.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0be1c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe04068bd0 0x7fbe040c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:30.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0be1c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe040c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:30.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0be1c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe040c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:30.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0be1c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040c8fc0 con 0x7fbe040b2390 2023-10-10T19:11:30.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0be1c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040ccf30 con 0x7fbe04068bd0 2023-10-10T19:11:30.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0be1c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040cd210 con 0x7fbe040b1fc0 2023-10-10T19:11:30.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe040c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:30.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe040c5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44764/0 (socket says 172.21.15.175:44764) 2023-10-10T19:11:30.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 -- 172.21.15.175:0/4170033615 learned_addr learned my addr 172.21.15.175:0/4170033615 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:30.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe09bb8700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe04068bd0 0x7fbe040c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:30.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe093b7700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe040c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:30.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 -- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 msgr2=0x7fbe040c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:30.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe040c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 -- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe04068bd0 msgr2=0x7fbe040c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:30.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe04068bd0 0x7fbe040c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 -- 172.21.15.175:0/4170033615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf8009420 con 0x7fbe040b2390 2023-10-10T19:11:30.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe093b7700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe040c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:11:30.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.298+0000 7fbe0a3b9700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe040c5f80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400c5b0 tx=0x7fbdf4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:30.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.299+0000 7fbe01ffb700 1 -- 172.21.15.175:0/4170033615 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4005b30 con 0x7fbe040b2390 2023-10-10T19:11:30.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.299+0000 7fbe01ffb700 1 -- 172.21.15.175:0/4170033615 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf4005c90 con 0x7fbe040b2390 2023-10-10T19:11:30.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.299+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe04071580 con 0x7fbe040b2390 2023-10-10T19:11:30.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.299+0000 7fbe01ffb700 1 -- 172.21.15.175:0/4170033615 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40072e0 con 0x7fbe040b2390 2023-10-10T19:11:30.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.299+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0415d570 con 0x7fbe040b2390 2023-10-10T19:11:30.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.300+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe040c3970 con 0x7fbe040b2390 2023-10-10T19:11:30.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.300+0000 7fbe01ffb700 1 -- 172.21.15.175:0/4170033615 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbdf4007690 con 0x7fbe040b2390 2023-10-10T19:11:30.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.301+0000 7fbe01ffb700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdf0066f60 0x7fbdf0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:30.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.301+0000 7fbe01ffb700 1 -- 172.21.15.175:0/4170033615 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7fbdf4083dc0 con 0x7fbe040b2390 2023-10-10T19:11:30.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.301+0000 7fbe09bb8700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdf0066f60 0x7fbdf0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:30.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.301+0000 7fbe09bb8700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdf0066f60 0x7fbdf0069410 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009fd0 tx=0x7fbdfc009880 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:30.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.305+0000 7fbe01ffb700 1 -- 172.21.15.175:0/4170033615 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbdf404b5a0 con 0x7fbe040b2390 2023-10-10T19:11:30.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.530+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe040cd3d0 con 0x7fbe040b2390 2023-10-10T19:11:30.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.530+0000 7fbe01ffb700 1 -- 172.21.15.175:0/4170033615 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5443 (secure 0 0 0) 0x7fbdf4017020 con 0x7fbe040b2390 2023-10-10T19:11:30.534 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:30.534 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:28.817183+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:30.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdf0066f60 msgr2=0x7fbdf0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdf0066f60 0x7fbdf0069410 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009fd0 tx=0x7fbdfc009880 comp rx=0 tx=0).stop 2023-10-10T19:11:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 msgr2=0x7fbe040c5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe040c5f80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400c5b0 tx=0x7fbdf4007d30 comp rx=0 tx=0).stop 2023-10-10T19:11:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 shutdown_connections 2023-10-10T19:11:30.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdf0066f60 0x7fbdf0069410 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbe040b2390 0x7fbe040c5f80 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe04068bd0 0x7fbe040c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 --2- 172.21.15.175:0/4170033615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbe040b1fc0 0x7fbe040c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 >> 172.21.15.175:0/4170033615 conn(0x7fbe04063230 msgr2=0x7fbe041535e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:30.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 shutdown_connections 2023-10-10T19:11:30.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:30.532+0000 7fbe0be1c700 1 -- 172.21.15.175:0/4170033615 wait complete. 2023-10-10T19:11:30.538 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 210 2023-10-10T19:11:30.546 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5634 2023-10-10T19:11:30.546 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.e 2023-10-10T19:11:34.345 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:11:34.346 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:11:34.346 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.3 2023-10-10T19:11:34.347 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.3 2023-10-10T19:11:34.347 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.3 2023-10-10T19:11:34.347 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.3 2023-10-10T19:11:34.348 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:11:34.547 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-10T19:11:34.548 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-10T19:11:34.548 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-10T19:11:34.550 INFO:tasks.ceph.mds.e:Started 2023-10-10T19:11:34.551 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:11:34.625 INFO:tasks.ceph.mds.e.smithi175.stdout:starting mds.e at 2023-10-10T19:11:34.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.669+0000 7feca9773700 1 Processor -- start 2023-10-10T19:11:34.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.669+0000 7feca9773700 1 -- start start 2023-10-10T19:11:34.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7feca9773700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feca40b1fc0 0x7feca4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:34.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7feca9773700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7feca40b2390 0x7feca4068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:34.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7feca9773700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca4068bd0 0x7feca40aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:34.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7feca9773700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca4150770 con 0x7feca40b2390 2023-10-10T19:11:34.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7feca9773700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca41574b0 con 0x7feca40b1fc0 2023-10-10T19:11:34.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7feca9773700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca40afad0 con 0x7feca4068bd0 2023-10-10T19:11:34.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7fec9bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7feca40b2390 0x7feca4068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:34.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7fec9bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7feca40b2390 0x7feca4068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57860/0 (socket says 172.21.15.175:57860) 2023-10-10T19:11:34.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7fec9bfff700 1 -- 172.21.15.175:0/3806433012 learned_addr learned my addr 172.21.15.175:0/3806433012 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:34.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7feca37fe700 1 --2- 172.21.15.175:0/3806433012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca4068bd0 0x7feca40aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:34.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7feca2ffd700 1 --2- 172.21.15.175:0/3806433012 >> v2:172.21.15.191:3300/0 conn(0x7feca40b1fc0 0x7feca4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:34.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7fec9bfff700 1 -- 172.21.15.175:0/3806433012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca4068bd0 msgr2=0x7feca40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:34.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.670+0000 7fec9bfff700 1 --2- 172.21.15.175:0/3806433012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca4068bd0 0x7feca40aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7fec9bfff700 1 -- 172.21.15.175:0/3806433012 >> v2:172.21.15.191:3300/0 conn(0x7feca40b1fc0 msgr2=0x7feca4155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:11:34.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7fec9bfff700 1 --2- 172.21.15.175:0/3806433012 >> v2:172.21.15.191:3300/0 conn(0x7feca40b1fc0 0x7feca4155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7fec9bfff700 1 -- 172.21.15.175:0/3806433012 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca40af4c0 con 0x7feca40b2390 2023-10-10T19:11:34.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca37fe700 1 --2- 172.21.15.175:0/3806433012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca4068bd0 0x7feca40aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:11:34.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca2ffd700 1 --2- 172.21.15.175:0/3806433012 >> v2:172.21.15.191:3300/0 conn(0x7feca40b1fc0 0x7feca4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:11:34.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7fec9bfff700 1 --2- 172.21.15.175:0/3806433012 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca40b2390 0x7feca4068690 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7feca40b28c0 tx=0x7fec8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa43649733f7147c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:34.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca27fc700 1 -- 172.21.15.175:0/3806433012 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c00b8f0 con 0x7feca40b2390 2023-10-10T19:11:34.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca27fc700 1 -- 172.21.15.175:0/3806433012 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c00ba50 con 0x7feca40b2390 2023-10-10T19:11:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca27fc700 1 -- 172.21.15.175:0/3806433012 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c00bdf0 con 0x7feca40b2390 2023-10-10T19:11:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 -- 172.21.15.175:0/3806433012 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca40b2390 msgr2=0x7feca4068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 --2- 172.21.15.175:0/3806433012 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca40b2390 0x7feca4068690 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7feca40b28c0 tx=0x7fec8c009250 comp rx=0 tx=0).stop 2023-10-10T19:11:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 -- 172.21.15.175:0/3806433012 shutdown_connections 2023-10-10T19:11:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 --2- 172.21.15.175:0/3806433012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca4068bd0 0x7feca40aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 --2- 172.21.15.175:0/3806433012 >> v2:172.21.15.191:3300/0 conn(0x7feca40b1fc0 0x7feca4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 --2- 172.21.15.175:0/3806433012 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca40b2390 0x7feca4068690 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 -- 172.21.15.175:0/3806433012 >> 172.21.15.175:0/3806433012 conn(0x7feca4063230 msgr2=0x7feca4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 -- 172.21.15.175:0/3806433012 shutdown_connections 2023-10-10T19:11:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.671+0000 7feca9773700 1 -- 172.21.15.175:0/3806433012 wait complete. 2023-10-10T19:11:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca9773700 1 Processor -- start 2023-10-10T19:11:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca9773700 1 -- start start 2023-10-10T19:11:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca9773700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca4068bd0 0x7feca4156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca9773700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca40b1fc0 0x7feca40c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca9773700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca40b2390 0x7feca40c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca9773700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca41553b0 con 0x7feca4068bd0 2023-10-10T19:11:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca9773700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca40c0000 con 0x7feca40b1fc0 2023-10-10T19:11:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca9773700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca40c3870 con 0x7feca40b2390 2023-10-10T19:11:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.673+0000 7feca2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca4068bd0 0x7feca4156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca4068bd0 0x7feca4156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57874/0 (socket says 172.21.15.175:57874) 2023-10-10T19:11:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca2ffd700 1 -- 172.21.15.175:0/813481663 learned_addr learned my addr 172.21.15.175:0/813481663 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:11:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7fec9bfff700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca40b1fc0 0x7feca40c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca37fe700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca40b2390 0x7feca40c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca2ffd700 1 -- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca40b2390 msgr2=0x7feca40c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca2ffd700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca40b2390 0x7feca40c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca2ffd700 1 -- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca40b1fc0 msgr2=0x7feca40c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca2ffd700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca40b1fc0 0x7feca40c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca2ffd700 1 -- 172.21.15.175:0/813481663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c0094f0 con 0x7feca4068bd0 2023-10-10T19:11:34.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca2ffd700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca4068bd0 0x7feca4156880 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fec9400c830 tx=0x7fec94008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:34.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7fec9affd700 1 -- 172.21.15.175:0/813481663 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94011600 con 0x7feca4068bd0 2023-10-10T19:11:34.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7fec9affd700 1 -- 172.21.15.175:0/813481663 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec94011760 con 0x7feca4068bd0 2023-10-10T19:11:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca40c69b0 con 0x7feca4068bd0 2023-10-10T19:11:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7fec9affd700 1 -- 172.21.15.175:0/813481663 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94011b20 con 0x7feca4068bd0 2023-10-10T19:11:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.674+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca4154d60 con 0x7feca4068bd0 2023-10-10T19:11:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.675+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca4067570 con 0x7feca4068bd0 2023-10-10T19:11:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.676+0000 7fec9affd700 1 -- 172.21.15.175:0/813481663 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fec94018360 con 0x7feca4068bd0 2023-10-10T19:11:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.676+0000 7fec9affd700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec84066f30 0x7fec840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:11:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.676+0000 7fec9affd700 1 -- 172.21.15.175:0/813481663 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7fec940849d0 con 0x7feca4068bd0 2023-10-10T19:11:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.676+0000 7fec9bfff700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec84066f30 0x7fec840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:11:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.677+0000 7fec9bfff700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec84066f30 0x7fec840693e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7feca40b28c0 tx=0x7fec8c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:11:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.680+0000 7fec9affd700 1 -- 172.21.15.175:0/813481663 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec94051710 con 0x7feca4068bd0 2023-10-10T19:11:34.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.930+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feca4069060 con 0x7feca4068bd0 2023-10-10T19:11:34.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.930+0000 7fec9affd700 1 -- 172.21.15.175:0/813481663 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+6203 (secure 0 0 0) 0x7fec94054d30 con 0x7feca4068bd0 2023-10-10T19:11:34.932 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:11:34.932 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7104,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":211}],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:11:28.817183+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:11:34.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec84066f30 msgr2=0x7fec840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec84066f30 0x7fec840693e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7feca40b28c0 tx=0x7fec8c017000 comp rx=0 tx=0).stop 2023-10-10T19:11:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca4068bd0 msgr2=0x7feca4156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:11:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca4068bd0 0x7feca4156880 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fec9400c830 tx=0x7fec94008040 comp rx=0 tx=0).stop 2023-10-10T19:11:34.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 shutdown_connections 2023-10-10T19:11:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec84066f30 0x7fec840693e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca40b2390 0x7feca40c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca4068bd0 0x7feca4156880 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 --2- 172.21.15.175:0/813481663 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca40b1fc0 0x7feca40c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:11:34.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 >> 172.21.15.175:0/813481663 conn(0x7feca4063230 msgr2=0x7feca41575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:11:34.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 shutdown_connections 2023-10-10T19:11:34.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:11:34.933+0000 7feca9773700 1 -- 172.21.15.175:0/813481663 wait complete. 2023-10-10T19:11:34.938 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 211 2023-10-10T19:11:34.947 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-10T19:11:34.948 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 119 secs before thrashing 2023-10-10T19:11:35.793 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T19:11:35.792+0000 7feff93e2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-10T19:11:35.793 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T19:11:35.792+0000 7feff93e2700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:11:35.794 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T19:11:35.792+0000 7feff93e2700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-10T19:11:35.794 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T19:11:35.792+0000 7feff93e2700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T19:11:35.794 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T19:11:35.792+0000 7feff93e2700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-10T19:11:40.175 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:11:40.176 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:11:40.176 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.4 2023-10-10T19:11:40.176 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.4 2023-10-10T19:11:40.177 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.4 2023-10-10T19:11:40.177 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.4 2023-10-10T19:11:40.177 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:11:44.340 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:11:44.341 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:11:44.341 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.5 2023-10-10T19:11:44.341 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.5 2023-10-10T19:11:44.342 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.5 2023-10-10T19:11:44.342 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.5 2023-10-10T19:11:44.342 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:11:49.763 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:11:49.765 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:11:49.765 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.6 2023-10-10T19:11:49.765 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.6 2023-10-10T19:11:49.766 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.6 2023-10-10T19:11:49.767 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.6 2023-10-10T19:11:49.767 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:11:52.930 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:11:52.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:11:55.695 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:11:55.697 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:11:55.697 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.7 2023-10-10T19:11:55.697 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.7 2023-10-10T19:11:55.698 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.7 2023-10-10T19:11:55.698 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.7 2023-10-10T19:11:55.698 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:11:59.582 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:11:59.583 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:11:59.583 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.8 2023-10-10T19:11:59.584 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.8 2023-10-10T19:11:59.584 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.8 2023-10-10T19:11:59.585 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.8 2023-10-10T19:11:59.585 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:05.326 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:05.327 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:05.327 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.9 2023-10-10T19:12:05.328 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.9 2023-10-10T19:12:05.328 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.9 2023-10-10T19:12:05.329 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.9 2023-10-10T19:12:05.329 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:10.392 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:10.395 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:10.395 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.10 2023-10-10T19:12:10.396 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.10 2023-10-10T19:12:10.396 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.10 2023-10-10T19:12:10.396 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.10 2023-10-10T19:12:10.397 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:16.431 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:16.431 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:16.431 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.11 2023-10-10T19:12:16.432 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.11 2023-10-10T19:12:16.432 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.11 2023-10-10T19:12:16.432 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.11 2023-10-10T19:12:16.433 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:22.306 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:22.307 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:22.307 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.12 2023-10-10T19:12:22.307 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.12 2023-10-10T19:12:22.308 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.12 2023-10-10T19:12:22.308 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.12 2023-10-10T19:12:22.309 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:22.998 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:12:23.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:12:28.691 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:28.691 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:28.692 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.13 2023-10-10T19:12:28.692 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.13 2023-10-10T19:12:28.692 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.13 2023-10-10T19:12:28.693 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.13 2023-10-10T19:12:28.693 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:35.678 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:35.679 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:35.679 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.14 2023-10-10T19:12:35.679 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.14 2023-10-10T19:12:35.679 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.14 2023-10-10T19:12:35.680 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.14 2023-10-10T19:12:35.680 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:45.880 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:45.882 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:45.882 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.15 2023-10-10T19:12:45.882 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.15 2023-10-10T19:12:45.882 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.15 2023-10-10T19:12:45.883 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.15 2023-10-10T19:12:45.884 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:52.242 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:52.242 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:52.243 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.16 2023-10-10T19:12:52.243 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.16 2023-10-10T19:12:52.244 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.16 2023-10-10T19:12:52.244 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.16 2023-10-10T19:12:52.244 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:12:53.071 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:12:53.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:12:57.702 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:12:57.703 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:12:57.703 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.17 2023-10-10T19:12:57.703 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.17 2023-10-10T19:12:57.704 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.17 2023-10-10T19:12:57.704 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.17 2023-10-10T19:12:57.704 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:13:06.802 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:13:06.802 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:13:06.802 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.18 2023-10-10T19:13:06.802 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.18 2023-10-10T19:13:06.803 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.18 2023-10-10T19:13:06.804 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.18 2023-10-10T19:13:06.804 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:13:11.453 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:13:11.454 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:13:11.454 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.19 2023-10-10T19:13:11.455 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.19 2023-10-10T19:13:11.455 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.19 2023-10-10T19:13:11.455 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.19 2023-10-10T19:13:11.455 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:13:23.144 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:13:23.148 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:13:24.240 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:13:24.240 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:13:24.241 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.20 2023-10-10T19:13:24.241 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.20 2023-10-10T19:13:24.242 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.20 2023-10-10T19:13:24.242 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.20 2023-10-10T19:13:24.242 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:13:33.949 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:34.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.075+0000 7f56ef4be700 1 Processor -- start 2023-10-10T19:13:34.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.076+0000 7f56ef4be700 1 -- start start 2023-10-10T19:13:34.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.076+0000 7f56ef4be700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f56e814cc10 0x7f56e806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.076+0000 7f56ef4be700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56e80c52a0 0x7f56e8152970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.076+0000 7f56ef4be700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e8155ca0 con 0x7f56e814c840 2023-10-10T19:13:34.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.076+0000 7f56ef4be700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e8156b00 con 0x7f56e814cc10 2023-10-10T19:13:34.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.076+0000 7f56ef4be700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e806d700 con 0x7f56e80c52a0 2023-10-10T19:13:34.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eca59700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f56e814cc10 0x7f56e806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eca59700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f56e814cc10 0x7f56e806e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48984/0 (socket says 172.21.15.175:48984) 2023-10-10T19:13:34.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eca59700 1 -- 172.21.15.175:0/2711671948 learned_addr learned my addr 172.21.15.175:0/2711671948 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:34.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56ed25a700 1 --1- 172.21.15.175:0/2711671948 >> v1:172.21.15.175:6789/0 conn(0x7f56e814c840 0x7f56e80c4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:59118/0 (socket says 172.21.15.175:59118) 2023-10-10T19:13:34.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eda5b700 1 --2- 172.21.15.175:0/2711671948 >> v2:172.21.15.191:3300/0 conn(0x7f56e80c52a0 0x7f56e8152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eca59700 1 -- 172.21.15.175:0/2711671948 >> v2:172.21.15.191:3300/0 conn(0x7f56e80c52a0 msgr2=0x7f56e8152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:34.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eca59700 1 --2- 172.21.15.175:0/2711671948 >> v2:172.21.15.191:3300/0 conn(0x7f56e80c52a0 0x7f56e8152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eca59700 1 -- 172.21.15.175:0/2711671948 >> v1:172.21.15.175:6789/0 conn(0x7f56e814c840 legacy=0x7f56e80c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eca59700 1 -- 172.21.15.175:0/2711671948 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e8155ef0 con 0x7f56e814cc10 2023-10-10T19:13:34.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56dffff700 1 -- 172.21.15.175:0/2711671948 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988221927 0 0) 0x7f56e8155ca0 con 0x7f56e814c840 2023-10-10T19:13:34.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56eca59700 1 --2- 172.21.15.175:0/2711671948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e814cc10 0x7f56e806e7a0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f56e80bf630 tx=0x7f56d800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1277eb26b53e88 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:34.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56dffff700 1 -- 172.21.15.175:0/2711671948 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d800dce0 con 0x7f56e814cc10 2023-10-10T19:13:34.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56dffff700 1 -- 172.21.15.175:0/2711671948 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d800de40 con 0x7f56e814cc10 2023-10-10T19:13:34.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.077+0000 7f56dffff700 1 -- 172.21.15.175:0/2711671948 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d80196d0 con 0x7f56e814cc10 2023-10-10T19:13:34.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.078+0000 7f56ef4be700 1 -- 172.21.15.175:0/2711671948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e814cc10 msgr2=0x7f56e806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.078+0000 7f56ef4be700 1 --2- 172.21.15.175:0/2711671948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e814cc10 0x7f56e806e7a0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f56e80bf630 tx=0x7f56d800b4f0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.078+0000 7f56ef4be700 1 -- 172.21.15.175:0/2711671948 shutdown_connections 2023-10-10T19:13:34.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.078+0000 7f56ef4be700 1 --2- 172.21.15.175:0/2711671948 >> v2:172.21.15.191:3300/0 conn(0x7f56e80c52a0 0x7f56e8152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.078+0000 7f56ef4be700 1 --2- 172.21.15.175:0/2711671948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e814cc10 0x7f56e806e7a0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.078+0000 7f56ef4be700 1 -- 172.21.15.175:0/2711671948 >> 172.21.15.175:0/2711671948 conn(0x7f56e8068280 msgr2=0x7f56e806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:34.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.078+0000 7f56ef4be700 1 -- 172.21.15.175:0/2711671948 shutdown_connections 2023-10-10T19:13:34.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.078+0000 7f56ef4be700 1 -- 172.21.15.175:0/2711671948 wait complete. 2023-10-10T19:13:34.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.079+0000 7f56ef4be700 1 Processor -- start 2023-10-10T19:13:34.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.079+0000 7f56ef4be700 1 -- start start 2023-10-10T19:13:34.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ef4be700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e80c52a0 0x7f56e8154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ef4be700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56e814c840 0x7f56e8159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ef4be700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56e814cc10 0x7f56e81596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ef4be700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e8159c10 con 0x7f56e80c52a0 2023-10-10T19:13:34.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ef4be700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e8158d00 con 0x7f56e814cc10 2023-10-10T19:13:34.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ef4be700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e814b9b0 con 0x7f56e814c840 2023-10-10T19:13:34.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e80c52a0 0x7f56e8154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e80c52a0 0x7f56e8154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49000/0 (socket says 172.21.15.175:49000) 2023-10-10T19:13:34.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 -- 172.21.15.175:0/1945048267 learned_addr learned my addr 172.21.15.175:0/1945048267 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:34.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 -- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56e814c840 msgr2=0x7f56e8159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56e814c840 0x7f56e8159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56eda5b700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56e814cc10 0x7f56e81596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 -- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56e814cc10 msgr2=0x7f56e81596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56e814cc10 0x7f56e81596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 -- 172.21.15.175:0/1945048267 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d800b790 con 0x7f56e80c52a0 2023-10-10T19:13:34.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.080+0000 7f56ed25a700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e80c52a0 0x7f56e8154fc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f56e400c810 tx=0x7f56e4007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:34.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.081+0000 7f56dd7fa700 1 -- 172.21.15.175:0/1945048267 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4007300 con 0x7f56e80c52a0 2023-10-10T19:13:34.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.081+0000 7f56dd7fa700 1 -- 172.21.15.175:0/1945048267 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e4007460 con 0x7f56e80c52a0 2023-10-10T19:13:34.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.081+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e814bc90 con 0x7f56e80c52a0 2023-10-10T19:13:34.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.081+0000 7f56dd7fa700 1 -- 172.21.15.175:0/1945048267 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e40057f0 con 0x7f56e80c52a0 2023-10-10T19:13:34.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.081+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e8157b00 con 0x7f56e80c52a0 2023-10-10T19:13:34.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.082+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e8067570 con 0x7f56e80c52a0 2023-10-10T19:13:34.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.082+0000 7f56dd7fa700 1 -- 172.21.15.175:0/1945048267 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f56e4005b80 con 0x7f56e80c52a0 2023-10-10T19:13:34.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.082+0000 7f56dd7fa700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56cc066f60 0x7f56cc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.082+0000 7f56dd7fa700 1 -- 172.21.15.175:0/1945048267 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f56e4084540 con 0x7f56e80c52a0 2023-10-10T19:13:34.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.083+0000 7f56eca59700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56cc066f60 0x7f56cc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.083+0000 7f56eca59700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56cc066f60 0x7f56cc069410 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f56e80bf630 tx=0x7f56d801b000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:34.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.087+0000 7f56dd7fa700 1 -- 172.21.15.175:0/1945048267 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56e4051300 con 0x7f56e80c52a0 2023-10-10T19:13:34.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.309+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56e80bf380 con 0x7f56e80c52a0 2023-10-10T19:13:34.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.310+0000 7f56dd7fa700 1 -- 172.21.15.175:0/1945048267 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+6204 (secure 0 0 0) 0x7f56e4054920 con 0x7f56e80c52a0 2023-10-10T19:13:34.311 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:34.311 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7104,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":211}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:29.720314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:34.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.311+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56cc066f60 msgr2=0x7f56cc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.311+0000 7f56ef4be700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56cc066f60 0x7f56cc069410 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f56e80bf630 tx=0x7f56d801b000 comp rx=0 tx=0).stop 2023-10-10T19:13:34.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.311+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e80c52a0 msgr2=0x7f56e8154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.311+0000 7f56ef4be700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e80c52a0 0x7f56e8154fc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f56e400c810 tx=0x7f56e4007d20 comp rx=0 tx=0).stop 2023-10-10T19:13:34.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.312+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 shutdown_connections 2023-10-10T19:13:34.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.312+0000 7f56ef4be700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f56cc066f60 0x7f56cc069410 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.312+0000 7f56ef4be700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56e814cc10 0x7f56e81596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.312+0000 7f56ef4be700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f56e80c52a0 0x7f56e8154fc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.312+0000 7f56ef4be700 1 --2- 172.21.15.175:0/1945048267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56e814c840 0x7f56e8159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.312+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 >> 172.21.15.175:0/1945048267 conn(0x7f56e8068280 msgr2=0x7f56e806e580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:34.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.312+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 shutdown_connections 2023-10-10T19:13:34.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.312+0000 7f56ef4be700 1 -- 172.21.15.175:0/1945048267 wait complete. 2023-10-10T19:13:34.316 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 219 2023-10-10T19:13:34.325 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=2) 2023-10-10T19:13:34.325 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-10T19:13:34.326 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:13:34.337 INFO:tasks.ceph.mds.b:Stopped 2023-10-10T19:13:34.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:34.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.451+0000 7f827ffff700 1 Processor -- start 2023-10-10T19:13:34.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.451+0000 7f827ffff700 1 -- start start 2023-10-10T19:13:34.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828014fd90 0x7f8280159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f82800cd040 0x7f82800684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827ffff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8280067fa0 con 0x7f828008a100 2023-10-10T19:13:34.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827ffff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82800c5600 con 0x7f82800cd040 2023-10-10T19:13:34.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827ffff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8280156b70 con 0x7f828014fd90 2023-10-10T19:13:34.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827f7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f82800cd040 0x7f82800684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827effd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828014fd90 0x7f8280159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827f7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f82800cd040 0x7f82800684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:47802/0 (socket says 172.21.15.175:47802) 2023-10-10T19:13:34.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827e7fc700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f828008a100 0x7f82800c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:42590/0 (socket says 172.21.15.175:42590) 2023-10-10T19:13:34.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827effd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828014fd90 0x7f8280159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33170/0 (socket says 172.21.15.175:33170) 2023-10-10T19:13:34.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.452+0000 7f827f7fe700 1 -- 172.21.15.175:0/3121105407 learned_addr learned my addr 172.21.15.175:0/3121105407 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:34.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827effd700 1 -- 172.21.15.175:0/3121105407 >> v2:172.21.15.191:3300/0 conn(0x7f82800cd040 msgr2=0x7f82800684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:34.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827effd700 1 --2- 172.21.15.175:0/3121105407 >> v2:172.21.15.191:3300/0 conn(0x7f82800cd040 0x7f82800684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827effd700 1 -- 172.21.15.175:0/3121105407 >> v1:172.21.15.191:6789/0 conn(0x7f828008a100 legacy=0x7f82800c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827dffb700 1 -- 172.21.15.175:0/3121105407 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817460445 0 0) 0x7f8280067fa0 con 0x7f828008a100 2023-10-10T19:13:34.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827effd700 1 -- 172.21.15.175:0/3121105407 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280150160 con 0x7f828014fd90 2023-10-10T19:13:34.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827f7fe700 1 --2- 172.21.15.175:0/3121105407 >> v2:172.21.15.191:3300/0 conn(0x7f82800cd040 0x7f82800684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:13:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827effd700 1 --2- 172.21.15.175:0/3121105407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828014fd90 0x7f8280159c30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8270009e60 tx=0x7f8270009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c08801da40466cfe server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827dffb700 1 -- 172.21.15.175:0/3121105407 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827000e070 con 0x7f828014fd90 2023-10-10T19:13:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827dffb700 1 -- 172.21.15.175:0/3121105407 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827000bb60 con 0x7f828014fd90 2023-10-10T19:13:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827dffb700 1 -- 172.21.15.175:0/3121105407 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270012430 con 0x7f828014fd90 2023-10-10T19:13:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827ffff700 1 -- 172.21.15.175:0/3121105407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828014fd90 msgr2=0x7f8280159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827ffff700 1 --2- 172.21.15.175:0/3121105407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828014fd90 0x7f8280159c30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8270009e60 tx=0x7f8270009780 comp rx=0 tx=0).stop 2023-10-10T19:13:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827ffff700 1 -- 172.21.15.175:0/3121105407 shutdown_connections 2023-10-10T19:13:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827ffff700 1 --2- 172.21.15.175:0/3121105407 >> v2:172.21.15.191:3300/0 conn(0x7f82800cd040 0x7f82800684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827ffff700 1 --2- 172.21.15.175:0/3121105407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828014fd90 0x7f8280159c30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.453+0000 7f827ffff700 1 -- 172.21.15.175:0/3121105407 >> 172.21.15.175:0/3121105407 conn(0x7f8280063d00 msgr2=0x7f82800cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:34.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.454+0000 7f827ffff700 1 -- 172.21.15.175:0/3121105407 shutdown_connections 2023-10-10T19:13:34.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.454+0000 7f827ffff700 1 -- 172.21.15.175:0/3121105407 wait complete. 2023-10-10T19:13:34.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.455+0000 7f827ffff700 1 Processor -- start 2023-10-10T19:13:34.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.455+0000 7f827ffff700 1 -- start start 2023-10-10T19:13:34.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828008a100 0x7f82800c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82800cd040 0x7f828014e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827ffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f828014fd90 0x7f8280169140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827ffff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800c9730 con 0x7f82800cd040 2023-10-10T19:13:34.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827ffff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800c9cb0 con 0x7f828014fd90 2023-10-10T19:13:34.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827ffff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82800c8870 con 0x7f828008a100 2023-10-10T19:13:34.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82800cd040 0x7f828014e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82800cd040 0x7f828014e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49010/0 (socket says 172.21.15.175:49010) 2023-10-10T19:13:34.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827e7fc700 1 -- 172.21.15.175:0/749980261 learned_addr learned my addr 172.21.15.175:0/749980261 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:34.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827f7fe700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f828014fd90 0x7f8280169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827effd700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828008a100 0x7f82800c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827e7fc700 1 -- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828008a100 msgr2=0x7f82800c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827e7fc700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828008a100 0x7f82800c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827e7fc700 1 -- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f828014fd90 msgr2=0x7f8280169140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827e7fc700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f828014fd90 0x7f8280169140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827e7fc700 1 -- 172.21.15.175:0/749980261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82780094f0 con 0x7f82800cd040 2023-10-10T19:13:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.456+0000 7f827effd700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828008a100 0x7f82800c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:13:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.457+0000 7f827e7fc700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82800cd040 0x7f828014e260 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f827400c7e0 tx=0x7f8274007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.457+0000 7f8266ffd700 1 -- 172.21.15.175:0/749980261 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274005a70 con 0x7f82800cd040 2023-10-10T19:13:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.457+0000 7f8266ffd700 1 -- 172.21.15.175:0/749980261 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8274005bd0 con 0x7f82800cd040 2023-10-10T19:13:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.457+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8270009a20 con 0x7f82800cd040 2023-10-10T19:13:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.457+0000 7f8266ffd700 1 -- 172.21.15.175:0/749980261 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82740070d0 con 0x7f82800cd040 2023-10-10T19:13:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.457+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82800c3380 con 0x7f82800cd040 2023-10-10T19:13:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.458+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82800c2c90 con 0x7f82800cd040 2023-10-10T19:13:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.458+0000 7f8266ffd700 1 -- 172.21.15.175:0/749980261 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8274007480 con 0x7f82800cd040 2023-10-10T19:13:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.459+0000 7f8266ffd700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f826c066f30 0x7f826c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.459+0000 7f8266ffd700 1 -- 172.21.15.175:0/749980261 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f8274083fd0 con 0x7f82800cd040 2023-10-10T19:13:34.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.459+0000 7f827effd700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f826c066f30 0x7f826c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:34.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.459+0000 7f827effd700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f826c066f30 0x7f826c0693e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f82700097c0 tx=0x7f8270019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:34.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.463+0000 7f8266ffd700 1 -- 172.21.15.175:0/749980261 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8274050dc0 con 0x7f82800cd040 2023-10-10T19:13:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.687+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8280159630 con 0x7f82800cd040 2023-10-10T19:13:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.687+0000 7f8266ffd700 1 -- 172.21.15.175:0/749980261 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+6204 (secure 0 0 0) 0x7f8274017090 con 0x7f82800cd040 2023-10-10T19:13:34.689 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:34.689 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7104,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":211}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:29.720314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f826c066f30 msgr2=0x7f826c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f826c066f30 0x7f826c0693e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f82700097c0 tx=0x7f8270019000 comp rx=0 tx=0).stop 2023-10-10T19:13:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82800cd040 msgr2=0x7f828014e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82800cd040 0x7f828014e260 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f827400c7e0 tx=0x7f8274007e10 comp rx=0 tx=0).stop 2023-10-10T19:13:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 shutdown_connections 2023-10-10T19:13:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f826c066f30 0x7f826c0693e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f828014fd90 0x7f8280169140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f828008a100 0x7f82800c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 --2- 172.21.15.175:0/749980261 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82800cd040 0x7f828014e260 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 >> 172.21.15.175:0/749980261 conn(0x7f8280063d00 msgr2=0x7f8280154b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.689+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 shutdown_connections 2023-10-10T19:13:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:34.690+0000 7f827ffff700 1 -- 172.21.15.175:0/749980261 wait complete. 2023-10-10T19:13:34.693 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 219 2023-10-10T19:13:34.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:13:34.803 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:13:34.803 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:13:34.803 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.21 2023-10-10T19:13:34.804 INFO:tasks.workunit.client.0.smithi175.stdout:checking v0.21 2023-10-10T19:13:34.804 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo checking v0.21 2023-10-10T19:13:34.805 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd .snap/v0.21 2023-10-10T19:13:34.805 INFO:tasks.workunit.client.0.smithi175.stderr:+ git diff --exit-code 2023-10-10T19:13:36.704 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:36.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.822+0000 7fab9f2a4700 1 Processor -- start 2023-10-10T19:13:36.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.822+0000 7fab9f2a4700 1 -- start start 2023-10-10T19:13:36.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab9f2a4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980b1fc0 0x7fab98155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:36.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab9f2a4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab981574b0 con 0x7fab980b2390 2023-10-10T19:13:36.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab9f2a4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab980afad0 con 0x7fab980aff60 2023-10-10T19:13:36.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab9f2a4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab98150770 con 0x7fab980b1fc0 2023-10-10T19:13:36.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab9c83f700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fab980b2390 0x7fab98068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36036/0 (socket says 172.21.15.175:36036) 2023-10-10T19:13:36.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab9c83f700 1 -- 172.21.15.175:0/165956697 learned_addr learned my addr 172.21.15.175:0/165956697 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:13:36.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab9d040700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980b1fc0 0x7fab98155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:36.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207473425 0 0) 0x7fab981574b0 con 0x7fab980b2390 2023-10-10T19:13:36.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab84003570 con 0x7fab980b2390 2023-10-10T19:13:36.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2483355081 0 0) 0x7fab84003570 con 0x7fab980b2390 2023-10-10T19:13:36.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980b1fc0 msgr2=0x7fab98155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:36.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980b1fc0 0x7fab98155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:36.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 >> v1:172.21.15.191:6789/0 conn(0x7fab980aff60 legacy=0x7fab98159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:36.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab980af4c0 con 0x7fab980b2390 2023-10-10T19:13:36.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fab88001470 con 0x7fab980b2390 2023-10-10T19:13:36.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 >> v1:172.21.15.175:6789/0 conn(0x7fab980b2390 legacy=0x7fab98068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:36.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab84003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:36.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980b1fc0 0x7fab84004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:36.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab84004d50 0x7fab840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:36.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab98150770 con 0x7fab84003a90 2023-10-10T19:13:36.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8400bb70 con 0x7fab84004d50 2023-10-10T19:13:36.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.823+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8400bd30 con 0x7fab980b1fc0 2023-10-10T19:13:36.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9d040700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab84003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:36.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9c83f700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980b1fc0 0x7fab84004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:36.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9d841700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab84004d50 0x7fab840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:36.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9d040700 1 -- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980b1fc0 msgr2=0x7fab84004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:36.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9d040700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980b1fc0 0x7fab84004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:36.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9d040700 1 -- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab84004d50 msgr2=0x7fab840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:36.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9d040700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab84004d50 0x7fab840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:36.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9d040700 1 -- 172.21.15.175:0/165956697 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84003570 con 0x7fab84003a90 2023-10-10T19:13:36.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab9d040700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab84003ea0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fab90009c00 tx=0x7fab900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184d361809004584 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:36.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9000e710 con 0x7fab84003a90 2023-10-10T19:13:36.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9000e870 con 0x7fab84003a90 2023-10-10T19:13:36.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.824+0000 7fab8ffff700 1 -- 172.21.15.175:0/165956697 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9000ec30 con 0x7fab84003a90 2023-10-10T19:13:36.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.825+0000 7fab9f2a4700 1 -- 172.21.15.175:0/165956697 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 msgr2=0x7fab84003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:36.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.825+0000 7fab9f2a4700 1 --2- 172.21.15.175:0/165956697 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab84003ea0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fab90009c00 tx=0x7fab900091a0 comp rx=0 tx=0).stop 2023-10-10T19:13:36.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.825+0000 7fab9d841700 1 -- 172.21.15.175:0/165956697 reap_dead start 2023-10-10T19:13:36.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.825+0000 7fab9f2a4700 1 -- 172.21.15.175:0/165956697 shutdown_connections 2023-10-10T19:13:36.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.825+0000 7fab9f2a4700 1 -- 172.21.15.175:0/165956697 >> 172.21.15.175:0/165956697 conn(0x7fab98063230 msgr2=0x7fab98063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:36.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.825+0000 7fab9f2a4700 1 -- 172.21.15.175:0/165956697 shutdown_connections 2023-10-10T19:13:36.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.825+0000 7fab9f2a4700 1 -- 172.21.15.175:0/165956697 wait complete. 2023-10-10T19:13:36.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.826+0000 7fab9f2a4700 1 Processor -- start 2023-10-10T19:13:36.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.826+0000 7fab9f2a4700 1 -- start start 2023-10-10T19:13:36.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9f2a4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab98157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:36.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9f2a4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab9814b790 0x7fab980ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:36.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9f2a4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980cae80 0x7fab980cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:36.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9f2a4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab980cacc0 con 0x7fab84003a90 2023-10-10T19:13:36.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9f2a4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9814b420 con 0x7fab9814b790 2023-10-10T19:13:36.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9f2a4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab981576c0 con 0x7fab980cae80 2023-10-10T19:13:36.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab98157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:36.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab98157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60960/0 (socket says 172.21.15.175:60960) 2023-10-10T19:13:36.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 -- 172.21.15.175:0/1994226317 learned_addr learned my addr 172.21.15.175:0/1994226317 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:36.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9c83f700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab9814b790 0x7fab980ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:36.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d841700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980cae80 0x7fab980cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:36.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 -- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980cae80 msgr2=0x7fab980cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:36.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980cae80 0x7fab980cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:36.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 -- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab9814b790 msgr2=0x7fab980ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:36.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab9814b790 0x7fab980ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:36.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 -- 172.21.15.175:0/1994226317 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab98159c80 con 0x7fab84003a90 2023-10-10T19:13:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d841700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980cae80 0x7fab980cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:13:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9c83f700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab9814b790 0x7fab980ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:13:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9d040700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab98157180 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fab9000e4b0 tx=0x7fab9001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab8d7fa700 1 -- 172.21.15.175:0/1994226317 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90022070 con 0x7fab84003a90 2023-10-10T19:13:36.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab8d7fa700 1 -- 172.21.15.175:0/1994226317 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9000ebf0 con 0x7fab84003a90 2023-10-10T19:13:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab980ca720 con 0x7fab84003a90 2023-10-10T19:13:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab8d7fa700 1 -- 172.21.15.175:0/1994226317 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9001c6c0 con 0x7fab84003a90 2023-10-10T19:13:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.827+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab980ca9e0 con 0x7fab84003a90 2023-10-10T19:13:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.828+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab980b0680 con 0x7fab84003a90 2023-10-10T19:13:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.829+0000 7fab8d7fa700 1 -- 172.21.15.175:0/1994226317 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fab9001ca70 con 0x7fab84003a90 2023-10-10T19:13:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.829+0000 7fab8d7fa700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab7c066f60 0x7fab7c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.829+0000 7fab8d7fa700 1 -- 172.21.15.175:0/1994226317 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7fab90086130 con 0x7fab84003a90 2023-10-10T19:13:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.829+0000 7fab9c83f700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab7c066f60 0x7fab7c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.830+0000 7fab9c83f700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab7c066f60 0x7fab7c069410 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fab88007fd0 tx=0x7fab88009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:36.833+0000 7fab8d7fa700 1 -- 172.21.15.175:0/1994226317 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab90052e70 con 0x7fab84003a90 2023-10-10T19:13:37.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.058+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab98067570 con 0x7fab84003a90 2023-10-10T19:13:37.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.058+0000 7fab8d7fa700 1 -- 172.21.15.175:0/1994226317 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fab90017090 con 0x7fab84003a90 2023-10-10T19:13:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:37.060 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7104,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":211}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:29.720314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:37.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab7c066f60 msgr2=0x7fab7c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:37.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab7c066f60 0x7fab7c069410 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fab88007fd0 tx=0x7fab88009040 comp rx=0 tx=0).stop 2023-10-10T19:13:37.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 msgr2=0x7fab98157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:37.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab98157180 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fab9000e4b0 tx=0x7fab9001a000 comp rx=0 tx=0).stop 2023-10-10T19:13:37.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 shutdown_connections 2023-10-10T19:13:37.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab7c066f60 0x7fab7c069410 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:37.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab980cae80 0x7fab980cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:37.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab84003a90 0x7fab98157180 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:37.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 --2- 172.21.15.175:0/1994226317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab9814b790 0x7fab980ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:37.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 >> 172.21.15.175:0/1994226317 conn(0x7fab98063230 msgr2=0x7fab98152540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:37.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 shutdown_connections 2023-10-10T19:13:37.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:37.060+0000 7fab9f2a4700 1 -- 172.21.15.175:0/1994226317 wait complete. 2023-10-10T19:13:37.064 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 219 2023-10-10T19:13:37.073 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:13:39.075 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:39.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.200+0000 7f5fa4869700 1 Processor -- start 2023-10-10T19:13:39.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.200+0000 7f5fa4869700 1 -- start start 2023-10-10T19:13:39.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5fa4869700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00b1fc0 0x7f5fa0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:39.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5fa4869700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5fa00aff60 0x7f5fa00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:39.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5fa4869700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa01574b0 con 0x7f5fa00b2390 2023-10-10T19:13:39.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5fa4869700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa0150770 con 0x7f5fa00aff60 2023-10-10T19:13:39.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5fa4869700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa00afad0 con 0x7f5fa00b1fc0 2023-10-10T19:13:39.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9ed9d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5fa00aff60 0x7f5fa00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:39.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9e59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00b1fc0 0x7f5fa0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:39.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9ed9d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5fa00aff60 0x7f5fa00b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:47522/0 (socket says 172.21.15.175:47522) 2023-10-10T19:13:39.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f95d9b700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f5fa00b2390 0x7f5fa0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:42262/0 (socket says 172.21.15.175:42262) 2023-10-10T19:13:39.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9ed9d700 1 -- 172.21.15.175:0/3853617975 learned_addr learned my addr 172.21.15.175:0/3853617975 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:39.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9e59c700 1 --2- 172.21.15.175:0/3853617975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00b1fc0 0x7f5fa0155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:58566/0 (socket says 172.21.15.175:58566) 2023-10-10T19:13:39.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9e59c700 1 -- 172.21.15.175:0/3853617975 >> v2:172.21.15.191:3300/0 conn(0x7f5fa00aff60 msgr2=0x7f5fa00b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:39.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9e59c700 1 --2- 172.21.15.175:0/3853617975 >> v2:172.21.15.191:3300/0 conn(0x7f5fa00aff60 0x7f5fa00b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9e59c700 1 -- 172.21.15.175:0/3853617975 >> v1:172.21.15.191:6789/0 conn(0x7f5fa00b2390 legacy=0x7f5fa0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:39.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9e59c700 1 -- 172.21.15.175:0/3853617975 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa00af4c0 con 0x7f5fa00b1fc0 2023-10-10T19:13:39.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.201+0000 7f5f9ed9d700 1 --2- 172.21.15.175:0/3853617975 >> v2:172.21.15.191:3300/0 conn(0x7f5fa00aff60 0x7f5fa00b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:13:39.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5f9e59c700 1 --2- 172.21.15.175:0/3853617975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00b1fc0 0x7f5fa0155fb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009910 tx=0x7f5f900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3841298d1b19e6c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:39.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5f9dd9b700 1 -- 172.21.15.175:0/3853617975 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000b6d0 con 0x7f5fa00b1fc0 2023-10-10T19:13:39.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5f9dd9b700 1 -- 172.21.15.175:0/3853617975 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9000b830 con 0x7f5fa00b1fc0 2023-10-10T19:13:39.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5f9dd9b700 1 -- 172.21.15.175:0/3853617975 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000bbf0 con 0x7f5fa00b1fc0 2023-10-10T19:13:39.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5fa4869700 1 -- 172.21.15.175:0/3853617975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00b1fc0 msgr2=0x7f5fa0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:39.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3853617975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00b1fc0 0x7f5fa0155fb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009910 tx=0x7f5f900091a0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5fa4869700 1 -- 172.21.15.175:0/3853617975 shutdown_connections 2023-10-10T19:13:39.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3853617975 >> v2:172.21.15.191:3300/0 conn(0x7f5fa00aff60 0x7f5fa00b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3853617975 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00b1fc0 0x7f5fa0155fb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5fa4869700 1 -- 172.21.15.175:0/3853617975 >> 172.21.15.175:0/3853617975 conn(0x7f5fa0063230 msgr2=0x7f5fa0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:39.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5fa4869700 1 -- 172.21.15.175:0/3853617975 shutdown_connections 2023-10-10T19:13:39.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.202+0000 7f5fa4869700 1 -- 172.21.15.175:0/3853617975 wait complete. 2023-10-10T19:13:39.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.204+0000 7f5fa4869700 1 Processor -- start 2023-10-10T19:13:39.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.204+0000 7f5fa4869700 1 -- start start 2023-10-10T19:13:39.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.204+0000 7f5fa4869700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00aff60 0x7f5fa015bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:39.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.204+0000 7f5fa4869700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fa00b1fc0 0x7f5fa0165e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:39.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.204+0000 7f5fa4869700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fa00b2390 0x7f5fa0166390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:39.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.204+0000 7f5fa4869700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa00cb090 con 0x7f5fa00b2390 2023-10-10T19:13:39.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.204+0000 7f5fa4869700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0169ef0 con 0x7f5fa00b1fc0 2023-10-10T19:13:39.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.204+0000 7f5fa4869700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa01692e0 con 0x7f5fa00aff60 2023-10-10T19:13:39.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fa00b2390 0x7f5fa0166390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:39.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9e59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00aff60 0x7f5fa015bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:39.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f95d9b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fa00b1fc0 0x7f5fa0165e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:39.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fa00b2390 0x7f5fa0166390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60968/0 (socket says 172.21.15.175:60968) 2023-10-10T19:13:39.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 -- 172.21.15.175:0/3423143077 learned_addr learned my addr 172.21.15.175:0/3423143077 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:39.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9e59c700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00aff60 0x7f5fa015bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:58570/0 (socket says 172.21.15.175:58570) 2023-10-10T19:13:39.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 -- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00aff60 msgr2=0x7f5fa015bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:39.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00aff60 0x7f5fa015bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 -- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fa00b1fc0 msgr2=0x7f5fa0165e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:39.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fa00b1fc0 0x7f5fa0165e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 -- 172.21.15.175:0/3423143077 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98009ae0 con 0x7f5fa00b2390 2023-10-10T19:13:39.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f9ed9d700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fa00b2390 0x7f5fa0166390 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009fd0 tx=0x7f5f9800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:39.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f96ffd700 1 -- 172.21.15.175:0/3423143077 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98018070 con 0x7f5fa00b2390 2023-10-10T19:13:39.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f96ffd700 1 -- 172.21.15.175:0/3423143077 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9800f040 con 0x7f5fa00b2390 2023-10-10T19:13:39.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f90009420 con 0x7f5fa00b2390 2023-10-10T19:13:39.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5f96ffd700 1 -- 172.21.15.175:0/3423143077 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9800cbd0 con 0x7f5fa00b2390 2023-10-10T19:13:39.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.205+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa01698a0 con 0x7f5fa00b2390 2023-10-10T19:13:39.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.206+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa0067570 con 0x7f5fa00b2390 2023-10-10T19:13:39.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.207+0000 7f5f96ffd700 1 -- 172.21.15.175:0/3423143077 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5f98003020 con 0x7f5fa00b2390 2023-10-10T19:13:39.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.207+0000 7f5f96ffd700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f80066f30 0x7f5f800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:39.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.207+0000 7f5f96ffd700 1 -- 172.21.15.175:0/3423143077 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f5f98084fd0 con 0x7f5fa00b2390 2023-10-10T19:13:39.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.207+0000 7f5f9e59c700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f80066f30 0x7f5f800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:39.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.208+0000 7f5f9e59c700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f80066f30 0x7f5f800693e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5f900098e0 tx=0x7f5f90017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:39.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.211+0000 7f5f96ffd700 1 -- 172.21.15.175:0/3423143077 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f98051c70 con 0x7f5fa00b2390 2023-10-10T19:13:39.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.437+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fa0068da0 con 0x7f5fa00b2390 2023-10-10T19:13:39.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.438+0000 7f5f96ffd700 1 -- 172.21.15.175:0/3423143077 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+6204 (secure 0 0 0) 0x7f5f98055290 con 0x7f5fa00b2390 2023-10-10T19:13:39.440 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:39.440 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7104,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":211}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:29.720314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:39.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f80066f30 msgr2=0x7f5f800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:39.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f80066f30 0x7f5f800693e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5f900098e0 tx=0x7f5f90017000 comp rx=0 tx=0).stop 2023-10-10T19:13:39.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fa00b2390 msgr2=0x7f5fa0166390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:39.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fa00b2390 0x7f5fa0166390 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009fd0 tx=0x7f5f9800b040 comp rx=0 tx=0).stop 2023-10-10T19:13:39.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 shutdown_connections 2023-10-10T19:13:39.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5f80066f30 0x7f5f800693e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5fa00b2390 0x7f5fa0166390 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5fa00aff60 0x7f5fa015bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 --2- 172.21.15.175:0/3423143077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fa00b1fc0 0x7f5fa0165e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:39.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 >> 172.21.15.175:0/3423143077 conn(0x7f5fa0063230 msgr2=0x7f5fa014fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:39.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 shutdown_connections 2023-10-10T19:13:39.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:39.440+0000 7f5fa4869700 1 -- 172.21.15.175:0/3423143077 wait complete. 2023-10-10T19:13:39.445 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 219 2023-10-10T19:13:39.453 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:13:41.455 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:41.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.580+0000 7f5a6a6d7700 1 Processor -- start 2023-10-10T19:13:41.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.580+0000 7f5a6a6d7700 1 -- start start 2023-10-10T19:13:41.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a6a6d7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5a641577a0 0x7f5a640c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:41.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a6a6d7700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a64154aa0 con 0x7f5a641573d0 2023-10-10T19:13:41.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a6a6d7700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a64063dc0 con 0x7f5a640c12a0 2023-10-10T19:13:41.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a6a6d7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a640bfdf0 con 0x7f5a641577a0 2023-10-10T19:13:41.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a63fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5a641573d0 0x7f5a640bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36040/0 (socket says 172.21.15.175:36040) 2023-10-10T19:13:41.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a63fff700 1 -- 172.21.15.175:0/4244542851 learned_addr learned my addr 172.21.15.175:0/4244542851 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:13:41.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a68c74700 1 --2- 172.21.15.175:0/4244542851 >> v2:172.21.15.191:3300/0 conn(0x7f5a641577a0 0x7f5a640c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:41.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a62ffd700 1 -- 172.21.15.175:0/4244542851 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101482358 0 0) 0x7f5a64154aa0 con 0x7f5a641573d0 2023-10-10T19:13:41.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a62ffd700 1 -- 172.21.15.175:0/4244542851 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a4c003570 con 0x7f5a641573d0 2023-10-10T19:13:41.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a62ffd700 1 -- 172.21.15.175:0/4244542851 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048206269 0 0) 0x7f5a4c003570 con 0x7f5a641573d0 2023-10-10T19:13:41.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a68c74700 1 -- 172.21.15.175:0/4244542851 >> v1:172.21.15.191:6789/0 conn(0x7f5a640c12a0 legacy=0x7f5a6414d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:41.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a68c74700 1 -- 172.21.15.175:0/4244542851 >> v1:172.21.15.175:6789/0 conn(0x7f5a641573d0 legacy=0x7f5a640bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:41.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.581+0000 7f5a68c74700 1 -- 172.21.15.175:0/4244542851 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a64157ba0 con 0x7f5a641577a0 2023-10-10T19:13:41.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a68c74700 1 --2- 172.21.15.175:0/4244542851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 0x7f5a640c4f20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009e60 tx=0x7f5a5c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a96b52816440d8e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:41.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a62ffd700 1 -- 172.21.15.175:0/4244542851 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c00e070 con 0x7f5a641577a0 2023-10-10T19:13:41.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a62ffd700 1 -- 172.21.15.175:0/4244542851 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5c00b850 con 0x7f5a641577a0 2023-10-10T19:13:41.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a62ffd700 1 -- 172.21.15.175:0/4244542851 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58001470 con 0x7f5a641577a0 2023-10-10T19:13:41.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/4244542851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 msgr2=0x7f5a640c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:41.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a6a6d7700 1 --2- 172.21.15.175:0/4244542851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 0x7f5a640c4f20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009e60 tx=0x7f5a5c009780 comp rx=0 tx=0).stop 2023-10-10T19:13:41.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/4244542851 shutdown_connections 2023-10-10T19:13:41.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a6a6d7700 1 --2- 172.21.15.175:0/4244542851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 0x7f5a640c4f20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:41.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/4244542851 >> 172.21.15.175:0/4244542851 conn(0x7f5a6406dc70 msgr2=0x7f5a64063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:41.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/4244542851 shutdown_connections 2023-10-10T19:13:41.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.582+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/4244542851 wait complete. 2023-10-10T19:13:41.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.583+0000 7f5a6a6d7700 1 Processor -- start 2023-10-10T19:13:41.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a6a6d7700 1 -- start start 2023-10-10T19:13:41.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a6a6d7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a640c12a0 0x7f5a6416a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:41.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a6a6d7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a641573d0 0x7f5a64152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:41.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a6a6d7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 0x7f5a640c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:41.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a6a6d7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a640c1ff0 con 0x7f5a640c12a0 2023-10-10T19:13:41.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a6a6d7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a64153b00 con 0x7f5a641577a0 2023-10-10T19:13:41.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a6a6d7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a64153de0 con 0x7f5a641573d0 2023-10-10T19:13:41.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a63fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a640c12a0 0x7f5a6416a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:41.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a63fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a640c12a0 0x7f5a6416a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60980/0 (socket says 172.21.15.175:60980) 2023-10-10T19:13:41.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a63fff700 1 -- 172.21.15.175:0/1026623091 learned_addr learned my addr 172.21.15.175:0/1026623091 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:41.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a68c74700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 0x7f5a640c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:41.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.584+0000 7f5a637fe700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a641573d0 0x7f5a64152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:41.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a63fff700 1 -- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a641573d0 msgr2=0x7f5a64152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:41.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a63fff700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a641573d0 0x7f5a64152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:41.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a63fff700 1 -- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 msgr2=0x7f5a640c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:41.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a63fff700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 0x7f5a640c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:41.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a63fff700 1 -- 172.21.15.175:0/1026623091 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c009a20 con 0x7f5a640c12a0 2023-10-10T19:13:41.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a637fe700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a641573d0 0x7f5a64152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:13:41.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a63fff700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a640c12a0 0x7f5a6416a410 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800cb90 tx=0x7f5a58001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:41.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a4bfff700 1 -- 172.21.15.175:0/1026623091 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58007e90 con 0x7f5a640c12a0 2023-10-10T19:13:41.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a4bfff700 1 -- 172.21.15.175:0/1026623091 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a58007360 con 0x7f5a640c12a0 2023-10-10T19:13:41.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a4bfff700 1 -- 172.21.15.175:0/1026623091 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58004830 con 0x7f5a640c12a0 2023-10-10T19:13:41.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a641540c0 con 0x7f5a640c12a0 2023-10-10T19:13:41.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.585+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a640c1c10 con 0x7f5a640c12a0 2023-10-10T19:13:41.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.586+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a64067570 con 0x7f5a640c12a0 2023-10-10T19:13:41.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.586+0000 7f5a4bfff700 1 -- 172.21.15.175:0/1026623091 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5a58004be0 con 0x7f5a640c12a0 2023-10-10T19:13:41.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.587+0000 7f5a4bfff700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a50066f30 0x7f5a500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:41.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.587+0000 7f5a4bfff700 1 -- 172.21.15.175:0/1026623091 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f5a58084070 con 0x7f5a640c12a0 2023-10-10T19:13:41.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.587+0000 7f5a637fe700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a50066f30 0x7f5a500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:41.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.587+0000 7f5a637fe700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a50066f30 0x7f5a500693e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5a6406b360 tx=0x7f5a54009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:41.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.591+0000 7f5a4bfff700 1 -- 172.21.15.175:0/1026623091 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a58050d90 con 0x7f5a640c12a0 2023-10-10T19:13:41.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.817+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a64157e50 con 0x7f5a640c12a0 2023-10-10T19:13:41.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.818+0000 7f5a4bfff700 1 -- 172.21.15.175:0/1026623091 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f5a58018020 con 0x7f5a640c12a0 2023-10-10T19:13:41.819 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:41.820 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7104,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":211}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:39.736736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:41.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a50066f30 msgr2=0x7f5a500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:41.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a50066f30 0x7f5a500693e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5a6406b360 tx=0x7f5a54009040 comp rx=0 tx=0).stop 2023-10-10T19:13:41.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a640c12a0 msgr2=0x7f5a6416a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:41.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a640c12a0 0x7f5a6416a410 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800cb90 tx=0x7f5a58001b50 comp rx=0 tx=0).stop 2023-10-10T19:13:41.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 shutdown_connections 2023-10-10T19:13:41.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5a50066f30 0x7f5a500693e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:41.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a641577a0 0x7f5a640c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:41.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a640c12a0 0x7f5a6416a410 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:41.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 --2- 172.21.15.175:0/1026623091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5a641573d0 0x7f5a64152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:41.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 >> 172.21.15.175:0/1026623091 conn(0x7f5a6406dc70 msgr2=0x7f5a64063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:41.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 shutdown_connections 2023-10-10T19:13:41.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:41.820+0000 7f5a6a6d7700 1 -- 172.21.15.175:0/1026623091 wait complete. 2023-10-10T19:13:41.824 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 220 2023-10-10T19:13:41.833 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:13:43.834 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:43.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.960+0000 7f0db8614700 1 Processor -- start 2023-10-10T19:13:43.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.960+0000 7f0db8614700 1 -- start start 2023-10-10T19:13:43.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db8614700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b1fc0 0x7f0db0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:43.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db8614700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0db00b2390 0x7f0db0068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:43.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db8614700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db01574b0 con 0x7f0db0068bd0 2023-10-10T19:13:43.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db8614700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db0150770 con 0x7f0db00b2390 2023-10-10T19:13:43.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db8614700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00afad0 con 0x7f0db00b1fc0 2023-10-10T19:13:43.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db5baf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0db00b2390 0x7f0db0068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:43.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db6bb1700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f0db0068bd0 0x7f0db00aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36044/0 (socket says 172.21.15.175:36044) 2023-10-10T19:13:43.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db6bb1700 1 -- 172.21.15.175:0/455363034 learned_addr learned my addr 172.21.15.175:0/455363034 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:13:43.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db63b0700 1 --2- 172.21.15.175:0/455363034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b1fc0 0x7f0db0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:43.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db5baf700 1 -- 172.21.15.175:0/455363034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b1fc0 msgr2=0x7f0db0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:43.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db5baf700 1 --2- 172.21.15.175:0/455363034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b1fc0 0x7f0db0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:43.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db5baf700 1 -- 172.21.15.175:0/455363034 >> v1:172.21.15.175:6789/0 conn(0x7f0db0068bd0 legacy=0x7f0db00aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:43.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db53ae700 1 -- 172.21.15.175:0/455363034 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114875938 0 0) 0x7f0db01574b0 con 0x7f0db0068bd0 2023-10-10T19:13:43.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db5baf700 1 -- 172.21.15.175:0/455363034 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db00af4c0 con 0x7f0db00b2390 2023-10-10T19:13:43.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db63b0700 1 --2- 172.21.15.175:0/455363034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b1fc0 0x7f0db0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:13:43.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db5baf700 1 --2- 172.21.15.175:0/455363034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db00b2390 0x7f0db0068690 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0da00098e0 tx=0x7f0da00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d6ba2dff5b2479 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:43.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db53ae700 1 -- 172.21.15.175:0/455363034 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0009e80 con 0x7f0db00b2390 2023-10-10T19:13:43.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.961+0000 7f0db53ae700 1 -- 172.21.15.175:0/455363034 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da000b8f0 con 0x7f0db00b2390 2023-10-10T19:13:43.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db53ae700 1 -- 172.21.15.175:0/455363034 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da000bc70 con 0x7f0db00b2390 2023-10-10T19:13:43.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db8614700 1 -- 172.21.15.175:0/455363034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db00b2390 msgr2=0x7f0db0068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:43.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db8614700 1 --2- 172.21.15.175:0/455363034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db00b2390 0x7f0db0068690 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0da00098e0 tx=0x7f0da00091a0 comp rx=0 tx=0).stop 2023-10-10T19:13:43.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db8614700 1 -- 172.21.15.175:0/455363034 shutdown_connections 2023-10-10T19:13:43.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db8614700 1 --2- 172.21.15.175:0/455363034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b1fc0 0x7f0db0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:43.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db8614700 1 --2- 172.21.15.175:0/455363034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db00b2390 0x7f0db0068690 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:43.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db8614700 1 -- 172.21.15.175:0/455363034 >> 172.21.15.175:0/455363034 conn(0x7f0db0063230 msgr2=0x7f0db0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:43.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db8614700 1 -- 172.21.15.175:0/455363034 shutdown_connections 2023-10-10T19:13:43.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.962+0000 7f0db8614700 1 -- 172.21.15.175:0/455363034 wait complete. 2023-10-10T19:13:43.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.963+0000 7f0db8614700 1 Processor -- start 2023-10-10T19:13:43.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.963+0000 7f0db8614700 1 -- start start 2023-10-10T19:13:43.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db8614700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db0068bd0 0x7f0db00c4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:43.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db8614700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0db00b1fc0 0x7f0db0155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:43.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db8614700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b2390 0x7f0db01589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:43.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db8614700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00c3870 con 0x7f0db0068bd0 2023-10-10T19:13:43.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db8614700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db0158520 con 0x7f0db00b1fc0 2023-10-10T19:13:43.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db8614700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db01691d0 con 0x7f0db00b2390 2023-10-10T19:13:43.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db0068bd0 0x7f0db00c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:43.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db0068bd0 0x7f0db00c4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60994/0 (socket says 172.21.15.175:60994) 2023-10-10T19:13:43.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 -- 172.21.15.175:0/999062338 learned_addr learned my addr 172.21.15.175:0/999062338 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:43.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db5baf700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0db00b1fc0 0x7f0db0155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:43.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db6bb1700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b2390 0x7f0db01589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:43.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 -- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b2390 msgr2=0x7f0db01589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:43.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b2390 0x7f0db01589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:43.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 -- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0db00b1fc0 msgr2=0x7f0db0155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:43.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0db00b1fc0 0x7f0db0155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:43.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 -- 172.21.15.175:0/999062338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da0009420 con 0x7f0db0068bd0 2023-10-10T19:13:43.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.964+0000 7f0db63b0700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db0068bd0 0x7f0db00c4470 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f0da800e900 tx=0x7f0da8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:43.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.965+0000 7f0da67fc700 1 -- 172.21.15.175:0/999062338 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da800eea0 con 0x7f0db0068bd0 2023-10-10T19:13:43.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.965+0000 7f0da67fc700 1 -- 172.21.15.175:0/999062338 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da800c5e0 con 0x7f0db0068bd0 2023-10-10T19:13:43.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.965+0000 7f0da67fc700 1 -- 172.21.15.175:0/999062338 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da800c980 con 0x7f0db0068bd0 2023-10-10T19:13:43.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.965+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db00c3ac0 con 0x7f0db0068bd0 2023-10-10T19:13:43.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.965+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db00ca920 con 0x7f0db0068bd0 2023-10-10T19:13:43.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.966+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db0067570 con 0x7f0db0068bd0 2023-10-10T19:13:43.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.966+0000 7f0da67fc700 1 -- 172.21.15.175:0/999062338 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0da8013410 con 0x7f0db0068bd0 2023-10-10T19:13:43.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.966+0000 7f0da67fc700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d94066f30 0x7f0d940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:43.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.967+0000 7f0da67fc700 1 -- 172.21.15.175:0/999062338 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f0da80847d0 con 0x7f0db0068bd0 2023-10-10T19:13:43.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.967+0000 7f0db5baf700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d94066f30 0x7f0d940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:43.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.967+0000 7f0db5baf700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d94066f30 0x7f0d940693e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0da00091e0 tx=0x7f0da0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:43.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:43.971+0000 7f0da67fc700 1 -- 172.21.15.175:0/999062338 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0da8051510 con 0x7f0db0068bd0 2023-10-10T19:13:44.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.198+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0db0158800 con 0x7f0db0068bd0 2023-10-10T19:13:44.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.198+0000 7f0da67fc700 1 -- 172.21.15.175:0/999062338 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f0da8054b30 con 0x7f0db0068bd0 2023-10-10T19:13:44.200 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:44.200 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7104,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":211}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:39.736736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:44.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d94066f30 msgr2=0x7f0d940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:44.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d94066f30 0x7f0d940693e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0da00091e0 tx=0x7f0da0017000 comp rx=0 tx=0).stop 2023-10-10T19:13:44.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db0068bd0 msgr2=0x7f0db00c4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:44.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db0068bd0 0x7f0db00c4470 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f0da800e900 tx=0x7f0da8008040 comp rx=0 tx=0).stop 2023-10-10T19:13:44.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 shutdown_connections 2023-10-10T19:13:44.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0d94066f30 0x7f0d940693e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:44.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0db00b2390 0x7f0db01589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:44.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0db0068bd0 0x7f0db00c4470 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:44.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 --2- 172.21.15.175:0/999062338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0db00b1fc0 0x7f0db0155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:44.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.200+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 >> 172.21.15.175:0/999062338 conn(0x7f0db0063230 msgr2=0x7f0db00c9170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:44.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.201+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 shutdown_connections 2023-10-10T19:13:44.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:44.201+0000 7f0db8614700 1 -- 172.21.15.175:0/999062338 wait complete. 2023-10-10T19:13:44.205 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 220 2023-10-10T19:13:44.215 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:13:46.216 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:46.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.341+0000 7f4d1a8af700 1 Processor -- start 2023-10-10T19:13:46.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.341+0000 7f4d1a8af700 1 -- start start 2023-10-10T19:13:46.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.341+0000 7f4d1a8af700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4d140b1fc0 0x7f4d14155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:46.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.341+0000 7f4d1a8af700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4d140b2390 0x7f4d14068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:46.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.341+0000 7f4d1a8af700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d141574b0 con 0x7f4d14068bd0 2023-10-10T19:13:46.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.341+0000 7f4d1a8af700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d14150770 con 0x7f4d140b1fc0 2023-10-10T19:13:46.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.341+0000 7f4d1a8af700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d140afad0 con 0x7f4d140b2390 2023-10-10T19:13:46.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d13fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4d140b1fc0 0x7f4d14155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:46.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d13fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4d140b1fc0 0x7f4d14155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50064/0 (socket says 172.21.15.175:50064) 2023-10-10T19:13:46.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d13fff700 1 -- 172.21.15.175:0/3667818038 learned_addr learned my addr 172.21.15.175:0/3667818038 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:46.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d137fe700 1 --2- 172.21.15.175:0/3667818038 >> v2:172.21.15.191:3300/0 conn(0x7f4d140b2390 0x7f4d14068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:46.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d13fff700 1 -- 172.21.15.175:0/3667818038 >> v2:172.21.15.191:3300/0 conn(0x7f4d140b2390 msgr2=0x7f4d14068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:46.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d13fff700 1 --2- 172.21.15.175:0/3667818038 >> v2:172.21.15.191:3300/0 conn(0x7f4d140b2390 0x7f4d14068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d13fff700 1 -- 172.21.15.175:0/3667818038 >> v1:172.21.15.191:6789/0 conn(0x7f4d14068bd0 legacy=0x7f4d140aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:46.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d13fff700 1 -- 172.21.15.175:0/3667818038 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d140af4c0 con 0x7f4d140b1fc0 2023-10-10T19:13:46.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d137fe700 1 --2- 172.21.15.175:0/3667818038 >> v2:172.21.15.191:3300/0 conn(0x7f4d140b2390 0x7f4d14068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:13:46.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d13fff700 1 --2- 172.21.15.175:0/3667818038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155fb0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4d140c0290 tx=0x7f4d04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5157f3cd855c592 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:46.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d12ffd700 1 -- 172.21.15.175:0/3667818038 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0400b8b0 con 0x7f4d140b1fc0 2023-10-10T19:13:46.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d12ffd700 1 -- 172.21.15.175:0/3667818038 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0400ba10 con 0x7f4d140b1fc0 2023-10-10T19:13:46.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.342+0000 7f4d12ffd700 1 -- 172.21.15.175:0/3667818038 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0400bdb0 con 0x7f4d140b1fc0 2023-10-10T19:13:46.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.343+0000 7f4d1a8af700 1 -- 172.21.15.175:0/3667818038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 msgr2=0x7f4d14155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:46.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.343+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/3667818038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155fb0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4d140c0290 tx=0x7f4d04009250 comp rx=0 tx=0).stop 2023-10-10T19:13:46.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.343+0000 7f4d1a8af700 1 -- 172.21.15.175:0/3667818038 shutdown_connections 2023-10-10T19:13:46.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.343+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/3667818038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155fb0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.343+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/3667818038 >> v2:172.21.15.191:3300/0 conn(0x7f4d140b2390 0x7f4d14068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.343+0000 7f4d1a8af700 1 -- 172.21.15.175:0/3667818038 >> 172.21.15.175:0/3667818038 conn(0x7f4d14063230 msgr2=0x7f4d14063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:46.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.343+0000 7f4d1a8af700 1 -- 172.21.15.175:0/3667818038 shutdown_connections 2023-10-10T19:13:46.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.343+0000 7f4d1a8af700 1 -- 172.21.15.175:0/3667818038 wait complete. 2023-10-10T19:13:46.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.344+0000 7f4d1a8af700 1 Processor -- start 2023-10-10T19:13:46.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.344+0000 7f4d1a8af700 1 -- start start 2023-10-10T19:13:46.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d1a8af700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d14068bd0 0x7f4d1415d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:46.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d1a8af700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:46.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d1a8af700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d140b2390 0x7f4d1414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:46.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d1a8af700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d14169970 con 0x7f4d140b1fc0 2023-10-10T19:13:46.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d1a8af700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d14169e00 con 0x7f4d14068bd0 2023-10-10T19:13:46.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d1a8af700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d14153dc0 con 0x7f4d140b2390 2023-10-10T19:13:46.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:46.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50076/0 (socket says 172.21.15.175:50076) 2023-10-10T19:13:46.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 -- 172.21.15.175:0/2242157794 learned_addr learned my addr 172.21.15.175:0/2242157794 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:46.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d13fff700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d14068bd0 0x7f4d1415d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:46.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d18e4c700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d140b2390 0x7f4d1414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:46.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 -- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d140b2390 msgr2=0x7f4d1414fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:46.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d140b2390 0x7f4d1414fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 -- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d14068bd0 msgr2=0x7f4d1415d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:46.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d14068bd0 0x7f4d1415d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 -- 172.21.15.175:0/2242157794 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d040094f0 con 0x7f4d140b1fc0 2023-10-10T19:13:46.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d13fff700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d14068bd0 0x7f4d1415d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:13:46.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d18e4c700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d140b2390 0x7f4d1414fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:13:46.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d137fe700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155750 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800c850 tx=0x7f4d08008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:46.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4cfbfff700 1 -- 172.21.15.175:0/2242157794 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08011600 con 0x7f4d140b1fc0 2023-10-10T19:13:46.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4cfbfff700 1 -- 172.21.15.175:0/2242157794 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d08011760 con 0x7f4d140b1fc0 2023-10-10T19:13:46.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d14154060 con 0x7f4d140b1fc0 2023-10-10T19:13:46.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4cfbfff700 1 -- 172.21.15.175:0/2242157794 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08011b00 con 0x7f4d140b1fc0 2023-10-10T19:13:46.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.345+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d14169320 con 0x7f4d140b1fc0 2023-10-10T19:13:46.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.346+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d14169b20 con 0x7f4d140b1fc0 2023-10-10T19:13:46.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.347+0000 7f4cfbfff700 1 -- 172.21.15.175:0/2242157794 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4d080073b0 con 0x7f4d140b1fc0 2023-10-10T19:13:46.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.347+0000 7f4cfbfff700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d00066f30 0x7f4d000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:46.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.347+0000 7f4cfbfff700 1 -- 172.21.15.175:0/2242157794 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7025+0+0 (secure 0 0 0) 0x7f4d08084a40 con 0x7f4d140b1fc0 2023-10-10T19:13:46.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.348+0000 7f4d13fff700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d00066f30 0x7f4d000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:46.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.348+0000 7f4d13fff700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d00066f30 0x7f4d000693e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4d140c0290 tx=0x7f4d04016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:46.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.352+0000 7f4cfbfff700 1 -- 172.21.15.175:0/2242157794 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d14169b20 con 0x7f4d140b1fc0 2023-10-10T19:13:46.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.574+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d14069860 con 0x7f4d140b1fc0 2023-10-10T19:13:46.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.574+0000 7f4cfbfff700 1 -- 172.21.15.175:0/2242157794 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+6202 (secure 0 0 0) 0x7f4d08086e60 con 0x7f4d140b1fc0 2023-10-10T19:13:46.576 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:46.576 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7104,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":211}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:39.736736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":6106,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6106":{"gid":6106,"name":"b","rank":2,"incarnation":167,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6836/62572703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":62572703},{"type":"v1","addr":"172.21.15.191:6836","nonce":62572703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d00066f30 msgr2=0x7f4d000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d00066f30 0x7f4d000693e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4d140c0290 tx=0x7f4d04016000 comp rx=0 tx=0).stop 2023-10-10T19:13:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 msgr2=0x7f4d14155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155750 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800c850 tx=0x7f4d08008040 comp rx=0 tx=0).stop 2023-10-10T19:13:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 shutdown_connections 2023-10-10T19:13:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4d00066f30 0x7f4d000693e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d140b2390 0x7f4d1414fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d14068bd0 0x7f4d1415d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 --2- 172.21.15.175:0/2242157794 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4d140b1fc0 0x7f4d14155750 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:46.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 >> 172.21.15.175:0/2242157794 conn(0x7f4d14063230 msgr2=0x7f4d14158000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:46.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 shutdown_connections 2023-10-10T19:13:46.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:46.576+0000 7f4d1a8af700 1 -- 172.21.15.175:0/2242157794 wait complete. 2023-10-10T19:13:46.581 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 220 2023-10-10T19:13:46.589 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:13:48.591 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:48.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.717+0000 7f9816ba8700 1 Processor -- start 2023-10-10T19:13:48.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.717+0000 7f9816ba8700 1 -- start start 2023-10-10T19:13:48.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f9816ba8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f981006cf90 0x7f981006d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:48.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f9816ba8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 0x7f9810156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:48.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f9816ba8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f98100bf2b0 0x7f98101567d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:48.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f9816ba8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98100c0990 con 0x7f98100bf2b0 2023-10-10T19:13:48.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f9816ba8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981006d8a0 con 0x7f981006cf90 2023-10-10T19:13:48.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f9816ba8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98101595e0 con 0x7f98100c0f70 2023-10-10T19:13:48.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f98100bf2b0 0x7f98101567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:48.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f98100bf2b0 0x7f98101567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50078/0 (socket says 172.21.15.175:50078) 2023-10-10T19:13:48.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 -- 172.21.15.175:0/3621129748 learned_addr learned my addr 172.21.15.175:0/3621129748 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:48.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f9815ba6700 1 --2- 172.21.15.175:0/3621129748 >> v2:172.21.15.191:3300/0 conn(0x7f981006cf90 0x7f981006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:48.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98153a5700 1 --2- 172.21.15.175:0/3621129748 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 0x7f9810156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:48.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 -- 172.21.15.175:0/3621129748 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 msgr2=0x7f9810156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:48.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 --2- 172.21.15.175:0/3621129748 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 0x7f9810156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 -- 172.21.15.175:0/3621129748 >> v2:172.21.15.191:3300/0 conn(0x7f981006cf90 msgr2=0x7f981006d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:48.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 --2- 172.21.15.175:0/3621129748 >> v2:172.21.15.191:3300/0 conn(0x7f981006cf90 0x7f981006d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 -- 172.21.15.175:0/3621129748 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981015a440 con 0x7f98100bf2b0 2023-10-10T19:13:48.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.718+0000 7f98163a7700 1 --2- 172.21.15.175:0/3621129748 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98100bf2b0 0x7f98101567d0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f981006cc20 tx=0x7f9808009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e06e31e144ce1d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:48.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9814ba4700 1 -- 172.21.15.175:0/3621129748 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980800e070 con 0x7f98100bf2b0 2023-10-10T19:13:48.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9814ba4700 1 -- 172.21.15.175:0/3621129748 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980800b970 con 0x7f98100bf2b0 2023-10-10T19:13:48.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9814ba4700 1 -- 172.21.15.175:0/3621129748 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98080092e0 con 0x7f98100bf2b0 2023-10-10T19:13:48.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 -- 172.21.15.175:0/3621129748 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98100bf2b0 msgr2=0x7f98101567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:48.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 --2- 172.21.15.175:0/3621129748 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98100bf2b0 0x7f98101567d0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f981006cc20 tx=0x7f9808009840 comp rx=0 tx=0).stop 2023-10-10T19:13:48.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 -- 172.21.15.175:0/3621129748 shutdown_connections 2023-10-10T19:13:48.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 --2- 172.21.15.175:0/3621129748 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98100bf2b0 0x7f98101567d0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 --2- 172.21.15.175:0/3621129748 >> v2:172.21.15.191:3300/0 conn(0x7f981006cf90 0x7f981006d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 --2- 172.21.15.175:0/3621129748 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 0x7f9810156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 -- 172.21.15.175:0/3621129748 >> 172.21.15.175:0/3621129748 conn(0x7f981006b310 msgr2=0x7f981006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:48.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 -- 172.21.15.175:0/3621129748 shutdown_connections 2023-10-10T19:13:48.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.719+0000 7f9816ba8700 1 -- 172.21.15.175:0/3621129748 wait complete. 2023-10-10T19:13:48.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.720+0000 7f9816ba8700 1 Processor -- start 2023-10-10T19:13:48.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.720+0000 7f9816ba8700 1 -- start start 2023-10-10T19:13:48.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9816ba8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f981006cf90 0x7f98100c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:48.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9816ba8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98100bf2b0 0x7f98100c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:48.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9816ba8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 0x7f9810069730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:48.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9816ba8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98101497d0 con 0x7f981006cf90 2023-10-10T19:13:48.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9816ba8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9810166ed0 con 0x7f98100bf2b0 2023-10-10T19:13:48.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9816ba8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9810068f70 con 0x7f98100c0f70 2023-10-10T19:13:48.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9815ba6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f981006cf90 0x7f98100c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:48.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9815ba6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f981006cf90 0x7f98100c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50080/0 (socket says 172.21.15.175:50080) 2023-10-10T19:13:48.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9815ba6700 1 -- 172.21.15.175:0/1365586864 learned_addr learned my addr 172.21.15.175:0/1365586864 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:48.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f98163a7700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 0x7f9810069730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:48.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f98153a5700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98100bf2b0 0x7f98100c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:48.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9815ba6700 1 -- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 msgr2=0x7f9810069730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:48.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9815ba6700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 0x7f9810069730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9815ba6700 1 -- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98100bf2b0 msgr2=0x7f98100c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:48.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9815ba6700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98100bf2b0 0x7f98100c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.721+0000 7f9815ba6700 1 -- 172.21.15.175:0/1365586864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9808009ae0 con 0x7f981006cf90 2023-10-10T19:13:48.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.722+0000 7f9815ba6700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f981006cf90 0x7f98100c9480 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f980c00ec80 tx=0x7f980c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:48.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.722+0000 7f9805ffb700 1 -- 172.21.15.175:0/1365586864 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00c850 con 0x7f981006cf90 2023-10-10T19:13:48.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.722+0000 7f9805ffb700 1 -- 172.21.15.175:0/1365586864 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c00c9b0 con 0x7f981006cf90 2023-10-10T19:13:48.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.722+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98101494f0 con 0x7f981006cf90 2023-10-10T19:13:48.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.722+0000 7f9805ffb700 1 -- 172.21.15.175:0/1365586864 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00cd30 con 0x7f981006cf90 2023-10-10T19:13:48.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.722+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9810069210 con 0x7f981006cf90 2023-10-10T19:13:48.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.723+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9810067570 con 0x7f981006cf90 2023-10-10T19:13:48.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.729+0000 7f9805ffb700 1 -- 172.21.15.175:0/1365586864 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f980c013680 con 0x7f981006cf90 2023-10-10T19:13:48.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.729+0000 7f9805ffb700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f97f4066f30 0x7f97f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:48.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.730+0000 7f9805ffb700 1 -- 172.21.15.175:0/1365586864 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f980c085490 con 0x7f981006cf90 2023-10-10T19:13:48.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.730+0000 7f98153a5700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f97f4066f30 0x7f97f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:48.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.730+0000 7f98153a5700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f97f4066f30 0x7f97f40693e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f981014f1a0 tx=0x7f9800009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:48.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.733+0000 7f9805ffb700 1 -- 172.21.15.175:0/1365586864 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f980c052200 con 0x7f981006cf90 2023-10-10T19:13:48.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.956+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f981006a9a0 con 0x7f981006cf90 2023-10-10T19:13:48.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.956+0000 7f9805ffb700 1 -- 172.21.15.175:0/1365586864 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+5447 (secure 0 0 0) 0x7f980c055820 con 0x7f981006cf90 2023-10-10T19:13:48.958 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:48.958 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:48.722822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:48.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.958+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f97f4066f30 msgr2=0x7f97f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:48.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.958+0000 7f9816ba8700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f97f4066f30 0x7f97f40693e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f981014f1a0 tx=0x7f9800009120 comp rx=0 tx=0).stop 2023-10-10T19:13:48.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.958+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f981006cf90 msgr2=0x7f98100c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:48.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f981006cf90 0x7f98100c9480 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f980c00ec80 tx=0x7f980c009120 comp rx=0 tx=0).stop 2023-10-10T19:13:48.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 shutdown_connections 2023-10-10T19:13:48.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f97f4066f30 0x7f97f40693e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98100c0f70 0x7f9810069730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f981006cf90 0x7f98100c9480 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 --2- 172.21.15.175:0/1365586864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98100bf2b0 0x7f98100c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:48.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 >> 172.21.15.175:0/1365586864 conn(0x7f981006b310 msgr2=0x7f98100c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:48.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 shutdown_connections 2023-10-10T19:13:48.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:48.959+0000 7f9816ba8700 1 -- 172.21.15.175:0/1365586864 wait complete. 2023-10-10T19:13:48.963 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 221 2023-10-10T19:13:48.972 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-10T19:13:48.972 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:13:48.973 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:49.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.086+0000 7f637d2f7700 1 Processor -- start 2023-10-10T19:13:49.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.086+0000 7f637d2f7700 1 -- start start 2023-10-10T19:13:49.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f637d2f7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63780b1fc0 0x7f6378155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:49.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f637d2f7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63780b2390 0x7f6378068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:49.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f637d2f7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6378068bd0 0x7f63780aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:49.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f637d2f7700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6378150770 con 0x7f63780b2390 2023-10-10T19:13:49.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f637d2f7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63781574b0 con 0x7f6378068bd0 2023-10-10T19:13:49.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f637d2f7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63780afad0 con 0x7f63780b1fc0 2023-10-10T19:13:49.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63780b2390 0x7f6378068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:49.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63780b2390 0x7f6378068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50096/0 (socket says 172.21.15.175:50096) 2023-10-10T19:13:49.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 -- 172.21.15.175:0/4098858747 learned_addr learned my addr 172.21.15.175:0/4098858747 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:49.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63777fe700 1 --2- 172.21.15.175:0/4098858747 >> v2:172.21.15.191:3300/0 conn(0x7f6378068bd0 0x7f63780aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:49.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f6376ffd700 1 --2- 172.21.15.175:0/4098858747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63780b1fc0 0x7f6378155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:49.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 -- 172.21.15.175:0/4098858747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63780b1fc0 msgr2=0x7f6378155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:49.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 --2- 172.21.15.175:0/4098858747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63780b1fc0 0x7f6378155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 -- 172.21.15.175:0/4098858747 >> v2:172.21.15.191:3300/0 conn(0x7f6378068bd0 msgr2=0x7f63780aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:49.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 --2- 172.21.15.175:0/4098858747 >> v2:172.21.15.191:3300/0 conn(0x7f6378068bd0 0x7f63780aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 -- 172.21.15.175:0/4098858747 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780af4c0 con 0x7f63780b2390 2023-10-10T19:13:49.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63777fe700 1 --2- 172.21.15.175:0/4098858747 >> v2:172.21.15.191:3300/0 conn(0x7f6378068bd0 0x7f63780aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:13:49.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f6376ffd700 1 --2- 172.21.15.175:0/4098858747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63780b1fc0 0x7f6378155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:13:49.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f63767fc700 1 --2- 172.21.15.175:0/4098858747 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378068690 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f63780afa40 tx=0x7f636c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b76e6424857a05 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:49.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f6375ffb700 1 -- 172.21.15.175:0/4098858747 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c00b8f0 con 0x7f63780b2390 2023-10-10T19:13:49.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f6375ffb700 1 -- 172.21.15.175:0/4098858747 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c00ba50 con 0x7f63780b2390 2023-10-10T19:13:49.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.087+0000 7f6375ffb700 1 -- 172.21.15.175:0/4098858747 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c00bdf0 con 0x7f63780b2390 2023-10-10T19:13:49.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 -- 172.21.15.175:0/4098858747 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 msgr2=0x7f6378068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:49.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4098858747 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378068690 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f63780afa40 tx=0x7f636c009250 comp rx=0 tx=0).stop 2023-10-10T19:13:49.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 -- 172.21.15.175:0/4098858747 shutdown_connections 2023-10-10T19:13:49.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4098858747 >> v2:172.21.15.191:3300/0 conn(0x7f6378068bd0 0x7f63780aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4098858747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63780b1fc0 0x7f6378155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4098858747 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378068690 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 -- 172.21.15.175:0/4098858747 >> 172.21.15.175:0/4098858747 conn(0x7f6378063230 msgr2=0x7f6378063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:49.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 -- 172.21.15.175:0/4098858747 shutdown_connections 2023-10-10T19:13:49.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.088+0000 7f637d2f7700 1 -- 172.21.15.175:0/4098858747 wait complete. 2023-10-10T19:13:49.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.089+0000 7f637d2f7700 1 Processor -- start 2023-10-10T19:13:49.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.089+0000 7f637d2f7700 1 -- start start 2023-10-10T19:13:49.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f637d2f7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6378068bd0 0x7f63780c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:49.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f637d2f7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63780b1fc0 0x7f63780c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:49.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f637d2f7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378169460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:49.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f637d2f7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63780c4870 con 0x7f63780b2390 2023-10-10T19:13:49.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f637d2f7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6378156fc0 con 0x7f63780b1fc0 2023-10-10T19:13:49.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f637d2f7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6378150e10 con 0x7f6378068bd0 2023-10-10T19:13:49.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63777fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378169460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:49.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63777fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378169460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50104/0 (socket says 172.21.15.175:50104) 2023-10-10T19:13:49.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63777fe700 1 -- 172.21.15.175:0/4014995408 learned_addr learned my addr 172.21.15.175:0/4014995408 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:49.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63767fc700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63780b1fc0 0x7f63780c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:49.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f6376ffd700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6378068bd0 0x7f63780c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:49.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63777fe700 1 -- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6378068bd0 msgr2=0x7f63780c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:49.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63777fe700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6378068bd0 0x7f63780c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63777fe700 1 -- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63780b1fc0 msgr2=0x7f63780c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:49.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63777fe700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63780b1fc0 0x7f63780c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f63777fe700 1 -- 172.21.15.175:0/4014995408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0094f0 con 0x7f63780b2390 2023-10-10T19:13:49.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.090+0000 7f6376ffd700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6378068bd0 0x7f63780c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:13:49.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.091+0000 7f63777fe700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378169460 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f636800cf30 tx=0x7f6368007840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:49.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.091+0000 7f635effd700 1 -- 172.21.15.175:0/4014995408 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6368015440 con 0x7f63780b2390 2023-10-10T19:13:49.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.091+0000 7f635effd700 1 -- 172.21.15.175:0/4014995408 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63680155a0 con 0x7f63780b2390 2023-10-10T19:13:49.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.091+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6378154290 con 0x7f63780b2390 2023-10-10T19:13:49.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.091+0000 7f635effd700 1 -- 172.21.15.175:0/4014995408 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6368015960 con 0x7f63780b2390 2023-10-10T19:13:49.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.091+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63780c5930 con 0x7f63780b2390 2023-10-10T19:13:49.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.092+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6378067570 con 0x7f63780b2390 2023-10-10T19:13:49.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.092+0000 7f635effd700 1 -- 172.21.15.175:0/4014995408 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f636800d440 con 0x7f63780b2390 2023-10-10T19:13:49.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.092+0000 7f635effd700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6364066f30 0x7f63640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:49.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.092+0000 7f635effd700 1 -- 172.21.15.175:0/4014995408 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f63680856c0 con 0x7f63780b2390 2023-10-10T19:13:49.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.093+0000 7f6376ffd700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6364066f30 0x7f63640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:49.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.093+0000 7f6376ffd700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6364066f30 0x7f63640693e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6370008b20 tx=0x7f6370008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:49.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.097+0000 7f635effd700 1 -- 172.21.15.175:0/4014995408 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63680523b0 con 0x7f63780b2390 2023-10-10T19:13:49.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.320+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6378068470 con 0x7f63780b2390 2023-10-10T19:13:49.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.321+0000 7f635effd700 1 -- 172.21.15.175:0/4014995408 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+5447 (secure 0 0 0) 0x7f63680559d0 con 0x7f63780b2390 2023-10-10T19:13:49.323 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:49.323 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:48.722822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:49.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6364066f30 msgr2=0x7f63640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:49.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6364066f30 0x7f63640693e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6370008b20 tx=0x7f6370008040 comp rx=0 tx=0).stop 2023-10-10T19:13:49.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 msgr2=0x7f6378169460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:49.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378169460 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f636800cf30 tx=0x7f6368007840 comp rx=0 tx=0).stop 2023-10-10T19:13:49.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 shutdown_connections 2023-10-10T19:13:49.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6364066f30 0x7f63640693e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63780b2390 0x7f6378169460 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6378068bd0 0x7f63780c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 --2- 172.21.15.175:0/4014995408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63780b1fc0 0x7f63780c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:49.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 >> 172.21.15.175:0/4014995408 conn(0x7f6378063230 msgr2=0x7f63781517c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:49.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 shutdown_connections 2023-10-10T19:13:49.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:49.323+0000 7f637d2f7700 1 -- 172.21.15.175:0/4014995408 wait complete. 2023-10-10T19:13:49.331 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 221 2023-10-10T19:13:49.337 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 221, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 221, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:13:48.722822+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 6117, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:13:51.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:51.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.463+0000 7fc75c03e700 1 Processor -- start 2023-10-10T19:13:51.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75c03e700 1 -- start start 2023-10-10T19:13:51.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75c03e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 0x7fc75406b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:51.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75c03e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc75414ddb0 0x7fc75414e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:51.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75c03e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc754154e20 con 0x7fc7540c1be0 2023-10-10T19:13:51.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75c03e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75406bca0 con 0x7fc75414ddb0 2023-10-10T19:13:51.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75c03e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc754154000 con 0x7fc7540c1fb0 2023-10-10T19:13:51.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75a5db700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc75414ddb0 0x7fc75414e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:51.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75a5db700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc75414ddb0 0x7fc75414e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50112/0 (socket says 172.21.15.175:50112) 2023-10-10T19:13:51.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.464+0000 7fc75a5db700 1 -- 172.21.15.175:0/2143753964 learned_addr learned my addr 172.21.15.175:0/2143753964 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:51.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.465+0000 7fc7595d9700 1 --2- 172.21.15.175:0/2143753964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 0x7fc75406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:51.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.465+0000 7fc75a5db700 1 -- 172.21.15.175:0/2143753964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 msgr2=0x7fc75406b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:51.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.465+0000 7fc75a5db700 1 --2- 172.21.15.175:0/2143753964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 0x7fc75406b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.465+0000 7fc75a5db700 1 -- 172.21.15.175:0/2143753964 >> v1:172.21.15.191:6789/0 conn(0x7fc7540c1be0 legacy=0x7fc75406a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:51.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.465+0000 7fc75a5db700 1 -- 172.21.15.175:0/2143753964 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc754154210 con 0x7fc75414ddb0 2023-10-10T19:13:51.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.465+0000 7fc75a5db700 1 --2- 172.21.15.175:0/2143753964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc75414ddb0 0x7fc75414e250 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009f20 tx=0x7fc74c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594c9ff28600b5cb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:51.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.466+0000 7fc758dd8700 1 -- 172.21.15.175:0/2143753964 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c00e070 con 0x7fc75414ddb0 2023-10-10T19:13:51.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.466+0000 7fc758dd8700 1 -- 172.21.15.175:0/2143753964 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c00ba80 con 0x7fc75414ddb0 2023-10-10T19:13:51.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.466+0000 7fc758dd8700 1 -- 172.21.15.175:0/2143753964 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0092e0 con 0x7fc75414ddb0 2023-10-10T19:13:51.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.467+0000 7fc75c03e700 1 -- 172.21.15.175:0/2143753964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc75414ddb0 msgr2=0x7fc75414e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:51.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.467+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2143753964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc75414ddb0 0x7fc75414e250 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009f20 tx=0x7fc74c009840 comp rx=0 tx=0).stop 2023-10-10T19:13:51.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.467+0000 7fc75c03e700 1 -- 172.21.15.175:0/2143753964 shutdown_connections 2023-10-10T19:13:51.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.467+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2143753964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc75414ddb0 0x7fc75414e250 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.467+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2143753964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 0x7fc75406b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.467+0000 7fc75c03e700 1 -- 172.21.15.175:0/2143753964 >> 172.21.15.175:0/2143753964 conn(0x7fc754068280 msgr2=0x7fc75406af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:51.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.467+0000 7fc75c03e700 1 -- 172.21.15.175:0/2143753964 shutdown_connections 2023-10-10T19:13:51.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.467+0000 7fc75c03e700 1 -- 172.21.15.175:0/2143753964 wait complete. 2023-10-10T19:13:51.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.468+0000 7fc75c03e700 1 Processor -- start 2023-10-10T19:13:51.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.468+0000 7fc75c03e700 1 -- start start 2023-10-10T19:13:51.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75c03e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7540c1be0 0x7fc7541694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:51.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75c03e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 0x7fc75414c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:51.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75c03e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc75414ddb0 0x7fc7540c41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:51.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75c03e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc754169a10 con 0x7fc7540c1be0 2023-10-10T19:13:51.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75c03e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc754148610 con 0x7fc75414ddb0 2023-10-10T19:13:51.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75c03e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7541596c0 con 0x7fc7540c1fb0 2023-10-10T19:13:51.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7540c1be0 0x7fc7541694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:51.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7540c1be0 0x7fc7541694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50124/0 (socket says 172.21.15.175:50124) 2023-10-10T19:13:51.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 -- 172.21.15.175:0/2927967515 learned_addr learned my addr 172.21.15.175:0/2927967515 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:51.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75a5db700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc75414ddb0 0x7fc7540c41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:51.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc7595d9700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 0x7fc75414c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:51.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 -- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 msgr2=0x7fc75414c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:51.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 0x7fc75414c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 -- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc75414ddb0 msgr2=0x7fc7540c41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:51.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc75414ddb0 0x7fc7540c41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 -- 172.21.15.175:0/2927967515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c009ae0 con 0x7fc7540c1be0 2023-10-10T19:13:51.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75a5db700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc75414ddb0 0x7fc7540c41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:13:51.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc759dda700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7540c1be0 0x7fc7541694d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c5e0 tx=0x7fc750007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:51.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc749ffb700 1 -- 172.21.15.175:0/2927967515 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750012630 con 0x7fc7540c1be0 2023-10-10T19:13:51.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc749ffb700 1 -- 172.21.15.175:0/2927967515 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750012790 con 0x7fc7540c1be0 2023-10-10T19:13:51.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc754147fe0 con 0x7fc7540c1be0 2023-10-10T19:13:51.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.469+0000 7fc749ffb700 1 -- 172.21.15.175:0/2927967515 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750012b90 con 0x7fc7540c1be0 2023-10-10T19:13:51.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.470+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75415a440 con 0x7fc7540c1be0 2023-10-10T19:13:51.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.471+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc754067570 con 0x7fc7540c1be0 2023-10-10T19:13:51.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.471+0000 7fc749ffb700 1 -- 172.21.15.175:0/2927967515 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc75001b400 con 0x7fc7540c1be0 2023-10-10T19:13:51.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.471+0000 7fc749ffb700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc738066f30 0x7fc7380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:51.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.471+0000 7fc749ffb700 1 -- 172.21.15.175:0/2927967515 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7fc7500845b0 con 0x7fc7540c1be0 2023-10-10T19:13:51.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.472+0000 7fc7595d9700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc738066f30 0x7fc7380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:51.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.473+0000 7fc7595d9700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc738066f30 0x7fc7380693e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc754158b60 tx=0x7fc744008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:51.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.476+0000 7fc749ffb700 1 -- 172.21.15.175:0/2927967515 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7500512a0 con 0x7fc7540c1be0 2023-10-10T19:13:51.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.701+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7540c5c00 con 0x7fc7540c1be0 2023-10-10T19:13:51.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.701+0000 7fc749ffb700 1 -- 172.21.15.175:0/2927967515 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+5447 (secure 0 0 0) 0x7fc7500548c0 con 0x7fc7540c1be0 2023-10-10T19:13:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:51.703 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:48.722822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:51.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc738066f30 msgr2=0x7fc7380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:51.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc738066f30 0x7fc7380693e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc754158b60 tx=0x7fc744008040 comp rx=0 tx=0).stop 2023-10-10T19:13:51.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7540c1be0 msgr2=0x7fc7541694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:51.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7540c1be0 0x7fc7541694d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c5e0 tx=0x7fc750007f30 comp rx=0 tx=0).stop 2023-10-10T19:13:51.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 shutdown_connections 2023-10-10T19:13:51.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc738066f30 0x7fc7380693e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc75414ddb0 0x7fc7540c41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc7540c1be0 0x7fc7541694d0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 --2- 172.21.15.175:0/2927967515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c1fb0 0x7fc75414c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:51.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 >> 172.21.15.175:0/2927967515 conn(0x7fc754068280 msgr2=0x7fc754153290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:51.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 shutdown_connections 2023-10-10T19:13:51.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:51.703+0000 7fc75c03e700 1 -- 172.21.15.175:0/2927967515 wait complete. 2023-10-10T19:13:51.707 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 221 2023-10-10T19:13:51.716 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:13:53.225 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:13:53.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:13:53.718 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:53.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.849+0000 7f0846d55700 1 Processor -- start 2023-10-10T19:13:53.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.850+0000 7f0846d55700 1 -- start start 2023-10-10T19:13:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.850+0000 7f0846d55700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f084014d190 0x7f0840152d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.850+0000 7f0846d55700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0840153240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.850+0000 7f0846d55700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084006e940 con 0x7f084014cdc0 2023-10-10T19:13:53.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.850+0000 7f0846d55700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08400c1c90 con 0x7f084014d190 2023-10-10T19:13:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.850+0000 7f0846d55700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08400c6e20 con 0x7f08400c2df0 2023-10-10T19:13:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.850+0000 7f0844af1700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f084014cdc0 0x7f08400c26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58232/0 (socket says 172.21.15.175:58232) 2023-10-10T19:13:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.850+0000 7f0844af1700 1 -- 172.21.15.175:0/1995137392 learned_addr learned my addr 172.21.15.175:0/1995137392 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:13:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083ffff700 1 --2- 172.21.15.175:0/1995137392 >> v2:172.21.15.191:3300/0 conn(0x7f084014d190 0x7f0840152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f08452f2700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0840153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305121718 0 0) 0x7f084006e940 con 0x7f084014cdc0 2023-10-10T19:13:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0830003570 con 0x7f084014cdc0 2023-10-10T19:13:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2583857247 0 0) 0x7f0830003570 con 0x7f084014cdc0 2023-10-10T19:13:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 msgr2=0x7f0840153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:53.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0840153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 >> v2:172.21.15.191:3300/0 conn(0x7f084014d190 msgr2=0x7f0840152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 --2- 172.21.15.175:0/1995137392 >> v2:172.21.15.191:3300/0 conn(0x7f084014d190 0x7f0840152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084006ca40 con 0x7f084014cdc0 2023-10-10T19:13:53.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f08340015f0 con 0x7f084014cdc0 2023-10-10T19:13:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 >> v1:172.21.15.175:6789/0 conn(0x7f084014cdc0 legacy=0x7f08400c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0830003a60 0x7f0830003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0830004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0830004d50 0x7f08300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400c1c90 con 0x7f0830004d50 2023-10-10T19:13:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400c6e20 con 0x7f0830003a60 2023-10-10T19:13:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083000dc70 con 0x7f08400c2df0 2023-10-10T19:13:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f0844af1700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0830003a60 0x7f0830003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f083ffff700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0830004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.851+0000 7f08452f2700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0830004d50 0x7f08300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083ffff700 1 -- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0830003a60 msgr2=0x7f0830003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:53.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083ffff700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0830003a60 0x7f0830003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:53.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083ffff700 1 -- 172.21.15.175:0/1995137392 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0830004d50 msgr2=0x7f08300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:53.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083ffff700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0830004d50 0x7f08300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:53.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083ffff700 1 -- 172.21.15.175:0/1995137392 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0830003570 con 0x7f08400c2df0 2023-10-10T19:13:53.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f0844af1700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0830003a60 0x7f0830003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:13:53.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083ffff700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0830004810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f084014d5d0 tx=0x7f082c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f24e9fe8c6923f09 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:53.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00e6b0 con 0x7f08400c2df0 2023-10-10T19:13:53.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c00e810 con 0x7f08400c2df0 2023-10-10T19:13:53.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f083f7fe700 1 -- 172.21.15.175:0/1995137392 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00ebd0 con 0x7f08400c2df0 2023-10-10T19:13:53.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f0846d55700 1 -- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 msgr2=0x7f0830004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:53.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f0846d55700 1 --2- 172.21.15.175:0/1995137392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0830004810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f084014d5d0 tx=0x7f082c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:13:53.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f08452f2700 1 -- 172.21.15.175:0/1995137392 reap_dead start 2023-10-10T19:13:53.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f0846d55700 1 -- 172.21.15.175:0/1995137392 shutdown_connections 2023-10-10T19:13:53.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f0846d55700 1 -- 172.21.15.175:0/1995137392 >> 172.21.15.175:0/1995137392 conn(0x7f0840068900 msgr2=0x7f0840068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:53.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f0846d55700 1 -- 172.21.15.175:0/1995137392 shutdown_connections 2023-10-10T19:13:53.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.852+0000 7f0846d55700 1 -- 172.21.15.175:0/1995137392 wait complete. 2023-10-10T19:13:53.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.853+0000 7f0846d55700 1 Processor -- start 2023-10-10T19:13:53.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f0846d55700 1 -- start start 2023-10-10T19:13:53.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f0846d55700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0840162120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f0846d55700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084014fd70 0x7f0840148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f0846d55700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08401501c0 0x7f0840148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f0846d55700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0840147bb0 con 0x7f084014fd70 2023-10-10T19:13:53.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f0846d55700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084014d600 con 0x7f08401501c0 2023-10-10T19:13:53.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f0846d55700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084006f610 con 0x7f08400c2df0 2023-10-10T19:13:53.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f083ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084014fd70 0x7f0840148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f083ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084014fd70 0x7f0840148040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50142/0 (socket says 172.21.15.175:50142) 2023-10-10T19:13:53.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.854+0000 7f083ffff700 1 -- 172.21.15.175:0/2946103567 learned_addr learned my addr 172.21.15.175:0/2946103567 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:53.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f0844af1700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0840162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f08452f2700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08401501c0 0x7f0840148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083ffff700 1 -- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 msgr2=0x7f0840162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:53.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083ffff700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0840162120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:53.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083ffff700 1 -- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08401501c0 msgr2=0x7f0840148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:53.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083ffff700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08401501c0 0x7f0840148580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:53.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083ffff700 1 -- 172.21.15.175:0/2946103567 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084014d8c0 con 0x7f084014fd70 2023-10-10T19:13:53.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083ffff700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084014fd70 0x7f0840148040 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f083000de00 tx=0x7f082c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:53.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083cff9700 1 -- 172.21.15.175:0/2946103567 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c022070 con 0x7f084014fd70 2023-10-10T19:13:53.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083cff9700 1 -- 172.21.15.175:0/2946103567 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c00e980 con 0x7f084014fd70 2023-10-10T19:13:53.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084014dba0 con 0x7f084014fd70 2023-10-10T19:13:53.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f083cff9700 1 -- 172.21.15.175:0/2946103567 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00e6b0 con 0x7f084014fd70 2023-10-10T19:13:53.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.855+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084014e160 con 0x7f084014fd70 2023-10-10T19:13:53.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.856+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0840147d60 con 0x7f084014fd70 2023-10-10T19:13:53.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.857+0000 7f083cff9700 1 -- 172.21.15.175:0/2946103567 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f082c027680 con 0x7f084014fd70 2023-10-10T19:13:53.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.857+0000 7f083cff9700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0824066f30 0x7f08240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:53.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.857+0000 7f083cff9700 1 -- 172.21.15.175:0/2946103567 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f082c086430 con 0x7f084014fd70 2023-10-10T19:13:53.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.857+0000 7f0844af1700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0824066f30 0x7f08240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:53.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.858+0000 7f0844af1700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0824066f30 0x7f08240693e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0834001760 tx=0x7f0834009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:53.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:53.861+0000 7f083cff9700 1 -- 172.21.15.175:0/2946103567 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0840147d60 con 0x7f084014fd70 2023-10-10T19:13:54.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.099+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0840147d60 con 0x7f084014fd70 2023-10-10T19:13:54.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.100+0000 7f083cff9700 1 -- 172.21.15.175:0/2946103567 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+5447 (secure 0 0 0) 0x7f0840147d60 con 0x7f084014fd70 2023-10-10T19:13:54.101 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:54.102 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:48.722822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:54.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0824066f30 msgr2=0x7f08240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:54.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0824066f30 0x7f08240693e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0834001760 tx=0x7f0834009040 comp rx=0 tx=0).stop 2023-10-10T19:13:54.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084014fd70 msgr2=0x7f0840148040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:54.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084014fd70 0x7f0840148040 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f083000de00 tx=0x7f082c01a000 comp rx=0 tx=0).stop 2023-10-10T19:13:54.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 shutdown_connections 2023-10-10T19:13:54.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0824066f30 0x7f08240693e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:54.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08401501c0 0x7f0840148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:54.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08400c2df0 0x7f0840162120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:54.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 --2- 172.21.15.175:0/2946103567 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f084014fd70 0x7f0840148040 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:54.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 >> 172.21.15.175:0/2946103567 conn(0x7f0840068900 msgr2=0x7f084006ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:54.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 shutdown_connections 2023-10-10T19:13:54.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:54.103+0000 7f0846d55700 1 -- 172.21.15.175:0/2946103567 wait complete. 2023-10-10T19:13:54.107 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 221 2023-10-10T19:13:54.117 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:13:56.117 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:56.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.242+0000 7f984f1c5700 1 Processor -- start 2023-10-10T19:13:56.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.242+0000 7f984f1c5700 1 -- start start 2023-10-10T19:13:56.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984f1c5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9848155fb0 0x7f9848159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984f1c5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98481574b0 con 0x7f98480b2390 2023-10-10T19:13:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984f1c5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98480afad0 con 0x7f98480b1fc0 2023-10-10T19:13:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984f1c5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9848150770 con 0x7f9848155fb0 2023-10-10T19:13:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984d762700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9848155fb0 0x7f9848159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:56.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f9847fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f98480b2390 0x7f98480aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56540/0 (socket says 172.21.15.175:56540) 2023-10-10T19:13:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f9847fff700 1 -- 172.21.15.175:0/1355261819 learned_addr learned my addr 172.21.15.175:0/1355261819 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:13:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984d762700 1 -- 172.21.15.175:0/1355261819 >> v1:172.21.15.191:6789/0 conn(0x7f98480b1fc0 legacy=0x7f9848068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984d762700 1 -- 172.21.15.175:0/1355261819 >> v1:172.21.15.175:6789/0 conn(0x7f98480b2390 legacy=0x7f98480aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984d762700 1 -- 172.21.15.175:0/1355261819 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98480af4c0 con 0x7f9848155fb0 2023-10-10T19:13:56.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f98477fe700 1 -- 172.21.15.175:0/1355261819 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227617026 0 0) 0x7f98481574b0 con 0x7f98480b2390 2023-10-10T19:13:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f984d762700 1 --2- 172.21.15.175:0/1355261819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f9848159920 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9840009e60 tx=0x7f9840009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dd54b99455c233 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f98477fe700 1 -- 172.21.15.175:0/1355261819 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984000e070 con 0x7f9848155fb0 2023-10-10T19:13:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f98477fe700 1 -- 172.21.15.175:0/1355261819 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984000bcc0 con 0x7f9848155fb0 2023-10-10T19:13:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.243+0000 7f98477fe700 1 -- 172.21.15.175:0/1355261819 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840012680 con 0x7f9848155fb0 2023-10-10T19:13:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.244+0000 7f984f1c5700 1 -- 172.21.15.175:0/1355261819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 msgr2=0x7f9848159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.244+0000 7f984f1c5700 1 --2- 172.21.15.175:0/1355261819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f9848159920 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9840009e60 tx=0x7f9840009780 comp rx=0 tx=0).stop 2023-10-10T19:13:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.244+0000 7f984f1c5700 1 -- 172.21.15.175:0/1355261819 shutdown_connections 2023-10-10T19:13:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.244+0000 7f984f1c5700 1 --2- 172.21.15.175:0/1355261819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f9848159920 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.244+0000 7f984f1c5700 1 -- 172.21.15.175:0/1355261819 >> 172.21.15.175:0/1355261819 conn(0x7f9848063230 msgr2=0x7f9848063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:56.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.244+0000 7f984f1c5700 1 -- 172.21.15.175:0/1355261819 shutdown_connections 2023-10-10T19:13:56.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.244+0000 7f984f1c5700 1 -- 172.21.15.175:0/1355261819 wait complete. 2023-10-10T19:13:56.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.245+0000 7f984f1c5700 1 Processor -- start 2023-10-10T19:13:56.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.245+0000 7f984f1c5700 1 -- start start 2023-10-10T19:13:56.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984f1c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98480b1fc0 0x7f98480c1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:56.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984f1c5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98480b2390 0x7f98480c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:56.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984f1c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f98480c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984f1c5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98480c0c80 con 0x7f9848155fb0 2023-10-10T19:13:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984f1c5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98480ccf50 con 0x7f98480b2390 2023-10-10T19:13:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984f1c5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98480c3b30 con 0x7f98480b1fc0 2023-10-10T19:13:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f98480c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:56.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f98480c53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34242/0 (socket says 172.21.15.175:34242) 2023-10-10T19:13:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 -- 172.21.15.175:0/3608888363 learned_addr learned my addr 172.21.15.175:0/3608888363 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f9847fff700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98480b2390 0x7f98480c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984cf61700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98480b1fc0 0x7f98480c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 -- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98480b1fc0 msgr2=0x7f98480c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:56.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98480b1fc0 0x7f98480c1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 -- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98480b2390 msgr2=0x7f98480c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98480b2390 0x7f98480c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 -- 172.21.15.175:0/3608888363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9840009a20 con 0x7f9848155fb0 2023-10-10T19:13:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.246+0000 7f984d762700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f98480c53a0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f984000b6c0 tx=0x7f9840019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:56.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.247+0000 7f9844ff9700 1 -- 172.21.15.175:0/3608888363 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984000beb0 con 0x7f9848155fb0 2023-10-10T19:13:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.247+0000 7f9844ff9700 1 -- 172.21.15.175:0/3608888363 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984001b630 con 0x7f9848155fb0 2023-10-10T19:13:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.247+0000 7f9844ff9700 1 -- 172.21.15.175:0/3608888363 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984001ba10 con 0x7f9848155fb0 2023-10-10T19:13:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.247+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98480c0f00 con 0x7f9848155fb0 2023-10-10T19:13:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.247+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98480cc6b0 con 0x7f9848155fb0 2023-10-10T19:13:56.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.248+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98480c1090 con 0x7f9848155fb0 2023-10-10T19:13:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.248+0000 7f9844ff9700 1 -- 172.21.15.175:0/3608888363 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f984001c400 con 0x7f9848155fb0 2023-10-10T19:13:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.248+0000 7f9844ff9700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f982c066f30 0x7f982c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.249+0000 7f9844ff9700 1 -- 172.21.15.175:0/3608888363 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f9840086ae0 con 0x7f9848155fb0 2023-10-10T19:13:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.249+0000 7f984cf61700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f982c066f30 0x7f982c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.249+0000 7f984cf61700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f982c066f30 0x7f982c0693e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f983c003a80 tx=0x7f983c007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:56.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.253+0000 7f9844ff9700 1 -- 172.21.15.175:0/3608888363 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f98400537d0 con 0x7f9848155fb0 2023-10-10T19:13:56.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.476+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9848068da0 con 0x7f9848155fb0 2023-10-10T19:13:56.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.476+0000 7f9844ff9700 1 -- 172.21.15.175:0/3608888363 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+5449 (secure 0 0 0) 0x7f984002a090 con 0x7f9848155fb0 2023-10-10T19:13:56.477 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:56.477 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:54.771130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:resolve","state_seq":37,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:56.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.478+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f982c066f30 msgr2=0x7f982c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:56.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.478+0000 7f984f1c5700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f982c066f30 0x7f982c0693e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f983c003a80 tx=0x7f983c007ec0 comp rx=0 tx=0).stop 2023-10-10T19:13:56.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.478+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 msgr2=0x7f98480c53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:56.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.478+0000 7f984f1c5700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f98480c53a0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f984000b6c0 tx=0x7f9840019000 comp rx=0 tx=0).stop 2023-10-10T19:13:56.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.479+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 shutdown_connections 2023-10-10T19:13:56.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.479+0000 7f984f1c5700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f982c066f30 0x7f982c0693e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:56.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.479+0000 7f984f1c5700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9848155fb0 0x7f98480c53a0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:56.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.479+0000 7f984f1c5700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98480b1fc0 0x7f98480c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:56.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.479+0000 7f984f1c5700 1 --2- 172.21.15.175:0/3608888363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98480b2390 0x7f98480c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:56.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.479+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 >> 172.21.15.175:0/3608888363 conn(0x7f9848063230 msgr2=0x7f98481508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:56.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.479+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 shutdown_connections 2023-10-10T19:13:56.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:56.479+0000 7f984f1c5700 1 -- 172.21.15.175:0/3608888363 wait complete. 2023-10-10T19:13:56.482 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 222 2023-10-10T19:13:56.491 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:13:58.492 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:13:58.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.618+0000 7f85d4d1b700 1 Processor -- start 2023-10-10T19:13:58.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.618+0000 7f85d4d1b700 1 -- start start 2023-10-10T19:13:58.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85d4d1b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f85d0155fb0 0x7f85d0159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:58.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85d4d1b700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d01574b0 con 0x7f85d00b2390 2023-10-10T19:13:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85d4d1b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d00afad0 con 0x7f85d00b1fc0 2023-10-10T19:13:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85d4d1b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d0150770 con 0x7f85d0155fb0 2023-10-10T19:13:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85ced9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f85d0155fb0 0x7f85d0159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:58.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85c5d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f85d00b2390 0x7f85d00aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56544/0 (socket says 172.21.15.175:56544) 2023-10-10T19:13:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85c5d9b700 1 -- 172.21.15.175:0/4257236822 learned_addr learned my addr 172.21.15.175:0/4257236822 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:13:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85ced9d700 1 -- 172.21.15.175:0/4257236822 >> v1:172.21.15.191:6789/0 conn(0x7f85d00b1fc0 legacy=0x7f85d0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:13:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85cdd9b700 1 -- 172.21.15.175:0/4257236822 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145749261 0 0) 0x7f85d01574b0 con 0x7f85d00b2390 2023-10-10T19:13:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85ced9d700 1 -- 172.21.15.175:0/4257236822 >> v1:172.21.15.175:6789/0 conn(0x7f85d00b2390 legacy=0x7f85d00aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:58.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.619+0000 7f85ced9d700 1 -- 172.21.15.175:0/4257236822 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d00af4c0 con 0x7f85d0155fb0 2023-10-10T19:13:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85ced9d700 1 --2- 172.21.15.175:0/4257236822 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d0155fb0 0x7f85d0159920 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009e60 tx=0x7f85c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0887f62c0a7f26c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85cdd9b700 1 -- 172.21.15.175:0/4257236822 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c800e070 con 0x7f85d0155fb0 2023-10-10T19:13:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85cdd9b700 1 -- 172.21.15.175:0/4257236822 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c800bcc0 con 0x7f85d0155fb0 2023-10-10T19:13:58.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85cdd9b700 1 -- 172.21.15.175:0/4257236822 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8012680 con 0x7f85d0155fb0 2023-10-10T19:13:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85d4d1b700 1 -- 172.21.15.175:0/4257236822 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d0155fb0 msgr2=0x7f85d0159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85d4d1b700 1 --2- 172.21.15.175:0/4257236822 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d0155fb0 0x7f85d0159920 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009e60 tx=0x7f85c8009780 comp rx=0 tx=0).stop 2023-10-10T19:13:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85d4d1b700 1 -- 172.21.15.175:0/4257236822 shutdown_connections 2023-10-10T19:13:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85d4d1b700 1 --2- 172.21.15.175:0/4257236822 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d0155fb0 0x7f85d0159920 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:58.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85d4d1b700 1 -- 172.21.15.175:0/4257236822 >> 172.21.15.175:0/4257236822 conn(0x7f85d0063230 msgr2=0x7f85d0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:58.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85d4d1b700 1 -- 172.21.15.175:0/4257236822 shutdown_connections 2023-10-10T19:13:58.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.620+0000 7f85d4d1b700 1 -- 172.21.15.175:0/4257236822 wait complete. 2023-10-10T19:13:58.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85d4d1b700 1 Processor -- start 2023-10-10T19:13:58.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85d4d1b700 1 -- start start 2023-10-10T19:13:58.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85d4d1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d00b1fc0 0x7f85d00c1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:58.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85d4d1b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85d00b2390 0x7f85d00c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:58.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85d4d1b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d0155fb0 0x7f85d00c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:58.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85d4d1b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00c0c80 con 0x7f85d00b1fc0 2023-10-10T19:13:58.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85d4d1b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00ccf50 con 0x7f85d0155fb0 2023-10-10T19:13:58.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85d4d1b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00c3b30 con 0x7f85d00b2390 2023-10-10T19:13:58.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85ce59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d00b1fc0 0x7f85d00c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:58.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85ce59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d00b1fc0 0x7f85d00c1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34260/0 (socket says 172.21.15.175:34260) 2023-10-10T19:13:58.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85ce59c700 1 -- 172.21.15.175:0/2452484314 learned_addr learned my addr 172.21.15.175:0/2452484314 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:13:58.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85ced9d700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d0155fb0 0x7f85d00c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:58.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.622+0000 7f85c5d9b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85d00b2390 0x7f85d00c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:58.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85ce59c700 1 -- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85d00b2390 msgr2=0x7f85d00c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:58.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85ce59c700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85d00b2390 0x7f85d00c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:58.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85ce59c700 1 -- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d0155fb0 msgr2=0x7f85d00c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:58.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85ce59c700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d0155fb0 0x7f85d00c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:58.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85ce59c700 1 -- 172.21.15.175:0/2452484314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8009a20 con 0x7f85d00b1fc0 2023-10-10T19:13:58.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85ce59c700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d00b1fc0 0x7f85d00c1390 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f85c000c5c0 tx=0x7f85c000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:58.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85c6ffd700 1 -- 172.21.15.175:0/2452484314 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c0011590 con 0x7f85d00b1fc0 2023-10-10T19:13:58.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85c6ffd700 1 -- 172.21.15.175:0/2452484314 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c0011710 con 0x7f85d00b1fc0 2023-10-10T19:13:58.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85c6ffd700 1 -- 172.21.15.175:0/2452484314 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c0011ad0 con 0x7f85d00b1fc0 2023-10-10T19:13:58.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d00c0f00 con 0x7f85d00b1fc0 2023-10-10T19:13:58.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.623+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d00cc6b0 con 0x7f85d00b1fc0 2023-10-10T19:13:58.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.624+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d0067570 con 0x7f85d00b1fc0 2023-10-10T19:13:58.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.624+0000 7f85c6ffd700 1 -- 172.21.15.175:0/2452484314 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f85c001b3b0 con 0x7f85d00b1fc0 2023-10-10T19:13:58.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.625+0000 7f85c6ffd700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f85b0066f30 0x7f85b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:13:58.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.625+0000 7f85c6ffd700 1 -- 172.21.15.175:0/2452484314 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f85c0083f30 con 0x7f85d00b1fc0 2023-10-10T19:13:58.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.625+0000 7f85c5d9b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f85b0066f30 0x7f85b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:13:58.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.626+0000 7f85c5d9b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f85b0066f30 0x7f85b00693e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f85d015a580 tx=0x7f85b8007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:13:58.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.629+0000 7f85c6ffd700 1 -- 172.21.15.175:0/2452484314 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f85c0050c20 con 0x7f85d00b1fc0 2023-10-10T19:13:58.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.857+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85d00c11e0 con 0x7f85d00b1fc0 2023-10-10T19:13:58.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.858+0000 7f85c6ffd700 1 -- 172.21.15.175:0/2452484314 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+5449 (secure 0 0 0) 0x7f85c0017070 con 0x7f85d00b1fc0 2023-10-10T19:13:58.859 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:13:58.859 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:13:54.771130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:resolve","state_seq":37,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:13:58.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f85b0066f30 msgr2=0x7f85b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:58.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f85b0066f30 0x7f85b00693e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f85d015a580 tx=0x7f85b8007d30 comp rx=0 tx=0).stop 2023-10-10T19:13:58.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d00b1fc0 msgr2=0x7f85d00c1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:13:58.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d00b1fc0 0x7f85d00c1390 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f85c000c5c0 tx=0x7f85c000c8d0 comp rx=0 tx=0).stop 2023-10-10T19:13:58.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 shutdown_connections 2023-10-10T19:13:58.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f85b0066f30 0x7f85b00693e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:58.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d0155fb0 0x7f85d00c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:58.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85d00b1fc0 0x7f85d00c1390 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:58.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 --2- 172.21.15.175:0/2452484314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85d00b2390 0x7f85d00c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:13:58.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 >> 172.21.15.175:0/2452484314 conn(0x7f85d0063230 msgr2=0x7f85d01508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:13:58.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 shutdown_connections 2023-10-10T19:13:58.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:13:58.860+0000 7f85d4d1b700 1 -- 172.21.15.175:0/2452484314 wait complete. 2023-10-10T19:13:58.863 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 222 2023-10-10T19:13:58.872 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:00.873 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:00.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.996+0000 7ff21ccd3700 1 Processor -- start 2023-10-10T19:14:00.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.996+0000 7ff21ccd3700 1 -- start start 2023-10-10T19:14:00.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff21ccd3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff2180b1fc0 0x7ff218155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff21ccd3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff21ccd3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff218068bd0 0x7ff2180aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff21ccd3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff218150770 con 0x7ff218068bd0 2023-10-10T19:14:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff21ccd3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2181574b0 con 0x7ff2180b1fc0 2023-10-10T19:14:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff21ccd3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2180afad0 con 0x7ff2180b2390 2023-10-10T19:14:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff216d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff218068bd0 0x7ff2180aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff216d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff218068bd0 0x7ff2180aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34264/0 (socket says 172.21.15.175:34264) 2023-10-10T19:14:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.997+0000 7ff216d9d700 1 -- 172.21.15.175:0/28454796 learned_addr learned my addr 172.21.15.175:0/28454796 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:01.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff21659c700 1 --2- 172.21.15.175:0/28454796 >> v2:172.21.15.191:3300/0 conn(0x7ff2180b1fc0 0x7ff218155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:01.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff215d9b700 1 --2- 172.21.15.175:0/28454796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:01.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff216d9d700 1 -- 172.21.15.175:0/28454796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 msgr2=0x7ff218068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:01.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff216d9d700 1 --2- 172.21.15.175:0/28454796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff216d9d700 1 -- 172.21.15.175:0/28454796 >> v2:172.21.15.191:3300/0 conn(0x7ff2180b1fc0 msgr2=0x7ff218155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:01.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff216d9d700 1 --2- 172.21.15.175:0/28454796 >> v2:172.21.15.191:3300/0 conn(0x7ff2180b1fc0 0x7ff218155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff216d9d700 1 -- 172.21.15.175:0/28454796 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2180af4c0 con 0x7ff218068bd0 2023-10-10T19:14:01.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff21659c700 1 --2- 172.21.15.175:0/28454796 >> v2:172.21.15.191:3300/0 conn(0x7ff2180b1fc0 0x7ff218155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:01.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff215d9b700 1 --2- 172.21.15.175:0/28454796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:01.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff216d9d700 1 --2- 172.21.15.175:0/28454796 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff2180aff60 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7ff2180b28c0 tx=0x7ff208009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb69b3652372901 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:01.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff21559a700 1 -- 172.21.15.175:0/28454796 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20800b8f0 con 0x7ff218068bd0 2023-10-10T19:14:01.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff21559a700 1 -- 172.21.15.175:0/28454796 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20800ba50 con 0x7ff218068bd0 2023-10-10T19:14:01.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff21559a700 1 -- 172.21.15.175:0/28454796 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20800bdf0 con 0x7ff218068bd0 2023-10-10T19:14:01.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff21ccd3700 1 -- 172.21.15.175:0/28454796 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 msgr2=0x7ff2180aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:01.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.998+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/28454796 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff2180aff60 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7ff2180b28c0 tx=0x7ff208009250 comp rx=0 tx=0).stop 2023-10-10T19:14:01.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.999+0000 7ff21ccd3700 1 -- 172.21.15.175:0/28454796 shutdown_connections 2023-10-10T19:14:01.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.999+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/28454796 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff2180aff60 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.999+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/28454796 >> v2:172.21.15.191:3300/0 conn(0x7ff2180b1fc0 0x7ff218155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.999+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/28454796 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.999+0000 7ff21ccd3700 1 -- 172.21.15.175:0/28454796 >> 172.21.15.175:0/28454796 conn(0x7ff218063230 msgr2=0x7ff218063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:01.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.999+0000 7ff21ccd3700 1 -- 172.21.15.175:0/28454796 shutdown_connections 2023-10-10T19:14:01.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:00.999+0000 7ff21ccd3700 1 -- 172.21.15.175:0/28454796 wait complete. 2023-10-10T19:14:01.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21ccd3700 1 Processor -- start 2023-10-10T19:14:01.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21ccd3700 1 -- start start 2023-10-10T19:14:01.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21ccd3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff218153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:01.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21ccd3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2180b1fc0 0x7ff2181561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:01.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21ccd3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:01.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21ccd3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2180c6ad0 con 0x7ff218068bd0 2023-10-10T19:14:01.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21ccd3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2180c43a0 con 0x7ff2180b1fc0 2023-10-10T19:14:01.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21ccd3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2180c3f70 con 0x7ff2180b2390 2023-10-10T19:14:01.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff218153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:01.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff218153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34272/0 (socket says 172.21.15.175:34272) 2023-10-10T19:14:01.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.000+0000 7ff21659c700 1 -- 172.21.15.175:0/1564172271 learned_addr learned my addr 172.21.15.175:0/1564172271 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:01.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff216d9d700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:01.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff215d9b700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2180b1fc0 0x7ff2181561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:01.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff21659c700 1 -- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 msgr2=0x7ff218156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:01.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff21659c700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff21659c700 1 -- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2180b1fc0 msgr2=0x7ff2181561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:01.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff21659c700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2180b1fc0 0x7ff2181561e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff21659c700 1 -- 172.21.15.175:0/1564172271 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080094f0 con 0x7ff218068bd0 2023-10-10T19:14:01.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff215d9b700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2180b1fc0 0x7ff2181561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:01.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff21659c700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff218153f20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7ff21000a400 tx=0x7ff210007880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:01.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff2067fc700 1 -- 172.21.15.175:0/1564172271 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21000d650 con 0x7ff218068bd0 2023-10-10T19:14:01.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff2067fc700 1 -- 172.21.15.175:0/1564172271 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21000d7b0 con 0x7ff218068bd0 2023-10-10T19:14:01.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff2067fc700 1 -- 172.21.15.175:0/1564172271 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21000db50 con 0x7ff218068bd0 2023-10-10T19:14:01.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff218166450 con 0x7ff218068bd0 2023-10-10T19:14:01.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.001+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2180c6510 con 0x7ff218068bd0 2023-10-10T19:14:01.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.002+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff218067570 con 0x7ff218068bd0 2023-10-10T19:14:01.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.002+0000 7ff2067fc700 1 -- 172.21.15.175:0/1564172271 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff210015440 con 0x7ff218068bd0 2023-10-10T19:14:01.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.003+0000 7ff2067fc700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff1f8066f30 0x7ff1f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:01.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.003+0000 7ff2067fc700 1 -- 172.21.15.175:0/1564172271 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7ff210085390 con 0x7ff218068bd0 2023-10-10T19:14:01.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.003+0000 7ff215d9b700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff1f8066f30 0x7ff1f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:01.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.004+0000 7ff215d9b700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff1f8066f30 0x7ff1f80693e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff20c006fd0 tx=0x7ff20c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:01.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.007+0000 7ff2067fc700 1 -- 172.21.15.175:0/1564172271 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff21004e5a0 con 0x7ff218068bd0 2023-10-10T19:14:01.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.233+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff218069060 con 0x7ff218068bd0 2023-10-10T19:14:01.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.233+0000 7ff2067fc700 1 -- 172.21.15.175:0/1564172271 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+5451 (secure 0 0 0) 0x7ff210052130 con 0x7ff218068bd0 2023-10-10T19:14:01.234 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:01.234 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":223,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:00.795351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:01.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff1f8066f30 msgr2=0x7ff1f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:01.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff1f8066f30 0x7ff1f80693e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff20c006fd0 tx=0x7ff20c008040 comp rx=0 tx=0).stop 2023-10-10T19:14:01.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 msgr2=0x7ff218153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:01.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff218153f20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7ff21000a400 tx=0x7ff210007880 comp rx=0 tx=0).stop 2023-10-10T19:14:01.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 shutdown_connections 2023-10-10T19:14:01.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff1f8066f30 0x7ff1f80693e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2180b2390 0x7ff218156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff218068bd0 0x7ff218153f20 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 --2- 172.21.15.175:0/1564172271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2180b1fc0 0x7ff2181561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:01.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 >> 172.21.15.175:0/1564172271 conn(0x7ff218063230 msgr2=0x7ff2180c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:01.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 shutdown_connections 2023-10-10T19:14:01.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:01.235+0000 7ff21ccd3700 1 -- 172.21.15.175:0/1564172271 wait complete. 2023-10-10T19:14:01.238 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 223 2023-10-10T19:14:01.248 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 223, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 223, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6117": { "addr": "172.21.15.175:6834/1872591144", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 1872591144, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 1872591144, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6117, "incarnation": 134, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 39 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:14:00.795351+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 6117, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:14:03.250 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:03.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.378+0000 7fe9d1c73700 1 Processor -- start 2023-10-10T19:14:03.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.378+0000 7fe9d1c73700 1 -- start start 2023-10-10T19:14:03.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.378+0000 7fe9d1c73700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9cc0c35e0 0x7fe9cc159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:03.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9d1c73700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe9cc0c9390 0x7fe9cc0c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:03.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9d1c73700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0c5e40 con 0x7fe9cc0c3210 2023-10-10T19:14:03.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9d1c73700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9cc0c4c50 con 0x7fe9cc0c9390 2023-10-10T19:14:03.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9d1c73700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9cc0ca400 con 0x7fe9cc0c35e0 2023-10-10T19:14:03.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9cbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe9cc0c9390 0x7fe9cc0c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:03.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9cbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe9cc0c9390 0x7fe9cc0c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34282/0 (socket says 172.21.15.175:34282) 2023-10-10T19:14:03.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9cbfff700 1 -- 172.21.15.175:0/1450149064 learned_addr learned my addr 172.21.15.175:0/1450149064 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:03.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9caffd700 1 --2- 172.21.15.175:0/1450149064 >> v2:172.21.15.191:3300/0 conn(0x7fe9cc0c35e0 0x7fe9cc159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:03.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9cbfff700 1 -- 172.21.15.175:0/1450149064 >> v2:172.21.15.191:3300/0 conn(0x7fe9cc0c35e0 msgr2=0x7fe9cc159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:03.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9cbfff700 1 --2- 172.21.15.175:0/1450149064 >> v2:172.21.15.191:3300/0 conn(0x7fe9cc0c35e0 0x7fe9cc159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9cbfff700 1 -- 172.21.15.175:0/1450149064 >> v1:172.21.15.191:6789/0 conn(0x7fe9cc0c3210 legacy=0x7fe9cc0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:03.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9cbfff700 1 -- 172.21.15.175:0/1450149064 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc0c9880 con 0x7fe9cc0c9390 2023-10-10T19:14:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9cbfff700 1 --2- 172.21.15.175:0/1450149064 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc0c62d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc064890 tx=0x7fe9bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1b2f809d2b1ed3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9ca7fc700 1 -- 172.21.15.175:0/1450149064 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc00b7e0 con 0x7fe9cc0c9390 2023-10-10T19:14:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9ca7fc700 1 -- 172.21.15.175:0/1450149064 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc00b940 con 0x7fe9cc0c9390 2023-10-10T19:14:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.379+0000 7fe9ca7fc700 1 -- 172.21.15.175:0/1450149064 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc00bce0 con 0x7fe9cc0c9390 2023-10-10T19:14:03.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.380+0000 7fe9d1c73700 1 -- 172.21.15.175:0/1450149064 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 msgr2=0x7fe9cc0c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.380+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/1450149064 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc0c62d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc064890 tx=0x7fe9bc009250 comp rx=0 tx=0).stop 2023-10-10T19:14:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.380+0000 7fe9d1c73700 1 -- 172.21.15.175:0/1450149064 shutdown_connections 2023-10-10T19:14:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.380+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/1450149064 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc0c62d0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.380+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/1450149064 >> v2:172.21.15.191:3300/0 conn(0x7fe9cc0c35e0 0x7fe9cc159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.380+0000 7fe9d1c73700 1 -- 172.21.15.175:0/1450149064 >> 172.21.15.175:0/1450149064 conn(0x7fe9cc06c110 msgr2=0x7fe9cc06c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.380+0000 7fe9d1c73700 1 -- 172.21.15.175:0/1450149064 shutdown_connections 2023-10-10T19:14:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.380+0000 7fe9d1c73700 1 -- 172.21.15.175:0/1450149064 wait complete. 2023-10-10T19:14:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.381+0000 7fe9d1c73700 1 Processor -- start 2023-10-10T19:14:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.381+0000 7fe9d1c73700 1 -- start start 2023-10-10T19:14:03.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9d1c73700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9cc0c3210 0x7fe9cc13df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9d1c73700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9cc0c35e0 0x7fe9cc1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9d1c73700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9d1c73700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc0c5840 con 0x7fe9cc0c9390 2023-10-10T19:14:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9d1c73700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc152a20 con 0x7fe9cc0c35e0 2023-10-10T19:14:03.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9d1c73700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc169d00 con 0x7fe9cc0c3210 2023-10-10T19:14:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34284/0 (socket says 172.21.15.175:34284) 2023-10-10T19:14:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 -- 172.21.15.175:0/2530793678 learned_addr learned my addr 172.21.15.175:0/2530793678 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9caffd700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9cc0c35e0 0x7fe9cc1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:03.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cb7fe700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9cc0c3210 0x7fe9cc13df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:03.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 -- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9cc0c3210 msgr2=0x7fe9cc13df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:03.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9cc0c3210 0x7fe9cc13df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 -- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9cc0c35e0 msgr2=0x7fe9cc1591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:03.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9cc0c35e0 0x7fe9cc1591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 -- 172.21.15.175:0/2530793678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0094f0 con 0x7fe9cc0c9390 2023-10-10T19:14:03.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cb7fe700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9cc0c3210 0x7fe9cc13df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:14:03.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9cbfff700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc159700 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009290 tx=0x7fe9bc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:03.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.382+0000 7fe9b37fe700 1 -- 172.21.15.175:0/2530793678 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc00b7e0 con 0x7fe9cc0c9390 2023-10-10T19:14:03.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.383+0000 7fe9b37fe700 1 -- 172.21.15.175:0/2530793678 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc020070 con 0x7fe9cc0c9390 2023-10-10T19:14:03.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.383+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc169f80 con 0x7fe9cc0c9390 2023-10-10T19:14:03.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.383+0000 7fe9b37fe700 1 -- 172.21.15.175:0/2530793678 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc01b6a0 con 0x7fe9cc0c9390 2023-10-10T19:14:03.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.383+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc16a540 con 0x7fe9cc0c9390 2023-10-10T19:14:03.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.384+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc067050 con 0x7fe9cc0c9390 2023-10-10T19:14:03.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.384+0000 7fe9b37fe700 1 -- 172.21.15.175:0/2530793678 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe9bc022030 con 0x7fe9cc0c9390 2023-10-10T19:14:03.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.384+0000 7fe9b37fe700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9b8066f30 0x7fe9b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:03.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.385+0000 7fe9b37fe700 1 -- 172.21.15.175:0/2530793678 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7fe9bc086aa0 con 0x7fe9cc0c9390 2023-10-10T19:14:03.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.385+0000 7fe9cb7fe700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9b8066f30 0x7fe9b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:03.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.385+0000 7fe9cb7fe700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9b8066f30 0x7fe9b80693e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4006fd0 tx=0x7fe9c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:03.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.389+0000 7fe9b37fe700 1 -- 172.21.15.175:0/2530793678 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe9bc053790 con 0x7fe9cc0c9390 2023-10-10T19:14:03.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.620+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9cc067050 con 0x7fe9cc0c9390 2023-10-10T19:14:03.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.621+0000 7fe9b37fe700 1 -- 172.21.15.175:0/2530793678 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+5451 (secure 0 0 0) 0x7fe9bc084e60 con 0x7fe9cc0c9390 2023-10-10T19:14:03.622 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:03.622 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":223,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:00.795351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:03.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9b8066f30 msgr2=0x7fe9b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:03.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9b8066f30 0x7fe9b80693e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4006fd0 tx=0x7fe9c4008040 comp rx=0 tx=0).stop 2023-10-10T19:14:03.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 msgr2=0x7fe9cc159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:03.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc159700 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009290 tx=0x7fe9bc016000 comp rx=0 tx=0).stop 2023-10-10T19:14:03.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 shutdown_connections 2023-10-10T19:14:03.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe9b8066f30 0x7fe9b80693e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe9cc0c9390 0x7fe9cc159700 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9cc0c3210 0x7fe9cc13df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 --2- 172.21.15.175:0/2530793678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9cc0c35e0 0x7fe9cc1591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:03.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 >> 172.21.15.175:0/2530793678 conn(0x7fe9cc06c110 msgr2=0x7fe9cc0c17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:03.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 shutdown_connections 2023-10-10T19:14:03.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:03.623+0000 7fe9d1c73700 1 -- 172.21.15.175:0/2530793678 wait complete. 2023-10-10T19:14:03.628 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 223 2023-10-10T19:14:03.636 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:05.637 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:05.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.765+0000 7f907d090700 1 Processor -- start 2023-10-10T19:14:05.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.765+0000 7f907d090700 1 -- start start 2023-10-10T19:14:05.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f907d090700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907808a100 0x7f9078159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f907d090700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90780cd040 0x7f90780684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f907d090700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9078067fa0 con 0x7f907814fd90 2023-10-10T19:14:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f907d090700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90780c5600 con 0x7f90780cd040 2023-10-10T19:14:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f907d090700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9078156b70 con 0x7f907808a100 2023-10-10T19:14:05.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f907c88f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90780cd040 0x7f90780684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f9077fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f907814fd90 0x7f90780c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40134/0 (socket says 172.21.15.175:40134) 2023-10-10T19:14:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f9077fff700 1 -- 172.21.15.175:0/1620942274 learned_addr learned my addr 172.21.15.175:0/1620942274 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f90777fe700 1 --2- 172.21.15.175:0/1620942274 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907808a100 0x7f9078159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f9076ffd700 1 -- 172.21.15.175:0/1620942274 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299916811 0 0) 0x7f9078067fa0 con 0x7f907814fd90 2023-10-10T19:14:05.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.766+0000 7f9076ffd700 1 -- 172.21.15.175:0/1620942274 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9060003570 con 0x7f907814fd90 2023-10-10T19:14:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907c88f700 1 -- 172.21.15.175:0/1620942274 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907808a100 msgr2=0x7f9078159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907c88f700 1 --2- 172.21.15.175:0/1620942274 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907808a100 0x7f9078159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907c88f700 1 -- 172.21.15.175:0/1620942274 >> v1:172.21.15.175:6789/0 conn(0x7f907814fd90 legacy=0x7f90780c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907c88f700 1 -- 172.21.15.175:0/1620942274 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9078150160 con 0x7f90780cd040 2023-10-10T19:14:05.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f90777fe700 1 --2- 172.21.15.175:0/1620942274 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907808a100 0x7f9078159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:14:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907c88f700 1 --2- 172.21.15.175:0/1620942274 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90780684f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f9070009e60 tx=0x7f9070009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d841d2a431d9e20 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f9076ffd700 1 -- 172.21.15.175:0/1620942274 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907000e070 con 0x7f90780cd040 2023-10-10T19:14:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f9076ffd700 1 -- 172.21.15.175:0/1620942274 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907000bb60 con 0x7f90780cd040 2023-10-10T19:14:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f9076ffd700 1 -- 172.21.15.175:0/1620942274 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070012430 con 0x7f90780cd040 2023-10-10T19:14:05.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907d090700 1 -- 172.21.15.175:0/1620942274 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 msgr2=0x7f90780684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:05.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907d090700 1 --2- 172.21.15.175:0/1620942274 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90780684f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f9070009e60 tx=0x7f9070009780 comp rx=0 tx=0).stop 2023-10-10T19:14:05.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907d090700 1 -- 172.21.15.175:0/1620942274 shutdown_connections 2023-10-10T19:14:05.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907d090700 1 --2- 172.21.15.175:0/1620942274 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90780684f0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:05.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907d090700 1 --2- 172.21.15.175:0/1620942274 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907808a100 0x7f9078159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:05.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907d090700 1 -- 172.21.15.175:0/1620942274 >> 172.21.15.175:0/1620942274 conn(0x7f9078063d00 msgr2=0x7f90780cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:05.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907d090700 1 -- 172.21.15.175:0/1620942274 shutdown_connections 2023-10-10T19:14:05.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.767+0000 7f907d090700 1 -- 172.21.15.175:0/1620942274 wait complete. 2023-10-10T19:14:05.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.768+0000 7f907d090700 1 Processor -- start 2023-10-10T19:14:05.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.769+0000 7f907d090700 1 -- start start 2023-10-10T19:14:05.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.769+0000 7f907d090700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907808a100 0x7f907814e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:05.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.769+0000 7f907d090700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90781660c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:05.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.769+0000 7f907d090700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907814fd90 0x7f9078166600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:05.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.769+0000 7f907d090700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9078165bc0 con 0x7f90780cd040 2023-10-10T19:14:05.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.769+0000 7f907d090700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9078145670 con 0x7f907808a100 2023-10-10T19:14:05.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.769+0000 7f907d090700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9078145950 con 0x7f907814fd90 2023-10-10T19:14:05.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90781660c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:05.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90781660c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60332/0 (socket says 172.21.15.175:60332) 2023-10-10T19:14:05.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 -- 172.21.15.175:0/3564271525 learned_addr learned my addr 172.21.15.175:0/3564271525 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:05.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f9077fff700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907808a100 0x7f907814e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:05.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f907c88f700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907814fd90 0x7f9078166600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:05.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 -- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907814fd90 msgr2=0x7f9078166600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:05.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907814fd90 0x7f9078166600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:05.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 -- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907808a100 msgr2=0x7f907814e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:05.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907808a100 0x7f907814e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:05.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 -- 172.21.15.175:0/3564271525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9070009a20 con 0x7f90780cd040 2023-10-10T19:14:05.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f90777fe700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90781660c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f906c00d860 tx=0x7f906c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:05.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f905ffff700 1 -- 172.21.15.175:0/3564271525 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c0125f0 con 0x7f90780cd040 2023-10-10T19:14:05.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f905ffff700 1 -- 172.21.15.175:0/3564271525 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c012750 con 0x7f90780cd040 2023-10-10T19:14:05.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9078145c30 con 0x7f90780cd040 2023-10-10T19:14:05.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f905ffff700 1 -- 172.21.15.175:0/3564271525 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c012ad0 con 0x7f90780cd040 2023-10-10T19:14:05.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.770+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9078146340 con 0x7f90780cd040 2023-10-10T19:14:05.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.771+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9078067570 con 0x7f90780cd040 2023-10-10T19:14:05.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.772+0000 7f905ffff700 1 -- 172.21.15.175:0/3564271525 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f906c014430 con 0x7f90780cd040 2023-10-10T19:14:05.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.772+0000 7f905ffff700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9064066f30 0x7f90640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:05.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.772+0000 7f905ffff700 1 -- 172.21.15.175:0/3564271525 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f906c084f00 con 0x7f90780cd040 2023-10-10T19:14:05.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.772+0000 7f9077fff700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9064066f30 0x7f90640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:05.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.773+0000 7f9077fff700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9064066f30 0x7f90640693e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f90680058d0 tx=0x7f9068009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:05.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:05.776+0000 7f905ffff700 1 -- 172.21.15.175:0/3564271525 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f906c055680 con 0x7f90780cd040 2023-10-10T19:14:06.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.004+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9078155f80 con 0x7f90780cd040 2023-10-10T19:14:06.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.005+0000 7f905ffff700 1 -- 172.21.15.175:0/3564271525 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f906c055210 con 0x7f90780cd040 2023-10-10T19:14:06.005 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:06.006 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:05.817477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:06.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9064066f30 msgr2=0x7f90640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:06.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9064066f30 0x7f90640693e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f90680058d0 tx=0x7f9068009040 comp rx=0 tx=0).stop 2023-10-10T19:14:06.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 msgr2=0x7f90781660c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:06.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90781660c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f906c00d860 tx=0x7f906c008040 comp rx=0 tx=0).stop 2023-10-10T19:14:06.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 shutdown_connections 2023-10-10T19:14:06.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9064066f30 0x7f90640693e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:06.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f907814fd90 0x7f9078166600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:06.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f907808a100 0x7f907814e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:06.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 --2- 172.21.15.175:0/3564271525 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90780cd040 0x7f90781660c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:06.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 >> 172.21.15.175:0/3564271525 conn(0x7f9078063d00 msgr2=0x7f9078159100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:06.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 shutdown_connections 2023-10-10T19:14:06.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:06.007+0000 7f907d090700 1 -- 172.21.15.175:0/3564271525 wait complete. 2023-10-10T19:14:06.010 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 224 2023-10-10T19:14:06.019 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:08.021 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:08.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.147+0000 7f1255506700 1 Processor -- start 2023-10-10T19:14:08.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.147+0000 7f1255506700 1 -- start start 2023-10-10T19:14:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.148+0000 7f1255506700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f12501573d0 0x7f12500c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.148+0000 7f1255506700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1250154aa0 con 0x7f12500c4f20 2023-10-10T19:14:08.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.148+0000 7f1255506700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1250157c20 con 0x7f12501577a0 2023-10-10T19:14:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.148+0000 7f1255506700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12500bfdf0 con 0x7f12501573d0 2023-10-10T19:14:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.148+0000 7f124effd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f12501573d0 0x7f12500c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.148+0000 7f124f7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f12500c4f20 0x7f125014d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40150/0 (socket says 172.21.15.175:40150) 2023-10-10T19:14:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.148+0000 7f124f7fe700 1 -- 172.21.15.175:0/84786176 learned_addr learned my addr 172.21.15.175:0/84786176 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:08.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f124effd700 1 -- 172.21.15.175:0/84786176 >> v1:172.21.15.191:6789/0 conn(0x7f12501577a0 legacy=0x7f12500bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f124effd700 1 -- 172.21.15.175:0/84786176 >> v1:172.21.15.175:6789/0 conn(0x7f12500c4f20 legacy=0x7f125014d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f124effd700 1 -- 172.21.15.175:0/84786176 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1250063dc0 con 0x7f12501573d0 2023-10-10T19:14:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f124effd700 1 --2- 172.21.15.175:0/84786176 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12501573d0 0x7f12500c12a0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1240009e60 tx=0x7f1240009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41981887cab0079 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f124dffb700 1 -- 172.21.15.175:0/84786176 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124000e070 con 0x7f12501573d0 2023-10-10T19:14:08.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f124dffb700 1 -- 172.21.15.175:0/84786176 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124000bbf0 con 0x7f12501573d0 2023-10-10T19:14:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f124dffb700 1 -- 172.21.15.175:0/84786176 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1240012430 con 0x7f12501573d0 2023-10-10T19:14:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f1255506700 1 -- 172.21.15.175:0/84786176 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12501573d0 msgr2=0x7f12500c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f1255506700 1 --2- 172.21.15.175:0/84786176 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12501573d0 0x7f12500c12a0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1240009e60 tx=0x7f1240009780 comp rx=0 tx=0).stop 2023-10-10T19:14:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f1255506700 1 -- 172.21.15.175:0/84786176 shutdown_connections 2023-10-10T19:14:08.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f1255506700 1 --2- 172.21.15.175:0/84786176 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12501573d0 0x7f12500c12a0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:08.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.149+0000 7f1255506700 1 -- 172.21.15.175:0/84786176 >> 172.21.15.175:0/84786176 conn(0x7f125006dc70 msgr2=0x7f1250063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:08.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.150+0000 7f1255506700 1 -- 172.21.15.175:0/84786176 shutdown_connections 2023-10-10T19:14:08.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.150+0000 7f1255506700 1 -- 172.21.15.175:0/84786176 wait complete. 2023-10-10T19:14:08.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f1255506700 1 Processor -- start 2023-10-10T19:14:08.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f1255506700 1 -- start start 2023-10-10T19:14:08.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f1255506700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12500c4f20 0x7f12500c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:08.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f1255506700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12501573d0 0x7f1250147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:08.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f1255506700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12501577a0 0x7f12500cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:08.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f1255506700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12500c8dc0 con 0x7f12500c4f20 2023-10-10T19:14:08.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f1255506700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1250164250 con 0x7f12501577a0 2023-10-10T19:14:08.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f1255506700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12500ca490 con 0x7f12501573d0 2023-10-10T19:14:08.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.151+0000 7f124effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12500c4f20 0x7f12500c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:08.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12500c4f20 0x7f12500c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60346/0 (socket says 172.21.15.175:60346) 2023-10-10T19:14:08.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124effd700 1 -- 172.21.15.175:0/2744723710 learned_addr learned my addr 172.21.15.175:0/2744723710 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:08.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124f7fe700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12501577a0 0x7f12500cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:08.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124e7fc700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12501573d0 0x7f1250147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:08.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124effd700 1 -- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12501573d0 msgr2=0x7f1250147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:08.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124effd700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12501573d0 0x7f1250147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:08.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124effd700 1 -- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12501577a0 msgr2=0x7f12500cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:08.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124effd700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12501577a0 0x7f12500cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:08.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124effd700 1 -- 172.21.15.175:0/2744723710 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1240009a20 con 0x7f12500c4f20 2023-10-10T19:14:08.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124f7fe700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12501577a0 0x7f12500cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:14:08.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f124effd700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12500c4f20 0x7f12500c9250 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f125006e6e0 tx=0x7f1240018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:08.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f1236ffd700 1 -- 172.21.15.175:0/2744723710 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1240027470 con 0x7f12500c4f20 2023-10-10T19:14:08.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f1236ffd700 1 -- 172.21.15.175:0/2744723710 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12400275f0 con 0x7f12500c4f20 2023-10-10T19:14:08.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f1236ffd700 1 -- 172.21.15.175:0/2744723710 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12400279f0 con 0x7f12500c4f20 2023-10-10T19:14:08.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1250164510 con 0x7f12500c4f20 2023-10-10T19:14:08.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.152+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12500c8770 con 0x7f12500c4f20 2023-10-10T19:14:08.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.153+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12500c90a0 con 0x7f12500c4f20 2023-10-10T19:14:08.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.154+0000 7f1236ffd700 1 -- 172.21.15.175:0/2744723710 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1240024070 con 0x7f12500c4f20 2023-10-10T19:14:08.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.154+0000 7f1236ffd700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f123c066f30 0x7f123c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:08.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.154+0000 7f1236ffd700 1 -- 172.21.15.175:0/2744723710 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f1240086ff0 con 0x7f12500c4f20 2023-10-10T19:14:08.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.154+0000 7f124e7fc700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f123c066f30 0x7f123c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:08.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.155+0000 7f124e7fc700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f123c066f30 0x7f123c0693e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1250156eb0 tx=0x7f1244009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:08.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.158+0000 7f1236ffd700 1 -- 172.21.15.175:0/2744723710 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1240053ce0 con 0x7f12500c4f20 2023-10-10T19:14:08.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.385+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1250067570 con 0x7f12500c4f20 2023-10-10T19:14:08.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.385+0000 7f1236ffd700 1 -- 172.21.15.175:0/2744723710 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f1240057300 con 0x7f12500c4f20 2023-10-10T19:14:08.386 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:08.386 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:05.817477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:08.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.387+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f123c066f30 msgr2=0x7f123c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:08.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.387+0000 7f1255506700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f123c066f30 0x7f123c0693e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1250156eb0 tx=0x7f1244009040 comp rx=0 tx=0).stop 2023-10-10T19:14:08.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.387+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12500c4f20 msgr2=0x7f12500c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:08.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.387+0000 7f1255506700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12500c4f20 0x7f12500c9250 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f125006e6e0 tx=0x7f1240018000 comp rx=0 tx=0).stop 2023-10-10T19:14:08.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.387+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 shutdown_connections 2023-10-10T19:14:08.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.387+0000 7f1255506700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f123c066f30 0x7f123c0693e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:08.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.388+0000 7f1255506700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12501577a0 0x7f12500cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:08.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.388+0000 7f1255506700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12500c4f20 0x7f12500c9250 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:08.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.388+0000 7f1255506700 1 --2- 172.21.15.175:0/2744723710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12501573d0 0x7f1250147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:08.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.388+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 >> 172.21.15.175:0/2744723710 conn(0x7f125006dc70 msgr2=0x7f125014bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:08.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.388+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 shutdown_connections 2023-10-10T19:14:08.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:08.388+0000 7f1255506700 1 -- 172.21.15.175:0/2744723710 wait complete. 2023-10-10T19:14:08.391 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 224 2023-10-10T19:14:08.400 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:10.401 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:10.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.526+0000 7f3987473700 1 Processor -- start 2023-10-10T19:14:10.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.526+0000 7f3987473700 1 -- start start 2023-10-10T19:14:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3987473700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f39800b1fc0 0x7f3980155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3987473700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f39800b2390 0x7f3980068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3987473700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39801574b0 con 0x7f3980068bd0 2023-10-10T19:14:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3987473700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3980150770 con 0x7f39800b2390 2023-10-10T19:14:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3987473700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800afad0 con 0x7f39800b1fc0 2023-10-10T19:14:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3984a0e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f39800b2390 0x7f3980068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3984a0e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f39800b2390 0x7f3980068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60350/0 (socket says 172.21.15.175:60350) 2023-10-10T19:14:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3984a0e700 1 -- 172.21.15.175:0/532033019 learned_addr learned my addr 172.21.15.175:0/532033019 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f398520f700 1 --2- 172.21.15.175:0/532033019 >> v2:172.21.15.191:3300/0 conn(0x7f39800b1fc0 0x7f3980155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:10.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3984a0e700 1 -- 172.21.15.175:0/532033019 >> v2:172.21.15.191:3300/0 conn(0x7f39800b1fc0 msgr2=0x7f3980155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3984a0e700 1 --2- 172.21.15.175:0/532033019 >> v2:172.21.15.191:3300/0 conn(0x7f39800b1fc0 0x7f3980155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3984a0e700 1 -- 172.21.15.175:0/532033019 >> v1:172.21.15.191:6789/0 conn(0x7f3980068bd0 legacy=0x7f39800aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.527+0000 7f3984a0e700 1 -- 172.21.15.175:0/532033019 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800af4c0 con 0x7f39800b2390 2023-10-10T19:14:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f398520f700 1 --2- 172.21.15.175:0/532033019 >> v2:172.21.15.191:3300/0 conn(0x7f39800b1fc0 0x7f3980155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3984a0e700 1 --2- 172.21.15.175:0/532033019 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b2390 0x7f3980068690 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f39800b28c0 tx=0x7f3970009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ff34ccffcad27b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3977fff700 1 -- 172.21.15.175:0/532033019 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397000b8b0 con 0x7f39800b2390 2023-10-10T19:14:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3977fff700 1 -- 172.21.15.175:0/532033019 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397000ba10 con 0x7f39800b2390 2023-10-10T19:14:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3977fff700 1 -- 172.21.15.175:0/532033019 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397000bdb0 con 0x7f39800b2390 2023-10-10T19:14:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3987473700 1 -- 172.21.15.175:0/532033019 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b2390 msgr2=0x7f3980068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3987473700 1 --2- 172.21.15.175:0/532033019 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b2390 0x7f3980068690 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f39800b28c0 tx=0x7f3970009250 comp rx=0 tx=0).stop 2023-10-10T19:14:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3987473700 1 -- 172.21.15.175:0/532033019 shutdown_connections 2023-10-10T19:14:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3987473700 1 --2- 172.21.15.175:0/532033019 >> v2:172.21.15.191:3300/0 conn(0x7f39800b1fc0 0x7f3980155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3987473700 1 --2- 172.21.15.175:0/532033019 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b2390 0x7f3980068690 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3987473700 1 -- 172.21.15.175:0/532033019 >> 172.21.15.175:0/532033019 conn(0x7f3980063230 msgr2=0x7f3980063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3987473700 1 -- 172.21.15.175:0/532033019 shutdown_connections 2023-10-10T19:14:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.528+0000 7f3987473700 1 -- 172.21.15.175:0/532033019 wait complete. 2023-10-10T19:14:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.529+0000 7f3987473700 1 Processor -- start 2023-10-10T19:14:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.530+0000 7f3987473700 1 -- start start 2023-10-10T19:14:10.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.530+0000 7f3987473700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3980068bd0 0x7f398015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:10.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.530+0000 7f3987473700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b1fc0 0x7f398014d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:10.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.530+0000 7f3987473700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39800b2390 0x7f398014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:10.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.530+0000 7f3987473700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3980152220 con 0x7f39800b1fc0 2023-10-10T19:14:10.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.530+0000 7f3987473700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39801530a0 con 0x7f39800b2390 2023-10-10T19:14:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.530+0000 7f3987473700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3980166a70 con 0x7f3980068bd0 2023-10-10T19:14:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.530+0000 7f3984a0e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b1fc0 0x7f398014d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3984a0e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b1fc0 0x7f398014d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60364/0 (socket says 172.21.15.175:60364) 2023-10-10T19:14:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3984a0e700 1 -- 172.21.15.175:0/442938482 learned_addr learned my addr 172.21.15.175:0/442938482 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:10.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f398520f700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3980068bd0 0x7f398015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3985a10700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39800b2390 0x7f398014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3984a0e700 1 -- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3980068bd0 msgr2=0x7f398015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3984a0e700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3980068bd0 0x7f398015ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3984a0e700 1 -- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39800b2390 msgr2=0x7f398014d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:10.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3984a0e700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39800b2390 0x7f398014d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3984a0e700 1 -- 172.21.15.175:0/442938482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39700094f0 con 0x7f39800b1fc0 2023-10-10T19:14:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3985a10700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39800b2390 0x7f398014d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:14:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3984a0e700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b1fc0 0x7f398014d2c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f3970003080 tx=0x7f3970016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f39757fa700 1 -- 172.21.15.175:0/442938482 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397001c460 con 0x7f39800b1fc0 2023-10-10T19:14:10.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f39757fa700 1 -- 172.21.15.175:0/442938482 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397001c5c0 con 0x7f39800b1fc0 2023-10-10T19:14:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3980152ae0 con 0x7f39800b1fc0 2023-10-10T19:14:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f39757fa700 1 -- 172.21.15.175:0/442938482 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397001c980 con 0x7f39800b1fc0 2023-10-10T19:14:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.531+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3980151eb0 con 0x7f39800b1fc0 2023-10-10T19:14:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.532+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39801523d0 con 0x7f39800b1fc0 2023-10-10T19:14:10.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.533+0000 7f39757fa700 1 -- 172.21.15.175:0/442938482 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f397001a450 con 0x7f39800b1fc0 2023-10-10T19:14:10.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.533+0000 7f39757fa700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3964066f30 0x7f39640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:10.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.533+0000 7f39757fa700 1 -- 172.21.15.175:0/442938482 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f3970086ad0 con 0x7f39800b1fc0 2023-10-10T19:14:10.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.533+0000 7f398520f700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3964066f30 0x7f39640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:10.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.534+0000 7f398520f700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3964066f30 0x7f39640693e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f397c00adc0 tx=0x7f397c006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:10.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.537+0000 7f39757fa700 1 -- 172.21.15.175:0/442938482 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39700537c0 con 0x7f39800b1fc0 2023-10-10T19:14:10.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.761+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3980152040 con 0x7f39800b1fc0 2023-10-10T19:14:10.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.761+0000 7f39757fa700 1 -- 172.21.15.175:0/442938482 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f397002b090 con 0x7f39800b1fc0 2023-10-10T19:14:10.763 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:10.763 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:05.817477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:10.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3964066f30 msgr2=0x7f39640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:10.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3964066f30 0x7f39640693e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f397c00adc0 tx=0x7f397c006f90 comp rx=0 tx=0).stop 2023-10-10T19:14:10.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b1fc0 msgr2=0x7f398014d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:10.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b1fc0 0x7f398014d2c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f3970003080 tx=0x7f3970016000 comp rx=0 tx=0).stop 2023-10-10T19:14:10.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 shutdown_connections 2023-10-10T19:14:10.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3964066f30 0x7f39640693e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39800b2390 0x7f398014d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3980068bd0 0x7f398015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 --2- 172.21.15.175:0/442938482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39800b1fc0 0x7f398014d2c0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:10.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.763+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 >> 172.21.15.175:0/442938482 conn(0x7f3980063230 msgr2=0x7f39800c88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:10.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.764+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 shutdown_connections 2023-10-10T19:14:10.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:10.764+0000 7f3987473700 1 -- 172.21.15.175:0/442938482 wait complete. 2023-10-10T19:14:10.769 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 224 2023-10-10T19:14:10.776 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:12.777 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:12.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.905+0000 7f6a80c0c700 1 Processor -- start 2023-10-10T19:14:12.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.905+0000 7f6a80c0c700 1 -- start start 2023-10-10T19:14:12.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a80c0c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6a7c14fd90 0x7f6a7c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a80c0c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c08a100 0x7f6a7c0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a80c0c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6a7c0cd040 0x7f6a7c0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a80c0c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7c0c5600 con 0x7f6a7c14fd90 2023-10-10T19:14:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a80c0c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7c067fa0 con 0x7f6a7c0cd040 2023-10-10T19:14:12.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a80c0c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c156b70 con 0x7f6a7c08a100 2023-10-10T19:14:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6a7c14fd90 0x7f6a7c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6a7c14fd90 0x7f6a7c159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60372/0 (socket says 172.21.15.175:60372) 2023-10-10T19:14:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 -- 172.21.15.175:0/2654601384 learned_addr learned my addr 172.21.15.175:0/2654601384 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7bfff700 1 --2- 172.21.15.175:0/2654601384 >> v2:172.21.15.191:3300/0 conn(0x7f6a7c0cd040 0x7f6a7c0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:12.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 -- 172.21.15.175:0/2654601384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c08a100 msgr2=0x7f6a7c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7affd700 1 --2- 172.21.15.175:0/2654601384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c08a100 0x7f6a7c0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 --2- 172.21.15.175:0/2654601384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c08a100 0x7f6a7c0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 -- 172.21.15.175:0/2654601384 >> v2:172.21.15.191:3300/0 conn(0x7f6a7c0cd040 msgr2=0x7f6a7c0c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 --2- 172.21.15.175:0/2654601384 >> v2:172.21.15.191:3300/0 conn(0x7f6a7c0cd040 0x7f6a7c0c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 -- 172.21.15.175:0/2654601384 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c150160 con 0x7f6a7c14fd90 2023-10-10T19:14:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7bfff700 1 --2- 172.21.15.175:0/2654601384 >> v2:172.21.15.191:3300/0 conn(0x7f6a7c0cd040 0x7f6a7c0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7affd700 1 --2- 172.21.15.175:0/2654601384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c08a100 0x7f6a7c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.906+0000 7f6a7b7fe700 1 --2- 172.21.15.175:0/2654601384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c14fd90 0x7f6a7c159c30 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c069e40 tx=0x7f6a6c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecfe0c6c81f3cf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a7a7fc700 1 -- 172.21.15.175:0/2654601384 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c00e070 con 0x7f6a7c14fd90 2023-10-10T19:14:12.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a7a7fc700 1 -- 172.21.15.175:0/2654601384 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c00ba50 con 0x7f6a7c14fd90 2023-10-10T19:14:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a7a7fc700 1 -- 172.21.15.175:0/2654601384 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0092e0 con 0x7f6a7c14fd90 2023-10-10T19:14:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2654601384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c14fd90 msgr2=0x7f6a7c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2654601384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c14fd90 0x7f6a7c159c30 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c069e40 tx=0x7f6a6c009840 comp rx=0 tx=0).stop 2023-10-10T19:14:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2654601384 shutdown_connections 2023-10-10T19:14:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2654601384 >> v2:172.21.15.191:3300/0 conn(0x7f6a7c0cd040 0x7f6a7c0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2654601384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c14fd90 0x7f6a7c159c30 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2654601384 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c08a100 0x7f6a7c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2654601384 >> 172.21.15.175:0/2654601384 conn(0x7f6a7c063d00 msgr2=0x7f6a7c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2654601384 shutdown_connections 2023-10-10T19:14:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.907+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2654601384 wait complete. 2023-10-10T19:14:12.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.908+0000 7f6a80c0c700 1 Processor -- start 2023-10-10T19:14:12.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.908+0000 7f6a80c0c700 1 -- start start 2023-10-10T19:14:12.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a80c0c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c08a100 0x7f6a7c0a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:12.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a80c0c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a7c0cd040 0x7f6a7c152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:12.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a80c0c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c14fd90 0x7f6a7c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:12.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a80c0c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0a8b60 con 0x7f6a7c08a100 2023-10-10T19:14:12.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a80c0c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0c9df0 con 0x7f6a7c0cd040 2023-10-10T19:14:12.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a80c0c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c155210 con 0x7f6a7c14fd90 2023-10-10T19:14:12.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a7b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c08a100 0x7f6a7c0a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:12.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a7b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c08a100 0x7f6a7c0a7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60376/0 (socket says 172.21.15.175:60376) 2023-10-10T19:14:12.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a7b7fe700 1 -- 172.21.15.175:0/2987137180 learned_addr learned my addr 172.21.15.175:0/2987137180 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:12.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a7affd700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a7c0cd040 0x7f6a7c152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:12.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.909+0000 7f6a7bfff700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c14fd90 0x7f6a7c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:12.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a7b7fe700 1 -- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c14fd90 msgr2=0x7f6a7c14c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:12.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a7b7fe700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c14fd90 0x7f6a7c14c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:12.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a7b7fe700 1 -- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a7c0cd040 msgr2=0x7f6a7c152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:12.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a7b7fe700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a7c0cd040 0x7f6a7c152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:12.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a7b7fe700 1 -- 172.21.15.175:0/2987137180 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c009ae0 con 0x7f6a7c08a100 2023-10-10T19:14:12.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a7b7fe700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c08a100 0x7f6a7c0a7e50 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c064c20 tx=0x7f6a6c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:12.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a637fe700 1 -- 172.21.15.175:0/2987137180 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c008a80 con 0x7f6a7c08a100 2023-10-10T19:14:12.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a637fe700 1 -- 172.21.15.175:0/2987137180 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c008c00 con 0x7f6a7c08a100 2023-10-10T19:14:12.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7c0a8390 con 0x7f6a7c08a100 2023-10-10T19:14:12.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a637fe700 1 -- 172.21.15.175:0/2987137180 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c026460 con 0x7f6a7c08a100 2023-10-10T19:14:12.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.910+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7c0c9a80 con 0x7f6a7c08a100 2023-10-10T19:14:12.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.911+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7c0c98e0 con 0x7f6a7c08a100 2023-10-10T19:14:12.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.911+0000 7f6a637fe700 1 -- 172.21.15.175:0/2987137180 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6a6c003040 con 0x7f6a7c08a100 2023-10-10T19:14:12.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.912+0000 7f6a637fe700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6a68066f30 0x7f6a680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:12.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.912+0000 7f6a637fe700 1 -- 172.21.15.175:0/2987137180 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f6a6c086890 con 0x7f6a7c08a100 2023-10-10T19:14:12.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.912+0000 7f6a7affd700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6a68066f30 0x7f6a680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:12.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.912+0000 7f6a7affd700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6a68066f30 0x7f6a680693e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0696d0 tx=0x7f6a70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:12.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:12.916+0000 7f6a637fe700 1 -- 172.21.15.175:0/2987137180 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a7c0c98e0 con 0x7f6a7c08a100 2023-10-10T19:14:13.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.139+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a7c0c98e0 con 0x7f6a7c08a100 2023-10-10T19:14:13.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.139+0000 7f6a637fe700 1 -- 172.21.15.175:0/2987137180 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f6a7c0c98e0 con 0x7f6a7c08a100 2023-10-10T19:14:13.140 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:13.140 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:11.828953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:13.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6a68066f30 msgr2=0x7f6a680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:13.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6a68066f30 0x7f6a680693e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0696d0 tx=0x7f6a70008040 comp rx=0 tx=0).stop 2023-10-10T19:14:13.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c08a100 msgr2=0x7f6a7c0a7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:13.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c08a100 0x7f6a7c0a7e50 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c064c20 tx=0x7f6a6c018000 comp rx=0 tx=0).stop 2023-10-10T19:14:13.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 shutdown_connections 2023-10-10T19:14:13.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6a68066f30 0x7f6a680693e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:13.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6a7c14fd90 0x7f6a7c14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:13.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a7c08a100 0x7f6a7c0a7e50 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:13.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 --2- 172.21.15.175:0/2987137180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a7c0cd040 0x7f6a7c152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:13.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 >> 172.21.15.175:0/2987137180 conn(0x7f6a7c063d00 msgr2=0x7f6a7c158310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:13.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 shutdown_connections 2023-10-10T19:14:13.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:13.141+0000 7f6a80c0c700 1 -- 172.21.15.175:0/2987137180 wait complete. 2023-10-10T19:14:13.145 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 225 2023-10-10T19:14:13.154 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=6106 2023-10-10T19:14:13.154 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2023-10-10T19:14:15.155 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-10T19:14:15.155 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-10T19:14:15.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-10T19:14:15.159 INFO:tasks.ceph.mds.b:Started 2023-10-10T19:14:15.159 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:15.239 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-10T19:14:15.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.287+0000 7f09f4253700 1 Processor -- start 2023-10-10T19:14:15.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.288+0000 7f09f4253700 1 -- start start 2023-10-10T19:14:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.288+0000 7f09f4253700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09ec0b2390 0x7f09ec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.288+0000 7f09f4253700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0aff60 0x7f09ec0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f4253700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec1574b0 con 0x7f09ec0b1fc0 2023-10-10T19:14:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f4253700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec150770 con 0x7f09ec0b2390 2023-10-10T19:14:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f4253700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0afad0 con 0x7f09ec0aff60 2023-10-10T19:14:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f1fef700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f09ec0b1fc0 0x7f09ec068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45202/0 (socket says 172.21.15.175:45202) 2023-10-10T19:14:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f1fef700 1 -- 172.21.15.175:0/1398310205 learned_addr learned my addr 172.21.15.175:0/1398310205 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f17ee700 1 --2- 172.21.15.175:0/1398310205 >> v2:172.21.15.191:3300/0 conn(0x7f09ec0b2390 0x7f09ec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:15.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f27f0700 1 --2- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0aff60 0x7f09ec0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:15.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f0fed700 1 -- 172.21.15.175:0/1398310205 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678122648 0 0) 0x7f09ec1574b0 con 0x7f09ec0b1fc0 2023-10-10T19:14:15.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f0fed700 1 -- 172.21.15.175:0/1398310205 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d8003570 con 0x7f09ec0b1fc0 2023-10-10T19:14:15.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f17ee700 1 -- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0aff60 msgr2=0x7f09ec0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:15.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f17ee700 1 --2- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0aff60 0x7f09ec0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f0fed700 1 -- 172.21.15.175:0/1398310205 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27553425 0 0) 0x7f09d8003570 con 0x7f09ec0b1fc0 2023-10-10T19:14:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f17ee700 1 -- 172.21.15.175:0/1398310205 >> v1:172.21.15.175:6789/0 conn(0x7f09ec0b1fc0 legacy=0x7f09ec068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f17ee700 1 -- 172.21.15.175:0/1398310205 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec0af4c0 con 0x7f09ec0b2390 2023-10-10T19:14:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f27f0700 1 --2- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0aff60 0x7f09ec0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:14:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f17ee700 1 --2- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 0x7f09ec155fb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0098e0 tx=0x7f09dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47911793ef02dcca server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.289+0000 7f09f0fed700 1 -- 172.21.15.175:0/1398310205 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e80017f0 con 0x7f09ec0b2390 2023-10-10T19:14:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f0fed700 1 -- 172.21.15.175:0/1398310205 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc011420 con 0x7f09ec0b2390 2023-10-10T19:14:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f0fed700 1 -- 172.21.15.175:0/1398310205 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc0117c0 con 0x7f09ec0b2390 2023-10-10T19:14:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f4253700 1 -- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 msgr2=0x7f09ec155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f4253700 1 --2- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 0x7f09ec155fb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0098e0 tx=0x7f09dc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f4253700 1 -- 172.21.15.175:0/1398310205 shutdown_connections 2023-10-10T19:14:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f4253700 1 --2- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0aff60 0x7f09ec0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f4253700 1 --2- 172.21.15.175:0/1398310205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 0x7f09ec155fb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f4253700 1 -- 172.21.15.175:0/1398310205 >> 172.21.15.175:0/1398310205 conn(0x7f09ec063230 msgr2=0x7f09ec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f4253700 1 -- 172.21.15.175:0/1398310205 shutdown_connections 2023-10-10T19:14:15.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.290+0000 7f09f4253700 1 -- 172.21.15.175:0/1398310205 wait complete. 2023-10-10T19:14:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.291+0000 7f09f4253700 1 Processor -- start 2023-10-10T19:14:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.291+0000 7f09f4253700 1 -- start start 2023-10-10T19:14:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f4253700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09ec0aff60 0x7f09ec15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:15.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f4253700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0b1fc0 0x7f09ec0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f4253700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 0x7f09ec0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f4253700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec166950 con 0x7f09ec0aff60 2023-10-10T19:14:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f4253700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec151210 con 0x7f09ec0b2390 2023-10-10T19:14:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f4253700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec150f70 con 0x7f09ec0b1fc0 2023-10-10T19:14:15.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f1fef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09ec0aff60 0x7f09ec15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f1fef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09ec0aff60 0x7f09ec15b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46830/0 (socket says 172.21.15.175:46830) 2023-10-10T19:14:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f1fef700 1 -- 172.21.15.175:0/857175514 learned_addr learned my addr 172.21.15.175:0/857175514 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f17ee700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0b1fc0 0x7f09ec0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:15.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f27f0700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 0x7f09ec0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f1fef700 1 -- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0b1fc0 msgr2=0x7f09ec0cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f1fef700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0b1fc0 0x7f09ec0cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f1fef700 1 -- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 msgr2=0x7f09ec0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f1fef700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 0x7f09ec0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.292+0000 7f09f1fef700 1 -- 172.21.15.175:0/857175514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4009420 con 0x7f09ec0aff60 2023-10-10T19:14:15.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.293+0000 7f09f1fef700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09ec0aff60 0x7f09ec15b0c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f09e800ccd0 tx=0x7f09e80016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:15.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.293+0000 7f09e1ffb700 1 -- 172.21.15.175:0/857175514 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8004dc0 con 0x7f09ec0aff60 2023-10-10T19:14:15.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.293+0000 7f09e1ffb700 1 -- 172.21.15.175:0/857175514 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e800a770 con 0x7f09ec0aff60 2023-10-10T19:14:15.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.293+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc009420 con 0x7f09ec0aff60 2023-10-10T19:14:15.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.293+0000 7f09e1ffb700 1 -- 172.21.15.175:0/857175514 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e800ab30 con 0x7f09ec0aff60 2023-10-10T19:14:15.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.293+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec0cd2b0 con 0x7f09ec0aff60 2023-10-10T19:14:15.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.294+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec067570 con 0x7f09ec0aff60 2023-10-10T19:14:15.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.294+0000 7f09e1ffb700 1 -- 172.21.15.175:0/857175514 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f09e8007150 con 0x7f09ec0aff60 2023-10-10T19:14:15.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.295+0000 7f09e1ffb700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f09d0066f30 0x7f09d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:15.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.295+0000 7f09e1ffb700 1 -- 172.21.15.175:0/857175514 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f09e8084570 con 0x7f09ec0aff60 2023-10-10T19:14:15.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.295+0000 7f09f17ee700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f09d0066f30 0x7f09d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:15.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.295+0000 7f09f17ee700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f09d0066f30 0x7f09d00693e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0093f0 tx=0x7f09dc016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:15.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.299+0000 7f09e1ffb700 1 -- 172.21.15.175:0/857175514 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09e8051320 con 0x7f09ec0aff60 2023-10-10T19:14:15.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.522+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09ec068da0 con 0x7f09ec0aff60 2023-10-10T19:14:15.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.522+0000 7f09e1ffb700 1 -- 172.21.15.175:0/857175514 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5448 (secure 0 0 0) 0x7f09e8054940 con 0x7f09ec0aff60 2023-10-10T19:14:15.523 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:15.524 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:11.828953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:15.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f09d0066f30 msgr2=0x7f09d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:15.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f09d0066f30 0x7f09d00693e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0093f0 tx=0x7f09dc016000 comp rx=0 tx=0).stop 2023-10-10T19:14:15.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09ec0aff60 msgr2=0x7f09ec15b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:15.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09ec0aff60 0x7f09ec15b0c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f09e800ccd0 tx=0x7f09e80016f0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 shutdown_connections 2023-10-10T19:14:15.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f09d0066f30 0x7f09d00693e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09ec0b2390 0x7f09ec0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09ec0aff60 0x7f09ec15b0c0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.524+0000 7f09f4253700 1 --2- 172.21.15.175:0/857175514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09ec0b1fc0 0x7f09ec0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:15.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.525+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 >> 172.21.15.175:0/857175514 conn(0x7f09ec063230 msgr2=0x7f09ec15a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:15.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.525+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 shutdown_connections 2023-10-10T19:14:15.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:15.525+0000 7f09f4253700 1 -- 172.21.15.175:0/857175514 wait complete. 2023-10-10T19:14:15.528 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 225 2023-10-10T19:14:15.537 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-10T19:14:17.538 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:17.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.664+0000 7f33349ea700 1 Processor -- start 2023-10-10T19:14:17.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.665+0000 7f33349ea700 1 -- start start 2023-10-10T19:14:17.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.665+0000 7f33349ea700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f333006b640 0x7f333014cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:17.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.665+0000 7f33349ea700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33301533e0 con 0x7f33300c3330 2023-10-10T19:14:17.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.665+0000 7f33349ea700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333006e130 con 0x7f33300c3700 2023-10-10T19:14:17.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.665+0000 7f33349ea700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333006a970 con 0x7f333006b640 2023-10-10T19:14:17.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.665+0000 7f332e59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f33300c3330 0x7f33300c3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45210/0 (socket says 172.21.15.175:45210) 2023-10-10T19:14:17.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.665+0000 7f332e59c700 1 -- 172.21.15.175:0/3303406321 learned_addr learned my addr 172.21.15.175:0/3303406321 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:17.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f3325d9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f333006b640 0x7f333014cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:17.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806903666 0 0) 0x7f33301533e0 con 0x7f33300c3330 2023-10-10T19:14:17.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f331c003570 con 0x7f33300c3330 2023-10-10T19:14:17.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402434614 0 0) 0x7f333006e130 con 0x7f33300c3700 2023-10-10T19:14:17.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33301533e0 con 0x7f33300c3700 2023-10-10T19:14:17.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2735351184 0 0) 0x7f331c003570 con 0x7f33300c3330 2023-10-10T19:14:17.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f333006b640 msgr2=0x7f333014cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:17.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f333006b640 0x7f333014cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 >> v1:172.21.15.191:6789/0 conn(0x7f33300c3700 legacy=0x7f333015d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:17.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33300c43b0 con 0x7f33300c3330 2023-10-10T19:14:17.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f33200015f0 con 0x7f33300c3330 2023-10-10T19:14:17.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 >> v1:172.21.15.175:6789/0 conn(0x7f33300c3330 legacy=0x7f33300c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331c003c20 0x7f331c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f331c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f333006b640 0x7f331c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333006a970 con 0x7f331c004810 2023-10-10T19:14:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c00bb40 con 0x7f331c003c20 2023-10-10T19:14:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c00bd00 con 0x7f333006b640 2023-10-10T19:14:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f3325d9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f331c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332e59c700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331c003c20 0x7f331c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.666+0000 7f332ed9d700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f333006b640 0x7f331c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f3325d9b700 1 -- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f333006b640 msgr2=0x7f331c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f3325d9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f333006b640 0x7f331c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f3325d9b700 1 -- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331c003c20 msgr2=0x7f331c004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f3325d9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331c003c20 0x7f331c004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f3325d9b700 1 -- 172.21.15.175:0/3303406321 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c003570 con 0x7f331c004810 2023-10-10T19:14:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f3325d9b700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f331c004c80 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f333006ba10 tx=0x7f33180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ba2b3b4bf0ac1c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:17.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331800e710 con 0x7f331c004810 2023-10-10T19:14:17.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331800e870 con 0x7f331c004810 2023-10-10T19:14:17.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.667+0000 7f332dd9b700 1 -- 172.21.15.175:0/3303406321 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331800ec30 con 0x7f331c004810 2023-10-10T19:14:17.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.668+0000 7f33349ea700 1 -- 172.21.15.175:0/3303406321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 msgr2=0x7f331c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:17.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.668+0000 7f33349ea700 1 --2- 172.21.15.175:0/3303406321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f331c004c80 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f333006ba10 tx=0x7f33180091a0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.668+0000 7f332ed9d700 1 -- 172.21.15.175:0/3303406321 reap_dead start 2023-10-10T19:14:17.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.668+0000 7f33349ea700 1 -- 172.21.15.175:0/3303406321 shutdown_connections 2023-10-10T19:14:17.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.668+0000 7f33349ea700 1 -- 172.21.15.175:0/3303406321 >> 172.21.15.175:0/3303406321 conn(0x7f3330068280 msgr2=0x7f333006af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:17.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.668+0000 7f33349ea700 1 -- 172.21.15.175:0/3303406321 shutdown_connections 2023-10-10T19:14:17.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.668+0000 7f33349ea700 1 -- 172.21.15.175:0/3303406321 wait complete. 2023-10-10T19:14:17.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.669+0000 7f33349ea700 1 Processor -- start 2023-10-10T19:14:17.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.669+0000 7f33349ea700 1 -- start start 2023-10-10T19:14:17.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f33349ea700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f3330164a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:17.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f33349ea700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33300c1660 0x7f33300697d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:17.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f33349ea700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3330069d10 0x7f3330167640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:17.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f33349ea700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3330164590 con 0x7f331c004810 2023-10-10T19:14:17.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f33349ea700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333014c430 con 0x7f33300c1660 2023-10-10T19:14:17.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f33349ea700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333006a780 con 0x7f3330069d10 2023-10-10T19:14:17.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f3330164a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:17.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f3330164a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46842/0 (socket says 172.21.15.175:46842) 2023-10-10T19:14:17.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 -- 172.21.15.175:0/2643717947 learned_addr learned my addr 172.21.15.175:0/2643717947 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:17.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332ed9d700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3330069d10 0x7f3330167640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:17.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f3325d9b700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33300c1660 0x7f33300697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:17.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 -- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3330069d10 msgr2=0x7f3330167640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:17.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3330069d10 0x7f3330167640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 -- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33300c1660 msgr2=0x7f33300697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:17.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33300c1660 0x7f33300697d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 -- 172.21.15.175:0/2643717947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330164260 con 0x7f331c004810 2023-10-10T19:14:17.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f332e59c700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f3330164a20 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f332000fc00 tx=0x7f3320009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:17.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f3326ffd700 1 -- 172.21.15.175:0/2643717947 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33200072e0 con 0x7f331c004810 2023-10-10T19:14:17.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f3326ffd700 1 -- 172.21.15.175:0/2643717947 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3320007440 con 0x7f331c004810 2023-10-10T19:14:17.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333006a1c0 con 0x7f331c004810 2023-10-10T19:14:17.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f3326ffd700 1 -- 172.21.15.175:0/2643717947 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33200057f0 con 0x7f331c004810 2023-10-10T19:14:17.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.670+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3330164f60 con 0x7f331c004810 2023-10-10T19:14:17.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.671+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33301640c0 con 0x7f331c004810 2023-10-10T19:14:17.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.672+0000 7f3326ffd700 1 -- 172.21.15.175:0/2643717947 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3320005c00 con 0x7f331c004810 2023-10-10T19:14:17.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.672+0000 7f3326ffd700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3310066f30 0x7f33100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:17.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.672+0000 7f3326ffd700 1 -- 172.21.15.175:0/2643717947 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f3320084bf0 con 0x7f331c004810 2023-10-10T19:14:17.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.672+0000 7f3325d9b700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3310066f30 0x7f33100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:17.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.673+0000 7f3325d9b700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3310066f30 0x7f33100693e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f33180091e0 tx=0x7f331801a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:17.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.676+0000 7f3326ffd700 1 -- 172.21.15.175:0/2643717947 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33301640c0 con 0x7f331c004810 2023-10-10T19:14:17.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.907+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33301640c0 con 0x7f331c004810 2023-10-10T19:14:17.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.907+0000 7f3326ffd700 1 -- 172.21.15.175:0/2643717947 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) v1 ==== 95+0+6208 (secure 0 0 0) 0x7f33301640c0 con 0x7f331c004810 2023-10-10T19:14:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:17.908 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:11.828953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:17.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3310066f30 msgr2=0x7f33100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:17.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3310066f30 0x7f33100693e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f33180091e0 tx=0x7f331801a000 comp rx=0 tx=0).stop 2023-10-10T19:14:17.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 msgr2=0x7f3330164a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:17.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f3330164a20 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f332000fc00 tx=0x7f3320009040 comp rx=0 tx=0).stop 2023-10-10T19:14:17.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 shutdown_connections 2023-10-10T19:14:17.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3310066f30 0x7f33100693e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3330069d10 0x7f3330167640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f331c004810 0x7f3330164a20 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 --2- 172.21.15.175:0/2643717947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33300c1660 0x7f33300697d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:17.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 >> 172.21.15.175:0/2643717947 conn(0x7f3330068280 msgr2=0x7f3330153590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:17.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 shutdown_connections 2023-10-10T19:14:17.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:17.909+0000 7f33349ea700 1 -- 172.21.15.175:0/2643717947 wait complete. 2023-10-10T19:14:17.912 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 226 2023-10-10T19:14:17.922 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-10T19:14:17.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-10-10T19:14:23.303 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:14:23.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:14:24.762 INFO:tasks.workunit.client.0.smithi175.stderr:+ cd ../.. 2023-10-10T19:14:24.762 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.763 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.1 2023-10-10T19:14:24.763 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.1 2023-10-10T19:14:24.771 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.771 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.2 2023-10-10T19:14:24.772 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.2 2023-10-10T19:14:24.778 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.778 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.3 2023-10-10T19:14:24.779 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.3 2023-10-10T19:14:24.785 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.785 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.4 2023-10-10T19:14:24.785 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.4 2023-10-10T19:14:24.791 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.792 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.5 2023-10-10T19:14:24.792 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.5 2023-10-10T19:14:24.798 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.799 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.6 2023-10-10T19:14:24.799 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.6 2023-10-10T19:14:24.805 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.806 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.7 2023-10-10T19:14:24.806 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.7 2023-10-10T19:14:24.812 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.813 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.8 2023-10-10T19:14:24.813 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.8 2023-10-10T19:14:24.819 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.819 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.9 2023-10-10T19:14:24.819 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.9 2023-10-10T19:14:24.827 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.828 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.10 2023-10-10T19:14:24.828 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.10 2023-10-10T19:14:24.833 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.834 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.11 2023-10-10T19:14:24.834 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.11 2023-10-10T19:14:24.840 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.840 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.12 2023-10-10T19:14:24.840 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.12 2023-10-10T19:14:24.846 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.846 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.13 2023-10-10T19:14:24.847 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.13 2023-10-10T19:14:24.852 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.852 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.14 2023-10-10T19:14:24.852 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.14 2023-10-10T19:14:24.858 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.858 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.15 2023-10-10T19:14:24.859 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.15 2023-10-10T19:14:24.864 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:24.865 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.16 2023-10-10T19:14:24.865 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.16 2023-10-10T19:14:25.378 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:25.378 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.17 2023-10-10T19:14:25.378 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.17 2023-10-10T19:14:25.387 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:25.387 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.18 2023-10-10T19:14:25.387 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.18 2023-10-10T19:14:25.394 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:25.395 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.19 2023-10-10T19:14:25.395 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.19 2023-10-10T19:14:26.146 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:26.146 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.20 2023-10-10T19:14:26.147 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.20 2023-10-10T19:14:26.846 INFO:tasks.workunit.client.0.smithi175.stderr:+ for v in $versions 2023-10-10T19:14:26.847 INFO:tasks.workunit.client.0.smithi175.stderr:+ ver=v0.21 2023-10-10T19:14:26.847 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/v0.21 2023-10-10T19:14:26.860 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:14:26.861 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T19:14:26.861 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:14:26.861 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:14:26.924 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:27.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.041+0000 7f5ef754f700 1 Processor -- start 2023-10-10T19:14:27.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef754f700 1 -- start start 2023-10-10T19:14:27.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef754f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5ef014cdc0 0x7f5ef00c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef754f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5ef006f620 0x7f5ef00c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef754f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef006e940 con 0x7f5ef014d190 2023-10-10T19:14:27.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef754f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef00c1c90 con 0x7f5ef014cdc0 2023-10-10T19:14:27.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef754f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef00c6e20 con 0x7f5ef006f620 2023-10-10T19:14:27.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef52eb700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5ef014cdc0 0x7f5ef00c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef4aea700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5ef014d190 0x7f5ef0156300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33112/0 (socket says 172.21.15.175:33112) 2023-10-10T19:14:27.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.042+0000 7f5ef4aea700 1 -- 172.21.15.175:0/103997298 learned_addr learned my addr 172.21.15.175:0/103997298 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:27.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef5aec700 1 --2- 172.21.15.175:0/103997298 >> v2:172.21.15.191:3300/0 conn(0x7f5ef006f620 0x7f5ef00c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef52eb700 1 -- 172.21.15.175:0/103997298 >> v2:172.21.15.191:3300/0 conn(0x7f5ef006f620 msgr2=0x7f5ef00c2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:27.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef52eb700 1 --2- 172.21.15.175:0/103997298 >> v2:172.21.15.191:3300/0 conn(0x7f5ef006f620 0x7f5ef00c2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef52eb700 1 -- 172.21.15.175:0/103997298 >> v1:172.21.15.175:6789/0 conn(0x7f5ef014d190 legacy=0x7f5ef0156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ee7fff700 1 -- 172.21.15.175:0/103997298 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225395624 0 0) 0x7f5ef006e940 con 0x7f5ef014d190 2023-10-10T19:14:27.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef52eb700 1 -- 172.21.15.175:0/103997298 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef006ca40 con 0x7f5ef014cdc0 2023-10-10T19:14:27.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef52eb700 1 --2- 172.21.15.175:0/103997298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef00c26e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0098d0 tx=0x7f5eec009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a391642cae7221b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:27.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ee7fff700 1 -- 172.21.15.175:0/103997298 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec009e70 con 0x7f5ef014cdc0 2023-10-10T19:14:27.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ee7fff700 1 -- 172.21.15.175:0/103997298 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec005c10 con 0x7f5ef014cdc0 2023-10-10T19:14:27.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ee7fff700 1 -- 172.21.15.175:0/103997298 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec00b610 con 0x7f5ef014cdc0 2023-10-10T19:14:27.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef754f700 1 -- 172.21.15.175:0/103997298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 msgr2=0x7f5ef00c26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef754f700 1 --2- 172.21.15.175:0/103997298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef00c26e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0098d0 tx=0x7f5eec009120 comp rx=0 tx=0).stop 2023-10-10T19:14:27.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef754f700 1 -- 172.21.15.175:0/103997298 shutdown_connections 2023-10-10T19:14:27.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef754f700 1 --2- 172.21.15.175:0/103997298 >> v2:172.21.15.191:3300/0 conn(0x7f5ef006f620 0x7f5ef00c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.043+0000 7f5ef754f700 1 --2- 172.21.15.175:0/103997298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef00c26e0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.044+0000 7f5ef754f700 1 -- 172.21.15.175:0/103997298 >> 172.21.15.175:0/103997298 conn(0x7f5ef0068900 msgr2=0x7f5ef0068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:27.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.044+0000 7f5ef754f700 1 -- 172.21.15.175:0/103997298 shutdown_connections 2023-10-10T19:14:27.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.044+0000 7f5ef754f700 1 -- 172.21.15.175:0/103997298 wait complete. 2023-10-10T19:14:27.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.045+0000 7f5ef754f700 1 Processor -- start 2023-10-10T19:14:27.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.045+0000 7f5ef754f700 1 -- start start 2023-10-10T19:14:27.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.045+0000 7f5ef754f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ef006f620 0x7f5ef014aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.045+0000 7f5ef754f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef0150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.045+0000 7f5ef754f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ef014d190 0x7f5ef0164e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.045+0000 7f5ef754f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00c0320 con 0x7f5ef014cdc0 2023-10-10T19:14:27.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.045+0000 7f5ef754f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01646d0 con 0x7f5ef006f620 2023-10-10T19:14:27.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.045+0000 7f5ef754f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00694c0 con 0x7f5ef014d190 2023-10-10T19:14:27.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef0150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef0150480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57408/0 (socket says 172.21.15.175:57408) 2023-10-10T19:14:27.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 -- 172.21.15.175:0/133691571 learned_addr learned my addr 172.21.15.175:0/133691571 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:27.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef52eb700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ef006f620 0x7f5ef014aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef5aec700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ef014d190 0x7f5ef0164e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 -- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ef014d190 msgr2=0x7f5ef0164e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ef014d190 0x7f5ef0164e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 -- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ef006f620 msgr2=0x7f5ef014aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ef006f620 0x7f5ef014aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 -- 172.21.15.175:0/133691571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec0093c0 con 0x7f5ef014cdc0 2023-10-10T19:14:27.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef4aea700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef0150480 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000c5e0 tx=0x7f5ee0000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:27.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ee57fa700 1 -- 172.21.15.175:0/133691571 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0005b30 con 0x7f5ef014cdc0 2023-10-10T19:14:27.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ee57fa700 1 -- 172.21.15.175:0/133691571 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0005c90 con 0x7f5ef014cdc0 2023-10-10T19:14:27.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ee57fa700 1 -- 172.21.15.175:0/133691571 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0007300 con 0x7f5ef014cdc0 2023-10-10T19:14:27.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef0164970 con 0x7f5ef014cdc0 2023-10-10T19:14:27.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.046+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef0166ab0 con 0x7f5ef014cdc0 2023-10-10T19:14:27.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.047+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef0067570 con 0x7f5ef014cdc0 2023-10-10T19:14:27.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.047+0000 7f5ee57fa700 1 -- 172.21.15.175:0/133691571 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5ee001b3b0 con 0x7f5ef014cdc0 2023-10-10T19:14:27.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.048+0000 7f5ee57fa700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.048+0000 7f5ee57fa700 1 -- 172.21.15.175:0/133691571 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f5ee0083eb0 con 0x7f5ef014cdc0 2023-10-10T19:14:27.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.048+0000 7f5ef52eb700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.049+0000 7f5ef52eb700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009160 tx=0x7f5eec016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:27.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.052+0000 7f5ee57fa700 1 -- 172.21.15.175:0/133691571 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ee0050ba0 con 0x7f5ef014cdc0 2023-10-10T19:14:27.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.285+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ef00c5100 con 0x7f5ef014cdc0 2023-10-10T19:14:27.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.285+0000 7f5ee57fa700 1 -- 172.21.15.175:0/133691571 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+6207 (secure 0 0 0) 0x7f5ee0017090 con 0x7f5ef014cdc0 2023-10-10T19:14:27.286 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:27.286 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:24.730718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:27.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 msgr2=0x7f5ed40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009160 tx=0x7f5eec016000 comp rx=0 tx=0).stop 2023-10-10T19:14:27.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 msgr2=0x7f5ef0150480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef0150480 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000c5e0 tx=0x7f5ee0000fe0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 shutdown_connections 2023-10-10T19:14:27.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5ed4066f30 0x7f5ed40693e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ef014d190 0x7f5ef0164e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ef006f620 0x7f5ef014aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 --2- 172.21.15.175:0/133691571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ef014cdc0 0x7f5ef0150480 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 >> 172.21.15.175:0/133691571 conn(0x7f5ef0068900 msgr2=0x7f5ef00c44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:27.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 shutdown_connections 2023-10-10T19:14:27.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.287+0000 7f5ef754f700 1 -- 172.21.15.175:0/133691571 wait complete. 2023-10-10T19:14:27.292 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 227 2023-10-10T19:14:27.300 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-10T19:14:27.300 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-10T19:14:27.300 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:14:27.311 INFO:tasks.ceph.mds.a:Stopped 2023-10-10T19:14:27.311 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:27.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.458+0000 7ff6f8ef0700 1 Processor -- start 2023-10-10T19:14:27.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.458+0000 7ff6f8ef0700 1 -- start start 2023-10-10T19:14:27.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f8ef0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff6f414fd90 0x7ff6f4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f8ef0700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f4067fa0 con 0x7ff6f408a100 2023-10-10T19:14:27.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f8ef0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f4156b70 con 0x7ff6f40cd040 2023-10-10T19:14:27.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f8ef0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f40c5600 con 0x7ff6f414fd90 2023-10-10T19:14:27.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f2ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff6f408a100 0x7ff6f40c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33122/0 (socket says 172.21.15.175:33122) 2023-10-10T19:14:27.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f2ffd700 1 -- 172.21.15.175:0/3904906378 learned_addr learned my addr 172.21.15.175:0/3904906378 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:27.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f37fe700 1 --2- 172.21.15.175:0/3904906378 >> v2:172.21.15.191:3300/0 conn(0x7ff6f414fd90 0x7ff6f4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f27fc700 1 -- 172.21.15.175:0/3904906378 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013051709 0 0) 0x7ff6f4067fa0 con 0x7ff6f408a100 2023-10-10T19:14:27.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f27fc700 1 -- 172.21.15.175:0/3904906378 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e0003570 con 0x7ff6f408a100 2023-10-10T19:14:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f37fe700 1 -- 172.21.15.175:0/3904906378 >> v1:172.21.15.191:6789/0 conn(0x7ff6f40cd040 legacy=0x7ff6f40684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f27fc700 1 -- 172.21.15.175:0/3904906378 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626803718 0 0) 0x7ff6e0003570 con 0x7ff6f408a100 2023-10-10T19:14:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f37fe700 1 -- 172.21.15.175:0/3904906378 >> v1:172.21.15.175:6789/0 conn(0x7ff6f408a100 legacy=0x7ff6f40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.459+0000 7ff6f37fe700 1 -- 172.21.15.175:0/3904906378 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f4150160 con 0x7ff6f414fd90 2023-10-10T19:14:27.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f37fe700 1 --2- 172.21.15.175:0/3904906378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 0x7ff6f4159c30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0098e0 tx=0x7ff6dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6f666a8f9c4f0b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f27fc700 1 -- 172.21.15.175:0/3904906378 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4001470 con 0x7ff6f414fd90 2023-10-10T19:14:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f27fc700 1 -- 172.21.15.175:0/3904906378 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc011420 con 0x7ff6f414fd90 2023-10-10T19:14:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f27fc700 1 -- 172.21.15.175:0/3904906378 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0117c0 con 0x7ff6f414fd90 2023-10-10T19:14:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/3904906378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 msgr2=0x7ff6f4159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f8ef0700 1 --2- 172.21.15.175:0/3904906378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 0x7ff6f4159c30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0098e0 tx=0x7ff6dc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/3904906378 shutdown_connections 2023-10-10T19:14:27.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f8ef0700 1 --2- 172.21.15.175:0/3904906378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 0x7ff6f4159c30 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/3904906378 >> 172.21.15.175:0/3904906378 conn(0x7ff6f4063d00 msgr2=0x7ff6f40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:27.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/3904906378 shutdown_connections 2023-10-10T19:14:27.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.460+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/3904906378 wait complete. 2023-10-10T19:14:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.461+0000 7ff6f8ef0700 1 Processor -- start 2023-10-10T19:14:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f8ef0700 1 -- start start 2023-10-10T19:14:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f8ef0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6f408a100 0x7ff6f4158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f8ef0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6f40cd040 0x7ff6f4157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f8ef0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 0x7ff6f40b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f8ef0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40a8620 con 0x7ff6f408a100 2023-10-10T19:14:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f8ef0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4166390 con 0x7ff6f414fd90 2023-10-10T19:14:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f8ef0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4166650 con 0x7ff6f40cd040 2023-10-10T19:14:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6f408a100 0x7ff6f4158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6f408a100 0x7ff6f4158930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57414/0 (socket says 172.21.15.175:57414) 2023-10-10T19:14:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f37fe700 1 -- 172.21.15.175:0/1338557147 learned_addr learned my addr 172.21.15.175:0/1338557147 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f2ffd700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6f40cd040 0x7ff6f4157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.462+0000 7ff6f3fff700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 0x7ff6f40b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6f37fe700 1 -- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6f40cd040 msgr2=0x7ff6f4157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6f37fe700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6f40cd040 0x7ff6f4157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6f37fe700 1 -- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 msgr2=0x7ff6f40b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6f37fe700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 0x7ff6f40b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6f37fe700 1 -- 172.21.15.175:0/1338557147 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc009420 con 0x7ff6f408a100 2023-10-10T19:14:27.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6f37fe700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6f408a100 0x7ff6f4158930 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0030c0 tx=0x7ff6dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:27.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6eb7fe700 1 -- 172.21.15.175:0/1338557147 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc010ac0 con 0x7ff6f408a100 2023-10-10T19:14:27.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6eb7fe700 1 -- 172.21.15.175:0/1338557147 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc010c40 con 0x7ff6f408a100 2023-10-10T19:14:27.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f4166930 con 0x7ff6f408a100 2023-10-10T19:14:27.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6eb7fe700 1 -- 172.21.15.175:0/1338557147 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc01d430 con 0x7ff6f408a100 2023-10-10T19:14:27.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.463+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f4166ef0 con 0x7ff6f408a100 2023-10-10T19:14:27.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.464+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f4067570 con 0x7ff6f408a100 2023-10-10T19:14:27.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.464+0000 7ff6eb7fe700 1 -- 172.21.15.175:0/1338557147 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff6dc01d7a0 con 0x7ff6f408a100 2023-10-10T19:14:27.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.465+0000 7ff6eb7fe700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff6d4066f60 0x7ff6d4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:27.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.465+0000 7ff6eb7fe700 1 -- 172.21.15.175:0/1338557147 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7111+0+0 (secure 0 0 0) 0x7ff6dc086470 con 0x7ff6f408a100 2023-10-10T19:14:27.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.465+0000 7ff6f2ffd700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff6d4066f60 0x7ff6d4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:27.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.465+0000 7ff6f2ffd700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff6d4066f60 0x7ff6d4069410 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4160bb0 tx=0x7ff6e4000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:27.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.469+0000 7ff6eb7fe700 1 -- 172.21.15.175:0/1338557147 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff6dc053160 con 0x7ff6f408a100 2023-10-10T19:14:27.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.704+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f4150320 con 0x7ff6f408a100 2023-10-10T19:14:27.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.704+0000 7ff6eb7fe700 1 -- 172.21.15.175:0/1338557147 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+6207 (secure 0 0 0) 0x7ff6dc056780 con 0x7ff6f408a100 2023-10-10T19:14:27.705 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:27.705 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:24.730718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:27.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff6d4066f60 msgr2=0x7ff6d4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff6d4066f60 0x7ff6d4069410 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4160bb0 tx=0x7ff6e4000bc0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6f408a100 msgr2=0x7ff6f4158930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:27.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6f408a100 0x7ff6f4158930 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0030c0 tx=0x7ff6dc017000 comp rx=0 tx=0).stop 2023-10-10T19:14:27.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 shutdown_connections 2023-10-10T19:14:27.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff6d4066f60 0x7ff6d4069410 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6f414fd90 0x7ff6f40b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6f408a100 0x7ff6f4158930 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 --2- 172.21.15.175:0/1338557147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6f40cd040 0x7ff6f4157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:27.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.706+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 >> 172.21.15.175:0/1338557147 conn(0x7ff6f4063d00 msgr2=0x7ff6f40c71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:27.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.707+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 shutdown_connections 2023-10-10T19:14:27.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:27.707+0000 7ff6f8ef0700 1 -- 172.21.15.175:0/1338557147 wait complete. 2023-10-10T19:14:27.709 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 227 2023-10-10T19:14:27.719 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:14:29.720 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:29.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.903+0000 7f4331e1b700 1 Processor -- start 2023-10-10T19:14:29.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.903+0000 7f4331e1b700 1 -- start start 2023-10-10T19:14:29.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.904+0000 7f4331e1b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f432c069d90 0x7f432c06b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:29.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.904+0000 7f4331e1b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c06d3f0 0x7f432c14bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:29.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.904+0000 7f4331e1b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432c153180 con 0x7f432c0c3870 2023-10-10T19:14:29.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.904+0000 7f4331e1b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c06bd60 con 0x7f432c069d90 2023-10-10T19:14:29.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.904+0000 7f4331e1b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c152570 con 0x7f432c06d3f0 2023-10-10T19:14:29.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.904+0000 7f432affd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f432c069d90 0x7f432c06b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:29.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.904+0000 7f432affd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f432c069d90 0x7f432c06b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:56326/0 (socket says 172.21.15.175:56326) 2023-10-10T19:14:29.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.904+0000 7f432affd700 1 -- 172.21.15.175:0/736215338 learned_addr learned my addr 172.21.15.175:0/736215338 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:29.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.905+0000 7f432bfff700 1 --2- 172.21.15.175:0/736215338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c06d3f0 0x7f432c14bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:29.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.905+0000 7f432affd700 1 -- 172.21.15.175:0/736215338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c06d3f0 msgr2=0x7f432c14bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:29.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.905+0000 7f432affd700 1 --2- 172.21.15.175:0/736215338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c06d3f0 0x7f432c14bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:29.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.905+0000 7f432affd700 1 -- 172.21.15.175:0/736215338 >> v1:172.21.15.191:6789/0 conn(0x7f432c0c3870 legacy=0x7f432c069680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:29.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.905+0000 7f432affd700 1 -- 172.21.15.175:0/736215338 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c152850 con 0x7f432c069d90 2023-10-10T19:14:29.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.905+0000 7f432affd700 1 --2- 172.21.15.175:0/736215338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c069d90 0x7f432c06b6b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f431c0099b0 tx=0x7f431c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28cdd2d71e63a44d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:29.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.906+0000 7f432a7fc700 1 -- 172.21.15.175:0/736215338 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00b860 con 0x7f432c069d90 2023-10-10T19:14:29.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.906+0000 7f432a7fc700 1 -- 172.21.15.175:0/736215338 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c00b9c0 con 0x7f432c069d90 2023-10-10T19:14:29.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.906+0000 7f432a7fc700 1 -- 172.21.15.175:0/736215338 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00bd60 con 0x7f432c069d90 2023-10-10T19:14:29.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.907+0000 7f4331e1b700 1 -- 172.21.15.175:0/736215338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c069d90 msgr2=0x7f432c06b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:29.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.907+0000 7f4331e1b700 1 --2- 172.21.15.175:0/736215338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c069d90 0x7f432c06b6b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f431c0099b0 tx=0x7f431c009250 comp rx=0 tx=0).stop 2023-10-10T19:14:29.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.907+0000 7f4331e1b700 1 -- 172.21.15.175:0/736215338 shutdown_connections 2023-10-10T19:14:29.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.907+0000 7f4331e1b700 1 --2- 172.21.15.175:0/736215338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c06d3f0 0x7f432c14bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:29.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.907+0000 7f4331e1b700 1 --2- 172.21.15.175:0/736215338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c069d90 0x7f432c06b6b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:29.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.907+0000 7f4331e1b700 1 -- 172.21.15.175:0/736215338 >> 172.21.15.175:0/736215338 conn(0x7f432c068280 msgr2=0x7f432c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:29.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.908+0000 7f4331e1b700 1 -- 172.21.15.175:0/736215338 shutdown_connections 2023-10-10T19:14:29.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.909+0000 7f4331e1b700 1 -- 172.21.15.175:0/736215338 wait complete. 2023-10-10T19:14:29.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.910+0000 7f4331e1b700 1 Processor -- start 2023-10-10T19:14:29.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.910+0000 7f4331e1b700 1 -- start start 2023-10-10T19:14:29.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f4331e1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f432c069d90 0x7f432c151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:29.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f4331e1b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c06d3f0 0x7f432c150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:29.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f4331e1b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c0c3870 0x7f432c14cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:29.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f4331e1b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c14f710 con 0x7f432c069d90 2023-10-10T19:14:29.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f4331e1b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c1409b0 con 0x7f432c06d3f0 2023-10-10T19:14:29.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f4331e1b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c141070 con 0x7f432c0c3870 2023-10-10T19:14:29.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f432b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f432c069d90 0x7f432c151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:29.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f432b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f432c069d90 0x7f432c151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57418/0 (socket says 172.21.15.175:57418) 2023-10-10T19:14:29.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f432b7fe700 1 -- 172.21.15.175:0/1924980034 learned_addr learned my addr 172.21.15.175:0/1924980034 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:29.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f432affd700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c06d3f0 0x7f432c150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:29.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.911+0000 7f432bfff700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c0c3870 0x7f432c14cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:29.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f432b7fe700 1 -- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c0c3870 msgr2=0x7f432c14cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:29.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f432b7fe700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c0c3870 0x7f432c14cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:29.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f432b7fe700 1 -- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c06d3f0 msgr2=0x7f432c150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:29.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f432b7fe700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c06d3f0 0x7f432c150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:29.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f432b7fe700 1 -- 172.21.15.175:0/1924980034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0094f0 con 0x7f432c069d90 2023-10-10T19:14:29.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f432affd700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c06d3f0 0x7f432c150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:14:29.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f432b7fe700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f432c069d90 0x7f432c151a40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f432400c940 tx=0x7f4324001740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:29.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f43137fe700 1 -- 172.21.15.175:0/1924980034 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324012630 con 0x7f432c069d90 2023-10-10T19:14:29.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f43137fe700 1 -- 172.21.15.175:0/1924980034 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324012790 con 0x7f432c069d90 2023-10-10T19:14:29.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c140110 con 0x7f432c069d90 2023-10-10T19:14:29.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f43137fe700 1 -- 172.21.15.175:0/1924980034 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324012b50 con 0x7f432c069d90 2023-10-10T19:14:29.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.912+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c1406d0 con 0x7f432c069d90 2023-10-10T19:14:29.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.913+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c067570 con 0x7f432c069d90 2023-10-10T19:14:29.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.914+0000 7f43137fe700 1 -- 172.21.15.175:0/1924980034 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4324003030 con 0x7f432c069d90 2023-10-10T19:14:29.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.914+0000 7f43137fe700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4318066f30 0x7f43180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:29.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.914+0000 7f43137fe700 1 -- 172.21.15.175:0/1924980034 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7199+0+0 (secure 0 0 0) 0x7f4324084950 con 0x7f432c069d90 2023-10-10T19:14:29.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.915+0000 7f432affd700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4318066f30 0x7f43180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:29.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.915+0000 7f432affd700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4318066f30 0x7f43180693e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f431c0099b0 tx=0x7f431c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:29.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:29.919+0000 7f43137fe700 1 -- 172.21.15.175:0/1924980034 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f43240515f0 con 0x7f432c069d90 2023-10-10T19:14:30.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.274+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f432c069360 con 0x7f432c069d90 2023-10-10T19:14:30.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.275+0000 7f43137fe700 1 -- 172.21.15.175:0/1924980034 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+6208 (secure 0 0 0) 0x7f4324054c10 con 0x7f432c069d90 2023-10-10T19:14:30.275 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:30.276 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:29.731950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:30.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4318066f30 msgr2=0x7f43180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:30.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4318066f30 0x7f43180693e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f431c0099b0 tx=0x7f431c017000 comp rx=0 tx=0).stop 2023-10-10T19:14:30.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f432c069d90 msgr2=0x7f432c151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:30.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f432c069d90 0x7f432c151a40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f432400c940 tx=0x7f4324001740 comp rx=0 tx=0).stop 2023-10-10T19:14:30.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 shutdown_connections 2023-10-10T19:14:30.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4318066f30 0x7f43180693e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:30.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f432c0c3870 0x7f432c14cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:30.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f432c069d90 0x7f432c151a40 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:30.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 --2- 172.21.15.175:0/1924980034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f432c06d3f0 0x7f432c150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:30.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 >> 172.21.15.175:0/1924980034 conn(0x7f432c068280 msgr2=0x7f432c06dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:30.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.278+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 shutdown_connections 2023-10-10T19:14:30.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:30.279+0000 7f4331e1b700 1 -- 172.21.15.175:0/1924980034 wait complete. 2023-10-10T19:14:30.282 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 228 2023-10-10T19:14:30.297 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:14:32.299 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:32.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.413+0000 7f262cbe1700 1 Processor -- start 2023-10-10T19:14:32.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.413+0000 7f262cbe1700 1 -- start start 2023-10-10T19:14:32.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262cbe1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26280c1fb0 0x7f262806b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:32.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262cbe1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2628154e20 con 0x7f262814ddb0 2023-10-10T19:14:32.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262cbe1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2628154210 con 0x7f26280c1be0 2023-10-10T19:14:32.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262cbe1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262806bca0 con 0x7f26280c1fb0 2023-10-10T19:14:32.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f2626d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f262814ddb0 0x7f262806de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33130/0 (socket says 172.21.15.175:33130) 2023-10-10T19:14:32.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f2626d9d700 1 -- 172.21.15.175:0/699785778 learned_addr learned my addr 172.21.15.175:0/699785778 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:32.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f2625d9b700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26280c1fb0 0x7f262806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:32.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709920960 0 0) 0x7f2628154e20 con 0x7f262814ddb0 2023-10-10T19:14:32.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2610003570 con 0x7f262814ddb0 2023-10-10T19:14:32.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564577442 0 0) 0x7f2628154210 con 0x7f26280c1be0 2023-10-10T19:14:32.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2628154e20 con 0x7f26280c1be0 2023-10-10T19:14:32.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2912651644 0 0) 0x7f2610003570 con 0x7f262814ddb0 2023-10-10T19:14:32.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26280c1fb0 msgr2=0x7f262806b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:32.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262559a700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26280c1fb0 0x7f262806b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.414+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 >> v1:172.21.15.191:6789/0 conn(0x7f26280c1be0 legacy=0x7f262806a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:32.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26281544f0 con 0x7f262814ddb0 2023-10-10T19:14:32.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f2618001470 con 0x7f262814ddb0 2023-10-10T19:14:32.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 >> v1:172.21.15.175:6789/0 conn(0x7f262814ddb0 legacy=0x7f262806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:32.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2610003c20 0x7f2610004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:32.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f2610004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:32.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26280c1fb0 0x7f26100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:32.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262806bca0 con 0x7f2610004810 2023-10-10T19:14:32.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261000bb70 con 0x7f2610003c20 2023-10-10T19:14:32.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261000bd30 con 0x7f26280c1fb0 2023-10-10T19:14:32.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f2625d9b700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f2610004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:32.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262659c700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2610003c20 0x7f2610004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:32.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f2626d9d700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26280c1fb0 0x7f26100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:32.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f2625d9b700 1 -- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26280c1fb0 msgr2=0x7f26100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:32.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f2625d9b700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26280c1fb0 0x7f26100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f2625d9b700 1 -- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2610003c20 msgr2=0x7f2610004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:32.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f2625d9b700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2610003c20 0x7f2610004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f2625d9b700 1 -- 172.21.15.175:0/699785778 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2610003570 con 0x7f2610004810 2023-10-10T19:14:32.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f2625d9b700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f2610004c80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f261c009c00 tx=0x7f261c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77eb1759f71d9ff server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:32.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c00e710 con 0x7f2610004810 2023-10-10T19:14:32.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c00e870 con 0x7f2610004810 2023-10-10T19:14:32.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.415+0000 7f262559a700 1 -- 172.21.15.175:0/699785778 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c00ec30 con 0x7f2610004810 2023-10-10T19:14:32.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.416+0000 7f262cbe1700 1 -- 172.21.15.175:0/699785778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 msgr2=0x7f2610004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:32.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.416+0000 7f262cbe1700 1 --2- 172.21.15.175:0/699785778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f2610004c80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f261c009c00 tx=0x7f261c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.416+0000 7f2626d9d700 1 -- 172.21.15.175:0/699785778 reap_dead start 2023-10-10T19:14:32.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.416+0000 7f262cbe1700 1 -- 172.21.15.175:0/699785778 shutdown_connections 2023-10-10T19:14:32.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.416+0000 7f262cbe1700 1 -- 172.21.15.175:0/699785778 >> 172.21.15.175:0/699785778 conn(0x7f2628068280 msgr2=0x7f262806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:32.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.416+0000 7f262cbe1700 1 -- 172.21.15.175:0/699785778 shutdown_connections 2023-10-10T19:14:32.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.416+0000 7f262cbe1700 1 -- 172.21.15.175:0/699785778 wait complete. 2023-10-10T19:14:32.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.417+0000 7f262cbe1700 1 Processor -- start 2023-10-10T19:14:32.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.417+0000 7f262cbe1700 1 -- start start 2023-10-10T19:14:32.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.417+0000 7f262cbe1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f26281673e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:32.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.417+0000 7f262cbe1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26281551d0 0x7f262814ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:32.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.417+0000 7f262cbe1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2628167920 0x7f2628150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:32.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262cbe1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26281494e0 con 0x7f2610004810 2023-10-10T19:14:32.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262cbe1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26281680b0 con 0x7f2628167920 2023-10-10T19:14:32.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262cbe1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262814fa80 con 0x7f26281551d0 2023-10-10T19:14:32.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f26281673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:32.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f26281673e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57432/0 (socket says 172.21.15.175:57432) 2023-10-10T19:14:32.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 -- 172.21.15.175:0/924111204 learned_addr learned my addr 172.21.15.175:0/924111204 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:32.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f2625d9b700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26281551d0 0x7f262814ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:32.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f2626d9d700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2628167920 0x7f2628150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:32.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 -- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26281551d0 msgr2=0x7f262814ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:32.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26281551d0 0x7f262814ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 -- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2628167920 msgr2=0x7f2628150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:32.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2628167920 0x7f2628150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 -- 172.21.15.175:0/924111204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26281496e0 con 0x7f2610004810 2023-10-10T19:14:32.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f2626d9d700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2628167920 0x7f2628150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:32.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262659c700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f26281673e0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f262000c440 tx=0x7f2620001bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:32.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f26167fc700 1 -- 172.21.15.175:0/924111204 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26200135f0 con 0x7f2610004810 2023-10-10T19:14:32.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f26167fc700 1 -- 172.21.15.175:0/924111204 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2620013770 con 0x7f2610004810 2023-10-10T19:14:32.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f26167fc700 1 -- 172.21.15.175:0/924111204 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2620013b30 con 0x7f2610004810 2023-10-10T19:14:32.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2628167dd0 con 0x7f2610004810 2023-10-10T19:14:32.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.418+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262814ce60 con 0x7f2610004810 2023-10-10T19:14:32.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.419+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2628067570 con 0x7f2610004810 2023-10-10T19:14:32.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.420+0000 7f26167fc700 1 -- 172.21.15.175:0/924111204 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2620014480 con 0x7f2610004810 2023-10-10T19:14:32.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.420+0000 7f26167fc700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2608066f30 0x7f26080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:32.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.420+0000 7f26167fc700 1 -- 172.21.15.175:0/924111204 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7183+0+0 (secure 0 0 0) 0x7f2620085a00 con 0x7f2610004810 2023-10-10T19:14:32.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.420+0000 7f2625d9b700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2608066f30 0x7f26080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:32.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.421+0000 7f2625d9b700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2608066f30 0x7f26080693e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2628151120 tx=0x7f261c01a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:32.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.424+0000 7f26167fc700 1 -- 172.21.15.175:0/924111204 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26200526b0 con 0x7f2610004810 2023-10-10T19:14:32.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.649+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26280c5c00 con 0x7f2610004810 2023-10-10T19:14:32.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.650+0000 7f26167fc700 1 -- 172.21.15.175:0/924111204 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+6208 (secure 0 0 0) 0x7f2620055cd0 con 0x7f2610004810 2023-10-10T19:14:32.651 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:32.651 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:29.731950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:32.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2608066f30 msgr2=0x7f26080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:32.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2608066f30 0x7f26080693e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2628151120 tx=0x7f261c01a000 comp rx=0 tx=0).stop 2023-10-10T19:14:32.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 msgr2=0x7f26281673e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:32.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f26281673e0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f262000c440 tx=0x7f2620001bd0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 shutdown_connections 2023-10-10T19:14:32.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2608066f30 0x7f26080693e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2628167920 0x7f2628150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2610004810 0x7f26281673e0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 --2- 172.21.15.175:0/924111204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26281551d0 0x7f262814ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:32.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 >> 172.21.15.175:0/924111204 conn(0x7f2628068280 msgr2=0x7f262814b460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:32.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 shutdown_connections 2023-10-10T19:14:32.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:32.652+0000 7f262cbe1700 1 -- 172.21.15.175:0/924111204 wait complete. 2023-10-10T19:14:32.655 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 228 2023-10-10T19:14:32.664 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:14:34.665 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:34.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.791+0000 7f379e151700 1 Processor -- start 2023-10-10T19:14:34.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f379e151700 1 -- start start 2023-10-10T19:14:34.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f379e151700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f37980b1fc0 0x7f3798155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:34.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f379e151700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37980aff60 0x7f37980b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:34.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f379e151700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37981574b0 con 0x7f37980b2390 2023-10-10T19:14:34.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f379e151700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3798150770 con 0x7f37980aff60 2023-10-10T19:14:34.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f379e151700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37980afad0 con 0x7f37980b1fc0 2023-10-10T19:14:34.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f3797fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37980aff60 0x7f37980b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:34.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f3797fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f37980aff60 0x7f37980b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45416/0 (socket says 172.21.15.175:45416) 2023-10-10T19:14:34.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.792+0000 7f3797fff700 1 -- 172.21.15.175:0/2664272150 learned_addr learned my addr 172.21.15.175:0/2664272150 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:34.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f37977fe700 1 --2- 172.21.15.175:0/2664272150 >> v2:172.21.15.191:3300/0 conn(0x7f37980b1fc0 0x7f3798155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:34.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f3797fff700 1 -- 172.21.15.175:0/2664272150 >> v2:172.21.15.191:3300/0 conn(0x7f37980b1fc0 msgr2=0x7f3798155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:34.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f3797fff700 1 --2- 172.21.15.175:0/2664272150 >> v2:172.21.15.191:3300/0 conn(0x7f37980b1fc0 0x7f3798155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:34.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f3797fff700 1 -- 172.21.15.175:0/2664272150 >> v1:172.21.15.175:6789/0 conn(0x7f37980b2390 legacy=0x7f3798068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:34.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f3797fff700 1 -- 172.21.15.175:0/2664272150 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37980af4c0 con 0x7f37980aff60 2023-10-10T19:14:34.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f3797fff700 1 --2- 172.21.15.175:0/2664272150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980aff60 0x7f37980b03e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f37980b28c0 tx=0x7f3790009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a97dd35e64b33b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:34.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f37967fc700 1 -- 172.21.15.175:0/2664272150 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379000b8f0 con 0x7f37980aff60 2023-10-10T19:14:34.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f37967fc700 1 -- 172.21.15.175:0/2664272150 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379000ba50 con 0x7f37980aff60 2023-10-10T19:14:34.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.793+0000 7f37967fc700 1 -- 172.21.15.175:0/2664272150 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379000bdf0 con 0x7f37980aff60 2023-10-10T19:14:34.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.794+0000 7f379e151700 1 -- 172.21.15.175:0/2664272150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980aff60 msgr2=0x7f37980b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:34.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.794+0000 7f379e151700 1 --2- 172.21.15.175:0/2664272150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980aff60 0x7f37980b03e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f37980b28c0 tx=0x7f3790009250 comp rx=0 tx=0).stop 2023-10-10T19:14:34.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.794+0000 7f379e151700 1 -- 172.21.15.175:0/2664272150 shutdown_connections 2023-10-10T19:14:34.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.794+0000 7f379e151700 1 --2- 172.21.15.175:0/2664272150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980aff60 0x7f37980b03e0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:34.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.794+0000 7f379e151700 1 --2- 172.21.15.175:0/2664272150 >> v2:172.21.15.191:3300/0 conn(0x7f37980b1fc0 0x7f3798155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:34.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.794+0000 7f379e151700 1 -- 172.21.15.175:0/2664272150 >> 172.21.15.175:0/2664272150 conn(0x7f3798063230 msgr2=0x7f3798063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:34.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.794+0000 7f379e151700 1 -- 172.21.15.175:0/2664272150 shutdown_connections 2023-10-10T19:14:34.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.794+0000 7f379e151700 1 -- 172.21.15.175:0/2664272150 wait complete. 2023-10-10T19:14:34.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.795+0000 7f379e151700 1 Processor -- start 2023-10-10T19:14:34.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.795+0000 7f379e151700 1 -- start start 2023-10-10T19:14:34.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f379e151700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37980aff60 0x7f3798166760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:34.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f379e151700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980b1fc0 0x7f37980c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:34.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f379e151700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37980b2390 0x7f37981552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:34.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f379e151700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3798166ca0 con 0x7f37980b1fc0 2023-10-10T19:14:34.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f379e151700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37980c6400 con 0x7f37980aff60 2023-10-10T19:14:34.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f379e151700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37980c5860 con 0x7f37980b2390 2023-10-10T19:14:34.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980b1fc0 0x7f37980c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:34.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980b1fc0 0x7f37980c3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45424/0 (socket says 172.21.15.175:45424) 2023-10-10T19:14:34.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 -- 172.21.15.175:0/1728280353 learned_addr learned my addr 172.21.15.175:0/1728280353 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:34.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f37977fe700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37980aff60 0x7f3798166760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:34.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3797fff700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37980b2390 0x7f37981552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:34.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 -- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37980b2390 msgr2=0x7f37981552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:34.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37980b2390 0x7f37981552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:34.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 -- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37980aff60 msgr2=0x7f3798166760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:34.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37980aff60 0x7f3798166760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:34.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 -- 172.21.15.175:0/1728280353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900094f0 con 0x7f37980b1fc0 2023-10-10T19:14:34.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f3796ffd700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980b1fc0 0x7f37980c3a30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c510 tx=0x7f378c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:34.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f377f7fe700 1 -- 172.21.15.175:0/1728280353 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c011590 con 0x7f37980b1fc0 2023-10-10T19:14:34.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f377f7fe700 1 -- 172.21.15.175:0/1728280353 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c011710 con 0x7f37980b1fc0 2023-10-10T19:14:34.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37980c6890 con 0x7f37980b1fc0 2023-10-10T19:14:34.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f377f7fe700 1 -- 172.21.15.175:0/1728280353 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c011ad0 con 0x7f37980b1fc0 2023-10-10T19:14:34.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.796+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37980c5ad0 con 0x7f37980b1fc0 2023-10-10T19:14:34.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.797+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3798071340 con 0x7f37980b1fc0 2023-10-10T19:14:34.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.798+0000 7f377f7fe700 1 -- 172.21.15.175:0/1728280353 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f378c01b3b0 con 0x7f37980b1fc0 2023-10-10T19:14:34.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.798+0000 7f377f7fe700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3784066f30 0x7f37840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:34.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.798+0000 7f377f7fe700 1 -- 172.21.15.175:0/1728280353 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7183+0+0 (secure 0 0 0) 0x7f378c083e40 con 0x7f37980b1fc0 2023-10-10T19:14:34.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.798+0000 7f37977fe700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3784066f30 0x7f37840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:34.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.799+0000 7f37977fe700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3784066f30 0x7f37840693e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3788009fd0 tx=0x7f3788009880 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:34.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:34.803+0000 7f377f7fe700 1 -- 172.21.15.175:0/1728280353 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f378c050af0 con 0x7f37980b1fc0 2023-10-10T19:14:35.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.028+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37980c6540 con 0x7f37980b1fc0 2023-10-10T19:14:35.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.029+0000 7f377f7fe700 1 -- 172.21.15.175:0/1728280353 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+6208 (secure 0 0 0) 0x7f378c017070 con 0x7f37980b1fc0 2023-10-10T19:14:35.031 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:35.031 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:29.731950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:35.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3784066f30 msgr2=0x7f37840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:35.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3784066f30 0x7f37840693e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3788009fd0 tx=0x7f3788009880 comp rx=0 tx=0).stop 2023-10-10T19:14:35.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980b1fc0 msgr2=0x7f37980c3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:35.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980b1fc0 0x7f37980c3a30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c510 tx=0x7f378c00c820 comp rx=0 tx=0).stop 2023-10-10T19:14:35.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 shutdown_connections 2023-10-10T19:14:35.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3784066f30 0x7f37840693e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:35.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f37980b2390 0x7f37981552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:35.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37980aff60 0x7f3798166760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:35.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 --2- 172.21.15.175:0/1728280353 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37980b1fc0 0x7f37980c3a30 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:35.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 >> 172.21.15.175:0/1728280353 conn(0x7f3798063230 msgr2=0x7f379814fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:35.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 shutdown_connections 2023-10-10T19:14:35.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:35.031+0000 7f379e151700 1 -- 172.21.15.175:0/1728280353 wait complete. 2023-10-10T19:14:35.036 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 228 2023-10-10T19:14:35.045 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:14:37.046 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:37.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.171+0000 7f86df59e700 1 Processor -- start 2023-10-10T19:14:37.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.171+0000 7f86df59e700 1 -- start start 2023-10-10T19:14:37.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86df59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e014fd90 0x7f86e0159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:37.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86df59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86e008a100 0x7f86e00c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86df59e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e0067fa0 con 0x7f86e00cd040 2023-10-10T19:14:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86df59e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e00c5600 con 0x7f86e008a100 2023-10-10T19:14:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86df59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e0156b70 con 0x7f86e014fd90 2023-10-10T19:14:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86de59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e014fd90 0x7f86e0159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:37.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86d5d9b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86e008a100 0x7f86e00c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86d5d9b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86e008a100 0x7f86e00c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:58376/0 (socket says 172.21.15.175:58376) 2023-10-10T19:14:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.172+0000 7f86d5d9b700 1 -- 172.21.15.175:0/1774278895 learned_addr learned my addr 172.21.15.175:0/1774278895 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86d5d9b700 1 -- 172.21.15.175:0/1774278895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e014fd90 msgr2=0x7f86e0159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:37.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86d5d9b700 1 --2- 172.21.15.175:0/1774278895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e014fd90 0x7f86e0159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86d5d9b700 1 -- 172.21.15.175:0/1774278895 >> v1:172.21.15.191:6789/0 conn(0x7f86e00cd040 legacy=0x7f86e00684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:37.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86d5d9b700 1 -- 172.21.15.175:0/1774278895 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e0150160 con 0x7f86e008a100 2023-10-10T19:14:37.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86de59c700 1 --2- 172.21.15.175:0/1774278895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e014fd90 0x7f86e0159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:14:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86d5d9b700 1 --2- 172.21.15.175:0/1774278895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e008a100 0x7f86e00c5eb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f86c80099b0 tx=0x7f86c8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a92df411eb307dee server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86ddd9b700 1 -- 172.21.15.175:0/1774278895 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8009ea0 con 0x7f86e008a100 2023-10-10T19:14:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86ddd9b700 1 -- 172.21.15.175:0/1774278895 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c800b860 con 0x7f86e008a100 2023-10-10T19:14:37.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.173+0000 7f86ddd9b700 1 -- 172.21.15.175:0/1774278895 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c800bc00 con 0x7f86e008a100 2023-10-10T19:14:37.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.174+0000 7f86df59e700 1 -- 172.21.15.175:0/1774278895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e008a100 msgr2=0x7f86e00c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:37.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.174+0000 7f86df59e700 1 --2- 172.21.15.175:0/1774278895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e008a100 0x7f86e00c5eb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f86c80099b0 tx=0x7f86c8009250 comp rx=0 tx=0).stop 2023-10-10T19:14:37.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.174+0000 7f86df59e700 1 -- 172.21.15.175:0/1774278895 shutdown_connections 2023-10-10T19:14:37.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.174+0000 7f86df59e700 1 --2- 172.21.15.175:0/1774278895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e014fd90 0x7f86e0159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.174+0000 7f86df59e700 1 --2- 172.21.15.175:0/1774278895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e008a100 0x7f86e00c5eb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.174+0000 7f86df59e700 1 -- 172.21.15.175:0/1774278895 >> 172.21.15.175:0/1774278895 conn(0x7f86e0063d00 msgr2=0x7f86e00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:37.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.174+0000 7f86df59e700 1 -- 172.21.15.175:0/1774278895 shutdown_connections 2023-10-10T19:14:37.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.174+0000 7f86df59e700 1 -- 172.21.15.175:0/1774278895 wait complete. 2023-10-10T19:14:37.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.175+0000 7f86df59e700 1 Processor -- start 2023-10-10T19:14:37.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.175+0000 7f86df59e700 1 -- start start 2023-10-10T19:14:37.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.175+0000 7f86df59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e008a100 0x7f86e0169d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:37.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86df59e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e00cd040 0x7f86e0154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:37.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86df59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86e014fd90 0x7f86e014b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:37.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86df59e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e0166d20 con 0x7f86e014fd90 2023-10-10T19:14:37.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86df59e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e014f420 con 0x7f86e00cd040 2023-10-10T19:14:37.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86df59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e01698a0 con 0x7f86e008a100 2023-10-10T19:14:37.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86e014fd90 0x7f86e014b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:37.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86e014fd90 0x7f86e014b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45434/0 (socket says 172.21.15.175:45434) 2023-10-10T19:14:37.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 -- 172.21.15.175:0/3359242759 learned_addr learned my addr 172.21.15.175:0/3359242759 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:37.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86de59c700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e008a100 0x7f86e0169d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:37.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86d5d9b700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e00cd040 0x7f86e0154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:37.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 -- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e008a100 msgr2=0x7f86e0169d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:37.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e008a100 0x7f86e0169d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 -- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e00cd040 msgr2=0x7f86e0154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:37.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e00cd040 0x7f86e0154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 -- 172.21.15.175:0/3359242759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0009420 con 0x7f86e014fd90 2023-10-10T19:14:37.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86d5d9b700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e00cd040 0x7f86e0154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:14:37.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86ded9d700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86e014fd90 0x7f86e014b7f0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f86d800c940 tx=0x7f86d8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:37.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86d6ffd700 1 -- 172.21.15.175:0/3359242759 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d800a760 con 0x7f86e014fd90 2023-10-10T19:14:37.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86d6ffd700 1 -- 172.21.15.175:0/3359242759 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d800a8c0 con 0x7f86e014fd90 2023-10-10T19:14:37.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c80094f0 con 0x7f86e014fd90 2023-10-10T19:14:37.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86d6ffd700 1 -- 172.21.15.175:0/3359242759 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d800ac80 con 0x7f86e014fd90 2023-10-10T19:14:37.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.176+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e014f140 con 0x7f86e014fd90 2023-10-10T19:14:37.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.177+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e0067570 con 0x7f86e014fd90 2023-10-10T19:14:37.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.178+0000 7f86d6ffd700 1 -- 172.21.15.175:0/3359242759 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f86d8012420 con 0x7f86e014fd90 2023-10-10T19:14:37.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.178+0000 7f86d6ffd700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86c0066f30 0x7f86c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:37.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.178+0000 7f86d6ffd700 1 -- 172.21.15.175:0/3359242759 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7183+0+0 (secure 0 0 0) 0x7f86d8084db0 con 0x7f86e014fd90 2023-10-10T19:14:37.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.178+0000 7f86de59c700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86c0066f30 0x7f86c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:37.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.179+0000 7f86de59c700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86c0066f30 0x7f86c00693e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f86d00093f0 tx=0x7f86d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:37.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.183+0000 7f86d6ffd700 1 -- 172.21.15.175:0/3359242759 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f86d8051a60 con 0x7f86e014fd90 2023-10-10T19:14:37.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.407+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86e01699e0 con 0x7f86e014fd90 2023-10-10T19:14:37.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.407+0000 7f86d6ffd700 1 -- 172.21.15.175:0/3359242759 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+6208 (secure 0 0 0) 0x7f86d8055080 con 0x7f86e014fd90 2023-10-10T19:14:37.409 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:37.410 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:29.731950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:37.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.409+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86c0066f30 msgr2=0x7f86c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:37.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.409+0000 7f86df59e700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86c0066f30 0x7f86c00693e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f86d00093f0 tx=0x7f86d0008040 comp rx=0 tx=0).stop 2023-10-10T19:14:37.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.409+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86e014fd90 msgr2=0x7f86e014b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:37.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.409+0000 7f86df59e700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86e014fd90 0x7f86e014b7f0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f86d800c940 tx=0x7f86d8001710 comp rx=0 tx=0).stop 2023-10-10T19:14:37.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.410+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 shutdown_connections 2023-10-10T19:14:37.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.410+0000 7f86df59e700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f86c0066f30 0x7f86c00693e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.410+0000 7f86df59e700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86e014fd90 0x7f86e014b7f0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.410+0000 7f86df59e700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86e008a100 0x7f86e0169d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.410+0000 7f86df59e700 1 --2- 172.21.15.175:0/3359242759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86e00cd040 0x7f86e0154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:37.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.410+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 >> 172.21.15.175:0/3359242759 conn(0x7f86e0063d00 msgr2=0x7f86e00cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:37.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.410+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 shutdown_connections 2023-10-10T19:14:37.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:37.410+0000 7f86df59e700 1 -- 172.21.15.175:0/3359242759 wait complete. 2023-10-10T19:14:37.414 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 228 2023-10-10T19:14:37.423 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:14:39.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:39.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.553+0000 7fab093d0700 1 Processor -- start 2023-10-10T19:14:39.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.553+0000 7fab093d0700 1 -- start start 2023-10-10T19:14:39.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab093d0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c5850 0x7fab040c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab093d0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fab040c4f20 0x7fab040c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab093d0700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040c6fb0 con 0x7fab040c12d0 2023-10-10T19:14:39.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab093d0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab04063ea0 con 0x7fab040c4f20 2023-10-10T19:14:39.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab093d0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab04155f30 con 0x7fab040c5850 2023-10-10T19:14:39.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab027fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fab040c12d0 0x7fab04156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35228/0 (socket says 172.21.15.175:35228) 2023-10-10T19:14:39.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab027fc700 1 -- 172.21.15.175:0/3502097696 learned_addr learned my addr 172.21.15.175:0/3502097696 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:39.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab037fe700 1 --2- 172.21.15.175:0/3502097696 >> v2:172.21.15.191:3300/0 conn(0x7fab040c4f20 0x7fab040c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab02ffd700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c5850 0x7fab040c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867320574 0 0) 0x7fab040c6fb0 con 0x7fab040c12d0 2023-10-10T19:14:39.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaec003570 con 0x7fab040c12d0 2023-10-10T19:14:39.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833249047 0 0) 0x7faaec003570 con 0x7fab040c12d0 2023-10-10T19:14:39.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c5850 msgr2=0x7fab040c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:39.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c5850 0x7fab040c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 >> v2:172.21.15.191:3300/0 conn(0x7fab040c4f20 msgr2=0x7fab040c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:39.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 --2- 172.21.15.175:0/3502097696 >> v2:172.21.15.191:3300/0 conn(0x7fab040c4f20 0x7fab040c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040c6160 con 0x7fab040c12d0 2023-10-10T19:14:39.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7faaf8001470 con 0x7fab040c12d0 2023-10-10T19:14:39.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.554+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 >> v1:172.21.15.175:6789/0 conn(0x7fab040c12d0 legacy=0x7fab04156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:39.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c5850 0x7faaec003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaec003fe0 0x7faaec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7faaec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04063ea0 con 0x7faaec004d50 2023-10-10T19:14:39.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04155f30 con 0x7faaec003fe0 2023-10-10T19:14:39.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec00ba90 con 0x7fab040c5850 2023-10-10T19:14:39.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab037fe700 1 --2- 172.21.15.175:0/3502097696 >> v2:172.21.15.191:3300/0 conn(0x7fab040c4f20 0x7fab040c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:14:39.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab02ffd700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c5850 0x7faaec003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab037fe700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7faaec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab027fc700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaec003fe0 0x7faaec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab037fe700 1 -- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c5850 msgr2=0x7faaec003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:39.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab037fe700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c5850 0x7faaec003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab037fe700 1 -- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaec003fe0 msgr2=0x7faaec004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:39.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab037fe700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faaec003fe0 0x7faaec004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab037fe700 1 -- 172.21.15.175:0/3502097696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec003570 con 0x7faaec004d50 2023-10-10T19:14:39.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab037fe700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7faaec0051c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7faafc009db0 tx=0x7faafc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe6f1a5466cf016 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:39.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc009740 con 0x7faaec004d50 2023-10-10T19:14:39.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc00e8c0 con 0x7faaec004d50 2023-10-10T19:14:39.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.555+0000 7fab01ffb700 1 -- 172.21.15.175:0/3502097696 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc00ec40 con 0x7faaec004d50 2023-10-10T19:14:39.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.556+0000 7fab093d0700 1 -- 172.21.15.175:0/3502097696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 msgr2=0x7faaec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:39.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.556+0000 7fab093d0700 1 --2- 172.21.15.175:0/3502097696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7faaec0051c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7faafc009db0 tx=0x7faafc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.556+0000 7fab037fe700 1 -- 172.21.15.175:0/3502097696 reap_dead start 2023-10-10T19:14:39.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.556+0000 7fab093d0700 1 -- 172.21.15.175:0/3502097696 shutdown_connections 2023-10-10T19:14:39.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.556+0000 7fab093d0700 1 -- 172.21.15.175:0/3502097696 >> 172.21.15.175:0/3502097696 conn(0x7fab0406dc70 msgr2=0x7fab04063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:39.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.556+0000 7fab093d0700 1 -- 172.21.15.175:0/3502097696 shutdown_connections 2023-10-10T19:14:39.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.556+0000 7fab093d0700 1 -- 172.21.15.175:0/3502097696 wait complete. 2023-10-10T19:14:39.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.557+0000 7fab093d0700 1 Processor -- start 2023-10-10T19:14:39.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.557+0000 7fab093d0700 1 -- start start 2023-10-10T19:14:39.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab093d0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7fab040c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab093d0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab04152e60 0x7fab04152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab093d0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c88d0 0x7fab04152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab093d0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab040cad50 con 0x7faaec004d50 2023-10-10T19:14:39.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab093d0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04152b90 con 0x7fab04152e60 2023-10-10T19:14:39.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab093d0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab04168e70 con 0x7fab040c88d0 2023-10-10T19:14:39.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7fab040c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7fab040c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45456/0 (socket says 172.21.15.175:45456) 2023-10-10T19:14:39.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 -- 172.21.15.175:0/3983320976 learned_addr learned my addr 172.21.15.175:0/3983320976 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:39.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab037fe700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c88d0 0x7fab04152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab027fc700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab04152e60 0x7fab04152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 -- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c88d0 msgr2=0x7fab04152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:39.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c88d0 0x7fab04152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 -- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab04152e60 msgr2=0x7fab04152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:39.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab04152e60 0x7fab04152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 -- 172.21.15.175:0/3983320976 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab04154ae0 con 0x7faaec004d50 2023-10-10T19:14:39.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7fab02ffd700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7fab040c8390 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7faaf4011f30 tx=0x7faaf400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:39.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7faaeaffd700 1 -- 172.21.15.175:0/3983320976 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf400e040 con 0x7faaec004d50 2023-10-10T19:14:39.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.558+0000 7faaeaffd700 1 -- 172.21.15.175:0/3983320976 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf400f040 con 0x7faaec004d50 2023-10-10T19:14:39.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.559+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab040caa70 con 0x7faaec004d50 2023-10-10T19:14:39.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.559+0000 7faaeaffd700 1 -- 172.21.15.175:0/3983320976 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf4015640 con 0x7faaec004d50 2023-10-10T19:14:39.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.559+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab04168390 con 0x7faaec004d50 2023-10-10T19:14:39.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.560+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab04067570 con 0x7faaec004d50 2023-10-10T19:14:39.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.560+0000 7faaeaffd700 1 -- 172.21.15.175:0/3983320976 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faaf4015a70 con 0x7faaec004d50 2023-10-10T19:14:39.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.560+0000 7faaeaffd700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faaf0066f30 0x7faaf00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:39.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.560+0000 7faaeaffd700 1 -- 172.21.15.175:0/3983320976 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 7139+0+0 (secure 0 0 0) 0x7faaf4085820 con 0x7faaec004d50 2023-10-10T19:14:39.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.560+0000 7fab027fc700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faaf0066f30 0x7faaf00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:39.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.561+0000 7fab027fc700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faaf0066f30 0x7faaf00693e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fab04153230 tx=0x7faaf8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:39.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.565+0000 7faaeaffd700 1 -- 172.21.15.175:0/3983320976 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faaf40524f0 con 0x7faaec004d50 2023-10-10T19:14:39.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.791+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab04154c70 con 0x7faaec004d50 2023-10-10T19:14:39.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.792+0000 7faaeaffd700 1 -- 172.21.15.175:0/3983320976 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+6208 (secure 0 0 0) 0x7fab04154c70 con 0x7faaec004d50 2023-10-10T19:14:39.794 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:39.794 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:29.731950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:39.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faaf0066f30 msgr2=0x7faaf00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:39.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faaf0066f30 0x7faaf00693e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fab04153230 tx=0x7faaf8009040 comp rx=0 tx=0).stop 2023-10-10T19:14:39.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 msgr2=0x7fab040c8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:39.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7fab040c8390 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7faaf4011f30 tx=0x7faaf400b040 comp rx=0 tx=0).stop 2023-10-10T19:14:39.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 shutdown_connections 2023-10-10T19:14:39.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faaf0066f30 0x7faaf00693e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab040c88d0 0x7fab04152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faaec004d50 0x7fab040c8390 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 --2- 172.21.15.175:0/3983320976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab04152e60 0x7fab04152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:39.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 >> 172.21.15.175:0/3983320976 conn(0x7fab0406dc70 msgr2=0x7fab0406a720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:39.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 shutdown_connections 2023-10-10T19:14:39.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:39.794+0000 7fab093d0700 1 -- 172.21.15.175:0/3983320976 wait complete. 2023-10-10T19:14:39.798 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 228 2023-10-10T19:14:39.807 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:14:41.808 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:41.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.933+0000 7f820e983700 1 Processor -- start 2023-10-10T19:14:41.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.934+0000 7f820e983700 1 -- start start 2023-10-10T19:14:41.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.934+0000 7f820e983700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f82080b1fc0 0x7f8208155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:41.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.934+0000 7f820e983700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f82080b2390 0x7f8208068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:41.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.934+0000 7f820e983700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 0x7f82080aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:41.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.934+0000 7f820e983700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8208150770 con 0x7f82080b2390 2023-10-10T19:14:41.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.934+0000 7f820e983700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82081574b0 con 0x7f82080b1fc0 2023-10-10T19:14:41.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.934+0000 7f820e983700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82080afad0 con 0x7f8208068bd0 2023-10-10T19:14:41.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f82080b2390 0x7f8208068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:41.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f82080b2390 0x7f8208068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45458/0 (socket says 172.21.15.175:45458) 2023-10-10T19:14:41.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 -- 172.21.15.175:0/606858591 learned_addr learned my addr 172.21.15.175:0/606858591 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:41.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f820cf20700 1 --2- 172.21.15.175:0/606858591 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 0x7f82080aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:41.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f8207fff700 1 --2- 172.21.15.175:0/606858591 >> v2:172.21.15.191:3300/0 conn(0x7f82080b1fc0 0x7f8208155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:41.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 -- 172.21.15.175:0/606858591 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 msgr2=0x7f82080aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:41.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 --2- 172.21.15.175:0/606858591 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 0x7f82080aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:41.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 -- 172.21.15.175:0/606858591 >> v2:172.21.15.191:3300/0 conn(0x7f82080b1fc0 msgr2=0x7f8208155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:41.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 --2- 172.21.15.175:0/606858591 >> v2:172.21.15.191:3300/0 conn(0x7f82080b1fc0 0x7f8208155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:41.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 -- 172.21.15.175:0/606858591 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080af4c0 con 0x7f82080b2390 2023-10-10T19:14:41.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f82077fe700 1 --2- 172.21.15.175:0/606858591 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b2390 0x7f8208068690 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f82080b28c0 tx=0x7f81fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0fd488e855b4b8d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:41.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f8206ffd700 1 -- 172.21.15.175:0/606858591 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc00b8f0 con 0x7f82080b2390 2023-10-10T19:14:41.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f8206ffd700 1 -- 172.21.15.175:0/606858591 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc00ba50 con 0x7f82080b2390 2023-10-10T19:14:41.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.935+0000 7f8206ffd700 1 -- 172.21.15.175:0/606858591 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc00bdf0 con 0x7f82080b2390 2023-10-10T19:14:41.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 -- 172.21.15.175:0/606858591 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b2390 msgr2=0x7f8208068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:41.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 --2- 172.21.15.175:0/606858591 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b2390 0x7f8208068690 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f82080b28c0 tx=0x7f81fc009250 comp rx=0 tx=0).stop 2023-10-10T19:14:41.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 -- 172.21.15.175:0/606858591 shutdown_connections 2023-10-10T19:14:41.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 --2- 172.21.15.175:0/606858591 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 0x7f82080aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:41.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 --2- 172.21.15.175:0/606858591 >> v2:172.21.15.191:3300/0 conn(0x7f82080b1fc0 0x7f8208155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:41.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 --2- 172.21.15.175:0/606858591 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b2390 0x7f8208068690 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:41.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 -- 172.21.15.175:0/606858591 >> 172.21.15.175:0/606858591 conn(0x7f8208063230 msgr2=0x7f8208063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:41.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 -- 172.21.15.175:0/606858591 shutdown_connections 2023-10-10T19:14:41.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.936+0000 7f820e983700 1 -- 172.21.15.175:0/606858591 wait complete. 2023-10-10T19:14:41.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.937+0000 7f820e983700 1 Processor -- start 2023-10-10T19:14:41.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.937+0000 7f820e983700 1 -- start start 2023-10-10T19:14:41.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f820e983700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 0x7f820814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:41.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f820e983700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b1fc0 0x7f82081491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:41.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f820e983700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82080b2390 0x7f8208149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:41.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f820e983700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82080cb6b0 con 0x7f82080b1fc0 2023-10-10T19:14:41.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f820e983700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8208148d40 con 0x7f82080b2390 2023-10-10T19:14:41.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f820e983700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82081481f0 con 0x7f8208068bd0 2023-10-10T19:14:41.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b1fc0 0x7f82081491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:41.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b1fc0 0x7f82081491d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45474/0 (socket says 172.21.15.175:45474) 2023-10-10T19:14:41.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 -- 172.21.15.175:0/861897468 learned_addr learned my addr 172.21.15.175:0/861897468 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:41.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f8207fff700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 0x7f820814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:41.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f820cf20700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82080b2390 0x7f8208149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:41.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 -- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 msgr2=0x7f820814fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:41.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 0x7f820814fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:41.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 -- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82080b2390 msgr2=0x7f8208149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:41.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82080b2390 0x7f8208149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:41.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 -- 172.21.15.175:0/861897468 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc0094f0 con 0x7f82080b1fc0 2023-10-10T19:14:41.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f82077fe700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b1fc0 0x7f82081491d0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f81fc006fd0 tx=0x7f81fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:41.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f81effff700 1 -- 172.21.15.175:0/861897468 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc01a450 con 0x7f82080b1fc0 2023-10-10T19:14:41.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f81effff700 1 -- 172.21.15.175:0/861897468 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc01a5b0 con 0x7f82080b1fc0 2023-10-10T19:14:41.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820814af30 con 0x7f82080b1fc0 2023-10-10T19:14:41.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.938+0000 7f81effff700 1 -- 172.21.15.175:0/861897468 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc01a970 con 0x7f82080b1fc0 2023-10-10T19:14:41.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.939+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82080cb8e0 con 0x7f82080b1fc0 2023-10-10T19:14:41.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.940+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8208067570 con 0x7f82080b1fc0 2023-10-10T19:14:41.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.940+0000 7f81effff700 1 -- 172.21.15.175:0/861897468 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f81fc01ad20 con 0x7f82080b1fc0 2023-10-10T19:14:41.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.940+0000 7f81effff700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81f4066f30 0x7f81f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:41.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.940+0000 7f81effff700 1 -- 172.21.15.175:0/861897468 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 7139+0+0 (secure 0 0 0) 0x7f81fc086810 con 0x7f82080b1fc0 2023-10-10T19:14:41.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.940+0000 7f8207fff700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81f4066f30 0x7f81f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:41.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.941+0000 7f8207fff700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81f4066f30 0x7f81f40693e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f81f8006fd0 tx=0x7f81f8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:41.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:41.945+0000 7f81effff700 1 -- 172.21.15.175:0/861897468 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81fc0534e0 con 0x7f82080b1fc0 2023-10-10T19:14:42.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.169+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8208069060 con 0x7f82080b1fc0 2023-10-10T19:14:42.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.169+0000 7f81effff700 1 -- 172.21.15.175:0/861897468 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+6208 (secure 0 0 0) 0x7f81fc056b00 con 0x7f82080b1fc0 2023-10-10T19:14:42.171 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:42.171 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":226}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:29.731950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6117,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6117":{"gid":6117,"name":"a","rank":1,"incarnation":134,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/1872591144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1872591144},{"type":"v1","addr":"172.21.15.175:6834","nonce":1872591144}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:42.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81f4066f30 msgr2=0x7f81f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:42.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81f4066f30 0x7f81f40693e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f81f8006fd0 tx=0x7f81f8008040 comp rx=0 tx=0).stop 2023-10-10T19:14:42.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b1fc0 msgr2=0x7f82081491d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:42.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b1fc0 0x7f82081491d0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f81fc006fd0 tx=0x7f81fc017000 comp rx=0 tx=0).stop 2023-10-10T19:14:42.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 shutdown_connections 2023-10-10T19:14:42.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81f4066f30 0x7f81f40693e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:42.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82080b2390 0x7f8208149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:42.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8208068bd0 0x7f820814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:42.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 --2- 172.21.15.175:0/861897468 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82080b1fc0 0x7f82081491d0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:42.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 >> 172.21.15.175:0/861897468 conn(0x7f8208063230 msgr2=0x7f8208063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:42.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 shutdown_connections 2023-10-10T19:14:42.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:42.172+0000 7f820e983700 1 -- 172.21.15.175:0/861897468 wait complete. 2023-10-10T19:14:42.177 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 228 2023-10-10T19:14:42.185 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:14:44.186 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:44.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.312+0000 7f34e057a700 1 Processor -- start 2023-10-10T19:14:44.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.312+0000 7f34e057a700 1 -- start start 2023-10-10T19:14:44.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34e057a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34d80c7190 0x7f34d815a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34e057a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d8157030 con 0x7f34d80c4580 2023-10-10T19:14:44.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34e057a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d8156420 con 0x7f34d80c6330 2023-10-10T19:14:44.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34e057a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d80c5000 con 0x7f34d80c7190 2023-10-10T19:14:44.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34ddb15700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34d80c7190 0x7f34d815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34ddb15700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34d80c7190 0x7f34d815a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45484/0 (socket says 172.21.15.175:45484) 2023-10-10T19:14:44.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34ddb15700 1 -- 172.21.15.175:0/3794336203 learned_addr learned my addr 172.21.15.175:0/3794336203 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:44.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34ddb15700 1 -- 172.21.15.175:0/3794336203 >> v1:172.21.15.191:6789/0 conn(0x7f34d80c6330 legacy=0x7f34d80c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34ddb15700 1 -- 172.21.15.175:0/3794336203 >> v1:172.21.15.175:6789/0 conn(0x7f34d80c4580 legacy=0x7f34d814fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34ddb15700 1 -- 172.21.15.175:0/3794336203 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8156700 con 0x7f34d80c7190 2023-10-10T19:14:44.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34ddb15700 1 --2- 172.21.15.175:0/3794336203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d815a030 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0099b0 tx=0x7f34cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55b41445eca56a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:44.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34dd314700 1 -- 172.21.15.175:0/3794336203 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00b8f0 con 0x7f34d80c7190 2023-10-10T19:14:44.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34dd314700 1 -- 172.21.15.175:0/3794336203 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc00ba50 con 0x7f34d80c7190 2023-10-10T19:14:44.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.313+0000 7f34dd314700 1 -- 172.21.15.175:0/3794336203 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00bdf0 con 0x7f34d80c7190 2023-10-10T19:14:44.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.314+0000 7f34e057a700 1 -- 172.21.15.175:0/3794336203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 msgr2=0x7f34d815a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.314+0000 7f34e057a700 1 --2- 172.21.15.175:0/3794336203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d815a030 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0099b0 tx=0x7f34cc009250 comp rx=0 tx=0).stop 2023-10-10T19:14:44.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.314+0000 7f34e057a700 1 -- 172.21.15.175:0/3794336203 shutdown_connections 2023-10-10T19:14:44.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.314+0000 7f34e057a700 1 --2- 172.21.15.175:0/3794336203 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d815a030 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.314+0000 7f34e057a700 1 -- 172.21.15.175:0/3794336203 >> 172.21.15.175:0/3794336203 conn(0x7f34d806ef20 msgr2=0x7f34d8063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:44.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.314+0000 7f34e057a700 1 -- 172.21.15.175:0/3794336203 shutdown_connections 2023-10-10T19:14:44.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.314+0000 7f34e057a700 1 -- 172.21.15.175:0/3794336203 wait complete. 2023-10-10T19:14:44.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.315+0000 7f34e057a700 1 Processor -- start 2023-10-10T19:14:44.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.315+0000 7f34e057a700 1 -- start start 2023-10-10T19:14:44.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34e057a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d80c4580 0x7f34d80cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34e057a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d80c6330 0x7f34d8169640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34e057a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d8169b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34e057a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8152090 con 0x7f34d80c7190 2023-10-10T19:14:44.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34e057a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80c96c0 con 0x7f34d80c4580 2023-10-10T19:14:44.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34e057a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8152c40 con 0x7f34d80c6330 2023-10-10T19:14:44.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d8169b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d8169b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45498/0 (socket says 172.21.15.175:45498) 2023-10-10T19:14:44.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 -- 172.21.15.175:0/4117323241 learned_addr learned my addr 172.21.15.175:0/4117323241 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:44.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34de316700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d80c4580 0x7f34d80cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34ddb15700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d80c6330 0x7f34d8169640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 -- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d80c6330 msgr2=0x7f34d8169640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d80c6330 0x7f34d8169640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 -- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d80c4580 msgr2=0x7f34d80cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d80c4580 0x7f34d80cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 -- 172.21.15.175:0/4117323241 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc0094f0 con 0x7f34d80c7190 2023-10-10T19:14:44.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34deb17700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d8169b80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f34c800c8e0 tx=0x7f34c8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:44.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34d67fc700 1 -- 172.21.15.175:0/4117323241 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8012630 con 0x7f34d80c7190 2023-10-10T19:14:44.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34d67fc700 1 -- 172.21.15.175:0/4117323241 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c8012790 con 0x7f34d80c7190 2023-10-10T19:14:44.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.316+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d8152f20 con 0x7f34d80c7190 2023-10-10T19:14:44.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.317+0000 7f34d67fc700 1 -- 172.21.15.175:0/4117323241 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8012b50 con 0x7f34d80c7190 2023-10-10T19:14:44.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.317+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d8151a40 con 0x7f34d80c7190 2023-10-10T19:14:44.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.317+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d8152240 con 0x7f34d80c7190 2023-10-10T19:14:44.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.318+0000 7f34d67fc700 1 -- 172.21.15.175:0/4117323241 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f34c801a3a0 con 0x7f34d80c7190 2023-10-10T19:14:44.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.318+0000 7f34d67fc700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34c4066f30 0x7f34c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.318+0000 7f34d67fc700 1 -- 172.21.15.175:0/4117323241 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f34c8084c70 con 0x7f34d80c7190 2023-10-10T19:14:44.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.318+0000 7f34de316700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34c4066f30 0x7f34c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.319+0000 7f34de316700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34c4066f30 0x7f34c40693e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f34d0003aa0 tx=0x7f34d0000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:44.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.322+0000 7f34d67fc700 1 -- 172.21.15.175:0/4117323241 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34d8152240 con 0x7f34d80c7190 2023-10-10T19:14:44.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.548+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34d8152240 con 0x7f34d80c7190 2023-10-10T19:14:44.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.548+0000 7f34d67fc700 1 -- 172.21.15.175:0/4117323241 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f34d8152240 con 0x7f34d80c7190 2023-10-10T19:14:44.550 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:44.550 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:43.733551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:44.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34c4066f30 msgr2=0x7f34c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34c4066f30 0x7f34c40693e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f34d0003aa0 tx=0x7f34d0000bc0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 msgr2=0x7f34d8169b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d8169b80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f34c800c8e0 tx=0x7f34c8001710 comp rx=0 tx=0).stop 2023-10-10T19:14:44.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 shutdown_connections 2023-10-10T19:14:44.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f34c4066f30 0x7f34c40693e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34d80c7190 0x7f34d8169b80 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34d80c4580 0x7f34d80cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 --2- 172.21.15.175:0/4117323241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34d80c6330 0x7f34d8169640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.550+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 >> 172.21.15.175:0/4117323241 conn(0x7f34d806ef20 msgr2=0x7f34d8063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:44.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.551+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 shutdown_connections 2023-10-10T19:14:44.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.551+0000 7f34e057a700 1 -- 172.21.15.175:0/4117323241 wait complete. 2023-10-10T19:14:44.555 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 229 2023-10-10T19:14:44.564 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-10T19:14:44.564 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:14:44.565 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:44.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.678+0000 7f0723df2700 1 Processor -- start 2023-10-10T19:14:44.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.678+0000 7f0723df2700 1 -- start start 2023-10-10T19:14:44.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f0723df2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 0x7f071c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f0723df2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f071c0b2390 0x7f071c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f0723df2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f071c068bd0 0x7f071c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f0723df2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071c150770 con 0x7f071c0b2390 2023-10-10T19:14:44.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f0723df2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071c1574b0 con 0x7f071c068bd0 2023-10-10T19:14:44.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f0723df2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071c0afad0 con 0x7f071c0b1fc0 2023-10-10T19:14:44.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072138d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f071c0b2390 0x7f071c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072138d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f071c0b2390 0x7f071c068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46760/0 (socket says 172.21.15.175:46760) 2023-10-10T19:14:44.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072138d700 1 -- 172.21.15.175:0/3680199834 learned_addr learned my addr 172.21.15.175:0/3680199834 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:44.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f0721b8e700 1 --2- 172.21.15.175:0/3680199834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 0x7f071c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072238f700 1 --2- 172.21.15.175:0/3680199834 >> v2:172.21.15.191:3300/0 conn(0x7f071c068bd0 0x7f071c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072138d700 1 -- 172.21.15.175:0/3680199834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 msgr2=0x7f071c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:44.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072138d700 1 --2- 172.21.15.175:0/3680199834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 0x7f071c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072138d700 1 -- 172.21.15.175:0/3680199834 >> v2:172.21.15.191:3300/0 conn(0x7f071c068bd0 msgr2=0x7f071c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:44.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072138d700 1 --2- 172.21.15.175:0/3680199834 >> v2:172.21.15.191:3300/0 conn(0x7f071c068bd0 0x7f071c0aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.679+0000 7f072138d700 1 -- 172.21.15.175:0/3680199834 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c0af4c0 con 0x7f071c0b2390 2023-10-10T19:14:44.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f072138d700 1 --2- 172.21.15.175:0/3680199834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c068690 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f071c0afa40 tx=0x7f070c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d3f13803b629e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:44.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0720b8c700 1 -- 172.21.15.175:0/3680199834 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c00b6d0 con 0x7f071c0b2390 2023-10-10T19:14:44.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0720b8c700 1 -- 172.21.15.175:0/3680199834 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c00b830 con 0x7f071c0b2390 2023-10-10T19:14:44.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0720b8c700 1 -- 172.21.15.175:0/3680199834 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c00bbf0 con 0x7f071c0b2390 2023-10-10T19:14:44.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 -- 172.21.15.175:0/3680199834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 msgr2=0x7f071c068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 --2- 172.21.15.175:0/3680199834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c068690 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f071c0afa40 tx=0x7f070c009250 comp rx=0 tx=0).stop 2023-10-10T19:14:44.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 -- 172.21.15.175:0/3680199834 shutdown_connections 2023-10-10T19:14:44.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 --2- 172.21.15.175:0/3680199834 >> v2:172.21.15.191:3300/0 conn(0x7f071c068bd0 0x7f071c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 --2- 172.21.15.175:0/3680199834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 0x7f071c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 --2- 172.21.15.175:0/3680199834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c068690 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 -- 172.21.15.175:0/3680199834 >> 172.21.15.175:0/3680199834 conn(0x7f071c063230 msgr2=0x7f071c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:44.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 -- 172.21.15.175:0/3680199834 shutdown_connections 2023-10-10T19:14:44.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.680+0000 7f0723df2700 1 -- 172.21.15.175:0/3680199834 wait complete. 2023-10-10T19:14:44.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.681+0000 7f0723df2700 1 Processor -- start 2023-10-10T19:14:44.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.681+0000 7f0723df2700 1 -- start start 2023-10-10T19:14:44.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f0723df2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f071c068bd0 0x7f071c157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f0723df2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 0x7f071c155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f0723df2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f0723df2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c0cd740 con 0x7f071c0b2390 2023-10-10T19:14:44.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f0723df2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c08a4c0 con 0x7f071c068bd0 2023-10-10T19:14:44.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f0723df2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071c0cd460 con 0x7f071c0b1fc0 2023-10-10T19:14:44.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f072238f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f072238f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46774/0 (socket says 172.21.15.175:46774) 2023-10-10T19:14:44.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f072238f700 1 -- 172.21.15.175:0/2243823708 learned_addr learned my addr 172.21.15.175:0/2243823708 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:44.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f0721b8e700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f071c068bd0 0x7f071c157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.682+0000 7f072138d700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 0x7f071c155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f072238f700 1 -- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 msgr2=0x7f071c155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f072238f700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 0x7f071c155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f072238f700 1 -- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f071c068bd0 msgr2=0x7f071c157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f072238f700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f071c068bd0 0x7f071c157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f072238f700 1 -- 172.21.15.175:0/2243823708 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c0094f0 con 0x7f071c0b2390 2023-10-10T19:14:44.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f072238f700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c1563d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f071400ea50 tx=0x7f07140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:44.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f0711ffb700 1 -- 172.21.15.175:0/2243823708 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071400ee90 con 0x7f071c0b2390 2023-10-10T19:14:44.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f0711ffb700 1 -- 172.21.15.175:0/2243823708 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071400c7f0 con 0x7f071c0b2390 2023-10-10T19:14:44.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f0711ffb700 1 -- 172.21.15.175:0/2243823708 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071400cbb0 con 0x7f071c0b2390 2023-10-10T19:14:44.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c1550d0 con 0x7f071c0b2390 2023-10-10T19:14:44.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.683+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c155690 con 0x7f071c0b2390 2023-10-10T19:14:44.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.684+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c067570 con 0x7f071c0b2390 2023-10-10T19:14:44.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.684+0000 7f0711ffb700 1 -- 172.21.15.175:0/2243823708 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0714013470 con 0x7f071c0b2390 2023-10-10T19:14:44.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.685+0000 7f0711ffb700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0700066f30 0x7f07000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:44.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.685+0000 7f0711ffb700 1 -- 172.21.15.175:0/2243823708 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f0714084cf0 con 0x7f071c0b2390 2023-10-10T19:14:44.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.685+0000 7f0721b8e700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0700066f30 0x7f07000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:44.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.685+0000 7f0721b8e700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0700066f30 0x7f07000693e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f071800f570 tx=0x7f071800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:44.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.689+0000 7f0711ffb700 1 -- 172.21.15.175:0/2243823708 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f07140518f0 con 0x7f071c0b2390 2023-10-10T19:14:44.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.915+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f071c069060 con 0x7f071c0b2390 2023-10-10T19:14:44.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.915+0000 7f0711ffb700 1 -- 172.21.15.175:0/2243823708 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f0714054f10 con 0x7f071c0b2390 2023-10-10T19:14:44.917 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:44.917 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:43.733551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:44.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0700066f30 msgr2=0x7f07000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0700066f30 0x7f07000693e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f071800f570 tx=0x7f071800b040 comp rx=0 tx=0).stop 2023-10-10T19:14:44.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 msgr2=0x7f071c1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:44.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c1563d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f071400ea50 tx=0x7f07140091a0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 shutdown_connections 2023-10-10T19:14:44.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0700066f30 0x7f07000693e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f071c0b2390 0x7f071c1563d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f071c068bd0 0x7f071c157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 --2- 172.21.15.175:0/2243823708 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f071c0b1fc0 0x7f071c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:44.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 >> 172.21.15.175:0/2243823708 conn(0x7f071c063230 msgr2=0x7f071c14fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:44.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 shutdown_connections 2023-10-10T19:14:44.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:44.917+0000 7f0723df2700 1 -- 172.21.15.175:0/2243823708 wait complete. 2023-10-10T19:14:44.921 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 229 2023-10-10T19:14:44.931 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 229, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 229, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6080": { "addr": "172.21.15.191:6836/4084912544", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 4084912544, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 4084912544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6080, "incarnation": 229, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 } }, "last_failure": 0, "last_failure_osd_epoch": 89, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:14:43.733551+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 6080, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:14:46.932 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:47.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.058+0000 7f82a8367700 1 Processor -- start 2023-10-10T19:14:47.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.059+0000 7f82a8367700 1 -- start start 2023-10-10T19:14:47.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.059+0000 7f82a8367700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f82a0156c30 0x7f82a00c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:47.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.059+0000 7f82a8367700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00c6df0 con 0x7f82a006bf00 2023-10-10T19:14:47.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.059+0000 7f82a8367700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00c6920 con 0x7f82a00c5e40 2023-10-10T19:14:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.059+0000 7f82a8367700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a01560d0 con 0x7f82a0156c30 2023-10-10T19:14:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a6904700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f82a0156c30 0x7f82a00c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a6904700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f82a0156c30 0x7f82a00c47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46784/0 (socket says 172.21.15.175:46784) 2023-10-10T19:14:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a6904700 1 -- 172.21.15.175:0/1108066517 learned_addr learned my addr 172.21.15.175:0/1108066517 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:47.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a5902700 1 --1- 172.21.15.175:0/1108066517 >> v1:172.21.15.175:6789/0 conn(0x7f82a006bf00 0x7f82a0156520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36382/0 (socket says 172.21.15.175:36382) 2023-10-10T19:14:47.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a6904700 1 -- 172.21.15.175:0/1108066517 >> v1:172.21.15.191:6789/0 conn(0x7f82a00c5e40 legacy=0x7f82a00c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:47.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a6904700 1 -- 172.21.15.175:0/1108066517 >> v1:172.21.15.175:6789/0 conn(0x7f82a006bf00 legacy=0x7f82a0156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:47.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a6904700 1 -- 172.21.15.175:0/1108066517 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a01597e0 con 0x7f82a0156c30 2023-10-10T19:14:47.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a6904700 1 --2- 172.21.15.175:0/1108066517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a00c47d0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f82980099b0 tx=0x7f8298009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1e7946bede6eee5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:47.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a5101700 1 -- 172.21.15.175:0/1108066517 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829800b8f0 con 0x7f82a0156c30 2023-10-10T19:14:47.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a5101700 1 -- 172.21.15.175:0/1108066517 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829800ba50 con 0x7f82a0156c30 2023-10-10T19:14:47.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.060+0000 7f82a5101700 1 -- 172.21.15.175:0/1108066517 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829800bdf0 con 0x7f82a0156c30 2023-10-10T19:14:47.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.061+0000 7f82a8367700 1 -- 172.21.15.175:0/1108066517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 msgr2=0x7f82a00c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:47.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.061+0000 7f82a8367700 1 --2- 172.21.15.175:0/1108066517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a00c47d0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f82980099b0 tx=0x7f8298009250 comp rx=0 tx=0).stop 2023-10-10T19:14:47.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.061+0000 7f82a8367700 1 -- 172.21.15.175:0/1108066517 shutdown_connections 2023-10-10T19:14:47.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.061+0000 7f82a8367700 1 --2- 172.21.15.175:0/1108066517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a00c47d0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:47.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.061+0000 7f82a8367700 1 -- 172.21.15.175:0/1108066517 >> 172.21.15.175:0/1108066517 conn(0x7f82a0068900 msgr2=0x7f82a0068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:47.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.061+0000 7f82a8367700 1 -- 172.21.15.175:0/1108066517 shutdown_connections 2023-10-10T19:14:47.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.061+0000 7f82a8367700 1 -- 172.21.15.175:0/1108066517 wait complete. 2023-10-10T19:14:47.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.062+0000 7f82a8367700 1 Processor -- start 2023-10-10T19:14:47.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.062+0000 7f82a8367700 1 -- start start 2023-10-10T19:14:47.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a8367700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82a006bf00 0x7f82a00c15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:47.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a8367700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82a00c5e40 0x7f82a0166040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:47.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a8367700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a015a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:47.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a8367700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a014cf60 con 0x7f82a0156c30 2023-10-10T19:14:47.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a8367700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a00c8f90 con 0x7f82a00c5e40 2023-10-10T19:14:47.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a8367700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a016a150 con 0x7f82a006bf00 2023-10-10T19:14:47.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a015a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:47.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a015a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46786/0 (socket says 172.21.15.175:46786) 2023-10-10T19:14:47.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 -- 172.21.15.175:0/75720363 learned_addr learned my addr 172.21.15.175:0/75720363 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:47.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a5902700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82a00c5e40 0x7f82a0166040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:47.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6103700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82a006bf00 0x7f82a00c15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:47.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 -- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82a006bf00 msgr2=0x7f82a00c15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:47.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82a006bf00 0x7f82a00c15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:47.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 -- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82a00c5e40 msgr2=0x7f82a0166040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:47.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82a00c5e40 0x7f82a0166040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:47.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 -- 172.21.15.175:0/75720363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82980094f0 con 0x7f82a0156c30 2023-10-10T19:14:47.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a5902700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82a00c5e40 0x7f82a0166040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:47.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a6904700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a015a5a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f82980094c0 tx=0x7f8298017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:47.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82967fc700 1 -- 172.21.15.175:0/75720363 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829801a630 con 0x7f82a0156c30 2023-10-10T19:14:47.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82967fc700 1 -- 172.21.15.175:0/75720363 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829801a790 con 0x7f82a0156c30 2023-10-10T19:14:47.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82967fc700 1 -- 172.21.15.175:0/75720363 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829801ab30 con 0x7f82a0156c30 2023-10-10T19:14:47.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a00c2040 con 0x7f82a0156c30 2023-10-10T19:14:47.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.063+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a014d3b0 con 0x7f82a0156c30 2023-10-10T19:14:47.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.064+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a014d0d0 con 0x7f82a0156c30 2023-10-10T19:14:47.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.065+0000 7f82967fc700 1 -- 172.21.15.175:0/75720363 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f82980193b0 con 0x7f82a0156c30 2023-10-10T19:14:47.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.065+0000 7f82967fc700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8284066f60 0x7f8284069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:47.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.065+0000 7f82967fc700 1 -- 172.21.15.175:0/75720363 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f82980868c0 con 0x7f82a0156c30 2023-10-10T19:14:47.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.065+0000 7f82a6103700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8284066f60 0x7f8284069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:47.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.066+0000 7f82a6103700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8284066f60 0x7f8284069410 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f829c00d150 tx=0x7f829c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:47.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.069+0000 7f82967fc700 1 -- 172.21.15.175:0/75720363 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f82a014d0d0 con 0x7f82a0156c30 2023-10-10T19:14:47.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.292+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82a014d0a0 con 0x7f82a0156c30 2023-10-10T19:14:47.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.293+0000 7f82967fc700 1 -- 172.21.15.175:0/75720363 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+5440 (secure 0 0 0) 0x7f8298053560 con 0x7f82a0156c30 2023-10-10T19:14:47.295 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:47.295 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:43.733551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:47.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8284066f60 msgr2=0x7f8284069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:47.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8284066f60 0x7f8284069410 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f829c00d150 tx=0x7f829c009040 comp rx=0 tx=0).stop 2023-10-10T19:14:47.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 msgr2=0x7f82a015a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:47.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a015a5a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f82980094c0 tx=0x7f8298017000 comp rx=0 tx=0).stop 2023-10-10T19:14:47.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 shutdown_connections 2023-10-10T19:14:47.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8284066f60 0x7f8284069410 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:47.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82a0156c30 0x7f82a015a5a0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:47.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82a006bf00 0x7f82a00c15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:47.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 --2- 172.21.15.175:0/75720363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82a00c5e40 0x7f82a0166040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:47.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 >> 172.21.15.175:0/75720363 conn(0x7f82a0068900 msgr2=0x7f82a0156520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:47.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 shutdown_connections 2023-10-10T19:14:47.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:47.295+0000 7f82a8367700 1 -- 172.21.15.175:0/75720363 wait complete. 2023-10-10T19:14:47.299 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 229 2023-10-10T19:14:47.308 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:49.309 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:49.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.431+0000 7fd72b22f700 1 Processor -- start 2023-10-10T19:14:49.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.431+0000 7fd72b22f700 1 -- start start 2023-10-10T19:14:49.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd72b22f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd7240c5850 0x7fd7240c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:49.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd72b22f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c4f20 0x7fd7240c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:49.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd72b22f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7240c6fb0 con 0x7fd7240c12d0 2023-10-10T19:14:49.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd72b22f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd724063ea0 con 0x7fd7240c5850 2023-10-10T19:14:49.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd72b22f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd724155f30 con 0x7fd7240c4f20 2023-10-10T19:14:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd71bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd7240c12d0 0x7fd724156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36394/0 (socket says 172.21.15.175:36394) 2023-10-10T19:14:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd71bfff700 1 -- 172.21.15.175:0/3241054249 learned_addr learned my addr 172.21.15.175:0/3241054249 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd7297cc700 1 --2- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c4f20 0x7fd7240c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.432+0000 7fd728fcb700 1 --2- 172.21.15.175:0/3241054249 >> v2:172.21.15.191:3300/0 conn(0x7fd7240c5850 0x7fd7240c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:49.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd71b7fe700 1 -- 172.21.15.175:0/3241054249 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306688098 0 0) 0x7fd7240c6fb0 con 0x7fd7240c12d0 2023-10-10T19:14:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd71b7fe700 1 -- 172.21.15.175:0/3241054249 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd70c003570 con 0x7fd7240c12d0 2023-10-10T19:14:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd728fcb700 1 -- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c4f20 msgr2=0x7fd7240c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd728fcb700 1 --2- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c4f20 0x7fd7240c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd728fcb700 1 -- 172.21.15.175:0/3241054249 >> v1:172.21.15.175:6789/0 conn(0x7fd7240c12d0 legacy=0x7fd724156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:49.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd728fcb700 1 -- 172.21.15.175:0/3241054249 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7240c6160 con 0x7fd7240c5850 2023-10-10T19:14:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd7297cc700 1 --2- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c4f20 0x7fd7240c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:14:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd728fcb700 1 --2- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c5850 0x7fd7240c5c20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd720009e60 tx=0x7fd720009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c07e7c865e55a55 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd71b7fe700 1 -- 172.21.15.175:0/3241054249 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72000e070 con 0x7fd7240c5850 2023-10-10T19:14:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd71b7fe700 1 -- 172.21.15.175:0/3241054249 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72000bab0 con 0x7fd7240c5850 2023-10-10T19:14:49.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.433+0000 7fd71b7fe700 1 -- 172.21.15.175:0/3241054249 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72000be30 con 0x7fd7240c5850 2023-10-10T19:14:49.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.434+0000 7fd72b22f700 1 -- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c5850 msgr2=0x7fd7240c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:49.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.434+0000 7fd72b22f700 1 --2- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c5850 0x7fd7240c5c20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd720009e60 tx=0x7fd720009780 comp rx=0 tx=0).stop 2023-10-10T19:14:49.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.434+0000 7fd72b22f700 1 -- 172.21.15.175:0/3241054249 shutdown_connections 2023-10-10T19:14:49.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.434+0000 7fd72b22f700 1 --2- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c4f20 0x7fd7240c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.434+0000 7fd72b22f700 1 --2- 172.21.15.175:0/3241054249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c5850 0x7fd7240c5c20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.434+0000 7fd72b22f700 1 -- 172.21.15.175:0/3241054249 >> 172.21.15.175:0/3241054249 conn(0x7fd72406dc70 msgr2=0x7fd724063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:49.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.434+0000 7fd72b22f700 1 -- 172.21.15.175:0/3241054249 shutdown_connections 2023-10-10T19:14:49.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.434+0000 7fd72b22f700 1 -- 172.21.15.175:0/3241054249 wait complete. 2023-10-10T19:14:49.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.435+0000 7fd72b22f700 1 Processor -- start 2023-10-10T19:14:49.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.435+0000 7fd72b22f700 1 -- start start 2023-10-10T19:14:49.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd72b22f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7240c12d0 0x7fd7240c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:49.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd72b22f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c4f20 0x7fd72414e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:49.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd72b22f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c5850 0x7fd72414ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:49.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd72b22f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd724151400 con 0x7fd7240c12d0 2023-10-10T19:14:49.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd72b22f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd724166800 con 0x7fd7240c4f20 2023-10-10T19:14:49.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd72b22f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd724150600 con 0x7fd7240c5850 2023-10-10T19:14:49.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7240c12d0 0x7fd7240c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:49.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7240c12d0 0x7fd7240c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46796/0 (socket says 172.21.15.175:46796) 2023-10-10T19:14:49.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 -- 172.21.15.175:0/180305938 learned_addr learned my addr 172.21.15.175:0/180305938 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:49.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd71bfff700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c4f20 0x7fd72414e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:49.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd7297cc700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c5850 0x7fd72414ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:49.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 -- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c5850 msgr2=0x7fd72414ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:49.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c5850 0x7fd72414ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 -- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c4f20 msgr2=0x7fd72414e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:49.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c4f20 0x7fd72414e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 -- 172.21.15.175:0/180305938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c009420 con 0x7fd7240c12d0 2023-10-10T19:14:49.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd728fcb700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7240c12d0 0x7fd7240c8d50 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fd7200097c0 tx=0x7fd720019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:49.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd718ff9700 1 -- 172.21.15.175:0/180305938 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720012a40 con 0x7fd7240c12d0 2023-10-10T19:14:49.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd718ff9700 1 -- 172.21.15.175:0/180305938 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd720021070 con 0x7fd7240c12d0 2023-10-10T19:14:49.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd720009a20 con 0x7fd7240c12d0 2023-10-10T19:14:49.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd718ff9700 1 -- 172.21.15.175:0/180305938 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720012e10 con 0x7fd7240c12d0 2023-10-10T19:14:49.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.436+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd724165ed0 con 0x7fd7240c12d0 2023-10-10T19:14:49.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.437+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd724064200 con 0x7fd7240c12d0 2023-10-10T19:14:49.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.438+0000 7fd718ff9700 1 -- 172.21.15.175:0/180305938 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd720023030 con 0x7fd7240c12d0 2023-10-10T19:14:49.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.438+0000 7fd718ff9700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd710066f60 0x7fd710069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:49.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.438+0000 7fd718ff9700 1 -- 172.21.15.175:0/180305938 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7fd7200868f0 con 0x7fd7240c12d0 2023-10-10T19:14:49.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.438+0000 7fd71bfff700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd710066f60 0x7fd710069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:49.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.439+0000 7fd71bfff700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd710066f60 0x7fd710069410 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd724159480 tx=0x7fd714009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:49.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.442+0000 7fd718ff9700 1 -- 172.21.15.175:0/180305938 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd720053650 con 0x7fd7240c12d0 2023-10-10T19:14:49.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.670+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd724067570 con 0x7fd7240c12d0 2023-10-10T19:14:49.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.671+0000 7fd718ff9700 1 -- 172.21.15.175:0/180305938 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+5440 (secure 0 0 0) 0x7fd720056c70 con 0x7fd7240c12d0 2023-10-10T19:14:49.673 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:49.673 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:43.733551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:49.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd710066f60 msgr2=0x7fd710069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:49.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd710066f60 0x7fd710069410 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd724159480 tx=0x7fd714009040 comp rx=0 tx=0).stop 2023-10-10T19:14:49.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7240c12d0 msgr2=0x7fd7240c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:49.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7240c12d0 0x7fd7240c8d50 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fd7200097c0 tx=0x7fd720019000 comp rx=0 tx=0).stop 2023-10-10T19:14:49.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 shutdown_connections 2023-10-10T19:14:49.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd710066f60 0x7fd710069410 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7240c5850 0x7fd72414ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7240c12d0 0x7fd7240c8d50 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 --2- 172.21.15.175:0/180305938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7240c4f20 0x7fd72414e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:49.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 >> 172.21.15.175:0/180305938 conn(0x7fd72406dc70 msgr2=0x7fd724153210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:49.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 shutdown_connections 2023-10-10T19:14:49.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:49.673+0000 7fd72b22f700 1 -- 172.21.15.175:0/180305938 wait complete. 2023-10-10T19:14:49.678 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 229 2023-10-10T19:14:49.687 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:51.688 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:51.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.814+0000 7fb92311a700 1 Processor -- start 2023-10-10T19:14:51.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.814+0000 7fb92311a700 1 -- start start 2023-10-10T19:14:51.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.814+0000 7fb92311a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb91c0b1fc0 0x7fb91c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:51.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.814+0000 7fb92311a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:51.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.814+0000 7fb92311a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb91c068bd0 0x7fb91c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:51.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.814+0000 7fb92311a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91c150770 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.814+0000 7fb92311a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91c1574b0 con 0x7fb91c068bd0 2023-10-10T19:14:51.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.814+0000 7fb92311a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0afad0 con 0x7fb91c0b2390 2023-10-10T19:14:51.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb91c0b1fc0 0x7fb91c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:51.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb91c0b1fc0 0x7fb91c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46810/0 (socket says 172.21.15.175:46810) 2023-10-10T19:14:51.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 -- 172.21.15.175:0/102164421 learned_addr learned my addr 172.21.15.175:0/102164421 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:51.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb91bfff700 1 --2- 172.21.15.175:0/102164421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:51.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb9216b7700 1 --2- 172.21.15.175:0/102164421 >> v2:172.21.15.191:3300/0 conn(0x7fb91c068bd0 0x7fb91c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:51.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 -- 172.21.15.175:0/102164421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 msgr2=0x7fb91c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:51.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 --2- 172.21.15.175:0/102164421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:51.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 -- 172.21.15.175:0/102164421 >> v2:172.21.15.191:3300/0 conn(0x7fb91c068bd0 msgr2=0x7fb91c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:51.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 --2- 172.21.15.175:0/102164421 >> v2:172.21.15.191:3300/0 conn(0x7fb91c068bd0 0x7fb91c0aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:51.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 -- 172.21.15.175:0/102164421 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c0af4c0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb920eb6700 1 --2- 172.21.15.175:0/102164421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c155fb0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0afa40 tx=0x7fb910009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e3dda8fa4e4327 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:51.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb91b7fe700 1 -- 172.21.15.175:0/102164421 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91000b6d0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb91b7fe700 1 -- 172.21.15.175:0/102164421 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91000b830 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.815+0000 7fb91b7fe700 1 -- 172.21.15.175:0/102164421 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91000bbf0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 -- 172.21.15.175:0/102164421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 msgr2=0x7fb91c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:51.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 --2- 172.21.15.175:0/102164421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c155fb0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0afa40 tx=0x7fb910009250 comp rx=0 tx=0).stop 2023-10-10T19:14:51.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 -- 172.21.15.175:0/102164421 shutdown_connections 2023-10-10T19:14:51.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 --2- 172.21.15.175:0/102164421 >> v2:172.21.15.191:3300/0 conn(0x7fb91c068bd0 0x7fb91c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 --2- 172.21.15.175:0/102164421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c155fb0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 --2- 172.21.15.175:0/102164421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 -- 172.21.15.175:0/102164421 >> 172.21.15.175:0/102164421 conn(0x7fb91c063230 msgr2=0x7fb91c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 -- 172.21.15.175:0/102164421 shutdown_connections 2023-10-10T19:14:51.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.816+0000 7fb92311a700 1 -- 172.21.15.175:0/102164421 wait complete. 2023-10-10T19:14:51.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.817+0000 7fb92311a700 1 Processor -- start 2023-10-10T19:14:51.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.817+0000 7fb92311a700 1 -- start start 2023-10-10T19:14:51.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb92311a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb91c068bd0 0x7fb91c155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:51.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb92311a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c0c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:51.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb92311a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c1660f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:51.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb92311a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0c59c0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb92311a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0c6820 con 0x7fb91c068bd0 2023-10-10T19:14:51.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb92311a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0c5c10 con 0x7fb91c0b2390 2023-10-10T19:14:51.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c0c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:51.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c0c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46812/0 (socket says 172.21.15.175:46812) 2023-10-10T19:14:51.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 -- 172.21.15.175:0/3153681546 learned_addr learned my addr 172.21.15.175:0/3153681546 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:51.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb9216b7700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c1660f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:51.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb920eb6700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb91c068bd0 0x7fb91c155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:51.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 -- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 msgr2=0x7fb91c1660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:51.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c1660f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:51.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 -- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb91c068bd0 msgr2=0x7fb91c155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:51.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb91c068bd0 0x7fb91c155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:51.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 -- 172.21.15.175:0/3153681546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9100094f0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb920eb6700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb91c068bd0 0x7fb91c155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:51.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb9216b7700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c1660f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:51.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.818+0000 7fb91bfff700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c0c8600 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb90800e930 tx=0x7fb9080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:51.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.819+0000 7fb918ff9700 1 -- 172.21.15.175:0/3153681546 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90800eed0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.819+0000 7fb918ff9700 1 -- 172.21.15.175:0/3153681546 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90800c850 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.819+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91c0c5ef0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.819+0000 7fb918ff9700 1 -- 172.21.15.175:0/3153681546 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90800cbf0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.819+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91c0c64b0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.820+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91c15c0e0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.820+0000 7fb918ff9700 1 -- 172.21.15.175:0/3153681546 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb908013470 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.820+0000 7fb918ff9700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:51.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.821+0000 7fb918ff9700 1 -- 172.21.15.175:0/3153681546 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7fb908084cf0 con 0x7fb91c0b1fc0 2023-10-10T19:14:51.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.821+0000 7fb920eb6700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:51.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.821+0000 7fb920eb6700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0afa40 tx=0x7fb910017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:51.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:51.825+0000 7fb918ff9700 1 -- 172.21.15.175:0/3153681546 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb908089070 con 0x7fb91c0b1fc0 2023-10-10T19:14:52.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.049+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb91c067570 con 0x7fb91c0b1fc0 2023-10-10T19:14:52.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.050+0000 7fb918ff9700 1 -- 172.21.15.175:0/3153681546 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5442 (secure 0 0 0) 0x7fb908051990 con 0x7fb91c0b1fc0 2023-10-10T19:14:52.052 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:52.052 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:51.808261+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:resolve","state_seq":10,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:52.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 msgr2=0x7fb9000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:52.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0afa40 tx=0x7fb910017000 comp rx=0 tx=0).stop 2023-10-10T19:14:52.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 msgr2=0x7fb91c0c8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:52.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c0c8600 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb90800e930 tx=0x7fb9080091a0 comp rx=0 tx=0).stop 2023-10-10T19:14:52.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 shutdown_connections 2023-10-10T19:14:52.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb900066f30 0x7fb9000693e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:52.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb91c0b2390 0x7fb91c1660f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:52.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb91c068bd0 0x7fb91c155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:52.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 --2- 172.21.15.175:0/3153681546 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb91c0b1fc0 0x7fb91c0c8600 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:52.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 >> 172.21.15.175:0/3153681546 conn(0x7fb91c063230 msgr2=0x7fb91c15b430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:52.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 shutdown_connections 2023-10-10T19:14:52.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:52.052+0000 7fb92311a700 1 -- 172.21.15.175:0/3153681546 wait complete. 2023-10-10T19:14:52.057 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 230 2023-10-10T19:14:52.065 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:53.402 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:14:53.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:14:54.066 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:54.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.191+0000 7f36323b9700 1 Processor -- start 2023-10-10T19:14:54.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.191+0000 7f36323b9700 1 -- start start 2023-10-10T19:14:54.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f36323b9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f362c068c00 0x7f362c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:54.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f36323b9700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362c067fa0 con 0x7f362c08a100 2023-10-10T19:14:54.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f36323b9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362c156b70 con 0x7f362c14fd90 2023-10-10T19:14:54.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f36323b9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362c0c5600 con 0x7f362c068c00 2023-10-10T19:14:54.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f3630bb6700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f362c08a100 0x7f362c0684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:36408/0 (socket says 172.21.15.175:36408) 2023-10-10T19:14:54.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f3630bb6700 1 -- 172.21.15.175:0/2453819543 learned_addr learned my addr 172.21.15.175:0/2453819543 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:54.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f3631bb8700 1 --2- 172.21.15.175:0/2453819543 >> v2:172.21.15.191:3300/0 conn(0x7f362c068c00 0x7f362c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:54.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870341490 0 0) 0x7f362c067fa0 con 0x7f362c08a100 2023-10-10T19:14:54.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.192+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3618003570 con 0x7f362c08a100 2023-10-10T19:14:54.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 369144391 0 0) 0x7f3618003570 con 0x7f362c08a100 2023-10-10T19:14:54.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 >> v2:172.21.15.191:3300/0 conn(0x7f362c068c00 msgr2=0x7f362c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:54.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 --2- 172.21.15.175:0/2453819543 >> v2:172.21.15.191:3300/0 conn(0x7f362c068c00 0x7f362c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 >> v1:172.21.15.191:6789/0 conn(0x7f362c14fd90 legacy=0x7f362c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:54.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c150160 con 0x7f362c08a100 2023-10-10T19:14:54.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f361c001470 con 0x7f362c08a100 2023-10-10T19:14:54.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 >> v1:172.21.15.175:6789/0 conn(0x7f362c08a100 legacy=0x7f362c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:54.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3618003a60 0x7f3618003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:54.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3618004810 0x7f3618004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:54.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36180051c0 0x7f361800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:54.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0c5600 con 0x7f36180051c0 2023-10-10T19:14:54.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361800c0d0 con 0x7f3618003a60 2023-10-10T19:14:54.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361800c290 con 0x7f3618004810 2023-10-10T19:14:54.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3631bb8700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36180051c0 0x7f361800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3630bb6700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3618004810 0x7f3618004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f36313b7700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3618003a60 0x7f3618003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3631bb8700 1 -- 172.21.15.175:0/2453819543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3618004810 msgr2=0x7f3618004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3631bb8700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3618004810 0x7f3618004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3631bb8700 1 -- 172.21.15.175:0/2453819543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3618003a60 msgr2=0x7f3618003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3631bb8700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3618003a60 0x7f3618003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3631bb8700 1 -- 172.21.15.175:0/2453819543 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3618003570 con 0x7f36180051c0 2023-10-10T19:14:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3631bb8700 1 -- 172.21.15.175:0/2453819543 reap_dead start 2023-10-10T19:14:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3631bb8700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36180051c0 0x7f361800ba60 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f362400e210 tx=0x7f362400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f2d160e758f254 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362400eac0 con 0x7f36180051c0 2023-10-10T19:14:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362400f040 con 0x7f36180051c0 2023-10-10T19:14:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.193+0000 7f3623fff700 1 -- 172.21.15.175:0/2453819543 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362400ee70 con 0x7f36180051c0 2023-10-10T19:14:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.194+0000 7f36323b9700 1 -- 172.21.15.175:0/2453819543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36180051c0 msgr2=0x7f361800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.194+0000 7f36323b9700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36180051c0 0x7f361800ba60 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f362400e210 tx=0x7f362400b040 comp rx=0 tx=0).stop 2023-10-10T19:14:54.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.194+0000 7f36323b9700 1 -- 172.21.15.175:0/2453819543 shutdown_connections 2023-10-10T19:14:54.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.194+0000 7f36323b9700 1 --2- 172.21.15.175:0/2453819543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36180051c0 0x7f361800ba60 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.194+0000 7f36323b9700 1 -- 172.21.15.175:0/2453819543 >> 172.21.15.175:0/2453819543 conn(0x7f362c063d00 msgr2=0x7f362c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:54.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.194+0000 7f36323b9700 1 -- 172.21.15.175:0/2453819543 shutdown_connections 2023-10-10T19:14:54.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.194+0000 7f36323b9700 1 -- 172.21.15.175:0/2453819543 wait complete. 2023-10-10T19:14:54.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.195+0000 7f36323b9700 1 Processor -- start 2023-10-10T19:14:54.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.195+0000 7f36323b9700 1 -- start start 2023-10-10T19:14:54.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f36323b9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36180051c0 0x7f362c0c1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:54.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f36323b9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f362c158c70 0x7f362c14b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:54.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f36323b9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f362c0c1980 0x7f362c14bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:54.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f36323b9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c1583a0 con 0x7f362c0c1980 2023-10-10T19:14:54.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f36323b9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c158830 con 0x7f362c158c70 2023-10-10T19:14:54.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f36323b9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362c150990 con 0x7f36180051c0 2023-10-10T19:14:54.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f3631bb8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f362c0c1980 0x7f362c14bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:54.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f3631bb8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f362c0c1980 0x7f362c14bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46816/0 (socket says 172.21.15.175:46816) 2023-10-10T19:14:54.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f3631bb8700 1 -- 172.21.15.175:0/931578124 learned_addr learned my addr 172.21.15.175:0/931578124 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:54.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f3630bb6700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f362c158c70 0x7f362c14b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:54.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.196+0000 7f36313b7700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36180051c0 0x7f362c0c1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:54.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f3631bb8700 1 -- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36180051c0 msgr2=0x7f362c0c1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:54.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f3631bb8700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36180051c0 0x7f362c0c1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f3631bb8700 1 -- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f362c158c70 msgr2=0x7f362c14b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:54.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f3631bb8700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f362c158c70 0x7f362c14b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f3631bb8700 1 -- 172.21.15.175:0/931578124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c157a70 con 0x7f362c0c1980 2023-10-10T19:14:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f3631bb8700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f362c0c1980 0x7f362c14bda0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f362400afd0 tx=0x7f3624019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f36217fa700 1 -- 172.21.15.175:0/931578124 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362401d660 con 0x7f362c0c1980 2023-10-10T19:14:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f36217fa700 1 -- 172.21.15.175:0/931578124 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624022070 con 0x7f362c0c1980 2023-10-10T19:14:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362c157d50 con 0x7f362c0c1980 2023-10-10T19:14:54.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f36217fa700 1 -- 172.21.15.175:0/931578124 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362401da30 con 0x7f362c0c1980 2023-10-10T19:14:54.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.197+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362c157730 con 0x7f362c0c1980 2023-10-10T19:14:54.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.198+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362c1581f0 con 0x7f362c0c1980 2023-10-10T19:14:54.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.198+0000 7f36217fa700 1 -- 172.21.15.175:0/931578124 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3624003020 con 0x7f362c0c1980 2023-10-10T19:14:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.199+0000 7f36217fa700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3610066f30 0x7f36100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.199+0000 7f36217fa700 1 -- 172.21.15.175:0/931578124 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f3624024070 con 0x7f362c0c1980 2023-10-10T19:14:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.199+0000 7f36313b7700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3610066f30 0x7f36100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.199+0000 7f36313b7700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3610066f30 0x7f36100693e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3628007e40 tx=0x7f3628007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:54.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.203+0000 7f36217fa700 1 -- 172.21.15.175:0/931578124 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3624053d00 con 0x7f362c0c1980 2023-10-10T19:14:54.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.426+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f362c067570 con 0x7f362c0c1980 2023-10-10T19:14:54.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.426+0000 7f36217fa700 1 -- 172.21.15.175:0/931578124 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5442 (secure 0 0 0) 0x7f3624018020 con 0x7f362c0c1980 2023-10-10T19:14:54.428 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:54.428 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:51.808261+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:resolve","state_seq":10,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:54.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.428+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3610066f30 msgr2=0x7f36100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:54.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.428+0000 7f36323b9700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3610066f30 0x7f36100693e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3628007e40 tx=0x7f3628007d30 comp rx=0 tx=0).stop 2023-10-10T19:14:54.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f362c0c1980 msgr2=0x7f362c14bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:54.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f362c0c1980 0x7f362c14bda0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f362400afd0 tx=0x7f3624019000 comp rx=0 tx=0).stop 2023-10-10T19:14:54.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 shutdown_connections 2023-10-10T19:14:54.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3610066f30 0x7f36100693e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f362c0c1980 0x7f362c14bda0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36180051c0 0x7f362c0c1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 --2- 172.21.15.175:0/931578124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f362c158c70 0x7f362c14b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:54.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 >> 172.21.15.175:0/931578124 conn(0x7f362c063d00 msgr2=0x7f362c0c5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:54.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 shutdown_connections 2023-10-10T19:14:54.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:54.429+0000 7f36323b9700 1 -- 172.21.15.175:0/931578124 wait complete. 2023-10-10T19:14:54.432 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 230 2023-10-10T19:14:54.442 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:14:56.443 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:56.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.568+0000 7f364e7e7700 1 Processor -- start 2023-10-10T19:14:56.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.569+0000 7f364e7e7700 1 -- start start 2023-10-10T19:14:56.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.569+0000 7f364e7e7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f364814c840 0x7f364806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.569+0000 7f364e7e7700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3648155ca0 con 0x7f36480c52a0 2023-10-10T19:14:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.569+0000 7f364e7e7700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3648156b00 con 0x7f364814cc10 2023-10-10T19:14:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.569+0000 7f364e7e7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f364806d700 con 0x7f364814c840 2023-10-10T19:14:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.569+0000 7f364cd84700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f36480c52a0 0x7f3648152970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:34058/0 (socket says 172.21.15.175:34058) 2023-10-10T19:14:56.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.569+0000 7f364cd84700 1 -- 172.21.15.175:0/3412404641 learned_addr learned my addr 172.21.15.175:0/3412404641 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:14:56.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3647fff700 1 --2- 172.21.15.175:0/3412404641 >> v2:172.21.15.191:3300/0 conn(0x7f364814c840 0x7f364806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:56.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112830783 0 0) 0x7f3648155ca0 con 0x7f36480c52a0 2023-10-10T19:14:56.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630003570 con 0x7f36480c52a0 2023-10-10T19:14:56.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2304580559 0 0) 0x7f3630003570 con 0x7f36480c52a0 2023-10-10T19:14:56.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 >> v2:172.21.15.191:3300/0 conn(0x7f364814c840 msgr2=0x7f364806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:56.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 --2- 172.21.15.175:0/3412404641 >> v2:172.21.15.191:3300/0 conn(0x7f364814c840 0x7f364806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 >> v1:172.21.15.191:6789/0 conn(0x7f364814cc10 legacy=0x7f36480c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:56.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648155ef0 con 0x7f36480c52a0 2023-10-10T19:14:56.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f36400017f0 con 0x7f36480c52a0 2023-10-10T19:14:56.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3647fff700 1 --2- 172.21.15.175:0/3412404641 >> v2:172.21.15.191:3300/0 conn(0x7f364814c840 0x7f364806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:14:56.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 >> v1:172.21.15.175:6789/0 conn(0x7f36480c52a0 legacy=0x7f3648152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:56.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3630003a60 0x7f3630003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:56.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3630004810 0x7f3630004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:56.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36300051c0 0x7f363000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:56.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364806d700 con 0x7f3630003a60 2023-10-10T19:14:56.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363000c0d0 con 0x7f3630004810 2023-10-10T19:14:56.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363000c290 con 0x7f36300051c0 2023-10-10T19:14:56.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3647fff700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3630003a60 0x7f3630003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:56.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f36477fe700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3630004810 0x7f3630004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:56.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f364cd84700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36300051c0 0x7f363000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:56.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3647fff700 1 -- 172.21.15.175:0/3412404641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36300051c0 msgr2=0x7f363000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:56.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3647fff700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36300051c0 0x7f363000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3647fff700 1 -- 172.21.15.175:0/3412404641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3630004810 msgr2=0x7f3630004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:14:56.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.570+0000 7f3647fff700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3630004810 0x7f3630004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f3647fff700 1 -- 172.21.15.175:0/3412404641 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630003570 con 0x7f3630003a60 2023-10-10T19:14:56.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f364cd84700 1 -- 172.21.15.175:0/3412404641 reap_dead start 2023-10-10T19:14:56.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f3647fff700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3630003a60 0x7f3630003e70 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f3638009de0 tx=0x7f3638009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4656ad872fdb8cc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:56.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638009730 con 0x7f3630003a60 2023-10-10T19:14:56.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3638005a00 con 0x7f3630003a60 2023-10-10T19:14:56.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f3646ffd700 1 -- 172.21.15.175:0/3412404641 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638005dc0 con 0x7f3630003a60 2023-10-10T19:14:56.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f364e7e7700 1 -- 172.21.15.175:0/3412404641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3630003a60 msgr2=0x7f3630003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:56.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f364e7e7700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3630003a60 0x7f3630003e70 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f3638009de0 tx=0x7f3638009120 comp rx=0 tx=0).stop 2023-10-10T19:14:56.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f364e7e7700 1 -- 172.21.15.175:0/3412404641 shutdown_connections 2023-10-10T19:14:56.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f364e7e7700 1 --2- 172.21.15.175:0/3412404641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3630003a60 0x7f3630003e70 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f364e7e7700 1 -- 172.21.15.175:0/3412404641 >> 172.21.15.175:0/3412404641 conn(0x7f3648068280 msgr2=0x7f364806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:56.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f364e7e7700 1 -- 172.21.15.175:0/3412404641 shutdown_connections 2023-10-10T19:14:56.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.571+0000 7f364e7e7700 1 -- 172.21.15.175:0/3412404641 wait complete. 2023-10-10T19:14:56.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.572+0000 7f364e7e7700 1 Processor -- start 2023-10-10T19:14:56.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.573+0000 7f364e7e7700 1 -- start start 2023-10-10T19:14:56.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.573+0000 7f364e7e7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3630003a60 0x7f36480c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:56.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.573+0000 7f364e7e7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f364814e430 0x7f36480c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:56.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.573+0000 7f364e7e7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36480c6c30 0x7f3648140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:56.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.573+0000 7f364e7e7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36480c62a0 con 0x7f364814e430 2023-10-10T19:14:56.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.573+0000 7f364e7e7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36480c70a0 con 0x7f36480c6c30 2023-10-10T19:14:56.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.573+0000 7f364e7e7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3648152ed0 con 0x7f3630003a60 2023-10-10T19:14:56.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.573+0000 7f36477fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f364814e430 0x7f36480c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:56.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f36477fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f364814e430 0x7f36480c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54394/0 (socket says 172.21.15.175:54394) 2023-10-10T19:14:56.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f36477fe700 1 -- 172.21.15.175:0/2052598641 learned_addr learned my addr 172.21.15.175:0/2052598641 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:56.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f3647fff700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3630003a60 0x7f36480c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:56.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f364cd84700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36480c6c30 0x7f3648140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:56.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f36477fe700 1 -- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3630003a60 msgr2=0x7f36480c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:56.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f36477fe700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3630003a60 0x7f36480c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f36477fe700 1 -- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36480c6c30 msgr2=0x7f3648140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:56.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f36477fe700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36480c6c30 0x7f3648140b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f36477fe700 1 -- 172.21.15.175:0/2052598641 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648166bd0 con 0x7f364814e430 2023-10-10T19:14:56.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f36477fe700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f364814e430 0x7f36480c66f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f363c002950 tx=0x7f363c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:56.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f362ffff700 1 -- 172.21.15.175:0/2052598641 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c002e90 con 0x7f364814e430 2023-10-10T19:14:56.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f362ffff700 1 -- 172.21.15.175:0/2052598641 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363c003390 con 0x7f364814e430 2023-10-10T19:14:56.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3648151430 con 0x7f364814e430 2023-10-10T19:14:56.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f362ffff700 1 -- 172.21.15.175:0/2052598641 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c00fb00 con 0x7f364814e430 2023-10-10T19:14:56.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.574+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36481519f0 con 0x7f364814e430 2023-10-10T19:14:56.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.575+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3648067570 con 0x7f364814e430 2023-10-10T19:14:56.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.575+0000 7f362ffff700 1 -- 172.21.15.175:0/2052598641 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f363c006820 con 0x7f364814e430 2023-10-10T19:14:56.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.576+0000 7f362ffff700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3634066f30 0x7f36340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:56.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.576+0000 7f362ffff700 1 -- 172.21.15.175:0/2052598641 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f363c084aa0 con 0x7f364814e430 2023-10-10T19:14:56.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.576+0000 7f3647fff700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3634066f30 0x7f36340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:56.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.576+0000 7f3647fff700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3634066f30 0x7f36340693e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3638009160 tx=0x7f3638019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:56.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.580+0000 7f362ffff700 1 -- 172.21.15.175:0/2052598641 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f363c051740 con 0x7f364814e430 2023-10-10T19:14:56.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.806+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f364806da40 con 0x7f364814e430 2023-10-10T19:14:56.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.806+0000 7f362ffff700 1 -- 172.21.15.175:0/2052598641 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5442 (secure 0 0 0) 0x7f363c054d60 con 0x7f364814e430 2023-10-10T19:14:56.808 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:56.808 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:51.808261+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:resolve","state_seq":10,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:56.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3634066f30 msgr2=0x7f36340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:56.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3634066f30 0x7f36340693e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3638009160 tx=0x7f3638019000 comp rx=0 tx=0).stop 2023-10-10T19:14:56.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f364814e430 msgr2=0x7f36480c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:56.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f364814e430 0x7f36480c66f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f363c002950 tx=0x7f363c007d30 comp rx=0 tx=0).stop 2023-10-10T19:14:56.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 shutdown_connections 2023-10-10T19:14:56.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3634066f30 0x7f36340693e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36480c6c30 0x7f3648140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3630003a60 0x7f36480c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 --2- 172.21.15.175:0/2052598641 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f364814e430 0x7f36480c66f0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:56.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 >> 172.21.15.175:0/2052598641 conn(0x7f3648068280 msgr2=0x7f364806e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:56.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 shutdown_connections 2023-10-10T19:14:56.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:56.808+0000 7f364e7e7700 1 -- 172.21.15.175:0/2052598641 wait complete. 2023-10-10T19:14:56.812 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 230 2023-10-10T19:14:56.822 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 230, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 230, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6080": { "addr": "172.21.15.191:6836/4084912544", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 4084912544, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 4084912544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6080, "incarnation": 229, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:resolve", "state_seq": 10 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 } }, "last_failure": 0, "last_failure_osd_epoch": 89, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:14:51.808261+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 6080, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:14:58.824 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:14:58.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.951+0000 7fb6cb528700 1 Processor -- start 2023-10-10T19:14:58.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.951+0000 7fb6cb528700 1 -- start start 2023-10-10T19:14:58.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6cb528700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb6c4155fb0 0x7fb6c4159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:58.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6cb528700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c41574b0 con 0x7fb6c40b1fc0 2023-10-10T19:14:58.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6cb528700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c40afad0 con 0x7fb6c40b2390 2023-10-10T19:14:58.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6cb528700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c4150770 con 0x7fb6c4155fb0 2023-10-10T19:14:58.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6c9ac5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb6c4155fb0 0x7fb6c4159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:58.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6c9ac5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb6c4155fb0 0x7fb6c4159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54406/0 (socket says 172.21.15.175:54406) 2023-10-10T19:14:58.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6c9ac5700 1 -- 172.21.15.175:0/3462220391 learned_addr learned my addr 172.21.15.175:0/3462220391 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:58.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6c9ac5700 1 -- 172.21.15.175:0/3462220391 >> v1:172.21.15.191:6789/0 conn(0x7fb6c40b2390 legacy=0x7fb6c40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:58.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6c9ac5700 1 -- 172.21.15.175:0/3462220391 >> v1:172.21.15.175:6789/0 conn(0x7fb6c40b1fc0 legacy=0x7fb6c4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:58.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6c9ac5700 1 -- 172.21.15.175:0/3462220391 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c40af4c0 con 0x7fb6c4155fb0 2023-10-10T19:14:58.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6c9ac5700 1 --2- 172.21.15.175:0/3462220391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4159920 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0099b0 tx=0x7fb6bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d64b2ae868c5856 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:58.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6bbfff700 1 -- 172.21.15.175:0/3462220391 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00b8f0 con 0x7fb6c4155fb0 2023-10-10T19:14:58.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6bbfff700 1 -- 172.21.15.175:0/3462220391 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc00ba50 con 0x7fb6c4155fb0 2023-10-10T19:14:58.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.952+0000 7fb6bbfff700 1 -- 172.21.15.175:0/3462220391 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00bdf0 con 0x7fb6c4155fb0 2023-10-10T19:14:58.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.953+0000 7fb6cb528700 1 -- 172.21.15.175:0/3462220391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 msgr2=0x7fb6c4159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:58.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.953+0000 7fb6cb528700 1 --2- 172.21.15.175:0/3462220391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4159920 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0099b0 tx=0x7fb6bc009250 comp rx=0 tx=0).stop 2023-10-10T19:14:58.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.953+0000 7fb6cb528700 1 -- 172.21.15.175:0/3462220391 shutdown_connections 2023-10-10T19:14:58.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.953+0000 7fb6cb528700 1 --2- 172.21.15.175:0/3462220391 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4159920 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:58.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.953+0000 7fb6cb528700 1 -- 172.21.15.175:0/3462220391 >> 172.21.15.175:0/3462220391 conn(0x7fb6c4063230 msgr2=0x7fb6c4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:58.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.953+0000 7fb6cb528700 1 -- 172.21.15.175:0/3462220391 shutdown_connections 2023-10-10T19:14:58.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.953+0000 7fb6cb528700 1 -- 172.21.15.175:0/3462220391 wait complete. 2023-10-10T19:14:58.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.954+0000 7fb6cb528700 1 Processor -- start 2023-10-10T19:14:58.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.954+0000 7fb6cb528700 1 -- start start 2023-10-10T19:14:58.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6cb528700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c40b1fc0 0x7fb6c4167300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:58.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6cb528700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c40b2390 0x7fb6c4150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:58.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6cb528700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:58.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6cb528700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4166e80 con 0x7fb6c4155fb0 2023-10-10T19:14:58.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6cb528700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4166900 con 0x7fb6c40b2390 2023-10-10T19:14:58.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6cb528700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4166ba0 con 0x7fb6c40b1fc0 2023-10-10T19:14:58.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c9ac5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:58.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c9ac5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4151060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54414/0 (socket says 172.21.15.175:54414) 2023-10-10T19:14:58.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c9ac5700 1 -- 172.21.15.175:0/3785007929 learned_addr learned my addr 172.21.15.175:0/3785007929 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:14:58.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c8ac3700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c40b2390 0x7fb6c4150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:58.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c9ac5700 1 -- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c40b1fc0 msgr2=0x7fb6c4167300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:58.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c92c4700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c40b1fc0 0x7fb6c4167300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:58.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c9ac5700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c40b1fc0 0x7fb6c4167300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:58.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c9ac5700 1 -- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c40b2390 msgr2=0x7fb6c4150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:58.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c9ac5700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c40b2390 0x7fb6c4150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:58.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c9ac5700 1 -- 172.21.15.175:0/3785007929 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc0094f0 con 0x7fb6c4155fb0 2023-10-10T19:14:58.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.955+0000 7fb6c8ac3700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c40b2390 0x7fb6c4150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:58.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.956+0000 7fb6c92c4700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c40b1fc0 0x7fb6c4167300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:14:58.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.956+0000 7fb6c9ac5700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4151060 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0094c0 tx=0x7fb6bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:58.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.956+0000 7fb6b97fa700 1 -- 172.21.15.175:0/3785007929 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc01a630 con 0x7fb6c4155fb0 2023-10-10T19:14:58.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.956+0000 7fb6b97fa700 1 -- 172.21.15.175:0/3785007929 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc01a790 con 0x7fb6c4155fb0 2023-10-10T19:14:58.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.956+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c41518b0 con 0x7fb6c4155fb0 2023-10-10T19:14:58.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.956+0000 7fb6b97fa700 1 -- 172.21.15.175:0/3785007929 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc01ab30 con 0x7fb6c4155fb0 2023-10-10T19:14:58.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.956+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c4151e70 con 0x7fb6c4155fb0 2023-10-10T19:14:58.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.957+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c4067570 con 0x7fb6c4155fb0 2023-10-10T19:14:58.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.957+0000 7fb6b97fa700 1 -- 172.21.15.175:0/3785007929 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb6bc02e450 con 0x7fb6c4155fb0 2023-10-10T19:14:58.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.958+0000 7fb6b97fa700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb6a8066f60 0x7fb6a8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:14:58.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.958+0000 7fb6b97fa700 1 -- 172.21.15.175:0/3785007929 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7fb6bc086770 con 0x7fb6c4155fb0 2023-10-10T19:14:58.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.958+0000 7fb6c92c4700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb6a8066f60 0x7fb6a8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:14:58.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.959+0000 7fb6c92c4700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb6a8066f60 0x7fb6a8069410 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007fd0 tx=0x7fb6c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:14:58.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:58.962+0000 7fb6b97fa700 1 -- 172.21.15.175:0/3785007929 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb6bc053410 con 0x7fb6c4155fb0 2023-10-10T19:14:59.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.186+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6c4068da0 con 0x7fb6c4155fb0 2023-10-10T19:14:59.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.186+0000 7fb6b97fa700 1 -- 172.21.15.175:0/3785007929 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+5444 (secure 0 0 0) 0x7fb6bc056a30 con 0x7fb6c4155fb0 2023-10-10T19:14:59.188 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:14:59.188 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:57.822399+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:14:59.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.188+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb6a8066f60 msgr2=0x7fb6a8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:59.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb6a8066f60 0x7fb6a8069410 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007fd0 tx=0x7fb6c0009040 comp rx=0 tx=0).stop 2023-10-10T19:14:59.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 msgr2=0x7fb6c4151060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:14:59.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4151060 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0094c0 tx=0x7fb6bc017000 comp rx=0 tx=0).stop 2023-10-10T19:14:59.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 shutdown_connections 2023-10-10T19:14:59.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb6a8066f60 0x7fb6a8069410 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:59.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6c4155fb0 0x7fb6c4151060 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:59.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb6c40b1fc0 0x7fb6c4167300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:59.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 --2- 172.21.15.175:0/3785007929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6c40b2390 0x7fb6c4150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:14:59.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 >> 172.21.15.175:0/3785007929 conn(0x7fb6c4063230 msgr2=0x7fb6c415b860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:14:59.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 shutdown_connections 2023-10-10T19:14:59.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:14:59.189+0000 7fb6cb528700 1 -- 172.21.15.175:0/3785007929 wait complete. 2023-10-10T19:14:59.193 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 231 2023-10-10T19:14:59.202 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:15:01.203 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:01.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.330+0000 7f5e8a560700 1 Processor -- start 2023-10-10T19:15:01.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.330+0000 7f5e8a560700 1 -- start start 2023-10-10T19:15:01.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.330+0000 7f5e8a560700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e84155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:01.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e8a560700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5e840b2390 0x7f5e84068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:01.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e8a560700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e841574b0 con 0x7f5e84068bd0 2023-10-10T19:15:01.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e8a560700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e84150770 con 0x7f5e840b2390 2023-10-10T19:15:01.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e8a560700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840afad0 con 0x7f5e840b1fc0 2023-10-10T19:15:01.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e837fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5e840b2390 0x7f5e84068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:01.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e83fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e84155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:01.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e837fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5e840b2390 0x7f5e84068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:45516/0 (socket says 172.21.15.175:45516) 2023-10-10T19:15:01.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e83fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e84155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:51816/0 (socket says 172.21.15.175:51816) 2023-10-10T19:15:01.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e837fe700 1 -- 172.21.15.175:0/3916837747 learned_addr learned my addr 172.21.15.175:0/3916837747 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:01.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e88afd700 1 --1- 172.21.15.175:0/3916837747 >> v1:172.21.15.191:6789/0 conn(0x7f5e84068bd0 0x7f5e840aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:60744/0 (socket says 172.21.15.175:60744) 2023-10-10T19:15:01.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e83fff700 1 -- 172.21.15.175:0/3916837747 >> v2:172.21.15.191:3300/0 conn(0x7f5e840b2390 msgr2=0x7f5e84068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:01.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e83fff700 1 --2- 172.21.15.175:0/3916837747 >> v2:172.21.15.191:3300/0 conn(0x7f5e840b2390 0x7f5e84068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e83fff700 1 -- 172.21.15.175:0/3916837747 >> v1:172.21.15.191:6789/0 conn(0x7f5e84068bd0 legacy=0x7f5e840aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:01.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e83fff700 1 -- 172.21.15.175:0/3916837747 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840af4c0 con 0x7f5e840b1fc0 2023-10-10T19:15:01.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e837fe700 1 --2- 172.21.15.175:0/3916837747 >> v2:172.21.15.191:3300/0 conn(0x7f5e840b2390 0x7f5e84068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:15:01.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e82ffd700 1 -- 172.21.15.175:0/3916837747 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336032310 0 0) 0x7f5e841574b0 con 0x7f5e84068bd0 2023-10-10T19:15:01.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.331+0000 7f5e83fff700 1 --2- 172.21.15.175:0/3916837747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e84155fb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5e780098e0 tx=0x7f5e780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24763503874cc636 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:01.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e82ffd700 1 -- 172.21.15.175:0/3916837747 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800b6d0 con 0x7f5e840b1fc0 2023-10-10T19:15:01.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e82ffd700 1 -- 172.21.15.175:0/3916837747 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7800b830 con 0x7f5e840b1fc0 2023-10-10T19:15:01.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e82ffd700 1 -- 172.21.15.175:0/3916837747 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800bbd0 con 0x7f5e840b1fc0 2023-10-10T19:15:01.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e8a560700 1 -- 172.21.15.175:0/3916837747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 msgr2=0x7f5e84155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:01.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e8a560700 1 --2- 172.21.15.175:0/3916837747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e84155fb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5e780098e0 tx=0x7f5e780091a0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e8a560700 1 -- 172.21.15.175:0/3916837747 shutdown_connections 2023-10-10T19:15:01.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e8a560700 1 --2- 172.21.15.175:0/3916837747 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e84155fb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e8a560700 1 --2- 172.21.15.175:0/3916837747 >> v2:172.21.15.191:3300/0 conn(0x7f5e840b2390 0x7f5e84068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e8a560700 1 -- 172.21.15.175:0/3916837747 >> 172.21.15.175:0/3916837747 conn(0x7f5e84063230 msgr2=0x7f5e84063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e8a560700 1 -- 172.21.15.175:0/3916837747 shutdown_connections 2023-10-10T19:15:01.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.332+0000 7f5e8a560700 1 -- 172.21.15.175:0/3916837747 wait complete. 2023-10-10T19:15:01.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.333+0000 7f5e8a560700 1 Processor -- start 2023-10-10T19:15:01.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e8a560700 1 -- start start 2023-10-10T19:15:01.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e8a560700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e84068bd0 0x7f5e84158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:01.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e8a560700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e840cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:01.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e8a560700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e840b2390 0x7f5e840cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:01.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e8a560700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840cae60 con 0x7f5e840b2390 2023-10-10T19:15:01.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e8a560700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8415a1a0 con 0x7f5e84068bd0 2023-10-10T19:15:01.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e8a560700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840cd120 con 0x7f5e840b1fc0 2023-10-10T19:15:01.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e88afd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e840b2390 0x7f5e840cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:01.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e88afd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e840b2390 0x7f5e840cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54428/0 (socket says 172.21.15.175:54428) 2023-10-10T19:15:01.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e88afd700 1 -- 172.21.15.175:0/1638015626 learned_addr learned my addr 172.21.15.175:0/1638015626 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:01.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.334+0000 7f5e83fff700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e84068bd0 0x7f5e84158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:01.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e837fe700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e840cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:01.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e88afd700 1 -- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 msgr2=0x7f5e840cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:01.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e88afd700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e840cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e88afd700 1 -- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e84068bd0 msgr2=0x7f5e84158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:01.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e88afd700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e84068bd0 0x7f5e84158580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e88afd700 1 -- 172.21.15.175:0/1638015626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740094f0 con 0x7f5e840b2390 2023-10-10T19:15:01.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e88afd700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e840b2390 0x7f5e840cb830 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00cd70 tx=0x7f5e7c001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:01.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e6bfff700 1 -- 172.21.15.175:0/1638015626 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c00ce70 con 0x7f5e840b2390 2023-10-10T19:15:01.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e6bfff700 1 -- 172.21.15.175:0/1638015626 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c005df0 con 0x7f5e840b2390 2023-10-10T19:15:01.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e78009420 con 0x7f5e840b2390 2023-10-10T19:15:01.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e6bfff700 1 -- 172.21.15.175:0/1638015626 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c00aae0 con 0x7f5e840b2390 2023-10-10T19:15:01.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.335+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e840caaf0 con 0x7f5e840b2390 2023-10-10T19:15:01.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.336+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e840cafd0 con 0x7f5e840b2390 2023-10-10T19:15:01.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.336+0000 7f5e6bfff700 1 -- 172.21.15.175:0/1638015626 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5e7c0070f0 con 0x7f5e840b2390 2023-10-10T19:15:01.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.337+0000 7f5e6bfff700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e70066f60 0x7f5e70069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:01.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.337+0000 7f5e6bfff700 1 -- 172.21.15.175:0/1638015626 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f5e7c084b70 con 0x7f5e840b2390 2023-10-10T19:15:01.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.337+0000 7f5e83fff700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e70066f60 0x7f5e70069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:01.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.337+0000 7f5e83fff700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e70066f60 0x7f5e70069410 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5e780091e0 tx=0x7f5e78017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:01.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.341+0000 7f5e6bfff700 1 -- 172.21.15.175:0/1638015626 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e7c0517f0 con 0x7f5e840b2390 2023-10-10T19:15:01.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.565+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e84067570 con 0x7f5e840b2390 2023-10-10T19:15:01.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.565+0000 7f5e6bfff700 1 -- 172.21.15.175:0/1638015626 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+5444 (secure 0 0 0) 0x7f5e7c054e10 con 0x7f5e840b2390 2023-10-10T19:15:01.567 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:01.567 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:14:57.822399+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:01.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e70066f60 msgr2=0x7f5e70069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:01.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e70066f60 0x7f5e70069410 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5e780091e0 tx=0x7f5e78017000 comp rx=0 tx=0).stop 2023-10-10T19:15:01.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e840b2390 msgr2=0x7f5e840cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:01.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e840b2390 0x7f5e840cb830 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00cd70 tx=0x7f5e7c001aa0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 shutdown_connections 2023-10-10T19:15:01.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5e70066f60 0x7f5e70069410 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e840b2390 0x7f5e840cb830 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5e84068bd0 0x7f5e84158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 --2- 172.21.15.175:0/1638015626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5e840b1fc0 0x7f5e840cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:01.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 >> 172.21.15.175:0/1638015626 conn(0x7f5e84063230 msgr2=0x7f5e8414fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:01.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 shutdown_connections 2023-10-10T19:15:01.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:01.568+0000 7f5e8a560700 1 -- 172.21.15.175:0/1638015626 wait complete. 2023-10-10T19:15:01.572 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 231 2023-10-10T19:15:01.580 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:15:03.581 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:03.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.708+0000 7f79af59e700 1 Processor -- start 2023-10-10T19:15:03.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.708+0000 7f79af59e700 1 -- start start 2023-10-10T19:15:03.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79af59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f79b01595a0 0x7f79b006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:03.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79af59e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b006a8c0 con 0x7f79b006bda0 2023-10-10T19:15:03.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79af59e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b00c5ab0 con 0x7f79b00b0240 2023-10-10T19:15:03.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79af59e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b0159fb0 con 0x7f79b01595a0 2023-10-10T19:15:03.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79add9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f79b006bda0 0x7f79b00afb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:34086/0 (socket says 172.21.15.175:34086) 2023-10-10T19:15:03.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79add9b700 1 -- 172.21.15.175:0/3855405299 learned_addr learned my addr 172.21.15.175:0/3855405299 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:15:03.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79ae59c700 1 --2- 172.21.15.175:0/3855405299 >> v2:172.21.15.191:3300/0 conn(0x7f79b01595a0 0x7f79b006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:03.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197584438 0 0) 0x7f79b006a8c0 con 0x7f79b006bda0 2023-10-10T19:15:03.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.709+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7998003570 con 0x7f79b006bda0 2023-10-10T19:15:03.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741060427 0 0) 0x7f79b00c5ab0 con 0x7f79b00b0240 2023-10-10T19:15:03.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79b006a8c0 con 0x7f79b00b0240 2023-10-10T19:15:03.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2429048314 0 0) 0x7f7998003570 con 0x7f79b006bda0 2023-10-10T19:15:03.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 >> v2:172.21.15.191:3300/0 conn(0x7f79b01595a0 msgr2=0x7f79b006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:03.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 --2- 172.21.15.175:0/3855405299 >> v2:172.21.15.191:3300/0 conn(0x7f79b01595a0 0x7f79b006dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 >> v1:172.21.15.191:6789/0 conn(0x7f79b00b0240 legacy=0x7f79b0160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:03.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b00c57d0 con 0x7f79b006bda0 2023-10-10T19:15:03.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f79a4001470 con 0x7f79b006bda0 2023-10-10T19:15:03.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ae59c700 1 --2- 172.21.15.175:0/3855405299 >> v2:172.21.15.191:3300/0 conn(0x7f79b01595a0 0x7f79b006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:15:03.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 >> v1:172.21.15.175:6789/0 conn(0x7f79b006bda0 legacy=0x7f79b00afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:03.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998003bf0 0x7f7998003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:03.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7998004810 0x7f7998004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:03.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79980051c0 0x7f799800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:03.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b0159fb0 con 0x7f7998004810 2023-10-10T19:15:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799800c0b0 con 0x7f79980051c0 2023-10-10T19:15:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799800c270 con 0x7f7998003bf0 2023-10-10T19:15:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79add9b700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7998004810 0x7f7998004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79ae59c700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998003bf0 0x7f7998003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:03.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79aed9d700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79980051c0 0x7f799800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:03.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79add9b700 1 -- 172.21.15.175:0/3855405299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998003bf0 msgr2=0x7f7998003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:03.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79add9b700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998003bf0 0x7f7998003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79add9b700 1 -- 172.21.15.175:0/3855405299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79980051c0 msgr2=0x7f799800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:03.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79add9b700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79980051c0 0x7f799800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79add9b700 1 -- 172.21.15.175:0/3855405299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7998003570 con 0x7f7998004810 2023-10-10T19:15:03.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79aed9d700 1 -- 172.21.15.175:0/3855405299 reap_dead start 2023-10-10T19:15:03.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.710+0000 7f79add9b700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7998004810 0x7f7998004c80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f79b006c1d0 tx=0x7f79b0159f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae0e8b11c19eb49 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:03.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a400cba0 con 0x7f7998004810 2023-10-10T19:15:03.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a400cd00 con 0x7f7998004810 2023-10-10T19:15:03.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79ad59a700 1 -- 172.21.15.175:0/3855405299 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4005b70 con 0x7f7998004810 2023-10-10T19:15:03.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79af59e700 1 -- 172.21.15.175:0/3855405299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7998004810 msgr2=0x7f7998004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:03.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79af59e700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7998004810 0x7f7998004c80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f79b006c1d0 tx=0x7f79b0159f20 comp rx=0 tx=0).stop 2023-10-10T19:15:03.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79af59e700 1 -- 172.21.15.175:0/3855405299 shutdown_connections 2023-10-10T19:15:03.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79af59e700 1 --2- 172.21.15.175:0/3855405299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7998004810 0x7f7998004c80 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79af59e700 1 -- 172.21.15.175:0/3855405299 >> 172.21.15.175:0/3855405299 conn(0x7f79b0063cd0 msgr2=0x7f79b006d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:03.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79af59e700 1 -- 172.21.15.175:0/3855405299 shutdown_connections 2023-10-10T19:15:03.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.711+0000 7f79af59e700 1 -- 172.21.15.175:0/3855405299 wait complete. 2023-10-10T19:15:03.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.712+0000 7f79af59e700 1 Processor -- start 2023-10-10T19:15:03.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.712+0000 7f79af59e700 1 -- start start 2023-10-10T19:15:03.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79af59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998004810 0x7f79b00ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:03.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79af59e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79b00c3e00 0x7f79b0169850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:03.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79af59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79b0169d90 0x7f79b0155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:03.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79af59e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00ad300 con 0x7f79b0169d90 2023-10-10T19:15:03.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79af59e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00c2dc0 con 0x7f79b00c3e00 2023-10-10T19:15:03.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79af59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00c3980 con 0x7f7998004810 2023-10-10T19:15:03.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79aed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79b0169d90 0x7f79b0155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:03.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79add9b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79b00c3e00 0x7f79b0169850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:03.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79ae59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998004810 0x7f79b00ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:03.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79aed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79b0169d90 0x7f79b0155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54450/0 (socket says 172.21.15.175:54450) 2023-10-10T19:15:03.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79aed9d700 1 -- 172.21.15.175:0/2707890348 learned_addr learned my addr 172.21.15.175:0/2707890348 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:03.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79add9b700 1 -- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998004810 msgr2=0x7f79b00ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:03.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79add9b700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998004810 0x7f79b00ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79add9b700 1 -- 172.21.15.175:0/2707890348 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79b0169d90 msgr2=0x7f79b0155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:03.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79add9b700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79b0169d90 0x7f79b0155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79add9b700 1 -- 172.21.15.175:0/2707890348 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b00c2490 con 0x7f79b00c3e00 2023-10-10T19:15:03.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.713+0000 7f79aed9d700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79b0169d90 0x7f79b0155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:15:03.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.714+0000 7f79add9b700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79b00c3e00 0x7f79b0169850 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f79b006e1d0 tx=0x7f79a4015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:03.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.714+0000 7f799e7fc700 1 -- 172.21.15.175:0/2707890348 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40070d0 con 0x7f79b00c3e00 2023-10-10T19:15:03.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.714+0000 7f799e7fc700 1 -- 172.21.15.175:0/2707890348 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a4007230 con 0x7f79b00c3e00 2023-10-10T19:15:03.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.714+0000 7f799e7fc700 1 -- 172.21.15.175:0/2707890348 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40075d0 con 0x7f79b00c3e00 2023-10-10T19:15:03.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.714+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b00c2770 con 0x7f79b00c3e00 2023-10-10T19:15:03.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.714+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b00c3250 con 0x7f79b00c3e00 2023-10-10T19:15:03.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.715+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b00c2f70 con 0x7f79b00c3e00 2023-10-10T19:15:03.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.715+0000 7f799e7fc700 1 -- 172.21.15.175:0/2707890348 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f79a401e430 con 0x7f79b00c3e00 2023-10-10T19:15:03.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.716+0000 7f799e7fc700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7990066f30 0x7f79900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:03.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.716+0000 7f799e7fc700 1 -- 172.21.15.175:0/2707890348 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f79a4085650 con 0x7f79b00c3e00 2023-10-10T19:15:03.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.716+0000 7f79ae59c700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7990066f30 0x7f79900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:03.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.716+0000 7f79ae59c700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7990066f30 0x7f79900693e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f79a00094d0 tx=0x7f79a000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:03.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.720+0000 7f799e7fc700 1 -- 172.21.15.175:0/2707890348 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f79a4052310 con 0x7f79b00c3e00 2023-10-10T19:15:03.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.947+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79b00c2f70 con 0x7f79b00c3e00 2023-10-10T19:15:03.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.948+0000 7f799e7fc700 1 -- 172.21.15.175:0/2707890348 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5441 (secure 0 0 0) 0x7f79b00c2f70 con 0x7f79b00c3e00 2023-10-10T19:15:03.948 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:03.949 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:02.839857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:03.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7990066f30 msgr2=0x7f79900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:03.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7990066f30 0x7f79900693e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f79a00094d0 tx=0x7f79a000b040 comp rx=0 tx=0).stop 2023-10-10T19:15:03.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79b00c3e00 msgr2=0x7f79b0169850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:03.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79b00c3e00 0x7f79b0169850 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f79b006e1d0 tx=0x7f79a4015000 comp rx=0 tx=0).stop 2023-10-10T19:15:03.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 shutdown_connections 2023-10-10T19:15:03.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7990066f30 0x7f79900693e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79b0169d90 0x7f79b0155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7998004810 0x7f79b00ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 --2- 172.21.15.175:0/2707890348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79b00c3e00 0x7f79b0169850 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:03.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 >> 172.21.15.175:0/2707890348 conn(0x7f79b0063cd0 msgr2=0x7f79b00c73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:03.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 shutdown_connections 2023-10-10T19:15:03.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:03.950+0000 7f79af59e700 1 -- 172.21.15.175:0/2707890348 wait complete. 2023-10-10T19:15:03.954 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 232 2023-10-10T19:15:03.963 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:15:05.964 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:06.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.086+0000 7f3436db1700 1 Processor -- start 2023-10-10T19:15:06.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.086+0000 7f3436db1700 1 -- start start 2023-10-10T19:15:06.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.087+0000 7f3436db1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f34300b1fc0 0x7f3430155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:06.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.087+0000 7f3436db1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34301574b0 con 0x7f34300b2390 2023-10-10T19:15:06.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.087+0000 7f3436db1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34300afad0 con 0x7f34300aff60 2023-10-10T19:15:06.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.087+0000 7f3436db1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3430150770 con 0x7f34300b1fc0 2023-10-10T19:15:06.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.087+0000 7f3427fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f34300b2390 0x7f3430068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54718/0 (socket says 172.21.15.175:54718) 2023-10-10T19:15:06.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.087+0000 7f3427fff700 1 -- 172.21.15.175:0/3093837219 learned_addr learned my addr 172.21.15.175:0/3093837219 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:15:06.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f3434b4d700 1 --2- 172.21.15.175:0/3093837219 >> v2:172.21.15.191:3300/0 conn(0x7f34300b1fc0 0x7f3430155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:06.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f34277fe700 1 -- 172.21.15.175:0/3093837219 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204595021 0 0) 0x7f34301574b0 con 0x7f34300b2390 2023-10-10T19:15:06.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f34277fe700 1 -- 172.21.15.175:0/3093837219 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3418003570 con 0x7f34300b2390 2023-10-10T19:15:06.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f34277fe700 1 -- 172.21.15.175:0/3093837219 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341059771 0 0) 0x7f34300afad0 con 0x7f34300aff60 2023-10-10T19:15:06.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f34277fe700 1 -- 172.21.15.175:0/3093837219 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34301574b0 con 0x7f34300aff60 2023-10-10T19:15:06.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f3434b4d700 1 -- 172.21.15.175:0/3093837219 >> v1:172.21.15.191:6789/0 conn(0x7f34300aff60 legacy=0x7f3430159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:06.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f3434b4d700 1 -- 172.21.15.175:0/3093837219 >> v1:172.21.15.175:6789/0 conn(0x7f34300b2390 legacy=0x7f3430068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:06.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f3434b4d700 1 -- 172.21.15.175:0/3093837219 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300af4c0 con 0x7f34300b1fc0 2023-10-10T19:15:06.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f3434b4d700 1 --2- 172.21.15.175:0/3093837219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300b1fc0 0x7f3430155fb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f342c0098e0 tx=0x7f342c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8a900b65c3b6974 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:06.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f34277fe700 1 -- 172.21.15.175:0/3093837219 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c011630 con 0x7f34300b1fc0 2023-10-10T19:15:06.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f34277fe700 1 -- 172.21.15.175:0/3093837219 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c011790 con 0x7f34300b1fc0 2023-10-10T19:15:06.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.088+0000 7f34277fe700 1 -- 172.21.15.175:0/3093837219 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c011b30 con 0x7f34300b1fc0 2023-10-10T19:15:06.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.089+0000 7f3436db1700 1 -- 172.21.15.175:0/3093837219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300b1fc0 msgr2=0x7f3430155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:06.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.089+0000 7f3436db1700 1 --2- 172.21.15.175:0/3093837219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300b1fc0 0x7f3430155fb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f342c0098e0 tx=0x7f342c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.089+0000 7f3436db1700 1 -- 172.21.15.175:0/3093837219 shutdown_connections 2023-10-10T19:15:06.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.089+0000 7f3436db1700 1 --2- 172.21.15.175:0/3093837219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300b1fc0 0x7f3430155fb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.089+0000 7f3436db1700 1 -- 172.21.15.175:0/3093837219 >> 172.21.15.175:0/3093837219 conn(0x7f3430063230 msgr2=0x7f3430063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:06.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.089+0000 7f3436db1700 1 -- 172.21.15.175:0/3093837219 shutdown_connections 2023-10-10T19:15:06.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.089+0000 7f3436db1700 1 -- 172.21.15.175:0/3093837219 wait complete. 2023-10-10T19:15:06.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.090+0000 7f3436db1700 1 Processor -- start 2023-10-10T19:15:06.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.090+0000 7f3436db1700 1 -- start start 2023-10-10T19:15:06.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3436db1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300aff60 0x7f34300cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:06.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3436db1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34300b1fc0 0x7f3430157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:06.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3436db1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34300b2390 0x7f343014da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:06.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3436db1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34301662d0 con 0x7f34300b2390 2023-10-10T19:15:06.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3436db1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34300cbe10 con 0x7f34300aff60 2023-10-10T19:15:06.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3436db1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34300cc7e0 con 0x7f34300b1fc0 2023-10-10T19:15:06.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34300b2390 0x7f343014da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:06.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34300b2390 0x7f343014da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50664/0 (socket says 172.21.15.175:50664) 2023-10-10T19:15:06.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 -- 172.21.15.175:0/2368687488 learned_addr learned my addr 172.21.15.175:0/2368687488 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:06.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3434b4d700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300aff60 0x7f34300cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:06.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3427fff700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34300b1fc0 0x7f3430157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:06.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 -- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34300b1fc0 msgr2=0x7f3430157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:06.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34300b1fc0 0x7f3430157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 -- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300aff60 msgr2=0x7f34300cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:06.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300aff60 0x7f34300cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 -- 172.21.15.175:0/2368687488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c009420 con 0x7f34300b2390 2023-10-10T19:15:06.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f343534e700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34300b2390 0x7f343014da00 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f342800c910 tx=0x7f34280027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:06.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.091+0000 7f3424ff9700 1 -- 172.21.15.175:0/2368687488 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428012630 con 0x7f34300b2390 2023-10-10T19:15:06.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.092+0000 7f3424ff9700 1 -- 172.21.15.175:0/2368687488 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34280127b0 con 0x7f34300b2390 2023-10-10T19:15:06.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.092+0000 7f3424ff9700 1 -- 172.21.15.175:0/2368687488 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428012b70 con 0x7f34300b2390 2023-10-10T19:15:06.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.092+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34300cd4c0 con 0x7f34300b2390 2023-10-10T19:15:06.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.092+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34300cbaa0 con 0x7f34300b2390 2023-10-10T19:15:06.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.092+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34300cbf80 con 0x7f34300b2390 2023-10-10T19:15:06.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.093+0000 7f3424ff9700 1 -- 172.21.15.175:0/2368687488 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f342801b400 con 0x7f34300b2390 2023-10-10T19:15:06.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.093+0000 7f3424ff9700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f341c066f30 0x7f341c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:06.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.093+0000 7f3424ff9700 1 -- 172.21.15.175:0/2368687488 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f3428084e10 con 0x7f34300b2390 2023-10-10T19:15:06.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.093+0000 7f3434b4d700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f341c066f30 0x7f341c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:06.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.094+0000 7f3434b4d700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f341c066f30 0x7f341c0693e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f342c0093f0 tx=0x7f342c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:06.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.098+0000 7f3424ff9700 1 -- 172.21.15.175:0/2368687488 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3428051b60 con 0x7f34300b2390 2023-10-10T19:15:06.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.320+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3430067570 con 0x7f34300b2390 2023-10-10T19:15:06.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.320+0000 7f3424ff9700 1 -- 172.21.15.175:0/2368687488 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5441 (secure 0 0 0) 0x7f3428055180 con 0x7f34300b2390 2023-10-10T19:15:06.321 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:06.321 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:02.839857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:06.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f341c066f30 msgr2=0x7f341c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:06.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f341c066f30 0x7f341c0693e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f342c0093f0 tx=0x7f342c017000 comp rx=0 tx=0).stop 2023-10-10T19:15:06.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34300b2390 msgr2=0x7f343014da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:06.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34300b2390 0x7f343014da00 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f342800c910 tx=0x7f34280027a0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 shutdown_connections 2023-10-10T19:15:06.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f341c066f30 0x7f341c0693e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34300b2390 0x7f343014da00 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34300aff60 0x7f34300cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 --2- 172.21.15.175:0/2368687488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34300b1fc0 0x7f3430157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:06.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.322+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 >> 172.21.15.175:0/2368687488 conn(0x7f3430063230 msgr2=0x7f3430150c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:06.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.323+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 shutdown_connections 2023-10-10T19:15:06.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:06.323+0000 7f3436db1700 1 -- 172.21.15.175:0/2368687488 wait complete. 2023-10-10T19:15:06.326 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 232 2023-10-10T19:15:06.335 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:15:08.336 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:08.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.465+0000 7f52a6c1b700 1 Processor -- start 2023-10-10T19:15:08.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.465+0000 7f52a6c1b700 1 -- start start 2023-10-10T19:15:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.465+0000 7f52a6c1b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52a014fbf0 0x7f52a0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.465+0000 7f52a6c1b700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a0156800 con 0x7f52a00681e0 2023-10-10T19:15:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.465+0000 7f52a6c1b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a0063e30 con 0x7f52a014ffc0 2023-10-10T19:15:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.465+0000 7f52a6c1b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a00c59e0 con 0x7f52a014fbf0 2023-10-10T19:15:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f52a51b8700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f52a00681e0 0x7f52a00c5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54734/0 (socket says 172.21.15.175:54734) 2023-10-10T19:15:08.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f52a51b8700 1 -- 172.21.15.175:0/1152582083 learned_addr learned my addr 172.21.15.175:0/1152582083 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:15:08.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f52a49b7700 1 --2- 172.21.15.175:0/1152582083 >> v2:172.21.15.191:3300/0 conn(0x7f52a014fbf0 0x7f52a0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:08.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f529f7fe700 1 -- 172.21.15.175:0/1152582083 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859400250 0 0) 0x7f52a0156800 con 0x7f52a00681e0 2023-10-10T19:15:08.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f529f7fe700 1 -- 172.21.15.175:0/1152582083 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c003570 con 0x7f52a00681e0 2023-10-10T19:15:08.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f52a49b7700 1 -- 172.21.15.175:0/1152582083 >> v1:172.21.15.191:6789/0 conn(0x7f52a014ffc0 legacy=0x7f52a0063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:08.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f52a49b7700 1 -- 172.21.15.175:0/1152582083 >> v1:172.21.15.175:6789/0 conn(0x7f52a00681e0 legacy=0x7f52a00c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:08.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f529f7fe700 1 -- 172.21.15.175:0/1152582083 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3318860019 0 0) 0x7f528c003570 con 0x7f52a00681e0 2023-10-10T19:15:08.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f52a49b7700 1 -- 172.21.15.175:0/1152582083 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0156c90 con 0x7f52a014fbf0 2023-10-10T19:15:08.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f52a49b7700 1 --2- 172.21.15.175:0/1152582083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014fbf0 0x7f52a0159d70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5290009eb0 tx=0x7f5290009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9557204a40b7d9c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:08.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f529f7fe700 1 -- 172.21.15.175:0/1152582083 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529000e070 con 0x7f52a014fbf0 2023-10-10T19:15:08.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f529f7fe700 1 -- 172.21.15.175:0/1152582083 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529000b920 con 0x7f52a014fbf0 2023-10-10T19:15:08.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.466+0000 7f529f7fe700 1 -- 172.21.15.175:0/1152582083 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298001470 con 0x7f52a014fbf0 2023-10-10T19:15:08.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.467+0000 7f52a6c1b700 1 -- 172.21.15.175:0/1152582083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014fbf0 msgr2=0x7f52a0159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:08.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.467+0000 7f52a6c1b700 1 --2- 172.21.15.175:0/1152582083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014fbf0 0x7f52a0159d70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5290009eb0 tx=0x7f5290009780 comp rx=0 tx=0).stop 2023-10-10T19:15:08.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.467+0000 7f52a6c1b700 1 -- 172.21.15.175:0/1152582083 shutdown_connections 2023-10-10T19:15:08.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.467+0000 7f52a6c1b700 1 --2- 172.21.15.175:0/1152582083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014fbf0 0x7f52a0159d70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:08.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.467+0000 7f52a6c1b700 1 -- 172.21.15.175:0/1152582083 >> 172.21.15.175:0/1152582083 conn(0x7f52a00cc760 msgr2=0x7f52a00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:08.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.467+0000 7f52a6c1b700 1 -- 172.21.15.175:0/1152582083 shutdown_connections 2023-10-10T19:15:08.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.467+0000 7f52a6c1b700 1 -- 172.21.15.175:0/1152582083 wait complete. 2023-10-10T19:15:08.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.468+0000 7f52a6c1b700 1 Processor -- start 2023-10-10T19:15:08.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.468+0000 7f52a6c1b700 1 -- start start 2023-10-10T19:15:08.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a6c1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52a00681e0 0x7f52a0166dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:08.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a6c1b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52a014fbf0 0x7f52a0158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:08.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a6c1b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014ffc0 0x7f52a0151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:08.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a6c1b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00b22c0 con 0x7f52a00681e0 2023-10-10T19:15:08.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a6c1b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0153780 con 0x7f52a014ffc0 2023-10-10T19:15:08.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a6c1b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0166930 con 0x7f52a014fbf0 2023-10-10T19:15:08.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a49b7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52a00681e0 0x7f52a0166dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:08.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a49b7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52a00681e0 0x7f52a0166dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50676/0 (socket says 172.21.15.175:50676) 2023-10-10T19:15:08.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a49b7700 1 -- 172.21.15.175:0/2965652653 learned_addr learned my addr 172.21.15.175:0/2965652653 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:08.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a51b8700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014ffc0 0x7f52a0151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:08.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f529ffff700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52a014fbf0 0x7f52a0158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:08.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a49b7700 1 -- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52a014fbf0 msgr2=0x7f52a0158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:08.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a49b7700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52a014fbf0 0x7f52a0158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:08.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a49b7700 1 -- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014ffc0 msgr2=0x7f52a0151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:08.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a49b7700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014ffc0 0x7f52a0151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:08.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a49b7700 1 -- 172.21.15.175:0/2965652653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290009a20 con 0x7f52a00681e0 2023-10-10T19:15:08.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.469+0000 7f52a51b8700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014ffc0 0x7f52a0151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:15:08.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.470+0000 7f52a49b7700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52a00681e0 0x7f52a0166dc0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f52a0089f50 tx=0x7f5290017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:08.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.470+0000 7f529cff9700 1 -- 172.21.15.175:0/2965652653 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290013c60 con 0x7f52a00681e0 2023-10-10T19:15:08.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.470+0000 7f529cff9700 1 -- 172.21.15.175:0/2965652653 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5290013de0 con 0x7f52a00681e0 2023-10-10T19:15:08.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.470+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a0153240 con 0x7f52a00681e0 2023-10-10T19:15:08.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.470+0000 7f529cff9700 1 -- 172.21.15.175:0/2965652653 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290026650 con 0x7f52a00681e0 2023-10-10T19:15:08.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.470+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a0154950 con 0x7f52a00681e0 2023-10-10T19:15:08.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.471+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a0166c10 con 0x7f52a00681e0 2023-10-10T19:15:08.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.471+0000 7f529cff9700 1 -- 172.21.15.175:0/2965652653 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5290019030 con 0x7f52a00681e0 2023-10-10T19:15:08.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.472+0000 7f529cff9700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5280066f30 0x7f52800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:08.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.472+0000 7f529cff9700 1 -- 172.21.15.175:0/2965652653 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f52900867f0 con 0x7f52a00681e0 2023-10-10T19:15:08.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.472+0000 7f529ffff700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5280066f30 0x7f52800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:08.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.472+0000 7f529ffff700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5280066f30 0x7f52800693e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f52a00c6980 tx=0x7f5288009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:08.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.476+0000 7f529cff9700 1 -- 172.21.15.175:0/2965652653 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5290053600 con 0x7f52a00681e0 2023-10-10T19:15:08.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.704+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52a00cb880 con 0x7f52a00681e0 2023-10-10T19:15:08.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.704+0000 7f529cff9700 1 -- 172.21.15.175:0/2965652653 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5441 (secure 0 0 0) 0x7f5290056c20 con 0x7f52a00681e0 2023-10-10T19:15:08.705 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:08.705 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:02.839857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:08.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5280066f30 msgr2=0x7f52800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:08.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5280066f30 0x7f52800693e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f52a00c6980 tx=0x7f5288009040 comp rx=0 tx=0).stop 2023-10-10T19:15:08.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52a00681e0 msgr2=0x7f52a0166dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:08.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52a00681e0 0x7f52a0166dc0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f52a0089f50 tx=0x7f5290017000 comp rx=0 tx=0).stop 2023-10-10T19:15:08.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 shutdown_connections 2023-10-10T19:15:08.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5280066f30 0x7f52800693e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:08.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52a014ffc0 0x7f52a0151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:08.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f52a00681e0 0x7f52a0166dc0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:08.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 --2- 172.21.15.175:0/2965652653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52a014fbf0 0x7f52a0158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:08.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 >> 172.21.15.175:0/2965652653 conn(0x7f52a00cc760 msgr2=0x7f52a01593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:08.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 shutdown_connections 2023-10-10T19:15:08.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:08.706+0000 7f52a6c1b700 1 -- 172.21.15.175:0/2965652653 wait complete. 2023-10-10T19:15:08.710 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 232 2023-10-10T19:15:08.719 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 232, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 232, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6080": { "addr": "172.21.15.191:6836/4084912544", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 4084912544, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 4084912544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6080, "incarnation": 229, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:rejoin", "state_seq": 14 }, "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 } }, "last_failure": 0, "last_failure_osd_epoch": 89, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:15:02.839857+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 6080, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:15:10.721 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:10.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.845+0000 7f4bc05b5700 1 Processor -- start 2023-10-10T19:15:10.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.845+0000 7f4bc05b5700 1 -- start start 2023-10-10T19:15:10.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.846+0000 7f4bc05b5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb80c2cf0 0x7f4bb806e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.846+0000 7f4bc05b5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4bb80c30c0 0x7f4bb814db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.846+0000 7f4bc05b5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb8153f70 con 0x7f4bb80c0270 2023-10-10T19:15:10.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.846+0000 7f4bc05b5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb8067bd0 con 0x7f4bb80c30c0 2023-10-10T19:15:10.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.846+0000 7f4bc05b5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb8154210 con 0x7f4bb80c2cf0 2023-10-10T19:15:10.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbeb52700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f4bb80c0270 0x7f4bb8157460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54744/0 (socket says 172.21.15.175:54744) 2023-10-10T19:15:10.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbeb52700 1 -- 172.21.15.175:0/129975807 learned_addr learned my addr 172.21.15.175:0/129975807 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:15:10.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbdb50700 1 --2- 172.21.15.175:0/129975807 >> v2:172.21.15.191:3300/0 conn(0x7f4bb80c30c0 0x7f4bb814db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbe351700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb80c2cf0 0x7f4bb806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789554545 0 0) 0x7f4bb8153f70 con 0x7f4bb80c0270 2023-10-10T19:15:10.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba4003570 con 0x7f4bb80c0270 2023-10-10T19:15:10.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187706384 0 0) 0x7f4ba4003570 con 0x7f4bb80c0270 2023-10-10T19:15:10.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb80c2cf0 msgr2=0x7f4bb806e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:10.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb80c2cf0 0x7f4bb806e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:10.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 >> v2:172.21.15.191:3300/0 conn(0x7f4bb80c30c0 msgr2=0x7f4bb814db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:10.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 --2- 172.21.15.175:0/129975807 >> v2:172.21.15.191:3300/0 conn(0x7f4bb80c30c0 0x7f4bb814db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:10.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb81544f0 con 0x7f4bb80c0270 2023-10-10T19:15:10.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f4bb00017f0 con 0x7f4bb80c0270 2023-10-10T19:15:10.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 >> v1:172.21.15.175:6789/0 conn(0x7f4bb80c0270 legacy=0x7f4bb8157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:10.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb80c2cf0 0x7f4ba4003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4003fe0 0x7f4ba4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ba4004d50 0x7f4ba40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb8067bd0 con 0x7f4ba4004d50 2023-10-10T19:15:10.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb8154210 con 0x7f4ba4003fe0 2023-10-10T19:15:10.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba400ba90 con 0x7f4bb80c2cf0 2023-10-10T19:15:10.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbeb52700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ba4004d50 0x7f4ba40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbdb50700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4003fe0 0x7f4ba4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.847+0000 7f4bbe351700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb80c2cf0 0x7f4ba4003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbeb52700 1 -- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb80c2cf0 msgr2=0x7f4ba4003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:10.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbeb52700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb80c2cf0 0x7f4ba4003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:10.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbeb52700 1 -- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4003fe0 msgr2=0x7f4ba4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:10.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbeb52700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4003fe0 0x7f4ba4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:10.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbeb52700 1 -- 172.21.15.175:0/129975807 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4003570 con 0x7f4ba4004d50 2023-10-10T19:15:10.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbeb52700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ba4004d50 0x7f4ba40051c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000a970 tx=0x7f4bb80c5100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c035830671874761 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:10.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb00070d0 con 0x7f4ba4004d50 2023-10-10T19:15:10.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb0007230 con 0x7f4ba4004d50 2023-10-10T19:15:10.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbd34f700 1 -- 172.21.15.175:0/129975807 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb00075f0 con 0x7f4ba4004d50 2023-10-10T19:15:10.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bc05b5700 1 -- 172.21.15.175:0/129975807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ba4004d50 msgr2=0x7f4ba40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:10.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bc05b5700 1 --2- 172.21.15.175:0/129975807 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ba4004d50 0x7f4ba40051c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000a970 tx=0x7f4bb80c5100 comp rx=0 tx=0).stop 2023-10-10T19:15:10.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.848+0000 7f4bbeb52700 1 -- 172.21.15.175:0/129975807 reap_dead start 2023-10-10T19:15:10.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.849+0000 7f4bc05b5700 1 -- 172.21.15.175:0/129975807 shutdown_connections 2023-10-10T19:15:10.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.849+0000 7f4bc05b5700 1 -- 172.21.15.175:0/129975807 >> 172.21.15.175:0/129975807 conn(0x7f4bb8068900 msgr2=0x7f4bb8068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:10.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.849+0000 7f4bc05b5700 1 -- 172.21.15.175:0/129975807 shutdown_connections 2023-10-10T19:15:10.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.849+0000 7f4bc05b5700 1 -- 172.21.15.175:0/129975807 wait complete. 2023-10-10T19:15:10.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bc05b5700 1 Processor -- start 2023-10-10T19:15:10.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bc05b5700 1 -- start start 2023-10-10T19:15:10.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bc05b5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4004d50 0x7f4bb813d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bc05b5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb814bab0 0x7f4bb8143530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bc05b5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bb8143a70 0x7f4bb814aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bc05b5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb806f7d0 con 0x7f4bb8143a70 2023-10-10T19:15:10.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bc05b5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb813ddf0 con 0x7f4ba4004d50 2023-10-10T19:15:10.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bc05b5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb814a650 con 0x7f4bb814bab0 2023-10-10T19:15:10.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bbeb52700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bb8143a70 0x7f4bb814aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bbeb52700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bb8143a70 0x7f4bb814aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50692/0 (socket says 172.21.15.175:50692) 2023-10-10T19:15:10.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bbeb52700 1 -- 172.21.15.175:0/4181837752 learned_addr learned my addr 172.21.15.175:0/4181837752 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:10.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bbe351700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4004d50 0x7f4bb813d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.850+0000 7f4bbdb50700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb814bab0 0x7f4bb8143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bbeb52700 1 -- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb814bab0 msgr2=0x7f4bb8143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:10.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bbeb52700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb814bab0 0x7f4bb8143530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:10.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bbeb52700 1 -- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4004d50 msgr2=0x7f4bb813d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:10.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bbeb52700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4004d50 0x7f4bb813d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:10.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bbeb52700 1 -- 172.21.15.175:0/4181837752 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8143ee0 con 0x7f4bb8143a70 2023-10-10T19:15:10.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bbeb52700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bb8143a70 0x7f4bb814aae0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000a430 tx=0x7f4bb0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:10.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bae7fc700 1 -- 172.21.15.175:0/4181837752 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb00017b0 con 0x7f4bb8143a70 2023-10-10T19:15:10.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bae7fc700 1 -- 172.21.15.175:0/4181837752 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb001e640 con 0x7f4bb8143a70 2023-10-10T19:15:10.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb814d060 con 0x7f4bb8143a70 2023-10-10T19:15:10.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bae7fc700 1 -- 172.21.15.175:0/4181837752 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb001ea20 con 0x7f4bb8143a70 2023-10-10T19:15:10.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.851+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb8167240 con 0x7f4bb8143a70 2023-10-10T19:15:10.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.852+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb814a800 con 0x7f4bb8143a70 2023-10-10T19:15:10.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.852+0000 7f4bae7fc700 1 -- 172.21.15.175:0/4181837752 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4bb0026410 con 0x7f4bb8143a70 2023-10-10T19:15:10.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.853+0000 7f4bae7fc700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b9c066f60 0x7f4b9c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:10.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.853+0000 7f4bae7fc700 1 -- 172.21.15.175:0/4181837752 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f4bb0086c90 con 0x7f4bb8143a70 2023-10-10T19:15:10.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.853+0000 7f4bbe351700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b9c066f60 0x7f4b9c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:10.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.853+0000 7f4bbe351700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b9c066f60 0x7f4b9c069410 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4006e50 tx=0x7f4bb4009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:10.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:10.857+0000 7f4bae7fc700 1 -- 172.21.15.175:0/4181837752 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4bb814a800 con 0x7f4bb8143a70 2023-10-10T19:15:11.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.082+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bb814a800 con 0x7f4bb8143a70 2023-10-10T19:15:11.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.083+0000 7f4bae7fc700 1 -- 172.21.15.175:0/4181837752 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+5447 (secure 0 0 0) 0x7f4bb814a800 con 0x7f4bb8143a70 2023-10-10T19:15:11.083 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:11.084 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":233,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:08.853814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:11.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b9c066f60 msgr2=0x7f4b9c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:11.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b9c066f60 0x7f4b9c069410 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4006e50 tx=0x7f4bb4009120 comp rx=0 tx=0).stop 2023-10-10T19:15:11.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bb8143a70 msgr2=0x7f4bb814aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:11.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bb8143a70 0x7f4bb814aae0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000a430 tx=0x7f4bb0016000 comp rx=0 tx=0).stop 2023-10-10T19:15:11.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 shutdown_connections 2023-10-10T19:15:11.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4b9c066f60 0x7f4b9c069410 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:11.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bb8143a70 0x7f4bb814aae0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:11.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ba4004d50 0x7f4bb813d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:11.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 --2- 172.21.15.175:0/4181837752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bb814bab0 0x7f4bb8143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:11.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 >> 172.21.15.175:0/4181837752 conn(0x7f4bb8068900 msgr2=0x7f4bb806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:11.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 shutdown_connections 2023-10-10T19:15:11.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:11.085+0000 7f4bc05b5700 1 -- 172.21.15.175:0/4181837752 wait complete. 2023-10-10T19:15:11.088 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 233 2023-10-10T19:15:11.098 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:15:13.100 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:13.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.226+0000 7ffb2a4e4700 1 Processor -- start 2023-10-10T19:15:13.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.226+0000 7ffb2a4e4700 1 -- start start 2023-10-10T19:15:13.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.226+0000 7ffb2a4e4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb24068a40 0x7ffb24063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:13.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.226+0000 7ffb2a4e4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb240c4ee0 con 0x7ffb24068e10 2023-10-10T19:15:13.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.226+0000 7ffb2a4e4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb240c5780 con 0x7ffb240cac80 2023-10-10T19:15:13.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.226+0000 7ffb2a4e4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb24064180 con 0x7ffb24068a40 2023-10-10T19:15:13.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb28a81700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ffb24068e10 0x7ffb240af910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54748/0 (socket says 172.21.15.175:54748) 2023-10-10T19:15:13.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb28a81700 1 -- 172.21.15.175:0/100496798 learned_addr learned my addr 172.21.15.175:0/100496798 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:15:13.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb23fff700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb24068a40 0x7ffb24063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:13.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759371059 0 0) 0x7ffb240c4ee0 con 0x7ffb24068e10 2023-10-10T19:15:13.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb0c003570 con 0x7ffb24068e10 2023-10-10T19:15:13.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867160612 0 0) 0x7ffb240c5780 con 0x7ffb240cac80 2023-10-10T19:15:13.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb240c4ee0 con 0x7ffb240cac80 2023-10-10T19:15:13.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146892828 0 0) 0x7ffb0c003570 con 0x7ffb24068e10 2023-10-10T19:15:13.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb24068a40 msgr2=0x7ffb24063c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:13.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb24068a40 0x7ffb24063c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 >> v1:172.21.15.191:6789/0 conn(0x7ffb240cac80 legacy=0x7ffb2406b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:13.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb240c6710 con 0x7ffb24068e10 2023-10-10T19:15:13.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7ffb1c0017f0 con 0x7ffb24068e10 2023-10-10T19:15:13.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 >> v1:172.21.15.175:6789/0 conn(0x7ffb24068e10 legacy=0x7ffb240af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:13.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb0c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:13.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb24068a40 0x7ffb0c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:13.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb0c004d50 0x7ffb0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:13.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb24064180 con 0x7ffb0c003c20 2023-10-10T19:15:13.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c00bc40 con 0x7ffb0c004d50 2023-10-10T19:15:13.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c00be00 con 0x7ffb24068a40 2023-10-10T19:15:13.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb23fff700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb0c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:13.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.227+0000 7ffb237fe700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb24068a40 0x7ffb0c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:13.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb28a81700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb0c004d50 0x7ffb0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:13.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb23fff700 1 -- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb24068a40 msgr2=0x7ffb0c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:13.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb23fff700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb24068a40 0x7ffb0c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb23fff700 1 -- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb0c004d50 msgr2=0x7ffb0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:13.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb23fff700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb0c004d50 0x7ffb0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb23fff700 1 -- 172.21.15.175:0/100496798 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c003570 con 0x7ffb0c003c20 2023-10-10T19:15:13.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb28a81700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb0c004d50 0x7ffb0c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:15:13.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb23fff700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb0c004010 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009bf0 tx=0x7ffb14009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832b7717288383bd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:13.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14005a00 con 0x7ffb0c003c20 2023-10-10T19:15:13.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14005b60 con 0x7ffb0c003c20 2023-10-10T19:15:13.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb22ffd700 1 -- 172.21.15.175:0/100496798 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1400e710 con 0x7ffb0c003c20 2023-10-10T19:15:13.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/100496798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 msgr2=0x7ffb0c004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:13.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb2a4e4700 1 --2- 172.21.15.175:0/100496798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb0c004010 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009bf0 tx=0x7ffb14009120 comp rx=0 tx=0).stop 2023-10-10T19:15:13.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.228+0000 7ffb28a81700 1 -- 172.21.15.175:0/100496798 reap_dead start 2023-10-10T19:15:13.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.229+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/100496798 shutdown_connections 2023-10-10T19:15:13.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.229+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/100496798 >> 172.21.15.175:0/100496798 conn(0x7ffb2406dbe0 msgr2=0x7ffb240ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:13.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.229+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/100496798 shutdown_connections 2023-10-10T19:15:13.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.229+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/100496798 wait complete. 2023-10-10T19:15:13.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.230+0000 7ffb2a4e4700 1 Processor -- start 2023-10-10T19:15:13.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.230+0000 7ffb2a4e4700 1 -- start start 2023-10-10T19:15:13.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.230+0000 7ffb2a4e4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb2415ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:13.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb2a4e4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb2415b1c0 0x7ffb24153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:13.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb2a4e4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb240c2c30 0x7ffb24153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:13.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb2a4e4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2414efe0 con 0x7ffb0c003c20 2023-10-10T19:15:13.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb2a4e4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2414fa70 con 0x7ffb2415b1c0 2023-10-10T19:15:13.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb2a4e4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb241520a0 con 0x7ffb240c2c30 2023-10-10T19:15:13.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb2415ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:13.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb2415ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50706/0 (socket says 172.21.15.175:50706) 2023-10-10T19:15:13.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 -- 172.21.15.175:0/2949986141 learned_addr learned my addr 172.21.15.175:0/2949986141 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:13.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 -- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb240c2c30 msgr2=0x7ffb24153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:13.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb28a81700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb240c2c30 0x7ffb24153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:13.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb240c2c30 0x7ffb24153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 -- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb2415b1c0 msgr2=0x7ffb24153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:13.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb237fe700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb2415b1c0 0x7ffb24153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:13.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb2415b1c0 0x7ffb24153500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 -- 172.21.15.175:0/2949986141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24152340 con 0x7ffb0c003c20 2023-10-10T19:15:13.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb23fff700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb2415ac80 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c00bc10 tx=0x7ffb14019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:13.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb0bfff700 1 -- 172.21.15.175:0/2949986141 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14006310 con 0x7ffb0c003c20 2023-10-10T19:15:13.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb0bfff700 1 -- 172.21.15.175:0/2949986141 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14006490 con 0x7ffb0c003c20 2023-10-10T19:15:13.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb24152620 con 0x7ffb0c003c20 2023-10-10T19:15:13.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb0bfff700 1 -- 172.21.15.175:0/2949986141 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14006850 con 0x7ffb0c003c20 2023-10-10T19:15:13.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.231+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb24152be0 con 0x7ffb0c003c20 2023-10-10T19:15:13.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.232+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb24067570 con 0x7ffb0c003c20 2023-10-10T19:15:13.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.233+0000 7ffb0bfff700 1 -- 172.21.15.175:0/2949986141 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffb14013680 con 0x7ffb0c003c20 2023-10-10T19:15:13.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.233+0000 7ffb0bfff700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb10066f30 0x7ffb100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:13.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.233+0000 7ffb0bfff700 1 -- 172.21.15.175:0/2949986141 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7ffb14086280 con 0x7ffb0c003c20 2023-10-10T19:15:13.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.233+0000 7ffb237fe700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb10066f30 0x7ffb100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:13.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.234+0000 7ffb237fe700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb10066f30 0x7ffb100693e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffb18000c00 tx=0x7ffb18009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:13.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.237+0000 7ffb0bfff700 1 -- 172.21.15.175:0/2949986141 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffb14052f60 con 0x7ffb0c003c20 2023-10-10T19:15:13.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.461+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb240c6990 con 0x7ffb0c003c20 2023-10-10T19:15:13.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.461+0000 7ffb0bfff700 1 -- 172.21.15.175:0/2949986141 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+5447 (secure 0 0 0) 0x7ffb14056540 con 0x7ffb0c003c20 2023-10-10T19:15:13.462 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:13.462 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":233,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:08.853814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:13.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb10066f30 msgr2=0x7ffb100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:13.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb10066f30 0x7ffb100693e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffb18000c00 tx=0x7ffb18009040 comp rx=0 tx=0).stop 2023-10-10T19:15:13.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 msgr2=0x7ffb2415ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:13.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb2415ac80 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c00bc10 tx=0x7ffb14019000 comp rx=0 tx=0).stop 2023-10-10T19:15:13.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 shutdown_connections 2023-10-10T19:15:13.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb10066f30 0x7ffb100693e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb240c2c30 0x7ffb24153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb0c003c20 0x7ffb2415ac80 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 --2- 172.21.15.175:0/2949986141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb2415b1c0 0x7ffb24153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:13.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 >> 172.21.15.175:0/2949986141 conn(0x7ffb2406dbe0 msgr2=0x7ffb2406b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:13.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 shutdown_connections 2023-10-10T19:15:13.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:13.463+0000 7ffb2a4e4700 1 -- 172.21.15.175:0/2949986141 wait complete. 2023-10-10T19:15:13.466 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 233 2023-10-10T19:15:13.475 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:15:15.477 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:15.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.600+0000 7f7a14de0700 1 Processor -- start 2023-10-10T19:15:15.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.600+0000 7f7a14de0700 1 -- start start 2023-10-10T19:15:15.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a14de0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7a1014fd90 0x7f7a10159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:15.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a14de0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a1008a100 0x7f7a100c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:15.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a14de0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7a100cd040 0x7f7a100c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:15.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a14de0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a100c5600 con 0x7f7a100cd040 2023-10-10T19:15:15.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a14de0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a10067fa0 con 0x7f7a1014fd90 2023-10-10T19:15:15.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a14de0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a10156b70 con 0x7f7a1008a100 2023-10-10T19:15:15.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7a100cd040 0x7f7a100c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:15.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7a100cd040 0x7f7a100c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49544/0 (socket says 172.21.15.175:49544) 2023-10-10T19:15:15.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0ffff700 1 -- 172.21.15.175:0/1133299580 learned_addr learned my addr 172.21.15.175:0/1133299580 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:15.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0f7fe700 1 --2- 172.21.15.175:0/1133299580 >> v2:172.21.15.191:3300/0 conn(0x7f7a1014fd90 0x7f7a10159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:15.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0effd700 1 --2- 172.21.15.175:0/1133299580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a1008a100 0x7f7a100c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:15.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0ffff700 1 -- 172.21.15.175:0/1133299580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a1008a100 msgr2=0x7f7a100c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:15.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0ffff700 1 --2- 172.21.15.175:0/1133299580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a1008a100 0x7f7a100c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0ffff700 1 -- 172.21.15.175:0/1133299580 >> v2:172.21.15.191:3300/0 conn(0x7f7a1014fd90 msgr2=0x7f7a10159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:15.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0ffff700 1 --2- 172.21.15.175:0/1133299580 >> v2:172.21.15.191:3300/0 conn(0x7f7a1014fd90 0x7f7a10159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.601+0000 7f7a0ffff700 1 -- 172.21.15.175:0/1133299580 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a10150160 con 0x7f7a100cd040 2023-10-10T19:15:15.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a0ffff700 1 --2- 172.21.15.175:0/1133299580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a100cd040 0x7f7a100c63f0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f7a10065080 tx=0x7f7a08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44b03b2c09714fe server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:15.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a0e7fc700 1 -- 172.21.15.175:0/1133299580 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0800e070 con 0x7f7a100cd040 2023-10-10T19:15:15.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a0e7fc700 1 -- 172.21.15.175:0/1133299580 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0800ba50 con 0x7f7a100cd040 2023-10-10T19:15:15.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a0e7fc700 1 -- 172.21.15.175:0/1133299580 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a080092e0 con 0x7f7a100cd040 2023-10-10T19:15:15.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 -- 172.21.15.175:0/1133299580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a100cd040 msgr2=0x7f7a100c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:15.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 --2- 172.21.15.175:0/1133299580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a100cd040 0x7f7a100c63f0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f7a10065080 tx=0x7f7a08009840 comp rx=0 tx=0).stop 2023-10-10T19:15:15.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 -- 172.21.15.175:0/1133299580 shutdown_connections 2023-10-10T19:15:15.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 --2- 172.21.15.175:0/1133299580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a100cd040 0x7f7a100c63f0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 --2- 172.21.15.175:0/1133299580 >> v2:172.21.15.191:3300/0 conn(0x7f7a1014fd90 0x7f7a10159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 --2- 172.21.15.175:0/1133299580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a1008a100 0x7f7a100c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 -- 172.21.15.175:0/1133299580 >> 172.21.15.175:0/1133299580 conn(0x7f7a10063d00 msgr2=0x7f7a100cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:15.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 -- 172.21.15.175:0/1133299580 shutdown_connections 2023-10-10T19:15:15.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.602+0000 7f7a14de0700 1 -- 172.21.15.175:0/1133299580 wait complete. 2023-10-10T19:15:15.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a14de0700 1 Processor -- start 2023-10-10T19:15:15.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a14de0700 1 -- start start 2023-10-10T19:15:15.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a14de0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a1008a100 0x7f7a100c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:15.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a14de0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a100cd040 0x7f7a101661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:15.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a14de0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a1014fd90 0x7f7a100cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:15.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a14de0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a10165d50 con 0x7f7a1008a100 2023-10-10T19:15:15.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a14de0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100c7ba0 con 0x7f7a1014fd90 2023-10-10T19:15:15.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a14de0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100c9d20 con 0x7f7a100cd040 2023-10-10T19:15:15.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a0f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a1008a100 0x7f7a100c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:15.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a0f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a1008a100 0x7f7a100c7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49558/0 (socket says 172.21.15.175:49558) 2023-10-10T19:15:15.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.604+0000 7f7a0f7fe700 1 -- 172.21.15.175:0/3197491353 learned_addr learned my addr 172.21.15.175:0/3197491353 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:15.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0ffff700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a1014fd90 0x7f7a100cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:15.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0effd700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a100cd040 0x7f7a101661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:15.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0f7fe700 1 -- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a100cd040 msgr2=0x7f7a101661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:15.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0f7fe700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a100cd040 0x7f7a101661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0f7fe700 1 -- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a1014fd90 msgr2=0x7f7a100cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:15.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0f7fe700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a1014fd90 0x7f7a100cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0f7fe700 1 -- 172.21.15.175:0/3197491353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a08009ae0 con 0x7f7a1008a100 2023-10-10T19:15:15.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0ffff700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a1014fd90 0x7f7a100cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:15:15.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a0f7fe700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a1008a100 0x7f7a100c7660 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000d890 tx=0x7f7a00008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:15.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f79f77fe700 1 -- 172.21.15.175:0/3197491353 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000b690 con 0x7f7a1008a100 2023-10-10T19:15:15.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f79f77fe700 1 -- 172.21.15.175:0/3197491353 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0000b7f0 con 0x7f7a1008a100 2023-10-10T19:15:15.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a100c9480 con 0x7f7a1008a100 2023-10-10T19:15:15.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f79f77fe700 1 -- 172.21.15.175:0/3197491353 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000bb70 con 0x7f7a1008a100 2023-10-10T19:15:15.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.605+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a101659e0 con 0x7f7a1008a100 2023-10-10T19:15:15.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.606+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a10067570 con 0x7f7a1008a100 2023-10-10T19:15:15.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.606+0000 7f79f77fe700 1 -- 172.21.15.175:0/3197491353 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7a0001e460 con 0x7f7a1008a100 2023-10-10T19:15:15.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.607+0000 7f79f77fe700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79fc066f60 0x7f79fc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:15.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.607+0000 7f79f77fe700 1 -- 172.21.15.175:0/3197491353 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f7a00085250 con 0x7f7a1008a100 2023-10-10T19:15:15.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.607+0000 7f7a0effd700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79fc066f60 0x7f79fc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:15.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.608+0000 7f7a0effd700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79fc066f60 0x7f79fc069410 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7a1004c490 tx=0x7f7a04008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:15.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.611+0000 7f79f77fe700 1 -- 172.21.15.175:0/3197491353 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a00051ef0 con 0x7f7a1008a100 2023-10-10T19:15:15.807 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T19:15:15.807+0000 7feff93e2700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:15:15.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.836+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a10155f80 con 0x7f7a1008a100 2023-10-10T19:15:15.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.837+0000 7f79f77fe700 1 -- 172.21.15.175:0/3197491353 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5441 (secure 0 0 0) 0x7f7a00055510 con 0x7f7a1008a100 2023-10-10T19:15:15.837 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:15.838 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:14.892946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:15.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.838+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79fc066f60 msgr2=0x7f79fc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:15.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.838+0000 7f7a14de0700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79fc066f60 0x7f79fc069410 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7a1004c490 tx=0x7f7a04008040 comp rx=0 tx=0).stop 2023-10-10T19:15:15.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.838+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a1008a100 msgr2=0x7f7a100c7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:15.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.838+0000 7f7a14de0700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a1008a100 0x7f7a100c7660 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000d890 tx=0x7f7a00008040 comp rx=0 tx=0).stop 2023-10-10T19:15:15.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.839+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 shutdown_connections 2023-10-10T19:15:15.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.839+0000 7f7a14de0700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f79fc066f60 0x7f79fc069410 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.839+0000 7f7a14de0700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a1014fd90 0x7f7a100cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.839+0000 7f7a14de0700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7a1008a100 0x7f7a100c7660 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.839+0000 7f7a14de0700 1 --2- 172.21.15.175:0/3197491353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a100cd040 0x7f7a101661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:15.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.839+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 >> 172.21.15.175:0/3197491353 conn(0x7f7a10063d00 msgr2=0x7f7a10158150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:15.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.839+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 shutdown_connections 2023-10-10T19:15:15.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:15.839+0000 7f7a14de0700 1 -- 172.21.15.175:0/3197491353 wait complete. 2023-10-10T19:15:15.842 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 234 2023-10-10T19:15:15.851 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=1, replacing gid=6117 2023-10-10T19:15:15.851 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2023-10-10T19:15:17.853 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-10T19:15:17.854 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-10T19:15:17.854 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-10T19:15:17.857 INFO:tasks.ceph.mds.a:Started 2023-10-10T19:15:17.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:17.942 INFO:tasks.ceph.mds.a.smithi175.stdout:starting mds.a at 2023-10-10T19:15:17.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.987+0000 7f4e83130700 1 Processor -- start 2023-10-10T19:15:17.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.988+0000 7f4e83130700 1 -- start start 2023-10-10T19:15:17.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.988+0000 7f4e83130700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 0x7f4e7c0c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:17.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.988+0000 7f4e83130700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4e7c06d6e0 0x7f4e7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:17.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.988+0000 7f4e83130700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7c0677f0 con 0x7f4e7c0c4c20 2023-10-10T19:15:17.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.988+0000 7f4e83130700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7c159590 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.988+0000 7f4e83130700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0c4250 con 0x7f4e7c1500b0 2023-10-10T19:15:18.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.988+0000 7f4e7bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4e7c06d6e0 0x7f4e7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:18.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4e7c06d6e0 0x7f4e7c06dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49598/0 (socket says 172.21.15.175:49598) 2023-10-10T19:15:18.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7bfff700 1 -- 172.21.15.175:0/548741093 learned_addr learned my addr 172.21.15.175:0/548741093 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:18.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e80ecc700 1 --2- 172.21.15.175:0/548741093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 0x7f4e7c0c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:18.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7bfff700 1 -- 172.21.15.175:0/548741093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 msgr2=0x7f4e7c0c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:18.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7bfff700 1 --2- 172.21.15.175:0/548741093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 0x7f4e7c0c46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7bfff700 1 -- 172.21.15.175:0/548741093 >> v1:172.21.15.191:6789/0 conn(0x7f4e7c0c4c20 legacy=0x7f4e7c156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:15:18.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7bfff700 1 -- 172.21.15.175:0/548741093 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c063760 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7bfff700 1 --2- 172.21.15.175:0/548741093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c06dcd0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4e680099b0 tx=0x7f4e68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd0d83989049d5d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:18.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7b7fe700 1 -- 172.21.15.175:0/548741093 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800b8b0 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7b7fe700 1 -- 172.21.15.175:0/548741093 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6800ba10 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.989+0000 7f4e7b7fe700 1 -- 172.21.15.175:0/548741093 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800bdb0 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.990+0000 7f4e83130700 1 -- 172.21.15.175:0/548741093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 msgr2=0x7f4e7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:18.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.990+0000 7f4e83130700 1 --2- 172.21.15.175:0/548741093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c06dcd0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4e680099b0 tx=0x7f4e68009250 comp rx=0 tx=0).stop 2023-10-10T19:15:18.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.990+0000 7f4e83130700 1 -- 172.21.15.175:0/548741093 shutdown_connections 2023-10-10T19:15:18.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.990+0000 7f4e83130700 1 --2- 172.21.15.175:0/548741093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 0x7f4e7c0c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.990+0000 7f4e83130700 1 --2- 172.21.15.175:0/548741093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c06dcd0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.990+0000 7f4e83130700 1 -- 172.21.15.175:0/548741093 >> 172.21.15.175:0/548741093 conn(0x7f4e7c0c5650 msgr2=0x7f4e7c0c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:18.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.990+0000 7f4e83130700 1 -- 172.21.15.175:0/548741093 shutdown_connections 2023-10-10T19:15:18.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.990+0000 7f4e83130700 1 -- 172.21.15.175:0/548741093 wait complete. 2023-10-10T19:15:18.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.991+0000 7f4e83130700 1 Processor -- start 2023-10-10T19:15:18.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.991+0000 7f4e83130700 1 -- start start 2023-10-10T19:15:18.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e83130700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c0bbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:18.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e83130700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e7c0c4c20 0x7f4e7c1671d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e83130700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 0x7f4e7c14d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e83130700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0c3f80 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e83130700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c068e40 con 0x7f4e7c0c4c20 2023-10-10T19:15:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e83130700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0bafe0 con 0x7f4e7c1500b0 2023-10-10T19:15:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c0bbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c0bbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49600/0 (socket says 172.21.15.175:49600) 2023-10-10T19:15:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 -- 172.21.15.175:0/950849009 learned_addr learned my addr 172.21.15.175:0/950849009 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e816cd700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 0x7f4e7c14d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e7bfff700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e7c0c4c20 0x7f4e7c1671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:18.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 -- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 msgr2=0x7f4e7c14d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 0x7f4e7c14d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 -- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e7c0c4c20 msgr2=0x7f4e7c1671d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e7c0c4c20 0x7f4e7c1671d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 -- 172.21.15.175:0/950849009 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e680094f0 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e80ecc700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c0bbda0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c7d0 tx=0x7f4e70008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e78ff9700 1 -- 172.21.15.175:0/950849009 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70011600 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e78ff9700 1 -- 172.21.15.175:0/950849009 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e70011760 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7c0bb2c0 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e78ff9700 1 -- 172.21.15.175:0/950849009 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70011b20 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.992+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7c0bb880 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.993+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7c067570 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.994+0000 7f4e78ff9700 1 -- 172.21.15.175:0/950849009 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4e700193f0 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.994+0000 7f4e78ff9700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4e60066f60 0x7f4e60069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.994+0000 7f4e78ff9700 1 -- 172.21.15.175:0/950849009 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f4e70084f50 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.994+0000 7f4e7bfff700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4e60066f60 0x7f4e60069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:18.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.995+0000 7f4e7bfff700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4e60066f60 0x7f4e60069410 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4e680099b0 tx=0x7f4e68016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:18.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:17.998+0000 7f4e78ff9700 1 -- 172.21.15.175:0/950849009 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4e70051bf0 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.225+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e7c069c40 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.226+0000 7f4e78ff9700 1 -- 172.21.15.175:0/950849009 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5441 (secure 0 0 0) 0x7f4e70055210 con 0x7f4e7c06d6e0 2023-10-10T19:15:18.227 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:18.227 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:14.892946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:18.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4e60066f60 msgr2=0x7f4e60069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:18.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4e60066f60 0x7f4e60069410 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4e680099b0 tx=0x7f4e68016000 comp rx=0 tx=0).stop 2023-10-10T19:15:18.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 msgr2=0x7f4e7c0bbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:18.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c0bbda0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c7d0 tx=0x7f4e70008040 comp rx=0 tx=0).stop 2023-10-10T19:15:18.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 shutdown_connections 2023-10-10T19:15:18.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4e60066f60 0x7f4e60069410 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7c1500b0 0x7f4e7c14d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c06d6e0 0x7f4e7c0bbda0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 --2- 172.21.15.175:0/950849009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e7c0c4c20 0x7f4e7c1671d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:18.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 >> 172.21.15.175:0/950849009 conn(0x7f4e7c0c5650 msgr2=0x7f4e7c0c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:18.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 shutdown_connections 2023-10-10T19:15:18.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:18.228+0000 7f4e83130700 1 -- 172.21.15.175:0/950849009 wait complete. 2023-10-10T19:15:18.231 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 234 2023-10-10T19:15:18.241 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-10T19:15:20.241 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:15:20.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.364+0000 7fec9ffff700 1 Processor -- start 2023-10-10T19:15:20.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.364+0000 7fec9ffff700 1 -- start start 2023-10-10T19:15:20.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.365+0000 7fec9ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feca0068c00 0x7feca0159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:20.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.365+0000 7fec9ffff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca0067fa0 con 0x7feca008a100 2023-10-10T19:15:20.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.365+0000 7fec9ffff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca0156b70 con 0x7feca014fd90 2023-10-10T19:15:20.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.365+0000 7fec9ffff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca00c5600 con 0x7feca0068c00 2023-10-10T19:15:20.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.365+0000 7fec967fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7feca008a100 0x7feca00684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50682/0 (socket says 172.21.15.175:50682) 2023-10-10T19:15:20.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.365+0000 7fec967fc700 1 -- 172.21.15.175:0/3535224285 learned_addr learned my addr 172.21.15.175:0/3535224285 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:15:20.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.365+0000 7fec9f7fe700 1 --2- 172.21.15.175:0/3535224285 >> v2:172.21.15.191:3300/0 conn(0x7feca0068c00 0x7feca0159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:20.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9e7fc700 1 -- 172.21.15.175:0/3535224285 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487909553 0 0) 0x7feca0067fa0 con 0x7feca008a100 2023-10-10T19:15:20.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9e7fc700 1 -- 172.21.15.175:0/3535224285 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c003570 con 0x7feca008a100 2023-10-10T19:15:20.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9e7fc700 1 -- 172.21.15.175:0/3535224285 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427044896 0 0) 0x7feca0156b70 con 0x7feca014fd90 2023-10-10T19:15:20.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9e7fc700 1 -- 172.21.15.175:0/3535224285 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca0067fa0 con 0x7feca014fd90 2023-10-10T19:15:20.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9f7fe700 1 -- 172.21.15.175:0/3535224285 >> v1:172.21.15.191:6789/0 conn(0x7feca014fd90 legacy=0x7feca00c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:20.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9f7fe700 1 -- 172.21.15.175:0/3535224285 >> v1:172.21.15.175:6789/0 conn(0x7feca008a100 legacy=0x7feca00684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:20.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9f7fe700 1 -- 172.21.15.175:0/3535224285 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca0150160 con 0x7feca0068c00 2023-10-10T19:15:20.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9f7fe700 1 --2- 172.21.15.175:0/3535224285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca0068c00 0x7feca0159c30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fec98009e60 tx=0x7fec98009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20a2fe5c57473f5e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:20.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9e7fc700 1 -- 172.21.15.175:0/3535224285 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9800e070 con 0x7feca0068c00 2023-10-10T19:15:20.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9e7fc700 1 -- 172.21.15.175:0/3535224285 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9800bab0 con 0x7feca0068c00 2023-10-10T19:15:20.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.366+0000 7fec9e7fc700 1 -- 172.21.15.175:0/3535224285 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9800be70 con 0x7feca0068c00 2023-10-10T19:15:20.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.367+0000 7fec9ffff700 1 -- 172.21.15.175:0/3535224285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca0068c00 msgr2=0x7feca0159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:20.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.367+0000 7fec9ffff700 1 --2- 172.21.15.175:0/3535224285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca0068c00 0x7feca0159c30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fec98009e60 tx=0x7fec98009780 comp rx=0 tx=0).stop 2023-10-10T19:15:20.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.367+0000 7fec9ffff700 1 -- 172.21.15.175:0/3535224285 shutdown_connections 2023-10-10T19:15:20.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.367+0000 7fec9ffff700 1 --2- 172.21.15.175:0/3535224285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca0068c00 0x7feca0159c30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:20.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.367+0000 7fec9ffff700 1 -- 172.21.15.175:0/3535224285 >> 172.21.15.175:0/3535224285 conn(0x7feca0063d00 msgr2=0x7feca00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:20.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.367+0000 7fec9ffff700 1 -- 172.21.15.175:0/3535224285 shutdown_connections 2023-10-10T19:15:20.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.367+0000 7fec9ffff700 1 -- 172.21.15.175:0/3535224285 wait complete. 2023-10-10T19:15:20.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.368+0000 7fec9ffff700 1 Processor -- start 2023-10-10T19:15:20.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.368+0000 7fec9ffff700 1 -- start start 2023-10-10T19:15:20.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca0068c00 0x7feca01567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:20.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9ffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca008a100 0x7feca00ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:20.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca014fd90 0x7feca00cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:20.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9ffff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca0155b20 con 0x7feca0068c00 2023-10-10T19:15:20.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9ffff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca01536d0 con 0x7feca008a100 2023-10-10T19:15:20.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9ffff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca0155560 con 0x7feca014fd90 2023-10-10T19:15:20.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca0068c00 0x7feca01567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:20.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca0068c00 0x7feca01567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49604/0 (socket says 172.21.15.175:49604) 2023-10-10T19:15:20.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 -- 172.21.15.175:0/1786421019 learned_addr learned my addr 172.21.15.175:0/1786421019 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:15:20.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec967fc700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca008a100 0x7feca00ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:20.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9f7fe700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca014fd90 0x7feca00cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:20.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 -- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca014fd90 msgr2=0x7feca00cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:20.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca014fd90 0x7feca00cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:20.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 -- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca008a100 msgr2=0x7feca00ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:20.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca008a100 0x7feca00ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:20.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 -- 172.21.15.175:0/1786421019 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec98009a20 con 0x7feca0068c00 2023-10-10T19:15:20.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9f7fe700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca014fd90 0x7feca00cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:15:20.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9effd700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca0068c00 0x7feca01567c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fec9000c580 tx=0x7fec90007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:20.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec977fe700 1 -- 172.21.15.175:0/1786421019 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec90010510 con 0x7feca0068c00 2023-10-10T19:15:20.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec977fe700 1 -- 172.21.15.175:0/1786421019 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec90010670 con 0x7feca0068c00 2023-10-10T19:15:20.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca0155840 con 0x7feca0068c00 2023-10-10T19:15:20.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec977fe700 1 -- 172.21.15.175:0/1786421019 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec90010a30 con 0x7feca0068c00 2023-10-10T19:15:20.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.369+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca0153360 con 0x7feca0068c00 2023-10-10T19:15:20.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.370+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca0152ee0 con 0x7feca0068c00 2023-10-10T19:15:20.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.371+0000 7fec977fe700 1 -- 172.21.15.175:0/1786421019 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fec9001b400 con 0x7feca0068c00 2023-10-10T19:15:20.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.371+0000 7fec977fe700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec80066f30 0x7fec800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:15:20.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.371+0000 7fec977fe700 1 -- 172.21.15.175:0/1786421019 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7197+0+0 (secure 0 0 0) 0x7fec900848b0 con 0x7feca0068c00 2023-10-10T19:15:20.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.372+0000 7fec967fc700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec80066f30 0x7fec800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:15:20.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.372+0000 7fec967fc700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec80066f30 0x7fec800693e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7feca00c7a60 tx=0x7fec88009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:15:20.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.376+0000 7fec977fe700 1 -- 172.21.15.175:0/1786421019 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feca0152ee0 con 0x7feca0068c00 2023-10-10T19:15:20.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.611+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feca0152ee0 con 0x7feca0068c00 2023-10-10T19:15:20.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.611+0000 7fec977fe700 1 -- 172.21.15.175:0/1786421019 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6201 (secure 0 0 0) 0x7feca0152ee0 con 0x7feca0068c00 2023-10-10T19:15:20.612 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:15:20.612 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:14.892946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:15:20.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec80066f30 msgr2=0x7fec800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:20.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec80066f30 0x7fec800693e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7feca00c7a60 tx=0x7fec88009040 comp rx=0 tx=0).stop 2023-10-10T19:15:20.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca0068c00 msgr2=0x7feca01567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:15:20.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca0068c00 0x7feca01567c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fec9000c580 tx=0x7fec90007e10 comp rx=0 tx=0).stop 2023-10-10T19:15:20.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 shutdown_connections 2023-10-10T19:15:20.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec80066f30 0x7fec800693e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:20.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feca014fd90 0x7feca00cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:20.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feca0068c00 0x7feca01567c0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:20.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 --2- 172.21.15.175:0/1786421019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca008a100 0x7feca00ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:15:20.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 >> 172.21.15.175:0/1786421019 conn(0x7feca0063d00 msgr2=0x7feca00c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:15:20.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 shutdown_connections 2023-10-10T19:15:20.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:15:20.613+0000 7fec9ffff700 1 -- 172.21.15.175:0/1786421019 wait complete. 2023-10-10T19:15:20.617 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 235 2023-10-10T19:15:20.627 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-10T19:15:20.627 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-10-10T19:15:23.500 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:15:23.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:15:34.581 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-hardlink.sh... 2023-10-10T19:15:34.582 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-hardlink.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-hardlink.sh 2023-10-10T19:15:34.611 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T19:15:34.611 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir 1 2 2023-10-10T19:15:34.614 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo asdf 2023-10-10T19:15:34.615 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo asdf 2023-10-10T19:15:34.615 INFO:tasks.workunit.client.0.smithi175.stderr:+ ln 1/file1 2/file1 2023-10-10T19:15:34.617 INFO:tasks.workunit.client.0.smithi175.stderr:+ ln 1/file2 2/file2 2023-10-10T19:15:34.619 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir 2/.snap/s1 2023-10-10T19:15:34.647 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo qwer 2023-10-10T19:15:34.655 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep asdf 2/.snap/s1/file1 2023-10-10T19:15:34.669 INFO:tasks.workunit.client.0.smithi175.stdout:asdf 2023-10-10T19:15:34.669 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -f 1/file2 2023-10-10T19:15:34.907 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep asdf 2/.snap/s1/file2 2023-10-10T19:15:35.417 INFO:tasks.workunit.client.0.smithi175.stdout:asdf 2023-10-10T19:15:35.418 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -f 2/file2 2023-10-10T19:15:35.647 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep asdf 2/.snap/s1/file2 2023-10-10T19:15:39.917 INFO:tasks.workunit.client.0.smithi175.stdout:asdf 2023-10-10T19:15:39.918 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir 2/.snap/s1 2023-10-10T19:15:40.796 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -rf 1 2 2023-10-10T19:15:41.631 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:15:41.632 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T19:15:41.632 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:15:41.632 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:15:41.712 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-intodir.sh... 2023-10-10T19:15:41.713 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-intodir.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-intodir.sh 2023-10-10T19:15:41.847 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir foo 2023-10-10T19:15:41.849 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir foo/.snap/one 2023-10-10T19:15:42.061 INFO:tasks.workunit.client.0.smithi175.stderr:+ touch bar 2023-10-10T19:15:42.063 INFO:tasks.workunit.client.0.smithi175.stderr:+ mv bar foo 2023-10-10T19:15:42.065 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T19:15:42.896 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir baz 2023-10-10T19:15:42.898 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir baz/.snap/two 2023-10-10T19:15:42.922 INFO:tasks.workunit.client.0.smithi175.stderr:+ mv baz foo 2023-10-10T19:15:42.925 INFO:tasks.workunit.client.0.smithi175.stderr:+ sync 2023-10-10T19:15:44.910 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir foo/baz/.snap/two 2023-10-10T19:15:44.973 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir foo/.snap/one 2023-10-10T19:15:44.979 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -r foo 2023-10-10T19:15:44.994 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T19:15:44.995 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:15:44.996 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:15:44.996 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:15:46.681 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-multiple-capsnaps.sh... 2023-10-10T19:15:46.681 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-multiple-capsnaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh 2023-10-10T19:15:50.619 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T19:15:50.619 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo asdf 2023-10-10T19:15:50.620 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/1 2023-10-10T19:15:50.645 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 777 a 2023-10-10T19:15:50.646 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/2 2023-10-10T19:15:50.668 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo qwer 2023-10-10T19:15:50.679 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/3 2023-10-10T19:15:50.705 INFO:tasks.workunit.client.0.smithi175.stderr:+ chmod 666 a 2023-10-10T19:15:50.706 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/4 2023-10-10T19:15:51.428 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo zxcv 2023-10-10T19:15:51.772 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/5 2023-10-10T19:15:52.074 INFO:tasks.workunit.client.0.smithi175.stderr:+ ls -al .snap/1/a .snap/2/a .snap/3/a .snap/4/a .snap/5/a 2023-10-10T19:15:52.075 INFO:tasks.workunit.client.0.smithi175.stdout:-rw-rw-r--. 1 ubuntu ubuntu 5 Oct 10 19:15 .snap/1/a 2023-10-10T19:15:52.075 INFO:tasks.workunit.client.0.smithi175.stdout:-rwxrwxrwx. 1 ubuntu ubuntu 5 Oct 10 19:15 .snap/2/a 2023-10-10T19:15:52.075 INFO:tasks.workunit.client.0.smithi175.stdout:-rwxrwxrwx. 1 ubuntu ubuntu 5 Oct 10 19:15 .snap/3/a 2023-10-10T19:15:52.076 INFO:tasks.workunit.client.0.smithi175.stdout:-rw-rw-rw-. 1 ubuntu ubuntu 5 Oct 10 19:15 .snap/4/a 2023-10-10T19:15:52.076 INFO:tasks.workunit.client.0.smithi175.stdout:-rw-rw-rw-. 1 ubuntu ubuntu 5 Oct 10 19:15 .snap/5/a 2023-10-10T19:15:52.076 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep asdf .snap/1/a 2023-10-10T19:15:52.077 INFO:tasks.workunit.client.0.smithi175.stdout:asdf 2023-10-10T19:15:52.077 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/1/a 2023-10-10T19:15:52.077 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'Size: 5' 2023-10-10T19:15:52.078 INFO:tasks.workunit.client.0.smithi175.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-10T19:15:52.078 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep asdf .snap/2/a 2023-10-10T19:15:52.079 INFO:tasks.workunit.client.0.smithi175.stdout:asdf 2023-10-10T19:15:52.080 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/2/a 2023-10-10T19:15:52.080 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'Size: 5' 2023-10-10T19:15:52.080 INFO:tasks.workunit.client.0.smithi175.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-10T19:15:52.081 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/2/a 2023-10-10T19:15:52.081 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep -- -rwxrwxrwx 2023-10-10T19:15:52.082 INFO:tasks.workunit.client.0.smithi175.stdout:Access: (0777/-rwxrwxrwx) Uid: ( 1000/ ubuntu) Gid: ( 1275/ ubuntu) 2023-10-10T19:15:52.082 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep qwer .snap/3/a 2023-10-10T19:15:52.083 INFO:tasks.workunit.client.0.smithi175.stdout:qwer 2023-10-10T19:15:52.083 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/3/a 2023-10-10T19:15:52.083 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'Size: 5' 2023-10-10T19:15:52.084 INFO:tasks.workunit.client.0.smithi175.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-10T19:15:52.085 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/3/a 2023-10-10T19:15:52.085 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep -- -rwxrwxrwx 2023-10-10T19:15:52.085 INFO:tasks.workunit.client.0.smithi175.stdout:Access: (0777/-rwxrwxrwx) Uid: ( 1000/ ubuntu) Gid: ( 1275/ ubuntu) 2023-10-10T19:15:52.086 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep qwer .snap/4/a 2023-10-10T19:15:52.087 INFO:tasks.workunit.client.0.smithi175.stdout:qwer 2023-10-10T19:15:52.087 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/4/a 2023-10-10T19:15:52.087 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'Size: 5' 2023-10-10T19:15:52.088 INFO:tasks.workunit.client.0.smithi175.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-10T19:15:52.088 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/4/a 2023-10-10T19:15:52.089 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep -- -rw-rw-rw- 2023-10-10T19:15:52.089 INFO:tasks.workunit.client.0.smithi175.stdout:Access: (0666/-rw-rw-rw-) Uid: ( 1000/ ubuntu) Gid: ( 1275/ ubuntu) 2023-10-10T19:15:52.089 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep zxcv .snap/5/a 2023-10-10T19:15:52.090 INFO:tasks.workunit.client.0.smithi175.stdout:zxcv 2023-10-10T19:15:52.091 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/5/a 2023-10-10T19:15:52.091 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep 'Size: 5' 2023-10-10T19:15:52.092 INFO:tasks.workunit.client.0.smithi175.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-10T19:15:52.092 INFO:tasks.workunit.client.0.smithi175.stderr:+ stat .snap/5/a 2023-10-10T19:15:52.092 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep -- -rw-rw-rw- 2023-10-10T19:15:52.093 INFO:tasks.workunit.client.0.smithi175.stdout:Access: (0666/-rw-rw-rw-) Uid: ( 1000/ ubuntu) Gid: ( 1275/ ubuntu) 2023-10-10T19:15:52.093 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/1 .snap/2 .snap/3 .snap/4 .snap/5 2023-10-10T19:15:52.119 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:15:52.120 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T19:15:52.120 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:15:52.120 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:15:52.157 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-parents.sh... 2023-10-10T19:15:52.157 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-parents.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-parents.sh 2023-10-10T19:15:52.227 INFO:tasks.workunit.client.0.smithi175.stdout:making directory tree and files 2023-10-10T19:15:52.234 INFO:tasks.workunit.client.0.smithi175.stdout:snapshotting 2023-10-10T19:15:52.262 INFO:tasks.workunit.client.0.smithi175.stdout:moving tree 2023-10-10T19:15:52.265 INFO:tasks.workunit.client.0.smithi175.stdout:checking snapshot contains tree... 2023-10-10T19:15:53.104 INFO:tasks.workunit.client.0.smithi175.stdout:Success! 2023-10-10T19:15:53.105 INFO:tasks.workunit.client.0.smithi175.stdout:adding folder and file to tree... 2023-10-10T19:15:53.107 INFO:tasks.workunit.client.0.smithi175.stdout:snapshotting tree 2 2023-10-10T19:15:53.134 INFO:tasks.workunit.client.0.smithi175.stdout:comparing snapshots 2023-10-10T19:15:53.521 INFO:tasks.workunit.client.0.smithi175.stdout:Success! 2023-10-10T19:15:53.521 INFO:tasks.workunit.client.0.smithi175.stdout:moving subtree to first folder 2023-10-10T19:15:53.526 INFO:tasks.workunit.client.0.smithi175.stdout:comparing snapshots and new tree 2023-10-10T19:15:53.698 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:15:53.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:15:54.963 INFO:tasks.workunit.client.0.smithi175.stdout:Success! 2023-10-10T19:15:55.003 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:15:55.004 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:15:55.004 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:15:56.281 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-realm-split.sh... 2023-10-10T19:15:56.282 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-realm-split.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh 2023-10-10T19:15:58.943 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T19:15:58.944 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir -p 1/a 2023-10-10T19:15:58.946 INFO:tasks.workunit.client.0.smithi175.stderr:+ exec 2023-10-10T19:15:58.947 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo -n a 2023-10-10T19:15:58.948 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir 1/.snap/s1 2023-10-10T19:15:59.482 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo -n b 2023-10-10T19:15:59.486 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir 2 2023-10-10T19:15:59.488 INFO:tasks.workunit.client.0.smithi175.stderr:+ mv 1/a 2 2023-10-10T19:15:59.491 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir 2/.snap/s2 2023-10-10T19:15:59.517 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo -n c 2023-10-10T19:15:59.521 INFO:tasks.workunit.client.0.smithi175.stderr:+ exec 2023-10-10T19:15:59.521 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep '^a$' 1/.snap/s1/a/file1 2023-10-10T19:15:59.522 INFO:tasks.workunit.client.0.smithi175.stdout:a 2023-10-10T19:15:59.522 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep '^ab$' 2/.snap/s2/a/file1 2023-10-10T19:15:59.533 INFO:tasks.workunit.client.0.smithi175.stdout:ab 2023-10-10T19:15:59.534 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep '^abc$' 2/a/file1 2023-10-10T19:15:59.535 INFO:tasks.workunit.client.0.smithi175.stdout:abc 2023-10-10T19:15:59.535 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir 1/.snap/s1 2023-10-10T19:15:59.540 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir 2/.snap/s2 2023-10-10T19:15:59.546 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -rf 1 2 2023-10-10T19:15:59.930 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:15:59.931 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T19:15:59.931 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:15:59.932 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:16:00.154 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-snap-rename.sh... 2023-10-10T19:16:00.154 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-snap-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rename.sh 2023-10-10T19:16:04.929 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T19:16:04.929 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir -p d1/d2 2023-10-10T19:16:04.932 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir -p d1/d3 2023-10-10T19:16:04.933 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir d1/.snap/foo 2023-10-10T19:16:04.959 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir d1/d2/.snap/foo 2023-10-10T19:16:04.981 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir d1/d3/.snap/foo 2023-10-10T19:16:05.003 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir d1/d3/.snap/bar 2023-10-10T19:16:05.024 INFO:tasks.workunit.client.0.smithi175.stderr:+ mv d1/d2/.snap/foo d1/d2/.snap/bar 2023-10-10T19:16:05.257 INFO:tasks.workunit.client.0.smithi175.stderr:+ expect_failure mv d1/d2/.snap/bar d1/d2/.snap/_bar 2023-10-10T19:16:05.257 INFO:tasks.workunit.client.0.smithi175.stderr:+ mv d1/d2/.snap/bar d1/d2/.snap/_bar 2023-10-10T19:16:05.259 INFO:tasks.workunit.client.0.smithi175.stderr:mv: cannot move 'd1/d2/.snap/bar' to a subdirectory of itself, 'd1/d2/.snap/_bar' 2023-10-10T19:16:05.259 INFO:tasks.workunit.client.0.smithi175.stderr:+ return 0 2023-10-10T19:16:05.260 INFO:tasks.workunit.client.0.smithi175.stderr:+ expect_failure mv d1/d2/.snap/_foo_2199023259936 d1/d2/.snap/foo 2023-10-10T19:16:05.260 INFO:tasks.workunit.client.0.smithi175.stderr:+ mv d1/d2/.snap/_foo_2199023259936 d1/d2/.snap/foo 2023-10-10T19:16:05.262 INFO:tasks.workunit.client.0.smithi175.stderr:mv: cannot move 'd1/d2/.snap/_foo_2199023259936' to a subdirectory of itself, 'd1/d2/.snap/foo' 2023-10-10T19:16:05.263 INFO:tasks.workunit.client.0.smithi175.stderr:+ return 0 2023-10-10T19:16:05.263 INFO:tasks.workunit.client.0.smithi175.stderr:+ expect_failure mv d1/d2/.snap/_foo_2199023259936 d1/d2/.snap/_foo_1 2023-10-10T19:16:05.263 INFO:tasks.workunit.client.0.smithi175.stderr:+ mv d1/d2/.snap/_foo_2199023259936 d1/d2/.snap/_foo_1 2023-10-10T19:16:05.265 INFO:tasks.workunit.client.0.smithi175.stderr:mv: cannot move 'd1/d2/.snap/_foo_2199023259936' to a subdirectory of itself, 'd1/d2/.snap/_foo_1' 2023-10-10T19:16:05.265 INFO:tasks.workunit.client.0.smithi175.stderr:+ return 0 2023-10-10T19:16:05.265 INFO:tasks.workunit.client.0.smithi175.stderr:+ expect_failure mv d1/d2/.snap/bar d1/.snap/ 2023-10-10T19:16:05.265 INFO:tasks.workunit.client.0.smithi175.stderr:+ mv d1/d2/.snap/bar d1/.snap/ 2023-10-10T19:16:05.266 INFO:tasks.workunit.client.0.smithi175.stderr:mv: cannot move 'd1/d2/.snap/bar' to 'd1/.snap/bar': Read-only file system 2023-10-10T19:16:05.267 INFO:tasks.workunit.client.0.smithi175.stderr:+ return 0 2023-10-10T19:16:05.267 INFO:tasks.workunit.client.0.smithi175.stderr:+ expect_failure python3 -c 'import os; os.rename('\''d1/d3/.snap/foo'\'', '\''d1/d3/.snap/bar'\'')' 2023-10-10T19:16:05.267 INFO:tasks.workunit.client.0.smithi175.stderr:+ python3 -c 'import os; os.rename('\''d1/d3/.snap/foo'\'', '\''d1/d3/.snap/bar'\'')' 2023-10-10T19:16:05.288 INFO:tasks.workunit.client.0.smithi175.stderr:Traceback (most recent call last): 2023-10-10T19:16:05.288 INFO:tasks.workunit.client.0.smithi175.stderr: File "", line 1, in 2023-10-10T19:16:05.289 INFO:tasks.workunit.client.0.smithi175.stderr:FileExistsError: [Errno 17] File exists: 'd1/d3/.snap/foo' -> 'd1/d3/.snap/bar' 2023-10-10T19:16:05.291 INFO:tasks.workunit.client.0.smithi175.stderr:+ return 0 2023-10-10T19:16:05.291 INFO:tasks.workunit.client.0.smithi175.stderr:+ expect_failure python3 -c 'import os; os.rename('\''d1/.snap/foo'\'', '\''d1/foo'\'')' 2023-10-10T19:16:05.291 INFO:tasks.workunit.client.0.smithi175.stderr:+ python3 -c 'import os; os.rename('\''d1/.snap/foo'\'', '\''d1/foo'\'')' 2023-10-10T19:16:05.310 INFO:tasks.workunit.client.0.smithi175.stderr:Traceback (most recent call last): 2023-10-10T19:16:05.310 INFO:tasks.workunit.client.0.smithi175.stderr: File "", line 1, in 2023-10-10T19:16:05.310 INFO:tasks.workunit.client.0.smithi175.stderr:OSError: [Errno 18] Invalid cross-device link: 'd1/.snap/foo' -> 'd1/foo' 2023-10-10T19:16:05.312 INFO:tasks.workunit.client.0.smithi175.stderr:+ return 0 2023-10-10T19:16:05.312 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir d1/.snap/foo 2023-10-10T19:16:05.320 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir d1/d2/.snap/bar 2023-10-10T19:16:05.325 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir d1/d3/.snap/foo 2023-10-10T19:16:05.331 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir d1/d3/.snap/bar 2023-10-10T19:16:05.337 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -rf d1 2023-10-10T19:16:05.358 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T19:16:05.359 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:16:05.359 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:16:05.360 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:16:06.188 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-snap-rm-cmp.sh... 2023-10-10T19:16:06.188 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-snap-rm-cmp.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rm-cmp.sh 2023-10-10T19:16:06.219 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T19:16:06.220 INFO:tasks.workunit.client.0.smithi175.stderr:+ file=linux-2.6.33.tar.bz2 2023-10-10T19:16:06.220 INFO:tasks.workunit.client.0.smithi175.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-10T19:16:10.399 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum linux-2.6.33.tar.bz2 2023-10-10T19:16:10.399 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:10.511 INFO:tasks.workunit.client.0.smithi175.stderr:+ real=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:10.512 INFO:tasks.workunit.client.0.smithi175.stderr:++ seq 1 20 2023-10-10T19:16:10.512 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:10.512 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 1 2023-10-10T19:16:10.513 INFO:tasks.workunit.client.0.smithi175.stdout:1 2023-10-10T19:16:10.514 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:11.038 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:11.061 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:11.065 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:11.319 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:11.319 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:11.417 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:11.417 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:11.417 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:11.427 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:11.427 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 2 2023-10-10T19:16:11.427 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:11.428 INFO:tasks.workunit.client.0.smithi175.stdout:2 2023-10-10T19:16:11.466 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:11.487 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:11.490 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:11.985 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:11.985 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:12.084 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:12.084 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:12.085 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:12.718 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:12.718 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 3 2023-10-10T19:16:12.718 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:12.719 INFO:tasks.workunit.client.0.smithi175.stdout:3 2023-10-10T19:16:12.761 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:13.649 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:13.653 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:14.523 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:14.523 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:14.623 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:14.623 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:14.624 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:14.636 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:14.636 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 4 2023-10-10T19:16:14.636 INFO:tasks.workunit.client.0.smithi175.stdout:4 2023-10-10T19:16:14.637 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:14.680 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:15.427 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:15.430 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:16.454 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:16.455 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:16.555 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:16.556 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:16.556 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:16.566 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:16.567 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 5 2023-10-10T19:16:16.567 INFO:tasks.workunit.client.0.smithi175.stdout:5 2023-10-10T19:16:16.568 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:16.622 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:16.644 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:16.648 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:17.147 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:17.147 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:17.246 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:17.247 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:17.247 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:17.257 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:17.257 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 6 2023-10-10T19:16:17.257 INFO:tasks.workunit.client.0.smithi175.stdout:6 2023-10-10T19:16:17.258 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:17.307 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:17.332 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:17.335 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:18.762 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:18.763 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:18.864 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:18.864 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:18.865 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:18.878 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:18.879 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 7 2023-10-10T19:16:18.879 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:18.879 INFO:tasks.workunit.client.0.smithi175.stdout:7 2023-10-10T19:16:18.924 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:18.947 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:18.952 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:19.509 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:19.510 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:19.610 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:19.611 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:19.611 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:19.622 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:19.622 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 8 2023-10-10T19:16:19.623 INFO:tasks.workunit.client.0.smithi175.stdout:8 2023-10-10T19:16:19.623 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:19.664 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:19.686 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:19.689 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:20.252 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:20.253 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:20.352 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:20.352 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:20.352 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:20.363 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:20.364 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 9 2023-10-10T19:16:20.364 INFO:tasks.workunit.client.0.smithi175.stdout:9 2023-10-10T19:16:20.364 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:20.409 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:20.510 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:20.514 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:20.996 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:20.997 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:21.096 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:21.096 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:21.096 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:21.105 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:21.106 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 10 2023-10-10T19:16:21.106 INFO:tasks.workunit.client.0.smithi175.stdout:10 2023-10-10T19:16:21.106 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:21.148 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:21.484 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:21.488 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:22.027 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:22.027 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:22.125 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:22.125 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:22.125 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:22.136 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:22.136 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 11 2023-10-10T19:16:22.137 INFO:tasks.workunit.client.0.smithi175.stdout:11 2023-10-10T19:16:22.137 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:22.181 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:23.147 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:23.150 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:23.766 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:16:23.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:16:23.953 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:23.954 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:24.054 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:24.054 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:24.055 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:24.063 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:24.064 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 12 2023-10-10T19:16:24.064 INFO:tasks.workunit.client.0.smithi175.stdout:12 2023-10-10T19:16:24.065 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:24.099 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:24.121 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:24.123 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:24.677 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:24.677 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:24.780 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:24.781 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:24.781 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:24.802 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:24.802 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 13 2023-10-10T19:16:24.803 INFO:tasks.workunit.client.0.smithi175.stdout:13 2023-10-10T19:16:24.803 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:24.844 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:24.865 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:24.867 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:25.442 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:25.442 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:25.543 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:25.543 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:25.543 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:26.132 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:26.132 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 14 2023-10-10T19:16:26.133 INFO:tasks.workunit.client.0.smithi175.stdout:14 2023-10-10T19:16:26.133 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:26.173 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:26.194 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:26.197 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:27.508 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:27.508 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:27.609 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:27.610 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:27.610 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:27.635 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:27.636 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 15 2023-10-10T19:16:27.636 INFO:tasks.workunit.client.0.smithi175.stdout:15 2023-10-10T19:16:27.636 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:27.673 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:27.695 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:27.698 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:28.237 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:28.238 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:28.338 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:28.338 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:28.338 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:28.348 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:28.348 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 16 2023-10-10T19:16:28.349 INFO:tasks.workunit.client.0.smithi175.stdout:16 2023-10-10T19:16:28.349 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:28.394 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:28.957 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:28.960 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:29.500 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:29.500 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:29.599 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:29.600 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:29.600 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:29.610 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:29.611 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 17 2023-10-10T19:16:29.611 INFO:tasks.workunit.client.0.smithi175.stdout:17 2023-10-10T19:16:29.611 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:29.656 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:29.678 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:29.681 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:30.248 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:30.248 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:30.348 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:30.349 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:30.349 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:30.957 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:30.958 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 18 2023-10-10T19:16:30.958 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:30.958 INFO:tasks.workunit.client.0.smithi175.stdout:18 2023-10-10T19:16:31.068 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:31.090 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:31.093 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:33.190 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:33.190 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:33.289 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:33.289 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:33.289 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:33.299 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:33.300 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 19 2023-10-10T19:16:33.300 INFO:tasks.workunit.client.0.smithi175.stdout:19 2023-10-10T19:16:33.300 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:33.351 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:34.282 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:34.285 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:34.785 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:34.785 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:34.889 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:34.889 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:34.889 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:34.899 INFO:tasks.workunit.client.0.smithi175.stderr:+ for f in `seq 1 20` 2023-10-10T19:16:34.899 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 20 2023-10-10T19:16:34.900 INFO:tasks.workunit.client.0.smithi175.stdout:20 2023-10-10T19:16:34.900 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-10-10T19:16:34.943 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s 2023-10-10T19:16:34.967 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm a 2023-10-10T19:16:34.969 INFO:tasks.workunit.client.0.smithi175.stderr:+ cp .snap/s/a /tmp/a 2023-10-10T19:16:35.484 INFO:tasks.workunit.client.0.smithi175.stderr:++ md5sum /tmp/a 2023-10-10T19:16:35.484 INFO:tasks.workunit.client.0.smithi175.stderr:++ awk '{print $1}' 2023-10-10T19:16:35.584 INFO:tasks.workunit.client.0.smithi175.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-10-10T19:16:35.585 INFO:tasks.workunit.client.0.smithi175.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-10-10T19:16:35.585 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s 2023-10-10T19:16:35.954 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm linux-2.6.33.tar.bz2 2023-10-10T19:16:35.957 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:16:35.958 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:16:36.552 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-upchildrealms.sh... 2023-10-10T19:16:36.552 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-upchildrealms.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-upchildrealms.sh 2023-10-10T19:16:39.929 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T19:16:39.929 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T19:16:39.931 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/b 2023-10-10T19:16:39.933 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/a1 2023-10-10T19:16:39.958 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/b/.snap/b1 2023-10-10T19:16:39.980 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo asdf 2023-10-10T19:16:39.981 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/a2 2023-10-10T19:16:40.005 INFO:tasks.workunit.client.0.smithi175.stderr:+ ln a/b/foo a/b/bar 2023-10-10T19:16:40.008 INFO:tasks.workunit.client.0.smithi175.stdout:this should not hang... 2023-10-10T19:16:40.008 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 'this should not hang...' 2023-10-10T19:16:40.186 INFO:tasks.workunit.client.0.smithi175.stderr:+ cat a/b/.snap/_a2_2199023259962/foo 2023-10-10T19:16:40.187 INFO:tasks.workunit.client.0.smithi175.stdout:asdf 2023-10-10T19:16:40.187 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 'good, it did not hang.' 2023-10-10T19:16:40.188 INFO:tasks.workunit.client.0.smithi175.stdout:good, it did not hang. 2023-10-10T19:16:40.188 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/b/.snap/b1 2023-10-10T19:16:40.195 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/a1 2023-10-10T19:16:40.201 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/a2 2023-10-10T19:16:40.211 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -r a 2023-10-10T19:16:40.257 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:16:40.258 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T19:16:40.258 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:16:40.259 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:16:40.294 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-xattrwb.sh... 2023-10-10T19:16:40.295 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/snaptest-xattrwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-xattrwb.sh 2023-10-10T19:16:44.927 INFO:tasks.workunit.client.0.smithi175.stderr:+ set -e 2023-10-10T19:16:44.927 INFO:tasks.workunit.client.0.smithi175.stdout:testing simple xattr wb 2023-10-10T19:16:44.928 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 'testing simple xattr wb' 2023-10-10T19:16:44.928 INFO:tasks.workunit.client.0.smithi175.stderr:+ touch x 2023-10-10T19:16:44.929 INFO:tasks.workunit.client.0.smithi175.stderr:+ setfattr -n user.foo x 2023-10-10T19:16:45.237 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir .snap/s1 2023-10-10T19:16:45.502 INFO:tasks.workunit.client.0.smithi175.stderr:+ getfattr -n user.foo .snap/s1/x 2023-10-10T19:16:45.502 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep user.foo 2023-10-10T19:16:45.508 INFO:tasks.workunit.client.0.smithi175.stdout:user.foo="" 2023-10-10T19:16:45.508 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm x 2023-10-10T19:16:45.514 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir .snap/s1 2023-10-10T19:16:45.526 INFO:tasks.workunit.client.0.smithi175.stdout:testing wb with pre-wb server cow 2023-10-10T19:16:45.526 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo 'testing wb with pre-wb server cow' 2023-10-10T19:16:45.526 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a 2023-10-10T19:16:45.527 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/b 2023-10-10T19:16:45.529 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/b/c 2023-10-10T19:16:45.531 INFO:tasks.workunit.client.0.smithi175.stderr:+ setfattr -n user.foo a/b 2023-10-10T19:16:45.532 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/.snap/s 2023-10-10T19:16:45.719 INFO:tasks.workunit.client.0.smithi175.stderr:+ mkdir a/b/cc 2023-10-10T19:16:45.722 INFO:tasks.workunit.client.0.smithi175.stderr:+ getfattr -n user.foo a/b 2023-10-10T19:16:45.722 INFO:tasks.workunit.client.0.smithi175.stdout:# file: a/b 2023-10-10T19:16:45.723 INFO:tasks.workunit.client.0.smithi175.stdout:user.foo="" 2023-10-10T19:16:45.723 INFO:tasks.workunit.client.0.smithi175.stdout: 2023-10-10T19:16:45.723 INFO:tasks.workunit.client.0.smithi175.stderr:+ getfattr -n user.foo a/.snap/s/b 2023-10-10T19:16:45.724 INFO:tasks.workunit.client.0.smithi175.stderr:+ grep user.foo 2023-10-10T19:16:45.961 INFO:tasks.workunit.client.0.smithi175.stdout:user.foo="" 2023-10-10T19:16:45.961 INFO:tasks.workunit.client.0.smithi175.stderr:+ rmdir a/.snap/s 2023-10-10T19:16:45.972 INFO:tasks.workunit.client.0.smithi175.stderr:+ rm -r a 2023-10-10T19:16:47.008 INFO:tasks.workunit.client.0.smithi175.stdout:OK 2023-10-10T19:16:47.009 INFO:tasks.workunit.client.0.smithi175.stderr:+ echo OK 2023-10-10T19:16:47.009 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T19:16:47.010 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T19:16:47.479 INFO:tasks.workunit:Running workunit fs/snaps/untar_snap_rm.sh... 2023-10-10T19:16:47.480 DEBUG:teuthology.orchestra.run.smithi175:workunit test fs/snaps/untar_snap_rm.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh 2023-10-10T19:16:49.530 INFO:tasks.workunit.client.0.smithi175.stderr:--2023-10-10 19:16:49-- http://download.ceph.com/qa/coreutils_8.5.orig.tar.gz 2023-10-10T19:16:49.530 INFO:tasks.workunit.client.0.smithi175.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2023-10-10T19:16:49.571 INFO:tasks.workunit.client.0.smithi175.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2023-10-10T19:16:49.606 INFO:tasks.workunit.client.0.smithi175.stderr:HTTP request sent, awaiting response... 200 OK 2023-10-10T19:16:49.607 INFO:tasks.workunit.client.0.smithi175.stderr:Length: 10740866 (10M) [application/octet-stream] 2023-10-10T19:16:49.608 INFO:tasks.workunit.client.0.smithi175.stderr:Saving to: ‘coreutils_8.5.orig.tar.gz’ 2023-10-10T19:16:49.608 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T19:16:49.678 INFO:tasks.workunit.client.0.smithi175.stderr: 0K .......... .......... .......... .......... .......... 0% 717K 15s 2023-10-10T19:16:49.715 INFO:tasks.workunit.client.0.smithi175.stderr: 50K .......... .......... .......... .......... .......... 0% 1.37M 11s 2023-10-10T19:16:49.715 INFO:tasks.workunit.client.0.smithi175.stderr: 100K .......... .......... .......... .......... .......... 1% 63.9M 7s 2023-10-10T19:16:49.717 INFO:tasks.workunit.client.0.smithi175.stderr: 150K .......... .......... .......... .......... .......... 1% 94.0M 5s 2023-10-10T19:16:49.757 INFO:tasks.workunit.client.0.smithi175.stderr: 200K .......... .......... .......... .......... .......... 2% 1.28M 6s 2023-10-10T19:16:49.758 INFO:tasks.workunit.client.0.smithi175.stderr: 250K .......... .......... .......... .......... .......... 2% 114M 5s 2023-10-10T19:16:49.758 INFO:tasks.workunit.client.0.smithi175.stderr: 300K .......... .......... .......... .......... .......... 3% 66.5M 4s 2023-10-10T19:16:49.816 INFO:tasks.workunit.client.0.smithi175.stderr: 350K .......... .......... .......... .......... .......... 3% 77.4M 4s 2023-10-10T19:16:49.817 INFO:tasks.workunit.client.0.smithi175.stderr: 400K .......... .......... .......... .......... .......... 4% 1.00M 4s 2023-10-10T19:16:49.817 INFO:tasks.workunit.client.0.smithi175.stderr: 450K .......... .......... .......... .......... .......... 4% 91.9M 4s 2023-10-10T19:16:49.817 INFO:tasks.workunit.client.0.smithi175.stderr: 500K .......... .......... .......... .......... .......... 5% 95.2M 4s 2023-10-10T19:16:49.817 INFO:tasks.workunit.client.0.smithi175.stderr: 550K .......... .......... .......... .......... .......... 5% 38.9M 3s 2023-10-10T19:16:49.817 INFO:tasks.workunit.client.0.smithi175.stderr: 600K .......... .......... .......... .......... .......... 6% 67.9M 3s 2023-10-10T19:16:49.818 INFO:tasks.workunit.client.0.smithi175.stderr: 650K .......... .......... .......... .......... .......... 6% 59.7M 3s 2023-10-10T19:16:49.818 INFO:tasks.workunit.client.0.smithi175.stderr: 700K .......... .......... .......... .......... .......... 7% 58.3M 3s 2023-10-10T19:16:49.818 INFO:tasks.workunit.client.0.smithi175.stderr: 750K .......... .......... .......... .......... .......... 7% 15.9M 2s 2023-10-10T19:16:49.878 INFO:tasks.workunit.client.0.smithi175.stderr: 800K .......... .......... .......... .......... .......... 8% 951K 3s 2023-10-10T19:16:49.878 INFO:tasks.workunit.client.0.smithi175.stderr: 850K .......... .......... .......... .......... .......... 8% 47.6M 3s 2023-10-10T19:16:49.879 INFO:tasks.workunit.client.0.smithi175.stderr: 900K .......... .......... .......... .......... .......... 9% 55.8M 3s 2023-10-10T19:16:49.879 INFO:tasks.workunit.client.0.smithi175.stderr: 950K .......... .......... .......... .......... .......... 9% 39.5M 2s 2023-10-10T19:16:49.879 INFO:tasks.workunit.client.0.smithi175.stderr: 1000K .......... .......... .......... .......... .......... 10% 56.9M 2s 2023-10-10T19:16:49.879 INFO:tasks.workunit.client.0.smithi175.stderr: 1050K .......... .......... .......... .......... .......... 10% 27.8M 2s 2023-10-10T19:16:49.879 INFO:tasks.workunit.client.0.smithi175.stderr: 1100K .......... .......... .......... .......... .......... 10% 42.8M 2s 2023-10-10T19:16:49.880 INFO:tasks.workunit.client.0.smithi175.stderr: 1150K .......... .......... .......... .......... .......... 11% 53.8M 2s 2023-10-10T19:16:49.880 INFO:tasks.workunit.client.0.smithi175.stderr: 1200K .......... .......... .......... .......... .......... 11% 13.5M 2s 2023-10-10T19:16:49.880 INFO:tasks.workunit.client.0.smithi175.stderr: 1250K .......... .......... .......... .......... .......... 12% 21.9M 2s 2023-10-10T19:16:49.931 INFO:tasks.workunit.client.0.smithi175.stderr: 1300K .......... .......... .......... .......... .......... 12% 1.34M 2s 2023-10-10T19:16:49.931 INFO:tasks.workunit.client.0.smithi175.stderr: 1350K .......... .......... .......... .......... .......... 13% 37.0M 2s 2023-10-10T19:16:49.932 INFO:tasks.workunit.client.0.smithi175.stderr: 1400K .......... .......... .......... .......... .......... 13% 39.3M 2s 2023-10-10T19:16:49.932 INFO:tasks.workunit.client.0.smithi175.stderr: 1450K .......... .......... .......... .......... .......... 14% 36.2M 2s 2023-10-10T19:16:49.932 INFO:tasks.workunit.client.0.smithi175.stderr: 1500K .......... .......... .......... .......... .......... 14% 31.2M 2s 2023-10-10T19:16:49.932 INFO:tasks.workunit.client.0.smithi175.stderr: 1550K .......... .......... .......... .......... .......... 15% 30.2M 2s 2023-10-10T19:16:49.932 INFO:tasks.workunit.client.0.smithi175.stderr: 1600K .......... .......... .......... .......... .......... 15% 37.4M 2s 2023-10-10T19:16:49.933 INFO:tasks.workunit.client.0.smithi175.stderr: 1650K .......... .......... .......... .......... .......... 16% 29.4M 2s 2023-10-10T19:16:49.933 INFO:tasks.workunit.client.0.smithi175.stderr: 1700K .......... .......... .......... .......... .......... 16% 16.7M 2s 2023-10-10T19:16:49.933 INFO:tasks.workunit.client.0.smithi175.stderr: 1750K .......... .......... .......... .......... .......... 17% 16.4M 2s 2023-10-10T19:16:49.976 INFO:tasks.workunit.client.0.smithi175.stderr: 1800K .......... .......... .......... .......... .......... 17% 1.33M 2s 2023-10-10T19:16:49.976 INFO:tasks.workunit.client.0.smithi175.stderr: 1850K .......... .......... .......... .......... .......... 18% 32.3M 2s 2023-10-10T19:16:49.977 INFO:tasks.workunit.client.0.smithi175.stderr: 1900K .......... .......... .......... .......... .......... 18% 59.2M 2s 2023-10-10T19:16:49.977 INFO:tasks.workunit.client.0.smithi175.stderr: 1950K .......... .......... .......... .......... .......... 19% 28.4M 2s 2023-10-10T19:16:49.977 INFO:tasks.workunit.client.0.smithi175.stderr: 2000K .......... .......... .......... .......... .......... 19% 29.2M 1s 2023-10-10T19:16:49.977 INFO:tasks.workunit.client.0.smithi175.stderr: 2050K .......... .......... .......... .......... .......... 20% 14.2M 1s 2023-10-10T19:16:49.979 INFO:tasks.workunit.client.0.smithi175.stderr: 2100K .......... .......... .......... .......... .......... 20% 18.0M 1s 2023-10-10T19:16:49.983 INFO:tasks.workunit.client.0.smithi175.stderr: 2150K .......... .......... .......... .......... .......... 20% 19.0M 1s 2023-10-10T19:16:49.984 INFO:tasks.workunit.client.0.smithi175.stderr: 2200K .......... .......... .......... .......... .......... 21% 16.8M 1s 2023-10-10T19:16:49.988 INFO:tasks.workunit.client.0.smithi175.stderr: 2250K .......... .......... .......... .......... .......... 21% 10.7M 1s 2023-10-10T19:16:49.991 INFO:tasks.workunit.client.0.smithi175.stderr: 2300K .......... .......... .......... .......... .......... 22% 17.6M 1s 2023-10-10T19:16:50.030 INFO:tasks.workunit.client.0.smithi175.stderr: 2350K .......... .......... .......... .......... .......... 22% 1.25M 1s 2023-10-10T19:16:50.034 INFO:tasks.workunit.client.0.smithi175.stderr: 2400K .......... .......... .......... .......... .......... 23% 23.2M 1s 2023-10-10T19:16:50.034 INFO:tasks.workunit.client.0.smithi175.stderr: 2450K .......... .......... .......... .......... .......... 23% 42.2M 1s 2023-10-10T19:16:50.035 INFO:tasks.workunit.client.0.smithi175.stderr: 2500K .......... .......... .......... .......... .......... 24% 25.4M 1s 2023-10-10T19:16:50.037 INFO:tasks.workunit.client.0.smithi175.stderr: 2550K .......... .......... .......... .......... .......... 24% 28.1M 1s 2023-10-10T19:16:50.041 INFO:tasks.workunit.client.0.smithi175.stderr: 2600K .......... .......... .......... .......... .......... 25% 20.6M 1s 2023-10-10T19:16:50.042 INFO:tasks.workunit.client.0.smithi175.stderr: 2650K .......... .......... .......... .......... .......... 25% 19.0M 1s 2023-10-10T19:16:50.046 INFO:tasks.workunit.client.0.smithi175.stderr: 2700K .......... .......... .......... .......... .......... 26% 17.2M 1s 2023-10-10T19:16:50.047 INFO:tasks.workunit.client.0.smithi175.stderr: 2750K .......... .......... .......... .......... .......... 26% 21.4M 1s 2023-10-10T19:16:50.050 INFO:tasks.workunit.client.0.smithi175.stderr: 2800K .......... .......... .......... .......... .......... 27% 21.6M 1s 2023-10-10T19:16:50.053 INFO:tasks.workunit.client.0.smithi175.stderr: 2850K .......... .......... .......... .......... .......... 27% 14.6M 1s 2023-10-10T19:16:50.096 INFO:tasks.workunit.client.0.smithi175.stderr: 2900K .......... .......... .......... .......... .......... 28% 1.30M 1s 2023-10-10T19:16:50.097 INFO:tasks.workunit.client.0.smithi175.stderr: 2950K .......... .......... .......... .......... .......... 28% 21.3M 1s 2023-10-10T19:16:50.097 INFO:tasks.workunit.client.0.smithi175.stderr: 3000K .......... .......... .......... .......... .......... 29% 50.5M 1s 2023-10-10T19:16:50.097 INFO:tasks.workunit.client.0.smithi175.stderr: 3050K .......... .......... .......... .......... .......... 29% 17.4M 1s 2023-10-10T19:16:50.099 INFO:tasks.workunit.client.0.smithi175.stderr: 3100K .......... .......... .......... .......... .......... 30% 22.4M 1s 2023-10-10T19:16:50.111 INFO:tasks.workunit.client.0.smithi175.stderr: 3150K .......... .......... .......... .......... .......... 30% 13.6M 1s 2023-10-10T19:16:50.112 INFO:tasks.workunit.client.0.smithi175.stderr: 3200K .......... .......... .......... .......... .......... 30% 43.1M 1s 2023-10-10T19:16:50.112 INFO:tasks.workunit.client.0.smithi175.stderr: 3250K .......... .......... .......... .......... .......... 31% 18.8M 1s 2023-10-10T19:16:50.112 INFO:tasks.workunit.client.0.smithi175.stderr: 3300K .......... .......... .......... .......... .......... 31% 21.8M 1s 2023-10-10T19:16:50.112 INFO:tasks.workunit.client.0.smithi175.stderr: 3350K .......... .......... .......... .......... .......... 32% 24.5M 1s 2023-10-10T19:16:50.113 INFO:tasks.workunit.client.0.smithi175.stderr: 3400K .......... .......... .......... .......... .......... 32% 16.3M 1s 2023-10-10T19:16:50.116 INFO:tasks.workunit.client.0.smithi175.stderr: 3450K .......... .......... .......... .......... .......... 33% 22.4M 1s 2023-10-10T19:16:50.150 INFO:tasks.workunit.client.0.smithi175.stderr: 3500K .......... .......... .......... .......... .......... 33% 1.47M 1s 2023-10-10T19:16:50.154 INFO:tasks.workunit.client.0.smithi175.stderr: 3550K .......... .......... .......... .......... .......... 34% 16.1M 1s 2023-10-10T19:16:50.154 INFO:tasks.workunit.client.0.smithi175.stderr: 3600K .......... .......... .......... .......... .......... 34% 53.2M 1s 2023-10-10T19:16:50.155 INFO:tasks.workunit.client.0.smithi175.stderr: 3650K .......... .......... .......... .......... .......... 35% 13.7M 1s 2023-10-10T19:16:50.159 INFO:tasks.workunit.client.0.smithi175.stderr: 3700K .......... .......... .......... .......... .......... 35% 15.5M 1s 2023-10-10T19:16:50.163 INFO:tasks.workunit.client.0.smithi175.stderr: 3750K .......... .......... .......... .......... .......... 36% 11.4M 1s 2023-10-10T19:16:50.168 INFO:tasks.workunit.client.0.smithi175.stderr: 3800K .......... .......... .......... .......... .......... 36% 35.1M 1s 2023-10-10T19:16:50.169 INFO:tasks.workunit.client.0.smithi175.stderr: 3850K .......... .......... .......... .......... .......... 37% 13.8M 1s 2023-10-10T19:16:50.171 INFO:tasks.workunit.client.0.smithi175.stderr: 3900K .......... .......... .......... .......... .......... 37% 16.7M 1s 2023-10-10T19:16:50.173 INFO:tasks.workunit.client.0.smithi175.stderr: 3950K .......... .......... .......... .......... .......... 38% 20.6M 1s 2023-10-10T19:16:50.176 INFO:tasks.workunit.client.0.smithi175.stderr: 4000K .......... .......... .......... .......... .......... 38% 25.1M 1s 2023-10-10T19:16:50.182 INFO:tasks.workunit.client.0.smithi175.stderr: 4050K .......... .......... .......... .......... .......... 39% 12.5M 1s 2023-10-10T19:16:50.216 INFO:tasks.workunit.client.0.smithi175.stderr: 4100K .......... .......... .......... .......... .......... 39% 1.42M 1s 2023-10-10T19:16:50.216 INFO:tasks.workunit.client.0.smithi175.stderr: 4150K .......... .......... .......... .......... .......... 40% 32.2M 1s 2023-10-10T19:16:50.221 INFO:tasks.workunit.client.0.smithi175.stderr: 4200K .......... .......... .......... .......... .......... 40% 17.1M 1s 2023-10-10T19:16:50.221 INFO:tasks.workunit.client.0.smithi175.stderr: 4250K .......... .......... .......... .......... .......... 40% 26.5M 1s 2023-10-10T19:16:50.223 INFO:tasks.workunit.client.0.smithi175.stderr: 4300K .......... .......... .......... .......... .......... 41% 15.5M 1s 2023-10-10T19:16:50.226 INFO:tasks.workunit.client.0.smithi175.stderr: 4350K .......... .......... .......... .......... .......... 41% 14.0M 1s 2023-10-10T19:16:50.233 INFO:tasks.workunit.client.0.smithi175.stderr: 4400K .......... .......... .......... .......... .......... 42% 16.4M 1s 2023-10-10T19:16:50.233 INFO:tasks.workunit.client.0.smithi175.stderr: 4450K .......... .......... .......... .......... .......... 42% 44.8M 1s 2023-10-10T19:16:50.245 INFO:tasks.workunit.client.0.smithi175.stderr: 4500K .......... .......... .......... .......... .......... 43% 9.03M 1s 2023-10-10T19:16:50.245 INFO:tasks.workunit.client.0.smithi175.stderr: 4550K .......... .......... .......... .......... .......... 43% 31.1M 1s 2023-10-10T19:16:50.245 INFO:tasks.workunit.client.0.smithi175.stderr: 4600K .......... .......... .......... .......... .......... 44% 27.9M 1s 2023-10-10T19:16:50.246 INFO:tasks.workunit.client.0.smithi175.stderr: 4650K .......... .......... .......... .......... .......... 44% 22.1M 1s 2023-10-10T19:16:50.249 INFO:tasks.workunit.client.0.smithi175.stderr: 4700K .......... .......... .......... .......... .......... 45% 10.4M 1s 2023-10-10T19:16:50.279 INFO:tasks.workunit.client.0.smithi175.stderr: 4750K .......... .......... .......... .......... .......... 45% 1.59M 1s 2023-10-10T19:16:50.280 INFO:tasks.workunit.client.0.smithi175.stderr: 4800K .......... .......... .......... .......... .......... 46% 15.3M 1s 2023-10-10T19:16:50.286 INFO:tasks.workunit.client.0.smithi175.stderr: 4850K .......... .......... .......... .......... .......... 46% 16.7M 1s 2023-10-10T19:16:50.286 INFO:tasks.workunit.client.0.smithi175.stderr: 4900K .......... .......... .......... .......... .......... 47% 21.7M 1s 2023-10-10T19:16:50.289 INFO:tasks.workunit.client.0.smithi175.stderr: 4950K .......... .......... .......... .......... .......... 47% 15.2M 1s 2023-10-10T19:16:50.292 INFO:tasks.workunit.client.0.smithi175.stderr: 5000K .......... .......... .......... .......... .......... 48% 18.4M 1s 2023-10-10T19:16:50.297 INFO:tasks.workunit.client.0.smithi175.stderr: 5050K .......... .......... .......... .......... .......... 48% 16.8M 1s 2023-10-10T19:16:50.297 INFO:tasks.workunit.client.0.smithi175.stderr: 5100K .......... .......... .......... .......... .......... 49% 72.1M 1s 2023-10-10T19:16:50.299 INFO:tasks.workunit.client.0.smithi175.stderr: 5150K .......... .......... .......... .......... .......... 49% 10.9M 1s 2023-10-10T19:16:50.313 INFO:tasks.workunit.client.0.smithi175.stderr: 5200K .......... .......... .......... .......... .......... 50% 23.8M 1s 2023-10-10T19:16:50.313 INFO:tasks.workunit.client.0.smithi175.stderr: 5250K .......... .......... .......... .......... .......... 50% 22.0M 1s 2023-10-10T19:16:50.314 INFO:tasks.workunit.client.0.smithi175.stderr: 5300K .......... .......... .......... .......... .......... 51% 20.6M 1s 2023-10-10T19:16:50.314 INFO:tasks.workunit.client.0.smithi175.stderr: 5350K .......... .......... .......... .......... .......... 51% 12.7M 1s 2023-10-10T19:16:50.314 INFO:tasks.workunit.client.0.smithi175.stderr: 5400K .......... .......... .......... .......... .......... 51% 76.8M 1s 2023-10-10T19:16:50.340 INFO:tasks.workunit.client.0.smithi175.stderr: 5450K .......... .......... .......... .......... .......... 52% 1.71M 1s 2023-10-10T19:16:50.347 INFO:tasks.workunit.client.0.smithi175.stderr: 5500K .......... .......... .......... .......... .......... 52% 16.5M 1s 2023-10-10T19:16:50.347 INFO:tasks.workunit.client.0.smithi175.stderr: 5550K .......... .......... .......... .......... .......... 53% 38.6M 1s 2023-10-10T19:16:50.347 INFO:tasks.workunit.client.0.smithi175.stderr: 5600K .......... .......... .......... .......... .......... 53% 25.6M 1s 2023-10-10T19:16:50.349 INFO:tasks.workunit.client.0.smithi175.stderr: 5650K .......... .......... .......... .......... .......... 54% 12.4M 1s 2023-10-10T19:16:50.362 INFO:tasks.workunit.client.0.smithi175.stderr: 5700K .......... .......... .......... .......... .......... 54% 17.3M 1s 2023-10-10T19:16:50.363 INFO:tasks.workunit.client.0.smithi175.stderr: 5750K .......... .......... .......... .......... .......... 55% 20.6M 1s 2023-10-10T19:16:50.363 INFO:tasks.workunit.client.0.smithi175.stderr: 5800K .......... .......... .......... .......... .......... 55% 66.9M 1s 2023-10-10T19:16:50.363 INFO:tasks.workunit.client.0.smithi175.stderr: 5850K .......... .......... .......... .......... .......... 56% 21.1M 1s 2023-10-10T19:16:50.363 INFO:tasks.workunit.client.0.smithi175.stderr: 5900K .......... .......... .......... .......... .......... 56% 13.8M 1s 2023-10-10T19:16:50.378 INFO:tasks.workunit.client.0.smithi175.stderr: 5950K .......... .......... .......... .......... .......... 57% 16.8M 1s 2023-10-10T19:16:50.379 INFO:tasks.workunit.client.0.smithi175.stderr: 6000K .......... .......... .......... .......... .......... 57% 23.1M 1s 2023-10-10T19:16:50.379 INFO:tasks.workunit.client.0.smithi175.stderr: 6050K .......... .......... .......... .......... .......... 58% 16.5M 1s 2023-10-10T19:16:50.379 INFO:tasks.workunit.client.0.smithi175.stderr: 6100K .......... .......... .......... .......... .......... 58% 30.6M 1s 2023-10-10T19:16:50.396 INFO:tasks.workunit.client.0.smithi175.stderr: 6150K .......... .......... .......... .......... .......... 59% 1.89M 1s 2023-10-10T19:16:50.399 INFO:tasks.workunit.client.0.smithi175.stderr: 6200K .......... .......... .......... .......... .......... 59% 20.6M 1s 2023-10-10T19:16:50.408 INFO:tasks.workunit.client.0.smithi175.stderr: 6250K .......... .......... .......... .......... .......... 60% 16.8M 1s 2023-10-10T19:16:50.409 INFO:tasks.workunit.client.0.smithi175.stderr: 6300K .......... .......... .......... .......... .......... 60% 29.0M 1s 2023-10-10T19:16:50.409 INFO:tasks.workunit.client.0.smithi175.stderr: 6350K .......... .......... .......... .......... .......... 61% 14.0M 1s 2023-10-10T19:16:50.421 INFO:tasks.workunit.client.0.smithi175.stderr: 6400K .......... .......... .......... .......... .......... 61% 17.9M 1s 2023-10-10T19:16:50.421 INFO:tasks.workunit.client.0.smithi175.stderr: 6450K .......... .......... .......... .......... .......... 61% 24.5M 0s 2023-10-10T19:16:50.422 INFO:tasks.workunit.client.0.smithi175.stderr: 6500K .......... .......... .......... .......... .......... 62% 27.4M 0s 2023-10-10T19:16:50.422 INFO:tasks.workunit.client.0.smithi175.stderr: 6550K .......... .......... .......... .......... .......... 62% 23.4M 0s 2023-10-10T19:16:50.422 INFO:tasks.workunit.client.0.smithi175.stderr: 6600K .......... .......... .......... .......... .......... 63% 13.9M 0s 2023-10-10T19:16:50.422 INFO:tasks.workunit.client.0.smithi175.stderr: 6650K .......... .......... .......... .......... .......... 63% 57.2M 0s 2023-10-10T19:16:50.431 INFO:tasks.workunit.client.0.smithi175.stderr: 6700K .......... .......... .......... .......... .......... 64% 15.9M 0s 2023-10-10T19:16:50.432 INFO:tasks.workunit.client.0.smithi175.stderr: 6750K .......... .......... .......... .......... .......... 64% 23.9M 0s 2023-10-10T19:16:50.432 INFO:tasks.workunit.client.0.smithi175.stderr: 6800K .......... .......... .......... .......... .......... 65% 27.3M 0s 2023-10-10T19:16:50.432 INFO:tasks.workunit.client.0.smithi175.stderr: 6850K .......... .......... .......... .......... .......... 65% 23.4M 0s 2023-10-10T19:16:50.451 INFO:tasks.workunit.client.0.smithi175.stderr: 6900K .......... .......... .......... .......... .......... 66% 2.26M 0s 2023-10-10T19:16:50.453 INFO:tasks.workunit.client.0.smithi175.stderr: 6950K .......... .......... .......... .......... .......... 66% 22.7M 0s 2023-10-10T19:16:50.460 INFO:tasks.workunit.client.0.smithi175.stderr: 7000K .......... .......... .......... .......... .......... 67% 19.7M 0s 2023-10-10T19:16:50.460 INFO:tasks.workunit.client.0.smithi175.stderr: 7050K .......... .......... .......... .......... .......... 67% 20.7M 0s 2023-10-10T19:16:50.461 INFO:tasks.workunit.client.0.smithi175.stderr: 7100K .......... .......... .......... .......... .......... 68% 11.4M 0s 2023-10-10T19:16:50.473 INFO:tasks.workunit.client.0.smithi175.stderr: 7150K .......... .......... .......... .......... .......... 68% 26.0M 0s 2023-10-10T19:16:50.473 INFO:tasks.workunit.client.0.smithi175.stderr: 7200K .......... .......... .......... .......... .......... 69% 29.1M 0s 2023-10-10T19:16:50.473 INFO:tasks.workunit.client.0.smithi175.stderr: 7250K .......... .......... .......... .......... .......... 69% 20.0M 0s 2023-10-10T19:16:50.473 INFO:tasks.workunit.client.0.smithi175.stderr: 7300K .......... .......... .......... .......... .......... 70% 52.3M 0s 2023-10-10T19:16:50.474 INFO:tasks.workunit.client.0.smithi175.stderr: 7350K .......... .......... .......... .......... .......... 70% 21.5M 0s 2023-10-10T19:16:50.476 INFO:tasks.workunit.client.0.smithi175.stderr: 7400K .......... .......... .......... .......... .......... 71% 10.9M 0s 2023-10-10T19:16:50.488 INFO:tasks.workunit.client.0.smithi175.stderr: 7450K .......... .......... .......... .......... .......... 71% 12.4M 0s 2023-10-10T19:16:50.489 INFO:tasks.workunit.client.0.smithi175.stderr: 7500K .......... .......... .......... .......... .......... 71% 74.6M 0s 2023-10-10T19:16:50.489 INFO:tasks.workunit.client.0.smithi175.stderr: 7550K .......... .......... .......... .......... .......... 72% 24.5M 0s 2023-10-10T19:16:50.489 INFO:tasks.workunit.client.0.smithi175.stderr: 7600K .......... .......... .......... .......... .......... 72% 17.5M 0s 2023-10-10T19:16:50.489 INFO:tasks.workunit.client.0.smithi175.stderr: 7650K .......... .......... .......... .......... .......... 73% 32.2M 0s 2023-10-10T19:16:50.516 INFO:tasks.workunit.client.0.smithi175.stderr: 7700K .......... .......... .......... .......... .......... 73% 2.49M 0s 2023-10-10T19:16:50.516 INFO:tasks.workunit.client.0.smithi175.stderr: 7750K .......... .......... .......... .......... .......... 74% 17.4M 0s 2023-10-10T19:16:50.516 INFO:tasks.workunit.client.0.smithi175.stderr: 7800K .......... .......... .......... .......... .......... 74% 59.3M 0s 2023-10-10T19:16:50.517 INFO:tasks.workunit.client.0.smithi175.stderr: 7850K .......... .......... .......... .......... .......... 75% 17.0M 0s 2023-10-10T19:16:50.517 INFO:tasks.workunit.client.0.smithi175.stderr: 7900K .......... .......... .......... .......... .......... 75% 12.2M 0s 2023-10-10T19:16:50.529 INFO:tasks.workunit.client.0.smithi175.stderr: 7950K .......... .......... .......... .......... .......... 76% 23.5M 0s 2023-10-10T19:16:50.529 INFO:tasks.workunit.client.0.smithi175.stderr: 8000K .......... .......... .......... .......... .......... 76% 23.1M 0s 2023-10-10T19:16:50.529 INFO:tasks.workunit.client.0.smithi175.stderr: 8050K .......... .......... .......... .......... .......... 77% 16.0M 0s 2023-10-10T19:16:50.529 INFO:tasks.workunit.client.0.smithi175.stderr: 8100K .......... .......... .......... .......... .......... 77% 61.1M 0s 2023-10-10T19:16:50.530 INFO:tasks.workunit.client.0.smithi175.stderr: 8150K .......... .......... .......... .......... .......... 78% 18.4M 0s 2023-10-10T19:16:50.537 INFO:tasks.workunit.client.0.smithi175.stderr: 8200K .......... .......... .......... .......... .......... 78% 15.1M 0s 2023-10-10T19:16:50.538 INFO:tasks.workunit.client.0.smithi175.stderr: 8250K .......... .......... .......... .......... .......... 79% 24.9M 0s 2023-10-10T19:16:50.538 INFO:tasks.workunit.client.0.smithi175.stderr: 8300K .......... .......... .......... .......... .......... 79% 23.0M 0s 2023-10-10T19:16:50.538 INFO:tasks.workunit.client.0.smithi175.stderr: 8350K .......... .......... .......... .......... .......... 80% 18.4M 0s 2023-10-10T19:16:50.544 INFO:tasks.workunit.client.0.smithi175.stderr: 8400K .......... .......... .......... .......... .......... 80% 31.9M 0s 2023-10-10T19:16:50.544 INFO:tasks.workunit.client.0.smithi175.stderr: 8450K .......... .......... .......... .......... .......... 81% 18.4M 0s 2023-10-10T19:16:50.559 INFO:tasks.workunit.client.0.smithi175.stderr: 8500K .......... .......... .......... .......... .......... 81% 2.76M 0s 2023-10-10T19:16:50.611 INFO:tasks.workunit.client.0.smithi175.stderr: 8550K .......... .......... .......... .......... .......... 81% 32.8M 0s 2023-10-10T19:16:50.612 INFO:tasks.workunit.client.0.smithi175.stderr: 8600K .......... .......... .......... .......... .......... 82% 11.9M 0s 2023-10-10T19:16:50.612 INFO:tasks.workunit.client.0.smithi175.stderr: 8650K .......... .......... .......... .......... .......... 82% 24.9M 0s 2023-10-10T19:16:50.612 INFO:tasks.workunit.client.0.smithi175.stderr: 8700K .......... .......... .......... .......... .......... 83% 22.7M 0s 2023-10-10T19:16:50.612 INFO:tasks.workunit.client.0.smithi175.stderr: 8750K .......... .......... .......... .......... .......... 83% 11.8M 0s 2023-10-10T19:16:50.612 INFO:tasks.workunit.client.0.smithi175.stderr: 8800K .......... .......... .......... .......... .......... 84% 35.2M 0s 2023-10-10T19:16:50.613 INFO:tasks.workunit.client.0.smithi175.stderr: 8850K .......... .......... .......... .......... .......... 84% 13.0M 0s 2023-10-10T19:16:50.613 INFO:tasks.workunit.client.0.smithi175.stderr: 8900K .......... .......... .......... .......... .......... 85% 26.4M 0s 2023-10-10T19:16:50.613 INFO:tasks.workunit.client.0.smithi175.stderr: 8950K .......... .......... .......... .......... .......... 85% 51.0M 0s 2023-10-10T19:16:50.613 INFO:tasks.workunit.client.0.smithi175.stderr: 9000K .......... .......... .......... .......... .......... 86% 14.0M 0s 2023-10-10T19:16:50.613 INFO:tasks.workunit.client.0.smithi175.stderr: 9050K .......... .......... .......... .......... .......... 86% 12.4M 0s 2023-10-10T19:16:50.614 INFO:tasks.workunit.client.0.smithi175.stderr: 9100K .......... .......... .......... .......... .......... 87% 28.8M 0s 2023-10-10T19:16:50.614 INFO:tasks.workunit.client.0.smithi175.stderr: 9150K .......... .......... .......... .......... .......... 87% 28.1M 0s 2023-10-10T19:16:50.614 INFO:tasks.workunit.client.0.smithi175.stderr: 9200K .......... .......... .......... .......... .......... 88% 28.4M 0s 2023-10-10T19:16:50.614 INFO:tasks.workunit.client.0.smithi175.stderr: 9250K .......... .......... .......... .......... .......... 88% 21.8M 0s 2023-10-10T19:16:50.615 INFO:tasks.workunit.client.0.smithi175.stderr: 9300K .......... .......... .......... .......... .......... 89% 35.0M 0s 2023-10-10T19:16:50.620 INFO:tasks.workunit.client.0.smithi175.stderr: 9350K .......... .......... .......... .......... .......... 89% 2.41M 0s 2023-10-10T19:16:50.620 INFO:tasks.workunit.client.0.smithi175.stderr: 9400K .......... .......... .......... .......... .......... 90% 52.1M 0s 2023-10-10T19:16:50.625 INFO:tasks.workunit.client.0.smithi175.stderr: 9450K .......... .......... .......... .......... .......... 90% 12.5M 0s 2023-10-10T19:16:50.626 INFO:tasks.workunit.client.0.smithi175.stderr: 9500K .......... .......... .......... .......... .......... 91% 70.0M 0s 2023-10-10T19:16:50.629 INFO:tasks.workunit.client.0.smithi175.stderr: 9550K .......... .......... .......... .......... .......... 91% 12.8M 0s 2023-10-10T19:16:50.664 INFO:tasks.workunit.client.0.smithi175.stderr: 9600K .......... .......... .......... .......... .......... 92% 11.8M 0s 2023-10-10T19:16:50.665 INFO:tasks.workunit.client.0.smithi175.stderr: 9650K .......... .......... .......... .......... .......... 92% 44.1M 0s 2023-10-10T19:16:50.665 INFO:tasks.workunit.client.0.smithi175.stderr: 9700K .......... .......... .......... .......... .......... 92% 24.1M 0s 2023-10-10T19:16:50.666 INFO:tasks.workunit.client.0.smithi175.stderr: 9750K .......... .......... .......... .......... .......... 93% 26.4M 0s 2023-10-10T19:16:50.666 INFO:tasks.workunit.client.0.smithi175.stderr: 9800K .......... .......... .......... .......... .......... 93% 25.8M 0s 2023-10-10T19:16:50.666 INFO:tasks.workunit.client.0.smithi175.stderr: 9850K .......... .......... .......... .......... .......... 94% 22.3M 0s 2023-10-10T19:16:50.667 INFO:tasks.workunit.client.0.smithi175.stderr: 9900K .......... .......... .......... .......... .......... 94% 23.2M 0s 2023-10-10T19:16:50.667 INFO:tasks.workunit.client.0.smithi175.stderr: 9950K .......... .......... .......... .......... .......... 95% 17.3M 0s 2023-10-10T19:16:50.668 INFO:tasks.workunit.client.0.smithi175.stderr: 10000K .......... .......... .......... .......... .......... 95% 47.1M 0s 2023-10-10T19:16:50.668 INFO:tasks.workunit.client.0.smithi175.stderr: 10050K .......... .......... .......... .......... .......... 96% 24.0M 0s 2023-10-10T19:16:50.668 INFO:tasks.workunit.client.0.smithi175.stderr: 10100K .......... .......... .......... .......... .......... 96% 22.7M 0s 2023-10-10T19:16:50.669 INFO:tasks.workunit.client.0.smithi175.stderr: 10150K .......... .......... .......... .......... .......... 97% 33.2M 0s 2023-10-10T19:16:50.669 INFO:tasks.workunit.client.0.smithi175.stderr: 10200K .......... .......... .......... .......... .......... 97% 26.0M 0s 2023-10-10T19:16:50.670 INFO:tasks.workunit.client.0.smithi175.stderr: 10250K .......... .......... .......... .......... .......... 98% 40.6M 0s 2023-10-10T19:16:50.683 INFO:tasks.workunit.client.0.smithi175.stderr: 10300K .......... .......... .......... .......... .......... 98% 2.47M 0s 2023-10-10T19:16:50.684 INFO:tasks.workunit.client.0.smithi175.stderr: 10350K .......... .......... .......... .......... .......... 99% 38.6M 0s 2023-10-10T19:16:50.684 INFO:tasks.workunit.client.0.smithi175.stderr: 10400K .......... .......... .......... .......... .......... 99% 13.5M 0s 2023-10-10T19:16:50.685 INFO:tasks.workunit.client.0.smithi175.stderr: 10450K .......... .......... .......... ......... 100% 25.5M=1.1s 2023-10-10T19:16:50.685 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T19:16:50.685 INFO:tasks.workunit.client.0.smithi175.stderr:2023-10-10 19:16:50 (9.55 MB/s) - ‘coreutils_8.5.orig.tar.gz’ saved [10740866/10740866] 2023-10-10T19:16:50.686 INFO:tasks.workunit.client.0.smithi175.stderr: 2023-10-10T19:16:50.766 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/ 2023-10-10T19:16:50.767 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/maint.mk 2023-10-10T19:16:50.768 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/THANKS 2023-10-10T19:16:50.769 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/ABOUT-NLS 2023-10-10T19:16:50.770 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/ 2023-10-10T19:16:50.770 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/textutils/ 2023-10-10T19:16:50.771 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/textutils/ChangeLog 2023-10-10T19:16:50.774 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/textutils/NEWS 2023-10-10T19:16:50.774 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/fileutils/ 2023-10-10T19:16:50.775 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/fileutils/ChangeLog 2023-10-10T19:16:50.778 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/fileutils/ChangeLog-1997 2023-10-10T19:16:50.779 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/fileutils/NEWS 2023-10-10T19:16:50.780 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/sh-utils/ 2023-10-10T19:16:50.781 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/sh-utils/ChangeLog 2023-10-10T19:16:50.783 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/sh-utils/NEWS 2023-10-10T19:16:50.784 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/old/sh-utils/ChangeLog.0 2023-10-10T19:16:50.784 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_prohibit_strcmp 2023-10-10T19:16:50.785 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/ 2023-10-10T19:16:50.786 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/config.rpath 2023-10-10T19:16:50.787 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/cvsu 2023-10-10T19:16:50.788 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/mdate-sh 2023-10-10T19:16:50.789 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/gitlog-to-changelog 2023-10-10T19:16:50.789 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/gnu-web-doc-update 2023-10-10T19:16:50.790 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/useless-if-before-free 2023-10-10T19:16:50.792 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/config.sub 2023-10-10T19:16:50.792 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/missing 2023-10-10T19:16:50.792 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/compile 2023-10-10T19:16:50.793 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/install-sh 2023-10-10T19:16:50.793 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/gendocs.sh 2023-10-10T19:16:50.794 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/update-copyright 2023-10-10T19:16:50.795 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/texinfo.tex 2023-10-10T19:16:50.797 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/unused-parameter.h 2023-10-10T19:16:50.798 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/c++defs.h 2023-10-10T19:16:50.798 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/config.guess 2023-10-10T19:16:50.799 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/do-release-commit-and-tag 2023-10-10T19:16:50.799 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/git-version-gen 2023-10-10T19:16:50.800 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/arg-nonnull.h 2023-10-10T19:16:50.801 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/gnupload 2023-10-10T19:16:50.802 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/warn-on-use.h 2023-10-10T19:16:50.803 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/vc-list-files 2023-10-10T19:16:50.804 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/ylwrap 2023-10-10T19:16:50.804 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/ChangeLog-2007 2023-10-10T19:16:50.805 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/announce-gen 2023-10-10T19:16:50.806 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/build-aux/depcomp 2023-10-10T19:16:50.807 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_error_message_uppercase 2023-10-10T19:16:50.808 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_obsolete_symbols 2023-10-10T19:16:50.809 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/INSTALL 2023-10-10T19:16:50.810 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gl/ 2023-10-10T19:16:50.811 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gl/modules/ 2023-10-10T19:16:50.812 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gl/modules/getloadavg.diff 2023-10-10T19:16:50.813 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/configure.ac 2023-10-10T19:16:50.814 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_prohibit_tab_based_indentation 2023-10-10T19:16:50.814 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/GNUmakefile 2023-10-10T19:16:50.815 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/ChangeLog 2023-10-10T19:16:50.818 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/COPYING 2023-10-10T19:16:50.819 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ 2023-10-10T19:16:50.820 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/closeout.m4 2023-10-10T19:16:50.821 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/dirent-safer.m4 2023-10-10T19:16:50.821 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/settime.m4 2023-10-10T19:16:50.822 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/readtokens.m4 2023-10-10T19:16:50.823 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/clock_time.m4 2023-10-10T19:16:50.824 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/config-h.m4 2023-10-10T19:16:50.824 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/euidaccess.m4 2023-10-10T19:16:50.825 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/printf-frexpl.m4 2023-10-10T19:16:50.826 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/posix-shell.m4 2023-10-10T19:16:50.827 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/servent.m4 2023-10-10T19:16:50.828 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xstrtol.m4 2023-10-10T19:16:50.828 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mkdir-p.m4 2023-10-10T19:16:50.829 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/00gnulib.m4 2023-10-10T19:16:50.830 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbsinit.m4 2023-10-10T19:16:50.831 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/dup2.m4 2023-10-10T19:16:50.832 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/eealloc.m4 2023-10-10T19:16:50.832 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/savewd.m4 2023-10-10T19:16:50.833 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memrchr.m4 2023-10-10T19:16:50.834 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/d-ino.m4 2023-10-10T19:16:50.835 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/bison.m4 2023-10-10T19:16:50.836 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/isnanl.m4 2023-10-10T19:16:50.837 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lib-ld.m4 2023-10-10T19:16:50.838 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stpcpy.m4 2023-10-10T19:16:50.838 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/progtest.m4 2023-10-10T19:16:50.839 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/unistd_h.m4 2023-10-10T19:16:50.840 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/time_h.m4 2023-10-10T19:16:50.841 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mkancesdirs.m4 2023-10-10T19:16:50.841 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/signalblocking.m4 2023-10-10T19:16:50.842 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/unlink.m4 2023-10-10T19:16:50.843 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/backupfile.m4 2023-10-10T19:16:50.843 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/remove.m4 2023-10-10T19:16:50.844 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fts.m4 2023-10-10T19:16:50.845 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xalloc.m4 2023-10-10T19:16:50.845 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/inet_ntop.m4 2023-10-10T19:16:50.846 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/iconv_open.m4 2023-10-10T19:16:50.847 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/extensions.m4 2023-10-10T19:16:50.848 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbrtowc.m4 2023-10-10T19:16:50.849 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/check-decl.m4 2023-10-10T19:16:50.850 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/cycle-check.m4 2023-10-10T19:16:50.851 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/signal_h.m4 2023-10-10T19:16:50.852 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sys_socket_h.m4 2023-10-10T19:16:50.852 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fprintftime.m4 2023-10-10T19:16:50.853 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lib-check.m4 2023-10-10T19:16:50.854 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/isblank.m4 2023-10-10T19:16:50.855 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xstrtod.m4 2023-10-10T19:16:50.855 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stdarg.m4 2023-10-10T19:16:50.856 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/selinux-selinux-h.m4 2023-10-10T19:16:50.857 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fstypename.m4 2023-10-10T19:16:50.858 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/link.m4 2023-10-10T19:16:50.858 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/md5.m4 2023-10-10T19:16:50.859 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/multiarch.m4 2023-10-10T19:16:50.860 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ioctl.m4 2023-10-10T19:16:50.861 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/flexmember.m4 2023-10-10T19:16:50.861 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/locale-fr.m4 2023-10-10T19:16:50.862 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fdopendir.m4 2023-10-10T19:16:50.863 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/exponentf.m4 2023-10-10T19:16:50.864 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/utimecmp.m4 2023-10-10T19:16:50.864 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/human.m4 2023-10-10T19:16:50.865 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strerror.m4 2023-10-10T19:16:50.866 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/socklen.m4 2023-10-10T19:16:50.867 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/dirent_h.m4 2023-10-10T19:16:50.868 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/locale-zh.m4 2023-10-10T19:16:50.869 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getdate.m4 2023-10-10T19:16:50.869 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/cloexec.m4 2023-10-10T19:16:50.870 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/i-ring.m4 2023-10-10T19:16:50.871 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/group-member.m4 2023-10-10T19:16:50.872 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strtod.m4 2023-10-10T19:16:50.873 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/chdir-long.m4 2023-10-10T19:16:50.874 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strnlen.m4 2023-10-10T19:16:50.874 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strtoimax.m4 2023-10-10T19:16:50.875 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getgroups.m4 2023-10-10T19:16:50.875 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/yield.m4 2023-10-10T19:16:50.876 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/write.m4 2023-10-10T19:16:50.877 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/localcharset.m4 2023-10-10T19:16:50.878 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/usleep.m4 2023-10-10T19:16:50.878 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memmove.m4 2023-10-10T19:16:50.879 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/close.m4 2023-10-10T19:16:50.880 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memcmp.m4 2023-10-10T19:16:50.881 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/autobuild.m4 2023-10-10T19:16:50.881 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fchdir.m4 2023-10-10T19:16:50.882 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fd-reopen.m4 2023-10-10T19:16:50.883 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/jm-winsz2.m4 2023-10-10T19:16:50.884 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/posixver.m4 2023-10-10T19:16:50.884 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/locale-ja.m4 2023-10-10T19:16:50.885 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stat-prog.m4 2023-10-10T19:16:50.886 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/wint_t.m4 2023-10-10T19:16:50.887 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/st_dm_mode.m4 2023-10-10T19:16:50.887 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/tempname.m4 2023-10-10T19:16:50.888 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/frexp.m4 2023-10-10T19:16:50.889 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/dirfd.m4 2023-10-10T19:16:50.889 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sys_select_h.m4 2023-10-10T19:16:50.890 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/btowc.m4 2023-10-10T19:16:50.891 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/filenamecat.m4 2023-10-10T19:16:50.891 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gmp.m4 2023-10-10T19:16:50.892 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/isapipe.m4 2023-10-10T19:16:50.893 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/wctype_h.m4 2023-10-10T19:16:50.893 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ldexpl.m4 2023-10-10T19:16:50.894 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getloadavg.m4 2023-10-10T19:16:50.895 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbchar.m4 2023-10-10T19:16:50.896 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/version-etc.m4 2023-10-10T19:16:50.897 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fcntl_h.m4 2023-10-10T19:16:50.897 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gettime.m4 2023-10-10T19:16:50.898 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/wchar_h.m4 2023-10-10T19:16:50.899 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/vasprintf-posix.m4 2023-10-10T19:16:50.899 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/selinux-context-h.m4 2023-10-10T19:16:50.900 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getpass.m4 2023-10-10T19:16:50.901 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mempcpy.m4 2023-10-10T19:16:50.902 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/save-cwd.m4 2023-10-10T19:16:50.902 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fclose.m4 2023-10-10T19:16:50.903 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/modechange.m4 2023-10-10T19:16:50.904 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strstr.m4 2023-10-10T19:16:50.904 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getdtablesize.m4 2023-10-10T19:16:50.905 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/wchar_t.m4 2023-10-10T19:16:51.167 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mkdir.m4 2023-10-10T19:16:51.167 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/idcache.m4 2023-10-10T19:16:51.168 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/tm_gmtoff.m4 2023-10-10T19:16:51.169 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/dirname.m4 2023-10-10T19:16:51.170 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbiter.m4 2023-10-10T19:16:51.172 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fcntl-safer.m4 2023-10-10T19:16:51.172 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/manywarnings.m4 2023-10-10T19:16:51.581 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/same.m4 2023-10-10T19:16:51.582 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getdelim.m4 2023-10-10T19:16:51.583 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mountlist.m4 2023-10-10T19:16:51.584 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lchmod.m4 2023-10-10T19:16:51.586 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/readlink.m4 2023-10-10T19:16:51.587 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gnu-make.m4 2023-10-10T19:16:51.588 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/thread.m4 2023-10-10T19:16:51.589 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/hostent.m4 2023-10-10T19:16:51.591 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/math_h.m4 2023-10-10T19:16:51.592 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fpending.m4 2023-10-10T19:16:51.593 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xgetcwd.m4 2023-10-10T19:16:51.594 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/utimens.m4 2023-10-10T19:16:51.595 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/exponentd.m4 2023-10-10T19:16:51.596 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getaddrinfo.m4 2023-10-10T19:16:51.597 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sys_ioctl_h.m4 2023-10-10T19:16:51.599 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stdlib_h.m4 2023-10-10T19:16:51.600 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mgetgroups.m4 2023-10-10T19:16:51.601 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/open.m4 2023-10-10T19:16:51.602 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strpbrk.m4 2023-10-10T19:16:51.603 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ssize_t.m4 2023-10-10T19:16:51.604 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/priv-set.m4 2023-10-10T19:16:51.605 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fcntl-o.m4 2023-10-10T19:16:51.607 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/inttostr.m4 2023-10-10T19:16:51.607 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/malloca.m4 2023-10-10T19:16:51.608 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/unlocked-io.m4 2023-10-10T19:16:51.609 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getugroups.m4 2023-10-10T19:16:51.610 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sha1.m4 2023-10-10T19:16:51.611 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fpurge.m4 2023-10-10T19:16:51.611 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gnulib-comp.m4 2023-10-10T19:16:51.612 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/wcrtomb.m4 2023-10-10T19:16:51.613 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/putenv.m4 2023-10-10T19:16:51.614 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lstat.m4 2023-10-10T19:16:54.076 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:16:54.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:16:54.988 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strsignal.m4 2023-10-10T19:16:54.989 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbslen.m4 2023-10-10T19:16:54.990 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/inttypes.m4 2023-10-10T19:16:54.990 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stdlib-safer.m4 2023-10-10T19:16:54.991 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strtol.m4 2023-10-10T19:16:54.991 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lib-ignore.m4 2023-10-10T19:16:54.992 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/readutmp.m4 2023-10-10T19:16:54.993 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strtoll.m4 2023-10-10T19:16:54.993 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stdbool.m4 2023-10-10T19:16:54.994 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/setenv.m4 2023-10-10T19:16:54.995 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/alloca.m4 2023-10-10T19:16:54.995 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/size_max.m4 2023-10-10T19:16:54.996 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/include_next.m4 2023-10-10T19:16:54.997 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sys_stat_h.m4 2023-10-10T19:16:54.997 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strndup.m4 2023-10-10T19:16:54.998 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/link-follow.m4 2023-10-10T19:16:54.999 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ctype.m4 2023-10-10T19:16:55.000 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/timespec.m4 2023-10-10T19:16:55.001 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getline.m4 2023-10-10T19:16:55.002 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/calloc.m4 2023-10-10T19:16:55.002 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strftime.m4 2023-10-10T19:16:55.003 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/arpa_inet_h.m4 2023-10-10T19:16:55.004 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sig2str.m4 2023-10-10T19:16:55.004 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/printf.m4 2023-10-10T19:16:55.005 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/environ.m4 2023-10-10T19:16:55.006 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/vprintf-posix.m4 2023-10-10T19:16:55.007 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/intlmacosx.m4 2023-10-10T19:16:55.008 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/locale-tr.m4 2023-10-10T19:16:55.008 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mktime.m4 2023-10-10T19:16:55.009 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/iconv.m4 2023-10-10T19:16:55.010 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/pathmax.m4 2023-10-10T19:16:55.011 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/printf-frexp.m4 2023-10-10T19:16:55.012 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/userspec.m4 2023-10-10T19:16:55.013 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/host-os.m4 2023-10-10T19:16:55.014 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/file-type.m4 2023-10-10T19:16:55.015 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ls-mntd-fs.m4 2023-10-10T19:16:55.019 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sha512.m4 2023-10-10T19:16:55.019 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stdio-safer.m4 2023-10-10T19:16:55.020 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lock.m4 2023-10-10T19:16:55.020 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/c-strtod.m4 2023-10-10T19:16:55.020 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/closein.m4 2023-10-10T19:16:55.023 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/frexpl.m4 2023-10-10T19:16:55.023 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/posixtm.m4 2023-10-10T19:16:55.023 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/rmdir.m4 2023-10-10T19:16:55.023 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/chown.m4 2023-10-10T19:16:55.024 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stdint.m4 2023-10-10T19:16:55.024 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stddef_h.m4 2023-10-10T19:16:55.025 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sigaction.m4 2023-10-10T19:16:55.025 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/error.m4 2023-10-10T19:16:55.026 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/vasnprintf.m4 2023-10-10T19:16:55.027 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/utimbuf.m4 2023-10-10T19:16:55.027 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sockets.m4 2023-10-10T19:16:55.028 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/canonicalize.m4 2023-10-10T19:16:55.029 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sys_time_h.m4 2023-10-10T19:16:55.029 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/unlink-busy.m4 2023-10-10T19:16:55.030 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gettimeofday.m4 2023-10-10T19:16:55.031 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/write-any-file.m4 2023-10-10T19:16:55.032 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strdup.m4 2023-10-10T19:16:55.033 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/regex.m4 2023-10-10T19:16:55.034 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sleep.m4 2023-10-10T19:16:55.035 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/float_h.m4 2023-10-10T19:16:55.035 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/warn-on-use.m4 2023-10-10T19:16:55.036 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/atexit.m4 2023-10-10T19:16:55.037 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getcwd-path-max.m4 2023-10-10T19:16:55.038 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fnmatch.m4 2023-10-10T19:16:55.039 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strnumcmp.m4 2023-10-10T19:16:55.040 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gnulib-common.m4 2023-10-10T19:16:55.041 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/codeset.m4 2023-10-10T19:16:55.041 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/time_r.m4 2023-10-10T19:16:55.042 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memcoll.m4 2023-10-10T19:16:55.043 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/string_h.m4 2023-10-10T19:16:55.044 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sys_utsname_h.m4 2023-10-10T19:16:55.045 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stdint_h.m4 2023-10-10T19:16:55.045 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/nproc.m4 2023-10-10T19:16:55.047 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/symlink.m4 2023-10-10T19:16:55.047 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/rename.m4 2023-10-10T19:16:55.048 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/read-file.m4 2023-10-10T19:16:55.049 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strtoumax.m4 2023-10-10T19:16:55.050 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getopt.m4 2023-10-10T19:16:55.051 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fopen.m4 2023-10-10T19:16:55.052 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ungetc.m4 2023-10-10T19:16:55.053 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xsize.m4 2023-10-10T19:16:55.054 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fflush.m4 2023-10-10T19:16:55.055 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/acl.m4 2023-10-10T19:16:55.056 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fsusage.m4 2023-10-10T19:16:55.058 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gethrxtime.m4 2023-10-10T19:16:55.059 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/nls.m4 2023-10-10T19:16:55.060 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lib-link.m4 2023-10-10T19:16:55.061 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mpsort.m4 2023-10-10T19:16:55.063 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/netinet_in_h.m4 2023-10-10T19:16:55.064 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fseeko.m4 2023-10-10T19:16:55.065 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/inet_pton.m4 2023-10-10T19:16:55.066 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/yesno.m4 2023-10-10T19:16:55.067 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/physmem.m4 2023-10-10T19:16:55.068 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/intmax_t.m4 2023-10-10T19:16:55.069 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/inttypes-pri.m4 2023-10-10T19:16:55.070 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/malloc.m4 2023-10-10T19:16:55.071 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/jm-winsz1.m4 2023-10-10T19:16:55.072 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/close-stream.m4 2023-10-10T19:16:55.074 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/nocrash.m4 2023-10-10T19:16:55.075 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/realloc.m4 2023-10-10T19:16:55.075 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/canon-host.m4 2023-10-10T19:16:55.076 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memcpy.m4 2023-10-10T19:16:55.077 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/tls.m4 2023-10-10T19:16:55.077 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/perl.m4 2023-10-10T19:16:55.078 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/signbit.m4 2023-10-10T19:16:55.079 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memchr.m4 2023-10-10T19:16:55.080 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/d-type.m4 2023-10-10T19:16:55.081 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lchown.m4 2023-10-10T19:16:55.082 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lib-prefix.m4 2023-10-10T19:16:55.082 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getcwd.m4 2023-10-10T19:16:55.083 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/locale_h.m4 2023-10-10T19:16:55.084 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memxfrm.m4 2023-10-10T19:16:55.085 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbstate_t.m4 2023-10-10T19:16:55.085 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/filemode.m4 2023-10-10T19:16:55.086 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memset.m4 2023-10-10T19:16:55.087 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gethostname.m4 2023-10-10T19:16:55.088 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/vasprintf.m4 2023-10-10T19:16:55.089 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/unistd-safer.m4 2023-10-10T19:16:55.089 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xfts.m4 2023-10-10T19:16:55.090 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/uptime.m4 2023-10-10T19:16:55.091 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/prereq.m4 2023-10-10T19:16:55.092 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/symlinkat.m4 2023-10-10T19:16:55.092 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getusershell.m4 2023-10-10T19:16:55.093 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/isnand.m4 2023-10-10T19:16:55.094 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/select.m4 2023-10-10T19:16:55.095 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/wctob.m4 2023-10-10T19:16:55.095 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/double-slash-root.m4 2023-10-10T19:16:55.096 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/long-options.m4 2023-10-10T19:16:55.097 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/glibc21.m4 2023-10-10T19:16:55.097 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stat-time.m4 2023-10-10T19:16:55.098 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/rpmatch.m4 2023-10-10T19:16:55.099 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xattr.m4 2023-10-10T19:16:55.100 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/gettext.m4 2023-10-10T19:16:55.101 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fsync.m4 2023-10-10T19:16:55.101 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/exclude.m4 2023-10-10T19:16:55.102 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/freopen.m4 2023-10-10T19:16:55.103 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strtoul.m4 2023-10-10T19:16:55.104 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/netdb_h.m4 2023-10-10T19:16:55.105 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/snprintf.m4 2023-10-10T19:16:55.105 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stpncpy.m4 2023-10-10T19:16:55.106 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xnanosleep.m4 2023-10-10T19:16:55.107 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/warnings.m4 2023-10-10T19:16:55.107 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/lseek.m4 2023-10-10T19:16:55.108 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/hard-locale.m4 2023-10-10T19:16:55.109 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/include-exclude-prog.m4 2023-10-10T19:16:55.110 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mmap-anon.m4 2023-10-10T19:16:55.110 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mkstemp.m4 2023-10-10T19:16:55.111 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/openat.m4 2023-10-10T19:16:55.112 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbrlen.m4 2023-10-10T19:16:55.113 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/jm-macros.m4 2023-10-10T19:16:55.114 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ftello.m4 2023-10-10T19:16:55.114 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stdio_h.m4 2023-10-10T19:16:55.115 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/exponentl.m4 2023-10-10T19:16:55.116 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getpagesize.m4 2023-10-10T19:16:55.117 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/argmatch.m4 2023-10-10T19:16:55.118 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/perror.m4 2023-10-10T19:16:55.118 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/inline.m4 2023-10-10T19:16:55.119 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/inttypes_h.m4 2023-10-10T19:16:55.120 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/iconv_h.m4 2023-10-10T19:16:55.120 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/base64.m4 2023-10-10T19:16:55.121 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/stat.m4 2023-10-10T19:16:55.122 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/tzset.m4 2023-10-10T19:16:55.123 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/unicodeio.m4 2023-10-10T19:16:55.124 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sha256.m4 2023-10-10T19:16:55.125 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/errno_h.m4 2023-10-10T19:16:55.125 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/unlinkdir.m4 2023-10-10T19:16:55.126 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/boottime.m4 2023-10-10T19:16:55.127 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/freading.m4 2023-10-10T19:16:55.128 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mode_t.m4 2023-10-10T19:16:55.129 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getndelim2.m4 2023-10-10T19:16:55.129 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xvasprintf.m4 2023-10-10T19:16:55.130 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbsrtowcs.m4 2023-10-10T19:16:55.131 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/hash.m4 2023-10-10T19:16:55.131 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/safe-write.m4 2023-10-10T19:16:55.132 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/po.m4 2023-10-10T19:16:55.133 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/isnanf.m4 2023-10-10T19:16:55.134 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/longlong.m4 2023-10-10T19:16:55.135 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/xstrndup.m4 2023-10-10T19:16:55.136 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/euidaccess-stat.m4 2023-10-10T19:16:55.136 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/getcwd-abort-bug.m4 2023-10-10T19:16:55.137 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/faccessat.m4 2023-10-10T19:16:55.138 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/nanosleep.m4 2023-10-10T19:16:55.139 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/ChangeLog-2007 2023-10-10T19:16:55.140 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/utimes.m4 2023-10-10T19:16:55.141 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fileblocks.m4 2023-10-10T19:16:55.142 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/sockpfaf.m4 2023-10-10T19:16:55.142 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strtoull.m4 2023-10-10T19:16:55.143 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/assert.m4 2023-10-10T19:16:55.144 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/quotearg.m4 2023-10-10T19:16:55.145 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/uname.m4 2023-10-10T19:16:55.146 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/savedir.m4 2023-10-10T19:16:55.146 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/threadlib.m4 2023-10-10T19:16:55.147 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/langinfo_h.m4 2023-10-10T19:16:55.148 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/linkat.m4 2023-10-10T19:16:55.149 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fpieee.m4 2023-10-10T19:16:55.150 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/memcasecmp.m4 2023-10-10T19:16:55.151 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/vfprintf-posix.m4 2023-10-10T19:16:55.152 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/dos.m4 2023-10-10T19:16:55.153 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/root-dev-ino.m4 2023-10-10T19:16:55.154 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/quote.m4 2023-10-10T19:16:55.155 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/mbswidth.m4 2023-10-10T19:16:55.155 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/nl_langinfo.m4 2023-10-10T19:16:55.156 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/fcntl.m4 2023-10-10T19:16:55.157 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/strcspn.m4 2023-10-10T19:16:55.158 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/safe-read.m4 2023-10-10T19:16:55.159 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/m4/wcwidth.m4 2023-10-10T19:16:55.160 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/Makefile.am 2023-10-10T19:16:55.161 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_po_check 2023-10-10T19:16:55.162 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/dist-check.mk 2023-10-10T19:16:55.163 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_GPL_version 2023-10-10T19:16:55.164 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_unmarked_diagnostics 2023-10-10T19:16:55.165 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.prev-version 2023-10-10T19:16:55.166 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/bootstrap.conf 2023-10-10T19:16:55.167 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/ChangeLog-2005 2023-10-10T19:16:55.171 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/aclocal.m4 2023-10-10T19:16:55.172 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/AUTHORS 2023-10-10T19:16:55.173 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/cfg.mk 2023-10-10T19:16:55.173 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_file_system 2023-10-10T19:16:55.174 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.version 2023-10-10T19:16:55.175 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/TODO 2023-10-10T19:16:55.175 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_space_tab 2023-10-10T19:16:55.176 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_prohibit_empty_lines_at_EOF 2023-10-10T19:16:55.177 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/Makefile.in 2023-10-10T19:16:55.178 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/ 2023-10-10T19:16:55.179 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/cat.x 2023-10-10T19:16:55.179 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/date.x 2023-10-10T19:16:55.180 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/basename.1 2023-10-10T19:16:55.182 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/pathchk.x 2023-10-10T19:16:55.183 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tail.x 2023-10-10T19:16:55.183 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/yes.x 2023-10-10T19:16:55.183 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/pinky.1 2023-10-10T19:16:55.184 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/kill.x 2023-10-10T19:16:55.184 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/pathchk.1 2023-10-10T19:16:55.185 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sync.1 2023-10-10T19:16:55.185 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/dir.1 2023-10-10T19:16:55.186 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mv.1 2023-10-10T19:16:55.187 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/rm.1 2023-10-10T19:16:55.188 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/dircolors.x 2023-10-10T19:16:55.188 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/rmdir.1 2023-10-10T19:16:55.189 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tty.1 2023-10-10T19:16:55.190 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/fold.x 2023-10-10T19:16:55.190 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/fold.1 2023-10-10T19:16:55.191 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/dir.x 2023-10-10T19:16:55.192 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/fmt.x 2023-10-10T19:16:55.192 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/paste.1 2023-10-10T19:16:55.193 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chmod.x 2023-10-10T19:16:55.194 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/od.1 2023-10-10T19:16:55.194 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/ptx.x 2023-10-10T19:16:55.195 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/nice.1 2023-10-10T19:16:55.196 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/install.1 2023-10-10T19:16:55.196 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/dd.x 2023-10-10T19:16:55.197 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/cksum.1 2023-10-10T19:16:55.198 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/test.1 2023-10-10T19:16:55.198 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chgrp.1 2023-10-10T19:16:55.199 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/unlink.1 2023-10-10T19:16:55.200 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chmod.1 2023-10-10T19:16:55.201 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/dd.1 2023-10-10T19:16:55.201 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/du.1 2023-10-10T19:16:55.202 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tsort.1 2023-10-10T19:16:55.203 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/ls.1 2023-10-10T19:16:55.203 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/split.x 2023-10-10T19:16:55.204 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/nohup.x 2023-10-10T19:16:55.205 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha1sum.1 2023-10-10T19:16:55.206 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tr.1 2023-10-10T19:16:55.207 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/head.x 2023-10-10T19:16:55.207 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/false.1 2023-10-10T19:16:55.208 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/pwd.x 2023-10-10T19:16:55.209 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/logname.x 2023-10-10T19:16:55.209 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/paste.x 2023-10-10T19:16:55.210 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/nl.x 2023-10-10T19:16:55.211 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha224sum.1 2023-10-10T19:16:55.212 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mkdir.1 2023-10-10T19:16:55.212 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/nl.1 2023-10-10T19:16:55.213 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha256sum.1 2023-10-10T19:16:55.213 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/printf.x 2023-10-10T19:16:55.214 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/md5sum.1 2023-10-10T19:16:55.215 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/Makefile.am 2023-10-10T19:16:55.216 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/dirname.1 2023-10-10T19:16:55.217 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/shuf.x 2023-10-10T19:16:55.217 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/echo.1 2023-10-10T19:16:55.218 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chcon.x 2023-10-10T19:16:55.219 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chroot.1 2023-10-10T19:16:55.219 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha384sum.x 2023-10-10T19:16:55.220 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/stdbuf.1 2023-10-10T19:16:55.220 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/users.1 2023-10-10T19:16:55.221 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/readlink.x 2023-10-10T19:16:55.222 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/cut.x 2023-10-10T19:16:55.223 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/groups.1 2023-10-10T19:16:55.223 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/seq.1 2023-10-10T19:16:55.224 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/help2man 2023-10-10T19:16:55.225 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/dirname.x 2023-10-10T19:16:55.226 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/uname.x 2023-10-10T19:16:55.226 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/uptime.x 2023-10-10T19:16:55.227 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/cut.1 2023-10-10T19:16:55.228 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/cksum.x 2023-10-10T19:16:55.229 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/nohup.1 2023-10-10T19:16:55.230 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mv.x 2023-10-10T19:16:55.230 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sort.1 2023-10-10T19:16:55.231 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chown.1 2023-10-10T19:16:55.232 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/pr.x 2023-10-10T19:16:55.233 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/true.x 2023-10-10T19:16:55.234 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/base64.1 2023-10-10T19:16:55.234 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/factor.1 2023-10-10T19:16:55.235 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/ln.x 2023-10-10T19:16:55.236 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/head.1 2023-10-10T19:16:55.237 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/unexpand.1 2023-10-10T19:16:55.237 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/df.x 2023-10-10T19:16:55.238 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/join.x 2023-10-10T19:16:55.239 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/du.x 2023-10-10T19:16:55.239 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/dircolors.1 2023-10-10T19:16:55.240 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/uniq.x 2023-10-10T19:16:55.241 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/split.1 2023-10-10T19:16:55.241 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tty.x 2023-10-10T19:16:55.242 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/uname.1 2023-10-10T19:16:55.243 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/uptime.1 2023-10-10T19:16:55.503 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sort.x 2023-10-10T19:16:55.503 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha256sum.x 2023-10-10T19:16:55.504 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha512sum.x 2023-10-10T19:16:55.505 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/nice.x 2023-10-10T19:16:55.505 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/pinky.x 2023-10-10T19:16:55.506 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/fmt.1 2023-10-10T19:16:55.507 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/expr.1 2023-10-10T19:16:55.508 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/wc.1 2023-10-10T19:16:55.508 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/true.1 2023-10-10T19:16:55.509 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/comm.x 2023-10-10T19:16:55.510 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/truncate.x 2023-10-10T19:16:55.510 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/install.x 2023-10-10T19:16:55.511 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/wc.x 2023-10-10T19:16:55.512 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/uniq.1 2023-10-10T19:16:55.513 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/arch.x 2023-10-10T19:16:55.514 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sum.x 2023-10-10T19:16:55.515 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/timeout.x 2023-10-10T19:16:55.516 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/link.x 2023-10-10T19:16:56.485 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tac.1 2023-10-10T19:16:56.485 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sync.x 2023-10-10T19:16:56.487 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/Makefile.in 2023-10-10T19:16:56.488 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mkfifo.1 2023-10-10T19:16:56.489 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/od.x 2023-10-10T19:16:56.490 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/vdir.1 2023-10-10T19:16:56.491 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/timeout.1 2023-10-10T19:16:56.492 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mkfifo.x 2023-10-10T19:16:56.492 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/basename.x 2023-10-10T19:16:56.494 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/expr.x 2023-10-10T19:16:56.495 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mktemp.x 2023-10-10T19:16:56.496 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/vdir.x 2023-10-10T19:16:56.497 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/su.x 2023-10-10T19:16:56.498 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/env.1 2023-10-10T19:16:56.499 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/who.x 2023-10-10T19:16:56.500 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/hostid.1 2023-10-10T19:16:56.501 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tee.x 2023-10-10T19:16:56.502 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/rmdir.x 2023-10-10T19:16:56.503 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/hostid.x 2023-10-10T19:16:56.504 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/id.x 2023-10-10T19:16:56.505 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/unexpand.x 2023-10-10T19:16:56.506 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/pr.1 2023-10-10T19:16:56.507 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/csplit.1 2023-10-10T19:16:56.508 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/truncate.1 2023-10-10T19:16:56.509 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tee.1 2023-10-10T19:16:56.510 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/env.x 2023-10-10T19:16:56.511 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chroot.x 2023-10-10T19:16:56.512 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/cp.x 2023-10-10T19:16:56.512 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/whoami.x 2023-10-10T19:16:56.513 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/logname.1 2023-10-10T19:16:56.514 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mknod.1 2023-10-10T19:16:56.515 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/groups.x 2023-10-10T19:16:56.515 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/cp.1 2023-10-10T19:16:56.516 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/runcon.x 2023-10-10T19:16:56.517 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/yes.1 2023-10-10T19:16:56.518 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/stat.1 2023-10-10T19:16:56.519 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/nproc.x 2023-10-10T19:16:56.520 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/unlink.x 2023-10-10T19:16:56.521 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha384sum.1 2023-10-10T19:16:56.521 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/nproc.1 2023-10-10T19:16:56.522 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/seq.x 2023-10-10T19:16:56.523 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha512sum.1 2023-10-10T19:16:56.524 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/stat.x 2023-10-10T19:16:58.628 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:16:58.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.753+0000 7fbd04bcd700 1 Processor -- start 2023-10-10T19:16:58.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.753+0000 7fbd04bcd700 1 -- start start 2023-10-10T19:16:58.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.754+0000 7fbd04bcd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbd000c2b80 0x7fbd00156580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:58.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.754+0000 7fbd04bcd700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0015a6c0 con 0x7fbd000c35a0 2023-10-10T19:16:58.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.754+0000 7fbd04bcd700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd00159830 con 0x7fbd00156ac0 2023-10-10T19:16:58.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.754+0000 7fbd04bcd700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0014d4a0 con 0x7fbd000c2b80 2023-10-10T19:16:58.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.754+0000 7fbcfdd9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbd000c35a0 0x7fbd000c6a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51932/0 (socket says 172.21.15.175:51932) 2023-10-10T19:16:58.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.754+0000 7fbcfdd9b700 1 -- 172.21.15.175:0/958226868 learned_addr learned my addr 172.21.15.175:0/958226868 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:16:58.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfe59c700 1 --2- 172.21.15.175:0/958226868 >> v2:172.21.15.191:3300/0 conn(0x7fbd000c2b80 0x7fbd00156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:58.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfd59a700 1 -- 172.21.15.175:0/958226868 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601007811 0 0) 0x7fbd0015a6c0 con 0x7fbd000c35a0 2023-10-10T19:16:58.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfd59a700 1 -- 172.21.15.175:0/958226868 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce8003570 con 0x7fbd000c35a0 2023-10-10T19:16:58.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfe59c700 1 -- 172.21.15.175:0/958226868 >> v1:172.21.15.191:6789/0 conn(0x7fbd00156ac0 legacy=0x7fbd000c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:58.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfd59a700 1 -- 172.21.15.175:0/958226868 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2760605153 0 0) 0x7fbce8003570 con 0x7fbd000c35a0 2023-10-10T19:16:58.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfe59c700 1 -- 172.21.15.175:0/958226868 >> v1:172.21.15.175:6789/0 conn(0x7fbd000c35a0 legacy=0x7fbd000c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:58.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfe59c700 1 -- 172.21.15.175:0/958226868 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd00159ab0 con 0x7fbd000c2b80 2023-10-10T19:16:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfe59c700 1 --2- 172.21.15.175:0/958226868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 0x7fbd00156580 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009e60 tx=0x7fbcf0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d61858285829a14e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:16:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfd59a700 1 -- 172.21.15.175:0/958226868 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf000e070 con 0x7fbd000c2b80 2023-10-10T19:16:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfd59a700 1 -- 172.21.15.175:0/958226868 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf000b850 con 0x7fbd000c2b80 2023-10-10T19:16:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.755+0000 7fbcfd59a700 1 -- 172.21.15.175:0/958226868 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4001470 con 0x7fbd000c2b80 2023-10-10T19:16:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.756+0000 7fbd04bcd700 1 -- 172.21.15.175:0/958226868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 msgr2=0x7fbd00156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.756+0000 7fbd04bcd700 1 --2- 172.21.15.175:0/958226868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 0x7fbd00156580 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009e60 tx=0x7fbcf0009780 comp rx=0 tx=0).stop 2023-10-10T19:16:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.756+0000 7fbd04bcd700 1 -- 172.21.15.175:0/958226868 shutdown_connections 2023-10-10T19:16:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.756+0000 7fbd04bcd700 1 --2- 172.21.15.175:0/958226868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 0x7fbd00156580 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.756+0000 7fbd04bcd700 1 -- 172.21.15.175:0/958226868 >> 172.21.15.175:0/958226868 conn(0x7fbd000c56a0 msgr2=0x7fbd000c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:16:58.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.756+0000 7fbd04bcd700 1 -- 172.21.15.175:0/958226868 shutdown_connections 2023-10-10T19:16:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.756+0000 7fbd04bcd700 1 -- 172.21.15.175:0/958226868 wait complete. 2023-10-10T19:16:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.757+0000 7fbd04bcd700 1 Processor -- start 2023-10-10T19:16:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.757+0000 7fbd04bcd700 1 -- start start 2023-10-10T19:16:58.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbd04bcd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 0x7fbd0016a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbd04bcd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd000c35a0 0x7fbd001549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbd04bcd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd00156ac0 0x7fbd0015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbd04bcd700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd000bab50 con 0x7fbd00156ac0 2023-10-10T19:16:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbd04bcd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0016a150 con 0x7fbd000c2b80 2023-10-10T19:16:58.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbd04bcd700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd001560e0 con 0x7fbd000c35a0 2023-10-10T19:16:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd00156ac0 0x7fbd0015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd00156ac0 0x7fbd0015abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40018/0 (socket says 172.21.15.175:40018) 2023-10-10T19:16:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 -- 172.21.15.175:0/3908134483 learned_addr learned my addr 172.21.15.175:0/3908134483 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:16:58.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfdd9b700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd000c35a0 0x7fbd001549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfe59c700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 0x7fbd0016a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 -- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd000c35a0 msgr2=0x7fbd001549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd000c35a0 0x7fbd001549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 -- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 msgr2=0x7fbd0016a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:58.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 0x7fbd0016a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:58.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 -- 172.21.15.175:0/3908134483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf0009a20 con 0x7fbd00156ac0 2023-10-10T19:16:58.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfe59c700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 0x7fbd0016a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:16:58.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.758+0000 7fbcfed9d700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd00156ac0 0x7fbd0015abe0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800c810 tx=0x7fbcf8007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:16:58.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.759+0000 7fbcee7fc700 1 -- 172.21.15.175:0/3908134483 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8012630 con 0x7fbd00156ac0 2023-10-10T19:16:58.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.759+0000 7fbcee7fc700 1 -- 172.21.15.175:0/3908134483 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf80127b0 con 0x7fbd00156ac0 2023-10-10T19:16:58.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.759+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0006bed0 con 0x7fbd00156ac0 2023-10-10T19:16:58.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.759+0000 7fbcee7fc700 1 -- 172.21.15.175:0/3908134483 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8012b70 con 0x7fbd00156ac0 2023-10-10T19:16:58.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.759+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0006c490 con 0x7fbd00156ac0 2023-10-10T19:16:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.760+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0006a140 con 0x7fbd00156ac0 2023-10-10T19:16:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.760+0000 7fbcee7fc700 1 -- 172.21.15.175:0/3908134483 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbcf801b400 con 0x7fbd00156ac0 2023-10-10T19:16:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.760+0000 7fbcee7fc700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbce0066f30 0x7fbce00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.761+0000 7fbcee7fc700 1 -- 172.21.15.175:0/3908134483 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 7253+0+0 (secure 0 0 0) 0x7fbcf80846e0 con 0x7fbd00156ac0 2023-10-10T19:16:58.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.761+0000 7fbcfe59c700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbce0066f30 0x7fbce00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:58.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.761+0000 7fbcfe59c700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbce0066f30 0x7fbce00693e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbcf00099a0 tx=0x7fbcf0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:16:58.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.765+0000 7fbcee7fc700 1 -- 172.21.15.175:0/3908134483 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcf8051340 con 0x7fbd00156ac0 2023-10-10T19:16:58.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.996+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd0016a2c0 con 0x7fbd00156ac0 2023-10-10T19:16:58.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.996+0000 7fbcee7fc700 1 -- 172.21.15.175:0/3908134483 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+6232 (secure 0 0 0) 0x7fbcf8054960 con 0x7fbd00156ac0 2023-10-10T19:16:58.998 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:16:58.998 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:39.782385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:16:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.998+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbce0066f30 msgr2=0x7fbce00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.998+0000 7fbd04bcd700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbce0066f30 0x7fbce00693e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbcf00099a0 tx=0x7fbcf0017000 comp rx=0 tx=0).stop 2023-10-10T19:16:59.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.998+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd00156ac0 msgr2=0x7fbd0015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.998+0000 7fbd04bcd700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd00156ac0 0x7fbd0015abe0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800c810 tx=0x7fbcf8007e60 comp rx=0 tx=0).stop 2023-10-10T19:16:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.999+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 shutdown_connections 2023-10-10T19:16:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.999+0000 7fbd04bcd700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbce0066f30 0x7fbce00693e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.999+0000 7fbd04bcd700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd00156ac0 0x7fbd0015abe0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.999+0000 7fbd04bcd700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd000c2b80 0x7fbd0016a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.999+0000 7fbd04bcd700 1 --2- 172.21.15.175:0/3908134483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd000c35a0 0x7fbd001549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.999+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 >> 172.21.15.175:0/3908134483 conn(0x7fbd000c56a0 msgr2=0x7fbd0015c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:16:59.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.999+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 shutdown_connections 2023-10-10T19:16:59.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:58.999+0000 7fbd04bcd700 1 -- 172.21.15.175:0/3908134483 wait complete. 2023-10-10T19:16:59.004 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 243 2023-10-10T19:16:59.012 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=1) 2023-10-10T19:16:59.012 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-10T19:16:59.013 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:16:59.022 INFO:tasks.ceph.mds.b:Stopped 2023-10-10T19:16:59.023 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:16:59.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.136+0000 7fea2f59e700 1 Processor -- start 2023-10-10T19:16:59.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.136+0000 7fea2f59e700 1 -- start start 2023-10-10T19:16:59.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2f59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea3014fd90 0x7fea30159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:59.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2f59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea300c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:59.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2f59e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea30067fa0 con 0x7fea300cd040 2023-10-10T19:16:59.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2f59e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea300c5600 con 0x7fea3014fd90 2023-10-10T19:16:59.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2f59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea30156b70 con 0x7fea3008a100 2023-10-10T19:16:59.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2ed9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fea300cd040 0x7fea300684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51942/0 (socket says 172.21.15.175:51942) 2023-10-10T19:16:59.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2ed9d700 1 -- 172.21.15.175:0/43573578 learned_addr learned my addr 172.21.15.175:0/43573578 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:16:59.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2e59c700 1 --2- 172.21.15.175:0/43573578 >> v2:172.21.15.191:3300/0 conn(0x7fea3014fd90 0x7fea30159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:59.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2dd9b700 1 --2- 172.21.15.175:0/43573578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea300c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:59.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2d59a700 1 -- 172.21.15.175:0/43573578 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185480994 0 0) 0x7fea30067fa0 con 0x7fea300cd040 2023-10-10T19:16:59.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.137+0000 7fea2d59a700 1 -- 172.21.15.175:0/43573578 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea18003570 con 0x7fea300cd040 2023-10-10T19:16:59.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2dd9b700 1 -- 172.21.15.175:0/43573578 >> v2:172.21.15.191:3300/0 conn(0x7fea3014fd90 msgr2=0x7fea30159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:16:59.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2dd9b700 1 --2- 172.21.15.175:0/43573578 >> v2:172.21.15.191:3300/0 conn(0x7fea3014fd90 0x7fea30159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2dd9b700 1 -- 172.21.15.175:0/43573578 >> v1:172.21.15.175:6789/0 conn(0x7fea300cd040 legacy=0x7fea300684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:59.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2d59a700 1 -- 172.21.15.175:0/43573578 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 904798191 0 0) 0x7fea18003570 con 0x7fea300cd040 2023-10-10T19:16:59.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2dd9b700 1 -- 172.21.15.175:0/43573578 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30150160 con 0x7fea3008a100 2023-10-10T19:16:59.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2e59c700 1 --2- 172.21.15.175:0/43573578 >> v2:172.21.15.191:3300/0 conn(0x7fea3014fd90 0x7fea30159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:16:59.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2dd9b700 1 --2- 172.21.15.175:0/43573578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea300c5eb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fea240098e0 tx=0x7fea240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=760b0d19c6e9a47 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:16:59.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2d59a700 1 -- 172.21.15.175:0/43573578 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28001470 con 0x7fea3008a100 2023-10-10T19:16:59.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2d59a700 1 -- 172.21.15.175:0/43573578 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea24011420 con 0x7fea3008a100 2023-10-10T19:16:59.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2d59a700 1 -- 172.21.15.175:0/43573578 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea240117a0 con 0x7fea3008a100 2023-10-10T19:16:59.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2f59e700 1 -- 172.21.15.175:0/43573578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 msgr2=0x7fea300c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:59.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2f59e700 1 --2- 172.21.15.175:0/43573578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea300c5eb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fea240098e0 tx=0x7fea240091a0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2f59e700 1 -- 172.21.15.175:0/43573578 shutdown_connections 2023-10-10T19:16:59.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2f59e700 1 --2- 172.21.15.175:0/43573578 >> v2:172.21.15.191:3300/0 conn(0x7fea3014fd90 0x7fea30159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2f59e700 1 --2- 172.21.15.175:0/43573578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea300c5eb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2f59e700 1 -- 172.21.15.175:0/43573578 >> 172.21.15.175:0/43573578 conn(0x7fea30063d00 msgr2=0x7fea300cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:16:59.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2f59e700 1 -- 172.21.15.175:0/43573578 shutdown_connections 2023-10-10T19:16:59.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.138+0000 7fea2f59e700 1 -- 172.21.15.175:0/43573578 wait complete. 2023-10-10T19:16:59.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.139+0000 7fea2f59e700 1 Processor -- start 2023-10-10T19:16:59.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2f59e700 1 -- start start 2023-10-10T19:16:59.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2f59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea301515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:59.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2f59e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea300cd040 0x7fea30151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:59.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2f59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea3014fd90 0x7fea300c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:59.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2f59e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30152e30 con 0x7fea3014fd90 2023-10-10T19:16:59.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2f59e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea301506a0 con 0x7fea300cd040 2023-10-10T19:16:59.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2f59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea30151150 con 0x7fea3008a100 2023-10-10T19:16:59.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea3014fd90 0x7fea300c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:59.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea3014fd90 0x7fea300c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40022/0 (socket says 172.21.15.175:40022) 2023-10-10T19:16:59.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2ed9d700 1 -- 172.21.15.175:0/3481095096 learned_addr learned my addr 172.21.15.175:0/3481095096 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:16:59.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2e59c700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea301515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:59.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.140+0000 7fea2dd9b700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea300cd040 0x7fea30151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:59.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea2ed9d700 1 -- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 msgr2=0x7fea301515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:59.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea2ed9d700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea301515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea2ed9d700 1 -- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea300cd040 msgr2=0x7fea30151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:59.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea2ed9d700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea300cd040 0x7fea30151b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea2ed9d700 1 -- 172.21.15.175:0/3481095096 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea20009a20 con 0x7fea3014fd90 2023-10-10T19:16:59.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea2ed9d700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea3014fd90 0x7fea300c9360 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fea2800cb90 tx=0x7fea28001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:16:59.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea1e7fc700 1 -- 172.21.15.175:0/3481095096 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28001d80 con 0x7fea3014fd90 2023-10-10T19:16:59.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea1e7fc700 1 -- 172.21.15.175:0/3481095096 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28007360 con 0x7fea3014fd90 2023-10-10T19:16:59.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea24009420 con 0x7fea3014fd90 2023-10-10T19:16:59.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea1e7fc700 1 -- 172.21.15.175:0/3481095096 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28004830 con 0x7fea3014fd90 2023-10-10T19:16:59.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.141+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea301668c0 con 0x7fea3014fd90 2023-10-10T19:16:59.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.142+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea30067570 con 0x7fea3014fd90 2023-10-10T19:16:59.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.142+0000 7fea1e7fc700 1 -- 172.21.15.175:0/3481095096 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fea28004ba0 con 0x7fea3014fd90 2023-10-10T19:16:59.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.142+0000 7fea1e7fc700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea10066f30 0x7fea100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:16:59.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.143+0000 7fea1e7fc700 1 -- 172.21.15.175:0/3481095096 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 7253+0+0 (secure 0 0 0) 0x7fea28084370 con 0x7fea3014fd90 2023-10-10T19:16:59.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.143+0000 7fea2e59c700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea10066f30 0x7fea100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:16:59.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.143+0000 7fea2e59c700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea10066f30 0x7fea100693e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fea200099f0 tx=0x7fea2000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:16:59.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.147+0000 7fea1e7fc700 1 -- 172.21.15.175:0/3481095096 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea28051050 con 0x7fea3014fd90 2023-10-10T19:16:59.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.369+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea30155f80 con 0x7fea3014fd90 2023-10-10T19:16:59.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.370+0000 7fea1e7fc700 1 -- 172.21.15.175:0/3481095096 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+6232 (secure 0 0 0) 0x7fea28054670 con 0x7fea3014fd90 2023-10-10T19:16:59.371 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:16:59.372 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:39.782385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:16:59.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea10066f30 msgr2=0x7fea100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:59.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea10066f30 0x7fea100693e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fea200099f0 tx=0x7fea2000b040 comp rx=0 tx=0).stop 2023-10-10T19:16:59.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea3014fd90 msgr2=0x7fea300c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:16:59.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea3014fd90 0x7fea300c9360 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fea2800cb90 tx=0x7fea28001a10 comp rx=0 tx=0).stop 2023-10-10T19:16:59.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 shutdown_connections 2023-10-10T19:16:59.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea10066f30 0x7fea100693e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea3014fd90 0x7fea300c9360 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea3008a100 0x7fea301515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 --2- 172.21.15.175:0/3481095096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea300cd040 0x7fea30151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:16:59.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 >> 172.21.15.175:0/3481095096 conn(0x7fea30063d00 msgr2=0x7fea30153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:16:59.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 shutdown_connections 2023-10-10T19:16:59.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:16:59.372+0000 7fea2f59e700 1 -- 172.21.15.175:0/3481095096 wait complete. 2023-10-10T19:16:59.377 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 243 2023-10-10T19:16:59.385 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:17:01.386 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:01.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.513+0000 7fbebee71700 1 Processor -- start 2023-10-10T19:17:01.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.513+0000 7fbebee71700 1 -- start start 2023-10-10T19:17:01.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.513+0000 7fbebee71700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbeb814c840 0x7fbeb806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:01.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.513+0000 7fbebee71700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb80c52a0 0x7fbeb8152970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:01.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.513+0000 7fbebee71700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb8155ca0 con 0x7fbeb814cc10 2023-10-10T19:17:01.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.513+0000 7fbebee71700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb8156b00 con 0x7fbeb814c840 2023-10-10T19:17:01.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.513+0000 7fbebee71700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb806d700 con 0x7fbeb80c52a0 2023-10-10T19:17:01.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebd40e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb80c52a0 0x7fbeb8152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:01.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebcc0d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbeb814c840 0x7fbeb806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:01.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbeaffff700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fbeb814cc10 0x7fbeb80c4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:55014/0 (socket says 172.21.15.175:55014) 2023-10-10T19:17:01.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebd40e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb80c52a0 0x7fbeb8152970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:55770/0 (socket says 172.21.15.175:55770) 2023-10-10T19:17:01.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbeaffff700 1 -- 172.21.15.175:0/504350665 learned_addr learned my addr 172.21.15.175:0/504350665 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:01.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebd40e700 1 -- 172.21.15.175:0/504350665 >> v2:172.21.15.191:3300/0 conn(0x7fbeb814c840 msgr2=0x7fbeb806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:01.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebd40e700 1 --2- 172.21.15.175:0/504350665 >> v2:172.21.15.191:3300/0 conn(0x7fbeb814c840 0x7fbeb806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebd40e700 1 -- 172.21.15.175:0/504350665 >> v1:172.21.15.191:6789/0 conn(0x7fbeb814cc10 legacy=0x7fbeb80c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:01.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebd40e700 1 -- 172.21.15.175:0/504350665 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb8155ef0 con 0x7fbeb80c52a0 2023-10-10T19:17:01.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebcc0d700 1 --2- 172.21.15.175:0/504350665 >> v2:172.21.15.191:3300/0 conn(0x7fbeb814c840 0x7fbeb806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:17:01.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbebd40e700 1 --2- 172.21.15.175:0/504350665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb80c52a0 0x7fbeb8152970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000aa90 tx=0x7fbeb000a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ad02d28074a4bc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:01.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbeaf7fe700 1 -- 172.21.15.175:0/504350665 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb000ca00 con 0x7fbeb80c52a0 2023-10-10T19:17:01.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbeaf7fe700 1 -- 172.21.15.175:0/504350665 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb000cb60 con 0x7fbeb80c52a0 2023-10-10T19:17:01.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.514+0000 7fbeaf7fe700 1 -- 172.21.15.175:0/504350665 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0018400 con 0x7fbeb80c52a0 2023-10-10T19:17:01.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.515+0000 7fbebee71700 1 -- 172.21.15.175:0/504350665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb80c52a0 msgr2=0x7fbeb8152970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:01.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.515+0000 7fbebee71700 1 --2- 172.21.15.175:0/504350665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb80c52a0 0x7fbeb8152970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000aa90 tx=0x7fbeb000a330 comp rx=0 tx=0).stop 2023-10-10T19:17:01.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.515+0000 7fbebee71700 1 -- 172.21.15.175:0/504350665 shutdown_connections 2023-10-10T19:17:01.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.515+0000 7fbebee71700 1 --2- 172.21.15.175:0/504350665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb80c52a0 0x7fbeb8152970 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.515+0000 7fbebee71700 1 --2- 172.21.15.175:0/504350665 >> v2:172.21.15.191:3300/0 conn(0x7fbeb814c840 0x7fbeb806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.515+0000 7fbebee71700 1 -- 172.21.15.175:0/504350665 >> 172.21.15.175:0/504350665 conn(0x7fbeb8068280 msgr2=0x7fbeb806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:01.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.515+0000 7fbebee71700 1 -- 172.21.15.175:0/504350665 shutdown_connections 2023-10-10T19:17:01.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.515+0000 7fbebee71700 1 -- 172.21.15.175:0/504350665 wait complete. 2023-10-10T19:17:01.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.516+0000 7fbebee71700 1 Processor -- start 2023-10-10T19:17:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.516+0000 7fbebee71700 1 -- start start 2023-10-10T19:17:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebee71700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbeb80c52a0 0x7fbeb8144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebee71700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb814c840 0x7fbeb8143550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebee71700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeb814cc10 0x7fbeb8148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:01.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebee71700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8145910 con 0x7fbeb80c52a0 2023-10-10T19:17:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebee71700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8069d50 con 0x7fbeb814cc10 2023-10-10T19:17:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebee71700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8069ff0 con 0x7fbeb814c840 2023-10-10T19:17:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebcc0d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbeb80c52a0 0x7fbeb8144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:01.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebcc0d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbeb80c52a0 0x7fbeb8144950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40034/0 (socket says 172.21.15.175:40034) 2023-10-10T19:17:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebcc0d700 1 -- 172.21.15.175:0/939238513 learned_addr learned my addr 172.21.15.175:0/939238513 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebd40e700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeb814cc10 0x7fbeb8148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbeaffff700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb814c840 0x7fbeb8143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebcc0d700 1 -- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb814c840 msgr2=0x7fbeb8143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebcc0d700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb814c840 0x7fbeb8143550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebcc0d700 1 -- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeb814cc10 msgr2=0x7fbeb8148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebcc0d700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeb814cc10 0x7fbeb8148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.517+0000 7fbebcc0d700 1 -- 172.21.15.175:0/939238513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb4009a20 con 0x7fbeb80c52a0 2023-10-10T19:17:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.518+0000 7fbebcc0d700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbeb80c52a0 0x7fbeb8144950 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400c1f0 tx=0x7fbeb400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.518+0000 7fbeacff9700 1 -- 172.21.15.175:0/939238513 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb400ce30 con 0x7fbeb80c52a0 2023-10-10T19:17:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.518+0000 7fbeacff9700 1 -- 172.21.15.175:0/939238513 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb400f040 con 0x7fbeb80c52a0 2023-10-10T19:17:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.518+0000 7fbeacff9700 1 -- 172.21.15.175:0/939238513 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4014690 con 0x7fbeb80c52a0 2023-10-10T19:17:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.518+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb000a5d0 con 0x7fbeb80c52a0 2023-10-10T19:17:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.518+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb81452c0 con 0x7fbeb80c52a0 2023-10-10T19:17:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.519+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb8067570 con 0x7fbeb80c52a0 2023-10-10T19:17:01.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.519+0000 7fbeacff9700 1 -- 172.21.15.175:0/939238513 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbeb4014a00 con 0x7fbeb80c52a0 2023-10-10T19:17:01.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.520+0000 7fbeacff9700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbea4066f60 0x7fbea4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:01.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.520+0000 7fbeacff9700 1 -- 172.21.15.175:0/939238513 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 7253+0+0 (secure 0 0 0) 0x7fbeb4085b00 con 0x7fbeb80c52a0 2023-10-10T19:17:01.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.520+0000 7fbeaffff700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbea4066f60 0x7fbea4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:01.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.520+0000 7fbeaffff700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbea4066f60 0x7fbea4069410 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8144e90 tx=0x7fbea8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:01.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.524+0000 7fbeacff9700 1 -- 172.21.15.175:0/939238513 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbeb40561f0 con 0x7fbeb80c52a0 2023-10-10T19:17:01.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.749+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbeb806cf40 con 0x7fbeb80c52a0 2023-10-10T19:17:01.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.750+0000 7fbeacff9700 1 -- 172.21.15.175:0/939238513 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+6232 (secure 0 0 0) 0x7fbeb4014cb0 con 0x7fbeb80c52a0 2023-10-10T19:17:01.752 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:01.752 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:39.782385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:01.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbea4066f60 msgr2=0x7fbea4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:01.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbea4066f60 0x7fbea4069410 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8144e90 tx=0x7fbea8009040 comp rx=0 tx=0).stop 2023-10-10T19:17:01.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbeb80c52a0 msgr2=0x7fbeb8144950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:01.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbeb80c52a0 0x7fbeb8144950 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400c1f0 tx=0x7fbeb400b040 comp rx=0 tx=0).stop 2023-10-10T19:17:01.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 shutdown_connections 2023-10-10T19:17:01.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbea4066f60 0x7fbea4069410 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeb814cc10 0x7fbeb8148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbeb80c52a0 0x7fbeb8144950 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 --2- 172.21.15.175:0/939238513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbeb814c840 0x7fbeb8143550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:01.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 >> 172.21.15.175:0/939238513 conn(0x7fbeb8068280 msgr2=0x7fbeb814ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:01.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 shutdown_connections 2023-10-10T19:17:01.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:01.752+0000 7fbebee71700 1 -- 172.21.15.175:0/939238513 wait complete. 2023-10-10T19:17:01.757 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 243 2023-10-10T19:17:01.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:17:03.767 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:03.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.897+0000 7f8d94c46700 1 Processor -- start 2023-10-10T19:17:03.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.897+0000 7f8d94c46700 1 -- start start 2023-10-10T19:17:03.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d94c46700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8d900b1fc0 0x7f8d90155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:03.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d94c46700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8d900b2390 0x7f8d90068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:03.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d94c46700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d901574b0 con 0x7f8d90068bd0 2023-10-10T19:17:03.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d94c46700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d90150770 con 0x7f8d900b1fc0 2023-10-10T19:17:03.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d94c46700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d900afad0 con 0x7f8d900b2390 2023-10-10T19:17:03.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d8e59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8d900b1fc0 0x7f8d90155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:03.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d8ed9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8d90068bd0 0x7f8d900aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51950/0 (socket says 172.21.15.175:51950) 2023-10-10T19:17:03.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d8ed9d700 1 -- 172.21.15.175:0/2638177582 learned_addr learned my addr 172.21.15.175:0/2638177582 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:03.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d85d9b700 1 --2- 172.21.15.175:0/2638177582 >> v2:172.21.15.191:3300/0 conn(0x7f8d900b2390 0x7f8d90068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:03.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d8e59c700 1 -- 172.21.15.175:0/2638177582 >> v2:172.21.15.191:3300/0 conn(0x7f8d900b2390 msgr2=0x7f8d90068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:03.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d8e59c700 1 --2- 172.21.15.175:0/2638177582 >> v2:172.21.15.191:3300/0 conn(0x7f8d900b2390 0x7f8d90068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:03.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d8e59c700 1 -- 172.21.15.175:0/2638177582 >> v1:172.21.15.175:6789/0 conn(0x7f8d90068bd0 legacy=0x7f8d900aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:03.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d8e59c700 1 -- 172.21.15.175:0/2638177582 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900af4c0 con 0x7f8d900b1fc0 2023-10-10T19:17:03.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.898+0000 7f8d8dd9b700 1 -- 172.21.15.175:0/2638177582 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586074973 0 0) 0x7f8d901574b0 con 0x7f8d90068bd0 2023-10-10T19:17:03.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d8e59c700 1 --2- 172.21.15.175:0/2638177582 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d90155fb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f8d900c0290 tx=0x7f8d800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d7b3a7f34a528c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:03.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d8dd9b700 1 -- 172.21.15.175:0/2638177582 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8000b690 con 0x7f8d900b1fc0 2023-10-10T19:17:03.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d8dd9b700 1 -- 172.21.15.175:0/2638177582 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8000b7f0 con 0x7f8d900b1fc0 2023-10-10T19:17:03.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d8dd9b700 1 -- 172.21.15.175:0/2638177582 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8000bb90 con 0x7f8d900b1fc0 2023-10-10T19:17:03.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d94c46700 1 -- 172.21.15.175:0/2638177582 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 msgr2=0x7f8d90155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:03.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d94c46700 1 --2- 172.21.15.175:0/2638177582 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d90155fb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f8d900c0290 tx=0x7f8d800091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:03.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d94c46700 1 -- 172.21.15.175:0/2638177582 shutdown_connections 2023-10-10T19:17:03.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d94c46700 1 --2- 172.21.15.175:0/2638177582 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d90155fb0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:03.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d94c46700 1 --2- 172.21.15.175:0/2638177582 >> v2:172.21.15.191:3300/0 conn(0x7f8d900b2390 0x7f8d90068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:03.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d94c46700 1 -- 172.21.15.175:0/2638177582 >> 172.21.15.175:0/2638177582 conn(0x7f8d90063230 msgr2=0x7f8d90063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:03.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d94c46700 1 -- 172.21.15.175:0/2638177582 shutdown_connections 2023-10-10T19:17:03.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.899+0000 7f8d94c46700 1 -- 172.21.15.175:0/2638177582 wait complete. 2023-10-10T19:17:03.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d94c46700 1 Processor -- start 2023-10-10T19:17:03.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d94c46700 1 -- start start 2023-10-10T19:17:03.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d94c46700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d90068bd0 0x7f8d90159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:03.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d94c46700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d901572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:03.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d94c46700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d900b2390 0x7f8d90157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:03.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d94c46700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90159b60 con 0x7f8d900b1fc0 2023-10-10T19:17:03.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d94c46700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900cc830 con 0x7f8d900b2390 2023-10-10T19:17:03.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d94c46700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900ccfe0 con 0x7f8d90068bd0 2023-10-10T19:17:03.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d85d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d901572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:03.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d85d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d901572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40058/0 (socket says 172.21.15.175:40058) 2023-10-10T19:17:03.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.901+0000 7f8d85d9b700 1 -- 172.21.15.175:0/3790606157 learned_addr learned my addr 172.21.15.175:0/3790606157 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:03.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d8ed9d700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d900b2390 0x7f8d90157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:03.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d8e59c700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d90068bd0 0x7f8d90159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:03.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d85d9b700 1 -- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d90068bd0 msgr2=0x7f8d90159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:03.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d85d9b700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d90068bd0 0x7f8d90159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:03.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d85d9b700 1 -- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d900b2390 msgr2=0x7f8d90157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:03.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d85d9b700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d900b2390 0x7f8d90157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:03.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d85d9b700 1 -- 172.21.15.175:0/3790606157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d80009420 con 0x7f8d900b1fc0 2023-10-10T19:17:03.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d8e59c700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d90068bd0 0x7f8d90159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:03.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d85d9b700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d901572f0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800e930 tx=0x7f8d780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:03.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d86ffd700 1 -- 172.21.15.175:0/3790606157 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78013680 con 0x7f8d900b1fc0 2023-10-10T19:17:03.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d86ffd700 1 -- 172.21.15.175:0/3790606157 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d780137e0 con 0x7f8d900b1fc0 2023-10-10T19:17:03.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d86ffd700 1 -- 172.21.15.175:0/3790606157 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78013b80 con 0x7f8d900b1fc0 2023-10-10T19:17:03.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9015a2d0 con 0x7f8d900b1fc0 2023-10-10T19:17:03.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.902+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d900ccd00 con 0x7f8d900b1fc0 2023-10-10T19:17:03.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.903+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d90067570 con 0x7f8d900b1fc0 2023-10-10T19:17:03.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.903+0000 7f8d86ffd700 1 -- 172.21.15.175:0/3790606157 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8d7800c460 con 0x7f8d900b1fc0 2023-10-10T19:17:03.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.904+0000 7f8d86ffd700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8d70066f60 0x7f8d70069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:03.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.904+0000 7f8d86ffd700 1 -- 172.21.15.175:0/3790606157 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f8d78084c90 con 0x7f8d900b1fc0 2023-10-10T19:17:03.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.904+0000 7f8d8e59c700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8d70066f60 0x7f8d70069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:03.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.904+0000 7f8d8e59c700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8d70066f60 0x7f8d70069410 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8d800093f0 tx=0x7f8d80017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:03.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:03.908+0000 7f8d86ffd700 1 -- 172.21.15.175:0/3790606157 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d78051910 con 0x7f8d900b1fc0 2023-10-10T19:17:04.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.130+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d90069060 con 0x7f8d900b1fc0 2023-10-10T19:17:04.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.131+0000 7f8d86ffd700 1 -- 172.21.15.175:0/3790606157 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+6232 (secure 0 0 0) 0x7f8d78054f30 con 0x7f8d900b1fc0 2023-10-10T19:17:04.133 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:04.133 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:39.782385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8d70066f60 msgr2=0x7f8d70069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8d70066f60 0x7f8d70069410 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8d800093f0 tx=0x7f8d80017000 comp rx=0 tx=0).stop 2023-10-10T19:17:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 msgr2=0x7f8d901572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:04.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d901572f0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800e930 tx=0x7f8d780091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 shutdown_connections 2023-10-10T19:17:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8d70066f60 0x7f8d70069410 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d900b2390 0x7f8d90157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:04.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d90068bd0 0x7f8d90159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:04.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 --2- 172.21.15.175:0/3790606157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d900b1fc0 0x7f8d901572f0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:04.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 >> 172.21.15.175:0/3790606157 conn(0x7f8d90063230 msgr2=0x7f8d9014fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:04.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 shutdown_connections 2023-10-10T19:17:04.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:04.133+0000 7f8d94c46700 1 -- 172.21.15.175:0/3790606157 wait complete. 2023-10-10T19:17:04.138 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 243 2023-10-10T19:17:04.147 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:17:06.147 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:06.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.274+0000 7f05b5ec0700 1 Processor -- start 2023-10-10T19:17:06.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.274+0000 7f05b5ec0700 1 -- start start 2023-10-10T19:17:06.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.275+0000 7f05b5ec0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c12d0 0x7f05b00c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:06.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.275+0000 7f05b5ec0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f05b00c4f20 0x7f05b00c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:06.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.275+0000 7f05b5ec0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b00c6fb0 con 0x7f05b00c5850 2023-10-10T19:17:06.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.275+0000 7f05b5ec0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b0063ea0 con 0x7f05b00c4f20 2023-10-10T19:17:06.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.275+0000 7f05b5ec0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0155f30 con 0x7f05b00c12d0 2023-10-10T19:17:06.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.275+0000 7f05aeffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c12d0 0x7f05b00c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:06.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05affff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f05b00c4f20 0x7f05b00c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:06.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05af7fe700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f05b00c5850 0x7f05b0156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:47324/0 (socket says 172.21.15.175:47324) 2023-10-10T19:17:06.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05aeffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c12d0 0x7f05b00c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:50204/0 (socket says 172.21.15.175:50204) 2023-10-10T19:17:06.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05af7fe700 1 -- 172.21.15.175:0/381569842 learned_addr learned my addr 172.21.15.175:0/381569842 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:06.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05affff700 1 -- 172.21.15.175:0/381569842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c12d0 msgr2=0x7f05b00c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:06.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05affff700 1 --2- 172.21.15.175:0/381569842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c12d0 0x7f05b00c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05affff700 1 -- 172.21.15.175:0/381569842 >> v1:172.21.15.191:6789/0 conn(0x7f05b00c5850 legacy=0x7f05b0156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:06.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05affff700 1 -- 172.21.15.175:0/381569842 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b00c6160 con 0x7f05b00c4f20 2023-10-10T19:17:06.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05aeffd700 1 --2- 172.21.15.175:0/381569842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c12d0 0x7f05b00c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:17:06.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05ae7fc700 1 -- 172.21.15.175:0/381569842 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479817000 0 0) 0x7f05b00c6fb0 con 0x7f05b00c5850 2023-10-10T19:17:06.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05affff700 1 --2- 172.21.15.175:0/381569842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c4f20 0x7f05b00c7370 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f05a80098e0 tx=0x7f05a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef6ab270ce2b2f62 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:06.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05ae7fc700 1 -- 172.21.15.175:0/381569842 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a800b6d0 con 0x7f05b00c4f20 2023-10-10T19:17:06.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05ae7fc700 1 -- 172.21.15.175:0/381569842 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a800b830 con 0x7f05b00c4f20 2023-10-10T19:17:06.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.276+0000 7f05ae7fc700 1 -- 172.21.15.175:0/381569842 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a800bbd0 con 0x7f05b00c4f20 2023-10-10T19:17:06.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.277+0000 7f05b5ec0700 1 -- 172.21.15.175:0/381569842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c4f20 msgr2=0x7f05b00c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:06.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.277+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/381569842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c4f20 0x7f05b00c7370 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f05a80098e0 tx=0x7f05a80091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.277+0000 7f05b5ec0700 1 -- 172.21.15.175:0/381569842 shutdown_connections 2023-10-10T19:17:06.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.277+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/381569842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c4f20 0x7f05b00c7370 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.277+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/381569842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c12d0 0x7f05b00c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.277+0000 7f05b5ec0700 1 -- 172.21.15.175:0/381569842 >> 172.21.15.175:0/381569842 conn(0x7f05b006dc70 msgr2=0x7f05b0063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:06.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.277+0000 7f05b5ec0700 1 -- 172.21.15.175:0/381569842 shutdown_connections 2023-10-10T19:17:06.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.277+0000 7f05b5ec0700 1 -- 172.21.15.175:0/381569842 wait complete. 2023-10-10T19:17:06.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.278+0000 7f05b5ec0700 1 Processor -- start 2023-10-10T19:17:06.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.278+0000 7f05b5ec0700 1 -- start start 2023-10-10T19:17:06.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05b5ec0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c12d0 0x7f05b0166160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:06.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05b5ec0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05b00c4f20 0x7f05b014eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:06.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05b5ec0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c5850 0x7f05b0154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:06.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05b5ec0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0150e40 con 0x7f05b00c4f20 2023-10-10T19:17:06.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05b5ec0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0150880 con 0x7f05b00c12d0 2023-10-10T19:17:06.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05b5ec0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0150b60 con 0x7f05b00c5850 2023-10-10T19:17:06.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05b00c4f20 0x7f05b014eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:06.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05b00c4f20 0x7f05b014eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43192/0 (socket says 172.21.15.175:43192) 2023-10-10T19:17:06.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 -- 172.21.15.175:0/150835156 learned_addr learned my addr 172.21.15.175:0/150835156 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:06.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05af7fe700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c12d0 0x7f05b0166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:06.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05affff700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c5850 0x7f05b0154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:06.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 -- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c5850 msgr2=0x7f05b0154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:06.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c5850 0x7f05b0154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 -- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c12d0 msgr2=0x7f05b0166160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:06.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c12d0 0x7f05b0166160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 -- 172.21.15.175:0/150835156 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a4009420 con 0x7f05b00c4f20 2023-10-10T19:17:06.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05aeffd700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05b00c4f20 0x7f05b014eeb0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f05a400ec80 tx=0x7f05a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:06.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05977fe700 1 -- 172.21.15.175:0/150835156 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a400ca60 con 0x7f05b00c4f20 2023-10-10T19:17:06.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05977fe700 1 -- 172.21.15.175:0/150835156 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a400cbc0 con 0x7f05b00c4f20 2023-10-10T19:17:06.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.279+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a8009420 con 0x7f05b00c4f20 2023-10-10T19:17:06.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.280+0000 7f05977fe700 1 -- 172.21.15.175:0/150835156 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4013470 con 0x7f05b00c4f20 2023-10-10T19:17:06.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.280+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b0165960 con 0x7f05b00c4f20 2023-10-10T19:17:06.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.280+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b0067570 con 0x7f05b00c4f20 2023-10-10T19:17:06.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.281+0000 7f05977fe700 1 -- 172.21.15.175:0/150835156 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f05a4013820 con 0x7f05b00c4f20 2023-10-10T19:17:06.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.281+0000 7f05977fe700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f059c066f30 0x7f059c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:06.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.281+0000 7f05977fe700 1 -- 172.21.15.175:0/150835156 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f05a4084f00 con 0x7f05b00c4f20 2023-10-10T19:17:06.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.281+0000 7f05af7fe700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f059c066f30 0x7f059c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:06.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.282+0000 7f05af7fe700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f059c066f30 0x7f059c0693e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f05a0001560 tx=0x7f05a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:06.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.285+0000 7f05977fe700 1 -- 172.21.15.175:0/150835156 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05a4051b80 con 0x7f05b00c4f20 2023-10-10T19:17:06.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.509+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05b00bf640 con 0x7f05b00c4f20 2023-10-10T19:17:06.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.510+0000 7f05977fe700 1 -- 172.21.15.175:0/150835156 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+6232 (secure 0 0 0) 0x7f05a40551a0 con 0x7f05b00c4f20 2023-10-10T19:17:06.512 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:06.512 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:39.782385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:06.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f059c066f30 msgr2=0x7f059c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:06.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f059c066f30 0x7f059c0693e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f05a0001560 tx=0x7f05a0009040 comp rx=0 tx=0).stop 2023-10-10T19:17:06.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05b00c4f20 msgr2=0x7f05b014eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:06.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05b00c4f20 0x7f05b014eeb0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f05a400ec80 tx=0x7f05a40091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 shutdown_connections 2023-10-10T19:17:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f059c066f30 0x7f059c0693e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f05b00c5850 0x7f05b0154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05b00c12d0 0x7f05b0166160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 --2- 172.21.15.175:0/150835156 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05b00c4f20 0x7f05b014eeb0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:06.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 >> 172.21.15.175:0/150835156 conn(0x7f05b006dc70 msgr2=0x7f05b0152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:06.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 shutdown_connections 2023-10-10T19:17:06.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:06.512+0000 7f05b5ec0700 1 -- 172.21.15.175:0/150835156 wait complete. 2023-10-10T19:17:06.516 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 243 2023-10-10T19:17:06.525 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:17:08.527 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:08.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.653+0000 7f0299353700 1 Processor -- start 2023-10-10T19:17:08.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.653+0000 7f0299353700 1 -- start start 2023-10-10T19:17:08.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.654+0000 7f0299353700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02940c5c50 0x7f0294155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.654+0000 7f0299353700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0294068280 con 0x7f029406dd70 2023-10-10T19:17:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.654+0000 7f0299353700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0294159b70 con 0x7f0294063230 2023-10-10T19:17:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.654+0000 7f0299353700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029415a0d0 con 0x7f02940c5c50 2023-10-10T19:17:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.654+0000 7f0292ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f029406dd70 0x7f0294068710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35334/0 (socket says 172.21.15.175:35334) 2023-10-10T19:17:08.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.654+0000 7f0292ffd700 1 -- 172.21.15.175:0/4246556817 learned_addr learned my addr 172.21.15.175:0/4246556817 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.654+0000 7f02927fc700 1 --2- 172.21.15.175:0/4246556817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02940c5c50 0x7f0294155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f0291ffb700 1 -- 172.21.15.175:0/4246556817 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089667161 0 0) 0x7f0294068280 con 0x7f029406dd70 2023-10-10T19:17:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f0291ffb700 1 -- 172.21.15.175:0/4246556817 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0280003570 con 0x7f029406dd70 2023-10-10T19:17:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f0291ffb700 1 -- 172.21.15.175:0/4246556817 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819246967 0 0) 0x7f0294159b70 con 0x7f0294063230 2023-10-10T19:17:08.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f0291ffb700 1 -- 172.21.15.175:0/4246556817 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294068280 con 0x7f0294063230 2023-10-10T19:17:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f02927fc700 1 -- 172.21.15.175:0/4246556817 >> v1:172.21.15.191:6789/0 conn(0x7f0294063230 legacy=0x7f0294156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f02927fc700 1 -- 172.21.15.175:0/4246556817 >> v1:172.21.15.175:6789/0 conn(0x7f029406dd70 legacy=0x7f0294068710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f02927fc700 1 -- 172.21.15.175:0/4246556817 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294067990 con 0x7f02940c5c50 2023-10-10T19:17:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f02927fc700 1 --2- 172.21.15.175:0/4246556817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02940c5c50 0x7f0294155c30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f027c009910 tx=0x7f027c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbd18c0b053eea06 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f0291ffb700 1 -- 172.21.15.175:0/4246556817 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c00b690 con 0x7f02940c5c50 2023-10-10T19:17:08.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f0291ffb700 1 -- 172.21.15.175:0/4246556817 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c00b7f0 con 0x7f02940c5c50 2023-10-10T19:17:08.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.655+0000 7f0291ffb700 1 -- 172.21.15.175:0/4246556817 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c00bb90 con 0x7f02940c5c50 2023-10-10T19:17:08.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.656+0000 7f0299353700 1 -- 172.21.15.175:0/4246556817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02940c5c50 msgr2=0x7f0294155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:08.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.656+0000 7f0299353700 1 --2- 172.21.15.175:0/4246556817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02940c5c50 0x7f0294155c30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f027c009910 tx=0x7f027c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.656+0000 7f0299353700 1 -- 172.21.15.175:0/4246556817 shutdown_connections 2023-10-10T19:17:08.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.656+0000 7f0299353700 1 --2- 172.21.15.175:0/4246556817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02940c5c50 0x7f0294155c30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.656+0000 7f0299353700 1 -- 172.21.15.175:0/4246556817 >> 172.21.15.175:0/4246556817 conn(0x7f02940c95a0 msgr2=0x7f02940c5540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:08.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.656+0000 7f0299353700 1 -- 172.21.15.175:0/4246556817 shutdown_connections 2023-10-10T19:17:08.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.656+0000 7f0299353700 1 -- 172.21.15.175:0/4246556817 wait complete. 2023-10-10T19:17:08.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.657+0000 7f0299353700 1 Processor -- start 2023-10-10T19:17:08.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.657+0000 7f0299353700 1 -- start start 2023-10-10T19:17:08.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f0299353700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0294063230 0x7f0294169b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:08.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f0299353700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f029406dd70 0x7f029414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:08.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f0299353700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02940c5c50 0x7f02940c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:08.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f0299353700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02940c2e50 con 0x7f02940c5c50 2023-10-10T19:17:08.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f0299353700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029414e7d0 con 0x7f029406dd70 2023-10-10T19:17:08.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f0299353700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02940c1d30 con 0x7f0294063230 2023-10-10T19:17:08.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02940c5c50 0x7f02940c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:08.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02940c5c50 0x7f02940c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43200/0 (socket says 172.21.15.175:43200) 2023-10-10T19:17:08.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 -- 172.21.15.175:0/1373450014 learned_addr learned my addr 172.21.15.175:0/1373450014 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:08.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f0292ffd700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0294063230 0x7f0294169b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:08.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02927fc700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f029406dd70 0x7f029414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:08.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 -- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0294063230 msgr2=0x7f0294169b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:08.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0294063230 0x7f0294169b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 -- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f029406dd70 msgr2=0x7f029414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:08.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f029406dd70 0x7f029414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 -- 172.21.15.175:0/1373450014 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c009420 con 0x7f02940c5c50 2023-10-10T19:17:08.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f02937fe700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02940c5c50 0x7f02940c7350 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f028400c5c0 tx=0x7f0284007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:08.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f028affd700 1 -- 172.21.15.175:0/1373450014 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284011590 con 0x7f02940c5c50 2023-10-10T19:17:08.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f028affd700 1 -- 172.21.15.175:0/1373450014 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0284011710 con 0x7f02940c5c50 2023-10-10T19:17:08.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.658+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0294156860 con 0x7f02940c5c50 2023-10-10T19:17:08.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.659+0000 7f028affd700 1 -- 172.21.15.175:0/1373450014 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284011ad0 con 0x7f02940c5c50 2023-10-10T19:17:08.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.659+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02940c2240 con 0x7f02940c5c50 2023-10-10T19:17:08.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.659+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0294157fd0 con 0x7f02940c5c50 2023-10-10T19:17:08.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.660+0000 7f028affd700 1 -- 172.21.15.175:0/1373450014 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f028401c460 con 0x7f02940c5c50 2023-10-10T19:17:08.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.660+0000 7f028affd700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0274066f60 0x7f0274069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:08.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.660+0000 7f028affd700 1 -- 172.21.15.175:0/1373450014 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f0284084730 con 0x7f02940c5c50 2023-10-10T19:17:08.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.660+0000 7f0292ffd700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0274066f60 0x7f0274069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:08.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.661+0000 7f0292ffd700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0274066f60 0x7f0274069410 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f028c005c50 tx=0x7f028c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:08.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.665+0000 7f028affd700 1 -- 172.21.15.175:0/1373450014 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0284051310 con 0x7f02940c5c50 2023-10-10T19:17:08.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.891+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0294067570 con 0x7f02940c5c50 2023-10-10T19:17:08.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.892+0000 7f028affd700 1 -- 172.21.15.175:0/1373450014 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+6232 (secure 0 0 0) 0x7f0284054930 con 0x7f02940c5c50 2023-10-10T19:17:08.893 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:08.893 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:39.782385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:08.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.893+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0274066f60 msgr2=0x7f0274069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:08.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.893+0000 7f0299353700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0274066f60 0x7f0274069410 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f028c005c50 tx=0x7f028c009040 comp rx=0 tx=0).stop 2023-10-10T19:17:08.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.893+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02940c5c50 msgr2=0x7f02940c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:08.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.893+0000 7f0299353700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02940c5c50 0x7f02940c7350 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f028400c5c0 tx=0x7f0284007cf0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.893+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 shutdown_connections 2023-10-10T19:17:08.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.894+0000 7f0299353700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0274066f60 0x7f0274069410 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.894+0000 7f0299353700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02940c5c50 0x7f02940c7350 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.894+0000 7f0299353700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0294063230 0x7f0294169b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.894+0000 7f0299353700 1 --2- 172.21.15.175:0/1373450014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f029406dd70 0x7f029414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:08.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.894+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 >> 172.21.15.175:0/1373450014 conn(0x7f02940c95a0 msgr2=0x7f02940c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:08.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.894+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 shutdown_connections 2023-10-10T19:17:08.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:08.894+0000 7f0299353700 1 -- 172.21.15.175:0/1373450014 wait complete. 2023-10-10T19:17:08.898 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 243 2023-10-10T19:17:08.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:17:10.908 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:11.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f097a40a700 1 Processor -- start 2023-10-10T19:17:11.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f097a40a700 1 -- start start 2023-10-10T19:17:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f097a40a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f09740c5e40 0x7f09740c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f097a40a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f097406bf00 0x7f09740c6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f097a40a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09740c6df0 con 0x7f09740c6750 2023-10-10T19:17:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f097a40a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09741560d0 con 0x7f09740c5e40 2023-10-10T19:17:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f097a40a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09741597e0 con 0x7f097406bf00 2023-10-10T19:17:11.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0973fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f09740c5e40 0x7f09740c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:11.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f09789a7700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f09740c6750 0x7f0974156520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35350/0 (socket says 172.21.15.175:35350) 2023-10-10T19:17:11.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f09789a7700 1 -- 172.21.15.175:0/4286664062 learned_addr learned my addr 172.21.15.175:0/4286664062 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:11.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0973fff700 1 --2- 172.21.15.175:0/4286664062 >> v2:172.21.15.175:3300/0 conn(0x7f09740c5e40 0x7f09740c47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43202/0 (socket says 172.21.15.175:43202) 2023-10-10T19:17:11.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f09737fe700 1 --2- 172.21.15.175:0/4286664062 >> v2:172.21.15.191:3300/0 conn(0x7f097406bf00 0x7f09740c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:11.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0973fff700 1 -- 172.21.15.175:0/4286664062 >> v2:172.21.15.191:3300/0 conn(0x7f097406bf00 msgr2=0x7f09740c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:11.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0973fff700 1 --2- 172.21.15.175:0/4286664062 >> v2:172.21.15.191:3300/0 conn(0x7f097406bf00 0x7f09740c6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0973fff700 1 -- 172.21.15.175:0/4286664062 >> v1:172.21.15.175:6789/0 conn(0x7f09740c6750 legacy=0x7f0974156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:11.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0973fff700 1 -- 172.21.15.175:0/4286664062 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0974159ac0 con 0x7f09740c5e40 2023-10-10T19:17:11.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0972ffd700 1 -- 172.21.15.175:0/4286664062 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767292769 0 0) 0x7f09740c6df0 con 0x7f09740c6750 2023-10-10T19:17:11.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0973fff700 1 --2- 172.21.15.175:0/4286664062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09740c5e40 0x7f09740c47d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f097406fb10 tx=0x7f0968009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69090e827f0cad35 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:11.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0972ffd700 1 -- 172.21.15.175:0/4286664062 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968005c10 con 0x7f09740c5e40 2023-10-10T19:17:11.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0972ffd700 1 -- 172.21.15.175:0/4286664062 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0968005d70 con 0x7f09740c5e40 2023-10-10T19:17:11.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.036+0000 7f0972ffd700 1 -- 172.21.15.175:0/4286664062 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968006260 con 0x7f09740c5e40 2023-10-10T19:17:11.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.037+0000 7f097a40a700 1 -- 172.21.15.175:0/4286664062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09740c5e40 msgr2=0x7f09740c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:11.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.037+0000 7f097a40a700 1 --2- 172.21.15.175:0/4286664062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09740c5e40 0x7f09740c47d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f097406fb10 tx=0x7f0968009120 comp rx=0 tx=0).stop 2023-10-10T19:17:11.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.037+0000 7f097a40a700 1 -- 172.21.15.175:0/4286664062 shutdown_connections 2023-10-10T19:17:11.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.037+0000 7f097a40a700 1 --2- 172.21.15.175:0/4286664062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09740c5e40 0x7f09740c47d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.037+0000 7f097a40a700 1 --2- 172.21.15.175:0/4286664062 >> v2:172.21.15.191:3300/0 conn(0x7f097406bf00 0x7f09740c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.037+0000 7f097a40a700 1 -- 172.21.15.175:0/4286664062 >> 172.21.15.175:0/4286664062 conn(0x7f0974068900 msgr2=0x7f0974068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:11.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.037+0000 7f097a40a700 1 -- 172.21.15.175:0/4286664062 shutdown_connections 2023-10-10T19:17:11.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.037+0000 7f097a40a700 1 -- 172.21.15.175:0/4286664062 wait complete. 2023-10-10T19:17:11.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.038+0000 7f097a40a700 1 Processor -- start 2023-10-10T19:17:11.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.038+0000 7f097a40a700 1 -- start start 2023-10-10T19:17:11.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f097a40a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f097406bf00 0x7f09740c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:11.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f097a40a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09740c5e40 0x7f09741662d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:11.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f097a40a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09741567e0 0x7f09741587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:11.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f097a40a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0974169800 con 0x7f097406bf00 2023-10-10T19:17:11.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f097a40a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0974155f20 con 0x7f09740c5e40 2023-10-10T19:17:11.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f097a40a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097416a040 con 0x7f09741567e0 2023-10-10T19:17:11.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f0973fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f097406bf00 0x7f09740c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:11.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f0973fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f097406bf00 0x7f09740c7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43208/0 (socket says 172.21.15.175:43208) 2023-10-10T19:17:11.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f0973fff700 1 -- 172.21.15.175:0/805950744 learned_addr learned my addr 172.21.15.175:0/805950744 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:11.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f09737fe700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09740c5e40 0x7f09741662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:11.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f09789a7700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09741567e0 0x7f09741587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:11.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f0973fff700 1 -- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09741567e0 msgr2=0x7f09741587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:11.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f0973fff700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09741567e0 0x7f09741587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f0973fff700 1 -- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09740c5e40 msgr2=0x7f09741662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:11.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f0973fff700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09740c5e40 0x7f09741662d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.039+0000 7f0973fff700 1 -- 172.21.15.175:0/805950744 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680093c0 con 0x7f097406bf00 2023-10-10T19:17:11.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.040+0000 7f0973fff700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f097406bf00 0x7f09740c7290 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f0968006360 tx=0x7f0968016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:11.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.040+0000 7f095bfff700 1 -- 172.21.15.175:0/805950744 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096800bbe0 con 0x7f097406bf00 2023-10-10T19:17:11.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.040+0000 7f095bfff700 1 -- 172.21.15.175:0/805950744 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096800bd60 con 0x7f097406bf00 2023-10-10T19:17:11.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.040+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097415abc0 con 0x7f097406bf00 2023-10-10T19:17:11.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.040+0000 7f095bfff700 1 -- 172.21.15.175:0/805950744 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096801a670 con 0x7f097406bf00 2023-10-10T19:17:11.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.040+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09741699f0 con 0x7f097406bf00 2023-10-10T19:17:11.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.041+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0974067570 con 0x7f097406bf00 2023-10-10T19:17:11.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.041+0000 7f095bfff700 1 -- 172.21.15.175:0/805950744 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f096801aa20 con 0x7f097406bf00 2023-10-10T19:17:11.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.041+0000 7f095bfff700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0960066f60 0x7f0960069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:11.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.042+0000 7f095bfff700 1 -- 172.21.15.175:0/805950744 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f0968086360 con 0x7f097406bf00 2023-10-10T19:17:11.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.042+0000 7f09737fe700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0960066f60 0x7f0960069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:11.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.042+0000 7f09737fe700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0960066f60 0x7f0960069410 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f09740c0770 tx=0x7f0964008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:11.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.047+0000 7f095bfff700 1 -- 172.21.15.175:0/805950744 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0968052fe0 con 0x7f097406bf00 2023-10-10T19:17:11.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.271+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f097406c690 con 0x7f097406bf00 2023-10-10T19:17:11.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.271+0000 7f095bfff700 1 -- 172.21.15.175:0/805950744 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+6232 (secure 0 0 0) 0x7f0968056600 con 0x7f097406bf00 2023-10-10T19:17:11.273 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:11.273 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:39.782385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:11.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0960066f60 msgr2=0x7f0960069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:11.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0960066f60 0x7f0960069410 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f09740c0770 tx=0x7f0964008040 comp rx=0 tx=0).stop 2023-10-10T19:17:11.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f097406bf00 msgr2=0x7f09740c7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:11.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f097406bf00 0x7f09740c7290 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f0968006360 tx=0x7f0968016000 comp rx=0 tx=0).stop 2023-10-10T19:17:11.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 shutdown_connections 2023-10-10T19:17:11.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0960066f60 0x7f0960069410 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09741567e0 0x7f09741587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f097406bf00 0x7f09740c7290 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.273+0000 7f097a40a700 1 --2- 172.21.15.175:0/805950744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09740c5e40 0x7f09741662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:11.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.274+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 >> 172.21.15.175:0/805950744 conn(0x7f0974068900 msgr2=0x7f097414f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:11.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.274+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 shutdown_connections 2023-10-10T19:17:11.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:11.274+0000 7f097a40a700 1 -- 172.21.15.175:0/805950744 wait complete. 2023-10-10T19:17:11.278 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 243 2023-10-10T19:17:11.287 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:17:13.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:13.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.418+0000 7f1ede310700 1 Processor -- start 2023-10-10T19:17:13.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.418+0000 7f1ede310700 1 -- start start 2023-10-10T19:17:13.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.418+0000 7f1ede310700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1ed814c840 0x7f1ed806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:13.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.418+0000 7f1ede310700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed80c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:13.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.418+0000 7f1ede310700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed8155ca0 con 0x7f1ed80c59d0 2023-10-10T19:17:13.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.418+0000 7f1ede310700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed806bd50 con 0x7f1ed814c840 2023-10-10T19:17:13.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.418+0000 7f1ede310700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed8156b00 con 0x7f1ed814cc10 2023-10-10T19:17:13.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1edc8ad700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1ed80c59d0 0x7f1ed806b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35364/0 (socket says 172.21.15.175:35364) 2023-10-10T19:17:13.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1edc8ad700 1 -- 172.21.15.175:0/1915954118 learned_addr learned my addr 172.21.15.175:0/1915954118 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:13.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed7fff700 1 --2- 172.21.15.175:0/1915954118 >> v2:172.21.15.191:3300/0 conn(0x7f1ed814c840 0x7f1ed806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:13.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed77fe700 1 --2- 172.21.15.175:0/1915954118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed80c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:13.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed6ffd700 1 -- 172.21.15.175:0/1915954118 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254848794 0 0) 0x7f1ed8155ca0 con 0x7f1ed80c59d0 2023-10-10T19:17:13.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed6ffd700 1 -- 172.21.15.175:0/1915954118 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec0003570 con 0x7f1ed80c59d0 2023-10-10T19:17:13.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed77fe700 1 -- 172.21.15.175:0/1915954118 >> v2:172.21.15.191:3300/0 conn(0x7f1ed814c840 msgr2=0x7f1ed806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:13.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed77fe700 1 --2- 172.21.15.175:0/1915954118 >> v2:172.21.15.191:3300/0 conn(0x7f1ed814c840 0x7f1ed806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed6ffd700 1 -- 172.21.15.175:0/1915954118 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1739647377 0 0) 0x7f1ec0003570 con 0x7f1ed80c59d0 2023-10-10T19:17:13.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed77fe700 1 -- 172.21.15.175:0/1915954118 >> v1:172.21.15.175:6789/0 conn(0x7f1ed80c59d0 legacy=0x7f1ed806b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:13.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed77fe700 1 -- 172.21.15.175:0/1915954118 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed806d700 con 0x7f1ed814cc10 2023-10-10T19:17:13.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed7fff700 1 --2- 172.21.15.175:0/1915954118 >> v2:172.21.15.191:3300/0 conn(0x7f1ed814c840 0x7f1ed806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:17:13.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed77fe700 1 --2- 172.21.15.175:0/1915954118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed80c4b90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0098e0 tx=0x7f1ecc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44ec340ecaebc4fc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:13.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.419+0000 7f1ed6ffd700 1 -- 172.21.15.175:0/1915954118 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00017f0 con 0x7f1ed814cc10 2023-10-10T19:17:13.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ed6ffd700 1 -- 172.21.15.175:0/1915954118 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc011420 con 0x7f1ed814cc10 2023-10-10T19:17:13.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ed6ffd700 1 -- 172.21.15.175:0/1915954118 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0117c0 con 0x7f1ed814cc10 2023-10-10T19:17:13.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ede310700 1 -- 172.21.15.175:0/1915954118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 msgr2=0x7f1ed80c4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:13.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ede310700 1 --2- 172.21.15.175:0/1915954118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed80c4b90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0098e0 tx=0x7f1ecc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ede310700 1 -- 172.21.15.175:0/1915954118 shutdown_connections 2023-10-10T19:17:13.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ede310700 1 --2- 172.21.15.175:0/1915954118 >> v2:172.21.15.191:3300/0 conn(0x7f1ed814c840 0x7f1ed806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ede310700 1 --2- 172.21.15.175:0/1915954118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed80c4b90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ede310700 1 -- 172.21.15.175:0/1915954118 >> 172.21.15.175:0/1915954118 conn(0x7f1ed8068280 msgr2=0x7f1ed806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:13.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ede310700 1 -- 172.21.15.175:0/1915954118 shutdown_connections 2023-10-10T19:17:13.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.420+0000 7f1ede310700 1 -- 172.21.15.175:0/1915954118 wait complete. 2023-10-10T19:17:13.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.421+0000 7f1ede310700 1 Processor -- start 2023-10-10T19:17:13.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.421+0000 7f1ede310700 1 -- start start 2023-10-10T19:17:13.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ede310700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ed80c59d0 0x7f1ed8164d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:13.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ede310700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ed814c840 0x7f1ed814d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:13.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ede310700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed8148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:13.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ede310700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed80c5740 con 0x7f1ed814c840 2023-10-10T19:17:13.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ede310700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed8152530 con 0x7f1ed80c59d0 2023-10-10T19:17:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ede310700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed8159580 con 0x7f1ed814cc10 2023-10-10T19:17:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ed814c840 0x7f1ed814d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ed814c840 0x7f1ed814d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:43218/0 (socket says 172.21.15.175:43218) 2023-10-10T19:17:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed77fe700 1 -- 172.21.15.175:0/4051626034 learned_addr learned my addr 172.21.15.175:0/4051626034 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1edc8ad700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed8148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed7fff700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ed80c59d0 0x7f1ed8164d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed77fe700 1 -- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 msgr2=0x7f1ed8148800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed77fe700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed8148800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed77fe700 1 -- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ed80c59d0 msgr2=0x7f1ed8164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:13.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed77fe700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ed80c59d0 0x7f1ed8164d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed77fe700 1 -- 172.21.15.175:0/4051626034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80093c0 con 0x7f1ed814c840 2023-10-10T19:17:13.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.422+0000 7f1ed7fff700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ed80c59d0 0x7f1ed8164d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:13.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.423+0000 7f1ed77fe700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ed814c840 0x7f1ed814d020 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0030c0 tx=0x7f1ecc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:13.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.423+0000 7f1ebffff700 1 -- 172.21.15.175:0/4051626034 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc01cd60 con 0x7f1ed814c840 2023-10-10T19:17:13.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.423+0000 7f1ebffff700 1 -- 172.21.15.175:0/4051626034 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc01a450 con 0x7f1ed814c840 2023-10-10T19:17:13.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.423+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc009420 con 0x7f1ed814c840 2023-10-10T19:17:13.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.423+0000 7f1ebffff700 1 -- 172.21.15.175:0/4051626034 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc01a810 con 0x7f1ed814c840 2023-10-10T19:17:13.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.423+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed81539a0 con 0x7f1ed814c840 2023-10-10T19:17:13.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.424+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed806d3f0 con 0x7f1ed814c840 2023-10-10T19:17:13.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.424+0000 7f1ebffff700 1 -- 172.21.15.175:0/4051626034 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1ecc01ac40 con 0x7f1ed814c840 2023-10-10T19:17:13.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.425+0000 7f1ebffff700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ec4066f60 0x7f1ec4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:13.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.425+0000 7f1ebffff700 1 -- 172.21.15.175:0/4051626034 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 7225+0+0 (secure 0 0 0) 0x7f1ecc086520 con 0x7f1ed814c840 2023-10-10T19:17:13.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.425+0000 7f1ed7fff700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ec4066f60 0x7f1ec4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:13.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.425+0000 7f1ed7fff700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ec4066f60 0x7f1ec4069410 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009390 tx=0x7f1ec8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:13.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.429+0000 7f1ebffff700 1 -- 172.21.15.175:0/4051626034 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ecc053180 con 0x7f1ed814c840 2023-10-10T19:17:13.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.655+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ed8067570 con 0x7f1ed814c840 2023-10-10T19:17:13.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.655+0000 7f1ebffff700 1 -- 172.21.15.175:0/4051626034 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+6232 (secure 0 0 0) 0x7f1ecc0567a0 con 0x7f1ed814c840 2023-10-10T19:17:13.656 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:13.656 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:15:39.782385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":6080,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6080":{"gid":6080,"name":"b","rank":1,"incarnation":229,"state":"up:active","state_seq":18,"addr":"172.21.15.191:6836/4084912544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":4084912544},{"type":"v1","addr":"172.21.15.191:6836","nonce":4084912544}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:13.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ec4066f60 msgr2=0x7f1ec4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:13.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ec4066f60 0x7f1ec4069410 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009390 tx=0x7f1ec8008040 comp rx=0 tx=0).stop 2023-10-10T19:17:13.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ed814c840 msgr2=0x7f1ed814d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:13.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ed814c840 0x7f1ed814d020 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0030c0 tx=0x7f1ecc016000 comp rx=0 tx=0).stop 2023-10-10T19:17:13.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 shutdown_connections 2023-10-10T19:17:13.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1ec4066f60 0x7f1ec4069410 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ed814cc10 0x7f1ed8148800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ed80c59d0 0x7f1ed8164d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 --2- 172.21.15.175:0/4051626034 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ed814c840 0x7f1ed814d020 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:13.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 >> 172.21.15.175:0/4051626034 conn(0x7f1ed8068280 msgr2=0x7f1ed80c2940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:13.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 shutdown_connections 2023-10-10T19:17:13.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:13.657+0000 7f1ede310700 1 -- 172.21.15.175:0/4051626034 wait complete. 2023-10-10T19:17:13.662 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 243 2023-10-10T19:17:13.669 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-10T19:17:15.670 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:15.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.794+0000 7fd338f75700 1 Processor -- start 2023-10-10T19:17:15.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.794+0000 7fd338f75700 1 -- start start 2023-10-10T19:17:15.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.795+0000 7fd338f75700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd334150120 0x7fd3340c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:15.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.795+0000 7fd338f75700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3341504f0 0x7fd3340c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:15.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.795+0000 7fd338f75700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd3340c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:15.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.795+0000 7fd338f75700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33406f3e0 con 0x7fd334150120 2023-10-10T19:17:15.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.795+0000 7fd338f75700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33406d970 con 0x7fd3341504f0 2023-10-10T19:17:15.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.795+0000 7fd338f75700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd334156fe0 con 0x7fd3340c9450 2023-10-10T19:17:15.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd334150120 0x7fd3340c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:15.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd334150120 0x7fd3340c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60462/0 (socket says 172.21.15.175:60462) 2023-10-10T19:17:15.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 -- 172.21.15.175:0/315070851 learned_addr learned my addr 172.21.15.175:0/315070851 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:15.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd332d9d700 1 --2- 172.21.15.175:0/315070851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd3340c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:15.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd331d9b700 1 --2- 172.21.15.175:0/315070851 >> v2:172.21.15.191:3300/0 conn(0x7fd3341504f0 0x7fd3340c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:15.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 -- 172.21.15.175:0/315070851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 msgr2=0x7fd3340c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:15.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 --2- 172.21.15.175:0/315070851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd3340c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:15.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 -- 172.21.15.175:0/315070851 >> v2:172.21.15.191:3300/0 conn(0x7fd3341504f0 msgr2=0x7fd3340c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:15.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 --2- 172.21.15.175:0/315070851 >> v2:172.21.15.191:3300/0 conn(0x7fd3341504f0 0x7fd3340c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:15.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 -- 172.21.15.175:0/315070851 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334156460 con 0x7fd334150120 2023-10-10T19:17:15.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd332d9d700 1 --2- 172.21.15.175:0/315070851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd3340c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:15.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd331d9b700 1 --2- 172.21.15.175:0/315070851 >> v2:172.21.15.191:3300/0 conn(0x7fd3341504f0 0x7fd3340c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:15.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.796+0000 7fd33259c700 1 --2- 172.21.15.175:0/315070851 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334150120 0x7fd3340c8f10 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fd3340650f0 tx=0x7fd324009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a1d4394e61d493 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:15.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.797+0000 7fd33159a700 1 -- 172.21.15.175:0/315070851 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32400e070 con 0x7fd334150120 2023-10-10T19:17:15.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd33159a700 1 -- 172.21.15.175:0/315070851 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32400ba50 con 0x7fd334150120 2023-10-10T19:17:15.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd33159a700 1 -- 172.21.15.175:0/315070851 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3240092e0 con 0x7fd334150120 2023-10-10T19:17:15.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 -- 172.21.15.175:0/315070851 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334150120 msgr2=0x7fd3340c8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:15.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 --2- 172.21.15.175:0/315070851 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334150120 0x7fd3340c8f10 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fd3340650f0 tx=0x7fd324009840 comp rx=0 tx=0).stop 2023-10-10T19:17:15.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 -- 172.21.15.175:0/315070851 shutdown_connections 2023-10-10T19:17:15.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 --2- 172.21.15.175:0/315070851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd3340c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:15.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 --2- 172.21.15.175:0/315070851 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd334150120 0x7fd3340c8f10 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:15.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 --2- 172.21.15.175:0/315070851 >> v2:172.21.15.191:3300/0 conn(0x7fd3341504f0 0x7fd3340c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:15.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 -- 172.21.15.175:0/315070851 >> 172.21.15.175:0/315070851 conn(0x7fd3340682c0 msgr2=0x7fd3340c87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:15.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 -- 172.21.15.175:0/315070851 shutdown_connections 2023-10-10T19:17:15.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.798+0000 7fd338f75700 1 -- 172.21.15.175:0/315070851 wait complete. 2023-10-10T19:17:15.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.799+0000 7fd338f75700 1 Processor -- start 2023-10-10T19:17:15.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.799+0000 7fd338f75700 1 -- start start 2023-10-10T19:17:15.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd338f75700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd334169060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:15.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd338f75700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334150120 0x7fd334166560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:15.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd338f75700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3341504f0 0x7fd334145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:15.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd338f75700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd334159cc0 con 0x7fd3341504f0 2023-10-10T19:17:15.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd338f75700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3341660d0 con 0x7fd334150120 2023-10-10T19:17:15.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd338f75700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd334165aa0 con 0x7fd3340c9450 2023-10-10T19:17:15.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd332d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3341504f0 0x7fd334145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:15.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd332d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3341504f0 0x7fd334145160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60478/0 (socket says 172.21.15.175:60478) 2023-10-10T19:17:15.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd332d9d700 1 -- 172.21.15.175:0/2803848716 learned_addr learned my addr 172.21.15.175:0/2803848716 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:15.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd331d9b700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334150120 0x7fd334166560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:15.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd33259c700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd334169060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:15.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd332d9d700 1 -- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 msgr2=0x7fd334169060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:15.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd332d9d700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd334169060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:15.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd332d9d700 1 -- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334150120 msgr2=0x7fd334166560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:15.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd332d9d700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334150120 0x7fd334166560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:15.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.800+0000 7fd332d9d700 1 -- 172.21.15.175:0/2803848716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd324009ae0 con 0x7fd3341504f0 2023-10-10T19:17:15.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.801+0000 7fd332d9d700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3341504f0 0x7fd334145160 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00e8a0 tx=0x7fd32c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:15.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.801+0000 7fd32a7fc700 1 -- 172.21.15.175:0/2803848716 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c013410 con 0x7fd3341504f0 2023-10-10T19:17:15.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.801+0000 7fd32a7fc700 1 -- 172.21.15.175:0/2803848716 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c013570 con 0x7fd3341504f0 2023-10-10T19:17:15.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.801+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd334159f10 con 0x7fd3341504f0 2023-10-10T19:17:15.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.801+0000 7fd32a7fc700 1 -- 172.21.15.175:0/2803848716 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0138f0 con 0x7fd3341504f0 2023-10-10T19:17:15.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.801+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd334165d60 con 0x7fd3341504f0 2023-10-10T19:17:15.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.802+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd334067570 con 0x7fd3341504f0 2023-10-10T19:17:15.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.802+0000 7fd32a7fc700 1 -- 172.21.15.175:0/2803848716 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd32c013ce0 con 0x7fd3341504f0 2023-10-10T19:17:15.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.802+0000 7fd32a7fc700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:15.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.803+0000 7fd32a7fc700 1 -- 172.21.15.175:0/2803848716 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fd32c084790 con 0x7fd3341504f0 2023-10-10T19:17:15.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.803+0000 7fd33259c700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:15.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.803+0000 7fd33259c700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd324009880 tx=0x7fd324018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:15.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:15.807+0000 7fd32a7fc700 1 -- 172.21.15.175:0/2803848716 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd32c054e60 con 0x7fd3341504f0 2023-10-10T19:17:16.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.031+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd334166210 con 0x7fd3341504f0 2023-10-10T19:17:16.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.031+0000 7fd32a7fc700 1 -- 172.21.15.175:0/2803848716 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+5465 (secure 0 0 0) 0x7fd32c0549f0 con 0x7fd3341504f0 2023-10-10T19:17:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:16.032 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:13.770581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.033+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 msgr2=0x7fd3140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.033+0000 7fd338f75700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd324009880 tx=0x7fd324018000 comp rx=0 tx=0).stop 2023-10-10T19:17:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.033+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3341504f0 msgr2=0x7fd334145160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.033+0000 7fd338f75700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3341504f0 0x7fd334145160 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00e8a0 tx=0x7fd32c008040 comp rx=0 tx=0).stop 2023-10-10T19:17:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.033+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 shutdown_connections 2023-10-10T19:17:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.033+0000 7fd338f75700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.033+0000 7fd338f75700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3341504f0 0x7fd334145160 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.034+0000 7fd338f75700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3340c9450 0x7fd334169060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.034+0000 7fd338f75700 1 --2- 172.21.15.175:0/2803848716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd334150120 0x7fd334166560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.034+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 >> 172.21.15.175:0/2803848716 conn(0x7fd3340682c0 msgr2=0x7fd33414d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.034+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 shutdown_connections 2023-10-10T19:17:16.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.034+0000 7fd338f75700 1 -- 172.21.15.175:0/2803848716 wait complete. 2023-10-10T19:17:16.037 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 244 2023-10-10T19:17:16.046 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-10T19:17:16.046 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:17:16.047 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:16.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.161+0000 7fcf32862700 1 Processor -- start 2023-10-10T19:17:16.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.161+0000 7fcf32862700 1 -- start start 2023-10-10T19:17:16.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.162+0000 7fcf32862700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:16.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.162+0000 7fcf32862700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c156800 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.162+0000 7fcf32862700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c063e30 con 0x7fcf2c14fbf0 2023-10-10T19:17:16.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.162+0000 7fcf32862700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0c59e0 con 0x7fcf2c159d70 2023-10-10T19:17:16.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.162+0000 7fcf2b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcf2c14ffc0 0x7fcf2c0681e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42776/0 (socket says 172.21.15.175:42776) 2023-10-10T19:17:16.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.162+0000 7fcf2b7fe700 1 -- 172.21.15.175:0/3387443112 learned_addr learned my addr 172.21.15.175:0/3387443112 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:16.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.162+0000 7fcf30dff700 1 --2- 172.21.15.175:0/3387443112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:16.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.162+0000 7fcf2affd700 1 -- 172.21.15.175:0/3387443112 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165785214 0 0) 0x7fcf2c156800 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf2affd700 1 -- 172.21.15.175:0/3387443112 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf14003570 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf2affd700 1 -- 172.21.15.175:0/3387443112 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134744918 0 0) 0x7fcf2c063e30 con 0x7fcf2c14fbf0 2023-10-10T19:17:16.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf2affd700 1 -- 172.21.15.175:0/3387443112 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf2c156800 con 0x7fcf2c14fbf0 2023-10-10T19:17:16.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf30dff700 1 -- 172.21.15.175:0/3387443112 >> v1:172.21.15.191:6789/0 conn(0x7fcf2c14fbf0 legacy=0x7fcf2c063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf30dff700 1 -- 172.21.15.175:0/3387443112 >> v1:172.21.15.175:6789/0 conn(0x7fcf2c14ffc0 legacy=0x7fcf2c0681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf2affd700 1 -- 172.21.15.175:0/3387443112 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1596918431 0 0) 0x7fcf14003570 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf30dff700 1 -- 172.21.15.175:0/3387443112 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c156c90 con 0x7fcf2c159d70 2023-10-10T19:17:16.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf30dff700 1 --2- 172.21.15.175:0/3387443112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c0c5fb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009e90 tx=0x7fcf24009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5af8cd539d26bb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:16.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf2affd700 1 -- 172.21.15.175:0/3387443112 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2400e070 con 0x7fcf2c159d70 2023-10-10T19:17:16.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf2affd700 1 -- 172.21.15.175:0/3387443112 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2400b870 con 0x7fcf2c159d70 2023-10-10T19:17:16.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.163+0000 7fcf2affd700 1 -- 172.21.15.175:0/3387443112 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20001470 con 0x7fcf2c159d70 2023-10-10T19:17:16.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.164+0000 7fcf32862700 1 -- 172.21.15.175:0/3387443112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 msgr2=0x7fcf2c0c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.164+0000 7fcf32862700 1 --2- 172.21.15.175:0/3387443112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c0c5fb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009e90 tx=0x7fcf24009780 comp rx=0 tx=0).stop 2023-10-10T19:17:16.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.164+0000 7fcf32862700 1 -- 172.21.15.175:0/3387443112 shutdown_connections 2023-10-10T19:17:16.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.164+0000 7fcf32862700 1 --2- 172.21.15.175:0/3387443112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c0c5fb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.164+0000 7fcf32862700 1 -- 172.21.15.175:0/3387443112 >> 172.21.15.175:0/3387443112 conn(0x7fcf2c0cc760 msgr2=0x7fcf2c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:16.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.164+0000 7fcf32862700 1 -- 172.21.15.175:0/3387443112 shutdown_connections 2023-10-10T19:17:16.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.164+0000 7fcf32862700 1 -- 172.21.15.175:0/3387443112 wait complete. 2023-10-10T19:17:16.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.165+0000 7fcf32862700 1 Processor -- start 2023-10-10T19:17:16.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.165+0000 7fcf32862700 1 -- start start 2023-10-10T19:17:16.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf32862700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf2c14fbf0 0x7fcf2c15a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:16.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf32862700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf2c14ffc0 0x7fcf2c14c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:16.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf32862700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c14c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:16.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf32862700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c1667e0 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf32862700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c155430 con 0x7fcf2c14fbf0 2023-10-10T19:17:16.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf32862700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c1556f0 con 0x7fcf2c159d70 2023-10-10T19:17:16.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf2c14ffc0 0x7fcf2c14c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:16.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf2c14ffc0 0x7fcf2c14c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60488/0 (socket says 172.21.15.175:60488) 2023-10-10T19:17:16.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 -- 172.21.15.175:0/1607278734 learned_addr learned my addr 172.21.15.175:0/1607278734 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:16.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2bfff700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf2c14fbf0 0x7fcf2c15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:16.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf30dff700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c14c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:16.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 -- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 msgr2=0x7fcf2c14c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c14c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 -- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf2c14fbf0 msgr2=0x7fcf2c15a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf2c14fbf0 0x7fcf2c15a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 -- 172.21.15.175:0/1607278734 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf24009a20 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2bfff700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf2c14fbf0 0x7fcf2c15a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:16.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf30dff700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c14c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:16.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf2b7fe700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf2c14ffc0 0x7fcf2c14c210 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000cb60 tx=0x7fcf20001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:16.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf13fff700 1 -- 172.21.15.175:0/1607278734 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20007e90 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf13fff700 1 -- 172.21.15.175:0/1607278734 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf20007360 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c165bd0 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf13fff700 1 -- 172.21.15.175:0/1607278734 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20004830 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.166+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c166190 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.167+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c067570 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.168+0000 7fcf13fff700 1 -- 172.21.15.175:0/1607278734 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcf20004ba0 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.168+0000 7fcf13fff700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf18066f30 0x7fcf180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:16.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.168+0000 7fcf13fff700 1 -- 172.21.15.175:0/1607278734 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fcf200843b0 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.168+0000 7fcf2bfff700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf18066f30 0x7fcf180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:16.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.169+0000 7fcf2bfff700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf18066f30 0x7fcf180693e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c001170 tx=0x7fcf1c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:16.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.172+0000 7fcf13fff700 1 -- 172.21.15.175:0/1607278734 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf20051070 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.395+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf2c154a10 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.396+0000 7fcf13fff700 1 -- 172.21.15.175:0/1607278734 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+5465 (secure 0 0 0) 0x7fcf20054690 con 0x7fcf2c14ffc0 2023-10-10T19:17:16.396 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:16.397 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:13.770581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:16.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf18066f30 msgr2=0x7fcf180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf18066f30 0x7fcf180693e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c001170 tx=0x7fcf1c009040 comp rx=0 tx=0).stop 2023-10-10T19:17:16.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf2c14ffc0 msgr2=0x7fcf2c14c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:16.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf2c14ffc0 0x7fcf2c14c210 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000cb60 tx=0x7fcf20001ad0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 shutdown_connections 2023-10-10T19:17:16.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcf18066f30 0x7fcf180693e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf2c159d70 0x7fcf2c14c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf2c14fbf0 0x7fcf2c15a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 --2- 172.21.15.175:0/1607278734 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcf2c14ffc0 0x7fcf2c14c210 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:16.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.397+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 >> 172.21.15.175:0/1607278734 conn(0x7fcf2c0cc760 msgr2=0x7fcf2c14ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:16.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.398+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 shutdown_connections 2023-10-10T19:17:16.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:16.398+0000 7fcf32862700 1 -- 172.21.15.175:0/1607278734 wait complete. 2023-10-10T19:17:16.401 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 244 2023-10-10T19:17:16.411 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 244, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 244, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 125, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:17:13.770581+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:17:18.412 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:18.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.529+0000 7f099d92c700 1 Processor -- start 2023-10-10T19:17:18.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.530+0000 7f099d92c700 1 -- start start 2023-10-10T19:17:18.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.530+0000 7f099d92c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c5850 0x7f09980c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:18.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.530+0000 7f099d92c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09980c4f20 0x7f09980c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:18.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.530+0000 7f099d92c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09980c6fb0 con 0x7f09980c12d0 2023-10-10T19:17:18.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.530+0000 7f099d92c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0998063ea0 con 0x7f09980c4f20 2023-10-10T19:17:18.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.530+0000 7f099d92c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0998155f30 con 0x7f09980c5850 2023-10-10T19:17:18.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0996ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c5850 0x7f09980c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:18.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f09977fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09980c4f20 0x7f09980c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:18.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0996ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c5850 0x7f09980c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:57950/0 (socket says 172.21.15.175:57950) 2023-10-10T19:17:18.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0996ffd700 1 -- 172.21.15.175:0/1664451419 learned_addr learned my addr 172.21.15.175:0/1664451419 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:18.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0996ffd700 1 -- 172.21.15.175:0/1664451419 >> v2:172.21.15.191:3300/0 conn(0x7f09980c4f20 msgr2=0x7f09980c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:18.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0996ffd700 1 --2- 172.21.15.175:0/1664451419 >> v2:172.21.15.191:3300/0 conn(0x7f09980c4f20 0x7f09980c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0996ffd700 1 -- 172.21.15.175:0/1664451419 >> v1:172.21.15.191:6789/0 conn(0x7f09980c12d0 legacy=0x7f0998156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:18.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0996ffd700 1 -- 172.21.15.175:0/1664451419 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09980c6160 con 0x7f09980c5850 2023-10-10T19:17:18.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0995ffb700 1 -- 172.21.15.175:0/1664451419 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323215254 0 0) 0x7f09980c6fb0 con 0x7f09980c12d0 2023-10-10T19:17:18.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f09977fe700 1 --2- 172.21.15.175:0/1664451419 >> v2:172.21.15.191:3300/0 conn(0x7f09980c4f20 0x7f09980c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:17:18.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0996ffd700 1 --2- 172.21.15.175:0/1664451419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c5850 0x7f09980c5c20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0988009f20 tx=0x7f0988009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff49f6a4a49e64e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:18.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0995ffb700 1 -- 172.21.15.175:0/1664451419 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098800e070 con 0x7f09980c5850 2023-10-10T19:17:18.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0995ffb700 1 -- 172.21.15.175:0/1664451419 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098800ba60 con 0x7f09980c5850 2023-10-10T19:17:18.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.531+0000 7f0995ffb700 1 -- 172.21.15.175:0/1664451419 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09880092e0 con 0x7f09980c5850 2023-10-10T19:17:18.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.532+0000 7f099d92c700 1 -- 172.21.15.175:0/1664451419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c5850 msgr2=0x7f09980c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:18.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.532+0000 7f099d92c700 1 --2- 172.21.15.175:0/1664451419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c5850 0x7f09980c5c20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0988009f20 tx=0x7f0988009840 comp rx=0 tx=0).stop 2023-10-10T19:17:18.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.532+0000 7f099d92c700 1 -- 172.21.15.175:0/1664451419 shutdown_connections 2023-10-10T19:17:18.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.532+0000 7f099d92c700 1 --2- 172.21.15.175:0/1664451419 >> v2:172.21.15.191:3300/0 conn(0x7f09980c4f20 0x7f09980c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.532+0000 7f099d92c700 1 --2- 172.21.15.175:0/1664451419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c5850 0x7f09980c5c20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.532+0000 7f099d92c700 1 -- 172.21.15.175:0/1664451419 >> 172.21.15.175:0/1664451419 conn(0x7f099806dc70 msgr2=0x7f0998063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:18.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.532+0000 7f099d92c700 1 -- 172.21.15.175:0/1664451419 shutdown_connections 2023-10-10T19:17:18.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.532+0000 7f099d92c700 1 -- 172.21.15.175:0/1664451419 wait complete. 2023-10-10T19:17:18.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.533+0000 7f099d92c700 1 Processor -- start 2023-10-10T19:17:18.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.533+0000 7f099d92c700 1 -- start start 2023-10-10T19:17:18.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f099d92c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09980c12d0 0x7f09980c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:18.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f099d92c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c4f20 0x7f099806af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:18.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f099d92c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09980c5850 0x7f09981570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:18.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f099d92c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0998158a10 con 0x7f09980c5850 2023-10-10T19:17:18.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f099d92c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09980c8390 con 0x7f09980c12d0 2023-10-10T19:17:18.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f099d92c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0998158f70 con 0x7f09980c4f20 2023-10-10T19:17:18.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09980c5850 0x7f09981570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:18.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09980c5850 0x7f09981570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60496/0 (socket says 172.21.15.175:60496) 2023-10-10T19:17:18.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 -- 172.21.15.175:0/949698875 learned_addr learned my addr 172.21.15.175:0/949698875 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:18.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f0996ffd700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09980c12d0 0x7f09980c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:18.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09967fc700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c4f20 0x7f099806af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:18.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 -- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c4f20 msgr2=0x7f099806af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:18.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c4f20 0x7f099806af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 -- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09980c12d0 msgr2=0x7f09980c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:18.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09980c12d0 0x7f09980c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 -- 172.21.15.175:0/949698875 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990009420 con 0x7f09980c5850 2023-10-10T19:17:18.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f09977fe700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09980c5850 0x7f09981570a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f099000ea20 tx=0x7f09900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:18.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f097effd700 1 -- 172.21.15.175:0/949698875 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099000ee60 con 0x7f09980c5850 2023-10-10T19:17:18.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f097effd700 1 -- 172.21.15.175:0/949698875 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099000c850 con 0x7f09980c5850 2023-10-10T19:17:18.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f097effd700 1 -- 172.21.15.175:0/949698875 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099000cc10 con 0x7f09980c5850 2023-10-10T19:17:18.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.534+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0988009ae0 con 0x7f09980c5850 2023-10-10T19:17:18.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.535+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09980c88c0 con 0x7f09980c5850 2023-10-10T19:17:18.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.535+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0998067570 con 0x7f09980c5850 2023-10-10T19:17:18.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.536+0000 7f097effd700 1 -- 172.21.15.175:0/949698875 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0990013470 con 0x7f09980c5850 2023-10-10T19:17:18.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.536+0000 7f097effd700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0984066f60 0x7f0984069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:18.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.536+0000 7f097effd700 1 -- 172.21.15.175:0/949698875 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f0990084ac0 con 0x7f09980c5850 2023-10-10T19:17:18.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.536+0000 7f0996ffd700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0984066f60 0x7f0984069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:18.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.537+0000 7f0996ffd700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0984066f60 0x7f0984069410 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0988009880 tx=0x7f0988018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:18.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.540+0000 7f097effd700 1 -- 172.21.15.175:0/949698875 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0990051700 con 0x7f09980c5850 2023-10-10T19:17:18.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.764+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09981590b0 con 0x7f09980c5850 2023-10-10T19:17:18.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.765+0000 7f097effd700 1 -- 172.21.15.175:0/949698875 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+5465 (secure 0 0 0) 0x7f0990054d20 con 0x7f09980c5850 2023-10-10T19:17:18.765 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:18.766 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:13.770581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:18.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0984066f60 msgr2=0x7f0984069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0984066f60 0x7f0984069410 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0988009880 tx=0x7f0988018000 comp rx=0 tx=0).stop 2023-10-10T19:17:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09980c5850 msgr2=0x7f09981570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09980c5850 0x7f09981570a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f099000ea20 tx=0x7f09900091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 shutdown_connections 2023-10-10T19:17:18.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0984066f60 0x7f0984069410 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09980c5850 0x7f09981570a0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09980c12d0 0x7f09980c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 --2- 172.21.15.175:0/949698875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09980c4f20 0x7f099806af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 >> 172.21.15.175:0/949698875 conn(0x7f099806dc70 msgr2=0x7f0998063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:18.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 shutdown_connections 2023-10-10T19:17:18.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:18.767+0000 7f099d92c700 1 -- 172.21.15.175:0/949698875 wait complete. 2023-10-10T19:17:18.770 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 244 2023-10-10T19:17:18.781 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:20.782 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:20.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba82fa8700 1 Processor -- start 2023-10-10T19:17:20.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba82fa8700 1 -- start start 2023-10-10T19:17:20.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba82fa8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 0x7fba7c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:20.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba82fa8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba7c0b2390 0x7fba7c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:20.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba82fa8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7c1574b0 con 0x7fba7c068bd0 2023-10-10T19:17:20.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba82fa8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba7c150770 con 0x7fba7c0b2390 2023-10-10T19:17:20.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba82fa8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c0afad0 con 0x7fba7c0b1fc0 2023-10-10T19:17:20.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba7bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba7c0b2390 0x7fba7c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:20.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.902+0000 7fba7bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fba7c0b2390 0x7fba7c068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60506/0 (socket says 172.21.15.175:60506) 2023-10-10T19:17:20.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7bfff700 1 -- 172.21.15.175:0/3706286425 learned_addr learned my addr 172.21.15.175:0/3706286425 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:20.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba80d44700 1 --2- 172.21.15.175:0/3706286425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 0x7fba7c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:20.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7bfff700 1 -- 172.21.15.175:0/3706286425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 msgr2=0x7fba7c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:20.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7bfff700 1 --2- 172.21.15.175:0/3706286425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 0x7fba7c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:20.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7bfff700 1 -- 172.21.15.175:0/3706286425 >> v1:172.21.15.191:6789/0 conn(0x7fba7c068bd0 legacy=0x7fba7c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:20.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7bfff700 1 -- 172.21.15.175:0/3706286425 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c0af4c0 con 0x7fba7c0b2390 2023-10-10T19:17:20.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7bfff700 1 --2- 172.21.15.175:0/3706286425 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c068690 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fba680099b0 tx=0x7fba68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa010bddcd63dac0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:20.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7b7fe700 1 -- 172.21.15.175:0/3706286425 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68009ea0 con 0x7fba7c0b2390 2023-10-10T19:17:20.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7b7fe700 1 -- 172.21.15.175:0/3706286425 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6800b8f0 con 0x7fba7c0b2390 2023-10-10T19:17:20.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.903+0000 7fba7b7fe700 1 -- 172.21.15.175:0/3706286425 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6800bc90 con 0x7fba7c0b2390 2023-10-10T19:17:20.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.904+0000 7fba82fa8700 1 -- 172.21.15.175:0/3706286425 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 msgr2=0x7fba7c068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:20.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.904+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3706286425 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c068690 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fba680099b0 tx=0x7fba68009250 comp rx=0 tx=0).stop 2023-10-10T19:17:20.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.904+0000 7fba82fa8700 1 -- 172.21.15.175:0/3706286425 shutdown_connections 2023-10-10T19:17:20.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.904+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3706286425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 0x7fba7c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:20.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.904+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3706286425 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c068690 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:20.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.904+0000 7fba82fa8700 1 -- 172.21.15.175:0/3706286425 >> 172.21.15.175:0/3706286425 conn(0x7fba7c063230 msgr2=0x7fba7c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:20.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.904+0000 7fba82fa8700 1 -- 172.21.15.175:0/3706286425 shutdown_connections 2023-10-10T19:17:20.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.904+0000 7fba82fa8700 1 -- 172.21.15.175:0/3706286425 wait complete. 2023-10-10T19:17:20.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.905+0000 7fba82fa8700 1 Processor -- start 2023-10-10T19:17:20.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.905+0000 7fba82fa8700 1 -- start start 2023-10-10T19:17:20.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba82fa8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba7c068bd0 0x7fba7c1598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:20.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba82fa8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 0x7fba7c159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:20.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba82fa8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c165f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:20.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba82fa8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c0c4950 con 0x7fba7c0b2390 2023-10-10T19:17:20.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba82fa8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c1550a0 con 0x7fba7c068bd0 2023-10-10T19:17:20.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba82fa8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c16a110 con 0x7fba7c0b1fc0 2023-10-10T19:17:20.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:20.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba80d44700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba7c068bd0 0x7fba7c1598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:20.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c165f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60516/0 (socket says 172.21.15.175:60516) 2023-10-10T19:17:20.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 -- 172.21.15.175:0/3764860173 learned_addr learned my addr 172.21.15.175:0/3764860173 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:20.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba7bfff700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 0x7fba7c159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:20.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 -- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 msgr2=0x7fba7c159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:20.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 0x7fba7c159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:20.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 -- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba7c068bd0 msgr2=0x7fba7c1598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:20.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba7c068bd0 0x7fba7c1598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:20.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 -- 172.21.15.175:0/3764860173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680094f0 con 0x7fba7c0b2390 2023-10-10T19:17:20.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba81545700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c165f80 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c910 tx=0x7fba740016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:20.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.906+0000 7fba78ff9700 1 -- 172.21.15.175:0/3764860173 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74012630 con 0x7fba7c0b2390 2023-10-10T19:17:20.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.907+0000 7fba78ff9700 1 -- 172.21.15.175:0/3764860173 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba74012790 con 0x7fba7c0b2390 2023-10-10T19:17:20.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.907+0000 7fba78ff9700 1 -- 172.21.15.175:0/3764860173 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74012b50 con 0x7fba7c0b2390 2023-10-10T19:17:20.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.907+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c1552d0 con 0x7fba7c0b2390 2023-10-10T19:17:20.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.907+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c1697e0 con 0x7fba7c0b2390 2023-10-10T19:17:20.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.908+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c151530 con 0x7fba7c0b2390 2023-10-10T19:17:20.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.908+0000 7fba78ff9700 1 -- 172.21.15.175:0/3764860173 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fba7401b450 con 0x7fba7c0b2390 2023-10-10T19:17:20.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.909+0000 7fba78ff9700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba60066f60 0x7fba60069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:20.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.909+0000 7fba78ff9700 1 -- 172.21.15.175:0/3764860173 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fba74084cb0 con 0x7fba7c0b2390 2023-10-10T19:17:20.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.909+0000 7fba80d44700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba60066f60 0x7fba60069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:20.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.909+0000 7fba80d44700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba60066f60 0x7fba60069410 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fba70009db0 tx=0x7fba700091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:20.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:20.917+0000 7fba78ff9700 1 -- 172.21.15.175:0/3764860173 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba74055380 con 0x7fba7c0b2390 2023-10-10T19:17:21.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.151+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba7c067570 con 0x7fba7c0b2390 2023-10-10T19:17:21.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.151+0000 7fba78ff9700 1 -- 172.21.15.175:0/3764860173 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5463 (secure 0 0 0) 0x7fba74054f10 con 0x7fba7c0b2390 2023-10-10T19:17:21.152 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:21.152 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:19.801603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:21.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba60066f60 msgr2=0x7fba60069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:21.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba60066f60 0x7fba60069410 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fba70009db0 tx=0x7fba700091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:21.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 msgr2=0x7fba7c165f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:21.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c165f80 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c910 tx=0x7fba740016f0 comp rx=0 tx=0).stop 2023-10-10T19:17:21.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 shutdown_connections 2023-10-10T19:17:21.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba60066f60 0x7fba60069410 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:21.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba7c0b2390 0x7fba7c165f80 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:21.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba7c068bd0 0x7fba7c1598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:21.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 --2- 172.21.15.175:0/3764860173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba7c0b1fc0 0x7fba7c159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:21.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.153+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 >> 172.21.15.175:0/3764860173 conn(0x7fba7c063230 msgr2=0x7fba7c14fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:21.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.154+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 shutdown_connections 2023-10-10T19:17:21.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:21.154+0000 7fba82fa8700 1 -- 172.21.15.175:0/3764860173 wait complete. 2023-10-10T19:17:21.157 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 245 2023-10-10T19:17:21.166 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:23.167 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:23.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.295+0000 7f7537b29700 1 Processor -- start 2023-10-10T19:17:23.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.295+0000 7f7537b29700 1 -- start start 2023-10-10T19:17:23.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f7537b29700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f7537b29700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f753014ffc0 0x7f7530063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f7537b29700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7530156800 con 0x7f75300681e0 2023-10-10T19:17:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f7537b29700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75300c59e0 con 0x7f753014ffc0 2023-10-10T19:17:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f7537b29700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753015a2b0 con 0x7f753014fbf0 2023-10-10T19:17:23.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f75358c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f75350c4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f753014ffc0 0x7f7530063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f75360c6700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f75300681e0 0x7f75300c5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:34618/0 (socket says 172.21.15.175:34618) 2023-10-10T19:17:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f75350c4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f753014ffc0 0x7f7530063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:33498/0 (socket says 172.21.15.175:33498) 2023-10-10T19:17:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f75358c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:57982/0 (socket says 172.21.15.175:57982) 2023-10-10T19:17:23.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.296+0000 7f75360c6700 1 -- 172.21.15.175:0/414100449 learned_addr learned my addr 172.21.15.175:0/414100449 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75350c4700 1 -- 172.21.15.175:0/414100449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 msgr2=0x7f7530159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75350c4700 1 --2- 172.21.15.175:0/414100449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75350c4700 1 -- 172.21.15.175:0/414100449 >> v1:172.21.15.191:6789/0 conn(0x7f75300681e0 legacy=0x7f75300c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75350c4700 1 -- 172.21.15.175:0/414100449 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7530063e30 con 0x7f753014ffc0 2023-10-10T19:17:23.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75358c5700 1 --2- 172.21.15.175:0/414100449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:17:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75350c4700 1 --2- 172.21.15.175:0/414100449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 0x7f7530063230 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7520009910 tx=0x7f75200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab3f3b5cdbcfed7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75348c3700 1 -- 172.21.15.175:0/414100449 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752000b640 con 0x7f753014ffc0 2023-10-10T19:17:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75348c3700 1 -- 172.21.15.175:0/414100449 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752000b7a0 con 0x7f753014ffc0 2023-10-10T19:17:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.297+0000 7f75348c3700 1 -- 172.21.15.175:0/414100449 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752000bb60 con 0x7f753014ffc0 2023-10-10T19:17:23.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.298+0000 7f7537b29700 1 -- 172.21.15.175:0/414100449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 msgr2=0x7f7530063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.298+0000 7f7537b29700 1 --2- 172.21.15.175:0/414100449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 0x7f7530063230 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7520009910 tx=0x7f75200091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.298+0000 7f7537b29700 1 -- 172.21.15.175:0/414100449 shutdown_connections 2023-10-10T19:17:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.298+0000 7f7537b29700 1 --2- 172.21.15.175:0/414100449 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.298+0000 7f7537b29700 1 --2- 172.21.15.175:0/414100449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 0x7f7530063230 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.298+0000 7f7537b29700 1 -- 172.21.15.175:0/414100449 >> 172.21.15.175:0/414100449 conn(0x7f75300cc760 msgr2=0x7f75300ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:23.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.298+0000 7f7537b29700 1 -- 172.21.15.175:0/414100449 shutdown_connections 2023-10-10T19:17:23.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.298+0000 7f7537b29700 1 -- 172.21.15.175:0/414100449 wait complete. 2023-10-10T19:17:23.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.299+0000 7f7537b29700 1 Processor -- start 2023-10-10T19:17:23.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.299+0000 7f7537b29700 1 -- start start 2023-10-10T19:17:23.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.299+0000 7f7537b29700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75300681e0 0x7f75300c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:23.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.299+0000 7f7537b29700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530169270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:23.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.299+0000 7f7537b29700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 0x7f753014b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:23.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.299+0000 7f7537b29700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75301667b0 con 0x7f75300681e0 2023-10-10T19:17:23.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.299+0000 7f7537b29700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530166bd0 con 0x7f753014ffc0 2023-10-10T19:17:23.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.299+0000 7f7537b29700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75300c7df0 con 0x7f753014fbf0 2023-10-10T19:17:23.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75300681e0 0x7f75300c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75300681e0 0x7f75300c6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60522/0 (socket says 172.21.15.175:60522) 2023-10-10T19:17:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 -- 172.21.15.175:0/60699588 learned_addr learned my addr 172.21.15.175:0/60699588 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75360c6700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 0x7f753014b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75350c4700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530169270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:23.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 -- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 msgr2=0x7f7530169270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530169270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 -- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 msgr2=0x7f753014b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 0x7f753014b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 -- 172.21.15.175:0/60699588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c0093c0 con 0x7f75300681e0 2023-10-10T19:17:23.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f75358c5700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75300681e0 0x7f75300c6090 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f752c00ecb0 tx=0x7f752c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f7525ffb700 1 -- 172.21.15.175:0/60699588 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c005a00 con 0x7f75300681e0 2023-10-10T19:17:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f7525ffb700 1 -- 172.21.15.175:0/60699588 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752c005b60 con 0x7f75300681e0 2023-10-10T19:17:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7520009420 con 0x7f75300681e0 2023-10-10T19:17:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.300+0000 7f7525ffb700 1 -- 172.21.15.175:0/60699588 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c00c920 con 0x7f75300681e0 2023-10-10T19:17:23.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.301+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75300c7ba0 con 0x7f75300681e0 2023-10-10T19:17:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.301+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7530067570 con 0x7f75300681e0 2023-10-10T19:17:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.302+0000 7f7525ffb700 1 -- 172.21.15.175:0/60699588 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f752c00cd10 con 0x7f75300681e0 2023-10-10T19:17:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.302+0000 7f7525ffb700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7514066f30 0x7f75140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.302+0000 7f7525ffb700 1 -- 172.21.15.175:0/60699588 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f752c084fb0 con 0x7f75300681e0 2023-10-10T19:17:23.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.302+0000 7f75350c4700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7514066f30 0x7f75140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:23.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.303+0000 7f75350c4700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7514066f30 0x7f75140693e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f75200091e0 tx=0x7f7520017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:23.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.306+0000 7f7525ffb700 1 -- 172.21.15.175:0/60699588 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f752c0b65a0 con 0x7f75300681e0 2023-10-10T19:17:23.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.534+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7530069780 con 0x7f75300681e0 2023-10-10T19:17:23.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.534+0000 7f7525ffb700 1 -- 172.21.15.175:0/60699588 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+5465 (secure 0 0 0) 0x7f752c020020 con 0x7f75300681e0 2023-10-10T19:17:23.535 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:23.535 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:21.807520+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:resolve","state_seq":32,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:23.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7514066f30 msgr2=0x7f75140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:23.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7514066f30 0x7f75140693e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f75200091e0 tx=0x7f7520017000 comp rx=0 tx=0).stop 2023-10-10T19:17:23.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75300681e0 msgr2=0x7f75300c6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:23.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75300681e0 0x7f75300c6090 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f752c00ecb0 tx=0x7f752c009120 comp rx=0 tx=0).stop 2023-10-10T19:17:23.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 shutdown_connections 2023-10-10T19:17:23.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7514066f30 0x7f75140693e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753014ffc0 0x7f753014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75300681e0 0x7f75300c6090 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 --2- 172.21.15.175:0/60699588 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f753014fbf0 0x7f7530169270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:23.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 >> 172.21.15.175:0/60699588 conn(0x7f75300cc760 msgr2=0x7f7530159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:23.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 shutdown_connections 2023-10-10T19:17:23.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:23.536+0000 7f7537b29700 1 -- 172.21.15.175:0/60699588 wait complete. 2023-10-10T19:17:23.539 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 246 2023-10-10T19:17:23.548 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:24.198 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:17:24.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:17:25.550 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:25.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.672+0000 7f540f6b9700 1 Processor -- start 2023-10-10T19:17:25.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.672+0000 7f540f6b9700 1 -- start start 2023-10-10T19:17:25.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.673+0000 7f540f6b9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f54080c2cf0 0x7f540806e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.673+0000 7f540f6b9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f540814e290 0x7f540815e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.673+0000 7f540f6b9700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5408153f70 con 0x7f54080c30c0 2023-10-10T19:17:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.673+0000 7f540f6b9700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5408067bd0 con 0x7f54080c2cf0 2023-10-10T19:17:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.673+0000 7f540f6b9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5408154210 con 0x7f540814e290 2023-10-10T19:17:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.673+0000 7f540d455700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f54080c2cf0 0x7f540806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.673+0000 7f540cc54700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f54080c30c0 0x7f540814db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42232/0 (socket says 172.21.15.175:42232) 2023-10-10T19:17:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.673+0000 7f540cc54700 1 -- 172.21.15.175:0/30138773 learned_addr learned my addr 172.21.15.175:0/30138773 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f540dc56700 1 --2- 172.21.15.175:0/30138773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f540814e290 0x7f540815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f540d455700 1 -- 172.21.15.175:0/30138773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f540814e290 msgr2=0x7f540815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f540d455700 1 --2- 172.21.15.175:0/30138773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f540814e290 0x7f540815e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f540d455700 1 -- 172.21.15.175:0/30138773 >> v1:172.21.15.175:6789/0 conn(0x7f54080c30c0 legacy=0x7f540814db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:25.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f540d455700 1 -- 172.21.15.175:0/30138773 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54081544f0 con 0x7f54080c2cf0 2023-10-10T19:17:25.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f53fffff700 1 -- 172.21.15.175:0/30138773 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799435945 0 0) 0x7f5408153f70 con 0x7f54080c30c0 2023-10-10T19:17:25.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f540d455700 1 --2- 172.21.15.175:0/30138773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c2cf0 0x7f540806e500 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f54000098d0 tx=0x7f5400009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6621312081f152f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:25.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f53fffff700 1 -- 172.21.15.175:0/30138773 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400009e70 con 0x7f54080c2cf0 2023-10-10T19:17:25.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f53fffff700 1 -- 172.21.15.175:0/30138773 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5400005c10 con 0x7f54080c2cf0 2023-10-10T19:17:25.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f53fffff700 1 -- 172.21.15.175:0/30138773 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540000b6e0 con 0x7f54080c2cf0 2023-10-10T19:17:25.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f540f6b9700 1 -- 172.21.15.175:0/30138773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c2cf0 msgr2=0x7f540806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:25.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.674+0000 7f540f6b9700 1 --2- 172.21.15.175:0/30138773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c2cf0 0x7f540806e500 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f54000098d0 tx=0x7f5400009120 comp rx=0 tx=0).stop 2023-10-10T19:17:25.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.675+0000 7f540f6b9700 1 -- 172.21.15.175:0/30138773 shutdown_connections 2023-10-10T19:17:25.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.675+0000 7f540f6b9700 1 --2- 172.21.15.175:0/30138773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f540814e290 0x7f540815e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.675+0000 7f540f6b9700 1 --2- 172.21.15.175:0/30138773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c2cf0 0x7f540806e500 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.675+0000 7f540f6b9700 1 -- 172.21.15.175:0/30138773 >> 172.21.15.175:0/30138773 conn(0x7f5408068900 msgr2=0x7f5408068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:25.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.675+0000 7f540f6b9700 1 -- 172.21.15.175:0/30138773 shutdown_connections 2023-10-10T19:17:25.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.675+0000 7f540f6b9700 1 -- 172.21.15.175:0/30138773 wait complete. 2023-10-10T19:17:25.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.676+0000 7f540f6b9700 1 Processor -- start 2023-10-10T19:17:25.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.676+0000 7f540f6b9700 1 -- start start 2023-10-10T19:17:25.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540f6b9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54080c2cf0 0x7f54080c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:25.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540f6b9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c30c0 0x7f54081695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:25.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540f6b9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f540814e290 0x7f5408148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:25.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540f6b9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5408156c20 con 0x7f54080c30c0 2023-10-10T19:17:25.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540f6b9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5408157480 con 0x7f540814e290 2023-10-10T19:17:25.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540f6b9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54080ca1c0 con 0x7f54080c2cf0 2023-10-10T19:17:25.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c30c0 0x7f54081695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:25.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c30c0 0x7f54081695b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52496/0 (socket says 172.21.15.175:52496) 2023-10-10T19:17:25.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 -- 172.21.15.175:0/3136096604 learned_addr learned my addr 172.21.15.175:0/3136096604 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:25.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540dc56700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f540814e290 0x7f5408148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:25.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540d455700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54080c2cf0 0x7f54080c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:25.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 -- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54080c2cf0 msgr2=0x7f54080c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:25.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54080c2cf0 0x7f54080c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 -- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f540814e290 msgr2=0x7f5408148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:25.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f540814e290 0x7f5408148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 -- 172.21.15.175:0/3136096604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54000093c0 con 0x7f54080c30c0 2023-10-10T19:17:25.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540dc56700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f540814e290 0x7f5408148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:25.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540cc54700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c30c0 0x7f54081695b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f53f800c810 tx=0x7f53f8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:25.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f53fd7fa700 1 -- 172.21.15.175:0/3136096604 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8005b30 con 0x7f54080c30c0 2023-10-10T19:17:25.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f53fd7fa700 1 -- 172.21.15.175:0/3136096604 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f8005c90 con 0x7f54080c30c0 2023-10-10T19:17:25.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540816a050 con 0x7f54080c30c0 2023-10-10T19:17:25.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f53fd7fa700 1 -- 172.21.15.175:0/3136096604 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f80072e0 con 0x7f54080c30c0 2023-10-10T19:17:25.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.677+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5408157110 con 0x7f54080c30c0 2023-10-10T19:17:25.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.678+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5408067570 con 0x7f54080c30c0 2023-10-10T19:17:25.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.679+0000 7f53fd7fa700 1 -- 172.21.15.175:0/3136096604 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f53f80076d0 con 0x7f54080c30c0 2023-10-10T19:17:25.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.679+0000 7f53fd7fa700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f53ec066f30 0x7f53ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:25.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.679+0000 7f53fd7fa700 1 -- 172.21.15.175:0/3136096604 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f53f8083fc0 con 0x7f54080c30c0 2023-10-10T19:17:25.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.679+0000 7f540d455700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f53ec066f30 0x7f53ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:25.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.680+0000 7f540d455700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f53ec066f30 0x7f53ec0693e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5400009160 tx=0x7f5400016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:25.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.683+0000 7f53fd7fa700 1 -- 172.21.15.175:0/3136096604 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f53f804b5a0 con 0x7f54080c30c0 2023-10-10T19:17:25.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.906+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54080694a0 con 0x7f54080c30c0 2023-10-10T19:17:25.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.906+0000 7f53fd7fa700 1 -- 172.21.15.175:0/3136096604 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5461 (secure 0 0 0) 0x7f53f8017090 con 0x7f54080c30c0 2023-10-10T19:17:25.907 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:25.908 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:24.817338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:resolve","state_seq":32,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:25.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.908+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f53ec066f30 msgr2=0x7f53ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:25.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.908+0000 7f540f6b9700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f53ec066f30 0x7f53ec0693e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5400009160 tx=0x7f5400016000 comp rx=0 tx=0).stop 2023-10-10T19:17:25.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.908+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c30c0 msgr2=0x7f54081695b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:25.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.908+0000 7f540f6b9700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c30c0 0x7f54081695b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f53f800c810 tx=0x7f53f8007df0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.908+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 shutdown_connections 2023-10-10T19:17:25.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.909+0000 7f540f6b9700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f53ec066f30 0x7f53ec0693e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.909+0000 7f540f6b9700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f540814e290 0x7f5408148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.909+0000 7f540f6b9700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54080c2cf0 0x7f54080c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.909+0000 7f540f6b9700 1 --2- 172.21.15.175:0/3136096604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54080c30c0 0x7f54081695b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:25.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.909+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 >> 172.21.15.175:0/3136096604 conn(0x7f5408068900 msgr2=0x7f540806edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:25.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.909+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 shutdown_connections 2023-10-10T19:17:25.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:25.909+0000 7f540f6b9700 1 -- 172.21.15.175:0/3136096604 wait complete. 2023-10-10T19:17:25.912 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 247 2023-10-10T19:17:25.921 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:27.922 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:28.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.051+0000 7f903c863700 1 Processor -- start 2023-10-10T19:17:28.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.051+0000 7f903c863700 1 -- start start 2023-10-10T19:17:28.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.052+0000 7f903c863700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90380cac80 0x7f9038063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:28.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.052+0000 7f903c863700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90380c4ee0 con 0x7f9038068e10 2023-10-10T19:17:28.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.052+0000 7f903c863700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90380c5780 con 0x7f9038068a40 2023-10-10T19:17:28.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.052+0000 7f903c863700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9038064180 con 0x7f90380cac80 2023-10-10T19:17:28.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.052+0000 7f9035d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90380cac80 0x7f9038063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:28.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.052+0000 7f9036d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9038068e10 0x7f90380af910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42242/0 (socket says 172.21.15.175:42242) 2023-10-10T19:17:28.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.052+0000 7f9036d9d700 1 -- 172.21.15.175:0/2515363629 learned_addr learned my addr 172.21.15.175:0/2515363629 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:28.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f9035d9b700 1 -- 172.21.15.175:0/2515363629 >> v1:172.21.15.191:6789/0 conn(0x7f9038068a40 legacy=0x7f903806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:28.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f9035d9b700 1 -- 172.21.15.175:0/2515363629 >> v1:172.21.15.175:6789/0 conn(0x7f9038068e10 legacy=0x7f90380af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:28.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f9035d9b700 1 -- 172.21.15.175:0/2515363629 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90380c6710 con 0x7f90380cac80 2023-10-10T19:17:28.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f903559a700 1 -- 172.21.15.175:0/2515363629 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400255151 0 0) 0x7f90380c4ee0 con 0x7f9038068e10 2023-10-10T19:17:28.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f9035d9b700 1 --2- 172.21.15.175:0/2515363629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90380cac80 0x7f9038063c40 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f90200098d0 tx=0x7f9020009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfb34c64ca237a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:28.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f903559a700 1 -- 172.21.15.175:0/2515363629 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9020009e70 con 0x7f90380cac80 2023-10-10T19:17:28.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f903559a700 1 -- 172.21.15.175:0/2515363629 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9020005c10 con 0x7f90380cac80 2023-10-10T19:17:28.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f903559a700 1 -- 172.21.15.175:0/2515363629 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902000b650 con 0x7f90380cac80 2023-10-10T19:17:28.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f903c863700 1 -- 172.21.15.175:0/2515363629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90380cac80 msgr2=0x7f9038063c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:28.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.053+0000 7f903c863700 1 --2- 172.21.15.175:0/2515363629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90380cac80 0x7f9038063c40 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f90200098d0 tx=0x7f9020009120 comp rx=0 tx=0).stop 2023-10-10T19:17:28.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.054+0000 7f903c863700 1 -- 172.21.15.175:0/2515363629 shutdown_connections 2023-10-10T19:17:28.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.054+0000 7f903c863700 1 --2- 172.21.15.175:0/2515363629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90380cac80 0x7f9038063c40 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:28.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.054+0000 7f903c863700 1 -- 172.21.15.175:0/2515363629 >> 172.21.15.175:0/2515363629 conn(0x7f903806dbe0 msgr2=0x7f90380ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:28.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.054+0000 7f903c863700 1 -- 172.21.15.175:0/2515363629 shutdown_connections 2023-10-10T19:17:28.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.054+0000 7f903c863700 1 -- 172.21.15.175:0/2515363629 wait complete. 2023-10-10T19:17:28.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903c863700 1 Processor -- start 2023-10-10T19:17:28.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903c863700 1 -- start start 2023-10-10T19:17:28.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903c863700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9038068a40 0x7f90380c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:28.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903c863700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9038068e10 0x7f903815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:28.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903c863700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90380cac80 0x7f90381693f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:28.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903c863700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9038166800 con 0x7f9038068e10 2023-10-10T19:17:28.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903c863700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90380caa80 con 0x7f9038068a40 2023-10-10T19:17:28.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903c863700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9038166a90 con 0x7f90380cac80 2023-10-10T19:17:28.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f9035d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9038068e10 0x7f903815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:28.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f9035d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9038068e10 0x7f903815ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52504/0 (socket says 172.21.15.175:52504) 2023-10-10T19:17:28.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f9035d9b700 1 -- 172.21.15.175:0/2104550898 learned_addr learned my addr 172.21.15.175:0/2104550898 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:28.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f903659c700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9038068a40 0x7f90380c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:28.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.055+0000 7f9036d9d700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90380cac80 0x7f90381693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:28.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f9035d9b700 1 -- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90380cac80 msgr2=0x7f90381693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:28.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f9035d9b700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90380cac80 0x7f90381693f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:28.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f9035d9b700 1 -- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9038068a40 msgr2=0x7f90380c0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:28.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f9035d9b700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9038068a40 0x7f90380c0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:28.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f9035d9b700 1 -- 172.21.15.175:0/2104550898 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90200093c0 con 0x7f9038068e10 2023-10-10T19:17:28.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f9035d9b700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9038068e10 0x7f903815ad10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f9020005fb0 tx=0x7f9020016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:28.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f902e7fc700 1 -- 172.21.15.175:0/2104550898 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902001b690 con 0x7f9038068e10 2023-10-10T19:17:28.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f902e7fc700 1 -- 172.21.15.175:0/2104550898 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902001b7f0 con 0x7f9038068e10 2023-10-10T19:17:28.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90380c4d30 con 0x7f9038068e10 2023-10-10T19:17:28.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f902e7fc700 1 -- 172.21.15.175:0/2104550898 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902001bbb0 con 0x7f9038068e10 2023-10-10T19:17:28.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.056+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90380afd60 con 0x7f9038068e10 2023-10-10T19:17:28.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.057+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9038067570 con 0x7f9038068e10 2023-10-10T19:17:28.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.057+0000 7f902e7fc700 1 -- 172.21.15.175:0/2104550898 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f902002f440 con 0x7f9038068e10 2023-10-10T19:17:28.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.058+0000 7f902e7fc700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9018066f00 0x7f90180693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:28.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.058+0000 7f902e7fc700 1 -- 172.21.15.175:0/2104550898 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f9020086b00 con 0x7f9038068e10 2023-10-10T19:17:28.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.058+0000 7f903659c700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9018066f00 0x7f90180693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:28.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.059+0000 7f903659c700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9018066f00 0x7f90180693b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f902800c580 tx=0x7f9028001b20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:28.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.062+0000 7f902e7fc700 1 -- 172.21.15.175:0/2104550898 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9020053740 con 0x7f9038068e10 2023-10-10T19:17:28.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.293+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90380c6990 con 0x7f9038068e10 2023-10-10T19:17:28.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.294+0000 7f902e7fc700 1 -- 172.21.15.175:0/2104550898 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5459 (secure 0 0 0) 0x7f9020056d60 con 0x7f9038068e10 2023-10-10T19:17:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:28.295 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:27.828332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9018066f00 msgr2=0x7f90180693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:28.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9018066f00 0x7f90180693b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f902800c580 tx=0x7f9028001b20 comp rx=0 tx=0).stop 2023-10-10T19:17:28.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9038068e10 msgr2=0x7f903815ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:28.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9038068e10 0x7f903815ad10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f9020005fb0 tx=0x7f9020016000 comp rx=0 tx=0).stop 2023-10-10T19:17:28.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 shutdown_connections 2023-10-10T19:17:28.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9018066f00 0x7f90180693b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:28.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90380cac80 0x7f90381693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:28.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9038068a40 0x7f90380c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:28.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 --2- 172.21.15.175:0/2104550898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9038068e10 0x7f903815ad10 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:28.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 >> 172.21.15.175:0/2104550898 conn(0x7f903806dbe0 msgr2=0x7f903806b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:28.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 shutdown_connections 2023-10-10T19:17:28.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:28.296+0000 7f903c863700 1 -- 172.21.15.175:0/2104550898 wait complete. 2023-10-10T19:17:28.299 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 248 2023-10-10T19:17:28.309 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 248, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 248, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 34 } }, "last_failure": 0, "last_failure_osd_epoch": 125, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:17:27.828332+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:17:30.310 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:30.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.435+0000 7fdeb5855700 1 Processor -- start 2023-10-10T19:17:30.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.435+0000 7fdeb5855700 1 -- start start 2023-10-10T19:17:30.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.436+0000 7fdeb5855700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdeb00b2390 0x7fdeb0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:30.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.436+0000 7fdeb5855700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb01574b0 con 0x7fdeb00b1fc0 2023-10-10T19:17:30.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.436+0000 7fdeb5855700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb00afad0 con 0x7fdeb00aff60 2023-10-10T19:17:30.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.436+0000 7fdeb5855700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb0150770 con 0x7fdeb00b2390 2023-10-10T19:17:30.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.436+0000 7fdeae7fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdeb00b2390 0x7fdeb0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:30.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.436+0000 7fdeaeffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fdeb00b1fc0 0x7fdeb0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42254/0 (socket says 172.21.15.175:42254) 2023-10-10T19:17:30.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.436+0000 7fdeaeffd700 1 -- 172.21.15.175:0/2175749860 learned_addr learned my addr 172.21.15.175:0/2175749860 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:30.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.437+0000 7fdeae7fc700 1 -- 172.21.15.175:0/2175749860 >> v1:172.21.15.191:6789/0 conn(0x7fdeb00aff60 legacy=0x7fdeb0159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:30.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.437+0000 7fdeae7fc700 1 -- 172.21.15.175:0/2175749860 >> v1:172.21.15.175:6789/0 conn(0x7fdeb00b1fc0 legacy=0x7fdeb0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:30.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.437+0000 7fdeae7fc700 1 -- 172.21.15.175:0/2175749860 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb00af4c0 con 0x7fdeb00b2390 2023-10-10T19:17:30.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.437+0000 7fdeae7fc700 1 --2- 172.21.15.175:0/2175749860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb0155fb0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fdea40098e0 tx=0x7fdea40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423d77f91a2a6385 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:30.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.437+0000 7fdeadffb700 1 -- 172.21.15.175:0/2175749860 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4009e80 con 0x7fdeb00b2390 2023-10-10T19:17:30.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.437+0000 7fdeadffb700 1 -- 172.21.15.175:0/2175749860 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea400b8f0 con 0x7fdeb00b2390 2023-10-10T19:17:30.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.437+0000 7fdeadffb700 1 -- 172.21.15.175:0/2175749860 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea400bc70 con 0x7fdeb00b2390 2023-10-10T19:17:30.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.438+0000 7fdeb5855700 1 -- 172.21.15.175:0/2175749860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 msgr2=0x7fdeb0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:30.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.438+0000 7fdeb5855700 1 --2- 172.21.15.175:0/2175749860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb0155fb0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fdea40098e0 tx=0x7fdea40091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.438+0000 7fdeb5855700 1 -- 172.21.15.175:0/2175749860 shutdown_connections 2023-10-10T19:17:30.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.438+0000 7fdeb5855700 1 --2- 172.21.15.175:0/2175749860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb0155fb0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.438+0000 7fdeb5855700 1 -- 172.21.15.175:0/2175749860 >> 172.21.15.175:0/2175749860 conn(0x7fdeb0063230 msgr2=0x7fdeb0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:30.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.438+0000 7fdeb5855700 1 -- 172.21.15.175:0/2175749860 shutdown_connections 2023-10-10T19:17:30.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.438+0000 7fdeb5855700 1 -- 172.21.15.175:0/2175749860 wait complete. 2023-10-10T19:17:30.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.439+0000 7fdeb5855700 1 Processor -- start 2023-10-10T19:17:30.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.439+0000 7fdeb5855700 1 -- start start 2023-10-10T19:17:30.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeb5855700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb00aff60 0x7fdeb0166b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:30.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeb5855700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb00b1fc0 0x7fdeb00c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:30.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeb5855700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb01594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:30.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeb5855700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00cb720 con 0x7fdeb00b2390 2023-10-10T19:17:30.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeb5855700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb0167080 con 0x7fdeb00b1fc0 2023-10-10T19:17:30.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeb5855700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00cbf80 con 0x7fdeb00aff60 2023-10-10T19:17:30.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb01594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:30.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb01594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52524/0 (socket says 172.21.15.175:52524) 2023-10-10T19:17:30.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 -- 172.21.15.175:0/3012111952 learned_addr learned my addr 172.21.15.175:0/3012111952 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:30.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaeffd700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb00aff60 0x7fdeb0166b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:30.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeae7fc700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb00b1fc0 0x7fdeb00c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:30.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 -- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb00aff60 msgr2=0x7fdeb0166b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:30.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb00aff60 0x7fdeb0166b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 -- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb00b1fc0 msgr2=0x7fdeb00c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:30.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb00b1fc0 0x7fdeb00c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 -- 172.21.15.175:0/3012111952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea4009420 con 0x7fdeb00b2390 2023-10-10T19:17:30.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeaf7fe700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb01594b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fdea800c590 tx=0x7fdea8007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:30.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fde96ffd700 1 -- 172.21.15.175:0/3012111952 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8011590 con 0x7fdeb00b2390 2023-10-10T19:17:30.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fde96ffd700 1 -- 172.21.15.175:0/3012111952 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea80116f0 con 0x7fdeb00b2390 2023-10-10T19:17:30.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb00cccb0 con 0x7fdeb00b2390 2023-10-10T19:17:30.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fde96ffd700 1 -- 172.21.15.175:0/3012111952 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8011ab0 con 0x7fdeb00b2390 2023-10-10T19:17:30.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.440+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb00ccf10 con 0x7fdeb00b2390 2023-10-10T19:17:30.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.441+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb00cc0f0 con 0x7fdeb00b2390 2023-10-10T19:17:30.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.442+0000 7fde96ffd700 1 -- 172.21.15.175:0/3012111952 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdea801b3b0 con 0x7fdeb00b2390 2023-10-10T19:17:30.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.442+0000 7fde96ffd700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde9c066f30 0x7fde9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:30.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.442+0000 7fde96ffd700 1 -- 172.21.15.175:0/3012111952 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fdea8083fe0 con 0x7fdeb00b2390 2023-10-10T19:17:30.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.442+0000 7fdeaeffd700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde9c066f30 0x7fde9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:30.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.443+0000 7fdeaeffd700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde9c066f30 0x7fde9c0693e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdea00020c0 tx=0x7fdea0001a10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:30.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.446+0000 7fde96ffd700 1 -- 172.21.15.175:0/3012111952 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdea8050c20 con 0x7fdeb00b2390 2023-10-10T19:17:30.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.673+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdeb0067570 con 0x7fdeb00b2390 2023-10-10T19:17:30.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.674+0000 7fde96ffd700 1 -- 172.21.15.175:0/3012111952 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5459 (secure 0 0 0) 0x7fdea8017020 con 0x7fdeb00b2390 2023-10-10T19:17:30.674 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:30.675 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:27.828332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:30.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.675+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde9c066f30 msgr2=0x7fde9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:30.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.675+0000 7fdeb5855700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde9c066f30 0x7fde9c0693e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdea00020c0 tx=0x7fdea0001a10 comp rx=0 tx=0).stop 2023-10-10T19:17:30.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 msgr2=0x7fdeb01594b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:30.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb01594b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fdea800c590 tx=0x7fdea8007ef0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 shutdown_connections 2023-10-10T19:17:30.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde9c066f30 0x7fde9c0693e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdeb00b2390 0x7fdeb01594b0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdeb00aff60 0x7fdeb0166b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 --2- 172.21.15.175:0/3012111952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeb00b1fc0 0x7fdeb00c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:30.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 >> 172.21.15.175:0/3012111952 conn(0x7fdeb0063230 msgr2=0x7fdeb015a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:30.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 shutdown_connections 2023-10-10T19:17:30.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:30.676+0000 7fdeb5855700 1 -- 172.21.15.175:0/3012111952 wait complete. 2023-10-10T19:17:30.679 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 248 2023-10-10T19:17:30.688 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:32.689 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:32.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.816+0000 7f7b14b5a700 1 Processor -- start 2023-10-10T19:17:32.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.816+0000 7f7b14b5a700 1 -- start start 2023-10-10T19:17:32.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.817+0000 7f7b14b5a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7b100b2390 0x7f7b10155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:32.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.817+0000 7f7b14b5a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b101574b0 con 0x7f7b100b1fc0 2023-10-10T19:17:32.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.817+0000 7f7b14b5a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b100afad0 con 0x7f7b100aff60 2023-10-10T19:17:32.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.817+0000 7f7b14b5a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b10150770 con 0x7f7b100b2390 2023-10-10T19:17:32.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.817+0000 7f7b0e59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7b100b1fc0 0x7f7b10068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42264/0 (socket says 172.21.15.175:42264) 2023-10-10T19:17:32.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.817+0000 7f7b0e59c700 1 -- 172.21.15.175:0/3929047462 learned_addr learned my addr 172.21.15.175:0/3929047462 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:32.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0dd9b700 1 --2- 172.21.15.175:0/3929047462 >> v2:172.21.15.191:3300/0 conn(0x7f7b100b2390 0x7f7b10155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:32.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0d59a700 1 -- 172.21.15.175:0/3929047462 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710422940 0 0) 0x7f7b101574b0 con 0x7f7b100b1fc0 2023-10-10T19:17:32.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0d59a700 1 -- 172.21.15.175:0/3929047462 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7afc003570 con 0x7f7b100b1fc0 2023-10-10T19:17:32.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0d59a700 1 -- 172.21.15.175:0/3929047462 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585432171 0 0) 0x7f7b100afad0 con 0x7f7b100aff60 2023-10-10T19:17:32.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0d59a700 1 -- 172.21.15.175:0/3929047462 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b101574b0 con 0x7f7b100aff60 2023-10-10T19:17:32.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0d59a700 1 -- 172.21.15.175:0/3929047462 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4276930341 0 0) 0x7f7afc003570 con 0x7f7b100b1fc0 2023-10-10T19:17:32.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0dd9b700 1 -- 172.21.15.175:0/3929047462 >> v1:172.21.15.191:6789/0 conn(0x7f7b100aff60 legacy=0x7f7b10159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:32.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0dd9b700 1 -- 172.21.15.175:0/3929047462 >> v1:172.21.15.175:6789/0 conn(0x7f7b100b1fc0 legacy=0x7f7b10068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:32.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0dd9b700 1 -- 172.21.15.175:0/3929047462 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b100af4c0 con 0x7f7b100b2390 2023-10-10T19:17:32.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0dd9b700 1 --2- 172.21.15.175:0/3929047462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100b2390 0x7f7b10155fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7af80098e0 tx=0x7f7af80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ef44d9e7657f350 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:32.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0d59a700 1 -- 172.21.15.175:0/3929047462 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b000017f0 con 0x7f7b100b2390 2023-10-10T19:17:32.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0d59a700 1 -- 172.21.15.175:0/3929047462 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8011420 con 0x7f7b100b2390 2023-10-10T19:17:32.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.818+0000 7f7b0d59a700 1 -- 172.21.15.175:0/3929047462 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af80117c0 con 0x7f7b100b2390 2023-10-10T19:17:32.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.819+0000 7f7b14b5a700 1 -- 172.21.15.175:0/3929047462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100b2390 msgr2=0x7f7b10155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:32.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.819+0000 7f7b14b5a700 1 --2- 172.21.15.175:0/3929047462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100b2390 0x7f7b10155fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7af80098e0 tx=0x7f7af80091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:32.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.819+0000 7f7b14b5a700 1 -- 172.21.15.175:0/3929047462 shutdown_connections 2023-10-10T19:17:32.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.819+0000 7f7b14b5a700 1 --2- 172.21.15.175:0/3929047462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100b2390 0x7f7b10155fb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:32.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.819+0000 7f7b14b5a700 1 -- 172.21.15.175:0/3929047462 >> 172.21.15.175:0/3929047462 conn(0x7f7b10063230 msgr2=0x7f7b10063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:32.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.819+0000 7f7b14b5a700 1 -- 172.21.15.175:0/3929047462 shutdown_connections 2023-10-10T19:17:32.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.819+0000 7f7b14b5a700 1 -- 172.21.15.175:0/3929047462 wait complete. 2023-10-10T19:17:32.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.820+0000 7f7b14b5a700 1 Processor -- start 2023-10-10T19:17:32.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.820+0000 7f7b14b5a700 1 -- start start 2023-10-10T19:17:32.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b14b5a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100aff60 0x7f7b100cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:32.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b14b5a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b100b1fc0 0x7f7b10152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:32.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b14b5a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b100b2390 0x7f7b10146f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:32.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b14b5a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1014bbb0 con 0x7f7b100b2390 2023-10-10T19:17:32.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b14b5a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b101667f0 con 0x7f7b100aff60 2023-10-10T19:17:32.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b14b5a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10166530 con 0x7f7b100b1fc0 2023-10-10T19:17:32.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b100b2390 0x7f7b10146f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:32.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b100b2390 0x7f7b10146f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52538/0 (socket says 172.21.15.175:52538) 2023-10-10T19:17:32.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 -- 172.21.15.175:0/2383723470 learned_addr learned my addr 172.21.15.175:0/2383723470 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:32.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0e59c700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100aff60 0x7f7b100cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:32.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0dd9b700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b100b1fc0 0x7f7b10152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:32.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 -- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b100b1fc0 msgr2=0x7f7b10152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:32.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b100b1fc0 0x7f7b10152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:32.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 -- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100aff60 msgr2=0x7f7b100cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:32.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100aff60 0x7f7b100cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:32.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 -- 172.21.15.175:0/2383723470 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8009420 con 0x7f7b100b2390 2023-10-10T19:17:32.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0e59c700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100aff60 0x7f7b100cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:17:32.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0dd9b700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b100b1fc0 0x7f7b10152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:17:32.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b0ed9d700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b100b2390 0x7f7b10146f60 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800c590 tx=0x7f7b08007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:32.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.821+0000 7f7b067fc700 1 -- 172.21.15.175:0/2383723470 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08016070 con 0x7f7b100b2390 2023-10-10T19:17:32.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.822+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1014b280 con 0x7f7b100b2390 2023-10-10T19:17:32.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.822+0000 7f7b067fc700 1 -- 172.21.15.175:0/2383723470 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b08011590 con 0x7f7b100b2390 2023-10-10T19:17:32.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.822+0000 7f7b067fc700 1 -- 172.21.15.175:0/2383723470 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08011970 con 0x7f7b100b2390 2023-10-10T19:17:32.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.822+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1014b840 con 0x7f7b100b2390 2023-10-10T19:17:32.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.823+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1014d130 con 0x7f7b100b2390 2023-10-10T19:17:32.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.823+0000 7f7b067fc700 1 -- 172.21.15.175:0/2383723470 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7b08012420 con 0x7f7b100b2390 2023-10-10T19:17:32.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.823+0000 7f7b067fc700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7af0066f30 0x7f7af00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:32.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.823+0000 7f7b067fc700 1 -- 172.21.15.175:0/2383723470 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f7b080847c0 con 0x7f7b100b2390 2023-10-10T19:17:32.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.823+0000 7f7b0e59c700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7af0066f30 0x7f7af00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:32.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.824+0000 7f7b0e59c700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7af0066f30 0x7f7af00693e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7b000019e0 tx=0x7f7b00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:32.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:32.827+0000 7f7b067fc700 1 -- 172.21.15.175:0/2383723470 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b1014d130 con 0x7f7b100b2390 2023-10-10T19:17:33.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.061+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b1014d130 con 0x7f7b100b2390 2023-10-10T19:17:33.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.062+0000 7f7b067fc700 1 -- 172.21.15.175:0/2383723470 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+5457 (secure 0 0 0) 0x7f7b1014d130 con 0x7f7b100b2390 2023-10-10T19:17:33.063 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:33.063 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:32.839797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:33.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7af0066f30 msgr2=0x7f7af00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:33.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7af0066f30 0x7f7af00693e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7b000019e0 tx=0x7f7b00009040 comp rx=0 tx=0).stop 2023-10-10T19:17:33.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b100b2390 msgr2=0x7f7b10146f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:33.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b100b2390 0x7f7b10146f60 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800c590 tx=0x7f7b08007d80 comp rx=0 tx=0).stop 2023-10-10T19:17:33.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 shutdown_connections 2023-10-10T19:17:33.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7af0066f30 0x7f7af00693e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:33.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b100b2390 0x7f7b10146f60 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:33.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b100aff60 0x7f7b100cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:33.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 --2- 172.21.15.175:0/2383723470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b100b1fc0 0x7f7b10152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:33.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 >> 172.21.15.175:0/2383723470 conn(0x7f7b10063230 msgr2=0x7f7b10157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:33.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 shutdown_connections 2023-10-10T19:17:33.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:33.064+0000 7f7b14b5a700 1 -- 172.21.15.175:0/2383723470 wait complete. 2023-10-10T19:17:33.068 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 249 2023-10-10T19:17:33.076 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:35.077 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:35.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.195+0000 7fa128aed700 1 Processor -- start 2023-10-10T19:17:35.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.195+0000 7fa128aed700 1 -- start start 2023-10-10T19:17:35.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa128aed700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1240b1fc0 0x7fa124155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:35.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa128aed700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1240b2390 0x7fa124068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:35.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa128aed700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1241574b0 con 0x7fa124068bd0 2023-10-10T19:17:35.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa128aed700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa124150770 con 0x7fa1240b1fc0 2023-10-10T19:17:35.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa128aed700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240afad0 con 0x7fa1240b2390 2023-10-10T19:17:35.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa12259c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1240b1fc0 0x7fa124155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:35.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa121d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1240b2390 0x7fa124068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:35.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa122d9d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa124068bd0 0x7fa1240aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:58096/0 (socket says 172.21.15.175:58096) 2023-10-10T19:17:35.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa121d9b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1240b2390 0x7fa124068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:37768/0 (socket says 172.21.15.175:37768) 2023-10-10T19:17:35.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa12259c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1240b1fc0 0x7fa124155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:51260/0 (socket says 172.21.15.175:51260) 2023-10-10T19:17:35.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.196+0000 7fa122d9d700 1 -- 172.21.15.175:0/1954669205 learned_addr learned my addr 172.21.15.175:0/1954669205 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:35.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12259c700 1 -- 172.21.15.175:0/1954669205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1240b2390 msgr2=0x7fa124068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:35.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12259c700 1 --2- 172.21.15.175:0/1954669205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1240b2390 0x7fa124068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12259c700 1 -- 172.21.15.175:0/1954669205 >> v1:172.21.15.191:6789/0 conn(0x7fa124068bd0 legacy=0x7fa1240aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:35.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12259c700 1 -- 172.21.15.175:0/1954669205 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1240af4c0 con 0x7fa1240b1fc0 2023-10-10T19:17:35.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12159a700 1 -- 172.21.15.175:0/1954669205 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637630610 0 0) 0x7fa1241574b0 con 0x7fa124068bd0 2023-10-10T19:17:35.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa121d9b700 1 --2- 172.21.15.175:0/1954669205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1240b2390 0x7fa124068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:17:35.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12259c700 1 --2- 172.21.15.175:0/1954669205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b1fc0 0x7fa124155fb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa114009e60 tx=0x7fa114009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df19ab4fa018ea76 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:35.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12159a700 1 -- 172.21.15.175:0/1954669205 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11400e070 con 0x7fa1240b1fc0 2023-10-10T19:17:35.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12159a700 1 -- 172.21.15.175:0/1954669205 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11400bbf0 con 0x7fa1240b1fc0 2023-10-10T19:17:35.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.197+0000 7fa12159a700 1 -- 172.21.15.175:0/1954669205 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114012430 con 0x7fa1240b1fc0 2023-10-10T19:17:35.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.198+0000 7fa128aed700 1 -- 172.21.15.175:0/1954669205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b1fc0 msgr2=0x7fa124155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:35.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.198+0000 7fa128aed700 1 --2- 172.21.15.175:0/1954669205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b1fc0 0x7fa124155fb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa114009e60 tx=0x7fa114009780 comp rx=0 tx=0).stop 2023-10-10T19:17:35.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.198+0000 7fa128aed700 1 -- 172.21.15.175:0/1954669205 shutdown_connections 2023-10-10T19:17:35.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.198+0000 7fa128aed700 1 --2- 172.21.15.175:0/1954669205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b1fc0 0x7fa124155fb0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.198+0000 7fa128aed700 1 --2- 172.21.15.175:0/1954669205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1240b2390 0x7fa124068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.198+0000 7fa128aed700 1 -- 172.21.15.175:0/1954669205 >> 172.21.15.175:0/1954669205 conn(0x7fa124063230 msgr2=0x7fa124063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:35.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.198+0000 7fa128aed700 1 -- 172.21.15.175:0/1954669205 shutdown_connections 2023-10-10T19:17:35.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.198+0000 7fa128aed700 1 -- 172.21.15.175:0/1954669205 wait complete. 2023-10-10T19:17:35.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.199+0000 7fa128aed700 1 Processor -- start 2023-10-10T19:17:35.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.199+0000 7fa128aed700 1 -- start start 2023-10-10T19:17:35.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.199+0000 7fa128aed700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa124068bd0 0x7fa12415b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:35.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.199+0000 7fa128aed700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1240b1fc0 0x7fa1240c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:35.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.199+0000 7fa128aed700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b2390 0x7fa124165e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:35.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.199+0000 7fa128aed700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240c4750 con 0x7fa1240b1fc0 2023-10-10T19:17:35.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.199+0000 7fa128aed700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240c6a90 con 0x7fa1240b2390 2023-10-10T19:17:35.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.199+0000 7fa128aed700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240c51b0 con 0x7fa124068bd0 2023-10-10T19:17:35.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1240b1fc0 0x7fa1240c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:35.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1240b1fc0 0x7fa1240c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58394/0 (socket says 172.21.15.175:58394) 2023-10-10T19:17:35.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 -- 172.21.15.175:0/4015201361 learned_addr learned my addr 172.21.15.175:0/4015201361 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:35.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa12259c700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa124068bd0 0x7fa12415b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:35.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa122d9d700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b2390 0x7fa124165e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:35.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 -- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa124068bd0 msgr2=0x7fa12415b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:35.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa124068bd0 0x7fa12415b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 -- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b2390 msgr2=0x7fa124165e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:35.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b2390 0x7fa124165e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 -- 172.21.15.175:0/4015201361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10c009420 con 0x7fa1240b1fc0 2023-10-10T19:17:35.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa121d9b700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1240b1fc0 0x7fa1240c5a90 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00ecb0 tx=0x7fa10c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:35.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa11a7fc700 1 -- 172.21.15.175:0/4015201361 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00c9d0 con 0x7fa1240b1fc0 2023-10-10T19:17:35.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa11a7fc700 1 -- 172.21.15.175:0/4015201361 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c00cb30 con 0x7fa1240b1fc0 2023-10-10T19:17:35.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa114009a20 con 0x7fa1240b1fc0 2023-10-10T19:17:35.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa11a7fc700 1 -- 172.21.15.175:0/4015201361 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00ceb0 con 0x7fa1240b1fc0 2023-10-10T19:17:35.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.200+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12415bf80 con 0x7fa1240b1fc0 2023-10-10T19:17:35.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.201+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa124067570 con 0x7fa1240b1fc0 2023-10-10T19:17:35.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.202+0000 7fa11a7fc700 1 -- 172.21.15.175:0/4015201361 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa10c013680 con 0x7fa1240b1fc0 2023-10-10T19:17:35.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.203+0000 7fa11a7fc700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa104066f30 0x7fa1040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:35.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.203+0000 7fa11a7fc700 1 -- 172.21.15.175:0/4015201361 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fa10c085690 con 0x7fa1240b1fc0 2023-10-10T19:17:35.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.203+0000 7fa12259c700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa104066f30 0x7fa1040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:35.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.203+0000 7fa12259c700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa104066f30 0x7fa1040693e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa1140097c0 tx=0x7fa114019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:35.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.206+0000 7fa11a7fc700 1 -- 172.21.15.175:0/4015201361 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa10c055de0 con 0x7fa1240b1fc0 2023-10-10T19:17:35.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.429+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa124069060 con 0x7fa1240b1fc0 2023-10-10T19:17:35.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.429+0000 7fa11a7fc700 1 -- 172.21.15.175:0/4015201361 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+5457 (secure 0 0 0) 0x7fa10c055970 con 0x7fa1240b1fc0 2023-10-10T19:17:35.430 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:35.430 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:32.839797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:35.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa104066f30 msgr2=0x7fa1040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:35.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa104066f30 0x7fa1040693e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa1140097c0 tx=0x7fa114019000 comp rx=0 tx=0).stop 2023-10-10T19:17:35.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1240b1fc0 msgr2=0x7fa1240c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:35.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1240b1fc0 0x7fa1240c5a90 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00ecb0 tx=0x7fa10c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 shutdown_connections 2023-10-10T19:17:35.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa104066f30 0x7fa1040693e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1240b2390 0x7fa124165e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa124068bd0 0x7fa12415b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 --2- 172.21.15.175:0/4015201361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1240b1fc0 0x7fa1240c5a90 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:35.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 >> 172.21.15.175:0/4015201361 conn(0x7fa124063230 msgr2=0x7fa1240c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:35.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 shutdown_connections 2023-10-10T19:17:35.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:35.431+0000 7fa128aed700 1 -- 172.21.15.175:0/4015201361 wait complete. 2023-10-10T19:17:35.435 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 249 2023-10-10T19:17:35.444 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:37.445 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:37.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.571+0000 7f84b8285700 1 Processor -- start 2023-10-10T19:17:37.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.571+0000 7f84b8285700 1 -- start start 2023-10-10T19:17:37.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b8285700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84b00c6330 0x7f84b015a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:37.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b8285700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f84b00c4580 0x7f84b0063c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:37.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b8285700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b0157030 con 0x7f84b00c7190 2023-10-10T19:17:37.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b8285700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b00c5000 con 0x7f84b00c4580 2023-10-10T19:17:37.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b8285700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b0156420 con 0x7f84b00c6330 2023-10-10T19:17:37.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6822700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f84b00c4580 0x7f84b0063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:37.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6822700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f84b00c4580 0x7f84b0063c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58406/0 (socket says 172.21.15.175:58406) 2023-10-10T19:17:37.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6822700 1 -- 172.21.15.175:0/1688179052 learned_addr learned my addr 172.21.15.175:0/1688179052 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:37.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6021700 1 --2- 172.21.15.175:0/1688179052 >> v2:172.21.15.191:3300/0 conn(0x7f84b00c6330 0x7f84b015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:37.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6822700 1 -- 172.21.15.175:0/1688179052 >> v2:172.21.15.191:3300/0 conn(0x7f84b00c6330 msgr2=0x7f84b015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:37.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6822700 1 --2- 172.21.15.175:0/1688179052 >> v2:172.21.15.191:3300/0 conn(0x7f84b00c6330 0x7f84b015a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6822700 1 -- 172.21.15.175:0/1688179052 >> v1:172.21.15.191:6789/0 conn(0x7f84b00c7190 legacy=0x7f84b00c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6822700 1 -- 172.21.15.175:0/1688179052 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0156700 con 0x7f84b00c4580 2023-10-10T19:17:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6021700 1 --2- 172.21.15.175:0/1688179052 >> v2:172.21.15.191:3300/0 conn(0x7f84b00c6330 0x7f84b015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:17:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b6822700 1 --2- 172.21.15.175:0/1688179052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c4580 0x7f84b0063c20 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f84b00c01e0 tx=0x7f84a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af204b89a5fff312 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:37.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b501f700 1 -- 172.21.15.175:0/1688179052 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a800e070 con 0x7f84b00c4580 2023-10-10T19:17:37.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b501f700 1 -- 172.21.15.175:0/1688179052 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a800baa0 con 0x7f84b00c4580 2023-10-10T19:17:37.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.572+0000 7f84b501f700 1 -- 172.21.15.175:0/1688179052 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a80092e0 con 0x7f84b00c4580 2023-10-10T19:17:37.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.573+0000 7f84b8285700 1 -- 172.21.15.175:0/1688179052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c4580 msgr2=0x7f84b0063c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:37.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.573+0000 7f84b8285700 1 --2- 172.21.15.175:0/1688179052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c4580 0x7f84b0063c20 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f84b00c01e0 tx=0x7f84a8009840 comp rx=0 tx=0).stop 2023-10-10T19:17:37.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.573+0000 7f84b8285700 1 -- 172.21.15.175:0/1688179052 shutdown_connections 2023-10-10T19:17:37.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.573+0000 7f84b8285700 1 --2- 172.21.15.175:0/1688179052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c4580 0x7f84b0063c20 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.573+0000 7f84b8285700 1 --2- 172.21.15.175:0/1688179052 >> v2:172.21.15.191:3300/0 conn(0x7f84b00c6330 0x7f84b015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.573+0000 7f84b8285700 1 -- 172.21.15.175:0/1688179052 >> 172.21.15.175:0/1688179052 conn(0x7f84b006ef20 msgr2=0x7f84b0063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:37.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.573+0000 7f84b8285700 1 -- 172.21.15.175:0/1688179052 shutdown_connections 2023-10-10T19:17:37.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.573+0000 7f84b8285700 1 -- 172.21.15.175:0/1688179052 wait complete. 2023-10-10T19:17:37.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.574+0000 7f84b8285700 1 Processor -- start 2023-10-10T19:17:37.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.574+0000 7f84b8285700 1 -- start start 2023-10-10T19:17:37.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b8285700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84b00c4580 0x7f84b0157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:37.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b8285700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00c6330 0x7f84b014dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:37.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b8285700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c7190 0x7f84b014e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:37.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b8285700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b014d860 con 0x7f84b00c7190 2023-10-10T19:17:37.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b8285700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b0158e80 con 0x7f84b00c6330 2023-10-10T19:17:37.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b8285700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00cbf00 con 0x7f84b00c4580 2023-10-10T19:17:37.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c7190 0x7f84b014e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:37.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c7190 0x7f84b014e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58408/0 (socket says 172.21.15.175:58408) 2023-10-10T19:17:37.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 -- 172.21.15.175:0/4291005427 learned_addr learned my addr 172.21.15.175:0/4291005427 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:37.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b5820700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00c6330 0x7f84b014dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:37.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6021700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84b00c4580 0x7f84b0157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:37.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 -- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84b00c4580 msgr2=0x7f84b0157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:37.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84b00c4580 0x7f84b0157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 -- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00c6330 msgr2=0x7f84b014dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:37.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00c6330 0x7f84b014dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 -- 172.21.15.175:0/4291005427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a8009ae0 con 0x7f84b00c7190 2023-10-10T19:17:37.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6021700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84b00c4580 0x7f84b0157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:37.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.575+0000 7f84b6822700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c7190 0x7f84b014e1c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f84b0069880 tx=0x7f84a8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:37.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.576+0000 7f84a67fc700 1 -- 172.21.15.175:0/4291005427 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8012d00 con 0x7f84b00c7190 2023-10-10T19:17:37.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.576+0000 7f84a67fc700 1 -- 172.21.15.175:0/4291005427 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a8012e60 con 0x7f84b00c7190 2023-10-10T19:17:37.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.576+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b00cc1a0 con 0x7f84b00c7190 2023-10-10T19:17:37.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.576+0000 7f84a67fc700 1 -- 172.21.15.175:0/4291005427 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a801d670 con 0x7f84b00c7190 2023-10-10T19:17:37.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.576+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b00cc700 con 0x7f84b00c7190 2023-10-10T19:17:37.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.576+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b014d9a0 con 0x7f84b00c7190 2023-10-10T19:17:37.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.577+0000 7f84a67fc700 1 -- 172.21.15.175:0/4291005427 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f84a801da20 con 0x7f84b00c7190 2023-10-10T19:17:37.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.577+0000 7f84a67fc700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8494066f30 0x7f84940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:37.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.577+0000 7f84a67fc700 1 -- 172.21.15.175:0/4291005427 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f84a8086530 con 0x7f84b00c7190 2023-10-10T19:17:37.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.578+0000 7f84b6021700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8494066f30 0x7f84940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:37.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.578+0000 7f84b6021700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8494066f30 0x7f84940693e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00ac10 tx=0x7f84ac006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:37.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.582+0000 7f84a67fc700 1 -- 172.21.15.175:0/4291005427 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84b014d9a0 con 0x7f84b00c7190 2023-10-10T19:17:37.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.808+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84b014d9a0 con 0x7f84b00c7190 2023-10-10T19:17:37.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.808+0000 7f84a67fc700 1 -- 172.21.15.175:0/4291005427 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 250 v250) v1 ==== 95+0+5454 (secure 0 0 0) 0x7f84b014d9a0 con 0x7f84b00c7190 2023-10-10T19:17:37.809 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:37.809 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":250,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":250,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:36.857528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:37.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8494066f30 msgr2=0x7f84940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:37.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8494066f30 0x7f84940693e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00ac10 tx=0x7f84ac006f90 comp rx=0 tx=0).stop 2023-10-10T19:17:37.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c7190 msgr2=0x7f84b014e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:37.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c7190 0x7f84b014e1c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f84b0069880 tx=0x7f84a8017000 comp rx=0 tx=0).stop 2023-10-10T19:17:37.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 shutdown_connections 2023-10-10T19:17:37.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8494066f30 0x7f84940693e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84b00c7190 0x7f84b014e1c0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84b00c4580 0x7f84b0157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 --2- 172.21.15.175:0/4291005427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00c6330 0x7f84b014dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:37.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 >> 172.21.15.175:0/4291005427 conn(0x7f84b006ef20 msgr2=0x7f84b0068690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:37.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 shutdown_connections 2023-10-10T19:17:37.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:37.810+0000 7f84b8285700 1 -- 172.21.15.175:0/4291005427 wait complete. 2023-10-10T19:17:37.813 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 250 2023-10-10T19:17:37.823 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:39.824 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.951+0000 7fc1f5258700 1 Processor -- start 2023-10-10T19:17:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.951+0000 7fc1f5258700 1 -- start start 2023-10-10T19:17:39.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.952+0000 7fc1f5258700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b2390 0x7fc1f0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.952+0000 7fc1f5258700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc1f00aff60 0x7fc1f00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.952+0000 7fc1f5258700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f01574b0 con 0x7fc1f00b1fc0 2023-10-10T19:17:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.952+0000 7fc1f5258700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f0150770 con 0x7fc1f00aff60 2023-10-10T19:17:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.952+0000 7fc1f5258700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00afad0 con 0x7fc1f00b2390 2023-10-10T19:17:39.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ef59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc1f00aff60 0x7fc1f00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:39.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ee59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b2390 0x7fc1f0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:39.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1eed9d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc1f00b1fc0 0x7fc1f0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:58112/0 (socket says 172.21.15.175:58112) 2023-10-10T19:17:39.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1eed9d700 1 -- 172.21.15.175:0/3433077481 learned_addr learned my addr 172.21.15.175:0/3433077481 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:39.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ef59e700 1 --2- 172.21.15.175:0/3433077481 >> v2:172.21.15.191:3300/0 conn(0x7fc1f00aff60 0x7fc1f00b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:51310/0 (socket says 172.21.15.175:51310) 2023-10-10T19:17:39.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ef59e700 1 -- 172.21.15.175:0/3433077481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b2390 msgr2=0x7fc1f0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:39.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ef59e700 1 --2- 172.21.15.175:0/3433077481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b2390 0x7fc1f0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:39.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ef59e700 1 -- 172.21.15.175:0/3433077481 >> v1:172.21.15.191:6789/0 conn(0x7fc1f00b1fc0 legacy=0x7fc1f0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:39.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ef59e700 1 -- 172.21.15.175:0/3433077481 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00af4c0 con 0x7fc1f00aff60 2023-10-10T19:17:39.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ee59c700 1 --2- 172.21.15.175:0/3433077481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b2390 0x7fc1f0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:17:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1ef59e700 1 --2- 172.21.15.175:0/3433077481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 0x7fc1f00b03e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8009910 tx=0x7fc1e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c61917ca1af156 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1edd9b700 1 -- 172.21.15.175:0/3433077481 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800b6d0 con 0x7fc1f00aff60 2023-10-10T19:17:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1edd9b700 1 -- 172.21.15.175:0/3433077481 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e800b830 con 0x7fc1f00aff60 2023-10-10T19:17:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.953+0000 7fc1edd9b700 1 -- 172.21.15.175:0/3433077481 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800bbf0 con 0x7fc1f00aff60 2023-10-10T19:17:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.954+0000 7fc1f5258700 1 -- 172.21.15.175:0/3433077481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 msgr2=0x7fc1f00b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.954+0000 7fc1f5258700 1 --2- 172.21.15.175:0/3433077481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 0x7fc1f00b03e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8009910 tx=0x7fc1e80091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.954+0000 7fc1f5258700 1 -- 172.21.15.175:0/3433077481 shutdown_connections 2023-10-10T19:17:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.954+0000 7fc1f5258700 1 --2- 172.21.15.175:0/3433077481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 0x7fc1f00b03e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.954+0000 7fc1f5258700 1 --2- 172.21.15.175:0/3433077481 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b2390 0x7fc1f0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:39.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.954+0000 7fc1f5258700 1 -- 172.21.15.175:0/3433077481 >> 172.21.15.175:0/3433077481 conn(0x7fc1f0063230 msgr2=0x7fc1f0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:39.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.954+0000 7fc1f5258700 1 -- 172.21.15.175:0/3433077481 shutdown_connections 2023-10-10T19:17:39.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.954+0000 7fc1f5258700 1 -- 172.21.15.175:0/3433077481 wait complete. 2023-10-10T19:17:39.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.955+0000 7fc1f5258700 1 Processor -- start 2023-10-10T19:17:39.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.955+0000 7fc1f5258700 1 -- start start 2023-10-10T19:17:39.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1f5258700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 0x7fc1f00c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:39.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1f5258700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b1fc0 0x7fc1f0153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:39.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1f5258700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f00b2390 0x7fc1f014da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:39.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1f5258700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0166bb0 con 0x7fc1f00b2390 2023-10-10T19:17:39.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1f5258700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0166670 con 0x7fc1f00aff60 2023-10-10T19:17:39.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1f5258700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0154cc0 con 0x7fc1f00b1fc0 2023-10-10T19:17:39.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f00b2390 0x7fc1f014da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:39.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f00b2390 0x7fc1f014da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58422/0 (socket says 172.21.15.175:58422) 2023-10-10T19:17:39.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 -- 172.21.15.175:0/2969519575 learned_addr learned my addr 172.21.15.175:0/2969519575 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:39.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1eed9d700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 0x7fc1f00c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:39.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ee59c700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b1fc0 0x7fc1f0153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:39.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 -- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b1fc0 msgr2=0x7fc1f0153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:39.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b1fc0 0x7fc1f0153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:39.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 -- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 msgr2=0x7fc1f00c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:39.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 0x7fc1f00c3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:39.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 -- 172.21.15.175:0/2969519575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d8009420 con 0x7fc1f00b2390 2023-10-10T19:17:39.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.956+0000 7fc1ef59e700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f00b2390 0x7fc1f014da70 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8009fb0 tx=0x7fc1e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:39.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.957+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/2969519575 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e801f070 con 0x7fc1f00b2390 2023-10-10T19:17:39.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.957+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/2969519575 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e8011420 con 0x7fc1f00b2390 2023-10-10T19:17:39.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.957+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/2969519575 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e80117c0 con 0x7fc1f00b2390 2023-10-10T19:17:39.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.957+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e8009420 con 0x7fc1f00b2390 2023-10-10T19:17:39.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.957+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f0154f60 con 0x7fc1f00b2390 2023-10-10T19:17:39.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.958+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f01559d0 con 0x7fc1f00b2390 2023-10-10T19:17:39.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.958+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/2969519575 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc1e8011bb0 con 0x7fc1f00b2390 2023-10-10T19:17:39.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.958+0000 7fc1e6ffd700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d0066f30 0x7fc1d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:39.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.959+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/2969519575 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fc1e80861a0 con 0x7fc1f00b2390 2023-10-10T19:17:39.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.959+0000 7fc1eed9d700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d0066f30 0x7fc1d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:39.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.959+0000 7fc1eed9d700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d0066f30 0x7fc1d00693e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000e160 tx=0x7fc1e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:39.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:39.963+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/2969519575 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1e8052de0 con 0x7fc1f00b2390 2023-10-10T19:17:40.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.187+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1f0067570 con 0x7fc1f00b2390 2023-10-10T19:17:40.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.188+0000 7fc1e6ffd700 1 -- 172.21.15.175:0/2969519575 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) v1 ==== 95+0+5452 (secure 0 0 0) 0x7fc1e8011e60 con 0x7fc1f00b2390 2023-10-10T19:17:40.188 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:40.189 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":251,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":251,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:39.776275+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:40.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.189+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d0066f30 msgr2=0x7fc1d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:40.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.189+0000 7fc1f5258700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d0066f30 0x7fc1d00693e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000e160 tx=0x7fc1e0009040 comp rx=0 tx=0).stop 2023-10-10T19:17:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.189+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f00b2390 msgr2=0x7fc1f014da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.189+0000 7fc1f5258700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f00b2390 0x7fc1f014da70 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8009fb0 tx=0x7fc1e8017000 comp rx=0 tx=0).stop 2023-10-10T19:17:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.190+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 shutdown_connections 2023-10-10T19:17:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.190+0000 7fc1f5258700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1d0066f30 0x7fc1d00693e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.190+0000 7fc1f5258700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f00b2390 0x7fc1f014da70 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.190+0000 7fc1f5258700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1f00aff60 0x7fc1f00c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.190+0000 7fc1f5258700 1 --2- 172.21.15.175:0/2969519575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1f00b1fc0 0x7fc1f0153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.190+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 >> 172.21.15.175:0/2969519575 conn(0x7fc1f0063230 msgr2=0x7fc1f014fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.190+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 shutdown_connections 2023-10-10T19:17:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:40.190+0000 7fc1f5258700 1 -- 172.21.15.175:0/2969519575 wait complete. 2023-10-10T19:17:40.193 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 251 2023-10-10T19:17:40.203 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 251, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 251, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 3 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 4 ], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:rejoin", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 125, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:17:39.776275+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [] } 2023-10-10T19:17:42.204 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:42.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.328+0000 7f168b6c4700 1 Processor -- start 2023-10-10T19:17:42.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.328+0000 7f168b6c4700 1 -- start start 2023-10-10T19:17:42.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f168b6c4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840b1fc0 0x7f1684155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f168b6c4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f16840b2390 0x7f1684068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f168b6c4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16841574b0 con 0x7f1684068bd0 2023-10-10T19:17:42.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f168b6c4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1684150770 con 0x7f16840b2390 2023-10-10T19:17:42.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f168b6c4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16840afad0 con 0x7f16840b1fc0 2023-10-10T19:17:42.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f1689c61700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1684068bd0 0x7f16840aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:34808/0 (socket says 172.21.15.175:34808) 2023-10-10T19:17:42.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f1689c61700 1 -- 172.21.15.175:0/892330968 learned_addr learned my addr 172.21.15.175:0/892330968 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:17:42.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f1689460700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840b1fc0 0x7f1684155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f1688c5f700 1 --2- 172.21.15.175:0/892330968 >> v2:172.21.15.191:3300/0 conn(0x7f16840b2390 0x7f1684068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631718534 0 0) 0x7f16841574b0 con 0x7f1684068bd0 2023-10-10T19:17:42.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1670003570 con 0x7f1684068bd0 2023-10-10T19:17:42.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2141691988 0 0) 0x7f1670003570 con 0x7f1684068bd0 2023-10-10T19:17:42.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840b1fc0 msgr2=0x7f1684155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:42.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840b1fc0 0x7f1684155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 >> v2:172.21.15.191:3300/0 conn(0x7f16840b2390 msgr2=0x7f1684068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:42.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 --2- 172.21.15.175:0/892330968 >> v2:172.21.15.191:3300/0 conn(0x7f16840b2390 0x7f1684068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840af4c0 con 0x7f1684068bd0 2023-10-10T19:17:42.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f167c0017f0 con 0x7f1684068bd0 2023-10-10T19:17:42.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 >> v1:172.21.15.175:6789/0 conn(0x7f1684068bd0 legacy=0x7f16840aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:42.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f1670003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840b1fc0 0x7f1670004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1670004d50 0x7f16700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1684150770 con 0x7f1670003a60 2023-10-10T19:17:42.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840afad0 con 0x7f1670004d50 2023-10-10T19:17:42.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167000ba60 con 0x7f16840b1fc0 2023-10-10T19:17:42.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.329+0000 7f1689460700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f1670003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f1688c5f700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840b1fc0 0x7f1670004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f1689c61700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1670004d50 0x7f16700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f1689460700 1 -- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840b1fc0 msgr2=0x7f1670004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:42.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f1689460700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840b1fc0 0x7f1670004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f1689460700 1 -- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1670004d50 msgr2=0x7f16700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:42.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f1689460700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1670004d50 0x7f16700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f1689460700 1 -- 172.21.15.175:0/892330968 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1670003570 con 0x7f1670003a60 2023-10-10T19:17:42.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f1689460700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f1670003e70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1680009c00 tx=0x7f16800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc884372ba0bbed server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:42.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168000e6b0 con 0x7f1670003a60 2023-10-10T19:17:42.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168000e810 con 0x7f1670003a60 2023-10-10T19:17:42.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.330+0000 7f167bfff700 1 -- 172.21.15.175:0/892330968 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168000ebd0 con 0x7f1670003a60 2023-10-10T19:17:42.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.331+0000 7f168b6c4700 1 -- 172.21.15.175:0/892330968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 msgr2=0x7f1670003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:42.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.331+0000 7f168b6c4700 1 --2- 172.21.15.175:0/892330968 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f1670003e70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1680009c00 tx=0x7f16800091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.331+0000 7f1689c61700 1 -- 172.21.15.175:0/892330968 reap_dead start 2023-10-10T19:17:42.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.331+0000 7f168b6c4700 1 -- 172.21.15.175:0/892330968 shutdown_connections 2023-10-10T19:17:42.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.331+0000 7f168b6c4700 1 -- 172.21.15.175:0/892330968 >> 172.21.15.175:0/892330968 conn(0x7f1684063230 msgr2=0x7f1684063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:42.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.331+0000 7f168b6c4700 1 -- 172.21.15.175:0/892330968 shutdown_connections 2023-10-10T19:17:42.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.331+0000 7f168b6c4700 1 -- 172.21.15.175:0/892330968 wait complete. 2023-10-10T19:17:42.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.332+0000 7f168b6c4700 1 Processor -- start 2023-10-10T19:17:42.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.332+0000 7f168b6c4700 1 -- start start 2023-10-10T19:17:42.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f168b6c4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f16840cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f168b6c4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840cd170 0x7f16840c1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f168b6c4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16840caac0 0x7f16840cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f168b6c4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840ca630 con 0x7f1670003a60 2023-10-10T19:17:42.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f168b6c4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840c17d0 con 0x7f16840caac0 2023-10-10T19:17:42.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f168b6c4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168416a0a0 con 0x7f16840cd170 2023-10-10T19:17:42.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f16840cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f16840cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58442/0 (socket says 172.21.15.175:58442) 2023-10-10T19:17:42.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 -- 172.21.15.175:0/3870236433 learned_addr learned my addr 172.21.15.175:0/3870236433 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:42.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689c61700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16840caac0 0x7f16840cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1688c5f700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840cd170 0x7f16840c1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 -- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840cd170 msgr2=0x7f16840c1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:42.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840cd170 0x7f16840c1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 -- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16840caac0 msgr2=0x7f16840cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:42.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16840caac0 0x7f16840cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 -- 172.21.15.175:0/3870236433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1684159e50 con 0x7f1670003a60 2023-10-10T19:17:42.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f1689460700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f16840cbb20 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f168000e4b0 tx=0x7f168001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:42.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f16797fa700 1 -- 172.21.15.175:0/3870236433 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680022070 con 0x7f1670003a60 2023-10-10T19:17:42.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f16797fa700 1 -- 172.21.15.175:0/3870236433 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168000e980 con 0x7f1670003a60 2023-10-10T19:17:42.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168415a130 con 0x7f1670003a60 2023-10-10T19:17:42.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f16797fa700 1 -- 172.21.15.175:0/3870236433 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680027470 con 0x7f1670003a60 2023-10-10T19:17:42.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.333+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1684157180 con 0x7f1670003a60 2023-10-10T19:17:42.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.334+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1684159cb0 con 0x7f1670003a60 2023-10-10T19:17:42.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.335+0000 7f16797fa700 1 -- 172.21.15.175:0/3870236433 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f16800278a0 con 0x7f1670003a60 2023-10-10T19:17:42.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.335+0000 7f16797fa700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1668066f60 0x7f1668069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:42.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.335+0000 7f16797fa700 1 -- 172.21.15.175:0/3870236433 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f16800864f0 con 0x7f1670003a60 2023-10-10T19:17:42.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.335+0000 7f1688c5f700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1668066f60 0x7f1668069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:42.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.336+0000 7f1688c5f700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1668066f60 0x7f1668069410 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f168415a410 tx=0x7f16740091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:42.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.339+0000 7f16797fa700 1 -- 172.21.15.175:0/3870236433 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1684159cb0 con 0x7f1670003a60 2023-10-10T19:17:42.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.564+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1684159cb0 con 0x7f1670003a60 2023-10-10T19:17:42.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.564+0000 7f16797fa700 1 -- 172.21.15.175:0/3870236433 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) v1 ==== 95+0+5452 (secure 0 0 0) 0x7f1684159cb0 con 0x7f1670003a60 2023-10-10T19:17:42.565 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:42.565 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":251,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":251,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:39.776275+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:42.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.566+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1668066f60 msgr2=0x7f1668069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:42.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.566+0000 7f168b6c4700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1668066f60 0x7f1668069410 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f168415a410 tx=0x7f16740091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.566+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 msgr2=0x7f16840cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:42.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.566+0000 7f168b6c4700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f16840cbb20 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f168000e4b0 tx=0x7f168001a000 comp rx=0 tx=0).stop 2023-10-10T19:17:42.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.567+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 shutdown_connections 2023-10-10T19:17:42.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.567+0000 7f168b6c4700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1668066f60 0x7f1668069410 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.567+0000 7f168b6c4700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16840caac0 0x7f16840cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.567+0000 7f168b6c4700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1670003a60 0x7f16840cbb20 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.567+0000 7f168b6c4700 1 --2- 172.21.15.175:0/3870236433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16840cd170 0x7f16840c1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:42.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.567+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 >> 172.21.15.175:0/3870236433 conn(0x7f1684063230 msgr2=0x7f168415c800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:42.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.567+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 shutdown_connections 2023-10-10T19:17:42.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:42.567+0000 7f168b6c4700 1 -- 172.21.15.175:0/3870236433 wait complete. 2023-10-10T19:17:42.570 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 251 2023-10-10T19:17:42.579 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:44.580 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:44.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.708+0000 7fd3d95b0700 1 Processor -- start 2023-10-10T19:17:44.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.708+0000 7fd3d95b0700 1 -- start start 2023-10-10T19:17:44.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.708+0000 7fd3d95b0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d40b1fc0 0x7fd3d4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:44.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.708+0000 7fd3d95b0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd3d40b2390 0x7fd3d4068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:44.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.708+0000 7fd3d95b0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3d4068bd0 0x7fd3d40aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:44.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.708+0000 7fd3d95b0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d4150770 con 0x7fd3d40b2390 2023-10-10T19:17:44.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.708+0000 7fd3d95b0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d41574b0 con 0x7fd3d4068bd0 2023-10-10T19:17:44.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.708+0000 7fd3d95b0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d40afad0 con 0x7fd3d40b1fc0 2023-10-10T19:17:44.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd3d40b2390 0x7fd3d4068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:44.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd3d40b2390 0x7fd3d4068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55928/0 (socket says 172.21.15.175:55928) 2023-10-10T19:17:44.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 -- 172.21.15.175:0/4010324039 learned_addr learned my addr 172.21.15.175:0/4010324039 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:44.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 -- 172.21.15.175:0/4010324039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d40b1fc0 msgr2=0x7fd3d4155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:44.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3d2ffd700 1 --2- 172.21.15.175:0/4010324039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d40b1fc0 0x7fd3d4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:44.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3d37fe700 1 --2- 172.21.15.175:0/4010324039 >> v2:172.21.15.191:3300/0 conn(0x7fd3d4068bd0 0x7fd3d40aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:44.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 --2- 172.21.15.175:0/4010324039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d40b1fc0 0x7fd3d4155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 -- 172.21.15.175:0/4010324039 >> v2:172.21.15.191:3300/0 conn(0x7fd3d4068bd0 msgr2=0x7fd3d40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:44.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 --2- 172.21.15.175:0/4010324039 >> v2:172.21.15.191:3300/0 conn(0x7fd3d4068bd0 0x7fd3d40aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 -- 172.21.15.175:0/4010324039 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d40af4c0 con 0x7fd3d40b2390 2023-10-10T19:17:44.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3d37fe700 1 --2- 172.21.15.175:0/4010324039 >> v2:172.21.15.191:3300/0 conn(0x7fd3d4068bd0 0x7fd3d40aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:17:44.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3cbfff700 1 --2- 172.21.15.175:0/4010324039 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d4068690 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40afa40 tx=0x7fd3bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9540985e32a6cb75 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:44.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3d27fc700 1 -- 172.21.15.175:0/4010324039 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00b8f0 con 0x7fd3d40b2390 2023-10-10T19:17:44.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3d27fc700 1 -- 172.21.15.175:0/4010324039 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3bc00ba50 con 0x7fd3d40b2390 2023-10-10T19:17:44.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.709+0000 7fd3d27fc700 1 -- 172.21.15.175:0/4010324039 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00bdf0 con 0x7fd3d40b2390 2023-10-10T19:17:44.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 -- 172.21.15.175:0/4010324039 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 msgr2=0x7fd3d4068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:44.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/4010324039 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d4068690 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40afa40 tx=0x7fd3bc009250 comp rx=0 tx=0).stop 2023-10-10T19:17:44.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 -- 172.21.15.175:0/4010324039 shutdown_connections 2023-10-10T19:17:44.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/4010324039 >> v2:172.21.15.191:3300/0 conn(0x7fd3d4068bd0 0x7fd3d40aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/4010324039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d40b1fc0 0x7fd3d4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/4010324039 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d4068690 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 -- 172.21.15.175:0/4010324039 >> 172.21.15.175:0/4010324039 conn(0x7fd3d4063230 msgr2=0x7fd3d4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:44.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 -- 172.21.15.175:0/4010324039 shutdown_connections 2023-10-10T19:17:44.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.710+0000 7fd3d95b0700 1 -- 172.21.15.175:0/4010324039 wait complete. 2023-10-10T19:17:44.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.711+0000 7fd3d95b0700 1 Processor -- start 2023-10-10T19:17:44.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.711+0000 7fd3d95b0700 1 -- start start 2023-10-10T19:17:44.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.711+0000 7fd3d95b0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d4068bd0 0x7fd3d40c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:44.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d95b0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3d40b1fc0 0x7fd3d4169270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:44.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d95b0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d414ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:44.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d95b0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d40c1e80 con 0x7fd3d40b2390 2023-10-10T19:17:44.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d95b0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d40c3230 con 0x7fd3d40b1fc0 2023-10-10T19:17:44.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d95b0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d4169c60 con 0x7fd3d4068bd0 2023-10-10T19:17:44.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d414ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:44.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d414ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55940/0 (socket says 172.21.15.175:55940) 2023-10-10T19:17:44.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 -- 172.21.15.175:0/608647233 learned_addr learned my addr 172.21.15.175:0/608647233 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:44.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3cbfff700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3d40b1fc0 0x7fd3d4169270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:44.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d2ffd700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d4068bd0 0x7fd3d40c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:44.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 -- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d4068bd0 msgr2=0x7fd3d40c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:44.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d4068bd0 0x7fd3d40c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 -- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3d40b1fc0 msgr2=0x7fd3d4169270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:44.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3d40b1fc0 0x7fd3d4169270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 -- 172.21.15.175:0/608647233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc0094f0 con 0x7fd3d40b2390 2023-10-10T19:17:44.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d37fe700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d414ea80 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00cfc0 tx=0x7fd3cc0078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:44.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3caffd700 1 -- 172.21.15.175:0/608647233 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc00ae80 con 0x7fd3d40b2390 2023-10-10T19:17:44.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3caffd700 1 -- 172.21.15.175:0/608647233 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3cc00d650 con 0x7fd3d40b2390 2023-10-10T19:17:44.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d4153e10 con 0x7fd3d40b2390 2023-10-10T19:17:44.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3caffd700 1 -- 172.21.15.175:0/608647233 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc00d9f0 con 0x7fd3d40b2390 2023-10-10T19:17:44.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.712+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d40c04c0 con 0x7fd3d40b2390 2023-10-10T19:17:44.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.713+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d4154460 con 0x7fd3d40b2390 2023-10-10T19:17:44.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.714+0000 7fd3caffd700 1 -- 172.21.15.175:0/608647233 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd3cc01e3b0 con 0x7fd3d40b2390 2023-10-10T19:17:44.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.714+0000 7fd3caffd700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd3b4066f60 0x7fd3b4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:44.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.714+0000 7fd3caffd700 1 -- 172.21.15.175:0/608647233 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fd3cc0855e0 con 0x7fd3d40b2390 2023-10-10T19:17:44.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.714+0000 7fd3d2ffd700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd3b4066f60 0x7fd3b4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:44.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.715+0000 7fd3d2ffd700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd3b4066f60 0x7fd3b4069410 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400ab30 tx=0x7fd3c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:44.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.719+0000 7fd3caffd700 1 -- 172.21.15.175:0/608647233 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd3cc052220 con 0x7fd3d40b2390 2023-10-10T19:17:44.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.945+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3d4153c70 con 0x7fd3d40b2390 2023-10-10T19:17:44.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.946+0000 7fd3caffd700 1 -- 172.21.15.175:0/608647233 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+5454 (secure 0 0 0) 0x7fd3d4153c70 con 0x7fd3d40b2390 2023-10-10T19:17:44.947 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:44.947 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:44.794088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:clientreplay","state_seq":39,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:44.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd3b4066f60 msgr2=0x7fd3b4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:44.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd3b4066f60 0x7fd3b4069410 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400ab30 tx=0x7fd3c4008040 comp rx=0 tx=0).stop 2023-10-10T19:17:44.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 msgr2=0x7fd3d414ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:44.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d414ea80 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00cfc0 tx=0x7fd3cc0078d0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 shutdown_connections 2023-10-10T19:17:44.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd3b4066f60 0x7fd3b4069410 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40b2390 0x7fd3d414ea80 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3d4068bd0 0x7fd3d40c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 --2- 172.21.15.175:0/608647233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3d40b1fc0 0x7fd3d4169270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:44.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 >> 172.21.15.175:0/608647233 conn(0x7fd3d4063230 msgr2=0x7fd3d4158f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:44.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 shutdown_connections 2023-10-10T19:17:44.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:44.948+0000 7fd3d95b0700 1 -- 172.21.15.175:0/608647233 wait complete. 2023-10-10T19:17:44.951 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 253 2023-10-10T19:17:44.960 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:46.961 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:47.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.087+0000 7fc9d7543700 1 Processor -- start 2023-10-10T19:17:47.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.087+0000 7fc9d7543700 1 -- start start 2023-10-10T19:17:47.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d7543700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 0x7fc9d0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:47.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d7543700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc9d00aff60 0x7fc9d00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:47.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d7543700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d01574b0 con 0x7fc9d00b1fc0 2023-10-10T19:17:47.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d7543700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d0150770 con 0x7fc9d00aff60 2023-10-10T19:17:47.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d7543700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00afad0 con 0x7fc9d00b2390 2023-10-10T19:17:47.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d5ae0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc9d00aff60 0x7fc9d00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:47.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d5ae0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc9d00aff60 0x7fc9d00b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55956/0 (socket says 172.21.15.175:55956) 2023-10-10T19:17:47.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d5ae0700 1 -- 172.21.15.175:0/3556410008 learned_addr learned my addr 172.21.15.175:0/3556410008 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:47.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d4ade700 1 --2- 172.21.15.175:0/3556410008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 0x7fc9d0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:47.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d5ae0700 1 -- 172.21.15.175:0/3556410008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 msgr2=0x7fc9d0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:47.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d5ae0700 1 --2- 172.21.15.175:0/3556410008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 0x7fc9d0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.088+0000 7fc9d5ae0700 1 -- 172.21.15.175:0/3556410008 >> v1:172.21.15.191:6789/0 conn(0x7fc9d00b1fc0 legacy=0x7fc9d0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:47.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d5ae0700 1 -- 172.21.15.175:0/3556410008 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d00af4c0 con 0x7fc9d00aff60 2023-10-10T19:17:47.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d5ae0700 1 --2- 172.21.15.175:0/3556410008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d00b03e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80099e0 tx=0x7fc9c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48073574827161a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:47.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9c7fff700 1 -- 172.21.15.175:0/3556410008 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c800b8b0 con 0x7fc9d00aff60 2023-10-10T19:17:47.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9c7fff700 1 -- 172.21.15.175:0/3556410008 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c800ba10 con 0x7fc9d00aff60 2023-10-10T19:17:47.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9c7fff700 1 -- 172.21.15.175:0/3556410008 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c800bdb0 con 0x7fc9d00aff60 2023-10-10T19:17:47.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d7543700 1 -- 172.21.15.175:0/3556410008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 msgr2=0x7fc9d00b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:47.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3556410008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d00b03e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80099e0 tx=0x7fc9c8009250 comp rx=0 tx=0).stop 2023-10-10T19:17:47.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d7543700 1 -- 172.21.15.175:0/3556410008 shutdown_connections 2023-10-10T19:17:47.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3556410008 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d00b03e0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3556410008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 0x7fc9d0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d7543700 1 -- 172.21.15.175:0/3556410008 >> 172.21.15.175:0/3556410008 conn(0x7fc9d0063230 msgr2=0x7fc9d0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:47.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d7543700 1 -- 172.21.15.175:0/3556410008 shutdown_connections 2023-10-10T19:17:47.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.089+0000 7fc9d7543700 1 -- 172.21.15.175:0/3556410008 wait complete. 2023-10-10T19:17:47.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.090+0000 7fc9d7543700 1 Processor -- start 2023-10-10T19:17:47.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.091+0000 7fc9d7543700 1 -- start start 2023-10-10T19:17:47.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.091+0000 7fc9d7543700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d0156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:47.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.091+0000 7fc9d7543700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9d00b1fc0 0x7fc9d00cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:47.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.091+0000 7fc9d7543700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 0x7fc9d00cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:47.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.091+0000 7fc9d7543700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00c4150 con 0x7fc9d00aff60 2023-10-10T19:17:47.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.091+0000 7fc9d7543700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00c5d00 con 0x7fc9d00b1fc0 2023-10-10T19:17:47.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.091+0000 7fc9d7543700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00cb6a0 con 0x7fc9d00b2390 2023-10-10T19:17:47.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.091+0000 7fc9d52df700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d0156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:47.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d52df700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d0156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55960/0 (socket says 172.21.15.175:55960) 2023-10-10T19:17:47.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d52df700 1 -- 172.21.15.175:0/3571788100 learned_addr learned my addr 172.21.15.175:0/3571788100 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:47.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d4ade700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9d00b1fc0 0x7fc9d00cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:47.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d5ae0700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 0x7fc9d00cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:47.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d52df700 1 -- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 msgr2=0x7fc9d00cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:47.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d52df700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 0x7fc9d00cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d52df700 1 -- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9d00b1fc0 msgr2=0x7fc9d00cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:47.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d52df700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9d00b1fc0 0x7fc9d00cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d52df700 1 -- 172.21.15.175:0/3571788100 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c80094f0 con 0x7fc9d00aff60 2023-10-10T19:17:47.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d4ade700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9d00b1fc0 0x7fc9d00cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:17:47.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d52df700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d0156650 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00cb10 tx=0x7fc9cc0017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:47.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9c57fa700 1 -- 172.21.15.175:0/3571788100 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc00a7f0 con 0x7fc9d00aff60 2023-10-10T19:17:47.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9c57fa700 1 -- 172.21.15.175:0/3571788100 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9cc00a970 con 0x7fc9d00aff60 2023-10-10T19:17:47.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d00c6ab0 con 0x7fc9d00aff60 2023-10-10T19:17:47.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9c57fa700 1 -- 172.21.15.175:0/3571788100 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc00ad30 con 0x7fc9d00aff60 2023-10-10T19:17:47.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.092+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d00c64f0 con 0x7fc9d00aff60 2023-10-10T19:17:47.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.093+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d0067570 con 0x7fc9d00aff60 2023-10-10T19:17:47.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.093+0000 7fc9c57fa700 1 -- 172.21.15.175:0/3571788100 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc9cc003030 con 0x7fc9d00aff60 2023-10-10T19:17:47.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.094+0000 7fc9c57fa700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9b4066f30 0x7fc9b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:47.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.094+0000 7fc9c57fa700 1 -- 172.21.15.175:0/3571788100 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fc9cc084c90 con 0x7fc9d00aff60 2023-10-10T19:17:47.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.094+0000 7fc9d4ade700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9b4066f30 0x7fc9b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:47.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.094+0000 7fc9d4ade700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9b4066f30 0x7fc9b40693e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc9d01510e0 tx=0x7fc9c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:47.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.098+0000 7fc9c57fa700 1 -- 172.21.15.175:0/3571788100 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9cc0518d0 con 0x7fc9d00aff60 2023-10-10T19:17:47.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.323+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9d0068da0 con 0x7fc9d00aff60 2023-10-10T19:17:47.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.323+0000 7fc9c57fa700 1 -- 172.21.15.175:0/3571788100 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+5454 (secure 0 0 0) 0x7fc9cc054ef0 con 0x7fc9d00aff60 2023-10-10T19:17:47.324 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:47.324 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:44.794088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:clientreplay","state_seq":39,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:47.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9b4066f30 msgr2=0x7fc9b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:47.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9b4066f30 0x7fc9b40693e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc9d01510e0 tx=0x7fc9c0008040 comp rx=0 tx=0).stop 2023-10-10T19:17:47.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 msgr2=0x7fc9d0156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:47.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d0156650 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00cb10 tx=0x7fc9cc0017b0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 shutdown_connections 2023-10-10T19:17:47.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc9b4066f30 0x7fc9b40693e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9d00b2390 0x7fc9d00cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc9d00aff60 0x7fc9d0156650 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 --2- 172.21.15.175:0/3571788100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9d00b1fc0 0x7fc9d00cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:47.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 >> 172.21.15.175:0/3571788100 conn(0x7fc9d0063230 msgr2=0x7fc9d0159060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:47.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 shutdown_connections 2023-10-10T19:17:47.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:47.326+0000 7fc9d7543700 1 -- 172.21.15.175:0/3571788100 wait complete. 2023-10-10T19:17:47.329 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 253 2023-10-10T19:17:47.338 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:49.339 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:49.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.453+0000 7f60c0f4f700 1 Processor -- start 2023-10-10T19:17:49.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.453+0000 7f60c0f4f700 1 -- start start 2023-10-10T19:17:49.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.454+0000 7f60c0f4f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f60bc06d6e0 0x7f60bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:49.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.454+0000 7f60c0f4f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 0x7f60bc06e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:49.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.454+0000 7f60c0f4f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60bc0677f0 con 0x7f60bc1500b0 2023-10-10T19:17:49.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.454+0000 7f60c0f4f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc159590 con 0x7f60bc06d6e0 2023-10-10T19:17:49.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.454+0000 7f60c0f4f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0c4250 con 0x7f60bc150480 2023-10-10T19:17:49.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b9d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f60bc06d6e0 0x7f60bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:49.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b9d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f60bc06d6e0 0x7f60bc06dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55968/0 (socket says 172.21.15.175:55968) 2023-10-10T19:17:49.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b9d9b700 1 -- 172.21.15.175:0/3669407537 learned_addr learned my addr 172.21.15.175:0/3669407537 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:49.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60bad9d700 1 --2- 172.21.15.175:0/3669407537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 0x7f60bc06e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:49.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b9d9b700 1 -- 172.21.15.175:0/3669407537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 msgr2=0x7f60bc06e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:49.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b9d9b700 1 --2- 172.21.15.175:0/3669407537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 0x7f60bc06e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b9d9b700 1 -- 172.21.15.175:0/3669407537 >> v1:172.21.15.191:6789/0 conn(0x7f60bc1500b0 legacy=0x7f60bc0c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:49.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b9d9b700 1 -- 172.21.15.175:0/3669407537 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc063760 con 0x7f60bc06d6e0 2023-10-10T19:17:49.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b959a700 1 -- 172.21.15.175:0/3669407537 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614796031 0 0) 0x7f60bc0677f0 con 0x7f60bc1500b0 2023-10-10T19:17:49.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b9d9b700 1 --2- 172.21.15.175:0/3669407537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc06d6e0 0x7f60bc06dcd0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f60a40099b0 tx=0x7f60a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb7201c4260ac81 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:49.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b959a700 1 -- 172.21.15.175:0/3669407537 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4009ea0 con 0x7f60bc06d6e0 2023-10-10T19:17:49.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b959a700 1 -- 172.21.15.175:0/3669407537 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a400b8f0 con 0x7f60bc06d6e0 2023-10-10T19:17:49.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.455+0000 7f60b959a700 1 -- 172.21.15.175:0/3669407537 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a400bc90 con 0x7f60bc06d6e0 2023-10-10T19:17:49.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.456+0000 7f60c0f4f700 1 -- 172.21.15.175:0/3669407537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc06d6e0 msgr2=0x7f60bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:49.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.456+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/3669407537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc06d6e0 0x7f60bc06dcd0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f60a40099b0 tx=0x7f60a4009250 comp rx=0 tx=0).stop 2023-10-10T19:17:49.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.456+0000 7f60c0f4f700 1 -- 172.21.15.175:0/3669407537 shutdown_connections 2023-10-10T19:17:49.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.456+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/3669407537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 0x7f60bc06e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.456+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/3669407537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc06d6e0 0x7f60bc06dcd0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.456+0000 7f60c0f4f700 1 -- 172.21.15.175:0/3669407537 >> 172.21.15.175:0/3669407537 conn(0x7f60bc0c5650 msgr2=0x7f60bc0c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:49.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.456+0000 7f60c0f4f700 1 -- 172.21.15.175:0/3669407537 shutdown_connections 2023-10-10T19:17:49.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.456+0000 7f60c0f4f700 1 -- 172.21.15.175:0/3669407537 wait complete. 2023-10-10T19:17:49.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.457+0000 7f60c0f4f700 1 Processor -- start 2023-10-10T19:17:49.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.457+0000 7f60c0f4f700 1 -- start start 2023-10-10T19:17:49.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60c0f4f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc06d6e0 0x7f60bc155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:49.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60c0f4f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc1500b0 0x7f60bc0c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:49.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60c0f4f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 0x7f60bc0c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:49.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60c0f4f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc154720 con 0x7f60bc1500b0 2023-10-10T19:17:49.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60c0f4f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0c0e20 con 0x7f60bc06d6e0 2023-10-10T19:17:49.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60c0f4f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc156cd0 con 0x7f60bc150480 2023-10-10T19:17:49.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc1500b0 0x7f60bc0c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:49.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc1500b0 0x7f60bc0c1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55984/0 (socket says 172.21.15.175:55984) 2023-10-10T19:17:49.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 -- 172.21.15.175:0/509698852 learned_addr learned my addr 172.21.15.175:0/509698852 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:49.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60ba59c700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc06d6e0 0x7f60bc155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:49.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60bad9d700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 0x7f60bc0c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 -- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 msgr2=0x7f60bc0c17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 0x7f60bc0c17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 -- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc06d6e0 msgr2=0x7f60bc155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc06d6e0 0x7f60bc155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 -- 172.21.15.175:0/509698852 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a40094f0 con 0x7f60bc1500b0 2023-10-10T19:17:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b9d9b700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc1500b0 0x7f60bc0c1270 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f60bc067260 tx=0x7f60a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b27fc700 1 -- 172.21.15.175:0/509698852 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a400bc70 con 0x7f60bc1500b0 2023-10-10T19:17:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b27fc700 1 -- 172.21.15.175:0/509698852 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a4011420 con 0x7f60bc1500b0 2023-10-10T19:17:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc156f70 con 0x7f60bc1500b0 2023-10-10T19:17:49.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60b27fc700 1 -- 172.21.15.175:0/509698852 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a40117c0 con 0x7f60bc1500b0 2023-10-10T19:17:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.458+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc166d60 con 0x7f60bc1500b0 2023-10-10T19:17:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.459+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc0c0f90 con 0x7f60bc1500b0 2023-10-10T19:17:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.460+0000 7f60b27fc700 1 -- 172.21.15.175:0/509698852 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f60a4011b70 con 0x7f60bc1500b0 2023-10-10T19:17:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.460+0000 7f60b27fc700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f609c066f60 0x7f609c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.460+0000 7f60b27fc700 1 -- 172.21.15.175:0/509698852 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f60a4085e50 con 0x7f60bc1500b0 2023-10-10T19:17:49.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.460+0000 7f60ba59c700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f609c066f60 0x7f609c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:49.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.460+0000 7f60ba59c700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f609c066f60 0x7f609c069410 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f60ac001560 tx=0x7f60ac0016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:49.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.464+0000 7f60b27fc700 1 -- 172.21.15.175:0/509698852 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f60bc0c0f90 con 0x7f60bc1500b0 2023-10-10T19:17:49.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.690+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60bc1558b0 con 0x7f60bc1500b0 2023-10-10T19:17:49.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.690+0000 7f60b27fc700 1 -- 172.21.15.175:0/509698852 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+5454 (secure 0 0 0) 0x7f60a4015090 con 0x7f60bc1500b0 2023-10-10T19:17:49.691 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:49.691 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:44.794088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:clientreplay","state_seq":39,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:49.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f609c066f60 msgr2=0x7f609c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:49.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f609c066f60 0x7f609c069410 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f60ac001560 tx=0x7f60ac0016f0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc1500b0 msgr2=0x7f60bc0c1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:49.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc1500b0 0x7f60bc0c1270 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f60bc067260 tx=0x7f60a4017000 comp rx=0 tx=0).stop 2023-10-10T19:17:49.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 shutdown_connections 2023-10-10T19:17:49.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f609c066f60 0x7f609c069410 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f60bc150480 0x7f60bc0c17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60bc06d6e0 0x7f60bc155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 --2- 172.21.15.175:0/509698852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60bc1500b0 0x7f60bc0c1270 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:49.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 >> 172.21.15.175:0/509698852 conn(0x7f60bc0c5650 msgr2=0x7f60bc14f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:49.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 shutdown_connections 2023-10-10T19:17:49.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:49.692+0000 7f60c0f4f700 1 -- 172.21.15.175:0/509698852 wait complete. 2023-10-10T19:17:49.696 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 253 2023-10-10T19:17:49.704 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:17:50.816 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-10T19:17:50.815+0000 7feff93e2700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:17:51.705 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:17:51.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.833+0000 7fb35f534700 1 Processor -- start 2023-10-10T19:17:51.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.834+0000 7fb35f534700 1 -- start start 2023-10-10T19:17:51.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.834+0000 7fb35f534700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb35806bf00 0x7fb3580c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:51.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.834+0000 7fb35f534700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb358156520 0x7fb3581569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:51.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.834+0000 7fb35f534700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358156f00 con 0x7fb3580c5e40 2023-10-10T19:17:51.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.834+0000 7fb35f534700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3581560d0 con 0x7fb35806bf00 2023-10-10T19:17:51.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.834+0000 7fb35f534700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580c6df0 con 0x7fb358156520 2023-10-10T19:17:51.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35dad1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb358156520 0x7fb3581569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:51.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35cacf700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb35806bf00 0x7fb3580c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:51.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35dad1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb358156520 0x7fb3581569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:43270/0 (socket says 172.21.15.175:43270) 2023-10-10T19:17:51.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35dad1700 1 -- 172.21.15.175:0/37826649 learned_addr learned my addr 172.21.15.175:0/37826649 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:51.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35dad1700 1 -- 172.21.15.175:0/37826649 >> v2:172.21.15.191:3300/0 conn(0x7fb35806bf00 msgr2=0x7fb3580c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:17:51.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35dad1700 1 --2- 172.21.15.175:0/37826649 >> v2:172.21.15.191:3300/0 conn(0x7fb35806bf00 0x7fb3580c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:51.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35dad1700 1 -- 172.21.15.175:0/37826649 >> v1:172.21.15.191:6789/0 conn(0x7fb3580c5e40 legacy=0x7fb3580c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:51.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35dad1700 1 -- 172.21.15.175:0/37826649 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3580c6920 con 0x7fb358156520 2023-10-10T19:17:51.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb34ffff700 1 -- 172.21.15.175:0/37826649 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198346510 0 0) 0x7fb358156f00 con 0x7fb3580c5e40 2023-10-10T19:17:51.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35cacf700 1 --2- 172.21.15.175:0/37826649 >> v2:172.21.15.191:3300/0 conn(0x7fb35806bf00 0x7fb3580c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:17:51.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb35dad1700 1 --2- 172.21.15.175:0/37826649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb358156520 0x7fb3581569c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb358156210 tx=0x7fb350009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da4a9044416431b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:51.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb34ffff700 1 -- 172.21.15.175:0/37826649 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35000e070 con 0x7fb358156520 2023-10-10T19:17:51.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.835+0000 7fb34ffff700 1 -- 172.21.15.175:0/37826649 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35000bab0 con 0x7fb358156520 2023-10-10T19:17:51.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb34ffff700 1 -- 172.21.15.175:0/37826649 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35000be70 con 0x7fb358156520 2023-10-10T19:17:51.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb35f534700 1 -- 172.21.15.175:0/37826649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb358156520 msgr2=0x7fb3581569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:51.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb35f534700 1 --2- 172.21.15.175:0/37826649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb358156520 0x7fb3581569c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb358156210 tx=0x7fb350009840 comp rx=0 tx=0).stop 2023-10-10T19:17:51.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb35f534700 1 -- 172.21.15.175:0/37826649 shutdown_connections 2023-10-10T19:17:51.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb35f534700 1 --2- 172.21.15.175:0/37826649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb358156520 0x7fb3581569c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:51.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb35f534700 1 --2- 172.21.15.175:0/37826649 >> v2:172.21.15.191:3300/0 conn(0x7fb35806bf00 0x7fb3580c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:51.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb35f534700 1 -- 172.21.15.175:0/37826649 >> 172.21.15.175:0/37826649 conn(0x7fb358068900 msgr2=0x7fb358068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:51.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb35f534700 1 -- 172.21.15.175:0/37826649 shutdown_connections 2023-10-10T19:17:51.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.836+0000 7fb35f534700 1 -- 172.21.15.175:0/37826649 wait complete. 2023-10-10T19:17:51.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.837+0000 7fb35f534700 1 Processor -- start 2023-10-10T19:17:51.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.837+0000 7fb35f534700 1 -- start start 2023-10-10T19:17:51.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35f534700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb35806bf00 0x7fb35815ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:51.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35f534700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3580c5e40 0x7fb3580c1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:51.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35f534700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358156520 0x7fb35815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:51.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35f534700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35815b610 con 0x7fb35806bf00 2023-10-10T19:17:51.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35f534700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35814d840 con 0x7fb358156520 2023-10-10T19:17:51.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35f534700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35815b180 con 0x7fb3580c5e40 2023-10-10T19:17:51.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35d2d0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb35806bf00 0x7fb35815ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:51.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35d2d0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb35806bf00 0x7fb35815ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55994/0 (socket says 172.21.15.175:55994) 2023-10-10T19:17:51.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35d2d0700 1 -- 172.21.15.175:0/1341003922 learned_addr learned my addr 172.21.15.175:0/1341003922 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:17:51.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35dad1700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358156520 0x7fb35815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:51.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.838+0000 7fb35cacf700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3580c5e40 0x7fb3580c1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:51.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb35d2d0700 1 -- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3580c5e40 msgr2=0x7fb3580c1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:51.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb35d2d0700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3580c5e40 0x7fb3580c1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:51.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb35d2d0700 1 -- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358156520 msgr2=0x7fb35815baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:51.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb35d2d0700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358156520 0x7fb35815baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:51.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb35d2d0700 1 -- 172.21.15.175:0/1341003922 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb348009420 con 0x7fb35806bf00 2023-10-10T19:17:51.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb35d2d0700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb35806bf00 0x7fb35815ac40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fb35400c810 tx=0x7fb354007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:51.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb34d7fa700 1 -- 172.21.15.175:0/1341003922 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354011590 con 0x7fb35806bf00 2023-10-10T19:17:51.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb34d7fa700 1 -- 172.21.15.175:0/1341003922 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb354011710 con 0x7fb35806bf00 2023-10-10T19:17:51.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb350009ae0 con 0x7fb35806bf00 2023-10-10T19:17:51.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb34d7fa700 1 -- 172.21.15.175:0/1341003922 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354011ad0 con 0x7fb35806bf00 2023-10-10T19:17:51.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.839+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35814cfa0 con 0x7fb35806bf00 2023-10-10T19:17:51.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.840+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb358067570 con 0x7fb35806bf00 2023-10-10T19:17:51.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.840+0000 7fb34d7fa700 1 -- 172.21.15.175:0/1341003922 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb35401b3b0 con 0x7fb35806bf00 2023-10-10T19:17:51.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.841+0000 7fb34d7fa700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb33c066f30 0x7fb33c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:17:51.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.841+0000 7fb34d7fa700 1 -- 172.21.15.175:0/1341003922 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fb354083f80 con 0x7fb35806bf00 2023-10-10T19:17:51.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.841+0000 7fb35cacf700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb33c066f30 0x7fb33c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:17:51.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.842+0000 7fb35cacf700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb33c066f30 0x7fb33c0693e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb3580c1df0 tx=0x7fb3480091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:17:51.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:51.845+0000 7fb34d7fa700 1 -- 172.21.15.175:0/1341003922 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb354050c70 con 0x7fb35806bf00 2023-10-10T19:17:52.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.071+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb35806c690 con 0x7fb35806bf00 2023-10-10T19:17:52.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.072+0000 7fb34d7fa700 1 -- 172.21.15.175:0/1341003922 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 254 v254) v1 ==== 95+0+5448 (secure 0 0 0) 0x7fb354017070 con 0x7fb35806bf00 2023-10-10T19:17:52.074 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:17:52.074 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":254,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":254,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:49.813710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:17:52.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb33c066f30 msgr2=0x7fb33c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:52.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb33c066f30 0x7fb33c0693e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb3580c1df0 tx=0x7fb3480091a0 comp rx=0 tx=0).stop 2023-10-10T19:17:52.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb35806bf00 msgr2=0x7fb35815ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:17:52.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb35806bf00 0x7fb35815ac40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fb35400c810 tx=0x7fb354007dd0 comp rx=0 tx=0).stop 2023-10-10T19:17:52.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 shutdown_connections 2023-10-10T19:17:52.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb33c066f30 0x7fb33c0693e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:52.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb358156520 0x7fb35815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:52.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb35806bf00 0x7fb35815ac40 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:52.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 --2- 172.21.15.175:0/1341003922 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3580c5e40 0x7fb3580c1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:17:52.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 >> 172.21.15.175:0/1341003922 conn(0x7fb358068900 msgr2=0x7fb3580c3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:17:52.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 shutdown_connections 2023-10-10T19:17:52.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:17:52.074+0000 7fb35f534700 1 -- 172.21.15.175:0/1341003922 wait complete. 2023-10-10T19:17:52.078 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 254 2023-10-10T19:17:52.087 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=6080 2023-10-10T19:17:52.087 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.b 2023-10-10T19:17:54.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:17:54.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:17:54.890 INFO:tasks.workunit.client.0.smithi175.stderr:tar: coreutils-8.5/man/stat.x: Cannot open: Input/output error 2023-10-10T19:17:54.890 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sleep.x 2023-10-10T19:18:00.089 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-10T19:18:00.089 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-10T19:18:00.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-10T19:18:00.092 INFO:tasks.ceph.mds.b:Started 2023-10-10T19:18:00.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:18:00.173 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-10T19:18:00.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.207+0000 7fe56bdfc700 1 Processor -- start 2023-10-10T19:18:00.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.207+0000 7fe56bdfc700 1 -- start start 2023-10-10T19:18:00.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe56bdfc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe56414e090 0x7fe564153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:00.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe56bdfc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe564063640 0x7fe5640c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:00.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe56bdfc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe5640c2d20 0x7fe5640c2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:00.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe56bdfc700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5640c5200 con 0x7fe564063640 2023-10-10T19:18:00.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe56bdfc700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564157b10 con 0x7fe5640c2d20 2023-10-10T19:18:00.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe56bdfc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564068ab0 con 0x7fe56414e090 2023-10-10T19:18:00.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe569397700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe564063640 0x7fe5640c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:00.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe569397700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe564063640 0x7fe5640c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39552/0 (socket says 172.21.15.175:39552) 2023-10-10T19:18:00.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.208+0000 7fe569397700 1 -- 172.21.15.175:0/2358747157 learned_addr learned my addr 172.21.15.175:0/2358747157 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:18:00.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe56a399700 1 --2- 172.21.15.175:0/2358747157 >> v2:172.21.15.191:3300/0 conn(0x7fe5640c2d20 0x7fe5640c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:00.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe569b98700 1 --2- 172.21.15.175:0/2358747157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe56414e090 0x7fe564153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:00.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe569397700 1 -- 172.21.15.175:0/2358747157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe56414e090 msgr2=0x7fe564153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:18:00.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe569397700 1 --2- 172.21.15.175:0/2358747157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe56414e090 0x7fe564153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe569397700 1 -- 172.21.15.175:0/2358747157 >> v2:172.21.15.191:3300/0 conn(0x7fe5640c2d20 msgr2=0x7fe5640c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:18:00.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe569397700 1 --2- 172.21.15.175:0/2358747157 >> v2:172.21.15.191:3300/0 conn(0x7fe5640c2d20 0x7fe5640c2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe569397700 1 -- 172.21.15.175:0/2358747157 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564068d70 con 0x7fe564063640 2023-10-10T19:18:00.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe569397700 1 --2- 172.21.15.175:0/2358747157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe564063640 0x7fe5640c21d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fe564071830 tx=0x7fe554009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a864db764bd2b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:18:00.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe568b96700 1 -- 172.21.15.175:0/2358747157 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55400b860 con 0x7fe564063640 2023-10-10T19:18:00.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe568b96700 1 -- 172.21.15.175:0/2358747157 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55400b9c0 con 0x7fe564063640 2023-10-10T19:18:00.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe568b96700 1 -- 172.21.15.175:0/2358747157 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55400bd60 con 0x7fe564063640 2023-10-10T19:18:00.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe56bdfc700 1 -- 172.21.15.175:0/2358747157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe564063640 msgr2=0x7fe5640c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:00.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/2358747157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe564063640 0x7fe5640c21d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fe564071830 tx=0x7fe554009250 comp rx=0 tx=0).stop 2023-10-10T19:18:00.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe56bdfc700 1 -- 172.21.15.175:0/2358747157 shutdown_connections 2023-10-10T19:18:00.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/2358747157 >> v2:172.21.15.191:3300/0 conn(0x7fe5640c2d20 0x7fe5640c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/2358747157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe56414e090 0x7fe564153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/2358747157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe564063640 0x7fe5640c21d0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.209+0000 7fe56bdfc700 1 -- 172.21.15.175:0/2358747157 >> 172.21.15.175:0/2358747157 conn(0x7fe5640c5650 msgr2=0x7fe5640c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:18:00.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.210+0000 7fe56bdfc700 1 -- 172.21.15.175:0/2358747157 shutdown_connections 2023-10-10T19:18:00.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.210+0000 7fe56bdfc700 1 -- 172.21.15.175:0/2358747157 wait complete. 2023-10-10T19:18:00.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56bdfc700 1 Processor -- start 2023-10-10T19:18:00.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56bdfc700 1 -- start start 2023-10-10T19:18:00.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56bdfc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe564063640 0x7fe564164190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:00.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56bdfc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5640c2d20 0x7fe564165690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:00.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56bdfc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe56414e090 0x7fe564153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:00.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56bdfc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe564152cd0 con 0x7fe56414e090 2023-10-10T19:18:00.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56bdfc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe564151e20 con 0x7fe564063640 2023-10-10T19:18:00.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56bdfc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5640c8750 con 0x7fe5640c2d20 2023-10-10T19:18:00.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56a399700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe56414e090 0x7fe564153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:00.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe569b98700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe564063640 0x7fe564164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:00.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56a399700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe56414e090 0x7fe564153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39568/0 (socket says 172.21.15.175:39568) 2023-10-10T19:18:00.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe56a399700 1 -- 172.21.15.175:0/17119089 learned_addr learned my addr 172.21.15.175:0/17119089 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:18:00.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.211+0000 7fe569397700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5640c2d20 0x7fe564165690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:00.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe56a399700 1 -- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5640c2d20 msgr2=0x7fe564165690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:00.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe56a399700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5640c2d20 0x7fe564165690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe56a399700 1 -- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe564063640 msgr2=0x7fe564164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:00.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe56a399700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe564063640 0x7fe564164190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe56a399700 1 -- 172.21.15.175:0/17119089 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5540094f0 con 0x7fe56414e090 2023-10-10T19:18:00.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe56a399700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe56414e090 0x7fe564153160 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00d890 tx=0x7fe55c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:18:00.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe559ffb700 1 -- 172.21.15.175:0/17119089 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c0125f0 con 0x7fe56414e090 2023-10-10T19:18:00.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe559ffb700 1 -- 172.21.15.175:0/17119089 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c012750 con 0x7fe56414e090 2023-10-10T19:18:00.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5641520c0 con 0x7fe56414e090 2023-10-10T19:18:00.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe559ffb700 1 -- 172.21.15.175:0/17119089 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c012b10 con 0x7fe56414e090 2023-10-10T19:18:00.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.212+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe564152680 con 0x7fe56414e090 2023-10-10T19:18:00.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.213+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe564152e80 con 0x7fe56414e090 2023-10-10T19:18:00.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.214+0000 7fe559ffb700 1 -- 172.21.15.175:0/17119089 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe55c013480 con 0x7fe56414e090 2023-10-10T19:18:00.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.214+0000 7fe559ffb700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe548066f30 0x7fe5480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:00.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.214+0000 7fe559ffb700 1 -- 172.21.15.175:0/17119089 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fe55c084ee0 con 0x7fe56414e090 2023-10-10T19:18:00.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.214+0000 7fe569b98700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe548066f30 0x7fe5480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:00.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.215+0000 7fe569b98700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe548066f30 0x7fe5480693e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe5600075a0 tx=0x7fe5600097e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:18:00.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.219+0000 7fe559ffb700 1 -- 172.21.15.175:0/17119089 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe564152e80 con 0x7fe56414e090 2023-10-10T19:18:00.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.444+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe564152e80 con 0x7fe56414e090 2023-10-10T19:18:00.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.445+0000 7fe559ffb700 1 -- 172.21.15.175:0/17119089 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 256 v256) v1 ==== 95+0+5445 (secure 0 0 0) 0x7fe564152e80 con 0x7fe56414e090 2023-10-10T19:18:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:18:00.447 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":256,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":256,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:17:59.844959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:18:00.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe548066f30 msgr2=0x7fe5480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:00.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe548066f30 0x7fe5480693e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe5600075a0 tx=0x7fe5600097e0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe56414e090 msgr2=0x7fe564153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:00.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe56414e090 0x7fe564153160 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00d890 tx=0x7fe55c008040 comp rx=0 tx=0).stop 2023-10-10T19:18:00.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 shutdown_connections 2023-10-10T19:18:00.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe548066f30 0x7fe5480693e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe56414e090 0x7fe564153160 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe564063640 0x7fe564164190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 --2- 172.21.15.175:0/17119089 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe5640c2d20 0x7fe564165690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:00.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 >> 172.21.15.175:0/17119089 conn(0x7fe5640c5650 msgr2=0x7fe564154470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:18:00.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 shutdown_connections 2023-10-10T19:18:00.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:00.447+0000 7fe56bdfc700 1 -- 172.21.15.175:0/17119089 wait complete. 2023-10-10T19:18:00.451 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 256 2023-10-10T19:18:00.460 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-10T19:18:02.461 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:18:02.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.588+0000 7f81dafcc700 1 Processor -- start 2023-10-10T19:18:02.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.588+0000 7f81dafcc700 1 -- start start 2023-10-10T19:18:02.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.588+0000 7f81dafcc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d4155fb0 0x7f81d4159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:02.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.588+0000 7f81dafcc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d41574b0 con 0x7f81d40b1fc0 2023-10-10T19:18:02.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.588+0000 7f81dafcc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d40afad0 con 0x7f81d40b2390 2023-10-10T19:18:02.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.588+0000 7f81dafcc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d4150770 con 0x7f81d4155fb0 2023-10-10T19:18:02.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d8d68700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f81d40b1fc0 0x7f81d4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48818/0 (socket says 172.21.15.175:48818) 2023-10-10T19:18:02.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d8d68700 1 -- 172.21.15.175:0/1163008196 learned_addr learned my addr 172.21.15.175:0/1163008196 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:18:02.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d9569700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d4155fb0 0x7f81d4159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:02.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172697742 0 0) 0x7f81d41574b0 con 0x7f81d40b1fc0 2023-10-10T19:18:02.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81bc003570 con 0x7f81d40b1fc0 2023-10-10T19:18:02.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2013526935 0 0) 0x7f81bc003570 con 0x7f81d40b1fc0 2023-10-10T19:18:02.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d4155fb0 msgr2=0x7f81d4159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:18:02.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d4155fb0 0x7f81d4159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 >> v1:172.21.15.191:6789/0 conn(0x7f81d40b2390 legacy=0x7f81d40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:02.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d40af4c0 con 0x7f81d40b1fc0 2023-10-10T19:18:02.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f81c40017f0 con 0x7f81d40b1fc0 2023-10-10T19:18:02.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 >> v1:172.21.15.175:6789/0 conn(0x7f81d40b1fc0 legacy=0x7f81d4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:02.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc003a90 0x7f81bc003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:02.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81bc004810 0x7f81bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:02.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d4155fb0 0x7f81bc0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:02.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d4150770 con 0x7f81bc004810 2023-10-10T19:18:02.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc00bb60 con 0x7f81bc003a90 2023-10-10T19:18:02.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc00bd20 con 0x7f81d4155fb0 2023-10-10T19:18:02.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d3fff700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81bc004810 0x7f81bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:02.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d8d68700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc003a90 0x7f81bc003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:02.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.589+0000 7f81d9569700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d4155fb0 0x7f81bc0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:02.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d3fff700 1 -- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d4155fb0 msgr2=0x7f81bc0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:18:02.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d3fff700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d4155fb0 0x7f81bc0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d3fff700 1 -- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc003a90 msgr2=0x7f81bc003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:18:02.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d3fff700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc003a90 0x7f81bc003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d3fff700 1 -- 172.21.15.175:0/1163008196 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc003570 con 0x7f81bc004810 2023-10-10T19:18:02.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d8d68700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc003a90 0x7f81bc003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:18:02.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d3fff700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81bc004810 0x7f81bc004c80 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f81c80018b0 tx=0x7f81c8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147a3c9b78596e1b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:18:02.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8012420 con 0x7f81bc004810 2023-10-10T19:18:02.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c8012580 con 0x7f81bc004810 2023-10-10T19:18:02.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d37fe700 1 -- 172.21.15.175:0/1163008196 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8012920 con 0x7f81bc004810 2023-10-10T19:18:02.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81dafcc700 1 -- 172.21.15.175:0/1163008196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81bc004810 msgr2=0x7f81bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:02.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81dafcc700 1 --2- 172.21.15.175:0/1163008196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81bc004810 0x7f81bc004c80 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f81c80018b0 tx=0x7f81c8007d30 comp rx=0 tx=0).stop 2023-10-10T19:18:02.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81d9569700 1 -- 172.21.15.175:0/1163008196 reap_dead start 2023-10-10T19:18:02.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81dafcc700 1 -- 172.21.15.175:0/1163008196 shutdown_connections 2023-10-10T19:18:02.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81dafcc700 1 -- 172.21.15.175:0/1163008196 >> 172.21.15.175:0/1163008196 conn(0x7f81d4063230 msgr2=0x7f81d4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:18:02.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81dafcc700 1 -- 172.21.15.175:0/1163008196 shutdown_connections 2023-10-10T19:18:02.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.590+0000 7f81dafcc700 1 -- 172.21.15.175:0/1163008196 wait complete. 2023-10-10T19:18:02.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.591+0000 7f81dafcc700 1 Processor -- start 2023-10-10T19:18:02.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.592+0000 7f81dafcc700 1 -- start start 2023-10-10T19:18:02.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.592+0000 7f81dafcc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc004810 0x7f81d4169af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:02.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.592+0000 7f81dafcc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d414c610 0x7f81d41661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:02.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.592+0000 7f81dafcc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81d414ca80 0x7f81d4166720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:02.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.592+0000 7f81dafcc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d416a030 con 0x7f81d414ca80 2023-10-10T19:18:02.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.592+0000 7f81dafcc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d4165d50 con 0x7f81bc004810 2023-10-10T19:18:02.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.592+0000 7f81dafcc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d4152e50 con 0x7f81d414c610 2023-10-10T19:18:02.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.592+0000 7f81d9569700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81d414ca80 0x7f81d4166720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:02.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d9569700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81d414ca80 0x7f81d4166720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39586/0 (socket says 172.21.15.175:39586) 2023-10-10T19:18:02.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d9569700 1 -- 172.21.15.175:0/736497260 learned_addr learned my addr 172.21.15.175:0/736497260 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:18:02.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d8d68700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc004810 0x7f81d4169af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:02.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d3fff700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d414c610 0x7f81d41661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:02.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d9569700 1 -- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d414c610 msgr2=0x7f81d41661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:02.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d9569700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d414c610 0x7f81d41661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d9569700 1 -- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc004810 msgr2=0x7f81d4169af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:02.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d9569700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc004810 0x7f81d4169af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d9569700 1 -- 172.21.15.175:0/736497260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d41530f0 con 0x7f81d414ca80 2023-10-10T19:18:02.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d9569700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81d414ca80 0x7f81d4166720 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f81cc006f00 tx=0x7f81cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:18:02.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d0ff9700 1 -- 172.21.15.175:0/736497260 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc013750 con 0x7f81d414ca80 2023-10-10T19:18:02.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d0ff9700 1 -- 172.21.15.175:0/736497260 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc0138b0 con 0x7f81d414ca80 2023-10-10T19:18:02.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81d0ff9700 1 -- 172.21.15.175:0/736497260 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc013c50 con 0x7f81d414ca80 2023-10-10T19:18:02.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d41533d0 con 0x7f81d414ca80 2023-10-10T19:18:02.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.593+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d4153990 con 0x7f81d414ca80 2023-10-10T19:18:02.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.594+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d4165f00 con 0x7f81d414ca80 2023-10-10T19:18:02.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.595+0000 7f81d0ff9700 1 -- 172.21.15.175:0/736497260 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f81cc014650 con 0x7f81d414ca80 2023-10-10T19:18:02.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.595+0000 7f81d0ff9700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81c0066f30 0x7f81c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:18:02.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.595+0000 7f81d0ff9700 1 -- 172.21.15.175:0/736497260 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f81cc0851f0 con 0x7f81d414ca80 2023-10-10T19:18:02.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.595+0000 7f81d8d68700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81c0066f30 0x7f81c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:18:02.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.596+0000 7f81d8d68700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81c0066f30 0x7f81c00693e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f81c400a3f0 tx=0x7f81c4006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:18:02.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.599+0000 7f81d0ff9700 1 -- 172.21.15.175:0/736497260 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81d4165f00 con 0x7f81d414ca80 2023-10-10T19:18:02.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.824+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81d4165f00 con 0x7f81d414ca80 2023-10-10T19:18:02.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.824+0000 7f81d0ff9700 1 -- 172.21.15.175:0/736497260 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6201 (secure 0 0 0) 0x7f81d4165f00 con 0x7f81d414ca80 2023-10-10T19:18:02.826 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:18:02.827 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:18:01.861061+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:18:02.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81c0066f30 msgr2=0x7f81c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:02.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81c0066f30 0x7f81c00693e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f81c400a3f0 tx=0x7f81c4006040 comp rx=0 tx=0).stop 2023-10-10T19:18:02.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81d414ca80 msgr2=0x7f81d4166720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:18:02.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81d414ca80 0x7f81d4166720 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f81cc006f00 tx=0x7f81cc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 shutdown_connections 2023-10-10T19:18:02.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81c0066f30 0x7f81c00693e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81d414ca80 0x7f81d4166720 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81bc004810 0x7f81d4169af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 --2- 172.21.15.175:0/736497260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81d414c610 0x7f81d41661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:18:02.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 >> 172.21.15.175:0/736497260 conn(0x7f81d4063230 msgr2=0x7f81d4068690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:18:02.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.826+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 shutdown_connections 2023-10-10T19:18:02.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:18:02.827+0000 7f81dafcc700 1 -- 172.21.15.175:0/736497260 wait complete. 2023-10-10T19:18:02.831 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 258 2023-10-10T19:18:02.840 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-10T19:18:02.840 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-10-10T19:18:24.478 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:18:24.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:18:54.608 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:18:54.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:19:15.841 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:15.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.966+0000 7fced5c42700 1 Processor -- start 2023-10-10T19:19:15.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.966+0000 7fced5c42700 1 -- start start 2023-10-10T19:19:15.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.966+0000 7fced5c42700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fced014fbf0 0x7fced0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:15.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.966+0000 7fced5c42700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:15.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.966+0000 7fced5c42700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced0156800 con 0x7fced00681e0 2023-10-10T19:19:15.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.966+0000 7fced5c42700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced00c59e0 con 0x7fced014fbf0 2023-10-10T19:19:15.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.966+0000 7fced5c42700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced015a2b0 con 0x7fced014ffc0 2023-10-10T19:19:15.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fceceffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:15.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fcecf7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fced014fbf0 0x7fced0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:15.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fcecffff700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fced00681e0 0x7fced00c5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:41274/0 (socket says 172.21.15.175:41274) 2023-10-10T19:19:15.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fcecffff700 1 -- 172.21.15.175:0/1056676945 learned_addr learned my addr 172.21.15.175:0/1056676945 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:15.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fcecf7fe700 1 -- 172.21.15.175:0/1056676945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 msgr2=0x7fced0063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:15.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fcecf7fe700 1 --2- 172.21.15.175:0/1056676945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0063230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:15.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fcecf7fe700 1 -- 172.21.15.175:0/1056676945 >> v1:172.21.15.191:6789/0 conn(0x7fced00681e0 legacy=0x7fced00c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:15.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fcecf7fe700 1 -- 172.21.15.175:0/1056676945 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced0063e30 con 0x7fced014fbf0 2023-10-10T19:19:15.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fceceffd700 1 --2- 172.21.15.175:0/1056676945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0063230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:15.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.967+0000 7fcecf7fe700 1 --2- 172.21.15.175:0/1056676945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced014fbf0 0x7fced0159d70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009ee0 tx=0x7fcec0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63dfb6a55922e962 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:15.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fcece7fc700 1 -- 172.21.15.175:0/1056676945 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec000e070 con 0x7fced014fbf0 2023-10-10T19:19:15.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fcece7fc700 1 -- 172.21.15.175:0/1056676945 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec000bbf0 con 0x7fced014fbf0 2023-10-10T19:19:15.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fcece7fc700 1 -- 172.21.15.175:0/1056676945 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0012430 con 0x7fced014fbf0 2023-10-10T19:19:15.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fced5c42700 1 -- 172.21.15.175:0/1056676945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced014fbf0 msgr2=0x7fced0159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:15.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fced5c42700 1 --2- 172.21.15.175:0/1056676945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced014fbf0 0x7fced0159d70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009ee0 tx=0x7fcec0009780 comp rx=0 tx=0).stop 2023-10-10T19:19:15.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fced5c42700 1 -- 172.21.15.175:0/1056676945 shutdown_connections 2023-10-10T19:19:15.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fced5c42700 1 --2- 172.21.15.175:0/1056676945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced014fbf0 0x7fced0159d70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:15.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fced5c42700 1 --2- 172.21.15.175:0/1056676945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0063230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:15.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fced5c42700 1 -- 172.21.15.175:0/1056676945 >> 172.21.15.175:0/1056676945 conn(0x7fced00cc760 msgr2=0x7fced00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:15.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fced5c42700 1 -- 172.21.15.175:0/1056676945 shutdown_connections 2023-10-10T19:19:15.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.968+0000 7fced5c42700 1 -- 172.21.15.175:0/1056676945 wait complete. 2023-10-10T19:19:15.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.969+0000 7fced5c42700 1 Processor -- start 2023-10-10T19:19:15.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.969+0000 7fced5c42700 1 -- start start 2023-10-10T19:19:15.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fced5c42700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced00681e0 0x7fced0165a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:15.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fced5c42700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fced014fbf0 0x7fced0151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:15.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fced5c42700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:15.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fced5c42700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0152b30 con 0x7fced014fbf0 2023-10-10T19:19:15.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fced5c42700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0151c40 con 0x7fced00681e0 2023-10-10T19:19:15.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fced5c42700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced0151f20 con 0x7fced014ffc0 2023-10-10T19:19:15.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fceceffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fced014fbf0 0x7fced0151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:15.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fceceffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fced014fbf0 0x7fced0151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56352/0 (socket says 172.21.15.175:56352) 2023-10-10T19:19:15.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fceceffd700 1 -- 172.21.15.175:0/1564743902 learned_addr learned my addr 172.21.15.175:0/1564743902 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:15.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fcecf7fe700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced00681e0 0x7fced0165a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:15.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.970+0000 7fcecffff700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:15.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceceffd700 1 -- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 msgr2=0x7fced0152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:15.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceceffd700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:15.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceceffd700 1 -- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced00681e0 msgr2=0x7fced0165a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:15.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceceffd700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced00681e0 0x7fced0165a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:15.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceceffd700 1 -- 172.21.15.175:0/1564743902 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4009420 con 0x7fced014fbf0 2023-10-10T19:19:15.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fcecf7fe700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced00681e0 0x7fced0165a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:15.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceceffd700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fced014fbf0 0x7fced0151620 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fcec400ec80 tx=0x7fcec40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:15.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceb77fe700 1 -- 172.21.15.175:0/1564743902 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec400c9d0 con 0x7fced014fbf0 2023-10-10T19:19:15.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceb77fe700 1 -- 172.21.15.175:0/1564743902 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec400cb30 con 0x7fced014fbf0 2023-10-10T19:19:15.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec0009a20 con 0x7fced014fbf0 2023-10-10T19:19:15.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fceb77fe700 1 -- 172.21.15.175:0/1564743902 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec400ceb0 con 0x7fced014fbf0 2023-10-10T19:19:15.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.971+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced01524e0 con 0x7fced014fbf0 2023-10-10T19:19:15.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.972+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced0067570 con 0x7fced014fbf0 2023-10-10T19:19:15.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.972+0000 7fceb77fe700 1 -- 172.21.15.175:0/1564743902 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcec4013680 con 0x7fced014fbf0 2023-10-10T19:19:15.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.973+0000 7fceb77fe700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcebc066f30 0x7fcebc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:15.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.973+0000 7fceb77fe700 1 -- 172.21.15.175:0/1564743902 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fcec4084f50 con 0x7fced014fbf0 2023-10-10T19:19:15.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.973+0000 7fcecf7fe700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcebc066f30 0x7fcebc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:15.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.973+0000 7fcecf7fe700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcebc066f30 0x7fcebc0693e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcec00097c0 tx=0x7fcec0019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:15.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:15.977+0000 7fceb77fe700 1 -- 172.21.15.175:0/1564743902 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcec4051b90 con 0x7fced014fbf0 2023-10-10T19:19:16.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.199+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fced00c7380 con 0x7fced014fbf0 2023-10-10T19:19:16.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.200+0000 7fceb77fe700 1 -- 172.21.15.175:0/1564743902 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) v1 ==== 95+0+6196 (secure 0 0 0) 0x7fcec40551b0 con 0x7fced014fbf0 2023-10-10T19:19:16.202 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:16.202 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":261,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:18:19.799555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:16.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcebc066f30 msgr2=0x7fcebc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:16.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcebc066f30 0x7fcebc0693e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcec00097c0 tx=0x7fcec0019000 comp rx=0 tx=0).stop 2023-10-10T19:19:16.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fced014fbf0 msgr2=0x7fced0151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:16.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fced014fbf0 0x7fced0151620 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fcec400ec80 tx=0x7fcec40091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 shutdown_connections 2023-10-10T19:19:16.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcebc066f30 0x7fcebc0693e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fced014ffc0 0x7fced0152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fced00681e0 0x7fced0165a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 --2- 172.21.15.175:0/1564743902 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fced014fbf0 0x7fced0151620 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 >> 172.21.15.175:0/1564743902 conn(0x7fced00cc760 msgr2=0x7fced00ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 shutdown_connections 2023-10-10T19:19:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.202+0000 7fced5c42700 1 -- 172.21.15.175:0/1564743902 wait complete. 2023-10-10T19:19:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 261 2023-10-10T19:19:16.215 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 5 -> 4 2023-10-10T19:19:16.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 4 2023-10-10T19:19:16.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.331+0000 7fe1cdfbf700 1 Processor -- start 2023-10-10T19:19:16.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.331+0000 7fe1cdfbf700 1 -- start start 2023-10-10T19:19:16.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1cdfbf700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe1c80c7620 0x7fe1c80aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1cdfbf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c80b0490 0x7fe1c814f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1cdfbf700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80c7ca0 con 0x7fe1c80633f0 2023-10-10T19:19:16.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1cdfbf700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c80af540 con 0x7fe1c80c7620 2023-10-10T19:19:16.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1cdfbf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8156830 con 0x7fe1c80b0490 2023-10-10T19:19:16.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1c77fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe1c80633f0 0x7fe1c8156010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46782/0 (socket says 172.21.15.175:46782) 2023-10-10T19:19:16.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1c77fe700 1 -- 172.21.15.175:0/2383572376 learned_addr learned my addr 172.21.15.175:0/2383572376 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:16.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1c7fff700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c80b0490 0x7fe1c814f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1c6ffd700 1 --2- 172.21.15.175:0/2383572376 >> v2:172.21.15.191:3300/0 conn(0x7fe1c80c7620 0x7fe1c80aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472552176 0 0) 0x7fe1c80c7ca0 con 0x7fe1c80633f0 2023-10-10T19:19:16.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.332+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b0003570 con 0x7fe1c80633f0 2023-10-10T19:19:16.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10606609 0 0) 0x7fe1b0003570 con 0x7fe1c80633f0 2023-10-10T19:19:16.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c80b0490 msgr2=0x7fe1c814f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:16.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c80b0490 0x7fe1c814f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 >> v2:172.21.15.191:3300/0 conn(0x7fe1c80c7620 msgr2=0x7fe1c80aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:16.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 --2- 172.21.15.175:0/2383572376 >> v2:172.21.15.191:3300/0 conn(0x7fe1c80c7620 0x7fe1c80aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8156ad0 con 0x7fe1c80633f0 2023-10-10T19:19:16.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fe1b80017f0 con 0x7fe1c80633f0 2023-10-10T19:19:16.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 >> v1:172.21.15.175:6789/0 conn(0x7fe1c80633f0 legacy=0x7fe1c8156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:16.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1b0003a60 0x7fe1b0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c80b0490 0x7fe1b0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1b0004d50 0x7fe1b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80af540 con 0x7fe1b0004d50 2023-10-10T19:19:16.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8156830 con 0x7fe1b0003a60 2023-10-10T19:19:16.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b000ba60 con 0x7fe1c80b0490 2023-10-10T19:19:16.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c7fff700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1b0004d50 0x7fe1b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c77fe700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1b0003a60 0x7fe1b0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c6ffd700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c80b0490 0x7fe1b0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c7fff700 1 -- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c80b0490 msgr2=0x7fe1b0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:16.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c7fff700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c80b0490 0x7fe1b0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c7fff700 1 -- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1b0003a60 msgr2=0x7fe1b0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:16.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c7fff700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1b0003a60 0x7fe1b0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c7fff700 1 -- 172.21.15.175:0/2383572376 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b0003570 con 0x7fe1b0004d50 2023-10-10T19:19:16.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c7fff700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1b0004d50 0x7fe1b00051c0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80b0930 tx=0x7fe1c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a6b76ffe11b95f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:16.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000e6b0 con 0x7fe1b0004d50 2023-10-10T19:19:16.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c000e810 con 0x7fe1b0004d50 2023-10-10T19:19:16.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.333+0000 7fe1c67fc700 1 -- 172.21.15.175:0/2383572376 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000ebd0 con 0x7fe1b0004d50 2023-10-10T19:19:16.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.334+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/2383572376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1b0004d50 msgr2=0x7fe1b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:16.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.334+0000 7fe1cdfbf700 1 --2- 172.21.15.175:0/2383572376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1b0004d50 0x7fe1b00051c0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80b0930 tx=0x7fe1c00091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.334+0000 7fe1c7fff700 1 -- 172.21.15.175:0/2383572376 reap_dead start 2023-10-10T19:19:16.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.334+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/2383572376 shutdown_connections 2023-10-10T19:19:16.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.334+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/2383572376 >> 172.21.15.175:0/2383572376 conn(0x7fe1c80cc7a0 msgr2=0x7fe1c80ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:16.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.334+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/2383572376 shutdown_connections 2023-10-10T19:19:16.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.334+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/2383572376 wait complete. 2023-10-10T19:19:16.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.335+0000 7fe1cdfbf700 1 Processor -- start 2023-10-10T19:19:16.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.335+0000 7fe1cdfbf700 1 -- start start 2023-10-10T19:19:16.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1cdfbf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c8154df0 0x7fe1c81551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1cdfbf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80c6bb0 0x7fe1c81508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1cdfbf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c8150e30 0x7fe1c8166380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1cdfbf700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8159ce0 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1cdfbf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8155b70 con 0x7fe1c8150e30 2023-10-10T19:19:16.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1cdfbf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8154660 con 0x7fe1c8154df0 2023-10-10T19:19:16.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80c6bb0 0x7fe1c81508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80c6bb0 0x7fe1c81508f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56368/0 (socket says 172.21.15.175:56368) 2023-10-10T19:19:16.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 -- 172.21.15.175:0/1307879724 learned_addr learned my addr 172.21.15.175:0/1307879724 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:16.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c77fe700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c8154df0 0x7fe1c81551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c7fff700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c8150e30 0x7fe1c8166380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 -- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c8154df0 msgr2=0x7fe1c81551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:16.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c8154df0 0x7fe1c81551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 -- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c8150e30 msgr2=0x7fe1c8166380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:16.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c8150e30 0x7fe1c8166380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 -- 172.21.15.175:0/1307879724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8153620 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1c6ffd700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80c6bb0 0x7fe1c81508f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc011b40 tx=0x7fe1bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:16.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.336+0000 7fe1af7fe700 1 -- 172.21.15.175:0/1307879724 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc00f940 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.337+0000 7fe1af7fe700 1 -- 172.21.15.175:0/1307879724 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc00faa0 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.337+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c8155890 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.337+0000 7fe1af7fe700 1 -- 172.21.15.175:0/1307879724 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc00fe20 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.337+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c814d530 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.338+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c8067110 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.338+0000 7fe1af7fe700 1 -- 172.21.15.175:0/1307879724 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe1bc014650 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.338+0000 7fe1af7fe700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b4066f30 0x7fe1b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:16.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.338+0000 7fe1af7fe700 1 -- 172.21.15.175:0/1307879724 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fe1bc018030 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.338+0000 7fe1c77fe700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b4066f30 0x7fe1b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:16.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.339+0000 7fe1c77fe700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b4066f30 0x7fe1b40693e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80020a0 tx=0x7fe1b8006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:16.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.342+0000 7fe1af7fe700 1 -- 172.21.15.175:0/1307879724 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1bc052080 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.568+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "4"} v 0) v1 -- 0x7fe1c8067570 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.899+0000 7fe1af7fe700 1 -- 172.21.15.175:0/1307879724 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "4"}]=0 v262) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe1bc0556a0 con 0x7fe1c80c6bb0 2023-10-10T19:19:16.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b4066f30 msgr2=0x7fe1b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:16.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b4066f30 0x7fe1b40693e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80020a0 tx=0x7fe1b8006040 comp rx=0 tx=0).stop 2023-10-10T19:19:16.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80c6bb0 msgr2=0x7fe1c81508f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:16.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80c6bb0 0x7fe1c81508f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc011b40 tx=0x7fe1bc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 shutdown_connections 2023-10-10T19:19:16.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1b4066f30 0x7fe1b40693e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c8150e30 0x7fe1c8166380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c8154df0 0x7fe1c81551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.901+0000 7fe1cdfbf700 1 --2- 172.21.15.175:0/1307879724 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80c6bb0 0x7fe1c81508f0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:16.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.902+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 >> 172.21.15.175:0/1307879724 conn(0x7fe1c80cc7a0 msgr2=0x7fe1c8068400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:16.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.902+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 shutdown_connections 2023-10-10T19:19:16.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:16.902+0000 7fe1cdfbf700 1 -- 172.21.15.175:0/1307879724 wait complete. 2023-10-10T19:19:16.915 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:19:16.916 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:17.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.029+0000 7fd67fc1a700 1 Processor -- start 2023-10-10T19:19:17.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.029+0000 7fd67fc1a700 1 -- start start 2023-10-10T19:19:17.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67fc1a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd6780b1fc0 0x7fd678155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:17.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67fc1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd678068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:17.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67fc1a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6781574b0 con 0x7fd678068bd0 2023-10-10T19:19:17.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67fc1a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd678150770 con 0x7fd6780b1fc0 2023-10-10T19:19:17.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67fc1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780afad0 con 0x7fd6780b2390 2023-10-10T19:19:17.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67d9b6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd6780b1fc0 0x7fd678155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:17.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67d1b5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd678068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:17.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67d9b6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd6780b1fc0 0x7fd678155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:55412/0 (socket says 172.21.15.175:55412) 2023-10-10T19:19:17.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67e1b7700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd678068bd0 0x7fd6780aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:41276/0 (socket says 172.21.15.175:41276) 2023-10-10T19:19:17.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67d1b5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd678068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:55616/0 (socket says 172.21.15.175:55616) 2023-10-10T19:19:17.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.030+0000 7fd67d9b6700 1 -- 172.21.15.175:0/3971290850 learned_addr learned my addr 172.21.15.175:0/3971290850 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:17.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67d9b6700 1 -- 172.21.15.175:0/3971290850 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 msgr2=0x7fd678068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:17.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67d9b6700 1 --2- 172.21.15.175:0/3971290850 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd678068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67d9b6700 1 -- 172.21.15.175:0/3971290850 >> v1:172.21.15.191:6789/0 conn(0x7fd678068bd0 legacy=0x7fd6780aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:17.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67d9b6700 1 -- 172.21.15.175:0/3971290850 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780af4c0 con 0x7fd6780b1fc0 2023-10-10T19:19:17.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67d1b5700 1 --2- 172.21.15.175:0/3971290850 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd678068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:17.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67d9b6700 1 --2- 172.21.15.175:0/3971290850 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6780b1fc0 0x7fd678155fb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd668009f20 tx=0x7fd668009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e80cb44793f07354 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:17.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67c9b4700 1 -- 172.21.15.175:0/3971290850 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66800e070 con 0x7fd6780b1fc0 2023-10-10T19:19:17.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67c9b4700 1 -- 172.21.15.175:0/3971290850 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66800ba60 con 0x7fd6780b1fc0 2023-10-10T19:19:17.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.031+0000 7fd67c9b4700 1 -- 172.21.15.175:0/3971290850 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6680092e0 con 0x7fd6780b1fc0 2023-10-10T19:19:17.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.032+0000 7fd67fc1a700 1 -- 172.21.15.175:0/3971290850 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6780b1fc0 msgr2=0x7fd678155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:17.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.032+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/3971290850 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6780b1fc0 0x7fd678155fb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd668009f20 tx=0x7fd668009840 comp rx=0 tx=0).stop 2023-10-10T19:19:17.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.032+0000 7fd67fc1a700 1 -- 172.21.15.175:0/3971290850 shutdown_connections 2023-10-10T19:19:17.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.032+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/3971290850 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6780b1fc0 0x7fd678155fb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.032+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/3971290850 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd678068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.032+0000 7fd67fc1a700 1 -- 172.21.15.175:0/3971290850 >> 172.21.15.175:0/3971290850 conn(0x7fd678063230 msgr2=0x7fd678063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:17.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.032+0000 7fd67fc1a700 1 -- 172.21.15.175:0/3971290850 shutdown_connections 2023-10-10T19:19:17.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.032+0000 7fd67fc1a700 1 -- 172.21.15.175:0/3971290850 wait complete. 2023-10-10T19:19:17.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.033+0000 7fd67fc1a700 1 Processor -- start 2023-10-10T19:19:17.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.033+0000 7fd67fc1a700 1 -- start start 2023-10-10T19:19:17.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67fc1a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd678068bd0 0x7fd67814ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:17.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67fc1a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6780b1fc0 0x7fd6780c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:17.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67fc1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd67814db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:17.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67fc1a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd678156e60 con 0x7fd6780b1fc0 2023-10-10T19:19:17.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67fc1a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd678155a70 con 0x7fd678068bd0 2023-10-10T19:19:17.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67fc1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd678169fc0 con 0x7fd6780b2390 2023-10-10T19:19:17.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6780b1fc0 0x7fd6780c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:17.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6780b1fc0 0x7fd6780c4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56384/0 (socket says 172.21.15.175:56384) 2023-10-10T19:19:17.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 -- 172.21.15.175:0/1954977935 learned_addr learned my addr 172.21.15.175:0/1954977935 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:17.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67e1b7700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd67814db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:17.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d9b6700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd678068bd0 0x7fd67814ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:17.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 -- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 msgr2=0x7fd67814db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:17.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd67814db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 -- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd678068bd0 msgr2=0x7fd67814ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:17.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd678068bd0 0x7fd67814ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 -- 172.21.15.175:0/1954977935 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c009420 con 0x7fd6780b1fc0 2023-10-10T19:19:17.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67d1b5700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6780b1fc0 0x7fd6780c4bd0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00ec80 tx=0x7fd66c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:17.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd675ffb700 1 -- 172.21.15.175:0/1954977935 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c00c9d0 con 0x7fd6780b1fc0 2023-10-10T19:19:17.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd675ffb700 1 -- 172.21.15.175:0/1954977935 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c00cb30 con 0x7fd6780b1fc0 2023-10-10T19:19:17.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd668009ae0 con 0x7fd6780b1fc0 2023-10-10T19:19:17.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.034+0000 7fd675ffb700 1 -- 172.21.15.175:0/1954977935 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c00ceb0 con 0x7fd6780b1fc0 2023-10-10T19:19:17.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.035+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd678155140 con 0x7fd6780b1fc0 2023-10-10T19:19:17.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.036+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd678155c00 con 0x7fd6780b1fc0 2023-10-10T19:19:17.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.036+0000 7fd675ffb700 1 -- 172.21.15.175:0/1954977935 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd66c013680 con 0x7fd6780b1fc0 2023-10-10T19:19:17.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.036+0000 7fd675ffb700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd664066f30 0x7fd6640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:17.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.036+0000 7fd675ffb700 1 -- 172.21.15.175:0/1954977935 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fd66c084f50 con 0x7fd6780b1fc0 2023-10-10T19:19:17.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.037+0000 7fd67d9b6700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd664066f30 0x7fd6640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:17.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.037+0000 7fd67d9b6700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd664066f30 0x7fd6640693e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd668009880 tx=0x7fd668018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:17.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.041+0000 7fd675ffb700 1 -- 172.21.15.175:0/1954977935 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd66c051b90 con 0x7fd6780b1fc0 2023-10-10T19:19:17.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.265+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd678067570 con 0x7fd6780b1fc0 2023-10-10T19:19:17.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.266+0000 7fd675ffb700 1 -- 172.21.15.175:0/1954977935 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+6198 (secure 0 0 0) 0x7fd66c0551b0 con 0x7fd6780b1fc0 2023-10-10T19:19:17.267 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:17.268 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:16.901254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:17.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd664066f30 msgr2=0x7fd6640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:17.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd664066f30 0x7fd6640693e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd668009880 tx=0x7fd668018000 comp rx=0 tx=0).stop 2023-10-10T19:19:17.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6780b1fc0 msgr2=0x7fd6780c4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:17.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6780b1fc0 0x7fd6780c4bd0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00ec80 tx=0x7fd66c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 shutdown_connections 2023-10-10T19:19:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd664066f30 0x7fd6640693e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6780b2390 0x7fd67814db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd678068bd0 0x7fd67814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 --2- 172.21.15.175:0/1954977935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6780b1fc0 0x7fd6780c4bd0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 >> 172.21.15.175:0/1954977935 conn(0x7fd678063230 msgr2=0x7fd678159310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 shutdown_connections 2023-10-10T19:19:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:17.268+0000 7fd67fc1a700 1 -- 172.21.15.175:0/1954977935 wait complete. 2023-10-10T19:19:17.272 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 263 2023-10-10T19:19:17.281 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-10T19:19:17.282 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 263, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 263, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:stopping", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 125, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:19:16.901254+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:19:19.284 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:19.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.404+0000 7fed5193a700 1 Processor -- start 2023-10-10T19:19:19.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.404+0000 7fed5193a700 1 -- start start 2023-10-10T19:19:19.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed5193a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fed4c14fd90 0x7fed4c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:19.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed5193a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c0cd040 0x7fed4c0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:19.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed5193a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c067fa0 con 0x7fed4c08a100 2023-10-10T19:19:19.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed5193a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4c0c5600 con 0x7fed4c14fd90 2023-10-10T19:19:19.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed5193a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c156b70 con 0x7fed4c0cd040 2023-10-10T19:19:19.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed50938700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fed4c14fd90 0x7fed4c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:19.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed4bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fed4c08a100 0x7fed4c0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46786/0 (socket says 172.21.15.175:46786) 2023-10-10T19:19:19.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed4bfff700 1 -- 172.21.15.175:0/2248083643 learned_addr learned my addr 172.21.15.175:0/2248083643 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:19.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed51139700 1 --2- 172.21.15.175:0/2248083643 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c0cd040 0x7fed4c0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:19.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed50938700 1 -- 172.21.15.175:0/2248083643 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c0cd040 msgr2=0x7fed4c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:19.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed50938700 1 --2- 172.21.15.175:0/2248083643 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c0cd040 0x7fed4c0684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed50938700 1 -- 172.21.15.175:0/2248083643 >> v1:172.21.15.175:6789/0 conn(0x7fed4c08a100 legacy=0x7fed4c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:19.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed4b7fe700 1 -- 172.21.15.175:0/2248083643 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941312256 0 0) 0x7fed4c067fa0 con 0x7fed4c08a100 2023-10-10T19:19:19.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.405+0000 7fed50938700 1 -- 172.21.15.175:0/2248083643 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c150160 con 0x7fed4c14fd90 2023-10-10T19:19:19.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed50938700 1 --2- 172.21.15.175:0/2248083643 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c159c30 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fed340098e0 tx=0x7fed340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3a5e61e72dc9b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:19.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed4b7fe700 1 -- 172.21.15.175:0/2248083643 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed34009e80 con 0x7fed4c14fd90 2023-10-10T19:19:19.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed4b7fe700 1 -- 172.21.15.175:0/2248083643 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3400b860 con 0x7fed4c14fd90 2023-10-10T19:19:19.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed4b7fe700 1 -- 172.21.15.175:0/2248083643 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3400bbe0 con 0x7fed4c14fd90 2023-10-10T19:19:19.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed5193a700 1 -- 172.21.15.175:0/2248083643 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 msgr2=0x7fed4c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:19.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed5193a700 1 --2- 172.21.15.175:0/2248083643 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c159c30 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fed340098e0 tx=0x7fed340091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed5193a700 1 -- 172.21.15.175:0/2248083643 shutdown_connections 2023-10-10T19:19:19.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed5193a700 1 --2- 172.21.15.175:0/2248083643 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c0cd040 0x7fed4c0684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed5193a700 1 --2- 172.21.15.175:0/2248083643 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c159c30 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed5193a700 1 -- 172.21.15.175:0/2248083643 >> 172.21.15.175:0/2248083643 conn(0x7fed4c063d00 msgr2=0x7fed4c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:19.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed5193a700 1 -- 172.21.15.175:0/2248083643 shutdown_connections 2023-10-10T19:19:19.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.406+0000 7fed5193a700 1 -- 172.21.15.175:0/2248083643 wait complete. 2023-10-10T19:19:19.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.407+0000 7fed5193a700 1 Processor -- start 2023-10-10T19:19:19.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.407+0000 7fed5193a700 1 -- start start 2023-10-10T19:19:19.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed5193a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c08a100 0x7fed4c1663f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:19.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed5193a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed4c0cd040 0x7fed4c1518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:19.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed5193a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c0cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:19.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed5193a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0c7450 con 0x7fed4c14fd90 2023-10-10T19:19:19.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed5193a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c155480 con 0x7fed4c0cd040 2023-10-10T19:19:19.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed5193a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0c7680 con 0x7fed4c08a100 2023-10-10T19:19:19.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed51139700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c0cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:19.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed51139700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c0cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56404/0 (socket says 172.21.15.175:56404) 2023-10-10T19:19:19.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed51139700 1 -- 172.21.15.175:0/2345386440 learned_addr learned my addr 172.21.15.175:0/2345386440 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:19.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed50938700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c08a100 0x7fed4c1663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:19.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.408+0000 7fed4bfff700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed4c0cd040 0x7fed4c1518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:19.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed51139700 1 -- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c08a100 msgr2=0x7fed4c1663f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:19.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed51139700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c08a100 0x7fed4c1663f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed51139700 1 -- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed4c0cd040 msgr2=0x7fed4c1518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:19.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed51139700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed4c0cd040 0x7fed4c1518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed51139700 1 -- 172.21.15.175:0/2345386440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed34009420 con 0x7fed4c14fd90 2023-10-10T19:19:19.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed50938700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c08a100 0x7fed4c1663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:19.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed51139700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c0cb760 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fed4400ceb0 tx=0x7fed44007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:19.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed48ff9700 1 -- 172.21.15.175:0/2345386440 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4400ae40 con 0x7fed4c14fd90 2023-10-10T19:19:19.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed48ff9700 1 -- 172.21.15.175:0/2345386440 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44016430 con 0x7fed4c14fd90 2023-10-10T19:19:19.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4c0c7960 con 0x7fed4c14fd90 2023-10-10T19:19:19.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed48ff9700 1 -- 172.21.15.175:0/2345386440 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed440167d0 con 0x7fed4c14fd90 2023-10-10T19:19:19.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.409+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4c154e30 con 0x7fed4c14fd90 2023-10-10T19:19:19.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.410+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4c067570 con 0x7fed4c14fd90 2023-10-10T19:19:19.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.410+0000 7fed48ff9700 1 -- 172.21.15.175:0/2345386440 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fed44003020 con 0x7fed4c14fd90 2023-10-10T19:19:19.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.411+0000 7fed48ff9700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed2c066f30 0x7fed2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:19.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.411+0000 7fed48ff9700 1 -- 172.21.15.175:0/2345386440 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fed44012070 con 0x7fed4c14fd90 2023-10-10T19:19:19.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.411+0000 7fed50938700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed2c066f30 0x7fed2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:19.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.411+0000 7fed50938700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed2c066f30 0x7fed2c0693e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fed340091e0 tx=0x7fed34017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:19.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.415+0000 7fed48ff9700 1 -- 172.21.15.175:0/2345386440 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed44052630 con 0x7fed4c14fd90 2023-10-10T19:19:19.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.641+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed4c155f80 con 0x7fed4c14fd90 2023-10-10T19:19:19.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.641+0000 7fed48ff9700 1 -- 172.21.15.175:0/2345386440 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+6198 (secure 0 0 0) 0x7fed44055c50 con 0x7fed4c14fd90 2023-10-10T19:19:19.643 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:19.643 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:16.901254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:19.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed2c066f30 msgr2=0x7fed2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed2c066f30 0x7fed2c0693e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fed340091e0 tx=0x7fed34017000 comp rx=0 tx=0).stop 2023-10-10T19:19:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 msgr2=0x7fed4c0cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c0cb760 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fed4400ceb0 tx=0x7fed44007870 comp rx=0 tx=0).stop 2023-10-10T19:19:19.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 shutdown_connections 2023-10-10T19:19:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed2c066f30 0x7fed2c0693e0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed4c14fd90 0x7fed4c0cb760 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed4c08a100 0x7fed4c1663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 --2- 172.21.15.175:0/2345386440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed4c0cd040 0x7fed4c1518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:19.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 >> 172.21.15.175:0/2345386440 conn(0x7fed4c063d00 msgr2=0x7fed4c0c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:19.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 shutdown_connections 2023-10-10T19:19:19.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:19.643+0000 7fed5193a700 1 -- 172.21.15.175:0/2345386440 wait complete. 2023-10-10T19:19:19.647 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 263 2023-10-10T19:19:19.656 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:21.657 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:21.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.782+0000 7f736d469700 1 Processor -- start 2023-10-10T19:19:21.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.782+0000 7f736d469700 1 -- start start 2023-10-10T19:19:21.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.782+0000 7f736d469700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f73680b2390 0x7f7368155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:21.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.782+0000 7f736d469700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:21.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.782+0000 7f736d469700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73681574b0 con 0x7f73680b1fc0 2023-10-10T19:19:21.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.782+0000 7f736d469700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7368150770 con 0x7f73680b2390 2023-10-10T19:19:21.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.782+0000 7f736d469700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73680afad0 con 0x7f73680aff60 2023-10-10T19:19:21.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.782+0000 7f7366ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f73680b1fc0 0x7f7368068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46802/0 (socket says 172.21.15.175:46802) 2023-10-10T19:19:21.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f7366ffd700 1 -- 172.21.15.175:0/2484342505 learned_addr learned my addr 172.21.15.175:0/2484342505 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:21.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f73677fe700 1 --2- 172.21.15.175:0/2484342505 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:21.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f735e7fc700 1 --2- 172.21.15.175:0/2484342505 >> v2:172.21.15.191:3300/0 conn(0x7f73680b2390 0x7f7368155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:21.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f73667fc700 1 -- 172.21.15.175:0/2484342505 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804288706 0 0) 0x7f73681574b0 con 0x7f73680b1fc0 2023-10-10T19:19:21.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f73667fc700 1 -- 172.21.15.175:0/2484342505 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7354003570 con 0x7f73680b1fc0 2023-10-10T19:19:21.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f735e7fc700 1 -- 172.21.15.175:0/2484342505 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 msgr2=0x7f73680b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:21.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f73667fc700 1 -- 172.21.15.175:0/2484342505 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1807266790 0 0) 0x7f7354003570 con 0x7f73680b1fc0 2023-10-10T19:19:21.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f735e7fc700 1 --2- 172.21.15.175:0/2484342505 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:21.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f735e7fc700 1 -- 172.21.15.175:0/2484342505 >> v1:172.21.15.175:6789/0 conn(0x7f73680b1fc0 legacy=0x7f7368068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:21.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f735e7fc700 1 -- 172.21.15.175:0/2484342505 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73680af4c0 con 0x7f73680b2390 2023-10-10T19:19:21.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f735e7fc700 1 --2- 172.21.15.175:0/2484342505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 0x7f7368155fb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f73500098e0 tx=0x7f73500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=934c7147b212b7cd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:21.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f73667fc700 1 -- 172.21.15.175:0/2484342505 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358001470 con 0x7f73680b2390 2023-10-10T19:19:21.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f73667fc700 1 -- 172.21.15.175:0/2484342505 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7350011420 con 0x7f73680b2390 2023-10-10T19:19:21.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.783+0000 7f73667fc700 1 -- 172.21.15.175:0/2484342505 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73500117c0 con 0x7f73680b2390 2023-10-10T19:19:21.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.784+0000 7f736d469700 1 -- 172.21.15.175:0/2484342505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 msgr2=0x7f7368155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:21.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.784+0000 7f736d469700 1 --2- 172.21.15.175:0/2484342505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 0x7f7368155fb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f73500098e0 tx=0x7f73500091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:21.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.784+0000 7f736d469700 1 -- 172.21.15.175:0/2484342505 shutdown_connections 2023-10-10T19:19:21.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.784+0000 7f736d469700 1 --2- 172.21.15.175:0/2484342505 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:21.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.784+0000 7f736d469700 1 --2- 172.21.15.175:0/2484342505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 0x7f7368155fb0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:21.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.784+0000 7f736d469700 1 -- 172.21.15.175:0/2484342505 >> 172.21.15.175:0/2484342505 conn(0x7f7368063230 msgr2=0x7f7368063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:21.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.784+0000 7f736d469700 1 -- 172.21.15.175:0/2484342505 shutdown_connections 2023-10-10T19:19:21.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.784+0000 7f736d469700 1 -- 172.21.15.175:0/2484342505 wait complete. 2023-10-10T19:19:21.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.785+0000 7f736d469700 1 Processor -- start 2023-10-10T19:19:21.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.785+0000 7f736d469700 1 -- start start 2023-10-10T19:19:21.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f736d469700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:21.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f736d469700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73680b1fc0 0x7f736814db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:21.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f736d469700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 0x7f73680ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:21.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f736d469700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73680cd430 con 0x7f73680b1fc0 2023-10-10T19:19:21.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f736d469700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73680cc9f0 con 0x7f73680b2390 2023-10-10T19:19:21.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f736d469700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73680cc020 con 0x7f73680aff60 2023-10-10T19:19:21.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73680b1fc0 0x7f736814db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:21.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73680b1fc0 0x7f736814db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56420/0 (socket says 172.21.15.175:56420) 2023-10-10T19:19:21.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 -- 172.21.15.175:0/810219224 learned_addr learned my addr 172.21.15.175:0/810219224 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:21.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f7366ffd700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:21.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f73677fe700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 0x7f73680ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:21.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 -- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 msgr2=0x7f73680cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:21.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:21.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 -- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 msgr2=0x7f73680ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:21.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 0x7f73680ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:21.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 -- 172.21.15.175:0/810219224 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7350009420 con 0x7f73680b1fc0 2023-10-10T19:19:21.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f7366ffd700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:19:21.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f73677fe700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 0x7f73680ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:19:21.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.786+0000 7f735e7fc700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73680b1fc0 0x7f736814db30 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7350009ed0 tx=0x7f7350016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:21.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.787+0000 7f735f7fe700 1 -- 172.21.15.175:0/810219224 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735001cc40 con 0x7f73680b1fc0 2023-10-10T19:19:21.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.787+0000 7f735f7fe700 1 -- 172.21.15.175:0/810219224 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735001cda0 con 0x7f73680b1fc0 2023-10-10T19:19:21.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.787+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73680cb730 con 0x7f73680b1fc0 2023-10-10T19:19:21.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.787+0000 7f735f7fe700 1 -- 172.21.15.175:0/810219224 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735001a660 con 0x7f73680b1fc0 2023-10-10T19:19:21.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.787+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7368166d80 con 0x7f73680b1fc0 2023-10-10T19:19:21.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.788+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7368063e90 con 0x7f73680b1fc0 2023-10-10T19:19:21.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.788+0000 7f735f7fe700 1 -- 172.21.15.175:0/810219224 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f735001aa50 con 0x7f73680b1fc0 2023-10-10T19:19:21.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.788+0000 7f735f7fe700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7348066f30 0x7f73480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:21.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.788+0000 7f735f7fe700 1 -- 172.21.15.175:0/810219224 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f73500864c0 con 0x7f73680b1fc0 2023-10-10T19:19:21.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.789+0000 7f7366ffd700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7348066f30 0x7f73480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:21.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.789+0000 7f7366ffd700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7348066f30 0x7f73480693e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f73580011e0 tx=0x7f7358009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:21.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:21.793+0000 7f735f7fe700 1 -- 172.21.15.175:0/810219224 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73500531b0 con 0x7f73680b1fc0 2023-10-10T19:19:22.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.018+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7368067570 con 0x7f73680b1fc0 2023-10-10T19:19:22.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.019+0000 7f735f7fe700 1 -- 172.21.15.175:0/810219224 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+6198 (secure 0 0 0) 0x7f73500567d0 con 0x7f73680b1fc0 2023-10-10T19:19:22.020 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:22.021 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:16.901254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:22.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7348066f30 msgr2=0x7f73480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:22.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7348066f30 0x7f73480693e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f73580011e0 tx=0x7f7358009040 comp rx=0 tx=0).stop 2023-10-10T19:19:22.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73680b1fc0 msgr2=0x7f736814db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:22.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73680b1fc0 0x7f736814db30 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7350009ed0 tx=0x7f7350016000 comp rx=0 tx=0).stop 2023-10-10T19:19:22.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 shutdown_connections 2023-10-10T19:19:22.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7348066f30 0x7f73480693e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:22.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73680b2390 0x7f73680ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:22.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73680aff60 0x7f73680cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:22.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 --2- 172.21.15.175:0/810219224 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73680b1fc0 0x7f736814db30 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:22.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 >> 172.21.15.175:0/810219224 conn(0x7f7368063230 msgr2=0x7f7368154020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:22.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 shutdown_connections 2023-10-10T19:19:22.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:22.022+0000 7f736d469700 1 -- 172.21.15.175:0/810219224 wait complete. 2023-10-10T19:19:22.025 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 263 2023-10-10T19:19:22.034 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:24.035 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:24.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.173+0000 7fb27dafa700 1 Processor -- start 2023-10-10T19:19:24.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.173+0000 7fb27dafa700 1 -- start start 2023-10-10T19:19:24.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb27dafa700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb27806b640 0x7fb27814cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:24.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb27dafa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2780c3700 0x7fb27814d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:24.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb27dafa700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2781533e0 con 0x7fb2780c3330 2023-10-10T19:19:24.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb27dafa700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27814d820 con 0x7fb27806b640 2023-10-10T19:19:24.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb27dafa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27806e130 con 0x7fb2780c3700 2023-10-10T19:19:24.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb276ffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb27806b640 0x7fb27814cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:24.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb277fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2780c3700 0x7fb27814d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:24.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb276ffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb27806b640 0x7fb27814cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:55462/0 (socket says 172.21.15.175:55462) 2023-10-10T19:19:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb276ffd700 1 -- 172.21.15.175:0/710631893 learned_addr learned my addr 172.21.15.175:0/710631893 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.174+0000 7fb2777fe700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb2780c3330 0x7fb2780c3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:41290/0 (socket says 172.21.15.175:41290) 2023-10-10T19:19:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb276ffd700 1 -- 172.21.15.175:0/710631893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2780c3700 msgr2=0x7fb27814d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb276ffd700 1 --2- 172.21.15.175:0/710631893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2780c3700 0x7fb27814d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb276ffd700 1 -- 172.21.15.175:0/710631893 >> v1:172.21.15.191:6789/0 conn(0x7fb2780c3330 legacy=0x7fb2780c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:24.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb276ffd700 1 -- 172.21.15.175:0/710631893 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27806a970 con 0x7fb27806b640 2023-10-10T19:19:24.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb2767fc700 1 -- 172.21.15.175:0/710631893 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738843589 0 0) 0x7fb2781533e0 con 0x7fb2780c3330 2023-10-10T19:19:24.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb277fff700 1 --2- 172.21.15.175:0/710631893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2780c3700 0x7fb27814d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:24.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb276ffd700 1 --2- 172.21.15.175:0/710631893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb27806b640 0x7fb27814cda0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0099b0 tx=0x7fb26c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1b1ae8b29b0483a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:24.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb2767fc700 1 -- 172.21.15.175:0/710631893 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c009ea0 con 0x7fb27806b640 2023-10-10T19:19:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb2767fc700 1 -- 172.21.15.175:0/710631893 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c00b860 con 0x7fb27806b640 2023-10-10T19:19:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb2767fc700 1 -- 172.21.15.175:0/710631893 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c00bc00 con 0x7fb27806b640 2023-10-10T19:19:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb27dafa700 1 -- 172.21.15.175:0/710631893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb27806b640 msgr2=0x7fb27814cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb27dafa700 1 --2- 172.21.15.175:0/710631893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb27806b640 0x7fb27814cda0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0099b0 tx=0x7fb26c009250 comp rx=0 tx=0).stop 2023-10-10T19:19:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb27dafa700 1 -- 172.21.15.175:0/710631893 shutdown_connections 2023-10-10T19:19:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb27dafa700 1 --2- 172.21.15.175:0/710631893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2780c3700 0x7fb27814d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb27dafa700 1 --2- 172.21.15.175:0/710631893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb27806b640 0x7fb27814cda0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb27dafa700 1 -- 172.21.15.175:0/710631893 >> 172.21.15.175:0/710631893 conn(0x7fb278068280 msgr2=0x7fb27806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.175+0000 7fb27dafa700 1 -- 172.21.15.175:0/710631893 shutdown_connections 2023-10-10T19:19:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.176+0000 7fb27dafa700 1 -- 172.21.15.175:0/710631893 wait complete. 2023-10-10T19:19:24.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.177+0000 7fb27dafa700 1 Processor -- start 2023-10-10T19:19:24.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.177+0000 7fb27dafa700 1 -- start start 2023-10-10T19:19:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.177+0000 7fb27dafa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb27806b640 0x7fb278166d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.177+0000 7fb27dafa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2780c3330 0x7fb27814a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.177+0000 7fb27dafa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2780c3700 0x7fb278151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.177+0000 7fb27dafa700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb278151e80 con 0x7fb2780c3330 2023-10-10T19:19:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.177+0000 7fb27dafa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27814c540 con 0x7fb2780c3700 2023-10-10T19:19:24.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.177+0000 7fb27dafa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb278169a70 con 0x7fb27806b640 2023-10-10T19:19:24.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2780c3330 0x7fb27814a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:24.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2780c3330 0x7fb27814a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56430/0 (socket says 172.21.15.175:56430) 2023-10-10T19:19:24.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 -- 172.21.15.175:0/1394709443 learned_addr learned my addr 172.21.15.175:0/1394709443 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb2777fe700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb27806b640 0x7fb278166d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb277fff700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2780c3700 0x7fb278151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 -- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb27806b640 msgr2=0x7fb278166d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb27806b640 0x7fb278166d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 -- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2780c3700 msgr2=0x7fb278151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:24.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2780c3700 0x7fb278151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 -- 172.21.15.175:0/1394709443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb270009420 con 0x7fb2780c3330 2023-10-10T19:19:24.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb276ffd700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2780c3330 0x7fb27814a210 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb26c006fd0 tx=0x7fb26c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:24.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb25f7fe700 1 -- 172.21.15.175:0/1394709443 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c00bbe0 con 0x7fb2780c3330 2023-10-10T19:19:24.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb25f7fe700 1 -- 172.21.15.175:0/1394709443 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c011420 con 0x7fb2780c3330 2023-10-10T19:19:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb25f7fe700 1 -- 172.21.15.175:0/1394709443 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0117c0 con 0x7fb2780c3330 2023-10-10T19:19:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c0094f0 con 0x7fb2780c3330 2023-10-10T19:19:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.178+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2781668f0 con 0x7fb2780c3330 2023-10-10T19:19:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.179+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb278166a80 con 0x7fb2780c3330 2023-10-10T19:19:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.179+0000 7fb25f7fe700 1 -- 172.21.15.175:0/1394709443 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb26c011bf0 con 0x7fb2780c3330 2023-10-10T19:19:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.180+0000 7fb25f7fe700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb264066f30 0x7fb2640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.180+0000 7fb25f7fe700 1 -- 172.21.15.175:0/1394709443 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fb26c0861f0 con 0x7fb2780c3330 2023-10-10T19:19:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.180+0000 7fb2777fe700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb264066f30 0x7fb2640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.180+0000 7fb2777fe700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb264066f30 0x7fb2640693e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb268001330 tx=0x7fb2680019e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.184+0000 7fb25f7fe700 1 -- 172.21.15.175:0/1394709443 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb26c052ef0 con 0x7fb2780c3330 2023-10-10T19:19:24.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.409+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb278067570 con 0x7fb2780c3330 2023-10-10T19:19:24.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.409+0000 7fb25f7fe700 1 -- 172.21.15.175:0/1394709443 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+6198 (secure 0 0 0) 0x7fb26c056510 con 0x7fb2780c3330 2023-10-10T19:19:24.410 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:24.411 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:16.901254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:24.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb264066f30 msgr2=0x7fb2640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:24.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb264066f30 0x7fb2640693e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb268001330 tx=0x7fb2680019e0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2780c3330 msgr2=0x7fb27814a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:24.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2780c3330 0x7fb27814a210 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb26c006fd0 tx=0x7fb26c017000 comp rx=0 tx=0).stop 2023-10-10T19:19:24.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 shutdown_connections 2023-10-10T19:19:24.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb264066f30 0x7fb2640693e0 secure :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb268001330 tx=0x7fb2680019e0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2780c3700 0x7fb278151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb27806b640 0x7fb278166d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 --2- 172.21.15.175:0/1394709443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb2780c3330 0x7fb27814a210 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:24.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 >> 172.21.15.175:0/1394709443 conn(0x7fb278068280 msgr2=0x7fb2780c3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:24.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 shutdown_connections 2023-10-10T19:19:24.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:24.412+0000 7fb27dafa700 1 -- 172.21.15.175:0/1394709443 wait complete. 2023-10-10T19:19:24.416 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 263 2023-10-10T19:19:24.424 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:24.754 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:19:24.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:19:26.425 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:26.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.553+0000 7f365d87d700 1 Processor -- start 2023-10-10T19:19:26.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.553+0000 7f365d87d700 1 -- start start 2023-10-10T19:19:26.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f365d87d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36580b2390 0x7f3658155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f365d87d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f36580aff60 0x7f36580b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f365d87d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36581574b0 con 0x7f36580b1fc0 2023-10-10T19:19:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f365d87d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3658150770 con 0x7f36580aff60 2023-10-10T19:19:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f365d87d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36580afad0 con 0x7f36580b2390 2023-10-10T19:19:26.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f3656ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f36580b1fc0 0x7f3658068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37514/0 (socket says 172.21.15.175:37514) 2023-10-10T19:19:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f3656ffd700 1 -- 172.21.15.175:0/1657873037 learned_addr learned my addr 172.21.15.175:0/1657873037 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9366379 0 0) 0x7f36581574b0 con 0x7f36580b1fc0 2023-10-10T19:19:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3644003570 con 0x7f36580b1fc0 2023-10-10T19:19:26.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f36577fe700 1 --2- 172.21.15.175:0/1657873037 >> v2:172.21.15.191:3300/0 conn(0x7f36580aff60 0x7f36580b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f364e7fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36580b2390 0x7f3658155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.554+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1855086787 0 0) 0x7f3644003570 con 0x7f36580b1fc0 2023-10-10T19:19:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36580b2390 msgr2=0x7f3658155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36580b2390 0x7f3658155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 >> v2:172.21.15.191:3300/0 conn(0x7f36580aff60 msgr2=0x7f36580b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 --2- 172.21.15.175:0/1657873037 >> v2:172.21.15.191:3300/0 conn(0x7f36580aff60 0x7f36580b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36580af4c0 con 0x7f36580b1fc0 2023-10-10T19:19:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f3648001470 con 0x7f36580b1fc0 2023-10-10T19:19:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 >> v1:172.21.15.175:6789/0 conn(0x7f36580b1fc0 legacy=0x7f3658068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:26.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36580b2390 0x7f3644003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f3644004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3644004d50 0x7f36440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3658150770 con 0x7f36440040b0 2023-10-10T19:19:26.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36580afad0 con 0x7f3644004d50 2023-10-10T19:19:26.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364400bb60 con 0x7f36580b2390 2023-10-10T19:19:26.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f364e7fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f3644004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f3656ffd700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36580b2390 0x7f3644003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36577fe700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3644004d50 0x7f36440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f364e7fc700 1 -- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36580b2390 msgr2=0x7f3644003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:26.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f364e7fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36580b2390 0x7f3644003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f364e7fc700 1 -- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3644004d50 msgr2=0x7f36440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:26.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f364e7fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3644004d50 0x7f36440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f364e7fc700 1 -- 172.21.15.175:0/1657873037 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3644003570 con 0x7f36440040b0 2023-10-10T19:19:26.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f364e7fc700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f3644004810 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f3640006f00 tx=0x7f36400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aaf7c75e36baf51 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:26.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364000e710 con 0x7f36440040b0 2023-10-10T19:19:26.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364000e870 con 0x7f36440040b0 2023-10-10T19:19:26.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.555+0000 7f36567fc700 1 -- 172.21.15.175:0/1657873037 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364000ec30 con 0x7f36440040b0 2023-10-10T19:19:26.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.556+0000 7f365d87d700 1 -- 172.21.15.175:0/1657873037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 msgr2=0x7f3644004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:26.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.556+0000 7f365d87d700 1 --2- 172.21.15.175:0/1657873037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f3644004810 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f3640006f00 tx=0x7f36400091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.556+0000 7f36577fe700 1 -- 172.21.15.175:0/1657873037 reap_dead start 2023-10-10T19:19:26.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.556+0000 7f365d87d700 1 -- 172.21.15.175:0/1657873037 shutdown_connections 2023-10-10T19:19:26.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.556+0000 7f365d87d700 1 -- 172.21.15.175:0/1657873037 >> 172.21.15.175:0/1657873037 conn(0x7f3658063230 msgr2=0x7f3658063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:26.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.556+0000 7f365d87d700 1 -- 172.21.15.175:0/1657873037 shutdown_connections 2023-10-10T19:19:26.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.556+0000 7f365d87d700 1 -- 172.21.15.175:0/1657873037 wait complete. 2023-10-10T19:19:26.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.557+0000 7f365d87d700 1 Processor -- start 2023-10-10T19:19:26.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.557+0000 7f365d87d700 1 -- start start 2023-10-10T19:19:26.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f365d87d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f36580c3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f365d87d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3658157b90 0x7f365814df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f365d87d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f365814ef00 0x7f365814f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f365d87d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36580c36b0 con 0x7f36440040b0 2023-10-10T19:19:26.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f365d87d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365814e4d0 con 0x7f365814ef00 2023-10-10T19:19:26.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f365d87d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36580cd260 con 0x7f3658157b90 2023-10-10T19:19:26.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f36580c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f36580c3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50820/0 (socket says 172.21.15.175:50820) 2023-10-10T19:19:26.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 -- 172.21.15.175:0/3655809029 learned_addr learned my addr 172.21.15.175:0/3655809029 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:26.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f36577fe700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f365814ef00 0x7f365814f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f364e7fc700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3658157b90 0x7f365814df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 -- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3658157b90 msgr2=0x7f365814df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:26.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3658157b90 0x7f365814df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 -- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f365814ef00 msgr2=0x7f365814f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:26.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f365814ef00 0x7f365814f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 -- 172.21.15.175:0/3655809029 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3658158250 con 0x7f36440040b0 2023-10-10T19:19:26.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f364e7fc700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3658157b90 0x7f365814df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:26.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f3656ffd700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f36580c3090 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f36480073c0 tx=0x7f36580afa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:26.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f364f7fe700 1 -- 172.21.15.175:0/3655809029 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648018030 con 0x7f36440040b0 2023-10-10T19:19:26.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f364f7fe700 1 -- 172.21.15.175:0/3655809029 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3648007780 con 0x7f36440040b0 2023-10-10T19:19:26.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.558+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3658158530 con 0x7f36440040b0 2023-10-10T19:19:26.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.559+0000 7f364f7fe700 1 -- 172.21.15.175:0/3655809029 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364800fa40 con 0x7f36440040b0 2023-10-10T19:19:26.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.559+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36580ccef0 con 0x7f36440040b0 2023-10-10T19:19:26.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.559+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36580713a0 con 0x7f36440040b0 2023-10-10T19:19:26.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.560+0000 7f364f7fe700 1 -- 172.21.15.175:0/3655809029 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f364801b070 con 0x7f36440040b0 2023-10-10T19:19:26.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.560+0000 7f364f7fe700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3638066f60 0x7f3638069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:26.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.560+0000 7f364f7fe700 1 -- 172.21.15.175:0/3655809029 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f3648019070 con 0x7f36440040b0 2023-10-10T19:19:26.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.560+0000 7f364e7fc700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3638066f60 0x7f3638069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:26.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.561+0000 7f364e7fc700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3638066f60 0x7f3638069410 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f36400091e0 tx=0x7f364001a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:26.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.565+0000 7f364f7fe700 1 -- 172.21.15.175:0/3655809029 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36580713a0 con 0x7f36440040b0 2023-10-10T19:19:26.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.789+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36580713a0 con 0x7f36440040b0 2023-10-10T19:19:26.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.790+0000 7f364f7fe700 1 -- 172.21.15.175:0/3655809029 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+6198 (secure 0 0 0) 0x7f36580713a0 con 0x7f36440040b0 2023-10-10T19:19:26.791 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:26.791 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:16.901254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:26.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3638066f60 msgr2=0x7f3638069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:26.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3638066f60 0x7f3638069410 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f36400091e0 tx=0x7f364001a000 comp rx=0 tx=0).stop 2023-10-10T19:19:26.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 msgr2=0x7f36580c3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:26.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f36580c3090 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f36480073c0 tx=0x7f36580afa40 comp rx=0 tx=0).stop 2023-10-10T19:19:26.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 shutdown_connections 2023-10-10T19:19:26.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3638066f60 0x7f3638069410 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f365814ef00 0x7f365814f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36440040b0 0x7f36580c3090 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 --2- 172.21.15.175:0/3655809029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3658157b90 0x7f365814df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:26.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 >> 172.21.15.175:0/3655809029 conn(0x7f3658063230 msgr2=0x7f365814fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:26.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 shutdown_connections 2023-10-10T19:19:26.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:26.792+0000 7f365d87d700 1 -- 172.21.15.175:0/3655809029 wait complete. 2023-10-10T19:19:26.796 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 263 2023-10-10T19:19:26.804 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:28.806 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:28.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.933+0000 7f272c358700 1 Processor -- start 2023-10-10T19:19:28.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.933+0000 7f272c358700 1 -- start start 2023-10-10T19:19:28.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272c358700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2724068a40 0x7f2724063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:28.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272c358700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2724068e10 0x7f27240af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:28.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272c358700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27240c4ee0 con 0x7f27240cac80 2023-10-10T19:19:28.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272c358700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2724064180 con 0x7f2724068a40 2023-10-10T19:19:28.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272c358700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27240c5780 con 0x7f2724068e10 2023-10-10T19:19:28.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a8f5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2724068e10 0x7f27240af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:28.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a0f4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2724068a40 0x7f2724063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:28.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a8f5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2724068e10 0x7f27240af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:53112/0 (socket says 172.21.15.175:53112) 2023-10-10T19:19:28.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a8f5700 1 -- 172.21.15.175:0/726417832 learned_addr learned my addr 172.21.15.175:0/726417832 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:28.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f27298f3700 1 --1- 172.21.15.175:0/726417832 >> v1:172.21.15.191:6789/0 conn(0x7f27240cac80 0x7f272406b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:37170/0 (socket says 172.21.15.175:37170) 2023-10-10T19:19:28.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f27290f2700 1 -- 172.21.15.175:0/726417832 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320540834 0 0) 0x7f27240c4ee0 con 0x7f27240cac80 2023-10-10T19:19:28.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f27290f2700 1 -- 172.21.15.175:0/726417832 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710003570 con 0x7f27240cac80 2023-10-10T19:19:28.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a8f5700 1 -- 172.21.15.175:0/726417832 >> v2:172.21.15.191:3300/0 conn(0x7f2724068a40 msgr2=0x7f2724063c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:28.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a8f5700 1 --2- 172.21.15.175:0/726417832 >> v2:172.21.15.191:3300/0 conn(0x7f2724068a40 0x7f2724063c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:28.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a8f5700 1 -- 172.21.15.175:0/726417832 >> v1:172.21.15.191:6789/0 conn(0x7f27240cac80 legacy=0x7f272406b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:28.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a8f5700 1 -- 172.21.15.175:0/726417832 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27240c6710 con 0x7f2724068e10 2023-10-10T19:19:28.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.934+0000 7f272a0f4700 1 --2- 172.21.15.175:0/726417832 >> v2:172.21.15.191:3300/0 conn(0x7f2724068a40 0x7f2724063c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272a8f5700 1 --2- 172.21.15.175:0/726417832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2724068e10 0x7f27240af910 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f271c009f20 tx=0x7f271c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd62cd7acdb82602 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f27290f2700 1 -- 172.21.15.175:0/726417832 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c00e070 con 0x7f2724068e10 2023-10-10T19:19:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f27290f2700 1 -- 172.21.15.175:0/726417832 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c00b970 con 0x7f2724068e10 2023-10-10T19:19:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f27290f2700 1 -- 172.21.15.175:0/726417832 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c0092e0 con 0x7f2724068e10 2023-10-10T19:19:28.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272c358700 1 -- 172.21.15.175:0/726417832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2724068e10 msgr2=0x7f27240af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272c358700 1 --2- 172.21.15.175:0/726417832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2724068e10 0x7f27240af910 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f271c009f20 tx=0x7f271c009840 comp rx=0 tx=0).stop 2023-10-10T19:19:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272c358700 1 -- 172.21.15.175:0/726417832 shutdown_connections 2023-10-10T19:19:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272c358700 1 --2- 172.21.15.175:0/726417832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2724068e10 0x7f27240af910 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272c358700 1 --2- 172.21.15.175:0/726417832 >> v2:172.21.15.191:3300/0 conn(0x7f2724068a40 0x7f2724063c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:28.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272c358700 1 -- 172.21.15.175:0/726417832 >> 172.21.15.175:0/726417832 conn(0x7f272406dbe0 msgr2=0x7f27240ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:28.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272c358700 1 -- 172.21.15.175:0/726417832 shutdown_connections 2023-10-10T19:19:28.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.935+0000 7f272c358700 1 -- 172.21.15.175:0/726417832 wait complete. 2023-10-10T19:19:28.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.936+0000 7f272c358700 1 Processor -- start 2023-10-10T19:19:28.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.936+0000 7f272c358700 1 -- start start 2023-10-10T19:19:28.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f272c358700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2724068a40 0x7f2724169300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:28.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f272c358700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2724068e10 0x7f2724152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:28.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f272c358700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27240cac80 0x7f27241531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:28.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f272c358700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27241527f0 con 0x7f2724068e10 2023-10-10T19:19:28.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f272c358700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2724064f50 con 0x7f2724068a40 2023-10-10T19:19:28.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f272c358700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2724169da0 con 0x7f27240cac80 2023-10-10T19:19:28.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f27298f3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2724068e10 0x7f2724152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:28.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f27298f3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2724068e10 0x7f2724152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50836/0 (socket says 172.21.15.175:50836) 2023-10-10T19:19:28.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f27298f3700 1 -- 172.21.15.175:0/3189801085 learned_addr learned my addr 172.21.15.175:0/3189801085 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f272a0f4700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2724068a40 0x7f2724169300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.937+0000 7f272a8f5700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27240cac80 0x7f27241531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f27298f3700 1 -- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27240cac80 msgr2=0x7f27241531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f27298f3700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27240cac80 0x7f27241531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f27298f3700 1 -- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2724068a40 msgr2=0x7f2724169300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:28.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f27298f3700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2724068a40 0x7f2724169300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f27298f3700 1 -- 172.21.15.175:0/3189801085 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27200093c0 con 0x7f2724068e10 2023-10-10T19:19:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f27298f3700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2724068e10 0x7f2724152c80 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f271400c580 tx=0x7f2714007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f271a7fc700 1 -- 172.21.15.175:0/3189801085 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714005ac0 con 0x7f2724068e10 2023-10-10T19:19:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f271a7fc700 1 -- 172.21.15.175:0/3189801085 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2714005c20 con 0x7f2724068e10 2023-10-10T19:19:28.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c009ae0 con 0x7f2724068e10 2023-10-10T19:19:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f271a7fc700 1 -- 172.21.15.175:0/3189801085 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140070d0 con 0x7f2724068e10 2023-10-10T19:19:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.938+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2724151ec0 con 0x7f2724068e10 2023-10-10T19:19:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.939+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27241529a0 con 0x7f2724068e10 2023-10-10T19:19:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.940+0000 7f271a7fc700 1 -- 172.21.15.175:0/3189801085 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2714007440 con 0x7f2724068e10 2023-10-10T19:19:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.940+0000 7f271a7fc700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2708066f30 0x7f27080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:28.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.940+0000 7f271a7fc700 1 -- 172.21.15.175:0/3189801085 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f2714083ed0 con 0x7f2724068e10 2023-10-10T19:19:28.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.940+0000 7f272a0f4700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2708066f30 0x7f27080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:28.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.941+0000 7f272a0f4700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2708066f30 0x7f27080693e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2720009f30 tx=0x7f2720009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:28.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:28.946+0000 7f271a7fc700 1 -- 172.21.15.175:0/3189801085 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f271404b5a0 con 0x7f2724068e10 2023-10-10T19:19:29.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.171+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2724067570 con 0x7f2724068e10 2023-10-10T19:19:29.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.171+0000 7f271a7fc700 1 -- 172.21.15.175:0/3189801085 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6199 (secure 0 0 0) 0x7f27241529a0 con 0x7f2724068e10 2023-10-10T19:19:29.172 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:29.173 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:26.925419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:29.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2708066f30 msgr2=0x7f27080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:29.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2708066f30 0x7f27080693e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2720009f30 tx=0x7f2720009120 comp rx=0 tx=0).stop 2023-10-10T19:19:29.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2724068e10 msgr2=0x7f2724152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:29.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2724068e10 0x7f2724152c80 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f271400c580 tx=0x7f2714007df0 comp rx=0 tx=0).stop 2023-10-10T19:19:29.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 shutdown_connections 2023-10-10T19:19:29.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2708066f30 0x7f27080693e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:29.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27240cac80 0x7f27241531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:29.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2724068a40 0x7f2724169300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:29.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 --2- 172.21.15.175:0/3189801085 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2724068e10 0x7f2724152c80 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:29.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 >> 172.21.15.175:0/3189801085 conn(0x7f272406dbe0 msgr2=0x7f27240c4190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:29.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 shutdown_connections 2023-10-10T19:19:29.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:29.174+0000 7f272c358700 1 -- 172.21.15.175:0/3189801085 wait complete. 2023-10-10T19:19:29.177 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 264 2023-10-10T19:19:29.186 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-10T19:19:29.187 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 264, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 264, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:stopping", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 125, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:19:26.925419+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:19:31.188 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:31.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.313+0000 7f8b2aa0d700 1 Processor -- start 2023-10-10T19:19:31.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.314+0000 7f8b2aa0d700 1 -- start start 2023-10-10T19:19:31.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.314+0000 7f8b2aa0d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b240c5e40 0x7f8b240c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:31.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.314+0000 7f8b2aa0d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8b24156520 0x7f8b241569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:31.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.314+0000 7f8b2aa0d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b24156ee0 con 0x7f8b2406bf00 2023-10-10T19:19:31.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.314+0000 7f8b2aa0d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b241560d0 con 0x7f8b240c5e40 2023-10-10T19:19:31.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.314+0000 7f8b2aa0d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b240c6df0 con 0x7f8b24156520 2023-10-10T19:19:31.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.314+0000 7f8b23fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b240c5e40 0x7f8b240c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:31.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.314+0000 7f8b23fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b240c5e40 0x7f8b240c47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50838/0 (socket says 172.21.15.175:50838) 2023-10-10T19:19:31.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b23fff700 1 -- 172.21.15.175:0/69110903 learned_addr learned my addr 172.21.15.175:0/69110903 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:31.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b28faa700 1 --2- 172.21.15.175:0/69110903 >> v2:172.21.15.191:3300/0 conn(0x7f8b24156520 0x7f8b241569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:31.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b23fff700 1 -- 172.21.15.175:0/69110903 >> v2:172.21.15.191:3300/0 conn(0x7f8b24156520 msgr2=0x7f8b241569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:31.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b23fff700 1 --2- 172.21.15.175:0/69110903 >> v2:172.21.15.191:3300/0 conn(0x7f8b24156520 0x7f8b241569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b23fff700 1 -- 172.21.15.175:0/69110903 >> v1:172.21.15.191:6789/0 conn(0x7f8b2406bf00 legacy=0x7f8b240c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:31.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b23fff700 1 -- 172.21.15.175:0/69110903 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b240c6920 con 0x7f8b240c5e40 2023-10-10T19:19:31.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b28faa700 1 --2- 172.21.15.175:0/69110903 >> v2:172.21.15.191:3300/0 conn(0x7f8b24156520 0x7f8b241569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:31.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b23fff700 1 --2- 172.21.15.175:0/69110903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b240c5e40 0x7f8b240c47d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f8b2406c8b0 tx=0x7f8b14009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21640d97da162f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:31.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b22ffd700 1 -- 172.21.15.175:0/69110903 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14005c10 con 0x7f8b240c5e40 2023-10-10T19:19:31.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b22ffd700 1 -- 172.21.15.175:0/69110903 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b14005d70 con 0x7f8b240c5e40 2023-10-10T19:19:31.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.315+0000 7f8b22ffd700 1 -- 172.21.15.175:0/69110903 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400b910 con 0x7f8b240c5e40 2023-10-10T19:19:31.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.316+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/69110903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b240c5e40 msgr2=0x7f8b240c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:31.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.316+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/69110903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b240c5e40 0x7f8b240c47d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f8b2406c8b0 tx=0x7f8b14009220 comp rx=0 tx=0).stop 2023-10-10T19:19:31.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.316+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/69110903 shutdown_connections 2023-10-10T19:19:31.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.316+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/69110903 >> v2:172.21.15.191:3300/0 conn(0x7f8b24156520 0x7f8b241569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.316+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/69110903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b240c5e40 0x7f8b240c47d0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.316+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/69110903 >> 172.21.15.175:0/69110903 conn(0x7f8b24068900 msgr2=0x7f8b24068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:31.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.316+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/69110903 shutdown_connections 2023-10-10T19:19:31.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.316+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/69110903 wait complete. 2023-10-10T19:19:31.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.317+0000 7f8b2aa0d700 1 Processor -- start 2023-10-10T19:19:31.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.317+0000 7f8b2aa0d700 1 -- start start 2023-10-10T19:19:31.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b2aa0d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b2406bf00 0x7f8b24153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:31.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b2aa0d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b240c5e40 0x7f8b2416e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:31.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b2aa0d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b24156520 0x7f8b24155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:31.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b2aa0d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b24151550 con 0x7f8b2406bf00 2023-10-10T19:19:31.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b2aa0d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b24154420 con 0x7f8b24156520 2023-10-10T19:19:31.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b2aa0d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2416e920 con 0x7f8b240c5e40 2023-10-10T19:19:31.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b2406bf00 0x7f8b24153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:31.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b2406bf00 0x7f8b24153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50854/0 (socket says 172.21.15.175:50854) 2023-10-10T19:19:31.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 -- 172.21.15.175:0/2851980251 learned_addr learned my addr 172.21.15.175:0/2851980251 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:31.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b28faa700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b24156520 0x7f8b24155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:31.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b237fe700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b240c5e40 0x7f8b2416e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:31.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 -- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b240c5e40 msgr2=0x7f8b2416e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:31.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b240c5e40 0x7f8b2416e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 -- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b24156520 msgr2=0x7f8b24155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:31.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b24156520 0x7f8b24155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 -- 172.21.15.175:0/2851980251 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b14009480 con 0x7f8b2406bf00 2023-10-10T19:19:31.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b237fe700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b240c5e40 0x7f8b2416e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:31.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b23fff700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b2406bf00 0x7f8b24153ee0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400b570 tx=0x7f8b14016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:31.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b0bfff700 1 -- 172.21.15.175:0/2851980251 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400bcd0 con 0x7f8b2406bf00 2023-10-10T19:19:31.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b0bfff700 1 -- 172.21.15.175:0/2851980251 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b14020070 con 0x7f8b2406bf00 2023-10-10T19:19:31.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.318+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b24150960 con 0x7f8b2406bf00 2023-10-10T19:19:31.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.319+0000 7f8b0bfff700 1 -- 172.21.15.175:0/2851980251 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400b6e0 con 0x7f8b2406bf00 2023-10-10T19:19:31.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.319+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b24150c20 con 0x7f8b2406bf00 2023-10-10T19:19:31.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.319+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b24067570 con 0x7f8b2406bf00 2023-10-10T19:19:31.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.320+0000 7f8b0bfff700 1 -- 172.21.15.175:0/2851980251 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8b14022030 con 0x7f8b2406bf00 2023-10-10T19:19:31.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.320+0000 7f8b0bfff700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b10066f30 0x7f8b100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:31.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.320+0000 7f8b0bfff700 1 -- 172.21.15.175:0/2851980251 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f8b14086620 con 0x7f8b2406bf00 2023-10-10T19:19:31.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.320+0000 7f8b237fe700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b10066f30 0x7f8b100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:31.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.321+0000 7f8b237fe700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b10066f30 0x7f8b100693e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8b2406fa10 tx=0x7f8b18009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:31.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.324+0000 7f8b0bfff700 1 -- 172.21.15.175:0/2851980251 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b14053260 con 0x7f8b2406bf00 2023-10-10T19:19:31.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.549+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b24150050 con 0x7f8b2406bf00 2023-10-10T19:19:31.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.549+0000 7f8b0bfff700 1 -- 172.21.15.175:0/2851980251 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6199 (secure 0 0 0) 0x7f8b14056880 con 0x7f8b2406bf00 2023-10-10T19:19:31.550 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:31.551 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:26.925419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:31.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b10066f30 msgr2=0x7f8b100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:31.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b10066f30 0x7f8b100693e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8b2406fa10 tx=0x7f8b18009040 comp rx=0 tx=0).stop 2023-10-10T19:19:31.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b2406bf00 msgr2=0x7f8b24153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:31.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b2406bf00 0x7f8b24153ee0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400b570 tx=0x7f8b14016000 comp rx=0 tx=0).stop 2023-10-10T19:19:31.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 shutdown_connections 2023-10-10T19:19:31.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b10066f30 0x7f8b100693e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b24156520 0x7f8b24155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b2406bf00 0x7f8b24153ee0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 --2- 172.21.15.175:0/2851980251 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b240c5e40 0x7f8b2416e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:31.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 >> 172.21.15.175:0/2851980251 conn(0x7f8b24068900 msgr2=0x7f8b240c0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:31.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 shutdown_connections 2023-10-10T19:19:31.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:31.551+0000 7f8b2aa0d700 1 -- 172.21.15.175:0/2851980251 wait complete. 2023-10-10T19:19:31.555 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 264 2023-10-10T19:19:31.564 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:33.564 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:33.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.692+0000 7fb12ade7700 1 Processor -- start 2023-10-10T19:19:33.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.692+0000 7fb12ade7700 1 -- start start 2023-10-10T19:19:33.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.692+0000 7fb12ade7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb1240b1fc0 0x7fb124155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:33.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.692+0000 7fb12ade7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb1240b2390 0x7fb124068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:33.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.692+0000 7fb12ade7700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1241574b0 con 0x7fb124068bd0 2023-10-10T19:19:33.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.692+0000 7fb12ade7700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb124150770 con 0x7fb1240b1fc0 2023-10-10T19:19:33.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.692+0000 7fb12ade7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1240afad0 con 0x7fb1240b2390 2023-10-10T19:19:33.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb128b83700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb1240b1fc0 0x7fb124155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:33.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb128b83700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb1240b1fc0 0x7fb124155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50868/0 (socket says 172.21.15.175:50868) 2023-10-10T19:19:33.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb128b83700 1 -- 172.21.15.175:0/3150585474 learned_addr learned my addr 172.21.15.175:0/3150585474 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:33.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb123fff700 1 --2- 172.21.15.175:0/3150585474 >> v2:172.21.15.191:3300/0 conn(0x7fb1240b2390 0x7fb124068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:33.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb128b83700 1 -- 172.21.15.175:0/3150585474 >> v2:172.21.15.191:3300/0 conn(0x7fb1240b2390 msgr2=0x7fb124068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:33.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb128b83700 1 --2- 172.21.15.175:0/3150585474 >> v2:172.21.15.191:3300/0 conn(0x7fb1240b2390 0x7fb124068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb128b83700 1 -- 172.21.15.175:0/3150585474 >> v1:172.21.15.191:6789/0 conn(0x7fb124068bd0 legacy=0x7fb1240aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:33.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb128b83700 1 -- 172.21.15.175:0/3150585474 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1240af4c0 con 0x7fb1240b1fc0 2023-10-10T19:19:33.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb123fff700 1 --2- 172.21.15.175:0/3150585474 >> v2:172.21.15.191:3300/0 conn(0x7fb1240b2390 0x7fb124068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:19:33.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb128b83700 1 --2- 172.21.15.175:0/3150585474 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1240b1fc0 0x7fb124155fb0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fb1240c0290 tx=0x7fb114009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b59c5cfd82f0905 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:33.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb1237fe700 1 -- 172.21.15.175:0/3150585474 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11400e070 con 0x7fb1240b1fc0 2023-10-10T19:19:33.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb1237fe700 1 -- 172.21.15.175:0/3150585474 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11400baa0 con 0x7fb1240b1fc0 2023-10-10T19:19:33.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.693+0000 7fb1237fe700 1 -- 172.21.15.175:0/3150585474 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1140092e0 con 0x7fb1240b1fc0 2023-10-10T19:19:33.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.694+0000 7fb12ade7700 1 -- 172.21.15.175:0/3150585474 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1240b1fc0 msgr2=0x7fb124155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:33.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.694+0000 7fb12ade7700 1 --2- 172.21.15.175:0/3150585474 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1240b1fc0 0x7fb124155fb0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fb1240c0290 tx=0x7fb114009840 comp rx=0 tx=0).stop 2023-10-10T19:19:33.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.694+0000 7fb12ade7700 1 -- 172.21.15.175:0/3150585474 shutdown_connections 2023-10-10T19:19:33.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.694+0000 7fb12ade7700 1 --2- 172.21.15.175:0/3150585474 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1240b1fc0 0x7fb124155fb0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.694+0000 7fb12ade7700 1 --2- 172.21.15.175:0/3150585474 >> v2:172.21.15.191:3300/0 conn(0x7fb1240b2390 0x7fb124068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.694+0000 7fb12ade7700 1 -- 172.21.15.175:0/3150585474 >> 172.21.15.175:0/3150585474 conn(0x7fb124063230 msgr2=0x7fb124063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:33.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.694+0000 7fb12ade7700 1 -- 172.21.15.175:0/3150585474 shutdown_connections 2023-10-10T19:19:33.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.694+0000 7fb12ade7700 1 -- 172.21.15.175:0/3150585474 wait complete. 2023-10-10T19:19:33.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.695+0000 7fb12ade7700 1 Processor -- start 2023-10-10T19:19:33.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.695+0000 7fb12ade7700 1 -- start start 2023-10-10T19:19:33.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb12ade7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb124068bd0 0x7fb1240c4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:33.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb12ade7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1240b1fc0 0x7fb1240c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:33.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb12ade7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1240b2390 0x7fb12416d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:33.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb12ade7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1240c5260 con 0x7fb124068bd0 2023-10-10T19:19:33.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb12ade7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12415d830 con 0x7fb1240b1fc0 2023-10-10T19:19:33.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb12ade7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1240c3b80 con 0x7fb1240b2390 2023-10-10T19:19:33.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb124068bd0 0x7fb1240c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:33.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb124068bd0 0x7fb1240c4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50884/0 (socket says 172.21.15.175:50884) 2023-10-10T19:19:33.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 -- 172.21.15.175:0/1826824933 learned_addr learned my addr 172.21.15.175:0/1826824933 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:33.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb123fff700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1240b1fc0 0x7fb1240c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:33.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb129384700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1240b2390 0x7fb12416d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:33.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 -- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1240b2390 msgr2=0x7fb12416d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:33.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1240b2390 0x7fb12416d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 -- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1240b1fc0 msgr2=0x7fb1240c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:33.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1240b1fc0 0x7fb1240c4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 -- 172.21.15.175:0/1826824933 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb114009ae0 con 0x7fb124068bd0 2023-10-10T19:19:33.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb123fff700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1240b1fc0 0x7fb1240c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:19:33.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb128b83700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb124068bd0 0x7fb1240c4330 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fb12406fd90 tx=0x7fb114017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:33.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb120ff9700 1 -- 172.21.15.175:0/1826824933 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114012af0 con 0x7fb124068bd0 2023-10-10T19:19:33.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb120ff9700 1 -- 172.21.15.175:0/1826824933 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb114012c90 con 0x7fb124068bd0 2023-10-10T19:19:33.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1240c3e60 con 0x7fb124068bd0 2023-10-10T19:19:33.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.696+0000 7fb120ff9700 1 -- 172.21.15.175:0/1826824933 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11401d670 con 0x7fb124068bd0 2023-10-10T19:19:33.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.697+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb124154b10 con 0x7fb124068bd0 2023-10-10T19:19:33.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.697+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb124067570 con 0x7fb124068bd0 2023-10-10T19:19:33.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.698+0000 7fb120ff9700 1 -- 172.21.15.175:0/1826824933 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb11401da60 con 0x7fb124068bd0 2023-10-10T19:19:33.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.698+0000 7fb120ff9700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb104066f30 0x7fb1040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:33.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.698+0000 7fb120ff9700 1 -- 172.21.15.175:0/1826824933 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fb1140863e0 con 0x7fb124068bd0 2023-10-10T19:19:33.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.698+0000 7fb123fff700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb104066f30 0x7fb1040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:33.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.699+0000 7fb123fff700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb104066f30 0x7fb1040693e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb10c006fd0 tx=0x7fb10c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:33.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.703+0000 7fb120ff9700 1 -- 172.21.15.175:0/1826824933 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb114053020 con 0x7fb124068bd0 2023-10-10T19:19:33.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.928+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb124069060 con 0x7fb124068bd0 2023-10-10T19:19:33.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.929+0000 7fb120ff9700 1 -- 172.21.15.175:0/1826824933 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6199 (secure 0 0 0) 0x7fb114056640 con 0x7fb124068bd0 2023-10-10T19:19:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:33.930 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:26.925419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:33.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb104066f30 msgr2=0x7fb1040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:33.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb104066f30 0x7fb1040693e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb10c006fd0 tx=0x7fb10c008040 comp rx=0 tx=0).stop 2023-10-10T19:19:33.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb124068bd0 msgr2=0x7fb1240c4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:33.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb124068bd0 0x7fb1240c4330 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fb12406fd90 tx=0x7fb114017000 comp rx=0 tx=0).stop 2023-10-10T19:19:33.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 shutdown_connections 2023-10-10T19:19:33.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb104066f30 0x7fb1040693e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1240b2390 0x7fb12416d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb124068bd0 0x7fb1240c4330 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 --2- 172.21.15.175:0/1826824933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1240b1fc0 0x7fb1240c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:33.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 >> 172.21.15.175:0/1826824933 conn(0x7fb124063230 msgr2=0x7fb124159290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:33.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 shutdown_connections 2023-10-10T19:19:33.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:33.931+0000 7fb12ade7700 1 -- 172.21.15.175:0/1826824933 wait complete. 2023-10-10T19:19:33.934 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 264 2023-10-10T19:19:33.945 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:35.946 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:36.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.069+0000 7fec6fee2700 1 Processor -- start 2023-10-10T19:19:36.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.069+0000 7fec6fee2700 1 -- start start 2023-10-10T19:19:36.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6fee2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b1fc0 0x7fec68155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:36.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6fee2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fec680b2390 0x7fec68068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:36.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6fee2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec681574b0 con 0x7fec68068bd0 2023-10-10T19:19:36.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6fee2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec68150770 con 0x7fec680b2390 2023-10-10T19:19:36.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6fee2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec680afad0 con 0x7fec680b1fc0 2023-10-10T19:19:36.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6e47f700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fec68068bd0 0x7fec680aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51778/0 (socket says 172.21.15.175:51778) 2023-10-10T19:19:36.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6e47f700 1 -- 172.21.15.175:0/762081954 learned_addr learned my addr 172.21.15.175:0/762081954 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:36.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6d47d700 1 --2- 172.21.15.175:0/762081954 >> v2:172.21.15.191:3300/0 conn(0x7fec680b2390 0x7fec68068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:36.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6dc7e700 1 --2- 172.21.15.175:0/762081954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b1fc0 0x7fec68155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:36.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6cc7c700 1 -- 172.21.15.175:0/762081954 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275075446 0 0) 0x7fec681574b0 con 0x7fec68068bd0 2023-10-10T19:19:36.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6cc7c700 1 -- 172.21.15.175:0/762081954 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec54003570 con 0x7fec68068bd0 2023-10-10T19:19:36.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6dc7e700 1 -- 172.21.15.175:0/762081954 >> v2:172.21.15.191:3300/0 conn(0x7fec680b2390 msgr2=0x7fec68068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:36.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6dc7e700 1 --2- 172.21.15.175:0/762081954 >> v2:172.21.15.191:3300/0 conn(0x7fec680b2390 0x7fec68068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6dc7e700 1 -- 172.21.15.175:0/762081954 >> v1:172.21.15.175:6789/0 conn(0x7fec68068bd0 legacy=0x7fec680aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:36.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6dc7e700 1 -- 172.21.15.175:0/762081954 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec680af4c0 con 0x7fec680b1fc0 2023-10-10T19:19:36.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6d47d700 1 --2- 172.21.15.175:0/762081954 >> v2:172.21.15.191:3300/0 conn(0x7fec680b2390 0x7fec68068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:36.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.070+0000 7fec6dc7e700 1 --2- 172.21.15.175:0/762081954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b1fc0 0x7fec68155fb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fec640098e0 tx=0x7fec640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba0d72c429a08656 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:36.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6cc7c700 1 -- 172.21.15.175:0/762081954 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64011630 con 0x7fec680b1fc0 2023-10-10T19:19:36.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6cc7c700 1 -- 172.21.15.175:0/762081954 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec64011790 con 0x7fec680b1fc0 2023-10-10T19:19:36.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6cc7c700 1 -- 172.21.15.175:0/762081954 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64011b10 con 0x7fec680b1fc0 2023-10-10T19:19:36.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6fee2700 1 -- 172.21.15.175:0/762081954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b1fc0 msgr2=0x7fec68155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:36.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6fee2700 1 --2- 172.21.15.175:0/762081954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b1fc0 0x7fec68155fb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fec640098e0 tx=0x7fec640091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6fee2700 1 -- 172.21.15.175:0/762081954 shutdown_connections 2023-10-10T19:19:36.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6fee2700 1 --2- 172.21.15.175:0/762081954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b1fc0 0x7fec68155fb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6fee2700 1 --2- 172.21.15.175:0/762081954 >> v2:172.21.15.191:3300/0 conn(0x7fec680b2390 0x7fec68068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6fee2700 1 -- 172.21.15.175:0/762081954 >> 172.21.15.175:0/762081954 conn(0x7fec68063230 msgr2=0x7fec68063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:36.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6fee2700 1 -- 172.21.15.175:0/762081954 shutdown_connections 2023-10-10T19:19:36.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.071+0000 7fec6fee2700 1 -- 172.21.15.175:0/762081954 wait complete. 2023-10-10T19:19:36.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.072+0000 7fec6fee2700 1 Processor -- start 2023-10-10T19:19:36.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.072+0000 7fec6fee2700 1 -- start start 2023-10-10T19:19:36.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6fee2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec68068bd0 0x7fec68167670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:36.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6fee2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec680b1fc0 0x7fec68167bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:36.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6fee2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b2390 0x7fec68169610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:36.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6fee2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68159d60 con 0x7fec680b1fc0 2023-10-10T19:19:36.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6fee2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68157ce0 con 0x7fec68068bd0 2023-10-10T19:19:36.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6fee2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec68158e70 con 0x7fec680b2390 2023-10-10T19:19:36.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6d47d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec680b1fc0 0x7fec68167bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:36.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6d47d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec680b1fc0 0x7fec68167bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56226/0 (socket says 172.21.15.175:56226) 2023-10-10T19:19:36.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6d47d700 1 -- 172.21.15.175:0/122644217 learned_addr learned my addr 172.21.15.175:0/122644217 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:36.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6e47f700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b2390 0x7fec68169610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:36.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6dc7e700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec68068bd0 0x7fec68167670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:36.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6d47d700 1 -- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b2390 msgr2=0x7fec68169610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:36.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6d47d700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b2390 0x7fec68169610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6d47d700 1 -- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec68068bd0 msgr2=0x7fec68167670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:36.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6d47d700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec68068bd0 0x7fec68167670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.073+0000 7fec6d47d700 1 -- 172.21.15.175:0/122644217 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58009420 con 0x7fec680b1fc0 2023-10-10T19:19:36.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.074+0000 7fec6d47d700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec680b1fc0 0x7fec68167bb0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fec5800ec80 tx=0x7fec580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:36.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.074+0000 7fec5dffb700 1 -- 172.21.15.175:0/122644217 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5800c850 con 0x7fec680b1fc0 2023-10-10T19:19:36.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.074+0000 7fec5dffb700 1 -- 172.21.15.175:0/122644217 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5800c9b0 con 0x7fec680b1fc0 2023-10-10T19:19:36.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.074+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec64009420 con 0x7fec680b1fc0 2023-10-10T19:19:36.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.074+0000 7fec5dffb700 1 -- 172.21.15.175:0/122644217 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5800cd30 con 0x7fec680b1fc0 2023-10-10T19:19:36.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.074+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec68159430 con 0x7fec680b1fc0 2023-10-10T19:19:36.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.075+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec68167390 con 0x7fec680b1fc0 2023-10-10T19:19:36.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.075+0000 7fec5dffb700 1 -- 172.21.15.175:0/122644217 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fec58013680 con 0x7fec680b1fc0 2023-10-10T19:19:36.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.076+0000 7fec5dffb700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec4c066f30 0x7fec4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:36.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.076+0000 7fec5dffb700 1 -- 172.21.15.175:0/122644217 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fec58085110 con 0x7fec680b1fc0 2023-10-10T19:19:36.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.076+0000 7fec6dc7e700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec4c066f30 0x7fec4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:36.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.076+0000 7fec6dc7e700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec4c066f30 0x7fec4c0693e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fec640091e0 tx=0x7fec64017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:36.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.080+0000 7fec5dffb700 1 -- 172.21.15.175:0/122644217 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec68167390 con 0x7fec680b1fc0 2023-10-10T19:19:36.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.308+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec68167390 con 0x7fec680b1fc0 2023-10-10T19:19:36.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.308+0000 7fec5dffb700 1 -- 172.21.15.175:0/122644217 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6199 (secure 0 0 0) 0x7fec68167390 con 0x7fec680b1fc0 2023-10-10T19:19:36.309 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:36.309 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:26.925419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:36.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec4c066f30 msgr2=0x7fec4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:36.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec4c066f30 0x7fec4c0693e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fec640091e0 tx=0x7fec64017000 comp rx=0 tx=0).stop 2023-10-10T19:19:36.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec680b1fc0 msgr2=0x7fec68167bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:36.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec680b1fc0 0x7fec68167bb0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fec5800ec80 tx=0x7fec580091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 shutdown_connections 2023-10-10T19:19:36.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fec4c066f30 0x7fec4c0693e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec680b2390 0x7fec68169610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec68068bd0 0x7fec68167670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 --2- 172.21.15.175:0/122644217 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec680b1fc0 0x7fec68167bb0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:36.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 >> 172.21.15.175:0/122644217 conn(0x7fec68063230 msgr2=0x7fec6814fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:36.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 shutdown_connections 2023-10-10T19:19:36.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:36.311+0000 7fec6fee2700 1 -- 172.21.15.175:0/122644217 wait complete. 2023-10-10T19:19:36.314 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 264 2023-10-10T19:19:36.323 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:38.324 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:38.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.450+0000 7fc863fff700 1 Processor -- start 2023-10-10T19:19:38.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.450+0000 7fc863fff700 1 -- start start 2023-10-10T19:19:38.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc863fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc86414d590 0x7fc86414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:38.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc863fff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86406d8e0 con 0x7fc864157400 2023-10-10T19:19:38.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc863fff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86406dbc0 con 0x7fc8640c4640 2023-10-10T19:19:38.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc863fff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86406e050 con 0x7fc86414d590 2023-10-10T19:19:38.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc8637fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc86414d590 0x7fc86414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:38.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc862ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc864157400 0x7fc86406bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51780/0 (socket says 172.21.15.175:51780) 2023-10-10T19:19:38.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc862ffd700 1 -- 172.21.15.175:0/1253424906 learned_addr learned my addr 172.21.15.175:0/1253424906 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:38.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc8637fe700 1 -- 172.21.15.175:0/1253424906 >> v1:172.21.15.191:6789/0 conn(0x7fc8640c4640 legacy=0x7fc8641577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:38.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc8637fe700 1 -- 172.21.15.175:0/1253424906 >> v1:172.21.15.175:6789/0 conn(0x7fc864157400 legacy=0x7fc86406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:38.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc8637fe700 1 -- 172.21.15.175:0/1253424906 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640c1030 con 0x7fc86414d590 2023-10-10T19:19:38.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.451+0000 7fc8627fc700 1 -- 172.21.15.175:0/1253424906 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099102153 0 0) 0x7fc86406d8e0 con 0x7fc864157400 2023-10-10T19:19:38.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc8637fe700 1 --2- 172.21.15.175:0/1253424906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc86414d590 0x7fc86414d9c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009e60 tx=0x7fc85c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc079b6abebe8296 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:38.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc8627fc700 1 -- 172.21.15.175:0/1253424906 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c00e070 con 0x7fc86414d590 2023-10-10T19:19:38.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc8627fc700 1 -- 172.21.15.175:0/1253424906 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c00bbf0 con 0x7fc86414d590 2023-10-10T19:19:38.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc8627fc700 1 -- 172.21.15.175:0/1253424906 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c012430 con 0x7fc86414d590 2023-10-10T19:19:38.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc863fff700 1 -- 172.21.15.175:0/1253424906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc86414d590 msgr2=0x7fc86414d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:38.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc863fff700 1 --2- 172.21.15.175:0/1253424906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc86414d590 0x7fc86414d9c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009e60 tx=0x7fc85c009780 comp rx=0 tx=0).stop 2023-10-10T19:19:38.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc863fff700 1 -- 172.21.15.175:0/1253424906 shutdown_connections 2023-10-10T19:19:38.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc863fff700 1 --2- 172.21.15.175:0/1253424906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc86414d590 0x7fc86414d9c0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:38.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc863fff700 1 -- 172.21.15.175:0/1253424906 >> 172.21.15.175:0/1253424906 conn(0x7fc864063cd0 msgr2=0x7fc86406d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:38.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc863fff700 1 -- 172.21.15.175:0/1253424906 shutdown_connections 2023-10-10T19:19:38.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.452+0000 7fc863fff700 1 -- 172.21.15.175:0/1253424906 wait complete. 2023-10-10T19:19:38.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc863fff700 1 Processor -- start 2023-10-10T19:19:38.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc863fff700 1 -- start start 2023-10-10T19:19:38.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc863fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c4640 0x7fc8641561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:38.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc863fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc86414d590 0x7fc8640c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:38.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc863fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc864157400 0x7fc864143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:38.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc863fff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640c6860 con 0x7fc8640c4640 2023-10-10T19:19:38.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc863fff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc864145950 con 0x7fc86414d590 2023-10-10T19:19:38.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc863fff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc864163e60 con 0x7fc864157400 2023-10-10T19:19:38.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc862ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c4640 0x7fc8641561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:38.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc862ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c4640 0x7fc8641561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56240/0 (socket says 172.21.15.175:56240) 2023-10-10T19:19:38.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc862ffd700 1 -- 172.21.15.175:0/1457267952 learned_addr learned my addr 172.21.15.175:0/1457267952 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:38.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc8637fe700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc864157400 0x7fc864143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:38.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.454+0000 7fc85a7fc700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc86414d590 0x7fc8640c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:38.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc862ffd700 1 -- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc864157400 msgr2=0x7fc864143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:38.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc862ffd700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc864157400 0x7fc864143b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:38.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc862ffd700 1 -- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc86414d590 msgr2=0x7fc8640c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:38.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc862ffd700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc86414d590 0x7fc8640c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:38.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc862ffd700 1 -- 172.21.15.175:0/1457267952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c009a20 con 0x7fc8640c4640 2023-10-10T19:19:38.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc862ffd700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c4640 0x7fc8641561b0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc85400c5b0 tx=0x7fc854001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:38.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc85b7fe700 1 -- 172.21.15.175:0/1457267952 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854005c00 con 0x7fc8640c4640 2023-10-10T19:19:38.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc85b7fe700 1 -- 172.21.15.175:0/1457267952 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc854005d60 con 0x7fc8640c4640 2023-10-10T19:19:38.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc864164140 con 0x7fc8640c4640 2023-10-10T19:19:38.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc85b7fe700 1 -- 172.21.15.175:0/1457267952 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854007300 con 0x7fc8640c4640 2023-10-10T19:19:38.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.455+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc864145300 con 0x7fc8640c4640 2023-10-10T19:19:38.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.456+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc864067570 con 0x7fc8640c4640 2023-10-10T19:19:38.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.456+0000 7fc85b7fe700 1 -- 172.21.15.175:0/1457267952 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc8540076b0 con 0x7fc8640c4640 2023-10-10T19:19:38.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.457+0000 7fc85b7fe700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc844066f30 0x7fc8440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:38.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.457+0000 7fc85b7fe700 1 -- 172.21.15.175:0/1457267952 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fc8540840b0 con 0x7fc8640c4640 2023-10-10T19:19:38.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.457+0000 7fc85a7fc700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc844066f30 0x7fc8440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:38.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.457+0000 7fc85a7fc700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc844066f30 0x7fc8440693e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc8640c4d40 tx=0x7fc84c007ea0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:38.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.461+0000 7fc85b7fe700 1 -- 172.21.15.175:0/1457267952 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc854050cf0 con 0x7fc8640c4640 2023-10-10T19:19:38.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.690+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8640c1320 con 0x7fc8640c4640 2023-10-10T19:19:38.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.691+0000 7fc85b7fe700 1 -- 172.21.15.175:0/1457267952 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6199 (secure 0 0 0) 0x7fc854054310 con 0x7fc8640c4640 2023-10-10T19:19:38.692 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:38.692 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:26.925419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:38.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc844066f30 msgr2=0x7fc8440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:38.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc844066f30 0x7fc8440693e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc8640c4d40 tx=0x7fc84c007ea0 comp rx=0 tx=0).stop 2023-10-10T19:19:38.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c4640 msgr2=0x7fc8641561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:38.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c4640 0x7fc8641561b0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc85400c5b0 tx=0x7fc854001070 comp rx=0 tx=0).stop 2023-10-10T19:19:38.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 shutdown_connections 2023-10-10T19:19:38.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc844066f30 0x7fc8440693e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:38.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc864157400 0x7fc864143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:38.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8640c4640 0x7fc8641561b0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:38.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 --2- 172.21.15.175:0/1457267952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc86414d590 0x7fc8640c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:38.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 >> 172.21.15.175:0/1457267952 conn(0x7fc864063cd0 msgr2=0x7fc8641479e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:38.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 shutdown_connections 2023-10-10T19:19:38.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:38.693+0000 7fc863fff700 1 -- 172.21.15.175:0/1457267952 wait complete. 2023-10-10T19:19:38.696 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 264 2023-10-10T19:19:38.705 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:40.707 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:40.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.833+0000 7f3b6b1f5700 1 Processor -- start 2023-10-10T19:19:40.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.833+0000 7f3b6b1f5700 1 -- start start 2023-10-10T19:19:40.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6b1f5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3b6414fd90 0x7f3b64159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:40.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6b1f5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b64067fa0 con 0x7f3b640cd040 2023-10-10T19:19:40.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6b1f5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b64156b70 con 0x7f3b6408a100 2023-10-10T19:19:40.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6b1f5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b640c5600 con 0x7f3b6414fd90 2023-10-10T19:19:40.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6a9f4700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f3b640cd040 0x7f3b640684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51786/0 (socket says 172.21.15.175:51786) 2023-10-10T19:19:40.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6a9f4700 1 -- 172.21.15.175:0/4110498290 learned_addr learned my addr 172.21.15.175:0/4110498290 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:40.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6a1f3700 1 --2- 172.21.15.175:0/4110498290 >> v2:172.21.15.191:3300/0 conn(0x7f3b6414fd90 0x7f3b64159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:40.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b691f1700 1 -- 172.21.15.175:0/4110498290 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315646782 0 0) 0x7f3b64067fa0 con 0x7f3b640cd040 2023-10-10T19:19:40.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b691f1700 1 -- 172.21.15.175:0/4110498290 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b50003570 con 0x7f3b640cd040 2023-10-10T19:19:40.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6a1f3700 1 -- 172.21.15.175:0/4110498290 >> v1:172.21.15.191:6789/0 conn(0x7f3b6408a100 legacy=0x7f3b640c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:40.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6a1f3700 1 -- 172.21.15.175:0/4110498290 >> v1:172.21.15.175:6789/0 conn(0x7f3b640cd040 legacy=0x7f3b640684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:40.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.834+0000 7f3b6a1f3700 1 -- 172.21.15.175:0/4110498290 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b64150160 con 0x7f3b6414fd90 2023-10-10T19:19:40.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b6a1f3700 1 --2- 172.21.15.175:0/4110498290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b6414fd90 0x7f3b64159c30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009e60 tx=0x7f3b60009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70377686d1b11360 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:40.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b691f1700 1 -- 172.21.15.175:0/4110498290 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6000e070 con 0x7f3b6414fd90 2023-10-10T19:19:40.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b691f1700 1 -- 172.21.15.175:0/4110498290 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6000bbf0 con 0x7f3b6414fd90 2023-10-10T19:19:40.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b691f1700 1 -- 172.21.15.175:0/4110498290 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60012430 con 0x7f3b6414fd90 2023-10-10T19:19:40.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/4110498290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b6414fd90 msgr2=0x7f3b64159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:40.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b6b1f5700 1 --2- 172.21.15.175:0/4110498290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b6414fd90 0x7f3b64159c30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009e60 tx=0x7f3b60009780 comp rx=0 tx=0).stop 2023-10-10T19:19:40.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/4110498290 shutdown_connections 2023-10-10T19:19:40.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b6b1f5700 1 --2- 172.21.15.175:0/4110498290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b6414fd90 0x7f3b64159c30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:40.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/4110498290 >> 172.21.15.175:0/4110498290 conn(0x7f3b64063d00 msgr2=0x7f3b640cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:40.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/4110498290 shutdown_connections 2023-10-10T19:19:40.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.835+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/4110498290 wait complete. 2023-10-10T19:19:40.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.836+0000 7f3b6b1f5700 1 Processor -- start 2023-10-10T19:19:40.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.836+0000 7f3b6b1f5700 1 -- start start 2023-10-10T19:19:40.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6b1f5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b6408a100 0x7f3b64169830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:40.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6b1f5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b640cd040 0x7f3b6414b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:40.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6b1f5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b6414fd90 0x7f3b6414bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:40.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6b1f5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6414e2a0 con 0x7f3b6408a100 2023-10-10T19:19:40.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6b1f5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b641526a0 con 0x7f3b640cd040 2023-10-10T19:19:40.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6b1f5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b641693a0 con 0x7f3b6414fd90 2023-10-10T19:19:40.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6a1f3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b6408a100 0x7f3b64169830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:40.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6a1f3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b6408a100 0x7f3b64169830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56252/0 (socket says 172.21.15.175:56252) 2023-10-10T19:19:40.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6a1f3700 1 -- 172.21.15.175:0/2618015820 learned_addr learned my addr 172.21.15.175:0/2618015820 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:40.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6a9f4700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b6414fd90 0x7f3b6414bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:40.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b699f2700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b640cd040 0x7f3b6414b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:40.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6a1f3700 1 -- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b6414fd90 msgr2=0x7f3b6414bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:40.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.837+0000 7f3b6a1f3700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b6414fd90 0x7f3b6414bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:40.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b6a1f3700 1 -- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b640cd040 msgr2=0x7f3b6414b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:40.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b6a1f3700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b640cd040 0x7f3b6414b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:40.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b6a1f3700 1 -- 172.21.15.175:0/2618015820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b60009a20 con 0x7f3b6408a100 2023-10-10T19:19:40.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b699f2700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b640cd040 0x7f3b6414b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:40.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b6a1f3700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b6408a100 0x7f3b64169830 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000b6c0 tx=0x7f3b60019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:40.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b5a7fc700 1 -- 172.21.15.175:0/2618015820 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60027470 con 0x7f3b6408a100 2023-10-10T19:19:40.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b5a7fc700 1 -- 172.21.15.175:0/2618015820 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b600275d0 con 0x7f3b6408a100 2023-10-10T19:19:40.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b64159780 con 0x7f3b6408a100 2023-10-10T19:19:40.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b5a7fc700 1 -- 172.21.15.175:0/2618015820 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60027990 con 0x7f3b6408a100 2023-10-10T19:19:40.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.838+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b640cc560 con 0x7f3b6408a100 2023-10-10T19:19:40.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.839+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b64067570 con 0x7f3b6408a100 2023-10-10T19:19:40.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.839+0000 7f3b5a7fc700 1 -- 172.21.15.175:0/2618015820 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3b60024070 con 0x7f3b6408a100 2023-10-10T19:19:40.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.839+0000 7f3b5a7fc700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b48066f60 0x7f3b48069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:40.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.840+0000 7f3b5a7fc700 1 -- 172.21.15.175:0/2618015820 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f3b60087480 con 0x7f3b6408a100 2023-10-10T19:19:40.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.840+0000 7f3b699f2700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b48066f60 0x7f3b48069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:40.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.840+0000 7f3b699f2700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b48066f60 0x7f3b48069410 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3b54007fd0 tx=0x7f3b54009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:40.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:40.844+0000 7f3b5a7fc700 1 -- 172.21.15.175:0/2618015820 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3b600540c0 con 0x7f3b6408a100 2023-10-10T19:19:41.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.069+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b64155f80 con 0x7f3b6408a100 2023-10-10T19:19:41.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.069+0000 7f3b5a7fc700 1 -- 172.21.15.175:0/2618015820 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6199 (secure 0 0 0) 0x7f3b600576e0 con 0x7f3b6408a100 2023-10-10T19:19:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:41.071 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:26.925419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:41.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b48066f60 msgr2=0x7f3b48069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:41.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b48066f60 0x7f3b48069410 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3b54007fd0 tx=0x7f3b54009040 comp rx=0 tx=0).stop 2023-10-10T19:19:41.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b6408a100 msgr2=0x7f3b64169830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:41.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b6408a100 0x7f3b64169830 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000b6c0 tx=0x7f3b60019000 comp rx=0 tx=0).stop 2023-10-10T19:19:41.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 shutdown_connections 2023-10-10T19:19:41.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3b48066f60 0x7f3b48069410 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:41.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b6414fd90 0x7f3b6414bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:41.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b6408a100 0x7f3b64169830 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:41.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 --2- 172.21.15.175:0/2618015820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b640cd040 0x7f3b6414b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:41.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.071+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 >> 172.21.15.175:0/2618015820 conn(0x7f3b64063d00 msgr2=0x7f3b64157fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:41.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.072+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 shutdown_connections 2023-10-10T19:19:41.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:41.072+0000 7f3b6b1f5700 1 -- 172.21.15.175:0/2618015820 wait complete. 2023-10-10T19:19:41.075 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 264 2023-10-10T19:19:41.084 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-10T19:19:41.084 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 264, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 264, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:stopping", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 125, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:19:26.925419+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:19:43.086 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:43.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.211+0000 7f0472cde700 1 Processor -- start 2023-10-10T19:19:43.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0472cde700 1 -- start start 2023-10-10T19:19:43.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0472cde700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f046c0b1fc0 0x7f046c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:43.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0472cde700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:43.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0472cde700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f046c068bd0 0x7f046c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:43.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0472cde700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046c150770 con 0x7f046c0b1fc0 2023-10-10T19:19:43.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0472cde700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046c1574b0 con 0x7f046c068bd0 2023-10-10T19:19:43.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0472cde700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0afad0 con 0x7f046c0b2390 2023-10-10T19:19:43.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0470a7a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f046c0b1fc0 0x7f046c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:43.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0470a7a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f046c0b1fc0 0x7f046c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56266/0 (socket says 172.21.15.175:56266) 2023-10-10T19:19:43.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.212+0000 7f0470a7a700 1 -- 172.21.15.175:0/62640149 learned_addr learned my addr 172.21.15.175:0/62640149 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:43.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f046bfff700 1 --2- 172.21.15.175:0/62640149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:43.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f047127b700 1 --2- 172.21.15.175:0/62640149 >> v2:172.21.15.191:3300/0 conn(0x7f046c068bd0 0x7f046c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:43.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f0470a7a700 1 -- 172.21.15.175:0/62640149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 msgr2=0x7f046c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:43.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f0470a7a700 1 --2- 172.21.15.175:0/62640149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f0470a7a700 1 -- 172.21.15.175:0/62640149 >> v2:172.21.15.191:3300/0 conn(0x7f046c068bd0 msgr2=0x7f046c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:43.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f0470a7a700 1 --2- 172.21.15.175:0/62640149 >> v2:172.21.15.191:3300/0 conn(0x7f046c068bd0 0x7f046c0aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f0470a7a700 1 -- 172.21.15.175:0/62640149 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c0af4c0 con 0x7f046c0b1fc0 2023-10-10T19:19:43.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f047127b700 1 --2- 172.21.15.175:0/62640149 >> v2:172.21.15.191:3300/0 conn(0x7f046c068bd0 0x7f046c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:43.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f046bfff700 1 --2- 172.21.15.175:0/62640149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:43.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f0470a7a700 1 --2- 172.21.15.175:0/62640149 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c155fb0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f046c0afa40 tx=0x7f045c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e0eb65e17d0ace server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:43.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f046b7fe700 1 -- 172.21.15.175:0/62640149 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c00e070 con 0x7f046c0b1fc0 2023-10-10T19:19:43.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f046b7fe700 1 -- 172.21.15.175:0/62640149 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c00bae0 con 0x7f046c0b1fc0 2023-10-10T19:19:43.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.213+0000 7f046b7fe700 1 -- 172.21.15.175:0/62640149 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c0092e0 con 0x7f046c0b1fc0 2023-10-10T19:19:43.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 -- 172.21.15.175:0/62640149 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 msgr2=0x7f046c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:43.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 --2- 172.21.15.175:0/62640149 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c155fb0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f046c0afa40 tx=0x7f045c009840 comp rx=0 tx=0).stop 2023-10-10T19:19:43.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 -- 172.21.15.175:0/62640149 shutdown_connections 2023-10-10T19:19:43.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 --2- 172.21.15.175:0/62640149 >> v2:172.21.15.191:3300/0 conn(0x7f046c068bd0 0x7f046c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 --2- 172.21.15.175:0/62640149 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c155fb0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 --2- 172.21.15.175:0/62640149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 -- 172.21.15.175:0/62640149 >> 172.21.15.175:0/62640149 conn(0x7f046c063230 msgr2=0x7f046c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:43.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 -- 172.21.15.175:0/62640149 shutdown_connections 2023-10-10T19:19:43.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.214+0000 7f0472cde700 1 -- 172.21.15.175:0/62640149 wait complete. 2023-10-10T19:19:43.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.215+0000 7f0472cde700 1 Processor -- start 2023-10-10T19:19:43.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.215+0000 7f0472cde700 1 -- start start 2023-10-10T19:19:43.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.215+0000 7f0472cde700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f046c068bd0 0x7f046c14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:43.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0472cde700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:43.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0472cde700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c0c21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:43.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0472cde700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c154b40 con 0x7f046c0b1fc0 2023-10-10T19:19:43.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0472cde700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0c02c0 con 0x7f046c068bd0 2023-10-10T19:19:43.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0472cde700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c154070 con 0x7f046c0b2390 2023-10-10T19:19:43.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:43.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c14fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56274/0 (socket says 172.21.15.175:56274) 2023-10-10T19:19:43.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 -- 172.21.15.175:0/883553528 learned_addr learned my addr 172.21.15.175:0/883553528 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:43.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0470a7a700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f046c068bd0 0x7f046c14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:43.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f047127b700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c0c21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:43.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 -- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 msgr2=0x7f046c0c21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:43.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c0c21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 -- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f046c068bd0 msgr2=0x7f046c14fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:43.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f046c068bd0 0x7f046c14fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 -- 172.21.15.175:0/883553528 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c009ae0 con 0x7f046c0b1fc0 2023-10-10T19:19:43.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0470a7a700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f046c068bd0 0x7f046c14fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:43.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f046bfff700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c14fff0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f045400d860 tx=0x7f0454008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:43.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0468ff9700 1 -- 172.21.15.175:0/883553528 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454009610 con 0x7f046c0b1fc0 2023-10-10T19:19:43.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0468ff9700 1 -- 172.21.15.175:0/883553528 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0454009770 con 0x7f046c0b1fc0 2023-10-10T19:19:43.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046c154350 con 0x7f046c0b1fc0 2023-10-10T19:19:43.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0468ff9700 1 -- 172.21.15.175:0/883553528 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454009b10 con 0x7f046c0b1fc0 2023-10-10T19:19:43.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.216+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046c0c07f0 con 0x7f046c0b1fc0 2023-10-10T19:19:43.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.217+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046c067570 con 0x7f046c0b1fc0 2023-10-10T19:19:43.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.218+0000 7f0468ff9700 1 -- 172.21.15.175:0/883553528 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0454014430 con 0x7f046c0b1fc0 2023-10-10T19:19:43.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.218+0000 7f0468ff9700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f044c066f30 0x7f044c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:43.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.218+0000 7f0468ff9700 1 -- 172.21.15.175:0/883553528 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f0454084f90 con 0x7f046c0b1fc0 2023-10-10T19:19:43.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.218+0000 7f0470a7a700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f044c066f30 0x7f044c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:43.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.219+0000 7f0470a7a700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f044c066f30 0x7f044c0693e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f045c009880 tx=0x7f045c018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:43.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.222+0000 7f0468ff9700 1 -- 172.21.15.175:0/883553528 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0454051bd0 con 0x7f046c0b1fc0 2023-10-10T19:19:43.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.448+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f046c069060 con 0x7f046c0b1fc0 2023-10-10T19:19:43.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.449+0000 7f0468ff9700 1 -- 172.21.15.175:0/883553528 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6199 (secure 0 0 0) 0x7f04540551f0 con 0x7f046c0b1fc0 2023-10-10T19:19:43.449 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:43.450 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:26.925419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:43.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.450+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f044c066f30 msgr2=0x7f044c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:43.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.450+0000 7f0472cde700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f044c066f30 0x7f044c0693e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f045c009880 tx=0x7f045c018000 comp rx=0 tx=0).stop 2023-10-10T19:19:43.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.450+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 msgr2=0x7f046c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:43.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.450+0000 7f0472cde700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c14fff0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f045400d860 tx=0x7f0454008040 comp rx=0 tx=0).stop 2023-10-10T19:19:43.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.450+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 shutdown_connections 2023-10-10T19:19:43.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.450+0000 7f0472cde700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f044c066f30 0x7f044c0693e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.450+0000 7f0472cde700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f046c0b2390 0x7f046c0c21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.450+0000 7f0472cde700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f046c068bd0 0x7f046c14fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.451+0000 7f0472cde700 1 --2- 172.21.15.175:0/883553528 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f046c0b1fc0 0x7f046c14fff0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:43.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.451+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 >> 172.21.15.175:0/883553528 conn(0x7f046c063230 msgr2=0x7f046c157a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:43.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.451+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 shutdown_connections 2023-10-10T19:19:43.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:43.451+0000 7f0472cde700 1 -- 172.21.15.175:0/883553528 wait complete. 2023-10-10T19:19:43.454 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 264 2023-10-10T19:19:43.463 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:45.464 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:45.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.589+0000 7f0101110700 1 Processor -- start 2023-10-10T19:19:45.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.589+0000 7f0101110700 1 -- start start 2023-10-10T19:19:45.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f0101110700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f00fc06b640 0x7f00fc14cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:45.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f0101110700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00fc0c3700 0x7f00fc14d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:45.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f0101110700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc1533e0 con 0x7f00fc0c3330 2023-10-10T19:19:45.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f0101110700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc14d820 con 0x7f00fc0c3700 2023-10-10T19:19:45.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f0101110700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc06e130 con 0x7f00fc06b640 2023-10-10T19:19:45.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f00fb59e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00fc0c3700 0x7f00fc14d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:45.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f00fb59e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00fc0c3700 0x7f00fc14d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52292/0 (socket says 172.21.15.175:52292) 2023-10-10T19:19:45.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f00fb59e700 1 -- 172.21.15.175:0/571837692 learned_addr learned my addr 172.21.15.175:0/571837692 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:45.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.590+0000 7f00fad9d700 1 --1- 172.21.15.175:0/571837692 >> v1:172.21.15.175:6789/0 conn(0x7f00fc0c3330 0x7f00fc0c3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54622/0 (socket says 172.21.15.175:54622) 2023-10-10T19:19:45.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00f259c700 1 --2- 172.21.15.175:0/571837692 >> v2:172.21.15.191:3300/0 conn(0x7f00fc06b640 0x7f00fc14cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:45.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fb59e700 1 -- 172.21.15.175:0/571837692 >> v2:172.21.15.191:3300/0 conn(0x7f00fc06b640 msgr2=0x7f00fc14cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:45.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fb59e700 1 --2- 172.21.15.175:0/571837692 >> v2:172.21.15.191:3300/0 conn(0x7f00fc06b640 0x7f00fc14cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fb59e700 1 -- 172.21.15.175:0/571837692 >> v1:172.21.15.175:6789/0 conn(0x7f00fc0c3330 legacy=0x7f00fc0c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:45.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fb59e700 1 -- 172.21.15.175:0/571837692 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc06a970 con 0x7f00fc0c3700 2023-10-10T19:19:45.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fa59c700 1 -- 172.21.15.175:0/571837692 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154311305 0 0) 0x7f00fc1533e0 con 0x7f00fc0c3330 2023-10-10T19:19:45.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fb59e700 1 --2- 172.21.15.175:0/571837692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3700 0x7f00fc14d2e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f00fc06ce00 tx=0x7f00f400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a85ed775346c28 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:45.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fa59c700 1 -- 172.21.15.175:0/571837692 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4010070 con 0x7f00fc0c3700 2023-10-10T19:19:45.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fa59c700 1 -- 172.21.15.175:0/571837692 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f400de90 con 0x7f00fc0c3700 2023-10-10T19:19:45.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f00fa59c700 1 -- 172.21.15.175:0/571837692 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f400b320 con 0x7f00fc0c3700 2023-10-10T19:19:45.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f0101110700 1 -- 172.21.15.175:0/571837692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3700 msgr2=0x7f00fc14d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:45.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f0101110700 1 --2- 172.21.15.175:0/571837692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3700 0x7f00fc14d2e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f00fc06ce00 tx=0x7f00f400ba50 comp rx=0 tx=0).stop 2023-10-10T19:19:45.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f0101110700 1 -- 172.21.15.175:0/571837692 shutdown_connections 2023-10-10T19:19:45.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f0101110700 1 --2- 172.21.15.175:0/571837692 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3700 0x7f00fc14d2e0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f0101110700 1 --2- 172.21.15.175:0/571837692 >> v2:172.21.15.191:3300/0 conn(0x7f00fc06b640 0x7f00fc14cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.591+0000 7f0101110700 1 -- 172.21.15.175:0/571837692 >> 172.21.15.175:0/571837692 conn(0x7f00fc068280 msgr2=0x7f00fc06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:45.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.592+0000 7f0101110700 1 -- 172.21.15.175:0/571837692 shutdown_connections 2023-10-10T19:19:45.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.592+0000 7f0101110700 1 -- 172.21.15.175:0/571837692 wait complete. 2023-10-10T19:19:45.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.593+0000 7f0101110700 1 Processor -- start 2023-10-10T19:19:45.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.593+0000 7f0101110700 1 -- start start 2023-10-10T19:19:45.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.593+0000 7f0101110700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00fc06b640 0x7f00fc1499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:45.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.593+0000 7f0101110700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3330 0x7f00fc0c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:45.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.593+0000 7f0101110700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00fc0c3700 0x7f00fc145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:45.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.593+0000 7f0101110700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc14b0f0 con 0x7f00fc0c3330 2023-10-10T19:19:45.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.593+0000 7f0101110700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc165950 con 0x7f00fc0c3700 2023-10-10T19:19:45.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.593+0000 7f0101110700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc149f20 con 0x7f00fc06b640 2023-10-10T19:19:45.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3330 0x7f00fc0c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:45.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3330 0x7f00fc0c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52294/0 (socket says 172.21.15.175:52294) 2023-10-10T19:19:45.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 -- 172.21.15.175:0/4141314120 learned_addr learned my addr 172.21.15.175:0/4141314120 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:45.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00fad9d700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00fc06b640 0x7f00fc1499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:45.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00fb59e700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00fc0c3700 0x7f00fc145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:45.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 -- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00fc06b640 msgr2=0x7f00fc1499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:45.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00fc06b640 0x7f00fc1499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 -- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00fc0c3700 msgr2=0x7f00fc145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:45.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00fc0c3700 0x7f00fc145c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 -- 172.21.15.175:0/4141314120 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f400bcf0 con 0x7f00fc0c3330 2023-10-10T19:19:45.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f259c700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3330 0x7f00fc0c6260 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f00e400d890 tx=0x7f00e4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:45.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f37fe700 1 -- 172.21.15.175:0/4141314120 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e40125f0 con 0x7f00fc0c3330 2023-10-10T19:19:45.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f37fe700 1 -- 172.21.15.175:0/4141314120 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e4012750 con 0x7f00fc0c3330 2023-10-10T19:19:45.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc165de0 con 0x7f00fc0c3330 2023-10-10T19:19:45.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f00f37fe700 1 -- 172.21.15.175:0/4141314120 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4012af0 con 0x7f00fc0c3330 2023-10-10T19:19:45.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.594+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc14a950 con 0x7f00fc0c3330 2023-10-10T19:19:45.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.595+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc069740 con 0x7f00fc0c3330 2023-10-10T19:19:45.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.596+0000 7f00f37fe700 1 -- 172.21.15.175:0/4141314120 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f00e401e460 con 0x7f00fc0c3330 2023-10-10T19:19:45.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.596+0000 7f00f37fe700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f00dc066f60 0x7f00dc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:45.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.596+0000 7f00f37fe700 1 -- 172.21.15.175:0/4141314120 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f00e4085090 con 0x7f00fc0c3330 2023-10-10T19:19:45.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.596+0000 7f00fad9d700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f00dc066f60 0x7f00dc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:45.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.597+0000 7f00fad9d700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f00dc066f60 0x7f00dc069410 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0012a0 tx=0x7f00ec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:45.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.600+0000 7f00f37fe700 1 -- 172.21.15.175:0/4141314120 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00e4051cd0 con 0x7f00fc0c3330 2023-10-10T19:19:45.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.824+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00fc067570 con 0x7f00fc0c3330 2023-10-10T19:19:45.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.824+0000 7f00f37fe700 1 -- 172.21.15.175:0/4141314120 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6199 (secure 0 0 0) 0x7f00e40552f0 con 0x7f00fc0c3330 2023-10-10T19:19:45.825 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:45.825 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:26.925419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:45.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f00dc066f60 msgr2=0x7f00dc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:45.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f00dc066f60 0x7f00dc069410 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0012a0 tx=0x7f00ec009040 comp rx=0 tx=0).stop 2023-10-10T19:19:45.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3330 msgr2=0x7f00fc0c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:45.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3330 0x7f00fc0c6260 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f00e400d890 tx=0x7f00e4008040 comp rx=0 tx=0).stop 2023-10-10T19:19:45.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 shutdown_connections 2023-10-10T19:19:45.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f00dc066f60 0x7f00dc069410 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00fc0c3700 0x7f00fc145c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00fc06b640 0x7f00fc1499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 --2- 172.21.15.175:0/4141314120 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00fc0c3330 0x7f00fc0c6260 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:45.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.826+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 >> 172.21.15.175:0/4141314120 conn(0x7f00fc068280 msgr2=0x7f00fc069b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:45.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.827+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 shutdown_connections 2023-10-10T19:19:45.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:45.827+0000 7f0101110700 1 -- 172.21.15.175:0/4141314120 wait complete. 2023-10-10T19:19:45.830 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 264 2023-10-10T19:19:45.840 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:47.841 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:47.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.966+0000 7fefd35a6700 1 Processor -- start 2023-10-10T19:19:47.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.966+0000 7fefd35a6700 1 -- start start 2023-10-10T19:19:47.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.967+0000 7fefd35a6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fefcc0c5850 0x7fefcc0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.967+0000 7fefd35a6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc0c12d0 0x7fefcc156660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.967+0000 7fefd35a6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefcc0c6fb0 con 0x7fefcc156ba0 2023-10-10T19:19:47.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.967+0000 7fefd35a6700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc063ea0 con 0x7fefcc0c5850 2023-10-10T19:19:47.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.967+0000 7fefd35a6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc155f30 con 0x7fefcc0c12d0 2023-10-10T19:19:47.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.967+0000 7fefd1b43700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fefcc156ba0 0x7fefcc0c7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54630/0 (socket says 172.21.15.175:54630) 2023-10-10T19:19:47.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.967+0000 7fefd1b43700 1 -- 172.21.15.175:0/3508560623 learned_addr learned my addr 172.21.15.175:0/3508560623 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:47.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd1342700 1 --2- 172.21.15.175:0/3508560623 >> v2:172.21.15.191:3300/0 conn(0x7fefcc0c5850 0x7fefcc0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd0b41700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc0c12d0 0x7fefcc156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963518740 0 0) 0x7fefcc0c6fb0 con 0x7fefcc156ba0 2023-10-10T19:19:47.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb4003570 con 0x7fefcc156ba0 2023-10-10T19:19:47.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 225300714 0 0) 0x7fefb4003570 con 0x7fefcc156ba0 2023-10-10T19:19:47.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc0c12d0 msgr2=0x7fefcc156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:47.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc0c12d0 0x7fefcc156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:47.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 >> v2:172.21.15.191:3300/0 conn(0x7fefcc0c5850 msgr2=0x7fefcc0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:47.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 --2- 172.21.15.175:0/3508560623 >> v2:172.21.15.191:3300/0 conn(0x7fefcc0c5850 0x7fefcc0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:47.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc0c6160 con 0x7fefcc156ba0 2023-10-10T19:19:47.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fefc4001470 con 0x7fefcc156ba0 2023-10-10T19:19:47.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 >> v1:172.21.15.175:6789/0 conn(0x7fefcc156ba0 legacy=0x7fefcc0c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:47.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb4003a60 0x7fefb4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb40051c0 0x7fefb400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc063ea0 con 0x7fefb4004810 2023-10-10T19:19:47.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc155f30 con 0x7fefb4003a60 2023-10-10T19:19:47.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb400bfa0 con 0x7fefb40051c0 2023-10-10T19:19:47.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd1342700 1 --2- 172.21.15.175:0/3508560623 >> v2:172.21.15.191:3300/0 conn(0x7fefcc0c5850 0x7fefcc0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:47.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd0b41700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc0c12d0 0x7fefcc156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:47.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd0b41700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd1b43700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb40051c0 0x7fefb400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd1342700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb4003a60 0x7fefb4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd0b41700 1 -- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb40051c0 msgr2=0x7fefb400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:47.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.968+0000 7fefd0b41700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb40051c0 0x7fefb400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:47.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd0b41700 1 -- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb4003a60 msgr2=0x7fefb4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:47.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd0b41700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb4003a60 0x7fefb4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:47.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd0b41700 1 -- 172.21.15.175:0/3508560623 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4003570 con 0x7fefb4004810 2023-10-10T19:19:47.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd0b41700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefb4004c80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009c50 tx=0x7fefc00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d141b5d11d1d58b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:47.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc000e710 con 0x7fefb4004810 2023-10-10T19:19:47.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc000e870 con 0x7fefb4004810 2023-10-10T19:19:47.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefcbfff700 1 -- 172.21.15.175:0/3508560623 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc000ec10 con 0x7fefb4004810 2023-10-10T19:19:47.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd35a6700 1 -- 172.21.15.175:0/3508560623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 msgr2=0x7fefb4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:47.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd35a6700 1 --2- 172.21.15.175:0/3508560623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefb4004c80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009c50 tx=0x7fefc00091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:47.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd1b43700 1 -- 172.21.15.175:0/3508560623 reap_dead start 2023-10-10T19:19:47.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd35a6700 1 -- 172.21.15.175:0/3508560623 shutdown_connections 2023-10-10T19:19:47.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd35a6700 1 -- 172.21.15.175:0/3508560623 >> 172.21.15.175:0/3508560623 conn(0x7fefcc06dc70 msgr2=0x7fefcc063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:47.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.969+0000 7fefd35a6700 1 -- 172.21.15.175:0/3508560623 shutdown_connections 2023-10-10T19:19:47.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.970+0000 7fefd35a6700 1 -- 172.21.15.175:0/3508560623 wait complete. 2023-10-10T19:19:47.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.970+0000 7fefd35a6700 1 Processor -- start 2023-10-10T19:19:47.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd35a6700 1 -- start start 2023-10-10T19:19:47.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd35a6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefcc0c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd35a6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc158be0 0x7fefcc14b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd35a6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc159030 0x7fefcc14ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd35a6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0c2510 con 0x7fefb4004810 2023-10-10T19:19:47.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd35a6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc158750 con 0x7fefcc159030 2023-10-10T19:19:47.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd35a6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc14bf80 con 0x7fefcc158be0 2023-10-10T19:19:47.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd1342700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefcc0c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd1342700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefcc0c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52310/0 (socket says 172.21.15.175:52310) 2023-10-10T19:19:47.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.971+0000 7fefd1342700 1 -- 172.21.15.175:0/1537395451 learned_addr learned my addr 172.21.15.175:0/1537395451 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:47.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd0b41700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc158be0 0x7fefcc14b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd1b43700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc159030 0x7fefcc14ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd1342700 1 -- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc158be0 msgr2=0x7fefcc14b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:47.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd1342700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc158be0 0x7fefcc14b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:47.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd1342700 1 -- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc159030 msgr2=0x7fefcc14ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:47.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd1342700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc159030 0x7fefcc14ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:47.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd1342700 1 -- 172.21.15.175:0/1537395451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc0c1cb0 con 0x7fefb4004810 2023-10-10T19:19:47.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd1b43700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc159030 0x7fefcc14ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:47.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd1342700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefcc0c3d90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00f340 tx=0x7fefbc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:47.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefc97fa700 1 -- 172.21.15.175:0/1537395451 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc00e040 con 0x7fefb4004810 2023-10-10T19:19:47.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefc97fa700 1 -- 172.21.15.175:0/1537395451 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc00f040 con 0x7fefb4004810 2023-10-10T19:19:47.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc0c1f50 con 0x7fefb4004810 2023-10-10T19:19:47.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefc97fa700 1 -- 172.21.15.175:0/1537395451 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc015680 con 0x7fefb4004810 2023-10-10T19:19:47.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.972+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc1583e0 con 0x7fefb4004810 2023-10-10T19:19:47.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.973+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc067570 con 0x7fefb4004810 2023-10-10T19:19:47.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.973+0000 7fefc97fa700 1 -- 172.21.15.175:0/1537395451 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fefbc015a30 con 0x7fefb4004810 2023-10-10T19:19:47.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.974+0000 7fefc97fa700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb8066f60 0x7fefb8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:47.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.974+0000 7fefc97fa700 1 -- 172.21.15.175:0/1537395451 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fefbc085c00 con 0x7fefb4004810 2023-10-10T19:19:47.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.974+0000 7fefd0b41700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb8066f60 0x7fefb8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:47.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.974+0000 7fefd0b41700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb8066f60 0x7fefb8069410 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fefcc15d610 tx=0x7fefc0019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:47.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:47.978+0000 7fefc97fa700 1 -- 172.21.15.175:0/1537395451 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fefbc08c080 con 0x7fefb4004810 2023-10-10T19:19:48.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.203+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefcc063830 con 0x7fefb4004810 2023-10-10T19:19:48.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.203+0000 7fefc97fa700 1 -- 172.21.15.175:0/1537395451 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 265 v265) v1 ==== 95+0+6198 (secure 0 0 0) 0x7fefbc052840 con 0x7fefb4004810 2023-10-10T19:19:48.204 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:48.204 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":265,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":265,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:45.827946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:48.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb8066f60 msgr2=0x7fefb8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:48.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb8066f60 0x7fefb8069410 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fefcc15d610 tx=0x7fefc0019000 comp rx=0 tx=0).stop 2023-10-10T19:19:48.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 msgr2=0x7fefcc0c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:48.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefcc0c3d90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00f340 tx=0x7fefbc00b040 comp rx=0 tx=0).stop 2023-10-10T19:19:48.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 shutdown_connections 2023-10-10T19:19:48.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb8066f60 0x7fefb8069410 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:48.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc159030 0x7fefcc14ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:48.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb4004810 0x7fefcc0c3d90 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:48.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 --2- 172.21.15.175:0/1537395451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc158be0 0x7fefcc14b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:48.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 >> 172.21.15.175:0/1537395451 conn(0x7fefcc06dc70 msgr2=0x7fefcc154560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:48.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 shutdown_connections 2023-10-10T19:19:48.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:48.206+0000 7fefd35a6700 1 -- 172.21.15.175:0/1537395451 wait complete. 2023-10-10T19:19:48.209 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 265 2023-10-10T19:19:48.218 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:50.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:50.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.346+0000 7f31aba62700 1 Processor -- start 2023-10-10T19:19:50.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.346+0000 7f31aba62700 1 -- start start 2023-10-10T19:19:50.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f31aba62700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f31a4155fb0 0x7f31a4159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:50.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f31aba62700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a41574b0 con 0x7f31a40b2390 2023-10-10T19:19:50.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f31aba62700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a40afad0 con 0x7f31a40b1fc0 2023-10-10T19:19:50.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f31aba62700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a4150770 con 0x7f31a4155fb0 2023-10-10T19:19:50.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f31a8ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f31a40b2390 0x7f31a40aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54644/0 (socket says 172.21.15.175:54644) 2023-10-10T19:19:50.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f31a8ffd700 1 -- 172.21.15.175:0/1895923061 learned_addr learned my addr 172.21.15.175:0/1895923061 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:50.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f31a9fff700 1 --2- 172.21.15.175:0/1895923061 >> v2:172.21.15.191:3300/0 conn(0x7f31a4155fb0 0x7f31a4159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:50.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f319bfff700 1 -- 172.21.15.175:0/1895923061 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436195312 0 0) 0x7f31a41574b0 con 0x7f31a40b2390 2023-10-10T19:19:50.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.347+0000 7f319bfff700 1 -- 172.21.15.175:0/1895923061 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3190003570 con 0x7f31a40b2390 2023-10-10T19:19:50.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.348+0000 7f31a9fff700 1 -- 172.21.15.175:0/1895923061 >> v1:172.21.15.191:6789/0 conn(0x7f31a40b1fc0 legacy=0x7f31a4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:50.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.348+0000 7f31a9fff700 1 -- 172.21.15.175:0/1895923061 >> v1:172.21.15.175:6789/0 conn(0x7f31a40b2390 legacy=0x7f31a40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:50.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.348+0000 7f31a9fff700 1 -- 172.21.15.175:0/1895923061 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a40af4c0 con 0x7f31a4155fb0 2023-10-10T19:19:50.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.348+0000 7f31a9fff700 1 --2- 172.21.15.175:0/1895923061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 0x7f31a4159920 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f319c0098e0 tx=0x7f319c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8a3975155bbe0c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:50.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.348+0000 7f319bfff700 1 -- 172.21.15.175:0/1895923061 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c00b6d0 con 0x7f31a4155fb0 2023-10-10T19:19:50.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.348+0000 7f319bfff700 1 -- 172.21.15.175:0/1895923061 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c00b830 con 0x7f31a4155fb0 2023-10-10T19:19:50.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.348+0000 7f319bfff700 1 -- 172.21.15.175:0/1895923061 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c00bbd0 con 0x7f31a4155fb0 2023-10-10T19:19:50.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.349+0000 7f31aba62700 1 -- 172.21.15.175:0/1895923061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 msgr2=0x7f31a4159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:50.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.349+0000 7f31aba62700 1 --2- 172.21.15.175:0/1895923061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 0x7f31a4159920 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f319c0098e0 tx=0x7f319c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:50.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.349+0000 7f31aba62700 1 -- 172.21.15.175:0/1895923061 shutdown_connections 2023-10-10T19:19:50.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.349+0000 7f31aba62700 1 --2- 172.21.15.175:0/1895923061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 0x7f31a4159920 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:50.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.349+0000 7f31aba62700 1 -- 172.21.15.175:0/1895923061 >> 172.21.15.175:0/1895923061 conn(0x7f31a4063230 msgr2=0x7f31a4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:50.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.349+0000 7f31aba62700 1 -- 172.21.15.175:0/1895923061 shutdown_connections 2023-10-10T19:19:50.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.349+0000 7f31aba62700 1 -- 172.21.15.175:0/1895923061 wait complete. 2023-10-10T19:19:50.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.350+0000 7f31aba62700 1 Processor -- start 2023-10-10T19:19:50.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.350+0000 7f31aba62700 1 -- start start 2023-10-10T19:19:50.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31aba62700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31a40b1fc0 0x7f31a4159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:50.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31aba62700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a40b2390 0x7f31a40c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:50.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31aba62700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 0x7f31a40c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:50.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31aba62700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a4158540 con 0x7f31a40b2390 2023-10-10T19:19:50.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31aba62700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a4158de0 con 0x7f31a4155fb0 2023-10-10T19:19:50.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31aba62700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40cd210 con 0x7f31a40b1fc0 2023-10-10T19:19:50.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a40b2390 0x7f31a40c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:50.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a40b2390 0x7f31a40c55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52314/0 (socket says 172.21.15.175:52314) 2023-10-10T19:19:50.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 -- 172.21.15.175:0/1734834824 learned_addr learned my addr 172.21.15.175:0/1734834824 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:50.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a9fff700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 0x7f31a40c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:50.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a97fe700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31a40b1fc0 0x7f31a4159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:50.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 -- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31a40b1fc0 msgr2=0x7f31a4159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:50.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31a40b1fc0 0x7f31a4159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:50.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 -- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 msgr2=0x7f31a40c5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:50.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 0x7f31a40c5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:50.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 -- 172.21.15.175:0/1734834824 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c009420 con 0x7f31a40b2390 2023-10-10T19:19:50.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a9fff700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 0x7f31a40c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:19:50.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31a8ffd700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a40b2390 0x7f31a40c55b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f319400c7f0 tx=0x7f3194007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:50.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31997fa700 1 -- 172.21.15.175:0/1734834824 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3194005b30 con 0x7f31a40b2390 2023-10-10T19:19:50.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31997fa700 1 -- 172.21.15.175:0/1734834824 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3194005c90 con 0x7f31a40b2390 2023-10-10T19:19:50.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31997fa700 1 -- 172.21.15.175:0/1734834824 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3194007340 con 0x7f31a40b2390 2023-10-10T19:19:50.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a40c3a10 con 0x7f31a40b2390 2023-10-10T19:19:50.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.351+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a4158a70 con 0x7f31a40b2390 2023-10-10T19:19:50.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.352+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a4067570 con 0x7f31a40b2390 2023-10-10T19:19:50.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.353+0000 7f31997fa700 1 -- 172.21.15.175:0/1734834824 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f319401b400 con 0x7f31a40b2390 2023-10-10T19:19:50.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.353+0000 7f31997fa700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3188066f30 0x7f31880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:50.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.353+0000 7f31997fa700 1 -- 172.21.15.175:0/1734834824 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f3194084660 con 0x7f31a40b2390 2023-10-10T19:19:50.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.353+0000 7f31a97fe700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3188066f30 0x7f31880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:50.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.354+0000 7f31a97fe700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3188066f30 0x7f31880693e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f31a0007fd0 tx=0x7f31a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:50.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.358+0000 7f31997fa700 1 -- 172.21.15.175:0/1734834824 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31940513d0 con 0x7f31a40b2390 2023-10-10T19:19:50.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.583+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31a4153400 con 0x7f31a40b2390 2023-10-10T19:19:50.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.583+0000 7f31997fa700 1 -- 172.21.15.175:0/1734834824 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 265 v265) v1 ==== 95+0+6198 (secure 0 0 0) 0x7f31940076f0 con 0x7f31a40b2390 2023-10-10T19:19:50.584 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:50.584 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":265,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":265,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:45.827946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:50.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.585+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3188066f30 msgr2=0x7f31880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:50.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.585+0000 7f31aba62700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3188066f30 0x7f31880693e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f31a0007fd0 tx=0x7f31a0009040 comp rx=0 tx=0).stop 2023-10-10T19:19:50.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.585+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a40b2390 msgr2=0x7f31a40c55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:50.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.585+0000 7f31aba62700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a40b2390 0x7f31a40c55b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f319400c7f0 tx=0x7f3194007e00 comp rx=0 tx=0).stop 2023-10-10T19:19:50.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.586+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 shutdown_connections 2023-10-10T19:19:50.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.586+0000 7f31aba62700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3188066f30 0x7f31880693e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:50.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.586+0000 7f31aba62700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31a4155fb0 0x7f31a40c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:50.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.586+0000 7f31aba62700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31a40b1fc0 0x7f31a4159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:50.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.586+0000 7f31aba62700 1 --2- 172.21.15.175:0/1734834824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a40b2390 0x7f31a40c55b0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:50.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.586+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 >> 172.21.15.175:0/1734834824 conn(0x7f31a4063230 msgr2=0x7f31a4152750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:50.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.586+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 shutdown_connections 2023-10-10T19:19:50.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:50.586+0000 7f31aba62700 1 -- 172.21.15.175:0/1734834824 wait complete. 2023-10-10T19:19:50.589 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 265 2023-10-10T19:19:50.598 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:52.599 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:52.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.725+0000 7fa820caa700 1 Processor -- start 2023-10-10T19:19:52.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.725+0000 7fa820caa700 1 -- start start 2023-10-10T19:19:52.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.726+0000 7fa820caa700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa81c14cc10 0x7fa81c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:52.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.726+0000 7fa820caa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c0c52a0 0x7fa81c152970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:52.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.726+0000 7fa820caa700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c155ca0 con 0x7fa81c14c840 2023-10-10T19:19:52.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.726+0000 7fa820caa700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c156b00 con 0x7fa81c14cc10 2023-10-10T19:19:52.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.726+0000 7fa820caa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c06d700 con 0x7fa81c0c52a0 2023-10-10T19:19:52.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa819d9b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa81c14cc10 0x7fa81c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:52.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81ad9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c0c52a0 0x7fa81c152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:52.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81a59c700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa81c14c840 0x7fa81c0c4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:57356/0 (socket says 172.21.15.175:57356) 2023-10-10T19:19:52.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81a59c700 1 -- 172.21.15.175:0/823256429 learned_addr learned my addr 172.21.15.175:0/823256429 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:52.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81ad9d700 1 -- 172.21.15.175:0/823256429 >> v2:172.21.15.191:3300/0 conn(0x7fa81c14cc10 msgr2=0x7fa81c06e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:52.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81ad9d700 1 --2- 172.21.15.175:0/823256429 >> v2:172.21.15.191:3300/0 conn(0x7fa81c14cc10 0x7fa81c06e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81ad9d700 1 -- 172.21.15.175:0/823256429 >> v1:172.21.15.191:6789/0 conn(0x7fa81c14c840 legacy=0x7fa81c0c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:52.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81ad9d700 1 -- 172.21.15.175:0/823256429 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c155ef0 con 0x7fa81c0c52a0 2023-10-10T19:19:52.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81959a700 1 -- 172.21.15.175:0/823256429 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051997198 0 0) 0x7fa81c155ca0 con 0x7fa81c14c840 2023-10-10T19:19:52.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa819d9b700 1 --2- 172.21.15.175:0/823256429 >> v2:172.21.15.191:3300/0 conn(0x7fa81c14cc10 0x7fa81c06e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:52.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81ad9d700 1 --2- 172.21.15.175:0/823256429 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c0c52a0 0x7fa81c152970 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa8140098e0 tx=0x7fa8140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=760230f3bbb0c3d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:52.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81959a700 1 -- 172.21.15.175:0/823256429 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81400b6d0 con 0x7fa81c0c52a0 2023-10-10T19:19:52.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81959a700 1 -- 172.21.15.175:0/823256429 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81400b830 con 0x7fa81c0c52a0 2023-10-10T19:19:52.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.727+0000 7fa81959a700 1 -- 172.21.15.175:0/823256429 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81400bbd0 con 0x7fa81c0c52a0 2023-10-10T19:19:52.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.728+0000 7fa820caa700 1 -- 172.21.15.175:0/823256429 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c0c52a0 msgr2=0x7fa81c152970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:52.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.728+0000 7fa820caa700 1 --2- 172.21.15.175:0/823256429 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c0c52a0 0x7fa81c152970 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa8140098e0 tx=0x7fa8140091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.728+0000 7fa820caa700 1 -- 172.21.15.175:0/823256429 shutdown_connections 2023-10-10T19:19:52.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.728+0000 7fa820caa700 1 --2- 172.21.15.175:0/823256429 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c0c52a0 0x7fa81c152970 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.728+0000 7fa820caa700 1 --2- 172.21.15.175:0/823256429 >> v2:172.21.15.191:3300/0 conn(0x7fa81c14cc10 0x7fa81c06e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.728+0000 7fa820caa700 1 -- 172.21.15.175:0/823256429 >> 172.21.15.175:0/823256429 conn(0x7fa81c068280 msgr2=0x7fa81c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:52.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.728+0000 7fa820caa700 1 -- 172.21.15.175:0/823256429 shutdown_connections 2023-10-10T19:19:52.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.728+0000 7fa820caa700 1 -- 172.21.15.175:0/823256429 wait complete. 2023-10-10T19:19:52.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.729+0000 7fa820caa700 1 Processor -- start 2023-10-10T19:19:52.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.729+0000 7fa820caa700 1 -- start start 2023-10-10T19:19:52.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa820caa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa81c0c52a0 0x7fa81c1597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:52.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa820caa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81c14c840 0x7fa81c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:52.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa820caa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c14cc10 0x7fa81c169480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:52.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa820caa700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c151b20 con 0x7fa81c0c52a0 2023-10-10T19:19:52.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa820caa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c154520 con 0x7fa81c14c840 2023-10-10T19:19:52.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa820caa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c151510 con 0x7fa81c14cc10 2023-10-10T19:19:52.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa81c0c52a0 0x7fa81c1597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:52.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa81c0c52a0 0x7fa81c1597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52322/0 (socket says 172.21.15.175:52322) 2023-10-10T19:19:52.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81a59c700 1 -- 172.21.15.175:0/3661895533 learned_addr learned my addr 172.21.15.175:0/3661895533 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:52.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa819d9b700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81c14c840 0x7fa81c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:52.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81ad9d700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c14cc10 0x7fa81c169480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:52.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81a59c700 1 -- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c14cc10 msgr2=0x7fa81c169480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:52.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81a59c700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c14cc10 0x7fa81c169480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81a59c700 1 -- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81c14c840 msgr2=0x7fa81c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:52.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81a59c700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81c14c840 0x7fa81c151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.730+0000 7fa81a59c700 1 -- 172.21.15.175:0/3661895533 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa810009420 con 0x7fa81c0c52a0 2023-10-10T19:19:52.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.731+0000 7fa81a59c700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa81c0c52a0 0x7fa81c1597d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00cd50 tx=0x7fa80c001af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:52.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.731+0000 7fa80a7fc700 1 -- 172.21.15.175:0/3661895533 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00ce50 con 0x7fa81c0c52a0 2023-10-10T19:19:52.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.731+0000 7fa80a7fc700 1 -- 172.21.15.175:0/3661895533 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80c00aac0 con 0x7fa81c0c52a0 2023-10-10T19:19:52.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.731+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa814009420 con 0x7fa81c0c52a0 2023-10-10T19:19:52.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.731+0000 7fa80a7fc700 1 -- 172.21.15.175:0/3661895533 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00ae80 con 0x7fa81c0c52a0 2023-10-10T19:19:52.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.731+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c1547c0 con 0x7fa81c0c52a0 2023-10-10T19:19:52.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.732+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c067570 con 0x7fa81c0c52a0 2023-10-10T19:19:52.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.732+0000 7fa80a7fc700 1 -- 172.21.15.175:0/3661895533 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa80c012610 con 0x7fa81c0c52a0 2023-10-10T19:19:52.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.732+0000 7fa80a7fc700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7fc066f60 0x7fa7fc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:52.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.733+0000 7fa80a7fc700 1 -- 172.21.15.175:0/3661895533 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fa80c084ea0 con 0x7fa81c0c52a0 2023-10-10T19:19:52.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.733+0000 7fa819d9b700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7fc066f60 0x7fa7fc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:52.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.733+0000 7fa819d9b700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7fc066f60 0x7fa7fc069410 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa81c160830 tx=0x7fa8100091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:52.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.737+0000 7fa80a7fc700 1 -- 172.21.15.175:0/3661895533 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa80c051ae0 con 0x7fa81c0c52a0 2023-10-10T19:19:52.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.963+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa81c154a80 con 0x7fa81c0c52a0 2023-10-10T19:19:52.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.964+0000 7fa80a7fc700 1 -- 172.21.15.175:0/3661895533 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 265 v265) v1 ==== 95+0+6198 (secure 0 0 0) 0x7fa80c055100 con 0x7fa81c0c52a0 2023-10-10T19:19:52.965 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:52.965 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":265,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":265,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:45.827946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:52.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7fc066f60 msgr2=0x7fa7fc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:52.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7fc066f60 0x7fa7fc069410 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa81c160830 tx=0x7fa8100091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa81c0c52a0 msgr2=0x7fa81c1597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:52.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa81c0c52a0 0x7fa81c1597d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00cd50 tx=0x7fa80c001af0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 shutdown_connections 2023-10-10T19:19:52.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa7fc066f60 0x7fa7fc069410 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa81c14cc10 0x7fa81c169480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa81c0c52a0 0x7fa81c1597d0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 --2- 172.21.15.175:0/3661895533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa81c14c840 0x7fa81c151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:52.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 >> 172.21.15.175:0/3661895533 conn(0x7fa81c068280 msgr2=0x7fa81c06e780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:52.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 shutdown_connections 2023-10-10T19:19:52.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:52.967+0000 7fa820caa700 1 -- 172.21.15.175:0/3661895533 wait complete. 2023-10-10T19:19:52.971 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 265 2023-10-10T19:19:52.980 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-10T19:19:52.981 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 265, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 265, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6373": { "addr": "172.21.15.191:6839/4045179206", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 4045179206, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 4045179206, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138320759226367, "flags": 0, "gid": 6373, "incarnation": 195, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:stopping", "state_seq": 19 }, "gid_6511": { "addr": "172.21.15.191:6837/1851345295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 1851345295, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 1851345295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6511, "incarnation": 204, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 19 }, "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 125, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:19:45.827946+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6511, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618, "mds_4": 6373 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:19:54.892 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:19:54.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:19:54.982 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:55.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.105+0000 7fc41c1d4700 1 Processor -- start 2023-10-10T19:19:55.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.105+0000 7fc41c1d4700 1 -- start start 2023-10-10T19:19:55.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.105+0000 7fc41c1d4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4140b1fc0 0x7fc414155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:55.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41c1d4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc4140b2390 0x7fc414068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:55.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41c1d4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc414068bd0 0x7fc4140aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:55.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41c1d4700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc414150770 con 0x7fc414068bd0 2023-10-10T19:19:55.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41c1d4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4141574b0 con 0x7fc4140b2390 2023-10-10T19:19:55.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41c1d4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140afad0 con 0x7fc4140b1fc0 2023-10-10T19:19:55.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc414068bd0 0x7fc4140aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:55.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc414068bd0 0x7fc4140aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55626/0 (socket says 172.21.15.175:55626) 2023-10-10T19:19:55.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 -- 172.21.15.175:0/810738894 learned_addr learned my addr 172.21.15.175:0/810738894 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:55.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41976f700 1 --2- 172.21.15.175:0/810738894 >> v2:172.21.15.191:3300/0 conn(0x7fc4140b2390 0x7fc414068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:55.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc419f70700 1 --2- 172.21.15.175:0/810738894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4140b1fc0 0x7fc414155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:55.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 -- 172.21.15.175:0/810738894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4140b1fc0 msgr2=0x7fc414155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:55.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 --2- 172.21.15.175:0/810738894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4140b1fc0 0x7fc414155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 -- 172.21.15.175:0/810738894 >> v2:172.21.15.191:3300/0 conn(0x7fc4140b2390 msgr2=0x7fc414068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:55.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 --2- 172.21.15.175:0/810738894 >> v2:172.21.15.191:3300/0 conn(0x7fc4140b2390 0x7fc414068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 -- 172.21.15.175:0/810738894 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4140af4c0 con 0x7fc414068bd0 2023-10-10T19:19:55.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc419f70700 1 --2- 172.21.15.175:0/810738894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4140b1fc0 0x7fc414155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:19:55.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc41a771700 1 --2- 172.21.15.175:0/810738894 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc414068bd0 0x7fc4140aff60 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fc414071640 tx=0x7fc40c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca23b84335a0ec33 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:55.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc418f6e700 1 -- 172.21.15.175:0/810738894 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c00e070 con 0x7fc414068bd0 2023-10-10T19:19:55.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc418f6e700 1 -- 172.21.15.175:0/810738894 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40c00bab0 con 0x7fc414068bd0 2023-10-10T19:19:55.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.106+0000 7fc418f6e700 1 -- 172.21.15.175:0/810738894 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c0092e0 con 0x7fc414068bd0 2023-10-10T19:19:55.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 -- 172.21.15.175:0/810738894 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc414068bd0 msgr2=0x7fc4140aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:55.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/810738894 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc414068bd0 0x7fc4140aff60 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fc414071640 tx=0x7fc40c009840 comp rx=0 tx=0).stop 2023-10-10T19:19:55.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 -- 172.21.15.175:0/810738894 shutdown_connections 2023-10-10T19:19:55.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/810738894 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc414068bd0 0x7fc4140aff60 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/810738894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4140b1fc0 0x7fc414155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/810738894 >> v2:172.21.15.191:3300/0 conn(0x7fc4140b2390 0x7fc414068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 -- 172.21.15.175:0/810738894 >> 172.21.15.175:0/810738894 conn(0x7fc414063230 msgr2=0x7fc414063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:55.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 -- 172.21.15.175:0/810738894 shutdown_connections 2023-10-10T19:19:55.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.107+0000 7fc41c1d4700 1 -- 172.21.15.175:0/810738894 wait complete. 2023-10-10T19:19:55.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.108+0000 7fc41c1d4700 1 Processor -- start 2023-10-10T19:19:55.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.108+0000 7fc41c1d4700 1 -- start start 2023-10-10T19:19:55.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41c1d4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc414068bd0 0x7fc414169000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:55.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41c1d4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4140b1fc0 0x7fc4140c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:55.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41c1d4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4140b2390 0x7fc4140c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:55.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41c1d4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc414169bd0 con 0x7fc4140b2390 2023-10-10T19:19:55.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41c1d4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140c45c0 con 0x7fc4140b1fc0 2023-10-10T19:19:55.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41c1d4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41414fab0 con 0x7fc414068bd0 2023-10-10T19:19:55.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4140b2390 0x7fc4140c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:55.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4140b2390 0x7fc4140c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55632/0 (socket says 172.21.15.175:55632) 2023-10-10T19:19:55.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 -- 172.21.15.175:0/1419189633 learned_addr learned my addr 172.21.15.175:0/1419189633 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:55.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc419f70700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc414068bd0 0x7fc414169000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:55.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41976f700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4140b1fc0 0x7fc4140c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:55.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 -- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc414068bd0 msgr2=0x7fc414169000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:55.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc414068bd0 0x7fc414169000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 -- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4140b1fc0 msgr2=0x7fc4140c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:55.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4140b1fc0 0x7fc4140c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 -- 172.21.15.175:0/1419189633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c009ae0 con 0x7fc4140b2390 2023-10-10T19:19:55.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc41a771700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4140b2390 0x7fc4140c5ba0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fc40c012000 tx=0x7fc40c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:55.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.109+0000 7fc409ffb700 1 -- 172.21.15.175:0/1419189633 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c0032f0 con 0x7fc4140b2390 2023-10-10T19:19:55.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.110+0000 7fc409ffb700 1 -- 172.21.15.175:0/1419189633 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40c008d20 con 0x7fc4140b2390 2023-10-10T19:19:55.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.110+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4140c51d0 con 0x7fc4140b2390 2023-10-10T19:19:55.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.110+0000 7fc409ffb700 1 -- 172.21.15.175:0/1419189633 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c01d670 con 0x7fc4140b2390 2023-10-10T19:19:55.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.110+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc414068900 con 0x7fc4140b2390 2023-10-10T19:19:55.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.111+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4140c4fa0 con 0x7fc4140b2390 2023-10-10T19:19:55.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.111+0000 7fc409ffb700 1 -- 172.21.15.175:0/1419189633 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc40c003040 con 0x7fc4140b2390 2023-10-10T19:19:55.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.111+0000 7fc409ffb700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc3f8066f30 0x7fc3f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:55.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.111+0000 7fc409ffb700 1 -- 172.21.15.175:0/1419189633 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fc40c086670 con 0x7fc4140b2390 2023-10-10T19:19:55.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.112+0000 7fc419f70700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc3f8066f30 0x7fc3f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:55.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.112+0000 7fc419f70700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc3f8066f30 0x7fc3f80693e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc410006f00 tx=0x7fc410008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:55.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.116+0000 7fc409ffb700 1 -- 172.21.15.175:0/1419189633 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4140c4fa0 con 0x7fc4140b2390 2023-10-10T19:19:55.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.337+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4140c4fa0 con 0x7fc4140b2390 2023-10-10T19:19:55.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.338+0000 7fc409ffb700 1 -- 172.21.15.175:0/1419189633 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 265 v265) v1 ==== 95+0+6198 (secure 0 0 0) 0x7fc4140c4fa0 con 0x7fc4140b2390 2023-10-10T19:19:55.339 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:55.339 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":265,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":265,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:45.827946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:55.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc3f8066f30 msgr2=0x7fc3f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:55.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc3f8066f30 0x7fc3f80693e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc410006f00 tx=0x7fc410008040 comp rx=0 tx=0).stop 2023-10-10T19:19:55.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4140b2390 msgr2=0x7fc4140c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:55.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4140b2390 0x7fc4140c5ba0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fc40c012000 tx=0x7fc40c018000 comp rx=0 tx=0).stop 2023-10-10T19:19:55.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 shutdown_connections 2023-10-10T19:19:55.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc3f8066f30 0x7fc3f80693e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4140b2390 0x7fc4140c5ba0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc414068bd0 0x7fc414169000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 --2- 172.21.15.175:0/1419189633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4140b1fc0 0x7fc4140c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:55.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 >> 172.21.15.175:0/1419189633 conn(0x7fc414063230 msgr2=0x7fc414157620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:55.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 shutdown_connections 2023-10-10T19:19:55.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:55.340+0000 7fc41c1d4700 1 -- 172.21.15.175:0/1419189633 wait complete. 2023-10-10T19:19:55.344 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 265 2023-10-10T19:19:55.352 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:57.353 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:57.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.480+0000 7fdc75294700 1 Processor -- start 2023-10-10T19:19:57.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.480+0000 7fdc75294700 1 -- start start 2023-10-10T19:19:57.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc75294700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdc7008a100 0x7fdc70159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc75294700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc700cd040 0x7fdc700684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc75294700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc70067fa0 con 0x7fdc7014fd90 2023-10-10T19:19:57.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc75294700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc700c5600 con 0x7fdc7008a100 2023-10-10T19:19:57.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc75294700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70156b70 con 0x7fdc700cd040 2023-10-10T19:19:57.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fdc7014fd90 0x7fdc700c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47858/0 (socket says 172.21.15.175:47858) 2023-10-10T19:19:57.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6ffff700 1 -- 172.21.15.175:0/2014692487 learned_addr learned my addr 172.21.15.175:0/2014692487 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:57.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6f7fe700 1 --2- 172.21.15.175:0/2014692487 >> v2:172.21.15.191:3300/0 conn(0x7fdc7008a100 0x7fdc70159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc74a93700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc700cd040 0x7fdc700684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15834337 0 0) 0x7fdc70067fa0 con 0x7fdc7014fd90 2023-10-10T19:19:57.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc58003570 con 0x7fdc7014fd90 2023-10-10T19:19:57.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1985025414 0 0) 0x7fdc58003570 con 0x7fdc7014fd90 2023-10-10T19:19:57.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc700cd040 msgr2=0x7fdc700684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:57.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6effd700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc700cd040 0x7fdc700684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 >> v2:172.21.15.191:3300/0 conn(0x7fdc7008a100 msgr2=0x7fdc70159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:57.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.481+0000 7fdc6effd700 1 --2- 172.21.15.175:0/2014692487 >> v2:172.21.15.191:3300/0 conn(0x7fdc7008a100 0x7fdc70159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70150160 con 0x7fdc7014fd90 2023-10-10T19:19:57.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fdc60001470 con 0x7fdc7014fd90 2023-10-10T19:19:57.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 >> v1:172.21.15.175:6789/0 conn(0x7fdc7014fd90 legacy=0x7fdc700c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:57.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc58003a60 0x7fdc58003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58004810 0x7fdc58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc700cd040 0x7fdc580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc700c5600 con 0x7fdc58003a60 2023-10-10T19:19:57.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70156b70 con 0x7fdc58004810 2023-10-10T19:19:57.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5800ba60 con 0x7fdc700cd040 2023-10-10T19:19:57.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6f7fe700 1 --2- 172.21.15.175:0/2014692487 >> v2:172.21.15.191:3300/0 conn(0x7fdc7008a100 0x7fdc70159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:19:57.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6ffff700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc58003a60 0x7fdc58003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc74a93700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc700cd040 0x7fdc580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6f7fe700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58004810 0x7fdc58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6ffff700 1 -- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc700cd040 msgr2=0x7fdc580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:57.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6ffff700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc700cd040 0x7fdc580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6ffff700 1 -- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58004810 msgr2=0x7fdc58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:57.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6ffff700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58004810 0x7fdc58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6ffff700 1 -- 172.21.15.175:0/2014692487 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc58003570 con 0x7fdc58003a60 2023-10-10T19:19:57.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6ffff700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc58003a60 0x7fdc58003e70 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000c3f0 tx=0x7fdc70089e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8668e4e68ae37a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:57.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6000cc80 con 0x7fdc58003a60 2023-10-10T19:19:57.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6000cde0 con 0x7fdc58003a60 2023-10-10T19:19:57.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.482+0000 7fdc6effd700 1 -- 172.21.15.175:0/2014692487 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60005b70 con 0x7fdc58003a60 2023-10-10T19:19:57.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.483+0000 7fdc75294700 1 -- 172.21.15.175:0/2014692487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc58003a60 msgr2=0x7fdc58003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:57.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.483+0000 7fdc75294700 1 --2- 172.21.15.175:0/2014692487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc58003a60 0x7fdc58003e70 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000c3f0 tx=0x7fdc70089e40 comp rx=0 tx=0).stop 2023-10-10T19:19:57.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.483+0000 7fdc74a93700 1 -- 172.21.15.175:0/2014692487 reap_dead start 2023-10-10T19:19:57.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.483+0000 7fdc75294700 1 -- 172.21.15.175:0/2014692487 shutdown_connections 2023-10-10T19:19:57.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.483+0000 7fdc75294700 1 -- 172.21.15.175:0/2014692487 >> 172.21.15.175:0/2014692487 conn(0x7fdc70063d00 msgr2=0x7fdc700cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:57.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.483+0000 7fdc75294700 1 -- 172.21.15.175:0/2014692487 shutdown_connections 2023-10-10T19:19:57.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.483+0000 7fdc75294700 1 -- 172.21.15.175:0/2014692487 wait complete. 2023-10-10T19:19:57.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.484+0000 7fdc75294700 1 Processor -- start 2023-10-10T19:19:57.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.484+0000 7fdc75294700 1 -- start start 2023-10-10T19:19:57.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc75294700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58003a60 0x7fdc70151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc75294700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc700ca420 0x7fdc700cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc75294700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc701666b0 0x7fdc700c3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc75294700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70150500 con 0x7fdc700ca420 2023-10-10T19:19:57.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc75294700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc700cb480 con 0x7fdc58003a60 2023-10-10T19:19:57.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc75294700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70166220 con 0x7fdc701666b0 2023-10-10T19:19:57.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc700ca420 0x7fdc700cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc700ca420 0x7fdc700cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55648/0 (socket says 172.21.15.175:55648) 2023-10-10T19:19:57.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 -- 172.21.15.175:0/445591209 learned_addr learned my addr 172.21.15.175:0/445591209 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:57.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc74a93700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc701666b0 0x7fdc700c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6ffff700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58003a60 0x7fdc70151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 -- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc701666b0 msgr2=0x7fdc700c3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:57.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc701666b0 0x7fdc700c3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 -- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58003a60 msgr2=0x7fdc70151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:57.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58003a60 0x7fdc70151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 -- 172.21.15.175:0/445591209 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70165ef0 con 0x7fdc700ca420 2023-10-10T19:19:57.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.485+0000 7fdc6f7fe700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc700ca420 0x7fdc700cbf80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fdc64011d20 tx=0x7fdc640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:57.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.486+0000 7fdc57fff700 1 -- 172.21.15.175:0/445591209 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6400fa10 con 0x7fdc700ca420 2023-10-10T19:19:57.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.486+0000 7fdc57fff700 1 -- 172.21.15.175:0/445591209 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6400fb70 con 0x7fdc700ca420 2023-10-10T19:19:57.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.486+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc70150710 con 0x7fdc700ca420 2023-10-10T19:19:57.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.486+0000 7fdc57fff700 1 -- 172.21.15.175:0/445591209 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64014440 con 0x7fdc700ca420 2023-10-10T19:19:57.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.486+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc70150cd0 con 0x7fdc700ca420 2023-10-10T19:19:57.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.487+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc70067570 con 0x7fdc700ca420 2023-10-10T19:19:57.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.487+0000 7fdc57fff700 1 -- 172.21.15.175:0/445591209 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdc64014a80 con 0x7fdc700ca420 2023-10-10T19:19:57.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.487+0000 7fdc57fff700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc5c066f60 0x7fdc5c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:57.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.487+0000 7fdc57fff700 1 -- 172.21.15.175:0/445591209 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fdc64018030 con 0x7fdc700ca420 2023-10-10T19:19:57.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.487+0000 7fdc6ffff700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc5c066f60 0x7fdc5c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:57.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.488+0000 7fdc6ffff700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc5c066f60 0x7fdc5c069410 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000c310 tx=0x7fdc60015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:57.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.492+0000 7fdc57fff700 1 -- 172.21.15.175:0/445591209 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc64052460 con 0x7fdc700ca420 2023-10-10T19:19:57.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.716+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc70165d50 con 0x7fdc700ca420 2023-10-10T19:19:57.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.717+0000 7fdc57fff700 1 -- 172.21.15.175:0/445591209 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 265 v265) v1 ==== 95+0+6198 (secure 0 0 0) 0x7fdc70165d50 con 0x7fdc700ca420 2023-10-10T19:19:57.718 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:19:57.718 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":265,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":265,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:45.827946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618,"mds_4":6373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6373":{"gid":6373,"name":"f","rank":4,"incarnation":195,"state":"up:stopping","state_seq":19,"addr":"172.21.15.191:6839/4045179206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":4045179206},{"type":"v1","addr":"172.21.15.191:6839","nonce":4045179206}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:19:57.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc5c066f60 msgr2=0x7fdc5c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:57.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc5c066f60 0x7fdc5c069410 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000c310 tx=0x7fdc60015000 comp rx=0 tx=0).stop 2023-10-10T19:19:57.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc700ca420 msgr2=0x7fdc700cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:57.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc700ca420 0x7fdc700cbf80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fdc64011d20 tx=0x7fdc640091a0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 shutdown_connections 2023-10-10T19:19:57.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fdc5c066f60 0x7fdc5c069410 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc701666b0 0x7fdc700c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc58003a60 0x7fdc70151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 --2- 172.21.15.175:0/445591209 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc700ca420 0x7fdc700cbf80 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:57.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 >> 172.21.15.175:0/445591209 conn(0x7fdc70063d00 msgr2=0x7fdc700c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:57.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 shutdown_connections 2023-10-10T19:19:57.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:57.719+0000 7fdc75294700 1 -- 172.21.15.175:0/445591209 wait complete. 2023-10-10T19:19:57.723 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 265 2023-10-10T19:19:57.731 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:19:57.906 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-10T19:19:59.733 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:19:59.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.857+0000 7f900c6a6700 1 Processor -- start 2023-10-10T19:19:59.858 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.857+0000 7f900c6a6700 1 -- start start 2023-10-10T19:19:59.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f900c6a6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040b1fc0 0x7f9004155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:59.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f900c6a6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90040aff60 0x7f90040b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:59.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f900c6a6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90041574b0 con 0x7f90040b2390 2023-10-10T19:19:59.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f900c6a6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9004150770 con 0x7f90040aff60 2023-10-10T19:19:59.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f900c6a6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90040afad0 con 0x7f90040b1fc0 2023-10-10T19:19:59.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f900ac43700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90040aff60 0x7f90040b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:59.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f9009c41700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f90040b2390 0x7f9004068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47872/0 (socket says 172.21.15.175:47872) 2023-10-10T19:19:59.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f900ac43700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90040aff60 0x7f90040b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55660/0 (socket says 172.21.15.175:55660) 2023-10-10T19:19:59.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.858+0000 7f9009c41700 1 -- 172.21.15.175:0/3123360361 learned_addr learned my addr 172.21.15.175:0/3123360361 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:19:59.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f900a442700 1 --2- 172.21.15.175:0/3123360361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040b1fc0 0x7f9004155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:59.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f900ac43700 1 -- 172.21.15.175:0/3123360361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040b1fc0 msgr2=0x7f9004155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:19:59.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f900ac43700 1 --2- 172.21.15.175:0/3123360361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040b1fc0 0x7f9004155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:59.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f900ac43700 1 -- 172.21.15.175:0/3123360361 >> v1:172.21.15.175:6789/0 conn(0x7f90040b2390 legacy=0x7f9004068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:59.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f9009440700 1 -- 172.21.15.175:0/3123360361 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110570771 0 0) 0x7f90041574b0 con 0x7f90040b2390 2023-10-10T19:19:59.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f900ac43700 1 -- 172.21.15.175:0/3123360361 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040af4c0 con 0x7f90040aff60 2023-10-10T19:19:59.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f900ac43700 1 --2- 172.21.15.175:0/3123360361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040aff60 0x7f90040b03e0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009e90 tx=0x7f8ffc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d097f7340f70777c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:59.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f9009440700 1 -- 172.21.15.175:0/3123360361 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00e070 con 0x7f90040aff60 2023-10-10T19:19:59.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f9009440700 1 -- 172.21.15.175:0/3123360361 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc00bc80 con 0x7f90040aff60 2023-10-10T19:19:59.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.859+0000 7f9009440700 1 -- 172.21.15.175:0/3123360361 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc012430 con 0x7f90040aff60 2023-10-10T19:19:59.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.860+0000 7f900c6a6700 1 -- 172.21.15.175:0/3123360361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040aff60 msgr2=0x7f90040b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:59.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.860+0000 7f900c6a6700 1 --2- 172.21.15.175:0/3123360361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040aff60 0x7f90040b03e0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009e90 tx=0x7f8ffc009780 comp rx=0 tx=0).stop 2023-10-10T19:19:59.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.860+0000 7f900c6a6700 1 -- 172.21.15.175:0/3123360361 shutdown_connections 2023-10-10T19:19:59.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.860+0000 7f900c6a6700 1 --2- 172.21.15.175:0/3123360361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040aff60 0x7f90040b03e0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:59.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.860+0000 7f900c6a6700 1 --2- 172.21.15.175:0/3123360361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040b1fc0 0x7f9004155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:59.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.860+0000 7f900c6a6700 1 -- 172.21.15.175:0/3123360361 >> 172.21.15.175:0/3123360361 conn(0x7f9004063230 msgr2=0x7f9004063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:19:59.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.860+0000 7f900c6a6700 1 -- 172.21.15.175:0/3123360361 shutdown_connections 2023-10-10T19:19:59.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.860+0000 7f900c6a6700 1 -- 172.21.15.175:0/3123360361 wait complete. 2023-10-10T19:19:59.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.861+0000 7f900c6a6700 1 Processor -- start 2023-10-10T19:19:59.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.861+0000 7f900c6a6700 1 -- start start 2023-10-10T19:19:59.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.861+0000 7f900c6a6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040aff60 0x7f90041664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:59.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.861+0000 7f900c6a6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040b1fc0 0x7f90040ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:59.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.861+0000 7f900c6a6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90040b2390 0x7f90040c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:59.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.861+0000 7f900c6a6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040c40f0 con 0x7f90040b1fc0 2023-10-10T19:19:59.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.861+0000 7f900c6a6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040ccf20 con 0x7f90040b2390 2023-10-10T19:19:59.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.861+0000 7f900c6a6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90040c4ec0 con 0x7f90040aff60 2023-10-10T19:19:59.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040b1fc0 0x7f90040ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:59.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040b1fc0 0x7f90040ca960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55674/0 (socket says 172.21.15.175:55674) 2023-10-10T19:19:59.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 -- 172.21.15.175:0/1225183361 learned_addr learned my addr 172.21.15.175:0/1225183361 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:19:59.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f900a442700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040aff60 0x7f90041664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:59.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f900ac43700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90040b2390 0x7f90040c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:59.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 -- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040aff60 msgr2=0x7f90041664e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:59.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040aff60 0x7f90041664e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:59.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 -- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90040b2390 msgr2=0x7f90040c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:19:59.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90040b2390 0x7f90040c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:19:59.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 -- 172.21.15.175:0/1225183361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc009a20 con 0x7f90040b1fc0 2023-10-10T19:19:59.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f9009c41700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040b1fc0 0x7f90040ca960 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400c5b0 tx=0x7f8ff4001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:59.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f8ffa7fc700 1 -- 172.21.15.175:0/1225183361 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4005b30 con 0x7f90040b1fc0 2023-10-10T19:19:59.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f8ffa7fc700 1 -- 172.21.15.175:0/1225183361 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff4005c90 con 0x7f90040b1fc0 2023-10-10T19:19:59.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90040c3870 con 0x7f90040b1fc0 2023-10-10T19:19:59.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f8ffa7fc700 1 -- 172.21.15.175:0/1225183361 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4007320 con 0x7f90040b1fc0 2023-10-10T19:19:59.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.862+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90040c3e10 con 0x7f90040b1fc0 2023-10-10T19:19:59.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.863+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004067570 con 0x7f90040b1fc0 2023-10-10T19:19:59.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.864+0000 7f8ffa7fc700 1 -- 172.21.15.175:0/1225183361 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8ff401b3b0 con 0x7f90040b1fc0 2023-10-10T19:19:59.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.864+0000 7f8ffa7fc700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f30 0x7f8fe80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:19:59.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.864+0000 7f8ffa7fc700 1 -- 172.21.15.175:0/1225183361 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f8ff4084030 con 0x7f90040b1fc0 2023-10-10T19:19:59.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.864+0000 7f900a442700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f30 0x7f8fe80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:19:59.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.865+0000 7f900a442700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f30 0x7f8fe80693e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f9000009d40 tx=0x7f9000008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:19:59.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:19:59.868+0000 7f8ffa7fc700 1 -- 172.21.15.175:0/1225183361 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ff4050c70 con 0x7f90040b1fc0 2023-10-10T19:20:00.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.092+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9004068da0 con 0x7f90040b1fc0 2023-10-10T19:20:00.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.092+0000 7f8ffa7fc700 1 -- 172.21.15.175:0/1225183361 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f8ff4054290 con 0x7f90040b1fc0 2023-10-10T19:20:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:00.093 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7204,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2765861262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2765861262},{"type":"v1","addr":"172.21.15.191:6839","nonce":2765861262}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":267}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:00.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f30 msgr2=0x7f8fe80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f30 0x7f8fe80693e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f9000009d40 tx=0x7f9000008040 comp rx=0 tx=0).stop 2023-10-10T19:20:00.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040b1fc0 msgr2=0x7f90040ca960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040b1fc0 0x7f90040ca960 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400c5b0 tx=0x7f8ff4001070 comp rx=0 tx=0).stop 2023-10-10T19:20:00.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 shutdown_connections 2023-10-10T19:20:00.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fe8066f30 0x7f8fe80693e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90040b2390 0x7f90040c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90040aff60 0x7f90041664e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 --2- 172.21.15.175:0/1225183361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90040b1fc0 0x7f90040ca960 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 >> 172.21.15.175:0/1225183361 conn(0x7f9004063230 msgr2=0x7f9004153dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:00.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 shutdown_connections 2023-10-10T19:20:00.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.094+0000 7f900c6a6700 1 -- 172.21.15.175:0/1225183361 wait complete. 2023-10-10T19:20:00.098 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 267 2023-10-10T19:20:00.106 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 4 alive and active, now stable! 2023-10-10T19:20:00.106 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=4) 2023-10-10T19:20:00.107 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-10T19:20:00.107 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:20:00.117 INFO:tasks.ceph.mds.f:Stopped 2023-10-10T19:20:00.118 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:00.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.233+0000 7f19b5052700 1 Processor -- start 2023-10-10T19:20:00.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.233+0000 7f19b5052700 1 -- start start 2023-10-10T19:20:00.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.233+0000 7f19b5052700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b1fc0 0x7f19b0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:00.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.233+0000 7f19b5052700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b01574b0 con 0x7f19b00aff60 2023-10-10T19:20:00.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.233+0000 7f19b5052700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b00afad0 con 0x7f19b00b2390 2023-10-10T19:20:00.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.233+0000 7f19b5052700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b0150770 con 0x7f19b00b1fc0 2023-10-10T19:20:00.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19af59e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f19b00aff60 0x7f19b0159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47874/0 (socket says 172.21.15.175:47874) 2023-10-10T19:20:00.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19af59e700 1 -- 172.21.15.175:0/1002639616 learned_addr learned my addr 172.21.15.175:0/1002639616 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:20:00.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19aed9d700 1 --2- 172.21.15.175:0/1002639616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b1fc0 0x7f19b0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:00.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19add9b700 1 -- 172.21.15.175:0/1002639616 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437773637 0 0) 0x7f19b01574b0 con 0x7f19b00aff60 2023-10-10T19:20:00.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19add9b700 1 -- 172.21.15.175:0/1002639616 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199c003570 con 0x7f19b00aff60 2023-10-10T19:20:00.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19add9b700 1 -- 172.21.15.175:0/1002639616 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842970174 0 0) 0x7f19b00afad0 con 0x7f19b00b2390 2023-10-10T19:20:00.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19add9b700 1 -- 172.21.15.175:0/1002639616 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b01574b0 con 0x7f19b00b2390 2023-10-10T19:20:00.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19aed9d700 1 -- 172.21.15.175:0/1002639616 >> v1:172.21.15.191:6789/0 conn(0x7f19b00b2390 legacy=0x7f19b0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19aed9d700 1 -- 172.21.15.175:0/1002639616 >> v1:172.21.15.175:6789/0 conn(0x7f19b00aff60 legacy=0x7f19b0159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19add9b700 1 -- 172.21.15.175:0/1002639616 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520112212 0 0) 0x7f199c003570 con 0x7f19b00aff60 2023-10-10T19:20:00.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19aed9d700 1 -- 172.21.15.175:0/1002639616 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b00af4c0 con 0x7f19b00b1fc0 2023-10-10T19:20:00.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19aed9d700 1 --2- 172.21.15.175:0/1002639616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b1fc0 0x7f19b0155fb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009ec0 tx=0x7f19a8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f750c691d335ad server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:00.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19add9b700 1 -- 172.21.15.175:0/1002639616 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a800e070 con 0x7f19b00b1fc0 2023-10-10T19:20:00.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19add9b700 1 -- 172.21.15.175:0/1002639616 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a800bbd0 con 0x7f19b00b1fc0 2023-10-10T19:20:00.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.234+0000 7f19add9b700 1 -- 172.21.15.175:0/1002639616 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a8012430 con 0x7f19b00b1fc0 2023-10-10T19:20:00.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.235+0000 7f19b5052700 1 -- 172.21.15.175:0/1002639616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b1fc0 msgr2=0x7f19b0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.235+0000 7f19b5052700 1 --2- 172.21.15.175:0/1002639616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b1fc0 0x7f19b0155fb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009ec0 tx=0x7f19a8009780 comp rx=0 tx=0).stop 2023-10-10T19:20:00.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.235+0000 7f19b5052700 1 -- 172.21.15.175:0/1002639616 shutdown_connections 2023-10-10T19:20:00.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.235+0000 7f19b5052700 1 --2- 172.21.15.175:0/1002639616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b1fc0 0x7f19b0155fb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.235+0000 7f19b5052700 1 -- 172.21.15.175:0/1002639616 >> 172.21.15.175:0/1002639616 conn(0x7f19b0063230 msgr2=0x7f19b0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:00.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.235+0000 7f19b5052700 1 -- 172.21.15.175:0/1002639616 shutdown_connections 2023-10-10T19:20:00.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.235+0000 7f19b5052700 1 -- 172.21.15.175:0/1002639616 wait complete. 2023-10-10T19:20:00.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.236+0000 7f19b5052700 1 Processor -- start 2023-10-10T19:20:00.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.236+0000 7f19b5052700 1 -- start start 2023-10-10T19:20:00.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19b5052700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19b00aff60 0x7f19b0156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:00.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19b5052700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19b00b1fc0 0x7f19b00c2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:00.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19b5052700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b2390 0x7f19b00c25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:00.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19b5052700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00c0950 con 0x7f19b00b1fc0 2023-10-10T19:20:00.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19b5052700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00c0670 con 0x7f19b00aff60 2023-10-10T19:20:00.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19b5052700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b0155070 con 0x7f19b00b2390 2023-10-10T19:20:00.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19b00b1fc0 0x7f19b00c2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:00.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19b00b1fc0 0x7f19b00c2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55684/0 (socket says 172.21.15.175:55684) 2023-10-10T19:20:00.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 -- 172.21.15.175:0/18566739 learned_addr learned my addr 172.21.15.175:0/18566739 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:00.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19aed9d700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19b00aff60 0x7f19b0156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:00.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19af59e700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b2390 0x7f19b00c25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:00.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 -- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b2390 msgr2=0x7f19b00c25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b2390 0x7f19b00c25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 -- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19b00aff60 msgr2=0x7f19b0156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19b00aff60 0x7f19b0156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 -- 172.21.15.175:0/18566739 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a8009a20 con 0x7f19b00b1fc0 2023-10-10T19:20:00.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19ae59c700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19b00b1fc0 0x7f19b00c2060 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f199800c580 tx=0x7f1998007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:00.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19a6ffd700 1 -- 172.21.15.175:0/18566739 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998010510 con 0x7f19b00b1fc0 2023-10-10T19:20:00.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19a6ffd700 1 -- 172.21.15.175:0/18566739 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1998010670 con 0x7f19b00b1fc0 2023-10-10T19:20:00.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.237+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b0155350 con 0x7f19b00b1fc0 2023-10-10T19:20:00.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.238+0000 7f19a6ffd700 1 -- 172.21.15.175:0/18566739 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998010a30 con 0x7f19b00b1fc0 2023-10-10T19:20:00.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.238+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b0155910 con 0x7f19b00b1fc0 2023-10-10T19:20:00.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.238+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b0063930 con 0x7f19b00b1fc0 2023-10-10T19:20:00.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.239+0000 7f19a6ffd700 1 -- 172.21.15.175:0/18566739 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f199801b400 con 0x7f19b00b1fc0 2023-10-10T19:20:00.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.239+0000 7f19a6ffd700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1990066f30 0x7f19900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:00.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.239+0000 7f19a6ffd700 1 -- 172.21.15.175:0/18566739 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f19980848d0 con 0x7f19b00b1fc0 2023-10-10T19:20:00.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.239+0000 7f19aed9d700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1990066f30 0x7f19900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:00.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.240+0000 7f19aed9d700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1990066f30 0x7f19900693e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009de0 tx=0x7f19a8019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:00.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.243+0000 7f19a6ffd700 1 -- 172.21.15.175:0/18566739 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1998051510 con 0x7f19b00b1fc0 2023-10-10T19:20:00.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.468+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19b0067570 con 0x7f19b00b1fc0 2023-10-10T19:20:00.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.469+0000 7f19a6ffd700 1 -- 172.21.15.175:0/18566739 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f1998054b30 con 0x7f19b00b1fc0 2023-10-10T19:20:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:00.470 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7204,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2765861262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2765861262},{"type":"v1","addr":"172.21.15.191:6839","nonce":2765861262}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":267}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:00.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1990066f30 msgr2=0x7f19900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1990066f30 0x7f19900693e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009de0 tx=0x7f19a8019000 comp rx=0 tx=0).stop 2023-10-10T19:20:00.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19b00b1fc0 msgr2=0x7f19b00c2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:00.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19b00b1fc0 0x7f19b00c2060 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f199800c580 tx=0x7f1998007e10 comp rx=0 tx=0).stop 2023-10-10T19:20:00.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 shutdown_connections 2023-10-10T19:20:00.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1990066f30 0x7f19900693e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19b00b2390 0x7f19b00c25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19b00aff60 0x7f19b0156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 --2- 172.21.15.175:0/18566739 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19b00b1fc0 0x7f19b00c2060 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:00.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 >> 172.21.15.175:0/18566739 conn(0x7f19b0063230 msgr2=0x7f19b0157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:00.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 shutdown_connections 2023-10-10T19:20:00.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:00.471+0000 7f19b5052700 1 -- 172.21.15.175:0/18566739 wait complete. 2023-10-10T19:20:00.474 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 267 2023-10-10T19:20:00.483 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:20:02.483 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:02.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.609+0000 7fb3a4be8700 1 Processor -- start 2023-10-10T19:20:02.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.609+0000 7fb3a4be8700 1 -- start start 2023-10-10T19:20:02.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb3a4be8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 0x7fb3a00c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:02.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb3a4be8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb3a00c41e0 0x7fb3a014cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:02.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb3a4be8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3a006b640 0x7fb3a014d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:02.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb3a4be8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a014d820 con 0x7fb3a00c41e0 2023-10-10T19:20:02.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb3a4be8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a01533e0 con 0x7fb3a006b640 2023-10-10T19:20:02.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb3a4be8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a006e130 con 0x7fb3a00c3330 2023-10-10T19:20:02.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb3a00c41e0 0x7fb3a014cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:02.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb3a00c41e0 0x7fb3a014cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55688/0 (socket says 172.21.15.175:55688) 2023-10-10T19:20:02.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 -- 172.21.15.175:0/979111035 learned_addr learned my addr 172.21.15.175:0/979111035 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:02.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39ed9d700 1 --2- 172.21.15.175:0/979111035 >> v2:172.21.15.191:3300/0 conn(0x7fb3a006b640 0x7fb3a014d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:02.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39e59c700 1 --2- 172.21.15.175:0/979111035 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 0x7fb3a00c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:02.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 -- 172.21.15.175:0/979111035 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 msgr2=0x7fb3a00c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:02.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 --2- 172.21.15.175:0/979111035 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 0x7fb3a00c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 -- 172.21.15.175:0/979111035 >> v2:172.21.15.191:3300/0 conn(0x7fb3a006b640 msgr2=0x7fb3a014d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:02.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 --2- 172.21.15.175:0/979111035 >> v2:172.21.15.191:3300/0 conn(0x7fb3a006b640 0x7fb3a014d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 -- 172.21.15.175:0/979111035 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a006a970 con 0x7fb3a00c41e0 2023-10-10T19:20:02.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39ed9d700 1 --2- 172.21.15.175:0/979111035 >> v2:172.21.15.191:3300/0 conn(0x7fb3a006b640 0x7fb3a014d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:20:02.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.610+0000 7fb39dd9b700 1 --2- 172.21.15.175:0/979111035 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a014cda0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00c6230 tx=0x7fb39400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a9a0a98e41e7b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:02.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb39d59a700 1 -- 172.21.15.175:0/979111035 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39400dc50 con 0x7fb3a00c41e0 2023-10-10T19:20:02.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb39d59a700 1 -- 172.21.15.175:0/979111035 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39400ddb0 con 0x7fb3a00c41e0 2023-10-10T19:20:02.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb39d59a700 1 -- 172.21.15.175:0/979111035 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3940196d0 con 0x7fb3a00c41e0 2023-10-10T19:20:02.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 -- 172.21.15.175:0/979111035 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 msgr2=0x7fb3a014cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:02.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/979111035 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a014cda0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00c6230 tx=0x7fb39400b4f0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 -- 172.21.15.175:0/979111035 shutdown_connections 2023-10-10T19:20:02.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/979111035 >> v2:172.21.15.191:3300/0 conn(0x7fb3a006b640 0x7fb3a014d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/979111035 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 0x7fb3a00c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/979111035 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a014cda0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 -- 172.21.15.175:0/979111035 >> 172.21.15.175:0/979111035 conn(0x7fb3a0068280 msgr2=0x7fb3a006af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:02.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 -- 172.21.15.175:0/979111035 shutdown_connections 2023-10-10T19:20:02.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.611+0000 7fb3a4be8700 1 -- 172.21.15.175:0/979111035 wait complete. 2023-10-10T19:20:02.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb3a4be8700 1 Processor -- start 2023-10-10T19:20:02.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb3a4be8700 1 -- start start 2023-10-10T19:20:02.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb3a4be8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a006b640 0x7fb3a01659b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:02.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb3a4be8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 0x7fb3a0148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:02.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb3a4be8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a0144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:02.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb3a4be8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a0149bb0 con 0x7fb3a00c41e0 2023-10-10T19:20:02.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb3a4be8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a0144090 con 0x7fb3a006b640 2023-10-10T19:20:02.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb3a4be8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a0154a60 con 0x7fb3a00c3330 2023-10-10T19:20:02.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.613+0000 7fb39ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a0144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:02.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a0144520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55702/0 (socket says 172.21.15.175:55702) 2023-10-10T19:20:02.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39ed9d700 1 -- 172.21.15.175:0/3518475909 learned_addr learned my addr 172.21.15.175:0/3518475909 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:02.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39e59c700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a006b640 0x7fb3a01659b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:02.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39dd9b700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 0x7fb3a0148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:02.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39ed9d700 1 -- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 msgr2=0x7fb3a0148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:02.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39ed9d700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 0x7fb3a0148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39ed9d700 1 -- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a006b640 msgr2=0x7fb3a01659b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:02.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39ed9d700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a006b640 0x7fb3a01659b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39ed9d700 1 -- 172.21.15.175:0/3518475909 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39400b790 con 0x7fb3a00c41e0 2023-10-10T19:20:02.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb39ed9d700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a0144520 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb39800d150 tx=0x7fb398009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:02.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb38e7fc700 1 -- 172.21.15.175:0/3518475909 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39800dc30 con 0x7fb3a00c41e0 2023-10-10T19:20:02.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb38e7fc700 1 -- 172.21.15.175:0/3518475909 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39800f040 con 0x7fb3a00c41e0 2023-10-10T19:20:02.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb38e7fc700 1 -- 172.21.15.175:0/3518475909 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398015400 con 0x7fb3a00c41e0 2023-10-10T19:20:02.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a0149050 con 0x7fb3a00c41e0 2023-10-10T19:20:02.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.614+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a01495f0 con 0x7fb3a00c41e0 2023-10-10T19:20:02.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.615+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a0144200 con 0x7fb3a00c41e0 2023-10-10T19:20:02.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.616+0000 7fb38e7fc700 1 -- 172.21.15.175:0/3518475909 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb3980157b0 con 0x7fb3a00c41e0 2023-10-10T19:20:02.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.616+0000 7fb38e7fc700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb380066f30 0x7fb3800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:02.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.616+0000 7fb38e7fc700 1 -- 172.21.15.175:0/3518475909 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fb398085350 con 0x7fb3a00c41e0 2023-10-10T19:20:02.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.616+0000 7fb39e59c700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb380066f30 0x7fb3800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:02.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.617+0000 7fb39e59c700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb380066f30 0x7fb3800693e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb390006aa0 tx=0x7fb390008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:02.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.620+0000 7fb38e7fc700 1 -- 172.21.15.175:0/3518475909 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb398051f90 con 0x7fb3a00c41e0 2023-10-10T19:20:02.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.847+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3a0067570 con 0x7fb3a00c41e0 2023-10-10T19:20:02.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.847+0000 7fb38e7fc700 1 -- 172.21.15.175:0/3518475909 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6181 (secure 0 0 0) 0x7fb3980555b0 con 0x7fb3a00c41e0 2023-10-10T19:20:02.849 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:02.849 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7204,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2765861262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2765861262},{"type":"v1","addr":"172.21.15.191:6839","nonce":2765861262}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":267}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:02.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.849+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb380066f30 msgr2=0x7fb3800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:02.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb380066f30 0x7fb3800693e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb390006aa0 tx=0x7fb390008040 comp rx=0 tx=0).stop 2023-10-10T19:20:02.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 msgr2=0x7fb3a0144520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:02.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a0144520 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb39800d150 tx=0x7fb398009a50 comp rx=0 tx=0).stop 2023-10-10T19:20:02.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 shutdown_connections 2023-10-10T19:20:02.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb380066f30 0x7fb3800693e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3a00c41e0 0x7fb3a0144520 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a006b640 0x7fb3a01659b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 --2- 172.21.15.175:0/3518475909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3a00c3330 0x7fb3a0148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:02.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 >> 172.21.15.175:0/3518475909 conn(0x7fb3a0068280 msgr2=0x7fb3a006a200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:02.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 shutdown_connections 2023-10-10T19:20:02.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:02.850+0000 7fb3a4be8700 1 -- 172.21.15.175:0/3518475909 wait complete. 2023-10-10T19:20:02.854 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 267 2023-10-10T19:20:02.863 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:20:04.864 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:04.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.985+0000 7f396279d700 1 Processor -- start 2023-10-10T19:20:04.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.985+0000 7f396279d700 1 -- start start 2023-10-10T19:20:04.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f396279d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f395c0c1fb0 0x7f395c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:04.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f396279d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c154e20 con 0x7f395c14ddb0 2023-10-10T19:20:04.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f396279d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c154210 con 0x7f395c0c1be0 2023-10-10T19:20:04.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f396279d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395c06bca0 con 0x7f395c0c1fb0 2023-10-10T19:20:04.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f3960d3a700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f395c14ddb0 0x7f395c14e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52862/0 (socket says 172.21.15.175:52862) 2023-10-10T19:20:04.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f3960d3a700 1 -- 172.21.15.175:0/816664658 learned_addr learned my addr 172.21.15.175:0/816664658 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:20:04.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478037873 0 0) 0x7f395c154e20 con 0x7f395c14ddb0 2023-10-10T19:20:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395b7fe700 1 --2- 172.21.15.175:0/816664658 >> v2:172.21.15.191:3300/0 conn(0x7f395c0c1fb0 0x7f395c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3944003570 con 0x7f395c14ddb0 2023-10-10T19:20:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2500250145 0 0) 0x7f3944003570 con 0x7f395c14ddb0 2023-10-10T19:20:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 >> v2:172.21.15.191:3300/0 conn(0x7f395c0c1fb0 msgr2=0x7f395c06b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 --2- 172.21.15.175:0/816664658 >> v2:172.21.15.191:3300/0 conn(0x7f395c0c1fb0 0x7f395c06b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:04.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 >> v1:172.21.15.191:6789/0 conn(0x7f395c0c1be0 legacy=0x7f395c06a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:04.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c1544f0 con 0x7f395c14ddb0 2023-10-10T19:20:04.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f39540017f0 con 0x7f395c14ddb0 2023-10-10T19:20:04.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 >> v1:172.21.15.175:6789/0 conn(0x7f395c14ddb0 legacy=0x7f395c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:04.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944003a60 0x7f3944003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:04.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3944004810 0x7f3944004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:04.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39440051c0 0x7f394400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:04.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c06bca0 con 0x7f3944004810 2023-10-10T19:20:04.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394400c0d0 con 0x7f3944003a60 2023-10-10T19:20:04.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394400c290 con 0x7f39440051c0 2023-10-10T19:20:04.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.986+0000 7f395b7fe700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3944004810 0x7f3944004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:04.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395bfff700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944003a60 0x7f3944003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:04.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f3960d3a700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39440051c0 0x7f394400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:05.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395b7fe700 1 -- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39440051c0 msgr2=0x7f394400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:05.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395b7fe700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39440051c0 0x7f394400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395b7fe700 1 -- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944003a60 msgr2=0x7f3944003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:05.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395b7fe700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944003a60 0x7f3944003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395b7fe700 1 -- 172.21.15.175:0/816664658 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3944003570 con 0x7f3944004810 2023-10-10T19:20:05.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f3960d3a700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39440051c0 0x7f394400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:20:05.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f3960d3a700 1 -- 172.21.15.175:0/816664658 reap_dead start 2023-10-10T19:20:05.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395b7fe700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3944004810 0x7f3944004c80 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f3950009c00 tx=0x7f39500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd025e06d8cf9217 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:05.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395000e710 con 0x7f3944004810 2023-10-10T19:20:05.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395000e870 con 0x7f3944004810 2023-10-10T19:20:05.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.987+0000 7f395affd700 1 -- 172.21.15.175:0/816664658 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395000ec30 con 0x7f3944004810 2023-10-10T19:20:05.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.988+0000 7f396279d700 1 -- 172.21.15.175:0/816664658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3944004810 msgr2=0x7f3944004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:05.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.988+0000 7f396279d700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3944004810 0x7f3944004c80 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f3950009c00 tx=0x7f39500091a0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.988+0000 7f396279d700 1 -- 172.21.15.175:0/816664658 shutdown_connections 2023-10-10T19:20:05.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.988+0000 7f396279d700 1 --2- 172.21.15.175:0/816664658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3944004810 0x7f3944004c80 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.988+0000 7f396279d700 1 -- 172.21.15.175:0/816664658 >> 172.21.15.175:0/816664658 conn(0x7f395c068280 msgr2=0x7f395c06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:05.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.988+0000 7f396279d700 1 -- 172.21.15.175:0/816664658 shutdown_connections 2023-10-10T19:20:05.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.988+0000 7f396279d700 1 -- 172.21.15.175:0/816664658 wait complete. 2023-10-10T19:20:05.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.989+0000 7f396279d700 1 Processor -- start 2023-10-10T19:20:05.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.989+0000 7f396279d700 1 -- start start 2023-10-10T19:20:05.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f396279d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944004810 0x7f395c0c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:05.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f396279d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f395c0c42d0 0x7f395c165200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:05.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f396279d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f395c0c4740 0x7f395c165740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:05.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f396279d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c0c55f0 con 0x7f395c0c4740 2023-10-10T19:20:05.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f396279d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c148520 con 0x7f3944004810 2023-10-10T19:20:05.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f396279d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395c148aa0 con 0x7f395c0c42d0 2023-10-10T19:20:05.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f395c0c4740 0x7f395c165740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:05.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f395c0c4740 0x7f395c165740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44176/0 (socket says 172.21.15.175:44176) 2023-10-10T19:20:05.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 -- 172.21.15.175:0/500232094 learned_addr learned my addr 172.21.15.175:0/500232094 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:05.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f395bfff700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944004810 0x7f395c0c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:05.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f395b7fe700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f395c0c42d0 0x7f395c165200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:05.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 -- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f395c0c42d0 msgr2=0x7f395c165200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:05.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f395c0c42d0 0x7f395c165200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 -- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944004810 msgr2=0x7f395c0c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:05.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944004810 0x7f395c0c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 -- 172.21.15.175:0/500232094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c147980 con 0x7f395c0c4740 2023-10-10T19:20:05.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f395bfff700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944004810 0x7f395c0c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:20:05.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f395b7fe700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f395c0c42d0 0x7f395c165200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:20:05.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3960d3a700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f395c0c4740 0x7f395c165740 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3954000f80 tx=0x7f39540017f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:05.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3943fff700 1 -- 172.21.15.175:0/500232094 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400ecc0 con 0x7f395c0c4740 2023-10-10T19:20:05.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3943fff700 1 -- 172.21.15.175:0/500232094 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395400ee20 con 0x7f395c0c4740 2023-10-10T19:20:05.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c147c60 con 0x7f395c0c4740 2023-10-10T19:20:05.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f3943fff700 1 -- 172.21.15.175:0/500232094 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954013670 con 0x7f395c0c4740 2023-10-10T19:20:05.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.990+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c147670 con 0x7f395c0c4740 2023-10-10T19:20:05.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.991+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c148c50 con 0x7f395c0c4740 2023-10-10T19:20:05.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.992+0000 7f3943fff700 1 -- 172.21.15.175:0/500232094 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3954013a20 con 0x7f395c0c4740 2023-10-10T19:20:05.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.992+0000 7f3943fff700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3948066f30 0x7f39480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:05.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.992+0000 7f3943fff700 1 -- 172.21.15.175:0/500232094 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f3954085240 con 0x7f395c0c4740 2023-10-10T19:20:05.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.992+0000 7f395bfff700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3948066f30 0x7f39480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:05.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.993+0000 7f395bfff700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3948066f30 0x7f39480693e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f394c007c10 tx=0x7f394c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:05.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:04.996+0000 7f3943fff700 1 -- 172.21.15.175:0/500232094 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f395c148c50 con 0x7f395c0c4740 2023-10-10T19:20:05.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.221+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f395c148c10 con 0x7f395c0c4740 2023-10-10T19:20:05.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.222+0000 7f3943fff700 1 -- 172.21.15.175:0/500232094 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f3954051e80 con 0x7f395c0c4740 2023-10-10T19:20:05.224 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:05.224 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7204,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2765861262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2765861262},{"type":"v1","addr":"172.21.15.191:6839","nonce":2765861262}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":267}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:05.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.223+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3948066f30 msgr2=0x7f39480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:05.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.223+0000 7f396279d700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3948066f30 0x7f39480693e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f394c007c10 tx=0x7f394c009040 comp rx=0 tx=0).stop 2023-10-10T19:20:05.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.223+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f395c0c4740 msgr2=0x7f395c165740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:05.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.223+0000 7f396279d700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f395c0c4740 0x7f395c165740 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3954000f80 tx=0x7f39540017f0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.224+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 shutdown_connections 2023-10-10T19:20:05.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.224+0000 7f396279d700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3948066f30 0x7f39480693e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.224+0000 7f396279d700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f395c0c4740 0x7f395c165740 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.224+0000 7f396279d700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3944004810 0x7f395c0c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.224+0000 7f396279d700 1 --2- 172.21.15.175:0/500232094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f395c0c42d0 0x7f395c165200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:05.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.224+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 >> 172.21.15.175:0/500232094 conn(0x7f395c068280 msgr2=0x7f395c148ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:05.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.224+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 shutdown_connections 2023-10-10T19:20:05.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:05.224+0000 7f396279d700 1 -- 172.21.15.175:0/500232094 wait complete. 2023-10-10T19:20:05.228 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 267 2023-10-10T19:20:05.237 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:20:07.239 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:07.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.361+0000 7fc16d155700 1 Processor -- start 2023-10-10T19:20:07.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.361+0000 7fc16d155700 1 -- start start 2023-10-10T19:20:07.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.362+0000 7fc16d155700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1681573d0 0x7fc1680c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:07.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.362+0000 7fc16d155700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc168154aa0 con 0x7fc1680c4f20 2023-10-10T19:20:07.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.362+0000 7fc16d155700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc168157c20 con 0x7fc1681577a0 2023-10-10T19:20:07.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.362+0000 7fc16d155700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1680bfdf0 con 0x7fc1681573d0 2023-10-10T19:20:07.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.362+0000 7fc166d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1681573d0 0x7fc1680c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:07.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.362+0000 7fc16759e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc1680c4f20 0x7fc16814d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52874/0 (socket says 172.21.15.175:52874) 2023-10-10T19:20:07.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.362+0000 7fc16759e700 1 -- 172.21.15.175:0/3059173428 learned_addr learned my addr 172.21.15.175:0/3059173428 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:20:07.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.363+0000 7fc166d9d700 1 -- 172.21.15.175:0/3059173428 >> v1:172.21.15.191:6789/0 conn(0x7fc1681577a0 legacy=0x7fc1680bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:07.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.363+0000 7fc166d9d700 1 -- 172.21.15.175:0/3059173428 >> v1:172.21.15.175:6789/0 conn(0x7fc1680c4f20 legacy=0x7fc16814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:07.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.363+0000 7fc166d9d700 1 -- 172.21.15.175:0/3059173428 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc168063dc0 con 0x7fc1681573d0 2023-10-10T19:20:07.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.363+0000 7fc16659c700 1 -- 172.21.15.175:0/3059173428 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044938227 0 0) 0x7fc168154aa0 con 0x7fc1680c4f20 2023-10-10T19:20:07.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.363+0000 7fc166d9d700 1 --2- 172.21.15.175:0/3059173428 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1681573d0 0x7fc1680c12a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc1580098e0 tx=0x7fc1580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e216d2670907d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:07.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.363+0000 7fc16659c700 1 -- 172.21.15.175:0/3059173428 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158009e80 con 0x7fc1681573d0 2023-10-10T19:20:07.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.363+0000 7fc16659c700 1 -- 172.21.15.175:0/3059173428 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15800b860 con 0x7fc1681573d0 2023-10-10T19:20:07.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.363+0000 7fc16659c700 1 -- 172.21.15.175:0/3059173428 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15800bbe0 con 0x7fc1681573d0 2023-10-10T19:20:07.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.364+0000 7fc16d155700 1 -- 172.21.15.175:0/3059173428 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1681573d0 msgr2=0x7fc1680c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:07.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.364+0000 7fc16d155700 1 --2- 172.21.15.175:0/3059173428 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1681573d0 0x7fc1680c12a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc1580098e0 tx=0x7fc1580091a0 comp rx=0 tx=0).stop 2023-10-10T19:20:07.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.364+0000 7fc16d155700 1 -- 172.21.15.175:0/3059173428 shutdown_connections 2023-10-10T19:20:07.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.364+0000 7fc16d155700 1 --2- 172.21.15.175:0/3059173428 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1681573d0 0x7fc1680c12a0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:07.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.364+0000 7fc16d155700 1 -- 172.21.15.175:0/3059173428 >> 172.21.15.175:0/3059173428 conn(0x7fc16806dc70 msgr2=0x7fc168063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:07.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.364+0000 7fc16d155700 1 -- 172.21.15.175:0/3059173428 shutdown_connections 2023-10-10T19:20:07.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.364+0000 7fc16d155700 1 -- 172.21.15.175:0/3059173428 wait complete. 2023-10-10T19:20:07.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.365+0000 7fc16d155700 1 Processor -- start 2023-10-10T19:20:07.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.365+0000 7fc16d155700 1 -- start start 2023-10-10T19:20:07.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16d155700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1680c4f20 0x7fc1681557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:07.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16d155700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1681573d0 0x7fc168167e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:07.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16d155700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1681577a0 0x7fc168152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:07.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16d155700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc168154240 con 0x7fc1680c4f20 2023-10-10T19:20:07.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16d155700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1681480d0 con 0x7fc1681573d0 2023-10-10T19:20:07.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16d155700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc168153cc0 con 0x7fc1681577a0 2023-10-10T19:20:07.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1680c4f20 0x7fc1681557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:07.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1680c4f20 0x7fc1681557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44196/0 (socket says 172.21.15.175:44196) 2023-10-10T19:20:07.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 -- 172.21.15.175:0/3108715192 learned_addr learned my addr 172.21.15.175:0/3108715192 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:07.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 -- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1681577a0 msgr2=0x7fc168152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:07.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16759e700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1681577a0 0x7fc168152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:07.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc15e59c700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1681573d0 0x7fc168167e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:07.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1681577a0 0x7fc168152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:07.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 -- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1681573d0 msgr2=0x7fc168167e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:07.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1681573d0 0x7fc168167e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:07.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 -- 172.21.15.175:0/3108715192 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc158009420 con 0x7fc1680c4f20 2023-10-10T19:20:07.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc15e59c700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1681573d0 0x7fc168167e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:20:07.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc166d9d700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1680c4f20 0x7fc1681557d0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc16806e6e0 tx=0x7fc158017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:07.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc15f7fe700 1 -- 172.21.15.175:0/3108715192 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15801f070 con 0x7fc1680c4f20 2023-10-10T19:20:07.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc15f7fe700 1 -- 172.21.15.175:0/3108715192 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158011420 con 0x7fc1680c4f20 2023-10-10T19:20:07.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc168153f60 con 0x7fc1680c4f20 2023-10-10T19:20:07.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc15f7fe700 1 -- 172.21.15.175:0/3108715192 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15800bbe0 con 0x7fc1680c4f20 2023-10-10T19:20:07.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.366+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc168144ce0 con 0x7fc1680c4f20 2023-10-10T19:20:07.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.367+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc168147bc0 con 0x7fc1680c4f20 2023-10-10T19:20:07.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.368+0000 7fc15f7fe700 1 -- 172.21.15.175:0/3108715192 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc158011a10 con 0x7fc1680c4f20 2023-10-10T19:20:07.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.368+0000 7fc15f7fe700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc148066f30 0x7fc1480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:07.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.368+0000 7fc15f7fe700 1 -- 172.21.15.175:0/3108715192 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fc1580860a0 con 0x7fc1680c4f20 2023-10-10T19:20:07.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.368+0000 7fc15e59c700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc148066f30 0x7fc1480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:07.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.369+0000 7fc15e59c700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc148066f30 0x7fc1480693e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc1680c6940 tx=0x7fc150009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:07.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.372+0000 7fc15f7fe700 1 -- 172.21.15.175:0/3108715192 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc168147bc0 con 0x7fc1680c4f20 2023-10-10T19:20:07.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.599+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc168147bc0 con 0x7fc1680c4f20 2023-10-10T19:20:07.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.599+0000 7fc15f7fe700 1 -- 172.21.15.175:0/3108715192 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6181 (secure 0 0 0) 0x7fc168147bc0 con 0x7fc1680c4f20 2023-10-10T19:20:07.601 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:07.602 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7204,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2765861262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2765861262},{"type":"v1","addr":"172.21.15.191:6839","nonce":2765861262}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":267}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:07.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc148066f30 msgr2=0x7fc1480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:07.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc148066f30 0x7fc1480693e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc1680c6940 tx=0x7fc150009040 comp rx=0 tx=0).stop 2023-10-10T19:20:07.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1680c4f20 msgr2=0x7fc1681557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:07.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1680c4f20 0x7fc1681557d0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc16806e6e0 tx=0x7fc158017000 comp rx=0 tx=0).stop 2023-10-10T19:20:07.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 shutdown_connections 2023-10-10T19:20:07.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc148066f30 0x7fc1480693e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:07.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1681577a0 0x7fc168152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:07.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1680c4f20 0x7fc1681557d0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:07.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 --2- 172.21.15.175:0/3108715192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1681573d0 0x7fc168167e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:07.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 >> 172.21.15.175:0/3108715192 conn(0x7fc16806dc70 msgr2=0x7fc16814b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:07.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 shutdown_connections 2023-10-10T19:20:07.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:07.601+0000 7fc16d155700 1 -- 172.21.15.175:0/3108715192 wait complete. 2023-10-10T19:20:07.607 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 267 2023-10-10T19:20:07.615 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:20:09.616 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:09.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.743+0000 7f71610c5700 1 Processor -- start 2023-10-10T19:20:09.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.743+0000 7f71610c5700 1 -- start start 2023-10-10T19:20:09.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.743+0000 7f71610c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:09.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.743+0000 7f71610c5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f715c0681e0 0x7f715c0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:09.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.743+0000 7f71610c5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c156800 con 0x7f715c14fbf0 2023-10-10T19:20:09.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.743+0000 7f71610c5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715c0c59e0 con 0x7f715c0681e0 2023-10-10T19:20:09.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.743+0000 7f71610c5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c15a2b0 con 0x7f715c14ffc0 2023-10-10T19:20:09.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715b59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f715c0681e0 0x7f715c0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:09.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715259c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:09.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715259c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:35676/0 (socket says 172.21.15.175:35676) 2023-10-10T19:20:09.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715ad9d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f715c14fbf0 0x7f715c063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:34566/0 (socket says 172.21.15.175:34566) 2023-10-10T19:20:09.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715b59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f715c0681e0 0x7f715c0c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:39584/0 (socket says 172.21.15.175:39584) 2023-10-10T19:20:09.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715259c700 1 -- 172.21.15.175:0/2125839731 learned_addr learned my addr 172.21.15.175:0/2125839731 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:09.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715b59e700 1 -- 172.21.15.175:0/2125839731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 msgr2=0x7f715c159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:09.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715b59e700 1 --2- 172.21.15.175:0/2125839731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715b59e700 1 -- 172.21.15.175:0/2125839731 >> v1:172.21.15.191:6789/0 conn(0x7f715c14fbf0 legacy=0x7f715c063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:09.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715b59e700 1 -- 172.21.15.175:0/2125839731 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c063e30 con 0x7f715c0681e0 2023-10-10T19:20:09.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715a59c700 1 -- 172.21.15.175:0/2125839731 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662946678 0 0) 0x7f715c156800 con 0x7f715c14fbf0 2023-10-10T19:20:09.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715259c700 1 --2- 172.21.15.175:0/2125839731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:20:09.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715b59e700 1 --2- 172.21.15.175:0/2125839731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c0681e0 0x7f715c0c5fb0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7154009e90 tx=0x7f7154009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=316817128be5d8d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:09.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715a59c700 1 -- 172.21.15.175:0/2125839731 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715400e070 con 0x7f715c0681e0 2023-10-10T19:20:09.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715a59c700 1 -- 172.21.15.175:0/2125839731 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715400bbf0 con 0x7f715c0681e0 2023-10-10T19:20:09.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.744+0000 7f715a59c700 1 -- 172.21.15.175:0/2125839731 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154012430 con 0x7f715c0681e0 2023-10-10T19:20:09.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.745+0000 7f71610c5700 1 -- 172.21.15.175:0/2125839731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c0681e0 msgr2=0x7f715c0c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:09.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.745+0000 7f71610c5700 1 --2- 172.21.15.175:0/2125839731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c0681e0 0x7f715c0c5fb0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7154009e90 tx=0x7f7154009780 comp rx=0 tx=0).stop 2023-10-10T19:20:09.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.745+0000 7f71610c5700 1 -- 172.21.15.175:0/2125839731 shutdown_connections 2023-10-10T19:20:09.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.745+0000 7f71610c5700 1 --2- 172.21.15.175:0/2125839731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c0681e0 0x7f715c0c5fb0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.745+0000 7f71610c5700 1 --2- 172.21.15.175:0/2125839731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.745+0000 7f71610c5700 1 -- 172.21.15.175:0/2125839731 >> 172.21.15.175:0/2125839731 conn(0x7f715c0cc760 msgr2=0x7f715c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:09.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.745+0000 7f71610c5700 1 -- 172.21.15.175:0/2125839731 shutdown_connections 2023-10-10T19:20:09.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.745+0000 7f71610c5700 1 -- 172.21.15.175:0/2125839731 wait complete. 2023-10-10T19:20:09.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.746+0000 7f71610c5700 1 Processor -- start 2023-10-10T19:20:09.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.746+0000 7f71610c5700 1 -- start start 2023-10-10T19:20:09.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f71610c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f715c0681e0 0x7f715c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:09.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f71610c5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c14fbf0 0x7f715c1484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:09.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f71610c5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:09.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f71610c5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c155860 con 0x7f715c0681e0 2023-10-10T19:20:09.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f71610c5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c165c20 con 0x7f715c14fbf0 2023-10-10T19:20:09.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f71610c5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c166a40 con 0x7f715c14ffc0 2023-10-10T19:20:09.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f715c0681e0 0x7f715c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:09.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f715c0681e0 0x7f715c155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44208/0 (socket says 172.21.15.175:44208) 2023-10-10T19:20:09.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715ad9d700 1 -- 172.21.15.175:0/2789115810 learned_addr learned my addr 172.21.15.175:0/2789115810 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:09.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715259c700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c14fbf0 0x7f715c1484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:09.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715b59e700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:09.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715ad9d700 1 -- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 msgr2=0x7f715c14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:09.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715ad9d700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c14a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715ad9d700 1 -- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c14fbf0 msgr2=0x7f715c1484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:09.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715ad9d700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c14fbf0 0x7f715c1484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.747+0000 7f715ad9d700 1 -- 172.21.15.175:0/2789115810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71440094f0 con 0x7f715c0681e0 2023-10-10T19:20:09.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.748+0000 7f715ad9d700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f715c0681e0 0x7f715c155a30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f714c00c840 tx=0x7f714c007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:09.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.748+0000 7f71537fe700 1 -- 172.21.15.175:0/2789115810 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c005b90 con 0x7f715c0681e0 2023-10-10T19:20:09.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.748+0000 7f71537fe700 1 -- 172.21.15.175:0/2789115810 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c005cf0 con 0x7f715c0681e0 2023-10-10T19:20:09.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.748+0000 7f71537fe700 1 -- 172.21.15.175:0/2789115810 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c007320 con 0x7f715c0681e0 2023-10-10T19:20:09.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.748+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7154009a20 con 0x7f715c0681e0 2023-10-10T19:20:09.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.748+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c1663f0 con 0x7f715c0681e0 2023-10-10T19:20:09.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.749+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c067570 con 0x7f715c0681e0 2023-10-10T19:20:09.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.749+0000 7f71537fe700 1 -- 172.21.15.175:0/2789115810 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f714c01b3b0 con 0x7f715c0681e0 2023-10-10T19:20:09.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.749+0000 7f71537fe700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f713c066f30 0x7f713c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:09.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.750+0000 7f71537fe700 1 -- 172.21.15.175:0/2789115810 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f714c0840e0 con 0x7f715c0681e0 2023-10-10T19:20:09.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.750+0000 7f715259c700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f713c066f30 0x7f713c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:09.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.750+0000 7f715259c700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f713c066f30 0x7f713c0693e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f715c14ed00 tx=0x7f7144009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:09.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.754+0000 7f71537fe700 1 -- 172.21.15.175:0/2789115810 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f714c050d20 con 0x7f715c0681e0 2023-10-10T19:20:09.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.982+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f715c1492e0 con 0x7f715c0681e0 2023-10-10T19:20:09.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.982+0000 7f71537fe700 1 -- 172.21.15.175:0/2789115810 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f714c005e60 con 0x7f715c0681e0 2023-10-10T19:20:09.984 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:09.984 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7204,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2765861262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2765861262},{"type":"v1","addr":"172.21.15.191:6839","nonce":2765861262}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":267}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:09.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f713c066f30 msgr2=0x7f713c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:09.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f713c066f30 0x7f713c0693e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f715c14ed00 tx=0x7f7144009250 comp rx=0 tx=0).stop 2023-10-10T19:20:09.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f715c0681e0 msgr2=0x7f715c155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:09.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f715c0681e0 0x7f715c155a30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f714c00c840 tx=0x7f714c007de0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 shutdown_connections 2023-10-10T19:20:09.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f713c066f30 0x7f713c0693e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f715c14ffc0 0x7f715c14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f715c0681e0 0x7f715c155a30 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 --2- 172.21.15.175:0/2789115810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f715c14fbf0 0x7f715c1484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:09.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 >> 172.21.15.175:0/2789115810 conn(0x7f715c0cc760 msgr2=0x7f715c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:09.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.984+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 shutdown_connections 2023-10-10T19:20:09.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:09.985+0000 7f71610c5700 1 -- 172.21.15.175:0/2789115810 wait complete. 2023-10-10T19:20:09.989 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 267 2023-10-10T19:20:09.998 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:20:11.998 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:12.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.124+0000 7fed77a7c700 1 Processor -- start 2023-10-10T19:20:12.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.125+0000 7fed77a7c700 1 -- start start 2023-10-10T19:20:12.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.125+0000 7fed77a7c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fed700c3330 0x7fed700c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:12.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.125+0000 7fed77a7c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fed7006b640 0x7fed7015d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:12.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.125+0000 7fed77a7c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed701533e0 con 0x7fed700c41e0 2023-10-10T19:20:12.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.125+0000 7fed77a7c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7006e130 con 0x7fed7006b640 2023-10-10T19:20:12.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.125+0000 7fed77a7c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7006a970 con 0x7fed700c3330 2023-10-10T19:20:12.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed76019700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fed7006b640 0x7fed7015d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:12.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed76019700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fed7006b640 0x7fed7015d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44220/0 (socket says 172.21.15.175:44220) 2023-10-10T19:20:12.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed76019700 1 -- 172.21.15.175:0/266204570 learned_addr learned my addr 172.21.15.175:0/266204570 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:12.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed75818700 1 --2- 172.21.15.175:0/266204570 >> v2:172.21.15.191:3300/0 conn(0x7fed700c3330 0x7fed700c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:12.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed76019700 1 -- 172.21.15.175:0/266204570 >> v2:172.21.15.191:3300/0 conn(0x7fed700c3330 msgr2=0x7fed700c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:12.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed76019700 1 --2- 172.21.15.175:0/266204570 >> v2:172.21.15.191:3300/0 conn(0x7fed700c3330 0x7fed700c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed76019700 1 -- 172.21.15.175:0/266204570 >> v1:172.21.15.191:6789/0 conn(0x7fed700c41e0 legacy=0x7fed7014cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:12.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed76019700 1 -- 172.21.15.175:0/266204570 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7006d520 con 0x7fed7006b640 2023-10-10T19:20:12.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed76019700 1 --2- 172.21.15.175:0/266204570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed7006b640 0x7fed7015d8d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fed7006cb20 tx=0x7fed68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fe32e391867bfd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:12.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed74816700 1 -- 172.21.15.175:0/266204570 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6800e070 con 0x7fed7006b640 2023-10-10T19:20:12.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed74816700 1 -- 172.21.15.175:0/266204570 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6800ba20 con 0x7fed7006b640 2023-10-10T19:20:12.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.126+0000 7fed74816700 1 -- 172.21.15.175:0/266204570 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680092e0 con 0x7fed7006b640 2023-10-10T19:20:12.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.127+0000 7fed77a7c700 1 -- 172.21.15.175:0/266204570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed7006b640 msgr2=0x7fed7015d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:12.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.127+0000 7fed77a7c700 1 --2- 172.21.15.175:0/266204570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed7006b640 0x7fed7015d8d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fed7006cb20 tx=0x7fed68009840 comp rx=0 tx=0).stop 2023-10-10T19:20:12.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.127+0000 7fed77a7c700 1 -- 172.21.15.175:0/266204570 shutdown_connections 2023-10-10T19:20:12.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.127+0000 7fed77a7c700 1 --2- 172.21.15.175:0/266204570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed7006b640 0x7fed7015d8d0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.127+0000 7fed77a7c700 1 --2- 172.21.15.175:0/266204570 >> v2:172.21.15.191:3300/0 conn(0x7fed700c3330 0x7fed700c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.127+0000 7fed77a7c700 1 -- 172.21.15.175:0/266204570 >> 172.21.15.175:0/266204570 conn(0x7fed70068280 msgr2=0x7fed7006af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:12.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.127+0000 7fed77a7c700 1 -- 172.21.15.175:0/266204570 shutdown_connections 2023-10-10T19:20:12.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.127+0000 7fed77a7c700 1 -- 172.21.15.175:0/266204570 wait complete. 2023-10-10T19:20:12.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.128+0000 7fed77a7c700 1 Processor -- start 2023-10-10T19:20:12.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.128+0000 7fed77a7c700 1 -- start start 2023-10-10T19:20:12.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed77a7c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed7006b640 0x7fed70142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:12.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed77a7c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed700c3330 0x7fed70143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:12.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed77a7c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed700c41e0 0x7fed701444e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:12.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed77a7c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed700c7f10 con 0x7fed700c41e0 2023-10-10T19:20:12.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed77a7c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed70164bf0 con 0x7fed7006b640 2023-10-10T19:20:12.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed77a7c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed70163d20 con 0x7fed700c3330 2023-10-10T19:20:12.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed700c41e0 0x7fed701444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:12.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed700c41e0 0x7fed701444e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44222/0 (socket says 172.21.15.175:44222) 2023-10-10T19:20:12.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 -- 172.21.15.175:0/1182166748 learned_addr learned my addr 172.21.15.175:0/1182166748 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:12.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed75818700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed7006b640 0x7fed70142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:12.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed75017700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed700c3330 0x7fed70143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:12.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 -- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed700c3330 msgr2=0x7fed70143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:12.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed700c3330 0x7fed70143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 -- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed7006b640 msgr2=0x7fed70142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:12.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed7006b640 0x7fed70142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 -- 172.21.15.175:0/1182166748 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed68009ae0 con 0x7fed700c41e0 2023-10-10T19:20:12.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed76019700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed700c41e0 0x7fed701444e0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fed68009880 tx=0x7fed68018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:12.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed65ffb700 1 -- 172.21.15.175:0/1182166748 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68008a80 con 0x7fed700c41e0 2023-10-10T19:20:12.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed65ffb700 1 -- 172.21.15.175:0/1182166748 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68008c00 con 0x7fed700c41e0 2023-10-10T19:20:12.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed70163fe0 con 0x7fed700c41e0 2023-10-10T19:20:12.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.129+0000 7fed65ffb700 1 -- 172.21.15.175:0/1182166748 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68026460 con 0x7fed700c41e0 2023-10-10T19:20:12.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.130+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701645a0 con 0x7fed700c41e0 2023-10-10T19:20:12.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.131+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed70067570 con 0x7fed700c41e0 2023-10-10T19:20:12.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.131+0000 7fed65ffb700 1 -- 172.21.15.175:0/1182166748 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fed68003040 con 0x7fed700c41e0 2023-10-10T19:20:12.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.131+0000 7fed65ffb700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed54066f30 0x7fed540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:12.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.132+0000 7fed65ffb700 1 -- 172.21.15.175:0/1182166748 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fed68086440 con 0x7fed700c41e0 2023-10-10T19:20:12.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.132+0000 7fed75818700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed54066f30 0x7fed540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:12.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.132+0000 7fed75818700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed54066f30 0x7fed540693e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009da0 tx=0x7fed6c009120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:12.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.136+0000 7fed65ffb700 1 -- 172.21.15.175:0/1182166748 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed68053080 con 0x7fed700c41e0 2023-10-10T19:20:12.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.357+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed7014a600 con 0x7fed700c41e0 2023-10-10T19:20:12.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.357+0000 7fed65ffb700 1 -- 172.21.15.175:0/1182166748 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6181 (secure 0 0 0) 0x7fed680566a0 con 0x7fed700c41e0 2023-10-10T19:20:12.359 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:12.360 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7204,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2765861262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2765861262},{"type":"v1","addr":"172.21.15.191:6839","nonce":2765861262}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":267}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:12.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed54066f30 msgr2=0x7fed540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:12.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed54066f30 0x7fed540693e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009da0 tx=0x7fed6c009120 comp rx=0 tx=0).stop 2023-10-10T19:20:12.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed700c41e0 msgr2=0x7fed701444e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:12.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed700c41e0 0x7fed701444e0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fed68009880 tx=0x7fed68018000 comp rx=0 tx=0).stop 2023-10-10T19:20:12.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 shutdown_connections 2023-10-10T19:20:12.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fed54066f30 0x7fed540693e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed700c41e0 0x7fed701444e0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed7006b640 0x7fed70142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 --2- 172.21.15.175:0/1182166748 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed700c3330 0x7fed70143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:12.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.359+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 >> 172.21.15.175:0/1182166748 conn(0x7fed70068280 msgr2=0x7fed7006e950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:12.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.360+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 shutdown_connections 2023-10-10T19:20:12.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:12.360+0000 7fed77a7c700 1 -- 172.21.15.175:0/1182166748 wait complete. 2023-10-10T19:20:12.364 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 267 2023-10-10T19:20:12.373 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:20:14.374 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:14.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.503+0000 7fefd3b4f700 1 Processor -- start 2023-10-10T19:20:14.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.503+0000 7fefd3b4f700 1 -- start start 2023-10-10T19:20:14.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd3b4f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 0x7fefcc0c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd3b4f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fefcc06f620 0x7fefcc0c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd3b4f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefcc06e940 con 0x7fefcc14d190 2023-10-10T19:20:14.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd3b4f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc0c1c90 con 0x7fefcc06f620 2023-10-10T19:20:14.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd3b4f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0c6e20 con 0x7fefcc14cdc0 2023-10-10T19:20:14.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd20ec700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fefcc06f620 0x7fefcc0c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd20ec700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fefcc06f620 0x7fefcc0c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44236/0 (socket says 172.21.15.175:44236) 2023-10-10T19:20:14.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd20ec700 1 -- 172.21.15.175:0/831245622 learned_addr learned my addr 172.21.15.175:0/831245622 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:14.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd10ea700 1 --1- 172.21.15.175:0/831245622 >> v1:172.21.15.175:6789/0 conn(0x7fefcc14d190 0x7fefcc156300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52886/0 (socket says 172.21.15.175:52886) 2023-10-10T19:20:14.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd18eb700 1 --2- 172.21.15.175:0/831245622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 0x7fefcc0c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd20ec700 1 -- 172.21.15.175:0/831245622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 msgr2=0x7fefcc0c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:14.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd20ec700 1 --2- 172.21.15.175:0/831245622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 0x7fefcc0c26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd20ec700 1 -- 172.21.15.175:0/831245622 >> v1:172.21.15.175:6789/0 conn(0x7fefcc14d190 legacy=0x7fefcc156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd20ec700 1 -- 172.21.15.175:0/831245622 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc06ca40 con 0x7fefcc06f620 2023-10-10T19:20:14.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd08e9700 1 -- 172.21.15.175:0/831245622 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583756269 0 0) 0x7fefcc06e940 con 0x7fefcc14d190 2023-10-10T19:20:14.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd20ec700 1 --2- 172.21.15.175:0/831245622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc0c2c20 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fefc40099b0 tx=0x7fefc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b34c4767df67a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:14.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd08e9700 1 -- 172.21.15.175:0/831245622 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc400b860 con 0x7fefcc06f620 2023-10-10T19:20:14.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.504+0000 7fefd08e9700 1 -- 172.21.15.175:0/831245622 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc400b9c0 con 0x7fefcc06f620 2023-10-10T19:20:14.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd08e9700 1 -- 172.21.15.175:0/831245622 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc400bd60 con 0x7fefcc06f620 2023-10-10T19:20:14.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd3b4f700 1 -- 172.21.15.175:0/831245622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 msgr2=0x7fefcc0c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/831245622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc0c2c20 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fefc40099b0 tx=0x7fefc4009250 comp rx=0 tx=0).stop 2023-10-10T19:20:14.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd3b4f700 1 -- 172.21.15.175:0/831245622 shutdown_connections 2023-10-10T19:20:14.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/831245622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc0c2c20 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/831245622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 0x7fefcc0c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd3b4f700 1 -- 172.21.15.175:0/831245622 >> 172.21.15.175:0/831245622 conn(0x7fefcc068900 msgr2=0x7fefcc068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:14.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd3b4f700 1 -- 172.21.15.175:0/831245622 shutdown_connections 2023-10-10T19:20:14.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.505+0000 7fefd3b4f700 1 -- 172.21.15.175:0/831245622 wait complete. 2023-10-10T19:20:14.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.506+0000 7fefd3b4f700 1 Processor -- start 2023-10-10T19:20:14.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.506+0000 7fefd3b4f700 1 -- start start 2023-10-10T19:20:14.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd3b4f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd3b4f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 0x7fefcc142ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd3b4f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc14d190 0x7fefcc14bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd3b4f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc161270 con 0x7fefcc06f620 2023-10-10T19:20:14.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd3b4f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc161730 con 0x7fefcc14d190 2023-10-10T19:20:14.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd3b4f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc161450 con 0x7fefcc14cdc0 2023-10-10T19:20:14.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc142580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44240/0 (socket says 172.21.15.175:44240) 2023-10-10T19:20:14.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 -- 172.21.15.175:0/1552334876 learned_addr learned my addr 172.21.15.175:0/1552334876 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:14.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd10ea700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 0x7fefcc142ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd20ec700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc14d190 0x7fefcc14bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 -- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 msgr2=0x7fefcc142ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 0x7fefcc142ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 -- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc14d190 msgr2=0x7fefcc14bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc14d190 0x7fefcc14bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 -- 172.21.15.175:0/1552334876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc40094f0 con 0x7fefcc06f620 2023-10-10T19:20:14.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd20ec700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc14d190 0x7fefcc14bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:20:14.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.507+0000 7fefd18eb700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc142580 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fefc800a320 tx=0x7fefc8007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:14.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.508+0000 7fefc1ffb700 1 -- 172.21.15.175:0/1552334876 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc800ae40 con 0x7fefcc06f620 2023-10-10T19:20:14.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.508+0000 7fefc1ffb700 1 -- 172.21.15.175:0/1552334876 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc800e660 con 0x7fefcc06f620 2023-10-10T19:20:14.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.508+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc163d90 con 0x7fefcc06f620 2023-10-10T19:20:14.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.508+0000 7fefc1ffb700 1 -- 172.21.15.175:0/1552334876 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc800ea00 con 0x7fefcc06f620 2023-10-10T19:20:14.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.508+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc164350 con 0x7fefcc06f620 2023-10-10T19:20:14.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.509+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc067570 con 0x7fefcc06f620 2023-10-10T19:20:14.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.509+0000 7fefc1ffb700 1 -- 172.21.15.175:0/1552334876 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fefc8003020 con 0x7fefcc06f620 2023-10-10T19:20:14.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.509+0000 7fefc1ffb700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb0066f60 0x7fefb0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.510+0000 7fefc1ffb700 1 -- 172.21.15.175:0/1552334876 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fefc8012070 con 0x7fefcc06f620 2023-10-10T19:20:14.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.510+0000 7fefd10ea700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb0066f60 0x7fefb0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.510+0000 7fefd10ea700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb0066f60 0x7fefb0069410 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fefbc007fd0 tx=0x7fefbc007e10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:14.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.514+0000 7fefc1ffb700 1 -- 172.21.15.175:0/1552334876 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fefc8052900 con 0x7fefcc06f620 2023-10-10T19:20:14.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.738+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefcc0c8c50 con 0x7fefcc06f620 2023-10-10T19:20:14.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.739+0000 7fefc1ffb700 1 -- 172.21.15.175:0/1552334876 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+5420 (secure 0 0 0) 0x7fefc8055f20 con 0x7fefcc06f620 2023-10-10T19:20:14.740 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:14.740 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:14.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb0066f60 msgr2=0x7fefb0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb0066f60 0x7fefb0069410 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fefbc007fd0 tx=0x7fefbc007e10 comp rx=0 tx=0).stop 2023-10-10T19:20:14.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 msgr2=0x7fefcc142580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc142580 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fefc800a320 tx=0x7fefc8007900 comp rx=0 tx=0).stop 2023-10-10T19:20:14.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 shutdown_connections 2023-10-10T19:20:14.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefb0066f60 0x7fefb0069410 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefcc14d190 0x7fefcc14bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefcc06f620 0x7fefcc142580 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 --2- 172.21.15.175:0/1552334876 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefcc14cdc0 0x7fefcc142ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 >> 172.21.15.175:0/1552334876 conn(0x7fefcc068900 msgr2=0x7fefcc0c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:14.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 shutdown_connections 2023-10-10T19:20:14.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.741+0000 7fefd3b4f700 1 -- 172.21.15.175:0/1552334876 wait complete. 2023-10-10T19:20:14.745 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 268 2023-10-10T19:20:14.754 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-10T19:20:14.754 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:20:14.755 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:14.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.868+0000 7f1a43de5700 1 Processor -- start 2023-10-10T19:20:14.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.868+0000 7f1a43de5700 1 -- start start 2023-10-10T19:20:14.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a43de5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1a3c0c12a0 0x7f1a3c1577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a43de5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c4f20 0x7f1a3c14d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a43de5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3c154aa0 con 0x7f1a3c1573d0 2023-10-10T19:20:14.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a43de5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3c0bfdf0 con 0x7f1a3c0c12a0 2023-10-10T19:20:14.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a43de5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c063dc0 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a41380700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1a3c0c12a0 0x7f1a3c1577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a41b81700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1a3c1573d0 0x7f1a3c0bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49432/0 (socket says 172.21.15.175:49432) 2023-10-10T19:20:14.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a41b81700 1 -- 172.21.15.175:0/763511218 learned_addr learned my addr 172.21.15.175:0/763511218 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:20:14.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a42382700 1 --2- 172.21.15.175:0/763511218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c4f20 0x7f1a3c14d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a41380700 1 -- 172.21.15.175:0/763511218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c4f20 msgr2=0x7f1a3c14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:14.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a41380700 1 --2- 172.21.15.175:0/763511218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c4f20 0x7f1a3c14d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a41380700 1 -- 172.21.15.175:0/763511218 >> v1:172.21.15.175:6789/0 conn(0x7f1a3c1573d0 legacy=0x7f1a3c0bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a41380700 1 -- 172.21.15.175:0/763511218 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0c5340 con 0x7f1a3c0c12a0 2023-10-10T19:20:14.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a40b7f700 1 -- 172.21.15.175:0/763511218 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308331030 0 0) 0x7f1a3c154aa0 con 0x7f1a3c1573d0 2023-10-10T19:20:14.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a42382700 1 --2- 172.21.15.175:0/763511218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c4f20 0x7f1a3c14d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:20:14.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a41380700 1 --2- 172.21.15.175:0/763511218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c12a0 0x7f1a3c1577a0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0098e0 tx=0x7f1a2c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4a8633103a771 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:14.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a40b7f700 1 -- 172.21.15.175:0/763511218 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c009e80 con 0x7f1a3c0c12a0 2023-10-10T19:20:14.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a40b7f700 1 -- 172.21.15.175:0/763511218 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c00b860 con 0x7f1a3c0c12a0 2023-10-10T19:20:14.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.869+0000 7f1a40b7f700 1 -- 172.21.15.175:0/763511218 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00bbe0 con 0x7f1a3c0c12a0 2023-10-10T19:20:14.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.870+0000 7f1a43de5700 1 -- 172.21.15.175:0/763511218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c12a0 msgr2=0x7f1a3c1577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.870+0000 7f1a43de5700 1 --2- 172.21.15.175:0/763511218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c12a0 0x7f1a3c1577a0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0098e0 tx=0x7f1a2c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.870+0000 7f1a43de5700 1 -- 172.21.15.175:0/763511218 shutdown_connections 2023-10-10T19:20:14.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.870+0000 7f1a43de5700 1 --2- 172.21.15.175:0/763511218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c4f20 0x7f1a3c14d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.870+0000 7f1a43de5700 1 --2- 172.21.15.175:0/763511218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c12a0 0x7f1a3c1577a0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.870+0000 7f1a43de5700 1 -- 172.21.15.175:0/763511218 >> 172.21.15.175:0/763511218 conn(0x7f1a3c06dc70 msgr2=0x7f1a3c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:14.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.870+0000 7f1a43de5700 1 -- 172.21.15.175:0/763511218 shutdown_connections 2023-10-10T19:20:14.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.870+0000 7f1a43de5700 1 -- 172.21.15.175:0/763511218 wait complete. 2023-10-10T19:20:14.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.871+0000 7f1a43de5700 1 Processor -- start 2023-10-10T19:20:14.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.871+0000 7f1a43de5700 1 -- start start 2023-10-10T19:20:14.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a43de5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c12a0 0x7f1a3c151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a43de5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c4f20 0x7f1a3c069d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a43de5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a3c1573d0 0x7f1a3c152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a43de5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0b0640 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a43de5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0697e0 con 0x7f1a3c1573d0 2023-10-10T19:20:14.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a43de5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c069540 con 0x7f1a3c0c12a0 2023-10-10T19:20:14.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c4f20 0x7f1a3c069d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c4f20 0x7f1a3c069d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46458/0 (socket says 172.21.15.175:46458) 2023-10-10T19:20:14.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 -- 172.21.15.175:0/2835449443 learned_addr learned my addr 172.21.15.175:0/2835449443 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:14.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41b81700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c12a0 0x7f1a3c151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a42382700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a3c1573d0 0x7f1a3c152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 -- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c12a0 msgr2=0x7f1a3c151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c12a0 0x7f1a3c151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 -- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a3c1573d0 msgr2=0x7f1a3c152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:14.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a3c1573d0 0x7f1a3c152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:14.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 -- 172.21.15.175:0/2835449443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c009420 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a41380700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c4f20 0x7f1a3c069d20 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c06e6e0 tx=0x7f1a2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:14.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.872+0000 7f1a31ffb700 1 -- 172.21.15.175:0/2835449443 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00bbe0 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.873+0000 7f1a31ffb700 1 -- 172.21.15.175:0/2835449443 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c011420 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.873+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3c152590 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.873+0000 7f1a31ffb700 1 -- 172.21.15.175:0/2835449443 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c0117e0 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.873+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c0b0080 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.874+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3c067570 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.874+0000 7f1a31ffb700 1 -- 172.21.15.175:0/2835449443 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1a2c011b90 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.874+0000 7f1a31ffb700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1a20066f30 0x7f1a200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:14.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.875+0000 7f1a31ffb700 1 -- 172.21.15.175:0/2835449443 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f1a2c086680 con 0x7f1a3c0c4f20 2023-10-10T19:20:14.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.875+0000 7f1a41b81700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1a20066f30 0x7f1a200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:14.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.875+0000 7f1a41b81700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1a20066f30 0x7f1a200693e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1a380016b0 tx=0x7f1a38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:14.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:14.879+0000 7f1a31ffb700 1 -- 172.21.15.175:0/2835449443 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1a2c0532c0 con 0x7f1a3c0c4f20 2023-10-10T19:20:15.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.103+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a3c152720 con 0x7f1a3c0c4f20 2023-10-10T19:20:15.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.104+0000 7f1a31ffb700 1 -- 172.21.15.175:0/2835449443 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+5420 (secure 0 0 0) 0x7f1a2c0568e0 con 0x7f1a3c0c4f20 2023-10-10T19:20:15.106 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:15.106 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:15.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1a20066f30 msgr2=0x7f1a200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:15.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1a20066f30 0x7f1a200693e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1a380016b0 tx=0x7f1a38009040 comp rx=0 tx=0).stop 2023-10-10T19:20:15.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c4f20 msgr2=0x7f1a3c069d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:15.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c4f20 0x7f1a3c069d20 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c06e6e0 tx=0x7f1a2c017000 comp rx=0 tx=0).stop 2023-10-10T19:20:15.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 shutdown_connections 2023-10-10T19:20:15.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1a20066f30 0x7f1a200693e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:15.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a3c1573d0 0x7f1a3c152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:15.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1a3c0c12a0 0x7f1a3c151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:15.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 --2- 172.21.15.175:0/2835449443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a3c0c4f20 0x7f1a3c069d20 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:15.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 >> 172.21.15.175:0/2835449443 conn(0x7f1a3c06dc70 msgr2=0x7f1a3c0c5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:15.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 shutdown_connections 2023-10-10T19:20:15.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:15.106+0000 7f1a43de5700 1 -- 172.21.15.175:0/2835449443 wait complete. 2023-10-10T19:20:15.110 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 268 2023-10-10T19:20:15.119 INFO:tasks.mds_thrash.fs.[cephfs]:cluster has 4 actives (max_mds is 4), no MDS can replace rank 4 2023-10-10T19:20:15.119 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2023-10-10T19:20:22.120 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-10T19:20:22.120 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-10T19:20:22.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-10T19:20:22.124 INFO:tasks.ceph.mds.f:Started 2023-10-10T19:20:22.124 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:22.204 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-10T19:20:22.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.252+0000 7f1e498ab700 1 Processor -- start 2023-10-10T19:20:22.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.252+0000 7f1e498ab700 1 -- start start 2023-10-10T19:20:22.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e498ab700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e4408a100 0x7f1e44159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:22.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e498ab700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1e440cd040 0x7f1e440684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:22.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e498ab700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e44067fa0 con 0x7f1e4414fd90 2023-10-10T19:20:22.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e498ab700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e440c5600 con 0x7f1e4408a100 2023-10-10T19:20:22.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e498ab700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e44156b70 con 0x7f1e440cd040 2023-10-10T19:20:22.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e43fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e4408a100 0x7f1e44159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:22.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e43fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e4408a100 0x7f1e44159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46472/0 (socket says 172.21.15.175:46472) 2023-10-10T19:20:22.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e43fff700 1 -- 172.21.15.175:0/887050860 learned_addr learned my addr 172.21.15.175:0/887050860 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:22.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e490aa700 1 --2- 172.21.15.175:0/887050860 >> v2:172.21.15.191:3300/0 conn(0x7f1e440cd040 0x7f1e440684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:22.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e43fff700 1 -- 172.21.15.175:0/887050860 >> v2:172.21.15.191:3300/0 conn(0x7f1e440cd040 msgr2=0x7f1e440684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:22.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e43fff700 1 --2- 172.21.15.175:0/887050860 >> v2:172.21.15.191:3300/0 conn(0x7f1e440cd040 0x7f1e440684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e43fff700 1 -- 172.21.15.175:0/887050860 >> v1:172.21.15.191:6789/0 conn(0x7f1e4414fd90 legacy=0x7f1e440c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:22.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e43fff700 1 -- 172.21.15.175:0/887050860 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e44150160 con 0x7f1e4408a100 2023-10-10T19:20:22.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.253+0000 7f1e43fff700 1 --2- 172.21.15.175:0/887050860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4408a100 0x7f1e44159c30 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f1e4408a6e0 tx=0x7f1e30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d18e550668b3cbd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:22.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e437fe700 1 -- 172.21.15.175:0/887050860 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3000b8b0 con 0x7f1e4408a100 2023-10-10T19:20:22.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e437fe700 1 -- 172.21.15.175:0/887050860 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3000ba10 con 0x7f1e4408a100 2023-10-10T19:20:22.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e437fe700 1 -- 172.21.15.175:0/887050860 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3000bdb0 con 0x7f1e4408a100 2023-10-10T19:20:22.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e498ab700 1 -- 172.21.15.175:0/887050860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4408a100 msgr2=0x7f1e44159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:22.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e498ab700 1 --2- 172.21.15.175:0/887050860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4408a100 0x7f1e44159c30 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f1e4408a6e0 tx=0x7f1e30009250 comp rx=0 tx=0).stop 2023-10-10T19:20:22.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e498ab700 1 -- 172.21.15.175:0/887050860 shutdown_connections 2023-10-10T19:20:22.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e498ab700 1 --2- 172.21.15.175:0/887050860 >> v2:172.21.15.191:3300/0 conn(0x7f1e440cd040 0x7f1e440684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e498ab700 1 --2- 172.21.15.175:0/887050860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4408a100 0x7f1e44159c30 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e498ab700 1 -- 172.21.15.175:0/887050860 >> 172.21.15.175:0/887050860 conn(0x7f1e44063d00 msgr2=0x7f1e440cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:22.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e498ab700 1 -- 172.21.15.175:0/887050860 shutdown_connections 2023-10-10T19:20:22.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.254+0000 7f1e498ab700 1 -- 172.21.15.175:0/887050860 wait complete. 2023-10-10T19:20:22.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.255+0000 7f1e498ab700 1 Processor -- start 2023-10-10T19:20:22.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.255+0000 7f1e498ab700 1 -- start start 2023-10-10T19:20:22.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e498ab700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e4408a100 0x7f1e4414d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:22.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e498ab700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e440cd040 0x7f1e4414d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:22.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e498ab700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4414fd90 0x7f1e44169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:22.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e498ab700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4414e660 con 0x7f1e4414fd90 2023-10-10T19:20:22.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e498ab700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4414f260 con 0x7f1e4408a100 2023-10-10T19:20:22.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e498ab700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4414ef30 con 0x7f1e440cd040 2023-10-10T19:20:22.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e490aa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4414fd90 0x7f1e44169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:22.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e490aa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4414fd90 0x7f1e44169960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46476/0 (socket says 172.21.15.175:46476) 2023-10-10T19:20:22.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e488a9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e4408a100 0x7f1e4414d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:22.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e490aa700 1 -- 172.21.15.175:0/1303233419 learned_addr learned my addr 172.21.15.175:0/1303233419 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:22.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.256+0000 7f1e43fff700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e440cd040 0x7f1e4414d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:22.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e490aa700 1 -- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e440cd040 msgr2=0x7f1e4414d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:22.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e490aa700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e440cd040 0x7f1e4414d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e490aa700 1 -- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e4408a100 msgr2=0x7f1e4414d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:22.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e490aa700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e4408a100 0x7f1e4414d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e490aa700 1 -- 172.21.15.175:0/1303233419 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e300094f0 con 0x7f1e4414fd90 2023-10-10T19:20:22.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e490aa700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4414fd90 0x7f1e44169960 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009fd0 tx=0x7f1e3c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:22.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e40ff9700 1 -- 172.21.15.175:0/1303233419 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c00ebf0 con 0x7f1e4414fd90 2023-10-10T19:20:22.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e40ff9700 1 -- 172.21.15.175:0/1303233419 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c00ed50 con 0x7f1e4414fd90 2023-10-10T19:20:22.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e44150fa0 con 0x7f1e4414fd90 2023-10-10T19:20:22.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e40ff9700 1 -- 172.21.15.175:0/1303233419 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c014670 con 0x7f1e4414fd90 2023-10-10T19:20:22.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.257+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e440c4330 con 0x7f1e4414fd90 2023-10-10T19:20:22.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.258+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e44067570 con 0x7f1e4414fd90 2023-10-10T19:20:22.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.258+0000 7f1e40ff9700 1 -- 172.21.15.175:0/1303233419 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1e3c0149e0 con 0x7f1e4414fd90 2023-10-10T19:20:22.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.259+0000 7f1e40ff9700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:22.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.259+0000 7f1e40ff9700 1 -- 172.21.15.175:0/1303233419 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f1e3c085230 con 0x7f1e4414fd90 2023-10-10T19:20:22.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.259+0000 7f1e488a9700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:22.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.259+0000 7f1e488a9700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009af0 tx=0x7f1e38009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:22.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.263+0000 7f1e40ff9700 1 -- 172.21.15.175:0/1303233419 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1e3c051dd0 con 0x7f1e4414fd90 2023-10-10T19:20:22.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.491+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e4414f3a0 con 0x7f1e4414fd90 2023-10-10T19:20:22.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.491+0000 7f1e40ff9700 1 -- 172.21.15.175:0/1303233419 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+5420 (secure 0 0 0) 0x7f1e3c0553f0 con 0x7f1e4414fd90 2023-10-10T19:20:22.493 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:22.493 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:22.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.493+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 msgr2=0x7f1e28069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:22.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.493+0000 7f1e498ab700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009af0 tx=0x7f1e38009060 comp rx=0 tx=0).stop 2023-10-10T19:20:22.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.493+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4414fd90 msgr2=0x7f1e44169960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:22.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.493+0000 7f1e498ab700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4414fd90 0x7f1e44169960 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009fd0 tx=0x7f1e3c009900 comp rx=0 tx=0).stop 2023-10-10T19:20:22.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.494+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 shutdown_connections 2023-10-10T19:20:22.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.494+0000 7f1e498ab700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.494+0000 7f1e498ab700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e4414fd90 0x7f1e44169960 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.494+0000 7f1e498ab700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e4408a100 0x7f1e4414d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.494+0000 7f1e498ab700 1 --2- 172.21.15.175:0/1303233419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e440cd040 0x7f1e4414d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:22.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.494+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 >> 172.21.15.175:0/1303233419 conn(0x7f1e44063d00 msgr2=0x7f1e441582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:22.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.494+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 shutdown_connections 2023-10-10T19:20:22.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:22.494+0000 7f1e498ab700 1 -- 172.21.15.175:0/1303233419 wait complete. 2023-10-10T19:20:22.498 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 268 2023-10-10T19:20:22.507 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-10T19:20:24.508 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:24.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.633+0000 7f5b366d3700 1 Processor -- start 2023-10-10T19:20:24.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.633+0000 7f5b366d3700 1 -- start start 2023-10-10T19:20:24.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.634+0000 7f5b366d3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b3008a100 0x7f5b30159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:24.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.634+0000 7f5b366d3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b30067fa0 con 0x7f5b300cd040 2023-10-10T19:20:24.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.634+0000 7f5b366d3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b30156b70 con 0x7f5b3014fd90 2023-10-10T19:20:24.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.634+0000 7f5b366d3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300c5600 con 0x7f5b3008a100 2023-10-10T19:20:24.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.634+0000 7f5b35ed2700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5b300cd040 0x7f5b300684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56260/0 (socket says 172.21.15.175:56260) 2023-10-10T19:20:24.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.634+0000 7f5b35ed2700 1 -- 172.21.15.175:0/2604699319 learned_addr learned my addr 172.21.15.175:0/2604699319 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:20:24.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.634+0000 7f5b34ed0700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b3008a100 0x7f5b30159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:24.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.634+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927062740 0 0) 0x7f5b30067fa0 con 0x7f5b300cd040 2023-10-10T19:20:24.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c003570 con 0x7f5b300cd040 2023-10-10T19:20:24.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3881391986 0 0) 0x7f5b1c003570 con 0x7f5b300cd040 2023-10-10T19:20:24.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b3008a100 msgr2=0x7f5b30159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:24.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b3008a100 0x7f5b30159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 >> v1:172.21.15.191:6789/0 conn(0x7f5b3014fd90 legacy=0x7f5b300c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:24.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b30150160 con 0x7f5b300cd040 2023-10-10T19:20:24.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f5b280017f0 con 0x7f5b300cd040 2023-10-10T19:20:24.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 >> v1:172.21.15.175:6789/0 conn(0x7f5b300cd040 legacy=0x7f5b300684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:24.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b3008a100 0x7f5b1c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:24.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b1c004010 0x7f5b1c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:24.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b1c004d50 0x7f5b1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:24.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300c5600 con 0x7f5b1c004010 2023-10-10T19:20:24.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c00bbf0 con 0x7f5b1c004d50 2023-10-10T19:20:24.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c00bdb0 con 0x7f5b3008a100 2023-10-10T19:20:24.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b34ed0700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b1c004010 0x7f5b1c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:24.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b356d1700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b3008a100 0x7f5b1c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:24.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b35ed2700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b1c004d50 0x7f5b1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:24.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b34ed0700 1 -- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b3008a100 msgr2=0x7f5b1c003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:24.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b34ed0700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b3008a100 0x7f5b1c003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b34ed0700 1 -- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b1c004d50 msgr2=0x7f5b1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:24.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b34ed0700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b1c004d50 0x7f5b1c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b34ed0700 1 -- 172.21.15.175:0/2604699319 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c003570 con 0x7f5b1c004010 2023-10-10T19:20:24.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.635+0000 7f5b34ed0700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b1c004010 0x7f5b1c004810 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f5b20006f00 tx=0x7f5b200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113f52806389852 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:24.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2000e710 con 0x7f5b1c004010 2023-10-10T19:20:24.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2000e870 con 0x7f5b1c004010 2023-10-10T19:20:24.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b27fff700 1 -- 172.21.15.175:0/2604699319 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2000ec30 con 0x7f5b1c004010 2023-10-10T19:20:24.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b366d3700 1 -- 172.21.15.175:0/2604699319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b1c004010 msgr2=0x7f5b1c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:24.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b366d3700 1 --2- 172.21.15.175:0/2604699319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b1c004010 0x7f5b1c004810 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f5b20006f00 tx=0x7f5b200091a0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b35ed2700 1 -- 172.21.15.175:0/2604699319 reap_dead start 2023-10-10T19:20:24.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b366d3700 1 -- 172.21.15.175:0/2604699319 shutdown_connections 2023-10-10T19:20:24.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b366d3700 1 -- 172.21.15.175:0/2604699319 >> 172.21.15.175:0/2604699319 conn(0x7f5b30063d00 msgr2=0x7f5b300cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:24.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b366d3700 1 -- 172.21.15.175:0/2604699319 shutdown_connections 2023-10-10T19:20:24.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.636+0000 7f5b366d3700 1 -- 172.21.15.175:0/2604699319 wait complete. 2023-10-10T19:20:24.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.637+0000 7f5b366d3700 1 Processor -- start 2023-10-10T19:20:24.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.637+0000 7f5b366d3700 1 -- start start 2023-10-10T19:20:24.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b366d3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b1c004010 0x7f5b300b16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:24.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b366d3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b30158cc0 0x7f5b30166660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:24.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b366d3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b30159160 0x7f5b30169950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:24.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b366d3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300c8a40 con 0x7f5b30159160 2023-10-10T19:20:24.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b366d3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b301661d0 con 0x7f5b30158cc0 2023-10-10T19:20:24.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b366d3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30165e80 con 0x7f5b1c004010 2023-10-10T19:20:24.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b35ed2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b30159160 0x7f5b30169950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:24.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b35ed2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b30159160 0x7f5b30169950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:41052/0 (socket says 172.21.15.175:41052) 2023-10-10T19:20:24.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b35ed2700 1 -- 172.21.15.175:0/2334131696 learned_addr learned my addr 172.21.15.175:0/2334131696 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:24.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b34ed0700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b30158cc0 0x7f5b30166660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:24.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.638+0000 7f5b356d1700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b1c004010 0x7f5b300b16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:24.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b35ed2700 1 -- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b1c004010 msgr2=0x7f5b300b16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:24.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b35ed2700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b1c004010 0x7f5b300b16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b35ed2700 1 -- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b30158cc0 msgr2=0x7f5b30166660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:24.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b35ed2700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b30158cc0 0x7f5b30166660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b35ed2700 1 -- 172.21.15.175:0/2334131696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300c8c90 con 0x7f5b30159160 2023-10-10T19:20:24.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b35ed2700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b30159160 0x7f5b30169950 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f5b300c35e0 tx=0x7f5b300c5dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:24.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b257fa700 1 -- 172.21.15.175:0/2334131696 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800ab60 con 0x7f5b30159160 2023-10-10T19:20:24.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b257fa700 1 -- 172.21.15.175:0/2334131696 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2800acc0 con 0x7f5b30159160 2023-10-10T19:20:24.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b300a7c10 con 0x7f5b30159160 2023-10-10T19:20:24.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b257fa700 1 -- 172.21.15.175:0/2334131696 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28013620 con 0x7f5b30159160 2023-10-10T19:20:24.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.639+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b300c6fa0 con 0x7f5b30159160 2023-10-10T19:20:24.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.640+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b300a80b0 con 0x7f5b30159160 2023-10-10T19:20:24.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.640+0000 7f5b257fa700 1 -- 172.21.15.175:0/2334131696 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5b28013a50 con 0x7f5b30159160 2023-10-10T19:20:24.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.641+0000 7f5b257fa700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5b14066f60 0x7f5b14069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:24.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.641+0000 7f5b257fa700 1 -- 172.21.15.175:0/2334131696 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f5b28085150 con 0x7f5b30159160 2023-10-10T19:20:24.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.641+0000 7f5b356d1700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5b14066f60 0x7f5b14069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:24.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.641+0000 7f5b356d1700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5b14066f60 0x7f5b14069410 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c003b80 tx=0x7f5b2c007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:24.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.645+0000 7f5b257fa700 1 -- 172.21.15.175:0/2334131696 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b300a80b0 con 0x7f5b30159160 2023-10-10T19:20:24.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.870+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b300a80b0 con 0x7f5b30159160 2023-10-10T19:20:24.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.870+0000 7f5b257fa700 1 -- 172.21.15.175:0/2334131696 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f5b300a80b0 con 0x7f5b30159160 2023-10-10T19:20:24.872 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:24.872 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":269}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:24.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5b14066f60 msgr2=0x7f5b14069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:24.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5b14066f60 0x7f5b14069410 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c003b80 tx=0x7f5b2c007d30 comp rx=0 tx=0).stop 2023-10-10T19:20:24.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b30159160 msgr2=0x7f5b30169950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:24.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b30159160 0x7f5b30169950 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f5b300c35e0 tx=0x7f5b300c5dd0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 shutdown_connections 2023-10-10T19:20:24.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5b14066f60 0x7f5b14069410 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5b30159160 0x7f5b30169950 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b1c004010 0x7f5b300b16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 --2- 172.21.15.175:0/2334131696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b30158cc0 0x7f5b30166660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:24.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 >> 172.21.15.175:0/2334131696 conn(0x7f5b30063d00 msgr2=0x7f5b300c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:24.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 shutdown_connections 2023-10-10T19:20:24.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:24.872+0000 7f5b366d3700 1 -- 172.21.15.175:0/2334131696 wait complete. 2023-10-10T19:20:24.877 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 269 2023-10-10T19:20:24.885 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-10T19:20:24.886 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 26 secs before thrashing 2023-10-10T19:20:24.990 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:20:24.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:20:50.887 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:51.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.018+0000 7f6ffd5f0700 1 Processor -- start 2023-10-10T19:20:51.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.018+0000 7f6ffd5f0700 1 -- start start 2023-10-10T19:20:51.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ffd5f0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ffd5f0700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff80c4ee0 con 0x7f6ff80cac80 2023-10-10T19:20:51.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ffd5f0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff80c5780 con 0x7f6ff8068e10 2023-10-10T19:20:51.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ffd5f0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8064180 con 0x7f6ff8068a40 2023-10-10T19:20:51.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff67fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6ff80cac80 0x7f6ff806b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42638/0 (socket says 172.21.15.175:42638) 2023-10-10T19:20:51.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff67fc700 1 -- 172.21.15.175:0/2490825113 learned_addr learned my addr 172.21.15.175:0/2490825113 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:20:51.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff6ffd700 1 --2- 172.21.15.175:0/2490825113 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff5ffb700 1 -- 172.21.15.175:0/2490825113 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146191356 0 0) 0x7f6ff80c4ee0 con 0x7f6ff80cac80 2023-10-10T19:20:51.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff5ffb700 1 -- 172.21.15.175:0/2490825113 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe0003570 con 0x7f6ff80cac80 2023-10-10T19:20:51.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff5ffb700 1 -- 172.21.15.175:0/2490825113 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354205472 0 0) 0x7f6ff80c5780 con 0x7f6ff8068e10 2023-10-10T19:20:51.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff5ffb700 1 -- 172.21.15.175:0/2490825113 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff80c4ee0 con 0x7f6ff8068e10 2023-10-10T19:20:51.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff5ffb700 1 -- 172.21.15.175:0/2490825113 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180751745 0 0) 0x7f6fe0003570 con 0x7f6ff80cac80 2023-10-10T19:20:51.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff6ffd700 1 -- 172.21.15.175:0/2490825113 >> v1:172.21.15.191:6789/0 conn(0x7f6ff8068e10 legacy=0x7f6ff80af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff6ffd700 1 -- 172.21.15.175:0/2490825113 >> v1:172.21.15.175:6789/0 conn(0x7f6ff80cac80 legacy=0x7f6ff806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.019+0000 7f6ff6ffd700 1 -- 172.21.15.175:0/2490825113 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80c6710 con 0x7f6ff8068a40 2023-10-10T19:20:51.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ff6ffd700 1 --2- 172.21.15.175:0/2490825113 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8063c40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009900 tx=0x7f6fec009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6c0bc7bf42dbf16 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ff5ffb700 1 -- 172.21.15.175:0/2490825113 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe80015f0 con 0x7f6ff8068a40 2023-10-10T19:20:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ff5ffb700 1 -- 172.21.15.175:0/2490825113 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec011420 con 0x7f6ff8068a40 2023-10-10T19:20:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ff5ffb700 1 -- 172.21.15.175:0/2490825113 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec0117e0 con 0x7f6ff8068a40 2023-10-10T19:20:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/2490825113 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 msgr2=0x7f6ff8063c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ffd5f0700 1 --2- 172.21.15.175:0/2490825113 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8063c40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009900 tx=0x7f6fec009120 comp rx=0 tx=0).stop 2023-10-10T19:20:51.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/2490825113 shutdown_connections 2023-10-10T19:20:51.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ffd5f0700 1 --2- 172.21.15.175:0/2490825113 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8063c40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/2490825113 >> 172.21.15.175:0/2490825113 conn(0x7f6ff806dbe0 msgr2=0x7f6ff80ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:51.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/2490825113 shutdown_connections 2023-10-10T19:20:51.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.020+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/2490825113 wait complete. 2023-10-10T19:20:51.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ffd5f0700 1 Processor -- start 2023-10-10T19:20:51.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ffd5f0700 1 -- start start 2023-10-10T19:20:51.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ffd5f0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ffd5f0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ff8068e10 0x7f6ff8150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ffd5f0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ff80cac80 0x7f6ff81512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ffd5f0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff81532c0 con 0x7f6ff8068e10 2023-10-10T19:20:51.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ffd5f0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8153d50 con 0x7f6ff80cac80 2023-10-10T19:20:51.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ffd5f0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff815dbf0 con 0x7f6ff8068a40 2023-10-10T19:20:51.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ff67fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ff8068e10 0x7f6ff8150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ff67fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ff8068e10 0x7f6ff8150d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37776/0 (socket says 172.21.15.175:37776) 2023-10-10T19:20:51.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.022+0000 7f6ff67fc700 1 -- 172.21.15.175:0/286175403 learned_addr learned my addr 172.21.15.175:0/286175403 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:51.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff6ffd700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff77fe700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ff80cac80 0x7f6ff81512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff67fc700 1 -- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 msgr2=0x7f6ff8155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff67fc700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff67fc700 1 -- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ff80cac80 msgr2=0x7f6ff81512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff67fc700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ff80cac80 0x7f6ff81512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff67fc700 1 -- 172.21.15.175:0/286175403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0093c0 con 0x7f6ff8068e10 2023-10-10T19:20:51.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff6ffd700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:20:51.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ff67fc700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ff8068e10 0x7f6ff8150d70 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800cba0 tx=0x7f6fe8002b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:51.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6fdeffd700 1 -- 172.21.15.175:0/286175403 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8007e40 con 0x7f6ff8068e10 2023-10-10T19:20:51.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6fdeffd700 1 -- 172.21.15.175:0/286175403 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe8007300 con 0x7f6ff8068e10 2023-10-10T19:20:51.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6fdeffd700 1 -- 172.21.15.175:0/286175403 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe80057f0 con 0x7f6ff8068e10 2023-10-10T19:20:51.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff8155e90 con 0x7f6ff8068e10 2023-10-10T19:20:51.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.023+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff8152990 con 0x7f6ff8068e10 2023-10-10T19:20:51.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.024+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8067570 con 0x7f6ff8068e10 2023-10-10T19:20:51.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.025+0000 7f6fdeffd700 1 -- 172.21.15.175:0/286175403 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6fe8005b60 con 0x7f6ff8068e10 2023-10-10T19:20:51.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.025+0000 7f6fdeffd700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe4066f60 0x7f6fe4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.025+0000 7f6fdeffd700 1 -- 172.21.15.175:0/286175403 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f6fe8084920 con 0x7f6ff8068e10 2023-10-10T19:20:51.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.025+0000 7f6ff6ffd700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe4066f60 0x7f6fe4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.026+0000 7f6ff6ffd700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe4066f60 0x7f6fe4069410 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009850 tx=0x7f6fec016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:51.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.029+0000 7f6fdeffd700 1 -- 172.21.15.175:0/286175403 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fe8051560 con 0x7f6ff8068e10 2023-10-10T19:20:51.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.265+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ff80c6990 con 0x7f6ff8068e10 2023-10-10T19:20:51.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.266+0000 7f6fdeffd700 1 -- 172.21.15.175:0/286175403 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f6fe8054b80 con 0x7f6ff8068e10 2023-10-10T19:20:51.267 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:51.267 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":269}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:51.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe4066f60 msgr2=0x7f6fe4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe4066f60 0x7f6fe4069410 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009850 tx=0x7f6fec016000 comp rx=0 tx=0).stop 2023-10-10T19:20:51.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ff8068e10 msgr2=0x7f6ff8150d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ff8068e10 0x7f6ff8150d70 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800cba0 tx=0x7f6fe8002b10 comp rx=0 tx=0).stop 2023-10-10T19:20:51.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 shutdown_connections 2023-10-10T19:20:51.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe4066f60 0x7f6fe4069410 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ff80cac80 0x7f6ff81512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ff8068a40 0x7f6ff8155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 --2- 172.21.15.175:0/286175403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ff8068e10 0x7f6ff8150d70 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 >> 172.21.15.175:0/286175403 conn(0x7f6ff806dbe0 msgr2=0x7f6ff806b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:51.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 shutdown_connections 2023-10-10T19:20:51.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.268+0000 7f6ffd5f0700 1 -- 172.21.15.175:0/286175403 wait complete. 2023-10-10T19:20:51.272 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 269 2023-10-10T19:20:51.281 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-10T19:20:51.281 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-10T19:20:51.282 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:20:51.293 INFO:tasks.ceph.mds.d:Stopped 2023-10-10T19:20:51.294 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:51.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.413+0000 7f16f8dae700 1 Processor -- start 2023-10-10T19:20:51.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.413+0000 7f16f8dae700 1 -- start start 2023-10-10T19:20:51.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.414+0000 7f16f8dae700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f16f40b1fc0 0x7f16f4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.414+0000 7f16f8dae700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f16f40b2390 0x7f16f4068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.414+0000 7f16f8dae700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f41574b0 con 0x7f16f4068bd0 2023-10-10T19:20:51.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.414+0000 7f16f8dae700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f4150770 con 0x7f16f40b1fc0 2023-10-10T19:20:51.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.414+0000 7f16f8dae700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f40afad0 con 0x7f16f40b2390 2023-10-10T19:20:51.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.414+0000 7f16f259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f16f40b1fc0 0x7f16f4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f16f40b1fc0 0x7f16f4155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37778/0 (socket says 172.21.15.175:37778) 2023-10-10T19:20:51.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f259c700 1 -- 172.21.15.175:0/643483908 learned_addr learned my addr 172.21.15.175:0/643483908 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:51.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f1d9b700 1 --2- 172.21.15.175:0/643483908 >> v2:172.21.15.191:3300/0 conn(0x7f16f40b2390 0x7f16f4068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f259c700 1 -- 172.21.15.175:0/643483908 >> v2:172.21.15.191:3300/0 conn(0x7f16f40b2390 msgr2=0x7f16f4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:51.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f259c700 1 --2- 172.21.15.175:0/643483908 >> v2:172.21.15.191:3300/0 conn(0x7f16f40b2390 0x7f16f4068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f259c700 1 -- 172.21.15.175:0/643483908 >> v1:172.21.15.191:6789/0 conn(0x7f16f4068bd0 legacy=0x7f16f40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:51.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f259c700 1 -- 172.21.15.175:0/643483908 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40af4c0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f1d9b700 1 --2- 172.21.15.175:0/643483908 >> v2:172.21.15.191:3300/0 conn(0x7f16f40b2390 0x7f16f4068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:20:51.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f259c700 1 --2- 172.21.15.175:0/643483908 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f4155fb0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f16f40c0290 tx=0x7f16e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217ff1a427a36b11 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:51.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f159a700 1 -- 172.21.15.175:0/643483908 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e800b8b0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f159a700 1 -- 172.21.15.175:0/643483908 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e800ba10 con 0x7f16f40b1fc0 2023-10-10T19:20:51.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.415+0000 7f16f159a700 1 -- 172.21.15.175:0/643483908 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e800bdb0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.416+0000 7f16f8dae700 1 -- 172.21.15.175:0/643483908 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 msgr2=0x7f16f4155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.416+0000 7f16f8dae700 1 --2- 172.21.15.175:0/643483908 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f4155fb0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f16f40c0290 tx=0x7f16e8009250 comp rx=0 tx=0).stop 2023-10-10T19:20:51.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.416+0000 7f16f8dae700 1 -- 172.21.15.175:0/643483908 shutdown_connections 2023-10-10T19:20:51.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.416+0000 7f16f8dae700 1 --2- 172.21.15.175:0/643483908 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f4155fb0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.416+0000 7f16f8dae700 1 --2- 172.21.15.175:0/643483908 >> v2:172.21.15.191:3300/0 conn(0x7f16f40b2390 0x7f16f4068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.416+0000 7f16f8dae700 1 -- 172.21.15.175:0/643483908 >> 172.21.15.175:0/643483908 conn(0x7f16f4063230 msgr2=0x7f16f4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:51.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.416+0000 7f16f8dae700 1 -- 172.21.15.175:0/643483908 shutdown_connections 2023-10-10T19:20:51.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.416+0000 7f16f8dae700 1 -- 172.21.15.175:0/643483908 wait complete. 2023-10-10T19:20:51.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.417+0000 7f16f8dae700 1 Processor -- start 2023-10-10T19:20:51.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.417+0000 7f16f8dae700 1 -- start start 2023-10-10T19:20:51.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f8dae700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16f4068bd0 0x7f16f4154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f8dae700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f41567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f8dae700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16f40b2390 0x7f16f40c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f8dae700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4165dd0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f8dae700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4166070 con 0x7f16f40b2390 2023-10-10T19:20:51.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f8dae700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4166350 con 0x7f16f4068bd0 2023-10-10T19:20:51.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f41567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f41567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37794/0 (socket says 172.21.15.175:37794) 2023-10-10T19:20:51.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 -- 172.21.15.175:0/210511780 learned_addr learned my addr 172.21.15.175:0/210511780 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:51.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f2d9d700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16f40b2390 0x7f16f40c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 -- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16f4068bd0 msgr2=0x7f16f4154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16f4068bd0 0x7f16f4154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 -- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16f40b2390 msgr2=0x7f16f40c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16f40b2390 0x7f16f40c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 -- 172.21.15.175:0/210511780 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e80094f0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f2d9d700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16f40b2390 0x7f16f40c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:20:51.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f1d9b700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f41567d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f16e400c850 tx=0x7f16e4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:51.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16e27fc700 1 -- 172.21.15.175:0/210511780 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4007650 con 0x7f16f40b1fc0 2023-10-10T19:20:51.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16e27fc700 1 -- 172.21.15.175:0/210511780 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e40077b0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f4166630 con 0x7f16f40b1fc0 2023-10-10T19:20:51.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16e27fc700 1 -- 172.21.15.175:0/210511780 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4007b50 con 0x7f16f40b1fc0 2023-10-10T19:20:51.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.418+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f4166bf0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.420+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f4067570 con 0x7f16f40b1fc0 2023-10-10T19:20:51.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.420+0000 7f16e27fc700 1 -- 172.21.15.175:0/210511780 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f16e40113f0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.420+0000 7f16e27fc700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f16d4066f30 0x7f16d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:51.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.421+0000 7f16e27fc700 1 -- 172.21.15.175:0/210511780 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f16e4084c20 con 0x7f16f40b1fc0 2023-10-10T19:20:51.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.421+0000 7f16f259c700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f16d4066f30 0x7f16d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:51.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.421+0000 7f16f259c700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f16d4066f30 0x7f16d40693e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f16f40c0290 tx=0x7f16e8016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:51.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.425+0000 7f16e27fc700 1 -- 172.21.15.175:0/210511780 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16e40517c0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.669+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16f4069060 con 0x7f16f40b1fc0 2023-10-10T19:20:51.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.670+0000 7f16e27fc700 1 -- 172.21.15.175:0/210511780 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f16e4054de0 con 0x7f16f40b1fc0 2023-10-10T19:20:51.671 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:51.671 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":269}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:51.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f16d4066f30 msgr2=0x7f16d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f16d4066f30 0x7f16d40693e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f16f40c0290 tx=0x7f16e8016000 comp rx=0 tx=0).stop 2023-10-10T19:20:51.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 msgr2=0x7f16f41567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:51.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f41567d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f16e400c850 tx=0x7f16e4008040 comp rx=0 tx=0).stop 2023-10-10T19:20:51.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 shutdown_connections 2023-10-10T19:20:51.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f16d4066f30 0x7f16d40693e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16f40b2390 0x7f16f40c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16f4068bd0 0x7f16f4154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 --2- 172.21.15.175:0/210511780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f16f40b1fc0 0x7f16f41567d0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:51.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 >> 172.21.15.175:0/210511780 conn(0x7f16f4063230 msgr2=0x7f16f4159080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:51.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 shutdown_connections 2023-10-10T19:20:51.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:51.672+0000 7f16f8dae700 1 -- 172.21.15.175:0/210511780 wait complete. 2023-10-10T19:20:51.676 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 269 2023-10-10T19:20:51.685 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:20:53.686 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:53.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.804+0000 7f412486a700 1 Processor -- start 2023-10-10T19:20:53.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.804+0000 7f412486a700 1 -- start start 2023-10-10T19:20:53.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f412486a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f41200c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:53.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f412486a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f41200c41e0 0x7f412014cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:53.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f412486a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41201533e0 con 0x7f412006b640 2023-10-10T19:20:53.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f412486a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412006e130 con 0x7f41200c41e0 2023-10-10T19:20:53.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f412486a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412006a970 con 0x7f41200c3330 2023-10-10T19:20:53.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f4115d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f41200c41e0 0x7f412014cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:53.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f411ed9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f412006b640 0x7f4120156590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42652/0 (socket says 172.21.15.175:42652) 2023-10-10T19:20:53.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f411ed9d700 1 -- 172.21.15.175:0/2523776404 learned_addr learned my addr 172.21.15.175:0/2523776404 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:20:53.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f411e59c700 1 --2- 172.21.15.175:0/2523776404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f41200c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:53.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f411dd9b700 1 -- 172.21.15.175:0/2523776404 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79142572 0 0) 0x7f41201533e0 con 0x7f412006b640 2023-10-10T19:20:53.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f411dd9b700 1 -- 172.21.15.175:0/2523776404 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c003570 con 0x7f412006b640 2023-10-10T19:20:53.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f4115d9b700 1 -- 172.21.15.175:0/2523776404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 msgr2=0x7f41200c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:53.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f4115d9b700 1 --2- 172.21.15.175:0/2523776404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f41200c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:53.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f4115d9b700 1 -- 172.21.15.175:0/2523776404 >> v1:172.21.15.175:6789/0 conn(0x7f412006b640 legacy=0x7f4120156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:53.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.805+0000 7f4115d9b700 1 -- 172.21.15.175:0/2523776404 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412006d520 con 0x7f41200c41e0 2023-10-10T19:20:53.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f4115d9b700 1 --2- 172.21.15.175:0/2523776404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f412014cda0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f41080098e0 tx=0x7f41080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd52812691f36bba server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:53.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f411dd9b700 1 -- 172.21.15.175:0/2523776404 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410800f6b0 con 0x7f41200c41e0 2023-10-10T19:20:53.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f411dd9b700 1 -- 172.21.15.175:0/2523776404 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410800f810 con 0x7f41200c41e0 2023-10-10T19:20:53.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f411dd9b700 1 -- 172.21.15.175:0/2523776404 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410800fbb0 con 0x7f41200c41e0 2023-10-10T19:20:53.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f412486a700 1 -- 172.21.15.175:0/2523776404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 msgr2=0x7f412014cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:53.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f412486a700 1 --2- 172.21.15.175:0/2523776404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f412014cda0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f41080098e0 tx=0x7f41080091a0 comp rx=0 tx=0).stop 2023-10-10T19:20:53.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f412486a700 1 -- 172.21.15.175:0/2523776404 shutdown_connections 2023-10-10T19:20:53.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f412486a700 1 --2- 172.21.15.175:0/2523776404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f41200c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:53.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f412486a700 1 --2- 172.21.15.175:0/2523776404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f412014cda0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:53.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f412486a700 1 -- 172.21.15.175:0/2523776404 >> 172.21.15.175:0/2523776404 conn(0x7f4120068280 msgr2=0x7f412006af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:53.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f412486a700 1 -- 172.21.15.175:0/2523776404 shutdown_connections 2023-10-10T19:20:53.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.806+0000 7f412486a700 1 -- 172.21.15.175:0/2523776404 wait complete. 2023-10-10T19:20:53.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.807+0000 7f412486a700 1 Processor -- start 2023-10-10T19:20:53.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f412486a700 1 -- start start 2023-10-10T19:20:53.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f412486a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f412006b640 0x7f4120162ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:53.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f412486a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f4120164da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:53.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f412486a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f41201652e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:53.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f412486a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412014f6e0 con 0x7f41200c41e0 2023-10-10T19:20:53.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f412486a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412014f250 con 0x7f412006b640 2023-10-10T19:20:53.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f412486a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41201659c0 con 0x7f41200c3330 2023-10-10T19:20:53.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f411ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f41201652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:53.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f411ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f41201652e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:37802/0 (socket says 172.21.15.175:37802) 2023-10-10T19:20:53.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f411ed9d700 1 -- 172.21.15.175:0/3443286221 learned_addr learned my addr 172.21.15.175:0/3443286221 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:53.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f4115d9b700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f4120164da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:53.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.808+0000 7f411e59c700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f412006b640 0x7f4120162ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:53.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f411ed9d700 1 -- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 msgr2=0x7f4120164da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:53.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f411ed9d700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f4120164da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:53.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f411ed9d700 1 -- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f412006b640 msgr2=0x7f4120162ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:53.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f411ed9d700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f412006b640 0x7f4120162ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:53.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f411ed9d700 1 -- 172.21.15.175:0/3443286221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108009420 con 0x7f41200c41e0 2023-10-10T19:20:53.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f4115d9b700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f4120164da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:20:53.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f411ed9d700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f41201652e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f411800cc40 tx=0x7f4118001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:53.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f4116ffd700 1 -- 172.21.15.175:0/3443286221 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800aa10 con 0x7f41200c41e0 2023-10-10T19:20:53.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f4116ffd700 1 -- 172.21.15.175:0/3443286221 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411800ab70 con 0x7f41200c41e0 2023-10-10T19:20:53.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41200b8ed0 con 0x7f41200c41e0 2023-10-10T19:20:53.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f4116ffd700 1 -- 172.21.15.175:0/3443286221 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41180070f0 con 0x7f41200c41e0 2023-10-10T19:20:53.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.809+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412014e920 con 0x7f41200c41e0 2023-10-10T19:20:53.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.810+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412014f3e0 con 0x7f41200c41e0 2023-10-10T19:20:53.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.810+0000 7f4116ffd700 1 -- 172.21.15.175:0/3443286221 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f41180074e0 con 0x7f41200c41e0 2023-10-10T19:20:53.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.810+0000 7f4116ffd700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4100066f30 0x7f41000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:53.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.811+0000 7f4116ffd700 1 -- 172.21.15.175:0/3443286221 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f41180850e0 con 0x7f41200c41e0 2023-10-10T19:20:53.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.811+0000 7f411e59c700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4100066f30 0x7f41000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:53.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.811+0000 7f411e59c700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4100066f30 0x7f41000693e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4110009d30 tx=0x7f4110008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:53.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:53.815+0000 7f4116ffd700 1 -- 172.21.15.175:0/3443286221 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4118051d10 con 0x7f41200c41e0 2023-10-10T19:20:54.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.042+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4120067570 con 0x7f41200c41e0 2023-10-10T19:20:54.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.043+0000 7f4116ffd700 1 -- 172.21.15.175:0/3443286221 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f4118055330 con 0x7f41200c41e0 2023-10-10T19:20:54.044 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:54.044 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":269}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4100066f30 msgr2=0x7f41000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4100066f30 0x7f41000693e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4110009d30 tx=0x7f4110008040 comp rx=0 tx=0).stop 2023-10-10T19:20:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 msgr2=0x7f41201652e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f41201652e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f411800cc40 tx=0x7f4118001a80 comp rx=0 tx=0).stop 2023-10-10T19:20:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 shutdown_connections 2023-10-10T19:20:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4100066f30 0x7f41000693e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41200c41e0 0x7f41201652e0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f412006b640 0x7f4120162ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:54.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 --2- 172.21.15.175:0/3443286221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41200c3330 0x7f4120164da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 >> 172.21.15.175:0/3443286221 conn(0x7f4120068280 msgr2=0x7f41200c4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 shutdown_connections 2023-10-10T19:20:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:54.045+0000 7f412486a700 1 -- 172.21.15.175:0/3443286221 wait complete. 2023-10-10T19:20:54.048 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 269 2023-10-10T19:20:54.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:20:55.040 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:20:55.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:20:56.058 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:56.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.184+0000 7f04ca89e700 1 Processor -- start 2023-10-10T19:20:56.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.184+0000 7f04ca89e700 1 -- start start 2023-10-10T19:20:56.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04ca89e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f04c40b2390 0x7f04c4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:56.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04ca89e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c41574b0 con 0x7f04c40b1fc0 2023-10-10T19:20:56.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04ca89e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c40afad0 con 0x7f04c40aff60 2023-10-10T19:20:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04ca89e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c4150770 con 0x7f04c40b2390 2023-10-10T19:20:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c3fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f04c40b1fc0 0x7f04c4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60266/0 (socket says 172.21.15.175:60266) 2023-10-10T19:20:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c3fff700 1 -- 172.21.15.175:0/1142975991 learned_addr learned my addr 172.21.15.175:0/1142975991 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:20:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261172296 0 0) 0x7f04c41574b0 con 0x7f04c40b1fc0 2023-10-10T19:20:56.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ac003570 con 0x7f04c40b1fc0 2023-10-10T19:20:56.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c37fe700 1 --2- 172.21.15.175:0/1142975991 >> v2:172.21.15.191:3300/0 conn(0x7f04c40b2390 0x7f04c4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:56.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 779462328 0 0) 0x7f04ac003570 con 0x7f04c40b1fc0 2023-10-10T19:20:56.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 >> v2:172.21.15.191:3300/0 conn(0x7f04c40b2390 msgr2=0x7f04c4155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:56.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 --2- 172.21.15.175:0/1142975991 >> v2:172.21.15.191:3300/0 conn(0x7f04c40b2390 0x7f04c4155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 >> v1:172.21.15.191:6789/0 conn(0x7f04c40aff60 legacy=0x7f04c4159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:56.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c40af4c0 con 0x7f04c40b1fc0 2023-10-10T19:20:56.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f04b4001470 con 0x7f04c40b1fc0 2023-10-10T19:20:56.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 >> v1:172.21.15.175:6789/0 conn(0x7f04c40b1fc0 legacy=0x7f04c4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:56.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac003a60 0x7f04ac003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:56.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04ac004810 0x7f04ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:56.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac0051c0 0x7f04ac00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:56.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4150770 con 0x7f04ac004810 2023-10-10T19:20:56.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00c0d0 con 0x7f04ac0051c0 2023-10-10T19:20:56.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00c290 con 0x7f04ac003a60 2023-10-10T19:20:56.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.185+0000 7f04c37fe700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04ac004810 0x7f04ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:56.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c3fff700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac003a60 0x7f04ac003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:56.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c8e3b700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac0051c0 0x7f04ac00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:56.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c37fe700 1 -- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac003a60 msgr2=0x7f04ac003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:56.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c37fe700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac003a60 0x7f04ac003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c37fe700 1 -- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac0051c0 msgr2=0x7f04ac00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:56.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c37fe700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac0051c0 0x7f04ac00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c37fe700 1 -- 172.21.15.175:0/1142975991 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac003570 con 0x7f04ac004810 2023-10-10T19:20:56.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c8e3b700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac0051c0 0x7f04ac00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:20:56.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c8e3b700 1 -- 172.21.15.175:0/1142975991 reap_dead start 2023-10-10T19:20:56.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c37fe700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04ac004810 0x7f04ac004c80 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009c00 tx=0x7f04b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f06b96728e7dce server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:56.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b800e710 con 0x7f04ac004810 2023-10-10T19:20:56.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b800e870 con 0x7f04ac004810 2023-10-10T19:20:56.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.186+0000 7f04c2ffd700 1 -- 172.21.15.175:0/1142975991 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b800ec30 con 0x7f04ac004810 2023-10-10T19:20:56.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.187+0000 7f04ca89e700 1 -- 172.21.15.175:0/1142975991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04ac004810 msgr2=0x7f04ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:56.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.187+0000 7f04ca89e700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04ac004810 0x7f04ac004c80 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009c00 tx=0x7f04b80091a0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.187+0000 7f04ca89e700 1 -- 172.21.15.175:0/1142975991 shutdown_connections 2023-10-10T19:20:56.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.187+0000 7f04ca89e700 1 --2- 172.21.15.175:0/1142975991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04ac004810 0x7f04ac004c80 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.187+0000 7f04ca89e700 1 -- 172.21.15.175:0/1142975991 >> 172.21.15.175:0/1142975991 conn(0x7f04c4063230 msgr2=0x7f04c4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:56.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.187+0000 7f04ca89e700 1 -- 172.21.15.175:0/1142975991 shutdown_connections 2023-10-10T19:20:56.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.187+0000 7f04ca89e700 1 -- 172.21.15.175:0/1142975991 wait complete. 2023-10-10T19:20:56.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.188+0000 7f04ca89e700 1 Processor -- start 2023-10-10T19:20:56.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.188+0000 7f04ca89e700 1 -- start start 2023-10-10T19:20:56.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.188+0000 7f04ca89e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac004810 0x7f04c4159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.188+0000 7f04ca89e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c415a090 0x7f04c40c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.188+0000 7f04ca89e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04c40c5fb0 0x7f04c40c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04ca89e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4165e20 con 0x7f04c415a090 2023-10-10T19:20:56.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04ca89e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c40c55e0 con 0x7f04ac004810 2023-10-10T19:20:56.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04ca89e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4166c20 con 0x7f04c40c5fb0 2023-10-10T19:20:56.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c415a090 0x7f04c40c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:56.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c3fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac004810 0x7f04c4159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:56.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c415a090 0x7f04c40c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50654/0 (socket says 172.21.15.175:50654) 2023-10-10T19:20:56.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 -- 172.21.15.175:0/2400614439 learned_addr learned my addr 172.21.15.175:0/2400614439 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:56.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c3fff700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac004810 0x7f04c4159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:51744/0 (socket says 172.21.15.175:51744) 2023-10-10T19:20:56.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c8e3b700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04c40c5fb0 0x7f04c40c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:56.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 -- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04c40c5fb0 msgr2=0x7f04c40c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:56.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04c40c5fb0 0x7f04c40c3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 -- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac004810 msgr2=0x7f04c4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:56.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac004810 0x7f04c4159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 -- 172.21.15.175:0/2400614439 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c40c6430 con 0x7f04c415a090 2023-10-10T19:20:56.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04c37fe700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c415a090 0x7f04c40c5a70 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f04b800e4b0 tx=0x7f04b801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:56.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04abfff700 1 -- 172.21.15.175:0/2400614439 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b8022070 con 0x7f04c415a090 2023-10-10T19:20:56.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04abfff700 1 -- 172.21.15.175:0/2400614439 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b800e9e0 con 0x7f04c415a090 2023-10-10T19:20:56.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04abfff700 1 -- 172.21.15.175:0/2400614439 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b80276a0 con 0x7f04c415a090 2023-10-10T19:20:56.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c4166010 con 0x7f04c415a090 2023-10-10T19:20:56.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.189+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c41665d0 con 0x7f04c415a090 2023-10-10T19:20:56.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.190+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c4067570 con 0x7f04c415a090 2023-10-10T19:20:56.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.191+0000 7f04abfff700 1 -- 172.21.15.175:0/2400614439 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f04b801c450 con 0x7f04c415a090 2023-10-10T19:20:56.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.191+0000 7f04abfff700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f04b0066f60 0x7f04b0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:56.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.191+0000 7f04abfff700 1 -- 172.21.15.175:0/2400614439 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f04b8086610 con 0x7f04c415a090 2023-10-10T19:20:56.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.191+0000 7f04c3fff700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f04b0066f60 0x7f04b0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:56.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.192+0000 7f04c3fff700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f04b0066f60 0x7f04b0069410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f04b4001620 tx=0x7f04c40af650 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:56.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.195+0000 7f04abfff700 1 -- 172.21.15.175:0/2400614439 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04b8053350 con 0x7f04c415a090 2023-10-10T19:20:56.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.418+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04c40c5720 con 0x7f04c415a090 2023-10-10T19:20:56.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.419+0000 7f04abfff700 1 -- 172.21.15.175:0/2400614439 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f04b8056930 con 0x7f04c415a090 2023-10-10T19:20:56.419 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:56.420 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":269}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:56.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f04b0066f60 msgr2=0x7f04b0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:56.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f04b0066f60 0x7f04b0069410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f04b4001620 tx=0x7f04c40af650 comp rx=0 tx=0).stop 2023-10-10T19:20:56.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c415a090 msgr2=0x7f04c40c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:56.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c415a090 0x7f04c40c5a70 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f04b800e4b0 tx=0x7f04b801a000 comp rx=0 tx=0).stop 2023-10-10T19:20:56.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 shutdown_connections 2023-10-10T19:20:56.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f04b0066f60 0x7f04b0069410 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04c40c5fb0 0x7f04c40c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac004810 0x7f04c4159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 --2- 172.21.15.175:0/2400614439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c415a090 0x7f04c40c5a70 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:56.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 >> 172.21.15.175:0/2400614439 conn(0x7f04c4063230 msgr2=0x7f04c40b0360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:56.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 shutdown_connections 2023-10-10T19:20:56.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:56.421+0000 7f04ca89e700 1 -- 172.21.15.175:0/2400614439 wait complete. 2023-10-10T19:20:56.425 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 269 2023-10-10T19:20:56.433 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:20:58.434 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:20:58.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.561+0000 7fe1c5fc8700 1 Processor -- start 2023-10-10T19:20:58.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.561+0000 7fe1c5fc8700 1 -- start start 2023-10-10T19:20:58.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.562+0000 7fe1c5fc8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe1c00b1fc0 0x7fe1c0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:58.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.562+0000 7fe1c5fc8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00aff60 0x7fe1c00b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:58.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.562+0000 7fe1c5fc8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c01574b0 con 0x7fe1c00b2390 2023-10-10T19:20:58.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.562+0000 7fe1c5fc8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c0150770 con 0x7fe1c00b1fc0 2023-10-10T19:20:58.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.562+0000 7fe1c5fc8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00afad0 con 0x7fe1c00aff60 2023-10-10T19:20:58.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00aff60 0x7fe1c00b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:58.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bf7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe1c00b1fc0 0x7fe1c0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:58.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bf7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe1c00b1fc0 0x7fe1c0155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:51746/0 (socket says 172.21.15.175:51746) 2023-10-10T19:20:58.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00aff60 0x7fe1c00b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:60402/0 (socket says 172.21.15.175:60402) 2023-10-10T19:20:58.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1beffd700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe1c00b2390 0x7fe1c0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:33976/0 (socket says 172.21.15.175:33976) 2023-10-10T19:20:58.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bf7fe700 1 -- 172.21.15.175:0/2787568877 learned_addr learned my addr 172.21.15.175:0/2787568877 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:58.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bf7fe700 1 -- 172.21.15.175:0/2787568877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00aff60 msgr2=0x7fe1c00b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:20:58.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bf7fe700 1 --2- 172.21.15.175:0/2787568877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00aff60 0x7fe1c00b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bf7fe700 1 -- 172.21.15.175:0/2787568877 >> v1:172.21.15.191:6789/0 conn(0x7fe1c00b2390 legacy=0x7fe1c0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:58.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bf7fe700 1 -- 172.21.15.175:0/2787568877 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c00af4c0 con 0x7fe1c00b1fc0 2023-10-10T19:20:58.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bffff700 1 --2- 172.21.15.175:0/2787568877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00aff60 0x7fe1c00b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:20:58.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1be7fc700 1 -- 172.21.15.175:0/2787568877 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789725750 0 0) 0x7fe1c01574b0 con 0x7fe1c00b2390 2023-10-10T19:20:58.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1bf7fe700 1 --2- 172.21.15.175:0/2787568877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b1fc0 0x7fe1c0155fb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009f20 tx=0x7fe1b0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9ac7d7bb016299d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:58.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1be7fc700 1 -- 172.21.15.175:0/2787568877 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b000e070 con 0x7fe1c00b1fc0 2023-10-10T19:20:58.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1be7fc700 1 -- 172.21.15.175:0/2787568877 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b000ba60 con 0x7fe1c00b1fc0 2023-10-10T19:20:58.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.563+0000 7fe1be7fc700 1 -- 172.21.15.175:0/2787568877 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00092e0 con 0x7fe1c00b1fc0 2023-10-10T19:20:58.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.564+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2787568877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b1fc0 msgr2=0x7fe1c0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:58.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.564+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2787568877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b1fc0 0x7fe1c0155fb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009f20 tx=0x7fe1b0009840 comp rx=0 tx=0).stop 2023-10-10T19:20:58.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.564+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2787568877 shutdown_connections 2023-10-10T19:20:58.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.564+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2787568877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00aff60 0x7fe1c00b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.564+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2787568877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b1fc0 0x7fe1c0155fb0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.564+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2787568877 >> 172.21.15.175:0/2787568877 conn(0x7fe1c0063230 msgr2=0x7fe1c0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:58.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.564+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2787568877 shutdown_connections 2023-10-10T19:20:58.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.564+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2787568877 wait complete. 2023-10-10T19:20:58.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.565+0000 7fe1c5fc8700 1 Processor -- start 2023-10-10T19:20:58.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.565+0000 7fe1c5fc8700 1 -- start start 2023-10-10T19:20:58.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1c5fc8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c00aff60 0x7fe1c0159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:58.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1c5fc8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00b1fc0 0x7fe1c00c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:58.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1c5fc8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b2390 0x7fe1c00c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:58.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1c5fc8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00c4120 con 0x7fe1c00aff60 2023-10-10T19:20:58.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1c5fc8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00c3b60 con 0x7fe1c00b2390 2023-10-10T19:20:58.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1c5fc8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00c38e0 con 0x7fe1c00b1fc0 2023-10-10T19:20:58.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bf7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c00aff60 0x7fe1c0159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:58.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bf7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c00aff60 0x7fe1c0159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50666/0 (socket says 172.21.15.175:50666) 2023-10-10T19:20:58.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bf7fe700 1 -- 172.21.15.175:0/2388757985 learned_addr learned my addr 172.21.15.175:0/2388757985 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:20:58.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bffff700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b2390 0x7fe1c00c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:58.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1beffd700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00b1fc0 0x7fe1c00c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:58.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bf7fe700 1 -- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00b1fc0 msgr2=0x7fe1c00c45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:58.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bf7fe700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00b1fc0 0x7fe1c00c45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bf7fe700 1 -- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b2390 msgr2=0x7fe1c00c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:58.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bf7fe700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b2390 0x7fe1c00c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.566+0000 7fe1bf7fe700 1 -- 172.21.15.175:0/2388757985 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b8009420 con 0x7fe1c00aff60 2023-10-10T19:20:58.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.567+0000 7fe1bf7fe700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c00aff60 0x7fe1c0159b50 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fe1c006fd90 tx=0x7fe1b0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:58.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.567+0000 7fe1a77fe700 1 -- 172.21.15.175:0/2388757985 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00089b0 con 0x7fe1c00aff60 2023-10-10T19:20:58.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.567+0000 7fe1a77fe700 1 -- 172.21.15.175:0/2388757985 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b0008b30 con 0x7fe1c00aff60 2023-10-10T19:20:58.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.567+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b0009ae0 con 0x7fe1c00aff60 2023-10-10T19:20:58.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.567+0000 7fe1a77fe700 1 -- 172.21.15.175:0/2388757985 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0026460 con 0x7fe1c00aff60 2023-10-10T19:20:58.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.567+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c00c63d0 con 0x7fe1c00aff60 2023-10-10T19:20:58.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.568+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c0067570 con 0x7fe1c00aff60 2023-10-10T19:20:58.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.568+0000 7fe1a77fe700 1 -- 172.21.15.175:0/2388757985 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe1b0003040 con 0x7fe1c00aff60 2023-10-10T19:20:58.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.568+0000 7fe1a77fe700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1ac066f30 0x7fe1ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:20:58.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.569+0000 7fe1a77fe700 1 -- 172.21.15.175:0/2388757985 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fe1b0086860 con 0x7fe1c00aff60 2023-10-10T19:20:58.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.569+0000 7fe1beffd700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1ac066f30 0x7fe1ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:20:58.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.569+0000 7fe1beffd700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1ac066f30 0x7fe1ac0693e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00cd890 tx=0x7fe1b4007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:20:58.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.573+0000 7fe1a77fe700 1 -- 172.21.15.175:0/2388757985 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1b0053560 con 0x7fe1c00aff60 2023-10-10T19:20:58.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.796+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1c0068da0 con 0x7fe1c00aff60 2023-10-10T19:20:58.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.797+0000 7fe1a77fe700 1 -- 172.21.15.175:0/2388757985 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+6181 (secure 0 0 0) 0x7fe1b0056b80 con 0x7fe1c00aff60 2023-10-10T19:20:58.798 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:20:58.798 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":269}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:20:58.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.798+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1ac066f30 msgr2=0x7fe1ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:58.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.798+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1ac066f30 0x7fe1ac0693e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00cd890 tx=0x7fe1b4007ef0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.798+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c00aff60 msgr2=0x7fe1c0159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:20:58.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.798+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c00aff60 0x7fe1c0159b50 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fe1c006fd90 tx=0x7fe1b0018000 comp rx=0 tx=0).stop 2023-10-10T19:20:58.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.799+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 shutdown_connections 2023-10-10T19:20:58.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.799+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe1ac066f30 0x7fe1ac0693e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.799+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1c00b2390 0x7fe1c00c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.799+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c00aff60 0x7fe1c0159b50 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.799+0000 7fe1c5fc8700 1 --2- 172.21.15.175:0/2388757985 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1c00b1fc0 0x7fe1c00c45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:20:58.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.799+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 >> 172.21.15.175:0/2388757985 conn(0x7fe1c0063230 msgr2=0x7fe1c014fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:20:58.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.799+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 shutdown_connections 2023-10-10T19:20:58.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:20:58.799+0000 7fe1c5fc8700 1 -- 172.21.15.175:0/2388757985 wait complete. 2023-10-10T19:20:58.802 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 269 2023-10-10T19:20:58.811 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:21:00.812 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:00.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.939+0000 7fd181349700 1 Processor -- start 2023-10-10T19:21:00.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.939+0000 7fd181349700 1 -- start start 2023-10-10T19:21:00.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.939+0000 7fd181349700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:00.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.939+0000 7fd181349700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd17c0aff60 0x7fd17c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:00.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.939+0000 7fd181349700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c1574b0 con 0x7fd17c0b1fc0 2023-10-10T19:21:00.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.939+0000 7fd181349700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c150770 con 0x7fd17c0aff60 2023-10-10T19:21:00.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.939+0000 7fd181349700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0afad0 con 0x7fd17c0b2390 2023-10-10T19:21:00.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17a7fc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:00.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17b7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd17c0aff60 0x7fd17c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:00.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17affd700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd17c0b1fc0 0x7fd17c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:33978/0 (socket says 172.21.15.175:33978) 2023-10-10T19:21:00.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17affd700 1 -- 172.21.15.175:0/3985278396 learned_addr learned my addr 172.21.15.175:0/3985278396 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:00.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17b7fe700 1 -- 172.21.15.175:0/3985278396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 msgr2=0x7fd17c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:00.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17b7fe700 1 --2- 172.21.15.175:0/3985278396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:00.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17b7fe700 1 -- 172.21.15.175:0/3985278396 >> v1:172.21.15.191:6789/0 conn(0x7fd17c0b1fc0 legacy=0x7fd17c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:00.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17b7fe700 1 -- 172.21.15.175:0/3985278396 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c0af4c0 con 0x7fd17c0aff60 2023-10-10T19:21:00.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd179ffb700 1 -- 172.21.15.175:0/3985278396 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188918607 0 0) 0x7fd17c1574b0 con 0x7fd17c0b1fc0 2023-10-10T19:21:00.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17a7fc700 1 --2- 172.21.15.175:0/3985278396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:21:00.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd17b7fe700 1 --2- 172.21.15.175:0/3985278396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0aff60 0x7fd17c0b03e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd174009910 tx=0x7fd1740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb5684954d08225 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:00.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd179ffb700 1 -- 172.21.15.175:0/3985278396 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17400b6d0 con 0x7fd17c0aff60 2023-10-10T19:21:00.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd179ffb700 1 -- 172.21.15.175:0/3985278396 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17400b830 con 0x7fd17c0aff60 2023-10-10T19:21:00.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.940+0000 7fd179ffb700 1 -- 172.21.15.175:0/3985278396 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17400bbf0 con 0x7fd17c0aff60 2023-10-10T19:21:00.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.941+0000 7fd181349700 1 -- 172.21.15.175:0/3985278396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0aff60 msgr2=0x7fd17c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:00.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.941+0000 7fd181349700 1 --2- 172.21.15.175:0/3985278396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0aff60 0x7fd17c0b03e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd174009910 tx=0x7fd1740091a0 comp rx=0 tx=0).stop 2023-10-10T19:21:00.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.941+0000 7fd181349700 1 -- 172.21.15.175:0/3985278396 shutdown_connections 2023-10-10T19:21:00.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.941+0000 7fd181349700 1 --2- 172.21.15.175:0/3985278396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0aff60 0x7fd17c0b03e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:00.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.941+0000 7fd181349700 1 --2- 172.21.15.175:0/3985278396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:00.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.941+0000 7fd181349700 1 -- 172.21.15.175:0/3985278396 >> 172.21.15.175:0/3985278396 conn(0x7fd17c063230 msgr2=0x7fd17c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:00.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.941+0000 7fd181349700 1 -- 172.21.15.175:0/3985278396 shutdown_connections 2023-10-10T19:21:00.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.941+0000 7fd181349700 1 -- 172.21.15.175:0/3985278396 wait complete. 2023-10-10T19:21:00.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.942+0000 7fd181349700 1 Processor -- start 2023-10-10T19:21:00.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.942+0000 7fd181349700 1 -- start start 2023-10-10T19:21:00.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd181349700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c0aff60 0x7fd17c155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:00.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd181349700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0b1fc0 0x7fd17c1556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:00.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd181349700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c0cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:00.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd181349700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c157180 con 0x7fd17c0aff60 2023-10-10T19:21:00.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd181349700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0c5a60 con 0x7fd17c0b1fc0 2023-10-10T19:21:00.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd181349700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c157760 con 0x7fd17c0b2390 2023-10-10T19:21:00.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c0aff60 0x7fd17c155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:00.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c0aff60 0x7fd17c155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50680/0 (socket says 172.21.15.175:50680) 2023-10-10T19:21:00.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17affd700 1 -- 172.21.15.175:0/2961855316 learned_addr learned my addr 172.21.15.175:0/2961855316 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:00.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17b7fe700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c0cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:00.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17a7fc700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0b1fc0 0x7fd17c1556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:00.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17affd700 1 -- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 msgr2=0x7fd17c0cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:00.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17affd700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c0cbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:00.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17affd700 1 -- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0b1fc0 msgr2=0x7fd17c1556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:00.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17affd700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0b1fc0 0x7fd17c1556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:00.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.943+0000 7fd17affd700 1 -- 172.21.15.175:0/2961855316 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164009420 con 0x7fd17c0aff60 2023-10-10T19:21:00.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.944+0000 7fd17b7fe700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c0cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:21:00.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.944+0000 7fd17affd700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c0aff60 0x7fd17c155160 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00cdb0 tx=0x7fd16c001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:00.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.944+0000 7fd172ffd700 1 -- 172.21.15.175:0/2961855316 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00aad0 con 0x7fd17c0aff60 2023-10-10T19:21:00.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.944+0000 7fd172ffd700 1 -- 172.21.15.175:0/2961855316 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c00ac30 con 0x7fd17c0aff60 2023-10-10T19:21:00.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.944+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd174009420 con 0x7fd17c0aff60 2023-10-10T19:21:00.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.944+0000 7fd172ffd700 1 -- 172.21.15.175:0/2961855316 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0070f0 con 0x7fd17c0aff60 2023-10-10T19:21:00.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.944+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c157410 con 0x7fd17c0aff60 2023-10-10T19:21:00.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.945+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c067570 con 0x7fd17c0aff60 2023-10-10T19:21:00.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.945+0000 7fd172ffd700 1 -- 172.21.15.175:0/2961855316 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd16c0074e0 con 0x7fd17c0aff60 2023-10-10T19:21:00.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.945+0000 7fd172ffd700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f60 0x7fd15c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:00.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.946+0000 7fd172ffd700 1 -- 172.21.15.175:0/2961855316 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fd16c084dd0 con 0x7fd17c0aff60 2023-10-10T19:21:00.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.946+0000 7fd17a7fc700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f60 0x7fd15c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:00.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.946+0000 7fd17a7fc700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f60 0x7fd15c069410 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0c9c40 tx=0x7fd1640091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:00.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:00.950+0000 7fd172ffd700 1 -- 172.21.15.175:0/2961855316 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd16c051a90 con 0x7fd17c0aff60 2023-10-10T19:21:01.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.175+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd17c1573b0 con 0x7fd17c0aff60 2023-10-10T19:21:01.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.176+0000 7fd172ffd700 1 -- 172.21.15.175:0/2961855316 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+6181 (secure 0 0 0) 0x7fd16c086090 con 0x7fd17c0aff60 2023-10-10T19:21:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:01.177 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":269}],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:19:57.863786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":41,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:01.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f60 msgr2=0x7fd15c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:01.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f60 0x7fd15c069410 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0c9c40 tx=0x7fd1640091a0 comp rx=0 tx=0).stop 2023-10-10T19:21:01.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c0aff60 msgr2=0x7fd17c155160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:01.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c0aff60 0x7fd17c155160 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00cdb0 tx=0x7fd16c001ae0 comp rx=0 tx=0).stop 2023-10-10T19:21:01.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 shutdown_connections 2023-10-10T19:21:01.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f60 0x7fd15c069410 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:01.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0b2390 0x7fd17c0cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:01.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c0aff60 0x7fd17c155160 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:01.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 --2- 172.21.15.175:0/2961855316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c0b1fc0 0x7fd17c1556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:01.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 >> 172.21.15.175:0/2961855316 conn(0x7fd17c063230 msgr2=0x7fd17c1508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:01.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 shutdown_connections 2023-10-10T19:21:01.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:01.178+0000 7fd181349700 1 -- 172.21.15.175:0/2961855316 wait complete. 2023-10-10T19:21:01.182 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 269 2023-10-10T19:21:01.191 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:21:03.191 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:03.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.317+0000 7fbd20b41700 1 Processor -- start 2023-10-10T19:21:03.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.318+0000 7fbd20b41700 1 -- start start 2023-10-10T19:21:03.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.318+0000 7fbd20b41700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbd1c155fb0 0x7fbd1c159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:03.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.318+0000 7fbd20b41700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1c1574b0 con 0x7fbd1c0b2390 2023-10-10T19:21:03.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.318+0000 7fbd20b41700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1c0afad0 con 0x7fbd1c0b1fc0 2023-10-10T19:21:03.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.318+0000 7fbd20b41700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd1c150770 con 0x7fbd1c155fb0 2023-10-10T19:21:03.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd19d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbd1c0b2390 0x7fbd1c0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60280/0 (socket says 172.21.15.175:60280) 2023-10-10T19:21:03.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd19d9b700 1 -- 172.21.15.175:0/1244374494 learned_addr learned my addr 172.21.15.175:0/1244374494 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:03.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1ad9d700 1 --2- 172.21.15.175:0/1244374494 >> v2:172.21.15.191:3300/0 conn(0x7fbd1c155fb0 0x7fbd1c159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:03.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1959a700 1 -- 172.21.15.175:0/1244374494 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316557404 0 0) 0x7fbd1c1574b0 con 0x7fbd1c0b2390 2023-10-10T19:21:03.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1959a700 1 -- 172.21.15.175:0/1244374494 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04003570 con 0x7fbd1c0b2390 2023-10-10T19:21:03.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1ad9d700 1 -- 172.21.15.175:0/1244374494 >> v1:172.21.15.191:6789/0 conn(0x7fbd1c0b1fc0 legacy=0x7fbd1c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:03.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1959a700 1 -- 172.21.15.175:0/1244374494 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290866833 0 0) 0x7fbd1c0afad0 con 0x7fbd1c0b1fc0 2023-10-10T19:21:03.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1ad9d700 1 -- 172.21.15.175:0/1244374494 >> v1:172.21.15.175:6789/0 conn(0x7fbd1c0b2390 legacy=0x7fbd1c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:03.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1ad9d700 1 -- 172.21.15.175:0/1244374494 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c0af4c0 con 0x7fbd1c155fb0 2023-10-10T19:21:03.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1ad9d700 1 --2- 172.21.15.175:0/1244374494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 0x7fbd1c159920 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbd140098e0 tx=0x7fbd140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97a1ad799083b8b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:03.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1959a700 1 -- 172.21.15.175:0/1244374494 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1400b6d0 con 0x7fbd1c155fb0 2023-10-10T19:21:03.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1959a700 1 -- 172.21.15.175:0/1244374494 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1400b830 con 0x7fbd1c155fb0 2023-10-10T19:21:03.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.319+0000 7fbd1959a700 1 -- 172.21.15.175:0/1244374494 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1400bbd0 con 0x7fbd1c155fb0 2023-10-10T19:21:03.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.320+0000 7fbd20b41700 1 -- 172.21.15.175:0/1244374494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 msgr2=0x7fbd1c159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:03.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.320+0000 7fbd20b41700 1 --2- 172.21.15.175:0/1244374494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 0x7fbd1c159920 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbd140098e0 tx=0x7fbd140091a0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.320+0000 7fbd20b41700 1 -- 172.21.15.175:0/1244374494 shutdown_connections 2023-10-10T19:21:03.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.320+0000 7fbd20b41700 1 --2- 172.21.15.175:0/1244374494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 0x7fbd1c159920 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.320+0000 7fbd20b41700 1 -- 172.21.15.175:0/1244374494 >> 172.21.15.175:0/1244374494 conn(0x7fbd1c063230 msgr2=0x7fbd1c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:03.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.320+0000 7fbd20b41700 1 -- 172.21.15.175:0/1244374494 shutdown_connections 2023-10-10T19:21:03.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.320+0000 7fbd20b41700 1 -- 172.21.15.175:0/1244374494 wait complete. 2023-10-10T19:21:03.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.321+0000 7fbd20b41700 1 Processor -- start 2023-10-10T19:21:03.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.321+0000 7fbd20b41700 1 -- start start 2023-10-10T19:21:03.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd20b41700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd1c0b1fc0 0x7fbd1c157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:03.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd20b41700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd1c0b2390 0x7fbd1c0ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:03.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd20b41700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 0x7fbd1c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:03.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd20b41700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c1594e0 con 0x7fbd1c0b2390 2023-10-10T19:21:03.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd20b41700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c166570 con 0x7fbd1c155fb0 2023-10-10T19:21:03.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd20b41700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c159050 con 0x7fbd1c0b1fc0 2023-10-10T19:21:03.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd1c0b2390 0x7fbd1c0ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:03.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd1c0b2390 0x7fbd1c0ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:50696/0 (socket says 172.21.15.175:50696) 2023-10-10T19:21:03.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 -- 172.21.15.175:0/2439695204 learned_addr learned my addr 172.21.15.175:0/2439695204 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:03.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd1ad9d700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 0x7fbd1c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:03.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd1a59c700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd1c0b1fc0 0x7fbd1c157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:03.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 -- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd1c0b1fc0 msgr2=0x7fbd1c157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:03.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd1c0b1fc0 0x7fbd1c157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 -- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 msgr2=0x7fbd1c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:03.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 0x7fbd1c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 -- 172.21.15.175:0/2439695204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd14009420 con 0x7fbd1c0b2390 2023-10-10T19:21:03.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd1ad9d700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 0x7fbd1c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:21:03.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd19d9b700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd1c0b2390 0x7fbd1c0ccc00 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000c7c0 tx=0x7fbd10007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:03.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd0a7fc700 1 -- 172.21.15.175:0/2439695204 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10005b30 con 0x7fbd1c0b2390 2023-10-10T19:21:03.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd0a7fc700 1 -- 172.21.15.175:0/2439695204 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd10005c90 con 0x7fbd1c0b2390 2023-10-10T19:21:03.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.322+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1c165ff0 con 0x7fbd1c0b2390 2023-10-10T19:21:03.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.323+0000 7fbd0a7fc700 1 -- 172.21.15.175:0/2439695204 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10007320 con 0x7fbd1c0b2390 2023-10-10T19:21:03.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.323+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1c166290 con 0x7fbd1c0b2390 2023-10-10T19:21:03.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.323+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd1c063e80 con 0x7fbd1c0b2390 2023-10-10T19:21:03.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.324+0000 7fbd0a7fc700 1 -- 172.21.15.175:0/2439695204 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbd10007690 con 0x7fbd1c0b2390 2023-10-10T19:21:03.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.324+0000 7fbd0a7fc700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbcfc066f30 0x7fbcfc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:03.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.324+0000 7fbd0a7fc700 1 -- 172.21.15.175:0/2439695204 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 7283+0+0 (secure 0 0 0) 0x7fbd10083fd0 con 0x7fbd1c0b2390 2023-10-10T19:21:03.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.324+0000 7fbd1a59c700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbcfc066f30 0x7fbcfc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:03.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.325+0000 7fbd1a59c700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbcfc066f30 0x7fbcfc0693e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c001560 tx=0x7fbd0c0016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:03.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.328+0000 7fbd0a7fc700 1 -- 172.21.15.175:0/2439695204 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbd10050c30 con 0x7fbd1c0b2390 2023-10-10T19:21:03.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.557+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd1c068470 con 0x7fbd1c0b2390 2023-10-10T19:21:03.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.558+0000 7fbd0a7fc700 1 -- 172.21.15.175:0/2439695204 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 270 v270) v1 ==== 95+0+6181 (secure 0 0 0) 0x7fbd10017020 con 0x7fbd1c0b2390 2023-10-10T19:21:03.558 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:03.559 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":270,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":269}],"filesystems":[{"mdsmap":{"epoch":270,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:02.015153+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6511,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6511":{"gid":6511,"name":"d","rank":0,"incarnation":204,"state":"up:active","state_seq":19,"addr":"172.21.15.191:6837/1851345295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1851345295},{"type":"v1","addr":"172.21.15.191:6837","nonce":1851345295}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":89,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:03.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbcfc066f30 msgr2=0x7fbcfc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:03.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbcfc066f30 0x7fbcfc0693e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c001560 tx=0x7fbd0c0016f0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd1c0b2390 msgr2=0x7fbd1c0ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:03.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd1c0b2390 0x7fbd1c0ccc00 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000c7c0 tx=0x7fbd10007df0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 shutdown_connections 2023-10-10T19:21:03.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbcfc066f30 0x7fbcfc0693e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c155fb0 0x7fbd1c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd1c0b1fc0 0x7fbd1c157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 --2- 172.21.15.175:0/2439695204 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd1c0b2390 0x7fbd1c0ccc00 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:03.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 >> 172.21.15.175:0/2439695204 conn(0x7fbd1c063230 msgr2=0x7fbd1c0c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:03.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 shutdown_connections 2023-10-10T19:21:03.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:03.560+0000 7fbd20b41700 1 -- 172.21.15.175:0/2439695204 wait complete. 2023-10-10T19:21:03.563 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 270 2023-10-10T19:21:03.572 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:21:05.574 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:05.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.699+0000 7ff4d8870700 1 Processor -- start 2023-10-10T19:21:05.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.699+0000 7ff4d8870700 1 -- start start 2023-10-10T19:21:05.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d8870700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff4d414fbf0 0x7ff4d4159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:05.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d8870700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4d414ffc0 0x7ff4d4063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:05.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d8870700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d4156800 con 0x7ff4d40681e0 2023-10-10T19:21:05.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d8870700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d40c59e0 con 0x7ff4d414fbf0 2023-10-10T19:21:05.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d8870700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d415a2b0 con 0x7ff4d414ffc0 2023-10-10T19:21:05.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff4d414fbf0 0x7ff4d4159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:05.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff4d414fbf0 0x7ff4d4159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38458/0 (socket says 172.21.15.175:38458) 2023-10-10T19:21:05.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d259c700 1 -- 172.21.15.175:0/1591727883 learned_addr learned my addr 172.21.15.175:0/1591727883 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:05.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d1d9b700 1 --2- 172.21.15.175:0/1591727883 >> v2:172.21.15.191:3300/0 conn(0x7ff4d414ffc0 0x7ff4d4063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:05.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d259c700 1 -- 172.21.15.175:0/1591727883 >> v2:172.21.15.191:3300/0 conn(0x7ff4d414ffc0 msgr2=0x7ff4d4063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:05.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d259c700 1 --2- 172.21.15.175:0/1591727883 >> v2:172.21.15.191:3300/0 conn(0x7ff4d414ffc0 0x7ff4d4063230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d259c700 1 -- 172.21.15.175:0/1591727883 >> v1:172.21.15.191:6789/0 conn(0x7ff4d40681e0 legacy=0x7ff4d40c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:05.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d259c700 1 -- 172.21.15.175:0/1591727883 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d4063e30 con 0x7ff4d414fbf0 2023-10-10T19:21:05.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d259c700 1 --2- 172.21.15.175:0/1591727883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414fbf0 0x7ff4d4159d70 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4089b70 tx=0x7ff4c4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291cdbd226d4ae2c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:05.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d159a700 1 -- 172.21.15.175:0/1591727883 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4005c10 con 0x7ff4d414fbf0 2023-10-10T19:21:05.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d159a700 1 -- 172.21.15.175:0/1591727883 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c4005d70 con 0x7ff4d414fbf0 2023-10-10T19:21:05.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.700+0000 7ff4d159a700 1 -- 172.21.15.175:0/1591727883 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c400b800 con 0x7ff4d414fbf0 2023-10-10T19:21:05.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.701+0000 7ff4d8870700 1 -- 172.21.15.175:0/1591727883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414fbf0 msgr2=0x7ff4d4159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:05.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.701+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1591727883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414fbf0 0x7ff4d4159d70 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4089b70 tx=0x7ff4c4009220 comp rx=0 tx=0).stop 2023-10-10T19:21:05.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.701+0000 7ff4d8870700 1 -- 172.21.15.175:0/1591727883 shutdown_connections 2023-10-10T19:21:05.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.701+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1591727883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414fbf0 0x7ff4d4159d70 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.701+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1591727883 >> v2:172.21.15.191:3300/0 conn(0x7ff4d414ffc0 0x7ff4d4063230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.701+0000 7ff4d8870700 1 -- 172.21.15.175:0/1591727883 >> 172.21.15.175:0/1591727883 conn(0x7ff4d40cc760 msgr2=0x7ff4d40ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:05.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.701+0000 7ff4d8870700 1 -- 172.21.15.175:0/1591727883 shutdown_connections 2023-10-10T19:21:05.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.701+0000 7ff4d8870700 1 -- 172.21.15.175:0/1591727883 wait complete. 2023-10-10T19:21:05.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.702+0000 7ff4d8870700 1 Processor -- start 2023-10-10T19:21:05.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.702+0000 7ff4d8870700 1 -- start start 2023-10-10T19:21:05.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d8870700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4d40681e0 0x7ff4d41640b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:05.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d8870700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4d414fbf0 0x7ff4d40b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:05.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d8870700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414ffc0 0x7ff4d40b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:05.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d8870700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4169510 con 0x7ff4d414ffc0 2023-10-10T19:21:05.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d8870700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d41681d0 con 0x7ff4d40681e0 2023-10-10T19:21:05.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d8870700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4168470 con 0x7ff4d414fbf0 2023-10-10T19:21:05.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414ffc0 0x7ff4d40b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:05.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414ffc0 0x7ff4d40b13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38472/0 (socket says 172.21.15.175:38472) 2023-10-10T19:21:05.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 -- 172.21.15.175:0/1174573315 learned_addr learned my addr 172.21.15.175:0/1174573315 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:05.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d259c700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4d40681e0 0x7ff4d41640b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:05.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d1d9b700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4d414fbf0 0x7ff4d40b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:05.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 -- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4d414fbf0 msgr2=0x7ff4d40b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:05.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4d414fbf0 0x7ff4d40b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 -- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4d40681e0 msgr2=0x7ff4d41640b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:05.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4d40681e0 0x7ff4d41640b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 -- 172.21.15.175:0/1174573315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4009480 con 0x7ff4d414ffc0 2023-10-10T19:21:05.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.703+0000 7ff4d2d9d700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414ffc0 0x7ff4d40b13f0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00c930 tx=0x7ff4cc0026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:05.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.704+0000 7ff4c27fc700 1 -- 172.21.15.175:0/1174573315 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc012630 con 0x7ff4d414ffc0 2023-10-10T19:21:05.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.704+0000 7ff4c27fc700 1 -- 172.21.15.175:0/1174573315 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc0127d0 con 0x7ff4d414ffc0 2023-10-10T19:21:05.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.704+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d4168750 con 0x7ff4d414ffc0 2023-10-10T19:21:05.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.704+0000 7ff4c27fc700 1 -- 172.21.15.175:0/1174573315 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc012b90 con 0x7ff4d414ffc0 2023-10-10T19:21:05.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.704+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d4168d10 con 0x7ff4d414ffc0 2023-10-10T19:21:05.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.705+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d4067570 con 0x7ff4d414ffc0 2023-10-10T19:21:05.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.705+0000 7ff4c27fc700 1 -- 172.21.15.175:0/1174573315 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff4cc01a3b0 con 0x7ff4d414ffc0 2023-10-10T19:21:05.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.705+0000 7ff4c27fc700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff4b4066f30 0x7ff4b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:05.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.705+0000 7ff4c27fc700 1 -- 172.21.15.175:0/1174573315 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7ff4cc0847d0 con 0x7ff4d414ffc0 2023-10-10T19:21:05.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.706+0000 7ff4d259c700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff4b4066f30 0x7ff4b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:05.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.706+0000 7ff4d259c700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff4b4066f30 0x7ff4b40693e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4007fd0 tx=0x7ff4c4015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:05.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.709+0000 7ff4c27fc700 1 -- 172.21.15.175:0/1174573315 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff4cc054e50 con 0x7ff4d414ffc0 2023-10-10T19:21:05.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.934+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4d408a4b0 con 0x7ff4d414ffc0 2023-10-10T19:21:05.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.935+0000 7ff4c27fc700 1 -- 172.21.15.175:0/1174573315 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 271 v271) v1 ==== 95+0+5419 (secure 0 0 0) 0x7ff4cc0549e0 con 0x7ff4d414ffc0 2023-10-10T19:21:05.937 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:05.937 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":271,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":271,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:03.818392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":89,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:05.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff4b4066f30 msgr2=0x7ff4b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:05.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff4b4066f30 0x7ff4b40693e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4007fd0 tx=0x7ff4c4015000 comp rx=0 tx=0).stop 2023-10-10T19:21:05.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414ffc0 msgr2=0x7ff4d40b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:05.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414ffc0 0x7ff4d40b13f0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00c930 tx=0x7ff4cc0026e0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 shutdown_connections 2023-10-10T19:21:05.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff4b4066f30 0x7ff4b40693e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff4d414ffc0 0x7ff4d40b13f0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4d40681e0 0x7ff4d41640b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 --2- 172.21.15.175:0/1174573315 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4d414fbf0 0x7ff4d40b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:05.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 >> 172.21.15.175:0/1174573315 conn(0x7ff4d40cc760 msgr2=0x7ff4d4159a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:05.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 shutdown_connections 2023-10-10T19:21:05.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:05.937+0000 7ff4d8870700 1 -- 172.21.15.175:0/1174573315 wait complete. 2023-10-10T19:21:05.942 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 271 2023-10-10T19:21:05.951 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-10T19:21:05.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:21:05.951 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:06.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.068+0000 7fc5c97e5700 1 Processor -- start 2023-10-10T19:21:06.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.068+0000 7fc5c97e5700 1 -- start start 2023-10-10T19:21:06.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.069+0000 7fc5c97e5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5c40c9390 0x7fc5c4159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:06.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.069+0000 7fc5c97e5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c40c5e40 con 0x7fc5c40c35e0 2023-10-10T19:21:06.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.069+0000 7fc5c97e5700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c40c9880 con 0x7fc5c40c3210 2023-10-10T19:21:06.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.069+0000 7fc5c97e5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c4c50 con 0x7fc5c40c9390 2023-10-10T19:21:06.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.069+0000 7fc5c27fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc5c40c35e0 0x7fc5c40c62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38750/0 (socket says 172.21.15.175:38750) 2023-10-10T19:21:06.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.069+0000 7fc5c27fc700 1 -- 172.21.15.175:0/2457606852 learned_addr learned my addr 172.21.15.175:0/2457606852 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:06.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.069+0000 7fc5c37fe700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5c40c9390 0x7fc5c4159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:06.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725664281 0 0) 0x7fc5c40c5e40 con 0x7fc5c40c35e0 2023-10-10T19:21:06.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0003570 con 0x7fc5c40c35e0 2023-10-10T19:21:06.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2650750647 0 0) 0x7fc5b0003570 con 0x7fc5c40c35e0 2023-10-10T19:21:06.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5c40c9390 msgr2=0x7fc5c4159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:06.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5c40c9390 0x7fc5c4159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 >> v1:172.21.15.191:6789/0 conn(0x7fc5c40c3210 legacy=0x7fc5c40c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:06.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c40c9b60 con 0x7fc5c40c35e0 2023-10-10T19:21:06.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fc5ac001470 con 0x7fc5c40c35e0 2023-10-10T19:21:06.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 >> v1:172.21.15.175:6789/0 conn(0x7fc5c40c35e0 legacy=0x7fc5c40c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:06.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5b0003a90 0x7fc5b0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:06.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5b0004810 0x7fc5b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:06.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5c40c9390 0x7fc5b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:06.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c4c50 con 0x7fc5b0003a90 2023-10-10T19:21:06.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000bb10 con 0x7fc5b0004810 2023-10-10T19:21:06.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000bcd0 con 0x7fc5c40c9390 2023-10-10T19:21:06.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c2ffd700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5b0003a90 0x7fc5b0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:06.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c27fc700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5b0004810 0x7fc5b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:06.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c37fe700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5c40c9390 0x7fc5b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:06.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c2ffd700 1 -- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5c40c9390 msgr2=0x7fc5b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:06.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c2ffd700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5c40c9390 0x7fc5b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c2ffd700 1 -- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5b0004810 msgr2=0x7fc5b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:06.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c2ffd700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5b0004810 0x7fc5b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c2ffd700 1 -- 172.21.15.175:0/2457606852 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0003570 con 0x7fc5b0003a90 2023-10-10T19:21:06.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.070+0000 7fc5c2ffd700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5b0003a90 0x7fc5b0003e60 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0019a0 tx=0x7fc5bc007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3309e3f0d24d7f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:06.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00c5d0 con 0x7fc5b0003a90 2023-10-10T19:21:06.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc00c730 con 0x7fc5b0003a90 2023-10-10T19:21:06.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c1ffb700 1 -- 172.21.15.175:0/2457606852 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00cad0 con 0x7fc5b0003a90 2023-10-10T19:21:06.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c97e5700 1 -- 172.21.15.175:0/2457606852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5b0003a90 msgr2=0x7fc5b0003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:06.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c97e5700 1 --2- 172.21.15.175:0/2457606852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5b0003a90 0x7fc5b0003e60 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0019a0 tx=0x7fc5bc007d10 comp rx=0 tx=0).stop 2023-10-10T19:21:06.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c37fe700 1 -- 172.21.15.175:0/2457606852 reap_dead start 2023-10-10T19:21:06.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c97e5700 1 -- 172.21.15.175:0/2457606852 shutdown_connections 2023-10-10T19:21:06.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c97e5700 1 -- 172.21.15.175:0/2457606852 >> 172.21.15.175:0/2457606852 conn(0x7fc5c406c110 msgr2=0x7fc5c406c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:06.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c97e5700 1 -- 172.21.15.175:0/2457606852 shutdown_connections 2023-10-10T19:21:06.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.071+0000 7fc5c97e5700 1 -- 172.21.15.175:0/2457606852 wait complete. 2023-10-10T19:21:06.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.072+0000 7fc5c97e5700 1 Processor -- start 2023-10-10T19:21:06.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.072+0000 7fc5c97e5700 1 -- start start 2023-10-10T19:21:06.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c97e5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5b0003a90 0x7fc5c4155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:06.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c97e5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5c4152b60 0x7fc5c4152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:06.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c97e5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5c4153eb0 0x7fc5c4154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:06.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c97e5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40ca400 con 0x7fc5c4153eb0 2023-10-10T19:21:06.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c97e5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c5840 con 0x7fc5c4152b60 2023-10-10T19:21:06.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c97e5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4156de0 con 0x7fc5b0003a90 2023-10-10T19:21:06.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5c4153eb0 0x7fc5c4154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:06.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5c4153eb0 0x7fc5c4154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38480/0 (socket says 172.21.15.175:38480) 2023-10-10T19:21:06.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c37fe700 1 -- 172.21.15.175:0/3700051408 learned_addr learned my addr 172.21.15.175:0/3700051408 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:06.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c27fc700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5c4152b60 0x7fc5c4152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:06.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c2ffd700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5b0003a90 0x7fc5c4155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:06.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c37fe700 1 -- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5b0003a90 msgr2=0x7fc5c4155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:06.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c37fe700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5b0003a90 0x7fc5c4155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c37fe700 1 -- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5c4152b60 msgr2=0x7fc5c4152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:06.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c37fe700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5c4152b60 0x7fc5c4152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c37fe700 1 -- 172.21.15.175:0/3700051408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c4155a00 con 0x7fc5c4153eb0 2023-10-10T19:21:06.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.073+0000 7fc5c27fc700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5c4152b60 0x7fc5c4152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:21:06.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.074+0000 7fc5c37fe700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5c4153eb0 0x7fc5c4154300 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40c97b0 tx=0x7fc5b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:06.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.074+0000 7fc5baffd700 1 -- 172.21.15.175:0/3700051408 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400e040 con 0x7fc5c4153eb0 2023-10-10T19:21:06.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.074+0000 7fc5baffd700 1 -- 172.21.15.175:0/3700051408 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b400f040 con 0x7fc5c4153eb0 2023-10-10T19:21:06.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.074+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c4156b00 con 0x7fc5c4153eb0 2023-10-10T19:21:06.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.074+0000 7fc5baffd700 1 -- 172.21.15.175:0/3700051408 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400fe50 con 0x7fc5c4153eb0 2023-10-10T19:21:06.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.074+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c41553b0 con 0x7fc5c4153eb0 2023-10-10T19:21:06.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.075+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c4152860 con 0x7fc5c4153eb0 2023-10-10T19:21:06.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.075+0000 7fc5baffd700 1 -- 172.21.15.175:0/3700051408 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc5b4015640 con 0x7fc5c4153eb0 2023-10-10T19:21:06.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.075+0000 7fc5baffd700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc5a4066f30 0x7fc5a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:06.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.076+0000 7fc5baffd700 1 -- 172.21.15.175:0/3700051408 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7fc5b40853d0 con 0x7fc5c4153eb0 2023-10-10T19:21:06.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.076+0000 7fc5c2ffd700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc5a4066f30 0x7fc5a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:06.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.076+0000 7fc5c2ffd700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc5a4066f30 0x7fc5a40693e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc000fd0 tx=0x7fc5bc015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:06.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.080+0000 7fc5baffd700 1 -- 172.21.15.175:0/3700051408 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5b4055a50 con 0x7fc5c4153eb0 2023-10-10T19:21:06.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.305+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5c4067570 con 0x7fc5c4153eb0 2023-10-10T19:21:06.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.306+0000 7fc5baffd700 1 -- 172.21.15.175:0/3700051408 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 272 v272) v1 ==== 95+0+5419 (secure 0 0 0) 0x7fc5b40555e0 con 0x7fc5c4153eb0 2023-10-10T19:21:06.308 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:06.308 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":272,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:06.035498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:06.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc5a4066f30 msgr2=0x7fc5a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:06.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc5a4066f30 0x7fc5a40693e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc000fd0 tx=0x7fc5bc015000 comp rx=0 tx=0).stop 2023-10-10T19:21:06.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5c4153eb0 msgr2=0x7fc5c4154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:06.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5c4153eb0 0x7fc5c4154300 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40c97b0 tx=0x7fc5b400b040 comp rx=0 tx=0).stop 2023-10-10T19:21:06.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 shutdown_connections 2023-10-10T19:21:06.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc5a4066f30 0x7fc5a40693e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5c4153eb0 0x7fc5c4154300 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5b0003a90 0x7fc5c4155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 --2- 172.21.15.175:0/3700051408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5c4152b60 0x7fc5c4152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:06.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 >> 172.21.15.175:0/3700051408 conn(0x7fc5c406c110 msgr2=0x7fc5c40c7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:06.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 shutdown_connections 2023-10-10T19:21:06.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:06.308+0000 7fc5c97e5700 1 -- 172.21.15.175:0/3700051408 wait complete. 2023-10-10T19:21:06.312 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 272 2023-10-10T19:21:06.322 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 272, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 272, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7252": { "addr": "172.21.15.191:6839/2393527600", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 2393527600, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 2393527600, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7252, "incarnation": 271, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 90 } }, "last_failure": 0, "last_failure_osd_epoch": 126, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:21:06.035498+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 7252, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:21:08.323 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:08.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.449+0000 7f6ec1927700 1 Processor -- start 2023-10-10T19:21:08.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.449+0000 7f6ec1927700 1 -- start start 2023-10-10T19:21:08.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ec1927700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ebc0b1fc0 0x7f6ebc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:08.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ec1927700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0aff60 0x7f6ebc0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:08.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ec1927700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc1574b0 con 0x7f6ebc0b2390 2023-10-10T19:21:08.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ec1927700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc150770 con 0x7f6ebc0b1fc0 2023-10-10T19:21:08.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ec1927700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0afad0 con 0x7f6ebc0aff60 2023-10-10T19:21:08.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebaffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ebc0b1fc0 0x7f6ebc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:08.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebaffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ebc0b1fc0 0x7f6ebc155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38484/0 (socket says 172.21.15.175:38484) 2023-10-10T19:21:08.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebaffd700 1 -- 172.21.15.175:0/543024178 learned_addr learned my addr 172.21.15.175:0/543024178 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:08.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebb7fe700 1 --2- 172.21.15.175:0/543024178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0aff60 0x7f6ebc0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:08.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebaffd700 1 -- 172.21.15.175:0/543024178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0aff60 msgr2=0x7f6ebc0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:08.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebaffd700 1 --2- 172.21.15.175:0/543024178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0aff60 0x7f6ebc0b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebaffd700 1 -- 172.21.15.175:0/543024178 >> v1:172.21.15.191:6789/0 conn(0x7f6ebc0b2390 legacy=0x7f6ebc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:08.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebaffd700 1 -- 172.21.15.175:0/543024178 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc0af4c0 con 0x7f6ebc0b1fc0 2023-10-10T19:21:08.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6ebaffd700 1 --2- 172.21.15.175:0/543024178 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0b1fc0 0x7f6ebc155fb0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009f20 tx=0x7f6eb4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f648d8ce781ab8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:08.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6eb9ffb700 1 -- 172.21.15.175:0/543024178 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb400e070 con 0x7f6ebc0b1fc0 2023-10-10T19:21:08.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6eb9ffb700 1 -- 172.21.15.175:0/543024178 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb400bad0 con 0x7f6ebc0b1fc0 2023-10-10T19:21:08.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.450+0000 7f6eb9ffb700 1 -- 172.21.15.175:0/543024178 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb40092e0 con 0x7f6ebc0b1fc0 2023-10-10T19:21:08.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.451+0000 7f6ec1927700 1 -- 172.21.15.175:0/543024178 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0b1fc0 msgr2=0x7f6ebc155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:08.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.451+0000 7f6ec1927700 1 --2- 172.21.15.175:0/543024178 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0b1fc0 0x7f6ebc155fb0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009f20 tx=0x7f6eb4009840 comp rx=0 tx=0).stop 2023-10-10T19:21:08.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.451+0000 7f6ec1927700 1 -- 172.21.15.175:0/543024178 shutdown_connections 2023-10-10T19:21:08.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.451+0000 7f6ec1927700 1 --2- 172.21.15.175:0/543024178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0aff60 0x7f6ebc0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.451+0000 7f6ec1927700 1 --2- 172.21.15.175:0/543024178 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0b1fc0 0x7f6ebc155fb0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.451+0000 7f6ec1927700 1 -- 172.21.15.175:0/543024178 >> 172.21.15.175:0/543024178 conn(0x7f6ebc063230 msgr2=0x7f6ebc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:08.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.451+0000 7f6ec1927700 1 -- 172.21.15.175:0/543024178 shutdown_connections 2023-10-10T19:21:08.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.451+0000 7f6ec1927700 1 -- 172.21.15.175:0/543024178 wait complete. 2023-10-10T19:21:08.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.452+0000 7f6ec1927700 1 Processor -- start 2023-10-10T19:21:08.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.452+0000 7f6ec1927700 1 -- start start 2023-10-10T19:21:08.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ec1927700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0aff60 0x7f6ebc0c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:08.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ec1927700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ebc0b1fc0 0x7f6ebc0c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:08.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ec1927700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0b2390 0x7f6ebc0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:08.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ec1927700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0c52a0 con 0x7f6ebc0aff60 2023-10-10T19:21:08.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ec1927700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0c6940 con 0x7f6ebc0b1fc0 2023-10-10T19:21:08.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ec1927700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc068b90 con 0x7f6ebc0b2390 2023-10-10T19:21:08.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0aff60 0x7f6ebc0c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:08.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0aff60 0x7f6ebc0c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38494/0 (socket says 172.21.15.175:38494) 2023-10-10T19:21:08.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 -- 172.21.15.175:0/3004133269 learned_addr learned my addr 172.21.15.175:0/3004133269 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:08.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebb7fe700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0b2390 0x7f6ebc0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:08.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6eba7fc700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ebc0b1fc0 0x7f6ebc0c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:08.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 -- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0b2390 msgr2=0x7f6ebc0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:08.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0b2390 0x7f6ebc0cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 -- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ebc0b1fc0 msgr2=0x7f6ebc0c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:08.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ebc0b1fc0 0x7f6ebc0c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 -- 172.21.15.175:0/3004133269 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb4009ae0 con 0x7f6ebc0aff60 2023-10-10T19:21:08.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6eba7fc700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ebc0b1fc0 0x7f6ebc0c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:21:08.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebb7fe700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0b2390 0x7f6ebc0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:21:08.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ebaffd700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0aff60 0x7f6ebc0c3ac0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc06fd90 tx=0x7f6eb4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:08.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.453+0000 7f6ea2ffd700 1 -- 172.21.15.175:0/3004133269 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4012b70 con 0x7f6ebc0aff60 2023-10-10T19:21:08.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.454+0000 7f6ea2ffd700 1 -- 172.21.15.175:0/3004133269 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb4012d10 con 0x7f6ebc0aff60 2023-10-10T19:21:08.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.454+0000 7f6ea2ffd700 1 -- 172.21.15.175:0/3004133269 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb401d670 con 0x7f6ebc0aff60 2023-10-10T19:21:08.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.454+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc0cb7a0 con 0x7f6ebc0aff60 2023-10-10T19:21:08.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.454+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc151690 con 0x7f6ebc0aff60 2023-10-10T19:21:08.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.455+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc067570 con 0x7f6ebc0aff60 2023-10-10T19:21:08.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.455+0000 7f6ea2ffd700 1 -- 172.21.15.175:0/3004133269 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6eb401da20 con 0x7f6ebc0aff60 2023-10-10T19:21:08.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.455+0000 7f6ea2ffd700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ea8066f60 0x7f6ea8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:08.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.455+0000 7f6ea2ffd700 1 -- 172.21.15.175:0/3004133269 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f6eb4086350 con 0x7f6ebc0aff60 2023-10-10T19:21:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.456+0000 7f6eba7fc700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ea8066f60 0x7f6ea8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.456+0000 7f6eba7fc700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ea8066f60 0x7f6ea8069410 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0006fd0 tx=0x7f6eb0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.460+0000 7f6ea2ffd700 1 -- 172.21.15.175:0/3004133269 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6eb4052f40 con 0x7f6ebc0aff60 2023-10-10T19:21:08.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.683+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ebc068da0 con 0x7f6ebc0aff60 2023-10-10T19:21:08.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.684+0000 7f6ea2ffd700 1 -- 172.21.15.175:0/3004133269 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 272 v272) v1 ==== 95+0+5419 (secure 0 0 0) 0x7f6eb4056560 con 0x7f6ebc0aff60 2023-10-10T19:21:08.686 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:08.686 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":272,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:06.035498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:08.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ea8066f60 msgr2=0x7f6ea8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:08.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ea8066f60 0x7f6ea8069410 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0006fd0 tx=0x7f6eb0008040 comp rx=0 tx=0).stop 2023-10-10T19:21:08.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0aff60 msgr2=0x7f6ebc0c3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:08.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0aff60 0x7f6ebc0c3ac0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc06fd90 tx=0x7f6eb4017000 comp rx=0 tx=0).stop 2023-10-10T19:21:08.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 shutdown_connections 2023-10-10T19:21:08.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ea8066f60 0x7f6ea8069410 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ebc0b2390 0x7f6ebc0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ebc0aff60 0x7f6ebc0c3ac0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 --2- 172.21.15.175:0/3004133269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ebc0b1fc0 0x7f6ebc0c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:08.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 >> 172.21.15.175:0/3004133269 conn(0x7f6ebc063230 msgr2=0x7f6ebc14fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:08.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 shutdown_connections 2023-10-10T19:21:08.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:08.686+0000 7f6ec1927700 1 -- 172.21.15.175:0/3004133269 wait complete. 2023-10-10T19:21:08.690 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 272 2023-10-10T19:21:08.699 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:21:10.700 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:10.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.824+0000 7f4784f7f700 1 Processor -- start 2023-10-10T19:21:10.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.824+0000 7f4784f7f700 1 -- start start 2023-10-10T19:21:10.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.825+0000 7f4784f7f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f47800c2b80 0x7f4780156580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:10.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.825+0000 7f4784f7f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4780156ac0 0x7f47800c7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:10.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.825+0000 7f4784f7f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478015a6c0 con 0x7f47800c35a0 2023-10-10T19:21:10.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.825+0000 7f4784f7f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478014d4a0 con 0x7f47800c2b80 2023-10-10T19:21:10.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.825+0000 7f4784f7f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4780159830 con 0x7f4780156ac0 2023-10-10T19:21:10.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477e59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f47800c2b80 0x7f4780156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:10.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477e59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f47800c2b80 0x7f4780156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38498/0 (socket says 172.21.15.175:38498) 2023-10-10T19:21:10.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477e59c700 1 -- 172.21.15.175:0/1969616547 learned_addr learned my addr 172.21.15.175:0/1969616547 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:10.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477ed9d700 1 --2- 172.21.15.175:0/1969616547 >> v2:172.21.15.191:3300/0 conn(0x7f4780156ac0 0x7f47800c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:10.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477e59c700 1 -- 172.21.15.175:0/1969616547 >> v2:172.21.15.191:3300/0 conn(0x7f4780156ac0 msgr2=0x7f47800c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:10.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477e59c700 1 --2- 172.21.15.175:0/1969616547 >> v2:172.21.15.191:3300/0 conn(0x7f4780156ac0 0x7f47800c7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:10.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477e59c700 1 -- 172.21.15.175:0/1969616547 >> v1:172.21.15.191:6789/0 conn(0x7f47800c35a0 legacy=0x7f47800c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:10.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477e59c700 1 -- 172.21.15.175:0/1969616547 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780159ab0 con 0x7f47800c2b80 2023-10-10T19:21:10.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477e59c700 1 --2- 172.21.15.175:0/1969616547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c2b80 0x7f4780156580 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f47800c5640 tx=0x7f4770009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd1e0556479fcd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:10.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477d59a700 1 -- 172.21.15.175:0/1969616547 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000e070 con 0x7f47800c2b80 2023-10-10T19:21:10.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477d59a700 1 -- 172.21.15.175:0/1969616547 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477000bab0 con 0x7f47800c2b80 2023-10-10T19:21:10.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.826+0000 7f477d59a700 1 -- 172.21.15.175:0/1969616547 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47700092e0 con 0x7f47800c2b80 2023-10-10T19:21:10.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.827+0000 7f4784f7f700 1 -- 172.21.15.175:0/1969616547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c2b80 msgr2=0x7f4780156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:10.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.827+0000 7f4784f7f700 1 --2- 172.21.15.175:0/1969616547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c2b80 0x7f4780156580 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f47800c5640 tx=0x7f4770009840 comp rx=0 tx=0).stop 2023-10-10T19:21:10.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.827+0000 7f4784f7f700 1 -- 172.21.15.175:0/1969616547 shutdown_connections 2023-10-10T19:21:10.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.827+0000 7f4784f7f700 1 --2- 172.21.15.175:0/1969616547 >> v2:172.21.15.191:3300/0 conn(0x7f4780156ac0 0x7f47800c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:10.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.827+0000 7f4784f7f700 1 --2- 172.21.15.175:0/1969616547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c2b80 0x7f4780156580 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:10.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.827+0000 7f4784f7f700 1 -- 172.21.15.175:0/1969616547 >> 172.21.15.175:0/1969616547 conn(0x7f47800c56a0 msgr2=0x7f47800c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:10.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.827+0000 7f4784f7f700 1 -- 172.21.15.175:0/1969616547 shutdown_connections 2023-10-10T19:21:10.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.827+0000 7f4784f7f700 1 -- 172.21.15.175:0/1969616547 wait complete. 2023-10-10T19:21:10.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.828+0000 7f4784f7f700 1 Processor -- start 2023-10-10T19:21:10.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.828+0000 7f4784f7f700 1 -- start start 2023-10-10T19:21:10.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f4784f7f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47800c2b80 0x7f47800b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:10.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f4784f7f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c35a0 0x7f4780158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:10.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f4784f7f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4780156ac0 0x7f4780167460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:10.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f4784f7f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47800ba490 con 0x7f47800c35a0 2023-10-10T19:21:10.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f4784f7f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4780068090 con 0x7f4780156ac0 2023-10-10T19:21:10.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f4784f7f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4780153dd0 con 0x7f47800c2b80 2023-10-10T19:21:10.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c35a0 0x7f4780158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:10.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c35a0 0x7f4780158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38508/0 (socket says 172.21.15.175:38508) 2023-10-10T19:21:10.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 -- 172.21.15.175:0/38932378 learned_addr learned my addr 172.21.15.175:0/38932378 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:10.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477ed9d700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4780156ac0 0x7f4780167460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:10.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477e59c700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47800c2b80 0x7f47800b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:10.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 -- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47800c2b80 msgr2=0x7f47800b9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:10.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47800c2b80 0x7f47800b9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:10.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 -- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4780156ac0 msgr2=0x7f4780167460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:10.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4780156ac0 0x7f4780167460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:10.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 -- 172.21.15.175:0/38932378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770009ae0 con 0x7f47800c35a0 2023-10-10T19:21:10.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f477dd9b700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c35a0 0x7f4780158eb0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f476800c7f0 tx=0x7f4768007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:10.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f47767fc700 1 -- 172.21.15.175:0/38932378 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768011590 con 0x7f47800c35a0 2023-10-10T19:21:10.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f47767fc700 1 -- 172.21.15.175:0/38932378 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4768011710 con 0x7f47800c35a0 2023-10-10T19:21:10.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4780158a20 con 0x7f47800c35a0 2023-10-10T19:21:10.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f47767fc700 1 -- 172.21.15.175:0/38932378 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768011ad0 con 0x7f47800c35a0 2023-10-10T19:21:10.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.829+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47801542c0 con 0x7f47800c35a0 2023-10-10T19:21:10.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.830+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478006a140 con 0x7f47800c35a0 2023-10-10T19:21:10.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.831+0000 7f47767fc700 1 -- 172.21.15.175:0/38932378 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4768012420 con 0x7f47800c35a0 2023-10-10T19:21:10.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.831+0000 7f47767fc700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4760066f60 0x7f4760069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:10.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.831+0000 7f47767fc700 1 -- 172.21.15.175:0/38932378 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f4768083dc0 con 0x7f47800c35a0 2023-10-10T19:21:10.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.831+0000 7f477e59c700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4760066f60 0x7f4760069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:10.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.832+0000 7f477e59c700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4760066f60 0x7f4760069410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4770009880 tx=0x7f4770018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:10.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:10.835+0000 7f47767fc700 1 -- 172.21.15.175:0/38932378 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f476804b5a0 con 0x7f47800c35a0 2023-10-10T19:21:11.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.062+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4780067570 con 0x7f47800c35a0 2023-10-10T19:21:11.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.063+0000 7f47767fc700 1 -- 172.21.15.175:0/38932378 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 272 v272) v1 ==== 95+0+5419 (secure 0 0 0) 0x7f4768017070 con 0x7f47800c35a0 2023-10-10T19:21:11.064 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:11.065 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":272,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:06.035498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:11.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4760066f60 msgr2=0x7f4760069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:11.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4760066f60 0x7f4760069410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4770009880 tx=0x7f4770018000 comp rx=0 tx=0).stop 2023-10-10T19:21:11.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c35a0 msgr2=0x7f4780158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:11.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c35a0 0x7f4780158eb0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f476800c7f0 tx=0x7f4768007f50 comp rx=0 tx=0).stop 2023-10-10T19:21:11.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 shutdown_connections 2023-10-10T19:21:11.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4760066f60 0x7f4760069410 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:11.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4780156ac0 0x7f4780167460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:11.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47800c2b80 0x7f47800b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:11.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 --2- 172.21.15.175:0/38932378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47800c35a0 0x7f4780158eb0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:11.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 >> 172.21.15.175:0/38932378 conn(0x7f47800c56a0 msgr2=0x7f47800c1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:11.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 shutdown_connections 2023-10-10T19:21:11.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:11.065+0000 7f4784f7f700 1 -- 172.21.15.175:0/38932378 wait complete. 2023-10-10T19:21:11.069 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 272 2023-10-10T19:21:11.079 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:21:13.080 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:13.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.204+0000 7f270b59e700 1 Processor -- start 2023-10-10T19:21:13.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.204+0000 7f270b59e700 1 -- start start 2023-10-10T19:21:13.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270b59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 0x7f270c14fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270b59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f270c0c4630 0x7f270c150220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270b59e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0c3a90 con 0x7f270c0c62f0 2023-10-10T19:21:13.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270b59e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270c150760 con 0x7f270c0c4630 2023-10-10T19:21:13.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270b59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c0c5b30 con 0x7f270c089fe0 2023-10-10T19:21:13.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270a59c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f270c0c62f0 0x7f270c0c3f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38752/0 (socket says 172.21.15.175:38752) 2023-10-10T19:21:13.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270a59c700 1 -- 172.21.15.175:0/41289665 learned_addr learned my addr 172.21.15.175:0/41289665 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:13.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270ad9d700 1 --2- 172.21.15.175:0/41289665 >> v2:172.21.15.191:3300/0 conn(0x7f270c0c4630 0x7f270c150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f2709d9b700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 0x7f270c14fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152207224 0 0) 0x7f270c0c3a90 con 0x7f270c0c62f0 2023-10-10T19:21:13.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f4003570 con 0x7f270c0c62f0 2023-10-10T19:21:13.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1857605360 0 0) 0x7f26f4003570 con 0x7f270c0c62f0 2023-10-10T19:21:13.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 msgr2=0x7f270c14fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:13.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 0x7f270c14fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 >> v2:172.21.15.191:3300/0 conn(0x7f270c0c4630 msgr2=0x7f270c150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:13.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 --2- 172.21.15.175:0/41289665 >> v2:172.21.15.191:3300/0 conn(0x7f270c0c4630 0x7f270c150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c156130 con 0x7f270c0c62f0 2023-10-10T19:21:13.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.205+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f26fc001470 con 0x7f270c0c62f0 2023-10-10T19:21:13.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 >> v1:172.21.15.175:6789/0 conn(0x7f270c0c62f0 legacy=0x7f270c0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:13.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26f4003a90 0x7f26f4003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26f4004810 0x7f26f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 0x7f26f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c150760 con 0x7f26f4003a90 2023-10-10T19:21:13.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f400ba60 con 0x7f26f4004810 2023-10-10T19:21:13.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f400bbd0 con 0x7f270c089fe0 2023-10-10T19:21:13.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270ad9d700 1 --2- 172.21.15.175:0/41289665 >> v2:172.21.15.191:3300/0 conn(0x7f270c0c4630 0x7f270c150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:21:13.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270a59c700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26f4003a90 0x7f26f4003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270ad9d700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 0x7f26f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f2709d9b700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26f4004810 0x7f26f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270a59c700 1 -- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 msgr2=0x7f26f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:13.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270a59c700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 0x7f26f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270a59c700 1 -- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26f4004810 msgr2=0x7f26f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:13.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270a59c700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26f4004810 0x7f26f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270a59c700 1 -- 172.21.15.175:0/41289665 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f4003570 con 0x7f26f4003a90 2023-10-10T19:21:13.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270ad9d700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f270c089fe0 0x7f26f40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:21:13.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f2709d9b700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26f4004810 0x7f26f4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:21:13.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270a59c700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26f4003a90 0x7f26f4003ea0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f26fc00c3c0 tx=0x7f26fc001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e032d1a2d1324e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:13.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc00cc30 con 0x7f26f4003a90 2023-10-10T19:21:13.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc00cd90 con 0x7f26f4003a90 2023-10-10T19:21:13.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.206+0000 7f270959a700 1 -- 172.21.15.175:0/41289665 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc005b70 con 0x7f26f4003a90 2023-10-10T19:21:13.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.207+0000 7f270b59e700 1 -- 172.21.15.175:0/41289665 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26f4003a90 msgr2=0x7f26f4003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:13.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.207+0000 7f270b59e700 1 --2- 172.21.15.175:0/41289665 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26f4003a90 0x7f26f4003ea0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f26fc00c3c0 tx=0x7f26fc001e50 comp rx=0 tx=0).stop 2023-10-10T19:21:13.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.207+0000 7f270ad9d700 1 -- 172.21.15.175:0/41289665 reap_dead start 2023-10-10T19:21:13.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.207+0000 7f270b59e700 1 -- 172.21.15.175:0/41289665 shutdown_connections 2023-10-10T19:21:13.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.207+0000 7f270b59e700 1 -- 172.21.15.175:0/41289665 >> 172.21.15.175:0/41289665 conn(0x7f270c063cc0 msgr2=0x7f270c0cc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:13.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.207+0000 7f270b59e700 1 -- 172.21.15.175:0/41289665 shutdown_connections 2023-10-10T19:21:13.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.207+0000 7f270b59e700 1 -- 172.21.15.175:0/41289665 wait complete. 2023-10-10T19:21:13.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.208+0000 7f270b59e700 1 Processor -- start 2023-10-10T19:21:13.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.208+0000 7f270b59e700 1 -- start start 2023-10-10T19:21:13.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270b59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26f4003a90 0x7f270c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270b59e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f270c152cf0 0x7f270c14a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270b59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c14ad70 0x7f270c1693c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270b59e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c153140 con 0x7f270c14ad70 2023-10-10T19:21:13.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270b59e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c1539d0 con 0x7f270c152cf0 2023-10-10T19:21:13.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270b59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c152320 con 0x7f26f4003a90 2023-10-10T19:21:13.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c14ad70 0x7f270c1693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c14ad70 0x7f270c1693c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38526/0 (socket says 172.21.15.175:38526) 2023-10-10T19:21:13.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 -- 172.21.15.175:0/1868406696 learned_addr learned my addr 172.21.15.175:0/1868406696 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:13.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270a59c700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26f4003a90 0x7f270c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f2709d9b700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f270c152cf0 0x7f270c14a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 -- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26f4003a90 msgr2=0x7f270c1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:13.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26f4003a90 0x7f270c1527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 -- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f270c152cf0 msgr2=0x7f270c14a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:13.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f270c152cf0 0x7f270c14a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 -- 172.21.15.175:0/1868406696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c1516a0 con 0x7f270c14ad70 2023-10-10T19:21:13.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270ad9d700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c14ad70 0x7f270c1693c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f270c0714d0 tx=0x7f270400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:13.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f26fa7fc700 1 -- 172.21.15.175:0/1868406696 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270400e070 con 0x7f270c14ad70 2023-10-10T19:21:13.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f26fa7fc700 1 -- 172.21.15.175:0/1868406696 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2704011db0 con 0x7f270c14ad70 2023-10-10T19:21:13.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f26fa7fc700 1 -- 172.21.15.175:0/1868406696 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27040166c0 con 0x7f270c14ad70 2023-10-10T19:21:13.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270c1555f0 con 0x7f270c14ad70 2023-10-10T19:21:13.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.209+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c151fb0 con 0x7f270c14ad70 2023-10-10T19:21:13.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.210+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c152460 con 0x7f270c14ad70 2023-10-10T19:21:13.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.211+0000 7f26fa7fc700 1 -- 172.21.15.175:0/1868406696 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2704016ab0 con 0x7f270c14ad70 2023-10-10T19:21:13.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.211+0000 7f26fa7fc700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26ec066f30 0x7f26ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:13.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.211+0000 7f26fa7fc700 1 -- 172.21.15.175:0/1868406696 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f27040858d0 con 0x7f270c14ad70 2023-10-10T19:21:13.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.211+0000 7f270a59c700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26ec066f30 0x7f26ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:13.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.212+0000 7f270a59c700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26ec066f30 0x7f26ec0693e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f26fc001e90 tx=0x7f26fc015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:13.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.215+0000 7f26fa7fc700 1 -- 172.21.15.175:0/1868406696 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27040524c0 con 0x7f270c14ad70 2023-10-10T19:21:13.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.439+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f270c04bb80 con 0x7f270c14ad70 2023-10-10T19:21:13.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.440+0000 7f26fa7fc700 1 -- 172.21.15.175:0/1868406696 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+5421 (secure 0 0 0) 0x7f2704055ae0 con 0x7f270c14ad70 2023-10-10T19:21:13.441 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:13.442 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:12.054604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:resolve","state_seq":14,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:13.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26ec066f30 msgr2=0x7f26ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:13.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26ec066f30 0x7f26ec0693e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f26fc001e90 tx=0x7f26fc015000 comp rx=0 tx=0).stop 2023-10-10T19:21:13.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c14ad70 msgr2=0x7f270c1693c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:13.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c14ad70 0x7f270c1693c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f270c0714d0 tx=0x7f270400b040 comp rx=0 tx=0).stop 2023-10-10T19:21:13.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 shutdown_connections 2023-10-10T19:21:13.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f26ec066f30 0x7f26ec0693e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c14ad70 0x7f270c1693c0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26f4003a90 0x7f270c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 --2- 172.21.15.175:0/1868406696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f270c152cf0 0x7f270c14a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:13.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 >> 172.21.15.175:0/1868406696 conn(0x7f270c063cc0 msgr2=0x7f270c15afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:13.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 shutdown_connections 2023-10-10T19:21:13.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:13.442+0000 7f270b59e700 1 -- 172.21.15.175:0/1868406696 wait complete. 2023-10-10T19:21:13.446 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 273 2023-10-10T19:21:13.455 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:21:15.456 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:15.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.581+0000 7f80148d0700 1 Processor -- start 2023-10-10T19:21:15.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.581+0000 7f80148d0700 1 -- start start 2023-10-10T19:21:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f80148d0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f801014fd90 0x7f8010159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f80148d0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f80100cd040 0x7f80100684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f80148d0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8010067fa0 con 0x7f801008a100 2023-10-10T19:21:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f80148d0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80100c5600 con 0x7f801014fd90 2023-10-10T19:21:15.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f80148d0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8010156b70 con 0x7f80100cd040 2023-10-10T19:21:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f801014fd90 0x7f8010159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f801014fd90 0x7f8010159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59452/0 (socket says 172.21.15.175:59452) 2023-10-10T19:21:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800ed9d700 1 -- 172.21.15.175:0/240781680 learned_addr learned my addr 172.21.15.175:0/240781680 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800f59e700 1 --2- 172.21.15.175:0/240781680 >> v2:172.21.15.191:3300/0 conn(0x7f80100cd040 0x7f80100684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:15.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800ed9d700 1 -- 172.21.15.175:0/240781680 >> v2:172.21.15.191:3300/0 conn(0x7f80100cd040 msgr2=0x7f80100684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:15.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800ed9d700 1 --2- 172.21.15.175:0/240781680 >> v2:172.21.15.191:3300/0 conn(0x7f80100cd040 0x7f80100684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800ed9d700 1 -- 172.21.15.175:0/240781680 >> v1:172.21.15.191:6789/0 conn(0x7f801008a100 legacy=0x7f80100c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:15.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800ed9d700 1 -- 172.21.15.175:0/240781680 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8010150160 con 0x7f801014fd90 2023-10-10T19:21:15.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.582+0000 7f800ed9d700 1 --2- 172.21.15.175:0/240781680 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f801014fd90 0x7f8010159c30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f801008a6e0 tx=0x7f8000009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4755de038fad149 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:15.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f800dd9b700 1 -- 172.21.15.175:0/240781680 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800000e070 con 0x7f801014fd90 2023-10-10T19:21:15.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f800dd9b700 1 -- 172.21.15.175:0/240781680 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800000baa0 con 0x7f801014fd90 2023-10-10T19:21:15.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f800dd9b700 1 -- 172.21.15.175:0/240781680 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000092e0 con 0x7f801014fd90 2023-10-10T19:21:15.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f80148d0700 1 -- 172.21.15.175:0/240781680 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f801014fd90 msgr2=0x7f8010159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:15.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f80148d0700 1 --2- 172.21.15.175:0/240781680 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f801014fd90 0x7f8010159c30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f801008a6e0 tx=0x7f8000009840 comp rx=0 tx=0).stop 2023-10-10T19:21:15.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f80148d0700 1 -- 172.21.15.175:0/240781680 shutdown_connections 2023-10-10T19:21:15.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f80148d0700 1 --2- 172.21.15.175:0/240781680 >> v2:172.21.15.191:3300/0 conn(0x7f80100cd040 0x7f80100684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f80148d0700 1 --2- 172.21.15.175:0/240781680 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f801014fd90 0x7f8010159c30 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f80148d0700 1 -- 172.21.15.175:0/240781680 >> 172.21.15.175:0/240781680 conn(0x7f8010063d00 msgr2=0x7f80100cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:15.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f80148d0700 1 -- 172.21.15.175:0/240781680 shutdown_connections 2023-10-10T19:21:15.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.583+0000 7f80148d0700 1 -- 172.21.15.175:0/240781680 wait complete. 2023-10-10T19:21:15.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.584+0000 7f80148d0700 1 Processor -- start 2023-10-10T19:21:15.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.584+0000 7f80148d0700 1 -- start start 2023-10-10T19:21:15.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f80148d0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f801008a100 0x7f8010169b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:15.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f80148d0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80100cd040 0x7f80101665a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:15.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f80148d0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f801014fd90 0x7f8010157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:15.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f80148d0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8010165bd0 con 0x7f80100cd040 2023-10-10T19:21:15.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f80148d0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8010159150 con 0x7f801014fd90 2023-10-10T19:21:15.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f80148d0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8010158940 con 0x7f801008a100 2023-10-10T19:21:15.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f800e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80100cd040 0x7f80101665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:15.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f800e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80100cd040 0x7f80101665a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59462/0 (socket says 172.21.15.175:59462) 2023-10-10T19:21:15.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f800e59c700 1 -- 172.21.15.175:0/251330283 learned_addr learned my addr 172.21.15.175:0/251330283 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:15.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f800ed9d700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f801008a100 0x7f8010169b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:15.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.585+0000 7f800f59e700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f801014fd90 0x7f8010157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:15.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f800e59c700 1 -- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f801008a100 msgr2=0x7f8010169b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:15.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f800e59c700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f801008a100 0x7f8010169b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f800e59c700 1 -- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f801014fd90 msgr2=0x7f8010157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:15.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f800e59c700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f801014fd90 0x7f8010157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f800e59c700 1 -- 172.21.15.175:0/251330283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8000009ae0 con 0x7f80100cd040 2023-10-10T19:21:15.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f800f59e700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f801014fd90 0x7f8010157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:21:15.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f800e59c700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80100cd040 0x7f80101665a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800b630 tx=0x7f7ff80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:15.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f8006ffd700 1 -- 172.21.15.175:0/251330283 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800f510 con 0x7f80100cd040 2023-10-10T19:21:15.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f8006ffd700 1 -- 172.21.15.175:0/251330283 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff800f690 con 0x7f80100cd040 2023-10-10T19:21:15.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8010166110 con 0x7f80100cd040 2023-10-10T19:21:15.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f8006ffd700 1 -- 172.21.15.175:0/251330283 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800fa50 con 0x7f80100cd040 2023-10-10T19:21:15.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.586+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8010165de0 con 0x7f80100cd040 2023-10-10T19:21:15.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.587+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8010158bf0 con 0x7f80100cd040 2023-10-10T19:21:15.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.587+0000 7f8006ffd700 1 -- 172.21.15.175:0/251330283 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7ff801a400 con 0x7f80100cd040 2023-10-10T19:21:15.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.588+0000 7f8006ffd700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ff0066f30 0x7f7ff00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:15.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.588+0000 7f8006ffd700 1 -- 172.21.15.175:0/251330283 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f7ff80846b0 con 0x7f80100cd040 2023-10-10T19:21:15.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.588+0000 7f800ed9d700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ff0066f30 0x7f7ff00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:15.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.588+0000 7f800ed9d700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ff0066f30 0x7f7ff00693e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8000009880 tx=0x7f8000017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:15.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.592+0000 7f8006ffd700 1 -- 172.21.15.175:0/251330283 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ff8051320 con 0x7f80100cd040 2023-10-10T19:21:15.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.820+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8010067570 con 0x7f80100cd040 2023-10-10T19:21:15.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.820+0000 7f8006ffd700 1 -- 172.21.15.175:0/251330283 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+5421 (secure 0 0 0) 0x7f7ff8054940 con 0x7f80100cd040 2023-10-10T19:21:15.822 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:15.822 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:12.054604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:resolve","state_seq":14,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:15.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ff0066f30 msgr2=0x7f7ff00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:15.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ff0066f30 0x7f7ff00693e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8000009880 tx=0x7f8000017000 comp rx=0 tx=0).stop 2023-10-10T19:21:15.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80100cd040 msgr2=0x7f80101665a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:15.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80100cd040 0x7f80101665a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800b630 tx=0x7f7ff80091a0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 shutdown_connections 2023-10-10T19:21:15.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7ff0066f30 0x7f7ff00693e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f801014fd90 0x7f8010157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f801008a100 0x7f8010169b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 --2- 172.21.15.175:0/251330283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f80100cd040 0x7f80101665a0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:15.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 >> 172.21.15.175:0/251330283 conn(0x7f8010063d00 msgr2=0x7f801014eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:15.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 shutdown_connections 2023-10-10T19:21:15.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:15.822+0000 7f80148d0700 1 -- 172.21.15.175:0/251330283 wait complete. 2023-10-10T19:21:15.827 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 273 2023-10-10T19:21:15.836 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:21:17.837 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:17.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.957+0000 7ffb1f497700 1 Processor -- start 2023-10-10T19:21:17.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.957+0000 7ffb1f497700 1 -- start start 2023-10-10T19:21:17.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.958+0000 7ffb1f497700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb180b1fc0 0x7ffb18155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:17.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.958+0000 7ffb1f497700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb181574b0 con 0x7ffb180b2390 2023-10-10T19:21:17.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.958+0000 7ffb1f497700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180afad0 con 0x7ffb180aff60 2023-10-10T19:21:17.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.958+0000 7ffb1f497700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18150770 con 0x7ffb180b1fc0 2023-10-10T19:21:17.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.958+0000 7ffb1ca32700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ffb180b2390 0x7ffb18068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49722/0 (socket says 172.21.15.175:49722) 2023-10-10T19:21:17.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.958+0000 7ffb1ca32700 1 -- 172.21.15.175:0/1733062599 learned_addr learned my addr 172.21.15.175:0/1733062599 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:17.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1d233700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb180b1fc0 0x7ffb18155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:17.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921798634 0 0) 0x7ffb181574b0 con 0x7ffb180b2390 2023-10-10T19:21:17.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04003570 con 0x7ffb180b2390 2023-10-10T19:21:17.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075881346 0 0) 0x7ffb180afad0 con 0x7ffb180aff60 2023-10-10T19:21:17.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb181574b0 con 0x7ffb180aff60 2023-10-10T19:21:17.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097469183 0 0) 0x7ffb04003570 con 0x7ffb180b2390 2023-10-10T19:21:17.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb180b1fc0 msgr2=0x7ffb18155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:17.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb180b1fc0 0x7ffb18155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:17.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 >> v1:172.21.15.191:6789/0 conn(0x7ffb180aff60 legacy=0x7ffb18159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:17.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180af4c0 con 0x7ffb180b2390 2023-10-10T19:21:17.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7ffb08001470 con 0x7ffb180b2390 2023-10-10T19:21:17.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 >> v1:172.21.15.175:6789/0 conn(0x7ffb180b2390 legacy=0x7ffb18068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:17.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb180b1fc0 0x7ffb04003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:17.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb04004180 0x7ffb04004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:17.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb04004d50 0x7ffb040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:17.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18150770 con 0x7ffb04004180 2023-10-10T19:21:17.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0400bba0 con 0x7ffb04004d50 2023-10-10T19:21:17.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0400bd60 con 0x7ffb180b1fc0 2023-10-10T19:21:17.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1ca32700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb04004180 0x7ffb04004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:17.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1d233700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb180b1fc0 0x7ffb04003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:17.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1da34700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb04004d50 0x7ffb040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:17.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1ca32700 1 -- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb180b1fc0 msgr2=0x7ffb04003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:17.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1ca32700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb180b1fc0 0x7ffb04003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:17.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1ca32700 1 -- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb04004d50 msgr2=0x7ffb040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:17.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1ca32700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb04004d50 0x7ffb040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:17.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.959+0000 7ffb1ca32700 1 -- 172.21.15.175:0/1733062599 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04003570 con 0x7ffb04004180 2023-10-10T19:21:17.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb1ca32700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb04004180 0x7ffb04004810 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800c2d0 tx=0x7ffb180afa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b3de88bce0d1a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:17.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0800cba0 con 0x7ffb04004180 2023-10-10T19:21:17.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0800cd20 con 0x7ffb04004180 2023-10-10T19:21:17.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb0ffff700 1 -- 172.21.15.175:0/1733062599 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08005b70 con 0x7ffb04004180 2023-10-10T19:21:17.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb1f497700 1 -- 172.21.15.175:0/1733062599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb04004180 msgr2=0x7ffb04004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:17.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb1f497700 1 --2- 172.21.15.175:0/1733062599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb04004180 0x7ffb04004810 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800c2d0 tx=0x7ffb180afa40 comp rx=0 tx=0).stop 2023-10-10T19:21:17.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb1da34700 1 -- 172.21.15.175:0/1733062599 reap_dead start 2023-10-10T19:21:17.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb1f497700 1 -- 172.21.15.175:0/1733062599 shutdown_connections 2023-10-10T19:21:17.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb1f497700 1 -- 172.21.15.175:0/1733062599 >> 172.21.15.175:0/1733062599 conn(0x7ffb18063230 msgr2=0x7ffb18063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:17.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb1f497700 1 -- 172.21.15.175:0/1733062599 shutdown_connections 2023-10-10T19:21:17.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.960+0000 7ffb1f497700 1 -- 172.21.15.175:0/1733062599 wait complete. 2023-10-10T19:21:17.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.961+0000 7ffb1f497700 1 Processor -- start 2023-10-10T19:21:17.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.961+0000 7ffb1f497700 1 -- start start 2023-10-10T19:21:17.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1f497700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb04004180 0x7ffb18169100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:17.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1f497700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb180c5710 0x7ffb18166e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:17.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1f497700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb181673a0 0x7ffb180c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:17.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1f497700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb180c2970 con 0x7ffb181673a0 2023-10-10T19:21:17.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1f497700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb180c6570 con 0x7ffb180c5710 2023-10-10T19:21:17.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1f497700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18169df0 con 0x7ffb04004180 2023-10-10T19:21:17.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1da34700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb181673a0 0x7ffb180c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:17.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1da34700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb181673a0 0x7ffb180c3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59488/0 (socket says 172.21.15.175:59488) 2023-10-10T19:21:17.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1da34700 1 -- 172.21.15.175:0/2030453207 learned_addr learned my addr 172.21.15.175:0/2030453207 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:17.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1ca32700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb180c5710 0x7ffb18166e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:17.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.962+0000 7ffb1d233700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb04004180 0x7ffb18169100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:17.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb1da34700 1 -- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb04004180 msgr2=0x7ffb18169100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:17.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb1da34700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb04004180 0x7ffb18169100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:17.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb1da34700 1 -- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb180c5710 msgr2=0x7ffb18166e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:17.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb1da34700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb180c5710 0x7ffb18166e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:17.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb1da34700 1 -- 172.21.15.175:0/2030453207 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180c69c0 con 0x7ffb181673a0 2023-10-10T19:21:17.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb1da34700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb181673a0 0x7ffb180c3870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ffb1808a030 tx=0x7ffb10007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:17.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb0d7fa700 1 -- 172.21.15.175:0/2030453207 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10006820 con 0x7ffb181673a0 2023-10-10T19:21:17.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb0d7fa700 1 -- 172.21.15.175:0/2030453207 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb10006980 con 0x7ffb181673a0 2023-10-10T19:21:17.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb180c5c80 con 0x7ffb181673a0 2023-10-10T19:21:17.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb0d7fa700 1 -- 172.21.15.175:0/2030453207 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10006d40 con 0x7ffb181673a0 2023-10-10T19:21:17.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.963+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb180c2fe0 con 0x7ffb181673a0 2023-10-10T19:21:17.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.964+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb180c66e0 con 0x7ffb181673a0 2023-10-10T19:21:17.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.964+0000 7ffb0d7fa700 1 -- 172.21.15.175:0/2030453207 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffb10007240 con 0x7ffb181673a0 2023-10-10T19:21:17.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.965+0000 7ffb0d7fa700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffafc066f30 0x7ffafc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:17.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.965+0000 7ffb0d7fa700 1 -- 172.21.15.175:0/2030453207 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7ffb10084680 con 0x7ffb181673a0 2023-10-10T19:21:17.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.965+0000 7ffb1d233700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffafc066f30 0x7ffafc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:17.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.965+0000 7ffb1d233700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffafc066f30 0x7ffafc0693e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffb140094d0 tx=0x7ffb1400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:17.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:17.969+0000 7ffb0d7fa700 1 -- 172.21.15.175:0/2030453207 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffb180c66e0 con 0x7ffb181673a0 2023-10-10T19:21:18.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.200+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb180c66e0 con 0x7ffb181673a0 2023-10-10T19:21:18.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.200+0000 7ffb0d7fa700 1 -- 172.21.15.175:0/2030453207 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 274 v274) v1 ==== 95+0+5423 (secure 0 0 0) 0x7ffb180c66e0 con 0x7ffb181673a0 2023-10-10T19:21:18.202 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:18.202 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":274,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":274,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:18.079855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:18.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffafc066f30 msgr2=0x7ffafc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:18.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffafc066f30 0x7ffafc0693e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffb140094d0 tx=0x7ffb1400b040 comp rx=0 tx=0).stop 2023-10-10T19:21:18.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb181673a0 msgr2=0x7ffb180c3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:18.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb181673a0 0x7ffb180c3870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ffb1808a030 tx=0x7ffb10007d30 comp rx=0 tx=0).stop 2023-10-10T19:21:18.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 shutdown_connections 2023-10-10T19:21:18.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffafc066f30 0x7ffafc0693e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:18.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb181673a0 0x7ffb180c3870 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:18.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb04004180 0x7ffb18169100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:18.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 --2- 172.21.15.175:0/2030453207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb180c5710 0x7ffb18166e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:18.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 >> 172.21.15.175:0/2030453207 conn(0x7ffb18063230 msgr2=0x7ffb180b04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:18.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 shutdown_connections 2023-10-10T19:21:18.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:18.202+0000 7ffb1f497700 1 -- 172.21.15.175:0/2030453207 wait complete. 2023-10-10T19:21:18.207 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 274 2023-10-10T19:21:18.217 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 274, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 274, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_6618": { "addr": "172.21.15.175:6836/972555107", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 972555107, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 972555107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6618, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7252": { "addr": "172.21.15.191:6839/2393527600", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 2393527600, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 2393527600, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7252, "incarnation": 271, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:reconnect", "state_seq": 16 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 90 } }, "last_failure": 0, "last_failure_osd_epoch": 126, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:21:18.079855+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 7252, "mds_1": 7401, "mds_2": 7104, "mds_3": 6618 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:21:20.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:20.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.343+0000 7f3e38309700 1 Processor -- start 2023-10-10T19:21:20.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.343+0000 7f3e38309700 1 -- start start 2023-10-10T19:21:20.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e38309700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e300b1fc0 0x7f3e30155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:20.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e38309700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3e300b2390 0x7f3e30068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:20.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e38309700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e301574b0 con 0x7f3e30068bd0 2023-10-10T19:21:20.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e38309700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e30150770 con 0x7f3e300b2390 2023-10-10T19:21:20.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e38309700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300afad0 con 0x7f3e300b1fc0 2023-10-10T19:21:20.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e358a4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3e300b2390 0x7f3e30068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:20.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e368a6700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f3e30068bd0 0x7f3e300aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49730/0 (socket says 172.21.15.175:49730) 2023-10-10T19:21:20.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e358a4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3e300b2390 0x7f3e30068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59494/0 (socket says 172.21.15.175:59494) 2023-10-10T19:21:20.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e368a6700 1 -- 172.21.15.175:0/189703244 learned_addr learned my addr 172.21.15.175:0/189703244 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:20.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e360a5700 1 --2- 172.21.15.175:0/189703244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e300b1fc0 0x7f3e30155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:20.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e358a4700 1 -- 172.21.15.175:0/189703244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e300b1fc0 msgr2=0x7f3e30155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:20.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e358a4700 1 --2- 172.21.15.175:0/189703244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e300b1fc0 0x7f3e30155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e358a4700 1 -- 172.21.15.175:0/189703244 >> v1:172.21.15.175:6789/0 conn(0x7f3e30068bd0 legacy=0x7f3e300aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:20.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e358a4700 1 -- 172.21.15.175:0/189703244 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e300af4c0 con 0x7f3e300b2390 2023-10-10T19:21:20.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e360a5700 1 --2- 172.21.15.175:0/189703244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e300b1fc0 0x7f3e30155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:21:20.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e350a3700 1 -- 172.21.15.175:0/189703244 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599965085 0 0) 0x7f3e301574b0 con 0x7f3e30068bd0 2023-10-10T19:21:20.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.344+0000 7f3e358a4700 1 --2- 172.21.15.175:0/189703244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e30068690 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3e200098e0 tx=0x7f3e200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413a45632c5b011f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:20.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e350a3700 1 -- 172.21.15.175:0/189703244 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20009e80 con 0x7f3e300b2390 2023-10-10T19:21:20.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e350a3700 1 -- 172.21.15.175:0/189703244 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2000b8f0 con 0x7f3e300b2390 2023-10-10T19:21:20.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e350a3700 1 -- 172.21.15.175:0/189703244 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000bc70 con 0x7f3e300b2390 2023-10-10T19:21:20.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e38309700 1 -- 172.21.15.175:0/189703244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 msgr2=0x7f3e30068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:20.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e38309700 1 --2- 172.21.15.175:0/189703244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e30068690 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3e200098e0 tx=0x7f3e200091a0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e38309700 1 -- 172.21.15.175:0/189703244 shutdown_connections 2023-10-10T19:21:20.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e38309700 1 --2- 172.21.15.175:0/189703244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e300b1fc0 0x7f3e30155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e38309700 1 --2- 172.21.15.175:0/189703244 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e30068690 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e38309700 1 -- 172.21.15.175:0/189703244 >> 172.21.15.175:0/189703244 conn(0x7f3e30063230 msgr2=0x7f3e30063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:20.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e38309700 1 -- 172.21.15.175:0/189703244 shutdown_connections 2023-10-10T19:21:20.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.345+0000 7f3e38309700 1 -- 172.21.15.175:0/189703244 wait complete. 2023-10-10T19:21:20.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.346+0000 7f3e38309700 1 Processor -- start 2023-10-10T19:21:20.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.346+0000 7f3e38309700 1 -- start start 2023-10-10T19:21:20.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e38309700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e30068bd0 0x7f3e300cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:20.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e38309700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e300b1fc0 0x7f3e30165420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:20.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e38309700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e300c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:20.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e38309700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300cc260 con 0x7f3e300b2390 2023-10-10T19:21:20.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e38309700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300ca5a0 con 0x7f3e300b1fc0 2023-10-10T19:21:20.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e38309700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300ca840 con 0x7f3e30068bd0 2023-10-10T19:21:20.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e368a6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e300c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:20.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e368a6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e300c6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59498/0 (socket says 172.21.15.175:59498) 2023-10-10T19:21:20.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e368a6700 1 -- 172.21.15.175:0/7763378 learned_addr learned my addr 172.21.15.175:0/7763378 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:20.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e360a5700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e30068bd0 0x7f3e300cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:20.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e358a4700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e300b1fc0 0x7f3e30165420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:20.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e368a6700 1 -- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e30068bd0 msgr2=0x7f3e300cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:20.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e368a6700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e30068bd0 0x7f3e300cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e368a6700 1 -- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e300b1fc0 msgr2=0x7f3e30165420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:20.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e368a6700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e300b1fc0 0x7f3e30165420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.347+0000 7f3e368a6700 1 -- 172.21.15.175:0/7763378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e20009420 con 0x7f3e300b2390 2023-10-10T19:21:20.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.348+0000 7f3e368a6700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e300c6060 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800ccc0 tx=0x7f3e28001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:20.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.348+0000 7f3e267fc700 1 -- 172.21.15.175:0/7763378 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800a910 con 0x7f3e300b2390 2023-10-10T19:21:20.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.348+0000 7f3e267fc700 1 -- 172.21.15.175:0/7763378 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2800aa70 con 0x7f3e300b2390 2023-10-10T19:21:20.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.348+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e300cab20 con 0x7f3e300b2390 2023-10-10T19:21:20.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.348+0000 7f3e267fc700 1 -- 172.21.15.175:0/7763378 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800ae10 con 0x7f3e300b2390 2023-10-10T19:21:20.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.348+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e300cbef0 con 0x7f3e300b2390 2023-10-10T19:21:20.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.349+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e30067570 con 0x7f3e300b2390 2023-10-10T19:21:20.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.349+0000 7f3e267fc700 1 -- 172.21.15.175:0/7763378 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3e28007300 con 0x7f3e300b2390 2023-10-10T19:21:20.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.349+0000 7f3e267fc700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e14066f30 0x7f3e140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:20.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.350+0000 7f3e267fc700 1 -- 172.21.15.175:0/7763378 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f3e28084f40 con 0x7f3e300b2390 2023-10-10T19:21:20.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.350+0000 7f3e360a5700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e14066f30 0x7f3e140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:20.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.350+0000 7f3e360a5700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e14066f30 0x7f3e140693e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00adc0 tx=0x7f3e2c006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:20.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.354+0000 7f3e267fc700 1 -- 172.21.15.175:0/7763378 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3e28051bb0 con 0x7f3e300b2390 2023-10-10T19:21:20.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.582+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e30069060 con 0x7f3e300b2390 2023-10-10T19:21:20.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.582+0000 7f3e267fc700 1 -- 172.21.15.175:0/7763378 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 274 v274) v1 ==== 95+0+5423 (secure 0 0 0) 0x7f3e280551d0 con 0x7f3e300b2390 2023-10-10T19:21:20.585 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:20.585 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":274,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":274,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:18.079855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:20.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e14066f30 msgr2=0x7f3e140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:20.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e14066f30 0x7f3e140693e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00adc0 tx=0x7f3e2c006f90 comp rx=0 tx=0).stop 2023-10-10T19:21:20.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 msgr2=0x7f3e300c6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:20.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e300c6060 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800ccc0 tx=0x7f3e28001a20 comp rx=0 tx=0).stop 2023-10-10T19:21:20.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 shutdown_connections 2023-10-10T19:21:20.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e14066f30 0x7f3e140693e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e300b2390 0x7f3e300c6060 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e30068bd0 0x7f3e300cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.584+0000 7f3e38309700 1 --2- 172.21.15.175:0/7763378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e300b1fc0 0x7f3e30165420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:20.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.585+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 >> 172.21.15.175:0/7763378 conn(0x7f3e30063230 msgr2=0x7f3e30152620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:20.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.585+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 shutdown_connections 2023-10-10T19:21:20.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:20.585+0000 7f3e38309700 1 -- 172.21.15.175:0/7763378 wait complete. 2023-10-10T19:21:20.592 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 274 2023-10-10T19:21:20.598 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:21:22.600 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:22.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.725+0000 7f81f9a00700 1 Processor -- start 2023-10-10T19:21:22.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f9a00700 1 -- start start 2023-10-10T19:21:22.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f9a00700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f81f414fd90 0x7f81f4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:22.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f9a00700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f81f40cd040 0x7f81f40684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:22.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f9a00700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f4067fa0 con 0x7f81f408a100 2023-10-10T19:21:22.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f9a00700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f40c5600 con 0x7f81f414fd90 2023-10-10T19:21:22.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f9a00700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f4156b70 con 0x7f81f40cd040 2023-10-10T19:21:22.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f89fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f81f414fd90 0x7f81f4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:22.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f3fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f81f408a100 0x7f81f40c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49736/0 (socket says 172.21.15.175:49736) 2023-10-10T19:21:22.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f89fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f81f414fd90 0x7f81f4159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59510/0 (socket says 172.21.15.175:59510) 2023-10-10T19:21:22.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.726+0000 7f81f3fff700 1 -- 172.21.15.175:0/1222298407 learned_addr learned my addr 172.21.15.175:0/1222298407 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:22.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f91ff700 1 --2- 172.21.15.175:0/1222298407 >> v2:172.21.15.191:3300/0 conn(0x7f81f40cd040 0x7f81f40684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f89fe700 1 -- 172.21.15.175:0/1222298407 >> v2:172.21.15.191:3300/0 conn(0x7f81f40cd040 msgr2=0x7f81f40684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f89fe700 1 --2- 172.21.15.175:0/1222298407 >> v2:172.21.15.191:3300/0 conn(0x7f81f40cd040 0x7f81f40684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f89fe700 1 -- 172.21.15.175:0/1222298407 >> v1:172.21.15.175:6789/0 conn(0x7f81f408a100 legacy=0x7f81f40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f89fe700 1 -- 172.21.15.175:0/1222298407 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4150160 con 0x7f81f414fd90 2023-10-10T19:21:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f37fe700 1 -- 172.21.15.175:0/1222298407 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185987471 0 0) 0x7f81f4067fa0 con 0x7f81f408a100 2023-10-10T19:21:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f89fe700 1 --2- 172.21.15.175:0/1222298407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4159c30 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f81f408a6e0 tx=0x7f81dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34cc3eefc76a4c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f37fe700 1 -- 172.21.15.175:0/1222298407 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00b740 con 0x7f81f414fd90 2023-10-10T19:21:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f37fe700 1 -- 172.21.15.175:0/1222298407 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc00b8a0 con 0x7f81f414fd90 2023-10-10T19:21:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.727+0000 7f81f37fe700 1 -- 172.21.15.175:0/1222298407 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00bc40 con 0x7f81f414fd90 2023-10-10T19:21:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.728+0000 7f81f9a00700 1 -- 172.21.15.175:0/1222298407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 msgr2=0x7f81f4159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.728+0000 7f81f9a00700 1 --2- 172.21.15.175:0/1222298407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4159c30 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f81f408a6e0 tx=0x7f81dc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.728+0000 7f81f9a00700 1 -- 172.21.15.175:0/1222298407 shutdown_connections 2023-10-10T19:21:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.728+0000 7f81f9a00700 1 --2- 172.21.15.175:0/1222298407 >> v2:172.21.15.191:3300/0 conn(0x7f81f40cd040 0x7f81f40684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.728+0000 7f81f9a00700 1 --2- 172.21.15.175:0/1222298407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4159c30 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.728+0000 7f81f9a00700 1 -- 172.21.15.175:0/1222298407 >> 172.21.15.175:0/1222298407 conn(0x7f81f4063d00 msgr2=0x7f81f40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:22.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.728+0000 7f81f9a00700 1 -- 172.21.15.175:0/1222298407 shutdown_connections 2023-10-10T19:21:22.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.728+0000 7f81f9a00700 1 -- 172.21.15.175:0/1222298407 wait complete. 2023-10-10T19:21:22.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.729+0000 7f81f9a00700 1 Processor -- start 2023-10-10T19:21:22.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.729+0000 7f81f9a00700 1 -- start start 2023-10-10T19:21:22.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f9a00700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81f408a100 0x7f81f4153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:22.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f9a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81f40cd040 0x7f81f4149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:22.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f9a00700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:22.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f9a00700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40c6c80 con 0x7f81f414fd90 2023-10-10T19:21:22.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f9a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40c74e0 con 0x7f81f40cd040 2023-10-10T19:21:22.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f9a00700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f4168f80 con 0x7f81f408a100 2023-10-10T19:21:22.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:22.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59520/0 (socket says 172.21.15.175:59520) 2023-10-10T19:21:22.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 -- 172.21.15.175:0/3949787139 learned_addr learned my addr 172.21.15.175:0/3949787139 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:22.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f89fe700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81f408a100 0x7f81f4153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:22.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f3fff700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81f40cd040 0x7f81f4149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:22.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 -- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81f408a100 msgr2=0x7f81f4153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:22.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81f408a100 0x7f81f4153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 -- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81f40cd040 msgr2=0x7f81f4149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:22.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81f40cd040 0x7f81f4149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 -- 172.21.15.175:0/3949787139 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc009420 con 0x7f81f414fd90 2023-10-10T19:21:22.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f3fff700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81f40cd040 0x7f81f4149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:21:22.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f91ff700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4149e80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f81ec00d8c0 tx=0x7f81ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:22.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f0ff9700 1 -- 172.21.15.175:0/3949787139 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec00b5d0 con 0x7f81f414fd90 2023-10-10T19:21:22.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f0ff9700 1 -- 172.21.15.175:0/3949787139 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ec00b750 con 0x7f81f414fd90 2023-10-10T19:21:22.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f41547d0 con 0x7f81f414fd90 2023-10-10T19:21:22.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f0ff9700 1 -- 172.21.15.175:0/3949787139 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec00bb10 con 0x7f81f414fd90 2023-10-10T19:21:22.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.730+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f40c7200 con 0x7f81f414fd90 2023-10-10T19:21:22.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.731+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f40697d0 con 0x7f81f414fd90 2023-10-10T19:21:22.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.732+0000 7f81f0ff9700 1 -- 172.21.15.175:0/3949787139 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f81ec01e460 con 0x7f81f414fd90 2023-10-10T19:21:22.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.732+0000 7f81f0ff9700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81d4066f30 0x7f81d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:22.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.732+0000 7f81f0ff9700 1 -- 172.21.15.175:0/3949787139 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f81ec0851a0 con 0x7f81f414fd90 2023-10-10T19:21:22.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.733+0000 7f81f89fe700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81d4066f30 0x7f81d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:22.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.733+0000 7f81f89fe700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81d4066f30 0x7f81d40693e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f81dc003080 tx=0x7f81dc017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:22.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.737+0000 7f81f0ff9700 1 -- 172.21.15.175:0/3949787139 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81ec051d90 con 0x7f81f414fd90 2023-10-10T19:21:22.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.965+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81f4067570 con 0x7f81f414fd90 2023-10-10T19:21:22.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.965+0000 7f81f0ff9700 1 -- 172.21.15.175:0/3949787139 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 274 v274) v1 ==== 95+0+5423 (secure 0 0 0) 0x7f81ec0553b0 con 0x7f81f414fd90 2023-10-10T19:21:22.967 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:22.967 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":274,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":274,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:18.079855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:22.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81d4066f30 msgr2=0x7f81d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:22.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81d4066f30 0x7f81d40693e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f81dc003080 tx=0x7f81dc017000 comp rx=0 tx=0).stop 2023-10-10T19:21:22.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 msgr2=0x7f81f4149e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:22.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4149e80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f81ec00d8c0 tx=0x7f81ec008040 comp rx=0 tx=0).stop 2023-10-10T19:21:22.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 shutdown_connections 2023-10-10T19:21:22.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f81d4066f30 0x7f81d40693e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81f414fd90 0x7f81f4149e80 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81f408a100 0x7f81f4153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 --2- 172.21.15.175:0/3949787139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81f40cd040 0x7f81f4149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:22.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 >> 172.21.15.175:0/3949787139 conn(0x7f81f4063d00 msgr2=0x7f81f414ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:22.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.967+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 shutdown_connections 2023-10-10T19:21:22.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:22.968+0000 7f81f9a00700 1 -- 172.21.15.175:0/3949787139 wait complete. 2023-10-10T19:21:22.971 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 274 2023-10-10T19:21:22.981 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:21:24.982 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:25.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.108+0000 7f659ccd9700 1 Processor -- start 2023-10-10T19:21:25.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.108+0000 7f659ccd9700 1 -- start start 2023-10-10T19:21:25.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f659ccd9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f65981595a0 0x7f659806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:25.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f659ccd9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65980b0240 0x7f65980c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:25.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f659ccd9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659806a8c0 con 0x7f659806bda0 2023-10-10T19:21:25.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f659ccd9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6598159fb0 con 0x7f65981595a0 2023-10-10T19:21:25.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f659ccd9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65980c5ab0 con 0x7f65980b0240 2023-10-10T19:21:25.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f6597fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65980b0240 0x7f65980c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:25.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f65977fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f65981595a0 0x7f659806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:25.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f6597fff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65980b0240 0x7f65980c0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:35346/0 (socket says 172.21.15.175:35346) 2023-10-10T19:21:25.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f65977fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f65981595a0 0x7f659806dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:56984/0 (socket says 172.21.15.175:56984) 2023-10-10T19:21:25.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f6597fff700 1 -- 172.21.15.175:0/2686280008 learned_addr learned my addr 172.21.15.175:0/2686280008 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:25.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.109+0000 7f658ffff700 1 --1- 172.21.15.175:0/2686280008 >> v1:172.21.15.191:6789/0 conn(0x7f659806bda0 0x7f65980afb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:57204/0 (socket says 172.21.15.175:57204) 2023-10-10T19:21:25.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6596ffd700 1 -- 172.21.15.175:0/2686280008 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909474112 0 0) 0x7f659806a8c0 con 0x7f659806bda0 2023-10-10T19:21:25.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6596ffd700 1 -- 172.21.15.175:0/2686280008 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6584003570 con 0x7f659806bda0 2023-10-10T19:21:25.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6597fff700 1 -- 172.21.15.175:0/2686280008 >> v2:172.21.15.191:3300/0 conn(0x7f65981595a0 msgr2=0x7f659806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:25.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6597fff700 1 --2- 172.21.15.175:0/2686280008 >> v2:172.21.15.191:3300/0 conn(0x7f65981595a0 0x7f659806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6597fff700 1 -- 172.21.15.175:0/2686280008 >> v1:172.21.15.191:6789/0 conn(0x7f659806bda0 legacy=0x7f65980afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:25.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6597fff700 1 -- 172.21.15.175:0/2686280008 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980c57d0 con 0x7f65980b0240 2023-10-10T19:21:25.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f65977fe700 1 --2- 172.21.15.175:0/2686280008 >> v2:172.21.15.191:3300/0 conn(0x7f65981595a0 0x7f659806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:21:25.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6597fff700 1 --2- 172.21.15.175:0/2686280008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65980b0240 0x7f65980c0ff0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6590009f20 tx=0x7f6590009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bb9580f7ba41321 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:25.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6596ffd700 1 -- 172.21.15.175:0/2686280008 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659000e070 con 0x7f65980b0240 2023-10-10T19:21:25.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6596ffd700 1 -- 172.21.15.175:0/2686280008 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659000ba00 con 0x7f65980b0240 2023-10-10T19:21:25.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.110+0000 7f6596ffd700 1 -- 172.21.15.175:0/2686280008 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65900092e0 con 0x7f65980b0240 2023-10-10T19:21:25.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.111+0000 7f659ccd9700 1 -- 172.21.15.175:0/2686280008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65980b0240 msgr2=0x7f65980c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:25.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.111+0000 7f659ccd9700 1 --2- 172.21.15.175:0/2686280008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65980b0240 0x7f65980c0ff0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6590009f20 tx=0x7f6590009840 comp rx=0 tx=0).stop 2023-10-10T19:21:25.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.111+0000 7f659ccd9700 1 -- 172.21.15.175:0/2686280008 shutdown_connections 2023-10-10T19:21:25.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.111+0000 7f659ccd9700 1 --2- 172.21.15.175:0/2686280008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65980b0240 0x7f65980c0ff0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.111+0000 7f659ccd9700 1 --2- 172.21.15.175:0/2686280008 >> v2:172.21.15.191:3300/0 conn(0x7f65981595a0 0x7f659806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.111+0000 7f659ccd9700 1 -- 172.21.15.175:0/2686280008 >> 172.21.15.175:0/2686280008 conn(0x7f6598063cd0 msgr2=0x7f659806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:25.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.111+0000 7f659ccd9700 1 -- 172.21.15.175:0/2686280008 shutdown_connections 2023-10-10T19:21:25.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.111+0000 7f659ccd9700 1 -- 172.21.15.175:0/2686280008 wait complete. 2023-10-10T19:21:25.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.112+0000 7f659ccd9700 1 Processor -- start 2023-10-10T19:21:25.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.112+0000 7f659ccd9700 1 -- start start 2023-10-10T19:21:25.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.112+0000 7f659ccd9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f659806bda0 0x7f6598155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:25.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.112+0000 7f659ccd9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65980b0240 0x7f6598151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:25.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f659ccd9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65981595a0 0x7f6598151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:25.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f659ccd9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6598151620 con 0x7f65980b0240 2023-10-10T19:21:25.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f659ccd9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980afe90 con 0x7f659806bda0 2023-10-10T19:21:25.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f659ccd9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65980c9990 con 0x7f65981595a0 2023-10-10T19:21:25.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65980b0240 0x7f6598151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:25.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65980b0240 0x7f6598151ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34948/0 (socket says 172.21.15.175:34948) 2023-10-10T19:21:25.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 -- 172.21.15.175:0/3091744657 learned_addr learned my addr 172.21.15.175:0/3091744657 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:25.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f65977fe700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f659806bda0 0x7f6598155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:25.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f6597fff700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65981595a0 0x7f6598151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:25.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 -- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65981595a0 msgr2=0x7f6598151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:25.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65981595a0 0x7f6598151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 -- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f659806bda0 msgr2=0x7f6598155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:25.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f659806bda0 0x7f6598155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 -- 172.21.15.175:0/3091744657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6588009420 con 0x7f65980b0240 2023-10-10T19:21:25.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658ffff700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65980b0240 0x7f6598151ab0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f658000c590 tx=0x7f6580007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:25.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658f7fe700 1 -- 172.21.15.175:0/3091744657 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580005a70 con 0x7f65980b0240 2023-10-10T19:21:25.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658f7fe700 1 -- 172.21.15.175:0/3091744657 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6580005bd0 con 0x7f65980b0240 2023-10-10T19:21:25.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6590009ae0 con 0x7f65980b0240 2023-10-10T19:21:25.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f658f7fe700 1 -- 172.21.15.175:0/3091744657 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65800070d0 con 0x7f65980b0240 2023-10-10T19:21:25.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.113+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65980afbb0 con 0x7f65980b0240 2023-10-10T19:21:25.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.114+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6598067570 con 0x7f65980b0240 2023-10-10T19:21:25.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.115+0000 7f658f7fe700 1 -- 172.21.15.175:0/3091744657 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6580007440 con 0x7f65980b0240 2023-10-10T19:21:25.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.115+0000 7f658f7fe700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6578066f30 0x7f65780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:25.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.115+0000 7f658f7fe700 1 -- 172.21.15.175:0/3091744657 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f65800841e0 con 0x7f65980b0240 2023-10-10T19:21:25.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.115+0000 7f65977fe700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6578066f30 0x7f65780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:25.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.116+0000 7f65977fe700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6578066f30 0x7f65780693e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f65880093f0 tx=0x7f6588008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:25.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.120+0000 7f658f7fe700 1 -- 172.21.15.175:0/3091744657 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6580050dd0 con 0x7f65980b0240 2023-10-10T19:21:25.155 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:21:25.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:21:25.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.348+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6598065140 con 0x7f65980b0240 2023-10-10T19:21:25.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.348+0000 7f658f7fe700 1 -- 172.21.15.175:0/3091744657 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 275 v275) v1 ==== 95+0+5420 (secure 0 0 0) 0x7f65800543f0 con 0x7f65980b0240 2023-10-10T19:21:25.350 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:25.350 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":275,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":275,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:23.100858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:25.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6578066f30 msgr2=0x7f65780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:25.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6578066f30 0x7f65780693e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f65880093f0 tx=0x7f6588008040 comp rx=0 tx=0).stop 2023-10-10T19:21:25.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65980b0240 msgr2=0x7f6598151ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:25.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65980b0240 0x7f6598151ab0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f658000c590 tx=0x7f6580007d60 comp rx=0 tx=0).stop 2023-10-10T19:21:25.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 shutdown_connections 2023-10-10T19:21:25.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6578066f30 0x7f65780693e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65981595a0 0x7f6598151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f659806bda0 0x7f6598155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 --2- 172.21.15.175:0/3091744657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65980b0240 0x7f6598151ab0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:25.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.350+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 >> 172.21.15.175:0/3091744657 conn(0x7f6598063cd0 msgr2=0x7f65980c84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:25.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.351+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 shutdown_connections 2023-10-10T19:21:25.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:25.351+0000 7f659ccd9700 1 -- 172.21.15.175:0/3091744657 wait complete. 2023-10-10T19:21:25.355 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 275 2023-10-10T19:21:25.364 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:21:27.364 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:27.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.490+0000 7f1ba4daf700 1 Processor -- start 2023-10-10T19:21:27.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.490+0000 7f1ba4daf700 1 -- start start 2023-10-10T19:21:27.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1ba4daf700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1ba00b2390 0x7f1ba0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:27.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1ba4daf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1ba00aff60 0x7f1ba00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:27.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1ba4daf700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba01574b0 con 0x7f1ba00b1fc0 2023-10-10T19:21:27.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1ba4daf700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba0150770 con 0x7f1ba00aff60 2023-10-10T19:21:27.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1ba4daf700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba00afad0 con 0x7f1ba00b2390 2023-10-10T19:21:27.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1b9ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1ba00aff60 0x7f1ba00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:27.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1b9ed9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1ba00aff60 0x7f1ba00b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34956/0 (socket says 172.21.15.175:34956) 2023-10-10T19:21:27.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1b9ed9d700 1 -- 172.21.15.175:0/1600673044 learned_addr learned my addr 172.21.15.175:0/1600673044 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:27.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.491+0000 7f1b9dd9b700 1 --2- 172.21.15.175:0/1600673044 >> v2:172.21.15.191:3300/0 conn(0x7f1ba00b2390 0x7f1ba0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:27.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1b9ed9d700 1 -- 172.21.15.175:0/1600673044 >> v2:172.21.15.191:3300/0 conn(0x7f1ba00b2390 msgr2=0x7f1ba0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:27.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1b9ed9d700 1 --2- 172.21.15.175:0/1600673044 >> v2:172.21.15.191:3300/0 conn(0x7f1ba00b2390 0x7f1ba0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1b9ed9d700 1 -- 172.21.15.175:0/1600673044 >> v1:172.21.15.191:6789/0 conn(0x7f1ba00b1fc0 legacy=0x7f1ba0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:27.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1b9ed9d700 1 -- 172.21.15.175:0/1600673044 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba00af4c0 con 0x7f1ba00aff60 2023-10-10T19:21:27.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1b9ed9d700 1 --2- 172.21.15.175:0/1600673044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba00b03e0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00c0290 tx=0x7f1b90009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af42f690ed8a39e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:27.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1b9d59a700 1 -- 172.21.15.175:0/1600673044 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000e070 con 0x7f1ba00aff60 2023-10-10T19:21:27.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1b9d59a700 1 -- 172.21.15.175:0/1600673044 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9000bab0 con 0x7f1ba00aff60 2023-10-10T19:21:27.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1b9d59a700 1 -- 172.21.15.175:0/1600673044 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b900092e0 con 0x7f1ba00aff60 2023-10-10T19:21:27.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1ba4daf700 1 -- 172.21.15.175:0/1600673044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 msgr2=0x7f1ba00b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:27.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.492+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/1600673044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba00b03e0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00c0290 tx=0x7f1b90009840 comp rx=0 tx=0).stop 2023-10-10T19:21:27.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.493+0000 7f1ba4daf700 1 -- 172.21.15.175:0/1600673044 shutdown_connections 2023-10-10T19:21:27.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.493+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/1600673044 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba00b03e0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.493+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/1600673044 >> v2:172.21.15.191:3300/0 conn(0x7f1ba00b2390 0x7f1ba0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.493+0000 7f1ba4daf700 1 -- 172.21.15.175:0/1600673044 >> 172.21.15.175:0/1600673044 conn(0x7f1ba0063230 msgr2=0x7f1ba0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:27.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.493+0000 7f1ba4daf700 1 -- 172.21.15.175:0/1600673044 shutdown_connections 2023-10-10T19:21:27.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.493+0000 7f1ba4daf700 1 -- 172.21.15.175:0/1600673044 wait complete. 2023-10-10T19:21:27.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.494+0000 7f1ba4daf700 1 Processor -- start 2023-10-10T19:21:27.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.494+0000 7f1ba4daf700 1 -- start start 2023-10-10T19:21:27.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.494+0000 7f1ba4daf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba01559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:27.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.494+0000 7f1ba4daf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ba00b1fc0 0x7f1ba01563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:27.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.494+0000 7f1ba4daf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ba00b2390 0x7f1ba00c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:27.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.494+0000 7f1ba4daf700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00cac50 con 0x7f1ba00aff60 2023-10-10T19:21:27.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.494+0000 7f1ba4daf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00c5930 con 0x7f1ba00b2390 2023-10-10T19:21:27.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.494+0000 7f1ba4daf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00c67f0 con 0x7f1ba00b1fc0 2023-10-10T19:21:27.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba01559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:27.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba01559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34960/0 (socket says 172.21.15.175:34960) 2023-10-10T19:21:27.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 -- 172.21.15.175:0/4023758655 learned_addr learned my addr 172.21.15.175:0/4023758655 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:27.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9ed9d700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ba00b2390 0x7f1ba00c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:27.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9dd9b700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ba00b1fc0 0x7f1ba01563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:27.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 -- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ba00b1fc0 msgr2=0x7f1ba01563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:27.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ba00b1fc0 0x7f1ba01563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 -- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ba00b2390 msgr2=0x7f1ba00c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:27.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ba00b2390 0x7f1ba00c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 -- 172.21.15.175:0/4023758655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b90009ae0 con 0x7f1ba00aff60 2023-10-10T19:21:27.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b9e59c700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba01559b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800c7c0 tx=0x7f1b98007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:27.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b967fc700 1 -- 172.21.15.175:0/4023758655 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98011590 con 0x7f1ba00aff60 2023-10-10T19:21:27.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b967fc700 1 -- 172.21.15.175:0/4023758655 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b98011710 con 0x7f1ba00aff60 2023-10-10T19:21:27.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba0169240 con 0x7f1ba00aff60 2023-10-10T19:21:27.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1b967fc700 1 -- 172.21.15.175:0/4023758655 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98011ad0 con 0x7f1ba00aff60 2023-10-10T19:21:27.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.495+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba00ca970 con 0x7f1ba00aff60 2023-10-10T19:21:27.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.496+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba00713a0 con 0x7f1ba00aff60 2023-10-10T19:21:27.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.497+0000 7f1b967fc700 1 -- 172.21.15.175:0/4023758655 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1b9801b3b0 con 0x7f1ba00aff60 2023-10-10T19:21:27.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.497+0000 7f1b967fc700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b80066f30 0x7f1b800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:27.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.497+0000 7f1b967fc700 1 -- 172.21.15.175:0/4023758655 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f1b98083fd0 con 0x7f1ba00aff60 2023-10-10T19:21:27.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.497+0000 7f1b9dd9b700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b80066f30 0x7f1b800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:27.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.497+0000 7f1b9dd9b700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b80066f30 0x7f1b800693e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00c0260 tx=0x7f1b880091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:27.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.501+0000 7f1b967fc700 1 -- 172.21.15.175:0/4023758655 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ba00713a0 con 0x7f1ba00aff60 2023-10-10T19:21:27.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.726+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba00713a0 con 0x7f1ba00aff60 2023-10-10T19:21:27.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.727+0000 7f1b967fc700 1 -- 172.21.15.175:0/4023758655 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 275 v275) v1 ==== 95+0+5420 (secure 0 0 0) 0x7f1ba00713a0 con 0x7f1ba00aff60 2023-10-10T19:21:27.729 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:27.729 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":275,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":275,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:23.100858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:27.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b80066f30 msgr2=0x7f1b800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:27.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b80066f30 0x7f1b800693e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00c0260 tx=0x7f1b880091a0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 msgr2=0x7f1ba01559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:27.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba01559b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800c7c0 tx=0x7f1b98007f40 comp rx=0 tx=0).stop 2023-10-10T19:21:27.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 shutdown_connections 2023-10-10T19:21:27.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1b80066f30 0x7f1b800693e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ba00b2390 0x7f1ba00c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1ba00aff60 0x7f1ba01559b0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 --2- 172.21.15.175:0/4023758655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ba00b1fc0 0x7f1ba01563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:27.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 >> 172.21.15.175:0/4023758655 conn(0x7f1ba0063230 msgr2=0x7f1ba0158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:27.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 shutdown_connections 2023-10-10T19:21:27.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:27.729+0000 7f1ba4daf700 1 -- 172.21.15.175:0/4023758655 wait complete. 2023-10-10T19:21:27.733 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 275 2023-10-10T19:21:27.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:21:29.743 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:29.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.869+0000 7fc67267e700 1 Processor -- start 2023-10-10T19:21:29.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.869+0000 7fc67267e700 1 -- start start 2023-10-10T19:21:29.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.870+0000 7fc67267e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c06f5f0 0x7fc66c156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:29.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.870+0000 7fc67267e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c0c9cc0 con 0x7fc66c0c64a0 2023-10-10T19:21:29.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.870+0000 7fc67267e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c06e650 con 0x7fc66c0c60d0 2023-10-10T19:21:29.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.870+0000 7fc67267e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0c4500 con 0x7fc66c06f5f0 2023-10-10T19:21:29.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.870+0000 7fc670c1b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc66c0c64a0 0x7fc66c14cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40854/0 (socket says 172.21.15.175:40854) 2023-10-10T19:21:29.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.870+0000 7fc670c1b700 1 -- 172.21.15.175:0/1339439599 learned_addr learned my addr 172.21.15.175:0/1339439599 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:29.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.870+0000 7fc66b7fe700 1 --2- 172.21.15.175:0/1339439599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c06f5f0 0x7fc66c156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:29.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66affd700 1 -- 172.21.15.175:0/1339439599 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112560512 0 0) 0x7fc66c0c9cc0 con 0x7fc66c0c64a0 2023-10-10T19:21:29.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66affd700 1 -- 172.21.15.175:0/1339439599 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc654003570 con 0x7fc66c0c64a0 2023-10-10T19:21:29.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66affd700 1 -- 172.21.15.175:0/1339439599 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086986605 0 0) 0x7fc66c06e650 con 0x7fc66c0c60d0 2023-10-10T19:21:29.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66affd700 1 -- 172.21.15.175:0/1339439599 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc66c0c9cc0 con 0x7fc66c0c60d0 2023-10-10T19:21:29.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66b7fe700 1 -- 172.21.15.175:0/1339439599 >> v1:172.21.15.191:6789/0 conn(0x7fc66c0c60d0 legacy=0x7fc66c0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:29.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66b7fe700 1 -- 172.21.15.175:0/1339439599 >> v1:172.21.15.175:6789/0 conn(0x7fc66c0c64a0 legacy=0x7fc66c14cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:29.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66b7fe700 1 -- 172.21.15.175:0/1339439599 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c06e930 con 0x7fc66c06f5f0 2023-10-10T19:21:29.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66b7fe700 1 --2- 172.21.15.175:0/1339439599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c06f5f0 0x7fc66c156c80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc660009910 tx=0x7fc6600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5113df199ae9e3a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:29.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66affd700 1 -- 172.21.15.175:0/1339439599 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc660011630 con 0x7fc66c06f5f0 2023-10-10T19:21:29.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66affd700 1 -- 172.21.15.175:0/1339439599 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc660011790 con 0x7fc66c06f5f0 2023-10-10T19:21:29.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.871+0000 7fc66affd700 1 -- 172.21.15.175:0/1339439599 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc660011b30 con 0x7fc66c06f5f0 2023-10-10T19:21:29.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.872+0000 7fc67267e700 1 -- 172.21.15.175:0/1339439599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c06f5f0 msgr2=0x7fc66c156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:29.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.872+0000 7fc67267e700 1 --2- 172.21.15.175:0/1339439599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c06f5f0 0x7fc66c156c80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc660009910 tx=0x7fc6600091a0 comp rx=0 tx=0).stop 2023-10-10T19:21:29.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.872+0000 7fc67267e700 1 -- 172.21.15.175:0/1339439599 shutdown_connections 2023-10-10T19:21:29.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.872+0000 7fc67267e700 1 --2- 172.21.15.175:0/1339439599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c06f5f0 0x7fc66c156c80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:29.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.872+0000 7fc67267e700 1 -- 172.21.15.175:0/1339439599 >> 172.21.15.175:0/1339439599 conn(0x7fc66c06c110 msgr2=0x7fc66c06c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:29.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.872+0000 7fc67267e700 1 -- 172.21.15.175:0/1339439599 shutdown_connections 2023-10-10T19:21:29.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.872+0000 7fc67267e700 1 -- 172.21.15.175:0/1339439599 wait complete. 2023-10-10T19:21:29.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.873+0000 7fc67267e700 1 Processor -- start 2023-10-10T19:21:29.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.873+0000 7fc67267e700 1 -- start start 2023-10-10T19:21:29.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc67267e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66c06f5f0 0x7fc66c14fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:29.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc67267e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c0c60d0 0x7fc66c14ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:29.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc67267e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc66c0c64a0 0x7fc66c14bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:29.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc67267e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c1571c0 con 0x7fc66c0c64a0 2023-10-10T19:21:29.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc67267e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c14f5e0 con 0x7fc66c06f5f0 2023-10-10T19:21:29.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc67267e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c14ed20 con 0x7fc66c0c60d0 2023-10-10T19:21:29.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc66c0c64a0 0x7fc66c14bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:29.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc66c0c64a0 0x7fc66c14bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34974/0 (socket says 172.21.15.175:34974) 2023-10-10T19:21:29.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 -- 172.21.15.175:0/541102446 learned_addr learned my addr 172.21.15.175:0/541102446 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:29.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc66bfff700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66c06f5f0 0x7fc66c14fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:29.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc66b7fe700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c0c60d0 0x7fc66c14ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:29.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 -- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c0c60d0 msgr2=0x7fc66c14ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:29.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c0c60d0 0x7fc66c14ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:29.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 -- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66c06f5f0 msgr2=0x7fc66c14fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:29.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66c06f5f0 0x7fc66c14fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:29.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 -- 172.21.15.175:0/541102446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc660009420 con 0x7fc66c0c64a0 2023-10-10T19:21:29.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc66bfff700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66c06f5f0 0x7fc66c14fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:21:29.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.874+0000 7fc670c1b700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc66c0c64a0 0x7fc66c14bcd0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fc66400cc80 tx=0x7fc664001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:29.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.875+0000 7fc653fff700 1 -- 172.21.15.175:0/541102446 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66400ce90 con 0x7fc66c0c64a0 2023-10-10T19:21:29.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.875+0000 7fc653fff700 1 -- 172.21.15.175:0/541102446 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66400aa90 con 0x7fc66c0c64a0 2023-10-10T19:21:29.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.875+0000 7fc653fff700 1 -- 172.21.15.175:0/541102446 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66400ae30 con 0x7fc66c0c64a0 2023-10-10T19:21:29.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.875+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66c0688e0 con 0x7fc66c0c64a0 2023-10-10T19:21:29.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.875+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66c14ef90 con 0x7fc66c0c64a0 2023-10-10T19:21:29.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.875+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66c067570 con 0x7fc66c0c64a0 2023-10-10T19:21:29.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.876+0000 7fc653fff700 1 -- 172.21.15.175:0/541102446 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc664007300 con 0x7fc66c0c64a0 2023-10-10T19:21:29.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.876+0000 7fc653fff700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc658066f60 0x7fc658069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:29.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.877+0000 7fc653fff700 1 -- 172.21.15.175:0/541102446 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7fc664084fd0 con 0x7fc66c0c64a0 2023-10-10T19:21:29.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.877+0000 7fc66bfff700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc658066f60 0x7fc658069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:29.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.877+0000 7fc66bfff700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc658066f60 0x7fc658069410 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc65c001120 tx=0x7fc65c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:29.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:29.881+0000 7fc653fff700 1 -- 172.21.15.175:0/541102446 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6640556d0 con 0x7fc66c0c64a0 2023-10-10T19:21:30.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.108+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc66c14f720 con 0x7fc66c0c64a0 2023-10-10T19:21:30.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.108+0000 7fc653fff700 1 -- 172.21.15.175:0/541102446 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 276 v276) v1 ==== 95+0+5420 (secure 0 0 0) 0x7fc664055260 con 0x7fc66c0c64a0 2023-10-10T19:21:30.110 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:30.110 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":276,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:30.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc658066f60 msgr2=0x7fc658069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:30.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc658066f60 0x7fc658069410 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc65c001120 tx=0x7fc65c009040 comp rx=0 tx=0).stop 2023-10-10T19:21:30.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc66c0c64a0 msgr2=0x7fc66c14bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:30.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc66c0c64a0 0x7fc66c14bcd0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fc66400cc80 tx=0x7fc664001a40 comp rx=0 tx=0).stop 2023-10-10T19:21:30.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 shutdown_connections 2023-10-10T19:21:30.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc658066f60 0x7fc658069410 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:30.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc66c0c64a0 0x7fc66c14bcd0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:30.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66c06f5f0 0x7fc66c14fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:30.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 --2- 172.21.15.175:0/541102446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66c0c60d0 0x7fc66c14ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:30.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 >> 172.21.15.175:0/541102446 conn(0x7fc66c06c110 msgr2=0x7fc66c0c89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:30.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 shutdown_connections 2023-10-10T19:21:30.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:30.110+0000 7fc67267e700 1 -- 172.21.15.175:0/541102446 wait complete. 2023-10-10T19:21:30.116 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 276 2023-10-10T19:21:30.124 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=6511 2023-10-10T19:21:30.124 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.d 2023-10-10T19:21:31.125 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-10T19:21:31.126 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-10T19:21:31.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-10T19:21:31.130 INFO:tasks.ceph.mds.d:Started 2023-10-10T19:21:31.130 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:31.211 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-10T19:21:31.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.253+0000 7efc77eb7700 1 Processor -- start 2023-10-10T19:21:31.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.254+0000 7efc77eb7700 1 -- start start 2023-10-10T19:21:31.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.254+0000 7efc77eb7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efc700b2390 0x7efc70155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:31.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.254+0000 7efc77eb7700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc701574b0 con 0x7efc700b1fc0 2023-10-10T19:21:31.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.254+0000 7efc77eb7700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc700afad0 con 0x7efc700aff60 2023-10-10T19:21:31.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.254+0000 7efc77eb7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc70150770 con 0x7efc700b2390 2023-10-10T19:21:31.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.254+0000 7efc75c53700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7efc700b1fc0 0x7efc70068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40864/0 (socket says 172.21.15.175:40864) 2023-10-10T19:21:31.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.254+0000 7efc75c53700 1 -- 172.21.15.175:0/2045735106 learned_addr learned my addr 172.21.15.175:0/2045735106 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:31.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc75452700 1 --2- 172.21.15.175:0/2045735106 >> v2:172.21.15.191:3300/0 conn(0x7efc700b2390 0x7efc70155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:31.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887849016 0 0) 0x7efc701574b0 con 0x7efc700b1fc0 2023-10-10T19:21:31.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc58003570 con 0x7efc700b1fc0 2023-10-10T19:21:31.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502083610 0 0) 0x7efc58003570 con 0x7efc700b1fc0 2023-10-10T19:21:31.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 >> v2:172.21.15.191:3300/0 conn(0x7efc700b2390 msgr2=0x7efc70155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:31.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 --2- 172.21.15.175:0/2045735106 >> v2:172.21.15.191:3300/0 conn(0x7efc700b2390 0x7efc70155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 >> v1:172.21.15.191:6789/0 conn(0x7efc700aff60 legacy=0x7efc70159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:31.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc700af4c0 con 0x7efc700b1fc0 2023-10-10T19:21:31.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7efc600017f0 con 0x7efc700b1fc0 2023-10-10T19:21:31.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 >> v1:172.21.15.175:6789/0 conn(0x7efc700b1fc0 legacy=0x7efc70068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:31.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc58003a60 0x7efc58003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:31.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58004810 0x7efc58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:31.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc580051c0 0x7efc5800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:31.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc70150770 con 0x7efc580051c0 2023-10-10T19:21:31.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5800c0d0 con 0x7efc58003a60 2023-10-10T19:21:31.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5800c290 con 0x7efc58004810 2023-10-10T19:21:31.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc75c53700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc58003a60 0x7efc58003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:31.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc76454700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc580051c0 0x7efc5800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:31.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc75452700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58004810 0x7efc58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:31.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc75c53700 1 -- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58004810 msgr2=0x7efc58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:31.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc75c53700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58004810 0x7efc58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc75c53700 1 -- 172.21.15.175:0/2045735106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc580051c0 msgr2=0x7efc5800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:31.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc75c53700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc580051c0 0x7efc5800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc75c53700 1 -- 172.21.15.175:0/2045735106 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc58003570 con 0x7efc58003a60 2023-10-10T19:21:31.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc76454700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc580051c0 0x7efc5800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:21:31.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.255+0000 7efc76454700 1 -- 172.21.15.175:0/2045735106 reap_dead start 2023-10-10T19:21:31.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc75c53700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc58003a60 0x7efc58003e70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efc6000a900 tx=0x7efc700af650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36f280fbfa1af881 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:31.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc600070d0 con 0x7efc58003a60 2023-10-10T19:21:31.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc60007230 con 0x7efc58003a60 2023-10-10T19:21:31.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc74c51700 1 -- 172.21.15.175:0/2045735106 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc600075f0 con 0x7efc58003a60 2023-10-10T19:21:31.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc77eb7700 1 -- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc58003a60 msgr2=0x7efc58003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:31.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc77eb7700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc58003a60 0x7efc58003e70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efc6000a900 tx=0x7efc700af650 comp rx=0 tx=0).stop 2023-10-10T19:21:31.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc77eb7700 1 -- 172.21.15.175:0/2045735106 shutdown_connections 2023-10-10T19:21:31.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc77eb7700 1 --2- 172.21.15.175:0/2045735106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc58003a60 0x7efc58003e70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc77eb7700 1 -- 172.21.15.175:0/2045735106 >> 172.21.15.175:0/2045735106 conn(0x7efc70063230 msgr2=0x7efc70063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:31.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc77eb7700 1 -- 172.21.15.175:0/2045735106 shutdown_connections 2023-10-10T19:21:31.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.256+0000 7efc77eb7700 1 -- 172.21.15.175:0/2045735106 wait complete. 2023-10-10T19:21:31.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.257+0000 7efc77eb7700 1 Processor -- start 2023-10-10T19:21:31.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.257+0000 7efc77eb7700 1 -- start start 2023-10-10T19:21:31.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc77eb7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58003a60 0x7efc70154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:31.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc77eb7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc700c5330 0x7efc700c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:31.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc77eb7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc70159370 0x7efc701597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:31.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc77eb7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc70156e00 con 0x7efc70159370 2023-10-10T19:21:31.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc77eb7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc70165e90 con 0x7efc700c5330 2023-10-10T19:21:31.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc77eb7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc700cbdc0 con 0x7efc58003a60 2023-10-10T19:21:31.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc76454700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc70159370 0x7efc701597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:31.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc76454700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc70159370 0x7efc701597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34992/0 (socket says 172.21.15.175:34992) 2023-10-10T19:21:31.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc76454700 1 -- 172.21.15.175:0/3447963661 learned_addr learned my addr 172.21.15.175:0/3447963661 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:31.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc75c53700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58003a60 0x7efc70154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:31.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc75452700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc700c5330 0x7efc700c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:31.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc76454700 1 -- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58003a60 msgr2=0x7efc70154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:31.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc76454700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58003a60 0x7efc70154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc76454700 1 -- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc700c5330 msgr2=0x7efc700c3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:31.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc76454700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc700c5330 0x7efc700c3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.258+0000 7efc76454700 1 -- 172.21.15.175:0/3447963661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc701666f0 con 0x7efc70159370 2023-10-10T19:21:31.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.259+0000 7efc76454700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc70159370 0x7efc701597a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7efc700b2760 tx=0x7efc68009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:31.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.259+0000 7efc6dffb700 1 -- 172.21.15.175:0/3447963661 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68018070 con 0x7efc70159370 2023-10-10T19:21:31.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.259+0000 7efc6dffb700 1 -- 172.21.15.175:0/3447963661 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc68006820 con 0x7efc70159370 2023-10-10T19:21:31.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.259+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc700cc010 con 0x7efc70159370 2023-10-10T19:21:31.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.259+0000 7efc6dffb700 1 -- 172.21.15.175:0/3447963661 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68006ba0 con 0x7efc70159370 2023-10-10T19:21:31.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.259+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc70156b60 con 0x7efc70159370 2023-10-10T19:21:31.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.260+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc70067570 con 0x7efc70159370 2023-10-10T19:21:31.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.260+0000 7efc6dffb700 1 -- 172.21.15.175:0/3447963661 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efc68007030 con 0x7efc70159370 2023-10-10T19:21:31.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.261+0000 7efc6dffb700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efc5c066f30 0x7efc5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:31.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.261+0000 7efc6dffb700 1 -- 172.21.15.175:0/3447963661 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7efc680844f0 con 0x7efc70159370 2023-10-10T19:21:31.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.261+0000 7efc75c53700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efc5c066f30 0x7efc5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:31.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.261+0000 7efc75c53700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efc5c066f30 0x7efc5c0693e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efc6000a3f0 tx=0x7efc60016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:31.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.265+0000 7efc6dffb700 1 -- 172.21.15.175:0/3447963661 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc680510e0 con 0x7efc70159370 2023-10-10T19:21:31.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.488+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc70069060 con 0x7efc70159370 2023-10-10T19:21:31.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.489+0000 7efc6dffb700 1 -- 172.21.15.175:0/3447963661 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 276 v276) v1 ==== 95+0+5420 (secure 0 0 0) 0x7efc68054700 con 0x7efc70159370 2023-10-10T19:21:31.491 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:31.491 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":276,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:31.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efc5c066f30 msgr2=0x7efc5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:31.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efc5c066f30 0x7efc5c0693e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efc6000a3f0 tx=0x7efc60016000 comp rx=0 tx=0).stop 2023-10-10T19:21:31.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc70159370 msgr2=0x7efc701597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc70159370 0x7efc701597a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7efc700b2760 tx=0x7efc68009040 comp rx=0 tx=0).stop 2023-10-10T19:21:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 shutdown_connections 2023-10-10T19:21:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efc5c066f30 0x7efc5c0693e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc70159370 0x7efc701597a0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc58003a60 0x7efc70154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 --2- 172.21.15.175:0/3447963661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc700c5330 0x7efc700c3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 >> 172.21.15.175:0/3447963661 conn(0x7efc70063230 msgr2=0x7efc7014fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 shutdown_connections 2023-10-10T19:21:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:31.491+0000 7efc77eb7700 1 -- 172.21.15.175:0/3447963661 wait complete. 2023-10-10T19:21:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 276 2023-10-10T19:21:31.504 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-10T19:21:33.506 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:21:33.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.633+0000 7f0b87523700 1 Processor -- start 2023-10-10T19:21:33.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.633+0000 7f0b87523700 1 -- start start 2023-10-10T19:21:33.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b87523700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0b800b1fc0 0x7f0b80155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:33.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b87523700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0b800aff60 0x7f0b800b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:33.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b87523700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b801574b0 con 0x7f0b800b2390 2023-10-10T19:21:33.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b87523700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b80150770 con 0x7f0b800b1fc0 2023-10-10T19:21:33.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b87523700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b800afad0 con 0x7f0b800aff60 2023-10-10T19:21:33.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b852bf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0b800b1fc0 0x7f0b80155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:33.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b84abe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f0b800b2390 0x7f0b80068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40876/0 (socket says 172.21.15.175:40876) 2023-10-10T19:21:33.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b84abe700 1 -- 172.21.15.175:0/2328237225 learned_addr learned my addr 172.21.15.175:0/2328237225 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:21:33.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b85ac0700 1 --2- 172.21.15.175:0/2328237225 >> v2:172.21.15.191:3300/0 conn(0x7f0b800aff60 0x7f0b800b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:33.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b852bf700 1 -- 172.21.15.175:0/2328237225 >> v2:172.21.15.191:3300/0 conn(0x7f0b800aff60 msgr2=0x7f0b800b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:21:33.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b852bf700 1 --2- 172.21.15.175:0/2328237225 >> v2:172.21.15.191:3300/0 conn(0x7f0b800aff60 0x7f0b800b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b852bf700 1 -- 172.21.15.175:0/2328237225 >> v1:172.21.15.175:6789/0 conn(0x7f0b800b2390 legacy=0x7f0b80068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:33.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b77fff700 1 -- 172.21.15.175:0/2328237225 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970586937 0 0) 0x7f0b801574b0 con 0x7f0b800b2390 2023-10-10T19:21:33.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b852bf700 1 -- 172.21.15.175:0/2328237225 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b800af4c0 con 0x7f0b800b1fc0 2023-10-10T19:21:33.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b852bf700 1 --2- 172.21.15.175:0/2328237225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800b1fc0 0x7f0b80155fb0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f0b800641f0 tx=0x7f0b78009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e01d42efc2c5dd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:33.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.634+0000 7f0b77fff700 1 -- 172.21.15.175:0/2328237225 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7800e070 con 0x7f0b800b1fc0 2023-10-10T19:21:33.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b77fff700 1 -- 172.21.15.175:0/2328237225 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7800bcc0 con 0x7f0b800b1fc0 2023-10-10T19:21:33.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b77fff700 1 -- 172.21.15.175:0/2328237225 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78012680 con 0x7f0b800b1fc0 2023-10-10T19:21:33.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b87523700 1 -- 172.21.15.175:0/2328237225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800b1fc0 msgr2=0x7f0b80155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:33.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b87523700 1 --2- 172.21.15.175:0/2328237225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800b1fc0 0x7f0b80155fb0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f0b800641f0 tx=0x7f0b78009780 comp rx=0 tx=0).stop 2023-10-10T19:21:33.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b87523700 1 -- 172.21.15.175:0/2328237225 shutdown_connections 2023-10-10T19:21:33.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b87523700 1 --2- 172.21.15.175:0/2328237225 >> v2:172.21.15.191:3300/0 conn(0x7f0b800aff60 0x7f0b800b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b87523700 1 --2- 172.21.15.175:0/2328237225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800b1fc0 0x7f0b80155fb0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b87523700 1 -- 172.21.15.175:0/2328237225 >> 172.21.15.175:0/2328237225 conn(0x7f0b80063230 msgr2=0x7f0b80063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:33.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b87523700 1 -- 172.21.15.175:0/2328237225 shutdown_connections 2023-10-10T19:21:33.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.635+0000 7f0b87523700 1 -- 172.21.15.175:0/2328237225 wait complete. 2023-10-10T19:21:33.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.636+0000 7f0b87523700 1 Processor -- start 2023-10-10T19:21:33.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.636+0000 7f0b87523700 1 -- start start 2023-10-10T19:21:33.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b87523700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800aff60 0x7f0b80165e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:33.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b87523700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b800b1fc0 0x7f0b800cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:33.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b87523700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b800b2390 0x7f0b800ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:33.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b87523700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b800c8fd0 con 0x7f0b800aff60 2023-10-10T19:21:33.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b87523700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b801583e0 con 0x7f0b800b2390 2023-10-10T19:21:33.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b87523700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b801586c0 con 0x7f0b800b1fc0 2023-10-10T19:21:33.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b852bf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800aff60 0x7f0b80165e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:33.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b852bf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800aff60 0x7f0b80165e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35002/0 (socket says 172.21.15.175:35002) 2023-10-10T19:21:33.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b852bf700 1 -- 172.21.15.175:0/2643416075 learned_addr learned my addr 172.21.15.175:0/2643416075 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:21:33.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b84abe700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b800b1fc0 0x7f0b800cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:33.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.637+0000 7f0b85ac0700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b800b2390 0x7f0b800ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:33.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b852bf700 1 -- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b800b1fc0 msgr2=0x7f0b800cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:33.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b852bf700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b800b1fc0 0x7f0b800cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b852bf700 1 -- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b800b2390 msgr2=0x7f0b800ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:33.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b852bf700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b800b2390 0x7f0b800ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b852bf700 1 -- 172.21.15.175:0/2643416075 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78009a20 con 0x7f0b800aff60 2023-10-10T19:21:33.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b85ac0700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b800b2390 0x7f0b800ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:21:33.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b84abe700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b800b1fc0 0x7f0b800cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:21:33.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b852bf700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800aff60 0x7f0b80165e80 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f0b780097c0 tx=0x7f0b78019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:33.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b757fa700 1 -- 172.21.15.175:0/2643416075 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78027470 con 0x7f0b800aff60 2023-10-10T19:21:33.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b757fa700 1 -- 172.21.15.175:0/2643416075 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b780275d0 con 0x7f0b800aff60 2023-10-10T19:21:33.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b757fa700 1 -- 172.21.15.175:0/2643416075 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78027990 con 0x7f0b800aff60 2023-10-10T19:21:33.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b801589a0 con 0x7f0b800aff60 2023-10-10T19:21:33.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.638+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b80158f60 con 0x7f0b800aff60 2023-10-10T19:21:33.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.639+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b80063e90 con 0x7f0b800aff60 2023-10-10T19:21:33.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.639+0000 7f0b757fa700 1 -- 172.21.15.175:0/2643416075 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0b78024070 con 0x7f0b800aff60 2023-10-10T19:21:33.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.640+0000 7f0b757fa700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0b64066f30 0x7f0b640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:21:33.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.640+0000 7f0b757fa700 1 -- 172.21.15.175:0/2643416075 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f0b78087640 con 0x7f0b800aff60 2023-10-10T19:21:33.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.640+0000 7f0b84abe700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0b64066f30 0x7f0b640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:21:33.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.640+0000 7f0b84abe700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0b64066f30 0x7f0b640693e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0b800712a0 tx=0x7f0b70009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:21:33.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.644+0000 7f0b757fa700 1 -- 172.21.15.175:0/2643416075 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b78054230 con 0x7f0b800aff60 2023-10-10T19:21:33.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.884+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b80067570 con 0x7f0b800aff60 2023-10-10T19:21:33.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.884+0000 7f0b757fa700 1 -- 172.21.15.175:0/2643416075 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f0b78057850 con 0x7f0b800aff60 2023-10-10T19:21:33.885 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:21:33.885 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":277}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:21:33.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.886+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0b64066f30 msgr2=0x7f0b640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:33.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.886+0000 7f0b87523700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0b64066f30 0x7f0b640693e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0b800712a0 tx=0x7f0b70009040 comp rx=0 tx=0).stop 2023-10-10T19:21:33.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800aff60 msgr2=0x7f0b80165e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:21:33.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800aff60 0x7f0b80165e80 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f0b780097c0 tx=0x7f0b78019000 comp rx=0 tx=0).stop 2023-10-10T19:21:33.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 shutdown_connections 2023-10-10T19:21:33.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0b64066f30 0x7f0b640693e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b800b2390 0x7f0b800ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b800aff60 0x7f0b80165e80 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 --2- 172.21.15.175:0/2643416075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b800b1fc0 0x7f0b800cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:21:33.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 >> 172.21.15.175:0/2643416075 conn(0x7f0b80063230 msgr2=0x7f0b80152640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:21:33.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 shutdown_connections 2023-10-10T19:21:33.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:21:33.887+0000 7f0b87523700 1 -- 172.21.15.175:0/2643416075 wait complete. 2023-10-10T19:21:33.890 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 277 2023-10-10T19:21:33.899 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-10T19:21:33.899 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 37 secs before thrashing 2023-10-10T19:21:37.949 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-10T19:21:37.948+0000 7fb93ba0c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-10T19:21:37.950 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-10T19:21:37.948+0000 7fb93ba0c700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:21:37.950 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-10T19:21:37.948+0000 7fb93ba0c700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-10T19:21:37.950 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-10T19:21:37.948+0000 7fb93ba0c700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T19:21:55.305 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:21:55.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:22:10.900 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:11.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.025+0000 7fefbf6ca700 1 Processor -- start 2023-10-10T19:22:11.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.026+0000 7fefbf6ca700 1 -- start start 2023-10-10T19:22:11.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.026+0000 7fefbf6ca700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fefb8155fb0 0x7fefb8159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.026+0000 7fefbf6ca700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb81574b0 con 0x7fefb80b2390 2023-10-10T19:22:11.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.026+0000 7fefbf6ca700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb80afad0 con 0x7fefb80b1fc0 2023-10-10T19:22:11.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.026+0000 7fefbf6ca700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb8150770 con 0x7fefb8155fb0 2023-10-10T19:22:11.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.026+0000 7fefbcc65700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fefb80b2390 0x7fefb80aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45704/0 (socket says 172.21.15.175:45704) 2023-10-10T19:22:11.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefbcc65700 1 -- 172.21.15.175:0/2543003545 learned_addr learned my addr 172.21.15.175:0/2543003545 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:11.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefbdc67700 1 --2- 172.21.15.175:0/2543003545 >> v2:172.21.15.191:3300/0 conn(0x7fefb8155fb0 0x7fefb8159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefb7fff700 1 -- 172.21.15.175:0/2543003545 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380372825 0 0) 0x7fefb81574b0 con 0x7fefb80b2390 2023-10-10T19:22:11.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefb7fff700 1 -- 172.21.15.175:0/2543003545 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa0003570 con 0x7fefb80b2390 2023-10-10T19:22:11.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefb7fff700 1 -- 172.21.15.175:0/2543003545 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435143051 0 0) 0x7fefb80afad0 con 0x7fefb80b1fc0 2023-10-10T19:22:11.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefb7fff700 1 -- 172.21.15.175:0/2543003545 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb81574b0 con 0x7fefb80b1fc0 2023-10-10T19:22:11.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefbdc67700 1 -- 172.21.15.175:0/2543003545 >> v1:172.21.15.191:6789/0 conn(0x7fefb80b1fc0 legacy=0x7fefb8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefbdc67700 1 -- 172.21.15.175:0/2543003545 >> v1:172.21.15.175:6789/0 conn(0x7fefb80b2390 legacy=0x7fefb80aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefbdc67700 1 -- 172.21.15.175:0/2543003545 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80af4c0 con 0x7fefb8155fb0 2023-10-10T19:22:11.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefbdc67700 1 --2- 172.21.15.175:0/2543003545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb8155fb0 0x7fefb8159920 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fefb00098e0 tx=0x7fefb00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d25eb597c650ae37 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:11.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefb7fff700 1 -- 172.21.15.175:0/2543003545 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb0011630 con 0x7fefb8155fb0 2023-10-10T19:22:11.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefb7fff700 1 -- 172.21.15.175:0/2543003545 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb0011790 con 0x7fefb8155fb0 2023-10-10T19:22:11.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.027+0000 7fefb7fff700 1 -- 172.21.15.175:0/2543003545 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb0011b10 con 0x7fefb8155fb0 2023-10-10T19:22:11.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.028+0000 7fefbf6ca700 1 -- 172.21.15.175:0/2543003545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb8155fb0 msgr2=0x7fefb8159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.028+0000 7fefbf6ca700 1 --2- 172.21.15.175:0/2543003545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb8155fb0 0x7fefb8159920 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fefb00098e0 tx=0x7fefb00091a0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.028+0000 7fefbf6ca700 1 -- 172.21.15.175:0/2543003545 shutdown_connections 2023-10-10T19:22:11.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.028+0000 7fefbf6ca700 1 --2- 172.21.15.175:0/2543003545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb8155fb0 0x7fefb8159920 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.028+0000 7fefbf6ca700 1 -- 172.21.15.175:0/2543003545 >> 172.21.15.175:0/2543003545 conn(0x7fefb8063230 msgr2=0x7fefb8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:11.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.028+0000 7fefbf6ca700 1 -- 172.21.15.175:0/2543003545 shutdown_connections 2023-10-10T19:22:11.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.028+0000 7fefbf6ca700 1 -- 172.21.15.175:0/2543003545 wait complete. 2023-10-10T19:22:11.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.029+0000 7fefbf6ca700 1 Processor -- start 2023-10-10T19:22:11.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.029+0000 7fefbf6ca700 1 -- start start 2023-10-10T19:22:11.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbf6ca700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb80b1fc0 0x7fefb80c2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbf6ca700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb80b2390 0x7fefb80c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbf6ca700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb8155fb0 0x7fefb814f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbf6ca700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c46a0 con 0x7fefb8155fb0 2023-10-10T19:22:11.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbf6ca700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c6920 con 0x7fefb80b1fc0 2023-10-10T19:22:11.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbf6ca700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c66b0 con 0x7fefb80b2390 2023-10-10T19:22:11.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbdc67700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb8155fb0 0x7fefb814f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbdc67700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb8155fb0 0x7fefb814f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53260/0 (socket says 172.21.15.175:53260) 2023-10-10T19:22:11.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbdc67700 1 -- 172.21.15.175:0/1398394403 learned_addr learned my addr 172.21.15.175:0/1398394403 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:11.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbcc65700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb80b2390 0x7fefb80c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbdc67700 1 -- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb80b2390 msgr2=0x7fefb80c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbd466700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb80b1fc0 0x7fefb80c2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbdc67700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb80b2390 0x7fefb80c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbdc67700 1 -- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb80b1fc0 msgr2=0x7fefb80c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbdc67700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb80b1fc0 0x7fefb80c2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbdc67700 1 -- 172.21.15.175:0/1398394403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0009420 con 0x7fefb8155fb0 2023-10-10T19:22:11.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbcc65700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb80b2390 0x7fefb80c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:22:11.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.030+0000 7fefbd466700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb80b1fc0 0x7fefb80c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:22:11.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.031+0000 7fefbdc67700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb8155fb0 0x7fefb814f920 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fefb0011f50 tx=0x7fefb0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:11.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.031+0000 7fefb57fa700 1 -- 172.21.15.175:0/1398394403 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb001a430 con 0x7fefb8155fb0 2023-10-10T19:22:11.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.031+0000 7fefb57fa700 1 -- 172.21.15.175:0/1398394403 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb001a590 con 0x7fefb8155fb0 2023-10-10T19:22:11.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.031+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb80c4210 con 0x7fefb8155fb0 2023-10-10T19:22:11.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.031+0000 7fefb57fa700 1 -- 172.21.15.175:0/1398394403 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb001a910 con 0x7fefb8155fb0 2023-10-10T19:22:11.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.031+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb80aeee0 con 0x7fefb8155fb0 2023-10-10T19:22:11.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.032+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb80c4450 con 0x7fefb8155fb0 2023-10-10T19:22:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.032+0000 7fefb57fa700 1 -- 172.21.15.175:0/1398394403 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fefb001acc0 con 0x7fefb8155fb0 2023-10-10T19:22:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.032+0000 7fefb57fa700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefa4066f30 0x7fefa40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.032+0000 7fefb57fa700 1 -- 172.21.15.175:0/1398394403 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7fefb0086a30 con 0x7fefb8155fb0 2023-10-10T19:22:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.033+0000 7fefbd466700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefa4066f30 0x7fefa40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.033+0000 7fefbd466700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefa4066f30 0x7fefa40693e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fefa8000f80 tx=0x7fefa8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:11.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.037+0000 7fefb57fa700 1 -- 172.21.15.175:0/1398394403 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fefb80c4450 con 0x7fefb8155fb0 2023-10-10T19:22:11.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.261+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefb80c4450 con 0x7fefb8155fb0 2023-10-10T19:22:11.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.261+0000 7fefb57fa700 1 -- 172.21.15.175:0/1398394403 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+6181 (secure 0 0 0) 0x7fefb80c4450 con 0x7fefb8155fb0 2023-10-10T19:22:11.263 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:11.263 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":277}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:11.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefa4066f30 msgr2=0x7fefa40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefa4066f30 0x7fefa40693e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fefa8000f80 tx=0x7fefa8009040 comp rx=0 tx=0).stop 2023-10-10T19:22:11.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb8155fb0 msgr2=0x7fefb814f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb8155fb0 0x7fefb814f920 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fefb0011f50 tx=0x7fefb0017000 comp rx=0 tx=0).stop 2023-10-10T19:22:11.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 shutdown_connections 2023-10-10T19:22:11.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fefa4066f30 0x7fefa40693e0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fefb8155fb0 0x7fefb814f920 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefb80b1fc0 0x7fefb80c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 --2- 172.21.15.175:0/1398394403 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fefb80b2390 0x7fefb80c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 >> 172.21.15.175:0/1398394403 conn(0x7fefb8063230 msgr2=0x7fefb8068690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:11.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 shutdown_connections 2023-10-10T19:22:11.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.263+0000 7fefbf6ca700 1 -- 172.21.15.175:0/1398394403 wait complete. 2023-10-10T19:22:11.267 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 277 2023-10-10T19:22:11.277 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=3) 2023-10-10T19:22:11.277 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-10T19:22:11.277 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:22:11.287 INFO:tasks.ceph.mds.c:Stopped 2023-10-10T19:22:11.288 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:11.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.439+0000 7f35d68d6700 1 Processor -- start 2023-10-10T19:22:11.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.439+0000 7f35d68d6700 1 -- start start 2023-10-10T19:22:11.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d68d6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f35d00b1fc0 0x7f35d0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d68d6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f35d00aff60 0x7f35d00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d68d6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d01574b0 con 0x7f35d00b2390 2023-10-10T19:22:11.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d68d6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d0150770 con 0x7f35d00aff60 2023-10-10T19:22:11.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d68d6700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d00afad0 con 0x7f35d00b1fc0 2023-10-10T19:22:11.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d4e73700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f35d00aff60 0x7f35d00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35cf7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f35d00b2390 0x7f35d0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45718/0 (socket says 172.21.15.175:45718) 2023-10-10T19:22:11.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35cf7fe700 1 -- 172.21.15.175:0/1564248368 learned_addr learned my addr 172.21.15.175:0/1564248368 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:11.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35cffff700 1 --2- 172.21.15.175:0/1564248368 >> v2:172.21.15.191:3300/0 conn(0x7f35d00b1fc0 0x7f35d0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d4e73700 1 -- 172.21.15.175:0/1564248368 >> v2:172.21.15.191:3300/0 conn(0x7f35d00b1fc0 msgr2=0x7f35d0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:11.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d4e73700 1 --2- 172.21.15.175:0/1564248368 >> v2:172.21.15.191:3300/0 conn(0x7f35d00b1fc0 0x7f35d0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d4e73700 1 -- 172.21.15.175:0/1564248368 >> v1:172.21.15.175:6789/0 conn(0x7f35d00b2390 legacy=0x7f35d0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.440+0000 7f35d4e73700 1 -- 172.21.15.175:0/1564248368 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d00af4c0 con 0x7f35d00aff60 2023-10-10T19:22:11.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d4e73700 1 --2- 172.21.15.175:0/1564248368 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d00b03e0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f35d00b28c0 tx=0x7f35c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269469fe992489f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:11.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35ceffd700 1 -- 172.21.15.175:0/1564248368 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c800e070 con 0x7f35d00aff60 2023-10-10T19:22:11.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35ceffd700 1 -- 172.21.15.175:0/1564248368 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c800bcc0 con 0x7f35d00aff60 2023-10-10T19:22:11.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35ceffd700 1 -- 172.21.15.175:0/1564248368 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8012680 con 0x7f35d00aff60 2023-10-10T19:22:11.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d68d6700 1 -- 172.21.15.175:0/1564248368 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 msgr2=0x7f35d00b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1564248368 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d00b03e0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f35d00b28c0 tx=0x7f35c8009780 comp rx=0 tx=0).stop 2023-10-10T19:22:11.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d68d6700 1 -- 172.21.15.175:0/1564248368 shutdown_connections 2023-10-10T19:22:11.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1564248368 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d00b03e0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1564248368 >> v2:172.21.15.191:3300/0 conn(0x7f35d00b1fc0 0x7f35d0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d68d6700 1 -- 172.21.15.175:0/1564248368 >> 172.21.15.175:0/1564248368 conn(0x7f35d0063230 msgr2=0x7f35d0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:11.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d68d6700 1 -- 172.21.15.175:0/1564248368 shutdown_connections 2023-10-10T19:22:11.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.441+0000 7f35d68d6700 1 -- 172.21.15.175:0/1564248368 wait complete. 2023-10-10T19:22:11.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.442+0000 7f35d68d6700 1 Processor -- start 2023-10-10T19:22:11.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35d68d6700 1 -- start start 2023-10-10T19:22:11.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35d68d6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d0166760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35d68d6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d00b1fc0 0x7f35d00c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35d68d6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d00b2390 0x7f35d01552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35d68d6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0166ca0 con 0x7f35d00aff60 2023-10-10T19:22:11.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35d68d6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00c6400 con 0x7f35d00b1fc0 2023-10-10T19:22:11.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35d68d6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00c5860 con 0x7f35d00b2390 2023-10-10T19:22:11.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35cffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d0166760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35cffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d0166760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53292/0 (socket says 172.21.15.175:53292) 2023-10-10T19:22:11.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35cffff700 1 -- 172.21.15.175:0/1731854626 learned_addr learned my addr 172.21.15.175:0/1731854626 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:11.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35cf7fe700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d00b1fc0 0x7f35d00c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35d4e73700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d00b2390 0x7f35d01552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35cffff700 1 -- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d00b2390 msgr2=0x7f35d01552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.443+0000 7f35cffff700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d00b2390 0x7f35d01552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35cffff700 1 -- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d00b1fc0 msgr2=0x7f35d00c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35cffff700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d00b1fc0 0x7f35d00c3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35cffff700 1 -- 172.21.15.175:0/1731854626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8009a20 con 0x7f35d00aff60 2023-10-10T19:22:11.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35cf7fe700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d00b1fc0 0x7f35d00c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:22:11.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35cffff700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d0166760 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f35c000d8c0 tx=0x7f35c0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:11.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35b7fff700 1 -- 172.21.15.175:0/1731854626 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c000de60 con 0x7f35d00aff60 2023-10-10T19:22:11.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35b7fff700 1 -- 172.21.15.175:0/1731854626 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c000b780 con 0x7f35d00aff60 2023-10-10T19:22:11.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d00c6890 con 0x7f35d00aff60 2023-10-10T19:22:11.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35b7fff700 1 -- 172.21.15.175:0/1731854626 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c000bb20 con 0x7f35d00aff60 2023-10-10T19:22:11.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.444+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d00c5ad0 con 0x7f35d00aff60 2023-10-10T19:22:11.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.445+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d0067570 con 0x7f35d00aff60 2023-10-10T19:22:11.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.445+0000 7f35b7fff700 1 -- 172.21.15.175:0/1731854626 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f35c001e460 con 0x7f35d00aff60 2023-10-10T19:22:11.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.445+0000 7f35b7fff700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35bc066f30 0x7f35bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:11.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.446+0000 7f35b7fff700 1 -- 172.21.15.175:0/1731854626 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f35c0085240 con 0x7f35d00aff60 2023-10-10T19:22:11.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.446+0000 7f35cf7fe700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35bc066f30 0x7f35bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:11.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.446+0000 7f35cf7fe700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35bc066f30 0x7f35bc0693e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f35d00ca9a0 tx=0x7f35c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:11.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.450+0000 7f35b7fff700 1 -- 172.21.15.175:0/1731854626 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35c0051e30 con 0x7f35d00aff60 2023-10-10T19:22:11.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.673+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35d0071340 con 0x7f35d00aff60 2023-10-10T19:22:11.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.673+0000 7f35b7fff700 1 -- 172.21.15.175:0/1731854626 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f35c0055450 con 0x7f35d00aff60 2023-10-10T19:22:11.675 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:11.676 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":277}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:11.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.675+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35bc066f30 msgr2=0x7f35bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.675+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35bc066f30 0x7f35bc0693e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f35d00ca9a0 tx=0x7f35c4009040 comp rx=0 tx=0).stop 2023-10-10T19:22:11.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.675+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 msgr2=0x7f35d0166760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:11.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.675+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d0166760 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f35c000d8c0 tx=0x7f35c0008040 comp rx=0 tx=0).stop 2023-10-10T19:22:11.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.676+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 shutdown_connections 2023-10-10T19:22:11.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.676+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f35bc066f30 0x7f35bc0693e0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.676+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35d00b2390 0x7f35d01552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.676+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35d00aff60 0x7f35d0166760 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.676+0000 7f35d68d6700 1 --2- 172.21.15.175:0/1731854626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35d00b1fc0 0x7f35d00c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:11.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.676+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 >> 172.21.15.175:0/1731854626 conn(0x7f35d0063230 msgr2=0x7f35d014fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:11.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.676+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 shutdown_connections 2023-10-10T19:22:11.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:11.676+0000 7f35d68d6700 1 -- 172.21.15.175:0/1731854626 wait complete. 2023-10-10T19:22:11.683 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 277 2023-10-10T19:22:11.689 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:22:13.690 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:13.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.817+0000 7f428dced700 1 Processor -- start 2023-10-10T19:22:13.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.817+0000 7f428dced700 1 -- start start 2023-10-10T19:22:13.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.817+0000 7f428dced700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f42881577a0 0x7f42880c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:13.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.817+0000 7f428dced700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4288154aa0 con 0x7f42880c12a0 2023-10-10T19:22:13.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.817+0000 7f428dced700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4288063dc0 con 0x7f42881573d0 2023-10-10T19:22:13.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.817+0000 7f428dced700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42880bfdf0 con 0x7f42881577a0 2023-10-10T19:22:13.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f427ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f42880c12a0 0x7f428814d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45734/0 (socket says 172.21.15.175:45734) 2023-10-10T19:22:13.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f427ffff700 1 -- 172.21.15.175:0/3689816545 learned_addr learned my addr 172.21.15.175:0/3689816545 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:13.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4287fff700 1 --2- 172.21.15.175:0/3689816545 >> v2:172.21.15.191:3300/0 conn(0x7f42881577a0 0x7f42880c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:13.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374570545 0 0) 0x7f4288154aa0 con 0x7f42880c12a0 2023-10-10T19:22:13.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4274003570 con 0x7f42880c12a0 2023-10-10T19:22:13.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1912588461 0 0) 0x7f4274003570 con 0x7f42880c12a0 2023-10-10T19:22:13.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 >> v2:172.21.15.191:3300/0 conn(0x7f42881577a0 msgr2=0x7f42880c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:13.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 --2- 172.21.15.175:0/3689816545 >> v2:172.21.15.191:3300/0 conn(0x7f42881577a0 0x7f42880c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:13.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 >> v1:172.21.15.191:6789/0 conn(0x7f42881573d0 legacy=0x7f42880bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:13.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4288157ba0 con 0x7f42880c12a0 2023-10-10T19:22:13.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f4270001470 con 0x7f42880c12a0 2023-10-10T19:22:13.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 >> v1:172.21.15.175:6789/0 conn(0x7f42880c12a0 legacy=0x7f428814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:13.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4274003a60 0x7f4274003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:13.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4274004810 0x7f4274004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:13.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42740051c0 0x7f427400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:13.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427400c050 con 0x7f42740051c0 2023-10-10T19:22:13.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427400c190 con 0x7f4274003a60 2023-10-10T19:22:13.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427400c350 con 0x7f4274004810 2023-10-10T19:22:13.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f4287fff700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42740051c0 0x7f427400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:13.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f42877fe700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4274003a60 0x7f4274003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:13.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.818+0000 7f427ffff700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4274004810 0x7f4274004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:13.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4287fff700 1 -- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4274004810 msgr2=0x7f4274004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:13.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4287fff700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4274004810 0x7f4274004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:13.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4287fff700 1 -- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4274003a60 msgr2=0x7f4274003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:13.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4287fff700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4274003a60 0x7f4274003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:13.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4287fff700 1 -- 172.21.15.175:0/3689816545 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4274003570 con 0x7f42740051c0 2023-10-10T19:22:13.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f427ffff700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4274004810 0x7f4274004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:22:13.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4287fff700 1 -- 172.21.15.175:0/3689816545 reap_dead start 2023-10-10T19:22:13.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4287fff700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42740051c0 0x7f427400ba60 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f428000e210 tx=0x7f428000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee549a69af7c6599 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:13.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428000eac0 con 0x7f42740051c0 2023-10-10T19:22:13.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428000f040 con 0x7f42740051c0 2023-10-10T19:22:13.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f4286ffd700 1 -- 172.21.15.175:0/3689816545 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428000ee70 con 0x7f42740051c0 2023-10-10T19:22:13.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f428dced700 1 -- 172.21.15.175:0/3689816545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42740051c0 msgr2=0x7f427400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:13.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f428dced700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42740051c0 0x7f427400ba60 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f428000e210 tx=0x7f428000b040 comp rx=0 tx=0).stop 2023-10-10T19:22:13.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f428dced700 1 -- 172.21.15.175:0/3689816545 shutdown_connections 2023-10-10T19:22:13.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f428dced700 1 --2- 172.21.15.175:0/3689816545 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42740051c0 0x7f427400ba60 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:13.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.819+0000 7f428dced700 1 -- 172.21.15.175:0/3689816545 >> 172.21.15.175:0/3689816545 conn(0x7f428806dc70 msgr2=0x7f4288063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:13.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.820+0000 7f428dced700 1 -- 172.21.15.175:0/3689816545 shutdown_connections 2023-10-10T19:22:13.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.820+0000 7f428dced700 1 -- 172.21.15.175:0/3689816545 wait complete. 2023-10-10T19:22:13.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f428dced700 1 Processor -- start 2023-10-10T19:22:13.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f428dced700 1 -- start start 2023-10-10T19:22:13.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f428dced700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42740051c0 0x7f42881584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:13.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f428dced700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4288151c30 0x7f4288158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:13.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f428dced700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428815a5b0 0x7f42880c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:13.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f428dced700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42880c47e0 con 0x7f428815a5b0 2023-10-10T19:22:13.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f428dced700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42880c35d0 con 0x7f4288151c30 2023-10-10T19:22:13.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f428dced700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42880c26e0 con 0x7f42740051c0 2023-10-10T19:22:13.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f4287fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428815a5b0 0x7f42880c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:13.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f4287fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428815a5b0 0x7f42880c1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53306/0 (socket says 172.21.15.175:53306) 2023-10-10T19:22:13.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f4287fff700 1 -- 172.21.15.175:0/2128606172 learned_addr learned my addr 172.21.15.175:0/2128606172 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:13.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f42877fe700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42740051c0 0x7f42881584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:13.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.821+0000 7f427ffff700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4288151c30 0x7f4288158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:13.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f4287fff700 1 -- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42740051c0 msgr2=0x7f42881584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:13.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f4287fff700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42740051c0 0x7f42881584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:13.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f4287fff700 1 -- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4288151c30 msgr2=0x7f4288158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:13.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f4287fff700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4288151c30 0x7f4288158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:13.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f4287fff700 1 -- 172.21.15.175:0/2128606172 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42880c29c0 con 0x7f428815a5b0 2023-10-10T19:22:13.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f4287fff700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428815a5b0 0x7f42880c1ae0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f428806ce40 tx=0x7f4280019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:13.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f427f7fe700 1 -- 172.21.15.175:0/2128606172 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428001d650 con 0x7f428815a5b0 2023-10-10T19:22:13.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f427f7fe700 1 -- 172.21.15.175:0/2128606172 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428001d7d0 con 0x7f428815a5b0 2023-10-10T19:22:13.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42880c2ca0 con 0x7f428815a5b0 2023-10-10T19:22:13.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f427f7fe700 1 -- 172.21.15.175:0/2128606172 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428001db90 con 0x7f428815a5b0 2023-10-10T19:22:13.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.822+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42880c3260 con 0x7f428815a5b0 2023-10-10T19:22:13.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.823+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42880c3960 con 0x7f428815a5b0 2023-10-10T19:22:13.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.823+0000 7f427f7fe700 1 -- 172.21.15.175:0/2128606172 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4280003020 con 0x7f428815a5b0 2023-10-10T19:22:13.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.824+0000 7f427f7fe700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4268066f60 0x7f4268069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:13.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.824+0000 7f427f7fe700 1 -- 172.21.15.175:0/2128606172 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f4280086b90 con 0x7f428815a5b0 2023-10-10T19:22:13.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.824+0000 7f42877fe700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4268066f60 0x7f4268069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:13.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.824+0000 7f42877fe700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4268066f60 0x7f4268069410 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4278000c00 tx=0x7f4278007d10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:13.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:13.828+0000 7f427f7fe700 1 -- 172.21.15.175:0/2128606172 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f42880c3960 con 0x7f428815a5b0 2023-10-10T19:22:14.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.052+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42880c3960 con 0x7f428815a5b0 2023-10-10T19:22:14.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.053+0000 7f427f7fe700 1 -- 172.21.15.175:0/2128606172 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f42880c3960 con 0x7f428815a5b0 2023-10-10T19:22:14.055 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:14.055 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":277}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:14.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4268066f60 msgr2=0x7f4268069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:14.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4268066f60 0x7f4268069410 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4278000c00 tx=0x7f4278007d10 comp rx=0 tx=0).stop 2023-10-10T19:22:14.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428815a5b0 msgr2=0x7f42880c1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:14.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428815a5b0 0x7f42880c1ae0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f428806ce40 tx=0x7f4280019000 comp rx=0 tx=0).stop 2023-10-10T19:22:14.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 shutdown_connections 2023-10-10T19:22:14.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4268066f60 0x7f4268069410 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:14.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f428815a5b0 0x7f42880c1ae0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:14.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42740051c0 0x7f42881584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:14.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 --2- 172.21.15.175:0/2128606172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4288151c30 0x7f4288158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:14.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 >> 172.21.15.175:0/2128606172 conn(0x7f428806dc70 msgr2=0x7f428814aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:14.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 shutdown_connections 2023-10-10T19:22:14.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:14.055+0000 7f428dced700 1 -- 172.21.15.175:0/2128606172 wait complete. 2023-10-10T19:22:14.059 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 277 2023-10-10T19:22:14.068 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:22:16.069 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:16.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.191+0000 7f27ab9d5700 1 Processor -- start 2023-10-10T19:22:16.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.191+0000 7f27ab9d5700 1 -- start start 2023-10-10T19:22:16.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.192+0000 7f27ab9d5700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f27a41573d0 0x7f27a40c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:16.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.192+0000 7f27ab9d5700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f27a40c4f20 0x7f27a414d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:16.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.192+0000 7f27ab9d5700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a4154aa0 con 0x7f27a41577a0 2023-10-10T19:22:16.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.192+0000 7f27ab9d5700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a40bfdf0 con 0x7f27a40c4f20 2023-10-10T19:22:16.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.192+0000 7f27ab9d5700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a4063dc0 con 0x7f27a41573d0 2023-10-10T19:22:16.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.192+0000 7f27a9f72700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f27a40c4f20 0x7f27a414d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:16.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.192+0000 7f27a8f70700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f27a41577a0 0x7f27a40bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39438/0 (socket says 172.21.15.175:39438) 2023-10-10T19:22:16.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.192+0000 7f27a8f70700 1 -- 172.21.15.175:0/4008747656 learned_addr learned my addr 172.21.15.175:0/4008747656 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:16.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a9771700 1 --2- 172.21.15.175:0/4008747656 >> v2:172.21.15.191:3300/0 conn(0x7f27a41573d0 0x7f27a40c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:16.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a9f72700 1 -- 172.21.15.175:0/4008747656 >> v2:172.21.15.191:3300/0 conn(0x7f27a41573d0 msgr2=0x7f27a40c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:16.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a9f72700 1 --2- 172.21.15.175:0/4008747656 >> v2:172.21.15.191:3300/0 conn(0x7f27a41573d0 0x7f27a40c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a9f72700 1 -- 172.21.15.175:0/4008747656 >> v1:172.21.15.175:6789/0 conn(0x7f27a41577a0 legacy=0x7f27a40bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:16.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a9f72700 1 -- 172.21.15.175:0/4008747656 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4157c20 con 0x7f27a40c4f20 2023-10-10T19:22:16.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a3fff700 1 -- 172.21.15.175:0/4008747656 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514276387 0 0) 0x7f27a4154aa0 con 0x7f27a41577a0 2023-10-10T19:22:16.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a9f72700 1 --2- 172.21.15.175:0/4008747656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a40c4f20 0x7f27a414d450 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f27a406e6b0 tx=0x7f279c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37301ed513b0a47a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:16.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a3fff700 1 -- 172.21.15.175:0/4008747656 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c011630 con 0x7f27a40c4f20 2023-10-10T19:22:16.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a3fff700 1 -- 172.21.15.175:0/4008747656 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279c011790 con 0x7f27a40c4f20 2023-10-10T19:22:16.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27a3fff700 1 -- 172.21.15.175:0/4008747656 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c011b30 con 0x7f27a40c4f20 2023-10-10T19:22:16.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.193+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4008747656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a40c4f20 msgr2=0x7f27a414d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:16.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.194+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4008747656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a40c4f20 0x7f27a414d450 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f27a406e6b0 tx=0x7f279c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.194+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4008747656 shutdown_connections 2023-10-10T19:22:16.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.194+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4008747656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a40c4f20 0x7f27a414d450 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.194+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4008747656 >> v2:172.21.15.191:3300/0 conn(0x7f27a41573d0 0x7f27a40c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.194+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4008747656 >> 172.21.15.175:0/4008747656 conn(0x7f27a406dc70 msgr2=0x7f27a4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:16.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.194+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4008747656 shutdown_connections 2023-10-10T19:22:16.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.194+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4008747656 wait complete. 2023-10-10T19:22:16.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.195+0000 7f27ab9d5700 1 Processor -- start 2023-10-10T19:22:16.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.195+0000 7f27ab9d5700 1 -- start start 2023-10-10T19:22:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.195+0000 7f27ab9d5700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a40c4f20 0x7f27a41557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27ab9d5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a41573d0 0x7f27a4155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27ab9d5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a41577a0 0x7f27a4152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27ab9d5700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a4154d50 con 0x7f27a41577a0 2023-10-10T19:22:16.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27ab9d5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a4155310 con 0x7f27a41573d0 2023-10-10T19:22:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27ab9d5700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a40c6970 con 0x7f27a40c4f20 2023-10-10T19:22:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a41577a0 0x7f27a4152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a41577a0 0x7f27a4152d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40788/0 (socket says 172.21.15.175:40788) 2023-10-10T19:22:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 -- 172.21.15.175:0/4125803506 learned_addr learned my addr 172.21.15.175:0/4125803506 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:16.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a8f70700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a41573d0 0x7f27a4155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9771700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a40c4f20 0x7f27a41557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 -- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a40c4f20 msgr2=0x7f27a41557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a40c4f20 0x7f27a41557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 -- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a41573d0 msgr2=0x7f27a4155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:16.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a41573d0 0x7f27a4155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 -- 172.21.15.175:0/4125803506 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c009420 con 0x7f27a41577a0 2023-10-10T19:22:16.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a9f72700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a41577a0 0x7f27a4152d10 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f27a406e6e0 tx=0x7f279c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:16.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a17fa700 1 -- 172.21.15.175:0/4125803506 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c011b10 con 0x7f27a41577a0 2023-10-10T19:22:16.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a17fa700 1 -- 172.21.15.175:0/4125803506 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279c0253f0 con 0x7f27a41577a0 2023-10-10T19:22:16.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a40c6c50 con 0x7f27a41577a0 2023-10-10T19:22:16.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27a17fa700 1 -- 172.21.15.175:0/4125803506 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c025790 con 0x7f27a41577a0 2023-10-10T19:22:16.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.196+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a4154fa0 con 0x7f27a41577a0 2023-10-10T19:22:16.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.197+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a4155480 con 0x7f27a41577a0 2023-10-10T19:22:16.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.198+0000 7f27a17fa700 1 -- 172.21.15.175:0/4125803506 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f279c025b00 con 0x7f27a41577a0 2023-10-10T19:22:16.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.198+0000 7f27a17fa700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2790066f60 0x7f2790069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:16.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.198+0000 7f27a17fa700 1 -- 172.21.15.175:0/4125803506 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f279c086590 con 0x7f27a41577a0 2023-10-10T19:22:16.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.198+0000 7f27a9771700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2790066f60 0x7f2790069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:16.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.198+0000 7f27a9771700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2790066f60 0x7f2790069410 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f279400f570 tx=0x7f279400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:16.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.202+0000 7f27a17fa700 1 -- 172.21.15.175:0/4125803506 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f279c053180 con 0x7f27a41577a0 2023-10-10T19:22:16.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.426+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27a4067570 con 0x7f27a41577a0 2023-10-10T19:22:16.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.426+0000 7f27a17fa700 1 -- 172.21.15.175:0/4125803506 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f279c0567a0 con 0x7f27a41577a0 2023-10-10T19:22:16.428 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:16.428 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":277}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:16.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2790066f60 msgr2=0x7f2790069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:16.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2790066f60 0x7f2790069410 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f279400f570 tx=0x7f279400b040 comp rx=0 tx=0).stop 2023-10-10T19:22:16.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a41577a0 msgr2=0x7f27a4152d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:16.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a41577a0 0x7f27a4152d10 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f27a406e6e0 tx=0x7f279c017000 comp rx=0 tx=0).stop 2023-10-10T19:22:16.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 shutdown_connections 2023-10-10T19:22:16.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2790066f60 0x7f2790069410 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a41577a0 0x7f27a4152d10 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a40c4f20 0x7f27a41557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 --2- 172.21.15.175:0/4125803506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27a41573d0 0x7f27a4155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:16.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 >> 172.21.15.175:0/4125803506 conn(0x7f27a406dc70 msgr2=0x7f27a414cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:16.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 shutdown_connections 2023-10-10T19:22:16.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:16.429+0000 7f27ab9d5700 1 -- 172.21.15.175:0/4125803506 wait complete. 2023-10-10T19:22:16.433 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 277 2023-10-10T19:22:16.442 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:22:18.443 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:18.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.569+0000 7f40bc6bc700 1 Processor -- start 2023-10-10T19:22:18.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.570+0000 7f40bc6bc700 1 -- start start 2023-10-10T19:22:18.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.570+0000 7f40bc6bc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f40b40c60d0 0x7f40b4156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:18.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.570+0000 7f40bc6bc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b40c9cc0 con 0x7f40b406f5f0 2023-10-10T19:22:18.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.570+0000 7f40bc6bc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b406e650 con 0x7f40b40c64a0 2023-10-10T19:22:18.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.570+0000 7f40bc6bc700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b41571c0 con 0x7f40b40c60d0 2023-10-10T19:22:18.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40ba458700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f40b40c60d0 0x7f40b4156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:18.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40b9c57700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f40b406f5f0 0x7f40b40c3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39452/0 (socket says 172.21.15.175:39452) 2023-10-10T19:22:18.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40b9c57700 1 -- 172.21.15.175:0/738812849 learned_addr learned my addr 172.21.15.175:0/738812849 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:18.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40ba458700 1 -- 172.21.15.175:0/738812849 >> v1:172.21.15.191:6789/0 conn(0x7f40b40c64a0 legacy=0x7f40b414cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:18.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40ba458700 1 -- 172.21.15.175:0/738812849 >> v1:172.21.15.175:6789/0 conn(0x7f40b406f5f0 legacy=0x7f40b40c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:18.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40ba458700 1 -- 172.21.15.175:0/738812849 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b406e930 con 0x7f40b40c60d0 2023-10-10T19:22:18.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40b9456700 1 -- 172.21.15.175:0/738812849 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493990251 0 0) 0x7f40b40c9cc0 con 0x7f40b406f5f0 2023-10-10T19:22:18.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40ba458700 1 --2- 172.21.15.175:0/738812849 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b40c60d0 0x7f40b4156c80 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0098e0 tx=0x7f40ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeea7052074234e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:18.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40b9456700 1 -- 172.21.15.175:0/738812849 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac009e80 con 0x7f40b40c60d0 2023-10-10T19:22:18.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40b9456700 1 -- 172.21.15.175:0/738812849 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac00b860 con 0x7f40b40c60d0 2023-10-10T19:22:18.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.571+0000 7f40b9456700 1 -- 172.21.15.175:0/738812849 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac00bbe0 con 0x7f40b40c60d0 2023-10-10T19:22:18.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.572+0000 7f40bc6bc700 1 -- 172.21.15.175:0/738812849 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b40c60d0 msgr2=0x7f40b4156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:18.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.572+0000 7f40bc6bc700 1 --2- 172.21.15.175:0/738812849 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b40c60d0 0x7f40b4156c80 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0098e0 tx=0x7f40ac0091a0 comp rx=0 tx=0).stop 2023-10-10T19:22:18.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.572+0000 7f40bc6bc700 1 -- 172.21.15.175:0/738812849 shutdown_connections 2023-10-10T19:22:18.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.572+0000 7f40bc6bc700 1 --2- 172.21.15.175:0/738812849 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b40c60d0 0x7f40b4156c80 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:18.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.572+0000 7f40bc6bc700 1 -- 172.21.15.175:0/738812849 >> 172.21.15.175:0/738812849 conn(0x7f40b406c110 msgr2=0x7f40b406c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:18.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.572+0000 7f40bc6bc700 1 -- 172.21.15.175:0/738812849 shutdown_connections 2023-10-10T19:22:18.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.572+0000 7f40bc6bc700 1 -- 172.21.15.175:0/738812849 wait complete. 2023-10-10T19:22:18.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.573+0000 7f40bc6bc700 1 Processor -- start 2023-10-10T19:22:18.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.573+0000 7f40bc6bc700 1 -- start start 2023-10-10T19:22:18.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.573+0000 7f40bc6bc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b406f5f0 0x7f40b415fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:18.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.573+0000 7f40bc6bc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40b40c60d0 0x7f40b40c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:18.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.573+0000 7f40bc6bc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40b40c64a0 0x7f40b40c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:18.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.573+0000 7f40bc6bc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40c4500 con 0x7f40b406f5f0 2023-10-10T19:22:18.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.573+0000 7f40bc6bc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b4165be0 con 0x7f40b40c64a0 2023-10-10T19:22:18.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.573+0000 7f40bc6bc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b4165ec0 con 0x7f40b40c60d0 2023-10-10T19:22:18.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b406f5f0 0x7f40b415fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:18.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b406f5f0 0x7f40b415fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40808/0 (socket says 172.21.15.175:40808) 2023-10-10T19:22:18.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 -- 172.21.15.175:0/2742000536 learned_addr learned my addr 172.21.15.175:0/2742000536 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:18.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40bac59700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40b40c64a0 0x7f40b40c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:18.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 -- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40b40c60d0 msgr2=0x7f40b40c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:18.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b9c57700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40b40c60d0 0x7f40b40c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:18.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40b40c60d0 0x7f40b40c7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:18.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 -- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40b40c64a0 msgr2=0x7f40b40c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:18.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40b40c64a0 0x7f40b40c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:18.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 -- 172.21.15.175:0/2742000536 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac009420 con 0x7f40b406f5f0 2023-10-10T19:22:18.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40bac59700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40b40c64a0 0x7f40b40c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:22:18.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b9c57700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40b40c60d0 0x7f40b40c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:22:18.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40ba458700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b406f5f0 0x7f40b415fd80 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00b3c0 tx=0x7f40ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:18.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b27fc700 1 -- 172.21.15.175:0/2742000536 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac01f070 con 0x7f40b406f5f0 2023-10-10T19:22:18.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b27fc700 1 -- 172.21.15.175:0/2742000536 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac00b9d0 con 0x7f40b406f5f0 2023-10-10T19:22:18.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b4153b40 con 0x7f40b406f5f0 2023-10-10T19:22:18.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b27fc700 1 -- 172.21.15.175:0/2742000536 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac025650 con 0x7f40b406f5f0 2023-10-10T19:22:18.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b4142d40 con 0x7f40b406f5f0 2023-10-10T19:22:18.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b4067570 con 0x7f40b406f5f0 2023-10-10T19:22:18.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b27fc700 1 -- 172.21.15.175:0/2742000536 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f40ac023070 con 0x7f40b406f5f0 2023-10-10T19:22:18.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b27fc700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40a0066f30 0x7f40a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:18.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b27fc700 1 -- 172.21.15.175:0/2742000536 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f40ac0861c0 con 0x7f40b406f5f0 2023-10-10T19:22:18.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b9c57700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40a0066f30 0x7f40a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:18.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.575+0000 7f40b9c57700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40a0066f30 0x7f40a00693e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f40b4156e30 tx=0x7f40a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:18.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.580+0000 7f40b27fc700 1 -- 172.21.15.175:0/2742000536 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f40ac052df0 con 0x7f40b406f5f0 2023-10-10T19:22:18.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.803+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40b4071d40 con 0x7f40b406f5f0 2023-10-10T19:22:18.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.804+0000 7f40b27fc700 1 -- 172.21.15.175:0/2742000536 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f40ac0563d0 con 0x7f40b406f5f0 2023-10-10T19:22:18.805 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:18.806 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":277}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:18.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40a0066f30 msgr2=0x7f40a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:18.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40a0066f30 0x7f40a00693e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f40b4156e30 tx=0x7f40a8009040 comp rx=0 tx=0).stop 2023-10-10T19:22:18.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b406f5f0 msgr2=0x7f40b415fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:18.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b406f5f0 0x7f40b415fd80 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00b3c0 tx=0x7f40ac017000 comp rx=0 tx=0).stop 2023-10-10T19:22:18.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 shutdown_connections 2023-10-10T19:22:18.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f40a0066f30 0x7f40a00693e0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:18.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40b40c64a0 0x7f40b40c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:18.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40b406f5f0 0x7f40b415fd80 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:18.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 --2- 172.21.15.175:0/2742000536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40b40c60d0 0x7f40b40c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:18.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 >> 172.21.15.175:0/2742000536 conn(0x7f40b406c110 msgr2=0x7f40b40c4640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:18.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 shutdown_connections 2023-10-10T19:22:18.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:18.806+0000 7f40bc6bc700 1 -- 172.21.15.175:0/2742000536 wait complete. 2023-10-10T19:22:18.813 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 277 2023-10-10T19:22:18.819 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:22:20.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:20.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.947+0000 7f2be47e8700 1 Processor -- start 2023-10-10T19:22:20.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.947+0000 7f2be47e8700 1 -- start start 2023-10-10T19:22:20.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be47e8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2bdc0c3210 0x7f2bdc159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:20.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be47e8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2bdc0c35e0 0x7f2bdc0c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:20.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be47e8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bdc0c5e40 con 0x7f2bdc0c9390 2023-10-10T19:22:20.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be47e8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bdc0c4c50 con 0x7f2bdc0c3210 2023-10-10T19:22:20.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be47e8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bdc0ca400 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be2584700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2bdc0c3210 0x7f2bdc159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:20.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be2584700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2bdc0c3210 0x7f2bdc159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40816/0 (socket says 172.21.15.175:40816) 2023-10-10T19:22:20.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be2584700 1 -- 172.21.15.175:0/4158012248 learned_addr learned my addr 172.21.15.175:0/4158012248 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:20.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be1d83700 1 --2- 172.21.15.175:0/4158012248 >> v2:172.21.15.191:3300/0 conn(0x7f2bdc0c35e0 0x7f2bdc0c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:20.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be2584700 1 -- 172.21.15.175:0/4158012248 >> v2:172.21.15.191:3300/0 conn(0x7f2bdc0c35e0 msgr2=0x7f2bdc0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:20.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be2584700 1 --2- 172.21.15.175:0/4158012248 >> v2:172.21.15.191:3300/0 conn(0x7f2bdc0c35e0 0x7f2bdc0c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:20.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be2584700 1 -- 172.21.15.175:0/4158012248 >> v1:172.21.15.191:6789/0 conn(0x7f2bdc0c9390 legacy=0x7f2bdc0c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:20.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be2584700 1 -- 172.21.15.175:0/4158012248 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc0c9880 con 0x7f2bdc0c3210 2023-10-10T19:22:20.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.948+0000 7f2be2584700 1 --2- 172.21.15.175:0/4158012248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c3210 0x7f2bdc159fa0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc064890 tx=0x7f2bd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9258276f6c1f17 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:20.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be1582700 1 -- 172.21.15.175:0/4158012248 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8009ea0 con 0x7f2bdc0c3210 2023-10-10T19:22:20.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be1582700 1 -- 172.21.15.175:0/4158012248 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd800b820 con 0x7f2bdc0c3210 2023-10-10T19:22:20.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be1582700 1 -- 172.21.15.175:0/4158012248 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd800bbc0 con 0x7f2bdc0c3210 2023-10-10T19:22:20.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be47e8700 1 -- 172.21.15.175:0/4158012248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c3210 msgr2=0x7f2bdc159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:20.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be47e8700 1 --2- 172.21.15.175:0/4158012248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c3210 0x7f2bdc159fa0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc064890 tx=0x7f2bd8009250 comp rx=0 tx=0).stop 2023-10-10T19:22:20.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be47e8700 1 -- 172.21.15.175:0/4158012248 shutdown_connections 2023-10-10T19:22:20.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be47e8700 1 --2- 172.21.15.175:0/4158012248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c3210 0x7f2bdc159fa0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:20.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be47e8700 1 --2- 172.21.15.175:0/4158012248 >> v2:172.21.15.191:3300/0 conn(0x7f2bdc0c35e0 0x7f2bdc0c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:20.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be47e8700 1 -- 172.21.15.175:0/4158012248 >> 172.21.15.175:0/4158012248 conn(0x7f2bdc06c110 msgr2=0x7f2bdc06c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:20.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be47e8700 1 -- 172.21.15.175:0/4158012248 shutdown_connections 2023-10-10T19:22:20.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.951+0000 7f2be47e8700 1 -- 172.21.15.175:0/4158012248 wait complete. 2023-10-10T19:22:20.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.952+0000 7f2be47e8700 1 Processor -- start 2023-10-10T19:22:20.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.952+0000 7f2be47e8700 1 -- start start 2023-10-10T19:22:20.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be47e8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bdc0c3210 0x7f2bdc166460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:20.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be47e8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c35e0 0x7f2bdc13ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:20.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be47e8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2bdc0c9390 0x7f2bdc156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:20.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be47e8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc0c5840 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be47e8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc154740 con 0x7f2bdc0c3210 2023-10-10T19:22:20.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be47e8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc159a60 con 0x7f2bdc0c9390 2023-10-10T19:22:20.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be1d83700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c35e0 0x7f2bdc13ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:20.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be1d83700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c35e0 0x7f2bdc13ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40828/0 (socket says 172.21.15.175:40828) 2023-10-10T19:22:20.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be1d83700 1 -- 172.21.15.175:0/2162203732 learned_addr learned my addr 172.21.15.175:0/2162203732 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:20.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be2584700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bdc0c3210 0x7f2bdc166460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:20.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.953+0000 7f2be2d85700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2bdc0c9390 0x7f2bdc156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:20.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2be1d83700 1 -- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2bdc0c9390 msgr2=0x7f2bdc156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:20.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2be1d83700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2bdc0c9390 0x7f2bdc156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:20.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2be1d83700 1 -- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bdc0c3210 msgr2=0x7f2bdc166460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:20.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2be1d83700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bdc0c3210 0x7f2bdc166460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:20.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2be1d83700 1 -- 172.21.15.175:0/2162203732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd80094f0 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2be1d83700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c35e0 0x7f2bdc13ad80 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00ecb0 tx=0x7f2bcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:20.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2bd27fc700 1 -- 172.21.15.175:0/2162203732 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc00c920 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2bd27fc700 1 -- 172.21.15.175:0/2162203732 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bcc00ca80 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc165fd0 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2bd27fc700 1 -- 172.21.15.175:0/2162203732 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc00ce20 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.954+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc159cb0 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.955+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc067570 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.955+0000 7f2bd27fc700 1 -- 172.21.15.175:0/2162203732 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2bcc013680 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.956+0000 7f2bd27fc700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2bc0066f30 0x7f2bc00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:20.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.956+0000 7f2bd27fc700 1 -- 172.21.15.175:0/2162203732 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f2bcc084f00 con 0x7f2bdc0c35e0 2023-10-10T19:22:20.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.956+0000 7f2be2584700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2bc0066f30 0x7f2bc00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:20.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.956+0000 7f2be2584700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2bc0066f30 0x7f2bc00693e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc064890 tx=0x7f2bd8017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:20.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:20.960+0000 7f2bd27fc700 1 -- 172.21.15.175:0/2162203732 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2bcc051af0 con 0x7f2bdc0c35e0 2023-10-10T19:22:21.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.184+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bdc06ea70 con 0x7f2bdc0c35e0 2023-10-10T19:22:21.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.184+0000 7f2bd27fc700 1 -- 172.21.15.175:0/2162203732 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f2bcc055110 con 0x7f2bdc0c35e0 2023-10-10T19:22:21.186 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:21.186 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":277}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:21.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2bc0066f30 msgr2=0x7f2bc00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:21.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2bc0066f30 0x7f2bc00693e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc064890 tx=0x7f2bd8017000 comp rx=0 tx=0).stop 2023-10-10T19:22:21.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c35e0 msgr2=0x7f2bdc13ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:21.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c35e0 0x7f2bdc13ad80 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00ecb0 tx=0x7f2bcc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:22:21.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 shutdown_connections 2023-10-10T19:22:21.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2bc0066f30 0x7f2bc00693e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:21.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2bdc0c9390 0x7f2bdc156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:21.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bdc0c3210 0x7f2bdc166460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:21.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 --2- 172.21.15.175:0/2162203732 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2bdc0c35e0 0x7f2bdc13ad80 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:21.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 >> 172.21.15.175:0/2162203732 conn(0x7f2bdc06c110 msgr2=0x7f2bdc0c1350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:21.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 shutdown_connections 2023-10-10T19:22:21.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:21.186+0000 7f2be47e8700 1 -- 172.21.15.175:0/2162203732 wait complete. 2023-10-10T19:22:21.191 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 277 2023-10-10T19:22:21.200 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:22:23.201 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:23.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.327+0000 7f8d11ff8700 1 Processor -- start 2023-10-10T19:22:23.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.327+0000 7f8d11ff8700 1 -- start start 2023-10-10T19:22:23.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.328+0000 7f8d11ff8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8d0c0c12d0 0x7f8d0c0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:23.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.328+0000 7f8d11ff8700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c0c6fb0 con 0x7f8d0c0c5850 2023-10-10T19:22:23.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.328+0000 7f8d11ff8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c0c6160 con 0x7f8d0c0c4f20 2023-10-10T19:22:23.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.328+0000 7f8d11ff8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0c063ea0 con 0x7f8d0c0c12d0 2023-10-10T19:22:23.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.328+0000 7f8d0b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8d0c0c5850 0x7f8d0c156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:39462/0 (socket says 172.21.15.175:39462) 2023-10-10T19:22:23.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.328+0000 7f8d0b7fe700 1 -- 172.21.15.175:0/1546484470 learned_addr learned my addr 172.21.15.175:0/1546484470 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:23.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0affd700 1 --2- 172.21.15.175:0/1546484470 >> v2:172.21.15.191:3300/0 conn(0x7f8d0c0c12d0 0x7f8d0c0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:23.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0a7fc700 1 -- 172.21.15.175:0/1546484470 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224190885 0 0) 0x7f8d0c0c6fb0 con 0x7f8d0c0c5850 2023-10-10T19:22:23.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0a7fc700 1 -- 172.21.15.175:0/1546484470 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cf4003570 con 0x7f8d0c0c5850 2023-10-10T19:22:23.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0a7fc700 1 -- 172.21.15.175:0/1546484470 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399280239 0 0) 0x7f8d0c0c6160 con 0x7f8d0c0c4f20 2023-10-10T19:22:23.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0a7fc700 1 -- 172.21.15.175:0/1546484470 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d0c0c6fb0 con 0x7f8d0c0c4f20 2023-10-10T19:22:23.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0a7fc700 1 -- 172.21.15.175:0/1546484470 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1151162396 0 0) 0x7f8cf4003570 con 0x7f8d0c0c5850 2023-10-10T19:22:23.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0affd700 1 -- 172.21.15.175:0/1546484470 >> v1:172.21.15.191:6789/0 conn(0x7f8d0c0c4f20 legacy=0x7f8d0c0c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:23.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0affd700 1 -- 172.21.15.175:0/1546484470 >> v1:172.21.15.175:6789/0 conn(0x7f8d0c0c5850 legacy=0x7f8d0c156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:23.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0affd700 1 -- 172.21.15.175:0/1546484470 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c063940 con 0x7f8d0c0c12d0 2023-10-10T19:22:23.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0affd700 1 --2- 172.21.15.175:0/1546484470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c12d0 0x7f8d0c0c5c20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8d000098e0 tx=0x7f8d000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b40e397d53a589c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:23.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0a7fc700 1 -- 172.21.15.175:0/1546484470 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc001470 con 0x7f8d0c0c12d0 2023-10-10T19:22:23.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0a7fc700 1 -- 172.21.15.175:0/1546484470 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d00011420 con 0x7f8d0c0c12d0 2023-10-10T19:22:23.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.329+0000 7f8d0a7fc700 1 -- 172.21.15.175:0/1546484470 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d000117c0 con 0x7f8d0c0c12d0 2023-10-10T19:22:23.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.330+0000 7f8d11ff8700 1 -- 172.21.15.175:0/1546484470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c12d0 msgr2=0x7f8d0c0c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:23.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.330+0000 7f8d11ff8700 1 --2- 172.21.15.175:0/1546484470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c12d0 0x7f8d0c0c5c20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8d000098e0 tx=0x7f8d000091a0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.330+0000 7f8d11ff8700 1 -- 172.21.15.175:0/1546484470 shutdown_connections 2023-10-10T19:22:23.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.330+0000 7f8d11ff8700 1 --2- 172.21.15.175:0/1546484470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c12d0 0x7f8d0c0c5c20 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.330+0000 7f8d11ff8700 1 -- 172.21.15.175:0/1546484470 >> 172.21.15.175:0/1546484470 conn(0x7f8d0c06dc70 msgr2=0x7f8d0c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:23.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.330+0000 7f8d11ff8700 1 -- 172.21.15.175:0/1546484470 shutdown_connections 2023-10-10T19:22:23.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.330+0000 7f8d11ff8700 1 -- 172.21.15.175:0/1546484470 wait complete. 2023-10-10T19:22:23.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.331+0000 7f8d11ff8700 1 Processor -- start 2023-10-10T19:22:23.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.331+0000 7f8d11ff8700 1 -- start start 2023-10-10T19:22:23.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d11ff8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d0c0c12d0 0x7f8d0c165da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:23.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d11ff8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c4f20 0x7f8d0c14c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:23.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d11ff8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0c0c5850 0x7f8d0c14c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:23.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d11ff8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c166970 con 0x7f8d0c0c5850 2023-10-10T19:22:23.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d11ff8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c14e100 con 0x7f8d0c0c4f20 2023-10-10T19:22:23.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d11ff8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0c8e10 con 0x7f8d0c0c12d0 2023-10-10T19:22:23.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0c0c5850 0x7f8d0c14c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:23.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0c0c5850 0x7f8d0c14c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40836/0 (socket says 172.21.15.175:40836) 2023-10-10T19:22:23.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 -- 172.21.15.175:0/4058497930 learned_addr learned my addr 172.21.15.175:0/4058497930 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:23.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0affd700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c4f20 0x7f8d0c14c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:23.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0b7fe700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d0c0c12d0 0x7f8d0c165da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:23.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 -- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d0c0c12d0 msgr2=0x7f8d0c165da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:23.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d0c0c12d0 0x7f8d0c165da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 -- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c4f20 msgr2=0x7f8d0c14c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:23.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c4f20 0x7f8d0c14c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 -- 172.21.15.175:0/4058497930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d00009420 con 0x7f8d0c0c5850 2023-10-10T19:22:23.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.332+0000 7f8d0bfff700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0c0c5850 0x7f8d0c14c6d0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400c910 tx=0x7f8d040027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:23.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.333+0000 7f8cf37fe700 1 -- 172.21.15.175:0/4058497930 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0400a870 con 0x7f8d0c0c5850 2023-10-10T19:22:23.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.333+0000 7f8cf37fe700 1 -- 172.21.15.175:0/4058497930 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0400a9f0 con 0x7f8d0c0c5850 2023-10-10T19:22:23.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.333+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c14ba20 con 0x7f8d0c0c5850 2023-10-10T19:22:23.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.333+0000 7f8cf37fe700 1 -- 172.21.15.175:0/4058497930 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0400adb0 con 0x7f8d0c0c5850 2023-10-10T19:22:23.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.333+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c14bd00 con 0x7f8d0c0c5850 2023-10-10T19:22:23.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.334+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0c1666f0 con 0x7f8d0c0c5850 2023-10-10T19:22:23.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.334+0000 7f8cf37fe700 1 -- 172.21.15.175:0/4058497930 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8d04012630 con 0x7f8d0c0c5850 2023-10-10T19:22:23.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.334+0000 7f8cf37fe700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cf8066f60 0x7f8cf8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:23.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.334+0000 7f8cf37fe700 1 -- 172.21.15.175:0/4058497930 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 7369+0+0 (secure 0 0 0) 0x7f8d04085080 con 0x7f8d0c0c5850 2023-10-10T19:22:23.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.335+0000 7f8d0b7fe700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cf8066f60 0x7f8cf8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:23.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.335+0000 7f8d0b7fe700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cf8066f60 0x7f8cf8069410 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc0015a0 tx=0x7f8cfc000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:23.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.339+0000 7f8cf37fe700 1 -- 172.21.15.175:0/4058497930 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d04051d20 con 0x7f8d0c0c5850 2023-10-10T19:22:23.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.568+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d0c067570 con 0x7f8d0c0c5850 2023-10-10T19:22:23.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.569+0000 7f8cf37fe700 1 -- 172.21.15.175:0/4058497930 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+6181 (secure 0 0 0) 0x7f8d04055340 con 0x7f8d0c0c5850 2023-10-10T19:22:23.570 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:23.571 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":277}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:21:28.780992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":6618},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6618":{"gid":6618,"name":"c","rank":3,"incarnation":177,"state":"up:active","state_seq":20,"addr":"172.21.15.175:6836/972555107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":972555107},{"type":"v1","addr":"172.21.15.175:6836","nonce":972555107}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:23.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cf8066f60 msgr2=0x7f8cf8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:23.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cf8066f60 0x7f8cf8069410 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc0015a0 tx=0x7f8cfc000bc0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0c0c5850 msgr2=0x7f8d0c14c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:23.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0c0c5850 0x7f8d0c14c6d0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400c910 tx=0x7f8d040027a0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 shutdown_connections 2023-10-10T19:22:23.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8cf8066f60 0x7f8cf8069410 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8d0c0c5850 0x7f8d0c14c6d0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d0c0c12d0 0x7f8d0c165da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 --2- 172.21.15.175:0/4058497930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d0c0c4f20 0x7f8d0c14c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:23.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 >> 172.21.15.175:0/4058497930 conn(0x7f8d0c06dc70 msgr2=0x7f8d0c151b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:23.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 shutdown_connections 2023-10-10T19:22:23.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:23.571+0000 7f8d11ff8700 1 -- 172.21.15.175:0/4058497930 wait complete. 2023-10-10T19:22:23.575 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 277 2023-10-10T19:22:23.584 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-10T19:22:25.450 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:22:25.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:22:25.585 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:25.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.705+0000 7fba2ffff700 1 Processor -- start 2023-10-10T19:22:25.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.705+0000 7fba2ffff700 1 -- start start 2023-10-10T19:22:25.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 0x7fba30159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:25.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2ffff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba30067fa0 con 0x7fba3008a100 2023-10-10T19:22:25.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2ffff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba30156b70 con 0x7fba3014fd90 2023-10-10T19:22:25.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2ffff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba300c5600 con 0x7fba30068c00 2023-10-10T19:22:25.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2e7fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fba3008a100 0x7fba300684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33194/0 (socket says 172.21.15.175:33194) 2023-10-10T19:22:25.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2e7fc700 1 -- 172.21.15.175:0/4272359054 learned_addr learned my addr 172.21.15.175:0/4272359054 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:25.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577032248 0 0) 0x7fba30067fa0 con 0x7fba3008a100 2023-10-10T19:22:25.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba1c003570 con 0x7fba3008a100 2023-10-10T19:22:25.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2f7fe700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 0x7fba30159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:25.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2448642432 0 0) 0x7fba1c003570 con 0x7fba3008a100 2023-10-10T19:22:25.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 msgr2=0x7fba30159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:25.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 0x7fba30159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 >> v1:172.21.15.191:6789/0 conn(0x7fba3014fd90 legacy=0x7fba300c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:25.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba30150160 con 0x7fba3008a100 2023-10-10T19:22:25.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fba18001470 con 0x7fba3008a100 2023-10-10T19:22:25.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2f7fe700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 0x7fba30159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:22:25.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.706+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 >> v1:172.21.15.175:6789/0 conn(0x7fba3008a100 legacy=0x7fba300684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:25.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 0x7fba1c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:25.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba1c004080 0x7fba1c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:25.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:25.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba300c5600 con 0x7fba1c004d50 2023-10-10T19:22:25.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c00bca0 con 0x7fba1c004080 2023-10-10T19:22:25.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c00be60 con 0x7fba30068c00 2023-10-10T19:22:25.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2effd700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 0x7fba1c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:25.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2e7fc700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba1c004080 0x7fba1c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:25.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2f7fe700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:25.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2f7fe700 1 -- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 msgr2=0x7fba1c003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:25.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2f7fe700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30068c00 0x7fba1c003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2f7fe700 1 -- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba1c004080 msgr2=0x7fba1c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:25.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2f7fe700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba1c004080 0x7fba1c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2f7fe700 1 -- 172.21.15.175:0/4272359054 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c003570 con 0x7fba1c004d50 2023-10-10T19:22:25.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2f7fe700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba1c0051c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fba28006fd0 tx=0x7fba28008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d27fac7e2ed35b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:25.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2800c070 con 0x7fba1c004d50 2023-10-10T19:22:25.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2800e5b0 con 0x7fba1c004d50 2023-10-10T19:22:25.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.707+0000 7fba2dffb700 1 -- 172.21.15.175:0/4272359054 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2800e760 con 0x7fba1c004d50 2023-10-10T19:22:25.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.708+0000 7fba2ffff700 1 -- 172.21.15.175:0/4272359054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 msgr2=0x7fba1c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:25.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.708+0000 7fba2ffff700 1 --2- 172.21.15.175:0/4272359054 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba1c0051c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fba28006fd0 tx=0x7fba28008040 comp rx=0 tx=0).stop 2023-10-10T19:22:25.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.708+0000 7fba2f7fe700 1 -- 172.21.15.175:0/4272359054 reap_dead start 2023-10-10T19:22:25.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.708+0000 7fba2ffff700 1 -- 172.21.15.175:0/4272359054 shutdown_connections 2023-10-10T19:22:25.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.708+0000 7fba2ffff700 1 -- 172.21.15.175:0/4272359054 >> 172.21.15.175:0/4272359054 conn(0x7fba30063d00 msgr2=0x7fba300cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:25.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.708+0000 7fba2ffff700 1 -- 172.21.15.175:0/4272359054 shutdown_connections 2023-10-10T19:22:25.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.708+0000 7fba2ffff700 1 -- 172.21.15.175:0/4272359054 wait complete. 2023-10-10T19:22:25.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.709+0000 7fba2ffff700 1 Processor -- start 2023-10-10T19:22:25.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.709+0000 7fba2ffff700 1 -- start start 2023-10-10T19:22:25.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba30158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:25.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2ffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba30153320 0x7fba30156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:25.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2ffff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30150550 0x7fba30149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:25.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2ffff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba30150c50 con 0x7fba1c004d50 2023-10-10T19:22:25.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2ffff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba30152ac0 con 0x7fba30153320 2023-10-10T19:22:25.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2ffff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba30165ce0 con 0x7fba30150550 2023-10-10T19:22:25.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba30158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:25.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba30158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56306/0 (socket says 172.21.15.175:56306) 2023-10-10T19:22:25.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 -- 172.21.15.175:0/518188042 learned_addr learned my addr 172.21.15.175:0/518188042 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:25.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 -- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30150550 msgr2=0x7fba30149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:25.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30150550 0x7fba30149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 -- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba30153320 msgr2=0x7fba30156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:25.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba30153320 0x7fba30156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 -- 172.21.15.175:0/518188042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba30151ef0 con 0x7fba1c004d50 2023-10-10T19:22:25.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba2effd700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba30158d30 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fba2000ebf0 tx=0x7fba20001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:25.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.710+0000 7fba26ffd700 1 -- 172.21.15.175:0/518188042 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2000c960 con 0x7fba1c004d50 2023-10-10T19:22:25.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.711+0000 7fba26ffd700 1 -- 172.21.15.175:0/518188042 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2000cac0 con 0x7fba1c004d50 2023-10-10T19:22:25.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.711+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba30152190 con 0x7fba1c004d50 2023-10-10T19:22:25.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.711+0000 7fba26ffd700 1 -- 172.21.15.175:0/518188042 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2000ce60 con 0x7fba1c004d50 2023-10-10T19:22:25.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.711+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba30152750 con 0x7fba1c004d50 2023-10-10T19:22:25.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.711+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba30067570 con 0x7fba1c004d50 2023-10-10T19:22:25.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.712+0000 7fba26ffd700 1 -- 172.21.15.175:0/518188042 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fba20015640 con 0x7fba1c004d50 2023-10-10T19:22:25.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.712+0000 7fba26ffd700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:25.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.712+0000 7fba26ffd700 1 -- 172.21.15.175:0/518188042 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fba20085480 con 0x7fba1c004d50 2023-10-10T19:22:25.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.712+0000 7fba2e7fc700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:25.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.713+0000 7fba2e7fc700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fba18007a70 tx=0x7fba300c5dd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:25.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.716+0000 7fba26ffd700 1 -- 172.21.15.175:0/518188042 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba20052020 con 0x7fba1c004d50 2023-10-10T19:22:25.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.942+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba30155f80 con 0x7fba1c004d50 2023-10-10T19:22:25.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.942+0000 7fba26ffd700 1 -- 172.21.15.175:0/518188042 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+5422 (secure 0 0 0) 0x7fba20055640 con 0x7fba1c004d50 2023-10-10T19:22:25.944 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:25.944 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":278,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:23.833930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:25.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 msgr2=0x7fba10069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:25.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fba18007a70 tx=0x7fba300c5dd0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 msgr2=0x7fba30158d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:25.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba30158d30 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fba2000ebf0 tx=0x7fba20001670 comp rx=0 tx=0).stop 2023-10-10T19:22:25.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 shutdown_connections 2023-10-10T19:22:25.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fba10066f60 0x7fba10069410 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba30150550 0x7fba30149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fba1c004d50 0x7fba30158d30 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 --2- 172.21.15.175:0/518188042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba30153320 0x7fba30156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:25.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 >> 172.21.15.175:0/518188042 conn(0x7fba30063d00 msgr2=0x7fba300c3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:25.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 shutdown_connections 2023-10-10T19:22:25.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:25.944+0000 7fba2ffff700 1 -- 172.21.15.175:0/518188042 wait complete. 2023-10-10T19:22:25.949 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 278 2023-10-10T19:22:25.958 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-10T19:22:25.958 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:22:25.958 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:26.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.073+0000 7fdde88c6700 1 Processor -- start 2023-10-10T19:22:26.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.073+0000 7fdde88c6700 1 -- start start 2023-10-10T19:22:26.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde88c6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdde414fd90 0x7fdde4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:26.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde88c6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdde408a100 0x7fdde40c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:26.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde88c6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde4067fa0 con 0x7fdde40cd040 2023-10-10T19:22:26.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde88c6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40c5600 con 0x7fdde414fd90 2023-10-10T19:22:26.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde88c6700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde4156b70 con 0x7fdde408a100 2023-10-10T19:22:26.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde2d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdde414fd90 0x7fdde4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:26.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde359e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fdde40cd040 0x7fdde40684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33196/0 (socket says 172.21.15.175:33196) 2023-10-10T19:22:26.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde359e700 1 -- 172.21.15.175:0/884705823 learned_addr learned my addr 172.21.15.175:0/884705823 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:26.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fddda59c700 1 --2- 172.21.15.175:0/884705823 >> v2:172.21.15.191:3300/0 conn(0x7fdde408a100 0x7fdde40c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:26.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde2d9d700 1 -- 172.21.15.175:0/884705823 >> v2:172.21.15.191:3300/0 conn(0x7fdde408a100 msgr2=0x7fdde40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:26.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde2d9d700 1 --2- 172.21.15.175:0/884705823 >> v2:172.21.15.191:3300/0 conn(0x7fdde408a100 0x7fdde40c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde2d9d700 1 -- 172.21.15.175:0/884705823 >> v1:172.21.15.175:6789/0 conn(0x7fdde40cd040 legacy=0x7fdde40684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:26.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde2d9d700 1 -- 172.21.15.175:0/884705823 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4150160 con 0x7fdde414fd90 2023-10-10T19:22:26.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde259c700 1 -- 172.21.15.175:0/884705823 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863649869 0 0) 0x7fdde4067fa0 con 0x7fdde40cd040 2023-10-10T19:22:26.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.074+0000 7fdde2d9d700 1 --2- 172.21.15.175:0/884705823 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde414fd90 0x7fdde4159c30 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fdde4069990 tx=0x7fddd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe96d498469a5cf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:26.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde259c700 1 -- 172.21.15.175:0/884705823 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd400b740 con 0x7fdde414fd90 2023-10-10T19:22:26.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde259c700 1 -- 172.21.15.175:0/884705823 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd400b8a0 con 0x7fdde414fd90 2023-10-10T19:22:26.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde259c700 1 -- 172.21.15.175:0/884705823 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd400bc40 con 0x7fdde414fd90 2023-10-10T19:22:26.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde88c6700 1 -- 172.21.15.175:0/884705823 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde414fd90 msgr2=0x7fdde4159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:26.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde88c6700 1 --2- 172.21.15.175:0/884705823 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde414fd90 0x7fdde4159c30 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fdde4069990 tx=0x7fddd40091a0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde88c6700 1 -- 172.21.15.175:0/884705823 shutdown_connections 2023-10-10T19:22:26.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde88c6700 1 --2- 172.21.15.175:0/884705823 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde414fd90 0x7fdde4159c30 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde88c6700 1 --2- 172.21.15.175:0/884705823 >> v2:172.21.15.191:3300/0 conn(0x7fdde408a100 0x7fdde40c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde88c6700 1 -- 172.21.15.175:0/884705823 >> 172.21.15.175:0/884705823 conn(0x7fdde4063d00 msgr2=0x7fdde40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:26.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde88c6700 1 -- 172.21.15.175:0/884705823 shutdown_connections 2023-10-10T19:22:26.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.075+0000 7fdde88c6700 1 -- 172.21.15.175:0/884705823 wait complete. 2023-10-10T19:22:26.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.076+0000 7fdde88c6700 1 Processor -- start 2023-10-10T19:22:26.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.076+0000 7fdde88c6700 1 -- start start 2023-10-10T19:22:26.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fdde88c6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdde408a100 0x7fdde416d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:26.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fdde88c6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde40cd040 0x7fdde4155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:26.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fdde88c6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdde414fd90 0x7fdde414c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:26.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fdde88c6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40cb730 con 0x7fdde40cd040 2023-10-10T19:22:26.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fdde88c6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40ccc20 con 0x7fdde408a100 2023-10-10T19:22:26.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fdde88c6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40ca8c0 con 0x7fdde414fd90 2023-10-10T19:22:26.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fddda59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde40cd040 0x7fdde4155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:26.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fddda59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde40cd040 0x7fdde4155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56326/0 (socket says 172.21.15.175:56326) 2023-10-10T19:22:26.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fddda59c700 1 -- 172.21.15.175:0/2365790835 learned_addr learned my addr 172.21.15.175:0/2365790835 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:26.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fdde2d9d700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdde408a100 0x7fdde416d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:26.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fdde359e700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdde414fd90 0x7fdde414c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:26.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fddda59c700 1 -- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdde414fd90 msgr2=0x7fdde414c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:26.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fddda59c700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdde414fd90 0x7fdde414c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fddda59c700 1 -- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdde408a100 msgr2=0x7fdde416d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:26.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fddda59c700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdde408a100 0x7fdde416d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.077+0000 7fddda59c700 1 -- 172.21.15.175:0/2365790835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd4009420 con 0x7fdde40cd040 2023-10-10T19:22:26.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.078+0000 7fddda59c700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde40cd040 0x7fdde4155210 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00d8c0 tx=0x7fddcc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:26.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.078+0000 7fdddb7fe700 1 -- 172.21.15.175:0/2365790835 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc0125f0 con 0x7fdde40cd040 2023-10-10T19:22:26.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.078+0000 7fdddb7fe700 1 -- 172.21.15.175:0/2365790835 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc012750 con 0x7fdde40cd040 2023-10-10T19:22:26.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.078+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde4157830 con 0x7fdde40cd040 2023-10-10T19:22:26.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.078+0000 7fdddb7fe700 1 -- 172.21.15.175:0/2365790835 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc012af0 con 0x7fdde40cd040 2023-10-10T19:22:26.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.078+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde40cc940 con 0x7fdde40cd040 2023-10-10T19:22:26.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.079+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde4067570 con 0x7fdde40cd040 2023-10-10T19:22:26.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.079+0000 7fdddb7fe700 1 -- 172.21.15.175:0/2365790835 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fddcc01e460 con 0x7fdde40cd040 2023-10-10T19:22:26.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.079+0000 7fdddb7fe700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fddc4066f60 0x7fddc4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:26.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.080+0000 7fdddb7fe700 1 -- 172.21.15.175:0/2365790835 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fddcc085100 con 0x7fdde40cd040 2023-10-10T19:22:26.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.080+0000 7fdde2d9d700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fddc4066f60 0x7fddc4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:26.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.080+0000 7fdde2d9d700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fddc4066f60 0x7fddc4069410 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fddd40093f0 tx=0x7fddd4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:26.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.084+0000 7fdddb7fe700 1 -- 172.21.15.175:0/2365790835 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fddcc055730 con 0x7fdde40cd040 2023-10-10T19:22:26.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.309+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdde40caa00 con 0x7fdde40cd040 2023-10-10T19:22:26.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.309+0000 7fdddb7fe700 1 -- 172.21.15.175:0/2365790835 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+5422 (secure 0 0 0) 0x7fddcc0552c0 con 0x7fdde40cd040 2023-10-10T19:22:26.311 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:26.312 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":278,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:23.833930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:26.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.311+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fddc4066f60 msgr2=0x7fddc4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:26.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.311+0000 7fdde88c6700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fddc4066f60 0x7fddc4069410 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fddd40093f0 tx=0x7fddd4017000 comp rx=0 tx=0).stop 2023-10-10T19:22:26.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.311+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde40cd040 msgr2=0x7fdde4155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:26.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.311+0000 7fdde88c6700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde40cd040 0x7fdde4155210 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00d8c0 tx=0x7fddcc008040 comp rx=0 tx=0).stop 2023-10-10T19:22:26.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.312+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 shutdown_connections 2023-10-10T19:22:26.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.312+0000 7fdde88c6700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fddc4066f60 0x7fddc4069410 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.312+0000 7fdde88c6700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdde414fd90 0x7fdde414c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.312+0000 7fdde88c6700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdde408a100 0x7fdde416d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.312+0000 7fdde88c6700 1 --2- 172.21.15.175:0/2365790835 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdde40cd040 0x7fdde4155210 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:26.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.312+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 >> 172.21.15.175:0/2365790835 conn(0x7fdde4063d00 msgr2=0x7fdde40c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:26.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.312+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 shutdown_connections 2023-10-10T19:22:26.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:26.312+0000 7fdde88c6700 1 -- 172.21.15.175:0/2365790835 wait complete. 2023-10-10T19:22:26.316 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 278 2023-10-10T19:22:26.325 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 278, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 278, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7252": { "addr": "172.21.15.191:6839/2393527600", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 2393527600, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 2393527600, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7252, "incarnation": 271, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 20 }, "gid_7384": { "addr": "172.21.15.191:6837/2914004704", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 2914004704, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2914004704, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7384, "incarnation": 278, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:replay", "state_seq": 1 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 90 } }, "last_failure": 0, "last_failure_osd_epoch": 127, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:22:23.833930+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 7252, "mds_1": 7401, "mds_2": 7104, "mds_3": 7384 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:22:28.327 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:28.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.451+0000 7fcb81ffa700 1 Processor -- start 2023-10-10T19:22:28.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.451+0000 7fcb81ffa700 1 -- start start 2023-10-10T19:22:28.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb81ffa700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcb7c0b2430 0x7fcb7c14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:28.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb81ffa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c150290 0x7fcb7c155f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:28.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb81ffa700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c0b0130 con 0x7fcb7c0b2060 2023-10-10T19:22:28.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb81ffa700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c150710 con 0x7fcb7c0b2430 2023-10-10T19:22:28.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb81ffa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c156c90 con 0x7fcb7c150290 2023-10-10T19:22:28.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb7affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcb7c0b2430 0x7fcb7c14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:28.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb7affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcb7c0b2430 0x7fcb7c14fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56340/0 (socket says 172.21.15.175:56340) 2023-10-10T19:22:28.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb7affd700 1 -- 172.21.15.175:0/1196528706 learned_addr learned my addr 172.21.15.175:0/1196528706 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:28.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.452+0000 7fcb7bfff700 1 --2- 172.21.15.175:0/1196528706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c150290 0x7fcb7c155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:28.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb7affd700 1 -- 172.21.15.175:0/1196528706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c150290 msgr2=0x7fcb7c155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:28.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb7affd700 1 --2- 172.21.15.175:0/1196528706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c150290 0x7fcb7c155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb7affd700 1 -- 172.21.15.175:0/1196528706 >> v1:172.21.15.191:6789/0 conn(0x7fcb7c0b2060 legacy=0x7fcb7c157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:28.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb7affd700 1 -- 172.21.15.175:0/1196528706 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0af800 con 0x7fcb7c0b2430 2023-10-10T19:22:28.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb7affd700 1 --2- 172.21.15.175:0/1196528706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2430 0x7fcb7c14fd50 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fcb700099b0 tx=0x7fcb70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4390c90a9ef4f81 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:28.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb7a7fc700 1 -- 172.21.15.175:0/1196528706 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7000b860 con 0x7fcb7c0b2430 2023-10-10T19:22:28.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb7a7fc700 1 -- 172.21.15.175:0/1196528706 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7000b9c0 con 0x7fcb7c0b2430 2023-10-10T19:22:28.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb7a7fc700 1 -- 172.21.15.175:0/1196528706 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7000bd60 con 0x7fcb7c0b2430 2023-10-10T19:22:28.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1196528706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2430 msgr2=0x7fcb7c14fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:28.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1196528706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2430 0x7fcb7c14fd50 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fcb700099b0 tx=0x7fcb70009250 comp rx=0 tx=0).stop 2023-10-10T19:22:28.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1196528706 shutdown_connections 2023-10-10T19:22:28.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1196528706 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c150290 0x7fcb7c155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1196528706 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2430 0x7fcb7c14fd50 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.453+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1196528706 >> 172.21.15.175:0/1196528706 conn(0x7fcb7c063230 msgr2=0x7fcb7c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:28.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.454+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1196528706 shutdown_connections 2023-10-10T19:22:28.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.454+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1196528706 wait complete. 2023-10-10T19:22:28.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.455+0000 7fcb81ffa700 1 Processor -- start 2023-10-10T19:22:28.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.455+0000 7fcb81ffa700 1 -- start start 2023-10-10T19:22:28.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.455+0000 7fcb81ffa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2060 0x7fcb7c0c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:28.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.455+0000 7fcb81ffa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c0b2430 0x7fcb7c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:28.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.455+0000 7fcb81ffa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb7c150290 0x7fcb7c0c28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:28.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.455+0000 7fcb81ffa700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c155810 con 0x7fcb7c0b2060 2023-10-10T19:22:28.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.455+0000 7fcb81ffa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c169410 con 0x7fcb7c150290 2023-10-10T19:22:28.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.455+0000 7fcb81ffa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c09b0 con 0x7fcb7c0b2430 2023-10-10T19:22:28.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2060 0x7fcb7c0c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:28.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2060 0x7fcb7c0c0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56342/0 (socket says 172.21.15.175:56342) 2023-10-10T19:22:28.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 -- 172.21.15.175:0/1887862020 learned_addr learned my addr 172.21.15.175:0/1887862020 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:28.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7affd700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c0b2430 0x7fcb7c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:28.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7bfff700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb7c150290 0x7fcb7c0c28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:28.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 -- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c0b2430 msgr2=0x7fcb7c1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:28.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c0b2430 0x7fcb7c1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 -- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb7c150290 msgr2=0x7fcb7c0c28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:28.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb7c150290 0x7fcb7c0c28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 -- 172.21.15.175:0/1887862020 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb700094f0 con 0x7fcb7c0b2060 2023-10-10T19:22:28.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb7b7fe700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2060 0x7fcb7c0c0e40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c5c0 tx=0x7fcb6c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:28.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb637fe700 1 -- 172.21.15.175:0/1887862020 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c011590 con 0x7fcb7c0b2060 2023-10-10T19:22:28.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb637fe700 1 -- 172.21.15.175:0/1887862020 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c011710 con 0x7fcb7c0b2060 2023-10-10T19:22:28.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c0c1380 con 0x7fcb7c0b2060 2023-10-10T19:22:28.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb637fe700 1 -- 172.21.15.175:0/1887862020 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c011ad0 con 0x7fcb7c0b2060 2023-10-10T19:22:28.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.456+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c068fc0 con 0x7fcb7c0b2060 2023-10-10T19:22:28.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.457+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c0c0c90 con 0x7fcb7c0b2060 2023-10-10T19:22:28.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.458+0000 7fcb637fe700 1 -- 172.21.15.175:0/1887862020 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcb6c01b3b0 con 0x7fcb7c0b2060 2023-10-10T19:22:28.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.458+0000 7fcb637fe700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb68066f30 0x7fcb680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:28.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.458+0000 7fcb637fe700 1 -- 172.21.15.175:0/1887862020 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fcb6c084010 con 0x7fcb7c0b2060 2023-10-10T19:22:28.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.458+0000 7fcb7affd700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb68066f30 0x7fcb680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:28.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.459+0000 7fcb7affd700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb68066f30 0x7fcb680693e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcb700099b0 tx=0x7fcb70017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:28.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.462+0000 7fcb637fe700 1 -- 172.21.15.175:0/1887862020 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb6c050bb0 con 0x7fcb7c0b2060 2023-10-10T19:22:28.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.685+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb7c0675e0 con 0x7fcb7c0b2060 2023-10-10T19:22:28.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.686+0000 7fcb637fe700 1 -- 172.21.15.175:0/1887862020 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+5422 (secure 0 0 0) 0x7fcb7c0675e0 con 0x7fcb7c0b2060 2023-10-10T19:22:28.700 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:28.700 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":278,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:23.833930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:28.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb68066f30 msgr2=0x7fcb680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:28.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb68066f30 0x7fcb680693e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcb700099b0 tx=0x7fcb70017000 comp rx=0 tx=0).stop 2023-10-10T19:22:28.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2060 msgr2=0x7fcb7c0c0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:28.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2060 0x7fcb7c0c0e40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c5c0 tx=0x7fcb6c00c8d0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 shutdown_connections 2023-10-10T19:22:28.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcb68066f30 0x7fcb680693e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb7c150290 0x7fcb7c0c28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb7c0b2060 0x7fcb7c0c0e40 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 --2- 172.21.15.175:0/1887862020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb7c0b2430 0x7fcb7c1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:28.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 >> 172.21.15.175:0/1887862020 conn(0x7fcb7c063230 msgr2=0x7fcb7c156110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:28.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 shutdown_connections 2023-10-10T19:22:28.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:28.688+0000 7fcb81ffa700 1 -- 172.21.15.175:0/1887862020 wait complete. 2023-10-10T19:22:28.704 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 278 2023-10-10T19:22:28.704 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:22:30.705 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:30.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.823+0000 7fda0ae9f700 1 Processor -- start 2023-10-10T19:22:30.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.823+0000 7fda0ae9f700 1 -- start start 2023-10-10T19:22:30.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0ae9f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda0408a100 0x7fda04159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:30.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0ae9f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda040cd040 0x7fda040684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:30.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0ae9f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda04067fa0 con 0x7fda0414fd90 2023-10-10T19:22:30.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0ae9f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda040c5600 con 0x7fda0408a100 2023-10-10T19:22:30.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0ae9f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda04156b70 con 0x7fda040cd040 2023-10-10T19:22:30.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0a69e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda040cd040 0x7fda040684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:30.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0969c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda0408a100 0x7fda04159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:30.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda09e9d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fda0414fd90 0x7fda040c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:39272/0 (socket says 172.21.15.175:39272) 2023-10-10T19:22:30.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda09e9d700 1 -- 172.21.15.175:0/666050451 learned_addr learned my addr 172.21.15.175:0/666050451 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:30.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0a69e700 1 -- 172.21.15.175:0/666050451 >> v2:172.21.15.191:3300/0 conn(0x7fda0408a100 msgr2=0x7fda04159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:30.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.824+0000 7fda0a69e700 1 --2- 172.21.15.175:0/666050451 >> v2:172.21.15.191:3300/0 conn(0x7fda0408a100 0x7fda04159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:30.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0a69e700 1 -- 172.21.15.175:0/666050451 >> v1:172.21.15.191:6789/0 conn(0x7fda0414fd90 legacy=0x7fda040c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:30.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda08e9b700 1 -- 172.21.15.175:0/666050451 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849355228 0 0) 0x7fda04067fa0 con 0x7fda0414fd90 2023-10-10T19:22:30.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0a69e700 1 -- 172.21.15.175:0/666050451 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda04150160 con 0x7fda040cd040 2023-10-10T19:22:30.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0969c700 1 --2- 172.21.15.175:0/666050451 >> v2:172.21.15.191:3300/0 conn(0x7fda0408a100 0x7fda04159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:22:30.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0a69e700 1 --2- 172.21.15.175:0/666050451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda040cd040 0x7fda040684f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009e60 tx=0x7fd9fc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77198c1680499ea server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:30.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda08e9b700 1 -- 172.21.15.175:0/666050451 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc00e070 con 0x7fda040cd040 2023-10-10T19:22:30.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda08e9b700 1 -- 172.21.15.175:0/666050451 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc00bb60 con 0x7fda040cd040 2023-10-10T19:22:30.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda08e9b700 1 -- 172.21.15.175:0/666050451 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc012430 con 0x7fda040cd040 2023-10-10T19:22:30.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0ae9f700 1 -- 172.21.15.175:0/666050451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda040cd040 msgr2=0x7fda040684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:30.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/666050451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda040cd040 0x7fda040684f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009e60 tx=0x7fd9fc009780 comp rx=0 tx=0).stop 2023-10-10T19:22:30.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0ae9f700 1 -- 172.21.15.175:0/666050451 shutdown_connections 2023-10-10T19:22:30.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/666050451 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda040cd040 0x7fda040684f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:30.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/666050451 >> v2:172.21.15.191:3300/0 conn(0x7fda0408a100 0x7fda04159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:30.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0ae9f700 1 -- 172.21.15.175:0/666050451 >> 172.21.15.175:0/666050451 conn(0x7fda04063d00 msgr2=0x7fda040cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:30.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0ae9f700 1 -- 172.21.15.175:0/666050451 shutdown_connections 2023-10-10T19:22:30.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.825+0000 7fda0ae9f700 1 -- 172.21.15.175:0/666050451 wait complete. 2023-10-10T19:22:30.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.826+0000 7fda0ae9f700 1 Processor -- start 2023-10-10T19:22:30.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.826+0000 7fda0ae9f700 1 -- start start 2023-10-10T19:22:30.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0ae9f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda0408a100 0x7fda04168d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:30.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0ae9f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda040cd040 0x7fda0414c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:30.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0ae9f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda0414fd90 0x7fda0414cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:30.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0ae9f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda040a81b0 con 0x7fda0414fd90 2023-10-10T19:22:30.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0ae9f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda041692a0 con 0x7fda040cd040 2023-10-10T19:22:30.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0ae9f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda04151a30 con 0x7fda0408a100 2023-10-10T19:22:30.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0a69e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda0414fd90 0x7fda0414cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:30.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0a69e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda0414fd90 0x7fda0414cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56352/0 (socket says 172.21.15.175:56352) 2023-10-10T19:22:30.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0a69e700 1 -- 172.21.15.175:0/2774382144 learned_addr learned my addr 172.21.15.175:0/2774382144 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:30.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda0969c700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda040cd040 0x7fda0414c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:30.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.827+0000 7fda09e9d700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda0408a100 0x7fda04168d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:30.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fda0a69e700 1 -- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda0408a100 msgr2=0x7fda04168d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:30.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fda0a69e700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda0408a100 0x7fda04168d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:30.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fda0a69e700 1 -- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda040cd040 msgr2=0x7fda0414c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:30.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fda0a69e700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda040cd040 0x7fda0414c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:30.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fda0a69e700 1 -- 172.21.15.175:0/2774382144 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f4009420 con 0x7fda0414fd90 2023-10-10T19:22:30.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fda0a69e700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda0414fd90 0x7fda0414cb80 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc012a50 tx=0x7fd9fc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:30.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fd9f9ffb700 1 -- 172.21.15.175:0/2774382144 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc027470 con 0x7fda0414fd90 2023-10-10T19:22:30.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fd9f9ffb700 1 -- 172.21.15.175:0/2774382144 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc0275d0 con 0x7fda0414fd90 2023-10-10T19:22:30.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9fc009a20 con 0x7fda0414fd90 2023-10-10T19:22:30.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fd9f9ffb700 1 -- 172.21.15.175:0/2774382144 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc027990 con 0x7fda0414fd90 2023-10-10T19:22:30.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.828+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0414ede0 con 0x7fda0414fd90 2023-10-10T19:22:30.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.829+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda04067860 con 0x7fda0414fd90 2023-10-10T19:22:30.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.829+0000 7fd9f9ffb700 1 -- 172.21.15.175:0/2774382144 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd9fc024070 con 0x7fda0414fd90 2023-10-10T19:22:30.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.830+0000 7fd9f9ffb700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd9e8066f60 0x7fd9e8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:30.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.830+0000 7fd9f9ffb700 1 -- 172.21.15.175:0/2774382144 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fd9fc087340 con 0x7fda0414fd90 2023-10-10T19:22:30.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.830+0000 7fda09e9d700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd9e8066f60 0x7fd9e8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:30.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.830+0000 7fda09e9d700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd9e8066f60 0x7fd9e8069410 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fda00007160 tx=0x7fda00007f50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:30.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:30.834+0000 7fd9f9ffb700 1 -- 172.21.15.175:0/2774382144 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9fc053ee0 con 0x7fda0414fd90 2023-10-10T19:22:31.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.058+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda04071210 con 0x7fda0414fd90 2023-10-10T19:22:31.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.059+0000 7fd9f9ffb700 1 -- 172.21.15.175:0/2774382144 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+5422 (secure 0 0 0) 0x7fd9fc057500 con 0x7fda0414fd90 2023-10-10T19:22:31.060 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:31.061 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":278,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:23.833930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:31.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd9e8066f60 msgr2=0x7fd9e8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:31.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd9e8066f60 0x7fd9e8069410 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fda00007160 tx=0x7fda00007f50 comp rx=0 tx=0).stop 2023-10-10T19:22:31.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda0414fd90 msgr2=0x7fda0414cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:31.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda0414fd90 0x7fda0414cb80 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc012a50 tx=0x7fd9fc019000 comp rx=0 tx=0).stop 2023-10-10T19:22:31.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 shutdown_connections 2023-10-10T19:22:31.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd9e8066f60 0x7fd9e8069410 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:31.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda0414fd90 0x7fda0414cb80 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:31.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fda0408a100 0x7fda04168d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:31.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 --2- 172.21.15.175:0/2774382144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda040cd040 0x7fda0414c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:31.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 >> 172.21.15.175:0/2774382144 conn(0x7fda04063d00 msgr2=0x7fda040c2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:31.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 shutdown_connections 2023-10-10T19:22:31.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:31.061+0000 7fda0ae9f700 1 -- 172.21.15.175:0/2774382144 wait complete. 2023-10-10T19:22:31.065 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 278 2023-10-10T19:22:31.074 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:22:33.075 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:33.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.200+0000 7fcdb28b2700 1 Processor -- start 2023-10-10T19:22:33.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.200+0000 7fcdb28b2700 1 -- start start 2023-10-10T19:22:33.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdb28b2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c52a0 0x7fcdac06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:33.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdb28b2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac155ca0 con 0x7fcdac14cc10 2023-10-10T19:22:33.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdb28b2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac156b00 con 0x7fcdac14c840 2023-10-10T19:22:33.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdb28b2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac06d700 con 0x7fcdac0c52a0 2023-10-10T19:22:33.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcda3fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcdac14cc10 0x7fcdac152970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33200/0 (socket says 172.21.15.175:33200) 2023-10-10T19:22:33.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcda3fff700 1 -- 172.21.15.175:0/3875278222 learned_addr learned my addr 172.21.15.175:0/3875278222 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:33.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdb0e4f700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c52a0 0x7fcdac06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:33.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971467767 0 0) 0x7fcdac155ca0 con 0x7fcdac14cc10 2023-10-10T19:22:33.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd98004680 con 0x7fcdac14cc10 2023-10-10T19:22:33.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3860581030 0 0) 0x7fcd98004680 con 0x7fcdac14cc10 2023-10-10T19:22:33.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c52a0 msgr2=0x7fcdac06e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:33.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c52a0 0x7fcdac06e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 >> v1:172.21.15.191:6789/0 conn(0x7fcdac14c840 legacy=0x7fcdac0c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:33.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac155ef0 con 0x7fcdac14cc10 2023-10-10T19:22:33.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fcd94001470 con 0x7fcdac14cc10 2023-10-10T19:22:33.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 >> v1:172.21.15.175:6789/0 conn(0x7fcdac14cc10 legacy=0x7fcdac152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:33.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd98004ba0 0x7fcd98004f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:33.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c52a0 0x7fcd98005920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:33.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98005e60 0x7fcd980062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:33.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac06d700 con 0x7fcd98004ba0 2023-10-10T19:22:33.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9800cc20 con 0x7fcd98005e60 2023-10-10T19:22:33.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.201+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd980035f0 con 0x7fcdac0c52a0 2023-10-10T19:22:33.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdabfff700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd98004ba0 0x7fcd98004f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:33.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcda3fff700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c52a0 0x7fcd98005920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:33.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdb0e4f700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98005e60 0x7fcd980062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:33.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdabfff700 1 -- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c52a0 msgr2=0x7fcd98005920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:33.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdabfff700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c52a0 0x7fcd98005920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdabfff700 1 -- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98005e60 msgr2=0x7fcd980062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:33.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdabfff700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98005e60 0x7fcd980062d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdabfff700 1 -- 172.21.15.175:0/3875278222 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd98004680 con 0x7fcd98004ba0 2023-10-10T19:22:33.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdabfff700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd98004ba0 0x7fcd98004f70 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0019c0 tx=0x7fcd9c007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a541c9ae6f55b87 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:33.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c001330 con 0x7fcd98004ba0 2023-10-10T19:22:33.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c00c5d0 con 0x7fcd98004ba0 2023-10-10T19:22:33.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.202+0000 7fcdab7fe700 1 -- 172.21.15.175:0/3875278222 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00c970 con 0x7fcd98004ba0 2023-10-10T19:22:33.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.203+0000 7fcdb28b2700 1 -- 172.21.15.175:0/3875278222 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd98004ba0 msgr2=0x7fcd98004f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:33.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.203+0000 7fcdb28b2700 1 --2- 172.21.15.175:0/3875278222 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd98004ba0 0x7fcd98004f70 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0019c0 tx=0x7fcd9c007d60 comp rx=0 tx=0).stop 2023-10-10T19:22:33.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.203+0000 7fcdb0e4f700 1 -- 172.21.15.175:0/3875278222 reap_dead start 2023-10-10T19:22:33.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.203+0000 7fcdb28b2700 1 -- 172.21.15.175:0/3875278222 shutdown_connections 2023-10-10T19:22:33.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.203+0000 7fcdb28b2700 1 -- 172.21.15.175:0/3875278222 >> 172.21.15.175:0/3875278222 conn(0x7fcdac068280 msgr2=0x7fcdac06af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:33.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.203+0000 7fcdb28b2700 1 -- 172.21.15.175:0/3875278222 shutdown_connections 2023-10-10T19:22:33.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.203+0000 7fcdb28b2700 1 -- 172.21.15.175:0/3875278222 wait complete. 2023-10-10T19:22:33.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.204+0000 7fcdb28b2700 1 Processor -- start 2023-10-10T19:22:33.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.204+0000 7fcdb28b2700 1 -- start start 2023-10-10T19:22:33.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.204+0000 7fcdb28b2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98004ba0 0x7fcdac14b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:33.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.204+0000 7fcdb28b2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdac0c2bd0 0x7fcdac146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:33.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.204+0000 7fcdb28b2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c8960 0x7fcdac0c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:33.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.204+0000 7fcdb28b2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac151da0 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.204+0000 7fcdb28b2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac166c80 con 0x7fcd98004ba0 2023-10-10T19:22:33.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.204+0000 7fcdb28b2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac16a230 con 0x7fcdac0c8960 2023-10-10T19:22:33.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdac0c2bd0 0x7fcdac146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:33.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdac0c2bd0 0x7fcdac146520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:56376/0 (socket says 172.21.15.175:56376) 2023-10-10T19:22:33.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 -- 172.21.15.175:0/944936512 learned_addr learned my addr 172.21.15.175:0/944936512 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:33.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcdabfff700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98004ba0 0x7fcdac14b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:33.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcdb0e4f700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c8960 0x7fcdac0c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:33.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 -- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c8960 msgr2=0x7fcdac0c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:33.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c8960 0x7fcdac0c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 -- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98004ba0 msgr2=0x7fcdac14b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:33.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98004ba0 0x7fcdac14b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 -- 172.21.15.175:0/944936512 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac166000 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda3fff700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdac0c2bd0 0x7fcdac146520 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fcd94007340 tx=0x7fcd94001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:33.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda8ff9700 1 -- 172.21.15.175:0/944936512 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94018030 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda8ff9700 1 -- 172.21.15.175:0/944936512 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9400f620 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac1662e0 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcda8ff9700 1 -- 172.21.15.175:0/944936512 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9400f9e0 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.205+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac1668a0 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.206+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac06cf40 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.206+0000 7fcda8ff9700 1 -- 172.21.15.175:0/944936512 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcd9401b070 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.207+0000 7fcda8ff9700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd8c066f60 0x7fcd8c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:33.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.207+0000 7fcda8ff9700 1 -- 172.21.15.175:0/944936512 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fcd94019070 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.207+0000 7fcdabfff700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd8c066f60 0x7fcd8c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:33.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.207+0000 7fcdabfff700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd8c066f60 0x7fcd8c069410 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c001020 tx=0x7fcd9c015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:33.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.211+0000 7fcda8ff9700 1 -- 172.21.15.175:0/944936512 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd94054fa0 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.434+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdac067570 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.434+0000 7fcda8ff9700 1 -- 172.21.15.175:0/944936512 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 279 v279) v1 ==== 95+0+5424 (secure 0 0 0) 0x7fcd94054b30 con 0x7fcdac0c2bd0 2023-10-10T19:22:33.436 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:33.436 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":279,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":279,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:31.984969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:resolve","state_seq":17,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:33.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd8c066f60 msgr2=0x7fcd8c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:33.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd8c066f60 0x7fcd8c069410 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c001020 tx=0x7fcd9c015000 comp rx=0 tx=0).stop 2023-10-10T19:22:33.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdac0c2bd0 msgr2=0x7fcdac146520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:33.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdac0c2bd0 0x7fcdac146520 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fcd94007340 tx=0x7fcd94001e70 comp rx=0 tx=0).stop 2023-10-10T19:22:33.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 shutdown_connections 2023-10-10T19:22:33.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd8c066f60 0x7fcd8c069410 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdac0c8960 0x7fcdac0c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd98004ba0 0x7fcdac14b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 --2- 172.21.15.175:0/944936512 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdac0c2bd0 0x7fcdac146520 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:33.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 >> 172.21.15.175:0/944936512 conn(0x7fcdac068280 msgr2=0x7fcdac0c0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:33.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 shutdown_connections 2023-10-10T19:22:33.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:33.436+0000 7fcdb28b2700 1 -- 172.21.15.175:0/944936512 wait complete. 2023-10-10T19:22:33.441 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 279 2023-10-10T19:22:33.450 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:22:35.451 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:35.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.577+0000 7ff7a8e69700 1 Processor -- start 2023-10-10T19:22:35.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.577+0000 7ff7a8e69700 1 -- start start 2023-10-10T19:22:35.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.577+0000 7ff7a8e69700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff7a40b1fc0 0x7ff7a4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:35.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.577+0000 7ff7a8e69700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff7a40aff60 0x7ff7a40b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:35.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.577+0000 7ff7a8e69700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a41574b0 con 0x7ff7a40b2390 2023-10-10T19:22:35.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.577+0000 7ff7a8e69700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a4150770 con 0x7ff7a40b1fc0 2023-10-10T19:22:35.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.577+0000 7ff7a8e69700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a40afad0 con 0x7ff7a40aff60 2023-10-10T19:22:35.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff7a40b1fc0 0x7ff7a4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:35.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a1d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff7a40b2390 0x7ff7a4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33354/0 (socket says 172.21.15.175:33354) 2023-10-10T19:22:35.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a1d9b700 1 -- 172.21.15.175:0/2757124442 learned_addr learned my addr 172.21.15.175:0/2757124442 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:35.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a2d9d700 1 --2- 172.21.15.175:0/2757124442 >> v2:172.21.15.191:3300/0 conn(0x7ff7a40aff60 0x7ff7a40b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:35.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a259c700 1 -- 172.21.15.175:0/2757124442 >> v2:172.21.15.191:3300/0 conn(0x7ff7a40aff60 msgr2=0x7ff7a40b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:35.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a259c700 1 --2- 172.21.15.175:0/2757124442 >> v2:172.21.15.191:3300/0 conn(0x7ff7a40aff60 0x7ff7a40b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a259c700 1 -- 172.21.15.175:0/2757124442 >> v1:172.21.15.175:6789/0 conn(0x7ff7a40b2390 legacy=0x7ff7a4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:35.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a259c700 1 -- 172.21.15.175:0/2757124442 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40af4c0 con 0x7ff7a40b1fc0 2023-10-10T19:22:35.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a159a700 1 -- 172.21.15.175:0/2757124442 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291227392 0 0) 0x7ff7a41574b0 con 0x7ff7a40b2390 2023-10-10T19:22:35.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a259c700 1 --2- 172.21.15.175:0/2757124442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40b1fc0 0x7ff7a4155fb0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7ff7a40641f0 tx=0x7ff794009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eda719f934ace16 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:35.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a159a700 1 -- 172.21.15.175:0/2757124442 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79400e070 con 0x7ff7a40b1fc0 2023-10-10T19:22:35.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a159a700 1 -- 172.21.15.175:0/2757124442 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79400bcc0 con 0x7ff7a40b1fc0 2023-10-10T19:22:35.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.578+0000 7ff7a159a700 1 -- 172.21.15.175:0/2757124442 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794012680 con 0x7ff7a40b1fc0 2023-10-10T19:22:35.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.579+0000 7ff7a8e69700 1 -- 172.21.15.175:0/2757124442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40b1fc0 msgr2=0x7ff7a4155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:35.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.579+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/2757124442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40b1fc0 0x7ff7a4155fb0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7ff7a40641f0 tx=0x7ff794009780 comp rx=0 tx=0).stop 2023-10-10T19:22:35.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.579+0000 7ff7a8e69700 1 -- 172.21.15.175:0/2757124442 shutdown_connections 2023-10-10T19:22:35.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.579+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/2757124442 >> v2:172.21.15.191:3300/0 conn(0x7ff7a40aff60 0x7ff7a40b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.579+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/2757124442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40b1fc0 0x7ff7a4155fb0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.579+0000 7ff7a8e69700 1 -- 172.21.15.175:0/2757124442 >> 172.21.15.175:0/2757124442 conn(0x7ff7a4063230 msgr2=0x7ff7a4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:35.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.579+0000 7ff7a8e69700 1 -- 172.21.15.175:0/2757124442 shutdown_connections 2023-10-10T19:22:35.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.579+0000 7ff7a8e69700 1 -- 172.21.15.175:0/2757124442 wait complete. 2023-10-10T19:22:35.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.580+0000 7ff7a8e69700 1 Processor -- start 2023-10-10T19:22:35.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.580+0000 7ff7a8e69700 1 -- start start 2023-10-10T19:22:35.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a8e69700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40aff60 0x7ff7a4166a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:35.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a8e69700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7a40b1fc0 0x7ff7a40c5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:35.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a8e69700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7a40b2390 0x7ff7a40c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:35.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a8e69700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a415a340 con 0x7ff7a40aff60 2023-10-10T19:22:35.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a8e69700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a4166590 con 0x7ff7a40b1fc0 2023-10-10T19:22:35.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a8e69700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a415a080 con 0x7ff7a40b2390 2023-10-10T19:22:35.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40aff60 0x7ff7a4166a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:35.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40aff60 0x7ff7a4166a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54708/0 (socket says 172.21.15.175:54708) 2023-10-10T19:22:35.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 -- 172.21.15.175:0/374521211 learned_addr learned my addr 172.21.15.175:0/374521211 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:35.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a1d9b700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7a40b1fc0 0x7ff7a40c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:35.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 -- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7a40b2390 msgr2=0x7ff7a40c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:35.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a2d9d700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7a40b2390 0x7ff7a40c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:35.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7a40b2390 0x7ff7a40c3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 -- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7a40b1fc0 msgr2=0x7ff7a40c5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:35.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7a40b1fc0 0x7ff7a40c5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 -- 172.21.15.175:0/374521211 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff794009a20 con 0x7ff7a40aff60 2023-10-10T19:22:35.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a259c700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40aff60 0x7ff7a4166a20 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7ff7940097c0 tx=0x7ff794019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:35.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff79a7fc700 1 -- 172.21.15.175:0/374521211 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794027470 con 0x7ff7a40aff60 2023-10-10T19:22:35.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff79a7fc700 1 -- 172.21.15.175:0/374521211 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7940275d0 con 0x7ff7a40aff60 2023-10-10T19:22:35.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a4151610 con 0x7ff7a40aff60 2023-10-10T19:22:35.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff79a7fc700 1 -- 172.21.15.175:0/374521211 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794027990 con 0x7ff7a40aff60 2023-10-10T19:22:35.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.581+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a4166220 con 0x7ff7a40aff60 2023-10-10T19:22:35.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.582+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a4067570 con 0x7ff7a40aff60 2023-10-10T19:22:35.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.583+0000 7ff79a7fc700 1 -- 172.21.15.175:0/374521211 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff794024070 con 0x7ff7a40aff60 2023-10-10T19:22:35.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.583+0000 7ff79a7fc700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff784066f30 0x7ff7840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:35.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.583+0000 7ff79a7fc700 1 -- 172.21.15.175:0/374521211 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7ff7940875b0 con 0x7ff7a40aff60 2023-10-10T19:22:35.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.583+0000 7ff7a1d9b700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff784066f30 0x7ff7840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:35.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.584+0000 7ff7a1d9b700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff784066f30 0x7ff7840693e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff7a40cd800 tx=0x7ff78c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:35.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.587+0000 7ff79a7fc700 1 -- 172.21.15.175:0/374521211 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff794054150 con 0x7ff7a40aff60 2023-10-10T19:22:35.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.810+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7a4166870 con 0x7ff7a40aff60 2023-10-10T19:22:35.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.811+0000 7ff79a7fc700 1 -- 172.21.15.175:0/374521211 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 279 v279) v1 ==== 95+0+5424 (secure 0 0 0) 0x7ff794057770 con 0x7ff7a40aff60 2023-10-10T19:22:35.813 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:35.813 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":279,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":279,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:31.984969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:resolve","state_seq":17,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:35.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff784066f30 msgr2=0x7ff7840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:35.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff784066f30 0x7ff7840693e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff7a40cd800 tx=0x7ff78c009040 comp rx=0 tx=0).stop 2023-10-10T19:22:35.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40aff60 msgr2=0x7ff7a4166a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:35.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40aff60 0x7ff7a4166a20 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7ff7940097c0 tx=0x7ff794019000 comp rx=0 tx=0).stop 2023-10-10T19:22:35.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 shutdown_connections 2023-10-10T19:22:35.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff784066f30 0x7ff7840693e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7a40b2390 0x7ff7a40c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7a40aff60 0x7ff7a4166a20 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 --2- 172.21.15.175:0/374521211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7a40b1fc0 0x7ff7a40c5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:35.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 >> 172.21.15.175:0/374521211 conn(0x7ff7a4063230 msgr2=0x7ff7a41527d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:35.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 shutdown_connections 2023-10-10T19:22:35.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:35.813+0000 7ff7a8e69700 1 -- 172.21.15.175:0/374521211 wait complete. 2023-10-10T19:22:35.818 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 279 2023-10-10T19:22:35.826 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:22:37.827 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:37.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.954+0000 7fe358886700 1 Processor -- start 2023-10-10T19:22:37.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.954+0000 7fe358886700 1 -- start start 2023-10-10T19:22:37.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.954+0000 7fe358886700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3540b2390 0x7fe354155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:37.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe358886700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe3540aff60 0x7fe3540b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:37.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe358886700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3541574b0 con 0x7fe3540b1fc0 2023-10-10T19:22:37.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe358886700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe354150770 con 0x7fe3540b2390 2023-10-10T19:22:37.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe358886700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3540afad0 con 0x7fe3540aff60 2023-10-10T19:22:37.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe351d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3540b2390 0x7fe354155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:37.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe351d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3540b2390 0x7fe354155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54724/0 (socket says 172.21.15.175:54724) 2023-10-10T19:22:37.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe351d9b700 1 -- 172.21.15.175:0/1706461373 learned_addr learned my addr 172.21.15.175:0/1706461373 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:37.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe352d9d700 1 --2- 172.21.15.175:0/1706461373 >> v2:172.21.15.191:3300/0 conn(0x7fe3540aff60 0x7fe3540b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:37.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe351d9b700 1 -- 172.21.15.175:0/1706461373 >> v2:172.21.15.191:3300/0 conn(0x7fe3540aff60 msgr2=0x7fe3540b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:37.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe351d9b700 1 --2- 172.21.15.175:0/1706461373 >> v2:172.21.15.191:3300/0 conn(0x7fe3540aff60 0x7fe3540b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:37.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe351d9b700 1 -- 172.21.15.175:0/1706461373 >> v1:172.21.15.191:6789/0 conn(0x7fe3540b1fc0 legacy=0x7fe354068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:37.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe351d9b700 1 -- 172.21.15.175:0/1706461373 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540af4c0 con 0x7fe3540b2390 2023-10-10T19:22:37.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe351d9b700 1 --2- 172.21.15.175:0/1706461373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354155fb0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fe3540641f0 tx=0x7fe33c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a520aeb5f4dc592e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:37.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe35159a700 1 -- 172.21.15.175:0/1706461373 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c00b8f0 con 0x7fe3540b2390 2023-10-10T19:22:37.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.955+0000 7fe35159a700 1 -- 172.21.15.175:0/1706461373 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c00ba50 con 0x7fe3540b2390 2023-10-10T19:22:37.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe35159a700 1 -- 172.21.15.175:0/1706461373 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c00bdf0 con 0x7fe3540b2390 2023-10-10T19:22:37.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe358886700 1 -- 172.21.15.175:0/1706461373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 msgr2=0x7fe354155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:37.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe358886700 1 --2- 172.21.15.175:0/1706461373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354155fb0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fe3540641f0 tx=0x7fe33c009250 comp rx=0 tx=0).stop 2023-10-10T19:22:37.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe358886700 1 -- 172.21.15.175:0/1706461373 shutdown_connections 2023-10-10T19:22:37.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe358886700 1 --2- 172.21.15.175:0/1706461373 >> v2:172.21.15.191:3300/0 conn(0x7fe3540aff60 0x7fe3540b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:37.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe358886700 1 --2- 172.21.15.175:0/1706461373 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354155fb0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:37.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe358886700 1 -- 172.21.15.175:0/1706461373 >> 172.21.15.175:0/1706461373 conn(0x7fe354063230 msgr2=0x7fe354063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:37.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe358886700 1 -- 172.21.15.175:0/1706461373 shutdown_connections 2023-10-10T19:22:37.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.956+0000 7fe358886700 1 -- 172.21.15.175:0/1706461373 wait complete. 2023-10-10T19:22:37.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.957+0000 7fe358886700 1 Processor -- start 2023-10-10T19:22:37.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.957+0000 7fe358886700 1 -- start start 2023-10-10T19:22:37.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe358886700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3540aff60 0x7fe354169950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:37.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe358886700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3540b1fc0 0x7fe354154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:37.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe358886700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:37.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe358886700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354156420 con 0x7fe3540b2390 2023-10-10T19:22:37.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe358886700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354154090 con 0x7fe3540b1fc0 2023-10-10T19:22:37.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe358886700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35416a340 con 0x7fe3540aff60 2023-10-10T19:22:37.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:37.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54740/0 (socket says 172.21.15.175:54740) 2023-10-10T19:22:37.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 -- 172.21.15.175:0/2775038101 learned_addr learned my addr 172.21.15.175:0/2775038101 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:37.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe351d9b700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3540b1fc0 0x7fe354154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:37.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe35259c700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3540aff60 0x7fe354169950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:37.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 -- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3540aff60 msgr2=0x7fe354169950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:37.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3540aff60 0x7fe354169950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:37.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 -- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3540b1fc0 msgr2=0x7fe354154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:37.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3540b1fc0 0x7fe354154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:37.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 -- 172.21.15.175:0/2775038101 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c0094f0 con 0x7fe3540b2390 2023-10-10T19:22:37.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe35259c700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3540aff60 0x7fe354169950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:22:37.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe352d9d700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354154a60 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00d8c0 tx=0x7fe34c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:37.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.958+0000 7fe34a7fc700 1 -- 172.21.15.175:0/2775038101 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0125f0 con 0x7fe3540b2390 2023-10-10T19:22:37.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.959+0000 7fe34a7fc700 1 -- 172.21.15.175:0/2775038101 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c012770 con 0x7fe3540b2390 2023-10-10T19:22:37.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.959+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe354156670 con 0x7fe3540b2390 2023-10-10T19:22:37.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.959+0000 7fe34a7fc700 1 -- 172.21.15.175:0/2775038101 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c012b30 con 0x7fe3540b2390 2023-10-10T19:22:37.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.959+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3540c41a0 con 0x7fe3540b2390 2023-10-10T19:22:37.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.959+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe354067570 con 0x7fe3540b2390 2023-10-10T19:22:37.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.960+0000 7fe34a7fc700 1 -- 172.21.15.175:0/2775038101 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe34c01e460 con 0x7fe3540b2390 2023-10-10T19:22:37.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.960+0000 7fe34a7fc700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe334066f30 0x7fe3340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:37.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.960+0000 7fe34a7fc700 1 -- 172.21.15.175:0/2775038101 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fe34c0851d0 con 0x7fe3540b2390 2023-10-10T19:22:37.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.960+0000 7fe35259c700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe334066f30 0x7fe3340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:37.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.961+0000 7fe35259c700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe334066f30 0x7fe3340693e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe344007fd0 tx=0x7fe344009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:37.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:37.964+0000 7fe34a7fc700 1 -- 172.21.15.175:0/2775038101 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe34c051d70 con 0x7fe3540b2390 2023-10-10T19:22:38.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.193+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3541541d0 con 0x7fe3540b2390 2023-10-10T19:22:38.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.194+0000 7fe34a7fc700 1 -- 172.21.15.175:0/2775038101 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 280 v280) v1 ==== 95+0+5426 (secure 0 0 0) 0x7fe34c055390 con 0x7fe3540b2390 2023-10-10T19:22:38.194 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:38.195 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":280,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":280,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:38.011399+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:38.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe334066f30 msgr2=0x7fe3340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:38.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe334066f30 0x7fe3340693e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe344007fd0 tx=0x7fe344009040 comp rx=0 tx=0).stop 2023-10-10T19:22:38.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 msgr2=0x7fe354154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:38.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354154a60 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00d8c0 tx=0x7fe34c008040 comp rx=0 tx=0).stop 2023-10-10T19:22:38.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 shutdown_connections 2023-10-10T19:22:38.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe334066f30 0x7fe3340693e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:38.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3540b2390 0x7fe354154a60 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:38.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3540aff60 0x7fe354169950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:38.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 --2- 172.21.15.175:0/2775038101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3540b1fc0 0x7fe354154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:38.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 >> 172.21.15.175:0/2775038101 conn(0x7fe354063230 msgr2=0x7fe35414fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:38.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 shutdown_connections 2023-10-10T19:22:38.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:38.196+0000 7fe358886700 1 -- 172.21.15.175:0/2775038101 wait complete. 2023-10-10T19:22:38.199 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 280 2023-10-10T19:22:38.209 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 280, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 280, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_7104": { "addr": "172.21.15.175:6837/3849771254", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 3849771254, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 3849771254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7104, "incarnation": 221, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 43 }, "gid_7252": { "addr": "172.21.15.191:6839/2393527600", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 2393527600, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 2393527600, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7252, "incarnation": 271, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 20 }, "gid_7384": { "addr": "172.21.15.191:6837/2914004704", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 2914004704, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2914004704, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7384, "incarnation": 278, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:reconnect", "state_seq": 19 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 90 } }, "last_failure": 0, "last_failure_osd_epoch": 127, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:22:38.011399+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 7252, "mds_1": 7401, "mds_2": 7104, "mds_3": 7384 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:22:40.210 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:40.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.339+0000 7f7c4f345700 1 Processor -- start 2023-10-10T19:22:40.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.339+0000 7f7c4f345700 1 -- start start 2023-10-10T19:22:40.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4f345700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7c481595a0 0x7f7c4806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:40.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4f345700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c4806bda0 0x7f7c480c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:40.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4f345700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4806a8c0 con 0x7f7c480c5e00 2023-10-10T19:22:40.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4f345700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c48159fb0 con 0x7f7c4806bda0 2023-10-10T19:22:40.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4f345700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c480c5ab0 con 0x7f7c481595a0 2023-10-10T19:22:40.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4db42700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c4806bda0 0x7f7c480c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:40.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4db42700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c4806bda0 0x7f7c480c0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54744/0 (socket says 172.21.15.175:54744) 2023-10-10T19:22:40.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4db42700 1 -- 172.21.15.175:0/3728699759 learned_addr learned my addr 172.21.15.175:0/3728699759 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:40.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.340+0000 7f7c4e343700 1 --2- 172.21.15.175:0/3728699759 >> v2:172.21.15.191:3300/0 conn(0x7f7c481595a0 0x7f7c4806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:40.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4db42700 1 -- 172.21.15.175:0/3728699759 >> v2:172.21.15.191:3300/0 conn(0x7f7c481595a0 msgr2=0x7f7c4806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:40.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4db42700 1 --2- 172.21.15.175:0/3728699759 >> v2:172.21.15.191:3300/0 conn(0x7f7c481595a0 0x7f7c4806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4db42700 1 -- 172.21.15.175:0/3728699759 >> v1:172.21.15.191:6789/0 conn(0x7f7c480c5e00 legacy=0x7f7c480afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:40.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4db42700 1 -- 172.21.15.175:0/3728699759 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c480c57d0 con 0x7f7c4806bda0 2023-10-10T19:22:40.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4db42700 1 --2- 172.21.15.175:0/3728699759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c4806bda0 0x7f7c480c0ff0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f7c48067120 tx=0x7f7c38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1410a270501a748 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:40.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4d341700 1 -- 172.21.15.175:0/3728699759 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38009e70 con 0x7f7c4806bda0 2023-10-10T19:22:40.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4d341700 1 -- 172.21.15.175:0/3728699759 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3800b8f0 con 0x7f7c4806bda0 2023-10-10T19:22:40.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4d341700 1 -- 172.21.15.175:0/3728699759 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3800bc90 con 0x7f7c4806bda0 2023-10-10T19:22:40.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4f345700 1 -- 172.21.15.175:0/3728699759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c4806bda0 msgr2=0x7f7c480c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:40.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4f345700 1 --2- 172.21.15.175:0/3728699759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c4806bda0 0x7f7c480c0ff0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f7c48067120 tx=0x7f7c38009250 comp rx=0 tx=0).stop 2023-10-10T19:22:40.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.341+0000 7f7c4f345700 1 -- 172.21.15.175:0/3728699759 shutdown_connections 2023-10-10T19:22:40.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.342+0000 7f7c4f345700 1 --2- 172.21.15.175:0/3728699759 >> v2:172.21.15.191:3300/0 conn(0x7f7c481595a0 0x7f7c4806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.342+0000 7f7c4f345700 1 --2- 172.21.15.175:0/3728699759 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c4806bda0 0x7f7c480c0ff0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.342+0000 7f7c4f345700 1 -- 172.21.15.175:0/3728699759 >> 172.21.15.175:0/3728699759 conn(0x7f7c48063cd0 msgr2=0x7f7c4806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:40.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.342+0000 7f7c4f345700 1 -- 172.21.15.175:0/3728699759 shutdown_connections 2023-10-10T19:22:40.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.342+0000 7f7c4f345700 1 -- 172.21.15.175:0/3728699759 wait complete. 2023-10-10T19:22:40.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4f345700 1 Processor -- start 2023-10-10T19:22:40.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4f345700 1 -- start start 2023-10-10T19:22:40.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4f345700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c4806bda0 0x7f7c480c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:40.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4f345700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c480c5e00 0x7f7c48152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:40.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4f345700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c481595a0 0x7f7c48165f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:40.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4f345700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480c27c0 con 0x7f7c481595a0 2023-10-10T19:22:40.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4f345700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c481657b0 con 0x7f7c480c5e00 2023-10-10T19:22:40.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4f345700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48166c30 con 0x7f7c4806bda0 2023-10-10T19:22:40.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4eb44700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c481595a0 0x7f7c48165f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:40.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4eb44700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c481595a0 0x7f7c48165f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54752/0 (socket says 172.21.15.175:54752) 2023-10-10T19:22:40.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4eb44700 1 -- 172.21.15.175:0/1600554908 learned_addr learned my addr 172.21.15.175:0/1600554908 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:40.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4db42700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c480c5e00 0x7f7c48152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:40.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.343+0000 7f7c4e343700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c4806bda0 0x7f7c480c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:40.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c4eb44700 1 -- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c4806bda0 msgr2=0x7f7c480c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:40.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c4eb44700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c4806bda0 0x7f7c480c2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c4eb44700 1 -- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c480c5e00 msgr2=0x7f7c48152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:40.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c4eb44700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c480c5e00 0x7f7c48152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c4eb44700 1 -- 172.21.15.175:0/1600554908 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380094f0 con 0x7f7c481595a0 2023-10-10T19:22:40.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c4eb44700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c481595a0 0x7f7c48165f50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c590 tx=0x7f7c40007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:40.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c3e7fc700 1 -- 172.21.15.175:0/1600554908 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40011590 con 0x7f7c481595a0 2023-10-10T19:22:40.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c3e7fc700 1 -- 172.21.15.175:0/1600554908 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c400116f0 con 0x7f7c481595a0 2023-10-10T19:22:40.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4815b5f0 con 0x7f7c481595a0 2023-10-10T19:22:40.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c3e7fc700 1 -- 172.21.15.175:0/1600554908 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40011ab0 con 0x7f7c481595a0 2023-10-10T19:22:40.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.344+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4815bb90 con 0x7f7c481595a0 2023-10-10T19:22:40.355 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.345+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c48067570 con 0x7f7c481595a0 2023-10-10T19:22:40.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.345+0000 7f7c3e7fc700 1 -- 172.21.15.175:0/1600554908 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7c4001b3b0 con 0x7f7c481595a0 2023-10-10T19:22:40.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.346+0000 7f7c3e7fc700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c2c066f30 0x7f7c2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:40.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.346+0000 7f7c3e7fc700 1 -- 172.21.15.175:0/1600554908 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f7c40083fb0 con 0x7f7c481595a0 2023-10-10T19:22:40.356 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.346+0000 7f7c4e343700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c2c066f30 0x7f7c2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:40.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.346+0000 7f7c4e343700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c2c066f30 0x7f7c2c0693e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009fd0 tx=0x7f7c4400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:40.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.350+0000 7f7c3e7fc700 1 -- 172.21.15.175:0/1600554908 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c40050b50 con 0x7f7c481595a0 2023-10-10T19:22:40.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.577+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c480bfa50 con 0x7f7c481595a0 2023-10-10T19:22:40.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.577+0000 7f7c3e7fc700 1 -- 172.21.15.175:0/1600554908 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 280 v280) v1 ==== 95+0+5426 (secure 0 0 0) 0x7f7c40017070 con 0x7f7c481595a0 2023-10-10T19:22:40.578 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:40.578 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":280,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":280,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:38.011399+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:40.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c2c066f30 msgr2=0x7f7c2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:40.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c2c066f30 0x7f7c2c0693e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009fd0 tx=0x7f7c4400b040 comp rx=0 tx=0).stop 2023-10-10T19:22:40.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c481595a0 msgr2=0x7f7c48165f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:40.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c481595a0 0x7f7c48165f50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c590 tx=0x7f7c40007ec0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 shutdown_connections 2023-10-10T19:22:40.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7c2c066f30 0x7f7c2c0693e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c481595a0 0x7f7c48165f50 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c4806bda0 0x7f7c480c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 --2- 172.21.15.175:0/1600554908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c480c5e00 0x7f7c48152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:40.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.579+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 >> 172.21.15.175:0/1600554908 conn(0x7f7c48063cd0 msgr2=0x7f7c480c0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:40.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.580+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 shutdown_connections 2023-10-10T19:22:40.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:40.580+0000 7f7c4f345700 1 -- 172.21.15.175:0/1600554908 wait complete. 2023-10-10T19:22:40.583 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 280 2023-10-10T19:22:40.592 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:22:42.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:42.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.720+0000 7fd6514ef700 1 Processor -- start 2023-10-10T19:22:42.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.720+0000 7fd6514ef700 1 -- start start 2023-10-10T19:22:42.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd6514ef700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd64c14fbf0 0x7fd64c159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:42.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd6514ef700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14ffc0 0x7fd64c063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:42.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd6514ef700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd64c0681e0 0x7fd64c0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:42.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd6514ef700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c0c59e0 con 0x7fd64c0681e0 2023-10-10T19:22:42.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd6514ef700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c15a2b0 con 0x7fd64c14fbf0 2023-10-10T19:22:42.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd6514ef700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c156800 con 0x7fd64c14ffc0 2023-10-10T19:22:42.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd64b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd64c0681e0 0x7fd64c0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:42.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd64b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd64c0681e0 0x7fd64c0c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54772/0 (socket says 172.21.15.175:54772) 2023-10-10T19:22:42.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd64b7fe700 1 -- 172.21.15.175:0/2675117780 learned_addr learned my addr 172.21.15.175:0/2675117780 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:42.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd64affd700 1 --2- 172.21.15.175:0/2675117780 >> v2:172.21.15.191:3300/0 conn(0x7fd64c14fbf0 0x7fd64c159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:42.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.721+0000 7fd64a7fc700 1 --2- 172.21.15.175:0/2675117780 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14ffc0 0x7fd64c063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:42.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd64b7fe700 1 -- 172.21.15.175:0/2675117780 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14ffc0 msgr2=0x7fd64c063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:42.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd64b7fe700 1 --2- 172.21.15.175:0/2675117780 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14ffc0 0x7fd64c063230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd64b7fe700 1 -- 172.21.15.175:0/2675117780 >> v2:172.21.15.191:3300/0 conn(0x7fd64c14fbf0 msgr2=0x7fd64c159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:42.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd64b7fe700 1 --2- 172.21.15.175:0/2675117780 >> v2:172.21.15.191:3300/0 conn(0x7fd64c14fbf0 0x7fd64c159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd64b7fe700 1 -- 172.21.15.175:0/2675117780 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c063e30 con 0x7fd64c0681e0 2023-10-10T19:22:42.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd64b7fe700 1 --2- 172.21.15.175:0/2675117780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c0681e0 0x7fd64c0c5fb0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd6440099b0 tx=0x7fd644009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8bb6d568f3705b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:42.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd649ffb700 1 -- 172.21.15.175:0/2675117780 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64400b820 con 0x7fd64c0681e0 2023-10-10T19:22:42.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd649ffb700 1 -- 172.21.15.175:0/2675117780 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64400b980 con 0x7fd64c0681e0 2023-10-10T19:22:42.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd649ffb700 1 -- 172.21.15.175:0/2675117780 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64400bd20 con 0x7fd64c0681e0 2023-10-10T19:22:42.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 -- 172.21.15.175:0/2675117780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c0681e0 msgr2=0x7fd64c0c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:42.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 --2- 172.21.15.175:0/2675117780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c0681e0 0x7fd64c0c5fb0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd6440099b0 tx=0x7fd644009250 comp rx=0 tx=0).stop 2023-10-10T19:22:42.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 -- 172.21.15.175:0/2675117780 shutdown_connections 2023-10-10T19:22:42.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 --2- 172.21.15.175:0/2675117780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c0681e0 0x7fd64c0c5fb0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 --2- 172.21.15.175:0/2675117780 >> v2:172.21.15.191:3300/0 conn(0x7fd64c14fbf0 0x7fd64c159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 --2- 172.21.15.175:0/2675117780 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14ffc0 0x7fd64c063230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 -- 172.21.15.175:0/2675117780 >> 172.21.15.175:0/2675117780 conn(0x7fd64c0cc760 msgr2=0x7fd64c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:42.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 -- 172.21.15.175:0/2675117780 shutdown_connections 2023-10-10T19:22:42.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.722+0000 7fd6514ef700 1 -- 172.21.15.175:0/2675117780 wait complete. 2023-10-10T19:22:42.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.724+0000 7fd6514ef700 1 Processor -- start 2023-10-10T19:22:42.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.724+0000 7fd6514ef700 1 -- start start 2023-10-10T19:22:42.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.724+0000 7fd6514ef700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd64c0681e0 0x7fd64c15ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:42.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.724+0000 7fd6514ef700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14fbf0 0x7fd64c165ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:42.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.724+0000 7fd6514ef700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c14ffc0 0x7fd64c166530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:42.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.724+0000 7fd6514ef700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c1516f0 con 0x7fd64c14ffc0 2023-10-10T19:22:42.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.724+0000 7fd6514ef700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c14b320 con 0x7fd64c0681e0 2023-10-10T19:22:42.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.724+0000 7fd6514ef700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c152970 con 0x7fd64c14fbf0 2023-10-10T19:22:42.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c14ffc0 0x7fd64c166530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:42.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c14ffc0 0x7fd64c166530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54776/0 (socket says 172.21.15.175:54776) 2023-10-10T19:22:42.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 -- 172.21.15.175:0/316784515 learned_addr learned my addr 172.21.15.175:0/316784515 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:42.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64affd700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd64c0681e0 0x7fd64c15ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:42.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64a7fc700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14fbf0 0x7fd64c165ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:42.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 -- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14fbf0 msgr2=0x7fd64c165ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:42.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14fbf0 0x7fd64c165ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 -- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd64c0681e0 msgr2=0x7fd64c15ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:42.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd64c0681e0 0x7fd64c15ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 -- 172.21.15.175:0/316784515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6440094f0 con 0x7fd64c14ffc0 2023-10-10T19:22:42.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd64b7fe700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c14ffc0 0x7fd64c166530 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd644009290 tx=0x7fd644017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:42.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd632ffd700 1 -- 172.21.15.175:0/316784515 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64400b820 con 0x7fd64c14ffc0 2023-10-10T19:22:42.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd632ffd700 1 -- 172.21.15.175:0/316784515 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64401a630 con 0x7fd64c14ffc0 2023-10-10T19:22:42.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c152c50 con 0x7fd64c14ffc0 2023-10-10T19:22:42.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd632ffd700 1 -- 172.21.15.175:0/316784515 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644011ea0 con 0x7fd64c14ffc0 2023-10-10T19:22:42.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.725+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c153210 con 0x7fd64c14ffc0 2023-10-10T19:22:42.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.726+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c067570 con 0x7fd64c14ffc0 2023-10-10T19:22:42.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.727+0000 7fd632ffd700 1 -- 172.21.15.175:0/316784515 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd64401ace0 con 0x7fd64c14ffc0 2023-10-10T19:22:42.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.727+0000 7fd632ffd700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd638066f30 0x7fd6380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:42.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.727+0000 7fd632ffd700 1 -- 172.21.15.175:0/316784515 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fd644086790 con 0x7fd64c14ffc0 2023-10-10T19:22:42.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.727+0000 7fd64affd700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd638066f30 0x7fd6380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:42.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.727+0000 7fd64affd700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd638066f30 0x7fd6380693e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd63c007680 tx=0x7fd63c0098d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:42.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.731+0000 7fd632ffd700 1 -- 172.21.15.175:0/316784515 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd644053330 con 0x7fd64c14ffc0 2023-10-10T19:22:42.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.956+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd64c165ca0 con 0x7fd64c14ffc0 2023-10-10T19:22:42.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.956+0000 7fd632ffd700 1 -- 172.21.15.175:0/316784515 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 280 v280) v1 ==== 95+0+5426 (secure 0 0 0) 0x7fd644056950 con 0x7fd64c14ffc0 2023-10-10T19:22:42.957 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:42.957 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":280,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":280,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:38.011399+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:42.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd638066f30 msgr2=0x7fd6380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:42.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd638066f30 0x7fd6380693e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd63c007680 tx=0x7fd63c0098d0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c14ffc0 msgr2=0x7fd64c166530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:42.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c14ffc0 0x7fd64c166530 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd644009290 tx=0x7fd644017000 comp rx=0 tx=0).stop 2023-10-10T19:22:42.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 shutdown_connections 2023-10-10T19:22:42.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd638066f30 0x7fd6380693e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd64c14ffc0 0x7fd64c166530 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd64c0681e0 0x7fd64c15ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 --2- 172.21.15.175:0/316784515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd64c14fbf0 0x7fd64c165ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:42.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.958+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 >> 172.21.15.175:0/316784515 conn(0x7fd64c0cc760 msgr2=0x7fd64c0c5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:42.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.959+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 shutdown_connections 2023-10-10T19:22:42.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:42.959+0000 7fd6514ef700 1 -- 172.21.15.175:0/316784515 wait complete. 2023-10-10T19:22:42.962 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 280 2023-10-10T19:22:42.971 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:22:44.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:45.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.100+0000 7f65c3ef4700 1 Processor -- start 2023-10-10T19:22:45.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.101+0000 7f65c3ef4700 1 -- start start 2023-10-10T19:22:45.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.101+0000 7f65c3ef4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc159a70 0x7f65bc159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.101+0000 7f65c3ef4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f65bc15a380 0x7f65bc0c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.101+0000 7f65c3ef4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc068c90 con 0x7f65bc06a650 2023-10-10T19:22:45.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.101+0000 7f65c3ef4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc069120 con 0x7f65bc15a380 2023-10-10T19:22:45.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.101+0000 7f65c3ef4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0c5b90 con 0x7f65bc159a70 2023-10-10T19:22:45.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.101+0000 7f65c2491700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f65bc06a650 0x7f65bc06b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49414/0 (socket says 172.21.15.175:49414) 2023-10-10T19:22:45.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.101+0000 7f65c2491700 1 -- 172.21.15.175:0/2945397215 learned_addr learned my addr 172.21.15.175:0/2945397215 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:45.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c1c90700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc159a70 0x7f65bc159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c148f700 1 --2- 172.21.15.175:0/2945397215 >> v2:172.21.15.191:3300/0 conn(0x7f65bc15a380 0x7f65bc0c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701572421 0 0) 0x7f65bc068c90 con 0x7f65bc06a650 2023-10-10T19:22:45.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a8003570 con 0x7f65bc06a650 2023-10-10T19:22:45.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 685591001 0 0) 0x7f65a8003570 con 0x7f65bc06a650 2023-10-10T19:22:45.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc159a70 msgr2=0x7f65bc159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:45.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc159a70 0x7f65bc159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 >> v2:172.21.15.191:3300/0 conn(0x7f65bc15a380 msgr2=0x7f65bc0c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:45.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 --2- 172.21.15.175:0/2945397215 >> v2:172.21.15.191:3300/0 conn(0x7f65bc15a380 0x7f65bc0c7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc0639e0 con 0x7f65bc06a650 2023-10-10T19:22:45.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f65b40017f0 con 0x7f65bc06a650 2023-10-10T19:22:45.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c148f700 1 --2- 172.21.15.175:0/2945397215 >> v2:172.21.15.191:3300/0 conn(0x7f65bc15a380 0x7f65bc0c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:22:45.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 >> v1:172.21.15.175:6789/0 conn(0x7f65bc06a650 legacy=0x7f65bc06b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:45.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a8003a60 0x7f65a8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a8004810 0x7f65a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65a80051c0 0x7f65a800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc069120 con 0x7f65a80051c0 2023-10-10T19:22:45.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0c5b90 con 0x7f65a8003a60 2023-10-10T19:22:45.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800bfa0 con 0x7f65a8004810 2023-10-10T19:22:45.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c1c90700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc159a70 0x7f65bc159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:22:45.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c2491700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65a80051c0 0x7f65a800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c148f700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a8004810 0x7f65a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c1c90700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a8003a60 0x7f65a8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c2491700 1 -- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a8004810 msgr2=0x7f65a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:45.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c2491700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65a8004810 0x7f65a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c2491700 1 -- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a8003a60 msgr2=0x7f65a8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:45.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c2491700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a8003a60 0x7f65a8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c2491700 1 -- 172.21.15.175:0/2945397215 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a8003570 con 0x7f65a80051c0 2023-10-10T19:22:45.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.102+0000 7f65c1c90700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65a8003a60 0x7f65a8003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:22:45.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c2491700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65a80051c0 0x7f65a800ba60 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f65b400a940 tx=0x7f65b4002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea5cc4013752ef5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:45.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4012440 con 0x7f65a80051c0 2023-10-10T19:22:45.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b40125c0 con 0x7f65a80051c0 2023-10-10T19:22:45.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c0c8e700 1 -- 172.21.15.175:0/2945397215 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4012980 con 0x7f65a80051c0 2023-10-10T19:22:45.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c3ef4700 1 -- 172.21.15.175:0/2945397215 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65a80051c0 msgr2=0x7f65a800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:45.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c3ef4700 1 --2- 172.21.15.175:0/2945397215 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65a80051c0 0x7f65a800ba60 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f65b400a940 tx=0x7f65b4002210 comp rx=0 tx=0).stop 2023-10-10T19:22:45.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c2491700 1 -- 172.21.15.175:0/2945397215 reap_dead start 2023-10-10T19:22:45.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c3ef4700 1 -- 172.21.15.175:0/2945397215 shutdown_connections 2023-10-10T19:22:45.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c3ef4700 1 -- 172.21.15.175:0/2945397215 >> 172.21.15.175:0/2945397215 conn(0x7f65bc06dc80 msgr2=0x7f65bc063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:45.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c3ef4700 1 -- 172.21.15.175:0/2945397215 shutdown_connections 2023-10-10T19:22:45.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.103+0000 7f65c3ef4700 1 -- 172.21.15.175:0/2945397215 wait complete. 2023-10-10T19:22:45.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.104+0000 7f65c3ef4700 1 Processor -- start 2023-10-10T19:22:45.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.104+0000 7f65c3ef4700 1 -- start start 2023-10-10T19:22:45.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c3ef4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc153db0 0x7f65bc153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c3ef4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65bc154180 0x7f65bc0c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c3ef4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65bc0c8900 0x7f65bc150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c3ef4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0c34f0 con 0x7f65bc0c8900 2023-10-10T19:22:45.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c3ef4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0c4070 con 0x7f65bc154180 2023-10-10T19:22:45.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c3ef4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0c4500 con 0x7f65bc153db0 2023-10-10T19:22:45.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c2491700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65bc0c8900 0x7f65bc150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c2491700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65bc0c8900 0x7f65bc150aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42034/0 (socket says 172.21.15.175:42034) 2023-10-10T19:22:45.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c2491700 1 -- 172.21.15.175:0/164014782 learned_addr learned my addr 172.21.15.175:0/164014782 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:45.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c1c90700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc153db0 0x7f65bc153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c148f700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65bc154180 0x7f65bc0c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c2491700 1 -- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc153db0 msgr2=0x7f65bc153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:45.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c2491700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc153db0 0x7f65bc153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c2491700 1 -- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65bc154180 msgr2=0x7f65bc0c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:45.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c2491700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65bc154180 0x7f65bc0c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.105+0000 7f65c2491700 1 -- 172.21.15.175:0/164014782 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc153800 con 0x7f65bc0c8900 2023-10-10T19:22:45.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.106+0000 7f65c2491700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65bc0c8900 0x7f65bc150aa0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f65b400a000 tx=0x7f65b4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:45.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.106+0000 7f65b1ffb700 1 -- 172.21.15.175:0/164014782 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4001360 con 0x7f65bc0c8900 2023-10-10T19:22:45.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.106+0000 7f65b1ffb700 1 -- 172.21.15.175:0/164014782 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4012440 con 0x7f65bc0c8900 2023-10-10T19:22:45.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.106+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc0c3740 con 0x7f65bc0c8900 2023-10-10T19:22:45.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.106+0000 7f65b1ffb700 1 -- 172.21.15.175:0/164014782 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40017b0 con 0x7f65bc0c8900 2023-10-10T19:22:45.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.106+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc0c3d00 con 0x7f65bc0c8900 2023-10-10T19:22:45.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.107+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc0c41e0 con 0x7f65bc0c8900 2023-10-10T19:22:45.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.107+0000 7f65b1ffb700 1 -- 172.21.15.175:0/164014782 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f65b4019030 con 0x7f65bc0c8900 2023-10-10T19:22:45.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.108+0000 7f65b1ffb700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f65a0066f30 0x7f65a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:45.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.108+0000 7f65b1ffb700 1 -- 172.21.15.175:0/164014782 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f65b4086970 con 0x7f65bc0c8900 2023-10-10T19:22:45.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.108+0000 7f65c1c90700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f65a0066f30 0x7f65a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:45.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.108+0000 7f65c1c90700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f65a0066f30 0x7f65a00693e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f65bc064c60 tx=0x7f65b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:45.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.112+0000 7f65b1ffb700 1 -- 172.21.15.175:0/164014782 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f65b40535d0 con 0x7f65bc0c8900 2023-10-10T19:22:45.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.334+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65bc0c41e0 con 0x7f65bc0c8900 2023-10-10T19:22:45.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.335+0000 7f65b1ffb700 1 -- 172.21.15.175:0/164014782 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 281 v281) v1 ==== 95+0+5423 (secure 0 0 0) 0x7f65b4001f30 con 0x7f65bc0c8900 2023-10-10T19:22:45.336 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:45.336 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":281,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":281,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:43.025395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:45.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f65a0066f30 msgr2=0x7f65a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:45.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f65a0066f30 0x7f65a00693e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f65bc064c60 tx=0x7f65b8008040 comp rx=0 tx=0).stop 2023-10-10T19:22:45.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65bc0c8900 msgr2=0x7f65bc150aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:45.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65bc0c8900 0x7f65bc150aa0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f65b400a000 tx=0x7f65b4015000 comp rx=0 tx=0).stop 2023-10-10T19:22:45.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 shutdown_connections 2023-10-10T19:22:45.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f65a0066f30 0x7f65a00693e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f65bc0c8900 0x7f65bc150aa0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65bc153db0 0x7f65bc153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 --2- 172.21.15.175:0/164014782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65bc154180 0x7f65bc0c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:45.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 >> 172.21.15.175:0/164014782 conn(0x7f65bc06dc80 msgr2=0x7f65bc0c6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:45.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 shutdown_connections 2023-10-10T19:22:45.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:45.337+0000 7f65c3ef4700 1 -- 172.21.15.175:0/164014782 wait complete. 2023-10-10T19:22:45.340 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 281 2023-10-10T19:22:45.349 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:22:47.350 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:47.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.478+0000 7f8595e08700 1 Processor -- start 2023-10-10T19:22:47.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.478+0000 7f8595e08700 1 -- start start 2023-10-10T19:22:47.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8595e08700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f859014fd90 0x7f8590159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:47.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8595e08700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 0x7f85900c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:47.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8595e08700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8590067fa0 con 0x7f85900cd040 2023-10-10T19:22:47.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8595e08700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85900c5600 con 0x7f859014fd90 2023-10-10T19:22:47.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8595e08700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8590156b70 con 0x7f859008a100 2023-10-10T19:22:47.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8594e06700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f859014fd90 0x7f8590159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:47.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8594e06700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f859014fd90 0x7f8590159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42048/0 (socket says 172.21.15.175:42048) 2023-10-10T19:22:47.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8594e06700 1 -- 172.21.15.175:0/3879864630 learned_addr learned my addr 172.21.15.175:0/3879864630 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:47.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f858ffff700 1 --2- 172.21.15.175:0/3879864630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 0x7f85900c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:47.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8594e06700 1 -- 172.21.15.175:0/3879864630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 msgr2=0x7f85900c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:47.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8594e06700 1 --2- 172.21.15.175:0/3879864630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 0x7f85900c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8594e06700 1 -- 172.21.15.175:0/3879864630 >> v1:172.21.15.191:6789/0 conn(0x7f85900cd040 legacy=0x7f85900684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:47.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8594e06700 1 -- 172.21.15.175:0/3879864630 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8590150160 con 0x7f859014fd90 2023-10-10T19:22:47.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.479+0000 7f8594e06700 1 --2- 172.21.15.175:0/3879864630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f859014fd90 0x7f8590159c30 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f85880099b0 tx=0x7f8588009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85886cf369642d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:47.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f858f7fe700 1 -- 172.21.15.175:0/3879864630 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858800b860 con 0x7f859014fd90 2023-10-10T19:22:47.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f858f7fe700 1 -- 172.21.15.175:0/3879864630 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858800b9c0 con 0x7f859014fd90 2023-10-10T19:22:47.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f858f7fe700 1 -- 172.21.15.175:0/3879864630 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858800bd60 con 0x7f859014fd90 2023-10-10T19:22:47.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f8595e08700 1 -- 172.21.15.175:0/3879864630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f859014fd90 msgr2=0x7f8590159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:47.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f8595e08700 1 --2- 172.21.15.175:0/3879864630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f859014fd90 0x7f8590159c30 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f85880099b0 tx=0x7f8588009250 comp rx=0 tx=0).stop 2023-10-10T19:22:47.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f8595e08700 1 -- 172.21.15.175:0/3879864630 shutdown_connections 2023-10-10T19:22:47.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f8595e08700 1 --2- 172.21.15.175:0/3879864630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f859014fd90 0x7f8590159c30 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f8595e08700 1 --2- 172.21.15.175:0/3879864630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 0x7f85900c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f8595e08700 1 -- 172.21.15.175:0/3879864630 >> 172.21.15.175:0/3879864630 conn(0x7f8590063d00 msgr2=0x7f85900cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:47.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f8595e08700 1 -- 172.21.15.175:0/3879864630 shutdown_connections 2023-10-10T19:22:47.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.480+0000 7f8595e08700 1 -- 172.21.15.175:0/3879864630 wait complete. 2023-10-10T19:22:47.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.481+0000 7f8595e08700 1 Processor -- start 2023-10-10T19:22:47.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8595e08700 1 -- start start 2023-10-10T19:22:47.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8595e08700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 0x7f8590152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:47.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8595e08700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85900cd040 0x7f85900c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:47.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8595e08700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f859014fd90 0x7f85900c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:47.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8595e08700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8590166bd0 con 0x7f85900cd040 2023-10-10T19:22:47.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8595e08700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85901666a0 con 0x7f859014fd90 2023-10-10T19:22:47.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8595e08700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85901668f0 con 0x7f859008a100 2023-10-10T19:22:47.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f858ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85900cd040 0x7f85900c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:47.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f858ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85900cd040 0x7f85900c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42062/0 (socket says 172.21.15.175:42062) 2023-10-10T19:22:47.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f858ffff700 1 -- 172.21.15.175:0/3431353495 learned_addr learned my addr 172.21.15.175:0/3431353495 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:47.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8594e06700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 0x7f8590152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:47.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.482+0000 7f8595607700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f859014fd90 0x7f85900c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:47.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858ffff700 1 -- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 msgr2=0x7f8590152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:47.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858ffff700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 0x7f8590152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858ffff700 1 -- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f859014fd90 msgr2=0x7f85900c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:47.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858ffff700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f859014fd90 0x7f85900c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858ffff700 1 -- 172.21.15.175:0/3431353495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85880094f0 con 0x7f85900cd040 2023-10-10T19:22:47.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858ffff700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85900cd040 0x7f85900c7060 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f858000e190 tx=0x7f8580009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:47.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858cff9700 1 -- 172.21.15.175:0/3431353495 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858000ecd0 con 0x7f85900cd040 2023-10-10T19:22:47.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858cff9700 1 -- 172.21.15.175:0/3431353495 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858000ee30 con 0x7f85900cd040 2023-10-10T19:22:47.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85900a7db0 con 0x7f85900cd040 2023-10-10T19:22:47.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f858cff9700 1 -- 172.21.15.175:0/3431353495 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580014670 con 0x7f85900cd040 2023-10-10T19:22:47.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.483+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85901588d0 con 0x7f85900cd040 2023-10-10T19:22:47.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.484+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85900a7c10 con 0x7f85900cd040 2023-10-10T19:22:47.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.485+0000 7f858cff9700 1 -- 172.21.15.175:0/3431353495 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f85800149e0 con 0x7f85900cd040 2023-10-10T19:22:47.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.485+0000 7f858cff9700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8570066f30 0x7f85700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:47.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.485+0000 7f858cff9700 1 -- 172.21.15.175:0/3431353495 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f85800852a0 con 0x7f85900cd040 2023-10-10T19:22:47.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.485+0000 7f8594e06700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8570066f30 0x7f85700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:47.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.486+0000 7f8594e06700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8570066f30 0x7f85700693e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f85880099b0 tx=0x7f8588017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:47.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.490+0000 7f858cff9700 1 -- 172.21.15.175:0/3431353495 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f85900a7c10 con 0x7f85900cd040 2023-10-10T19:22:47.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.716+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85900c6d80 con 0x7f85900cd040 2023-10-10T19:22:47.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.717+0000 7f858cff9700 1 -- 172.21.15.175:0/3431353495 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 281 v281) v1 ==== 95+0+5423 (secure 0 0 0) 0x7f85900c6d80 con 0x7f85900cd040 2023-10-10T19:22:47.718 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:47.718 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":281,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":281,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:43.025395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:47.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8570066f30 msgr2=0x7f85700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:47.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8570066f30 0x7f85700693e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f85880099b0 tx=0x7f8588017000 comp rx=0 tx=0).stop 2023-10-10T19:22:47.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85900cd040 msgr2=0x7f85900c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:47.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85900cd040 0x7f85900c7060 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f858000e190 tx=0x7f8580009920 comp rx=0 tx=0).stop 2023-10-10T19:22:47.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 shutdown_connections 2023-10-10T19:22:47.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8570066f30 0x7f85700693e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f859014fd90 0x7f85900c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f859008a100 0x7f8590152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 --2- 172.21.15.175:0/3431353495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85900cd040 0x7f85900c7060 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:47.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 >> 172.21.15.175:0/3431353495 conn(0x7f8590063d00 msgr2=0x7f8590154c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:47.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 shutdown_connections 2023-10-10T19:22:47.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:47.719+0000 7f8595e08700 1 -- 172.21.15.175:0/3431353495 wait complete. 2023-10-10T19:22:47.722 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 281 2023-10-10T19:22:47.732 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:22:49.733 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:49.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.854+0000 7f9460e14700 1 Processor -- start 2023-10-10T19:22:49.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.854+0000 7f9460e14700 1 -- start start 2023-10-10T19:22:49.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f9460e14700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 0x7f945c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:49.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f9460e14700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c067fa0 con 0x7f945c08a100 2023-10-10T19:22:49.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f9460e14700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c156b70 con 0x7f945c0cd040 2023-10-10T19:22:49.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f9460e14700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0c5600 con 0x7f945c14fd90 2023-10-10T19:22:49.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945affd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f945c08a100 0x7f945c0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49430/0 (socket says 172.21.15.175:49430) 2023-10-10T19:22:49.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945affd700 1 -- 172.21.15.175:0/2064562717 learned_addr learned my addr 172.21.15.175:0/2064562717 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:49.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945b7fe700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 0x7f945c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:49.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979075604 0 0) 0x7f945c067fa0 con 0x7f945c08a100 2023-10-10T19:22:49.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9444003570 con 0x7f945c08a100 2023-10-10T19:22:49.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236597568 0 0) 0x7f9444003570 con 0x7f945c08a100 2023-10-10T19:22:49.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 msgr2=0x7f945c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:49.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 0x7f945c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:49.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 >> v1:172.21.15.191:6789/0 conn(0x7f945c0cd040 legacy=0x7f945c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:49.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c150160 con 0x7f945c08a100 2023-10-10T19:22:49.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f9450001470 con 0x7f945c08a100 2023-10-10T19:22:49.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.855+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 >> v1:172.21.15.175:6789/0 conn(0x7f945c08a100 legacy=0x7f945c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:49.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9444003a90 0x7f9444003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:49.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 0x7f9444004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:49.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9444004d50 0x7f94440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:49.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0c5600 con 0x7f9444003a90 2023-10-10T19:22:49.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944400bb70 con 0x7f9444004d50 2023-10-10T19:22:49.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944400bd30 con 0x7f945c14fd90 2023-10-10T19:22:49.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945b7fe700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9444003a90 0x7f9444003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:49.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945affd700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 0x7f9444004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:49.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945bfff700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9444004d50 0x7f94440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:49.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945b7fe700 1 -- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 msgr2=0x7f9444004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:49.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945b7fe700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 0x7f9444004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:49.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945b7fe700 1 -- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9444004d50 msgr2=0x7f94440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:49.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945b7fe700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9444004d50 0x7f94440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:49.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945b7fe700 1 -- 172.21.15.175:0/2064562717 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444003570 con 0x7f9444003a90 2023-10-10T19:22:49.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945bfff700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9444004d50 0x7f94440051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:22:49.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945affd700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f945c14fd90 0x7f9444004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:22:49.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945b7fe700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9444003a90 0x7f9444003ea0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f944c00e210 tx=0x7f944c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322bbd0d6263452a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:49.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c00eac0 con 0x7f9444003a90 2023-10-10T19:22:49.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c00f040 con 0x7f9444003a90 2023-10-10T19:22:49.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.856+0000 7f945a7fc700 1 -- 172.21.15.175:0/2064562717 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c00ee70 con 0x7f9444003a90 2023-10-10T19:22:49.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.857+0000 7f9460e14700 1 -- 172.21.15.175:0/2064562717 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9444003a90 msgr2=0x7f9444003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:49.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.857+0000 7f9460e14700 1 --2- 172.21.15.175:0/2064562717 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9444003a90 0x7f9444003ea0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f944c00e210 tx=0x7f944c00b040 comp rx=0 tx=0).stop 2023-10-10T19:22:49.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.857+0000 7f945bfff700 1 -- 172.21.15.175:0/2064562717 reap_dead start 2023-10-10T19:22:49.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.857+0000 7f9460e14700 1 -- 172.21.15.175:0/2064562717 shutdown_connections 2023-10-10T19:22:49.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.857+0000 7f9460e14700 1 -- 172.21.15.175:0/2064562717 >> 172.21.15.175:0/2064562717 conn(0x7f945c063d00 msgr2=0x7f945c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:49.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.857+0000 7f9460e14700 1 -- 172.21.15.175:0/2064562717 shutdown_connections 2023-10-10T19:22:49.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.857+0000 7f9460e14700 1 -- 172.21.15.175:0/2064562717 wait complete. 2023-10-10T19:22:49.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.858+0000 7f9460e14700 1 Processor -- start 2023-10-10T19:22:49.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.858+0000 7f9460e14700 1 -- start start 2023-10-10T19:22:49.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f9460e14700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9444003a90 0x7f945c0cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:49.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f9460e14700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f945c0a7c10 0x7f945c165c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:49.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f9460e14700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945c0cc770 0x7f945c14c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:49.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f9460e14700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1528b0 con 0x7f945c0a7c10 2023-10-10T19:22:49.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f9460e14700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c154d70 con 0x7f945c0cc770 2023-10-10T19:22:49.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f9460e14700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1524f0 con 0x7f9444003a90 2023-10-10T19:22:49.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f945c0a7c10 0x7f945c165c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:49.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f945c0a7c10 0x7f945c165c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42074/0 (socket says 172.21.15.175:42074) 2023-10-10T19:22:49.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 -- 172.21.15.175:0/1463297398 learned_addr learned my addr 172.21.15.175:0/1463297398 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:49.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945bfff700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945c0cc770 0x7f945c14c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:49.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945b7fe700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9444003a90 0x7f945c0cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:49.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 -- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9444003a90 msgr2=0x7f945c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:49.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9444003a90 0x7f945c0cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:49.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 -- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945c0cc770 msgr2=0x7f945c14c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:49.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945c0cc770 0x7f945c14c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:49.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 -- 172.21.15.175:0/1463297398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c152b20 con 0x7f945c0a7c10 2023-10-10T19:22:49.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f945affd700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f945c0a7c10 0x7f945c165c40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f9450000ff0 tx=0x7f945c0c5dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:49.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f94437fe700 1 -- 172.21.15.175:0/1463297398 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450004a40 con 0x7f945c0a7c10 2023-10-10T19:22:49.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.859+0000 7f94437fe700 1 -- 172.21.15.175:0/1463297398 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9450004ba0 con 0x7f945c0a7c10 2023-10-10T19:22:49.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.860+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c152e00 con 0x7f945c0a7c10 2023-10-10T19:22:49.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.860+0000 7f94437fe700 1 -- 172.21.15.175:0/1463297398 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450005050 con 0x7f945c0a7c10 2023-10-10T19:22:49.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.860+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c0cc120 con 0x7f945c0a7c10 2023-10-10T19:22:49.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.860+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c152ff0 con 0x7f945c0a7c10 2023-10-10T19:22:49.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.861+0000 7f94437fe700 1 -- 172.21.15.175:0/1463297398 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f94500053c0 con 0x7f945c0a7c10 2023-10-10T19:22:49.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.861+0000 7f94437fe700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9448066f60 0x7f9448069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:49.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.861+0000 7f94437fe700 1 -- 172.21.15.175:0/1463297398 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f94500843f0 con 0x7f945c0a7c10 2023-10-10T19:22:49.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.861+0000 7f945b7fe700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9448066f60 0x7f9448069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:49.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.862+0000 7f945b7fe700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9448066f60 0x7f9448069410 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f944c00afd0 tx=0x7f944c01a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:49.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:49.866+0000 7f94437fe700 1 -- 172.21.15.175:0/1463297398 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f945004d5a0 con 0x7f945c0a7c10 2023-10-10T19:22:50.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.093+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f945c1534e0 con 0x7f945c0a7c10 2023-10-10T19:22:50.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.094+0000 7f94437fe700 1 -- 172.21.15.175:0/1463297398 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 282 v282) v1 ==== 95+0+5423 (secure 0 0 0) 0x7f9450051100 con 0x7f945c0a7c10 2023-10-10T19:22:50.095 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:50.095 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":282,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:49.060936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":90,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:50.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9448066f60 msgr2=0x7f9448069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:50.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9448066f60 0x7f9448069410 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f944c00afd0 tx=0x7f944c01a000 comp rx=0 tx=0).stop 2023-10-10T19:22:50.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f945c0a7c10 msgr2=0x7f945c165c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:50.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f945c0a7c10 0x7f945c165c40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f9450000ff0 tx=0x7f945c0c5dd0 comp rx=0 tx=0).stop 2023-10-10T19:22:50.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 shutdown_connections 2023-10-10T19:22:50.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9448066f60 0x7f9448069410 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:50.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945c0cc770 0x7f945c14c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:50.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9444003a90 0x7f945c0cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:50.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 --2- 172.21.15.175:0/1463297398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f945c0a7c10 0x7f945c165c40 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:50.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 >> 172.21.15.175:0/1463297398 conn(0x7f945c063d00 msgr2=0x7f945c158db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:50.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 shutdown_connections 2023-10-10T19:22:50.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:50.096+0000 7f9460e14700 1 -- 172.21.15.175:0/1463297398 wait complete. 2023-10-10T19:22:50.100 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 282 2023-10-10T19:22:50.109 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=3, replacing gid=6618 2023-10-10T19:22:50.110 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.c 2023-10-10T19:22:52.954 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-10T19:22:52.954+0000 7fb93ba0c700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T19:22:55.590 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:22:55.593 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:22:58.111 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-10T19:22:58.111 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-10T19:22:58.112 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-10T19:22:58.115 INFO:tasks.ceph.mds.c:Started 2023-10-10T19:22:58.115 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:22:58.197 INFO:tasks.ceph.mds.c.smithi175.stdout:starting mds.c at 2023-10-10T19:22:58.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.243+0000 7fbfe5818700 1 Processor -- start 2023-10-10T19:22:58.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.243+0000 7fbfe5818700 1 -- start start 2023-10-10T19:22:58.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfe5818700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe014fd90 0x7fbfe0159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:58.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfe5818700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe0067fa0 con 0x7fbfe008a100 2023-10-10T19:22:58.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfe5818700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe0156b70 con 0x7fbfe00cd040 2023-10-10T19:22:58.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfe5818700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00c5600 con 0x7fbfe014fd90 2023-10-10T19:22:58.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd7fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fbfe008a100 0x7fbfe00c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55342/0 (socket says 172.21.15.175:55342) 2023-10-10T19:22:58.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd7fff700 1 -- 172.21.15.175:0/3284124642 learned_addr learned my addr 172.21.15.175:0/3284124642 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:22:58.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfe4816700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe014fd90 0x7fbfe0159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:58.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722359171 0 0) 0x7fbfe0067fa0 con 0x7fbfe008a100 2023-10-10T19:22:58.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc8003570 con 0x7fbfe008a100 2023-10-10T19:22:58.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1858808951 0 0) 0x7fbfc8003570 con 0x7fbfe008a100 2023-10-10T19:22:58.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe014fd90 msgr2=0x7fbfe0159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:58.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd77fe700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe014fd90 0x7fbfe0159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 >> v1:172.21.15.191:6789/0 conn(0x7fbfe00cd040 legacy=0x7fbfe00684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:58.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0150160 con 0x7fbfe008a100 2023-10-10T19:22:58.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.244+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fbfd0001470 con 0x7fbfe008a100 2023-10-10T19:22:58.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 >> v1:172.21.15.175:6789/0 conn(0x7fbfe008a100 legacy=0x7fbfe00c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:58.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfc8003a90 0x7fbfc8003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:58.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc8004810 0x7fbfc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:58.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe014fd90 0x7fbfc80051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:58.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00c5600 con 0x7fbfc8003a90 2023-10-10T19:22:58.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc800bb60 con 0x7fbfc8004810 2023-10-10T19:22:58.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc800bd20 con 0x7fbfe014fd90 2023-10-10T19:22:58.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfe4816700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfc8003a90 0x7fbfc8003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:58.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd7fff700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc8004810 0x7fbfc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:58.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfe5017700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe014fd90 0x7fbfc80051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:58.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfe4816700 1 -- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe014fd90 msgr2=0x7fbfc80051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:58.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfe4816700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe014fd90 0x7fbfc80051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfe4816700 1 -- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc8004810 msgr2=0x7fbfc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:22:58.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfe4816700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc8004810 0x7fbfc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfe4816700 1 -- 172.21.15.175:0/3284124642 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8003570 con 0x7fbfc8003a90 2023-10-10T19:22:58.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfe4816700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfc8003a90 0x7fbfc8003ea0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009c00 tx=0x7fbfdc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ebb2aca8935641 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:58.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00e710 con 0x7fbfc8003a90 2023-10-10T19:22:58.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc00e870 con 0x7fbfc8003a90 2023-10-10T19:22:58.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.245+0000 7fbfd77fe700 1 -- 172.21.15.175:0/3284124642 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00ec30 con 0x7fbfc8003a90 2023-10-10T19:22:58.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.246+0000 7fbfe5818700 1 -- 172.21.15.175:0/3284124642 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfc8003a90 msgr2=0x7fbfc8003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:58.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.246+0000 7fbfe5818700 1 --2- 172.21.15.175:0/3284124642 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfc8003a90 0x7fbfc8003ea0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009c00 tx=0x7fbfdc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.246+0000 7fbfe5017700 1 -- 172.21.15.175:0/3284124642 reap_dead start 2023-10-10T19:22:58.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.246+0000 7fbfe5818700 1 -- 172.21.15.175:0/3284124642 shutdown_connections 2023-10-10T19:22:58.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.246+0000 7fbfe5818700 1 -- 172.21.15.175:0/3284124642 >> 172.21.15.175:0/3284124642 conn(0x7fbfe0063d00 msgr2=0x7fbfe00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:58.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.246+0000 7fbfe5818700 1 -- 172.21.15.175:0/3284124642 shutdown_connections 2023-10-10T19:22:58.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.246+0000 7fbfe5818700 1 -- 172.21.15.175:0/3284124642 wait complete. 2023-10-10T19:22:58.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.247+0000 7fbfe5818700 1 Processor -- start 2023-10-10T19:22:58.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.247+0000 7fbfe5818700 1 -- start start 2023-10-10T19:22:58.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5818700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfc8003a90 0x7fbfe00cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:58.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5818700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfe00a7c10 0x7fbfe0165c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:58.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5818700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe00cc770 0x7fbfe014c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:58.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5818700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe01528b0 con 0x7fbfe00a7c10 2023-10-10T19:22:58.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5818700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0154d70 con 0x7fbfe00cc770 2023-10-10T19:22:58.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5818700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe01524f0 con 0x7fbfc8003a90 2023-10-10T19:22:58.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfe00a7c10 0x7fbfe0165c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:58.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfe00a7c10 0x7fbfe0165c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51776/0 (socket says 172.21.15.175:51776) 2023-10-10T19:22:58.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 -- 172.21.15.175:0/668497606 learned_addr learned my addr 172.21.15.175:0/668497606 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:22:58.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5017700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe00cc770 0x7fbfe014c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:58.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe4816700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfc8003a90 0x7fbfe00cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:58.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 -- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfc8003a90 msgr2=0x7fbfe00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:58.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfc8003a90 0x7fbfe00cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 -- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe00cc770 msgr2=0x7fbfe014c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:58.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe00cc770 0x7fbfe014c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 -- 172.21.15.175:0/668497606 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0152b20 con 0x7fbfe00a7c10 2023-10-10T19:22:58.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5017700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe00cc770 0x7fbfe014c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:22:58.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd7fff700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfe00a7c10 0x7fbfe0165c40 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0007390 tx=0x7fbfe00c5dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:58.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd4ff9700 1 -- 172.21.15.175:0/668497606 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd000f8d0 con 0x7fbfe00a7c10 2023-10-10T19:22:58.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd4ff9700 1 -- 172.21.15.175:0/668497606 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd000fa70 con 0x7fbfe00a7c10 2023-10-10T19:22:58.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe0152e00 con 0x7fbfe00a7c10 2023-10-10T19:22:58.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfd4ff9700 1 -- 172.21.15.175:0/668497606 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd000fe30 con 0x7fbfe00a7c10 2023-10-10T19:22:58.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.248+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe00cc120 con 0x7fbfe00a7c10 2023-10-10T19:22:58.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.249+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe0152ff0 con 0x7fbfe00a7c10 2023-10-10T19:22:58.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.250+0000 7fbfd4ff9700 1 -- 172.21.15.175:0/668497606 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbfd0013600 con 0x7fbfe00a7c10 2023-10-10T19:22:58.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.250+0000 7fbfd4ff9700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbfcc066f60 0x7fbfcc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:22:58.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.250+0000 7fbfd4ff9700 1 -- 172.21.15.175:0/668497606 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fbfd0084e80 con 0x7fbfe00a7c10 2023-10-10T19:22:58.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.250+0000 7fbfe4816700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbfcc066f60 0x7fbfcc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:22:58.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.251+0000 7fbfe4816700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbfcc066f60 0x7fbfcc069410 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0091e0 tx=0x7fbfdc01a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:22:58.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.255+0000 7fbfd4ff9700 1 -- 172.21.15.175:0/668497606 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfd0051a20 con 0x7fbfe00a7c10 2023-10-10T19:22:58.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.482+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfe01534e0 con 0x7fbfe00a7c10 2023-10-10T19:22:58.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.482+0000 7fbfd4ff9700 1 -- 172.21.15.175:0/668497606 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+5424 (secure 0 0 0) 0x7fbfd0055040 con 0x7fbfe00a7c10 2023-10-10T19:22:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:22:58.483 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:22:58.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbfcc066f60 msgr2=0x7fbfcc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:58.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbfcc066f60 0x7fbfcc069410 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0091e0 tx=0x7fbfdc01a000 comp rx=0 tx=0).stop 2023-10-10T19:22:58.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfe00a7c10 msgr2=0x7fbfe0165c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:22:58.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfe00a7c10 0x7fbfe0165c40 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0007390 tx=0x7fbfe00c5dd0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 shutdown_connections 2023-10-10T19:22:58.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbfcc066f60 0x7fbfcc069410 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe00cc770 0x7fbfe014c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfc8003a90 0x7fbfe00cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 --2- 172.21.15.175:0/668497606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbfe00a7c10 0x7fbfe0165c40 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:22:58.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 >> 172.21.15.175:0/668497606 conn(0x7fbfe0063d00 msgr2=0x7fbfe0158db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:22:58.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 shutdown_connections 2023-10-10T19:22:58.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:22:58.484+0000 7fbfe5818700 1 -- 172.21.15.175:0/668497606 wait complete. 2023-10-10T19:22:58.488 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 283 2023-10-10T19:22:58.497 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-10T19:23:00.498 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:00.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.624+0000 7f988cfe1700 1 Processor -- start 2023-10-10T19:23:00.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.624+0000 7f988cfe1700 1 -- start start 2023-10-10T19:23:00.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.625+0000 7f988cfe1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f988806a650 0x7f98880c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:00.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.625+0000 7f988cfe1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9888068c90 con 0x7f98880c5630 2023-10-10T19:23:00.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.625+0000 7f988cfe1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9888069120 con 0x7f9888159a70 2023-10-10T19:23:00.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.625+0000 7f988cfe1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98880c5b90 con 0x7f988806a650 2023-10-10T19:23:00.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.625+0000 7f987dd9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f988806a650 0x7f98880c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:00.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.625+0000 7f9886d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f98880c5630 0x7f9888156470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:55344/0 (socket says 172.21.15.175:55344) 2023-10-10T19:23:00.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.625+0000 7f987dd9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f988806a650 0x7f98880c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51792/0 (socket says 172.21.15.175:51792) 2023-10-10T19:23:00.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.625+0000 7f9886d9d700 1 -- 172.21.15.175:0/3136213088 learned_addr learned my addr 172.21.15.175:0/3136213088 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:00.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f987dd9b700 1 -- 172.21.15.175:0/3136213088 >> v1:172.21.15.191:6789/0 conn(0x7f9888159a70 legacy=0x7f9888159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:00.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f987dd9b700 1 -- 172.21.15.175:0/3136213088 >> v1:172.21.15.175:6789/0 conn(0x7f98880c5630 legacy=0x7f9888156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:00.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f987dd9b700 1 -- 172.21.15.175:0/3136213088 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98880639e0 con 0x7f988806a650 2023-10-10T19:23:00.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f9885d9b700 1 -- 172.21.15.175:0/3136213088 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216449663 0 0) 0x7f9888068c90 con 0x7f98880c5630 2023-10-10T19:23:00.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f987dd9b700 1 --2- 172.21.15.175:0/3136213088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988806a650 0x7f98880c7060 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f98700098e0 tx=0x7f98700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0921e11efa2fb93 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:00.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f9885d9b700 1 -- 172.21.15.175:0/3136213088 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9870009e80 con 0x7f988806a650 2023-10-10T19:23:00.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f9885d9b700 1 -- 172.21.15.175:0/3136213088 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987000b8f0 con 0x7f988806a650 2023-10-10T19:23:00.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f9885d9b700 1 -- 172.21.15.175:0/3136213088 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987000bc70 con 0x7f988806a650 2023-10-10T19:23:00.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f988cfe1700 1 -- 172.21.15.175:0/3136213088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988806a650 msgr2=0x7f98880c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:00.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.626+0000 7f988cfe1700 1 --2- 172.21.15.175:0/3136213088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988806a650 0x7f98880c7060 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f98700098e0 tx=0x7f98700091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:00.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.627+0000 7f988cfe1700 1 -- 172.21.15.175:0/3136213088 shutdown_connections 2023-10-10T19:23:00.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.627+0000 7f988cfe1700 1 --2- 172.21.15.175:0/3136213088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988806a650 0x7f98880c7060 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:00.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.627+0000 7f988cfe1700 1 -- 172.21.15.175:0/3136213088 >> 172.21.15.175:0/3136213088 conn(0x7f988806dc80 msgr2=0x7f9888063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:00.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.627+0000 7f988cfe1700 1 -- 172.21.15.175:0/3136213088 shutdown_connections 2023-10-10T19:23:00.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.627+0000 7f988cfe1700 1 -- 172.21.15.175:0/3136213088 wait complete. 2023-10-10T19:23:00.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.628+0000 7f988cfe1700 1 Processor -- start 2023-10-10T19:23:00.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.628+0000 7f988cfe1700 1 -- start start 2023-10-10T19:23:00.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988cfe1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f988806a650 0x7f98880c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:00.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988cfe1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98880c5630 0x7f9888151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:00.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988cfe1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9888159a70 0x7f9888166f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:00.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988cfe1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98880c3c40 con 0x7f9888159a70 2023-10-10T19:23:00.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988cfe1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98880c48c0 con 0x7f98880c5630 2023-10-10T19:23:00.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988cfe1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98880c2d50 con 0x7f988806a650 2023-10-10T19:23:00.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9888159a70 0x7f9888166f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:00.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9888159a70 0x7f9888166f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51804/0 (socket says 172.21.15.175:51804) 2023-10-10T19:23:00.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 -- 172.21.15.175:0/1210545406 learned_addr learned my addr 172.21.15.175:0/1210545406 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f987dd9b700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98880c5630 0x7f9888151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988659c700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f988806a650 0x7f98880c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 -- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f988806a650 msgr2=0x7f98880c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f988806a650 0x7f98880c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 -- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98880c5630 msgr2=0x7f9888151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98880c5630 0x7f9888151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 -- 172.21.15.175:0/1210545406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9870009420 con 0x7f9888159a70 2023-10-10T19:23:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988659c700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f988806a650 0x7f98880c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:23:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f9886d9d700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9888159a70 0x7f9888166f00 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f988000ccc0 tx=0x7f9880001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f987effd700 1 -- 172.21.15.175:0/1210545406 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988000a950 con 0x7f9888159a70 2023-10-10T19:23:00.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f987effd700 1 -- 172.21.15.175:0/1210545406 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988000aab0 con 0x7f9888159a70 2023-10-10T19:23:00.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98880c2fc0 con 0x7f9888159a70 2023-10-10T19:23:00.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f987effd700 1 -- 172.21.15.175:0/1210545406 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988000ae70 con 0x7f9888159a70 2023-10-10T19:23:00.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.629+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98880c3580 con 0x7f9888159a70 2023-10-10T19:23:00.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.630+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98880c3d80 con 0x7f9888159a70 2023-10-10T19:23:00.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.631+0000 7f987effd700 1 -- 172.21.15.175:0/1210545406 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9880007300 con 0x7f9888159a70 2023-10-10T19:23:00.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.631+0000 7f987effd700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9868066f60 0x7f9868069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:00.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.631+0000 7f987effd700 1 -- 172.21.15.175:0/1210545406 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f9880085140 con 0x7f9888159a70 2023-10-10T19:23:00.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.631+0000 7f988659c700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9868066f60 0x7f9868069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:00.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.632+0000 7f988659c700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9868066f60 0x7f9868069410 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9878007fd0 tx=0x7f9878009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:00.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.635+0000 7f987effd700 1 -- 172.21.15.175:0/1210545406 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9880051c40 con 0x7f9888159a70 2023-10-10T19:23:00.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.859+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9888063cc0 con 0x7f9888159a70 2023-10-10T19:23:00.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.859+0000 7f987effd700 1 -- 172.21.15.175:0/1210545406 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7f9880055260 con 0x7f9888159a70 2023-10-10T19:23:00.860 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:00.860 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:00.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9868066f60 msgr2=0x7f9868069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:00.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9868066f60 0x7f9868069410 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9878007fd0 tx=0x7f9878009040 comp rx=0 tx=0).stop 2023-10-10T19:23:00.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9888159a70 msgr2=0x7f9888166f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:00.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9888159a70 0x7f9888166f00 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f988000ccc0 tx=0x7f9880001a80 comp rx=0 tx=0).stop 2023-10-10T19:23:00.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 shutdown_connections 2023-10-10T19:23:00.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9868066f60 0x7f9868069410 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:00.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9888159a70 0x7f9888166f00 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:00.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f988806a650 0x7f98880c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:00.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 --2- 172.21.15.175:0/1210545406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98880c5630 0x7f9888151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:00.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 >> 172.21.15.175:0/1210545406 conn(0x7f988806dc80 msgr2=0x7f98880c9430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:00.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 shutdown_connections 2023-10-10T19:23:00.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:00.861+0000 7f988cfe1700 1 -- 172.21.15.175:0/1210545406 wait complete. 2023-10-10T19:23:00.868 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:00.874 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-10T19:23:00.874 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-10-10T19:23:19.875 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:20.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.001+0000 7f4bc4be3700 1 Processor -- start 2023-10-10T19:23:20.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.001+0000 7f4bc4be3700 1 -- start start 2023-10-10T19:23:20.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.002+0000 7f4bc4be3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4bc00b1fc0 0x7f4bc0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.002+0000 7f4bc4be3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4bc00b2390 0x7f4bc0068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.002+0000 7f4bc4be3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc01574b0 con 0x7f4bc0068bd0 2023-10-10T19:23:20.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.002+0000 7f4bc4be3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc0150770 con 0x7f4bc00b2390 2023-10-10T19:23:20.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.002+0000 7f4bc4be3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc00afad0 con 0x7f4bc00b1fc0 2023-10-10T19:23:20.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.002+0000 7f4bbdd9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4bc00b2390 0x7f4bc0068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.002+0000 7f4bbdd9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4bc00b2390 0x7f4bc0068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57976/0 (socket says 172.21.15.175:57976) 2023-10-10T19:23:20.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.002+0000 7f4bbdd9b700 1 -- 172.21.15.175:0/1703150872 learned_addr learned my addr 172.21.15.175:0/1703150872 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:20.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbe59c700 1 --2- 172.21.15.175:0/1703150872 >> v2:172.21.15.191:3300/0 conn(0x7f4bc00b1fc0 0x7f4bc0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbdd9b700 1 -- 172.21.15.175:0/1703150872 >> v2:172.21.15.191:3300/0 conn(0x7f4bc00b1fc0 msgr2=0x7f4bc0155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:20.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbdd9b700 1 --2- 172.21.15.175:0/1703150872 >> v2:172.21.15.191:3300/0 conn(0x7f4bc00b1fc0 0x7f4bc0155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbdd9b700 1 -- 172.21.15.175:0/1703150872 >> v1:172.21.15.191:6789/0 conn(0x7f4bc0068bd0 legacy=0x7f4bc00aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:20.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbdd9b700 1 -- 172.21.15.175:0/1703150872 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc00af4c0 con 0x7f4bc00b2390 2023-10-10T19:23:20.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbdd9b700 1 --2- 172.21.15.175:0/1703150872 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc0068690 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00b28c0 tx=0x7f4bb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb77d0d48081a6d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:20.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbd59a700 1 -- 172.21.15.175:0/1703150872 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400b8f0 con 0x7f4bc00b2390 2023-10-10T19:23:20.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbd59a700 1 -- 172.21.15.175:0/1703150872 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb400ba50 con 0x7f4bc00b2390 2023-10-10T19:23:20.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.003+0000 7f4bbd59a700 1 -- 172.21.15.175:0/1703150872 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400bdf0 con 0x7f4bc00b2390 2023-10-10T19:23:20.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.004+0000 7f4bc4be3700 1 -- 172.21.15.175:0/1703150872 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 msgr2=0x7f4bc0068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.004+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/1703150872 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc0068690 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00b28c0 tx=0x7f4bb4009250 comp rx=0 tx=0).stop 2023-10-10T19:23:20.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.004+0000 7f4bc4be3700 1 -- 172.21.15.175:0/1703150872 shutdown_connections 2023-10-10T19:23:20.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.004+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/1703150872 >> v2:172.21.15.191:3300/0 conn(0x7f4bc00b1fc0 0x7f4bc0155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.004+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/1703150872 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc0068690 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.004+0000 7f4bc4be3700 1 -- 172.21.15.175:0/1703150872 >> 172.21.15.175:0/1703150872 conn(0x7f4bc0063230 msgr2=0x7f4bc0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:20.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.004+0000 7f4bc4be3700 1 -- 172.21.15.175:0/1703150872 shutdown_connections 2023-10-10T19:23:20.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.004+0000 7f4bc4be3700 1 -- 172.21.15.175:0/1703150872 wait complete. 2023-10-10T19:23:20.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.005+0000 7f4bc4be3700 1 Processor -- start 2023-10-10T19:23:20.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.005+0000 7f4bc4be3700 1 -- start start 2023-10-10T19:23:20.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.005+0000 7f4bc4be3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc0068bd0 0x7f4bc0157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.005+0000 7f4bc4be3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc00b1fc0 0x7f4bc00cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.005+0000 7f4bc4be3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc00cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bc4be3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00cb6e0 con 0x7f4bc00b2390 2023-10-10T19:23:20.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bc4be3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00cbf40 con 0x7f4bc00b1fc0 2023-10-10T19:23:20.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bc4be3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00cab60 con 0x7f4bc0068bd0 2023-10-10T19:23:20.014 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc00cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc00cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57978/0 (socket says 172.21.15.175:57978) 2023-10-10T19:23:20.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 -- 172.21.15.175:0/204171024 learned_addr learned my addr 172.21.15.175:0/204171024 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:20.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbdd9b700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc00b1fc0 0x7f4bc00cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbe59c700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc0068bd0 0x7f4bc0157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 -- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc0068bd0 msgr2=0x7f4bc0157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc0068bd0 0x7f4bc0157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 -- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc00b1fc0 msgr2=0x7f4bc00cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc00b1fc0 0x7f4bc00cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 -- 172.21.15.175:0/204171024 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb40094f0 con 0x7f4bc00b2390 2023-10-10T19:23:20.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbe59c700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc0068bd0 0x7f4bc0157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:23:20.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bbed9d700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc00cc960 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800c560 tx=0x7f4bb8007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:20.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bae7fc700 1 -- 172.21.15.175:0/204171024 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8011590 con 0x7f4bc00b2390 2023-10-10T19:23:20.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bae7fc700 1 -- 172.21.15.175:0/204171024 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb80116f0 con 0x7f4bc00b2390 2023-10-10T19:23:20.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc00cae40 con 0x7f4bc00b2390 2023-10-10T19:23:20.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bae7fc700 1 -- 172.21.15.175:0/204171024 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8011ab0 con 0x7f4bc00b2390 2023-10-10T19:23:20.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.006+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc00cb400 con 0x7f4bc00b2390 2023-10-10T19:23:20.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.007+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc00c36a0 con 0x7f4bc00b2390 2023-10-10T19:23:20.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.008+0000 7f4bae7fc700 1 -- 172.21.15.175:0/204171024 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4bb801b3b0 con 0x7f4bc00b2390 2023-10-10T19:23:20.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.008+0000 7f4bae7fc700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4ba0066f30 0x7f4ba00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.008+0000 7f4bae7fc700 1 -- 172.21.15.175:0/204171024 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f4bb8084060 con 0x7f4bc00b2390 2023-10-10T19:23:20.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.008+0000 7f4bbe59c700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4ba0066f30 0x7f4ba00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.009+0000 7f4bbe59c700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4ba0066f30 0x7f4ba00693e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4bb00075e0 tx=0x7f4bb00097e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:20.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.012+0000 7f4bae7fc700 1 -- 172.21.15.175:0/204171024 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4bb8050c00 con 0x7f4bc00b2390 2023-10-10T19:23:20.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.240+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bc00cc100 con 0x7f4bc00b2390 2023-10-10T19:23:20.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.241+0000 7f4bae7fc700 1 -- 172.21.15.175:0/204171024 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7f4bb8017020 con 0x7f4bc00b2390 2023-10-10T19:23:20.243 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:20.243 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:20.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4ba0066f30 msgr2=0x7f4ba00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4ba0066f30 0x7f4ba00693e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4bb00075e0 tx=0x7f4bb00097e0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 msgr2=0x7f4bc00cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc00cc960 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800c560 tx=0x7f4bb8007ef0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 shutdown_connections 2023-10-10T19:23:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4ba0066f30 0x7f4ba00693e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4bc00b2390 0x7f4bc00cc960 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc0068bd0 0x7f4bc0157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 --2- 172.21.15.175:0/204171024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc00b1fc0 0x7f4bc00cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 >> 172.21.15.175:0/204171024 conn(0x7f4bc0063230 msgr2=0x7f4bc00c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 shutdown_connections 2023-10-10T19:23:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.243+0000 7f4bc4be3700 1 -- 172.21.15.175:0/204171024 wait complete. 2023-10-10T19:23:20.247 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:20.256 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2023-10-10T19:23:20.257 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-10T19:23:20.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:23:20.266 INFO:tasks.ceph.mds.e:Stopped 2023-10-10T19:23:20.267 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:20.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.419+0000 7fd337191700 1 Processor -- start 2023-10-10T19:23:20.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.419+0000 7fd337191700 1 -- start start 2023-10-10T19:23:20.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd337191700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd3300c4580 0x7fd3301504f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd337191700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd330157030 con 0x7fd3300c7190 2023-10-10T19:23:20.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd337191700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd330156420 con 0x7fd3300c6330 2023-10-10T19:23:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd337191700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3300c5000 con 0x7fd3300c4580 2023-10-10T19:23:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd33572e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd3300c4580 0x7fd3301504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd32ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd3300c7190 0x7fd33014fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47758/0 (socket says 172.21.15.175:47758) 2023-10-10T19:23:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd32ffff700 1 -- 172.21.15.175:0/3408053439 learned_addr learned my addr 172.21.15.175:0/3408053439 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd33572e700 1 -- 172.21.15.175:0/3408053439 >> v1:172.21.15.191:6789/0 conn(0x7fd3300c6330 legacy=0x7fd3300c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:20.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd33572e700 1 -- 172.21.15.175:0/3408053439 >> v1:172.21.15.175:6789/0 conn(0x7fd3300c7190 legacy=0x7fd33014fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd32f7fe700 1 -- 172.21.15.175:0/3408053439 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853673422 0 0) 0x7fd330157030 con 0x7fd3300c7190 2023-10-10T19:23:20.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.420+0000 7fd33572e700 1 -- 172.21.15.175:0/3408053439 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd330156700 con 0x7fd3300c4580 2023-10-10T19:23:20.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd33572e700 1 --2- 172.21.15.175:0/3408053439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c4580 0x7fd3301504f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fd328009e60 tx=0x7fd328009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ae0b7a14328d68 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:20.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd32f7fe700 1 -- 172.21.15.175:0/3408053439 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32800e070 con 0x7fd3300c4580 2023-10-10T19:23:20.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd32f7fe700 1 -- 172.21.15.175:0/3408053439 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32800bc80 con 0x7fd3300c4580 2023-10-10T19:23:20.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd32f7fe700 1 -- 172.21.15.175:0/3408053439 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd328012430 con 0x7fd3300c4580 2023-10-10T19:23:20.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd337191700 1 -- 172.21.15.175:0/3408053439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c4580 msgr2=0x7fd3301504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd337191700 1 --2- 172.21.15.175:0/3408053439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c4580 0x7fd3301504f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fd328009e60 tx=0x7fd328009780 comp rx=0 tx=0).stop 2023-10-10T19:23:20.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd337191700 1 -- 172.21.15.175:0/3408053439 shutdown_connections 2023-10-10T19:23:20.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd337191700 1 --2- 172.21.15.175:0/3408053439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c4580 0x7fd3301504f0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd337191700 1 -- 172.21.15.175:0/3408053439 >> 172.21.15.175:0/3408053439 conn(0x7fd33006ef20 msgr2=0x7fd330063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:20.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd337191700 1 -- 172.21.15.175:0/3408053439 shutdown_connections 2023-10-10T19:23:20.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.421+0000 7fd337191700 1 -- 172.21.15.175:0/3408053439 wait complete. 2023-10-10T19:23:20.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.422+0000 7fd337191700 1 Processor -- start 2023-10-10T19:23:20.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd337191700 1 -- start start 2023-10-10T19:23:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd337191700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3300c4580 0x7fd330159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd337191700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c6330 0x7fd330158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd337191700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c7190 0x7fd330149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd337191700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3301589f0 con 0x7fd3300c6330 2023-10-10T19:23:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd337191700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300c8090 con 0x7fd3300c7190 2023-10-10T19:23:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd337191700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300cad10 con 0x7fd3300c4580 2023-10-10T19:23:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd32ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c6330 0x7fd330158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd32ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c6330 0x7fd330158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58000/0 (socket says 172.21.15.175:58000) 2023-10-10T19:23:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd32ffff700 1 -- 172.21.15.175:0/1168765324 learned_addr learned my addr 172.21.15.175:0/1168765324 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd334f2d700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3300c4580 0x7fd330159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.423+0000 7fd33572e700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c7190 0x7fd330149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32ffff700 1 -- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3300c4580 msgr2=0x7fd330159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32ffff700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3300c4580 0x7fd330159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32ffff700 1 -- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c7190 msgr2=0x7fd330149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32ffff700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c7190 0x7fd330149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32ffff700 1 -- 172.21.15.175:0/1168765324 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd328009a20 con 0x7fd3300c6330 2023-10-10T19:23:20.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32ffff700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c6330 0x7fd330158080 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00c5e0 tx=0x7fd31c001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:20.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32cff9700 1 -- 172.21.15.175:0/1168765324 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c005b30 con 0x7fd3300c6330 2023-10-10T19:23:20.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32cff9700 1 -- 172.21.15.175:0/1168765324 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31c005c90 con 0x7fd3300c6330 2023-10-10T19:23:20.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd32cff9700 1 -- 172.21.15.175:0/1168765324 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0072e0 con 0x7fd3300c6330 2023-10-10T19:23:20.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd330158710 con 0x7fd3300c6330 2023-10-10T19:23:20.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.424+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3300c7a40 con 0x7fd3300c6330 2023-10-10T19:23:20.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.425+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd330067570 con 0x7fd3300c6330 2023-10-10T19:23:20.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.425+0000 7fd32cff9700 1 -- 172.21.15.175:0/1168765324 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd31c007690 con 0x7fd3300c6330 2023-10-10T19:23:20.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.425+0000 7fd32cff9700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:20.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.426+0000 7fd32cff9700 1 -- 172.21.15.175:0/1168765324 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fd31c083ff0 con 0x7fd3300c6330 2023-10-10T19:23:20.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.426+0000 7fd334f2d700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:20.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.426+0000 7fd334f2d700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd324003870 tx=0x7fd324009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:20.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.430+0000 7fd32cff9700 1 -- 172.21.15.175:0/1168765324 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd31c04b5a0 con 0x7fd3300c6330 2023-10-10T19:23:20.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.659+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3301569e0 con 0x7fd3300c6330 2023-10-10T19:23:20.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.660+0000 7fd32cff9700 1 -- 172.21.15.175:0/1168765324 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7fd31c050c40 con 0x7fd3300c6330 2023-10-10T19:23:20.661 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:20.662 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:20.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 msgr2=0x7fd3140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd324003870 tx=0x7fd324009040 comp rx=0 tx=0).stop 2023-10-10T19:23:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c6330 msgr2=0x7fd330158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c6330 0x7fd330158080 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00c5e0 tx=0x7fd31c001070 comp rx=0 tx=0).stop 2023-10-10T19:23:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 shutdown_connections 2023-10-10T19:23:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd314066f30 0x7fd3140693e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3300c7190 0x7fd330149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3300c4580 0x7fd330159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 --2- 172.21.15.175:0/1168765324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3300c6330 0x7fd330158080 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:20.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 >> 172.21.15.175:0/1168765324 conn(0x7fd33006ef20 msgr2=0x7fd330153f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:20.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 shutdown_connections 2023-10-10T19:23:20.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:20.662+0000 7fd337191700 1 -- 172.21.15.175:0/1168765324 wait complete. 2023-10-10T19:23:20.666 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:20.675 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:23:22.676 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:22.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.800+0000 7f73f960c700 1 Processor -- start 2023-10-10T19:23:22.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.800+0000 7f73f960c700 1 -- start start 2023-10-10T19:23:22.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.801+0000 7f73f960c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f73f40c30c0 0x7f73f414db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:22.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.801+0000 7f73f960c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f414e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:22.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.801+0000 7f73f960c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f406ec10 con 0x7f73f40c2cf0 2023-10-10T19:23:22.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.801+0000 7f73f960c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f414e600 con 0x7f73f40c30c0 2023-10-10T19:23:22.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.801+0000 7f73f960c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4153f70 con 0x7f73f40c0270 2023-10-10T19:23:22.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f27fc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f73f40c30c0 0x7f73f414db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:22.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f37fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f414e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:22.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f2ffd700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f73f40c2cf0 0x7f73f406e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:33220/0 (socket says 172.21.15.175:33220) 2023-10-10T19:23:22.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f2ffd700 1 -- 172.21.15.175:0/3372141141 learned_addr learned my addr 172.21.15.175:0/3372141141 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:22.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f1ffb700 1 -- 172.21.15.175:0/3372141141 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071730011 0 0) 0x7f73f406ec10 con 0x7f73f40c2cf0 2023-10-10T19:23:22.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f1ffb700 1 -- 172.21.15.175:0/3372141141 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e0003570 con 0x7f73f40c2cf0 2023-10-10T19:23:22.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f27fc700 1 -- 172.21.15.175:0/3372141141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 msgr2=0x7f73f414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:22.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f27fc700 1 --2- 172.21.15.175:0/3372141141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f414e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:22.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f27fc700 1 -- 172.21.15.175:0/3372141141 >> v1:172.21.15.191:6789/0 conn(0x7f73f40c2cf0 legacy=0x7f73f406e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:22.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f27fc700 1 -- 172.21.15.175:0/3372141141 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f4067bd0 con 0x7f73f40c30c0 2023-10-10T19:23:22.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f37fe700 1 --2- 172.21.15.175:0/3372141141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:23:22.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f27fc700 1 --2- 172.21.15.175:0/3372141141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 0x7f73f414db80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0098e0 tx=0x7f73dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83c5b15cc0e9ecf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:22.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f1ffb700 1 -- 172.21.15.175:0/3372141141 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc011630 con 0x7f73f40c30c0 2023-10-10T19:23:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f1ffb700 1 -- 172.21.15.175:0/3372141141 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc011790 con 0x7f73f40c30c0 2023-10-10T19:23:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.802+0000 7f73f1ffb700 1 -- 172.21.15.175:0/3372141141 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc011b30 con 0x7f73f40c30c0 2023-10-10T19:23:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.803+0000 7f73f960c700 1 -- 172.21.15.175:0/3372141141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 msgr2=0x7f73f414db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.803+0000 7f73f960c700 1 --2- 172.21.15.175:0/3372141141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 0x7f73f414db80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0098e0 tx=0x7f73dc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.803+0000 7f73f960c700 1 -- 172.21.15.175:0/3372141141 shutdown_connections 2023-10-10T19:23:22.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.803+0000 7f73f960c700 1 --2- 172.21.15.175:0/3372141141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:22.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.803+0000 7f73f960c700 1 --2- 172.21.15.175:0/3372141141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 0x7f73f414db80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:22.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.803+0000 7f73f960c700 1 -- 172.21.15.175:0/3372141141 >> 172.21.15.175:0/3372141141 conn(0x7f73f4068900 msgr2=0x7f73f4068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:22.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.803+0000 7f73f960c700 1 -- 172.21.15.175:0/3372141141 shutdown_connections 2023-10-10T19:23:22.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.803+0000 7f73f960c700 1 -- 172.21.15.175:0/3372141141 wait complete. 2023-10-10T19:23:22.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.804+0000 7f73f960c700 1 Processor -- start 2023-10-10T19:23:22.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.804+0000 7f73f960c700 1 -- start start 2023-10-10T19:23:22.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f960c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f41490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:22.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f960c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73f40c2cf0 0x7f73f4149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:22.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f960c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 0x7f73f40c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:22.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f960c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f40c9990 con 0x7f73f40c2cf0 2023-10-10T19:23:22.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f960c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4158240 con 0x7f73f40c30c0 2023-10-10T19:23:22.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f960c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4152520 con 0x7f73f40c0270 2023-10-10T19:23:22.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f27fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73f40c2cf0 0x7f73f4149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:22.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f27fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73f40c2cf0 0x7f73f4149630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:58016/0 (socket says 172.21.15.175:58016) 2023-10-10T19:23:22.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f27fc700 1 -- 172.21.15.175:0/140909299 learned_addr learned my addr 172.21.15.175:0/140909299 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:22.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f2ffd700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f41490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:22.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f37fe700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 0x7f73f40c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:22.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f27fc700 1 -- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 msgr2=0x7f73f41490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:22.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f27fc700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f41490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:22.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f27fc700 1 -- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 msgr2=0x7f73f40c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:22.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f27fc700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 0x7f73f40c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:22.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.805+0000 7f73f27fc700 1 -- 172.21.15.175:0/140909299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec009420 con 0x7f73f40c2cf0 2023-10-10T19:23:22.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.806+0000 7f73f27fc700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73f40c2cf0 0x7f73f4149630 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009ed0 tx=0x7f73dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:22.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.806+0000 7f73eaffd700 1 -- 172.21.15.175:0/140909299 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc011900 con 0x7f73f40c2cf0 2023-10-10T19:23:22.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.806+0000 7f73eaffd700 1 -- 172.21.15.175:0/140909299 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc011a60 con 0x7f73f40c2cf0 2023-10-10T19:23:22.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.806+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73dc009420 con 0x7f73f40c2cf0 2023-10-10T19:23:22.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.806+0000 7f73eaffd700 1 -- 172.21.15.175:0/140909299 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc025670 con 0x7f73f40c2cf0 2023-10-10T19:23:22.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.806+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f41685d0 con 0x7f73f40c2cf0 2023-10-10T19:23:22.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.807+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f4067570 con 0x7f73f40c2cf0 2023-10-10T19:23:22.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.807+0000 7f73eaffd700 1 -- 172.21.15.175:0/140909299 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f73dc023070 con 0x7f73f40c2cf0 2023-10-10T19:23:22.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.808+0000 7f73eaffd700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:22.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.808+0000 7f73eaffd700 1 -- 172.21.15.175:0/140909299 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f73dc0860d0 con 0x7f73f40c2cf0 2023-10-10T19:23:22.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.808+0000 7f73f2ffd700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:22.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.808+0000 7f73f2ffd700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f73e4005ca0 tx=0x7f73e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:22.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:22.812+0000 7f73eaffd700 1 -- 172.21.15.175:0/140909299 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73dc052d30 con 0x7f73f40c2cf0 2023-10-10T19:23:23.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.042+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73f41528a0 con 0x7f73f40c2cf0 2023-10-10T19:23:23.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.042+0000 7f73eaffd700 1 -- 172.21.15.175:0/140909299 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7f73dc015090 con 0x7f73f40c2cf0 2023-10-10T19:23:23.044 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:23.044 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 msgr2=0x7f73d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f73e4005ca0 tx=0x7f73e4009040 comp rx=0 tx=0).stop 2023-10-10T19:23:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73f40c2cf0 msgr2=0x7f73f4149630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:23.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73f40c2cf0 0x7f73f4149630 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009ed0 tx=0x7f73dc017000 comp rx=0 tx=0).stop 2023-10-10T19:23:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 shutdown_connections 2023-10-10T19:23:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73f40c30c0 0x7f73f40c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73f40c0270 0x7f73f41490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:23.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 --2- 172.21.15.175:0/140909299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73f40c2cf0 0x7f73f4149630 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.044+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 >> 172.21.15.175:0/140909299 conn(0x7f73f4068900 msgr2=0x7f73f4153710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.045+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 shutdown_connections 2023-10-10T19:23:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:23.045+0000 7f73f960c700 1 -- 172.21.15.175:0/140909299 wait complete. 2023-10-10T19:23:23.048 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:23.058 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:23:25.060 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:25.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.186+0000 7feb0b913700 1 Processor -- start 2023-10-10T19:23:25.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.186+0000 7feb0b913700 1 -- start start 2023-10-10T19:23:25.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb0b913700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feb040c12a0 0x7feb041577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:25.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb0b913700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb0414d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:25.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb0b913700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb04154aa0 con 0x7feb041573d0 2023-10-10T19:23:25.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb0b913700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb040bfdf0 con 0x7feb040c12a0 2023-10-10T19:23:25.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb0b913700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb04063dc0 con 0x7feb040c4f20 2023-10-10T19:23:25.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb08eae700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feb040c12a0 0x7feb041577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:25.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb09eb0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb0414d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:25.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb08eae700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feb040c12a0 0x7feb041577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:54370/0 (socket says 172.21.15.175:54370) 2023-10-10T19:23:25.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb096af700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7feb041573d0 0x7feb040bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:44226/0 (socket says 172.21.15.175:44226) 2023-10-10T19:23:25.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb09eb0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb0414d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:54450/0 (socket says 172.21.15.175:54450) 2023-10-10T19:23:25.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.187+0000 7feb08eae700 1 -- 172.21.15.175:0/3130511999 learned_addr learned my addr 172.21.15.175:0/3130511999 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:25.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb09eb0700 1 -- 172.21.15.175:0/3130511999 >> v2:172.21.15.191:3300/0 conn(0x7feb040c12a0 msgr2=0x7feb041577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:25.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb09eb0700 1 --2- 172.21.15.175:0/3130511999 >> v2:172.21.15.191:3300/0 conn(0x7feb040c12a0 0x7feb041577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb09eb0700 1 -- 172.21.15.175:0/3130511999 >> v1:172.21.15.191:6789/0 conn(0x7feb041573d0 legacy=0x7feb040bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:25.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb09eb0700 1 -- 172.21.15.175:0/3130511999 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb040c5340 con 0x7feb040c4f20 2023-10-10T19:23:25.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb08eae700 1 --2- 172.21.15.175:0/3130511999 >> v2:172.21.15.191:3300/0 conn(0x7feb040c12a0 0x7feb041577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:23:25.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb09eb0700 1 --2- 172.21.15.175:0/3130511999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb0414d450 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feafc0098e0 tx=0x7feafc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c9539cfdfc3acd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:25.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb03fff700 1 -- 172.21.15.175:0/3130511999 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc00b640 con 0x7feb040c4f20 2023-10-10T19:23:25.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb03fff700 1 -- 172.21.15.175:0/3130511999 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc00b7a0 con 0x7feb040c4f20 2023-10-10T19:23:25.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.188+0000 7feb03fff700 1 -- 172.21.15.175:0/3130511999 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc00bb40 con 0x7feb040c4f20 2023-10-10T19:23:25.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.189+0000 7feb0b913700 1 -- 172.21.15.175:0/3130511999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 msgr2=0x7feb0414d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:25.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.189+0000 7feb0b913700 1 --2- 172.21.15.175:0/3130511999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb0414d450 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feafc0098e0 tx=0x7feafc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.189+0000 7feb0b913700 1 -- 172.21.15.175:0/3130511999 shutdown_connections 2023-10-10T19:23:25.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.189+0000 7feb0b913700 1 --2- 172.21.15.175:0/3130511999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb0414d450 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.189+0000 7feb0b913700 1 --2- 172.21.15.175:0/3130511999 >> v2:172.21.15.191:3300/0 conn(0x7feb040c12a0 0x7feb041577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.189+0000 7feb0b913700 1 -- 172.21.15.175:0/3130511999 >> 172.21.15.175:0/3130511999 conn(0x7feb0406dc70 msgr2=0x7feb04063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:25.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.189+0000 7feb0b913700 1 -- 172.21.15.175:0/3130511999 shutdown_connections 2023-10-10T19:23:25.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.189+0000 7feb0b913700 1 -- 172.21.15.175:0/3130511999 wait complete. 2023-10-10T19:23:25.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.190+0000 7feb0b913700 1 Processor -- start 2023-10-10T19:23:25.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.190+0000 7feb0b913700 1 -- start start 2023-10-10T19:23:25.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb0b913700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb040c12a0 0x7feb040cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:25.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb0b913700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb041529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:25.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb0b913700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb041573d0 0x7feb0413fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:25.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb0b913700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb04152650 con 0x7feb041573d0 2023-10-10T19:23:25.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb0b913700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb04153d10 con 0x7feb040c12a0 2023-10-10T19:23:25.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb0b913700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb041523f0 con 0x7feb040c4f20 2023-10-10T19:23:25.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb09eb0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb041573d0 0x7feb0413fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:25.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb09eb0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb041573d0 0x7feb0413fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39438/0 (socket says 172.21.15.175:39438) 2023-10-10T19:23:25.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb09eb0700 1 -- 172.21.15.175:0/1706338417 learned_addr learned my addr 172.21.15.175:0/1706338417 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:25.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb08eae700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb041529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:25.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb096af700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb040c12a0 0x7feb040cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:25.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb09eb0700 1 -- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 msgr2=0x7feb041529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:25.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb09eb0700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb041529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb09eb0700 1 -- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb040c12a0 msgr2=0x7feb040cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:25.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.191+0000 7feb09eb0700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb040c12a0 0x7feb040cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.192+0000 7feb09eb0700 1 -- 172.21.15.175:0/1706338417 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80094f0 con 0x7feb041573d0 2023-10-10T19:23:25.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.192+0000 7feb09eb0700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb041573d0 0x7feb0413fa60 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7feb04063f00 tx=0x7feafc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:25.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.192+0000 7feb017fa700 1 -- 172.21.15.175:0/1706338417 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc011420 con 0x7feb041573d0 2023-10-10T19:23:25.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.192+0000 7feb017fa700 1 -- 172.21.15.175:0/1706338417 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc011580 con 0x7feb041573d0 2023-10-10T19:23:25.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.192+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feafc009420 con 0x7feb041573d0 2023-10-10T19:23:25.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.192+0000 7feb017fa700 1 -- 172.21.15.175:0/1706338417 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc011940 con 0x7feb041573d0 2023-10-10T19:23:25.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.192+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb04151f60 con 0x7feb041573d0 2023-10-10T19:23:25.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.193+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0406f150 con 0x7feb041573d0 2023-10-10T19:23:25.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.193+0000 7feb017fa700 1 -- 172.21.15.175:0/1706338417 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feafc011cb0 con 0x7feb041573d0 2023-10-10T19:23:25.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.194+0000 7feb017fa700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feaf0066f30 0x7feaf00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:25.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.194+0000 7feb017fa700 1 -- 172.21.15.175:0/1706338417 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7feafc086700 con 0x7feb041573d0 2023-10-10T19:23:25.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.194+0000 7feb096af700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feaf0066f30 0x7feaf00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:25.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.194+0000 7feb096af700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feaf0066f30 0x7feaf00693e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7feaf4002020 tx=0x7feaf4001a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:25.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.198+0000 7feb017fa700 1 -- 172.21.15.175:0/1706338417 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feafc0532a0 con 0x7feb041573d0 2023-10-10T19:23:25.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.421+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb04153e50 con 0x7feb041573d0 2023-10-10T19:23:25.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.422+0000 7feb017fa700 1 -- 172.21.15.175:0/1706338417 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7feafc0568c0 con 0x7feb041573d0 2023-10-10T19:23:25.424 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:25.424 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:25.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feaf0066f30 msgr2=0x7feaf00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:25.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feaf0066f30 0x7feaf00693e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7feaf4002020 tx=0x7feaf4001a30 comp rx=0 tx=0).stop 2023-10-10T19:23:25.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb041573d0 msgr2=0x7feb0413fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:25.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb041573d0 0x7feb0413fa60 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7feb04063f00 tx=0x7feafc017000 comp rx=0 tx=0).stop 2023-10-10T19:23:25.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 shutdown_connections 2023-10-10T19:23:25.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7feaf0066f30 0x7feaf00693e0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb041573d0 0x7feb0413fa60 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb040c12a0 0x7feb040cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 --2- 172.21.15.175:0/1706338417 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb040c4f20 0x7feb041529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:25.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 >> 172.21.15.175:0/1706338417 conn(0x7feb0406dc70 msgr2=0x7feb04147c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:25.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 shutdown_connections 2023-10-10T19:23:25.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:25.424+0000 7feb0b913700 1 -- 172.21.15.175:0/1706338417 wait complete. 2023-10-10T19:23:25.429 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:25.438 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:23:25.735 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:23:25.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:23:27.439 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:27.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.565+0000 7f35a0774700 1 Processor -- start 2023-10-10T19:23:27.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.565+0000 7f35a0774700 1 -- start start 2023-10-10T19:23:27.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.566+0000 7f35a0774700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f359814e140 0x7f3598155270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.566+0000 7f35a0774700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f35980c1210 0x7f35980bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.566+0000 7f35a0774700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359806b740 con 0x7f359806edb0 2023-10-10T19:23:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.566+0000 7f35a0774700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35980c0960 con 0x7f359814e140 2023-10-10T19:23:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.566+0000 7f35a0774700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359806ca30 con 0x7f35980c1210 2023-10-10T19:23:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.566+0000 7f359e510700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f359814e140 0x7f3598155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:27.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.566+0000 7f359ed11700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f359806edb0 0x7f3598153df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:54940/0 (socket says 172.21.15.175:54940) 2023-10-10T19:23:27.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.566+0000 7f359ed11700 1 -- 172.21.15.175:0/141851959 learned_addr learned my addr 172.21.15.175:0/141851959 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:27.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359dd0f700 1 --2- 172.21.15.175:0/141851959 >> v2:172.21.15.191:3300/0 conn(0x7f35980c1210 0x7f35980bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:27.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359e510700 1 -- 172.21.15.175:0/141851959 >> v2:172.21.15.191:3300/0 conn(0x7f35980c1210 msgr2=0x7f35980bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:27.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359e510700 1 --2- 172.21.15.175:0/141851959 >> v2:172.21.15.191:3300/0 conn(0x7f35980c1210 0x7f35980bf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359e510700 1 -- 172.21.15.175:0/141851959 >> v1:172.21.15.175:6789/0 conn(0x7f359806edb0 legacy=0x7f3598153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:27.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359e510700 1 -- 172.21.15.175:0/141851959 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359806c5a0 con 0x7f359814e140 2023-10-10T19:23:27.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359d50e700 1 -- 172.21.15.175:0/141851959 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899462377 0 0) 0x7f359806b740 con 0x7f359806edb0 2023-10-10T19:23:27.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359dd0f700 1 --2- 172.21.15.175:0/141851959 >> v2:172.21.15.191:3300/0 conn(0x7f35980c1210 0x7f35980bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:23:27.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359e510700 1 --2- 172.21.15.175:0/141851959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359814e140 0x7f3598155270 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f359806ea00 tx=0x7f35880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c91788372b9e888 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:27.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359d50e700 1 -- 172.21.15.175:0/141851959 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358800b6b0 con 0x7f359814e140 2023-10-10T19:23:27.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359d50e700 1 -- 172.21.15.175:0/141851959 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358800b810 con 0x7f359814e140 2023-10-10T19:23:27.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.567+0000 7f359d50e700 1 -- 172.21.15.175:0/141851959 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358800bbb0 con 0x7f359814e140 2023-10-10T19:23:27.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.568+0000 7f35a0774700 1 -- 172.21.15.175:0/141851959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359814e140 msgr2=0x7f3598155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:27.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.568+0000 7f35a0774700 1 --2- 172.21.15.175:0/141851959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359814e140 0x7f3598155270 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f359806ea00 tx=0x7f35880091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.568+0000 7f35a0774700 1 -- 172.21.15.175:0/141851959 shutdown_connections 2023-10-10T19:23:27.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.568+0000 7f35a0774700 1 --2- 172.21.15.175:0/141851959 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f359814e140 0x7f3598155270 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.568+0000 7f35a0774700 1 --2- 172.21.15.175:0/141851959 >> v2:172.21.15.191:3300/0 conn(0x7f35980c1210 0x7f35980bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.568+0000 7f35a0774700 1 -- 172.21.15.175:0/141851959 >> 172.21.15.175:0/141851959 conn(0x7f3598063cb0 msgr2=0x7f359806d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:27.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.568+0000 7f35a0774700 1 -- 172.21.15.175:0/141851959 shutdown_connections 2023-10-10T19:23:27.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.568+0000 7f35a0774700 1 -- 172.21.15.175:0/141851959 wait complete. 2023-10-10T19:23:27.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.569+0000 7f35a0774700 1 Processor -- start 2023-10-10T19:23:27.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.569+0000 7f35a0774700 1 -- start start 2023-10-10T19:23:27.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.569+0000 7f35a0774700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359806edb0 0x7f3598142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:27.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.569+0000 7f35a0774700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359814e140 0x7f35980c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:27.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.569+0000 7f35a0774700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35980c2ea0 0x7f35980c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:27.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.569+0000 7f35a0774700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35981581f0 con 0x7f35980c2ea0 2023-10-10T19:23:27.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f35a0774700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35980c3ca0 con 0x7f359814e140 2023-10-10T19:23:27.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f35a0774700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35980c3f80 con 0x7f359806edb0 2023-10-10T19:23:27.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35980c2ea0 0x7f35980c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:27.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35980c2ea0 0x7f35980c50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39454/0 (socket says 172.21.15.175:39454) 2023-10-10T19:23:27.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 -- 172.21.15.175:0/824083427 learned_addr learned my addr 172.21.15.175:0/824083427 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:27.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359e510700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359806edb0 0x7f3598142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:27.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359dd0f700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359814e140 0x7f35980c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:27.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 -- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359806edb0 msgr2=0x7f3598142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:27.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359806edb0 0x7f3598142240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 -- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359814e140 msgr2=0x7f35980c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:27.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359814e140 0x7f35980c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 -- 172.21.15.175:0/824083427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588009420 con 0x7f35980c2ea0 2023-10-10T19:23:27.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f359ed11700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35980c2ea0 0x7f35980c50e0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f359000ccc0 tx=0x7f3590001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:27.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f358e7fc700 1 -- 172.21.15.175:0/824083427 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359000a910 con 0x7f35980c2ea0 2023-10-10T19:23:27.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f358e7fc700 1 -- 172.21.15.175:0/824083427 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359000aa70 con 0x7f35980c2ea0 2023-10-10T19:23:27.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35981578c0 con 0x7f35980c2ea0 2023-10-10T19:23:27.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f358e7fc700 1 -- 172.21.15.175:0/824083427 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359000ae10 con 0x7f35980c2ea0 2023-10-10T19:23:27.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.570+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3598157e80 con 0x7f35980c2ea0 2023-10-10T19:23:27.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.571+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3598167aa0 con 0x7f35980c2ea0 2023-10-10T19:23:27.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.572+0000 7f358e7fc700 1 -- 172.21.15.175:0/824083427 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3590007300 con 0x7f35980c2ea0 2023-10-10T19:23:27.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.572+0000 7f358e7fc700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f357c066f60 0x7f357c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:27.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.572+0000 7f358e7fc700 1 -- 172.21.15.175:0/824083427 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f3590084fa0 con 0x7f35980c2ea0 2023-10-10T19:23:27.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.572+0000 7f359e510700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f357c066f60 0x7f357c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:27.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.573+0000 7f359e510700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f357c066f60 0x7f357c069410 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f35880093f0 tx=0x7f3588017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:27.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.576+0000 7f358e7fc700 1 -- 172.21.15.175:0/824083427 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3598167aa0 con 0x7f35980c2ea0 2023-10-10T19:23:27.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.800+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f359806ead0 con 0x7f35980c2ea0 2023-10-10T19:23:27.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.800+0000 7f358e7fc700 1 -- 172.21.15.175:0/824083427 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7f3590051bc0 con 0x7f35980c2ea0 2023-10-10T19:23:27.802 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:27.802 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:27.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f357c066f60 msgr2=0x7f357c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:27.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f357c066f60 0x7f357c069410 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f35880093f0 tx=0x7f3588017000 comp rx=0 tx=0).stop 2023-10-10T19:23:27.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35980c2ea0 msgr2=0x7f35980c50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:27.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35980c2ea0 0x7f35980c50e0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f359000ccc0 tx=0x7f3590001a20 comp rx=0 tx=0).stop 2023-10-10T19:23:27.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 shutdown_connections 2023-10-10T19:23:27.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f357c066f60 0x7f357c069410 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35980c2ea0 0x7f35980c50e0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f359806edb0 0x7f3598142240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 --2- 172.21.15.175:0/824083427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f359814e140 0x7f35980c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:27.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 >> 172.21.15.175:0/824083427 conn(0x7f3598063cb0 msgr2=0x7f3598144190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:27.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 shutdown_connections 2023-10-10T19:23:27.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:27.802+0000 7f35a0774700 1 -- 172.21.15.175:0/824083427 wait complete. 2023-10-10T19:23:27.807 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:27.816 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:23:29.817 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.942+0000 7f13d10e7700 1 Processor -- start 2023-10-10T19:23:29.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.942+0000 7f13d10e7700 1 -- start start 2023-10-10T19:23:29.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13d10e7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f13cc14fd90 0x7f13cc159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:29.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13d10e7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc08a100 0x7f13cc0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:29.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13d10e7700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13cc067fa0 con 0x7f13cc0cd040 2023-10-10T19:23:29.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13d10e7700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13cc0c5600 con 0x7f13cc14fd90 2023-10-10T19:23:29.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13d10e7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc156b70 con 0x7f13cc08a100 2023-10-10T19:23:29.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f13cc14fd90 0x7f13cc159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f13cc14fd90 0x7f13cc159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39460/0 (socket says 172.21.15.175:39460) 2023-10-10T19:23:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cbfff700 1 -- 172.21.15.175:0/3640815400 learned_addr learned my addr 172.21.15.175:0/3640815400 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cbfff700 1 -- 172.21.15.175:0/3640815400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc08a100 msgr2=0x7f13cc0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cb7fe700 1 --2- 172.21.15.175:0/3640815400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc08a100 0x7f13cc0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cbfff700 1 --2- 172.21.15.175:0/3640815400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc08a100 0x7f13cc0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:29.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cbfff700 1 -- 172.21.15.175:0/3640815400 >> v1:172.21.15.191:6789/0 conn(0x7f13cc0cd040 legacy=0x7f13cc0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:29.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cbfff700 1 -- 172.21.15.175:0/3640815400 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc150160 con 0x7f13cc14fd90 2023-10-10T19:23:29.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.943+0000 7f13cb7fe700 1 --2- 172.21.15.175:0/3640815400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc08a100 0x7f13cc0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:23:29.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13cbfff700 1 --2- 172.21.15.175:0/3640815400 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc159c30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0099b0 tx=0x7f13bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749be85d7cfac37 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13caffd700 1 -- 172.21.15.175:0/3640815400 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc00b820 con 0x7f13cc14fd90 2023-10-10T19:23:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13caffd700 1 -- 172.21.15.175:0/3640815400 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13bc00b980 con 0x7f13cc14fd90 2023-10-10T19:23:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13caffd700 1 -- 172.21.15.175:0/3640815400 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc00bd20 con 0x7f13cc14fd90 2023-10-10T19:23:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13d10e7700 1 -- 172.21.15.175:0/3640815400 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 msgr2=0x7f13cc159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3640815400 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc159c30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0099b0 tx=0x7f13bc009250 comp rx=0 tx=0).stop 2023-10-10T19:23:29.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13d10e7700 1 -- 172.21.15.175:0/3640815400 shutdown_connections 2023-10-10T19:23:29.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3640815400 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc159c30 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:29.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3640815400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc08a100 0x7f13cc0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:29.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13d10e7700 1 -- 172.21.15.175:0/3640815400 >> 172.21.15.175:0/3640815400 conn(0x7f13cc063d00 msgr2=0x7f13cc0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:29.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13d10e7700 1 -- 172.21.15.175:0/3640815400 shutdown_connections 2023-10-10T19:23:29.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.944+0000 7f13d10e7700 1 -- 172.21.15.175:0/3640815400 wait complete. 2023-10-10T19:23:29.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.945+0000 7f13d10e7700 1 Processor -- start 2023-10-10T19:23:29.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d10e7700 1 -- start start 2023-10-10T19:23:29.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d10e7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13cc08a100 0x7f13cc169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:29.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d10e7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc0cd040 0x7f13cc165c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:29.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d10e7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc0a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:29.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d10e7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc158930 con 0x7f13cc14fd90 2023-10-10T19:23:29.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d10e7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc166180 con 0x7f13cc08a100 2023-10-10T19:23:29.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d10e7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc169810 con 0x7f13cc0cd040 2023-10-10T19:23:29.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d08e6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc0a7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:29.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.946+0000 7f13d08e6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc0a7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39472/0 (socket says 172.21.15.175:39472) 2023-10-10T19:23:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d08e6700 1 -- 172.21.15.175:0/3704929492 learned_addr learned my addr 172.21.15.175:0/3704929492 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13cb7fe700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc0cd040 0x7f13cc165c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13cbfff700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13cc08a100 0x7f13cc169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d08e6700 1 -- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc0cd040 msgr2=0x7f13cc165c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d08e6700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc0cd040 0x7f13cc165c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d08e6700 1 -- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13cc08a100 msgr2=0x7f13cc169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d08e6700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13cc08a100 0x7f13cc169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d08e6700 1 -- 172.21.15.175:0/3704929492 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0094f0 con 0x7f13cc14fd90 2023-10-10T19:23:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d08e6700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc0a7c10 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f13c400c930 tx=0x7f13c40026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13b3fff700 1 -- 172.21.15.175:0/3704929492 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4012630 con 0x7f13cc14fd90 2023-10-10T19:23:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13b3fff700 1 -- 172.21.15.175:0/3704929492 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c40127d0 con 0x7f13cc14fd90 2023-10-10T19:23:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13cc0a8150 con 0x7f13cc14fd90 2023-10-10T19:23:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13b3fff700 1 -- 172.21.15.175:0/3704929492 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4012b90 con 0x7f13cc14fd90 2023-10-10T19:23:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.947+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13cc158100 con 0x7f13cc14fd90 2023-10-10T19:23:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.948+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13cc169980 con 0x7f13cc14fd90 2023-10-10T19:23:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.949+0000 7f13b3fff700 1 -- 172.21.15.175:0/3704929492 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f13c401b400 con 0x7f13cc14fd90 2023-10-10T19:23:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.949+0000 7f13b3fff700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f13b8066f60 0x7f13b8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.949+0000 7f13b3fff700 1 -- 172.21.15.175:0/3704929492 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f13c4084e40 con 0x7f13cc14fd90 2023-10-10T19:23:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.949+0000 7f13cbfff700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f13b8066f60 0x7f13b8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.950+0000 7f13cbfff700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f13b8066f60 0x7f13b8069410 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0099b0 tx=0x7f13bc016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:29.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:29.953+0000 7f13b3fff700 1 -- 172.21.15.175:0/3704929492 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13c4055470 con 0x7f13cc14fd90 2023-10-10T19:23:30.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.176+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13cc158420 con 0x7f13cc14fd90 2023-10-10T19:23:30.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.177+0000 7f13b3fff700 1 -- 172.21.15.175:0/3704929492 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7f13c4055000 con 0x7f13cc14fd90 2023-10-10T19:23:30.178 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:30.179 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f13b8066f60 msgr2=0x7f13b8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f13b8066f60 0x7f13b8069410 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0099b0 tx=0x7f13bc016000 comp rx=0 tx=0).stop 2023-10-10T19:23:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 msgr2=0x7f13cc0a7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc0a7c10 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f13c400c930 tx=0x7f13c40026e0 comp rx=0 tx=0).stop 2023-10-10T19:23:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 shutdown_connections 2023-10-10T19:23:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f13b8066f60 0x7f13b8069410 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13cc14fd90 0x7f13cc0a7c10 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13cc08a100 0x7f13cc169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:30.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 --2- 172.21.15.175:0/3704929492 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13cc0cd040 0x7f13cc165c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:30.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 >> 172.21.15.175:0/3704929492 conn(0x7f13cc063d00 msgr2=0x7f13cc14d840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:30.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 shutdown_connections 2023-10-10T19:23:30.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:30.179+0000 7f13d10e7700 1 -- 172.21.15.175:0/3704929492 wait complete. 2023-10-10T19:23:30.184 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:30.193 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:23:32.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:32.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.320+0000 7ff320894700 1 Processor -- start 2023-10-10T19:23:32.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.320+0000 7ff320894700 1 -- start start 2023-10-10T19:23:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.321+0000 7ff320894700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff31c0b1fc0 0x7ff31c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.321+0000 7ff320894700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff31c0aff60 0x7ff31c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.321+0000 7ff320894700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c1574b0 con 0x7ff31c0b2390 2023-10-10T19:23:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.321+0000 7ff320894700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31c150770 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.321+0000 7ff320894700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31c0afad0 con 0x7ff31c0aff60 2023-10-10T19:23:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31a59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff31c0b1fc0 0x7ff31c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31a59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff31c0b1fc0 0x7ff31c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39474/0 (socket says 172.21.15.175:39474) 2023-10-10T19:23:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31a59c700 1 -- 172.21.15.175:0/1067282523 learned_addr learned my addr 172.21.15.175:0/1067282523 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31ad9d700 1 --2- 172.21.15.175:0/1067282523 >> v2:172.21.15.191:3300/0 conn(0x7ff31c0aff60 0x7ff31c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31a59c700 1 -- 172.21.15.175:0/1067282523 >> v2:172.21.15.191:3300/0 conn(0x7ff31c0aff60 msgr2=0x7ff31c0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31a59c700 1 --2- 172.21.15.175:0/1067282523 >> v2:172.21.15.191:3300/0 conn(0x7ff31c0aff60 0x7ff31c0b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31a59c700 1 -- 172.21.15.175:0/1067282523 >> v1:172.21.15.191:6789/0 conn(0x7ff31c0b2390 legacy=0x7ff31c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31a59c700 1 -- 172.21.15.175:0/1067282523 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c0af4c0 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31a59c700 1 --2- 172.21.15.175:0/1067282523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c155fb0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0641f0 tx=0x7ff30c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c5db4a9e6aea42 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31959a700 1 -- 172.21.15.175:0/1067282523 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c00e070 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31959a700 1 -- 172.21.15.175:0/1067282523 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c00bab0 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.322+0000 7ff31959a700 1 -- 172.21.15.175:0/1067282523 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c0092e0 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.323+0000 7ff320894700 1 -- 172.21.15.175:0/1067282523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 msgr2=0x7ff31c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.323+0000 7ff320894700 1 --2- 172.21.15.175:0/1067282523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c155fb0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0641f0 tx=0x7ff30c009840 comp rx=0 tx=0).stop 2023-10-10T19:23:32.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.323+0000 7ff320894700 1 -- 172.21.15.175:0/1067282523 shutdown_connections 2023-10-10T19:23:32.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.323+0000 7ff320894700 1 --2- 172.21.15.175:0/1067282523 >> v2:172.21.15.191:3300/0 conn(0x7ff31c0aff60 0x7ff31c0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.323+0000 7ff320894700 1 --2- 172.21.15.175:0/1067282523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c155fb0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.323+0000 7ff320894700 1 -- 172.21.15.175:0/1067282523 >> 172.21.15.175:0/1067282523 conn(0x7ff31c063230 msgr2=0x7ff31c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:32.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.323+0000 7ff320894700 1 -- 172.21.15.175:0/1067282523 shutdown_connections 2023-10-10T19:23:32.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.323+0000 7ff320894700 1 -- 172.21.15.175:0/1067282523 wait complete. 2023-10-10T19:23:32.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.324+0000 7ff320894700 1 Processor -- start 2023-10-10T19:23:32.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.324+0000 7ff320894700 1 -- start start 2023-10-10T19:23:32.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff320894700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff31c0aff60 0x7ff31c0cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:32.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff320894700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c0c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff320894700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff31c0b2390 0x7ff31c0c4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff320894700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c15c4d0 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff320894700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c166b60 con 0x7ff31c0b2390 2023-10-10T19:23:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff320894700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c166630 con 0x7ff31c0aff60 2023-10-10T19:23:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c0c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:32.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c0c3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:39476/0 (socket says 172.21.15.175:39476) 2023-10-10T19:23:32.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 -- 172.21.15.175:0/281129296 learned_addr learned my addr 172.21.15.175:0/281129296 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:32.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff31ad9d700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff31c0b2390 0x7ff31c0c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:32.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff31a59c700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff31c0aff60 0x7ff31c0cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 -- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff31c0aff60 msgr2=0x7ff31c0cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff31c0aff60 0x7ff31c0cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 -- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff31c0b2390 msgr2=0x7ff31c0c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff31c0b2390 0x7ff31c0c4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 -- 172.21.15.175:0/281129296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c009ae0 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff319d9b700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c0c3c50 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ff30400c4e0 tx=0x7ff30400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff3127fc700 1 -- 172.21.15.175:0/281129296 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304011590 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff3127fc700 1 -- 172.21.15.175:0/281129296 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff304011710 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c15a330 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.325+0000 7ff3127fc700 1 -- 172.21.15.175:0/281129296 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304011ad0 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.326+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c0c6820 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.326+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c067570 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.327+0000 7ff3127fc700 1 -- 172.21.15.175:0/281129296 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff30401b3b0 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.327+0000 7ff3127fc700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff2fc066f30 0x7ff2fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:32.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.328+0000 7ff3127fc700 1 -- 172.21.15.175:0/281129296 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7ff304084140 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.328+0000 7ff31a59c700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff2fc066f30 0x7ff2fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:32.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.328+0000 7ff31a59c700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff2fc066f30 0x7ff2fc0693e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009880 tx=0x7ff30c018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:32.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.332+0000 7ff3127fc700 1 -- 172.21.15.175:0/281129296 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff304050ce0 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.558+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff31c068470 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.559+0000 7ff3127fc700 1 -- 172.21.15.175:0/281129296 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7ff304017020 con 0x7ff31c0b1fc0 2023-10-10T19:23:32.561 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:32.561 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:32.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff2fc066f30 msgr2=0x7ff2fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:32.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff2fc066f30 0x7ff2fc0693e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009880 tx=0x7ff30c018000 comp rx=0 tx=0).stop 2023-10-10T19:23:32.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 msgr2=0x7ff31c0c3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:32.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c0c3c50 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ff30400c4e0 tx=0x7ff30400c8a0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 shutdown_connections 2023-10-10T19:23:32.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff2fc066f30 0x7ff2fc0693e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff31c0b2390 0x7ff31c0c4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff31c0aff60 0x7ff31c0cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 --2- 172.21.15.175:0/281129296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff31c0b1fc0 0x7ff31c0c3c50 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:32.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 >> 172.21.15.175:0/281129296 conn(0x7ff31c063230 msgr2=0x7ff31c14fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:32.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 shutdown_connections 2023-10-10T19:23:32.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:32.561+0000 7ff320894700 1 -- 172.21.15.175:0/281129296 wait complete. 2023-10-10T19:23:32.567 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:32.575 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:23:34.576 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:34.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.702+0000 7ff817173700 1 Processor -- start 2023-10-10T19:23:34.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.702+0000 7ff817173700 1 -- start start 2023-10-10T19:23:34.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.703+0000 7ff817173700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff81006e8f0 0x7ff8101569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:34.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.703+0000 7ff817173700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff8100c9d20 0x7ff81006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:34.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.703+0000 7ff817173700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81006cca0 con 0x7ff81006e520 2023-10-10T19:23:34.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.703+0000 7ff817173700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8100c33c0 con 0x7ff81006e8f0 2023-10-10T19:23:34.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.703+0000 7ff817173700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8100c07f0 con 0x7ff8100c9d20 2023-10-10T19:23:34.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.703+0000 7ff815970700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff81006e8f0 0x7ff8101569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:34.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff816171700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff81006e520 0x7ff8100c9610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50300/0 (socket says 172.21.15.175:50300) 2023-10-10T19:23:34.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff816171700 1 -- 172.21.15.175:0/11105276 learned_addr learned my addr 172.21.15.175:0/11105276 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:34.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff815970700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff81006e8f0 0x7ff8101569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38438/0 (socket says 172.21.15.175:38438) 2023-10-10T19:23:34.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff816972700 1 --2- 172.21.15.175:0/11105276 >> v2:172.21.15.191:3300/0 conn(0x7ff8100c9d20 0x7ff81006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:34.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff815970700 1 -- 172.21.15.175:0/11105276 >> v2:172.21.15.191:3300/0 conn(0x7ff8100c9d20 msgr2=0x7ff81006c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:34.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff815970700 1 --2- 172.21.15.175:0/11105276 >> v2:172.21.15.191:3300/0 conn(0x7ff8100c9d20 0x7ff81006c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff815970700 1 -- 172.21.15.175:0/11105276 >> v1:172.21.15.175:6789/0 conn(0x7ff81006e520 legacy=0x7ff8100c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:34.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff815970700 1 -- 172.21.15.175:0/11105276 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff810152b70 con 0x7ff81006e8f0 2023-10-10T19:23:34.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff81516f700 1 -- 172.21.15.175:0/11105276 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908286908 0 0) 0x7ff81006cca0 con 0x7ff81006e520 2023-10-10T19:23:34.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff815970700 1 --2- 172.21.15.175:0/11105276 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff81006e8f0 0x7ff8101569f0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff8100c15a0 tx=0x7ff8000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3b6626f8733490 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:34.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff81516f700 1 -- 172.21.15.175:0/11105276 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800009e80 con 0x7ff81006e8f0 2023-10-10T19:23:34.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff81516f700 1 -- 172.21.15.175:0/11105276 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80000b8b0 con 0x7ff81006e8f0 2023-10-10T19:23:34.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.704+0000 7ff81516f700 1 -- 172.21.15.175:0/11105276 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80000bc30 con 0x7ff81006e8f0 2023-10-10T19:23:34.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.705+0000 7ff817173700 1 -- 172.21.15.175:0/11105276 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff81006e8f0 msgr2=0x7ff8101569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:34.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.705+0000 7ff817173700 1 --2- 172.21.15.175:0/11105276 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff81006e8f0 0x7ff8101569f0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff8100c15a0 tx=0x7ff8000091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.705+0000 7ff817173700 1 -- 172.21.15.175:0/11105276 shutdown_connections 2023-10-10T19:23:34.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.705+0000 7ff817173700 1 --2- 172.21.15.175:0/11105276 >> v2:172.21.15.191:3300/0 conn(0x7ff8100c9d20 0x7ff81006c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.705+0000 7ff817173700 1 --2- 172.21.15.175:0/11105276 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff81006e8f0 0x7ff8101569f0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.705+0000 7ff817173700 1 -- 172.21.15.175:0/11105276 >> 172.21.15.175:0/11105276 conn(0x7ff810063230 msgr2=0x7ff81006f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:34.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.705+0000 7ff817173700 1 -- 172.21.15.175:0/11105276 shutdown_connections 2023-10-10T19:23:34.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.705+0000 7ff817173700 1 -- 172.21.15.175:0/11105276 wait complete. 2023-10-10T19:23:34.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.706+0000 7ff817173700 1 Processor -- start 2023-10-10T19:23:34.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.706+0000 7ff817173700 1 -- start start 2023-10-10T19:23:34.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff817173700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff81006e520 0x7ff810167160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:34.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff817173700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff81006e8f0 0x7ff81014f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:34.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff817173700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8100c9d20 0x7ff81014b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:34.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff817173700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff810165ce0 con 0x7ff8100c9d20 2023-10-10T19:23:34.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff817173700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff810166540 con 0x7ff81006e8f0 2023-10-10T19:23:34.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff817173700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81014fc50 con 0x7ff81006e520 2023-10-10T19:23:34.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8100c9d20 0x7ff81014b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:34.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8100c9d20 0x7ff81014b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38452/0 (socket says 172.21.15.175:38452) 2023-10-10T19:23:34.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 -- 172.21.15.175:0/2397478321 learned_addr learned my addr 172.21.15.175:0/2397478321 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:34.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff815970700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff81006e8f0 0x7ff81014f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:34.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816171700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff81006e520 0x7ff810167160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:34.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 -- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff81006e520 msgr2=0x7ff810167160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:34.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff81006e520 0x7ff810167160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 -- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff81006e8f0 msgr2=0x7ff81014f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:34.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff81006e8f0 0x7ff81014f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 -- 172.21.15.175:0/2397478321 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800009420 con 0x7ff8100c9d20 2023-10-10T19:23:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff816972700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8100c9d20 0x7ff81014b7e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7ff808007f30 tx=0x7ff8080098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff8067fc700 1 -- 172.21.15.175:0/2397478321 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80800ecf0 con 0x7ff8100c9d20 2023-10-10T19:23:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff8067fc700 1 -- 172.21.15.175:0/2397478321 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80800ee50 con 0x7ff8100c9d20 2023-10-10T19:23:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff8067fc700 1 -- 172.21.15.175:0/2397478321 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808014670 con 0x7ff8100c9d20 2023-10-10T19:23:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81014feb0 con 0x7ff8100c9d20 2023-10-10T19:23:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.707+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8101661d0 con 0x7ff8100c9d20 2023-10-10T19:23:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.708+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff810156d40 con 0x7ff8100c9d20 2023-10-10T19:23:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.709+0000 7ff8067fc700 1 -- 172.21.15.175:0/2397478321 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff8080149e0 con 0x7ff8100c9d20 2023-10-10T19:23:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.709+0000 7ff8067fc700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7f4066f30 0x7ff7f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.709+0000 7ff8067fc700 1 -- 172.21.15.175:0/2397478321 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7ff8080853d0 con 0x7ff8100c9d20 2023-10-10T19:23:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.709+0000 7ff816171700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7f4066f30 0x7ff7f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.710+0000 7ff816171700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7f4066f30 0x7ff7f40693e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff80c001750 tx=0x7ff80c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.714+0000 7ff8067fc700 1 -- 172.21.15.175:0/2397478321 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff808051f70 con 0x7ff8100c9d20 2023-10-10T19:23:34.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.939+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff810067570 con 0x7ff8100c9d20 2023-10-10T19:23:34.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.939+0000 7ff8067fc700 1 -- 172.21.15.175:0/2397478321 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7ff808055590 con 0x7ff8100c9d20 2023-10-10T19:23:34.941 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:34.942 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:34.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7f4066f30 msgr2=0x7ff7f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:34.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7f4066f30 0x7ff7f40693e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff80c001750 tx=0x7ff80c009040 comp rx=0 tx=0).stop 2023-10-10T19:23:34.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8100c9d20 msgr2=0x7ff81014b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:34.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8100c9d20 0x7ff81014b7e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7ff808007f30 tx=0x7ff8080098e0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 shutdown_connections 2023-10-10T19:23:34.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff7f4066f30 0x7ff7f40693e0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8100c9d20 0x7ff81014b7e0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff81006e520 0x7ff810167160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 --2- 172.21.15.175:0/2397478321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff81006e8f0 0x7ff81014f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:34.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 >> 172.21.15.175:0/2397478321 conn(0x7ff810063230 msgr2=0x7ff8100c16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:34.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 shutdown_connections 2023-10-10T19:23:34.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:34.942+0000 7ff817173700 1 -- 172.21.15.175:0/2397478321 wait complete. 2023-10-10T19:23:34.946 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:34.956 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:23:36.957 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:37.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.084+0000 7ffb6332a700 1 Processor -- start 2023-10-10T19:23:37.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.084+0000 7ffb6332a700 1 -- start start 2023-10-10T19:23:37.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb6332a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffb5c0b1fc0 0x7ffb5c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:37.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb6332a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:37.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb6332a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5c1574b0 con 0x7ffb5c068bd0 2023-10-10T19:23:37.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb6332a700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5c150770 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb6332a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c0afad0 con 0x7ffb5c0b2390 2023-10-10T19:23:37.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb610c6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffb5c0b1fc0 0x7ffb5c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:37.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb618c7700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ffb5c068bd0 0x7ffb5c0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50302/0 (socket says 172.21.15.175:50302) 2023-10-10T19:23:37.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb618c7700 1 -- 172.21.15.175:0/3214296995 learned_addr learned my addr 172.21.15.175:0/3214296995 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:37.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb610c6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffb5c0b1fc0 0x7ffb5c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38466/0 (socket says 172.21.15.175:38466) 2023-10-10T19:23:37.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb608c5700 1 --2- 172.21.15.175:0/3214296995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:37.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb610c6700 1 -- 172.21.15.175:0/3214296995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 msgr2=0x7ffb5c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:37.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb610c6700 1 --2- 172.21.15.175:0/3214296995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb610c6700 1 -- 172.21.15.175:0/3214296995 >> v1:172.21.15.175:6789/0 conn(0x7ffb5c068bd0 legacy=0x7ffb5c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:37.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb610c6700 1 -- 172.21.15.175:0/3214296995 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5c0af4c0 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb53fff700 1 -- 172.21.15.175:0/3214296995 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950290222 0 0) 0x7ffb5c1574b0 con 0x7ffb5c068bd0 2023-10-10T19:23:37.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb608c5700 1 --2- 172.21.15.175:0/3214296995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:23:37.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb610c6700 1 --2- 172.21.15.175:0/3214296995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c155fb0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7ffb540098e0 tx=0x7ffb540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32879de40137ee7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:37.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.085+0000 7ffb53fff700 1 -- 172.21.15.175:0/3214296995 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54009e80 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb53fff700 1 -- 172.21.15.175:0/3214296995 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5400b8f0 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb53fff700 1 -- 172.21.15.175:0/3214296995 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5400bc70 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb6332a700 1 -- 172.21.15.175:0/3214296995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 msgr2=0x7ffb5c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:37.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb6332a700 1 --2- 172.21.15.175:0/3214296995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c155fb0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7ffb540098e0 tx=0x7ffb540091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb6332a700 1 -- 172.21.15.175:0/3214296995 shutdown_connections 2023-10-10T19:23:37.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb6332a700 1 --2- 172.21.15.175:0/3214296995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c155fb0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb6332a700 1 --2- 172.21.15.175:0/3214296995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb6332a700 1 -- 172.21.15.175:0/3214296995 >> 172.21.15.175:0/3214296995 conn(0x7ffb5c063230 msgr2=0x7ffb5c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:37.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb6332a700 1 -- 172.21.15.175:0/3214296995 shutdown_connections 2023-10-10T19:23:37.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.086+0000 7ffb6332a700 1 -- 172.21.15.175:0/3214296995 wait complete. 2023-10-10T19:23:37.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.087+0000 7ffb6332a700 1 Processor -- start 2023-10-10T19:23:37.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.087+0000 7ffb6332a700 1 -- start start 2023-10-10T19:23:37.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb6332a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb5c068bd0 0x7ffb5c0cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:37.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb6332a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c169140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:37.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb6332a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c0cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:37.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb6332a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c0c02c0 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb6332a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c0c0840 con 0x7ffb5c068bd0 2023-10-10T19:23:37.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb6332a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c0c18f0 con 0x7ffb5c0b2390 2023-10-10T19:23:37.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb608c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:37.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb608c5700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c169140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38476/0 (socket says 172.21.15.175:38476) 2023-10-10T19:23:37.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb608c5700 1 -- 172.21.15.175:0/533144713 learned_addr learned my addr 172.21.15.175:0/533144713 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:37.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb610c6700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb5c068bd0 0x7ffb5c0cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:37.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb618c7700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c0cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:37.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb608c5700 1 -- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 msgr2=0x7ffb5c0cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:37.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb608c5700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c0cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb608c5700 1 -- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb5c068bd0 msgr2=0x7ffb5c0cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:37.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb608c5700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb5c068bd0 0x7ffb5c0cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.088+0000 7ffb608c5700 1 -- 172.21.15.175:0/533144713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb54009420 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.089+0000 7ffb608c5700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c169140 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00c880 tx=0x7ffb4c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:37.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.089+0000 7ffb517fa700 1 -- 172.21.15.175:0/533144713 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c011600 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.089+0000 7ffb517fa700 1 -- 172.21.15.175:0/533144713 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c011760 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.089+0000 7ffb517fa700 1 -- 172.21.15.175:0/533144713 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c011b00 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.089+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c0cc110 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.089+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c0c04d0 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.090+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c0c09d0 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.090+0000 7ffb517fa700 1 -- 172.21.15.175:0/533144713 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffb4c018350 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.090+0000 7ffb517fa700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb40066f30 0x7ffb400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:37.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.090+0000 7ffb517fa700 1 -- 172.21.15.175:0/533144713 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 7455+0+0 (secure 0 0 0) 0x7ffb4c084a50 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.091+0000 7ffb610c6700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb40066f30 0x7ffb400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:37.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.091+0000 7ffb610c6700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb40066f30 0x7ffb400693e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ffb540091e0 tx=0x7ffb54017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:37.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.095+0000 7ffb517fa700 1 -- 172.21.15.175:0/533144713 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffb4c055080 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.320+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb5c067570 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.320+0000 7ffb517fa700 1 -- 172.21.15.175:0/533144713 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+6185 (secure 0 0 0) 0x7ffb4c054c10 con 0x7ffb5c0b1fc0 2023-10-10T19:23:37.322 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:37.323 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8049,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":284}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:22:54.075338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":7104,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7104":{"gid":7104,"name":"e","rank":2,"incarnation":221,"state":"up:active","state_seq":43,"addr":"172.21.15.175:6837/3849771254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":3849771254},{"type":"v1","addr":"172.21.15.175:6837","nonce":3849771254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:37.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.322+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb40066f30 msgr2=0x7ffb400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:37.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.322+0000 7ffb6332a700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb40066f30 0x7ffb400693e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ffb540091e0 tx=0x7ffb54017000 comp rx=0 tx=0).stop 2023-10-10T19:23:37.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.322+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 msgr2=0x7ffb5c169140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:37.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.322+0000 7ffb6332a700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c169140 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00c880 tx=0x7ffb4c008040 comp rx=0 tx=0).stop 2023-10-10T19:23:37.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.322+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 shutdown_connections 2023-10-10T19:23:37.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.323+0000 7ffb6332a700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ffb40066f30 0x7ffb400693e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.323+0000 7ffb6332a700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb5c0b2390 0x7ffb5c0cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.323+0000 7ffb6332a700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb5c068bd0 0x7ffb5c0cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.323+0000 7ffb6332a700 1 --2- 172.21.15.175:0/533144713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb5c0b1fc0 0x7ffb5c169140 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:37.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.323+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 >> 172.21.15.175:0/533144713 conn(0x7ffb5c063230 msgr2=0x7ffb5c1508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:37.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.323+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 shutdown_connections 2023-10-10T19:23:37.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:37.323+0000 7ffb6332a700 1 -- 172.21.15.175:0/533144713 wait complete. 2023-10-10T19:23:37.327 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 284 2023-10-10T19:23:37.336 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-10T19:23:39.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:39.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.464+0000 7f3e55c98700 1 Processor -- start 2023-10-10T19:23:39.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.464+0000 7f3e55c98700 1 -- start start 2023-10-10T19:23:39.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.464+0000 7f3e55c98700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3e500b2390 0x7f3e50155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.464+0000 7f3e55c98700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e501574b0 con 0x7f3e500aff60 2023-10-10T19:23:39.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.464+0000 7f3e55c98700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e500afad0 con 0x7f3e500b1fc0 2023-10-10T19:23:39.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.464+0000 7f3e55c98700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e50150770 con 0x7f3e500b2390 2023-10-10T19:23:39.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.464+0000 7f3e4effd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3e500b2390 0x7f3e50155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f3e500aff60 0x7f3e50159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50314/0 (socket says 172.21.15.175:50314) 2023-10-10T19:23:39.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4ffff700 1 -- 172.21.15.175:0/105150712 learned_addr learned my addr 172.21.15.175:0/105150712 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:39.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4effd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3e500b2390 0x7f3e50155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38486/0 (socket says 172.21.15.175:38486) 2023-10-10T19:23:39.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4effd700 1 -- 172.21.15.175:0/105150712 >> v1:172.21.15.191:6789/0 conn(0x7f3e500b1fc0 legacy=0x7f3e50068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4effd700 1 -- 172.21.15.175:0/105150712 >> v1:172.21.15.175:6789/0 conn(0x7f3e500aff60 legacy=0x7f3e50159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4effd700 1 -- 172.21.15.175:0/105150712 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500af4c0 con 0x7f3e500b2390 2023-10-10T19:23:39.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4e7fc700 1 -- 172.21.15.175:0/105150712 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918136850 0 0) 0x7f3e501574b0 con 0x7f3e500aff60 2023-10-10T19:23:39.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4effd700 1 --2- 172.21.15.175:0/105150712 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e50155fb0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f3e440098e0 tx=0x7f3e440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1697bd1e2dd55f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:39.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4e7fc700 1 -- 172.21.15.175:0/105150712 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44009e80 con 0x7f3e500b2390 2023-10-10T19:23:39.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4e7fc700 1 -- 172.21.15.175:0/105150712 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4400b8f0 con 0x7f3e500b2390 2023-10-10T19:23:39.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.465+0000 7f3e4e7fc700 1 -- 172.21.15.175:0/105150712 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4400bc70 con 0x7f3e500b2390 2023-10-10T19:23:39.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.466+0000 7f3e55c98700 1 -- 172.21.15.175:0/105150712 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 msgr2=0x7f3e50155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.466+0000 7f3e55c98700 1 --2- 172.21.15.175:0/105150712 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e50155fb0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f3e440098e0 tx=0x7f3e440091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.466+0000 7f3e55c98700 1 -- 172.21.15.175:0/105150712 shutdown_connections 2023-10-10T19:23:39.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.466+0000 7f3e55c98700 1 --2- 172.21.15.175:0/105150712 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e50155fb0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.466+0000 7f3e55c98700 1 -- 172.21.15.175:0/105150712 >> 172.21.15.175:0/105150712 conn(0x7f3e50063230 msgr2=0x7f3e50063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:39.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.466+0000 7f3e55c98700 1 -- 172.21.15.175:0/105150712 shutdown_connections 2023-10-10T19:23:39.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.466+0000 7f3e55c98700 1 -- 172.21.15.175:0/105150712 wait complete. 2023-10-10T19:23:39.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.467+0000 7f3e55c98700 1 Processor -- start 2023-10-10T19:23:39.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.467+0000 7f3e55c98700 1 -- start start 2023-10-10T19:23:39.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e55c98700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e500aff60 0x7f3e50169750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e55c98700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e500b1fc0 0x7f3e50169c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e55c98700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e500c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e55c98700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500cc110 con 0x7f3e500b2390 2023-10-10T19:23:39.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e55c98700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5016a2b0 con 0x7f3e500b1fc0 2023-10-10T19:23:39.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e55c98700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e50159a60 con 0x7f3e500aff60 2023-10-10T19:23:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e500c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e500c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38490/0 (socket says 172.21.15.175:38490) 2023-10-10T19:23:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 -- 172.21.15.175:0/2332522037 learned_addr learned my addr 172.21.15.175:0/2332522037 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4effd700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e500b1fc0 0x7f3e50169c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4f7fe700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e500aff60 0x7f3e50169750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 -- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e500aff60 msgr2=0x7f3e50169750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e500aff60 0x7f3e50169750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 -- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e500b1fc0 msgr2=0x7f3e50169c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e500b1fc0 0x7f3e50169c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 -- 172.21.15.175:0/2332522037 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e44009420 con 0x7f3e500b2390 2023-10-10T19:23:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.468+0000 7f3e4ffff700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e500c5050 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800c7e0 tx=0x7f3e48007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.469+0000 7f3e377fe700 1 -- 172.21.15.175:0/2332522037 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48005c00 con 0x7f3e500b2390 2023-10-10T19:23:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.469+0000 7f3e377fe700 1 -- 172.21.15.175:0/2332522037 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48005d80 con 0x7f3e500b2390 2023-10-10T19:23:39.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.469+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e50159d40 con 0x7f3e500b2390 2023-10-10T19:23:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.469+0000 7f3e377fe700 1 -- 172.21.15.175:0/2332522037 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48007320 con 0x7f3e500b2390 2023-10-10T19:23:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.469+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e500c4b30 con 0x7f3e500b2390 2023-10-10T19:23:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.469+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e500c4d20 con 0x7f3e500b2390 2023-10-10T19:23:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.470+0000 7f3e377fe700 1 -- 172.21.15.175:0/2332522037 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3e4801b3b0 con 0x7f3e500b2390 2023-10-10T19:23:39.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.470+0000 7f3e377fe700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e3c066f60 0x7f3e3c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.470+0000 7f3e377fe700 1 -- 172.21.15.175:0/2332522037 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f3e48084070 con 0x7f3e500b2390 2023-10-10T19:23:39.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.470+0000 7f3e4f7fe700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e3c066f60 0x7f3e3c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.471+0000 7f3e4f7fe700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e3c066f60 0x7f3e3c069410 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3e40007fd0 tx=0x7f3e400016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:39.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.475+0000 7f3e377fe700 1 -- 172.21.15.175:0/2332522037 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3e48050bb0 con 0x7f3e500b2390 2023-10-10T19:23:39.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.701+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e50067570 con 0x7f3e500b2390 2023-10-10T19:23:39.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.701+0000 7f3e377fe700 1 -- 172.21.15.175:0/2332522037 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 285 v285) v1 ==== 95+0+5423 (secure 0 0 0) 0x7f3e48017090 con 0x7f3e500b2390 2023-10-10T19:23:39.703 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:39.704 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":285,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":285,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:38.849230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:39.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e3c066f60 msgr2=0x7f3e3c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e3c066f60 0x7f3e3c069410 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3e40007fd0 tx=0x7f3e400016f0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 msgr2=0x7f3e500c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e500c5050 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800c7e0 tx=0x7f3e48007d90 comp rx=0 tx=0).stop 2023-10-10T19:23:39.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 shutdown_connections 2023-10-10T19:23:39.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3e3c066f60 0x7f3e3c069410 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e500b2390 0x7f3e500c5050 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e500aff60 0x7f3e50169750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 --2- 172.21.15.175:0/2332522037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e500b1fc0 0x7f3e50169c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.703+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 >> 172.21.15.175:0/2332522037 conn(0x7f3e50063230 msgr2=0x7f3e5014fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:39.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.704+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 shutdown_connections 2023-10-10T19:23:39.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.704+0000 7f3e55c98700 1 -- 172.21.15.175:0/2332522037 wait complete. 2023-10-10T19:23:39.708 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 285 2023-10-10T19:23:39.717 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-10T19:23:39.718 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:23:39.718 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:39.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.831+0000 7f33f2824700 1 Processor -- start 2023-10-10T19:23:39.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.831+0000 7f33f2824700 1 -- start start 2023-10-10T19:23:39.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f2824700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f2824700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f33ec0b2390 0x7f33ec068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f2824700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f33ec068bd0 0x7f33ec0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f2824700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33ec150770 con 0x7f33ec068bd0 2023-10-10T19:23:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f2824700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33ec1574b0 con 0x7f33ec0b2390 2023-10-10T19:23:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f2824700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0afad0 con 0x7f33ec0b1fc0 2023-10-10T19:23:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f0dc1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f33ec068bd0 0x7f33ec0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f0dc1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f33ec068bd0 0x7f33ec0aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38504/0 (socket says 172.21.15.175:38504) 2023-10-10T19:23:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33f0dc1700 1 -- 172.21.15.175:0/2149872157 learned_addr learned my addr 172.21.15.175:0/2149872157 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:39.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33ebfff700 1 --2- 172.21.15.175:0/2149872157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.832+0000 7f33eb7fe700 1 --2- 172.21.15.175:0/2149872157 >> v2:172.21.15.191:3300/0 conn(0x7f33ec0b2390 0x7f33ec068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f0dc1700 1 -- 172.21.15.175:0/2149872157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 msgr2=0x7f33ec155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:39.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f0dc1700 1 --2- 172.21.15.175:0/2149872157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f0dc1700 1 -- 172.21.15.175:0/2149872157 >> v2:172.21.15.191:3300/0 conn(0x7f33ec0b2390 msgr2=0x7f33ec068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f0dc1700 1 --2- 172.21.15.175:0/2149872157 >> v2:172.21.15.191:3300/0 conn(0x7f33ec0b2390 0x7f33ec068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f0dc1700 1 -- 172.21.15.175:0/2149872157 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec0af4c0 con 0x7f33ec068bd0 2023-10-10T19:23:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33ebfff700 1 --2- 172.21.15.175:0/2149872157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:23:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33eb7fe700 1 --2- 172.21.15.175:0/2149872157 >> v2:172.21.15.191:3300/0 conn(0x7f33ec0b2390 0x7f33ec068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:23:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f0dc1700 1 --2- 172.21.15.175:0/2149872157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0aff60 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f33ec071640 tx=0x7f33e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b88a3a2213d3b6f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33eaffd700 1 -- 172.21.15.175:0/2149872157 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e400e070 con 0x7f33ec068bd0 2023-10-10T19:23:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33eaffd700 1 -- 172.21.15.175:0/2149872157 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e400bae0 con 0x7f33ec068bd0 2023-10-10T19:23:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33eaffd700 1 -- 172.21.15.175:0/2149872157 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e40092e0 con 0x7f33ec068bd0 2023-10-10T19:23:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 -- 172.21.15.175:0/2149872157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 msgr2=0x7f33ec0aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 --2- 172.21.15.175:0/2149872157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0aff60 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f33ec071640 tx=0x7f33e4009840 comp rx=0 tx=0).stop 2023-10-10T19:23:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 -- 172.21.15.175:0/2149872157 shutdown_connections 2023-10-10T19:23:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 --2- 172.21.15.175:0/2149872157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0aff60 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 --2- 172.21.15.175:0/2149872157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 --2- 172.21.15.175:0/2149872157 >> v2:172.21.15.191:3300/0 conn(0x7f33ec0b2390 0x7f33ec068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 -- 172.21.15.175:0/2149872157 >> 172.21.15.175:0/2149872157 conn(0x7f33ec063230 msgr2=0x7f33ec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 -- 172.21.15.175:0/2149872157 shutdown_connections 2023-10-10T19:23:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.833+0000 7f33f2824700 1 -- 172.21.15.175:0/2149872157 wait complete. 2023-10-10T19:23:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.834+0000 7f33f2824700 1 Processor -- start 2023-10-10T19:23:39.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33f2824700 1 -- start start 2023-10-10T19:23:39.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33f2824700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33f2824700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33f2824700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33ec0b2390 0x7f33ec0c2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33f2824700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0c2980 con 0x7f33ec068bd0 2023-10-10T19:23:39.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33f2824700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec156ef0 con 0x7f33ec0b2390 2023-10-10T19:23:39.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33f2824700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0c06e0 con 0x7f33ec0b1fc0 2023-10-10T19:23:39.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33ebfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33ebfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0c1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38506/0 (socket says 172.21.15.175:38506) 2023-10-10T19:23:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33ebfff700 1 -- 172.21.15.175:0/622000501 learned_addr learned my addr 172.21.15.175:0/622000501 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.835+0000 7f33f0dc1700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33ec0b2390 0x7f33ec0c2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33eb7fe700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33ebfff700 1 -- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 msgr2=0x7f33ec154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33ebfff700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33ebfff700 1 -- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33ec0b2390 msgr2=0x7f33ec0c2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33ebfff700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33ec0b2390 0x7f33ec0c2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33ebfff700 1 -- 172.21.15.175:0/622000501 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4009ae0 con 0x7f33ec068bd0 2023-10-10T19:23:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33ebfff700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0c1330 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00c880 tx=0x7f33dc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33d3fff700 1 -- 172.21.15.175:0/622000501 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc011600 con 0x7f33ec068bd0 2023-10-10T19:23:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33d3fff700 1 -- 172.21.15.175:0/622000501 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc011760 con 0x7f33ec068bd0 2023-10-10T19:23:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec0c2050 con 0x7f33ec068bd0 2023-10-10T19:23:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33d3fff700 1 -- 172.21.15.175:0/622000501 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc011ae0 con 0x7f33ec068bd0 2023-10-10T19:23:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.836+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec0c2610 con 0x7f33ec068bd0 2023-10-10T19:23:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.837+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec0c2c60 con 0x7f33ec068bd0 2023-10-10T19:23:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.837+0000 7f33d3fff700 1 -- 172.21.15.175:0/622000501 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f33dc018350 con 0x7f33ec068bd0 2023-10-10T19:23:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.838+0000 7f33d3fff700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f33d8066f30 0x7f33d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.838+0000 7f33d3fff700 1 -- 172.21.15.175:0/622000501 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f33dc084b90 con 0x7f33ec068bd0 2023-10-10T19:23:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.838+0000 7f33eb7fe700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f33d8066f30 0x7f33d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.838+0000 7f33eb7fe700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f33d8066f30 0x7f33d80693e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0cd550 tx=0x7f33e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:39.842+0000 7f33d3fff700 1 -- 172.21.15.175:0/622000501 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33dc055160 con 0x7f33ec068bd0 2023-10-10T19:23:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.068+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33ec150f70 con 0x7f33ec068bd0 2023-10-10T19:23:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.069+0000 7f33d3fff700 1 -- 172.21.15.175:0/622000501 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 285 v285) v1 ==== 95+0+5423 (secure 0 0 0) 0x7f33dc054cf0 con 0x7f33ec068bd0 2023-10-10T19:23:40.071 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:40.071 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":285,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":285,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:38.849230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f33d8066f30 msgr2=0x7f33d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f33d8066f30 0x7f33d80693e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0cd550 tx=0x7f33e0008040 comp rx=0 tx=0).stop 2023-10-10T19:23:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 msgr2=0x7f33ec0c1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0c1330 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00c880 tx=0x7f33dc008040 comp rx=0 tx=0).stop 2023-10-10T19:23:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 shutdown_connections 2023-10-10T19:23:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f33d8066f30 0x7f33d80693e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33ec0b2390 0x7f33ec0c2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f33ec068bd0 0x7f33ec0c1330 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 --2- 172.21.15.175:0/622000501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33ec0b1fc0 0x7f33ec154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 >> 172.21.15.175:0/622000501 conn(0x7f33ec063230 msgr2=0x7f33ec158fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 shutdown_connections 2023-10-10T19:23:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:40.071+0000 7f33f2824700 1 -- 172.21.15.175:0/622000501 wait complete. 2023-10-10T19:23:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 285 2023-10-10T19:23:40.085 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 285, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 285, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_7252": { "addr": "172.21.15.191:6839/2393527600", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 2393527600, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 2393527600, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7252, "incarnation": 271, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 20 }, "gid_7384": { "addr": "172.21.15.191:6837/2914004704", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 2914004704, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2914004704, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7384, "incarnation": 278, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 23 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 117 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 128, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:23:38.849230+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 7252, "mds_1": 7401, "mds_2": 8049, "mds_3": 7384 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:23:42.086 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:42.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.213+0000 7ff780c05700 1 Processor -- start 2023-10-10T19:23:42.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.213+0000 7ff780c05700 1 -- start start 2023-10-10T19:23:42.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff780c05700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff77c0b1fc0 0x7ff77c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:42.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff780c05700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff77c0aff60 0x7ff77c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:42.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff780c05700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c1574b0 con 0x7ff77c0b2390 2023-10-10T19:23:42.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff780c05700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c150770 con 0x7ff77c0b1fc0 2023-10-10T19:23:42.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff780c05700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c0afad0 con 0x7ff77c0aff60 2023-10-10T19:23:42.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff77a59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff77c0b1fc0 0x7ff77c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:42.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff77a59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff77c0b1fc0 0x7ff77c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38514/0 (socket says 172.21.15.175:38514) 2023-10-10T19:23:42.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff77a59c700 1 -- 172.21.15.175:0/3330796363 learned_addr learned my addr 172.21.15.175:0/3330796363 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:42.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff77ad9d700 1 --2- 172.21.15.175:0/3330796363 >> v2:172.21.15.191:3300/0 conn(0x7ff77c0aff60 0x7ff77c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:42.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff77a59c700 1 -- 172.21.15.175:0/3330796363 >> v2:172.21.15.191:3300/0 conn(0x7ff77c0aff60 msgr2=0x7ff77c0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:42.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff77a59c700 1 --2- 172.21.15.175:0/3330796363 >> v2:172.21.15.191:3300/0 conn(0x7ff77c0aff60 0x7ff77c0b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff77a59c700 1 -- 172.21.15.175:0/3330796363 >> v1:172.21.15.191:6789/0 conn(0x7ff77c0b2390 legacy=0x7ff77c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:42.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.214+0000 7ff77a59c700 1 -- 172.21.15.175:0/3330796363 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c0af4c0 con 0x7ff77c0b1fc0 2023-10-10T19:23:42.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff77a59c700 1 --2- 172.21.15.175:0/3330796363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0b1fc0 0x7ff77c155fb0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0641f0 tx=0x7ff76c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ac3cb3dedc1969 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:42.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff77959a700 1 -- 172.21.15.175:0/3330796363 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c009e70 con 0x7ff77c0b1fc0 2023-10-10T19:23:42.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff77959a700 1 -- 172.21.15.175:0/3330796363 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c00b8f0 con 0x7ff77c0b1fc0 2023-10-10T19:23:42.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff77959a700 1 -- 172.21.15.175:0/3330796363 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c00bc90 con 0x7ff77c0b1fc0 2023-10-10T19:23:42.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff780c05700 1 -- 172.21.15.175:0/3330796363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0b1fc0 msgr2=0x7ff77c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:42.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff780c05700 1 --2- 172.21.15.175:0/3330796363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0b1fc0 0x7ff77c155fb0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0641f0 tx=0x7ff76c009250 comp rx=0 tx=0).stop 2023-10-10T19:23:42.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff780c05700 1 -- 172.21.15.175:0/3330796363 shutdown_connections 2023-10-10T19:23:42.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff780c05700 1 --2- 172.21.15.175:0/3330796363 >> v2:172.21.15.191:3300/0 conn(0x7ff77c0aff60 0x7ff77c0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff780c05700 1 --2- 172.21.15.175:0/3330796363 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0b1fc0 0x7ff77c155fb0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff780c05700 1 -- 172.21.15.175:0/3330796363 >> 172.21.15.175:0/3330796363 conn(0x7ff77c063230 msgr2=0x7ff77c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:42.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff780c05700 1 -- 172.21.15.175:0/3330796363 shutdown_connections 2023-10-10T19:23:42.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.215+0000 7ff780c05700 1 -- 172.21.15.175:0/3330796363 wait complete. 2023-10-10T19:23:42.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.216+0000 7ff780c05700 1 Processor -- start 2023-10-10T19:23:42.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.216+0000 7ff780c05700 1 -- start start 2023-10-10T19:23:42.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff780c05700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0aff60 0x7ff77c154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:42.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff780c05700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff77c0b1fc0 0x7ff77c154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:42.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff780c05700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff77c0b2390 0x7ff77c169280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:42.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff780c05700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c1541d0 con 0x7ff77c0aff60 2023-10-10T19:23:42.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff780c05700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c159c00 con 0x7ff77c0b1fc0 2023-10-10T19:23:42.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff780c05700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c156410 con 0x7ff77c0b2390 2023-10-10T19:23:42.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0aff60 0x7ff77c154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:42.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0aff60 0x7ff77c154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38524/0 (socket says 172.21.15.175:38524) 2023-10-10T19:23:42.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77a59c700 1 -- 172.21.15.175:0/3787825907 learned_addr learned my addr 172.21.15.175:0/3787825907 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:42.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff779d9b700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff77c0b1fc0 0x7ff77c154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:42.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77ad9d700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff77c0b2390 0x7ff77c169280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:42.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77a59c700 1 -- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff77c0b2390 msgr2=0x7ff77c169280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:42.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77a59c700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff77c0b2390 0x7ff77c169280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77a59c700 1 -- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff77c0b1fc0 msgr2=0x7ff77c154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:42.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77a59c700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff77c0b1fc0 0x7ff77c154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77a59c700 1 -- 172.21.15.175:0/3787825907 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0094f0 con 0x7ff77c0aff60 2023-10-10T19:23:42.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.217+0000 7ff77ad9d700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff77c0b2390 0x7ff77c169280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:23:42.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.218+0000 7ff77a59c700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0aff60 0x7ff77c154660 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff77c06fd90 tx=0x7ff76c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:42.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.218+0000 7ff76a7fc700 1 -- 172.21.15.175:0/3787825907 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c00bc70 con 0x7ff77c0aff60 2023-10-10T19:23:42.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.218+0000 7ff76a7fc700 1 -- 172.21.15.175:0/3787825907 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c024440 con 0x7ff77c0aff60 2023-10-10T19:23:42.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.218+0000 7ff76a7fc700 1 -- 172.21.15.175:0/3787825907 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c0247e0 con 0x7ff77c0aff60 2023-10-10T19:23:42.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.218+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c1566f0 con 0x7ff77c0aff60 2023-10-10T19:23:42.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.218+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c159f10 con 0x7ff77c0aff60 2023-10-10T19:23:42.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.219+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c0713a0 con 0x7ff77c0aff60 2023-10-10T19:23:42.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.219+0000 7ff76a7fc700 1 -- 172.21.15.175:0/3787825907 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff76c02d440 con 0x7ff77c0aff60 2023-10-10T19:23:42.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.219+0000 7ff76a7fc700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff75c066f30 0x7ff75c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:42.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.219+0000 7ff76a7fc700 1 -- 172.21.15.175:0/3787825907 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7ff76c086650 con 0x7ff77c0aff60 2023-10-10T19:23:42.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.220+0000 7ff779d9b700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff75c066f30 0x7ff75c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:42.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.220+0000 7ff779d9b700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff75c066f30 0x7ff75c0693e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff77c156340 tx=0x7ff770007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:42.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.224+0000 7ff76a7fc700 1 -- 172.21.15.175:0/3787825907 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff77c0713a0 con 0x7ff77c0aff60 2023-10-10T19:23:42.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.449+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff77c0713a0 con 0x7ff77c0aff60 2023-10-10T19:23:42.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.449+0000 7ff76a7fc700 1 -- 172.21.15.175:0/3787825907 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 285 v285) v1 ==== 95+0+5423 (secure 0 0 0) 0x7ff77c0713a0 con 0x7ff77c0aff60 2023-10-10T19:23:42.451 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:42.451 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":285,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":285,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:38.849230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:42.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff75c066f30 msgr2=0x7ff75c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:42.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff75c066f30 0x7ff75c0693e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff77c156340 tx=0x7ff770007ec0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0aff60 msgr2=0x7ff77c154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:42.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0aff60 0x7ff77c154660 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff77c06fd90 tx=0x7ff76c017000 comp rx=0 tx=0).stop 2023-10-10T19:23:42.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 shutdown_connections 2023-10-10T19:23:42.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff75c066f30 0x7ff75c0693e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff77c0b2390 0x7ff77c169280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff77c0aff60 0x7ff77c154660 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 --2- 172.21.15.175:0/3787825907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff77c0b1fc0 0x7ff77c154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:42.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 >> 172.21.15.175:0/3787825907 conn(0x7ff77c063230 msgr2=0x7ff77c14fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:42.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 shutdown_connections 2023-10-10T19:23:42.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:42.451+0000 7ff780c05700 1 -- 172.21.15.175:0/3787825907 wait complete. 2023-10-10T19:23:42.456 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 285 2023-10-10T19:23:42.465 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:23:44.466 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:44.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.584+0000 7fc8d1389700 1 Processor -- start 2023-10-10T19:23:44.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.584+0000 7fc8d1389700 1 -- start start 2023-10-10T19:23:44.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.585+0000 7fc8d1389700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.585+0000 7fc8d1389700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc8cc157d10 0x7fc8cc14d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.585+0000 7fc8d1389700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc06d8e0 con 0x7fc8cc157400 2023-10-10T19:23:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.585+0000 7fc8d1389700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc06e050 con 0x7fc8cc157d10 2023-10-10T19:23:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.585+0000 7fc8d1389700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc06dbc0 con 0x7fc8cc0c4640 2023-10-10T19:23:44.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8d0b88700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc8cc157d10 0x7fc8cc14d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cb7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cbfff700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc8cc157400 0x7fc8cc06bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52506/0 (socket says 172.21.15.175:52506) 2023-10-10T19:23:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cb7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:55708/0 (socket says 172.21.15.175:55708) 2023-10-10T19:23:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8d0b88700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc8cc157d10 0x7fc8cc14d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:40996/0 (socket says 172.21.15.175:40996) 2023-10-10T19:23:44.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cbfff700 1 -- 172.21.15.175:0/620675114 learned_addr learned my addr 172.21.15.175:0/620675114 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:44.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cb7fe700 1 -- 172.21.15.175:0/620675114 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc157d10 msgr2=0x7fc8cc14d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:44.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cb7fe700 1 --2- 172.21.15.175:0/620675114 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc157d10 0x7fc8cc14d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cb7fe700 1 -- 172.21.15.175:0/620675114 >> v1:172.21.15.191:6789/0 conn(0x7fc8cc157400 legacy=0x7fc8cc06bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:44.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cb7fe700 1 -- 172.21.15.175:0/620675114 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0c1030 con 0x7fc8cc0c4640 2023-10-10T19:23:44.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8caffd700 1 -- 172.21.15.175:0/620675114 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441341731 0 0) 0x7fc8cc06d8e0 con 0x7fc8cc157400 2023-10-10T19:23:44.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8d0b88700 1 --2- 172.21.15.175:0/620675114 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc157d10 0x7fc8cc14d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:23:44.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8cb7fe700 1 --2- 172.21.15.175:0/620675114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1577d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00098e0 tx=0x7fc8c00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4295711fb84cb51 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:44.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8caffd700 1 -- 172.21.15.175:0/620675114 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000b640 con 0x7fc8cc0c4640 2023-10-10T19:23:44.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8caffd700 1 -- 172.21.15.175:0/620675114 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c000b7a0 con 0x7fc8cc0c4640 2023-10-10T19:23:44.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.586+0000 7fc8caffd700 1 -- 172.21.15.175:0/620675114 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000bb40 con 0x7fc8cc0c4640 2023-10-10T19:23:44.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.587+0000 7fc8d1389700 1 -- 172.21.15.175:0/620675114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 msgr2=0x7fc8cc1577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:44.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.587+0000 7fc8d1389700 1 --2- 172.21.15.175:0/620675114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1577d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00098e0 tx=0x7fc8c00091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.587+0000 7fc8d1389700 1 -- 172.21.15.175:0/620675114 shutdown_connections 2023-10-10T19:23:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.587+0000 7fc8d1389700 1 --2- 172.21.15.175:0/620675114 >> v2:172.21.15.191:3300/0 conn(0x7fc8cc157d10 0x7fc8cc14d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.587+0000 7fc8d1389700 1 --2- 172.21.15.175:0/620675114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1577d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.587+0000 7fc8d1389700 1 -- 172.21.15.175:0/620675114 >> 172.21.15.175:0/620675114 conn(0x7fc8cc063cd0 msgr2=0x7fc8cc06d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.587+0000 7fc8d1389700 1 -- 172.21.15.175:0/620675114 shutdown_connections 2023-10-10T19:23:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.587+0000 7fc8d1389700 1 -- 172.21.15.175:0/620675114 wait complete. 2023-10-10T19:23:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.588+0000 7fc8d1389700 1 Processor -- start 2023-10-10T19:23:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.588+0000 7fc8d1389700 1 -- start start 2023-10-10T19:23:44.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8d1389700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1642f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8d1389700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc157400 0x7fc8cc14da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8d1389700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc157d10 0x7fc8cc145070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8d1389700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0c2680 con 0x7fc8cc157400 2023-10-10T19:23:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8d1389700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc149560 con 0x7fc8cc157d10 2023-10-10T19:23:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8d1389700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc163e60 con 0x7fc8cc0c4640 2023-10-10T19:23:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc157400 0x7fc8cc14da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cb7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc157400 0x7fc8cc14da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46984/0 (socket says 172.21.15.175:46984) 2023-10-10T19:23:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cb7fe700 1 -- 172.21.15.175:0/3681608114 learned_addr learned my addr 172.21.15.175:0/3681608114 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:44.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cbfff700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1642f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:44.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8d0b88700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc157d10 0x7fc8cc145070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:44.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cb7fe700 1 -- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 msgr2=0x7fc8cc1642f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:44.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cb7fe700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1642f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cb7fe700 1 -- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc157d10 msgr2=0x7fc8cc145070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:44.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cb7fe700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc157d10 0x7fc8cc145070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cb7fe700 1 -- 172.21.15.175:0/3681608114 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c4009a20 con 0x7fc8cc157400 2023-10-10T19:23:44.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8d0b88700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc157d10 0x7fc8cc145070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:23:44.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.589+0000 7fc8cbfff700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1642f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:23:44.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.590+0000 7fc8cb7fe700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc157400 0x7fc8cc14da30 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc06dd00 tx=0x7fc8c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:44.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.590+0000 7fc8b3fff700 1 -- 172.21.15.175:0/3681608114 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000bb20 con 0x7fc8cc157400 2023-10-10T19:23:44.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.590+0000 7fc8b3fff700 1 -- 172.21.15.175:0/3681608114 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c0011420 con 0x7fc8cc157400 2023-10-10T19:23:44.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.590+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c0009420 con 0x7fc8cc157400 2023-10-10T19:23:44.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.590+0000 7fc8b3fff700 1 -- 172.21.15.175:0/3681608114 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c00117c0 con 0x7fc8cc157400 2023-10-10T19:23:44.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.590+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc0c6630 con 0x7fc8cc157400 2023-10-10T19:23:44.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.591+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc067570 con 0x7fc8cc157400 2023-10-10T19:23:44.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.592+0000 7fc8b3fff700 1 -- 172.21.15.175:0/3681608114 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc8c0011b30 con 0x7fc8cc157400 2023-10-10T19:23:44.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.592+0000 7fc8b3fff700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f30 0x7fc8b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:44.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.592+0000 7fc8b3fff700 1 -- 172.21.15.175:0/3681608114 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7fc8c0086350 con 0x7fc8cc157400 2023-10-10T19:23:44.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.592+0000 7fc8cbfff700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f30 0x7fc8b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:44.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.593+0000 7fc8cbfff700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f30 0x7fc8b80693e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0011e0 tx=0x7fc8bc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:44.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.600+0000 7fc8b3fff700 1 -- 172.21.15.175:0/3681608114 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8c0052e90 con 0x7fc8cc157400 2023-10-10T19:23:44.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.840+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8cc0c12f0 con 0x7fc8cc157400 2023-10-10T19:23:44.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.841+0000 7fc8b3fff700 1 -- 172.21.15.175:0/3681608114 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 285 v285) v1 ==== 95+0+5423 (secure 0 0 0) 0x7fc8c00564b0 con 0x7fc8cc157400 2023-10-10T19:23:44.841 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:44.842 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":285,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":285,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:38.849230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:44.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f30 msgr2=0x7fc8b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:44.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f30 0x7fc8b80693e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0011e0 tx=0x7fc8bc009040 comp rx=0 tx=0).stop 2023-10-10T19:23:44.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc157400 msgr2=0x7fc8cc14da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:44.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc157400 0x7fc8cc14da30 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc06dd00 tx=0x7fc8c0017000 comp rx=0 tx=0).stop 2023-10-10T19:23:44.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 shutdown_connections 2023-10-10T19:23:44.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc8b8066f30 0x7fc8b80693e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8cc157d10 0x7fc8cc145070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc8cc0c4640 0x7fc8cc1642f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 --2- 172.21.15.175:0/3681608114 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8cc157400 0x7fc8cc14da30 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:44.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 >> 172.21.15.175:0/3681608114 conn(0x7fc8cc063cd0 msgr2=0x7fc8cc14be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:44.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 shutdown_connections 2023-10-10T19:23:44.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:44.843+0000 7fc8d1389700 1 -- 172.21.15.175:0/3681608114 wait complete. 2023-10-10T19:23:44.847 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 285 2023-10-10T19:23:44.856 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:23:46.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:46.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.984+0000 7fac242c8700 1 Processor -- start 2023-10-10T19:23:46.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.984+0000 7fac242c8700 1 -- start start 2023-10-10T19:23:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac242c8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fac1c0b1fc0 0x7fac1c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac242c8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fac1c0aff60 0x7fac1c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac242c8700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1c1574b0 con 0x7fac1c0b2390 2023-10-10T19:23:46.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac242c8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c150770 con 0x7fac1c0b1fc0 2023-10-10T19:23:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac242c8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c0afad0 con 0x7fac1c0aff60 2023-10-10T19:23:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac22064700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fac1c0b1fc0 0x7fac1c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac21863700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fac1c0b2390 0x7fac1c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33606/0 (socket says 172.21.15.175:33606) 2023-10-10T19:23:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac21863700 1 -- 172.21.15.175:0/4054054386 learned_addr learned my addr 172.21.15.175:0/4054054386 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:46.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac22064700 1 -- 172.21.15.175:0/4054054386 >> v2:172.21.15.191:3300/0 conn(0x7fac1c0aff60 msgr2=0x7fac1c0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac22865700 1 --2- 172.21.15.175:0/4054054386 >> v2:172.21.15.191:3300/0 conn(0x7fac1c0aff60 0x7fac1c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac22064700 1 --2- 172.21.15.175:0/4054054386 >> v2:172.21.15.191:3300/0 conn(0x7fac1c0aff60 0x7fac1c0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac22064700 1 -- 172.21.15.175:0/4054054386 >> v1:172.21.15.175:6789/0 conn(0x7fac1c0b2390 legacy=0x7fac1c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac21062700 1 -- 172.21.15.175:0/4054054386 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383133028 0 0) 0x7fac1c1574b0 con 0x7fac1c0b2390 2023-10-10T19:23:46.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac22064700 1 -- 172.21.15.175:0/4054054386 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c0af4c0 con 0x7fac1c0b1fc0 2023-10-10T19:23:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac22064700 1 --2- 172.21.15.175:0/4054054386 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0b1fc0 0x7fac1c155fb0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0641f0 tx=0x7fac0c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0127b343a66f102 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.985+0000 7fac21062700 1 -- 172.21.15.175:0/4054054386 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c00e070 con 0x7fac1c0b1fc0 2023-10-10T19:23:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac21062700 1 -- 172.21.15.175:0/4054054386 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c00bcc0 con 0x7fac1c0b1fc0 2023-10-10T19:23:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac21062700 1 -- 172.21.15.175:0/4054054386 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c012680 con 0x7fac1c0b1fc0 2023-10-10T19:23:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac242c8700 1 -- 172.21.15.175:0/4054054386 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0b1fc0 msgr2=0x7fac1c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:46.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac242c8700 1 --2- 172.21.15.175:0/4054054386 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0b1fc0 0x7fac1c155fb0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0641f0 tx=0x7fac0c009780 comp rx=0 tx=0).stop 2023-10-10T19:23:46.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac242c8700 1 -- 172.21.15.175:0/4054054386 shutdown_connections 2023-10-10T19:23:46.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac242c8700 1 --2- 172.21.15.175:0/4054054386 >> v2:172.21.15.191:3300/0 conn(0x7fac1c0aff60 0x7fac1c0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:46.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac242c8700 1 --2- 172.21.15.175:0/4054054386 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0b1fc0 0x7fac1c155fb0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:46.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac242c8700 1 -- 172.21.15.175:0/4054054386 >> 172.21.15.175:0/4054054386 conn(0x7fac1c063230 msgr2=0x7fac1c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:46.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac242c8700 1 -- 172.21.15.175:0/4054054386 shutdown_connections 2023-10-10T19:23:46.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.986+0000 7fac242c8700 1 -- 172.21.15.175:0/4054054386 wait complete. 2023-10-10T19:23:46.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.987+0000 7fac242c8700 1 Processor -- start 2023-10-10T19:23:46.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.987+0000 7fac242c8700 1 -- start start 2023-10-10T19:23:46.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac242c8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0aff60 0x7fac1c1667b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:46.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac242c8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac1c0b1fc0 0x7fac1c159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:46.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac242c8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac1c0b2390 0x7fac1c14db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:46.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac242c8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c159690 con 0x7fac1c0aff60 2023-10-10T19:23:46.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac242c8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c158700 con 0x7fac1c0b2390 2023-10-10T19:23:46.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac242c8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c159080 con 0x7fac1c0b1fc0 2023-10-10T19:23:46.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22064700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0aff60 0x7fac1c1667b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:46.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22064700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0aff60 0x7fac1c1667b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47002/0 (socket says 172.21.15.175:47002) 2023-10-10T19:23:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22064700 1 -- 172.21.15.175:0/420128923 learned_addr learned my addr 172.21.15.175:0/420128923 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22865700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac1c0b2390 0x7fac1c14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac21863700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac1c0b1fc0 0x7fac1c159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:46.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22064700 1 -- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac1c0b1fc0 msgr2=0x7fac1c159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22064700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac1c0b1fc0 0x7fac1c159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22064700 1 -- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac1c0b2390 msgr2=0x7fac1c14db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22064700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac1c0b2390 0x7fac1c14db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac22064700 1 -- 172.21.15.175:0/420128923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c009a20 con 0x7fac1c0aff60 2023-10-10T19:23:46.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.988+0000 7fac21863700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac1c0b1fc0 0x7fac1c159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:23:46.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.989+0000 7fac22064700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0aff60 0x7fac1c1667b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0099f0 tx=0x7fac0c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:46.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.989+0000 7fac1a7fc700 1 -- 172.21.15.175:0/420128923 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c027470 con 0x7fac1c0aff60 2023-10-10T19:23:46.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.989+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c158140 con 0x7fac1c0aff60 2023-10-10T19:23:46.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.989+0000 7fac1a7fc700 1 -- 172.21.15.175:0/420128923 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c027610 con 0x7fac1c0aff60 2023-10-10T19:23:46.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.989+0000 7fac1a7fc700 1 -- 172.21.15.175:0/420128923 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c0279d0 con 0x7fac1c0aff60 2023-10-10T19:23:47.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.989+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c159320 con 0x7fac1c0aff60 2023-10-10T19:23:47.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.990+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c067570 con 0x7fac1c0aff60 2023-10-10T19:23:47.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.990+0000 7fac1a7fc700 1 -- 172.21.15.175:0/420128923 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fac0c024070 con 0x7fac1c0aff60 2023-10-10T19:23:47.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.991+0000 7fac1a7fc700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac08066f30 0x7fac080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:47.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.991+0000 7fac1a7fc700 1 -- 172.21.15.175:0/420128923 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7fac0c087360 con 0x7fac1c0aff60 2023-10-10T19:23:47.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.991+0000 7fac21863700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac08066f30 0x7fac080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:47.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.991+0000 7fac21863700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac08066f30 0x7fac080693e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0c35f0 tx=0x7fac10009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:47.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:46.996+0000 7fac1a7fc700 1 -- 172.21.15.175:0/420128923 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac0c053ea0 con 0x7fac1c0aff60 2023-10-10T19:23:47.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.223+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac1c068da0 con 0x7fac1c0aff60 2023-10-10T19:23:47.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.223+0000 7fac1a7fc700 1 -- 172.21.15.175:0/420128923 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 286 v286) v1 ==== 95+0+5425 (secure 0 0 0) 0x7fac0c0574c0 con 0x7fac1c0aff60 2023-10-10T19:23:47.224 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:47.224 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":286,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":286,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:44.972397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:resolve","state_seq":13,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:47.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac08066f30 msgr2=0x7fac080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:47.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac08066f30 0x7fac080693e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0c35f0 tx=0x7fac10009040 comp rx=0 tx=0).stop 2023-10-10T19:23:47.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0aff60 msgr2=0x7fac1c1667b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:47.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0aff60 0x7fac1c1667b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0099f0 tx=0x7fac0c019000 comp rx=0 tx=0).stop 2023-10-10T19:23:47.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 shutdown_connections 2023-10-10T19:23:47.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fac08066f30 0x7fac080693e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:47.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fac1c0b2390 0x7fac1c14db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:47.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac1c0aff60 0x7fac1c1667b0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:47.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 --2- 172.21.15.175:0/420128923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fac1c0b1fc0 0x7fac1c159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:47.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 >> 172.21.15.175:0/420128923 conn(0x7fac1c063230 msgr2=0x7fac1c14fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:47.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 shutdown_connections 2023-10-10T19:23:47.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:47.225+0000 7fac242c8700 1 -- 172.21.15.175:0/420128923 wait complete. 2023-10-10T19:23:47.228 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 286 2023-10-10T19:23:47.237 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:23:49.239 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:49.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.360+0000 7f6feee40700 1 Processor -- start 2023-10-10T19:23:49.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.361+0000 7f6feee40700 1 -- start start 2023-10-10T19:23:49.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.361+0000 7f6feee40700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c12a0 0x7f6fe81577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:49.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.361+0000 7f6feee40700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6fe80c4f20 0x7f6fe814d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:49.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.361+0000 7f6feee40700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe8154aa0 con 0x7f6fe81573d0 2023-10-10T19:23:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.361+0000 7f6feee40700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe80bfdf0 con 0x7f6fe80c4f20 2023-10-10T19:23:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.361+0000 7f6feee40700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe8063dc0 con 0x7f6fe80c12a0 2023-10-10T19:23:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.361+0000 7f6fed3dd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6fe80c4f20 0x7f6fe814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fed3dd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6fe80c4f20 0x7f6fe814d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47008/0 (socket says 172.21.15.175:47008) 2023-10-10T19:23:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fed3dd700 1 -- 172.21.15.175:0/44979173 learned_addr learned my addr 172.21.15.175:0/44979173 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:49.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fe7fff700 1 --2- 172.21.15.175:0/44979173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c12a0 0x7f6fe81577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:49.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fed3dd700 1 -- 172.21.15.175:0/44979173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c12a0 msgr2=0x7f6fe81577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:49.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fed3dd700 1 --2- 172.21.15.175:0/44979173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c12a0 0x7f6fe81577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fed3dd700 1 -- 172.21.15.175:0/44979173 >> v1:172.21.15.191:6789/0 conn(0x7f6fe81573d0 legacy=0x7f6fe80bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fe77fe700 1 -- 172.21.15.175:0/44979173 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527212487 0 0) 0x7f6fe8154aa0 con 0x7f6fe81573d0 2023-10-10T19:23:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fed3dd700 1 -- 172.21.15.175:0/44979173 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe80c5360 con 0x7f6fe80c4f20 2023-10-10T19:23:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fed3dd700 1 --2- 172.21.15.175:0/44979173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c4f20 0x7f6fe814d450 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00099b0 tx=0x7f6fe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5953693d849ae5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fe77fe700 1 -- 172.21.15.175:0/44979173 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0009ea0 con 0x7f6fe80c4f20 2023-10-10T19:23:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fe77fe700 1 -- 172.21.15.175:0/44979173 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe000b860 con 0x7f6fe80c4f20 2023-10-10T19:23:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.362+0000 7f6fe77fe700 1 -- 172.21.15.175:0/44979173 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe000bc00 con 0x7f6fe80c4f20 2023-10-10T19:23:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.363+0000 7f6feee40700 1 -- 172.21.15.175:0/44979173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c4f20 msgr2=0x7f6fe814d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.363+0000 7f6feee40700 1 --2- 172.21.15.175:0/44979173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c4f20 0x7f6fe814d450 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00099b0 tx=0x7f6fe0009250 comp rx=0 tx=0).stop 2023-10-10T19:23:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.363+0000 7f6feee40700 1 -- 172.21.15.175:0/44979173 shutdown_connections 2023-10-10T19:23:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.363+0000 7f6feee40700 1 --2- 172.21.15.175:0/44979173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c4f20 0x7f6fe814d450 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.363+0000 7f6feee40700 1 --2- 172.21.15.175:0/44979173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c12a0 0x7f6fe81577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.363+0000 7f6feee40700 1 -- 172.21.15.175:0/44979173 >> 172.21.15.175:0/44979173 conn(0x7f6fe806dc70 msgr2=0x7f6fe8063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.363+0000 7f6feee40700 1 -- 172.21.15.175:0/44979173 shutdown_connections 2023-10-10T19:23:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.363+0000 7f6feee40700 1 -- 172.21.15.175:0/44979173 wait complete. 2023-10-10T19:23:49.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.364+0000 7f6feee40700 1 Processor -- start 2023-10-10T19:23:49.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.364+0000 7f6feee40700 1 -- start start 2023-10-10T19:23:49.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6feee40700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c12a0 0x7f6fe80c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6feee40700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c4f20 0x7f6fe814b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6feee40700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fe81573d0 0x7f6fe81610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6feee40700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe806aae0 con 0x7f6fe80c12a0 2023-10-10T19:23:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6feee40700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe81655c0 con 0x7f6fe81573d0 2023-10-10T19:23:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6feee40700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe8165130 con 0x7f6fe80c4f20 2023-10-10T19:23:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c12a0 0x7f6fe80c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c12a0 0x7f6fe80c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47014/0 (socket says 172.21.15.175:47014) 2023-10-10T19:23:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 -- 172.21.15.175:0/4140899592 learned_addr learned my addr 172.21.15.175:0/4140899592 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fed3dd700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fe81573d0 0x7f6fe81610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fe7fff700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c4f20 0x7f6fe814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:49.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 -- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c4f20 msgr2=0x7f6fe814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:49.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c4f20 0x7f6fe814b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 -- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fe81573d0 msgr2=0x7f6fe81610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:49.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fe81573d0 0x7f6fe81610a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 -- 172.21.15.175:0/4140899592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe00094f0 con 0x7f6fe80c12a0 2023-10-10T19:23:49.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fecbdc700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c12a0 0x7f6fe80c83f0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800cbc0 tx=0x7f6fd80017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:49.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fe4ff9700 1 -- 172.21.15.175:0/4140899592 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd800a750 con 0x7f6fe80c12a0 2023-10-10T19:23:49.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fe4ff9700 1 -- 172.21.15.175:0/4140899592 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd800a8d0 con 0x7f6fe80c12a0 2023-10-10T19:23:49.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe8164840 con 0x7f6fe80c12a0 2023-10-10T19:23:49.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6fe4ff9700 1 -- 172.21.15.175:0/4140899592 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd800ac90 con 0x7f6fe80c12a0 2023-10-10T19:23:49.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.365+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe8164ae0 con 0x7f6fe80c12a0 2023-10-10T19:23:49.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.367+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe8067570 con 0x7f6fe80c12a0 2023-10-10T19:23:49.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.367+0000 7f6fe4ff9700 1 -- 172.21.15.175:0/4140899592 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6fd8003030 con 0x7f6fe80c12a0 2023-10-10T19:23:49.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.367+0000 7f6fe4ff9700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fd4066f30 0x7f6fd40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:49.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.367+0000 7f6fe4ff9700 1 -- 172.21.15.175:0/4140899592 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f6fd8084d30 con 0x7f6fe80c12a0 2023-10-10T19:23:49.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.367+0000 7f6fe7fff700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fd4066f30 0x7f6fd40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:49.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.368+0000 7f6fe7fff700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fd4066f30 0x7f6fd40693e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6fe80cb460 tx=0x7f6fdc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:49.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.371+0000 7f6fe4ff9700 1 -- 172.21.15.175:0/4140899592 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fd8051870 con 0x7f6fe80c12a0 2023-10-10T19:23:49.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.596+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fe8165410 con 0x7f6fe80c12a0 2023-10-10T19:23:49.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.596+0000 7f6fe4ff9700 1 -- 172.21.15.175:0/4140899592 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 286 v286) v1 ==== 95+0+5425 (secure 0 0 0) 0x7f6fd8054e90 con 0x7f6fe80c12a0 2023-10-10T19:23:49.597 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:49.597 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":286,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":286,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:44.972397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:resolve","state_seq":13,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:49.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.598+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fd4066f30 msgr2=0x7f6fd40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:49.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.598+0000 7f6feee40700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fd4066f30 0x7f6fd40693e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6fe80cb460 tx=0x7f6fdc008040 comp rx=0 tx=0).stop 2023-10-10T19:23:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.598+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c12a0 msgr2=0x7f6fe80c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.598+0000 7f6feee40700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c12a0 0x7f6fe80c83f0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800cbc0 tx=0x7f6fd80017a0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.599+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 shutdown_connections 2023-10-10T19:23:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.599+0000 7f6feee40700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fd4066f30 0x7f6fd40693e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.599+0000 7f6feee40700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fe81573d0 0x7f6fe81610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.599+0000 7f6feee40700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fe80c12a0 0x7f6fe80c83f0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.599+0000 7f6feee40700 1 --2- 172.21.15.175:0/4140899592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6fe80c4f20 0x7f6fe814b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:49.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.599+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 >> 172.21.15.175:0/4140899592 conn(0x7f6fe806dc70 msgr2=0x7f6fe806b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:49.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.599+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 shutdown_connections 2023-10-10T19:23:49.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:49.599+0000 7f6feee40700 1 -- 172.21.15.175:0/4140899592 wait complete. 2023-10-10T19:23:49.602 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 286 2023-10-10T19:23:49.611 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:23:51.611 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:51.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.735+0000 7fc09f50e700 1 Processor -- start 2023-10-10T19:23:51.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.735+0000 7fc09f50e700 1 -- start start 2023-10-10T19:23:51.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.735+0000 7fc09f50e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 0x7fc098155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:51.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09f50e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc0980b2390 0x7fc098068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:51.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09f50e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0981574b0 con 0x7fc098068bd0 2023-10-10T19:23:51.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09f50e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc098150770 con 0x7fc0980b2390 2023-10-10T19:23:51.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09f50e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0980afad0 con 0x7fc0980b1fc0 2023-10-10T19:23:51.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09caa9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc0980b2390 0x7fc098068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:51.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09caa9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc0980b2390 0x7fc098068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47026/0 (socket says 172.21.15.175:47026) 2023-10-10T19:23:51.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09caa9700 1 -- 172.21.15.175:0/2680572360 learned_addr learned my addr 172.21.15.175:0/2680572360 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:51.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09d2aa700 1 --2- 172.21.15.175:0/2680572360 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 0x7fc098155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:51.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09caa9700 1 -- 172.21.15.175:0/2680572360 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 msgr2=0x7fc098155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:51.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09caa9700 1 --2- 172.21.15.175:0/2680572360 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 0x7fc098155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09caa9700 1 -- 172.21.15.175:0/2680572360 >> v1:172.21.15.191:6789/0 conn(0x7fc098068bd0 legacy=0x7fc0980aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:51.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09caa9700 1 -- 172.21.15.175:0/2680572360 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980af4c0 con 0x7fc0980b2390 2023-10-10T19:23:51.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc09caa9700 1 --2- 172.21.15.175:0/2680572360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098068690 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fc0880099b0 tx=0x7fc088009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54461b7e3c934e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:51.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc08ffff700 1 -- 172.21.15.175:0/2680572360 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08800b8f0 con 0x7fc0980b2390 2023-10-10T19:23:51.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc08ffff700 1 -- 172.21.15.175:0/2680572360 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08800ba50 con 0x7fc0980b2390 2023-10-10T19:23:51.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.736+0000 7fc08ffff700 1 -- 172.21.15.175:0/2680572360 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08800bdf0 con 0x7fc0980b2390 2023-10-10T19:23:51.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.737+0000 7fc09f50e700 1 -- 172.21.15.175:0/2680572360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 msgr2=0x7fc098068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.737+0000 7fc09f50e700 1 --2- 172.21.15.175:0/2680572360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098068690 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fc0880099b0 tx=0x7fc088009250 comp rx=0 tx=0).stop 2023-10-10T19:23:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.737+0000 7fc09f50e700 1 -- 172.21.15.175:0/2680572360 shutdown_connections 2023-10-10T19:23:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.737+0000 7fc09f50e700 1 --2- 172.21.15.175:0/2680572360 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 0x7fc098155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.737+0000 7fc09f50e700 1 --2- 172.21.15.175:0/2680572360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098068690 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.737+0000 7fc09f50e700 1 -- 172.21.15.175:0/2680572360 >> 172.21.15.175:0/2680572360 conn(0x7fc098063230 msgr2=0x7fc098063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:51.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.737+0000 7fc09f50e700 1 -- 172.21.15.175:0/2680572360 shutdown_connections 2023-10-10T19:23:51.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.737+0000 7fc09f50e700 1 -- 172.21.15.175:0/2680572360 wait complete. 2023-10-10T19:23:51.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.738+0000 7fc09f50e700 1 Processor -- start 2023-10-10T19:23:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.738+0000 7fc09f50e700 1 -- start start 2023-10-10T19:23:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09f50e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc098068bd0 0x7fc0981598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09f50e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 0x7fc098159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09f50e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098165f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09f50e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0980c4950 con 0x7fc0980b2390 2023-10-10T19:23:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09f50e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0981550a0 con 0x7fc098068bd0 2023-10-10T19:23:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09f50e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09816a110 con 0x7fc0980b1fc0 2023-10-10T19:23:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098165f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47030/0 (socket says 172.21.15.175:47030) 2023-10-10T19:23:51.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 -- 172.21.15.175:0/3029714537 learned_addr learned my addr 172.21.15.175:0/3029714537 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:51.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09caa9700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 0x7fc098159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:51.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09d2aa700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc098068bd0 0x7fc0981598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:51.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 -- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 msgr2=0x7fc098159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:51.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 0x7fc098159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 -- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc098068bd0 msgr2=0x7fc0981598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:51.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc098068bd0 0x7fc0981598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 -- 172.21.15.175:0/3029714537 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0880094f0 con 0x7fc0980b2390 2023-10-10T19:23:51.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.739+0000 7fc09daab700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098165f80 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fc09000c590 tx=0x7fc090007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:51.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.740+0000 7fc08d7fa700 1 -- 172.21.15.175:0/3029714537 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090011590 con 0x7fc0980b2390 2023-10-10T19:23:51.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.740+0000 7fc08d7fa700 1 -- 172.21.15.175:0/3029714537 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0900116f0 con 0x7fc0980b2390 2023-10-10T19:23:51.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.740+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0981552d0 con 0x7fc0980b2390 2023-10-10T19:23:51.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.740+0000 7fc08d7fa700 1 -- 172.21.15.175:0/3029714537 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090011ab0 con 0x7fc0980b2390 2023-10-10T19:23:51.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.740+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0981697e0 con 0x7fc0980b2390 2023-10-10T19:23:51.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.740+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc098151530 con 0x7fc0980b2390 2023-10-10T19:23:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.741+0000 7fc08d7fa700 1 -- 172.21.15.175:0/3029714537 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc09001b3b0 con 0x7fc0980b2390 2023-10-10T19:23:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.741+0000 7fc08d7fa700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc07c066f60 0x7fc07c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.741+0000 7fc08d7fa700 1 -- 172.21.15.175:0/3029714537 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7fc090084010 con 0x7fc0980b2390 2023-10-10T19:23:51.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.742+0000 7fc09d2aa700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc07c066f60 0x7fc07c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:51.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.742+0000 7fc09d2aa700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc07c066f60 0x7fc07c069410 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc094009fd0 tx=0x7fc094009880 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:51.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.746+0000 7fc08d7fa700 1 -- 172.21.15.175:0/3029714537 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc090050b50 con 0x7fc0980b2390 2023-10-10T19:23:51.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.970+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc098067570 con 0x7fc0980b2390 2023-10-10T19:23:51.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.970+0000 7fc08d7fa700 1 -- 172.21.15.175:0/3029714537 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 287 v287) v1 ==== 95+0+5427 (secure 0 0 0) 0x7fc090017020 con 0x7fc0980b2390 2023-10-10T19:23:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:51.971 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":287,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:50.002845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:51.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc07c066f60 msgr2=0x7fc07c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:51.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc07c066f60 0x7fc07c069410 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc094009fd0 tx=0x7fc094009880 comp rx=0 tx=0).stop 2023-10-10T19:23:51.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 msgr2=0x7fc098165f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:51.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098165f80 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fc09000c590 tx=0x7fc090007ec0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 shutdown_connections 2023-10-10T19:23:51.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc07c066f60 0x7fc07c069410 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc0980b2390 0x7fc098165f80 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc098068bd0 0x7fc0981598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 --2- 172.21.15.175:0/3029714537 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980b1fc0 0x7fc098159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:51.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.972+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 >> 172.21.15.175:0/3029714537 conn(0x7fc098063230 msgr2=0x7fc09814fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:51.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.973+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 shutdown_connections 2023-10-10T19:23:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:51.973+0000 7fc09f50e700 1 -- 172.21.15.175:0/3029714537 wait complete. 2023-10-10T19:23:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 287 2023-10-10T19:23:51.985 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 287, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 287, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_7252": { "addr": "172.21.15.191:6839/2393527600", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 2393527600, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 2393527600, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7252, "incarnation": 271, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 20 }, "gid_7384": { "addr": "172.21.15.191:6837/2914004704", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 2914004704, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2914004704, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7384, "incarnation": 278, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 23 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 117 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 128, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:23:50.002845+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 7252, "mds_1": 7401, "mds_2": 8049, "mds_3": 7384 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:23:53.986 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:54.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.112+0000 7f748fe76700 1 Processor -- start 2023-10-10T19:23:54.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.112+0000 7f748fe76700 1 -- start start 2023-10-10T19:23:54.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748fe76700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7488155fb0 0x7f7488159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:54.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748fe76700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74881574b0 con 0x7f74880b2390 2023-10-10T19:23:54.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748fe76700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74880afad0 con 0x7f74880b1fc0 2023-10-10T19:23:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748fe76700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7488150770 con 0x7f7488155fb0 2023-10-10T19:23:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748d411700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f74880b2390 0x7f74880aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:33610/0 (socket says 172.21.15.175:33610) 2023-10-10T19:23:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748d411700 1 -- 172.21.15.175:0/663220882 learned_addr learned my addr 172.21.15.175:0/663220882 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:23:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748e413700 1 --2- 172.21.15.175:0/663220882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7488155fb0 0x7f7488159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748cc10700 1 -- 172.21.15.175:0/663220882 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619168154 0 0) 0x7f74881574b0 con 0x7f74880b2390 2023-10-10T19:23:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748cc10700 1 -- 172.21.15.175:0/663220882 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7474003570 con 0x7f74880b2390 2023-10-10T19:23:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748cc10700 1 -- 172.21.15.175:0/663220882 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580294378 0 0) 0x7f74880afad0 con 0x7f74880b1fc0 2023-10-10T19:23:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748cc10700 1 -- 172.21.15.175:0/663220882 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74881574b0 con 0x7f74880b1fc0 2023-10-10T19:23:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748cc10700 1 -- 172.21.15.175:0/663220882 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 758194992 0 0) 0x7f7474003570 con 0x7f74880b2390 2023-10-10T19:23:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748e413700 1 -- 172.21.15.175:0/663220882 >> v1:172.21.15.191:6789/0 conn(0x7f74880b1fc0 legacy=0x7f7488068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:54.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748e413700 1 -- 172.21.15.175:0/663220882 >> v1:172.21.15.175:6789/0 conn(0x7f74880b2390 legacy=0x7f74880aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:54.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.113+0000 7f748e413700 1 -- 172.21.15.175:0/663220882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74880af4c0 con 0x7f7488155fb0 2023-10-10T19:23:54.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748e413700 1 --2- 172.21.15.175:0/663220882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7488155fb0 0x7f7488159920 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7484009910 tx=0x7f74840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cce2758a1b9efa8a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:54.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748cc10700 1 -- 172.21.15.175:0/663220882 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478001470 con 0x7f7488155fb0 2023-10-10T19:23:54.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748cc10700 1 -- 172.21.15.175:0/663220882 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7484011420 con 0x7f7488155fb0 2023-10-10T19:23:54.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748cc10700 1 -- 172.21.15.175:0/663220882 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74840117c0 con 0x7f7488155fb0 2023-10-10T19:23:54.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748fe76700 1 -- 172.21.15.175:0/663220882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7488155fb0 msgr2=0x7f7488159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:54.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748fe76700 1 --2- 172.21.15.175:0/663220882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7488155fb0 0x7f7488159920 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7484009910 tx=0x7f74840091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:54.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748fe76700 1 -- 172.21.15.175:0/663220882 shutdown_connections 2023-10-10T19:23:54.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748fe76700 1 --2- 172.21.15.175:0/663220882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7488155fb0 0x7f7488159920 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:54.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748fe76700 1 -- 172.21.15.175:0/663220882 >> 172.21.15.175:0/663220882 conn(0x7f7488063230 msgr2=0x7f7488063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:54.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748fe76700 1 -- 172.21.15.175:0/663220882 shutdown_connections 2023-10-10T19:23:54.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.114+0000 7f748fe76700 1 -- 172.21.15.175:0/663220882 wait complete. 2023-10-10T19:23:54.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.115+0000 7f748fe76700 1 Processor -- start 2023-10-10T19:23:54.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.115+0000 7f748fe76700 1 -- start start 2023-10-10T19:23:54.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748fe76700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74880b1fc0 0x7f7488152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748fe76700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74880b2390 0x7f7488152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748fe76700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7488155fb0 0x7f748814e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748fe76700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74880c8e00 con 0x7f74880b2390 2023-10-10T19:23:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748fe76700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74880c3980 con 0x7f7488155fb0 2023-10-10T19:23:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748fe76700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7488151f60 con 0x7f74880b1fc0 2023-10-10T19:23:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748d411700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74880b2390 0x7f7488152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748d411700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74880b2390 0x7f7488152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47046/0 (socket says 172.21.15.175:47046) 2023-10-10T19:23:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748d411700 1 -- 172.21.15.175:0/1972303903 learned_addr learned my addr 172.21.15.175:0/1972303903 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748dc12700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74880b1fc0 0x7f7488152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:54.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748e413700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7488155fb0 0x7f748814e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:54.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748d411700 1 -- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74880b1fc0 msgr2=0x7f7488152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:54.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748d411700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74880b1fc0 0x7f7488152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:54.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.116+0000 7f748d411700 1 -- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7488155fb0 msgr2=0x7f748814e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:54.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.117+0000 7f748d411700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7488155fb0 0x7f748814e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:54.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.117+0000 7f748d411700 1 -- 172.21.15.175:0/1972303903 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484009420 con 0x7f74880b2390 2023-10-10T19:23:54.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.117+0000 7f748d411700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74880b2390 0x7f7488152d70 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f747800cb90 tx=0x7f7478001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:54.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.117+0000 7f747dffb700 1 -- 172.21.15.175:0/1972303903 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747800cea0 con 0x7f74880b2390 2023-10-10T19:23:54.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.117+0000 7f747dffb700 1 -- 172.21.15.175:0/1972303903 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7478007e90 con 0x7f74880b2390 2023-10-10T19:23:54.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.117+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7488152990 con 0x7f74880b2390 2023-10-10T19:23:54.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.117+0000 7f747dffb700 1 -- 172.21.15.175:0/1972303903 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74780075d0 con 0x7f74880b2390 2023-10-10T19:23:54.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.117+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748808a4a0 con 0x7f74880b2390 2023-10-10T19:23:54.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.118+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7488067570 con 0x7f74880b2390 2023-10-10T19:23:54.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.118+0000 7f747dffb700 1 -- 172.21.15.175:0/1972303903 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7478004a40 con 0x7f74880b2390 2023-10-10T19:23:54.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.118+0000 7f747dffb700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f746c066f30 0x7f746c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:54.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.119+0000 7f747dffb700 1 -- 172.21.15.175:0/1972303903 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f7478084340 con 0x7f74880b2390 2023-10-10T19:23:54.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.119+0000 7f748dc12700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f746c066f30 0x7f746c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:54.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.119+0000 7f748dc12700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f746c066f30 0x7f746c0693e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7480004140 tx=0x7f7480007f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:54.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.123+0000 7f747dffb700 1 -- 172.21.15.175:0/1972303903 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7478050f00 con 0x7f74880b2390 2023-10-10T19:23:54.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.348+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7488071140 con 0x7f74880b2390 2023-10-10T19:23:54.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.348+0000 7f747dffb700 1 -- 172.21.15.175:0/1972303903 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 287 v287) v1 ==== 95+0+5427 (secure 0 0 0) 0x7f7478054520 con 0x7f74880b2390 2023-10-10T19:23:54.349 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:54.349 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":287,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:50.002845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f746c066f30 msgr2=0x7f746c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f746c066f30 0x7f746c0693e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7480004140 tx=0x7f7480007f90 comp rx=0 tx=0).stop 2023-10-10T19:23:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74880b2390 msgr2=0x7f7488152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:54.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74880b2390 0x7f7488152d70 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f747800cb90 tx=0x7f7478001b50 comp rx=0 tx=0).stop 2023-10-10T19:23:54.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 shutdown_connections 2023-10-10T19:23:54.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f746c066f30 0x7f746c0693e0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:54.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7488155fb0 0x7f748814e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:54.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74880b1fc0 0x7f7488152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:54.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 --2- 172.21.15.175:0/1972303903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74880b2390 0x7f7488152d70 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:54.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 >> 172.21.15.175:0/1972303903 conn(0x7f7488063230 msgr2=0x7f74880693c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:54.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 shutdown_connections 2023-10-10T19:23:54.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:54.350+0000 7f748fe76700 1 -- 172.21.15.175:0/1972303903 wait complete. 2023-10-10T19:23:54.354 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 287 2023-10-10T19:23:54.363 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:23:55.876 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:23:55.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:23:56.364 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:56.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.491+0000 7f1c86de1700 1 Processor -- start 2023-10-10T19:23:56.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.491+0000 7f1c86de1700 1 -- start start 2023-10-10T19:23:56.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c86de1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c800b1fc0 0x7f1c80155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:56.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c86de1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1c800b2390 0x7f1c80068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:56.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c86de1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1c80068bd0 0x7f1c800aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:56.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c86de1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c80150770 con 0x7f1c800b2390 2023-10-10T19:23:56.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c86de1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c801574b0 con 0x7f1c80068bd0 2023-10-10T19:23:56.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c86de1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c800afad0 con 0x7f1c800b1fc0 2023-10-10T19:23:56.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1c800b2390 0x7f1c80068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:56.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1c800b2390 0x7f1c80068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48568/0 (socket says 172.21.15.175:48568) 2023-10-10T19:23:56.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 -- 172.21.15.175:0/3483606206 learned_addr learned my addr 172.21.15.175:0/3483606206 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c84b7d700 1 --2- 172.21.15.175:0/3483606206 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c800b1fc0 0x7f1c80155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c8537e700 1 --2- 172.21.15.175:0/3483606206 >> v2:172.21.15.191:3300/0 conn(0x7f1c80068bd0 0x7f1c800aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 -- 172.21.15.175:0/3483606206 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c800b1fc0 msgr2=0x7f1c80155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 --2- 172.21.15.175:0/3483606206 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c800b1fc0 0x7f1c80155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 -- 172.21.15.175:0/3483606206 >> v2:172.21.15.191:3300/0 conn(0x7f1c80068bd0 msgr2=0x7f1c800aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 --2- 172.21.15.175:0/3483606206 >> v2:172.21.15.191:3300/0 conn(0x7f1c80068bd0 0x7f1c800aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 -- 172.21.15.175:0/3483606206 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800af4c0 con 0x7f1c800b2390 2023-10-10T19:23:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7ffff700 1 --2- 172.21.15.175:0/3483606206 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c80068690 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f1c800afa40 tx=0x7f1c70009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c910eaab46fcb529 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7f7fe700 1 -- 172.21.15.175:0/3483606206 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000e070 con 0x7f1c800b2390 2023-10-10T19:23:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.492+0000 7f1c7f7fe700 1 -- 172.21.15.175:0/3483606206 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7000ba00 con 0x7f1c800b2390 2023-10-10T19:23:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c7f7fe700 1 -- 172.21.15.175:0/3483606206 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c700092e0 con 0x7f1c800b2390 2023-10-10T19:23:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 -- 172.21.15.175:0/3483606206 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 msgr2=0x7f1c80068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3483606206 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c80068690 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f1c800afa40 tx=0x7f1c70009840 comp rx=0 tx=0).stop 2023-10-10T19:23:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 -- 172.21.15.175:0/3483606206 shutdown_connections 2023-10-10T19:23:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3483606206 >> v2:172.21.15.191:3300/0 conn(0x7f1c80068bd0 0x7f1c800aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3483606206 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c800b1fc0 0x7f1c80155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3483606206 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c80068690 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 -- 172.21.15.175:0/3483606206 >> 172.21.15.175:0/3483606206 conn(0x7f1c80063230 msgr2=0x7f1c80063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 -- 172.21.15.175:0/3483606206 shutdown_connections 2023-10-10T19:23:56.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.493+0000 7f1c86de1700 1 -- 172.21.15.175:0/3483606206 wait complete. 2023-10-10T19:23:56.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.494+0000 7f1c86de1700 1 Processor -- start 2023-10-10T19:23:56.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.494+0000 7f1c86de1700 1 -- start start 2023-10-10T19:23:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c86de1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c80068bd0 0x7f1c80157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c86de1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c800b1fc0 0x7f1c80155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c86de1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c801563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c86de1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c800cd740 con 0x7f1c800b2390 2023-10-10T19:23:56.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c86de1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8008a4c0 con 0x7f1c800b1fc0 2023-10-10T19:23:56.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c86de1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c800cd460 con 0x7f1c80068bd0 2023-10-10T19:23:56.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c8537e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c801563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:56.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c8537e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c801563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48578/0 (socket says 172.21.15.175:48578) 2023-10-10T19:23:56.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c8537e700 1 -- 172.21.15.175:0/3474592965 learned_addr learned my addr 172.21.15.175:0/3474592965 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:56.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c84b7d700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c80068bd0 0x7f1c80157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:56.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c7ffff700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c800b1fc0 0x7f1c80155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:56.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c8537e700 1 -- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c80068bd0 msgr2=0x7f1c80157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:56.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c8537e700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c80068bd0 0x7f1c80157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c8537e700 1 -- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c800b1fc0 msgr2=0x7f1c80155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:56.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c8537e700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c800b1fc0 0x7f1c80155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.495+0000 7f1c8537e700 1 -- 172.21.15.175:0/3474592965 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c70009ae0 con 0x7f1c800b2390 2023-10-10T19:23:56.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.496+0000 7f1c8537e700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c801563d0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800ea50 tx=0x7f1c680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:56.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.496+0000 7f1c7cff9700 1 -- 172.21.15.175:0/3474592965 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6800ee90 con 0x7f1c800b2390 2023-10-10T19:23:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.496+0000 7f1c7cff9700 1 -- 172.21.15.175:0/3474592965 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6800c7f0 con 0x7f1c800b2390 2023-10-10T19:23:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.496+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c801550d0 con 0x7f1c800b2390 2023-10-10T19:23:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.496+0000 7f1c7cff9700 1 -- 172.21.15.175:0/3474592965 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6800cbb0 con 0x7f1c800b2390 2023-10-10T19:23:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.496+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c80155690 con 0x7f1c800b2390 2023-10-10T19:23:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.497+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c80067570 con 0x7f1c800b2390 2023-10-10T19:23:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.497+0000 7f1c7cff9700 1 -- 172.21.15.175:0/3474592965 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1c68013470 con 0x7f1c800b2390 2023-10-10T19:23:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.497+0000 7f1c7cff9700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c60066f30 0x7f1c600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.498+0000 7f1c7cff9700 1 -- 172.21.15.175:0/3474592965 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f1c68084da0 con 0x7f1c800b2390 2023-10-10T19:23:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.498+0000 7f1c84b7d700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c60066f30 0x7f1c600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.498+0000 7f1c84b7d700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c60066f30 0x7f1c600693e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009db0 tx=0x7f1c780091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.502+0000 7f1c7cff9700 1 -- 172.21.15.175:0/3474592965 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c680518e0 con 0x7f1c800b2390 2023-10-10T19:23:56.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.726+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c80069060 con 0x7f1c800b2390 2023-10-10T19:23:56.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.727+0000 7f1c7cff9700 1 -- 172.21.15.175:0/3474592965 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+5424 (secure 0 0 0) 0x7f1c68054f00 con 0x7f1c800b2390 2023-10-10T19:23:56.728 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:56.728 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":288,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:55.018728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:56.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c60066f30 msgr2=0x7f1c600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:56.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c60066f30 0x7f1c600693e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009db0 tx=0x7f1c780091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 msgr2=0x7f1c801563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:56.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c801563d0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800ea50 tx=0x7f1c680091a0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 shutdown_connections 2023-10-10T19:23:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c60066f30 0x7f1c600693e0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c800b2390 0x7f1c801563d0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c80068bd0 0x7f1c80157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 --2- 172.21.15.175:0/3474592965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c800b1fc0 0x7f1c80155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 >> 172.21.15.175:0/3474592965 conn(0x7f1c80063230 msgr2=0x7f1c8014fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:56.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 shutdown_connections 2023-10-10T19:23:56.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:56.729+0000 7f1c86de1700 1 -- 172.21.15.175:0/3474592965 wait complete. 2023-10-10T19:23:56.733 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 288 2023-10-10T19:23:56.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:23:58.743 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:23:58.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.871+0000 7fee741c1700 1 Processor -- start 2023-10-10T19:23:58.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.871+0000 7fee741c1700 1 -- start start 2023-10-10T19:23:58.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee741c1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fee6c0b2390 0x7fee6c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:58.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee741c1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0aff60 0x7fee6c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:58.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee741c1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c1574b0 con 0x7fee6c0b1fc0 2023-10-10T19:23:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee741c1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6c150770 con 0x7fee6c0b2390 2023-10-10T19:23:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee741c1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0afad0 con 0x7fee6c0aff60 2023-10-10T19:23:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7175c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fee6c0b2390 0x7fee6c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7175c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fee6c0b2390 0x7fee6c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48582/0 (socket says 172.21.15.175:48582) 2023-10-10T19:23:58.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7175c700 1 -- 172.21.15.175:0/1207542675 learned_addr learned my addr 172.21.15.175:0/1207542675 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:58.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7175c700 1 -- 172.21.15.175:0/1207542675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0aff60 msgr2=0x7fee6c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:58.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7275e700 1 --2- 172.21.15.175:0/1207542675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0aff60 0x7fee6c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:58.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7175c700 1 --2- 172.21.15.175:0/1207542675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0aff60 0x7fee6c0b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:58.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7175c700 1 -- 172.21.15.175:0/1207542675 >> v1:172.21.15.191:6789/0 conn(0x7fee6c0b1fc0 legacy=0x7fee6c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:23:58.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7175c700 1 -- 172.21.15.175:0/1207542675 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c0af4c0 con 0x7fee6c0b2390 2023-10-10T19:23:58.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee7175c700 1 --2- 172.21.15.175:0/1207542675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c155fb0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0099b0 tx=0x7fee5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e901b8ff3f80d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:58.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee70f5b700 1 -- 172.21.15.175:0/1207542675 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00b8b0 con 0x7fee6c0b2390 2023-10-10T19:23:58.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee70f5b700 1 -- 172.21.15.175:0/1207542675 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5c00ba10 con 0x7fee6c0b2390 2023-10-10T19:23:58.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.872+0000 7fee70f5b700 1 -- 172.21.15.175:0/1207542675 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00bdb0 con 0x7fee6c0b2390 2023-10-10T19:23:58.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.873+0000 7fee741c1700 1 -- 172.21.15.175:0/1207542675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 msgr2=0x7fee6c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:58.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.873+0000 7fee741c1700 1 --2- 172.21.15.175:0/1207542675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c155fb0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0099b0 tx=0x7fee5c009250 comp rx=0 tx=0).stop 2023-10-10T19:23:58.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.873+0000 7fee741c1700 1 -- 172.21.15.175:0/1207542675 shutdown_connections 2023-10-10T19:23:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.873+0000 7fee741c1700 1 --2- 172.21.15.175:0/1207542675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0aff60 0x7fee6c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.873+0000 7fee741c1700 1 --2- 172.21.15.175:0/1207542675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c155fb0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.873+0000 7fee741c1700 1 -- 172.21.15.175:0/1207542675 >> 172.21.15.175:0/1207542675 conn(0x7fee6c063230 msgr2=0x7fee6c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.873+0000 7fee741c1700 1 -- 172.21.15.175:0/1207542675 shutdown_connections 2023-10-10T19:23:58.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.873+0000 7fee741c1700 1 -- 172.21.15.175:0/1207542675 wait complete. 2023-10-10T19:23:58.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.874+0000 7fee741c1700 1 Processor -- start 2023-10-10T19:23:58.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.874+0000 7fee741c1700 1 -- start start 2023-10-10T19:23:58.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee741c1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee6c0aff60 0x7fee6c157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:58.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee741c1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0b1fc0 0x7fee6c157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:58.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee741c1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c14d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:58.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee741c1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0cc360 con 0x7fee6c0b2390 2023-10-10T19:23:58.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee741c1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0cc000 con 0x7fee6c0aff60 2023-10-10T19:23:58.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee741c1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c166720 con 0x7fee6c0b1fc0 2023-10-10T19:23:58.886 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c14d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:58.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c14d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48594/0 (socket says 172.21.15.175:48594) 2023-10-10T19:23:58.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 -- 172.21.15.175:0/3213530289 learned_addr learned my addr 172.21.15.175:0/3213530289 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:23:58.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7175c700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0b1fc0 0x7fee6c157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:58.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee71f5d700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee6c0aff60 0x7fee6c157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:58.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 -- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0b1fc0 msgr2=0x7fee6c157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:58.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0b1fc0 0x7fee6c157b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:58.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 -- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee6c0aff60 msgr2=0x7fee6c157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:58.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee6c0aff60 0x7fee6c157640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:58.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 -- 172.21.15.175:0/3213530289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c0094f0 con 0x7fee6c0b2390 2023-10-10T19:23:58.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.875+0000 7fee7275e700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c14d410 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fee6400ceb0 tx=0x7fee64007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:58.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.876+0000 7fee61ffb700 1 -- 172.21.15.175:0/3213530289 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6400ae40 con 0x7fee6c0b2390 2023-10-10T19:23:58.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.876+0000 7fee61ffb700 1 -- 172.21.15.175:0/3213530289 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee64016420 con 0x7fee6c0b2390 2023-10-10T19:23:58.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.876+0000 7fee61ffb700 1 -- 172.21.15.175:0/3213530289 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640167e0 con 0x7fee6c0b2390 2023-10-10T19:23:58.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.876+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c169d70 con 0x7fee6c0b2390 2023-10-10T19:23:58.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.876+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c1660d0 con 0x7fee6c0b2390 2023-10-10T19:23:58.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.877+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c1668d0 con 0x7fee6c0b2390 2023-10-10T19:23:58.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.877+0000 7fee61ffb700 1 -- 172.21.15.175:0/3213530289 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fee64009030 con 0x7fee6c0b2390 2023-10-10T19:23:58.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.877+0000 7fee61ffb700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fee50066f60 0x7fee50069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:23:58.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.877+0000 7fee61ffb700 1 -- 172.21.15.175:0/3213530289 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7fee64012070 con 0x7fee6c0b2390 2023-10-10T19:23:58.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.877+0000 7fee71f5d700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fee50066f60 0x7fee50069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:23:58.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.878+0000 7fee71f5d700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fee50066f60 0x7fee50069410 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fee680027e0 tx=0x7fee68009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:23:58.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:58.882+0000 7fee61ffb700 1 -- 172.21.15.175:0/3213530289 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee6404e5a0 con 0x7fee6c0b2390 2023-10-10T19:23:59.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.109+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee6c067570 con 0x7fee6c0b2390 2023-10-10T19:23:59.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.110+0000 7fee61ffb700 1 -- 172.21.15.175:0/3213530289 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+5424 (secure 0 0 0) 0x7fee6c1668d0 con 0x7fee6c0b2390 2023-10-10T19:23:59.110 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:23:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":288,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:23:55.018728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:23:59.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fee50066f60 msgr2=0x7fee50069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:59.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fee50066f60 0x7fee50069410 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fee680027e0 tx=0x7fee68009040 comp rx=0 tx=0).stop 2023-10-10T19:23:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 msgr2=0x7fee6c14d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:23:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c14d410 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fee6400ceb0 tx=0x7fee64007870 comp rx=0 tx=0).stop 2023-10-10T19:23:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 shutdown_connections 2023-10-10T19:23:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fee50066f60 0x7fee50069410 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee6c0b2390 0x7fee6c14d410 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:59.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee6c0aff60 0x7fee6c157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:59.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 --2- 172.21.15.175:0/3213530289 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6c0b1fc0 0x7fee6c157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:23:59.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 >> 172.21.15.175:0/3213530289 conn(0x7fee6c063230 msgr2=0x7fee6c063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:23:59.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 shutdown_connections 2023-10-10T19:23:59.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:23:59.112+0000 7fee741c1700 1 -- 172.21.15.175:0/3213530289 wait complete. 2023-10-10T19:23:59.115 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 288 2023-10-10T19:23:59.124 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:24:01.124 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:01.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.251+0000 7f026cfe4700 1 Processor -- start 2023-10-10T19:24:01.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.251+0000 7f026cfe4700 1 -- start start 2023-10-10T19:24:01.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f026cfe4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 0x7f0268155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f026cfe4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f02680aff60 0x7f02680b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f026cfe4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02681574b0 con 0x7f02680b2390 2023-10-10T19:24:01.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f026cfe4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0268150770 con 0x7f02680aff60 2023-10-10T19:24:01.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f026cfe4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02680afad0 con 0x7f02680b1fc0 2023-10-10T19:24:01.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f025ffff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f02680b2390 0x7f0268068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46160/0 (socket says 172.21.15.175:46160) 2023-10-10T19:24:01.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f025ffff700 1 -- 172.21.15.175:0/2150209657 learned_addr learned my addr 172.21.15.175:0/2150209657 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:24:01.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f026659c700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 0x7f0268155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0266d9d700 1 --2- 172.21.15.175:0/2150209657 >> v2:172.21.15.191:3300/0 conn(0x7f02680aff60 0x7f02680b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838648684 0 0) 0x7f02681574b0 con 0x7f02680b2390 2023-10-10T19:24:01.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0254003570 con 0x7f02680b2390 2023-10-10T19:24:01.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1612523102 0 0) 0x7f0254003570 con 0x7f02680b2390 2023-10-10T19:24:01.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 msgr2=0x7f0268155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:01.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 0x7f0268155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 >> v2:172.21.15.191:3300/0 conn(0x7f02680aff60 msgr2=0x7f02680b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:01.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 --2- 172.21.15.175:0/2150209657 >> v2:172.21.15.191:3300/0 conn(0x7f02680aff60 0x7f02680b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680af4c0 con 0x7f02680b2390 2023-10-10T19:24:01.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.252+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f0250001470 con 0x7f02680b2390 2023-10-10T19:24:01.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f026659c700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 0x7f0268155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:24:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 >> v1:172.21.15.175:6789/0 conn(0x7f02680b2390 legacy=0x7f0268068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0254003a60 0x7f0254003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 0x7f0254004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0254004d50 0x7f02540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0268150770 con 0x7f0254003a60 2023-10-10T19:24:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02680afad0 con 0x7f0254004d50 2023-10-10T19:24:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025400ba60 con 0x7f02680b1fc0 2023-10-10T19:24:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0266d9d700 1 --2- 172.21.15.175:0/2150209657 >> v2:172.21.15.191:3300/0 conn(0x7f02680aff60 0x7f02680b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:24:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f026659c700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0254003a60 0x7f0254003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f025ffff700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 0x7f0254004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0266d9d700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0254004d50 0x7f02540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f026659c700 1 -- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 msgr2=0x7f0254004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f026659c700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02680b1fc0 0x7f0254004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f026659c700 1 -- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0254004d50 msgr2=0x7f02540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f026659c700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0254004d50 0x7f02540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f026659c700 1 -- 172.21.15.175:0/2150209657 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258009420 con 0x7f0254003a60 2023-10-10T19:24:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f026659c700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0254003a60 0x7f0254003e70 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f0258009df0 tx=0x7f02580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218e77809a55f25d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0258009740 con 0x7f0254003a60 2023-10-10T19:24:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025800e8c0 con 0x7f0254003a60 2023-10-10T19:24:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.253+0000 7f0265d9b700 1 -- 172.21.15.175:0/2150209657 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025800ec80 con 0x7f0254003a60 2023-10-10T19:24:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.254+0000 7f026cfe4700 1 -- 172.21.15.175:0/2150209657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0254003a60 msgr2=0x7f0254003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.254+0000 7f026cfe4700 1 --2- 172.21.15.175:0/2150209657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0254003a60 0x7f0254003e70 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f0258009df0 tx=0x7f02580091a0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.254+0000 7f0266d9d700 1 -- 172.21.15.175:0/2150209657 reap_dead start 2023-10-10T19:24:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.254+0000 7f026cfe4700 1 -- 172.21.15.175:0/2150209657 shutdown_connections 2023-10-10T19:24:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.254+0000 7f026cfe4700 1 -- 172.21.15.175:0/2150209657 >> 172.21.15.175:0/2150209657 conn(0x7f0268063230 msgr2=0x7f0268063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.254+0000 7f026cfe4700 1 -- 172.21.15.175:0/2150209657 shutdown_connections 2023-10-10T19:24:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.254+0000 7f026cfe4700 1 -- 172.21.15.175:0/2150209657 wait complete. 2023-10-10T19:24:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.255+0000 7f026cfe4700 1 Processor -- start 2023-10-10T19:24:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.255+0000 7f026cfe4700 1 -- start start 2023-10-10T19:24:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026cfe4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0254003a60 0x7f0268166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026cfe4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02680c4990 0x7f0268155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026cfe4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02680c4e00 0x7f026814d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026cfe4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0268167230 con 0x7f02680c4e00 2023-10-10T19:24:01.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026cfe4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02681698a0 con 0x7f02680c4990 2023-10-10T19:24:01.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026cfe4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026816a140 con 0x7f0254003a60 2023-10-10T19:24:01.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02680c4e00 0x7f026814d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02680c4e00 0x7f026814d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:48614/0 (socket says 172.21.15.175:48614) 2023-10-10T19:24:01.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 -- 172.21.15.175:0/328879056 learned_addr learned my addr 172.21.15.175:0/328879056 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:01.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026659c700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0254003a60 0x7f0268166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f025ffff700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02680c4990 0x7f0268155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 -- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0254003a60 msgr2=0x7f0268166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:01.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0254003a60 0x7f0268166cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 -- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02680c4990 msgr2=0x7f0268155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:01.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02680c4990 0x7f0268155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 -- 172.21.15.175:0/328879056 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0268159b70 con 0x7f02680c4e00 2023-10-10T19:24:01.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f0266d9d700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02680c4e00 0x7f026814d990 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f02680c0290 tx=0x7f026000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:01.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f025e7fc700 1 -- 172.21.15.175:0/328879056 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026000e070 con 0x7f02680c4e00 2023-10-10T19:24:01.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f025e7fc700 1 -- 172.21.15.175:0/328879056 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0260011bf0 con 0x7f02680c4e00 2023-10-10T19:24:01.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0268159e10 con 0x7f02680c4e00 2023-10-10T19:24:01.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f025e7fc700 1 -- 172.21.15.175:0/328879056 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260016470 con 0x7f02680c4e00 2023-10-10T19:24:01.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.256+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0268169af0 con 0x7f02680c4e00 2023-10-10T19:24:01.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.257+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026816a2b0 con 0x7f02680c4e00 2023-10-10T19:24:01.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.258+0000 7f025e7fc700 1 -- 172.21.15.175:0/328879056 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0260016860 con 0x7f02680c4e00 2023-10-10T19:24:01.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.258+0000 7f025e7fc700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0248066f60 0x7f0248069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:01.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.258+0000 7f025e7fc700 1 -- 172.21.15.175:0/328879056 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f02600857b0 con 0x7f02680c4e00 2023-10-10T19:24:01.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.258+0000 7f026659c700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0248066f60 0x7f0248069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:01.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.259+0000 7f026659c700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0248066f60 0x7f0248069410 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f025800f010 tx=0x7f025801a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:01.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.262+0000 7f025e7fc700 1 -- 172.21.15.175:0/328879056 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f02600522f0 con 0x7f02680c4e00 2023-10-10T19:24:01.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.486+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0268067570 con 0x7f02680c4e00 2023-10-10T19:24:01.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.487+0000 7f025e7fc700 1 -- 172.21.15.175:0/328879056 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 289 v289) v1 ==== 95+0+5424 (secure 0 0 0) 0x7f0260055910 con 0x7f02680c4e00 2023-10-10T19:24:01.488 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:01.488 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":289,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:01.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0248066f60 msgr2=0x7f0248069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:01.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0248066f60 0x7f0248069410 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f025800f010 tx=0x7f025801a000 comp rx=0 tx=0).stop 2023-10-10T19:24:01.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02680c4e00 msgr2=0x7f026814d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:01.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02680c4e00 0x7f026814d990 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f02680c0290 tx=0x7f026000b040 comp rx=0 tx=0).stop 2023-10-10T19:24:01.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 shutdown_connections 2023-10-10T19:24:01.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0248066f60 0x7f0248069410 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02680c4e00 0x7f026814d990 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0254003a60 0x7f0268166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 --2- 172.21.15.175:0/328879056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02680c4990 0x7f0268155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:01.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 >> 172.21.15.175:0/328879056 conn(0x7f0268063230 msgr2=0x7f02681544b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:01.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 shutdown_connections 2023-10-10T19:24:01.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:01.489+0000 7f026cfe4700 1 -- 172.21.15.175:0/328879056 wait complete. 2023-10-10T19:24:01.493 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 289 2023-10-10T19:24:01.501 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=7104 2023-10-10T19:24:01.502 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.e 2023-10-10T19:24:02.959 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-10T19:24:02.959+0000 7fb93ba0c700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-10T19:24:10.503 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-10T19:24:10.503 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-10T19:24:10.504 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-10T19:24:10.506 INFO:tasks.ceph.mds.e:Started 2023-10-10T19:24:10.507 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:10.589 INFO:tasks.ceph.mds.e.smithi175.stdout:starting mds.e at 2023-10-10T19:24:10.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.636+0000 7f59d60f0700 1 Processor -- start 2023-10-10T19:24:10.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.636+0000 7f59d60f0700 1 -- start start 2023-10-10T19:24:10.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59d60f0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f59d00b1fc0 0x7f59d0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:10.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59d60f0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f59d00aff60 0x7f59d00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:10.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59d60f0700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d01574b0 con 0x7f59d00b2390 2023-10-10T19:24:10.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59d60f0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d0150770 con 0x7f59d00b1fc0 2023-10-10T19:24:10.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59d60f0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d00afad0 con 0x7f59d00aff60 2023-10-10T19:24:10.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59cf7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f59d00b1fc0 0x7f59d0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:10.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59ceffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f59d00b2390 0x7f59d0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46946/0 (socket says 172.21.15.175:46946) 2023-10-10T19:24:10.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59ceffd700 1 -- 172.21.15.175:0/1025818197 learned_addr learned my addr 172.21.15.175:0/1025818197 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:24:10.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59cffff700 1 --2- 172.21.15.175:0/1025818197 >> v2:172.21.15.191:3300/0 conn(0x7f59d00aff60 0x7f59d00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:10.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59cf7fe700 1 -- 172.21.15.175:0/1025818197 >> v2:172.21.15.191:3300/0 conn(0x7f59d00aff60 msgr2=0x7f59d00b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:10.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59cf7fe700 1 --2- 172.21.15.175:0/1025818197 >> v2:172.21.15.191:3300/0 conn(0x7f59d00aff60 0x7f59d00b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59cf7fe700 1 -- 172.21.15.175:0/1025818197 >> v1:172.21.15.175:6789/0 conn(0x7f59d00b2390 legacy=0x7f59d0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:10.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59cf7fe700 1 -- 172.21.15.175:0/1025818197 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d00af4c0 con 0x7f59d00b1fc0 2023-10-10T19:24:10.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59ce7fc700 1 -- 172.21.15.175:0/1025818197 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082309564 0 0) 0x7f59d01574b0 con 0x7f59d00b2390 2023-10-10T19:24:10.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.637+0000 7f59cf7fe700 1 --2- 172.21.15.175:0/1025818197 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b1fc0 0x7f59d0155fb0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f59d00641f0 tx=0x7f59c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66565370c1a8613c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:10.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59ce7fc700 1 -- 172.21.15.175:0/1025818197 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c000b690 con 0x7f59d00b1fc0 2023-10-10T19:24:10.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59ce7fc700 1 -- 172.21.15.175:0/1025818197 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c000b7f0 con 0x7f59d00b1fc0 2023-10-10T19:24:10.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59ce7fc700 1 -- 172.21.15.175:0/1025818197 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c000bb90 con 0x7f59d00b1fc0 2023-10-10T19:24:10.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59d60f0700 1 -- 172.21.15.175:0/1025818197 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b1fc0 msgr2=0x7f59d0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:10.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59d60f0700 1 --2- 172.21.15.175:0/1025818197 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b1fc0 0x7f59d0155fb0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f59d00641f0 tx=0x7f59c00091a0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59d60f0700 1 -- 172.21.15.175:0/1025818197 shutdown_connections 2023-10-10T19:24:10.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59d60f0700 1 --2- 172.21.15.175:0/1025818197 >> v2:172.21.15.191:3300/0 conn(0x7f59d00aff60 0x7f59d00b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59d60f0700 1 --2- 172.21.15.175:0/1025818197 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b1fc0 0x7f59d0155fb0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59d60f0700 1 -- 172.21.15.175:0/1025818197 >> 172.21.15.175:0/1025818197 conn(0x7f59d0063230 msgr2=0x7f59d0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:10.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59d60f0700 1 -- 172.21.15.175:0/1025818197 shutdown_connections 2023-10-10T19:24:10.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.638+0000 7f59d60f0700 1 -- 172.21.15.175:0/1025818197 wait complete. 2023-10-10T19:24:10.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.639+0000 7f59d60f0700 1 Processor -- start 2023-10-10T19:24:10.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.639+0000 7f59d60f0700 1 -- start start 2023-10-10T19:24:10.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59d60f0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59d00aff60 0x7f59d0166a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:10.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59d60f0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59d00b1fc0 0x7f59d00c5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:10.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59d60f0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b2390 0x7f59d00c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:10.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59d60f0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d015a340 con 0x7f59d00b2390 2023-10-10T19:24:10.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59d60f0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d0166590 con 0x7f59d00aff60 2023-10-10T19:24:10.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59d60f0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d015a080 con 0x7f59d00b1fc0 2023-10-10T19:24:10.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59cffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b2390 0x7f59d00c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:10.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59cffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b2390 0x7f59d00c3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34388/0 (socket says 172.21.15.175:34388) 2023-10-10T19:24:10.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59cffff700 1 -- 172.21.15.175:0/2079785972 learned_addr learned my addr 172.21.15.175:0/2079785972 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:10.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59ceffd700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59d00b1fc0 0x7f59d00c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:10.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.640+0000 7f59cf7fe700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59d00aff60 0x7f59d0166a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:10.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59cffff700 1 -- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59d00b1fc0 msgr2=0x7f59d00c5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:10.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59cffff700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59d00b1fc0 0x7f59d00c5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59cffff700 1 -- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59d00aff60 msgr2=0x7f59d0166a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:10.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59cffff700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59d00aff60 0x7f59d0166a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59cffff700 1 -- 172.21.15.175:0/2079785972 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0009420 con 0x7f59d00b2390 2023-10-10T19:24:10.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59cf7fe700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59d00aff60 0x7f59d0166a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:24:10.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59cffff700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b2390 0x7f59d00c3a30 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009e90 tx=0x7f59c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:10.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59b77fe700 1 -- 172.21.15.175:0/2079785972 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c800cc20 con 0x7f59d00b2390 2023-10-10T19:24:10.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59b77fe700 1 -- 172.21.15.175:0/2079785972 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c800f040 con 0x7f59d00b2390 2023-10-10T19:24:10.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59b77fe700 1 -- 172.21.15.175:0/2079785972 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8014480 con 0x7f59d00b2390 2023-10-10T19:24:10.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d0151610 con 0x7f59d00b2390 2023-10-10T19:24:10.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.641+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d0166220 con 0x7f59d00b2390 2023-10-10T19:24:10.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.642+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d0166720 con 0x7f59d00b2390 2023-10-10T19:24:10.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.642+0000 7f59b77fe700 1 -- 172.21.15.175:0/2079785972 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f59c8014830 con 0x7f59d00b2390 2023-10-10T19:24:10.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.643+0000 7f59b77fe700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f59bc066f30 0x7f59bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:10.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.643+0000 7f59b77fe700 1 -- 172.21.15.175:0/2079785972 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f59c80859e0 con 0x7f59d00b2390 2023-10-10T19:24:10.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.643+0000 7f59cf7fe700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f59bc066f30 0x7f59bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:10.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.643+0000 7f59cf7fe700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f59bc066f30 0x7f59bc0693e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f59c00093f0 tx=0x7f59c0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:10.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.647+0000 7f59b77fe700 1 -- 172.21.15.175:0/2079785972 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59c8055fb0 con 0x7f59d00b2390 2023-10-10T19:24:10.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.872+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59d0067570 con 0x7f59d00b2390 2023-10-10T19:24:10.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.873+0000 7f59b77fe700 1 -- 172.21.15.175:0/2079785972 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 289 v289) v1 ==== 95+0+5424 (secure 0 0 0) 0x7f59c805b070 con 0x7f59d00b2390 2023-10-10T19:24:10.874 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:10.874 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":289,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:10.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f59bc066f30 msgr2=0x7f59bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:10.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f59bc066f30 0x7f59bc0693e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f59c00093f0 tx=0x7f59c0017000 comp rx=0 tx=0).stop 2023-10-10T19:24:10.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b2390 msgr2=0x7f59d00c3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:10.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b2390 0x7f59d00c3a30 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009e90 tx=0x7f59c800b040 comp rx=0 tx=0).stop 2023-10-10T19:24:10.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 shutdown_connections 2023-10-10T19:24:10.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f59bc066f30 0x7f59bc0693e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59d00b2390 0x7f59d00c3a30 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59d00aff60 0x7f59d0166a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 --2- 172.21.15.175:0/2079785972 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59d00b1fc0 0x7f59d00c5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:10.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 >> 172.21.15.175:0/2079785972 conn(0x7f59d0063230 msgr2=0x7f59d01527d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:10.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 shutdown_connections 2023-10-10T19:24:10.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:10.875+0000 7f59d60f0700 1 -- 172.21.15.175:0/2079785972 wait complete. 2023-10-10T19:24:10.878 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 289 2023-10-10T19:24:10.887 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-10T19:24:12.888 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:13.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.018+0000 7f00664c1700 1 Processor -- start 2023-10-10T19:24:13.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.018+0000 7f00664c1700 1 -- start start 2023-10-10T19:24:13.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f00664c1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f0060155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:13.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f00664c1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00600aff60 0x7f00600b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:13.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f00664c1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00601574b0 con 0x7f00600b2390 2023-10-10T19:24:13.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f00664c1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0060150770 con 0x7f00600aff60 2023-10-10T19:24:13.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f00664c1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00600afad0 con 0x7f00600b1fc0 2023-10-10T19:24:13.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f0064a5e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00600aff60 0x7f00600b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:13.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f0064a5e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00600aff60 0x7f00600b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34394/0 (socket says 172.21.15.175:34394) 2023-10-10T19:24:13.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f005f7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f00600b2390 0x7f0060068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46962/0 (socket says 172.21.15.175:46962) 2023-10-10T19:24:13.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f0064a5e700 1 -- 172.21.15.175:0/3120063504 learned_addr learned my addr 172.21.15.175:0/3120063504 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:13.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f005ffff700 1 --2- 172.21.15.175:0/3120063504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f0060155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:13.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f0064a5e700 1 -- 172.21.15.175:0/3120063504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 msgr2=0x7f0060155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:13.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f0064a5e700 1 --2- 172.21.15.175:0/3120063504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f0060155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f0064a5e700 1 -- 172.21.15.175:0/3120063504 >> v1:172.21.15.175:6789/0 conn(0x7f00600b2390 legacy=0x7f0060068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:13.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f0064a5e700 1 -- 172.21.15.175:0/3120063504 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00600af4c0 con 0x7f00600aff60 2023-10-10T19:24:13.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f005effd700 1 -- 172.21.15.175:0/3120063504 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828663543 0 0) 0x7f00601574b0 con 0x7f00600b2390 2023-10-10T19:24:13.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f005ffff700 1 --2- 172.21.15.175:0/3120063504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f0060155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:24:13.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f0064a5e700 1 --2- 172.21.15.175:0/3120063504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600aff60 0x7f00600b03e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0058009f50 tx=0x7f0058009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251e61f7c6eaa16a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:13.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f005effd700 1 -- 172.21.15.175:0/3120063504 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005800e070 con 0x7f00600aff60 2023-10-10T19:24:13.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f005effd700 1 -- 172.21.15.175:0/3120063504 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005800bb00 con 0x7f00600aff60 2023-10-10T19:24:13.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.019+0000 7f005effd700 1 -- 172.21.15.175:0/3120063504 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00580092e0 con 0x7f00600aff60 2023-10-10T19:24:13.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.020+0000 7f00664c1700 1 -- 172.21.15.175:0/3120063504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600aff60 msgr2=0x7f00600b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:13.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.020+0000 7f00664c1700 1 --2- 172.21.15.175:0/3120063504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600aff60 0x7f00600b03e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0058009f50 tx=0x7f0058009840 comp rx=0 tx=0).stop 2023-10-10T19:24:13.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.020+0000 7f00664c1700 1 -- 172.21.15.175:0/3120063504 shutdown_connections 2023-10-10T19:24:13.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.020+0000 7f00664c1700 1 --2- 172.21.15.175:0/3120063504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600aff60 0x7f00600b03e0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.020+0000 7f00664c1700 1 --2- 172.21.15.175:0/3120063504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f0060155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.020+0000 7f00664c1700 1 -- 172.21.15.175:0/3120063504 >> 172.21.15.175:0/3120063504 conn(0x7f0060063230 msgr2=0x7f0060063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:13.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.020+0000 7f00664c1700 1 -- 172.21.15.175:0/3120063504 shutdown_connections 2023-10-10T19:24:13.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.020+0000 7f00664c1700 1 -- 172.21.15.175:0/3120063504 wait complete. 2023-10-10T19:24:13.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.021+0000 7f00664c1700 1 Processor -- start 2023-10-10T19:24:13.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.021+0000 7f00664c1700 1 -- start start 2023-10-10T19:24:13.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f00664c1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00600aff60 0x7f00600c1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:13.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f00664c1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f00600c5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:13.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f00664c1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600b2390 0x7f00600c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:13.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f00664c1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00600c2990 con 0x7f00600b2390 2023-10-10T19:24:13.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f00664c1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00600c02c0 con 0x7f00600aff60 2023-10-10T19:24:13.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f00664c1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00600af010 con 0x7f00600b1fc0 2023-10-10T19:24:13.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f0064a5e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600b2390 0x7f00600c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:13.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f0064a5e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600b2390 0x7f00600c5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:34410/0 (socket says 172.21.15.175:34410) 2023-10-10T19:24:13.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f0064a5e700 1 -- 172.21.15.175:0/3549427809 learned_addr learned my addr 172.21.15.175:0/3549427809 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:13.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f005ffff700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00600aff60 0x7f00600c1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:13.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f005f7fe700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f00600c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:13.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f0064a5e700 1 -- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 msgr2=0x7f00600c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:13.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f0064a5e700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f00600c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f0064a5e700 1 -- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00600aff60 msgr2=0x7f00600c1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:13.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f0064a5e700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00600aff60 0x7f00600c1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f0064a5e700 1 -- 172.21.15.175:0/3549427809 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0058009ae0 con 0x7f00600b2390 2023-10-10T19:24:13.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.022+0000 7f005ffff700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00600aff60 0x7f00600c1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:24:13.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.023+0000 7f0064a5e700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600b2390 0x7f00600c5870 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f006006fd90 tx=0x7f0058018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:13.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.023+0000 7f0057fff700 1 -- 172.21.15.175:0/3549427809 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00580032f0 con 0x7f00600b2390 2023-10-10T19:24:13.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.023+0000 7f0057fff700 1 -- 172.21.15.175:0/3549427809 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005801d420 con 0x7f00600b2390 2023-10-10T19:24:13.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.023+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00600c0560 con 0x7f00600b2390 2023-10-10T19:24:13.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.023+0000 7f0057fff700 1 -- 172.21.15.175:0/3549427809 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005801d7e0 con 0x7f00600b2390 2023-10-10T19:24:13.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.023+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0060169c40 con 0x7f00600b2390 2023-10-10T19:24:13.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.024+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00600c3f20 con 0x7f00600b2390 2023-10-10T19:24:13.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.024+0000 7f0057fff700 1 -- 172.21.15.175:0/3549427809 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0058003040 con 0x7f00600b2390 2023-10-10T19:24:13.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.025+0000 7f0057fff700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0040066f30 0x7f00400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:13.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.025+0000 7f0057fff700 1 -- 172.21.15.175:0/3549427809 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f0058086950 con 0x7f00600b2390 2023-10-10T19:24:13.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.025+0000 7f005ffff700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0040066f30 0x7f00400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:13.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.025+0000 7f005ffff700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0040066f30 0x7f00400693e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f0050006f00 tx=0x7f0050008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:13.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.029+0000 7f0057fff700 1 -- 172.21.15.175:0/3549427809 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0058053490 con 0x7f00600b2390 2023-10-10T19:24:13.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.257+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00600c3d30 con 0x7f00600b2390 2023-10-10T19:24:13.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.257+0000 7f0057fff700 1 -- 172.21.15.175:0/3549427809 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f0058056ab0 con 0x7f00600b2390 2023-10-10T19:24:13.258 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:13.258 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:13.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0040066f30 msgr2=0x7f00400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:13.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0040066f30 0x7f00400693e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f0050006f00 tx=0x7f0050008040 comp rx=0 tx=0).stop 2023-10-10T19:24:13.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600b2390 msgr2=0x7f00600c5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:13.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600b2390 0x7f00600c5870 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f006006fd90 tx=0x7f0058018000 comp rx=0 tx=0).stop 2023-10-10T19:24:13.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 shutdown_connections 2023-10-10T19:24:13.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0040066f30 0x7f00400693e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00600b2390 0x7f00600c5870 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00600aff60 0x7f00600c1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 --2- 172.21.15.175:0/3549427809 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00600b1fc0 0x7f00600c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:13.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 >> 172.21.15.175:0/3549427809 conn(0x7f0060063230 msgr2=0x7f00600c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:13.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 shutdown_connections 2023-10-10T19:24:13.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:13.259+0000 7f00664c1700 1 -- 172.21.15.175:0/3549427809 wait complete. 2023-10-10T19:24:13.263 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:13.272 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-10T19:24:13.272 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2023-10-10T19:24:26.126 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:24:26.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:24:40.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:40.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.399+0000 7f66841e6700 1 Processor -- start 2023-10-10T19:24:40.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.399+0000 7f66841e6700 1 -- start start 2023-10-10T19:24:40.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f66841e6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f667c0b1fc0 0x7f667c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f66841e6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c0b2390 0x7f667c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f66841e6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c1574b0 con 0x7f667c068bd0 2023-10-10T19:24:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f66841e6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667c150770 con 0x7f667c0b1fc0 2023-10-10T19:24:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f66841e6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0afad0 con 0x7f667c0b2390 2023-10-10T19:24:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681f82700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f667c0b1fc0 0x7f667c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6682783700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f667c068bd0 0x7f667c0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47230/0 (socket says 172.21.15.175:47230) 2023-10-10T19:24:40.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681f82700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f667c0b1fc0 0x7f667c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36862/0 (socket says 172.21.15.175:36862) 2023-10-10T19:24:40.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6682783700 1 -- 172.21.15.175:0/1818082169 learned_addr learned my addr 172.21.15.175:0/1818082169 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:24:40.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681781700 1 --2- 172.21.15.175:0/1818082169 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c0b2390 0x7f667c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681f82700 1 -- 172.21.15.175:0/1818082169 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c0b2390 msgr2=0x7f667c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681f82700 1 --2- 172.21.15.175:0/1818082169 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c0b2390 0x7f667c068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681f82700 1 -- 172.21.15.175:0/1818082169 >> v1:172.21.15.175:6789/0 conn(0x7f667c068bd0 legacy=0x7f667c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681f82700 1 -- 172.21.15.175:0/1818082169 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c0af4c0 con 0x7f667c0b1fc0 2023-10-10T19:24:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6680f80700 1 -- 172.21.15.175:0/1818082169 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912593213 0 0) 0x7f667c1574b0 con 0x7f667c068bd0 2023-10-10T19:24:40.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681781700 1 --2- 172.21.15.175:0/1818082169 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c0b2390 0x7f667c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:24:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.400+0000 7f6681f82700 1 --2- 172.21.15.175:0/1818082169 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b1fc0 0x7f667c155fb0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f66780098e0 tx=0x7f66780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b13c86db34a004a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f6680f80700 1 -- 172.21.15.175:0/1818082169 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678009e80 con 0x7f667c0b1fc0 2023-10-10T19:24:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f6680f80700 1 -- 172.21.15.175:0/1818082169 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667800b8f0 con 0x7f667c0b1fc0 2023-10-10T19:24:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f6680f80700 1 -- 172.21.15.175:0/1818082169 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667800bc70 con 0x7f667c0b1fc0 2023-10-10T19:24:40.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f66841e6700 1 -- 172.21.15.175:0/1818082169 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b1fc0 msgr2=0x7f667c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f66841e6700 1 --2- 172.21.15.175:0/1818082169 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b1fc0 0x7f667c155fb0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f66780098e0 tx=0x7f66780091a0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f66841e6700 1 -- 172.21.15.175:0/1818082169 shutdown_connections 2023-10-10T19:24:40.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f66841e6700 1 --2- 172.21.15.175:0/1818082169 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b1fc0 0x7f667c155fb0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f66841e6700 1 --2- 172.21.15.175:0/1818082169 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c0b2390 0x7f667c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f66841e6700 1 -- 172.21.15.175:0/1818082169 >> 172.21.15.175:0/1818082169 conn(0x7f667c063230 msgr2=0x7f667c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:40.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f66841e6700 1 -- 172.21.15.175:0/1818082169 shutdown_connections 2023-10-10T19:24:40.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.401+0000 7f66841e6700 1 -- 172.21.15.175:0/1818082169 wait complete. 2023-10-10T19:24:40.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.402+0000 7f66841e6700 1 Processor -- start 2023-10-10T19:24:40.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.402+0000 7f66841e6700 1 -- start start 2023-10-10T19:24:40.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f66841e6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c068bd0 0x7f667c0cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f66841e6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f667c0b1fc0 0x7f667c169140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f66841e6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b2390 0x7f667c0cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f66841e6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0c02c0 con 0x7f667c0b2390 2023-10-10T19:24:40.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f66841e6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0c0840 con 0x7f667c0b1fc0 2023-10-10T19:24:40.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f66841e6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0c18f0 con 0x7f667c068bd0 2023-10-10T19:24:40.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6682783700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b2390 0x7f667c0cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6682783700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b2390 0x7f667c0cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36876/0 (socket says 172.21.15.175:36876) 2023-10-10T19:24:40.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6682783700 1 -- 172.21.15.175:0/1707362462 learned_addr learned my addr 172.21.15.175:0/1707362462 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:40.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6681f82700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c068bd0 0x7f667c0cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6681781700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f667c0b1fc0 0x7f667c169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6682783700 1 -- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c068bd0 msgr2=0x7f667c0cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6682783700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c068bd0 0x7f667c0cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6682783700 1 -- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f667c0b1fc0 msgr2=0x7f667c169140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6682783700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f667c0b1fc0 0x7f667c169140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6682783700 1 -- 172.21.15.175:0/1707362462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678009420 con 0x7f667c0b2390 2023-10-10T19:24:40.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.403+0000 7f6681781700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f667c0b1fc0 0x7f667c169140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:24:40.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.404+0000 7f6682783700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b2390 0x7f667c0cb060 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f667400ccf0 tx=0x7f6674001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:40.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.404+0000 7f6671ffb700 1 -- 172.21.15.175:0/1707362462 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667400a9b0 con 0x7f667c0b2390 2023-10-10T19:24:40.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.404+0000 7f6671ffb700 1 -- 172.21.15.175:0/1707362462 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667400ab10 con 0x7f667c0b2390 2023-10-10T19:24:40.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.404+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c0cc110 con 0x7f667c0b2390 2023-10-10T19:24:40.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.404+0000 7f6671ffb700 1 -- 172.21.15.175:0/1707362462 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66740070f0 con 0x7f667c0b2390 2023-10-10T19:24:40.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.404+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c0c04d0 con 0x7f667c0b2390 2023-10-10T19:24:40.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.405+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667c0c09d0 con 0x7f667c0b2390 2023-10-10T19:24:40.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.405+0000 7f6671ffb700 1 -- 172.21.15.175:0/1707362462 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6674007520 con 0x7f667c0b2390 2023-10-10T19:24:40.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.405+0000 7f6671ffb700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.406+0000 7f6671ffb700 1 -- 172.21.15.175:0/1707362462 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f6674085080 con 0x7f667c0b2390 2023-10-10T19:24:40.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.406+0000 7f6681f82700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.406+0000 7f6681f82700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f66780091e0 tx=0x7f6678017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:40.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.410+0000 7f6671ffb700 1 -- 172.21.15.175:0/1707362462 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6674051c40 con 0x7f667c0b2390 2023-10-10T19:24:40.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.636+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f667c067570 con 0x7f667c0b2390 2023-10-10T19:24:40.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.637+0000 7f6671ffb700 1 -- 172.21.15.175:0/1707362462 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f6674055260 con 0x7f667c0b2390 2023-10-10T19:24:40.638 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:40.639 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:40.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 msgr2=0x7f66600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f66780091e0 tx=0x7f6678017000 comp rx=0 tx=0).stop 2023-10-10T19:24:40.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b2390 msgr2=0x7f667c0cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b2390 0x7f667c0cb060 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f667400ccf0 tx=0x7f6674001a20 comp rx=0 tx=0).stop 2023-10-10T19:24:40.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 shutdown_connections 2023-10-10T19:24:40.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f667c0b2390 0x7f667c0cb060 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f667c068bd0 0x7f667c0cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 --2- 172.21.15.175:0/1707362462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f667c0b1fc0 0x7f667c169140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 >> 172.21.15.175:0/1707362462 conn(0x7f667c063230 msgr2=0x7f667c1508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:40.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 shutdown_connections 2023-10-10T19:24:40.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.639+0000 7f66841e6700 1 -- 172.21.15.175:0/1707362462 wait complete. 2023-10-10T19:24:40.643 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:40.652 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-10T19:24:40.652 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-10T19:24:40.653 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:24:40.664 INFO:tasks.ceph.mds.f:Stopped 2023-10-10T19:24:40.665 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:40.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.777+0000 7f0584d1b700 1 Processor -- start 2023-10-10T19:24:40.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.777+0000 7f0584d1b700 1 -- start start 2023-10-10T19:24:40.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f0584d1b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f058014fd90 0x7f0580159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f0584d1b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f05800c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f0584d1b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0580067fa0 con 0x7f05800cd040 2023-10-10T19:24:40.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f0584d1b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05800c5600 con 0x7f058014fd90 2023-10-10T19:24:40.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f0584d1b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0580156b70 con 0x7f058008a100 2023-10-10T19:24:40.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f057effd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f05800c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f057f7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f058014fd90 0x7f0580159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f057f7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f058014fd90 0x7f0580159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:45930/0 (socket says 172.21.15.175:45930) 2023-10-10T19:24:40.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f057effd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f05800c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33716/0 (socket says 172.21.15.175:33716) 2023-10-10T19:24:40.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.778+0000 7f057f7fe700 1 -- 172.21.15.175:0/1855180718 learned_addr learned my addr 172.21.15.175:0/1855180718 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:40.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057effd700 1 -- 172.21.15.175:0/1855180718 >> v2:172.21.15.191:3300/0 conn(0x7f058014fd90 msgr2=0x7f0580159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:40.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057effd700 1 --2- 172.21.15.175:0/1855180718 >> v2:172.21.15.191:3300/0 conn(0x7f058014fd90 0x7f0580159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057effd700 1 -- 172.21.15.175:0/1855180718 >> v1:172.21.15.191:6789/0 conn(0x7f05800cd040 legacy=0x7f05800684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057effd700 1 -- 172.21.15.175:0/1855180718 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0580150160 con 0x7f058008a100 2023-10-10T19:24:40.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057f7fe700 1 --2- 172.21.15.175:0/1855180718 >> v2:172.21.15.191:3300/0 conn(0x7f058014fd90 0x7f0580159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:24:40.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057effd700 1 --2- 172.21.15.175:0/1855180718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f05800c5eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f05740098e0 tx=0x7f05740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a454d3a5e8d59417 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:40.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057e7fc700 1 -- 172.21.15.175:0/1855180718 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400b640 con 0x7f058008a100 2023-10-10T19:24:40.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057e7fc700 1 -- 172.21.15.175:0/1855180718 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057400b7a0 con 0x7f058008a100 2023-10-10T19:24:40.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f057e7fc700 1 -- 172.21.15.175:0/1855180718 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400bb40 con 0x7f058008a100 2023-10-10T19:24:40.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f0584d1b700 1 -- 172.21.15.175:0/1855180718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 msgr2=0x7f05800c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f0584d1b700 1 --2- 172.21.15.175:0/1855180718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f05800c5eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f05740098e0 tx=0x7f05740091a0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f0584d1b700 1 -- 172.21.15.175:0/1855180718 shutdown_connections 2023-10-10T19:24:40.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.779+0000 7f0584d1b700 1 --2- 172.21.15.175:0/1855180718 >> v2:172.21.15.191:3300/0 conn(0x7f058014fd90 0x7f0580159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.780+0000 7f0584d1b700 1 --2- 172.21.15.175:0/1855180718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f05800c5eb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.780+0000 7f0584d1b700 1 -- 172.21.15.175:0/1855180718 >> 172.21.15.175:0/1855180718 conn(0x7f0580063d00 msgr2=0x7f05800cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:40.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.780+0000 7f0584d1b700 1 -- 172.21.15.175:0/1855180718 shutdown_connections 2023-10-10T19:24:40.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.780+0000 7f0584d1b700 1 -- 172.21.15.175:0/1855180718 wait complete. 2023-10-10T19:24:40.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.781+0000 7f0584d1b700 1 Processor -- start 2023-10-10T19:24:40.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.781+0000 7f0584d1b700 1 -- start start 2023-10-10T19:24:40.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.781+0000 7f0584d1b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f0580169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.781+0000 7f0584d1b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05800cd040 0x7f05800cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.781+0000 7f0584d1b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f058014fd90 0x7f05800cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.781+0000 7f0584d1b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058016a200 con 0x7f05800cd040 2023-10-10T19:24:40.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.781+0000 7f0584d1b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580169500 con 0x7f058014fd90 2023-10-10T19:24:40.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.781+0000 7f0584d1b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800cc440 con 0x7f058008a100 2023-10-10T19:24:40.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05800cd040 0x7f05800cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05800cd040 0x7f05800cbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36878/0 (socket says 172.21.15.175:36878) 2023-10-10T19:24:40.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 -- 172.21.15.175:0/3909114418 learned_addr learned my addr 172.21.15.175:0/3909114418 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:40.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057f7fe700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f0580169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057ffff700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f058014fd90 0x7f05800cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 -- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 msgr2=0x7f0580169cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f0580169cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 -- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f058014fd90 msgr2=0x7f05800cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:40.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f058014fd90 0x7f05800cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:40.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 -- 172.21.15.175:0/3909114418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570009ae0 con 0x7f05800cd040 2023-10-10T19:24:40.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f057effd700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05800cd040 0x7f05800cbf00 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0574009f80 tx=0x7f0574017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:40.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f05677fe700 1 -- 172.21.15.175:0/3909114418 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400bb20 con 0x7f05800cd040 2023-10-10T19:24:40.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f05677fe700 1 -- 172.21.15.175:0/3909114418 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0574000c90 con 0x7f05800cd040 2023-10-10T19:24:40.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0574009420 con 0x7f05800cd040 2023-10-10T19:24:40.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f05677fe700 1 -- 172.21.15.175:0/3909114418 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574011650 con 0x7f05800cd040 2023-10-10T19:24:40.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.782+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0580150860 con 0x7f05800cd040 2023-10-10T19:24:40.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.783+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0580169360 con 0x7f05800cd040 2023-10-10T19:24:40.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.784+0000 7f05677fe700 1 -- 172.21.15.175:0/3909114418 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0574011a80 con 0x7f05800cd040 2023-10-10T19:24:40.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.784+0000 7f05677fe700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f056c066f30 0x7f056c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:40.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.784+0000 7f05677fe700 1 -- 172.21.15.175:0/3909114418 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f05740864c0 con 0x7f05800cd040 2023-10-10T19:24:40.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.784+0000 7f057f7fe700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f056c066f30 0x7f056c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:40.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.785+0000 7f057f7fe700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f056c066f30 0x7f056c0693e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f05700072d0 tx=0x7f057000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:40.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:40.788+0000 7f05677fe700 1 -- 172.21.15.175:0/3909114418 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0580169360 con 0x7f05800cd040 2023-10-10T19:24:41.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.014+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0580169360 con 0x7f05800cd040 2023-10-10T19:24:41.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.015+0000 7f05677fe700 1 -- 172.21.15.175:0/3909114418 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f0580169360 con 0x7f05800cd040 2023-10-10T19:24:41.017 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:41.017 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:41.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f056c066f30 msgr2=0x7f056c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:41.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f056c066f30 0x7f056c0693e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f05700072d0 tx=0x7f057000b040 comp rx=0 tx=0).stop 2023-10-10T19:24:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05800cd040 msgr2=0x7f05800cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05800cd040 0x7f05800cbf00 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0574009f80 tx=0x7f0574017000 comp rx=0 tx=0).stop 2023-10-10T19:24:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 shutdown_connections 2023-10-10T19:24:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f056c066f30 0x7f056c0693e0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:41.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f058014fd90 0x7f05800cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f058008a100 0x7f0580169cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 --2- 172.21.15.175:0/3909114418 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05800cd040 0x7f05800cbf00 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 >> 172.21.15.175:0/3909114418 conn(0x7f0580063d00 msgr2=0x7f0580153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 shutdown_connections 2023-10-10T19:24:41.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:41.017+0000 7f0584d1b700 1 -- 172.21.15.175:0/3909114418 wait complete. 2023-10-10T19:24:41.021 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:41.030 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:24:43.031 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:43.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.156+0000 7fec0475c700 1 Processor -- start 2023-10-10T19:24:43.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.156+0000 7fec0475c700 1 -- start start 2023-10-10T19:24:43.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.157+0000 7fec0475c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc1577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:43.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.157+0000 7fec0475c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7febfc0c4f20 0x7febfc14d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:43.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.157+0000 7fec0475c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc154aa0 con 0x7febfc1573d0 2023-10-10T19:24:43.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.157+0000 7fec0475c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febfc0bfdf0 con 0x7febfc0c4f20 2023-10-10T19:24:43.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.157+0000 7fec0475c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc063dc0 con 0x7febfc0c12a0 2023-10-10T19:24:43.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec02cf9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7febfc0c4f20 0x7febfc14d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:43.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec01cf7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc1577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:43.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec02cf9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7febfc0c4f20 0x7febfc14d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:45954/0 (socket says 172.21.15.175:45954) 2023-10-10T19:24:43.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec02cf9700 1 -- 172.21.15.175:0/2866746310 learned_addr learned my addr 172.21.15.175:0/2866746310 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:43.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec01cf7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc1577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:33730/0 (socket says 172.21.15.175:33730) 2023-10-10T19:24:43.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec02cf9700 1 -- 172.21.15.175:0/2866746310 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 msgr2=0x7febfc1577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:43.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec02cf9700 1 --2- 172.21.15.175:0/2866746310 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc1577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec02cf9700 1 -- 172.21.15.175:0/2866746310 >> v1:172.21.15.191:6789/0 conn(0x7febfc1573d0 legacy=0x7febfc0bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:43.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec02cf9700 1 -- 172.21.15.175:0/2866746310 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc0c5360 con 0x7febfc0c4f20 2023-10-10T19:24:43.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec01cf7700 1 --2- 172.21.15.175:0/2866746310 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc1577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:24:43.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec02cf9700 1 --2- 172.21.15.175:0/2866746310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc0c4f20 0x7febfc14d450 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7febf4009f20 tx=0x7febf4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b973008d76d45b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:43.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec014f6700 1 -- 172.21.15.175:0/2866746310 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf400e070 con 0x7febfc0c4f20 2023-10-10T19:24:43.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec014f6700 1 -- 172.21.15.175:0/2866746310 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf400b9d0 con 0x7febfc0c4f20 2023-10-10T19:24:43.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.158+0000 7fec014f6700 1 -- 172.21.15.175:0/2866746310 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf40092e0 con 0x7febfc0c4f20 2023-10-10T19:24:43.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.159+0000 7fec0475c700 1 -- 172.21.15.175:0/2866746310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc0c4f20 msgr2=0x7febfc14d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:43.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.159+0000 7fec0475c700 1 --2- 172.21.15.175:0/2866746310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc0c4f20 0x7febfc14d450 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7febf4009f20 tx=0x7febf4009840 comp rx=0 tx=0).stop 2023-10-10T19:24:43.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.159+0000 7fec0475c700 1 -- 172.21.15.175:0/2866746310 shutdown_connections 2023-10-10T19:24:43.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.159+0000 7fec0475c700 1 --2- 172.21.15.175:0/2866746310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc0c4f20 0x7febfc14d450 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.159+0000 7fec0475c700 1 --2- 172.21.15.175:0/2866746310 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc1577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.159+0000 7fec0475c700 1 -- 172.21.15.175:0/2866746310 >> 172.21.15.175:0/2866746310 conn(0x7febfc06dc70 msgr2=0x7febfc063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:43.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.159+0000 7fec0475c700 1 -- 172.21.15.175:0/2866746310 shutdown_connections 2023-10-10T19:24:43.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.159+0000 7fec0475c700 1 -- 172.21.15.175:0/2866746310 wait complete. 2023-10-10T19:24:43.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.160+0000 7fec0475c700 1 Processor -- start 2023-10-10T19:24:43.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.160+0000 7fec0475c700 1 -- start start 2023-10-10T19:24:43.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec0475c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:43.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec0475c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7febfc0c4f20 0x7febfc152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:43.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec0475c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc1573d0 0x7febfc167280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:43.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec0475c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc149b00 con 0x7febfc0c4f20 2023-10-10T19:24:43.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec0475c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc148320 con 0x7febfc1573d0 2023-10-10T19:24:43.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec0475c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc154d50 con 0x7febfc0c12a0 2023-10-10T19:24:43.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec01cf7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7febfc0c4f20 0x7febfc152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:43.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec01cf7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7febfc0c4f20 0x7febfc152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36882/0 (socket says 172.21.15.175:36882) 2023-10-10T19:24:43.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec01cf7700 1 -- 172.21.15.175:0/3462941276 learned_addr learned my addr 172.21.15.175:0/3462941276 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:43.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec02cf9700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc1573d0 0x7febfc167280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:43.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec024f8700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:43.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec01cf7700 1 -- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 msgr2=0x7febfc149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:43.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec01cf7700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc149500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec01cf7700 1 -- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc1573d0 msgr2=0x7febfc167280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:43.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec01cf7700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc1573d0 0x7febfc167280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.161+0000 7fec01cf7700 1 -- 172.21.15.175:0/3462941276 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec009420 con 0x7febfc0c4f20 2023-10-10T19:24:43.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.162+0000 7fec01cf7700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7febfc0c4f20 0x7febfc152b50 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7febec00ecb0 tx=0x7febec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:43.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.162+0000 7febf27fc700 1 -- 172.21.15.175:0/3462941276 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec00c9b0 con 0x7febfc0c4f20 2023-10-10T19:24:43.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.162+0000 7febf27fc700 1 -- 172.21.15.175:0/3462941276 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febec00cb10 con 0x7febfc0c4f20 2023-10-10T19:24:43.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.162+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf4009ae0 con 0x7febfc0c4f20 2023-10-10T19:24:43.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.162+0000 7febf27fc700 1 -- 172.21.15.175:0/3462941276 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec00ce90 con 0x7febfc0c4f20 2023-10-10T19:24:43.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.162+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febfc1526c0 con 0x7febfc0c4f20 2023-10-10T19:24:43.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.163+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febfc152870 con 0x7febfc0c4f20 2023-10-10T19:24:43.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.163+0000 7febf27fc700 1 -- 172.21.15.175:0/3462941276 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7febec013680 con 0x7febfc0c4f20 2023-10-10T19:24:43.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.163+0000 7febf27fc700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7febe0066f30 0x7febe00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:43.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.164+0000 7febf27fc700 1 -- 172.21.15.175:0/3462941276 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7febec085040 con 0x7febfc0c4f20 2023-10-10T19:24:43.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.164+0000 7fec024f8700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7febe0066f30 0x7febe00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:43.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.164+0000 7fec024f8700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7febe0066f30 0x7febe00693e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7febf80039e0 tx=0x7febf8007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:43.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.168+0000 7febf27fc700 1 -- 172.21.15.175:0/3462941276 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7febfc152870 con 0x7febfc0c4f20 2023-10-10T19:24:43.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.392+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febfc152870 con 0x7febfc0c4f20 2023-10-10T19:24:43.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.393+0000 7febf27fc700 1 -- 172.21.15.175:0/3462941276 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7febfc152870 con 0x7febfc0c4f20 2023-10-10T19:24:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:43.395 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:43.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7febe0066f30 msgr2=0x7febe00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:43.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7febe0066f30 0x7febe00693e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7febf80039e0 tx=0x7febf8007ec0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7febfc0c4f20 msgr2=0x7febfc152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:43.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7febfc0c4f20 0x7febfc152b50 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7febec00ecb0 tx=0x7febec0091a0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 shutdown_connections 2023-10-10T19:24:43.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7febe0066f30 0x7febe00693e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7febfc1573d0 0x7febfc167280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7febfc0c12a0 0x7febfc149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 --2- 172.21.15.175:0/3462941276 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7febfc0c4f20 0x7febfc152b50 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:43.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 >> 172.21.15.175:0/3462941276 conn(0x7febfc06dc70 msgr2=0x7febfc14aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:43.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 shutdown_connections 2023-10-10T19:24:43.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:43.395+0000 7fec0475c700 1 -- 172.21.15.175:0/3462941276 wait complete. 2023-10-10T19:24:43.399 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:43.408 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:24:45.409 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:45.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.536+0000 7f8b53161700 1 Processor -- start 2023-10-10T19:24:45.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.536+0000 7f8b53161700 1 -- start start 2023-10-10T19:24:45.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b53161700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b4c0b2390 0x7f8b4c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:45.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b53161700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8b4c0aff60 0x7f8b4c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:45.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b53161700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4c1574b0 con 0x7f8b4c0b1fc0 2023-10-10T19:24:45.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b53161700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4c150770 con 0x7f8b4c0b2390 2023-10-10T19:24:45.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b53161700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4c0afad0 con 0x7f8b4c0aff60 2023-10-10T19:24:45.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b4bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b4c0b2390 0x7f8b4c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:45.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b50efd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8b4c0b1fc0 0x7f8b4c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49196/0 (socket says 172.21.15.175:49196) 2023-10-10T19:24:45.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b50efd700 1 -- 172.21.15.175:0/3439699991 learned_addr learned my addr 172.21.15.175:0/3439699991 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:24:45.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b516fe700 1 --2- 172.21.15.175:0/3439699991 >> v2:172.21.15.191:3300/0 conn(0x7f8b4c0aff60 0x7f8b4c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:45.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3439699991 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859728075 0 0) 0x7f8b4c1574b0 con 0x7f8b4c0b1fc0 2023-10-10T19:24:45.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3439699991 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b3c003570 con 0x7f8b4c0b1fc0 2023-10-10T19:24:45.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b4bfff700 1 -- 172.21.15.175:0/3439699991 >> v2:172.21.15.191:3300/0 conn(0x7f8b4c0aff60 msgr2=0x7f8b4c0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:45.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b4bfff700 1 --2- 172.21.15.175:0/3439699991 >> v2:172.21.15.191:3300/0 conn(0x7f8b4c0aff60 0x7f8b4c0b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b4bfff700 1 -- 172.21.15.175:0/3439699991 >> v1:172.21.15.175:6789/0 conn(0x7f8b4c0b1fc0 legacy=0x7f8b4c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:45.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b4bfff700 1 -- 172.21.15.175:0/3439699991 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4c0af4c0 con 0x7f8b4c0b2390 2023-10-10T19:24:45.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.537+0000 7f8b4bfff700 1 --2- 172.21.15.175:0/3439699991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0b2390 0x7f8b4c155fb0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0641f0 tx=0x7f8b380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3fa9f98a632161 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:45.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3439699991 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38011630 con 0x7f8b4c0b2390 2023-10-10T19:24:45.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3439699991 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b380117b0 con 0x7f8b4c0b2390 2023-10-10T19:24:45.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b4b7fe700 1 -- 172.21.15.175:0/3439699991 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38011b70 con 0x7f8b4c0b2390 2023-10-10T19:24:45.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b53161700 1 -- 172.21.15.175:0/3439699991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0b2390 msgr2=0x7f8b4c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:45.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b53161700 1 --2- 172.21.15.175:0/3439699991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0b2390 0x7f8b4c155fb0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0641f0 tx=0x7f8b380091a0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b53161700 1 -- 172.21.15.175:0/3439699991 shutdown_connections 2023-10-10T19:24:45.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b53161700 1 --2- 172.21.15.175:0/3439699991 >> v2:172.21.15.191:3300/0 conn(0x7f8b4c0aff60 0x7f8b4c0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b53161700 1 --2- 172.21.15.175:0/3439699991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0b2390 0x7f8b4c155fb0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b53161700 1 -- 172.21.15.175:0/3439699991 >> 172.21.15.175:0/3439699991 conn(0x7f8b4c063230 msgr2=0x7f8b4c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:45.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b53161700 1 -- 172.21.15.175:0/3439699991 shutdown_connections 2023-10-10T19:24:45.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.538+0000 7f8b53161700 1 -- 172.21.15.175:0/3439699991 wait complete. 2023-10-10T19:24:45.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.539+0000 7f8b53161700 1 Processor -- start 2023-10-10T19:24:45.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.539+0000 7f8b53161700 1 -- start start 2023-10-10T19:24:45.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b53161700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0aff60 0x7f8b4c0cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:45.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b53161700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4c0b1fc0 0x7f8b4c0c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:45.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b53161700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4c0b2390 0x7f8b4c0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:45.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b53161700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c0c4fe0 con 0x7f8b4c0aff60 2023-10-10T19:24:45.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b53161700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c0ccbe0 con 0x7f8b4c0b2390 2023-10-10T19:24:45.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b53161700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c0ccec0 con 0x7f8b4c0b1fc0 2023-10-10T19:24:45.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0aff60 0x7f8b4c0cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:45.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0aff60 0x7f8b4c0cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52394/0 (socket says 172.21.15.175:52394) 2023-10-10T19:24:45.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 -- 172.21.15.175:0/1379005233 learned_addr learned my addr 172.21.15.175:0/1379005233 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:45.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b516fe700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4c0b2390 0x7f8b4c0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:45.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b4bfff700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4c0b1fc0 0x7f8b4c0c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:45.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 -- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4c0b1fc0 msgr2=0x7f8b4c0c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:45.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4c0b1fc0 0x7f8b4c0c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 -- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4c0b2390 msgr2=0x7f8b4c0c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:45.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4c0b2390 0x7f8b4c0c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 -- 172.21.15.175:0/1379005233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b38009420 con 0x7f8b4c0aff60 2023-10-10T19:24:45.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.540+0000 7f8b50efd700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0aff60 0x7f8b4c0cc550 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000c7c0 tx=0x7f8b40007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:45.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.541+0000 7f8b48ff9700 1 -- 172.21.15.175:0/1379005233 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40005b30 con 0x7f8b4c0aff60 2023-10-10T19:24:45.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.541+0000 7f8b48ff9700 1 -- 172.21.15.175:0/1379005233 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40005cb0 con 0x7f8b4c0aff60 2023-10-10T19:24:45.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.541+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4c0cd1a0 con 0x7f8b4c0aff60 2023-10-10T19:24:45.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.541+0000 7f8b48ff9700 1 -- 172.21.15.175:0/1379005233 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40007320 con 0x7f8b4c0aff60 2023-10-10T19:24:45.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.541+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4c0c4c70 con 0x7f8b4c0aff60 2023-10-10T19:24:45.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.542+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4c063e90 con 0x7f8b4c0aff60 2023-10-10T19:24:45.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.542+0000 7f8b48ff9700 1 -- 172.21.15.175:0/1379005233 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8b4001b3b0 con 0x7f8b4c0aff60 2023-10-10T19:24:45.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.542+0000 7f8b48ff9700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b30066f60 0x7f8b30069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:45.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.542+0000 7f8b48ff9700 1 -- 172.21.15.175:0/1379005233 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f8b40084290 con 0x7f8b4c0aff60 2023-10-10T19:24:45.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.543+0000 7f8b4bfff700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b30066f60 0x7f8b30069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:45.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.543+0000 7f8b4bfff700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b30066f60 0x7f8b30069410 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0641f0 tx=0x7f8b38017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:45.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.547+0000 7f8b48ff9700 1 -- 172.21.15.175:0/1379005233 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b40050d40 con 0x7f8b4c0aff60 2023-10-10T19:24:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.771+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b4c0c52c0 con 0x7f8b4c0aff60 2023-10-10T19:24:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.772+0000 7f8b48ff9700 1 -- 172.21.15.175:0/1379005233 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f8b40054360 con 0x7f8b4c0aff60 2023-10-10T19:24:45.774 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:45.774 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b30066f60 msgr2=0x7f8b30069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b30066f60 0x7f8b30069410 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0641f0 tx=0x7f8b38017000 comp rx=0 tx=0).stop 2023-10-10T19:24:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0aff60 msgr2=0x7f8b4c0cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0aff60 0x7f8b4c0cc550 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000c7c0 tx=0x7f8b40007e00 comp rx=0 tx=0).stop 2023-10-10T19:24:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 shutdown_connections 2023-10-10T19:24:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8b30066f60 0x7f8b30069410 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4c0b2390 0x7f8b4c0c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b4c0aff60 0x7f8b4c0cc550 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 --2- 172.21.15.175:0/1379005233 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4c0b1fc0 0x7f8b4c0c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 >> 172.21.15.175:0/1379005233 conn(0x7f8b4c063230 msgr2=0x7f8b4c14fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:45.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 shutdown_connections 2023-10-10T19:24:45.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:45.774+0000 7f8b53161700 1 -- 172.21.15.175:0/1379005233 wait complete. 2023-10-10T19:24:45.778 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:45.788 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:24:47.789 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:47.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f80a1fbb700 1 Processor -- start 2023-10-10T19:24:47.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f80a1fbb700 1 -- start start 2023-10-10T19:24:47.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f80a1fbb700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f809c14d190 0x7f809c152d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f80a1fbb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f809c0c2df0 0x7f809c153240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f80a1fbb700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809c06e940 con 0x7f809c14cdc0 2023-10-10T19:24:47.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f80a1fbb700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f809c0c1c90 con 0x7f809c14d190 2023-10-10T19:24:47.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f80a1fbb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0c6e20 con 0x7f809c0c2df0 2023-10-10T19:24:47.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f809c14cdc0 0x7f809c0c26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49204/0 (socket says 172.21.15.175:49204) 2023-10-10T19:24:47.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809b7fe700 1 -- 172.21.15.175:0/2439894682 learned_addr learned my addr 172.21.15.175:0/2439894682 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:24:47.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926827949 0 0) 0x7f809c06e940 con 0x7f809c14cdc0 2023-10-10T19:24:47.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809bfff700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f809c0c2df0 0x7f809c153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8088003570 con 0x7f809c14cdc0 2023-10-10T19:24:47.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809affd700 1 --2- 172.21.15.175:0/2439894682 >> v2:172.21.15.191:3300/0 conn(0x7f809c14d190 0x7f809c152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 511087798 0 0) 0x7f8088003570 con 0x7f809c14cdc0 2023-10-10T19:24:47.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f809c0c2df0 msgr2=0x7f809c153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:47.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f809c0c2df0 0x7f809c153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:47.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 >> v2:172.21.15.191:3300/0 conn(0x7f809c14d190 msgr2=0x7f809c152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:47.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 --2- 172.21.15.175:0/2439894682 >> v2:172.21.15.191:3300/0 conn(0x7f809c14d190 0x7f809c152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:47.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c06ca40 con 0x7f809c14cdc0 2023-10-10T19:24:47.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f808c001970 con 0x7f809c14cdc0 2023-10-10T19:24:47.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 >> v1:172.21.15.175:6789/0 conn(0x7f809c14cdc0 legacy=0x7f809c0c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:47.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8088003a60 0x7f8088003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f809c0c2df0 0x7f8088004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8088004d50 0x7f80880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0c1c90 con 0x7f8088004d50 2023-10-10T19:24:47.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0c6e20 con 0x7f8088003a60 2023-10-10T19:24:47.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f808800ba60 con 0x7f809c0c2df0 2023-10-10T19:24:47.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809bfff700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8088004d50 0x7f80880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809affd700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f809c0c2df0 0x7f8088004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809b7fe700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8088003a60 0x7f8088003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809bfff700 1 -- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f809c0c2df0 msgr2=0x7f8088004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:47.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.918+0000 7f809bfff700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f809c0c2df0 0x7f8088004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:47.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f809bfff700 1 -- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8088003a60 msgr2=0x7f8088003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:47.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f809bfff700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8088003a60 0x7f8088003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:47.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f809bfff700 1 -- 172.21.15.175:0/2439894682 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8088003570 con 0x7f8088004d50 2023-10-10T19:24:47.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f809bfff700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8088004d50 0x7f80880051c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f809c0c3250 tx=0x7f80940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c6f7fae46ad44d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:47.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809400e6b0 con 0x7f8088004d50 2023-10-10T19:24:47.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809400e810 con 0x7f8088004d50 2023-10-10T19:24:47.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f809a7fc700 1 -- 172.21.15.175:0/2439894682 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809400ebd0 con 0x7f8088004d50 2023-10-10T19:24:47.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f80a1fbb700 1 -- 172.21.15.175:0/2439894682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8088004d50 msgr2=0x7f80880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:47.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f80a1fbb700 1 --2- 172.21.15.175:0/2439894682 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8088004d50 0x7f80880051c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f809c0c3250 tx=0x7f80940091a0 comp rx=0 tx=0).stop 2023-10-10T19:24:47.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f809bfff700 1 -- 172.21.15.175:0/2439894682 reap_dead start 2023-10-10T19:24:47.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f80a1fbb700 1 -- 172.21.15.175:0/2439894682 shutdown_connections 2023-10-10T19:24:47.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f80a1fbb700 1 -- 172.21.15.175:0/2439894682 >> 172.21.15.175:0/2439894682 conn(0x7f809c068900 msgr2=0x7f809c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:47.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f80a1fbb700 1 -- 172.21.15.175:0/2439894682 shutdown_connections 2023-10-10T19:24:47.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.919+0000 7f80a1fbb700 1 -- 172.21.15.175:0/2439894682 wait complete. 2023-10-10T19:24:47.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f80a1fbb700 1 Processor -- start 2023-10-10T19:24:47.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f80a1fbb700 1 -- start start 2023-10-10T19:24:47.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f80a1fbb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8088004d50 0x7f809c162120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f80a1fbb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f809c150510 0x7f809c14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f80a1fbb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f809c149e70 0x7f809c14a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f80a1fbb700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c1501a0 con 0x7f809c149e70 2023-10-10T19:24:47.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f80a1fbb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c14a830 con 0x7f809c150510 2023-10-10T19:24:47.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f80a1fbb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c154000 con 0x7f8088004d50 2023-10-10T19:24:47.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f809bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f809c149e70 0x7f809c14a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f809b7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8088004d50 0x7f809c162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f809affd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f809c150510 0x7f809c14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f809bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f809c149e70 0x7f809c14a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52404/0 (socket says 172.21.15.175:52404) 2023-10-10T19:24:47.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f809b7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8088004d50 0x7f809c162120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:56366/0 (socket says 172.21.15.175:56366) 2023-10-10T19:24:47.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.921+0000 7f809bfff700 1 -- 172.21.15.175:0/3872659258 learned_addr learned my addr 172.21.15.175:0/3872659258 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:47.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f809bfff700 1 -- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8088004d50 msgr2=0x7f809c162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:47.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f809bfff700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8088004d50 0x7f809c162120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:47.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f809bfff700 1 -- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f809c150510 msgr2=0x7f809c14ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:47.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f809bfff700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f809c150510 0x7f809c14ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:47.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f809bfff700 1 -- 172.21.15.175:0/3872659258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c14e350 con 0x7f809c149e70 2023-10-10T19:24:47.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f809bfff700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f809c149e70 0x7f809c14a2f0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f8094016000 tx=0x7f809401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:47.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f80937fe700 1 -- 172.21.15.175:0/3872659258 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094023070 con 0x7f809c149e70 2023-10-10T19:24:47.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f80937fe700 1 -- 172.21.15.175:0/3872659258 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809401e280 con 0x7f809c149e70 2023-10-10T19:24:47.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f80937fe700 1 -- 172.21.15.175:0/3872659258 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809400e6b0 con 0x7f809c149e70 2023-10-10T19:24:47.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809c14e630 con 0x7f809c149e70 2023-10-10T19:24:47.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.922+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809c14ebf0 con 0x7f809c149e70 2023-10-10T19:24:47.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.923+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f809c1541b0 con 0x7f809c149e70 2023-10-10T19:24:47.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.923+0000 7f80937fe700 1 -- 172.21.15.175:0/3872659258 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8094013950 con 0x7f809c149e70 2023-10-10T19:24:47.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.924+0000 7f80937fe700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f807c066f20 0x7f807c0693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:47.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.924+0000 7f80937fe700 1 -- 172.21.15.175:0/3872659258 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f8094086440 con 0x7f809c149e70 2023-10-10T19:24:47.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.924+0000 7f809b7fe700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f807c066f20 0x7f807c0693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:47.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.925+0000 7f809b7fe700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f807c066f20 0x7f807c0693d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f808c0016e0 tx=0x7f808c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:47.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:47.928+0000 7f80937fe700 1 -- 172.21.15.175:0/3872659258 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f809c1541b0 con 0x7f809c149e70 2023-10-10T19:24:48.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.152+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f809c1541b0 con 0x7f809c149e70 2023-10-10T19:24:48.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.152+0000 7f80937fe700 1 -- 172.21.15.175:0/3872659258 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f809c1541b0 con 0x7f809c149e70 2023-10-10T19:24:48.155 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:48.155 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:48.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f807c066f20 msgr2=0x7f807c0693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:48.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f807c066f20 0x7f807c0693d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f808c0016e0 tx=0x7f808c009040 comp rx=0 tx=0).stop 2023-10-10T19:24:48.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f809c149e70 msgr2=0x7f809c14a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:48.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f809c149e70 0x7f809c14a2f0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f8094016000 tx=0x7f809401a000 comp rx=0 tx=0).stop 2023-10-10T19:24:48.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 shutdown_connections 2023-10-10T19:24:48.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f807c066f20 0x7f807c0693d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:48.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f809c149e70 0x7f809c14a2f0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:48.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8088004d50 0x7f809c162120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:48.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 --2- 172.21.15.175:0/3872659258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f809c150510 0x7f809c14ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:48.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 >> 172.21.15.175:0/3872659258 conn(0x7f809c068900 msgr2=0x7f809c0c3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:48.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 shutdown_connections 2023-10-10T19:24:48.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:48.154+0000 7f80a1fbb700 1 -- 172.21.15.175:0/3872659258 wait complete. 2023-10-10T19:24:48.159 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:48.168 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:24:50.170 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:50.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.286+0000 7f6867098700 1 Processor -- start 2023-10-10T19:24:50.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.286+0000 7f6867098700 1 -- start start 2023-10-10T19:24:50.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f6867098700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f68600b2390 0x7f6860155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:50.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f6867098700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68601574b0 con 0x7f68600b1fc0 2023-10-10T19:24:50.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f6867098700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68600afad0 con 0x7f68600aff60 2023-10-10T19:24:50.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f6867098700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6860150770 con 0x7f68600b2390 2023-10-10T19:24:50.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f685ffff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f68600b2390 0x7f6860155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:50.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f6864e34700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f68600b1fc0 0x7f6860068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:49208/0 (socket says 172.21.15.175:49208) 2023-10-10T19:24:50.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f6864e34700 1 -- 172.21.15.175:0/1940871201 learned_addr learned my addr 172.21.15.175:0/1940871201 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:24:50.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f685ffff700 1 -- 172.21.15.175:0/1940871201 >> v1:172.21.15.191:6789/0 conn(0x7f68600aff60 legacy=0x7f6860159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:50.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f685ffff700 1 -- 172.21.15.175:0/1940871201 >> v1:172.21.15.175:6789/0 conn(0x7f68600b1fc0 legacy=0x7f6860068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:50.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f685ffff700 1 -- 172.21.15.175:0/1940871201 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68600af4c0 con 0x7f68600b2390 2023-10-10T19:24:50.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f685ffff700 1 --2- 172.21.15.175:0/1940871201 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b2390 0x7f6860155fb0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f6850009e60 tx=0x7f6850009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c1a46c991e56e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:50.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.287+0000 7f685f7fe700 1 -- 172.21.15.175:0/1940871201 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685000e070 con 0x7f68600b2390 2023-10-10T19:24:50.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f685f7fe700 1 -- 172.21.15.175:0/1940871201 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685000bc80 con 0x7f68600b2390 2023-10-10T19:24:50.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f685f7fe700 1 -- 172.21.15.175:0/1940871201 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850012430 con 0x7f68600b2390 2023-10-10T19:24:50.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f6867098700 1 -- 172.21.15.175:0/1940871201 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b2390 msgr2=0x7f6860155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:50.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f6867098700 1 --2- 172.21.15.175:0/1940871201 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b2390 0x7f6860155fb0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f6850009e60 tx=0x7f6850009780 comp rx=0 tx=0).stop 2023-10-10T19:24:50.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f6867098700 1 -- 172.21.15.175:0/1940871201 shutdown_connections 2023-10-10T19:24:50.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f6867098700 1 --2- 172.21.15.175:0/1940871201 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b2390 0x7f6860155fb0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:50.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f6867098700 1 -- 172.21.15.175:0/1940871201 >> 172.21.15.175:0/1940871201 conn(0x7f6860063230 msgr2=0x7f6860063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:50.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f6867098700 1 -- 172.21.15.175:0/1940871201 shutdown_connections 2023-10-10T19:24:50.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.288+0000 7f6867098700 1 -- 172.21.15.175:0/1940871201 wait complete. 2023-10-10T19:24:50.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.289+0000 7f6867098700 1 Processor -- start 2023-10-10T19:24:50.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.289+0000 7f6867098700 1 -- start start 2023-10-10T19:24:50.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f6867098700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68600aff60 0x7f68600cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:50.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f6867098700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b1fc0 0x7f6860152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:50.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f6867098700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68600b2390 0x7f6860166630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:50.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f6867098700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68600cb8e0 con 0x7f68600b1fc0 2023-10-10T19:24:50.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f6867098700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68601661a0 con 0x7f68600b2390 2023-10-10T19:24:50.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f6867098700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68600c4440 con 0x7f68600aff60 2023-10-10T19:24:50.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b1fc0 0x7f6860152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:50.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b1fc0 0x7f6860152c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52430/0 (socket says 172.21.15.175:52430) 2023-10-10T19:24:50.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 -- 172.21.15.175:0/1796555330 learned_addr learned my addr 172.21.15.175:0/1796555330 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:50.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 -- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68600aff60 msgr2=0x7f68600cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:50.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68600aff60 0x7f68600cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:50.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 -- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68600b2390 msgr2=0x7f6860166630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:50.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68600b2390 0x7f6860166630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:50.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 -- 172.21.15.175:0/1796555330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6850009a20 con 0x7f68600b1fc0 2023-10-10T19:24:50.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685ffff700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b1fc0 0x7f6860152c50 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f685000b6c0 tx=0x7f6850018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:50.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.290+0000 7f685cff9700 1 -- 172.21.15.175:0/1796555330 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850009420 con 0x7f68600b1fc0 2023-10-10T19:24:50.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.291+0000 7f685cff9700 1 -- 172.21.15.175:0/1796555330 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6850012e40 con 0x7f68600b1fc0 2023-10-10T19:24:50.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.291+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68600ca6d0 con 0x7f68600b1fc0 2023-10-10T19:24:50.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.291+0000 7f685cff9700 1 -- 172.21.15.175:0/1796555330 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685001e870 con 0x7f68600b1fc0 2023-10-10T19:24:50.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.291+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68600cac90 con 0x7f68600b1fc0 2023-10-10T19:24:50.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.292+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686008a570 con 0x7f68600b1fc0 2023-10-10T19:24:50.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.292+0000 7f685cff9700 1 -- 172.21.15.175:0/1796555330 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f685001ec60 con 0x7f68600b1fc0 2023-10-10T19:24:50.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.292+0000 7f685cff9700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6840066f30 0x7f68400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:50.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.292+0000 7f685cff9700 1 -- 172.21.15.175:0/1796555330 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f68500871b0 con 0x7f68600b1fc0 2023-10-10T19:24:50.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.292+0000 7f6864e34700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6840066f30 0x7f68400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:50.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.293+0000 7f6864e34700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6840066f30 0x7f68400693e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f68580016a0 tx=0x7f6858007b30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:50.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.296+0000 7f685cff9700 1 -- 172.21.15.175:0/1796555330 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6850053cf0 con 0x7f68600b1fc0 2023-10-10T19:24:50.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.520+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6860166310 con 0x7f68600b1fc0 2023-10-10T19:24:50.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.520+0000 7f685cff9700 1 -- 172.21.15.175:0/1796555330 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f6850057310 con 0x7f68600b1fc0 2023-10-10T19:24:50.521 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:50.521 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:50.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.522+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6840066f30 msgr2=0x7f68400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:50.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.522+0000 7f6867098700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6840066f30 0x7f68400693e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f68580016a0 tx=0x7f6858007b30 comp rx=0 tx=0).stop 2023-10-10T19:24:50.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.522+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b1fc0 msgr2=0x7f6860152c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:50.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.522+0000 7f6867098700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b1fc0 0x7f6860152c50 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f685000b6c0 tx=0x7f6850018000 comp rx=0 tx=0).stop 2023-10-10T19:24:50.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.522+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 shutdown_connections 2023-10-10T19:24:50.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.522+0000 7f6867098700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6840066f30 0x7f68400693e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:50.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.523+0000 7f6867098700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68600b2390 0x7f6860166630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:50.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.523+0000 7f6867098700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68600aff60 0x7f68600cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:50.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.523+0000 7f6867098700 1 --2- 172.21.15.175:0/1796555330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68600b1fc0 0x7f6860152c50 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:50.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.523+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 >> 172.21.15.175:0/1796555330 conn(0x7f6860063230 msgr2=0x7f68600b04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:50.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.523+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 shutdown_connections 2023-10-10T19:24:50.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:50.523+0000 7f6867098700 1 -- 172.21.15.175:0/1796555330 wait complete. 2023-10-10T19:24:50.526 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:50.535 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:24:52.536 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:52.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.664+0000 7f3a2ee57700 1 Processor -- start 2023-10-10T19:24:52.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.664+0000 7f3a2ee57700 1 -- start start 2023-10-10T19:24:52.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.664+0000 7f3a2ee57700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3a280b1fc0 0x7f3a28155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:52.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.664+0000 7f3a2ee57700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b2390 0x7f3a28068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:52.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.664+0000 7f3a2ee57700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a281574b0 con 0x7f3a28068bd0 2023-10-10T19:24:52.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.664+0000 7f3a2ee57700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a28150770 con 0x7f3a280b1fc0 2023-10-10T19:24:52.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.664+0000 7f3a2ee57700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a280afad0 con 0x7f3a280b2390 2023-10-10T19:24:52.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a2cbf3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3a280b1fc0 0x7f3a28155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:52.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a2cbf3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3a280b1fc0 0x7f3a28155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52440/0 (socket says 172.21.15.175:52440) 2023-10-10T19:24:52.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a2cbf3700 1 -- 172.21.15.175:0/397526228 learned_addr learned my addr 172.21.15.175:0/397526228 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:52.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a27fff700 1 --2- 172.21.15.175:0/397526228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b2390 0x7f3a28068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:52.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a2cbf3700 1 -- 172.21.15.175:0/397526228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b2390 msgr2=0x7f3a28068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:52.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a2cbf3700 1 --2- 172.21.15.175:0/397526228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b2390 0x7f3a28068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a2cbf3700 1 -- 172.21.15.175:0/397526228 >> v1:172.21.15.191:6789/0 conn(0x7f3a28068bd0 legacy=0x7f3a280aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:52.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a2cbf3700 1 -- 172.21.15.175:0/397526228 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a280af4c0 con 0x7f3a280b1fc0 2023-10-10T19:24:52.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a2cbf3700 1 --2- 172.21.15.175:0/397526228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b1fc0 0x7f3a28155fb0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009f20 tx=0x7f3a18009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95e4013900bef97 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:52.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a277fe700 1 -- 172.21.15.175:0/397526228 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1800e070 con 0x7f3a280b1fc0 2023-10-10T19:24:52.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a277fe700 1 -- 172.21.15.175:0/397526228 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1800bb10 con 0x7f3a280b1fc0 2023-10-10T19:24:52.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.665+0000 7f3a277fe700 1 -- 172.21.15.175:0/397526228 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a180092e0 con 0x7f3a280b1fc0 2023-10-10T19:24:52.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.666+0000 7f3a2ee57700 1 -- 172.21.15.175:0/397526228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b1fc0 msgr2=0x7f3a28155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:52.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.666+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/397526228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b1fc0 0x7f3a28155fb0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009f20 tx=0x7f3a18009840 comp rx=0 tx=0).stop 2023-10-10T19:24:52.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.666+0000 7f3a2ee57700 1 -- 172.21.15.175:0/397526228 shutdown_connections 2023-10-10T19:24:52.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.666+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/397526228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b1fc0 0x7f3a28155fb0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.666+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/397526228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b2390 0x7f3a28068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.666+0000 7f3a2ee57700 1 -- 172.21.15.175:0/397526228 >> 172.21.15.175:0/397526228 conn(0x7f3a28063230 msgr2=0x7f3a28063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:52.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.666+0000 7f3a2ee57700 1 -- 172.21.15.175:0/397526228 shutdown_connections 2023-10-10T19:24:52.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.666+0000 7f3a2ee57700 1 -- 172.21.15.175:0/397526228 wait complete. 2023-10-10T19:24:52.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.667+0000 7f3a2ee57700 1 Processor -- start 2023-10-10T19:24:52.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.667+0000 7f3a2ee57700 1 -- start start 2023-10-10T19:24:52.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2ee57700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a28068bd0 0x7f3a2814ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:52.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2ee57700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b1fc0 0x7f3a280c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:52.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2ee57700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b2390 0x7f3a2814db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:52.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2ee57700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a28156e60 con 0x7f3a280b2390 2023-10-10T19:24:52.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2ee57700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a28155a70 con 0x7f3a28068bd0 2023-10-10T19:24:52.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2ee57700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a28169fc0 con 0x7f3a280b1fc0 2023-10-10T19:24:52.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b2390 0x7f3a2814db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:52.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b2390 0x7f3a2814db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52450/0 (socket says 172.21.15.175:52450) 2023-10-10T19:24:52.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 -- 172.21.15.175:0/22423442 learned_addr learned my addr 172.21.15.175:0/22423442 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:52.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2cbf3700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a28068bd0 0x7f3a2814ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:52.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a27fff700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b1fc0 0x7f3a280c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:52.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 -- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b1fc0 msgr2=0x7f3a280c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:52.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b1fc0 0x7f3a280c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 -- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a28068bd0 msgr2=0x7f3a2814ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:52.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a28068bd0 0x7f3a2814ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 -- 172.21.15.175:0/22423442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a18009ae0 con 0x7f3a280b2390 2023-10-10T19:24:52.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.668+0000 7f3a2d3f4700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b2390 0x7f3a2814db00 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000c590 tx=0x7f3a20007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:52.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.669+0000 7f3a24ff9700 1 -- 172.21.15.175:0/22423442 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20011590 con 0x7f3a280b2390 2023-10-10T19:24:52.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.669+0000 7f3a24ff9700 1 -- 172.21.15.175:0/22423442 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a200116f0 con 0x7f3a280b2390 2023-10-10T19:24:52.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.669+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2816a2a0 con 0x7f3a280b2390 2023-10-10T19:24:52.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.669+0000 7f3a24ff9700 1 -- 172.21.15.175:0/22423442 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20011ab0 con 0x7f3a280b2390 2023-10-10T19:24:52.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.669+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a28155420 con 0x7f3a280b2390 2023-10-10T19:24:52.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.670+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a28067570 con 0x7f3a280b2390 2023-10-10T19:24:52.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.670+0000 7f3a24ff9700 1 -- 172.21.15.175:0/22423442 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3a2001b3b0 con 0x7f3a280b2390 2023-10-10T19:24:52.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.670+0000 7f3a24ff9700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a08066f30 0x7f3a080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:52.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.670+0000 7f3a24ff9700 1 -- 172.21.15.175:0/22423442 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f3a20083fa0 con 0x7f3a280b2390 2023-10-10T19:24:52.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.670+0000 7f3a2cbf3700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a08066f30 0x7f3a080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:52.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.671+0000 7f3a2cbf3700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a08066f30 0x7f3a080693e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009880 tx=0x7f3a18018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:52.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.674+0000 7f3a24ff9700 1 -- 172.21.15.175:0/22423442 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a20050ae0 con 0x7f3a280b2390 2023-10-10T19:24:52.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.899+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a28155bb0 con 0x7f3a280b2390 2023-10-10T19:24:52.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.899+0000 7f3a24ff9700 1 -- 172.21.15.175:0/22423442 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f3a20017070 con 0x7f3a280b2390 2023-10-10T19:24:52.900 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:52.900 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:52.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a08066f30 msgr2=0x7f3a080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:52.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a08066f30 0x7f3a080693e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009880 tx=0x7f3a18018000 comp rx=0 tx=0).stop 2023-10-10T19:24:52.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b2390 msgr2=0x7f3a2814db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:52.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b2390 0x7f3a2814db00 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000c590 tx=0x7f3a20007ec0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 shutdown_connections 2023-10-10T19:24:52.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a08066f30 0x7f3a080693e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a280b2390 0x7f3a2814db00 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a28068bd0 0x7f3a2814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 --2- 172.21.15.175:0/22423442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a280b1fc0 0x7f3a280c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:52.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 >> 172.21.15.175:0/22423442 conn(0x7f3a28063230 msgr2=0x7f3a28159310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:52.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 shutdown_connections 2023-10-10T19:24:52.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:52.901+0000 7f3a2ee57700 1 -- 172.21.15.175:0/22423442 wait complete. 2023-10-10T19:24:52.905 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:52.915 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:24:54.916 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:55.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.042+0000 7f2a42a80700 1 Processor -- start 2023-10-10T19:24:55.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.042+0000 7f2a42a80700 1 -- start start 2023-10-10T19:24:55.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a42a80700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2a3c14fd90 0x7f2a3c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a42a80700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2a3c08a100 0x7f2a3c0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a42a80700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c0cd040 0x7f2a3c0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a42a80700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3c0c5600 con 0x7f2a3c14fd90 2023-10-10T19:24:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a42a80700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3c067fa0 con 0x7f2a3c08a100 2023-10-10T19:24:55.046 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a42a80700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c156b70 con 0x7f2a3c0cd040 2023-10-10T19:24:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2a3c14fd90 0x7f2a3c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2a3c14fd90 0x7f2a3c159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35888/0 (socket says 172.21.15.175:35888) 2023-10-10T19:24:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 -- 172.21.15.175:0/2182864263 learned_addr learned my addr 172.21.15.175:0/2182864263 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a4227f700 1 --2- 172.21.15.175:0/2182864263 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c0cd040 0x7f2a3c0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:55.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a4127d700 1 --2- 172.21.15.175:0/2182864263 >> v2:172.21.15.191:3300/0 conn(0x7f2a3c08a100 0x7f2a3c0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 -- 172.21.15.175:0/2182864263 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c0cd040 msgr2=0x7f2a3c0c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 --2- 172.21.15.175:0/2182864263 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c0cd040 0x7f2a3c0c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 -- 172.21.15.175:0/2182864263 >> v2:172.21.15.191:3300/0 conn(0x7f2a3c08a100 msgr2=0x7f2a3c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 --2- 172.21.15.175:0/2182864263 >> v2:172.21.15.191:3300/0 conn(0x7f2a3c08a100 0x7f2a3c0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 -- 172.21.15.175:0/2182864263 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c150160 con 0x7f2a3c14fd90 2023-10-10T19:24:55.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a4127d700 1 --2- 172.21.15.175:0/2182864263 >> v2:172.21.15.191:3300/0 conn(0x7f2a3c08a100 0x7f2a3c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:24:55.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.043+0000 7f2a41a7e700 1 --2- 172.21.15.175:0/2182864263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c14fd90 0x7f2a3c159c30 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c069990 tx=0x7f2a34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358f6015599d80f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:55.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a40a7c700 1 -- 172.21.15.175:0/2182864263 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3400b860 con 0x7f2a3c14fd90 2023-10-10T19:24:55.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a40a7c700 1 -- 172.21.15.175:0/2182864263 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3400b9c0 con 0x7f2a3c14fd90 2023-10-10T19:24:55.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a40a7c700 1 -- 172.21.15.175:0/2182864263 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3400bd60 con 0x7f2a3c14fd90 2023-10-10T19:24:55.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 -- 172.21.15.175:0/2182864263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c14fd90 msgr2=0x7f2a3c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:55.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 --2- 172.21.15.175:0/2182864263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c14fd90 0x7f2a3c159c30 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c069990 tx=0x7f2a34009250 comp rx=0 tx=0).stop 2023-10-10T19:24:55.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 -- 172.21.15.175:0/2182864263 shutdown_connections 2023-10-10T19:24:55.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 --2- 172.21.15.175:0/2182864263 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c0cd040 0x7f2a3c0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 --2- 172.21.15.175:0/2182864263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c14fd90 0x7f2a3c159c30 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 --2- 172.21.15.175:0/2182864263 >> v2:172.21.15.191:3300/0 conn(0x7f2a3c08a100 0x7f2a3c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 -- 172.21.15.175:0/2182864263 >> 172.21.15.175:0/2182864263 conn(0x7f2a3c063d00 msgr2=0x7f2a3c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:55.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 -- 172.21.15.175:0/2182864263 shutdown_connections 2023-10-10T19:24:55.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.044+0000 7f2a42a80700 1 -- 172.21.15.175:0/2182864263 wait complete. 2023-10-10T19:24:55.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.045+0000 7f2a42a80700 1 Processor -- start 2023-10-10T19:24:55.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.045+0000 7f2a42a80700 1 -- start start 2023-10-10T19:24:55.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a42a80700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c08a100 0x7f2a3c152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:55.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a42a80700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a3c0cd040 0x7f2a3c14c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:55.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a42a80700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c14fd90 0x7f2a3c14ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:55.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a42a80700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c166680 con 0x7f2a3c08a100 2023-10-10T19:24:55.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a42a80700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c165ad0 con 0x7f2a3c0cd040 2023-10-10T19:24:55.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a42a80700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0a7d90 con 0x7f2a3c14fd90 2023-10-10T19:24:55.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a41a7e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c08a100 0x7f2a3c152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:55.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a41a7e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c08a100 0x7f2a3c152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35900/0 (socket says 172.21.15.175:35900) 2023-10-10T19:24:55.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a41a7e700 1 -- 172.21.15.175:0/469239825 learned_addr learned my addr 172.21.15.175:0/469239825 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:55.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a4127d700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a3c0cd040 0x7f2a3c14c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:55.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a4227f700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c14fd90 0x7f2a3c14ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:55.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a41a7e700 1 -- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c14fd90 msgr2=0x7f2a3c14ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:55.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a41a7e700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c14fd90 0x7f2a3c14ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a41a7e700 1 -- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a3c0cd040 msgr2=0x7f2a3c14c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:55.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a41a7e700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a3c0cd040 0x7f2a3c14c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.046+0000 7f2a41a7e700 1 -- 172.21.15.175:0/469239825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a340094f0 con 0x7f2a3c08a100 2023-10-10T19:24:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.047+0000 7f2a41a7e700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c08a100 0x7f2a3c152df0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f2a340090b0 tx=0x7f2a34017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.047+0000 7f2a31ffb700 1 -- 172.21.15.175:0/469239825 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34026480 con 0x7f2a3c08a100 2023-10-10T19:24:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.047+0000 7f2a31ffb700 1 -- 172.21.15.175:0/469239825 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3401f070 con 0x7f2a3c08a100 2023-10-10T19:24:55.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.047+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c165d50 con 0x7f2a3c08a100 2023-10-10T19:24:55.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.047+0000 7f2a31ffb700 1 -- 172.21.15.175:0/469239825 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34026830 con 0x7f2a3c08a100 2023-10-10T19:24:55.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.047+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c166310 con 0x7f2a3c08a100 2023-10-10T19:24:55.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.048+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3c067570 con 0x7f2a3c08a100 2023-10-10T19:24:55.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.048+0000 7f2a31ffb700 1 -- 172.21.15.175:0/469239825 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2a34026ba0 con 0x7f2a3c08a100 2023-10-10T19:24:55.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.049+0000 7f2a31ffb700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2a20066f30 0x7f2a200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:55.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.049+0000 7f2a31ffb700 1 -- 172.21.15.175:0/469239825 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f2a34021070 con 0x7f2a3c08a100 2023-10-10T19:24:55.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.049+0000 7f2a4127d700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2a20066f30 0x7f2a200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:55.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.049+0000 7f2a4127d700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2a20066f30 0x7f2a200693e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c006f00 tx=0x7f2a2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:55.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.053+0000 7f2a31ffb700 1 -- 172.21.15.175:0/469239825 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a34053530 con 0x7f2a3c08a100 2023-10-10T19:24:55.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.276+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a3c155f80 con 0x7f2a3c08a100 2023-10-10T19:24:55.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.277+0000 7f2a31ffb700 1 -- 172.21.15.175:0/469239825 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f2a34056b50 con 0x7f2a3c08a100 2023-10-10T19:24:55.278 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:55.278 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:55.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2a20066f30 msgr2=0x7f2a200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:55.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2a20066f30 0x7f2a200693e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c006f00 tx=0x7f2a2c008040 comp rx=0 tx=0).stop 2023-10-10T19:24:55.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c08a100 msgr2=0x7f2a3c152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:55.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c08a100 0x7f2a3c152df0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f2a340090b0 tx=0x7f2a34017000 comp rx=0 tx=0).stop 2023-10-10T19:24:55.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 shutdown_connections 2023-10-10T19:24:55.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2a20066f30 0x7f2a200693e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2a3c14fd90 0x7f2a3c14ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a3c08a100 0x7f2a3c152df0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 --2- 172.21.15.175:0/469239825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a3c0cd040 0x7f2a3c14c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:55.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 >> 172.21.15.175:0/469239825 conn(0x7f2a3c063d00 msgr2=0x7f2a3c0c7420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:55.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 shutdown_connections 2023-10-10T19:24:55.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:55.279+0000 7f2a42a80700 1 -- 172.21.15.175:0/469239825 wait complete. 2023-10-10T19:24:55.283 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:55.291 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:24:56.198 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:24:56.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:24:57.293 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:57.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.420+0000 7f22738e7700 1 Processor -- start 2023-10-10T19:24:57.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.420+0000 7f22738e7700 1 -- start start 2023-10-10T19:24:57.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.420+0000 7f22738e7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f226c156c30 0x7f226c0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:57.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f22738e7700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c0c6df0 con 0x7f226c06bf00 2023-10-10T19:24:57.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f22738e7700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c0c6920 con 0x7f226c0c5e40 2023-10-10T19:24:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f22738e7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226c1560d0 con 0x7f226c156c30 2023-10-10T19:24:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f2270e82700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f226c06bf00 0x7f226c156520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58650/0 (socket says 172.21.15.175:58650) 2023-10-10T19:24:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f2270e82700 1 -- 172.21.15.175:0/18425906 learned_addr learned my addr 172.21.15.175:0/18425906 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:24:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f2271e84700 1 --2- 172.21.15.175:0/18425906 >> v2:172.21.15.191:3300/0 conn(0x7f226c156c30 0x7f226c0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:57.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305793356 0 0) 0x7f226c0c6df0 con 0x7f226c06bf00 2023-10-10T19:24:57.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2254003570 con 0x7f226c06bf00 2023-10-10T19:24:57.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1408659908 0 0) 0x7f2254003570 con 0x7f226c06bf00 2023-10-10T19:24:57.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 >> v2:172.21.15.191:3300/0 conn(0x7f226c156c30 msgr2=0x7f226c0c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:57.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 --2- 172.21.15.175:0/18425906 >> v2:172.21.15.191:3300/0 conn(0x7f226c156c30 0x7f226c0c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 >> v1:172.21.15.191:6789/0 conn(0x7f226c0c5e40 legacy=0x7f226c0c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:57.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c1597e0 con 0x7f226c06bf00 2023-10-10T19:24:57.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f2260001470 con 0x7f226c06bf00 2023-10-10T19:24:57.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 >> v1:172.21.15.175:6789/0 conn(0x7f226c06bf00 legacy=0x7f226c156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:57.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254003a60 0x7f2254003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:57.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2254004810 0x7f2254004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:57.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22540051c0 0x7f225400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:57.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c1560d0 con 0x7f2254004810 2023-10-10T19:24:57.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225400c0d0 con 0x7f2254003a60 2023-10-10T19:24:57.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225400c290 con 0x7f22540051c0 2023-10-10T19:24:57.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.421+0000 7f2270e82700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2254004810 0x7f2254004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:57.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2271683700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254003a60 0x7f2254003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:57.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2271e84700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22540051c0 0x7f225400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:57.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2270e82700 1 -- 172.21.15.175:0/18425906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22540051c0 msgr2=0x7f225400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:57.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2270e82700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22540051c0 0x7f225400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2270e82700 1 -- 172.21.15.175:0/18425906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254003a60 msgr2=0x7f2254003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:57.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2270e82700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254003a60 0x7f2254003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2270e82700 1 -- 172.21.15.175:0/18425906 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2254003570 con 0x7f2254004810 2023-10-10T19:24:57.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2271e84700 1 -- 172.21.15.175:0/18425906 reap_dead start 2023-10-10T19:24:57.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f2270e82700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2254004810 0x7f2254004c80 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f226000c3c0 tx=0x7f2260001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53318c21e70943c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:57.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226000cc90 con 0x7f2254004810 2023-10-10T19:24:57.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226000cdf0 con 0x7f2254004810 2023-10-10T19:24:57.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.422+0000 7f226bfff700 1 -- 172.21.15.175:0/18425906 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260005b50 con 0x7f2254004810 2023-10-10T19:24:57.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.423+0000 7f22738e7700 1 -- 172.21.15.175:0/18425906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2254004810 msgr2=0x7f2254004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:57.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.423+0000 7f22738e7700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2254004810 0x7f2254004c80 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f226000c3c0 tx=0x7f2260001e50 comp rx=0 tx=0).stop 2023-10-10T19:24:57.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.423+0000 7f22738e7700 1 -- 172.21.15.175:0/18425906 shutdown_connections 2023-10-10T19:24:57.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.423+0000 7f22738e7700 1 --2- 172.21.15.175:0/18425906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2254004810 0x7f2254004c80 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.423+0000 7f22738e7700 1 -- 172.21.15.175:0/18425906 >> 172.21.15.175:0/18425906 conn(0x7f226c068900 msgr2=0x7f226c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:57.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.423+0000 7f22738e7700 1 -- 172.21.15.175:0/18425906 shutdown_connections 2023-10-10T19:24:57.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.423+0000 7f22738e7700 1 -- 172.21.15.175:0/18425906 wait complete. 2023-10-10T19:24:57.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.424+0000 7f22738e7700 1 Processor -- start 2023-10-10T19:24:57.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.424+0000 7f22738e7700 1 -- start start 2023-10-10T19:24:57.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22738e7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254004810 0x7f226c0ba200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:57.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22738e7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f226c166de0 0x7f226c154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:57.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22738e7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f226c0c9ca0 0x7f226c153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:57.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22738e7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c1660f0 con 0x7f226c166de0 2023-10-10T19:24:57.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22738e7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c166950 con 0x7f2254004810 2023-10-10T19:24:57.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22738e7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c155130 con 0x7f226c0c9ca0 2023-10-10T19:24:57.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f226c166de0 0x7f226c154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:57.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f226c166de0 0x7f226c154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35918/0 (socket says 172.21.15.175:35918) 2023-10-10T19:24:57.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 -- 172.21.15.175:0/276283980 learned_addr learned my addr 172.21.15.175:0/276283980 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:57.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2271683700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254004810 0x7f226c0ba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:57.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2271e84700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f226c0c9ca0 0x7f226c153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:57.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 -- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f226c0c9ca0 msgr2=0x7f226c153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:57.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f226c0c9ca0 0x7f226c153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 -- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254004810 msgr2=0x7f226c0ba200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:57.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254004810 0x7f226c0ba200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 -- 172.21.15.175:0/276283980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c150f30 con 0x7f226c166de0 2023-10-10T19:24:57.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2271683700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254004810 0x7f226c0ba200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:24:57.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f2270e82700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f226c166de0 0x7f226c154bf0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f225400c0a0 tx=0x7f2260015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:57.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22697fa700 1 -- 172.21.15.175:0/276283980 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260001430 con 0x7f226c166de0 2023-10-10T19:24:57.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22697fa700 1 -- 172.21.15.175:0/276283980 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22600070d0 con 0x7f226c166de0 2023-10-10T19:24:57.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c151210 con 0x7f226c166de0 2023-10-10T19:24:57.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22697fa700 1 -- 172.21.15.175:0/276283980 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260007470 con 0x7f226c166de0 2023-10-10T19:24:57.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.425+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c1665e0 con 0x7f226c166de0 2023-10-10T19:24:57.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.426+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c067570 con 0x7f226c166de0 2023-10-10T19:24:57.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.427+0000 7f22697fa700 1 -- 172.21.15.175:0/276283980 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2260012400 con 0x7f226c166de0 2023-10-10T19:24:57.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.427+0000 7f22697fa700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2258066f30 0x7f22580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:57.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.427+0000 7f22697fa700 1 -- 172.21.15.175:0/276283980 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f2260085560 con 0x7f226c166de0 2023-10-10T19:24:57.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.427+0000 7f2271683700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2258066f30 0x7f22580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:57.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.428+0000 7f2271683700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2258066f30 0x7f22580693e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f225c0042e0 tx=0x7f225c001a70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:57.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.431+0000 7f22697fa700 1 -- 172.21.15.175:0/276283980 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f22600520a0 con 0x7f226c166de0 2023-10-10T19:24:57.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.655+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f226c166a90 con 0x7f226c166de0 2023-10-10T19:24:57.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.656+0000 7f22697fa700 1 -- 172.21.15.175:0/276283980 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f22600556c0 con 0x7f226c166de0 2023-10-10T19:24:57.657 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:24:57.657 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":290,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":8202,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":290}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:01.041408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":7252,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7252":{"gid":7252,"name":"f","rank":0,"incarnation":271,"state":"up:active","state_seq":20,"addr":"172.21.15.191:6839/2393527600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2393527600},{"type":"v1","addr":"172.21.15.191:6839","nonce":2393527600}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:24:57.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2258066f30 msgr2=0x7f22580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:57.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2258066f30 0x7f22580693e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f225c0042e0 tx=0x7f225c001a70 comp rx=0 tx=0).stop 2023-10-10T19:24:57.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f226c166de0 msgr2=0x7f226c154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:57.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f226c166de0 0x7f226c154bf0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f225400c0a0 tx=0x7f2260015000 comp rx=0 tx=0).stop 2023-10-10T19:24:57.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 shutdown_connections 2023-10-10T19:24:57.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2258066f30 0x7f22580693e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f226c0c9ca0 0x7f226c153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2254004810 0x7f226c0ba200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 --2- 172.21.15.175:0/276283980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f226c166de0 0x7f226c154bf0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:57.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 >> 172.21.15.175:0/276283980 conn(0x7f226c068900 msgr2=0x7f226c14f900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:57.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 shutdown_connections 2023-10-10T19:24:57.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:57.658+0000 7f22738e7700 1 -- 172.21.15.175:0/276283980 wait complete. 2023-10-10T19:24:57.661 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 290 2023-10-10T19:24:57.670 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-10T19:24:59.671 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:24:59.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.798+0000 7f14b2345700 1 Processor -- start 2023-10-10T19:24:59.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.798+0000 7f14b2345700 1 -- start start 2023-10-10T19:24:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.798+0000 7f14b2345700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f14ac0b1fc0 0x7f14ac155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.798+0000 7f14b2345700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac0b2390 0x7f14ac068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.798+0000 7f14b2345700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac1574b0 con 0x7f14ac068bd0 2023-10-10T19:24:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.798+0000 7f14b2345700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ac150770 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.798+0000 7f14b2345700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0afad0 con 0x7f14ac0b2390 2023-10-10T19:24:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.798+0000 7f14abfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f14ac0b1fc0 0x7f14ac155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14abfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f14ac0b1fc0 0x7f14ac155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35934/0 (socket says 172.21.15.175:35934) 2023-10-10T19:24:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14abfff700 1 -- 172.21.15.175:0/1951118778 learned_addr learned my addr 172.21.15.175:0/1951118778 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14b08e2700 1 --1- 172.21.15.175:0/1951118778 >> v1:172.21.15.175:6789/0 conn(0x7f14ac068bd0 0x7f14ac0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58666/0 (socket says 172.21.15.175:58666) 2023-10-10T19:24:59.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14ab7fe700 1 --2- 172.21.15.175:0/1951118778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac0b2390 0x7f14ac068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14abfff700 1 -- 172.21.15.175:0/1951118778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac0b2390 msgr2=0x7f14ac068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:24:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14abfff700 1 --2- 172.21.15.175:0/1951118778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac0b2390 0x7f14ac068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14abfff700 1 -- 172.21.15.175:0/1951118778 >> v1:172.21.15.175:6789/0 conn(0x7f14ac068bd0 legacy=0x7f14ac0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:59.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14aaffd700 1 -- 172.21.15.175:0/1951118778 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597739787 0 0) 0x7f14ac1574b0 con 0x7f14ac068bd0 2023-10-10T19:24:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14abfff700 1 -- 172.21.15.175:0/1951118778 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac0af4c0 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14abfff700 1 --2- 172.21.15.175:0/1951118778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac155fb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f149c0099b0 tx=0x7f149c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb71e87978e62d15 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14aaffd700 1 -- 172.21.15.175:0/1951118778 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c00b8f0 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14aaffd700 1 -- 172.21.15.175:0/1951118778 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c00ba50 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.799+0000 7f14aaffd700 1 -- 172.21.15.175:0/1951118778 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c00bdf0 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.800+0000 7f14b2345700 1 -- 172.21.15.175:0/1951118778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 msgr2=0x7f14ac155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:59.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.800+0000 7f14b2345700 1 --2- 172.21.15.175:0/1951118778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac155fb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f149c0099b0 tx=0x7f149c009250 comp rx=0 tx=0).stop 2023-10-10T19:24:59.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.800+0000 7f14b2345700 1 -- 172.21.15.175:0/1951118778 shutdown_connections 2023-10-10T19:24:59.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.800+0000 7f14b2345700 1 --2- 172.21.15.175:0/1951118778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac155fb0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:59.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.800+0000 7f14b2345700 1 --2- 172.21.15.175:0/1951118778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac0b2390 0x7f14ac068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:59.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.800+0000 7f14b2345700 1 -- 172.21.15.175:0/1951118778 >> 172.21.15.175:0/1951118778 conn(0x7f14ac063230 msgr2=0x7f14ac063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:24:59.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.800+0000 7f14b2345700 1 -- 172.21.15.175:0/1951118778 shutdown_connections 2023-10-10T19:24:59.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.800+0000 7f14b2345700 1 -- 172.21.15.175:0/1951118778 wait complete. 2023-10-10T19:24:59.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.801+0000 7f14b2345700 1 Processor -- start 2023-10-10T19:24:59.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.801+0000 7f14b2345700 1 -- start start 2023-10-10T19:24:59.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14b2345700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac068bd0 0x7f14ac0c02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:59.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14b2345700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac167000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:59.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14b2345700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac0b2390 0x7f14ac167540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:59.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14b2345700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac159d90 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14b2345700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac068720 con 0x7f14ac0b2390 2023-10-10T19:24:59.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14b2345700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac158ea0 con 0x7f14ac068bd0 2023-10-10T19:24:59.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac167000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:59.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac167000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35948/0 (socket says 172.21.15.175:35948) 2023-10-10T19:24:59.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 -- 172.21.15.175:0/46093743 learned_addr learned my addr 172.21.15.175:0/46093743 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:24:59.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14b08e2700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac0b2390 0x7f14ac167540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:59.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14abfff700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac068bd0 0x7f14ac0c02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:59.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 -- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac068bd0 msgr2=0x7f14ac0c02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:59.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac068bd0 0x7f14ac0c02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:59.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 -- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac0b2390 msgr2=0x7f14ac167540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:24:59.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac0b2390 0x7f14ac167540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:24:59.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 -- 172.21.15.175:0/46093743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c0094f0 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14abfff700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac068bd0 0x7f14ac0c02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:24:59.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14ab7fe700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac167000 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f14a000d890 tx=0x7f14a0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:59.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f1493fff700 1 -- 172.21.15.175:0/46093743 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a00125f0 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f1493fff700 1 -- 172.21.15.175:0/46093743 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a0012770 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f1493fff700 1 -- 172.21.15.175:0/46093743 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0012b30 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.802+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac159180 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.803+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac159740 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.804+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ac067570 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.804+0000 7f1493fff700 1 -- 172.21.15.175:0/46093743 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f14a001e460 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.804+0000 7f1493fff700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1498066f60 0x7f1498069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:24:59.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.804+0000 7f1493fff700 1 -- 172.21.15.175:0/46093743 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f14a0085330 con 0x7f14ac0b1fc0 2023-10-10T19:24:59.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.805+0000 7f14abfff700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1498066f60 0x7f1498069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:24:59.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.805+0000 7f14abfff700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1498066f60 0x7f1498069410 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f149c0099b0 tx=0x7f149c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:24:59.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:24:59.809+0000 7f1493fff700 1 -- 172.21.15.175:0/46093743 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f14a0051e20 con 0x7f14ac0b1fc0 2023-10-10T19:25:00.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.034+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14ac069060 con 0x7f14ac0b1fc0 2023-10-10T19:25:00.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.034+0000 7f1493fff700 1 -- 172.21.15.175:0/46093743 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 291 v291) v1 ==== 95+0+5417 (secure 0 0 0) 0x7f14a0055440 con 0x7f14ac0b1fc0 2023-10-10T19:25:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:00.036 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":291,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":291,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:58.864159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:00.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1498066f60 msgr2=0x7f1498069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1498066f60 0x7f1498069410 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f149c0099b0 tx=0x7f149c017000 comp rx=0 tx=0).stop 2023-10-10T19:25:00.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 msgr2=0x7f14ac167000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac167000 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f14a000d890 tx=0x7f14a0008040 comp rx=0 tx=0).stop 2023-10-10T19:25:00.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 shutdown_connections 2023-10-10T19:25:00.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1498066f60 0x7f1498069410 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac0b2390 0x7f14ac167540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14ac068bd0 0x7f14ac0c02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 --2- 172.21.15.175:0/46093743 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14ac0b1fc0 0x7f14ac167000 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.036+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 >> 172.21.15.175:0/46093743 conn(0x7f14ac063230 msgr2=0x7f14ac0637a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:00.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.037+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 shutdown_connections 2023-10-10T19:25:00.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.037+0000 7f14b2345700 1 -- 172.21.15.175:0/46093743 wait complete. 2023-10-10T19:25:00.042 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 291 2023-10-10T19:25:00.049 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-10T19:25:00.050 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:25:00.051 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:00.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.165+0000 7f1358a25700 1 Processor -- start 2023-10-10T19:25:00.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.165+0000 7f1358a25700 1 -- start start 2023-10-10T19:25:00.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1358a25700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:00.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1358a25700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1354067fa0 con 0x7f13540cd040 2023-10-10T19:25:00.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1358a25700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1354156b70 con 0x7f135408a100 2023-10-10T19:25:00.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1358a25700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13540c5600 con 0x7f135414fd90 2023-10-10T19:25:00.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f135359e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f13540cd040 0x7f13540684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:58682/0 (socket says 172.21.15.175:58682) 2023-10-10T19:25:00.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f135359e700 1 -- 172.21.15.175:0/2930835421 learned_addr learned my addr 172.21.15.175:0/2930835421 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:00.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1352d9d700 1 --2- 172.21.15.175:0/2930835421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:00.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1351d9b700 1 -- 172.21.15.175:0/2930835421 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372103845 0 0) 0x7f1354067fa0 con 0x7f13540cd040 2023-10-10T19:25:00.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1351d9b700 1 -- 172.21.15.175:0/2930835421 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c003570 con 0x7f13540cd040 2023-10-10T19:25:00.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1351d9b700 1 -- 172.21.15.175:0/2930835421 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587027799 0 0) 0x7f1354156b70 con 0x7f135408a100 2023-10-10T19:25:00.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1351d9b700 1 -- 172.21.15.175:0/2930835421 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1354067fa0 con 0x7f135408a100 2023-10-10T19:25:00.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1352d9d700 1 -- 172.21.15.175:0/2930835421 >> v1:172.21.15.191:6789/0 conn(0x7f135408a100 legacy=0x7f13540c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1352d9d700 1 -- 172.21.15.175:0/2930835421 >> v1:172.21.15.175:6789/0 conn(0x7f13540cd040 legacy=0x7f13540684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1352d9d700 1 -- 172.21.15.175:0/2930835421 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1354150160 con 0x7f135414fd90 2023-10-10T19:25:00.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.166+0000 7f1352d9d700 1 --2- 172.21.15.175:0/2930835421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354159c30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1344009910 tx=0x7f13440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1b2cf6f1dd92aac server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:00.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1351d9b700 1 -- 172.21.15.175:0/2930835421 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344011630 con 0x7f135414fd90 2023-10-10T19:25:00.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1351d9b700 1 -- 172.21.15.175:0/2930835421 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1344011790 con 0x7f135414fd90 2023-10-10T19:25:00.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1351d9b700 1 -- 172.21.15.175:0/2930835421 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344011b30 con 0x7f135414fd90 2023-10-10T19:25:00.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1358a25700 1 -- 172.21.15.175:0/2930835421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 msgr2=0x7f1354159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1358a25700 1 --2- 172.21.15.175:0/2930835421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354159c30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1344009910 tx=0x7f13440091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1358a25700 1 -- 172.21.15.175:0/2930835421 shutdown_connections 2023-10-10T19:25:00.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1358a25700 1 --2- 172.21.15.175:0/2930835421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354159c30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1358a25700 1 -- 172.21.15.175:0/2930835421 >> 172.21.15.175:0/2930835421 conn(0x7f1354063d00 msgr2=0x7f13540cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:00.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1358a25700 1 -- 172.21.15.175:0/2930835421 shutdown_connections 2023-10-10T19:25:00.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.167+0000 7f1358a25700 1 -- 172.21.15.175:0/2930835421 wait complete. 2023-10-10T19:25:00.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.168+0000 7f1358a25700 1 Processor -- start 2023-10-10T19:25:00.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.168+0000 7f1358a25700 1 -- start start 2023-10-10T19:25:00.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1358a25700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f135408a100 0x7f1354158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:00.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1358a25700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13540cd040 0x7f13541527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:00.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1358a25700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:00.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1358a25700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1354158eb0 con 0x7f135408a100 2023-10-10T19:25:00.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1358a25700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13541584e0 con 0x7f13540cd040 2023-10-10T19:25:00.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1358a25700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13540c6ef0 con 0x7f135414fd90 2023-10-10T19:25:00.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1352d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f135408a100 0x7f1354158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:00.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1352d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f135408a100 0x7f1354158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35952/0 (socket says 172.21.15.175:35952) 2023-10-10T19:25:00.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1352d9d700 1 -- 172.21.15.175:0/587685339 learned_addr learned my addr 172.21.15.175:0/587685339 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:00.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f135359e700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:00.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f135259c700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13540cd040 0x7f13541527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:00.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1352d9d700 1 -- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 msgr2=0x7f1354154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1352d9d700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1352d9d700 1 -- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13540cd040 msgr2=0x7f13541527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1352d9d700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13540cd040 0x7f13541527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.169+0000 7f1352d9d700 1 -- 172.21.15.175:0/587685339 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1344009420 con 0x7f135408a100 2023-10-10T19:25:00.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.170+0000 7f1352d9d700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f135408a100 0x7f1354158970 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f1354064c20 tx=0x7f1344017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:00.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.170+0000 7f133affd700 1 -- 172.21.15.175:0/587685339 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134401f070 con 0x7f135408a100 2023-10-10T19:25:00.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.170+0000 7f133affd700 1 -- 172.21.15.175:0/587685339 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1344002f70 con 0x7f135408a100 2023-10-10T19:25:00.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.170+0000 7f133affd700 1 -- 172.21.15.175:0/587685339 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344011900 con 0x7f135408a100 2023-10-10T19:25:00.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.170+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135414b810 con 0x7f135408a100 2023-10-10T19:25:00.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.170+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135414bdd0 con 0x7f135408a100 2023-10-10T19:25:00.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.171+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1354067570 con 0x7f135408a100 2023-10-10T19:25:00.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.171+0000 7f133affd700 1 -- 172.21.15.175:0/587685339 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1344023070 con 0x7f135408a100 2023-10-10T19:25:00.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.171+0000 7f133affd700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1340066f30 0x7f13400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:00.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.172+0000 7f133affd700 1 -- 172.21.15.175:0/587685339 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f1344085eb0 con 0x7f135408a100 2023-10-10T19:25:00.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.172+0000 7f135259c700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1340066f30 0x7f13400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:00.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.172+0000 7f135259c700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1340066f30 0x7f13400693e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f13541534e0 tx=0x7f1348007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:00.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.176+0000 7f133affd700 1 -- 172.21.15.175:0/587685339 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13440529a0 con 0x7f135408a100 2023-10-10T19:25:00.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.400+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1354155f80 con 0x7f135408a100 2023-10-10T19:25:00.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.401+0000 7f133affd700 1 -- 172.21.15.175:0/587685339 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 291 v291) v1 ==== 95+0+5417 (secure 0 0 0) 0x7f134402a090 con 0x7f135408a100 2023-10-10T19:25:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:00.401 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":291,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":291,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:58.864159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:00.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1340066f30 msgr2=0x7f13400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1340066f30 0x7f13400693e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f13541534e0 tx=0x7f1348007d50 comp rx=0 tx=0).stop 2023-10-10T19:25:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f135408a100 msgr2=0x7f1354158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f135408a100 0x7f1354158970 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f1354064c20 tx=0x7f1344017000 comp rx=0 tx=0).stop 2023-10-10T19:25:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 shutdown_connections 2023-10-10T19:25:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1340066f30 0x7f13400693e0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f135414fd90 0x7f1354154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f135408a100 0x7f1354158970 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 --2- 172.21.15.175:0/587685339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13540cd040 0x7f13541527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 >> 172.21.15.175:0/587685339 conn(0x7f1354063d00 msgr2=0x7f135414d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 shutdown_connections 2023-10-10T19:25:00.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:00.403+0000 7f1358a25700 1 -- 172.21.15.175:0/587685339 wait complete. 2023-10-10T19:25:00.406 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 291 2023-10-10T19:25:00.416 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 291, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 291, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_7384": { "addr": "172.21.15.191:6837/2914004704", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 2914004704, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2914004704, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7384, "incarnation": 278, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 23 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 117 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 129, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:24:58.864159+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 7401, "mds_2": 8049, "mds_3": 7384 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:25:02.417 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:02.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.543+0000 7f9fa5c10700 1 Processor -- start 2023-10-10T19:25:02.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.543+0000 7f9fa5c10700 1 -- start start 2023-10-10T19:25:02.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa5c10700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9fa006cf90 0x7f9fa006d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:02.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa5c10700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa01569a0 0x7f9fa0160980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:02.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa5c10700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa006d8a0 con 0x7f9fa00c0f70 2023-10-10T19:25:02.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa5c10700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa00c0990 con 0x7f9fa006cf90 2023-10-10T19:25:02.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa5c10700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa01595e0 con 0x7f9fa01569a0 2023-10-10T19:25:02.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa540f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa01569a0 0x7f9fa0160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:02.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa540f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa01569a0 0x7f9fa0160980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:39604/0 (socket says 172.21.15.175:39604) 2023-10-10T19:25:02.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa540f700 1 -- 172.21.15.175:0/3119917762 learned_addr learned my addr 172.21.15.175:0/3119917762 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:02.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.544+0000 7f9fa4c0e700 1 --2- 172.21.15.175:0/3119917762 >> v2:172.21.15.191:3300/0 conn(0x7f9fa006cf90 0x7f9fa006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:02.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9fa540f700 1 -- 172.21.15.175:0/3119917762 >> v2:172.21.15.191:3300/0 conn(0x7f9fa006cf90 msgr2=0x7f9fa006d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:02.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9fa540f700 1 --2- 172.21.15.175:0/3119917762 >> v2:172.21.15.191:3300/0 conn(0x7f9fa006cf90 0x7f9fa006d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9fa540f700 1 -- 172.21.15.175:0/3119917762 >> v1:172.21.15.191:6789/0 conn(0x7f9fa00c0f70 legacy=0x7f9fa0156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:02.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9fa540f700 1 -- 172.21.15.175:0/3119917762 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa015a440 con 0x7f9fa01569a0 2023-10-10T19:25:02.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9fa540f700 1 --2- 172.21.15.175:0/3119917762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa01569a0 0x7f9fa0160980 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9f980099b0 tx=0x7f9f98009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe30a47e5e539c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:02.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9f9f7fe700 1 -- 172.21.15.175:0/3119917762 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98009ea0 con 0x7f9fa01569a0 2023-10-10T19:25:02.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9f9f7fe700 1 -- 172.21.15.175:0/3119917762 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9800b860 con 0x7f9fa01569a0 2023-10-10T19:25:02.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9f9f7fe700 1 -- 172.21.15.175:0/3119917762 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9800bc00 con 0x7f9fa01569a0 2023-10-10T19:25:02.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3119917762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa01569a0 msgr2=0x7f9fa0160980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:02.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.545+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3119917762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa01569a0 0x7f9fa0160980 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9f980099b0 tx=0x7f9f98009250 comp rx=0 tx=0).stop 2023-10-10T19:25:02.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.546+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3119917762 shutdown_connections 2023-10-10T19:25:02.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.546+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3119917762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa01569a0 0x7f9fa0160980 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.546+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3119917762 >> v2:172.21.15.191:3300/0 conn(0x7f9fa006cf90 0x7f9fa006d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.546+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3119917762 >> 172.21.15.175:0/3119917762 conn(0x7f9fa006b310 msgr2=0x7f9fa006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:02.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.546+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3119917762 shutdown_connections 2023-10-10T19:25:02.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.546+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3119917762 wait complete. 2023-10-10T19:25:02.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.547+0000 7f9fa5c10700 1 Processor -- start 2023-10-10T19:25:02.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.547+0000 7f9fa5c10700 1 -- start start 2023-10-10T19:25:02.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.547+0000 7f9fa5c10700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa006cf90 0x7f9fa0154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:02.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.547+0000 7f9fa5c10700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa00c0f70 0x7f9fa016a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:02.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.547+0000 7f9fa5c10700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fa01569a0 0x7f9fa01669a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:02.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.547+0000 7f9fa5c10700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa0169a10 con 0x7f9fa01569a0 2023-10-10T19:25:02.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.547+0000 7f9fa5c10700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa0169cf0 con 0x7f9fa00c0f70 2023-10-10T19:25:02.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.547+0000 7f9fa5c10700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa01576d0 con 0x7f9fa006cf90 2023-10-10T19:25:02.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fa01569a0 0x7f9fa01669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:02.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fa01569a0 0x7f9fa01669a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35966/0 (socket says 172.21.15.175:35966) 2023-10-10T19:25:02.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 -- 172.21.15.175:0/3100831454 learned_addr learned my addr 172.21.15.175:0/3100831454 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:02.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9f9ffff700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa00c0f70 0x7f9fa016a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:02.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa4c0e700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa006cf90 0x7f9fa0154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:02.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 -- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa006cf90 msgr2=0x7f9fa0154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:02.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa006cf90 0x7f9fa0154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 -- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa00c0f70 msgr2=0x7f9fa016a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:02.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa00c0f70 0x7f9fa016a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 -- 172.21.15.175:0/3100831454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f980094f0 con 0x7f9fa01569a0 2023-10-10T19:25:02.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa540f700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fa01569a0 0x7f9fa01669a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0159720 tx=0x7f9f98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:02.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9f9cff9700 1 -- 172.21.15.175:0/3100831454 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98011420 con 0x7f9fa01569a0 2023-10-10T19:25:02.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9f9cff9700 1 -- 172.21.15.175:0/3100831454 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f98011580 con 0x7f9fa01569a0 2023-10-10T19:25:02.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa0157970 con 0x7f9fa01569a0 2023-10-10T19:25:02.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9f9cff9700 1 -- 172.21.15.175:0/3100831454 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98011940 con 0x7f9fa01569a0 2023-10-10T19:25:02.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.548+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa0157c30 con 0x7f9fa01569a0 2023-10-10T19:25:02.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.549+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa0067570 con 0x7f9fa01569a0 2023-10-10T19:25:02.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.550+0000 7f9f9cff9700 1 -- 172.21.15.175:0/3100831454 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9f98011cf0 con 0x7f9fa01569a0 2023-10-10T19:25:02.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.550+0000 7f9f9cff9700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9f8c066f30 0x7f9f8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:02.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.550+0000 7f9f9cff9700 1 -- 172.21.15.175:0/3100831454 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f9f98086200 con 0x7f9fa01569a0 2023-10-10T19:25:02.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.550+0000 7f9fa4c0e700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9f8c066f30 0x7f9f8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:02.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.551+0000 7f9fa4c0e700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9f8c066f30 0x7f9f8c0693e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000f5b0 tx=0x7f9f9000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:02.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.554+0000 7f9f9cff9700 1 -- 172.21.15.175:0/3100831454 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f98052d70 con 0x7f9fa01569a0 2023-10-10T19:25:02.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.779+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fa0169e30 con 0x7f9fa01569a0 2023-10-10T19:25:02.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.780+0000 7f9f9cff9700 1 -- 172.21.15.175:0/3100831454 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 291 v291) v1 ==== 95+0+5417 (secure 0 0 0) 0x7f9f98056390 con 0x7f9fa01569a0 2023-10-10T19:25:02.781 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:02.781 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":291,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":291,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:58.864159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:02.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9f8c066f30 msgr2=0x7f9f8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9f8c066f30 0x7f9f8c0693e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000f5b0 tx=0x7f9f9000b040 comp rx=0 tx=0).stop 2023-10-10T19:25:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fa01569a0 msgr2=0x7f9fa01669a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fa01569a0 0x7f9fa01669a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0159720 tx=0x7f9f98017000 comp rx=0 tx=0).stop 2023-10-10T19:25:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 shutdown_connections 2023-10-10T19:25:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9f8c066f30 0x7f9f8c0693e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fa01569a0 0x7f9fa01669a0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fa006cf90 0x7f9fa0154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 --2- 172.21.15.175:0/3100831454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa00c0f70 0x7f9fa016a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 >> 172.21.15.175:0/3100831454 conn(0x7f9fa006b310 msgr2=0x7f9fa01558d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 shutdown_connections 2023-10-10T19:25:02.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:02.782+0000 7f9fa5c10700 1 -- 172.21.15.175:0/3100831454 wait complete. 2023-10-10T19:25:02.785 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 291 2023-10-10T19:25:02.794 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:04.796 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:04.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.913+0000 7eff633f1700 1 Processor -- start 2023-10-10T19:25:04.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.913+0000 7eff633f1700 1 -- start start 2023-10-10T19:25:04.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.914+0000 7eff633f1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7eff5c0b2390 0x7eff5c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:04.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.914+0000 7eff633f1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c1574b0 con 0x7eff5c0aff60 2023-10-10T19:25:04.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.914+0000 7eff633f1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c0afad0 con 0x7eff5c0b1fc0 2023-10-10T19:25:04.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.914+0000 7eff633f1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5c150770 con 0x7eff5c0b2390 2023-10-10T19:25:04.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.914+0000 7eff6198e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7eff5c0aff60 0x7eff5c159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53154/0 (socket says 172.21.15.175:53154) 2023-10-10T19:25:04.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.914+0000 7eff6198e700 1 -- 172.21.15.175:0/2954351161 learned_addr learned my addr 172.21.15.175:0/2954351161 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:04.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.914+0000 7eff6098c700 1 --2- 172.21.15.175:0/2954351161 >> v2:172.21.15.191:3300/0 conn(0x7eff5c0b2390 0x7eff5c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:04.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff53fff700 1 -- 172.21.15.175:0/2954351161 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023702769 0 0) 0x7eff5c1574b0 con 0x7eff5c0aff60 2023-10-10T19:25:04.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff53fff700 1 -- 172.21.15.175:0/2954351161 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff48003570 con 0x7eff5c0aff60 2023-10-10T19:25:04.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff6098c700 1 -- 172.21.15.175:0/2954351161 >> v1:172.21.15.191:6789/0 conn(0x7eff5c0b1fc0 legacy=0x7eff5c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:04.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff6098c700 1 -- 172.21.15.175:0/2954351161 >> v1:172.21.15.175:6789/0 conn(0x7eff5c0aff60 legacy=0x7eff5c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:04.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff6098c700 1 -- 172.21.15.175:0/2954351161 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c0af4c0 con 0x7eff5c0b2390 2023-10-10T19:25:04.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff6098c700 1 --2- 172.21.15.175:0/2954351161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 0x7eff5c155fb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0098e0 tx=0x7eff4c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feff382c4f88da10 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:04.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff53fff700 1 -- 172.21.15.175:0/2954351161 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c00b6d0 con 0x7eff5c0b2390 2023-10-10T19:25:04.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff53fff700 1 -- 172.21.15.175:0/2954351161 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c00b830 con 0x7eff5c0b2390 2023-10-10T19:25:04.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.915+0000 7eff53fff700 1 -- 172.21.15.175:0/2954351161 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c00bbd0 con 0x7eff5c0b2390 2023-10-10T19:25:04.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.916+0000 7eff633f1700 1 -- 172.21.15.175:0/2954351161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 msgr2=0x7eff5c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:04.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.916+0000 7eff633f1700 1 --2- 172.21.15.175:0/2954351161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 0x7eff5c155fb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0098e0 tx=0x7eff4c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:04.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.916+0000 7eff633f1700 1 -- 172.21.15.175:0/2954351161 shutdown_connections 2023-10-10T19:25:04.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.916+0000 7eff633f1700 1 --2- 172.21.15.175:0/2954351161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 0x7eff5c155fb0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:04.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.916+0000 7eff633f1700 1 -- 172.21.15.175:0/2954351161 >> 172.21.15.175:0/2954351161 conn(0x7eff5c063230 msgr2=0x7eff5c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:04.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.916+0000 7eff633f1700 1 -- 172.21.15.175:0/2954351161 shutdown_connections 2023-10-10T19:25:04.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.916+0000 7eff633f1700 1 -- 172.21.15.175:0/2954351161 wait complete. 2023-10-10T19:25:04.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.917+0000 7eff633f1700 1 Processor -- start 2023-10-10T19:25:04.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.917+0000 7eff633f1700 1 -- start start 2023-10-10T19:25:04.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff633f1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7eff5c0aff60 0x7eff5c15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:04.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff633f1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff5c0b1fc0 0x7eff5c149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:04.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff633f1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 0x7eff5c1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:04.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff633f1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c149030 con 0x7eff5c0aff60 2023-10-10T19:25:04.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff633f1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c08a5c0 con 0x7eff5c0b2390 2023-10-10T19:25:04.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff633f1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c14c160 con 0x7eff5c0b1fc0 2023-10-10T19:25:04.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7eff5c0aff60 0x7eff5c15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:04.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7eff5c0aff60 0x7eff5c15b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59712/0 (socket says 172.21.15.175:59712) 2023-10-10T19:25:04.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 -- 172.21.15.175:0/1851676788 learned_addr learned my addr 172.21.15.175:0/1851676788 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:04.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6198e700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 0x7eff5c1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:04.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6098c700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff5c0b1fc0 0x7eff5c149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:04.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 -- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff5c0b1fc0 msgr2=0x7eff5c149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:04.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff5c0b1fc0 0x7eff5c149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:04.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 -- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 msgr2=0x7eff5c1499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:04.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 0x7eff5c1499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:04.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 -- 172.21.15.175:0/1851676788 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c009420 con 0x7eff5c0aff60 2023-10-10T19:25:04.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff6118d700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7eff5c0aff60 0x7eff5c15b9b0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c590 tx=0x7eff5800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:04.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff517fa700 1 -- 172.21.15.175:0/1851676788 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58011590 con 0x7eff5c0aff60 2023-10-10T19:25:04.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.918+0000 7eff517fa700 1 -- 172.21.15.175:0/1851676788 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff58011710 con 0x7eff5c0aff60 2023-10-10T19:25:04.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.919+0000 7eff517fa700 1 -- 172.21.15.175:0/1851676788 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58011ad0 con 0x7eff5c0aff60 2023-10-10T19:25:04.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.919+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c14c400 con 0x7eff5c0aff60 2023-10-10T19:25:04.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.919+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5c14e190 con 0x7eff5c0aff60 2023-10-10T19:25:04.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.920+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5c1492b0 con 0x7eff5c0aff60 2023-10-10T19:25:04.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.920+0000 7eff517fa700 1 -- 172.21.15.175:0/1851676788 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7eff5801b3b0 con 0x7eff5c0aff60 2023-10-10T19:25:04.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.921+0000 7eff517fa700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7eff40066f30 0x7eff400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:04.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.921+0000 7eff517fa700 1 -- 172.21.15.175:0/1851676788 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7eff58084240 con 0x7eff5c0aff60 2023-10-10T19:25:04.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.921+0000 7eff6098c700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7eff40066f30 0x7eff400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:04.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.922+0000 7eff6098c700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7eff40066f30 0x7eff400693e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0093f0 tx=0x7eff4c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:04.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:04.929+0000 7eff517fa700 1 -- 172.21.15.175:0/1851676788 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff58050d30 con 0x7eff5c0aff60 2023-10-10T19:25:05.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.166+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff5c067570 con 0x7eff5c0aff60 2023-10-10T19:25:05.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.167+0000 7eff517fa700 1 -- 172.21.15.175:0/1851676788 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 291 v291) v1 ==== 95+0+5417 (secure 0 0 0) 0x7eff58017090 con 0x7eff5c0aff60 2023-10-10T19:25:05.168 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:05.168 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":291,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":291,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:24:58.864159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:05.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7eff40066f30 msgr2=0x7eff400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:05.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7eff40066f30 0x7eff400693e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0093f0 tx=0x7eff4c017000 comp rx=0 tx=0).stop 2023-10-10T19:25:05.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7eff5c0aff60 msgr2=0x7eff5c15b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:05.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7eff5c0aff60 0x7eff5c15b9b0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c590 tx=0x7eff5800c8a0 comp rx=0 tx=0).stop 2023-10-10T19:25:05.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 shutdown_connections 2023-10-10T19:25:05.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7eff40066f30 0x7eff400693e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:05.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff5c0b2390 0x7eff5c1499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:05.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7eff5c0aff60 0x7eff5c15b9b0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 --2- 172.21.15.175:0/1851676788 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff5c0b1fc0 0x7eff5c149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 >> 172.21.15.175:0/1851676788 conn(0x7eff5c063230 msgr2=0x7eff5c14fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 shutdown_connections 2023-10-10T19:25:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:05.169+0000 7eff633f1700 1 -- 172.21.15.175:0/1851676788 wait complete. 2023-10-10T19:25:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 291 2023-10-10T19:25:05.181 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:07.185 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:07.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.301+0000 7f3483b16700 1 Processor -- start 2023-10-10T19:25:07.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.301+0000 7f3483b16700 1 -- start start 2023-10-10T19:25:07.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.302+0000 7f3483b16700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c5850 0x7f347c0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:07.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.302+0000 7f3483b16700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f347c0c12d0 0x7f347c156660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:07.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.302+0000 7f3483b16700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0c6fb0 con 0x7f347c156ba0 2023-10-10T19:25:07.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.302+0000 7f3483b16700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c063ea0 con 0x7f347c0c12d0 2023-10-10T19:25:07.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.302+0000 7f3483b16700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c155f30 con 0x7f347c0c5850 2023-10-10T19:25:07.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.302+0000 7f34810b1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f347c0c12d0 0x7f347c156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:07.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.302+0000 7f34820b3700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f347c156ba0 0x7f347c0c7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53156/0 (socket says 172.21.15.175:53156) 2023-10-10T19:25:07.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.302+0000 7f34820b3700 1 -- 172.21.15.175:0/191048014 learned_addr learned my addr 172.21.15.175:0/191048014 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:07.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34810b1700 1 -- 172.21.15.175:0/191048014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c5850 msgr2=0x7f347c0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:07.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34818b2700 1 --2- 172.21.15.175:0/191048014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c5850 0x7f347c0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:07.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34810b1700 1 --2- 172.21.15.175:0/191048014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c5850 0x7f347c0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34808b0700 1 -- 172.21.15.175:0/191048014 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270133008 0 0) 0x7f347c0c6fb0 con 0x7f347c156ba0 2023-10-10T19:25:07.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34810b1700 1 -- 172.21.15.175:0/191048014 >> v1:172.21.15.175:6789/0 conn(0x7f347c156ba0 legacy=0x7f347c0c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:07.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34810b1700 1 -- 172.21.15.175:0/191048014 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c0c6160 con 0x7f347c0c12d0 2023-10-10T19:25:07.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34818b2700 1 --2- 172.21.15.175:0/191048014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c5850 0x7f347c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:25:07.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34810b1700 1 --2- 172.21.15.175:0/191048014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c12d0 0x7f347c156660 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f34700098e0 tx=0x7f34700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b19d4d68d9ef0f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:07.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34808b0700 1 -- 172.21.15.175:0/191048014 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470009e80 con 0x7f347c0c12d0 2023-10-10T19:25:07.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34808b0700 1 -- 172.21.15.175:0/191048014 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347000b8f0 con 0x7f347c0c12d0 2023-10-10T19:25:07.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f34808b0700 1 -- 172.21.15.175:0/191048014 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347000bc70 con 0x7f347c0c12d0 2023-10-10T19:25:07.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f3483b16700 1 -- 172.21.15.175:0/191048014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c12d0 msgr2=0x7f347c156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:07.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f3483b16700 1 --2- 172.21.15.175:0/191048014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c12d0 0x7f347c156660 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f34700098e0 tx=0x7f34700091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.303+0000 7f3483b16700 1 -- 172.21.15.175:0/191048014 shutdown_connections 2023-10-10T19:25:07.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.304+0000 7f3483b16700 1 --2- 172.21.15.175:0/191048014 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c5850 0x7f347c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.304+0000 7f3483b16700 1 --2- 172.21.15.175:0/191048014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c12d0 0x7f347c156660 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.304+0000 7f3483b16700 1 -- 172.21.15.175:0/191048014 >> 172.21.15.175:0/191048014 conn(0x7f347c06dc70 msgr2=0x7f347c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:07.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.304+0000 7f3483b16700 1 -- 172.21.15.175:0/191048014 shutdown_connections 2023-10-10T19:25:07.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.304+0000 7f3483b16700 1 -- 172.21.15.175:0/191048014 wait complete. 2023-10-10T19:25:07.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f3483b16700 1 Processor -- start 2023-10-10T19:25:07.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f3483b16700 1 -- start start 2023-10-10T19:25:07.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f3483b16700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c12d0 0x7f347c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:07.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f3483b16700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c5850 0x7f347c14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:07.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f3483b16700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c156ba0 0x7f347c1572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:07.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f3483b16700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0c8ef0 con 0x7f347c0c5850 2023-10-10T19:25:07.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f3483b16700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c151100 con 0x7f347c156ba0 2023-10-10T19:25:07.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f3483b16700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c150810 con 0x7f347c0c12d0 2023-10-10T19:25:07.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f34810b1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c5850 0x7f347c14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:07.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f34810b1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c5850 0x7f347c14fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59730/0 (socket says 172.21.15.175:59730) 2023-10-10T19:25:07.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.305+0000 7f34810b1700 1 -- 172.21.15.175:0/848609546 learned_addr learned my addr 172.21.15.175:0/848609546 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:07.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f34818b2700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c12d0 0x7f347c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:07.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f34820b3700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c156ba0 0x7f347c1572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:07.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f34810b1700 1 -- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c12d0 msgr2=0x7f347c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:07.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f34810b1700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c12d0 0x7f347c0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f34810b1700 1 -- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c156ba0 msgr2=0x7f347c1572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:07.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f34810b1700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c156ba0 0x7f347c1572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f34810b1700 1 -- 172.21.15.175:0/848609546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3470009420 con 0x7f347c0c5850 2023-10-10T19:25:07.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f34810b1700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c5850 0x7f347c14fde0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f347000b3c0 tx=0x7f3470017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:07.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f3479ffb700 1 -- 172.21.15.175:0/848609546 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347000bc30 con 0x7f347c0c5850 2023-10-10T19:25:07.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f3479ffb700 1 -- 172.21.15.175:0/848609546 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3470025420 con 0x7f347c0c5850 2023-10-10T19:25:07.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c14ec10 con 0x7f347c0c5850 2023-10-10T19:25:07.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f3479ffb700 1 -- 172.21.15.175:0/848609546 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700257c0 con 0x7f347c0c5850 2023-10-10T19:25:07.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.306+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c150ab0 con 0x7f347c0c5850 2023-10-10T19:25:07.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.307+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c0c6360 con 0x7f347c0c5850 2023-10-10T19:25:07.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.308+0000 7f3479ffb700 1 -- 172.21.15.175:0/848609546 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3470023070 con 0x7f347c0c5850 2023-10-10T19:25:07.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.308+0000 7f3479ffb700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3468066f60 0x7f3468069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:07.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.308+0000 7f3479ffb700 1 -- 172.21.15.175:0/848609546 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f3470086400 con 0x7f347c0c5850 2023-10-10T19:25:07.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.308+0000 7f34818b2700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3468066f60 0x7f3468069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:07.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.309+0000 7f34818b2700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3468066f60 0x7f3468069410 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f346c00e4b0 tx=0x7f346c007670 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:07.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.312+0000 7f3479ffb700 1 -- 172.21.15.175:0/848609546 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3470052ef0 con 0x7f347c0c5850 2023-10-10T19:25:07.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.535+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f347c067570 con 0x7f347c0c5850 2023-10-10T19:25:07.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.535+0000 7f3479ffb700 1 -- 172.21.15.175:0/848609546 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) v1 ==== 95+0+5419 (secure 0 0 0) 0x7f3470056510 con 0x7f347c0c5850 2023-10-10T19:25:07.536 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:07.536 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":292,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:06.939264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:resolve","state_seq":15,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:07.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3468066f60 msgr2=0x7f3468069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:07.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3468066f60 0x7f3468069410 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f346c00e4b0 tx=0x7f346c007670 comp rx=0 tx=0).stop 2023-10-10T19:25:07.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c5850 msgr2=0x7f347c14fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:07.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c5850 0x7f347c14fde0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f347000b3c0 tx=0x7f3470017000 comp rx=0 tx=0).stop 2023-10-10T19:25:07.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 shutdown_connections 2023-10-10T19:25:07.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3468066f60 0x7f3468069410 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c156ba0 0x7f347c1572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0c12d0 0x7f347c0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 --2- 172.21.15.175:0/848609546 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f347c0c5850 0x7f347c14fde0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:07.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 >> 172.21.15.175:0/848609546 conn(0x7f347c06dc70 msgr2=0x7f347c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:07.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 shutdown_connections 2023-10-10T19:25:07.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:07.537+0000 7f3483b16700 1 -- 172.21.15.175:0/848609546 wait complete. 2023-10-10T19:25:07.541 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 292 2023-10-10T19:25:07.549 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:09.550 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:09.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.674+0000 7f25552ee700 1 Processor -- start 2023-10-10T19:25:09.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.674+0000 7f25552ee700 1 -- start start 2023-10-10T19:25:09.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.675+0000 7f25552ee700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2550068e10 0x7f25500b0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:09.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.675+0000 7f25552ee700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25500c4ee0 con 0x7f25500cac80 2023-10-10T19:25:09.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.675+0000 7f25552ee700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25500c5780 con 0x7f2550068a40 2023-10-10T19:25:09.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.675+0000 7f25552ee700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25500c6710 con 0x7f2550068e10 2023-10-10T19:25:09.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.675+0000 7f254f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2550068e10 0x7f25500b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:09.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.675+0000 7f25467fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f25500cac80 0x7f25500af910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53160/0 (socket says 172.21.15.175:53160) 2023-10-10T19:25:09.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.675+0000 7f25467fc700 1 -- 172.21.15.175:0/2910777110 learned_addr learned my addr 172.21.15.175:0/2910777110 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:09.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f254f7fe700 1 -- 172.21.15.175:0/2910777110 >> v1:172.21.15.191:6789/0 conn(0x7f2550068a40 legacy=0x7f255006b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:09.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f254f7fe700 1 -- 172.21.15.175:0/2910777110 >> v1:172.21.15.175:6789/0 conn(0x7f25500cac80 legacy=0x7f25500af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:09.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f254e7fc700 1 -- 172.21.15.175:0/2910777110 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460793307 0 0) 0x7f25500c4ee0 con 0x7f25500cac80 2023-10-10T19:25:09.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f254f7fe700 1 -- 172.21.15.175:0/2910777110 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25500c6990 con 0x7f2550068e10 2023-10-10T19:25:09.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f254f7fe700 1 --2- 172.21.15.175:0/2910777110 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550068e10 0x7f25500b0020 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f2548009e60 tx=0x7f2548009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce6a4d741090dd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:09.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f254e7fc700 1 -- 172.21.15.175:0/2910777110 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254800e070 con 0x7f2550068e10 2023-10-10T19:25:09.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f254e7fc700 1 -- 172.21.15.175:0/2910777110 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254800bc30 con 0x7f2550068e10 2023-10-10T19:25:09.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f254e7fc700 1 -- 172.21.15.175:0/2910777110 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548012430 con 0x7f2550068e10 2023-10-10T19:25:09.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f25552ee700 1 -- 172.21.15.175:0/2910777110 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550068e10 msgr2=0x7f25500b0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:09.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.676+0000 7f25552ee700 1 --2- 172.21.15.175:0/2910777110 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550068e10 0x7f25500b0020 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f2548009e60 tx=0x7f2548009780 comp rx=0 tx=0).stop 2023-10-10T19:25:09.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.677+0000 7f25552ee700 1 -- 172.21.15.175:0/2910777110 shutdown_connections 2023-10-10T19:25:09.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.677+0000 7f25552ee700 1 --2- 172.21.15.175:0/2910777110 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550068e10 0x7f25500b0020 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:09.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.677+0000 7f25552ee700 1 -- 172.21.15.175:0/2910777110 >> 172.21.15.175:0/2910777110 conn(0x7f255006dbe0 msgr2=0x7f25500ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:09.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.677+0000 7f25552ee700 1 -- 172.21.15.175:0/2910777110 shutdown_connections 2023-10-10T19:25:09.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.677+0000 7f25552ee700 1 -- 172.21.15.175:0/2910777110 wait complete. 2023-10-10T19:25:09.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.678+0000 7f25552ee700 1 Processor -- start 2023-10-10T19:25:09.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.678+0000 7f25552ee700 1 -- start start 2023-10-10T19:25:09.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.678+0000 7f25552ee700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2550068a40 0x7f255015ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:09.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.678+0000 7f25552ee700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2550068e10 0x7f2550168110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:09.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.678+0000 7f25552ee700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f25500cac80 0x7f2550168650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:09.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.678+0000 7f25552ee700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255015ad00 con 0x7f25500cac80 2023-10-10T19:25:09.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.678+0000 7f25552ee700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2550166cd0 con 0x7f2550068e10 2023-10-10T19:25:09.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.678+0000 7f25552ee700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2550157500 con 0x7f2550068a40 2023-10-10T19:25:09.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f25500cac80 0x7f2550168650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:09.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f25500cac80 0x7f2550168650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59746/0 (socket says 172.21.15.175:59746) 2023-10-10T19:25:09.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 -- 172.21.15.175:0/3281928457 learned_addr learned my addr 172.21.15.175:0/3281928457 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:09.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f25467fc700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2550068e10 0x7f2550168110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:09.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254effd700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2550068a40 0x7f255015ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:09.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 -- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2550068a40 msgr2=0x7f255015ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:09.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2550068a40 0x7f255015ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:09.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 -- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2550068e10 msgr2=0x7f2550168110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:09.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2550068e10 0x7f2550168110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:09.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 -- 172.21.15.175:0/3281928457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2548009a20 con 0x7f25500cac80 2023-10-10T19:25:09.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254effd700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2550068a40 0x7f255015ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:09.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f254f7fe700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f25500cac80 0x7f2550168650 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f255006b290 tx=0x7f2548019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:09.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f25477fe700 1 -- 172.21.15.175:0/3281928457 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548012ea0 con 0x7f25500cac80 2023-10-10T19:25:09.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f25477fe700 1 -- 172.21.15.175:0/3281928457 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254801b600 con 0x7f25500cac80 2023-10-10T19:25:09.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25500bdab0 con 0x7f25500cac80 2023-10-10T19:25:09.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f25477fe700 1 -- 172.21.15.175:0/3281928457 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254801b980 con 0x7f25500cac80 2023-10-10T19:25:09.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.679+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2550167630 con 0x7f25500cac80 2023-10-10T19:25:09.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.680+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2550067570 con 0x7f25500cac80 2023-10-10T19:25:09.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.681+0000 7f25477fe700 1 -- 172.21.15.175:0/3281928457 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2548003020 con 0x7f25500cac80 2023-10-10T19:25:09.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.681+0000 7f25477fe700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2530066f60 0x7f2530069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:09.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.681+0000 7f25477fe700 1 -- 172.21.15.175:0/3281928457 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f2548086e00 con 0x7f25500cac80 2023-10-10T19:25:09.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.681+0000 7f254effd700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2530066f60 0x7f2530069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:09.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.682+0000 7f254effd700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2530066f60 0x7f2530069410 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f2540001850 tx=0x7f2540009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:09.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.685+0000 7f25477fe700 1 -- 172.21.15.175:0/3281928457 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f25480538f0 con 0x7f25500cac80 2023-10-10T19:25:09.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.912+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25500cb260 con 0x7f25500cac80 2023-10-10T19:25:09.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.913+0000 7f25477fe700 1 -- 172.21.15.175:0/3281928457 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) v1 ==== 95+0+5419 (secure 0 0 0) 0x7f2548056f10 con 0x7f25500cac80 2023-10-10T19:25:09.914 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:09.914 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":292,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:06.939264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:resolve","state_seq":15,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:09.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2530066f60 msgr2=0x7f2530069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:09.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2530066f60 0x7f2530069410 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f2540001850 tx=0x7f2540009040 comp rx=0 tx=0).stop 2023-10-10T19:25:09.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f25500cac80 msgr2=0x7f2550168650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:09.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f25500cac80 0x7f2550168650 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f255006b290 tx=0x7f2548019000 comp rx=0 tx=0).stop 2023-10-10T19:25:09.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 shutdown_connections 2023-10-10T19:25:09.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2530066f60 0x7f2530069410 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:09.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f25500cac80 0x7f2550168650 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:09.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2550068a40 0x7f255015ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:09.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 --2- 172.21.15.175:0/3281928457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2550068e10 0x7f2550168110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:09.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 >> 172.21.15.175:0/3281928457 conn(0x7f255006dbe0 msgr2=0x7f2550063ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:09.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 shutdown_connections 2023-10-10T19:25:09.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:09.915+0000 7f25552ee700 1 -- 172.21.15.175:0/3281928457 wait complete. 2023-10-10T19:25:09.918 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 292 2023-10-10T19:25:09.927 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:11.928 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:12.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.058+0000 7f6b26dcf700 1 Processor -- start 2023-10-10T19:25:12.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.058+0000 7f6b26dcf700 1 -- start start 2023-10-10T19:25:12.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b26dcf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6b200b1fc0 0x7f6b20155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:12.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b26dcf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b200b2390 0x7f6b20068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:12.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b26dcf700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b201574b0 con 0x7f6b20068bd0 2023-10-10T19:25:12.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b26dcf700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b20150770 con 0x7f6b200b1fc0 2023-10-10T19:25:12.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b26dcf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200afad0 con 0x7f6b200b2390 2023-10-10T19:25:12.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b24b6b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6b200b1fc0 0x7f6b20155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:12.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b2536c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6b20068bd0 0x7f6b200aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53162/0 (socket says 172.21.15.175:53162) 2023-10-10T19:25:12.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b2536c700 1 -- 172.21.15.175:0/2652025094 learned_addr learned my addr 172.21.15.175:0/2652025094 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:12.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b1ffff700 1 --2- 172.21.15.175:0/2652025094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b200b2390 0x7f6b20068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:12.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b24b6b700 1 -- 172.21.15.175:0/2652025094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b200b2390 msgr2=0x7f6b20068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:12.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b24b6b700 1 --2- 172.21.15.175:0/2652025094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b200b2390 0x7f6b20068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b24b6b700 1 -- 172.21.15.175:0/2652025094 >> v1:172.21.15.175:6789/0 conn(0x7f6b20068bd0 legacy=0x7f6b200aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:12.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b24b6b700 1 -- 172.21.15.175:0/2652025094 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200af4c0 con 0x7f6b200b1fc0 2023-10-10T19:25:12.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b1f7fe700 1 -- 172.21.15.175:0/2652025094 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747248601 0 0) 0x7f6b201574b0 con 0x7f6b20068bd0 2023-10-10T19:25:12.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b1ffff700 1 --2- 172.21.15.175:0/2652025094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b200b2390 0x7f6b20068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:12.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b24b6b700 1 --2- 172.21.15.175:0/2652025094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b1fc0 0x7f6b20155fb0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009e60 tx=0x7f6b18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef29972b6a6a60d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:12.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b1f7fe700 1 -- 172.21.15.175:0/2652025094 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800e070 con 0x7f6b200b1fc0 2023-10-10T19:25:12.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b1f7fe700 1 -- 172.21.15.175:0/2652025094 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1800bcc0 con 0x7f6b200b1fc0 2023-10-10T19:25:12.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.059+0000 7f6b1f7fe700 1 -- 172.21.15.175:0/2652025094 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18012680 con 0x7f6b200b1fc0 2023-10-10T19:25:12.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.060+0000 7f6b26dcf700 1 -- 172.21.15.175:0/2652025094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b1fc0 msgr2=0x7f6b20155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:12.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.060+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/2652025094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b1fc0 0x7f6b20155fb0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009e60 tx=0x7f6b18009780 comp rx=0 tx=0).stop 2023-10-10T19:25:12.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.060+0000 7f6b26dcf700 1 -- 172.21.15.175:0/2652025094 shutdown_connections 2023-10-10T19:25:12.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.060+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/2652025094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b1fc0 0x7f6b20155fb0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.060+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/2652025094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b200b2390 0x7f6b20068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.060+0000 7f6b26dcf700 1 -- 172.21.15.175:0/2652025094 >> 172.21.15.175:0/2652025094 conn(0x7f6b20063230 msgr2=0x7f6b20063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:12.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.060+0000 7f6b26dcf700 1 -- 172.21.15.175:0/2652025094 shutdown_connections 2023-10-10T19:25:12.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.060+0000 7f6b26dcf700 1 -- 172.21.15.175:0/2652025094 wait complete. 2023-10-10T19:25:12.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.061+0000 7f6b26dcf700 1 Processor -- start 2023-10-10T19:25:12.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.061+0000 7f6b26dcf700 1 -- start start 2023-10-10T19:25:12.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b26dcf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b20068bd0 0x7f6b200cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:12.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b26dcf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b200b1fc0 0x7f6b200c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:12.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b26dcf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b2390 0x7f6b2014fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:12.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b26dcf700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200c3b60 con 0x7f6b200b2390 2023-10-10T19:25:12.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b26dcf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200c6960 con 0x7f6b200b1fc0 2023-10-10T19:25:12.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b26dcf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200c4f50 con 0x7f6b20068bd0 2023-10-10T19:25:12.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b2390 0x7f6b2014fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:12.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b2390 0x7f6b2014fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59754/0 (socket says 172.21.15.175:59754) 2023-10-10T19:25:12.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 -- 172.21.15.175:0/3188844741 learned_addr learned my addr 172.21.15.175:0/3188844741 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:12.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b24b6b700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b20068bd0 0x7f6b200cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:12.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 -- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b20068bd0 msgr2=0x7f6b200cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:12.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b1ffff700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b200b1fc0 0x7f6b200c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:12.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b20068bd0 0x7f6b200cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 -- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b200b1fc0 msgr2=0x7f6b200c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:12.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b200b1fc0 0x7f6b200c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 -- 172.21.15.175:0/3188844741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b18009a20 con 0x7f6b200b2390 2023-10-10T19:25:12.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.062+0000 7f6b2536c700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b2390 0x7f6b2014fa20 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800c5e0 tx=0x7f6b08007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:12.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.063+0000 7f6b1cff9700 1 -- 172.21.15.175:0/3188844741 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08005b30 con 0x7f6b200b2390 2023-10-10T19:25:12.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.063+0000 7f6b1cff9700 1 -- 172.21.15.175:0/3188844741 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b08005c90 con 0x7f6b200b2390 2023-10-10T19:25:12.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.063+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b200c51f0 con 0x7f6b200b2390 2023-10-10T19:25:12.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.063+0000 7f6b1cff9700 1 -- 172.21.15.175:0/3188844741 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08004a40 con 0x7f6b200b2390 2023-10-10T19:25:12.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.063+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b200c4ac0 con 0x7f6b200b2390 2023-10-10T19:25:12.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.064+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b200c4640 con 0x7f6b200b2390 2023-10-10T19:25:12.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.064+0000 7f6b1cff9700 1 -- 172.21.15.175:0/3188844741 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6b08005030 con 0x7f6b200b2390 2023-10-10T19:25:12.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.064+0000 7f6b1cff9700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6b00066f60 0x7f6b00069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:12.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.064+0000 7f6b1cff9700 1 -- 172.21.15.175:0/3188844741 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f6b08084160 con 0x7f6b200b2390 2023-10-10T19:25:12.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.065+0000 7f6b24b6b700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6b00066f60 0x7f6b00069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:12.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.065+0000 7f6b24b6b700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6b00066f60 0x7f6b00069410 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800c010 tx=0x7f6b18019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:12.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.069+0000 7f6b1cff9700 1 -- 172.21.15.175:0/3188844741 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b200c4640 con 0x7f6b200b2390 2023-10-10T19:25:12.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.294+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b200c3d10 con 0x7f6b200b2390 2023-10-10T19:25:12.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.295+0000 7f6b1cff9700 1 -- 172.21.15.175:0/3188844741 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) v1 ==== 95+0+5419 (secure 0 0 0) 0x7f6b200c3d10 con 0x7f6b200b2390 2023-10-10T19:25:12.296 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:12.296 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":292,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:06.939264+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:resolve","state_seq":15,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:12.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6b00066f60 msgr2=0x7f6b00069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:12.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6b00066f60 0x7f6b00069410 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800c010 tx=0x7f6b18019000 comp rx=0 tx=0).stop 2023-10-10T19:25:12.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b2390 msgr2=0x7f6b2014fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:12.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b2390 0x7f6b2014fa20 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800c5e0 tx=0x7f6b08007d30 comp rx=0 tx=0).stop 2023-10-10T19:25:12.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 shutdown_connections 2023-10-10T19:25:12.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6b00066f60 0x7f6b00069410 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b200b2390 0x7f6b2014fa20 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b20068bd0 0x7f6b200cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 --2- 172.21.15.175:0/3188844741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b200b1fc0 0x7f6b200c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:12.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 >> 172.21.15.175:0/3188844741 conn(0x7f6b20063230 msgr2=0x7f6b201508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:12.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 shutdown_connections 2023-10-10T19:25:12.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:12.297+0000 7f6b26dcf700 1 -- 172.21.15.175:0/3188844741 wait complete. 2023-10-10T19:25:12.301 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 292 2023-10-10T19:25:12.310 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 292, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 292, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_7384": { "addr": "172.21.15.191:6837/2914004704", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 2914004704, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2914004704, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7384, "incarnation": 278, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 23 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 117 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 129, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:25:06.939264+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 7401, "mds_2": 8049, "mds_3": 7384 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:25:14.312 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:14.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.439+0000 7f30ea594700 1 Processor -- start 2023-10-10T19:25:14.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.439+0000 7f30ea594700 1 -- start start 2023-10-10T19:25:14.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30ea594700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f30e414fd90 0x7f30e4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:14.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30ea594700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e40c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:14.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30ea594700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e4067fa0 con 0x7f30e40cd040 2023-10-10T19:25:14.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30ea594700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e40c5600 con 0x7f30e414fd90 2023-10-10T19:25:14.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30ea594700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4156b70 con 0x7f30e408a100 2023-10-10T19:25:14.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e9592700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f30e414fd90 0x7f30e4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:14.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e9592700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f30e414fd90 0x7f30e4159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59756/0 (socket says 172.21.15.175:59756) 2023-10-10T19:25:14.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e9592700 1 -- 172.21.15.175:0/3413924116 learned_addr learned my addr 172.21.15.175:0/3413924116 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:14.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e9d93700 1 --1- 172.21.15.175:0/3413924116 >> v1:172.21.15.175:6789/0 conn(0x7f30e40cd040 0x7f30e40684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53172/0 (socket says 172.21.15.175:53172) 2023-10-10T19:25:14.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e8d91700 1 --2- 172.21.15.175:0/3413924116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e40c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:14.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e9592700 1 -- 172.21.15.175:0/3413924116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 msgr2=0x7f30e40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:14.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e9592700 1 --2- 172.21.15.175:0/3413924116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e40c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e9592700 1 -- 172.21.15.175:0/3413924116 >> v1:172.21.15.175:6789/0 conn(0x7f30e40cd040 legacy=0x7f30e40684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:14.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30dbfff700 1 -- 172.21.15.175:0/3413924116 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963480444 0 0) 0x7f30e4067fa0 con 0x7f30e40cd040 2023-10-10T19:25:14.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e9592700 1 -- 172.21.15.175:0/3413924116 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e4150160 con 0x7f30e414fd90 2023-10-10T19:25:14.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.440+0000 7f30e8d91700 1 --2- 172.21.15.175:0/3413924116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e40c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:14.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30e9592700 1 --2- 172.21.15.175:0/3413924116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e414fd90 0x7f30e4159c30 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f30e00099b0 tx=0x7f30e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ef98a87a62b33d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:14.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30dbfff700 1 -- 172.21.15.175:0/3413924116 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e000b860 con 0x7f30e414fd90 2023-10-10T19:25:14.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30dbfff700 1 -- 172.21.15.175:0/3413924116 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e000b9c0 con 0x7f30e414fd90 2023-10-10T19:25:14.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30dbfff700 1 -- 172.21.15.175:0/3413924116 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e000bd60 con 0x7f30e414fd90 2023-10-10T19:25:14.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30ea594700 1 -- 172.21.15.175:0/3413924116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e414fd90 msgr2=0x7f30e4159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:14.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30ea594700 1 --2- 172.21.15.175:0/3413924116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e414fd90 0x7f30e4159c30 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f30e00099b0 tx=0x7f30e0009250 comp rx=0 tx=0).stop 2023-10-10T19:25:14.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30ea594700 1 -- 172.21.15.175:0/3413924116 shutdown_connections 2023-10-10T19:25:14.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30ea594700 1 --2- 172.21.15.175:0/3413924116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e414fd90 0x7f30e4159c30 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30ea594700 1 --2- 172.21.15.175:0/3413924116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e40c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30ea594700 1 -- 172.21.15.175:0/3413924116 >> 172.21.15.175:0/3413924116 conn(0x7f30e4063d00 msgr2=0x7f30e40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:14.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30ea594700 1 -- 172.21.15.175:0/3413924116 shutdown_connections 2023-10-10T19:25:14.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.441+0000 7f30ea594700 1 -- 172.21.15.175:0/3413924116 wait complete. 2023-10-10T19:25:14.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.442+0000 7f30ea594700 1 Processor -- start 2023-10-10T19:25:14.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.443+0000 7f30ea594700 1 -- start start 2023-10-10T19:25:14.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.443+0000 7f30ea594700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e4166680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:14.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.443+0000 7f30ea594700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e40cd040 0x7f30e4151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:14.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.443+0000 7f30ea594700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e414fd90 0x7f30e414b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:14.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.443+0000 7f30ea594700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4152830 con 0x7f30e40cd040 2023-10-10T19:25:14.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.443+0000 7f30ea594700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e40c9cc0 con 0x7f30e414fd90 2023-10-10T19:25:14.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.443+0000 7f30ea594700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4150e40 con 0x7f30e408a100 2023-10-10T19:25:14.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e40cd040 0x7f30e4151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:14.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e40cd040 0x7f30e4151a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59758/0 (socket says 172.21.15.175:59758) 2023-10-10T19:25:14.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 -- 172.21.15.175:0/1056188584 learned_addr learned my addr 172.21.15.175:0/1056188584 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:14.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e9592700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e4166680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:14.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e9d93700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e414fd90 0x7f30e414b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:14.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 -- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 msgr2=0x7f30e4166680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:14.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e4166680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 -- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e414fd90 msgr2=0x7f30e414b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:14.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e414fd90 0x7f30e414b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 -- 172.21.15.175:0/1056188584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e00094f0 con 0x7f30e40cd040 2023-10-10T19:25:14.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e9592700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e4166680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:14.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30e8d91700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e40cd040 0x7f30e4151a90 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f30d400e900 tx=0x7f30d4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:14.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30d97fa700 1 -- 172.21.15.175:0/1056188584 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d400eea0 con 0x7f30e40cd040 2023-10-10T19:25:14.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30d97fa700 1 -- 172.21.15.175:0/1056188584 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d400c670 con 0x7f30e40cd040 2023-10-10T19:25:14.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30d97fa700 1 -- 172.21.15.175:0/1056188584 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d400ca10 con 0x7f30e40cd040 2023-10-10T19:25:14.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e41661f0 con 0x7f30e40cd040 2023-10-10T19:25:14.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.444+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e4150b60 con 0x7f30e40cd040 2023-10-10T19:25:14.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.445+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e4166380 con 0x7f30e40cd040 2023-10-10T19:25:14.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.445+0000 7f30d97fa700 1 -- 172.21.15.175:0/1056188584 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f30d4013410 con 0x7f30e40cd040 2023-10-10T19:25:14.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.446+0000 7f30d97fa700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30c8066f30 0x7f30c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:14.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.446+0000 7f30d97fa700 1 -- 172.21.15.175:0/1056188584 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f30d4084b80 con 0x7f30e40cd040 2023-10-10T19:25:14.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.446+0000 7f30e9592700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30c8066f30 0x7f30c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:14.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.447+0000 7f30e9592700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30c8066f30 0x7f30c80693e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f30e00099b0 tx=0x7f30e0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:14.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.450+0000 7f30d97fa700 1 -- 172.21.15.175:0/1056188584 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f30d4055100 con 0x7f30e40cd040 2023-10-10T19:25:14.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.673+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30e4067570 con 0x7f30e40cd040 2023-10-10T19:25:14.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.673+0000 7f30d97fa700 1 -- 172.21.15.175:0/1056188584 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+5421 (secure 0 0 0) 0x7f30d4054c90 con 0x7f30e40cd040 2023-10-10T19:25:14.674 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:14.675 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":293,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:12.961735+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:14.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30c8066f30 msgr2=0x7f30c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:14.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30c8066f30 0x7f30c80693e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f30e00099b0 tx=0x7f30e0017000 comp rx=0 tx=0).stop 2023-10-10T19:25:14.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e40cd040 msgr2=0x7f30e4151a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:14.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e40cd040 0x7f30e4151a90 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f30d400e900 tx=0x7f30d4008040 comp rx=0 tx=0).stop 2023-10-10T19:25:14.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 shutdown_connections 2023-10-10T19:25:14.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f30c8066f30 0x7f30c80693e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e414fd90 0x7f30e414b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30e408a100 0x7f30e4166680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 --2- 172.21.15.175:0/1056188584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f30e40cd040 0x7f30e4151a90 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:14.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.675+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 >> 172.21.15.175:0/1056188584 conn(0x7f30e4063d00 msgr2=0x7f30e4153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:14.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.676+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 shutdown_connections 2023-10-10T19:25:14.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:14.676+0000 7f30ea594700 1 -- 172.21.15.175:0/1056188584 wait complete. 2023-10-10T19:25:14.679 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 293 2023-10-10T19:25:14.688 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:16.689 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:16.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.816+0000 7f27bacad700 1 Processor -- start 2023-10-10T19:25:16.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.816+0000 7f27bacad700 1 -- start start 2023-10-10T19:25:16.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.817+0000 7f27bacad700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f27b4156c90 0x7f27b40c7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:16.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.817+0000 7f27bacad700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b415a6c0 con 0x7f27b40c35a0 2023-10-10T19:25:16.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.817+0000 7f27bacad700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b4159830 con 0x7f27b40c2b80 2023-10-10T19:25:16.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.817+0000 7f27bacad700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b414d4a0 con 0x7f27b4156c90 2023-10-10T19:25:16.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.817+0000 7f27b924a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f27b4156c90 0x7f27b40c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:16.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.817+0000 7f27b3fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f27b40c35a0 0x7f27b40c6a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47088/0 (socket says 172.21.15.175:47088) 2023-10-10T19:25:16.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.817+0000 7f27b924a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f27b4156c90 0x7f27b40c7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49850/0 (socket says 172.21.15.175:49850) 2023-10-10T19:25:16.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.817+0000 7f27b3fff700 1 -- 172.21.15.175:0/3722514182 learned_addr learned my addr 172.21.15.175:0/3722514182 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:16.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27b924a700 1 -- 172.21.15.175:0/3722514182 >> v1:172.21.15.191:6789/0 conn(0x7f27b40c2b80 legacy=0x7f27b4156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:16.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27b924a700 1 -- 172.21.15.175:0/3722514182 >> v1:172.21.15.175:6789/0 conn(0x7f27b40c35a0 legacy=0x7f27b40c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:16.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27b924a700 1 -- 172.21.15.175:0/3722514182 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4159ab0 con 0x7f27b4156c90 2023-10-10T19:25:16.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27b37fe700 1 -- 172.21.15.175:0/3722514182 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894926099 0 0) 0x7f27b415a6c0 con 0x7f27b40c35a0 2023-10-10T19:25:16.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27b924a700 1 --2- 172.21.15.175:0/3722514182 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b4156c90 0x7f27b40c7140 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0098e0 tx=0x7f27ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c6f6515390f713 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:16.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27b37fe700 1 -- 172.21.15.175:0/3722514182 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac009e80 con 0x7f27b4156c90 2023-10-10T19:25:16.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27b37fe700 1 -- 172.21.15.175:0/3722514182 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac00b8f0 con 0x7f27b4156c90 2023-10-10T19:25:16.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27b37fe700 1 -- 172.21.15.175:0/3722514182 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac00bc70 con 0x7f27b4156c90 2023-10-10T19:25:16.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27bacad700 1 -- 172.21.15.175:0/3722514182 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b4156c90 msgr2=0x7f27b40c7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:16.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.818+0000 7f27bacad700 1 --2- 172.21.15.175:0/3722514182 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b4156c90 0x7f27b40c7140 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0098e0 tx=0x7f27ac0091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:16.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.819+0000 7f27bacad700 1 -- 172.21.15.175:0/3722514182 shutdown_connections 2023-10-10T19:25:16.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.819+0000 7f27bacad700 1 --2- 172.21.15.175:0/3722514182 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b4156c90 0x7f27b40c7140 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:16.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.819+0000 7f27bacad700 1 -- 172.21.15.175:0/3722514182 >> 172.21.15.175:0/3722514182 conn(0x7f27b40c56a0 msgr2=0x7f27b40c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:16.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.819+0000 7f27bacad700 1 -- 172.21.15.175:0/3722514182 shutdown_connections 2023-10-10T19:25:16.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.819+0000 7f27bacad700 1 -- 172.21.15.175:0/3722514182 wait complete. 2023-10-10T19:25:16.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.820+0000 7f27bacad700 1 Processor -- start 2023-10-10T19:25:16.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.820+0000 7f27bacad700 1 -- start start 2023-10-10T19:25:16.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.820+0000 7f27bacad700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27b40c2b80 0x7f27b40c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:16.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.820+0000 7f27bacad700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b40c35a0 0x7f27b4165dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:16.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27bacad700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27b4156c90 0x7f27b4144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:16.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27bacad700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b4169d10 con 0x7f27b40c35a0 2023-10-10T19:25:16.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27bacad700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b416a1a0 con 0x7f27b4156c90 2023-10-10T19:25:16.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27bacad700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b41578d0 con 0x7f27b40c2b80 2023-10-10T19:25:16.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b40c35a0 0x7f27b4165dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b40c35a0 0x7f27b4165dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49864/0 (socket says 172.21.15.175:49864) 2023-10-10T19:25:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 -- 172.21.15.175:0/3671387446 learned_addr learned my addr 172.21.15.175:0/3671387446 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b924a700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27b4156c90 0x7f27b4144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b8a49700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27b40c2b80 0x7f27b40c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 -- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27b40c2b80 msgr2=0x7f27b40c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:16.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27b40c2b80 0x7f27b40c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:16.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 -- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27b4156c90 msgr2=0x7f27b4144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:16.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27b4156c90 0x7f27b4144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:16.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 -- 172.21.15.175:0/3671387446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac009420 con 0x7f27b40c35a0 2023-10-10T19:25:16.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b924a700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27b4156c90 0x7f27b4144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:16.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b3fff700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b40c35a0 0x7f27b4165dc0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f279c00c7e0 tx=0x7f279c007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:16.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b0ff9700 1 -- 172.21.15.175:0/3671387446 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c005c00 con 0x7f27b40c35a0 2023-10-10T19:25:16.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b0ff9700 1 -- 172.21.15.175:0/3671387446 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279c005d80 con 0x7f27b40c35a0 2023-10-10T19:25:16.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27b0ff9700 1 -- 172.21.15.175:0/3671387446 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c007320 con 0x7f27b40c35a0 2023-10-10T19:25:16.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b41693e0 con 0x7f27b40c35a0 2023-10-10T19:25:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.821+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b41699a0 con 0x7f27b40c35a0 2023-10-10T19:25:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.822+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b406a140 con 0x7f27b40c35a0 2023-10-10T19:25:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.823+0000 7f27b0ff9700 1 -- 172.21.15.175:0/3671387446 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f279c01b3b0 con 0x7f27b40c35a0 2023-10-10T19:25:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.823+0000 7f27b0ff9700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2794066f60 0x7f2794069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.823+0000 7f27b0ff9700 1 -- 172.21.15.175:0/3671387446 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f279c084250 con 0x7f27b40c35a0 2023-10-10T19:25:16.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.823+0000 7f27b8a49700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2794066f60 0x7f2794069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:16.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.824+0000 7f27b8a49700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2794066f60 0x7f2794069410 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f27a4007fd0 tx=0x7f27a4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:16.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:16.827+0000 7f27b0ff9700 1 -- 172.21.15.175:0/3671387446 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f279c050d40 con 0x7f27b40c35a0 2023-10-10T19:25:17.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.055+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27b4067570 con 0x7f27b40c35a0 2023-10-10T19:25:17.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.055+0000 7f27b0ff9700 1 -- 172.21.15.175:0/3671387446 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+5421 (secure 0 0 0) 0x7f279c054360 con 0x7f27b40c35a0 2023-10-10T19:25:17.057 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:17.057 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":293,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:12.961735+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:17.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2794066f60 msgr2=0x7f2794069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:17.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2794066f60 0x7f2794069410 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f27a4007fd0 tx=0x7f27a4009040 comp rx=0 tx=0).stop 2023-10-10T19:25:17.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b40c35a0 msgr2=0x7f27b4165dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:17.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b40c35a0 0x7f27b4165dc0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f279c00c7e0 tx=0x7f279c007d70 comp rx=0 tx=0).stop 2023-10-10T19:25:17.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 shutdown_connections 2023-10-10T19:25:17.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2794066f60 0x7f2794069410 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:17.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27b4156c90 0x7f27b4144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:17.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27b40c2b80 0x7f27b40c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:17.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 --2- 172.21.15.175:0/3671387446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27b40c35a0 0x7f27b4165dc0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:17.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 >> 172.21.15.175:0/3671387446 conn(0x7f27b40c56a0 msgr2=0x7f27b40c5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:17.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 shutdown_connections 2023-10-10T19:25:17.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:17.058+0000 7f27bacad700 1 -- 172.21.15.175:0/3671387446 wait complete. 2023-10-10T19:25:17.061 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 293 2023-10-10T19:25:17.070 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:19.071 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:19.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.198+0000 7f62cb55b700 1 Processor -- start 2023-10-10T19:25:19.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.198+0000 7f62cb55b700 1 -- start start 2023-10-10T19:25:19.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62cb55b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f62c414fd90 0x7f62c4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:19.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62cb55b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c40cd040 0x7f62c40684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62cb55b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c4067fa0 con 0x7f62c408a100 2023-10-10T19:25:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62cb55b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c40c5600 con 0x7f62c414fd90 2023-10-10T19:25:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62cb55b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4156b70 con 0x7f62c40cd040 2023-10-10T19:25:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62ca559700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f62c414fd90 0x7f62c4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:19.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62ca559700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f62c414fd90 0x7f62c4159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49880/0 (socket says 172.21.15.175:49880) 2023-10-10T19:25:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62ca559700 1 -- 172.21.15.175:0/2149787494 learned_addr learned my addr 172.21.15.175:0/2149787494 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.199+0000 7f62cad5a700 1 --2- 172.21.15.175:0/2149787494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c40cd040 0x7f62c40684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62ca559700 1 -- 172.21.15.175:0/2149787494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c40cd040 msgr2=0x7f62c40684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62ca559700 1 --2- 172.21.15.175:0/2149787494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c40cd040 0x7f62c40684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62ca559700 1 -- 172.21.15.175:0/2149787494 >> v1:172.21.15.191:6789/0 conn(0x7f62c408a100 legacy=0x7f62c40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:19.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62ca559700 1 -- 172.21.15.175:0/2149787494 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c4150160 con 0x7f62c414fd90 2023-10-10T19:25:19.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62ca559700 1 --2- 172.21.15.175:0/2149787494 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c414fd90 0x7f62c4159c30 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f62c00099b0 tx=0x7f62c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdce2e99ffa248f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:19.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62c9557700 1 -- 172.21.15.175:0/2149787494 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0009ea0 con 0x7f62c414fd90 2023-10-10T19:25:19.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62c9557700 1 -- 172.21.15.175:0/2149787494 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c000b860 con 0x7f62c414fd90 2023-10-10T19:25:19.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62c9557700 1 -- 172.21.15.175:0/2149787494 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000bc00 con 0x7f62c414fd90 2023-10-10T19:25:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62cb55b700 1 -- 172.21.15.175:0/2149787494 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c414fd90 msgr2=0x7f62c4159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62cb55b700 1 --2- 172.21.15.175:0/2149787494 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c414fd90 0x7f62c4159c30 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f62c00099b0 tx=0x7f62c0009250 comp rx=0 tx=0).stop 2023-10-10T19:25:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62cb55b700 1 -- 172.21.15.175:0/2149787494 shutdown_connections 2023-10-10T19:25:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62cb55b700 1 --2- 172.21.15.175:0/2149787494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c40cd040 0x7f62c40684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62cb55b700 1 --2- 172.21.15.175:0/2149787494 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c414fd90 0x7f62c4159c30 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62cb55b700 1 -- 172.21.15.175:0/2149787494 >> 172.21.15.175:0/2149787494 conn(0x7f62c4063d00 msgr2=0x7f62c40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:19.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62cb55b700 1 -- 172.21.15.175:0/2149787494 shutdown_connections 2023-10-10T19:25:19.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.200+0000 7f62cb55b700 1 -- 172.21.15.175:0/2149787494 wait complete. 2023-10-10T19:25:19.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.201+0000 7f62cb55b700 1 Processor -- start 2023-10-10T19:25:19.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62cb55b700 1 -- start start 2023-10-10T19:25:19.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62cb55b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c408a100 0x7f62c4157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:19.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62cb55b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c40cd040 0x7f62c41498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62cb55b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62c414fd90 0x7f62c4149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62cb55b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4166410 con 0x7f62c40cd040 2023-10-10T19:25:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62cb55b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4152ca0 con 0x7f62c414fd90 2023-10-10T19:25:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62cb55b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4152a00 con 0x7f62c408a100 2023-10-10T19:25:19.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62c9d58700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c40cd040 0x7f62c41498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62ca559700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c408a100 0x7f62c4157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62c9d58700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c40cd040 0x7f62c41498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49886/0 (socket says 172.21.15.175:49886) 2023-10-10T19:25:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62cad5a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62c414fd90 0x7f62c4149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.202+0000 7f62c9d58700 1 -- 172.21.15.175:0/1167184036 learned_addr learned my addr 172.21.15.175:0/1167184036 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62c9d58700 1 -- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c408a100 msgr2=0x7f62c4157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:19.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62c9d58700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c408a100 0x7f62c4157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62c9d58700 1 -- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62c414fd90 msgr2=0x7f62c4149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62c9d58700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62c414fd90 0x7f62c4149e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62c9d58700 1 -- 172.21.15.175:0/1167184036 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00094f0 con 0x7f62c40cd040 2023-10-10T19:25:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62c9d58700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c40cd040 0x7f62c41498d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f62b400c590 tx=0x7f62b400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62ba7fc700 1 -- 172.21.15.175:0/1167184036 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b4011590 con 0x7f62c40cd040 2023-10-10T19:25:19.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62ba7fc700 1 -- 172.21.15.175:0/1167184036 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b4011710 con 0x7f62c40cd040 2023-10-10T19:25:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c4152ec0 con 0x7f62c40cd040 2023-10-10T19:25:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62ba7fc700 1 -- 172.21.15.175:0/1167184036 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b4011ad0 con 0x7f62c40cd040 2023-10-10T19:25:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.203+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c4153420 con 0x7f62c40cd040 2023-10-10T19:25:19.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.204+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c4067570 con 0x7f62c40cd040 2023-10-10T19:25:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.204+0000 7f62ba7fc700 1 -- 172.21.15.175:0/1167184036 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f62b401b3b0 con 0x7f62c40cd040 2023-10-10T19:25:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.205+0000 7f62ba7fc700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f62a8066f30 0x7f62a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:19.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.205+0000 7f62ba7fc700 1 -- 172.21.15.175:0/1167184036 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f62b4084030 con 0x7f62c40cd040 2023-10-10T19:25:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.205+0000 7f62ca559700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f62a8066f30 0x7f62a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:19.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.205+0000 7f62ca559700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f62a8066f30 0x7f62a80693e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f62c00099b0 tx=0x7f62c0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.209+0000 7f62ba7fc700 1 -- 172.21.15.175:0/1167184036 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f62b4050b20 con 0x7f62c40cd040 2023-10-10T19:25:19.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.433+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62c4155f80 con 0x7f62c40cd040 2023-10-10T19:25:19.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.433+0000 7f62ba7fc700 1 -- 172.21.15.175:0/1167184036 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 294 v294) v1 ==== 95+0+5418 (secure 0 0 0) 0x7f62b4017020 con 0x7f62c40cd040 2023-10-10T19:25:19.434 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:19.434 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":294,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":294,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:17.981159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:19.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f62a8066f30 msgr2=0x7f62a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:19.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f62a8066f30 0x7f62a80693e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f62c00099b0 tx=0x7f62c0017000 comp rx=0 tx=0).stop 2023-10-10T19:25:19.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c40cd040 msgr2=0x7f62c41498d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:19.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c40cd040 0x7f62c41498d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f62b400c590 tx=0x7f62b400c8a0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 shutdown_connections 2023-10-10T19:25:19.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f62a8066f30 0x7f62a80693e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62c414fd90 0x7f62c4149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62c408a100 0x7f62c4157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 --2- 172.21.15.175:0/1167184036 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62c40cd040 0x7f62c41498d0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:19.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 >> 172.21.15.175:0/1167184036 conn(0x7f62c4063d00 msgr2=0x7f62c414ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:19.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 shutdown_connections 2023-10-10T19:25:19.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:19.435+0000 7f62cb55b700 1 -- 172.21.15.175:0/1167184036 wait complete. 2023-10-10T19:25:19.439 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 294 2023-10-10T19:25:19.447 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:21.448 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:21.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.572+0000 7f667bff2700 1 Processor -- start 2023-10-10T19:25:21.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.573+0000 7f667bff2700 1 -- start start 2023-10-10T19:25:21.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.573+0000 7f667bff2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f66740c5850 0x7f66740c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:21.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.573+0000 7f667bff2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740c6fb0 con 0x7f66740c4f20 2023-10-10T19:25:21.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f667bff2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740c6160 con 0x7f66740c12d0 2023-10-10T19:25:21.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f667bff2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6674063ea0 con 0x7f66740c5850 2023-10-10T19:25:21.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6679d8e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f66740c5850 0x7f66740c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:21.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6679d8e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f66740c5850 0x7f66740c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49888/0 (socket says 172.21.15.175:49888) 2023-10-10T19:25:21.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6679d8e700 1 -- 172.21.15.175:0/2840073383 learned_addr learned my addr 172.21.15.175:0/2840073383 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:21.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f667a58f700 1 --1- 172.21.15.175:0/2840073383 >> v1:172.21.15.175:6789/0 conn(0x7f66740c4f20 0x7f66740c7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47092/0 (socket says 172.21.15.175:47092) 2023-10-10T19:25:21.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6679d8e700 1 -- 172.21.15.175:0/2840073383 >> v1:172.21.15.191:6789/0 conn(0x7f66740c12d0 legacy=0x7f6674156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:21.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6679d8e700 1 -- 172.21.15.175:0/2840073383 >> v1:172.21.15.175:6789/0 conn(0x7f66740c4f20 legacy=0x7f66740c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:21.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6679d8e700 1 -- 172.21.15.175:0/2840073383 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6674063940 con 0x7f66740c5850 2023-10-10T19:25:21.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6678d8c700 1 -- 172.21.15.175:0/2840073383 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002352307 0 0) 0x7f66740c6fb0 con 0x7f66740c4f20 2023-10-10T19:25:21.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6679d8e700 1 --2- 172.21.15.175:0/2840073383 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c5850 0x7f66740c5c20 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f6664009f20 tx=0x7f6664009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8894c4351b6a2df6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:21.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6678d8c700 1 -- 172.21.15.175:0/2840073383 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666400e070 con 0x7f66740c5850 2023-10-10T19:25:21.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6678d8c700 1 -- 172.21.15.175:0/2840073383 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666400bad0 con 0x7f66740c5850 2023-10-10T19:25:21.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.574+0000 7f6678d8c700 1 -- 172.21.15.175:0/2840073383 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66640092e0 con 0x7f66740c5850 2023-10-10T19:25:21.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.575+0000 7f667bff2700 1 -- 172.21.15.175:0/2840073383 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c5850 msgr2=0x7f66740c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:21.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.575+0000 7f667bff2700 1 --2- 172.21.15.175:0/2840073383 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c5850 0x7f66740c5c20 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f6664009f20 tx=0x7f6664009840 comp rx=0 tx=0).stop 2023-10-10T19:25:21.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.575+0000 7f667bff2700 1 -- 172.21.15.175:0/2840073383 shutdown_connections 2023-10-10T19:25:21.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.575+0000 7f667bff2700 1 --2- 172.21.15.175:0/2840073383 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c5850 0x7f66740c5c20 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:21.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.575+0000 7f667bff2700 1 -- 172.21.15.175:0/2840073383 >> 172.21.15.175:0/2840073383 conn(0x7f667406dc70 msgr2=0x7f6674063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:21.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.575+0000 7f667bff2700 1 -- 172.21.15.175:0/2840073383 shutdown_connections 2023-10-10T19:25:21.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.575+0000 7f667bff2700 1 -- 172.21.15.175:0/2840073383 wait complete. 2023-10-10T19:25:21.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.576+0000 7f667bff2700 1 Processor -- start 2023-10-10T19:25:21.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.576+0000 7f667bff2700 1 -- start start 2023-10-10T19:25:21.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667bff2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66740c12d0 0x7f66740c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:21.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667bff2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c4f20 0x7f6674169860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:21.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667bff2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66740c5850 0x7f6674169da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667bff2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667416a2e0 con 0x7f66740c4f20 2023-10-10T19:25:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667bff2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6674156110 con 0x7f66740c12d0 2023-10-10T19:25:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667bff2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667416a4d0 con 0x7f66740c5850 2023-10-10T19:25:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c4f20 0x7f6674169860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:21.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c4f20 0x7f6674169860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49890/0 (socket says 172.21.15.175:49890) 2023-10-10T19:25:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 -- 172.21.15.175:0/1137238303 learned_addr learned my addr 172.21.15.175:0/1137238303 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f6679d8e700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66740c12d0 0x7f66740c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667a58f700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66740c5850 0x7f6674169da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:21.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 -- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66740c5850 msgr2=0x7f6674169da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66740c5850 0x7f6674169da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 -- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66740c12d0 msgr2=0x7f66740c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66740c12d0 0x7f66740c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 -- 172.21.15.175:0/1137238303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6664009ae0 con 0x7f66740c4f20 2023-10-10T19:25:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667a58f700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66740c5850 0x7f6674169da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667958d700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c4f20 0x7f6674169860 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f666800b610 tx=0x7f6668009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f6671ffb700 1 -- 172.21.15.175:0/1137238303 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666800f510 con 0x7f66740c4f20 2023-10-10T19:25:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f6671ffb700 1 -- 172.21.15.175:0/1137238303 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666800f6b0 con 0x7f66740c4f20 2023-10-10T19:25:21.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66740c92e0 con 0x7f66740c4f20 2023-10-10T19:25:21.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.577+0000 7f6671ffb700 1 -- 172.21.15.175:0/1137238303 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666800fa70 con 0x7f66740c4f20 2023-10-10T19:25:21.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.578+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66740c9cb0 con 0x7f66740c4f20 2023-10-10T19:25:21.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.579+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6674068bb0 con 0x7f66740c4f20 2023-10-10T19:25:21.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.579+0000 7f6671ffb700 1 -- 172.21.15.175:0/1137238303 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f666801a400 con 0x7f66740c4f20 2023-10-10T19:25:21.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.579+0000 7f6671ffb700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:21.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.580+0000 7f6671ffb700 1 -- 172.21.15.175:0/1137238303 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f6668084760 con 0x7f66740c4f20 2023-10-10T19:25:21.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.580+0000 7f6679d8e700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:21.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.580+0000 7f6679d8e700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6664009880 tx=0x7f6664017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:21.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.584+0000 7f6671ffb700 1 -- 172.21.15.175:0/1137238303 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66680512d0 con 0x7f66740c4f20 2023-10-10T19:25:21.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.809+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66740c9470 con 0x7f66740c4f20 2023-10-10T19:25:21.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.810+0000 7f6671ffb700 1 -- 172.21.15.175:0/1137238303 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 294 v294) v1 ==== 95+0+5418 (secure 0 0 0) 0x7f66680548f0 con 0x7f66740c4f20 2023-10-10T19:25:21.811 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:21.811 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":294,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":294,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:17.981159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:21.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 msgr2=0x7f66600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:21.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6664009880 tx=0x7f6664017000 comp rx=0 tx=0).stop 2023-10-10T19:25:21.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c4f20 msgr2=0x7f6674169860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:21.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c4f20 0x7f6674169860 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f666800b610 tx=0x7f6668009120 comp rx=0 tx=0).stop 2023-10-10T19:25:21.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 shutdown_connections 2023-10-10T19:25:21.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6660066f30 0x7f66600693e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:21.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66740c5850 0x7f6674169da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:21.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66740c12d0 0x7f66740c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:21.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 --2- 172.21.15.175:0/1137238303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f66740c4f20 0x7f6674169860 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:21.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 >> 172.21.15.175:0/1137238303 conn(0x7f667406dc70 msgr2=0x7f667406b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:21.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 shutdown_connections 2023-10-10T19:25:21.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:21.812+0000 7f667bff2700 1 -- 172.21.15.175:0/1137238303 wait complete. 2023-10-10T19:25:21.815 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 294 2023-10-10T19:25:21.824 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:23.825 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:23.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.948+0000 7f989c807700 1 Processor -- start 2023-10-10T19:25:23.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.948+0000 7f989c807700 1 -- start start 2023-10-10T19:25:23.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f989c807700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f98980b1fc0 0x7f9898155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:23.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f989c807700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98980b2390 0x7f9898068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:23.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f989c807700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98981574b0 con 0x7f9898068bd0 2023-10-10T19:25:23.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f989c807700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9898150770 con 0x7f98980b1fc0 2023-10-10T19:25:23.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f989c807700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98980afad0 con 0x7f98980b2390 2023-10-10T19:25:23.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f9896d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9898068bd0 0x7f98980aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47096/0 (socket says 172.21.15.175:47096) 2023-10-10T19:25:23.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f9896d9d700 1 -- 172.21.15.175:0/4106394004 learned_addr learned my addr 172.21.15.175:0/4106394004 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:23.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f989659c700 1 --2- 172.21.15.175:0/4106394004 >> v2:172.21.15.191:3300/0 conn(0x7f98980b1fc0 0x7f9898155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:23.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f9895d9b700 1 --2- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98980b2390 0x7f9898068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:23.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.949+0000 7f989559a700 1 -- 172.21.15.175:0/4106394004 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300641892 0 0) 0x7f98981574b0 con 0x7f9898068bd0 2023-10-10T19:25:23.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989559a700 1 -- 172.21.15.175:0/4106394004 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9880003570 con 0x7f9898068bd0 2023-10-10T19:25:23.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989659c700 1 -- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98980b2390 msgr2=0x7f9898068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:23.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989659c700 1 --2- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98980b2390 0x7f9898068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:23.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989659c700 1 -- 172.21.15.175:0/4106394004 >> v1:172.21.15.175:6789/0 conn(0x7f9898068bd0 legacy=0x7f98980aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:23.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989659c700 1 -- 172.21.15.175:0/4106394004 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980af4c0 con 0x7f98980b1fc0 2023-10-10T19:25:23.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f9895d9b700 1 --2- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98980b2390 0x7f9898068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:25:23.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989659c700 1 --2- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b1fc0 0x7f9898155fb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9888009e60 tx=0x7f9888009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=835b4396abf9df49 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:23.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989559a700 1 -- 172.21.15.175:0/4106394004 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988800e070 con 0x7f98980b1fc0 2023-10-10T19:25:23.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989559a700 1 -- 172.21.15.175:0/4106394004 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988800bb40 con 0x7f98980b1fc0 2023-10-10T19:25:23.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.950+0000 7f989559a700 1 -- 172.21.15.175:0/4106394004 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888012430 con 0x7f98980b1fc0 2023-10-10T19:25:23.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.951+0000 7f989c807700 1 -- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b1fc0 msgr2=0x7f9898155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:23.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.951+0000 7f989c807700 1 --2- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b1fc0 0x7f9898155fb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9888009e60 tx=0x7f9888009780 comp rx=0 tx=0).stop 2023-10-10T19:25:23.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.951+0000 7f989c807700 1 -- 172.21.15.175:0/4106394004 shutdown_connections 2023-10-10T19:25:23.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.951+0000 7f989c807700 1 --2- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b1fc0 0x7f9898155fb0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:23.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.951+0000 7f989c807700 1 --2- 172.21.15.175:0/4106394004 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98980b2390 0x7f9898068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:23.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.951+0000 7f989c807700 1 -- 172.21.15.175:0/4106394004 >> 172.21.15.175:0/4106394004 conn(0x7f9898063230 msgr2=0x7f9898063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:23.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.951+0000 7f989c807700 1 -- 172.21.15.175:0/4106394004 shutdown_connections 2023-10-10T19:25:23.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.951+0000 7f989c807700 1 -- 172.21.15.175:0/4106394004 wait complete. 2023-10-10T19:25:23.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.952+0000 7f989c807700 1 Processor -- start 2023-10-10T19:25:23.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.952+0000 7f989c807700 1 -- start start 2023-10-10T19:25:23.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.952+0000 7f989c807700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9898068bd0 0x7f98980c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:23.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.952+0000 7f989c807700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98980b1fc0 0x7f98980c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:23.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.952+0000 7f989c807700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b2390 0x7f98980cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:23.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.952+0000 7f989c807700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989814e750 con 0x7f98980b1fc0 2023-10-10T19:25:23.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.952+0000 7f989c807700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989815a120 con 0x7f98980b2390 2023-10-10T19:25:23.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.952+0000 7f989c807700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98981697a0 con 0x7f9898068bd0 2023-10-10T19:25:23.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98980b1fc0 0x7f98980c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:23.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98980b1fc0 0x7f98980c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49894/0 (socket says 172.21.15.175:49894) 2023-10-10T19:25:23.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 -- 172.21.15.175:0/3022218145 learned_addr learned my addr 172.21.15.175:0/3022218145 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:23.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9896d9d700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b2390 0x7f98980cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:23.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f989659c700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9898068bd0 0x7f98980c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:23.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 -- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9898068bd0 msgr2=0x7f98980c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:23.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9898068bd0 0x7f98980c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:23.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 -- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b2390 msgr2=0x7f98980cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:23.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b2390 0x7f98980cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:23.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 -- 172.21.15.175:0/3022218145 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c009420 con 0x7f98980b1fc0 2023-10-10T19:25:23.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9896d9d700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b2390 0x7f98980cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:23.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f9895d9b700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98980b1fc0 0x7f98980c3db0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f988c00ecb0 tx=0x7f988c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:23.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f98867fc700 1 -- 172.21.15.175:0/3022218145 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c00c9d0 con 0x7f98980b1fc0 2023-10-10T19:25:23.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f98867fc700 1 -- 172.21.15.175:0/3022218145 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988c00cb30 con 0x7f98980b1fc0 2023-10-10T19:25:23.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9888009a20 con 0x7f98980b1fc0 2023-10-10T19:25:23.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f98867fc700 1 -- 172.21.15.175:0/3022218145 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c00ceb0 con 0x7f98980b1fc0 2023-10-10T19:25:23.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.953+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989814ea10 con 0x7f98980b1fc0 2023-10-10T19:25:23.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.954+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9898067570 con 0x7f98980b1fc0 2023-10-10T19:25:23.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.954+0000 7f98867fc700 1 -- 172.21.15.175:0/3022218145 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f988c013680 con 0x7f98980b1fc0 2023-10-10T19:25:23.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.955+0000 7f98867fc700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9878066f30 0x7f98780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:23.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.955+0000 7f98867fc700 1 -- 172.21.15.175:0/3022218145 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f988c085050 con 0x7f98980b1fc0 2023-10-10T19:25:23.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.955+0000 7f989659c700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9878066f30 0x7f98780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:23.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.956+0000 7f989659c700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9878066f30 0x7f98780693e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f98880097c0 tx=0x7f9888019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:23.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:23.959+0000 7f98867fc700 1 -- 172.21.15.175:0/3022218145 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f988c0555d0 con 0x7f98980b1fc0 2023-10-10T19:25:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.189+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9898069060 con 0x7f98980b1fc0 2023-10-10T19:25:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.190+0000 7f98867fc700 1 -- 172.21.15.175:0/3022218145 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 295 v295) v1 ==== 95+0+5418 (secure 0 0 0) 0x7f988c055160 con 0x7f98980b1fc0 2023-10-10T19:25:24.190 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:24.191 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":295,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:24.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9878066f30 msgr2=0x7f98780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:24.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9878066f30 0x7f98780693e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f98880097c0 tx=0x7f9888019000 comp rx=0 tx=0).stop 2023-10-10T19:25:24.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98980b1fc0 msgr2=0x7f98980c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:24.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98980b1fc0 0x7f98980c3db0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f988c00ecb0 tx=0x7f988c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:24.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 shutdown_connections 2023-10-10T19:25:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9878066f30 0x7f98780693e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98980b2390 0x7f98980cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9898068bd0 0x7f98980c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 --2- 172.21.15.175:0/3022218145 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98980b1fc0 0x7f98980c3db0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 >> 172.21.15.175:0/3022218145 conn(0x7f9898063230 msgr2=0x7f98981508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 shutdown_connections 2023-10-10T19:25:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:24.192+0000 7f989c807700 1 -- 172.21.15.175:0/3022218145 wait complete. 2023-10-10T19:25:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 295 2023-10-10T19:25:24.204 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=7252 2023-10-10T19:25:24.205 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.f 2023-10-10T19:25:26.206 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-10T19:25:26.206 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-10T19:25:26.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-10T19:25:26.209 INFO:tasks.ceph.mds.f:Started 2023-10-10T19:25:26.209 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:26.289 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-10T19:25:26.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.333+0000 7f81831ac700 1 Processor -- start 2023-10-10T19:25:26.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.333+0000 7f81831ac700 1 -- start start 2023-10-10T19:25:26.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81831ac700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f817c08a100 0x7f817c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:26.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81831ac700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c067fa0 con 0x7f817c14fd90 2023-10-10T19:25:26.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81831ac700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c156b70 con 0x7f817c0cd040 2023-10-10T19:25:26.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81831ac700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817c0c5600 con 0x7f817c08a100 2023-10-10T19:25:26.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81821aa700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f817c14fd90 0x7f817c0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37834/0 (socket says 172.21.15.175:37834) 2023-10-10T19:25:26.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81821aa700 1 -- 172.21.15.175:0/883965198 learned_addr learned my addr 172.21.15.175:0/883965198 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:26.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81819a9700 1 --2- 172.21.15.175:0/883965198 >> v2:172.21.15.191:3300/0 conn(0x7f817c08a100 0x7f817c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:26.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81811a8700 1 -- 172.21.15.175:0/883965198 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088753828 0 0) 0x7f817c067fa0 con 0x7f817c14fd90 2023-10-10T19:25:26.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.334+0000 7f81811a8700 1 -- 172.21.15.175:0/883965198 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8168003570 con 0x7f817c14fd90 2023-10-10T19:25:26.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81819a9700 1 -- 172.21.15.175:0/883965198 >> v1:172.21.15.191:6789/0 conn(0x7f817c0cd040 legacy=0x7f817c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:26.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81819a9700 1 -- 172.21.15.175:0/883965198 >> v1:172.21.15.175:6789/0 conn(0x7f817c14fd90 legacy=0x7f817c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:26.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81811a8700 1 -- 172.21.15.175:0/883965198 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3078359199 0 0) 0x7f8168003570 con 0x7f817c14fd90 2023-10-10T19:25:26.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81819a9700 1 -- 172.21.15.175:0/883965198 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c150160 con 0x7f817c08a100 2023-10-10T19:25:26.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81819a9700 1 --2- 172.21.15.175:0/883965198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c08a100 0x7f817c159c30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f816c0098e0 tx=0x7f816c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9980bd6f402939bb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:26.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81811a8700 1 -- 172.21.15.175:0/883965198 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178001470 con 0x7f817c08a100 2023-10-10T19:25:26.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81811a8700 1 -- 172.21.15.175:0/883965198 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c011420 con 0x7f817c08a100 2023-10-10T19:25:26.340 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81811a8700 1 -- 172.21.15.175:0/883965198 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c0117c0 con 0x7f817c08a100 2023-10-10T19:25:26.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81831ac700 1 -- 172.21.15.175:0/883965198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c08a100 msgr2=0x7f817c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:26.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81831ac700 1 --2- 172.21.15.175:0/883965198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c08a100 0x7f817c159c30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f816c0098e0 tx=0x7f816c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:26.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81831ac700 1 -- 172.21.15.175:0/883965198 shutdown_connections 2023-10-10T19:25:26.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.335+0000 7f81831ac700 1 --2- 172.21.15.175:0/883965198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c08a100 0x7f817c159c30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:26.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.336+0000 7f81831ac700 1 -- 172.21.15.175:0/883965198 >> 172.21.15.175:0/883965198 conn(0x7f817c063d00 msgr2=0x7f817c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:26.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.336+0000 7f81831ac700 1 -- 172.21.15.175:0/883965198 shutdown_connections 2023-10-10T19:25:26.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.336+0000 7f81831ac700 1 -- 172.21.15.175:0/883965198 wait complete. 2023-10-10T19:25:26.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81831ac700 1 Processor -- start 2023-10-10T19:25:26.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81831ac700 1 -- start start 2023-10-10T19:25:26.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81831ac700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f817c08a100 0x7f817c1660b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:26.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81831ac700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f817c0cd040 0x7f817c150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:26.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81831ac700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c14fd90 0x7f817c0c3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:26.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81831ac700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c150a90 con 0x7f817c08a100 2023-10-10T19:25:26.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81831ac700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c150c90 con 0x7f817c14fd90 2023-10-10T19:25:26.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81831ac700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c150f30 con 0x7f817c0cd040 2023-10-10T19:25:26.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81821aa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f817c08a100 0x7f817c1660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:26.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81821aa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f817c08a100 0x7f817c1660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33316/0 (socket says 172.21.15.175:33316) 2023-10-10T19:25:26.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.337+0000 7f81821aa700 1 -- 172.21.15.175:0/1429175787 learned_addr learned my addr 172.21.15.175:0/1429175787 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:26.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81829ab700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c14fd90 0x7f817c0c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:26.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81819a9700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f817c0cd040 0x7f817c150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:26.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81821aa700 1 -- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f817c0cd040 msgr2=0x7f817c150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:26.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81821aa700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f817c0cd040 0x7f817c150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:26.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81821aa700 1 -- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c14fd90 msgr2=0x7f817c0c3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:26.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81821aa700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c14fd90 0x7f817c0c3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:26.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81821aa700 1 -- 172.21.15.175:0/1429175787 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c009420 con 0x7f817c08a100 2023-10-10T19:25:26.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81819a9700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f817c0cd040 0x7f817c150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:25:26.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81821aa700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f817c08a100 0x7f817c1660b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f817800cb90 tx=0x7f8178001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81727fc700 1 -- 172.21.15.175:0/1429175787 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178007150 con 0x7f817c08a100 2023-10-10T19:25:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81727fc700 1 -- 172.21.15.175:0/1429175787 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81780072d0 con 0x7f817c08a100 2023-10-10T19:25:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c1511f0 con 0x7f817c08a100 2023-10-10T19:25:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81727fc700 1 -- 172.21.15.175:0/1429175787 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178007690 con 0x7f817c08a100 2023-10-10T19:25:26.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.338+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c151d50 con 0x7f817c08a100 2023-10-10T19:25:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.339+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c067570 con 0x7f817c08a100 2023-10-10T19:25:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.339+0000 7f81727fc700 1 -- 172.21.15.175:0/1429175787 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8178004a40 con 0x7f817c08a100 2023-10-10T19:25:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.340+0000 7f81727fc700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8160066f30 0x7f81600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.340+0000 7f81727fc700 1 -- 172.21.15.175:0/1429175787 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f8178084450 con 0x7f817c08a100 2023-10-10T19:25:26.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.340+0000 7f81819a9700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8160066f30 0x7f81600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:26.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.340+0000 7f81819a9700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8160066f30 0x7f81600693e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f817c071760 tx=0x7f816c016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:26.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.344+0000 7f81727fc700 1 -- 172.21.15.175:0/1429175787 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8178050fc0 con 0x7f817c08a100 2023-10-10T19:25:26.354 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:25:26.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:25:26.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.577+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f817c155f80 con 0x7f817c08a100 2023-10-10T19:25:26.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.578+0000 7f81727fc700 1 -- 172.21.15.175:0/1429175787 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 295 v295) v1 ==== 95+0+5418 (secure 0 0 0) 0x7f81780545e0 con 0x7f817c08a100 2023-10-10T19:25:26.578 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:26.579 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":295,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:26.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8160066f30 msgr2=0x7f81600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8160066f30 0x7f81600693e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f817c071760 tx=0x7f816c016000 comp rx=0 tx=0).stop 2023-10-10T19:25:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f817c08a100 msgr2=0x7f817c1660b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f817c08a100 0x7f817c1660b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f817800cb90 tx=0x7f8178001a30 comp rx=0 tx=0).stop 2023-10-10T19:25:26.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 shutdown_connections 2023-10-10T19:25:26.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8160066f30 0x7f81600693e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:26.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f817c14fd90 0x7f817c0c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:26.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f817c08a100 0x7f817c1660b0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:26.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 --2- 172.21.15.175:0/1429175787 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f817c0cd040 0x7f817c150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:26.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 >> 172.21.15.175:0/1429175787 conn(0x7f817c063d00 msgr2=0x7f817c153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:26.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 shutdown_connections 2023-10-10T19:25:26.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:26.580+0000 7f81831ac700 1 -- 172.21.15.175:0/1429175787 wait complete. 2023-10-10T19:25:26.583 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 295 2023-10-10T19:25:26.592 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-10T19:25:28.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:28.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.719+0000 7f9fb9fb2700 1 Processor -- start 2023-10-10T19:25:28.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.719+0000 7f9fb9fb2700 1 -- start start 2023-10-10T19:25:28.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb9fb2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9fb40b1fc0 0x7f9fb4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:28.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb9fb2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb41574b0 con 0x7f9fb40aff60 2023-10-10T19:25:28.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb9fb2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb40afad0 con 0x7f9fb40b2390 2023-10-10T19:25:28.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb9fb2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb4150770 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb37fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9fb40b1fc0 0x7f9fb4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb37fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9fb40b1fc0 0x7f9fb4155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33328/0 (socket says 172.21.15.175:33328) 2023-10-10T19:25:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb37fe700 1 -- 172.21.15.175:0/3286227526 learned_addr learned my addr 172.21.15.175:0/3286227526 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:28.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb3fff700 1 --1- 172.21.15.175:0/3286227526 >> v1:172.21.15.175:6789/0 conn(0x7f9fb40aff60 0x7f9fb4159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37850/0 (socket says 172.21.15.175:37850) 2023-10-10T19:25:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb37fe700 1 -- 172.21.15.175:0/3286227526 >> v1:172.21.15.191:6789/0 conn(0x7f9fb40b2390 legacy=0x7f9fb4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb37fe700 1 -- 172.21.15.175:0/3286227526 >> v1:172.21.15.175:6789/0 conn(0x7f9fb40aff60 legacy=0x7f9fb4159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb37fe700 1 -- 172.21.15.175:0/3286227526 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb40af4c0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb27fc700 1 -- 172.21.15.175:0/3286227526 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793242544 0 0) 0x7f9fb41574b0 con 0x7f9fb40aff60 2023-10-10T19:25:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.720+0000 7f9fb37fe700 1 --2- 172.21.15.175:0/3286227526 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4155fb0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009f20 tx=0x7f9fa4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b5c8bc164599e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:28.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb27fc700 1 -- 172.21.15.175:0/3286227526 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa400e070 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb27fc700 1 -- 172.21.15.175:0/3286227526 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa400bb10 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb27fc700 1 -- 172.21.15.175:0/3286227526 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40092e0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/3286227526 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 msgr2=0x7f9fb4155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb9fb2700 1 --2- 172.21.15.175:0/3286227526 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4155fb0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009f20 tx=0x7f9fa4009840 comp rx=0 tx=0).stop 2023-10-10T19:25:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/3286227526 shutdown_connections 2023-10-10T19:25:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb9fb2700 1 --2- 172.21.15.175:0/3286227526 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4155fb0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/3286227526 >> 172.21.15.175:0/3286227526 conn(0x7f9fb4063230 msgr2=0x7f9fb4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/3286227526 shutdown_connections 2023-10-10T19:25:28.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.721+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/3286227526 wait complete. 2023-10-10T19:25:28.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.722+0000 7f9fb9fb2700 1 Processor -- start 2023-10-10T19:25:28.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.722+0000 7f9fb9fb2700 1 -- start start 2023-10-10T19:25:28.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb9fb2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40aff60 0x7f9fb4166360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb9fb2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb9fb2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40b2390 0x7f9fb4158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb9fb2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb40cd5e0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb9fb2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb414f2d0 con 0x7f9fb40b2390 2023-10-10T19:25:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb9fb2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb414e7e0 con 0x7f9fb40aff60 2023-10-10T19:25:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33344/0 (socket says 172.21.15.175:33344) 2023-10-10T19:25:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb2ffd700 1 -- 172.21.15.175:0/2310920431 learned_addr learned my addr 172.21.15.175:0/2310920431 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb3fff700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40b2390 0x7f9fb4158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.723+0000 7f9fb37fe700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40aff60 0x7f9fb4166360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9fb2ffd700 1 -- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40aff60 msgr2=0x7f9fb4166360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9fb2ffd700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40aff60 0x7f9fb4166360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9fb2ffd700 1 -- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40b2390 msgr2=0x7f9fb4158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9fb2ffd700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40b2390 0x7f9fb4158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9fb2ffd700 1 -- 172.21.15.175:0/2310920431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa4009ae0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9fb2ffd700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4159bf0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800c510 tx=0x7f9fa800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9f9b7fe700 1 -- 172.21.15.175:0/2310920431 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8011590 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9f9b7fe700 1 -- 172.21.15.175:0/2310920431 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa8011710 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb4158690 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9f9b7fe700 1 -- 172.21.15.175:0/2310920431 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8011ad0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.724+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb414ed10 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.725+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb41588b0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.725+0000 7f9f9b7fe700 1 -- 172.21.15.175:0/2310920431 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9fa801b3b0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.726+0000 7f9f9b7fe700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9fa0066f60 0x7f9fa0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.726+0000 7f9f9b7fe700 1 -- 172.21.15.175:0/2310920431 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f9fa80840e0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.726+0000 7f9fb37fe700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9fa0066f60 0x7f9fa0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.726+0000 7f9fb37fe700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9fa0066f60 0x7f9fa0069410 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009880 tx=0x7f9fa4018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.730+0000 7f9f9b7fe700 1 -- 172.21.15.175:0/2310920431 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9fb41588b0 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.955+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fb4158880 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.956+0000 7f9f9b7fe700 1 -- 172.21.15.175:0/2310920431 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f9fa8017070 con 0x7f9fb40b1fc0 2023-10-10T19:25:28.957 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:28.958 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":296,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7771,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":296}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:28.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9fa0066f60 msgr2=0x7f9fa0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:28.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9fa0066f60 0x7f9fa0069410 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009880 tx=0x7f9fa4018000 comp rx=0 tx=0).stop 2023-10-10T19:25:28.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 msgr2=0x7f9fb4159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:28.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4159bf0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800c510 tx=0x7f9fa800c820 comp rx=0 tx=0).stop 2023-10-10T19:25:28.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 shutdown_connections 2023-10-10T19:25:28.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9fa0066f60 0x7f9fa0069410 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:28.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb40b2390 0x7f9fb4158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:28.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fb40aff60 0x7f9fb4166360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:28.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 --2- 172.21.15.175:0/2310920431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fb40b1fc0 0x7f9fb4159bf0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:28.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 >> 172.21.15.175:0/2310920431 conn(0x7f9fb4063230 msgr2=0x7f9fb414fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:28.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 shutdown_connections 2023-10-10T19:25:28.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:28.958+0000 7f9fb9fb2700 1 -- 172.21.15.175:0/2310920431 wait complete. 2023-10-10T19:25:28.962 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 296 2023-10-10T19:25:28.971 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-10T19:25:28.971 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2023-10-10T19:25:30.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:31.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.099+0000 7f952b703700 1 Processor -- start 2023-10-10T19:25:31.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.099+0000 7f952b703700 1 -- start start 2023-10-10T19:25:31.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.100+0000 7f952b703700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9524155fb0 0x7f9524159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.100+0000 7f952b703700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95241574b0 con 0x7f95240b2390 2023-10-10T19:25:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.100+0000 7f952b703700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240afad0 con 0x7f95240b1fc0 2023-10-10T19:25:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.100+0000 7f952b703700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9524150770 con 0x7f9524155fb0 2023-10-10T19:25:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.100+0000 7f9528c9e700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f95240b2390 0x7f95240aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37866/0 (socket says 172.21.15.175:37866) 2023-10-10T19:25:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.100+0000 7f9528c9e700 1 -- 172.21.15.175:0/3431621094 learned_addr learned my addr 172.21.15.175:0/3431621094 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.100+0000 7f9529ca0700 1 --2- 172.21.15.175:0/3431621094 >> v2:172.21.15.191:3300/0 conn(0x7f9524155fb0 0x7f9524159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59969505 0 0) 0x7f95241574b0 con 0x7f95240b2390 2023-10-10T19:25:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510003570 con 0x7f95240b2390 2023-10-10T19:25:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1985449902 0 0) 0x7f9510003570 con 0x7f95240b2390 2023-10-10T19:25:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 >> v2:172.21.15.191:3300/0 conn(0x7f9524155fb0 msgr2=0x7f9524159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 --2- 172.21.15.175:0/3431621094 >> v2:172.21.15.191:3300/0 conn(0x7f9524155fb0 0x7f9524159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 >> v1:172.21.15.191:6789/0 conn(0x7f95240b1fc0 legacy=0x7f9524068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95240af4c0 con 0x7f95240b2390 2023-10-10T19:25:31.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f9514001470 con 0x7f95240b2390 2023-10-10T19:25:31.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 >> v1:172.21.15.175:6789/0 conn(0x7f95240b2390 legacy=0x7f95240aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9510003a60 0x7f9510003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9510004810 0x7f9510004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95100051c0 0x7f951000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9524150770 con 0x7f95100051c0 2023-10-10T19:25:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951000c0d0 con 0x7f9510003a60 2023-10-10T19:25:31.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951000c290 con 0x7f9510004810 2023-10-10T19:25:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 --2- 172.21.15.175:0/3431621094 >> v2:172.21.15.191:3300/0 conn(0x7f9524155fb0 0x7f9524159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:25:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95100051c0 0x7f951000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f952949f700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9510003a60 0x7f9510003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9528c9e700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9510004810 0x7f9510004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 -- 172.21.15.175:0/3431621094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9510004810 msgr2=0x7f9510004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:31.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9510004810 0x7f9510004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 -- 172.21.15.175:0/3431621094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9510003a60 msgr2=0x7f9510003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:31.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9510003a60 0x7f9510003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 -- 172.21.15.175:0/3431621094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9510003570 con 0x7f95100051c0 2023-10-10T19:25:31.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 -- 172.21.15.175:0/3431621094 reap_dead start 2023-10-10T19:25:31.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.101+0000 7f9529ca0700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95100051c0 0x7f951000ba60 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f951c009db0 tx=0x7f951c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab45c48135bb8d29 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:31.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c009740 con 0x7f95100051c0 2023-10-10T19:25:31.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c00e920 con 0x7f95100051c0 2023-10-10T19:25:31.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f951bfff700 1 -- 172.21.15.175:0/3431621094 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c00eca0 con 0x7f95100051c0 2023-10-10T19:25:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f952b703700 1 -- 172.21.15.175:0/3431621094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95100051c0 msgr2=0x7f951000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f952b703700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95100051c0 0x7f951000ba60 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f951c009db0 tx=0x7f951c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f952b703700 1 -- 172.21.15.175:0/3431621094 shutdown_connections 2023-10-10T19:25:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f952b703700 1 --2- 172.21.15.175:0/3431621094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95100051c0 0x7f951000ba60 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f952b703700 1 -- 172.21.15.175:0/3431621094 >> 172.21.15.175:0/3431621094 conn(0x7f9524063230 msgr2=0x7f9524063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f952b703700 1 -- 172.21.15.175:0/3431621094 shutdown_connections 2023-10-10T19:25:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.102+0000 7f952b703700 1 -- 172.21.15.175:0/3431621094 wait complete. 2023-10-10T19:25:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.103+0000 7f952b703700 1 Processor -- start 2023-10-10T19:25:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.103+0000 7f952b703700 1 -- start start 2023-10-10T19:25:31.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f952b703700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95100051c0 0x7f95240ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f952b703700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f952414c350 0x7f9524165fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f952b703700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f952414bb20 0x7f9524166520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f952b703700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952416a0a0 con 0x7f952414bb20 2023-10-10T19:25:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f952b703700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95240c3f70 con 0x7f952414c350 2023-10-10T19:25:31.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f952b703700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95240c3c90 con 0x7f95100051c0 2023-10-10T19:25:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f9529ca0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f952414bb20 0x7f9524166520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f9529ca0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f952414bb20 0x7f9524166520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33356/0 (socket says 172.21.15.175:33356) 2023-10-10T19:25:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f9529ca0700 1 -- 172.21.15.175:0/2349143433 learned_addr learned my addr 172.21.15.175:0/2349143433 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f952949f700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95100051c0 0x7f95240ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.104+0000 7f9528c9e700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f952414c350 0x7f9524165fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f9529ca0700 1 -- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95100051c0 msgr2=0x7f95240ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f9529ca0700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95100051c0 0x7f95240ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f9529ca0700 1 -- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f952414c350 msgr2=0x7f9524165fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f9529ca0700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f952414c350 0x7f9524165fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f9529ca0700 1 -- 172.21.15.175:0/2349143433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9524169700 con 0x7f952414bb20 2023-10-10T19:25:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f9529ca0700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f952414bb20 0x7f9524166520 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f952408a030 tx=0x7f951c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f95197fa700 1 -- 172.21.15.175:0/2349143433 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c00eca0 con 0x7f952414bb20 2023-10-10T19:25:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f95197fa700 1 -- 172.21.15.175:0/2349143433 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c021040 con 0x7f952414bb20 2023-10-10T19:25:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f95197fa700 1 -- 172.21.15.175:0/2349143433 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c025640 con 0x7f952414bb20 2023-10-10T19:25:31.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95241699e0 con 0x7f952414bb20 2023-10-10T19:25:31.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.105+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9524155cb0 con 0x7f952414bb20 2023-10-10T19:25:31.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.106+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9524169e80 con 0x7f952414bb20 2023-10-10T19:25:31.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.106+0000 7f95197fa700 1 -- 172.21.15.175:0/2349143433 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f951c025a30 con 0x7f952414bb20 2023-10-10T19:25:31.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.107+0000 7f95197fa700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9508066f30 0x7f95080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.107+0000 7f95197fa700 1 -- 172.21.15.175:0/2349143433 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f951c086ce0 con 0x7f952414bb20 2023-10-10T19:25:31.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.107+0000 7f952949f700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9508066f30 0x7f95080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.107+0000 7f952949f700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9508066f30 0x7f95080693e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9520001750 tx=0x7f9520001640 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:31.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.111+0000 7f95197fa700 1 -- 172.21.15.175:0/2349143433 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f951c053890 con 0x7f952414bb20 2023-10-10T19:25:31.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.337+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9524067570 con 0x7f952414bb20 2023-10-10T19:25:31.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.338+0000 7f95197fa700 1 -- 172.21.15.175:0/2349143433 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f951c056eb0 con 0x7f952414bb20 2023-10-10T19:25:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:31.340 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":296,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7771,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":296}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:31.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9508066f30 msgr2=0x7f95080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9508066f30 0x7f95080693e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9520001750 tx=0x7f9520001640 comp rx=0 tx=0).stop 2023-10-10T19:25:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f952414bb20 msgr2=0x7f9524166520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f952414bb20 0x7f9524166520 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f952408a030 tx=0x7f951c019000 comp rx=0 tx=0).stop 2023-10-10T19:25:31.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 shutdown_connections 2023-10-10T19:25:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9508066f30 0x7f95080693e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f952414bb20 0x7f9524166520 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f95100051c0 0x7f95240ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 --2- 172.21.15.175:0/2349143433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f952414c350 0x7f9524165fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 >> 172.21.15.175:0/2349143433 conn(0x7f9524063230 msgr2=0x7f95240cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 shutdown_connections 2023-10-10T19:25:31.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.340+0000 7f952b703700 1 -- 172.21.15.175:0/2349143433 wait complete. 2023-10-10T19:25:31.345 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 296 2023-10-10T19:25:31.353 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=3) 2023-10-10T19:25:31.353 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-10T19:25:31.354 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:25:31.365 INFO:tasks.ceph.mds.d:Stopped 2023-10-10T19:25:31.366 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:31.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.479+0000 7f51e52d7700 1 Processor -- start 2023-10-10T19:25:31.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.479+0000 7f51e52d7700 1 -- start start 2023-10-10T19:25:31.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51e52d7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f51e006f1c0 0x7f51e0068900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51e52d7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f51e006f590 0x7f51e00c3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51e52d7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 0x7f51e0156300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51e52d7700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e0068e40 con 0x7f51e006f590 2023-10-10T19:25:31.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51e52d7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e00c21f0 con 0x7f51e006f1c0 2023-10-10T19:25:31.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51e52d7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e00632d0 con 0x7f51e0155eb0 2023-10-10T19:25:31.484 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51d67fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f51e006f590 0x7f51e00c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51d67fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f51e006f590 0x7f51e00c3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33366/0 (socket says 172.21.15.175:33366) 2023-10-10T19:25:31.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51d67fc700 1 -- 172.21.15.175:0/1131699840 learned_addr learned my addr 172.21.15.175:0/1131699840 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:31.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51df7fe700 1 --2- 172.21.15.175:0/1131699840 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 0x7f51e0156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51deffd700 1 --2- 172.21.15.175:0/1131699840 >> v2:172.21.15.191:3300/0 conn(0x7f51e006f1c0 0x7f51e0068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51d67fc700 1 -- 172.21.15.175:0/1131699840 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 msgr2=0x7f51e0156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:31.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51d67fc700 1 --2- 172.21.15.175:0/1131699840 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 0x7f51e0156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51d67fc700 1 -- 172.21.15.175:0/1131699840 >> v2:172.21.15.191:3300/0 conn(0x7f51e006f1c0 msgr2=0x7f51e0068900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:31.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51d67fc700 1 --2- 172.21.15.175:0/1131699840 >> v2:172.21.15.191:3300/0 conn(0x7f51e006f1c0 0x7f51e0068900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.480+0000 7f51d67fc700 1 -- 172.21.15.175:0/1131699840 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e0063870 con 0x7f51e006f590 2023-10-10T19:25:31.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51d67fc700 1 --2- 172.21.15.175:0/1131699840 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e00c3440 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f51e00c89e0 tx=0x7f51c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d669cca02204d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:31.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51de7fc700 1 -- 172.21.15.175:0/1131699840 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800b6d0 con 0x7f51e006f590 2023-10-10T19:25:31.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51de7fc700 1 -- 172.21.15.175:0/1131699840 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c800b830 con 0x7f51e006f590 2023-10-10T19:25:31.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51de7fc700 1 -- 172.21.15.175:0/1131699840 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800bbf0 con 0x7f51e006f590 2023-10-10T19:25:31.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 -- 172.21.15.175:0/1131699840 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 msgr2=0x7f51e00c3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.487 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1131699840 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e00c3440 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f51e00c89e0 tx=0x7f51c8009250 comp rx=0 tx=0).stop 2023-10-10T19:25:31.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 -- 172.21.15.175:0/1131699840 shutdown_connections 2023-10-10T19:25:31.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1131699840 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 0x7f51e0156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1131699840 >> v2:172.21.15.191:3300/0 conn(0x7f51e006f1c0 0x7f51e0068900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1131699840 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e00c3440 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.488 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 -- 172.21.15.175:0/1131699840 >> 172.21.15.175:0/1131699840 conn(0x7f51e00c95a0 msgr2=0x7f51e00c5540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:31.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 -- 172.21.15.175:0/1131699840 shutdown_connections 2023-10-10T19:25:31.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.481+0000 7f51e52d7700 1 -- 172.21.15.175:0/1131699840 wait complete. 2023-10-10T19:25:31.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.482+0000 7f51e52d7700 1 Processor -- start 2023-10-10T19:25:31.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51e52d7700 1 -- start start 2023-10-10T19:25:31.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51e52d7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51e006f1c0 0x7f51e00c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51e52d7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e0154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51e52d7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 0x7f51e0166670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51e52d7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e00c5020 con 0x7f51e006f590 2023-10-10T19:25:31.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51e52d7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e00c3d80 con 0x7f51e006f1c0 2023-10-10T19:25:31.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51e52d7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e0153f10 con 0x7f51e0155eb0 2023-10-10T19:25:31.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51d67fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e0154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51d67fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e0154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33378/0 (socket says 172.21.15.175:33378) 2023-10-10T19:25:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51d67fc700 1 -- 172.21.15.175:0/1498161094 learned_addr learned my addr 172.21.15.175:0/1498161094 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51deffd700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51e006f1c0 0x7f51e00c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.483+0000 7f51df7fe700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 0x7f51e0166670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d67fc700 1 -- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 msgr2=0x7f51e0166670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d67fc700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 0x7f51e0166670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d67fc700 1 -- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51e006f1c0 msgr2=0x7f51e00c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d67fc700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51e006f1c0 0x7f51e00c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d67fc700 1 -- 172.21.15.175:0/1498161094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c80094f0 con 0x7f51e006f590 2023-10-10T19:25:31.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d67fc700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e0154960 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f51c8006fd0 tx=0x7f51c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:31.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d77fe700 1 -- 172.21.15.175:0/1498161094 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8000d10 con 0x7f51e006f590 2023-10-10T19:25:31.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d77fe700 1 -- 172.21.15.175:0/1498161094 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c800bbb0 con 0x7f51e006f590 2023-10-10T19:25:31.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e00c4630 con 0x7f51e006f590 2023-10-10T19:25:31.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51d77fe700 1 -- 172.21.15.175:0/1498161094 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8011670 con 0x7f51e006f590 2023-10-10T19:25:31.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.484+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e00c4250 con 0x7f51e006f590 2023-10-10T19:25:31.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.485+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e006ebb0 con 0x7f51e006f590 2023-10-10T19:25:31.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.485+0000 7f51d77fe700 1 -- 172.21.15.175:0/1498161094 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f51c80119e0 con 0x7f51e006f590 2023-10-10T19:25:31.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.486+0000 7f51d77fe700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51c0066f30 0x7f51c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:31.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.486+0000 7f51d77fe700 1 -- 172.21.15.175:0/1498161094 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f51c8086430 con 0x7f51e006f590 2023-10-10T19:25:31.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.486+0000 7f51deffd700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51c0066f30 0x7f51c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:31.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.486+0000 7f51deffd700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51c0066f30 0x7f51c00693e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009d10 tx=0x7f51d00091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:31.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.490+0000 7f51d77fe700 1 -- 172.21.15.175:0/1498161094 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51c8052f20 con 0x7f51e006f590 2023-10-10T19:25:31.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-10-10T19:25:31.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.714+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51e0067570 con 0x7f51e006f590 2023-10-10T19:25:31.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.715+0000 7f51d77fe700 1 -- 172.21.15.175:0/1498161094 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f51c8056540 con 0x7f51e006f590 2023-10-10T19:25:31.716 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:31.717 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":296,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7771,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":296}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:31.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51c0066f30 msgr2=0x7f51c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51c0066f30 0x7f51c00693e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009d10 tx=0x7f51d00091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 msgr2=0x7f51e0154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:31.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e0154960 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f51c8006fd0 tx=0x7f51c8017000 comp rx=0 tx=0).stop 2023-10-10T19:25:31.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 shutdown_connections 2023-10-10T19:25:31.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51c0066f30 0x7f51c00693e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51e0155eb0 0x7f51e0166670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51e006f1c0 0x7f51e00c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 --2- 172.21.15.175:0/1498161094 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51e006f590 0x7f51e0154960 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:31.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 >> 172.21.15.175:0/1498161094 conn(0x7f51e00c95a0 msgr2=0x7f51e00c6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:31.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 shutdown_connections 2023-10-10T19:25:31.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:31.717+0000 7f51e52d7700 1 -- 172.21.15.175:0/1498161094 wait complete. 2023-10-10T19:25:31.721 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 296 2023-10-10T19:25:31.730 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:25:32.073 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:25:32.072+0000 7f7f74406700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-10T19:25:32.073 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:25:32.072+0000 7f7f74406700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:25:32.073 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:25:32.072+0000 7f7f74406700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-10T19:25:32.073 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:25:32.072+0000 7f7f74406700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-10T19:25:33.732 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:33.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.854+0000 7fc152f5c700 1 Processor -- start 2023-10-10T19:25:33.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.854+0000 7fc152f5c700 1 -- start start 2023-10-10T19:25:33.859 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc152f5c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc14c08a100 0x7fc14c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc152f5c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14c067fa0 con 0x7fc14c14fd90 2023-10-10T19:25:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc152f5c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14c156b70 con 0x7fc14c0cd040 2023-10-10T19:25:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc152f5c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14c0c5600 con 0x7fc14c08a100 2023-10-10T19:25:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc151f5a700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc14c14fd90 0x7fc14c0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37868/0 (socket says 172.21.15.175:37868) 2023-10-10T19:25:33.860 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc151f5a700 1 -- 172.21.15.175:0/796040896 learned_addr learned my addr 172.21.15.175:0/796040896 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc151759700 1 --2- 172.21.15.175:0/796040896 >> v2:172.21.15.191:3300/0 conn(0x7fc14c08a100 0x7fc14c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369228864 0 0) 0x7fc14c067fa0 con 0x7fc14c14fd90 2023-10-10T19:25:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.855+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc138003570 con 0x7fc14c14fd90 2023-10-10T19:25:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694255384 0 0) 0x7fc138003570 con 0x7fc14c14fd90 2023-10-10T19:25:33.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 >> v2:172.21.15.191:3300/0 conn(0x7fc14c08a100 msgr2=0x7fc14c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 --2- 172.21.15.175:0/796040896 >> v2:172.21.15.191:3300/0 conn(0x7fc14c08a100 0x7fc14c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 >> v1:172.21.15.191:6789/0 conn(0x7fc14c0cd040 legacy=0x7fc14c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c150160 con 0x7fc14c14fd90 2023-10-10T19:25:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fc1480017f0 con 0x7fc14c14fd90 2023-10-10T19:25:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 >> v1:172.21.15.175:6789/0 conn(0x7fc14c14fd90 legacy=0x7fc14c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138003a60 0x7fc138003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc138004810 0x7fc138004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1380051c0 0x7fc13800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c0c5600 con 0x7fc138004810 2023-10-10T19:25:33.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13800c0d0 con 0x7fc138003a60 2023-10-10T19:25:33.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13800c290 con 0x7fc1380051c0 2023-10-10T19:25:33.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc151759700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc138004810 0x7fc138004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:33.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc151f5a700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138003a60 0x7fc138003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:33.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc15275b700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1380051c0 0x7fc13800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:33.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc151759700 1 -- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1380051c0 msgr2=0x7fc13800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:33.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc151759700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1380051c0 0x7fc13800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:33.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc151759700 1 -- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138003a60 msgr2=0x7fc138003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:33.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc151759700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138003a60 0x7fc138003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:33.865 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc151759700 1 -- 172.21.15.175:0/796040896 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc138003570 con 0x7fc138004810 2023-10-10T19:25:33.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc15275b700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1380051c0 0x7fc13800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:33.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc15275b700 1 -- 172.21.15.175:0/796040896 reap_dead start 2023-10-10T19:25:33.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc151759700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc138004810 0x7fc138004c80 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009c00 tx=0x7fc13c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c71479b90649a75 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:33.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c00e710 con 0x7fc138004810 2023-10-10T19:25:33.866 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c00e870 con 0x7fc138004810 2023-10-10T19:25:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.856+0000 7fc150f58700 1 -- 172.21.15.175:0/796040896 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c00ec30 con 0x7fc138004810 2023-10-10T19:25:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.857+0000 7fc152f5c700 1 -- 172.21.15.175:0/796040896 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc138004810 msgr2=0x7fc138004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.857+0000 7fc152f5c700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc138004810 0x7fc138004c80 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009c00 tx=0x7fc13c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.857+0000 7fc152f5c700 1 -- 172.21.15.175:0/796040896 shutdown_connections 2023-10-10T19:25:33.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.857+0000 7fc152f5c700 1 --2- 172.21.15.175:0/796040896 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc138004810 0x7fc138004c80 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.857+0000 7fc152f5c700 1 -- 172.21.15.175:0/796040896 >> 172.21.15.175:0/796040896 conn(0x7fc14c063d00 msgr2=0x7fc14c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.857+0000 7fc152f5c700 1 -- 172.21.15.175:0/796040896 shutdown_connections 2023-10-10T19:25:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.857+0000 7fc152f5c700 1 -- 172.21.15.175:0/796040896 wait complete. 2023-10-10T19:25:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.858+0000 7fc152f5c700 1 Processor -- start 2023-10-10T19:25:33.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.858+0000 7fc152f5c700 1 -- start start 2023-10-10T19:25:33.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc152f5c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138004810 0x7fc14c168d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:33.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc152f5c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14c0c1c60 0x7fc14c166280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc152f5c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc14c0c6c80 0x7fc14c0c2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc152f5c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c150d80 con 0x7fc14c0c6c80 2023-10-10T19:25:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc152f5c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c165df0 con 0x7fc138004810 2023-10-10T19:25:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc152f5c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c1657e0 con 0x7fc14c0c1c60 2023-10-10T19:25:33.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc14c0c6c80 0x7fc14c0c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:33.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc14c0c6c80 0x7fc14c0c2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33384/0 (socket says 172.21.15.175:33384) 2023-10-10T19:25:33.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 -- 172.21.15.175:0/2413832671 learned_addr learned my addr 172.21.15.175:0/2413832671 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:33.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc151f5a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138004810 0x7fc14c168d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:33.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc151759700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14c0c1c60 0x7fc14c166280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:33.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 -- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14c0c1c60 msgr2=0x7fc14c166280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:33.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14c0c1c60 0x7fc14c166280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:33.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 -- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138004810 msgr2=0x7fc14c168d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:33.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138004810 0x7fc14c168d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:33.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 -- 172.21.15.175:0/2413832671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c0c7530 con 0x7fc14c0c6c80 2023-10-10T19:25:33.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc15275b700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc14c0c6c80 0x7fc14c0c2630 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0699f0 tx=0x7fc144009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:33.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.859+0000 7fc141ffb700 1 -- 172.21.15.175:0/2413832671 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14400ee40 con 0x7fc14c0c6c80 2023-10-10T19:25:33.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.860+0000 7fc141ffb700 1 -- 172.21.15.175:0/2413832671 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc144014480 con 0x7fc14c0c6c80 2023-10-10T19:25:33.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.860+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c150490 con 0x7fc14c0c6c80 2023-10-10T19:25:33.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.860+0000 7fc141ffb700 1 -- 172.21.15.175:0/2413832671 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144014840 con 0x7fc14c0c6c80 2023-10-10T19:25:33.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.860+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14c165a80 con 0x7fc14c0c6c80 2023-10-10T19:25:33.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.861+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14c067570 con 0x7fc14c0c6c80 2023-10-10T19:25:33.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.861+0000 7fc141ffb700 1 -- 172.21.15.175:0/2413832671 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc144014bb0 con 0x7fc14c0c6c80 2023-10-10T19:25:33.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.861+0000 7fc141ffb700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc130066f30 0x7fc1300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:33.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.861+0000 7fc141ffb700 1 -- 172.21.15.175:0/2413832671 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7fc1440841d0 con 0x7fc14c0c6c80 2023-10-10T19:25:33.876 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.861+0000 7fc151f5a700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc130066f30 0x7fc1300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:33.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.862+0000 7fc151f5a700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc130066f30 0x7fc1300693e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc1480013d0 tx=0x7fc148006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:33.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:33.865+0000 7fc141ffb700 1 -- 172.21.15.175:0/2413832671 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc14404d5a0 con 0x7fc14c0c6c80 2023-10-10T19:25:34.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.093+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc14c165f30 con 0x7fc14c0c6c80 2023-10-10T19:25:34.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.093+0000 7fc141ffb700 1 -- 172.21.15.175:0/2413832671 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) v1 ==== 95+0+6179 (secure 0 0 0) 0x7fc144019020 con 0x7fc14c0c6c80 2023-10-10T19:25:34.095 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:34.096 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":296,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7771,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":296}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:34.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc130066f30 msgr2=0x7fc1300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:34.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc130066f30 0x7fc1300693e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc1480013d0 tx=0x7fc148006040 comp rx=0 tx=0).stop 2023-10-10T19:25:34.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc14c0c6c80 msgr2=0x7fc14c0c2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:34.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc14c0c6c80 0x7fc14c0c2630 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0699f0 tx=0x7fc144009040 comp rx=0 tx=0).stop 2023-10-10T19:25:34.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 shutdown_connections 2023-10-10T19:25:34.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc130066f30 0x7fc1300693e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:34.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc14c0c6c80 0x7fc14c0c2630 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:34.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc138004810 0x7fc14c168d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:34.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 --2- 172.21.15.175:0/2413832671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc14c0c1c60 0x7fc14c166280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:34.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 >> 172.21.15.175:0/2413832671 conn(0x7fc14c063d00 msgr2=0x7fc14c14edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:34.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 shutdown_connections 2023-10-10T19:25:34.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:34.096+0000 7fc152f5c700 1 -- 172.21.15.175:0/2413832671 wait complete. 2023-10-10T19:25:34.100 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 296 2023-10-10T19:25:34.109 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:25:36.110 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:36.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.235+0000 7fad4a55c700 1 Processor -- start 2023-10-10T19:25:36.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.235+0000 7fad4a55c700 1 -- start start 2023-10-10T19:25:36.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad4a55c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fad440b1fc0 0x7fad44155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:36.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad4a55c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440aff60 0x7fad440b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad4a55c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad441574b0 con 0x7fad440b2390 2023-10-10T19:25:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad4a55c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad44150770 con 0x7fad440b1fc0 2023-10-10T19:25:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad4a55c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad440afad0 con 0x7fad440aff60 2023-10-10T19:25:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad43fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fad440b1fc0 0x7fad44155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:36.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad48af9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440aff60 0x7fad440b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad43fff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fad440b1fc0 0x7fad44155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:49352/0 (socket says 172.21.15.175:49352) 2023-10-10T19:25:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad48af9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440aff60 0x7fad440b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:48938/0 (socket says 172.21.15.175:48938) 2023-10-10T19:25:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad43fff700 1 -- 172.21.15.175:0/3647136900 learned_addr learned my addr 172.21.15.175:0/3647136900 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.236+0000 7fad3bfff700 1 --1- 172.21.15.175:0/3647136900 >> v1:172.21.15.191:6789/0 conn(0x7fad440b2390 0x7fad44068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:34026/0 (socket says 172.21.15.175:34026) 2023-10-10T19:25:36.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad43fff700 1 -- 172.21.15.175:0/3647136900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440aff60 msgr2=0x7fad440b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:36.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad43fff700 1 --2- 172.21.15.175:0/3647136900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440aff60 0x7fad440b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad43fff700 1 -- 172.21.15.175:0/3647136900 >> v1:172.21.15.191:6789/0 conn(0x7fad440b2390 legacy=0x7fad44068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:36.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad43fff700 1 -- 172.21.15.175:0/3647136900 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440af4c0 con 0x7fad440b1fc0 2023-10-10T19:25:36.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad48af9700 1 --2- 172.21.15.175:0/3647136900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440aff60 0x7fad440b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:25:36.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad43fff700 1 --2- 172.21.15.175:0/3647136900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 0x7fad44155fb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fad340099b0 tx=0x7fad34009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9f2dcc98732c144 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:36.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad437fe700 1 -- 172.21.15.175:0/3647136900 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34009ea0 con 0x7fad440b1fc0 2023-10-10T19:25:36.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad437fe700 1 -- 172.21.15.175:0/3647136900 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3400b8f0 con 0x7fad440b1fc0 2023-10-10T19:25:36.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad437fe700 1 -- 172.21.15.175:0/3647136900 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3400bc90 con 0x7fad440b1fc0 2023-10-10T19:25:36.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad4a55c700 1 -- 172.21.15.175:0/3647136900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 msgr2=0x7fad44155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:36.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.237+0000 7fad4a55c700 1 --2- 172.21.15.175:0/3647136900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 0x7fad44155fb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fad340099b0 tx=0x7fad34009250 comp rx=0 tx=0).stop 2023-10-10T19:25:36.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.238+0000 7fad4a55c700 1 -- 172.21.15.175:0/3647136900 shutdown_connections 2023-10-10T19:25:36.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.238+0000 7fad4a55c700 1 --2- 172.21.15.175:0/3647136900 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440aff60 0x7fad440b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.238+0000 7fad4a55c700 1 --2- 172.21.15.175:0/3647136900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 0x7fad44155fb0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.238+0000 7fad4a55c700 1 -- 172.21.15.175:0/3647136900 >> 172.21.15.175:0/3647136900 conn(0x7fad44063230 msgr2=0x7fad44063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:36.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.238+0000 7fad4a55c700 1 -- 172.21.15.175:0/3647136900 shutdown_connections 2023-10-10T19:25:36.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.238+0000 7fad4a55c700 1 -- 172.21.15.175:0/3647136900 wait complete. 2023-10-10T19:25:36.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.239+0000 7fad4a55c700 1 Processor -- start 2023-10-10T19:25:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.239+0000 7fad4a55c700 1 -- start start 2023-10-10T19:25:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.239+0000 7fad4a55c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad440aff60 0x7fad4415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.239+0000 7fad4a55c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 0x7fad44165db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.239+0000 7fad4a55c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440b2390 0x7fad4414d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.239+0000 7fad4a55c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad440cb5e0 con 0x7fad440aff60 2023-10-10T19:25:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.239+0000 7fad4a55c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad44169d00 con 0x7fad440b1fc0 2023-10-10T19:25:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.239+0000 7fad4a55c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4416a190 con 0x7fad440b2390 2023-10-10T19:25:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad440aff60 0x7fad4415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad440aff60 0x7fad4415b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55130/0 (socket says 172.21.15.175:55130) 2023-10-10T19:25:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 -- 172.21.15.175:0/1842200508 learned_addr learned my addr 172.21.15.175:0/1842200508 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad3bfff700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 0x7fad44165db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad48af9700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440b2390 0x7fad4414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 -- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440b2390 msgr2=0x7fad4414d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:36.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440b2390 0x7fad4414d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 -- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 msgr2=0x7fad44165db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:36.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 0x7fad44165db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 -- 172.21.15.175:0/1842200508 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c009a20 con 0x7fad440aff60 2023-10-10T19:25:36.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad43fff700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad440aff60 0x7fad4415b6d0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fad34006fd0 tx=0x7fad34017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:36.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad40ff9700 1 -- 172.21.15.175:0/1842200508 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3400bb30 con 0x7fad440aff60 2023-10-10T19:25:36.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad40ff9700 1 -- 172.21.15.175:0/1842200508 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad34026470 con 0x7fad440aff60 2023-10-10T19:25:36.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad40ff9700 1 -- 172.21.15.175:0/1842200508 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34026830 con 0x7fad440aff60 2023-10-10T19:25:36.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad340094f0 con 0x7fad440aff60 2023-10-10T19:25:36.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.240+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad441693d0 con 0x7fad440aff60 2023-10-10T19:25:36.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.241+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad44169fe0 con 0x7fad440aff60 2023-10-10T19:25:36.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.242+0000 7fad40ff9700 1 -- 172.21.15.175:0/1842200508 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fad34023070 con 0x7fad440aff60 2023-10-10T19:25:36.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.242+0000 7fad40ff9700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad24066f30 0x7fad240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:36.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.242+0000 7fad40ff9700 1 -- 172.21.15.175:0/1842200508 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7fad34086470 con 0x7fad440aff60 2023-10-10T19:25:36.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.242+0000 7fad3bfff700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad24066f30 0x7fad240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:36.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.243+0000 7fad3bfff700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad24066f30 0x7fad240693e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fad2c007fd0 tx=0x7fad2c007d70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:36.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.246+0000 7fad40ff9700 1 -- 172.21.15.175:0/1842200508 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fad34052fe0 con 0x7fad440aff60 2023-10-10T19:25:36.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.468+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad44067570 con 0x7fad440aff60 2023-10-10T19:25:36.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.469+0000 7fad40ff9700 1 -- 172.21.15.175:0/1842200508 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) v1 ==== 95+0+6179 (secure 0 0 0) 0x7fad34056600 con 0x7fad440aff60 2023-10-10T19:25:36.471 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:36.471 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":296,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7771,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":296}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:36.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad24066f30 msgr2=0x7fad240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad24066f30 0x7fad240693e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fad2c007fd0 tx=0x7fad2c007d70 comp rx=0 tx=0).stop 2023-10-10T19:25:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad440aff60 msgr2=0x7fad4415b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad440aff60 0x7fad4415b6d0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fad34006fd0 tx=0x7fad34017000 comp rx=0 tx=0).stop 2023-10-10T19:25:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 shutdown_connections 2023-10-10T19:25:36.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad24066f30 0x7fad240693e0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad440b2390 0x7fad4414d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad440aff60 0x7fad4415b6d0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 --2- 172.21.15.175:0/1842200508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad440b1fc0 0x7fad44165db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 >> 172.21.15.175:0/1842200508 conn(0x7fad44063230 msgr2=0x7fad4414fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 shutdown_connections 2023-10-10T19:25:36.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:36.471+0000 7fad4a55c700 1 -- 172.21.15.175:0/1842200508 wait complete. 2023-10-10T19:25:36.475 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 296 2023-10-10T19:25:36.484 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:25:38.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-10-10T19:25:38.485 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:38.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.611+0000 7f94fa7c6700 1 Processor -- start 2023-10-10T19:25:38.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.611+0000 7f94fa7c6700 1 -- start start 2023-10-10T19:25:38.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.612+0000 7f94fa7c6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94f41573d0 0x7f94f40c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:38.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.612+0000 7f94fa7c6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f41577a0 0x7f94f40c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:38.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.612+0000 7f94fa7c6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f4154aa0 con 0x7f94f40bf6e0 2023-10-10T19:25:38.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.612+0000 7f94fa7c6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f4063dc0 con 0x7f94f41573d0 2023-10-10T19:25:38.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.612+0000 7f94fa7c6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4157bf0 con 0x7f94f41577a0 2023-10-10T19:25:38.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.612+0000 7f94f3fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94f41573d0 0x7f94f40c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:38.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.612+0000 7f94f3fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94f41573d0 0x7f94f40c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55140/0 (socket says 172.21.15.175:55140) 2023-10-10T19:25:38.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.612+0000 7f94f3fff700 1 -- 172.21.15.175:0/3596644165 learned_addr learned my addr 172.21.15.175:0/3596644165 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:38.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f37fe700 1 --2- 172.21.15.175:0/3596644165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f41577a0 0x7f94f40c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:38.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f3fff700 1 -- 172.21.15.175:0/3596644165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f41577a0 msgr2=0x7f94f40c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:38.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f3fff700 1 --2- 172.21.15.175:0/3596644165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f41577a0 0x7f94f40c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f3fff700 1 -- 172.21.15.175:0/3596644165 >> v1:172.21.15.191:6789/0 conn(0x7f94f40bf6e0 legacy=0x7f94f40c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:38.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f3fff700 1 -- 172.21.15.175:0/3596644165 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f4157e50 con 0x7f94f41573d0 2023-10-10T19:25:38.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f3fff700 1 --2- 172.21.15.175:0/3596644165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41573d0 0x7f94f40c12a0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f94e40099b0 tx=0x7f94e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa8e40373831902 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:38.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f2ffd700 1 -- 172.21.15.175:0/3596644165 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4009ea0 con 0x7f94f41573d0 2023-10-10T19:25:38.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f2ffd700 1 -- 172.21.15.175:0/3596644165 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e400b860 con 0x7f94f41573d0 2023-10-10T19:25:38.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94f2ffd700 1 -- 172.21.15.175:0/3596644165 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e400bc00 con 0x7f94f41573d0 2023-10-10T19:25:38.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3596644165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41573d0 msgr2=0x7f94f40c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:38.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.613+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3596644165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41573d0 0x7f94f40c12a0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f94e40099b0 tx=0x7f94e4009250 comp rx=0 tx=0).stop 2023-10-10T19:25:38.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.614+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3596644165 shutdown_connections 2023-10-10T19:25:38.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.614+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3596644165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41573d0 0x7f94f40c12a0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.614+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3596644165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f41577a0 0x7f94f40c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.614+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3596644165 >> 172.21.15.175:0/3596644165 conn(0x7f94f406dc70 msgr2=0x7f94f4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:38.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.614+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3596644165 shutdown_connections 2023-10-10T19:25:38.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.614+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3596644165 wait complete. 2023-10-10T19:25:38.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.615+0000 7f94fa7c6700 1 Processor -- start 2023-10-10T19:25:38.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.615+0000 7f94fa7c6700 1 -- start start 2023-10-10T19:25:38.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94fa7c6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f40bf6e0 0x7f94f4158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:38.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94fa7c6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94f41573d0 0x7f94f41469d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94fa7c6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41577a0 0x7f94f4151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94fa7c6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4068900 con 0x7f94f41577a0 2023-10-10T19:25:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94fa7c6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4153410 con 0x7f94f41573d0 2023-10-10T19:25:38.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94fa7c6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4151a40 con 0x7f94f40bf6e0 2023-10-10T19:25:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41577a0 0x7f94f4151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41577a0 0x7f94f4151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55152/0 (socket says 172.21.15.175:55152) 2023-10-10T19:25:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 -- 172.21.15.175:0/3922323048 learned_addr learned my addr 172.21.15.175:0/3922323048 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f37fe700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94f41573d0 0x7f94f41469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:38.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f3fff700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f40bf6e0 0x7f94f4158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:38.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 -- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f40bf6e0 msgr2=0x7f94f4158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:38.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f40bf6e0 0x7f94f4158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 -- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94f41573d0 msgr2=0x7f94f41469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:38.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94f41573d0 0x7f94f41469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 -- 172.21.15.175:0/3922323048 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e40094f0 con 0x7f94f41577a0 2023-10-10T19:25:38.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94f8d63700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41577a0 0x7f94f4151ed0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00c910 tx=0x7f94ec001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:38.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94dbfff700 1 -- 172.21.15.175:0/3922323048 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec012630 con 0x7f94f41577a0 2023-10-10T19:25:38.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94dbfff700 1 -- 172.21.15.175:0/3922323048 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec0127b0 con 0x7f94f41577a0 2023-10-10T19:25:38.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f4068b30 con 0x7f94f41577a0 2023-10-10T19:25:38.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94dbfff700 1 -- 172.21.15.175:0/3922323048 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec012b70 con 0x7f94f41577a0 2023-10-10T19:25:38.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.616+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f4153130 con 0x7f94f41577a0 2023-10-10T19:25:38.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.617+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f406f030 con 0x7f94f41577a0 2023-10-10T19:25:38.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.618+0000 7f94dbfff700 1 -- 172.21.15.175:0/3922323048 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f94ec003030 con 0x7f94f41577a0 2023-10-10T19:25:38.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.618+0000 7f94dbfff700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f94e0066f30 0x7f94e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:38.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.618+0000 7f94dbfff700 1 -- 172.21.15.175:0/3922323048 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7f94ec084b90 con 0x7f94f41577a0 2023-10-10T19:25:38.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.618+0000 7f94f3fff700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f94e0066f30 0x7f94e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:38.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.619+0000 7f94f3fff700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f94e0066f30 0x7f94e00693e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f94e40099b0 tx=0x7f94e4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:38.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.623+0000 7f94dbfff700 1 -- 172.21.15.175:0/3922323048 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f94ec051680 con 0x7f94f41577a0 2023-10-10T19:25:38.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.849+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94f4151b80 con 0x7f94f41577a0 2023-10-10T19:25:38.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.849+0000 7f94dbfff700 1 -- 172.21.15.175:0/3922323048 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) v1 ==== 95+0+6179 (secure 0 0 0) 0x7f94ec054ca0 con 0x7f94f41577a0 2023-10-10T19:25:38.851 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:38.851 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":296,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7771,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":296}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:38.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.851+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f94e0066f30 msgr2=0x7f94e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:38.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.851+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f94e0066f30 0x7f94e00693e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f94e40099b0 tx=0x7f94e4017000 comp rx=0 tx=0).stop 2023-10-10T19:25:38.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.851+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41577a0 msgr2=0x7f94f4151ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:38.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.851+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41577a0 0x7f94f4151ed0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00c910 tx=0x7f94ec001710 comp rx=0 tx=0).stop 2023-10-10T19:25:38.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.852+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 shutdown_connections 2023-10-10T19:25:38.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.852+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f94e0066f30 0x7f94e00693e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.852+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f41577a0 0x7f94f4151ed0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.852+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94f40bf6e0 0x7f94f4158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.852+0000 7f94fa7c6700 1 --2- 172.21.15.175:0/3922323048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94f41573d0 0x7f94f41469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:38.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.852+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 >> 172.21.15.175:0/3922323048 conn(0x7f94f406dc70 msgr2=0x7f94f40c6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:38.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.852+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 shutdown_connections 2023-10-10T19:25:38.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:38.852+0000 7f94fa7c6700 1 -- 172.21.15.175:0/3922323048 wait complete. 2023-10-10T19:25:38.856 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 296 2023-10-10T19:25:38.865 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:25:40.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:40.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.993+0000 7fcce359e700 1 Processor -- start 2023-10-10T19:25:40.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.993+0000 7fcce359e700 1 -- start start 2023-10-10T19:25:40.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fcce359e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcce4157400 0x7fcce406bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:40.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fcce359e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcce40c4640 0x7fcce41577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:40.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fcce359e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce406d8e0 con 0x7fcce4157d10 2023-10-10T19:25:40.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fcce359e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce406e050 con 0x7fcce40c4640 2023-10-10T19:25:40.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fcce359e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce406dbc0 con 0x7fcce4157400 2023-10-10T19:25:40.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fccd9d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcce40c4640 0x7fcce41577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:40.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fccd9d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcce40c4640 0x7fcce41577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55164/0 (socket says 172.21.15.175:55164) 2023-10-10T19:25:40.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fccd9d9b700 1 -- 172.21.15.175:0/2200539009 learned_addr learned my addr 172.21.15.175:0/2200539009 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:40.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fcce259c700 1 --2- 172.21.15.175:0/2200539009 >> v2:172.21.15.191:3300/0 conn(0x7fcce4157400 0x7fcce406bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:40.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fccd9d9b700 1 -- 172.21.15.175:0/2200539009 >> v2:172.21.15.191:3300/0 conn(0x7fcce4157400 msgr2=0x7fcce406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:41.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fccd9d9b700 1 --2- 172.21.15.175:0/2200539009 >> v2:172.21.15.191:3300/0 conn(0x7fcce4157400 0x7fcce406bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fccd9d9b700 1 -- 172.21.15.175:0/2200539009 >> v1:172.21.15.191:6789/0 conn(0x7fcce4157d10 legacy=0x7fcce414d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:41.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fccd9d9b700 1 -- 172.21.15.175:0/2200539009 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce40c1030 con 0x7fcce40c4640 2023-10-10T19:25:41.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.994+0000 7fccd9d9b700 1 --2- 172.21.15.175:0/2200539009 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce40c4640 0x7fcce41577d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fcce4067120 tx=0x7fcccc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f0d6b4a37fd47c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:41.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce1d9b700 1 -- 172.21.15.175:0/2200539009 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc00b820 con 0x7fcce40c4640 2023-10-10T19:25:41.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce1d9b700 1 -- 172.21.15.175:0/2200539009 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc00b980 con 0x7fcce40c4640 2023-10-10T19:25:41.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce1d9b700 1 -- 172.21.15.175:0/2200539009 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc00bd20 con 0x7fcce40c4640 2023-10-10T19:25:41.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce359e700 1 -- 172.21.15.175:0/2200539009 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce40c4640 msgr2=0x7fcce41577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:41.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce359e700 1 --2- 172.21.15.175:0/2200539009 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce40c4640 0x7fcce41577d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fcce4067120 tx=0x7fcccc009250 comp rx=0 tx=0).stop 2023-10-10T19:25:41.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce359e700 1 -- 172.21.15.175:0/2200539009 shutdown_connections 2023-10-10T19:25:41.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce359e700 1 --2- 172.21.15.175:0/2200539009 >> v2:172.21.15.191:3300/0 conn(0x7fcce4157400 0x7fcce406bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce359e700 1 --2- 172.21.15.175:0/2200539009 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce40c4640 0x7fcce41577d0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce359e700 1 -- 172.21.15.175:0/2200539009 >> 172.21.15.175:0/2200539009 conn(0x7fcce4063cd0 msgr2=0x7fcce406d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:41.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce359e700 1 -- 172.21.15.175:0/2200539009 shutdown_connections 2023-10-10T19:25:41.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.995+0000 7fcce359e700 1 -- 172.21.15.175:0/2200539009 wait complete. 2023-10-10T19:25:41.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.996+0000 7fcce359e700 1 Processor -- start 2023-10-10T19:25:41.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce359e700 1 -- start start 2023-10-10T19:25:41.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce359e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcce40c4640 0x7fcce40c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:41.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce359e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce4157400 0x7fcce40c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:41.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce359e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcce4157d10 0x7fcce4164340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:41.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce359e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce40c7780 con 0x7fcce4157400 2023-10-10T19:25:41.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce359e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce40c9cb0 con 0x7fcce4157d10 2023-10-10T19:25:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce359e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce4163eb0 con 0x7fcce40c4640 2023-10-10T19:25:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fccd9d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce4157400 0x7fcce40c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fccd9d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce4157400 0x7fcce40c5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55168/0 (socket says 172.21.15.175:55168) 2023-10-10T19:25:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fccd9d9b700 1 -- 172.21.15.175:0/676393378 learned_addr learned my addr 172.21.15.175:0/676393378 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce259c700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcce40c4640 0x7fcce40c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.997+0000 7fcce2d9d700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcce4157d10 0x7fcce4164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccd9d9b700 1 -- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcce40c4640 msgr2=0x7fcce40c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccd9d9b700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcce40c4640 0x7fcce40c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccd9d9b700 1 -- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcce4157d10 msgr2=0x7fcce4164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:41.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccd9d9b700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcce4157d10 0x7fcce4164340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccd9d9b700 1 -- 172.21.15.175:0/676393378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0094f0 con 0x7fcce4157400 2023-10-10T19:25:41.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccd9d9b700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce4157400 0x7fcce40c5990 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fcce406e390 tx=0x7fcccc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:41.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccdaffd700 1 -- 172.21.15.175:0/676393378 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc01b470 con 0x7fcce4157400 2023-10-10T19:25:41.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccdaffd700 1 -- 172.21.15.175:0/676393378 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc01b5f0 con 0x7fcce4157400 2023-10-10T19:25:41.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fccdaffd700 1 -- 172.21.15.175:0/676393378 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc01b9b0 con 0x7fcce4157400 2023-10-10T19:25:41.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce4163ba0 con 0x7fcce4157400 2023-10-10T19:25:41.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.998+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce40c9f00 con 0x7fcce4157400 2023-10-10T19:25:41.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.999+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce4067570 con 0x7fcce4157400 2023-10-10T19:25:41.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:40.999+0000 7fccdaffd700 1 -- 172.21.15.175:0/676393378 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcccc01c450 con 0x7fcce4157400 2023-10-10T19:25:41.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.000+0000 7fccdaffd700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fccc4066f30 0x7fccc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:41.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.000+0000 7fccdaffd700 1 -- 172.21.15.175:0/676393378 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7fcccc086860 con 0x7fcce4157400 2023-10-10T19:25:41.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.000+0000 7fcce259c700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fccc4066f30 0x7fccc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:41.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.000+0000 7fcce259c700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fccc4066f30 0x7fccc40693e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fccd4006fd0 tx=0x7fccd4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:41.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.004+0000 7fccdaffd700 1 -- 172.21.15.175:0/676393378 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcccc053350 con 0x7fcce4157400 2023-10-10T19:25:41.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.230+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcce40c12f0 con 0x7fcce4157400 2023-10-10T19:25:41.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.230+0000 7fccdaffd700 1 -- 172.21.15.175:0/676393378 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) v1 ==== 95+0+6179 (secure 0 0 0) 0x7fcccc056970 con 0x7fcce4157400 2023-10-10T19:25:41.232 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:41.232 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":296,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7771,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":296}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:41.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fccc4066f30 msgr2=0x7fccc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:41.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fccc4066f30 0x7fccc40693e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fccd4006fd0 tx=0x7fccd4008040 comp rx=0 tx=0).stop 2023-10-10T19:25:41.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce4157400 msgr2=0x7fcce40c5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:41.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce4157400 0x7fcce40c5990 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fcce406e390 tx=0x7fcccc016000 comp rx=0 tx=0).stop 2023-10-10T19:25:41.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 shutdown_connections 2023-10-10T19:25:41.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fccc4066f30 0x7fccc40693e0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcce4157d10 0x7fcce4164340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcce40c4640 0x7fcce40c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 --2- 172.21.15.175:0/676393378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcce4157400 0x7fcce40c5990 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:41.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 >> 172.21.15.175:0/676393378 conn(0x7fcce4063cd0 msgr2=0x7fcce406bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:41.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 shutdown_connections 2023-10-10T19:25:41.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:41.232+0000 7fcce359e700 1 -- 172.21.15.175:0/676393378 wait complete. 2023-10-10T19:25:41.237 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 296 2023-10-10T19:25:41.245 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:25:43.247 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:43.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.372+0000 7fcd58870700 1 Processor -- start 2023-10-10T19:25:43.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.372+0000 7fcd58870700 1 -- start start 2023-10-10T19:25:43.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd58870700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcd5408a100 0x7fcd54159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:43.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd58870700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd540cd040 0x7fcd540684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:43.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd58870700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd54067fa0 con 0x7fcd5414fd90 2023-10-10T19:25:43.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd58870700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd540c5600 con 0x7fcd5408a100 2023-10-10T19:25:43.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd58870700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54156b70 con 0x7fcd540cd040 2023-10-10T19:25:43.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd537fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcd5414fd90 0x7fcd540c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41092/0 (socket says 172.21.15.175:41092) 2023-10-10T19:25:43.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd52ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcd5408a100 0x7fcd54159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:43.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd537fe700 1 -- 172.21.15.175:0/609177735 learned_addr learned my addr 172.21.15.175:0/609177735 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:43.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd53fff700 1 --2- 172.21.15.175:0/609177735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd540cd040 0x7fcd540684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:43.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd527fc700 1 -- 172.21.15.175:0/609177735 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977403570 0 0) 0x7fcd54067fa0 con 0x7fcd5414fd90 2023-10-10T19:25:43.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd527fc700 1 -- 172.21.15.175:0/609177735 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd3c003570 con 0x7fcd5414fd90 2023-10-10T19:25:43.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd52ffd700 1 -- 172.21.15.175:0/609177735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd540cd040 msgr2=0x7fcd540684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:43.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.373+0000 7fcd52ffd700 1 --2- 172.21.15.175:0/609177735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd540cd040 0x7fcd540684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd52ffd700 1 -- 172.21.15.175:0/609177735 >> v1:172.21.15.175:6789/0 conn(0x7fcd5414fd90 legacy=0x7fcd540c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:43.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd52ffd700 1 -- 172.21.15.175:0/609177735 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd54150160 con 0x7fcd5408a100 2023-10-10T19:25:43.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd52ffd700 1 --2- 172.21.15.175:0/609177735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd5408a100 0x7fcd54159c30 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fcd480098e0 tx=0x7fcd480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70c3194c6e8af16 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:43.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd527fc700 1 -- 172.21.15.175:0/609177735 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48011630 con 0x7fcd5408a100 2023-10-10T19:25:43.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd527fc700 1 -- 172.21.15.175:0/609177735 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd48011790 con 0x7fcd5408a100 2023-10-10T19:25:43.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd527fc700 1 -- 172.21.15.175:0/609177735 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48011b30 con 0x7fcd5408a100 2023-10-10T19:25:43.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd58870700 1 -- 172.21.15.175:0/609177735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd5408a100 msgr2=0x7fcd54159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:43.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd58870700 1 --2- 172.21.15.175:0/609177735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd5408a100 0x7fcd54159c30 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fcd480098e0 tx=0x7fcd480091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd58870700 1 -- 172.21.15.175:0/609177735 shutdown_connections 2023-10-10T19:25:43.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd58870700 1 --2- 172.21.15.175:0/609177735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd540cd040 0x7fcd540684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd58870700 1 --2- 172.21.15.175:0/609177735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd5408a100 0x7fcd54159c30 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd58870700 1 -- 172.21.15.175:0/609177735 >> 172.21.15.175:0/609177735 conn(0x7fcd54063d00 msgr2=0x7fcd540cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:43.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd58870700 1 -- 172.21.15.175:0/609177735 shutdown_connections 2023-10-10T19:25:43.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.374+0000 7fcd58870700 1 -- 172.21.15.175:0/609177735 wait complete. 2023-10-10T19:25:43.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd58870700 1 Processor -- start 2023-10-10T19:25:43.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd58870700 1 -- start start 2023-10-10T19:25:43.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd58870700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd5408a100 0x7fcd54158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:43.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd58870700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd540cd040 0x7fcd54158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:43.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd58870700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd5414fd90 0x7fcd54154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:43.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd58870700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54158490 con 0x7fcd540cd040 2023-10-10T19:25:43.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd58870700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd540cbd80 con 0x7fcd5408a100 2023-10-10T19:25:43.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd58870700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd540caf40 con 0x7fcd5414fd90 2023-10-10T19:25:43.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd52ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd540cd040 0x7fcd54158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:43.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd52ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd540cd040 0x7fcd54158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55182/0 (socket says 172.21.15.175:55182) 2023-10-10T19:25:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.376+0000 7fcd52ffd700 1 -- 172.21.15.175:0/2521247544 learned_addr learned my addr 172.21.15.175:0/2521247544 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd537fe700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd5408a100 0x7fcd54158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd53fff700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd5414fd90 0x7fcd54154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd52ffd700 1 -- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd5414fd90 msgr2=0x7fcd54154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:43.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd52ffd700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd5414fd90 0x7fcd54154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd52ffd700 1 -- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd5408a100 msgr2=0x7fcd54158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd52ffd700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd5408a100 0x7fcd54158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd52ffd700 1 -- 172.21.15.175:0/2521247544 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd48009420 con 0x7fcd540cd040 2023-10-10T19:25:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd537fe700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd5408a100 0x7fcd54158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:25:43.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd52ffd700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd540cd040 0x7fcd54158e60 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fcd54064c20 tx=0x7fcd48017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd3b7fe700 1 -- 172.21.15.175:0/2521247544 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48011b10 con 0x7fcd540cd040 2023-10-10T19:25:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd3b7fe700 1 -- 172.21.15.175:0/2521247544 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd480253f0 con 0x7fcd540cd040 2023-10-10T19:25:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd540cb200 con 0x7fcd540cd040 2023-10-10T19:25:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd3b7fe700 1 -- 172.21.15.175:0/2521247544 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd480257b0 con 0x7fcd540cd040 2023-10-10T19:25:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.377+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd540cbaa0 con 0x7fcd540cd040 2023-10-10T19:25:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.378+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd54159630 con 0x7fcd540cd040 2023-10-10T19:25:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.378+0000 7fcd3b7fe700 1 -- 172.21.15.175:0/2521247544 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcd48025ba0 con 0x7fcd540cd040 2023-10-10T19:25:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.379+0000 7fcd3b7fe700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd40066f30 0x7fcd400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.379+0000 7fcd3b7fe700 1 -- 172.21.15.175:0/2521247544 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 7627+0+0 (secure 0 0 0) 0x7fcd480865d0 con 0x7fcd540cd040 2023-10-10T19:25:43.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.379+0000 7fcd537fe700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd40066f30 0x7fcd400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:43.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.379+0000 7fcd537fe700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd40066f30 0x7fcd400693e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcd440058d0 tx=0x7fcd44009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:43.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.383+0000 7fcd3b7fe700 1 -- 172.21.15.175:0/2521247544 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd480530c0 con 0x7fcd540cd040 2023-10-10T19:25:43.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.611+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd54067570 con 0x7fcd540cd040 2023-10-10T19:25:43.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.612+0000 7fcd3b7fe700 1 -- 172.21.15.175:0/2521247544 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) v1 ==== 95+0+6179 (secure 0 0 0) 0x7fcd480566e0 con 0x7fcd540cd040 2023-10-10T19:25:43.614 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:43.614 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":296,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257},{"gid":7771,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":296}],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:24.002935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7384},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7384":{"gid":7384,"name":"d","rank":3,"incarnation":278,"state":"up:active","state_seq":23,"addr":"172.21.15.191:6837/2914004704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2914004704},{"type":"v1","addr":"172.21.15.191:6837","nonce":2914004704}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:43.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd40066f30 msgr2=0x7fcd400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:43.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd40066f30 0x7fcd400693e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcd440058d0 tx=0x7fcd44009040 comp rx=0 tx=0).stop 2023-10-10T19:25:43.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd540cd040 msgr2=0x7fcd54158e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:43.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd540cd040 0x7fcd54158e60 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fcd54064c20 tx=0x7fcd48017000 comp rx=0 tx=0).stop 2023-10-10T19:25:43.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 shutdown_connections 2023-10-10T19:25:43.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcd40066f30 0x7fcd400693e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd5414fd90 0x7fcd54154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd5408a100 0x7fcd54158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 --2- 172.21.15.175:0/2521247544 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd540cd040 0x7fcd54158e60 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:43.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 >> 172.21.15.175:0/2521247544 conn(0x7fcd54063d00 msgr2=0x7fcd540c6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:43.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 shutdown_connections 2023-10-10T19:25:43.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:43.614+0000 7fcd58870700 1 -- 172.21.15.175:0/2521247544 wait complete. 2023-10-10T19:25:43.618 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 296 2023-10-10T19:25:43.627 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:25:45.628 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:45.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.754+0000 7fcecbd1c700 1 Processor -- start 2023-10-10T19:25:45.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcecbd1c700 1 -- start start 2023-10-10T19:25:45.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcecbd1c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcec40b1fc0 0x7fcec4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:45.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcecbd1c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 0x7fcec4068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:45.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcecbd1c700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec41574b0 con 0x7fcec4068bd0 2023-10-10T19:25:45.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcecbd1c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec4150770 con 0x7fcec40b1fc0 2023-10-10T19:25:45.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcecbd1c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec40afad0 con 0x7fcec40b2390 2023-10-10T19:25:45.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcec9ab8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcec40b1fc0 0x7fcec4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:45.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcec9ab8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcec40b1fc0 0x7fcec4155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59414/0 (socket says 172.21.15.175:59414) 2023-10-10T19:25:45.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.755+0000 7fcec9ab8700 1 -- 172.21.15.175:0/471921536 learned_addr learned my addr 172.21.15.175:0/471921536 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:45.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec92b7700 1 --2- 172.21.15.175:0/471921536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 0x7fcec4068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:45.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec9ab8700 1 -- 172.21.15.175:0/471921536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 msgr2=0x7fcec4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:45.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec9ab8700 1 --2- 172.21.15.175:0/471921536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 0x7fcec4068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec9ab8700 1 -- 172.21.15.175:0/471921536 >> v1:172.21.15.191:6789/0 conn(0x7fcec4068bd0 legacy=0x7fcec40aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:45.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec9ab8700 1 -- 172.21.15.175:0/471921536 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec40af4c0 con 0x7fcec40b1fc0 2023-10-10T19:25:45.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec9ab8700 1 --2- 172.21.15.175:0/471921536 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec40b1fc0 0x7fcec4155fb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009f20 tx=0x7fcec0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04d50736686cbdc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:45.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec8ab6700 1 -- 172.21.15.175:0/471921536 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec000e070 con 0x7fcec40b1fc0 2023-10-10T19:25:45.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec8ab6700 1 -- 172.21.15.175:0/471921536 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec000bad0 con 0x7fcec40b1fc0 2023-10-10T19:25:45.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.756+0000 7fcec8ab6700 1 -- 172.21.15.175:0/471921536 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec00092e0 con 0x7fcec40b1fc0 2023-10-10T19:25:45.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.757+0000 7fcecbd1c700 1 -- 172.21.15.175:0/471921536 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec40b1fc0 msgr2=0x7fcec4155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:45.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.757+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/471921536 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec40b1fc0 0x7fcec4155fb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009f20 tx=0x7fcec0009840 comp rx=0 tx=0).stop 2023-10-10T19:25:45.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.757+0000 7fcecbd1c700 1 -- 172.21.15.175:0/471921536 shutdown_connections 2023-10-10T19:25:45.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.757+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/471921536 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec40b1fc0 0x7fcec4155fb0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.757+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/471921536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 0x7fcec4068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.757+0000 7fcecbd1c700 1 -- 172.21.15.175:0/471921536 >> 172.21.15.175:0/471921536 conn(0x7fcec4063230 msgr2=0x7fcec4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:45.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.757+0000 7fcecbd1c700 1 -- 172.21.15.175:0/471921536 shutdown_connections 2023-10-10T19:25:45.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.757+0000 7fcecbd1c700 1 -- 172.21.15.175:0/471921536 wait complete. 2023-10-10T19:25:45.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.758+0000 7fcecbd1c700 1 Processor -- start 2023-10-10T19:25:45.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.758+0000 7fcecbd1c700 1 -- start start 2023-10-10T19:25:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcecbd1c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec4068bd0 0x7fcec41569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcecbd1c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcec40b1fc0 0x7fcec41543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcecbd1c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 0x7fcec40c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:45.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcecbd1c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec4156f00 con 0x7fcec4068bd0 2023-10-10T19:25:45.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcecbd1c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec4166440 con 0x7fcec40b1fc0 2023-10-10T19:25:45.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcecbd1c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec4166720 con 0x7fcec40b2390 2023-10-10T19:25:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec4068bd0 0x7fcec41569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec4068bd0 0x7fcec41569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59420/0 (socket says 172.21.15.175:59420) 2023-10-10T19:25:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 -- 172.21.15.175:0/1035148033 learned_addr learned my addr 172.21.15.175:0/1035148033 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:45.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec92b7700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcec40b1fc0 0x7fcec41543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fceca2b9700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 0x7fcec40c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 -- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 msgr2=0x7fcec40c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 0x7fcec40c5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 -- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcec40b1fc0 msgr2=0x7fcec41543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:45.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcec40b1fc0 0x7fcec41543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 -- 172.21.15.175:0/1035148033 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec0009ae0 con 0x7fcec4068bd0 2023-10-10T19:25:45.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcec9ab8700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec4068bd0 0x7fcec41569c0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fcec406fd90 tx=0x7fcec0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fceb9ffb700 1 -- 172.21.15.175:0/1035148033 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec00129d0 con 0x7fcec4068bd0 2023-10-10T19:25:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fceb9ffb700 1 -- 172.21.15.175:0/1035148033 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec0012b30 con 0x7fcec4068bd0 2023-10-10T19:25:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec4166a00 con 0x7fcec4068bd0 2023-10-10T19:25:45.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.759+0000 7fceb9ffb700 1 -- 172.21.15.175:0/1035148033 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec001d420 con 0x7fcec4068bd0 2023-10-10T19:25:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.760+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec40c57a0 con 0x7fcec4068bd0 2023-10-10T19:25:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.761+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec4067570 con 0x7fcec4068bd0 2023-10-10T19:25:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.761+0000 7fceb9ffb700 1 -- 172.21.15.175:0/1035148033 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcec001d810 con 0x7fcec4068bd0 2023-10-10T19:25:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.761+0000 7fceb9ffb700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcea8066f60 0x7fcea8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:45.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.761+0000 7fceb9ffb700 1 -- 172.21.15.175:0/1035148033 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fcec00863c0 con 0x7fcec4068bd0 2023-10-10T19:25:45.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.762+0000 7fcec92b7700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcea8066f60 0x7fcea8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:45.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.762+0000 7fcec92b7700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcea8066f60 0x7fcea8069410 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcec4158bf0 tx=0x7fceb4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:45.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.766+0000 7fceb9ffb700 1 -- 172.21.15.175:0/1035148033 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcec0052e50 con 0x7fcec4068bd0 2023-10-10T19:25:45.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.991+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcec4069060 con 0x7fcec4068bd0 2023-10-10T19:25:45.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.991+0000 7fceb9ffb700 1 -- 172.21.15.175:0/1035148033 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+5417 (secure 0 0 0) 0x7fcec0056470 con 0x7fcec4068bd0 2023-10-10T19:25:45.993 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:45.993 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":297,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:43.873007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:45.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcea8066f60 msgr2=0x7fcea8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:45.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcea8066f60 0x7fcea8069410 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcec4158bf0 tx=0x7fceb4008040 comp rx=0 tx=0).stop 2023-10-10T19:25:45.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec4068bd0 msgr2=0x7fcec41569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:45.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec4068bd0 0x7fcec41569c0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fcec406fd90 tx=0x7fcec0017000 comp rx=0 tx=0).stop 2023-10-10T19:25:45.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 shutdown_connections 2023-10-10T19:25:45.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcea8066f60 0x7fcea8069410 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcec40b2390 0x7fcec40c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcec4068bd0 0x7fcec41569c0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 --2- 172.21.15.175:0/1035148033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcec40b1fc0 0x7fcec41543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:45.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 >> 172.21.15.175:0/1035148033 conn(0x7fcec4063230 msgr2=0x7fcec41592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:45.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 shutdown_connections 2023-10-10T19:25:45.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:45.994+0000 7fcecbd1c700 1 -- 172.21.15.175:0/1035148033 wait complete. 2023-10-10T19:25:45.998 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 297 2023-10-10T19:25:46.007 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-10T19:25:46.007 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:25:46.007 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:46.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.123+0000 7fbdd29fb700 1 Processor -- start 2023-10-10T19:25:46.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.123+0000 7fbdd29fb700 1 -- start start 2023-10-10T19:25:46.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd29fb700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbdcc14fd90 0x7fbdcc159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:46.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd29fb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc08a100 0x7fbdcc0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:46.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd29fb700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbdcc0cd040 0x7fbdcc0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:46.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd29fb700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdcc0c5600 con 0x7fbdcc0cd040 2023-10-10T19:25:46.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd29fb700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdcc067fa0 con 0x7fbdcc14fd90 2023-10-10T19:25:46.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd29fb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc156b70 con 0x7fbdcc08a100 2023-10-10T19:25:46.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbdcc0cd040 0x7fbdcc0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:46.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbdcc0cd040 0x7fbdcc0c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59436/0 (socket says 172.21.15.175:59436) 2023-10-10T19:25:46.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 -- 172.21.15.175:0/2897079825 learned_addr learned my addr 172.21.15.175:0/2897079825 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:46.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd19f9700 1 --2- 172.21.15.175:0/2897079825 >> v2:172.21.15.191:3300/0 conn(0x7fbdcc14fd90 0x7fbdcc159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:46.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd11f8700 1 --2- 172.21.15.175:0/2897079825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc08a100 0x7fbdcc0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:46.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 -- 172.21.15.175:0/2897079825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc08a100 msgr2=0x7fbdcc0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:46.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 --2- 172.21.15.175:0/2897079825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc08a100 0x7fbdcc0c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 -- 172.21.15.175:0/2897079825 >> v2:172.21.15.191:3300/0 conn(0x7fbdcc14fd90 msgr2=0x7fbdcc159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:46.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 --2- 172.21.15.175:0/2897079825 >> v2:172.21.15.191:3300/0 conn(0x7fbdcc14fd90 0x7fbdcc159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 -- 172.21.15.175:0/2897079825 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc150160 con 0x7fbdcc0cd040 2023-10-10T19:25:46.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd19f9700 1 --2- 172.21.15.175:0/2897079825 >> v2:172.21.15.191:3300/0 conn(0x7fbdcc14fd90 0x7fbdcc159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:25:46.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd21fa700 1 --2- 172.21.15.175:0/2897079825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc0cd040 0x7fbdcc0c63f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc065080 tx=0x7fbdc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672027a5c396d85d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:46.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.124+0000 7fbdd09f7700 1 -- 172.21.15.175:0/2897079825 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800b860 con 0x7fbdcc0cd040 2023-10-10T19:25:46.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd09f7700 1 -- 172.21.15.175:0/2897079825 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc800b9c0 con 0x7fbdcc0cd040 2023-10-10T19:25:46.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd09f7700 1 -- 172.21.15.175:0/2897079825 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800bd60 con 0x7fbdcc0cd040 2023-10-10T19:25:46.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 -- 172.21.15.175:0/2897079825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc0cd040 msgr2=0x7fbdcc0c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:46.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/2897079825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc0cd040 0x7fbdcc0c63f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc065080 tx=0x7fbdc8009250 comp rx=0 tx=0).stop 2023-10-10T19:25:46.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 -- 172.21.15.175:0/2897079825 shutdown_connections 2023-10-10T19:25:46.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/2897079825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc0cd040 0x7fbdcc0c63f0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/2897079825 >> v2:172.21.15.191:3300/0 conn(0x7fbdcc14fd90 0x7fbdcc159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/2897079825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc08a100 0x7fbdcc0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 -- 172.21.15.175:0/2897079825 >> 172.21.15.175:0/2897079825 conn(0x7fbdcc063d00 msgr2=0x7fbdcc0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:46.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 -- 172.21.15.175:0/2897079825 shutdown_connections 2023-10-10T19:25:46.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.125+0000 7fbdd29fb700 1 -- 172.21.15.175:0/2897079825 wait complete. 2023-10-10T19:25:46.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.126+0000 7fbdd29fb700 1 Processor -- start 2023-10-10T19:25:46.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.126+0000 7fbdd29fb700 1 -- start start 2023-10-10T19:25:46.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd29fb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc08a100 0x7fbdcc152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:46.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd29fb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc0cd040 0x7fbdcc153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:46.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd29fb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdcc14fd90 0x7fbdcc0cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:46.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd29fb700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc1536e0 con 0x7fbdcc08a100 2023-10-10T19:25:46.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd29fb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc0cab10 con 0x7fbdcc14fd90 2023-10-10T19:25:46.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd29fb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc150550 con 0x7fbdcc0cd040 2023-10-10T19:25:46.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc08a100 0x7fbdcc152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:46.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc08a100 0x7fbdcc152ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59448/0 (socket says 172.21.15.175:59448) 2023-10-10T19:25:46.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 -- 172.21.15.175:0/702312011 learned_addr learned my addr 172.21.15.175:0/702312011 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:46.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd21fa700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdcc14fd90 0x7fbdcc0cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:46.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd11f8700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc0cd040 0x7fbdcc153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:46.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 -- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc0cd040 msgr2=0x7fbdcc153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:46.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc0cd040 0x7fbdcc153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 -- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdcc14fd90 msgr2=0x7fbdcc0cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:46.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdcc14fd90 0x7fbdcc0cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 -- 172.21.15.175:0/702312011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80094f0 con 0x7fbdcc08a100 2023-10-10T19:25:46.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd21fa700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdcc14fd90 0x7fbdcc0cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:25:46.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.127+0000 7fbdd19f9700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc08a100 0x7fbdcc152ae0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400a430 tx=0x7fbdc4007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:46.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.128+0000 7fbdc1ffb700 1 -- 172.21.15.175:0/702312011 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc400d650 con 0x7fbdcc08a100 2023-10-10T19:25:46.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.128+0000 7fbdc1ffb700 1 -- 172.21.15.175:0/702312011 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc400d7b0 con 0x7fbdcc08a100 2023-10-10T19:25:46.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.128+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdcc169410 con 0x7fbdcc08a100 2023-10-10T19:25:46.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.128+0000 7fbdc1ffb700 1 -- 172.21.15.175:0/702312011 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc400db50 con 0x7fbdcc08a100 2023-10-10T19:25:46.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.128+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdcc0ca4c0 con 0x7fbdcc08a100 2023-10-10T19:25:46.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.129+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdcc0cadf0 con 0x7fbdcc08a100 2023-10-10T19:25:46.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.129+0000 7fbdc1ffb700 1 -- 172.21.15.175:0/702312011 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbdc4015440 con 0x7fbdcc08a100 2023-10-10T19:25:46.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.129+0000 7fbdc1ffb700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdb0066f30 0x7fbdb00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:46.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.129+0000 7fbdc1ffb700 1 -- 172.21.15.175:0/702312011 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fbdc40855d0 con 0x7fbdcc08a100 2023-10-10T19:25:46.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.130+0000 7fbdd11f8700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdb0066f30 0x7fbdb00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:46.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.130+0000 7fbdd11f8700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdb0066f30 0x7fbdb00693e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc1589d0 tx=0x7fbdbc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:46.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.134+0000 7fbdc1ffb700 1 -- 172.21.15.175:0/702312011 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbdc404e5a0 con 0x7fbdcc08a100 2023-10-10T19:25:46.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.358+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdcc067570 con 0x7fbdcc08a100 2023-10-10T19:25:46.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.359+0000 7fbdc1ffb700 1 -- 172.21.15.175:0/702312011 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+5417 (secure 0 0 0) 0x7fbdc4052110 con 0x7fbdcc08a100 2023-10-10T19:25:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:46.361 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":297,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:43.873007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:46.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdb0066f30 msgr2=0x7fbdb00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:46.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdb0066f30 0x7fbdb00693e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc1589d0 tx=0x7fbdbc008040 comp rx=0 tx=0).stop 2023-10-10T19:25:46.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc08a100 msgr2=0x7fbdcc152ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:46.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc08a100 0x7fbdcc152ae0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400a430 tx=0x7fbdc4007910 comp rx=0 tx=0).stop 2023-10-10T19:25:46.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 shutdown_connections 2023-10-10T19:25:46.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdb0066f30 0x7fbdb00693e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdcc14fd90 0x7fbdcc0cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdcc08a100 0x7fbdcc152ae0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 --2- 172.21.15.175:0/702312011 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdcc0cd040 0x7fbdcc153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:46.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 >> 172.21.15.175:0/702312011 conn(0x7fbdcc063d00 msgr2=0x7fbdcc15c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:46.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 shutdown_connections 2023-10-10T19:25:46.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:46.361+0000 7fbdd29fb700 1 -- 172.21.15.175:0/702312011 wait complete. 2023-10-10T19:25:46.365 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 297 2023-10-10T19:25:46.375 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 297, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 297, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 117 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:replay", "state_seq": 1 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 130, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:25:43.873007+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 7401, "mds_2": 8049, "mds_3": 7771 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:25:48.376 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:48.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.501+0000 7fe117309700 1 Processor -- start 2023-10-10T19:25:48.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.502+0000 7fe117309700 1 -- start start 2023-10-10T19:25:48.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.502+0000 7fe117309700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe1100b1fc0 0x7fe110155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:48.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.502+0000 7fe117309700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1101574b0 con 0x7fe1100b2390 2023-10-10T19:25:48.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.502+0000 7fe117309700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1100afad0 con 0x7fe1100aff60 2023-10-10T19:25:48.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.502+0000 7fe117309700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe110150770 con 0x7fe1100b1fc0 2023-10-10T19:25:48.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe1148a4700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe1100b2390 0x7fe110068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52750/0 (socket says 172.21.15.175:52750) 2023-10-10T19:25:48.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe1148a4700 1 -- 172.21.15.175:0/3626597467 learned_addr learned my addr 172.21.15.175:0/3626597467 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:48.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe1150a5700 1 --2- 172.21.15.175:0/3626597467 >> v2:172.21.15.191:3300/0 conn(0x7fe1100b1fc0 0x7fe110155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:48.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe107fff700 1 -- 172.21.15.175:0/3626597467 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444531110 0 0) 0x7fe1101574b0 con 0x7fe1100b2390 2023-10-10T19:25:48.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe107fff700 1 -- 172.21.15.175:0/3626597467 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0fc003570 con 0x7fe1100b2390 2023-10-10T19:25:48.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe107fff700 1 -- 172.21.15.175:0/3626597467 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3212058828 0 0) 0x7fe0fc003570 con 0x7fe1100b2390 2023-10-10T19:25:48.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe1150a5700 1 -- 172.21.15.175:0/3626597467 >> v1:172.21.15.191:6789/0 conn(0x7fe1100aff60 legacy=0x7fe110159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:48.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe1150a5700 1 -- 172.21.15.175:0/3626597467 >> v1:172.21.15.175:6789/0 conn(0x7fe1100b2390 legacy=0x7fe110068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:48.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe1150a5700 1 -- 172.21.15.175:0/3626597467 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1100af4c0 con 0x7fe1100b1fc0 2023-10-10T19:25:48.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.503+0000 7fe1150a5700 1 --2- 172.21.15.175:0/3626597467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100b1fc0 0x7fe110155fb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009e60 tx=0x7fe10c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3447c38cd5d066af server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:48.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe107fff700 1 -- 172.21.15.175:0/3626597467 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00e070 con 0x7fe1100b1fc0 2023-10-10T19:25:48.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe107fff700 1 -- 172.21.15.175:0/3626597467 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c00b8e0 con 0x7fe1100b1fc0 2023-10-10T19:25:48.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe107fff700 1 -- 172.21.15.175:0/3626597467 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe100001470 con 0x7fe1100b1fc0 2023-10-10T19:25:48.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe117309700 1 -- 172.21.15.175:0/3626597467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100b1fc0 msgr2=0x7fe110155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:48.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe117309700 1 --2- 172.21.15.175:0/3626597467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100b1fc0 0x7fe110155fb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009e60 tx=0x7fe10c009780 comp rx=0 tx=0).stop 2023-10-10T19:25:48.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe117309700 1 -- 172.21.15.175:0/3626597467 shutdown_connections 2023-10-10T19:25:48.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe117309700 1 --2- 172.21.15.175:0/3626597467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100b1fc0 0x7fe110155fb0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:48.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe117309700 1 -- 172.21.15.175:0/3626597467 >> 172.21.15.175:0/3626597467 conn(0x7fe110063230 msgr2=0x7fe110063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:48.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe117309700 1 -- 172.21.15.175:0/3626597467 shutdown_connections 2023-10-10T19:25:48.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.504+0000 7fe117309700 1 -- 172.21.15.175:0/3626597467 wait complete. 2023-10-10T19:25:48.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.505+0000 7fe117309700 1 Processor -- start 2023-10-10T19:25:48.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.505+0000 7fe117309700 1 -- start start 2023-10-10T19:25:48.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe117309700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100aff60 0x7fe110158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:48.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe117309700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1100b1fc0 0x7fe1101581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:48.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe117309700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1100b2390 0x7fe1100cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:48.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe117309700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe110166380 con 0x7fe1100b1fc0 2023-10-10T19:25:48.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe117309700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1100cd4f0 con 0x7fe1100aff60 2023-10-10T19:25:48.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe117309700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe110159400 con 0x7fe1100b2390 2023-10-10T19:25:48.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe1148a4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1100b1fc0 0x7fe1101581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:48.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe1148a4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1100b1fc0 0x7fe1101581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59454/0 (socket says 172.21.15.175:59454) 2023-10-10T19:25:48.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe1148a4700 1 -- 172.21.15.175:0/2700268311 learned_addr learned my addr 172.21.15.175:0/2700268311 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:48.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe1158a6700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1100b2390 0x7fe1100cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:48.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.506+0000 7fe1150a5700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100aff60 0x7fe110158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:48.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1148a4700 1 -- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1100b2390 msgr2=0x7fe1100cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:48.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1148a4700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1100b2390 0x7fe1100cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:48.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1148a4700 1 -- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100aff60 msgr2=0x7fe110158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:48.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1148a4700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100aff60 0x7fe110158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:48.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1148a4700 1 -- 172.21.15.175:0/2700268311 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c009a20 con 0x7fe1100b1fc0 2023-10-10T19:25:48.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1148a4700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1100b1fc0 0x7fe1101581b0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fe10000cbf0 tx=0x7fe100001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:48.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1057fa700 1 -- 172.21.15.175:0/2700268311 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe100007150 con 0x7fe1100b1fc0 2023-10-10T19:25:48.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1057fa700 1 -- 172.21.15.175:0/2700268311 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1000072d0 con 0x7fe1100b1fc0 2023-10-10T19:25:48.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe1057fa700 1 -- 172.21.15.175:0/2700268311 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe100007690 con 0x7fe1100b1fc0 2023-10-10T19:25:48.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1100cd250 con 0x7fe1100b1fc0 2023-10-10T19:25:48.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.507+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe110166010 con 0x7fe1100b1fc0 2023-10-10T19:25:48.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.508+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe110166530 con 0x7fe1100b1fc0 2023-10-10T19:25:48.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.508+0000 7fe1057fa700 1 -- 172.21.15.175:0/2700268311 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe100004a40 con 0x7fe1100b1fc0 2023-10-10T19:25:48.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.509+0000 7fe1057fa700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe0f4066f30 0x7fe0f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:48.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.509+0000 7fe1057fa700 1 -- 172.21.15.175:0/2700268311 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fe100083e60 con 0x7fe1100b1fc0 2023-10-10T19:25:48.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.509+0000 7fe1150a5700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe0f4066f30 0x7fe0f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:48.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.509+0000 7fe1150a5700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe0f4066f30 0x7fe0f40693e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0099f0 tx=0x7fe10c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:48.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.513+0000 7fe1057fa700 1 -- 172.21.15.175:0/2700268311 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe110166530 con 0x7fe1100b1fc0 2023-10-10T19:25:48.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.737+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe110067570 con 0x7fe1100b1fc0 2023-10-10T19:25:48.739 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.738+0000 7fe1057fa700 1 -- 172.21.15.175:0/2700268311 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+5417 (secure 0 0 0) 0x7fe100018090 con 0x7fe1100b1fc0 2023-10-10T19:25:48.739 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:48.739 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":297,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:43.873007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:48.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.739+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe0f4066f30 msgr2=0x7fe0f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:48.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.739+0000 7fe117309700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe0f4066f30 0x7fe0f40693e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0099f0 tx=0x7fe10c017000 comp rx=0 tx=0).stop 2023-10-10T19:25:48.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.739+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1100b1fc0 msgr2=0x7fe1101581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:48.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.739+0000 7fe117309700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1100b1fc0 0x7fe1101581b0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fe10000cbf0 tx=0x7fe100001a30 comp rx=0 tx=0).stop 2023-10-10T19:25:48.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.740+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 shutdown_connections 2023-10-10T19:25:48.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.740+0000 7fe117309700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe0f4066f30 0x7fe0f40693e0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:48.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.740+0000 7fe117309700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1100b2390 0x7fe1100cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:48.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.740+0000 7fe117309700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1100aff60 0x7fe110158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:48.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.740+0000 7fe117309700 1 --2- 172.21.15.175:0/2700268311 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1100b1fc0 0x7fe1101581b0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:48.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.740+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 >> 172.21.15.175:0/2700268311 conn(0x7fe110063230 msgr2=0x7fe11014fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:48.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.740+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 shutdown_connections 2023-10-10T19:25:48.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:48.740+0000 7fe117309700 1 -- 172.21.15.175:0/2700268311 wait complete. 2023-10-10T19:25:48.744 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 297 2023-10-10T19:25:48.753 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:50.754 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:50.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.879+0000 7f0ab48e7700 1 Processor -- start 2023-10-10T19:25:50.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.879+0000 7f0ab48e7700 1 -- start start 2023-10-10T19:25:50.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0ab48e7700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0ab014fd90 0x7f0ab0159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:50.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0ab48e7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab00c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:50.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0ab48e7700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0ab00cd040 0x7f0ab00c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:50.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0ab48e7700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab00c5600 con 0x7f0ab00cd040 2023-10-10T19:25:50.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0ab48e7700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab0067fa0 con 0x7f0ab014fd90 2023-10-10T19:25:50.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0ab48e7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab0156b70 con 0x7f0ab008a100 2023-10-10T19:25:50.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0ab00cd040 0x7f0ab00c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:50.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaed9d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0ab014fd90 0x7f0ab0159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:50.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0ab00cd040 0x7f0ab00c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59460/0 (socket says 172.21.15.175:59460) 2023-10-10T19:25:50.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 -- 172.21.15.175:0/3331293377 learned_addr learned my addr 172.21.15.175:0/3331293377 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:50.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aae59c700 1 --2- 172.21.15.175:0/3331293377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab00c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:50.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 -- 172.21.15.175:0/3331293377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 msgr2=0x7f0ab00c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:50.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 --2- 172.21.15.175:0/3331293377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab00c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:50.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 -- 172.21.15.175:0/3331293377 >> v2:172.21.15.191:3300/0 conn(0x7f0ab014fd90 msgr2=0x7f0ab0159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:50.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 --2- 172.21.15.175:0/3331293377 >> v2:172.21.15.191:3300/0 conn(0x7f0ab014fd90 0x7f0ab0159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:50.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 -- 172.21.15.175:0/3331293377 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0150160 con 0x7f0ab00cd040 2023-10-10T19:25:50.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.880+0000 7f0aaf59e700 1 --2- 172.21.15.175:0/3331293377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab00cd040 0x7f0ab00c63f0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0065080 tx=0x7f0aa8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26748bcbc13a3611 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:50.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0aadd9b700 1 -- 172.21.15.175:0/3331293377 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa800e070 con 0x7f0ab00cd040 2023-10-10T19:25:50.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0aadd9b700 1 -- 172.21.15.175:0/3331293377 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa800b970 con 0x7f0ab00cd040 2023-10-10T19:25:50.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0aadd9b700 1 -- 172.21.15.175:0/3331293377 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa80092e0 con 0x7f0ab00cd040 2023-10-10T19:25:50.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 -- 172.21.15.175:0/3331293377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab00cd040 msgr2=0x7f0ab00c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:50.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/3331293377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab00cd040 0x7f0ab00c63f0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0065080 tx=0x7f0aa8009840 comp rx=0 tx=0).stop 2023-10-10T19:25:50.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 -- 172.21.15.175:0/3331293377 shutdown_connections 2023-10-10T19:25:50.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/3331293377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab00cd040 0x7f0ab00c63f0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:50.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/3331293377 >> v2:172.21.15.191:3300/0 conn(0x7f0ab014fd90 0x7f0ab0159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:50.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/3331293377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab00c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:50.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 -- 172.21.15.175:0/3331293377 >> 172.21.15.175:0/3331293377 conn(0x7f0ab0063d00 msgr2=0x7f0ab00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:50.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 -- 172.21.15.175:0/3331293377 shutdown_connections 2023-10-10T19:25:50.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.881+0000 7f0ab48e7700 1 -- 172.21.15.175:0/3331293377 wait complete. 2023-10-10T19:25:50.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.882+0000 7f0ab48e7700 1 Processor -- start 2023-10-10T19:25:50.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0ab48e7700 1 -- start start 2023-10-10T19:25:50.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0ab48e7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab0169c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:50.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0ab48e7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ab00cd040 0x7f0ab0158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:50.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0ab48e7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab014fd90 0x7f0ab0150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:50.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0ab48e7700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab014f030 con 0x7f0ab014fd90 2023-10-10T19:25:50.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0ab48e7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab0153b50 con 0x7f0ab00cd040 2023-10-10T19:25:50.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0ab48e7700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab016a1b0 con 0x7f0ab008a100 2023-10-10T19:25:50.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0aaf59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab014fd90 0x7f0ab0150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:50.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0aae59c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ab00cd040 0x7f0ab0158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:50.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0aaf59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab014fd90 0x7f0ab0150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59476/0 (socket says 172.21.15.175:59476) 2023-10-10T19:25:50.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0aaed9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab0169c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:50.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0aaf59e700 1 -- 172.21.15.175:0/2930413532 learned_addr learned my addr 172.21.15.175:0/2930413532 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:50.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.883+0000 7f0aae59c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ab00cd040 0x7f0ab0158230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:55972/0 (socket says 172.21.15.175:55972) 2023-10-10T19:25:50.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0aaf59e700 1 -- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 msgr2=0x7f0ab0169c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:50.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0aaf59e700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab0169c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:50.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0aaf59e700 1 -- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ab00cd040 msgr2=0x7f0ab0158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:50.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0aaf59e700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ab00cd040 0x7f0ab0158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:50.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0aaf59e700 1 -- 172.21.15.175:0/2930413532 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa8009ae0 con 0x7f0ab014fd90 2023-10-10T19:25:50.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0aaed9d700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab0169c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:25:50.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0aae59c700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ab00cd040 0x7f0ab0158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:25:50.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0aaf59e700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab014fd90 0x7f0ab0150550 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8009880 tx=0x7f0aa8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:50.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0a96ffd700 1 -- 172.21.15.175:0/2930413532 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8008b10 con 0x7f0ab014fd90 2023-10-10T19:25:50.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0a96ffd700 1 -- 172.21.15.175:0/2930413532 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8008c70 con 0x7f0ab014fd90 2023-10-10T19:25:50.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab0166b40 con 0x7f0ab014fd90 2023-10-10T19:25:50.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0a96ffd700 1 -- 172.21.15.175:0/2930413532 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8026460 con 0x7f0ab014fd90 2023-10-10T19:25:50.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.884+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab014ed50 con 0x7f0ab014fd90 2023-10-10T19:25:50.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.885+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab0067570 con 0x7f0ab014fd90 2023-10-10T19:25:50.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.885+0000 7f0a96ffd700 1 -- 172.21.15.175:0/2930413532 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0aa8003040 con 0x7f0ab014fd90 2023-10-10T19:25:50.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.886+0000 7f0a96ffd700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a9c066f60 0x7f0a9c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:50.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.886+0000 7f0a96ffd700 1 -- 172.21.15.175:0/2930413532 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f0aa80864a0 con 0x7f0ab014fd90 2023-10-10T19:25:50.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.886+0000 7f0aaed9d700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a9c066f60 0x7f0a9c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:50.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.886+0000 7f0aaed9d700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a9c066f60 0x7f0a9c069410 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00113e0 tx=0x7f0aa00091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:50.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:50.890+0000 7f0a96ffd700 1 -- 172.21.15.175:0/2930413532 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0aa8052fe0 con 0x7f0ab014fd90 2023-10-10T19:25:51.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.115+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ab0155f80 con 0x7f0ab014fd90 2023-10-10T19:25:51.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.116+0000 7f0a96ffd700 1 -- 172.21.15.175:0/2930413532 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+5417 (secure 0 0 0) 0x7f0aa8056600 con 0x7f0ab014fd90 2023-10-10T19:25:51.118 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:51.118 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":297,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:43.873007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:51.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a9c066f60 msgr2=0x7f0a9c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:51.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a9c066f60 0x7f0a9c069410 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00113e0 tx=0x7f0aa00091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:51.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab014fd90 msgr2=0x7f0ab0150550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:51.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab014fd90 0x7f0ab0150550 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8009880 tx=0x7f0aa8018000 comp rx=0 tx=0).stop 2023-10-10T19:25:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 shutdown_connections 2023-10-10T19:25:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0a9c066f60 0x7f0a9c069410 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ab014fd90 0x7f0ab0150550 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:51.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ab008a100 0x7f0ab0169c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 --2- 172.21.15.175:0/2930413532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ab00cd040 0x7f0ab0158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 >> 172.21.15.175:0/2930413532 conn(0x7f0ab0063d00 msgr2=0x7f0ab0154be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 shutdown_connections 2023-10-10T19:25:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:51.118+0000 7f0ab48e7700 1 -- 172.21.15.175:0/2930413532 wait complete. 2023-10-10T19:25:51.124 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 297 2023-10-10T19:25:51.131 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:53.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:53.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.257+0000 7fd552269700 1 Processor -- start 2023-10-10T19:25:53.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.257+0000 7fd552269700 1 -- start start 2023-10-10T19:25:53.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.257+0000 7fd552269700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd54c155fb0 0x7fd54c159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:53.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd552269700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c1574b0 con 0x7fd54c0b1fc0 2023-10-10T19:25:53.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd552269700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0afad0 con 0x7fd54c0b2390 2023-10-10T19:25:53.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd552269700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c150770 con 0x7fd54c155fb0 2023-10-10T19:25:53.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd54c0b1fc0 0x7fd54c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52752/0 (socket says 172.21.15.175:52752) 2023-10-10T19:25:53.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54bfff700 1 -- 172.21.15.175:0/131254994 learned_addr learned my addr 172.21.15.175:0/131254994 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:53.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd550806700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd54c155fb0 0x7fd54c159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:53.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319958038 0 0) 0x7fd54c1574b0 con 0x7fd54c0b1fc0 2023-10-10T19:25:53.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534003570 con 0x7fd54c0b1fc0 2023-10-10T19:25:53.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 649916865 0 0) 0x7fd534003570 con 0x7fd54c0b1fc0 2023-10-10T19:25:53.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd54c155fb0 msgr2=0x7fd54c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:53.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd54c155fb0 0x7fd54c159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 >> v1:172.21.15.191:6789/0 conn(0x7fd54c0b2390 legacy=0x7fd54c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:53.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0af4c0 con 0x7fd54c0b1fc0 2023-10-10T19:25:53.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fd53c001470 con 0x7fd54c0b1fc0 2023-10-10T19:25:53.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 >> v1:172.21.15.175:6789/0 conn(0x7fd54c0b1fc0 legacy=0x7fd54c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:53.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd534003a90 0x7fd534003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:53.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd534004810 0x7fd534004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:53.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd54c155fb0 0x7fd5340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:53.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c150770 con 0x7fd534003a90 2023-10-10T19:25:53.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53400bb10 con 0x7fd534004810 2023-10-10T19:25:53.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53400bcd0 con 0x7fd54c155fb0 2023-10-10T19:25:53.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.258+0000 7fd54bfff700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd534003a90 0x7fd534003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:53.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54b7fe700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd534004810 0x7fd534004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:53.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd550806700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd54c155fb0 0x7fd5340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:53.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54bfff700 1 -- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd54c155fb0 msgr2=0x7fd5340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:53.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54bfff700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd54c155fb0 0x7fd5340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54bfff700 1 -- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd534004810 msgr2=0x7fd534004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:53.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54bfff700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd534004810 0x7fd534004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54bfff700 1 -- 172.21.15.175:0/131254994 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534003570 con 0x7fd534003a90 2023-10-10T19:25:53.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54bfff700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd534003a90 0x7fd534003e60 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00c2d0 tx=0x7fd54c0712f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b538666cbeba2996 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:53.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00cc00 con 0x7fd534003a90 2023-10-10T19:25:53.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c00cd80 con 0x7fd534003a90 2023-10-10T19:25:53.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.259+0000 7fd54affd700 1 -- 172.21.15.175:0/131254994 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c005b70 con 0x7fd534003a90 2023-10-10T19:25:53.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.260+0000 7fd552269700 1 -- 172.21.15.175:0/131254994 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd534003a90 msgr2=0x7fd534003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:53.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.260+0000 7fd552269700 1 --2- 172.21.15.175:0/131254994 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd534003a90 0x7fd534003e60 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00c2d0 tx=0x7fd54c0712f0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.260+0000 7fd550806700 1 -- 172.21.15.175:0/131254994 reap_dead start 2023-10-10T19:25:53.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.260+0000 7fd552269700 1 -- 172.21.15.175:0/131254994 shutdown_connections 2023-10-10T19:25:53.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.260+0000 7fd552269700 1 -- 172.21.15.175:0/131254994 >> 172.21.15.175:0/131254994 conn(0x7fd54c063230 msgr2=0x7fd54c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:53.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.260+0000 7fd552269700 1 -- 172.21.15.175:0/131254994 shutdown_connections 2023-10-10T19:25:53.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.260+0000 7fd552269700 1 -- 172.21.15.175:0/131254994 wait complete. 2023-10-10T19:25:53.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.261+0000 7fd552269700 1 Processor -- start 2023-10-10T19:25:53.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.261+0000 7fd552269700 1 -- start start 2023-10-10T19:25:53.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd552269700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd534003a90 0x7fd54c169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:53.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd552269700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd54c0c92d0 0x7fd54c14f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:53.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd552269700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd54c169850 0x7fd54c14f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:53.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd552269700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0cbaf0 con 0x7fd54c169850 2023-10-10T19:25:53.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd552269700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c169fe0 con 0x7fd54c0c92d0 2023-10-10T19:25:53.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd552269700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c1578d0 con 0x7fd534003a90 2023-10-10T19:25:53.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd54c169850 0x7fd54c14f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:53.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd54c169850 0x7fd54c14f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59488/0 (socket says 172.21.15.175:59488) 2023-10-10T19:25:53.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 -- 172.21.15.175:0/1020106126 learned_addr learned my addr 172.21.15.175:0/1020106126 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:53.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd54bfff700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd534003a90 0x7fd54c169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:53.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd54b7fe700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd54c0c92d0 0x7fd54c14f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:53.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 -- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd534003a90 msgr2=0x7fd54c169310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:53.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd534003a90 0x7fd54c169310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 -- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd54c0c92d0 msgr2=0x7fd54c14f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:53.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd54c0c92d0 0x7fd54c14f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 -- 172.21.15.175:0/1020106126 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0cb7c0 con 0x7fd54c169850 2023-10-10T19:25:53.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd550806700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd54c169850 0x7fd54c14f6e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd54c08a030 tx=0x7fd54400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:53.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd533fff700 1 -- 172.21.15.175:0/1020106126 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400e040 con 0x7fd54c169850 2023-10-10T19:25:53.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd533fff700 1 -- 172.21.15.175:0/1020106126 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54400f040 con 0x7fd54c169850 2023-10-10T19:25:53.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c169d00 con 0x7fd54c169850 2023-10-10T19:25:53.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd533fff700 1 -- 172.21.15.175:0/1020106126 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400fe40 con 0x7fd54c169850 2023-10-10T19:25:53.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.262+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c152ef0 con 0x7fd54c169850 2023-10-10T19:25:53.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.263+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c0cbca0 con 0x7fd54c169850 2023-10-10T19:25:53.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.264+0000 7fd533fff700 1 -- 172.21.15.175:0/1020106126 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd544015640 con 0x7fd54c169850 2023-10-10T19:25:53.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.264+0000 7fd533fff700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd538066f30 0x7fd5380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:53.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.264+0000 7fd533fff700 1 -- 172.21.15.175:0/1020106126 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fd544085620 con 0x7fd54c169850 2023-10-10T19:25:53.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.264+0000 7fd54bfff700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd538066f30 0x7fd5380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:53.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.265+0000 7fd54bfff700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd538066f30 0x7fd5380693e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd53c001ed0 tx=0x7fd53c015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:53.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.268+0000 7fd533fff700 1 -- 172.21.15.175:0/1020106126 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd5440520b0 con 0x7fd54c169850 2023-10-10T19:25:53.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.495+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd54c067570 con 0x7fd54c169850 2023-10-10T19:25:53.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.495+0000 7fd533fff700 1 -- 172.21.15.175:0/1020106126 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+5418 (secure 0 0 0) 0x7fd5440556d0 con 0x7fd54c169850 2023-10-10T19:25:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:53.496 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":298,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:52.095664+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:resolve","state_seq":8,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:53.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd538066f30 msgr2=0x7fd5380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:53.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd538066f30 0x7fd5380693e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd53c001ed0 tx=0x7fd53c015000 comp rx=0 tx=0).stop 2023-10-10T19:25:53.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd54c169850 msgr2=0x7fd54c14f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:53.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd54c169850 0x7fd54c14f6e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd54c08a030 tx=0x7fd54400b040 comp rx=0 tx=0).stop 2023-10-10T19:25:53.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 shutdown_connections 2023-10-10T19:25:53.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd538066f30 0x7fd5380693e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd54c169850 0x7fd54c14f6e0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd534003a90 0x7fd54c169310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 --2- 172.21.15.175:0/1020106126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd54c0c92d0 0x7fd54c14f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:53.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 >> 172.21.15.175:0/1020106126 conn(0x7fd54c063230 msgr2=0x7fd54c151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:53.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 shutdown_connections 2023-10-10T19:25:53.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:53.498+0000 7fd552269700 1 -- 172.21.15.175:0/1020106126 wait complete. 2023-10-10T19:25:53.501 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 298 2023-10-10T19:25:53.510 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:55.511 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:55.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.635+0000 7fa652795700 1 Processor -- start 2023-10-10T19:25:55.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa652795700 1 -- start start 2023-10-10T19:25:55.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa652795700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa64c0b2390 0x7fa64c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:55.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa652795700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c1574b0 con 0x7fa64c0b1fc0 2023-10-10T19:25:55.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa652795700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0afad0 con 0x7fa64c0aff60 2023-10-10T19:25:55.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa652795700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c150770 con 0x7fa64c0b2390 2023-10-10T19:25:55.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa64b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa64c0b2390 0x7fa64c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:55.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa64bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa64c0b1fc0 0x7fa64c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50670/0 (socket says 172.21.15.175:50670) 2023-10-10T19:25:55.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa64bfff700 1 -- 172.21.15.175:0/3506151378 learned_addr learned my addr 172.21.15.175:0/3506151378 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:55.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.636+0000 7fa64b7fe700 1 --2- 172.21.15.175:0/3506151378 >> v2:172.21.15.175:3300/0 conn(0x7fa64c0b2390 0x7fa64c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44502/0 (socket says 172.21.15.175:44502) 2023-10-10T19:25:55.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa64b7fe700 1 -- 172.21.15.175:0/3506151378 >> v1:172.21.15.191:6789/0 conn(0x7fa64c0aff60 legacy=0x7fa64c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:55.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa64b7fe700 1 -- 172.21.15.175:0/3506151378 >> v1:172.21.15.175:6789/0 conn(0x7fa64c0b1fc0 legacy=0x7fa64c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:55.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa64b7fe700 1 -- 172.21.15.175:0/3506151378 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0af4c0 con 0x7fa64c0b2390 2023-10-10T19:25:55.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa64affd700 1 -- 172.21.15.175:0/3506151378 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785072690 0 0) 0x7fa64c1574b0 con 0x7fa64c0b1fc0 2023-10-10T19:25:55.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa64b7fe700 1 --2- 172.21.15.175:0/3506151378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c155fb0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fa6400098e0 tx=0x7fa6400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe097039c363599 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:55.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa64affd700 1 -- 172.21.15.175:0/3506151378 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640009e80 con 0x7fa64c0b2390 2023-10-10T19:25:55.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa64affd700 1 -- 172.21.15.175:0/3506151378 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64000b8f0 con 0x7fa64c0b2390 2023-10-10T19:25:55.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa64affd700 1 -- 172.21.15.175:0/3506151378 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64000bc70 con 0x7fa64c0b2390 2023-10-10T19:25:55.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa652795700 1 -- 172.21.15.175:0/3506151378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 msgr2=0x7fa64c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:55.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.637+0000 7fa652795700 1 --2- 172.21.15.175:0/3506151378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c155fb0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fa6400098e0 tx=0x7fa6400091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:55.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.638+0000 7fa652795700 1 -- 172.21.15.175:0/3506151378 shutdown_connections 2023-10-10T19:25:55.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.638+0000 7fa652795700 1 --2- 172.21.15.175:0/3506151378 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c155fb0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:55.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.638+0000 7fa652795700 1 -- 172.21.15.175:0/3506151378 >> 172.21.15.175:0/3506151378 conn(0x7fa64c063230 msgr2=0x7fa64c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:55.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.638+0000 7fa652795700 1 -- 172.21.15.175:0/3506151378 shutdown_connections 2023-10-10T19:25:55.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.638+0000 7fa652795700 1 -- 172.21.15.175:0/3506151378 wait complete. 2023-10-10T19:25:55.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.639+0000 7fa652795700 1 Processor -- start 2023-10-10T19:25:55.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.639+0000 7fa652795700 1 -- start start 2023-10-10T19:25:55.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.639+0000 7fa652795700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa64c0aff60 0x7fa64c158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:55.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa652795700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa64c0b1fc0 0x7fa64c158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:55.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa652795700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c1576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:55.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa652795700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0cc010 con 0x7fa64c0b2390 2023-10-10T19:25:55.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa652795700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0ccd10 con 0x7fa64c0aff60 2023-10-10T19:25:55.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa652795700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c166ad0 con 0x7fa64c0b1fc0 2023-10-10T19:25:55.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c1576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:55.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c1576c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44504/0 (socket says 172.21.15.175:44504) 2023-10-10T19:25:55.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 -- 172.21.15.175:0/749498605 learned_addr learned my addr 172.21.15.175:0/749498605 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:55.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa64bfff700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa64c0aff60 0x7fa64c158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:55.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa64b7fe700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa64c0b1fc0 0x7fa64c158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:55.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 -- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa64c0b1fc0 msgr2=0x7fa64c158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:55.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa64c0b1fc0 0x7fa64c158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:55.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 -- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa64c0aff60 msgr2=0x7fa64c158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:55.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa64c0aff60 0x7fa64c158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:55.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 -- 172.21.15.175:0/749498605 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa640009420 con 0x7fa64c0b2390 2023-10-10T19:25:55.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa650d32700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c1576c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fa64400c910 tx=0x7fa644001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:55.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa633fff700 1 -- 172.21.15.175:0/749498605 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400a7a0 con 0x7fa64c0b2390 2023-10-10T19:25:55.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa633fff700 1 -- 172.21.15.175:0/749498605 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64400a900 con 0x7fa64c0b2390 2023-10-10T19:25:55.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c0cc650 con 0x7fa64c0b2390 2023-10-10T19:25:55.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa633fff700 1 -- 172.21.15.175:0/749498605 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400acc0 con 0x7fa64c0b2390 2023-10-10T19:25:55.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.640+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c158340 con 0x7fa64c0b2390 2023-10-10T19:25:55.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.641+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c067570 con 0x7fa64c0b2390 2023-10-10T19:25:55.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.642+0000 7fa633fff700 1 -- 172.21.15.175:0/749498605 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa644012630 con 0x7fa64c0b2390 2023-10-10T19:25:55.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.642+0000 7fa633fff700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa638066f30 0x7fa6380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:55.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.642+0000 7fa633fff700 1 -- 172.21.15.175:0/749498605 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fa644085050 con 0x7fa64c0b2390 2023-10-10T19:25:55.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.642+0000 7fa64bfff700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa638066f30 0x7fa6380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:55.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.643+0000 7fa64bfff700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa638066f30 0x7fa6380693e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0011e0 tx=0x7fa63c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:55.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.647+0000 7fa633fff700 1 -- 172.21.15.175:0/749498605 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa644051ae0 con 0x7fa64c0b2390 2023-10-10T19:25:55.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.868+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa64c068da0 con 0x7fa64c0b2390 2023-10-10T19:25:55.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.869+0000 7fa633fff700 1 -- 172.21.15.175:0/749498605 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+5418 (secure 0 0 0) 0x7fa644055100 con 0x7fa64c0b2390 2023-10-10T19:25:55.870 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:55.870 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":298,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:52.095664+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:resolve","state_seq":8,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:55.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa638066f30 msgr2=0x7fa6380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:55.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa638066f30 0x7fa6380693e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0011e0 tx=0x7fa63c009040 comp rx=0 tx=0).stop 2023-10-10T19:25:55.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 msgr2=0x7fa64c1576c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:55.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c1576c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fa64400c910 tx=0x7fa644001710 comp rx=0 tx=0).stop 2023-10-10T19:25:55.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 shutdown_connections 2023-10-10T19:25:55.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa638066f30 0x7fa6380693e0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:55.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa64c0b2390 0x7fa64c1576c0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:55.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa64c0aff60 0x7fa64c158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:55.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 --2- 172.21.15.175:0/749498605 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa64c0b1fc0 0x7fa64c158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:55.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 >> 172.21.15.175:0/749498605 conn(0x7fa64c063230 msgr2=0x7fa64c154140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:55.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 shutdown_connections 2023-10-10T19:25:55.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:55.871+0000 7fa652795700 1 -- 172.21.15.175:0/749498605 wait complete. 2023-10-10T19:25:55.875 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 298 2023-10-10T19:25:55.883 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:25:56.485 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:25:56.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:25:57.885 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:25:58.012 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.012+0000 7f3212669700 1 Processor -- start 2023-10-10T19:25:58.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.012+0000 7f3212669700 1 -- start start 2023-10-10T19:25:58.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.013+0000 7f3212669700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f320c14fd90 0x7f320c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:58.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.013+0000 7f3212669700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c067fa0 con 0x7f320c0cd040 2023-10-10T19:25:58.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.013+0000 7f3212669700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c156b70 con 0x7f320c08a100 2023-10-10T19:25:58.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.013+0000 7f3212669700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0c5600 con 0x7f320c14fd90 2023-10-10T19:25:58.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.013+0000 7f3211e68700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f320c0cd040 0x7f320c0684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50676/0 (socket says 172.21.15.175:50676) 2023-10-10T19:25:58.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.013+0000 7f3211e68700 1 -- 172.21.15.175:0/527104622 learned_addr learned my addr 172.21.15.175:0/527104622 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:25:58.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3211667700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f320c14fd90 0x7f320c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:58.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896483197 0 0) 0x7f320c067fa0 con 0x7f320c0cd040 2023-10-10T19:25:58.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f8003570 con 0x7f320c0cd040 2023-10-10T19:25:58.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1508821853 0 0) 0x7f31f8003570 con 0x7f320c0cd040 2023-10-10T19:25:58.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f320c14fd90 msgr2=0x7f320c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:58.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f320c14fd90 0x7f320c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 >> v1:172.21.15.191:6789/0 conn(0x7f320c08a100 legacy=0x7f320c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:58.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c150160 con 0x7f320c0cd040 2023-10-10T19:25:58.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f32040017f0 con 0x7f320c0cd040 2023-10-10T19:25:58.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 >> v1:172.21.15.175:6789/0 conn(0x7f320c0cd040 legacy=0x7f320c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:58.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31f8003a90 0x7f31f8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:58.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31f8004810 0x7f31f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:58.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f320c14fd90 0x7f31f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:58.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0c5600 con 0x7f31f8004810 2023-10-10T19:25:58.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800bb10 con 0x7f31f8003a90 2023-10-10T19:25:58.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800bcd0 con 0x7f320c14fd90 2023-10-10T19:25:58.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3210e66700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31f8004810 0x7f31f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:58.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3211667700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31f8003a90 0x7f31f8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:58.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3211e68700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f320c14fd90 0x7f31f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:58.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3210e66700 1 -- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f320c14fd90 msgr2=0x7f31f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:58.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3210e66700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f320c14fd90 0x7f31f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3210e66700 1 -- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31f8003a90 msgr2=0x7f31f8003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:25:58.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3210e66700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31f8003a90 0x7f31f8003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3210e66700 1 -- 172.21.15.175:0/527104622 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f8003570 con 0x7f31f8004810 2023-10-10T19:25:58.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.014+0000 7f3210e66700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31f8004810 0x7f31f8004c80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f31fc001a60 tx=0x7f31fc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966e991e9455442a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:58.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc001ea0 con 0x7f31f8004810 2023-10-10T19:25:58.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc00c590 con 0x7f31f8004810 2023-10-10T19:25:58.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3203fff700 1 -- 172.21.15.175:0/527104622 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00c930 con 0x7f31f8004810 2023-10-10T19:25:58.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3212669700 1 -- 172.21.15.175:0/527104622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31f8004810 msgr2=0x7f31f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:58.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3212669700 1 --2- 172.21.15.175:0/527104622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31f8004810 0x7f31f8004c80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f31fc001a60 tx=0x7f31fc007d30 comp rx=0 tx=0).stop 2023-10-10T19:25:58.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3211e68700 1 -- 172.21.15.175:0/527104622 reap_dead start 2023-10-10T19:25:58.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3212669700 1 -- 172.21.15.175:0/527104622 shutdown_connections 2023-10-10T19:25:58.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3212669700 1 -- 172.21.15.175:0/527104622 >> 172.21.15.175:0/527104622 conn(0x7f320c063d00 msgr2=0x7f320c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:58.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3212669700 1 -- 172.21.15.175:0/527104622 shutdown_connections 2023-10-10T19:25:58.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.015+0000 7f3212669700 1 -- 172.21.15.175:0/527104622 wait complete. 2023-10-10T19:25:58.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.016+0000 7f3212669700 1 Processor -- start 2023-10-10T19:25:58.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.016+0000 7f3212669700 1 -- start start 2023-10-10T19:25:58.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3212669700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31f8004810 0x7f320c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:58.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3212669700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f320c156da0 0x7f320c154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:58.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3212669700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f320c1553f0 0x7f320c168a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:58.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3212669700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0cbba0 con 0x7f320c1553f0 2023-10-10T19:25:58.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3212669700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0ca5c0 con 0x7f320c156da0 2023-10-10T19:25:58.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3212669700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0ca8a0 con 0x7f31f8004810 2023-10-10T19:25:58.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3211e68700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f320c1553f0 0x7f320c168a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:58.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3211e68700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f320c1553f0 0x7f320c168a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44518/0 (socket says 172.21.15.175:44518) 2023-10-10T19:25:58.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3211e68700 1 -- 172.21.15.175:0/782009670 learned_addr learned my addr 172.21.15.175:0/782009670 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:25:58.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3211667700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31f8004810 0x7f320c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:58.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.017+0000 7f3210e66700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f320c156da0 0x7f320c154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:58.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f3211e68700 1 -- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31f8004810 msgr2=0x7f320c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:58.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f3211e68700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31f8004810 0x7f320c0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f3211e68700 1 -- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f320c156da0 msgr2=0x7f320c154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:58.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f3211e68700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f320c156da0 0x7f320c154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f3211e68700 1 -- 172.21.15.175:0/782009670 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c0cab80 con 0x7f320c1553f0 2023-10-10T19:25:58.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f3211e68700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f320c1553f0 0x7f320c168a00 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3204000f80 tx=0x7f320c0c5dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:58.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f32017fa700 1 -- 172.21.15.175:0/782009670 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204007b60 con 0x7f320c1553f0 2023-10-10T19:25:58.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f32017fa700 1 -- 172.21.15.175:0/782009670 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320400ac90 con 0x7f320c1553f0 2023-10-10T19:25:58.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c0cae60 con 0x7f320c1553f0 2023-10-10T19:25:58.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f32017fa700 1 -- 172.21.15.175:0/782009670 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204013620 con 0x7f320c1553f0 2023-10-10T19:25:58.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.018+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c155890 con 0x7f320c1553f0 2023-10-10T19:25:58.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.019+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c14f6b0 con 0x7f320c1553f0 2023-10-10T19:25:58.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.019+0000 7f32017fa700 1 -- 172.21.15.175:0/782009670 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f32040139d0 con 0x7f320c1553f0 2023-10-10T19:25:58.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.020+0000 7f32017fa700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31f0066f60 0x7f31f0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:25:58.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.020+0000 7f32017fa700 1 -- 172.21.15.175:0/782009670 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f3204085420 con 0x7f320c1553f0 2023-10-10T19:25:58.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.020+0000 7f3211667700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31f0066f60 0x7f31f0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:25:58.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.020+0000 7f3211667700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31f0066f60 0x7f31f0069410 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3208006f00 tx=0x7f32080091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:25:58.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.024+0000 7f32017fa700 1 -- 172.21.15.175:0/782009670 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3204055940 con 0x7f320c1553f0 2023-10-10T19:25:58.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.254+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f320c067570 con 0x7f320c1553f0 2023-10-10T19:25:58.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.255+0000 7f32017fa700 1 -- 172.21.15.175:0/782009670 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 299 v299) v1 ==== 95+0+5421 (secure 0 0 0) 0x7f32040554d0 con 0x7f320c1553f0 2023-10-10T19:25:58.255 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:25:58.256 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":299,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":299,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:58.114537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:25:58.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31f0066f60 msgr2=0x7f31f0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:58.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31f0066f60 0x7f31f0069410 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3208006f00 tx=0x7f32080091a0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f320c1553f0 msgr2=0x7f320c168a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:25:58.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f320c1553f0 0x7f320c168a00 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3204000f80 tx=0x7f320c0c5dd0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 shutdown_connections 2023-10-10T19:25:58.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f31f0066f60 0x7f31f0069410 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f320c1553f0 0x7f320c168a00 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31f8004810 0x7f320c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 --2- 172.21.15.175:0/782009670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f320c156da0 0x7f320c154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:25:58.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 >> 172.21.15.175:0/782009670 conn(0x7f320c063d00 msgr2=0x7f320c0683f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:25:58.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 shutdown_connections 2023-10-10T19:25:58.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:25:58.257+0000 7f3212669700 1 -- 172.21.15.175:0/782009670 wait complete. 2023-10-10T19:25:58.260 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 299 2023-10-10T19:25:58.271 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 299, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 299, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 117 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:reconnect", "state_seq": 10 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 130, "max_file_size": 1099511627776, "max_mds": 4, "metadata_pool": 2, "modified": "2023-10-10T19:25:58.114537+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 7401, "mds_2": 8049, "mds_3": 7771 } } } ], "standbys": [ { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 257, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:26:00.271 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:26:00.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.398+0000 7fb1e4fa1700 1 Processor -- start 2023-10-10T19:26:00.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.398+0000 7fb1e4fa1700 1 -- start start 2023-10-10T19:26:00.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1e4fa1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb1e014e140 0x7fb1e0155270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:00.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1e4fa1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e006b740 con 0x7fb1e006edb0 2023-10-10T19:26:00.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1e4fa1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e006c5a0 con 0x7fb1e00c1210 2023-10-10T19:26:00.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1e4fa1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e00c0960 con 0x7fb1e014e140 2023-10-10T19:26:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1ded9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fb1e006edb0 0x7fb1e015e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50682/0 (socket says 172.21.15.175:50682) 2023-10-10T19:26:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1ded9d700 1 -- 172.21.15.175:0/2192388435 learned_addr learned my addr 172.21.15.175:0/2192388435 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:26:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276703330 0 0) 0x7fb1e006b740 con 0x7fb1e006edb0 2023-10-10T19:26:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8003570 con 0x7fb1e006edb0 2023-10-10T19:26:00.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1de59c700 1 --2- 172.21.15.175:0/2192388435 >> v2:172.21.15.191:3300/0 conn(0x7fb1e014e140 0x7fb1e0155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.399+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 698009859 0 0) 0x7fb1c8003570 con 0x7fb1e006edb0 2023-10-10T19:26:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 >> v2:172.21.15.191:3300/0 conn(0x7fb1e014e140 msgr2=0x7fb1e0155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 --2- 172.21.15.175:0/2192388435 >> v2:172.21.15.191:3300/0 conn(0x7fb1e014e140 0x7fb1e0155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 >> v1:172.21.15.191:6789/0 conn(0x7fb1e00c1210 legacy=0x7fb1e00bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:00.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e014e510 con 0x7fb1e006edb0 2023-10-10T19:26:00.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fb1d80017f0 con 0x7fb1e006edb0 2023-10-10T19:26:00.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 >> v1:172.21.15.175:6789/0 conn(0x7fb1e006edb0 legacy=0x7fb1e015e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:00.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1c8003a60 0x7fb1c8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:00.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1c8004810 0x7fb1c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:00.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c80051c0 0x7fb1c800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:00.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00c0960 con 0x7fb1c8003a60 2023-10-10T19:26:00.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c800c0d0 con 0x7fb1c80051c0 2023-10-10T19:26:00.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c800c290 con 0x7fb1c8004810 2023-10-10T19:26:00.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1de59c700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1c8003a60 0x7fb1c8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:00.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1ded9d700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c80051c0 0x7fb1c800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:00.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1ddd9b700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1c8004810 0x7fb1c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:00.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1de59c700 1 -- 172.21.15.175:0/2192388435 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1c8004810 msgr2=0x7fb1c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:00.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1de59c700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1c8004810 0x7fb1c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1de59c700 1 -- 172.21.15.175:0/2192388435 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c80051c0 msgr2=0x7fb1c800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:00.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1de59c700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c80051c0 0x7fb1c800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1ded9d700 1 -- 172.21.15.175:0/2192388435 reap_dead start 2023-10-10T19:26:00.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1de59c700 1 -- 172.21.15.175:0/2192388435 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c8003570 con 0x7fb1c8003a60 2023-10-10T19:26:00.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1de59c700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1c8003a60 0x7fb1c8003e70 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009c00 tx=0x7fb1d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8a3c1fe6bb79db server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:00.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000e710 con 0x7fb1c8003a60 2023-10-10T19:26:00.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d000e870 con 0x7fb1c8003a60 2023-10-10T19:26:00.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.400+0000 7fb1dd59a700 1 -- 172.21.15.175:0/2192388435 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000ec30 con 0x7fb1c8003a60 2023-10-10T19:26:00.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.401+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2192388435 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1c8003a60 msgr2=0x7fb1c8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:00.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.401+0000 7fb1e4fa1700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1c8003a60 0x7fb1c8003e70 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009c00 tx=0x7fb1d00091a0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.401+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2192388435 shutdown_connections 2023-10-10T19:26:00.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.401+0000 7fb1e4fa1700 1 --2- 172.21.15.175:0/2192388435 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1c8003a60 0x7fb1c8003e70 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.401+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2192388435 >> 172.21.15.175:0/2192388435 conn(0x7fb1e0063cb0 msgr2=0x7fb1e006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:00.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.401+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2192388435 shutdown_connections 2023-10-10T19:26:00.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.401+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2192388435 wait complete. 2023-10-10T19:26:00.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.402+0000 7fb1e4fa1700 1 Processor -- start 2023-10-10T19:26:00.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.402+0000 7fb1e4fa1700 1 -- start start 2023-10-10T19:26:00.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1e4fa1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c8003a60 0x7fb1e00c3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:00.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1e4fa1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1e00c2ff0 0x7fb1e0167480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:00.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1e4fa1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1e00c65e0 0x7fb1e0146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:00.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1e4fa1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e01500e0 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1e4fa1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e01648d0 con 0x7fb1c8003a60 2023-10-10T19:26:00.415 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1e4fa1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00c7310 con 0x7fb1e00c65e0 2023-10-10T19:26:00.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1e00c2ff0 0x7fb1e0167480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:00.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1de59c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c8003a60 0x7fb1e00c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:00.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1e00c2ff0 0x7fb1e0167480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44542/0 (socket says 172.21.15.175:44542) 2023-10-10T19:26:00.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 -- 172.21.15.175:0/2778047175 learned_addr learned my addr 172.21.15.175:0/2778047175 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:00.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ded9d700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1e00c65e0 0x7fb1e0146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:00.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 -- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1e00c65e0 msgr2=0x7fb1e0146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:00.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1e00c65e0 0x7fb1e0146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 -- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c8003a60 msgr2=0x7fb1e00c3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:00.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c8003a60 0x7fb1e00c3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 -- 172.21.15.175:0/2778047175 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e0164b70 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1de59c700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c8003a60 0x7fb1e00c3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:26:00.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ddd9b700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1e00c2ff0 0x7fb1e0167480 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4002950 tx=0x7fb1d4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:00.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ce7fc700 1 -- 172.21.15.175:0/2778047175 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4003390 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.403+0000 7fb1ce7fc700 1 -- 172.21.15.175:0/2778047175 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d4006820 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.404+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e0151170 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.404+0000 7fb1ce7fc700 1 -- 172.21.15.175:0/2778047175 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4006c00 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.404+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e014a600 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.404+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e0164db0 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.405+0000 7fb1ce7fc700 1 -- 172.21.15.175:0/2778047175 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb1d4007230 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.405+0000 7fb1ce7fc700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1c0066f60 0x7fb1c0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:00.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.405+0000 7fb1ce7fc700 1 -- 172.21.15.175:0/2778047175 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fb1d4084c50 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.405+0000 7fb1de59c700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1c0066f60 0x7fb1c0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:00.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.406+0000 7fb1de59c700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1c0066f60 0x7fb1c0069410 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00091e0 tx=0x7fb1d001a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:00.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.410+0000 7fb1ce7fc700 1 -- 172.21.15.175:0/2778047175 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1e0164db0 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.635+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1e0150fd0 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.635+0000 7fb1ce7fc700 1 -- 172.21.15.175:0/2778047175 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 299 v299) v1 ==== 95+0+5421 (secure 0 0 0) 0x7fb1e0150fd0 con 0x7fb1e00c2ff0 2023-10-10T19:26:00.636 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:26:00.636 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":299,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":299,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:58.114537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:26:00.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1c0066f60 msgr2=0x7fb1c0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1c0066f60 0x7fb1c0069410 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00091e0 tx=0x7fb1d001a000 comp rx=0 tx=0).stop 2023-10-10T19:26:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1e00c2ff0 msgr2=0x7fb1e0167480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1e00c2ff0 0x7fb1e0167480 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4002950 tx=0x7fb1d4007d30 comp rx=0 tx=0).stop 2023-10-10T19:26:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 shutdown_connections 2023-10-10T19:26:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb1c0066f60 0x7fb1c0069410 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1e00c65e0 0x7fb1e0146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1c8003a60 0x7fb1e00c3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 --2- 172.21.15.175:0/2778047175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1e00c2ff0 0x7fb1e0167480 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:00.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 >> 172.21.15.175:0/2778047175 conn(0x7fb1e0063cb0 msgr2=0x7fb1e0157820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:00.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 shutdown_connections 2023-10-10T19:26:00.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:00.637+0000 7fb1e4fa1700 1 -- 172.21.15.175:0/2778047175 wait complete. 2023-10-10T19:26:00.640 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 299 2023-10-10T19:26:00.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:26:02.650 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:26:02.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.777+0000 7faf3015a700 1 Processor -- start 2023-10-10T19:26:02.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.777+0000 7faf3015a700 1 -- start start 2023-10-10T19:26:02.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf3015a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faf28063e20 0x7faf280cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:02.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf3015a700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf280c4170 con 0x7faf280c6780 2023-10-10T19:26:02.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf3015a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf28154be0 con 0x7faf280c6b50 2023-10-10T19:26:02.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf3015a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf280c2c30 con 0x7faf28063e20 2023-10-10T19:26:02.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2def6700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7faf280c6780 0x7faf2806b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50684/0 (socket says 172.21.15.175:50684) 2023-10-10T19:26:02.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2def6700 1 -- 172.21.15.175:0/2527788961 learned_addr learned my addr 172.21.15.175:0/2527788961 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:26:02.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2d6f5700 1 --2- 172.21.15.175:0/2527788961 >> v2:172.21.15.191:3300/0 conn(0x7faf28063e20 0x7faf280cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337197678 0 0) 0x7faf280c4170 con 0x7faf280c6780 2023-10-10T19:26:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf14003570 con 0x7faf280c6780 2023-10-10T19:26:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 429083569 0 0) 0x7faf14003570 con 0x7faf280c6780 2023-10-10T19:26:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 >> v2:172.21.15.191:3300/0 conn(0x7faf28063e20 msgr2=0x7faf280cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 --2- 172.21.15.175:0/2527788961 >> v2:172.21.15.191:3300/0 conn(0x7faf28063e20 0x7faf280cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 >> v1:172.21.15.191:6789/0 conn(0x7faf280c6b50 legacy=0x7faf28063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280c3e90 con 0x7faf280c6780 2023-10-10T19:26:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7faf24001970 con 0x7faf280c6780 2023-10-10T19:26:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 >> v1:172.21.15.175:6789/0 conn(0x7faf280c6780 legacy=0x7faf2806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14003a60 0x7faf14003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:02.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:02.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140051c0 0x7faf1400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:02.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280c2c30 con 0x7faf14004810 2023-10-10T19:26:02.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1400c160 con 0x7faf140051c0 2023-10-10T19:26:02.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1400c320 con 0x7faf14003a60 2023-10-10T19:26:02.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.778+0000 7faf2d6f5700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:02.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2def6700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14003a60 0x7faf14003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:02.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2e6f7700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140051c0 0x7faf1400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:02.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2d6f5700 1 -- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14003a60 msgr2=0x7faf14003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:02.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2d6f5700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14003a60 0x7faf14003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:02.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2d6f5700 1 -- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140051c0 msgr2=0x7faf1400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:02.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2d6f5700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140051c0 0x7faf1400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:02.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2d6f5700 1 -- 172.21.15.175:0/2527788961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf14003570 con 0x7faf14004810 2023-10-10T19:26:02.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2e6f7700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140051c0 0x7faf1400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:26:02.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2def6700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14003a60 0x7faf14003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:26:02.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2e6f7700 1 -- 172.21.15.175:0/2527788961 reap_dead start 2023-10-10T19:26:02.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2d6f5700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf14004c80 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7faf18009bf0 tx=0x7faf18009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd91d5786c683491 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:02.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf18005a00 con 0x7faf14004810 2023-10-10T19:26:02.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf18005b60 con 0x7faf14004810 2023-10-10T19:26:02.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.779+0000 7faf2cef4700 1 -- 172.21.15.175:0/2527788961 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1800e710 con 0x7faf14004810 2023-10-10T19:26:02.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.780+0000 7faf3015a700 1 -- 172.21.15.175:0/2527788961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 msgr2=0x7faf14004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:02.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.780+0000 7faf3015a700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf14004c80 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7faf18009bf0 tx=0x7faf18009120 comp rx=0 tx=0).stop 2023-10-10T19:26:02.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.780+0000 7faf3015a700 1 -- 172.21.15.175:0/2527788961 shutdown_connections 2023-10-10T19:26:02.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.780+0000 7faf3015a700 1 --2- 172.21.15.175:0/2527788961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf14004c80 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:02.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.780+0000 7faf3015a700 1 -- 172.21.15.175:0/2527788961 >> 172.21.15.175:0/2527788961 conn(0x7faf2806dbe0 msgr2=0x7faf280ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:02.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.780+0000 7faf3015a700 1 -- 172.21.15.175:0/2527788961 shutdown_connections 2023-10-10T19:26:02.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.780+0000 7faf3015a700 1 -- 172.21.15.175:0/2527788961 wait complete. 2023-10-10T19:26:02.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.781+0000 7faf3015a700 1 Processor -- start 2023-10-10T19:26:02.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.781+0000 7faf3015a700 1 -- start start 2023-10-10T19:26:02.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.781+0000 7faf3015a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf281587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:02.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.781+0000 7faf3015a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf280c9250 0x7faf28155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:02.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.781+0000 7faf3015a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf28159d30 0x7faf281557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:02.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.781+0000 7faf3015a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf281639b0 con 0x7faf14004810 2023-10-10T19:26:02.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.781+0000 7faf3015a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280bf6b0 con 0x7faf280c9250 2023-10-10T19:26:02.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.781+0000 7faf3015a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf280bfd60 con 0x7faf28159d30 2023-10-10T19:26:02.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf281587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:02.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf281587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:44548/0 (socket says 172.21.15.175:44548) 2023-10-10T19:26:02.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 -- 172.21.15.175:0/3782577826 learned_addr learned my addr 172.21.15.175:0/3782577826 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:02.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2e6f7700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf28159d30 0x7faf281557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:02.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2d6f5700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf280c9250 0x7faf28155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:02.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 -- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf28159d30 msgr2=0x7faf281557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:02.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf28159d30 0x7faf281557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:02.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 -- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf280c9250 msgr2=0x7faf28155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:02.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf280c9250 0x7faf28155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:02.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 -- 172.21.15.175:0/3782577826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf28158cf0 con 0x7faf14004810 2023-10-10T19:26:02.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf2def6700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf281587b0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7faf24003260 tx=0x7faf24009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:02.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf1dffb700 1 -- 172.21.15.175:0/3782577826 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24005d80 con 0x7faf14004810 2023-10-10T19:26:02.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf1dffb700 1 -- 172.21.15.175:0/3782577826 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2400c8d0 con 0x7faf14004810 2023-10-10T19:26:02.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf1dffb700 1 -- 172.21.15.175:0/3782577826 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2400cc50 con 0x7faf14004810 2023-10-10T19:26:02.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf280c8e00 con 0x7faf14004810 2023-10-10T19:26:02.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.782+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf28163e80 con 0x7faf14004810 2023-10-10T19:26:02.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.783+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf28067570 con 0x7faf14004810 2023-10-10T19:26:02.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.783+0000 7faf1dffb700 1 -- 172.21.15.175:0/3782577826 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faf240072e0 con 0x7faf14004810 2023-10-10T19:26:02.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.784+0000 7faf1dffb700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faf0c066f30 0x7faf0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:02.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.784+0000 7faf1dffb700 1 -- 172.21.15.175:0/3782577826 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7faf240851e0 con 0x7faf14004810 2023-10-10T19:26:02.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.784+0000 7faf2d6f5700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faf0c066f30 0x7faf0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:02.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.784+0000 7faf2d6f5700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faf0c066f30 0x7faf0c0693e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7faf28159780 tx=0x7faf18019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:02.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:02.788+0000 7faf1dffb700 1 -- 172.21.15.175:0/3782577826 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf24051c70 con 0x7faf14004810 2023-10-10T19:26:03.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.014+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf280cb260 con 0x7faf14004810 2023-10-10T19:26:03.015 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.015+0000 7faf1dffb700 1 -- 172.21.15.175:0/3782577826 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 299 v299) v1 ==== 95+0+5421 (secure 0 0 0) 0x7faf2400cdb0 con 0x7faf14004810 2023-10-10T19:26:03.016 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:26:03.016 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":299,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":299,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:25:58.114537+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:26:03.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faf0c066f30 msgr2=0x7faf0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:03.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faf0c066f30 0x7faf0c0693e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7faf28159780 tx=0x7faf18019000 comp rx=0 tx=0).stop 2023-10-10T19:26:03.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 msgr2=0x7faf281587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:03.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf281587b0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7faf24003260 tx=0x7faf24009040 comp rx=0 tx=0).stop 2023-10-10T19:26:03.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 shutdown_connections 2023-10-10T19:26:03.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faf0c066f30 0x7faf0c0693e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:03.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf28159d30 0x7faf281557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:03.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf14004810 0x7faf281587b0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:03.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 --2- 172.21.15.175:0/3782577826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf280c9250 0x7faf28155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:03.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 >> 172.21.15.175:0/3782577826 conn(0x7faf2806dbe0 msgr2=0x7faf280c8350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:03.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 shutdown_connections 2023-10-10T19:26:03.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:03.017+0000 7faf3015a700 1 -- 172.21.15.175:0/3782577826 wait complete. 2023-10-10T19:26:03.020 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 299 2023-10-10T19:26:03.029 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:26:05.031 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:26:05.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.157+0000 7fde71f2e700 1 Processor -- start 2023-10-10T19:26:05.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.157+0000 7fde71f2e700 1 -- start start 2023-10-10T19:26:05.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.158+0000 7fde71f2e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0b1fc0 0x7fde6c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:05.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.158+0000 7fde71f2e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fde6c0aff60 0x7fde6c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:05.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.158+0000 7fde71f2e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c1574b0 con 0x7fde6c0b2390 2023-10-10T19:26:05.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.158+0000 7fde71f2e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c150770 con 0x7fde6c0aff60 2023-10-10T19:26:05.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.158+0000 7fde71f2e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0afad0 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.158+0000 7fde6bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fde6c0aff60 0x7fde6c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:05.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.158+0000 7fde6affd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fde6c0b2390 0x7fde6c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35420/0 (socket says 172.21.15.175:35420) 2023-10-10T19:26:05.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.158+0000 7fde6affd700 1 -- 172.21.15.175:0/2892271131 learned_addr learned my addr 172.21.15.175:0/2892271131 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:26:05.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6b7fe700 1 --2- 172.21.15.175:0/2892271131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0b1fc0 0x7fde6c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:05.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6bfff700 1 -- 172.21.15.175:0/2892271131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0b1fc0 msgr2=0x7fde6c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:05.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6bfff700 1 --2- 172.21.15.175:0/2892271131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0b1fc0 0x7fde6c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6bfff700 1 -- 172.21.15.175:0/2892271131 >> v1:172.21.15.175:6789/0 conn(0x7fde6c0b2390 legacy=0x7fde6c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:05.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6bfff700 1 -- 172.21.15.175:0/2892271131 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0af4c0 con 0x7fde6c0aff60 2023-10-10T19:26:05.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6a7fc700 1 -- 172.21.15.175:0/2892271131 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812623231 0 0) 0x7fde6c1574b0 con 0x7fde6c0b2390 2023-10-10T19:26:05.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6bfff700 1 --2- 172.21.15.175:0/2892271131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0aff60 0x7fde6c0b03e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fde64009910 tx=0x7fde640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6b62afd48bece3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:05.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6a7fc700 1 -- 172.21.15.175:0/2892271131 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6400b8b0 con 0x7fde6c0aff60 2023-10-10T19:26:05.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6a7fc700 1 -- 172.21.15.175:0/2892271131 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6400ba10 con 0x7fde6c0aff60 2023-10-10T19:26:05.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde6a7fc700 1 -- 172.21.15.175:0/2892271131 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6400bdb0 con 0x7fde6c0aff60 2023-10-10T19:26:05.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde71f2e700 1 -- 172.21.15.175:0/2892271131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0aff60 msgr2=0x7fde6c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:05.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.159+0000 7fde71f2e700 1 --2- 172.21.15.175:0/2892271131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0aff60 0x7fde6c0b03e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fde64009910 tx=0x7fde640091a0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.160+0000 7fde71f2e700 1 -- 172.21.15.175:0/2892271131 shutdown_connections 2023-10-10T19:26:05.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.160+0000 7fde71f2e700 1 --2- 172.21.15.175:0/2892271131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0aff60 0x7fde6c0b03e0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.160+0000 7fde71f2e700 1 --2- 172.21.15.175:0/2892271131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0b1fc0 0x7fde6c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.160+0000 7fde71f2e700 1 -- 172.21.15.175:0/2892271131 >> 172.21.15.175:0/2892271131 conn(0x7fde6c063230 msgr2=0x7fde6c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:05.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.160+0000 7fde71f2e700 1 -- 172.21.15.175:0/2892271131 shutdown_connections 2023-10-10T19:26:05.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.160+0000 7fde71f2e700 1 -- 172.21.15.175:0/2892271131 wait complete. 2023-10-10T19:26:05.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.161+0000 7fde71f2e700 1 Processor -- start 2023-10-10T19:26:05.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.161+0000 7fde71f2e700 1 -- start start 2023-10-10T19:26:05.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.161+0000 7fde71f2e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0aff60 0x7fde6c1664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:05.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.161+0000 7fde71f2e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0b1fc0 0x7fde6c0ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:05.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.161+0000 7fde71f2e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c0b2390 0x7fde6c0c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:05.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.161+0000 7fde71f2e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0c40f0 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.161+0000 7fde71f2e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0ccf20 con 0x7fde6c0b2390 2023-10-10T19:26:05.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.161+0000 7fde71f2e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0c4ec0 con 0x7fde6c0aff60 2023-10-10T19:26:05.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0b1fc0 0x7fde6c0ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0b1fc0 0x7fde6c0ca960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55934/0 (socket says 172.21.15.175:55934) 2023-10-10T19:26:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 -- 172.21.15.175:0/1176460666 learned_addr learned my addr 172.21.15.175:0/1176460666 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6b7fe700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0aff60 0x7fde6c1664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6bfff700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c0b2390 0x7fde6c0c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:05.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 -- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0aff60 msgr2=0x7fde6c1664e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:05.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0aff60 0x7fde6c1664e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 -- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c0b2390 msgr2=0x7fde6c0c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:05.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c0b2390 0x7fde6c0c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 -- 172.21.15.175:0/1176460666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64009420 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde6affd700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0b1fc0 0x7fde6c0ca960 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c5b0 tx=0x7fde60001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:05.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde537fe700 1 -- 172.21.15.175:0/1176460666 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60005b30 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde537fe700 1 -- 172.21.15.175:0/1176460666 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde60005c90 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde537fe700 1 -- 172.21.15.175:0/1176460666 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60007320 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c0c3870 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.162+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c0c3e10 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.163+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c067570 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.164+0000 7fde537fe700 1 -- 172.21.15.175:0/1176460666 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fde6001b3b0 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.164+0000 7fde537fe700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde58066f30 0x7fde580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:05.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.164+0000 7fde537fe700 1 -- 172.21.15.175:0/1176460666 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fde600840e0 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.164+0000 7fde6b7fe700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde58066f30 0x7fde580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:05.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.165+0000 7fde6b7fe700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde58066f30 0x7fde580693e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009fd0 tx=0x7fde5c009880 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:05.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.168+0000 7fde537fe700 1 -- 172.21.15.175:0/1176460666 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde60050b70 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.393+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde6c068da0 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.394+0000 7fde537fe700 1 -- 172.21.15.175:0/1176460666 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 300 v300) v1 ==== 95+0+5418 (secure 0 0 0) 0x7fde60054190 con 0x7fde6c0b1fc0 2023-10-10T19:26:05.396 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:26:05.396 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":300,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":300,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:26:03.127785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:26:05.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde58066f30 msgr2=0x7fde580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:05.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde58066f30 0x7fde580693e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009fd0 tx=0x7fde5c009880 comp rx=0 tx=0).stop 2023-10-10T19:26:05.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0b1fc0 msgr2=0x7fde6c0ca960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:05.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0b1fc0 0x7fde6c0ca960 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c5b0 tx=0x7fde60001070 comp rx=0 tx=0).stop 2023-10-10T19:26:05.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 shutdown_connections 2023-10-10T19:26:05.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fde58066f30 0x7fde580693e0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde6c0b2390 0x7fde6c0c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6c0aff60 0x7fde6c1664e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 --2- 172.21.15.175:0/1176460666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fde6c0b1fc0 0x7fde6c0ca960 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:05.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 >> 172.21.15.175:0/1176460666 conn(0x7fde6c063230 msgr2=0x7fde6c153dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:05.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.396+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 shutdown_connections 2023-10-10T19:26:05.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:05.397+0000 7fde71f2e700 1 -- 172.21.15.175:0/1176460666 wait complete. 2023-10-10T19:26:05.400 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 300 2023-10-10T19:26:05.409 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:26:07.410 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:26:07.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.539+0000 7f27a593f700 1 Processor -- start 2023-10-10T19:26:07.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.539+0000 7f27a593f700 1 -- start start 2023-10-10T19:26:07.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.540+0000 7f27a593f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f27a00c5850 0x7f27a00c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:07.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.540+0000 7f27a593f700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a00c6fb0 con 0x7f27a00c12d0 2023-10-10T19:26:07.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.540+0000 7f27a593f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a00c6160 con 0x7f27a00c4f20 2023-10-10T19:26:07.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.540+0000 7f27a593f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a0063ea0 con 0x7f27a00c5850 2023-10-10T19:26:07.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.540+0000 7f279e7fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f27a00c12d0 0x7f27a0156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35428/0 (socket says 172.21.15.175:35428) 2023-10-10T19:26:07.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.540+0000 7f279e7fc700 1 -- 172.21.15.175:0/3536612906 learned_addr learned my addr 172.21.15.175:0/3536612906 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:26:07.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279effd700 1 --2- 172.21.15.175:0/3536612906 >> v2:172.21.15.191:3300/0 conn(0x7f27a00c5850 0x7f27a00c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:07.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923308950 0 0) 0x7f27a00c6fb0 con 0x7f27a00c12d0 2023-10-10T19:26:07.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2788003570 con 0x7f27a00c12d0 2023-10-10T19:26:07.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132837129 0 0) 0x7f2788003570 con 0x7f27a00c12d0 2023-10-10T19:26:07.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 >> v2:172.21.15.191:3300/0 conn(0x7f27a00c5850 msgr2=0x7f27a00c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:07.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 --2- 172.21.15.175:0/3536612906 >> v2:172.21.15.191:3300/0 conn(0x7f27a00c5850 0x7f27a00c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 >> v1:172.21.15.191:6789/0 conn(0x7f27a00c4f20 legacy=0x7f27a00c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:07.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a0063940 con 0x7f27a00c12d0 2023-10-10T19:26:07.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f2794001470 con 0x7f27a00c12d0 2023-10-10T19:26:07.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 >> v1:172.21.15.175:6789/0 conn(0x7f27a00c12d0 legacy=0x7f27a0156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:07.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2788003a60 0x7f2788003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:07.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788004810 0x7f2788004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:07.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27880051c0 0x7f278800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:07.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0063ea0 con 0x7f2788003a60 2023-10-10T19:26:07.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278800c0d0 con 0x7f2788004810 2023-10-10T19:26:07.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278800c290 con 0x7f27880051c0 2023-10-10T19:26:07.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279effd700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2788003a60 0x7f2788003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:07.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279e7fc700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788004810 0x7f2788004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:07.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279f7fe700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27880051c0 0x7f278800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:07.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279effd700 1 -- 172.21.15.175:0/3536612906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27880051c0 msgr2=0x7f278800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:07.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279effd700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27880051c0 0x7f278800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279effd700 1 -- 172.21.15.175:0/3536612906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788004810 msgr2=0x7f2788004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:07.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279effd700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788004810 0x7f2788004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279f7fe700 1 -- 172.21.15.175:0/3536612906 reap_dead start 2023-10-10T19:26:07.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279effd700 1 -- 172.21.15.175:0/3536612906 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788003570 con 0x7f2788003a60 2023-10-10T19:26:07.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.541+0000 7f279effd700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2788003a60 0x7f2788003e70 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f279000e210 tx=0x7f279000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5ae2b6484e6e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:07.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279000eac0 con 0x7f2788003a60 2023-10-10T19:26:07.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279000f040 con 0x7f2788003a60 2023-10-10T19:26:07.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f279dffb700 1 -- 172.21.15.175:0/3536612906 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279000ee50 con 0x7f2788003a60 2023-10-10T19:26:07.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f27a593f700 1 -- 172.21.15.175:0/3536612906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2788003a60 msgr2=0x7f2788003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:07.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f27a593f700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2788003a60 0x7f2788003e70 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f279000e210 tx=0x7f279000b040 comp rx=0 tx=0).stop 2023-10-10T19:26:07.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f27a593f700 1 -- 172.21.15.175:0/3536612906 shutdown_connections 2023-10-10T19:26:07.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f27a593f700 1 --2- 172.21.15.175:0/3536612906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2788003a60 0x7f2788003e70 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f27a593f700 1 -- 172.21.15.175:0/3536612906 >> 172.21.15.175:0/3536612906 conn(0x7f27a006dc70 msgr2=0x7f27a0063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:07.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f27a593f700 1 -- 172.21.15.175:0/3536612906 shutdown_connections 2023-10-10T19:26:07.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.542+0000 7f27a593f700 1 -- 172.21.15.175:0/3536612906 wait complete. 2023-10-10T19:26:07.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.543+0000 7f27a593f700 1 Processor -- start 2023-10-10T19:26:07.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f27a593f700 1 -- start start 2023-10-10T19:26:07.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f27a593f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788003a60 0x7f27a0169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:07.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f27a593f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a0165b80 0x7f27a0150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:07.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f27a593f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00cb040 0x7f27a014e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:07.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f27a593f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01691a0 con 0x7f27a0165b80 2023-10-10T19:26:07.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f27a593f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00c88f0 con 0x7f2788003a60 2023-10-10T19:26:07.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f27a593f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a014ffb0 con 0x7f27a00cb040 2023-10-10T19:26:07.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f279e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a0165b80 0x7f27a0150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:07.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f279effd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788003a60 0x7f27a0169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:07.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f279e7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a0165b80 0x7f27a0150aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55942/0 (socket says 172.21.15.175:55942) 2023-10-10T19:26:07.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f279e7fc700 1 -- 172.21.15.175:0/3733976398 learned_addr learned my addr 172.21.15.175:0/3733976398 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:07.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.544+0000 7f279f7fe700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00cb040 0x7f27a014e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:07.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f279e7fc700 1 -- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00cb040 msgr2=0x7f27a014e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:07.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f279e7fc700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00cb040 0x7f27a014e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f279e7fc700 1 -- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788003a60 msgr2=0x7f27a0169930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:07.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f279e7fc700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788003a60 0x7f27a0169930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f279e7fc700 1 -- 172.21.15.175:0/3733976398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a0150290 con 0x7f27a0165b80 2023-10-10T19:26:07.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f279e7fc700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a0165b80 0x7f27a0150aa0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f2794007370 tx=0x7f2794001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:07.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f2786ffd700 1 -- 172.21.15.175:0/3733976398 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794018030 con 0x7f27a0165b80 2023-10-10T19:26:07.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f2786ffd700 1 -- 172.21.15.175:0/3733976398 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2794007750 con 0x7f27a0165b80 2023-10-10T19:26:07.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a0169410 con 0x7f27a0165b80 2023-10-10T19:26:07.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f2786ffd700 1 -- 172.21.15.175:0/3733976398 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279400fbd0 con 0x7f27a0165b80 2023-10-10T19:26:07.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.545+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a0150570 con 0x7f27a0165b80 2023-10-10T19:26:07.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.546+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a0067570 con 0x7f27a0165b80 2023-10-10T19:26:07.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.546+0000 7f2786ffd700 1 -- 172.21.15.175:0/3733976398 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f279401b070 con 0x7f27a0165b80 2023-10-10T19:26:07.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.547+0000 7f2786ffd700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f278c066f30 0x7f278c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:07.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.547+0000 7f2786ffd700 1 -- 172.21.15.175:0/3733976398 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f2794019070 con 0x7f27a0165b80 2023-10-10T19:26:07.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.547+0000 7f279effd700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f278c066f30 0x7f278c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:07.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.547+0000 7f279effd700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f278c066f30 0x7f278c0693e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f279000afd0 tx=0x7f279001a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:07.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.551+0000 7f2786ffd700 1 -- 172.21.15.175:0/3733976398 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27940516c0 con 0x7f27a0165b80 2023-10-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.777+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27a00afc40 con 0x7f27a0165b80 2023-10-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.778+0000 7f2786ffd700 1 -- 172.21.15.175:0/3733976398 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 300 v300) v1 ==== 95+0+5418 (secure 0 0 0) 0x7f2794054ce0 con 0x7f27a0165b80 2023-10-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":300,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":300,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:26:03.127785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f278c066f30 msgr2=0x7f278c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f278c066f30 0x7f278c0693e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f279000afd0 tx=0x7f279001a000 comp rx=0 tx=0).stop 2023-10-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a0165b80 msgr2=0x7f27a0150aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a0165b80 0x7f27a0150aa0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f2794007370 tx=0x7f2794001e50 comp rx=0 tx=0).stop 2023-10-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 shutdown_connections 2023-10-10T19:26:07.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f278c066f30 0x7f278c0693e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27a00cb040 0x7f27a014e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2788003a60 0x7f27a0169930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 --2- 172.21.15.175:0/3733976398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27a0165b80 0x7f27a0150aa0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:07.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 >> 172.21.15.175:0/3733976398 conn(0x7f27a006dc70 msgr2=0x7f27a0153180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 shutdown_connections 2023-10-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:07.780+0000 7f27a593f700 1 -- 172.21.15.175:0/3733976398 wait complete. 2023-10-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 300 2023-10-10T19:26:07.792 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:26:09.793 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:26:09.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.919+0000 7f6821a59700 1 Processor -- start 2023-10-10T19:26:09.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.919+0000 7f6821a59700 1 -- start start 2023-10-10T19:26:09.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f6821a59700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f681c068cf0 0x7f681c067b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:09.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f6821a59700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f681c159dd0 0x7f681c0c2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:09.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f6821a59700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c068b30 con 0x7f681c14fd20 2023-10-10T19:26:09.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f6821a59700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c0c1c70 con 0x7f681c159dd0 2023-10-10T19:26:09.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f6821a59700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c0c2640 con 0x7f681c068cf0 2023-10-10T19:26:09.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f681bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f681c159dd0 0x7f681c0c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:09.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f681bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f681c159dd0 0x7f681c0c2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55946/0 (socket says 172.21.15.175:55946) 2023-10-10T19:26:09.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f681bfff700 1 -- 172.21.15.175:0/2448671103 learned_addr learned my addr 172.21.15.175:0/2448671103 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:09.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.920+0000 7f6821258700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f681c14fd20 0x7f681c0690c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35440/0 (socket says 172.21.15.175:35440) 2023-10-10T19:26:09.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f6820a57700 1 --2- 172.21.15.175:0/2448671103 >> v2:172.21.15.191:3300/0 conn(0x7f681c068cf0 0x7f681c067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:09.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681bfff700 1 -- 172.21.15.175:0/2448671103 >> v2:172.21.15.191:3300/0 conn(0x7f681c068cf0 msgr2=0x7f681c067b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:09.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681bfff700 1 --2- 172.21.15.175:0/2448671103 >> v2:172.21.15.191:3300/0 conn(0x7f681c068cf0 0x7f681c067b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:09.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681bfff700 1 -- 172.21.15.175:0/2448671103 >> v1:172.21.15.175:6789/0 conn(0x7f681c14fd20 legacy=0x7f681c0690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:09.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681bfff700 1 -- 172.21.15.175:0/2448671103 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c0cd2b0 con 0x7f681c159dd0 2023-10-10T19:26:09.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681b7fe700 1 -- 172.21.15.175:0/2448671103 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933595033 0 0) 0x7f681c068b30 con 0x7f681c14fd20 2023-10-10T19:26:09.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f6820a57700 1 --2- 172.21.15.175:0/2448671103 >> v2:172.21.15.191:3300/0 conn(0x7f681c068cf0 0x7f681c067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:26:09.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681bfff700 1 --2- 172.21.15.175:0/2448671103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c159dd0 0x7f681c0c2100 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f6810009f20 tx=0x7f6810009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6e83d4456837be server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:09.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681b7fe700 1 -- 172.21.15.175:0/2448671103 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681000e070 con 0x7f681c159dd0 2023-10-10T19:26:09.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681b7fe700 1 -- 172.21.15.175:0/2448671103 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681000bad0 con 0x7f681c159dd0 2023-10-10T19:26:09.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.921+0000 7f681b7fe700 1 -- 172.21.15.175:0/2448671103 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68100092e0 con 0x7f681c159dd0 2023-10-10T19:26:09.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.922+0000 7f6821a59700 1 -- 172.21.15.175:0/2448671103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c159dd0 msgr2=0x7f681c0c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:09.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.922+0000 7f6821a59700 1 --2- 172.21.15.175:0/2448671103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c159dd0 0x7f681c0c2100 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f6810009f20 tx=0x7f6810009840 comp rx=0 tx=0).stop 2023-10-10T19:26:09.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.922+0000 7f6821a59700 1 -- 172.21.15.175:0/2448671103 shutdown_connections 2023-10-10T19:26:09.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.922+0000 7f6821a59700 1 --2- 172.21.15.175:0/2448671103 >> v2:172.21.15.191:3300/0 conn(0x7f681c068cf0 0x7f681c067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:09.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.922+0000 7f6821a59700 1 --2- 172.21.15.175:0/2448671103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c159dd0 0x7f681c0c2100 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:09.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.922+0000 7f6821a59700 1 -- 172.21.15.175:0/2448671103 >> 172.21.15.175:0/2448671103 conn(0x7f681c063cc0 msgr2=0x7f681c0cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:09.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.922+0000 7f6821a59700 1 -- 172.21.15.175:0/2448671103 shutdown_connections 2023-10-10T19:26:09.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.922+0000 7f6821a59700 1 -- 172.21.15.175:0/2448671103 wait complete. 2023-10-10T19:26:09.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.923+0000 7f6821a59700 1 Processor -- start 2023-10-10T19:26:09.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.923+0000 7f6821a59700 1 -- start start 2023-10-10T19:26:09.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821a59700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f681c068cf0 0x7f681c0c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:09.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821a59700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c14fd20 0x7f681c0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:09.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821a59700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f681c159dd0 0x7f681c153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:09.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821a59700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c155320 con 0x7f681c14fd20 2023-10-10T19:26:09.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821a59700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0c4530 con 0x7f681c159dd0 2023-10-10T19:26:09.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821a59700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c156630 con 0x7f681c068cf0 2023-10-10T19:26:09.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c14fd20 0x7f681c0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:09.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c14fd20 0x7f681c0cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55956/0 (socket says 172.21.15.175:55956) 2023-10-10T19:26:09.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 -- 172.21.15.175:0/3607162328 learned_addr learned my addr 172.21.15.175:0/3607162328 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:09.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821258700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f681c159dd0 0x7f681c153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:09.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6820a57700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f681c068cf0 0x7f681c0c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:09.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 -- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f681c068cf0 msgr2=0x7f681c0c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:09.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f681c068cf0 0x7f681c0c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:09.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 -- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f681c159dd0 msgr2=0x7f681c153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:09.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f681c159dd0 0x7f681c153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:09.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 -- 172.21.15.175:0/3607162328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810009ae0 con 0x7f681c14fd20 2023-10-10T19:26:09.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f681bfff700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c14fd20 0x7f681c0cc020 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6810009ab0 tx=0x7f6810018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:09.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6818ff9700 1 -- 172.21.15.175:0/3607162328 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810008a80 con 0x7f681c14fd20 2023-10-10T19:26:09.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6818ff9700 1 -- 172.21.15.175:0/3607162328 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6810008be0 con 0x7f681c14fd20 2023-10-10T19:26:09.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c0c4770 con 0x7f681c14fd20 2023-10-10T19:26:09.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6818ff9700 1 -- 172.21.15.175:0/3607162328 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681001c410 con 0x7f681c14fd20 2023-10-10T19:26:09.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.924+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c1568d0 con 0x7f681c14fd20 2023-10-10T19:26:09.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.925+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c156470 con 0x7f681c14fd20 2023-10-10T19:26:09.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.926+0000 7f6818ff9700 1 -- 172.21.15.175:0/3607162328 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6810003040 con 0x7f681c14fd20 2023-10-10T19:26:09.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.926+0000 7f6818ff9700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6808066f30 0x7f68080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:09.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.926+0000 7f6818ff9700 1 -- 172.21.15.175:0/3607162328 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f6810086290 con 0x7f681c14fd20 2023-10-10T19:26:09.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.926+0000 7f6820a57700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6808066f30 0x7f68080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:09.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.927+0000 7f6820a57700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6808066f30 0x7f68080693e0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f680c006f00 tx=0x7f680c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:09.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:09.931+0000 7f6818ff9700 1 -- 172.21.15.175:0/3607162328 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6810052d20 con 0x7f681c14fd20 2023-10-10T19:26:10.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.156+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f681c067570 con 0x7f681c14fd20 2023-10-10T19:26:10.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.156+0000 7f6818ff9700 1 -- 172.21.15.175:0/3607162328 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 301 v301) v1 ==== 95+0+5418 (secure 0 0 0) 0x7f6810056340 con 0x7f681c14fd20 2023-10-10T19:26:10.157 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:26:10.157 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":301,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":301,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:26:08.830544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:26:10.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6808066f30 msgr2=0x7f68080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:10.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6808066f30 0x7f68080693e0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f680c006f00 tx=0x7f680c008040 comp rx=0 tx=0).stop 2023-10-10T19:26:10.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c14fd20 msgr2=0x7f681c0cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:10.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c14fd20 0x7f681c0cc020 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6810009ab0 tx=0x7f6810018000 comp rx=0 tx=0).stop 2023-10-10T19:26:10.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 shutdown_connections 2023-10-10T19:26:10.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6808066f30 0x7f68080693e0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:10.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f681c159dd0 0x7f681c153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:10.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f681c068cf0 0x7f681c0c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:10.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 --2- 172.21.15.175:0/3607162328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f681c14fd20 0x7f681c0cc020 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:10.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 >> 172.21.15.175:0/3607162328 conn(0x7f681c063cc0 msgr2=0x7f681c0c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:10.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 shutdown_connections 2023-10-10T19:26:10.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:10.158+0000 7f6821a59700 1 -- 172.21.15.175:0/3607162328 wait complete. 2023-10-10T19:26:10.162 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 301 2023-10-10T19:26:10.171 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=3, replacing gid=7384 2023-10-10T19:26:10.171 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.d 2023-10-10T19:26:12.172 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-10T19:26:12.172 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-10T19:26:12.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-10T19:26:12.175 INFO:tasks.ceph.mds.d:Started 2023-10-10T19:26:12.176 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:26:12.254 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-10T19:26:12.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.304+0000 7f12f66fc700 1 Processor -- start 2023-10-10T19:26:12.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.304+0000 7f12f66fc700 1 -- start start 2023-10-10T19:26:12.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.305+0000 7f12f66fc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f12f00b1fc0 0x7f12f0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:12.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.305+0000 7f12f66fc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f01574b0 con 0x7f12f00b2390 2023-10-10T19:26:12.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.305+0000 7f12f66fc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f00afad0 con 0x7f12f00aff60 2023-10-10T19:26:12.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.305+0000 7f12f66fc700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f0150770 con 0x7f12f00b1fc0 2023-10-10T19:26:12.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.305+0000 7f12effff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f12f00b1fc0 0x7f12f0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:12.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.305+0000 7f12effff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f12f00b1fc0 0x7f12f0155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55958/0 (socket says 172.21.15.175:55958) 2023-10-10T19:26:12.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.305+0000 7f12effff700 1 -- 172.21.15.175:0/112885912 learned_addr learned my addr 172.21.15.175:0/112885912 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:12.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.305+0000 7f12ef7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f12f00b2390 0x7f12f0068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:35450/0 (socket says 172.21.15.175:35450) 2023-10-10T19:26:12.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12effff700 1 -- 172.21.15.175:0/112885912 >> v1:172.21.15.191:6789/0 conn(0x7f12f00aff60 legacy=0x7f12f0159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:12.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12effff700 1 -- 172.21.15.175:0/112885912 >> v1:172.21.15.175:6789/0 conn(0x7f12f00b2390 legacy=0x7f12f0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:12.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12effff700 1 -- 172.21.15.175:0/112885912 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f00af4c0 con 0x7f12f00b1fc0 2023-10-10T19:26:12.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12effff700 1 --2- 172.21.15.175:0/112885912 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b1fc0 0x7f12f0155fb0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f12e00099b0 tx=0x7f12e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71020ac4361b24d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:12.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12eeffd700 1 -- 172.21.15.175:0/112885912 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e000b8f0 con 0x7f12f00b1fc0 2023-10-10T19:26:12.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12eeffd700 1 -- 172.21.15.175:0/112885912 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e000ba50 con 0x7f12f00b1fc0 2023-10-10T19:26:12.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12eeffd700 1 -- 172.21.15.175:0/112885912 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e000bdf0 con 0x7f12f00b1fc0 2023-10-10T19:26:12.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12f66fc700 1 -- 172.21.15.175:0/112885912 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b1fc0 msgr2=0x7f12f0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:12.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12f66fc700 1 --2- 172.21.15.175:0/112885912 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b1fc0 0x7f12f0155fb0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f12e00099b0 tx=0x7f12e0009250 comp rx=0 tx=0).stop 2023-10-10T19:26:12.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12f66fc700 1 -- 172.21.15.175:0/112885912 shutdown_connections 2023-10-10T19:26:12.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12f66fc700 1 --2- 172.21.15.175:0/112885912 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b1fc0 0x7f12f0155fb0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:12.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12f66fc700 1 -- 172.21.15.175:0/112885912 >> 172.21.15.175:0/112885912 conn(0x7f12f0063230 msgr2=0x7f12f0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:12.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12f66fc700 1 -- 172.21.15.175:0/112885912 shutdown_connections 2023-10-10T19:26:12.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.306+0000 7f12f66fc700 1 -- 172.21.15.175:0/112885912 wait complete. 2023-10-10T19:26:12.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.307+0000 7f12f66fc700 1 Processor -- start 2023-10-10T19:26:12.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.308+0000 7f12f66fc700 1 -- start start 2023-10-10T19:26:12.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.308+0000 7f12f66fc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12f00aff60 0x7f12f0156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:12.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.308+0000 7f12f66fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12f00b1fc0 0x7f12f00ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:12.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.308+0000 7f12f66fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b2390 0x7f12f00ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:12.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.308+0000 7f12f66fc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f01541d0 con 0x7f12f00b2390 2023-10-10T19:26:12.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.308+0000 7f12f66fc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0155a70 con 0x7f12f00b1fc0 2023-10-10T19:26:12.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.308+0000 7f12f66fc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0068900 con 0x7f12f00aff60 2023-10-10T19:26:12.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f4c99700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b2390 0x7f12f00ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:12.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12ef7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12f00b1fc0 0x7f12f00ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:12.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12effff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12f00aff60 0x7f12f0156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:12.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12ef7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12f00b1fc0 0x7f12f00ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:45680/0 (socket says 172.21.15.175:45680) 2023-10-10T19:26:12.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12ef7fe700 1 -- 172.21.15.175:0/3335083248 learned_addr learned my addr 172.21.15.175:0/3335083248 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:12.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f4c99700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b2390 0x7f12f00ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55970/0 (socket says 172.21.15.175:55970) 2023-10-10T19:26:12.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f4c99700 1 -- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12f00aff60 msgr2=0x7f12f0156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:12.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f4c99700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12f00aff60 0x7f12f0156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:12.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f4c99700 1 -- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12f00b1fc0 msgr2=0x7f12f00ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:12.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f4c99700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12f00b1fc0 0x7f12f00ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:12.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f4c99700 1 -- 172.21.15.175:0/3335083248 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e00094f0 con 0x7f12f00b2390 2023-10-10T19:26:12.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12ef7fe700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12f00b1fc0 0x7f12f00ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:26:12.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f4c99700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b2390 0x7f12f00ccd20 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f12e800c830 tx=0x7f12e8001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:12.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12d7fff700 1 -- 172.21.15.175:0/3335083248 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8012630 con 0x7f12f00b2390 2023-10-10T19:26:12.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12d7fff700 1 -- 172.21.15.175:0/3335083248 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e8012790 con 0x7f12f00b2390 2023-10-10T19:26:12.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f0155d10 con 0x7f12f00b2390 2023-10-10T19:26:12.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12d7fff700 1 -- 172.21.15.175:0/3335083248 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8012b50 con 0x7f12f00b2390 2023-10-10T19:26:12.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.309+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f01562d0 con 0x7f12f00b2390 2023-10-10T19:26:12.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.310+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f0156920 con 0x7f12f00b2390 2023-10-10T19:26:12.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.311+0000 7f12d7fff700 1 -- 172.21.15.175:0/3335083248 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f12e801a3a0 con 0x7f12f00b2390 2023-10-10T19:26:12.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.311+0000 7f12d7fff700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12dc066f60 0x7f12dc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:12.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.311+0000 7f12d7fff700 1 -- 172.21.15.175:0/3335083248 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f12e8084e40 con 0x7f12f00b2390 2023-10-10T19:26:12.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.311+0000 7f12effff700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12dc066f60 0x7f12dc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:12.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.312+0000 7f12effff700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12dc066f60 0x7f12dc069410 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f12e00099b0 tx=0x7f12e0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:12.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.315+0000 7f12d7fff700 1 -- 172.21.15.175:0/3335083248 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12e80518d0 con 0x7f12f00b2390 2023-10-10T19:26:12.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.540+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12f0067570 con 0x7f12f00b2390 2023-10-10T19:26:12.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.540+0000 7f12d7fff700 1 -- 172.21.15.175:0/3335083248 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 301 v301) v1 ==== 95+0+5418 (secure 0 0 0) 0x7f12e8054ef0 con 0x7f12f00b2390 2023-10-10T19:26:12.541 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:26:12.541 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":301,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":301,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:26:08.830544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:26:12.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12dc066f60 msgr2=0x7f12dc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:12.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12dc066f60 0x7f12dc069410 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f12e00099b0 tx=0x7f12e0017000 comp rx=0 tx=0).stop 2023-10-10T19:26:12.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b2390 msgr2=0x7f12f00ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:12.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b2390 0x7f12f00ccd20 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f12e800c830 tx=0x7f12e8001690 comp rx=0 tx=0).stop 2023-10-10T19:26:12.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 shutdown_connections 2023-10-10T19:26:12.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f12dc066f60 0x7f12dc069410 secure :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f12e00099b0 tx=0x7f12e0017000 comp rx=0 tx=0).stop 2023-10-10T19:26:12.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12f00b2390 0x7f12f00ccd20 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:12.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12f00aff60 0x7f12f0156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:12.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 --2- 172.21.15.175:0/3335083248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12f00b1fc0 0x7f12f00ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:12.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.542+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 >> 172.21.15.175:0/3335083248 conn(0x7f12f0063230 msgr2=0x7f12f0150c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:12.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.543+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 shutdown_connections 2023-10-10T19:26:12.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:12.543+0000 7f12f66fc700 1 -- 172.21.15.175:0/3335083248 wait complete. 2023-10-10T19:26:12.546 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 301 2023-10-10T19:26:12.555 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-10T19:26:14.556 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:26:14.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.684+0000 7fe799831700 1 Processor -- start 2023-10-10T19:26:14.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.684+0000 7fe799831700 1 -- start start 2023-10-10T19:26:14.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe799831700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe7940b1fc0 0x7fe794155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:14.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe799831700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 0x7fe794068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:14.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe799831700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe794068bd0 0x7fe7940aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:14.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe799831700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe794150770 con 0x7fe794068bd0 2023-10-10T19:26:14.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe799831700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7941574b0 con 0x7fe7940b1fc0 2023-10-10T19:26:14.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe799831700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940afad0 con 0x7fe7940b2390 2023-10-10T19:26:14.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe7937fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe794068bd0 0x7fe7940aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:14.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe7937fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe794068bd0 0x7fe7940aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45960/0 (socket says 172.21.15.175:45960) 2023-10-10T19:26:14.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe7937fe700 1 -- 172.21.15.175:0/1646771508 learned_addr learned my addr 172.21.15.175:0/1646771508 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:14.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe792ffd700 1 --2- 172.21.15.175:0/1646771508 >> v2:172.21.15.191:3300/0 conn(0x7fe7940b1fc0 0x7fe794155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:14.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe78a7fc700 1 --2- 172.21.15.175:0/1646771508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 0x7fe794068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:14.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe7937fe700 1 -- 172.21.15.175:0/1646771508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 msgr2=0x7fe794068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:14.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe7937fe700 1 --2- 172.21.15.175:0/1646771508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 0x7fe794068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe7937fe700 1 -- 172.21.15.175:0/1646771508 >> v2:172.21.15.191:3300/0 conn(0x7fe7940b1fc0 msgr2=0x7fe794155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:26:14.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe7937fe700 1 --2- 172.21.15.175:0/1646771508 >> v2:172.21.15.191:3300/0 conn(0x7fe7940b1fc0 0x7fe794155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.685+0000 7fe7937fe700 1 -- 172.21.15.175:0/1646771508 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940af4c0 con 0x7fe794068bd0 2023-10-10T19:26:14.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe7937fe700 1 --2- 172.21.15.175:0/1646771508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe794068bd0 0x7fe7940aff60 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fe7940b28c0 tx=0x7fe78c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef01bcbefa2206a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:14.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe7927fc700 1 -- 172.21.15.175:0/1646771508 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c00e070 con 0x7fe794068bd0 2023-10-10T19:26:14.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe7927fc700 1 -- 172.21.15.175:0/1646771508 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c00ba00 con 0x7fe794068bd0 2023-10-10T19:26:14.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe7927fc700 1 -- 172.21.15.175:0/1646771508 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0092e0 con 0x7fe794068bd0 2023-10-10T19:26:14.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 -- 172.21.15.175:0/1646771508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe794068bd0 msgr2=0x7fe7940aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:14.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 --2- 172.21.15.175:0/1646771508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe794068bd0 0x7fe7940aff60 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fe7940b28c0 tx=0x7fe78c009840 comp rx=0 tx=0).stop 2023-10-10T19:26:14.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 -- 172.21.15.175:0/1646771508 shutdown_connections 2023-10-10T19:26:14.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 --2- 172.21.15.175:0/1646771508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe794068bd0 0x7fe7940aff60 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 --2- 172.21.15.175:0/1646771508 >> v2:172.21.15.191:3300/0 conn(0x7fe7940b1fc0 0x7fe794155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 --2- 172.21.15.175:0/1646771508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 0x7fe794068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 -- 172.21.15.175:0/1646771508 >> 172.21.15.175:0/1646771508 conn(0x7fe794063230 msgr2=0x7fe794063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:14.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 -- 172.21.15.175:0/1646771508 shutdown_connections 2023-10-10T19:26:14.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.686+0000 7fe799831700 1 -- 172.21.15.175:0/1646771508 wait complete. 2023-10-10T19:26:14.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.688+0000 7fe799831700 1 Processor -- start 2023-10-10T19:26:14.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.688+0000 7fe799831700 1 -- start start 2023-10-10T19:26:14.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.688+0000 7fe799831700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe794068bd0 0x7fe7940c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:14.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.688+0000 7fe799831700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7940b1fc0 0x7fe794168e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:14.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.688+0000 7fe799831700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 0x7fe79414f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:14.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.688+0000 7fe799831700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794154750 con 0x7fe7940b1fc0 2023-10-10T19:26:14.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.688+0000 7fe799831700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794154470 con 0x7fe794068bd0 2023-10-10T19:26:14.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.688+0000 7fe799831700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe794169800 con 0x7fe7940b2390 2023-10-10T19:26:14.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7940b1fc0 0x7fe794168e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:14.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7940b1fc0 0x7fe794168e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:45970/0 (socket says 172.21.15.175:45970) 2023-10-10T19:26:14.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 -- 172.21.15.175:0/1180817302 learned_addr learned my addr 172.21.15.175:0/1180817302 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:26:14.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe792ffd700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe794068bd0 0x7fe7940c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:14.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe7937fe700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 0x7fe79414f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:14.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 -- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 msgr2=0x7fe79414f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:14.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 0x7fe79414f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 -- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe794068bd0 msgr2=0x7fe7940c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:14.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe794068bd0 0x7fe7940c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 -- 172.21.15.175:0/1180817302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c009ae0 con 0x7fe7940b1fc0 2023-10-10T19:26:14.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe792ffd700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe794068bd0 0x7fe7940c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:26:14.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78a7fc700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7940b1fc0 0x7fe794168e10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00e930 tx=0x7fe77c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:14.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78b7fe700 1 -- 172.21.15.175:0/1180817302 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c00eed0 con 0x7fe7940b1fc0 2023-10-10T19:26:14.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78b7fe700 1 -- 172.21.15.175:0/1180817302 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77c00c8e0 con 0x7fe7940b1fc0 2023-10-10T19:26:14.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe78b7fe700 1 -- 172.21.15.175:0/1180817302 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c00cc80 con 0x7fe7940b1fc0 2023-10-10T19:26:14.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7940c5240 con 0x7fe7940b1fc0 2023-10-10T19:26:14.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.689+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7940c4120 con 0x7fe7940b1fc0 2023-10-10T19:26:14.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.690+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7940c5080 con 0x7fe7940b1fc0 2023-10-10T19:26:14.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.691+0000 7fe78b7fe700 1 -- 172.21.15.175:0/1180817302 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe77c013680 con 0x7fe7940b1fc0 2023-10-10T19:26:14.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.691+0000 7fe78b7fe700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe774066f60 0x7fe774069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:26:14.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.691+0000 7fe78b7fe700 1 -- 172.21.15.175:0/1180817302 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fe77c084fb0 con 0x7fe7940b1fc0 2023-10-10T19:26:14.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.691+0000 7fe792ffd700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe774066f60 0x7fe774069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:26:14.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.692+0000 7fe792ffd700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe774066f60 0x7fe774069410 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe784009d80 tx=0x7fe784008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:26:14.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.695+0000 7fe78b7fe700 1 -- 172.21.15.175:0/1180817302 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe77c051a40 con 0x7fe7940b1fc0 2023-10-10T19:26:14.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.920+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe794069060 con 0x7fe7940b1fc0 2023-10-10T19:26:14.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.921+0000 7fe78b7fe700 1 -- 172.21.15.175:0/1180817302 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 302 v302) v1 ==== 95+0+6176 (secure 0 0 0) 0x7fe77c055060 con 0x7fe7940b1fc0 2023-10-10T19:26:14.921 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:26:14.922 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":302,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302},{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":301,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:26:08.830544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:26:14.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe774066f60 msgr2=0x7fe774069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:14.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe774066f60 0x7fe774069410 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe784009d80 tx=0x7fe784008040 comp rx=0 tx=0).stop 2023-10-10T19:26:14.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7940b1fc0 msgr2=0x7fe794168e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:26:14.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7940b1fc0 0x7fe794168e10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00e930 tx=0x7fe77c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 shutdown_connections 2023-10-10T19:26:14.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe774066f60 0x7fe774069410 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7940b2390 0x7fe79414f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe794068bd0 0x7fe7940c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 --2- 172.21.15.175:0/1180817302 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7940b1fc0 0x7fe794168e10 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:26:14.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 >> 172.21.15.175:0/1180817302 conn(0x7fe794063230 msgr2=0x7fe7941575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:26:14.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 shutdown_connections 2023-10-10T19:26:14.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:26:14.923+0000 7fe799831700 1 -- 172.21.15.175:0/1180817302 wait complete. 2023-10-10T19:26:14.926 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 302 2023-10-10T19:26:14.935 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-10T19:26:14.936 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 49 secs before thrashing 2023-10-10T19:26:26.888 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:26:26.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:26:57.014 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:26:57.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:27:03.937 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:04.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.065+0000 7f84932ab700 1 Processor -- start 2023-10-10T19:27:04.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.065+0000 7f84932ab700 1 -- start start 2023-10-10T19:27:04.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f84932ab700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f848c0c6330 0x7f848c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f84932ab700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f848c0c7190 0x7f848c063c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f84932ab700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c0c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f84932ab700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848c0c5000 con 0x7f848c0c7190 2023-10-10T19:27:04.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f84932ab700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848c157030 con 0x7f848c0c6330 2023-10-10T19:27:04.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f84932ab700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848c156420 con 0x7f848c0c6700 2023-10-10T19:27:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f848c0c7190 0x7f848c063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f848c0c7190 0x7f848c063c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35880/0 (socket says 172.21.15.175:35880) 2023-10-10T19:27:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 -- 172.21.15.175:0/2610385723 learned_addr learned my addr 172.21.15.175:0/2610385723 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8491047700 1 --2- 172.21.15.175:0/2610385723 >> v2:172.21.15.191:3300/0 conn(0x7f848c0c6330 0x7f848c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8491848700 1 --2- 172.21.15.175:0/2610385723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c0c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 -- 172.21.15.175:0/2610385723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 msgr2=0x7f848c0c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 --2- 172.21.15.175:0/2610385723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c0c3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 -- 172.21.15.175:0/2610385723 >> v2:172.21.15.191:3300/0 conn(0x7f848c0c6330 msgr2=0x7f848c15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 --2- 172.21.15.175:0/2610385723 >> v2:172.21.15.191:3300/0 conn(0x7f848c0c6330 0x7f848c15a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 -- 172.21.15.175:0/2610385723 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c156700 con 0x7f848c0c7190 2023-10-10T19:27:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8491848700 1 --2- 172.21.15.175:0/2610385723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c0c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:27:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8490846700 1 --2- 172.21.15.175:0/2610385723 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c7190 0x7f848c063c20 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f848c0c01e0 tx=0x7f847c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b89a0b6a3228c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.066+0000 7f8483fff700 1 -- 172.21.15.175:0/2610385723 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c00b8f0 con 0x7f848c0c7190 2023-10-10T19:27:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f8483fff700 1 -- 172.21.15.175:0/2610385723 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c00ba50 con 0x7f848c0c7190 2023-10-10T19:27:04.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f8483fff700 1 -- 172.21.15.175:0/2610385723 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c00bdf0 con 0x7f848c0c7190 2023-10-10T19:27:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 -- 172.21.15.175:0/2610385723 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c7190 msgr2=0x7f848c063c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 --2- 172.21.15.175:0/2610385723 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c7190 0x7f848c063c20 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f848c0c01e0 tx=0x7f847c009250 comp rx=0 tx=0).stop 2023-10-10T19:27:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 -- 172.21.15.175:0/2610385723 shutdown_connections 2023-10-10T19:27:04.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 --2- 172.21.15.175:0/2610385723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c0c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 --2- 172.21.15.175:0/2610385723 >> v2:172.21.15.191:3300/0 conn(0x7f848c0c6330 0x7f848c15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 --2- 172.21.15.175:0/2610385723 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c7190 0x7f848c063c20 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 -- 172.21.15.175:0/2610385723 >> 172.21.15.175:0/2610385723 conn(0x7f848c06ef20 msgr2=0x7f848c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 -- 172.21.15.175:0/2610385723 shutdown_connections 2023-10-10T19:27:04.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.067+0000 7f84932ab700 1 -- 172.21.15.175:0/2610385723 wait complete. 2023-10-10T19:27:04.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.068+0000 7f84932ab700 1 Processor -- start 2023-10-10T19:27:04.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.068+0000 7f84932ab700 1 -- start start 2023-10-10T19:27:04.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f84932ab700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c6330 0x7f848c169450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f84932ab700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f84932ab700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848c0c7190 0x7f848c0cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f84932ab700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c152750 con 0x7f848c0c6330 2023-10-10T19:27:04.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f84932ab700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c159460 con 0x7f848c0c7190 2023-10-10T19:27:04.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f84932ab700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848c16a010 con 0x7f848c0c6700 2023-10-10T19:27:04.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491047700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c6330 0x7f848c169450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491047700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c6330 0x7f848c169450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35882/0 (socket says 172.21.15.175:35882) 2023-10-10T19:27:04.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491047700 1 -- 172.21.15.175:0/4276697585 learned_addr learned my addr 172.21.15.175:0/4276697585 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:04.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8490846700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491848700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848c0c7190 0x7f848c0cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491047700 1 -- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 msgr2=0x7f848c169990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491047700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c169990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491047700 1 -- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848c0c7190 msgr2=0x7f848c0cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491047700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848c0c7190 0x7f848c0cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.069+0000 7f8491047700 1 -- 172.21.15.175:0/4276697585 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c0094f0 con 0x7f848c0c6330 2023-10-10T19:27:04.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.070+0000 7f8491047700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c6330 0x7f848c169450 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f848400d890 tx=0x7f8484008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:04.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.070+0000 7f84817fa700 1 -- 172.21.15.175:0/4276697585 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84840125f0 con 0x7f848c0c6330 2023-10-10T19:27:04.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.070+0000 7f84817fa700 1 -- 172.21.15.175:0/4276697585 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8484012750 con 0x7f848c0c6330 2023-10-10T19:27:04.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.070+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848c1596c0 con 0x7f848c0c6330 2023-10-10T19:27:04.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.070+0000 7f84817fa700 1 -- 172.21.15.175:0/4276697585 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8484012af0 con 0x7f848c0c6330 2023-10-10T19:27:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.070+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848c152eb0 con 0x7f848c0c6330 2023-10-10T19:27:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.071+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848c067570 con 0x7f848c0c6330 2023-10-10T19:27:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.071+0000 7f84817fa700 1 -- 172.21.15.175:0/4276697585 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f848401e460 con 0x7f848c0c6330 2023-10-10T19:27:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.072+0000 7f84817fa700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8470066f30 0x7f84700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.072+0000 7f84817fa700 1 -- 172.21.15.175:0/4276697585 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f8484085280 con 0x7f848c0c6330 2023-10-10T19:27:04.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.072+0000 7f8490846700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8470066f30 0x7f84700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.072+0000 7f8490846700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8470066f30 0x7f84700693e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f848c0c01e0 tx=0x7f847c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:04.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.076+0000 7f84817fa700 1 -- 172.21.15.175:0/4276697585 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8484051d10 con 0x7f848c0c6330 2023-10-10T19:27:04.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.300+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f848c1569e0 con 0x7f848c0c6330 2023-10-10T19:27:04.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.300+0000 7f84817fa700 1 -- 172.21.15.175:0/4276697585 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 302 v302) v1 ==== 95+0+6176 (secure 0 0 0) 0x7f8484055330 con 0x7f848c0c6330 2023-10-10T19:27:04.301 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:04.301 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":302,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302},{"gid":7057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":301,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:26:08.830544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:04.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8470066f30 msgr2=0x7f84700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8470066f30 0x7f84700693e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f848c0c01e0 tx=0x7f847c017000 comp rx=0 tx=0).stop 2023-10-10T19:27:04.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c6330 msgr2=0x7f848c169450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c6330 0x7f848c169450 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f848400d890 tx=0x7f8484008040 comp rx=0 tx=0).stop 2023-10-10T19:27:04.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 shutdown_connections 2023-10-10T19:27:04.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8470066f30 0x7f84700693e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f848c0c7190 0x7f848c0cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f848c0c6330 0x7f848c169450 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 --2- 172.21.15.175:0/4276697585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f848c0c6700 0x7f848c169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 >> 172.21.15.175:0/4276697585 conn(0x7f848c06ef20 msgr2=0x7f848c15afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:04.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 shutdown_connections 2023-10-10T19:27:04.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.303+0000 7f84932ab700 1 -- 172.21.15.175:0/4276697585 wait complete. 2023-10-10T19:27:04.306 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 302 2023-10-10T19:27:04.315 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 4 -> 5 2023-10-10T19:27:04.315 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-10-10T19:27:04.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.432+0000 7f0488977700 1 Processor -- start 2023-10-10T19:27:04.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.432+0000 7f0488977700 1 -- start start 2023-10-10T19:27:04.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.433+0000 7f0488977700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f04840c67a0 0x7f0484063cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.433+0000 7f0488977700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048406c480 con 0x7f048406a540 2023-10-10T19:27:04.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.433+0000 7f0488977700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840c5360 con 0x7f04840c6b70 2023-10-10T19:27:04.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.433+0000 7f0488977700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04840c7bb0 con 0x7f04840c67a0 2023-10-10T19:27:04.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.433+0000 7f048259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f04840c67a0 0x7f0484063cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.433+0000 7f0482d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f048406a540 0x7f048406b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:47790/0 (socket says 172.21.15.175:47790) 2023-10-10T19:27:04.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.433+0000 7f0482d9d700 1 -- 172.21.15.175:0/2178221598 learned_addr learned my addr 172.21.15.175:0/2178221598 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:27:04.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f048259c700 1 -- 172.21.15.175:0/2178221598 >> v1:172.21.15.191:6789/0 conn(0x7f04840c6b70 legacy=0x7f04840c1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f048259c700 1 -- 172.21.15.175:0/2178221598 >> v1:172.21.15.175:6789/0 conn(0x7f048406a540 legacy=0x7f048406b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f048259c700 1 -- 172.21.15.175:0/2178221598 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0484068be0 con 0x7f04840c67a0 2023-10-10T19:27:04.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f048159a700 1 -- 172.21.15.175:0/2178221598 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930202270 0 0) 0x7f048406c480 con 0x7f048406a540 2023-10-10T19:27:04.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f048259c700 1 --2- 172.21.15.175:0/2178221598 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c67a0 0x7f0484063cf0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f0474009e60 tx=0x7f0474009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdde1c1e0ac49aa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:04.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f048159a700 1 -- 172.21.15.175:0/2178221598 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047400e070 con 0x7f04840c67a0 2023-10-10T19:27:04.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f048159a700 1 -- 172.21.15.175:0/2178221598 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047400bc30 con 0x7f04840c67a0 2023-10-10T19:27:04.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f048159a700 1 -- 172.21.15.175:0/2178221598 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474012430 con 0x7f04840c67a0 2023-10-10T19:27:04.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f0488977700 1 -- 172.21.15.175:0/2178221598 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c67a0 msgr2=0x7f0484063cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f0488977700 1 --2- 172.21.15.175:0/2178221598 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c67a0 0x7f0484063cf0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f0474009e60 tx=0x7f0474009780 comp rx=0 tx=0).stop 2023-10-10T19:27:04.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f0488977700 1 -- 172.21.15.175:0/2178221598 shutdown_connections 2023-10-10T19:27:04.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f0488977700 1 --2- 172.21.15.175:0/2178221598 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c67a0 0x7f0484063cf0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.434+0000 7f0488977700 1 -- 172.21.15.175:0/2178221598 >> 172.21.15.175:0/2178221598 conn(0x7f048406dcc0 msgr2=0x7f0484063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:04.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.435+0000 7f0488977700 1 -- 172.21.15.175:0/2178221598 shutdown_connections 2023-10-10T19:27:04.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.435+0000 7f0488977700 1 -- 172.21.15.175:0/2178221598 wait complete. 2023-10-10T19:27:04.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.435+0000 7f0488977700 1 Processor -- start 2023-10-10T19:27:04.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0488977700 1 -- start start 2023-10-10T19:27:04.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0488977700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f048406a540 0x7f0484159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0488977700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04840c67a0 0x7f0484155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0488977700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c6b70 0x7f0484155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0488977700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04841668d0 con 0x7f04840c6b70 2023-10-10T19:27:04.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0488977700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0484155290 con 0x7f048406a540 2023-10-10T19:27:04.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0488977700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0484166370 con 0x7f04840c67a0 2023-10-10T19:27:04.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0482d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c6b70 0x7f0484155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0482d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c6b70 0x7f0484155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35894/0 (socket says 172.21.15.175:35894) 2023-10-10T19:27:04.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0482d9d700 1 -- 172.21.15.175:0/1673403666 learned_addr learned my addr 172.21.15.175:0/1673403666 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:04.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f048259c700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f048406a540 0x7f0484159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.436+0000 7f0481d9b700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04840c67a0 0x7f0484155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f0482d9d700 1 -- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04840c67a0 msgr2=0x7f0484155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f0482d9d700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04840c67a0 0x7f0484155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f0482d9d700 1 -- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f048406a540 msgr2=0x7f0484159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f0482d9d700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f048406a540 0x7f0484159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f0482d9d700 1 -- 172.21.15.175:0/1673403666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474009a20 con 0x7f04840c6b70 2023-10-10T19:27:04.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f0482d9d700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c6b70 0x7f0484155c60 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f047c00c520 tx=0x7f047c007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:04.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f047a7fc700 1 -- 172.21.15.175:0/1673403666 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c005b30 con 0x7f04840c6b70 2023-10-10T19:27:04.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f047a7fc700 1 -- 172.21.15.175:0/1673403666 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c005cb0 con 0x7f04840c6b70 2023-10-10T19:27:04.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0484156ac0 con 0x7f04840c6b70 2023-10-10T19:27:04.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f047a7fc700 1 -- 172.21.15.175:0/1673403666 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c007320 con 0x7f04840c6b70 2023-10-10T19:27:04.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.437+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0484154f20 con 0x7f04840c6b70 2023-10-10T19:27:04.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.438+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0484154d80 con 0x7f04840c6b70 2023-10-10T19:27:04.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.438+0000 7f047a7fc700 1 -- 172.21.15.175:0/1673403666 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f047c01b3b0 con 0x7f04840c6b70 2023-10-10T19:27:04.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.438+0000 7f047a7fc700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0464066fe0 0x7f0464069490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:04.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.439+0000 7f047a7fc700 1 -- 172.21.15.175:0/1673403666 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f047c084200 con 0x7f04840c6b70 2023-10-10T19:27:04.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.439+0000 7f048259c700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0464066fe0 0x7f0464069490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:04.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.439+0000 7f048259c700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0464066fe0 0x7f0464069490 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0474009e60 tx=0x7f0474019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:04.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.443+0000 7f047a7fc700 1 -- 172.21.15.175:0/1673403666 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0484154d80 con 0x7f04840c6b70 2023-10-10T19:27:04.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.668+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f0484154d80 con 0x7f04840c6b70 2023-10-10T19:27:04.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.993+0000 7f047a7fc700 1 -- 172.21.15.175:0/1673403666 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v303) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0484154d80 con 0x7f04840c6b70 2023-10-10T19:27:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0464066fe0 msgr2=0x7f0464069490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0464066fe0 0x7f0464069490 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0474009e60 tx=0x7f0474019000 comp rx=0 tx=0).stop 2023-10-10T19:27:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c6b70 msgr2=0x7f0484155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c6b70 0x7f0484155c60 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f047c00c520 tx=0x7f047c007f50 comp rx=0 tx=0).stop 2023-10-10T19:27:04.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 shutdown_connections 2023-10-10T19:27:04.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0464066fe0 0x7f0464069490 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04840c6b70 0x7f0484155c60 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f048406a540 0x7f0484159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 --2- 172.21.15.175:0/1673403666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04840c67a0 0x7f0484155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:04.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 >> 172.21.15.175:0/1673403666 conn(0x7f048406dcc0 msgr2=0x7f048415bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:04.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 shutdown_connections 2023-10-10T19:27:04.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:04.995+0000 7f0488977700 1 -- 172.21.15.175:0/1673403666 wait complete. 2023-10-10T19:27:05.008 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:27:05.008 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:05.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.122+0000 7fd17bfff700 1 Processor -- start 2023-10-10T19:27:05.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.122+0000 7fd17bfff700 1 -- start start 2023-10-10T19:27:05.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd17c14fd90 0x7fd17c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:05.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17bfff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd17c0cd040 0x7fd17c0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:05.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17bfff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c067fa0 con 0x7fd17c08a100 2023-10-10T19:27:05.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17bfff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c0c5600 con 0x7fd17c14fd90 2023-10-10T19:27:05.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17bfff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c156b70 con 0x7fd17c0cd040 2023-10-10T19:27:05.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd17c14fd90 0x7fd17c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:05.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17a7fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd17c08a100 0x7fd17c0c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56880/0 (socket says 172.21.15.175:56880) 2023-10-10T19:27:05.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17a7fc700 1 -- 172.21.15.175:0/882264675 learned_addr learned my addr 172.21.15.175:0/882264675 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:27:05.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd17c14fd90 0x7fd17c159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55642/0 (socket says 172.21.15.175:55642) 2023-10-10T19:27:05.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17b7fe700 1 --2- 172.21.15.175:0/882264675 >> v2:172.21.15.191:3300/0 conn(0x7fd17c0cd040 0x7fd17c0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:05.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17affd700 1 -- 172.21.15.175:0/882264675 >> v2:172.21.15.191:3300/0 conn(0x7fd17c0cd040 msgr2=0x7fd17c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:05.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17affd700 1 --2- 172.21.15.175:0/882264675 >> v2:172.21.15.191:3300/0 conn(0x7fd17c0cd040 0x7fd17c0684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17affd700 1 -- 172.21.15.175:0/882264675 >> v1:172.21.15.175:6789/0 conn(0x7fd17c08a100 legacy=0x7fd17c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd17affd700 1 -- 172.21.15.175:0/882264675 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c150160 con 0x7fd17c14fd90 2023-10-10T19:27:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.123+0000 7fd179ffb700 1 -- 172.21.15.175:0/882264675 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254529723 0 0) 0x7fd17c067fa0 con 0x7fd17c08a100 2023-10-10T19:27:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17affd700 1 --2- 172.21.15.175:0/882264675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c14fd90 0x7fd17c159c30 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fd17c08a6e0 tx=0x7fd16c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1392927ba988304 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd179ffb700 1 -- 172.21.15.175:0/882264675 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00e070 con 0x7fd17c14fd90 2023-10-10T19:27:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd179ffb700 1 -- 172.21.15.175:0/882264675 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c00bcc0 con 0x7fd17c14fd90 2023-10-10T19:27:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd179ffb700 1 -- 172.21.15.175:0/882264675 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c012680 con 0x7fd17c14fd90 2023-10-10T19:27:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17bfff700 1 -- 172.21.15.175:0/882264675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c14fd90 msgr2=0x7fd17c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17bfff700 1 --2- 172.21.15.175:0/882264675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c14fd90 0x7fd17c159c30 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fd17c08a6e0 tx=0x7fd16c009780 comp rx=0 tx=0).stop 2023-10-10T19:27:05.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17bfff700 1 -- 172.21.15.175:0/882264675 shutdown_connections 2023-10-10T19:27:05.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17bfff700 1 --2- 172.21.15.175:0/882264675 >> v2:172.21.15.191:3300/0 conn(0x7fd17c0cd040 0x7fd17c0684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17bfff700 1 --2- 172.21.15.175:0/882264675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c14fd90 0x7fd17c159c30 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17bfff700 1 -- 172.21.15.175:0/882264675 >> 172.21.15.175:0/882264675 conn(0x7fd17c063d00 msgr2=0x7fd17c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:05.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17bfff700 1 -- 172.21.15.175:0/882264675 shutdown_connections 2023-10-10T19:27:05.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.124+0000 7fd17bfff700 1 -- 172.21.15.175:0/882264675 wait complete. 2023-10-10T19:27:05.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.125+0000 7fd17bfff700 1 Processor -- start 2023-10-10T19:27:05.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17bfff700 1 -- start start 2023-10-10T19:27:05.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c08a100 0x7fd17c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:05.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0cd040 0x7fd17c14dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:05.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17bfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c14fd90 0x7fd17c169530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:05.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17bfff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0c9800 con 0x7fd17c08a100 2023-10-10T19:27:05.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17bfff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c166e70 con 0x7fd17c14fd90 2023-10-10T19:27:05.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17bfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c14f300 con 0x7fd17c0cd040 2023-10-10T19:27:05.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c08a100 0x7fd17c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:05.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c08a100 0x7fd17c14d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55652/0 (socket says 172.21.15.175:55652) 2023-10-10T19:27:05.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17affd700 1 -- 172.21.15.175:0/1656118314 learned_addr learned my addr 172.21.15.175:0/1656118314 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:05.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17a7fc700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0cd040 0x7fd17c14dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:05.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.126+0000 7fd17b7fe700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c14fd90 0x7fd17c169530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:05.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd17affd700 1 -- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0cd040 msgr2=0x7fd17c14dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:05.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd17affd700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0cd040 0x7fd17c14dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd17affd700 1 -- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c14fd90 msgr2=0x7fd17c169530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:05.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd17affd700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c14fd90 0x7fd17c169530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd17affd700 1 -- 172.21.15.175:0/1656118314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c009a20 con 0x7fd17c08a100 2023-10-10T19:27:05.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd17affd700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c08a100 0x7fd17c14d860 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0097c0 tx=0x7fd16c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:05.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd172ffd700 1 -- 172.21.15.175:0/1656118314 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c012e20 con 0x7fd17c08a100 2023-10-10T19:27:05.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd172ffd700 1 -- 172.21.15.175:0/1656118314 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c027470 con 0x7fd17c08a100 2023-10-10T19:27:05.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c14e6f0 con 0x7fd17c08a100 2023-10-10T19:27:05.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd172ffd700 1 -- 172.21.15.175:0/1656118314 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c027830 con 0x7fd17c08a100 2023-10-10T19:27:05.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.127+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c14ecb0 con 0x7fd17c08a100 2023-10-10T19:27:05.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.128+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c14f5e0 con 0x7fd17c08a100 2023-10-10T19:27:05.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.128+0000 7fd172ffd700 1 -- 172.21.15.175:0/1656118314 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd16c024070 con 0x7fd17c08a100 2023-10-10T19:27:05.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.129+0000 7fd172ffd700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f30 0x7fd15c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:05.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.129+0000 7fd172ffd700 1 -- 172.21.15.175:0/1656118314 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fd16c0877a0 con 0x7fd17c08a100 2023-10-10T19:27:05.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.129+0000 7fd17a7fc700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f30 0x7fd15c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:05.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.129+0000 7fd17a7fc700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f30 0x7fd15c0693e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0c3510 tx=0x7fd164007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:05.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.133+0000 7fd172ffd700 1 -- 172.21.15.175:0/1656118314 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd16c054230 con 0x7fd17c08a100 2023-10-10T19:27:05.357 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.357+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd17c067570 con 0x7fd17c08a100 2023-10-10T19:27:05.358 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.358+0000 7fd172ffd700 1 -- 172.21.15.175:0/1656118314 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 304 v304) v1 ==== 95+0+6191 (secure 0 0 0) 0x7fd16c057850 con 0x7fd17c08a100 2023-10-10T19:27:05.358 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:05.359 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":304,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":304,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:04.993581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:starting","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:05.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.359+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f30 msgr2=0x7fd15c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:05.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.359+0000 7fd17bfff700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f30 0x7fd15c0693e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0c3510 tx=0x7fd164007d30 comp rx=0 tx=0).stop 2023-10-10T19:27:05.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.359+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c08a100 msgr2=0x7fd17c14d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:05.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.359+0000 7fd17bfff700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c08a100 0x7fd17c14d860 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0097c0 tx=0x7fd16c019000 comp rx=0 tx=0).stop 2023-10-10T19:27:05.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.359+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 shutdown_connections 2023-10-10T19:27:05.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.359+0000 7fd17bfff700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd15c066f30 0x7fd15c0693e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.360+0000 7fd17bfff700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd17c14fd90 0x7fd17c169530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.360+0000 7fd17bfff700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd17c08a100 0x7fd17c14d860 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.360+0000 7fd17bfff700 1 --2- 172.21.15.175:0/1656118314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd17c0cd040 0x7fd17c14dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:05.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.360+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 >> 172.21.15.175:0/1656118314 conn(0x7fd17c063d00 msgr2=0x7fd17c0cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:05.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.360+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 shutdown_connections 2023-10-10T19:27:05.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:05.360+0000 7fd17bfff700 1 -- 172.21.15.175:0/1656118314 wait complete. 2023-10-10T19:27:05.363 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 304 2023-10-10T19:27:05.372 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 304, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 304, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_7057": { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 304, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:starting", "state_seq": 1 }, "gid_7401": { "addr": "172.21.15.175:6834/3457422611", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3457422611, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3457422611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7401, "incarnation": 244, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 117 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 130, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:27:04.993581+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 7401, "mds_2": 8049, "mds_3": 7771, "mds_4": 7057 } } } ], "standbys": [ { "addr": "172.21.15.191:6837/840318079", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 840318079, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 840318079, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 302, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6893, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-10T19:27:07.373 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:07.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.498+0000 7f1c2a2bf700 1 Processor -- start 2023-10-10T19:27:07.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.498+0000 7f1c2a2bf700 1 -- start start 2023-10-10T19:27:07.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.498+0000 7f1c2a2bf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1c2406bf00 0x7f1c240c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:07.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.498+0000 7f1c2a2bf700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240c6df0 con 0x7f1c240c5e40 2023-10-10T19:27:07.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.498+0000 7f1c2a2bf700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240c6920 con 0x7f1c24156520 2023-10-10T19:27:07.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.498+0000 7f1c2a2bf700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c241560d0 con 0x7f1c2406bf00 2023-10-10T19:27:07.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c237fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1c2406bf00 0x7f1c240c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:07.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c237fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1c2406bf00 0x7f1c240c47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55660/0 (socket says 172.21.15.175:55660) 2023-10-10T19:27:07.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c23fff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1c240c5e40 0x7f1c240c6210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56882/0 (socket says 172.21.15.175:56882) 2023-10-10T19:27:07.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c237fe700 1 -- 172.21.15.175:0/836366849 learned_addr learned my addr 172.21.15.175:0/836366849 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:07.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c237fe700 1 -- 172.21.15.175:0/836366849 >> v1:172.21.15.191:6789/0 conn(0x7f1c24156520 legacy=0x7f1c241569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:07.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c237fe700 1 -- 172.21.15.175:0/836366849 >> v1:172.21.15.175:6789/0 conn(0x7f1c240c5e40 legacy=0x7f1c240c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:07.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c237fe700 1 -- 172.21.15.175:0/836366849 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c241597e0 con 0x7f1c2406bf00 2023-10-10T19:27:07.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c22ffd700 1 -- 172.21.15.175:0/836366849 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057806081 0 0) 0x7f1c240c6df0 con 0x7f1c240c5e40 2023-10-10T19:27:07.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c237fe700 1 --2- 172.21.15.175:0/836366849 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c47d0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f1c180099b0 tx=0x7f1c18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138c03a0cc54c53e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:07.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c22ffd700 1 -- 172.21.15.175:0/836366849 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1800b8b0 con 0x7f1c2406bf00 2023-10-10T19:27:07.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c22ffd700 1 -- 172.21.15.175:0/836366849 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1800ba10 con 0x7f1c2406bf00 2023-10-10T19:27:07.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.499+0000 7f1c22ffd700 1 -- 172.21.15.175:0/836366849 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1800bdb0 con 0x7f1c2406bf00 2023-10-10T19:27:07.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.500+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/836366849 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 msgr2=0x7f1c240c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:07.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.500+0000 7f1c2a2bf700 1 --2- 172.21.15.175:0/836366849 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c47d0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f1c180099b0 tx=0x7f1c18009250 comp rx=0 tx=0).stop 2023-10-10T19:27:07.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.500+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/836366849 shutdown_connections 2023-10-10T19:27:07.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.500+0000 7f1c2a2bf700 1 --2- 172.21.15.175:0/836366849 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c47d0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:07.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.500+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/836366849 >> 172.21.15.175:0/836366849 conn(0x7f1c24068900 msgr2=0x7f1c24068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:07.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.500+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/836366849 shutdown_connections 2023-10-10T19:27:07.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.500+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/836366849 wait complete. 2023-10-10T19:27:07.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.501+0000 7f1c2a2bf700 1 Processor -- start 2023-10-10T19:27:07.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.501+0000 7f1c2a2bf700 1 -- start start 2023-10-10T19:27:07.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c2a2bf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:07.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c2a2bf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c240c5e40 0x7f1c2415be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:07.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c2a2bf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c24156520 0x7f1c2413d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:07.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c2a2bf700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2413bb30 con 0x7f1c2406bf00 2023-10-10T19:27:07.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c2a2bf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240c2510 con 0x7f1c240c5e40 2023-10-10T19:27:07.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c2a2bf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240c29e0 con 0x7f1c24156520 2023-10-10T19:27:07.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:07.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55668/0 (socket says 172.21.15.175:55668) 2023-10-10T19:27:07.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 -- 172.21.15.175:0/3460544937 learned_addr learned my addr 172.21.15.175:0/3460544937 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:07.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c237fe700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c240c5e40 0x7f1c2415be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:07.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 -- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c24156520 msgr2=0x7f1c2413d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:07.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c24156520 0x7f1c2413d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:07.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 -- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c240c5e40 msgr2=0x7f1c2415be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:07.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c240c5e40 0x7f1c2415be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:07.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 -- 172.21.15.175:0/3460544937 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c180094f0 con 0x7f1c2406bf00 2023-10-10T19:27:07.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c23fff700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c7ee0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400c7c0 tx=0x7f1c14007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:07.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c0bfff700 1 -- 172.21.15.175:0/3460544937 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14001ea0 con 0x7f1c2406bf00 2023-10-10T19:27:07.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c0bfff700 1 -- 172.21.15.175:0/3460544937 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c14007300 con 0x7f1c2406bf00 2023-10-10T19:27:07.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2413d0b0 con 0x7f1c2406bf00 2023-10-10T19:27:07.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c0bfff700 1 -- 172.21.15.175:0/3460544937 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c140076a0 con 0x7f1c2406bf00 2023-10-10T19:27:07.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.502+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c240c2700 con 0x7f1c2406bf00 2023-10-10T19:27:07.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.503+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c24067570 con 0x7f1c2406bf00 2023-10-10T19:27:07.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.504+0000 7f1c0bfff700 1 -- 172.21.15.175:0/3460544937 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1c14005a00 con 0x7f1c2406bf00 2023-10-10T19:27:07.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.504+0000 7f1c0bfff700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c10066f30 0x7f1c100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:07.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.504+0000 7f1c0bfff700 1 -- 172.21.15.175:0/3460544937 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f1c14084200 con 0x7f1c2406bf00 2023-10-10T19:27:07.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.504+0000 7f1c237fe700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c10066f30 0x7f1c100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:07.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.505+0000 7f1c237fe700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c10066f30 0x7f1c100693e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1c240c4d10 tx=0x7f1c18016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:07.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.509+0000 7f1c0bfff700 1 -- 172.21.15.175:0/3460544937 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c14050d10 con 0x7f1c2406bf00 2023-10-10T19:27:07.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.733+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c240c56f0 con 0x7f1c2406bf00 2023-10-10T19:27:07.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.733+0000 7f1c0bfff700 1 -- 172.21.15.175:0/3460544937 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 304 v304) v1 ==== 95+0+6191 (secure 0 0 0) 0x7f1c14018020 con 0x7f1c2406bf00 2023-10-10T19:27:07.734 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:07.735 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":304,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":304,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:04.993581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:starting","state_seq":1,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:07.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.735+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c10066f30 msgr2=0x7f1c100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:07.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.735+0000 7f1c2a2bf700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c10066f30 0x7f1c100693e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1c240c4d10 tx=0x7f1c18016000 comp rx=0 tx=0).stop 2023-10-10T19:27:07.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.735+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 msgr2=0x7f1c240c7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:07.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.735+0000 7f1c2a2bf700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c7ee0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400c7c0 tx=0x7f1c14007d20 comp rx=0 tx=0).stop 2023-10-10T19:27:07.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.735+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 shutdown_connections 2023-10-10T19:27:07.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.735+0000 7f1c2a2bf700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1c10066f30 0x7f1c100693e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:07.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.736+0000 7f1c2a2bf700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c24156520 0x7f1c2413d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:07.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.736+0000 7f1c2a2bf700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c2406bf00 0x7f1c240c7ee0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:07.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.736+0000 7f1c2a2bf700 1 --2- 172.21.15.175:0/3460544937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c240c5e40 0x7f1c2415be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:07.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.736+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 >> 172.21.15.175:0/3460544937 conn(0x7f1c24068900 msgr2=0x7f1c241371f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:07.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.736+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 shutdown_connections 2023-10-10T19:27:07.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:07.736+0000 7f1c2a2bf700 1 -- 172.21.15.175:0/3460544937 wait complete. 2023-10-10T19:27:07.739 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 304 2023-10-10T19:27:07.748 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:09.749 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:09.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.867+0000 7fe3ab0cc700 1 Processor -- start 2023-10-10T19:27:09.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.867+0000 7fe3ab0cc700 1 -- start start 2023-10-10T19:27:09.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3ab0cc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3a40b2390 0x7fe3a4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:09.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3ab0cc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a41574b0 con 0x7fe3a40b1fc0 2023-10-10T19:27:09.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3ab0cc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40afad0 con 0x7fe3a40aff60 2023-10-10T19:27:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3ab0cc700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a4150770 con 0x7fe3a40b2390 2023-10-10T19:27:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3a3fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3a40b2390 0x7fe3a4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3a8e68700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe3a40b1fc0 0x7fe3a4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:56894/0 (socket says 172.21.15.175:56894) 2023-10-10T19:27:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3a8e68700 1 -- 172.21.15.175:0/280216910 learned_addr learned my addr 172.21.15.175:0/280216910 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:27:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3a3fff700 1 -- 172.21.15.175:0/280216910 >> v1:172.21.15.191:6789/0 conn(0x7fe3a40aff60 legacy=0x7fe3a4159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:09.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.868+0000 7fe3a3fff700 1 -- 172.21.15.175:0/280216910 >> v1:172.21.15.175:6789/0 conn(0x7fe3a40b1fc0 legacy=0x7fe3a4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:09.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3a3fff700 1 -- 172.21.15.175:0/280216910 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40af4c0 con 0x7fe3a40b2390 2023-10-10T19:27:09.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3a37fe700 1 -- 172.21.15.175:0/280216910 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567354694 0 0) 0x7fe3a41574b0 con 0x7fe3a40b1fc0 2023-10-10T19:27:09.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3a3fff700 1 --2- 172.21.15.175:0/280216910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a4155fb0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fe398009e60 tx=0x7fe398009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7044aeab9cda31 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:09.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3a37fe700 1 -- 172.21.15.175:0/280216910 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39800e070 con 0x7fe3a40b2390 2023-10-10T19:27:09.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3a37fe700 1 -- 172.21.15.175:0/280216910 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39800bcc0 con 0x7fe3a40b2390 2023-10-10T19:27:09.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3a37fe700 1 -- 172.21.15.175:0/280216910 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398012680 con 0x7fe3a40b2390 2023-10-10T19:27:09.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/280216910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 msgr2=0x7fe3a4155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:09.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3ab0cc700 1 --2- 172.21.15.175:0/280216910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a4155fb0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fe398009e60 tx=0x7fe398009780 comp rx=0 tx=0).stop 2023-10-10T19:27:09.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/280216910 shutdown_connections 2023-10-10T19:27:09.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3ab0cc700 1 --2- 172.21.15.175:0/280216910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a4155fb0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:09.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/280216910 >> 172.21.15.175:0/280216910 conn(0x7fe3a4063230 msgr2=0x7fe3a4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:09.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/280216910 shutdown_connections 2023-10-10T19:27:09.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.869+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/280216910 wait complete. 2023-10-10T19:27:09.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.870+0000 7fe3ab0cc700 1 Processor -- start 2023-10-10T19:27:09.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.871+0000 7fe3ab0cc700 1 -- start start 2023-10-10T19:27:09.877 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.871+0000 7fe3ab0cc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3a40aff60 0x7fe3a4165e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:09.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.871+0000 7fe3ab0cc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3a40b1fc0 0x7fe3a414de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:09.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.871+0000 7fe3ab0cc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a40caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:09.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.871+0000 7fe3ab0cc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40cd340 con 0x7fe3a40b2390 2023-10-10T19:27:09.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.871+0000 7fe3ab0cc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4169bb0 con 0x7fe3a40b1fc0 2023-10-10T19:27:09.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.871+0000 7fe3ab0cc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415bef0 con 0x7fe3a40aff60 2023-10-10T19:27:09.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a40caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:09.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a40caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55694/0 (socket says 172.21.15.175:55694) 2023-10-10T19:27:09.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 -- 172.21.15.175:0/2643481066 learned_addr learned my addr 172.21.15.175:0/2643481066 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:09.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a3fff700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3a40b1fc0 0x7fe3a414de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:09.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a8e68700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3a40aff60 0x7fe3a4165e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:09.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 -- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3a40aff60 msgr2=0x7fe3a4165e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:09.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3a40aff60 0x7fe3a4165e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:09.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 -- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3a40b1fc0 msgr2=0x7fe3a414de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:09.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3a40b1fc0 0x7fe3a414de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:09.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 -- 172.21.15.175:0/2643481066 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe398009a20 con 0x7fe3a40b2390 2023-10-10T19:27:09.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a9669700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a40caaa0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00c590 tx=0x7fe39c007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:09.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a0ff9700 1 -- 172.21.15.175:0/2643481066 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c011590 con 0x7fe3a40b2390 2023-10-10T19:27:09.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a0ff9700 1 -- 172.21.15.175:0/2643481066 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c0116f0 con 0x7fe3a40b2390 2023-10-10T19:27:09.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a4166390 con 0x7fe3a40b2390 2023-10-10T19:27:09.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3a0ff9700 1 -- 172.21.15.175:0/2643481066 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c011ab0 con 0x7fe3a40b2390 2023-10-10T19:27:09.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.872+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a415bc10 con 0x7fe3a40b2390 2023-10-10T19:27:09.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.873+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a4067570 con 0x7fe3a40b2390 2023-10-10T19:27:09.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.873+0000 7fe3a0ff9700 1 -- 172.21.15.175:0/2643481066 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe39c01b3b0 con 0x7fe3a40b2390 2023-10-10T19:27:09.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.874+0000 7fe3a0ff9700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe390066f30 0x7fe3900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:09.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.874+0000 7fe3a0ff9700 1 -- 172.21.15.175:0/2643481066 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fe39c084080 con 0x7fe3a40b2390 2023-10-10T19:27:09.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.874+0000 7fe3a8e68700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe390066f30 0x7fe3900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:09.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.874+0000 7fe3a8e68700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe390066f30 0x7fe3900693e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe394007fd0 tx=0x7fe394007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:09.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:09.878+0000 7fe3a0ff9700 1 -- 172.21.15.175:0/2643481066 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe39c050b10 con 0x7fe3a40b2390 2023-10-10T19:27:10.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.107+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3a4169cf0 con 0x7fe3a40b2390 2023-10-10T19:27:10.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.107+0000 7fe3a0ff9700 1 -- 172.21.15.175:0/2643481066 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7fe39c017020 con 0x7fe3a40b2390 2023-10-10T19:27:10.108 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:10.108 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:10.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.109+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe390066f30 msgr2=0x7fe3900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:10.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.109+0000 7fe3ab0cc700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe390066f30 0x7fe3900693e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe394007fd0 tx=0x7fe394007d30 comp rx=0 tx=0).stop 2023-10-10T19:27:10.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 msgr2=0x7fe3a40caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:10.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a40caaa0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00c590 tx=0x7fe39c007ef0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 shutdown_connections 2023-10-10T19:27:10.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe390066f30 0x7fe3900693e0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3a40b2390 0x7fe3a40caaa0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3a40aff60 0x7fe3a4165e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 --2- 172.21.15.175:0/2643481066 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3a40b1fc0 0x7fe3a414de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 >> 172.21.15.175:0/2643481066 conn(0x7fe3a4063230 msgr2=0x7fe3a414fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:10.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 shutdown_connections 2023-10-10T19:27:10.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.110+0000 7fe3ab0cc700 1 -- 172.21.15.175:0/2643481066 wait complete. 2023-10-10T19:27:10.113 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:10.122 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 5 alive and active, now stable! 2023-10-10T19:27:10.122 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-10T19:27:10.123 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-10T19:27:10.123 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:27:10.134 INFO:tasks.ceph.mds.a:Stopped 2023-10-10T19:27:10.134 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:10.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.288+0000 7fd7bbfff700 1 Processor -- start 2023-10-10T19:27:10.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.288+0000 7fd7bbfff700 1 -- start start 2023-10-10T19:27:10.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bbfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc14fd90 0x7fd7bc159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:10.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bbfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd7bc0cd040 0x7fd7bc0684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:10.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bbfff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc067fa0 con 0x7fd7bc08a100 2023-10-10T19:27:10.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bbfff700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7bc0c5600 con 0x7fd7bc0cd040 2023-10-10T19:27:10.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bbfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc156b70 con 0x7fd7bc14fd90 2023-10-10T19:27:10.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bb7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd7bc0cd040 0x7fd7bc0684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:10.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bb7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd7bc0cd040 0x7fd7bc0684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55696/0 (socket says 172.21.15.175:55696) 2023-10-10T19:27:10.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bb7fe700 1 -- 172.21.15.175:0/1505074239 learned_addr learned my addr 172.21.15.175:0/1505074239 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:10.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7baffd700 1 --2- 172.21.15.175:0/1505074239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc14fd90 0x7fd7bc159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:10.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bb7fe700 1 -- 172.21.15.175:0/1505074239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc14fd90 msgr2=0x7fd7bc159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:10.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bb7fe700 1 --2- 172.21.15.175:0/1505074239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc14fd90 0x7fd7bc159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bb7fe700 1 -- 172.21.15.175:0/1505074239 >> v1:172.21.15.191:6789/0 conn(0x7fd7bc08a100 legacy=0x7fd7bc0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:10.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7bb7fe700 1 -- 172.21.15.175:0/1505074239 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc150160 con 0x7fd7bc0cd040 2023-10-10T19:27:10.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.289+0000 7fd7baffd700 1 --2- 172.21.15.175:0/1505074239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc14fd90 0x7fd7bc159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:27:10.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bb7fe700 1 --2- 172.21.15.175:0/1505074239 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc0cd040 0x7fd7bc0684f0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009f20 tx=0x7fd7b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fad29834474ef8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:10.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7b9ffb700 1 -- 172.21.15.175:0/1505074239 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400e070 con 0x7fd7bc0cd040 2023-10-10T19:27:10.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7b9ffb700 1 -- 172.21.15.175:0/1505074239 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b400ba80 con 0x7fd7bc0cd040 2023-10-10T19:27:10.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7b9ffb700 1 -- 172.21.15.175:0/1505074239 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b40092e0 con 0x7fd7bc0cd040 2023-10-10T19:27:10.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bbfff700 1 -- 172.21.15.175:0/1505074239 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc0cd040 msgr2=0x7fd7bc0684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:10.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/1505074239 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc0cd040 0x7fd7bc0684f0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009f20 tx=0x7fd7b4009840 comp rx=0 tx=0).stop 2023-10-10T19:27:10.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bbfff700 1 -- 172.21.15.175:0/1505074239 shutdown_connections 2023-10-10T19:27:10.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/1505074239 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc0cd040 0x7fd7bc0684f0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/1505074239 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc14fd90 0x7fd7bc159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bbfff700 1 -- 172.21.15.175:0/1505074239 >> 172.21.15.175:0/1505074239 conn(0x7fd7bc063d00 msgr2=0x7fd7bc0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:10.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bbfff700 1 -- 172.21.15.175:0/1505074239 shutdown_connections 2023-10-10T19:27:10.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.290+0000 7fd7bbfff700 1 -- 172.21.15.175:0/1505074239 wait complete. 2023-10-10T19:27:10.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.291+0000 7fd7bbfff700 1 Processor -- start 2023-10-10T19:27:10.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7bbfff700 1 -- start start 2023-10-10T19:27:10.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7bbfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc08a100 0x7fd7bc1504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:10.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7bbfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc0cd040 0x7fd7bc165850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:10.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7bbfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7bc14fd90 0x7fd7bc14ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:10.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7bbfff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc165d90 con 0x7fd7bc08a100 2023-10-10T19:27:10.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7bbfff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc169540 con 0x7fd7bc14fd90 2023-10-10T19:27:10.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7bbfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc150af0 con 0x7fd7bc0cd040 2023-10-10T19:27:10.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7baffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc08a100 0x7fd7bc1504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:10.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7baffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc08a100 0x7fd7bc1504b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55700/0 (socket says 172.21.15.175:55700) 2023-10-10T19:27:10.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7baffd700 1 -- 172.21.15.175:0/4036164778 learned_addr learned my addr 172.21.15.175:0/4036164778 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:10.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7bb7fe700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7bc14fd90 0x7fd7bc14ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:10.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.292+0000 7fd7ba7fc700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc0cd040 0x7fd7bc165850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:10.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7baffd700 1 -- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc0cd040 msgr2=0x7fd7bc165850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:10.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7baffd700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc0cd040 0x7fd7bc165850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7baffd700 1 -- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7bc14fd90 msgr2=0x7fd7bc14ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:10.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7baffd700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7bc14fd90 0x7fd7bc14ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7baffd700 1 -- 172.21.15.175:0/4036164778 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b4009ae0 con 0x7fd7bc08a100 2023-10-10T19:27:10.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7baffd700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc08a100 0x7fd7bc1504b0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00d860 tx=0x7fd7ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:10.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7a2ffd700 1 -- 172.21.15.175:0/4036164778 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0125f0 con 0x7fd7bc08a100 2023-10-10T19:27:10.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7a2ffd700 1 -- 172.21.15.175:0/4036164778 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac012750 con 0x7fd7bc08a100 2023-10-10T19:27:10.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7bc150dd0 con 0x7fd7bc08a100 2023-10-10T19:27:10.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7a2ffd700 1 -- 172.21.15.175:0/4036164778 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac012af0 con 0x7fd7bc08a100 2023-10-10T19:27:10.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.293+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7bc168ef0 con 0x7fd7bc08a100 2023-10-10T19:27:10.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.294+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7bc067570 con 0x7fd7bc08a100 2023-10-10T19:27:10.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.294+0000 7fd7a2ffd700 1 -- 172.21.15.175:0/4036164778 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd7ac013480 con 0x7fd7bc08a100 2023-10-10T19:27:10.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.294+0000 7fd7a2ffd700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7a8066f60 0x7fd7a8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:10.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.295+0000 7fd7a2ffd700 1 -- 172.21.15.175:0/4036164778 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fd7ac084e20 con 0x7fd7bc08a100 2023-10-10T19:27:10.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.295+0000 7fd7ba7fc700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7a8066f60 0x7fd7a8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:10.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.295+0000 7fd7ba7fc700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7a8066f60 0x7fd7a8069410 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0b1d80 tx=0x7fd7b0007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:10.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.299+0000 7fd7a2ffd700 1 -- 172.21.15.175:0/4036164778 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7ac0518b0 con 0x7fd7bc08a100 2023-10-10T19:27:10.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.529+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7bc155f80 con 0x7fd7bc08a100 2023-10-10T19:27:10.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.530+0000 7fd7a2ffd700 1 -- 172.21.15.175:0/4036164778 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7fd7ac054ed0 con 0x7fd7bc08a100 2023-10-10T19:27:10.531 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:10.531 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:10.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7a8066f60 msgr2=0x7fd7a8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7a8066f60 0x7fd7a8069410 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0b1d80 tx=0x7fd7b0007ec0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc08a100 msgr2=0x7fd7bc1504b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc08a100 0x7fd7bc1504b0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00d860 tx=0x7fd7ac008040 comp rx=0 tx=0).stop 2023-10-10T19:27:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 shutdown_connections 2023-10-10T19:27:10.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd7a8066f60 0x7fd7a8069410 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7bc14fd90 0x7fd7bc14ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd7bc08a100 0x7fd7bc1504b0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 --2- 172.21.15.175:0/4036164778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7bc0cd040 0x7fd7bc165850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:10.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 >> 172.21.15.175:0/4036164778 conn(0x7fd7bc063d00 msgr2=0x7fd7bc0cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 shutdown_connections 2023-10-10T19:27:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:10.532+0000 7fd7bbfff700 1 -- 172.21.15.175:0/4036164778 wait complete. 2023-10-10T19:27:10.535 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:10.544 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:27:12.079 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:27:12.078+0000 7f7f74406700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-10T19:27:12.546 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:12.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.673+0000 7fa17da45700 1 Processor -- start 2023-10-10T19:27:12.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.673+0000 7fa17da45700 1 -- start start 2023-10-10T19:27:12.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa17da45700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa1780b1fc0 0x7fa178155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:12.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa17da45700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1780aff60 0x7fa1780b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:12.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa17da45700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1781574b0 con 0x7fa1780b2390 2023-10-10T19:27:12.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa17da45700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa178150770 con 0x7fa1780b1fc0 2023-10-10T19:27:12.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa17da45700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1780afad0 con 0x7fa1780aff60 2023-10-10T19:27:12.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa176ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa1780b1fc0 0x7fa178155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:12.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa176ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa1780b1fc0 0x7fa178155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55710/0 (socket says 172.21.15.175:55710) 2023-10-10T19:27:12.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa176ffd700 1 -- 172.21.15.175:0/2087312041 learned_addr learned my addr 172.21.15.175:0/2087312041 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:12.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa1777fe700 1 --2- 172.21.15.175:0/2087312041 >> v2:172.21.15.191:3300/0 conn(0x7fa1780aff60 0x7fa1780b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:12.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa176ffd700 1 -- 172.21.15.175:0/2087312041 >> v2:172.21.15.191:3300/0 conn(0x7fa1780aff60 msgr2=0x7fa1780b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:12.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa176ffd700 1 --2- 172.21.15.175:0/2087312041 >> v2:172.21.15.191:3300/0 conn(0x7fa1780aff60 0x7fa1780b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa176ffd700 1 -- 172.21.15.175:0/2087312041 >> v1:172.21.15.191:6789/0 conn(0x7fa1780b2390 legacy=0x7fa178068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:12.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.674+0000 7fa176ffd700 1 -- 172.21.15.175:0/2087312041 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1780af4c0 con 0x7fa1780b1fc0 2023-10-10T19:27:12.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa176ffd700 1 --2- 172.21.15.175:0/2087312041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b1fc0 0x7fa178155fb0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fa1780641f0 tx=0x7fa168009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5662d1918d97fa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:12.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa175ffb700 1 -- 172.21.15.175:0/2087312041 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16800b8b0 con 0x7fa1780b1fc0 2023-10-10T19:27:12.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa175ffb700 1 -- 172.21.15.175:0/2087312041 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16800ba10 con 0x7fa1780b1fc0 2023-10-10T19:27:12.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa175ffb700 1 -- 172.21.15.175:0/2087312041 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16800bdb0 con 0x7fa1780b1fc0 2023-10-10T19:27:12.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa17da45700 1 -- 172.21.15.175:0/2087312041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b1fc0 msgr2=0x7fa178155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:12.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa17da45700 1 --2- 172.21.15.175:0/2087312041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b1fc0 0x7fa178155fb0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fa1780641f0 tx=0x7fa168009250 comp rx=0 tx=0).stop 2023-10-10T19:27:12.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa17da45700 1 -- 172.21.15.175:0/2087312041 shutdown_connections 2023-10-10T19:27:12.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa17da45700 1 --2- 172.21.15.175:0/2087312041 >> v2:172.21.15.191:3300/0 conn(0x7fa1780aff60 0x7fa1780b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa17da45700 1 --2- 172.21.15.175:0/2087312041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b1fc0 0x7fa178155fb0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa17da45700 1 -- 172.21.15.175:0/2087312041 >> 172.21.15.175:0/2087312041 conn(0x7fa178063230 msgr2=0x7fa178063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:12.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa17da45700 1 -- 172.21.15.175:0/2087312041 shutdown_connections 2023-10-10T19:27:12.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.675+0000 7fa17da45700 1 -- 172.21.15.175:0/2087312041 wait complete. 2023-10-10T19:27:12.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa17da45700 1 Processor -- start 2023-10-10T19:27:12.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa17da45700 1 -- start start 2023-10-10T19:27:12.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa17da45700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1780aff60 0x7fa1780cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:12.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa17da45700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1780b1fc0 0x7fa1780c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:12.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa17da45700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b2390 0x7fa1780c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:12.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa17da45700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780ccae0 con 0x7fa1780b2390 2023-10-10T19:27:12.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa17da45700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17814fcd0 con 0x7fa1780aff60 2023-10-10T19:27:12.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa17da45700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780ccd40 con 0x7fa1780b1fc0 2023-10-10T19:27:12.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa1777fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b2390 0x7fa1780c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:12.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa1777fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b2390 0x7fa1780c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55720/0 (socket says 172.21.15.175:55720) 2023-10-10T19:27:12.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.677+0000 7fa1777fe700 1 -- 172.21.15.175:0/1610970865 learned_addr learned my addr 172.21.15.175:0/1610970865 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:12.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa176ffd700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1780aff60 0x7fa1780cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:12.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa1767fc700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1780b1fc0 0x7fa1780c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:12.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa1777fe700 1 -- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1780b1fc0 msgr2=0x7fa1780c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:12.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa1777fe700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1780b1fc0 0x7fa1780c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa1777fe700 1 -- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1780aff60 msgr2=0x7fa1780cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:12.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa1777fe700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1780aff60 0x7fa1780cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa1777fe700 1 -- 172.21.15.175:0/1610970865 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1680094f0 con 0x7fa1780b2390 2023-10-10T19:27:12.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa1777fe700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b2390 0x7fa1780c55a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fa170009fa0 tx=0x7fa170009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:12.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa16effd700 1 -- 172.21.15.175:0/1610970865 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17000ebc0 con 0x7fa1780b2390 2023-10-10T19:27:12.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa16effd700 1 -- 172.21.15.175:0/1610970865 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17000ed20 con 0x7fa1780b2390 2023-10-10T19:27:12.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1780cd300 con 0x7fa1780b2390 2023-10-10T19:27:12.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa16effd700 1 -- 172.21.15.175:0/1610970865 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170014670 con 0x7fa1780b2390 2023-10-10T19:27:12.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.678+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1781668b0 con 0x7fa1780b2390 2023-10-10T19:27:12.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.679+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1780c4eb0 con 0x7fa1780b2390 2023-10-10T19:27:12.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.679+0000 7fa16effd700 1 -- 172.21.15.175:0/1610970865 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa1700149e0 con 0x7fa1780b2390 2023-10-10T19:27:12.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.680+0000 7fa16effd700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa158066f30 0x7fa1580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:12.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.680+0000 7fa16effd700 1 -- 172.21.15.175:0/1610970865 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fa170085380 con 0x7fa1780b2390 2023-10-10T19:27:12.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.680+0000 7fa176ffd700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa158066f30 0x7fa1580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:12.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.680+0000 7fa176ffd700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa158066f30 0x7fa1580693e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa1780641f0 tx=0x7fa168016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:12.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.684+0000 7fa16effd700 1 -- 172.21.15.175:0/1610970865 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa170051e10 con 0x7fa1780b2390 2023-10-10T19:27:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.910+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa178067570 con 0x7fa1780b2390 2023-10-10T19:27:12.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.911+0000 7fa16effd700 1 -- 172.21.15.175:0/1610970865 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7fa170055430 con 0x7fa1780b2390 2023-10-10T19:27:12.912 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:12.912 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa158066f30 msgr2=0x7fa1580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa158066f30 0x7fa1580693e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa1780641f0 tx=0x7fa168016000 comp rx=0 tx=0).stop 2023-10-10T19:27:12.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b2390 msgr2=0x7fa1780c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b2390 0x7fa1780c55a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fa170009fa0 tx=0x7fa170009900 comp rx=0 tx=0).stop 2023-10-10T19:27:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 shutdown_connections 2023-10-10T19:27:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa158066f30 0x7fa1580693e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1780b2390 0x7fa1780c55a0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1780aff60 0x7fa1780cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 --2- 172.21.15.175:0/1610970865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1780b1fc0 0x7fa1780c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:12.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 >> 172.21.15.175:0/1610970865 conn(0x7fa178063230 msgr2=0x7fa17815a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:12.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 shutdown_connections 2023-10-10T19:27:12.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:12.913+0000 7fa17da45700 1 -- 172.21.15.175:0/1610970865 wait complete. 2023-10-10T19:27:12.916 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:12.925 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:27:14.926 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:15.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.055+0000 7f2c8de77700 1 Processor -- start 2023-10-10T19:27:15.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c8de77700 1 -- start start 2023-10-10T19:27:15.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c8de77700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2c8814cc10 0x7f2c8806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:15.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c8de77700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c880c52a0 0x7f2c88152970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:15.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c8de77700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c88155ca0 con 0x7f2c8814c840 2023-10-10T19:27:15.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c8de77700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c88156b00 con 0x7f2c8814cc10 2023-10-10T19:27:15.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c8de77700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8806d700 con 0x7f2c880c52a0 2023-10-10T19:27:15.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c877fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2c8814c840 0x7f2c880c4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45584/0 (socket says 172.21.15.175:45584) 2023-10-10T19:27:15.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c86ffd700 1 --2- 172.21.15.175:0/1017440662 >> v2:172.21.15.175:3300/0 conn(0x7f2c8814cc10 0x7f2c8806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:15.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.056+0000 7f2c877fe700 1 -- 172.21.15.175:0/1017440662 learned_addr learned my addr 172.21.15.175:0/1017440662 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:27:15.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c87fff700 1 --2- 172.21.15.175:0/1017440662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c880c52a0 0x7f2c88152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:15.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c86ffd700 1 -- 172.21.15.175:0/1017440662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c880c52a0 msgr2=0x7f2c88152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:15.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c86ffd700 1 --2- 172.21.15.175:0/1017440662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c880c52a0 0x7f2c88152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c86ffd700 1 -- 172.21.15.175:0/1017440662 >> v1:172.21.15.175:6789/0 conn(0x7f2c8814c840 legacy=0x7f2c880c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:15.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c867fc700 1 -- 172.21.15.175:0/1017440662 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288050618 0 0) 0x7f2c88155ca0 con 0x7f2c8814c840 2023-10-10T19:27:15.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c86ffd700 1 -- 172.21.15.175:0/1017440662 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c88155ef0 con 0x7f2c8814cc10 2023-10-10T19:27:15.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c87fff700 1 --2- 172.21.15.175:0/1017440662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c880c52a0 0x7f2c88152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:27:15.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c86ffd700 1 --2- 172.21.15.175:0/1017440662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8814cc10 0x7f2c8806e7a0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c0098e0 tx=0x7f2c7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23e629ed2835785 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:15.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c867fc700 1 -- 172.21.15.175:0/1017440662 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c009e80 con 0x7f2c8814cc10 2023-10-10T19:27:15.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c867fc700 1 -- 172.21.15.175:0/1017440662 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c00b8f0 con 0x7f2c8814cc10 2023-10-10T19:27:15.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.057+0000 7f2c867fc700 1 -- 172.21.15.175:0/1017440662 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c00bc70 con 0x7f2c8814cc10 2023-10-10T19:27:15.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.058+0000 7f2c8de77700 1 -- 172.21.15.175:0/1017440662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8814cc10 msgr2=0x7f2c8806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:15.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.058+0000 7f2c8de77700 1 --2- 172.21.15.175:0/1017440662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8814cc10 0x7f2c8806e7a0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c0098e0 tx=0x7f2c7c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.058+0000 7f2c8de77700 1 -- 172.21.15.175:0/1017440662 shutdown_connections 2023-10-10T19:27:15.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.058+0000 7f2c8de77700 1 --2- 172.21.15.175:0/1017440662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c880c52a0 0x7f2c88152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.058+0000 7f2c8de77700 1 --2- 172.21.15.175:0/1017440662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c8814cc10 0x7f2c8806e7a0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.058+0000 7f2c8de77700 1 -- 172.21.15.175:0/1017440662 >> 172.21.15.175:0/1017440662 conn(0x7f2c88068280 msgr2=0x7f2c8806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:15.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.058+0000 7f2c8de77700 1 -- 172.21.15.175:0/1017440662 shutdown_connections 2023-10-10T19:27:15.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.058+0000 7f2c8de77700 1 -- 172.21.15.175:0/1017440662 wait complete. 2023-10-10T19:27:15.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.059+0000 7f2c8de77700 1 Processor -- start 2023-10-10T19:27:15.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.059+0000 7f2c8de77700 1 -- start start 2023-10-10T19:27:15.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c8de77700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c880c52a0 0x7f2c88146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:15.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c8de77700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8814c840 0x7f2c88147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:15.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c8de77700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8814cc10 0x7f2c88150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:15.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c8de77700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c880c78d0 con 0x7f2c880c52a0 2023-10-10T19:27:15.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c8de77700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c880c69e0 con 0x7f2c8814c840 2023-10-10T19:27:15.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c8de77700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c881626f0 con 0x7f2c8814cc10 2023-10-10T19:27:15.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c880c52a0 0x7f2c88146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:15.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c880c52a0 0x7f2c88146d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49454/0 (socket says 172.21.15.175:49454) 2023-10-10T19:27:15.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 -- 172.21.15.175:0/3583687820 learned_addr learned my addr 172.21.15.175:0/3583687820 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:15.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c87fff700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8814cc10 0x7f2c88150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:15.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c86ffd700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8814c840 0x7f2c88147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:15.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 -- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8814cc10 msgr2=0x7f2c88150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:15.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8814cc10 0x7f2c88150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 -- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8814c840 msgr2=0x7f2c88147280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:15.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8814c840 0x7f2c88147280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 -- 172.21.15.175:0/3583687820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c009420 con 0x7f2c880c52a0 2023-10-10T19:27:15.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c87fff700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8814cc10 0x7f2c88150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:27:15.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c86ffd700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8814c840 0x7f2c88147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:27:15.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c877fe700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c880c52a0 0x7f2c88146d40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800c810 tx=0x7f2c78007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:15.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c6f7fe700 1 -- 172.21.15.175:0/3583687820 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78007300 con 0x7f2c880c52a0 2023-10-10T19:27:15.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c6f7fe700 1 -- 172.21.15.175:0/3583687820 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c78007460 con 0x7f2c880c52a0 2023-10-10T19:27:15.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c881629c0 con 0x7f2c880c52a0 2023-10-10T19:27:15.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c6f7fe700 1 -- 172.21.15.175:0/3583687820 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c780057f0 con 0x7f2c880c52a0 2023-10-10T19:27:15.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.060+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c880c7560 con 0x7f2c880c52a0 2023-10-10T19:27:15.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.061+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c880c70c0 con 0x7f2c880c52a0 2023-10-10T19:27:15.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.062+0000 7f2c6f7fe700 1 -- 172.21.15.175:0/3583687820 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2c78005c00 con 0x7f2c880c52a0 2023-10-10T19:27:15.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.062+0000 7f2c6f7fe700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c74066f60 0x7f2c74069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:15.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.062+0000 7f2c6f7fe700 1 -- 172.21.15.175:0/3583687820 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f2c78084850 con 0x7f2c880c52a0 2023-10-10T19:27:15.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.062+0000 7f2c86ffd700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c74066f60 0x7f2c74069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:15.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.063+0000 7f2c86ffd700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c74066f60 0x7f2c74069410 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2c881477c0 tx=0x7f2c7c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:15.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.066+0000 7f2c6f7fe700 1 -- 172.21.15.175:0/3583687820 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c78051360 con 0x7f2c880c52a0 2023-10-10T19:27:15.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.291+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c880c1690 con 0x7f2c880c52a0 2023-10-10T19:27:15.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.291+0000 7f2c6f7fe700 1 -- 172.21.15.175:0/3583687820 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7f2c78054980 con 0x7f2c880c52a0 2023-10-10T19:27:15.292 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:15.293 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c74066f60 msgr2=0x7f2c74069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c74066f60 0x7f2c74069410 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2c881477c0 tx=0x7f2c7c017000 comp rx=0 tx=0).stop 2023-10-10T19:27:15.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c880c52a0 msgr2=0x7f2c88146d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c880c52a0 0x7f2c88146d40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800c810 tx=0x7f2c78007c80 comp rx=0 tx=0).stop 2023-10-10T19:27:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 shutdown_connections 2023-10-10T19:27:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2c74066f60 0x7f2c74069410 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c8814cc10 0x7f2c88150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2c880c52a0 0x7f2c88146d40 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 --2- 172.21.15.175:0/3583687820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8814c840 0x7f2c88147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.293+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 >> 172.21.15.175:0/3583687820 conn(0x7f2c88068280 msgr2=0x7f2c880c4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.294+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 shutdown_connections 2023-10-10T19:27:15.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:15.294+0000 7f2c8de77700 1 -- 172.21.15.175:0/3583687820 wait complete. 2023-10-10T19:27:15.297 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:15.306 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:27:17.307 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:17.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.434+0000 7efdace6d700 1 Processor -- start 2023-10-10T19:27:17.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.434+0000 7efdace6d700 1 -- start start 2023-10-10T19:27:17.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.435+0000 7efdace6d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b1fc0 0x7efda8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:17.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.435+0000 7efdace6d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7efda80b2390 0x7efda8068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:17.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.435+0000 7efdace6d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda81574b0 con 0x7efda8068bd0 2023-10-10T19:27:17.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.435+0000 7efdace6d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda8150770 con 0x7efda80b2390 2023-10-10T19:27:17.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.435+0000 7efdace6d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda80afad0 con 0x7efda80b1fc0 2023-10-10T19:27:17.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.435+0000 7efda5d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7efda80b2390 0x7efda8068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:17.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.435+0000 7efda5d9b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7efda80b2390 0x7efda8068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49462/0 (socket says 172.21.15.175:49462) 2023-10-10T19:27:17.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.435+0000 7efda5d9b700 1 -- 172.21.15.175:0/3232251951 learned_addr learned my addr 172.21.15.175:0/3232251951 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:17.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda659c700 1 --2- 172.21.15.175:0/3232251951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b1fc0 0x7efda8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:17.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda5d9b700 1 -- 172.21.15.175:0/3232251951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b1fc0 msgr2=0x7efda8155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:17.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda5d9b700 1 --2- 172.21.15.175:0/3232251951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b1fc0 0x7efda8155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda5d9b700 1 -- 172.21.15.175:0/3232251951 >> v1:172.21.15.175:6789/0 conn(0x7efda8068bd0 legacy=0x7efda80aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:17.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda5d9b700 1 -- 172.21.15.175:0/3232251951 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda80af4c0 con 0x7efda80b2390 2023-10-10T19:27:17.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda659c700 1 --2- 172.21.15.175:0/3232251951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b1fc0 0x7efda8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:27:17.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda5d9b700 1 --2- 172.21.15.175:0/3232251951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda80b2390 0x7efda8068690 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7efd900099b0 tx=0x7efd90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ec6a7886d2c772 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:17.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda559a700 1 -- 172.21.15.175:0/3232251951 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000b8f0 con 0x7efda80b2390 2023-10-10T19:27:17.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda559a700 1 -- 172.21.15.175:0/3232251951 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9000ba50 con 0x7efda80b2390 2023-10-10T19:27:17.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efda559a700 1 -- 172.21.15.175:0/3232251951 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000bdf0 con 0x7efda80b2390 2023-10-10T19:27:17.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efdace6d700 1 -- 172.21.15.175:0/3232251951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda80b2390 msgr2=0x7efda8068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:17.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.436+0000 7efdace6d700 1 --2- 172.21.15.175:0/3232251951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda80b2390 0x7efda8068690 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7efd900099b0 tx=0x7efd90009250 comp rx=0 tx=0).stop 2023-10-10T19:27:17.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.437+0000 7efdace6d700 1 -- 172.21.15.175:0/3232251951 shutdown_connections 2023-10-10T19:27:17.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.437+0000 7efdace6d700 1 --2- 172.21.15.175:0/3232251951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b1fc0 0x7efda8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.437+0000 7efdace6d700 1 --2- 172.21.15.175:0/3232251951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda80b2390 0x7efda8068690 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.437+0000 7efdace6d700 1 -- 172.21.15.175:0/3232251951 >> 172.21.15.175:0/3232251951 conn(0x7efda8063230 msgr2=0x7efda8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:17.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.437+0000 7efdace6d700 1 -- 172.21.15.175:0/3232251951 shutdown_connections 2023-10-10T19:27:17.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.437+0000 7efdace6d700 1 -- 172.21.15.175:0/3232251951 wait complete. 2023-10-10T19:27:17.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.438+0000 7efdace6d700 1 Processor -- start 2023-10-10T19:27:17.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.438+0000 7efdace6d700 1 -- start start 2023-10-10T19:27:17.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.438+0000 7efdace6d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda8068bd0 0x7efda8166670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:17.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.438+0000 7efdace6d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efda80b1fc0 0x7efda80ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:17.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.438+0000 7efdace6d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b2390 0x7efda814fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:17.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.438+0000 7efdace6d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda8156a60 con 0x7efda8068bd0 2023-10-10T19:27:17.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.438+0000 7efdace6d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda8166bb0 con 0x7efda80b1fc0 2023-10-10T19:27:17.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.438+0000 7efdace6d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda80cd0b0 con 0x7efda80b2390 2023-10-10T19:27:17.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda8068bd0 0x7efda8166670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:17.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda8068bd0 0x7efda8166670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49476/0 (socket says 172.21.15.175:49476) 2023-10-10T19:27:17.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 -- 172.21.15.175:0/2828768162 learned_addr learned my addr 172.21.15.175:0/2828768162 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:17.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda5d9b700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efda80b1fc0 0x7efda80ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:17.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda6d9d700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b2390 0x7efda814fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:17.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 -- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b2390 msgr2=0x7efda814fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:17.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b2390 0x7efda814fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 -- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efda80b1fc0 msgr2=0x7efda80ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:17.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efda80b1fc0 0x7efda80ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 -- 172.21.15.175:0/2828768162 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd900094f0 con 0x7efda8068bd0 2023-10-10T19:27:17.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda5d9b700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efda80b1fc0 0x7efda80ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:27:17.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efda659c700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda8068bd0 0x7efda8166670 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7efd9800a430 tx=0x7efd98007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:17.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efd9e7fc700 1 -- 172.21.15.175:0/2828768162 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9800d650 con 0x7efda8068bd0 2023-10-10T19:27:17.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efd9e7fc700 1 -- 172.21.15.175:0/2828768162 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9800d7b0 con 0x7efda8068bd0 2023-10-10T19:27:17.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda8155050 con 0x7efda8068bd0 2023-10-10T19:27:17.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efd9e7fc700 1 -- 172.21.15.175:0/2828768162 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9800db50 con 0x7efda8068bd0 2023-10-10T19:27:17.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.439+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda8154bc0 con 0x7efda8068bd0 2023-10-10T19:27:17.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.440+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda8067570 con 0x7efda8068bd0 2023-10-10T19:27:17.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.440+0000 7efd9e7fc700 1 -- 172.21.15.175:0/2828768162 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efd98015440 con 0x7efda8068bd0 2023-10-10T19:27:17.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.441+0000 7efd9e7fc700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efd88066f30 0x7efd880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:17.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.441+0000 7efd9e7fc700 1 -- 172.21.15.175:0/2828768162 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7efd98085440 con 0x7efda8068bd0 2023-10-10T19:27:17.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.441+0000 7efda5d9b700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efd88066f30 0x7efd880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:17.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.441+0000 7efda5d9b700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efd88066f30 0x7efd880693e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efd900099b0 tx=0x7efd90017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:17.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.445+0000 7efd9e7fc700 1 -- 172.21.15.175:0/2828768162 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efd9804e5a0 con 0x7efda8068bd0 2023-10-10T19:27:17.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.669+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efda8155030 con 0x7efda8068bd0 2023-10-10T19:27:17.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.670+0000 7efd9e7fc700 1 -- 172.21.15.175:0/2828768162 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7efd98051f80 con 0x7efda8068bd0 2023-10-10T19:27:17.670 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:17.671 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:17.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efd88066f30 msgr2=0x7efd880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efd88066f30 0x7efd880693e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efd900099b0 tx=0x7efd90017000 comp rx=0 tx=0).stop 2023-10-10T19:27:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda8068bd0 msgr2=0x7efda8166670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda8068bd0 0x7efda8166670 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7efd9800a430 tx=0x7efd98007910 comp rx=0 tx=0).stop 2023-10-10T19:27:17.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 shutdown_connections 2023-10-10T19:27:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7efd88066f30 0x7efd880693e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efda80b2390 0x7efda814fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efda8068bd0 0x7efda8166670 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 --2- 172.21.15.175:0/2828768162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efda80b1fc0 0x7efda80ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 >> 172.21.15.175:0/2828768162 conn(0x7efda8063230 msgr2=0x7efda81508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 shutdown_connections 2023-10-10T19:27:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:17.672+0000 7efdace6d700 1 -- 172.21.15.175:0/2828768162 wait complete. 2023-10-10T19:27:17.676 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:17.684 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:27:19.686 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:19.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.813+0000 7f7872022700 1 Processor -- start 2023-10-10T19:27:19.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.813+0000 7f7872022700 1 -- start start 2023-10-10T19:27:19.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.813+0000 7f7872022700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0b1fc0 0x7f786c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:19.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.813+0000 7f7872022700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f786c0aff60 0x7f786c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:19.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.813+0000 7f7872022700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786c1574b0 con 0x7f786c0b2390 2023-10-10T19:27:19.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.813+0000 7f7872022700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786c150770 con 0x7f786c0aff60 2023-10-10T19:27:19.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.813+0000 7f7872022700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0afad0 con 0x7f786c0b1fc0 2023-10-10T19:27:19.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f786c0aff60 0x7f786c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:19.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f786c0aff60 0x7f786c0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49478/0 (socket says 172.21.15.175:49478) 2023-10-10T19:27:19.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786bfff700 1 -- 172.21.15.175:0/2953211284 learned_addr learned my addr 172.21.15.175:0/2953211284 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:19.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786b7fe700 1 --2- 172.21.15.175:0/2953211284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0b1fc0 0x7f786c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:19.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786bfff700 1 -- 172.21.15.175:0/2953211284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0b1fc0 msgr2=0x7f786c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:19.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786bfff700 1 --2- 172.21.15.175:0/2953211284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0b1fc0 0x7f786c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:19.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786bfff700 1 -- 172.21.15.175:0/2953211284 >> v1:172.21.15.175:6789/0 conn(0x7f786c0b2390 legacy=0x7f786c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:19.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786bfff700 1 -- 172.21.15.175:0/2953211284 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c0af4c0 con 0x7f786c0aff60 2023-10-10T19:27:19.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786b7fe700 1 --2- 172.21.15.175:0/2953211284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0b1fc0 0x7f786c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:27:19.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786bfff700 1 --2- 172.21.15.175:0/2953211284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0aff60 0x7f786c0b03e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7864009f50 tx=0x7f7864009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5ec7749d77fd9b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:19.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786a7fc700 1 -- 172.21.15.175:0/2953211284 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786400e070 con 0x7f786c0aff60 2023-10-10T19:27:19.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786a7fc700 1 -- 172.21.15.175:0/2953211284 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786400bb00 con 0x7f786c0aff60 2023-10-10T19:27:19.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.814+0000 7f786a7fc700 1 -- 172.21.15.175:0/2953211284 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78640092e0 con 0x7f786c0aff60 2023-10-10T19:27:19.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.815+0000 7f7872022700 1 -- 172.21.15.175:0/2953211284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0aff60 msgr2=0x7f786c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:19.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.815+0000 7f7872022700 1 --2- 172.21.15.175:0/2953211284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0aff60 0x7f786c0b03e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7864009f50 tx=0x7f7864009840 comp rx=0 tx=0).stop 2023-10-10T19:27:19.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.815+0000 7f7872022700 1 -- 172.21.15.175:0/2953211284 shutdown_connections 2023-10-10T19:27:19.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.815+0000 7f7872022700 1 --2- 172.21.15.175:0/2953211284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0aff60 0x7f786c0b03e0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:19.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.815+0000 7f7872022700 1 --2- 172.21.15.175:0/2953211284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0b1fc0 0x7f786c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:19.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.815+0000 7f7872022700 1 -- 172.21.15.175:0/2953211284 >> 172.21.15.175:0/2953211284 conn(0x7f786c063230 msgr2=0x7f786c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:19.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.815+0000 7f7872022700 1 -- 172.21.15.175:0/2953211284 shutdown_connections 2023-10-10T19:27:19.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.815+0000 7f7872022700 1 -- 172.21.15.175:0/2953211284 wait complete. 2023-10-10T19:27:19.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.816+0000 7f7872022700 1 Processor -- start 2023-10-10T19:27:19.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.816+0000 7f7872022700 1 -- start start 2023-10-10T19:27:19.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f7872022700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0aff60 0x7f786c0c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:19.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f7872022700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0b1fc0 0x7f786c0c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:19.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f7872022700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f786c0b2390 0x7f786c0c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:19.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f7872022700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0c4840 con 0x7f786c0b1fc0 2023-10-10T19:27:19.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f7872022700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c165480 con 0x7f786c0b2390 2023-10-10T19:27:19.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f7872022700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c165fc0 con 0x7f786c0aff60 2023-10-10T19:27:19.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0b1fc0 0x7f786c0c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:19.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0b1fc0 0x7f786c0c24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49484/0 (socket says 172.21.15.175:49484) 2023-10-10T19:27:19.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 -- 172.21.15.175:0/2588986157 learned_addr learned my addr 172.21.15.175:0/2588986157 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:19.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786b7fe700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0aff60 0x7f786c0c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:19.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786bfff700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f786c0b2390 0x7f786c0c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:19.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 -- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0aff60 msgr2=0x7f786c0c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:19.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0aff60 0x7f786c0c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:19.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 -- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f786c0b2390 msgr2=0x7f786c0c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:19.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f786c0b2390 0x7f786c0c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:19.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 -- 172.21.15.175:0/2588986157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7864009ae0 con 0x7f786c0b1fc0 2023-10-10T19:27:19.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.817+0000 7f786affd700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0b1fc0 0x7f786c0c24b0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f785c00c4e0 tx=0x7f785c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:19.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.818+0000 7f78637fe700 1 -- 172.21.15.175:0/2588986157 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c015b70 con 0x7f786c0b1fc0 2023-10-10T19:27:19.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.818+0000 7f78637fe700 1 -- 172.21.15.175:0/2588986157 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c015cf0 con 0x7f786c0b1fc0 2023-10-10T19:27:19.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.818+0000 7f78637fe700 1 -- 172.21.15.175:0/2588986157 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c01d640 con 0x7f786c0b1fc0 2023-10-10T19:27:19.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.818+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c169750 con 0x7f786c0b1fc0 2023-10-10T19:27:19.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.818+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c165970 con 0x7f786c0b1fc0 2023-10-10T19:27:19.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.819+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c067570 con 0x7f786c0b1fc0 2023-10-10T19:27:19.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.819+0000 7f78637fe700 1 -- 172.21.15.175:0/2588986157 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f785c0263a0 con 0x7f786c0b1fc0 2023-10-10T19:27:19.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.819+0000 7f78637fe700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f784c066f30 0x7f784c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:19.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.820+0000 7f78637fe700 1 -- 172.21.15.175:0/2588986157 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f785c08c760 con 0x7f786c0b1fc0 2023-10-10T19:27:19.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.820+0000 7f786b7fe700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f784c066f30 0x7f784c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:19.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.820+0000 7f786b7fe700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f784c066f30 0x7f784c0693e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f785400adc0 tx=0x7f7854006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:19.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:19.824+0000 7f78637fe700 1 -- 172.21.15.175:0/2588986157 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f785c0591f0 con 0x7f786c0b1fc0 2023-10-10T19:27:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.051+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f786c068da0 con 0x7f786c0b1fc0 2023-10-10T19:27:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.052+0000 7f78637fe700 1 -- 172.21.15.175:0/2588986157 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7f785c015e60 con 0x7f786c0b1fc0 2023-10-10T19:27:20.053 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:20.053 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:20.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f784c066f30 msgr2=0x7f784c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:20.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f784c066f30 0x7f784c0693e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f785400adc0 tx=0x7f7854006f90 comp rx=0 tx=0).stop 2023-10-10T19:27:20.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0b1fc0 msgr2=0x7f786c0c24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:20.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0b1fc0 0x7f786c0c24b0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f785c00c4e0 tx=0x7f785c00c7f0 comp rx=0 tx=0).stop 2023-10-10T19:27:20.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 shutdown_connections 2023-10-10T19:27:20.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f784c066f30 0x7f784c0693e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:20.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f786c0b2390 0x7f786c0c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:20.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f786c0aff60 0x7f786c0c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:20.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 --2- 172.21.15.175:0/2588986157 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f786c0b1fc0 0x7f786c0c24b0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:20.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 >> 172.21.15.175:0/2588986157 conn(0x7f786c063230 msgr2=0x7f786c150ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:20.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 shutdown_connections 2023-10-10T19:27:20.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:20.054+0000 7f7872022700 1 -- 172.21.15.175:0/2588986157 wait complete. 2023-10-10T19:27:20.058 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:20.067 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:27:22.068 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:22.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.198+0000 7fa49d6a6700 1 Processor -- start 2023-10-10T19:27:22.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.198+0000 7fa49d6a6700 1 -- start start 2023-10-10T19:27:22.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa49d6a6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4980b2060 0x7fa498157470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:22.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa49d6a6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4980b0130 con 0x7fa4980b2430 2023-10-10T19:27:22.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa49d6a6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa498156c90 con 0x7fa498150460 2023-10-10T19:27:22.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa49d6a6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980af800 con 0x7fa4980b2060 2023-10-10T19:27:22.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa4967fc700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa4980b2430 0x7fa49814fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:45608/0 (socket says 172.21.15.175:45608) 2023-10-10T19:27:22.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa4967fc700 1 -- 172.21.15.175:0/3969483895 learned_addr learned my addr 172.21.15.175:0/3969483895 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:27:22.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa496ffd700 1 --2- 172.21.15.175:0/3969483895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4980b2060 0x7fa498157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:22.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa495ffb700 1 -- 172.21.15.175:0/3969483895 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811145478 0 0) 0x7fa4980b0130 con 0x7fa4980b2430 2023-10-10T19:27:22.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa495ffb700 1 -- 172.21.15.175:0/3969483895 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa484003570 con 0x7fa4980b2430 2023-10-10T19:27:22.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa495ffb700 1 -- 172.21.15.175:0/3969483895 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779631007 0 0) 0x7fa498156c90 con 0x7fa498150460 2023-10-10T19:27:22.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa495ffb700 1 -- 172.21.15.175:0/3969483895 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4980b0130 con 0x7fa498150460 2023-10-10T19:27:22.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa496ffd700 1 -- 172.21.15.175:0/3969483895 >> v1:172.21.15.191:6789/0 conn(0x7fa498150460 legacy=0x7fa498155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:22.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa496ffd700 1 -- 172.21.15.175:0/3969483895 >> v1:172.21.15.175:6789/0 conn(0x7fa4980b2430 legacy=0x7fa49814fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:22.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.199+0000 7fa496ffd700 1 -- 172.21.15.175:0/3969483895 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4980afae0 con 0x7fa4980b2060 2023-10-10T19:27:22.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa496ffd700 1 --2- 172.21.15.175:0/3969483895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4980b2060 0x7fa498157470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa488009e90 tx=0x7fa488009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35452a54caa87583 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:22.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa495ffb700 1 -- 172.21.15.175:0/3969483895 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48800e070 con 0x7fa4980b2060 2023-10-10T19:27:22.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa495ffb700 1 -- 172.21.15.175:0/3969483895 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48800bb80 con 0x7fa4980b2060 2023-10-10T19:27:22.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa495ffb700 1 -- 172.21.15.175:0/3969483895 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488012430 con 0x7fa4980b2060 2023-10-10T19:27:22.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa49d6a6700 1 -- 172.21.15.175:0/3969483895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4980b2060 msgr2=0x7fa498157470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:22.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa49d6a6700 1 --2- 172.21.15.175:0/3969483895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4980b2060 0x7fa498157470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa488009e90 tx=0x7fa488009780 comp rx=0 tx=0).stop 2023-10-10T19:27:22.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa49d6a6700 1 -- 172.21.15.175:0/3969483895 shutdown_connections 2023-10-10T19:27:22.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa49d6a6700 1 --2- 172.21.15.175:0/3969483895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4980b2060 0x7fa498157470 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:22.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa49d6a6700 1 -- 172.21.15.175:0/3969483895 >> 172.21.15.175:0/3969483895 conn(0x7fa498063230 msgr2=0x7fa498063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:22.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa49d6a6700 1 -- 172.21.15.175:0/3969483895 shutdown_connections 2023-10-10T19:27:22.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.200+0000 7fa49d6a6700 1 -- 172.21.15.175:0/3969483895 wait complete. 2023-10-10T19:27:22.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa49d6a6700 1 Processor -- start 2023-10-10T19:27:22.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa49d6a6700 1 -- start start 2023-10-10T19:27:22.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa49d6a6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980b2060 0x7fa4980c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:22.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa49d6a6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4980b2430 0x7fa498165e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:22.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa49d6a6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa498150460 0x7fa49814e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:22.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa49d6a6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980c4870 con 0x7fa4980b2060 2023-10-10T19:27:22.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa49d6a6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980c3910 con 0x7fa4980b2430 2023-10-10T19:27:22.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa49d6a6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980c3bf0 con 0x7fa498150460 2023-10-10T19:27:22.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa496ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980b2060 0x7fa4980c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:22.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa496ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980b2060 0x7fa4980c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49496/0 (socket says 172.21.15.175:49496) 2023-10-10T19:27:22.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.202+0000 7fa496ffd700 1 -- 172.21.15.175:0/874131407 learned_addr learned my addr 172.21.15.175:0/874131407 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:22.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa4967fc700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4980b2430 0x7fa498165e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:22.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa4977fe700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa498150460 0x7fa49814e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:22.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa496ffd700 1 -- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa498150460 msgr2=0x7fa49814e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:22.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa496ffd700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa498150460 0x7fa49814e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:22.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa496ffd700 1 -- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4980b2430 msgr2=0x7fa498165e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:22.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa496ffd700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4980b2430 0x7fa498165e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:22.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa496ffd700 1 -- 172.21.15.175:0/874131407 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa488009a20 con 0x7fa4980b2060 2023-10-10T19:27:22.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa496ffd700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980b2060 0x7fa4980c5280 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa4880097c0 tx=0x7fa488019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:22.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa48effd700 1 -- 172.21.15.175:0/874131407 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488012d80 con 0x7fa4980b2060 2023-10-10T19:27:22.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa48effd700 1 -- 172.21.15.175:0/874131407 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48801b420 con 0x7fa4980b2060 2023-10-10T19:27:22.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4980c3ed0 con 0x7fa4980b2060 2023-10-10T19:27:22.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa48effd700 1 -- 172.21.15.175:0/874131407 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48801b7e0 con 0x7fa4980b2060 2023-10-10T19:27:22.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.203+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4980c4490 con 0x7fa4980b2060 2023-10-10T19:27:22.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.204+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa498069b30 con 0x7fa4980b2060 2023-10-10T19:27:22.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.204+0000 7fa48effd700 1 -- 172.21.15.175:0/874131407 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa48801bb50 con 0x7fa4980b2060 2023-10-10T19:27:22.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.205+0000 7fa48effd700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa478066f30 0x7fa4780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:22.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.205+0000 7fa48effd700 1 -- 172.21.15.175:0/874131407 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7fa488086930 con 0x7fa4980b2060 2023-10-10T19:27:22.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.205+0000 7fa4967fc700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa478066f30 0x7fa4780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:22.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.206+0000 7fa4967fc700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa478066f30 0x7fa4780693e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa4981639f0 tx=0x7fa480009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:22.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.209+0000 7fa48effd700 1 -- 172.21.15.175:0/874131407 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa488056e50 con 0x7fa4980b2060 2023-10-10T19:27:22.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.434+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa498067570 con 0x7fa4980b2060 2023-10-10T19:27:22.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.435+0000 7fa48effd700 1 -- 172.21.15.175:0/874131407 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7fa4880569e0 con 0x7fa4980b2060 2023-10-10T19:27:22.437 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:22.437 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:22.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa478066f30 msgr2=0x7fa4780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:22.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa478066f30 0x7fa4780693e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa4981639f0 tx=0x7fa480009040 comp rx=0 tx=0).stop 2023-10-10T19:27:22.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980b2060 msgr2=0x7fa4980c5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:22.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980b2060 0x7fa4980c5280 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa4880097c0 tx=0x7fa488019000 comp rx=0 tx=0).stop 2023-10-10T19:27:22.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 shutdown_connections 2023-10-10T19:27:22.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fa478066f30 0x7fa4780693e0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:22.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa498150460 0x7fa49814e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:22.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980b2060 0x7fa4980c5280 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:22.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 --2- 172.21.15.175:0/874131407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4980b2430 0x7fa498165e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:22.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 >> 172.21.15.175:0/874131407 conn(0x7fa498063230 msgr2=0x7fa4980ca880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:22.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 shutdown_connections 2023-10-10T19:27:22.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:22.437+0000 7fa49d6a6700 1 -- 172.21.15.175:0/874131407 wait complete. 2023-10-10T19:27:22.443 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:22.451 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:27:24.452 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:24.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.578+0000 7f3467ac2700 1 Processor -- start 2023-10-10T19:27:24.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.579+0000 7f3467ac2700 1 -- start start 2023-10-10T19:27:24.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.579+0000 7f3467ac2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34600c3330 0x7f34600c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:24.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.579+0000 7f3467ac2700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f34600c41e0 0x7f346014cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:24.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.579+0000 7f3467ac2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34601533e0 con 0x7f346006b640 2023-10-10T19:27:24.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.579+0000 7f3467ac2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346006e130 con 0x7f34600c3330 2023-10-10T19:27:24.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.579+0000 7f3467ac2700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346006a970 con 0x7f34600c41e0 2023-10-10T19:27:24.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346585e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34600c3330 0x7f34600c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:24.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346585e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f34600c3330 0x7f34600c3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33546/0 (socket says 172.21.15.175:33546) 2023-10-10T19:27:24.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346585e700 1 -- 172.21.15.175:0/153278485 learned_addr learned my addr 172.21.15.175:0/153278485 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:24.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346505d700 1 --2- 172.21.15.175:0/153278485 >> v2:172.21.15.191:3300/0 conn(0x7f34600c41e0 0x7f346014cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:24.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346585e700 1 -- 172.21.15.175:0/153278485 >> v2:172.21.15.191:3300/0 conn(0x7f34600c41e0 msgr2=0x7f346014cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:24.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346585e700 1 --2- 172.21.15.175:0/153278485 >> v2:172.21.15.191:3300/0 conn(0x7f34600c41e0 0x7f346014cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346585e700 1 -- 172.21.15.175:0/153278485 >> v1:172.21.15.175:6789/0 conn(0x7f346006b640 legacy=0x7f3460156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:24.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346585e700 1 -- 172.21.15.175:0/153278485 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346006d520 con 0x7f34600c3330 2023-10-10T19:27:24.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346505d700 1 --2- 172.21.15.175:0/153278485 >> v2:172.21.15.191:3300/0 conn(0x7f34600c41e0 0x7f346014cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:27:24.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346585e700 1 --2- 172.21.15.175:0/153278485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f34600c3ca0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f346006ce00 tx=0x7f345c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1567cb104cea7be1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:24.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346485c700 1 -- 172.21.15.175:0/153278485 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c005c10 con 0x7f34600c3330 2023-10-10T19:27:24.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346485c700 1 -- 172.21.15.175:0/153278485 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c005d70 con 0x7f34600c3330 2023-10-10T19:27:24.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.580+0000 7f346485c700 1 -- 172.21.15.175:0/153278485 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c00b880 con 0x7f34600c3330 2023-10-10T19:27:24.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.581+0000 7f3467ac2700 1 -- 172.21.15.175:0/153278485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 msgr2=0x7f34600c3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:24.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.581+0000 7f3467ac2700 1 --2- 172.21.15.175:0/153278485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f34600c3ca0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f346006ce00 tx=0x7f345c009220 comp rx=0 tx=0).stop 2023-10-10T19:27:24.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.581+0000 7f3467ac2700 1 -- 172.21.15.175:0/153278485 shutdown_connections 2023-10-10T19:27:24.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.581+0000 7f3467ac2700 1 --2- 172.21.15.175:0/153278485 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f34600c3ca0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.581+0000 7f3467ac2700 1 --2- 172.21.15.175:0/153278485 >> v2:172.21.15.191:3300/0 conn(0x7f34600c41e0 0x7f346014cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.581+0000 7f3467ac2700 1 -- 172.21.15.175:0/153278485 >> 172.21.15.175:0/153278485 conn(0x7f3460068280 msgr2=0x7f346006af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:24.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.581+0000 7f3467ac2700 1 -- 172.21.15.175:0/153278485 shutdown_connections 2023-10-10T19:27:24.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.581+0000 7f3467ac2700 1 -- 172.21.15.175:0/153278485 wait complete. 2023-10-10T19:27:24.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.582+0000 7f3467ac2700 1 Processor -- start 2023-10-10T19:27:24.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.582+0000 7f3467ac2700 1 -- start start 2023-10-10T19:27:24.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3467ac2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f346006b640 0x7f346015aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:24.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3467ac2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f3460151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:24.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3467ac2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34600c41e0 0x7f34601668a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:24.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3467ac2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460157230 con 0x7f34600c3330 2023-10-10T19:27:24.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3467ac2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460150c40 con 0x7f34600c41e0 2023-10-10T19:27:24.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3467ac2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34601479b0 con 0x7f346006b640 2023-10-10T19:27:24.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f3460151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:24.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f3460151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33550/0 (socket says 172.21.15.175:33550) 2023-10-10T19:27:24.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 -- 172.21.15.175:0/1144138333 learned_addr learned my addr 172.21.15.175:0/1144138333 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:24.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346605f700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34600c41e0 0x7f34601668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:24.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 -- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f346006b640 msgr2=0x7f346015aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:24.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f346006b640 0x7f346015aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 -- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34600c41e0 msgr2=0x7f34601668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:24.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34600c41e0 0x7f34601668a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 -- 172.21.15.175:0/1144138333 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c009480 con 0x7f34600c3330 2023-10-10T19:27:24.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f346505d700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f3460151640 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f345000d860 tx=0x7f3450008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:24.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3455ffb700 1 -- 172.21.15.175:0/1144138333 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34500125f0 con 0x7f34600c3330 2023-10-10T19:27:24.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3455ffb700 1 -- 172.21.15.175:0/1144138333 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3450012750 con 0x7f34600c3330 2023-10-10T19:27:24.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34601511b0 con 0x7f34600c3330 2023-10-10T19:27:24.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.583+0000 7f3455ffb700 1 -- 172.21.15.175:0/1144138333 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450012ad0 con 0x7f34600c3330 2023-10-10T19:27:24.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.584+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3460152730 con 0x7f34600c3330 2023-10-10T19:27:24.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.584+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3460151490 con 0x7f34600c3330 2023-10-10T19:27:24.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.585+0000 7f3455ffb700 1 -- 172.21.15.175:0/1144138333 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3450014430 con 0x7f34600c3330 2023-10-10T19:27:24.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.585+0000 7f3455ffb700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3444066f60 0x7f3444069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:24.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.585+0000 7f3455ffb700 1 -- 172.21.15.175:0/1144138333 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f3450085060 con 0x7f34600c3330 2023-10-10T19:27:24.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.585+0000 7f346585e700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3444066f60 0x7f3444069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:24.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.586+0000 7f346585e700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3444066f60 0x7f3444069410 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f345c009260 tx=0x7f345c016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:24.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.590+0000 7f3455ffb700 1 -- 172.21.15.175:0/1144138333 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3450051af0 con 0x7f34600c3330 2023-10-10T19:27:24.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.812+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34600642b0 con 0x7f34600c3330 2023-10-10T19:27:24.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.812+0000 7f3455ffb700 1 -- 172.21.15.175:0/1144138333 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7f3450055110 con 0x7f34600c3330 2023-10-10T19:27:24.813 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:24.813 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:24.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3444066f60 msgr2=0x7f3444069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:24.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3444066f60 0x7f3444069410 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f345c009260 tx=0x7f345c016000 comp rx=0 tx=0).stop 2023-10-10T19:27:24.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 msgr2=0x7f3460151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:24.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f3460151640 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f345000d860 tx=0x7f3450008040 comp rx=0 tx=0).stop 2023-10-10T19:27:24.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 shutdown_connections 2023-10-10T19:27:24.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3444066f60 0x7f3444069410 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34600c41e0 0x7f34601668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f346006b640 0x7f346015aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 --2- 172.21.15.175:0/1144138333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f34600c3330 0x7f3460151640 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:24.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 >> 172.21.15.175:0/1144138333 conn(0x7f3460068280 msgr2=0x7f34600c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:24.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 shutdown_connections 2023-10-10T19:27:24.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:24.814+0000 7f3467ac2700 1 -- 172.21.15.175:0/1144138333 wait complete. 2023-10-10T19:27:24.818 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:24.827 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:27:26.828 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:26.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.953+0000 7f6b08be6700 1 Processor -- start 2023-10-10T19:27:26.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.953+0000 7f6b08be6700 1 -- start start 2023-10-10T19:27:26.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.954+0000 7f6b08be6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6b040c4f20 0x7f6b040c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:26.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.954+0000 7f6b08be6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b040c6fb0 con 0x7f6b040c12d0 2023-10-10T19:27:26.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.954+0000 7f6b08be6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b04063940 con 0x7f6b040c5850 2023-10-10T19:27:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.954+0000 7f6b08be6700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b04063ea0 con 0x7f6b040c4f20 2023-10-10T19:27:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b01d9b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6b040c12d0 0x7f6b040c7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50528/0 (socket says 172.21.15.175:50528) 2023-10-10T19:27:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b01d9b700 1 -- 172.21.15.175:0/2859109741 learned_addr learned my addr 172.21.15.175:0/2859109741 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:27:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b02d9d700 1 --2- 172.21.15.175:0/2859109741 >> v2:172.21.15.191:3300/0 conn(0x7f6b040c4f20 0x7f6b040c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299884436 0 0) 0x7f6b040c6fb0 con 0x7f6b040c12d0 2023-10-10T19:27:26.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af0003570 con 0x7f6b040c12d0 2023-10-10T19:27:26.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2073980087 0 0) 0x7f6af0003570 con 0x7f6b040c12d0 2023-10-10T19:27:26.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 >> v2:172.21.15.191:3300/0 conn(0x7f6b040c4f20 msgr2=0x7f6b040c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:26.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 --2- 172.21.15.175:0/2859109741 >> v2:172.21.15.191:3300/0 conn(0x7f6b040c4f20 0x7f6b040c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:26.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 >> v1:172.21.15.191:6789/0 conn(0x7f6b040c5850 legacy=0x7f6b04156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:26.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b04063bc0 con 0x7f6b040c12d0 2023-10-10T19:27:26.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f6aec001470 con 0x7f6b040c12d0 2023-10-10T19:27:26.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 >> v1:172.21.15.175:6789/0 conn(0x7f6b040c12d0 legacy=0x7f6b040c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:26.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6af0003a60 0x7f6af0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:26.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af0004810 0x7f6af0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:26.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6af00051c0 0x7f6af000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:26.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b04063ea0 con 0x7f6af00051c0 2023-10-10T19:27:26.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af000c0d0 con 0x7f6af0004810 2023-10-10T19:27:26.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af000c290 con 0x7f6af0003a60 2023-10-10T19:27:26.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b02d9d700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6af00051c0 0x7f6af000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:26.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b01d9b700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af0004810 0x7f6af0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:26.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.955+0000 7f6b0259c700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6af0003a60 0x7f6af0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:26.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b02d9d700 1 -- 172.21.15.175:0/2859109741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6af0003a60 msgr2=0x7f6af0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:26.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b02d9d700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6af0003a60 0x7f6af0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:26.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b02d9d700 1 -- 172.21.15.175:0/2859109741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af0004810 msgr2=0x7f6af0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:26.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b02d9d700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af0004810 0x7f6af0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:26.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b02d9d700 1 -- 172.21.15.175:0/2859109741 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0003570 con 0x7f6af00051c0 2023-10-10T19:27:26.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b02d9d700 1 -- 172.21.15.175:0/2859109741 reap_dead start 2023-10-10T19:27:26.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b02d9d700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6af00051c0 0x7f6af000ba60 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009c00 tx=0x7f6afc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed913197f88005a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:26.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc00e710 con 0x7f6af00051c0 2023-10-10T19:27:26.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc00e890 con 0x7f6af00051c0 2023-10-10T19:27:26.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b0159a700 1 -- 172.21.15.175:0/2859109741 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc00ec50 con 0x7f6af00051c0 2023-10-10T19:27:26.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b08be6700 1 -- 172.21.15.175:0/2859109741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6af00051c0 msgr2=0x7f6af000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:26.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.956+0000 7f6b08be6700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6af00051c0 0x7f6af000ba60 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009c00 tx=0x7f6afc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:27:26.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.957+0000 7f6b08be6700 1 -- 172.21.15.175:0/2859109741 shutdown_connections 2023-10-10T19:27:26.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.957+0000 7f6b08be6700 1 --2- 172.21.15.175:0/2859109741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6af00051c0 0x7f6af000ba60 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:26.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.957+0000 7f6b08be6700 1 -- 172.21.15.175:0/2859109741 >> 172.21.15.175:0/2859109741 conn(0x7f6b0406dc70 msgr2=0x7f6b04063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:26.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.957+0000 7f6b08be6700 1 -- 172.21.15.175:0/2859109741 shutdown_connections 2023-10-10T19:27:26.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.957+0000 7f6b08be6700 1 -- 172.21.15.175:0/2859109741 wait complete. 2023-10-10T19:27:26.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b08be6700 1 Processor -- start 2023-10-10T19:27:26.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b08be6700 1 -- start start 2023-10-10T19:27:26.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b08be6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af00051c0 0x7f6b040ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:26.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b08be6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b040c34d0 0x7f6b040c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:26.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b08be6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b041665a0 0x7f6b040bf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:26.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b08be6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040cb500 con 0x7f6b040c34d0 2023-10-10T19:27:26.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b08be6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040cacf0 con 0x7f6af00051c0 2023-10-10T19:27:26.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b08be6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040c8d90 con 0x7f6b041665a0 2023-10-10T19:27:26.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b01d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b040c34d0 0x7f6b040c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:26.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b0259c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af00051c0 0x7f6b040ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:26.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b0259c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af00051c0 0x7f6b040ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:39542/0 (socket says 172.21.15.175:39542) 2023-10-10T19:27:26.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b01d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b040c34d0 0x7f6b040c9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33570/0 (socket says 172.21.15.175:33570) 2023-10-10T19:27:26.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b02d9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b041665a0 0x7f6b040bf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:26.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.958+0000 7f6b01d9b700 1 -- 172.21.15.175:0/1658817390 learned_addr learned my addr 172.21.15.175:0/1658817390 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:26.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b01d9b700 1 -- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b041665a0 msgr2=0x7f6b040bf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:26.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b01d9b700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b041665a0 0x7f6b040bf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:26.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b01d9b700 1 -- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af00051c0 msgr2=0x7f6b040ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:26.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b01d9b700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af00051c0 0x7f6b040ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:26.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b01d9b700 1 -- 172.21.15.175:0/1658817390 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b04166a10 con 0x7f6b040c34d0 2023-10-10T19:27:26.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b0259c700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af00051c0 0x7f6b040ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:27:26.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b01d9b700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b040c34d0 0x7f6b040c9220 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0073f0 tx=0x7f6aec001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:26.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6afa7fc700 1 -- 172.21.15.175:0/1658817390 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00f960 con 0x7f6b040c34d0 2023-10-10T19:27:26.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6afa7fc700 1 -- 172.21.15.175:0/1658817390 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec00fae0 con 0x7f6b040c34d0 2023-10-10T19:27:26.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6afa7fc700 1 -- 172.21.15.175:0/1658817390 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00fea0 con 0x7f6b040c34d0 2023-10-10T19:27:26.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b040caf40 con 0x7f6b040c34d0 2023-10-10T19:27:26.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.959+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b040c8740 con 0x7f6b040c34d0 2023-10-10T19:27:26.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.960+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b040c9760 con 0x7f6b040c34d0 2023-10-10T19:27:26.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.960+0000 7f6afa7fc700 1 -- 172.21.15.175:0/1658817390 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6aec0135c0 con 0x7f6b040c34d0 2023-10-10T19:27:26.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.961+0000 7f6afa7fc700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ae4066f30 0x7f6ae40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:26.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.961+0000 7f6afa7fc700 1 -- 172.21.15.175:0/1658817390 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 7713+0+0 (secure 0 0 0) 0x7f6aec0845f0 con 0x7f6b040c34d0 2023-10-10T19:27:26.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.961+0000 7f6b0259c700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ae4066f30 0x7f6ae40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:26.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.961+0000 7f6b0259c700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ae4066f30 0x7f6ae40693e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6af4001750 tx=0x7f6af4001640 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:26.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:26.965+0000 7f6afa7fc700 1 -- 172.21.15.175:0/1658817390 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6aec051140 con 0x7f6b040c34d0 2023-10-10T19:27:27.154 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:27:27.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:27:27.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.193+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b040c85a0 con 0x7f6b040c34d0 2023-10-10T19:27:27.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.194+0000 7f6afa7fc700 1 -- 172.21.15.175:0/1658817390 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) v1 ==== 95+0+6191 (secure 0 0 0) 0x7f6b040c85a0 con 0x7f6b040c34d0 2023-10-10T19:27:27.195 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:27.195 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":305,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":302}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:10.009463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":130,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":7401,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7401":{"gid":7401,"name":"a","rank":1,"incarnation":244,"state":"up:active","state_seq":117,"addr":"172.21.15.175:6834/3457422611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3457422611},{"type":"v1","addr":"172.21.15.175:6834","nonce":3457422611}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:27.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ae4066f30 msgr2=0x7f6ae40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:27.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ae4066f30 0x7f6ae40693e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6af4001750 tx=0x7f6af4001640 comp rx=0 tx=0).stop 2023-10-10T19:27:27.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b040c34d0 msgr2=0x7f6b040c9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:27.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b040c34d0 0x7f6b040c9220 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0073f0 tx=0x7f6aec001e50 comp rx=0 tx=0).stop 2023-10-10T19:27:27.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 shutdown_connections 2023-10-10T19:27:27.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ae4066f30 0x7f6ae40693e0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:27.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b041665a0 0x7f6b040bf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:27.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6af00051c0 0x7f6b040ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:27.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 --2- 172.21.15.175:0/1658817390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6b040c34d0 0x7f6b040c9220 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:27.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 >> 172.21.15.175:0/1658817390 conn(0x7f6b0406dc70 msgr2=0x7f6b040c7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:27.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 shutdown_connections 2023-10-10T19:27:27.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:27.196+0000 7f6b08be6700 1 -- 172.21.15.175:0/1658817390 wait complete. 2023-10-10T19:27:27.199 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 305 2023-10-10T19:27:27.209 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-10T19:27:29.210 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:29.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.335+0000 7fc721f1a700 1 Processor -- start 2023-10-10T19:27:29.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.335+0000 7fc721f1a700 1 -- start start 2023-10-10T19:27:29.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc721f1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b2390 0x7fc71c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc721f1a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc71c0aff60 0x7fc71c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc721f1a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c1574b0 con 0x7fc71c0b1fc0 2023-10-10T19:27:29.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc721f1a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71c150770 con 0x7fc71c0aff60 2023-10-10T19:27:29.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc721f1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c0afad0 con 0x7fc71c0b2390 2023-10-10T19:27:29.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc71affd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b2390 0x7fc71c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc71bfff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc71c0aff60 0x7fc71c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.339 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc71affd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b2390 0x7fc71c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:39110/0 (socket says 172.21.15.175:39110) 2023-10-10T19:27:29.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.336+0000 7fc71affd700 1 -- 172.21.15.175:0/3300223387 learned_addr learned my addr 172.21.15.175:0/3300223387 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:29.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.337+0000 7fc71affd700 1 -- 172.21.15.175:0/3300223387 >> v2:172.21.15.191:3300/0 conn(0x7fc71c0aff60 msgr2=0x7fc71c0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:29.341 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.337+0000 7fc71affd700 1 --2- 172.21.15.175:0/3300223387 >> v2:172.21.15.191:3300/0 conn(0x7fc71c0aff60 0x7fc71c0b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.337+0000 7fc71affd700 1 -- 172.21.15.175:0/3300223387 >> v1:172.21.15.191:6789/0 conn(0x7fc71c0b1fc0 legacy=0x7fc71c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.337+0000 7fc71affd700 1 -- 172.21.15.175:0/3300223387 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c0af4c0 con 0x7fc71c0b2390 2023-10-10T19:27:29.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.337+0000 7fc71affd700 1 --2- 172.21.15.175:0/3300223387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b2390 0x7fc71c155fb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc7100099e0 tx=0x7fc710009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c1f02e7bf7828d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:29.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.337+0000 7fc71a7fc700 1 -- 172.21.15.175:0/3300223387 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71000b8b0 con 0x7fc71c0b2390 2023-10-10T19:27:29.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.337+0000 7fc71a7fc700 1 -- 172.21.15.175:0/3300223387 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71000ba10 con 0x7fc71c0b2390 2023-10-10T19:27:29.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.337+0000 7fc71a7fc700 1 -- 172.21.15.175:0/3300223387 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71000bdb0 con 0x7fc71c0b2390 2023-10-10T19:27:29.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.338+0000 7fc721f1a700 1 -- 172.21.15.175:0/3300223387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b2390 msgr2=0x7fc71c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.338+0000 7fc721f1a700 1 --2- 172.21.15.175:0/3300223387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b2390 0x7fc71c155fb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc7100099e0 tx=0x7fc710009250 comp rx=0 tx=0).stop 2023-10-10T19:27:29.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.338+0000 7fc721f1a700 1 -- 172.21.15.175:0/3300223387 shutdown_connections 2023-10-10T19:27:29.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.338+0000 7fc721f1a700 1 --2- 172.21.15.175:0/3300223387 >> v2:172.21.15.191:3300/0 conn(0x7fc71c0aff60 0x7fc71c0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.338+0000 7fc721f1a700 1 --2- 172.21.15.175:0/3300223387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b2390 0x7fc71c155fb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.338+0000 7fc721f1a700 1 -- 172.21.15.175:0/3300223387 >> 172.21.15.175:0/3300223387 conn(0x7fc71c063230 msgr2=0x7fc71c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:29.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.338+0000 7fc721f1a700 1 -- 172.21.15.175:0/3300223387 shutdown_connections 2023-10-10T19:27:29.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.338+0000 7fc721f1a700 1 -- 172.21.15.175:0/3300223387 wait complete. 2023-10-10T19:27:29.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.339+0000 7fc721f1a700 1 Processor -- start 2023-10-10T19:27:29.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.339+0000 7fc721f1a700 1 -- start start 2023-10-10T19:27:29.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc721f1a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc71c0aff60 0x7fc71c153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc721f1a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b1fc0 0x7fc71c15bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc721f1a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc71c0b2390 0x7fc71c14dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc721f1a700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c154720 con 0x7fc71c0b2390 2023-10-10T19:27:29.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc721f1a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c166430 con 0x7fc71c0aff60 2023-10-10T19:27:29.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc721f1a700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c155590 con 0x7fc71c0b1fc0 2023-10-10T19:27:29.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc71c0b2390 0x7fc71c14dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc71c0b2390 0x7fc71c14dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33572/0 (socket says 172.21.15.175:33572) 2023-10-10T19:27:29.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 -- 172.21.15.175:0/2825313249 learned_addr learned my addr 172.21.15.175:0/2825313249 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:29.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71affd700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b1fc0 0x7fc71c15bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71b7fe700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc71c0aff60 0x7fc71c153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 -- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b1fc0 msgr2=0x7fc71c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b1fc0 0x7fc71c15bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 -- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc71c0aff60 msgr2=0x7fc71c153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc71c0aff60 0x7fc71c153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 -- 172.21.15.175:0/2825313249 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7100094f0 con 0x7fc71c0b2390 2023-10-10T19:27:29.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71affd700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b1fc0 0x7fc71c15bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:27:29.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc71bfff700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc71c0b2390 0x7fc71c14dba0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fc71400eab0 tx=0x7fc7140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:29.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc7037fe700 1 -- 172.21.15.175:0/2825313249 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71400c850 con 0x7fc71c0b2390 2023-10-10T19:27:29.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.340+0000 7fc7037fe700 1 -- 172.21.15.175:0/2825313249 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71400c9d0 con 0x7fc71c0b2390 2023-10-10T19:27:29.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.341+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c166660 con 0x7fc71c0b2390 2023-10-10T19:27:29.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.341+0000 7fc7037fe700 1 -- 172.21.15.175:0/2825313249 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71400cd90 con 0x7fc71c0b2390 2023-10-10T19:27:29.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.341+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c166be0 con 0x7fc71c0b2390 2023-10-10T19:27:29.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.342+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c155760 con 0x7fc71c0b2390 2023-10-10T19:27:29.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.342+0000 7fc7037fe700 1 -- 172.21.15.175:0/2825313249 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc714013680 con 0x7fc71c0b2390 2023-10-10T19:27:29.353 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.342+0000 7fc7037fe700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc708066f30 0x7fc7080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.342+0000 7fc7037fe700 1 -- 172.21.15.175:0/2825313249 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7fc714085310 con 0x7fc71c0b2390 2023-10-10T19:27:29.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.342+0000 7fc71b7fe700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc708066f30 0x7fc7080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.343+0000 7fc71b7fe700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc708066f30 0x7fc7080693e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc70c007fd0 tx=0x7fc70c0016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:29.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.346+0000 7fc7037fe700 1 -- 172.21.15.175:0/2825313249 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc71c155760 con 0x7fc71c0b2390 2023-10-10T19:27:29.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.573+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc71c155760 con 0x7fc71c0b2390 2023-10-10T19:27:29.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.573+0000 7fc7037fe700 1 -- 172.21.15.175:0/2825313249 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 306 v306) v1 ==== 95+0+5429 (secure 0 0 0) 0x7fc71c155760 con 0x7fc71c0b2390 2023-10-10T19:27:29.574 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:29.574 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":306,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":306,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:28.894127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:29.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc708066f30 msgr2=0x7fc7080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc708066f30 0x7fc7080693e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc70c007fd0 tx=0x7fc70c0016f0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc71c0b2390 msgr2=0x7fc71c14dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc71c0b2390 0x7fc71c14dba0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fc71400eab0 tx=0x7fc7140091a0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 shutdown_connections 2023-10-10T19:27:29.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc708066f30 0x7fc7080693e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc71c0b2390 0x7fc71c14dba0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc71c0aff60 0x7fc71c153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 --2- 172.21.15.175:0/2825313249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc71c0b1fc0 0x7fc71c15bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 >> 172.21.15.175:0/2825313249 conn(0x7fc71c063230 msgr2=0x7fc71c14fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:29.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 shutdown_connections 2023-10-10T19:27:29.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.575+0000 7fc721f1a700 1 -- 172.21.15.175:0/2825313249 wait complete. 2023-10-10T19:27:29.578 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 306 2023-10-10T19:27:29.588 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-10T19:27:29.588 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:27:29.588 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:29.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.703+0000 7f3c48846700 1 Processor -- start 2023-10-10T19:27:29.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.703+0000 7f3c48846700 1 -- start start 2023-10-10T19:27:29.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c48846700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3c4414fd90 0x7f3c44159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c48846700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3c4408a100 0x7f3c440c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c48846700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440cd040 0x7f3c440c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c48846700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c440c5600 con 0x7f3c4408a100 2023-10-10T19:27:29.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c48846700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c44067fa0 con 0x7f3c4414fd90 2023-10-10T19:27:29.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c48846700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44156b70 con 0x7f3c440cd040 2023-10-10T19:27:29.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3c4408a100 0x7f3c440c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3c4408a100 0x7f3c440c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33578/0 (socket says 172.21.15.175:33578) 2023-10-10T19:27:29.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 -- 172.21.15.175:0/1905589194 learned_addr learned my addr 172.21.15.175:0/1905589194 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:29.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c42d9d700 1 --2- 172.21.15.175:0/1905589194 >> v2:172.21.15.191:3300/0 conn(0x7f3c4414fd90 0x7f3c44159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4359e700 1 --2- 172.21.15.175:0/1905589194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440cd040 0x7f3c440c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 -- 172.21.15.175:0/1905589194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440cd040 msgr2=0x7f3c440c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:29.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 --2- 172.21.15.175:0/1905589194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440cd040 0x7f3c440c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 -- 172.21.15.175:0/1905589194 >> v2:172.21.15.191:3300/0 conn(0x7f3c4414fd90 msgr2=0x7f3c44159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:29.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 --2- 172.21.15.175:0/1905589194 >> v2:172.21.15.191:3300/0 conn(0x7f3c4414fd90 0x7f3c44159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 -- 172.21.15.175:0/1905589194 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c44150160 con 0x7f3c4408a100 2023-10-10T19:27:29.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c4259c700 1 --2- 172.21.15.175:0/1905589194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c4408a100 0x7f3c440c5eb0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f3c44065080 tx=0x7f3c38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a212057e1207c3db server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:29.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.704+0000 7f3c41d9b700 1 -- 172.21.15.175:0/1905589194 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3800b640 con 0x7f3c4408a100 2023-10-10T19:27:29.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c41d9b700 1 -- 172.21.15.175:0/1905589194 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3800b7a0 con 0x7f3c4408a100 2023-10-10T19:27:29.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c41d9b700 1 -- 172.21.15.175:0/1905589194 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3800bb60 con 0x7f3c4408a100 2023-10-10T19:27:29.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 -- 172.21.15.175:0/1905589194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c4408a100 msgr2=0x7f3c440c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 --2- 172.21.15.175:0/1905589194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c4408a100 0x7f3c440c5eb0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f3c44065080 tx=0x7f3c38009250 comp rx=0 tx=0).stop 2023-10-10T19:27:29.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 -- 172.21.15.175:0/1905589194 shutdown_connections 2023-10-10T19:27:29.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 --2- 172.21.15.175:0/1905589194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440cd040 0x7f3c440c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 --2- 172.21.15.175:0/1905589194 >> v2:172.21.15.191:3300/0 conn(0x7f3c4414fd90 0x7f3c44159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 --2- 172.21.15.175:0/1905589194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c4408a100 0x7f3c440c5eb0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 -- 172.21.15.175:0/1905589194 >> 172.21.15.175:0/1905589194 conn(0x7f3c44063d00 msgr2=0x7f3c440cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:29.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 -- 172.21.15.175:0/1905589194 shutdown_connections 2023-10-10T19:27:29.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.705+0000 7f3c48846700 1 -- 172.21.15.175:0/1905589194 wait complete. 2023-10-10T19:27:29.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.706+0000 7f3c48846700 1 Processor -- start 2023-10-10T19:27:29.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.706+0000 7f3c48846700 1 -- start start 2023-10-10T19:27:29.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c48846700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c4408a100 0x7f3c44168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c48846700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c440cd040 0x7f3c44163e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c48846700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c4414fd90 0x7f3c440b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c48846700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44168a10 con 0x7f3c440cd040 2023-10-10T19:27:29.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c48846700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44150c70 con 0x7f3c4408a100 2023-10-10T19:27:29.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c48846700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c441680e0 con 0x7f3c4414fd90 2023-10-10T19:27:29.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c440cd040 0x7f3c44163e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c42d9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c4408a100 0x7f3c44168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4259c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c440cd040 0x7f3c44163e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33592/0 (socket says 172.21.15.175:33592) 2023-10-10T19:27:29.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4259c700 1 -- 172.21.15.175:0/3370827446 learned_addr learned my addr 172.21.15.175:0/3370827446 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:29.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4359e700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c4414fd90 0x7f3c440b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4259c700 1 -- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c4414fd90 msgr2=0x7f3c440b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4259c700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c4414fd90 0x7f3c440b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4259c700 1 -- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c4408a100 msgr2=0x7f3c44168ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4259c700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c4408a100 0x7f3c44168ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c4259c700 1 -- 172.21.15.175:0/3370827446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c380094f0 con 0x7f3c440cd040 2023-10-10T19:27:29.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.707+0000 7f3c42d9d700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c4408a100 0x7f3c44168ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:27:29.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.708+0000 7f3c4259c700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c440cd040 0x7f3c44163e80 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f3c44064c20 tx=0x7f3c38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:29.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.708+0000 7f3c2affd700 1 -- 172.21.15.175:0/3370827446 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3800b910 con 0x7f3c440cd040 2023-10-10T19:27:29.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.708+0000 7f3c2affd700 1 -- 172.21.15.175:0/3370827446 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3800ba70 con 0x7f3c440cd040 2023-10-10T19:27:29.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.708+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c441683c0 con 0x7f3c440cd040 2023-10-10T19:27:29.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.708+0000 7f3c2affd700 1 -- 172.21.15.175:0/3370827446 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38011630 con 0x7f3c440cd040 2023-10-10T19:27:29.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.708+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c440b2190 con 0x7f3c440cd040 2023-10-10T19:27:29.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.709+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c44067570 con 0x7f3c440cd040 2023-10-10T19:27:29.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.709+0000 7f3c2affd700 1 -- 172.21.15.175:0/3370827446 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3c380119a0 con 0x7f3c440cd040 2023-10-10T19:27:29.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.709+0000 7f3c2affd700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c30066f30 0x7f3c300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:29.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.710+0000 7f3c2affd700 1 -- 172.21.15.175:0/3370827446 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f3c38086460 con 0x7f3c440cd040 2023-10-10T19:27:29.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.710+0000 7f3c42d9d700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c30066f30 0x7f3c300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:29.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.710+0000 7f3c42d9d700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c30066f30 0x7f3c300693e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3c340113e0 tx=0x7f3c340091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:29.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.714+0000 7f3c2affd700 1 -- 172.21.15.175:0/3370827446 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c38052ea0 con 0x7f3c440cd040 2023-10-10T19:27:29.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.938+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c44155f80 con 0x7f3c440cd040 2023-10-10T19:27:29.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.938+0000 7f3c2affd700 1 -- 172.21.15.175:0/3370827446 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 306 v306) v1 ==== 95+0+5429 (secure 0 0 0) 0x7f3c380564c0 con 0x7f3c440cd040 2023-10-10T19:27:29.939 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:29.939 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":306,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":306,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:28.894127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c30066f30 msgr2=0x7f3c300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c30066f30 0x7f3c300693e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3c340113e0 tx=0x7f3c340091a0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c440cd040 msgr2=0x7f3c44163e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:29.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c440cd040 0x7f3c44163e80 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f3c44064c20 tx=0x7f3c38017000 comp rx=0 tx=0).stop 2023-10-10T19:27:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 shutdown_connections 2023-10-10T19:27:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3c30066f30 0x7f3c300693e0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c4414fd90 0x7f3c440b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c4408a100 0x7f3c44168ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 --2- 172.21.15.175:0/3370827446 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3c440cd040 0x7f3c44163e80 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 >> 172.21.15.175:0/3370827446 conn(0x7f3c44063d00 msgr2=0x7f3c4414ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 shutdown_connections 2023-10-10T19:27:29.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:29.940+0000 7f3c48846700 1 -- 172.21.15.175:0/3370827446 wait complete. 2023-10-10T19:27:29.944 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 306 2023-10-10T19:27:29.956 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 306, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 306, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6893": { "addr": "172.21.15.191:6837/840318079", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 840318079, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 840318079, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6893, "incarnation": 306, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_7057": { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 304, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 139 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 131, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:27:28.894127+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 6893, "mds_2": 8049, "mds_3": 7771, "mds_4": 7057 } } } ], "standbys": [] } 2023-10-10T19:27:31.957 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:32.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.085+0000 7fd218a83700 1 Processor -- start 2023-10-10T19:27:32.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.085+0000 7fd218a83700 1 -- start start 2023-10-10T19:27:32.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd218a83700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd21414fd90 0x7fd214159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:32.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd218a83700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:32.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd218a83700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd214067fa0 con 0x7fd2140cd040 2023-10-10T19:27:32.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd218a83700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2140c5600 con 0x7fd21414fd90 2023-10-10T19:27:32.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd218a83700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd214156b70 con 0x7fd21408a100 2023-10-10T19:27:32.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd2137fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd21414fd90 0x7fd214159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:32.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd2137fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd21414fd90 0x7fd214159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33596/0 (socket says 172.21.15.175:33596) 2023-10-10T19:27:32.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd2137fe700 1 -- 172.21.15.175:0/1835748897 learned_addr learned my addr 172.21.15.175:0/1835748897 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:32.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd212ffd700 1 --2- 172.21.15.175:0/1835748897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:32.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd2137fe700 1 -- 172.21.15.175:0/1835748897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 msgr2=0x7fd2140c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:32.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd2137fe700 1 --2- 172.21.15.175:0/1835748897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd2137fe700 1 -- 172.21.15.175:0/1835748897 >> v1:172.21.15.175:6789/0 conn(0x7fd2140cd040 legacy=0x7fd2140684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:32.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd2137fe700 1 -- 172.21.15.175:0/1835748897 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd214150160 con 0x7fd21414fd90 2023-10-10T19:27:32.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.086+0000 7fd212ffd700 1 --2- 172.21.15.175:0/1835748897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:27:32.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd2137fe700 1 --2- 172.21.15.175:0/1835748897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd21414fd90 0x7fd214159c30 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fd204009f20 tx=0x7fd204009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ffb308aa89e0ce server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:32.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd2127fc700 1 -- 172.21.15.175:0/1835748897 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20400e070 con 0x7fd21414fd90 2023-10-10T19:27:32.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd2127fc700 1 -- 172.21.15.175:0/1835748897 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20400ba80 con 0x7fd21414fd90 2023-10-10T19:27:32.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd2127fc700 1 -- 172.21.15.175:0/1835748897 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2040092e0 con 0x7fd21414fd90 2023-10-10T19:27:32.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd218a83700 1 -- 172.21.15.175:0/1835748897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd21414fd90 msgr2=0x7fd214159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:32.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd218a83700 1 --2- 172.21.15.175:0/1835748897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd21414fd90 0x7fd214159c30 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fd204009f20 tx=0x7fd204009840 comp rx=0 tx=0).stop 2023-10-10T19:27:32.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd218a83700 1 -- 172.21.15.175:0/1835748897 shutdown_connections 2023-10-10T19:27:32.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd218a83700 1 --2- 172.21.15.175:0/1835748897 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd21414fd90 0x7fd214159c30 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd218a83700 1 --2- 172.21.15.175:0/1835748897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd218a83700 1 -- 172.21.15.175:0/1835748897 >> 172.21.15.175:0/1835748897 conn(0x7fd214063d00 msgr2=0x7fd2140cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:32.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd218a83700 1 -- 172.21.15.175:0/1835748897 shutdown_connections 2023-10-10T19:27:32.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.087+0000 7fd218a83700 1 -- 172.21.15.175:0/1835748897 wait complete. 2023-10-10T19:27:32.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd218a83700 1 Processor -- start 2023-10-10T19:27:32.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd218a83700 1 -- start start 2023-10-10T19:27:32.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd218a83700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:32.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd218a83700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2140cd040 0x7fd214166680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:32.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd218a83700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21414fd90 0x7fd2140c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:32.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd218a83700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2140a8bd0 con 0x7fd2140cd040 2023-10-10T19:27:32.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd218a83700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2140a8150 con 0x7fd21414fd90 2023-10-10T19:27:32.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd218a83700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2141661f0 con 0x7fd21408a100 2023-10-10T19:27:32.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd212ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2140cd040 0x7fd214166680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:32.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd212ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2140cd040 0x7fd214166680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33598/0 (socket says 172.21.15.175:33598) 2023-10-10T19:27:32.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd212ffd700 1 -- 172.21.15.175:0/2118307863 learned_addr learned my addr 172.21.15.175:0/2118307863 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:32.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd2137fe700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140a7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:32.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.089+0000 7fd213fff700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21414fd90 0x7fd2140c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:32.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd212ffd700 1 -- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 msgr2=0x7fd2140a7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:32.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd212ffd700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140a7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd212ffd700 1 -- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21414fd90 msgr2=0x7fd2140c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:32.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd212ffd700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21414fd90 0x7fd2140c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd212ffd700 1 -- 172.21.15.175:0/2118307863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd204009ae0 con 0x7fd2140cd040 2023-10-10T19:27:32.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd212ffd700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2140cd040 0x7fd214166680 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd20800c860 tx=0x7fd208008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:32.102 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd1fb7fe700 1 -- 172.21.15.175:0/2118307863 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208011600 con 0x7fd2140cd040 2023-10-10T19:27:32.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd1fb7fe700 1 -- 172.21.15.175:0/2118307863 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd208011760 con 0x7fd2140cd040 2023-10-10T19:27:32.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2140c9670 con 0x7fd2140cd040 2023-10-10T19:27:32.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd1fb7fe700 1 -- 172.21.15.175:0/2118307863 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208011b20 con 0x7fd2140cd040 2023-10-10T19:27:32.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.090+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2140c9bb0 con 0x7fd2140cd040 2023-10-10T19:27:32.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.091+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd214067570 con 0x7fd2140cd040 2023-10-10T19:27:32.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.091+0000 7fd1fb7fe700 1 -- 172.21.15.175:0/2118307863 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd2080193b0 con 0x7fd2140cd040 2023-10-10T19:27:32.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.092+0000 7fd1fb7fe700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd200066f60 0x7fd200069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:32.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.092+0000 7fd1fb7fe700 1 -- 172.21.15.175:0/2118307863 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7fd208084d70 con 0x7fd2140cd040 2023-10-10T19:27:32.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.092+0000 7fd2137fe700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd200066f60 0x7fd200069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:32.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.092+0000 7fd2137fe700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd200066f60 0x7fd200069410 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd204009880 tx=0x7fd204018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:32.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.096+0000 7fd1fb7fe700 1 -- 172.21.15.175:0/2118307863 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2080517b0 con 0x7fd2140cd040 2023-10-10T19:27:32.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.322+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd214166330 con 0x7fd2140cd040 2023-10-10T19:27:32.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.322+0000 7fd1fb7fe700 1 -- 172.21.15.175:0/2118307863 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 306 v306) v1 ==== 95+0+5429 (secure 0 0 0) 0x7fd20801f090 con 0x7fd2140cd040 2023-10-10T19:27:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:32.324 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":306,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":306,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:28.894127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd200066f60 msgr2=0x7fd200069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd200066f60 0x7fd200069410 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd204009880 tx=0x7fd204018000 comp rx=0 tx=0).stop 2023-10-10T19:27:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2140cd040 msgr2=0x7fd214166680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2140cd040 0x7fd214166680 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd20800c860 tx=0x7fd208008040 comp rx=0 tx=0).stop 2023-10-10T19:27:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 shutdown_connections 2023-10-10T19:27:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd200066f60 0x7fd200069410 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21414fd90 0x7fd2140c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd21408a100 0x7fd2140a7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.324+0000 7fd218a83700 1 --2- 172.21.15.175:0/2118307863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2140cd040 0x7fd214166680 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.325+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 >> 172.21.15.175:0/2118307863 conn(0x7fd214063d00 msgr2=0x7fd2140c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.325+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 shutdown_connections 2023-10-10T19:27:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:32.325+0000 7fd218a83700 1 -- 172.21.15.175:0/2118307863 wait complete. 2023-10-10T19:27:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 306 2023-10-10T19:27:32.337 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:34.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:34.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.454+0000 7f1f68685700 1 Processor -- start 2023-10-10T19:27:34.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.454+0000 7f1f68685700 1 -- start start 2023-10-10T19:27:34.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.455+0000 7f1f68685700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1f60155fb0 0x7f1f60159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:34.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.455+0000 7f1f68685700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f601574b0 con 0x7f1f600b2390 2023-10-10T19:27:34.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.455+0000 7f1f68685700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f600afad0 con 0x7f1f600b1fc0 2023-10-10T19:27:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.455+0000 7f1f68685700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f60150770 con 0x7f1f60155fb0 2023-10-10T19:27:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.455+0000 7f1f65c20700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1f600b2390 0x7f1f600aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:50552/0 (socket says 172.21.15.175:50552) 2023-10-10T19:27:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.455+0000 7f1f65c20700 1 -- 172.21.15.175:0/287688105 learned_addr learned my addr 172.21.15.175:0/287688105 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:27:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.455+0000 7f1f66c22700 1 --2- 172.21.15.175:0/287688105 >> v2:172.21.15.191:3300/0 conn(0x7f1f60155fb0 0x7f1f60159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:34.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f6541f700 1 -- 172.21.15.175:0/287688105 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845974608 0 0) 0x7f1f601574b0 con 0x7f1f600b2390 2023-10-10T19:27:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f6541f700 1 -- 172.21.15.175:0/287688105 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f4c003570 con 0x7f1f600b2390 2023-10-10T19:27:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f66c22700 1 -- 172.21.15.175:0/287688105 >> v1:172.21.15.191:6789/0 conn(0x7f1f600b1fc0 legacy=0x7f1f60068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f66c22700 1 -- 172.21.15.175:0/287688105 >> v1:172.21.15.175:6789/0 conn(0x7f1f600b2390 legacy=0x7f1f600aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f6541f700 1 -- 172.21.15.175:0/287688105 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 323359796 0 0) 0x7f1f4c003570 con 0x7f1f600b2390 2023-10-10T19:27:34.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f66c22700 1 -- 172.21.15.175:0/287688105 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f600af4c0 con 0x7f1f60155fb0 2023-10-10T19:27:34.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f66c22700 1 --2- 172.21.15.175:0/287688105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f60155fb0 0x7f1f60159920 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1f580098e0 tx=0x7f1f580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f81c3cf22b5d863c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:34.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f6541f700 1 -- 172.21.15.175:0/287688105 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50001470 con 0x7f1f60155fb0 2023-10-10T19:27:34.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f6541f700 1 -- 172.21.15.175:0/287688105 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58011420 con 0x7f1f60155fb0 2023-10-10T19:27:34.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.456+0000 7f1f6541f700 1 -- 172.21.15.175:0/287688105 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f580117c0 con 0x7f1f60155fb0 2023-10-10T19:27:34.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.457+0000 7f1f68685700 1 -- 172.21.15.175:0/287688105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f60155fb0 msgr2=0x7f1f60159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:34.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.457+0000 7f1f68685700 1 --2- 172.21.15.175:0/287688105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f60155fb0 0x7f1f60159920 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1f580098e0 tx=0x7f1f580091a0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.457+0000 7f1f68685700 1 -- 172.21.15.175:0/287688105 shutdown_connections 2023-10-10T19:27:34.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.457+0000 7f1f68685700 1 --2- 172.21.15.175:0/287688105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f60155fb0 0x7f1f60159920 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.457+0000 7f1f68685700 1 -- 172.21.15.175:0/287688105 >> 172.21.15.175:0/287688105 conn(0x7f1f60063230 msgr2=0x7f1f60063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:34.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.457+0000 7f1f68685700 1 -- 172.21.15.175:0/287688105 shutdown_connections 2023-10-10T19:27:34.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.457+0000 7f1f68685700 1 -- 172.21.15.175:0/287688105 wait complete. 2023-10-10T19:27:34.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.458+0000 7f1f68685700 1 Processor -- start 2023-10-10T19:27:34.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.458+0000 7f1f68685700 1 -- start start 2023-10-10T19:27:34.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f68685700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f600b1fc0 0x7f1f600ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:34.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f68685700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f600b2390 0x7f1f600ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:34.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f68685700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f60155fb0 0x7f1f600caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f68685700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600ca5a0 con 0x7f1f600b1fc0 2023-10-10T19:27:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f68685700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600c5e30 con 0x7f1f600b2390 2023-10-10T19:27:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f68685700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600c0010 con 0x7f1f60155fb0 2023-10-10T19:27:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f600b1fc0 0x7f1f600ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:34.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f600b1fc0 0x7f1f600ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:33606/0 (socket says 172.21.15.175:33606) 2023-10-10T19:27:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 -- 172.21.15.175:0/1105091292 learned_addr learned my addr 172.21.15.175:0/1105091292 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f65c20700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f600b2390 0x7f1f600ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66c22700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f60155fb0 0x7f1f600caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 -- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f60155fb0 msgr2=0x7f1f600caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:34.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f60155fb0 0x7f1f600caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 -- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f600b2390 msgr2=0x7f1f600ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f600b2390 0x7f1f600ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 -- 172.21.15.175:0/1105091292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f58009420 con 0x7f1f600b1fc0 2023-10-10T19:27:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f66421700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f600b1fc0 0x7f1f600ccec0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00cb40 tx=0x7f1f5c0017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f567fc700 1 -- 172.21.15.175:0/1105091292 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c00ce50 con 0x7f1f600b1fc0 2023-10-10T19:27:34.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f567fc700 1 -- 172.21.15.175:0/1105091292 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5c00a7a0 con 0x7f1f600b1fc0 2023-10-10T19:27:34.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f600c60a0 con 0x7f1f600b1fc0 2023-10-10T19:27:34.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f567fc700 1 -- 172.21.15.175:0/1105091292 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c00ab60 con 0x7f1f600b1fc0 2023-10-10T19:27:34.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.459+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f600c6660 con 0x7f1f600b1fc0 2023-10-10T19:27:34.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.460+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f60067570 con 0x7f1f600b1fc0 2023-10-10T19:27:34.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.461+0000 7f1f567fc700 1 -- 172.21.15.175:0/1105091292 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1f5c003030 con 0x7f1f600b1fc0 2023-10-10T19:27:34.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.461+0000 7f1f567fc700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f44066f30 0x7f1f440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:34.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.461+0000 7f1f567fc700 1 -- 172.21.15.175:0/1105091292 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f1f5c084d30 con 0x7f1f600b1fc0 2023-10-10T19:27:34.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.461+0000 7f1f65c20700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f44066f30 0x7f1f440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:34.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.461+0000 7f1f65c20700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f44066f30 0x7f1f440693e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1f50001ed0 tx=0x7f1f50000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:34.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.465+0000 7f1f567fc700 1 -- 172.21.15.175:0/1105091292 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f5c051770 con 0x7f1f600b1fc0 2023-10-10T19:27:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.693+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f60068da0 con 0x7f1f600b1fc0 2023-10-10T19:27:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.694+0000 7f1f567fc700 1 -- 172.21.15.175:0/1105091292 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 306 v306) v1 ==== 95+0+5429 (secure 0 0 0) 0x7f1f5c054d90 con 0x7f1f600b1fc0 2023-10-10T19:27:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:34.695 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":306,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":306,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:28.894127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f44066f30 msgr2=0x7f1f440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f44066f30 0x7f1f440693e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1f50001ed0 tx=0x7f1f50000bc0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f600b1fc0 msgr2=0x7f1f600ccec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f600b1fc0 0x7f1f600ccec0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00cb40 tx=0x7f1f5c0017b0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 shutdown_connections 2023-10-10T19:27:34.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f44066f30 0x7f1f440693e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f60155fb0 0x7f1f600caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f600b1fc0 0x7f1f600ccec0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 --2- 172.21.15.175:0/1105091292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f600b2390 0x7f1f600ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:34.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 >> 172.21.15.175:0/1105091292 conn(0x7f1f60063230 msgr2=0x7f1f600c7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:34.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 shutdown_connections 2023-10-10T19:27:34.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:34.696+0000 7f1f68685700 1 -- 172.21.15.175:0/1105091292 wait complete. 2023-10-10T19:27:34.699 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 306 2023-10-10T19:27:34.709 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:36.710 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:36.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.839+0000 7f8fe8623700 1 Processor -- start 2023-10-10T19:27:36.841 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.839+0000 7f8fe8623700 1 -- start start 2023-10-10T19:27:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.840+0000 7f8fe8623700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 0x7f8fe0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:36.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.840+0000 7f8fe8623700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8fe014ffc0 0x7f8fe0063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.840+0000 7f8fe8623700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe0156800 con 0x7f8fe00681e0 2023-10-10T19:27:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.840+0000 7f8fe8623700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe00c59e0 con 0x7f8fe014ffc0 2023-10-10T19:27:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.840+0000 7f8fe8623700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe015a2b0 con 0x7f8fe014fbf0 2023-10-10T19:27:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.840+0000 7f8fe5bbe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8fe014ffc0 0x7f8fe0063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:36.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.840+0000 7f8fe5bbe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8fe014ffc0 0x7f8fe0063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47544/0 (socket says 172.21.15.175:47544) 2023-10-10T19:27:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.840+0000 7f8fe5bbe700 1 -- 172.21.15.175:0/4177864694 learned_addr learned my addr 172.21.15.175:0/4177864694 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe63bf700 1 --2- 172.21.15.175:0/4177864694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 0x7f8fe0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe5bbe700 1 -- 172.21.15.175:0/4177864694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 msgr2=0x7f8fe0159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe5bbe700 1 --2- 172.21.15.175:0/4177864694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 0x7f8fe0159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:36.845 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe5bbe700 1 -- 172.21.15.175:0/4177864694 >> v1:172.21.15.191:6789/0 conn(0x7f8fe00681e0 legacy=0x7f8fe00c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe5bbe700 1 -- 172.21.15.175:0/4177864694 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe0063e30 con 0x7f8fe014ffc0 2023-10-10T19:27:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe5bbe700 1 --2- 172.21.15.175:0/4177864694 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0063230 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40099e0 tx=0x7f8fd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbbef76e3ded634 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe53bd700 1 -- 172.21.15.175:0/4177864694 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd400b820 con 0x7f8fe014ffc0 2023-10-10T19:27:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe53bd700 1 -- 172.21.15.175:0/4177864694 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd400b980 con 0x7f8fe014ffc0 2023-10-10T19:27:36.846 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe53bd700 1 -- 172.21.15.175:0/4177864694 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd400bd20 con 0x7f8fe014ffc0 2023-10-10T19:27:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe8623700 1 -- 172.21.15.175:0/4177864694 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 msgr2=0x7f8fe0063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.841+0000 7f8fe8623700 1 --2- 172.21.15.175:0/4177864694 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0063230 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40099e0 tx=0x7f8fd4009250 comp rx=0 tx=0).stop 2023-10-10T19:27:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.842+0000 7f8fe8623700 1 -- 172.21.15.175:0/4177864694 shutdown_connections 2023-10-10T19:27:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.842+0000 7f8fe8623700 1 --2- 172.21.15.175:0/4177864694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 0x7f8fe0159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.842+0000 7f8fe8623700 1 --2- 172.21.15.175:0/4177864694 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0063230 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:36.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.842+0000 7f8fe8623700 1 -- 172.21.15.175:0/4177864694 >> 172.21.15.175:0/4177864694 conn(0x7f8fe00cc760 msgr2=0x7f8fe00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:36.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.842+0000 7f8fe8623700 1 -- 172.21.15.175:0/4177864694 shutdown_connections 2023-10-10T19:27:36.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.842+0000 7f8fe8623700 1 -- 172.21.15.175:0/4177864694 wait complete. 2023-10-10T19:27:36.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.843+0000 7f8fe8623700 1 Processor -- start 2023-10-10T19:27:36.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.843+0000 7f8fe8623700 1 -- start start 2023-10-10T19:27:36.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe8623700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fe00681e0 0x7f8fe0167210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:36.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe8623700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 0x7f8fe00c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:36.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe8623700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe8623700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0151620 con 0x7f8fe014ffc0 2023-10-10T19:27:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe8623700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0151ec0 con 0x7f8fe00681e0 2023-10-10T19:27:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe8623700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01668b0 con 0x7f8fe014fbf0 2023-10-10T19:27:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:36.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0169310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47558/0 (socket says 172.21.15.175:47558) 2023-10-10T19:27:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 -- 172.21.15.175:0/327021660 learned_addr learned my addr 172.21.15.175:0/327021660 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe63bf700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fe00681e0 0x7f8fe0167210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe5bbe700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 0x7f8fe00c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 -- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 msgr2=0x7f8fe00c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:36.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 0x7f8fe00c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 -- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fe00681e0 msgr2=0x7f8fe0167210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fe00681e0 0x7f8fe0167210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 -- 172.21.15.175:0/327021660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40094f0 con 0x7f8fe014ffc0 2023-10-10T19:27:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe6bc0700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0169310 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800c560 tx=0x7f8fd8007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:36.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fde7fc700 1 -- 172.21.15.175:0/327021660 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8011590 con 0x7f8fe014ffc0 2023-10-10T19:27:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fde7fc700 1 -- 172.21.15.175:0/327021660 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd80116f0 con 0x7f8fe014ffc0 2023-10-10T19:27:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe0166b50 con 0x7f8fe014ffc0 2023-10-10T19:27:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.844+0000 7f8fde7fc700 1 -- 172.21.15.175:0/327021660 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8011ab0 con 0x7f8fe014ffc0 2023-10-10T19:27:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.845+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe0151b50 con 0x7f8fe014ffc0 2023-10-10T19:27:36.855 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.845+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe0152030 con 0x7f8fe014ffc0 2023-10-10T19:27:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.846+0000 7f8fde7fc700 1 -- 172.21.15.175:0/327021660 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8fd801b3b0 con 0x7f8fe014ffc0 2023-10-10T19:27:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.846+0000 7f8fde7fc700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fcc066f60 0x7f8fcc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.846+0000 7f8fde7fc700 1 -- 172.21.15.175:0/327021660 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f8fd8084120 con 0x7f8fe014ffc0 2023-10-10T19:27:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.846+0000 7f8fe63bf700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fcc066f60 0x7f8fcc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:36.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.847+0000 7f8fe63bf700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fcc066f60 0x7f8fcc069410 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009fd0 tx=0x7f8fd00098a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:36.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:36.851+0000 7f8fde7fc700 1 -- 172.21.15.175:0/327021660 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8fd8050b60 con 0x7f8fe014ffc0 2023-10-10T19:27:37.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.081+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fe0067570 con 0x7f8fe014ffc0 2023-10-10T19:27:37.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.081+0000 7f8fde7fc700 1 -- 172.21.15.175:0/327021660 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) v1 ==== 95+0+5431 (secure 0 0 0) 0x7f8fd8017070 con 0x7f8fe014ffc0 2023-10-10T19:27:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:37.083 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":307,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:37.076349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:resolve","state_seq":22,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:37.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fcc066f60 msgr2=0x7f8fcc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:37.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fcc066f60 0x7f8fcc069410 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009fd0 tx=0x7f8fd00098a0 comp rx=0 tx=0).stop 2023-10-10T19:27:37.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 msgr2=0x7f8fe0169310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:37.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0169310 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800c560 tx=0x7f8fd8007ef0 comp rx=0 tx=0).stop 2023-10-10T19:27:37.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 shutdown_connections 2023-10-10T19:27:37.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8fcc066f60 0x7f8fcc069410 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:37.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8fe014ffc0 0x7f8fe0169310 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:37.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fe00681e0 0x7f8fe0167210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:37.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 --2- 172.21.15.175:0/327021660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fe014fbf0 0x7f8fe00c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:37.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.083+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 >> 172.21.15.175:0/327021660 conn(0x7f8fe00cc760 msgr2=0x7f8fe0155640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:37.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.084+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 shutdown_connections 2023-10-10T19:27:37.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:37.084+0000 7f8fe8623700 1 -- 172.21.15.175:0/327021660 wait complete. 2023-10-10T19:27:37.088 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 307 2023-10-10T19:27:37.097 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:39.098 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:39.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.223+0000 7f6ac9c61700 1 Processor -- start 2023-10-10T19:27:39.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.223+0000 7f6ac9c61700 1 -- start start 2023-10-10T19:27:39.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac9c61700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ac41573d0 0x7f6ac40c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:39.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac9c61700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6ac40c4f20 0x7f6ac414d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:39.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac9c61700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac4154aa0 con 0x7f6ac41577a0 2023-10-10T19:27:39.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac9c61700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac40bfdf0 con 0x7f6ac41573d0 2023-10-10T19:27:39.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac9c61700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac4063dc0 con 0x7f6ac40c4f20 2023-10-10T19:27:39.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac37fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ac41573d0 0x7f6ac40c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:39.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac37fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ac41573d0 0x7f6ac40c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47574/0 (socket says 172.21.15.175:47574) 2023-10-10T19:27:39.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac37fe700 1 -- 172.21.15.175:0/1818936355 learned_addr learned my addr 172.21.15.175:0/1818936355 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:39.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac3fff700 1 --2- 172.21.15.175:0/1818936355 >> v2:172.21.15.191:3300/0 conn(0x7f6ac40c4f20 0x7f6ac414d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:39.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac37fe700 1 -- 172.21.15.175:0/1818936355 >> v2:172.21.15.191:3300/0 conn(0x7f6ac40c4f20 msgr2=0x7f6ac414d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:39.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac37fe700 1 --2- 172.21.15.175:0/1818936355 >> v2:172.21.15.191:3300/0 conn(0x7f6ac40c4f20 0x7f6ac414d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac37fe700 1 -- 172.21.15.175:0/1818936355 >> v1:172.21.15.175:6789/0 conn(0x7f6ac41577a0 legacy=0x7f6ac40bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:39.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac37fe700 1 -- 172.21.15.175:0/1818936355 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac4157c20 con 0x7f6ac41573d0 2023-10-10T19:27:39.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac37fe700 1 --2- 172.21.15.175:0/1818936355 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c12a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40c6ee0 tx=0x7f6ab8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d49a53366d3dd8d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:39.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac27fc700 1 -- 172.21.15.175:0/1818936355 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab800b860 con 0x7f6ac41573d0 2023-10-10T19:27:39.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac27fc700 1 -- 172.21.15.175:0/1818936355 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab800b9c0 con 0x7f6ac41573d0 2023-10-10T19:27:39.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.224+0000 7f6ac27fc700 1 -- 172.21.15.175:0/1818936355 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab800bd60 con 0x7f6ac41573d0 2023-10-10T19:27:39.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.225+0000 7f6ac9c61700 1 -- 172.21.15.175:0/1818936355 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 msgr2=0x7f6ac40c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:39.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.225+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/1818936355 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c12a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40c6ee0 tx=0x7f6ab8009250 comp rx=0 tx=0).stop 2023-10-10T19:27:39.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.225+0000 7f6ac9c61700 1 -- 172.21.15.175:0/1818936355 shutdown_connections 2023-10-10T19:27:39.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.225+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/1818936355 >> v2:172.21.15.191:3300/0 conn(0x7f6ac40c4f20 0x7f6ac414d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.225+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/1818936355 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c12a0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.225+0000 7f6ac9c61700 1 -- 172.21.15.175:0/1818936355 >> 172.21.15.175:0/1818936355 conn(0x7f6ac406dc70 msgr2=0x7f6ac4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:39.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.225+0000 7f6ac9c61700 1 -- 172.21.15.175:0/1818936355 shutdown_connections 2023-10-10T19:27:39.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.225+0000 7f6ac9c61700 1 -- 172.21.15.175:0/1818936355 wait complete. 2023-10-10T19:27:39.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.226+0000 7f6ac9c61700 1 Processor -- start 2023-10-10T19:27:39.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.226+0000 7f6ac9c61700 1 -- start start 2023-10-10T19:27:39.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac9c61700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ac40c4f20 0x7f6ac41568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:39.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac9c61700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:39.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac9c61700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ac41577a0 0x7f6ac4145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:39.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac9c61700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4156440 con 0x7f6ac41573d0 2023-10-10T19:27:39.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac9c61700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40c25b0 con 0x7f6ac41577a0 2023-10-10T19:27:39.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac9c61700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40c2340 con 0x7f6ac40c4f20 2023-10-10T19:27:39.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:39.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47584/0 (socket says 172.21.15.175:47584) 2023-10-10T19:27:39.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac2ffd700 1 -- 172.21.15.175:0/2802935121 learned_addr learned my addr 172.21.15.175:0/2802935121 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:39.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac37fe700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ac40c4f20 0x7f6ac41568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:39.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac3fff700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ac41577a0 0x7f6ac4145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:39.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac2ffd700 1 -- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ac40c4f20 msgr2=0x7f6ac41568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:39.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac2ffd700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ac40c4f20 0x7f6ac41568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac2ffd700 1 -- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ac41577a0 msgr2=0x7f6ac4145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:39.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac2ffd700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ac41577a0 0x7f6ac4145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.227+0000 7f6ac2ffd700 1 -- 172.21.15.175:0/2802935121 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab80094f0 con 0x7f6ac41573d0 2023-10-10T19:27:39.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.228+0000 7f6ac2ffd700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c3d70 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400c5c0 tx=0x7f6ab400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:39.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.228+0000 7f6aab7fe700 1 -- 172.21.15.175:0/2802935121 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4011590 con 0x7f6ac41573d0 2023-10-10T19:27:39.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.228+0000 7f6aab7fe700 1 -- 172.21.15.175:0/2802935121 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab4011710 con 0x7f6ac41573d0 2023-10-10T19:27:39.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.228+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac415af00 con 0x7f6ac41573d0 2023-10-10T19:27:39.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.228+0000 7f6aab7fe700 1 -- 172.21.15.175:0/2802935121 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4011ad0 con 0x7f6ac41573d0 2023-10-10T19:27:39.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.228+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac4164aa0 con 0x7f6ac41573d0 2023-10-10T19:27:39.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.229+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac4155f30 con 0x7f6ac41573d0 2023-10-10T19:27:39.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.229+0000 7f6aab7fe700 1 -- 172.21.15.175:0/2802935121 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6ab401c440 con 0x7f6ac41573d0 2023-10-10T19:27:39.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.229+0000 7f6aab7fe700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ab0066f30 0x7f6ab00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:39.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.230+0000 7f6aab7fe700 1 -- 172.21.15.175:0/2802935121 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f6ab4084140 con 0x7f6ac41573d0 2023-10-10T19:27:39.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.230+0000 7f6ac37fe700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ab0066f30 0x7f6ab00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:39.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.230+0000 7f6ac37fe700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ab0066f30 0x7f6ab00693e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40c6ee0 tx=0x7f6ab8017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:39.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.234+0000 7f6aab7fe700 1 -- 172.21.15.175:0/2802935121 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ac4155f30 con 0x7f6ac41573d0 2023-10-10T19:27:39.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.458+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ac41565f0 con 0x7f6ac41573d0 2023-10-10T19:27:39.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.459+0000 7f6aab7fe700 1 -- 172.21.15.175:0/2802935121 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) v1 ==== 95+0+5431 (secure 0 0 0) 0x7f6ac41565f0 con 0x7f6ac41573d0 2023-10-10T19:27:39.461 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:39.461 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":307,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:37.076349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:resolve","state_seq":22,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:39.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ab0066f30 msgr2=0x7f6ab00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:39.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ab0066f30 0x7f6ab00693e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40c6ee0 tx=0x7f6ab8017000 comp rx=0 tx=0).stop 2023-10-10T19:27:39.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 msgr2=0x7f6ac40c3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:39.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c3d70 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400c5c0 tx=0x7f6ab400c8d0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 shutdown_connections 2023-10-10T19:27:39.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6ab0066f30 0x7f6ab00693e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ac41577a0 0x7f6ac4145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ac40c4f20 0x7f6ac41568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 --2- 172.21.15.175:0/2802935121 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac41573d0 0x7f6ac40c3d70 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:39.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 >> 172.21.15.175:0/2802935121 conn(0x7f6ac406dc70 msgr2=0x7f6ac40c5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:39.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 shutdown_connections 2023-10-10T19:27:39.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:39.461+0000 7f6ac9c61700 1 -- 172.21.15.175:0/2802935121 wait complete. 2023-10-10T19:27:39.465 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 307 2023-10-10T19:27:39.474 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:41.476 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:41.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.602+0000 7fe69b536700 1 Processor -- start 2023-10-10T19:27:41.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.603+0000 7fe69b536700 1 -- start start 2023-10-10T19:27:41.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.603+0000 7fe69b536700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe6940c3330 0x7fe6940c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:41.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.603+0000 7fe69b536700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c41e0 0x7fe69414cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:41.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.603+0000 7fe69b536700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6941533e0 con 0x7fe69406b640 2023-10-10T19:27:41.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.603+0000 7fe69b536700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69406e130 con 0x7fe6940c3330 2023-10-10T19:27:41.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.603+0000 7fe69b536700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69406a970 con 0x7fe6940c41e0 2023-10-10T19:27:41.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe6992d2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe6940c3330 0x7fe6940c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:41.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe6992d2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe6940c3330 0x7fe6940c3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47586/0 (socket says 172.21.15.175:47586) 2023-10-10T19:27:41.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe6992d2700 1 -- 172.21.15.175:0/1706904439 learned_addr learned my addr 172.21.15.175:0/1706904439 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:41.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe699ad3700 1 --1- 172.21.15.175:0/1706904439 >> v1:172.21.15.175:6789/0 conn(0x7fe69406b640 0x7fe694156590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:38424/0 (socket says 172.21.15.175:38424) 2023-10-10T19:27:41.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe698ad1700 1 --2- 172.21.15.175:0/1706904439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c41e0 0x7fe69414cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:41.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe6992d2700 1 -- 172.21.15.175:0/1706904439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c41e0 msgr2=0x7fe69414cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:41.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe6992d2700 1 --2- 172.21.15.175:0/1706904439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c41e0 0x7fe69414cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe6992d2700 1 -- 172.21.15.175:0/1706904439 >> v1:172.21.15.175:6789/0 conn(0x7fe69406b640 legacy=0x7fe694156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:41.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe6992d2700 1 -- 172.21.15.175:0/1706904439 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69406d520 con 0x7fe6940c3330 2023-10-10T19:27:41.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe6992d2700 1 --2- 172.21.15.175:0/1706904439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c3330 0x7fe6940c3ca0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fe684009f70 tx=0x7fe684009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f193d1b2f90d00ad server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:41.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe693fff700 1 -- 172.21.15.175:0/1706904439 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68400e070 con 0x7fe6940c3330 2023-10-10T19:27:41.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe693fff700 1 -- 172.21.15.175:0/1706904439 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68400bad0 con 0x7fe6940c3330 2023-10-10T19:27:41.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.604+0000 7fe693fff700 1 -- 172.21.15.175:0/1706904439 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6840092e0 con 0x7fe6940c3330 2023-10-10T19:27:41.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.605+0000 7fe69b536700 1 -- 172.21.15.175:0/1706904439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c3330 msgr2=0x7fe6940c3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:41.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.605+0000 7fe69b536700 1 --2- 172.21.15.175:0/1706904439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c3330 0x7fe6940c3ca0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fe684009f70 tx=0x7fe684009840 comp rx=0 tx=0).stop 2023-10-10T19:27:41.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.605+0000 7fe69b536700 1 -- 172.21.15.175:0/1706904439 shutdown_connections 2023-10-10T19:27:41.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.605+0000 7fe69b536700 1 --2- 172.21.15.175:0/1706904439 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c3330 0x7fe6940c3ca0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.605+0000 7fe69b536700 1 --2- 172.21.15.175:0/1706904439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c41e0 0x7fe69414cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.605+0000 7fe69b536700 1 -- 172.21.15.175:0/1706904439 >> 172.21.15.175:0/1706904439 conn(0x7fe694068280 msgr2=0x7fe69406af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:41.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.605+0000 7fe69b536700 1 -- 172.21.15.175:0/1706904439 shutdown_connections 2023-10-10T19:27:41.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.605+0000 7fe69b536700 1 -- 172.21.15.175:0/1706904439 wait complete. 2023-10-10T19:27:41.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.606+0000 7fe69b536700 1 Processor -- start 2023-10-10T19:27:41.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.606+0000 7fe69b536700 1 -- start start 2023-10-10T19:27:41.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe69b536700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe69406b640 0x7fe694150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:41.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe69b536700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c3330 0x7fe6941676e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:41.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe69b536700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c41e0 0x7fe694166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:41.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe69b536700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c18e0 con 0x7fe6940c41e0 2023-10-10T19:27:41.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe69b536700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe694151660 con 0x7fe69406b640 2023-10-10T19:27:41.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe69b536700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe694151900 con 0x7fe6940c3330 2023-10-10T19:27:41.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c41e0 0x7fe694166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:41.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c41e0 0x7fe694166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47596/0 (socket says 172.21.15.175:47596) 2023-10-10T19:27:41.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 -- 172.21.15.175:0/638855315 learned_addr learned my addr 172.21.15.175:0/638855315 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:41.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe698ad1700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c3330 0x7fe6941676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:41.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe6992d2700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe69406b640 0x7fe694150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:41.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 -- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c3330 msgr2=0x7fe6941676e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:41.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c3330 0x7fe6941676e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 -- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe69406b640 msgr2=0x7fe694150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:41.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe69406b640 0x7fe694150e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 -- 172.21.15.175:0/638855315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe684009ae0 con 0x7fe6940c41e0 2023-10-10T19:27:41.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe699ad3700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c41e0 0x7fe694166ad0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c560 tx=0x7fe68c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:41.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe6917fa700 1 -- 172.21.15.175:0/638855315 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0062e0 con 0x7fe6940c41e0 2023-10-10T19:27:41.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe6917fa700 1 -- 172.21.15.175:0/638855315 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c006440 con 0x7fe6940c41e0 2023-10-10T19:27:41.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe694151ba0 con 0x7fe6940c41e0 2023-10-10T19:27:41.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.607+0000 7fe6917fa700 1 -- 172.21.15.175:0/638855315 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c004810 con 0x7fe6940c41e0 2023-10-10T19:27:41.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.608+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe694151e40 con 0x7fe6940c41e0 2023-10-10T19:27:41.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.608+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe694067570 con 0x7fe6940c41e0 2023-10-10T19:27:41.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.609+0000 7fe6917fa700 1 -- 172.21.15.175:0/638855315 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe68c004c00 con 0x7fe6940c41e0 2023-10-10T19:27:41.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.609+0000 7fe6917fa700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe680066f30 0x7fe6800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:41.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.609+0000 7fe6917fa700 1 -- 172.21.15.175:0/638855315 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7fe68c084260 con 0x7fe6940c41e0 2023-10-10T19:27:41.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.609+0000 7fe6992d2700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe680066f30 0x7fe6800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:41.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.610+0000 7fe6992d2700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe680066f30 0x7fe6800693e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe684009880 tx=0x7fe684018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:41.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.613+0000 7fe6917fa700 1 -- 172.21.15.175:0/638855315 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe68c050ca0 con 0x7fe6940c41e0 2023-10-10T19:27:41.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.838+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe694069360 con 0x7fe6940c41e0 2023-10-10T19:27:41.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.838+0000 7fe6917fa700 1 -- 172.21.15.175:0/638855315 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) v1 ==== 95+0+5431 (secure 0 0 0) 0x7fe68c017070 con 0x7fe6940c41e0 2023-10-10T19:27:41.840 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:41.840 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":307,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:37.076349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:resolve","state_seq":22,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:41.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe680066f30 msgr2=0x7fe6800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:41.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe680066f30 0x7fe6800693e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe684009880 tx=0x7fe684018000 comp rx=0 tx=0).stop 2023-10-10T19:27:41.842 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c41e0 msgr2=0x7fe694166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:41.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c41e0 0x7fe694166ad0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c560 tx=0x7fe68c007ea0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 shutdown_connections 2023-10-10T19:27:41.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe680066f30 0x7fe6800693e0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6940c41e0 0x7fe694166ad0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.843 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe69406b640 0x7fe694150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 --2- 172.21.15.175:0/638855315 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe6940c3330 0x7fe6941676e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:41.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.840+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 >> 172.21.15.175:0/638855315 conn(0x7fe694068280 msgr2=0x7fe694069580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:41.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.841+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 shutdown_connections 2023-10-10T19:27:41.844 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:41.841+0000 7fe69b536700 1 -- 172.21.15.175:0/638855315 wait complete. 2023-10-10T19:27:41.845 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 307 2023-10-10T19:27:41.854 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 307, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 307, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6893": { "addr": "172.21.15.191:6837/840318079", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 840318079, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 840318079, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6893, "incarnation": 306, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:resolve", "state_seq": 22 }, "gid_7057": { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 304, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 139 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 131, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:27:37.076349+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 6893, "mds_2": 8049, "mds_3": 7771, "mds_4": 7057 } } } ], "standbys": [] } 2023-10-10T19:27:43.855 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:43.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.982+0000 7f7c09888700 1 Processor -- start 2023-10-10T19:27:43.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.982+0000 7f7c09888700 1 -- start start 2023-10-10T19:27:43.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.982+0000 7f7c09888700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c040b1fc0 0x7f7c04155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:43.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c09888700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c041574b0 con 0x7f7c040aff60 2023-10-10T19:27:43.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c09888700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c040afad0 con 0x7f7c040b2390 2023-10-10T19:27:43.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c09888700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c04150770 con 0x7f7c040b1fc0 2023-10-10T19:27:43.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c02ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c040b1fc0 0x7f7c04155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:43.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c02ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7c040b1fc0 0x7f7c04155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47602/0 (socket says 172.21.15.175:47602) 2023-10-10T19:27:43.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c02ffd700 1 -- 172.21.15.175:0/3957447041 learned_addr learned my addr 172.21.15.175:0/3957447041 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:43.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c02ffd700 1 -- 172.21.15.175:0/3957447041 >> v1:172.21.15.191:6789/0 conn(0x7f7c040b2390 legacy=0x7f7c04068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:43.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c02ffd700 1 -- 172.21.15.175:0/3957447041 >> v1:172.21.15.175:6789/0 conn(0x7f7c040aff60 legacy=0x7f7c04159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:43.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c02ffd700 1 -- 172.21.15.175:0/3957447041 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c040af4c0 con 0x7f7c040b1fc0 2023-10-10T19:27:43.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c02ffd700 1 --2- 172.21.15.175:0/3957447041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b1fc0 0x7f7c04155fb0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009f20 tx=0x7f7bfc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da3f81d795d3115 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:43.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c01ffb700 1 -- 172.21.15.175:0/3957447041 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00e070 con 0x7f7c040b1fc0 2023-10-10T19:27:43.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c01ffb700 1 -- 172.21.15.175:0/3957447041 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc00bad0 con 0x7f7c040b1fc0 2023-10-10T19:27:43.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.983+0000 7f7c01ffb700 1 -- 172.21.15.175:0/3957447041 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0092e0 con 0x7f7c040b1fc0 2023-10-10T19:27:43.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.984+0000 7f7c09888700 1 -- 172.21.15.175:0/3957447041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b1fc0 msgr2=0x7f7c04155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:43.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.984+0000 7f7c09888700 1 --2- 172.21.15.175:0/3957447041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b1fc0 0x7f7c04155fb0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009f20 tx=0x7f7bfc009840 comp rx=0 tx=0).stop 2023-10-10T19:27:43.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.984+0000 7f7c09888700 1 -- 172.21.15.175:0/3957447041 shutdown_connections 2023-10-10T19:27:43.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.984+0000 7f7c09888700 1 --2- 172.21.15.175:0/3957447041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b1fc0 0x7f7c04155fb0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:43.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.984+0000 7f7c09888700 1 -- 172.21.15.175:0/3957447041 >> 172.21.15.175:0/3957447041 conn(0x7f7c04063230 msgr2=0x7f7c04063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:43.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.984+0000 7f7c09888700 1 -- 172.21.15.175:0/3957447041 shutdown_connections 2023-10-10T19:27:43.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.984+0000 7f7c09888700 1 -- 172.21.15.175:0/3957447041 wait complete. 2023-10-10T19:27:43.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.985+0000 7f7c09888700 1 Processor -- start 2023-10-10T19:27:43.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.985+0000 7f7c09888700 1 -- start start 2023-10-10T19:27:43.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c09888700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c040aff60 0x7f7c04166c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:43.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c09888700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c040b1fc0 0x7f7c0414d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:43.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c09888700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b2390 0x7f7c0414ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:43.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c09888700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c041697a0 con 0x7f7c040b2390 2023-10-10T19:27:43.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c09888700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0416a600 con 0x7f7c040aff60 2023-10-10T19:27:43.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c09888700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04156c90 con 0x7f7c040b1fc0 2023-10-10T19:27:43.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b2390 0x7f7c0414ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:43.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b2390 0x7f7c0414ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47614/0 (socket says 172.21.15.175:47614) 2023-10-10T19:27:43.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 -- 172.21.15.175:0/467657308 learned_addr learned my addr 172.21.15.175:0/467657308 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:43.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c02ffd700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c040aff60 0x7f7c04166c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:43.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c027fc700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c040b1fc0 0x7f7c0414d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:43.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 -- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c040b1fc0 msgr2=0x7f7c0414d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:43.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c040b1fc0 0x7f7c0414d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:43.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 -- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c040aff60 msgr2=0x7f7c04166c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:43.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c040aff60 0x7f7c04166c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:43.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 -- 172.21.15.175:0/467657308 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc009ae0 con 0x7f7c040b2390 2023-10-10T19:27:43.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7c037fe700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b2390 0x7f7c0414ba50 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400c590 tx=0x7f7bf4007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:43.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7beaffd700 1 -- 172.21.15.175:0/467657308 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4011590 con 0x7f7c040b2390 2023-10-10T19:27:43.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.986+0000 7f7beaffd700 1 -- 172.21.15.175:0/467657308 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf40116f0 con 0x7f7c040b2390 2023-10-10T19:27:43.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.987+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c041699f0 con 0x7f7c040b2390 2023-10-10T19:27:43.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.987+0000 7f7beaffd700 1 -- 172.21.15.175:0/467657308 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4011ab0 con 0x7f7c040b2390 2023-10-10T19:27:43.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.987+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c04169fb0 con 0x7f7c040b2390 2023-10-10T19:27:43.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.987+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c04067570 con 0x7f7c040b2390 2023-10-10T19:27:43.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.988+0000 7f7beaffd700 1 -- 172.21.15.175:0/467657308 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7bf401b3b0 con 0x7f7c040b2390 2023-10-10T19:27:43.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.988+0000 7f7beaffd700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7bf0066f60 0x7f7bf0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:43.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.988+0000 7f7beaffd700 1 -- 172.21.15.175:0/467657308 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f7bf4084120 con 0x7f7c040b2390 2023-10-10T19:27:43.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.988+0000 7f7c02ffd700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7bf0066f60 0x7f7bf0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:43.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.989+0000 7f7c02ffd700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7bf0066f60 0x7f7bf0069410 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009880 tx=0x7f7bfc017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:43.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:43.993+0000 7f7beaffd700 1 -- 172.21.15.175:0/467657308 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7bf4050b60 con 0x7f7c040b2390 2023-10-10T19:27:44.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.219+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c04068da0 con 0x7f7c040b2390 2023-10-10T19:27:44.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.220+0000 7f7beaffd700 1 -- 172.21.15.175:0/467657308 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+5433 (secure 0 0 0) 0x7f7bf4017070 con 0x7f7c040b2390 2023-10-10T19:27:44.222 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:44.222 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":308,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:43.101259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:44.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7bf0066f60 msgr2=0x7f7bf0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:44.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7bf0066f60 0x7f7bf0069410 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009880 tx=0x7f7bfc017000 comp rx=0 tx=0).stop 2023-10-10T19:27:44.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b2390 msgr2=0x7f7c0414ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:44.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b2390 0x7f7c0414ba50 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400c590 tx=0x7f7bf4007ef0 comp rx=0 tx=0).stop 2023-10-10T19:27:44.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 shutdown_connections 2023-10-10T19:27:44.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7bf0066f60 0x7f7bf0069410 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:44.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7c040b2390 0x7f7c0414ba50 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:44.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c040aff60 0x7f7c04166c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:44.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 --2- 172.21.15.175:0/467657308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7c040b1fc0 0x7f7c0414d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:44.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 >> 172.21.15.175:0/467657308 conn(0x7f7c04063230 msgr2=0x7f7c04157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:44.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 shutdown_connections 2023-10-10T19:27:44.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:44.222+0000 7f7c09888700 1 -- 172.21.15.175:0/467657308 wait complete. 2023-10-10T19:27:44.226 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 308 2023-10-10T19:27:44.235 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:46.237 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:46.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.363+0000 7f0feb169700 1 Processor -- start 2023-10-10T19:27:46.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.363+0000 7f0feb169700 1 -- start start 2023-10-10T19:27:46.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0feb169700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0fe414fd90 0x7f0fe4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:46.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0feb169700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0fe408a100 0x7f0fe40c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:46.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0feb169700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe4067fa0 con 0x7f0fe40cd040 2023-10-10T19:27:46.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0feb169700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe40c5600 con 0x7f0fe408a100 2023-10-10T19:27:46.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0feb169700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe4156b70 con 0x7f0fe414fd90 2023-10-10T19:27:46.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0fe9966700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0fe408a100 0x7f0fe40c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:46.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0fe9966700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0fe408a100 0x7f0fe40c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35542/0 (socket says 172.21.15.175:35542) 2023-10-10T19:27:46.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0fe9966700 1 -- 172.21.15.175:0/3818466728 learned_addr learned my addr 172.21.15.175:0/3818466728 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:46.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0fea167700 1 --2- 172.21.15.175:0/3818466728 >> v2:172.21.15.191:3300/0 conn(0x7f0fe414fd90 0x7f0fe4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:46.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0fe9966700 1 -- 172.21.15.175:0/3818466728 >> v2:172.21.15.191:3300/0 conn(0x7f0fe414fd90 msgr2=0x7f0fe4159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:46.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0fe9966700 1 --2- 172.21.15.175:0/3818466728 >> v2:172.21.15.191:3300/0 conn(0x7f0fe414fd90 0x7f0fe4159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0fe9966700 1 -- 172.21.15.175:0/3818466728 >> v1:172.21.15.191:6789/0 conn(0x7f0fe40cd040 legacy=0x7f0fe40684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:46.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.364+0000 7f0fe9966700 1 -- 172.21.15.175:0/3818466728 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4150160 con 0x7f0fe408a100 2023-10-10T19:27:46.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0fe9966700 1 --2- 172.21.15.175:0/3818466728 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe408a100 0x7f0fe40c5eb0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4065080 tx=0x7f0fd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfbd595d519da7e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:46.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0fe9165700 1 -- 172.21.15.175:0/3818466728 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd400b8b0 con 0x7f0fe408a100 2023-10-10T19:27:46.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0fe9165700 1 -- 172.21.15.175:0/3818466728 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd400ba10 con 0x7f0fe408a100 2023-10-10T19:27:46.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0fe9165700 1 -- 172.21.15.175:0/3818466728 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd400bdb0 con 0x7f0fe408a100 2023-10-10T19:27:46.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0feb169700 1 -- 172.21.15.175:0/3818466728 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe408a100 msgr2=0x7f0fe40c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:46.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0feb169700 1 --2- 172.21.15.175:0/3818466728 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe408a100 0x7f0fe40c5eb0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4065080 tx=0x7f0fd4009250 comp rx=0 tx=0).stop 2023-10-10T19:27:46.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0feb169700 1 -- 172.21.15.175:0/3818466728 shutdown_connections 2023-10-10T19:27:46.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0feb169700 1 --2- 172.21.15.175:0/3818466728 >> v2:172.21.15.191:3300/0 conn(0x7f0fe414fd90 0x7f0fe4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0feb169700 1 --2- 172.21.15.175:0/3818466728 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe408a100 0x7f0fe40c5eb0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0feb169700 1 -- 172.21.15.175:0/3818466728 >> 172.21.15.175:0/3818466728 conn(0x7f0fe4063d00 msgr2=0x7f0fe40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:46.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0feb169700 1 -- 172.21.15.175:0/3818466728 shutdown_connections 2023-10-10T19:27:46.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.365+0000 7f0feb169700 1 -- 172.21.15.175:0/3818466728 wait complete. 2023-10-10T19:27:46.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.366+0000 7f0feb169700 1 Processor -- start 2023-10-10T19:27:46.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.366+0000 7f0feb169700 1 -- start start 2023-10-10T19:27:46.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0feb169700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe408a100 0x7f0fe4149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:46.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0feb169700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe40cd040 0x7f0fe4149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:46.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0feb169700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fe414fd90 0x7f0fe40ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:46.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0feb169700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe4150d60 con 0x7f0fe40cd040 2023-10-10T19:27:46.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0feb169700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41504c0 con 0x7f0fe408a100 2023-10-10T19:27:46.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0feb169700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41669e0 con 0x7f0fe414fd90 2023-10-10T19:27:46.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fe9966700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe40cd040 0x7f0fe4149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:46.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fe9966700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe40cd040 0x7f0fe4149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35558/0 (socket says 172.21.15.175:35558) 2023-10-10T19:27:46.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fe9966700 1 -- 172.21.15.175:0/2356808185 learned_addr learned my addr 172.21.15.175:0/2356808185 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:46.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fea167700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe408a100 0x7f0fe4149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:46.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fea968700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fe414fd90 0x7f0fe40ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:46.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fe9966700 1 -- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fe414fd90 msgr2=0x7f0fe40ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:46.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fe9966700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fe414fd90 0x7f0fe40ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fe9966700 1 -- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe408a100 msgr2=0x7f0fe4149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:46.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.367+0000 7f0fe9966700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe408a100 0x7f0fe4149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.368+0000 7f0fe9966700 1 -- 172.21.15.175:0/2356808185 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40094f0 con 0x7f0fe40cd040 2023-10-10T19:27:46.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.368+0000 7f0fe9966700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe40cd040 0x7f0fe4149e80 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4003080 tx=0x7f0fd4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:46.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.368+0000 7f0fda7fc700 1 -- 172.21.15.175:0/2356808185 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd401b470 con 0x7f0fe40cd040 2023-10-10T19:27:46.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.368+0000 7f0fda7fc700 1 -- 172.21.15.175:0/2356808185 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd401b5d0 con 0x7f0fe40cd040 2023-10-10T19:27:46.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.368+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe4166cc0 con 0x7f0fe40cd040 2023-10-10T19:27:46.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.368+0000 7f0fda7fc700 1 -- 172.21.15.175:0/2356808185 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd401b990 con 0x7f0fe40cd040 2023-10-10T19:27:46.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.368+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe41509f0 con 0x7f0fe40cd040 2023-10-10T19:27:46.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.369+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe4067570 con 0x7f0fe40cd040 2023-10-10T19:27:46.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.369+0000 7f0fda7fc700 1 -- 172.21.15.175:0/2356808185 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0fd401bd00 con 0x7f0fe40cd040 2023-10-10T19:27:46.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.369+0000 7f0fda7fc700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fc8066f30 0x7f0fc80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:46.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.370+0000 7f0fda7fc700 1 -- 172.21.15.175:0/2356808185 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f0fd4086950 con 0x7f0fe40cd040 2023-10-10T19:27:46.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.370+0000 7f0fea167700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fc8066f30 0x7f0fc80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:46.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.370+0000 7f0fea167700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fc8066f30 0x7f0fc80693e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0006fd0 tx=0x7f0fe0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:46.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.374+0000 7f0fda7fc700 1 -- 172.21.15.175:0/2356808185 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0fd4053390 con 0x7f0fe40cd040 2023-10-10T19:27:46.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.598+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fe4155f80 con 0x7f0fe40cd040 2023-10-10T19:27:46.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.599+0000 7f0fda7fc700 1 -- 172.21.15.175:0/2356808185 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+5433 (secure 0 0 0) 0x7f0fd402a090 con 0x7f0fe40cd040 2023-10-10T19:27:46.601 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:46.601 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":308,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:43.101259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:46.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fc8066f30 msgr2=0x7f0fc80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:46.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fc8066f30 0x7f0fc80693e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0006fd0 tx=0x7f0fe0008040 comp rx=0 tx=0).stop 2023-10-10T19:27:46.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe40cd040 msgr2=0x7f0fe4149e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:46.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe40cd040 0x7f0fe4149e80 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4003080 tx=0x7f0fd4016000 comp rx=0 tx=0).stop 2023-10-10T19:27:46.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 shutdown_connections 2023-10-10T19:27:46.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fc8066f30 0x7f0fc80693e0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fe414fd90 0x7f0fe40ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe408a100 0x7f0fe4149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 --2- 172.21.15.175:0/2356808185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fe40cd040 0x7f0fe4149e80 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:46.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 >> 172.21.15.175:0/2356808185 conn(0x7f0fe4063d00 msgr2=0x7f0fe414dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:46.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 shutdown_connections 2023-10-10T19:27:46.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:46.601+0000 7f0feb169700 1 -- 172.21.15.175:0/2356808185 wait complete. 2023-10-10T19:27:46.606 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 308 2023-10-10T19:27:46.615 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:48.616 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:48.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.743+0000 7faf1ef9e700 1 Processor -- start 2023-10-10T19:27:48.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.743+0000 7faf1ef9e700 1 -- start start 2023-10-10T19:27:48.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1ef9e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faf180b1fc0 0x7faf18155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:48.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1ef9e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faf180aff60 0x7faf180b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:48.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1ef9e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf181574b0 con 0x7faf180b2390 2023-10-10T19:27:48.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1ef9e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf18150770 con 0x7faf180aff60 2023-10-10T19:27:48.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1ef9e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf180afad0 con 0x7faf180b1fc0 2023-10-10T19:27:48.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1d53b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faf180aff60 0x7faf180b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:48.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1d53b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faf180aff60 0x7faf180b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35568/0 (socket says 172.21.15.175:35568) 2023-10-10T19:27:48.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1d53b700 1 -- 172.21.15.175:0/3653383811 learned_addr learned my addr 172.21.15.175:0/3653383811 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:48.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1cd3a700 1 --2- 172.21.15.175:0/3653383811 >> v2:172.21.15.191:3300/0 conn(0x7faf180b1fc0 0x7faf18155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:48.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1d53b700 1 -- 172.21.15.175:0/3653383811 >> v2:172.21.15.191:3300/0 conn(0x7faf180b1fc0 msgr2=0x7faf18155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:48.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1d53b700 1 --2- 172.21.15.175:0/3653383811 >> v2:172.21.15.191:3300/0 conn(0x7faf180b1fc0 0x7faf18155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1d53b700 1 -- 172.21.15.175:0/3653383811 >> v1:172.21.15.191:6789/0 conn(0x7faf180b2390 legacy=0x7faf18068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:48.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1d53b700 1 -- 172.21.15.175:0/3653383811 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf180af4c0 con 0x7faf180aff60 2023-10-10T19:27:48.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf1d53b700 1 --2- 172.21.15.175:0/3653383811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180aff60 0x7faf180b03e0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7faf180b28c0 tx=0x7faf10009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad737b203b8ebe2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:48.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf177fe700 1 -- 172.21.15.175:0/3653383811 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1000e070 con 0x7faf180aff60 2023-10-10T19:27:48.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.744+0000 7faf177fe700 1 -- 172.21.15.175:0/3653383811 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1000baa0 con 0x7faf180aff60 2023-10-10T19:27:48.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf177fe700 1 -- 172.21.15.175:0/3653383811 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf100092e0 con 0x7faf180aff60 2023-10-10T19:27:48.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf1ef9e700 1 -- 172.21.15.175:0/3653383811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180aff60 msgr2=0x7faf180b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:48.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/3653383811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180aff60 0x7faf180b03e0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7faf180b28c0 tx=0x7faf10009840 comp rx=0 tx=0).stop 2023-10-10T19:27:48.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf1ef9e700 1 -- 172.21.15.175:0/3653383811 shutdown_connections 2023-10-10T19:27:48.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/3653383811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180aff60 0x7faf180b03e0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/3653383811 >> v2:172.21.15.191:3300/0 conn(0x7faf180b1fc0 0x7faf18155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf1ef9e700 1 -- 172.21.15.175:0/3653383811 >> 172.21.15.175:0/3653383811 conn(0x7faf18063230 msgr2=0x7faf18063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:48.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf1ef9e700 1 -- 172.21.15.175:0/3653383811 shutdown_connections 2023-10-10T19:27:48.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.745+0000 7faf1ef9e700 1 -- 172.21.15.175:0/3653383811 wait complete. 2023-10-10T19:27:48.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.746+0000 7faf1ef9e700 1 Processor -- start 2023-10-10T19:27:48.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.746+0000 7faf1ef9e700 1 -- start start 2023-10-10T19:27:48.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf1ef9e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf180aff60 0x7faf181557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:48.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf1ef9e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180b1fc0 0x7faf18169d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:48.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf1ef9e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf180b2390 0x7faf18153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:48.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf1ef9e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf181698b0 con 0x7faf180b1fc0 2023-10-10T19:27:48.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf1ef9e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf180c4160 con 0x7faf180aff60 2023-10-10T19:27:48.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf1ef9e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf181692a0 con 0x7faf180b2390 2023-10-10T19:27:48.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf17fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180b1fc0 0x7faf18169d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:48.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf17fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180b1fc0 0x7faf18169d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35584/0 (socket says 172.21.15.175:35584) 2023-10-10T19:27:48.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf17fff700 1 -- 172.21.15.175:0/2794122841 learned_addr learned my addr 172.21.15.175:0/2794122841 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:48.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf1cd3a700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf180aff60 0x7faf181557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:48.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf1d53b700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf180b2390 0x7faf18153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:48.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf17fff700 1 -- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf180b2390 msgr2=0x7faf18153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:48.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf17fff700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf180b2390 0x7faf18153f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf17fff700 1 -- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf180aff60 msgr2=0x7faf181557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:48.758 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf17fff700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf180aff60 0x7faf181557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.747+0000 7faf17fff700 1 -- 172.21.15.175:0/2794122841 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf10009ae0 con 0x7faf180b1fc0 2023-10-10T19:27:48.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.748+0000 7faf17fff700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180b1fc0 0x7faf18169d40 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7faf0400b5e0 tx=0x7faf04009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:48.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.748+0000 7faf14ff9700 1 -- 172.21.15.175:0/2794122841 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0400f510 con 0x7faf180b1fc0 2023-10-10T19:27:48.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.748+0000 7faf14ff9700 1 -- 172.21.15.175:0/2794122841 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0400f670 con 0x7faf180b1fc0 2023-10-10T19:27:48.759 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.748+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf180cca50 con 0x7faf180b1fc0 2023-10-10T19:27:48.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.748+0000 7faf14ff9700 1 -- 172.21.15.175:0/2794122841 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0400fa30 con 0x7faf180b1fc0 2023-10-10T19:27:48.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.748+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf180cd000 con 0x7faf180b1fc0 2023-10-10T19:27:48.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.749+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf18169a20 con 0x7faf180b1fc0 2023-10-10T19:27:48.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.749+0000 7faf14ff9700 1 -- 172.21.15.175:0/2794122841 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faf0401a400 con 0x7faf180b1fc0 2023-10-10T19:27:48.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.749+0000 7faf14ff9700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faefc066f30 0x7faefc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:48.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.750+0000 7faf14ff9700 1 -- 172.21.15.175:0/2794122841 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7faf040848e0 con 0x7faf180b1fc0 2023-10-10T19:27:48.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.750+0000 7faf1cd3a700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faefc066f30 0x7faefc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:48.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.750+0000 7faf1cd3a700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faefc066f30 0x7faefc0693e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7faf0c006fd0 tx=0x7faf0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:48.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.754+0000 7faf14ff9700 1 -- 172.21.15.175:0/2794122841 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf040513a0 con 0x7faf180b1fc0 2023-10-10T19:27:48.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.980+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf18067570 con 0x7faf180b1fc0 2023-10-10T19:27:48.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.981+0000 7faf14ff9700 1 -- 172.21.15.175:0/2794122841 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 309 v309) v1 ==== 95+0+5430 (secure 0 0 0) 0x7faf040549c0 con 0x7faf180b1fc0 2023-10-10T19:27:48.983 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:48.983 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":309,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":309,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:48.122197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:48.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faefc066f30 msgr2=0x7faefc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:48.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faefc066f30 0x7faefc0693e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7faf0c006fd0 tx=0x7faf0c008040 comp rx=0 tx=0).stop 2023-10-10T19:27:48.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180b1fc0 msgr2=0x7faf18169d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:48.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180b1fc0 0x7faf18169d40 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7faf0400b5e0 tx=0x7faf04009120 comp rx=0 tx=0).stop 2023-10-10T19:27:48.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 shutdown_connections 2023-10-10T19:27:48.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7faefc066f30 0x7faefc0693e0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf180b2390 0x7faf18153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf180aff60 0x7faf181557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 --2- 172.21.15.175:0/2794122841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf180b1fc0 0x7faf18169d40 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:48.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 >> 172.21.15.175:0/2794122841 conn(0x7faf18063230 msgr2=0x7faf181508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:48.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 shutdown_connections 2023-10-10T19:27:48.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:48.983+0000 7faf1ef9e700 1 -- 172.21.15.175:0/2794122841 wait complete. 2023-10-10T19:27:48.988 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 309 2023-10-10T19:27:48.997 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:50.998 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:51.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.124+0000 7fc1cff37700 1 Processor -- start 2023-10-10T19:27:51.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1cff37700 1 -- start start 2023-10-10T19:27:51.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1cff37700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc1c80c5850 0x7fc1c80c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:51.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1cff37700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1c80c4f20 0x7fc1c80c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1cff37700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80c6fb0 con 0x7fc1c80c12d0 2023-10-10T19:27:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1cff37700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c8063ea0 con 0x7fc1c80c4f20 2023-10-10T19:27:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1cff37700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c8155f30 con 0x7fc1c80c5850 2023-10-10T19:27:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1ce4d4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1c80c4f20 0x7fc1c80c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:51.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1ce4d4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1c80c4f20 0x7fc1c80c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35586/0 (socket says 172.21.15.175:35586) 2023-10-10T19:27:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.125+0000 7fc1ce4d4700 1 -- 172.21.15.175:0/1134218209 learned_addr learned my addr 172.21.15.175:0/1134218209 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1cdcd3700 1 --2- 172.21.15.175:0/1134218209 >> v2:172.21.15.191:3300/0 conn(0x7fc1c80c5850 0x7fc1c80c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1ce4d4700 1 -- 172.21.15.175:0/1134218209 >> v2:172.21.15.191:3300/0 conn(0x7fc1c80c5850 msgr2=0x7fc1c80c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1ce4d4700 1 --2- 172.21.15.175:0/1134218209 >> v2:172.21.15.191:3300/0 conn(0x7fc1c80c5850 0x7fc1c80c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1ce4d4700 1 -- 172.21.15.175:0/1134218209 >> v1:172.21.15.175:6789/0 conn(0x7fc1c80c12d0 legacy=0x7fc1c8156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1ce4d4700 1 -- 172.21.15.175:0/1134218209 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c80c6160 con 0x7fc1c80c4f20 2023-10-10T19:27:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1ce4d4700 1 --2- 172.21.15.175:0/1134218209 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c80c7370 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80cb5c0 tx=0x7fc1c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435ead0a6beb6e97 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1cccd1700 1 -- 172.21.15.175:0/1134218209 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0009e70 con 0x7fc1c80c4f20 2023-10-10T19:27:51.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1cccd1700 1 -- 172.21.15.175:0/1134218209 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c000b8f0 con 0x7fc1c80c4f20 2023-10-10T19:27:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.126+0000 7fc1cccd1700 1 -- 172.21.15.175:0/1134218209 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c000bc90 con 0x7fc1c80c4f20 2023-10-10T19:27:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.127+0000 7fc1cff37700 1 -- 172.21.15.175:0/1134218209 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 msgr2=0x7fc1c80c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.127+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1134218209 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c80c7370 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80cb5c0 tx=0x7fc1c0009250 comp rx=0 tx=0).stop 2023-10-10T19:27:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.127+0000 7fc1cff37700 1 -- 172.21.15.175:0/1134218209 shutdown_connections 2023-10-10T19:27:51.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.127+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1134218209 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c80c7370 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.127+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1134218209 >> v2:172.21.15.191:3300/0 conn(0x7fc1c80c5850 0x7fc1c80c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.127+0000 7fc1cff37700 1 -- 172.21.15.175:0/1134218209 >> 172.21.15.175:0/1134218209 conn(0x7fc1c806dc70 msgr2=0x7fc1c8063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:51.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.127+0000 7fc1cff37700 1 -- 172.21.15.175:0/1134218209 shutdown_connections 2023-10-10T19:27:51.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.127+0000 7fc1cff37700 1 -- 172.21.15.175:0/1134218209 wait complete. 2023-10-10T19:27:51.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.128+0000 7fc1cff37700 1 Processor -- start 2023-10-10T19:27:51.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.128+0000 7fc1cff37700 1 -- start start 2023-10-10T19:27:51.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.128+0000 7fc1cff37700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1c80c12d0 0x7fc1c8168bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:51.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.128+0000 7fc1cff37700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c8165fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:51.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.128+0000 7fc1cff37700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1c80c5850 0x7fc1c814da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:51.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.128+0000 7fc1cff37700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8169b20 con 0x7fc1c80c4f20 2023-10-10T19:27:51.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cff37700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c81655d0 con 0x7fc1c80c12d0 2023-10-10T19:27:51.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cff37700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8166520 con 0x7fc1c80c5850 2023-10-10T19:27:51.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c8165fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:51.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c8165fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35594/0 (socket says 172.21.15.175:35594) 2023-10-10T19:27:51.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 -- 172.21.15.175:0/1540361260 learned_addr learned my addr 172.21.15.175:0/1540361260 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:51.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cdcd3700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1c80c12d0 0x7fc1c8168bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:51.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1ce4d4700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1c80c5850 0x7fc1c814da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:51.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 -- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1c80c5850 msgr2=0x7fc1c814da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:51.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1c80c5850 0x7fc1c814da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 -- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1c80c12d0 msgr2=0x7fc1c8168bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:51.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1c80c12d0 0x7fc1c8168bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 -- 172.21.15.175:0/1540361260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00094f0 con 0x7fc1c80c4f20 2023-10-10T19:27:51.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cdcd3700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1c80c12d0 0x7fc1c8168bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:27:51.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cd4d2700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c8165fe0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c510 tx=0x7fc1b800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:51.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1bdffb700 1 -- 172.21.15.175:0/1540361260 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8011590 con 0x7fc1c80c4f20 2023-10-10T19:27:51.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1bdffb700 1 -- 172.21.15.175:0/1540361260 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b8011710 con 0x7fc1c80c4f20 2023-10-10T19:27:51.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c8165b50 con 0x7fc1c80c4f20 2023-10-10T19:27:51.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.129+0000 7fc1bdffb700 1 -- 172.21.15.175:0/1540361260 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8011ad0 con 0x7fc1c80c4f20 2023-10-10T19:27:51.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.130+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c8169280 con 0x7fc1c80c4f20 2023-10-10T19:27:51.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.130+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c8165ce0 con 0x7fc1c80c4f20 2023-10-10T19:27:51.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.131+0000 7fc1bdffb700 1 -- 172.21.15.175:0/1540361260 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc1b801b3b0 con 0x7fc1c80c4f20 2023-10-10T19:27:51.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.131+0000 7fc1bdffb700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1ac066f60 0x7fc1ac069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:51.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.131+0000 7fc1bdffb700 1 -- 172.21.15.175:0/1540361260 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7fc1b8084250 con 0x7fc1c80c4f20 2023-10-10T19:27:51.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.131+0000 7fc1cdcd3700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1ac066f60 0x7fc1ac069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:51.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.132+0000 7fc1cdcd3700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1ac066f60 0x7fc1ac069410 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40076c0 tx=0x7fc1c400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:51.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.135+0000 7fc1bdffb700 1 -- 172.21.15.175:0/1540361260 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1b8050c90 con 0x7fc1c80c4f20 2023-10-10T19:27:51.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.358+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1c8067570 con 0x7fc1c80c4f20 2023-10-10T19:27:51.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.359+0000 7fc1bdffb700 1 -- 172.21.15.175:0/1540361260 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 309 v309) v1 ==== 95+0+5430 (secure 0 0 0) 0x7fc1b8017070 con 0x7fc1c80c4f20 2023-10-10T19:27:51.361 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:51.361 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":309,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":309,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:48.122197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:51.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1ac066f60 msgr2=0x7fc1ac069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:51.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1ac066f60 0x7fc1ac069410 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40076c0 tx=0x7fc1c400b040 comp rx=0 tx=0).stop 2023-10-10T19:27:51.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 msgr2=0x7fc1c8165fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:51.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c8165fe0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c510 tx=0x7fc1b800c8d0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 shutdown_connections 2023-10-10T19:27:51.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc1ac066f60 0x7fc1ac069410 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1c80c5850 0x7fc1c814da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1c80c12d0 0x7fc1c8168bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 --2- 172.21.15.175:0/1540361260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c80c4f20 0x7fc1c8165fe0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:51.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 >> 172.21.15.175:0/1540361260 conn(0x7fc1c806dc70 msgr2=0x7fc1c8156560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:51.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 shutdown_connections 2023-10-10T19:27:51.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:51.361+0000 7fc1cff37700 1 -- 172.21.15.175:0/1540361260 wait complete. 2023-10-10T19:27:51.366 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 309 2023-10-10T19:27:51.375 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:27:53.376 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:53.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.503+0000 7ff111fb3700 1 Processor -- start 2023-10-10T19:27:53.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.503+0000 7ff111fb3700 1 -- start start 2023-10-10T19:27:53.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff111fb3700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff10c0b1fc0 0x7ff10c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:53.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff111fb3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 0x7ff10c068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:53.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff111fb3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff10c068bd0 0x7ff10c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:53.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff111fb3700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10c150770 con 0x7ff10c0b1fc0 2023-10-10T19:27:53.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff111fb3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10c1574b0 con 0x7ff10c068bd0 2023-10-10T19:27:53.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff111fb3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c0afad0 con 0x7ff10c0b2390 2023-10-10T19:27:53.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff10c0b1fc0 0x7ff10c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:53.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff10c0b1fc0 0x7ff10c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35608/0 (socket says 172.21.15.175:35608) 2023-10-10T19:27:53.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10b7fe700 1 -- 172.21.15.175:0/295770257 learned_addr learned my addr 172.21.15.175:0/295770257 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:53.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10bfff700 1 --2- 172.21.15.175:0/295770257 >> v2:172.21.15.191:3300/0 conn(0x7ff10c068bd0 0x7ff10c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:53.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10b7fe700 1 -- 172.21.15.175:0/295770257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 msgr2=0x7ff10c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:53.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10affd700 1 --2- 172.21.15.175:0/295770257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 0x7ff10c068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:53.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10b7fe700 1 --2- 172.21.15.175:0/295770257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 0x7ff10c068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10b7fe700 1 -- 172.21.15.175:0/295770257 >> v2:172.21.15.191:3300/0 conn(0x7ff10c068bd0 msgr2=0x7ff10c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:53.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10b7fe700 1 --2- 172.21.15.175:0/295770257 >> v2:172.21.15.191:3300/0 conn(0x7ff10c068bd0 0x7ff10c0aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10b7fe700 1 -- 172.21.15.175:0/295770257 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c0af4c0 con 0x7ff10c0b1fc0 2023-10-10T19:27:53.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.504+0000 7ff10bfff700 1 --2- 172.21.15.175:0/295770257 >> v2:172.21.15.191:3300/0 conn(0x7ff10c068bd0 0x7ff10c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:27:53.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff10b7fe700 1 --2- 172.21.15.175:0/295770257 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c0b1fc0 0x7ff10c155fb0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0afa40 tx=0x7ff0fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb5631e5c2aa7dd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:53.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff10a7fc700 1 -- 172.21.15.175:0/295770257 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00e070 con 0x7ff10c0b1fc0 2023-10-10T19:27:53.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff10a7fc700 1 -- 172.21.15.175:0/295770257 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc00bae0 con 0x7ff10c0b1fc0 2023-10-10T19:27:53.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff10a7fc700 1 -- 172.21.15.175:0/295770257 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0092e0 con 0x7ff10c0b1fc0 2023-10-10T19:27:53.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 -- 172.21.15.175:0/295770257 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c0b1fc0 msgr2=0x7ff10c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:53.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 --2- 172.21.15.175:0/295770257 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c0b1fc0 0x7ff10c155fb0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0afa40 tx=0x7ff0fc009840 comp rx=0 tx=0).stop 2023-10-10T19:27:53.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 -- 172.21.15.175:0/295770257 shutdown_connections 2023-10-10T19:27:53.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 --2- 172.21.15.175:0/295770257 >> v2:172.21.15.191:3300/0 conn(0x7ff10c068bd0 0x7ff10c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 --2- 172.21.15.175:0/295770257 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c0b1fc0 0x7ff10c155fb0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 --2- 172.21.15.175:0/295770257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 0x7ff10c068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 -- 172.21.15.175:0/295770257 >> 172.21.15.175:0/295770257 conn(0x7ff10c063230 msgr2=0x7ff10c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:53.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 -- 172.21.15.175:0/295770257 shutdown_connections 2023-10-10T19:27:53.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.505+0000 7ff111fb3700 1 -- 172.21.15.175:0/295770257 wait complete. 2023-10-10T19:27:53.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff111fb3700 1 Processor -- start 2023-10-10T19:27:53.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff111fb3700 1 -- start start 2023-10-10T19:27:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff111fb3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c068bd0 0x7ff10c156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff111fb3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff10c0b1fc0 0x7ff10c0c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff111fb3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 0x7ff10c0c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:53.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff111fb3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c0c5e30 con 0x7ff10c068bd0 2023-10-10T19:27:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff111fb3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c169dd0 con 0x7ff10c0b1fc0 2023-10-10T19:27:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff111fb3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c154130 con 0x7ff10c0b2390 2023-10-10T19:27:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff10b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c068bd0 0x7ff10c156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:53.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff10b7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c068bd0 0x7ff10c156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35622/0 (socket says 172.21.15.175:35622) 2023-10-10T19:27:53.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.507+0000 7ff10b7fe700 1 -- 172.21.15.175:0/3297987028 learned_addr learned my addr 172.21.15.175:0/3297987028 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:53.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10affd700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff10c0b1fc0 0x7ff10c0c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:53.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10bfff700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 0x7ff10c0c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:53.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10b7fe700 1 -- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 msgr2=0x7ff10c0c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:53.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10b7fe700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 0x7ff10c0c58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10b7fe700 1 -- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff10c0b1fc0 msgr2=0x7ff10c0c53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:53.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10b7fe700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff10c0b1fc0 0x7ff10c0c53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10b7fe700 1 -- 172.21.15.175:0/3297987028 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc009ae0 con 0x7ff10c068bd0 2023-10-10T19:27:53.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10affd700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff10c0b1fc0 0x7ff10c0c53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:27:53.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff10b7fe700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c068bd0 0x7ff10c156850 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009fd0 tx=0x7ff0fc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:53.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff0f37fe700 1 -- 172.21.15.175:0/3297987028 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc008ca0 con 0x7ff10c068bd0 2023-10-10T19:27:53.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff0f37fe700 1 -- 172.21.15.175:0/3297987028 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc008e20 con 0x7ff10c068bd0 2023-10-10T19:27:53.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c0af030 con 0x7ff10c068bd0 2023-10-10T19:27:53.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff0f37fe700 1 -- 172.21.15.175:0/3297987028 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc01d670 con 0x7ff10c068bd0 2023-10-10T19:27:53.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.508+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c0c39a0 con 0x7ff10c068bd0 2023-10-10T19:27:53.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.509+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10c067570 con 0x7ff10c068bd0 2023-10-10T19:27:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.510+0000 7ff0f37fe700 1 -- 172.21.15.175:0/3297987028 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff0fc003040 con 0x7ff10c068bd0 2023-10-10T19:27:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.510+0000 7ff0f37fe700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0f8066f30 0x7ff0f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.510+0000 7ff0f37fe700 1 -- 172.21.15.175:0/3297987028 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7ff0fc086980 con 0x7ff10c068bd0 2023-10-10T19:27:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.510+0000 7ff10affd700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0f8066f30 0x7ff0f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.511+0000 7ff10affd700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0f8066f30 0x7ff0f80693e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff100006fd0 tx=0x7ff100008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.514+0000 7ff0f37fe700 1 -- 172.21.15.175:0/3297987028 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff0fc0533c0 con 0x7ff10c068bd0 2023-10-10T19:27:53.740 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.739+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff10c069060 con 0x7ff10c068bd0 2023-10-10T19:27:53.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.739+0000 7ff0f37fe700 1 -- 172.21.15.175:0/3297987028 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 309 v309) v1 ==== 95+0+5430 (secure 0 0 0) 0x7ff0fc022090 con 0x7ff10c068bd0 2023-10-10T19:27:53.741 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:53.741 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":309,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":309,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:48.122197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:53.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0f8066f30 msgr2=0x7ff0f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:53.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0f8066f30 0x7ff0f80693e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff100006fd0 tx=0x7ff100008040 comp rx=0 tx=0).stop 2023-10-10T19:27:53.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c068bd0 msgr2=0x7ff10c156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:53.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c068bd0 0x7ff10c156850 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009fd0 tx=0x7ff0fc018000 comp rx=0 tx=0).stop 2023-10-10T19:27:53.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 shutdown_connections 2023-10-10T19:27:53.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff0f8066f30 0x7ff0f80693e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff10c0b2390 0x7ff10c0c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff10c068bd0 0x7ff10c156850 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 --2- 172.21.15.175:0/3297987028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff10c0b1fc0 0x7ff10c0c53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:53.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.741+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 >> 172.21.15.175:0/3297987028 conn(0x7ff10c063230 msgr2=0x7ff10c159210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:53.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.742+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 shutdown_connections 2023-10-10T19:27:53.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:53.742+0000 7ff111fb3700 1 -- 172.21.15.175:0/3297987028 wait complete. 2023-10-10T19:27:53.746 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 309 2023-10-10T19:27:53.756 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 309, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 309, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_6893": { "addr": "172.21.15.191:6837/840318079", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 840318079, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 840318079, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 6893, "incarnation": 306, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:rejoin", "state_seq": 26 }, "gid_7057": { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 304, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 139 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 131, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:27:48.122197+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 6893, "mds_2": 8049, "mds_3": 7771, "mds_4": 7057 } } } ], "standbys": [] } 2023-10-10T19:27:55.757 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:27:55.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.884+0000 7f81603dc700 1 Processor -- start 2023-10-10T19:27:55.885 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.884+0000 7f81603dc700 1 -- start start 2023-10-10T19:27:55.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f81603dc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f81580c6330 0x7f815815a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f81603dc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 0x7f8158063c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f81603dc700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8158157030 con 0x7f81580c7190 2023-10-10T19:27:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f81603dc700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81580c5000 con 0x7f81580c6330 2023-10-10T19:27:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f81603dc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8158156420 con 0x7f81580c4580 2023-10-10T19:27:55.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e178700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f81580c6330 0x7f815815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e178700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f81580c6330 0x7f815815a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52262/0 (socket says 172.21.15.175:52262) 2023-10-10T19:27:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e178700 1 -- 172.21.15.175:0/2310465773 learned_addr learned my addr 172.21.15.175:0/2310465773 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e979700 1 --2- 172.21.15.175:0/2310465773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 0x7f8158063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e178700 1 -- 172.21.15.175:0/2310465773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 msgr2=0x7f8158063c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:27:55.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e178700 1 --2- 172.21.15.175:0/2310465773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 0x7f8158063c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e178700 1 -- 172.21.15.175:0/2310465773 >> v1:172.21.15.191:6789/0 conn(0x7f81580c7190 legacy=0x7f81580c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e178700 1 -- 172.21.15.175:0/2310465773 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8158156700 con 0x7f81580c6330 2023-10-10T19:27:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815e178700 1 --2- 172.21.15.175:0/2310465773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f815815a030 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f81540099b0 tx=0x7f8154009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31570ae2cdbc0487 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:55.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815d176700 1 -- 172.21.15.175:0/2310465773 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154009ea0 con 0x7f81580c6330 2023-10-10T19:27:55.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815d176700 1 -- 172.21.15.175:0/2310465773 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815400b8f0 con 0x7f81580c6330 2023-10-10T19:27:55.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.885+0000 7f815d176700 1 -- 172.21.15.175:0/2310465773 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815400bc90 con 0x7f81580c6330 2023-10-10T19:27:55.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.886+0000 7f81603dc700 1 -- 172.21.15.175:0/2310465773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 msgr2=0x7f815815a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:55.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.886+0000 7f81603dc700 1 --2- 172.21.15.175:0/2310465773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f815815a030 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f81540099b0 tx=0x7f8154009250 comp rx=0 tx=0).stop 2023-10-10T19:27:55.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.886+0000 7f81603dc700 1 -- 172.21.15.175:0/2310465773 shutdown_connections 2023-10-10T19:27:55.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.886+0000 7f81603dc700 1 --2- 172.21.15.175:0/2310465773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 0x7f8158063c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:55.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.886+0000 7f81603dc700 1 --2- 172.21.15.175:0/2310465773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f815815a030 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:55.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.886+0000 7f81603dc700 1 -- 172.21.15.175:0/2310465773 >> 172.21.15.175:0/2310465773 conn(0x7f815806ef20 msgr2=0x7f8158063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:55.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.886+0000 7f81603dc700 1 -- 172.21.15.175:0/2310465773 shutdown_connections 2023-10-10T19:27:55.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.886+0000 7f81603dc700 1 -- 172.21.15.175:0/2310465773 wait complete. 2023-10-10T19:27:55.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.887+0000 7f81603dc700 1 Processor -- start 2023-10-10T19:27:55.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.887+0000 7f81603dc700 1 -- start start 2023-10-10T19:27:55.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f81603dc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 0x7f81580c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:55.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f81603dc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f8158167710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:55.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f81603dc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81580c7190 0x7f8158167c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:55.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f81603dc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81580cbc30 con 0x7f81580c6330 2023-10-10T19:27:55.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f81603dc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8158167260 con 0x7f81580c7190 2023-10-10T19:27:55.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f81603dc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8158155060 con 0x7f81580c4580 2023-10-10T19:27:55.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f8158167710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:55.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f8158167710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52274/0 (socket says 172.21.15.175:52274) 2023-10-10T19:27:55.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 -- 172.21.15.175:0/2828345194 learned_addr learned my addr 172.21.15.175:0/2828345194 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:27:55.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815e979700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81580c7190 0x7f8158167c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:55.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815e178700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 0x7f81580c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:55.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 -- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 msgr2=0x7f81580c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:55.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 0x7f81580c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:55.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 -- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81580c7190 msgr2=0x7f8158167c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:55.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81580c7190 0x7f8158167c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:55.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 -- 172.21.15.175:0/2828345194 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81540094f0 con 0x7f81580c6330 2023-10-10T19:27:55.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f815d977700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f8158167710 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f814800c510 tx=0x7f814800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:55.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f814e7fc700 1 -- 172.21.15.175:0/2828345194 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148011590 con 0x7f81580c6330 2023-10-10T19:27:55.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.888+0000 7f814e7fc700 1 -- 172.21.15.175:0/2828345194 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8148011710 con 0x7f81580c6330 2023-10-10T19:27:55.899 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.889+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81580cbe60 con 0x7f81580c6330 2023-10-10T19:27:55.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.889+0000 7f814e7fc700 1 -- 172.21.15.175:0/2828345194 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148011ad0 con 0x7f81580c6330 2023-10-10T19:27:55.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.889+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81580cc400 con 0x7f81580c6330 2023-10-10T19:27:55.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.889+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81581673f0 con 0x7f81580c6330 2023-10-10T19:27:55.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.890+0000 7f814e7fc700 1 -- 172.21.15.175:0/2828345194 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f814801b3b0 con 0x7f81580c6330 2023-10-10T19:27:55.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.890+0000 7f814e7fc700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f813c066f30 0x7f813c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:27:55.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.890+0000 7f814e7fc700 1 -- 172.21.15.175:0/2828345194 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f81480841f0 con 0x7f81580c6330 2023-10-10T19:27:55.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.890+0000 7f815e178700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f813c066f30 0x7f813c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:27:55.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.891+0000 7f815e178700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f813c066f30 0x7f813c0693e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f81540099b0 tx=0x7f8154017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:27:55.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:55.895+0000 7f814e7fc700 1 -- 172.21.15.175:0/2828345194 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8148050c30 con 0x7f81580c6330 2023-10-10T19:27:56.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.121+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8158067570 con 0x7f81580c6330 2023-10-10T19:27:56.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.122+0000 7f814e7fc700 1 -- 172.21.15.175:0/2828345194 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 310 v310) v1 ==== 95+0+5430 (secure 0 0 0) 0x7f8148017070 con 0x7f81580c6330 2023-10-10T19:27:56.124 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:27:56.124 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":310,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":310,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:54.146901+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:27:56.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f813c066f30 msgr2=0x7f813c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:56.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f813c066f30 0x7f813c0693e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f81540099b0 tx=0x7f8154017000 comp rx=0 tx=0).stop 2023-10-10T19:27:56.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 msgr2=0x7f8158167710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:27:56.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f8158167710 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f814800c510 tx=0x7f814800c820 comp rx=0 tx=0).stop 2023-10-10T19:27:56.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 shutdown_connections 2023-10-10T19:27:56.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f813c066f30 0x7f813c0693e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:56.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81580c7190 0x7f8158167c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:56.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81580c4580 0x7f81580c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:56.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 --2- 172.21.15.175:0/2828345194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f81580c6330 0x7f8158167710 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:27:56.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 >> 172.21.15.175:0/2828345194 conn(0x7f815806ef20 msgr2=0x7f8158154950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:27:56.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 shutdown_connections 2023-10-10T19:27:56.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:27:56.124+0000 7f81603dc700 1 -- 172.21.15.175:0/2828345194 wait complete. 2023-10-10T19:27:56.128 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 310 2023-10-10T19:27:56.137 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=1, replacing gid=7401 2023-10-10T19:27:56.137 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2023-10-10T19:27:57.081 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:27:57.080+0000 7f7f74406700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:27:57.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:27:57.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:27:59.263 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/printenv.1 2023-10-10T19:28:02.139 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-10T19:28:02.139 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-10T19:28:02.139 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-10T19:28:02.142 INFO:tasks.ceph.mds.a:Started 2023-10-10T19:28:02.143 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:02.225 INFO:tasks.ceph.mds.a.smithi175.stdout:starting mds.a at 2023-10-10T19:28:02.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a97e6700 1 Processor -- start 2023-10-10T19:28:02.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a97e6700 1 -- start start 2023-10-10T19:28:02.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a97e6700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f46a40c5850 0x7f46a40c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:02.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a97e6700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40c6fb0 con 0x7f46a40c4f20 2023-10-10T19:28:02.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a97e6700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40c6160 con 0x7f46a40c12d0 2023-10-10T19:28:02.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a97e6700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a4063ea0 con 0x7f46a40c5850 2023-10-10T19:28:02.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a2ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f46a40c5850 0x7f46a40c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:02.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a2ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f46a40c5850 0x7f46a40c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52316/0 (socket says 172.21.15.175:52316) 2023-10-10T19:28:02.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a37fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f46a40c4f20 0x7f46a40c7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:40934/0 (socket says 172.21.15.175:40934) 2023-10-10T19:28:02.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a2ffd700 1 -- 172.21.15.175:0/3669415662 learned_addr learned my addr 172.21.15.175:0/3669415662 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:02.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a2ffd700 1 -- 172.21.15.175:0/3669415662 >> v1:172.21.15.191:6789/0 conn(0x7f46a40c12d0 legacy=0x7f46a4156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:02.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a2ffd700 1 -- 172.21.15.175:0/3669415662 >> v1:172.21.15.175:6789/0 conn(0x7f46a40c4f20 legacy=0x7f46a40c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:02.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a2ffd700 1 -- 172.21.15.175:0/3669415662 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a4063940 con 0x7f46a40c5850 2023-10-10T19:28:02.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a1ffb700 1 -- 172.21.15.175:0/3669415662 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278049608 0 0) 0x7f46a40c6fb0 con 0x7f46a40c4f20 2023-10-10T19:28:02.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a2ffd700 1 --2- 172.21.15.175:0/3669415662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c5850 0x7f46a40c5c20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f46940099b0 tx=0x7f4694009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94dd8b97667b4489 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:02.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a1ffb700 1 -- 172.21.15.175:0/3669415662 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469400b8f0 con 0x7f46a40c5850 2023-10-10T19:28:02.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a1ffb700 1 -- 172.21.15.175:0/3669415662 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469400ba50 con 0x7f46a40c5850 2023-10-10T19:28:02.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.272+0000 7f46a1ffb700 1 -- 172.21.15.175:0/3669415662 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469400bdf0 con 0x7f46a40c5850 2023-10-10T19:28:02.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.273+0000 7f46a97e6700 1 -- 172.21.15.175:0/3669415662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c5850 msgr2=0x7f46a40c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:02.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.273+0000 7f46a97e6700 1 --2- 172.21.15.175:0/3669415662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c5850 0x7f46a40c5c20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f46940099b0 tx=0x7f4694009250 comp rx=0 tx=0).stop 2023-10-10T19:28:02.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.273+0000 7f46a97e6700 1 -- 172.21.15.175:0/3669415662 shutdown_connections 2023-10-10T19:28:02.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.273+0000 7f46a97e6700 1 --2- 172.21.15.175:0/3669415662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c5850 0x7f46a40c5c20 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:02.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.273+0000 7f46a97e6700 1 -- 172.21.15.175:0/3669415662 >> 172.21.15.175:0/3669415662 conn(0x7f46a406dc70 msgr2=0x7f46a4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:02.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.273+0000 7f46a97e6700 1 -- 172.21.15.175:0/3669415662 shutdown_connections 2023-10-10T19:28:02.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.273+0000 7f46a97e6700 1 -- 172.21.15.175:0/3669415662 wait complete. 2023-10-10T19:28:02.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.274+0000 7f46a97e6700 1 Processor -- start 2023-10-10T19:28:02.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.274+0000 7f46a97e6700 1 -- start start 2023-10-10T19:28:02.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a97e6700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c12d0 0x7f46a414c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:02.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a97e6700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46a40c4f20 0x7f46a4150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:02.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a97e6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46a40c5850 0x7f46a4168f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:02.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a97e6700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a414c3a0 con 0x7f46a40c12d0 2023-10-10T19:28:02.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a97e6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a414bc10 con 0x7f46a40c5850 2023-10-10T19:28:02.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a97e6700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a40afcb0 con 0x7f46a40c4f20 2023-10-10T19:28:02.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c12d0 0x7f46a414c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:02.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a2ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c12d0 0x7f46a414c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:52324/0 (socket says 172.21.15.175:52324) 2023-10-10T19:28:02.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a2ffd700 1 -- 172.21.15.175:0/614599462 learned_addr learned my addr 172.21.15.175:0/614599462 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:02.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a37fe700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46a40c5850 0x7f46a4168f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:02.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a27fc700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46a40c4f20 0x7f46a4150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:02.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a2ffd700 1 -- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46a40c4f20 msgr2=0x7f46a4150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:02.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a2ffd700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46a40c4f20 0x7f46a4150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:02.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a2ffd700 1 -- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46a40c5850 msgr2=0x7f46a4168f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:02.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a2ffd700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46a40c5850 0x7f46a4168f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:02.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.275+0000 7f46a2ffd700 1 -- 172.21.15.175:0/614599462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46940094f0 con 0x7f46a40c12d0 2023-10-10T19:28:02.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.276+0000 7f46a2ffd700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c12d0 0x7f46a414c570 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f4694006f50 tx=0x7f4694017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:02.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.276+0000 7f468affd700 1 -- 172.21.15.175:0/614599462 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469401f070 con 0x7f46a40c12d0 2023-10-10T19:28:02.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.276+0000 7f468affd700 1 -- 172.21.15.175:0/614599462 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4694011c80 con 0x7f46a40c12d0 2023-10-10T19:28:02.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.276+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a414af00 con 0x7f46a40c12d0 2023-10-10T19:28:02.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.276+0000 7f468affd700 1 -- 172.21.15.175:0/614599462 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694025420 con 0x7f46a40c12d0 2023-10-10T19:28:02.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.276+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a414b780 con 0x7f46a40c12d0 2023-10-10T19:28:02.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.277+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a414ba60 con 0x7f46a40c12d0 2023-10-10T19:28:02.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.277+0000 7f468affd700 1 -- 172.21.15.175:0/614599462 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4694023070 con 0x7f46a40c12d0 2023-10-10T19:28:02.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.278+0000 7f468affd700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4690066f30 0x7f46900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:02.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.278+0000 7f468affd700 1 -- 172.21.15.175:0/614599462 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f46940863a0 con 0x7f46a40c12d0 2023-10-10T19:28:02.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.278+0000 7f46a27fc700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4690066f30 0x7f46900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:02.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.278+0000 7f46a27fc700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4690066f30 0x7f46900693e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f46980037c0 tx=0x7f4698009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:02.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.283+0000 7f468affd700 1 -- 172.21.15.175:0/614599462 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4694052de0 con 0x7f46a40c12d0 2023-10-10T19:28:02.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.508+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46a4067570 con 0x7f46a40c12d0 2023-10-10T19:28:02.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.509+0000 7f468affd700 1 -- 172.21.15.175:0/614599462 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 310 v310) v1 ==== 95+0+5430 (secure 0 0 0) 0x7f469402a090 con 0x7f46a40c12d0 2023-10-10T19:28:02.510 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:02.510 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":310,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":310,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:54.146901+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:02.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4690066f30 msgr2=0x7f46900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:02.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4690066f30 0x7f46900693e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f46980037c0 tx=0x7f4698009040 comp rx=0 tx=0).stop 2023-10-10T19:28:02.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c12d0 msgr2=0x7f46a414c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:02.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c12d0 0x7f46a414c570 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f4694006f50 tx=0x7f4694017000 comp rx=0 tx=0).stop 2023-10-10T19:28:02.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 shutdown_connections 2023-10-10T19:28:02.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4690066f30 0x7f46900693e0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:02.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46a40c5850 0x7f46a4168f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:02.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46a40c12d0 0x7f46a414c570 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:02.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 --2- 172.21.15.175:0/614599462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46a40c4f20 0x7f46a4150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:02.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 >> 172.21.15.175:0/614599462 conn(0x7f46a406dc70 msgr2=0x7f46a4063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:02.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 shutdown_connections 2023-10-10T19:28:02.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:02.511+0000 7f46a97e6700 1 -- 172.21.15.175:0/614599462 wait complete. 2023-10-10T19:28:02.514 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 310 2023-10-10T19:28:02.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-10T19:28:04.234 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tr.x 2023-10-10T19:28:04.235 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tail.1 2023-10-10T19:28:04.237 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/md5sum.x 2023-10-10T19:28:04.237 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/link.1 2023-10-10T19:28:04.238 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/echo.x 2023-10-10T19:28:04.239 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/ptx.1 2023-10-10T19:28:04.240 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/factor.x 2023-10-10T19:28:04.241 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/who.1 2023-10-10T19:28:04.242 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mktemp.1 2023-10-10T19:28:04.243 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/shred.1 2023-10-10T19:28:04.244 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/runcon.1 2023-10-10T19:28:04.245 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/stty.x 2023-10-10T19:28:04.245 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/comm.1 2023-10-10T19:28:04.247 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/shuf.1 2023-10-10T19:28:04.248 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/stty.1 2023-10-10T19:28:04.249 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/printenv.x 2023-10-10T19:28:04.250 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/join.1 2023-10-10T19:28:04.252 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/hostname.x 2023-10-10T19:28:04.253 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/ln.1 2023-10-10T19:28:04.253 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/false.x 2023-10-10T19:28:04.254 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/cat.1 2023-10-10T19:28:04.255 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/users.x 2023-10-10T19:28:04.256 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/printf.1 2023-10-10T19:28:04.257 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/ls.x 2023-10-10T19:28:04.258 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha224sum.x 2023-10-10T19:28:04.259 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/whoami.1 2023-10-10T19:28:04.260 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/expand.1 2023-10-10T19:28:04.261 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chgrp.x 2023-10-10T19:28:04.524 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:04.528 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sleep.1 2023-10-10T19:28:04.528 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/csplit.x 2023-10-10T19:28:04.529 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/pwd.1 2023-10-10T19:28:04.529 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/kill.1 2023-10-10T19:28:04.530 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/touch.x 2023-10-10T19:28:04.531 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/df.1 2023-10-10T19:28:04.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.651+0000 7f6f8bfdf700 1 Processor -- start 2023-10-10T19:28:04.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.652+0000 7f6f8bfdf700 1 -- start start 2023-10-10T19:28:04.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.652+0000 7f6f8bfdf700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6f840c5850 0x7f6f840c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:04.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.652+0000 7f6f8bfdf700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6f840c12d0 0x7f6f84156660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:04.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.652+0000 7f6f8bfdf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f840c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:04.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8bfdf700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f84063ea0 con 0x7f6f840c12d0 2023-10-10T19:28:04.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8bfdf700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f84155f30 con 0x7f6f840c5850 2023-10-10T19:28:04.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8bfdf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f840c6fb0 con 0x7f6f84156ba0 2023-10-10T19:28:04.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6f840c12d0 0x7f6f84156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:04.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6f840c12d0 0x7f6f84156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42728/0 (socket says 172.21.15.175:42728) 2023-10-10T19:28:04.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 -- 172.21.15.175:0/838660162 learned_addr learned my addr 172.21.15.175:0/838660162 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:04.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f89d7b700 1 --2- 172.21.15.175:0/838660162 >> v2:172.21.15.191:3300/0 conn(0x7f6f840c5850 0x7f6f840c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:04.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8a57c700 1 --2- 172.21.15.175:0/838660162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f840c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:04.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 -- 172.21.15.175:0/838660162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 msgr2=0x7f6f840c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:04.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 --2- 172.21.15.175:0/838660162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f840c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 -- 172.21.15.175:0/838660162 >> v2:172.21.15.191:3300/0 conn(0x7f6f840c5850 msgr2=0x7f6f840c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:04.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 --2- 172.21.15.175:0/838660162 >> v2:172.21.15.191:3300/0 conn(0x7f6f840c5850 0x7f6f840c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 -- 172.21.15.175:0/838660162 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f840c6160 con 0x7f6f840c12d0 2023-10-10T19:28:04.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f8957a700 1 --2- 172.21.15.175:0/838660162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84156660 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f6f840cb5c0 tx=0x7f6f74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504198c7c425dabc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:04.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f88d79700 1 -- 172.21.15.175:0/838660162 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7400b8b0 con 0x7f6f840c12d0 2023-10-10T19:28:04.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f88d79700 1 -- 172.21.15.175:0/838660162 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7400ba10 con 0x7f6f840c12d0 2023-10-10T19:28:04.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.653+0000 7f6f88d79700 1 -- 172.21.15.175:0/838660162 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7400bdb0 con 0x7f6f840c12d0 2023-10-10T19:28:04.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/838660162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 msgr2=0x7f6f84156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:04.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/838660162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84156660 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f6f840cb5c0 tx=0x7f6f74009250 comp rx=0 tx=0).stop 2023-10-10T19:28:04.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/838660162 shutdown_connections 2023-10-10T19:28:04.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/838660162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f840c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/838660162 >> v2:172.21.15.191:3300/0 conn(0x7f6f840c5850 0x7f6f840c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/838660162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84156660 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/838660162 >> 172.21.15.175:0/838660162 conn(0x7f6f8406dc70 msgr2=0x7f6f84063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:04.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/838660162 shutdown_connections 2023-10-10T19:28:04.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.654+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/838660162 wait complete. 2023-10-10T19:28:04.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.655+0000 7f6f8bfdf700 1 Processor -- start 2023-10-10T19:28:04.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.655+0000 7f6f8bfdf700 1 -- start start 2023-10-10T19:28:04.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8bfdf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:04.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8bfdf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840c5850 0x7f6f8414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:04.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8bfdf700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f84169a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:04.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8bfdf700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84150a70 con 0x7f6f840c12d0 2023-10-10T19:28:04.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8bfdf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841695b0 con 0x7f6f840c5850 2023-10-10T19:28:04.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8bfdf700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8414fe70 con 0x7f6f84156ba0 2023-10-10T19:28:04.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:04.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42738/0 (socket says 172.21.15.175:42738) 2023-10-10T19:28:04.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 -- 172.21.15.175:0/781889228 learned_addr learned my addr 172.21.15.175:0/781889228 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:04.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8957a700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840c5850 0x7f6f8414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:04.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8a57c700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f84169a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:04.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 -- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 msgr2=0x7f6f84169a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:04.666 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f84169a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 -- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840c5850 msgr2=0x7f6f8414df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:04.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840c5850 0x7f6f8414df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 -- 172.21.15.175:0/781889228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740094f0 con 0x7f6f840c12d0 2023-10-10T19:28:04.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f8a57c700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f84169a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:28:04.667 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f89d7b700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84157a90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000d860 tx=0x7f6f80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:04.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.656+0000 7f6f79ffb700 1 -- 172.21.15.175:0/781889228 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80009610 con 0x7f6f840c12d0 2023-10-10T19:28:04.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.657+0000 7f6f79ffb700 1 -- 172.21.15.175:0/781889228 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f80009770 con 0x7f6f840c12d0 2023-10-10T19:28:04.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.657+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f84150150 con 0x7f6f840c12d0 2023-10-10T19:28:04.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.657+0000 7f6f79ffb700 1 -- 172.21.15.175:0/781889228 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80009b30 con 0x7f6f840c12d0 2023-10-10T19:28:04.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.657+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f84150710 con 0x7f6f840c12d0 2023-10-10T19:28:04.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.658+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f84069640 con 0x7f6f840c12d0 2023-10-10T19:28:04.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.658+0000 7f6f79ffb700 1 -- 172.21.15.175:0/781889228 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6f8001e460 con 0x7f6f840c12d0 2023-10-10T19:28:04.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.658+0000 7f6f79ffb700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f68066f30 0x7f6f680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:04.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.658+0000 7f6f79ffb700 1 -- 172.21.15.175:0/781889228 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f6f800853d0 con 0x7f6f840c12d0 2023-10-10T19:28:04.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.659+0000 7f6f8957a700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f68066f30 0x7f6f680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:04.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.659+0000 7f6f8957a700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f68066f30 0x7f6f680693e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f6f840cb5c0 tx=0x7f6f74017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:04.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.663+0000 7f6f79ffb700 1 -- 172.21.15.175:0/781889228 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f80051e10 con 0x7f6f840c12d0 2023-10-10T19:28:04.857 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/stdbuf.x 2023-10-10T19:28:04.862 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mkdir.x 2023-10-10T19:28:04.863 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/id.1 2023-10-10T19:28:04.863 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/mknod.x 2023-10-10T19:28:04.864 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sum.1 2023-10-10T19:28:04.865 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/rm.x 2023-10-10T19:28:04.866 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/date.1 2023-10-10T19:28:04.867 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chown.x 2023-10-10T19:28:04.867 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/touch.1 2023-10-10T19:28:04.868 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tac.x 2023-10-10T19:28:04.869 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/base64.x 2023-10-10T19:28:04.870 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/tsort.x 2023-10-10T19:28:04.871 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/shred.x 2023-10-10T19:28:04.872 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/readlink.1 2023-10-10T19:28:04.872 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/sha1sum.x 2023-10-10T19:28:04.873 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/expand.x 2023-10-10T19:28:04.874 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/test.x 2023-10-10T19:28:04.875 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/man/chcon.1 2023-10-10T19:28:04.876 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.tarball-version 2023-10-10T19:28:04.877 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/NEWS 2023-10-10T19:28:04.878 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_sun_os_names 2023-10-10T19:28:04.880 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/bootstrap 2023-10-10T19:28:04.889 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/ 2023-10-10T19:28:04.890 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbsalign.h 2023-10-10T19:28:04.891 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.891+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f84169890 con 0x7f6f840c12d0 2023-10-10T19:28:04.892 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/streq.h 2023-10-10T19:28:04.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.892+0000 7f6f79ffb700 1 -- 172.21.15.175:0/781889228 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 311 v311) v1 ==== 95+0+6190 (secure 0 0 0) 0x7f6f80055430 con 0x7f6f840c12d0 2023-10-10T19:28:04.893 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:04.893 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":311,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":311}],"filesystems":[{"mdsmap":{"epoch":310,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:27:54.146901+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:04.894 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/asprintf.c 2023-10-10T19:28:04.894 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strnlen1.c 2023-10-10T19:28:04.894 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirfd.c 2023-10-10T19:28:04.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f68066f30 msgr2=0x7f6f680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:04.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f68066f30 0x7f6f680693e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f6f840cb5c0 tx=0x7f6f74017000 comp rx=0 tx=0).stop 2023-10-10T19:28:04.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 msgr2=0x7f6f84157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:04.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84157a90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000d860 tx=0x7f6f80008040 comp rx=0 tx=0).stop 2023-10-10T19:28:04.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 shutdown_connections 2023-10-10T19:28:04.896 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/safe-write.h 2023-10-10T19:28:04.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6f68066f30 0x7f6f680693e0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f84156ba0 0x7f6f84169a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f840c12d0 0x7f6f84157a90 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 --2- 172.21.15.175:0/781889228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f840c5850 0x7f6f8414df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:04.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 >> 172.21.15.175:0/781889228 conn(0x7f6f8406dc70 msgr2=0x7f6f840c00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:04.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 shutdown_connections 2023-10-10T19:28:04.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:04.894+0000 7f6f8bfdf700 1 -- 172.21.15.175:0/781889228 wait complete. 2023-10-10T19:28:04.898 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/snprintf.c 2023-10-10T19:28:04.899 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 311 2023-10-10T19:28:04.900 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbiter.h 2023-10-10T19:28:04.900 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mountlist.c 2023-10-10T19:28:04.900 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/propername.h 2023-10-10T19:28:04.900 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memset.c 2023-10-10T19:28:04.901 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/nproc.c 2023-10-10T19:28:04.901 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getdelim.c 2023-10-10T19:28:04.901 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/signbitl.c 2023-10-10T19:28:04.902 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/group-member.h 2023-10-10T19:28:04.902 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/long-options.c 2023-10-10T19:28:04.902 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fts_.h 2023-10-10T19:28:04.903 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fts.c 2023-10-10T19:28:04.903 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/verify.h 2023-10-10T19:28:04.904 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/str-two-way.h 2023-10-10T19:28:04.904 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/areadlink.c 2023-10-10T19:28:04.905 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/c-strtod.h 2023-10-10T19:28:04.906 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sha256.h 2023-10-10T19:28:04.907 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-10T19:28:04.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 15 secs before thrashing 2023-10-10T19:28:04.907 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stddef.in.h 2023-10-10T19:28:04.908 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/randint.c 2023-10-10T19:28:04.908 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dup-safer.c 2023-10-10T19:28:04.909 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getpass.h 2023-10-10T19:28:04.909 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/gettext.h 2023-10-10T19:28:04.910 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/randint.h 2023-10-10T19:28:04.911 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdio-write.c 2023-10-10T19:28:04.912 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xgethostname.h 2023-10-10T19:28:04.913 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/asnprintf.c 2023-10-10T19:28:04.913 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-solaris.gperf 2023-10-10T19:28:04.914 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdio-impl.h 2023-10-10T19:28:04.915 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mpsort.h 2023-10-10T19:28:04.916 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/closein.c 2023-10-10T19:28:04.917 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/hard-locale.c 2023-10-10T19:28:04.918 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/utimecmp.c 2023-10-10T19:28:04.918 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/vasnprintf.c 2023-10-10T19:28:04.920 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/chown.c 2023-10-10T19:28:04.920 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/printf-args.c 2023-10-10T19:28:04.921 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/netdb.in.h 2023-10-10T19:28:04.921 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strnlen.c 2023-10-10T19:28:04.922 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/errno.in.h 2023-10-10T19:28:04.923 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbslen.c 2023-10-10T19:28:04.923 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/obstack.h 2023-10-10T19:28:04.924 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/file-type.h 2023-10-10T19:28:04.925 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbsrtowcs.c 2023-10-10T19:28:04.925 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fnmatch_loop.c 2023-10-10T19:28:04.926 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/safe-write.c 2023-10-10T19:28:04.927 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mpsort.c 2023-10-10T19:28:04.927 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/netinet_in.in.h 2023-10-10T19:28:04.928 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xfreopen.h 2023-10-10T19:28:04.929 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/wctype.in.h 2023-10-10T19:28:04.929 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stripslash.c 2023-10-10T19:28:04.930 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/closein.h 2023-10-10T19:28:04.930 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memchr2.c 2023-10-10T19:28:04.931 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memcoll.c 2023-10-10T19:28:04.932 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/malloca.h 2023-10-10T19:28:04.933 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stpcpy.c 2023-10-10T19:28:04.934 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/striconv.c 2023-10-10T19:28:04.934 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/basename.c 2023-10-10T19:28:04.935 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sha256.c 2023-10-10T19:28:04.936 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdint.in.h 2023-10-10T19:28:04.936 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/acl-internal.h 2023-10-10T19:28:04.937 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/atexit.c 2023-10-10T19:28:04.937 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/size_max.h 2023-10-10T19:28:04.938 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/buffer-lcm.h 2023-10-10T19:28:04.939 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/idcache.h 2023-10-10T19:28:04.939 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/root-dev-ino.c 2023-10-10T19:28:04.940 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/physmem.c 2023-10-10T19:28:04.941 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/glthread/ 2023-10-10T19:28:04.941 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/glthread/lock.h 2023-10-10T19:28:04.942 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/glthread/threadlib.c 2023-10-10T19:28:04.943 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/glthread/lock.c 2023-10-10T19:28:04.943 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/glthread/tls.h 2023-10-10T19:28:04.944 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/glthread/tls.c 2023-10-10T19:28:04.945 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unistr/ 2023-10-10T19:28:04.945 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unistr/u8-uctomb.c 2023-10-10T19:28:04.946 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unistr/u8-uctomb-aux.c 2023-10-10T19:28:04.947 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mkancesdirs.c 2023-10-10T19:28:04.947 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/readtokens0.c 2023-10-10T19:28:04.948 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fnmatch.in.h 2023-10-10T19:28:04.949 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirname.h 2023-10-10T19:28:04.949 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/regex_internal.c 2023-10-10T19:28:04.950 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strcspn.c 2023-10-10T19:28:04.950 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getugroups.h 2023-10-10T19:28:04.951 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/file-set.c 2023-10-10T19:28:04.952 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sys_socket.in.h 2023-10-10T19:28:04.952 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/inttostr.h 2023-10-10T19:28:04.953 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mkdirat.c 2023-10-10T19:28:04.954 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/readtokens.h 2023-10-10T19:28:04.954 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fd-reopen.h 2023-10-10T19:28:04.954 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/w32sock.h 2023-10-10T19:28:04.955 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memcasecmp.c 2023-10-10T19:28:04.956 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/at-func.c 2023-10-10T19:28:04.956 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unistd-safer.h 2023-10-10T19:28:04.957 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/filevercmp.h 2023-10-10T19:28:04.958 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/c-strcasecmp.c 2023-10-10T19:28:04.959 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fopen-safer.c 2023-10-10T19:28:04.959 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/modechange.h 2023-10-10T19:28:04.960 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sys_stat.in.h 2023-10-10T19:28:04.961 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freading.c 2023-10-10T19:28:04.961 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/utimens.h 2023-10-10T19:28:04.962 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/md5.h 2023-10-10T19:28:04.963 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isnanf.c 2023-10-10T19:28:04.963 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sha1.c 2023-10-10T19:28:04.964 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/filenamecat-lgpl.c 2023-10-10T19:28:04.964 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/config.hin 2023-10-10T19:28:04.965 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/ignore-value.h 2023-10-10T19:28:04.966 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strstr.c 2023-10-10T19:28:04.967 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/verror.c 2023-10-10T19:28:04.967 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/randperm.h 2023-10-10T19:28:04.968 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/vasprintf.c 2023-10-10T19:28:04.969 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/c-strncasecmp.c 2023-10-10T19:28:04.970 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/basename-lgpl.c 2023-10-10T19:28:04.970 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/human.h 2023-10-10T19:28:04.971 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/regex.h 2023-10-10T19:28:04.972 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/regcomp.c 2023-10-10T19:28:04.972 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freadahead.c 2023-10-10T19:28:04.973 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/trim.h 2023-10-10T19:28:04.974 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getgroups.c 2023-10-10T19:28:04.975 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xprintf.c 2023-10-10T19:28:04.975 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/euidaccess-stat.h 2023-10-10T19:28:04.976 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/rand-isaac.h 2023-10-10T19:28:04.976 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mgetgroups.h 2023-10-10T19:28:04.977 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/selinux-at.h 2023-10-10T19:28:04.977 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-solaris.h 2023-10-10T19:28:04.978 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/setenv.c 2023-10-10T19:28:04.978 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/locale.in.h 2023-10-10T19:28:04.979 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/base64.h 2023-10-10T19:28:04.980 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/open-safer.c 2023-10-10T19:28:04.980 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/exitfail.c 2023-10-10T19:28:04.981 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/closeout.h 2023-10-10T19:28:04.981 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/idcache.c 2023-10-10T19:28:04.982 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unlink.c 2023-10-10T19:28:04.983 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xfts.c 2023-10-10T19:28:04.983 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memrchr.c 2023-10-10T19:28:04.984 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/Makefile.am 2023-10-10T19:28:04.984 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtol.h 2023-10-10T19:28:04.985 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/gettimeofday.c 2023-10-10T19:28:04.986 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isnanf-nolibm.h 2023-10-10T19:28:04.987 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/chdir-long.c 2023-10-10T19:28:04.987 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbchar.c 2023-10-10T19:28:04.988 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/canon-host.h 2023-10-10T19:28:04.989 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unsetenv.c 2023-10-10T19:28:04.989 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freadseek.c 2023-10-10T19:28:04.990 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xnanosleep.c 2023-10-10T19:28:04.991 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/nanosleep.c 2023-10-10T19:28:04.991 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/ctype.in.h 2023-10-10T19:28:04.992 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xmalloc.c 2023-10-10T19:28:04.992 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strtoul.c 2023-10-10T19:28:04.993 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sha512.h 2023-10-10T19:28:04.993 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirchownmod.h 2023-10-10T19:28:04.994 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freadptr.h 2023-10-10T19:28:04.995 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getfilecon.c 2023-10-10T19:28:04.996 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/alignof.h 2023-10-10T19:28:04.996 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mempcpy.c 2023-10-10T19:28:04.997 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/savedir.h 2023-10-10T19:28:09.450 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/areadlink-with-size.c 2023-10-10T19:28:09.461 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/inttypes.in.h 2023-10-10T19:28:09.491 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/config.charset 2023-10-10T19:28:09.544 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtold.c 2023-10-10T19:28:09.545 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fcntl-safer.h 2023-10-10T19:28:09.546 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrndup.c 2023-10-10T19:28:09.547 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/write.c 2023-10-10T19:28:09.547 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/math.in.h 2023-10-10T19:28:09.548 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/openat-priv.h 2023-10-10T19:28:09.549 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/vasnprintf.h 2023-10-10T19:28:09.550 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getcwd.c 2023-10-10T19:28:09.551 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdbool.in.h 2023-10-10T19:28:09.551 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strtol.c 2023-10-10T19:28:09.552 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/argmatch.c 2023-10-10T19:28:09.553 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/acl_entries.c 2023-10-10T19:28:09.554 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/progname.c 2023-10-10T19:28:09.555 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mktime-internal.h 2023-10-10T19:28:09.555 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/printf-frexpl.c 2023-10-10T19:28:09.556 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/offtostr.c 2023-10-10T19:28:09.557 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memchr2.h 2023-10-10T19:28:09.558 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtoumax.c 2023-10-10T19:28:09.559 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getdate.y 2023-10-10T19:28:09.560 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/close-hook.h 2023-10-10T19:28:09.560 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/printf-parse.c 2023-10-10T19:28:09.561 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isnand.c 2023-10-10T19:28:09.562 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/timespec.h 2023-10-10T19:28:09.563 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getusershell.c 2023-10-10T19:28:09.564 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtod.c 2023-10-10T19:28:09.565 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/verror.h 2023-10-10T19:28:09.565 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/same.c 2023-10-10T19:28:09.566 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/quote.c 2023-10-10T19:28:09.567 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/raise.c 2023-10-10T19:28:09.567 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/link.c 2023-10-10T19:28:09.568 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/cloexec.h 2023-10-10T19:28:09.569 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sys_utsname.in.h 2023-10-10T19:28:09.570 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/cloexec.c 2023-10-10T19:28:09.571 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/vfprintf.c 2023-10-10T19:28:09.572 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/uniwidth/ 2023-10-10T19:28:09.573 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/uniwidth/cjk.h 2023-10-10T19:28:09.574 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/uniwidth/width.c 2023-10-10T19:28:09.587 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strtoull.c 2023-10-10T19:28:09.588 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/opendir-safer.c 2023-10-10T19:28:09.588 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/hash-pjw.h 2023-10-10T19:28:09.589 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/uname.c 2023-10-10T19:28:09.590 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/full-read.c 2023-10-10T19:28:09.591 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/priv-set.h 2023-10-10T19:28:09.592 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/close.c 2023-10-10T19:28:09.593 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fpending.h 2023-10-10T19:28:09.594 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/i-ring.h 2023-10-10T19:28:09.595 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/nl_langinfo.c 2023-10-10T19:28:09.595 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xmemxfrm.h 2023-10-10T19:28:09.596 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/c-strcase.h 2023-10-10T19:28:09.597 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fchdir.c 2023-10-10T19:28:09.598 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/canon-host.c 2023-10-10T19:28:09.599 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/signbitf.c 2023-10-10T19:28:09.599 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xnanosleep.h 2023-10-10T19:28:09.600 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/tempname.c 2023-10-10T19:28:09.601 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xreadlink.c 2023-10-10T19:28:09.602 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xgetcwd.h 2023-10-10T19:28:09.602 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/regex.c 2023-10-10T19:28:09.603 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/diacrit.h 2023-10-10T19:28:09.604 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/posixver.c 2023-10-10T19:28:09.605 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getugroups.c 2023-10-10T19:28:09.606 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/readutmp.h 2023-10-10T19:28:09.607 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/quote.h 2023-10-10T19:28:09.608 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/quotearg.c 2023-10-10T19:28:09.609 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/filevercmp.c 2023-10-10T19:28:09.609 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/printf-args.h 2023-10-10T19:28:09.611 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mountlist.h 2023-10-10T19:28:09.612 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtol-error.c 2023-10-10T19:28:09.613 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/rename.c 2023-10-10T19:28:09.613 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sha1.h 2023-10-10T19:28:09.614 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strndup.c 2023-10-10T19:28:09.615 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/priv-set.c 2023-10-10T19:28:09.615 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xfts.h 2023-10-10T19:28:09.616 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/argv-iter.c 2023-10-10T19:28:09.617 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdio--.h 2023-10-10T19:28:09.618 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/yesno.c 2023-10-10T19:28:09.619 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getopt.c 2023-10-10T19:28:09.620 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strpbrk.c 2023-10-10T19:28:09.622 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirname-lgpl.c 2023-10-10T19:28:09.622 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/filemode.h 2023-10-10T19:28:09.623 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbrtowc.c 2023-10-10T19:28:09.625 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open.c 2023-10-10T19:28:09.625 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/euidaccess-stat.c 2023-10-10T19:28:09.626 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stat-time.h 2023-10-10T19:28:09.626 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/frexpl.c 2023-10-10T19:28:09.627 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sys_select.in.h 2023-10-10T19:28:09.628 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strftime.h 2023-10-10T19:28:09.629 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strftime.c 2023-10-10T19:28:09.629 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-aix.gperf 2023-10-10T19:28:09.630 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/buffer-lcm.c 2023-10-10T19:28:09.631 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unistr.h 2023-10-10T19:28:09.632 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/modechange.c 2023-10-10T19:28:09.633 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/tempname.h 2023-10-10T19:28:09.634 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/faccessat.c 2023-10-10T19:28:09.635 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unicodeio.h 2023-10-10T19:28:09.636 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/filemode.c 2023-10-10T19:28:09.637 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strdup.c 2023-10-10T19:28:09.638 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strsignal.c 2023-10-10T19:28:09.639 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/euidaccess.c 2023-10-10T19:28:09.640 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/select.c 2023-10-10T19:28:09.641 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memcoll.h 2023-10-10T19:28:09.641 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fseeko.c 2023-10-10T19:28:09.643 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/langinfo.in.h 2023-10-10T19:28:09.644 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getopt.in.h 2023-10-10T19:28:09.645 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/localcharset.c 2023-10-10T19:28:09.646 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isapipe.c 2023-10-10T19:28:09.646 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/settime.c 2023-10-10T19:28:09.648 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/bitrotate.h 2023-10-10T19:28:09.648 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isnanl-nolibm.h 2023-10-10T19:28:09.649 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fcntl.c 2023-10-10T19:28:09.650 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/exclude.h 2023-10-10T19:28:09.651 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/symlinkat.c 2023-10-10T19:28:09.651 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fpending.c 2023-10-10T19:28:09.652 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/savewd.c 2023-10-10T19:28:09.652 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fchownat.c 2023-10-10T19:28:09.653 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/root-dev-ino.h 2023-10-10T19:28:09.654 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/randread.h 2023-10-10T19:28:09.655 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freadseek.h 2023-10-10T19:28:09.656 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strtoll.c 2023-10-10T19:28:09.657 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdio-safer.h 2023-10-10T19:28:09.657 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strtoumax.c 2023-10-10T19:28:09.658 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/argv-iter.h 2023-10-10T19:28:09.659 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirent-safer.h 2023-10-10T19:28:09.660 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirent.in.h 2023-10-10T19:28:09.660 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/malloca.valgrind 2023-10-10T19:28:09.661 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/savewd.h 2023-10-10T19:28:09.662 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/uinttostr.c 2023-10-10T19:28:09.663 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getline.c 2023-10-10T19:28:09.664 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/chdir-long.h 2023-10-10T19:28:09.665 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/cycle-check.h 2023-10-10T19:28:09.665 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fts-cycle.c 2023-10-10T19:28:09.666 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/file-has-acl.c 2023-10-10T19:28:09.667 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/areadlink.h 2023-10-10T19:28:09.668 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sig2str.h 2023-10-10T19:28:09.669 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memcpy.c 2023-10-10T19:28:09.670 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freading.h 2023-10-10T19:28:09.670 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/full-write.h 2023-10-10T19:28:09.671 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unistd.in.h 2023-10-10T19:28:09.672 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/symlink.c 2023-10-10T19:28:09.673 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/se-selinux.in.h 2023-10-10T19:28:09.674 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/write-any-file.h 2023-10-10T19:28:09.674 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/printf-frexpl.h 2023-10-10T19:28:09.675 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/Makefile.in 2023-10-10T19:28:09.677 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/filenamecat.h 2023-10-10T19:28:09.677 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/malloca.c 2023-10-10T19:28:09.678 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/utimecmp.h 2023-10-10T19:28:09.679 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdlib.in.h 2023-10-10T19:28:09.680 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdio.in.h 2023-10-10T19:28:09.681 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbsinit.c 2023-10-10T19:28:09.682 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/diacrit.c 2023-10-10T19:28:09.682 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/se-context.in.h 2023-10-10T19:28:09.683 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xvasprintf.c 2023-10-10T19:28:09.684 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xgetcwd.c 2023-10-10T19:28:09.685 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sockets.h 2023-10-10T19:28:09.686 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/striconv.h 2023-10-10T19:28:09.686 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mkstemp-safer.c 2023-10-10T19:28:09.687 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/u64.h 2023-10-10T19:28:09.688 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtoimax.c 2023-10-10T19:28:09.689 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/time_r.c 2023-10-10T19:28:09.689 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbchar.h 2023-10-10T19:28:09.690 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xtime.h 2023-10-10T19:28:09.691 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/set-mode-acl.c 2023-10-10T19:28:09.693 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/version-etc.c 2023-10-10T19:28:09.693 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dup2.c 2023-10-10T19:28:09.694 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unlinkat.c 2023-10-10T19:28:09.695 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sigprocmask.c 2023-10-10T19:28:09.696 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/hash-triple.h 2023-10-10T19:28:09.697 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fprintftime.c 2023-10-10T19:28:09.698 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unistd--.h 2023-10-10T19:28:09.699 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/umaxtostr.c 2023-10-10T19:28:09.699 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/i-ring.c 2023-10-10T19:28:09.700 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/signbitd.c 2023-10-10T19:28:09.701 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/ftello.c 2023-10-10T19:28:09.931 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/lchown.c 2023-10-10T19:28:09.969 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/lstat.c 2023-10-10T19:28:09.971 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fpurge.c 2023-10-10T19:28:10.264 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fstatat.c 2023-10-10T19:28:10.265 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/lseek.c 2023-10-10T19:28:10.266 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fchmodat.c 2023-10-10T19:28:10.266 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/alloca.c 2023-10-10T19:28:10.267 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memcmp.c 2023-10-10T19:28:10.268 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrndup.h 2023-10-10T19:28:10.269 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/frexp.c 2023-10-10T19:28:10.270 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/nproc.h 2023-10-10T19:28:10.270 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fsusage.h 2023-10-10T19:28:10.271 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fd-safer-flag.c 2023-10-10T19:28:10.272 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getdtablesize.c 2023-10-10T19:28:10.273 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/savedir.c 2023-10-10T19:28:10.274 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/wcwidth.c 2023-10-10T19:28:10.275 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbsstr.c 2023-10-10T19:28:10.276 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fpucw.h 2023-10-10T19:28:10.276 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-hpux.h 2023-10-10T19:28:10.277 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/readtokens0.h 2023-10-10T19:28:10.278 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getdate.c 2023-10-10T19:28:10.279 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/utimens.c 2023-10-10T19:28:10.280 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getndelim2.c 2023-10-10T19:28:10.281 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fflush.c 2023-10-10T19:28:10.281 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/version-etc-fsf.c 2023-10-10T19:28:10.282 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xsize.h 2023-10-10T19:28:10.283 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xgethostname.c 2023-10-10T19:28:10.413 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strnlen1.h 2023-10-10T19:28:10.504 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isnanl.c 2023-10-10T19:28:10.505 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/same-inode.h 2023-10-10T19:28:10.506 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/regex_internal.h 2023-10-10T19:28:10.507 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/full-read.h 2023-10-10T19:28:10.507 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/intprops.h 2023-10-10T19:28:10.508 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fsusage.c 2023-10-10T19:28:10.509 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/save-cwd.c 2023-10-10T19:28:10.510 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memchr.valgrind 2023-10-10T19:28:10.511 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/exitfail.h 2023-10-10T19:28:10.511 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/printf-frexp.h 2023-10-10T19:28:10.512 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isblank.c 2023-10-10T19:28:10.513 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/arpa_inet.in.h 2023-10-10T19:28:10.514 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strerror.c 2023-10-10T19:28:10.514 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freadptr.c 2023-10-10T19:28:10.515 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/openat.h 2023-10-10T19:28:10.516 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-irix.h 2023-10-10T19:28:10.517 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/open.c 2023-10-10T19:28:10.518 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/vprintf.c 2023-10-10T19:28:10.519 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/same.h 2023-10-10T19:28:10.519 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/gai_strerror.c 2023-10-10T19:28:10.520 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/imaxtostr.c 2023-10-10T19:28:10.885 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memcmp2.h 2023-10-10T19:28:10.886 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xfreopen.c 2023-10-10T19:28:10.887 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/rand-isaac.c 2023-10-10T19:28:10.888 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/full-write.c 2023-10-10T19:28:10.889 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/rmdir.c 2023-10-10T19:28:10.890 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/save-cwd.h 2023-10-10T19:28:10.891 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/at-func2.c 2023-10-10T19:28:10.892 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/gethrxtime.c 2023-10-10T19:28:10.893 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strtod.c 2023-10-10T19:28:10.894 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbsalign.c 2023-10-10T19:28:10.895 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-osf.gperf 2023-10-10T19:28:10.895 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/ref-add.sin 2023-10-10T19:28:10.896 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memcmp2.c 2023-10-10T19:28:10.897 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/gnulib.mk 2023-10-10T19:28:10.898 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/ref-del.sin 2023-10-10T19:28:10.899 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stpncpy.c 2023-10-10T19:28:10.900 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv.in.h 2023-10-10T19:28:10.900 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/posixtm.h 2023-10-10T19:28:10.901 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtol.c 2023-10-10T19:28:10.902 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getopt1.c 2023-10-10T19:28:10.903 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/quotearg.h 2023-10-10T19:28:10.904 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/cycle-check.c 2023-10-10T19:28:10.905 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/calloc.c 2023-10-10T19:28:10.906 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/areadlinkat.c 2023-10-10T19:28:10.907 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/hash-triple.c 2023-10-10T19:28:10.908 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getpass.c 2023-10-10T19:28:10.909 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstriconv.c 2023-10-10T19:28:10.910 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirent--.h 2023-10-10T19:28:10.911 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sigaction.c 2023-10-10T19:28:10.911 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/linkat.c 2023-10-10T19:28:10.912 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unitypes.h 2023-10-10T19:28:10.913 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/linebuffer.c 2023-10-10T19:28:10.914 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fnmatch.c 2023-10-10T19:28:10.915 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xasprintf.c 2023-10-10T19:28:10.916 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/printf-parse.h 2023-10-10T19:28:10.916 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/string.in.h 2023-10-10T19:28:10.917 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mkdir.c 2023-10-10T19:28:10.918 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fchown-stub.c 2023-10-10T19:28:10.919 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/human.c 2023-10-10T19:28:10.920 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sockets.c 2023-10-10T19:28:10.921 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/canonicalize.h 2023-10-10T19:28:10.922 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getopt_int.h 2023-10-10T19:28:10.923 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/version-etc.h 2023-10-10T19:28:10.923 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdlib--.h 2023-10-10T19:28:10.924 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/backupfile.h 2023-10-10T19:28:10.925 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/file-type.c 2023-10-10T19:28:10.926 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/file-set.h 2023-10-10T19:28:10.926 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbuiter.h 2023-10-10T19:28:10.927 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fcntl.in.h 2023-10-10T19:28:10.928 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/inet_ntop.c 2023-10-10T19:28:10.929 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fseterr.c 2023-10-10T19:28:10.930 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/close-stream.h 2023-10-10T19:28:10.930 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freopen-safer.c 2023-10-10T19:28:10.931 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stat-macros.h 2023-10-10T19:28:10.932 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirname.c 2023-10-10T19:28:10.933 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dev-ino.h 2023-10-10T19:28:10.934 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xalloc.h 2023-10-10T19:28:10.935 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memcasecmp.h 2023-10-10T19:28:10.935 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/hash.c 2023-10-10T19:28:10.936 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/userspec.c 2023-10-10T19:28:10.937 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/hard-locale.h 2023-10-10T19:28:10.938 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xvasprintf.h 2023-10-10T19:28:10.939 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/wcrtomb.c 2023-10-10T19:28:10.940 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/gethrxtime.h 2023-10-10T19:28:10.941 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fprintftime.h 2023-10-10T19:28:10.942 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mgetgroups.c 2023-10-10T19:28:10.943 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/randread.c 2023-10-10T19:28:10.943 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstriconv.h 2023-10-10T19:28:10.945 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dup-safer-flag.c 2023-10-10T19:28:10.945 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/yesno.h 2023-10-10T19:28:10.946 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbscasecmp.c 2023-10-10T19:28:10.947 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strtoimax.c 2023-10-10T19:28:10.948 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbsrtowcs-state.c 2023-10-10T19:28:10.949 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mkstemp.c 2023-10-10T19:28:10.949 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xmemcoll.c 2023-10-10T19:28:10.950 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strintcmp.c 2023-10-10T19:28:10.951 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fclose.c 2023-10-10T19:28:10.952 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fd-reopen.c 2023-10-10T19:28:10.953 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/inttostr.c 2023-10-10T19:28:10.953 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/c-ctype.c 2023-10-10T19:28:10.955 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/openat-die.c 2023-10-10T19:28:10.956 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/error.c 2023-10-10T19:28:10.957 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sig2str.c 2023-10-10T19:28:10.958 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unlocked-io.h 2023-10-10T19:28:10.959 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memxfrm.c 2023-10-10T19:28:10.960 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/pipe-safer.c 2023-10-10T19:28:10.960 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/backupfile.c 2023-10-10T19:28:10.962 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-osf.h 2023-10-10T19:28:10.962 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/linebuffer.h 2023-10-10T19:28:10.963 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xprintf.h 2023-10-10T19:28:10.964 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/exclude.c 2023-10-10T19:28:10.966 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/userspec.h 2023-10-10T19:28:11.301 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mkancesdirs.h 2023-10-10T19:28:11.302 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/hash-pjw.c 2023-10-10T19:28:11.303 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fd-safer.c 2023-10-10T19:28:11.304 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/long-options.h 2023-10-10T19:28:11.305 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freadahead.h 2023-10-10T19:28:11.305 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/gethostname.c 2023-10-10T19:28:11.306 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/putenv.c 2023-10-10T19:28:11.307 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbrlen.c 2023-10-10T19:28:11.308 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/group-member.c 2023-10-10T19:28:11.309 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sha512.c 2023-10-10T19:28:11.310 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/signal.in.h 2023-10-10T19:28:11.311 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mktime.c 2023-10-10T19:28:11.312 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/readlink.c 2023-10-10T19:28:11.313 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mkdir-p.c 2023-10-10T19:28:11.313 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getndelim2.h 2023-10-10T19:28:11.314 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memxfrm.h 2023-10-10T19:28:11.315 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/c-strtod.c 2023-10-10T19:28:11.316 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fseterr.h 2023-10-10T19:28:11.317 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-hpux.gperf 2023-10-10T19:28:11.317 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/filenamecat.c 2023-10-10T19:28:11.318 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memmove.c 2023-10-10T19:28:11.319 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/str-kmp.h 2023-10-10T19:28:11.320 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/base64.c 2023-10-10T19:28:11.321 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/close-hook.c 2023-10-10T19:28:11.583 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/dirchownmod.c 2023-10-10T19:28:11.583 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/openat-safer.c 2023-10-10T19:28:11.584 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/posixtm.c 2023-10-10T19:28:11.585 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/printf-frexp.c 2023-10-10T19:28:11.586 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/safe-read.h 2023-10-10T19:28:11.587 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/regexec.c 2023-10-10T19:28:11.588 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/uniwidth.h 2023-10-10T19:28:11.589 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/write-any-file.c 2023-10-10T19:28:11.590 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-aix.h 2023-10-10T19:28:11.591 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/unicodeio.c 2023-10-10T19:28:11.591 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/gettime.c 2023-10-10T19:28:11.592 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strnumcmp.h 2023-10-10T19:28:11.593 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/btowc.c 2023-10-10T19:28:11.594 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sys_time.in.h 2023-10-10T19:28:11.594 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isnan.c 2023-10-10T19:28:11.595 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getaddrinfo.c 2023-10-10T19:28:11.596 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getpagesize.c 2023-10-10T19:28:11.597 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/posixver.h 2023-10-10T19:28:11.598 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/openat.c 2023-10-10T19:28:11.598 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/closeout.c 2023-10-10T19:28:11.599 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fopen.c 2023-10-10T19:28:11.600 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memchr2.valgrind 2023-10-10T19:28:11.601 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/time.in.h 2023-10-10T19:28:11.602 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/physmem.h 2023-10-10T19:28:11.603 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/iconv_open-irix.gperf 2023-10-10T19:28:11.604 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stat.c 2023-10-10T19:28:11.604 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/siglist.h 2023-10-10T19:28:11.606 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbswidth.h 2023-10-10T19:28:11.606 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtod.h 2023-10-10T19:28:11.607 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/md5.c 2023-10-10T19:28:11.608 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getloadavg.c 2023-10-10T19:28:11.609 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/getdate.h 2023-10-10T19:28:11.609 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdlib-safer.h 2023-10-10T19:28:11.610 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/remove.c 2023-10-10T19:28:11.611 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/malloc.c 2023-10-10T19:28:11.612 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/progname.h 2023-10-10T19:28:11.612 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/argmatch.h 2023-10-10T19:28:11.613 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/ChangeLog-2007 2023-10-10T19:28:11.615 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mbswidth.c 2023-10-10T19:28:11.616 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/error.h 2023-10-10T19:28:11.617 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/obstack.c 2023-10-10T19:28:11.618 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/acl.h 2023-10-10T19:28:11.618 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strnumcmp-in.h 2023-10-10T19:28:11.619 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xstrtoul.c 2023-10-10T19:28:11.620 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/float+.h 2023-10-10T19:28:11.621 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/canonicalize.c 2023-10-10T19:28:11.622 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fcntl--.h 2023-10-10T19:28:11.622 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/wchar.in.h 2023-10-10T19:28:11.623 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/trim.c 2023-10-10T19:28:11.624 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/rpmatch.c 2023-10-10T19:28:11.625 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isapipe.h 2023-10-10T19:28:11.625 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/memchr.c 2023-10-10T19:28:11.626 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/localcharset.h 2023-10-10T19:28:11.627 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/sig-handler.h 2023-10-10T19:28:11.886 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/openat-proc.c 2023-10-10T19:28:11.887 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/pathmax.h 2023-10-10T19:28:11.888 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/close-stream.c 2023-10-10T19:28:11.889 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/readutmp.c 2023-10-10T19:28:11.890 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/float.in.h 2023-10-10T19:28:11.890 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/creat-safer.c 2023-10-10T19:28:11.891 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/randperm.c 2023-10-10T19:28:11.892 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/readtokens.c 2023-10-10T19:28:11.893 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/safe-read.c 2023-10-10T19:28:11.893 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xreadlink.h 2023-10-10T19:28:11.894 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/stdarg.in.h 2023-10-10T19:28:11.895 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xalloc-die.c 2023-10-10T19:28:11.896 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/copy-acl.c 2023-10-10T19:28:11.897 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fsync.c 2023-10-10T19:28:11.898 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/strnumcmp.c 2023-10-10T19:28:12.160 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/selinux-at.c 2023-10-10T19:28:12.160 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/realloc.c 2023-10-10T19:28:12.162 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fdopendir.c 2023-10-10T19:28:12.163 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/alloca.in.h 2023-10-10T19:28:12.164 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/freopen.c 2023-10-10T19:28:12.164 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/mkdir-p.h 2023-10-10T19:28:12.165 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/isnand-nolibm.h 2023-10-10T19:28:12.431 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/propername.c 2023-10-10T19:28:12.431 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/fileblocks.c 2023-10-10T19:28:12.432 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xmemcoll.h 2023-10-10T19:28:12.433 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/hash.h 2023-10-10T19:28:12.434 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/c-strtold.c 2023-10-10T19:28:12.435 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/c-ctype.h 2023-10-10T19:28:12.436 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/lib/xmemxfrm.c 2023-10-10T19:28:12.437 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_require_config_h_first 2023-10-10T19:28:12.437 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ 2023-10-10T19:28:12.439 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/pt.po 2023-10-10T19:28:12.442 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/sl.po 2023-10-10T19:28:12.445 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/zh_TW.po 2023-10-10T19:28:12.448 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/sk.po 2023-10-10T19:28:12.452 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/el.gmo 2023-10-10T19:28:12.453 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/en.gmo 2023-10-10T19:28:12.454 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/lg.po 2023-10-10T19:28:12.457 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/sv.gmo 2023-10-10T19:28:12.460 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/pl.gmo 2023-10-10T19:28:12.462 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/insert-header.sin 2023-10-10T19:28:12.463 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/sk.gmo 2023-10-10T19:28:12.464 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ja.gmo 2023-10-10T19:28:12.465 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ko.po 2023-10-10T19:28:12.468 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/da.gmo 2023-10-10T19:28:12.470 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/LINGUAS 2023-10-10T19:28:12.471 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/zh_TW.gmo 2023-10-10T19:28:12.472 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/tr.gmo 2023-10-10T19:28:12.474 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/gl.gmo 2023-10-10T19:28:12.475 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/fi.po 2023-10-10T19:28:12.478 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/da.po 2023-10-10T19:28:12.480 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/fi.gmo 2023-10-10T19:28:12.482 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/quot.sed 2023-10-10T19:28:12.483 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/stamp-po 2023-10-10T19:28:12.484 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ms.gmo 2023-10-10T19:28:12.485 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/pt_BR.po 2023-10-10T19:28:12.487 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/Makevars 2023-10-10T19:28:12.489 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/boldquot.sed 2023-10-10T19:28:12.490 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/coreutils.pot 2023-10-10T19:28:12.491 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/de.po 2023-10-10T19:28:12.495 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/pt.gmo 2023-10-10T19:28:12.497 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/Makefile.in.in 2023-10-10T19:28:12.498 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/nl.gmo 2023-10-10T19:28:12.500 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/fr.gmo 2023-10-10T19:28:12.502 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ga.gmo 2023-10-10T19:28:12.504 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/cs.po 2023-10-10T19:28:12.507 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ru.po 2023-10-10T19:28:12.511 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/hu.po 2023-10-10T19:28:12.514 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/id.po 2023-10-10T19:28:12.517 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ru.gmo 2023-10-10T19:28:12.520 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/sl.gmo 2023-10-10T19:28:12.523 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/remove-potcdate.sin 2023-10-10T19:28:12.524 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/hu.gmo 2023-10-10T19:28:12.526 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/et.po 2023-10-10T19:28:12.529 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/gl.po 2023-10-10T19:28:12.533 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/vi.gmo 2023-10-10T19:28:12.535 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/bg.po 2023-10-10T19:28:12.538 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/pl.po 2023-10-10T19:28:12.542 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/nl.po 2023-10-10T19:28:12.544 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/pt_BR.gmo 2023-10-10T19:28:12.547 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ko.gmo 2023-10-10T19:28:12.547 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/de.gmo 2023-10-10T19:28:12.550 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/POTFILES.in 2023-10-10T19:28:12.551 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/sv.po 2023-10-10T19:28:12.553 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/it.po 2023-10-10T19:28:12.556 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/eu.gmo 2023-10-10T19:28:12.557 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/en.po 2023-10-10T19:28:12.559 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/it.gmo 2023-10-10T19:28:12.561 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/en@boldquot.header 2023-10-10T19:28:12.562 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/et.gmo 2023-10-10T19:28:12.563 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ja.po 2023-10-10T19:28:12.567 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/zh_CN.gmo 2023-10-10T19:28:12.569 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/uk.po 2023-10-10T19:28:12.573 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ca.gmo 2023-10-10T19:28:12.575 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/Rules-quot 2023-10-10T19:28:12.576 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ro.po 2023-10-10T19:28:12.578 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/nb.gmo 2023-10-10T19:28:12.579 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/bg.gmo 2023-10-10T19:28:12.581 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/be.po 2023-10-10T19:28:12.583 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/vi.po 2023-10-10T19:28:12.586 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/lt.gmo 2023-10-10T19:28:12.587 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/uk.gmo 2023-10-10T19:28:12.589 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/id.gmo 2023-10-10T19:28:12.592 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/af.po 2023-10-10T19:28:12.594 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/eu.po 2023-10-10T19:28:12.596 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ms.po 2023-10-10T19:28:12.598 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/en@quot.header 2023-10-10T19:28:12.600 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/af.gmo 2023-10-10T19:28:12.601 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/es.gmo 2023-10-10T19:28:12.602 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ga.po 2023-10-10T19:28:12.605 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ro.gmo 2023-10-10T19:28:12.607 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ca.po 2023-10-10T19:28:12.609 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/lg.gmo 2023-10-10T19:28:12.612 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/zh_CN.po 2023-10-10T19:28:12.615 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/cs.gmo 2023-10-10T19:28:12.618 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/nb.po 2023-10-10T19:28:12.620 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/ChangeLog-2007 2023-10-10T19:28:12.621 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/tr.po 2023-10-10T19:28:12.624 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/be.gmo 2023-10-10T19:28:12.625 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/fr.po 2023-10-10T19:28:12.627 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/el.po 2023-10-10T19:28:12.631 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/lt.po 2023-10-10T19:28:12.633 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/po/es.po 2023-10-10T19:28:12.636 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ 2023-10-10T19:28:12.638 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/ 2023-10-10T19:28:12.639 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/skip-seek 2023-10-10T19:28:12.640 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/unblock-sync 2023-10-10T19:28:12.640 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/skip-seek2 2023-10-10T19:28:12.641 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/unblock 2023-10-10T19:28:12.642 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/stderr 2023-10-10T19:28:12.643 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/direct 2023-10-10T19:28:12.643 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/skip-seek-past-file 2023-10-10T19:28:12.644 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/not-rewound 2023-10-10T19:28:12.645 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/skip-seek-past-dev 2023-10-10T19:28:12.646 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/misc 2023-10-10T19:28:12.647 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/dd/reblock 2023-10-10T19:28:12.648 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ 2023-10-10T19:28:12.649 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tt-FF 2023-10-10T19:28:12.650 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/t_tab_ 2023-10-10T19:28:13.342 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/FF 2023-10-10T19:28:13.343 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/FnFn 2023-10-10T19:28:13.344 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3ml24f-bl-tn 2023-10-10T19:28:13.345 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+5-8b3l17f-FF 2023-10-10T19:28:13.346 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+3ml24f-bl-tn 2023-10-10T19:28:13.347 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/FFtn 2023-10-10T19:28:13.348 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a2l15-FF 2023-10-10T19:28:13.349 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tFnFt 2023-10-10T19:28:13.350 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tn2e5o3-t_tab 2023-10-10T19:28:13.351 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3ml24f-t 2023-10-10T19:28:13.352 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nSml20-t-tFFFF 2023-10-10T19:28:13.353 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tn2e8-t_tab 2023-10-10T19:28:13.355 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b3f-FF 2023-10-10T19:28:13.356 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/2w60f-t_notab 2023-10-10T19:28:13.357 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/FFn 2023-10-10T19:28:13.358 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/t_tab 2023-10-10T19:28:13.358 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b3-0F 2023-10-10T19:28:13.359 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+4-8a2l17-FF 2023-10-10T19:28:13.360 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+4b2l17f-0FF 2023-10-10T19:28:13.361 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3a3l15-t 2023-10-10T19:28:13.362 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tFFn 2023-10-10T19:28:13.363 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3l24f-t 2023-10-10T19:28:13.363 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/0FnFt 2023-10-10T19:28:13.364 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3-FF 2023-10-10T19:28:13.365 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/t-FF 2023-10-10T19:28:13.366 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nSml24-bl-FF 2023-10-10T19:28:13.367 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+3l24f-bl 2023-10-10T19:28:13.367 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b3l15-t 2023-10-10T19:28:13.368 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3b3f-FF 2023-10-10T19:28:13.369 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3b3f-0F 2023-10-10T19:28:13.370 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W35b3l24f-lm 2023-10-10T19:28:13.371 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/0Ft 2023-10-10T19:28:13.371 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tt-bl 2023-10-10T19:28:13.372 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3ml24-t-FF 2023-10-10T19:28:13.373 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tFFt-lm 2023-10-10T19:28:13.374 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nSml20-bl-FF 2023-10-10T19:28:13.375 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tn2e8o3-t_tab 2023-10-10T19:28:13.376 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W28l24f-ll 2023-10-10T19:28:13.377 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nSml24-t-tFFFF 2023-10-10T19:28:13.377 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/t-0FF 2023-10-10T19:28:13.378 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tb3-FF 2023-10-10T19:28:13.379 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3b3Sl24f-tn 2023-10-10T19:28:13.380 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3mSl24f-bl-tn 2023-10-10T19:28:13.381 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3b2l17-FF 2023-10-10T19:28:13.381 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/2f-t_notab 2023-10-10T19:28:13.382 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b2l15-FF 2023-10-10T19:28:13.383 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3b3f-0FF 2023-10-10T19:28:13.384 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a3l15-t 2023-10-10T19:28:18.639 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3Jml24f-lm-lo 2023-10-10T19:28:18.643 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tb3-0FF 2023-10-10T19:28:18.644 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/l24-t 2023-10-10T19:28:18.645 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W35ml24f-lm-lo 2023-10-10T19:28:18.647 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a3-0F 2023-10-10T19:28:18.648 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ml24f-0F 2023-10-10T19:28:18.649 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/loli 2023-10-10T19:28:18.651 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3ml24-t 2023-10-10T19:28:18.652 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a3l24f-lm 2023-10-10T19:28:18.653 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/Jb3l24f-lm 2023-10-10T19:28:18.654 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/t-t 2023-10-10T19:28:18.655 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/0F 2023-10-10T19:28:18.656 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/w72l24f-ll 2023-10-10T19:28:18.657 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ml24f-t 2023-10-10T19:28:18.658 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b3l15f-t 2023-10-10T19:28:18.660 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/0FnFnt 2023-10-10T19:28:18.661 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b3l24f-lm 2023-10-10T19:28:18.662 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3mSnl24fbltn 2023-10-10T19:28:18.664 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nJml24f-lmlolm 2023-10-10T19:28:18.665 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3a3Sl24f-tn 2023-10-10T19:28:18.666 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tne8-t_tab 2023-10-10T19:28:18.667 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3ml24-FF 2023-10-10T19:28:18.669 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a3f-0FF 2023-10-10T19:28:18.670 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ml20-FF-t 2023-10-10T19:28:18.671 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/pr-tests 2023-10-10T19:28:18.672 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W35Jb3l24f-lm 2023-10-10T19:28:18.673 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ttb3-0FF 2023-10-10T19:28:18.675 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nl24f-bl 2023-10-10T19:28:18.676 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nSml24-t-t-FF 2023-10-10T19:28:18.677 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3b3l15f-t 2023-10-10T19:28:18.678 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/0FF 2023-10-10T19:28:18.679 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3f-0F 2023-10-10T19:28:18.680 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/Ja3l24f-lm 2023-10-10T19:28:18.681 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3a2l17-FF 2023-10-10T19:28:18.682 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ml24f-t-0F 2023-10-10T19:28:18.683 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tn_2e8-t_tab 2023-10-10T19:28:18.684 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3a3Snl24f-tn 2023-10-10T19:28:18.685 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/2s_w60f-t_nota 2023-10-10T19:28:18.687 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/t-bl 2023-10-10T19:28:18.688 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ml24-t-FF 2023-10-10T19:28:18.689 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+3-7l24-FF 2023-10-10T19:28:18.690 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/t_notab 2023-10-10T19:28:18.691 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W72Jl24f-ll 2023-10-10T19:28:18.693 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a2l17-FF 2023-10-10T19:28:18.693 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ml24-FF 2023-10-10T19:28:18.694 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b3f-0F 2023-10-10T19:28:18.696 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+3l24f-0FF 2023-10-10T19:28:18.697 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+3ml20f-bl-FF 2023-10-10T19:28:18.698 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a3f-0F 2023-10-10T19:28:18.699 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W35Jml24f-lmlo 2023-10-10T19:28:18.700 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W27l24f-ll 2023-10-10T19:28:18.701 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ml24-t 2023-10-10T19:28:18.702 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a3f-FF 2023-10-10T19:28:18.704 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tFFt-bl 2023-10-10T19:28:18.705 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tta3-FF 2023-10-10T19:28:18.706 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3-0F 2023-10-10T19:28:18.707 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/2sf-t_notab 2023-10-10T19:28:18.708 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3a3l15f-t 2023-10-10T19:28:18.709 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W26l24f-ll 2023-10-10T19:28:18.711 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+7l24-FF 2023-10-10T19:28:18.712 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ml24f-lm-lo 2023-10-10T19:28:18.713 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/l24-FF 2023-10-10T19:28:18.714 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+2-5l24f-0FF 2023-10-10T19:28:18.715 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3f-FF 2023-10-10T19:28:18.716 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+3ml24f-tn-bl 2023-10-10T19:28:18.717 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ta3-0FF 2023-10-10T19:28:18.718 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3b3l15-t 2023-10-10T19:28:18.720 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+2l24f-0FF 2023-10-10T19:28:18.721 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tn_2e8S-t_tab 2023-10-10T19:28:18.722 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ta3-FF 2023-10-10T19:28:18.723 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tFFt-ll 2023-10-10T19:28:18.724 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+6b3l13f-FF 2023-10-10T19:28:18.725 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/2-Sf-t_notab 2023-10-10T19:28:18.726 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/0FFnt 2023-10-10T19:28:18.727 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W20l24f-ll 2023-10-10T19:28:18.728 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/Jml24f-lm-lo 2023-10-10T19:28:18.729 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3l24-t 2023-10-10T19:28:18.730 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3b3l24f-tn 2023-10-10T19:28:18.731 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W35a3l24f-lm 2023-10-10T19:28:18.733 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/2-S_f-t_notab 2023-10-10T19:28:18.734 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3-5l24f-t 2023-10-10T19:28:18.735 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3b3Snl24f-tn 2023-10-10T19:28:18.736 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+2l24f-bl 2023-10-10T19:28:18.738 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W35Ja3l24f-lm 2023-10-10T19:28:18.739 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+8l20-FF 2023-10-10T19:28:18.740 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/0FFt 2023-10-10T19:28:18.741 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/o3a3l24f-tn 2023-10-10T19:28:18.742 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/2s_f-t_notab 2023-10-10T19:28:18.743 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nN15l24f-bl 2023-10-10T19:28:18.744 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/0Fnt 2023-10-10T19:28:18.745 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/4-7l24-FF 2023-10-10T19:28:18.746 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tn 2023-10-10T19:28:18.747 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b2l17-FF 2023-10-10T19:28:18.748 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nJml24f-lmlmlo 2023-10-10T19:28:18.749 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tta3-0FF 2023-10-10T19:28:18.750 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+5a3l13f-0FF 2023-10-10T19:28:18.751 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nN1+3l24f-bl 2023-10-10T19:28:18.752 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/2sw60f-t_notab 2023-10-10T19:28:18.753 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/W-72l24f-ll 2023-10-10T19:28:18.754 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tt-0FF 2023-10-10T19:28:18.755 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/4l24-FF 2023-10-10T19:28:18.756 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/3a3f-0F 2023-10-10T19:28:18.798 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tFFt 2023-10-10T19:28:18.799 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/n+6a2l17-FF 2023-10-10T19:28:18.799 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/a3l15f-t 2023-10-10T19:28:18.800 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/l24f-t 2023-10-10T19:28:18.801 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/b3f-0FF 2023-10-10T19:28:18.801 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tt-t 2023-10-10T19:28:18.802 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/ttb3-FF 2023-10-10T19:28:18.803 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/nSml20-t-t-FF 2023-10-10T19:28:18.804 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/pr/tne8o3-t_tab 2023-10-10T19:28:18.805 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/Coreutils.pm 2023-10-10T19:28:18.805 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rmdir/ 2023-10-10T19:28:18.806 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rmdir/t-slash 2023-10-10T19:28:18.807 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rmdir/ignore 2023-10-10T19:28:18.808 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rmdir/fail-perm 2023-10-10T19:28:18.808 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/ 2023-10-10T19:28:18.809 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/ignorable 2023-10-10T19:28:18.810 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/r-2 2023-10-10T19:28:18.810 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/fail-2eperm 2023-10-10T19:28:18.811 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/empty-inacc 2023-10-10T19:28:18.812 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/cycle 2023-10-10T19:28:18.813 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/sunos-1 2023-10-10T19:28:18.813 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/v-slash 2023-10-10T19:28:18.814 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/no-give-up 2023-10-10T19:28:18.815 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/dot-rel 2023-10-10T19:28:18.816 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/one-file-system 2023-10-10T19:28:18.817 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/i-never 2023-10-10T19:28:18.817 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/rm1 2023-10-10T19:28:18.818 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/unread2 2023-10-10T19:28:18.819 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/read-only 2023-10-10T19:28:18.819 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/deep-2 2023-10-10T19:28:18.820 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/r-4 2023-10-10T19:28:18.821 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/dir-no-w 2023-10-10T19:28:18.822 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/interactive-always 2023-10-10T19:28:18.822 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/one-file-system2 2023-10-10T19:28:18.823 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/unreadable 2023-10-10T19:28:18.824 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/r-1 2023-10-10T19:28:18.825 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/fail-eacces 2023-10-10T19:28:18.826 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/inaccessible 2023-10-10T19:28:18.826 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/unread3 2023-10-10T19:28:18.828 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/ir-1 2023-10-10T19:28:18.829 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/rm3 2023-10-10T19:28:18.829 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/rm5 2023-10-10T19:28:18.830 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/deep-1 2023-10-10T19:28:18.831 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/ext3-perf 2023-10-10T19:28:18.831 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/dangling-symlink 2023-10-10T19:28:18.832 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/isatty 2023-10-10T19:28:18.833 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/i-1 2023-10-10T19:28:18.833 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/dir-nonrecur 2023-10-10T19:28:18.834 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/interactive-once 2023-10-10T19:28:18.835 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/f-1 2023-10-10T19:28:18.836 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/fail-eperm 2023-10-10T19:28:18.836 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/empty-name 2023-10-10T19:28:18.837 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/hash 2023-10-10T19:28:18.838 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/i-no-r 2023-10-10T19:28:18.839 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/r-3 2023-10-10T19:28:18.839 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/readdir-bug 2023-10-10T19:28:18.840 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/rm2 2023-10-10T19:28:18.841 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/rm/rm4 2023-10-10T19:28:18.841 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/test-lib.sh 2023-10-10T19:28:18.842 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/sample-test 2023-10-10T19:28:18.843 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chown/ 2023-10-10T19:28:18.844 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chown/basic 2023-10-10T19:28:18.845 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chown/separator 2023-10-10T19:28:18.845 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chown/deref 2023-10-10T19:28:18.846 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chown/preserve-root 2023-10-10T19:28:18.847 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/id/ 2023-10-10T19:28:18.848 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/id/no-context 2023-10-10T19:28:18.849 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/CuTmpdir.pm 2023-10-10T19:28:18.850 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/Makefile.am 2023-10-10T19:28:18.850 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/ 2023-10-10T19:28:18.851 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/dir-slash 2023-10-10T19:28:18.852 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/preserve-2 2023-10-10T19:28:18.853 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/dir-vs-file 2023-10-10T19:28:18.854 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/preserve-slink-time 2023-10-10T19:28:18.855 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/same-file 2023-10-10T19:28:18.855 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/src-base-dot 2023-10-10T19:28:18.856 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/cp-a-selinux 2023-10-10T19:28:19.018 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/r-vs-symlink 2023-10-10T19:28:19.024 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/capability 2023-10-10T19:28:19.908 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:20.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.037+0000 7fe451c91700 1 Processor -- start 2023-10-10T19:28:20.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.037+0000 7fe451c91700 1 -- start start 2023-10-10T19:28:20.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe451c91700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe44c0c7190 0x7fe44c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe451c91700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 0x7fe44c063c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe451c91700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44c157030 con 0x7fe44c0c6330 2023-10-10T19:28:20.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe451c91700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44c0c5000 con 0x7fe44c0c7190 2023-10-10T19:28:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe451c91700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c156420 con 0x7fe44c0c4580 2023-10-10T19:28:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe44c0c7190 0x7fe44c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44affd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe44c0c7190 0x7fe44c15a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53718/0 (socket says 172.21.15.175:53718) 2023-10-10T19:28:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44affd700 1 -- 172.21.15.175:0/3926664196 learned_addr learned my addr 172.21.15.175:0/3926664196 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:20.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44bfff700 1 --2- 172.21.15.175:0/3926664196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 0x7fe44c063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44affd700 1 -- 172.21.15.175:0/3926664196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 msgr2=0x7fe44c063c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:20.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44affd700 1 --2- 172.21.15.175:0/3926664196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 0x7fe44c063c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44affd700 1 -- 172.21.15.175:0/3926664196 >> v1:172.21.15.191:6789/0 conn(0x7fe44c0c6330 legacy=0x7fe44c0c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:20.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44affd700 1 -- 172.21.15.175:0/3926664196 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c156700 con 0x7fe44c0c7190 2023-10-10T19:28:20.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44affd700 1 --2- 172.21.15.175:0/3926664196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c0c7190 0x7fe44c15a030 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe434009f20 tx=0x7fe434009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bc94e9febab7b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:20.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.038+0000 7fe44a7fc700 1 -- 172.21.15.175:0/3926664196 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43400e070 con 0x7fe44c0c7190 2023-10-10T19:28:20.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe44a7fc700 1 -- 172.21.15.175:0/3926664196 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43400bad0 con 0x7fe44c0c7190 2023-10-10T19:28:20.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe44a7fc700 1 -- 172.21.15.175:0/3926664196 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4340092e0 con 0x7fe44c0c7190 2023-10-10T19:28:20.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe451c91700 1 -- 172.21.15.175:0/3926664196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c0c7190 msgr2=0x7fe44c15a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe451c91700 1 --2- 172.21.15.175:0/3926664196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c0c7190 0x7fe44c15a030 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe434009f20 tx=0x7fe434009840 comp rx=0 tx=0).stop 2023-10-10T19:28:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe451c91700 1 -- 172.21.15.175:0/3926664196 shutdown_connections 2023-10-10T19:28:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe451c91700 1 --2- 172.21.15.175:0/3926664196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 0x7fe44c063c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe451c91700 1 --2- 172.21.15.175:0/3926664196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c0c7190 0x7fe44c15a030 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe451c91700 1 -- 172.21.15.175:0/3926664196 >> 172.21.15.175:0/3926664196 conn(0x7fe44c06ef20 msgr2=0x7fe44c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:20.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe451c91700 1 -- 172.21.15.175:0/3926664196 shutdown_connections 2023-10-10T19:28:20.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.039+0000 7fe451c91700 1 -- 172.21.15.175:0/3926664196 wait complete. 2023-10-10T19:28:20.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.040+0000 7fe451c91700 1 Processor -- start 2023-10-10T19:28:20.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.040+0000 7fe451c91700 1 -- start start 2023-10-10T19:28:20.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe451c91700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 0x7fe44c152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe451c91700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44c0c6330 0x7fe44c1666a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe451c91700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c14f590 0x7fe44c166be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe451c91700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c0cb410 con 0x7fe44c14f590 2023-10-10T19:28:20.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe451c91700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c151e40 con 0x7fe44c0c6330 2023-10-10T19:28:20.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe451c91700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c0c14e0 con 0x7fe44c0c4580 2023-10-10T19:28:20.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe44bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c14f590 0x7fe44c166be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe44bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c14f590 0x7fe44c166be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53722/0 (socket says 172.21.15.175:53722) 2023-10-10T19:28:20.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe44bfff700 1 -- 172.21.15.175:0/937696956 learned_addr learned my addr 172.21.15.175:0/937696956 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:20.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe44affd700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44c0c6330 0x7fe44c1666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.041+0000 7fe44b7fe700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 0x7fe44c152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe44bfff700 1 -- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 msgr2=0x7fe44c152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe44bfff700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 0x7fe44c152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe44bfff700 1 -- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44c0c6330 msgr2=0x7fe44c1666a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe44bfff700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44c0c6330 0x7fe44c1666a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe44bfff700 1 -- 172.21.15.175:0/937696956 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe434009ae0 con 0x7fe44c14f590 2023-10-10T19:28:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe44bfff700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c14f590 0x7fe44c166be0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fe44400d890 tx=0x7fe444008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe4437fe700 1 -- 172.21.15.175:0/937696956 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4440125f0 con 0x7fe44c14f590 2023-10-10T19:28:20.051 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe4437fe700 1 -- 172.21.15.175:0/937696956 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe444012750 con 0x7fe44c14f590 2023-10-10T19:28:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe4437fe700 1 -- 172.21.15.175:0/937696956 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444012b10 con 0x7fe44c14f590 2023-10-10T19:28:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44c1520c0 con 0x7fe44c14f590 2023-10-10T19:28:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.042+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44c167220 con 0x7fe44c14f590 2023-10-10T19:28:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.043+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44c152250 con 0x7fe44c14f590 2023-10-10T19:28:20.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.043+0000 7fe4437fe700 1 -- 172.21.15.175:0/937696956 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe444013480 con 0x7fe44c14f590 2023-10-10T19:28:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.044+0000 7fe4437fe700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe42c066f30 0x7fe42c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.044+0000 7fe4437fe700 1 -- 172.21.15.175:0/937696956 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7fe444085030 con 0x7fe44c14f590 2023-10-10T19:28:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.044+0000 7fe44b7fe700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe42c066f30 0x7fe42c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.044+0000 7fe44b7fe700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe42c066f30 0x7fe42c0693e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009af0 tx=0x7fe43c009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:20.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.048+0000 7fe4437fe700 1 -- 172.21.15.175:0/937696956 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe444051a70 con 0x7fe44c14f590 2023-10-10T19:28:20.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.274+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe44c067570 con 0x7fe44c14f590 2023-10-10T19:28:20.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.275+0000 7fe4437fe700 1 -- 172.21.15.175:0/937696956 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+6193 (secure 0 0 0) 0x7fe444055090 con 0x7fe44c14f590 2023-10-10T19:28:20.275 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:20.276 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":311}],"filesystems":[{"mdsmap":{"epoch":313,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:19.941989+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:20.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe42c066f30 msgr2=0x7fe42c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe42c066f30 0x7fe42c0693e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009af0 tx=0x7fe43c009060 comp rx=0 tx=0).stop 2023-10-10T19:28:20.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c14f590 msgr2=0x7fe44c166be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c14f590 0x7fe44c166be0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fe44400d890 tx=0x7fe444008040 comp rx=0 tx=0).stop 2023-10-10T19:28:20.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 shutdown_connections 2023-10-10T19:28:20.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe42c066f30 0x7fe42c0693e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe44c14f590 0x7fe44c166be0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe44c0c4580 0x7fe44c152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 --2- 172.21.15.175:0/937696956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44c0c6330 0x7fe44c1666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 >> 172.21.15.175:0/937696956 conn(0x7fe44c06ef20 msgr2=0x7fe44c14ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:20.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 shutdown_connections 2023-10-10T19:28:20.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.277+0000 7fe451c91700 1 -- 172.21.15.175:0/937696956 wait complete. 2023-10-10T19:28:20.281 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 313 2023-10-10T19:28:20.289 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=1) 2023-10-10T19:28:20.290 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-10T19:28:20.290 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:28:20.302 INFO:tasks.ceph.mds.d:Stopped 2023-10-10T19:28:20.302 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:20.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.416+0000 7f728dac9700 1 Processor -- start 2023-10-10T19:28:20.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.416+0000 7f728dac9700 1 -- start start 2023-10-10T19:28:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728dac9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f7288159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728dac9700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f72880cd040 0x7f72880684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728dac9700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7288067fa0 con 0x7f728808a100 2023-10-10T19:28:20.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728dac9700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72880c5600 con 0x7f72880cd040 2023-10-10T19:28:20.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728dac9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7288156b70 con 0x7f728814fd90 2023-10-10T19:28:20.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728cac7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f7288159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728d2c8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f72880cd040 0x7f72880684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728cac7700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f7288159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:49574/0 (socket says 172.21.15.175:49574) 2023-10-10T19:28:20.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728d2c8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f72880cd040 0x7f72880684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:47396/0 (socket says 172.21.15.175:47396) 2023-10-10T19:28:20.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.417+0000 7f728cac7700 1 -- 172.21.15.175:0/417050012 learned_addr learned my addr 172.21.15.175:0/417050012 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:20.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f728cac7700 1 -- 172.21.15.175:0/417050012 >> v2:172.21.15.191:3300/0 conn(0x7f72880cd040 msgr2=0x7f72880684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:20.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f728cac7700 1 --2- 172.21.15.175:0/417050012 >> v2:172.21.15.191:3300/0 conn(0x7f72880cd040 0x7f72880684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f728cac7700 1 -- 172.21.15.175:0/417050012 >> v1:172.21.15.191:6789/0 conn(0x7f728808a100 legacy=0x7f72880c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f728cac7700 1 -- 172.21.15.175:0/417050012 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7288150160 con 0x7f728814fd90 2023-10-10T19:28:20.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f728d2c8700 1 --2- 172.21.15.175:0/417050012 >> v2:172.21.15.191:3300/0 conn(0x7f72880cd040 0x7f72880684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:28:20.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f728cac7700 1 --2- 172.21.15.175:0/417050012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f7288159c30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72780099b0 tx=0x7f7278009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b0d3192122e679a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:20.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f72877fe700 1 -- 172.21.15.175:0/417050012 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7278009ea0 con 0x7f728814fd90 2023-10-10T19:28:20.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f72877fe700 1 -- 172.21.15.175:0/417050012 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727800b860 con 0x7f728814fd90 2023-10-10T19:28:20.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.418+0000 7f72877fe700 1 -- 172.21.15.175:0/417050012 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727800bc00 con 0x7f728814fd90 2023-10-10T19:28:20.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.419+0000 7f728dac9700 1 -- 172.21.15.175:0/417050012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 msgr2=0x7f7288159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.419+0000 7f728dac9700 1 --2- 172.21.15.175:0/417050012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f7288159c30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72780099b0 tx=0x7f7278009250 comp rx=0 tx=0).stop 2023-10-10T19:28:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.419+0000 7f728dac9700 1 -- 172.21.15.175:0/417050012 shutdown_connections 2023-10-10T19:28:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.419+0000 7f728dac9700 1 --2- 172.21.15.175:0/417050012 >> v2:172.21.15.191:3300/0 conn(0x7f72880cd040 0x7f72880684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.419+0000 7f728dac9700 1 --2- 172.21.15.175:0/417050012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f7288159c30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.419+0000 7f728dac9700 1 -- 172.21.15.175:0/417050012 >> 172.21.15.175:0/417050012 conn(0x7f7288063d00 msgr2=0x7f72880cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:20.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.419+0000 7f728dac9700 1 -- 172.21.15.175:0/417050012 shutdown_connections 2023-10-10T19:28:20.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.419+0000 7f728dac9700 1 -- 172.21.15.175:0/417050012 wait complete. 2023-10-10T19:28:20.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.420+0000 7f728dac9700 1 Processor -- start 2023-10-10T19:28:20.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.420+0000 7f728dac9700 1 -- start start 2023-10-10T19:28:20.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728dac9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f728808a100 0x7f7288152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728dac9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72880cd040 0x7f72880c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728dac9700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f72880c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728dac9700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7288166bf0 con 0x7f728808a100 2023-10-10T19:28:20.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728dac9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72881666c0 con 0x7f72880cd040 2023-10-10T19:28:20.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728dac9700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7288166910 con 0x7f728814fd90 2023-10-10T19:28:20.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f728808a100 0x7f7288152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f728808a100 0x7f7288152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53732/0 (socket says 172.21.15.175:53732) 2023-10-10T19:28:20.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 -- 172.21.15.175:0/2063832482 learned_addr learned my addr 172.21.15.175:0/2063832482 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:20.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f7287fff700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72880cd040 0x7f72880c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728d2c8700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f72880c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 -- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 msgr2=0x7f72880c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f72880c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 -- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72880cd040 msgr2=0x7f72880c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72880cd040 0x7f72880c7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 -- 172.21.15.175:0/2063832482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7280009a20 con 0x7f728808a100 2023-10-10T19:28:20.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728cac7700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f728808a100 0x7f7288152be0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f7288156cb0 tx=0x7f7278017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f7284ff9700 1 -- 172.21.15.175:0/2063832482 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727800bbe0 con 0x7f728808a100 2023-10-10T19:28:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f7284ff9700 1 -- 172.21.15.175:0/2063832482 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727801a450 con 0x7f728808a100 2023-10-10T19:28:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72780094f0 con 0x7f728808a100 2023-10-10T19:28:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f7284ff9700 1 -- 172.21.15.175:0/2063832482 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727801a810 con 0x7f728808a100 2023-10-10T19:28:20.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.421+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72880a8090 con 0x7f728808a100 2023-10-10T19:28:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.422+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72880a7c10 con 0x7f728808a100 2023-10-10T19:28:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.423+0000 7f7284ff9700 1 -- 172.21.15.175:0/2063832482 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f727801ab80 con 0x7f728808a100 2023-10-10T19:28:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.423+0000 7f7284ff9700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7274066f30 0x7f72740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.423+0000 7f7284ff9700 1 -- 172.21.15.175:0/2063832482 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f72780865a0 con 0x7f728808a100 2023-10-10T19:28:20.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.423+0000 7f7287fff700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7274066f30 0x7f72740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:20.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.424+0000 7f7287fff700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7274066f30 0x7f72740693e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f728814f5f0 tx=0x7f727c007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:20.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.427+0000 7f7284ff9700 1 -- 172.21.15.175:0/2063832482 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f72880a7c10 con 0x7f728808a100 2023-10-10T19:28:20.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.652+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72880a7c10 con 0x7f728808a100 2023-10-10T19:28:20.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.652+0000 7f7284ff9700 1 -- 172.21.15.175:0/2063832482 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+6193 (secure 0 0 0) 0x7f72880a7c10 con 0x7f728808a100 2023-10-10T19:28:20.653 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:20.653 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":311}],"filesystems":[{"mdsmap":{"epoch":313,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:19.941989+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:20.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7274066f30 msgr2=0x7f72740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7274066f30 0x7f72740693e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f728814f5f0 tx=0x7f727c007ec0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f728808a100 msgr2=0x7f7288152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:20.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f728808a100 0x7f7288152be0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f7288156cb0 tx=0x7f7278017000 comp rx=0 tx=0).stop 2023-10-10T19:28:20.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 shutdown_connections 2023-10-10T19:28:20.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f7274066f30 0x7f72740693e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f728814fd90 0x7f72880c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f728808a100 0x7f7288152be0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 --2- 172.21.15.175:0/2063832482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72880cd040 0x7f72880c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:20.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 >> 172.21.15.175:0/2063832482 conn(0x7f7288063d00 msgr2=0x7f7288154c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:20.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 shutdown_connections 2023-10-10T19:28:20.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:20.654+0000 7f728dac9700 1 -- 172.21.15.175:0/2063832482 wait complete. 2023-10-10T19:28:20.658 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 313 2023-10-10T19:28:20.666 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:28:22.668 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:22.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.795+0000 7f8dce38e700 1 Processor -- start 2023-10-10T19:28:22.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.795+0000 7f8dce38e700 1 -- start start 2023-10-10T19:28:22.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.795+0000 7f8dce38e700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8dc80b1fc0 0x7f8dc8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:22.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.795+0000 7f8dce38e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8dc80b2390 0x7f8dc8068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:22.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.795+0000 7f8dce38e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc81574b0 con 0x7f8dc8068bd0 2023-10-10T19:28:22.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.795+0000 7f8dce38e700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc8150770 con 0x7f8dc80b1fc0 2023-10-10T19:28:22.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.795+0000 7f8dce38e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc80afad0 con 0x7f8dc80b2390 2023-10-10T19:28:22.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8dc80b1fc0 0x7f8dc8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:22.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc7fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8dc80b1fc0 0x7f8dc8155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53736/0 (socket says 172.21.15.175:53736) 2023-10-10T19:28:22.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc7fff700 1 -- 172.21.15.175:0/3187089570 learned_addr learned my addr 172.21.15.175:0/3187089570 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:22.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc77fe700 1 --2- 172.21.15.175:0/3187089570 >> v2:172.21.15.191:3300/0 conn(0x7f8dc80b2390 0x7f8dc8068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:22.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc7fff700 1 -- 172.21.15.175:0/3187089570 >> v2:172.21.15.191:3300/0 conn(0x7f8dc80b2390 msgr2=0x7f8dc8068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:22.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc7fff700 1 --2- 172.21.15.175:0/3187089570 >> v2:172.21.15.191:3300/0 conn(0x7f8dc80b2390 0x7f8dc8068690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:22.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc7fff700 1 -- 172.21.15.175:0/3187089570 >> v1:172.21.15.191:6789/0 conn(0x7f8dc8068bd0 legacy=0x7f8dc80aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:22.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc7fff700 1 -- 172.21.15.175:0/3187089570 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80af4c0 con 0x7f8dc80b1fc0 2023-10-10T19:28:22.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc77fe700 1 --2- 172.21.15.175:0/3187089570 >> v2:172.21.15.191:3300/0 conn(0x7f8dc80b2390 0x7f8dc8068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:28:22.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc7fff700 1 --2- 172.21.15.175:0/3187089570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b1fc0 0x7f8dc8155fb0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80c0290 tx=0x7f8db8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f7a1ff457d0a31 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:22.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc6ffd700 1 -- 172.21.15.175:0/3187089570 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db800b8b0 con 0x7f8dc80b1fc0 2023-10-10T19:28:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc6ffd700 1 -- 172.21.15.175:0/3187089570 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db800ba10 con 0x7f8dc80b1fc0 2023-10-10T19:28:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.796+0000 7f8dc6ffd700 1 -- 172.21.15.175:0/3187089570 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db800bdb0 con 0x7f8dc80b1fc0 2023-10-10T19:28:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.797+0000 7f8dce38e700 1 -- 172.21.15.175:0/3187089570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b1fc0 msgr2=0x7f8dc8155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:22.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.797+0000 7f8dce38e700 1 --2- 172.21.15.175:0/3187089570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b1fc0 0x7f8dc8155fb0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80c0290 tx=0x7f8db8009250 comp rx=0 tx=0).stop 2023-10-10T19:28:22.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.797+0000 7f8dce38e700 1 -- 172.21.15.175:0/3187089570 shutdown_connections 2023-10-10T19:28:22.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.797+0000 7f8dce38e700 1 --2- 172.21.15.175:0/3187089570 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b1fc0 0x7f8dc8155fb0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:22.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.797+0000 7f8dce38e700 1 --2- 172.21.15.175:0/3187089570 >> v2:172.21.15.191:3300/0 conn(0x7f8dc80b2390 0x7f8dc8068690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:22.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.797+0000 7f8dce38e700 1 -- 172.21.15.175:0/3187089570 >> 172.21.15.175:0/3187089570 conn(0x7f8dc8063230 msgr2=0x7f8dc8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:22.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.797+0000 7f8dce38e700 1 -- 172.21.15.175:0/3187089570 shutdown_connections 2023-10-10T19:28:22.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.797+0000 7f8dce38e700 1 -- 172.21.15.175:0/3187089570 wait complete. 2023-10-10T19:28:22.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.798+0000 7f8dce38e700 1 Processor -- start 2023-10-10T19:28:22.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.798+0000 7f8dce38e700 1 -- start start 2023-10-10T19:28:22.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dce38e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dc8068bd0 0x7f8dc8154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:22.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dce38e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dc80b1fc0 0x7f8dc8154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:22.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dce38e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b2390 0x7f8dc8169400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:22.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dce38e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80cd290 con 0x7f8dc80b2390 2023-10-10T19:28:22.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dce38e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc81540f0 con 0x7f8dc80b1fc0 2023-10-10T19:28:22.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dce38e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8152b10 con 0x7f8dc8068bd0 2023-10-10T19:28:22.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b2390 0x7f8dc8169400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:22.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b2390 0x7f8dc8169400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:53752/0 (socket says 172.21.15.175:53752) 2023-10-10T19:28:22.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 -- 172.21.15.175:0/2346386518 learned_addr learned my addr 172.21.15.175:0/2346386518 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:22.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dc77fe700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dc80b1fc0 0x7f8dc8154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:22.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dc7fff700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dc8068bd0 0x7f8dc8154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:22.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 -- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dc8068bd0 msgr2=0x7f8dc8154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:22.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dc8068bd0 0x7f8dc8154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:22.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 -- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dc80b1fc0 msgr2=0x7f8dc8154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:22.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dc80b1fc0 0x7f8dc8154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:22.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 -- 172.21.15.175:0/2346386518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db80094f0 con 0x7f8dc80b2390 2023-10-10T19:28:22.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.799+0000 7f8dcc92b700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b2390 0x7f8dc8169400 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000bbb0 tx=0x7f8dc0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:22.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.800+0000 7f8daffff700 1 -- 172.21.15.175:0/2346386518 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc00092e0 con 0x7f8dc80b2390 2023-10-10T19:28:22.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.800+0000 7f8daffff700 1 -- 172.21.15.175:0/2346386518 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc000c6f0 con 0x7f8dc80b2390 2023-10-10T19:28:22.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.800+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc8153e20 con 0x7f8dc80b2390 2023-10-10T19:28:22.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.800+0000 7f8daffff700 1 -- 172.21.15.175:0/2346386518 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc000ca90 con 0x7f8dc80b2390 2023-10-10T19:28:22.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.800+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc8156a30 con 0x7f8dc80b2390 2023-10-10T19:28:22.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.801+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc8067570 con 0x7f8dc80b2390 2023-10-10T19:28:22.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.801+0000 7f8daffff700 1 -- 172.21.15.175:0/2346386518 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8dc0012070 con 0x7f8dc80b2390 2023-10-10T19:28:22.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.801+0000 7f8daffff700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8db4066f30 0x7f8db40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:22.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.802+0000 7f8daffff700 1 -- 172.21.15.175:0/2346386518 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f8dc00848a0 con 0x7f8dc80b2390 2023-10-10T19:28:22.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.802+0000 7f8dc7fff700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8db4066f30 0x7f8db40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:22.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.802+0000 7f8dc7fff700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8db4066f30 0x7f8db40693e0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80c0290 tx=0x7f8db8016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:22.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:22.806+0000 7f8daffff700 1 -- 172.21.15.175:0/2346386518 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8dc0051240 con 0x7f8dc80b2390 2023-10-10T19:28:23.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.032+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dc8154230 con 0x7f8dc80b2390 2023-10-10T19:28:23.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.033+0000 7f8daffff700 1 -- 172.21.15.175:0/2346386518 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+6193 (secure 0 0 0) 0x7f8dc0054860 con 0x7f8dc80b2390 2023-10-10T19:28:23.033 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:23.034 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":311}],"filesystems":[{"mdsmap":{"epoch":313,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:19.941989+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8db4066f30 msgr2=0x7f8db40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8db4066f30 0x7f8db40693e0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80c0290 tx=0x7f8db8016000 comp rx=0 tx=0).stop 2023-10-10T19:28:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b2390 msgr2=0x7f8dc8169400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:23.036 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b2390 0x7f8dc8169400 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000bbb0 tx=0x7f8dc0009840 comp rx=0 tx=0).stop 2023-10-10T19:28:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 shutdown_connections 2023-10-10T19:28:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f8db4066f30 0x7f8db40693e0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8dc80b2390 0x7f8dc8169400 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dc8068bd0 0x7f8dc8154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:23.037 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 --2- 172.21.15.175:0/2346386518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dc80b1fc0 0x7f8dc8154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:23.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 >> 172.21.15.175:0/2346386518 conn(0x7f8dc8063230 msgr2=0x7f8dc814fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:23.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 shutdown_connections 2023-10-10T19:28:23.038 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:23.035+0000 7f8dce38e700 1 -- 172.21.15.175:0/2346386518 wait complete. 2023-10-10T19:28:23.039 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 313 2023-10-10T19:28:23.047 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:28:25.048 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:25.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.163+0000 7f0b1176f700 1 Processor -- start 2023-10-10T19:28:25.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.163+0000 7f0b1176f700 1 -- start start 2023-10-10T19:28:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b1176f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b1fc0 0x7f0b0c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b1176f700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0b0c0aff60 0x7f0b0c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b1176f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c1574b0 con 0x7f0b0c0b2390 2023-10-10T19:28:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b1176f700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c150770 con 0x7f0b0c0aff60 2023-10-10T19:28:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b1176f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0afad0 con 0x7f0b0c0b1fc0 2023-10-10T19:28:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0b0c0aff60 0x7f0b0c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0b0c0aff60 0x7f0b0c0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49378/0 (socket says 172.21.15.175:49378) 2023-10-10T19:28:25.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0b7fe700 1 -- 172.21.15.175:0/442926488 learned_addr learned my addr 172.21.15.175:0/442926488 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:25.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0affd700 1 --2- 172.21.15.175:0/442926488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b1fc0 0x7f0b0c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:25.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0b7fe700 1 -- 172.21.15.175:0/442926488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b1fc0 msgr2=0x7f0b0c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:25.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0b7fe700 1 --2- 172.21.15.175:0/442926488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b1fc0 0x7f0b0c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0b7fe700 1 -- 172.21.15.175:0/442926488 >> v1:172.21.15.191:6789/0 conn(0x7f0b0c0b2390 legacy=0x7f0b0c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:25.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0b7fe700 1 -- 172.21.15.175:0/442926488 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0af4c0 con 0x7f0b0c0aff60 2023-10-10T19:28:25.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b09ffb700 1 -- 172.21.15.175:0/442926488 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222851219 0 0) 0x7f0b0c1574b0 con 0x7f0b0c0b2390 2023-10-10T19:28:25.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.164+0000 7f0b0b7fe700 1 --2- 172.21.15.175:0/442926488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c0b03e0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009f50 tx=0x7f0b04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07cc3865c222132 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:25.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b09ffb700 1 -- 172.21.15.175:0/442926488 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0400e070 con 0x7f0b0c0aff60 2023-10-10T19:28:25.169 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b09ffb700 1 -- 172.21.15.175:0/442926488 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0400ba50 con 0x7f0b0c0aff60 2023-10-10T19:28:25.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b09ffb700 1 -- 172.21.15.175:0/442926488 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b040092e0 con 0x7f0b0c0aff60 2023-10-10T19:28:25.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b1176f700 1 -- 172.21.15.175:0/442926488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 msgr2=0x7f0b0c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:25.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b1176f700 1 --2- 172.21.15.175:0/442926488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c0b03e0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009f50 tx=0x7f0b04009840 comp rx=0 tx=0).stop 2023-10-10T19:28:25.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b1176f700 1 -- 172.21.15.175:0/442926488 shutdown_connections 2023-10-10T19:28:25.170 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b1176f700 1 --2- 172.21.15.175:0/442926488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c0b03e0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b1176f700 1 --2- 172.21.15.175:0/442926488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b1fc0 0x7f0b0c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b1176f700 1 -- 172.21.15.175:0/442926488 >> 172.21.15.175:0/442926488 conn(0x7f0b0c063230 msgr2=0x7f0b0c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:25.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b1176f700 1 -- 172.21.15.175:0/442926488 shutdown_connections 2023-10-10T19:28:25.171 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.165+0000 7f0b1176f700 1 -- 172.21.15.175:0/442926488 wait complete. 2023-10-10T19:28:25.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.166+0000 7f0b1176f700 1 Processor -- start 2023-10-10T19:28:25.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.166+0000 7f0b1176f700 1 -- start start 2023-10-10T19:28:25.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b1176f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b1176f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b0c0b1fc0 0x7f0b0c1529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b1176f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b2390 0x7f0b0c0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b1176f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c166cf0 con 0x7f0b0c0aff60 2023-10-10T19:28:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b1176f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c158730 con 0x7f0b0c0b1fc0 2023-10-10T19:28:25.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b1176f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0c4070 con 0x7f0b0c0b2390 2023-10-10T19:28:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b0affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b0affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49394/0 (socket says 172.21.15.175:49394) 2023-10-10T19:28:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b0affd700 1 -- 172.21.15.175:0/2448824107 learned_addr learned my addr 172.21.15.175:0/2448824107 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b0a7fc700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b0c0b1fc0 0x7f0b0c1529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:25.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.167+0000 7f0b0b7fe700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b2390 0x7f0b0c0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:25.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b0affd700 1 -- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b2390 msgr2=0x7f0b0c0cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:25.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b0affd700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b2390 0x7f0b0c0cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b0affd700 1 -- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b0c0b1fc0 msgr2=0x7f0b0c1529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:25.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b0affd700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b0c0b1fc0 0x7f0b0c1529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b0affd700 1 -- 172.21.15.175:0/2448824107 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04009ae0 con 0x7f0b0c0aff60 2023-10-10T19:28:25.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b0a7fc700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b0c0b1fc0 0x7f0b0c1529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:28:25.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b0affd700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c158bc0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00ec80 tx=0x7f0afc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:25.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0af2ffd700 1 -- 172.21.15.175:0/2448824107 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc00c850 con 0x7f0b0c0aff60 2023-10-10T19:28:25.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0af2ffd700 1 -- 172.21.15.175:0/2448824107 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc00c9b0 con 0x7f0b0c0aff60 2023-10-10T19:28:25.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c158400 con 0x7f0b0c0aff60 2023-10-10T19:28:25.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0af2ffd700 1 -- 172.21.15.175:0/2448824107 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc00cd50 con 0x7f0b0c0aff60 2023-10-10T19:28:25.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.168+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c159a60 con 0x7f0b0c0aff60 2023-10-10T19:28:25.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.169+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c159e00 con 0x7f0b0c0aff60 2023-10-10T19:28:25.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.169+0000 7f0af2ffd700 1 -- 172.21.15.175:0/2448824107 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0afc013680 con 0x7f0b0c0aff60 2023-10-10T19:28:25.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.170+0000 7f0af2ffd700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0af8066f30 0x7f0af80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:25.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.170+0000 7f0af2ffd700 1 -- 172.21.15.175:0/2448824107 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f0afc084f90 con 0x7f0b0c0aff60 2023-10-10T19:28:25.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.170+0000 7f0b0a7fc700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0af8066f30 0x7f0af80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:25.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.170+0000 7f0b0a7fc700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0af8066f30 0x7f0af80693e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c063720 tx=0x7f0b00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:25.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.174+0000 7f0af2ffd700 1 -- 172.21.15.175:0/2448824107 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0afc0519d0 con 0x7f0b0c0aff60 2023-10-10T19:28:25.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.403+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b0c067570 con 0x7f0b0c0aff60 2023-10-10T19:28:25.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.404+0000 7f0af2ffd700 1 -- 172.21.15.175:0/2448824107 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+6193 (secure 0 0 0) 0x7f0afc054ff0 con 0x7f0b0c0aff60 2023-10-10T19:28:25.405 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:25.405 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":311}],"filesystems":[{"mdsmap":{"epoch":313,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:19.941989+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:25.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0af8066f30 msgr2=0x7f0af80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:25.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0af8066f30 0x7f0af80693e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c063720 tx=0x7f0b00009040 comp rx=0 tx=0).stop 2023-10-10T19:28:25.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 msgr2=0x7f0b0c158bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:25.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c158bc0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00ec80 tx=0x7f0afc0091a0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 shutdown_connections 2023-10-10T19:28:25.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0af8066f30 0x7f0af80693e0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0b0c0b2390 0x7f0b0c0cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b0c0aff60 0x7f0b0c158bc0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 --2- 172.21.15.175:0/2448824107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0b0c0b1fc0 0x7f0b0c1529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:25.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 >> 172.21.15.175:0/2448824107 conn(0x7f0b0c063230 msgr2=0x7f0b0c14fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:25.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 shutdown_connections 2023-10-10T19:28:25.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:25.406+0000 7f0b1176f700 1 -- 172.21.15.175:0/2448824107 wait complete. 2023-10-10T19:28:25.411 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 313 2023-10-10T19:28:25.418 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:28:27.389 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:28:27.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:28:27.419 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:27.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.555+0000 7fb81da5f700 1 Processor -- start 2023-10-10T19:28:27.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.555+0000 7fb81da5f700 1 -- start start 2023-10-10T19:28:27.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb81da5f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b1fc0 0x7fb818155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:27.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb81da5f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb8180b2390 0x7fb818068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:27.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb81da5f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8181574b0 con 0x7fb818068bd0 2023-10-10T19:28:27.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb81da5f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb818150770 con 0x7fb8180b2390 2023-10-10T19:28:27.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb81da5f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8180afad0 con 0x7fb8180b1fc0 2023-10-10T19:28:27.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb816ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b1fc0 0x7fb818155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:27.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb8167fc700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb8180b2390 0x7fb818068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:27.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb816ffd700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b1fc0 0x7fb818155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:48052/0 (socket says 172.21.15.175:48052) 2023-10-10T19:28:27.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.556+0000 7fb816ffd700 1 -- 172.21.15.175:0/4107601872 learned_addr learned my addr 172.21.15.175:0/4107601872 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:27.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb8167fc700 1 -- 172.21.15.175:0/4107601872 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b1fc0 msgr2=0x7fb818155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:27.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb8167fc700 1 --2- 172.21.15.175:0/4107601872 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b1fc0 0x7fb818155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb8167fc700 1 -- 172.21.15.175:0/4107601872 >> v1:172.21.15.191:6789/0 conn(0x7fb818068bd0 legacy=0x7fb8180aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:27.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb8167fc700 1 -- 172.21.15.175:0/4107601872 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8180af4c0 con 0x7fb8180b2390 2023-10-10T19:28:27.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb816ffd700 1 --2- 172.21.15.175:0/4107601872 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b1fc0 0x7fb818155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:28:27.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb8167fc700 1 --2- 172.21.15.175:0/4107601872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b2390 0x7fb818068690 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0098e0 tx=0x7fb80c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d79f7fe3f2901192 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:27.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb815ffb700 1 -- 172.21.15.175:0/4107601872 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00b6d0 con 0x7fb8180b2390 2023-10-10T19:28:27.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb815ffb700 1 -- 172.21.15.175:0/4107601872 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80c00b830 con 0x7fb8180b2390 2023-10-10T19:28:27.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.557+0000 7fb815ffb700 1 -- 172.21.15.175:0/4107601872 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00bbd0 con 0x7fb8180b2390 2023-10-10T19:28:27.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.558+0000 7fb81da5f700 1 -- 172.21.15.175:0/4107601872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b2390 msgr2=0x7fb818068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:27.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.558+0000 7fb81da5f700 1 --2- 172.21.15.175:0/4107601872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b2390 0x7fb818068690 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0098e0 tx=0x7fb80c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.558+0000 7fb81da5f700 1 -- 172.21.15.175:0/4107601872 shutdown_connections 2023-10-10T19:28:27.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.558+0000 7fb81da5f700 1 --2- 172.21.15.175:0/4107601872 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b1fc0 0x7fb818155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.558+0000 7fb81da5f700 1 --2- 172.21.15.175:0/4107601872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b2390 0x7fb818068690 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.558+0000 7fb81da5f700 1 -- 172.21.15.175:0/4107601872 >> 172.21.15.175:0/4107601872 conn(0x7fb818063230 msgr2=0x7fb818063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:27.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.558+0000 7fb81da5f700 1 -- 172.21.15.175:0/4107601872 shutdown_connections 2023-10-10T19:28:27.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.558+0000 7fb81da5f700 1 -- 172.21.15.175:0/4107601872 wait complete. 2023-10-10T19:28:27.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.559+0000 7fb81da5f700 1 Processor -- start 2023-10-10T19:28:27.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.559+0000 7fb81da5f700 1 -- start start 2023-10-10T19:28:27.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb81da5f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb818068bd0 0x7fb8181662f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:27.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb81da5f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b1fc0 0x7fb81815b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:27.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb81da5f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b2390 0x7fb818168fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:27.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb81da5f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb818165e60 con 0x7fb818068bd0 2023-10-10T19:28:27.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb81da5f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb818153dc0 con 0x7fb8180b1fc0 2023-10-10T19:28:27.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb81da5f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb818154060 con 0x7fb8180b2390 2023-10-10T19:28:27.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb818068bd0 0x7fb8181662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:27.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb818068bd0 0x7fb8181662f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49398/0 (socket says 172.21.15.175:49398) 2023-10-10T19:28:27.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 -- 172.21.15.175:0/3976702658 learned_addr learned my addr 172.21.15.175:0/3976702658 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:27.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb8167fc700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b1fc0 0x7fb81815b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:27.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb8177fe700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b2390 0x7fb818168fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:27.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 -- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b2390 msgr2=0x7fb818168fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:27.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b2390 0x7fb818168fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 -- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b1fc0 msgr2=0x7fb81815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:27.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b1fc0 0x7fb81815b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 -- 172.21.15.175:0/3976702658 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080094f0 con 0x7fb818068bd0 2023-10-10T19:28:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb8177fe700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b2390 0x7fb818168fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:28:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb816ffd700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb818068bd0 0x7fb8181662f0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fb80800ec50 tx=0x7fb808009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb7feffd700 1 -- 172.21.15.175:0/3976702658 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8080097d0 con 0x7fb818068bd0 2023-10-10T19:28:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c009420 con 0x7fb818068bd0 2023-10-10T19:28:27.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.560+0000 7fb7feffd700 1 -- 172.21.15.175:0/3976702658 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80800c920 con 0x7fb818068bd0 2023-10-10T19:28:27.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.561+0000 7fb7feffd700 1 -- 172.21.15.175:0/3976702658 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80800cca0 con 0x7fb818068bd0 2023-10-10T19:28:27.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.561+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb818154300 con 0x7fb818068bd0 2023-10-10T19:28:27.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.561+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb818166140 con 0x7fb818068bd0 2023-10-10T19:28:27.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.562+0000 7fb7feffd700 1 -- 172.21.15.175:0/3976702658 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb808013680 con 0x7fb818068bd0 2023-10-10T19:28:27.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.562+0000 7fb7feffd700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb804066f30 0x7fb8040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:27.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.562+0000 7fb7feffd700 1 -- 172.21.15.175:0/3976702658 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7fb808085490 con 0x7fb818068bd0 2023-10-10T19:28:27.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.562+0000 7fb8167fc700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb804066f30 0x7fb8040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:27.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.563+0000 7fb8167fc700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb804066f30 0x7fb8040693e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0091e0 tx=0x7fb80c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:27.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.566+0000 7fb7feffd700 1 -- 172.21.15.175:0/3976702658 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb808051ed0 con 0x7fb818068bd0 2023-10-10T19:28:27.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.796+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb818067570 con 0x7fb818068bd0 2023-10-10T19:28:27.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.796+0000 7fb7feffd700 1 -- 172.21.15.175:0/3976702658 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+6193 (secure 0 0 0) 0x7fb8080554f0 con 0x7fb818068bd0 2023-10-10T19:28:27.797 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:27.797 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":311}],"filesystems":[{"mdsmap":{"epoch":313,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:19.941989+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:27.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb804066f30 msgr2=0x7fb8040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:27.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb804066f30 0x7fb8040693e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0091e0 tx=0x7fb80c017000 comp rx=0 tx=0).stop 2023-10-10T19:28:27.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb818068bd0 msgr2=0x7fb8181662f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:27.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb818068bd0 0x7fb8181662f0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fb80800ec50 tx=0x7fb808009250 comp rx=0 tx=0).stop 2023-10-10T19:28:27.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 shutdown_connections 2023-10-10T19:28:27.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb804066f30 0x7fb8040693e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb8180b2390 0x7fb818168fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb818068bd0 0x7fb8181662f0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 --2- 172.21.15.175:0/3976702658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8180b1fc0 0x7fb81815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:27.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 >> 172.21.15.175:0/3976702658 conn(0x7fb818063230 msgr2=0x7fb818150ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:27.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 shutdown_connections 2023-10-10T19:28:27.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:27.798+0000 7fb81da5f700 1 -- 172.21.15.175:0/3976702658 wait complete. 2023-10-10T19:28:27.801 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 313 2023-10-10T19:28:27.811 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:28:29.812 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.941+0000 7ff94292c700 1 Processor -- start 2023-10-10T19:28:29.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.941+0000 7ff94292c700 1 -- start start 2023-10-10T19:28:29.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff94292c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff93c14fbf0 0x7ff93c159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:29.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff94292c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 0x7ff93c063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:29.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff94292c700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93c156800 con 0x7ff93c0681e0 2023-10-10T19:28:29.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff94292c700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff93c0c59e0 con 0x7ff93c14fbf0 2023-10-10T19:28:29.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff94292c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c15a2b0 con 0x7ff93c14ffc0 2023-10-10T19:28:29.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff93c14fbf0 0x7ff93c159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:29.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff940ec9700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff93c0681e0 0x7ff93c0c5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60546/0 (socket says 172.21.15.175:60546) 2023-10-10T19:28:29.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff940ec9700 1 -- 172.21.15.175:0/3977020543 learned_addr learned my addr 172.21.15.175:0/3977020543 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:29.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93b7fe700 1 --2- 172.21.15.175:0/3977020543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 0x7ff93c063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:29.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93bfff700 1 -- 172.21.15.175:0/3977020543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 msgr2=0x7ff93c063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:29.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93bfff700 1 --2- 172.21.15.175:0/3977020543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 0x7ff93c063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:29.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93bfff700 1 -- 172.21.15.175:0/3977020543 >> v1:172.21.15.175:6789/0 conn(0x7ff93c0681e0 legacy=0x7ff93c0c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93bfff700 1 -- 172.21.15.175:0/3977020543 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93c063e30 con 0x7ff93c14fbf0 2023-10-10T19:28:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93affd700 1 -- 172.21.15.175:0/3977020543 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115979310 0 0) 0x7ff93c156800 con 0x7ff93c0681e0 2023-10-10T19:28:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93bfff700 1 --2- 172.21.15.175:0/3977020543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c14fbf0 0x7ff93c159d70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff930009900 tx=0x7ff930009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76768f5ee9d026d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93affd700 1 -- 172.21.15.175:0/3977020543 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930009ea0 con 0x7ff93c14fbf0 2023-10-10T19:28:29.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93affd700 1 -- 172.21.15.175:0/3977020543 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff930005c10 con 0x7ff93c14fbf0 2023-10-10T19:28:29.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.942+0000 7ff93affd700 1 -- 172.21.15.175:0/3977020543 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93000b610 con 0x7ff93c14fbf0 2023-10-10T19:28:29.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.943+0000 7ff94292c700 1 -- 172.21.15.175:0/3977020543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c14fbf0 msgr2=0x7ff93c159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:29.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.943+0000 7ff94292c700 1 --2- 172.21.15.175:0/3977020543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c14fbf0 0x7ff93c159d70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff930009900 tx=0x7ff930009120 comp rx=0 tx=0).stop 2023-10-10T19:28:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.943+0000 7ff94292c700 1 -- 172.21.15.175:0/3977020543 shutdown_connections 2023-10-10T19:28:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.943+0000 7ff94292c700 1 --2- 172.21.15.175:0/3977020543 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c14fbf0 0x7ff93c159d70 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.943+0000 7ff94292c700 1 --2- 172.21.15.175:0/3977020543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 0x7ff93c063230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.943+0000 7ff94292c700 1 -- 172.21.15.175:0/3977020543 >> 172.21.15.175:0/3977020543 conn(0x7ff93c0cc760 msgr2=0x7ff93c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:29.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.943+0000 7ff94292c700 1 -- 172.21.15.175:0/3977020543 shutdown_connections 2023-10-10T19:28:29.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.943+0000 7ff94292c700 1 -- 172.21.15.175:0/3977020543 wait complete. 2023-10-10T19:28:29.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.944+0000 7ff94292c700 1 Processor -- start 2023-10-10T19:28:29.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.944+0000 7ff94292c700 1 -- start start 2023-10-10T19:28:29.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff94292c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c0681e0 0x7ff93c0c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:29.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff94292c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff93c14fbf0 0x7ff93c1594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:29.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff94292c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 0x7ff93c152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:29.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff94292c700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0c9f30 con 0x7ff93c0681e0 2023-10-10T19:28:29.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff94292c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c169d90 con 0x7ff93c14fbf0 2023-10-10T19:28:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff94292c700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0c93b0 con 0x7ff93c14ffc0 2023-10-10T19:28:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c0681e0 0x7ff93c0c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c0681e0 0x7ff93c0c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49416/0 (socket says 172.21.15.175:49416) 2023-10-10T19:28:29.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 -- 172.21.15.175:0/1031959832 learned_addr learned my addr 172.21.15.175:0/1031959832 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff940ec9700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 0x7ff93c152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93b7fe700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff93c14fbf0 0x7ff93c1594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 -- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 msgr2=0x7ff93c152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 0x7ff93c152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:29.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 -- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff93c14fbf0 msgr2=0x7ff93c1594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff93c14fbf0 0x7ff93c1594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 -- 172.21.15.175:0/1031959832 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9300093c0 con 0x7ff93c0681e0 2023-10-10T19:28:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.945+0000 7ff93bfff700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c0681e0 0x7ff93c0c8850 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7ff930009160 tx=0x7ff930016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.946+0000 7ff923fff700 1 -- 172.21.15.175:0/1031959832 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930006440 con 0x7ff93c0681e0 2023-10-10T19:28:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.946+0000 7ff923fff700 1 -- 172.21.15.175:0/1031959832 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9300065a0 con 0x7ff93c0681e0 2023-10-10T19:28:29.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.946+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93c0c9690 con 0x7ff93c0681e0 2023-10-10T19:28:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.946+0000 7ff923fff700 1 -- 172.21.15.175:0/1031959832 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930006960 con 0x7ff93c0681e0 2023-10-10T19:28:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.946+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93c0c9c50 con 0x7ff93c0681e0 2023-10-10T19:28:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.947+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff93c1528b0 con 0x7ff93c0681e0 2023-10-10T19:28:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.947+0000 7ff923fff700 1 -- 172.21.15.175:0/1031959832 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff93001b690 con 0x7ff93c0681e0 2023-10-10T19:28:29.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.947+0000 7ff923fff700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff928066f30 0x7ff9280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:29.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.947+0000 7ff923fff700 1 -- 172.21.15.175:0/1031959832 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7ff930086fd0 con 0x7ff93c0681e0 2023-10-10T19:28:29.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.948+0000 7ff93b7fe700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff928066f30 0x7ff9280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:29.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.948+0000 7ff93b7fe700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff928066f30 0x7ff9280693e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff92c006fd0 tx=0x7ff92c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:29.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:29.952+0000 7ff923fff700 1 -- 172.21.15.175:0/1031959832 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff930053a10 con 0x7ff93c0681e0 2023-10-10T19:28:30.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.178+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff93c067570 con 0x7ff93c0681e0 2023-10-10T19:28:30.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.178+0000 7ff923fff700 1 -- 172.21.15.175:0/1031959832 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+6193 (secure 0 0 0) 0x7ff930057030 con 0x7ff93c0681e0 2023-10-10T19:28:30.179 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:30.179 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":311}],"filesystems":[{"mdsmap":{"epoch":313,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:19.941989+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff928066f30 msgr2=0x7ff9280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff928066f30 0x7ff9280693e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff92c006fd0 tx=0x7ff92c008040 comp rx=0 tx=0).stop 2023-10-10T19:28:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c0681e0 msgr2=0x7ff93c0c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c0681e0 0x7ff93c0c8850 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7ff930009160 tx=0x7ff930016000 comp rx=0 tx=0).stop 2023-10-10T19:28:30.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 shutdown_connections 2023-10-10T19:28:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff928066f30 0x7ff9280693e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff93c14ffc0 0x7ff93c152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff93c0681e0 0x7ff93c0c8850 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 --2- 172.21.15.175:0/1031959832 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff93c14fbf0 0x7ff93c1594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:30.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 >> 172.21.15.175:0/1031959832 conn(0x7ff93c0cc760 msgr2=0x7ff93c0ca6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:30.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 shutdown_connections 2023-10-10T19:28:30.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:30.180+0000 7ff94292c700 1 -- 172.21.15.175:0/1031959832 wait complete. 2023-10-10T19:28:30.183 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 313 2023-10-10T19:28:30.192 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:28:32.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:32.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.321+0000 7f3fc8fd8700 1 Processor -- start 2023-10-10T19:28:32.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.321+0000 7f3fc8fd8700 1 -- start start 2023-10-10T19:28:32.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc8fd8700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3fc40b1fc0 0x7f3fc4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:32.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc8fd8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3fc40aff60 0x7f3fc40b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc8fd8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc41574b0 con 0x7f3fc40b2390 2023-10-10T19:28:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc8fd8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc4150770 con 0x7f3fc40aff60 2023-10-10T19:28:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc8fd8700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc40afad0 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc2d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3fc40aff60 0x7f3fc40b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc2d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3fc40aff60 0x7f3fc40b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49422/0 (socket says 172.21.15.175:49422) 2023-10-10T19:28:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc2d9d700 1 -- 172.21.15.175:0/1474380399 learned_addr learned my addr 172.21.15.175:0/1474380399 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc259c700 1 --2- 172.21.15.175:0/1474380399 >> v2:172.21.15.191:3300/0 conn(0x7f3fc40b1fc0 0x7f3fc4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc2d9d700 1 -- 172.21.15.175:0/1474380399 >> v2:172.21.15.191:3300/0 conn(0x7f3fc40b1fc0 msgr2=0x7f3fc4155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:32.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc2d9d700 1 --2- 172.21.15.175:0/1474380399 >> v2:172.21.15.191:3300/0 conn(0x7f3fc40b1fc0 0x7f3fc4155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc2d9d700 1 -- 172.21.15.175:0/1474380399 >> v1:172.21.15.191:6789/0 conn(0x7f3fc40b2390 legacy=0x7f3fc4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc2d9d700 1 -- 172.21.15.175:0/1474380399 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc40af4c0 con 0x7f3fc40aff60 2023-10-10T19:28:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc2d9d700 1 --2- 172.21.15.175:0/1474380399 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40aff60 0x7f3fc40b03e0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40b28c0 tx=0x7f3fb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d256ac0427172d66 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc159a700 1 -- 172.21.15.175:0/1474380399 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4009e70 con 0x7f3fc40aff60 2023-10-10T19:28:32.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc159a700 1 -- 172.21.15.175:0/1474380399 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb400b8f0 con 0x7f3fc40aff60 2023-10-10T19:28:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.322+0000 7f3fc159a700 1 -- 172.21.15.175:0/1474380399 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb400bc90 con 0x7f3fc40aff60 2023-10-10T19:28:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.323+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/1474380399 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40aff60 msgr2=0x7f3fc40b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.323+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/1474380399 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40aff60 0x7f3fc40b03e0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40b28c0 tx=0x7f3fb4009250 comp rx=0 tx=0).stop 2023-10-10T19:28:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.323+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/1474380399 shutdown_connections 2023-10-10T19:28:32.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.323+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/1474380399 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40aff60 0x7f3fc40b03e0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.323+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/1474380399 >> v2:172.21.15.191:3300/0 conn(0x7f3fc40b1fc0 0x7f3fc4155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.323+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/1474380399 >> 172.21.15.175:0/1474380399 conn(0x7f3fc4063230 msgr2=0x7f3fc4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.323+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/1474380399 shutdown_connections 2023-10-10T19:28:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.323+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/1474380399 wait complete. 2023-10-10T19:28:32.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.324+0000 7f3fc8fd8700 1 Processor -- start 2023-10-10T19:28:32.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.324+0000 7f3fc8fd8700 1 -- start start 2023-10-10T19:28:32.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc8fd8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc40aff60 0x7f3fc4166760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:32.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc8fd8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40b1fc0 0x7f3fc40c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:32.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc8fd8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc40b2390 0x7f3fc41552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:32.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc8fd8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc4166ca0 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc8fd8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc40c6400 con 0x7f3fc40aff60 2023-10-10T19:28:32.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc8fd8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc40c5860 con 0x7f3fc40b2390 2023-10-10T19:28:32.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40b1fc0 0x7f3fc40c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:32.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40b1fc0 0x7f3fc40c3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:49436/0 (socket says 172.21.15.175:49436) 2023-10-10T19:28:32.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 -- 172.21.15.175:0/3964173509 learned_addr learned my addr 172.21.15.175:0/3964173509 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:32.334 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc259c700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc40aff60 0x7f3fc4166760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc2d9d700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc40b2390 0x7f3fc41552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 -- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc40b2390 msgr2=0x7f3fc41552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc40b2390 0x7f3fc41552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 -- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc40aff60 msgr2=0x7f3fc4166760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:32.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc40aff60 0x7f3fc4166760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 -- 172.21.15.175:0/3964173509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb40094f0 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.325+0000 7f3fc1d9b700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40b1fc0 0x7f3fc40c3a30 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c510 tx=0x7f3fb800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:32.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.326+0000 7f3fb27fc700 1 -- 172.21.15.175:0/3964173509 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011590 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.326+0000 7f3fb27fc700 1 -- 172.21.15.175:0/3964173509 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb8011710 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.326+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc40c6890 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.326+0000 7f3fb27fc700 1 -- 172.21.15.175:0/3964173509 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011ad0 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.326+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc40c5ad0 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.326+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc4071340 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.337 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.327+0000 7f3fb27fc700 1 -- 172.21.15.175:0/3964173509 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3fb801b3b0 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.327+0000 7f3fb27fc700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3fa4066f30 0x7f3fa40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.327+0000 7f3fb27fc700 1 -- 172.21.15.175:0/3964173509 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 7799+0+0 (secure 0 0 0) 0x7f3fb8084090 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.327+0000 7f3fc259c700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3fa4066f30 0x7f3fa40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.328+0000 7f3fc259c700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3fa4066f30 0x7f3fa40693e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00f570 tx=0x7f3fbc00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:32.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.332+0000 7f3fb27fc700 1 -- 172.21.15.175:0/3964173509 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3fb8050ad0 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.559+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc40c6540 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.560+0000 7f3fb27fc700 1 -- 172.21.15.175:0/3964173509 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+6193 (secure 0 0 0) 0x7f3fb8017070 con 0x7f3fc40b1fc0 2023-10-10T19:28:32.560 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:32.561 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8733,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":311}],"filesystems":[{"mdsmap":{"epoch":313,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:19.941989+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":6893,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6893":{"gid":6893,"name":"d","rank":1,"incarnation":306,"state":"up:active","state_seq":28,"addr":"172.21.15.191:6837/840318079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":840318079},{"type":"v1","addr":"172.21.15.191:6837","nonce":840318079}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:32.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3fa4066f30 msgr2=0x7f3fa40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:32.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3fa4066f30 0x7f3fa40693e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00f570 tx=0x7f3fbc00b040 comp rx=0 tx=0).stop 2023-10-10T19:28:32.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40b1fc0 msgr2=0x7f3fc40c3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:32.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40b1fc0 0x7f3fc40c3a30 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c510 tx=0x7f3fb800c820 comp rx=0 tx=0).stop 2023-10-10T19:28:32.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 shutdown_connections 2023-10-10T19:28:32.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3fa4066f30 0x7f3fa40693e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc40b2390 0x7f3fc41552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc40aff60 0x7f3fc4166760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 --2- 172.21.15.175:0/3964173509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fc40b1fc0 0x7f3fc40c3a30 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:32.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 >> 172.21.15.175:0/3964173509 conn(0x7f3fc4063230 msgr2=0x7f3fc414fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:32.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 shutdown_connections 2023-10-10T19:28:32.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:32.562+0000 7f3fc8fd8700 1 -- 172.21.15.175:0/3964173509 wait complete. 2023-10-10T19:28:32.565 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 313 2023-10-10T19:28:32.574 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-10T19:28:34.575 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:34.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.700+0000 7f1512baa700 1 Processor -- start 2023-10-10T19:28:34.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.701+0000 7f1512baa700 1 -- start start 2023-10-10T19:28:34.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1512baa700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f150c14fd90 0x7f150c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:34.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1512baa700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f150c08a100 0x7f150c0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:34.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1512baa700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c067fa0 con 0x7f150c0cd040 2023-10-10T19:28:34.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1512baa700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150c0c5600 con 0x7f150c14fd90 2023-10-10T19:28:34.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1512baa700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150c156b70 con 0x7f150c08a100 2023-10-10T19:28:34.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1511ba8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f150c14fd90 0x7f150c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f15123a9700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f150c0cd040 0x7f150c0684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37878/0 (socket says 172.21.15.175:37878) 2023-10-10T19:28:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f15123a9700 1 -- 172.21.15.175:0/2423048397 learned_addr learned my addr 172.21.15.175:0/2423048397 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f15113a7700 1 --2- 172.21.15.175:0/2423048397 >> v2:172.21.15.191:3300/0 conn(0x7f150c08a100 0x7f150c0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:34.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1511ba8700 1 -- 172.21.15.175:0/2423048397 >> v2:172.21.15.191:3300/0 conn(0x7f150c08a100 msgr2=0x7f150c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1511ba8700 1 --2- 172.21.15.175:0/2423048397 >> v2:172.21.15.191:3300/0 conn(0x7f150c08a100 0x7f150c0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1511ba8700 1 -- 172.21.15.175:0/2423048397 >> v1:172.21.15.175:6789/0 conn(0x7f150c0cd040 legacy=0x7f150c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1511ba8700 1 -- 172.21.15.175:0/2423048397 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c150160 con 0x7f150c14fd90 2023-10-10T19:28:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1510ba6700 1 -- 172.21.15.175:0/2423048397 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288850085 0 0) 0x7f150c067fa0 con 0x7f150c0cd040 2023-10-10T19:28:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1511ba8700 1 --2- 172.21.15.175:0/2423048397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c159c30 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f150c069990 tx=0x7f14fc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9071ba29f5cf0a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:34.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1510ba6700 1 -- 172.21.15.175:0/2423048397 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00e070 con 0x7f150c14fd90 2023-10-10T19:28:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1510ba6700 1 -- 172.21.15.175:0/2423048397 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc00bbf0 con 0x7f150c14fd90 2023-10-10T19:28:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.702+0000 7f1510ba6700 1 -- 172.21.15.175:0/2423048397 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc012430 con 0x7f150c14fd90 2023-10-10T19:28:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.703+0000 7f1512baa700 1 -- 172.21.15.175:0/2423048397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 msgr2=0x7f150c159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.703+0000 7f1512baa700 1 --2- 172.21.15.175:0/2423048397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c159c30 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f150c069990 tx=0x7f14fc009780 comp rx=0 tx=0).stop 2023-10-10T19:28:34.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.703+0000 7f1512baa700 1 -- 172.21.15.175:0/2423048397 shutdown_connections 2023-10-10T19:28:34.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.703+0000 7f1512baa700 1 --2- 172.21.15.175:0/2423048397 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c159c30 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.703+0000 7f1512baa700 1 --2- 172.21.15.175:0/2423048397 >> v2:172.21.15.191:3300/0 conn(0x7f150c08a100 0x7f150c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.703+0000 7f1512baa700 1 -- 172.21.15.175:0/2423048397 >> 172.21.15.175:0/2423048397 conn(0x7f150c063d00 msgr2=0x7f150c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:34.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.703+0000 7f1512baa700 1 -- 172.21.15.175:0/2423048397 shutdown_connections 2023-10-10T19:28:34.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.703+0000 7f1512baa700 1 -- 172.21.15.175:0/2423048397 wait complete. 2023-10-10T19:28:34.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.704+0000 7f1512baa700 1 Processor -- start 2023-10-10T19:28:34.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.704+0000 7f1512baa700 1 -- start start 2023-10-10T19:28:34.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f1512baa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150c08a100 0x7f150c1672b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:34.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f1512baa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f150c0cd040 0x7f150c169600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:34.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f1512baa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c169b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:34.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f1512baa700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c0cb9b0 con 0x7f150c14fd90 2023-10-10T19:28:34.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f1512baa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c0cadd0 con 0x7f150c08a100 2023-10-10T19:28:34.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f1512baa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150c0cc9b0 con 0x7f150c0cd040 2023-10-10T19:28:34.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c169b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:34.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c169b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36156/0 (socket says 172.21.15.175:36156) 2023-10-10T19:28:34.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 -- 172.21.15.175:0/854022560 learned_addr learned my addr 172.21.15.175:0/854022560 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:34.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f1511ba8700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150c08a100 0x7f150c1672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:34.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15113a7700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f150c0cd040 0x7f150c169600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:34.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 -- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f150c0cd040 msgr2=0x7f150c169600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:34.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f150c0cd040 0x7f150c169600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 -- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150c08a100 msgr2=0x7f150c1672b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:34.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150c08a100 0x7f150c1672b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 -- 172.21.15.175:0/854022560 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc009a20 con 0x7f150c14fd90 2023-10-10T19:28:34.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.705+0000 7f15123a9700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c169b40 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f150400c7e0 tx=0x7f1504007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:34.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.706+0000 7f1509ffb700 1 -- 172.21.15.175:0/854022560 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504005c00 con 0x7f150c14fd90 2023-10-10T19:28:34.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.706+0000 7f1509ffb700 1 -- 172.21.15.175:0/854022560 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1504005d80 con 0x7f150c14fd90 2023-10-10T19:28:34.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.706+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c0cb080 con 0x7f150c14fd90 2023-10-10T19:28:34.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.706+0000 7f1509ffb700 1 -- 172.21.15.175:0/854022560 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504007320 con 0x7f150c14fd90 2023-10-10T19:28:34.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.706+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150c0cb640 con 0x7f150c14fd90 2023-10-10T19:28:34.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.707+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c0cbb60 con 0x7f150c14fd90 2023-10-10T19:28:34.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.707+0000 7f1509ffb700 1 -- 172.21.15.175:0/854022560 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f150401b3b0 con 0x7f150c14fd90 2023-10-10T19:28:34.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.707+0000 7f1509ffb700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f14f8066f30 0x7f14f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:34.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.708+0000 7f1509ffb700 1 -- 172.21.15.175:0/854022560 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f15040841f0 con 0x7f150c14fd90 2023-10-10T19:28:34.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.708+0000 7f1511ba8700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f14f8066f30 0x7f14f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:34.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.708+0000 7f1511ba8700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f14f8066f30 0x7f14f80693e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0099f0 tx=0x7f14fc019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:34.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.712+0000 7f1509ffb700 1 -- 172.21.15.175:0/854022560 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f150c0cbb60 con 0x7f150c14fd90 2023-10-10T19:28:34.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.939+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f150c0cbb60 con 0x7f150c14fd90 2023-10-10T19:28:34.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.940+0000 7f1509ffb700 1 -- 172.21.15.175:0/854022560 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) v1 ==== 95+0+5432 (secure 0 0 0) 0x7f150c0cbb60 con 0x7f150c14fd90 2023-10-10T19:28:34.941 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:34.941 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":314,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":314,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:33.906493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:34.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f14f8066f30 msgr2=0x7f14f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:34.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f14f8066f30 0x7f14f80693e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0099f0 tx=0x7f14fc019000 comp rx=0 tx=0).stop 2023-10-10T19:28:34.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 msgr2=0x7f150c169b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:34.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c169b40 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f150400c7e0 tx=0x7f1504007d90 comp rx=0 tx=0).stop 2023-10-10T19:28:34.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 shutdown_connections 2023-10-10T19:28:34.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f14f8066f30 0x7f14f80693e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150c14fd90 0x7f150c169b40 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150c08a100 0x7f150c1672b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 --2- 172.21.15.175:0/854022560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f150c0cd040 0x7f150c169600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:34.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 >> 172.21.15.175:0/854022560 conn(0x7f150c063d00 msgr2=0x7f150c14ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:34.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 shutdown_connections 2023-10-10T19:28:34.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:34.942+0000 7f1512baa700 1 -- 172.21.15.175:0/854022560 wait complete. 2023-10-10T19:28:34.945 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 314 2023-10-10T19:28:34.955 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-10T19:28:34.955 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-10T19:28:34.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:35.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.070+0000 7f1e7ee36700 1 Processor -- start 2023-10-10T19:28:35.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.070+0000 7f1e7ee36700 1 -- start start 2023-10-10T19:28:35.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7ee36700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1e7814fd90 0x7f1e78159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:35.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7ee36700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e7808a100 0x7f1e780c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:35.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7ee36700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e78067fa0 con 0x7f1e780cd040 2023-10-10T19:28:35.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7ee36700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e780c5600 con 0x7f1e7808a100 2023-10-10T19:28:35.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7ee36700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e78156b70 con 0x7f1e7814fd90 2023-10-10T19:28:35.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7d633700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e7808a100 0x7f1e780c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:35.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7e635700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1e780cd040 0x7f1e780684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37894/0 (socket says 172.21.15.175:37894) 2023-10-10T19:28:35.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7e635700 1 -- 172.21.15.175:0/3658561334 learned_addr learned my addr 172.21.15.175:0/3658561334 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:35.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7d633700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e7808a100 0x7f1e780c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36164/0 (socket says 172.21.15.175:36164) 2023-10-10T19:28:35.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7de34700 1 --2- 172.21.15.175:0/3658561334 >> v2:172.21.15.191:3300/0 conn(0x7f1e7814fd90 0x7f1e78159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:35.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7d633700 1 -- 172.21.15.175:0/3658561334 >> v2:172.21.15.191:3300/0 conn(0x7f1e7814fd90 msgr2=0x7f1e78159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:35.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7d633700 1 --2- 172.21.15.175:0/3658561334 >> v2:172.21.15.191:3300/0 conn(0x7f1e7814fd90 0x7f1e78159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7d633700 1 -- 172.21.15.175:0/3658561334 >> v1:172.21.15.175:6789/0 conn(0x7f1e780cd040 legacy=0x7f1e780684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:35.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7ce32700 1 -- 172.21.15.175:0/3658561334 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260186440 0 0) 0x7f1e78067fa0 con 0x7f1e780cd040 2023-10-10T19:28:35.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7d633700 1 -- 172.21.15.175:0/3658561334 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e78150160 con 0x7f1e7808a100 2023-10-10T19:28:35.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7d633700 1 --2- 172.21.15.175:0/3658561334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e7808a100 0x7f1e780c5eb0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1e78065080 tx=0x7f1e680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db544e498a71af36 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:35.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7ce32700 1 -- 172.21.15.175:0/3658561334 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6800b690 con 0x7f1e7808a100 2023-10-10T19:28:35.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.071+0000 7f1e7ce32700 1 -- 172.21.15.175:0/3658561334 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6800b7f0 con 0x7f1e7808a100 2023-10-10T19:28:35.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ce32700 1 -- 172.21.15.175:0/3658561334 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6800bb90 con 0x7f1e7808a100 2023-10-10T19:28:35.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3658561334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e7808a100 msgr2=0x7f1e780c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:35.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3658561334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e7808a100 0x7f1e780c5eb0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1e78065080 tx=0x7f1e680091a0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3658561334 shutdown_connections 2023-10-10T19:28:35.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3658561334 >> v2:172.21.15.191:3300/0 conn(0x7f1e7814fd90 0x7f1e78159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3658561334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e7808a100 0x7f1e780c5eb0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3658561334 >> 172.21.15.175:0/3658561334 conn(0x7f1e78063d00 msgr2=0x7f1e780cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:35.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3658561334 shutdown_connections 2023-10-10T19:28:35.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.072+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3658561334 wait complete. 2023-10-10T19:28:35.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.073+0000 7f1e7ee36700 1 Processor -- start 2023-10-10T19:28:35.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.073+0000 7f1e7ee36700 1 -- start start 2023-10-10T19:28:35.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7ee36700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e7808a100 0x7f1e781531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:35.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7ee36700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e780cd040 0x7f1e7814bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:35.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7ee36700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e7814fd90 0x7f1e781671d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:35.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7ee36700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78152280 con 0x7f1e780cd040 2023-10-10T19:28:35.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7ee36700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78151390 con 0x7f1e7808a100 2023-10-10T19:28:35.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7ee36700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78151670 con 0x7f1e7814fd90 2023-10-10T19:28:35.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e780cd040 0x7f1e7814bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:35.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e780cd040 0x7f1e7814bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36176/0 (socket says 172.21.15.175:36176) 2023-10-10T19:28:35.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 -- 172.21.15.175:0/3130334123 learned_addr learned my addr 172.21.15.175:0/3130334123 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:35.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7de34700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e7808a100 0x7f1e781531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:35.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7e635700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e7814fd90 0x7f1e781671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:35.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 -- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e7814fd90 msgr2=0x7f1e781671d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:35.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e7814fd90 0x7f1e781671d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 -- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e7808a100 msgr2=0x7f1e781531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:35.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e7808a100 0x7f1e781531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 -- 172.21.15.175:0/3130334123 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e68009420 con 0x7f1e780cd040 2023-10-10T19:28:35.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e7d633700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e780cd040 0x7f1e7814bdf0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1e78064c20 tx=0x7f1e68017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:35.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.074+0000 7f1e6dffb700 1 -- 172.21.15.175:0/3130334123 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68011420 con 0x7f1e780cd040 2023-10-10T19:28:35.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.075+0000 7f1e6dffb700 1 -- 172.21.15.175:0/3130334123 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e68011580 con 0x7f1e780cd040 2023-10-10T19:28:35.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.075+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e78151950 con 0x7f1e780cd040 2023-10-10T19:28:35.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.075+0000 7f1e6dffb700 1 -- 172.21.15.175:0/3130334123 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68011940 con 0x7f1e780cd040 2023-10-10T19:28:35.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.075+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e78151f10 con 0x7f1e780cd040 2023-10-10T19:28:35.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.076+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e781523f0 con 0x7f1e780cd040 2023-10-10T19:28:35.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.076+0000 7f1e6dffb700 1 -- 172.21.15.175:0/3130334123 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1e68011d30 con 0x7f1e780cd040 2023-10-10T19:28:35.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.076+0000 7f1e6dffb700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e5c066f30 0x7f1e5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:35.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.077+0000 7f1e6dffb700 1 -- 172.21.15.175:0/3130334123 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f1e68086470 con 0x7f1e780cd040 2023-10-10T19:28:35.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.077+0000 7f1e7de34700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e5c066f30 0x7f1e5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:35.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.077+0000 7f1e7de34700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e5c066f30 0x7f1e5c0693e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009ca0 tx=0x7f1e74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:35.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.081+0000 7f1e6dffb700 1 -- 172.21.15.175:0/3130334123 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1e68052e60 con 0x7f1e780cd040 2023-10-10T19:28:35.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.305+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e78067570 con 0x7f1e780cd040 2023-10-10T19:28:35.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.306+0000 7f1e6dffb700 1 -- 172.21.15.175:0/3130334123 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) v1 ==== 95+0+5432 (secure 0 0 0) 0x7f1e68056480 con 0x7f1e780cd040 2023-10-10T19:28:35.306 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:35.307 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":314,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":314,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:33.906493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:35.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e5c066f30 msgr2=0x7f1e5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e5c066f30 0x7f1e5c0693e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009ca0 tx=0x7f1e74008040 comp rx=0 tx=0).stop 2023-10-10T19:28:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e780cd040 msgr2=0x7f1e7814bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e780cd040 0x7f1e7814bdf0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1e78064c20 tx=0x7f1e68017000 comp rx=0 tx=0).stop 2023-10-10T19:28:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 shutdown_connections 2023-10-10T19:28:35.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e5c066f30 0x7f1e5c0693e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e7814fd90 0x7f1e781671d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e7808a100 0x7f1e781531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 --2- 172.21.15.175:0/3130334123 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e780cd040 0x7f1e7814bdf0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 >> 172.21.15.175:0/3130334123 conn(0x7f1e78063d00 msgr2=0x7f1e780c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:35.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 shutdown_connections 2023-10-10T19:28:35.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:35.308+0000 7f1e7ee36700 1 -- 172.21.15.175:0/3130334123 wait complete. 2023-10-10T19:28:35.311 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 314 2023-10-10T19:28:35.321 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 314, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 314, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_7057": { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 304, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 139 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_8733": { "addr": "172.21.15.175:6834/3954290407", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3954290407, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3954290407, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8733, "incarnation": 314, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 132, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:28:33.906493+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 8733, "mds_2": 8049, "mds_3": 7771, "mds_4": 7057 } } } ], "standbys": [] } 2023-10-10T19:28:37.322 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:37.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.447+0000 7f73f00b8700 1 Processor -- start 2023-10-10T19:28:37.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.447+0000 7f73f00b8700 1 -- start start 2023-10-10T19:28:37.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.448+0000 7f73f00b8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f73e81573d0 0x7f73e80c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:37.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.448+0000 7f73f00b8700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e8154aa0 con 0x7f73e81577a0 2023-10-10T19:28:37.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.448+0000 7f73f00b8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e8157c20 con 0x7f73e80c4f20 2023-10-10T19:28:37.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.448+0000 7f73f00b8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e80bfdf0 con 0x7f73e81573d0 2023-10-10T19:28:37.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.448+0000 7f73ede54700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f73e81573d0 0x7f73e80c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:37.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.448+0000 7f73ed653700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f73e81577a0 0x7f73e80bf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:37896/0 (socket says 172.21.15.175:37896) 2023-10-10T19:28:37.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.448+0000 7f73ed653700 1 -- 172.21.15.175:0/1055311761 learned_addr learned my addr 172.21.15.175:0/1055311761 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:37.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.449+0000 7f73ede54700 1 -- 172.21.15.175:0/1055311761 >> v1:172.21.15.191:6789/0 conn(0x7f73e80c4f20 legacy=0x7f73e814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:37.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.449+0000 7f73ede54700 1 -- 172.21.15.175:0/1055311761 >> v1:172.21.15.175:6789/0 conn(0x7f73e81577a0 legacy=0x7f73e80bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:37.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.449+0000 7f73ece52700 1 -- 172.21.15.175:0/1055311761 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387845539 0 0) 0x7f73e8154aa0 con 0x7f73e81577a0 2023-10-10T19:28:37.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.449+0000 7f73ede54700 1 -- 172.21.15.175:0/1055311761 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8063dc0 con 0x7f73e81573d0 2023-10-10T19:28:37.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.449+0000 7f73ede54700 1 --2- 172.21.15.175:0/1055311761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e80c12a0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009e60 tx=0x7f73d8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa91bcc901e0d84 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:37.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.449+0000 7f73ece52700 1 -- 172.21.15.175:0/1055311761 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d800e070 con 0x7f73e81573d0 2023-10-10T19:28:37.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.449+0000 7f73ece52700 1 -- 172.21.15.175:0/1055311761 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d800bbf0 con 0x7f73e81573d0 2023-10-10T19:28:37.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.449+0000 7f73ece52700 1 -- 172.21.15.175:0/1055311761 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8012430 con 0x7f73e81573d0 2023-10-10T19:28:37.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.450+0000 7f73f00b8700 1 -- 172.21.15.175:0/1055311761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 msgr2=0x7f73e80c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:37.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.450+0000 7f73f00b8700 1 --2- 172.21.15.175:0/1055311761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e80c12a0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009e60 tx=0x7f73d8009780 comp rx=0 tx=0).stop 2023-10-10T19:28:37.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.450+0000 7f73f00b8700 1 -- 172.21.15.175:0/1055311761 shutdown_connections 2023-10-10T19:28:37.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.450+0000 7f73f00b8700 1 --2- 172.21.15.175:0/1055311761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e80c12a0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:37.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.450+0000 7f73f00b8700 1 -- 172.21.15.175:0/1055311761 >> 172.21.15.175:0/1055311761 conn(0x7f73e806dc70 msgr2=0x7f73e8063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:37.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.450+0000 7f73f00b8700 1 -- 172.21.15.175:0/1055311761 shutdown_connections 2023-10-10T19:28:37.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.450+0000 7f73f00b8700 1 -- 172.21.15.175:0/1055311761 wait complete. 2023-10-10T19:28:37.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.451+0000 7f73f00b8700 1 Processor -- start 2023-10-10T19:28:37.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.451+0000 7f73f00b8700 1 -- start start 2023-10-10T19:28:37.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73f00b8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73e80c4f20 0x7f73e8068b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:37.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73f00b8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e81584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:37.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73f00b8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73e81577a0 0x7f73e8069950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:37.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73f00b8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e81531c0 con 0x7f73e81573d0 2023-10-10T19:28:37.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73f00b8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e8152d30 con 0x7f73e80c4f20 2023-10-10T19:28:37.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73f00b8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e80691b0 con 0x7f73e81577a0 2023-10-10T19:28:37.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e81584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:37.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e81584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36194/0 (socket says 172.21.15.175:36194) 2023-10-10T19:28:37.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 -- 172.21.15.175:0/1975248879 learned_addr learned my addr 172.21.15.175:0/1975248879 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:37.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ede54700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73e80c4f20 0x7f73e8068b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:37.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ee655700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73e81577a0 0x7f73e8069950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:37.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 -- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73e81577a0 msgr2=0x7f73e8069950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:37.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73e81577a0 0x7f73e8069950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:37.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 -- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73e80c4f20 msgr2=0x7f73e8068b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:37.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73e80c4f20 0x7f73e8068b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:37.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 -- 172.21.15.175:0/1975248879 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d8009a20 con 0x7f73e81573d0 2023-10-10T19:28:37.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ee655700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73e81577a0 0x7f73e8069950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:28:37.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73ed653700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e81584f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00c7e0 tx=0x7f73dc007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:37.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73e5ffb700 1 -- 172.21.15.175:0/1975248879 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc005b30 con 0x7f73e81573d0 2023-10-10T19:28:37.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73e5ffb700 1 -- 172.21.15.175:0/1975248879 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc005c90 con 0x7f73e81573d0 2023-10-10T19:28:37.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e8069490 con 0x7f73e81573d0 2023-10-10T19:28:37.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.452+0000 7f73e5ffb700 1 -- 172.21.15.175:0/1975248879 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc007300 con 0x7f73e81573d0 2023-10-10T19:28:37.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.453+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e81529c0 con 0x7f73e81573d0 2023-10-10T19:28:37.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.453+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e8152ea0 con 0x7f73e81573d0 2023-10-10T19:28:37.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.454+0000 7f73e5ffb700 1 -- 172.21.15.175:0/1975248879 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f73dc01b400 con 0x7f73e81573d0 2023-10-10T19:28:37.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.454+0000 7f73e5ffb700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:37.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.454+0000 7f73e5ffb700 1 -- 172.21.15.175:0/1975248879 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f73dc0848c0 con 0x7f73e81573d0 2023-10-10T19:28:37.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.454+0000 7f73ede54700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:37.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.455+0000 7f73ede54700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f73d80097c0 tx=0x7f73d8018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:37.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.458+0000 7f73e5ffb700 1 -- 172.21.15.175:0/1975248879 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73dc051330 con 0x7f73e81573d0 2023-10-10T19:28:37.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.681+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73e8067570 con 0x7f73e81573d0 2023-10-10T19:28:37.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.682+0000 7f73e5ffb700 1 -- 172.21.15.175:0/1975248879 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) v1 ==== 95+0+5432 (secure 0 0 0) 0x7f73dc054950 con 0x7f73e81573d0 2023-10-10T19:28:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:37.683 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":314,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":314,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:33.906493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 msgr2=0x7f73d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f73d80097c0 tx=0x7f73d8018000 comp rx=0 tx=0).stop 2023-10-10T19:28:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 msgr2=0x7f73e81584f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e81584f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00c7e0 tx=0x7f73dc007df0 comp rx=0 tx=0).stop 2023-10-10T19:28:37.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 shutdown_connections 2023-10-10T19:28:37.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f73d4066f30 0x7f73d40693e0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:37.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f73e81577a0 0x7f73e8069950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:37.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73e80c4f20 0x7f73e8068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:37.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 --2- 172.21.15.175:0/1975248879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f73e81573d0 0x7f73e81584f0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:37.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 >> 172.21.15.175:0/1975248879 conn(0x7f73e806dc70 msgr2=0x7f73e80c5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:37.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 shutdown_connections 2023-10-10T19:28:37.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:37.684+0000 7f73f00b8700 1 -- 172.21.15.175:0/1975248879 wait complete. 2023-10-10T19:28:37.687 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 314 2023-10-10T19:28:37.696 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:28:39.697 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:39.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.814+0000 7fd8348b4700 1 Processor -- start 2023-10-10T19:28:39.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.814+0000 7fd8348b4700 1 -- start start 2023-10-10T19:28:39.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.814+0000 7fd8348b4700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd830068cf0 0x7fd830067b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:39.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.814+0000 7fd8348b4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd830159dd0 0x7fd8300c2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:39.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd8348b4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd830068b30 con 0x7fd83014fd20 2023-10-10T19:28:39.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd8348b4700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8300c1c70 con 0x7fd830159dd0 2023-10-10T19:28:39.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd8348b4700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8300c2640 con 0x7fd830068cf0 2023-10-10T19:28:39.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82e59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd830159dd0 0x7fd8300c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:39.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82e59c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd830159dd0 0x7fd8300c2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36210/0 (socket says 172.21.15.175:36210) 2023-10-10T19:28:39.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82e59c700 1 -- 172.21.15.175:0/1581689049 learned_addr learned my addr 172.21.15.175:0/1581689049 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:39.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82ed9d700 1 --2- 172.21.15.175:0/1581689049 >> v2:172.21.15.191:3300/0 conn(0x7fd830068cf0 0x7fd830067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:39.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82e59c700 1 -- 172.21.15.175:0/1581689049 >> v2:172.21.15.191:3300/0 conn(0x7fd830068cf0 msgr2=0x7fd830067b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:39.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82e59c700 1 --2- 172.21.15.175:0/1581689049 >> v2:172.21.15.191:3300/0 conn(0x7fd830068cf0 0x7fd830067b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:39.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82e59c700 1 -- 172.21.15.175:0/1581689049 >> v1:172.21.15.191:6789/0 conn(0x7fd83014fd20 legacy=0x7fd8300690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:39.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82e59c700 1 -- 172.21.15.175:0/1581689049 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8300cd2b0 con 0x7fd830159dd0 2023-10-10T19:28:39.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82e59c700 1 --2- 172.21.15.175:0/1581689049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830159dd0 0x7fd8300c2100 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fd8240099b0 tx=0x7fd824009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34d8f5e64cc4676 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:39.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82dd9b700 1 -- 172.21.15.175:0/1581689049 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82400b8b0 con 0x7fd830159dd0 2023-10-10T19:28:39.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82dd9b700 1 -- 172.21.15.175:0/1581689049 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82400ba10 con 0x7fd830159dd0 2023-10-10T19:28:39.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.815+0000 7fd82dd9b700 1 -- 172.21.15.175:0/1581689049 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82400bdb0 con 0x7fd830159dd0 2023-10-10T19:28:39.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.816+0000 7fd8348b4700 1 -- 172.21.15.175:0/1581689049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830159dd0 msgr2=0x7fd8300c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:39.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.816+0000 7fd8348b4700 1 --2- 172.21.15.175:0/1581689049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830159dd0 0x7fd8300c2100 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fd8240099b0 tx=0x7fd824009250 comp rx=0 tx=0).stop 2023-10-10T19:28:39.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.816+0000 7fd8348b4700 1 -- 172.21.15.175:0/1581689049 shutdown_connections 2023-10-10T19:28:39.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.816+0000 7fd8348b4700 1 --2- 172.21.15.175:0/1581689049 >> v2:172.21.15.191:3300/0 conn(0x7fd830068cf0 0x7fd830067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:39.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.816+0000 7fd8348b4700 1 --2- 172.21.15.175:0/1581689049 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830159dd0 0x7fd8300c2100 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:39.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.816+0000 7fd8348b4700 1 -- 172.21.15.175:0/1581689049 >> 172.21.15.175:0/1581689049 conn(0x7fd830063cc0 msgr2=0x7fd8300cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:39.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.816+0000 7fd8348b4700 1 -- 172.21.15.175:0/1581689049 shutdown_connections 2023-10-10T19:28:39.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.816+0000 7fd8348b4700 1 -- 172.21.15.175:0/1581689049 wait complete. 2023-10-10T19:28:39.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.817+0000 7fd8348b4700 1 Processor -- start 2023-10-10T19:28:39.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.817+0000 7fd8348b4700 1 -- start start 2023-10-10T19:28:39.825 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd8348b4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830068cf0 0x7fd830155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:39.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd8348b4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd83014fd20 0x7fd83015b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:39.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd8348b4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd830159dd0 0x7fd830153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:39.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd8348b4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300c6970 con 0x7fd830068cf0 2023-10-10T19:28:39.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd8348b4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830156920 con 0x7fd83014fd20 2023-10-10T19:28:39.826 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd8348b4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd830156c00 con 0x7fd830159dd0 2023-10-10T19:28:39.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830068cf0 0x7fd830155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:39.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82ed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830068cf0 0x7fd830155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36218/0 (socket says 172.21.15.175:36218) 2023-10-10T19:28:39.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82ed9d700 1 -- 172.21.15.175:0/4135940713 learned_addr learned my addr 172.21.15.175:0/4135940713 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:39.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82e59c700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd83014fd20 0x7fd83015b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:39.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82f59e700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd830159dd0 0x7fd830153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:39.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82ed9d700 1 -- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd830159dd0 msgr2=0x7fd830153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:39.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82ed9d700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd830159dd0 0x7fd830153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:39.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82ed9d700 1 -- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd83014fd20 msgr2=0x7fd83015b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:39.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82ed9d700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd83014fd20 0x7fd83015b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:39.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82ed9d700 1 -- 172.21.15.175:0/4135940713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8240094f0 con 0x7fd830068cf0 2023-10-10T19:28:39.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.818+0000 7fd82f59e700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd830159dd0 0x7fd830153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:28:39.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.819+0000 7fd82ed9d700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830068cf0 0x7fd830155980 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fd82000d890 tx=0x7fd820008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:39.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.819+0000 7fd816ffd700 1 -- 172.21.15.175:0/4135940713 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8200125f0 con 0x7fd830068cf0 2023-10-10T19:28:39.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.819+0000 7fd816ffd700 1 -- 172.21.15.175:0/4135940713 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd820012750 con 0x7fd830068cf0 2023-10-10T19:28:39.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.819+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8300c6040 con 0x7fd830068cf0 2023-10-10T19:28:39.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.819+0000 7fd816ffd700 1 -- 172.21.15.175:0/4135940713 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820012ad0 con 0x7fd830068cf0 2023-10-10T19:28:39.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.819+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8300c6600 con 0x7fd830068cf0 2023-10-10T19:28:39.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.820+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd830067570 con 0x7fd830068cf0 2023-10-10T19:28:39.831 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.820+0000 7fd816ffd700 1 -- 172.21.15.175:0/4135940713 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd82001e460 con 0x7fd830068cf0 2023-10-10T19:28:39.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.821+0000 7fd816ffd700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd81c066f30 0x7fd81c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:39.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.821+0000 7fd816ffd700 1 -- 172.21.15.175:0/4135940713 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fd820085300 con 0x7fd830068cf0 2023-10-10T19:28:39.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.821+0000 7fd82e59c700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd81c066f30 0x7fd81c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:39.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.821+0000 7fd82e59c700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd81c066f30 0x7fd81c0693e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd8240099b0 tx=0x7fd824017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:39.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:39.828+0000 7fd816ffd700 1 -- 172.21.15.175:0/4135940713 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd820051cf0 con 0x7fd830068cf0 2023-10-10T19:28:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.065+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd83008a4d0 con 0x7fd830068cf0 2023-10-10T19:28:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.066+0000 7fd816ffd700 1 -- 172.21.15.175:0/4135940713 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) v1 ==== 95+0+5432 (secure 0 0 0) 0x7fd820055310 con 0x7fd830068cf0 2023-10-10T19:28:40.067 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:40.067 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":314,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":314,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:33.906493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:replay","state_seq":1,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd81c066f30 msgr2=0x7fd81c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd81c066f30 0x7fd81c0693e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd8240099b0 tx=0x7fd824017000 comp rx=0 tx=0).stop 2023-10-10T19:28:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830068cf0 msgr2=0x7fd830155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830068cf0 0x7fd830155980 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fd82000d890 tx=0x7fd820008040 comp rx=0 tx=0).stop 2023-10-10T19:28:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 shutdown_connections 2023-10-10T19:28:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd81c066f30 0x7fd81c0693e0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd830159dd0 0x7fd830153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd830068cf0 0x7fd830155980 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 --2- 172.21.15.175:0/4135940713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd83014fd20 0x7fd83015b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 >> 172.21.15.175:0/4135940713 conn(0x7fd830063cc0 msgr2=0x7fd8300b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 shutdown_connections 2023-10-10T19:28:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:40.068+0000 7fd8348b4700 1 -- 172.21.15.175:0/4135940713 wait complete. 2023-10-10T19:28:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 314 2023-10-10T19:28:40.084 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:28:42.086 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:42.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.200+0000 7fab1a4aa700 1 Processor -- start 2023-10-10T19:28:42.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.200+0000 7fab1a4aa700 1 -- start start 2023-10-10T19:28:42.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab1a4aa700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fab1414cdc0 0x7fab140c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:42.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab1a4aa700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fab1406f620 0x7fab140c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:42.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab1a4aa700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1406e940 con 0x7fab1414d190 2023-10-10T19:28:42.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab1a4aa700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab140c1c90 con 0x7fab1406f620 2023-10-10T19:28:42.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab1a4aa700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab140c6e20 con 0x7fab1414cdc0 2023-10-10T19:28:42.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab18a47700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fab1406f620 0x7fab140c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:42.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab18a47700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fab1406f620 0x7fab140c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36234/0 (socket says 172.21.15.175:36234) 2023-10-10T19:28:42.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab18a47700 1 -- 172.21.15.175:0/1935263956 learned_addr learned my addr 172.21.15.175:0/1935263956 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:42.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab13fff700 1 --2- 172.21.15.175:0/1935263956 >> v2:172.21.15.191:3300/0 conn(0x7fab1414cdc0 0x7fab140c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:42.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab18a47700 1 -- 172.21.15.175:0/1935263956 >> v2:172.21.15.191:3300/0 conn(0x7fab1414cdc0 msgr2=0x7fab140c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:42.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab18a47700 1 --2- 172.21.15.175:0/1935263956 >> v2:172.21.15.191:3300/0 conn(0x7fab1414cdc0 0x7fab140c26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab18a47700 1 -- 172.21.15.175:0/1935263956 >> v1:172.21.15.191:6789/0 conn(0x7fab1414d190 legacy=0x7fab14156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:42.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab18a47700 1 -- 172.21.15.175:0/1935263956 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1406ca40 con 0x7fab1406f620 2023-10-10T19:28:42.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab18a47700 1 --2- 172.21.15.175:0/1935263956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab140c2c20 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009f20 tx=0x7fab0c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde06b6659d7e2cc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:42.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.201+0000 7fab12ffd700 1 -- 172.21.15.175:0/1935263956 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c00e070 con 0x7fab1406f620 2023-10-10T19:28:42.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab12ffd700 1 -- 172.21.15.175:0/1935263956 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0c00ba00 con 0x7fab1406f620 2023-10-10T19:28:42.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab12ffd700 1 -- 172.21.15.175:0/1935263956 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c0092e0 con 0x7fab1406f620 2023-10-10T19:28:42.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1935263956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 msgr2=0x7fab140c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:42.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1935263956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab140c2c20 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009f20 tx=0x7fab0c009840 comp rx=0 tx=0).stop 2023-10-10T19:28:42.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1935263956 shutdown_connections 2023-10-10T19:28:42.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1935263956 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab140c2c20 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1935263956 >> v2:172.21.15.191:3300/0 conn(0x7fab1414cdc0 0x7fab140c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1935263956 >> 172.21.15.175:0/1935263956 conn(0x7fab14068900 msgr2=0x7fab14068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:42.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1935263956 shutdown_connections 2023-10-10T19:28:42.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.202+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1935263956 wait complete. 2023-10-10T19:28:42.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.203+0000 7fab1a4aa700 1 Processor -- start 2023-10-10T19:28:42.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.203+0000 7fab1a4aa700 1 -- start start 2023-10-10T19:28:42.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab1a4aa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab1414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:42.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab1a4aa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab1414cdc0 0x7fab141671b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:42.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab1a4aa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab1414d190 0x7fab141676f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:42.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab1a4aa700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab14150510 con 0x7fab1406f620 2023-10-10T19:28:42.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab1a4aa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab141507a0 con 0x7fab1414cdc0 2023-10-10T19:28:42.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab1a4aa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab140c8470 con 0x7fab1414d190 2023-10-10T19:28:42.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab1414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:42.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab1414bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36244/0 (socket says 172.21.15.175:36244) 2023-10-10T19:28:42.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 -- 172.21.15.175:0/1014004580 learned_addr learned my addr 172.21.15.175:0/1014004580 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:42.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab137fe700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab1414cdc0 0x7fab141671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:42.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab18a47700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab1414d190 0x7fab141676f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:42.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 -- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab1414d190 msgr2=0x7fab141676f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:42.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab1414d190 0x7fab141676f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 -- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab1414cdc0 msgr2=0x7fab141671b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:42.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab1414cdc0 0x7fab141671b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 -- 172.21.15.175:0/1014004580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c009ae0 con 0x7fab1406f620 2023-10-10T19:28:42.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.204+0000 7fab13fff700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab1414bfc0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fab0400c850 tx=0x7fab04008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:42.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.205+0000 7faafbfff700 1 -- 172.21.15.175:0/1014004580 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04011600 con 0x7fab1406f620 2023-10-10T19:28:42.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.205+0000 7faafbfff700 1 -- 172.21.15.175:0/1014004580 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab04011760 con 0x7fab1406f620 2023-10-10T19:28:42.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.205+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab140c0310 con 0x7fab1406f620 2023-10-10T19:28:42.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.205+0000 7faafbfff700 1 -- 172.21.15.175:0/1014004580 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04011b00 con 0x7fab1406f620 2023-10-10T19:28:42.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.205+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab140c05f0 con 0x7fab1406f620 2023-10-10T19:28:42.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.206+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab14067570 con 0x7fab1406f620 2023-10-10T19:28:42.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.206+0000 7faafbfff700 1 -- 172.21.15.175:0/1014004580 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fab040193f0 con 0x7fab1406f620 2023-10-10T19:28:42.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.206+0000 7faafbfff700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab00066f30 0x7fab000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:42.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.206+0000 7faafbfff700 1 -- 172.21.15.175:0/1014004580 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fab04085120 con 0x7fab1406f620 2023-10-10T19:28:42.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.206+0000 7fab137fe700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab00066f30 0x7fab000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:42.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.207+0000 7fab137fe700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab00066f30 0x7fab000693e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fab14150ae0 tx=0x7fab08009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:42.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.210+0000 7faafbfff700 1 -- 172.21.15.175:0/1014004580 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab040555a0 con 0x7fab1406f620 2023-10-10T19:28:42.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.436+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab14166dd0 con 0x7fab1406f620 2023-10-10T19:28:42.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.436+0000 7faafbfff700 1 -- 172.21.15.175:0/1014004580 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 315 v315) v1 ==== 95+0+5434 (secure 0 0 0) 0x7fab04055130 con 0x7fab1406f620 2023-10-10T19:28:42.438 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:42.438 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":315,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":315,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:41.995092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:42.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab00066f30 msgr2=0x7fab000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:42.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab00066f30 0x7fab000693e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fab14150ae0 tx=0x7fab08009060 comp rx=0 tx=0).stop 2023-10-10T19:28:42.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 msgr2=0x7fab1414bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:42.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab1414bfc0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fab0400c850 tx=0x7fab04008040 comp rx=0 tx=0).stop 2023-10-10T19:28:42.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 shutdown_connections 2023-10-10T19:28:42.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fab00066f30 0x7fab000693e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fab1414d190 0x7fab141676f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fab1406f620 0x7fab1414bfc0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 --2- 172.21.15.175:0/1014004580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab1414cdc0 0x7fab141671b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:42.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 >> 172.21.15.175:0/1014004580 conn(0x7fab14068900 msgr2=0x7fab140c8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:42.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 shutdown_connections 2023-10-10T19:28:42.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:42.438+0000 7fab1a4aa700 1 -- 172.21.15.175:0/1014004580 wait complete. 2023-10-10T19:28:42.443 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 315 2023-10-10T19:28:42.451 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:28:44.452 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:44.578 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.577+0000 7f6869b00700 1 Processor -- start 2023-10-10T19:28:44.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.577+0000 7f6869b00700 1 -- start start 2023-10-10T19:28:44.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f6869b00700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68640b2390 0x7f6864155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:44.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f6869b00700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68641574b0 con 0x7f68640b1fc0 2023-10-10T19:28:44.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f6869b00700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68640afad0 con 0x7f68640aff60 2023-10-10T19:28:44.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f6869b00700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6864150770 con 0x7f68640b2390 2023-10-10T19:28:44.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68637fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f68640b1fc0 0x7f6864068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41596/0 (socket says 172.21.15.175:41596) 2023-10-10T19:28:44.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68637fe700 1 -- 172.21.15.175:0/4253384727 learned_addr learned my addr 172.21.15.175:0/4253384727 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:44.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f6862ffd700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68640b2390 0x7f6864155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:44.583 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284390768 0 0) 0x7f68641574b0 con 0x7f68640b1fc0 2023-10-10T19:28:44.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684c003570 con 0x7f68640b1fc0 2023-10-10T19:28:44.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3876003999 0 0) 0x7f684c003570 con 0x7f68640b1fc0 2023-10-10T19:28:44.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68640b2390 msgr2=0x7f6864155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:44.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68640b2390 0x7f6864155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.584 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 >> v1:172.21.15.191:6789/0 conn(0x7f68640aff60 legacy=0x7f6864159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:44.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68640af4c0 con 0x7f68640b1fc0 2023-10-10T19:28:44.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f6854001470 con 0x7f68640b1fc0 2023-10-10T19:28:44.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 >> v1:172.21.15.175:6789/0 conn(0x7f68640b1fc0 legacy=0x7f6864068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:44.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.578+0000 7f68627fc700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f684c003a90 0x7f684c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:44.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68627fc700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68640b2390 0x7f684c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68627fc700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f684c004d50 0x7f684c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864150770 con 0x7f684c004d50 2023-10-10T19:28:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c00bba0 con 0x7f684c003a90 2023-10-10T19:28:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c00bd60 con 0x7f68640b2390 2023-10-10T19:28:44.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f6863fff700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f684c004d50 0x7f684c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:44.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68637fe700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f684c003a90 0x7f684c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:44.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f6862ffd700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68640b2390 0x7f684c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:44.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f6863fff700 1 -- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68640b2390 msgr2=0x7f684c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:44.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f6863fff700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68640b2390 0x7f684c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f6863fff700 1 -- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f684c003a90 msgr2=0x7f684c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:44.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f6863fff700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f684c003a90 0x7f684c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f6863fff700 1 -- 172.21.15.175:0/4253384727 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c003570 con 0x7f684c004d50 2023-10-10T19:28:44.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f6863fff700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f684c004d50 0x7f684c0051c0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f685c001fc0 tx=0x7f685c001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77197ef1795608b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:44.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c001950 con 0x7f684c004d50 2023-10-10T19:28:44.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c00c8b0 con 0x7f684c004d50 2023-10-10T19:28:44.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.579+0000 7f68627fc700 1 -- 172.21.15.175:0/4253384727 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c00cc50 con 0x7f684c004d50 2023-10-10T19:28:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.580+0000 7f6869b00700 1 -- 172.21.15.175:0/4253384727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f684c004d50 msgr2=0x7f684c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.580+0000 7f6869b00700 1 --2- 172.21.15.175:0/4253384727 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f684c004d50 0x7f684c0051c0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f685c001fc0 tx=0x7f685c001640 comp rx=0 tx=0).stop 2023-10-10T19:28:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.580+0000 7f6863fff700 1 -- 172.21.15.175:0/4253384727 reap_dead start 2023-10-10T19:28:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.580+0000 7f6869b00700 1 -- 172.21.15.175:0/4253384727 shutdown_connections 2023-10-10T19:28:44.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.580+0000 7f6869b00700 1 -- 172.21.15.175:0/4253384727 >> 172.21.15.175:0/4253384727 conn(0x7f6864063230 msgr2=0x7f6864063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:44.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.580+0000 7f6869b00700 1 -- 172.21.15.175:0/4253384727 shutdown_connections 2023-10-10T19:28:44.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.580+0000 7f6869b00700 1 -- 172.21.15.175:0/4253384727 wait complete. 2023-10-10T19:28:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.581+0000 7f6869b00700 1 Processor -- start 2023-10-10T19:28:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.581+0000 7f6869b00700 1 -- start start 2023-10-10T19:28:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6869b00700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6864152f10 0x7f6864157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6869b00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68641532e0 0x7f6864146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:44.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6869b00700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6864146fd0 0x7f6864158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6869b00700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864147420 con 0x7f6864152f10 2023-10-10T19:28:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6869b00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864159f90 con 0x7f68641532e0 2023-10-10T19:28:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6869b00700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686415a250 con 0x7f6864146fd0 2023-10-10T19:28:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6864152f10 0x7f6864157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6864152f10 0x7f6864157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55746/0 (socket says 172.21.15.175:55746) 2023-10-10T19:28:44.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 -- 172.21.15.175:0/1712516165 learned_addr learned my addr 172.21.15.175:0/1712516165 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:44.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6862ffd700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68641532e0 0x7f6864146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:44.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6863fff700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6864146fd0 0x7f6864158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:44.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 -- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6864146fd0 msgr2=0x7f6864158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:44.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6864146fd0 0x7f6864158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 -- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68641532e0 msgr2=0x7f6864146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:44.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68641532e0 0x7f6864146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 -- 172.21.15.175:0/1712516165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686414cda0 con 0x7f6864152f10 2023-10-10T19:28:44.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f6863fff700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6864146fd0 0x7f6864158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:28:44.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f68637fe700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6864152f10 0x7f6864157180 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f6854007390 tx=0x7f68640afa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.582+0000 7f684b7fe700 1 -- 172.21.15.175:0/1712516165 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6854018030 con 0x7f6864152f10 2023-10-10T19:28:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.583+0000 7f684b7fe700 1 -- 172.21.15.175:0/1712516165 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685400f960 con 0x7f6864152f10 2023-10-10T19:28:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.583+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686414d080 con 0x7f6864152f10 2023-10-10T19:28:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.583+0000 7f684b7fe700 1 -- 172.21.15.175:0/1712516165 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685400fd20 con 0x7f6864152f10 2023-10-10T19:28:44.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.583+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686414d640 con 0x7f6864152f10 2023-10-10T19:28:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.584+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686414d830 con 0x7f6864152f10 2023-10-10T19:28:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.584+0000 7f684b7fe700 1 -- 172.21.15.175:0/1712516165 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f685401b070 con 0x7f6864152f10 2023-10-10T19:28:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.584+0000 7f684b7fe700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6850066f60 0x7f6850069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.584+0000 7f684b7fe700 1 -- 172.21.15.175:0/1712516165 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f6854019070 con 0x7f6864152f10 2023-10-10T19:28:44.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.584+0000 7f6862ffd700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6850066f60 0x7f6850069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:44.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.585+0000 7f6862ffd700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6850066f60 0x7f6850069410 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f68640c7d00 tx=0x7f6858008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:44.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.589+0000 7f684b7fe700 1 -- 172.21.15.175:0/1712516165 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f68540517a0 con 0x7f6864152f10 2023-10-10T19:28:44.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.815+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6864067570 con 0x7f6864152f10 2023-10-10T19:28:44.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.816+0000 7f684b7fe700 1 -- 172.21.15.175:0/1712516165 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 315 v315) v1 ==== 95+0+5434 (secure 0 0 0) 0x7f6854054dc0 con 0x7f6864152f10 2023-10-10T19:28:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:44.818 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":315,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":315,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:41.995092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:44.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6850066f60 msgr2=0x7f6850069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:44.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6850066f60 0x7f6850069410 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f68640c7d00 tx=0x7f6858008040 comp rx=0 tx=0).stop 2023-10-10T19:28:44.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6864152f10 msgr2=0x7f6864157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:44.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6864152f10 0x7f6864157180 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f6854007390 tx=0x7f68640afa40 comp rx=0 tx=0).stop 2023-10-10T19:28:44.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 shutdown_connections 2023-10-10T19:28:44.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6850066f60 0x7f6850069410 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6864146fd0 0x7f6864158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6864152f10 0x7f6864157180 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 --2- 172.21.15.175:0/1712516165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68641532e0 0x7f6864146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:44.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 >> 172.21.15.175:0/1712516165 conn(0x7f6864063230 msgr2=0x7f68641508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:44.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 shutdown_connections 2023-10-10T19:28:44.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:44.818+0000 7f6869b00700 1 -- 172.21.15.175:0/1712516165 wait complete. 2023-10-10T19:28:44.822 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 315 2023-10-10T19:28:44.831 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:28:46.832 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:46.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.957+0000 7f192ab9d700 1 Processor -- start 2023-10-10T19:28:46.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.957+0000 7f192ab9d700 1 -- start start 2023-10-10T19:28:46.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f192ab9d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f19240b1fc0 0x7f1924155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:46.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f192ab9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f19240b2390 0x7f1924068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f192ab9d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19241574b0 con 0x7f1924068bd0 2023-10-10T19:28:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f192ab9d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1924150770 con 0x7f19240b2390 2023-10-10T19:28:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f192ab9d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19240afad0 con 0x7f19240b1fc0 2023-10-10T19:28:46.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f1923fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f19240b2390 0x7f1924068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f192913a700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f1924068bd0 0x7f19240aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41598/0 (socket says 172.21.15.175:41598) 2023-10-10T19:28:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f1923fff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f19240b2390 0x7f1924068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55760/0 (socket says 172.21.15.175:55760) 2023-10-10T19:28:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f192913a700 1 -- 172.21.15.175:0/2884572089 learned_addr learned my addr 172.21.15.175:0/2884572089 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f1928939700 1 --2- 172.21.15.175:0/2884572089 >> v2:172.21.15.191:3300/0 conn(0x7f19240b1fc0 0x7f1924155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:46.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f1923fff700 1 -- 172.21.15.175:0/2884572089 >> v2:172.21.15.191:3300/0 conn(0x7f19240b1fc0 msgr2=0x7f1924155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f1923fff700 1 --2- 172.21.15.175:0/2884572089 >> v2:172.21.15.191:3300/0 conn(0x7f19240b1fc0 0x7f1924155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f1923fff700 1 -- 172.21.15.175:0/2884572089 >> v1:172.21.15.175:6789/0 conn(0x7f1924068bd0 legacy=0x7f19240aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f1923fff700 1 -- 172.21.15.175:0/2884572089 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19240af4c0 con 0x7f19240b2390 2023-10-10T19:28:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f19237fe700 1 -- 172.21.15.175:0/2884572089 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422468933 0 0) 0x7f19241574b0 con 0x7f1924068bd0 2023-10-10T19:28:46.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f1923fff700 1 --2- 172.21.15.175:0/2884572089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924068690 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f19240b28c0 tx=0x7f1914009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f4ac128cca1d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:46.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f19237fe700 1 -- 172.21.15.175:0/2884572089 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191400e070 con 0x7f19240b2390 2023-10-10T19:28:46.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f19237fe700 1 -- 172.21.15.175:0/2884572089 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191400bcc0 con 0x7f19240b2390 2023-10-10T19:28:46.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.958+0000 7f19237fe700 1 -- 172.21.15.175:0/2884572089 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1914012680 con 0x7f19240b2390 2023-10-10T19:28:46.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.959+0000 7f192ab9d700 1 -- 172.21.15.175:0/2884572089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 msgr2=0x7f1924068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:46.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.959+0000 7f192ab9d700 1 --2- 172.21.15.175:0/2884572089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924068690 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f19240b28c0 tx=0x7f1914009780 comp rx=0 tx=0).stop 2023-10-10T19:28:46.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.959+0000 7f192ab9d700 1 -- 172.21.15.175:0/2884572089 shutdown_connections 2023-10-10T19:28:46.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.959+0000 7f192ab9d700 1 --2- 172.21.15.175:0/2884572089 >> v2:172.21.15.191:3300/0 conn(0x7f19240b1fc0 0x7f1924155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:46.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.959+0000 7f192ab9d700 1 --2- 172.21.15.175:0/2884572089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924068690 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:46.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.959+0000 7f192ab9d700 1 -- 172.21.15.175:0/2884572089 >> 172.21.15.175:0/2884572089 conn(0x7f1924063230 msgr2=0x7f1924063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:46.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.959+0000 7f192ab9d700 1 -- 172.21.15.175:0/2884572089 shutdown_connections 2023-10-10T19:28:46.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.959+0000 7f192ab9d700 1 -- 172.21.15.175:0/2884572089 wait complete. 2023-10-10T19:28:46.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.960+0000 7f192ab9d700 1 Processor -- start 2023-10-10T19:28:46.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.960+0000 7f192ab9d700 1 -- start start 2023-10-10T19:28:46.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192ab9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1924068bd0 0x7f19240cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:46.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192ab9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19240b1fc0 0x7f1924157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:46.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192ab9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:46.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192ab9d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1924169b10 con 0x7f19240b2390 2023-10-10T19:28:46.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192ab9d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1924168f70 con 0x7f1924068bd0 2023-10-10T19:28:46.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192ab9d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19241691e0 con 0x7f19240b1fc0 2023-10-10T19:28:46.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192913a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:46.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192913a700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924157340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55766/0 (socket says 172.21.15.175:55766) 2023-10-10T19:28:46.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192913a700 1 -- 172.21.15.175:0/3465952509 learned_addr learned my addr 172.21.15.175:0/3465952509 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:46.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f1928939700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1924068bd0 0x7f19240cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:46.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f1923fff700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19240b1fc0 0x7f1924157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:46.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192913a700 1 -- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19240b1fc0 msgr2=0x7f1924157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:46.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192913a700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19240b1fc0 0x7f1924157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:46.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192913a700 1 -- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1924068bd0 msgr2=0x7f19240cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:46.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192913a700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1924068bd0 0x7f19240cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:46.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.961+0000 7f192913a700 1 -- 172.21.15.175:0/3465952509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1914009a20 con 0x7f19240b2390 2023-10-10T19:28:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.962+0000 7f192913a700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924157340 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f190c00c5d0 tx=0x7f190c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.962+0000 7f1920ff9700 1 -- 172.21.15.175:0/3465952509 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c005cd0 con 0x7f19240b2390 2023-10-10T19:28:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.962+0000 7f1920ff9700 1 -- 172.21.15.175:0/3465952509 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c005e30 con 0x7f19240b2390 2023-10-10T19:28:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.962+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19241694c0 con 0x7f19240b2390 2023-10-10T19:28:46.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.962+0000 7f1920ff9700 1 -- 172.21.15.175:0/3465952509 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c007320 con 0x7f19240b2390 2023-10-10T19:28:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.962+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19240c3f30 con 0x7f19240b2390 2023-10-10T19:28:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.963+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1924158bc0 con 0x7f19240b2390 2023-10-10T19:28:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.963+0000 7f1920ff9700 1 -- 172.21.15.175:0/3465952509 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f190c007690 con 0x7f19240b2390 2023-10-10T19:28:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.964+0000 7f1920ff9700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1904066f60 0x7f1904069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:46.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.964+0000 7f1920ff9700 1 -- 172.21.15.175:0/3465952509 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f190c084440 con 0x7f19240b2390 2023-10-10T19:28:46.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.964+0000 7f1928939700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1904066f60 0x7f1904069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:46.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.964+0000 7f1928939700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1904066f60 0x7f1904069410 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f191c006fd0 tx=0x7f191c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:46.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:46.968+0000 7f1920ff9700 1 -- 172.21.15.175:0/3465952509 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1924158bc0 con 0x7f19240b2390 2023-10-10T19:28:47.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.213+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1924158bc0 con 0x7f19240b2390 2023-10-10T19:28:47.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.214+0000 7f1920ff9700 1 -- 172.21.15.175:0/3465952509 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 315 v315) v1 ==== 95+0+5434 (secure 0 0 0) 0x7f1924158bc0 con 0x7f19240b2390 2023-10-10T19:28:47.216 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:47.216 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":315,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":315,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:41.995092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:resolve","state_seq":11,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:47.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1904066f60 msgr2=0x7f1904069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:47.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1904066f60 0x7f1904069410 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f191c006fd0 tx=0x7f191c008040 comp rx=0 tx=0).stop 2023-10-10T19:28:47.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 msgr2=0x7f1924157340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:47.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924157340 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f190c00c5d0 tx=0x7f190c007d30 comp rx=0 tx=0).stop 2023-10-10T19:28:47.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 shutdown_connections 2023-10-10T19:28:47.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1904066f60 0x7f1904069410 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:47.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19240b2390 0x7f1924157340 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:47.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1924068bd0 0x7f19240cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:47.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 --2- 172.21.15.175:0/3465952509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f19240b1fc0 0x7f1924157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:47.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 >> 172.21.15.175:0/3465952509 conn(0x7f1924063230 msgr2=0x7f1924063600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:47.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 shutdown_connections 2023-10-10T19:28:47.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:47.216+0000 7f192ab9d700 1 -- 172.21.15.175:0/3465952509 wait complete. 2023-10-10T19:28:47.221 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 315 2023-10-10T19:28:47.230 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 315, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 315, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_7057": { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 304, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 139 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_8733": { "addr": "172.21.15.175:6834/3954290407", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3954290407, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3954290407, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8733, "incarnation": 314, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:resolve", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 132, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:28:41.995092+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 8733, "mds_2": 8049, "mds_3": 7771, "mds_4": 7057 } } } ], "standbys": [] } 2023-10-10T19:28:49.232 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:49.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.359+0000 7fc55e24e700 1 Processor -- start 2023-10-10T19:28:49.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.359+0000 7fc55e24e700 1 -- start start 2023-10-10T19:28:49.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc55e24e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc55814fd90 0x7fc558159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc55e24e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc558067fa0 con 0x7fc55808a100 2023-10-10T19:28:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc55e24e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc558156b70 con 0x7fc5580cd040 2023-10-10T19:28:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc55e24e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5580c5600 con 0x7fc55814fd90 2023-10-10T19:28:49.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc55ca4b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc55808a100 0x7fc5580c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41602/0 (socket says 172.21.15.175:41602) 2023-10-10T19:28:49.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc55ca4b700 1 -- 172.21.15.175:0/1766025381 learned_addr learned my addr 172.21.15.175:0/1766025381 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:49.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc55d24c700 1 --2- 172.21.15.175:0/1766025381 >> v2:172.21.15.191:3300/0 conn(0x7fc55814fd90 0x7fc558159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:49.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199047734 0 0) 0x7fc558067fa0 con 0x7fc55808a100 2023-10-10T19:28:49.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.360+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc544003570 con 0x7fc55808a100 2023-10-10T19:28:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 190776285 0 0) 0x7fc544003570 con 0x7fc55808a100 2023-10-10T19:28:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 >> v2:172.21.15.191:3300/0 conn(0x7fc55814fd90 msgr2=0x7fc558159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 --2- 172.21.15.175:0/1766025381 >> v2:172.21.15.191:3300/0 conn(0x7fc55814fd90 0x7fc558159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 >> v1:172.21.15.191:6789/0 conn(0x7fc5580cd040 legacy=0x7fc5580684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:49.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc558150160 con 0x7fc55808a100 2023-10-10T19:28:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fc548001470 con 0x7fc55808a100 2023-10-10T19:28:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 >> v1:172.21.15.175:6789/0 conn(0x7fc55808a100 legacy=0x7fc5580c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc544003a60 0x7fc544003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc544004810 0x7fc544004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:49.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5440051c0 0x7fc54400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580c5600 con 0x7fc544004810 2023-10-10T19:28:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54400c0d0 con 0x7fc544003a60 2023-10-10T19:28:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54400c290 con 0x7fc5440051c0 2023-10-10T19:28:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55ca4b700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc544004810 0x7fc544004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:49.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55d24c700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc544003a60 0x7fc544003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:49.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55da4d700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5440051c0 0x7fc54400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:49.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55ca4b700 1 -- 172.21.15.175:0/1766025381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5440051c0 msgr2=0x7fc54400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:49.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55ca4b700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5440051c0 0x7fc54400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55ca4b700 1 -- 172.21.15.175:0/1766025381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc544003a60 msgr2=0x7fc544003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:49.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55ca4b700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc544003a60 0x7fc544003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55da4d700 1 -- 172.21.15.175:0/1766025381 reap_dead start 2023-10-10T19:28:49.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55ca4b700 1 -- 172.21.15.175:0/1766025381 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc544003570 con 0x7fc544004810 2023-10-10T19:28:49.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc55ca4b700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc544004810 0x7fc544004c80 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fc54800c340 tx=0x7fc558156ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bad7821c4d28c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:49.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54800cc30 con 0x7fc544004810 2023-10-10T19:28:49.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54800cdb0 con 0x7fc544004810 2023-10-10T19:28:49.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.361+0000 7fc54ffff700 1 -- 172.21.15.175:0/1766025381 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc548005b70 con 0x7fc544004810 2023-10-10T19:28:49.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.362+0000 7fc55e24e700 1 -- 172.21.15.175:0/1766025381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc544004810 msgr2=0x7fc544004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:49.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.362+0000 7fc55e24e700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc544004810 0x7fc544004c80 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fc54800c340 tx=0x7fc558156ae0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.362+0000 7fc55e24e700 1 -- 172.21.15.175:0/1766025381 shutdown_connections 2023-10-10T19:28:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.362+0000 7fc55e24e700 1 --2- 172.21.15.175:0/1766025381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc544004810 0x7fc544004c80 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.362+0000 7fc55e24e700 1 -- 172.21.15.175:0/1766025381 >> 172.21.15.175:0/1766025381 conn(0x7fc558063d00 msgr2=0x7fc5580cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.362+0000 7fc55e24e700 1 -- 172.21.15.175:0/1766025381 shutdown_connections 2023-10-10T19:28:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.362+0000 7fc55e24e700 1 -- 172.21.15.175:0/1766025381 wait complete. 2023-10-10T19:28:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.363+0000 7fc55e24e700 1 Processor -- start 2023-10-10T19:28:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.363+0000 7fc55e24e700 1 -- start start 2023-10-10T19:28:49.377 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55e24e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc544004810 0x7fc5580c1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:49.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55e24e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5580cc010 0x7fc558157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:49.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55e24e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5580c22a0 0x7fc558157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:49.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55e24e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580c2710 con 0x7fc5580cc010 2023-10-10T19:28:49.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55e24e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580cc590 con 0x7fc5580c22a0 2023-10-10T19:28:49.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55e24e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580c0080 con 0x7fc544004810 2023-10-10T19:28:49.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55ca4b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5580cc010 0x7fc558157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:49.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55ca4b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5580cc010 0x7fc558157be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55782/0 (socket says 172.21.15.175:55782) 2023-10-10T19:28:49.379 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55ca4b700 1 -- 172.21.15.175:0/1558802675 learned_addr learned my addr 172.21.15.175:0/1558802675 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:49.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55d24c700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc544004810 0x7fc5580c1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:49.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.364+0000 7fc55da4d700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5580c22a0 0x7fc558157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:49.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc55ca4b700 1 -- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc544004810 msgr2=0x7fc5580c1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:49.380 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc55ca4b700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc544004810 0x7fc5580c1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc55ca4b700 1 -- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5580c22a0 msgr2=0x7fc558157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:49.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc55ca4b700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5580c22a0 0x7fc558157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc55ca4b700 1 -- 172.21.15.175:0/1558802675 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5580c2c30 con 0x7fc5580cc010 2023-10-10T19:28:49.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc55ca4b700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5580cc010 0x7fc558157be0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fc54800c930 tx=0x7fc548012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:49.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc54d7fa700 1 -- 172.21.15.175:0/1558802675 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc548001710 con 0x7fc5580cc010 2023-10-10T19:28:49.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc54d7fa700 1 -- 172.21.15.175:0/1558802675 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5480070d0 con 0x7fc5580cc010 2023-10-10T19:28:49.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5580c2f10 con 0x7fc5580cc010 2023-10-10T19:28:49.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc54d7fa700 1 -- 172.21.15.175:0/1558802675 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc548007490 con 0x7fc5580cc010 2023-10-10T19:28:49.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.365+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5580c0600 con 0x7fc5580cc010 2023-10-10T19:28:49.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.366+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc558067570 con 0x7fc5580cc010 2023-10-10T19:28:49.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.366+0000 7fc54d7fa700 1 -- 172.21.15.175:0/1558802675 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc548012400 con 0x7fc5580cc010 2023-10-10T19:28:49.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.367+0000 7fc54d7fa700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc53c066f30 0x7fc53c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:49.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.367+0000 7fc54d7fa700 1 -- 172.21.15.175:0/1558802675 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fc5480855e0 con 0x7fc5580cc010 2023-10-10T19:28:49.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.367+0000 7fc55d24c700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc53c066f30 0x7fc53c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:49.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.367+0000 7fc55d24c700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc53c066f30 0x7fc53c0693e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc558066290 tx=0x7fc55400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:49.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.371+0000 7fc54d7fa700 1 -- 172.21.15.175:0/1558802675 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc548051f30 con 0x7fc5580cc010 2023-10-10T19:28:49.595 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.594+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc558155f80 con 0x7fc5580cc010 2023-10-10T19:28:49.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.594+0000 7fc54d7fa700 1 -- 172.21.15.175:0/1558802675 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) v1 ==== 95+0+5436 (secure 0 0 0) 0x7fc548055550 con 0x7fc5580cc010 2023-10-10T19:28:49.596 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:49.597 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":316,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":316,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:48.007178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:49.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.596+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc53c066f30 msgr2=0x7fc53c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:49.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc53c066f30 0x7fc53c0693e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc558066290 tx=0x7fc55400b040 comp rx=0 tx=0).stop 2023-10-10T19:28:49.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5580cc010 msgr2=0x7fc558157be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:49.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5580cc010 0x7fc558157be0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fc54800c930 tx=0x7fc548012000 comp rx=0 tx=0).stop 2023-10-10T19:28:49.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 shutdown_connections 2023-10-10T19:28:49.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc53c066f30 0x7fc53c0693e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5580c22a0 0x7fc558157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc544004810 0x7fc5580c1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 --2- 172.21.15.175:0/1558802675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc5580cc010 0x7fc558157be0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 >> 172.21.15.175:0/1558802675 conn(0x7fc558063d00 msgr2=0x7fc55814ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:49.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 shutdown_connections 2023-10-10T19:28:49.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:49.597+0000 7fc55e24e700 1 -- 172.21.15.175:0/1558802675 wait complete. 2023-10-10T19:28:49.601 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 316 2023-10-10T19:28:49.610 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:28:51.611 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:51.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.737+0000 7f700393d700 1 Processor -- start 2023-10-10T19:28:51.738 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.737+0000 7f700393d700 1 -- start start 2023-10-10T19:28:51.741 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f700393d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6ffc0b1fc0 0x7f6ffc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f700393d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ffc0b2390 0x7f6ffc068690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f700393d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc1574b0 con 0x7f6ffc068bd0 2023-10-10T19:28:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f700393d700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc150770 con 0x7f6ffc0b2390 2023-10-10T19:28:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f700393d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0afad0 con 0x7f6ffc0b1fc0 2023-10-10T19:28:51.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7000ed8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ffc0b2390 0x7f6ffc068690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:51.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7001eda700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f6ffc068bd0 0x7f6ffc0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41604/0 (socket says 172.21.15.175:41604) 2023-10-10T19:28:51.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7001eda700 1 -- 172.21.15.175:0/3642393561 learned_addr learned my addr 172.21.15.175:0/3642393561 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:51.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7000ed8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ffc0b2390 0x7f6ffc068690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55798/0 (socket says 172.21.15.175:55798) 2023-10-10T19:28:51.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f70016d9700 1 --2- 172.21.15.175:0/3642393561 >> v2:172.21.15.191:3300/0 conn(0x7f6ffc0b1fc0 0x7f6ffc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:51.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7000ed8700 1 -- 172.21.15.175:0/3642393561 >> v2:172.21.15.191:3300/0 conn(0x7f6ffc0b1fc0 msgr2=0x7f6ffc155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:51.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7000ed8700 1 --2- 172.21.15.175:0/3642393561 >> v2:172.21.15.191:3300/0 conn(0x7f6ffc0b1fc0 0x7f6ffc155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7000ed8700 1 -- 172.21.15.175:0/3642393561 >> v1:172.21.15.175:6789/0 conn(0x7f6ffc068bd0 legacy=0x7f6ffc0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:51.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7000ed8700 1 -- 172.21.15.175:0/3642393561 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc0af4c0 con 0x7f6ffc0b2390 2023-10-10T19:28:51.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f6ff3fff700 1 -- 172.21.15.175:0/3642393561 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609521807 0 0) 0x7f6ffc1574b0 con 0x7f6ffc068bd0 2023-10-10T19:28:51.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f7000ed8700 1 --2- 172.21.15.175:0/3642393561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc068690 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0b28c0 tx=0x7f6fec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc604f335c61a184 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:51.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f6ff3fff700 1 -- 172.21.15.175:0/3642393561 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00b740 con 0x7f6ffc0b2390 2023-10-10T19:28:51.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f6ff3fff700 1 -- 172.21.15.175:0/3642393561 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec00b8a0 con 0x7f6ffc0b2390 2023-10-10T19:28:51.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.738+0000 7f6ff3fff700 1 -- 172.21.15.175:0/3642393561 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00bc40 con 0x7f6ffc0b2390 2023-10-10T19:28:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.739+0000 7f700393d700 1 -- 172.21.15.175:0/3642393561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 msgr2=0x7f6ffc068690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.739+0000 7f700393d700 1 --2- 172.21.15.175:0/3642393561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc068690 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0b28c0 tx=0x7f6fec0091a0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.739+0000 7f700393d700 1 -- 172.21.15.175:0/3642393561 shutdown_connections 2023-10-10T19:28:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.739+0000 7f700393d700 1 --2- 172.21.15.175:0/3642393561 >> v2:172.21.15.191:3300/0 conn(0x7f6ffc0b1fc0 0x7f6ffc155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.739+0000 7f700393d700 1 --2- 172.21.15.175:0/3642393561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc068690 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.739+0000 7f700393d700 1 -- 172.21.15.175:0/3642393561 >> 172.21.15.175:0/3642393561 conn(0x7f6ffc063230 msgr2=0x7f6ffc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.739+0000 7f700393d700 1 -- 172.21.15.175:0/3642393561 shutdown_connections 2023-10-10T19:28:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.739+0000 7f700393d700 1 -- 172.21.15.175:0/3642393561 wait complete. 2023-10-10T19:28:51.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.740+0000 7f700393d700 1 Processor -- start 2023-10-10T19:28:51.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.740+0000 7f700393d700 1 -- start start 2023-10-10T19:28:51.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f700393d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ffc068bd0 0x7f6ffc14da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:51.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f700393d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ffc0b1fc0 0x7f6ffc14dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:51.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f700393d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc164060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:51.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f700393d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc1690f0 con 0x7f6ffc0b2390 2023-10-10T19:28:51.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f700393d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc168200 con 0x7f6ffc0b1fc0 2023-10-10T19:28:51.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f700393d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc1684e0 con 0x7f6ffc068bd0 2023-10-10T19:28:51.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f7001eda700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc164060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f7001eda700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc164060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55806/0 (socket says 172.21.15.175:55806) 2023-10-10T19:28:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f7001eda700 1 -- 172.21.15.175:0/425412483 learned_addr learned my addr 172.21.15.175:0/425412483 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f7000ed8700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ffc0b1fc0 0x7f6ffc14dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f70016d9700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ffc068bd0 0x7f6ffc14da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f7001eda700 1 -- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ffc068bd0 msgr2=0x7f6ffc14da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:51.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.741+0000 7f7001eda700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ffc068bd0 0x7f6ffc14da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f7001eda700 1 -- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ffc0b1fc0 msgr2=0x7f6ffc14dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:51.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f7001eda700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ffc0b1fc0 0x7f6ffc14dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f7001eda700 1 -- 172.21.15.175:0/425412483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec009420 con 0x7f6ffc0b2390 2023-10-10T19:28:51.753 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f7001eda700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc164060 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400cc90 tx=0x7f6ff4001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:51.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f6ff17fa700 1 -- 172.21.15.175:0/425412483 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400a8b0 con 0x7f6ffc0b2390 2023-10-10T19:28:51.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f6ff17fa700 1 -- 172.21.15.175:0/425412483 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff400aa10 con 0x7f6ffc0b2390 2023-10-10T19:28:51.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc1687c0 con 0x7f6ffc0b2390 2023-10-10T19:28:51.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f6ff17fa700 1 -- 172.21.15.175:0/425412483 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400add0 con 0x7f6ffc0b2390 2023-10-10T19:28:51.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.742+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc168d80 con 0x7f6ffc0b2390 2023-10-10T19:28:51.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.743+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc163d80 con 0x7f6ffc0b2390 2023-10-10T19:28:51.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.743+0000 7f6ff17fa700 1 -- 172.21.15.175:0/425412483 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6ff4007300 con 0x7f6ffc0b2390 2023-10-10T19:28:51.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.744+0000 7f6ff17fa700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe0066f30 0x7f6fe00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:51.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.744+0000 7f6ff17fa700 1 -- 172.21.15.175:0/425412483 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f6ff40850d0 con 0x7f6ffc0b2390 2023-10-10T19:28:51.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.744+0000 7f70016d9700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe0066f30 0x7f6fe00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:51.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.744+0000 7f70016d9700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe0066f30 0x7f6fe00693e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8006fd0 tx=0x7f6ff8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:51.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.748+0000 7f6ff17fa700 1 -- 172.21.15.175:0/425412483 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ffc163d80 con 0x7f6ffc0b2390 2023-10-10T19:28:51.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.970+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ffc163d80 con 0x7f6ffc0b2390 2023-10-10T19:28:51.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.971+0000 7f6ff17fa700 1 -- 172.21.15.175:0/425412483 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) v1 ==== 95+0+5436 (secure 0 0 0) 0x7f6ffc163d80 con 0x7f6ffc0b2390 2023-10-10T19:28:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:51.973 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":316,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":316,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:48.007178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:51.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe0066f30 msgr2=0x7f6fe00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:51.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe0066f30 0x7f6fe00693e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8006fd0 tx=0x7f6ff8008040 comp rx=0 tx=0).stop 2023-10-10T19:28:51.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 msgr2=0x7f6ffc164060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc164060 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400cc90 tx=0x7f6ff4001a80 comp rx=0 tx=0).stop 2023-10-10T19:28:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 shutdown_connections 2023-10-10T19:28:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6fe0066f30 0x7f6fe00693e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ffc0b2390 0x7f6ffc164060 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6ffc068bd0 0x7f6ffc14da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 --2- 172.21.15.175:0/425412483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ffc0b1fc0 0x7f6ffc14dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 >> 172.21.15.175:0/425412483 conn(0x7f6ffc063230 msgr2=0x7f6ffc14fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 shutdown_connections 2023-10-10T19:28:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:51.973+0000 7f700393d700 1 -- 172.21.15.175:0/425412483 wait complete. 2023-10-10T19:28:51.977 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 316 2023-10-10T19:28:51.986 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:28:53.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:54.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.108+0000 7fc4f2394700 1 Processor -- start 2023-10-10T19:28:54.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.108+0000 7fc4f2394700 1 -- start start 2023-10-10T19:28:54.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f2394700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0b2390 0x7fc4ec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:54.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f2394700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc4ec0aff60 0x7fc4ec0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:54.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f2394700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec1574b0 con 0x7fc4ec0b1fc0 2023-10-10T19:28:54.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f2394700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec150770 con 0x7fc4ec0aff60 2023-10-10T19:28:54.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f2394700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0afad0 con 0x7fc4ec0b2390 2023-10-10T19:28:54.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4ebfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fc4ec0b1fc0 0x7fc4ec068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41618/0 (socket says 172.21.15.175:41618) 2023-10-10T19:28:54.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4ebfff700 1 -- 172.21.15.175:0/1297486212 learned_addr learned my addr 172.21.15.175:0/1297486212 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:54.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f0931700 1 --2- 172.21.15.175:0/1297486212 >> v2:172.21.15.175:3300/0 conn(0x7fc4ec0aff60 0x7fc4ec0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:54.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4eb7fe700 1 --2- 172.21.15.175:0/1297486212 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0b2390 0x7fc4ec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:54.114 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4eaffd700 1 -- 172.21.15.175:0/1297486212 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924912839 0 0) 0x7fc4ec1574b0 con 0x7fc4ec0b1fc0 2023-10-10T19:28:54.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4eaffd700 1 -- 172.21.15.175:0/1297486212 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d4003570 con 0x7fc4ec0b1fc0 2023-10-10T19:28:54.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f0931700 1 -- 172.21.15.175:0/1297486212 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0b2390 msgr2=0x7fc4ec155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:54.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f0931700 1 --2- 172.21.15.175:0/1297486212 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0b2390 0x7fc4ec155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f0931700 1 -- 172.21.15.175:0/1297486212 >> v1:172.21.15.175:6789/0 conn(0x7fc4ec0b1fc0 legacy=0x7fc4ec068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f0931700 1 -- 172.21.15.175:0/1297486212 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0af4c0 con 0x7fc4ec0aff60 2023-10-10T19:28:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4f0931700 1 --2- 172.21.15.175:0/1297486212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0aff60 0x7fc4ec0b03e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009e90 tx=0x7fc4e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81bd8d2b8041428 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4eaffd700 1 -- 172.21.15.175:0/1297486212 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400e070 con 0x7fc4ec0aff60 2023-10-10T19:28:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4eaffd700 1 -- 172.21.15.175:0/1297486212 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e400bbd0 con 0x7fc4ec0aff60 2023-10-10T19:28:54.116 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.109+0000 7fc4eaffd700 1 -- 172.21.15.175:0/1297486212 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4012430 con 0x7fc4ec0aff60 2023-10-10T19:28:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.110+0000 7fc4f2394700 1 -- 172.21.15.175:0/1297486212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0aff60 msgr2=0x7fc4ec0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.110+0000 7fc4f2394700 1 --2- 172.21.15.175:0/1297486212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0aff60 0x7fc4ec0b03e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009e90 tx=0x7fc4e4009780 comp rx=0 tx=0).stop 2023-10-10T19:28:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.110+0000 7fc4f2394700 1 -- 172.21.15.175:0/1297486212 shutdown_connections 2023-10-10T19:28:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.110+0000 7fc4f2394700 1 --2- 172.21.15.175:0/1297486212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0aff60 0x7fc4ec0b03e0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.117 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.110+0000 7fc4f2394700 1 --2- 172.21.15.175:0/1297486212 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0b2390 0x7fc4ec155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.110+0000 7fc4f2394700 1 -- 172.21.15.175:0/1297486212 >> 172.21.15.175:0/1297486212 conn(0x7fc4ec063230 msgr2=0x7fc4ec063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:54.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.110+0000 7fc4f2394700 1 -- 172.21.15.175:0/1297486212 shutdown_connections 2023-10-10T19:28:54.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.110+0000 7fc4f2394700 1 -- 172.21.15.175:0/1297486212 wait complete. 2023-10-10T19:28:54.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.111+0000 7fc4f2394700 1 Processor -- start 2023-10-10T19:28:54.119 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.111+0000 7fc4f2394700 1 -- start start 2023-10-10T19:28:54.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f2394700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0aff60 0x7fc4ec0c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:54.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f2394700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ec0b1fc0 0x7fc4ec14f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:54.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f2394700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0b2390 0x7fc4ec0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:54.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f2394700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0cd410 con 0x7fc4ec0b2390 2023-10-10T19:28:54.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f2394700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec158ea0 con 0x7fc4ec0b1fc0 2023-10-10T19:28:54.122 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f2394700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0ccc20 con 0x7fc4ec0aff60 2023-10-10T19:28:54.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0b2390 0x7fc4ec0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:54.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0b2390 0x7fc4ec0cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55816/0 (socket says 172.21.15.175:55816) 2023-10-10T19:28:54.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 -- 172.21.15.175:0/4188054069 learned_addr learned my addr 172.21.15.175:0/4188054069 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:54.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4ebfff700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0aff60 0x7fc4ec0c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:54.123 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4eb7fe700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ec0b1fc0 0x7fc4ec14f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:54.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 -- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0aff60 msgr2=0x7fc4ec0c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:54.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0aff60 0x7fc4ec0c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 -- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ec0b1fc0 msgr2=0x7fc4ec14f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:54.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ec0b1fc0 0x7fc4ec14f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.124 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 -- 172.21.15.175:0/4188054069 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e4009a20 con 0x7fc4ec0b2390 2023-10-10T19:28:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f0931700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0b2390 0x7fc4ec0cc060 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec06fd90 tx=0x7fc4e4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4d3fff700 1 -- 172.21.15.175:0/4188054069 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4012dd0 con 0x7fc4ec0b2390 2023-10-10T19:28:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4d3fff700 1 -- 172.21.15.175:0/4188054069 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e4021070 con 0x7fc4ec0b2390 2023-10-10T19:28:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ec166710 con 0x7fc4ec0b2390 2023-10-10T19:28:54.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4d3fff700 1 -- 172.21.15.175:0/4188054069 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4026660 con 0x7fc4ec0b2390 2023-10-10T19:28:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.112+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec0cd130 con 0x7fc4ec0b2390 2023-10-10T19:28:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.113+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ec071de0 con 0x7fc4ec0b2390 2023-10-10T19:28:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.114+0000 7fc4d3fff700 1 -- 172.21.15.175:0/4188054069 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc4e4023030 con 0x7fc4ec0b2390 2023-10-10T19:28:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.114+0000 7fc4d3fff700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4d8066f30 0x7fc4d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:54.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.114+0000 7fc4d3fff700 1 -- 172.21.15.175:0/4188054069 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fc4e4087110 con 0x7fc4ec0b2390 2023-10-10T19:28:54.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.114+0000 7fc4ebfff700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4d8066f30 0x7fc4d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:54.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.115+0000 7fc4ebfff700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4d8066f30 0x7fc4d80693e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0058d0 tx=0x7fc4dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:54.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.118+0000 7fc4d3fff700 1 -- 172.21.15.175:0/4188054069 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4e4053b00 con 0x7fc4ec0b2390 2023-10-10T19:28:54.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.345+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4ec1533e0 con 0x7fc4ec0b2390 2023-10-10T19:28:54.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.346+0000 7fc4d3fff700 1 -- 172.21.15.175:0/4188054069 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 317 v317) v1 ==== 95+0+5433 (secure 0 0 0) 0x7fc4e4057120 con 0x7fc4ec0b2390 2023-10-10T19:28:54.347 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:54.348 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":317,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:53.023359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4d8066f30 msgr2=0x7fc4d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4d8066f30 0x7fc4d80693e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0058d0 tx=0x7fc4dc009040 comp rx=0 tx=0).stop 2023-10-10T19:28:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0b2390 msgr2=0x7fc4ec0cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:54.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0b2390 0x7fc4ec0cc060 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec06fd90 tx=0x7fc4e4019000 comp rx=0 tx=0).stop 2023-10-10T19:28:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 shutdown_connections 2023-10-10T19:28:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fc4d8066f30 0x7fc4d80693e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4ec0b2390 0x7fc4ec0cc060 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4ec0aff60 0x7fc4ec0c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 --2- 172.21.15.175:0/4188054069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ec0b1fc0 0x7fc4ec14f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:54.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 >> 172.21.15.175:0/4188054069 conn(0x7fc4ec063230 msgr2=0x7fc4ec155790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:54.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 shutdown_connections 2023-10-10T19:28:54.352 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:54.348+0000 7fc4f2394700 1 -- 172.21.15.175:0/4188054069 wait complete. 2023-10-10T19:28:54.353 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 317 2023-10-10T19:28:54.361 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:28:56.363 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:56.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.490+0000 7f5791d08700 1 Processor -- start 2023-10-10T19:28:56.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.490+0000 7f5791d08700 1 -- start start 2023-10-10T19:28:56.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f5791d08700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f578c155fb0 0x7f578c159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:56.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f5791d08700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578c1574b0 con 0x7f578c0b2390 2023-10-10T19:28:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f5791d08700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578c0afad0 con 0x7f578c0b1fc0 2023-10-10T19:28:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f5791d08700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578c150770 con 0x7f578c155fb0 2023-10-10T19:28:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578affd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f578c0b2390 0x7f578c0aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51872/0 (socket says 172.21.15.175:51872) 2023-10-10T19:28:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578affd700 1 -- 172.21.15.175:0/1775066971 learned_addr learned my addr 172.21.15.175:0/1775066971 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:56.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578bfff700 1 --2- 172.21.15.175:0/1775066971 >> v2:172.21.15.191:3300/0 conn(0x7f578c155fb0 0x7f578c159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528406975 0 0) 0x7f578c1574b0 con 0x7f578c0b2390 2023-10-10T19:28:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5774003570 con 0x7f578c0b2390 2023-10-10T19:28:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952161548 0 0) 0x7f5774003570 con 0x7f578c0b2390 2023-10-10T19:28:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 >> v2:172.21.15.191:3300/0 conn(0x7f578c155fb0 msgr2=0x7f578c159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:56.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 --2- 172.21.15.175:0/1775066971 >> v2:172.21.15.191:3300/0 conn(0x7f578c155fb0 0x7f578c159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 >> v1:172.21.15.191:6789/0 conn(0x7f578c0b1fc0 legacy=0x7f578c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c0af4c0 con 0x7f578c0b2390 2023-10-10T19:28:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f5780001470 con 0x7f578c0b2390 2023-10-10T19:28:56.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 >> v1:172.21.15.175:6789/0 conn(0x7f578c0b2390 legacy=0x7f578c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5774003a60 0x7f5774003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774004810 0x7f5774004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57740051c0 0x7f577400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c150770 con 0x7f5774003a60 2023-10-10T19:28:56.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577400c0d0 con 0x7f5774004810 2023-10-10T19:28:56.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577400c290 con 0x7f57740051c0 2023-10-10T19:28:56.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.491+0000 7f578b7fe700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5774003a60 0x7f5774003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:56.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578affd700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774004810 0x7f5774004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:56.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578bfff700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57740051c0 0x7f577400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:56.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578b7fe700 1 -- 172.21.15.175:0/1775066971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57740051c0 msgr2=0x7f577400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:56.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578b7fe700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57740051c0 0x7f577400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578b7fe700 1 -- 172.21.15.175:0/1775066971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774004810 msgr2=0x7f5774004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:56.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578b7fe700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774004810 0x7f5774004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578bfff700 1 -- 172.21.15.175:0/1775066971 reap_dead start 2023-10-10T19:28:56.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578b7fe700 1 -- 172.21.15.175:0/1775066971 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5774003570 con 0x7f5774003a60 2023-10-10T19:28:56.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578b7fe700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5774003a60 0x7f5774003e70 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f577c0018b0 tx=0x7f577c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8910636101c381e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:56.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c012420 con 0x7f5774003a60 2023-10-10T19:28:56.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c0125a0 con 0x7f5774003a60 2023-10-10T19:28:56.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.492+0000 7f578a7fc700 1 -- 172.21.15.175:0/1775066971 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c012960 con 0x7f5774003a60 2023-10-10T19:28:56.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.493+0000 7f5791d08700 1 -- 172.21.15.175:0/1775066971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5774003a60 msgr2=0x7f5774003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.493+0000 7f5791d08700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5774003a60 0x7f5774003e70 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f577c0018b0 tx=0x7f577c007d30 comp rx=0 tx=0).stop 2023-10-10T19:28:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.493+0000 7f5791d08700 1 -- 172.21.15.175:0/1775066971 shutdown_connections 2023-10-10T19:28:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.493+0000 7f5791d08700 1 --2- 172.21.15.175:0/1775066971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5774003a60 0x7f5774003e70 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.493+0000 7f5791d08700 1 -- 172.21.15.175:0/1775066971 >> 172.21.15.175:0/1775066971 conn(0x7f578c063230 msgr2=0x7f578c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:56.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.493+0000 7f5791d08700 1 -- 172.21.15.175:0/1775066971 shutdown_connections 2023-10-10T19:28:56.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.493+0000 7f5791d08700 1 -- 172.21.15.175:0/1775066971 wait complete. 2023-10-10T19:28:56.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.494+0000 7f5791d08700 1 Processor -- start 2023-10-10T19:28:56.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.494+0000 7f5791d08700 1 -- start start 2023-10-10T19:28:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f5791d08700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774003a60 0x7f578c15a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f5791d08700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f578c14ed40 0x7f578c14f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f5791d08700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f578c152d30 0x7f578c1531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:56.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f5791d08700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c150d50 con 0x7f578c14ed40 2023-10-10T19:28:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f5791d08700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c14f6d0 con 0x7f5774003a60 2023-10-10T19:28:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f5791d08700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c168f00 con 0x7f578c152d30 2023-10-10T19:28:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f578c14ed40 0x7f578c14f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f578c14ed40 0x7f578c14f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35958/0 (socket says 172.21.15.175:35958) 2023-10-10T19:28:56.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 -- 172.21.15.175:0/2590159205 learned_addr learned my addr 172.21.15.175:0/2590159205 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:56.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578b7fe700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774003a60 0x7f578c15a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:56.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578bfff700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f578c152d30 0x7f578c1531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:56.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 -- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f578c152d30 msgr2=0x7f578c1531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:56.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f578c152d30 0x7f578c1531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 -- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774003a60 msgr2=0x7f578c15a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:56.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774003a60 0x7f578c15a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 -- 172.21.15.175:0/2590159205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c1528a0 con 0x7f578c14ed40 2023-10-10T19:28:56.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578b7fe700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774003a60 0x7f578c15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:28:56.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f578affd700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f578c14ed40 0x7f578c14f190 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f5780007340 tx=0x7f578c0af650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:56.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f57737fe700 1 -- 172.21.15.175:0/2590159205 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780018030 con 0x7f578c14ed40 2023-10-10T19:28:56.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f57737fe700 1 -- 172.21.15.175:0/2590159205 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578000f8d0 con 0x7f578c14ed40 2023-10-10T19:28:56.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c1536e0 con 0x7f578c14ed40 2023-10-10T19:28:56.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f57737fe700 1 -- 172.21.15.175:0/2590159205 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578000fc70 con 0x7f578c14ed40 2023-10-10T19:28:56.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.495+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c151f70 con 0x7f578c14ed40 2023-10-10T19:28:56.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.497+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578c152a30 con 0x7f578c14ed40 2023-10-10T19:28:56.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.497+0000 7f57737fe700 1 -- 172.21.15.175:0/2590159205 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f578001b070 con 0x7f578c14ed40 2023-10-10T19:28:56.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.497+0000 7f57737fe700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5778066f60 0x7f5778069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:56.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.497+0000 7f57737fe700 1 -- 172.21.15.175:0/2590159205 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f5780019070 con 0x7f578c14ed40 2023-10-10T19:28:56.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.497+0000 7f578b7fe700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5778066f60 0x7f5778069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:56.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.498+0000 7f578b7fe700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5778066f60 0x7f5778069410 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f577c001690 tx=0x7f577c015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:56.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.501+0000 7f57737fe700 1 -- 172.21.15.175:0/2590159205 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5780051560 con 0x7f578c14ed40 2023-10-10T19:28:56.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.724+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f578c069060 con 0x7f578c14ed40 2023-10-10T19:28:56.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.725+0000 7f57737fe700 1 -- 172.21.15.175:0/2590159205 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 317 v317) v1 ==== 95+0+5433 (secure 0 0 0) 0x7f5780054b80 con 0x7f578c14ed40 2023-10-10T19:28:56.727 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:56.727 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":317,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:53.023359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:56.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5778066f60 msgr2=0x7f5778069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:56.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5778066f60 0x7f5778069410 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f577c001690 tx=0x7f577c015000 comp rx=0 tx=0).stop 2023-10-10T19:28:56.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f578c14ed40 msgr2=0x7f578c14f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:56.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f578c14ed40 0x7f578c14f190 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f5780007340 tx=0x7f578c0af650 comp rx=0 tx=0).stop 2023-10-10T19:28:56.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 shutdown_connections 2023-10-10T19:28:56.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5778066f60 0x7f5778069410 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f578c152d30 0x7f578c1531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5774003a60 0x7f578c15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 --2- 172.21.15.175:0/2590159205 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f578c14ed40 0x7f578c14f190 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 >> 172.21.15.175:0/2590159205 conn(0x7f578c063230 msgr2=0x7f578c0afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 shutdown_connections 2023-10-10T19:28:56.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:56.727+0000 7f5791d08700 1 -- 172.21.15.175:0/2590159205 wait complete. 2023-10-10T19:28:56.732 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 317 2023-10-10T19:28:56.741 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:28:57.536 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:28:57.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:28:58.742 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:28:58.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.866+0000 7ff5726e1700 1 Processor -- start 2023-10-10T19:28:58.867 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.866+0000 7ff5726e1700 1 -- start start 2023-10-10T19:28:58.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.866+0000 7ff5726e1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff56c0b2390 0x7ff56c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:58.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.866+0000 7ff5726e1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0aff60 0x7ff56c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:58.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.866+0000 7ff5726e1700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c1574b0 con 0x7ff56c0b1fc0 2023-10-10T19:28:58.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff5726e1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c150770 con 0x7ff56c0b2390 2023-10-10T19:28:58.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff5726e1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0afad0 con 0x7ff56c0aff60 2023-10-10T19:28:58.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff56c0b2390 0x7ff56c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:58.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff56c0b1fc0 0x7ff56c068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51874/0 (socket says 172.21.15.175:51874) 2023-10-10T19:28:58.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56bfff700 1 -- 172.21.15.175:0/4213007367 learned_addr learned my addr 172.21.15.175:0/4213007367 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:28:58.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56b7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff56c0b2390 0x7ff56c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35962/0 (socket says 172.21.15.175:35962) 2023-10-10T19:28:58.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56b7fe700 1 -- 172.21.15.175:0/4213007367 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0aff60 msgr2=0x7ff56c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:28:58.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff570c7e700 1 --2- 172.21.15.175:0/4213007367 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0aff60 0x7ff56c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:58.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56affd700 1 -- 172.21.15.175:0/4213007367 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829941202 0 0) 0x7ff56c1574b0 con 0x7ff56c0b1fc0 2023-10-10T19:28:58.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56b7fe700 1 --2- 172.21.15.175:0/4213007367 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0aff60 0x7ff56c0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:58.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56b7fe700 1 -- 172.21.15.175:0/4213007367 >> v1:172.21.15.175:6789/0 conn(0x7ff56c0b1fc0 legacy=0x7ff56c068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:58.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56b7fe700 1 -- 172.21.15.175:0/4213007367 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c0af4c0 con 0x7ff56c0b2390 2023-10-10T19:28:58.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff570c7e700 1 --2- 172.21.15.175:0/4213007367 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0aff60 0x7ff56c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:28:58.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56b7fe700 1 --2- 172.21.15.175:0/4213007367 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0b2390 0x7ff56c155fb0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0098e0 tx=0x7ff55c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51a5a67b5e4d58e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:58.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56affd700 1 -- 172.21.15.175:0/4213007367 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c009e80 con 0x7ff56c0b2390 2023-10-10T19:28:58.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56affd700 1 -- 172.21.15.175:0/4213007367 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55c00b8f0 con 0x7ff56c0b2390 2023-10-10T19:28:58.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.867+0000 7ff56affd700 1 -- 172.21.15.175:0/4213007367 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c00bc70 con 0x7ff56c0b2390 2023-10-10T19:28:58.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.868+0000 7ff5726e1700 1 -- 172.21.15.175:0/4213007367 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0b2390 msgr2=0x7ff56c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:58.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.868+0000 7ff5726e1700 1 --2- 172.21.15.175:0/4213007367 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0b2390 0x7ff56c155fb0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0098e0 tx=0x7ff55c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:28:58.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.868+0000 7ff5726e1700 1 -- 172.21.15.175:0/4213007367 shutdown_connections 2023-10-10T19:28:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.868+0000 7ff5726e1700 1 --2- 172.21.15.175:0/4213007367 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0aff60 0x7ff56c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.868+0000 7ff5726e1700 1 --2- 172.21.15.175:0/4213007367 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0b2390 0x7ff56c155fb0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.868+0000 7ff5726e1700 1 -- 172.21.15.175:0/4213007367 >> 172.21.15.175:0/4213007367 conn(0x7ff56c063230 msgr2=0x7ff56c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.868+0000 7ff5726e1700 1 -- 172.21.15.175:0/4213007367 shutdown_connections 2023-10-10T19:28:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.868+0000 7ff5726e1700 1 -- 172.21.15.175:0/4213007367 wait complete. 2023-10-10T19:28:58.875 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.869+0000 7ff5726e1700 1 Processor -- start 2023-10-10T19:28:58.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.869+0000 7ff5726e1700 1 -- start start 2023-10-10T19:28:58.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff5726e1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0aff60 0x7ff56c1658a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:58.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff5726e1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0b1fc0 0x7ff56c0c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:58.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff5726e1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff56c0b2390 0x7ff56c14fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:58.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff5726e1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0c4000 con 0x7ff56c0aff60 2023-10-10T19:28:58.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff5726e1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0cbf30 con 0x7ff56c0b2390 2023-10-10T19:28:58.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff5726e1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c166550 con 0x7ff56c0b1fc0 2023-10-10T19:28:58.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0aff60 0x7ff56c1658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0aff60 0x7ff56c1658a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35976/0 (socket says 172.21.15.175:35976) 2023-10-10T19:28:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 -- 172.21.15.175:0/3999470201 learned_addr learned my addr 172.21.15.175:0/3999470201 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:28:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56b7fe700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0b1fc0 0x7ff56c0c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff570c7e700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff56c0b2390 0x7ff56c14fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:58.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 -- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0b1fc0 msgr2=0x7ff56c0c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:58.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0b1fc0 0x7ff56c0c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:58.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 -- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff56c0b2390 msgr2=0x7ff56c14fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:58.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff56c0b2390 0x7ff56c14fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:58.881 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 -- 172.21.15.175:0/3999470201 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c009420 con 0x7ff56c0aff60 2023-10-10T19:28:58.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.870+0000 7ff56bfff700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0aff60 0x7ff56c1658a0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7ff56000c810 tx=0x7ff560007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:58.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.871+0000 7ff553fff700 1 -- 172.21.15.175:0/3999470201 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560005b30 con 0x7ff56c0aff60 2023-10-10T19:28:58.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.871+0000 7ff553fff700 1 -- 172.21.15.175:0/3999470201 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff560005c90 con 0x7ff56c0aff60 2023-10-10T19:28:58.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.871+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c0cca10 con 0x7ff56c0aff60 2023-10-10T19:28:58.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.871+0000 7ff553fff700 1 -- 172.21.15.175:0/3999470201 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560007300 con 0x7ff56c0aff60 2023-10-10T19:28:58.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.871+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c0ccea0 con 0x7ff56c0aff60 2023-10-10T19:28:58.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.872+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c0cccf0 con 0x7ff56c0aff60 2023-10-10T19:28:58.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.872+0000 7ff553fff700 1 -- 172.21.15.175:0/3999470201 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff56001b400 con 0x7ff56c0aff60 2023-10-10T19:28:58.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.872+0000 7ff553fff700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff558066f30 0x7ff5580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:28:58.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.872+0000 7ff553fff700 1 -- 172.21.15.175:0/3999470201 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7ff560084930 con 0x7ff56c0aff60 2023-10-10T19:28:58.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.872+0000 7ff56b7fe700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff558066f30 0x7ff5580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:28:58.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.873+0000 7ff56b7fe700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff558066f30 0x7ff5580693e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0091e0 tx=0x7ff55c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:28:58.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:58.876+0000 7ff553fff700 1 -- 172.21.15.175:0/3999470201 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5600513a0 con 0x7ff56c0aff60 2023-10-10T19:28:59.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.106+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff56c1662d0 con 0x7ff56c0aff60 2023-10-10T19:28:59.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.107+0000 7ff553fff700 1 -- 172.21.15.175:0/3999470201 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+5439 (secure 0 0 0) 0x7ff5600549c0 con 0x7ff56c0aff60 2023-10-10T19:28:59.109 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:28:59.109 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:59.041204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:28:59.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff558066f30 msgr2=0x7ff5580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:59.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff558066f30 0x7ff5580693e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0091e0 tx=0x7ff55c017000 comp rx=0 tx=0).stop 2023-10-10T19:28:59.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0aff60 msgr2=0x7ff56c1658a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:28:59.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0aff60 0x7ff56c1658a0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7ff56000c810 tx=0x7ff560007df0 comp rx=0 tx=0).stop 2023-10-10T19:28:59.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 shutdown_connections 2023-10-10T19:28:59.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7ff558066f30 0x7ff5580693e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:59.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff56c0b2390 0x7ff56c14fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:59.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff56c0aff60 0x7ff56c1658a0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:59.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 --2- 172.21.15.175:0/3999470201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff56c0b1fc0 0x7ff56c0c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:28:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 >> 172.21.15.175:0/3999470201 conn(0x7ff56c063230 msgr2=0x7ff56c15a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:28:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 shutdown_connections 2023-10-10T19:28:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:28:59.109+0000 7ff5726e1700 1 -- 172.21.15.175:0/3999470201 wait complete. 2023-10-10T19:28:59.113 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 318 2023-10-10T19:28:59.123 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 318, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-10T18:45:10.774833+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 318, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_7057": { "addr": "172.21.15.191:6836/437857850", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6835", "nonce": 437857850, "type": "v2" }, { "addr": "172.21.15.191:6836", "nonce": 437857850, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7057, "incarnation": 304, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 139 }, "gid_7771": { "addr": "172.21.15.191:6839/3029933182", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6838", "nonce": 3029933182, "type": "v2" }, { "addr": "172.21.15.191:6839", "nonce": 3029933182, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 7771, "incarnation": 297, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_8049": { "addr": "172.21.15.175:6836/2026792975", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6833", "nonce": 2026792975, "type": "v2" }, { "addr": "172.21.15.175:6836", "nonce": 2026792975, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8049, "incarnation": 285, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_8202": { "addr": "172.21.15.175:6837/74990995", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6835", "nonce": 74990995, "type": "v2" }, { "addr": "172.21.15.175:6837", "nonce": 74990995, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8202, "incarnation": 291, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_8733": { "addr": "172.21.15.175:6834/3954290407", "addrs": { "addrvec": [ { "addr": "172.21.15.175:6832", "nonce": 3954290407, "type": "v2" }, { "addr": "172.21.15.175:6834", "nonce": 3954290407, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 8733, "incarnation": 314, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:clientreplay", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 132, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 2, "modified": "2023-10-10T19:28:59.041204+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8202, "mds_1": 8733, "mds_2": 8049, "mds_3": 7771, "mds_4": 7057 } } } ], "standbys": [] } 2023-10-10T19:29:01.124 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:29:01.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.251+0000 7f29b61b4700 1 Processor -- start 2023-10-10T19:29:01.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.251+0000 7f29b61b4700 1 -- start start 2023-10-10T19:29:01.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29b61b4700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f29b00b2390 0x7f29b0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:01.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29b61b4700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b01574b0 con 0x7f29b00aff60 2023-10-10T19:29:01.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29b61b4700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b00afad0 con 0x7f29b00b1fc0 2023-10-10T19:29:01.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29b61b4700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b0150770 con 0x7f29b00b2390 2023-10-10T19:29:01.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29aeffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f29b00b2390 0x7f29b0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:01.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29affff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f29b00aff60 0x7f29b0159920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51882/0 (socket says 172.21.15.175:51882) 2023-10-10T19:29:01.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29affff700 1 -- 172.21.15.175:0/4294913857 learned_addr learned my addr 172.21.15.175:0/4294913857 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:01.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29aeffd700 1 -- 172.21.15.175:0/4294913857 >> v1:172.21.15.191:6789/0 conn(0x7f29b00b1fc0 legacy=0x7f29b0068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:01.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29aeffd700 1 -- 172.21.15.175:0/4294913857 >> v1:172.21.15.175:6789/0 conn(0x7f29b00aff60 legacy=0x7f29b0159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:01.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29aeffd700 1 -- 172.21.15.175:0/4294913857 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b00af4c0 con 0x7f29b00b2390 2023-10-10T19:29:01.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29ae7fc700 1 -- 172.21.15.175:0/4294913857 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425082811 0 0) 0x7f29b01574b0 con 0x7f29b00aff60 2023-10-10T19:29:01.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29aeffd700 1 --2- 172.21.15.175:0/4294913857 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b0155fb0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f29a40098e0 tx=0x7f29a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fe6bfe76d024f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:01.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29ae7fc700 1 -- 172.21.15.175:0/4294913857 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4009e80 con 0x7f29b00b2390 2023-10-10T19:29:01.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29ae7fc700 1 -- 172.21.15.175:0/4294913857 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a400b8f0 con 0x7f29b00b2390 2023-10-10T19:29:01.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.252+0000 7f29ae7fc700 1 -- 172.21.15.175:0/4294913857 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a400bc70 con 0x7f29b00b2390 2023-10-10T19:29:01.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.253+0000 7f29b61b4700 1 -- 172.21.15.175:0/4294913857 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 msgr2=0x7f29b0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:01.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.253+0000 7f29b61b4700 1 --2- 172.21.15.175:0/4294913857 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b0155fb0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f29a40098e0 tx=0x7f29a40091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.253+0000 7f29b61b4700 1 -- 172.21.15.175:0/4294913857 shutdown_connections 2023-10-10T19:29:01.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.253+0000 7f29b61b4700 1 --2- 172.21.15.175:0/4294913857 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b0155fb0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.253+0000 7f29b61b4700 1 -- 172.21.15.175:0/4294913857 >> 172.21.15.175:0/4294913857 conn(0x7f29b0063230 msgr2=0x7f29b0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:01.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.253+0000 7f29b61b4700 1 -- 172.21.15.175:0/4294913857 shutdown_connections 2023-10-10T19:29:01.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.253+0000 7f29b61b4700 1 -- 172.21.15.175:0/4294913857 wait complete. 2023-10-10T19:29:01.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.254+0000 7f29b61b4700 1 Processor -- start 2023-10-10T19:29:01.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.254+0000 7f29b61b4700 1 -- start start 2023-10-10T19:29:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29b61b4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b00aff60 0x7f29b0169750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29b61b4700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b00b1fc0 0x7f29b0169c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29b61b4700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b00c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:01.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29b61b4700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b00cc110 con 0x7f29b00b2390 2023-10-10T19:29:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29b61b4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b016a2b0 con 0x7f29b00aff60 2023-10-10T19:29:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29b61b4700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0159a60 con 0x7f29b00b1fc0 2023-10-10T19:29:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29affff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b00c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29affff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b00c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:35986/0 (socket says 172.21.15.175:35986) 2023-10-10T19:29:01.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29affff700 1 -- 172.21.15.175:0/3497633923 learned_addr learned my addr 172.21.15.175:0/3497633923 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29af7fe700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b00aff60 0x7f29b0169750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29aeffd700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b00b1fc0 0x7f29b0169c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29affff700 1 -- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b00b1fc0 msgr2=0x7f29b0169c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29affff700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b00b1fc0 0x7f29b0169c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29affff700 1 -- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b00aff60 msgr2=0x7f29b0169750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29affff700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b00aff60 0x7f29b0169750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.255+0000 7f29affff700 1 -- 172.21.15.175:0/3497633923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a4009420 con 0x7f29b00b2390 2023-10-10T19:29:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.256+0000 7f29affff700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b00c5050 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f29a000cc90 tx=0x7f29a0001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:01.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.256+0000 7f29977fe700 1 -- 172.21.15.175:0/3497633923 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a000a8b0 con 0x7f29b00b2390 2023-10-10T19:29:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.256+0000 7f29977fe700 1 -- 172.21.15.175:0/3497633923 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a000aa10 con 0x7f29b00b2390 2023-10-10T19:29:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.256+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b0159d40 con 0x7f29b00b2390 2023-10-10T19:29:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.256+0000 7f29977fe700 1 -- 172.21.15.175:0/3497633923 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a000add0 con 0x7f29b00b2390 2023-10-10T19:29:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.256+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b00c4b30 con 0x7f29b00b2390 2023-10-10T19:29:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.257+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b00c4d20 con 0x7f29b00b2390 2023-10-10T19:29:01.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.257+0000 7f29977fe700 1 -- 172.21.15.175:0/3497633923 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f29a0007300 con 0x7f29b00b2390 2023-10-10T19:29:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.258+0000 7f29977fe700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f299c066f60 0x7f299c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.258+0000 7f29977fe700 1 -- 172.21.15.175:0/3497633923 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f29a0085090 con 0x7f29b00b2390 2023-10-10T19:29:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.258+0000 7f29af7fe700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f299c066f60 0x7f299c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:01.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.258+0000 7f29af7fe700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f299c066f60 0x7f299c069410 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f29a8003a80 tx=0x7f29a8007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:01.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.262+0000 7f29977fe700 1 -- 172.21.15.175:0/3497633923 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f29a0051b00 con 0x7f29b00b2390 2023-10-10T19:29:01.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.488+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29b0067570 con 0x7f29b00b2390 2023-10-10T19:29:01.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.488+0000 7f29977fe700 1 -- 172.21.15.175:0/3497633923 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+5439 (secure 0 0 0) 0x7f29a0055120 con 0x7f29b00b2390 2023-10-10T19:29:01.490 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:01.490 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:59.041204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:29:01.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f299c066f60 msgr2=0x7f299c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:01.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f299c066f60 0x7f299c069410 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f29a8003a80 tx=0x7f29a8007ec0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 msgr2=0x7f29b00c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:01.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b00c5050 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f29a000cc90 tx=0x7f29a0001a80 comp rx=0 tx=0).stop 2023-10-10T19:29:01.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 shutdown_connections 2023-10-10T19:29:01.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f299c066f60 0x7f299c069410 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f29b00b2390 0x7f29b00c5050 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b00aff60 0x7f29b0169750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 --2- 172.21.15.175:0/3497633923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b00b1fc0 0x7f29b0169c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:01.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 >> 172.21.15.175:0/3497633923 conn(0x7f29b0063230 msgr2=0x7f29b014fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:01.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 shutdown_connections 2023-10-10T19:29:01.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:01.490+0000 7f29b61b4700 1 -- 172.21.15.175:0/3497633923 wait complete. 2023-10-10T19:29:01.495 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 318 2023-10-10T19:29:01.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:29:03.504 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:29:03.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.631+0000 7fcdd3c06700 1 Processor -- start 2023-10-10T19:29:03.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.631+0000 7fcdd3c06700 1 -- start start 2023-10-10T19:29:03.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.631+0000 7fcdd3c06700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcdcc155fb0 0x7fcdcc159920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:03.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.631+0000 7fcdd3c06700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdcc1574b0 con 0x7fcdcc0b1fc0 2023-10-10T19:29:03.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.631+0000 7fcdd3c06700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdcc0afad0 con 0x7fcdcc0b2390 2023-10-10T19:29:03.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.631+0000 7fcdd3c06700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdcc150770 con 0x7fcdcc155fb0 2023-10-10T19:29:03.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd19a2700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fcdcc0b1fc0 0x7fcdcc068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:51894/0 (socket says 172.21.15.175:51894) 2023-10-10T19:29:03.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd19a2700 1 -- 172.21.15.175:0/717524837 learned_addr learned my addr 172.21.15.175:0/717524837 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:03.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd21a3700 1 --2- 172.21.15.175:0/717524837 >> v2:172.21.15.191:3300/0 conn(0x7fcdcc155fb0 0x7fcdcc159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:03.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93148488 0 0) 0x7fcdcc1574b0 con 0x7fcdcc0b1fc0 2023-10-10T19:29:03.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb8003570 con 0x7fcdcc0b1fc0 2023-10-10T19:29:03.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4037564311 0 0) 0x7fcdb8003570 con 0x7fcdcc0b1fc0 2023-10-10T19:29:03.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 >> v2:172.21.15.191:3300/0 conn(0x7fcdcc155fb0 msgr2=0x7fcdcc159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:03.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 --2- 172.21.15.175:0/717524837 >> v2:172.21.15.191:3300/0 conn(0x7fcdcc155fb0 0x7fcdcc159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 >> v1:172.21.15.191:6789/0 conn(0x7fcdcc0b2390 legacy=0x7fcdcc0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:03.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc0af4c0 con 0x7fcdcc0b1fc0 2023-10-10T19:29:03.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fcdc80017f0 con 0x7fcdcc0b1fc0 2023-10-10T19:29:03.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 >> v1:172.21.15.175:6789/0 conn(0x7fcdcc0b1fc0 legacy=0x7fcdcc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:03.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8003a60 0x7fcdb8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:03.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb8004810 0x7fcdb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:03.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb80051c0 0x7fcdb800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:03.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc150770 con 0x7fcdb8004810 2023-10-10T19:29:03.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800c0d0 con 0x7fcdb8003a60 2023-10-10T19:29:03.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800c290 con 0x7fcdb80051c0 2023-10-10T19:29:03.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd11a1700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb8004810 0x7fcdb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:03.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd19a2700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8003a60 0x7fcdb8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:03.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.632+0000 7fcdd21a3700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb80051c0 0x7fcdb800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:03.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd11a1700 1 -- 172.21.15.175:0/717524837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb80051c0 msgr2=0x7fcdb800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:03.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd11a1700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb80051c0 0x7fcdb800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd11a1700 1 -- 172.21.15.175:0/717524837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8003a60 msgr2=0x7fcdb8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:03.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd11a1700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8003a60 0x7fcdb8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd21a3700 1 -- 172.21.15.175:0/717524837 reap_dead start 2023-10-10T19:29:03.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd11a1700 1 -- 172.21.15.175:0/717524837 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8003570 con 0x7fcdb8004810 2023-10-10T19:29:03.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd11a1700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb8004810 0x7fcdb8004c80 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0018b0 tx=0x7fcdbc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98af4fa5d874b5a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:03.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc012420 con 0x7fcdb8004810 2023-10-10T19:29:03.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc0125a0 con 0x7fcdb8004810 2023-10-10T19:29:03.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd09a0700 1 -- 172.21.15.175:0/717524837 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc012960 con 0x7fcdb8004810 2023-10-10T19:29:03.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd3c06700 1 -- 172.21.15.175:0/717524837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb8004810 msgr2=0x7fcdb8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:03.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.633+0000 7fcdd3c06700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb8004810 0x7fcdb8004c80 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0018b0 tx=0x7fcdbc007d30 comp rx=0 tx=0).stop 2023-10-10T19:29:03.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.634+0000 7fcdd3c06700 1 -- 172.21.15.175:0/717524837 shutdown_connections 2023-10-10T19:29:03.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.634+0000 7fcdd3c06700 1 --2- 172.21.15.175:0/717524837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb8004810 0x7fcdb8004c80 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.634+0000 7fcdd3c06700 1 -- 172.21.15.175:0/717524837 >> 172.21.15.175:0/717524837 conn(0x7fcdcc063230 msgr2=0x7fcdcc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:03.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.634+0000 7fcdd3c06700 1 -- 172.21.15.175:0/717524837 shutdown_connections 2023-10-10T19:29:03.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.634+0000 7fcdd3c06700 1 -- 172.21.15.175:0/717524837 wait complete. 2023-10-10T19:29:03.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd3c06700 1 Processor -- start 2023-10-10T19:29:03.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd3c06700 1 -- start start 2023-10-10T19:29:03.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd3c06700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8004810 0x7fcdcc15ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:03.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd3c06700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdcc0cb3d0 0x7fcdcc0c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:03.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd3c06700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdcc155240 0x7fcdcc169480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:03.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd3c06700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc15a6a0 con 0x7fcdcc155240 2023-10-10T19:29:03.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd3c06700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc0638f0 con 0x7fcdb8004810 2023-10-10T19:29:03.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd3c06700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc15a070 con 0x7fcdcc0cb3d0 2023-10-10T19:29:03.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd21a3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdcc155240 0x7fcdcc169480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:03.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd19a2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8004810 0x7fcdcc15ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:03.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd21a3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdcc155240 0x7fcdcc169480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:36000/0 (socket says 172.21.15.175:36000) 2023-10-10T19:29:03.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.635+0000 7fcdd21a3700 1 -- 172.21.15.175:0/1860358836 learned_addr learned my addr 172.21.15.175:0/1860358836 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:03.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd11a1700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdcc0cb3d0 0x7fcdcc0c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:03.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd21a3700 1 -- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdcc0cb3d0 msgr2=0x7fcdcc0c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:03.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd21a3700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdcc0cb3d0 0x7fcdcc0c4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd21a3700 1 -- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8004810 msgr2=0x7fcdcc15ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:03.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd21a3700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8004810 0x7fcdcc15ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd21a3700 1 -- 172.21.15.175:0/1860358836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc169dc0 con 0x7fcdcc155240 2023-10-10T19:29:03.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd19a2700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8004810 0x7fcdcc15ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:03.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd21a3700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdcc155240 0x7fcdcc169480 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc08a030 tx=0x7fcdc40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:03.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdc1ffb700 1 -- 172.21.15.175:0/1860358836 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400f7d0 con 0x7fcdcc155240 2023-10-10T19:29:03.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdc1ffb700 1 -- 172.21.15.175:0/1860358836 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc400f930 con 0x7fcdcc155240 2023-10-10T19:29:03.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdcc15a330 con 0x7fcdcc155240 2023-10-10T19:29:03.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdc1ffb700 1 -- 172.21.15.175:0/1860358836 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400fcd0 con 0x7fcdcc155240 2023-10-10T19:29:03.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.636+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdcc16a0a0 con 0x7fcdcc155240 2023-10-10T19:29:03.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.637+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdcc15a810 con 0x7fcdcc155240 2023-10-10T19:29:03.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.638+0000 7fcdc1ffb700 1 -- 172.21.15.175:0/1860358836 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcdc4014650 con 0x7fcdcc155240 2023-10-10T19:29:03.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.638+0000 7fcdc1ffb700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcdb0066f30 0x7fcdb00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:03.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.638+0000 7fcdc1ffb700 1 -- 172.21.15.175:0/1860358836 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fcdc4085380 con 0x7fcdcc155240 2023-10-10T19:29:03.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.638+0000 7fcdd19a2700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcdb0066f30 0x7fcdb00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:03.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.639+0000 7fcdd19a2700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcdb0066f30 0x7fcdb00693e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8001920 tx=0x7fcdc8006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:03.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.642+0000 7fcdc1ffb700 1 -- 172.21.15.175:0/1860358836 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcdc4051eb0 con 0x7fcdcc155240 2023-10-10T19:29:03.868 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.867+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdcc067570 con 0x7fcdcc155240 2023-10-10T19:29:03.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.868+0000 7fcdc1ffb700 1 -- 172.21.15.175:0/1860358836 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+5439 (secure 0 0 0) 0x7fcdc4022090 con 0x7fcdcc155240 2023-10-10T19:29:03.869 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:03.870 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:28:59.041204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:29:03.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcdb0066f30 msgr2=0x7fcdb00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:03.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcdb0066f30 0x7fcdb00693e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8001920 tx=0x7fcdc8006040 comp rx=0 tx=0).stop 2023-10-10T19:29:03.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdcc155240 msgr2=0x7fcdcc169480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:03.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdcc155240 0x7fcdcc169480 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc08a030 tx=0x7fcdc40091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 shutdown_connections 2023-10-10T19:29:03.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fcdb0066f30 0x7fcdb00693e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdcc155240 0x7fcdcc169480 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb8004810 0x7fcdcc15ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 --2- 172.21.15.175:0/1860358836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdcc0cb3d0 0x7fcdcc0c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:03.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 >> 172.21.15.175:0/1860358836 conn(0x7fcdcc063230 msgr2=0x7fcdcc0ca5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:03.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 shutdown_connections 2023-10-10T19:29:03.873 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:03.870+0000 7fcdd3c06700 1 -- 172.21.15.175:0/1860358836 wait complete. 2023-10-10T19:29:03.874 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 318 2023-10-10T19:29:03.883 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-10T19:29:05.884 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:29:06.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.011+0000 7f0fbaf6f700 1 Processor -- start 2023-10-10T19:29:06.013 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.012+0000 7f0fbaf6f700 1 -- start start 2023-10-10T19:29:06.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.012+0000 7f0fbaf6f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c2b80 0x7f0fb4156580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:06.016 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.012+0000 7f0fbaf6f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0fb40c35a0 0x7f0fb4156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:06.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.012+0000 7f0fbaf6f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb415a6c0 con 0x7f0fb40c6a30 2023-10-10T19:29:06.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.012+0000 7f0fbaf6f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb414d4a0 con 0x7f0fb40c35a0 2023-10-10T19:29:06.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.012+0000 7f0fbaf6f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4159830 con 0x7f0fb40c2b80 2023-10-10T19:29:06.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb8d0b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c2b80 0x7f0fb4156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:06.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fabfff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0fb40c35a0 0x7f0fb4156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:06.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb8d0b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c2b80 0x7f0fb4156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:34048/0 (socket says 172.21.15.175:34048) 2023-10-10T19:29:06.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fabfff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0fb40c35a0 0x7f0fb4156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:45742/0 (socket says 172.21.15.175:45742) 2023-10-10T19:29:06.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb8d0b700 1 -- 172.21.15.175:0/260181093 learned_addr learned my addr 172.21.15.175:0/260181093 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:06.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb950c700 1 --1- 172.21.15.175:0/260181093 >> v1:172.21.15.191:6789/0 conn(0x7f0fb40c6a30 0x7f0fb40c6eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:39770/0 (socket says 172.21.15.175:39770) 2023-10-10T19:29:06.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb8d0b700 1 -- 172.21.15.175:0/260181093 >> v2:172.21.15.191:3300/0 conn(0x7f0fb40c35a0 msgr2=0x7f0fb4156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:06.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb8d0b700 1 --2- 172.21.15.175:0/260181093 >> v2:172.21.15.191:3300/0 conn(0x7f0fb40c35a0 0x7f0fb4156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb8d0b700 1 -- 172.21.15.175:0/260181093 >> v1:172.21.15.191:6789/0 conn(0x7f0fb40c6a30 legacy=0x7f0fb40c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:06.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb8d0b700 1 -- 172.21.15.175:0/260181093 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4159ab0 con 0x7f0fb40c2b80 2023-10-10T19:29:06.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fb8d0b700 1 --2- 172.21.15.175:0/260181093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c2b80 0x7f0fb4156580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00099b0 tx=0x7f0fb0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5006b01f0e101ea5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:06.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fab7fe700 1 -- 172.21.15.175:0/260181093 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0009ea0 con 0x7f0fb40c2b80 2023-10-10T19:29:06.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fab7fe700 1 -- 172.21.15.175:0/260181093 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb000b8f0 con 0x7f0fb40c2b80 2023-10-10T19:29:06.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.013+0000 7f0fab7fe700 1 -- 172.21.15.175:0/260181093 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb000bc90 con 0x7f0fb40c2b80 2023-10-10T19:29:06.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.014+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/260181093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c2b80 msgr2=0x7f0fb4156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:06.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.014+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/260181093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c2b80 0x7f0fb4156580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00099b0 tx=0x7f0fb0009250 comp rx=0 tx=0).stop 2023-10-10T19:29:06.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.014+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/260181093 shutdown_connections 2023-10-10T19:29:06.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.014+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/260181093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c2b80 0x7f0fb4156580 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.014+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/260181093 >> v2:172.21.15.191:3300/0 conn(0x7f0fb40c35a0 0x7f0fb4156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.014+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/260181093 >> 172.21.15.175:0/260181093 conn(0x7f0fb40c56a0 msgr2=0x7f0fb40c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:06.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.014+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/260181093 shutdown_connections 2023-10-10T19:29:06.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.014+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/260181093 wait complete. 2023-10-10T19:29:06.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.015+0000 7f0fbaf6f700 1 Processor -- start 2023-10-10T19:29:06.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.015+0000 7f0fbaf6f700 1 -- start start 2023-10-10T19:29:06.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fbaf6f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fb40c2b80 0x7f0fb40679b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:06.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fbaf6f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fb40c35a0 0x7f0fb41662f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:06.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fbaf6f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c6a30 0x7f0fb4166830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:06.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fbaf6f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb416a830 con 0x7f0fb40c35a0 2023-10-10T19:29:06.027 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fbaf6f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb416a3a0 con 0x7f0fb40c2b80 2023-10-10T19:29:06.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fbaf6f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb406f890 con 0x7f0fb40c6a30 2023-10-10T19:29:06.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fb40c35a0 0x7f0fb41662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:06.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fb40c35a0 0x7f0fb41662f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42882/0 (socket says 172.21.15.175:42882) 2023-10-10T19:29:06.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 -- 172.21.15.175:0/2253285199 learned_addr learned my addr 172.21.15.175:0/2253285199 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:06.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fb950c700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c6a30 0x7f0fb4166830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:06.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fb8d0b700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fb40c2b80 0x7f0fb40679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:06.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 -- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c6a30 msgr2=0x7f0fb4166830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:06.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c6a30 0x7f0fb4166830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 -- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fb40c2b80 msgr2=0x7f0fb40679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:06.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fb40c2b80 0x7f0fb40679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 -- 172.21.15.175:0/2253285199 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00094f0 con 0x7f0fb40c35a0 2023-10-10T19:29:06.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fb8d0b700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fb40c2b80 0x7f0fb40679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:06.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fb950c700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c6a30 0x7f0fb4166830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:06.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fabfff700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fb40c35a0 0x7f0fb41662f0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400ecb0 tx=0x7f0fa40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:06.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fa8ff9700 1 -- 172.21.15.175:0/2253285199 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa400c9d0 con 0x7f0fb40c35a0 2023-10-10T19:29:06.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fa8ff9700 1 -- 172.21.15.175:0/2253285199 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa400cb30 con 0x7f0fb40c35a0 2023-10-10T19:29:06.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb4169a70 con 0x7f0fb40c35a0 2023-10-10T19:29:06.031 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fa8ff9700 1 -- 172.21.15.175:0/2253285199 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4013470 con 0x7f0fb40c35a0 2023-10-10T19:29:06.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.017+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb416a030 con 0x7f0fb40c35a0 2023-10-10T19:29:06.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.018+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb416a1f0 con 0x7f0fb40c35a0 2023-10-10T19:29:06.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.018+0000 7f0fa8ff9700 1 -- 172.21.15.175:0/2253285199 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0fa40137e0 con 0x7f0fb40c35a0 2023-10-10T19:29:06.032 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.018+0000 7f0fa8ff9700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fa0066f30 0x7f0fa00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:06.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.019+0000 7f0fa8ff9700 1 -- 172.21.15.175:0/2253285199 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f0fa4085120 con 0x7f0fb40c35a0 2023-10-10T19:29:06.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.019+0000 7f0fb8d0b700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fa0066f30 0x7f0fa00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:06.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.019+0000 7f0fb8d0b700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fa0066f30 0x7f0fa00693e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00099b0 tx=0x7f0fb0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:06.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.024+0000 7f0fa8ff9700 1 -- 172.21.15.175:0/2253285199 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0fa4051a70 con 0x7f0fb40c35a0 2023-10-10T19:29:06.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.250+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fb40bb1f0 con 0x7f0fb40c35a0 2023-10-10T19:29:06.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.251+0000 7f0fa8ff9700 1 -- 172.21.15.175:0/2253285199 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 319 v319) v1 ==== 95+0+5433 (secure 0 0 0) 0x7f0fa4055090 con 0x7f0fb40c35a0 2023-10-10T19:29:06.252 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:06.253 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":319,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":319,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:29:05.061764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:29:06.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.252+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fa0066f30 msgr2=0x7f0fa00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:06.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.252+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fa0066f30 0x7f0fa00693e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00099b0 tx=0x7f0fb0017000 comp rx=0 tx=0).stop 2023-10-10T19:29:06.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.252+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fb40c35a0 msgr2=0x7f0fb41662f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:06.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.252+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fb40c35a0 0x7f0fb41662f0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400ecb0 tx=0x7f0fa40091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.253+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 shutdown_connections 2023-10-10T19:29:06.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.253+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0fa0066f30 0x7f0fa00693e0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.253+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fb40c6a30 0x7f0fb4166830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.253+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fb40c2b80 0x7f0fb40679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.253+0000 7f0fbaf6f700 1 --2- 172.21.15.175:0/2253285199 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0fb40c35a0 0x7f0fb41662f0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:06.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.253+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 >> 172.21.15.175:0/2253285199 conn(0x7f0fb40c56a0 msgr2=0x7f0fb40c17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:06.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.253+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 shutdown_connections 2023-10-10T19:29:06.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:06.253+0000 7f0fbaf6f700 1 -- 172.21.15.175:0/2253285199 wait complete. 2023-10-10T19:29:06.257 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 319 2023-10-10T19:29:06.266 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=6893 2023-10-10T19:29:06.266 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.d 2023-10-10T19:29:07.086 INFO:tasks.ceph.mds.e.smithi175.stderr:2023-10-10T19:29:07.085+0000 7f7f74406700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:29:10.095 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/existing-perm-race 2023-10-10T19:29:10.137 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/parent-perm-race 2023-10-10T19:29:10.139 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/reflink-auto 2023-10-10T19:29:10.152 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/cp-mv-enotsup-xattr 2023-10-10T19:29:10.153 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/no-deref-link2 2023-10-10T19:29:10.154 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/cp-mv-backup 2023-10-10T19:29:10.154 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/link-no-deref 2023-10-10T19:29:10.155 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/file-perm-race 2023-10-10T19:29:10.156 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/abuse 2023-10-10T19:29:10.156 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/link-heap 2023-10-10T19:29:10.157 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/backup-dir 2023-10-10T19:29:10.158 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/preserve-gid 2023-10-10T19:29:10.158 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/link 2023-10-10T19:29:10.159 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/cp-parents 2023-10-10T19:29:10.159 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/no-deref-link1 2023-10-10T19:29:10.160 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/into-self 2023-10-10T19:29:10.161 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/parent-perm 2023-10-10T19:29:10.161 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/cp-deref 2023-10-10T19:29:10.162 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/proc-short-read 2023-10-10T19:29:10.162 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/backup-is-src 2023-10-10T19:29:10.163 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/cp-i 2023-10-10T19:29:10.164 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/special-f 2023-10-10T19:29:10.164 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/proc-zero-len 2023-10-10T19:29:10.165 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/slink-2-slink 2023-10-10T19:29:10.166 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/special-bits 2023-10-10T19:29:10.166 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/fail-perm 2023-10-10T19:29:10.167 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/link-preserve 2023-10-10T19:29:10.167 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/dir-rm-dest 2023-10-10T19:29:10.168 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/backup-1 2023-10-10T19:29:10.169 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/cp-HL 2023-10-10T19:29:10.169 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/thru-dangling 2023-10-10T19:29:10.170 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/sparse 2023-10-10T19:29:10.170 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/deref-slink 2023-10-10T19:29:10.171 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/symlink-slash 2023-10-10T19:29:10.172 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/acl 2023-10-10T19:29:10.172 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/perm 2023-10-10T19:29:10.173 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/reflink-perm 2023-10-10T19:29:10.173 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/cp/no-deref-link3 2023-10-10T19:29:10.174 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/ 2023-10-10T19:29:10.175 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/t-slash 2023-10-10T19:29:10.176 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/p-v 2023-10-10T19:29:10.176 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/p-3 2023-10-10T19:29:10.177 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/writable-under-readonly 2023-10-10T19:29:10.177 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/p-slashdot 2023-10-10T19:29:10.178 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/selinux 2023-10-10T19:29:10.179 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/p-1 2023-10-10T19:29:10.179 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/p-2 2023-10-10T19:29:10.180 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/parents 2023-10-10T19:29:10.180 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/special-1 2023-10-10T19:29:10.181 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/perm 2023-10-10T19:29:10.181 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mkdir/p-thru-slink 2023-10-10T19:29:10.182 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/require-perl 2023-10-10T19:29:10.183 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ln/ 2023-10-10T19:29:10.184 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ln/target-1 2023-10-10T19:29:10.184 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ln/hard-to-sym 2023-10-10T19:29:10.185 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ln/hard-backup 2023-10-10T19:29:10.185 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ln/sf-1 2023-10-10T19:29:10.186 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ln/backup-1 2023-10-10T19:29:10.187 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ln/slash-decorated-nonexistent-dest 2023-10-10T19:29:10.187 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ln/misc 2023-10-10T19:29:10.446 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/lang-default 2023-10-10T19:29:10.447 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/check.mk 2023-10-10T19:29:10.447 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/Makefile.in 2023-10-10T19:29:10.448 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/ 2023-10-10T19:29:10.449 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/create-leading 2023-10-10T19:29:10.450 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/install-C-selinux 2023-10-10T19:29:10.451 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/trap 2023-10-10T19:29:10.451 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/basic-1 2023-10-10T19:29:10.452 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/strip-program 2023-10-10T19:29:10.452 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/d-slashdot 2023-10-10T19:29:10.453 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/install-C 2023-10-10T19:29:10.454 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/install/install-C-root 2023-10-10T19:29:10.454 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/ 2023-10-10T19:29:10.455 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/c-option 2023-10-10T19:29:10.456 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/octal 2023-10-10T19:29:10.457 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/umask-x 2023-10-10T19:29:10.457 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/inaccessible 2023-10-10T19:29:10.458 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/equal-x 2023-10-10T19:29:10.459 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/silent 2023-10-10T19:29:10.459 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/equals 2023-10-10T19:29:10.460 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/usage 2023-10-10T19:29:10.460 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/thru-dangling 2023-10-10T19:29:10.461 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/setgid 2023-10-10T19:29:11.306 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chmod/no-x 2023-10-10T19:29:11.307 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/df/ 2023-10-10T19:29:11.308 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/df/unreadable 2023-10-10T19:29:11.309 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/df/total-verify 2023-10-10T19:29:11.310 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/other-fs-tmpdir 2023-10-10T19:29:11.310 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/ 2023-10-10T19:29:11.311 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/basic 2023-10-10T19:29:11.312 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/exclude 2023-10-10T19:29:11.313 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/one-file-system 2023-10-10T19:29:11.313 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/long-sloop 2023-10-10T19:29:11.314 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/slink 2023-10-10T19:29:11.315 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/inacc-dir 2023-10-10T19:29:11.315 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/inaccessible-cwd 2023-10-10T19:29:11.316 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/inacc-dest 2023-10-10T19:29:11.317 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/deref-args 2023-10-10T19:29:11.317 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/deref 2023-10-10T19:29:11.318 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/trailing-slash 2023-10-10T19:29:11.318 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/files0-from 2023-10-10T19:29:11.319 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/two-args 2023-10-10T19:29:11.320 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/hard-link 2023-10-10T19:29:11.320 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/long-from-unreadable 2023-10-10T19:29:11.321 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/8gb 2023-10-10T19:29:11.322 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/2g 2023-10-10T19:29:11.322 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/restore-wd 2023-10-10T19:29:11.583 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/slash 2023-10-10T19:29:11.583 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/no-x 2023-10-10T19:29:11.584 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/fd-leak 2023-10-10T19:29:11.585 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/du/no-deref 2023-10-10T19:29:11.585 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/ 2023-10-10T19:29:11.587 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/recursive 2023-10-10T19:29:11.587 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/readdir-mountpoint-inode 2023-10-10T19:29:11.588 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/multihardlink 2023-10-10T19:29:11.589 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/color-dtype-dir 2023-10-10T19:29:11.590 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/file-type 2023-10-10T19:29:11.590 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/rt-1 2023-10-10T19:29:11.591 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/stat-free-symlinks 2023-10-10T19:29:11.592 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/no-cap 2023-10-10T19:29:11.855 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/capability 2023-10-10T19:29:11.855 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/inode 2023-10-10T19:29:11.856 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/color-norm 2023-10-10T19:29:11.856 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/abmon-align 2023-10-10T19:29:11.857 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/dangle 2023-10-10T19:29:11.858 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/dired 2023-10-10T19:29:12.911 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/m-option 2023-10-10T19:29:12.912 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/nameless-uid 2023-10-10T19:29:12.913 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/proc-selinux-segfault 2023-10-10T19:29:12.914 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/stat-vs-dirent 2023-10-10T19:29:12.914 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/color-clear-to-eol 2023-10-10T19:29:12.915 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/no-arg 2023-10-10T19:29:12.916 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/follow-slink 2023-10-10T19:29:12.917 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/symlink-slash 2023-10-10T19:29:12.917 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/stat-dtype 2023-10-10T19:29:12.918 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/x-option 2023-10-10T19:29:12.919 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/infloop 2023-10-10T19:29:12.920 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/ls/stat-failed 2023-10-10T19:29:12.920 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/ 2023-10-10T19:29:12.922 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/hard-3 2023-10-10T19:29:12.923 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/mv-n 2023-10-10T19:29:12.924 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/to-symlink 2023-10-10T19:29:12.924 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/part-fail 2023-10-10T19:29:12.925 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/hard-4 2023-10-10T19:29:12.926 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/leak-fd 2023-10-10T19:29:12.927 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/no-target-dir 2023-10-10T19:29:12.927 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/part-rename 2023-10-10T19:29:12.928 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/dir-file 2023-10-10T19:29:12.929 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/hard-2 2023-10-10T19:29:12.929 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/part-hardlink 2023-10-10T19:29:12.930 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/sticky-to-xpart 2023-10-10T19:29:12.931 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/mv-special-1 2023-10-10T19:29:12.932 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/trailing-slash 2023-10-10T19:29:12.932 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/dir2dir 2023-10-10T19:29:12.933 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/into-self-2 2023-10-10T19:29:12.934 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/backup-dir 2023-10-10T19:29:12.934 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/diag 2023-10-10T19:29:12.935 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/atomic 2023-10-10T19:29:12.936 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/force 2023-10-10T19:29:12.936 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/into-self-3 2023-10-10T19:29:12.937 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/into-self 2023-10-10T19:29:12.938 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/i-2 2023-10-10T19:29:12.938 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/childproof 2023-10-10T19:29:12.939 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/perm-1 2023-10-10T19:29:12.940 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/i-3 2023-10-10T19:29:12.941 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/partition-perm 2023-10-10T19:29:12.942 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/part-symlink 2023-10-10T19:29:12.942 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/backup-is-src 2023-10-10T19:29:12.943 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/i-1 2023-10-10T19:29:12.944 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/update 2023-10-10T19:29:12.945 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/i-4 2023-10-10T19:29:12.946 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/hard-verbose 2023-10-10T19:29:12.946 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/dup-source 2023-10-10T19:29:12.947 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/i-link-no 2023-10-10T19:29:12.947 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/atomic2 2023-10-10T19:29:12.948 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/into-self-4 2023-10-10T19:29:12.949 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/hard-link-1 2023-10-10T19:29:12.950 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/i-5 2023-10-10T19:29:12.950 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/mv/acl 2023-10-10T19:29:12.951 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/ 2023-10-10T19:29:12.952 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/fifo 2023-10-10T19:29:12.952 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/empty-file 2023-10-10T19:29:12.953 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/read-only 2023-10-10T19:29:12.954 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/fail-diag 2023-10-10T19:29:12.954 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/trailing-slash 2023-10-10T19:29:12.955 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/no-dereference 2023-10-10T19:29:12.956 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/dir-1 2023-10-10T19:29:12.956 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/dangling-symlink 2023-10-10T19:29:12.957 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/now-owned-by-other 2023-10-10T19:29:12.958 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/no-create-missing 2023-10-10T19:29:12.958 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/no-rights 2023-10-10T19:29:12.959 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/obsolescent 2023-10-10T19:29:12.960 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/60-seconds 2023-10-10T19:29:12.960 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/not-owner 2023-10-10T19:29:12.961 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/touch/relative 2023-10-10T19:29:12.962 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/ 2023-10-10T19:29:12.963 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort-merge-fdlimit 2023-10-10T19:29:12.964 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/basename 2023-10-10T19:29:12.965 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/stat-fmt 2023-10-10T19:29:12.965 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/pr 2023-10-10T19:29:12.966 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/dirname 2023-10-10T19:29:12.967 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/chcon-fail 2023-10-10T19:29:12.967 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/head-elide-tail 2023-10-10T19:29:12.968 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/chroot-credentials 2023-10-10T19:29:12.969 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/nl 2023-10-10T19:29:12.969 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/tee-dash 2023-10-10T19:29:12.970 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/stat-hyphen 2023-10-10T19:29:12.971 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort-rand 2023-10-10T19:29:12.971 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/expr 2023-10-10T19:29:12.972 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/fmt-long-line 2023-10-10T19:29:12.973 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/split-fail 2023-10-10T19:29:12.974 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-relative 2023-10-10T19:29:12.974 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-dir-fail 2023-10-10T19:29:12.975 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/csplit 2023-10-10T19:29:12.975 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/od-N 2023-10-10T19:29:12.976 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/xattr 2023-10-10T19:29:12.977 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/mktemp 2023-10-10T19:29:12.977 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/cut 2023-10-10T19:29:12.978 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/cat-proc 2023-10-10T19:29:12.979 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/comm 2023-10-10T19:29:12.979 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/timeout-parameters 2023-10-10T19:29:12.980 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/tac-continue 2023-10-10T19:29:12.981 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/tty-eof 2023-10-10T19:29:12.981 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/df-P 2023-10-10T19:29:12.982 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/shuf 2023-10-10T19:29:12.983 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/env 2023-10-10T19:29:12.983 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/od 2023-10-10T19:29:12.984 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/split-a 2023-10-10T19:29:12.985 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/tail 2023-10-10T19:29:12.985 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/stat-slash 2023-10-10T19:29:12.986 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/timeout 2023-10-10T19:29:12.987 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/close-stdout 2023-10-10T19:29:12.987 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/wc-files0-from 2023-10-10T19:29:12.988 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sum 2023-10-10T19:29:12.989 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/expand 2023-10-10T19:29:12.989 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sum-sysv 2023-10-10T19:29:12.990 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/wc-parallel 2023-10-10T19:29:12.991 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/md5sum-newline 2023-10-10T19:29:12.991 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sha224sum 2023-10-10T19:29:12.992 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/stty 2023-10-10T19:29:12.992 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/printf-hex 2023-10-10T19:29:12.993 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/id-context 2023-10-10T19:29:12.994 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/stat-printf 2023-10-10T19:29:12.994 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/pwd-option 2023-10-10T19:29:12.995 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sha256sum 2023-10-10T19:29:12.995 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort-version 2023-10-10T19:29:12.996 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/groups-dash 2023-10-10T19:29:12.997 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/printenv 2023-10-10T19:29:12.997 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/seq 2023-10-10T19:29:12.998 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/fmt 2023-10-10T19:29:12.998 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/cat-buf 2023-10-10T19:29:12.999 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/dircolors 2023-10-10T19:29:13.000 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/fold 2023-10-10T19:29:13.000 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/selinux 2023-10-10T19:29:13.001 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/md5sum-parallel 2023-10-10T19:29:13.002 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/md5sum 2023-10-10T19:29:13.002 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/nice-fail 2023-10-10T19:29:13.003 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/groups-version 2023-10-10T19:29:13.004 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/date-sec 2023-10-10T19:29:13.004 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/split-l 2023-10-10T19:29:13.005 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/false-status 2023-10-10T19:29:13.005 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sha384sum 2023-10-10T19:29:13.006 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-dangling-symlink 2023-10-10T19:29:13.007 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/od-multiple-t 2023-10-10T19:29:13.007 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/wc-files0 2023-10-10T19:29:13.008 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/nohup 2023-10-10T19:29:13.008 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-no-create-missing 2023-10-10T19:29:13.009 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/head-pos 2023-10-10T19:29:13.010 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/uniq 2023-10-10T19:29:13.010 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/test-diag 2023-10-10T19:29:13.011 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/tac 2023-10-10T19:29:13.012 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort-continue 2023-10-10T19:29:13.012 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/head 2023-10-10T19:29:13.013 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/ptx 2023-10-10T19:29:13.014 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/nproc-avail 2023-10-10T19:29:13.014 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/su-fail 2023-10-10T19:29:13.015 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/od-x8 2023-10-10T19:29:13.015 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/nice 2023-10-10T19:29:13.016 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/chcon 2023-10-10T19:29:13.016 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/head-c 2023-10-10T19:29:13.017 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort-compress 2023-10-10T19:29:13.018 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/df 2023-10-10T19:29:13.018 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/chroot-fail 2023-10-10T19:29:13.019 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/date-next-dow 2023-10-10T19:29:13.020 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sha1sum 2023-10-10T19:29:13.020 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sha1sum-vec 2023-10-10T19:29:13.022 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/pathchk1 2023-10-10T19:29:13.022 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/arch 2023-10-10T19:29:13.023 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort-month 2023-10-10T19:29:13.023 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/shred-passes 2023-10-10T19:29:13.024 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sha512sum 2023-10-10T19:29:13.025 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/paste 2023-10-10T19:29:13.025 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/wc 2023-10-10T19:29:13.026 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/ls-time 2023-10-10T19:29:13.027 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/pwd-unreadable-parent 2023-10-10T19:29:13.027 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/test 2023-10-10T19:29:13.028 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-parameters 2023-10-10T19:29:13.028 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort-files0-from 2023-10-10T19:29:13.029 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/base64 2023-10-10T19:29:13.030 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/ptx-overrun 2023-10-10T19:29:13.030 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/printf-cov 2023-10-10T19:29:13.267 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-10T19:29:13.267 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-10T19:29:13.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-10T19:29:13.270 INFO:tasks.ceph.mds.d:Started 2023-10-10T19:29:13.270 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:29:13.350 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-10T19:29:13.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.386+0000 7f6121da1700 1 Processor -- start 2023-10-10T19:29:13.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.386+0000 7f6121da1700 1 -- start start 2023-10-10T19:29:13.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f6121da1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2430 0x7f611c14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:13.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f6121da1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f611c150290 0x7f611c155f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:13.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f6121da1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611c0b0130 con 0x7f611c0b2060 2023-10-10T19:29:13.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f6121da1700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f611c1506d0 con 0x7f611c150290 2023-10-10T19:29:13.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f6121da1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611c156c90 con 0x7f611c0b2430 2023-10-10T19:29:13.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f611bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f611c150290 0x7f611c155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:13.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f611bfff700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f611c150290 0x7f611c155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42890/0 (socket says 172.21.15.175:42890) 2023-10-10T19:29:13.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f611bfff700 1 -- 172.21.15.175:0/1936672150 learned_addr learned my addr 172.21.15.175:0/1936672150 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:13.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f611affd700 1 --2- 172.21.15.175:0/1936672150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2430 0x7f611c14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:13.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f611bfff700 1 -- 172.21.15.175:0/1936672150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2430 msgr2=0x7f611c14fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:13.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f611bfff700 1 --2- 172.21.15.175:0/1936672150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2430 0x7f611c14fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f611bfff700 1 -- 172.21.15.175:0/1936672150 >> v1:172.21.15.191:6789/0 conn(0x7f611c0b2060 legacy=0x7f611c157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:13.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.387+0000 7f611bfff700 1 -- 172.21.15.175:0/1936672150 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c0af800 con 0x7f611c150290 2023-10-10T19:29:13.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f611bfff700 1 --2- 172.21.15.175:0/1936672150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 0x7f611c155f70 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f61140099b0 tx=0x7f6114009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c3a39abf490376 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:13.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f611a7fc700 1 -- 172.21.15.175:0/1936672150 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114009ea0 con 0x7f611c150290 2023-10-10T19:29:13.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f611a7fc700 1 -- 172.21.15.175:0/1936672150 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611400b860 con 0x7f611c150290 2023-10-10T19:29:13.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f611a7fc700 1 -- 172.21.15.175:0/1936672150 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611400bc00 con 0x7f611c150290 2023-10-10T19:29:13.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f6121da1700 1 -- 172.21.15.175:0/1936672150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 msgr2=0x7f611c155f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:13.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f6121da1700 1 --2- 172.21.15.175:0/1936672150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 0x7f611c155f70 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f61140099b0 tx=0x7f6114009250 comp rx=0 tx=0).stop 2023-10-10T19:29:13.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f6121da1700 1 -- 172.21.15.175:0/1936672150 shutdown_connections 2023-10-10T19:29:13.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f6121da1700 1 --2- 172.21.15.175:0/1936672150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 0x7f611c155f70 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f6121da1700 1 --2- 172.21.15.175:0/1936672150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2430 0x7f611c14fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f6121da1700 1 -- 172.21.15.175:0/1936672150 >> 172.21.15.175:0/1936672150 conn(0x7f611c063230 msgr2=0x7f611c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:13.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f6121da1700 1 -- 172.21.15.175:0/1936672150 shutdown_connections 2023-10-10T19:29:13.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.388+0000 7f6121da1700 1 -- 172.21.15.175:0/1936672150 wait complete. 2023-10-10T19:29:13.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.389+0000 7f6121da1700 1 Processor -- start 2023-10-10T19:29:13.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f6121da1700 1 -- start start 2023-10-10T19:29:13.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f6121da1700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2060 0x7f611c0c2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:13.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f6121da1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f611c0b2430 0x7f611c0c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:13.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f6121da1700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 0x7f611c1691c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:13.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f6121da1700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c1534b0 con 0x7f611c150290 2023-10-10T19:29:13.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f6121da1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c153700 con 0x7f611c0b2430 2023-10-10T19:29:13.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f6121da1700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611c0c16f0 con 0x7f611c0b2060 2023-10-10T19:29:13.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f611bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 0x7f611c1691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:13.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f611affd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f611c0b2430 0x7f611c0c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:13.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f611b7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2060 0x7f611c0c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:13.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f611affd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f611c0b2430 0x7f611c0c2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:45758/0 (socket says 172.21.15.175:45758) 2023-10-10T19:29:13.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f611bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 0x7f611c1691c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:42892/0 (socket says 172.21.15.175:42892) 2023-10-10T19:29:13.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.390+0000 7f611affd700 1 -- 172.21.15.175:0/806781852 learned_addr learned my addr 172.21.15.175:0/806781852 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:13.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f611affd700 1 -- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2060 msgr2=0x7f611c0c2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:13.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f611affd700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2060 0x7f611c0c2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f611affd700 1 -- 172.21.15.175:0/806781852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 msgr2=0x7f611c1691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:13.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f611affd700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 0x7f611c1691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f611affd700 1 -- 172.21.15.175:0/806781852 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61140094f0 con 0x7f611c0b2430 2023-10-10T19:29:13.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f611b7fe700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2060 0x7f611c0c2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:13.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f611affd700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f611c0b2430 0x7f611c0c2a70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f611000ec80 tx=0x7f61100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:13.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f61037fe700 1 -- 172.21.15.175:0/806781852 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611000c850 con 0x7f611c0b2430 2023-10-10T19:29:13.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f61037fe700 1 -- 172.21.15.175:0/806781852 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611000c9b0 con 0x7f611c0b2430 2023-10-10T19:29:13.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c0c19d0 con 0x7f611c0b2430 2023-10-10T19:29:13.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f61037fe700 1 -- 172.21.15.175:0/806781852 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611000cd50 con 0x7f611c0b2430 2023-10-10T19:29:13.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.391+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c0c02c0 con 0x7f611c0b2430 2023-10-10T19:29:13.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.392+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c0c1520 con 0x7f611c0b2430 2023-10-10T19:29:13.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.392+0000 7f61037fe700 1 -- 172.21.15.175:0/806781852 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6110013680 con 0x7f611c0b2430 2023-10-10T19:29:13.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.393+0000 7f61037fe700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6108066f30 0x7f61080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:13.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.393+0000 7f61037fe700 1 -- 172.21.15.175:0/806781852 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f6110084fd0 con 0x7f611c0b2430 2023-10-10T19:29:13.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.393+0000 7f611b7fe700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6108066f30 0x7f61080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:13.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.393+0000 7f611b7fe700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6108066f30 0x7f61080693e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f610c007fd0 tx=0x7f610c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:13.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.397+0000 7f61037fe700 1 -- 172.21.15.175:0/806781852 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61100519c0 con 0x7f611c0b2430 2023-10-10T19:29:13.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.622+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f611c067570 con 0x7f611c0b2430 2023-10-10T19:29:13.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.623+0000 7f61037fe700 1 -- 172.21.15.175:0/806781852 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 319 v319) v1 ==== 95+0+5433 (secure 0 0 0) 0x7f6110054fe0 con 0x7f611c0b2430 2023-10-10T19:29:13.623 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:13.623 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":319,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":319,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:29:05.061764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:29:13.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6108066f30 msgr2=0x7f61080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:13.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6108066f30 0x7f61080693e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f610c007fd0 tx=0x7f610c009040 comp rx=0 tx=0).stop 2023-10-10T19:29:13.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f611c0b2430 msgr2=0x7f611c0c2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:13.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f611c0b2430 0x7f611c0c2a70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f611000ec80 tx=0x7f61100091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 shutdown_connections 2023-10-10T19:29:13.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f6108066f30 0x7f61080693e0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f611c150290 0x7f611c1691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f611c0b2060 0x7f611c0c2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 --2- 172.21.15.175:0/806781852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f611c0b2430 0x7f611c0c2a70 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:13.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 >> 172.21.15.175:0/806781852 conn(0x7f611c063230 msgr2=0x7f611c14ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:13.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 shutdown_connections 2023-10-10T19:29:13.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:13.625+0000 7f6121da1700 1 -- 172.21.15.175:0/806781852 wait complete. 2023-10-10T19:29:13.628 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 319 2023-10-10T19:29:13.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-10T19:29:15.638 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:29:15.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.755+0000 7f039d057700 1 Processor -- start 2023-10-10T19:29:15.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.755+0000 7f039d057700 1 -- start start 2023-10-10T19:29:15.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f039d057700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980b1fc0 0x7f0398155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:15.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f039d057700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03981574b0 con 0x7f03980b2390 2023-10-10T19:29:15.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f039d057700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03980afad0 con 0x7f03980aff60 2023-10-10T19:29:15.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f039d057700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0398150770 con 0x7f03980b1fc0 2023-10-10T19:29:15.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f039659c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f03980b2390 0x7f0398068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46374/0 (socket says 172.21.15.175:46374) 2023-10-10T19:29:15.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f039659c700 1 -- 172.21.15.175:0/811525954 learned_addr learned my addr 172.21.15.175:0/811525954 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:15.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f0396d9d700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980b1fc0 0x7f0398155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:15.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178783142 0 0) 0x7f03981574b0 con 0x7f03980b2390 2023-10-10T19:29:15.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.756+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0380003570 con 0x7f03980b2390 2023-10-10T19:29:15.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527470830 0 0) 0x7f03980afad0 con 0x7f03980aff60 2023-10-10T19:29:15.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03981574b0 con 0x7f03980aff60 2023-10-10T19:29:15.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4282574554 0 0) 0x7f0380003570 con 0x7f03980b2390 2023-10-10T19:29:15.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980b1fc0 msgr2=0x7f0398155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:15.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980b1fc0 0x7f0398155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 >> v1:172.21.15.191:6789/0 conn(0x7f03980aff60 legacy=0x7f0398159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:15.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03980af4c0 con 0x7f03980b2390 2023-10-10T19:29:15.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f038c001470 con 0x7f03980b2390 2023-10-10T19:29:15.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 >> v1:172.21.15.175:6789/0 conn(0x7f03980b2390 legacy=0x7f0398068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:15.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0380003c20 0x7f0380004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:15.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0380004810 0x7f0380004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:15.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03800051c0 0x7f038000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:15.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0398150770 con 0x7f0380003c20 2023-10-10T19:29:15.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038000c080 con 0x7f03800051c0 2023-10-10T19:29:15.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038000c240 con 0x7f0380004810 2023-10-10T19:29:15.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0396d9d700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980b1fc0 0x7f0398155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:15.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0396d9d700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0380003c20 0x7f0380004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:15.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f039759e700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03800051c0 0x7f038000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:15.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f039659c700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0380004810 0x7f0380004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:15.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0396d9d700 1 -- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0380004810 msgr2=0x7f0380004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:15.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0396d9d700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0380004810 0x7f0380004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0396d9d700 1 -- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03800051c0 msgr2=0x7f038000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:15.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0396d9d700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03800051c0 0x7f038000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0396d9d700 1 -- 172.21.15.175:0/811525954 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0380003570 con 0x7f0380003c20 2023-10-10T19:29:15.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0396d9d700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0380003c20 0x7f0380004010 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f03980cd830 tx=0x7f038800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d16afd204c15b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:15.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038800eaa0 con 0x7f0380003c20 2023-10-10T19:29:15.768 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038800f040 con 0x7f0380003c20 2023-10-10T19:29:15.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.757+0000 7f0395d9b700 1 -- 172.21.15.175:0/811525954 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038800ee30 con 0x7f0380003c20 2023-10-10T19:29:15.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.758+0000 7f039d057700 1 -- 172.21.15.175:0/811525954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0380003c20 msgr2=0x7f0380004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:15.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.758+0000 7f039d057700 1 --2- 172.21.15.175:0/811525954 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0380003c20 0x7f0380004010 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f03980cd830 tx=0x7f038800b040 comp rx=0 tx=0).stop 2023-10-10T19:29:15.769 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.758+0000 7f039759e700 1 -- 172.21.15.175:0/811525954 reap_dead start 2023-10-10T19:29:15.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.758+0000 7f039d057700 1 -- 172.21.15.175:0/811525954 shutdown_connections 2023-10-10T19:29:15.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.758+0000 7f039d057700 1 -- 172.21.15.175:0/811525954 >> 172.21.15.175:0/811525954 conn(0x7f0398063230 msgr2=0x7f0398063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:15.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.758+0000 7f039d057700 1 -- 172.21.15.175:0/811525954 shutdown_connections 2023-10-10T19:29:15.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.758+0000 7f039d057700 1 -- 172.21.15.175:0/811525954 wait complete. 2023-10-10T19:29:15.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.759+0000 7f039d057700 1 Processor -- start 2023-10-10T19:29:15.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.759+0000 7f039d057700 1 -- start start 2023-10-10T19:29:15.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039d057700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380003c20 0x7f0398166790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:15.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039d057700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03980c02c0 0x7f0398152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:15.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039d057700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980c65c0 0x7f03980c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:15.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039d057700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03980cd480 con 0x7f03980c02c0 2023-10-10T19:29:15.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039d057700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0398166300 con 0x7f0380003c20 2023-10-10T19:29:15.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039d057700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03980c4fb0 con 0x7f03980c65c0 2023-10-10T19:29:15.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03980c02c0 0x7f0398152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:15.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03980c02c0 0x7f0398152790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54424/0 (socket says 172.21.15.175:54424) 2023-10-10T19:29:15.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 -- 172.21.15.175:0/2402889302 learned_addr learned my addr 172.21.15.175:0/2402889302 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:15.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039759e700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980c65c0 0x7f03980c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:15.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f0396d9d700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380003c20 0x7f0398166790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:15.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 -- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980c65c0 msgr2=0x7f03980c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:15.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980c65c0 0x7f03980c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 -- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380003c20 msgr2=0x7f0398166790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:15.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380003c20 0x7f0398166790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 -- 172.21.15.175:0/2402889302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03980c5290 con 0x7f03980c02c0 2023-10-10T19:29:15.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039759e700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980c65c0 0x7f03980c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:29:15.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f039659c700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03980c02c0 0x7f0398152790 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f038c0073f0 tx=0x7f03980afa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:15.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f037effd700 1 -- 172.21.15.175:0/2402889302 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038c00f960 con 0x7f03980c02c0 2023-10-10T19:29:15.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.760+0000 7f037effd700 1 -- 172.21.15.175:0/2402889302 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038c00fb00 con 0x7f03980c02c0 2023-10-10T19:29:15.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.761+0000 7f037effd700 1 -- 172.21.15.175:0/2402889302 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038c00fec0 con 0x7f03980c02c0 2023-10-10T19:29:15.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.761+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03980c5570 con 0x7f03980c02c0 2023-10-10T19:29:15.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.761+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0398165f90 con 0x7f03980c02c0 2023-10-10T19:29:15.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.761+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03980c9650 con 0x7f03980c02c0 2023-10-10T19:29:15.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.762+0000 7f037effd700 1 -- 172.21.15.175:0/2402889302 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f038c013600 con 0x7f03980c02c0 2023-10-10T19:29:15.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.762+0000 7f037effd700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0384066f30 0x7f03840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:15.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.762+0000 7f037effd700 1 -- 172.21.15.175:0/2402889302 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f038c085170 con 0x7f03980c02c0 2023-10-10T19:29:15.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.762+0000 7f0396d9d700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0384066f30 0x7f03840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:15.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.763+0000 7f0396d9d700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0384066f30 0x7f03840693e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f03880099f0 tx=0x7f0388016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:15.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.766+0000 7f037effd700 1 -- 172.21.15.175:0/2402889302 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f038c051ac0 con 0x7f03980c02c0 2023-10-10T19:29:15.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.992+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0398067570 con 0x7f03980c02c0 2023-10-10T19:29:15.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.992+0000 7f037effd700 1 -- 172.21.15.175:0/2402889302 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 321 v321) v1 ==== 95+0+6194 (secure 0 0 0) 0x7f038c0550e0 con 0x7f03980c02c0 2023-10-10T19:29:15.993 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:15.993 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":321,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8188,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1993477403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1993477403},{"type":"v1","addr":"172.21.15.191:6837","nonce":1993477403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":320}],"filesystems":[{"mdsmap":{"epoch":321,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:29:15.092973+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:29:15.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0384066f30 msgr2=0x7f03840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:15.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0384066f30 0x7f03840693e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f03880099f0 tx=0x7f0388016000 comp rx=0 tx=0).stop 2023-10-10T19:29:15.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03980c02c0 msgr2=0x7f0398152790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:15.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03980c02c0 0x7f0398152790 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f038c0073f0 tx=0x7f03980afa40 comp rx=0 tx=0).stop 2023-10-10T19:29:15.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 shutdown_connections 2023-10-10T19:29:15.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0384066f30 0x7f03840693e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03980c65c0 0x7f03980c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0380003c20 0x7f0398166790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 --2- 172.21.15.175:0/2402889302 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03980c02c0 0x7f0398152790 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:15.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 >> 172.21.15.175:0/2402889302 conn(0x7f0398063230 msgr2=0x7f039814fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:15.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 shutdown_connections 2023-10-10T19:29:15.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:15.995+0000 7f039d057700 1 -- 172.21.15.175:0/2402889302 wait complete. 2023-10-10T19:29:15.998 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 321 2023-10-10T19:29:16.007 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-10T19:29:16.007 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 45 secs before thrashing 2023-10-10T19:29:19.056 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/tee 2023-10-10T19:29:19.060 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/mknod 2023-10-10T19:29:19.061 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/printf-surprise 2023-10-10T19:29:19.061 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/ls-misc 2023-10-10T19:29:19.062 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/nproc-positive 2023-10-10T19:29:19.063 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/join 2023-10-10T19:29:19.064 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort-merge 2023-10-10T19:29:19.065 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/factor 2023-10-10T19:29:19.065 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/sort 2023-10-10T19:29:19.066 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-fifo 2023-10-10T19:29:19.067 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-fail-diag 2023-10-10T19:29:19.068 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/stty-invalid 2023-10-10T19:29:19.068 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/env-null 2023-10-10T19:29:19.069 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/printf 2023-10-10T19:29:19.070 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/seq-long-double 2023-10-10T19:29:19.071 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/pwd-long 2023-10-10T19:29:19.071 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/tsort 2023-10-10T19:29:19.072 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/runcon-no-reorder 2023-10-10T19:29:19.073 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/date 2023-10-10T19:29:19.074 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/shred-exact 2023-10-10T19:29:19.075 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/shred-remove 2023-10-10T19:29:19.075 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/stdbuf 2023-10-10T19:29:19.076 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-overflow 2023-10-10T19:29:19.077 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/stty-row-col 2023-10-10T19:29:19.078 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/truncate-owned-by-other 2023-10-10T19:29:19.078 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/xstrtol 2023-10-10T19:29:19.079 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/invalid-opt 2023-10-10T19:29:19.081 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/readlink-fp-loop 2023-10-10T19:29:19.082 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/tr 2023-10-10T19:29:19.083 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/help-version 2023-10-10T19:29:19.083 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/id-groups 2023-10-10T19:29:19.084 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/misc/unexpand 2023-10-10T19:29:19.085 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chgrp/ 2023-10-10T19:29:19.087 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chgrp/basic 2023-10-10T19:29:19.088 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chgrp/default-no-deref 2023-10-10T19:29:19.089 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chgrp/deref 2023-10-10T19:29:19.090 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chgrp/posix-H 2023-10-10T19:29:19.091 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chgrp/recurse 2023-10-10T19:29:19.092 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/chgrp/no-x 2023-10-10T19:29:19.093 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/ 2023-10-10T19:29:19.095 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/big-4gb 2023-10-10T19:29:19.096 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/follow-stdin 2023-10-10T19:29:19.097 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/F-vs-rename 2023-10-10T19:29:19.098 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/inotify-rotate 2023-10-10T19:29:19.099 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/pid 2023-10-10T19:29:19.100 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/infloop-1 2023-10-10T19:29:19.101 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/assert-2 2023-10-10T19:29:19.102 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/pipe-f 2023-10-10T19:29:19.102 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/pipe-f2 2023-10-10T19:29:19.103 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/start-middle 2023-10-10T19:29:19.104 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/inotify-hash-abuse2 2023-10-10T19:29:19.105 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/inotify-hash-abuse 2023-10-10T19:29:19.106 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/inotify-race 2023-10-10T19:29:19.107 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/append-only 2023-10-10T19:29:19.108 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/tail-n0f 2023-10-10T19:29:19.108 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/flush-initial 2023-10-10T19:29:19.109 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/assert 2023-10-10T19:29:19.110 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/wait 2023-10-10T19:29:19.110 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/tail-2/proc-ksyms 2023-10-10T19:29:19.116 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/envvar-check 2023-10-10T19:29:19.117 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/readlink/ 2023-10-10T19:29:19.117 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/readlink/can-f 2023-10-10T19:29:19.117 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/readlink/can-m 2023-10-10T19:29:19.117 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/readlink/rl-1 2023-10-10T19:29:19.117 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/tests/readlink/can-e 2023-10-10T19:29:19.118 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_trailing_blank 2023-10-10T19:29:19.118 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/ 2023-10-10T19:29:19.119 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/coreutils.texi 2023-10-10T19:29:19.121 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/perm.texi 2023-10-10T19:29:19.122 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/version.texi 2023-10-10T19:29:19.122 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/Makefile.am 2023-10-10T19:29:19.123 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/coreutils.info 2023-10-10T19:29:19.127 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/stamp-vti 2023-10-10T19:29:19.128 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/fdl.texi 2023-10-10T19:29:19.128 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/Makefile.in 2023-10-10T19:29:19.129 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/getdate.texi 2023-10-10T19:29:19.130 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/constants.texi 2023-10-10T19:29:19.130 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/doc/ChangeLog-2007 2023-10-10T19:29:19.131 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/ 2023-10-10T19:29:19.132 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/split.c 2023-10-10T19:29:19.133 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/dircolors.hin 2023-10-10T19:29:19.134 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/test.c 2023-10-10T19:29:19.134 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/nproc.c 2023-10-10T19:29:19.135 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/cp-hash.c 2023-10-10T19:29:19.136 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/rm.c 2023-10-10T19:29:19.136 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/uname-arch.c 2023-10-10T19:29:19.137 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/chroot.c 2023-10-10T19:29:19.137 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/remove.h 2023-10-10T19:29:19.138 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/users.c 2023-10-10T19:29:19.139 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/dircolors.c 2023-10-10T19:29:19.139 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/join.c 2023-10-10T19:29:19.140 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/chown.c 2023-10-10T19:29:19.140 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/shuf.c 2023-10-10T19:29:19.141 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/sum.c 2023-10-10T19:29:19.141 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/csplit.c 2023-10-10T19:29:19.142 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/libstdbuf.c 2023-10-10T19:29:19.143 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/fold.c 2023-10-10T19:29:19.143 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/basename.c 2023-10-10T19:29:19.144 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/prog-fprintf.h 2023-10-10T19:29:19.145 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/sync.c 2023-10-10T19:29:19.145 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/lbracket.c 2023-10-10T19:29:19.146 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/ln.c 2023-10-10T19:29:19.146 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/tsort.c 2023-10-10T19:29:19.147 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/wc.c 2023-10-10T19:29:19.148 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/extract-magic 2023-10-10T19:29:19.148 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/hostid.c 2023-10-10T19:29:19.149 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/dd.c 2023-10-10T19:29:19.150 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/shred.c 2023-10-10T19:29:19.150 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/uname.h 2023-10-10T19:29:19.151 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/unlink.c 2023-10-10T19:29:19.151 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/Makefile.am 2023-10-10T19:29:19.152 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/seq.c 2023-10-10T19:29:19.153 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/uname-uname.c 2023-10-10T19:29:19.153 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/ptx.c 2023-10-10T19:29:19.154 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/wheel-size.h 2023-10-10T19:29:19.154 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/stty.c 2023-10-10T19:29:19.155 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/chcon.c 2023-10-10T19:29:19.156 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/dcgen 2023-10-10T19:29:19.156 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/uptime.c 2023-10-10T19:29:19.157 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/link.c 2023-10-10T19:29:19.158 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/install.c 2023-10-10T19:29:19.158 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/uname.c 2023-10-10T19:29:19.159 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/mknod.c 2023-10-10T19:29:19.159 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/su.c 2023-10-10T19:29:19.160 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/expand.c 2023-10-10T19:29:19.161 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/mv.c 2023-10-10T19:29:19.161 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/ls.c 2023-10-10T19:29:19.162 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/whoami.c 2023-10-10T19:29:19.163 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/ls-ls.c 2023-10-10T19:29:19.164 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/ls.h 2023-10-10T19:29:19.164 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/id.c 2023-10-10T19:29:19.165 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/Makefile.in 2023-10-10T19:29:19.166 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/du.c 2023-10-10T19:29:19.166 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/cut.c 2023-10-10T19:29:19.167 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/comm.c 2023-10-10T19:29:19.167 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/od.c 2023-10-10T19:29:19.168 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/expr.c 2023-10-10T19:29:19.169 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/nohup.c 2023-10-10T19:29:19.169 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/system.h 2023-10-10T19:29:19.170 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/cat.c 2023-10-10T19:29:19.170 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/mktemp.c 2023-10-10T19:29:19.171 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/tac.c 2023-10-10T19:29:19.172 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/ls-dir.c 2023-10-10T19:29:19.172 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/tac-pipe.c 2023-10-10T19:29:19.173 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/touch.c 2023-10-10T19:29:19.173 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/false.c 2023-10-10T19:29:19.174 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/echo.c 2023-10-10T19:29:19.175 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/printenv.c 2023-10-10T19:29:19.175 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/logname.c 2023-10-10T19:29:19.176 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/group-list.c 2023-10-10T19:29:19.176 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/pr.c 2023-10-10T19:29:19.177 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/chgrp.c 2023-10-10T19:29:19.178 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/cp-hash.h 2023-10-10T19:29:19.178 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/getlimits.c 2023-10-10T19:29:19.179 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/rmdir.c 2023-10-10T19:29:19.179 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/tail.c 2023-10-10T19:29:19.180 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/timeout.c 2023-10-10T19:29:19.180 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/sort.c 2023-10-10T19:29:19.181 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/operand2sig.c 2023-10-10T19:29:19.182 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/factor.c 2023-10-10T19:29:19.182 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/paste.c 2023-10-10T19:29:19.183 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/chmod.c 2023-10-10T19:29:19.184 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/pinky.c 2023-10-10T19:29:19.184 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/unexpand.c 2023-10-10T19:29:19.185 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/cksum.c 2023-10-10T19:29:19.185 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/date.c 2023-10-10T19:29:19.186 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/mkdir.c 2023-10-10T19:29:19.187 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/printf.c 2023-10-10T19:29:19.187 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/true.c 2023-10-10T19:29:19.188 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/c99-to-c89.diff 2023-10-10T19:29:19.188 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/groups.c 2023-10-10T19:29:19.189 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/dirname.c 2023-10-10T19:29:19.190 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/chown-core.h 2023-10-10T19:29:19.190 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/fs.h 2023-10-10T19:29:19.191 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/nl.c 2023-10-10T19:29:19.191 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/tee.c 2023-10-10T19:29:19.192 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/dircolors.h 2023-10-10T19:29:19.193 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/head.c 2023-10-10T19:29:19.193 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/uniq.c 2023-10-10T19:29:19.194 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/chown-core.c 2023-10-10T19:29:19.194 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/runcon.c 2023-10-10T19:29:19.195 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/sleep.c 2023-10-10T19:29:19.195 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/md5sum.c 2023-10-10T19:29:19.196 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/yes.c 2023-10-10T19:29:19.197 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/readlink.c 2023-10-10T19:29:19.197 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/env.c 2023-10-10T19:29:19.198 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/df.c 2023-10-10T19:29:19.198 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/nice.c 2023-10-10T19:29:19.199 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/fmt.c 2023-10-10T19:29:19.200 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/base64.c 2023-10-10T19:29:19.200 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/tty.c 2023-10-10T19:29:19.201 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/truncate.c 2023-10-10T19:29:19.201 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/mkfifo.c 2023-10-10T19:29:19.202 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/operand2sig.h 2023-10-10T19:29:19.202 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/copy.h 2023-10-10T19:29:19.203 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/kill.c 2023-10-10T19:29:19.204 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/who.c 2023-10-10T19:29:19.204 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/stat.c 2023-10-10T19:29:19.205 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/remove.c 2023-10-10T19:29:19.205 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/pwd.c 2023-10-10T19:29:19.206 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/prog-fprintf.c 2023-10-10T19:29:19.207 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/ls-vdir.c 2023-10-10T19:29:19.207 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/copy.c 2023-10-10T19:29:19.208 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/hostname.c 2023-10-10T19:29:19.208 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/wheel-gen.pl 2023-10-10T19:29:19.209 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/group-list.h 2023-10-10T19:29:19.210 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/cp.c 2023-10-10T19:29:19.210 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/setuidgid.c 2023-10-10T19:29:19.211 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/tr.c 2023-10-10T19:29:19.211 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/stdbuf.c 2023-10-10T19:29:19.212 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/pathchk.c 2023-10-10T19:29:19.441 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/src/wheel.h 2023-10-10T19:29:19.442 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/configure 2023-10-10T19:29:19.448 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/README 2023-10-10T19:29:19.449 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.vg-suppressions 2023-10-10T19:29:19.449 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_prohibit_atoi_atof 2023-10-10T19:29:19.450 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_prohibit_stat_st_blocks 2023-10-10T19:29:19.451 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/ChangeLog-2006 2023-10-10T19:29:19.452 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_program_name 2023-10-10T19:29:19.452 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/THANKStt.in 2023-10-10T19:29:19.453 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/ChangeLog-2008 2023-10-10T19:29:19.454 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/ChangeLog-2007 2023-10-10T19:29:19.455 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/ 2023-10-10T19:29:19.456 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-isnand.h 2023-10-10T19:29:19.457 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-ftello.sh 2023-10-10T19:29:19.457 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-snprintf.c 2023-10-10T19:29:19.458 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-openat.c 2023-10-10T19:29:19.459 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-dirent-safer.c 2023-10-10T19:29:19.459 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-cloexec.c 2023-10-10T19:29:19.460 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fstatat.c 2023-10-10T19:29:19.460 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fnmatch.c 2023-10-10T19:29:19.461 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getndelim2.c 2023-10-10T19:29:19.461 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-wcrtomb.sh 2023-10-10T19:29:19.462 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-filenamecat.c 2023-10-10T19:29:19.463 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fchownat.c 2023-10-10T19:29:19.463 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-stdlib.c 2023-10-10T19:29:19.464 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-read-file.c 2023-10-10T19:29:19.464 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-exclude7.sh 2023-10-10T19:29:19.465 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-closein.c 2023-10-10T19:29:19.465 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/connect.c 2023-10-10T19:29:19.466 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-rename.h 2023-10-10T19:29:19.467 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-memchr2.c 2023-10-10T19:29:19.467 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sys_time.c 2023-10-10T19:29:19.468 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/read-file.h 2023-10-10T19:29:19.468 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mkdirat.c 2023-10-10T19:29:19.469 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-isnand-nolibm.c 2023-10-10T19:29:19.469 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fopen-safer.c 2023-10-10T19:29:19.470 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-stdio.c 2023-10-10T19:29:19.470 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fseeko.c 2023-10-10T19:29:19.471 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-arpa_inet.c 2023-10-10T19:29:19.472 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freadahead.sh 2023-10-10T19:29:19.472 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-openat-safer.c 2023-10-10T19:29:19.473 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-unlink.c 2023-10-10T19:29:19.473 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsstr2.sh 2023-10-10T19:29:19.474 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-lchown.c 2023-10-10T19:29:19.475 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-nl_langinfo.c 2023-10-10T19:29:19.475 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-open.h 2023-10-10T19:29:19.476 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-memcmp.c 2023-10-10T19:29:19.476 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vfprintf-posix.c 2023-10-10T19:29:19.477 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freadahead.c 2023-10-10T19:29:19.477 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/ioctl.c 2023-10-10T19:29:19.478 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-iconv-h.c 2023-10-10T19:29:19.478 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mkdir.c 2023-10-10T19:29:19.479 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-posixtm.c 2023-10-10T19:29:19.479 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/wctob.c 2023-10-10T19:29:19.480 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/glthread/ 2023-10-10T19:29:19.481 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/glthread/thread.c 2023-10-10T19:29:19.481 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/glthread/yield.h 2023-10-10T19:29:19.482 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/glthread/thread.h 2023-10-10T19:29:19.483 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/unistr/ 2023-10-10T19:29:19.483 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/unistr/test-u8-uctomb.c 2023-10-10T19:29:19.484 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-exclude5.sh 2023-10-10T19:29:19.485 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-rmdir.h 2023-10-10T19:29:19.485 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-symlink.h 2023-10-10T19:29:19.486 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-unsetenv.c 2023-10-10T19:29:19.486 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-exclude4.sh 2023-10-10T19:29:19.487 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/listen.c 2023-10-10T19:29:19.487 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-printf-posix.output 2023-10-10T19:29:19.488 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc3.sh 2023-10-10T19:29:19.488 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-dirent.c 2023-10-10T19:29:19.489 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-utimens-common.h 2023-10-10T19:29:19.489 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsinit.c 2023-10-10T19:29:19.490 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/w32sock.h 2023-10-10T19:29:19.490 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-printf-posix.h 2023-10-10T19:29:19.491 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fchdir.c 2023-10-10T19:29:19.491 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/read-file.c 2023-10-10T19:29:19.492 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-ctype.c 2023-10-10T19:29:19.493 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/binary-io.h 2023-10-10T19:29:19.493 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/nap.h 2023-10-10T19:29:19.494 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/sys_ioctl.in.h 2023-10-10T19:29:19.494 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-ftello2.sh 2023-10-10T19:29:19.495 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-memchr.c 2023-10-10T19:29:19.496 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs3.sh 2023-10-10T19:29:19.496 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fseeko.sh 2023-10-10T19:29:19.497 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-select-out.sh 2023-10-10T19:29:19.497 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/inet_pton.c 2023-10-10T19:29:19.498 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-stdbool.c 2023-10-10T19:29:19.499 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-nanosleep.c 2023-10-10T19:29:19.500 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/bind.c 2023-10-10T19:29:19.501 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/unlinkdir.h 2023-10-10T19:29:19.502 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getline.c 2023-10-10T19:29:19.502 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-md5.c 2023-10-10T19:29:19.503 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/Makefile.am 2023-10-10T19:29:19.504 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-select-in.sh 2023-10-10T19:29:19.505 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-uname.c 2023-10-10T19:29:19.506 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-readlink.h 2023-10-10T19:29:19.507 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-string.c 2023-10-10T19:29:19.508 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-argmatch.c 2023-10-10T19:29:19.509 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-nl_langinfo.sh 2023-10-10T19:29:19.510 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sys_stat.c 2023-10-10T19:29:19.510 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-version-etc.c 2023-10-10T19:29:19.511 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-areadlink.h 2023-10-10T19:29:19.512 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-version-etc.sh 2023-10-10T19:29:19.513 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-binary-io.c 2023-10-10T19:29:19.514 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-exclude2.sh 2023-10-10T19:29:19.515 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xfprintf-posix.c 2023-10-10T19:29:19.516 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-set-mode-acl.c 2023-10-10T19:29:19.517 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-wchar.c 2023-10-10T19:29:19.518 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vasprintf.c 2023-10-10T19:29:19.518 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xstrtoimax.sh 2023-10-10T19:29:19.519 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freadseek.sh 2023-10-10T19:29:19.520 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-isblank.c 2023-10-10T19:29:19.521 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fpending.sh 2023-10-10T19:29:19.522 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-lstat.h 2023-10-10T19:29:19.523 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-chown.c 2023-10-10T19:29:19.524 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xstrtoumax.sh 2023-10-10T19:29:19.525 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/nan.h 2023-10-10T19:29:19.526 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xprintf-posix.c 2023-10-10T19:29:19.526 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-inet_pton.c 2023-10-10T19:29:19.527 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-dup-safer.c 2023-10-10T19:29:19.528 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-closein.sh 2023-10-10T19:29:19.529 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-canonicalize.c 2023-10-10T19:29:19.530 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-isnanl-nolibm.c 2023-10-10T19:29:19.531 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xstrtol.c 2023-10-10T19:29:19.532 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-open.c 2023-10-10T19:29:19.533 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fprintf-posix.h 2023-10-10T19:29:19.534 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc1.sh 2023-10-10T19:29:19.535 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-errno.c 2023-10-10T19:29:19.535 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-netdb.c 2023-10-10T19:29:19.536 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-wctype.c 2023-10-10T19:29:19.537 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-strsignal.c 2023-10-10T19:29:19.538 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsstr3.sh 2023-10-10T19:29:19.539 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-yesno.c 2023-10-10T19:29:19.540 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-userspec.c 2023-10-10T19:29:19.541 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freopen.c 2023-10-10T19:29:19.543 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/uniwidth/ 2023-10-10T19:29:19.544 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.c 2023-10-10T19:29:19.544 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/uniwidth/test-uc_width.c 2023-10-10T19:29:19.545 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.sh 2023-10-10T19:29:19.545 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sockets.c 2023-10-10T19:29:19.546 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-link.h 2023-10-10T19:29:19.547 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-select-fd.c 2023-10-10T19:29:19.548 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xstrtoimax.c 2023-10-10T19:29:19.806 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xvasprintf.c 2023-10-10T19:29:19.807 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-yesno.sh 2023-10-10T19:29:19.808 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fflush.c 2023-10-10T19:29:19.809 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vc-list-files-cvs.sh 2023-10-10T19:29:19.810 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbscasecmp.sh 2023-10-10T19:29:19.811 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-lseek.sh 2023-10-10T19:29:19.812 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/usleep.c 2023-10-10T19:29:19.813 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sleep.c 2023-10-10T19:29:19.813 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-link.c 2023-10-10T19:29:19.814 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-perror.c 2023-10-10T19:29:19.815 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs.c 2023-10-10T19:29:19.816 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xalloc-die.sh 2023-10-10T19:29:19.817 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-i-ring.c 2023-10-10T19:29:19.817 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-priv-set.c 2023-10-10T19:29:19.818 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-btowc.c 2023-10-10T19:29:19.819 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sys_select.c 2023-10-10T19:29:19.819 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc.c 2023-10-10T19:29:19.820 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-strerror.c 2023-10-10T19:29:19.820 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-update-copyright.sh 2023-10-10T19:29:19.821 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-btowc2.sh 2023-10-10T19:29:19.822 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getcwd.c 2023-10-10T19:29:19.822 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/socket.c 2023-10-10T19:29:19.823 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-set-mode-acl.sh 2023-10-10T19:29:20.087 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-linkat.c 2023-10-10T19:29:20.087 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/perror.c 2023-10-10T19:29:20.088 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-wcrtomb.c 2023-10-10T19:29:20.089 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vasnprintf.c 2023-10-10T19:29:20.089 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-memrchr.c 2023-10-10T19:29:20.090 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-ftello.c 2023-10-10T19:29:20.192 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-chown.h 2023-10-10T19:29:20.193 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-stat.h 2023-10-10T19:29:20.194 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/Makefile.in 2023-10-10T19:29:20.195 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getdelim.c 2023-10-10T19:29:20.196 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vprintf-posix.sh 2023-10-10T19:29:20.196 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fseeko2.sh 2023-10-10T19:29:20.197 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-c-strcase.sh 2023-10-10T19:29:20.198 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsalign.c 2023-10-10T19:29:20.199 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fcntl-h.c 2023-10-10T19:29:20.203 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vprintf-posix.c 2023-10-10T19:29:20.207 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-rename.c 2023-10-10T19:29:20.208 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-lstat.c 2023-10-10T19:29:20.209 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-malloca.c 2023-10-10T19:29:20.209 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xstrtoul.c 2023-10-10T19:29:20.210 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-perror.sh 2023-10-10T19:29:20.210 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vasprintf-posix.c 2023-10-10T19:29:20.211 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-frexp.c 2023-10-10T19:29:20.212 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-environ.c 2023-10-10T19:29:20.213 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-stdint.c 2023-10-10T19:29:20.214 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-isnanl.h 2023-10-10T19:29:20.215 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fcntl-safer.c 2023-10-10T19:29:20.216 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-locale.c 2023-10-10T19:29:20.217 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc4.sh 2023-10-10T19:29:20.218 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-argv-iter.c 2023-10-10T19:29:20.219 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-utimens.h 2023-10-10T19:29:20.219 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xalloc-die.c 2023-10-10T19:29:20.220 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fcntl.c 2023-10-10T19:29:20.221 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-alloca-opt.c 2023-10-10T19:29:20.222 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-dirname.c 2023-10-10T19:29:20.223 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-setenv.c 2023-10-10T19:29:20.224 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-printf-frexpl.c 2023-10-10T19:29:20.225 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-atexit.sh 2023-10-10T19:29:20.226 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-stddef.c 2023-10-10T19:29:20.227 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-c-ctype.c 2023-10-10T19:29:20.228 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xstrtol.sh 2023-10-10T19:29:20.229 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getopt_long.h 2023-10-10T19:29:20.229 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-quotearg.sh 2023-10-10T19:29:20.230 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-langinfo.c 2023-10-10T19:29:20.231 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-inttypes.c 2023-10-10T19:29:20.232 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-filevercmp.c 2023-10-10T19:29:20.233 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-rmdir.c 2023-10-10T19:29:20.234 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-futimens.h 2023-10-10T19:29:20.235 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/setsockopt.c 2023-10-10T19:29:20.236 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-exclude1.sh 2023-10-10T19:29:20.237 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freopen-safer.c 2023-10-10T19:29:20.238 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getaddrinfo.c 2023-10-10T19:29:20.239 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-iconv.c 2023-10-10T19:29:20.240 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/zerosize-ptr.h 2023-10-10T19:29:20.241 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-lseek.c 2023-10-10T19:29:20.241 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-usleep.c 2023-10-10T19:29:20.242 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/gnulib.mk 2023-10-10T19:29:20.243 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-unlinkat.c 2023-10-10T19:29:20.244 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-quotearg.c 2023-10-10T19:29:20.245 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-base64.c 2023-10-10T19:29:20.246 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-frexpl.c 2023-10-10T19:29:20.247 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-gethostname.c 2023-10-10T19:29:20.248 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mkdir.h 2023-10-10T19:29:20.249 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-binary-io.sh 2023-10-10T19:29:20.250 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sys_ioctl.c 2023-10-10T19:29:20.250 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-time.c 2023-10-10T19:29:20.251 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/signature.h 2023-10-10T19:29:20.252 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-isnanf.h 2023-10-10T19:29:20.253 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-isnanf-nolibm.c 2023-10-10T19:29:20.254 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-wcwidth.c 2023-10-10T19:29:20.255 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freadseek.c 2023-10-10T19:29:20.256 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-areadlink-with-size.c 2023-10-10T19:29:20.257 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-select-stdin.c 2023-10-10T19:29:20.258 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsstr1.c 2023-10-10T19:29:20.259 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-exclude3.sh 2023-10-10T19:29:20.260 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fopen.c 2023-10-10T19:29:20.260 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-lock.c 2023-10-10T19:29:20.263 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/locale/ 2023-10-10T19:29:20.263 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/locale/fr/ 2023-10-10T19:29:20.265 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/ 2023-10-10T19:29:20.265 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.mo 2023-10-10T19:29:20.265 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.po 2023-10-10T19:29:20.267 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fpending.c 2023-10-10T19:29:20.268 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-netinet_in.c 2023-10-10T19:29:20.268 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-bitrotate.c 2023-10-10T19:29:20.269 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-c-strcasecmp.c 2023-10-10T19:29:20.270 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsstr3.c 2023-10-10T19:29:20.271 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-hash.c 2023-10-10T19:29:20.271 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-copy-acl.c 2023-10-10T19:29:20.271 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sigaction.c 2023-10-10T19:29:20.271 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xstrtoumax.c 2023-10-10T19:29:20.272 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fseterr.c 2023-10-10T19:29:20.272 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-areadlink.c 2023-10-10T19:29:20.273 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-signbit.c 2023-10-10T19:29:20.273 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-btowc1.sh 2023-10-10T19:29:20.273 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-strtod.c 2023-10-10T19:29:20.274 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-atexit.c 2023-10-10T19:29:20.274 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getdate.c 2023-10-10T19:29:20.275 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fpurge.c 2023-10-10T19:29:20.275 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-select.c 2023-10-10T19:29:20.276 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-tls.c 2023-10-10T19:29:20.277 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/sleep.c 2023-10-10T19:29:20.277 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc2.sh 2023-10-10T19:29:20.278 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-stat.c 2023-10-10T19:29:20.278 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freadptr.sh 2023-10-10T19:29:20.279 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/accept.c 2023-10-10T19:29:20.280 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-unistd.c 2023-10-10T19:29:20.280 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fopen.h 2023-10-10T19:29:20.281 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-math.c 2023-10-10T19:29:20.282 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-printf-frexp.c 2023-10-10T19:29:20.282 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vfprintf-posix.sh 2023-10-10T19:29:20.283 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-exclude.c 2023-10-10T19:29:20.283 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freading.c 2023-10-10T19:29:20.284 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getdtablesize.c 2023-10-10T19:29:20.285 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/unlinkdir.c 2023-10-10T19:29:20.285 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fflush2.c 2023-10-10T19:29:20.286 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-stat-time.c 2023-10-10T19:29:20.286 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-inet_ntop.c 2023-10-10T19:29:20.287 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-vc-list-files-git.sh 2023-10-10T19:29:20.288 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getopt.c 2023-10-10T19:29:20.288 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-alignof.c 2023-10-10T19:29:20.289 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/macros.h 2023-10-10T19:29:20.289 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-symlinkat.c 2023-10-10T19:29:20.290 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getgroups.c 2023-10-10T19:29:20.291 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-xprintf-posix.sh 2023-10-10T19:29:20.292 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fdopendir.c 2023-10-10T19:29:20.293 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sys_socket.c 2023-10-10T19:29:20.294 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbscasecmp.c 2023-10-10T19:29:20.295 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-getopt.h 2023-10-10T19:29:20.296 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs1.sh 2023-10-10T19:29:20.297 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsinit.sh 2023-10-10T19:29:20.298 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-gettimeofday.c 2023-10-10T19:29:20.298 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsstr2.c 2023-10-10T19:29:20.299 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/init.sh 2023-10-10T19:29:20.300 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-symlink.c 2023-10-10T19:29:20.300 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-utimens.c 2023-10-10T19:29:20.301 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-readlink.c 2023-10-10T19:29:20.301 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-lchown.h 2023-10-10T19:29:20.302 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-c-strncasecmp.c 2023-10-10T19:29:20.303 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-lutimens.h 2023-10-10T19:29:20.303 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-copy-acl.sh 2023-10-10T19:29:20.304 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-exclude6.sh 2023-10-10T19:29:20.305 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freadptr2.c 2023-10-10T19:29:20.305 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fflush2.sh 2023-10-10T19:29:20.306 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sys_utsname.c 2023-10-10T19:29:20.306 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs2.sh 2023-10-10T19:29:20.307 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-file-has-acl.c 2023-10-10T19:29:20.308 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-signal.c 2023-10-10T19:29:20.308 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sha1.c 2023-10-10T19:29:20.309 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freadptr2.sh 2023-10-10T19:29:20.309 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-areadlinkat.c 2023-10-10T19:29:20.310 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-unlink.h 2023-10-10T19:29:20.311 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-dup2.c 2023-10-10T19:29:20.312 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-freadptr.c 2023-10-10T19:29:20.313 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-file-has-acl.sh 2023-10-10T19:29:20.314 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-u64.c 2023-10-10T19:29:20.315 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-striconv.c 2023-10-10T19:29:20.316 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-sameacls.c 2023-10-10T19:29:20.317 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-remove.c 2023-10-10T19:29:20.318 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-fsync.c 2023-10-10T19:29:20.319 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs4.sh 2023-10-10T19:29:20.320 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_require_config_h 2023-10-10T19:29:20.321 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_system_h_headers 2023-10-10T19:29:20.322 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_prohibit_fail_0 2023-10-10T19:29:20.322 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_prohibit_magic_number_exit 2023-10-10T19:29:20.323 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/THANKS-to-translators 2023-10-10T19:29:20.324 INFO:tasks.workunit.client.0.smithi175.stdout:coreutils-8.5/.x-sc_useless_cpp_parens 2023-10-10T19:29:20.325 INFO:tasks.workunit.client.0.smithi175.stderr:tar: Exiting with failure status due to previous errors 2023-10-10T19:29:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-10T19:29:20.327 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-10T19:29:20.327 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-10T19:29:20.591 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi175 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2023-10-10T19:29:20.679 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=85ea02d3d92f4d5d92f33221cbd2708d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_46d1f59b549730ba7f93d940a3d32b94c45a4cf7/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi175 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2023-10-10T19:29:20.682 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-10T19:29:20.692 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-10T19:29:20.695 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 2 2023-10-10T19:29:20.695 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-10-10T19:29:20.696 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 25 2023-10-10T19:29:20.696 INFO:tasks.mds_thrash:done joining 2023-10-10T19:29:20.697 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-10T19:29:20.705 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-10T19:29:20.706 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-10T19:29:20.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-10T19:29:20.707 DEBUG:teuthology.orchestra.run.smithi175:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-10T19:29:22.334 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T19:29:22.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T19:29:22.335 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-10T19:29:22.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T19:29:22.350 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2023-10-10T19:29:22.406 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-10T19:29:22.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T19:29:22.408 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-10T19:29:22.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T19:29:22.483 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link delete ceph-brx 2023-10-10T19:29:22.578 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-10T19:29:22.589 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-10T19:29:22.590 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T19:29:22.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.705+0000 7fd6de5b3700 1 Processor -- start 2023-10-10T19:29:22.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.706+0000 7fd6de5b3700 1 -- start start 2023-10-10T19:29:22.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.706+0000 7fd6de5b3700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd6d80c52a0 0x7fd6d806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.706+0000 7fd6de5b3700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d8155ca0 con 0x7fd6d814cc10 2023-10-10T19:29:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.706+0000 7fd6de5b3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d8156b00 con 0x7fd6d814c840 2023-10-10T19:29:22.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.706+0000 7fd6de5b3700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d806d700 con 0x7fd6d80c52a0 2023-10-10T19:29:22.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d77fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd6d814cc10 0x7fd6d8152970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46388/0 (socket says 172.21.15.175:46388) 2023-10-10T19:29:22.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d77fe700 1 -- 172.21.15.175:0/835870557 learned_addr learned my addr 172.21.15.175:0/835870557 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:22.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6dcb50700 1 --2- 172.21.15.175:0/835870557 >> v2:172.21.15.191:3300/0 conn(0x7fd6d80c52a0 0x7fd6d806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:22.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057652872 0 0) 0x7fd6d8155ca0 con 0x7fd6d814cc10 2023-10-10T19:29:22.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c0004680 con 0x7fd6d814cc10 2023-10-10T19:29:22.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196979595 0 0) 0x7fd6c0004680 con 0x7fd6d814cc10 2023-10-10T19:29:22.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 >> v2:172.21.15.191:3300/0 conn(0x7fd6d80c52a0 msgr2=0x7fd6d806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:22.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 --2- 172.21.15.175:0/835870557 >> v2:172.21.15.191:3300/0 conn(0x7fd6d80c52a0 0x7fd6d806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 >> v1:172.21.15.191:6789/0 conn(0x7fd6d814c840 legacy=0x7fd6d80c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:22.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d8155ef0 con 0x7fd6d814cc10 2023-10-10T19:29:22.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fd6cc001470 con 0x7fd6d814cc10 2023-10-10T19:29:22.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 >> v1:172.21.15.175:6789/0 conn(0x7fd6d814cc10 legacy=0x7fd6d8152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:22.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0004b70 0x7fd6c0004f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:22.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6c0005920 0x7fd6c0005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:22.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6c00062d0 0x7fd6c000cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:22.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d806d700 con 0x7fd6c0005920 2023-10-10T19:29:22.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c000d1e0 con 0x7fd6c00062d0 2023-10-10T19:29:22.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.707+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c000d3a0 con 0x7fd6c0004b70 2023-10-10T19:29:22.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d77fe700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6c0005920 0x7fd6c0005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:22.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6dcb50700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6c00062d0 0x7fd6c000cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:22.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d7fff700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0004b70 0x7fd6c0004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:22.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d77fe700 1 -- 172.21.15.175:0/835870557 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0004b70 msgr2=0x7fd6c0004f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:22.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d77fe700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0004b70 0x7fd6c0004f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d77fe700 1 -- 172.21.15.175:0/835870557 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6c00062d0 msgr2=0x7fd6c000cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:22.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d77fe700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6c00062d0 0x7fd6c000cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d77fe700 1 -- 172.21.15.175:0/835870557 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c0004680 con 0x7fd6c0005920 2023-10-10T19:29:22.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6dcb50700 1 -- 172.21.15.175:0/835870557 reap_dead start 2023-10-10T19:29:22.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d77fe700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6c0005920 0x7fd6c0005d90 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c3c0 tx=0x7fd6cc001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfc00d33f44e875 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:22.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00cc10 con 0x7fd6c0005920 2023-10-10T19:29:22.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc00cd70 con 0x7fd6c0005920 2023-10-10T19:29:22.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.708+0000 7fd6d6ffd700 1 -- 172.21.15.175:0/835870557 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc005b50 con 0x7fd6c0005920 2023-10-10T19:29:22.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.709+0000 7fd6de5b3700 1 -- 172.21.15.175:0/835870557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6c0005920 msgr2=0x7fd6c0005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:22.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.709+0000 7fd6de5b3700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6c0005920 0x7fd6c0005d90 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c3c0 tx=0x7fd6cc001e50 comp rx=0 tx=0).stop 2023-10-10T19:29:22.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.709+0000 7fd6de5b3700 1 -- 172.21.15.175:0/835870557 shutdown_connections 2023-10-10T19:29:22.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.709+0000 7fd6de5b3700 1 --2- 172.21.15.175:0/835870557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6c0005920 0x7fd6c0005d90 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.709+0000 7fd6de5b3700 1 -- 172.21.15.175:0/835870557 >> 172.21.15.175:0/835870557 conn(0x7fd6d8068280 msgr2=0x7fd6d806af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:22.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.709+0000 7fd6de5b3700 1 -- 172.21.15.175:0/835870557 shutdown_connections 2023-10-10T19:29:22.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.709+0000 7fd6de5b3700 1 -- 172.21.15.175:0/835870557 wait complete. 2023-10-10T19:29:22.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.710+0000 7fd6de5b3700 1 Processor -- start 2023-10-10T19:29:22.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.710+0000 7fd6de5b3700 1 -- start start 2023-10-10T19:29:22.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6de5b3700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0005920 0x7fd6d8149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:22.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6de5b3700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6d81621f0 0x7fd6d8145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:22.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6de5b3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6d8150380 0x7fd6d8165770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:22.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6de5b3700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8144d80 con 0x7fd6d81621f0 2023-10-10T19:29:22.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6de5b3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8149950 con 0x7fd6d8150380 2023-10-10T19:29:22.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6de5b3700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8151dc0 con 0x7fd6c0005920 2023-10-10T19:29:22.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6d81621f0 0x7fd6d8145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:22.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6d81621f0 0x7fd6d8145830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54454/0 (socket says 172.21.15.175:54454) 2023-10-10T19:29:22.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 -- 172.21.15.175:0/2627807765 learned_addr learned my addr 172.21.15.175:0/2627807765 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:22.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d7fff700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0005920 0x7fd6d8149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:22.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6dcb50700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6d8150380 0x7fd6d8165770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:22.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 -- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0005920 msgr2=0x7fd6d8149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0005920 0x7fd6d8149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 -- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6d8150380 msgr2=0x7fd6d8165770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6d8150380 0x7fd6d8165770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 -- 172.21.15.175:0/2627807765 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d81520a0 con 0x7fd6d81621f0 2023-10-10T19:29:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6d77fe700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6d81621f0 0x7fd6d8145830 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc007cd0 tx=0x7fd6cc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6bffff700 1 -- 172.21.15.175:0/2627807765 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00cc10 con 0x7fd6d81621f0 2023-10-10T19:29:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6bffff700 1 -- 172.21.15.175:0/2627807765 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc004e50 con 0x7fd6d81621f0 2023-10-10T19:29:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d81447c0 con 0x7fd6d81621f0 2023-10-10T19:29:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6bffff700 1 -- 172.21.15.175:0/2627807765 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00cee0 con 0x7fd6d81621f0 2023-10-10T19:29:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.711+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d8148ef0 con 0x7fd6d81621f0 2023-10-10T19:29:22.734 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.712+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d81490b0 con 0x7fd6d81621f0 2023-10-10T19:29:22.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.713+0000 7fd6bffff700 1 -- 172.21.15.175:0/2627807765 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd6cc007590 con 0x7fd6d81621f0 2023-10-10T19:29:22.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.713+0000 7fd6bffff700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6c4066f30 0x7fd6c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:22.735 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.713+0000 7fd6bffff700 1 -- 172.21.15.175:0/2627807765 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fd6cc085420 con 0x7fd6d81621f0 2023-10-10T19:29:22.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.713+0000 7fd6d7fff700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6c4066f30 0x7fd6c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:22.736 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.714+0000 7fd6d7fff700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6c4066f30 0x7fd6c40693e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8000c00 tx=0x7fd6c8007d80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:22.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.717+0000 7fd6bffff700 1 -- 172.21.15.175:0/2627807765 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd6cc051e10 con 0x7fd6d81621f0 2023-10-10T19:29:22.900 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.900+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6d8067570 con 0x7fd6d81621f0 2023-10-10T19:29:22.901 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.901+0000 7fd6bffff700 1 -- 172.21.15.175:0/2627807765 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+18765 (secure 0 0 0) 0x7fd6cc017090 con 0x7fd6d81621f0 2023-10-10T19:29:22.902 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:22.902 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":132,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T19:28:33.906285+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":629,"snap_epoch":123,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:6834/840318079":"2023-10-11T19:28:33.906197+0000","172.21.15.175:6834/3457422611":"2023-10-11T19:27:28.893896+0000","172.21.15.191:6834/2914004704":"2023-10-11T19:25:43.872713+0000","172.21.15.191:6839/2393527600":"2023-10-11T19:24:58.863855+0000","172.21.15.175:6836/972555107":"2023-10-11T19:22:23.833708+0000","172.21.15.191:6838/2393527600":"2023-10-11T19:24:58.863855+0000","172.21.15.175:6833/972555107":"2023-10-11T19:22:23.833708+0000","172.21.15.175:6832/3874110823":"2023-10-11T19:00:33.566369+0000","172.21.15.175:6837/1881948738":"2023-10-11T19:11:03.690607+0000","172.21.15.191:6836/3199867402":"2023-10-11T18:59:08.550308+0000","172.21.15.191:6838/1598171176":"2023-10-11T19:03:13.596165+0000","172.21.15.191:6835/3199867402":"2023-10-11T18:59:08.550308+0000","172.21.15.175:6835/2966961140":"2023-10-11T18:55:38.509904+0000","172.21.15.175:6834/3874110823":"2023-10-11T19:00:33.566369+0000","172.21.15.191:6838/246221790":"2023-10-11T18:53:33.481931+0000","172.21.15.191:6837/2914004704":"2023-10-11T19:25:43.872713+0000","172.21.15.191:6837/1851345295":"2023-10-11T19:21:03.818037+0000","172.21.15.191:6837/3870390806":"2023-10-11T18:57:08.527329+0000","172.21.15.191:6839/246221790":"2023-10-11T18:53:33.481931+0000","172.21.15.191:6834/1851345295":"2023-10-11T19:21:03.818037+0000","172.21.15.175:6832/3457422611":"2023-10-11T19:27:28.893896+0000","172.21.15.191:6835/3007475551":"2023-10-11T18:47:43.409173+0000","172.21.15.175:6836/647425682":"2023-10-11T18:52:23.468892+0000","172.21.15.175:6834/688515950":"2023-10-11T18:49:33.433082+0000","172.21.15.175:6833/647425682":"2023-10-11T18:52:23.468892+0000","172.21.15.175:6837/2966961140":"2023-10-11T18:55:38.509904+0000","172.21.15.175:6832/688515950":"2023-10-11T18:49:33.433082+0000","172.21.15.191:6835/1945747421":"2023-10-11T19:05:28.622478+0000","172.21.15.191:6839/1598171176":"2023-10-11T19:03:13.596165+0000","172.21.15.191:6836/1945747421":"2023-10-11T19:05:28.622478+0000","172.21.15.175:6833/2375829435":"2023-10-11T19:07:23.643743+0000","172.21.15.191:6838/1910071809":"2023-10-11T19:08:28.655326+0000","172.21.15.191:6837/840318079":"2023-10-11T19:28:33.906197+0000","172.21.15.191:6839/1910071809":"2023-10-11T19:08:28.655326+0000","172.21.15.191:6834/824552134":"2023-10-11T19:09:53.672299+0000","172.21.15.175:6832/1872591144":"2023-10-11T19:14:43.733216+0000","172.21.15.175:6835/3849771254":"2023-10-11T19:23:38.848907+0000","172.21.15.175:6834/1872591144":"2023-10-11T19:14:43.733216+0000","172.21.15.175:6837/3849771254":"2023-10-11T19:23:38.848907+0000","172.21.15.191:6837/824552134":"2023-10-11T19:09:53.672299+0000","172.21.15.191:6834/3870390806":"2023-10-11T18:57:08.527329+0000","172.21.15.175:6835/1881948738":"2023-10-11T19:11:03.690607+0000","172.21.15.191:6835/4084912544":"2023-10-11T19:17:13.770352+0000","172.21.15.191:6836/3007475551":"2023-10-11T18:47:43.409173+0000","172.21.15.175:6836/2375829435":"2023-10-11T19:07:23.643743+0000","172.21.15.191:6836/4084912544":"2023-10-11T19:17:13.770352+0000","172.21.15.191:6835/62572703":"2023-10-11T19:13:48.722600+0000","172.21.15.191:6836/62572703":"2023-10-11T19:13:48.722600+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T19:29:22.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6c4066f30 msgr2=0x7fd6c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:22.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6c4066f30 0x7fd6c40693e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8000c00 tx=0x7fd6c8007d80 comp rx=0 tx=0).stop 2023-10-10T19:29:22.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6d81621f0 msgr2=0x7fd6d8145830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:22.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6d81621f0 0x7fd6d8145830 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc007cd0 tx=0x7fd6cc015000 comp rx=0 tx=0).stop 2023-10-10T19:29:22.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 shutdown_connections 2023-10-10T19:29:22.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fd6c4066f30 0x7fd6c40693e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6d8150380 0x7fd6d8165770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6c0005920 0x7fd6d8149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 --2- 172.21.15.175:0/2627807765 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd6d81621f0 0x7fd6d8145830 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:22.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 >> 172.21.15.175:0/2627807765 conn(0x7fd6d8068280 msgr2=0x7fd6d80c0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:22.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 shutdown_connections 2023-10-10T19:29:22.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:22.903+0000 7fd6de5b3700 1 -- 172.21.15.175:0/2627807765 wait complete. 2023-10-10T19:29:22.915 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-10T19:29:22.916 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T19:29:23.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.056+0000 7f9af8b23700 1 Processor -- start 2023-10-10T19:29:23.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.056+0000 7f9af8b23700 1 -- start start 2023-10-10T19:29:23.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af8b23700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9af40b2390 0x7f9af4155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af8b23700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40aff60 0x7f9af40b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af8b23700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af41574b0 con 0x7f9af40b1fc0 2023-10-10T19:29:23.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af8b23700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af4150770 con 0x7f9af40b2390 2023-10-10T19:29:23.061 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af8b23700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40afad0 con 0x7f9af40aff60 2023-10-10T19:29:23.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af2d9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40aff60 0x7f9af40b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af1d9b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9af40b2390 0x7f9af4155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.062 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af1d9b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9af40b2390 0x7f9af4155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:60188/0 (socket says 172.21.15.175:60188) 2023-10-10T19:29:23.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af259c700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f9af40b1fc0 0x7f9af4068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:54558/0 (socket says 172.21.15.175:54558) 2023-10-10T19:29:23.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af2d9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40aff60 0x7f9af40b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:43280/0 (socket says 172.21.15.175:43280) 2023-10-10T19:29:23.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.057+0000 7f9af1d9b700 1 -- 172.21.15.175:0/1219425025 learned_addr learned my addr 172.21.15.175:0/1219425025 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:23.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af1d9b700 1 -- 172.21.15.175:0/1219425025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40aff60 msgr2=0x7f9af40b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:23.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af1d9b700 1 --2- 172.21.15.175:0/1219425025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40aff60 0x7f9af40b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af1d9b700 1 -- 172.21.15.175:0/1219425025 >> v1:172.21.15.191:6789/0 conn(0x7f9af40b1fc0 legacy=0x7f9af4068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af159a700 1 -- 172.21.15.175:0/1219425025 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122893557 0 0) 0x7f9af41574b0 con 0x7f9af40b1fc0 2023-10-10T19:29:23.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af1d9b700 1 -- 172.21.15.175:0/1219425025 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40af4c0 con 0x7f9af40b2390 2023-10-10T19:29:23.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af2d9d700 1 --2- 172.21.15.175:0/1219425025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40aff60 0x7f9af40b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:23.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af1d9b700 1 --2- 172.21.15.175:0/1219425025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40b2390 0x7f9af4155fb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0099b0 tx=0x7f9adc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d9f22617e9a41e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af159a700 1 -- 172.21.15.175:0/1219425025 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc009ea0 con 0x7f9af40b2390 2023-10-10T19:29:23.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af159a700 1 -- 172.21.15.175:0/1219425025 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc00b8f0 con 0x7f9af40b2390 2023-10-10T19:29:23.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.058+0000 7f9af159a700 1 -- 172.21.15.175:0/1219425025 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00bc90 con 0x7f9af40b2390 2023-10-10T19:29:23.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.059+0000 7f9af8b23700 1 -- 172.21.15.175:0/1219425025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40b2390 msgr2=0x7f9af4155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.059+0000 7f9af8b23700 1 --2- 172.21.15.175:0/1219425025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40b2390 0x7f9af4155fb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0099b0 tx=0x7f9adc009250 comp rx=0 tx=0).stop 2023-10-10T19:29:23.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.059+0000 7f9af8b23700 1 -- 172.21.15.175:0/1219425025 shutdown_connections 2023-10-10T19:29:23.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.059+0000 7f9af8b23700 1 --2- 172.21.15.175:0/1219425025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40aff60 0x7f9af40b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.059+0000 7f9af8b23700 1 --2- 172.21.15.175:0/1219425025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40b2390 0x7f9af4155fb0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.059+0000 7f9af8b23700 1 -- 172.21.15.175:0/1219425025 >> 172.21.15.175:0/1219425025 conn(0x7f9af4063230 msgr2=0x7f9af4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:23.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.059+0000 7f9af8b23700 1 -- 172.21.15.175:0/1219425025 shutdown_connections 2023-10-10T19:29:23.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.059+0000 7f9af8b23700 1 -- 172.21.15.175:0/1219425025 wait complete. 2023-10-10T19:29:23.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.060+0000 7f9af8b23700 1 Processor -- start 2023-10-10T19:29:23.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.060+0000 7f9af8b23700 1 -- start start 2023-10-10T19:29:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af8b23700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40aff60 0x7f9af40caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af8b23700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40b1fc0 0x7f9af40cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af8b23700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9af40b2390 0x7f9af414d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af8b23700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40cb100 con 0x7f9af40b2390 2023-10-10T19:29:23.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af8b23700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40cc150 con 0x7f9af40aff60 2023-10-10T19:29:23.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af8b23700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af4159dc0 con 0x7f9af40b1fc0 2023-10-10T19:29:23.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9af40b2390 0x7f9af414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9af40b2390 0x7f9af414d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54462/0 (socket says 172.21.15.175:54462) 2023-10-10T19:29:23.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 -- 172.21.15.175:0/755395047 learned_addr learned my addr 172.21.15.175:0/755395047 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:23.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af259c700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40aff60 0x7f9af40caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af1d9b700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40b1fc0 0x7f9af40cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.075 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 -- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40b1fc0 msgr2=0x7f9af40cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40b1fc0 0x7f9af40cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 -- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40aff60 msgr2=0x7f9af40caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40aff60 0x7f9af40caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 -- 172.21.15.175:0/755395047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec009420 con 0x7f9af40b2390 2023-10-10T19:29:23.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af1d9b700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40b1fc0 0x7f9af40cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:23.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af2d9d700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9af40b2390 0x7f9af414d970 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00ecb0 tx=0x7f9aec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9aea7fc700 1 -- 172.21.15.175:0/755395047 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00c920 con 0x7f9af40b2390 2023-10-10T19:29:23.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9aea7fc700 1 -- 172.21.15.175:0/755395047 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec00ca80 con 0x7f9af40b2390 2023-10-10T19:29:23.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc0094f0 con 0x7f9af40b2390 2023-10-10T19:29:23.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9aea7fc700 1 -- 172.21.15.175:0/755395047 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00ce40 con 0x7f9af40b2390 2023-10-10T19:29:23.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.061+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af4159ae0 con 0x7f9af40b2390 2023-10-10T19:29:23.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.062+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af40cc300 con 0x7f9af40b2390 2023-10-10T19:29:23.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.063+0000 7f9aea7fc700 1 -- 172.21.15.175:0/755395047 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9aec013680 con 0x7f9af40b2390 2023-10-10T19:29:23.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.063+0000 7f9aea7fc700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ad4066f30 0x7f9ad40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.063+0000 7f9aea7fc700 1 -- 172.21.15.175:0/755395047 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f9aec084e40 con 0x7f9af40b2390 2023-10-10T19:29:23.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.063+0000 7f9af259c700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ad4066f30 0x7f9ad40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.064+0000 7f9af259c700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ad4066f30 0x7f9ad40693e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4001560 tx=0x7f9ae4001b00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.067+0000 7f9aea7fc700 1 -- 172.21.15.175:0/755395047 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9af40cc300 con 0x7f9af40b2390 2023-10-10T19:29:23.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.250+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9af40cc300 con 0x7f9ad4066f30 2023-10-10T19:29:23.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.259+0000 7f9aea7fc700 1 -- 172.21.15.175:0/755395047 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294172 (secure 0 0 0) 0x7f9af40cc300 con 0x7f9ad4066f30 2023-10-10T19:29:23.263 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:23.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ad4066f30 msgr2=0x7f9ad40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ad4066f30 0x7f9ad40693e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4001560 tx=0x7f9ae4001b00 comp rx=0 tx=0).stop 2023-10-10T19:29:23.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9af40b2390 msgr2=0x7f9af414d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9af40b2390 0x7f9af414d970 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00ecb0 tx=0x7f9aec0091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 shutdown_connections 2023-10-10T19:29:23.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9ad4066f30 0x7f9ad40693e0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9af40b2390 0x7f9af414d970 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9af40aff60 0x7f9af40caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 --2- 172.21.15.175:0/755395047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9af40b1fc0 0x7f9af40cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 >> 172.21.15.175:0/755395047 conn(0x7f9af4063230 msgr2=0x7f9af415a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:23.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 shutdown_connections 2023-10-10T19:29:23.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.262+0000 7f9af8b23700 1 -- 172.21.15.175:0/755395047 wait complete. 2023-10-10T19:29:23.270 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T19:29:23.274 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":1454,"stamp":"2023-10-10T19:29:21.817054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371836467,"num_objects":2019,"num_object_clones":0,"num_object_copies":4038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":7382,"num_read_kb":3484532,"num_write":62406,"num_write_kb":2973288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":338,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79214,"ondisk_log_size":79214,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2685672,"kb_used_data":250024,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752289048,"statfs":{"total":773094113280,"available":770343985152,"internally_reserved":0,"allocated":256024576,"data_stored":750760178,"data_compressed":110781050,"data_compressed_allocated":137502720,"data_compressed_original":645881856,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1436897,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":1564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003628"},"pg_stats":[{"pgid":"3.3f","version":"131'943","reported_seq":3563,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.259330+0000","last_change":"2023-10-10T19:16:54.796013+0000","last_active":"2023-10-10T19:28:34.259330+0000","last_peered":"2023-10-10T19:28:34.259330+0000","last_clean":"2023-10-10T19:28:34.259330+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T19:28:34.259330+0000","last_undegraded":"2023-10-10T19:28:34.259330+0000","last_fullsized":"2023-10-10T19:28:34.259330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017995507000000001,"stat_sum":{"num_bytes":795566,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":62,"num_read_kb":8068,"num_write":667,"num_write_kb":16726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"131'993","reported_seq":3612,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.284503+0000","last_change":"2023-10-10T19:16:54.920511+0000","last_active":"2023-10-10T19:29:04.284503+0000","last_peered":"2023-10-10T19:29:04.284503+0000","last_clean":"2023-10-10T19:29:04.284503+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T19:29:04.284503+0000","last_undegraded":"2023-10-10T19:29:04.284503+0000","last_fullsized":"2023-10-10T19:29:04.284503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083569260000000006,"stat_sum":{"num_bytes":832331,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":62,"num_read_kb":28805,"num_write":733,"num_write_kb":46104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"131'937","reported_seq":3564,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.912051+0000","last_change":"2023-10-10T19:16:54.980353+0000","last_active":"2023-10-10T19:28:33.912051+0000","last_peered":"2023-10-10T19:28:33.912051+0000","last_clean":"2023-10-10T19:28:33.912051+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T19:28:33.912051+0000","last_undegraded":"2023-10-10T19:28:33.912051+0000","last_fullsized":"2023-10-10T19:28:33.912051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":937,"log_dups_size":0,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011353221,"stat_sum":{"num_bytes":419659,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":65,"num_read_kb":20680,"num_write":679,"num_write_kb":24555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"131'941","reported_seq":3555,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.942186+0000","last_change":"2023-10-10T19:16:55.412877+0000","last_active":"2023-10-10T19:28:24.942186+0000","last_peered":"2023-10-10T19:28:24.942186+0000","last_clean":"2023-10-10T19:28:24.942186+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T19:28:24.942186+0000","last_undegraded":"2023-10-10T19:28:24.942186+0000","last_fullsized":"2023-10-10T19:28:24.942186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088633549999999998,"stat_sum":{"num_bytes":476985,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":62,"num_read_kb":30965,"num_write":662,"num_write_kb":37535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"131'971","reported_seq":3601,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.424759+0000","last_change":"2023-10-10T19:16:54.790962+0000","last_active":"2023-10-10T19:29:04.424759+0000","last_peered":"2023-10-10T19:29:04.424759+0000","last_clean":"2023-10-10T19:29:04.424759+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T19:29:04.424759+0000","last_undegraded":"2023-10-10T19:29:04.424759+0000","last_fullsized":"2023-10-10T19:29:04.424759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016189688000000001,"stat_sum":{"num_bytes":142623,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":75,"num_read_kb":17848,"num_write":693,"num_write_kb":31284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"131'918","reported_seq":3544,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704394+0000","last_change":"2023-10-10T19:16:55.414924+0000","last_active":"2023-10-10T19:28:34.704394+0000","last_peered":"2023-10-10T19:28:34.704394+0000","last_clean":"2023-10-10T19:28:34.704394+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T19:28:34.704394+0000","last_undegraded":"2023-10-10T19:28:34.704394+0000","last_fullsized":"2023-10-10T19:28:34.704394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010631753000000001,"stat_sum":{"num_bytes":567917,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":63,"num_read_kb":20939,"num_write":659,"num_write_kb":24217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"131'968","reported_seq":3589,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078564+0000","last_change":"2023-10-10T19:16:54.789065+0000","last_active":"2023-10-10T19:29:04.078564+0000","last_peered":"2023-10-10T19:29:04.078564+0000","last_clean":"2023-10-10T19:29:04.078564+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T19:29:04.078564+0000","last_undegraded":"2023-10-10T19:29:04.078564+0000","last_fullsized":"2023-10-10T19:29:04.078564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014667316999999999,"stat_sum":{"num_bytes":1084188,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":53600,"num_write":676,"num_write_kb":52841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"131'852","reported_seq":3490,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077662+0000","last_change":"2023-10-10T19:16:54.870456+0000","last_active":"2023-10-10T19:29:04.077662+0000","last_peered":"2023-10-10T19:29:04.077662+0000","last_clean":"2023-10-10T19:29:04.077662+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T19:29:04.077662+0000","last_undegraded":"2023-10-10T19:29:04.077662+0000","last_fullsized":"2023-10-10T19:29:04.077662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":852,"log_dups_size":0,"ondisk_log_size":852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012402231,"stat_sum":{"num_bytes":449638,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":58,"num_read_kb":28110,"num_write":635,"num_write_kb":35676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"131'959","reported_seq":3586,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078384+0000","last_change":"2023-10-10T19:16:54.795632+0000","last_active":"2023-10-10T19:29:04.078384+0000","last_peered":"2023-10-10T19:29:04.078384+0000","last_clean":"2023-10-10T19:29:04.078384+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T19:29:04.078384+0000","last_undegraded":"2023-10-10T19:29:04.078384+0000","last_fullsized":"2023-10-10T19:29:04.078384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017699890999999999,"stat_sum":{"num_bytes":30143,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":76,"num_read_kb":37435,"num_write":691,"num_write_kb":40783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"131'935","reported_seq":3561,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.401310+0000","last_change":"2023-10-10T19:16:54.783622+0000","last_active":"2023-10-10T19:29:04.401310+0000","last_peered":"2023-10-10T19:29:04.401310+0000","last_clean":"2023-10-10T19:29:04.401310+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T19:29:04.401310+0000","last_undegraded":"2023-10-10T19:29:04.401310+0000","last_fullsized":"2023-10-10T19:29:04.401310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079196160000000008,"stat_sum":{"num_bytes":128300,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":70,"num_read_kb":16825,"num_write":682,"num_write_kb":23273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"131'955","reported_seq":3604,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.424201+0000","last_change":"2023-10-10T19:16:54.784745+0000","last_active":"2023-10-10T19:29:04.424201+0000","last_peered":"2023-10-10T19:29:04.424201+0000","last_clean":"2023-10-10T19:29:04.424201+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T19:29:04.424201+0000","last_undegraded":"2023-10-10T19:29:04.424201+0000","last_fullsized":"2023-10-10T19:29:04.424201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088422029999999999,"stat_sum":{"num_bytes":198054,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":82,"num_read_kb":46829,"num_write":704,"num_write_kb":53046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"131'944","reported_seq":3579,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.484037+0000","last_change":"2023-10-10T19:16:54.989857+0000","last_active":"2023-10-10T19:29:04.484037+0000","last_peered":"2023-10-10T19:29:04.484037+0000","last_clean":"2023-10-10T19:29:04.484037+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T19:29:04.484037+0000","last_undegraded":"2023-10-10T19:29:04.484037+0000","last_fullsized":"2023-10-10T19:29:04.484037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078589550000000008,"stat_sum":{"num_bytes":271785,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":75,"num_read_kb":57532,"num_write":692,"num_write_kb":73449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"131'896","reported_seq":3526,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.275163+0000","last_change":"2023-10-10T19:16:54.872408+0000","last_active":"2023-10-10T19:29:04.275163+0000","last_peered":"2023-10-10T19:29:04.275163+0000","last_clean":"2023-10-10T19:29:04.275163+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T19:29:04.275163+0000","last_undegraded":"2023-10-10T19:29:04.275163+0000","last_fullsized":"2023-10-10T19:29:04.275163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014204069,"stat_sum":{"num_bytes":4542310,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":73,"num_read_kb":32208,"num_write":631,"num_write_kb":44582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"131'902","reported_seq":3526,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.393937+0000","last_change":"2023-10-10T19:16:54.978714+0000","last_active":"2023-10-10T19:29:04.393937+0000","last_peered":"2023-10-10T19:29:04.393937+0000","last_clean":"2023-10-10T19:29:04.393937+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T19:29:04.393937+0000","last_undegraded":"2023-10-10T19:29:04.393937+0000","last_fullsized":"2023-10-10T19:29:04.393937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097965510000000006,"stat_sum":{"num_bytes":686329,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":63,"num_read_kb":28420,"num_write":651,"num_write_kb":31132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"131'934","reported_seq":3567,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.271178+0000","last_change":"2023-10-10T19:16:54.917427+0000","last_active":"2023-10-10T19:29:04.271178+0000","last_peered":"2023-10-10T19:29:04.271178+0000","last_clean":"2023-10-10T19:29:04.271178+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T19:29:04.271178+0000","last_undegraded":"2023-10-10T19:29:04.271178+0000","last_fullsized":"2023-10-10T19:29:04.271178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059774370000000004,"stat_sum":{"num_bytes":1294620,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":77,"num_read_kb":52747,"num_write":659,"num_write_kb":72992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"131'977","reported_seq":3611,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077966+0000","last_change":"2023-10-10T19:16:55.412885+0000","last_active":"2023-10-10T19:29:04.077966+0000","last_peered":"2023-10-10T19:29:04.077966+0000","last_clean":"2023-10-10T19:29:04.077966+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T19:29:04.077966+0000","last_undegraded":"2023-10-10T19:29:04.077966+0000","last_fullsized":"2023-10-10T19:29:04.077966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087668889999999999,"stat_sum":{"num_bytes":446378,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":80,"num_read_kb":42995,"num_write":700,"num_write_kb":52933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"131'882","reported_seq":3536,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.705589+0000","last_change":"2023-10-10T19:16:55.417126+0000","last_active":"2023-10-10T19:28:34.705589+0000","last_peered":"2023-10-10T19:28:34.705589+0000","last_clean":"2023-10-10T19:28:34.705589+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T19:28:34.705589+0000","last_undegraded":"2023-10-10T19:28:34.705589+0000","last_fullsized":"2023-10-10T19:28:34.705589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012085218,"stat_sum":{"num_bytes":1843052,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":72,"num_read_kb":40372,"num_write":665,"num_write_kb":48809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"131'826","reported_seq":3498,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.912020+0000","last_change":"2023-10-10T19:16:54.780747+0000","last_active":"2023-10-10T19:28:33.912020+0000","last_peered":"2023-10-10T19:28:33.912020+0000","last_clean":"2023-10-10T19:28:33.912020+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T19:28:33.912020+0000","last_undegraded":"2023-10-10T19:28:33.912020+0000","last_fullsized":"2023-10-10T19:28:33.912020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00594963,"stat_sum":{"num_bytes":909657,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":72,"num_read_kb":44243,"num_write":651,"num_write_kb":60901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"131'1049","reported_seq":3646,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.588062+0000","last_change":"2023-10-10T19:16:54.792335+0000","last_active":"2023-10-10T19:29:04.588062+0000","last_peered":"2023-10-10T19:29:04.588062+0000","last_clean":"2023-10-10T19:29:04.588062+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T19:29:04.588062+0000","last_undegraded":"2023-10-10T19:29:04.588062+0000","last_fullsized":"2023-10-10T19:29:04.588062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01601,"stat_sum":{"num_bytes":624343,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":73,"num_read_kb":41187,"num_write":717,"num_write_kb":49713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"131'836","reported_seq":1001,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:18.796925+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T19:28:18.796925+0000","last_peered":"2023-10-10T19:28:18.796925+0000","last_clean":"2023-10-10T19:28:18.796925+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T19:28:18.796925+0000","last_undegraded":"2023-10-10T19:28:18.796925+0000","last_fullsized":"2023-10-10T19:28:18.796925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389252,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":82,"num_read_kb":61516,"num_write":780,"num_write_kb":21018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"131'974","reported_seq":3595,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.491287+0000","last_change":"2023-10-10T19:16:54.877116+0000","last_active":"2023-10-10T19:29:04.491287+0000","last_peered":"2023-10-10T19:29:04.491287+0000","last_clean":"2023-10-10T19:29:04.491287+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T19:29:04.491287+0000","last_undegraded":"2023-10-10T19:29:04.491287+0000","last_fullsized":"2023-10-10T19:29:04.491287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018086130999999998,"stat_sum":{"num_bytes":63039,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":63,"num_read_kb":13169,"num_write":717,"num_write_kb":31098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"131'759","reported_seq":903,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:04.229561+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T19:28:04.229561+0000","last_peered":"2023-10-10T19:28:04.229561+0000","last_clean":"2023-10-10T19:28:04.229561+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T19:28:04.229561+0000","last_undegraded":"2023-10-10T19:28:04.229561+0000","last_fullsized":"2023-10-10T19:28:04.229561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":29,"num_read_kb":69691,"num_write":709,"num_write_kb":16925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"131'971","reported_seq":3613,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078264+0000","last_change":"2023-10-10T19:16:54.978352+0000","last_active":"2023-10-10T19:29:04.078264+0000","last_peered":"2023-10-10T19:29:04.078264+0000","last_clean":"2023-10-10T19:29:04.078264+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T19:29:04.078264+0000","last_undegraded":"2023-10-10T19:29:04.078264+0000","last_fullsized":"2023-10-10T19:29:04.078264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093729599999999996,"stat_sum":{"num_bytes":213282,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":74,"num_read_kb":34573,"num_write":722,"num_write_kb":44041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"132'1694","reported_seq":1870,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:10.134874+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T19:29:10.134874+0000","last_peered":"2023-10-10T19:29:10.134874+0000","last_clean":"2023-10-10T19:29:10.134874+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T19:29:10.134874+0000","last_undegraded":"2023-10-10T19:29:10.134874+0000","last_fullsized":"2023-10-10T19:29:10.134874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1694,"log_dups_size":0,"ondisk_log_size":1694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":114,"num_read_kb":59511,"num_write":1903,"num_write_kb":22655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"131'936","reported_seq":3530,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.496246+0000","last_change":"2023-10-10T19:16:54.780201+0000","last_active":"2023-10-10T19:29:04.496246+0000","last_peered":"2023-10-10T19:29:04.496246+0000","last_clean":"2023-10-10T19:29:04.496246+0000","last_became_active":"2023-10-10T18:46:01.103860+0000","last_became_peered":"2023-10-10T18:46:01.103860+0000","last_unstale":"2023-10-10T19:29:04.496246+0000","last_undegraded":"2023-10-10T19:29:04.496246+0000","last_fullsized":"2023-10-10T19:29:04.496246+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:17.975194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054668909999999998,"stat_sum":{"num_bytes":80680,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":48,"num_read_kb":295,"num_write":639,"num_write_kb":7460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"130'750","reported_seq":903,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.970799+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T19:27:29.970799+0000","last_peered":"2023-10-10T19:27:29.970799+0000","last_clean":"2023-10-10T19:27:29.970799+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T19:27:29.970799+0000","last_undegraded":"2023-10-10T19:27:29.970799+0000","last_fullsized":"2023-10-10T19:27:29.970799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389162,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":55,"num_read_kb":36831,"num_write":683,"num_write_kb":16858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"131'970","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.274296+0000","last_change":"2023-10-10T19:16:55.416075+0000","last_active":"2023-10-10T19:29:04.274296+0000","last_peered":"2023-10-10T19:29:04.274296+0000","last_clean":"2023-10-10T19:29:04.274296+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T19:29:04.274296+0000","last_undegraded":"2023-10-10T19:29:04.274296+0000","last_fullsized":"2023-10-10T19:29:04.274296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011753095,"stat_sum":{"num_bytes":80024,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":81,"num_read_kb":35882,"num_write":686,"num_write_kb":42928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"131'612","reported_seq":760,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.922994+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T19:28:38.922994+0000","last_peered":"2023-10-10T19:28:38.922994+0000","last_clean":"2023-10-10T19:28:38.922994+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T19:28:38.922994+0000","last_undegraded":"2023-10-10T19:28:38.922994+0000","last_fullsized":"2023-10-10T19:28:38.922994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14118582,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":31,"num_read_kb":65583,"num_write":554,"num_write_kb":18318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"131'893","reported_seq":3521,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.393580+0000","last_change":"2023-10-10T19:16:54.987567+0000","last_active":"2023-10-10T19:29:04.393580+0000","last_peered":"2023-10-10T19:29:04.393580+0000","last_clean":"2023-10-10T19:29:04.393580+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T19:29:04.393580+0000","last_undegraded":"2023-10-10T19:29:04.393580+0000","last_fullsized":"2023-10-10T19:29:04.393580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062644010000000002,"stat_sum":{"num_bytes":763528,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":59,"num_read_kb":21826,"num_write":671,"num_write_kb":37641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"129'775","reported_seq":1107,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.932043+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T19:28:38.932043+0000","last_peered":"2023-10-10T19:28:38.932043+0000","last_clean":"2023-10-10T19:28:38.932043+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T19:28:38.932043+0000","last_undegraded":"2023-10-10T19:28:38.932043+0000","last_fullsized":"2023-10-10T19:28:38.932043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111936,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":251,"num_read_kb":81147,"num_write":700,"num_write_kb":25323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"131'819","reported_seq":3467,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.924982+0000","last_change":"2023-10-10T19:16:54.984860+0000","last_active":"2023-10-10T19:28:24.924982+0000","last_peered":"2023-10-10T19:28:24.924982+0000","last_clean":"2023-10-10T19:28:24.924982+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T19:28:24.924982+0000","last_undegraded":"2023-10-10T19:28:24.924982+0000","last_fullsized":"2023-10-10T19:28:24.924982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015269961,"stat_sum":{"num_bytes":449137,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":60,"num_read_kb":24046,"num_write":605,"num_write_kb":30663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"130'1494","reported_seq":1889,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.913899+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T19:28:38.913899+0000","last_peered":"2023-10-10T19:28:38.913899+0000","last_clean":"2023-10-10T19:28:38.913899+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T19:28:38.913899+0000","last_undegraded":"2023-10-10T19:28:38.913899+0000","last_fullsized":"2023-10-10T19:28:38.913899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4727705,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":295,"num_read_kb":15405,"num_write":1447,"num_write_kb":10055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"131'980","reported_seq":3623,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.268565+0000","last_change":"2023-10-10T19:16:54.874108+0000","last_active":"2023-10-10T19:29:04.268565+0000","last_peered":"2023-10-10T19:29:04.268565+0000","last_clean":"2023-10-10T19:29:04.268565+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T19:29:04.268565+0000","last_undegraded":"2023-10-10T19:29:04.268565+0000","last_fullsized":"2023-10-10T19:29:04.268565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015767953000000001,"stat_sum":{"num_bytes":541499,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":81,"num_read_kb":19092,"num_write":733,"num_write_kb":30243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"131'494","reported_seq":639,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704437+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T19:28:34.704437+0000","last_peered":"2023-10-10T19:28:34.704437+0000","last_clean":"2023-10-10T19:28:34.704437+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T19:28:34.704437+0000","last_undegraded":"2023-10-10T19:28:34.704437+0000","last_fullsized":"2023-10-10T19:28:34.704437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583466,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":65566,"num_write":451,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"131'1005","reported_seq":3644,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.395988+0000","last_change":"2023-10-10T19:16:54.918258+0000","last_active":"2023-10-10T19:29:04.395988+0000","last_peered":"2023-10-10T19:29:04.395988+0000","last_clean":"2023-10-10T19:29:04.395988+0000","last_became_active":"2023-10-10T18:46:00.606391+0000","last_became_peered":"2023-10-10T18:46:00.606391+0000","last_unstale":"2023-10-10T19:29:04.395988+0000","last_undegraded":"2023-10-10T19:29:04.395988+0000","last_fullsized":"2023-10-10T19:29:04.395988+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:57.665869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066939039999999997,"stat_sum":{"num_bytes":631366,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":80,"num_read_kb":42231,"num_write":729,"num_write_kb":52765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"132'274","reported_seq":415,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:15.078147+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T19:29:15.078147+0000","last_peered":"2023-10-10T19:29:15.078147+0000","last_clean":"2023-10-10T19:29:15.078147+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T19:29:15.078147+0000","last_undegraded":"2023-10-10T19:29:15.078147+0000","last_fullsized":"2023-10-10T19:29:15.078147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":34,"num_read_kb":54,"num_write":225,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"131'948","reported_seq":3583,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.705242+0000","last_change":"2023-10-10T19:16:55.413821+0000","last_active":"2023-10-10T19:28:34.705242+0000","last_peered":"2023-10-10T19:28:34.705242+0000","last_clean":"2023-10-10T19:28:34.705242+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T19:28:34.705242+0000","last_undegraded":"2023-10-10T19:28:34.705242+0000","last_fullsized":"2023-10-10T19:28:34.705242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096962509999999995,"stat_sum":{"num_bytes":519488,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":63,"num_read_kb":26256,"num_write":717,"num_write_kb":45994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"132'617","reported_seq":975,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:10.134272+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T19:29:10.134272+0000","last_peered":"2023-10-10T19:29:10.134272+0000","last_clean":"2023-10-10T19:29:10.134272+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T19:29:10.134272+0000","last_undegraded":"2023-10-10T19:29:10.134272+0000","last_fullsized":"2023-10-10T19:29:10.134272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7932987,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":262,"num_read_kb":30418,"num_write":555,"num_write_kb":12258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"131'906","reported_seq":3529,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.400410+0000","last_change":"2023-10-10T19:16:54.984649+0000","last_active":"2023-10-10T19:29:04.400410+0000","last_peered":"2023-10-10T19:29:04.400410+0000","last_clean":"2023-10-10T19:29:04.400410+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T19:29:04.400410+0000","last_undegraded":"2023-10-10T19:29:04.400410+0000","last_fullsized":"2023-10-10T19:29:04.400410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015222243,"stat_sum":{"num_bytes":277071,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":64,"num_read_kb":38795,"num_write":628,"num_write_kb":46072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"131'207","reported_seq":349,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.761037+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T19:28:34.761037+0000","last_peered":"2023-10-10T19:28:34.761037+0000","last_clean":"2023-10-10T19:28:34.761037+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T19:28:34.761037+0000","last_undegraded":"2023-10-10T19:28:34.761037+0000","last_fullsized":"2023-10-10T19:28:34.761037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":42,"num_read_kb":61,"num_write":171,"num_write_kb":225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"131'858","reported_seq":3490,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.320810+0000","last_change":"2023-10-10T19:16:54.989921+0000","last_active":"2023-10-10T19:29:04.320810+0000","last_peered":"2023-10-10T19:29:04.320810+0000","last_clean":"2023-10-10T19:29:04.320810+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T19:29:04.320810+0000","last_undegraded":"2023-10-10T19:29:04.320810+0000","last_fullsized":"2023-10-10T19:29:04.320810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080742209999999995,"stat_sum":{"num_bytes":82158,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":56,"num_read_kb":23103,"num_write":645,"num_write_kb":46075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"132'264","reported_seq":413,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.276956+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T19:29:04.276956+0000","last_peered":"2023-10-10T19:29:04.276956+0000","last_clean":"2023-10-10T19:29:04.276956+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T19:29:04.276956+0000","last_undegraded":"2023-10-10T19:29:04.276956+0000","last_fullsized":"2023-10-10T19:29:04.276956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194948,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":59,"num_read_kb":16449,"num_write":212,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"131'930","reported_seq":3560,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.595503+0000","last_change":"2023-10-10T19:16:54.793178+0000","last_active":"2023-10-10T19:29:04.595503+0000","last_peered":"2023-10-10T19:29:04.595503+0000","last_clean":"2023-10-10T19:29:04.595503+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T19:29:04.595503+0000","last_undegraded":"2023-10-10T19:29:04.595503+0000","last_fullsized":"2023-10-10T19:29:04.595503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015774979000000001,"stat_sum":{"num_bytes":110666,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":71,"num_read_kb":24956,"num_write":663,"num_write_kb":31333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"125'818","reported_seq":1254,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.911380+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T19:28:33.911380+0000","last_peered":"2023-10-10T19:28:33.911380+0000","last_clean":"2023-10-10T19:28:33.911380+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T19:28:33.911380+0000","last_undegraded":"2023-10-10T19:28:33.911380+0000","last_fullsized":"2023-10-10T19:28:33.911380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227398,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":359,"num_read_kb":46031,"num_write":769,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"131'911","reported_seq":3542,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078167+0000","last_change":"2023-10-10T19:16:54.981687+0000","last_active":"2023-10-10T19:29:04.078167+0000","last_peered":"2023-10-10T19:29:04.078167+0000","last_clean":"2023-10-10T19:29:04.078167+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T19:29:04.078167+0000","last_undegraded":"2023-10-10T19:29:04.078167+0000","last_fullsized":"2023-10-10T19:29:04.078167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01251216,"stat_sum":{"num_bytes":89325,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":62,"num_read_kb":21731,"num_write":674,"num_write_kb":32887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"128'713","reported_seq":873,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.927273+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T19:28:38.927273+0000","last_peered":"2023-10-10T19:28:38.927273+0000","last_clean":"2023-10-10T19:28:38.927273+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T19:28:38.927273+0000","last_undegraded":"2023-10-10T19:28:38.927273+0000","last_fullsized":"2023-10-10T19:28:38.927273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415841,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":58,"num_read_kb":76305,"num_write":648,"num_write_kb":19602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"131'951","reported_seq":3576,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.258906+0000","last_change":"2023-10-10T19:16:54.791948+0000","last_active":"2023-10-10T19:28:34.258906+0000","last_peered":"2023-10-10T19:28:34.258906+0000","last_clean":"2023-10-10T19:28:34.258906+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T19:28:34.258906+0000","last_undegraded":"2023-10-10T19:28:34.258906+0000","last_fullsized":"2023-10-10T19:28:34.258906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017093574,"stat_sum":{"num_bytes":514832,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":73,"num_read_kb":45017,"num_write":683,"num_write_kb":44061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"131'407","reported_seq":563,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.926155+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T19:28:38.926155+0000","last_peered":"2023-10-10T19:28:38.926155+0000","last_clean":"2023-10-10T19:28:38.926155+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T19:28:38.926155+0000","last_undegraded":"2023-10-10T19:28:38.926155+0000","last_fullsized":"2023-10-10T19:28:38.926155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":54,"num_read_kb":84439,"num_write":360,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"131'818","reported_seq":3450,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.403900+0000","last_change":"2023-10-10T19:16:54.782792+0000","last_active":"2023-10-10T19:29:04.403900+0000","last_peered":"2023-10-10T19:29:04.403900+0000","last_clean":"2023-10-10T19:29:04.403900+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T19:29:04.403900+0000","last_undegraded":"2023-10-10T19:29:04.403900+0000","last_fullsized":"2023-10-10T19:29:04.403900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075683979999999996,"stat_sum":{"num_bytes":1065275,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":52,"num_read_kb":24653,"num_write":599,"num_write_kb":45347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"131'804","reported_seq":983,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.911148+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T19:28:33.911148+0000","last_peered":"2023-10-10T19:28:33.911148+0000","last_clean":"2023-10-10T19:28:33.911148+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T19:28:33.911148+0000","last_undegraded":"2023-10-10T19:28:33.911148+0000","last_fullsized":"2023-10-10T19:28:33.911148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":804,"log_dups_size":0,"ondisk_log_size":804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":113,"num_read_kb":78302,"num_write":734,"num_write_kb":25074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"131'894","reported_seq":3531,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.913337+0000","last_change":"2023-10-10T19:16:54.790805+0000","last_active":"2023-10-10T19:28:24.913337+0000","last_peered":"2023-10-10T19:28:24.913337+0000","last_clean":"2023-10-10T19:28:24.913337+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T19:28:24.913337+0000","last_undegraded":"2023-10-10T19:28:24.913337+0000","last_fullsized":"2023-10-10T19:28:24.913337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01616896,"stat_sum":{"num_bytes":43538,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":67,"num_read_kb":45230,"num_write":650,"num_write_kb":49050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"131'604","reported_seq":741,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:19.022228+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T19:28:19.022228+0000","last_peered":"2023-10-10T19:28:19.022228+0000","last_clean":"2023-10-10T19:28:19.022228+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T19:28:19.022228+0000","last_undegraded":"2023-10-10T19:28:19.022228+0000","last_fullsized":"2023-10-10T19:28:19.022228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":23,"num_read_kb":12341,"num_write":746,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"131'918","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.406226+0000","last_change":"2023-10-10T19:16:54.793503+0000","last_active":"2023-10-10T19:29:04.406226+0000","last_peered":"2023-10-10T19:29:04.406226+0000","last_clean":"2023-10-10T19:29:04.406226+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T19:29:04.406226+0000","last_undegraded":"2023-10-10T19:29:04.406226+0000","last_fullsized":"2023-10-10T19:29:04.406226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015882960000000002,"stat_sum":{"num_bytes":652546,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":57,"num_read_kb":20900,"num_write":659,"num_write_kb":24656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"132'150","reported_seq":287,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:15.166343+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T19:29:15.166343+0000","last_peered":"2023-10-10T19:29:15.166343+0000","last_clean":"2023-10-10T19:29:15.166343+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T19:29:15.166343+0000","last_undegraded":"2023-10-10T19:29:15.166343+0000","last_fullsized":"2023-10-10T19:29:15.166343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311985,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":45123,"num_write":106,"num_write_kb":9377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"131'1026","reported_seq":3634,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.495833+0000","last_change":"2023-10-10T19:16:55.415960+0000","last_active":"2023-10-10T19:29:04.495833+0000","last_peered":"2023-10-10T19:29:04.495833+0000","last_clean":"2023-10-10T19:29:04.495833+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T19:29:04.495833+0000","last_undegraded":"2023-10-10T19:29:04.495833+0000","last_fullsized":"2023-10-10T19:29:04.495833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011491507999999999,"stat_sum":{"num_bytes":340961,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":71,"num_read_kb":20716,"num_write":714,"num_write_kb":35169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"130'341","reported_seq":489,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704918+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T19:28:34.704918+0000","last_peered":"2023-10-10T19:28:34.704918+0000","last_clean":"2023-10-10T19:28:34.704918+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T19:28:34.704918+0000","last_undegraded":"2023-10-10T19:28:34.704918+0000","last_fullsized":"2023-10-10T19:28:34.704918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":73794,"num_write":284,"num_write_kb":20986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"131'933","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758752+0000","last_change":"2023-10-10T19:16:54.988658+0000","last_active":"2023-10-10T19:28:34.758752+0000","last_peered":"2023-10-10T19:28:34.758752+0000","last_clean":"2023-10-10T19:28:34.758752+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T19:28:34.758752+0000","last_undegraded":"2023-10-10T19:28:34.758752+0000","last_fullsized":"2023-10-10T19:28:34.758752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071091339999999996,"stat_sum":{"num_bytes":108666,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":93,"num_read_kb":25139,"num_write":704,"num_write_kb":35589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"131'931","reported_seq":3556,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.447963+0000","last_change":"2023-10-10T19:16:54.919574+0000","last_active":"2023-10-10T19:29:04.447963+0000","last_peered":"2023-10-10T19:29:04.447963+0000","last_clean":"2023-10-10T19:29:04.447963+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T19:29:04.447963+0000","last_undegraded":"2023-10-10T19:29:04.447963+0000","last_fullsized":"2023-10-10T19:29:04.447963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077260940000000002,"stat_sum":{"num_bytes":89807,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":61,"num_read_kb":19697,"num_write":681,"num_write_kb":26677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"132'685","reported_seq":1109,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:18.043195+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T19:29:18.043195+0000","last_peered":"2023-10-10T19:29:18.043195+0000","last_clean":"2023-10-10T19:29:18.043195+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T19:29:18.043195+0000","last_undegraded":"2023-10-10T19:29:18.043195+0000","last_fullsized":"2023-10-10T19:29:18.043195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9108221,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":314,"num_read_kb":40841,"num_write":609,"num_write_kb":9447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'875","reported_seq":3545,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.281760+0000","last_change":"2023-10-10T19:16:54.875665+0000","last_active":"2023-10-10T19:29:04.281760+0000","last_peered":"2023-10-10T19:29:04.281760+0000","last_clean":"2023-10-10T19:29:04.281760+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T19:29:04.281760+0000","last_undegraded":"2023-10-10T19:29:04.281760+0000","last_fullsized":"2023-10-10T19:29:04.281760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017091021000000001,"stat_sum":{"num_bytes":821658,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":88,"num_read_kb":19324,"num_write":658,"num_write_kb":26590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"131'137","reported_seq":294,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.142425+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T19:28:39.142425+0000","last_peered":"2023-10-10T19:28:39.142425+0000","last_clean":"2023-10-10T19:28:39.142425+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T19:28:39.142425+0000","last_undegraded":"2023-10-10T19:28:39.142425+0000","last_fullsized":"2023-10-10T19:28:39.142425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388756,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":36939,"num_write":74,"num_write_kb":8306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":274,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.970643+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T19:27:29.970643+0000","last_peered":"2023-10-10T19:27:29.970643+0000","last_clean":"2023-10-10T19:27:29.970643+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T19:27:29.970643+0000","last_undegraded":"2023-10-10T19:27:29.970643+0000","last_fullsized":"2023-10-10T19:27:29.970643+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"131'887","reported_seq":3510,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078025+0000","last_change":"2023-10-10T19:16:54.979966+0000","last_active":"2023-10-10T19:29:04.078025+0000","last_peered":"2023-10-10T19:29:04.078025+0000","last_clean":"2023-10-10T19:29:04.078025+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T19:29:04.078025+0000","last_undegraded":"2023-10-10T19:29:04.078025+0000","last_fullsized":"2023-10-10T19:29:04.078025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010927930000000001,"stat_sum":{"num_bytes":3326937,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":55,"num_read_kb":20626,"num_write":650,"num_write_kb":30635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"131'973","reported_seq":1124,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:40.006368+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T19:28:40.006368+0000","last_peered":"2023-10-10T19:28:40.006368+0000","last_clean":"2023-10-10T19:28:40.006368+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T19:28:40.006368+0000","last_undegraded":"2023-10-10T19:28:40.006368+0000","last_fullsized":"2023-10-10T19:28:40.006368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":38,"num_read_kb":89926,"num_write":912,"num_write_kb":29260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"128'825","reported_seq":978,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.259463+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T19:28:34.259463+0000","last_peered":"2023-10-10T19:28:34.259463+0000","last_clean":"2023-10-10T19:28:34.259463+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T19:28:34.259463+0000","last_undegraded":"2023-10-10T19:28:34.259463+0000","last_fullsized":"2023-10-10T19:28:34.259463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":45,"num_read_kb":45594,"num_write":764,"num_write_kb":12838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"131'1007","reported_seq":3620,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077312+0000","last_change":"2023-10-10T19:16:54.919294+0000","last_active":"2023-10-10T19:29:04.077312+0000","last_peered":"2023-10-10T19:29:04.077312+0000","last_clean":"2023-10-10T19:29:04.077312+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T19:29:04.077312+0000","last_undegraded":"2023-10-10T19:29:04.077312+0000","last_fullsized":"2023-10-10T19:29:04.077312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075449510000000003,"stat_sum":{"num_bytes":492311,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":65,"num_read_kb":49387,"num_write":712,"num_write_kb":52375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'275","reported_seq":431,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:49.017495+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T19:28:49.017495+0000","last_peered":"2023-10-10T19:28:49.017495+0000","last_clean":"2023-10-10T19:28:49.017495+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T19:28:49.017495+0000","last_undegraded":"2023-10-10T19:28:49.017495+0000","last_fullsized":"2023-10-10T19:28:49.017495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":81,"num_read_kb":20675,"num_write":475,"num_write_kb":4740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"131'939","reported_seq":3565,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.285360+0000","last_change":"2023-10-10T19:16:54.920290+0000","last_active":"2023-10-10T19:29:04.285360+0000","last_peered":"2023-10-10T19:29:04.285360+0000","last_clean":"2023-10-10T19:29:04.285360+0000","last_became_active":"2023-10-10T18:46:01.267092+0000","last_became_peered":"2023-10-10T18:46:01.267092+0000","last_unstale":"2023-10-10T19:29:04.285360+0000","last_undegraded":"2023-10-10T19:29:04.285360+0000","last_fullsized":"2023-10-10T19:29:04.285360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:46:17.491022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083609700000000006,"stat_sum":{"num_bytes":2123745,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":68,"num_read_kb":33081,"num_write":679,"num_write_kb":41978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"132'154","reported_seq":301,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:41.419018+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T19:28:41.419018+0000","last_peered":"2023-10-10T19:28:41.419018+0000","last_clean":"2023-10-10T19:28:41.419018+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T19:28:41.419018+0000","last_undegraded":"2023-10-10T19:28:41.419018+0000","last_fullsized":"2023-10-10T19:28:41.419018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":58,"num_read_kb":8269,"num_write":92,"num_write_kb":8284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"131'867","reported_seq":3527,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.288386+0000","last_change":"2023-10-10T19:16:54.981885+0000","last_active":"2023-10-10T19:29:04.288386+0000","last_peered":"2023-10-10T19:29:04.288386+0000","last_clean":"2023-10-10T19:29:04.288386+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T19:29:04.288386+0000","last_undegraded":"2023-10-10T19:29:04.288386+0000","last_fullsized":"2023-10-10T19:29:04.288386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721872,"stat_sum":{"num_bytes":440031,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":85,"num_read_kb":32652,"num_write":635,"num_write_kb":36138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"131'228","reported_seq":371,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.557527+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T19:28:39.557527+0000","last_peered":"2023-10-10T19:28:39.557527+0000","last_clean":"2023-10-10T19:28:39.557527+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T19:28:39.557527+0000","last_undegraded":"2023-10-10T19:28:39.557527+0000","last_fullsized":"2023-10-10T19:28:39.557527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":57389,"num_write":173,"num_write_kb":12477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"131'931","reported_seq":3570,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758691+0000","last_change":"2023-10-10T19:16:54.990974+0000","last_active":"2023-10-10T19:28:34.758691+0000","last_peered":"2023-10-10T19:28:34.758691+0000","last_clean":"2023-10-10T19:28:34.758691+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T19:28:34.758691+0000","last_undegraded":"2023-10-10T19:28:34.758691+0000","last_fullsized":"2023-10-10T19:28:34.758691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089059480000000003,"stat_sum":{"num_bytes":796993,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":67,"num_read_kb":26489,"num_write":680,"num_write_kb":31111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"132'369","reported_seq":521,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.547536+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T19:29:04.547536+0000","last_peered":"2023-10-10T19:29:04.547536+0000","last_clean":"2023-10-10T19:29:04.547536+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T19:29:04.547536+0000","last_undegraded":"2023-10-10T19:29:04.547536+0000","last_fullsized":"2023-10-10T19:29:04.547536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":70,"num_read_kb":20656,"num_write":331,"num_write_kb":8737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"131'882","reported_seq":3516,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.406702+0000","last_change":"2023-10-10T19:16:54.870214+0000","last_active":"2023-10-10T19:29:04.406702+0000","last_peered":"2023-10-10T19:29:04.406702+0000","last_clean":"2023-10-10T19:29:04.406702+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T19:29:04.406702+0000","last_undegraded":"2023-10-10T19:29:04.406702+0000","last_fullsized":"2023-10-10T19:29:04.406702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012052205,"stat_sum":{"num_bytes":579663,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":63,"num_read_kb":40064,"num_write":647,"num_write_kb":56806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"132'1301","reported_seq":1456,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:18.043527+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T19:29:18.043527+0000","last_peered":"2023-10-10T19:29:18.043527+0000","last_clean":"2023-10-10T19:29:18.043527+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T19:29:18.043527+0000","last_undegraded":"2023-10-10T19:29:18.043527+0000","last_fullsized":"2023-10-10T19:29:18.043527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1301,"log_dups_size":0,"ondisk_log_size":1301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14953265,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":48,"num_read_kb":53317,"num_write":1709,"num_write_kb":24642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"131'871","reported_seq":3503,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.271899+0000","last_change":"2023-10-10T19:16:54.874295+0000","last_active":"2023-10-10T19:29:04.271899+0000","last_peered":"2023-10-10T19:29:04.271899+0000","last_clean":"2023-10-10T19:29:04.271899+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T19:29:04.271899+0000","last_undegraded":"2023-10-10T19:29:04.271899+0000","last_fullsized":"2023-10-10T19:29:04.271899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015827366999999998,"stat_sum":{"num_bytes":60469,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":58,"num_read_kb":28965,"num_write":636,"num_write_kb":32020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"132'1200","reported_seq":1354,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.409456+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T19:29:04.409456+0000","last_peered":"2023-10-10T19:29:04.409456+0000","last_clean":"2023-10-10T19:29:04.409456+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T19:29:04.409456+0000","last_undegraded":"2023-10-10T19:29:04.409456+0000","last_fullsized":"2023-10-10T19:29:04.409456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":49,"num_read_kb":89747,"num_write":1124,"num_write_kb":25429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"131'914","reported_seq":3542,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.885234+0000","last_change":"2023-10-10T19:16:54.781695+0000","last_active":"2023-10-10T19:28:24.885234+0000","last_peered":"2023-10-10T19:28:24.885234+0000","last_clean":"2023-10-10T19:28:24.885234+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T19:28:24.885234+0000","last_undegraded":"2023-10-10T19:28:24.885234+0000","last_fullsized":"2023-10-10T19:28:24.885234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065052069999999998,"stat_sum":{"num_bytes":415973,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":64,"num_read_kb":10527,"num_write":665,"num_write_kb":20000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"130'311","reported_seq":451,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.683868+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T19:27:29.683868+0000","last_peered":"2023-10-10T19:27:29.683868+0000","last_clean":"2023-10-10T19:27:29.683868+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T19:27:29.683868+0000","last_undegraded":"2023-10-10T19:27:29.683868+0000","last_fullsized":"2023-10-10T19:27:29.683868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194858,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":40985,"num_write":231,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"131'896","reported_seq":3546,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.595223+0000","last_change":"2023-10-10T19:16:54.794460+0000","last_active":"2023-10-10T19:29:04.595223+0000","last_peered":"2023-10-10T19:29:04.595223+0000","last_clean":"2023-10-10T19:29:04.595223+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T19:29:04.595223+0000","last_undegraded":"2023-10-10T19:29:04.595223+0000","last_fullsized":"2023-10-10T19:29:04.595223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016889265000000001,"stat_sum":{"num_bytes":535544,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":67,"num_read_kb":24822,"num_write":684,"num_write_kb":31728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"132'288","reported_seq":432,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.760584+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T19:28:34.760584+0000","last_peered":"2023-10-10T19:28:34.760584+0000","last_clean":"2023-10-10T19:28:34.760584+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T19:28:34.760584+0000","last_undegraded":"2023-10-10T19:28:34.760584+0000","last_fullsized":"2023-10-10T19:28:34.760584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388678,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":27,"num_read_kb":50704,"num_write":218,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"131'948","reported_seq":3575,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.280142+0000","last_change":"2023-10-10T19:16:55.417128+0000","last_active":"2023-10-10T19:29:04.280142+0000","last_peered":"2023-10-10T19:29:04.280142+0000","last_clean":"2023-10-10T19:29:04.280142+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T19:29:04.280142+0000","last_undegraded":"2023-10-10T19:29:04.280142+0000","last_fullsized":"2023-10-10T19:29:04.280142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012585610000000001,"stat_sum":{"num_bytes":184409,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":60,"num_read_kb":19964,"num_write":698,"num_write_kb":27162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"128'310","reported_seq":458,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.683824+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T19:27:29.683824+0000","last_peered":"2023-10-10T19:27:29.683824+0000","last_clean":"2023-10-10T19:27:29.683824+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T19:27:29.683824+0000","last_undegraded":"2023-10-10T19:27:29.683824+0000","last_fullsized":"2023-10-10T19:27:29.683824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":49,"num_read_kb":57404,"num_write":260,"num_write_kb":8464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"131'999","reported_seq":3628,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.407478+0000","last_change":"2023-10-10T19:16:54.794658+0000","last_active":"2023-10-10T19:29:04.407478+0000","last_peered":"2023-10-10T19:29:04.407478+0000","last_clean":"2023-10-10T19:29:04.407478+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T19:29:04.407478+0000","last_undegraded":"2023-10-10T19:29:04.407478+0000","last_fullsized":"2023-10-10T19:29:04.407478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016837566000000002,"stat_sum":{"num_bytes":339858,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":67,"num_read_kb":28062,"num_write":742,"num_write_kb":23663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"131'891","reported_seq":3551,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077900+0000","last_change":"2023-10-10T19:16:54.782474+0000","last_active":"2023-10-10T19:29:04.077900+0000","last_peered":"2023-10-10T19:29:04.077900+0000","last_clean":"2023-10-10T19:29:04.077900+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T19:29:04.077900+0000","last_undegraded":"2023-10-10T19:29:04.077900+0000","last_fullsized":"2023-10-10T19:29:04.077900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074078640000000001,"stat_sum":{"num_bytes":492374,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":77,"num_read_kb":25056,"num_write":669,"num_write_kb":32074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"131'973","reported_seq":3594,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.287531+0000","last_change":"2023-10-10T19:16:54.779827+0000","last_active":"2023-10-10T19:29:04.287531+0000","last_peered":"2023-10-10T19:29:04.287531+0000","last_clean":"2023-10-10T19:29:04.287531+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T19:29:04.287531+0000","last_undegraded":"2023-10-10T19:29:04.287531+0000","last_fullsized":"2023-10-10T19:29:04.287531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050938800000000003,"stat_sum":{"num_bytes":392421,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":69,"num_read_kb":25013,"num_write":694,"num_write_kb":37199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"131'956","reported_seq":3603,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.272570+0000","last_change":"2023-10-10T19:16:54.781092+0000","last_active":"2023-10-10T19:29:04.272570+0000","last_peered":"2023-10-10T19:29:04.272570+0000","last_clean":"2023-10-10T19:29:04.272570+0000","last_became_active":"2023-10-10T18:46:00.606420+0000","last_became_peered":"2023-10-10T18:46:00.606420+0000","last_unstale":"2023-10-10T19:29:04.272570+0000","last_undegraded":"2023-10-10T19:29:04.272570+0000","last_fullsized":"2023-10-10T19:29:04.272570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:47.189801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064370249999999999,"stat_sum":{"num_bytes":69276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":90,"num_read_kb":45570,"num_write":703,"num_write_kb":52466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"131'929","reported_seq":3555,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.288777+0000","last_change":"2023-10-10T19:16:54.983512+0000","last_active":"2023-10-10T19:29:04.288777+0000","last_peered":"2023-10-10T19:29:04.288777+0000","last_clean":"2023-10-10T19:29:04.288777+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T19:29:04.288777+0000","last_undegraded":"2023-10-10T19:29:04.288777+0000","last_fullsized":"2023-10-10T19:29:04.288777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014096529,"stat_sum":{"num_bytes":398978,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":70,"num_read_kb":16093,"num_write":658,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"131'895","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.417096+0000","last_change":"2023-10-10T19:16:54.875626+0000","last_active":"2023-10-10T19:29:04.417096+0000","last_peered":"2023-10-10T19:29:04.417096+0000","last_clean":"2023-10-10T19:29:04.417096+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T19:29:04.417096+0000","last_undegraded":"2023-10-10T19:29:04.417096+0000","last_fullsized":"2023-10-10T19:29:04.417096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016808101999999998,"stat_sum":{"num_bytes":587511,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":68,"num_read_kb":48841,"num_write":661,"num_write_kb":55904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"131'962","reported_seq":3564,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.760146+0000","last_change":"2023-10-10T19:16:54.918525+0000","last_active":"2023-10-10T19:28:34.760146+0000","last_peered":"2023-10-10T19:28:34.760146+0000","last_clean":"2023-10-10T19:28:34.760146+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T19:28:34.760146+0000","last_undegraded":"2023-10-10T19:28:34.760146+0000","last_fullsized":"2023-10-10T19:28:34.760146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068654069999999996,"stat_sum":{"num_bytes":680481,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":59,"num_read_kb":41116,"num_write":666,"num_write_kb":48352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"131'952","reported_seq":3574,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.272886+0000","last_change":"2023-10-10T19:16:54.783685+0000","last_active":"2023-10-10T19:29:04.272886+0000","last_peered":"2023-10-10T19:29:04.272886+0000","last_clean":"2023-10-10T19:29:04.272886+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T19:29:04.272886+0000","last_undegraded":"2023-10-10T19:29:04.272886+0000","last_fullsized":"2023-10-10T19:29:04.272886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081260870000000006,"stat_sum":{"num_bytes":4687487,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":70,"num_read_kb":15871,"num_write":681,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"131'922","reported_seq":3558,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.267637+0000","last_change":"2023-10-10T19:16:54.877012+0000","last_active":"2023-10-10T19:29:04.267637+0000","last_peered":"2023-10-10T19:29:04.267637+0000","last_clean":"2023-10-10T19:29:04.267637+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T19:29:04.267637+0000","last_undegraded":"2023-10-10T19:29:04.267637+0000","last_fullsized":"2023-10-10T19:29:04.267637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":922,"log_dups_size":0,"ondisk_log_size":922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017981724000000001,"stat_sum":{"num_bytes":798509,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":72,"num_read_kb":44513,"num_write":661,"num_write_kb":47223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"131'898","reported_seq":3541,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.478410+0000","last_change":"2023-10-10T19:16:55.415035+0000","last_active":"2023-10-10T19:29:04.478410+0000","last_peered":"2023-10-10T19:29:04.478410+0000","last_clean":"2023-10-10T19:29:04.478410+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T19:29:04.478410+0000","last_undegraded":"2023-10-10T19:29:04.478410+0000","last_fullsized":"2023-10-10T19:29:04.478410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010756557999999999,"stat_sum":{"num_bytes":615982,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":71,"num_read_kb":21097,"num_write":676,"num_write_kb":25400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"131'878","reported_seq":3515,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758569+0000","last_change":"2023-10-10T19:16:54.872190+0000","last_active":"2023-10-10T19:28:34.758569+0000","last_peered":"2023-10-10T19:28:34.758569+0000","last_clean":"2023-10-10T19:28:34.758569+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T19:28:34.758569+0000","last_undegraded":"2023-10-10T19:28:34.758569+0000","last_fullsized":"2023-10-10T19:28:34.758569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014012482999999999,"stat_sum":{"num_bytes":662827,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":69,"num_read_kb":49258,"num_write":627,"num_write_kb":56876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"131'935","reported_seq":3568,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.290882+0000","last_change":"2023-10-10T19:16:55.413906+0000","last_active":"2023-10-10T19:29:04.290882+0000","last_peered":"2023-10-10T19:29:04.290882+0000","last_clean":"2023-10-10T19:29:04.290882+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T19:29:04.290882+0000","last_undegraded":"2023-10-10T19:29:04.290882+0000","last_fullsized":"2023-10-10T19:29:04.290882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097177140000000006,"stat_sum":{"num_bytes":515125,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":59,"num_read_kb":20609,"num_write":704,"num_write_kb":28747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"131'873","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.482135+0000","last_change":"2023-10-10T19:16:54.983331+0000","last_active":"2023-10-10T19:29:04.482135+0000","last_peered":"2023-10-10T19:29:04.482135+0000","last_clean":"2023-10-10T19:29:04.482135+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T19:29:04.482135+0000","last_undegraded":"2023-10-10T19:29:04.482135+0000","last_fullsized":"2023-10-10T19:29:04.482135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013935637000000001,"stat_sum":{"num_bytes":335358,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":51992,"num_write":654,"num_write_kb":56542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"131'994","reported_seq":3636,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077991+0000","last_change":"2023-10-10T19:16:54.784919+0000","last_active":"2023-10-10T19:29:04.077991+0000","last_peered":"2023-10-10T19:29:04.077991+0000","last_clean":"2023-10-10T19:29:04.077991+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T19:29:04.077991+0000","last_undegraded":"2023-10-10T19:29:04.077991+0000","last_fullsized":"2023-10-10T19:29:04.077991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088155260000000006,"stat_sum":{"num_bytes":221031,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":37312,"num_write":732,"num_write_kb":49119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":43035717,"num_objects":1543,"num_object_clones":0,"num_object_copies":3086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":4408,"num_read_kb":1953419,"num_write":43238,"num_write_kb":2509746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":63954944,"data_stored":86071434,"data_compressed":25961548,"data_compressed_allocated":28180480,"data_compressed_original":58130432,"omap_allocated":0,"internal_metadata":0},"log_size":59428,"ondisk_log_size":59428,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":328210382,"num_objects":474,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":2858,"num_read_kb":1531013,"num_write":19009,"num_write_kb":461126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":338,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179290112,"data_stored":656420764,"data_compressed":84754974,"data_compressed_allocated":109232128,"data_compressed_original":586604544,"omap_allocated":0,"internal_metadata":0},"log_size":19669,"ondisk_log_size":19669,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575383,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346716,"kb_used_data":45332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94025124,"statfs":{"total":96636764160,"available":96281726976,"internally_reserved":0,"allocated":46419968,"data_stored":131329189,"data_compressed":19664789,"data_compressed_allocated":24576000,"data_compressed_original":111382528,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.53700000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.78400000000000003,"15min":1.018},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.32100000000000001,"5min":0.316,"15min":0.25},"max":{"1min":0.76500000000000001,"5min":0.80700000000000005,"15min":2.153},"last":0.36799999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.33100000000000002,"5min":0.318,"15min":0.26200000000000001},"max":{"1min":0.79500000000000004,"5min":0.82499999999999996,"15min":1.802},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":2.2170000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.623,"15min":0.61199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83899999999999997,"5min":2.871,"15min":2.871},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63400000000000001,"15min":0.63800000000000001},"min":{"1min":0.48299999999999998,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":7.6200000000000001},"last":0.55400000000000005}]},{"osd":3,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.874,"15min":1.0349999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.60099999999999998},"min":{"1min":0.375,"5min":0.373,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":7.6349999999999998},"last":0.502}]},{"osd":4,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.36599999999999999,"5min":0.307,"15min":0.28000000000000003},"max":{"1min":0.64000000000000001,"5min":0.72499999999999998,"15min":1.089},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.33800000000000002,"5min":0.32600000000000001,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.9419999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":0.629,"5min":0.752,"15min":1.71},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59299999999999997,"15min":0.57199999999999995},"min":{"1min":0.41799999999999998,"5min":0.378,"15min":0.311},"max":{"1min":0.80500000000000005,"5min":0.85099999999999998,"15min":1.177},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.31},"max":{"1min":0.76100000000000001,"5min":0.78000000000000003,"15min":0.872},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.33900000000000002,"15min":0.29699999999999999},"max":{"1min":0.72299999999999998,"5min":0.78100000000000003,"15min":2.2559999999999998},"last":0.57299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575382,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332164,"kb_used_data":34876,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94039676,"statfs":{"total":96636764160,"available":96296628224,"internally_reserved":0,"allocated":35713024,"data_stored":90587605,"data_compressed":11613979,"data_compressed_allocated":14393344,"data_compressed_original":70787072,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.31},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.216},"max":{"1min":0.71799999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42499999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.77200000000000002,"5min":0.77600000000000002,"15min":0.85299999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.58399999999999996,"15min":0.56599999999999995},"min":{"1min":0.46700000000000003,"5min":0.29599999999999999,"15min":0.222},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86799999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Tue Oct 10 19:28:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58199999999999996,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.312,"15min":0.30499999999999999},"max":{"1min":0.69899999999999995,"5min":0.77800000000000002,"15min":0.90500000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56899999999999995,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29999999999999999},"max":{"1min":0.71299999999999997,"5min":0.76500000000000001,"15min":0.97599999999999998},"last":0.51300000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57399999999999995,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76600000000000001,"5min":0.82799999999999996,"15min":0.88200000000000001},"last":0.628},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.35999999999999999},"max":{"1min":0.73599999999999999,"5min":0.85699999999999998,"15min":0.95099999999999996},"last":0.73199999999999998}]},{"osd":4,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.46400000000000002},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.248},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.749},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.52500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.73599999999999999,"15min":0.879},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.39700000000000002,"5min":0.30599999999999999,"15min":0.26400000000000001},"max":{"1min":0.63100000000000001,"5min":0.76800000000000002,"15min":0.99299999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.38200000000000001,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.88900000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.38,"5min":0.376,"15min":0.27700000000000002},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.86199999999999999},"last":0.60899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575381,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339956,"kb_used_data":30380,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94031884,"statfs":{"total":96636764160,"available":96288649216,"internally_reserved":0,"allocated":31109120,"data_stored":109263664,"data_compressed":16514475,"data_compressed_allocated":20389888,"data_compressed_original":99950592,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.314,"15min":0.313},"min":{"1min":0.187,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.432,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.505},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.66700000000000004,"15min":0.71699999999999997},"last":0.626}]},{"osd":2,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34599999999999997,"15min":0.35199999999999998},"min":{"1min":0.23799999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55200000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25,"15min":0.19900000000000001},"max":{"1min":0.54000000000000004,"5min":0.64900000000000002,"15min":0.73899999999999999},"last":0.56699999999999995}]},{"osd":3,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.45300000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.32600000000000001,"15min":0.27700000000000002},"max":{"1min":0.59699999999999998,"5min":0.72899999999999998,"15min":0.84299999999999997},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":1.175},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58699999999999997,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.87,"5min":0.87,"15min":2.2519999999999998},"last":0.80800000000000005}]},{"osd":5,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.76700000000000002,"15min":0.90900000000000003},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":1.0009999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.40500000000000003,"15min":0.37},"max":{"1min":0.83199999999999996,"5min":0.86199999999999999,"15min":1.038},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.123},"last":0.96699999999999997}]},{"osd":7,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60299999999999998,"15min":0.62},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":5.2720000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":5.2599999999999998},"last":1.0389999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575383,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335236,"kb_used_data":29756,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94036604,"statfs":{"total":96636764160,"available":96293482496,"internally_reserved":0,"allocated":30470144,"data_stored":81989013,"data_compressed":12938492,"data_compressed_allocated":16048128,"data_compressed_original":69398528,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.35799999999999998,"15min":0.35699999999999998},"min":{"1min":0.245,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":0.47699999999999998,"5min":0.57399999999999995,"15min":0.66500000000000004},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42499999999999999,"15min":0.41999999999999998},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.20300000000000001},"max":{"1min":0.60299999999999998,"5min":0.66000000000000003,"15min":0.66500000000000004},"last":0.35299999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35999999999999999,"15min":0.373},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.49299999999999999,"5min":0.57799999999999996,"15min":2.028},"last":0.308},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54900000000000004,"5min":0.61299999999999999,"15min":0.77300000000000002},"last":0.27300000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":0.54000000000000004,"5min":0.59699999999999998,"15min":0.79100000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.32500000000000001,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":0.70499999999999996,"5min":0.70599999999999996,"15min":0.80300000000000005},"last":0.375}]},{"osd":4,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.373,"15min":0.26000000000000001},"max":{"1min":0.80200000000000005,"5min":0.97599999999999998,"15min":1.9610000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.38400000000000001},"max":{"1min":0.82599999999999996,"5min":0.999,"15min":0.999},"last":0.53700000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.55100000000000005,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.79600000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.437},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.626,"15min":0.623},"min":{"1min":0.51600000000000001,"5min":0.38600000000000001,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.873,"15min":0.96799999999999997},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.52200000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":0.75800000000000001,"5min":0.96299999999999997,"15min":2.0019999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.621,"15min":0.623},"min":{"1min":0.501,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":0.84999999999999998,"5min":0.94099999999999995,"15min":0.95399999999999996},"last":0.43099999999999999}]},{"osd":7,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.40400000000000003,"15min":0.39100000000000001},"max":{"1min":0.77000000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41799999999999998},"max":{"1min":0.879,"5min":0.89500000000000002,"15min":2.0539999999999998},"last":0.498}]}]},{"osd":2,"up_from":13,"seq":55834575383,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318528,"kb_used_data":17144,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94053312,"statfs":{"total":96636764160,"available":96310591488,"internally_reserved":0,"allocated":17555456,"data_stored":56587147,"data_compressed":8367326,"data_compressed_allocated":10403840,"data_compressed_original":50835456,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34399999999999997,"15min":0.33500000000000002},"min":{"1min":0.224,"5min":0.219,"15min":0.186},"max":{"1min":0.45000000000000001,"5min":0.55400000000000005,"15min":0.60799999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.53500000000000003,"5min":0.65100000000000002,"15min":0.67500000000000004},"last":0.38}]},{"osd":1,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20999999999999999},"max":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.65700000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.73999999999999999},"last":0.443}]},{"osd":3,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.24399999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.42499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.23899999999999999},"max":{"1min":0.59099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.76900000000000002},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.26600000000000001},"max":{"1min":0.751,"5min":0.95599999999999996,"15min":48.729999999999997},"last":0.499},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.25900000000000001},"max":{"1min":0.74099999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.48599999999999999}]},{"osd":5,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63,"15min":0.628},"min":{"1min":0.436,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":1.02,"15min":1.02},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.32400000000000001},"max":{"1min":0.81000000000000005,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.78900000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":1.0760000000000001,"15min":48.685000000000002},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.38800000000000001},"max":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":0.97899999999999998},"last":0.51600000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64800000000000002,"15min":0.80700000000000005},"min":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.38700000000000001},"max":{"1min":0.85799999999999998,"5min":1.052,"15min":48.710000000000001},"last":0.748},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.034,"15min":1.1459999999999999},"last":0.55700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575382,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345608,"kb_used_data":36032,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94026232,"statfs":{"total":96636764160,"available":96282861568,"internally_reserved":0,"allocated":36896768,"data_stored":96360757,"data_compressed":13146180,"data_compressed_allocated":16756736,"data_compressed_original":78069760,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.37},"min":{"1min":0.23499999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.47599999999999998,"5min":0.64900000000000002,"15min":5.0670000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.371,"15min":0.38700000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.189},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.7799999999999998},"last":0.223}]},{"osd":1,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.36499999999999999,"15min":0.377},"min":{"1min":0.27200000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.628,"5min":0.628,"15min":2.758},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.41199999999999998},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51700000000000002,"5min":0.58399999999999996,"15min":0.64500000000000002},"last":0.29099999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28000000000000003,"15min":0.25},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":1.333},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.41899999999999998},"min":{"1min":0.30599999999999999,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.98699999999999999},"last":0.307}]},{"osd":4,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.48299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.879,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46200000000000002,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.23},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.45100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.2929999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.36199999999999999},"max":{"1min":0.90200000000000002,"5min":0.93600000000000005,"15min":1.103},"last":0.53900000000000003}]},{"osd":6,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63500000000000001,"15min":0.64000000000000001},"min":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.39900000000000002},"max":{"1min":0.91000000000000003,"5min":0.94199999999999995,"15min":1.198},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.49099999999999999,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.94899999999999995,"5min":0.95199999999999996,"15min":2.7309999999999999},"last":0.504}]},{"osd":7,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.46700000000000003,"5min":0.46500000000000002,"15min":0.39400000000000002},"max":{"1min":0.85099999999999998,"5min":0.97999999999999998,"15min":2.6619999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.66900000000000004},"min":{"1min":0.50800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.59699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575382,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338568,"kb_used_data":33088,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94033272,"statfs":{"total":96636764160,"available":96290070528,"internally_reserved":0,"allocated":33882112,"data_stored":101437325,"data_compressed":15509067,"data_compressed_allocated":19054592,"data_compressed_original":88412160,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56100000000000005,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.21199999999999999},"max":{"1min":0.746,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.23699999999999999},"max":{"1min":0.71899999999999997,"5min":0.86799999999999999,"15min":0.88900000000000001},"last":0.56999999999999995}]},{"osd":1,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":0.68300000000000005,"5min":0.71199999999999997,"15min":1.5920000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.55800000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.69899999999999995,"5min":0.749,"15min":1.0449999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.30199999999999999},"max":{"1min":0.752,"5min":0.78200000000000003,"15min":1.615},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.36699999999999999},"max":{"1min":0.73799999999999999,"5min":0.85799999999999998,"15min":1.071},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.58099999999999996},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.76200000000000001,"5min":0.85199999999999998,"15min":0.85799999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.28199999999999997},"max":{"1min":0.81799999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.60699999999999998}]},{"osd":5,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":1.087},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56699999999999995,"15min":0.55500000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88,"5min":0.88,"15min":1.111},"last":0.52700000000000002}]},{"osd":6,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56799999999999995,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.29799999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.5580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":0.90400000000000003,"15min":0.98799999999999999},"last":0.50800000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.312},"max":{"1min":1.016,"5min":1.016,"15min":1.0620000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64900000000000002,"15min":0.64200000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.33700000000000002},"max":{"1min":0.873,"5min":0.873,"15min":1.1160000000000001},"last":0.69199999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575382,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328896,"kb_used_data":23416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94042944,"statfs":{"total":96636764160,"available":96299974656,"internally_reserved":0,"allocated":23977984,"data_stored":83205478,"data_compressed":13026742,"data_compressed_allocated":15880192,"data_compressed_original":77045760,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.438,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":0.69499999999999995,"5min":0.75,"15min":0.82199999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56499999999999995,"15min":0.59699999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.63800000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46300000000000002}]},{"osd":1,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57599999999999996,"15min":0.55900000000000005},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.29699999999999999},"max":{"1min":0.745,"5min":0.79400000000000004,"15min":0.79800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.56999999999999995},"min":{"1min":0.48199999999999998,"5min":0.39200000000000002,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.432}]},{"osd":2,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.54000000000000004},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.30499999999999999},"max":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.442},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.39500000000000002,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.74099999999999999,"5min":0.75600000000000001,"15min":0.79000000000000004},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.86099999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59499999999999997,"15min":0.58799999999999997},"min":{"1min":0.46899999999999997,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.81799999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.50900000000000001}]},{"osd":4,"last update":"Tue Oct 10 19:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.45700000000000002},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.251},"max":{"1min":0.69999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58099999999999996,"15min":0.55400000000000005},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.23000000000000001},"max":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.49199999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.497},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.26700000000000002},"max":{"1min":0.72199999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53800000000000003},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.32100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.376}]},{"osd":7,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76600000000000001,"5min":0.86299999999999999,"15min":0.89400000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.27700000000000002},"max":{"1min":0.80500000000000005,"5min":0.84599999999999997,"15min":0.90800000000000003},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20283392,"data_stored":68677505,"data_compressed":9126818,"data_compressed_allocated":11898880,"data_compressed_original":60325888,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24403968,"data_stored":99763638,"data_compressed":12930657,"data_compressed_allocated":16519168,"data_compressed_original":91906048,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":51052075,"data_compressed":6450195,"data_compressed_allocated":8323072,"data_compressed_original":46788608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25817088,"data_stored":83490082,"data_compressed":10157145,"data_compressed_allocated":13479936,"data_compressed_original":71184384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24055808,"data_stored":89060236,"data_compressed":11845848,"data_compressed_allocated":15089664,"data_compressed_original":80117760,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16453632,"data_stored":73683110,"data_compressed":9606977,"data_compressed_allocated":12140544,"data_compressed_original":69410816,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21950464,"data_stored":75910818,"data_compressed":9436899,"data_compressed_allocated":12021760,"data_compressed_original":66007040,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":33710080,"data_stored":114783300,"data_compressed":15200435,"data_compressed_allocated":19759104,"data_compressed_original":100864000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8540160,"data_stored":11835204,"data_compressed":3779410,"data_compressed_allocated":4104192,"data_compressed_original":8499200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5124096,"data_stored":8614212,"data_compressed":3583818,"data_compressed_allocated":3870720,"data_compressed_original":8044544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3358720,"data_stored":4649136,"data_compressed":1917131,"data_compressed_allocated":2080768,"data_compressed_original":4046848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9498624,"data_stored":11984861,"data_compressed":2989035,"data_compressed_allocated":3276800,"data_compressed_original":6885376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8245248,"data_stored":11491153,"data_compressed":3663219,"data_compressed_allocated":3964928,"data_compressed_original":8294400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5943296,"data_stored":8636432,"data_compressed":3419765,"data_compressed_allocated":3739648,"data_compressed_original":7634944,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12181504,"data_stored":13790851,"data_compressed":2177080,"data_compressed_allocated":2371584,"data_compressed_original":4780032,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":15069585,"data_compressed":4432090,"data_compressed_allocated":4771840,"data_compressed_original":9945088,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T19:29:23.281 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T19:29:23.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.427+0000 7feaa0f73700 1 Processor -- start 2023-10-10T19:29:23.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.427+0000 7feaa0f73700 1 -- start start 2023-10-10T19:29:23.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7feaa0f73700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c14e090 0x7fea9c153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7feaa0f73700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea9c063640 0x7fea9c0c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7feaa0f73700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9c157b10 con 0x7fea9c0c2d20 2023-10-10T19:29:23.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7feaa0f73700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9c0c5200 con 0x7fea9c063640 2023-10-10T19:29:23.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7feaa0f73700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c068ab0 con 0x7fea9c14e090 2023-10-10T19:29:23.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7fea9ad9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fea9c0c2d20 0x7fea9c0677f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46402/0 (socket says 172.21.15.175:46402) 2023-10-10T19:29:23.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7fea9ad9d700 1 -- 172.21.15.175:0/1789392460 learned_addr learned my addr 172.21.15.175:0/1789392460 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:23.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7fea99d9b700 1 --2- 172.21.15.175:0/1789392460 >> v2:172.21.15.191:3300/0 conn(0x7fea9c063640 0x7fea9c0c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.428+0000 7fea9a59c700 1 --2- 172.21.15.175:0/1789392460 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c14e090 0x7fea9c153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9959a700 1 -- 172.21.15.175:0/1789392460 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886028565 0 0) 0x7fea9c157b10 con 0x7fea9c0c2d20 2023-10-10T19:29:23.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9959a700 1 -- 172.21.15.175:0/1789392460 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea84003570 con 0x7fea9c0c2d20 2023-10-10T19:29:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9959a700 1 -- 172.21.15.175:0/1789392460 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982666513 0 0) 0x7fea84003570 con 0x7fea9c0c2d20 2023-10-10T19:29:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9a59c700 1 -- 172.21.15.175:0/1789392460 >> v2:172.21.15.191:3300/0 conn(0x7fea9c063640 msgr2=0x7fea9c0c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9a59c700 1 --2- 172.21.15.175:0/1789392460 >> v2:172.21.15.191:3300/0 conn(0x7fea9c063640 0x7fea9c0c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9a59c700 1 -- 172.21.15.175:0/1789392460 >> v1:172.21.15.175:6789/0 conn(0x7fea9c0c2d20 legacy=0x7fea9c0677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9a59c700 1 -- 172.21.15.175:0/1789392460 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9c068d70 con 0x7fea9c14e090 2023-10-10T19:29:23.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea99d9b700 1 --2- 172.21.15.175:0/1789392460 >> v2:172.21.15.191:3300/0 conn(0x7fea9c063640 0x7fea9c0c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:23.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9a59c700 1 --2- 172.21.15.175:0/1789392460 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c14e090 0x7fea9c153ea0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009e60 tx=0x7fea8c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d94712f22d4100f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9959a700 1 -- 172.21.15.175:0/1789392460 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c00e070 con 0x7fea9c14e090 2023-10-10T19:29:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9959a700 1 -- 172.21.15.175:0/1789392460 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c00b7a0 con 0x7fea9c14e090 2023-10-10T19:29:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.429+0000 7fea9959a700 1 -- 172.21.15.175:0/1789392460 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea94001470 con 0x7fea9c14e090 2023-10-10T19:29:23.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.430+0000 7feaa0f73700 1 -- 172.21.15.175:0/1789392460 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c14e090 msgr2=0x7fea9c153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.430+0000 7feaa0f73700 1 --2- 172.21.15.175:0/1789392460 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c14e090 0x7fea9c153ea0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009e60 tx=0x7fea8c009780 comp rx=0 tx=0).stop 2023-10-10T19:29:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.430+0000 7feaa0f73700 1 -- 172.21.15.175:0/1789392460 shutdown_connections 2023-10-10T19:29:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.430+0000 7feaa0f73700 1 --2- 172.21.15.175:0/1789392460 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c14e090 0x7fea9c153ea0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.430+0000 7feaa0f73700 1 --2- 172.21.15.175:0/1789392460 >> v2:172.21.15.191:3300/0 conn(0x7fea9c063640 0x7fea9c0c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.430+0000 7feaa0f73700 1 -- 172.21.15.175:0/1789392460 >> 172.21.15.175:0/1789392460 conn(0x7fea9c0c5650 msgr2=0x7fea9c0c5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:23.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.430+0000 7feaa0f73700 1 -- 172.21.15.175:0/1789392460 shutdown_connections 2023-10-10T19:29:23.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.430+0000 7feaa0f73700 1 -- 172.21.15.175:0/1789392460 wait complete. 2023-10-10T19:29:23.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.431+0000 7feaa0f73700 1 Processor -- start 2023-10-10T19:29:23.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.431+0000 7feaa0f73700 1 -- start start 2023-10-10T19:29:23.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7feaa0f73700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c063640 0x7fea9c164600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7feaa0f73700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea9c0c2d20 0x7fea9c148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7feaa0f73700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea9c14e090 0x7fea9c1563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7feaa0f73700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c147b80 con 0x7fea9c0c2d20 2023-10-10T19:29:23.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7feaa0f73700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c149050 con 0x7fea9c14e090 2023-10-10T19:29:23.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7feaa0f73700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0bf3b0 con 0x7fea9c063640 2023-10-10T19:29:23.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea9c0c2d20 0x7fea9c148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea9c0c2d20 0x7fea9c148530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54466/0 (socket says 172.21.15.175:54466) 2023-10-10T19:29:23.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 -- 172.21.15.175:0/2647188370 learned_addr learned my addr 172.21.15.175:0/2647188370 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:23.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea9ad9d700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea9c14e090 0x7fea9c1563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea9a59c700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c063640 0x7fea9c164600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 -- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c063640 msgr2=0x7fea9c164600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c063640 0x7fea9c164600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 -- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea9c14e090 msgr2=0x7fea9c1563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea9c14e090 0x7fea9c1563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 -- 172.21.15.175:0/2647188370 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea90009420 con 0x7fea9c0c2d20 2023-10-10T19:29:23.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea99d9b700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea9c0c2d20 0x7fea9c148530 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fea9000ecb0 tx=0x7fea900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea8a7fc700 1 -- 172.21.15.175:0/2647188370 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea90009780 con 0x7fea9c0c2d20 2023-10-10T19:29:23.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8c009a20 con 0x7fea9c0c2d20 2023-10-10T19:29:23.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea8a7fc700 1 -- 172.21.15.175:0/2647188370 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9000cbe0 con 0x7fea9c0c2d20 2023-10-10T19:29:23.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7fea8a7fc700 1 -- 172.21.15.175:0/2647188370 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea90013470 con 0x7fea9c0c2d20 2023-10-10T19:29:23.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.432+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9c148d70 con 0x7fea9c0c2d20 2023-10-10T19:29:23.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.433+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9c147d30 con 0x7fea9c0c2d20 2023-10-10T19:29:23.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.434+0000 7fea8a7fc700 1 -- 172.21.15.175:0/2647188370 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fea900137e0 con 0x7fea9c0c2d20 2023-10-10T19:29:23.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.434+0000 7fea8a7fc700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea7c066f60 0x7fea7c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.434+0000 7fea8a7fc700 1 -- 172.21.15.175:0/2647188370 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fea90085150 con 0x7fea9c0c2d20 2023-10-10T19:29:23.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.434+0000 7fea9a59c700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea7c066f60 0x7fea7c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.435+0000 7fea9a59c700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea7c066f60 0x7fea7c069410 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fea8c003400 tx=0x7fea8c018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.438+0000 7fea8a7fc700 1 -- 172.21.15.175:0/2647188370 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea9c147d30 con 0x7fea9c0c2d20 2023-10-10T19:29:23.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.617+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea9c147d30 con 0x7fea7c066f60 2023-10-10T19:29:23.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.625+0000 7fea8a7fc700 1 -- 172.21.15.175:0/2647188370 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294172 (secure 0 0 0) 0x7fea9c147d30 con 0x7fea7c066f60 2023-10-10T19:29:23.629 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:23.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea7c066f60 msgr2=0x7fea7c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea7c066f60 0x7fea7c069410 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fea8c003400 tx=0x7fea8c018000 comp rx=0 tx=0).stop 2023-10-10T19:29:23.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea9c0c2d20 msgr2=0x7fea9c148530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea9c0c2d20 0x7fea9c148530 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fea9000ecb0 tx=0x7fea900091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 shutdown_connections 2023-10-10T19:29:23.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fea7c066f60 0x7fea7c069410 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea9c14e090 0x7fea9c1563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea9c063640 0x7fea9c164600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 --2- 172.21.15.175:0/2647188370 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea9c0c2d20 0x7fea9c148530 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 >> 172.21.15.175:0/2647188370 conn(0x7fea9c0c5650 msgr2=0x7fea9c0c21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:23.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 shutdown_connections 2023-10-10T19:29:23.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.629+0000 7feaa0f73700 1 -- 172.21.15.175:0/2647188370 wait complete. 2023-10-10T19:29:23.635 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T19:29:23.641 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":1454,"stamp":"2023-10-10T19:29:21.817054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371836467,"num_objects":2019,"num_object_clones":0,"num_object_copies":4038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":7382,"num_read_kb":3484532,"num_write":62406,"num_write_kb":2973288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":338,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79214,"ondisk_log_size":79214,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2685672,"kb_used_data":250024,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752289048,"statfs":{"total":773094113280,"available":770343985152,"internally_reserved":0,"allocated":256024576,"data_stored":750760178,"data_compressed":110781050,"data_compressed_allocated":137502720,"data_compressed_original":645881856,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1436897,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":1564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003628"},"pg_stats":[{"pgid":"3.3f","version":"131'943","reported_seq":3563,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.259330+0000","last_change":"2023-10-10T19:16:54.796013+0000","last_active":"2023-10-10T19:28:34.259330+0000","last_peered":"2023-10-10T19:28:34.259330+0000","last_clean":"2023-10-10T19:28:34.259330+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T19:28:34.259330+0000","last_undegraded":"2023-10-10T19:28:34.259330+0000","last_fullsized":"2023-10-10T19:28:34.259330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017995507000000001,"stat_sum":{"num_bytes":795566,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":62,"num_read_kb":8068,"num_write":667,"num_write_kb":16726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"131'993","reported_seq":3612,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.284503+0000","last_change":"2023-10-10T19:16:54.920511+0000","last_active":"2023-10-10T19:29:04.284503+0000","last_peered":"2023-10-10T19:29:04.284503+0000","last_clean":"2023-10-10T19:29:04.284503+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T19:29:04.284503+0000","last_undegraded":"2023-10-10T19:29:04.284503+0000","last_fullsized":"2023-10-10T19:29:04.284503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083569260000000006,"stat_sum":{"num_bytes":832331,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":62,"num_read_kb":28805,"num_write":733,"num_write_kb":46104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"131'937","reported_seq":3564,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.912051+0000","last_change":"2023-10-10T19:16:54.980353+0000","last_active":"2023-10-10T19:28:33.912051+0000","last_peered":"2023-10-10T19:28:33.912051+0000","last_clean":"2023-10-10T19:28:33.912051+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T19:28:33.912051+0000","last_undegraded":"2023-10-10T19:28:33.912051+0000","last_fullsized":"2023-10-10T19:28:33.912051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":937,"log_dups_size":0,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011353221,"stat_sum":{"num_bytes":419659,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":65,"num_read_kb":20680,"num_write":679,"num_write_kb":24555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"131'941","reported_seq":3555,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.942186+0000","last_change":"2023-10-10T19:16:55.412877+0000","last_active":"2023-10-10T19:28:24.942186+0000","last_peered":"2023-10-10T19:28:24.942186+0000","last_clean":"2023-10-10T19:28:24.942186+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T19:28:24.942186+0000","last_undegraded":"2023-10-10T19:28:24.942186+0000","last_fullsized":"2023-10-10T19:28:24.942186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088633549999999998,"stat_sum":{"num_bytes":476985,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":62,"num_read_kb":30965,"num_write":662,"num_write_kb":37535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"131'971","reported_seq":3601,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.424759+0000","last_change":"2023-10-10T19:16:54.790962+0000","last_active":"2023-10-10T19:29:04.424759+0000","last_peered":"2023-10-10T19:29:04.424759+0000","last_clean":"2023-10-10T19:29:04.424759+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T19:29:04.424759+0000","last_undegraded":"2023-10-10T19:29:04.424759+0000","last_fullsized":"2023-10-10T19:29:04.424759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016189688000000001,"stat_sum":{"num_bytes":142623,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":75,"num_read_kb":17848,"num_write":693,"num_write_kb":31284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"131'918","reported_seq":3544,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704394+0000","last_change":"2023-10-10T19:16:55.414924+0000","last_active":"2023-10-10T19:28:34.704394+0000","last_peered":"2023-10-10T19:28:34.704394+0000","last_clean":"2023-10-10T19:28:34.704394+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T19:28:34.704394+0000","last_undegraded":"2023-10-10T19:28:34.704394+0000","last_fullsized":"2023-10-10T19:28:34.704394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010631753000000001,"stat_sum":{"num_bytes":567917,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":63,"num_read_kb":20939,"num_write":659,"num_write_kb":24217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"131'968","reported_seq":3589,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078564+0000","last_change":"2023-10-10T19:16:54.789065+0000","last_active":"2023-10-10T19:29:04.078564+0000","last_peered":"2023-10-10T19:29:04.078564+0000","last_clean":"2023-10-10T19:29:04.078564+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T19:29:04.078564+0000","last_undegraded":"2023-10-10T19:29:04.078564+0000","last_fullsized":"2023-10-10T19:29:04.078564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014667316999999999,"stat_sum":{"num_bytes":1084188,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":53600,"num_write":676,"num_write_kb":52841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"131'852","reported_seq":3490,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077662+0000","last_change":"2023-10-10T19:16:54.870456+0000","last_active":"2023-10-10T19:29:04.077662+0000","last_peered":"2023-10-10T19:29:04.077662+0000","last_clean":"2023-10-10T19:29:04.077662+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T19:29:04.077662+0000","last_undegraded":"2023-10-10T19:29:04.077662+0000","last_fullsized":"2023-10-10T19:29:04.077662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":852,"log_dups_size":0,"ondisk_log_size":852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012402231,"stat_sum":{"num_bytes":449638,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":58,"num_read_kb":28110,"num_write":635,"num_write_kb":35676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"131'959","reported_seq":3586,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078384+0000","last_change":"2023-10-10T19:16:54.795632+0000","last_active":"2023-10-10T19:29:04.078384+0000","last_peered":"2023-10-10T19:29:04.078384+0000","last_clean":"2023-10-10T19:29:04.078384+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T19:29:04.078384+0000","last_undegraded":"2023-10-10T19:29:04.078384+0000","last_fullsized":"2023-10-10T19:29:04.078384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017699890999999999,"stat_sum":{"num_bytes":30143,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":76,"num_read_kb":37435,"num_write":691,"num_write_kb":40783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"131'935","reported_seq":3561,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.401310+0000","last_change":"2023-10-10T19:16:54.783622+0000","last_active":"2023-10-10T19:29:04.401310+0000","last_peered":"2023-10-10T19:29:04.401310+0000","last_clean":"2023-10-10T19:29:04.401310+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T19:29:04.401310+0000","last_undegraded":"2023-10-10T19:29:04.401310+0000","last_fullsized":"2023-10-10T19:29:04.401310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079196160000000008,"stat_sum":{"num_bytes":128300,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":70,"num_read_kb":16825,"num_write":682,"num_write_kb":23273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"131'955","reported_seq":3604,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.424201+0000","last_change":"2023-10-10T19:16:54.784745+0000","last_active":"2023-10-10T19:29:04.424201+0000","last_peered":"2023-10-10T19:29:04.424201+0000","last_clean":"2023-10-10T19:29:04.424201+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T19:29:04.424201+0000","last_undegraded":"2023-10-10T19:29:04.424201+0000","last_fullsized":"2023-10-10T19:29:04.424201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088422029999999999,"stat_sum":{"num_bytes":198054,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":82,"num_read_kb":46829,"num_write":704,"num_write_kb":53046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"131'944","reported_seq":3579,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.484037+0000","last_change":"2023-10-10T19:16:54.989857+0000","last_active":"2023-10-10T19:29:04.484037+0000","last_peered":"2023-10-10T19:29:04.484037+0000","last_clean":"2023-10-10T19:29:04.484037+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T19:29:04.484037+0000","last_undegraded":"2023-10-10T19:29:04.484037+0000","last_fullsized":"2023-10-10T19:29:04.484037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078589550000000008,"stat_sum":{"num_bytes":271785,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":75,"num_read_kb":57532,"num_write":692,"num_write_kb":73449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"131'896","reported_seq":3526,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.275163+0000","last_change":"2023-10-10T19:16:54.872408+0000","last_active":"2023-10-10T19:29:04.275163+0000","last_peered":"2023-10-10T19:29:04.275163+0000","last_clean":"2023-10-10T19:29:04.275163+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T19:29:04.275163+0000","last_undegraded":"2023-10-10T19:29:04.275163+0000","last_fullsized":"2023-10-10T19:29:04.275163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014204069,"stat_sum":{"num_bytes":4542310,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":73,"num_read_kb":32208,"num_write":631,"num_write_kb":44582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"131'902","reported_seq":3526,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.393937+0000","last_change":"2023-10-10T19:16:54.978714+0000","last_active":"2023-10-10T19:29:04.393937+0000","last_peered":"2023-10-10T19:29:04.393937+0000","last_clean":"2023-10-10T19:29:04.393937+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T19:29:04.393937+0000","last_undegraded":"2023-10-10T19:29:04.393937+0000","last_fullsized":"2023-10-10T19:29:04.393937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097965510000000006,"stat_sum":{"num_bytes":686329,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":63,"num_read_kb":28420,"num_write":651,"num_write_kb":31132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"131'934","reported_seq":3567,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.271178+0000","last_change":"2023-10-10T19:16:54.917427+0000","last_active":"2023-10-10T19:29:04.271178+0000","last_peered":"2023-10-10T19:29:04.271178+0000","last_clean":"2023-10-10T19:29:04.271178+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T19:29:04.271178+0000","last_undegraded":"2023-10-10T19:29:04.271178+0000","last_fullsized":"2023-10-10T19:29:04.271178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059774370000000004,"stat_sum":{"num_bytes":1294620,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":77,"num_read_kb":52747,"num_write":659,"num_write_kb":72992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"131'977","reported_seq":3611,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077966+0000","last_change":"2023-10-10T19:16:55.412885+0000","last_active":"2023-10-10T19:29:04.077966+0000","last_peered":"2023-10-10T19:29:04.077966+0000","last_clean":"2023-10-10T19:29:04.077966+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T19:29:04.077966+0000","last_undegraded":"2023-10-10T19:29:04.077966+0000","last_fullsized":"2023-10-10T19:29:04.077966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087668889999999999,"stat_sum":{"num_bytes":446378,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":80,"num_read_kb":42995,"num_write":700,"num_write_kb":52933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"131'882","reported_seq":3536,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.705589+0000","last_change":"2023-10-10T19:16:55.417126+0000","last_active":"2023-10-10T19:28:34.705589+0000","last_peered":"2023-10-10T19:28:34.705589+0000","last_clean":"2023-10-10T19:28:34.705589+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T19:28:34.705589+0000","last_undegraded":"2023-10-10T19:28:34.705589+0000","last_fullsized":"2023-10-10T19:28:34.705589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012085218,"stat_sum":{"num_bytes":1843052,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":72,"num_read_kb":40372,"num_write":665,"num_write_kb":48809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"131'826","reported_seq":3498,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.912020+0000","last_change":"2023-10-10T19:16:54.780747+0000","last_active":"2023-10-10T19:28:33.912020+0000","last_peered":"2023-10-10T19:28:33.912020+0000","last_clean":"2023-10-10T19:28:33.912020+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T19:28:33.912020+0000","last_undegraded":"2023-10-10T19:28:33.912020+0000","last_fullsized":"2023-10-10T19:28:33.912020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00594963,"stat_sum":{"num_bytes":909657,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":72,"num_read_kb":44243,"num_write":651,"num_write_kb":60901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"131'1049","reported_seq":3646,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.588062+0000","last_change":"2023-10-10T19:16:54.792335+0000","last_active":"2023-10-10T19:29:04.588062+0000","last_peered":"2023-10-10T19:29:04.588062+0000","last_clean":"2023-10-10T19:29:04.588062+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T19:29:04.588062+0000","last_undegraded":"2023-10-10T19:29:04.588062+0000","last_fullsized":"2023-10-10T19:29:04.588062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01601,"stat_sum":{"num_bytes":624343,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":73,"num_read_kb":41187,"num_write":717,"num_write_kb":49713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"131'836","reported_seq":1001,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:18.796925+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T19:28:18.796925+0000","last_peered":"2023-10-10T19:28:18.796925+0000","last_clean":"2023-10-10T19:28:18.796925+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T19:28:18.796925+0000","last_undegraded":"2023-10-10T19:28:18.796925+0000","last_fullsized":"2023-10-10T19:28:18.796925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389252,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":82,"num_read_kb":61516,"num_write":780,"num_write_kb":21018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"131'974","reported_seq":3595,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.491287+0000","last_change":"2023-10-10T19:16:54.877116+0000","last_active":"2023-10-10T19:29:04.491287+0000","last_peered":"2023-10-10T19:29:04.491287+0000","last_clean":"2023-10-10T19:29:04.491287+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T19:29:04.491287+0000","last_undegraded":"2023-10-10T19:29:04.491287+0000","last_fullsized":"2023-10-10T19:29:04.491287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018086130999999998,"stat_sum":{"num_bytes":63039,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":63,"num_read_kb":13169,"num_write":717,"num_write_kb":31098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"131'759","reported_seq":903,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:04.229561+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T19:28:04.229561+0000","last_peered":"2023-10-10T19:28:04.229561+0000","last_clean":"2023-10-10T19:28:04.229561+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T19:28:04.229561+0000","last_undegraded":"2023-10-10T19:28:04.229561+0000","last_fullsized":"2023-10-10T19:28:04.229561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":29,"num_read_kb":69691,"num_write":709,"num_write_kb":16925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"131'971","reported_seq":3613,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078264+0000","last_change":"2023-10-10T19:16:54.978352+0000","last_active":"2023-10-10T19:29:04.078264+0000","last_peered":"2023-10-10T19:29:04.078264+0000","last_clean":"2023-10-10T19:29:04.078264+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T19:29:04.078264+0000","last_undegraded":"2023-10-10T19:29:04.078264+0000","last_fullsized":"2023-10-10T19:29:04.078264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093729599999999996,"stat_sum":{"num_bytes":213282,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":74,"num_read_kb":34573,"num_write":722,"num_write_kb":44041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"132'1694","reported_seq":1870,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:10.134874+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T19:29:10.134874+0000","last_peered":"2023-10-10T19:29:10.134874+0000","last_clean":"2023-10-10T19:29:10.134874+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T19:29:10.134874+0000","last_undegraded":"2023-10-10T19:29:10.134874+0000","last_fullsized":"2023-10-10T19:29:10.134874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1694,"log_dups_size":0,"ondisk_log_size":1694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":114,"num_read_kb":59511,"num_write":1903,"num_write_kb":22655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"131'936","reported_seq":3530,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.496246+0000","last_change":"2023-10-10T19:16:54.780201+0000","last_active":"2023-10-10T19:29:04.496246+0000","last_peered":"2023-10-10T19:29:04.496246+0000","last_clean":"2023-10-10T19:29:04.496246+0000","last_became_active":"2023-10-10T18:46:01.103860+0000","last_became_peered":"2023-10-10T18:46:01.103860+0000","last_unstale":"2023-10-10T19:29:04.496246+0000","last_undegraded":"2023-10-10T19:29:04.496246+0000","last_fullsized":"2023-10-10T19:29:04.496246+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:17.975194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054668909999999998,"stat_sum":{"num_bytes":80680,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":48,"num_read_kb":295,"num_write":639,"num_write_kb":7460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"130'750","reported_seq":903,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.970799+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T19:27:29.970799+0000","last_peered":"2023-10-10T19:27:29.970799+0000","last_clean":"2023-10-10T19:27:29.970799+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T19:27:29.970799+0000","last_undegraded":"2023-10-10T19:27:29.970799+0000","last_fullsized":"2023-10-10T19:27:29.970799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389162,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":55,"num_read_kb":36831,"num_write":683,"num_write_kb":16858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"131'970","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.274296+0000","last_change":"2023-10-10T19:16:55.416075+0000","last_active":"2023-10-10T19:29:04.274296+0000","last_peered":"2023-10-10T19:29:04.274296+0000","last_clean":"2023-10-10T19:29:04.274296+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T19:29:04.274296+0000","last_undegraded":"2023-10-10T19:29:04.274296+0000","last_fullsized":"2023-10-10T19:29:04.274296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011753095,"stat_sum":{"num_bytes":80024,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":81,"num_read_kb":35882,"num_write":686,"num_write_kb":42928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"131'612","reported_seq":760,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.922994+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T19:28:38.922994+0000","last_peered":"2023-10-10T19:28:38.922994+0000","last_clean":"2023-10-10T19:28:38.922994+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T19:28:38.922994+0000","last_undegraded":"2023-10-10T19:28:38.922994+0000","last_fullsized":"2023-10-10T19:28:38.922994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14118582,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":31,"num_read_kb":65583,"num_write":554,"num_write_kb":18318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"131'893","reported_seq":3521,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.393580+0000","last_change":"2023-10-10T19:16:54.987567+0000","last_active":"2023-10-10T19:29:04.393580+0000","last_peered":"2023-10-10T19:29:04.393580+0000","last_clean":"2023-10-10T19:29:04.393580+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T19:29:04.393580+0000","last_undegraded":"2023-10-10T19:29:04.393580+0000","last_fullsized":"2023-10-10T19:29:04.393580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062644010000000002,"stat_sum":{"num_bytes":763528,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":59,"num_read_kb":21826,"num_write":671,"num_write_kb":37641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"129'775","reported_seq":1107,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.932043+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T19:28:38.932043+0000","last_peered":"2023-10-10T19:28:38.932043+0000","last_clean":"2023-10-10T19:28:38.932043+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T19:28:38.932043+0000","last_undegraded":"2023-10-10T19:28:38.932043+0000","last_fullsized":"2023-10-10T19:28:38.932043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111936,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":251,"num_read_kb":81147,"num_write":700,"num_write_kb":25323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"131'819","reported_seq":3467,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.924982+0000","last_change":"2023-10-10T19:16:54.984860+0000","last_active":"2023-10-10T19:28:24.924982+0000","last_peered":"2023-10-10T19:28:24.924982+0000","last_clean":"2023-10-10T19:28:24.924982+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T19:28:24.924982+0000","last_undegraded":"2023-10-10T19:28:24.924982+0000","last_fullsized":"2023-10-10T19:28:24.924982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015269961,"stat_sum":{"num_bytes":449137,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":60,"num_read_kb":24046,"num_write":605,"num_write_kb":30663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"130'1494","reported_seq":1889,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.913899+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T19:28:38.913899+0000","last_peered":"2023-10-10T19:28:38.913899+0000","last_clean":"2023-10-10T19:28:38.913899+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T19:28:38.913899+0000","last_undegraded":"2023-10-10T19:28:38.913899+0000","last_fullsized":"2023-10-10T19:28:38.913899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4727705,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":295,"num_read_kb":15405,"num_write":1447,"num_write_kb":10055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"131'980","reported_seq":3623,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.268565+0000","last_change":"2023-10-10T19:16:54.874108+0000","last_active":"2023-10-10T19:29:04.268565+0000","last_peered":"2023-10-10T19:29:04.268565+0000","last_clean":"2023-10-10T19:29:04.268565+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T19:29:04.268565+0000","last_undegraded":"2023-10-10T19:29:04.268565+0000","last_fullsized":"2023-10-10T19:29:04.268565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015767953000000001,"stat_sum":{"num_bytes":541499,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":81,"num_read_kb":19092,"num_write":733,"num_write_kb":30243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"131'494","reported_seq":639,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704437+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T19:28:34.704437+0000","last_peered":"2023-10-10T19:28:34.704437+0000","last_clean":"2023-10-10T19:28:34.704437+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T19:28:34.704437+0000","last_undegraded":"2023-10-10T19:28:34.704437+0000","last_fullsized":"2023-10-10T19:28:34.704437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583466,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":65566,"num_write":451,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"131'1005","reported_seq":3644,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.395988+0000","last_change":"2023-10-10T19:16:54.918258+0000","last_active":"2023-10-10T19:29:04.395988+0000","last_peered":"2023-10-10T19:29:04.395988+0000","last_clean":"2023-10-10T19:29:04.395988+0000","last_became_active":"2023-10-10T18:46:00.606391+0000","last_became_peered":"2023-10-10T18:46:00.606391+0000","last_unstale":"2023-10-10T19:29:04.395988+0000","last_undegraded":"2023-10-10T19:29:04.395988+0000","last_fullsized":"2023-10-10T19:29:04.395988+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:57.665869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066939039999999997,"stat_sum":{"num_bytes":631366,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":80,"num_read_kb":42231,"num_write":729,"num_write_kb":52765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"132'274","reported_seq":415,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:15.078147+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T19:29:15.078147+0000","last_peered":"2023-10-10T19:29:15.078147+0000","last_clean":"2023-10-10T19:29:15.078147+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T19:29:15.078147+0000","last_undegraded":"2023-10-10T19:29:15.078147+0000","last_fullsized":"2023-10-10T19:29:15.078147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":34,"num_read_kb":54,"num_write":225,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"131'948","reported_seq":3583,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.705242+0000","last_change":"2023-10-10T19:16:55.413821+0000","last_active":"2023-10-10T19:28:34.705242+0000","last_peered":"2023-10-10T19:28:34.705242+0000","last_clean":"2023-10-10T19:28:34.705242+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T19:28:34.705242+0000","last_undegraded":"2023-10-10T19:28:34.705242+0000","last_fullsized":"2023-10-10T19:28:34.705242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096962509999999995,"stat_sum":{"num_bytes":519488,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":63,"num_read_kb":26256,"num_write":717,"num_write_kb":45994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"132'617","reported_seq":975,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:10.134272+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T19:29:10.134272+0000","last_peered":"2023-10-10T19:29:10.134272+0000","last_clean":"2023-10-10T19:29:10.134272+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T19:29:10.134272+0000","last_undegraded":"2023-10-10T19:29:10.134272+0000","last_fullsized":"2023-10-10T19:29:10.134272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7932987,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":262,"num_read_kb":30418,"num_write":555,"num_write_kb":12258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"131'906","reported_seq":3529,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.400410+0000","last_change":"2023-10-10T19:16:54.984649+0000","last_active":"2023-10-10T19:29:04.400410+0000","last_peered":"2023-10-10T19:29:04.400410+0000","last_clean":"2023-10-10T19:29:04.400410+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T19:29:04.400410+0000","last_undegraded":"2023-10-10T19:29:04.400410+0000","last_fullsized":"2023-10-10T19:29:04.400410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015222243,"stat_sum":{"num_bytes":277071,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":64,"num_read_kb":38795,"num_write":628,"num_write_kb":46072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"131'207","reported_seq":349,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.761037+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T19:28:34.761037+0000","last_peered":"2023-10-10T19:28:34.761037+0000","last_clean":"2023-10-10T19:28:34.761037+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T19:28:34.761037+0000","last_undegraded":"2023-10-10T19:28:34.761037+0000","last_fullsized":"2023-10-10T19:28:34.761037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":42,"num_read_kb":61,"num_write":171,"num_write_kb":225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"131'858","reported_seq":3490,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.320810+0000","last_change":"2023-10-10T19:16:54.989921+0000","last_active":"2023-10-10T19:29:04.320810+0000","last_peered":"2023-10-10T19:29:04.320810+0000","last_clean":"2023-10-10T19:29:04.320810+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T19:29:04.320810+0000","last_undegraded":"2023-10-10T19:29:04.320810+0000","last_fullsized":"2023-10-10T19:29:04.320810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080742209999999995,"stat_sum":{"num_bytes":82158,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":56,"num_read_kb":23103,"num_write":645,"num_write_kb":46075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"132'264","reported_seq":413,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.276956+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T19:29:04.276956+0000","last_peered":"2023-10-10T19:29:04.276956+0000","last_clean":"2023-10-10T19:29:04.276956+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T19:29:04.276956+0000","last_undegraded":"2023-10-10T19:29:04.276956+0000","last_fullsized":"2023-10-10T19:29:04.276956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194948,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":59,"num_read_kb":16449,"num_write":212,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"131'930","reported_seq":3560,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.595503+0000","last_change":"2023-10-10T19:16:54.793178+0000","last_active":"2023-10-10T19:29:04.595503+0000","last_peered":"2023-10-10T19:29:04.595503+0000","last_clean":"2023-10-10T19:29:04.595503+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T19:29:04.595503+0000","last_undegraded":"2023-10-10T19:29:04.595503+0000","last_fullsized":"2023-10-10T19:29:04.595503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015774979000000001,"stat_sum":{"num_bytes":110666,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":71,"num_read_kb":24956,"num_write":663,"num_write_kb":31333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"125'818","reported_seq":1254,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.911380+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T19:28:33.911380+0000","last_peered":"2023-10-10T19:28:33.911380+0000","last_clean":"2023-10-10T19:28:33.911380+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T19:28:33.911380+0000","last_undegraded":"2023-10-10T19:28:33.911380+0000","last_fullsized":"2023-10-10T19:28:33.911380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227398,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":359,"num_read_kb":46031,"num_write":769,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"131'911","reported_seq":3542,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078167+0000","last_change":"2023-10-10T19:16:54.981687+0000","last_active":"2023-10-10T19:29:04.078167+0000","last_peered":"2023-10-10T19:29:04.078167+0000","last_clean":"2023-10-10T19:29:04.078167+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T19:29:04.078167+0000","last_undegraded":"2023-10-10T19:29:04.078167+0000","last_fullsized":"2023-10-10T19:29:04.078167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01251216,"stat_sum":{"num_bytes":89325,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":62,"num_read_kb":21731,"num_write":674,"num_write_kb":32887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"128'713","reported_seq":873,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.927273+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T19:28:38.927273+0000","last_peered":"2023-10-10T19:28:38.927273+0000","last_clean":"2023-10-10T19:28:38.927273+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T19:28:38.927273+0000","last_undegraded":"2023-10-10T19:28:38.927273+0000","last_fullsized":"2023-10-10T19:28:38.927273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415841,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":58,"num_read_kb":76305,"num_write":648,"num_write_kb":19602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"131'951","reported_seq":3576,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.258906+0000","last_change":"2023-10-10T19:16:54.791948+0000","last_active":"2023-10-10T19:28:34.258906+0000","last_peered":"2023-10-10T19:28:34.258906+0000","last_clean":"2023-10-10T19:28:34.258906+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T19:28:34.258906+0000","last_undegraded":"2023-10-10T19:28:34.258906+0000","last_fullsized":"2023-10-10T19:28:34.258906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017093574,"stat_sum":{"num_bytes":514832,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":73,"num_read_kb":45017,"num_write":683,"num_write_kb":44061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"131'407","reported_seq":563,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.926155+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T19:28:38.926155+0000","last_peered":"2023-10-10T19:28:38.926155+0000","last_clean":"2023-10-10T19:28:38.926155+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T19:28:38.926155+0000","last_undegraded":"2023-10-10T19:28:38.926155+0000","last_fullsized":"2023-10-10T19:28:38.926155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":54,"num_read_kb":84439,"num_write":360,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"131'818","reported_seq":3450,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.403900+0000","last_change":"2023-10-10T19:16:54.782792+0000","last_active":"2023-10-10T19:29:04.403900+0000","last_peered":"2023-10-10T19:29:04.403900+0000","last_clean":"2023-10-10T19:29:04.403900+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T19:29:04.403900+0000","last_undegraded":"2023-10-10T19:29:04.403900+0000","last_fullsized":"2023-10-10T19:29:04.403900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075683979999999996,"stat_sum":{"num_bytes":1065275,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":52,"num_read_kb":24653,"num_write":599,"num_write_kb":45347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"131'804","reported_seq":983,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.911148+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T19:28:33.911148+0000","last_peered":"2023-10-10T19:28:33.911148+0000","last_clean":"2023-10-10T19:28:33.911148+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T19:28:33.911148+0000","last_undegraded":"2023-10-10T19:28:33.911148+0000","last_fullsized":"2023-10-10T19:28:33.911148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":804,"log_dups_size":0,"ondisk_log_size":804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":113,"num_read_kb":78302,"num_write":734,"num_write_kb":25074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"131'894","reported_seq":3531,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.913337+0000","last_change":"2023-10-10T19:16:54.790805+0000","last_active":"2023-10-10T19:28:24.913337+0000","last_peered":"2023-10-10T19:28:24.913337+0000","last_clean":"2023-10-10T19:28:24.913337+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T19:28:24.913337+0000","last_undegraded":"2023-10-10T19:28:24.913337+0000","last_fullsized":"2023-10-10T19:28:24.913337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01616896,"stat_sum":{"num_bytes":43538,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":67,"num_read_kb":45230,"num_write":650,"num_write_kb":49050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"131'604","reported_seq":741,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:19.022228+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T19:28:19.022228+0000","last_peered":"2023-10-10T19:28:19.022228+0000","last_clean":"2023-10-10T19:28:19.022228+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T19:28:19.022228+0000","last_undegraded":"2023-10-10T19:28:19.022228+0000","last_fullsized":"2023-10-10T19:28:19.022228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":23,"num_read_kb":12341,"num_write":746,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"131'918","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.406226+0000","last_change":"2023-10-10T19:16:54.793503+0000","last_active":"2023-10-10T19:29:04.406226+0000","last_peered":"2023-10-10T19:29:04.406226+0000","last_clean":"2023-10-10T19:29:04.406226+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T19:29:04.406226+0000","last_undegraded":"2023-10-10T19:29:04.406226+0000","last_fullsized":"2023-10-10T19:29:04.406226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015882960000000002,"stat_sum":{"num_bytes":652546,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":57,"num_read_kb":20900,"num_write":659,"num_write_kb":24656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"132'150","reported_seq":287,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:15.166343+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T19:29:15.166343+0000","last_peered":"2023-10-10T19:29:15.166343+0000","last_clean":"2023-10-10T19:29:15.166343+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T19:29:15.166343+0000","last_undegraded":"2023-10-10T19:29:15.166343+0000","last_fullsized":"2023-10-10T19:29:15.166343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311985,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":45123,"num_write":106,"num_write_kb":9377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"131'1026","reported_seq":3634,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.495833+0000","last_change":"2023-10-10T19:16:55.415960+0000","last_active":"2023-10-10T19:29:04.495833+0000","last_peered":"2023-10-10T19:29:04.495833+0000","last_clean":"2023-10-10T19:29:04.495833+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T19:29:04.495833+0000","last_undegraded":"2023-10-10T19:29:04.495833+0000","last_fullsized":"2023-10-10T19:29:04.495833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011491507999999999,"stat_sum":{"num_bytes":340961,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":71,"num_read_kb":20716,"num_write":714,"num_write_kb":35169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"130'341","reported_seq":489,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704918+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T19:28:34.704918+0000","last_peered":"2023-10-10T19:28:34.704918+0000","last_clean":"2023-10-10T19:28:34.704918+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T19:28:34.704918+0000","last_undegraded":"2023-10-10T19:28:34.704918+0000","last_fullsized":"2023-10-10T19:28:34.704918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":73794,"num_write":284,"num_write_kb":20986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"131'933","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758752+0000","last_change":"2023-10-10T19:16:54.988658+0000","last_active":"2023-10-10T19:28:34.758752+0000","last_peered":"2023-10-10T19:28:34.758752+0000","last_clean":"2023-10-10T19:28:34.758752+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T19:28:34.758752+0000","last_undegraded":"2023-10-10T19:28:34.758752+0000","last_fullsized":"2023-10-10T19:28:34.758752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071091339999999996,"stat_sum":{"num_bytes":108666,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":93,"num_read_kb":25139,"num_write":704,"num_write_kb":35589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"131'931","reported_seq":3556,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.447963+0000","last_change":"2023-10-10T19:16:54.919574+0000","last_active":"2023-10-10T19:29:04.447963+0000","last_peered":"2023-10-10T19:29:04.447963+0000","last_clean":"2023-10-10T19:29:04.447963+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T19:29:04.447963+0000","last_undegraded":"2023-10-10T19:29:04.447963+0000","last_fullsized":"2023-10-10T19:29:04.447963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077260940000000002,"stat_sum":{"num_bytes":89807,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":61,"num_read_kb":19697,"num_write":681,"num_write_kb":26677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"132'685","reported_seq":1109,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:18.043195+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T19:29:18.043195+0000","last_peered":"2023-10-10T19:29:18.043195+0000","last_clean":"2023-10-10T19:29:18.043195+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T19:29:18.043195+0000","last_undegraded":"2023-10-10T19:29:18.043195+0000","last_fullsized":"2023-10-10T19:29:18.043195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9108221,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":314,"num_read_kb":40841,"num_write":609,"num_write_kb":9447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'875","reported_seq":3545,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.281760+0000","last_change":"2023-10-10T19:16:54.875665+0000","last_active":"2023-10-10T19:29:04.281760+0000","last_peered":"2023-10-10T19:29:04.281760+0000","last_clean":"2023-10-10T19:29:04.281760+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T19:29:04.281760+0000","last_undegraded":"2023-10-10T19:29:04.281760+0000","last_fullsized":"2023-10-10T19:29:04.281760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017091021000000001,"stat_sum":{"num_bytes":821658,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":88,"num_read_kb":19324,"num_write":658,"num_write_kb":26590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"131'137","reported_seq":294,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.142425+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T19:28:39.142425+0000","last_peered":"2023-10-10T19:28:39.142425+0000","last_clean":"2023-10-10T19:28:39.142425+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T19:28:39.142425+0000","last_undegraded":"2023-10-10T19:28:39.142425+0000","last_fullsized":"2023-10-10T19:28:39.142425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388756,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":36939,"num_write":74,"num_write_kb":8306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":274,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.970643+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T19:27:29.970643+0000","last_peered":"2023-10-10T19:27:29.970643+0000","last_clean":"2023-10-10T19:27:29.970643+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T19:27:29.970643+0000","last_undegraded":"2023-10-10T19:27:29.970643+0000","last_fullsized":"2023-10-10T19:27:29.970643+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"131'887","reported_seq":3510,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078025+0000","last_change":"2023-10-10T19:16:54.979966+0000","last_active":"2023-10-10T19:29:04.078025+0000","last_peered":"2023-10-10T19:29:04.078025+0000","last_clean":"2023-10-10T19:29:04.078025+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T19:29:04.078025+0000","last_undegraded":"2023-10-10T19:29:04.078025+0000","last_fullsized":"2023-10-10T19:29:04.078025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010927930000000001,"stat_sum":{"num_bytes":3326937,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":55,"num_read_kb":20626,"num_write":650,"num_write_kb":30635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"131'973","reported_seq":1124,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:40.006368+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T19:28:40.006368+0000","last_peered":"2023-10-10T19:28:40.006368+0000","last_clean":"2023-10-10T19:28:40.006368+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T19:28:40.006368+0000","last_undegraded":"2023-10-10T19:28:40.006368+0000","last_fullsized":"2023-10-10T19:28:40.006368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":38,"num_read_kb":89926,"num_write":912,"num_write_kb":29260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"128'825","reported_seq":978,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.259463+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T19:28:34.259463+0000","last_peered":"2023-10-10T19:28:34.259463+0000","last_clean":"2023-10-10T19:28:34.259463+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T19:28:34.259463+0000","last_undegraded":"2023-10-10T19:28:34.259463+0000","last_fullsized":"2023-10-10T19:28:34.259463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":45,"num_read_kb":45594,"num_write":764,"num_write_kb":12838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"131'1007","reported_seq":3620,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077312+0000","last_change":"2023-10-10T19:16:54.919294+0000","last_active":"2023-10-10T19:29:04.077312+0000","last_peered":"2023-10-10T19:29:04.077312+0000","last_clean":"2023-10-10T19:29:04.077312+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T19:29:04.077312+0000","last_undegraded":"2023-10-10T19:29:04.077312+0000","last_fullsized":"2023-10-10T19:29:04.077312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075449510000000003,"stat_sum":{"num_bytes":492311,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":65,"num_read_kb":49387,"num_write":712,"num_write_kb":52375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'275","reported_seq":431,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:49.017495+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T19:28:49.017495+0000","last_peered":"2023-10-10T19:28:49.017495+0000","last_clean":"2023-10-10T19:28:49.017495+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T19:28:49.017495+0000","last_undegraded":"2023-10-10T19:28:49.017495+0000","last_fullsized":"2023-10-10T19:28:49.017495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":81,"num_read_kb":20675,"num_write":475,"num_write_kb":4740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"131'939","reported_seq":3565,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.285360+0000","last_change":"2023-10-10T19:16:54.920290+0000","last_active":"2023-10-10T19:29:04.285360+0000","last_peered":"2023-10-10T19:29:04.285360+0000","last_clean":"2023-10-10T19:29:04.285360+0000","last_became_active":"2023-10-10T18:46:01.267092+0000","last_became_peered":"2023-10-10T18:46:01.267092+0000","last_unstale":"2023-10-10T19:29:04.285360+0000","last_undegraded":"2023-10-10T19:29:04.285360+0000","last_fullsized":"2023-10-10T19:29:04.285360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:46:17.491022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083609700000000006,"stat_sum":{"num_bytes":2123745,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":68,"num_read_kb":33081,"num_write":679,"num_write_kb":41978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"132'154","reported_seq":301,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:41.419018+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T19:28:41.419018+0000","last_peered":"2023-10-10T19:28:41.419018+0000","last_clean":"2023-10-10T19:28:41.419018+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T19:28:41.419018+0000","last_undegraded":"2023-10-10T19:28:41.419018+0000","last_fullsized":"2023-10-10T19:28:41.419018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":58,"num_read_kb":8269,"num_write":92,"num_write_kb":8284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"131'867","reported_seq":3527,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.288386+0000","last_change":"2023-10-10T19:16:54.981885+0000","last_active":"2023-10-10T19:29:04.288386+0000","last_peered":"2023-10-10T19:29:04.288386+0000","last_clean":"2023-10-10T19:29:04.288386+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T19:29:04.288386+0000","last_undegraded":"2023-10-10T19:29:04.288386+0000","last_fullsized":"2023-10-10T19:29:04.288386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721872,"stat_sum":{"num_bytes":440031,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":85,"num_read_kb":32652,"num_write":635,"num_write_kb":36138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"131'228","reported_seq":371,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.557527+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T19:28:39.557527+0000","last_peered":"2023-10-10T19:28:39.557527+0000","last_clean":"2023-10-10T19:28:39.557527+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T19:28:39.557527+0000","last_undegraded":"2023-10-10T19:28:39.557527+0000","last_fullsized":"2023-10-10T19:28:39.557527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":57389,"num_write":173,"num_write_kb":12477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"131'931","reported_seq":3570,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758691+0000","last_change":"2023-10-10T19:16:54.990974+0000","last_active":"2023-10-10T19:28:34.758691+0000","last_peered":"2023-10-10T19:28:34.758691+0000","last_clean":"2023-10-10T19:28:34.758691+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T19:28:34.758691+0000","last_undegraded":"2023-10-10T19:28:34.758691+0000","last_fullsized":"2023-10-10T19:28:34.758691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089059480000000003,"stat_sum":{"num_bytes":796993,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":67,"num_read_kb":26489,"num_write":680,"num_write_kb":31111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"132'369","reported_seq":521,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.547536+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T19:29:04.547536+0000","last_peered":"2023-10-10T19:29:04.547536+0000","last_clean":"2023-10-10T19:29:04.547536+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T19:29:04.547536+0000","last_undegraded":"2023-10-10T19:29:04.547536+0000","last_fullsized":"2023-10-10T19:29:04.547536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":70,"num_read_kb":20656,"num_write":331,"num_write_kb":8737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"131'882","reported_seq":3516,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.406702+0000","last_change":"2023-10-10T19:16:54.870214+0000","last_active":"2023-10-10T19:29:04.406702+0000","last_peered":"2023-10-10T19:29:04.406702+0000","last_clean":"2023-10-10T19:29:04.406702+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T19:29:04.406702+0000","last_undegraded":"2023-10-10T19:29:04.406702+0000","last_fullsized":"2023-10-10T19:29:04.406702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012052205,"stat_sum":{"num_bytes":579663,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":63,"num_read_kb":40064,"num_write":647,"num_write_kb":56806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"132'1301","reported_seq":1456,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:18.043527+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T19:29:18.043527+0000","last_peered":"2023-10-10T19:29:18.043527+0000","last_clean":"2023-10-10T19:29:18.043527+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T19:29:18.043527+0000","last_undegraded":"2023-10-10T19:29:18.043527+0000","last_fullsized":"2023-10-10T19:29:18.043527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1301,"log_dups_size":0,"ondisk_log_size":1301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14953265,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":48,"num_read_kb":53317,"num_write":1709,"num_write_kb":24642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"131'871","reported_seq":3503,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.271899+0000","last_change":"2023-10-10T19:16:54.874295+0000","last_active":"2023-10-10T19:29:04.271899+0000","last_peered":"2023-10-10T19:29:04.271899+0000","last_clean":"2023-10-10T19:29:04.271899+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T19:29:04.271899+0000","last_undegraded":"2023-10-10T19:29:04.271899+0000","last_fullsized":"2023-10-10T19:29:04.271899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015827366999999998,"stat_sum":{"num_bytes":60469,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":58,"num_read_kb":28965,"num_write":636,"num_write_kb":32020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"132'1200","reported_seq":1354,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.409456+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T19:29:04.409456+0000","last_peered":"2023-10-10T19:29:04.409456+0000","last_clean":"2023-10-10T19:29:04.409456+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T19:29:04.409456+0000","last_undegraded":"2023-10-10T19:29:04.409456+0000","last_fullsized":"2023-10-10T19:29:04.409456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":49,"num_read_kb":89747,"num_write":1124,"num_write_kb":25429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"131'914","reported_seq":3542,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.885234+0000","last_change":"2023-10-10T19:16:54.781695+0000","last_active":"2023-10-10T19:28:24.885234+0000","last_peered":"2023-10-10T19:28:24.885234+0000","last_clean":"2023-10-10T19:28:24.885234+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T19:28:24.885234+0000","last_undegraded":"2023-10-10T19:28:24.885234+0000","last_fullsized":"2023-10-10T19:28:24.885234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065052069999999998,"stat_sum":{"num_bytes":415973,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":64,"num_read_kb":10527,"num_write":665,"num_write_kb":20000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"130'311","reported_seq":451,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.683868+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T19:27:29.683868+0000","last_peered":"2023-10-10T19:27:29.683868+0000","last_clean":"2023-10-10T19:27:29.683868+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T19:27:29.683868+0000","last_undegraded":"2023-10-10T19:27:29.683868+0000","last_fullsized":"2023-10-10T19:27:29.683868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194858,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":40985,"num_write":231,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"131'896","reported_seq":3546,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.595223+0000","last_change":"2023-10-10T19:16:54.794460+0000","last_active":"2023-10-10T19:29:04.595223+0000","last_peered":"2023-10-10T19:29:04.595223+0000","last_clean":"2023-10-10T19:29:04.595223+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T19:29:04.595223+0000","last_undegraded":"2023-10-10T19:29:04.595223+0000","last_fullsized":"2023-10-10T19:29:04.595223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016889265000000001,"stat_sum":{"num_bytes":535544,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":67,"num_read_kb":24822,"num_write":684,"num_write_kb":31728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"132'288","reported_seq":432,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.760584+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T19:28:34.760584+0000","last_peered":"2023-10-10T19:28:34.760584+0000","last_clean":"2023-10-10T19:28:34.760584+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T19:28:34.760584+0000","last_undegraded":"2023-10-10T19:28:34.760584+0000","last_fullsized":"2023-10-10T19:28:34.760584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388678,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":27,"num_read_kb":50704,"num_write":218,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"131'948","reported_seq":3575,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.280142+0000","last_change":"2023-10-10T19:16:55.417128+0000","last_active":"2023-10-10T19:29:04.280142+0000","last_peered":"2023-10-10T19:29:04.280142+0000","last_clean":"2023-10-10T19:29:04.280142+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T19:29:04.280142+0000","last_undegraded":"2023-10-10T19:29:04.280142+0000","last_fullsized":"2023-10-10T19:29:04.280142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012585610000000001,"stat_sum":{"num_bytes":184409,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":60,"num_read_kb":19964,"num_write":698,"num_write_kb":27162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"128'310","reported_seq":458,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.683824+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T19:27:29.683824+0000","last_peered":"2023-10-10T19:27:29.683824+0000","last_clean":"2023-10-10T19:27:29.683824+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T19:27:29.683824+0000","last_undegraded":"2023-10-10T19:27:29.683824+0000","last_fullsized":"2023-10-10T19:27:29.683824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":49,"num_read_kb":57404,"num_write":260,"num_write_kb":8464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"131'999","reported_seq":3628,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.407478+0000","last_change":"2023-10-10T19:16:54.794658+0000","last_active":"2023-10-10T19:29:04.407478+0000","last_peered":"2023-10-10T19:29:04.407478+0000","last_clean":"2023-10-10T19:29:04.407478+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T19:29:04.407478+0000","last_undegraded":"2023-10-10T19:29:04.407478+0000","last_fullsized":"2023-10-10T19:29:04.407478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016837566000000002,"stat_sum":{"num_bytes":339858,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":67,"num_read_kb":28062,"num_write":742,"num_write_kb":23663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"131'891","reported_seq":3551,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077900+0000","last_change":"2023-10-10T19:16:54.782474+0000","last_active":"2023-10-10T19:29:04.077900+0000","last_peered":"2023-10-10T19:29:04.077900+0000","last_clean":"2023-10-10T19:29:04.077900+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T19:29:04.077900+0000","last_undegraded":"2023-10-10T19:29:04.077900+0000","last_fullsized":"2023-10-10T19:29:04.077900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074078640000000001,"stat_sum":{"num_bytes":492374,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":77,"num_read_kb":25056,"num_write":669,"num_write_kb":32074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"131'973","reported_seq":3594,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.287531+0000","last_change":"2023-10-10T19:16:54.779827+0000","last_active":"2023-10-10T19:29:04.287531+0000","last_peered":"2023-10-10T19:29:04.287531+0000","last_clean":"2023-10-10T19:29:04.287531+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T19:29:04.287531+0000","last_undegraded":"2023-10-10T19:29:04.287531+0000","last_fullsized":"2023-10-10T19:29:04.287531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050938800000000003,"stat_sum":{"num_bytes":392421,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":69,"num_read_kb":25013,"num_write":694,"num_write_kb":37199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"131'956","reported_seq":3603,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.272570+0000","last_change":"2023-10-10T19:16:54.781092+0000","last_active":"2023-10-10T19:29:04.272570+0000","last_peered":"2023-10-10T19:29:04.272570+0000","last_clean":"2023-10-10T19:29:04.272570+0000","last_became_active":"2023-10-10T18:46:00.606420+0000","last_became_peered":"2023-10-10T18:46:00.606420+0000","last_unstale":"2023-10-10T19:29:04.272570+0000","last_undegraded":"2023-10-10T19:29:04.272570+0000","last_fullsized":"2023-10-10T19:29:04.272570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:47.189801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064370249999999999,"stat_sum":{"num_bytes":69276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":90,"num_read_kb":45570,"num_write":703,"num_write_kb":52466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"131'929","reported_seq":3555,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.288777+0000","last_change":"2023-10-10T19:16:54.983512+0000","last_active":"2023-10-10T19:29:04.288777+0000","last_peered":"2023-10-10T19:29:04.288777+0000","last_clean":"2023-10-10T19:29:04.288777+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T19:29:04.288777+0000","last_undegraded":"2023-10-10T19:29:04.288777+0000","last_fullsized":"2023-10-10T19:29:04.288777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014096529,"stat_sum":{"num_bytes":398978,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":70,"num_read_kb":16093,"num_write":658,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"131'895","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.417096+0000","last_change":"2023-10-10T19:16:54.875626+0000","last_active":"2023-10-10T19:29:04.417096+0000","last_peered":"2023-10-10T19:29:04.417096+0000","last_clean":"2023-10-10T19:29:04.417096+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T19:29:04.417096+0000","last_undegraded":"2023-10-10T19:29:04.417096+0000","last_fullsized":"2023-10-10T19:29:04.417096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016808101999999998,"stat_sum":{"num_bytes":587511,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":68,"num_read_kb":48841,"num_write":661,"num_write_kb":55904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"131'962","reported_seq":3564,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.760146+0000","last_change":"2023-10-10T19:16:54.918525+0000","last_active":"2023-10-10T19:28:34.760146+0000","last_peered":"2023-10-10T19:28:34.760146+0000","last_clean":"2023-10-10T19:28:34.760146+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T19:28:34.760146+0000","last_undegraded":"2023-10-10T19:28:34.760146+0000","last_fullsized":"2023-10-10T19:28:34.760146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068654069999999996,"stat_sum":{"num_bytes":680481,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":59,"num_read_kb":41116,"num_write":666,"num_write_kb":48352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"131'952","reported_seq":3574,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.272886+0000","last_change":"2023-10-10T19:16:54.783685+0000","last_active":"2023-10-10T19:29:04.272886+0000","last_peered":"2023-10-10T19:29:04.272886+0000","last_clean":"2023-10-10T19:29:04.272886+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T19:29:04.272886+0000","last_undegraded":"2023-10-10T19:29:04.272886+0000","last_fullsized":"2023-10-10T19:29:04.272886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081260870000000006,"stat_sum":{"num_bytes":4687487,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":70,"num_read_kb":15871,"num_write":681,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"131'922","reported_seq":3558,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.267637+0000","last_change":"2023-10-10T19:16:54.877012+0000","last_active":"2023-10-10T19:29:04.267637+0000","last_peered":"2023-10-10T19:29:04.267637+0000","last_clean":"2023-10-10T19:29:04.267637+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T19:29:04.267637+0000","last_undegraded":"2023-10-10T19:29:04.267637+0000","last_fullsized":"2023-10-10T19:29:04.267637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":922,"log_dups_size":0,"ondisk_log_size":922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017981724000000001,"stat_sum":{"num_bytes":798509,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":72,"num_read_kb":44513,"num_write":661,"num_write_kb":47223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"131'898","reported_seq":3541,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.478410+0000","last_change":"2023-10-10T19:16:55.415035+0000","last_active":"2023-10-10T19:29:04.478410+0000","last_peered":"2023-10-10T19:29:04.478410+0000","last_clean":"2023-10-10T19:29:04.478410+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T19:29:04.478410+0000","last_undegraded":"2023-10-10T19:29:04.478410+0000","last_fullsized":"2023-10-10T19:29:04.478410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010756557999999999,"stat_sum":{"num_bytes":615982,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":71,"num_read_kb":21097,"num_write":676,"num_write_kb":25400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"131'878","reported_seq":3515,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758569+0000","last_change":"2023-10-10T19:16:54.872190+0000","last_active":"2023-10-10T19:28:34.758569+0000","last_peered":"2023-10-10T19:28:34.758569+0000","last_clean":"2023-10-10T19:28:34.758569+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T19:28:34.758569+0000","last_undegraded":"2023-10-10T19:28:34.758569+0000","last_fullsized":"2023-10-10T19:28:34.758569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014012482999999999,"stat_sum":{"num_bytes":662827,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":69,"num_read_kb":49258,"num_write":627,"num_write_kb":56876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"131'935","reported_seq":3568,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.290882+0000","last_change":"2023-10-10T19:16:55.413906+0000","last_active":"2023-10-10T19:29:04.290882+0000","last_peered":"2023-10-10T19:29:04.290882+0000","last_clean":"2023-10-10T19:29:04.290882+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T19:29:04.290882+0000","last_undegraded":"2023-10-10T19:29:04.290882+0000","last_fullsized":"2023-10-10T19:29:04.290882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097177140000000006,"stat_sum":{"num_bytes":515125,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":59,"num_read_kb":20609,"num_write":704,"num_write_kb":28747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"131'873","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.482135+0000","last_change":"2023-10-10T19:16:54.983331+0000","last_active":"2023-10-10T19:29:04.482135+0000","last_peered":"2023-10-10T19:29:04.482135+0000","last_clean":"2023-10-10T19:29:04.482135+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T19:29:04.482135+0000","last_undegraded":"2023-10-10T19:29:04.482135+0000","last_fullsized":"2023-10-10T19:29:04.482135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013935637000000001,"stat_sum":{"num_bytes":335358,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":51992,"num_write":654,"num_write_kb":56542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"131'994","reported_seq":3636,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077991+0000","last_change":"2023-10-10T19:16:54.784919+0000","last_active":"2023-10-10T19:29:04.077991+0000","last_peered":"2023-10-10T19:29:04.077991+0000","last_clean":"2023-10-10T19:29:04.077991+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T19:29:04.077991+0000","last_undegraded":"2023-10-10T19:29:04.077991+0000","last_fullsized":"2023-10-10T19:29:04.077991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088155260000000006,"stat_sum":{"num_bytes":221031,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":37312,"num_write":732,"num_write_kb":49119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":43035717,"num_objects":1543,"num_object_clones":0,"num_object_copies":3086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":4408,"num_read_kb":1953419,"num_write":43238,"num_write_kb":2509746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":63954944,"data_stored":86071434,"data_compressed":25961548,"data_compressed_allocated":28180480,"data_compressed_original":58130432,"omap_allocated":0,"internal_metadata":0},"log_size":59428,"ondisk_log_size":59428,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":328210382,"num_objects":474,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":2858,"num_read_kb":1531013,"num_write":19009,"num_write_kb":461126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":338,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179290112,"data_stored":656420764,"data_compressed":84754974,"data_compressed_allocated":109232128,"data_compressed_original":586604544,"omap_allocated":0,"internal_metadata":0},"log_size":19669,"ondisk_log_size":19669,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575383,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346716,"kb_used_data":45332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94025124,"statfs":{"total":96636764160,"available":96281726976,"internally_reserved":0,"allocated":46419968,"data_stored":131329189,"data_compressed":19664789,"data_compressed_allocated":24576000,"data_compressed_original":111382528,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.53700000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.78400000000000003,"15min":1.018},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.32100000000000001,"5min":0.316,"15min":0.25},"max":{"1min":0.76500000000000001,"5min":0.80700000000000005,"15min":2.153},"last":0.36799999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.33100000000000002,"5min":0.318,"15min":0.26200000000000001},"max":{"1min":0.79500000000000004,"5min":0.82499999999999996,"15min":1.802},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":2.2170000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.623,"15min":0.61199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83899999999999997,"5min":2.871,"15min":2.871},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63400000000000001,"15min":0.63800000000000001},"min":{"1min":0.48299999999999998,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":7.6200000000000001},"last":0.55400000000000005}]},{"osd":3,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.874,"15min":1.0349999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.60099999999999998},"min":{"1min":0.375,"5min":0.373,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":7.6349999999999998},"last":0.502}]},{"osd":4,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.36599999999999999,"5min":0.307,"15min":0.28000000000000003},"max":{"1min":0.64000000000000001,"5min":0.72499999999999998,"15min":1.089},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.33800000000000002,"5min":0.32600000000000001,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.9419999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":0.629,"5min":0.752,"15min":1.71},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59299999999999997,"15min":0.57199999999999995},"min":{"1min":0.41799999999999998,"5min":0.378,"15min":0.311},"max":{"1min":0.80500000000000005,"5min":0.85099999999999998,"15min":1.177},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.31},"max":{"1min":0.76100000000000001,"5min":0.78000000000000003,"15min":0.872},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.33900000000000002,"15min":0.29699999999999999},"max":{"1min":0.72299999999999998,"5min":0.78100000000000003,"15min":2.2559999999999998},"last":0.57299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575382,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332164,"kb_used_data":34876,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94039676,"statfs":{"total":96636764160,"available":96296628224,"internally_reserved":0,"allocated":35713024,"data_stored":90587605,"data_compressed":11613979,"data_compressed_allocated":14393344,"data_compressed_original":70787072,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.31},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.216},"max":{"1min":0.71799999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42499999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.77200000000000002,"5min":0.77600000000000002,"15min":0.85299999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.58399999999999996,"15min":0.56599999999999995},"min":{"1min":0.46700000000000003,"5min":0.29599999999999999,"15min":0.222},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86799999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Tue Oct 10 19:28:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58199999999999996,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.312,"15min":0.30499999999999999},"max":{"1min":0.69899999999999995,"5min":0.77800000000000002,"15min":0.90500000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56899999999999995,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29999999999999999},"max":{"1min":0.71299999999999997,"5min":0.76500000000000001,"15min":0.97599999999999998},"last":0.51300000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57399999999999995,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76600000000000001,"5min":0.82799999999999996,"15min":0.88200000000000001},"last":0.628},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.35999999999999999},"max":{"1min":0.73599999999999999,"5min":0.85699999999999998,"15min":0.95099999999999996},"last":0.73199999999999998}]},{"osd":4,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.46400000000000002},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.248},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.749},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.52500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.73599999999999999,"15min":0.879},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.39700000000000002,"5min":0.30599999999999999,"15min":0.26400000000000001},"max":{"1min":0.63100000000000001,"5min":0.76800000000000002,"15min":0.99299999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.38200000000000001,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.88900000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.38,"5min":0.376,"15min":0.27700000000000002},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.86199999999999999},"last":0.60899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575381,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339956,"kb_used_data":30380,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94031884,"statfs":{"total":96636764160,"available":96288649216,"internally_reserved":0,"allocated":31109120,"data_stored":109263664,"data_compressed":16514475,"data_compressed_allocated":20389888,"data_compressed_original":99950592,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.314,"15min":0.313},"min":{"1min":0.187,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.432,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.505},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.66700000000000004,"15min":0.71699999999999997},"last":0.626}]},{"osd":2,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34599999999999997,"15min":0.35199999999999998},"min":{"1min":0.23799999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55200000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25,"15min":0.19900000000000001},"max":{"1min":0.54000000000000004,"5min":0.64900000000000002,"15min":0.73899999999999999},"last":0.56699999999999995}]},{"osd":3,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.45300000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.32600000000000001,"15min":0.27700000000000002},"max":{"1min":0.59699999999999998,"5min":0.72899999999999998,"15min":0.84299999999999997},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":1.175},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58699999999999997,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.87,"5min":0.87,"15min":2.2519999999999998},"last":0.80800000000000005}]},{"osd":5,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.76700000000000002,"15min":0.90900000000000003},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":1.0009999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.40500000000000003,"15min":0.37},"max":{"1min":0.83199999999999996,"5min":0.86199999999999999,"15min":1.038},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.123},"last":0.96699999999999997}]},{"osd":7,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60299999999999998,"15min":0.62},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":5.2720000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":5.2599999999999998},"last":1.0389999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575383,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335236,"kb_used_data":29756,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94036604,"statfs":{"total":96636764160,"available":96293482496,"internally_reserved":0,"allocated":30470144,"data_stored":81989013,"data_compressed":12938492,"data_compressed_allocated":16048128,"data_compressed_original":69398528,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.35799999999999998,"15min":0.35699999999999998},"min":{"1min":0.245,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":0.47699999999999998,"5min":0.57399999999999995,"15min":0.66500000000000004},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42499999999999999,"15min":0.41999999999999998},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.20300000000000001},"max":{"1min":0.60299999999999998,"5min":0.66000000000000003,"15min":0.66500000000000004},"last":0.35299999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35999999999999999,"15min":0.373},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.49299999999999999,"5min":0.57799999999999996,"15min":2.028},"last":0.308},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54900000000000004,"5min":0.61299999999999999,"15min":0.77300000000000002},"last":0.27300000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":0.54000000000000004,"5min":0.59699999999999998,"15min":0.79100000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.32500000000000001,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":0.70499999999999996,"5min":0.70599999999999996,"15min":0.80300000000000005},"last":0.375}]},{"osd":4,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.373,"15min":0.26000000000000001},"max":{"1min":0.80200000000000005,"5min":0.97599999999999998,"15min":1.9610000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.38400000000000001},"max":{"1min":0.82599999999999996,"5min":0.999,"15min":0.999},"last":0.53700000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.55100000000000005,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.79600000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.437},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.626,"15min":0.623},"min":{"1min":0.51600000000000001,"5min":0.38600000000000001,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.873,"15min":0.96799999999999997},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.52200000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":0.75800000000000001,"5min":0.96299999999999997,"15min":2.0019999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.621,"15min":0.623},"min":{"1min":0.501,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":0.84999999999999998,"5min":0.94099999999999995,"15min":0.95399999999999996},"last":0.43099999999999999}]},{"osd":7,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.40400000000000003,"15min":0.39100000000000001},"max":{"1min":0.77000000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41799999999999998},"max":{"1min":0.879,"5min":0.89500000000000002,"15min":2.0539999999999998},"last":0.498}]}]},{"osd":2,"up_from":13,"seq":55834575383,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318528,"kb_used_data":17144,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94053312,"statfs":{"total":96636764160,"available":96310591488,"internally_reserved":0,"allocated":17555456,"data_stored":56587147,"data_compressed":8367326,"data_compressed_allocated":10403840,"data_compressed_original":50835456,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34399999999999997,"15min":0.33500000000000002},"min":{"1min":0.224,"5min":0.219,"15min":0.186},"max":{"1min":0.45000000000000001,"5min":0.55400000000000005,"15min":0.60799999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.53500000000000003,"5min":0.65100000000000002,"15min":0.67500000000000004},"last":0.38}]},{"osd":1,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20999999999999999},"max":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.65700000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.73999999999999999},"last":0.443}]},{"osd":3,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.24399999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.42499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.23899999999999999},"max":{"1min":0.59099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.76900000000000002},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.26600000000000001},"max":{"1min":0.751,"5min":0.95599999999999996,"15min":48.729999999999997},"last":0.499},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.25900000000000001},"max":{"1min":0.74099999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.48599999999999999}]},{"osd":5,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63,"15min":0.628},"min":{"1min":0.436,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":1.02,"15min":1.02},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.32400000000000001},"max":{"1min":0.81000000000000005,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.78900000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":1.0760000000000001,"15min":48.685000000000002},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.38800000000000001},"max":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":0.97899999999999998},"last":0.51600000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64800000000000002,"15min":0.80700000000000005},"min":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.38700000000000001},"max":{"1min":0.85799999999999998,"5min":1.052,"15min":48.710000000000001},"last":0.748},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.034,"15min":1.1459999999999999},"last":0.55700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575382,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345608,"kb_used_data":36032,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94026232,"statfs":{"total":96636764160,"available":96282861568,"internally_reserved":0,"allocated":36896768,"data_stored":96360757,"data_compressed":13146180,"data_compressed_allocated":16756736,"data_compressed_original":78069760,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.37},"min":{"1min":0.23499999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.47599999999999998,"5min":0.64900000000000002,"15min":5.0670000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.371,"15min":0.38700000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.189},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.7799999999999998},"last":0.223}]},{"osd":1,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.36499999999999999,"15min":0.377},"min":{"1min":0.27200000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.628,"5min":0.628,"15min":2.758},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.41199999999999998},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51700000000000002,"5min":0.58399999999999996,"15min":0.64500000000000002},"last":0.29099999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28000000000000003,"15min":0.25},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":1.333},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.41899999999999998},"min":{"1min":0.30599999999999999,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.98699999999999999},"last":0.307}]},{"osd":4,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.48299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.879,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46200000000000002,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.23},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.45100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.2929999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.36199999999999999},"max":{"1min":0.90200000000000002,"5min":0.93600000000000005,"15min":1.103},"last":0.53900000000000003}]},{"osd":6,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63500000000000001,"15min":0.64000000000000001},"min":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.39900000000000002},"max":{"1min":0.91000000000000003,"5min":0.94199999999999995,"15min":1.198},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.49099999999999999,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.94899999999999995,"5min":0.95199999999999996,"15min":2.7309999999999999},"last":0.504}]},{"osd":7,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.46700000000000003,"5min":0.46500000000000002,"15min":0.39400000000000002},"max":{"1min":0.85099999999999998,"5min":0.97999999999999998,"15min":2.6619999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.66900000000000004},"min":{"1min":0.50800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.59699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575382,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338568,"kb_used_data":33088,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94033272,"statfs":{"total":96636764160,"available":96290070528,"internally_reserved":0,"allocated":33882112,"data_stored":101437325,"data_compressed":15509067,"data_compressed_allocated":19054592,"data_compressed_original":88412160,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56100000000000005,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.21199999999999999},"max":{"1min":0.746,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.23699999999999999},"max":{"1min":0.71899999999999997,"5min":0.86799999999999999,"15min":0.88900000000000001},"last":0.56999999999999995}]},{"osd":1,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":0.68300000000000005,"5min":0.71199999999999997,"15min":1.5920000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.55800000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.69899999999999995,"5min":0.749,"15min":1.0449999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.30199999999999999},"max":{"1min":0.752,"5min":0.78200000000000003,"15min":1.615},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.36699999999999999},"max":{"1min":0.73799999999999999,"5min":0.85799999999999998,"15min":1.071},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.58099999999999996},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.76200000000000001,"5min":0.85199999999999998,"15min":0.85799999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.28199999999999997},"max":{"1min":0.81799999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.60699999999999998}]},{"osd":5,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":1.087},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56699999999999995,"15min":0.55500000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88,"5min":0.88,"15min":1.111},"last":0.52700000000000002}]},{"osd":6,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56799999999999995,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.29799999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.5580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":0.90400000000000003,"15min":0.98799999999999999},"last":0.50800000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.312},"max":{"1min":1.016,"5min":1.016,"15min":1.0620000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64900000000000002,"15min":0.64200000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.33700000000000002},"max":{"1min":0.873,"5min":0.873,"15min":1.1160000000000001},"last":0.69199999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575382,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328896,"kb_used_data":23416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94042944,"statfs":{"total":96636764160,"available":96299974656,"internally_reserved":0,"allocated":23977984,"data_stored":83205478,"data_compressed":13026742,"data_compressed_allocated":15880192,"data_compressed_original":77045760,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.438,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":0.69499999999999995,"5min":0.75,"15min":0.82199999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56499999999999995,"15min":0.59699999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.63800000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46300000000000002}]},{"osd":1,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57599999999999996,"15min":0.55900000000000005},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.29699999999999999},"max":{"1min":0.745,"5min":0.79400000000000004,"15min":0.79800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.56999999999999995},"min":{"1min":0.48199999999999998,"5min":0.39200000000000002,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.432}]},{"osd":2,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.54000000000000004},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.30499999999999999},"max":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.442},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.39500000000000002,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.74099999999999999,"5min":0.75600000000000001,"15min":0.79000000000000004},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.86099999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59499999999999997,"15min":0.58799999999999997},"min":{"1min":0.46899999999999997,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.81799999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.50900000000000001}]},{"osd":4,"last update":"Tue Oct 10 19:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.45700000000000002},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.251},"max":{"1min":0.69999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58099999999999996,"15min":0.55400000000000005},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.23000000000000001},"max":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.49199999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.497},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.26700000000000002},"max":{"1min":0.72199999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53800000000000003},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.32100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.376}]},{"osd":7,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76600000000000001,"5min":0.86299999999999999,"15min":0.89400000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.27700000000000002},"max":{"1min":0.80500000000000005,"5min":0.84599999999999997,"15min":0.90800000000000003},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20283392,"data_stored":68677505,"data_compressed":9126818,"data_compressed_allocated":11898880,"data_compressed_original":60325888,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24403968,"data_stored":99763638,"data_compressed":12930657,"data_compressed_allocated":16519168,"data_compressed_original":91906048,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":51052075,"data_compressed":6450195,"data_compressed_allocated":8323072,"data_compressed_original":46788608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25817088,"data_stored":83490082,"data_compressed":10157145,"data_compressed_allocated":13479936,"data_compressed_original":71184384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24055808,"data_stored":89060236,"data_compressed":11845848,"data_compressed_allocated":15089664,"data_compressed_original":80117760,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16453632,"data_stored":73683110,"data_compressed":9606977,"data_compressed_allocated":12140544,"data_compressed_original":69410816,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21950464,"data_stored":75910818,"data_compressed":9436899,"data_compressed_allocated":12021760,"data_compressed_original":66007040,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":33710080,"data_stored":114783300,"data_compressed":15200435,"data_compressed_allocated":19759104,"data_compressed_original":100864000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8540160,"data_stored":11835204,"data_compressed":3779410,"data_compressed_allocated":4104192,"data_compressed_original":8499200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5124096,"data_stored":8614212,"data_compressed":3583818,"data_compressed_allocated":3870720,"data_compressed_original":8044544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3358720,"data_stored":4649136,"data_compressed":1917131,"data_compressed_allocated":2080768,"data_compressed_original":4046848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9498624,"data_stored":11984861,"data_compressed":2989035,"data_compressed_allocated":3276800,"data_compressed_original":6885376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8245248,"data_stored":11491153,"data_compressed":3663219,"data_compressed_allocated":3964928,"data_compressed_original":8294400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5943296,"data_stored":8636432,"data_compressed":3419765,"data_compressed_allocated":3739648,"data_compressed_original":7634944,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12181504,"data_stored":13790851,"data_compressed":2177080,"data_compressed_allocated":2371584,"data_compressed_original":4780032,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":15069585,"data_compressed":4432090,"data_compressed_allocated":4771840,"data_compressed_original":9945088,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T19:29:23.647 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-10T19:29:23.648 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T19:29:23.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.794+0000 7f9616f6d700 1 Processor -- start 2023-10-10T19:29:23.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.794+0000 7f9616f6d700 1 -- start start 2023-10-10T19:29:23.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9616f6d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f961008a100 0x7f9610159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9616f6d700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9610067fa0 con 0x7f961014fd90 2023-10-10T19:29:23.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9616f6d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9610156b70 con 0x7f96100cd040 2023-10-10T19:29:23.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9616f6d700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96100c5600 con 0x7f961008a100 2023-10-10T19:29:23.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9615f6b700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f961014fd90 0x7f96100c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46416/0 (socket says 172.21.15.175:46416) 2023-10-10T19:29:23.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9615f6b700 1 -- 172.21.15.175:0/1958405366 learned_addr learned my addr 172.21.15.175:0/1958405366 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:23.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f961576a700 1 --2- 172.21.15.175:0/1958405366 >> v2:172.21.15.191:3300/0 conn(0x7f961008a100 0x7f9610159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9614f69700 1 -- 172.21.15.175:0/1958405366 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825180595 0 0) 0x7f9610067fa0 con 0x7f961014fd90 2023-10-10T19:29:23.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9614f69700 1 -- 172.21.15.175:0/1958405366 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f8003570 con 0x7f961014fd90 2023-10-10T19:29:23.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9614f69700 1 -- 172.21.15.175:0/1958405366 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036350719 0 0) 0x7f9610156b70 con 0x7f96100cd040 2023-10-10T19:29:23.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f9614f69700 1 -- 172.21.15.175:0/1958405366 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610067fa0 con 0x7f96100cd040 2023-10-10T19:29:23.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f961576a700 1 -- 172.21.15.175:0/1958405366 >> v1:172.21.15.191:6789/0 conn(0x7f96100cd040 legacy=0x7f96100684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f961576a700 1 -- 172.21.15.175:0/1958405366 >> v1:172.21.15.175:6789/0 conn(0x7f961014fd90 legacy=0x7f96100c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.795+0000 7f961576a700 1 -- 172.21.15.175:0/1958405366 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9610150160 con 0x7f961008a100 2023-10-10T19:29:23.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f961576a700 1 --2- 172.21.15.175:0/1958405366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 0x7f9610159c30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f96040098e0 tx=0x7f96040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=258f49552df25b65 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9614f69700 1 -- 172.21.15.175:0/1958405366 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604011630 con 0x7f961008a100 2023-10-10T19:29:23.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9614f69700 1 -- 172.21.15.175:0/1958405366 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9604011790 con 0x7f961008a100 2023-10-10T19:29:23.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9614f69700 1 -- 172.21.15.175:0/1958405366 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604011b10 con 0x7f961008a100 2023-10-10T19:29:23.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9616f6d700 1 -- 172.21.15.175:0/1958405366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 msgr2=0x7f9610159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9616f6d700 1 --2- 172.21.15.175:0/1958405366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 0x7f9610159c30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f96040098e0 tx=0x7f96040091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9616f6d700 1 -- 172.21.15.175:0/1958405366 shutdown_connections 2023-10-10T19:29:23.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9616f6d700 1 --2- 172.21.15.175:0/1958405366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 0x7f9610159c30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9616f6d700 1 -- 172.21.15.175:0/1958405366 >> 172.21.15.175:0/1958405366 conn(0x7f9610063d00 msgr2=0x7f96100cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:23.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9616f6d700 1 -- 172.21.15.175:0/1958405366 shutdown_connections 2023-10-10T19:29:23.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.796+0000 7f9616f6d700 1 -- 172.21.15.175:0/1958405366 wait complete. 2023-10-10T19:29:23.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.797+0000 7f9616f6d700 1 Processor -- start 2023-10-10T19:29:23.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.797+0000 7f9616f6d700 1 -- start start 2023-10-10T19:29:23.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f9616f6d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 0x7f96100c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f9616f6d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96100cd040 0x7f9610166720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f9616f6d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f961014fd90 0x7f96100c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f9616f6d700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9610166c60 con 0x7f961014fd90 2023-10-10T19:29:23.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f9616f6d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96100c85d0 con 0x7f961008a100 2023-10-10T19:29:23.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f9616f6d700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9610150500 con 0x7f96100cd040 2023-10-10T19:29:23.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f961676c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f961014fd90 0x7f96100c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f961676c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f961014fd90 0x7f96100c72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54472/0 (socket says 172.21.15.175:54472) 2023-10-10T19:29:23.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f961676c700 1 -- 172.21.15.175:0/2262806369 learned_addr learned my addr 172.21.15.175:0/2262806369 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:23.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f9615f6b700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 0x7f96100c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f961576a700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96100cd040 0x7f9610166720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f961676c700 1 -- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96100cd040 msgr2=0x7f9610166720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.798+0000 7f961676c700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96100cd040 0x7f9610166720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f961676c700 1 -- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 msgr2=0x7f96100c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:23.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f961676c700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 0x7f96100c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:23.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f961676c700 1 -- 172.21.15.175:0/2262806369 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9604009420 con 0x7f961014fd90 2023-10-10T19:29:23.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f9615f6b700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 0x7f96100c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:23.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f961676c700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f961014fd90 0x7f96100c72b0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f960800c590 tx=0x7f9608007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f960dffb700 1 -- 172.21.15.175:0/2262806369 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608016070 con 0x7f961014fd90 2023-10-10T19:29:23.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f960dffb700 1 -- 172.21.15.175:0/2262806369 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9608011590 con 0x7f961014fd90 2023-10-10T19:29:23.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f960dffb700 1 -- 172.21.15.175:0/2262806369 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608011970 con 0x7f961014fd90 2023-10-10T19:29:23.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96101507a0 con 0x7f961014fd90 2023-10-10T19:29:23.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.799+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96100c8260 con 0x7f961014fd90 2023-10-10T19:29:23.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.800+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9610067570 con 0x7f961014fd90 2023-10-10T19:29:23.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.800+0000 7f960dffb700 1 -- 172.21.15.175:0/2262806369 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9608012420 con 0x7f961014fd90 2023-10-10T19:29:23.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.801+0000 7f960dffb700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f95fc066f30 0x7f95fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:23.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.801+0000 7f960dffb700 1 -- 172.21.15.175:0/2262806369 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f96080849d0 con 0x7f961014fd90 2023-10-10T19:29:23.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.801+0000 7f9615f6b700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f95fc066f30 0x7f95fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:23.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.801+0000 7f9615f6b700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f95fc066f30 0x7f95fc0693e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9600005c50 tx=0x7f9600009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:23.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.805+0000 7f960dffb700 1 -- 172.21.15.175:0/2262806369 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f96080513c0 con 0x7f961014fd90 2023-10-10T19:29:23.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.987+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f96100c8710 con 0x7f95fc066f30 2023-10-10T19:29:23.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.995+0000 7f960dffb700 1 -- 172.21.15.175:0/2262806369 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294172 (secure 0 0 0) 0x7f96100c8710 con 0x7f95fc066f30 2023-10-10T19:29:23.999 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:24.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f95fc066f30 msgr2=0x7f95fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f95fc066f30 0x7f95fc0693e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9600005c50 tx=0x7f9600009040 comp rx=0 tx=0).stop 2023-10-10T19:29:24.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f961014fd90 msgr2=0x7f96100c72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f961014fd90 0x7f96100c72b0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f960800c590 tx=0x7f9608007d80 comp rx=0 tx=0).stop 2023-10-10T19:29:24.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 shutdown_connections 2023-10-10T19:29:24.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f95fc066f30 0x7f95fc0693e0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f961014fd90 0x7f96100c72b0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961008a100 0x7f96100c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 --2- 172.21.15.175:0/2262806369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96100cd040 0x7f9610166720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 >> 172.21.15.175:0/2262806369 conn(0x7f9610063d00 msgr2=0x7f96100c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:24.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 shutdown_connections 2023-10-10T19:29:24.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:23.998+0000 7f9616f6d700 1 -- 172.21.15.175:0/2262806369 wait complete. 2023-10-10T19:29:24.005 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T19:29:24.011 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":1455,"stamp":"2023-10-10T19:29:23.817608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371836467,"num_objects":2019,"num_object_clones":0,"num_object_copies":4038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":7382,"num_read_kb":3484532,"num_write":62406,"num_write_kb":2973288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":338,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79214,"ondisk_log_size":79214,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2685672,"kb_used_data":250024,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752289048,"statfs":{"total":773094113280,"available":770343985152,"internally_reserved":0,"allocated":256024576,"data_stored":750760178,"data_compressed":110781050,"data_compressed_allocated":137502720,"data_compressed_original":645881856,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1436897,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":1564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003444"},"pg_stats":[{"pgid":"3.3f","version":"131'943","reported_seq":3563,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.259330+0000","last_change":"2023-10-10T19:16:54.796013+0000","last_active":"2023-10-10T19:28:34.259330+0000","last_peered":"2023-10-10T19:28:34.259330+0000","last_clean":"2023-10-10T19:28:34.259330+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T19:28:34.259330+0000","last_undegraded":"2023-10-10T19:28:34.259330+0000","last_fullsized":"2023-10-10T19:28:34.259330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017995507000000001,"stat_sum":{"num_bytes":795566,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":62,"num_read_kb":8068,"num_write":667,"num_write_kb":16726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"131'993","reported_seq":3612,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.284503+0000","last_change":"2023-10-10T19:16:54.920511+0000","last_active":"2023-10-10T19:29:04.284503+0000","last_peered":"2023-10-10T19:29:04.284503+0000","last_clean":"2023-10-10T19:29:04.284503+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T19:29:04.284503+0000","last_undegraded":"2023-10-10T19:29:04.284503+0000","last_fullsized":"2023-10-10T19:29:04.284503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083569260000000006,"stat_sum":{"num_bytes":832331,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":62,"num_read_kb":28805,"num_write":733,"num_write_kb":46104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"131'937","reported_seq":3564,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.912051+0000","last_change":"2023-10-10T19:16:54.980353+0000","last_active":"2023-10-10T19:28:33.912051+0000","last_peered":"2023-10-10T19:28:33.912051+0000","last_clean":"2023-10-10T19:28:33.912051+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T19:28:33.912051+0000","last_undegraded":"2023-10-10T19:28:33.912051+0000","last_fullsized":"2023-10-10T19:28:33.912051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":937,"log_dups_size":0,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011353221,"stat_sum":{"num_bytes":419659,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":65,"num_read_kb":20680,"num_write":679,"num_write_kb":24555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"131'941","reported_seq":3555,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.942186+0000","last_change":"2023-10-10T19:16:55.412877+0000","last_active":"2023-10-10T19:28:24.942186+0000","last_peered":"2023-10-10T19:28:24.942186+0000","last_clean":"2023-10-10T19:28:24.942186+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T19:28:24.942186+0000","last_undegraded":"2023-10-10T19:28:24.942186+0000","last_fullsized":"2023-10-10T19:28:24.942186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088633549999999998,"stat_sum":{"num_bytes":476985,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":62,"num_read_kb":30965,"num_write":662,"num_write_kb":37535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"131'971","reported_seq":3601,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.424759+0000","last_change":"2023-10-10T19:16:54.790962+0000","last_active":"2023-10-10T19:29:04.424759+0000","last_peered":"2023-10-10T19:29:04.424759+0000","last_clean":"2023-10-10T19:29:04.424759+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T19:29:04.424759+0000","last_undegraded":"2023-10-10T19:29:04.424759+0000","last_fullsized":"2023-10-10T19:29:04.424759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016189688000000001,"stat_sum":{"num_bytes":142623,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":75,"num_read_kb":17848,"num_write":693,"num_write_kb":31284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"131'918","reported_seq":3544,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704394+0000","last_change":"2023-10-10T19:16:55.414924+0000","last_active":"2023-10-10T19:28:34.704394+0000","last_peered":"2023-10-10T19:28:34.704394+0000","last_clean":"2023-10-10T19:28:34.704394+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T19:28:34.704394+0000","last_undegraded":"2023-10-10T19:28:34.704394+0000","last_fullsized":"2023-10-10T19:28:34.704394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010631753000000001,"stat_sum":{"num_bytes":567917,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":63,"num_read_kb":20939,"num_write":659,"num_write_kb":24217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"131'968","reported_seq":3589,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078564+0000","last_change":"2023-10-10T19:16:54.789065+0000","last_active":"2023-10-10T19:29:04.078564+0000","last_peered":"2023-10-10T19:29:04.078564+0000","last_clean":"2023-10-10T19:29:04.078564+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T19:29:04.078564+0000","last_undegraded":"2023-10-10T19:29:04.078564+0000","last_fullsized":"2023-10-10T19:29:04.078564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014667316999999999,"stat_sum":{"num_bytes":1084188,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":53600,"num_write":676,"num_write_kb":52841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"131'852","reported_seq":3490,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077662+0000","last_change":"2023-10-10T19:16:54.870456+0000","last_active":"2023-10-10T19:29:04.077662+0000","last_peered":"2023-10-10T19:29:04.077662+0000","last_clean":"2023-10-10T19:29:04.077662+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T19:29:04.077662+0000","last_undegraded":"2023-10-10T19:29:04.077662+0000","last_fullsized":"2023-10-10T19:29:04.077662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":852,"log_dups_size":0,"ondisk_log_size":852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012402231,"stat_sum":{"num_bytes":449638,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":58,"num_read_kb":28110,"num_write":635,"num_write_kb":35676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"131'959","reported_seq":3586,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078384+0000","last_change":"2023-10-10T19:16:54.795632+0000","last_active":"2023-10-10T19:29:04.078384+0000","last_peered":"2023-10-10T19:29:04.078384+0000","last_clean":"2023-10-10T19:29:04.078384+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T19:29:04.078384+0000","last_undegraded":"2023-10-10T19:29:04.078384+0000","last_fullsized":"2023-10-10T19:29:04.078384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017699890999999999,"stat_sum":{"num_bytes":30143,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":76,"num_read_kb":37435,"num_write":691,"num_write_kb":40783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"131'935","reported_seq":3561,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.401310+0000","last_change":"2023-10-10T19:16:54.783622+0000","last_active":"2023-10-10T19:29:04.401310+0000","last_peered":"2023-10-10T19:29:04.401310+0000","last_clean":"2023-10-10T19:29:04.401310+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T19:29:04.401310+0000","last_undegraded":"2023-10-10T19:29:04.401310+0000","last_fullsized":"2023-10-10T19:29:04.401310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079196160000000008,"stat_sum":{"num_bytes":128300,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":70,"num_read_kb":16825,"num_write":682,"num_write_kb":23273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"131'955","reported_seq":3604,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.424201+0000","last_change":"2023-10-10T19:16:54.784745+0000","last_active":"2023-10-10T19:29:04.424201+0000","last_peered":"2023-10-10T19:29:04.424201+0000","last_clean":"2023-10-10T19:29:04.424201+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T19:29:04.424201+0000","last_undegraded":"2023-10-10T19:29:04.424201+0000","last_fullsized":"2023-10-10T19:29:04.424201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088422029999999999,"stat_sum":{"num_bytes":198054,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":82,"num_read_kb":46829,"num_write":704,"num_write_kb":53046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"131'944","reported_seq":3579,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.484037+0000","last_change":"2023-10-10T19:16:54.989857+0000","last_active":"2023-10-10T19:29:04.484037+0000","last_peered":"2023-10-10T19:29:04.484037+0000","last_clean":"2023-10-10T19:29:04.484037+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T19:29:04.484037+0000","last_undegraded":"2023-10-10T19:29:04.484037+0000","last_fullsized":"2023-10-10T19:29:04.484037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078589550000000008,"stat_sum":{"num_bytes":271785,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":75,"num_read_kb":57532,"num_write":692,"num_write_kb":73449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"131'896","reported_seq":3526,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.275163+0000","last_change":"2023-10-10T19:16:54.872408+0000","last_active":"2023-10-10T19:29:04.275163+0000","last_peered":"2023-10-10T19:29:04.275163+0000","last_clean":"2023-10-10T19:29:04.275163+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T19:29:04.275163+0000","last_undegraded":"2023-10-10T19:29:04.275163+0000","last_fullsized":"2023-10-10T19:29:04.275163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014204069,"stat_sum":{"num_bytes":4542310,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":73,"num_read_kb":32208,"num_write":631,"num_write_kb":44582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"131'902","reported_seq":3526,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.393937+0000","last_change":"2023-10-10T19:16:54.978714+0000","last_active":"2023-10-10T19:29:04.393937+0000","last_peered":"2023-10-10T19:29:04.393937+0000","last_clean":"2023-10-10T19:29:04.393937+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T19:29:04.393937+0000","last_undegraded":"2023-10-10T19:29:04.393937+0000","last_fullsized":"2023-10-10T19:29:04.393937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097965510000000006,"stat_sum":{"num_bytes":686329,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":63,"num_read_kb":28420,"num_write":651,"num_write_kb":31132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"131'934","reported_seq":3567,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.271178+0000","last_change":"2023-10-10T19:16:54.917427+0000","last_active":"2023-10-10T19:29:04.271178+0000","last_peered":"2023-10-10T19:29:04.271178+0000","last_clean":"2023-10-10T19:29:04.271178+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T19:29:04.271178+0000","last_undegraded":"2023-10-10T19:29:04.271178+0000","last_fullsized":"2023-10-10T19:29:04.271178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059774370000000004,"stat_sum":{"num_bytes":1294620,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":77,"num_read_kb":52747,"num_write":659,"num_write_kb":72992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"131'977","reported_seq":3611,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077966+0000","last_change":"2023-10-10T19:16:55.412885+0000","last_active":"2023-10-10T19:29:04.077966+0000","last_peered":"2023-10-10T19:29:04.077966+0000","last_clean":"2023-10-10T19:29:04.077966+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T19:29:04.077966+0000","last_undegraded":"2023-10-10T19:29:04.077966+0000","last_fullsized":"2023-10-10T19:29:04.077966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087668889999999999,"stat_sum":{"num_bytes":446378,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":80,"num_read_kb":42995,"num_write":700,"num_write_kb":52933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"131'882","reported_seq":3536,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.705589+0000","last_change":"2023-10-10T19:16:55.417126+0000","last_active":"2023-10-10T19:28:34.705589+0000","last_peered":"2023-10-10T19:28:34.705589+0000","last_clean":"2023-10-10T19:28:34.705589+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T19:28:34.705589+0000","last_undegraded":"2023-10-10T19:28:34.705589+0000","last_fullsized":"2023-10-10T19:28:34.705589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012085218,"stat_sum":{"num_bytes":1843052,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":72,"num_read_kb":40372,"num_write":665,"num_write_kb":48809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"131'826","reported_seq":3498,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.912020+0000","last_change":"2023-10-10T19:16:54.780747+0000","last_active":"2023-10-10T19:28:33.912020+0000","last_peered":"2023-10-10T19:28:33.912020+0000","last_clean":"2023-10-10T19:28:33.912020+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T19:28:33.912020+0000","last_undegraded":"2023-10-10T19:28:33.912020+0000","last_fullsized":"2023-10-10T19:28:33.912020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00594963,"stat_sum":{"num_bytes":909657,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":72,"num_read_kb":44243,"num_write":651,"num_write_kb":60901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"131'1049","reported_seq":3646,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.588062+0000","last_change":"2023-10-10T19:16:54.792335+0000","last_active":"2023-10-10T19:29:04.588062+0000","last_peered":"2023-10-10T19:29:04.588062+0000","last_clean":"2023-10-10T19:29:04.588062+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T19:29:04.588062+0000","last_undegraded":"2023-10-10T19:29:04.588062+0000","last_fullsized":"2023-10-10T19:29:04.588062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01601,"stat_sum":{"num_bytes":624343,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":73,"num_read_kb":41187,"num_write":717,"num_write_kb":49713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"131'836","reported_seq":1001,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:18.796925+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T19:28:18.796925+0000","last_peered":"2023-10-10T19:28:18.796925+0000","last_clean":"2023-10-10T19:28:18.796925+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T19:28:18.796925+0000","last_undegraded":"2023-10-10T19:28:18.796925+0000","last_fullsized":"2023-10-10T19:28:18.796925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389252,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":82,"num_read_kb":61516,"num_write":780,"num_write_kb":21018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"131'974","reported_seq":3595,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.491287+0000","last_change":"2023-10-10T19:16:54.877116+0000","last_active":"2023-10-10T19:29:04.491287+0000","last_peered":"2023-10-10T19:29:04.491287+0000","last_clean":"2023-10-10T19:29:04.491287+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T19:29:04.491287+0000","last_undegraded":"2023-10-10T19:29:04.491287+0000","last_fullsized":"2023-10-10T19:29:04.491287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018086130999999998,"stat_sum":{"num_bytes":63039,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":63,"num_read_kb":13169,"num_write":717,"num_write_kb":31098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"131'759","reported_seq":903,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:04.229561+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T19:28:04.229561+0000","last_peered":"2023-10-10T19:28:04.229561+0000","last_clean":"2023-10-10T19:28:04.229561+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T19:28:04.229561+0000","last_undegraded":"2023-10-10T19:28:04.229561+0000","last_fullsized":"2023-10-10T19:28:04.229561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":29,"num_read_kb":69691,"num_write":709,"num_write_kb":16925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"131'971","reported_seq":3613,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078264+0000","last_change":"2023-10-10T19:16:54.978352+0000","last_active":"2023-10-10T19:29:04.078264+0000","last_peered":"2023-10-10T19:29:04.078264+0000","last_clean":"2023-10-10T19:29:04.078264+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T19:29:04.078264+0000","last_undegraded":"2023-10-10T19:29:04.078264+0000","last_fullsized":"2023-10-10T19:29:04.078264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093729599999999996,"stat_sum":{"num_bytes":213282,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":74,"num_read_kb":34573,"num_write":722,"num_write_kb":44041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"132'1694","reported_seq":1870,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:10.134874+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T19:29:10.134874+0000","last_peered":"2023-10-10T19:29:10.134874+0000","last_clean":"2023-10-10T19:29:10.134874+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T19:29:10.134874+0000","last_undegraded":"2023-10-10T19:29:10.134874+0000","last_fullsized":"2023-10-10T19:29:10.134874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1694,"log_dups_size":0,"ondisk_log_size":1694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":114,"num_read_kb":59511,"num_write":1903,"num_write_kb":22655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"131'936","reported_seq":3530,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.496246+0000","last_change":"2023-10-10T19:16:54.780201+0000","last_active":"2023-10-10T19:29:04.496246+0000","last_peered":"2023-10-10T19:29:04.496246+0000","last_clean":"2023-10-10T19:29:04.496246+0000","last_became_active":"2023-10-10T18:46:01.103860+0000","last_became_peered":"2023-10-10T18:46:01.103860+0000","last_unstale":"2023-10-10T19:29:04.496246+0000","last_undegraded":"2023-10-10T19:29:04.496246+0000","last_fullsized":"2023-10-10T19:29:04.496246+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:17.975194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054668909999999998,"stat_sum":{"num_bytes":80680,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":48,"num_read_kb":295,"num_write":639,"num_write_kb":7460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"130'750","reported_seq":903,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.970799+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T19:27:29.970799+0000","last_peered":"2023-10-10T19:27:29.970799+0000","last_clean":"2023-10-10T19:27:29.970799+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T19:27:29.970799+0000","last_undegraded":"2023-10-10T19:27:29.970799+0000","last_fullsized":"2023-10-10T19:27:29.970799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389162,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":55,"num_read_kb":36831,"num_write":683,"num_write_kb":16858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"131'970","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.274296+0000","last_change":"2023-10-10T19:16:55.416075+0000","last_active":"2023-10-10T19:29:04.274296+0000","last_peered":"2023-10-10T19:29:04.274296+0000","last_clean":"2023-10-10T19:29:04.274296+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T19:29:04.274296+0000","last_undegraded":"2023-10-10T19:29:04.274296+0000","last_fullsized":"2023-10-10T19:29:04.274296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011753095,"stat_sum":{"num_bytes":80024,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":81,"num_read_kb":35882,"num_write":686,"num_write_kb":42928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"131'612","reported_seq":760,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.922994+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T19:28:38.922994+0000","last_peered":"2023-10-10T19:28:38.922994+0000","last_clean":"2023-10-10T19:28:38.922994+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T19:28:38.922994+0000","last_undegraded":"2023-10-10T19:28:38.922994+0000","last_fullsized":"2023-10-10T19:28:38.922994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14118582,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":31,"num_read_kb":65583,"num_write":554,"num_write_kb":18318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"131'893","reported_seq":3521,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.393580+0000","last_change":"2023-10-10T19:16:54.987567+0000","last_active":"2023-10-10T19:29:04.393580+0000","last_peered":"2023-10-10T19:29:04.393580+0000","last_clean":"2023-10-10T19:29:04.393580+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T19:29:04.393580+0000","last_undegraded":"2023-10-10T19:29:04.393580+0000","last_fullsized":"2023-10-10T19:29:04.393580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062644010000000002,"stat_sum":{"num_bytes":763528,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":59,"num_read_kb":21826,"num_write":671,"num_write_kb":37641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"129'775","reported_seq":1107,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.932043+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T19:28:38.932043+0000","last_peered":"2023-10-10T19:28:38.932043+0000","last_clean":"2023-10-10T19:28:38.932043+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T19:28:38.932043+0000","last_undegraded":"2023-10-10T19:28:38.932043+0000","last_fullsized":"2023-10-10T19:28:38.932043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111936,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":251,"num_read_kb":81147,"num_write":700,"num_write_kb":25323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"131'819","reported_seq":3467,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.924982+0000","last_change":"2023-10-10T19:16:54.984860+0000","last_active":"2023-10-10T19:28:24.924982+0000","last_peered":"2023-10-10T19:28:24.924982+0000","last_clean":"2023-10-10T19:28:24.924982+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T19:28:24.924982+0000","last_undegraded":"2023-10-10T19:28:24.924982+0000","last_fullsized":"2023-10-10T19:28:24.924982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015269961,"stat_sum":{"num_bytes":449137,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":60,"num_read_kb":24046,"num_write":605,"num_write_kb":30663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"130'1494","reported_seq":1889,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.913899+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T19:28:38.913899+0000","last_peered":"2023-10-10T19:28:38.913899+0000","last_clean":"2023-10-10T19:28:38.913899+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T19:28:38.913899+0000","last_undegraded":"2023-10-10T19:28:38.913899+0000","last_fullsized":"2023-10-10T19:28:38.913899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4727705,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":295,"num_read_kb":15405,"num_write":1447,"num_write_kb":10055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"131'980","reported_seq":3623,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.268565+0000","last_change":"2023-10-10T19:16:54.874108+0000","last_active":"2023-10-10T19:29:04.268565+0000","last_peered":"2023-10-10T19:29:04.268565+0000","last_clean":"2023-10-10T19:29:04.268565+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T19:29:04.268565+0000","last_undegraded":"2023-10-10T19:29:04.268565+0000","last_fullsized":"2023-10-10T19:29:04.268565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015767953000000001,"stat_sum":{"num_bytes":541499,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":81,"num_read_kb":19092,"num_write":733,"num_write_kb":30243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"131'494","reported_seq":639,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704437+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T19:28:34.704437+0000","last_peered":"2023-10-10T19:28:34.704437+0000","last_clean":"2023-10-10T19:28:34.704437+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T19:28:34.704437+0000","last_undegraded":"2023-10-10T19:28:34.704437+0000","last_fullsized":"2023-10-10T19:28:34.704437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583466,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":65566,"num_write":451,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"131'1005","reported_seq":3644,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.395988+0000","last_change":"2023-10-10T19:16:54.918258+0000","last_active":"2023-10-10T19:29:04.395988+0000","last_peered":"2023-10-10T19:29:04.395988+0000","last_clean":"2023-10-10T19:29:04.395988+0000","last_became_active":"2023-10-10T18:46:00.606391+0000","last_became_peered":"2023-10-10T18:46:00.606391+0000","last_unstale":"2023-10-10T19:29:04.395988+0000","last_undegraded":"2023-10-10T19:29:04.395988+0000","last_fullsized":"2023-10-10T19:29:04.395988+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1005,"log_dups_size":0,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:57.665869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066939039999999997,"stat_sum":{"num_bytes":631366,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":80,"num_read_kb":42231,"num_write":729,"num_write_kb":52765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"132'274","reported_seq":415,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:15.078147+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T19:29:15.078147+0000","last_peered":"2023-10-10T19:29:15.078147+0000","last_clean":"2023-10-10T19:29:15.078147+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T19:29:15.078147+0000","last_undegraded":"2023-10-10T19:29:15.078147+0000","last_fullsized":"2023-10-10T19:29:15.078147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":34,"num_read_kb":54,"num_write":225,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"131'948","reported_seq":3583,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.705242+0000","last_change":"2023-10-10T19:16:55.413821+0000","last_active":"2023-10-10T19:28:34.705242+0000","last_peered":"2023-10-10T19:28:34.705242+0000","last_clean":"2023-10-10T19:28:34.705242+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T19:28:34.705242+0000","last_undegraded":"2023-10-10T19:28:34.705242+0000","last_fullsized":"2023-10-10T19:28:34.705242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096962509999999995,"stat_sum":{"num_bytes":519488,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":63,"num_read_kb":26256,"num_write":717,"num_write_kb":45994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"132'617","reported_seq":975,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:10.134272+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T19:29:10.134272+0000","last_peered":"2023-10-10T19:29:10.134272+0000","last_clean":"2023-10-10T19:29:10.134272+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T19:29:10.134272+0000","last_undegraded":"2023-10-10T19:29:10.134272+0000","last_fullsized":"2023-10-10T19:29:10.134272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7932987,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":262,"num_read_kb":30418,"num_write":555,"num_write_kb":12258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"131'906","reported_seq":3529,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.400410+0000","last_change":"2023-10-10T19:16:54.984649+0000","last_active":"2023-10-10T19:29:04.400410+0000","last_peered":"2023-10-10T19:29:04.400410+0000","last_clean":"2023-10-10T19:29:04.400410+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T19:29:04.400410+0000","last_undegraded":"2023-10-10T19:29:04.400410+0000","last_fullsized":"2023-10-10T19:29:04.400410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015222243,"stat_sum":{"num_bytes":277071,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":64,"num_read_kb":38795,"num_write":628,"num_write_kb":46072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"131'207","reported_seq":349,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.761037+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T19:28:34.761037+0000","last_peered":"2023-10-10T19:28:34.761037+0000","last_clean":"2023-10-10T19:28:34.761037+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T19:28:34.761037+0000","last_undegraded":"2023-10-10T19:28:34.761037+0000","last_fullsized":"2023-10-10T19:28:34.761037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":42,"num_read_kb":61,"num_write":171,"num_write_kb":225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"131'858","reported_seq":3490,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.320810+0000","last_change":"2023-10-10T19:16:54.989921+0000","last_active":"2023-10-10T19:29:04.320810+0000","last_peered":"2023-10-10T19:29:04.320810+0000","last_clean":"2023-10-10T19:29:04.320810+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T19:29:04.320810+0000","last_undegraded":"2023-10-10T19:29:04.320810+0000","last_fullsized":"2023-10-10T19:29:04.320810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080742209999999995,"stat_sum":{"num_bytes":82158,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":56,"num_read_kb":23103,"num_write":645,"num_write_kb":46075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"132'264","reported_seq":413,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.276956+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T19:29:04.276956+0000","last_peered":"2023-10-10T19:29:04.276956+0000","last_clean":"2023-10-10T19:29:04.276956+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T19:29:04.276956+0000","last_undegraded":"2023-10-10T19:29:04.276956+0000","last_fullsized":"2023-10-10T19:29:04.276956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194948,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":59,"num_read_kb":16449,"num_write":212,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"131'930","reported_seq":3560,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.595503+0000","last_change":"2023-10-10T19:16:54.793178+0000","last_active":"2023-10-10T19:29:04.595503+0000","last_peered":"2023-10-10T19:29:04.595503+0000","last_clean":"2023-10-10T19:29:04.595503+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T19:29:04.595503+0000","last_undegraded":"2023-10-10T19:29:04.595503+0000","last_fullsized":"2023-10-10T19:29:04.595503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015774979000000001,"stat_sum":{"num_bytes":110666,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":71,"num_read_kb":24956,"num_write":663,"num_write_kb":31333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"125'818","reported_seq":1254,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.911380+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T19:28:33.911380+0000","last_peered":"2023-10-10T19:28:33.911380+0000","last_clean":"2023-10-10T19:28:33.911380+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T19:28:33.911380+0000","last_undegraded":"2023-10-10T19:28:33.911380+0000","last_fullsized":"2023-10-10T19:28:33.911380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227398,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":359,"num_read_kb":46031,"num_write":769,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"131'911","reported_seq":3542,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078167+0000","last_change":"2023-10-10T19:16:54.981687+0000","last_active":"2023-10-10T19:29:04.078167+0000","last_peered":"2023-10-10T19:29:04.078167+0000","last_clean":"2023-10-10T19:29:04.078167+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T19:29:04.078167+0000","last_undegraded":"2023-10-10T19:29:04.078167+0000","last_fullsized":"2023-10-10T19:29:04.078167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01251216,"stat_sum":{"num_bytes":89325,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":62,"num_read_kb":21731,"num_write":674,"num_write_kb":32887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"128'713","reported_seq":873,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.927273+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T19:28:38.927273+0000","last_peered":"2023-10-10T19:28:38.927273+0000","last_clean":"2023-10-10T19:28:38.927273+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T19:28:38.927273+0000","last_undegraded":"2023-10-10T19:28:38.927273+0000","last_fullsized":"2023-10-10T19:28:38.927273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415841,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":58,"num_read_kb":76305,"num_write":648,"num_write_kb":19602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"131'951","reported_seq":3576,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.258906+0000","last_change":"2023-10-10T19:16:54.791948+0000","last_active":"2023-10-10T19:28:34.258906+0000","last_peered":"2023-10-10T19:28:34.258906+0000","last_clean":"2023-10-10T19:28:34.258906+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T19:28:34.258906+0000","last_undegraded":"2023-10-10T19:28:34.258906+0000","last_fullsized":"2023-10-10T19:28:34.258906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017093574,"stat_sum":{"num_bytes":514832,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":73,"num_read_kb":45017,"num_write":683,"num_write_kb":44061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"131'407","reported_seq":563,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.926155+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T19:28:38.926155+0000","last_peered":"2023-10-10T19:28:38.926155+0000","last_clean":"2023-10-10T19:28:38.926155+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T19:28:38.926155+0000","last_undegraded":"2023-10-10T19:28:38.926155+0000","last_fullsized":"2023-10-10T19:28:38.926155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":54,"num_read_kb":84439,"num_write":360,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"131'818","reported_seq":3450,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.403900+0000","last_change":"2023-10-10T19:16:54.782792+0000","last_active":"2023-10-10T19:29:04.403900+0000","last_peered":"2023-10-10T19:29:04.403900+0000","last_clean":"2023-10-10T19:29:04.403900+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T19:29:04.403900+0000","last_undegraded":"2023-10-10T19:29:04.403900+0000","last_fullsized":"2023-10-10T19:29:04.403900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075683979999999996,"stat_sum":{"num_bytes":1065275,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":52,"num_read_kb":24653,"num_write":599,"num_write_kb":45347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"131'804","reported_seq":983,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.911148+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T19:28:33.911148+0000","last_peered":"2023-10-10T19:28:33.911148+0000","last_clean":"2023-10-10T19:28:33.911148+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T19:28:33.911148+0000","last_undegraded":"2023-10-10T19:28:33.911148+0000","last_fullsized":"2023-10-10T19:28:33.911148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":804,"log_dups_size":0,"ondisk_log_size":804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":113,"num_read_kb":78302,"num_write":734,"num_write_kb":25074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"131'894","reported_seq":3531,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.913337+0000","last_change":"2023-10-10T19:16:54.790805+0000","last_active":"2023-10-10T19:28:24.913337+0000","last_peered":"2023-10-10T19:28:24.913337+0000","last_clean":"2023-10-10T19:28:24.913337+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T19:28:24.913337+0000","last_undegraded":"2023-10-10T19:28:24.913337+0000","last_fullsized":"2023-10-10T19:28:24.913337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01616896,"stat_sum":{"num_bytes":43538,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":67,"num_read_kb":45230,"num_write":650,"num_write_kb":49050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"131'604","reported_seq":741,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:19.022228+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T19:28:19.022228+0000","last_peered":"2023-10-10T19:28:19.022228+0000","last_clean":"2023-10-10T19:28:19.022228+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T19:28:19.022228+0000","last_undegraded":"2023-10-10T19:28:19.022228+0000","last_fullsized":"2023-10-10T19:28:19.022228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":23,"num_read_kb":12341,"num_write":746,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"131'918","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.406226+0000","last_change":"2023-10-10T19:16:54.793503+0000","last_active":"2023-10-10T19:29:04.406226+0000","last_peered":"2023-10-10T19:29:04.406226+0000","last_clean":"2023-10-10T19:29:04.406226+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T19:29:04.406226+0000","last_undegraded":"2023-10-10T19:29:04.406226+0000","last_fullsized":"2023-10-10T19:29:04.406226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015882960000000002,"stat_sum":{"num_bytes":652546,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":57,"num_read_kb":20900,"num_write":659,"num_write_kb":24656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"132'150","reported_seq":287,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:15.166343+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T19:29:15.166343+0000","last_peered":"2023-10-10T19:29:15.166343+0000","last_clean":"2023-10-10T19:29:15.166343+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T19:29:15.166343+0000","last_undegraded":"2023-10-10T19:29:15.166343+0000","last_fullsized":"2023-10-10T19:29:15.166343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311985,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":45123,"num_write":106,"num_write_kb":9377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"131'1026","reported_seq":3634,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.495833+0000","last_change":"2023-10-10T19:16:55.415960+0000","last_active":"2023-10-10T19:29:04.495833+0000","last_peered":"2023-10-10T19:29:04.495833+0000","last_clean":"2023-10-10T19:29:04.495833+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T19:29:04.495833+0000","last_undegraded":"2023-10-10T19:29:04.495833+0000","last_fullsized":"2023-10-10T19:29:04.495833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011491507999999999,"stat_sum":{"num_bytes":340961,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":71,"num_read_kb":20716,"num_write":714,"num_write_kb":35169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"130'341","reported_seq":489,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704918+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T19:28:34.704918+0000","last_peered":"2023-10-10T19:28:34.704918+0000","last_clean":"2023-10-10T19:28:34.704918+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T19:28:34.704918+0000","last_undegraded":"2023-10-10T19:28:34.704918+0000","last_fullsized":"2023-10-10T19:28:34.704918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":73794,"num_write":284,"num_write_kb":20986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"131'933","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758752+0000","last_change":"2023-10-10T19:16:54.988658+0000","last_active":"2023-10-10T19:28:34.758752+0000","last_peered":"2023-10-10T19:28:34.758752+0000","last_clean":"2023-10-10T19:28:34.758752+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T19:28:34.758752+0000","last_undegraded":"2023-10-10T19:28:34.758752+0000","last_fullsized":"2023-10-10T19:28:34.758752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071091339999999996,"stat_sum":{"num_bytes":108666,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":93,"num_read_kb":25139,"num_write":704,"num_write_kb":35589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"131'931","reported_seq":3556,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.447963+0000","last_change":"2023-10-10T19:16:54.919574+0000","last_active":"2023-10-10T19:29:04.447963+0000","last_peered":"2023-10-10T19:29:04.447963+0000","last_clean":"2023-10-10T19:29:04.447963+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T19:29:04.447963+0000","last_undegraded":"2023-10-10T19:29:04.447963+0000","last_fullsized":"2023-10-10T19:29:04.447963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077260940000000002,"stat_sum":{"num_bytes":89807,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":61,"num_read_kb":19697,"num_write":681,"num_write_kb":26677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"132'685","reported_seq":1109,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:18.043195+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T19:29:18.043195+0000","last_peered":"2023-10-10T19:29:18.043195+0000","last_clean":"2023-10-10T19:29:18.043195+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T19:29:18.043195+0000","last_undegraded":"2023-10-10T19:29:18.043195+0000","last_fullsized":"2023-10-10T19:29:18.043195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9108221,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":314,"num_read_kb":40841,"num_write":609,"num_write_kb":9447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'875","reported_seq":3545,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.281760+0000","last_change":"2023-10-10T19:16:54.875665+0000","last_active":"2023-10-10T19:29:04.281760+0000","last_peered":"2023-10-10T19:29:04.281760+0000","last_clean":"2023-10-10T19:29:04.281760+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T19:29:04.281760+0000","last_undegraded":"2023-10-10T19:29:04.281760+0000","last_fullsized":"2023-10-10T19:29:04.281760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017091021000000001,"stat_sum":{"num_bytes":821658,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":88,"num_read_kb":19324,"num_write":658,"num_write_kb":26590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"131'137","reported_seq":294,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.142425+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T19:28:39.142425+0000","last_peered":"2023-10-10T19:28:39.142425+0000","last_clean":"2023-10-10T19:28:39.142425+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T19:28:39.142425+0000","last_undegraded":"2023-10-10T19:28:39.142425+0000","last_fullsized":"2023-10-10T19:28:39.142425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388756,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":36939,"num_write":74,"num_write_kb":8306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":274,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.970643+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T19:27:29.970643+0000","last_peered":"2023-10-10T19:27:29.970643+0000","last_clean":"2023-10-10T19:27:29.970643+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T19:27:29.970643+0000","last_undegraded":"2023-10-10T19:27:29.970643+0000","last_fullsized":"2023-10-10T19:27:29.970643+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"131'887","reported_seq":3510,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.078025+0000","last_change":"2023-10-10T19:16:54.979966+0000","last_active":"2023-10-10T19:29:04.078025+0000","last_peered":"2023-10-10T19:29:04.078025+0000","last_clean":"2023-10-10T19:29:04.078025+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T19:29:04.078025+0000","last_undegraded":"2023-10-10T19:29:04.078025+0000","last_fullsized":"2023-10-10T19:29:04.078025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010927930000000001,"stat_sum":{"num_bytes":3326937,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":55,"num_read_kb":20626,"num_write":650,"num_write_kb":30635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"131'973","reported_seq":1124,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:40.006368+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T19:28:40.006368+0000","last_peered":"2023-10-10T19:28:40.006368+0000","last_clean":"2023-10-10T19:28:40.006368+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T19:28:40.006368+0000","last_undegraded":"2023-10-10T19:28:40.006368+0000","last_fullsized":"2023-10-10T19:28:40.006368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":38,"num_read_kb":89926,"num_write":912,"num_write_kb":29260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"128'825","reported_seq":978,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.259463+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T19:28:34.259463+0000","last_peered":"2023-10-10T19:28:34.259463+0000","last_clean":"2023-10-10T19:28:34.259463+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T19:28:34.259463+0000","last_undegraded":"2023-10-10T19:28:34.259463+0000","last_fullsized":"2023-10-10T19:28:34.259463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":45,"num_read_kb":45594,"num_write":764,"num_write_kb":12838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"131'1007","reported_seq":3620,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077312+0000","last_change":"2023-10-10T19:16:54.919294+0000","last_active":"2023-10-10T19:29:04.077312+0000","last_peered":"2023-10-10T19:29:04.077312+0000","last_clean":"2023-10-10T19:29:04.077312+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T19:29:04.077312+0000","last_undegraded":"2023-10-10T19:29:04.077312+0000","last_fullsized":"2023-10-10T19:29:04.077312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075449510000000003,"stat_sum":{"num_bytes":492311,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":65,"num_read_kb":49387,"num_write":712,"num_write_kb":52375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'275","reported_seq":431,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:49.017495+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T19:28:49.017495+0000","last_peered":"2023-10-10T19:28:49.017495+0000","last_clean":"2023-10-10T19:28:49.017495+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T19:28:49.017495+0000","last_undegraded":"2023-10-10T19:28:49.017495+0000","last_fullsized":"2023-10-10T19:28:49.017495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":81,"num_read_kb":20675,"num_write":475,"num_write_kb":4740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"131'939","reported_seq":3565,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.285360+0000","last_change":"2023-10-10T19:16:54.920290+0000","last_active":"2023-10-10T19:29:04.285360+0000","last_peered":"2023-10-10T19:29:04.285360+0000","last_clean":"2023-10-10T19:29:04.285360+0000","last_became_active":"2023-10-10T18:46:01.267092+0000","last_became_peered":"2023-10-10T18:46:01.267092+0000","last_unstale":"2023-10-10T19:29:04.285360+0000","last_undegraded":"2023-10-10T19:29:04.285360+0000","last_fullsized":"2023-10-10T19:29:04.285360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:46:17.491022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083609700000000006,"stat_sum":{"num_bytes":2123745,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":68,"num_read_kb":33081,"num_write":679,"num_write_kb":41978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"132'154","reported_seq":301,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:41.419018+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T19:28:41.419018+0000","last_peered":"2023-10-10T19:28:41.419018+0000","last_clean":"2023-10-10T19:28:41.419018+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T19:28:41.419018+0000","last_undegraded":"2023-10-10T19:28:41.419018+0000","last_fullsized":"2023-10-10T19:28:41.419018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":58,"num_read_kb":8269,"num_write":92,"num_write_kb":8284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"131'867","reported_seq":3527,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.288386+0000","last_change":"2023-10-10T19:16:54.981885+0000","last_active":"2023-10-10T19:29:04.288386+0000","last_peered":"2023-10-10T19:29:04.288386+0000","last_clean":"2023-10-10T19:29:04.288386+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T19:29:04.288386+0000","last_undegraded":"2023-10-10T19:29:04.288386+0000","last_fullsized":"2023-10-10T19:29:04.288386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721872,"stat_sum":{"num_bytes":440031,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":85,"num_read_kb":32652,"num_write":635,"num_write_kb":36138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"131'228","reported_seq":371,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.557527+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T19:28:39.557527+0000","last_peered":"2023-10-10T19:28:39.557527+0000","last_clean":"2023-10-10T19:28:39.557527+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T19:28:39.557527+0000","last_undegraded":"2023-10-10T19:28:39.557527+0000","last_fullsized":"2023-10-10T19:28:39.557527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":57389,"num_write":173,"num_write_kb":12477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"131'931","reported_seq":3570,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758691+0000","last_change":"2023-10-10T19:16:54.990974+0000","last_active":"2023-10-10T19:28:34.758691+0000","last_peered":"2023-10-10T19:28:34.758691+0000","last_clean":"2023-10-10T19:28:34.758691+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T19:28:34.758691+0000","last_undegraded":"2023-10-10T19:28:34.758691+0000","last_fullsized":"2023-10-10T19:28:34.758691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089059480000000003,"stat_sum":{"num_bytes":796993,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":67,"num_read_kb":26489,"num_write":680,"num_write_kb":31111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"132'369","reported_seq":521,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.547536+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T19:29:04.547536+0000","last_peered":"2023-10-10T19:29:04.547536+0000","last_clean":"2023-10-10T19:29:04.547536+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T19:29:04.547536+0000","last_undegraded":"2023-10-10T19:29:04.547536+0000","last_fullsized":"2023-10-10T19:29:04.547536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":70,"num_read_kb":20656,"num_write":331,"num_write_kb":8737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"131'882","reported_seq":3516,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.406702+0000","last_change":"2023-10-10T19:16:54.870214+0000","last_active":"2023-10-10T19:29:04.406702+0000","last_peered":"2023-10-10T19:29:04.406702+0000","last_clean":"2023-10-10T19:29:04.406702+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T19:29:04.406702+0000","last_undegraded":"2023-10-10T19:29:04.406702+0000","last_fullsized":"2023-10-10T19:29:04.406702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012052205,"stat_sum":{"num_bytes":579663,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":63,"num_read_kb":40064,"num_write":647,"num_write_kb":56806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"132'1301","reported_seq":1456,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:18.043527+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T19:29:18.043527+0000","last_peered":"2023-10-10T19:29:18.043527+0000","last_clean":"2023-10-10T19:29:18.043527+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T19:29:18.043527+0000","last_undegraded":"2023-10-10T19:29:18.043527+0000","last_fullsized":"2023-10-10T19:29:18.043527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1301,"log_dups_size":0,"ondisk_log_size":1301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14953265,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":48,"num_read_kb":53317,"num_write":1709,"num_write_kb":24642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"131'871","reported_seq":3503,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.271899+0000","last_change":"2023-10-10T19:16:54.874295+0000","last_active":"2023-10-10T19:29:04.271899+0000","last_peered":"2023-10-10T19:29:04.271899+0000","last_clean":"2023-10-10T19:29:04.271899+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T19:29:04.271899+0000","last_undegraded":"2023-10-10T19:29:04.271899+0000","last_fullsized":"2023-10-10T19:29:04.271899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015827366999999998,"stat_sum":{"num_bytes":60469,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":58,"num_read_kb":28965,"num_write":636,"num_write_kb":32020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"132'1200","reported_seq":1354,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.409456+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T19:29:04.409456+0000","last_peered":"2023-10-10T19:29:04.409456+0000","last_clean":"2023-10-10T19:29:04.409456+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T19:29:04.409456+0000","last_undegraded":"2023-10-10T19:29:04.409456+0000","last_fullsized":"2023-10-10T19:29:04.409456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":49,"num_read_kb":89747,"num_write":1124,"num_write_kb":25429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"131'914","reported_seq":3542,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:28:24.885234+0000","last_change":"2023-10-10T19:16:54.781695+0000","last_active":"2023-10-10T19:28:24.885234+0000","last_peered":"2023-10-10T19:28:24.885234+0000","last_clean":"2023-10-10T19:28:24.885234+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T19:28:24.885234+0000","last_undegraded":"2023-10-10T19:28:24.885234+0000","last_fullsized":"2023-10-10T19:28:24.885234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065052069999999998,"stat_sum":{"num_bytes":415973,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":64,"num_read_kb":10527,"num_write":665,"num_write_kb":20000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"130'311","reported_seq":451,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.683868+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T19:27:29.683868+0000","last_peered":"2023-10-10T19:27:29.683868+0000","last_clean":"2023-10-10T19:27:29.683868+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T19:27:29.683868+0000","last_undegraded":"2023-10-10T19:27:29.683868+0000","last_fullsized":"2023-10-10T19:27:29.683868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194858,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":40985,"num_write":231,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"131'896","reported_seq":3546,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.595223+0000","last_change":"2023-10-10T19:16:54.794460+0000","last_active":"2023-10-10T19:29:04.595223+0000","last_peered":"2023-10-10T19:29:04.595223+0000","last_clean":"2023-10-10T19:29:04.595223+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T19:29:04.595223+0000","last_undegraded":"2023-10-10T19:29:04.595223+0000","last_fullsized":"2023-10-10T19:29:04.595223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016889265000000001,"stat_sum":{"num_bytes":535544,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":67,"num_read_kb":24822,"num_write":684,"num_write_kb":31728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"132'288","reported_seq":432,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.760584+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T19:28:34.760584+0000","last_peered":"2023-10-10T19:28:34.760584+0000","last_clean":"2023-10-10T19:28:34.760584+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T19:28:34.760584+0000","last_undegraded":"2023-10-10T19:28:34.760584+0000","last_fullsized":"2023-10-10T19:28:34.760584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388678,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":27,"num_read_kb":50704,"num_write":218,"num_write_kb":12492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"131'948","reported_seq":3575,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.280142+0000","last_change":"2023-10-10T19:16:55.417128+0000","last_active":"2023-10-10T19:29:04.280142+0000","last_peered":"2023-10-10T19:29:04.280142+0000","last_clean":"2023-10-10T19:29:04.280142+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T19:29:04.280142+0000","last_undegraded":"2023-10-10T19:29:04.280142+0000","last_fullsized":"2023-10-10T19:29:04.280142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012585610000000001,"stat_sum":{"num_bytes":184409,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":60,"num_read_kb":19964,"num_write":698,"num_write_kb":27162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"128'310","reported_seq":458,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.683824+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T19:27:29.683824+0000","last_peered":"2023-10-10T19:27:29.683824+0000","last_clean":"2023-10-10T19:27:29.683824+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T19:27:29.683824+0000","last_undegraded":"2023-10-10T19:27:29.683824+0000","last_fullsized":"2023-10-10T19:27:29.683824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":49,"num_read_kb":57404,"num_write":260,"num_write_kb":8464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"131'999","reported_seq":3628,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.407478+0000","last_change":"2023-10-10T19:16:54.794658+0000","last_active":"2023-10-10T19:29:04.407478+0000","last_peered":"2023-10-10T19:29:04.407478+0000","last_clean":"2023-10-10T19:29:04.407478+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T19:29:04.407478+0000","last_undegraded":"2023-10-10T19:29:04.407478+0000","last_fullsized":"2023-10-10T19:29:04.407478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016837566000000002,"stat_sum":{"num_bytes":339858,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":67,"num_read_kb":28062,"num_write":742,"num_write_kb":23663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"131'891","reported_seq":3551,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077900+0000","last_change":"2023-10-10T19:16:54.782474+0000","last_active":"2023-10-10T19:29:04.077900+0000","last_peered":"2023-10-10T19:29:04.077900+0000","last_clean":"2023-10-10T19:29:04.077900+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T19:29:04.077900+0000","last_undegraded":"2023-10-10T19:29:04.077900+0000","last_fullsized":"2023-10-10T19:29:04.077900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074078640000000001,"stat_sum":{"num_bytes":492374,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":77,"num_read_kb":25056,"num_write":669,"num_write_kb":32074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"131'973","reported_seq":3594,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.287531+0000","last_change":"2023-10-10T19:16:54.779827+0000","last_active":"2023-10-10T19:29:04.287531+0000","last_peered":"2023-10-10T19:29:04.287531+0000","last_clean":"2023-10-10T19:29:04.287531+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T19:29:04.287531+0000","last_undegraded":"2023-10-10T19:29:04.287531+0000","last_fullsized":"2023-10-10T19:29:04.287531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050938800000000003,"stat_sum":{"num_bytes":392421,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":69,"num_read_kb":25013,"num_write":694,"num_write_kb":37199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"131'956","reported_seq":3603,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.272570+0000","last_change":"2023-10-10T19:16:54.781092+0000","last_active":"2023-10-10T19:29:04.272570+0000","last_peered":"2023-10-10T19:29:04.272570+0000","last_clean":"2023-10-10T19:29:04.272570+0000","last_became_active":"2023-10-10T18:46:00.606420+0000","last_became_peered":"2023-10-10T18:46:00.606420+0000","last_unstale":"2023-10-10T19:29:04.272570+0000","last_undegraded":"2023-10-10T19:29:04.272570+0000","last_fullsized":"2023-10-10T19:29:04.272570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:47.189801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064370249999999999,"stat_sum":{"num_bytes":69276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":90,"num_read_kb":45570,"num_write":703,"num_write_kb":52466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"131'929","reported_seq":3555,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.288777+0000","last_change":"2023-10-10T19:16:54.983512+0000","last_active":"2023-10-10T19:29:04.288777+0000","last_peered":"2023-10-10T19:29:04.288777+0000","last_clean":"2023-10-10T19:29:04.288777+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T19:29:04.288777+0000","last_undegraded":"2023-10-10T19:29:04.288777+0000","last_fullsized":"2023-10-10T19:29:04.288777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014096529,"stat_sum":{"num_bytes":398978,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":70,"num_read_kb":16093,"num_write":658,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"131'895","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.417096+0000","last_change":"2023-10-10T19:16:54.875626+0000","last_active":"2023-10-10T19:29:04.417096+0000","last_peered":"2023-10-10T19:29:04.417096+0000","last_clean":"2023-10-10T19:29:04.417096+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T19:29:04.417096+0000","last_undegraded":"2023-10-10T19:29:04.417096+0000","last_fullsized":"2023-10-10T19:29:04.417096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016808101999999998,"stat_sum":{"num_bytes":587511,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":68,"num_read_kb":48841,"num_write":661,"num_write_kb":55904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"131'962","reported_seq":3564,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.760146+0000","last_change":"2023-10-10T19:16:54.918525+0000","last_active":"2023-10-10T19:28:34.760146+0000","last_peered":"2023-10-10T19:28:34.760146+0000","last_clean":"2023-10-10T19:28:34.760146+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T19:28:34.760146+0000","last_undegraded":"2023-10-10T19:28:34.760146+0000","last_fullsized":"2023-10-10T19:28:34.760146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068654069999999996,"stat_sum":{"num_bytes":680481,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":59,"num_read_kb":41116,"num_write":666,"num_write_kb":48352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"131'952","reported_seq":3574,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.272886+0000","last_change":"2023-10-10T19:16:54.783685+0000","last_active":"2023-10-10T19:29:04.272886+0000","last_peered":"2023-10-10T19:29:04.272886+0000","last_clean":"2023-10-10T19:29:04.272886+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T19:29:04.272886+0000","last_undegraded":"2023-10-10T19:29:04.272886+0000","last_fullsized":"2023-10-10T19:29:04.272886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081260870000000006,"stat_sum":{"num_bytes":4687487,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":70,"num_read_kb":15871,"num_write":681,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"131'922","reported_seq":3558,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.267637+0000","last_change":"2023-10-10T19:16:54.877012+0000","last_active":"2023-10-10T19:29:04.267637+0000","last_peered":"2023-10-10T19:29:04.267637+0000","last_clean":"2023-10-10T19:29:04.267637+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T19:29:04.267637+0000","last_undegraded":"2023-10-10T19:29:04.267637+0000","last_fullsized":"2023-10-10T19:29:04.267637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":922,"log_dups_size":0,"ondisk_log_size":922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017981724000000001,"stat_sum":{"num_bytes":798509,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":72,"num_read_kb":44513,"num_write":661,"num_write_kb":47223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"131'898","reported_seq":3541,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.478410+0000","last_change":"2023-10-10T19:16:55.415035+0000","last_active":"2023-10-10T19:29:04.478410+0000","last_peered":"2023-10-10T19:29:04.478410+0000","last_clean":"2023-10-10T19:29:04.478410+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T19:29:04.478410+0000","last_undegraded":"2023-10-10T19:29:04.478410+0000","last_fullsized":"2023-10-10T19:29:04.478410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010756557999999999,"stat_sum":{"num_bytes":615982,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":71,"num_read_kb":21097,"num_write":676,"num_write_kb":25400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"131'878","reported_seq":3515,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.758569+0000","last_change":"2023-10-10T19:16:54.872190+0000","last_active":"2023-10-10T19:28:34.758569+0000","last_peered":"2023-10-10T19:28:34.758569+0000","last_clean":"2023-10-10T19:28:34.758569+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T19:28:34.758569+0000","last_undegraded":"2023-10-10T19:28:34.758569+0000","last_fullsized":"2023-10-10T19:28:34.758569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014012482999999999,"stat_sum":{"num_bytes":662827,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":69,"num_read_kb":49258,"num_write":627,"num_write_kb":56876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"131'935","reported_seq":3568,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.290882+0000","last_change":"2023-10-10T19:16:55.413906+0000","last_active":"2023-10-10T19:29:04.290882+0000","last_peered":"2023-10-10T19:29:04.290882+0000","last_clean":"2023-10-10T19:29:04.290882+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T19:29:04.290882+0000","last_undegraded":"2023-10-10T19:29:04.290882+0000","last_fullsized":"2023-10-10T19:29:04.290882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097177140000000006,"stat_sum":{"num_bytes":515125,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":59,"num_read_kb":20609,"num_write":704,"num_write_kb":28747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"131'873","reported_seq":3532,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.482135+0000","last_change":"2023-10-10T19:16:54.983331+0000","last_active":"2023-10-10T19:29:04.482135+0000","last_peered":"2023-10-10T19:29:04.482135+0000","last_clean":"2023-10-10T19:29:04.482135+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T19:29:04.482135+0000","last_undegraded":"2023-10-10T19:29:04.482135+0000","last_fullsized":"2023-10-10T19:29:04.482135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013935637000000001,"stat_sum":{"num_bytes":335358,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":51992,"num_write":654,"num_write_kb":56542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"131'994","reported_seq":3636,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.077991+0000","last_change":"2023-10-10T19:16:54.784919+0000","last_active":"2023-10-10T19:29:04.077991+0000","last_peered":"2023-10-10T19:29:04.077991+0000","last_clean":"2023-10-10T19:29:04.077991+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T19:29:04.077991+0000","last_undegraded":"2023-10-10T19:29:04.077991+0000","last_fullsized":"2023-10-10T19:29:04.077991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088155260000000006,"stat_sum":{"num_bytes":221031,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":77,"num_read_kb":37312,"num_write":732,"num_write_kb":49119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":43035717,"num_objects":1543,"num_object_clones":0,"num_object_copies":3086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":4408,"num_read_kb":1953419,"num_write":43238,"num_write_kb":2509746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":63954944,"data_stored":86071434,"data_compressed":25961548,"data_compressed_allocated":28180480,"data_compressed_original":58130432,"omap_allocated":0,"internal_metadata":0},"log_size":59428,"ondisk_log_size":59428,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":328210382,"num_objects":474,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":2858,"num_read_kb":1531013,"num_write":19009,"num_write_kb":461126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":338,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179290112,"data_stored":656420764,"data_compressed":84754974,"data_compressed_allocated":109232128,"data_compressed_original":586604544,"omap_allocated":0,"internal_metadata":0},"log_size":19669,"ondisk_log_size":19669,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575383,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346716,"kb_used_data":45332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94025124,"statfs":{"total":96636764160,"available":96281726976,"internally_reserved":0,"allocated":46419968,"data_stored":131329189,"data_compressed":19664789,"data_compressed_allocated":24576000,"data_compressed_original":111382528,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.53700000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.78400000000000003,"15min":1.018},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.32100000000000001,"5min":0.316,"15min":0.25},"max":{"1min":0.76500000000000001,"5min":0.80700000000000005,"15min":2.153},"last":0.36799999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.33100000000000002,"5min":0.318,"15min":0.26200000000000001},"max":{"1min":0.79500000000000004,"5min":0.82499999999999996,"15min":1.802},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":2.2170000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.623,"15min":0.61199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83899999999999997,"5min":2.871,"15min":2.871},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63400000000000001,"15min":0.63800000000000001},"min":{"1min":0.48299999999999998,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":7.6200000000000001},"last":0.55400000000000005}]},{"osd":3,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.874,"15min":1.0349999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.60099999999999998},"min":{"1min":0.375,"5min":0.373,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":7.6349999999999998},"last":0.502}]},{"osd":4,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.36599999999999999,"5min":0.307,"15min":0.28000000000000003},"max":{"1min":0.64000000000000001,"5min":0.72499999999999998,"15min":1.089},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.33800000000000002,"5min":0.32600000000000001,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.9419999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":0.629,"5min":0.752,"15min":1.71},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59299999999999997,"15min":0.57199999999999995},"min":{"1min":0.41799999999999998,"5min":0.378,"15min":0.311},"max":{"1min":0.80500000000000005,"5min":0.85099999999999998,"15min":1.177},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.31},"max":{"1min":0.76100000000000001,"5min":0.78000000000000003,"15min":0.872},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.33900000000000002,"15min":0.29699999999999999},"max":{"1min":0.72299999999999998,"5min":0.78100000000000003,"15min":2.2559999999999998},"last":0.57299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575382,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332164,"kb_used_data":34876,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94039676,"statfs":{"total":96636764160,"available":96296628224,"internally_reserved":0,"allocated":35713024,"data_stored":90587605,"data_compressed":11613979,"data_compressed_allocated":14393344,"data_compressed_original":70787072,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.31},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.216},"max":{"1min":0.71799999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42499999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.77200000000000002,"5min":0.77600000000000002,"15min":0.85299999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.58399999999999996,"15min":0.56599999999999995},"min":{"1min":0.46700000000000003,"5min":0.29599999999999999,"15min":0.222},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86799999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Tue Oct 10 19:28:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58199999999999996,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.312,"15min":0.30499999999999999},"max":{"1min":0.69899999999999995,"5min":0.77800000000000002,"15min":0.90500000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56899999999999995,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29999999999999999},"max":{"1min":0.71299999999999997,"5min":0.76500000000000001,"15min":0.97599999999999998},"last":0.51300000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57399999999999995,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76600000000000001,"5min":0.82799999999999996,"15min":0.88200000000000001},"last":0.628},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.35999999999999999},"max":{"1min":0.73599999999999999,"5min":0.85699999999999998,"15min":0.95099999999999996},"last":0.73199999999999998}]},{"osd":4,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.46400000000000002},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.248},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.749},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.52500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.73599999999999999,"15min":0.879},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.39700000000000002,"5min":0.30599999999999999,"15min":0.26400000000000001},"max":{"1min":0.63100000000000001,"5min":0.76800000000000002,"15min":0.99299999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.38200000000000001,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.88900000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.38,"5min":0.376,"15min":0.27700000000000002},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.86199999999999999},"last":0.60899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575381,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339956,"kb_used_data":30380,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94031884,"statfs":{"total":96636764160,"available":96288649216,"internally_reserved":0,"allocated":31109120,"data_stored":109263664,"data_compressed":16514475,"data_compressed_allocated":20389888,"data_compressed_original":99950592,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.314,"15min":0.313},"min":{"1min":0.187,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.432,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.505},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.66700000000000004,"15min":0.71699999999999997},"last":0.626}]},{"osd":2,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34599999999999997,"15min":0.35199999999999998},"min":{"1min":0.23799999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55200000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25,"15min":0.19900000000000001},"max":{"1min":0.54000000000000004,"5min":0.64900000000000002,"15min":0.73899999999999999},"last":0.56699999999999995}]},{"osd":3,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.432,"5min":0.45300000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.32600000000000001,"15min":0.27700000000000002},"max":{"1min":0.59699999999999998,"5min":0.72899999999999998,"15min":0.84299999999999997},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":1.175},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58699999999999997,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.87,"5min":0.87,"15min":2.2519999999999998},"last":0.80800000000000005}]},{"osd":5,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.76700000000000002,"15min":0.90900000000000003},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":1.0009999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.40500000000000003,"15min":0.37},"max":{"1min":0.83199999999999996,"5min":0.86199999999999999,"15min":1.038},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.123},"last":0.96699999999999997}]},{"osd":7,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60299999999999998,"15min":0.62},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":5.2720000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":5.2599999999999998},"last":1.0389999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575383,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335236,"kb_used_data":29756,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94036604,"statfs":{"total":96636764160,"available":96293482496,"internally_reserved":0,"allocated":30470144,"data_stored":81989013,"data_compressed":12938492,"data_compressed_allocated":16048128,"data_compressed_original":69398528,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.35799999999999998,"15min":0.35699999999999998},"min":{"1min":0.245,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":0.47699999999999998,"5min":0.57399999999999995,"15min":0.66500000000000004},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42499999999999999,"15min":0.41999999999999998},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.20300000000000001},"max":{"1min":0.60299999999999998,"5min":0.66000000000000003,"15min":0.66500000000000004},"last":0.35299999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35999999999999999,"15min":0.373},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.49299999999999999,"5min":0.57799999999999996,"15min":2.028},"last":0.308},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54900000000000004,"5min":0.61299999999999999,"15min":0.77300000000000002},"last":0.27300000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":0.54000000000000004,"5min":0.59699999999999998,"15min":0.79100000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.32500000000000001,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":0.70499999999999996,"5min":0.70599999999999996,"15min":0.80300000000000005},"last":0.375}]},{"osd":4,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.373,"15min":0.26000000000000001},"max":{"1min":0.80200000000000005,"5min":0.97599999999999998,"15min":1.9610000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.38400000000000001},"max":{"1min":0.82599999999999996,"5min":0.999,"15min":0.999},"last":0.53700000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.55100000000000005,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.79600000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.437},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.626,"15min":0.623},"min":{"1min":0.51600000000000001,"5min":0.38600000000000001,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.873,"15min":0.96799999999999997},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.52200000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":0.75800000000000001,"5min":0.96299999999999997,"15min":2.0019999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.621,"15min":0.623},"min":{"1min":0.501,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":0.84999999999999998,"5min":0.94099999999999995,"15min":0.95399999999999996},"last":0.43099999999999999}]},{"osd":7,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.40400000000000003,"15min":0.39100000000000001},"max":{"1min":0.77000000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41799999999999998},"max":{"1min":0.879,"5min":0.89500000000000002,"15min":2.0539999999999998},"last":0.498}]}]},{"osd":2,"up_from":13,"seq":55834575383,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318528,"kb_used_data":17144,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94053312,"statfs":{"total":96636764160,"available":96310591488,"internally_reserved":0,"allocated":17555456,"data_stored":56587147,"data_compressed":8367326,"data_compressed_allocated":10403840,"data_compressed_original":50835456,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34399999999999997,"15min":0.33500000000000002},"min":{"1min":0.224,"5min":0.219,"15min":0.186},"max":{"1min":0.45000000000000001,"5min":0.55400000000000005,"15min":0.60799999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.53500000000000003,"5min":0.65100000000000002,"15min":0.67500000000000004},"last":0.38}]},{"osd":1,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20999999999999999},"max":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.65700000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.73999999999999999},"last":0.443}]},{"osd":3,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.24399999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.42499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.23899999999999999},"max":{"1min":0.59099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.76900000000000002},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.26600000000000001},"max":{"1min":0.751,"5min":0.95599999999999996,"15min":48.729999999999997},"last":0.499},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.25900000000000001},"max":{"1min":0.74099999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.48599999999999999}]},{"osd":5,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63,"15min":0.628},"min":{"1min":0.436,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":1.02,"15min":1.02},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.32400000000000001},"max":{"1min":0.81000000000000005,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.78900000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":1.0760000000000001,"15min":48.685000000000002},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.38800000000000001},"max":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":0.97899999999999998},"last":0.51600000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64800000000000002,"15min":0.80700000000000005},"min":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.38700000000000001},"max":{"1min":0.85799999999999998,"5min":1.052,"15min":48.710000000000001},"last":0.748},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.034,"15min":1.1459999999999999},"last":0.55700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575382,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345608,"kb_used_data":36032,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94026232,"statfs":{"total":96636764160,"available":96282861568,"internally_reserved":0,"allocated":36896768,"data_stored":96360757,"data_compressed":13146180,"data_compressed_allocated":16756736,"data_compressed_original":78069760,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.37},"min":{"1min":0.23499999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.47599999999999998,"5min":0.64900000000000002,"15min":5.0670000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.371,"15min":0.38700000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.189},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.7799999999999998},"last":0.223}]},{"osd":1,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.36499999999999999,"15min":0.377},"min":{"1min":0.27200000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.628,"5min":0.628,"15min":2.758},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.41199999999999998},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51700000000000002,"5min":0.58399999999999996,"15min":0.64500000000000002},"last":0.29099999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28000000000000003,"15min":0.25},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":1.333},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.41899999999999998},"min":{"1min":0.30599999999999999,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.98699999999999999},"last":0.307}]},{"osd":4,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.48299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.879,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46200000000000002,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.23},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.45100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.2929999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.36199999999999999},"max":{"1min":0.90200000000000002,"5min":0.93600000000000005,"15min":1.103},"last":0.53900000000000003}]},{"osd":6,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63500000000000001,"15min":0.64000000000000001},"min":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.39900000000000002},"max":{"1min":0.91000000000000003,"5min":0.94199999999999995,"15min":1.198},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.49099999999999999,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.94899999999999995,"5min":0.95199999999999996,"15min":2.7309999999999999},"last":0.504}]},{"osd":7,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.46700000000000003,"5min":0.46500000000000002,"15min":0.39400000000000002},"max":{"1min":0.85099999999999998,"5min":0.97999999999999998,"15min":2.6619999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.66900000000000004},"min":{"1min":0.50800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.59699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575382,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338568,"kb_used_data":33088,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94033272,"statfs":{"total":96636764160,"available":96290070528,"internally_reserved":0,"allocated":33882112,"data_stored":101437325,"data_compressed":15509067,"data_compressed_allocated":19054592,"data_compressed_original":88412160,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56100000000000005,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.21199999999999999},"max":{"1min":0.746,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.23699999999999999},"max":{"1min":0.71899999999999997,"5min":0.86799999999999999,"15min":0.88900000000000001},"last":0.56999999999999995}]},{"osd":1,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":0.68300000000000005,"5min":0.71199999999999997,"15min":1.5920000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.55800000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.69899999999999995,"5min":0.749,"15min":1.0449999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.30199999999999999},"max":{"1min":0.752,"5min":0.78200000000000003,"15min":1.615},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.36699999999999999},"max":{"1min":0.73799999999999999,"5min":0.85799999999999998,"15min":1.071},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.58099999999999996},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.76200000000000001,"5min":0.85199999999999998,"15min":0.85799999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.28199999999999997},"max":{"1min":0.81799999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.60699999999999998}]},{"osd":5,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":1.087},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56699999999999995,"15min":0.55500000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88,"5min":0.88,"15min":1.111},"last":0.52700000000000002}]},{"osd":6,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56799999999999995,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.29799999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.5580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":0.90400000000000003,"15min":0.98799999999999999},"last":0.50800000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.312},"max":{"1min":1.016,"5min":1.016,"15min":1.0620000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64900000000000002,"15min":0.64200000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.33700000000000002},"max":{"1min":0.873,"5min":0.873,"15min":1.1160000000000001},"last":0.69199999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575382,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328896,"kb_used_data":23416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94042944,"statfs":{"total":96636764160,"available":96299974656,"internally_reserved":0,"allocated":23977984,"data_stored":83205478,"data_compressed":13026742,"data_compressed_allocated":15880192,"data_compressed_original":77045760,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.438,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":0.69499999999999995,"5min":0.75,"15min":0.82199999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56499999999999995,"15min":0.59699999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.63800000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46300000000000002}]},{"osd":1,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57599999999999996,"15min":0.55900000000000005},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.29699999999999999},"max":{"1min":0.745,"5min":0.79400000000000004,"15min":0.79800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.56999999999999995},"min":{"1min":0.48199999999999998,"5min":0.39200000000000002,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.432}]},{"osd":2,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.54000000000000004},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.30499999999999999},"max":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.442},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.39500000000000002,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.74099999999999999,"5min":0.75600000000000001,"15min":0.79000000000000004},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.86099999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59499999999999997,"15min":0.58799999999999997},"min":{"1min":0.46899999999999997,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.81799999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.50900000000000001}]},{"osd":4,"last update":"Tue Oct 10 19:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.45700000000000002},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.251},"max":{"1min":0.69999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58099999999999996,"15min":0.55400000000000005},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.23000000000000001},"max":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.49199999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.497},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.26700000000000002},"max":{"1min":0.72199999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53800000000000003},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.32100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.376}]},{"osd":7,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76600000000000001,"5min":0.86299999999999999,"15min":0.89400000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.27700000000000002},"max":{"1min":0.80500000000000005,"5min":0.84599999999999997,"15min":0.90800000000000003},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20283392,"data_stored":68677505,"data_compressed":9126818,"data_compressed_allocated":11898880,"data_compressed_original":60325888,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24403968,"data_stored":99763638,"data_compressed":12930657,"data_compressed_allocated":16519168,"data_compressed_original":91906048,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":51052075,"data_compressed":6450195,"data_compressed_allocated":8323072,"data_compressed_original":46788608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25817088,"data_stored":83490082,"data_compressed":10157145,"data_compressed_allocated":13479936,"data_compressed_original":71184384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24055808,"data_stored":89060236,"data_compressed":11845848,"data_compressed_allocated":15089664,"data_compressed_original":80117760,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16453632,"data_stored":73683110,"data_compressed":9606977,"data_compressed_allocated":12140544,"data_compressed_original":69410816,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21950464,"data_stored":75910818,"data_compressed":9436899,"data_compressed_allocated":12021760,"data_compressed_original":66007040,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":33710080,"data_stored":114783300,"data_compressed":15200435,"data_compressed_allocated":19759104,"data_compressed_original":100864000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8540160,"data_stored":11835204,"data_compressed":3779410,"data_compressed_allocated":4104192,"data_compressed_original":8499200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5124096,"data_stored":8614212,"data_compressed":3583818,"data_compressed_allocated":3870720,"data_compressed_original":8044544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3358720,"data_stored":4649136,"data_compressed":1917131,"data_compressed_allocated":2080768,"data_compressed_original":4046848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9498624,"data_stored":11984861,"data_compressed":2989035,"data_compressed_allocated":3276800,"data_compressed_original":6885376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8245248,"data_stored":11491153,"data_compressed":3663219,"data_compressed_allocated":3964928,"data_compressed_original":8294400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5943296,"data_stored":8636432,"data_compressed":3419765,"data_compressed_allocated":3739648,"data_compressed_original":7634944,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12181504,"data_stored":13790851,"data_compressed":2177080,"data_compressed_allocated":2371584,"data_compressed_original":4780032,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":15069585,"data_compressed":4432090,"data_compressed_allocated":4771840,"data_compressed_original":9945088,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T19:29:24.016 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T19:29:24.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.166+0000 7f2d7759e700 1 Processor -- start 2023-10-10T19:29:24.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.166+0000 7f2d7759e700 1 -- start start 2023-10-10T19:29:24.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7759e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2d7814fd90 0x7f2d78159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:24.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7759e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d78067fa0 con 0x7f2d780cd040 2023-10-10T19:29:24.172 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7759e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d78156b70 con 0x7f2d7808a100 2023-10-10T19:29:24.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7759e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d780c5600 con 0x7f2d7814fd90 2023-10-10T19:29:24.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d76d9d700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2d780cd040 0x7f2d780684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:46426/0 (socket says 172.21.15.175:46426) 2023-10-10T19:29:24.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d76d9d700 1 -- 172.21.15.175:0/1651311377 learned_addr learned my addr 172.21.15.175:0/1651311377 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:24.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7659c700 1 --2- 172.21.15.175:0/1651311377 >> v2:172.21.15.191:3300/0 conn(0x7f2d7814fd90 0x7f2d78159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:24.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010066056 0 0) 0x7f2d78067fa0 con 0x7f2d780cd040 2023-10-10T19:29:24.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d60003570 con 0x7f2d780cd040 2023-10-10T19:29:24.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3547380676 0 0) 0x7f2d60003570 con 0x7f2d780cd040 2023-10-10T19:29:24.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 >> v2:172.21.15.191:3300/0 conn(0x7f2d7814fd90 msgr2=0x7f2d78159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:24.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 --2- 172.21.15.175:0/1651311377 >> v2:172.21.15.191:3300/0 conn(0x7f2d7814fd90 0x7f2d78159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 >> v1:172.21.15.191:6789/0 conn(0x7f2d7808a100 legacy=0x7f2d780c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d78150160 con 0x7f2d780cd040 2023-10-10T19:29:24.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f2d700017f0 con 0x7f2d780cd040 2023-10-10T19:29:24.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 >> v1:172.21.15.175:6789/0 conn(0x7f2d780cd040 legacy=0x7f2d780684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.167+0000 7f2d7559a700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d60003a60 0x7f2d60003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:24.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7559a700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d60004810 0x7f2d60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:24.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7559a700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d6000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:24.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780c5600 con 0x7f2d600051c0 2023-10-10T19:29:24.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000c0d0 con 0x7f2d60003a60 2023-10-10T19:29:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000c290 con 0x7f2d60004810 2023-10-10T19:29:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7659c700 1 --2- 172.21.15.175:0/1651311377 >> v2:172.21.15.191:3300/0 conn(0x7f2d7814fd90 0x7f2d78159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d76d9d700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d6000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:24.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7659c700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d60003a60 0x7f2d60003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:24.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d75d9b700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d60004810 0x7f2d60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:24.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d76d9d700 1 -- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d60004810 msgr2=0x7f2d60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:24.179 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d76d9d700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d60004810 0x7f2d60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d76d9d700 1 -- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d60003a60 msgr2=0x7f2d60003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d76d9d700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d60003a60 0x7f2d60003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d76d9d700 1 -- 172.21.15.175:0/1651311377 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d60003570 con 0x7f2d600051c0 2023-10-10T19:29:24.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d75d9b700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d60004810 0x7f2d60004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d76d9d700 1 -- 172.21.15.175:0/1651311377 reap_dead start 2023-10-10T19:29:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d76d9d700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d6000ba60 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000a980 tx=0x7f2d78156ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce650e2ddf7b077f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d700070f0 con 0x7f2d600051c0 2023-10-10T19:29:24.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d70007270 con 0x7f2d600051c0 2023-10-10T19:29:24.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.168+0000 7f2d7559a700 1 -- 172.21.15.175:0/1651311377 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70007630 con 0x7f2d600051c0 2023-10-10T19:29:24.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.169+0000 7f2d7759e700 1 -- 172.21.15.175:0/1651311377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 msgr2=0x7f2d6000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.169+0000 7f2d7759e700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d6000ba60 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000a980 tx=0x7f2d78156ae0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.169+0000 7f2d7759e700 1 -- 172.21.15.175:0/1651311377 shutdown_connections 2023-10-10T19:29:24.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.169+0000 7f2d7759e700 1 --2- 172.21.15.175:0/1651311377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d6000ba60 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.169+0000 7f2d7759e700 1 -- 172.21.15.175:0/1651311377 >> 172.21.15.175:0/1651311377 conn(0x7f2d78063d00 msgr2=0x7f2d780cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:24.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.169+0000 7f2d7759e700 1 -- 172.21.15.175:0/1651311377 shutdown_connections 2023-10-10T19:29:24.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.169+0000 7f2d7759e700 1 -- 172.21.15.175:0/1651311377 wait complete. 2023-10-10T19:29:24.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.170+0000 7f2d7759e700 1 Processor -- start 2023-10-10T19:29:24.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.170+0000 7f2d7759e700 1 -- start start 2023-10-10T19:29:24.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7759e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d781668c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7759e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d78155000 0x7f2d7815b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7759e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d7815bab0 0x7f2d7814cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7759e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780c8be0 con 0x7f2d600051c0 2023-10-10T19:29:24.185 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7759e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7814c620 con 0x7f2d7815bab0 2023-10-10T19:29:24.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7759e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7814e1f0 con 0x7f2d78155000 2023-10-10T19:29:24.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d781668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:24.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d781668c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:54488/0 (socket says 172.21.15.175:54488) 2023-10-10T19:29:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 -- 172.21.15.175:0/3936924577 learned_addr learned my addr 172.21.15.175:0/3936924577 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d76d9d700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d7815bab0 0x7f2d7814cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d75d9b700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d78155000 0x7f2d7815b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:24.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 -- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d78155000 msgr2=0x7f2d7815b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d78155000 0x7f2d7815b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 -- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d7815bab0 msgr2=0x7f2d7814cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.188 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d7815bab0 0x7f2d7814cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 -- 172.21.15.175:0/3936924577 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7814e4d0 con 0x7f2d600051c0 2023-10-10T19:29:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d75d9b700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d78155000 0x7f2d7815b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d76d9d700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d7815bab0 0x7f2d7814cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:24.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7659c700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d781668c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f2d68011d70 tx=0x7f2d680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d667fc700 1 -- 172.21.15.175:0/3936924577 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800fd60 con 0x7f2d600051c0 2023-10-10T19:29:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d667fc700 1 -- 172.21.15.175:0/3936924577 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6800fec0 con 0x7f2d600051c0 2023-10-10T19:29:24.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7814e7b0 con 0x7f2d600051c0 2023-10-10T19:29:24.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d667fc700 1 -- 172.21.15.175:0/3936924577 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6801d6d0 con 0x7f2d600051c0 2023-10-10T19:29:24.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.171+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7814ed70 con 0x7f2d600051c0 2023-10-10T19:29:24.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.172+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7814c970 con 0x7f2d600051c0 2023-10-10T19:29:24.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.173+0000 7f2d667fc700 1 -- 172.21.15.175:0/3936924577 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2d6801dac0 con 0x7f2d600051c0 2023-10-10T19:29:24.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.173+0000 7f2d667fc700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d58066f30 0x7f2d580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:24.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.173+0000 7f2d667fc700 1 -- 172.21.15.175:0/3936924577 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f2d68018070 con 0x7f2d600051c0 2023-10-10T19:29:24.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.173+0000 7f2d75d9b700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d58066f30 0x7f2d580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:24.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.174+0000 7f2d75d9b700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d58066f30 0x7f2d580693e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2d781553d0 tx=0x7f2d6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.177+0000 7f2d667fc700 1 -- 172.21.15.175:0/3936924577 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d680527f0 con 0x7f2d600051c0 2023-10-10T19:29:24.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.365+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d78071210 con 0x7f2d600051c0 2023-10-10T19:29:24.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.366+0000 7f2d667fc700 1 -- 172.21.15.175:0/3936924577 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+18765 (secure 0 0 0) 0x7f2d78071210 con 0x7f2d600051c0 2023-10-10T19:29:24.367 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:24.367 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":132,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T19:28:33.906285+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":629,"snap_epoch":123,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:6834/840318079":"2023-10-11T19:28:33.906197+0000","172.21.15.175:6834/3457422611":"2023-10-11T19:27:28.893896+0000","172.21.15.191:6834/2914004704":"2023-10-11T19:25:43.872713+0000","172.21.15.191:6839/2393527600":"2023-10-11T19:24:58.863855+0000","172.21.15.175:6836/972555107":"2023-10-11T19:22:23.833708+0000","172.21.15.191:6838/2393527600":"2023-10-11T19:24:58.863855+0000","172.21.15.175:6833/972555107":"2023-10-11T19:22:23.833708+0000","172.21.15.175:6832/3874110823":"2023-10-11T19:00:33.566369+0000","172.21.15.175:6837/1881948738":"2023-10-11T19:11:03.690607+0000","172.21.15.191:6836/3199867402":"2023-10-11T18:59:08.550308+0000","172.21.15.191:6838/1598171176":"2023-10-11T19:03:13.596165+0000","172.21.15.191:6835/3199867402":"2023-10-11T18:59:08.550308+0000","172.21.15.175:6835/2966961140":"2023-10-11T18:55:38.509904+0000","172.21.15.175:6834/3874110823":"2023-10-11T19:00:33.566369+0000","172.21.15.191:6838/246221790":"2023-10-11T18:53:33.481931+0000","172.21.15.191:6837/2914004704":"2023-10-11T19:25:43.872713+0000","172.21.15.191:6837/1851345295":"2023-10-11T19:21:03.818037+0000","172.21.15.191:6837/3870390806":"2023-10-11T18:57:08.527329+0000","172.21.15.191:6839/246221790":"2023-10-11T18:53:33.481931+0000","172.21.15.191:6834/1851345295":"2023-10-11T19:21:03.818037+0000","172.21.15.175:6832/3457422611":"2023-10-11T19:27:28.893896+0000","172.21.15.191:6835/3007475551":"2023-10-11T18:47:43.409173+0000","172.21.15.175:6836/647425682":"2023-10-11T18:52:23.468892+0000","172.21.15.175:6834/688515950":"2023-10-11T18:49:33.433082+0000","172.21.15.175:6833/647425682":"2023-10-11T18:52:23.468892+0000","172.21.15.175:6837/2966961140":"2023-10-11T18:55:38.509904+0000","172.21.15.175:6832/688515950":"2023-10-11T18:49:33.433082+0000","172.21.15.191:6835/1945747421":"2023-10-11T19:05:28.622478+0000","172.21.15.191:6839/1598171176":"2023-10-11T19:03:13.596165+0000","172.21.15.191:6836/1945747421":"2023-10-11T19:05:28.622478+0000","172.21.15.175:6833/2375829435":"2023-10-11T19:07:23.643743+0000","172.21.15.191:6838/1910071809":"2023-10-11T19:08:28.655326+0000","172.21.15.191:6837/840318079":"2023-10-11T19:28:33.906197+0000","172.21.15.191:6839/1910071809":"2023-10-11T19:08:28.655326+0000","172.21.15.191:6834/824552134":"2023-10-11T19:09:53.672299+0000","172.21.15.175:6832/1872591144":"2023-10-11T19:14:43.733216+0000","172.21.15.175:6835/3849771254":"2023-10-11T19:23:38.848907+0000","172.21.15.175:6834/1872591144":"2023-10-11T19:14:43.733216+0000","172.21.15.175:6837/3849771254":"2023-10-11T19:23:38.848907+0000","172.21.15.191:6837/824552134":"2023-10-11T19:09:53.672299+0000","172.21.15.191:6834/3870390806":"2023-10-11T18:57:08.527329+0000","172.21.15.175:6835/1881948738":"2023-10-11T19:11:03.690607+0000","172.21.15.191:6835/4084912544":"2023-10-11T19:17:13.770352+0000","172.21.15.191:6836/3007475551":"2023-10-11T18:47:43.409173+0000","172.21.15.175:6836/2375829435":"2023-10-11T19:07:23.643743+0000","172.21.15.191:6836/4084912544":"2023-10-11T19:17:13.770352+0000","172.21.15.191:6835/62572703":"2023-10-11T19:13:48.722600+0000","172.21.15.191:6836/62572703":"2023-10-11T19:13:48.722600+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T19:29:24.368 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d58066f30 msgr2=0x7f2d580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d58066f30 0x7f2d580693e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2d781553d0 tx=0x7f2d6c009040 comp rx=0 tx=0).stop 2023-10-10T19:29:24.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 msgr2=0x7f2d781668c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:24.369 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d781668c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f2d68011d70 tx=0x7f2d680091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 shutdown_connections 2023-10-10T19:29:24.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f2d58066f30 0x7f2d580693e0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d7815bab0 0x7f2d7814cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d600051c0 0x7f2d781668c0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 --2- 172.21.15.175:0/3936924577 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d78155000 0x7f2d7815b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:24.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 >> 172.21.15.175:0/3936924577 conn(0x7f2d78063d00 msgr2=0x7f2d780cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:24.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 shutdown_connections 2023-10-10T19:29:24.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:24.368+0000 7f2d7759e700 1 -- 172.21.15.175:0/3936924577 wait complete. 2023-10-10T19:29:25.381 INFO:tasks.ceph:Scrubbing osd.0 2023-10-10T19:29:25.382 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-10T19:29:25.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.502+0000 7f415d33b700 1 Processor -- start 2023-10-10T19:29:25.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.502+0000 7f415d33b700 1 -- start start 2023-10-10T19:29:25.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f415d33b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f41580bc720 0x7f41581569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f415d33b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580c5130 0x7f41580c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f415d33b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580c4ca0 con 0x7f41580bcaf0 2023-10-10T19:29:25.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f415d33b700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4158068470 con 0x7f41580bc720 2023-10-10T19:29:25.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f415d33b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4158156f30 con 0x7f41580c5130 2023-10-10T19:29:25.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f4156ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f41580bc720 0x7f41581569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f4156ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f41580bc720 0x7f41581569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60522/0 (socket says 172.21.15.175:60522) 2023-10-10T19:29:25.507 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f4156ffd700 1 -- 172.21.15.175:0/1176547343 learned_addr learned my addr 172.21.15.175:0/1176547343 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:25.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.503+0000 7f41577fe700 1 --2- 172.21.15.175:0/1176547343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580c5130 0x7f41580c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f4156ffd700 1 -- 172.21.15.175:0/1176547343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580c5130 msgr2=0x7f41580c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:25.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f4156ffd700 1 --2- 172.21.15.175:0/1176547343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580c5130 0x7f41580c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.508 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f4156ffd700 1 -- 172.21.15.175:0/1176547343 >> v1:172.21.15.191:6789/0 conn(0x7f41580bcaf0 legacy=0x7f415806e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f4156ffd700 1 -- 172.21.15.175:0/1176547343 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580c5b10 con 0x7f41580bc720 2023-10-10T19:29:25.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f4156ffd700 1 --2- 172.21.15.175:0/1176547343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 0x7f41581569f0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f41480099e0 tx=0x7f4148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e7cedea7df5aa4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:25.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f4155ffb700 1 -- 172.21.15.175:0/1176547343 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414800b8b0 con 0x7f41580bc720 2023-10-10T19:29:25.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f4155ffb700 1 -- 172.21.15.175:0/1176547343 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414800ba10 con 0x7f41580bc720 2023-10-10T19:29:25.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f4155ffb700 1 -- 172.21.15.175:0/1176547343 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414800bdb0 con 0x7f41580bc720 2023-10-10T19:29:25.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f415d33b700 1 -- 172.21.15.175:0/1176547343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 msgr2=0x7f41581569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f415d33b700 1 --2- 172.21.15.175:0/1176547343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 0x7f41581569f0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f41480099e0 tx=0x7f4148009250 comp rx=0 tx=0).stop 2023-10-10T19:29:25.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f415d33b700 1 -- 172.21.15.175:0/1176547343 shutdown_connections 2023-10-10T19:29:25.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f415d33b700 1 --2- 172.21.15.175:0/1176547343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580c5130 0x7f41580c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f415d33b700 1 --2- 172.21.15.175:0/1176547343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 0x7f41581569f0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f415d33b700 1 -- 172.21.15.175:0/1176547343 >> 172.21.15.175:0/1176547343 conn(0x7f41580c9460 msgr2=0x7f41580c9860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:25.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f415d33b700 1 -- 172.21.15.175:0/1176547343 shutdown_connections 2023-10-10T19:29:25.512 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.504+0000 7f415d33b700 1 -- 172.21.15.175:0/1176547343 wait complete. 2023-10-10T19:29:25.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f415d33b700 1 Processor -- start 2023-10-10T19:29:25.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f415d33b700 1 -- start start 2023-10-10T19:29:25.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f415d33b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 0x7f41580c2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f415d33b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580bcaf0 0x7f41580c2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f415d33b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41580c5130 0x7f415814bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f415d33b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41580c1890 con 0x7f41580bc720 2023-10-10T19:29:25.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f415d33b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41580c1e70 con 0x7f41580c5130 2023-10-10T19:29:25.517 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f415d33b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415815a320 con 0x7f41580bcaf0 2023-10-10T19:29:25.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.506+0000 7f4156ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 0x7f41580c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f4156ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 0x7f41580c2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60528/0 (socket says 172.21.15.175:60528) 2023-10-10T19:29:25.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f4156ffd700 1 -- 172.21.15.175:0/3498620338 learned_addr learned my addr 172.21.15.175:0/3498620338 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:25.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f41577fe700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41580c5130 0x7f415814bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f41567fc700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580bcaf0 0x7f41580c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f4156ffd700 1 -- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580bcaf0 msgr2=0x7f41580c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.519 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f4156ffd700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41580bcaf0 0x7f41580c2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f4156ffd700 1 -- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41580c5130 msgr2=0x7f415814bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f4156ffd700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41580c5130 0x7f415814bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f4156ffd700 1 -- 172.21.15.175:0/3498620338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41480094f0 con 0x7f41580bc720 2023-10-10T19:29:25.520 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f4156ffd700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 0x7f41580c2300 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f4148009290 tx=0x7f4148017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:25.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148011d70 con 0x7f41580bc720 2023-10-10T19:29:25.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4148024440 con 0x7f41580bc720 2023-10-10T19:29:25.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415815a5e0 con 0x7f41580bc720 2023-10-10T19:29:25.521 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148024800 con 0x7f41580bc720 2023-10-10T19:29:25.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.507+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415815aba0 con 0x7f41580bc720 2023-10-10T19:29:25.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.508+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4158067570 con 0x7f41580bc720 2023-10-10T19:29:25.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.508+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f414801a420 con 0x7f41580bc720 2023-10-10T19:29:25.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f413effd700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4144066f60 0x7f4144069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f4148085990 con 0x7f41580bc720 2023-10-10T19:29:25.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f41567fc700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4144066f60 0x7f4144069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f413effd700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7f41440705a0 0x7f4144072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 --> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41440730e0 con 0x7f41440705a0 2023-10-10T19:29:25.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7f414808a240 con 0x7f41580bc720 2023-10-10T19:29:25.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f41577fe700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7f41440705a0 0x7f4144072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.524 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f41567fc700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4144066f60 0x7f4144069410 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f41580c3e50 tx=0x7f414c007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:25.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.509+0000 7f41577fe700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7f41440705a0 0x7f4144072a10 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:25.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.510+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 <== osd.0 v2:172.21.15.175:6808/90849 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f41440730e0 con 0x7f41440705a0 2023-10-10T19:29:25.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 --> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f41580670b0 con 0x7f41440705a0 2023-10-10T19:29:25.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f413effd700 1 -- 172.21.15.175:0/3498620338 <== osd.0 v2:172.21.15.175:6808/90849 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f41580670b0 con 0x7f41440705a0 2023-10-10T19:29:25.527 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T19:29:25.527 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-10T19:29:25.527 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T19:29:25.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7f41440705a0 msgr2=0x7f4144072a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.175:6808/90849,v1:172.21.15.175:6809/90849] conn(0x7f41440705a0 0x7f4144072a10 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4144066f60 msgr2=0x7f4144069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f4144066f60 0x7f4144069410 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f41580c3e50 tx=0x7f414c007ec0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 msgr2=0x7f41580c2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 --2- 172.21.15.175:0/3498620338 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41580bc720 0x7f41580c2300 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f4148009290 tx=0x7f4148017000 comp rx=0 tx=0).stop 2023-10-10T19:29:25.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f41577fe700 1 -- 172.21.15.175:0/3498620338 reap_dead start 2023-10-10T19:29:25.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 shutdown_connections 2023-10-10T19:29:25.530 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 >> 172.21.15.175:0/3498620338 conn(0x7f41580c9460 msgr2=0x7f415815b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:25.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 shutdown_connections 2023-10-10T19:29:25.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.524+0000 7f415d33b700 1 -- 172.21.15.175:0/3498620338 wait complete. 2023-10-10T19:29:25.536 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-10T19:29:25.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.651+0000 7f9cb1be0700 1 Processor -- start 2023-10-10T19:29:25.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.651+0000 7f9cb1be0700 1 -- start start 2023-10-10T19:29:25.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cb1be0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9cac158af0 0x7f9cac155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cb1be0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac158ec0 0x7f9cac159310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cb1be0700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac157210 con 0x7f9cac0c6f50 2023-10-10T19:29:25.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cb1be0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cac15a260 con 0x7f9cac158af0 2023-10-10T19:29:25.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cb1be0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac150590 con 0x7f9cac158ec0 2023-10-10T19:29:25.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cb13df700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9cac0c6f50 0x7f9cac0afd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41970/0 (socket says 172.21.15.175:41970) 2023-10-10T19:29:25.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cb13df700 1 -- 172.21.15.175:0/1295260832 learned_addr learned my addr 172.21.15.175:0/1295260832 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:25.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cabfff700 1 --2- 172.21.15.175:0/1295260832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac158ec0 0x7f9cac159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cb0bde700 1 --2- 172.21.15.175:0/1295260832 >> v2:172.21.15.191:3300/0 conn(0x7f9cac158af0 0x7f9cac155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cab7fe700 1 -- 172.21.15.175:0/1295260832 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709774208 0 0) 0x7f9cac157210 con 0x7f9cac0c6f50 2023-10-10T19:29:25.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.652+0000 7f9cab7fe700 1 -- 172.21.15.175:0/1295260832 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94003570 con 0x7f9cac0c6f50 2023-10-10T19:29:25.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cabfff700 1 -- 172.21.15.175:0/1295260832 >> v2:172.21.15.191:3300/0 conn(0x7f9cac158af0 msgr2=0x7f9cac155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:25.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cabfff700 1 --2- 172.21.15.175:0/1295260832 >> v2:172.21.15.191:3300/0 conn(0x7f9cac158af0 0x7f9cac155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cabfff700 1 -- 172.21.15.175:0/1295260832 >> v1:172.21.15.175:6789/0 conn(0x7f9cac0c6f50 legacy=0x7f9cac0afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cab7fe700 1 -- 172.21.15.175:0/1295260832 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106924198 0 0) 0x7f9c94003570 con 0x7f9cac0c6f50 2023-10-10T19:29:25.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cabfff700 1 -- 172.21.15.175:0/1295260832 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac0af8b0 con 0x7f9cac158ec0 2023-10-10T19:29:25.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cb0bde700 1 --2- 172.21.15.175:0/1295260832 >> v2:172.21.15.191:3300/0 conn(0x7f9cac158af0 0x7f9cac155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:25.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cabfff700 1 --2- 172.21.15.175:0/1295260832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac158ec0 0x7f9cac159310 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00098e0 tx=0x7f9ca00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63ea5303ca38783e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:25.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cab7fe700 1 -- 172.21.15.175:0/1295260832 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40017f0 con 0x7f9cac158ec0 2023-10-10T19:29:25.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cab7fe700 1 -- 172.21.15.175:0/1295260832 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca0011420 con 0x7f9cac158ec0 2023-10-10T19:29:25.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.653+0000 7f9cab7fe700 1 -- 172.21.15.175:0/1295260832 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca00117a0 con 0x7f9cac158ec0 2023-10-10T19:29:25.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.654+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1295260832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac158ec0 msgr2=0x7f9cac159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.654+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1295260832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac158ec0 0x7f9cac159310 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00098e0 tx=0x7f9ca00091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.654+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1295260832 shutdown_connections 2023-10-10T19:29:25.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.654+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1295260832 >> v2:172.21.15.191:3300/0 conn(0x7f9cac158af0 0x7f9cac155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.654+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1295260832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac158ec0 0x7f9cac159310 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.654+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1295260832 >> 172.21.15.175:0/1295260832 conn(0x7f9cac063230 msgr2=0x7f9cac063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:25.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.654+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1295260832 shutdown_connections 2023-10-10T19:29:25.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.654+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1295260832 wait complete. 2023-10-10T19:29:25.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.655+0000 7f9cb1be0700 1 Processor -- start 2023-10-10T19:29:25.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.655+0000 7f9cb1be0700 1 -- start start 2023-10-10T19:29:25.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb1be0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac0c6f50 0x7f9cac1538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb1be0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9cac158af0 0x7f9cac152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb1be0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cac152d80 0x7f9cac150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb1be0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac0cc070 con 0x7f9cac158af0 2023-10-10T19:29:25.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb1be0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac1523f0 con 0x7f9cac152d80 2023-10-10T19:29:25.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb1be0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac153220 con 0x7f9cac0c6f50 2023-10-10T19:29:25.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9cac158af0 0x7f9cac152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9cac158af0 0x7f9cac152840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60536/0 (socket says 172.21.15.175:60536) 2023-10-10T19:29:25.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 -- 172.21.15.175:0/1624216773 learned_addr learned my addr 172.21.15.175:0/1624216773 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:25.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb13df700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cac152d80 0x7f9cac150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb0bde700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac0c6f50 0x7f9cac1538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 -- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac0c6f50 msgr2=0x7f9cac1538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac0c6f50 0x7f9cac1538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 -- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cac152d80 msgr2=0x7f9cac150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cac152d80 0x7f9cac150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 -- 172.21.15.175:0/1624216773 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c009420 con 0x7f9cac158af0 2023-10-10T19:29:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cabfff700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9cac158af0 0x7f9cac152840 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f9cac0b1a50 tx=0x7f9ca0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9ca8ff9700 1 -- 172.21.15.175:0/1624216773 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca001caf0 con 0x7f9cac158af0 2023-10-10T19:29:25.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9ca8ff9700 1 -- 172.21.15.175:0/1624216773 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca001cc50 con 0x7f9cac158af0 2023-10-10T19:29:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca0009420 con 0x7f9cac158af0 2023-10-10T19:29:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9ca8ff9700 1 -- 172.21.15.175:0/1624216773 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca001a450 con 0x7f9cac158af0 2023-10-10T19:29:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.656+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac151630 con 0x7f9cac158af0 2023-10-10T19:29:25.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.657+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac152560 con 0x7f9cac158af0 2023-10-10T19:29:25.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.658+0000 7f9ca8ff9700 1 -- 172.21.15.175:0/1624216773 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9ca001a840 con 0x7f9cac158af0 2023-10-10T19:29:25.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.658+0000 7f9ca8ff9700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c98066f30 0x7f9c980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.658+0000 7f9ca8ff9700 1 -- 172.21.15.175:0/1624216773 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f9ca00866b0 con 0x7f9cac158af0 2023-10-10T19:29:25.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.658+0000 7f9cb0bde700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c98066f30 0x7f9c980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.659+0000 7f9cb0bde700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c98066f30 0x7f9c980693e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009ea0 tx=0x7f9c9c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:25.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.662+0000 7f9ca8ff9700 1 -- 172.21.15.175:0/1624216773 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9cac152560 con 0x7f9cac158af0 2023-10-10T19:29:25.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.847+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cac152560 con 0x7f9c98066f30 2023-10-10T19:29:25.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.848+0000 7f9ca8ff9700 1 -- 172.21.15.175:0/1624216773 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9cac152560 con 0x7f9c98066f30 2023-10-10T19:29:25.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c98066f30 msgr2=0x7f9c980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c98066f30 0x7f9c980693e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009ea0 tx=0x7f9c9c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9cac158af0 msgr2=0x7f9cac152840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9cac158af0 0x7f9cac152840 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f9cac0b1a50 tx=0x7f9ca0016000 comp rx=0 tx=0).stop 2023-10-10T19:29:25.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 shutdown_connections 2023-10-10T19:29:25.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9c98066f30 0x7f9c980693e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cac152d80 0x7f9cac150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cac0c6f50 0x7f9cac1538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 --2- 172.21.15.175:0/1624216773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9cac158af0 0x7f9cac152840 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 >> 172.21.15.175:0/1624216773 conn(0x7f9cac063230 msgr2=0x7f9cac14f920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:25.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 shutdown_connections 2023-10-10T19:29:25.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.850+0000 7f9cb1be0700 1 -- 172.21.15.175:0/1624216773 wait complete. 2023-10-10T19:29:25.854 INFO:teuthology.orchestra.run.smithi175.stderr:instructed osd(s) 0 to deep-scrub 2023-10-10T19:29:25.862 INFO:tasks.ceph:Scrubbing osd.1 2023-10-10T19:29:25.863 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-10T19:29:25.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.981+0000 7fb7cedd2700 1 Processor -- start 2023-10-10T19:29:25.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.981+0000 7fb7cedd2700 1 -- start start 2023-10-10T19:29:25.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7cedd2700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb7c80c6490 0x7fb7c8150130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7cedd2700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c80c82b0 con 0x7fb7c80c6f00 2023-10-10T19:29:25.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7cedd2700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c80c3e10 con 0x7fb7c80c60c0 2023-10-10T19:29:25.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7cedd2700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c80c49b0 con 0x7fb7c80c6490 2023-10-10T19:29:25.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ce5d1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb7c80c6490 0x7fb7c8150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ce5d1700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb7c80c6490 0x7fb7c8150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60538/0 (socket says 172.21.15.175:60538) 2023-10-10T19:29:25.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ce5d1700 1 -- 172.21.15.175:0/2834073023 learned_addr learned my addr 172.21.15.175:0/2834073023 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:25.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ce5d1700 1 -- 172.21.15.175:0/2834073023 >> v1:172.21.15.191:6789/0 conn(0x7fb7c80c60c0 legacy=0x7fb7c80c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ce5d1700 1 -- 172.21.15.175:0/2834073023 >> v1:172.21.15.175:6789/0 conn(0x7fb7c80c6f00 legacy=0x7fb7c814fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ce5d1700 1 -- 172.21.15.175:0/2834073023 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80c7a10 con 0x7fb7c80c6490 2023-10-10T19:29:25.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ce5d1700 1 --2- 172.21.15.175:0/2834073023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 0x7fb7c8150130 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009f20 tx=0x7fb7c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbe411ce0520e16 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:25.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ccdce700 1 -- 172.21.15.175:0/2834073023 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c400e070 con 0x7fb7c80c6490 2023-10-10T19:29:25.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ccdce700 1 -- 172.21.15.175:0/2834073023 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c400ba80 con 0x7fb7c80c6490 2023-10-10T19:29:25.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.982+0000 7fb7ccdce700 1 -- 172.21.15.175:0/2834073023 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c40092e0 con 0x7fb7c80c6490 2023-10-10T19:29:25.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.983+0000 7fb7cedd2700 1 -- 172.21.15.175:0/2834073023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 msgr2=0x7fb7c8150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.983+0000 7fb7cedd2700 1 --2- 172.21.15.175:0/2834073023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 0x7fb7c8150130 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009f20 tx=0x7fb7c4009840 comp rx=0 tx=0).stop 2023-10-10T19:29:25.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.983+0000 7fb7cedd2700 1 -- 172.21.15.175:0/2834073023 shutdown_connections 2023-10-10T19:29:25.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.983+0000 7fb7cedd2700 1 --2- 172.21.15.175:0/2834073023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 0x7fb7c8150130 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.983+0000 7fb7cedd2700 1 -- 172.21.15.175:0/2834073023 >> 172.21.15.175:0/2834073023 conn(0x7fb7c8063230 msgr2=0x7fb7c8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:25.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.983+0000 7fb7cedd2700 1 -- 172.21.15.175:0/2834073023 shutdown_connections 2023-10-10T19:29:25.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.983+0000 7fb7cedd2700 1 -- 172.21.15.175:0/2834073023 wait complete. 2023-10-10T19:29:25.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.984+0000 7fb7cedd2700 1 Processor -- start 2023-10-10T19:29:25.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.984+0000 7fb7cedd2700 1 -- start start 2023-10-10T19:29:25.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cedd2700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c80c60c0 0x7fb7c8152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cedd2700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 0x7fb7c8152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cedd2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c80c6f00 0x7fb7c814e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:25.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cedd2700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c815afe0 con 0x7fb7c80c6490 2023-10-10T19:29:25.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cedd2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8165f80 con 0x7fb7c80c6f00 2023-10-10T19:29:25.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cedd2700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8166240 con 0x7fb7c80c60c0 2023-10-10T19:29:25.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 0x7fb7c8152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 0x7fb7c8152f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60542/0 (socket says 172.21.15.175:60542) 2023-10-10T19:29:25.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 -- 172.21.15.175:0/240762110 learned_addr learned my addr 172.21.15.175:0/240762110 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:25.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7ce5d1700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c80c6f00 0x7fb7c814e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cddd0700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c80c60c0 0x7fb7c8152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:25.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 -- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c80c60c0 msgr2=0x7fb7c8152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c80c60c0 0x7fb7c8152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 -- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c80c6f00 msgr2=0x7fb7c814e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:25.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c80c6f00 0x7fb7c814e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:25.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 -- 172.21.15.175:0/240762110 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4009ae0 con 0x7fb7c80c6490 2023-10-10T19:29:25.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cddd0700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7c80c60c0 0x7fb7c8152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:25.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7ce5d1700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7c80c6f00 0x7fb7c814e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:25.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cd5cf700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 0x7fb7c8152f50 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800c7c0 tx=0x7fb7b8007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8012630 con 0x7fb7c80c6490 2023-10-10T19:29:26.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b8012790 con 0x7fb7c80c6490 2023-10-10T19:29:26.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c815a3d0 con 0x7fb7c80c6490 2023-10-10T19:29:26.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.985+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8012b50 con 0x7fb7c80c6490 2023-10-10T19:29:26.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.986+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c815a990 con 0x7fb7c80c6490 2023-10-10T19:29:26.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.986+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb7c804bb80 con 0x7fb7c80c6490 2023-10-10T19:29:26.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.987+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb7b801b400 con 0x7fb7c80c6490 2023-10-10T19:29:26.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.987+0000 7fb7bdffb700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7ac066f00 0x7fb7ac0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.987+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fb7b8084960 con 0x7fb7c80c6490 2023-10-10T19:29:26.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.987+0000 7fb7bdffb700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fb7ac070540 0x7fb7ac0729b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.987+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 --> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7ac073080 con 0x7fb7ac070540 2023-10-10T19:29:26.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.987+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7b8089240 con 0x7fb7c80c6490 2023-10-10T19:29:26.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.987+0000 7fb7cddd0700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7ac066f00 0x7fb7ac0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.987+0000 7fb7ce5d1700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fb7ac070540 0x7fb7ac0729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.988+0000 7fb7ce5d1700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fb7ac070540 0x7fb7ac0729b0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.988+0000 7fb7cddd0700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7ac066f00 0x7fb7ac0693b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0007fd0 tx=0x7fb7c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:25.988+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 <== osd.1 v2:172.21.15.175:6800/90850 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fb7ac073080 con 0x7fb7ac070540 2023-10-10T19:29:26.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.002+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 --> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb7c8066b60 con 0x7fb7ac070540 2023-10-10T19:29:26.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.002+0000 7fb7bdffb700 1 -- 172.21.15.175:0/240762110 <== osd.1 v2:172.21.15.175:6800/90850 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb7c8066b60 con 0x7fb7ac070540 2023-10-10T19:29:26.006 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T19:29:26.006 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-10T19:29:26.006 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T19:29:26.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fb7ac070540 msgr2=0x7fb7ac0729b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.175:6800/90850,v1:172.21.15.175:6801/90850] conn(0x7fb7ac070540 0x7fb7ac0729b0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7ac066f00 msgr2=0x7fb7ac0693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fb7ac066f00 0x7fb7ac0693b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0007fd0 tx=0x7fb7c0009040 comp rx=0 tx=0).stop 2023-10-10T19:29:26.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 msgr2=0x7fb7c8152f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 --2- 172.21.15.175:0/240762110 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7c80c6490 0x7fb7c8152f50 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800c7c0 tx=0x7fb7b8007f50 comp rx=0 tx=0).stop 2023-10-10T19:29:26.009 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7ce5d1700 1 -- 172.21.15.175:0/240762110 reap_dead start 2023-10-10T19:29:26.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 shutdown_connections 2023-10-10T19:29:26.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 >> 172.21.15.175:0/240762110 conn(0x7fb7c8063230 msgr2=0x7fb7c80c9110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 shutdown_connections 2023-10-10T19:29:26.011 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.003+0000 7fb7cedd2700 1 -- 172.21.15.175:0/240762110 wait complete. 2023-10-10T19:29:26.014 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-10T19:29:26.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.129+0000 7f579abd0700 1 Processor -- start 2023-10-10T19:29:26.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.129+0000 7f579abd0700 1 -- start start 2023-10-10T19:29:26.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579abd0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f579414dd90 0x7f57940c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579abd0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57940ca200 0x7f57940c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579abd0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579406ceb0 con 0x7f579414e160 2023-10-10T19:29:26.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579abd0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f579406c2c0 con 0x7f579414dd90 2023-10-10T19:29:26.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579abd0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579406c580 con 0x7f57940ca200 2023-10-10T19:29:26.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579916d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57940ca200 0x7f57940c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579896c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f579414dd90 0x7f57940c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579896c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f579414dd90 0x7f57940c2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:38702/0 (socket says 172.21.15.175:38702) 2023-10-10T19:29:26.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f5793fff700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f579414e160 0x7f5794153f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52928/0 (socket says 172.21.15.175:52928) 2023-10-10T19:29:26.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579916d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57940ca200 0x7f57940c3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:40064/0 (socket says 172.21.15.175:40064) 2023-10-10T19:29:26.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.130+0000 7f579896c700 1 -- 172.21.15.175:0/1820244397 learned_addr learned my addr 172.21.15.175:0/1820244397 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:26.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579916d700 1 -- 172.21.15.175:0/1820244397 >> v2:172.21.15.191:3300/0 conn(0x7f579414dd90 msgr2=0x7f57940c2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:26.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579916d700 1 --2- 172.21.15.175:0/1820244397 >> v2:172.21.15.191:3300/0 conn(0x7f579414dd90 0x7f57940c2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579916d700 1 -- 172.21.15.175:0/1820244397 >> v1:172.21.15.191:6789/0 conn(0x7f579414e160 legacy=0x7f5794153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.136 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579916d700 1 -- 172.21.15.175:0/1820244397 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579406c860 con 0x7f57940ca200 2023-10-10T19:29:26.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f57937fe700 1 -- 172.21.15.175:0/1820244397 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297645344 0 0) 0x7f579406ceb0 con 0x7f579414e160 2023-10-10T19:29:26.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579896c700 1 --2- 172.21.15.175:0/1820244397 >> v2:172.21.15.191:3300/0 conn(0x7f579414dd90 0x7f57940c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:26.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579916d700 1 --2- 172.21.15.175:0/1820244397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57940ca200 0x7f57940c3020 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f578c0098e0 tx=0x7f578c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12be1152adce8e39 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f57937fe700 1 -- 172.21.15.175:0/1820244397 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c00b640 con 0x7f57940ca200 2023-10-10T19:29:26.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f57937fe700 1 -- 172.21.15.175:0/1820244397 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c00b7a0 con 0x7f57940ca200 2023-10-10T19:29:26.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f57937fe700 1 -- 172.21.15.175:0/1820244397 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c00bb40 con 0x7f57940ca200 2023-10-10T19:29:26.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579abd0700 1 -- 172.21.15.175:0/1820244397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57940ca200 msgr2=0x7f57940c3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579abd0700 1 --2- 172.21.15.175:0/1820244397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57940ca200 0x7f57940c3020 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f578c0098e0 tx=0x7f578c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579abd0700 1 -- 172.21.15.175:0/1820244397 shutdown_connections 2023-10-10T19:29:26.139 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579abd0700 1 --2- 172.21.15.175:0/1820244397 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57940ca200 0x7f57940c3020 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579abd0700 1 --2- 172.21.15.175:0/1820244397 >> v2:172.21.15.191:3300/0 conn(0x7f579414dd90 0x7f57940c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.131+0000 7f579abd0700 1 -- 172.21.15.175:0/1820244397 >> 172.21.15.175:0/1820244397 conn(0x7f57940c94f0 msgr2=0x7f57940c98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.132+0000 7f579abd0700 1 -- 172.21.15.175:0/1820244397 shutdown_connections 2023-10-10T19:29:26.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.132+0000 7f579abd0700 1 -- 172.21.15.175:0/1820244397 wait complete. 2023-10-10T19:29:26.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.133+0000 7f579abd0700 1 Processor -- start 2023-10-10T19:29:26.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.133+0000 7f579abd0700 1 -- start start 2023-10-10T19:29:26.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.133+0000 7f579abd0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57940ca200 0x7f579413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.133+0000 7f579abd0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f579414dd90 0x7f5794156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.133+0000 7f579abd0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f579414e160 0x7f5794140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.133+0000 7f579abd0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579414cfc0 con 0x7f579414e160 2023-10-10T19:29:26.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.133+0000 7f579abd0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579414c6e0 con 0x7f57940ca200 2023-10-10T19:29:26.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.133+0000 7f579abd0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5794157600 con 0x7f579414dd90 2023-10-10T19:29:26.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f579414e160 0x7f5794140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f579414e160 0x7f5794140fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60550/0 (socket says 172.21.15.175:60550) 2023-10-10T19:29:26.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 -- 172.21.15.175:0/1149455742 learned_addr learned my addr 172.21.15.175:0/1149455742 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:26.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579896c700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57940ca200 0x7f579413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f5793fff700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f579414dd90 0x7f5794156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 -- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f579414dd90 msgr2=0x7f5794156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f579414dd90 0x7f5794156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 -- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57940ca200 msgr2=0x7f579413f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57940ca200 0x7f579413f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 -- 172.21.15.175:0/1149455742 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5784009ae0 con 0x7f579414e160 2023-10-10T19:29:26.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f5793fff700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f579414dd90 0x7f5794156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:26.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579916d700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f579414e160 0x7f5794140fb0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f579406c6c0 tx=0x7f578c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f5790ff9700 1 -- 172.21.15.175:0/1149455742 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c00b9e0 con 0x7f579414e160 2023-10-10T19:29:26.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f5790ff9700 1 -- 172.21.15.175:0/1149455742 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c025420 con 0x7f579414e160 2023-10-10T19:29:26.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c009420 con 0x7f579414e160 2023-10-10T19:29:26.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f5790ff9700 1 -- 172.21.15.175:0/1149455742 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c0257e0 con 0x7f579414e160 2023-10-10T19:29:26.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.134+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579414c400 con 0x7f579414e160 2023-10-10T19:29:26.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.135+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5794154a70 con 0x7f579414e160 2023-10-10T19:29:26.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.136+0000 7f5790ff9700 1 -- 172.21.15.175:0/1149455742 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f578c023070 con 0x7f579414e160 2023-10-10T19:29:26.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.136+0000 7f5790ff9700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5780066f30 0x7f57800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.136+0000 7f5790ff9700 1 -- 172.21.15.175:0/1149455742 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f578c086500 con 0x7f579414e160 2023-10-10T19:29:26.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.136+0000 7f579896c700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5780066f30 0x7f57800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.137+0000 7f579896c700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5780066f30 0x7f57800693e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f57840072d0 tx=0x7f578400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.141+0000 7f5790ff9700 1 -- 172.21.15.175:0/1149455742 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f578c052f30 con 0x7f579414e160 2023-10-10T19:29:26.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.324+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5794067570 con 0x7f5780066f30 2023-10-10T19:29:26.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.325+0000 7f5790ff9700 1 -- 172.21.15.175:0/1149455742 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5794067570 con 0x7f5780066f30 2023-10-10T19:29:26.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5780066f30 msgr2=0x7f57800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5780066f30 0x7f57800693e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f57840072d0 tx=0x7f578400b040 comp rx=0 tx=0).stop 2023-10-10T19:29:26.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f579414e160 msgr2=0x7f5794140fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f579414e160 0x7f5794140fb0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f579406c6c0 tx=0x7f578c017000 comp rx=0 tx=0).stop 2023-10-10T19:29:26.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 shutdown_connections 2023-10-10T19:29:26.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f5780066f30 0x7f57800693e0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f579414e160 0x7f5794140fb0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57940ca200 0x7f579413f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 --2- 172.21.15.175:0/1149455742 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f579414dd90 0x7f5794156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 >> 172.21.15.175:0/1149455742 conn(0x7f57940c94f0 msgr2=0x7f579406e400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 shutdown_connections 2023-10-10T19:29:26.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.327+0000 7f579abd0700 1 -- 172.21.15.175:0/1149455742 wait complete. 2023-10-10T19:29:26.331 INFO:teuthology.orchestra.run.smithi175.stderr:instructed osd(s) 1 to deep-scrub 2023-10-10T19:29:26.339 INFO:tasks.ceph:Scrubbing osd.2 2023-10-10T19:29:26.340 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-10T19:29:26.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.458+0000 7fadaa250700 1 Processor -- start 2023-10-10T19:29:26.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.458+0000 7fadaa250700 1 -- start start 2023-10-10T19:29:26.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fadaa250700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fada40c6240 0x7fada40690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fadaa250700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada4069620 0x7fada40c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fadaa250700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada40c4b60 con 0x7fada40c6610 2023-10-10T19:29:26.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fadaa250700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada4068c50 con 0x7fada40c6240 2023-10-10T19:29:26.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fadaa250700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada4157420 con 0x7fada4069620 2023-10-10T19:29:26.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada37fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fada40c6240 0x7fada40690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada37fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fada40c6240 0x7fada40690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60552/0 (socket says 172.21.15.175:60552) 2023-10-10T19:29:26.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada37fe700 1 -- 172.21.15.175:0/153255022 learned_addr learned my addr 172.21.15.175:0/153255022 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:26.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada2ffd700 1 --2- 172.21.15.175:0/153255022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada4069620 0x7fada40c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada37fe700 1 -- 172.21.15.175:0/153255022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada4069620 msgr2=0x7fada40c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:26.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada37fe700 1 --2- 172.21.15.175:0/153255022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada4069620 0x7fada40c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada37fe700 1 -- 172.21.15.175:0/153255022 >> v1:172.21.15.191:6789/0 conn(0x7fada40c6610 legacy=0x7fada40c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada37fe700 1 -- 172.21.15.175:0/153255022 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40c4880 con 0x7fada40c6240 2023-10-10T19:29:26.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada2ffd700 1 --2- 172.21.15.175:0/153255022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada4069620 0x7fada40c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:26.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada37fe700 1 --2- 172.21.15.175:0/153255022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada40c6240 0x7fada40690e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fad980099b0 tx=0x7fad98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a32569b74df4a08 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada27fc700 1 -- 172.21.15.175:0/153255022 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9800b860 con 0x7fada40c6240 2023-10-10T19:29:26.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada27fc700 1 -- 172.21.15.175:0/153255022 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9800b9c0 con 0x7fada40c6240 2023-10-10T19:29:26.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.459+0000 7fada27fc700 1 -- 172.21.15.175:0/153255022 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9800bd60 con 0x7fada40c6240 2023-10-10T19:29:26.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.460+0000 7fadaa250700 1 -- 172.21.15.175:0/153255022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada40c6240 msgr2=0x7fada40690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.460+0000 7fadaa250700 1 --2- 172.21.15.175:0/153255022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada40c6240 0x7fada40690e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fad980099b0 tx=0x7fad98009250 comp rx=0 tx=0).stop 2023-10-10T19:29:26.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.460+0000 7fadaa250700 1 -- 172.21.15.175:0/153255022 shutdown_connections 2023-10-10T19:29:26.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.460+0000 7fadaa250700 1 --2- 172.21.15.175:0/153255022 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada40c6240 0x7fada40690e0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.460+0000 7fadaa250700 1 --2- 172.21.15.175:0/153255022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada4069620 0x7fada40c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.460+0000 7fadaa250700 1 -- 172.21.15.175:0/153255022 >> 172.21.15.175:0/153255022 conn(0x7fada4063230 msgr2=0x7fada4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.460+0000 7fadaa250700 1 -- 172.21.15.175:0/153255022 shutdown_connections 2023-10-10T19:29:26.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.460+0000 7fadaa250700 1 -- 172.21.15.175:0/153255022 wait complete. 2023-10-10T19:29:26.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.461+0000 7fadaa250700 1 Processor -- start 2023-10-10T19:29:26.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.461+0000 7fadaa250700 1 -- start start 2023-10-10T19:29:26.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fadaa250700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada4069620 0x7fada4152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fadaa250700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fada40c6240 0x7fada4151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fadaa250700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada40c6610 0x7fada4151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fadaa250700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada41523e0 con 0x7fada4069620 2023-10-10T19:29:26.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fadaa250700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada41514d0 con 0x7fada40c6240 2023-10-10T19:29:26.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fadaa250700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada41539d0 con 0x7fada40c6610 2023-10-10T19:29:26.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada4069620 0x7fada4152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada4069620 0x7fada4152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60556/0 (socket says 172.21.15.175:60556) 2023-10-10T19:29:26.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 -- 172.21.15.175:0/1594903836 learned_addr learned my addr 172.21.15.175:0/1594903836 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:26.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada2ffd700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fada40c6240 0x7fada4151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada3fff700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada40c6610 0x7fada4151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 -- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada40c6610 msgr2=0x7fada4151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fada40c6610 0x7fada4151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 -- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fada40c6240 msgr2=0x7fada4151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fada40c6240 0x7fada4151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 -- 172.21.15.175:0/1594903836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980094f0 con 0x7fada4069620 2023-10-10T19:29:26.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.462+0000 7fada37fe700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada4069620 0x7fada4152900 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fad980090b0 tx=0x7fad98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.463+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98011e00 con 0x7fada4069620 2023-10-10T19:29:26.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.463+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9801f070 con 0x7fada4069620 2023-10-10T19:29:26.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.463+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada4165dc0 con 0x7fada4069620 2023-10-10T19:29:26.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.463+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980266d0 con 0x7fada4069620 2023-10-10T19:29:26.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.463+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada4150950 con 0x7fada4069620 2023-10-10T19:29:26.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.464+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fada4067570 con 0x7fada4069620 2023-10-10T19:29:26.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.464+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fad9801a430 con 0x7fada4069620 2023-10-10T19:29:26.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.464+0000 7fad8b7fe700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad90066f60 0x7fad90069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.464+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fad98021070 con 0x7fada4069620 2023-10-10T19:29:26.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.464+0000 7fada2ffd700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad90066f60 0x7fad90069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.465+0000 7fad8b7fe700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7fad900705a0 0x7fad90072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.465+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 --> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad900730e0 con 0x7fad900705a0 2023-10-10T19:29:26.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.465+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad9808b240 con 0x7fada4069620 2023-10-10T19:29:26.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.465+0000 7fada3fff700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7fad900705a0 0x7fad90072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.465+0000 7fada2ffd700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad90066f60 0x7fad90069410 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fada41546c0 tx=0x7fad94008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.465+0000 7fada3fff700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7fad900705a0 0x7fad90072a10 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.466+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 <== osd.2 v2:172.21.15.175:6814/90853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fad900730e0 con 0x7fad900705a0 2023-10-10T19:29:26.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 --> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fada4068320 con 0x7fad900705a0 2023-10-10T19:29:26.483 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fad8b7fe700 1 -- 172.21.15.175:0/1594903836 <== osd.2 v2:172.21.15.175:6814/90853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fada4068320 con 0x7fad900705a0 2023-10-10T19:29:26.488 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T19:29:26.489 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-10T19:29:26.489 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T19:29:26.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7fad900705a0 msgr2=0x7fad90072a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.175:6814/90853,v1:172.21.15.175:6817/90853] conn(0x7fad900705a0 0x7fad90072a10 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad90066f60 msgr2=0x7fad90069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fad90066f60 0x7fad90069410 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fada41546c0 tx=0x7fad94008040 comp rx=0 tx=0).stop 2023-10-10T19:29:26.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada4069620 msgr2=0x7fada4152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 --2- 172.21.15.175:0/1594903836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fada4069620 0x7fada4152900 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fad980090b0 tx=0x7fad98017000 comp rx=0 tx=0).stop 2023-10-10T19:29:26.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fada3fff700 1 -- 172.21.15.175:0/1594903836 reap_dead start 2023-10-10T19:29:26.492 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 shutdown_connections 2023-10-10T19:29:26.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 >> 172.21.15.175:0/1594903836 conn(0x7fada4063230 msgr2=0x7fada4153c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 shutdown_connections 2023-10-10T19:29:26.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.479+0000 7fadaa250700 1 -- 172.21.15.175:0/1594903836 wait complete. 2023-10-10T19:29:26.494 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-10T19:29:26.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.609+0000 7f3a2c471700 1 Processor -- start 2023-10-10T19:29:26.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.609+0000 7f3a2c471700 1 -- start start 2023-10-10T19:29:26.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.610+0000 7f3a2c471700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3a241575c0 0x7f3a240c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.610+0000 7f3a2c471700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c41b0 0x7f3a24153df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.610+0000 7f3a2c471700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a240c35f0 con 0x7f3a24157990 2023-10-10T19:29:26.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.610+0000 7f3a2c471700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a2406c0e0 con 0x7f3a241575c0 2023-10-10T19:29:26.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.610+0000 7f3a2c471700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2406c3c0 con 0x7f3a240c41b0 2023-10-10T19:29:26.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.610+0000 7f3a29a0c700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f3a24157990 0x7f3a240c3aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:41994/0 (socket says 172.21.15.175:41994) 2023-10-10T19:29:26.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.610+0000 7f3a29a0c700 1 -- 172.21.15.175:0/3290899683 learned_addr learned my addr 172.21.15.175:0/3290899683 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:26.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2a20d700 1 --2- 172.21.15.175:0/3290899683 >> v2:172.21.15.191:3300/0 conn(0x7f3a241575c0 0x7f3a240c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2aa0e700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c41b0 0x7f3a24153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541883441 0 0) 0x7f3a240c35f0 con 0x7f3a24157990 2023-10-10T19:29:26.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a0c003570 con 0x7f3a24157990 2023-10-10T19:29:26.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2634871107 0 0) 0x7f3a0c003570 con 0x7f3a24157990 2023-10-10T19:29:26.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c41b0 msgr2=0x7f3a24153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:26.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c41b0 0x7f3a24153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 >> v2:172.21.15.191:3300/0 conn(0x7f3a241575c0 msgr2=0x7f3a240c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:26.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 --2- 172.21.15.175:0/3290899683 >> v2:172.21.15.191:3300/0 conn(0x7f3a241575c0 0x7f3a240c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2406d870 con 0x7f3a24157990 2023-10-10T19:29:26.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f3a18001470 con 0x7f3a24157990 2023-10-10T19:29:26.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 >> v1:172.21.15.175:6789/0 conn(0x7f3a24157990 legacy=0x7f3a240c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a0c003a60 0x7f3a0c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c004810 0x7f3a0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c41b0 0x7f3a0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2406c0e0 con 0x7f3a0c003a60 2023-10-10T19:29:26.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2406c3c0 con 0x7f3a0c004810 2023-10-10T19:29:26.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c00ba60 con 0x7f3a240c41b0 2023-10-10T19:29:26.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2a20d700 1 --2- 172.21.15.175:0/3290899683 >> v2:172.21.15.191:3300/0 conn(0x7f3a241575c0 0x7f3a240c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:26.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2a20d700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a0c003a60 0x7f3a0c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2aa0e700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c41b0 0x7f3a0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a29a0c700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c004810 0x7f3a0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2a20d700 1 -- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c41b0 msgr2=0x7f3a0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:26.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2a20d700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c41b0 0x7f3a0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2a20d700 1 -- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c004810 msgr2=0x7f3a0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:26.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2a20d700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c004810 0x7f3a0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.611+0000 7f3a2a20d700 1 -- 172.21.15.175:0/3290899683 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c003570 con 0x7f3a0c003a60 2023-10-10T19:29:26.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2a20d700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a0c003a60 0x7f3a0c003e70 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400e3f0 tx=0x7f3a1400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e917fd9ca7cea9e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1400eca0 con 0x7f3a0c003a60 2023-10-10T19:29:26.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1400f040 con 0x7f3a0c003a60 2023-10-10T19:29:26.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2920b700 1 -- 172.21.15.175:0/3290899683 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a14014480 con 0x7f3a0c003a60 2023-10-10T19:29:26.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2c471700 1 -- 172.21.15.175:0/3290899683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a0c003a60 msgr2=0x7f3a0c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2c471700 1 --2- 172.21.15.175:0/3290899683 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a0c003a60 0x7f3a0c003e70 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400e3f0 tx=0x7f3a1400b040 comp rx=0 tx=0).stop 2023-10-10T19:29:26.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2aa0e700 1 -- 172.21.15.175:0/3290899683 reap_dead start 2023-10-10T19:29:26.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2c471700 1 -- 172.21.15.175:0/3290899683 shutdown_connections 2023-10-10T19:29:26.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2c471700 1 -- 172.21.15.175:0/3290899683 >> 172.21.15.175:0/3290899683 conn(0x7f3a24063cc0 msgr2=0x7f3a2406d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2c471700 1 -- 172.21.15.175:0/3290899683 shutdown_connections 2023-10-10T19:29:26.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.612+0000 7f3a2c471700 1 -- 172.21.15.175:0/3290899683 wait complete. 2023-10-10T19:29:26.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.613+0000 7f3a2c471700 1 Processor -- start 2023-10-10T19:29:26.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.613+0000 7f3a2c471700 1 -- start start 2023-10-10T19:29:26.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2c471700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c003a60 0x7f3a240c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2c471700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c6710 0x7f3a241495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2c471700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a24149b00 0x7f3a24163710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2c471700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240c0800 con 0x7f3a24149b00 2023-10-10T19:29:26.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2c471700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240c0240 con 0x7f3a0c003a60 2023-10-10T19:29:26.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2c471700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240c0520 con 0x7f3a240c6710 2023-10-10T19:29:26.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2aa0e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a24149b00 0x7f3a24163710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2aa0e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a24149b00 0x7f3a24163710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60576/0 (socket says 172.21.15.175:60576) 2023-10-10T19:29:26.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2a20d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c003a60 0x7f3a240c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2aa0e700 1 -- 172.21.15.175:0/4189027386 learned_addr learned my addr 172.21.15.175:0/4189027386 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:26.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a29a0c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c6710 0x7f3a241495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2aa0e700 1 -- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c6710 msgr2=0x7f3a241495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2aa0e700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c6710 0x7f3a241495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2aa0e700 1 -- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c003a60 msgr2=0x7f3a240c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2aa0e700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c003a60 0x7f3a240c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.614+0000 7f3a2aa0e700 1 -- 172.21.15.175:0/4189027386 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a240c92c0 con 0x7f3a24149b00 2023-10-10T19:29:26.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.615+0000 7f3a2aa0e700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a24149b00 0x7f3a24163710 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c006f00 tx=0x7f3a1c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.615+0000 7f3a227fc700 1 -- 172.21.15.175:0/4189027386 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c013680 con 0x7f3a24149b00 2023-10-10T19:29:26.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.615+0000 7f3a227fc700 1 -- 172.21.15.175:0/4189027386 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c013800 con 0x7f3a24149b00 2023-10-10T19:29:26.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.615+0000 7f3a227fc700 1 -- 172.21.15.175:0/4189027386 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c013bc0 con 0x7f3a24149b00 2023-10-10T19:29:26.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.615+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a240c95a0 con 0x7f3a24149b00 2023-10-10T19:29:26.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.615+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a240ca2e0 con 0x7f3a24149b00 2023-10-10T19:29:26.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.616+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2406b1b0 con 0x7f3a24149b00 2023-10-10T19:29:26.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.616+0000 7f3a227fc700 1 -- 172.21.15.175:0/4189027386 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3a1c014440 con 0x7f3a24149b00 2023-10-10T19:29:26.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.616+0000 7f3a227fc700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.616+0000 7f3a227fc700 1 -- 172.21.15.175:0/4189027386 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f3a1c0851e0 con 0x7f3a24149b00 2023-10-10T19:29:26.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.617+0000 7f3a2a20d700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.617+0000 7f3a2a20d700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400e3f0 tx=0x7f3a14019000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.621+0000 7f3a227fc700 1 -- 172.21.15.175:0/4189027386 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a1c051c50 con 0x7f3a24149b00 2023-10-10T19:29:26.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.803+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a240c9140 con 0x7f3a10066f30 2023-10-10T19:29:26.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.804+0000 7f3a227fc700 1 -- 172.21.15.175:0/4189027386 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3a1400ed50 con 0x7f3a10066f30 2023-10-10T19:29:26.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 msgr2=0x7f3a100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400e3f0 tx=0x7f3a14019000 comp rx=0 tx=0).stop 2023-10-10T19:29:26.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a24149b00 msgr2=0x7f3a24163710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a24149b00 0x7f3a24163710 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c006f00 tx=0x7f3a1c0091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 shutdown_connections 2023-10-10T19:29:26.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f3a10066f30 0x7f3a100693e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a24149b00 0x7f3a24163710 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a0c003a60 0x7f3a240c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 --2- 172.21.15.175:0/4189027386 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a240c6710 0x7f3a241495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 >> 172.21.15.175:0/4189027386 conn(0x7f3a24063cc0 msgr2=0x7f3a240c3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 shutdown_connections 2023-10-10T19:29:26.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.806+0000 7f3a2c471700 1 -- 172.21.15.175:0/4189027386 wait complete. 2023-10-10T19:29:26.810 INFO:teuthology.orchestra.run.smithi175.stderr:instructed osd(s) 2 to deep-scrub 2023-10-10T19:29:26.818 INFO:tasks.ceph:Scrubbing osd.3 2023-10-10T19:29:26.818 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-10T19:29:26.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.936+0000 7fbdf67de700 1 Processor -- start 2023-10-10T19:29:26.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.936+0000 7fbdf67de700 1 -- start start 2023-10-10T19:29:26.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf67de700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 0x7fbdf00690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf67de700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbdf0069620 0x7fbdf00c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf67de700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbdf00c6610 0x7fbdf00c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf67de700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf00c4b60 con 0x7fbdf00c6610 2023-10-10T19:29:26.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf67de700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf0068c50 con 0x7fbdf0069620 2023-10-10T19:29:26.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf67de700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0157420 con 0x7fbdf00c6240 2023-10-10T19:29:26.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf4d7b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbdf00c6610 0x7fbdf00c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf4d7b700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbdf00c6610 0x7fbdf00c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60586/0 (socket says 172.21.15.175:60586) 2023-10-10T19:29:26.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdf4d7b700 1 -- 172.21.15.175:0/3217033649 learned_addr learned my addr 172.21.15.175:0/3217033649 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:26.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbde7fff700 1 --2- 172.21.15.175:0/3217033649 >> v2:172.21.15.191:3300/0 conn(0x7fbdf0069620 0x7fbdf00c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.937+0000 7fbdeffff700 1 --2- 172.21.15.175:0/3217033649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 0x7fbdf00690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdf4d7b700 1 -- 172.21.15.175:0/3217033649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 msgr2=0x7fbdf00690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:26.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdf4d7b700 1 --2- 172.21.15.175:0/3217033649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 0x7fbdf00690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdf4d7b700 1 -- 172.21.15.175:0/3217033649 >> v2:172.21.15.191:3300/0 conn(0x7fbdf0069620 msgr2=0x7fbdf00c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:26.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdf4d7b700 1 --2- 172.21.15.175:0/3217033649 >> v2:172.21.15.191:3300/0 conn(0x7fbdf0069620 0x7fbdf00c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdf4d7b700 1 -- 172.21.15.175:0/3217033649 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf00c4880 con 0x7fbdf00c6610 2023-10-10T19:29:26.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbde7fff700 1 --2- 172.21.15.175:0/3217033649 >> v2:172.21.15.191:3300/0 conn(0x7fbdf0069620 0x7fbdf00c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:26.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdf4d7b700 1 --2- 172.21.15.175:0/3217033649 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf00c6610 0x7fbdf00c4030 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0089e60 tx=0x7fbde8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ba3a6c0bbcda4d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdef7fe700 1 -- 172.21.15.175:0/3217033649 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde800e070 con 0x7fbdf00c6610 2023-10-10T19:29:26.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdef7fe700 1 -- 172.21.15.175:0/3217033649 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde800ba50 con 0x7fbdf00c6610 2023-10-10T19:29:26.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.938+0000 7fbdef7fe700 1 -- 172.21.15.175:0/3217033649 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80092e0 con 0x7fbdf00c6610 2023-10-10T19:29:26.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 -- 172.21.15.175:0/3217033649 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf00c6610 msgr2=0x7fbdf00c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 --2- 172.21.15.175:0/3217033649 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf00c6610 0x7fbdf00c4030 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0089e60 tx=0x7fbde8009840 comp rx=0 tx=0).stop 2023-10-10T19:29:26.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 -- 172.21.15.175:0/3217033649 shutdown_connections 2023-10-10T19:29:26.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 --2- 172.21.15.175:0/3217033649 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf00c6610 0x7fbdf00c4030 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 --2- 172.21.15.175:0/3217033649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 0x7fbdf00690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 --2- 172.21.15.175:0/3217033649 >> v2:172.21.15.191:3300/0 conn(0x7fbdf0069620 0x7fbdf00c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 -- 172.21.15.175:0/3217033649 >> 172.21.15.175:0/3217033649 conn(0x7fbdf0063230 msgr2=0x7fbdf0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 -- 172.21.15.175:0/3217033649 shutdown_connections 2023-10-10T19:29:26.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.939+0000 7fbdf67de700 1 -- 172.21.15.175:0/3217033649 wait complete. 2023-10-10T19:29:26.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.940+0000 7fbdf67de700 1 Processor -- start 2023-10-10T19:29:26.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.940+0000 7fbdf67de700 1 -- start start 2023-10-10T19:29:26.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.940+0000 7fbdf67de700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf0069620 0x7fbdf00ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.940+0000 7fbdf67de700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 0x7fbdf00cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.940+0000 7fbdf67de700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c6610 0x7fbdf00c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.940+0000 7fbdf67de700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00c0170 con 0x7fbdf0069620 2023-10-10T19:29:26.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.940+0000 7fbdf67de700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00c03c0 con 0x7fbdf00c6610 2023-10-10T19:29:26.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.940+0000 7fbdf67de700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00c06a0 con 0x7fbdf00c6240 2023-10-10T19:29:26.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf0069620 0x7fbdf00ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf0069620 0x7fbdf00ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60598/0 (socket says 172.21.15.175:60598) 2023-10-10T19:29:26.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 -- 172.21.15.175:0/615112173 learned_addr learned my addr 172.21.15.175:0/615112173 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:26.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdf4d7b700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c6610 0x7fbdf00c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbde7fff700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 0x7fbdf00cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 -- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 msgr2=0x7fbdf00cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c6240 0x7fbdf00cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 -- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c6610 msgr2=0x7fbdf00c1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c6610 0x7fbdf00c1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 -- 172.21.15.175:0/615112173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8009ae0 con 0x7fbdf0069620 2023-10-10T19:29:26.959 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdeffff700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf0069620 0x7fbdf00ca8d0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fbde000d890 tx=0x7fbde0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde000b600 con 0x7fbdf0069620 2023-10-10T19:29:26.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde000b760 con 0x7fbdf0069620 2023-10-10T19:29:26.960 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf00cd120 con 0x7fbdf0069620 2023-10-10T19:29:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde000bae0 con 0x7fbdf0069620 2023-10-10T19:29:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.941+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf00c0f70 con 0x7fbdf0069620 2023-10-10T19:29:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.942+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbdf0067570 con 0x7fbdf0069620 2023-10-10T19:29:26.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.942+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbde001e460 con 0x7fbdf0069620 2023-10-10T19:29:26.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbdecff9700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdd0066f60 0x7fbdd0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fbde0085ec0 con 0x7fbdf0069620 2023-10-10T19:29:26.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbde7fff700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdd0066f60 0x7fbdd0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.964 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbdecff9700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7fbdd00705a0 0x7fbdd0072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:26.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 --> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdd00730e0 con 0x7fbdd00705a0 2023-10-10T19:29:26.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbde0086290 con 0x7fbdf0069620 2023-10-10T19:29:26.965 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbdf4d7b700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7fbdd00705a0 0x7fbdd0072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:26.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbde7fff700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdd0066f60 0x7fbdd0069410 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8006f00 tx=0x7fbdd8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.943+0000 7fbdf4d7b700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7fbdd00705a0 0x7fbdd0072a10 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:26.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.944+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 <== osd.3 v2:172.21.15.175:6824/90874 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fbdd00730e0 con 0x7fbdd00705a0 2023-10-10T19:29:26.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 --> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbdf0068320 con 0x7fbdd00705a0 2023-10-10T19:29:26.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdecff9700 1 -- 172.21.15.175:0/615112173 <== osd.3 v2:172.21.15.175:6824/90874 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fbdf0068320 con 0x7fbdd00705a0 2023-10-10T19:29:26.967 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T19:29:26.968 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-10T19:29:26.968 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T19:29:26.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7fbdd00705a0 msgr2=0x7fbdd0072a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.175:6824/90874,v1:172.21.15.175:6825/90874] conn(0x7fbdd00705a0 0x7fbdd0072a10 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:26.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdd0066f60 msgr2=0x7fbdd0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbdd0066f60 0x7fbdd0069410 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8006f00 tx=0x7fbdd8008040 comp rx=0 tx=0).stop 2023-10-10T19:29:26.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf0069620 msgr2=0x7fbdf00ca8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:26.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 --2- 172.21.15.175:0/615112173 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbdf0069620 0x7fbdf00ca8d0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fbde000d890 tx=0x7fbde0008040 comp rx=0 tx=0).stop 2023-10-10T19:29:26.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf4d7b700 1 -- 172.21.15.175:0/615112173 reap_dead start 2023-10-10T19:29:26.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 shutdown_connections 2023-10-10T19:29:26.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 >> 172.21.15.175:0/615112173 conn(0x7fbdf0063230 msgr2=0x7fbdf00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:26.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 shutdown_connections 2023-10-10T19:29:26.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:26.958+0000 7fbdf67de700 1 -- 172.21.15.175:0/615112173 wait complete. 2023-10-10T19:29:26.974 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-10T19:29:27.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.089+0000 7f622b59e700 1 Processor -- start 2023-10-10T19:29:27.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.089+0000 7f622b59e700 1 -- start start 2023-10-10T19:29:27.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622b59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c158af0 0x7f622c155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622b59e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f622c0c6f50 0x7f622c159310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622b59e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c157210 con 0x7f622c158ec0 2023-10-10T19:29:27.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622b59e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622c15a260 con 0x7f622c0c6f50 2023-10-10T19:29:27.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622b59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c150590 con 0x7f622c158af0 2023-10-10T19:29:27.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622a59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c158af0 0x7f622c155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622ad9d700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f622c0c6f50 0x7f622c159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622a59c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c158af0 0x7f622c155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:40154/0 (socket says 172.21.15.175:40154) 2023-10-10T19:29:27.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f622a59c700 1 -- 172.21.15.175:0/4111369279 learned_addr learned my addr 172.21.15.175:0/4111369279 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:27.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.090+0000 7f6229d9b700 1 --1- 172.21.15.175:0/4111369279 >> v1:172.21.15.191:6789/0 conn(0x7f622c158ec0 0x7f622c0afd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52932/0 (socket says 172.21.15.175:52932) 2023-10-10T19:29:27.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622a59c700 1 -- 172.21.15.175:0/4111369279 >> v2:172.21.15.191:3300/0 conn(0x7f622c0c6f50 msgr2=0x7f622c159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:27.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622a59c700 1 --2- 172.21.15.175:0/4111369279 >> v2:172.21.15.191:3300/0 conn(0x7f622c0c6f50 0x7f622c159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622a59c700 1 -- 172.21.15.175:0/4111369279 >> v1:172.21.15.191:6789/0 conn(0x7f622c158ec0 legacy=0x7f622c0afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622a59c700 1 -- 172.21.15.175:0/4111369279 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c0af8b0 con 0x7f622c158af0 2023-10-10T19:29:27.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622ad9d700 1 --2- 172.21.15.175:0/4111369279 >> v2:172.21.15.191:3300/0 conn(0x7f622c0c6f50 0x7f622c159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:27.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622a59c700 1 --2- 172.21.15.175:0/4111369279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c158af0 0x7f622c155d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f621c0099b0 tx=0x7f621c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42ade1d6c0d5567f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622959a700 1 -- 172.21.15.175:0/4111369279 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c009ea0 con 0x7f622c158af0 2023-10-10T19:29:27.098 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622959a700 1 -- 172.21.15.175:0/4111369279 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c00b860 con 0x7f622c158af0 2023-10-10T19:29:27.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622959a700 1 -- 172.21.15.175:0/4111369279 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c00bc00 con 0x7f622c158af0 2023-10-10T19:29:27.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622b59e700 1 -- 172.21.15.175:0/4111369279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c158af0 msgr2=0x7f622c155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.091+0000 7f622b59e700 1 --2- 172.21.15.175:0/4111369279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c158af0 0x7f622c155d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f621c0099b0 tx=0x7f621c009250 comp rx=0 tx=0).stop 2023-10-10T19:29:27.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.092+0000 7f622b59e700 1 -- 172.21.15.175:0/4111369279 shutdown_connections 2023-10-10T19:29:27.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.092+0000 7f622b59e700 1 --2- 172.21.15.175:0/4111369279 >> v2:172.21.15.191:3300/0 conn(0x7f622c0c6f50 0x7f622c159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.092+0000 7f622b59e700 1 --2- 172.21.15.175:0/4111369279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c158af0 0x7f622c155d90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.092+0000 7f622b59e700 1 -- 172.21.15.175:0/4111369279 >> 172.21.15.175:0/4111369279 conn(0x7f622c063230 msgr2=0x7f622c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:27.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.092+0000 7f622b59e700 1 -- 172.21.15.175:0/4111369279 shutdown_connections 2023-10-10T19:29:27.101 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.092+0000 7f622b59e700 1 -- 172.21.15.175:0/4111369279 wait complete. 2023-10-10T19:29:27.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622b59e700 1 Processor -- start 2023-10-10T19:29:27.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622b59e700 1 -- start start 2023-10-10T19:29:27.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622b59e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f622c0c6f50 0x7f622c167130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622b59e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c158af0 0x7f622c1693f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622b59e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c167670 0x7f622c169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622b59e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0c6320 con 0x7f622c0c6f50 2023-10-10T19:29:27.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622b59e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0c5e90 con 0x7f622c158af0 2023-10-10T19:29:27.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622b59e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c1519b0 con 0x7f622c167670 2023-10-10T19:29:27.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f622c0c6f50 0x7f622c167130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622a59c700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f622c0c6f50 0x7f622c167130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60604/0 (socket says 172.21.15.175:60604) 2023-10-10T19:29:27.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f6229d9b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c158af0 0x7f622c1693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622a59c700 1 -- 172.21.15.175:0/3552566292 learned_addr learned my addr 172.21.15.175:0/3552566292 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:27.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.093+0000 7f622ad9d700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c167670 0x7f622c169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f622a59c700 1 -- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c167670 msgr2=0x7f622c169930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f622a59c700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c167670 0x7f622c169930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.108 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f622a59c700 1 -- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c158af0 msgr2=0x7f622c1693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f622a59c700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c158af0 0x7f622c1693f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f622a59c700 1 -- 172.21.15.175:0/3552566292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6224009a20 con 0x7f622c0c6f50 2023-10-10T19:29:27.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f6229d9b700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c158af0 0x7f622c1693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:27.109 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f622a59c700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f622c0c6f50 0x7f622c167130 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f621c006fd0 tx=0x7f621c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f621a7fc700 1 -- 172.21.15.175:0/3552566292 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c00b9d0 con 0x7f622c0c6f50 2023-10-10T19:29:27.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f621a7fc700 1 -- 172.21.15.175:0/3552566292 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c00bb30 con 0x7f622c0c6f50 2023-10-10T19:29:27.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f621a7fc700 1 -- 172.21.15.175:0/3552566292 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c025630 con 0x7f622c0c6f50 2023-10-10T19:29:27.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621c0094f0 con 0x7f622c0c6f50 2023-10-10T19:29:27.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.094+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c0c55e0 con 0x7f622c0c6f50 2023-10-10T19:29:27.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.095+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c157cf0 con 0x7f622c0c6f50 2023-10-10T19:29:27.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.095+0000 7f621a7fc700 1 -- 172.21.15.175:0/3552566292 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f621c023070 con 0x7f622c0c6f50 2023-10-10T19:29:27.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.096+0000 7f621a7fc700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f620c066f60 0x7f620c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.096+0000 7f621a7fc700 1 -- 172.21.15.175:0/3552566292 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f621c086100 con 0x7f622c0c6f50 2023-10-10T19:29:27.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.096+0000 7f6229d9b700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f620c066f60 0x7f620c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.096+0000 7f6229d9b700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f620c066f60 0x7f620c069410 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f622c163860 tx=0x7f6220007eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.100+0000 7f621a7fc700 1 -- 172.21.15.175:0/3552566292 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f621c052c60 con 0x7f622c0c6f50 2023-10-10T19:29:27.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.285+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f622c04bb80 con 0x7f620c066f60 2023-10-10T19:29:27.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.285+0000 7f621a7fc700 1 -- 172.21.15.175:0/3552566292 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f622c04bb80 con 0x7f620c066f60 2023-10-10T19:29:27.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f620c066f60 msgr2=0x7f620c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f620c066f60 0x7f620c069410 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f622c163860 tx=0x7f6220007eb0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f622c0c6f50 msgr2=0x7f622c167130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f622c0c6f50 0x7f622c167130 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f621c006fd0 tx=0x7f621c017000 comp rx=0 tx=0).stop 2023-10-10T19:29:27.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 shutdown_connections 2023-10-10T19:29:27.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f620c066f60 0x7f620c069410 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f622c167670 0x7f622c169930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f622c0c6f50 0x7f622c167130 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 --2- 172.21.15.175:0/3552566292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c158af0 0x7f622c1693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.287+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 >> 172.21.15.175:0/3552566292 conn(0x7f622c063230 msgr2=0x7f622c0b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:27.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.288+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 shutdown_connections 2023-10-10T19:29:27.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.288+0000 7f622b59e700 1 -- 172.21.15.175:0/3552566292 wait complete. 2023-10-10T19:29:27.292 INFO:teuthology.orchestra.run.smithi175.stderr:instructed osd(s) 3 to deep-scrub 2023-10-10T19:29:27.300 INFO:tasks.ceph:Scrubbing osd.4 2023-10-10T19:29:27.300 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-10T19:29:27.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.428+0000 7f1f9746b700 1 Processor -- start 2023-10-10T19:29:27.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.428+0000 7f1f9746b700 1 -- start start 2023-10-10T19:29:27.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f9746b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1f900c6240 0x7f1f900690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f9746b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f90069620 0x7f1f900c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f9746b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900c4b60 con 0x7f1f900c6610 2023-10-10T19:29:27.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f9746b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f90068c50 con 0x7f1f900c6240 2023-10-10T19:29:27.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f9746b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f90157420 con 0x7f1f90069620 2023-10-10T19:29:27.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f94a06700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f90069620 0x7f1f900c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f95207700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1f900c6240 0x7f1f900690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f95a08700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1f900c6610 0x7f1f900c4030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.175:52936/0 (socket says 172.21.15.175:52936) 2023-10-10T19:29:27.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.429+0000 7f1f95a08700 1 -- 172.21.15.175:0/2609421102 learned_addr learned my addr 172.21.15.175:0/2609421102 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:27.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.430+0000 7f1f94a06700 1 -- 172.21.15.175:0/2609421102 >> v2:172.21.15.191:3300/0 conn(0x7f1f900c6240 msgr2=0x7f1f900690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:27.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.430+0000 7f1f94a06700 1 --2- 172.21.15.175:0/2609421102 >> v2:172.21.15.191:3300/0 conn(0x7f1f900c6240 0x7f1f900690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.430+0000 7f1f94a06700 1 -- 172.21.15.175:0/2609421102 >> v1:172.21.15.191:6789/0 conn(0x7f1f900c6610 legacy=0x7f1f900c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.430+0000 7f1f94a06700 1 -- 172.21.15.175:0/2609421102 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f900c4880 con 0x7f1f90069620 2023-10-10T19:29:27.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.430+0000 7f1f94a06700 1 --2- 172.21.15.175:0/2609421102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f90069620 0x7f1f900c3af0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f840098e0 tx=0x7f1f840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d6a0902034362df server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.430+0000 7f1f8ffff700 1 -- 172.21.15.175:0/2609421102 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400b640 con 0x7f1f90069620 2023-10-10T19:29:27.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.430+0000 7f1f8ffff700 1 -- 172.21.15.175:0/2609421102 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8400b7a0 con 0x7f1f90069620 2023-10-10T19:29:27.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.430+0000 7f1f8ffff700 1 -- 172.21.15.175:0/2609421102 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400bb40 con 0x7f1f90069620 2023-10-10T19:29:27.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.431+0000 7f1f9746b700 1 -- 172.21.15.175:0/2609421102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f90069620 msgr2=0x7f1f900c3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.431+0000 7f1f9746b700 1 --2- 172.21.15.175:0/2609421102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f90069620 0x7f1f900c3af0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f840098e0 tx=0x7f1f840091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.431+0000 7f1f9746b700 1 -- 172.21.15.175:0/2609421102 shutdown_connections 2023-10-10T19:29:27.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.431+0000 7f1f9746b700 1 --2- 172.21.15.175:0/2609421102 >> v2:172.21.15.191:3300/0 conn(0x7f1f900c6240 0x7f1f900690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.431+0000 7f1f9746b700 1 --2- 172.21.15.175:0/2609421102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f90069620 0x7f1f900c3af0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.431+0000 7f1f9746b700 1 -- 172.21.15.175:0/2609421102 >> 172.21.15.175:0/2609421102 conn(0x7f1f90063230 msgr2=0x7f1f90063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:27.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.431+0000 7f1f9746b700 1 -- 172.21.15.175:0/2609421102 shutdown_connections 2023-10-10T19:29:27.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.431+0000 7f1f9746b700 1 -- 172.21.15.175:0/2609421102 wait complete. 2023-10-10T19:29:27.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.432+0000 7f1f9746b700 1 Processor -- start 2023-10-10T19:29:27.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.432+0000 7f1f9746b700 1 -- start start 2023-10-10T19:29:27.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f9746b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f90069620 0x7f1f90166d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f9746b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f900c6240 0x7f1f900b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f9746b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f900c6610 0x7f1f900cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f9746b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f90167240 con 0x7f1f90069620 2023-10-10T19:29:27.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f9746b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900ca9b0 con 0x7f1f900c6610 2023-10-10T19:29:27.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f9746b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900cd620 con 0x7f1f900c6240 2023-10-10T19:29:27.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f90069620 0x7f1f90166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f90069620 0x7f1f90166d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60620/0 (socket says 172.21.15.175:60620) 2023-10-10T19:29:27.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 -- 172.21.15.175:0/1245330523 learned_addr learned my addr 172.21.15.175:0/1245330523 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:27.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f94a06700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f900c6240 0x7f1f900b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95a08700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f900c6610 0x7f1f900cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 -- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f900c6240 msgr2=0x7f1f900b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f900c6240 0x7f1f900b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 -- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f900c6610 msgr2=0x7f1f900cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f900c6610 0x7f1f900cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 -- 172.21.15.175:0/1245330523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84009420 con 0x7f1f90069620 2023-10-10T19:29:27.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f95207700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f90069620 0x7f1f90166d00 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009ef0 tx=0x7f1f8000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8000cb70 con 0x7f1f90069620 2023-10-10T19:29:27.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8000f040 con 0x7f1f90069620 2023-10-10T19:29:27.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f900ccd80 con 0x7f1f90069620 2023-10-10T19:29:27.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80014480 con 0x7f1f90069620 2023-10-10T19:29:27.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.433+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f900cd340 con 0x7f1f90069620 2023-10-10T19:29:27.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.434+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1f90067570 con 0x7f1f90069620 2023-10-10T19:29:27.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.435+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1f800147f0 con 0x7f1f90069620 2023-10-10T19:29:27.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.435+0000 7f1f8d7fa700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f7c066f30 0x7f1f7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.435+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f1f800856e0 con 0x7f1f90069620 2023-10-10T19:29:27.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.435+0000 7f1f8d7fa700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7f1f7c070570 0x7f1f7c0729e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.435+0000 7f1f94a06700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f7c066f30 0x7f1f7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.435+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 --> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f7c0730b0 con 0x7f1f7c070570 2023-10-10T19:29:27.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.435+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f80087d20 con 0x7f1f90069620 2023-10-10T19:29:27.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.435+0000 7f1f95a08700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7f1f7c070570 0x7f1f7c0729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.436+0000 7f1f94a06700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f7c066f30 0x7f1f7c0693e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1f840093f0 tx=0x7f1f84017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.436+0000 7f1f95a08700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7f1f7c070570 0x7f1f7c0729e0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.436+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 <== osd.4 v2:172.21.15.191:6800/91481 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f1f7c0730b0 con 0x7f1f7c070570 2023-10-10T19:29:27.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.450+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 --> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1f90068320 con 0x7f1f7c070570 2023-10-10T19:29:27.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.450+0000 7f1f8d7fa700 1 -- 172.21.15.175:0/1245330523 <== osd.4 v2:172.21.15.191:6800/91481 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1f90068320 con 0x7f1f7c070570 2023-10-10T19:29:27.455 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T19:29:27.455 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-10T19:29:27.455 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T19:29:27.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7f1f7c070570 msgr2=0x7f1f7c0729e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6800/91481,v1:172.21.15.191:6801/91481] conn(0x7f1f7c070570 0x7f1f7c0729e0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f7c066f30 msgr2=0x7f1f7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1f7c066f30 0x7f1f7c0693e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1f840093f0 tx=0x7f1f84017000 comp rx=0 tx=0).stop 2023-10-10T19:29:27.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f90069620 msgr2=0x7f1f90166d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 --2- 172.21.15.175:0/1245330523 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f90069620 0x7f1f90166d00 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009ef0 tx=0x7f1f8000b040 comp rx=0 tx=0).stop 2023-10-10T19:29:27.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f95a08700 1 -- 172.21.15.175:0/1245330523 reap_dead start 2023-10-10T19:29:27.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 shutdown_connections 2023-10-10T19:29:27.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 >> 172.21.15.175:0/1245330523 conn(0x7f1f90063230 msgr2=0x7f1f90158570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:27.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 shutdown_connections 2023-10-10T19:29:27.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.451+0000 7f1f9746b700 1 -- 172.21.15.175:0/1245330523 wait complete. 2023-10-10T19:29:27.463 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-10T19:29:27.581 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.581+0000 7fed1c09f700 1 Processor -- start 2023-10-10T19:29:27.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.581+0000 7fed1c09f700 1 -- start start 2023-10-10T19:29:27.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1c09f700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fed14157490 0x7fed140c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1c09f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed140c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1c09f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed14063e70 con 0x7fed14157d50 2023-10-10T19:29:27.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1c09f700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed14063940 con 0x7fed14157490 2023-10-10T19:29:27.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1c09f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1406bda0 con 0x7fed14157860 2023-10-10T19:29:27.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1a63c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed140c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed19e3b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fed14157490 0x7fed140c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1a63c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed140c0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:40178/0 (socket says 172.21.15.175:40178) 2023-10-10T19:29:27.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1a63c700 1 -- 172.21.15.175:0/2550867308 learned_addr learned my addr 172.21.15.175:0/2550867308 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:27.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.582+0000 7fed1a63c700 1 -- 172.21.15.175:0/2550867308 >> v2:172.21.15.191:3300/0 conn(0x7fed14157490 msgr2=0x7fed140c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:27.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1a63c700 1 --2- 172.21.15.175:0/2550867308 >> v2:172.21.15.191:3300/0 conn(0x7fed14157490 0x7fed140c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1a63c700 1 -- 172.21.15.175:0/2550867308 >> v1:172.21.15.191:6789/0 conn(0x7fed14157d50 legacy=0x7fed1406c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1a63c700 1 -- 172.21.15.175:0/2550867308 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed140bf500 con 0x7fed14157860 2023-10-10T19:29:27.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1a63c700 1 --2- 172.21.15.175:0/2550867308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed140c0f60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009f20 tx=0x7fed0c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=152154a5a9f37faa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed18e39700 1 -- 172.21.15.175:0/2550867308 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00e070 con 0x7fed14157860 2023-10-10T19:29:27.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed18e39700 1 -- 172.21.15.175:0/2550867308 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c00b8d0 con 0x7fed14157860 2023-10-10T19:29:27.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed18e39700 1 -- 172.21.15.175:0/2550867308 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0092e0 con 0x7fed14157860 2023-10-10T19:29:27.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1c09f700 1 -- 172.21.15.175:0/2550867308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 msgr2=0x7fed140c0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1c09f700 1 --2- 172.21.15.175:0/2550867308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed140c0f60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009f20 tx=0x7fed0c009840 comp rx=0 tx=0).stop 2023-10-10T19:29:27.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1c09f700 1 -- 172.21.15.175:0/2550867308 shutdown_connections 2023-10-10T19:29:27.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1c09f700 1 --2- 172.21.15.175:0/2550867308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed140c0f60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1c09f700 1 --2- 172.21.15.175:0/2550867308 >> v2:172.21.15.191:3300/0 conn(0x7fed14157490 0x7fed140c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.591 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1c09f700 1 -- 172.21.15.175:0/2550867308 >> 172.21.15.175:0/2550867308 conn(0x7fed1406dc60 msgr2=0x7fed14063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:27.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.583+0000 7fed1c09f700 1 -- 172.21.15.175:0/2550867308 shutdown_connections 2023-10-10T19:29:27.592 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.584+0000 7fed1c09f700 1 -- 172.21.15.175:0/2550867308 wait complete. 2023-10-10T19:29:27.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1c09f700 1 Processor -- start 2023-10-10T19:29:27.593 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1c09f700 1 -- start start 2023-10-10T19:29:27.596 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1c09f700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed14157490 0x7fed140c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1c09f700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed14069240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1c09f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed14157d50 0x7fed14069780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.597 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1c09f700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1406a9d0 con 0x7fed14157490 2023-10-10T19:29:27.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1c09f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed140c61b0 con 0x7fed14157d50 2023-10-10T19:29:27.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1c09f700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed14169410 con 0x7fed14157860 2023-10-10T19:29:27.598 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed19e3b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed14157490 0x7fed140c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed19e3b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed14157490 0x7fed140c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60622/0 (socket says 172.21.15.175:60622) 2023-10-10T19:29:27.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1963a700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed14069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed19e3b700 1 -- 172.21.15.175:0/1871104863 learned_addr learned my addr 172.21.15.175:0/1871104863 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:27.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.585+0000 7fed1a63c700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed14157d50 0x7fed14069780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed19e3b700 1 -- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 msgr2=0x7fed14069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed19e3b700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed14069240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed19e3b700 1 -- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed14157d50 msgr2=0x7fed14069780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed19e3b700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed14157d50 0x7fed14069780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed19e3b700 1 -- 172.21.15.175:0/1871104863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c009ae0 con 0x7fed14157490 2023-10-10T19:29:27.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed1963a700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed14069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:27.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed19e3b700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed14157490 0x7fed140c7280 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fed1000ecb0 tx=0x7fed100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.602 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed09ffb700 1 -- 172.21.15.175:0/1871104863 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000c850 con 0x7fed14157490 2023-10-10T19:29:27.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed09ffb700 1 -- 172.21.15.175:0/1871104863 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1000c9b0 con 0x7fed14157490 2023-10-10T19:29:27.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed14068900 con 0x7fed14157490 2023-10-10T19:29:27.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed09ffb700 1 -- 172.21.15.175:0/1871104863 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000cd70 con 0x7fed14157490 2023-10-10T19:29:27.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.586+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed140c5870 con 0x7fed14157490 2023-10-10T19:29:27.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.587+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed14169970 con 0x7fed14157490 2023-10-10T19:29:27.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.588+0000 7fed09ffb700 1 -- 172.21.15.175:0/1871104863 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fed10013680 con 0x7fed14157490 2023-10-10T19:29:27.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.588+0000 7fed09ffb700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fecf8066f30 0x7fecf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.588+0000 7fed09ffb700 1 -- 172.21.15.175:0/1871104863 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fed10085540 con 0x7fed14157490 2023-10-10T19:29:27.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.588+0000 7fed1963a700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fecf8066f30 0x7fecf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.589+0000 7fed1963a700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fecf8066f30 0x7fecf80693e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fed1414e3e0 tx=0x7fed04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.592+0000 7fed09ffb700 1 -- 172.21.15.175:0/1871104863 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed10051fb0 con 0x7fed14157490 2023-10-10T19:29:27.667 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:29:27.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:29:27.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.781+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fed140c55a0 con 0x7fecf8066f30 2023-10-10T19:29:27.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.782+0000 7fed09ffb700 1 -- 172.21.15.175:0/1871104863 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fed0400fc60 con 0x7fecf8066f30 2023-10-10T19:29:27.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fecf8066f30 msgr2=0x7fecf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fecf8066f30 0x7fecf80693e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fed1414e3e0 tx=0x7fed04009040 comp rx=0 tx=0).stop 2023-10-10T19:29:27.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed14157490 msgr2=0x7fed140c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed14157490 0x7fed140c7280 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fed1000ecb0 tx=0x7fed100091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 shutdown_connections 2023-10-10T19:29:27.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fecf8066f30 0x7fecf80693e0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed14157d50 0x7fed14069780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fed14157490 0x7fed140c7280 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 --2- 172.21.15.175:0/1871104863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fed14157860 0x7fed14069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 >> 172.21.15.175:0/1871104863 conn(0x7fed1406dc60 msgr2=0x7fed141587f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:27.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 shutdown_connections 2023-10-10T19:29:27.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.784+0000 7fed1c09f700 1 -- 172.21.15.175:0/1871104863 wait complete. 2023-10-10T19:29:27.789 INFO:teuthology.orchestra.run.smithi175.stderr:instructed osd(s) 4 to deep-scrub 2023-10-10T19:29:27.796 INFO:tasks.ceph:Scrubbing osd.5 2023-10-10T19:29:27.797 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-10T19:29:27.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.915+0000 7fe40e18e700 1 Processor -- start 2023-10-10T19:29:27.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.915+0000 7fe40e18e700 1 -- start start 2023-10-10T19:29:27.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe40e18e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080c6240 0x7fe4080690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe40e18e700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4080c4b60 con 0x7fe408069620 2023-10-10T19:29:27.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe40e18e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe408068c50 con 0x7fe4080c6610 2023-10-10T19:29:27.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe40e18e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe408157420 con 0x7fe4080c6240 2023-10-10T19:29:27.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe406ffd700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe408069620 0x7fe4080c3af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42010/0 (socket says 172.21.15.175:42010) 2023-10-10T19:29:27.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe406ffd700 1 -- 172.21.15.175:0/1874382131 learned_addr learned my addr 172.21.15.175:0/1874382131 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:27.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe4077fe700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080c6240 0x7fe4080690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221471584 0 0) 0x7fe3fc001110 con 0x7fe408069620 2023-10-10T19:29:27.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.916+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f0003570 con 0x7fe408069620 2023-10-10T19:29:27.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569621931 0 0) 0x7fe408068c50 con 0x7fe4080c6610 2023-10-10T19:29:27.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3fc001110 con 0x7fe4080c6610 2023-10-10T19:29:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2715397846 0 0) 0x7fe3f0003570 con 0x7fe408069620 2023-10-10T19:29:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080c6240 msgr2=0x7fe4080690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080c6240 0x7fe4080690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 >> v1:172.21.15.191:6789/0 conn(0x7fe4080c6610 legacy=0x7fe408159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080c4880 con 0x7fe408069620 2023-10-10T19:29:27.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7fe3fc001600 con 0x7fe408069620 2023-10-10T19:29:27.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 >> v1:172.21.15.175:6789/0 conn(0x7fe408069620 legacy=0x7fe4080c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f0003c20 0x7fe3f0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080c6240 0x7fe3f0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0004d50 0x7fe3f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408157420 con 0x7fe3f0003c20 2023-10-10T19:29:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f000bb40 con 0x7fe3f0004d50 2023-10-10T19:29:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f000bd00 con 0x7fe4080c6240 2023-10-10T19:29:27.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4077fe700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f0003c20 0x7fe3f0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe406ffd700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080c6240 0x7fe3f0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe407fff700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0004d50 0x7fe3f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.933 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4077fe700 1 -- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080c6240 msgr2=0x7fe3f0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:27.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4077fe700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080c6240 0x7fe3f0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4077fe700 1 -- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0004d50 msgr2=0x7fe3f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:27.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4077fe700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0004d50 0x7fe3f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4077fe700 1 -- 172.21.15.175:0/1874382131 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f0003570 con 0x7fe3f0003c20 2023-10-10T19:29:27.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe407fff700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0004d50 0x7fe3f00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:27.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4077fe700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f0003c20 0x7fe3f0004010 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800e210 tx=0x7fe3f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86656dbb407abe74 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.935 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800eac0 con 0x7fe3f0003c20 2023-10-10T19:29:27.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f800f040 con 0x7fe3f0003c20 2023-10-10T19:29:27.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.917+0000 7fe4067fc700 1 -- 172.21.15.175:0/1874382131 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800ee70 con 0x7fe3f0003c20 2023-10-10T19:29:27.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.918+0000 7fe40e18e700 1 -- 172.21.15.175:0/1874382131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f0003c20 msgr2=0x7fe3f0004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.918+0000 7fe40e18e700 1 --2- 172.21.15.175:0/1874382131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f0003c20 0x7fe3f0004010 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800e210 tx=0x7fe3f800b040 comp rx=0 tx=0).stop 2023-10-10T19:29:27.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.918+0000 7fe407fff700 1 -- 172.21.15.175:0/1874382131 reap_dead start 2023-10-10T19:29:27.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.918+0000 7fe40e18e700 1 -- 172.21.15.175:0/1874382131 shutdown_connections 2023-10-10T19:29:27.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.918+0000 7fe40e18e700 1 -- 172.21.15.175:0/1874382131 >> 172.21.15.175:0/1874382131 conn(0x7fe408063230 msgr2=0x7fe408063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:27.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.918+0000 7fe40e18e700 1 -- 172.21.15.175:0/1874382131 shutdown_connections 2023-10-10T19:29:27.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.918+0000 7fe40e18e700 1 -- 172.21.15.175:0/1874382131 wait complete. 2023-10-10T19:29:27.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.919+0000 7fe40e18e700 1 Processor -- start 2023-10-10T19:29:27.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.919+0000 7fe40e18e700 1 -- start start 2023-10-10T19:29:27.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe40e18e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0003c20 0x7fe4080c0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe40e18e700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4080cc0e0 0x7fe40814d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.939 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe40e18e700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080cc530 0x7fe40814da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe40e18e700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40814df70 con 0x7fe4080cc0e0 2023-10-10T19:29:27.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe40e18e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408166590 con 0x7fe3f0003c20 2023-10-10T19:29:27.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe40e18e700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080ccdc0 con 0x7fe4080cc530 2023-10-10T19:29:27.940 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4080cc0e0 0x7fe40814d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4080cc0e0 0x7fe40814d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60640/0 (socket says 172.21.15.175:60640) 2023-10-10T19:29:27.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 -- 172.21.15.175:0/1597839541 learned_addr learned my addr 172.21.15.175:0/1597839541 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:27.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe4077fe700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0003c20 0x7fe4080c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe407fff700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080cc530 0x7fe40814da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 -- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080cc530 msgr2=0x7fe40814da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe4080cc530 0x7fe40814da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 -- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0003c20 msgr2=0x7fe4080c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f0003c20 0x7fe4080c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 -- 172.21.15.175:0/1597839541 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080cd060 con 0x7fe4080cc0e0 2023-10-10T19:29:27.943 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.920+0000 7fe406ffd700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4080cc0e0 0x7fe40814d4f0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc000fd0 tx=0x7fe3fc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.921+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc018030 con 0x7fe4080cc0e0 2023-10-10T19:29:27.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.921+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc00f9c0 con 0x7fe4080cc0e0 2023-10-10T19:29:27.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.921+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4081547e0 con 0x7fe4080cc0e0 2023-10-10T19:29:27.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.921+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc00fd60 con 0x7fe4080cc0e0 2023-10-10T19:29:27.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.921+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe408165f40 con 0x7fe4080cc0e0 2023-10-10T19:29:27.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.921+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe408154ce0 con 0x7fe4080cc0e0 2023-10-10T19:29:27.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.922+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe3fc01b070 con 0x7fe4080cc0e0 2023-10-10T19:29:27.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.922+0000 7fe3ef7fe700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe3f4066f60 0x7fe3f4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.923+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fe3fc019070 con 0x7fe4080cc0e0 2023-10-10T19:29:27.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.923+0000 7fe3ef7fe700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fe3f40705a0 0x7fe3f4072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:27.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.923+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 --> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3f40730e0 con 0x7fe3f40705a0 2023-10-10T19:29:27.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.923+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3fc089220 con 0x7fe4080cc0e0 2023-10-10T19:29:27.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.923+0000 7fe4077fe700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe3f4066f60 0x7fe3f4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.923+0000 7fe407fff700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fe3f40705a0 0x7fe3f4072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:27.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.923+0000 7fe4077fe700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe3f4066f60 0x7fe3f4069410 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800afd0 tx=0x7fe3f801a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.923+0000 7fe407fff700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fe3f40705a0 0x7fe3f4072a10 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:27.948 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.924+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 <== osd.5 v2:172.21.15.191:6816/91495 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fe3f40730e0 con 0x7fe3f40705a0 2023-10-10T19:29:27.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.937+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 --> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe4080c4b60 con 0x7fe3f40705a0 2023-10-10T19:29:27.949 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe3ef7fe700 1 -- 172.21.15.175:0/1597839541 <== osd.5 v2:172.21.15.191:6816/91495 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe4080c4b60 con 0x7fe3f40705a0 2023-10-10T19:29:27.950 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T19:29:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-10T19:29:27.951 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T19:29:27.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fe3f40705a0 msgr2=0x7fe3f4072a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.952 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6816/91495,v1:172.21.15.191:6817/91495] conn(0x7fe3f40705a0 0x7fe3f4072a10 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:27.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe3f4066f60 msgr2=0x7fe3f4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fe3f4066f60 0x7fe3f4069410 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800afd0 tx=0x7fe3f801a000 comp rx=0 tx=0).stop 2023-10-10T19:29:27.953 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4080cc0e0 msgr2=0x7fe40814d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:27.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 --2- 172.21.15.175:0/1597839541 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4080cc0e0 0x7fe40814d4f0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc000fd0 tx=0x7fe3fc007a00 comp rx=0 tx=0).stop 2023-10-10T19:29:27.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe407fff700 1 -- 172.21.15.175:0/1597839541 reap_dead start 2023-10-10T19:29:27.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 shutdown_connections 2023-10-10T19:29:27.954 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 >> 172.21.15.175:0/1597839541 conn(0x7fe408063230 msgr2=0x7fe408063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:27.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 shutdown_connections 2023-10-10T19:29:27.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:27.938+0000 7fe40e18e700 1 -- 172.21.15.175:0/1597839541 wait complete. 2023-10-10T19:29:27.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-10T19:29:28.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.073+0000 7f83ea09b700 1 Processor -- start 2023-10-10T19:29:28.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.073+0000 7f83ea09b700 1 -- start start 2023-10-10T19:29:28.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.073+0000 7f83ea09b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e41575c0 0x7f83e40c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.073+0000 7f83ea09b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f83e4157990 0x7f83e40c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.073+0000 7f83ea09b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e40c35f0 con 0x7f83e4157dc0 2023-10-10T19:29:28.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.073+0000 7f83ea09b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e406c0e0 con 0x7f83e4157990 2023-10-10T19:29:28.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.073+0000 7f83ea09b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e406c3c0 con 0x7f83e41575c0 2023-10-10T19:29:28.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e37fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e41575c0 0x7f83e40c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e2ffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f83e4157990 0x7f83e40c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e2ffd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f83e4157990 0x7f83e40c3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.175:38858/0 (socket says 172.21.15.175:38858) 2023-10-10T19:29:28.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e37fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e41575c0 0x7f83e40c19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:40228/0 (socket says 172.21.15.175:40228) 2023-10-10T19:29:28.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e2ffd700 1 -- 172.21.15.175:0/1853620518 learned_addr learned my addr 172.21.15.175:0/1853620518 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:28.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e37fe700 1 -- 172.21.15.175:0/1853620518 >> v2:172.21.15.191:3300/0 conn(0x7f83e4157990 msgr2=0x7f83e40c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:28.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e37fe700 1 --2- 172.21.15.175:0/1853620518 >> v2:172.21.15.191:3300/0 conn(0x7f83e4157990 0x7f83e40c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e37fe700 1 -- 172.21.15.175:0/1853620518 >> v1:172.21.15.191:6789/0 conn(0x7f83e4157dc0 legacy=0x7f83e414d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e37fe700 1 -- 172.21.15.175:0/1853620518 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e406d870 con 0x7f83e41575c0 2023-10-10T19:29:28.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e2ffd700 1 --2- 172.21.15.175:0/1853620518 >> v2:172.21.15.191:3300/0 conn(0x7f83e4157990 0x7f83e40c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:28.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e37fe700 1 --2- 172.21.15.175:0/1853620518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e41575c0 0x7f83e40c19b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f83d400f300 tx=0x7f83d400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85ad85016f3a8b99 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e27fc700 1 -- 172.21.15.175:0/1853620518 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4012070 con 0x7f83e41575c0 2023-10-10T19:29:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e27fc700 1 -- 172.21.15.175:0/1853620518 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d400d0d0 con 0x7f83e41575c0 2023-10-10T19:29:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.074+0000 7f83e27fc700 1 -- 172.21.15.175:0/1853620518 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d400d490 con 0x7f83e41575c0 2023-10-10T19:29:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.075+0000 7f83ea09b700 1 -- 172.21.15.175:0/1853620518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e41575c0 msgr2=0x7f83e40c19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.075+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1853620518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e41575c0 0x7f83e40c19b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f83d400f300 tx=0x7f83d400db90 comp rx=0 tx=0).stop 2023-10-10T19:29:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.075+0000 7f83ea09b700 1 -- 172.21.15.175:0/1853620518 shutdown_connections 2023-10-10T19:29:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.075+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1853620518 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e41575c0 0x7f83e40c19b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.075+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1853620518 >> v2:172.21.15.191:3300/0 conn(0x7f83e4157990 0x7f83e40c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.075+0000 7f83ea09b700 1 -- 172.21.15.175:0/1853620518 >> 172.21.15.175:0/1853620518 conn(0x7f83e4063cc0 msgr2=0x7f83e406d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:28.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.075+0000 7f83ea09b700 1 -- 172.21.15.175:0/1853620518 shutdown_connections 2023-10-10T19:29:28.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.075+0000 7f83ea09b700 1 -- 172.21.15.175:0/1853620518 wait complete. 2023-10-10T19:29:28.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.076+0000 7f83ea09b700 1 Processor -- start 2023-10-10T19:29:28.086 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.076+0000 7f83ea09b700 1 -- start start 2023-10-10T19:29:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83ea09b700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e4157990 0x7f83e40c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83ea09b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e4157dc0 0x7f83e40c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83ea09b700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40c0180 0x7f83e4165280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83ea09b700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40c1060 con 0x7f83e40c0180 2023-10-10T19:29:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83ea09b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40cb0f0 con 0x7f83e4157dc0 2023-10-10T19:29:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83ea09b700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40cb390 con 0x7f83e4157990 2023-10-10T19:29:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40c0180 0x7f83e4165280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40c0180 0x7f83e4165280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60656/0 (socket says 172.21.15.175:60656) 2023-10-10T19:29:28.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 -- 172.21.15.175:0/1697519152 learned_addr learned my addr 172.21.15.175:0/1697519152 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:28.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e2ffd700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e4157dc0 0x7f83e40c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.089 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e37fe700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e4157990 0x7f83e40c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 -- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e4157990 msgr2=0x7f83e40c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e4157990 0x7f83e40c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.090 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 -- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e4157dc0 msgr2=0x7f83e40c3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e4157dc0 0x7f83e40c3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 -- 172.21.15.175:0/1697519152 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d80094f0 con 0x7f83e40c0180 2023-10-10T19:29:28.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.077+0000 7f83e3fff700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40c0180 0x7f83e4165280 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00c4c0 tx=0x7f83dc007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.078+0000 7f83cb7fe700 1 -- 172.21.15.175:0/1697519152 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc00ce70 con 0x7f83e40c0180 2023-10-10T19:29:28.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.078+0000 7f83cb7fe700 1 -- 172.21.15.175:0/1697519152 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83dc011590 con 0x7f83e40c0180 2023-10-10T19:29:28.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.078+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d400de30 con 0x7f83e40c0180 2023-10-10T19:29:28.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.078+0000 7f83cb7fe700 1 -- 172.21.15.175:0/1697519152 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc011970 con 0x7f83e40c0180 2023-10-10T19:29:28.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.078+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e40bfe70 con 0x7f83e40c0180 2023-10-10T19:29:28.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.078+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e40c83b0 con 0x7f83e40c0180 2023-10-10T19:29:28.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.079+0000 7f83cb7fe700 1 -- 172.21.15.175:0/1697519152 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f83dc011ce0 con 0x7f83e40c0180 2023-10-10T19:29:28.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.079+0000 7f83cb7fe700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83d0066f60 0x7f83d0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.079+0000 7f83cb7fe700 1 -- 172.21.15.175:0/1697519152 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f83dc084160 con 0x7f83e40c0180 2023-10-10T19:29:28.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.079+0000 7f83e37fe700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83d0066f60 0x7f83d0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.080+0000 7f83e37fe700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83d0066f60 0x7f83d0069410 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f83e4071510 tx=0x7f83d401c000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.083+0000 7f83cb7fe700 1 -- 172.21.15.175:0/1697519152 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f83e40c83b0 con 0x7f83e40c0180 2023-10-10T19:29:28.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.269+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f83e40c83b0 con 0x7f83d0066f60 2023-10-10T19:29:28.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.269+0000 7f83cb7fe700 1 -- 172.21.15.175:0/1697519152 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83e40c83b0 con 0x7f83d0066f60 2023-10-10T19:29:28.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83d0066f60 msgr2=0x7f83d0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83d0066f60 0x7f83d0069410 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f83e4071510 tx=0x7f83d401c000 comp rx=0 tx=0).stop 2023-10-10T19:29:28.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40c0180 msgr2=0x7f83e4165280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40c0180 0x7f83e4165280 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00c4c0 tx=0x7f83dc007f50 comp rx=0 tx=0).stop 2023-10-10T19:29:28.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 shutdown_connections 2023-10-10T19:29:28.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f83d0066f60 0x7f83d0069410 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40c0180 0x7f83e4165280 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83e4157990 0x7f83e40c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 --2- 172.21.15.175:0/1697519152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e4157dc0 0x7f83e40c3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 >> 172.21.15.175:0/1697519152 conn(0x7f83e4063cc0 msgr2=0x7f83e40c6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:28.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 shutdown_connections 2023-10-10T19:29:28.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.272+0000 7f83ea09b700 1 -- 172.21.15.175:0/1697519152 wait complete. 2023-10-10T19:29:28.276 INFO:teuthology.orchestra.run.smithi175.stderr:instructed osd(s) 5 to deep-scrub 2023-10-10T19:29:28.284 INFO:tasks.ceph:Scrubbing osd.6 2023-10-10T19:29:28.285 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-10T19:29:28.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.402+0000 7f63bf8bb700 1 Processor -- start 2023-10-10T19:29:28.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bf8bb700 1 -- start start 2023-10-10T19:29:28.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bf8bb700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63b81597a0 0x7f63b80c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bf8bb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b8069480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bf8bb700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f63b8159b70 0x7f63b8159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bf8bb700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b815a530 con 0x7f63b81597a0 2023-10-10T19:29:28.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bf8bb700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b80c6b70 con 0x7f63b8159b70 2023-10-10T19:29:28.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bf8bb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b8068c60 con 0x7f63b80c7fb0 2023-10-10T19:29:28.407 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bd657700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63b81597a0 0x7f63b80c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bd657700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f63b81597a0 0x7f63b80c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60668/0 (socket says 172.21.15.175:60668) 2023-10-10T19:29:28.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.403+0000 7f63bd657700 1 -- 172.21.15.175:0/3970513886 learned_addr learned my addr 172.21.15.175:0/3970513886 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:28.408 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bde58700 1 --2- 172.21.15.175:0/3970513886 >> v2:172.21.15.191:3300/0 conn(0x7f63b8159b70 0x7f63b8159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bd657700 1 -- 172.21.15.175:0/3970513886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 msgr2=0x7f63b8069480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:28.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bce56700 1 --2- 172.21.15.175:0/3970513886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b8069480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bd657700 1 --2- 172.21.15.175:0/3970513886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b8069480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.409 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bd657700 1 -- 172.21.15.175:0/3970513886 >> v2:172.21.15.191:3300/0 conn(0x7f63b8159b70 msgr2=0x7f63b8159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:28.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bd657700 1 --2- 172.21.15.175:0/3970513886 >> v2:172.21.15.191:3300/0 conn(0x7f63b8159b70 0x7f63b8159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bd657700 1 -- 172.21.15.175:0/3970513886 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b80699c0 con 0x7f63b81597a0 2023-10-10T19:29:28.410 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bde58700 1 --2- 172.21.15.175:0/3970513886 >> v2:172.21.15.191:3300/0 conn(0x7f63b8159b70 0x7f63b8159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:29:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bce56700 1 --2- 172.21.15.175:0/3970513886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b8069480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:29:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63bd657700 1 --2- 172.21.15.175:0/3970513886 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 0x7f63b80c2230 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f63b80ca070 tx=0x7f63b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40d0ceeb969608 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63affff700 1 -- 172.21.15.175:0/3970513886 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b400e070 con 0x7f63b81597a0 2023-10-10T19:29:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63affff700 1 -- 172.21.15.175:0/3970513886 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b400bae0 con 0x7f63b81597a0 2023-10-10T19:29:28.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.404+0000 7f63affff700 1 -- 172.21.15.175:0/3970513886 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b40092e0 con 0x7f63b81597a0 2023-10-10T19:29:28.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 -- 172.21.15.175:0/3970513886 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 msgr2=0x7f63b80c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 --2- 172.21.15.175:0/3970513886 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 0x7f63b80c2230 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f63b80ca070 tx=0x7f63b4009840 comp rx=0 tx=0).stop 2023-10-10T19:29:28.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 -- 172.21.15.175:0/3970513886 shutdown_connections 2023-10-10T19:29:28.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 --2- 172.21.15.175:0/3970513886 >> v2:172.21.15.191:3300/0 conn(0x7f63b8159b70 0x7f63b8159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 --2- 172.21.15.175:0/3970513886 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 0x7f63b80c2230 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.413 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 --2- 172.21.15.175:0/3970513886 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b8069480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 -- 172.21.15.175:0/3970513886 >> 172.21.15.175:0/3970513886 conn(0x7f63b8063230 msgr2=0x7f63b8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:28.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 -- 172.21.15.175:0/3970513886 shutdown_connections 2023-10-10T19:29:28.414 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.405+0000 7f63bf8bb700 1 -- 172.21.15.175:0/3970513886 wait complete. 2023-10-10T19:29:28.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.406+0000 7f63bf8bb700 1 Processor -- start 2023-10-10T19:29:28.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.406+0000 7f63bf8bb700 1 -- start start 2023-10-10T19:29:28.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bf8bb700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b80c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bf8bb700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 0x7f63b80c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bf8bb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b8159b70 0x7f63b814aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bf8bb700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80c4590 con 0x7f63b81597a0 2023-10-10T19:29:28.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bf8bb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b81564d0 con 0x7f63b8159b70 2023-10-10T19:29:28.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bf8bb700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80c9810 con 0x7f63b80c7fb0 2023-10-10T19:29:28.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 0x7f63b80c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 0x7f63b80c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60674/0 (socket says 172.21.15.175:60674) 2023-10-10T19:29:28.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 -- 172.21.15.175:0/189971012 learned_addr learned my addr 172.21.15.175:0/189971012 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:28.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bd657700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b80c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bde58700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b8159b70 0x7f63b814aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 -- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 msgr2=0x7f63b80c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b80c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 -- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b8159b70 msgr2=0x7f63b814aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b8159b70 0x7f63b814aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 -- 172.21.15.175:0/189971012 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4009ae0 con 0x7f63b81597a0 2023-10-10T19:29:28.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bd657700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63b80c7fb0 0x7f63b80c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:28.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bce56700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 0x7f63b80c8540 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f63a800c880 tx=0x7f63a8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8011600 con 0x7f63b81597a0 2023-10-10T19:29:28.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a8011760 con 0x7f63b81597a0 2023-10-10T19:29:28.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b8155eb0 con 0x7f63b81597a0 2023-10-10T19:29:28.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8011b20 con 0x7f63b81597a0 2023-10-10T19:29:28.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.407+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b80c3f60 con 0x7f63b81597a0 2023-10-10T19:29:28.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.408+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63b8067110 con 0x7f63b81597a0 2023-10-10T19:29:28.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.409+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f63a8018360 con 0x7f63b81597a0 2023-10-10T19:29:28.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.409+0000 7f63ad7fa700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f639c066f30 0x7f639c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.409+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f63a8084d60 con 0x7f63b81597a0 2023-10-10T19:29:28.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.409+0000 7f63bd657700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f639c066f30 0x7f639c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.409+0000 7f63ad7fa700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f639c0704f0 0x7f639c072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.409+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 --> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f639c073030 con 0x7f639c0704f0 2023-10-10T19:29:28.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.409+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63a80873a0 con 0x7f63b81597a0 2023-10-10T19:29:28.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.410+0000 7f63bde58700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f639c0704f0 0x7f639c072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.410+0000 7f63bd657700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f639c066f30 0x7f639c0693e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009880 tx=0x7f63b4018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.410+0000 7f63bde58700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f639c0704f0 0x7f639c072960 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.411+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 <== osd.6 v2:172.21.15.191:6808/91482 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f639c073030 con 0x7f639c0704f0 2023-10-10T19:29:28.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.424+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 --> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f63b8067570 con 0x7f639c0704f0 2023-10-10T19:29:28.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63ad7fa700 1 -- 172.21.15.175:0/189971012 <== osd.6 v2:172.21.15.191:6808/91482 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f63b8067570 con 0x7f639c0704f0 2023-10-10T19:29:28.429 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T19:29:28.430 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-10T19:29:28.430 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T19:29:28.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f639c0704f0 msgr2=0x7f639c072960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6808/91482,v1:172.21.15.191:6809/91482] conn(0x7f639c0704f0 0x7f639c072960 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f639c066f30 msgr2=0x7f639c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f639c066f30 0x7f639c0693e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009880 tx=0x7f63b4018000 comp rx=0 tx=0).stop 2023-10-10T19:29:28.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 msgr2=0x7f63b80c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 --2- 172.21.15.175:0/189971012 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f63b81597a0 0x7f63b80c8540 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f63a800c880 tx=0x7f63a8008040 comp rx=0 tx=0).stop 2023-10-10T19:29:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bde58700 1 -- 172.21.15.175:0/189971012 reap_dead start 2023-10-10T19:29:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 shutdown_connections 2023-10-10T19:29:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 >> 172.21.15.175:0/189971012 conn(0x7f63b8063230 msgr2=0x7f63b80cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:28.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 shutdown_connections 2023-10-10T19:29:28.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.425+0000 7f63bf8bb700 1 -- 172.21.15.175:0/189971012 wait complete. 2023-10-10T19:29:28.437 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-10T19:29:28.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.552+0000 7f1e4d349700 1 Processor -- start 2023-10-10T19:29:28.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.552+0000 7f1e4d349700 1 -- start start 2023-10-10T19:29:28.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e4d349700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e481597f0 0x7f1e480c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e4d349700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c1f60 0x7f1e48159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e4d349700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e480c74b0 con 0x7f1e480c86f0 2023-10-10T19:29:28.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e4d349700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e480683c0 con 0x7f1e481597f0 2023-10-10T19:29:28.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e4d349700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e48068f40 con 0x7f1e480c1f60 2023-10-10T19:29:28.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e46ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e481597f0 0x7f1e480c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e46ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1e481597f0 0x7f1e480c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60684/0 (socket says 172.21.15.175:60684) 2023-10-10T19:29:28.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e46ffd700 1 -- 172.21.15.175:0/17295402 learned_addr learned my addr 172.21.15.175:0/17295402 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:28.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e467fc700 1 --2- 172.21.15.175:0/17295402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c1f60 0x7f1e48159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e46ffd700 1 -- 172.21.15.175:0/17295402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c1f60 msgr2=0x7f1e48159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:28.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e46ffd700 1 --2- 172.21.15.175:0/17295402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c1f60 0x7f1e48159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e46ffd700 1 -- 172.21.15.175:0/17295402 >> v1:172.21.15.175:6789/0 conn(0x7f1e480c86f0 legacy=0x7f1e4814fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e46ffd700 1 -- 172.21.15.175:0/17295402 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e480693d0 con 0x7f1e481597f0 2023-10-10T19:29:28.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e467fc700 1 --2- 172.21.15.175:0/17295402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c1f60 0x7f1e48159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:29:28.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e46ffd700 1 --2- 172.21.15.175:0/17295402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e480c81b0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009f20 tx=0x7f1e38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf487bd4a28a970 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e45ffb700 1 -- 172.21.15.175:0/17295402 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800e070 con 0x7f1e481597f0 2023-10-10T19:29:28.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.553+0000 7f1e45ffb700 1 -- 172.21.15.175:0/17295402 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3800ba80 con 0x7f1e481597f0 2023-10-10T19:29:28.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e45ffb700 1 -- 172.21.15.175:0/17295402 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e380092e0 con 0x7f1e481597f0 2023-10-10T19:29:28.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e4d349700 1 -- 172.21.15.175:0/17295402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 msgr2=0x7f1e480c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e4d349700 1 --2- 172.21.15.175:0/17295402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e480c81b0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009f20 tx=0x7f1e38009840 comp rx=0 tx=0).stop 2023-10-10T19:29:28.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e4d349700 1 -- 172.21.15.175:0/17295402 shutdown_connections 2023-10-10T19:29:28.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e4d349700 1 --2- 172.21.15.175:0/17295402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e480c81b0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e4d349700 1 --2- 172.21.15.175:0/17295402 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c1f60 0x7f1e48159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e4d349700 1 -- 172.21.15.175:0/17295402 >> 172.21.15.175:0/17295402 conn(0x7f1e48063230 msgr2=0x7f1e48063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:28.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e4d349700 1 -- 172.21.15.175:0/17295402 shutdown_connections 2023-10-10T19:29:28.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.554+0000 7f1e4d349700 1 -- 172.21.15.175:0/17295402 wait complete. 2023-10-10T19:29:28.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.555+0000 7f1e4d349700 1 Processor -- start 2023-10-10T19:29:28.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.555+0000 7f1e4d349700 1 -- start start 2023-10-10T19:29:28.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e4d349700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e480c1f60 0x7f1e4815a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e4d349700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c86f0 0x7f1e4813f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e4d349700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e4813f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e4d349700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e48158000 con 0x7f1e481597f0 2023-10-10T19:29:28.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e4d349700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e481696a0 con 0x7f1e480c1f60 2023-10-10T19:29:28.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e4d349700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e48169980 con 0x7f1e480c86f0 2023-10-10T19:29:28.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e477fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e4813f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e477fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e4813f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60694/0 (socket says 172.21.15.175:60694) 2023-10-10T19:29:28.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e477fe700 1 -- 172.21.15.175:0/2244329377 learned_addr learned my addr 172.21.15.175:0/2244329377 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:28.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e467fc700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c86f0 0x7f1e4813f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e46ffd700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e480c1f60 0x7f1e4815a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e477fe700 1 -- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c86f0 msgr2=0x7f1e4813f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e477fe700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c86f0 0x7f1e4813f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e477fe700 1 -- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e480c1f60 msgr2=0x7f1e4815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e477fe700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e480c1f60 0x7f1e4815a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e477fe700 1 -- 172.21.15.175:0/2244329377 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e38009ae0 con 0x7f1e481597f0 2023-10-10T19:29:28.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.556+0000 7f1e467fc700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c86f0 0x7f1e4813f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:28.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.557+0000 7f1e477fe700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e4813f8d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000c560 tx=0x7f1e40007e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.557+0000 7f1e3effd700 1 -- 172.21.15.175:0/2244329377 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40011590 con 0x7f1e481597f0 2023-10-10T19:29:28.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.557+0000 7f1e3effd700 1 -- 172.21.15.175:0/2244329377 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e400116f0 con 0x7f1e481597f0 2023-10-10T19:29:28.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.557+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e48169c60 con 0x7f1e481597f0 2023-10-10T19:29:28.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.557+0000 7f1e3effd700 1 -- 172.21.15.175:0/2244329377 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40011a90 con 0x7f1e481597f0 2023-10-10T19:29:28.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.557+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4815ae20 con 0x7f1e481597f0 2023-10-10T19:29:28.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.558+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e48071720 con 0x7f1e481597f0 2023-10-10T19:29:28.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.558+0000 7f1e3effd700 1 -- 172.21.15.175:0/2244329377 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1e4001b3b0 con 0x7f1e481597f0 2023-10-10T19:29:28.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.558+0000 7f1e3effd700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.559+0000 7f1e3effd700 1 -- 172.21.15.175:0/2244329377 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f1e40084240 con 0x7f1e481597f0 2023-10-10T19:29:28.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.559+0000 7f1e46ffd700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.559+0000 7f1e46ffd700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009880 tx=0x7f1e38018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.563+0000 7f1e3effd700 1 -- 172.21.15.175:0/2244329377 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1e40050c30 con 0x7f1e481597f0 2023-10-10T19:29:28.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.756+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e48067570 con 0x7f1e28066f60 2023-10-10T19:29:28.757 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.757+0000 7f1e3effd700 1 -- 172.21.15.175:0/2244329377 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1e48067570 con 0x7f1e28066f60 2023-10-10T19:29:28.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 msgr2=0x7f1e28069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.760 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009880 tx=0x7f1e38018000 comp rx=0 tx=0).stop 2023-10-10T19:29:28.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 msgr2=0x7f1e4813f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e4813f8d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000c560 tx=0x7f1e40007e90 comp rx=0 tx=0).stop 2023-10-10T19:29:28.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 shutdown_connections 2023-10-10T19:29:28.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f1e28066f60 0x7f1e28069410 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1e481597f0 0x7f1e4813f8d0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.762 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e480c1f60 0x7f1e4815a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 --2- 172.21.15.175:0/2244329377 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e480c86f0 0x7f1e4813f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 >> 172.21.15.175:0/2244329377 conn(0x7f1e48063230 msgr2=0x7f1e48159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:28.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 shutdown_connections 2023-10-10T19:29:28.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.759+0000 7f1e4d349700 1 -- 172.21.15.175:0/2244329377 wait complete. 2023-10-10T19:29:28.764 INFO:teuthology.orchestra.run.smithi175.stderr:instructed osd(s) 6 to deep-scrub 2023-10-10T19:29:28.771 INFO:tasks.ceph:Scrubbing osd.7 2023-10-10T19:29:28.772 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-10T19:29:28.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.889+0000 7f92bbf89700 1 Processor -- start 2023-10-10T19:29:28.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.889+0000 7f92bbf89700 1 -- start start 2023-10-10T19:29:28.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.890+0000 7f92bbf89700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f92b40c6610 0x7f92b40c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.890+0000 7f92bbf89700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b40c4b60 con 0x7f92b40c4030 2023-10-10T19:29:28.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.890+0000 7f92bbf89700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b4068c50 con 0x7f92b40c6240 2023-10-10T19:29:28.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.890+0000 7f92bbf89700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b4157420 con 0x7f92b40c6610 2023-10-10T19:29:28.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.890+0000 7f92b9524700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f92b40c6610 0x7f92b40c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.890+0000 7f92b9524700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f92b40c6610 0x7f92b40c3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60700/0 (socket says 172.21.15.175:60700) 2023-10-10T19:29:28.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.890+0000 7f92b9524700 1 -- 172.21.15.175:0/520315216 learned_addr learned my addr 172.21.15.175:0/520315216 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:28.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92b9524700 1 -- 172.21.15.175:0/520315216 >> v1:172.21.15.191:6789/0 conn(0x7f92b40c6240 legacy=0x7f92b40690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:28.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92b9524700 1 -- 172.21.15.175:0/520315216 >> v1:172.21.15.175:6789/0 conn(0x7f92b40c4030 legacy=0x7f92b4159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92b9524700 1 -- 172.21.15.175:0/520315216 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b40c4880 con 0x7f92b40c6610 2023-10-10T19:29:28.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92b9524700 1 --2- 172.21.15.175:0/520315216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6610 0x7f92b40c3af0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f92a80099b0 tx=0x7f92a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4a80a62361e578 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92b8d23700 1 -- 172.21.15.175:0/520315216 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800b8b0 con 0x7f92b40c6610 2023-10-10T19:29:28.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92b8d23700 1 -- 172.21.15.175:0/520315216 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a800ba10 con 0x7f92b40c6610 2023-10-10T19:29:28.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92b8d23700 1 -- 172.21.15.175:0/520315216 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800bdb0 con 0x7f92b40c6610 2023-10-10T19:29:28.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92bbf89700 1 -- 172.21.15.175:0/520315216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6610 msgr2=0x7f92b40c3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92bbf89700 1 --2- 172.21.15.175:0/520315216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6610 0x7f92b40c3af0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f92a80099b0 tx=0x7f92a8009250 comp rx=0 tx=0).stop 2023-10-10T19:29:28.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92bbf89700 1 -- 172.21.15.175:0/520315216 shutdown_connections 2023-10-10T19:29:28.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92bbf89700 1 --2- 172.21.15.175:0/520315216 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6610 0x7f92b40c3af0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.891+0000 7f92bbf89700 1 -- 172.21.15.175:0/520315216 >> 172.21.15.175:0/520315216 conn(0x7f92b4063230 msgr2=0x7f92b4063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:28.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.892+0000 7f92bbf89700 1 -- 172.21.15.175:0/520315216 shutdown_connections 2023-10-10T19:29:28.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.892+0000 7f92bbf89700 1 -- 172.21.15.175:0/520315216 wait complete. 2023-10-10T19:29:28.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92bbf89700 1 Processor -- start 2023-10-10T19:29:28.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92bbf89700 1 -- start start 2023-10-10T19:29:28.902 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92bbf89700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92b40c4030 0x7f92b4152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92bbf89700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6240 0x7f92b4152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92bbf89700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92b40c6610 0x7f92b414d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.903 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92bbf89700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b40c08c0 con 0x7f92b40c6240 2023-10-10T19:29:28.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92bbf89700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b41524d0 con 0x7f92b40c6610 2023-10-10T19:29:28.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92bbf89700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b4068560 con 0x7f92b40c4030 2023-10-10T19:29:28.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92b9524700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6240 0x7f92b4152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.904 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92b9524700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6240 0x7f92b4152ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60702/0 (socket says 172.21.15.175:60702) 2023-10-10T19:29:28.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92b9524700 1 -- 172.21.15.175:0/4021388802 learned_addr learned my addr 172.21.15.175:0/4021388802 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:28.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.893+0000 7f92ba526700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92b40c6610 0x7f92b414d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b9d25700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92b40c4030 0x7f92b4152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b9524700 1 -- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92b40c4030 msgr2=0x7f92b4152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b9524700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92b40c4030 0x7f92b4152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b9524700 1 -- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92b40c6610 msgr2=0x7f92b414d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b9524700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92b40c6610 0x7f92b414d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b9524700 1 -- 172.21.15.175:0/4021388802 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a80094f0 con 0x7f92b40c6240 2023-10-10T19:29:28.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b9d25700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92b40c4030 0x7f92b4152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:29:28.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92ba526700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92b40c6610 0x7f92b414d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:28.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b9524700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6240 0x7f92b4152ea0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009290 tx=0x7f92a8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8011ea0 con 0x7f92b40c6240 2023-10-10T19:29:28.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a801c670 con 0x7f92b40c6240 2023-10-10T19:29:28.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b4151800 con 0x7f92b40c6240 2023-10-10T19:29:28.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a801ca30 con 0x7f92b40c6240 2023-10-10T19:29:28.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.894+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b4152160 con 0x7f92b40c6240 2023-10-10T19:29:28.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.895+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f92b4151fc0 con 0x7f92b40c6240 2023-10-10T19:29:28.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.895+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f92a801a450 con 0x7f92b40c6240 2023-10-10T19:29:28.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.896+0000 7f92b1ffb700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92a0066f30 0x7f92a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.896+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f92a8086f10 con 0x7f92b40c6240 2023-10-10T19:29:28.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.896+0000 7f92b9d25700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92a0066f30 0x7f92a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.896+0000 7f92b1ffb700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7f92a0070570 0x7f92a00729e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:28.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.896+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 --> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a00730b0 con 0x7f92a0070570 2023-10-10T19:29:28.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.896+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92a80894f0 con 0x7f92b40c6240 2023-10-10T19:29:28.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.896+0000 7f92ba526700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7f92a0070570 0x7f92a00729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:28.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.896+0000 7f92b9d25700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92a0066f30 0x7f92a00693e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f92a4007fd0 tx=0x7f92a4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.897+0000 7f92ba526700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7f92a0070570 0x7f92a00729e0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:28.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.897+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 <== osd.7 v2:172.21.15.191:6824/91512 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f92a00730b0 con 0x7f92a0070570 2023-10-10T19:29:28.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.911+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 --> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f92b4151d00 con 0x7f92a0070570 2023-10-10T19:29:28.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.911+0000 7f92b1ffb700 1 -- 172.21.15.175:0/4021388802 <== osd.7 v2:172.21.15.191:6824/91512 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f92b4151d00 con 0x7f92a0070570 2023-10-10T19:29:28.920 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-10-10T19:29:28.921 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-10T19:29:28.921 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-10-10T19:29:28.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7f92a0070570 msgr2=0x7f92a00729e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6824/91512,v1:172.21.15.191:6825/91512] conn(0x7f92a0070570 0x7f92a00729e0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:28.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92a0066f30 msgr2=0x7f92a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92a0066f30 0x7f92a00693e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f92a4007fd0 tx=0x7f92a4009040 comp rx=0 tx=0).stop 2023-10-10T19:29:28.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6240 msgr2=0x7f92b4152ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:28.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 --2- 172.21.15.175:0/4021388802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92b40c6240 0x7f92b4152ea0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009290 tx=0x7f92a8016000 comp rx=0 tx=0).stop 2023-10-10T19:29:28.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92ba526700 1 -- 172.21.15.175:0/4021388802 reap_dead start 2023-10-10T19:29:28.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 shutdown_connections 2023-10-10T19:29:28.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 >> 172.21.15.175:0/4021388802 conn(0x7f92b4063230 msgr2=0x7f92b414fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:28.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 shutdown_connections 2023-10-10T19:29:28.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:28.912+0000 7f92bbf89700 1 -- 172.21.15.175:0/4021388802 wait complete. 2023-10-10T19:29:28.929 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-10T19:29:29.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.049+0000 7f61dd345700 1 Processor -- start 2023-10-10T19:29:29.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.049+0000 7f61dd345700 1 -- start start 2023-10-10T19:29:29.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61dd345700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f61d80c58f0 0x7f61d8063c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61dd345700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c5cc0 0x7f61d80c6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61dd345700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d8068bf0 con 0x7f61d80b0180 2023-10-10T19:29:29.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61dd345700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d80c4fd0 con 0x7f61d80c58f0 2023-10-10T19:29:29.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61dd345700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d8156b40 con 0x7f61d80c5cc0 2023-10-10T19:29:29.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d6ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f61d80c58f0 0x7f61d8063c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.054 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d6ffd700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f61d80c58f0 0x7f61d8063c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60716/0 (socket says 172.21.15.175:60716) 2023-10-10T19:29:29.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d6ffd700 1 -- 172.21.15.175:0/3375848863 learned_addr learned my addr 172.21.15.175:0/3375848863 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:29.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d77fe700 1 --1- 172.21.15.175:0/3375848863 >> v1:172.21.15.175:6789/0 conn(0x7f61d80b0180 0x7f61d8159530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42026/0 (socket says 172.21.15.175:42026) 2023-10-10T19:29:29.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d6ffd700 1 -- 172.21.15.175:0/3375848863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c5cc0 msgr2=0x7f61d80c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:29.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d6ffd700 1 --2- 172.21.15.175:0/3375848863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c5cc0 0x7f61d80c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d6ffd700 1 -- 172.21.15.175:0/3375848863 >> v1:172.21.15.175:6789/0 conn(0x7f61d80b0180 legacy=0x7f61d8159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d6ffd700 1 -- 172.21.15.175:0/3375848863 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d80c5270 con 0x7f61d80c58f0 2023-10-10T19:29:29.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.050+0000 7f61d5ffb700 1 -- 172.21.15.175:0/3375848863 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785411232 0 0) 0x7f61d8068bf0 con 0x7f61d80b0180 2023-10-10T19:29:29.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61d6ffd700 1 --2- 172.21.15.175:0/3375848863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c58f0 0x7f61d8063c90 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009f20 tx=0x7f61c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952817ae21b746d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:29.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61d5ffb700 1 -- 172.21.15.175:0/3375848863 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c800e070 con 0x7f61d80c58f0 2023-10-10T19:29:29.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61d5ffb700 1 -- 172.21.15.175:0/3375848863 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c800bb10 con 0x7f61d80c58f0 2023-10-10T19:29:29.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61d5ffb700 1 -- 172.21.15.175:0/3375848863 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c80092e0 con 0x7f61d80c58f0 2023-10-10T19:29:29.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61dd345700 1 -- 172.21.15.175:0/3375848863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c58f0 msgr2=0x7f61d8063c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61dd345700 1 --2- 172.21.15.175:0/3375848863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c58f0 0x7f61d8063c90 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009f20 tx=0x7f61c8009840 comp rx=0 tx=0).stop 2023-10-10T19:29:29.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61dd345700 1 -- 172.21.15.175:0/3375848863 shutdown_connections 2023-10-10T19:29:29.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61dd345700 1 --2- 172.21.15.175:0/3375848863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c58f0 0x7f61d8063c90 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61dd345700 1 --2- 172.21.15.175:0/3375848863 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c5cc0 0x7f61d80c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.059 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61dd345700 1 -- 172.21.15.175:0/3375848863 >> 172.21.15.175:0/3375848863 conn(0x7f61d806dc60 msgr2=0x7f61d8063230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:29.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61dd345700 1 -- 172.21.15.175:0/3375848863 shutdown_connections 2023-10-10T19:29:29.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.051+0000 7f61dd345700 1 -- 172.21.15.175:0/3375848863 wait complete. 2023-10-10T19:29:29.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.052+0000 7f61dd345700 1 Processor -- start 2023-10-10T19:29:29.060 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.052+0000 7f61dd345700 1 -- start start 2023-10-10T19:29:29.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61dd345700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80b0180 0x7f61d80c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61dd345700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c58f0 0x7f61d8155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61dd345700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c5cc0 0x7f61d80c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61dd345700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80c4720 con 0x7f61d80c5cc0 2023-10-10T19:29:29.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61dd345700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80c2530 con 0x7f61d80b0180 2023-10-10T19:29:29.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61dd345700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80c1940 con 0x7f61d80c58f0 2023-10-10T19:29:29.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61d77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c5cc0 0x7f61d80c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61d77fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c5cc0 0x7f61d80c9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60732/0 (socket says 172.21.15.175:60732) 2023-10-10T19:29:29.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61d77fe700 1 -- 172.21.15.175:0/1067713930 learned_addr learned my addr 172.21.15.175:0/1067713930 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:29.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61d67fc700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c58f0 0x7f61d8155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.053+0000 7f61d6ffd700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80b0180 0x7f61d80c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61d77fe700 1 -- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c58f0 msgr2=0x7f61d8155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61d77fe700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c58f0 0x7f61d8155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61d77fe700 1 -- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80b0180 msgr2=0x7f61d80c3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61d77fe700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80b0180 0x7f61d80c3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61d77fe700 1 -- 172.21.15.175:0/1067713930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8009ae0 con 0x7f61d80c5cc0 2023-10-10T19:29:29.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61d77fe700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c5cc0 0x7f61d80c9f10 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c5b0 tx=0x7f61d0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:29.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61beffd700 1 -- 172.21.15.175:0/1067713930 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0005c10 con 0x7f61d80c5cc0 2023-10-10T19:29:29.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61beffd700 1 -- 172.21.15.175:0/1067713930 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d0005d70 con 0x7f61d80c5cc0 2023-10-10T19:29:29.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61beffd700 1 -- 172.21.15.175:0/1067713930 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0007300 con 0x7f61d80c5cc0 2023-10-10T19:29:29.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d80c1c00 con 0x7f61d80c5cc0 2023-10-10T19:29:29.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.054+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d80c21c0 con 0x7f61d80c5cc0 2023-10-10T19:29:29.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.055+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d80c28c0 con 0x7f61d80c5cc0 2023-10-10T19:29:29.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.055+0000 7f61beffd700 1 -- 172.21.15.175:0/1067713930 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f61d001b3b0 con 0x7f61d80c5cc0 2023-10-10T19:29:29.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.056+0000 7f61beffd700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61c4066f60 0x7f61c4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.056+0000 7f61beffd700 1 -- 172.21.15.175:0/1067713930 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f61d0084470 con 0x7f61d80c5cc0 2023-10-10T19:29:29.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.056+0000 7f61d6ffd700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61c4066f60 0x7f61c4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.056+0000 7f61d6ffd700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61c4066f60 0x7f61c4069410 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009880 tx=0x7f61c8018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:29.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.060+0000 7f61beffd700 1 -- 172.21.15.175:0/1067713930 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61d80c28c0 con 0x7f61d80c5cc0 2023-10-10T19:29:29.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.245+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f61d80c28c0 con 0x7f61c4066f60 2023-10-10T19:29:29.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.246+0000 7f61beffd700 1 -- 172.21.15.175:0/1067713930 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f61d80c28c0 con 0x7f61c4066f60 2023-10-10T19:29:29.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61c4066f60 msgr2=0x7f61c4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61c4066f60 0x7f61c4069410 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009880 tx=0x7f61c8018000 comp rx=0 tx=0).stop 2023-10-10T19:29:29.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c5cc0 msgr2=0x7f61d80c9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c5cc0 0x7f61d80c9f10 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c5b0 tx=0x7f61d0007d30 comp rx=0 tx=0).stop 2023-10-10T19:29:29.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 shutdown_connections 2023-10-10T19:29:29.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f61c4066f60 0x7f61c4069410 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f61d80c5cc0 0x7f61d80c9f10 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61d80b0180 0x7f61d80c3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 --2- 172.21.15.175:0/1067713930 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61d80c58f0 0x7f61d8155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 >> 172.21.15.175:0/1067713930 conn(0x7f61d806dc60 msgr2=0x7f61d8154d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:29.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 shutdown_connections 2023-10-10T19:29:29.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.248+0000 7f61dd345700 1 -- 172.21.15.175:0/1067713930 wait complete. 2023-10-10T19:29:29.252 INFO:teuthology.orchestra.run.smithi175.stderr:instructed osd(s) 7 to deep-scrub 2023-10-10T19:29:29.260 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T19:29:29.375 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.375+0000 7f9b22965700 1 Processor -- start 2023-10-10T19:29:29.376 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.375+0000 7f9b22965700 1 -- start start 2023-10-10T19:29:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b22965700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9b1c14fd90 0x7f9b1c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b22965700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c08a100 0x7f9b1c0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b22965700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1c067fa0 con 0x7f9b1c0cd040 2023-10-10T19:29:29.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b22965700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1c0c5600 con 0x7f9b1c14fd90 2023-10-10T19:29:29.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b22965700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c156b70 con 0x7f9b1c08a100 2023-10-10T19:29:29.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b22164700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f9b1c0cd040 0x7f9b1c0684f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:42034/0 (socket says 172.21.15.175:42034) 2023-10-10T19:29:29.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b22164700 1 -- 172.21.15.175:0/2489930602 learned_addr learned my addr 172.21.15.175:0/2489930602 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:29.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b21963700 1 --2- 172.21.15.175:0/2489930602 >> v2:172.21.15.191:3300/0 conn(0x7f9b1c14fd90 0x7f9b1c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b21162700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c08a100 0x7f9b1c0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199763119 0 0) 0x7f9b1c067fa0 con 0x7f9b1c0cd040 2023-10-10T19:29:29.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.376+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b04003570 con 0x7f9b1c0cd040 2023-10-10T19:29:29.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1337423843 0 0) 0x7f9b04003570 con 0x7f9b1c0cd040 2023-10-10T19:29:29.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c08a100 msgr2=0x7f9b1c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:29.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c08a100 0x7f9b1c0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 >> v2:172.21.15.191:3300/0 conn(0x7f9b1c14fd90 msgr2=0x7f9b1c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:29.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 --2- 172.21.15.175:0/2489930602 >> v2:172.21.15.191:3300/0 conn(0x7f9b1c14fd90 0x7f9b1c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c150160 con 0x7f9b1c0cd040 2023-10-10T19:29:29.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7f9b14001470 con 0x7f9b1c0cd040 2023-10-10T19:29:29.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 >> v1:172.21.15.175:6789/0 conn(0x7f9b1c0cd040 legacy=0x7f9b1c0684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04003a60 0x7f9b04003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b04004810 0x7f9b04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b040051c0 0x7f9b0400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c0c5600 con 0x7f9b04004810 2023-10-10T19:29:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c156b70 con 0x7f9b04003a60 2023-10-10T19:29:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0400bfa0 con 0x7f9b040051c0 2023-10-10T19:29:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21162700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c08a100 0x7f9b1c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21963700 1 --2- 172.21.15.175:0/2489930602 >> v2:172.21.15.191:3300/0 conn(0x7f9b1c14fd90 0x7f9b1c159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:29:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21162700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b04004810 0x7f9b04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b22164700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b040051c0 0x7f9b0400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21963700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04003a60 0x7f9b04003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21162700 1 -- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b040051c0 msgr2=0x7f9b0400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21162700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b040051c0 0x7f9b0400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21162700 1 -- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04003a60 msgr2=0x7f9b04003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:29.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21162700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04003a60 0x7f9b04003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21162700 1 -- 172.21.15.175:0/2489930602 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b04003570 con 0x7f9b04004810 2023-10-10T19:29:29.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21963700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04003a60 0x7f9b04003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:29:29.393 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b21162700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b04004810 0x7f9b04004c80 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0cd6b0 tx=0x7f9b100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ccfc14cc3f8b6e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:29.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1000e710 con 0x7f9b04004810 2023-10-10T19:29:29.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1000e870 con 0x7f9b04004810 2023-10-10T19:29:29.394 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.377+0000 7f9b20961700 1 -- 172.21.15.175:0/2489930602 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1000ec10 con 0x7f9b04004810 2023-10-10T19:29:29.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.378+0000 7f9b22965700 1 -- 172.21.15.175:0/2489930602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b04004810 msgr2=0x7f9b04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.378+0000 7f9b22965700 1 --2- 172.21.15.175:0/2489930602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b04004810 0x7f9b04004c80 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0cd6b0 tx=0x7f9b100091a0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.378+0000 7f9b22164700 1 -- 172.21.15.175:0/2489930602 reap_dead start 2023-10-10T19:29:29.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.378+0000 7f9b22965700 1 -- 172.21.15.175:0/2489930602 shutdown_connections 2023-10-10T19:29:29.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.378+0000 7f9b22965700 1 -- 172.21.15.175:0/2489930602 >> 172.21.15.175:0/2489930602 conn(0x7f9b1c063d00 msgr2=0x7f9b1c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:29.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.378+0000 7f9b22965700 1 -- 172.21.15.175:0/2489930602 shutdown_connections 2023-10-10T19:29:29.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.378+0000 7f9b22965700 1 -- 172.21.15.175:0/2489930602 wait complete. 2023-10-10T19:29:29.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.379+0000 7f9b22965700 1 Processor -- start 2023-10-10T19:29:29.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.379+0000 7f9b22965700 1 -- start start 2023-10-10T19:29:29.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22965700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04004810 0x7f9b1c166430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22965700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c14e970 0x7f9b1c14c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22965700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b1c0c3650 0x7f9b1c0c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22965700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c166e90 con 0x7f9b1c0c3650 2023-10-10T19:29:29.398 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22965700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c157670 con 0x7f9b04004810 2023-10-10T19:29:29.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22965700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c1676c0 con 0x7f9b1c14e970 2023-10-10T19:29:29.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b1c0c3650 0x7f9b1c0c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b1c0c3650 0x7f9b1c0c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60740/0 (socket says 172.21.15.175:60740) 2023-10-10T19:29:29.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b21963700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04004810 0x7f9b1c166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 -- 172.21.15.175:0/2383760453 learned_addr learned my addr 172.21.15.175:0/2383760453 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:29.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b21162700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c14e970 0x7f9b1c14c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 -- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c14e970 msgr2=0x7f9b1c14c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c14e970 0x7f9b1c14c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 -- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04004810 msgr2=0x7f9b1c166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04004810 0x7f9b1c166430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 -- 172.21.15.175:0/2383760453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c0c9860 con 0x7f9b1c0c3650 2023-10-10T19:29:29.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22164700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b1c0c3650 0x7f9b1c0c3ac0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c065030 tx=0x7f9b1c0c5dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:29.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b19ffb700 1 -- 172.21.15.175:0/2383760453 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1400f8a0 con 0x7f9b1c0c3650 2023-10-10T19:29:29.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b19ffb700 1 -- 172.21.15.175:0/2383760453 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1400fa20 con 0x7f9b1c0c3650 2023-10-10T19:29:29.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1c0c9b40 con 0x7f9b1c0c3650 2023-10-10T19:29:29.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.380+0000 7f9b19ffb700 1 -- 172.21.15.175:0/2383760453 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1400fde0 con 0x7f9b1c0c3650 2023-10-10T19:29:29.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.381+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1c167350 con 0x7f9b1c0c3650 2023-10-10T19:29:29.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.382+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b1c166970 con 0x7f9b1c0c3650 2023-10-10T19:29:29.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.382+0000 7f9b19ffb700 1 -- 172.21.15.175:0/2383760453 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9b14012560 con 0x7f9b1c0c3650 2023-10-10T19:29:29.404 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.382+0000 7f9b19ffb700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b08066f30 0x7f9b080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:29.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.382+0000 7f9b19ffb700 1 -- 172.21.15.175:0/2383760453 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f9b140845f0 con 0x7f9b1c0c3650 2023-10-10T19:29:29.405 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.383+0000 7f9b21963700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b08066f30 0x7f9b080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:29.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.383+0000 7f9b21963700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b08066f30 0x7f9b080693e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c007a70 tx=0x7f9b0c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:29.406 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.387+0000 7f9b19ffb700 1 -- 172.21.15.175:0/2383760453 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b14050fe0 con 0x7f9b1c0c3650 2023-10-10T19:29:29.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.568+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9b1c167870 con 0x7f9b08066f30 2023-10-10T19:29:29.577 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.577+0000 7f9b19ffb700 1 -- 172.21.15.175:0/2383760453 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294292 (secure 0 0 0) 0x7f9b1c167870 con 0x7f9b08066f30 2023-10-10T19:29:29.582 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:29.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b08066f30 msgr2=0x7f9b080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b08066f30 0x7f9b080693e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c007a70 tx=0x7f9b0c00b040 comp rx=0 tx=0).stop 2023-10-10T19:29:29.585 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b1c0c3650 msgr2=0x7f9b1c0c3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:29.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b1c0c3650 0x7f9b1c0c3ac0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c065030 tx=0x7f9b1c0c5dd0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.586 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 shutdown_connections 2023-10-10T19:29:29.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f9b08066f30 0x7f9b080693e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b1c0c3650 0x7f9b1c0c3ac0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.587 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b04004810 0x7f9b1c166430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 --2- 172.21.15.175:0/2383760453 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9b1c14e970 0x7f9b1c14c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:29.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 >> 172.21.15.175:0/2383760453 conn(0x7f9b1c063d00 msgr2=0x7f9b1c158200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:29.588 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 shutdown_connections 2023-10-10T19:29:29.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:29.580+0000 7f9b22965700 1 -- 172.21.15.175:0/2383760453 wait complete. 2023-10-10T19:29:29.589 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T19:29:29.593 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":1457,"stamp":"2023-10-10T19:29:27.818656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":384400818,"num_objects":2861,"num_object_clones":0,"num_object_copies":5722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":7382,"num_read_kb":3484532,"num_write":63300,"num_write_kb":2986306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":354,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80588,"ondisk_log_size":80588,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2699936,"kb_used_data":264288,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752274784,"statfs":{"total":773094113280,"available":770329378816,"internally_reserved":0,"allocated":270630912,"data_stored":775877414,"data_compressed":116711114,"data_compressed_allocated":144269312,"data_compressed_original":667049984,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":95,"apply_latency_ms":95,"commit_latency_ns":95000000,"apply_latency_ns":95000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13368205,"num_objects":851,"num_object_clones":0,"num_object_copies":1702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":923,"num_write_kb":13952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003442"},"pg_stats":[{"pgid":"3.3f","version":"132'966","reported_seq":3586,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.164790+0000","last_change":"2023-10-10T19:16:54.796013+0000","last_active":"2023-10-10T19:29:22.164790+0000","last_peered":"2023-10-10T19:29:22.164790+0000","last_clean":"2023-10-10T19:29:22.164790+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T19:29:22.164790+0000","last_undegraded":"2023-10-10T19:29:22.164790+0000","last_fullsized":"2023-10-10T19:29:22.164790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:23:16.822373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017995507000000001,"stat_sum":{"num_bytes":821890,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":62,"num_read_kb":8068,"num_write":683,"num_write_kb":16759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"132'1012","reported_seq":3631,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.147890+0000","last_change":"2023-10-10T19:16:54.920511+0000","last_active":"2023-10-10T19:29:22.147890+0000","last_peered":"2023-10-10T19:29:22.147890+0000","last_clean":"2023-10-10T19:29:22.147890+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T19:29:22.147890+0000","last_undegraded":"2023-10-10T19:29:22.147890+0000","last_fullsized":"2023-10-10T19:29:22.147890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:27.163351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083569260000000006,"stat_sum":{"num_bytes":1412883,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":62,"num_read_kb":28805,"num_write":746,"num_write_kb":46678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"132'962","reported_seq":3589,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.148214+0000","last_change":"2023-10-10T19:16:54.980353+0000","last_active":"2023-10-10T19:29:22.148214+0000","last_peered":"2023-10-10T19:29:22.148214+0000","last_clean":"2023-10-10T19:29:22.148214+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T19:29:22.148214+0000","last_undegraded":"2023-10-10T19:29:22.148214+0000","last_fullsized":"2023-10-10T19:29:22.148214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:30:15.449825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011353221,"stat_sum":{"num_bytes":671969,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":65,"num_read_kb":20680,"num_write":694,"num_write_kb":24810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"132'961","reported_seq":3575,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.155273+0000","last_change":"2023-10-10T19:16:55.412877+0000","last_active":"2023-10-10T19:29:22.155273+0000","last_peered":"2023-10-10T19:29:22.155273+0000","last_clean":"2023-10-10T19:29:22.155273+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T19:29:22.155273+0000","last_undegraded":"2023-10-10T19:29:22.155273+0000","last_fullsized":"2023-10-10T19:29:22.155273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:17.012457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088633549999999998,"stat_sum":{"num_bytes":651243,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":62,"num_read_kb":30965,"num_write":672,"num_write_kb":37710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"132'987","reported_seq":3617,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.156309+0000","last_change":"2023-10-10T19:16:54.790962+0000","last_active":"2023-10-10T19:29:22.156309+0000","last_peered":"2023-10-10T19:29:22.156309+0000","last_clean":"2023-10-10T19:29:22.156309+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T19:29:22.156309+0000","last_undegraded":"2023-10-10T19:29:22.156309+0000","last_fullsized":"2023-10-10T19:29:22.156309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:29.393232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016189688000000001,"stat_sum":{"num_bytes":219054,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":75,"num_read_kb":17848,"num_write":702,"num_write_kb":31364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"132'942","reported_seq":3568,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162027+0000","last_change":"2023-10-10T19:16:55.414924+0000","last_active":"2023-10-10T19:29:22.162027+0000","last_peered":"2023-10-10T19:29:22.162027+0000","last_clean":"2023-10-10T19:29:22.162027+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T19:29:22.162027+0000","last_undegraded":"2023-10-10T19:29:22.162027+0000","last_fullsized":"2023-10-10T19:29:22.162027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:07.049472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010631753000000001,"stat_sum":{"num_bytes":2108369,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":63,"num_read_kb":20939,"num_write":675,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"132'987","reported_seq":3608,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162863+0000","last_change":"2023-10-10T19:16:54.789065+0000","last_active":"2023-10-10T19:29:22.162863+0000","last_peered":"2023-10-10T19:29:22.162863+0000","last_clean":"2023-10-10T19:29:22.162863+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T19:29:22.162863+0000","last_undegraded":"2023-10-10T19:29:22.162863+0000","last_fullsized":"2023-10-10T19:29:22.162863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:17:57.307092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014667316999999999,"stat_sum":{"num_bytes":1119068,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":77,"num_read_kb":53600,"num_write":685,"num_write_kb":52880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"132'880","reported_seq":3518,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.161954+0000","last_change":"2023-10-10T19:16:54.870456+0000","last_active":"2023-10-10T19:29:22.161954+0000","last_peered":"2023-10-10T19:29:22.161954+0000","last_clean":"2023-10-10T19:29:22.161954+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T19:29:22.161954+0000","last_undegraded":"2023-10-10T19:29:22.161954+0000","last_fullsized":"2023-10-10T19:29:22.161954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:49.526367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012402231,"stat_sum":{"num_bytes":484780,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":58,"num_read_kb":28110,"num_write":652,"num_write_kb":35721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"132'976","reported_seq":3603,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.164602+0000","last_change":"2023-10-10T19:16:54.795632+0000","last_active":"2023-10-10T19:29:22.164602+0000","last_peered":"2023-10-10T19:29:22.164602+0000","last_clean":"2023-10-10T19:29:22.164602+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T19:29:22.164602+0000","last_undegraded":"2023-10-10T19:29:22.164602+0000","last_fullsized":"2023-10-10T19:29:22.164602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":976,"log_dups_size":0,"ondisk_log_size":976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:40.198136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017699890999999999,"stat_sum":{"num_bytes":94035,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":76,"num_read_kb":37435,"num_write":702,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"132'955","reported_seq":3581,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.153792+0000","last_change":"2023-10-10T19:16:54.783622+0000","last_active":"2023-10-10T19:29:22.153792+0000","last_peered":"2023-10-10T19:29:22.153792+0000","last_clean":"2023-10-10T19:29:22.153792+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T19:29:22.153792+0000","last_undegraded":"2023-10-10T19:29:22.153792+0000","last_fullsized":"2023-10-10T19:29:22.153792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:37.540092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079196160000000008,"stat_sum":{"num_bytes":269975,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":70,"num_read_kb":16825,"num_write":697,"num_write_kb":23418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"132'975","reported_seq":3624,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163041+0000","last_change":"2023-10-10T19:16:54.784745+0000","last_active":"2023-10-10T19:29:22.163041+0000","last_peered":"2023-10-10T19:29:22.163041+0000","last_clean":"2023-10-10T19:29:22.163041+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T19:29:22.163041+0000","last_undegraded":"2023-10-10T19:29:22.163041+0000","last_fullsized":"2023-10-10T19:29:22.163041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:49.196216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088422029999999999,"stat_sum":{"num_bytes":213661,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":82,"num_read_kb":46829,"num_write":712,"num_write_kb":53066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"132'972","reported_seq":3607,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163846+0000","last_change":"2023-10-10T19:16:54.989857+0000","last_active":"2023-10-10T19:29:22.163846+0000","last_peered":"2023-10-10T19:29:22.163846+0000","last_clean":"2023-10-10T19:29:22.163846+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T19:29:22.163846+0000","last_undegraded":"2023-10-10T19:29:22.163846+0000","last_fullsized":"2023-10-10T19:29:22.163846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:05.579860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078589550000000008,"stat_sum":{"num_bytes":337999,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":75,"num_read_kb":57532,"num_write":710,"num_write_kb":73524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"132'929","reported_seq":3559,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157282+0000","last_change":"2023-10-10T19:16:54.872408+0000","last_active":"2023-10-10T19:29:22.157282+0000","last_peered":"2023-10-10T19:29:22.157282+0000","last_clean":"2023-10-10T19:29:22.157282+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T19:29:22.157282+0000","last_undegraded":"2023-10-10T19:29:22.157282+0000","last_fullsized":"2023-10-10T19:29:22.157282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:25.228986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014204069,"stat_sum":{"num_bytes":4832489,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":73,"num_read_kb":32208,"num_write":656,"num_write_kb":44878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"132'929","reported_seq":3553,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.154826+0000","last_change":"2023-10-10T19:16:54.978714+0000","last_active":"2023-10-10T19:29:22.154826+0000","last_peered":"2023-10-10T19:29:22.154826+0000","last_clean":"2023-10-10T19:29:22.154826+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T19:29:22.154826+0000","last_undegraded":"2023-10-10T19:29:22.154826+0000","last_fullsized":"2023-10-10T19:29:22.154826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:50:30.808854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097965510000000006,"stat_sum":{"num_bytes":724304,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":63,"num_read_kb":28420,"num_write":663,"num_write_kb":31175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"132'955","reported_seq":3588,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.154959+0000","last_change":"2023-10-10T19:16:54.917427+0000","last_active":"2023-10-10T19:29:22.154959+0000","last_peered":"2023-10-10T19:29:22.154959+0000","last_clean":"2023-10-10T19:29:22.154959+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T19:29:22.154959+0000","last_undegraded":"2023-10-10T19:29:22.154959+0000","last_fullsized":"2023-10-10T19:29:22.154959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:48:38.409279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059774370000000004,"stat_sum":{"num_bytes":1353080,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":77,"num_read_kb":52747,"num_write":668,"num_write_kb":73053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"132'998","reported_seq":3632,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162046+0000","last_change":"2023-10-10T19:16:55.412885+0000","last_active":"2023-10-10T19:29:22.162046+0000","last_peered":"2023-10-10T19:29:22.162046+0000","last_clean":"2023-10-10T19:29:22.162046+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T19:29:22.162046+0000","last_undegraded":"2023-10-10T19:29:22.162046+0000","last_fullsized":"2023-10-10T19:29:22.162046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:12:56.786237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087668889999999999,"stat_sum":{"num_bytes":582240,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":80,"num_read_kb":42995,"num_write":709,"num_write_kb":53070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"132'905","reported_seq":3559,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.165956+0000","last_change":"2023-10-10T19:16:55.417126+0000","last_active":"2023-10-10T19:29:22.165956+0000","last_peered":"2023-10-10T19:29:22.165956+0000","last_clean":"2023-10-10T19:29:22.165956+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T19:29:22.165956+0000","last_undegraded":"2023-10-10T19:29:22.165956+0000","last_fullsized":"2023-10-10T19:29:22.165956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":905,"log_dups_size":0,"ondisk_log_size":905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:02.166845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012085218,"stat_sum":{"num_bytes":1912035,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":72,"num_read_kb":40372,"num_write":683,"num_write_kb":48886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"132'847","reported_seq":3519,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.149914+0000","last_change":"2023-10-10T19:16:54.780747+0000","last_active":"2023-10-10T19:29:22.149914+0000","last_peered":"2023-10-10T19:29:22.149914+0000","last_clean":"2023-10-10T19:29:22.149914+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T19:29:22.149914+0000","last_undegraded":"2023-10-10T19:29:22.149914+0000","last_fullsized":"2023-10-10T19:29:22.149914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":847,"log_dups_size":0,"ondisk_log_size":847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:28:07.768101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00594963,"stat_sum":{"num_bytes":953364,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":72,"num_read_kb":44243,"num_write":664,"num_write_kb":60950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"132'1073","reported_seq":3670,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163145+0000","last_change":"2023-10-10T19:16:54.792335+0000","last_active":"2023-10-10T19:29:22.163145+0000","last_peered":"2023-10-10T19:29:22.163145+0000","last_clean":"2023-10-10T19:29:22.163145+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T19:29:22.163145+0000","last_undegraded":"2023-10-10T19:29:22.163145+0000","last_fullsized":"2023-10-10T19:29:22.163145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:32.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01601,"stat_sum":{"num_bytes":689975,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":73,"num_read_kb":41187,"num_write":728,"num_write_kb":49784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"132'837","reported_seq":1002,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.912821+0000","last_change":"2023-10-10T18:45:10.413649+0000","last_active":"2023-10-10T19:28:33.912821+0000","last_peered":"2023-10-10T19:28:33.912821+0000","last_clean":"2023-10-10T19:28:33.912821+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T19:28:33.912821+0000","last_undegraded":"2023-10-10T19:28:33.912821+0000","last_fullsized":"2023-10-10T19:28:33.912821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:46.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389252,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":82,"num_read_kb":61516,"num_write":781,"num_write_kb":21019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"132'998","reported_seq":3619,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157863+0000","last_change":"2023-10-10T19:16:54.877116+0000","last_active":"2023-10-10T19:29:22.157863+0000","last_peered":"2023-10-10T19:29:22.157863+0000","last_clean":"2023-10-10T19:29:22.157863+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T19:29:22.157863+0000","last_undegraded":"2023-10-10T19:29:22.157863+0000","last_fullsized":"2023-10-10T19:29:22.157863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:08.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018086130999999998,"stat_sum":{"num_bytes":136255,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":63,"num_read_kb":13169,"num_write":738,"num_write_kb":31183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"132'761","reported_seq":905,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:20.191360+0000","last_change":"2023-10-10T18:45:10.408538+0000","last_active":"2023-10-10T19:29:20.191360+0000","last_peered":"2023-10-10T19:29:20.191360+0000","last_clean":"2023-10-10T19:29:20.191360+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T19:29:20.191360+0000","last_undegraded":"2023-10-10T19:29:20.191360+0000","last_fullsized":"2023-10-10T19:29:20.191360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:33:05.722855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":29,"num_read_kb":69691,"num_write":711,"num_write_kb":16940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"132'985","reported_seq":3627,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.167057+0000","last_change":"2023-10-10T19:16:54.978352+0000","last_active":"2023-10-10T19:29:22.167057+0000","last_peered":"2023-10-10T19:29:22.167057+0000","last_clean":"2023-10-10T19:29:22.167057+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T19:29:22.167057+0000","last_undegraded":"2023-10-10T19:29:22.167057+0000","last_fullsized":"2023-10-10T19:29:22.167057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":985,"log_dups_size":0,"ondisk_log_size":985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:56.701633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093729599999999996,"stat_sum":{"num_bytes":224018,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":74,"num_read_kb":34573,"num_write":731,"num_write_kb":44057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"132'1694","reported_seq":1870,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:10.134874+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T19:29:10.134874+0000","last_peered":"2023-10-10T19:29:10.134874+0000","last_clean":"2023-10-10T19:29:10.134874+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T19:29:10.134874+0000","last_undegraded":"2023-10-10T19:29:10.134874+0000","last_fullsized":"2023-10-10T19:29:10.134874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1694,"log_dups_size":0,"ondisk_log_size":1694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:22:21.397075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":114,"num_read_kb":59511,"num_write":1903,"num_write_kb":22655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"132'952","reported_seq":3546,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157482+0000","last_change":"2023-10-10T19:16:54.780201+0000","last_active":"2023-10-10T19:29:22.157482+0000","last_peered":"2023-10-10T19:29:22.157482+0000","last_clean":"2023-10-10T19:29:22.157482+0000","last_became_active":"2023-10-10T18:46:01.103860+0000","last_became_peered":"2023-10-10T18:46:01.103860+0000","last_unstale":"2023-10-10T19:29:22.157482+0000","last_undegraded":"2023-10-10T19:29:22.157482+0000","last_fullsized":"2023-10-10T19:29:22.157482+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:17.975194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054668909999999998,"stat_sum":{"num_bytes":121718,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":48,"num_read_kb":295,"num_write":648,"num_write_kb":7505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"130'750","reported_seq":903,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.970799+0000","last_change":"2023-10-10T18:45:10.413773+0000","last_active":"2023-10-10T19:27:29.970799+0000","last_peered":"2023-10-10T19:27:29.970799+0000","last_clean":"2023-10-10T19:27:29.970799+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T19:27:29.970799+0000","last_undegraded":"2023-10-10T19:27:29.970799+0000","last_fullsized":"2023-10-10T19:27:29.970799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:02.526945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389162,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":55,"num_read_kb":36831,"num_write":683,"num_write_kb":16858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"132'991","reported_seq":3621,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157486+0000","last_change":"2023-10-10T19:16:55.416075+0000","last_active":"2023-10-10T19:29:22.157486+0000","last_peered":"2023-10-10T19:29:22.157486+0000","last_clean":"2023-10-10T19:29:22.157486+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T19:29:22.157486+0000","last_undegraded":"2023-10-10T19:29:22.157486+0000","last_fullsized":"2023-10-10T19:29:22.157486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:42:47.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011753095,"stat_sum":{"num_bytes":171234,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":81,"num_read_kb":35882,"num_write":699,"num_write_kb":43024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"132'613","reported_seq":761,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.930535+0000","last_change":"2023-10-10T18:45:10.412395+0000","last_active":"2023-10-10T19:28:38.930535+0000","last_peered":"2023-10-10T19:28:38.930535+0000","last_clean":"2023-10-10T19:28:38.930535+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T19:28:38.930535+0000","last_undegraded":"2023-10-10T19:28:38.930535+0000","last_fullsized":"2023-10-10T19:28:38.930535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:28.926569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14118723,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":31,"num_read_kb":65583,"num_write":555,"num_write_kb":18319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"132'917","reported_seq":3545,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.149542+0000","last_change":"2023-10-10T19:16:54.987567+0000","last_active":"2023-10-10T19:29:22.149542+0000","last_peered":"2023-10-10T19:29:22.149542+0000","last_clean":"2023-10-10T19:29:22.149542+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T19:29:22.149542+0000","last_undegraded":"2023-10-10T19:29:22.149542+0000","last_fullsized":"2023-10-10T19:29:22.149542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:56:49.238733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062644010000000002,"stat_sum":{"num_bytes":838983,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":59,"num_read_kb":21826,"num_write":689,"num_write_kb":37727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"132'776","reported_seq":1108,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.423638+0000","last_change":"2023-10-10T18:45:10.400732+0000","last_active":"2023-10-10T19:28:39.423638+0000","last_peered":"2023-10-10T19:28:39.423638+0000","last_clean":"2023-10-10T19:28:39.423638+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T19:28:39.423638+0000","last_undegraded":"2023-10-10T19:28:39.423638+0000","last_fullsized":"2023-10-10T19:28:39.423638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:29:38.772157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111936,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":251,"num_read_kb":81147,"num_write":702,"num_write_kb":25397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"132'840","reported_seq":3488,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.158508+0000","last_change":"2023-10-10T19:16:54.984860+0000","last_active":"2023-10-10T19:29:22.158508+0000","last_peered":"2023-10-10T19:29:22.158508+0000","last_clean":"2023-10-10T19:29:22.158508+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T19:29:22.158508+0000","last_undegraded":"2023-10-10T19:29:22.158508+0000","last_fullsized":"2023-10-10T19:29:22.158508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:36.072962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015269961,"stat_sum":{"num_bytes":473837,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":60,"num_read_kb":24046,"num_write":615,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"130'1494","reported_seq":1889,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.913899+0000","last_change":"2023-10-10T18:45:10.407094+0000","last_active":"2023-10-10T19:28:38.913899+0000","last_peered":"2023-10-10T19:28:38.913899+0000","last_clean":"2023-10-10T19:28:38.913899+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T19:28:38.913899+0000","last_undegraded":"2023-10-10T19:28:38.913899+0000","last_fullsized":"2023-10-10T19:28:38.913899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:59:02.878600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4727705,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":295,"num_read_kb":15405,"num_write":1447,"num_write_kb":10055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"132'1002","reported_seq":3645,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.158566+0000","last_change":"2023-10-10T19:16:54.874108+0000","last_active":"2023-10-10T19:29:22.158566+0000","last_peered":"2023-10-10T19:29:22.158566+0000","last_clean":"2023-10-10T19:29:22.158566+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T19:29:22.158566+0000","last_undegraded":"2023-10-10T19:29:22.158566+0000","last_fullsized":"2023-10-10T19:29:22.158566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:13:47.865084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015767953000000001,"stat_sum":{"num_bytes":671315,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":81,"num_read_kb":19092,"num_write":747,"num_write_kb":30378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"131'494","reported_seq":639,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704437+0000","last_change":"2023-10-10T18:45:10.408236+0000","last_active":"2023-10-10T19:28:34.704437+0000","last_peered":"2023-10-10T19:28:34.704437+0000","last_clean":"2023-10-10T19:28:34.704437+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T19:28:34.704437+0000","last_undegraded":"2023-10-10T19:28:34.704437+0000","last_fullsized":"2023-10-10T19:28:34.704437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:02:43.919895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583466,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":65566,"num_write":451,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"132'1030","reported_seq":3669,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.145411+0000","last_change":"2023-10-10T19:16:54.918258+0000","last_active":"2023-10-10T19:29:22.145411+0000","last_peered":"2023-10-10T19:29:22.145411+0000","last_clean":"2023-10-10T19:29:22.145411+0000","last_became_active":"2023-10-10T18:46:00.606391+0000","last_became_peered":"2023-10-10T18:46:00.606391+0000","last_unstale":"2023-10-10T19:29:22.145411+0000","last_undegraded":"2023-10-10T19:29:22.145411+0000","last_fullsized":"2023-10-10T19:29:22.145411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:57.665869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066939039999999997,"stat_sum":{"num_bytes":730064,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":80,"num_read_kb":42231,"num_write":744,"num_write_kb":52869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"132'278","reported_seq":419,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:20.188061+0000","last_change":"2023-10-10T18:45:10.408649+0000","last_active":"2023-10-10T19:29:20.188061+0000","last_peered":"2023-10-10T19:29:20.188061+0000","last_clean":"2023-10-10T19:29:20.188061+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T19:29:20.188061+0000","last_undegraded":"2023-10-10T19:29:20.188061+0000","last_fullsized":"2023-10-10T19:29:20.188061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":34,"num_read_kb":54,"num_write":232,"num_write_kb":4550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"132'963","reported_seq":3598,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.165336+0000","last_change":"2023-10-10T19:16:55.413821+0000","last_active":"2023-10-10T19:29:22.165336+0000","last_peered":"2023-10-10T19:29:22.165336+0000","last_clean":"2023-10-10T19:29:22.165336+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T19:29:22.165336+0000","last_undegraded":"2023-10-10T19:29:22.165336+0000","last_fullsized":"2023-10-10T19:29:22.165336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":963,"log_dups_size":0,"ondisk_log_size":963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:47.700416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096962509999999995,"stat_sum":{"num_bytes":540276,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":63,"num_read_kb":26256,"num_write":725,"num_write_kb":46019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"132'619","reported_seq":977,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.098109+0000","last_change":"2023-10-10T18:45:10.408391+0000","last_active":"2023-10-10T19:29:22.098109+0000","last_peered":"2023-10-10T19:29:22.098109+0000","last_clean":"2023-10-10T19:29:22.098109+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T19:29:22.098109+0000","last_undegraded":"2023-10-10T19:29:22.098109+0000","last_fullsized":"2023-10-10T19:29:22.098109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:23.966299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7933208,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":262,"num_read_kb":30418,"num_write":558,"num_write_kb":12266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"132'930","reported_seq":3553,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.167256+0000","last_change":"2023-10-10T19:16:54.984649+0000","last_active":"2023-10-10T19:29:22.167256+0000","last_peered":"2023-10-10T19:29:22.167256+0000","last_clean":"2023-10-10T19:29:22.167256+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T19:29:22.167256+0000","last_undegraded":"2023-10-10T19:29:22.167256+0000","last_fullsized":"2023-10-10T19:29:22.167256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:50:59.472993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015222243,"stat_sum":{"num_bytes":331805,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":64,"num_read_kb":38795,"num_write":639,"num_write_kb":46131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"132'208","reported_seq":350,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.915056+0000","last_change":"2023-10-10T18:45:10.410066+0000","last_active":"2023-10-10T19:28:38.915056+0000","last_peered":"2023-10-10T19:28:38.915056+0000","last_clean":"2023-10-10T19:28:38.915056+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T19:28:38.915056+0000","last_undegraded":"2023-10-10T19:28:38.915056+0000","last_fullsized":"2023-10-10T19:28:38.915056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:45:02.633942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":61,"num_write":173,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"132'877","reported_seq":3509,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.165002+0000","last_change":"2023-10-10T19:16:54.989921+0000","last_active":"2023-10-10T19:29:22.165002+0000","last_peered":"2023-10-10T19:29:22.165002+0000","last_clean":"2023-10-10T19:29:22.165002+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T19:29:22.165002+0000","last_undegraded":"2023-10-10T19:29:22.165002+0000","last_fullsized":"2023-10-10T19:29:22.165002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:04:24.363813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080742209999999995,"stat_sum":{"num_bytes":103314,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":56,"num_read_kb":23103,"num_write":655,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"132'265","reported_seq":414,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.542051+0000","last_change":"2023-10-10T18:45:10.404054+0000","last_active":"2023-10-10T19:29:04.542051+0000","last_peered":"2023-10-10T19:29:04.542051+0000","last_clean":"2023-10-10T19:29:04.542051+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T19:29:04.542051+0000","last_undegraded":"2023-10-10T19:29:04.542051+0000","last_fullsized":"2023-10-10T19:29:04.542051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:02:39.484378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194948,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":59,"num_read_kb":16449,"num_write":213,"num_write_kb":4412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"132'954","reported_seq":3584,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.165984+0000","last_change":"2023-10-10T19:16:54.793178+0000","last_active":"2023-10-10T19:29:22.165984+0000","last_peered":"2023-10-10T19:29:22.165984+0000","last_clean":"2023-10-10T19:29:22.165984+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T19:29:22.165984+0000","last_undegraded":"2023-10-10T19:29:22.165984+0000","last_fullsized":"2023-10-10T19:29:22.165984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":954,"log_dups_size":0,"ondisk_log_size":954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:53.305231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015774979000000001,"stat_sum":{"num_bytes":487172,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":71,"num_read_kb":24956,"num_write":677,"num_write_kb":31709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"132'819","reported_seq":1255,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.546232+0000","last_change":"2023-10-10T18:45:10.394491+0000","last_active":"2023-10-10T19:29:04.546232+0000","last_peered":"2023-10-10T19:29:04.546232+0000","last_clean":"2023-10-10T19:29:04.546232+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T19:29:04.546232+0000","last_undegraded":"2023-10-10T19:29:04.546232+0000","last_fullsized":"2023-10-10T19:29:04.546232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:46:11.673628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227398,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":359,"num_read_kb":46031,"num_write":771,"num_write_kb":13519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"132'930","reported_seq":3561,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.165014+0000","last_change":"2023-10-10T19:16:54.981687+0000","last_active":"2023-10-10T19:29:22.165014+0000","last_peered":"2023-10-10T19:29:22.165014+0000","last_clean":"2023-10-10T19:29:22.165014+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T19:29:22.165014+0000","last_undegraded":"2023-10-10T19:29:22.165014+0000","last_fullsized":"2023-10-10T19:29:22.165014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:31:19.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01251216,"stat_sum":{"num_bytes":121104,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":62,"num_read_kb":21731,"num_write":687,"num_write_kb":32925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"132'717","reported_seq":877,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:20.770679+0000","last_change":"2023-10-10T18:45:10.407750+0000","last_active":"2023-10-10T19:29:20.770679+0000","last_peered":"2023-10-10T19:29:20.770679+0000","last_clean":"2023-10-10T19:29:20.770679+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T19:29:20.770679+0000","last_undegraded":"2023-10-10T19:29:20.770679+0000","last_fullsized":"2023-10-10T19:29:20.770679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:03.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15580825,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":58,"num_read_kb":76305,"num_write":652,"num_write_kb":19766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"132'969","reported_seq":3594,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163425+0000","last_change":"2023-10-10T19:16:54.791948+0000","last_active":"2023-10-10T19:29:22.163425+0000","last_peered":"2023-10-10T19:29:22.163425+0000","last_clean":"2023-10-10T19:29:22.163425+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T19:29:22.163425+0000","last_undegraded":"2023-10-10T19:29:22.163425+0000","last_fullsized":"2023-10-10T19:29:22.163425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":969,"log_dups_size":0,"ondisk_log_size":969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:24:52.006813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017093574,"stat_sum":{"num_bytes":558529,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":73,"num_read_kb":45017,"num_write":693,"num_write_kb":44107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"131'407","reported_seq":563,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.926155+0000","last_change":"2023-10-10T18:45:10.404033+0000","last_active":"2023-10-10T19:28:38.926155+0000","last_peered":"2023-10-10T19:28:38.926155+0000","last_clean":"2023-10-10T19:28:38.926155+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T19:28:38.926155+0000","last_undegraded":"2023-10-10T19:28:38.926155+0000","last_fullsized":"2023-10-10T19:28:38.926155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:51.980386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":54,"num_read_kb":84439,"num_write":360,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"132'834","reported_seq":3466,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.139699+0000","last_change":"2023-10-10T19:16:54.782792+0000","last_active":"2023-10-10T19:29:22.139699+0000","last_peered":"2023-10-10T19:29:22.139699+0000","last_clean":"2023-10-10T19:29:22.139699+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T19:29:22.139699+0000","last_undegraded":"2023-10-10T19:29:22.139699+0000","last_fullsized":"2023-10-10T19:29:22.139699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:44:19.698421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075683979999999996,"stat_sum":{"num_bytes":1111788,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":52,"num_read_kb":24653,"num_write":611,"num_write_kb":45399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"131'804","reported_seq":983,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.911148+0000","last_change":"2023-10-10T18:45:10.405809+0000","last_active":"2023-10-10T19:28:33.911148+0000","last_peered":"2023-10-10T19:28:33.911148+0000","last_clean":"2023-10-10T19:28:33.911148+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T19:28:33.911148+0000","last_undegraded":"2023-10-10T19:28:33.911148+0000","last_fullsized":"2023-10-10T19:28:33.911148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":804,"log_dups_size":0,"ondisk_log_size":804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:55:00.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":113,"num_read_kb":78302,"num_write":734,"num_write_kb":25074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"132'915","reported_seq":3552,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.155368+0000","last_change":"2023-10-10T19:16:54.790805+0000","last_active":"2023-10-10T19:29:22.155368+0000","last_peered":"2023-10-10T19:29:22.155368+0000","last_clean":"2023-10-10T19:29:22.155368+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T19:29:22.155368+0000","last_undegraded":"2023-10-10T19:29:22.155368+0000","last_fullsized":"2023-10-10T19:29:22.155368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":915,"log_dups_size":0,"ondisk_log_size":915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:57:03.782975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01616896,"stat_sum":{"num_bytes":90659,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":67,"num_read_kb":45230,"num_write":665,"num_write_kb":49103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"132'606","reported_seq":743,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:20.188497+0000","last_change":"2023-10-10T18:45:10.400115+0000","last_active":"2023-10-10T19:29:20.188497+0000","last_peered":"2023-10-10T19:29:20.188497+0000","last_clean":"2023-10-10T19:29:20.188497+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T19:29:20.188497+0000","last_undegraded":"2023-10-10T19:29:20.188497+0000","last_fullsized":"2023-10-10T19:29:20.188497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:03:28.011609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":23,"num_read_kb":12341,"num_write":750,"num_write_kb":8877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"132'938","reported_seq":3552,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.141031+0000","last_change":"2023-10-10T19:16:54.793503+0000","last_active":"2023-10-10T19:29:22.141031+0000","last_peered":"2023-10-10T19:29:22.141031+0000","last_clean":"2023-10-10T19:29:22.141031+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T19:29:22.141031+0000","last_undegraded":"2023-10-10T19:29:22.141031+0000","last_fullsized":"2023-10-10T19:29:22.141031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:13:32.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015882960000000002,"stat_sum":{"num_bytes":708648,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":57,"num_read_kb":20900,"num_write":676,"num_write_kb":24718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"132'163","reported_seq":300,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.098712+0000","last_change":"2023-10-10T18:45:10.406355+0000","last_active":"2023-10-10T19:29:22.098712+0000","last_peered":"2023-10-10T19:29:22.098712+0000","last_clean":"2023-10-10T19:29:22.098712+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T19:29:22.098712+0000","last_undegraded":"2023-10-10T19:29:22.098712+0000","last_fullsized":"2023-10-10T19:29:22.098712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:56.251170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":45123,"num_write":119,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"132'1049","reported_seq":3657,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157770+0000","last_change":"2023-10-10T19:16:55.415960+0000","last_active":"2023-10-10T19:29:22.157770+0000","last_peered":"2023-10-10T19:29:22.157770+0000","last_clean":"2023-10-10T19:29:22.157770+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T19:29:22.157770+0000","last_undegraded":"2023-10-10T19:29:22.157770+0000","last_fullsized":"2023-10-10T19:29:22.157770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:19.153248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011491507999999999,"stat_sum":{"num_bytes":461869,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":71,"num_read_kb":20716,"num_write":730,"num_write_kb":35297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"130'341","reported_seq":489,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:34.704918+0000","last_change":"2023-10-10T18:45:10.408858+0000","last_active":"2023-10-10T19:28:34.704918+0000","last_peered":"2023-10-10T19:28:34.704918+0000","last_clean":"2023-10-10T19:28:34.704918+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T19:28:34.704918+0000","last_undegraded":"2023-10-10T19:28:34.704918+0000","last_fullsized":"2023-10-10T19:28:34.704918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:26:55.547920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":73794,"num_write":284,"num_write_kb":20986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"132'953","reported_seq":3620,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.159602+0000","last_change":"2023-10-10T19:16:54.988658+0000","last_active":"2023-10-10T19:29:22.159602+0000","last_peered":"2023-10-10T19:29:22.159602+0000","last_clean":"2023-10-10T19:29:22.159602+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T19:29:22.159602+0000","last_undegraded":"2023-10-10T19:29:22.159602+0000","last_fullsized":"2023-10-10T19:29:22.159602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:04.319639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071091339999999996,"stat_sum":{"num_bytes":369103,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":93,"num_read_kb":25139,"num_write":713,"num_write_kb":35847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"132'945","reported_seq":3570,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.140067+0000","last_change":"2023-10-10T19:16:54.919574+0000","last_active":"2023-10-10T19:29:22.140067+0000","last_peered":"2023-10-10T19:29:22.140067+0000","last_clean":"2023-10-10T19:29:22.140067+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T19:29:22.140067+0000","last_undegraded":"2023-10-10T19:29:22.140067+0000","last_fullsized":"2023-10-10T19:29:22.140067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:36.722974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077260940000000002,"stat_sum":{"num_bytes":131785,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":61,"num_read_kb":19697,"num_write":689,"num_write_kb":26724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"132'686","reported_seq":1110,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:18.044315+0000","last_change":"2023-10-10T18:45:10.404513+0000","last_active":"2023-10-10T19:29:18.044315+0000","last_peered":"2023-10-10T19:29:18.044315+0000","last_clean":"2023-10-10T19:29:18.044315+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T19:29:18.044315+0000","last_undegraded":"2023-10-10T19:29:18.044315+0000","last_fullsized":"2023-10-10T19:29:18.044315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:38.975656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9108221,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":314,"num_read_kb":40841,"num_write":610,"num_write_kb":9448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'896","reported_seq":3566,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162414+0000","last_change":"2023-10-10T19:16:54.875665+0000","last_active":"2023-10-10T19:29:22.162414+0000","last_peered":"2023-10-10T19:29:22.162414+0000","last_clean":"2023-10-10T19:29:22.162414+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T19:29:22.162414+0000","last_undegraded":"2023-10-10T19:29:22.162414+0000","last_fullsized":"2023-10-10T19:29:22.162414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:36.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017091021000000001,"stat_sum":{"num_bytes":951410,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":88,"num_read_kb":19324,"num_write":674,"num_write_kb":26724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"131'137","reported_seq":294,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.142425+0000","last_change":"2023-10-10T18:45:10.404706+0000","last_active":"2023-10-10T19:28:39.142425+0000","last_peered":"2023-10-10T19:28:39.142425+0000","last_clean":"2023-10-10T19:28:39.142425+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T19:28:39.142425+0000","last_undegraded":"2023-10-10T19:28:39.142425+0000","last_fullsized":"2023-10-10T19:28:39.142425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:33.455996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388756,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":36939,"num_write":74,"num_write_kb":8306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":274,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.970643+0000","last_change":"2023-10-10T18:45:07.111885+0000","last_active":"2023-10-10T19:27:29.970643+0000","last_peered":"2023-10-10T19:27:29.970643+0000","last_clean":"2023-10-10T19:27:29.970643+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T19:27:29.970643+0000","last_undegraded":"2023-10-10T19:27:29.970643+0000","last_fullsized":"2023-10-10T19:27:29.970643+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:06.056019+0000","last_clean_scrub_stamp":"2023-10-10T18:45:06.056019+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:33:08.086335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"132'908","reported_seq":3531,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.158553+0000","last_change":"2023-10-10T19:16:54.979966+0000","last_active":"2023-10-10T19:29:22.158553+0000","last_peered":"2023-10-10T19:29:22.158553+0000","last_clean":"2023-10-10T19:29:22.158553+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T19:29:22.158553+0000","last_undegraded":"2023-10-10T19:29:22.158553+0000","last_fullsized":"2023-10-10T19:29:22.158553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:27:28.239443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010927930000000001,"stat_sum":{"num_bytes":3454752,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":55,"num_read_kb":20626,"num_write":663,"num_write_kb":30766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"132'974","reported_seq":1125,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:40.513525+0000","last_change":"2023-10-10T18:45:10.413074+0000","last_active":"2023-10-10T19:28:40.513525+0000","last_peered":"2023-10-10T19:28:40.513525+0000","last_clean":"2023-10-10T19:28:40.513525+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T19:28:40.513525+0000","last_undegraded":"2023-10-10T19:28:40.513525+0000","last_fullsized":"2023-10-10T19:28:40.513525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:08:23.904313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":38,"num_read_kb":89926,"num_write":914,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"132'826","reported_seq":979,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:38.911768+0000","last_change":"2023-10-10T18:45:10.415476+0000","last_active":"2023-10-10T19:28:38.911768+0000","last_peered":"2023-10-10T19:28:38.911768+0000","last_clean":"2023-10-10T19:28:38.911768+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T19:28:38.911768+0000","last_undegraded":"2023-10-10T19:28:38.911768+0000","last_fullsized":"2023-10-10T19:28:38.911768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:37:07.743570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":45,"num_read_kb":45594,"num_write":766,"num_write_kb":12847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"132'1032","reported_seq":3645,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.145283+0000","last_change":"2023-10-10T19:16:54.919294+0000","last_active":"2023-10-10T19:29:22.145283+0000","last_peered":"2023-10-10T19:29:22.145283+0000","last_clean":"2023-10-10T19:29:22.145283+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T19:29:22.145283+0000","last_undegraded":"2023-10-10T19:29:22.145283+0000","last_fullsized":"2023-10-10T19:29:22.145283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:28:56.855931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075449510000000003,"stat_sum":{"num_bytes":524227,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":65,"num_read_kb":49387,"num_write":726,"num_write_kb":52412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'277","reported_seq":433,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:24.018095+0000","last_change":"2023-10-10T18:45:10.406057+0000","last_active":"2023-10-10T19:29:24.018095+0000","last_peered":"2023-10-10T19:29:24.018095+0000","last_clean":"2023-10-10T19:29:24.018095+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T19:29:24.018095+0000","last_undegraded":"2023-10-10T19:29:24.018095+0000","last_fullsized":"2023-10-10T19:29:24.018095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:17:58.465101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":81,"num_read_kb":20675,"num_write":479,"num_write_kb":4749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"132'957","reported_seq":3583,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.155271+0000","last_change":"2023-10-10T19:16:54.920290+0000","last_active":"2023-10-10T19:29:22.155271+0000","last_peered":"2023-10-10T19:29:22.155271+0000","last_clean":"2023-10-10T19:29:22.155271+0000","last_became_active":"2023-10-10T18:46:01.267092+0000","last_became_peered":"2023-10-10T18:46:01.267092+0000","last_unstale":"2023-10-10T19:29:22.155271+0000","last_undegraded":"2023-10-10T19:29:22.155271+0000","last_fullsized":"2023-10-10T19:29:22.155271+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:46:17.491022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083609700000000006,"stat_sum":{"num_bytes":2285567,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":68,"num_read_kb":33081,"num_write":691,"num_write_kb":42143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"132'155","reported_seq":302,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.293003+0000","last_change":"2023-10-10T18:45:10.411043+0000","last_active":"2023-10-10T19:29:04.293003+0000","last_peered":"2023-10-10T19:29:04.293003+0000","last_clean":"2023-10-10T19:29:04.293003+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T19:29:04.293003+0000","last_undegraded":"2023-10-10T19:29:04.293003+0000","last_fullsized":"2023-10-10T19:29:04.293003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:00:13.775193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":8269,"num_write":94,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"132'885","reported_seq":3545,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.143091+0000","last_change":"2023-10-10T19:16:54.981885+0000","last_active":"2023-10-10T19:29:22.143091+0000","last_peered":"2023-10-10T19:29:22.143091+0000","last_clean":"2023-10-10T19:29:22.143091+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T19:29:22.143091+0000","last_undegraded":"2023-10-10T19:29:22.143091+0000","last_fullsized":"2023-10-10T19:29:22.143091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:11:17.201677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721872,"stat_sum":{"num_bytes":482904,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":85,"num_read_kb":32652,"num_write":648,"num_write_kb":36187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"131'228","reported_seq":371,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:39.557527+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T19:28:39.557527+0000","last_peered":"2023-10-10T19:28:39.557527+0000","last_clean":"2023-10-10T19:28:39.557527+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T19:28:39.557527+0000","last_undegraded":"2023-10-10T19:28:39.557527+0000","last_fullsized":"2023-10-10T19:28:39.557527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:57.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":57389,"num_write":173,"num_write_kb":12477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"132'951","reported_seq":3590,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.160879+0000","last_change":"2023-10-10T19:16:54.990974+0000","last_active":"2023-10-10T19:29:22.160879+0000","last_peered":"2023-10-10T19:29:22.160879+0000","last_clean":"2023-10-10T19:29:22.160879+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T19:29:22.160879+0000","last_undegraded":"2023-10-10T19:29:22.160879+0000","last_fullsized":"2023-10-10T19:29:22.160879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:11.589983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089059480000000003,"stat_sum":{"num_bytes":820479,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":67,"num_read_kb":26489,"num_write":692,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"132'370","reported_seq":522,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:18.043542+0000","last_change":"2023-10-10T18:45:10.412518+0000","last_active":"2023-10-10T19:29:18.043542+0000","last_peered":"2023-10-10T19:29:18.043542+0000","last_clean":"2023-10-10T19:29:18.043542+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T19:29:18.043542+0000","last_undegraded":"2023-10-10T19:29:18.043542+0000","last_fullsized":"2023-10-10T19:29:18.043542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:24.347236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":70,"num_read_kb":20656,"num_write":333,"num_write_kb":8747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"132'908","reported_seq":3542,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.144014+0000","last_change":"2023-10-10T19:16:54.870214+0000","last_active":"2023-10-10T19:29:22.144014+0000","last_peered":"2023-10-10T19:29:22.144014+0000","last_clean":"2023-10-10T19:29:22.144014+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T19:29:22.144014+0000","last_undegraded":"2023-10-10T19:29:22.144014+0000","last_fullsized":"2023-10-10T19:29:22.144014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:07:03.077401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012052205,"stat_sum":{"num_bytes":635543,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":63,"num_read_kb":40064,"num_write":667,"num_write_kb":56874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"132'1307","reported_seq":1462,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:20.770644+0000","last_change":"2023-10-10T18:45:10.406107+0000","last_active":"2023-10-10T19:29:20.770644+0000","last_peered":"2023-10-10T19:29:20.770644+0000","last_clean":"2023-10-10T19:29:20.770644+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T19:29:20.770644+0000","last_undegraded":"2023-10-10T19:29:20.770644+0000","last_fullsized":"2023-10-10T19:29:20.770644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:42.881791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15020210,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":48,"num_read_kb":53317,"num_write":1717,"num_write_kb":24748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"132'893","reported_seq":3525,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.158065+0000","last_change":"2023-10-10T19:16:54.874295+0000","last_active":"2023-10-10T19:29:22.158065+0000","last_peered":"2023-10-10T19:29:22.158065+0000","last_clean":"2023-10-10T19:29:22.158065+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T19:29:22.158065+0000","last_undegraded":"2023-10-10T19:29:22.158065+0000","last_fullsized":"2023-10-10T19:29:22.158065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:37:16.460140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015827366999999998,"stat_sum":{"num_bytes":178682,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":58,"num_read_kb":28965,"num_write":652,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"132'1202","reported_seq":1356,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:20.188102+0000","last_change":"2023-10-10T18:45:10.411172+0000","last_active":"2023-10-10T19:29:20.188102+0000","last_peered":"2023-10-10T19:29:20.188102+0000","last_clean":"2023-10-10T19:29:20.188102+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T19:29:20.188102+0000","last_undegraded":"2023-10-10T19:29:20.188102+0000","last_fullsized":"2023-10-10T19:29:20.188102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1202,"log_dups_size":0,"ondisk_log_size":1202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:19.677079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":49,"num_read_kb":89747,"num_write":1128,"num_write_kb":25455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"132'938","reported_seq":3566,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.144203+0000","last_change":"2023-10-10T19:16:54.781695+0000","last_active":"2023-10-10T19:29:22.144203+0000","last_peered":"2023-10-10T19:29:22.144203+0000","last_clean":"2023-10-10T19:29:22.144203+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T19:29:22.144203+0000","last_undegraded":"2023-10-10T19:29:22.144203+0000","last_fullsized":"2023-10-10T19:29:22.144203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:47:34.424057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065052069999999998,"stat_sum":{"num_bytes":479125,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":64,"num_read_kb":10527,"num_write":680,"num_write_kb":20069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"132'312","reported_seq":452,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:28:33.911550+0000","last_change":"2023-10-10T18:45:10.405876+0000","last_active":"2023-10-10T19:28:33.911550+0000","last_peered":"2023-10-10T19:28:33.911550+0000","last_clean":"2023-10-10T19:28:33.911550+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T19:28:33.911550+0000","last_undegraded":"2023-10-10T19:28:33.911550+0000","last_fullsized":"2023-10-10T19:28:33.911550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:25:40.597520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194858,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":28,"num_read_kb":40985,"num_write":233,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"132'919","reported_seq":3569,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.166522+0000","last_change":"2023-10-10T19:16:54.794460+0000","last_active":"2023-10-10T19:29:22.166522+0000","last_peered":"2023-10-10T19:29:22.166522+0000","last_clean":"2023-10-10T19:29:22.166522+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T19:29:22.166522+0000","last_undegraded":"2023-10-10T19:29:22.166522+0000","last_fullsized":"2023-10-10T19:29:22.166522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:48:41.118761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016889265000000001,"stat_sum":{"num_bytes":1346662,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":67,"num_read_kb":24822,"num_write":699,"num_write_kb":32529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"132'290","reported_seq":434,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.978567+0000","last_change":"2023-10-10T18:45:10.405116+0000","last_active":"2023-10-10T19:29:22.978567+0000","last_peered":"2023-10-10T19:29:22.978567+0000","last_clean":"2023-10-10T19:29:22.978567+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T19:29:22.978567+0000","last_undegraded":"2023-10-10T19:29:22.978567+0000","last_fullsized":"2023-10-10T19:29:22.978567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:27:54.069168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9558051,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":27,"num_read_kb":50704,"num_write":220,"num_write_kb":13635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"132'970","reported_seq":3597,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.155104+0000","last_change":"2023-10-10T19:16:55.417128+0000","last_active":"2023-10-10T19:29:22.155104+0000","last_peered":"2023-10-10T19:29:22.155104+0000","last_clean":"2023-10-10T19:29:22.155104+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T19:29:22.155104+0000","last_undegraded":"2023-10-10T19:29:22.155104+0000","last_fullsized":"2023-10-10T19:29:22.155104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:21:13.451660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012585610000000001,"stat_sum":{"num_bytes":222806,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":60,"num_read_kb":19964,"num_write":709,"num_write_kb":27205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"128'310","reported_seq":458,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-10T19:27:29.683824+0000","last_change":"2023-10-10T18:45:10.409296+0000","last_active":"2023-10-10T19:27:29.683824+0000","last_peered":"2023-10-10T19:27:29.683824+0000","last_clean":"2023-10-10T19:27:29.683824+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T19:27:29.683824+0000","last_undegraded":"2023-10-10T19:27:29.683824+0000","last_fullsized":"2023-10-10T19:27:29.683824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:11:02.506141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":49,"num_read_kb":57404,"num_write":260,"num_write_kb":8464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"132'1015","reported_seq":3644,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.160376+0000","last_change":"2023-10-10T19:16:54.794658+0000","last_active":"2023-10-10T19:29:22.160376+0000","last_peered":"2023-10-10T19:29:22.160376+0000","last_clean":"2023-10-10T19:29:22.160376+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T19:29:22.160376+0000","last_undegraded":"2023-10-10T19:29:22.160376+0000","last_fullsized":"2023-10-10T19:29:22.160376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1015,"log_dups_size":0,"ondisk_log_size":1015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:13.614065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016837566000000002,"stat_sum":{"num_bytes":394703,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":67,"num_read_kb":28062,"num_write":753,"num_write_kb":23722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"132'910","reported_seq":3570,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.149188+0000","last_change":"2023-10-10T19:16:54.782474+0000","last_active":"2023-10-10T19:29:22.149188+0000","last_peered":"2023-10-10T19:29:22.149188+0000","last_clean":"2023-10-10T19:29:22.149188+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T19:29:22.149188+0000","last_undegraded":"2023-10-10T19:29:22.149188+0000","last_fullsized":"2023-10-10T19:29:22.149188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:57:43.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074078640000000001,"stat_sum":{"num_bytes":519476,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":77,"num_read_kb":25056,"num_write":683,"num_write_kb":32109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"132'993","reported_seq":3614,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162607+0000","last_change":"2023-10-10T19:16:54.779827+0000","last_active":"2023-10-10T19:29:22.162607+0000","last_peered":"2023-10-10T19:29:22.162607+0000","last_clean":"2023-10-10T19:29:22.162607+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T19:29:22.162607+0000","last_undegraded":"2023-10-10T19:29:22.162607+0000","last_fullsized":"2023-10-10T19:29:22.162607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:19:39.641720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050938800000000003,"stat_sum":{"num_bytes":416129,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":69,"num_read_kb":25013,"num_write":702,"num_write_kb":37226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"132'973","reported_seq":3620,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.139137+0000","last_change":"2023-10-10T19:16:54.781092+0000","last_active":"2023-10-10T19:29:22.139137+0000","last_peered":"2023-10-10T19:29:22.139137+0000","last_clean":"2023-10-10T19:29:22.139137+0000","last_became_active":"2023-10-10T18:46:00.606420+0000","last_became_peered":"2023-10-10T18:46:00.606420+0000","last_unstale":"2023-10-10T19:29:22.139137+0000","last_undegraded":"2023-10-10T19:29:22.139137+0000","last_fullsized":"2023-10-10T19:29:22.139137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:47.189801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064370249999999999,"stat_sum":{"num_bytes":103727,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":90,"num_read_kb":45570,"num_write":716,"num_write_kb":52505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"132'946","reported_seq":3572,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157700+0000","last_change":"2023-10-10T19:16:54.983512+0000","last_active":"2023-10-10T19:29:22.157700+0000","last_peered":"2023-10-10T19:29:22.157700+0000","last_clean":"2023-10-10T19:29:22.157700+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T19:29:22.157700+0000","last_undegraded":"2023-10-10T19:29:22.157700+0000","last_fullsized":"2023-10-10T19:29:22.157700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:45:03.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014096529,"stat_sum":{"num_bytes":456616,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":70,"num_read_kb":16093,"num_write":668,"num_write_kb":20834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"132'918","reported_seq":3555,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157838+0000","last_change":"2023-10-10T19:16:54.875626+0000","last_active":"2023-10-10T19:29:22.157838+0000","last_peered":"2023-10-10T19:29:22.157838+0000","last_clean":"2023-10-10T19:29:22.157838+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T19:29:22.157838+0000","last_undegraded":"2023-10-10T19:29:22.157838+0000","last_fullsized":"2023-10-10T19:29:22.157838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:16:12.851840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016808101999999998,"stat_sum":{"num_bytes":708082,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":68,"num_read_kb":48841,"num_write":679,"num_write_kb":56033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"132'979","reported_seq":3581,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.143919+0000","last_change":"2023-10-10T19:16:54.918525+0000","last_active":"2023-10-10T19:29:22.143919+0000","last_peered":"2023-10-10T19:29:22.143919+0000","last_clean":"2023-10-10T19:29:22.143919+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T19:29:22.143919+0000","last_undegraded":"2023-10-10T19:29:22.143919+0000","last_fullsized":"2023-10-10T19:29:22.143919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":979,"log_dups_size":0,"ondisk_log_size":979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:42.298640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068654069999999996,"stat_sum":{"num_bytes":747011,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":59,"num_read_kb":41116,"num_write":675,"num_write_kb":48421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"132'978","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.145608+0000","last_change":"2023-10-10T19:16:54.783685+0000","last_active":"2023-10-10T19:29:22.145608+0000","last_peered":"2023-10-10T19:29:22.145608+0000","last_clean":"2023-10-10T19:29:22.145608+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T19:29:22.145608+0000","last_undegraded":"2023-10-10T19:29:22.145608+0000","last_fullsized":"2023-10-10T19:29:22.145608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:55.831082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081260870000000006,"stat_sum":{"num_bytes":4820268,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":70,"num_read_kb":15871,"num_write":698,"num_write_kb":28703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"132'933","reported_seq":3569,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.141176+0000","last_change":"2023-10-10T19:16:54.877012+0000","last_active":"2023-10-10T19:29:22.141176+0000","last_peered":"2023-10-10T19:29:22.141176+0000","last_clean":"2023-10-10T19:29:22.141176+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T19:29:22.141176+0000","last_undegraded":"2023-10-10T19:29:22.141176+0000","last_fullsized":"2023-10-10T19:29:22.141176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:20.354520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017981724000000001,"stat_sum":{"num_bytes":834726,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":72,"num_read_kb":44513,"num_write":669,"num_write_kb":47263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"132'911","reported_seq":3554,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.161844+0000","last_change":"2023-10-10T19:16:55.415035+0000","last_active":"2023-10-10T19:29:22.161844+0000","last_peered":"2023-10-10T19:29:22.161844+0000","last_clean":"2023-10-10T19:29:22.161844+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T19:29:22.161844+0000","last_undegraded":"2023-10-10T19:29:22.161844+0000","last_fullsized":"2023-10-10T19:29:22.161844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:51:49.094069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010756557999999999,"stat_sum":{"num_bytes":807796,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":71,"num_read_kb":21097,"num_write":685,"num_write_kb":25593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"132'896","reported_seq":3533,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.154243+0000","last_change":"2023-10-10T19:16:54.872190+0000","last_active":"2023-10-10T19:29:22.154243+0000","last_peered":"2023-10-10T19:29:22.154243+0000","last_clean":"2023-10-10T19:29:22.154243+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T19:29:22.154243+0000","last_undegraded":"2023-10-10T19:29:22.154243+0000","last_fullsized":"2023-10-10T19:29:22.154243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:12:58.541681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014012482999999999,"stat_sum":{"num_bytes":684647,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":69,"num_read_kb":49258,"num_write":638,"num_write_kb":56905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"132'959","reported_seq":3592,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.338185+0000","last_change":"2023-10-10T19:16:55.413906+0000","last_active":"2023-10-10T19:29:22.338185+0000","last_peered":"2023-10-10T19:29:22.338185+0000","last_clean":"2023-10-10T19:29:22.338185+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T19:29:22.338185+0000","last_undegraded":"2023-10-10T19:29:22.338185+0000","last_fullsized":"2023-10-10T19:29:22.338185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:54.424365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097177140000000006,"stat_sum":{"num_bytes":571513,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":59,"num_read_kb":20609,"num_write":714,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"132'887","reported_seq":3546,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.166117+0000","last_change":"2023-10-10T19:16:54.983331+0000","last_active":"2023-10-10T19:29:22.166117+0000","last_peered":"2023-10-10T19:29:22.166117+0000","last_clean":"2023-10-10T19:29:22.166117+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T19:29:22.166117+0000","last_undegraded":"2023-10-10T19:29:22.166117+0000","last_fullsized":"2023-10-10T19:29:22.166117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:22.969408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013935637000000001,"stat_sum":{"num_bytes":366510,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":77,"num_read_kb":51992,"num_write":664,"num_write_kb":56578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"132'1013","reported_seq":3655,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157580+0000","last_change":"2023-10-10T19:16:54.784919+0000","last_active":"2023-10-10T19:29:22.157580+0000","last_peered":"2023-10-10T19:29:22.157580+0000","last_clean":"2023-10-10T19:29:22.157580+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T19:29:22.157580+0000","last_undegraded":"2023-10-10T19:29:22.157580+0000","last_fullsized":"2023-10-10T19:29:22.157580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:19:52.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088155260000000006,"stat_sum":{"num_bytes":278227,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":77,"num_read_kb":37312,"num_write":744,"num_write_kb":49182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":50927477,"num_objects":2366,"num_object_clones":0,"num_object_copies":4732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":4408,"num_read_kb":1953419,"num_write":44061,"num_write_kb":2517898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":76832768,"data_stored":101854954,"data_compressed":30508016,"data_compressed_allocated":33341440,"data_compressed_original":70082560,"omap_allocated":0,"internal_metadata":0},"log_size":60751,"ondisk_log_size":60751,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":332882973,"num_objects":493,"num_object_clones":0,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":2858,"num_read_kb":1531013,"num_write":19080,"num_write_kb":465992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":354,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181026816,"data_stored":665765946,"data_compressed":86138570,"data_compressed_allocated":110837760,"data_compressed_original":595820544,"omap_allocated":0,"internal_metadata":0},"log_size":19720,"ondisk_log_size":19720,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575384,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348568,"kb_used_data":47184,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94023272,"statfs":{"total":96636764160,"available":96279830528,"internally_reserved":0,"allocated":48316416,"data_stored":133510727,"data_compressed":20347876,"data_compressed_allocated":25333760,"data_compressed_original":112963584,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.53700000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.78400000000000003,"15min":1.018},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.32100000000000001,"5min":0.316,"15min":0.25},"max":{"1min":0.76500000000000001,"5min":0.80700000000000005,"15min":2.153},"last":0.39800000000000002}]},{"osd":1,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.33100000000000002,"5min":0.318,"15min":0.26200000000000001},"max":{"1min":0.79500000000000004,"5min":0.82499999999999996,"15min":1.802},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":2.2170000000000001},"last":0.46999999999999997}]},{"osd":2,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.623,"15min":0.61199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83899999999999997,"5min":2.871,"15min":2.871},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63400000000000001,"15min":0.63800000000000001},"min":{"1min":0.48299999999999998,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":7.6200000000000001},"last":0.57799999999999996}]},{"osd":3,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.874,"15min":1.0349999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.60099999999999998},"min":{"1min":0.375,"5min":0.373,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":7.6349999999999998},"last":0.44900000000000001}]},{"osd":4,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.36599999999999999,"5min":0.307,"15min":0.28000000000000003},"max":{"1min":0.64000000000000001,"5min":0.72499999999999998,"15min":1.089},"last":0.436},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.33800000000000002,"5min":0.32600000000000001,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.9419999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":0.629,"5min":0.752,"15min":1.71},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59299999999999997,"15min":0.57199999999999995},"min":{"1min":0.41799999999999998,"5min":0.378,"15min":0.311},"max":{"1min":0.80500000000000005,"5min":0.85099999999999998,"15min":1.177},"last":0.627}]},{"osd":6,"last update":"Tue Oct 10 19:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.31},"max":{"1min":0.76100000000000001,"5min":0.78000000000000003,"15min":0.872},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.33900000000000002,"15min":0.29699999999999999},"max":{"1min":0.72299999999999998,"5min":0.78100000000000003,"15min":2.2559999999999998},"last":0.52200000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575383,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333252,"kb_used_data":35964,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94038588,"statfs":{"total":96636764160,"available":96295514112,"internally_reserved":0,"allocated":36827136,"data_stored":91571492,"data_compressed":11851024,"data_compressed_allocated":14680064,"data_compressed_original":71368704,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.31},"max":{"1min":0.748,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.435},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.216},"max":{"1min":0.71799999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.41899999999999998}]},{"osd":1,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.77200000000000002,"5min":0.77600000000000002,"15min":0.85299999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.58399999999999996,"15min":0.56599999999999995},"min":{"1min":0.46700000000000003,"5min":0.29599999999999999,"15min":0.222},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86799999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Tue Oct 10 19:28:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58199999999999996,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.312,"15min":0.30499999999999999},"max":{"1min":0.69899999999999995,"5min":0.77800000000000002,"15min":0.90500000000000003},"last":0.624},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56899999999999995,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29999999999999999},"max":{"1min":0.71299999999999997,"5min":0.76500000000000001,"15min":0.97599999999999998},"last":0.45900000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57399999999999995,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76600000000000001,"5min":0.82799999999999996,"15min":0.88200000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.35999999999999999},"max":{"1min":0.73599999999999999,"5min":0.85699999999999998,"15min":0.95099999999999996},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.46400000000000002},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.248},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.52500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59799999999999998}]},{"osd":5,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.73599999999999999,"15min":0.879},"last":0.503},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.39700000000000002,"5min":0.30599999999999999,"15min":0.26400000000000001},"max":{"1min":0.63100000000000001,"5min":0.76800000000000002,"15min":0.99299999999999999},"last":0.61099999999999999}]},{"osd":7,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.38200000000000001,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.88900000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.38,"5min":0.376,"15min":0.27700000000000002},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.86199999999999999},"last":0.58299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575382,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341836,"kb_used_data":32260,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94030004,"statfs":{"total":96636764160,"available":96286724096,"internally_reserved":0,"allocated":33034240,"data_stored":115017315,"data_compressed":17553054,"data_compressed_allocated":21581824,"data_compressed_original":105316352,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.314,"15min":0.313},"min":{"1min":0.187,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.432,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.66700000000000004,"15min":0.71699999999999997},"last":0.28499999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34599999999999997,"15min":0.35199999999999998},"min":{"1min":0.23799999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55200000000000005},"last":0.246},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25,"15min":0.19900000000000001},"max":{"1min":0.54000000000000004,"5min":0.64900000000000002,"15min":0.73899999999999999},"last":0.27000000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.45300000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.32600000000000001,"15min":0.27700000000000002},"max":{"1min":0.59699999999999998,"5min":0.72899999999999998,"15min":0.84299999999999997},"last":0.308}]},{"osd":4,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":1.175},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58699999999999997,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.87,"5min":0.87,"15min":2.2519999999999998},"last":0.45900000000000002}]},{"osd":5,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.76700000000000002,"15min":0.90900000000000003},"last":0.501},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53000000000000003}]},{"osd":6,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.40500000000000003,"15min":0.37},"max":{"1min":0.83199999999999996,"5min":0.86199999999999999,"15min":1.038},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.123},"last":0.45200000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60299999999999998,"15min":0.62},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":5.2720000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":5.2599999999999998},"last":0.47999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575384,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338068,"kb_used_data":32588,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94033772,"statfs":{"total":96636764160,"available":96290582528,"internally_reserved":0,"allocated":33370112,"data_stored":88805479,"data_compressed":14490668,"data_compressed_allocated":17825792,"data_compressed_original":75632640,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.35799999999999998,"15min":0.35699999999999998},"min":{"1min":0.245,"5min":0.19700000000000001,"15min":0.19600000000000001},"max":{"1min":0.47699999999999998,"5min":0.57399999999999995,"15min":0.66500000000000004},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42499999999999999,"15min":0.41999999999999998},"min":{"1min":0.379,"5min":0.25800000000000001,"15min":0.20300000000000001},"max":{"1min":0.60299999999999998,"5min":0.66000000000000003,"15min":0.66500000000000004},"last":0.33900000000000002}]},{"osd":2,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35999999999999999,"15min":0.373},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.49299999999999999,"5min":0.57799999999999996,"15min":2.028},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54900000000000004,"5min":0.61299999999999999,"15min":0.77300000000000002},"last":0.29999999999999999}]},{"osd":3,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":0.54000000000000004,"5min":0.59699999999999998,"15min":0.79100000000000004},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.32500000000000001,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":0.70499999999999996,"5min":0.70599999999999996,"15min":0.80300000000000005},"last":0.318}]},{"osd":4,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.373,"15min":0.26000000000000001},"max":{"1min":0.80200000000000005,"5min":0.97599999999999998,"15min":1.9610000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.38400000000000001},"max":{"1min":0.82599999999999996,"5min":0.999,"15min":0.999},"last":0.65700000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.55100000000000005,"5min":0.39000000000000001,"15min":0.35299999999999998},"max":{"1min":0.79600000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.628},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.626,"15min":0.623},"min":{"1min":0.51600000000000001,"5min":0.38600000000000001,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.873,"15min":0.96799999999999997},"last":0.498}]},{"osd":6,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.52200000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":0.75800000000000001,"5min":0.96299999999999997,"15min":2.0019999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.621,"15min":0.623},"min":{"1min":0.501,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":0.84999999999999998,"5min":0.94099999999999995,"15min":0.95399999999999996},"last":0.63700000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.40400000000000003,"15min":0.39100000000000001},"max":{"1min":0.77000000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41799999999999998},"max":{"1min":0.879,"5min":0.89500000000000002,"15min":2.0539999999999998},"last":0.73499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575384,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319288,"kb_used_data":17904,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94052552,"statfs":{"total":96636764160,"available":96309813248,"internally_reserved":0,"allocated":18333696,"data_stored":57320502,"data_compressed":8516380,"data_compressed_allocated":10579968,"data_compressed_original":51261440,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34399999999999997,"15min":0.33500000000000002},"min":{"1min":0.224,"5min":0.219,"15min":0.186},"max":{"1min":0.45000000000000001,"5min":0.55400000000000005,"15min":0.60799999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.53500000000000003,"5min":0.65100000000000002,"15min":0.67500000000000004},"last":0.41999999999999998}]},{"osd":1,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20999999999999999},"max":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.65700000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.73999999999999999},"last":0.443}]},{"osd":3,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.24399999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.42499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.23899999999999999},"max":{"1min":0.59099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.40300000000000002}]},{"osd":4,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.76900000000000002},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.26600000000000001},"max":{"1min":0.751,"5min":0.95599999999999996,"15min":48.729999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.25900000000000001},"max":{"1min":0.74099999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.69499999999999995}]},{"osd":5,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63,"15min":0.628},"min":{"1min":0.436,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":1.02,"15min":1.02},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.32400000000000001},"max":{"1min":0.81000000000000005,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.73299999999999998}]},{"osd":6,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.78900000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":1.0760000000000001,"15min":48.685000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.38800000000000001},"max":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":0.97899999999999998},"last":0.75600000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64800000000000002,"15min":0.80700000000000005},"min":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.38700000000000001},"max":{"1min":0.85799999999999998,"5min":1.052,"15min":48.710000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.034,"15min":1.1459999999999999},"last":0.71099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575383,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347532,"kb_used_data":37956,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94024308,"statfs":{"total":96636764160,"available":96280891392,"internally_reserved":0,"allocated":38866944,"data_stored":98664257,"data_compressed":13826517,"data_compressed_allocated":17530880,"data_compressed_original":79802368,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.37},"min":{"1min":0.23499999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.47599999999999998,"5min":0.64900000000000002,"15min":5.0670000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.371,"15min":0.38700000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.189},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.7799999999999998},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.36499999999999999,"15min":0.377},"min":{"1min":0.27200000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.628,"5min":0.628,"15min":2.758},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.41199999999999998},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51700000000000002,"5min":0.58399999999999996,"15min":0.64500000000000002},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28000000000000003,"15min":0.25},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":1.333},"last":0.316},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42099999999999999,"15min":0.41899999999999998},"min":{"1min":0.30599999999999999,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.98699999999999999},"last":0.32800000000000001}]},{"osd":4,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.48299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.879,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46200000000000002,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.23},"last":0.59299999999999997}]},{"osd":5,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.45100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.2929999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.36199999999999999},"max":{"1min":0.90200000000000002,"5min":0.93600000000000005,"15min":1.103},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63500000000000001,"15min":0.64000000000000001},"min":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.39900000000000002},"max":{"1min":0.91000000000000003,"5min":0.94199999999999995,"15min":1.198},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.49099999999999999,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.94899999999999995,"5min":0.95199999999999996,"15min":2.7309999999999999},"last":0.627}]},{"osd":7,"last update":"Tue Oct 10 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.46700000000000003,"5min":0.46500000000000002,"15min":0.39400000000000002},"max":{"1min":0.85099999999999998,"5min":0.97999999999999998,"15min":2.6619999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67900000000000005,"15min":0.66900000000000004},"min":{"1min":0.50800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.60299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575383,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340172,"kb_used_data":34692,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94031668,"statfs":{"total":96636764160,"available":96288428032,"internally_reserved":0,"allocated":35524608,"data_stored":103891535,"data_compressed":15969610,"data_compressed_allocated":19587072,"data_compressed_original":90329088,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56100000000000005,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.21199999999999999},"max":{"1min":0.746,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.23699999999999999},"max":{"1min":0.71899999999999997,"5min":0.86799999999999999,"15min":0.88900000000000001},"last":0.48999999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.499,"15min":0.51600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":0.68300000000000005,"5min":0.71199999999999997,"15min":1.5920000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.55800000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.69899999999999995,"5min":0.749,"15min":1.0449999999999999},"last":0.42299999999999999}]},{"osd":2,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.30199999999999999},"max":{"1min":0.752,"5min":0.78200000000000003,"15min":1.615},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.36699999999999999},"max":{"1min":0.73799999999999999,"5min":0.85799999999999998,"15min":1.071},"last":0.53700000000000003}]},{"osd":3,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.58099999999999996},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.76200000000000001,"5min":0.85199999999999998,"15min":0.85799999999999998},"last":0.376},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.28199999999999997},"max":{"1min":0.81799999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.624}]},{"osd":5,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":1.087},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56699999999999995,"15min":0.55500000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88,"5min":0.88,"15min":1.111},"last":0.60299999999999998}]},{"osd":6,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56799999999999995,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.29799999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.5580000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":0.90400000000000003,"15min":0.98799999999999999},"last":0.65800000000000003}]},{"osd":7,"last update":"Tue Oct 10 19:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.312},"max":{"1min":1.016,"5min":1.016,"15min":1.0620000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64900000000000002,"15min":0.64200000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.33700000000000002},"max":{"1min":0.873,"5min":0.873,"15min":1.1160000000000001},"last":0.67900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575383,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331220,"kb_used_data":25740,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94040620,"statfs":{"total":96636764160,"available":96297594880,"internally_reserved":0,"allocated":26357760,"data_stored":87096107,"data_compressed":14155985,"data_compressed_allocated":17149952,"data_compressed_original":80375808,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.438,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":0.69499999999999995,"5min":0.75,"15min":0.82199999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56499999999999995,"15min":0.59699999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.63800000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.41699999999999998}]},{"osd":1,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57599999999999996,"15min":0.55900000000000005},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.29699999999999999},"max":{"1min":0.745,"5min":0.79400000000000004,"15min":0.79800000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.56999999999999995},"min":{"1min":0.48199999999999998,"5min":0.39200000000000002,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.38}]},{"osd":2,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.54000000000000004},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.30499999999999999},"max":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.39500000000000002,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.74099999999999999,"5min":0.75600000000000001,"15min":0.79000000000000004},"last":0.39000000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.86099999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59499999999999997,"15min":0.58799999999999997},"min":{"1min":0.46899999999999997,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.81799999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.42799999999999999}]},{"osd":4,"last update":"Tue Oct 10 19:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.45700000000000002},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.251},"max":{"1min":0.69999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58099999999999996,"15min":0.55400000000000005},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.23000000000000001},"max":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.34499999999999997}]},{"osd":6,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.497},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.26700000000000002},"max":{"1min":0.72199999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.316},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53800000000000003},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.32100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.30299999999999999}]},{"osd":7,"last update":"Tue Oct 10 19:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76600000000000001,"5min":0.86299999999999999,"15min":0.89400000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.27700000000000002},"max":{"1min":0.80500000000000005,"5min":0.84599999999999997,"15min":0.90800000000000003},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20922368,"data_stored":71948432,"data_compressed":9641340,"data_compressed_allocated":12496896,"data_compressed_original":63553536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25231360,"data_stored":104270883,"data_compressed":13594240,"data_compressed_allocated":17289216,"data_compressed_original":96358400,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":51052075,"data_compressed":6450195,"data_compressed_allocated":8323072,"data_compressed_original":46788608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25874432,"data_stored":83722152,"data_compressed":10192717,"data_compressed_allocated":13520896,"data_compressed_original":71401472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24227840,"data_stored":90229609,"data_compressed":11987552,"data_compressed_allocated":15253504,"data_compressed_original":81281024,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16494592,"data_stored":73848315,"data_compressed":9635192,"data_compressed_allocated":12173312,"data_compressed_original":69566464,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21950464,"data_stored":75911039,"data_compressed":9436899,"data_compressed_allocated":12021760,"data_compressed_original":66007040,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":33710080,"data_stored":114783441,"data_compressed":15200435,"data_compressed_allocated":19759104,"data_compressed_original":100864000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10801152,"data_stored":15380743,"data_compressed":4817064,"data_compressed_allocated":5283840,"data_compressed_original":11505664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6225920,"data_stored":9866351,"data_compressed":3958814,"data_compressed_allocated":4292608,"data_compressed_original":8957952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4136960,"data_stored":5382491,"data_compressed":2066185,"data_compressed_allocated":2256896,"data_compressed_original":4472832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11411456,"data_stored":14056169,"data_compressed":3633800,"data_compressed_allocated":4009984,"data_compressed_original":8400896,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9719808,"data_stored":12781845,"data_compressed":3982058,"data_compressed_allocated":4333568,"data_compressed_original":9048064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8282112,"data_stored":12361856,"data_compressed":4520793,"data_compressed_allocated":4976640,"data_compressed_original":10809344,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13295616,"data_stored":14774517,"data_compressed":2414125,"data_compressed_allocated":2658304,"data_compressed_original":5361664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12959744,"data_stored":17250982,"data_compressed":5115177,"data_compressed_allocated":5529600,"data_compressed_original":11526144,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T19:29:29.601 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.602 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.602 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.602 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.603 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.603 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.603 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.604 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.604 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.605 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.605 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.605 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.606 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.606 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.607 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.607 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.607 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.608 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.608 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.608 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.609 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.609 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.609 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.610 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.610 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.611 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.611 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.611 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.612 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.612 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.612 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.613 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.613 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.614 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.614 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.614 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.615 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.615 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.616 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.616 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.616 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.617 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.617 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.617 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.618 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.618 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.619 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.619 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.619 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.620 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.620 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.620 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.621 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.621 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.622 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.622 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.623 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.623 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.623 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.624 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.624 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.625 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-10T18:45:06.056019+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=6, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.625 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.625 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.626 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.626 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.626 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.627 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.627 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.628 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.628 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.628 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.629 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.629 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.629 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.630 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.630 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.631 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.631 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.631 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.632 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.632 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.633 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.633 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.633 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.634 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.634 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.635 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.635 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.635 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.636 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.636 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.636 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.637 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.637 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.638 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.638 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:29.638 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-10T19:29:49.639 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T19:29:49.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.765+0000 7fb003475700 1 Processor -- start 2023-10-10T19:29:49.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.765+0000 7fb003475700 1 -- start start 2023-10-10T19:29:49.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb003475700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faffc0b2390 0x7faffc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb003475700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc0aff60 0x7faffc0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb003475700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc1574b0 con 0x7faffc0b1fc0 2023-10-10T19:29:49.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb003475700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc150770 con 0x7faffc0b2390 2023-10-10T19:29:49.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb003475700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0afad0 con 0x7faffc0aff60 2023-10-10T19:29:49.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb001211700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7faffc0b1fc0 0x7faffc068690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:48806/0 (socket says 172.21.15.175:48806) 2023-10-10T19:29:49.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb001211700 1 -- 172.21.15.175:0/162284599 learned_addr learned my addr 172.21.15.175:0/162284599 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:29:49.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb000a10700 1 --2- 172.21.15.175:0/162284599 >> v2:172.21.15.191:3300/0 conn(0x7faffc0b2390 0x7faffc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7fb001a12700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc0aff60 0x7faffc0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083854122 0 0) 0x7faffc1574b0 con 0x7faffc0b1fc0 2023-10-10T19:29:49.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe8003570 con 0x7faffc0b1fc0 2023-10-10T19:29:49.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154864199 0 0) 0x7fafe8003570 con 0x7faffc0b1fc0 2023-10-10T19:29:49.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc0aff60 msgr2=0x7faffc0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:49.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc0aff60 0x7faffc0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 >> v2:172.21.15.191:3300/0 conn(0x7faffc0b2390 msgr2=0x7faffc155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:49.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 --2- 172.21.15.175:0/162284599 >> v2:172.21.15.191:3300/0 conn(0x7faffc0b2390 0x7faffc155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc0af4c0 con 0x7faffc0b1fc0 2023-10-10T19:29:49.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 <== mon.0 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2336585211 0 0) 0x7faff4001470 con 0x7faffc0b1fc0 2023-10-10T19:29:49.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 >> v1:172.21.15.175:6789/0 conn(0x7faffc0b1fc0 legacy=0x7faffc068690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:49.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7fafe8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafe8004810 0x7fafe8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc0aff60 0x7fafe80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc150770 con 0x7fafe8003a60 2023-10-10T19:29:49.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0afad0 con 0x7fafe8004810 2023-10-10T19:29:49.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.766+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe800ba60 con 0x7faffc0aff60 2023-10-10T19:29:49.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb001211700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7fafe8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb000a10700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafe8004810 0x7fafe8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.781 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb001a12700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc0aff60 0x7fafe80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb001211700 1 -- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc0aff60 msgr2=0x7fafe80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:49.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb001211700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc0aff60 0x7fafe80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb001211700 1 -- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafe8004810 msgr2=0x7fafe8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:29:49.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb001211700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafe8004810 0x7fafe8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb001211700 1 -- 172.21.15.175:0/162284599 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe8003570 con 0x7fafe8003a60 2023-10-10T19:29:49.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7fb001211700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7fafe8003e70 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7faff400c3f0 tx=0x7faffc0712f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a946ad8a19657d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:49.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff400cc80 con 0x7fafe8003a60 2023-10-10T19:29:49.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff400cde0 con 0x7fafe8003a60 2023-10-10T19:29:49.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.767+0000 7faff3fff700 1 -- 172.21.15.175:0/162284599 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4005b70 con 0x7fafe8003a60 2023-10-10T19:29:49.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.768+0000 7fb003475700 1 -- 172.21.15.175:0/162284599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 msgr2=0x7fafe8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:49.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.768+0000 7fb003475700 1 --2- 172.21.15.175:0/162284599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7fafe8003e70 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7faff400c3f0 tx=0x7faffc0712f0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.768+0000 7fb001a12700 1 -- 172.21.15.175:0/162284599 reap_dead start 2023-10-10T19:29:49.785 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.768+0000 7fb003475700 1 -- 172.21.15.175:0/162284599 shutdown_connections 2023-10-10T19:29:49.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.768+0000 7fb003475700 1 -- 172.21.15.175:0/162284599 >> 172.21.15.175:0/162284599 conn(0x7faffc063230 msgr2=0x7faffc063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:49.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.768+0000 7fb003475700 1 -- 172.21.15.175:0/162284599 shutdown_connections 2023-10-10T19:29:49.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.768+0000 7fb003475700 1 -- 172.21.15.175:0/162284599 wait complete. 2023-10-10T19:29:49.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.769+0000 7fb003475700 1 Processor -- start 2023-10-10T19:29:49.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.769+0000 7fb003475700 1 -- start start 2023-10-10T19:29:49.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb003475700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7faffc0c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb003475700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faffc155e30 0x7faffc0c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb003475700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc169100 0x7faffc169570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb003475700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0cd5d0 con 0x7fafe8003a60 2023-10-10T19:29:49.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb003475700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc14f900 con 0x7faffc155e30 2023-10-10T19:29:49.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb003475700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0cc130 con 0x7faffc169100 2023-10-10T19:29:49.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7faffc0c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7faffc0c4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:57614/0 (socket says 172.21.15.175:57614) 2023-10-10T19:29:49.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 -- 172.21.15.175:0/2059611730 learned_addr learned my addr 172.21.15.175:0/2059611730 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:29:49.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb000a10700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faffc155e30 0x7faffc0c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001a12700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc169100 0x7faffc169570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 -- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc169100 msgr2=0x7faffc169570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:49.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc169100 0x7faffc169570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 -- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faffc155e30 msgr2=0x7faffc0c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:49.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faffc155e30 0x7faffc0c4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 -- 172.21.15.175:0/2059611730 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc0cccb0 con 0x7fafe8003a60 2023-10-10T19:29:49.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001a12700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc169100 0x7faffc169570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:29:49.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb000a10700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faffc155e30 0x7faffc0c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:29:49.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb001211700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7faffc0c4860 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7faff400c9e0 tx=0x7faff4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:49.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7faff17fa700 1 -- 172.21.15.175:0/2059611730 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff400cc80 con 0x7fafe8003a60 2023-10-10T19:29:49.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7faff17fa700 1 -- 172.21.15.175:0/2059611730 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff40072e0 con 0x7fafe8003a60 2023-10-10T19:29:49.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7faff17fa700 1 -- 172.21.15.175:0/2059611730 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40076c0 con 0x7fafe8003a60 2023-10-10T19:29:49.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.770+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc156300 con 0x7fafe8003a60 2023-10-10T19:29:49.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.771+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc0cc660 con 0x7fafe8003a60 2023-10-10T19:29:49.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.771+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc0ccf90 con 0x7fafe8003a60 2023-10-10T19:29:49.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.772+0000 7faff17fa700 1 -- 172.21.15.175:0/2059611730 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faff4019030 con 0x7fafe8003a60 2023-10-10T19:29:49.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.772+0000 7faff17fa700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafe0066f30 0x7fafe00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:29:49.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.772+0000 7faff17fa700 1 -- 172.21.15.175:0/2059611730 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7faff4086100 con 0x7fafe8003a60 2023-10-10T19:29:49.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.772+0000 7fb000a10700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafe0066f30 0x7fafe00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:29:49.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.773+0000 7fb000a10700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafe0066f30 0x7fafe00693e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fafec006dc0 tx=0x7fafec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:29:49.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.777+0000 7faff17fa700 1 -- 172.21.15.175:0/2059611730 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faff4052af0 con 0x7fafe8003a60 2023-10-10T19:29:49.958 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.957+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faffc0cd5d0 con 0x7fafe0066f30 2023-10-10T19:29:49.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.965+0000 7faff17fa700 1 -- 172.21.15.175:0/2059611730 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294310 (secure 0 0 0) 0x7faffc0cd5d0 con 0x7fafe0066f30 2023-10-10T19:29:49.971 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:29:49.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafe0066f30 msgr2=0x7fafe00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:49.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafe0066f30 0x7fafe00693e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fafec006dc0 tx=0x7fafec008040 comp rx=0 tx=0).stop 2023-10-10T19:29:49.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 msgr2=0x7faffc0c4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:29:49.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7faffc0c4860 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7faff400c9e0 tx=0x7faff4015000 comp rx=0 tx=0).stop 2023-10-10T19:29:49.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 shutdown_connections 2023-10-10T19:29:49.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fafe0066f30 0x7fafe00693e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faffc169100 0x7faffc169570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafe8003a60 0x7faffc0c4860 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 --2- 172.21.15.175:0/2059611730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faffc155e30 0x7faffc0c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:29:49.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 >> 172.21.15.175:0/2059611730 conn(0x7faffc063230 msgr2=0x7faffc154fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:29:49.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 shutdown_connections 2023-10-10T19:29:49.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:29:49.969+0000 7fb003475700 1 -- 172.21.15.175:0/2059611730 wait complete. 2023-10-10T19:29:49.977 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T19:29:49.982 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":1468,"stamp":"2023-10-10T19:29:49.824597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388268704,"num_objects":2861,"num_object_clones":0,"num_object_copies":5722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":7382,"num_read_kb":3484532,"num_write":63409,"num_write_kb":2990121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":353,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2328994,"num_omap_keys":4066,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81583,"ondisk_log_size":81583,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2706208,"kb_used_data":266464,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752268512,"statfs":{"total":773094113280,"available":770322956288,"internally_reserved":0,"allocated":272859136,"data_stored":783624896,"data_compressed":117819950,"data_compressed_allocated":145743872,"data_compressed_original":674045952,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5503,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":818982,"num_omap_keys":1417,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"3.3f","version":"132'966","reported_seq":3587,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.164884+0000","last_change":"2023-10-10T19:16:54.796013+0000","last_active":"2023-10-10T19:29:22.164884+0000","last_peered":"2023-10-10T19:29:22.164884+0000","last_clean":"2023-10-10T19:29:22.164884+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T19:29:22.164884+0000","last_undegraded":"2023-10-10T19:29:22.164884+0000","last_fullsized":"2023-10-10T19:29:22.164884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017995507000000001,"stat_sum":{"num_bytes":821890,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":62,"num_read_kb":8068,"num_write":683,"num_write_kb":16759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"132'1012","reported_seq":3632,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.148036+0000","last_change":"2023-10-10T19:16:54.920511+0000","last_active":"2023-10-10T19:29:22.148036+0000","last_peered":"2023-10-10T19:29:22.148036+0000","last_clean":"2023-10-10T19:29:22.148036+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T19:29:22.148036+0000","last_undegraded":"2023-10-10T19:29:22.148036+0000","last_fullsized":"2023-10-10T19:29:22.148036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083569260000000006,"stat_sum":{"num_bytes":1412883,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":62,"num_read_kb":28805,"num_write":746,"num_write_kb":46678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"132'962","reported_seq":3590,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.148281+0000","last_change":"2023-10-10T19:16:54.980353+0000","last_active":"2023-10-10T19:29:22.148281+0000","last_peered":"2023-10-10T19:29:22.148281+0000","last_clean":"2023-10-10T19:29:22.148281+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T19:29:22.148281+0000","last_undegraded":"2023-10-10T19:29:22.148281+0000","last_fullsized":"2023-10-10T19:29:22.148281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011353221,"stat_sum":{"num_bytes":671969,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":65,"num_read_kb":20680,"num_write":694,"num_write_kb":24810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"132'961","reported_seq":3576,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157013+0000","last_change":"2023-10-10T19:16:55.412877+0000","last_active":"2023-10-10T19:29:22.157013+0000","last_peered":"2023-10-10T19:29:22.157013+0000","last_clean":"2023-10-10T19:29:22.157013+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T19:29:22.157013+0000","last_undegraded":"2023-10-10T19:29:22.157013+0000","last_fullsized":"2023-10-10T19:29:22.157013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088633549999999998,"stat_sum":{"num_bytes":651243,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":62,"num_read_kb":30965,"num_write":672,"num_write_kb":37710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"132'987","reported_seq":3618,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163074+0000","last_change":"2023-10-10T19:16:54.790962+0000","last_active":"2023-10-10T19:29:22.163074+0000","last_peered":"2023-10-10T19:29:22.163074+0000","last_clean":"2023-10-10T19:29:22.163074+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T19:29:22.163074+0000","last_undegraded":"2023-10-10T19:29:22.163074+0000","last_fullsized":"2023-10-10T19:29:22.163074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016189688000000001,"stat_sum":{"num_bytes":219054,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":75,"num_read_kb":17848,"num_write":702,"num_write_kb":31364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"132'942","reported_seq":3569,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162830+0000","last_change":"2023-10-10T19:16:55.414924+0000","last_active":"2023-10-10T19:29:22.162830+0000","last_peered":"2023-10-10T19:29:22.162830+0000","last_clean":"2023-10-10T19:29:22.162830+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T19:29:22.162830+0000","last_undegraded":"2023-10-10T19:29:22.162830+0000","last_fullsized":"2023-10-10T19:29:22.162830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010631753000000001,"stat_sum":{"num_bytes":2108369,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":63,"num_read_kb":20939,"num_write":675,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"132'987","reported_seq":3609,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.164189+0000","last_change":"2023-10-10T19:16:54.789065+0000","last_active":"2023-10-10T19:29:22.164189+0000","last_peered":"2023-10-10T19:29:22.164189+0000","last_clean":"2023-10-10T19:29:22.164189+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T19:29:22.164189+0000","last_undegraded":"2023-10-10T19:29:22.164189+0000","last_fullsized":"2023-10-10T19:29:22.164189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014667316999999999,"stat_sum":{"num_bytes":1119068,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":77,"num_read_kb":53600,"num_write":685,"num_write_kb":52880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"132'880","reported_seq":3519,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162097+0000","last_change":"2023-10-10T19:16:54.870456+0000","last_active":"2023-10-10T19:29:22.162097+0000","last_peered":"2023-10-10T19:29:22.162097+0000","last_clean":"2023-10-10T19:29:22.162097+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T19:29:22.162097+0000","last_undegraded":"2023-10-10T19:29:22.162097+0000","last_fullsized":"2023-10-10T19:29:22.162097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012402231,"stat_sum":{"num_bytes":484780,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":58,"num_read_kb":28110,"num_write":652,"num_write_kb":35721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"132'976","reported_seq":3604,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.164700+0000","last_change":"2023-10-10T19:16:54.795632+0000","last_active":"2023-10-10T19:29:22.164700+0000","last_peered":"2023-10-10T19:29:22.164700+0000","last_clean":"2023-10-10T19:29:22.164700+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T19:29:22.164700+0000","last_undegraded":"2023-10-10T19:29:22.164700+0000","last_fullsized":"2023-10-10T19:29:22.164700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":976,"log_dups_size":0,"ondisk_log_size":976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017699890999999999,"stat_sum":{"num_bytes":94035,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":76,"num_read_kb":37435,"num_write":702,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"132'955","reported_seq":3582,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163444+0000","last_change":"2023-10-10T19:16:54.783622+0000","last_active":"2023-10-10T19:29:22.163444+0000","last_peered":"2023-10-10T19:29:22.163444+0000","last_clean":"2023-10-10T19:29:22.163444+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T19:29:22.163444+0000","last_undegraded":"2023-10-10T19:29:22.163444+0000","last_fullsized":"2023-10-10T19:29:22.163444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079196160000000008,"stat_sum":{"num_bytes":269975,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":70,"num_read_kb":16825,"num_write":697,"num_write_kb":23418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"132'975","reported_seq":3625,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163125+0000","last_change":"2023-10-10T19:16:54.784745+0000","last_active":"2023-10-10T19:29:22.163125+0000","last_peered":"2023-10-10T19:29:22.163125+0000","last_clean":"2023-10-10T19:29:22.163125+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T19:29:22.163125+0000","last_undegraded":"2023-10-10T19:29:22.163125+0000","last_fullsized":"2023-10-10T19:29:22.163125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088422029999999999,"stat_sum":{"num_bytes":213661,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":82,"num_read_kb":46829,"num_write":712,"num_write_kb":53066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"132'994","reported_seq":3641,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.048201+0000","last_change":"2023-10-10T19:29:48.048165+0000","last_active":"2023-10-10T19:29:48.048201+0000","last_peered":"2023-10-10T19:29:48.048201+0000","last_clean":"2023-10-10T19:29:48.048201+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T19:29:48.048201+0000","last_undegraded":"2023-10-10T19:29:48.048201+0000","last_fullsized":"2023-10-10T19:29:48.048201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'994","last_scrub_stamp":"2023-10-10T19:29:48.048044+0000","last_deep_scrub":"132'994","last_deep_scrub_stamp":"2023-10-10T19:29:48.048044+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.048044+0000","objects_scrubbed":40,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:22:32.633316+0000","scrub_duration":0.033909425999999999,"objects_trimmed":0,"snaptrim_duration":0.0078589550000000008,"stat_sum":{"num_bytes":337999,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":75,"num_read_kb":57532,"num_write":710,"num_write_kb":73524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"132'929","reported_seq":3560,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.158163+0000","last_change":"2023-10-10T19:16:54.872408+0000","last_active":"2023-10-10T19:29:22.158163+0000","last_peered":"2023-10-10T19:29:22.158163+0000","last_clean":"2023-10-10T19:29:22.158163+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T19:29:22.158163+0000","last_undegraded":"2023-10-10T19:29:22.158163+0000","last_fullsized":"2023-10-10T19:29:22.158163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014204069,"stat_sum":{"num_bytes":4832489,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":73,"num_read_kb":32208,"num_write":656,"num_write_kb":44878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"132'929","reported_seq":3554,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.155414+0000","last_change":"2023-10-10T19:16:54.978714+0000","last_active":"2023-10-10T19:29:22.155414+0000","last_peered":"2023-10-10T19:29:22.155414+0000","last_clean":"2023-10-10T19:29:22.155414+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T19:29:22.155414+0000","last_undegraded":"2023-10-10T19:29:22.155414+0000","last_fullsized":"2023-10-10T19:29:22.155414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097965510000000006,"stat_sum":{"num_bytes":724304,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":63,"num_read_kb":28420,"num_write":663,"num_write_kb":31175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"132'955","reported_seq":3589,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.155075+0000","last_change":"2023-10-10T19:16:54.917427+0000","last_active":"2023-10-10T19:29:22.155075+0000","last_peered":"2023-10-10T19:29:22.155075+0000","last_clean":"2023-10-10T19:29:22.155075+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T19:29:22.155075+0000","last_undegraded":"2023-10-10T19:29:22.155075+0000","last_fullsized":"2023-10-10T19:29:22.155075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059774370000000004,"stat_sum":{"num_bytes":1353080,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":77,"num_read_kb":52747,"num_write":668,"num_write_kb":73053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"132'998","reported_seq":3633,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162124+0000","last_change":"2023-10-10T19:16:55.412885+0000","last_active":"2023-10-10T19:29:22.162124+0000","last_peered":"2023-10-10T19:29:22.162124+0000","last_clean":"2023-10-10T19:29:22.162124+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T19:29:22.162124+0000","last_undegraded":"2023-10-10T19:29:22.162124+0000","last_fullsized":"2023-10-10T19:29:22.162124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087668889999999999,"stat_sum":{"num_bytes":582240,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":80,"num_read_kb":42995,"num_write":709,"num_write_kb":53070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"132'905","reported_seq":3560,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.335982+0000","last_change":"2023-10-10T19:16:55.417126+0000","last_active":"2023-10-10T19:29:22.335982+0000","last_peered":"2023-10-10T19:29:22.335982+0000","last_clean":"2023-10-10T19:29:22.335982+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T19:29:22.335982+0000","last_undegraded":"2023-10-10T19:29:22.335982+0000","last_fullsized":"2023-10-10T19:29:22.335982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":905,"log_dups_size":0,"ondisk_log_size":905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012085218,"stat_sum":{"num_bytes":1912035,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":72,"num_read_kb":40372,"num_write":683,"num_write_kb":48886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"132'871","reported_seq":3555,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.229972+0000","last_change":"2023-10-10T19:29:40.229945+0000","last_active":"2023-10-10T19:29:40.229972+0000","last_peered":"2023-10-10T19:29:40.229972+0000","last_clean":"2023-10-10T19:29:40.229972+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T19:29:40.229972+0000","last_undegraded":"2023-10-10T19:29:40.229972+0000","last_fullsized":"2023-10-10T19:29:40.229972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'871","last_scrub_stamp":"2023-10-10T19:29:40.229861+0000","last_deep_scrub":"132'871","last_deep_scrub_stamp":"2023-10-10T19:29:40.229861+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.229861+0000","objects_scrubbed":37,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:14:54.227939+0000","scrub_duration":0.028370732999999999,"objects_trimmed":0,"snaptrim_duration":0.00594963,"stat_sum":{"num_bytes":953364,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":72,"num_read_kb":44243,"num_write":664,"num_write_kb":60950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"132'1073","reported_seq":3671,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.166674+0000","last_change":"2023-10-10T19:16:54.792335+0000","last_active":"2023-10-10T19:29:22.166674+0000","last_peered":"2023-10-10T19:29:22.166674+0000","last_clean":"2023-10-10T19:29:22.166674+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T19:29:22.166674+0000","last_undegraded":"2023-10-10T19:29:22.166674+0000","last_fullsized":"2023-10-10T19:29:22.166674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01601,"stat_sum":{"num_bytes":689975,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":73,"num_read_kb":41187,"num_write":728,"num_write_kb":49784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"132'855","reported_seq":1031,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.287167+0000","last_change":"2023-10-10T19:29:28.287143+0000","last_active":"2023-10-10T19:29:28.287167+0000","last_peered":"2023-10-10T19:29:28.287167+0000","last_clean":"2023-10-10T19:29:28.287167+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T19:29:28.287167+0000","last_undegraded":"2023-10-10T19:29:28.287167+0000","last_fullsized":"2023-10-10T19:29:28.287167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'855","last_scrub_stamp":"2023-10-10T19:29:28.287071+0000","last_deep_scrub":"132'855","last_deep_scrub_stamp":"2023-10-10T19:29:28.287071+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.287071+0000","objects_scrubbed":19,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:01.117300+0000","scrub_duration":0.031438141000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389252,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":82,"num_read_kb":61516,"num_write":781,"num_write_kb":21019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":54468,"num_omap_keys":96,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"132'998","reported_seq":3620,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.158073+0000","last_change":"2023-10-10T19:16:54.877116+0000","last_active":"2023-10-10T19:29:22.158073+0000","last_peered":"2023-10-10T19:29:22.158073+0000","last_clean":"2023-10-10T19:29:22.158073+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T19:29:22.158073+0000","last_undegraded":"2023-10-10T19:29:22.158073+0000","last_fullsized":"2023-10-10T19:29:22.158073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018086130999999998,"stat_sum":{"num_bytes":136255,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":63,"num_read_kb":13169,"num_write":738,"num_write_kb":31183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"132'776","reported_seq":931,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:32.981768+0000","last_change":"2023-10-10T19:29:32.981742+0000","last_active":"2023-10-10T19:29:32.981768+0000","last_peered":"2023-10-10T19:29:32.981768+0000","last_clean":"2023-10-10T19:29:32.981768+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T19:29:32.981768+0000","last_undegraded":"2023-10-10T19:29:32.981768+0000","last_fullsized":"2023-10-10T19:29:32.981768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'776","last_scrub_stamp":"2023-10-10T19:29:32.981652+0000","last_deep_scrub":"132'776","last_deep_scrub_stamp":"2023-10-10T19:29:32.981652+0000","last_clean_scrub_stamp":"2023-10-10T19:29:32.981652+0000","objects_scrubbed":16,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:01:01.872125+0000","scrub_duration":0.042698237999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":29,"num_read_kb":69691,"num_write":711,"num_write_kb":16940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":122430,"num_omap_keys":218,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"132'1012","reported_seq":3666,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.277911+0000","last_change":"2023-10-10T19:29:47.277885+0000","last_active":"2023-10-10T19:29:47.277911+0000","last_peered":"2023-10-10T19:29:47.277911+0000","last_clean":"2023-10-10T19:29:47.277911+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T19:29:47.277911+0000","last_undegraded":"2023-10-10T19:29:47.277911+0000","last_fullsized":"2023-10-10T19:29:47.277911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1012","last_scrub_stamp":"2023-10-10T19:29:47.277804+0000","last_deep_scrub":"132'1012","last_deep_scrub_stamp":"2023-10-10T19:29:47.277804+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.277804+0000","objects_scrubbed":36,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:05.362050+0000","scrub_duration":0.026281229,"objects_trimmed":0,"snaptrim_duration":0.0093729599999999996,"stat_sum":{"num_bytes":224018,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":74,"num_read_kb":34573,"num_write":731,"num_write_kb":44057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"132'1695","reported_seq":1872,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:25.024703+0000","last_change":"2023-10-10T18:45:10.411980+0000","last_active":"2023-10-10T19:29:25.024703+0000","last_peered":"2023-10-10T19:29:25.024703+0000","last_clean":"2023-10-10T19:29:25.024703+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T19:29:25.024703+0000","last_undegraded":"2023-10-10T19:29:25.024703+0000","last_fullsized":"2023-10-10T19:29:25.024703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":1695,"log_dups_size":0,"ondisk_log_size":1695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":114,"num_read_kb":59511,"num_write":1905,"num_write_kb":22655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"132'971","reported_seq":3577,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.099971+0000","last_change":"2023-10-10T19:29:48.099945+0000","last_active":"2023-10-10T19:29:48.099971+0000","last_peered":"2023-10-10T19:29:48.099971+0000","last_clean":"2023-10-10T19:29:48.099971+0000","last_became_active":"2023-10-10T18:46:01.103860+0000","last_became_peered":"2023-10-10T18:46:01.103860+0000","last_unstale":"2023-10-10T19:29:48.099971+0000","last_undegraded":"2023-10-10T19:29:48.099971+0000","last_fullsized":"2023-10-10T19:29:48.099971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'971","last_scrub_stamp":"2023-10-10T19:29:48.099859+0000","last_deep_scrub":"132'971","last_deep_scrub_stamp":"2023-10-10T19:29:48.099859+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.099859+0000","objects_scrubbed":28,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:28:00.147441+0000","scrub_duration":0.027094722000000002,"objects_trimmed":0,"snaptrim_duration":0.0054668909999999998,"stat_sum":{"num_bytes":121718,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":48,"num_read_kb":295,"num_write":648,"num_write_kb":7505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"132'763","reported_seq":927,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.085034+0000","last_change":"2023-10-10T19:29:47.084994+0000","last_active":"2023-10-10T19:29:47.085034+0000","last_peered":"2023-10-10T19:29:47.085034+0000","last_clean":"2023-10-10T19:29:47.085034+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T19:29:47.085034+0000","last_undegraded":"2023-10-10T19:29:47.085034+0000","last_fullsized":"2023-10-10T19:29:47.085034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'763","last_scrub_stamp":"2023-10-10T19:29:47.084860+0000","last_deep_scrub":"132'763","last_deep_scrub_stamp":"2023-10-10T19:29:47.084860+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.084860+0000","objects_scrubbed":13,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:01.164246+0000","scrub_duration":0.026132728000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389162,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":55,"num_read_kb":36831,"num_write":683,"num_write_kb":16858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":72140,"num_omap_keys":127,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"132'991","reported_seq":3622,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157586+0000","last_change":"2023-10-10T19:16:55.416075+0000","last_active":"2023-10-10T19:29:22.157586+0000","last_peered":"2023-10-10T19:29:22.157586+0000","last_clean":"2023-10-10T19:29:22.157586+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T19:29:22.157586+0000","last_undegraded":"2023-10-10T19:29:22.157586+0000","last_fullsized":"2023-10-10T19:29:22.157586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011753095,"stat_sum":{"num_bytes":171234,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":81,"num_read_kb":35882,"num_write":699,"num_write_kb":43024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"132'652","reported_seq":811,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.206093+0000","last_change":"2023-10-10T19:29:29.999770+0000","last_active":"2023-10-10T19:29:30.206093+0000","last_peered":"2023-10-10T19:29:30.206093+0000","last_clean":"2023-10-10T19:29:30.206093+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T19:29:30.206093+0000","last_undegraded":"2023-10-10T19:29:30.206093+0000","last_fullsized":"2023-10-10T19:29:30.206093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'626","last_scrub_stamp":"2023-10-10T19:29:29.999705+0000","last_deep_scrub":"132'626","last_deep_scrub_stamp":"2023-10-10T19:29:29.999705+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.999705+0000","objects_scrubbed":14,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T07:13:56.479723+0000","scrub_duration":0.034014667999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15333167,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":31,"num_read_kb":65583,"num_write":581,"num_write_kb":19518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":50025,"num_omap_keys":89,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"132'942","reported_seq":3582,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:33.015438+0000","last_change":"2023-10-10T19:29:33.015412+0000","last_active":"2023-10-10T19:29:33.015438+0000","last_peered":"2023-10-10T19:29:33.015438+0000","last_clean":"2023-10-10T19:29:33.015438+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T19:29:33.015438+0000","last_undegraded":"2023-10-10T19:29:33.015438+0000","last_fullsized":"2023-10-10T19:29:33.015438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'942","last_scrub_stamp":"2023-10-10T19:29:33.015326+0000","last_deep_scrub":"132'942","last_deep_scrub_stamp":"2023-10-10T19:29:33.015326+0000","last_clean_scrub_stamp":"2023-10-10T19:29:33.015326+0000","objects_scrubbed":43,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:49:00.949756+0000","scrub_duration":0.032708036000000003,"objects_trimmed":0,"snaptrim_duration":0.0062644010000000002,"stat_sum":{"num_bytes":838983,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":59,"num_read_kb":21826,"num_write":689,"num_write_kb":37727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"132'791","reported_seq":1134,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.136520+0000","last_change":"2023-10-10T19:29:31.136493+0000","last_active":"2023-10-10T19:29:31.136520+0000","last_peered":"2023-10-10T19:29:31.136520+0000","last_clean":"2023-10-10T19:29:31.136520+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T19:29:31.136520+0000","last_undegraded":"2023-10-10T19:29:31.136520+0000","last_fullsized":"2023-10-10T19:29:31.136520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'791","last_scrub_stamp":"2023-10-10T19:29:31.136411+0000","last_deep_scrub":"132'791","last_deep_scrub_stamp":"2023-10-10T19:29:31.136411+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.136411+0000","objects_scrubbed":15,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:18:11.372189+0000","scrub_duration":0.042934018999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111936,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":251,"num_read_kb":81147,"num_write":704,"num_write_kb":25397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35794,"num_omap_keys":65,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"132'857","reported_seq":3517,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.291480+0000","last_change":"2023-10-10T19:29:42.291455+0000","last_active":"2023-10-10T19:29:42.291480+0000","last_peered":"2023-10-10T19:29:42.291480+0000","last_clean":"2023-10-10T19:29:42.291480+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T19:29:42.291480+0000","last_undegraded":"2023-10-10T19:29:42.291480+0000","last_fullsized":"2023-10-10T19:29:42.291480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'857","last_scrub_stamp":"2023-10-10T19:29:42.291374+0000","last_deep_scrub":"132'857","last_deep_scrub_stamp":"2023-10-10T19:29:42.291374+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.291374+0000","objects_scrubbed":27,"log_size":857,"log_dups_size":0,"ondisk_log_size":857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:23:42.700538+0000","scrub_duration":0.018625359000000001,"objects_trimmed":0,"snaptrim_duration":0.015269961,"stat_sum":{"num_bytes":473837,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":60,"num_read_kb":24046,"num_write":615,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"132'1506","reported_seq":1912,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.017721+0000","last_change":"2023-10-10T19:29:28.017678+0000","last_active":"2023-10-10T19:29:28.017721+0000","last_peered":"2023-10-10T19:29:28.017721+0000","last_clean":"2023-10-10T19:29:28.017721+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T19:29:28.017721+0000","last_undegraded":"2023-10-10T19:29:28.017721+0000","last_fullsized":"2023-10-10T19:29:28.017721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1506","last_scrub_stamp":"2023-10-10T19:29:28.017594+0000","last_deep_scrub":"132'1506","last_deep_scrub_stamp":"2023-10-10T19:29:28.017594+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.017594+0000","objects_scrubbed":12,"log_size":1506,"log_dups_size":0,"ondisk_log_size":1506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:59:47.232197+0000","scrub_duration":0.020028464999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4727705,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":295,"num_read_kb":15405,"num_write":1447,"num_write_kb":10055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66656,"num_omap_keys":116,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"132'1002","reported_seq":3646,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.158663+0000","last_change":"2023-10-10T19:16:54.874108+0000","last_active":"2023-10-10T19:29:22.158663+0000","last_peered":"2023-10-10T19:29:22.158663+0000","last_clean":"2023-10-10T19:29:22.158663+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T19:29:22.158663+0000","last_undegraded":"2023-10-10T19:29:22.158663+0000","last_fullsized":"2023-10-10T19:29:22.158663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015767953000000001,"stat_sum":{"num_bytes":671315,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":81,"num_read_kb":19092,"num_write":747,"num_write_kb":30378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"132'506","reported_seq":662,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.758375+0000","last_change":"2023-10-10T19:29:40.758347+0000","last_active":"2023-10-10T19:29:40.758375+0000","last_peered":"2023-10-10T19:29:40.758375+0000","last_clean":"2023-10-10T19:29:40.758375+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T19:29:40.758375+0000","last_undegraded":"2023-10-10T19:29:40.758375+0000","last_fullsized":"2023-10-10T19:29:40.758375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'506","last_scrub_stamp":"2023-10-10T19:29:40.758259+0000","last_deep_scrub":"132'506","last_deep_scrub_stamp":"2023-10-10T19:29:40.758259+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.758259+0000","objects_scrubbed":12,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:28:13.803240+0000","scrub_duration":0.033275374000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583466,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":65566,"num_write":451,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":76745,"num_omap_keys":113,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"132'1056","reported_seq":3707,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.597586+0000","last_change":"2023-10-10T19:29:45.597550+0000","last_active":"2023-10-10T19:29:45.597586+0000","last_peered":"2023-10-10T19:29:45.597586+0000","last_clean":"2023-10-10T19:29:45.597586+0000","last_became_active":"2023-10-10T18:46:00.606391+0000","last_became_peered":"2023-10-10T18:46:00.606391+0000","last_unstale":"2023-10-10T19:29:45.597586+0000","last_undegraded":"2023-10-10T19:29:45.597586+0000","last_fullsized":"2023-10-10T19:29:45.597586+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1056","last_scrub_stamp":"2023-10-10T19:29:45.597448+0000","last_deep_scrub":"132'1056","last_deep_scrub_stamp":"2023-10-10T19:29:45.597448+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.597448+0000","objects_scrubbed":41,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:39:53.993867+0000","scrub_duration":0.033388767999999999,"objects_trimmed":0,"snaptrim_duration":0.0066939039999999997,"stat_sum":{"num_bytes":730064,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":80,"num_read_kb":42231,"num_write":744,"num_write_kb":52869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"132'291","reported_seq":443,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.067549+0000","last_change":"2023-10-10T19:29:48.067525+0000","last_active":"2023-10-10T19:29:48.067549+0000","last_peered":"2023-10-10T19:29:48.067549+0000","last_clean":"2023-10-10T19:29:48.067549+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T19:29:48.067549+0000","last_undegraded":"2023-10-10T19:29:48.067549+0000","last_fullsized":"2023-10-10T19:29:48.067549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'291","last_scrub_stamp":"2023-10-10T19:29:48.067450+0000","last_deep_scrub":"132'291","last_deep_scrub_stamp":"2023-10-10T19:29:48.067450+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.067450+0000","objects_scrubbed":17,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:31:55.859793+0000","scrub_duration":0.010942558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":34,"num_read_kb":54,"num_write":233,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":87494,"num_omap_keys":156,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"132'963","reported_seq":3599,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.165435+0000","last_change":"2023-10-10T19:16:55.413821+0000","last_active":"2023-10-10T19:29:22.165435+0000","last_peered":"2023-10-10T19:29:22.165435+0000","last_clean":"2023-10-10T19:29:22.165435+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T19:29:22.165435+0000","last_undegraded":"2023-10-10T19:29:22.165435+0000","last_fullsized":"2023-10-10T19:29:22.165435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":963,"log_dups_size":0,"ondisk_log_size":963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096962509999999995,"stat_sum":{"num_bytes":540276,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":63,"num_read_kb":26256,"num_write":725,"num_write_kb":46019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"132'643","reported_seq":1012,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:43.718334+0000","last_change":"2023-10-10T19:29:43.718305+0000","last_active":"2023-10-10T19:29:43.718334+0000","last_peered":"2023-10-10T19:29:43.718334+0000","last_clean":"2023-10-10T19:29:43.718334+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T19:29:43.718334+0000","last_undegraded":"2023-10-10T19:29:43.718334+0000","last_fullsized":"2023-10-10T19:29:43.718334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'643","last_scrub_stamp":"2023-10-10T19:29:43.718205+0000","last_deep_scrub":"132'643","last_deep_scrub_stamp":"2023-10-10T19:29:43.718205+0000","last_clean_scrub_stamp":"2023-10-10T19:29:43.718205+0000","objects_scrubbed":15,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:24:50.063113+0000","scrub_duration":0.029979209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8573947,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":262,"num_read_kb":30418,"num_write":570,"num_write_kb":12898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":102578,"num_omap_keys":183,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"132'952","reported_seq":3587,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:33.425228+0000","last_change":"2023-10-10T19:29:33.425192+0000","last_active":"2023-10-10T19:29:33.425228+0000","last_peered":"2023-10-10T19:29:33.425228+0000","last_clean":"2023-10-10T19:29:33.425228+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T19:29:33.425228+0000","last_undegraded":"2023-10-10T19:29:33.425228+0000","last_fullsized":"2023-10-10T19:29:33.425228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'952","last_scrub_stamp":"2023-10-10T19:29:33.425101+0000","last_deep_scrub":"132'952","last_deep_scrub_stamp":"2023-10-10T19:29:33.425101+0000","last_clean_scrub_stamp":"2023-10-10T19:29:33.425101+0000","objects_scrubbed":33,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:50:37.033893+0000","scrub_duration":0.024688640000000001,"objects_trimmed":0,"snaptrim_duration":0.015222243,"stat_sum":{"num_bytes":331805,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":64,"num_read_kb":38795,"num_write":639,"num_write_kb":46131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"132'219","reported_seq":372,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.571663+0000","last_change":"2023-10-10T19:29:42.571637+0000","last_active":"2023-10-10T19:29:42.571663+0000","last_peered":"2023-10-10T19:29:42.571663+0000","last_clean":"2023-10-10T19:29:42.571663+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T19:29:42.571663+0000","last_undegraded":"2023-10-10T19:29:42.571663+0000","last_fullsized":"2023-10-10T19:29:42.571663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'219","last_scrub_stamp":"2023-10-10T19:29:42.571548+0000","last_deep_scrub":"132'219","last_deep_scrub_stamp":"2023-10-10T19:29:42.571548+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.571548+0000","objects_scrubbed":12,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:06:27.021509+0000","scrub_duration":0.010029698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":61,"num_write":173,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70925,"num_omap_keys":126,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"132'902","reported_seq":3546,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.010757+0000","last_change":"2023-10-10T19:29:42.010728+0000","last_active":"2023-10-10T19:29:42.010757+0000","last_peered":"2023-10-10T19:29:42.010757+0000","last_clean":"2023-10-10T19:29:42.010757+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T19:29:42.010757+0000","last_undegraded":"2023-10-10T19:29:42.010757+0000","last_fullsized":"2023-10-10T19:29:42.010757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'902","last_scrub_stamp":"2023-10-10T19:29:42.010643+0000","last_deep_scrub":"132'902","last_deep_scrub_stamp":"2023-10-10T19:29:42.010643+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.010643+0000","objects_scrubbed":35,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:51:41.084838+0000","scrub_duration":0.032300451000000001,"objects_trimmed":0,"snaptrim_duration":0.0080742209999999995,"stat_sum":{"num_bytes":103314,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":56,"num_read_kb":23103,"num_write":655,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"132'280","reported_seq":440,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:36.136095+0000","last_change":"2023-10-10T19:29:36.136030+0000","last_active":"2023-10-10T19:29:36.136095+0000","last_peered":"2023-10-10T19:29:36.136095+0000","last_clean":"2023-10-10T19:29:36.136095+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T19:29:36.136095+0000","last_undegraded":"2023-10-10T19:29:36.136095+0000","last_fullsized":"2023-10-10T19:29:36.136095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'279","last_scrub_stamp":"2023-10-10T19:29:36.135933+0000","last_deep_scrub":"132'279","last_deep_scrub_stamp":"2023-10-10T19:29:36.135933+0000","last_clean_scrub_stamp":"2023-10-10T19:29:36.135933+0000","objects_scrubbed":15,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:22:18.822287+0000","scrub_duration":0.016822342000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194948,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":59,"num_read_kb":16449,"num_write":214,"num_write_kb":4413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64730,"num_omap_keys":109,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"132'975","reported_seq":3617,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:35.143939+0000","last_change":"2023-10-10T19:29:35.143909+0000","last_active":"2023-10-10T19:29:35.143939+0000","last_peered":"2023-10-10T19:29:35.143939+0000","last_clean":"2023-10-10T19:29:35.143939+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T19:29:35.143939+0000","last_undegraded":"2023-10-10T19:29:35.143939+0000","last_fullsized":"2023-10-10T19:29:35.143939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'975","last_scrub_stamp":"2023-10-10T19:29:35.143816+0000","last_deep_scrub":"132'975","last_deep_scrub_stamp":"2023-10-10T19:29:35.143816+0000","last_clean_scrub_stamp":"2023-10-10T19:29:35.143816+0000","objects_scrubbed":35,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:54:49.422231+0000","scrub_duration":0.025232871,"objects_trimmed":0,"snaptrim_duration":0.015774979000000001,"stat_sum":{"num_bytes":487172,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":71,"num_read_kb":24956,"num_write":677,"num_write_kb":31709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"132'838","reported_seq":1285,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.356700+0000","last_change":"2023-10-10T19:29:30.356675+0000","last_active":"2023-10-10T19:29:30.356700+0000","last_peered":"2023-10-10T19:29:30.356700+0000","last_clean":"2023-10-10T19:29:30.356700+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T19:29:30.356700+0000","last_undegraded":"2023-10-10T19:29:30.356700+0000","last_fullsized":"2023-10-10T19:29:30.356700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'838","last_scrub_stamp":"2023-10-10T19:29:30.356605+0000","last_deep_scrub":"132'838","last_deep_scrub_stamp":"2023-10-10T19:29:30.356605+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.356605+0000","objects_scrubbed":20,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:03:47.996301+0000","scrub_duration":0.044624234999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227398,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":359,"num_read_kb":46031,"num_write":771,"num_write_kb":13519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":95526,"num_omap_keys":174,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"132'949","reported_seq":3592,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.362095+0000","last_change":"2023-10-10T19:29:31.362073+0000","last_active":"2023-10-10T19:29:31.362095+0000","last_peered":"2023-10-10T19:29:31.362095+0000","last_clean":"2023-10-10T19:29:31.362095+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T19:29:31.362095+0000","last_undegraded":"2023-10-10T19:29:31.362095+0000","last_fullsized":"2023-10-10T19:29:31.362095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'949","last_scrub_stamp":"2023-10-10T19:29:31.362007+0000","last_deep_scrub":"132'949","last_deep_scrub_stamp":"2023-10-10T19:29:31.362007+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.362007+0000","objects_scrubbed":32,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:44.834185+0000","scrub_duration":0.026536431999999999,"objects_trimmed":0,"snaptrim_duration":0.01251216,"stat_sum":{"num_bytes":121104,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":62,"num_read_kb":21731,"num_write":687,"num_write_kb":32925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"132'735","reported_seq":906,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.681955+0000","last_change":"2023-10-10T19:29:48.681927+0000","last_active":"2023-10-10T19:29:48.681955+0000","last_peered":"2023-10-10T19:29:48.681955+0000","last_clean":"2023-10-10T19:29:48.681955+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T19:29:48.681955+0000","last_undegraded":"2023-10-10T19:29:48.681955+0000","last_fullsized":"2023-10-10T19:29:48.681955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'735","last_scrub_stamp":"2023-10-10T19:29:48.681838+0000","last_deep_scrub":"132'735","last_deep_scrub_stamp":"2023-10-10T19:29:48.681838+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.681838+0000","objects_scrubbed":15,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:12.687746+0000","scrub_duration":0.036150337999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15771894,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":58,"num_read_kb":76305,"num_write":656,"num_write_kb":19955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":55118,"num_omap_keys":99,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"132'969","reported_seq":3595,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163527+0000","last_change":"2023-10-10T19:16:54.791948+0000","last_active":"2023-10-10T19:29:22.163527+0000","last_peered":"2023-10-10T19:29:22.163527+0000","last_clean":"2023-10-10T19:29:22.163527+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T19:29:22.163527+0000","last_undegraded":"2023-10-10T19:29:22.163527+0000","last_fullsized":"2023-10-10T19:29:22.163527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":969,"log_dups_size":0,"ondisk_log_size":969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017093574,"stat_sum":{"num_bytes":558529,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":73,"num_read_kb":45017,"num_write":693,"num_write_kb":44107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"132'423","reported_seq":590,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:34.186050+0000","last_change":"2023-10-10T19:29:34.186026+0000","last_active":"2023-10-10T19:29:34.186050+0000","last_peered":"2023-10-10T19:29:34.186050+0000","last_clean":"2023-10-10T19:29:34.186050+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T19:29:34.186050+0000","last_undegraded":"2023-10-10T19:29:34.186050+0000","last_fullsized":"2023-10-10T19:29:34.186050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'423","last_scrub_stamp":"2023-10-10T19:29:34.185945+0000","last_deep_scrub":"132'423","last_deep_scrub_stamp":"2023-10-10T19:29:34.185945+0000","last_clean_scrub_stamp":"2023-10-10T19:29:34.185945+0000","objects_scrubbed":16,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:58:02.739712+0000","scrub_duration":0.049211269000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":54,"num_read_kb":84439,"num_write":360,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":79730,"num_omap_keys":141,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"132'834","reported_seq":3467,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.139880+0000","last_change":"2023-10-10T19:16:54.782792+0000","last_active":"2023-10-10T19:29:22.139880+0000","last_peered":"2023-10-10T19:29:22.139880+0000","last_clean":"2023-10-10T19:29:22.139880+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T19:29:22.139880+0000","last_undegraded":"2023-10-10T19:29:22.139880+0000","last_fullsized":"2023-10-10T19:29:22.139880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075683979999999996,"stat_sum":{"num_bytes":1111788,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":52,"num_read_kb":24653,"num_write":611,"num_write_kb":45399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"132'823","reported_seq":1013,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:32.342694+0000","last_change":"2023-10-10T19:29:32.342669+0000","last_active":"2023-10-10T19:29:32.342694+0000","last_peered":"2023-10-10T19:29:32.342694+0000","last_clean":"2023-10-10T19:29:32.342694+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T19:29:32.342694+0000","last_undegraded":"2023-10-10T19:29:32.342694+0000","last_fullsized":"2023-10-10T19:29:32.342694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'823","last_scrub_stamp":"2023-10-10T19:29:32.342571+0000","last_deep_scrub":"132'823","last_deep_scrub_stamp":"2023-10-10T19:29:32.342571+0000","last_clean_scrub_stamp":"2023-10-10T19:29:32.342571+0000","objects_scrubbed":19,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:42.635501+0000","scrub_duration":0.051952427000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":113,"num_read_kb":78302,"num_write":734,"num_write_kb":25074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":59966,"num_omap_keys":106,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"132'933","reported_seq":3582,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:37.129812+0000","last_change":"2023-10-10T19:29:37.129774+0000","last_active":"2023-10-10T19:29:37.129812+0000","last_peered":"2023-10-10T19:29:37.129812+0000","last_clean":"2023-10-10T19:29:37.129812+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T19:29:37.129812+0000","last_undegraded":"2023-10-10T19:29:37.129812+0000","last_fullsized":"2023-10-10T19:29:37.129812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'933","last_scrub_stamp":"2023-10-10T19:29:37.129655+0000","last_deep_scrub":"132'933","last_deep_scrub_stamp":"2023-10-10T19:29:37.129655+0000","last_clean_scrub_stamp":"2023-10-10T19:29:37.129655+0000","objects_scrubbed":33,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:10.731137+0000","scrub_duration":0.028840863000000001,"objects_trimmed":0,"snaptrim_duration":0.01616896,"stat_sum":{"num_bytes":90659,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":67,"num_read_kb":45230,"num_write":665,"num_write_kb":49103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"132'621","reported_seq":769,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.983083+0000","last_change":"2023-10-10T19:29:28.983043+0000","last_active":"2023-10-10T19:29:28.983083+0000","last_peered":"2023-10-10T19:29:28.983083+0000","last_clean":"2023-10-10T19:29:28.983083+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T19:29:28.983083+0000","last_undegraded":"2023-10-10T19:29:28.983083+0000","last_fullsized":"2023-10-10T19:29:28.983083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'621","last_scrub_stamp":"2023-10-10T19:29:28.982898+0000","last_deep_scrub":"132'621","last_deep_scrub_stamp":"2023-10-10T19:29:28.982898+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.982898+0000","objects_scrubbed":17,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:02:20.298697+0000","scrub_duration":0.023284135000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":23,"num_read_kb":12341,"num_write":750,"num_write_kb":8877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":130710,"num_omap_keys":230,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"132'938","reported_seq":3553,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.164004+0000","last_change":"2023-10-10T19:16:54.793503+0000","last_active":"2023-10-10T19:29:22.164004+0000","last_peered":"2023-10-10T19:29:22.164004+0000","last_clean":"2023-10-10T19:29:22.164004+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T19:29:22.164004+0000","last_undegraded":"2023-10-10T19:29:22.164004+0000","last_fullsized":"2023-10-10T19:29:22.164004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015882960000000002,"stat_sum":{"num_bytes":708648,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":57,"num_read_kb":20900,"num_write":676,"num_write_kb":24718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"132'177","reported_seq":325,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.696712+0000","last_change":"2023-10-10T19:29:30.696675+0000","last_active":"2023-10-10T19:29:30.696712+0000","last_peered":"2023-10-10T19:29:30.696712+0000","last_clean":"2023-10-10T19:29:30.696712+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T19:29:30.696712+0000","last_undegraded":"2023-10-10T19:29:30.696712+0000","last_fullsized":"2023-10-10T19:29:30.696712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'177","last_scrub_stamp":"2023-10-10T19:29:30.696594+0000","last_deep_scrub":"132'177","last_deep_scrub_stamp":"2023-10-10T19:29:30.696594+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.696594+0000","objects_scrubbed":15,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:22:41.559522+0000","scrub_duration":0.035497603000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":45123,"num_write":119,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":122093,"num_omap_keys":206,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"132'1049","reported_seq":3658,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157851+0000","last_change":"2023-10-10T19:16:55.415960+0000","last_active":"2023-10-10T19:29:22.157851+0000","last_peered":"2023-10-10T19:29:22.157851+0000","last_clean":"2023-10-10T19:29:22.157851+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T19:29:22.157851+0000","last_undegraded":"2023-10-10T19:29:22.157851+0000","last_fullsized":"2023-10-10T19:29:22.157851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011491507999999999,"stat_sum":{"num_bytes":461869,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":71,"num_read_kb":20716,"num_write":730,"num_write_kb":35297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"132'359","reported_seq":518,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.696909+0000","last_change":"2023-10-10T19:29:28.696870+0000","last_active":"2023-10-10T19:29:28.696909+0000","last_peered":"2023-10-10T19:29:28.696909+0000","last_clean":"2023-10-10T19:29:28.696909+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T19:29:28.696909+0000","last_undegraded":"2023-10-10T19:29:28.696909+0000","last_fullsized":"2023-10-10T19:29:28.696909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'359","last_scrub_stamp":"2023-10-10T19:29:28.696790+0000","last_deep_scrub":"132'359","last_deep_scrub_stamp":"2023-10-10T19:29:28.696790+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.696790+0000","objects_scrubbed":18,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:28:30.574457+0000","scrub_duration":0.036111012999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":73794,"num_write":284,"num_write_kb":20986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":75814,"num_omap_keys":136,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"132'953","reported_seq":3621,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.159685+0000","last_change":"2023-10-10T19:16:54.988658+0000","last_active":"2023-10-10T19:29:22.159685+0000","last_peered":"2023-10-10T19:29:22.159685+0000","last_clean":"2023-10-10T19:29:22.159685+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T19:29:22.159685+0000","last_undegraded":"2023-10-10T19:29:22.159685+0000","last_fullsized":"2023-10-10T19:29:22.159685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071091339999999996,"stat_sum":{"num_bytes":369103,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":93,"num_read_kb":25139,"num_write":713,"num_write_kb":35847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"132'945","reported_seq":3571,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.142630+0000","last_change":"2023-10-10T19:16:54.919574+0000","last_active":"2023-10-10T19:29:22.142630+0000","last_peered":"2023-10-10T19:29:22.142630+0000","last_clean":"2023-10-10T19:29:22.142630+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T19:29:22.142630+0000","last_undegraded":"2023-10-10T19:29:22.142630+0000","last_fullsized":"2023-10-10T19:29:22.142630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077260940000000002,"stat_sum":{"num_bytes":131785,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":61,"num_read_kb":19697,"num_write":689,"num_write_kb":26724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"132'700","reported_seq":1135,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.351264+0000","last_change":"2023-10-10T19:29:30.351227+0000","last_active":"2023-10-10T19:29:30.351264+0000","last_peered":"2023-10-10T19:29:30.351264+0000","last_clean":"2023-10-10T19:29:30.351264+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T19:29:30.351264+0000","last_undegraded":"2023-10-10T19:29:30.351264+0000","last_fullsized":"2023-10-10T19:29:30.351264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'700","last_scrub_stamp":"2023-10-10T19:29:30.351098+0000","last_deep_scrub":"132'700","last_deep_scrub_stamp":"2023-10-10T19:29:30.351098+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.351098+0000","objects_scrubbed":17,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:16:26.607570+0000","scrub_duration":0.030203292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9108221,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":314,"num_read_kb":40841,"num_write":610,"num_write_kb":9448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":67648,"num_omap_keys":120,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'922","reported_seq":3604,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:39.851461+0000","last_change":"2023-10-10T19:29:39.851434+0000","last_active":"2023-10-10T19:29:39.851461+0000","last_peered":"2023-10-10T19:29:39.851461+0000","last_clean":"2023-10-10T19:29:39.851461+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T19:29:39.851461+0000","last_undegraded":"2023-10-10T19:29:39.851461+0000","last_fullsized":"2023-10-10T19:29:39.851461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'922","last_scrub_stamp":"2023-10-10T19:29:39.851353+0000","last_deep_scrub":"132'922","last_deep_scrub_stamp":"2023-10-10T19:29:39.851353+0000","last_clean_scrub_stamp":"2023-10-10T19:29:39.851353+0000","objects_scrubbed":42,"log_size":922,"log_dups_size":0,"ondisk_log_size":922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T07:19:01.743649+0000","scrub_duration":0.028531329000000001,"objects_trimmed":0,"snaptrim_duration":0.017091021000000001,"stat_sum":{"num_bytes":951410,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":88,"num_read_kb":19324,"num_write":674,"num_write_kb":26724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"132'152","reported_seq":320,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.682795+0000","last_change":"2023-10-10T19:29:29.682765+0000","last_active":"2023-10-10T19:29:29.682795+0000","last_peered":"2023-10-10T19:29:29.682795+0000","last_clean":"2023-10-10T19:29:29.682795+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T19:29:29.682795+0000","last_undegraded":"2023-10-10T19:29:29.682795+0000","last_fullsized":"2023-10-10T19:29:29.682795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'152","last_scrub_stamp":"2023-10-10T19:29:29.682678+0000","last_deep_scrub":"132'152","last_deep_scrub_stamp":"2023-10-10T19:29:29.682678+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.682678+0000","objects_scrubbed":15,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:06:08.947475+0000","scrub_duration":0.025814232999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388756,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":36939,"num_write":74,"num_write_kb":8306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":75158,"num_omap_keys":136,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"132'119","reported_seq":287,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.097607+0000","last_change":"2023-10-10T19:29:40.097581+0000","last_active":"2023-10-10T19:29:40.097607+0000","last_peered":"2023-10-10T19:29:40.097607+0000","last_clean":"2023-10-10T19:29:40.097607+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T19:29:40.097607+0000","last_undegraded":"2023-10-10T19:29:40.097607+0000","last_fullsized":"2023-10-10T19:29:40.097607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'119","last_scrub_stamp":"2023-10-10T19:29:40.097457+0000","last_deep_scrub":"132'119","last_deep_scrub_stamp":"2023-10-10T19:29:40.097457+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.097457+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:06:14.657014+0000","scrub_duration":0.0047236919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"132'934","reported_seq":3569,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:44.310730+0000","last_change":"2023-10-10T19:29:44.310686+0000","last_active":"2023-10-10T19:29:44.310730+0000","last_peered":"2023-10-10T19:29:44.310730+0000","last_clean":"2023-10-10T19:29:44.310730+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T19:29:44.310730+0000","last_undegraded":"2023-10-10T19:29:44.310730+0000","last_fullsized":"2023-10-10T19:29:44.310730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'934","last_scrub_stamp":"2023-10-10T19:29:44.310602+0000","last_deep_scrub":"132'934","last_deep_scrub_stamp":"2023-10-10T19:29:44.310602+0000","last_clean_scrub_stamp":"2023-10-10T19:29:44.310602+0000","objects_scrubbed":39,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:39.883204+0000","scrub_duration":0.034571985999999999,"objects_trimmed":0,"snaptrim_duration":0.010927930000000001,"stat_sum":{"num_bytes":3454752,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":55,"num_read_kb":20626,"num_write":663,"num_write_kb":30766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"132'990","reported_seq":1152,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:41.209314+0000","last_change":"2023-10-10T19:29:41.209289+0000","last_active":"2023-10-10T19:29:41.209314+0000","last_peered":"2023-10-10T19:29:41.209314+0000","last_clean":"2023-10-10T19:29:41.209314+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T19:29:41.209314+0000","last_undegraded":"2023-10-10T19:29:41.209314+0000","last_fullsized":"2023-10-10T19:29:41.209314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'990","last_scrub_stamp":"2023-10-10T19:29:41.209202+0000","last_deep_scrub":"132'990","last_deep_scrub_stamp":"2023-10-10T19:29:41.209202+0000","last_clean_scrub_stamp":"2023-10-10T19:29:41.209202+0000","objects_scrubbed":17,"log_size":990,"log_dups_size":0,"ondisk_log_size":990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:08.926653+0000","scrub_duration":0.069287455999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":38,"num_read_kb":89926,"num_write":914,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70568,"num_omap_keys":125,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"132'839","reported_seq":1003,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.127383+0000","last_change":"2023-10-10T19:29:42.127330+0000","last_active":"2023-10-10T19:29:42.127383+0000","last_peered":"2023-10-10T19:29:42.127383+0000","last_clean":"2023-10-10T19:29:42.127383+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T19:29:42.127383+0000","last_undegraded":"2023-10-10T19:29:42.127383+0000","last_fullsized":"2023-10-10T19:29:42.127383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'839","last_scrub_stamp":"2023-10-10T19:29:42.127227+0000","last_deep_scrub":"132'839","last_deep_scrub_stamp":"2023-10-10T19:29:42.127227+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.127227+0000","objects_scrubbed":14,"log_size":839,"log_dups_size":0,"ondisk_log_size":839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:46:38.680084+0000","scrub_duration":0.027082854999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":45,"num_read_kb":45594,"num_write":766,"num_write_kb":12847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":79898,"num_omap_keys":140,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"132'1056","reported_seq":3681,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.647554+0000","last_change":"2023-10-10T19:29:48.647516+0000","last_active":"2023-10-10T19:29:48.647554+0000","last_peered":"2023-10-10T19:29:48.647554+0000","last_clean":"2023-10-10T19:29:48.647554+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T19:29:48.647554+0000","last_undegraded":"2023-10-10T19:29:48.647554+0000","last_fullsized":"2023-10-10T19:29:48.647554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1056","last_scrub_stamp":"2023-10-10T19:29:48.647374+0000","last_deep_scrub":"132'1056","last_deep_scrub_stamp":"2023-10-10T19:29:48.647374+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.647374+0000","objects_scrubbed":38,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:49:57.030894+0000","scrub_duration":0.028559443,"objects_trimmed":0,"snaptrim_duration":0.0075449510000000003,"stat_sum":{"num_bytes":524227,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":65,"num_read_kb":49387,"num_write":726,"num_write_kb":52412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'292","reported_seq":459,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.731119+0000","last_change":"2023-10-10T19:29:29.731097+0000","last_active":"2023-10-10T19:29:29.731119+0000","last_peered":"2023-10-10T19:29:29.731119+0000","last_clean":"2023-10-10T19:29:29.731119+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T19:29:29.731119+0000","last_undegraded":"2023-10-10T19:29:29.731119+0000","last_fullsized":"2023-10-10T19:29:29.731119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'292","last_scrub_stamp":"2023-10-10T19:29:29.731010+0000","last_deep_scrub":"132'292","last_deep_scrub_stamp":"2023-10-10T19:29:29.731010+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.731010+0000","objects_scrubbed":18,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:44:14.879878+0000","scrub_duration":0.021479405999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":81,"num_read_kb":20675,"num_write":479,"num_write_kb":4749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":103356,"num_omap_keys":179,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"132'957","reported_seq":3584,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.155401+0000","last_change":"2023-10-10T19:16:54.920290+0000","last_active":"2023-10-10T19:29:22.155401+0000","last_peered":"2023-10-10T19:29:22.155401+0000","last_clean":"2023-10-10T19:29:22.155401+0000","last_became_active":"2023-10-10T18:46:01.267092+0000","last_became_peered":"2023-10-10T18:46:01.267092+0000","last_unstale":"2023-10-10T19:29:22.155401+0000","last_undegraded":"2023-10-10T19:29:22.155401+0000","last_fullsized":"2023-10-10T19:29:22.155401+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083609700000000006,"stat_sum":{"num_bytes":2285567,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":68,"num_read_kb":33081,"num_write":691,"num_write_kb":42143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"132'166","reported_seq":324,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.151113+0000","last_change":"2023-10-10T19:29:29.151081+0000","last_active":"2023-10-10T19:29:29.151113+0000","last_peered":"2023-10-10T19:29:29.151113+0000","last_clean":"2023-10-10T19:29:29.151113+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T19:29:29.151113+0000","last_undegraded":"2023-10-10T19:29:29.151113+0000","last_fullsized":"2023-10-10T19:29:29.151113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'166","last_scrub_stamp":"2023-10-10T19:29:29.150987+0000","last_deep_scrub":"132'166","last_deep_scrub_stamp":"2023-10-10T19:29:29.150987+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.150987+0000","objects_scrubbed":12,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:52:58.469784+0000","scrub_duration":0.0090081499999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":8269,"num_write":94,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64552,"num_omap_keys":116,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"132'914","reported_seq":3586,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:43.257893+0000","last_change":"2023-10-10T19:29:43.257858+0000","last_active":"2023-10-10T19:29:43.257893+0000","last_peered":"2023-10-10T19:29:43.257893+0000","last_clean":"2023-10-10T19:29:43.257893+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T19:29:43.257893+0000","last_undegraded":"2023-10-10T19:29:43.257893+0000","last_fullsized":"2023-10-10T19:29:43.257893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'914","last_scrub_stamp":"2023-10-10T19:29:43.257766+0000","last_deep_scrub":"132'914","last_deep_scrub_stamp":"2023-10-10T19:29:43.257766+0000","last_clean_scrub_stamp":"2023-10-10T19:29:43.257766+0000","objects_scrubbed":42,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:41.946443+0000","scrub_duration":0.029919424999999999,"objects_trimmed":0,"snaptrim_duration":0.012721872,"stat_sum":{"num_bytes":482904,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":85,"num_read_kb":32652,"num_write":648,"num_write_kb":36187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"131'228","reported_seq":372,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:04.557834+0000","last_change":"2023-10-10T18:45:10.415402+0000","last_active":"2023-10-10T19:29:04.557834+0000","last_peered":"2023-10-10T19:29:04.557834+0000","last_clean":"2023-10-10T19:29:04.557834+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T19:29:04.557834+0000","last_undegraded":"2023-10-10T19:29:04.557834+0000","last_fullsized":"2023-10-10T19:29:04.557834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:09.383211+0000","last_clean_scrub_stamp":"2023-10-10T18:45:09.383211+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":57389,"num_write":173,"num_write_kb":12477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"132'971","reported_seq":3622,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.000295+0000","last_change":"2023-10-10T19:29:45.000264+0000","last_active":"2023-10-10T19:29:45.000295+0000","last_peered":"2023-10-10T19:29:45.000295+0000","last_clean":"2023-10-10T19:29:45.000295+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T19:29:45.000295+0000","last_undegraded":"2023-10-10T19:29:45.000295+0000","last_fullsized":"2023-10-10T19:29:45.000295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'971","last_scrub_stamp":"2023-10-10T19:29:45.000170+0000","last_deep_scrub":"132'971","last_deep_scrub_stamp":"2023-10-10T19:29:45.000170+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.000170+0000","objects_scrubbed":32,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:53:24.335195+0000","scrub_duration":0.025238327000000001,"objects_trimmed":0,"snaptrim_duration":0.0089059480000000003,"stat_sum":{"num_bytes":820479,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":67,"num_read_kb":26489,"num_write":692,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"132'387","reported_seq":550,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.747339+0000","last_change":"2023-10-10T19:29:28.747314+0000","last_active":"2023-10-10T19:29:28.747339+0000","last_peered":"2023-10-10T19:29:28.747339+0000","last_clean":"2023-10-10T19:29:28.747339+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T19:29:28.747339+0000","last_undegraded":"2023-10-10T19:29:28.747339+0000","last_fullsized":"2023-10-10T19:29:28.747339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'387","last_scrub_stamp":"2023-10-10T19:29:28.747246+0000","last_deep_scrub":"132'387","last_deep_scrub_stamp":"2023-10-10T19:29:28.747246+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.747246+0000","objects_scrubbed":19,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:01:46.340517+0000","scrub_duration":0.020997531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":70,"num_read_kb":20656,"num_write":333,"num_write_kb":8747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":128481,"num_omap_keys":203,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"132'908","reported_seq":3543,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.151920+0000","last_change":"2023-10-10T19:16:54.870214+0000","last_active":"2023-10-10T19:29:22.151920+0000","last_peered":"2023-10-10T19:29:22.151920+0000","last_clean":"2023-10-10T19:29:22.151920+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T19:29:22.151920+0000","last_undegraded":"2023-10-10T19:29:22.151920+0000","last_fullsized":"2023-10-10T19:29:22.151920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012052205,"stat_sum":{"num_bytes":635543,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":63,"num_read_kb":40064,"num_write":667,"num_write_kb":56874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"132'1323","reported_seq":1489,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:34.027397+0000","last_change":"2023-10-10T19:29:31.691013+0000","last_active":"2023-10-10T19:29:34.027397+0000","last_peered":"2023-10-10T19:29:34.027397+0000","last_clean":"2023-10-10T19:29:34.027397+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T19:29:34.027397+0000","last_undegraded":"2023-10-10T19:29:34.027397+0000","last_fullsized":"2023-10-10T19:29:34.027397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1319","last_scrub_stamp":"2023-10-10T19:29:31.690927+0000","last_deep_scrub":"132'1319","last_deep_scrub_stamp":"2023-10-10T19:29:31.690927+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.690927+0000","objects_scrubbed":14,"log_size":1323,"log_dups_size":0,"ondisk_log_size":1323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:38:48.082901+0000","scrub_duration":0.043989502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15156641,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":48,"num_read_kb":53317,"num_write":1724,"num_write_kb":24884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40110,"num_omap_keys":72,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"132'893","reported_seq":3526,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.158256+0000","last_change":"2023-10-10T19:16:54.874295+0000","last_active":"2023-10-10T19:29:22.158256+0000","last_peered":"2023-10-10T19:29:22.158256+0000","last_clean":"2023-10-10T19:29:22.158256+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T19:29:22.158256+0000","last_undegraded":"2023-10-10T19:29:22.158256+0000","last_fullsized":"2023-10-10T19:29:22.158256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015827366999999998,"stat_sum":{"num_bytes":178682,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":58,"num_read_kb":28965,"num_write":652,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"132'1215","reported_seq":1380,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.115778+0000","last_change":"2023-10-10T19:29:45.115751+0000","last_active":"2023-10-10T19:29:45.115778+0000","last_peered":"2023-10-10T19:29:45.115778+0000","last_clean":"2023-10-10T19:29:45.115778+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T19:29:45.115778+0000","last_undegraded":"2023-10-10T19:29:45.115778+0000","last_fullsized":"2023-10-10T19:29:45.115778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1215","last_scrub_stamp":"2023-10-10T19:29:45.115661+0000","last_deep_scrub":"132'1215","last_deep_scrub_stamp":"2023-10-10T19:29:45.115661+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.115661+0000","objects_scrubbed":16,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:01.423877+0000","scrub_duration":0.048047320999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":49,"num_read_kb":89747,"num_write":1128,"num_write_kb":25455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70296,"num_omap_keys":124,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"132'938","reported_seq":3567,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.144929+0000","last_change":"2023-10-10T19:16:54.781695+0000","last_active":"2023-10-10T19:29:22.144929+0000","last_peered":"2023-10-10T19:29:22.144929+0000","last_clean":"2023-10-10T19:29:22.144929+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T19:29:22.144929+0000","last_undegraded":"2023-10-10T19:29:22.144929+0000","last_fullsized":"2023-10-10T19:29:22.144929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065052069999999998,"stat_sum":{"num_bytes":479125,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":64,"num_read_kb":10527,"num_write":680,"num_write_kb":20069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"132'323","reported_seq":474,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:34.384672+0000","last_change":"2023-10-10T19:29:34.384643+0000","last_active":"2023-10-10T19:29:34.384672+0000","last_peered":"2023-10-10T19:29:34.384672+0000","last_clean":"2023-10-10T19:29:34.384672+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T19:29:34.384672+0000","last_undegraded":"2023-10-10T19:29:34.384672+0000","last_fullsized":"2023-10-10T19:29:34.384672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'323","last_scrub_stamp":"2023-10-10T19:29:34.384547+0000","last_deep_scrub":"132'323","last_deep_scrub_stamp":"2023-10-10T19:29:34.384547+0000","last_clean_scrub_stamp":"2023-10-10T19:29:34.384547+0000","objects_scrubbed":12,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:06:08.523893+0000","scrub_duration":0.015112752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194858,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":28,"num_read_kb":40985,"num_write":233,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":71992,"num_omap_keys":129,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"132'919","reported_seq":3570,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.166601+0000","last_change":"2023-10-10T19:16:54.794460+0000","last_active":"2023-10-10T19:29:22.166601+0000","last_peered":"2023-10-10T19:29:22.166601+0000","last_clean":"2023-10-10T19:29:22.166601+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T19:29:22.166601+0000","last_undegraded":"2023-10-10T19:29:22.166601+0000","last_fullsized":"2023-10-10T19:29:22.166601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016889265000000001,"stat_sum":{"num_bytes":1346662,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":67,"num_read_kb":24822,"num_write":699,"num_write_kb":32529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"132'354","reported_seq":509,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.079495+0000","last_change":"2023-10-10T19:29:37.661226+0000","last_active":"2023-10-10T19:29:40.079495+0000","last_peered":"2023-10-10T19:29:40.079495+0000","last_clean":"2023-10-10T19:29:40.079495+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T19:29:40.079495+0000","last_undegraded":"2023-10-10T19:29:40.079495+0000","last_fullsized":"2023-10-10T19:29:40.079495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'352","last_scrub_stamp":"2023-10-10T19:29:37.661098+0000","last_deep_scrub":"132'352","last_deep_scrub_stamp":"2023-10-10T19:29:37.661098+0000","last_clean_scrub_stamp":"2023-10-10T19:29:37.661098+0000","objects_scrubbed":11,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:14:09.979735+0000","scrub_duration":0.032683222999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243254,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":27,"num_read_kb":50704,"num_write":274,"num_write_kb":15292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":68490,"num_omap_keys":115,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"132'970","reported_seq":3598,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.156927+0000","last_change":"2023-10-10T19:16:55.417128+0000","last_active":"2023-10-10T19:29:22.156927+0000","last_peered":"2023-10-10T19:29:22.156927+0000","last_clean":"2023-10-10T19:29:22.156927+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T19:29:22.156927+0000","last_undegraded":"2023-10-10T19:29:22.156927+0000","last_fullsized":"2023-10-10T19:29:22.156927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012585610000000001,"stat_sum":{"num_bytes":222806,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":60,"num_read_kb":19964,"num_write":709,"num_write_kb":27205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"132'322","reported_seq":481,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.316606+0000","last_change":"2023-10-10T19:29:31.316564+0000","last_active":"2023-10-10T19:29:31.316606+0000","last_peered":"2023-10-10T19:29:31.316606+0000","last_clean":"2023-10-10T19:29:31.316606+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T19:29:31.316606+0000","last_undegraded":"2023-10-10T19:29:31.316606+0000","last_fullsized":"2023-10-10T19:29:31.316606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'322","last_scrub_stamp":"2023-10-10T19:29:31.316447+0000","last_deep_scrub":"132'322","last_deep_scrub_stamp":"2023-10-10T19:29:31.316447+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.316447+0000","objects_scrubbed":12,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:27:45.094335+0000","scrub_duration":0.025432287000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":49,"num_read_kb":57404,"num_write":260,"num_write_kb":8464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":65503,"num_omap_keys":117,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"132'1015","reported_seq":3645,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.160463+0000","last_change":"2023-10-10T19:16:54.794658+0000","last_active":"2023-10-10T19:29:22.160463+0000","last_peered":"2023-10-10T19:29:22.160463+0000","last_clean":"2023-10-10T19:29:22.160463+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T19:29:22.160463+0000","last_undegraded":"2023-10-10T19:29:22.160463+0000","last_fullsized":"2023-10-10T19:29:22.160463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1015,"log_dups_size":0,"ondisk_log_size":1015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016837566000000002,"stat_sum":{"num_bytes":394703,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":67,"num_read_kb":28062,"num_write":753,"num_write_kb":23722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"132'910","reported_seq":3571,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.149293+0000","last_change":"2023-10-10T19:16:54.782474+0000","last_active":"2023-10-10T19:29:22.149293+0000","last_peered":"2023-10-10T19:29:22.149293+0000","last_clean":"2023-10-10T19:29:22.149293+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T19:29:22.149293+0000","last_undegraded":"2023-10-10T19:29:22.149293+0000","last_fullsized":"2023-10-10T19:29:22.149293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074078640000000001,"stat_sum":{"num_bytes":519476,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":77,"num_read_kb":25056,"num_write":683,"num_write_kb":32109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"132'1017","reported_seq":3650,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:41.182730+0000","last_change":"2023-10-10T19:29:41.182702+0000","last_active":"2023-10-10T19:29:41.182730+0000","last_peered":"2023-10-10T19:29:41.182730+0000","last_clean":"2023-10-10T19:29:41.182730+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T19:29:41.182730+0000","last_undegraded":"2023-10-10T19:29:41.182730+0000","last_fullsized":"2023-10-10T19:29:41.182730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1017","last_scrub_stamp":"2023-10-10T19:29:41.182614+0000","last_deep_scrub":"132'1017","last_deep_scrub_stamp":"2023-10-10T19:29:41.182614+0000","last_clean_scrub_stamp":"2023-10-10T19:29:41.182614+0000","objects_scrubbed":32,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:38:55.992116+0000","scrub_duration":0.028001608000000001,"objects_trimmed":0,"snaptrim_duration":0.0050938800000000003,"stat_sum":{"num_bytes":416129,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":69,"num_read_kb":25013,"num_write":702,"num_write_kb":37226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"132'1001","reported_seq":3660,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.132565+0000","last_change":"2023-10-10T19:29:47.132524+0000","last_active":"2023-10-10T19:29:47.132565+0000","last_peered":"2023-10-10T19:29:47.132565+0000","last_clean":"2023-10-10T19:29:47.132565+0000","last_became_active":"2023-10-10T18:46:00.606420+0000","last_became_peered":"2023-10-10T18:46:00.606420+0000","last_unstale":"2023-10-10T19:29:47.132565+0000","last_undegraded":"2023-10-10T19:29:47.132565+0000","last_fullsized":"2023-10-10T19:29:47.132565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1001","last_scrub_stamp":"2023-10-10T19:29:47.132433+0000","last_deep_scrub":"132'1001","last_deep_scrub_stamp":"2023-10-10T19:29:47.132433+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.132433+0000","objects_scrubbed":41,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:35:01.992844+0000","scrub_duration":0.032812747000000003,"objects_trimmed":0,"snaptrim_duration":0.0064370249999999999,"stat_sum":{"num_bytes":103727,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":90,"num_read_kb":45570,"num_write":716,"num_write_kb":52505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"132'973","reported_seq":3611,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:49.279974+0000","last_change":"2023-10-10T19:29:49.279944+0000","last_active":"2023-10-10T19:29:49.279974+0000","last_peered":"2023-10-10T19:29:49.279974+0000","last_clean":"2023-10-10T19:29:49.279974+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T19:29:49.279974+0000","last_undegraded":"2023-10-10T19:29:49.279974+0000","last_fullsized":"2023-10-10T19:29:49.279974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'973","last_scrub_stamp":"2023-10-10T19:29:49.279855+0000","last_deep_scrub":"132'973","last_deep_scrub_stamp":"2023-10-10T19:29:49.279855+0000","last_clean_scrub_stamp":"2023-10-10T19:29:49.279855+0000","objects_scrubbed":37,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:38.764535+0000","scrub_duration":0.031787211000000003,"objects_trimmed":0,"snaptrim_duration":0.014096529,"stat_sum":{"num_bytes":456616,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":70,"num_read_kb":16093,"num_write":668,"num_write_kb":20834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"132'918","reported_seq":3556,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157935+0000","last_change":"2023-10-10T19:16:54.875626+0000","last_active":"2023-10-10T19:29:22.157935+0000","last_peered":"2023-10-10T19:29:22.157935+0000","last_clean":"2023-10-10T19:29:22.157935+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T19:29:22.157935+0000","last_undegraded":"2023-10-10T19:29:22.157935+0000","last_fullsized":"2023-10-10T19:29:22.157935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016808101999999998,"stat_sum":{"num_bytes":708082,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":68,"num_read_kb":48841,"num_write":679,"num_write_kb":56033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"132'979","reported_seq":3582,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.144178+0000","last_change":"2023-10-10T19:16:54.918525+0000","last_active":"2023-10-10T19:29:22.144178+0000","last_peered":"2023-10-10T19:29:22.144178+0000","last_clean":"2023-10-10T19:29:22.144178+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T19:29:22.144178+0000","last_undegraded":"2023-10-10T19:29:22.144178+0000","last_fullsized":"2023-10-10T19:29:22.144178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":979,"log_dups_size":0,"ondisk_log_size":979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068654069999999996,"stat_sum":{"num_bytes":747011,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":59,"num_read_kb":41116,"num_write":675,"num_write_kb":48421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"132'978","reported_seq":3601,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.163335+0000","last_change":"2023-10-10T19:16:54.783685+0000","last_active":"2023-10-10T19:29:22.163335+0000","last_peered":"2023-10-10T19:29:22.163335+0000","last_clean":"2023-10-10T19:29:22.163335+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T19:29:22.163335+0000","last_undegraded":"2023-10-10T19:29:22.163335+0000","last_fullsized":"2023-10-10T19:29:22.163335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081260870000000006,"stat_sum":{"num_bytes":4820268,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":70,"num_read_kb":15871,"num_write":698,"num_write_kb":28703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"132'933","reported_seq":3570,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.149001+0000","last_change":"2023-10-10T19:16:54.877012+0000","last_active":"2023-10-10T19:29:22.149001+0000","last_peered":"2023-10-10T19:29:22.149001+0000","last_clean":"2023-10-10T19:29:22.149001+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T19:29:22.149001+0000","last_undegraded":"2023-10-10T19:29:22.149001+0000","last_fullsized":"2023-10-10T19:29:22.149001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017981724000000001,"stat_sum":{"num_bytes":834726,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":72,"num_read_kb":44513,"num_write":669,"num_write_kb":47263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"132'911","reported_seq":3555,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.161954+0000","last_change":"2023-10-10T19:16:55.415035+0000","last_active":"2023-10-10T19:29:22.161954+0000","last_peered":"2023-10-10T19:29:22.161954+0000","last_clean":"2023-10-10T19:29:22.161954+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T19:29:22.161954+0000","last_undegraded":"2023-10-10T19:29:22.161954+0000","last_fullsized":"2023-10-10T19:29:22.161954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010756557999999999,"stat_sum":{"num_bytes":807796,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":71,"num_read_kb":21097,"num_write":685,"num_write_kb":25593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"132'896","reported_seq":3534,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.154322+0000","last_change":"2023-10-10T19:16:54.872190+0000","last_active":"2023-10-10T19:29:22.154322+0000","last_peered":"2023-10-10T19:29:22.154322+0000","last_clean":"2023-10-10T19:29:22.154322+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T19:29:22.154322+0000","last_undegraded":"2023-10-10T19:29:22.154322+0000","last_fullsized":"2023-10-10T19:29:22.154322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014012482999999999,"stat_sum":{"num_bytes":684647,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":69,"num_read_kb":49258,"num_write":638,"num_write_kb":56905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"132'959","reported_seq":3593,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.338440+0000","last_change":"2023-10-10T19:16:55.413906+0000","last_active":"2023-10-10T19:29:22.338440+0000","last_peered":"2023-10-10T19:29:22.338440+0000","last_clean":"2023-10-10T19:29:22.338440+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T19:29:22.338440+0000","last_undegraded":"2023-10-10T19:29:22.338440+0000","last_fullsized":"2023-10-10T19:29:22.338440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097177140000000006,"stat_sum":{"num_bytes":571513,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":59,"num_read_kb":20609,"num_write":714,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"132'887","reported_seq":3547,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.166197+0000","last_change":"2023-10-10T19:16:54.983331+0000","last_active":"2023-10-10T19:29:22.166197+0000","last_peered":"2023-10-10T19:29:22.166197+0000","last_clean":"2023-10-10T19:29:22.166197+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T19:29:22.166197+0000","last_undegraded":"2023-10-10T19:29:22.166197+0000","last_fullsized":"2023-10-10T19:29:22.166197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013935637000000001,"stat_sum":{"num_bytes":366510,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":77,"num_read_kb":51992,"num_write":664,"num_write_kb":56578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"132'1013","reported_seq":3656,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.157657+0000","last_change":"2023-10-10T19:16:54.784919+0000","last_active":"2023-10-10T19:29:22.157657+0000","last_peered":"2023-10-10T19:29:22.157657+0000","last_clean":"2023-10-10T19:29:22.157657+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T19:29:22.157657+0000","last_undegraded":"2023-10-10T19:29:22.157657+0000","last_fullsized":"2023-10-10T19:29:22.157657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088155260000000006,"stat_sum":{"num_bytes":278227,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":77,"num_read_kb":37312,"num_write":744,"num_write_kb":49182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":50927477,"num_objects":2366,"num_object_clones":0,"num_object_copies":4732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":4408,"num_read_kb":1953419,"num_write":44061,"num_write_kb":2517898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":76832768,"data_stored":101854954,"data_compressed":30508016,"data_compressed_allocated":33341440,"data_compressed_original":70082560,"omap_allocated":0,"internal_metadata":0},"log_size":61220,"ondisk_log_size":61220,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":336750859,"num_objects":493,"num_object_clones":0,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":2858,"num_read_kb":1531013,"num_write":19189,"num_write_kb":469807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":353,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2328994,"num_omap_keys":4066,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183246848,"data_stored":673501718,"data_compressed":87247406,"data_compressed_allocated":112312320,"data_compressed_original":602816512,"omap_allocated":0,"internal_metadata":0},"log_size":20244,"ondisk_log_size":20244,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575389,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348880,"kb_used_data":47496,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94022960,"statfs":{"total":96636764160,"available":96279511040,"internally_reserved":0,"allocated":48635904,"data_stored":134725171,"data_compressed":20528468,"data_compressed_allocated":25550848,"data_compressed_original":114077696,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.53700000000000003},"min":{"1min":0.33500000000000002,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.72799999999999998,"5min":0.748,"15min":1.018},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.25},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":2.153},"last":0.54500000000000004}]},{"osd":1,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.34799999999999998,"5min":0.318,"15min":0.26200000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":1.802},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57899999999999996,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72599999999999998,"5min":0.82399999999999995,"15min":2.2170000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.44900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.76600000000000001,"5min":2.871,"15min":2.871},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":2.1989999999999998},"last":0.68899999999999995}]},{"osd":3,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.34699999999999998,"15min":0.317},"max":{"1min":0.67000000000000004,"5min":0.874,"15min":1.0349999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.28100000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.7649999999999999},"last":0.59499999999999997}]},{"osd":4,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.47599999999999998},"min":{"1min":0.35199999999999998,"5min":0.307,"15min":0.28000000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.089},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55800000000000005,"15min":0.54200000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.72399999999999998,"5min":0.78400000000000003,"15min":1.9419999999999999},"last":0.40500000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.502,"15min":0.51400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":1.71},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.42699999999999999,"5min":0.378,"15min":0.311},"max":{"1min":0.79800000000000004,"5min":0.80500000000000005,"15min":1.177},"last":0.49299999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60099999999999998,"15min":0.58599999999999997},"min":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.31},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.872},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.33900000000000002,"15min":0.29699999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":2.2559999999999998},"last":0.56399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575388,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333408,"kb_used_data":36120,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94038432,"statfs":{"total":96636764160,"available":96295354368,"internally_reserved":0,"allocated":36986880,"data_stored":92212231,"data_compressed":11950270,"data_compressed_allocated":14790656,"data_compressed_original":71962624,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.80600000000000005,"15min":0.81100000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.216},"max":{"1min":0.68300000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.372}]},{"osd":1,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58699999999999997,"15min":0.56000000000000005},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.85299999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.56499999999999995},"min":{"1min":0.41799999999999998,"5min":0.379,"15min":0.222},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.86799999999999999},"last":0.57499999999999996}]},{"osd":2,"last update":"Tue Oct 10 19:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.56200000000000006},"min":{"1min":0.48699999999999999,"5min":0.40400000000000003,"15min":0.30499999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90500000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.55300000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.29999999999999999},"max":{"1min":0.67500000000000004,"5min":0.76500000000000001,"15min":0.97599999999999998},"last":0.46200000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57999999999999996,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.32900000000000001},"max":{"1min":0.628,"5min":0.82799999999999996,"15min":0.88200000000000001},"last":0.433},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.437,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.85699999999999998,"15min":0.95099999999999996},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48199999999999998,"15min":0.46700000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.248},"max":{"1min":0.749,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54900000000000004,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.80300000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.51000000000000001}]},{"osd":5,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73599999999999999,"15min":0.879},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51900000000000002,"15min":0.53400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.26400000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.99299999999999999},"last":0.53200000000000003}]},{"osd":7,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.31},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.88900000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.38,"15min":0.27700000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.86199999999999999},"last":0.54600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575387,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342408,"kb_used_data":32832,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94029432,"statfs":{"total":96636764160,"available":96286138368,"internally_reserved":0,"allocated":33619968,"data_stored":116844804,"data_compressed":17796649,"data_compressed_allocated":21954560,"data_compressed_original":106926080,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.31,"15min":0.315},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17699999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.56899999999999995},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41999999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":0.66700000000000004,"15min":0.70799999999999996},"last":0.437}]},{"osd":2,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.34899999999999998,"15min":0.35399999999999998},"min":{"1min":0.246,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.55200000000000005},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38800000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.19900000000000001},"max":{"1min":0.56699999999999995,"5min":0.64900000000000002,"15min":0.68899999999999995},"last":0.39400000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56299999999999994,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.27700000000000002},"max":{"1min":0.66700000000000004,"5min":0.72899999999999998,"15min":0.84299999999999997},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.41999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.175},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.875,"5min":0.875,"15min":2.2519999999999998},"last":0.46600000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.90900000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.501,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.42199999999999999}]},{"osd":6,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.46100000000000002,"5min":0.40500000000000003,"15min":0.37},"max":{"1min":0.81200000000000006,"5min":0.86199999999999999,"15min":1.038},"last":0.503},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.38900000000000001,"15min":0.36299999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.123},"last":0.56399999999999995}]},{"osd":7,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.621},"min":{"1min":0.46700000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":5.2720000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61899999999999999,"15min":0.64800000000000002},"min":{"1min":0.47899999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":5.2599999999999998},"last":0.49199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575389,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338068,"kb_used_data":32588,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94033772,"statfs":{"total":96636764160,"available":96290582528,"internally_reserved":0,"allocated":33370112,"data_stored":88805479,"data_compressed":14490668,"data_compressed_allocated":17825792,"data_compressed_original":75632640,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.66500000000000004},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42399999999999999,"15min":0.41699999999999998},"min":{"1min":0.307,"5min":0.25800000000000001,"15min":0.20300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66000000000000003,"15min":0.66500000000000004},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35999999999999999,"15min":0.372},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.52000000000000002,"5min":0.52500000000000002,"15min":2.028},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40799999999999997},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53400000000000003,"5min":0.61299999999999999,"15min":0.77300000000000002},"last":0.41299999999999998}]},{"osd":3,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.39600000000000002},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.79100000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.80300000000000005},"last":0.42799999999999999}]},{"osd":4,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.629,"15min":0.63100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.26000000000000001},"max":{"1min":0.84399999999999997,"5min":0.97599999999999998,"15min":1.9610000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.38400000000000001},"max":{"1min":0.877,"5min":0.999,"15min":0.999},"last":0.73099999999999998}]},{"osd":5,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.35299999999999998},"max":{"1min":0.81799999999999995,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72699999999999998,"5min":0.873,"15min":0.96799999999999997},"last":0.71899999999999997}]},{"osd":6,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":0.85699999999999998,"5min":0.96299999999999997,"15min":2.0019999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.627,"15min":0.62},"min":{"1min":0.43099999999999999,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":0.73599999999999999,"5min":0.94099999999999995,"15min":0.95399999999999996},"last":0.56100000000000005}]},{"osd":7,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.63100000000000001,"15min":0.63400000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.63700000000000001,"15min":0.64900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.80100000000000005,"5min":0.89500000000000002,"15min":2.0539999999999998},"last":0.52200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575389,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319288,"kb_used_data":17904,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94052552,"statfs":{"total":96636764160,"available":96309813248,"internally_reserved":0,"allocated":18333696,"data_stored":57320502,"data_compressed":8516380,"data_compressed_allocated":10579968,"data_compressed_original":51261440,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34399999999999997,"15min":0.33500000000000002},"min":{"1min":0.224,"5min":0.219,"15min":0.186},"max":{"1min":0.45000000000000001,"5min":0.55400000000000005,"15min":0.60799999999999998},"last":0.254},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.53500000000000003,"5min":0.65100000000000002,"15min":0.67500000000000004},"last":0.36499999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20999999999999999},"max":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.65700000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.73999999999999999},"last":0.40899999999999997}]},{"osd":3,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.24399999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.42499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.23899999999999999},"max":{"1min":0.59099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.379}]},{"osd":4,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.76900000000000002},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.26600000000000001},"max":{"1min":0.751,"5min":0.95599999999999996,"15min":48.729999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.25900000000000001},"max":{"1min":0.74099999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.505}]},{"osd":5,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63,"15min":0.628},"min":{"1min":0.436,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":1.02,"15min":1.02},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.32400000000000001},"max":{"1min":0.81000000000000005,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.56699999999999995}]},{"osd":6,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.78900000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":1.0760000000000001,"15min":48.685000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.38800000000000001},"max":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":0.97899999999999998},"last":0.60899999999999999}]},{"osd":7,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64800000000000002,"15min":0.80700000000000005},"min":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.38700000000000001},"max":{"1min":0.85799999999999998,"5min":1.052,"15min":48.710000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.034,"15min":1.1459999999999999},"last":0.621}]}]},{"osd":3,"up_from":13,"seq":55834575388,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347936,"kb_used_data":38360,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94023904,"statfs":{"total":96636764160,"available":96280477696,"internally_reserved":0,"allocated":39280640,"data_stored":100206201,"data_compressed":14057817,"data_compressed_allocated":17813504,"data_compressed_original":81215488,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.37},"min":{"1min":0.23499999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.47599999999999998,"5min":0.64900000000000002,"15min":5.0670000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.371,"15min":0.38700000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.189},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.7799999999999998},"last":0.30399999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.373,"15min":0.379},"min":{"1min":0.25900000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.54600000000000004,"5min":0.628,"15min":2.758},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39300000000000002,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.64500000000000002},"last":0.34300000000000003}]},{"osd":2,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.25},"max":{"1min":0.60099999999999998,"5min":0.68400000000000005,"15min":1.333},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.41999999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.24199999999999999},"max":{"1min":0.56100000000000005,"5min":0.65400000000000003,"15min":0.98699999999999999},"last":0.41099999999999998}]},{"osd":4,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.623,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79600000000000004,"5min":0.89600000000000002,"15min":0.95499999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63400000000000001,"15min":0.63700000000000001},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":1.0169999999999999,"15min":1.23},"last":0.48099999999999998}]},{"osd":5,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.84199999999999997,"5min":0.97699999999999998,"15min":1.2929999999999999},"last":0.439},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.44800000000000001,"5min":0.40500000000000003,"15min":0.36199999999999999},"max":{"1min":0.875,"5min":0.93600000000000005,"15min":1.103},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.64100000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40300000000000002},"max":{"1min":0.90600000000000003,"5min":0.94199999999999995,"15min":1.198},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.88500000000000001,"5min":0.94899999999999995,"15min":2.7309999999999999},"last":0.51800000000000002}]},{"osd":7,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.39400000000000002},"max":{"1min":0.85699999999999998,"5min":0.97999999999999998,"15min":2.6619999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.49299999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86399999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.60199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575388,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344796,"kb_used_data":35220,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94027044,"statfs":{"total":96636764160,"available":96283693056,"internally_reserved":0,"allocated":36065280,"data_stored":105582593,"data_compressed":16193482,"data_compressed_allocated":19931136,"data_compressed_original":91820032,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33400000000000002,"5min":0.27600000000000002,"15min":0.21199999999999999},"max":{"1min":0.69599999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59899999999999998,"15min":0.58299999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.23699999999999999},"max":{"1min":0.74399999999999999,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.55400000000000005}]},{"osd":1,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.35599999999999998,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":0.67600000000000005,"5min":0.71199999999999997,"15min":1.5920000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":1.0169999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.42599999999999999,"5min":0.41999999999999998,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":1.615},"last":0.626},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.40100000000000002,"15min":0.36699999999999999},"max":{"1min":0.77800000000000002,"5min":0.85799999999999998,"15min":0.98199999999999998},"last":0.63900000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.57999999999999996},"min":{"1min":0.376,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.746,"5min":0.85199999999999998,"15min":0.85799999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.28199999999999997},"max":{"1min":0.78100000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.67700000000000005}]},{"osd":5,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.47099999999999997},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.26300000000000001},"max":{"1min":0.71599999999999997,"5min":0.84199999999999997,"15min":1.087},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.82399999999999995,"5min":0.88,"15min":1.111},"last":0.66200000000000003}]},{"osd":6,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.29799999999999999},"max":{"1min":0.82599999999999996,"5min":0.83099999999999996,"15min":1.5580000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.98799999999999999},"last":0.70699999999999996}]},{"osd":7,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.45200000000000001,"5min":0.37,"15min":0.312},"max":{"1min":0.86099999999999999,"5min":1.016,"15min":1.0620000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.64200000000000002},"min":{"1min":0.503,"5min":0.40000000000000002,"15min":0.33700000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":1.1160000000000001},"last":0.69899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331424,"kb_used_data":25944,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94040416,"statfs":{"total":96636764160,"available":96297385984,"internally_reserved":0,"allocated":26566656,"data_stored":87927915,"data_compressed":14286216,"data_compressed_allocated":17297408,"data_compressed_original":81149952,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.249},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.82199999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.752,"5min":0.76700000000000002,"15min":0.85099999999999998},"last":0.30099999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58499999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.314},"max":{"1min":0.78600000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.57499999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.28100000000000003},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81899999999999995},"last":0.35799999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.54200000000000004},"min":{"1min":0.442,"5min":0.35499999999999998,"15min":0.30499999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.28100000000000003},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.496}]},{"osd":3,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57199999999999995,"15min":0.55500000000000005},"min":{"1min":0.39600000000000002,"5min":0.377,"15min":0.32600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.76600000000000001,"5min":0.81799999999999995,"15min":0.94899999999999995},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.45800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57699999999999996,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.23000000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.50700000000000001}]},{"osd":6,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.502,"15min":0.496},"min":{"1min":0.316,"5min":0.316,"15min":0.26700000000000002},"max":{"1min":0.75800000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.77800000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.46600000000000003}]},{"osd":7,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79600000000000004,"5min":0.80000000000000004,"15min":0.89400000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61099999999999999,"15min":0.60199999999999998},"min":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.27700000000000002},"max":{"1min":0.82199999999999995,"5min":0.82599999999999996,"15min":0.90800000000000003},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20922368,"data_stored":71948432,"data_compressed":9641340,"data_compressed_allocated":12496896,"data_compressed_original":63553536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25812992,"data_stored":106092517,"data_compressed":13837835,"data_compressed_allocated":17661952,"data_compressed_original":97968128,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":51052075,"data_compressed":6450195,"data_compressed_allocated":8323072,"data_compressed_original":46788608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26288128,"data_stored":85264096,"data_compressed":10424017,"data_compressed_allocated":13803520,"data_compressed_original":72814592,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24764416,"data_stored":91914812,"data_compressed":12211424,"data_compressed_allocated":15597568,"data_compressed_original":82771968,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16703488,"data_stored":74680123,"data_compressed":9765423,"data_compressed_allocated":12320768,"data_compressed_original":70340608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":22110208,"data_stored":76551778,"data_compressed":9536145,"data_compressed_allocated":12132352,"data_compressed_original":66600960,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":34029568,"data_stored":115997885,"data_compressed":15381027,"data_compressed_allocated":19976192,"data_compressed_original":101978112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10801152,"data_stored":15380743,"data_compressed":4817064,"data_compressed_allocated":5283840,"data_compressed_original":11505664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6225920,"data_stored":9866351,"data_compressed":3958814,"data_compressed_allocated":4292608,"data_compressed_original":8957952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4136960,"data_stored":5382491,"data_compressed":2066185,"data_compressed_allocated":2256896,"data_compressed_original":4472832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11411456,"data_stored":14056169,"data_compressed":3633800,"data_compressed_allocated":4009984,"data_compressed_original":8400896,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9719808,"data_stored":12781845,"data_compressed":3982058,"data_compressed_allocated":4333568,"data_compressed_original":9048064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8282112,"data_stored":12361856,"data_compressed":4520793,"data_compressed_allocated":4976640,"data_compressed_original":10809344,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13295616,"data_stored":14774517,"data_compressed":2414125,"data_compressed_allocated":2658304,"data_compressed_original":5361664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12959744,"data_stored":17250982,"data_compressed":5115177,"data_compressed_allocated":5529600,"data_compressed_original":11526144,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T19:29:49.988 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.989 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.989 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.989 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.990 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.990 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.990 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.991 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.991 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.991 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.992 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.992 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.992 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.993 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.993 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.993 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.994 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.994 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.995 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.995 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.996 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.996 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.997 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.997 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.997 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.998 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.998 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.999 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.999 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:49.999 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-10T18:45:09.383211+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=9, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.000 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.000 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.000 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.001 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.001 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.002 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.002 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.002 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.003 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.003 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.004 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.004 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.004 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.005 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.005 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.005 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:29:50.006 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-10T19:29:57.812 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:29:57.815 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:30:10.007 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T19:30:10.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.136+0000 7f41f2adc700 1 Processor -- start 2023-10-10T19:30:10.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.136+0000 7f41f2adc700 1 -- start start 2023-10-10T19:30:10.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f2adc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 0x7f41ec159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:10.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f2adc700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f41ec0681e0 0x7f41ec0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:10.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f2adc700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec156800 con 0x7f41ec14ffc0 2023-10-10T19:30:10.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f2adc700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ec0c59e0 con 0x7f41ec0681e0 2023-10-10T19:30:10.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f2adc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec15a2b0 con 0x7f41ec14fbf0 2023-10-10T19:30:10.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f1079700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f41ec0681e0 0x7f41ec0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:10.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41ebfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f41ec14ffc0 0x7f41ec063230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:53780/0 (socket says 172.21.15.175:53780) 2023-10-10T19:30:10.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f1079700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f41ec0681e0 0x7f41ec0c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51220/0 (socket says 172.21.15.175:51220) 2023-10-10T19:30:10.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41ebfff700 1 -- 172.21.15.175:0/2478186630 learned_addr learned my addr 172.21.15.175:0/2478186630 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:30:10.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f0878700 1 --2- 172.21.15.175:0/2478186630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 0x7f41ec159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:10.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f1079700 1 -- 172.21.15.175:0/2478186630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 msgr2=0x7f41ec159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:10.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f1079700 1 --2- 172.21.15.175:0/2478186630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 0x7f41ec159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f1079700 1 -- 172.21.15.175:0/2478186630 >> v1:172.21.15.175:6789/0 conn(0x7f41ec14ffc0 legacy=0x7f41ec063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:10.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f1079700 1 -- 172.21.15.175:0/2478186630 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec063e30 con 0x7f41ec0681e0 2023-10-10T19:30:10.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41eb7fe700 1 -- 172.21.15.175:0/2478186630 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894479567 0 0) 0x7f41ec156800 con 0x7f41ec14ffc0 2023-10-10T19:30:10.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41f1079700 1 --2- 172.21.15.175:0/2478186630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec0681e0 0x7f41ec0c5fb0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009e90 tx=0x7f41e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d4f3bad44a9f3e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:10.145 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41eb7fe700 1 -- 172.21.15.175:0/2478186630 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e400e070 con 0x7f41ec0681e0 2023-10-10T19:30:10.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41eb7fe700 1 -- 172.21.15.175:0/2478186630 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e400bbf0 con 0x7f41ec0681e0 2023-10-10T19:30:10.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.137+0000 7f41eb7fe700 1 -- 172.21.15.175:0/2478186630 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4012430 con 0x7f41ec0681e0 2023-10-10T19:30:10.146 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.138+0000 7f41f2adc700 1 -- 172.21.15.175:0/2478186630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec0681e0 msgr2=0x7f41ec0c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:10.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.138+0000 7f41f2adc700 1 --2- 172.21.15.175:0/2478186630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec0681e0 0x7f41ec0c5fb0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009e90 tx=0x7f41e4009780 comp rx=0 tx=0).stop 2023-10-10T19:30:10.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.138+0000 7f41f2adc700 1 -- 172.21.15.175:0/2478186630 shutdown_connections 2023-10-10T19:30:10.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.138+0000 7f41f2adc700 1 --2- 172.21.15.175:0/2478186630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec0681e0 0x7f41ec0c5fb0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.147 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.138+0000 7f41f2adc700 1 --2- 172.21.15.175:0/2478186630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 0x7f41ec159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.138+0000 7f41f2adc700 1 -- 172.21.15.175:0/2478186630 >> 172.21.15.175:0/2478186630 conn(0x7f41ec0cc760 msgr2=0x7f41ec0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:10.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.138+0000 7f41f2adc700 1 -- 172.21.15.175:0/2478186630 shutdown_connections 2023-10-10T19:30:10.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.138+0000 7f41f2adc700 1 -- 172.21.15.175:0/2478186630 wait complete. 2023-10-10T19:30:10.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.139+0000 7f41f2adc700 1 Processor -- start 2023-10-10T19:30:10.149 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.139+0000 7f41f2adc700 1 -- start start 2023-10-10T19:30:10.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f2adc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41ec0681e0 0x7f41ec0c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:10.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f2adc700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 0x7f41ec167210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:10.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f2adc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec14ffc0 0x7f41ec14db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:10.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f2adc700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec0c9db0 con 0x7f41ec14ffc0 2023-10-10T19:30:10.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f2adc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec169d20 con 0x7f41ec0681e0 2023-10-10T19:30:10.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f2adc700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec169ff0 con 0x7f41ec14fbf0 2023-10-10T19:30:10.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f1079700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec14ffc0 0x7f41ec14db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:10.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f1079700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec14ffc0 0x7f41ec14db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:51224/0 (socket says 172.21.15.175:51224) 2023-10-10T19:30:10.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f1079700 1 -- 172.21.15.175:0/3145033558 learned_addr learned my addr 172.21.15.175:0/3145033558 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:10.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f0878700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41ec0681e0 0x7f41ec0c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:10.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41ebfff700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 0x7f41ec167210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:10.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f1079700 1 -- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 msgr2=0x7f41ec167210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:10.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f1079700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 0x7f41ec167210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f1079700 1 -- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41ec0681e0 msgr2=0x7f41ec0c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:10.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f1079700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41ec0681e0 0x7f41ec0c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.140+0000 7f41f1079700 1 -- 172.21.15.175:0/3145033558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e4009a20 con 0x7f41ec14ffc0 2023-10-10T19:30:10.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.141+0000 7f41f1079700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec14ffc0 0x7f41ec14db50 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f41ec089f50 tx=0x7f41e4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:10.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.141+0000 7f41e8ff9700 1 -- 172.21.15.175:0/3145033558 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4027470 con 0x7f41ec14ffc0 2023-10-10T19:30:10.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.141+0000 7f41e8ff9700 1 -- 172.21.15.175:0/3145033558 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e40275f0 con 0x7f41ec14ffc0 2023-10-10T19:30:10.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.141+0000 7f41e8ff9700 1 -- 172.21.15.175:0/3145033558 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e40279b0 con 0x7f41ec14ffc0 2023-10-10T19:30:10.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.141+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec1517c0 con 0x7f41ec14ffc0 2023-10-10T19:30:10.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.141+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec151d80 con 0x7f41ec14ffc0 2023-10-10T19:30:10.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.142+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec067570 con 0x7f41ec14ffc0 2023-10-10T19:30:10.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.142+0000 7f41e8ff9700 1 -- 172.21.15.175:0/3145033558 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f41e4024070 con 0x7f41ec14ffc0 2023-10-10T19:30:10.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.143+0000 7f41e8ff9700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f41d0066f30 0x7f41d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:10.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.143+0000 7f41e8ff9700 1 -- 172.21.15.175:0/3145033558 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f41e4087730 con 0x7f41ec14ffc0 2023-10-10T19:30:10.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.143+0000 7f41f0878700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f41d0066f30 0x7f41d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:10.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.143+0000 7f41f0878700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f41d0066f30 0x7f41d00693e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f41e0006aa0 tx=0x7f41e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:10.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.148+0000 7f41e8ff9700 1 -- 172.21.15.175:0/3145033558 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f41e4054120 con 0x7f41ec14ffc0 2023-10-10T19:30:10.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.325+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41ec063830 con 0x7f41d0066f30 2023-10-10T19:30:10.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.334+0000 7f41e8ff9700 1 -- 172.21.15.175:0/3145033558 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+296884 (secure 0 0 0) 0x7f41ec063830 con 0x7f41d0066f30 2023-10-10T19:30:10.339 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:30:10.342 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f41d0066f30 msgr2=0x7f41d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:10.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f41d0066f30 0x7f41d00693e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f41e0006aa0 tx=0x7f41e0008040 comp rx=0 tx=0).stop 2023-10-10T19:30:10.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec14ffc0 msgr2=0x7f41ec14db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:10.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec14ffc0 0x7f41ec14db50 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f41ec089f50 tx=0x7f41e4019000 comp rx=0 tx=0).stop 2023-10-10T19:30:10.344 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 shutdown_connections 2023-10-10T19:30:10.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f41d0066f30 0x7f41d00693e0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.345 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f41ec14ffc0 0x7f41ec14db50 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41ec0681e0 0x7f41ec0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.346 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 --2- 172.21.15.175:0/3145033558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41ec14fbf0 0x7f41ec167210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:10.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 >> 172.21.15.175:0/3145033558 conn(0x7f41ec0cc760 msgr2=0x7f41ec15bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:10.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 shutdown_connections 2023-10-10T19:30:10.347 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:10.337+0000 7f41f2adc700 1 -- 172.21.15.175:0/3145033558 wait complete. 2023-10-10T19:30:10.348 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T19:30:10.351 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":1478,"stamp":"2023-10-10T19:30:09.829961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388402487,"num_objects":2861,"num_object_clones":0,"num_object_copies":5722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":7382,"num_read_kb":3484532,"num_write":63416,"num_write_kb":2990256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":353,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2518057,"num_omap_keys":4386,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83004,"ondisk_log_size":83004,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2698072,"kb_used_data":266520,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752276648,"statfs":{"total":773094113280,"available":770331287552,"internally_reserved":0,"allocated":272916480,"data_stored":783892462,"data_compressed":117849180,"data_compressed_allocated":145776640,"data_compressed_original":674291712,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":66738,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":92062,"num_omap_keys":147,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002964"},"pg_stats":[{"pgid":"3.3f","version":"132'966","reported_seq":3587,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.164884+0000","last_change":"2023-10-10T19:16:54.796013+0000","last_active":"2023-10-10T19:29:22.164884+0000","last_peered":"2023-10-10T19:29:22.164884+0000","last_clean":"2023-10-10T19:29:22.164884+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T19:29:22.164884+0000","last_undegraded":"2023-10-10T19:29:22.164884+0000","last_fullsized":"2023-10-10T19:29:22.164884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017995507000000001,"stat_sum":{"num_bytes":821890,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":62,"num_read_kb":8068,"num_write":683,"num_write_kb":16759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"132'1048","reported_seq":3679,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:54.681549+0000","last_change":"2023-10-10T19:29:54.681522+0000","last_active":"2023-10-10T19:29:54.681549+0000","last_peered":"2023-10-10T19:29:54.681549+0000","last_clean":"2023-10-10T19:29:54.681549+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T19:29:54.681549+0000","last_undegraded":"2023-10-10T19:29:54.681549+0000","last_fullsized":"2023-10-10T19:29:54.681549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1048","last_scrub_stamp":"2023-10-10T19:29:54.681440+0000","last_deep_scrub":"132'1048","last_deep_scrub_stamp":"2023-10-10T19:29:54.681440+0000","last_clean_scrub_stamp":"2023-10-10T19:29:54.681440+0000","objects_scrubbed":36,"log_size":1048,"log_dups_size":0,"ondisk_log_size":1048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:09.327427+0000","scrub_duration":0.04117941,"objects_trimmed":0,"snaptrim_duration":0.0083569260000000006,"stat_sum":{"num_bytes":1412883,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":62,"num_read_kb":28805,"num_write":746,"num_write_kb":46678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"132'1002","reported_seq":3641,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:59.385738+0000","last_change":"2023-10-10T19:29:59.385701+0000","last_active":"2023-10-10T19:29:59.385738+0000","last_peered":"2023-10-10T19:29:59.385738+0000","last_clean":"2023-10-10T19:29:59.385738+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T19:29:59.385738+0000","last_undegraded":"2023-10-10T19:29:59.385738+0000","last_fullsized":"2023-10-10T19:29:59.385738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1002","last_scrub_stamp":"2023-10-10T19:29:59.385573+0000","last_deep_scrub":"132'1002","last_deep_scrub_stamp":"2023-10-10T19:29:59.385573+0000","last_clean_scrub_stamp":"2023-10-10T19:29:59.385573+0000","objects_scrubbed":40,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:26:17.112199+0000","scrub_duration":0.043775355000000002,"objects_trimmed":0,"snaptrim_duration":0.011353221,"stat_sum":{"num_bytes":671969,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":65,"num_read_kb":20680,"num_write":694,"num_write_kb":24810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"132'999","reported_seq":3625,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:01.613628+0000","last_change":"2023-10-10T19:30:01.613597+0000","last_active":"2023-10-10T19:30:01.613628+0000","last_peered":"2023-10-10T19:30:01.613628+0000","last_clean":"2023-10-10T19:30:01.613628+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T19:30:01.613628+0000","last_undegraded":"2023-10-10T19:30:01.613628+0000","last_fullsized":"2023-10-10T19:30:01.613628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'999","last_scrub_stamp":"2023-10-10T19:30:01.613517+0000","last_deep_scrub":"132'999","last_deep_scrub_stamp":"2023-10-10T19:30:01.613517+0000","last_clean_scrub_stamp":"2023-10-10T19:30:01.613517+0000","objects_scrubbed":38,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:00:14.282852+0000","scrub_duration":0.032420306000000003,"objects_trimmed":0,"snaptrim_duration":0.0088633549999999998,"stat_sum":{"num_bytes":651243,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":62,"num_read_kb":30965,"num_write":672,"num_write_kb":37710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"132'1023","reported_seq":3665,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:09.127081+0000","last_change":"2023-10-10T19:30:09.127044+0000","last_active":"2023-10-10T19:30:09.127081+0000","last_peered":"2023-10-10T19:30:09.127081+0000","last_clean":"2023-10-10T19:30:09.127081+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T19:30:09.127081+0000","last_undegraded":"2023-10-10T19:30:09.127081+0000","last_fullsized":"2023-10-10T19:30:09.127081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1023","last_scrub_stamp":"2023-10-10T19:30:09.126898+0000","last_deep_scrub":"132'1023","last_deep_scrub_stamp":"2023-10-10T19:30:09.126898+0000","last_clean_scrub_stamp":"2023-10-10T19:30:09.126898+0000","objects_scrubbed":36,"log_size":1023,"log_dups_size":0,"ondisk_log_size":1023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:44:47.815547+0000","scrub_duration":0.031728982000000003,"objects_trimmed":0,"snaptrim_duration":0.016189688000000001,"stat_sum":{"num_bytes":219054,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":75,"num_read_kb":17848,"num_write":702,"num_write_kb":31364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"132'942","reported_seq":3569,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162830+0000","last_change":"2023-10-10T19:16:55.414924+0000","last_active":"2023-10-10T19:29:22.162830+0000","last_peered":"2023-10-10T19:29:22.162830+0000","last_clean":"2023-10-10T19:29:22.162830+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T19:29:22.162830+0000","last_undegraded":"2023-10-10T19:29:22.162830+0000","last_fullsized":"2023-10-10T19:29:22.162830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010631753000000001,"stat_sum":{"num_bytes":2108369,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":63,"num_read_kb":20939,"num_write":675,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"132'1026","reported_seq":3659,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:04.095433+0000","last_change":"2023-10-10T19:30:04.095408+0000","last_active":"2023-10-10T19:30:04.095433+0000","last_peered":"2023-10-10T19:30:04.095433+0000","last_clean":"2023-10-10T19:30:04.095433+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T19:30:04.095433+0000","last_undegraded":"2023-10-10T19:30:04.095433+0000","last_fullsized":"2023-10-10T19:30:04.095433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1026","last_scrub_stamp":"2023-10-10T19:30:04.095327+0000","last_deep_scrub":"132'1026","last_deep_scrub_stamp":"2023-10-10T19:30:04.095327+0000","last_clean_scrub_stamp":"2023-10-10T19:30:04.095327+0000","objects_scrubbed":39,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T07:12:01.117837+0000","scrub_duration":0.034862328999999997,"objects_trimmed":0,"snaptrim_duration":0.014667316999999999,"stat_sum":{"num_bytes":1119068,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":77,"num_read_kb":53600,"num_write":685,"num_write_kb":52880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"132'914","reported_seq":3564,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:09.010756+0000","last_change":"2023-10-10T19:30:09.010731+0000","last_active":"2023-10-10T19:30:09.010756+0000","last_peered":"2023-10-10T19:30:09.010756+0000","last_clean":"2023-10-10T19:30:09.010756+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T19:30:09.010756+0000","last_undegraded":"2023-10-10T19:30:09.010756+0000","last_fullsized":"2023-10-10T19:30:09.010756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'914","last_scrub_stamp":"2023-10-10T19:30:09.010653+0000","last_deep_scrub":"132'914","last_deep_scrub_stamp":"2023-10-10T19:30:09.010653+0000","last_clean_scrub_stamp":"2023-10-10T19:30:09.010653+0000","objects_scrubbed":34,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:27:20.274857+0000","scrub_duration":0.030184883999999999,"objects_trimmed":0,"snaptrim_duration":0.012402231,"stat_sum":{"num_bytes":484780,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":58,"num_read_kb":28110,"num_write":652,"num_write_kb":35721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"132'976","reported_seq":3604,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.164700+0000","last_change":"2023-10-10T19:16:54.795632+0000","last_active":"2023-10-10T19:29:22.164700+0000","last_peered":"2023-10-10T19:29:22.164700+0000","last_clean":"2023-10-10T19:29:22.164700+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T19:29:22.164700+0000","last_undegraded":"2023-10-10T19:29:22.164700+0000","last_fullsized":"2023-10-10T19:29:22.164700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":976,"log_dups_size":0,"ondisk_log_size":976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017699890999999999,"stat_sum":{"num_bytes":94035,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":76,"num_read_kb":37435,"num_write":702,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"132'993","reported_seq":3631,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:58.196363+0000","last_change":"2023-10-10T19:29:58.196336+0000","last_active":"2023-10-10T19:29:58.196363+0000","last_peered":"2023-10-10T19:29:58.196363+0000","last_clean":"2023-10-10T19:29:58.196363+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T19:29:58.196363+0000","last_undegraded":"2023-10-10T19:29:58.196363+0000","last_fullsized":"2023-10-10T19:29:58.196363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'993","last_scrub_stamp":"2023-10-10T19:29:58.196244+0000","last_deep_scrub":"132'993","last_deep_scrub_stamp":"2023-10-10T19:29:58.196244+0000","last_clean_scrub_stamp":"2023-10-10T19:29:58.196244+0000","objects_scrubbed":38,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:09:04.497993+0000","scrub_duration":0.034923373000000001,"objects_trimmed":0,"snaptrim_duration":0.0079196160000000008,"stat_sum":{"num_bytes":269975,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":70,"num_read_kb":16825,"num_write":697,"num_write_kb":23418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"132'1021","reported_seq":3682,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:52.090985+0000","last_change":"2023-10-10T19:29:52.090959+0000","last_active":"2023-10-10T19:29:52.090985+0000","last_peered":"2023-10-10T19:29:52.090985+0000","last_clean":"2023-10-10T19:29:52.090985+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T19:29:52.090985+0000","last_undegraded":"2023-10-10T19:29:52.090985+0000","last_fullsized":"2023-10-10T19:29:52.090985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1021","last_scrub_stamp":"2023-10-10T19:29:52.090872+0000","last_deep_scrub":"132'1021","last_deep_scrub_stamp":"2023-10-10T19:29:52.090872+0000","last_clean_scrub_stamp":"2023-10-10T19:29:52.090872+0000","objects_scrubbed":46,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:30:48.456323+0000","scrub_duration":0.039720943000000002,"objects_trimmed":0,"snaptrim_duration":0.0088422029999999999,"stat_sum":{"num_bytes":213661,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":82,"num_read_kb":46829,"num_write":712,"num_write_kb":53066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"132'994","reported_seq":3641,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.048201+0000","last_change":"2023-10-10T19:29:48.048165+0000","last_active":"2023-10-10T19:29:48.048201+0000","last_peered":"2023-10-10T19:29:48.048201+0000","last_clean":"2023-10-10T19:29:48.048201+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T19:29:48.048201+0000","last_undegraded":"2023-10-10T19:29:48.048201+0000","last_fullsized":"2023-10-10T19:29:48.048201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'994","last_scrub_stamp":"2023-10-10T19:29:48.048044+0000","last_deep_scrub":"132'994","last_deep_scrub_stamp":"2023-10-10T19:29:48.048044+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.048044+0000","objects_scrubbed":40,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:22:32.633316+0000","scrub_duration":0.033909425999999999,"objects_trimmed":0,"snaptrim_duration":0.0078589550000000008,"stat_sum":{"num_bytes":337999,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":75,"num_read_kb":57532,"num_write":710,"num_write_kb":73524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"132'983","reported_seq":3626,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:06.001308+0000","last_change":"2023-10-10T19:30:06.001271+0000","last_active":"2023-10-10T19:30:06.001308+0000","last_peered":"2023-10-10T19:30:06.001308+0000","last_clean":"2023-10-10T19:30:06.001308+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T19:30:06.001308+0000","last_undegraded":"2023-10-10T19:30:06.001308+0000","last_fullsized":"2023-10-10T19:30:06.001308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'983","last_scrub_stamp":"2023-10-10T19:30:06.001152+0000","last_deep_scrub":"132'983","last_deep_scrub_stamp":"2023-10-10T19:30:06.001152+0000","last_clean_scrub_stamp":"2023-10-10T19:30:06.001152+0000","objects_scrubbed":54,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:56:00.208525+0000","scrub_duration":0.059218532999999997,"objects_trimmed":0,"snaptrim_duration":0.014204069,"stat_sum":{"num_bytes":4832489,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":73,"num_read_kb":32208,"num_write":656,"num_write_kb":44878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"132'956","reported_seq":3592,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:51.283187+0000","last_change":"2023-10-10T19:29:51.283150+0000","last_active":"2023-10-10T19:29:51.283187+0000","last_peered":"2023-10-10T19:29:51.283187+0000","last_clean":"2023-10-10T19:29:51.283187+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T19:29:51.283187+0000","last_undegraded":"2023-10-10T19:29:51.283187+0000","last_fullsized":"2023-10-10T19:29:51.283187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'956","last_scrub_stamp":"2023-10-10T19:29:51.283026+0000","last_deep_scrub":"132'956","last_deep_scrub_stamp":"2023-10-10T19:29:51.283026+0000","last_clean_scrub_stamp":"2023-10-10T19:29:51.283026+0000","objects_scrubbed":27,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:14:52.527044+0000","scrub_duration":0.027467595000000001,"objects_trimmed":0,"snaptrim_duration":0.0097965510000000006,"stat_sum":{"num_bytes":724304,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":63,"num_read_kb":28420,"num_write":663,"num_write_kb":31175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"132'991","reported_seq":3636,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.691055+0000","last_change":"2023-10-10T19:29:57.691018+0000","last_active":"2023-10-10T19:29:57.691055+0000","last_peered":"2023-10-10T19:29:57.691055+0000","last_clean":"2023-10-10T19:29:57.691055+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T19:29:57.691055+0000","last_undegraded":"2023-10-10T19:29:57.691055+0000","last_fullsized":"2023-10-10T19:29:57.691055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'991","last_scrub_stamp":"2023-10-10T19:29:57.690903+0000","last_deep_scrub":"132'991","last_deep_scrub_stamp":"2023-10-10T19:29:57.690903+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.690903+0000","objects_scrubbed":36,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:41:37.142029+0000","scrub_duration":0.034147700000000003,"objects_trimmed":0,"snaptrim_duration":0.0059774370000000004,"stat_sum":{"num_bytes":1353080,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":77,"num_read_kb":52747,"num_write":668,"num_write_kb":73053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"132'998","reported_seq":3633,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.162124+0000","last_change":"2023-10-10T19:16:55.412885+0000","last_active":"2023-10-10T19:29:22.162124+0000","last_peered":"2023-10-10T19:29:22.162124+0000","last_clean":"2023-10-10T19:29:22.162124+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T19:29:22.162124+0000","last_undegraded":"2023-10-10T19:29:22.162124+0000","last_fullsized":"2023-10-10T19:29:22.162124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087668889999999999,"stat_sum":{"num_bytes":582240,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":80,"num_read_kb":42995,"num_write":709,"num_write_kb":53070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"132'953","reported_seq":3619,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:04.618514+0000","last_change":"2023-10-10T19:30:04.618486+0000","last_active":"2023-10-10T19:30:04.618514+0000","last_peered":"2023-10-10T19:30:04.618514+0000","last_clean":"2023-10-10T19:30:04.618514+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T19:30:04.618514+0000","last_undegraded":"2023-10-10T19:30:04.618514+0000","last_fullsized":"2023-10-10T19:30:04.618514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'953","last_scrub_stamp":"2023-10-10T19:30:04.618402+0000","last_deep_scrub":"132'953","last_deep_scrub_stamp":"2023-10-10T19:30:04.618402+0000","last_clean_scrub_stamp":"2023-10-10T19:30:04.618402+0000","objects_scrubbed":48,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:28:28.649613+0000","scrub_duration":0.046332283000000002,"objects_trimmed":0,"snaptrim_duration":0.012085218,"stat_sum":{"num_bytes":1912035,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":72,"num_read_kb":40372,"num_write":683,"num_write_kb":48886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"132'871","reported_seq":3555,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.229972+0000","last_change":"2023-10-10T19:29:40.229945+0000","last_active":"2023-10-10T19:29:40.229972+0000","last_peered":"2023-10-10T19:29:40.229972+0000","last_clean":"2023-10-10T19:29:40.229972+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T19:29:40.229972+0000","last_undegraded":"2023-10-10T19:29:40.229972+0000","last_fullsized":"2023-10-10T19:29:40.229972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'871","last_scrub_stamp":"2023-10-10T19:29:40.229861+0000","last_deep_scrub":"132'871","last_deep_scrub_stamp":"2023-10-10T19:29:40.229861+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.229861+0000","objects_scrubbed":37,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:14:54.227939+0000","scrub_duration":0.028370732999999999,"objects_trimmed":0,"snaptrim_duration":0.00594963,"stat_sum":{"num_bytes":953364,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":72,"num_read_kb":44243,"num_write":664,"num_write_kb":60950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"132'1108","reported_seq":3717,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:07.082007+0000","last_change":"2023-10-10T19:30:07.081980+0000","last_active":"2023-10-10T19:30:07.082007+0000","last_peered":"2023-10-10T19:30:07.082007+0000","last_clean":"2023-10-10T19:30:07.082007+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T19:30:07.082007+0000","last_undegraded":"2023-10-10T19:30:07.082007+0000","last_fullsized":"2023-10-10T19:30:07.082007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1108","last_scrub_stamp":"2023-10-10T19:30:07.081901+0000","last_deep_scrub":"132'1108","last_deep_scrub_stamp":"2023-10-10T19:30:07.081901+0000","last_clean_scrub_stamp":"2023-10-10T19:30:07.081901+0000","objects_scrubbed":35,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:40:27.067627+0000","scrub_duration":0.032335704999999999,"objects_trimmed":0,"snaptrim_duration":0.01601,"stat_sum":{"num_bytes":689975,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":73,"num_read_kb":41187,"num_write":728,"num_write_kb":49784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"132'855","reported_seq":1031,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.287167+0000","last_change":"2023-10-10T19:29:28.287143+0000","last_active":"2023-10-10T19:29:28.287167+0000","last_peered":"2023-10-10T19:29:28.287167+0000","last_clean":"2023-10-10T19:29:28.287167+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T19:29:28.287167+0000","last_undegraded":"2023-10-10T19:29:28.287167+0000","last_fullsized":"2023-10-10T19:29:28.287167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'855","last_scrub_stamp":"2023-10-10T19:29:28.287071+0000","last_deep_scrub":"132'855","last_deep_scrub_stamp":"2023-10-10T19:29:28.287071+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.287071+0000","objects_scrubbed":19,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:01.117300+0000","scrub_duration":0.031438141000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389252,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":82,"num_read_kb":61516,"num_write":781,"num_write_kb":21019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":54468,"num_omap_keys":96,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"132'1045","reported_seq":3678,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:59.036152+0000","last_change":"2023-10-10T19:29:59.036124+0000","last_active":"2023-10-10T19:29:59.036152+0000","last_peered":"2023-10-10T19:29:59.036152+0000","last_clean":"2023-10-10T19:29:59.036152+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T19:29:59.036152+0000","last_undegraded":"2023-10-10T19:29:59.036152+0000","last_fullsized":"2023-10-10T19:29:59.036152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1045","last_scrub_stamp":"2023-10-10T19:29:59.036034+0000","last_deep_scrub":"132'1045","last_deep_scrub_stamp":"2023-10-10T19:29:59.036034+0000","last_clean_scrub_stamp":"2023-10-10T19:29:59.036034+0000","objects_scrubbed":47,"log_size":1045,"log_dups_size":0,"ondisk_log_size":1045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:49:51.599183+0000","scrub_duration":0.043332374,"objects_trimmed":0,"snaptrim_duration":0.018086130999999998,"stat_sum":{"num_bytes":136255,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":63,"num_read_kb":13169,"num_write":738,"num_write_kb":31183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"132'776","reported_seq":931,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:32.981768+0000","last_change":"2023-10-10T19:29:32.981742+0000","last_active":"2023-10-10T19:29:32.981768+0000","last_peered":"2023-10-10T19:29:32.981768+0000","last_clean":"2023-10-10T19:29:32.981768+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T19:29:32.981768+0000","last_undegraded":"2023-10-10T19:29:32.981768+0000","last_fullsized":"2023-10-10T19:29:32.981768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'776","last_scrub_stamp":"2023-10-10T19:29:32.981652+0000","last_deep_scrub":"132'776","last_deep_scrub_stamp":"2023-10-10T19:29:32.981652+0000","last_clean_scrub_stamp":"2023-10-10T19:29:32.981652+0000","objects_scrubbed":16,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:01:01.872125+0000","scrub_duration":0.042698237999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":29,"num_read_kb":69691,"num_write":711,"num_write_kb":16940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":122430,"num_omap_keys":218,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"132'1012","reported_seq":3666,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.277911+0000","last_change":"2023-10-10T19:29:47.277885+0000","last_active":"2023-10-10T19:29:47.277911+0000","last_peered":"2023-10-10T19:29:47.277911+0000","last_clean":"2023-10-10T19:29:47.277911+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T19:29:47.277911+0000","last_undegraded":"2023-10-10T19:29:47.277911+0000","last_fullsized":"2023-10-10T19:29:47.277911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1012","last_scrub_stamp":"2023-10-10T19:29:47.277804+0000","last_deep_scrub":"132'1012","last_deep_scrub_stamp":"2023-10-10T19:29:47.277804+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.277804+0000","objects_scrubbed":36,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:05.362050+0000","scrub_duration":0.026281229,"objects_trimmed":0,"snaptrim_duration":0.0093729599999999996,"stat_sum":{"num_bytes":224018,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":74,"num_read_kb":34573,"num_write":731,"num_write_kb":44057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"132'1717","reported_seq":1904,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:54.199844+0000","last_change":"2023-10-10T19:29:54.199820+0000","last_active":"2023-10-10T19:29:54.199844+0000","last_peered":"2023-10-10T19:29:54.199844+0000","last_clean":"2023-10-10T19:29:54.199844+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T19:29:54.199844+0000","last_undegraded":"2023-10-10T19:29:54.199844+0000","last_fullsized":"2023-10-10T19:29:54.199844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1717","last_scrub_stamp":"2023-10-10T19:29:54.199748+0000","last_deep_scrub":"132'1717","last_deep_scrub_stamp":"2023-10-10T19:29:54.199748+0000","last_clean_scrub_stamp":"2023-10-10T19:29:54.199748+0000","objects_scrubbed":23,"log_size":1717,"log_dups_size":0,"ondisk_log_size":1717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:36:21.023828+0000","scrub_duration":0.064385183999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":114,"num_read_kb":59511,"num_write":1905,"num_write_kb":22655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":97001,"num_omap_keys":173,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"132'971","reported_seq":3577,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.099971+0000","last_change":"2023-10-10T19:29:48.099945+0000","last_active":"2023-10-10T19:29:48.099971+0000","last_peered":"2023-10-10T19:29:48.099971+0000","last_clean":"2023-10-10T19:29:48.099971+0000","last_became_active":"2023-10-10T18:46:01.103860+0000","last_became_peered":"2023-10-10T18:46:01.103860+0000","last_unstale":"2023-10-10T19:29:48.099971+0000","last_undegraded":"2023-10-10T19:29:48.099971+0000","last_fullsized":"2023-10-10T19:29:48.099971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'971","last_scrub_stamp":"2023-10-10T19:29:48.099859+0000","last_deep_scrub":"132'971","last_deep_scrub_stamp":"2023-10-10T19:29:48.099859+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.099859+0000","objects_scrubbed":28,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:28:00.147441+0000","scrub_duration":0.027094722000000002,"objects_trimmed":0,"snaptrim_duration":0.0054668909999999998,"stat_sum":{"num_bytes":121718,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":48,"num_read_kb":295,"num_write":648,"num_write_kb":7505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"132'763","reported_seq":927,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.085034+0000","last_change":"2023-10-10T19:29:47.084994+0000","last_active":"2023-10-10T19:29:47.085034+0000","last_peered":"2023-10-10T19:29:47.085034+0000","last_clean":"2023-10-10T19:29:47.085034+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T19:29:47.085034+0000","last_undegraded":"2023-10-10T19:29:47.085034+0000","last_fullsized":"2023-10-10T19:29:47.085034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'763","last_scrub_stamp":"2023-10-10T19:29:47.084860+0000","last_deep_scrub":"132'763","last_deep_scrub_stamp":"2023-10-10T19:29:47.084860+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.084860+0000","objects_scrubbed":13,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:01.164246+0000","scrub_duration":0.026132728000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389162,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":55,"num_read_kb":36831,"num_write":683,"num_write_kb":16858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":72140,"num_omap_keys":127,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"132'1024","reported_seq":3666,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.614680+0000","last_change":"2023-10-10T19:29:57.614652+0000","last_active":"2023-10-10T19:29:57.614680+0000","last_peered":"2023-10-10T19:29:57.614680+0000","last_clean":"2023-10-10T19:29:57.614680+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T19:29:57.614680+0000","last_undegraded":"2023-10-10T19:29:57.614680+0000","last_fullsized":"2023-10-10T19:29:57.614680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1024","last_scrub_stamp":"2023-10-10T19:29:57.614566+0000","last_deep_scrub":"132'1024","last_deep_scrub_stamp":"2023-10-10T19:29:57.614566+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.614566+0000","objects_scrubbed":33,"log_size":1024,"log_dups_size":0,"ondisk_log_size":1024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:38:28.460701+0000","scrub_duration":0.038641201999999999,"objects_trimmed":0,"snaptrim_duration":0.011753095,"stat_sum":{"num_bytes":171234,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":81,"num_read_kb":35882,"num_write":699,"num_write_kb":43024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"132'652","reported_seq":811,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.206093+0000","last_change":"2023-10-10T19:29:29.999770+0000","last_active":"2023-10-10T19:29:30.206093+0000","last_peered":"2023-10-10T19:29:30.206093+0000","last_clean":"2023-10-10T19:29:30.206093+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T19:29:30.206093+0000","last_undegraded":"2023-10-10T19:29:30.206093+0000","last_fullsized":"2023-10-10T19:29:30.206093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'626","last_scrub_stamp":"2023-10-10T19:29:29.999705+0000","last_deep_scrub":"132'626","last_deep_scrub_stamp":"2023-10-10T19:29:29.999705+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.999705+0000","objects_scrubbed":14,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T07:13:56.479723+0000","scrub_duration":0.034014667999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15333167,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":31,"num_read_kb":65583,"num_write":581,"num_write_kb":19518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":50025,"num_omap_keys":89,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"132'942","reported_seq":3582,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:33.015438+0000","last_change":"2023-10-10T19:29:33.015412+0000","last_active":"2023-10-10T19:29:33.015438+0000","last_peered":"2023-10-10T19:29:33.015438+0000","last_clean":"2023-10-10T19:29:33.015438+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T19:29:33.015438+0000","last_undegraded":"2023-10-10T19:29:33.015438+0000","last_fullsized":"2023-10-10T19:29:33.015438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'942","last_scrub_stamp":"2023-10-10T19:29:33.015326+0000","last_deep_scrub":"132'942","last_deep_scrub_stamp":"2023-10-10T19:29:33.015326+0000","last_clean_scrub_stamp":"2023-10-10T19:29:33.015326+0000","objects_scrubbed":43,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:49:00.949756+0000","scrub_duration":0.032708036000000003,"objects_trimmed":0,"snaptrim_duration":0.0062644010000000002,"stat_sum":{"num_bytes":838983,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":59,"num_read_kb":21826,"num_write":689,"num_write_kb":37727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"132'791","reported_seq":1134,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.136520+0000","last_change":"2023-10-10T19:29:31.136493+0000","last_active":"2023-10-10T19:29:31.136520+0000","last_peered":"2023-10-10T19:29:31.136520+0000","last_clean":"2023-10-10T19:29:31.136520+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T19:29:31.136520+0000","last_undegraded":"2023-10-10T19:29:31.136520+0000","last_fullsized":"2023-10-10T19:29:31.136520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'791","last_scrub_stamp":"2023-10-10T19:29:31.136411+0000","last_deep_scrub":"132'791","last_deep_scrub_stamp":"2023-10-10T19:29:31.136411+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.136411+0000","objects_scrubbed":15,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:18:11.372189+0000","scrub_duration":0.042934018999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111936,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":251,"num_read_kb":81147,"num_write":704,"num_write_kb":25397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35794,"num_omap_keys":65,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"132'857","reported_seq":3517,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.291480+0000","last_change":"2023-10-10T19:29:42.291455+0000","last_active":"2023-10-10T19:29:42.291480+0000","last_peered":"2023-10-10T19:29:42.291480+0000","last_clean":"2023-10-10T19:29:42.291480+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T19:29:42.291480+0000","last_undegraded":"2023-10-10T19:29:42.291480+0000","last_fullsized":"2023-10-10T19:29:42.291480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'857","last_scrub_stamp":"2023-10-10T19:29:42.291374+0000","last_deep_scrub":"132'857","last_deep_scrub_stamp":"2023-10-10T19:29:42.291374+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.291374+0000","objects_scrubbed":27,"log_size":857,"log_dups_size":0,"ondisk_log_size":857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:23:42.700538+0000","scrub_duration":0.018625359000000001,"objects_trimmed":0,"snaptrim_duration":0.015269961,"stat_sum":{"num_bytes":473837,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":60,"num_read_kb":24046,"num_write":615,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"132'1506","reported_seq":1912,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.017721+0000","last_change":"2023-10-10T19:29:28.017678+0000","last_active":"2023-10-10T19:29:28.017721+0000","last_peered":"2023-10-10T19:29:28.017721+0000","last_clean":"2023-10-10T19:29:28.017721+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T19:29:28.017721+0000","last_undegraded":"2023-10-10T19:29:28.017721+0000","last_fullsized":"2023-10-10T19:29:28.017721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1506","last_scrub_stamp":"2023-10-10T19:29:28.017594+0000","last_deep_scrub":"132'1506","last_deep_scrub_stamp":"2023-10-10T19:29:28.017594+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.017594+0000","objects_scrubbed":12,"log_size":1506,"log_dups_size":0,"ondisk_log_size":1506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:59:47.232197+0000","scrub_duration":0.020028464999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4727705,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":295,"num_read_kb":15405,"num_write":1447,"num_write_kb":10055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66656,"num_omap_keys":116,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"132'1043","reported_seq":3698,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:54.022953+0000","last_change":"2023-10-10T19:29:54.022924+0000","last_active":"2023-10-10T19:29:54.022953+0000","last_peered":"2023-10-10T19:29:54.022953+0000","last_clean":"2023-10-10T19:29:54.022953+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T19:29:54.022953+0000","last_undegraded":"2023-10-10T19:29:54.022953+0000","last_fullsized":"2023-10-10T19:29:54.022953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1043","last_scrub_stamp":"2023-10-10T19:29:54.022842+0000","last_deep_scrub":"132'1043","last_deep_scrub_stamp":"2023-10-10T19:29:54.022842+0000","last_clean_scrub_stamp":"2023-10-10T19:29:54.022842+0000","objects_scrubbed":41,"log_size":1043,"log_dups_size":0,"ondisk_log_size":1043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:56:40.926906+0000","scrub_duration":0.038501266999999999,"objects_trimmed":0,"snaptrim_duration":0.015767953000000001,"stat_sum":{"num_bytes":671315,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":81,"num_read_kb":19092,"num_write":747,"num_write_kb":30378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"132'506","reported_seq":662,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.758375+0000","last_change":"2023-10-10T19:29:40.758347+0000","last_active":"2023-10-10T19:29:40.758375+0000","last_peered":"2023-10-10T19:29:40.758375+0000","last_clean":"2023-10-10T19:29:40.758375+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T19:29:40.758375+0000","last_undegraded":"2023-10-10T19:29:40.758375+0000","last_fullsized":"2023-10-10T19:29:40.758375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'506","last_scrub_stamp":"2023-10-10T19:29:40.758259+0000","last_deep_scrub":"132'506","last_deep_scrub_stamp":"2023-10-10T19:29:40.758259+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.758259+0000","objects_scrubbed":12,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:28:13.803240+0000","scrub_duration":0.033275374000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583466,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":65566,"num_write":451,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":76745,"num_omap_keys":113,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"132'1056","reported_seq":3707,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.597586+0000","last_change":"2023-10-10T19:29:45.597550+0000","last_active":"2023-10-10T19:29:45.597586+0000","last_peered":"2023-10-10T19:29:45.597586+0000","last_clean":"2023-10-10T19:29:45.597586+0000","last_became_active":"2023-10-10T18:46:00.606391+0000","last_became_peered":"2023-10-10T18:46:00.606391+0000","last_unstale":"2023-10-10T19:29:45.597586+0000","last_undegraded":"2023-10-10T19:29:45.597586+0000","last_fullsized":"2023-10-10T19:29:45.597586+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1056","last_scrub_stamp":"2023-10-10T19:29:45.597448+0000","last_deep_scrub":"132'1056","last_deep_scrub_stamp":"2023-10-10T19:29:45.597448+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.597448+0000","objects_scrubbed":41,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:39:53.993867+0000","scrub_duration":0.033388767999999999,"objects_trimmed":0,"snaptrim_duration":0.0066939039999999997,"stat_sum":{"num_bytes":730064,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":80,"num_read_kb":42231,"num_write":744,"num_write_kb":52869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"132'292","reported_seq":444,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.067574+0000","last_change":"2023-10-10T19:29:48.067525+0000","last_active":"2023-10-10T19:29:48.067574+0000","last_peered":"2023-10-10T19:29:48.067574+0000","last_clean":"2023-10-10T19:29:48.067574+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T19:29:48.067574+0000","last_undegraded":"2023-10-10T19:29:48.067574+0000","last_fullsized":"2023-10-10T19:29:48.067574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'291","last_scrub_stamp":"2023-10-10T19:29:48.067450+0000","last_deep_scrub":"132'291","last_deep_scrub_stamp":"2023-10-10T19:29:48.067450+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.067450+0000","objects_scrubbed":17,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:31:55.859793+0000","scrub_duration":0.010942558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":34,"num_read_kb":54,"num_write":234,"num_write_kb":4552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":87494,"num_omap_keys":156,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"132'988","reported_seq":3635,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:03.614708+0000","last_change":"2023-10-10T19:30:03.614679+0000","last_active":"2023-10-10T19:30:03.614708+0000","last_peered":"2023-10-10T19:30:03.614708+0000","last_clean":"2023-10-10T19:30:03.614708+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T19:30:03.614708+0000","last_undegraded":"2023-10-10T19:30:03.614708+0000","last_fullsized":"2023-10-10T19:30:03.614708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'988","last_scrub_stamp":"2023-10-10T19:30:03.614594+0000","last_deep_scrub":"132'988","last_deep_scrub_stamp":"2023-10-10T19:30:03.614594+0000","last_clean_scrub_stamp":"2023-10-10T19:30:03.614594+0000","objects_scrubbed":25,"log_size":988,"log_dups_size":0,"ondisk_log_size":988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:21:08.189640+0000","scrub_duration":0.025435242,"objects_trimmed":0,"snaptrim_duration":0.0096962509999999995,"stat_sum":{"num_bytes":540276,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":63,"num_read_kb":26256,"num_write":725,"num_write_kb":46019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"132'643","reported_seq":1012,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:43.718334+0000","last_change":"2023-10-10T19:29:43.718305+0000","last_active":"2023-10-10T19:29:43.718334+0000","last_peered":"2023-10-10T19:29:43.718334+0000","last_clean":"2023-10-10T19:29:43.718334+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T19:29:43.718334+0000","last_undegraded":"2023-10-10T19:29:43.718334+0000","last_fullsized":"2023-10-10T19:29:43.718334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'643","last_scrub_stamp":"2023-10-10T19:29:43.718205+0000","last_deep_scrub":"132'643","last_deep_scrub_stamp":"2023-10-10T19:29:43.718205+0000","last_clean_scrub_stamp":"2023-10-10T19:29:43.718205+0000","objects_scrubbed":15,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:24:50.063113+0000","scrub_duration":0.029979209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8573947,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":262,"num_read_kb":30418,"num_write":570,"num_write_kb":12898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":102578,"num_omap_keys":183,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"132'952","reported_seq":3587,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:33.425228+0000","last_change":"2023-10-10T19:29:33.425192+0000","last_active":"2023-10-10T19:29:33.425228+0000","last_peered":"2023-10-10T19:29:33.425228+0000","last_clean":"2023-10-10T19:29:33.425228+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T19:29:33.425228+0000","last_undegraded":"2023-10-10T19:29:33.425228+0000","last_fullsized":"2023-10-10T19:29:33.425228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'952","last_scrub_stamp":"2023-10-10T19:29:33.425101+0000","last_deep_scrub":"132'952","last_deep_scrub_stamp":"2023-10-10T19:29:33.425101+0000","last_clean_scrub_stamp":"2023-10-10T19:29:33.425101+0000","objects_scrubbed":33,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:50:37.033893+0000","scrub_duration":0.024688640000000001,"objects_trimmed":0,"snaptrim_duration":0.015222243,"stat_sum":{"num_bytes":331805,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":64,"num_read_kb":38795,"num_write":639,"num_write_kb":46131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"132'219","reported_seq":372,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.571663+0000","last_change":"2023-10-10T19:29:42.571637+0000","last_active":"2023-10-10T19:29:42.571663+0000","last_peered":"2023-10-10T19:29:42.571663+0000","last_clean":"2023-10-10T19:29:42.571663+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T19:29:42.571663+0000","last_undegraded":"2023-10-10T19:29:42.571663+0000","last_fullsized":"2023-10-10T19:29:42.571663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'219","last_scrub_stamp":"2023-10-10T19:29:42.571548+0000","last_deep_scrub":"132'219","last_deep_scrub_stamp":"2023-10-10T19:29:42.571548+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.571548+0000","objects_scrubbed":12,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:06:27.021509+0000","scrub_duration":0.010029698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":61,"num_write":173,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70925,"num_omap_keys":126,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"132'902","reported_seq":3546,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.010757+0000","last_change":"2023-10-10T19:29:42.010728+0000","last_active":"2023-10-10T19:29:42.010757+0000","last_peered":"2023-10-10T19:29:42.010757+0000","last_clean":"2023-10-10T19:29:42.010757+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T19:29:42.010757+0000","last_undegraded":"2023-10-10T19:29:42.010757+0000","last_fullsized":"2023-10-10T19:29:42.010757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'902","last_scrub_stamp":"2023-10-10T19:29:42.010643+0000","last_deep_scrub":"132'902","last_deep_scrub_stamp":"2023-10-10T19:29:42.010643+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.010643+0000","objects_scrubbed":35,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:51:41.084838+0000","scrub_duration":0.032300451000000001,"objects_trimmed":0,"snaptrim_duration":0.0080742209999999995,"stat_sum":{"num_bytes":103314,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":56,"num_read_kb":23103,"num_write":655,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"132'281","reported_seq":441,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:39.018236+0000","last_change":"2023-10-10T19:29:36.136030+0000","last_active":"2023-10-10T19:29:39.018236+0000","last_peered":"2023-10-10T19:29:39.018236+0000","last_clean":"2023-10-10T19:29:39.018236+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T19:29:39.018236+0000","last_undegraded":"2023-10-10T19:29:39.018236+0000","last_fullsized":"2023-10-10T19:29:39.018236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'279","last_scrub_stamp":"2023-10-10T19:29:36.135933+0000","last_deep_scrub":"132'279","last_deep_scrub_stamp":"2023-10-10T19:29:36.135933+0000","last_clean_scrub_stamp":"2023-10-10T19:29:36.135933+0000","objects_scrubbed":15,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:22:18.822287+0000","scrub_duration":0.016822342000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194948,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":59,"num_read_kb":16449,"num_write":215,"num_write_kb":4414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64730,"num_omap_keys":109,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"132'975","reported_seq":3617,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:35.143939+0000","last_change":"2023-10-10T19:29:35.143909+0000","last_active":"2023-10-10T19:29:35.143939+0000","last_peered":"2023-10-10T19:29:35.143939+0000","last_clean":"2023-10-10T19:29:35.143939+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T19:29:35.143939+0000","last_undegraded":"2023-10-10T19:29:35.143939+0000","last_fullsized":"2023-10-10T19:29:35.143939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'975","last_scrub_stamp":"2023-10-10T19:29:35.143816+0000","last_deep_scrub":"132'975","last_deep_scrub_stamp":"2023-10-10T19:29:35.143816+0000","last_clean_scrub_stamp":"2023-10-10T19:29:35.143816+0000","objects_scrubbed":35,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:54:49.422231+0000","scrub_duration":0.025232871,"objects_trimmed":0,"snaptrim_duration":0.015774979000000001,"stat_sum":{"num_bytes":487172,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":71,"num_read_kb":24956,"num_write":677,"num_write_kb":31709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"132'838","reported_seq":1285,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.356700+0000","last_change":"2023-10-10T19:29:30.356675+0000","last_active":"2023-10-10T19:29:30.356700+0000","last_peered":"2023-10-10T19:29:30.356700+0000","last_clean":"2023-10-10T19:29:30.356700+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T19:29:30.356700+0000","last_undegraded":"2023-10-10T19:29:30.356700+0000","last_fullsized":"2023-10-10T19:29:30.356700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'838","last_scrub_stamp":"2023-10-10T19:29:30.356605+0000","last_deep_scrub":"132'838","last_deep_scrub_stamp":"2023-10-10T19:29:30.356605+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.356605+0000","objects_scrubbed":20,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:03:47.996301+0000","scrub_duration":0.044624234999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227398,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":359,"num_read_kb":46031,"num_write":771,"num_write_kb":13519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":95526,"num_omap_keys":174,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"132'949","reported_seq":3592,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.362095+0000","last_change":"2023-10-10T19:29:31.362073+0000","last_active":"2023-10-10T19:29:31.362095+0000","last_peered":"2023-10-10T19:29:31.362095+0000","last_clean":"2023-10-10T19:29:31.362095+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T19:29:31.362095+0000","last_undegraded":"2023-10-10T19:29:31.362095+0000","last_fullsized":"2023-10-10T19:29:31.362095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'949","last_scrub_stamp":"2023-10-10T19:29:31.362007+0000","last_deep_scrub":"132'949","last_deep_scrub_stamp":"2023-10-10T19:29:31.362007+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.362007+0000","objects_scrubbed":32,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:44.834185+0000","scrub_duration":0.026536431999999999,"objects_trimmed":0,"snaptrim_duration":0.01251216,"stat_sum":{"num_bytes":121104,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":62,"num_read_kb":21731,"num_write":687,"num_write_kb":32925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"132'735","reported_seq":906,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.681955+0000","last_change":"2023-10-10T19:29:48.681927+0000","last_active":"2023-10-10T19:29:48.681955+0000","last_peered":"2023-10-10T19:29:48.681955+0000","last_clean":"2023-10-10T19:29:48.681955+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T19:29:48.681955+0000","last_undegraded":"2023-10-10T19:29:48.681955+0000","last_fullsized":"2023-10-10T19:29:48.681955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'735","last_scrub_stamp":"2023-10-10T19:29:48.681838+0000","last_deep_scrub":"132'735","last_deep_scrub_stamp":"2023-10-10T19:29:48.681838+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.681838+0000","objects_scrubbed":15,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:12.687746+0000","scrub_duration":0.036150337999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15771894,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":58,"num_read_kb":76305,"num_write":656,"num_write_kb":19955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":55118,"num_omap_keys":99,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"132'1000","reported_seq":3637,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:06.079446+0000","last_change":"2023-10-10T19:30:06.079422+0000","last_active":"2023-10-10T19:30:06.079446+0000","last_peered":"2023-10-10T19:30:06.079446+0000","last_clean":"2023-10-10T19:30:06.079446+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T19:30:06.079446+0000","last_undegraded":"2023-10-10T19:30:06.079446+0000","last_fullsized":"2023-10-10T19:30:06.079446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1000","last_scrub_stamp":"2023-10-10T19:30:06.079344+0000","last_deep_scrub":"132'1000","last_deep_scrub_stamp":"2023-10-10T19:30:06.079344+0000","last_clean_scrub_stamp":"2023-10-10T19:30:06.079344+0000","objects_scrubbed":31,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:30:20.873932+0000","scrub_duration":0.032878220999999999,"objects_trimmed":0,"snaptrim_duration":0.017093574,"stat_sum":{"num_bytes":558529,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":73,"num_read_kb":45017,"num_write":693,"num_write_kb":44107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"132'423","reported_seq":590,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:34.186050+0000","last_change":"2023-10-10T19:29:34.186026+0000","last_active":"2023-10-10T19:29:34.186050+0000","last_peered":"2023-10-10T19:29:34.186050+0000","last_clean":"2023-10-10T19:29:34.186050+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T19:29:34.186050+0000","last_undegraded":"2023-10-10T19:29:34.186050+0000","last_fullsized":"2023-10-10T19:29:34.186050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'423","last_scrub_stamp":"2023-10-10T19:29:34.185945+0000","last_deep_scrub":"132'423","last_deep_scrub_stamp":"2023-10-10T19:29:34.185945+0000","last_clean_scrub_stamp":"2023-10-10T19:29:34.185945+0000","objects_scrubbed":16,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:58:02.739712+0000","scrub_duration":0.049211269000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":54,"num_read_kb":84439,"num_write":360,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":79730,"num_omap_keys":141,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"132'876","reported_seq":3520,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.226230+0000","last_change":"2023-10-10T19:29:57.226206+0000","last_active":"2023-10-10T19:29:57.226230+0000","last_peered":"2023-10-10T19:29:57.226230+0000","last_clean":"2023-10-10T19:29:57.226230+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T19:29:57.226230+0000","last_undegraded":"2023-10-10T19:29:57.226230+0000","last_fullsized":"2023-10-10T19:29:57.226230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'876","last_scrub_stamp":"2023-10-10T19:29:57.226124+0000","last_deep_scrub":"132'876","last_deep_scrub_stamp":"2023-10-10T19:29:57.226124+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.226124+0000","objects_scrubbed":42,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:10.624743+0000","scrub_duration":0.034336075000000001,"objects_trimmed":0,"snaptrim_duration":0.0075683979999999996,"stat_sum":{"num_bytes":1111788,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":52,"num_read_kb":24653,"num_write":611,"num_write_kb":45399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"132'823","reported_seq":1013,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:32.342694+0000","last_change":"2023-10-10T19:29:32.342669+0000","last_active":"2023-10-10T19:29:32.342694+0000","last_peered":"2023-10-10T19:29:32.342694+0000","last_clean":"2023-10-10T19:29:32.342694+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T19:29:32.342694+0000","last_undegraded":"2023-10-10T19:29:32.342694+0000","last_fullsized":"2023-10-10T19:29:32.342694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'823","last_scrub_stamp":"2023-10-10T19:29:32.342571+0000","last_deep_scrub":"132'823","last_deep_scrub_stamp":"2023-10-10T19:29:32.342571+0000","last_clean_scrub_stamp":"2023-10-10T19:29:32.342571+0000","objects_scrubbed":19,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:42.635501+0000","scrub_duration":0.051952427000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":113,"num_read_kb":78302,"num_write":734,"num_write_kb":25074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":59966,"num_omap_keys":106,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"132'933","reported_seq":3582,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:37.129812+0000","last_change":"2023-10-10T19:29:37.129774+0000","last_active":"2023-10-10T19:29:37.129812+0000","last_peered":"2023-10-10T19:29:37.129812+0000","last_clean":"2023-10-10T19:29:37.129812+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T19:29:37.129812+0000","last_undegraded":"2023-10-10T19:29:37.129812+0000","last_fullsized":"2023-10-10T19:29:37.129812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'933","last_scrub_stamp":"2023-10-10T19:29:37.129655+0000","last_deep_scrub":"132'933","last_deep_scrub_stamp":"2023-10-10T19:29:37.129655+0000","last_clean_scrub_stamp":"2023-10-10T19:29:37.129655+0000","objects_scrubbed":33,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:10.731137+0000","scrub_duration":0.028840863000000001,"objects_trimmed":0,"snaptrim_duration":0.01616896,"stat_sum":{"num_bytes":90659,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":67,"num_read_kb":45230,"num_write":665,"num_write_kb":49103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"132'621","reported_seq":769,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.983083+0000","last_change":"2023-10-10T19:29:28.983043+0000","last_active":"2023-10-10T19:29:28.983083+0000","last_peered":"2023-10-10T19:29:28.983083+0000","last_clean":"2023-10-10T19:29:28.983083+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T19:29:28.983083+0000","last_undegraded":"2023-10-10T19:29:28.983083+0000","last_fullsized":"2023-10-10T19:29:28.983083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'621","last_scrub_stamp":"2023-10-10T19:29:28.982898+0000","last_deep_scrub":"132'621","last_deep_scrub_stamp":"2023-10-10T19:29:28.982898+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.982898+0000","objects_scrubbed":17,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:02:20.298697+0000","scrub_duration":0.023284135000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":23,"num_read_kb":12341,"num_write":750,"num_write_kb":8877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":130710,"num_omap_keys":230,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"132'976","reported_seq":3602,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:02.094991+0000","last_change":"2023-10-10T19:30:02.094965+0000","last_active":"2023-10-10T19:30:02.094991+0000","last_peered":"2023-10-10T19:30:02.094991+0000","last_clean":"2023-10-10T19:30:02.094991+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T19:30:02.094991+0000","last_undegraded":"2023-10-10T19:30:02.094991+0000","last_fullsized":"2023-10-10T19:30:02.094991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'976","last_scrub_stamp":"2023-10-10T19:30:02.094882+0000","last_deep_scrub":"132'976","last_deep_scrub_stamp":"2023-10-10T19:30:02.094882+0000","last_clean_scrub_stamp":"2023-10-10T19:30:02.094882+0000","objects_scrubbed":38,"log_size":976,"log_dups_size":0,"ondisk_log_size":976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:13:17.747270+0000","scrub_duration":0.041304615000000003,"objects_trimmed":0,"snaptrim_duration":0.015882960000000002,"stat_sum":{"num_bytes":708648,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":57,"num_read_kb":20900,"num_write":676,"num_write_kb":24718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"132'177","reported_seq":325,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.696712+0000","last_change":"2023-10-10T19:29:30.696675+0000","last_active":"2023-10-10T19:29:30.696712+0000","last_peered":"2023-10-10T19:29:30.696712+0000","last_clean":"2023-10-10T19:29:30.696712+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T19:29:30.696712+0000","last_undegraded":"2023-10-10T19:29:30.696712+0000","last_fullsized":"2023-10-10T19:29:30.696712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'177","last_scrub_stamp":"2023-10-10T19:29:30.696594+0000","last_deep_scrub":"132'177","last_deep_scrub_stamp":"2023-10-10T19:29:30.696594+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.696594+0000","objects_scrubbed":15,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:22:41.559522+0000","scrub_duration":0.035497603000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":45123,"num_write":119,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":122093,"num_omap_keys":206,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"132'1083","reported_seq":3703,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:56.639612+0000","last_change":"2023-10-10T19:29:56.639585+0000","last_active":"2023-10-10T19:29:56.639612+0000","last_peered":"2023-10-10T19:29:56.639612+0000","last_clean":"2023-10-10T19:29:56.639612+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T19:29:56.639612+0000","last_undegraded":"2023-10-10T19:29:56.639612+0000","last_fullsized":"2023-10-10T19:29:56.639612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1083","last_scrub_stamp":"2023-10-10T19:29:56.639500+0000","last_deep_scrub":"132'1083","last_deep_scrub_stamp":"2023-10-10T19:29:56.639500+0000","last_clean_scrub_stamp":"2023-10-10T19:29:56.639500+0000","objects_scrubbed":34,"log_size":1083,"log_dups_size":0,"ondisk_log_size":1083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:56.591064+0000","scrub_duration":0.031050833999999999,"objects_trimmed":0,"snaptrim_duration":0.011491507999999999,"stat_sum":{"num_bytes":461869,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":71,"num_read_kb":20716,"num_write":730,"num_write_kb":35297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"132'359","reported_seq":518,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.696909+0000","last_change":"2023-10-10T19:29:28.696870+0000","last_active":"2023-10-10T19:29:28.696909+0000","last_peered":"2023-10-10T19:29:28.696909+0000","last_clean":"2023-10-10T19:29:28.696909+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T19:29:28.696909+0000","last_undegraded":"2023-10-10T19:29:28.696909+0000","last_fullsized":"2023-10-10T19:29:28.696909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'359","last_scrub_stamp":"2023-10-10T19:29:28.696790+0000","last_deep_scrub":"132'359","last_deep_scrub_stamp":"2023-10-10T19:29:28.696790+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.696790+0000","objects_scrubbed":18,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:28:30.574457+0000","scrub_duration":0.036111012999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":73794,"num_write":284,"num_write_kb":20986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":75814,"num_omap_keys":136,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"132'979","reported_seq":3658,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:00.983629+0000","last_change":"2023-10-10T19:30:00.983588+0000","last_active":"2023-10-10T19:30:00.983629+0000","last_peered":"2023-10-10T19:30:00.983629+0000","last_clean":"2023-10-10T19:30:00.983629+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T19:30:00.983629+0000","last_undegraded":"2023-10-10T19:30:00.983629+0000","last_fullsized":"2023-10-10T19:30:00.983629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'979","last_scrub_stamp":"2023-10-10T19:30:00.983505+0000","last_deep_scrub":"132'979","last_deep_scrub_stamp":"2023-10-10T19:30:00.983505+0000","last_clean_scrub_stamp":"2023-10-10T19:30:00.983505+0000","objects_scrubbed":26,"log_size":979,"log_dups_size":0,"ondisk_log_size":979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:52:48.152482+0000","scrub_duration":0.025551553000000001,"objects_trimmed":0,"snaptrim_duration":0.0071091339999999996,"stat_sum":{"num_bytes":369103,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":93,"num_read_kb":25139,"num_write":713,"num_write_kb":35847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"132'984","reported_seq":3621,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:55.671168+0000","last_change":"2023-10-10T19:29:55.671132+0000","last_active":"2023-10-10T19:29:55.671168+0000","last_peered":"2023-10-10T19:29:55.671168+0000","last_clean":"2023-10-10T19:29:55.671168+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T19:29:55.671168+0000","last_undegraded":"2023-10-10T19:29:55.671168+0000","last_fullsized":"2023-10-10T19:29:55.671168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'984","last_scrub_stamp":"2023-10-10T19:29:55.671004+0000","last_deep_scrub":"132'984","last_deep_scrub_stamp":"2023-10-10T19:29:55.671004+0000","last_clean_scrub_stamp":"2023-10-10T19:29:55.671004+0000","objects_scrubbed":39,"log_size":984,"log_dups_size":0,"ondisk_log_size":984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:40:43.124234+0000","scrub_duration":0.038408824000000001,"objects_trimmed":0,"snaptrim_duration":0.0077260940000000002,"stat_sum":{"num_bytes":131785,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":61,"num_read_kb":19697,"num_write":689,"num_write_kb":26724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"132'700","reported_seq":1135,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.351264+0000","last_change":"2023-10-10T19:29:30.351227+0000","last_active":"2023-10-10T19:29:30.351264+0000","last_peered":"2023-10-10T19:29:30.351264+0000","last_clean":"2023-10-10T19:29:30.351264+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T19:29:30.351264+0000","last_undegraded":"2023-10-10T19:29:30.351264+0000","last_fullsized":"2023-10-10T19:29:30.351264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'700","last_scrub_stamp":"2023-10-10T19:29:30.351098+0000","last_deep_scrub":"132'700","last_deep_scrub_stamp":"2023-10-10T19:29:30.351098+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.351098+0000","objects_scrubbed":17,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:16:26.607570+0000","scrub_duration":0.030203292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9108221,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":314,"num_read_kb":40841,"num_write":610,"num_write_kb":9448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":67648,"num_omap_keys":120,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'922","reported_seq":3604,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:39.851461+0000","last_change":"2023-10-10T19:29:39.851434+0000","last_active":"2023-10-10T19:29:39.851461+0000","last_peered":"2023-10-10T19:29:39.851461+0000","last_clean":"2023-10-10T19:29:39.851461+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T19:29:39.851461+0000","last_undegraded":"2023-10-10T19:29:39.851461+0000","last_fullsized":"2023-10-10T19:29:39.851461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'922","last_scrub_stamp":"2023-10-10T19:29:39.851353+0000","last_deep_scrub":"132'922","last_deep_scrub_stamp":"2023-10-10T19:29:39.851353+0000","last_clean_scrub_stamp":"2023-10-10T19:29:39.851353+0000","objects_scrubbed":42,"log_size":922,"log_dups_size":0,"ondisk_log_size":922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T07:19:01.743649+0000","scrub_duration":0.028531329000000001,"objects_trimmed":0,"snaptrim_duration":0.017091021000000001,"stat_sum":{"num_bytes":951410,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":88,"num_read_kb":19324,"num_write":674,"num_write_kb":26724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"132'152","reported_seq":320,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.682795+0000","last_change":"2023-10-10T19:29:29.682765+0000","last_active":"2023-10-10T19:29:29.682795+0000","last_peered":"2023-10-10T19:29:29.682795+0000","last_clean":"2023-10-10T19:29:29.682795+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T19:29:29.682795+0000","last_undegraded":"2023-10-10T19:29:29.682795+0000","last_fullsized":"2023-10-10T19:29:29.682795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'152","last_scrub_stamp":"2023-10-10T19:29:29.682678+0000","last_deep_scrub":"132'152","last_deep_scrub_stamp":"2023-10-10T19:29:29.682678+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.682678+0000","objects_scrubbed":15,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:06:08.947475+0000","scrub_duration":0.025814232999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388756,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":36939,"num_write":74,"num_write_kb":8306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":75158,"num_omap_keys":136,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"132'119","reported_seq":287,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.097607+0000","last_change":"2023-10-10T19:29:40.097581+0000","last_active":"2023-10-10T19:29:40.097607+0000","last_peered":"2023-10-10T19:29:40.097607+0000","last_clean":"2023-10-10T19:29:40.097607+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T19:29:40.097607+0000","last_undegraded":"2023-10-10T19:29:40.097607+0000","last_fullsized":"2023-10-10T19:29:40.097607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'119","last_scrub_stamp":"2023-10-10T19:29:40.097457+0000","last_deep_scrub":"132'119","last_deep_scrub_stamp":"2023-10-10T19:29:40.097457+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.097457+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:06:14.657014+0000","scrub_duration":0.0047236919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"132'934","reported_seq":3569,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:44.310730+0000","last_change":"2023-10-10T19:29:44.310686+0000","last_active":"2023-10-10T19:29:44.310730+0000","last_peered":"2023-10-10T19:29:44.310730+0000","last_clean":"2023-10-10T19:29:44.310730+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T19:29:44.310730+0000","last_undegraded":"2023-10-10T19:29:44.310730+0000","last_fullsized":"2023-10-10T19:29:44.310730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'934","last_scrub_stamp":"2023-10-10T19:29:44.310602+0000","last_deep_scrub":"132'934","last_deep_scrub_stamp":"2023-10-10T19:29:44.310602+0000","last_clean_scrub_stamp":"2023-10-10T19:29:44.310602+0000","objects_scrubbed":39,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:39.883204+0000","scrub_duration":0.034571985999999999,"objects_trimmed":0,"snaptrim_duration":0.010927930000000001,"stat_sum":{"num_bytes":3454752,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":55,"num_read_kb":20626,"num_write":663,"num_write_kb":30766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"132'990","reported_seq":1152,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:41.209314+0000","last_change":"2023-10-10T19:29:41.209289+0000","last_active":"2023-10-10T19:29:41.209314+0000","last_peered":"2023-10-10T19:29:41.209314+0000","last_clean":"2023-10-10T19:29:41.209314+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T19:29:41.209314+0000","last_undegraded":"2023-10-10T19:29:41.209314+0000","last_fullsized":"2023-10-10T19:29:41.209314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'990","last_scrub_stamp":"2023-10-10T19:29:41.209202+0000","last_deep_scrub":"132'990","last_deep_scrub_stamp":"2023-10-10T19:29:41.209202+0000","last_clean_scrub_stamp":"2023-10-10T19:29:41.209202+0000","objects_scrubbed":17,"log_size":990,"log_dups_size":0,"ondisk_log_size":990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:08.926653+0000","scrub_duration":0.069287455999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":38,"num_read_kb":89926,"num_write":914,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70568,"num_omap_keys":125,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"132'839","reported_seq":1003,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.127383+0000","last_change":"2023-10-10T19:29:42.127330+0000","last_active":"2023-10-10T19:29:42.127383+0000","last_peered":"2023-10-10T19:29:42.127383+0000","last_clean":"2023-10-10T19:29:42.127383+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T19:29:42.127383+0000","last_undegraded":"2023-10-10T19:29:42.127383+0000","last_fullsized":"2023-10-10T19:29:42.127383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'839","last_scrub_stamp":"2023-10-10T19:29:42.127227+0000","last_deep_scrub":"132'839","last_deep_scrub_stamp":"2023-10-10T19:29:42.127227+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.127227+0000","objects_scrubbed":14,"log_size":839,"log_dups_size":0,"ondisk_log_size":839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:46:38.680084+0000","scrub_duration":0.027082854999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":45,"num_read_kb":45594,"num_write":766,"num_write_kb":12847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":79898,"num_omap_keys":140,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"132'1056","reported_seq":3681,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.647554+0000","last_change":"2023-10-10T19:29:48.647516+0000","last_active":"2023-10-10T19:29:48.647554+0000","last_peered":"2023-10-10T19:29:48.647554+0000","last_clean":"2023-10-10T19:29:48.647554+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T19:29:48.647554+0000","last_undegraded":"2023-10-10T19:29:48.647554+0000","last_fullsized":"2023-10-10T19:29:48.647554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1056","last_scrub_stamp":"2023-10-10T19:29:48.647374+0000","last_deep_scrub":"132'1056","last_deep_scrub_stamp":"2023-10-10T19:29:48.647374+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.647374+0000","objects_scrubbed":38,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:49:57.030894+0000","scrub_duration":0.028559443,"objects_trimmed":0,"snaptrim_duration":0.0075449510000000003,"stat_sum":{"num_bytes":524227,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":65,"num_read_kb":49387,"num_write":726,"num_write_kb":52412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'292","reported_seq":459,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.731119+0000","last_change":"2023-10-10T19:29:29.731097+0000","last_active":"2023-10-10T19:29:29.731119+0000","last_peered":"2023-10-10T19:29:29.731119+0000","last_clean":"2023-10-10T19:29:29.731119+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T19:29:29.731119+0000","last_undegraded":"2023-10-10T19:29:29.731119+0000","last_fullsized":"2023-10-10T19:29:29.731119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'292","last_scrub_stamp":"2023-10-10T19:29:29.731010+0000","last_deep_scrub":"132'292","last_deep_scrub_stamp":"2023-10-10T19:29:29.731010+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.731010+0000","objects_scrubbed":18,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:44:14.879878+0000","scrub_duration":0.021479405999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":81,"num_read_kb":20675,"num_write":479,"num_write_kb":4749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":103356,"num_omap_keys":179,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"132'1000","reported_seq":3638,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:58.722286+0000","last_change":"2023-10-10T19:29:58.722260+0000","last_active":"2023-10-10T19:29:58.722286+0000","last_peered":"2023-10-10T19:29:58.722286+0000","last_clean":"2023-10-10T19:29:58.722286+0000","last_became_active":"2023-10-10T18:46:01.267092+0000","last_became_peered":"2023-10-10T18:46:01.267092+0000","last_unstale":"2023-10-10T19:29:58.722286+0000","last_undegraded":"2023-10-10T19:29:58.722286+0000","last_fullsized":"2023-10-10T19:29:58.722286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1000","last_scrub_stamp":"2023-10-10T19:29:58.722178+0000","last_deep_scrub":"132'1000","last_deep_scrub_stamp":"2023-10-10T19:29:58.722178+0000","last_clean_scrub_stamp":"2023-10-10T19:29:58.722178+0000","objects_scrubbed":43,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:10:33.521494+0000","scrub_duration":0.046338531000000002,"objects_trimmed":0,"snaptrim_duration":0.0083609700000000006,"stat_sum":{"num_bytes":2285567,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":68,"num_read_kb":33081,"num_write":691,"num_write_kb":42143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"132'166","reported_seq":324,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.151113+0000","last_change":"2023-10-10T19:29:29.151081+0000","last_active":"2023-10-10T19:29:29.151113+0000","last_peered":"2023-10-10T19:29:29.151113+0000","last_clean":"2023-10-10T19:29:29.151113+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T19:29:29.151113+0000","last_undegraded":"2023-10-10T19:29:29.151113+0000","last_fullsized":"2023-10-10T19:29:29.151113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'166","last_scrub_stamp":"2023-10-10T19:29:29.150987+0000","last_deep_scrub":"132'166","last_deep_scrub_stamp":"2023-10-10T19:29:29.150987+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.150987+0000","objects_scrubbed":12,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:52:58.469784+0000","scrub_duration":0.0090081499999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":8269,"num_write":94,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64552,"num_omap_keys":116,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"132'914","reported_seq":3586,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:43.257893+0000","last_change":"2023-10-10T19:29:43.257858+0000","last_active":"2023-10-10T19:29:43.257893+0000","last_peered":"2023-10-10T19:29:43.257893+0000","last_clean":"2023-10-10T19:29:43.257893+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T19:29:43.257893+0000","last_undegraded":"2023-10-10T19:29:43.257893+0000","last_fullsized":"2023-10-10T19:29:43.257893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'914","last_scrub_stamp":"2023-10-10T19:29:43.257766+0000","last_deep_scrub":"132'914","last_deep_scrub_stamp":"2023-10-10T19:29:43.257766+0000","last_clean_scrub_stamp":"2023-10-10T19:29:43.257766+0000","objects_scrubbed":42,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:41.946443+0000","scrub_duration":0.029919424999999999,"objects_trimmed":0,"snaptrim_duration":0.012721872,"stat_sum":{"num_bytes":482904,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":85,"num_read_kb":32652,"num_write":648,"num_write_kb":36187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"132'241","reported_seq":395,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.155442+0000","last_change":"2023-10-10T19:29:57.155415+0000","last_active":"2023-10-10T19:29:57.155442+0000","last_peered":"2023-10-10T19:29:57.155442+0000","last_clean":"2023-10-10T19:29:57.155442+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T19:29:57.155442+0000","last_undegraded":"2023-10-10T19:29:57.155442+0000","last_fullsized":"2023-10-10T19:29:57.155442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'241","last_scrub_stamp":"2023-10-10T19:29:57.155334+0000","last_deep_scrub":"132'241","last_deep_scrub_stamp":"2023-10-10T19:29:57.155334+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.155334+0000","objects_scrubbed":13,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:40:22.341488+0000","scrub_duration":0.030119740999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":57389,"num_write":173,"num_write_kb":12477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":92062,"num_omap_keys":147,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"132'971","reported_seq":3622,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.000295+0000","last_change":"2023-10-10T19:29:45.000264+0000","last_active":"2023-10-10T19:29:45.000295+0000","last_peered":"2023-10-10T19:29:45.000295+0000","last_clean":"2023-10-10T19:29:45.000295+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T19:29:45.000295+0000","last_undegraded":"2023-10-10T19:29:45.000295+0000","last_fullsized":"2023-10-10T19:29:45.000295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'971","last_scrub_stamp":"2023-10-10T19:29:45.000170+0000","last_deep_scrub":"132'971","last_deep_scrub_stamp":"2023-10-10T19:29:45.000170+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.000170+0000","objects_scrubbed":32,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:53:24.335195+0000","scrub_duration":0.025238327000000001,"objects_trimmed":0,"snaptrim_duration":0.0089059480000000003,"stat_sum":{"num_bytes":820479,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":67,"num_read_kb":26489,"num_write":692,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"132'387","reported_seq":550,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.747339+0000","last_change":"2023-10-10T19:29:28.747314+0000","last_active":"2023-10-10T19:29:28.747339+0000","last_peered":"2023-10-10T19:29:28.747339+0000","last_clean":"2023-10-10T19:29:28.747339+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T19:29:28.747339+0000","last_undegraded":"2023-10-10T19:29:28.747339+0000","last_fullsized":"2023-10-10T19:29:28.747339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'387","last_scrub_stamp":"2023-10-10T19:29:28.747246+0000","last_deep_scrub":"132'387","last_deep_scrub_stamp":"2023-10-10T19:29:28.747246+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.747246+0000","objects_scrubbed":19,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:01:46.340517+0000","scrub_duration":0.020997531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":70,"num_read_kb":20656,"num_write":333,"num_write_kb":8747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":128481,"num_omap_keys":203,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"132'951","reported_seq":3597,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:03.064395+0000","last_change":"2023-10-10T19:30:03.064365+0000","last_active":"2023-10-10T19:30:03.064395+0000","last_peered":"2023-10-10T19:30:03.064395+0000","last_clean":"2023-10-10T19:30:03.064395+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T19:30:03.064395+0000","last_undegraded":"2023-10-10T19:30:03.064395+0000","last_fullsized":"2023-10-10T19:30:03.064395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'951","last_scrub_stamp":"2023-10-10T19:30:03.064283+0000","last_deep_scrub":"132'951","last_deep_scrub_stamp":"2023-10-10T19:30:03.064283+0000","last_clean_scrub_stamp":"2023-10-10T19:30:03.064283+0000","objects_scrubbed":43,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:39:24.338470+0000","scrub_duration":0.041115824000000002,"objects_trimmed":0,"snaptrim_duration":0.012052205,"stat_sum":{"num_bytes":635543,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":63,"num_read_kb":40064,"num_write":667,"num_write_kb":56874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"132'1325","reported_seq":1491,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:50.083396+0000","last_change":"2023-10-10T19:29:31.691013+0000","last_active":"2023-10-10T19:29:50.083396+0000","last_peered":"2023-10-10T19:29:50.083396+0000","last_clean":"2023-10-10T19:29:50.083396+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T19:29:50.083396+0000","last_undegraded":"2023-10-10T19:29:50.083396+0000","last_fullsized":"2023-10-10T19:29:50.083396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1319","last_scrub_stamp":"2023-10-10T19:29:31.690927+0000","last_deep_scrub":"132'1319","last_deep_scrub_stamp":"2023-10-10T19:29:31.690927+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.690927+0000","objects_scrubbed":14,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:38:48.082901+0000","scrub_duration":0.043989502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15286709,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":48,"num_read_kb":53317,"num_write":1726,"num_write_kb":25012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40110,"num_omap_keys":72,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"132'933","reported_seq":3577,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.037397+0000","last_change":"2023-10-10T19:29:57.037345+0000","last_active":"2023-10-10T19:29:57.037397+0000","last_peered":"2023-10-10T19:29:57.037397+0000","last_clean":"2023-10-10T19:29:57.037397+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T19:29:57.037397+0000","last_undegraded":"2023-10-10T19:29:57.037397+0000","last_fullsized":"2023-10-10T19:29:57.037397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'933","last_scrub_stamp":"2023-10-10T19:29:57.037218+0000","last_deep_scrub":"132'933","last_deep_scrub_stamp":"2023-10-10T19:29:57.037218+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.037218+0000","objects_scrubbed":40,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:12:09.618486+0000","scrub_duration":0.037571087000000003,"objects_trimmed":0,"snaptrim_duration":0.015827366999999998,"stat_sum":{"num_bytes":178682,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":58,"num_read_kb":28965,"num_write":652,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"132'1215","reported_seq":1380,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.115778+0000","last_change":"2023-10-10T19:29:45.115751+0000","last_active":"2023-10-10T19:29:45.115778+0000","last_peered":"2023-10-10T19:29:45.115778+0000","last_clean":"2023-10-10T19:29:45.115778+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T19:29:45.115778+0000","last_undegraded":"2023-10-10T19:29:45.115778+0000","last_fullsized":"2023-10-10T19:29:45.115778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1215","last_scrub_stamp":"2023-10-10T19:29:45.115661+0000","last_deep_scrub":"132'1215","last_deep_scrub_stamp":"2023-10-10T19:29:45.115661+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.115661+0000","objects_scrubbed":16,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:01.423877+0000","scrub_duration":0.048047320999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":49,"num_read_kb":89747,"num_write":1128,"num_write_kb":25455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70296,"num_omap_keys":124,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"132'938","reported_seq":3567,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.144929+0000","last_change":"2023-10-10T19:16:54.781695+0000","last_active":"2023-10-10T19:29:22.144929+0000","last_peered":"2023-10-10T19:29:22.144929+0000","last_clean":"2023-10-10T19:29:22.144929+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T19:29:22.144929+0000","last_undegraded":"2023-10-10T19:29:22.144929+0000","last_fullsized":"2023-10-10T19:29:22.144929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065052069999999998,"stat_sum":{"num_bytes":479125,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":64,"num_read_kb":10527,"num_write":680,"num_write_kb":20069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"132'323","reported_seq":474,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:34.384672+0000","last_change":"2023-10-10T19:29:34.384643+0000","last_active":"2023-10-10T19:29:34.384672+0000","last_peered":"2023-10-10T19:29:34.384672+0000","last_clean":"2023-10-10T19:29:34.384672+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T19:29:34.384672+0000","last_undegraded":"2023-10-10T19:29:34.384672+0000","last_fullsized":"2023-10-10T19:29:34.384672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'323","last_scrub_stamp":"2023-10-10T19:29:34.384547+0000","last_deep_scrub":"132'323","last_deep_scrub_stamp":"2023-10-10T19:29:34.384547+0000","last_clean_scrub_stamp":"2023-10-10T19:29:34.384547+0000","objects_scrubbed":12,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:06:08.523893+0000","scrub_duration":0.015112752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194858,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":28,"num_read_kb":40985,"num_write":233,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":71992,"num_omap_keys":129,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"132'967","reported_seq":3629,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:59.149786+0000","last_change":"2023-10-10T19:29:59.149750+0000","last_active":"2023-10-10T19:29:59.149786+0000","last_peered":"2023-10-10T19:29:59.149786+0000","last_clean":"2023-10-10T19:29:59.149786+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T19:29:59.149786+0000","last_undegraded":"2023-10-10T19:29:59.149786+0000","last_fullsized":"2023-10-10T19:29:59.149786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'967","last_scrub_stamp":"2023-10-10T19:29:59.149633+0000","last_deep_scrub":"132'967","last_deep_scrub_stamp":"2023-10-10T19:29:59.149633+0000","last_clean_scrub_stamp":"2023-10-10T19:29:59.149633+0000","objects_scrubbed":48,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:53:06.235612+0000","scrub_duration":0.046782177000000001,"objects_trimmed":0,"snaptrim_duration":0.016889265000000001,"stat_sum":{"num_bytes":1346662,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":67,"num_read_kb":24822,"num_write":699,"num_write_kb":32529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"132'357","reported_seq":512,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:55.078895+0000","last_change":"2023-10-10T19:29:37.661226+0000","last_active":"2023-10-10T19:29:55.078895+0000","last_peered":"2023-10-10T19:29:55.078895+0000","last_clean":"2023-10-10T19:29:55.078895+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T19:29:55.078895+0000","last_undegraded":"2023-10-10T19:29:55.078895+0000","last_fullsized":"2023-10-10T19:29:55.078895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'352","last_scrub_stamp":"2023-10-10T19:29:37.661098+0000","last_deep_scrub":"132'352","last_deep_scrub_stamp":"2023-10-10T19:29:37.661098+0000","last_clean_scrub_stamp":"2023-10-10T19:29:37.661098+0000","objects_scrubbed":11,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:14:09.979735+0000","scrub_duration":0.032683222999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11246969,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":27,"num_read_kb":50704,"num_write":277,"num_write_kb":15297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":68490,"num_omap_keys":115,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"132'970","reported_seq":3598,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.156927+0000","last_change":"2023-10-10T19:16:55.417128+0000","last_active":"2023-10-10T19:29:22.156927+0000","last_peered":"2023-10-10T19:29:22.156927+0000","last_clean":"2023-10-10T19:29:22.156927+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T19:29:22.156927+0000","last_undegraded":"2023-10-10T19:29:22.156927+0000","last_fullsized":"2023-10-10T19:29:22.156927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012585610000000001,"stat_sum":{"num_bytes":222806,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":60,"num_read_kb":19964,"num_write":709,"num_write_kb":27205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"132'322","reported_seq":481,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.316606+0000","last_change":"2023-10-10T19:29:31.316564+0000","last_active":"2023-10-10T19:29:31.316606+0000","last_peered":"2023-10-10T19:29:31.316606+0000","last_clean":"2023-10-10T19:29:31.316606+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T19:29:31.316606+0000","last_undegraded":"2023-10-10T19:29:31.316606+0000","last_fullsized":"2023-10-10T19:29:31.316606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'322","last_scrub_stamp":"2023-10-10T19:29:31.316447+0000","last_deep_scrub":"132'322","last_deep_scrub_stamp":"2023-10-10T19:29:31.316447+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.316447+0000","objects_scrubbed":12,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:27:45.094335+0000","scrub_duration":0.025432287000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":49,"num_read_kb":57404,"num_write":260,"num_write_kb":8464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":65503,"num_omap_keys":117,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"132'1049","reported_seq":3690,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:05.097815+0000","last_change":"2023-10-10T19:30:05.097774+0000","last_active":"2023-10-10T19:30:05.097815+0000","last_peered":"2023-10-10T19:30:05.097815+0000","last_clean":"2023-10-10T19:30:05.097815+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T19:30:05.097815+0000","last_undegraded":"2023-10-10T19:30:05.097815+0000","last_fullsized":"2023-10-10T19:30:05.097815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1049","last_scrub_stamp":"2023-10-10T19:30:05.097679+0000","last_deep_scrub":"132'1049","last_deep_scrub_stamp":"2023-10-10T19:30:05.097679+0000","last_clean_scrub_stamp":"2023-10-10T19:30:05.097679+0000","objects_scrubbed":34,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:53.287113+0000","scrub_duration":0.033390343000000003,"objects_trimmed":0,"snaptrim_duration":0.016837566000000002,"stat_sum":{"num_bytes":394703,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":67,"num_read_kb":28062,"num_write":753,"num_write_kb":23722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"132'910","reported_seq":3571,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:22.149293+0000","last_change":"2023-10-10T19:16:54.782474+0000","last_active":"2023-10-10T19:29:22.149293+0000","last_peered":"2023-10-10T19:29:22.149293+0000","last_clean":"2023-10-10T19:29:22.149293+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T19:29:22.149293+0000","last_undegraded":"2023-10-10T19:29:22.149293+0000","last_fullsized":"2023-10-10T19:29:22.149293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T18:45:10.384742+0000","last_clean_scrub_stamp":"2023-10-10T18:45:10.384742+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074078640000000001,"stat_sum":{"num_bytes":519476,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":77,"num_read_kb":25056,"num_write":683,"num_write_kb":32109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"132'1017","reported_seq":3650,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:41.182730+0000","last_change":"2023-10-10T19:29:41.182702+0000","last_active":"2023-10-10T19:29:41.182730+0000","last_peered":"2023-10-10T19:29:41.182730+0000","last_clean":"2023-10-10T19:29:41.182730+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T19:29:41.182730+0000","last_undegraded":"2023-10-10T19:29:41.182730+0000","last_fullsized":"2023-10-10T19:29:41.182730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1017","last_scrub_stamp":"2023-10-10T19:29:41.182614+0000","last_deep_scrub":"132'1017","last_deep_scrub_stamp":"2023-10-10T19:29:41.182614+0000","last_clean_scrub_stamp":"2023-10-10T19:29:41.182614+0000","objects_scrubbed":32,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:38:55.992116+0000","scrub_duration":0.028001608000000001,"objects_trimmed":0,"snaptrim_duration":0.0050938800000000003,"stat_sum":{"num_bytes":416129,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":69,"num_read_kb":25013,"num_write":702,"num_write_kb":37226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"132'1001","reported_seq":3660,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.132565+0000","last_change":"2023-10-10T19:29:47.132524+0000","last_active":"2023-10-10T19:29:47.132565+0000","last_peered":"2023-10-10T19:29:47.132565+0000","last_clean":"2023-10-10T19:29:47.132565+0000","last_became_active":"2023-10-10T18:46:00.606420+0000","last_became_peered":"2023-10-10T18:46:00.606420+0000","last_unstale":"2023-10-10T19:29:47.132565+0000","last_undegraded":"2023-10-10T19:29:47.132565+0000","last_fullsized":"2023-10-10T19:29:47.132565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1001","last_scrub_stamp":"2023-10-10T19:29:47.132433+0000","last_deep_scrub":"132'1001","last_deep_scrub_stamp":"2023-10-10T19:29:47.132433+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.132433+0000","objects_scrubbed":41,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:35:01.992844+0000","scrub_duration":0.032812747000000003,"objects_trimmed":0,"snaptrim_duration":0.0064370249999999999,"stat_sum":{"num_bytes":103727,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":90,"num_read_kb":45570,"num_write":716,"num_write_kb":52505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"132'973","reported_seq":3611,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:49.279974+0000","last_change":"2023-10-10T19:29:49.279944+0000","last_active":"2023-10-10T19:29:49.279974+0000","last_peered":"2023-10-10T19:29:49.279974+0000","last_clean":"2023-10-10T19:29:49.279974+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T19:29:49.279974+0000","last_undegraded":"2023-10-10T19:29:49.279974+0000","last_fullsized":"2023-10-10T19:29:49.279974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'973","last_scrub_stamp":"2023-10-10T19:29:49.279855+0000","last_deep_scrub":"132'973","last_deep_scrub_stamp":"2023-10-10T19:29:49.279855+0000","last_clean_scrub_stamp":"2023-10-10T19:29:49.279855+0000","objects_scrubbed":37,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:38.764535+0000","scrub_duration":0.031787211000000003,"objects_trimmed":0,"snaptrim_duration":0.014096529,"stat_sum":{"num_bytes":456616,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":70,"num_read_kb":16093,"num_write":668,"num_write_kb":20834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"132'959","reported_seq":3608,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:00.042069+0000","last_change":"2023-10-10T19:30:00.042044+0000","last_active":"2023-10-10T19:30:00.042069+0000","last_peered":"2023-10-10T19:30:00.042069+0000","last_clean":"2023-10-10T19:30:00.042069+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T19:30:00.042069+0000","last_undegraded":"2023-10-10T19:30:00.042069+0000","last_fullsized":"2023-10-10T19:30:00.042069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'959","last_scrub_stamp":"2023-10-10T19:30:00.041947+0000","last_deep_scrub":"132'959","last_deep_scrub_stamp":"2023-10-10T19:30:00.041947+0000","last_clean_scrub_stamp":"2023-10-10T19:30:00.041947+0000","objects_scrubbed":41,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:38:23.770402+0000","scrub_duration":0.034907344,"objects_trimmed":0,"snaptrim_duration":0.016808101999999998,"stat_sum":{"num_bytes":708082,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":68,"num_read_kb":48841,"num_write":679,"num_write_kb":56033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"132'1009","reported_seq":3623,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:53.644317+0000","last_change":"2023-10-10T19:29:53.644291+0000","last_active":"2023-10-10T19:29:53.644317+0000","last_peered":"2023-10-10T19:29:53.644317+0000","last_clean":"2023-10-10T19:29:53.644317+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T19:29:53.644317+0000","last_undegraded":"2023-10-10T19:29:53.644317+0000","last_fullsized":"2023-10-10T19:29:53.644317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1009","last_scrub_stamp":"2023-10-10T19:29:53.644211+0000","last_deep_scrub":"132'1009","last_deep_scrub_stamp":"2023-10-10T19:29:53.644211+0000","last_clean_scrub_stamp":"2023-10-10T19:29:53.644211+0000","objects_scrubbed":30,"log_size":1009,"log_dups_size":0,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:15:02.264513+0000","scrub_duration":0.035309926999999998,"objects_trimmed":0,"snaptrim_duration":0.0068654069999999996,"stat_sum":{"num_bytes":747011,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":59,"num_read_kb":41116,"num_write":675,"num_write_kb":48421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"132'1014","reported_seq":3648,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:08.003144+0000","last_change":"2023-10-10T19:30:08.003117+0000","last_active":"2023-10-10T19:30:08.003144+0000","last_peered":"2023-10-10T19:30:08.003144+0000","last_clean":"2023-10-10T19:30:08.003144+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T19:30:08.003144+0000","last_undegraded":"2023-10-10T19:30:08.003144+0000","last_fullsized":"2023-10-10T19:30:08.003144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1014","last_scrub_stamp":"2023-10-10T19:30:08.003025+0000","last_deep_scrub":"132'1014","last_deep_scrub_stamp":"2023-10-10T19:30:08.003025+0000","last_clean_scrub_stamp":"2023-10-10T19:30:08.003025+0000","objects_scrubbed":36,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:14:37.238471+0000","scrub_duration":0.041397751000000003,"objects_trimmed":0,"snaptrim_duration":0.0081260870000000006,"stat_sum":{"num_bytes":4820268,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":70,"num_read_kb":15871,"num_write":698,"num_write_kb":28703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"132'968","reported_seq":3616,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:01.048137+0000","last_change":"2023-10-10T19:30:01.048111+0000","last_active":"2023-10-10T19:30:01.048137+0000","last_peered":"2023-10-10T19:30:01.048137+0000","last_clean":"2023-10-10T19:30:01.048137+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T19:30:01.048137+0000","last_undegraded":"2023-10-10T19:30:01.048137+0000","last_fullsized":"2023-10-10T19:30:01.048137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'968","last_scrub_stamp":"2023-10-10T19:30:01.048031+0000","last_deep_scrub":"132'968","last_deep_scrub_stamp":"2023-10-10T19:30:01.048031+0000","last_clean_scrub_stamp":"2023-10-10T19:30:01.048031+0000","objects_scrubbed":35,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:51:18.799692+0000","scrub_duration":0.030600175,"objects_trimmed":0,"snaptrim_duration":0.017981724000000001,"stat_sum":{"num_bytes":834726,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":72,"num_read_kb":44513,"num_write":669,"num_write_kb":47263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"132'945","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:54.601048+0000","last_change":"2023-10-10T19:29:54.600997+0000","last_active":"2023-10-10T19:29:54.601048+0000","last_peered":"2023-10-10T19:29:54.601048+0000","last_clean":"2023-10-10T19:29:54.601048+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T19:29:54.601048+0000","last_undegraded":"2023-10-10T19:29:54.601048+0000","last_fullsized":"2023-10-10T19:29:54.601048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'945","last_scrub_stamp":"2023-10-10T19:29:54.600850+0000","last_deep_scrub":"132'945","last_deep_scrub_stamp":"2023-10-10T19:29:54.600850+0000","last_clean_scrub_stamp":"2023-10-10T19:29:54.600850+0000","objects_scrubbed":34,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:46:40.344869+0000","scrub_duration":0.031231462000000001,"objects_trimmed":0,"snaptrim_duration":0.010756557999999999,"stat_sum":{"num_bytes":807796,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":71,"num_read_kb":21097,"num_write":685,"num_write_kb":25593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"132'924","reported_seq":3573,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:52.038445+0000","last_change":"2023-10-10T19:29:52.038418+0000","last_active":"2023-10-10T19:29:52.038445+0000","last_peered":"2023-10-10T19:29:52.038445+0000","last_clean":"2023-10-10T19:29:52.038445+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T19:29:52.038445+0000","last_undegraded":"2023-10-10T19:29:52.038445+0000","last_fullsized":"2023-10-10T19:29:52.038445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'924","last_scrub_stamp":"2023-10-10T19:29:52.038297+0000","last_deep_scrub":"132'924","last_deep_scrub_stamp":"2023-10-10T19:29:52.038297+0000","last_clean_scrub_stamp":"2023-10-10T19:29:52.038297+0000","objects_scrubbed":28,"log_size":924,"log_dups_size":0,"ondisk_log_size":924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:34:00.960972+0000","scrub_duration":0.031392050999999997,"objects_trimmed":0,"snaptrim_duration":0.014012482999999999,"stat_sum":{"num_bytes":684647,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":69,"num_read_kb":49258,"num_write":638,"num_write_kb":56905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"132'993","reported_seq":3638,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:07.632893+0000","last_change":"2023-10-10T19:30:07.632853+0000","last_active":"2023-10-10T19:30:07.632893+0000","last_peered":"2023-10-10T19:30:07.632893+0000","last_clean":"2023-10-10T19:30:07.632893+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T19:30:07.632893+0000","last_undegraded":"2023-10-10T19:30:07.632893+0000","last_fullsized":"2023-10-10T19:30:07.632893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'993","last_scrub_stamp":"2023-10-10T19:30:07.632760+0000","last_deep_scrub":"132'993","last_deep_scrub_stamp":"2023-10-10T19:30:07.632760+0000","last_clean_scrub_stamp":"2023-10-10T19:30:07.632760+0000","objects_scrubbed":34,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:20:31.200915+0000","scrub_duration":0.035667892,"objects_trimmed":0,"snaptrim_duration":0.0097177140000000006,"stat_sum":{"num_bytes":571513,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":59,"num_read_kb":20609,"num_write":714,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"132'927","reported_seq":3598,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:00.406510+0000","last_change":"2023-10-10T19:30:00.406485+0000","last_active":"2023-10-10T19:30:00.406510+0000","last_peered":"2023-10-10T19:30:00.406510+0000","last_clean":"2023-10-10T19:30:00.406510+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T19:30:00.406510+0000","last_undegraded":"2023-10-10T19:30:00.406510+0000","last_fullsized":"2023-10-10T19:30:00.406510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'927","last_scrub_stamp":"2023-10-10T19:30:00.406402+0000","last_deep_scrub":"132'927","last_deep_scrub_stamp":"2023-10-10T19:30:00.406402+0000","last_clean_scrub_stamp":"2023-10-10T19:30:00.406402+0000","objects_scrubbed":40,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:36:31.446310+0000","scrub_duration":0.034815272000000001,"objects_trimmed":0,"snaptrim_duration":0.013935637000000001,"stat_sum":{"num_bytes":366510,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":77,"num_read_kb":51992,"num_write":664,"num_write_kb":56578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"132'1043","reported_seq":3697,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:50.062490+0000","last_change":"2023-10-10T19:29:50.062453+0000","last_active":"2023-10-10T19:29:50.062490+0000","last_peered":"2023-10-10T19:29:50.062490+0000","last_clean":"2023-10-10T19:29:50.062490+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T19:29:50.062490+0000","last_undegraded":"2023-10-10T19:29:50.062490+0000","last_fullsized":"2023-10-10T19:29:50.062490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1043","last_scrub_stamp":"2023-10-10T19:29:50.062332+0000","last_deep_scrub":"132'1043","last_deep_scrub_stamp":"2023-10-10T19:29:50.062332+0000","last_clean_scrub_stamp":"2023-10-10T19:29:50.062332+0000","objects_scrubbed":42,"log_size":1043,"log_dups_size":0,"ondisk_log_size":1043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:41:49.718435+0000","scrub_duration":0.035570905999999999,"objects_trimmed":0,"snaptrim_duration":0.0088155260000000006,"stat_sum":{"num_bytes":278227,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":77,"num_read_kb":37312,"num_write":744,"num_write_kb":49182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":50927477,"num_objects":2366,"num_object_clones":0,"num_object_copies":4732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":4408,"num_read_kb":1953419,"num_write":44061,"num_write_kb":2517898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":76832768,"data_stored":101854954,"data_compressed":30508016,"data_compressed_allocated":33341440,"data_compressed_original":70082560,"omap_allocated":0,"internal_metadata":0},"log_size":62599,"ondisk_log_size":62599,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":336884642,"num_objects":493,"num_object_clones":0,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":2858,"num_read_kb":1531013,"num_write":19196,"num_write_kb":469942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":353,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2518057,"num_omap_keys":4386,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183304192,"data_stored":673769284,"data_compressed":87276636,"data_compressed_allocated":112345088,"data_compressed_original":603062272,"omap_allocated":0,"internal_metadata":0},"log_size":20286,"ondisk_log_size":20286,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575393,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348880,"kb_used_data":47496,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94022960,"statfs":{"total":96636764160,"available":96279511040,"internally_reserved":0,"allocated":48635904,"data_stored":134725171,"data_compressed":20528468,"data_compressed_allocated":25550848,"data_compressed_original":114077696,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.53700000000000003},"min":{"1min":0.33500000000000002,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.72799999999999998,"5min":0.748,"15min":1.018},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.25},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":2.153},"last":0.55200000000000005}]},{"osd":1,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.34799999999999998,"5min":0.318,"15min":0.26200000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":1.802},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57899999999999996,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72599999999999998,"5min":0.82399999999999995,"15min":2.2170000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.44900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.76600000000000001,"5min":2.871,"15min":2.871},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":2.1989999999999998},"last":0.71899999999999997}]},{"osd":3,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.34699999999999998,"15min":0.317},"max":{"1min":0.67000000000000004,"5min":0.874,"15min":1.0349999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.28100000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.7649999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.47599999999999998},"min":{"1min":0.35199999999999998,"5min":0.307,"15min":0.28000000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.089},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55800000000000005,"15min":0.54200000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.72399999999999998,"5min":0.78400000000000003,"15min":1.9419999999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.502,"15min":0.51400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":1.71},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.42699999999999999,"5min":0.378,"15min":0.311},"max":{"1min":0.79800000000000004,"5min":0.80500000000000005,"15min":1.177},"last":0.46600000000000003}]},{"osd":6,"last update":"Tue Oct 10 19:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60099999999999998,"15min":0.58599999999999997},"min":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.31},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.872},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.33900000000000002,"15min":0.29699999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":2.2559999999999998},"last":0.52800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575392,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337504,"kb_used_data":36120,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94034336,"statfs":{"total":96636764160,"available":96291160064,"internally_reserved":0,"allocated":36986880,"data_stored":92212231,"data_compressed":11950270,"data_compressed_allocated":14790656,"data_compressed_original":71962624,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.80600000000000005,"15min":0.81100000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.216},"max":{"1min":0.68300000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.43099999999999999}]},{"osd":1,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58699999999999997,"15min":0.56000000000000005},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.85299999999999998},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.56499999999999995},"min":{"1min":0.41799999999999998,"5min":0.379,"15min":0.222},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.86799999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.56200000000000006},"min":{"1min":0.48699999999999999,"5min":0.40400000000000003,"15min":0.30499999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90500000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.55300000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.29999999999999999},"max":{"1min":0.67500000000000004,"5min":0.76500000000000001,"15min":0.97599999999999998},"last":0.49399999999999999}]},{"osd":3,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57999999999999996,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.32900000000000001},"max":{"1min":0.628,"5min":0.82799999999999996,"15min":0.88200000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.437,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.85699999999999998,"15min":0.95099999999999996},"last":0.51300000000000001}]},{"osd":4,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48199999999999998,"15min":0.46700000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.248},"max":{"1min":0.749,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54900000000000004,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.80300000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45700000000000002}]},{"osd":5,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73599999999999999,"15min":0.879},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51900000000000002,"15min":0.53400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.26400000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.99299999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.31},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.88900000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.38,"15min":0.27700000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.86199999999999999},"last":0.54600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575391,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342436,"kb_used_data":32860,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94029404,"statfs":{"total":96636764160,"available":96286109696,"internally_reserved":0,"allocated":33648640,"data_stored":116978587,"data_compressed":17811264,"data_compressed_allocated":21970944,"data_compressed_original":107048960,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.31,"15min":0.315},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17699999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.56899999999999995},"last":0.376},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41999999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":0.66700000000000004,"15min":0.70799999999999996},"last":0.504}]},{"osd":2,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.34899999999999998,"15min":0.35399999999999998},"min":{"1min":0.246,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.55200000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38800000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.19900000000000001},"max":{"1min":0.56699999999999995,"5min":0.64900000000000002,"15min":0.68899999999999995},"last":0.47599999999999998}]},{"osd":3,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56299999999999994,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.27700000000000002},"max":{"1min":0.66700000000000004,"5min":0.72899999999999998,"15min":0.84299999999999997},"last":0.54300000000000004}]},{"osd":4,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.41999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.175},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.875,"5min":0.875,"15min":2.2519999999999998},"last":0.63900000000000001}]},{"osd":5,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.90900000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.501,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.71399999999999997}]},{"osd":6,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.46100000000000002,"5min":0.40500000000000003,"15min":0.37},"max":{"1min":0.81200000000000006,"5min":0.86199999999999999,"15min":1.038},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.38900000000000001,"15min":0.36299999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.123},"last":0.79000000000000004}]},{"osd":7,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.621},"min":{"1min":0.46700000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":5.2720000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61899999999999999,"15min":0.64800000000000002},"min":{"1min":0.47899999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":5.2599999999999998},"last":0.752}]}]},{"osd":0,"up_from":13,"seq":55834575393,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338068,"kb_used_data":32588,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94033772,"statfs":{"total":96636764160,"available":96290582528,"internally_reserved":0,"allocated":33370112,"data_stored":88805479,"data_compressed":14490668,"data_compressed_allocated":17825792,"data_compressed_original":75632640,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.66500000000000004},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42399999999999999,"15min":0.41699999999999998},"min":{"1min":0.307,"5min":0.25800000000000001,"15min":0.20300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66000000000000003,"15min":0.66500000000000004},"last":0.60599999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35999999999999999,"15min":0.372},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.52000000000000002,"5min":0.52500000000000002,"15min":2.028},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40799999999999997},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53400000000000003,"5min":0.61299999999999999,"15min":0.77300000000000002},"last":0.42799999999999999}]},{"osd":3,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.39600000000000002},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.79100000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.80300000000000005},"last":0.57499999999999996}]},{"osd":4,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.629,"15min":0.63100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.26000000000000001},"max":{"1min":0.84399999999999997,"5min":0.97599999999999998,"15min":1.9610000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.38400000000000001},"max":{"1min":0.877,"5min":0.999,"15min":0.999},"last":0.81799999999999995}]},{"osd":5,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.35299999999999998},"max":{"1min":0.81799999999999995,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72699999999999998,"5min":0.873,"15min":0.96799999999999997},"last":0.65900000000000003}]},{"osd":6,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":0.85699999999999998,"5min":0.96299999999999997,"15min":2.0019999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.627,"15min":0.62},"min":{"1min":0.43099999999999999,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":0.73599999999999999,"5min":0.94099999999999995,"15min":0.95399999999999996},"last":0.67100000000000004}]},{"osd":7,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.63100000000000001,"15min":0.63400000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.63700000000000001,"15min":0.64900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.80100000000000005,"5min":0.89500000000000002,"15min":2.0539999999999998},"last":0.74199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575393,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319288,"kb_used_data":17904,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94052552,"statfs":{"total":96636764160,"available":96309813248,"internally_reserved":0,"allocated":18333696,"data_stored":57320502,"data_compressed":8516380,"data_compressed_allocated":10579968,"data_compressed_original":51261440,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34399999999999997,"15min":0.33500000000000002},"min":{"1min":0.224,"5min":0.219,"15min":0.186},"max":{"1min":0.45000000000000001,"5min":0.55400000000000005,"15min":0.60799999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.53500000000000003,"5min":0.65100000000000002,"15min":0.67500000000000004},"last":0.32400000000000001}]},{"osd":1,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20999999999999999},"max":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.65700000000000003},"last":0.379},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.73999999999999999},"last":0.40799999999999997}]},{"osd":3,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.24399999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.42499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.23899999999999999},"max":{"1min":0.59099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.76900000000000002},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.26600000000000001},"max":{"1min":0.751,"5min":0.95599999999999996,"15min":48.729999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.25900000000000001},"max":{"1min":0.74099999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.48399999999999999}]},{"osd":5,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63,"15min":0.628},"min":{"1min":0.436,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":1.02,"15min":1.02},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.32400000000000001},"max":{"1min":0.81000000000000005,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.51000000000000001}]},{"osd":6,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.78900000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":1.0760000000000001,"15min":48.685000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.38800000000000001},"max":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":0.97899999999999998},"last":0.435}]},{"osd":7,"last update":"Tue Oct 10 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64800000000000002,"15min":0.80700000000000005},"min":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.38700000000000001},"max":{"1min":0.85799999999999998,"5min":1.052,"15min":48.710000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.034,"15min":1.1459999999999999},"last":0.45000000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575392,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335672,"kb_used_data":38384,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94036168,"statfs":{"total":96636764160,"available":96293036032,"internally_reserved":0,"allocated":39305216,"data_stored":100336269,"data_compressed":14072432,"data_compressed_allocated":17829888,"data_compressed_original":81338368,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:55 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35099999999999998,"15min":0.37},"min":{"1min":0.21099999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.58799999999999997,"5min":0.64900000000000002,"15min":5.0670000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.374,"15min":0.39000000000000001},"min":{"1min":0.223,"5min":0.20899999999999999,"15min":0.189},"max":{"1min":0.49099999999999999,"5min":0.59899999999999998,"15min":2.7799999999999998},"last":0.33300000000000002}]},{"osd":1,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.373,"15min":0.379},"min":{"1min":0.25900000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.54600000000000004,"5min":0.628,"15min":2.758},"last":0.435},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39300000000000002,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.64500000000000002},"last":0.45700000000000002}]},{"osd":2,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.25},"max":{"1min":0.60099999999999998,"5min":0.68400000000000005,"15min":1.333},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.41999999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.24199999999999999},"max":{"1min":0.56100000000000005,"5min":0.65400000000000003,"15min":0.98699999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.623,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79600000000000004,"5min":0.89600000000000002,"15min":0.95499999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63400000000000001,"15min":0.63700000000000001},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":1.0169999999999999,"15min":1.23},"last":0.53300000000000003}]},{"osd":5,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.84199999999999997,"5min":0.97699999999999998,"15min":1.2929999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.44800000000000001,"5min":0.40500000000000003,"15min":0.36199999999999999},"max":{"1min":0.875,"5min":0.93600000000000005,"15min":1.103},"last":0.65200000000000002}]},{"osd":6,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.64100000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40300000000000002},"max":{"1min":0.90600000000000003,"5min":0.94199999999999995,"15min":1.198},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.88500000000000001,"5min":0.94899999999999995,"15min":2.7309999999999999},"last":0.93000000000000005}]},{"osd":7,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.39400000000000002},"max":{"1min":0.85699999999999998,"5min":0.97999999999999998,"15min":2.6619999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.49299999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86399999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.84699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575392,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344800,"kb_used_data":35224,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94027040,"statfs":{"total":96636764160,"available":96283688960,"internally_reserved":0,"allocated":36069376,"data_stored":105586308,"data_compressed":16193482,"data_compressed_allocated":19931136,"data_compressed_original":91820032,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33400000000000002,"5min":0.27600000000000002,"15min":0.21199999999999999},"max":{"1min":0.69599999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59899999999999998,"15min":0.58299999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.23699999999999999},"max":{"1min":0.74399999999999999,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.67700000000000005}]},{"osd":1,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.35599999999999998,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":0.67600000000000005,"5min":0.71199999999999997,"15min":1.5920000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":1.0169999999999999},"last":0.503}]},{"osd":2,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.42599999999999999,"5min":0.41999999999999998,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":1.615},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.40100000000000002,"15min":0.36699999999999999},"max":{"1min":0.77800000000000002,"5min":0.85799999999999998,"15min":0.98199999999999998},"last":0.63300000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.57999999999999996},"min":{"1min":0.376,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.746,"5min":0.85199999999999998,"15min":0.85799999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.28199999999999997},"max":{"1min":0.78100000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.76700000000000002}]},{"osd":5,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.47099999999999997},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.26300000000000001},"max":{"1min":0.71599999999999997,"5min":0.84199999999999997,"15min":1.087},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.82399999999999995,"5min":0.88,"15min":1.111},"last":0.46700000000000003}]},{"osd":6,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.29799999999999999},"max":{"1min":0.82599999999999996,"5min":0.83099999999999996,"15min":1.5580000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.98799999999999999},"last":0.53800000000000003}]},{"osd":7,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.45200000000000001,"5min":0.37,"15min":0.312},"max":{"1min":0.86099999999999999,"5min":1.016,"15min":1.0620000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.64200000000000002},"min":{"1min":0.503,"5min":0.40000000000000002,"15min":0.33700000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":1.1160000000000001},"last":0.49299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575392,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331424,"kb_used_data":25944,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94040416,"statfs":{"total":96636764160,"available":96297385984,"internally_reserved":0,"allocated":26566656,"data_stored":87927915,"data_compressed":14286216,"data_compressed_allocated":17297408,"data_compressed_original":81149952,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.249},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.82199999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.752,"5min":0.76700000000000002,"15min":0.85099999999999998},"last":0.503}]},{"osd":1,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58499999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.314},"max":{"1min":0.78600000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.57499999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.28100000000000003},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81899999999999995},"last":0.48099999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.54200000000000004},"min":{"1min":0.442,"5min":0.35499999999999998,"15min":0.30499999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.28100000000000003},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.42299999999999999}]},{"osd":3,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57199999999999995,"15min":0.55500000000000005},"min":{"1min":0.39600000000000002,"5min":0.377,"15min":0.32600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.76600000000000001,"5min":0.81799999999999995,"15min":0.94899999999999995},"last":0.61399999999999999}]},{"osd":4,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.45800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57699999999999996,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.23000000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.45100000000000001}]},{"osd":6,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.502,"15min":0.496},"min":{"1min":0.316,"5min":0.316,"15min":0.26700000000000002},"max":{"1min":0.75800000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.77800000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.39300000000000002}]},{"osd":7,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79600000000000004,"5min":0.80000000000000004,"15min":0.89400000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61099999999999999,"15min":0.60199999999999998},"min":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.27700000000000002},"max":{"1min":0.82199999999999995,"5min":0.82599999999999996,"15min":0.90800000000000003},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20922368,"data_stored":71948432,"data_compressed":9641340,"data_compressed_allocated":12496896,"data_compressed_original":63553536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25841664,"data_stored":106226300,"data_compressed":13852450,"data_compressed_allocated":17678336,"data_compressed_original":98091008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":51052075,"data_compressed":6450195,"data_compressed_allocated":8323072,"data_compressed_original":46788608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26312704,"data_stored":85394164,"data_compressed":10438632,"data_compressed_allocated":13819904,"data_compressed_original":72937472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24768512,"data_stored":91918527,"data_compressed":12211424,"data_compressed_allocated":15597568,"data_compressed_original":82771968,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16703488,"data_stored":74680123,"data_compressed":9765423,"data_compressed_allocated":12320768,"data_compressed_original":70340608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":22110208,"data_stored":76551778,"data_compressed":9536145,"data_compressed_allocated":12132352,"data_compressed_original":66600960,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":34029568,"data_stored":115997885,"data_compressed":15381027,"data_compressed_allocated":19976192,"data_compressed_original":101978112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10801152,"data_stored":15380743,"data_compressed":4817064,"data_compressed_allocated":5283840,"data_compressed_original":11505664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6225920,"data_stored":9866351,"data_compressed":3958814,"data_compressed_allocated":4292608,"data_compressed_original":8957952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4136960,"data_stored":5382491,"data_compressed":2066185,"data_compressed_allocated":2256896,"data_compressed_original":4472832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11411456,"data_stored":14056169,"data_compressed":3633800,"data_compressed_allocated":4009984,"data_compressed_original":8400896,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9719808,"data_stored":12781845,"data_compressed":3982058,"data_compressed_allocated":4333568,"data_compressed_original":9048064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8282112,"data_stored":12361856,"data_compressed":4520793,"data_compressed_allocated":4976640,"data_compressed_original":10809344,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13295616,"data_stored":14774517,"data_compressed":2414125,"data_compressed_allocated":2658304,"data_compressed_original":5361664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12959744,"data_stored":17250982,"data_compressed":5115177,"data_compressed_allocated":5529600,"data_compressed_original":11526144,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T19:30:10.408 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:30:10.409 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:30:10.409 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:30:10.410 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:30:10.411 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:30:10.411 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:30:10.412 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-10T18:45:10.384742+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=18, tm_min=45, tm_sec=10, tm_wday=1, tm_yday=283, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=10, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=1, tm_yday=283, tm_isdst=0) 2023-10-10T19:30:10.412 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-10T19:30:27.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:30:27.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:30:30.413 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T19:30:30.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92f5107700 1 Processor -- start 2023-10-10T19:30:30.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92f5107700 1 -- start start 2023-10-10T19:30:30.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92f5107700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f92f014fbf0 0x7f92f0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92f5107700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f92f014ffc0 0x7f92f0063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92f5107700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0156800 con 0x7f92f00681e0 2023-10-10T19:30:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92f5107700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f00c59e0 con 0x7f92f014ffc0 2023-10-10T19:30:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92f5107700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f015a2b0 con 0x7f92f014fbf0 2023-10-10T19:30:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92e659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f92f014ffc0 0x7f92f0063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.540+0000 7f92e659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f92f014ffc0 0x7f92f0063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60106/0 (socket says 172.21.15.175:60106) 2023-10-10T19:30:30.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92e659c700 1 -- 172.21.15.175:0/2357174136 learned_addr learned my addr 172.21.15.175:0/2357174136 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:30.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92eed9d700 1 --2- 172.21.15.175:0/2357174136 >> v2:172.21.15.191:3300/0 conn(0x7f92f014fbf0 0x7f92f0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92e659c700 1 -- 172.21.15.175:0/2357174136 >> v2:172.21.15.191:3300/0 conn(0x7f92f014fbf0 msgr2=0x7f92f0159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:30.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92e659c700 1 --2- 172.21.15.175:0/2357174136 >> v2:172.21.15.191:3300/0 conn(0x7f92f014fbf0 0x7f92f0159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92e659c700 1 -- 172.21.15.175:0/2357174136 >> v1:172.21.15.191:6789/0 conn(0x7f92f00681e0 legacy=0x7f92f00c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:30.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92e659c700 1 -- 172.21.15.175:0/2357174136 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f0063e30 con 0x7f92f014ffc0 2023-10-10T19:30:30.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92e659c700 1 --2- 172.21.15.175:0/2357174136 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f014ffc0 0x7f92f0063230 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f92f00cc6c0 tx=0x7f92d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f41686c0158db86 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:30.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92ee59c700 1 -- 172.21.15.175:0/2357174136 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d800b820 con 0x7f92f014ffc0 2023-10-10T19:30:30.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92ee59c700 1 -- 172.21.15.175:0/2357174136 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d800b980 con 0x7f92f014ffc0 2023-10-10T19:30:30.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92ee59c700 1 -- 172.21.15.175:0/2357174136 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d800bd20 con 0x7f92f014ffc0 2023-10-10T19:30:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92f5107700 1 -- 172.21.15.175:0/2357174136 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f014ffc0 msgr2=0x7f92f0063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.541+0000 7f92f5107700 1 --2- 172.21.15.175:0/2357174136 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f014ffc0 0x7f92f0063230 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f92f00cc6c0 tx=0x7f92d8009250 comp rx=0 tx=0).stop 2023-10-10T19:30:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.542+0000 7f92f5107700 1 -- 172.21.15.175:0/2357174136 shutdown_connections 2023-10-10T19:30:30.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.542+0000 7f92f5107700 1 --2- 172.21.15.175:0/2357174136 >> v2:172.21.15.191:3300/0 conn(0x7f92f014fbf0 0x7f92f0159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.542+0000 7f92f5107700 1 --2- 172.21.15.175:0/2357174136 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f014ffc0 0x7f92f0063230 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.542+0000 7f92f5107700 1 -- 172.21.15.175:0/2357174136 >> 172.21.15.175:0/2357174136 conn(0x7f92f00cc760 msgr2=0x7f92f00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.542+0000 7f92f5107700 1 -- 172.21.15.175:0/2357174136 shutdown_connections 2023-10-10T19:30:30.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.542+0000 7f92f5107700 1 -- 172.21.15.175:0/2357174136 wait complete. 2023-10-10T19:30:30.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.543+0000 7f92f5107700 1 Processor -- start 2023-10-10T19:30:30.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.543+0000 7f92f5107700 1 -- start start 2023-10-10T19:30:30.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.543+0000 7f92f5107700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f00681e0 0x7f92f0169630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.543+0000 7f92f5107700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92f014fbf0 0x7f92f014ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.543+0000 7f92f5107700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92f014ffc0 0x7f92f014bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92f5107700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00a80a0 con 0x7f92f00681e0 2023-10-10T19:30:30.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92f5107700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00c2db0 con 0x7f92f014fbf0 2023-10-10T19:30:30.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92f5107700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f0152220 con 0x7f92f014ffc0 2023-10-10T19:30:30.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f00681e0 0x7f92f0169630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f00681e0 0x7f92f0169630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60116/0 (socket says 172.21.15.175:60116) 2023-10-10T19:30:30.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 -- 172.21.15.175:0/4275357404 learned_addr learned my addr 172.21.15.175:0/4275357404 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:30.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 -- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92f014ffc0 msgr2=0x7f92f014bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92f014ffc0 0x7f92f014bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 -- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92f014fbf0 msgr2=0x7f92f014ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92e659c700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92f014fbf0 0x7f92f014ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92f014fbf0 0x7f92f014ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 -- 172.21.15.175:0/4275357404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d80094f0 con 0x7f92f00681e0 2023-10-10T19:30:30.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92eed9d700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f00681e0 0x7f92f0169630 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f92e000d810 tx=0x7f92e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:30.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92e77fe700 1 -- 172.21.15.175:0/4275357404 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e000b630 con 0x7f92f00681e0 2023-10-10T19:30:30.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92e77fe700 1 -- 172.21.15.175:0/4275357404 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e000b790 con 0x7f92f00681e0 2023-10-10T19:30:30.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f00c3580 con 0x7f92f00681e0 2023-10-10T19:30:30.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92e77fe700 1 -- 172.21.15.175:0/4275357404 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e000bb50 con 0x7f92f00681e0 2023-10-10T19:30:30.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.544+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f016a140 con 0x7f92f00681e0 2023-10-10T19:30:30.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.545+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f0067570 con 0x7f92f00681e0 2023-10-10T19:30:30.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.546+0000 7f92e77fe700 1 -- 172.21.15.175:0/4275357404 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f92e001e460 con 0x7f92f00681e0 2023-10-10T19:30:30.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.546+0000 7f92e77fe700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92d0066f30 0x7f92d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.546+0000 7f92e77fe700 1 -- 172.21.15.175:0/4275357404 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f92e0085540 con 0x7f92f00681e0 2023-10-10T19:30:30.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.546+0000 7f92e659c700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92d0066f30 0x7f92d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.547+0000 7f92e659c700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92d0066f30 0x7f92d00693e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f92f00cc6c0 tx=0x7f92d8017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:30.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.551+0000 7f92e77fe700 1 -- 172.21.15.175:0/4275357404 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f92e0051f30 con 0x7f92f00681e0 2023-10-10T19:30:30.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.729+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 --> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92f0152090 con 0x7f92d0066f30 2023-10-10T19:30:30.737 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.737+0000 7f92e77fe700 1 -- 172.21.15.175:0/4275357404 <== mgr.4103 v2:172.21.15.191:6832/90896 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+297212 (secure 0 0 0) 0x7f92f0152090 con 0x7f92d0066f30 2023-10-10T19:30:30.742 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:30:30.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92d0066f30 msgr2=0x7f92d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.744 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92d0066f30 0x7f92d00693e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f92f00cc6c0 tx=0x7f92d8017000 comp rx=0 tx=0).stop 2023-10-10T19:30:30.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f00681e0 msgr2=0x7f92f0169630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f00681e0 0x7f92f0169630 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f92e000d810 tx=0x7f92e0008040 comp rx=0 tx=0).stop 2023-10-10T19:30:30.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 shutdown_connections 2023-10-10T19:30:30.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f92d0066f30 0x7f92d00693e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92f014ffc0 0x7f92f014bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f92f00681e0 0x7f92f0169630 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 --2- 172.21.15.175:0/4275357404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92f014fbf0 0x7f92f014ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.740+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 >> 172.21.15.175:0/4275357404 conn(0x7f92f00cc760 msgr2=0x7f92f0155370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:30.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.741+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 shutdown_connections 2023-10-10T19:30:30.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.741+0000 7f92f5107700 1 -- 172.21.15.175:0/4275357404 wait complete. 2023-10-10T19:30:30.749 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-10-10T19:30:30.753 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":1488,"stamp":"2023-10-10T19:30:29.834996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388402487,"num_objects":2861,"num_object_clones":0,"num_object_copies":5722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":7382,"num_read_kb":3484532,"num_write":63416,"num_write_kb":2990256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":353,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2518057,"num_omap_keys":4386,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83256,"ondisk_log_size":83256,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2698072,"kb_used_data":266520,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752276648,"statfs":{"total":773094113280,"available":770331287552,"internally_reserved":0,"allocated":272916480,"data_stored":783892462,"data_compressed":117849180,"data_compressed_allocated":145776640,"data_compressed_original":674291712,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"3.3f","version":"132'1007","reported_seq":3639,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:15.190221+0000","last_change":"2023-10-10T19:30:15.190182+0000","last_active":"2023-10-10T19:30:15.190221+0000","last_peered":"2023-10-10T19:30:15.190221+0000","last_clean":"2023-10-10T19:30:15.190221+0000","last_became_active":"2023-10-10T18:45:11.418992+0000","last_became_peered":"2023-10-10T18:45:11.418992+0000","last_unstale":"2023-10-10T19:30:15.190221+0000","last_undegraded":"2023-10-10T19:30:15.190221+0000","last_fullsized":"2023-10-10T19:30:15.190221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1007","last_scrub_stamp":"2023-10-10T19:30:15.190062+0000","last_deep_scrub":"132'1007","last_deep_scrub_stamp":"2023-10-10T19:30:15.190062+0000","last_clean_scrub_stamp":"2023-10-10T19:30:15.190062+0000","objects_scrubbed":41,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:59:25.932362+0000","scrub_duration":0.039822509999999998,"objects_trimmed":0,"snaptrim_duration":0.017995507000000001,"stat_sum":{"num_bytes":821890,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":62,"num_read_kb":8068,"num_write":683,"num_write_kb":16759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"132'1048","reported_seq":3679,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:54.681549+0000","last_change":"2023-10-10T19:29:54.681522+0000","last_active":"2023-10-10T19:29:54.681549+0000","last_peered":"2023-10-10T19:29:54.681549+0000","last_clean":"2023-10-10T19:29:54.681549+0000","last_became_active":"2023-10-10T18:45:11.414295+0000","last_became_peered":"2023-10-10T18:45:11.414295+0000","last_unstale":"2023-10-10T19:29:54.681549+0000","last_undegraded":"2023-10-10T19:29:54.681549+0000","last_fullsized":"2023-10-10T19:29:54.681549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1048","last_scrub_stamp":"2023-10-10T19:29:54.681440+0000","last_deep_scrub":"132'1048","last_deep_scrub_stamp":"2023-10-10T19:29:54.681440+0000","last_clean_scrub_stamp":"2023-10-10T19:29:54.681440+0000","objects_scrubbed":36,"log_size":1048,"log_dups_size":0,"ondisk_log_size":1048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:09.327427+0000","scrub_duration":0.04117941,"objects_trimmed":0,"snaptrim_duration":0.0083569260000000006,"stat_sum":{"num_bytes":1412883,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":62,"num_read_kb":28805,"num_write":746,"num_write_kb":46678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"132'1002","reported_seq":3641,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:59.385738+0000","last_change":"2023-10-10T19:29:59.385701+0000","last_active":"2023-10-10T19:29:59.385738+0000","last_peered":"2023-10-10T19:29:59.385738+0000","last_clean":"2023-10-10T19:29:59.385738+0000","last_became_active":"2023-10-10T18:45:11.418856+0000","last_became_peered":"2023-10-10T18:45:11.418856+0000","last_unstale":"2023-10-10T19:29:59.385738+0000","last_undegraded":"2023-10-10T19:29:59.385738+0000","last_fullsized":"2023-10-10T19:29:59.385738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1002","last_scrub_stamp":"2023-10-10T19:29:59.385573+0000","last_deep_scrub":"132'1002","last_deep_scrub_stamp":"2023-10-10T19:29:59.385573+0000","last_clean_scrub_stamp":"2023-10-10T19:29:59.385573+0000","objects_scrubbed":40,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:26:17.112199+0000","scrub_duration":0.043775355000000002,"objects_trimmed":0,"snaptrim_duration":0.011353221,"stat_sum":{"num_bytes":671969,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":65,"num_read_kb":20680,"num_write":694,"num_write_kb":24810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"132'999","reported_seq":3625,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:01.613628+0000","last_change":"2023-10-10T19:30:01.613597+0000","last_active":"2023-10-10T19:30:01.613628+0000","last_peered":"2023-10-10T19:30:01.613628+0000","last_clean":"2023-10-10T19:30:01.613628+0000","last_became_active":"2023-10-10T18:45:11.421028+0000","last_became_peered":"2023-10-10T18:45:11.421028+0000","last_unstale":"2023-10-10T19:30:01.613628+0000","last_undegraded":"2023-10-10T19:30:01.613628+0000","last_fullsized":"2023-10-10T19:30:01.613628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'999","last_scrub_stamp":"2023-10-10T19:30:01.613517+0000","last_deep_scrub":"132'999","last_deep_scrub_stamp":"2023-10-10T19:30:01.613517+0000","last_clean_scrub_stamp":"2023-10-10T19:30:01.613517+0000","objects_scrubbed":38,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:00:14.282852+0000","scrub_duration":0.032420306000000003,"objects_trimmed":0,"snaptrim_duration":0.0088633549999999998,"stat_sum":{"num_bytes":651243,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":62,"num_read_kb":30965,"num_write":672,"num_write_kb":37710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"132'1023","reported_seq":3665,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:09.127081+0000","last_change":"2023-10-10T19:30:09.127044+0000","last_active":"2023-10-10T19:30:09.127081+0000","last_peered":"2023-10-10T19:30:09.127081+0000","last_clean":"2023-10-10T19:30:09.127081+0000","last_became_active":"2023-10-10T18:45:11.420800+0000","last_became_peered":"2023-10-10T18:45:11.420800+0000","last_unstale":"2023-10-10T19:30:09.127081+0000","last_undegraded":"2023-10-10T19:30:09.127081+0000","last_fullsized":"2023-10-10T19:30:09.127081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1023","last_scrub_stamp":"2023-10-10T19:30:09.126898+0000","last_deep_scrub":"132'1023","last_deep_scrub_stamp":"2023-10-10T19:30:09.126898+0000","last_clean_scrub_stamp":"2023-10-10T19:30:09.126898+0000","objects_scrubbed":36,"log_size":1023,"log_dups_size":0,"ondisk_log_size":1023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:44:47.815547+0000","scrub_duration":0.031728982000000003,"objects_trimmed":0,"snaptrim_duration":0.016189688000000001,"stat_sum":{"num_bytes":219054,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":75,"num_read_kb":17848,"num_write":702,"num_write_kb":31364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"132'979","reported_seq":3617,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:22.632836+0000","last_change":"2023-10-10T19:30:22.632810+0000","last_active":"2023-10-10T19:30:22.632836+0000","last_peered":"2023-10-10T19:30:22.632836+0000","last_clean":"2023-10-10T19:30:22.632836+0000","last_became_active":"2023-10-10T18:45:11.421188+0000","last_became_peered":"2023-10-10T18:45:11.421188+0000","last_unstale":"2023-10-10T19:30:22.632836+0000","last_undegraded":"2023-10-10T19:30:22.632836+0000","last_fullsized":"2023-10-10T19:30:22.632836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'979","last_scrub_stamp":"2023-10-10T19:30:22.632733+0000","last_deep_scrub":"132'979","last_deep_scrub_stamp":"2023-10-10T19:30:22.632733+0000","last_clean_scrub_stamp":"2023-10-10T19:30:22.632733+0000","objects_scrubbed":37,"log_size":979,"log_dups_size":0,"ondisk_log_size":979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:45:43.982558+0000","scrub_duration":0.035152910000000002,"objects_trimmed":0,"snaptrim_duration":0.010631753000000001,"stat_sum":{"num_bytes":2108369,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":63,"num_read_kb":20939,"num_write":675,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"132'1026","reported_seq":3659,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:04.095433+0000","last_change":"2023-10-10T19:30:04.095408+0000","last_active":"2023-10-10T19:30:04.095433+0000","last_peered":"2023-10-10T19:30:04.095433+0000","last_clean":"2023-10-10T19:30:04.095433+0000","last_became_active":"2023-10-10T18:45:11.419089+0000","last_became_peered":"2023-10-10T18:45:11.419089+0000","last_unstale":"2023-10-10T19:30:04.095433+0000","last_undegraded":"2023-10-10T19:30:04.095433+0000","last_fullsized":"2023-10-10T19:30:04.095433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1026","last_scrub_stamp":"2023-10-10T19:30:04.095327+0000","last_deep_scrub":"132'1026","last_deep_scrub_stamp":"2023-10-10T19:30:04.095327+0000","last_clean_scrub_stamp":"2023-10-10T19:30:04.095327+0000","objects_scrubbed":39,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T07:12:01.117837+0000","scrub_duration":0.034862328999999997,"objects_trimmed":0,"snaptrim_duration":0.014667316999999999,"stat_sum":{"num_bytes":1119068,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":77,"num_read_kb":53600,"num_write":685,"num_write_kb":52880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"132'914","reported_seq":3564,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:09.010756+0000","last_change":"2023-10-10T19:30:09.010731+0000","last_active":"2023-10-10T19:30:09.010756+0000","last_peered":"2023-10-10T19:30:09.010756+0000","last_clean":"2023-10-10T19:30:09.010756+0000","last_became_active":"2023-10-10T18:45:11.422655+0000","last_became_peered":"2023-10-10T18:45:11.422655+0000","last_unstale":"2023-10-10T19:30:09.010756+0000","last_undegraded":"2023-10-10T19:30:09.010756+0000","last_fullsized":"2023-10-10T19:30:09.010756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'914","last_scrub_stamp":"2023-10-10T19:30:09.010653+0000","last_deep_scrub":"132'914","last_deep_scrub_stamp":"2023-10-10T19:30:09.010653+0000","last_clean_scrub_stamp":"2023-10-10T19:30:09.010653+0000","objects_scrubbed":34,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:27:20.274857+0000","scrub_duration":0.030184883999999999,"objects_trimmed":0,"snaptrim_duration":0.012402231,"stat_sum":{"num_bytes":484780,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":58,"num_read_kb":28110,"num_write":652,"num_write_kb":35721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"132'1006","reported_seq":3645,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:10.133379+0000","last_change":"2023-10-10T19:30:10.133353+0000","last_active":"2023-10-10T19:30:10.133379+0000","last_peered":"2023-10-10T19:30:10.133379+0000","last_clean":"2023-10-10T19:30:10.133379+0000","last_became_active":"2023-10-10T18:45:11.422352+0000","last_became_peered":"2023-10-10T18:45:11.422352+0000","last_unstale":"2023-10-10T19:30:10.133379+0000","last_undegraded":"2023-10-10T19:30:10.133379+0000","last_fullsized":"2023-10-10T19:30:10.133379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1006","last_scrub_stamp":"2023-10-10T19:30:10.133264+0000","last_deep_scrub":"132'1006","last_deep_scrub_stamp":"2023-10-10T19:30:10.133264+0000","last_clean_scrub_stamp":"2023-10-10T19:30:10.133264+0000","objects_scrubbed":30,"log_size":1006,"log_dups_size":0,"ondisk_log_size":1006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:46:47.971900+0000","scrub_duration":0.029861723999999999,"objects_trimmed":0,"snaptrim_duration":0.017699890999999999,"stat_sum":{"num_bytes":94035,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":76,"num_read_kb":37435,"num_write":702,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"132'993","reported_seq":3631,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:58.196363+0000","last_change":"2023-10-10T19:29:58.196336+0000","last_active":"2023-10-10T19:29:58.196363+0000","last_peered":"2023-10-10T19:29:58.196363+0000","last_clean":"2023-10-10T19:29:58.196363+0000","last_became_active":"2023-10-10T18:45:11.425469+0000","last_became_peered":"2023-10-10T18:45:11.425469+0000","last_unstale":"2023-10-10T19:29:58.196363+0000","last_undegraded":"2023-10-10T19:29:58.196363+0000","last_fullsized":"2023-10-10T19:29:58.196363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'993","last_scrub_stamp":"2023-10-10T19:29:58.196244+0000","last_deep_scrub":"132'993","last_deep_scrub_stamp":"2023-10-10T19:29:58.196244+0000","last_clean_scrub_stamp":"2023-10-10T19:29:58.196244+0000","objects_scrubbed":38,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:09:04.497993+0000","scrub_duration":0.034923373000000001,"objects_trimmed":0,"snaptrim_duration":0.0079196160000000008,"stat_sum":{"num_bytes":269975,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":70,"num_read_kb":16825,"num_write":697,"num_write_kb":23418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"132'1021","reported_seq":3682,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:52.090985+0000","last_change":"2023-10-10T19:29:52.090959+0000","last_active":"2023-10-10T19:29:52.090985+0000","last_peered":"2023-10-10T19:29:52.090985+0000","last_clean":"2023-10-10T19:29:52.090985+0000","last_became_active":"2023-10-10T18:45:11.425554+0000","last_became_peered":"2023-10-10T18:45:11.425554+0000","last_unstale":"2023-10-10T19:29:52.090985+0000","last_undegraded":"2023-10-10T19:29:52.090985+0000","last_fullsized":"2023-10-10T19:29:52.090985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1021","last_scrub_stamp":"2023-10-10T19:29:52.090872+0000","last_deep_scrub":"132'1021","last_deep_scrub_stamp":"2023-10-10T19:29:52.090872+0000","last_clean_scrub_stamp":"2023-10-10T19:29:52.090872+0000","objects_scrubbed":46,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:30:48.456323+0000","scrub_duration":0.039720943000000002,"objects_trimmed":0,"snaptrim_duration":0.0088422029999999999,"stat_sum":{"num_bytes":213661,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":82,"num_read_kb":46829,"num_write":712,"num_write_kb":53066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"132'994","reported_seq":3641,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.048201+0000","last_change":"2023-10-10T19:29:48.048165+0000","last_active":"2023-10-10T19:29:48.048201+0000","last_peered":"2023-10-10T19:29:48.048201+0000","last_clean":"2023-10-10T19:29:48.048201+0000","last_became_active":"2023-10-10T18:45:11.424462+0000","last_became_peered":"2023-10-10T18:45:11.424462+0000","last_unstale":"2023-10-10T19:29:48.048201+0000","last_undegraded":"2023-10-10T19:29:48.048201+0000","last_fullsized":"2023-10-10T19:29:48.048201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'994","last_scrub_stamp":"2023-10-10T19:29:48.048044+0000","last_deep_scrub":"132'994","last_deep_scrub_stamp":"2023-10-10T19:29:48.048044+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.048044+0000","objects_scrubbed":40,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:22:32.633316+0000","scrub_duration":0.033909425999999999,"objects_trimmed":0,"snaptrim_duration":0.0078589550000000008,"stat_sum":{"num_bytes":337999,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":75,"num_read_kb":57532,"num_write":710,"num_write_kb":73524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"132'983","reported_seq":3626,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:06.001308+0000","last_change":"2023-10-10T19:30:06.001271+0000","last_active":"2023-10-10T19:30:06.001308+0000","last_peered":"2023-10-10T19:30:06.001308+0000","last_clean":"2023-10-10T19:30:06.001308+0000","last_became_active":"2023-10-10T18:45:11.417501+0000","last_became_peered":"2023-10-10T18:45:11.417501+0000","last_unstale":"2023-10-10T19:30:06.001308+0000","last_undegraded":"2023-10-10T19:30:06.001308+0000","last_fullsized":"2023-10-10T19:30:06.001308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'983","last_scrub_stamp":"2023-10-10T19:30:06.001152+0000","last_deep_scrub":"132'983","last_deep_scrub_stamp":"2023-10-10T19:30:06.001152+0000","last_clean_scrub_stamp":"2023-10-10T19:30:06.001152+0000","objects_scrubbed":54,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:56:00.208525+0000","scrub_duration":0.059218532999999997,"objects_trimmed":0,"snaptrim_duration":0.014204069,"stat_sum":{"num_bytes":4832489,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":73,"num_read_kb":32208,"num_write":656,"num_write_kb":44878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"132'956","reported_seq":3592,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:51.283187+0000","last_change":"2023-10-10T19:29:51.283150+0000","last_active":"2023-10-10T19:29:51.283187+0000","last_peered":"2023-10-10T19:29:51.283187+0000","last_clean":"2023-10-10T19:29:51.283187+0000","last_became_active":"2023-10-10T18:45:11.414498+0000","last_became_peered":"2023-10-10T18:45:11.414498+0000","last_unstale":"2023-10-10T19:29:51.283187+0000","last_undegraded":"2023-10-10T19:29:51.283187+0000","last_fullsized":"2023-10-10T19:29:51.283187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'956","last_scrub_stamp":"2023-10-10T19:29:51.283026+0000","last_deep_scrub":"132'956","last_deep_scrub_stamp":"2023-10-10T19:29:51.283026+0000","last_clean_scrub_stamp":"2023-10-10T19:29:51.283026+0000","objects_scrubbed":27,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:14:52.527044+0000","scrub_duration":0.027467595000000001,"objects_trimmed":0,"snaptrim_duration":0.0097965510000000006,"stat_sum":{"num_bytes":724304,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":63,"num_read_kb":28420,"num_write":663,"num_write_kb":31175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"132'991","reported_seq":3636,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.691055+0000","last_change":"2023-10-10T19:29:57.691018+0000","last_active":"2023-10-10T19:29:57.691055+0000","last_peered":"2023-10-10T19:29:57.691055+0000","last_clean":"2023-10-10T19:29:57.691055+0000","last_became_active":"2023-10-10T18:45:11.429194+0000","last_became_peered":"2023-10-10T18:45:11.429194+0000","last_unstale":"2023-10-10T19:29:57.691055+0000","last_undegraded":"2023-10-10T19:29:57.691055+0000","last_fullsized":"2023-10-10T19:29:57.691055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'991","last_scrub_stamp":"2023-10-10T19:29:57.690903+0000","last_deep_scrub":"132'991","last_deep_scrub_stamp":"2023-10-10T19:29:57.690903+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.690903+0000","objects_scrubbed":36,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:41:37.142029+0000","scrub_duration":0.034147700000000003,"objects_trimmed":0,"snaptrim_duration":0.0059774370000000004,"stat_sum":{"num_bytes":1353080,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":77,"num_read_kb":52747,"num_write":668,"num_write_kb":73053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"132'1032","reported_seq":3678,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:10.542221+0000","last_change":"2023-10-10T19:30:10.542196+0000","last_active":"2023-10-10T19:30:10.542221+0000","last_peered":"2023-10-10T19:30:10.542221+0000","last_clean":"2023-10-10T19:30:10.542221+0000","last_became_active":"2023-10-10T18:45:11.422646+0000","last_became_peered":"2023-10-10T18:45:11.422646+0000","last_unstale":"2023-10-10T19:30:10.542221+0000","last_undegraded":"2023-10-10T19:30:10.542221+0000","last_fullsized":"2023-10-10T19:30:10.542221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1032","last_scrub_stamp":"2023-10-10T19:30:10.542117+0000","last_deep_scrub":"132'1032","last_deep_scrub_stamp":"2023-10-10T19:30:10.542117+0000","last_clean_scrub_stamp":"2023-10-10T19:30:10.542117+0000","objects_scrubbed":34,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:18:00.691672+0000","scrub_duration":0.028262217999999999,"objects_trimmed":0,"snaptrim_duration":0.0087668889999999999,"stat_sum":{"num_bytes":582240,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":80,"num_read_kb":42995,"num_write":709,"num_write_kb":53070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"132'953","reported_seq":3619,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:04.618514+0000","last_change":"2023-10-10T19:30:04.618486+0000","last_active":"2023-10-10T19:30:04.618514+0000","last_peered":"2023-10-10T19:30:04.618514+0000","last_clean":"2023-10-10T19:30:04.618514+0000","last_became_active":"2023-10-10T18:45:11.430468+0000","last_became_peered":"2023-10-10T18:45:11.430468+0000","last_unstale":"2023-10-10T19:30:04.618514+0000","last_undegraded":"2023-10-10T19:30:04.618514+0000","last_fullsized":"2023-10-10T19:30:04.618514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'953","last_scrub_stamp":"2023-10-10T19:30:04.618402+0000","last_deep_scrub":"132'953","last_deep_scrub_stamp":"2023-10-10T19:30:04.618402+0000","last_clean_scrub_stamp":"2023-10-10T19:30:04.618402+0000","objects_scrubbed":48,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:28:28.649613+0000","scrub_duration":0.046332283000000002,"objects_trimmed":0,"snaptrim_duration":0.012085218,"stat_sum":{"num_bytes":1912035,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":72,"num_read_kb":40372,"num_write":683,"num_write_kb":48886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"132'871","reported_seq":3555,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.229972+0000","last_change":"2023-10-10T19:29:40.229945+0000","last_active":"2023-10-10T19:29:40.229972+0000","last_peered":"2023-10-10T19:29:40.229972+0000","last_clean":"2023-10-10T19:29:40.229972+0000","last_became_active":"2023-10-10T18:45:11.424621+0000","last_became_peered":"2023-10-10T18:45:11.424621+0000","last_unstale":"2023-10-10T19:29:40.229972+0000","last_undegraded":"2023-10-10T19:29:40.229972+0000","last_fullsized":"2023-10-10T19:29:40.229972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'871","last_scrub_stamp":"2023-10-10T19:29:40.229861+0000","last_deep_scrub":"132'871","last_deep_scrub_stamp":"2023-10-10T19:29:40.229861+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.229861+0000","objects_scrubbed":37,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:14:54.227939+0000","scrub_duration":0.028370732999999999,"objects_trimmed":0,"snaptrim_duration":0.00594963,"stat_sum":{"num_bytes":953364,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":72,"num_read_kb":44243,"num_write":664,"num_write_kb":60950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"132'1108","reported_seq":3717,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:07.082007+0000","last_change":"2023-10-10T19:30:07.081980+0000","last_active":"2023-10-10T19:30:07.082007+0000","last_peered":"2023-10-10T19:30:07.082007+0000","last_clean":"2023-10-10T19:30:07.082007+0000","last_became_active":"2023-10-10T18:45:11.420791+0000","last_became_peered":"2023-10-10T18:45:11.420791+0000","last_unstale":"2023-10-10T19:30:07.082007+0000","last_undegraded":"2023-10-10T19:30:07.082007+0000","last_fullsized":"2023-10-10T19:30:07.082007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1108","last_scrub_stamp":"2023-10-10T19:30:07.081901+0000","last_deep_scrub":"132'1108","last_deep_scrub_stamp":"2023-10-10T19:30:07.081901+0000","last_clean_scrub_stamp":"2023-10-10T19:30:07.081901+0000","objects_scrubbed":35,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:40:27.067627+0000","scrub_duration":0.032335704999999999,"objects_trimmed":0,"snaptrim_duration":0.01601,"stat_sum":{"num_bytes":689975,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":73,"num_read_kb":41187,"num_write":728,"num_write_kb":49784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"132'855","reported_seq":1031,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.287167+0000","last_change":"2023-10-10T19:29:28.287143+0000","last_active":"2023-10-10T19:29:28.287167+0000","last_peered":"2023-10-10T19:29:28.287167+0000","last_clean":"2023-10-10T19:29:28.287167+0000","last_became_active":"2023-10-10T18:45:10.413360+0000","last_became_peered":"2023-10-10T18:45:10.413360+0000","last_unstale":"2023-10-10T19:29:28.287167+0000","last_undegraded":"2023-10-10T19:29:28.287167+0000","last_fullsized":"2023-10-10T19:29:28.287167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'855","last_scrub_stamp":"2023-10-10T19:29:28.287071+0000","last_deep_scrub":"132'855","last_deep_scrub_stamp":"2023-10-10T19:29:28.287071+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.287071+0000","objects_scrubbed":19,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:01.117300+0000","scrub_duration":0.031438141000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389252,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":82,"num_read_kb":61516,"num_write":781,"num_write_kb":21019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":54468,"num_omap_keys":96,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"132'1045","reported_seq":3678,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:59.036152+0000","last_change":"2023-10-10T19:29:59.036124+0000","last_active":"2023-10-10T19:29:59.036152+0000","last_peered":"2023-10-10T19:29:59.036152+0000","last_clean":"2023-10-10T19:29:59.036152+0000","last_became_active":"2023-10-10T18:45:11.417195+0000","last_became_peered":"2023-10-10T18:45:11.417195+0000","last_unstale":"2023-10-10T19:29:59.036152+0000","last_undegraded":"2023-10-10T19:29:59.036152+0000","last_fullsized":"2023-10-10T19:29:59.036152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1045","last_scrub_stamp":"2023-10-10T19:29:59.036034+0000","last_deep_scrub":"132'1045","last_deep_scrub_stamp":"2023-10-10T19:29:59.036034+0000","last_clean_scrub_stamp":"2023-10-10T19:29:59.036034+0000","objects_scrubbed":47,"log_size":1045,"log_dups_size":0,"ondisk_log_size":1045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:49:51.599183+0000","scrub_duration":0.043332374,"objects_trimmed":0,"snaptrim_duration":0.018086130999999998,"stat_sum":{"num_bytes":136255,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":63,"num_read_kb":13169,"num_write":738,"num_write_kb":31183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"132'776","reported_seq":931,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:32.981768+0000","last_change":"2023-10-10T19:29:32.981742+0000","last_active":"2023-10-10T19:29:32.981768+0000","last_peered":"2023-10-10T19:29:32.981768+0000","last_clean":"2023-10-10T19:29:32.981768+0000","last_became_active":"2023-10-10T18:45:10.408310+0000","last_became_peered":"2023-10-10T18:45:10.408310+0000","last_unstale":"2023-10-10T19:29:32.981768+0000","last_undegraded":"2023-10-10T19:29:32.981768+0000","last_fullsized":"2023-10-10T19:29:32.981768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'776","last_scrub_stamp":"2023-10-10T19:29:32.981652+0000","last_deep_scrub":"132'776","last_deep_scrub_stamp":"2023-10-10T19:29:32.981652+0000","last_clean_scrub_stamp":"2023-10-10T19:29:32.981652+0000","objects_scrubbed":16,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:01:01.872125+0000","scrub_duration":0.042698237999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":29,"num_read_kb":69691,"num_write":711,"num_write_kb":16940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":122430,"num_omap_keys":218,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"132'1012","reported_seq":3666,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.277911+0000","last_change":"2023-10-10T19:29:47.277885+0000","last_active":"2023-10-10T19:29:47.277911+0000","last_peered":"2023-10-10T19:29:47.277911+0000","last_clean":"2023-10-10T19:29:47.277911+0000","last_became_active":"2023-10-10T18:45:11.421574+0000","last_became_peered":"2023-10-10T18:45:11.421574+0000","last_unstale":"2023-10-10T19:29:47.277911+0000","last_undegraded":"2023-10-10T19:29:47.277911+0000","last_fullsized":"2023-10-10T19:29:47.277911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1012","last_scrub_stamp":"2023-10-10T19:29:47.277804+0000","last_deep_scrub":"132'1012","last_deep_scrub_stamp":"2023-10-10T19:29:47.277804+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.277804+0000","objects_scrubbed":36,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:05.362050+0000","scrub_duration":0.026281229,"objects_trimmed":0,"snaptrim_duration":0.0093729599999999996,"stat_sum":{"num_bytes":224018,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":74,"num_read_kb":34573,"num_write":731,"num_write_kb":44057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"132'1717","reported_seq":1904,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:54.199844+0000","last_change":"2023-10-10T19:29:54.199820+0000","last_active":"2023-10-10T19:29:54.199844+0000","last_peered":"2023-10-10T19:29:54.199844+0000","last_clean":"2023-10-10T19:29:54.199844+0000","last_became_active":"2023-10-10T18:45:10.411728+0000","last_became_peered":"2023-10-10T18:45:10.411728+0000","last_unstale":"2023-10-10T19:29:54.199844+0000","last_undegraded":"2023-10-10T19:29:54.199844+0000","last_fullsized":"2023-10-10T19:29:54.199844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1717","last_scrub_stamp":"2023-10-10T19:29:54.199748+0000","last_deep_scrub":"132'1717","last_deep_scrub_stamp":"2023-10-10T19:29:54.199748+0000","last_clean_scrub_stamp":"2023-10-10T19:29:54.199748+0000","objects_scrubbed":23,"log_size":1717,"log_dups_size":0,"ondisk_log_size":1717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:36:21.023828+0000","scrub_duration":0.064385183999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":114,"num_read_kb":59511,"num_write":1905,"num_write_kb":22655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":97001,"num_omap_keys":173,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"132'971","reported_seq":3577,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.099971+0000","last_change":"2023-10-10T19:29:48.099945+0000","last_active":"2023-10-10T19:29:48.099971+0000","last_peered":"2023-10-10T19:29:48.099971+0000","last_clean":"2023-10-10T19:29:48.099971+0000","last_became_active":"2023-10-10T18:46:01.103860+0000","last_became_peered":"2023-10-10T18:46:01.103860+0000","last_unstale":"2023-10-10T19:29:48.099971+0000","last_undegraded":"2023-10-10T19:29:48.099971+0000","last_fullsized":"2023-10-10T19:29:48.099971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'971","last_scrub_stamp":"2023-10-10T19:29:48.099859+0000","last_deep_scrub":"132'971","last_deep_scrub_stamp":"2023-10-10T19:29:48.099859+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.099859+0000","objects_scrubbed":28,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:28:00.147441+0000","scrub_duration":0.027094722000000002,"objects_trimmed":0,"snaptrim_duration":0.0054668909999999998,"stat_sum":{"num_bytes":121718,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":48,"num_read_kb":295,"num_write":648,"num_write_kb":7505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"132'763","reported_seq":927,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.085034+0000","last_change":"2023-10-10T19:29:47.084994+0000","last_active":"2023-10-10T19:29:47.085034+0000","last_peered":"2023-10-10T19:29:47.085034+0000","last_clean":"2023-10-10T19:29:47.085034+0000","last_became_active":"2023-10-10T18:45:10.413576+0000","last_became_peered":"2023-10-10T18:45:10.413576+0000","last_unstale":"2023-10-10T19:29:47.085034+0000","last_undegraded":"2023-10-10T19:29:47.085034+0000","last_fullsized":"2023-10-10T19:29:47.085034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'763","last_scrub_stamp":"2023-10-10T19:29:47.084860+0000","last_deep_scrub":"132'763","last_deep_scrub_stamp":"2023-10-10T19:29:47.084860+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.084860+0000","objects_scrubbed":13,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:01.164246+0000","scrub_duration":0.026132728000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389162,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":55,"num_read_kb":36831,"num_write":683,"num_write_kb":16858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":72140,"num_omap_keys":127,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"132'1024","reported_seq":3666,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.614680+0000","last_change":"2023-10-10T19:29:57.614652+0000","last_active":"2023-10-10T19:29:57.614680+0000","last_peered":"2023-10-10T19:29:57.614680+0000","last_clean":"2023-10-10T19:29:57.614680+0000","last_became_active":"2023-10-10T18:45:11.429141+0000","last_became_peered":"2023-10-10T18:45:11.429141+0000","last_unstale":"2023-10-10T19:29:57.614680+0000","last_undegraded":"2023-10-10T19:29:57.614680+0000","last_fullsized":"2023-10-10T19:29:57.614680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1024","last_scrub_stamp":"2023-10-10T19:29:57.614566+0000","last_deep_scrub":"132'1024","last_deep_scrub_stamp":"2023-10-10T19:29:57.614566+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.614566+0000","objects_scrubbed":33,"log_size":1024,"log_dups_size":0,"ondisk_log_size":1024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:38:28.460701+0000","scrub_duration":0.038641201999999999,"objects_trimmed":0,"snaptrim_duration":0.011753095,"stat_sum":{"num_bytes":171234,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":81,"num_read_kb":35882,"num_write":699,"num_write_kb":43024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"132'652","reported_seq":811,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.206093+0000","last_change":"2023-10-10T19:29:29.999770+0000","last_active":"2023-10-10T19:29:30.206093+0000","last_peered":"2023-10-10T19:29:30.206093+0000","last_clean":"2023-10-10T19:29:30.206093+0000","last_became_active":"2023-10-10T18:45:10.412116+0000","last_became_peered":"2023-10-10T18:45:10.412116+0000","last_unstale":"2023-10-10T19:29:30.206093+0000","last_undegraded":"2023-10-10T19:29:30.206093+0000","last_fullsized":"2023-10-10T19:29:30.206093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'626","last_scrub_stamp":"2023-10-10T19:29:29.999705+0000","last_deep_scrub":"132'626","last_deep_scrub_stamp":"2023-10-10T19:29:29.999705+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.999705+0000","objects_scrubbed":14,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T07:13:56.479723+0000","scrub_duration":0.034014667999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15333167,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":31,"num_read_kb":65583,"num_write":581,"num_write_kb":19518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":50025,"num_omap_keys":89,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"132'942","reported_seq":3582,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:33.015438+0000","last_change":"2023-10-10T19:29:33.015412+0000","last_active":"2023-10-10T19:29:33.015438+0000","last_peered":"2023-10-10T19:29:33.015438+0000","last_clean":"2023-10-10T19:29:33.015438+0000","last_became_active":"2023-10-10T18:45:11.424753+0000","last_became_peered":"2023-10-10T18:45:11.424753+0000","last_unstale":"2023-10-10T19:29:33.015438+0000","last_undegraded":"2023-10-10T19:29:33.015438+0000","last_fullsized":"2023-10-10T19:29:33.015438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'942","last_scrub_stamp":"2023-10-10T19:29:33.015326+0000","last_deep_scrub":"132'942","last_deep_scrub_stamp":"2023-10-10T19:29:33.015326+0000","last_clean_scrub_stamp":"2023-10-10T19:29:33.015326+0000","objects_scrubbed":43,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:49:00.949756+0000","scrub_duration":0.032708036000000003,"objects_trimmed":0,"snaptrim_duration":0.0062644010000000002,"stat_sum":{"num_bytes":838983,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":59,"num_read_kb":21826,"num_write":689,"num_write_kb":37727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"132'791","reported_seq":1134,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.136520+0000","last_change":"2023-10-10T19:29:31.136493+0000","last_active":"2023-10-10T19:29:31.136520+0000","last_peered":"2023-10-10T19:29:31.136520+0000","last_clean":"2023-10-10T19:29:31.136520+0000","last_became_active":"2023-10-10T18:45:10.400379+0000","last_became_peered":"2023-10-10T18:45:10.400379+0000","last_unstale":"2023-10-10T19:29:31.136520+0000","last_undegraded":"2023-10-10T19:29:31.136520+0000","last_fullsized":"2023-10-10T19:29:31.136520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'791","last_scrub_stamp":"2023-10-10T19:29:31.136411+0000","last_deep_scrub":"132'791","last_deep_scrub_stamp":"2023-10-10T19:29:31.136411+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.136411+0000","objects_scrubbed":15,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:18:11.372189+0000","scrub_duration":0.042934018999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17111936,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":251,"num_read_kb":81147,"num_write":704,"num_write_kb":25397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35794,"num_omap_keys":65,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"132'857","reported_seq":3517,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.291480+0000","last_change":"2023-10-10T19:29:42.291455+0000","last_active":"2023-10-10T19:29:42.291480+0000","last_peered":"2023-10-10T19:29:42.291480+0000","last_clean":"2023-10-10T19:29:42.291480+0000","last_became_active":"2023-10-10T18:45:11.424833+0000","last_became_peered":"2023-10-10T18:45:11.424833+0000","last_unstale":"2023-10-10T19:29:42.291480+0000","last_undegraded":"2023-10-10T19:29:42.291480+0000","last_fullsized":"2023-10-10T19:29:42.291480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'857","last_scrub_stamp":"2023-10-10T19:29:42.291374+0000","last_deep_scrub":"132'857","last_deep_scrub_stamp":"2023-10-10T19:29:42.291374+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.291374+0000","objects_scrubbed":27,"log_size":857,"log_dups_size":0,"ondisk_log_size":857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:23:42.700538+0000","scrub_duration":0.018625359000000001,"objects_trimmed":0,"snaptrim_duration":0.015269961,"stat_sum":{"num_bytes":473837,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":60,"num_read_kb":24046,"num_write":615,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"132'1506","reported_seq":1912,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.017721+0000","last_change":"2023-10-10T19:29:28.017678+0000","last_active":"2023-10-10T19:29:28.017721+0000","last_peered":"2023-10-10T19:29:28.017721+0000","last_clean":"2023-10-10T19:29:28.017721+0000","last_became_active":"2023-10-10T18:45:10.406858+0000","last_became_peered":"2023-10-10T18:45:10.406858+0000","last_unstale":"2023-10-10T19:29:28.017721+0000","last_undegraded":"2023-10-10T19:29:28.017721+0000","last_fullsized":"2023-10-10T19:29:28.017721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1506","last_scrub_stamp":"2023-10-10T19:29:28.017594+0000","last_deep_scrub":"132'1506","last_deep_scrub_stamp":"2023-10-10T19:29:28.017594+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.017594+0000","objects_scrubbed":12,"log_size":1506,"log_dups_size":0,"ondisk_log_size":1506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:59:47.232197+0000","scrub_duration":0.020028464999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4727705,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":295,"num_read_kb":15405,"num_write":1447,"num_write_kb":10055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66656,"num_omap_keys":116,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"132'1043","reported_seq":3698,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:54.022953+0000","last_change":"2023-10-10T19:29:54.022924+0000","last_active":"2023-10-10T19:29:54.022953+0000","last_peered":"2023-10-10T19:29:54.022953+0000","last_clean":"2023-10-10T19:29:54.022953+0000","last_became_active":"2023-10-10T18:45:11.412365+0000","last_became_peered":"2023-10-10T18:45:11.412365+0000","last_unstale":"2023-10-10T19:29:54.022953+0000","last_undegraded":"2023-10-10T19:29:54.022953+0000","last_fullsized":"2023-10-10T19:29:54.022953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1043","last_scrub_stamp":"2023-10-10T19:29:54.022842+0000","last_deep_scrub":"132'1043","last_deep_scrub_stamp":"2023-10-10T19:29:54.022842+0000","last_clean_scrub_stamp":"2023-10-10T19:29:54.022842+0000","objects_scrubbed":41,"log_size":1043,"log_dups_size":0,"ondisk_log_size":1043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:56:40.926906+0000","scrub_duration":0.038501266999999999,"objects_trimmed":0,"snaptrim_duration":0.015767953000000001,"stat_sum":{"num_bytes":671315,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":81,"num_read_kb":19092,"num_write":747,"num_write_kb":30378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"132'506","reported_seq":662,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.758375+0000","last_change":"2023-10-10T19:29:40.758347+0000","last_active":"2023-10-10T19:29:40.758375+0000","last_peered":"2023-10-10T19:29:40.758375+0000","last_clean":"2023-10-10T19:29:40.758375+0000","last_became_active":"2023-10-10T18:45:10.406359+0000","last_became_peered":"2023-10-10T18:45:10.406359+0000","last_unstale":"2023-10-10T19:29:40.758375+0000","last_undegraded":"2023-10-10T19:29:40.758375+0000","last_fullsized":"2023-10-10T19:29:40.758375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'506","last_scrub_stamp":"2023-10-10T19:29:40.758259+0000","last_deep_scrub":"132'506","last_deep_scrub_stamp":"2023-10-10T19:29:40.758259+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.758259+0000","objects_scrubbed":12,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:28:13.803240+0000","scrub_duration":0.033275374000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583466,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":65566,"num_write":451,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":76745,"num_omap_keys":113,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"132'1056","reported_seq":3707,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.597586+0000","last_change":"2023-10-10T19:29:45.597550+0000","last_active":"2023-10-10T19:29:45.597586+0000","last_peered":"2023-10-10T19:29:45.597586+0000","last_clean":"2023-10-10T19:29:45.597586+0000","last_became_active":"2023-10-10T18:46:00.606391+0000","last_became_peered":"2023-10-10T18:46:00.606391+0000","last_unstale":"2023-10-10T19:29:45.597586+0000","last_undegraded":"2023-10-10T19:29:45.597586+0000","last_fullsized":"2023-10-10T19:29:45.597586+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1056","last_scrub_stamp":"2023-10-10T19:29:45.597448+0000","last_deep_scrub":"132'1056","last_deep_scrub_stamp":"2023-10-10T19:29:45.597448+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.597448+0000","objects_scrubbed":41,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:39:53.993867+0000","scrub_duration":0.033388767999999999,"objects_trimmed":0,"snaptrim_duration":0.0066939039999999997,"stat_sum":{"num_bytes":730064,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":80,"num_read_kb":42231,"num_write":744,"num_write_kb":52869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"132'292","reported_seq":444,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.067574+0000","last_change":"2023-10-10T19:29:48.067525+0000","last_active":"2023-10-10T19:29:48.067574+0000","last_peered":"2023-10-10T19:29:48.067574+0000","last_clean":"2023-10-10T19:29:48.067574+0000","last_became_active":"2023-10-10T18:45:10.408368+0000","last_became_peered":"2023-10-10T18:45:10.408368+0000","last_unstale":"2023-10-10T19:29:48.067574+0000","last_undegraded":"2023-10-10T19:29:48.067574+0000","last_fullsized":"2023-10-10T19:29:48.067574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'291","last_scrub_stamp":"2023-10-10T19:29:48.067450+0000","last_deep_scrub":"132'291","last_deep_scrub_stamp":"2023-10-10T19:29:48.067450+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.067450+0000","objects_scrubbed":17,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:31:55.859793+0000","scrub_duration":0.010942558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":34,"num_read_kb":54,"num_write":234,"num_write_kb":4552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":87494,"num_omap_keys":156,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"132'988","reported_seq":3635,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:03.614708+0000","last_change":"2023-10-10T19:30:03.614679+0000","last_active":"2023-10-10T19:30:03.614708+0000","last_peered":"2023-10-10T19:30:03.614708+0000","last_clean":"2023-10-10T19:30:03.614708+0000","last_became_active":"2023-10-10T18:45:11.426677+0000","last_became_peered":"2023-10-10T18:45:11.426677+0000","last_unstale":"2023-10-10T19:30:03.614708+0000","last_undegraded":"2023-10-10T19:30:03.614708+0000","last_fullsized":"2023-10-10T19:30:03.614708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'988","last_scrub_stamp":"2023-10-10T19:30:03.614594+0000","last_deep_scrub":"132'988","last_deep_scrub_stamp":"2023-10-10T19:30:03.614594+0000","last_clean_scrub_stamp":"2023-10-10T19:30:03.614594+0000","objects_scrubbed":25,"log_size":988,"log_dups_size":0,"ondisk_log_size":988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:21:08.189640+0000","scrub_duration":0.025435242,"objects_trimmed":0,"snaptrim_duration":0.0096962509999999995,"stat_sum":{"num_bytes":540276,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":63,"num_read_kb":26256,"num_write":725,"num_write_kb":46019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"132'643","reported_seq":1012,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:43.718334+0000","last_change":"2023-10-10T19:29:43.718305+0000","last_active":"2023-10-10T19:29:43.718334+0000","last_peered":"2023-10-10T19:29:43.718334+0000","last_clean":"2023-10-10T19:29:43.718334+0000","last_became_active":"2023-10-10T18:45:10.407948+0000","last_became_peered":"2023-10-10T18:45:10.407948+0000","last_unstale":"2023-10-10T19:29:43.718334+0000","last_undegraded":"2023-10-10T19:29:43.718334+0000","last_fullsized":"2023-10-10T19:29:43.718334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'643","last_scrub_stamp":"2023-10-10T19:29:43.718205+0000","last_deep_scrub":"132'643","last_deep_scrub_stamp":"2023-10-10T19:29:43.718205+0000","last_clean_scrub_stamp":"2023-10-10T19:29:43.718205+0000","objects_scrubbed":15,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:24:50.063113+0000","scrub_duration":0.029979209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8573947,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":262,"num_read_kb":30418,"num_write":570,"num_write_kb":12898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":102578,"num_omap_keys":183,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"132'952","reported_seq":3587,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:33.425228+0000","last_change":"2023-10-10T19:29:33.425192+0000","last_active":"2023-10-10T19:29:33.425228+0000","last_peered":"2023-10-10T19:29:33.425228+0000","last_clean":"2023-10-10T19:29:33.425228+0000","last_became_active":"2023-10-10T18:45:11.421534+0000","last_became_peered":"2023-10-10T18:45:11.421534+0000","last_unstale":"2023-10-10T19:29:33.425228+0000","last_undegraded":"2023-10-10T19:29:33.425228+0000","last_fullsized":"2023-10-10T19:29:33.425228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'952","last_scrub_stamp":"2023-10-10T19:29:33.425101+0000","last_deep_scrub":"132'952","last_deep_scrub_stamp":"2023-10-10T19:29:33.425101+0000","last_clean_scrub_stamp":"2023-10-10T19:29:33.425101+0000","objects_scrubbed":33,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:50:37.033893+0000","scrub_duration":0.024688640000000001,"objects_trimmed":0,"snaptrim_duration":0.015222243,"stat_sum":{"num_bytes":331805,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":64,"num_read_kb":38795,"num_write":639,"num_write_kb":46131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"132'219","reported_seq":372,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.571663+0000","last_change":"2023-10-10T19:29:42.571637+0000","last_active":"2023-10-10T19:29:42.571663+0000","last_peered":"2023-10-10T19:29:42.571663+0000","last_clean":"2023-10-10T19:29:42.571663+0000","last_became_active":"2023-10-10T18:45:10.409816+0000","last_became_peered":"2023-10-10T18:45:10.409816+0000","last_unstale":"2023-10-10T19:29:42.571663+0000","last_undegraded":"2023-10-10T19:29:42.571663+0000","last_fullsized":"2023-10-10T19:29:42.571663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'219","last_scrub_stamp":"2023-10-10T19:29:42.571548+0000","last_deep_scrub":"132'219","last_deep_scrub_stamp":"2023-10-10T19:29:42.571548+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.571548+0000","objects_scrubbed":12,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:06:27.021509+0000","scrub_duration":0.010029698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":61,"num_write":173,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70925,"num_omap_keys":126,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"132'902","reported_seq":3546,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.010757+0000","last_change":"2023-10-10T19:29:42.010728+0000","last_active":"2023-10-10T19:29:42.010757+0000","last_peered":"2023-10-10T19:29:42.010757+0000","last_clean":"2023-10-10T19:29:42.010757+0000","last_became_active":"2023-10-10T18:45:11.430908+0000","last_became_peered":"2023-10-10T18:45:11.430908+0000","last_unstale":"2023-10-10T19:29:42.010757+0000","last_undegraded":"2023-10-10T19:29:42.010757+0000","last_fullsized":"2023-10-10T19:29:42.010757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'902","last_scrub_stamp":"2023-10-10T19:29:42.010643+0000","last_deep_scrub":"132'902","last_deep_scrub_stamp":"2023-10-10T19:29:42.010643+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.010643+0000","objects_scrubbed":35,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:51:41.084838+0000","scrub_duration":0.032300451000000001,"objects_trimmed":0,"snaptrim_duration":0.0080742209999999995,"stat_sum":{"num_bytes":103314,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":56,"num_read_kb":23103,"num_write":655,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"132'281","reported_seq":441,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:39.018236+0000","last_change":"2023-10-10T19:29:36.136030+0000","last_active":"2023-10-10T19:29:39.018236+0000","last_peered":"2023-10-10T19:29:39.018236+0000","last_clean":"2023-10-10T19:29:39.018236+0000","last_became_active":"2023-10-10T18:45:10.403816+0000","last_became_peered":"2023-10-10T18:45:10.403816+0000","last_unstale":"2023-10-10T19:29:39.018236+0000","last_undegraded":"2023-10-10T19:29:39.018236+0000","last_fullsized":"2023-10-10T19:29:39.018236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'279","last_scrub_stamp":"2023-10-10T19:29:36.135933+0000","last_deep_scrub":"132'279","last_deep_scrub_stamp":"2023-10-10T19:29:36.135933+0000","last_clean_scrub_stamp":"2023-10-10T19:29:36.135933+0000","objects_scrubbed":15,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:22:18.822287+0000","scrub_duration":0.016822342000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194948,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":59,"num_read_kb":16449,"num_write":215,"num_write_kb":4414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64730,"num_omap_keys":109,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"132'975","reported_seq":3617,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:35.143939+0000","last_change":"2023-10-10T19:29:35.143909+0000","last_active":"2023-10-10T19:29:35.143939+0000","last_peered":"2023-10-10T19:29:35.143939+0000","last_clean":"2023-10-10T19:29:35.143939+0000","last_became_active":"2023-10-10T18:45:11.419480+0000","last_became_peered":"2023-10-10T18:45:11.419480+0000","last_unstale":"2023-10-10T19:29:35.143939+0000","last_undegraded":"2023-10-10T19:29:35.143939+0000","last_fullsized":"2023-10-10T19:29:35.143939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'975","last_scrub_stamp":"2023-10-10T19:29:35.143816+0000","last_deep_scrub":"132'975","last_deep_scrub_stamp":"2023-10-10T19:29:35.143816+0000","last_clean_scrub_stamp":"2023-10-10T19:29:35.143816+0000","objects_scrubbed":35,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:54:49.422231+0000","scrub_duration":0.025232871,"objects_trimmed":0,"snaptrim_duration":0.015774979000000001,"stat_sum":{"num_bytes":487172,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":71,"num_read_kb":24956,"num_write":677,"num_write_kb":31709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"132'838","reported_seq":1285,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.356700+0000","last_change":"2023-10-10T19:29:30.356675+0000","last_active":"2023-10-10T19:29:30.356700+0000","last_peered":"2023-10-10T19:29:30.356700+0000","last_clean":"2023-10-10T19:29:30.356700+0000","last_became_active":"2023-10-10T18:45:10.394257+0000","last_became_peered":"2023-10-10T18:45:10.394257+0000","last_unstale":"2023-10-10T19:29:30.356700+0000","last_undegraded":"2023-10-10T19:29:30.356700+0000","last_fullsized":"2023-10-10T19:29:30.356700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'838","last_scrub_stamp":"2023-10-10T19:29:30.356605+0000","last_deep_scrub":"132'838","last_deep_scrub_stamp":"2023-10-10T19:29:30.356605+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.356605+0000","objects_scrubbed":20,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:03:47.996301+0000","scrub_duration":0.044624234999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227398,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":359,"num_read_kb":46031,"num_write":771,"num_write_kb":13519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":95526,"num_omap_keys":174,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"132'949","reported_seq":3592,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.362095+0000","last_change":"2023-10-10T19:29:31.362073+0000","last_active":"2023-10-10T19:29:31.362095+0000","last_peered":"2023-10-10T19:29:31.362095+0000","last_clean":"2023-10-10T19:29:31.362095+0000","last_became_active":"2023-10-10T18:45:11.420264+0000","last_became_peered":"2023-10-10T18:45:11.420264+0000","last_unstale":"2023-10-10T19:29:31.362095+0000","last_undegraded":"2023-10-10T19:29:31.362095+0000","last_fullsized":"2023-10-10T19:29:31.362095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'949","last_scrub_stamp":"2023-10-10T19:29:31.362007+0000","last_deep_scrub":"132'949","last_deep_scrub_stamp":"2023-10-10T19:29:31.362007+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.362007+0000","objects_scrubbed":32,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:44.834185+0000","scrub_duration":0.026536431999999999,"objects_trimmed":0,"snaptrim_duration":0.01251216,"stat_sum":{"num_bytes":121104,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":62,"num_read_kb":21731,"num_write":687,"num_write_kb":32925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"132'735","reported_seq":906,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.681955+0000","last_change":"2023-10-10T19:29:48.681927+0000","last_active":"2023-10-10T19:29:48.681955+0000","last_peered":"2023-10-10T19:29:48.681955+0000","last_clean":"2023-10-10T19:29:48.681955+0000","last_became_active":"2023-10-10T18:45:10.405069+0000","last_became_peered":"2023-10-10T18:45:10.405069+0000","last_unstale":"2023-10-10T19:29:48.681955+0000","last_undegraded":"2023-10-10T19:29:48.681955+0000","last_fullsized":"2023-10-10T19:29:48.681955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'735","last_scrub_stamp":"2023-10-10T19:29:48.681838+0000","last_deep_scrub":"132'735","last_deep_scrub_stamp":"2023-10-10T19:29:48.681838+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.681838+0000","objects_scrubbed":15,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:31:12.687746+0000","scrub_duration":0.036150337999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15771894,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":58,"num_read_kb":76305,"num_write":656,"num_write_kb":19955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":55118,"num_omap_keys":99,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"132'1000","reported_seq":3637,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:06.079446+0000","last_change":"2023-10-10T19:30:06.079422+0000","last_active":"2023-10-10T19:30:06.079446+0000","last_peered":"2023-10-10T19:30:06.079446+0000","last_clean":"2023-10-10T19:30:06.079446+0000","last_became_active":"2023-10-10T18:45:11.426739+0000","last_became_peered":"2023-10-10T18:45:11.426739+0000","last_unstale":"2023-10-10T19:30:06.079446+0000","last_undegraded":"2023-10-10T19:30:06.079446+0000","last_fullsized":"2023-10-10T19:30:06.079446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1000","last_scrub_stamp":"2023-10-10T19:30:06.079344+0000","last_deep_scrub":"132'1000","last_deep_scrub_stamp":"2023-10-10T19:30:06.079344+0000","last_clean_scrub_stamp":"2023-10-10T19:30:06.079344+0000","objects_scrubbed":31,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:30:20.873932+0000","scrub_duration":0.032878220999999999,"objects_trimmed":0,"snaptrim_duration":0.017093574,"stat_sum":{"num_bytes":558529,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":73,"num_read_kb":45017,"num_write":693,"num_write_kb":44107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"132'423","reported_seq":590,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:34.186050+0000","last_change":"2023-10-10T19:29:34.186026+0000","last_active":"2023-10-10T19:29:34.186050+0000","last_peered":"2023-10-10T19:29:34.186050+0000","last_clean":"2023-10-10T19:29:34.186050+0000","last_became_active":"2023-10-10T18:45:10.403770+0000","last_became_peered":"2023-10-10T18:45:10.403770+0000","last_unstale":"2023-10-10T19:29:34.186050+0000","last_undegraded":"2023-10-10T19:29:34.186050+0000","last_fullsized":"2023-10-10T19:29:34.186050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'423","last_scrub_stamp":"2023-10-10T19:29:34.185945+0000","last_deep_scrub":"132'423","last_deep_scrub_stamp":"2023-10-10T19:29:34.185945+0000","last_clean_scrub_stamp":"2023-10-10T19:29:34.185945+0000","objects_scrubbed":16,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:58:02.739712+0000","scrub_duration":0.049211269000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":54,"num_read_kb":84439,"num_write":360,"num_write_kb":16787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":79730,"num_omap_keys":141,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"132'876","reported_seq":3520,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.226230+0000","last_change":"2023-10-10T19:29:57.226206+0000","last_active":"2023-10-10T19:29:57.226230+0000","last_peered":"2023-10-10T19:29:57.226230+0000","last_clean":"2023-10-10T19:29:57.226230+0000","last_became_active":"2023-10-10T18:45:11.416164+0000","last_became_peered":"2023-10-10T18:45:11.416164+0000","last_unstale":"2023-10-10T19:29:57.226230+0000","last_undegraded":"2023-10-10T19:29:57.226230+0000","last_fullsized":"2023-10-10T19:29:57.226230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'876","last_scrub_stamp":"2023-10-10T19:29:57.226124+0000","last_deep_scrub":"132'876","last_deep_scrub_stamp":"2023-10-10T19:29:57.226124+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.226124+0000","objects_scrubbed":42,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:10.624743+0000","scrub_duration":0.034336075000000001,"objects_trimmed":0,"snaptrim_duration":0.0075683979999999996,"stat_sum":{"num_bytes":1111788,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":52,"num_read_kb":24653,"num_write":611,"num_write_kb":45399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"132'823","reported_seq":1013,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:32.342694+0000","last_change":"2023-10-10T19:29:32.342669+0000","last_active":"2023-10-10T19:29:32.342694+0000","last_peered":"2023-10-10T19:29:32.342694+0000","last_clean":"2023-10-10T19:29:32.342694+0000","last_became_active":"2023-10-10T18:45:10.405518+0000","last_became_peered":"2023-10-10T18:45:10.405518+0000","last_unstale":"2023-10-10T19:29:32.342694+0000","last_undegraded":"2023-10-10T19:29:32.342694+0000","last_fullsized":"2023-10-10T19:29:32.342694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'823","last_scrub_stamp":"2023-10-10T19:29:32.342571+0000","last_deep_scrub":"132'823","last_deep_scrub_stamp":"2023-10-10T19:29:32.342571+0000","last_clean_scrub_stamp":"2023-10-10T19:29:32.342571+0000","objects_scrubbed":19,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:57:42.635501+0000","scrub_duration":0.051952427000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":113,"num_read_kb":78302,"num_write":734,"num_write_kb":25074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":59966,"num_omap_keys":106,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"132'933","reported_seq":3582,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:37.129812+0000","last_change":"2023-10-10T19:29:37.129774+0000","last_active":"2023-10-10T19:29:37.129812+0000","last_peered":"2023-10-10T19:29:37.129812+0000","last_clean":"2023-10-10T19:29:37.129812+0000","last_became_active":"2023-10-10T18:45:11.429282+0000","last_became_peered":"2023-10-10T18:45:11.429282+0000","last_unstale":"2023-10-10T19:29:37.129812+0000","last_undegraded":"2023-10-10T19:29:37.129812+0000","last_fullsized":"2023-10-10T19:29:37.129812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'933","last_scrub_stamp":"2023-10-10T19:29:37.129655+0000","last_deep_scrub":"132'933","last_deep_scrub_stamp":"2023-10-10T19:29:37.129655+0000","last_clean_scrub_stamp":"2023-10-10T19:29:37.129655+0000","objects_scrubbed":33,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:50:10.731137+0000","scrub_duration":0.028840863000000001,"objects_trimmed":0,"snaptrim_duration":0.01616896,"stat_sum":{"num_bytes":90659,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":67,"num_read_kb":45230,"num_write":665,"num_write_kb":49103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"132'621","reported_seq":769,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.983083+0000","last_change":"2023-10-10T19:29:28.983043+0000","last_active":"2023-10-10T19:29:28.983083+0000","last_peered":"2023-10-10T19:29:28.983083+0000","last_clean":"2023-10-10T19:29:28.983083+0000","last_became_active":"2023-10-10T18:45:10.399896+0000","last_became_peered":"2023-10-10T18:45:10.399896+0000","last_unstale":"2023-10-10T19:29:28.983083+0000","last_undegraded":"2023-10-10T19:29:28.983083+0000","last_fullsized":"2023-10-10T19:29:28.983083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'621","last_scrub_stamp":"2023-10-10T19:29:28.982898+0000","last_deep_scrub":"132'621","last_deep_scrub_stamp":"2023-10-10T19:29:28.982898+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.982898+0000","objects_scrubbed":17,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:02:20.298697+0000","scrub_duration":0.023284135000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":23,"num_read_kb":12341,"num_write":750,"num_write_kb":8877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":130710,"num_omap_keys":230,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"132'976","reported_seq":3602,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:02.094991+0000","last_change":"2023-10-10T19:30:02.094965+0000","last_active":"2023-10-10T19:30:02.094991+0000","last_peered":"2023-10-10T19:30:02.094991+0000","last_clean":"2023-10-10T19:30:02.094991+0000","last_became_active":"2023-10-10T18:45:11.427476+0000","last_became_peered":"2023-10-10T18:45:11.427476+0000","last_unstale":"2023-10-10T19:30:02.094991+0000","last_undegraded":"2023-10-10T19:30:02.094991+0000","last_fullsized":"2023-10-10T19:30:02.094991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'976","last_scrub_stamp":"2023-10-10T19:30:02.094882+0000","last_deep_scrub":"132'976","last_deep_scrub_stamp":"2023-10-10T19:30:02.094882+0000","last_clean_scrub_stamp":"2023-10-10T19:30:02.094882+0000","objects_scrubbed":38,"log_size":976,"log_dups_size":0,"ondisk_log_size":976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:13:17.747270+0000","scrub_duration":0.041304615000000003,"objects_trimmed":0,"snaptrim_duration":0.015882960000000002,"stat_sum":{"num_bytes":708648,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":57,"num_read_kb":20900,"num_write":676,"num_write_kb":24718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"132'177","reported_seq":325,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.696712+0000","last_change":"2023-10-10T19:29:30.696675+0000","last_active":"2023-10-10T19:29:30.696712+0000","last_peered":"2023-10-10T19:29:30.696712+0000","last_clean":"2023-10-10T19:29:30.696712+0000","last_became_active":"2023-10-10T18:45:10.406024+0000","last_became_peered":"2023-10-10T18:45:10.406024+0000","last_unstale":"2023-10-10T19:29:30.696712+0000","last_undegraded":"2023-10-10T19:29:30.696712+0000","last_fullsized":"2023-10-10T19:29:30.696712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'177","last_scrub_stamp":"2023-10-10T19:29:30.696594+0000","last_deep_scrub":"132'177","last_deep_scrub_stamp":"2023-10-10T19:29:30.696594+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.696594+0000","objects_scrubbed":15,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:22:41.559522+0000","scrub_duration":0.035497603000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":45123,"num_write":119,"num_write_kb":12578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":122093,"num_omap_keys":206,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"132'1083","reported_seq":3703,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:56.639612+0000","last_change":"2023-10-10T19:29:56.639585+0000","last_active":"2023-10-10T19:29:56.639612+0000","last_peered":"2023-10-10T19:29:56.639612+0000","last_clean":"2023-10-10T19:29:56.639612+0000","last_became_active":"2023-10-10T18:45:11.422876+0000","last_became_peered":"2023-10-10T18:45:11.422876+0000","last_unstale":"2023-10-10T19:29:56.639612+0000","last_undegraded":"2023-10-10T19:29:56.639612+0000","last_fullsized":"2023-10-10T19:29:56.639612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1083","last_scrub_stamp":"2023-10-10T19:29:56.639500+0000","last_deep_scrub":"132'1083","last_deep_scrub_stamp":"2023-10-10T19:29:56.639500+0000","last_clean_scrub_stamp":"2023-10-10T19:29:56.639500+0000","objects_scrubbed":34,"log_size":1083,"log_dups_size":0,"ondisk_log_size":1083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:56.591064+0000","scrub_duration":0.031050833999999999,"objects_trimmed":0,"snaptrim_duration":0.011491507999999999,"stat_sum":{"num_bytes":461869,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":71,"num_read_kb":20716,"num_write":730,"num_write_kb":35297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"132'359","reported_seq":518,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.696909+0000","last_change":"2023-10-10T19:29:28.696870+0000","last_active":"2023-10-10T19:29:28.696909+0000","last_peered":"2023-10-10T19:29:28.696909+0000","last_clean":"2023-10-10T19:29:28.696909+0000","last_became_active":"2023-10-10T18:45:10.408590+0000","last_became_peered":"2023-10-10T18:45:10.408590+0000","last_unstale":"2023-10-10T19:29:28.696909+0000","last_undegraded":"2023-10-10T19:29:28.696909+0000","last_fullsized":"2023-10-10T19:29:28.696909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'359","last_scrub_stamp":"2023-10-10T19:29:28.696790+0000","last_deep_scrub":"132'359","last_deep_scrub_stamp":"2023-10-10T19:29:28.696790+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.696790+0000","objects_scrubbed":18,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:28:30.574457+0000","scrub_duration":0.036111012999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":73794,"num_write":284,"num_write_kb":20986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":75814,"num_omap_keys":136,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"132'979","reported_seq":3658,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:00.983629+0000","last_change":"2023-10-10T19:30:00.983588+0000","last_active":"2023-10-10T19:30:00.983629+0000","last_peered":"2023-10-10T19:30:00.983629+0000","last_clean":"2023-10-10T19:30:00.983629+0000","last_became_active":"2023-10-10T18:45:11.425075+0000","last_became_peered":"2023-10-10T18:45:11.425075+0000","last_unstale":"2023-10-10T19:30:00.983629+0000","last_undegraded":"2023-10-10T19:30:00.983629+0000","last_fullsized":"2023-10-10T19:30:00.983629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'979","last_scrub_stamp":"2023-10-10T19:30:00.983505+0000","last_deep_scrub":"132'979","last_deep_scrub_stamp":"2023-10-10T19:30:00.983505+0000","last_clean_scrub_stamp":"2023-10-10T19:30:00.983505+0000","objects_scrubbed":26,"log_size":979,"log_dups_size":0,"ondisk_log_size":979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:52:48.152482+0000","scrub_duration":0.025551553000000001,"objects_trimmed":0,"snaptrim_duration":0.0071091339999999996,"stat_sum":{"num_bytes":369103,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":93,"num_read_kb":25139,"num_write":713,"num_write_kb":35847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"132'984","reported_seq":3621,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:55.671168+0000","last_change":"2023-10-10T19:29:55.671132+0000","last_active":"2023-10-10T19:29:55.671168+0000","last_peered":"2023-10-10T19:29:55.671168+0000","last_clean":"2023-10-10T19:29:55.671168+0000","last_became_active":"2023-10-10T18:45:11.418618+0000","last_became_peered":"2023-10-10T18:45:11.418618+0000","last_unstale":"2023-10-10T19:29:55.671168+0000","last_undegraded":"2023-10-10T19:29:55.671168+0000","last_fullsized":"2023-10-10T19:29:55.671168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'984","last_scrub_stamp":"2023-10-10T19:29:55.671004+0000","last_deep_scrub":"132'984","last_deep_scrub_stamp":"2023-10-10T19:29:55.671004+0000","last_clean_scrub_stamp":"2023-10-10T19:29:55.671004+0000","objects_scrubbed":39,"log_size":984,"log_dups_size":0,"ondisk_log_size":984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:40:43.124234+0000","scrub_duration":0.038408824000000001,"objects_trimmed":0,"snaptrim_duration":0.0077260940000000002,"stat_sum":{"num_bytes":131785,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":61,"num_read_kb":19697,"num_write":689,"num_write_kb":26724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"132'700","reported_seq":1135,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:30.351264+0000","last_change":"2023-10-10T19:29:30.351227+0000","last_active":"2023-10-10T19:29:30.351264+0000","last_peered":"2023-10-10T19:29:30.351264+0000","last_clean":"2023-10-10T19:29:30.351264+0000","last_became_active":"2023-10-10T18:45:10.404221+0000","last_became_peered":"2023-10-10T18:45:10.404221+0000","last_unstale":"2023-10-10T19:29:30.351264+0000","last_undegraded":"2023-10-10T19:29:30.351264+0000","last_fullsized":"2023-10-10T19:29:30.351264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'700","last_scrub_stamp":"2023-10-10T19:29:30.351098+0000","last_deep_scrub":"132'700","last_deep_scrub_stamp":"2023-10-10T19:29:30.351098+0000","last_clean_scrub_stamp":"2023-10-10T19:29:30.351098+0000","objects_scrubbed":17,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:16:26.607570+0000","scrub_duration":0.030203292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9108221,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":314,"num_read_kb":40841,"num_write":610,"num_write_kb":9448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":67648,"num_omap_keys":120,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'922","reported_seq":3604,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:39.851461+0000","last_change":"2023-10-10T19:29:39.851434+0000","last_active":"2023-10-10T19:29:39.851461+0000","last_peered":"2023-10-10T19:29:39.851461+0000","last_clean":"2023-10-10T19:29:39.851461+0000","last_became_active":"2023-10-10T18:45:11.422222+0000","last_became_peered":"2023-10-10T18:45:11.422222+0000","last_unstale":"2023-10-10T19:29:39.851461+0000","last_undegraded":"2023-10-10T19:29:39.851461+0000","last_fullsized":"2023-10-10T19:29:39.851461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'922","last_scrub_stamp":"2023-10-10T19:29:39.851353+0000","last_deep_scrub":"132'922","last_deep_scrub_stamp":"2023-10-10T19:29:39.851353+0000","last_clean_scrub_stamp":"2023-10-10T19:29:39.851353+0000","objects_scrubbed":42,"log_size":922,"log_dups_size":0,"ondisk_log_size":922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T07:19:01.743649+0000","scrub_duration":0.028531329000000001,"objects_trimmed":0,"snaptrim_duration":0.017091021000000001,"stat_sum":{"num_bytes":951410,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":88,"num_read_kb":19324,"num_write":674,"num_write_kb":26724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"132'152","reported_seq":320,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.682795+0000","last_change":"2023-10-10T19:29:29.682765+0000","last_active":"2023-10-10T19:29:29.682795+0000","last_peered":"2023-10-10T19:29:29.682795+0000","last_clean":"2023-10-10T19:29:29.682795+0000","last_became_active":"2023-10-10T18:45:10.404005+0000","last_became_peered":"2023-10-10T18:45:10.404005+0000","last_unstale":"2023-10-10T19:29:29.682795+0000","last_undegraded":"2023-10-10T19:29:29.682795+0000","last_fullsized":"2023-10-10T19:29:29.682795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'152","last_scrub_stamp":"2023-10-10T19:29:29.682678+0000","last_deep_scrub":"132'152","last_deep_scrub_stamp":"2023-10-10T19:29:29.682678+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.682678+0000","objects_scrubbed":15,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:06:08.947475+0000","scrub_duration":0.025814232999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388756,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":66,"num_read_kb":36939,"num_write":74,"num_write_kb":8306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":75158,"num_omap_keys":136,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"132'119","reported_seq":287,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:40.097607+0000","last_change":"2023-10-10T19:29:40.097581+0000","last_active":"2023-10-10T19:29:40.097607+0000","last_peered":"2023-10-10T19:29:40.097607+0000","last_clean":"2023-10-10T19:29:40.097607+0000","last_became_active":"2023-10-10T18:45:07.110949+0000","last_became_peered":"2023-10-10T18:45:07.110949+0000","last_unstale":"2023-10-10T19:29:40.097607+0000","last_undegraded":"2023-10-10T19:29:40.097607+0000","last_fullsized":"2023-10-10T19:29:40.097607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'119","last_scrub_stamp":"2023-10-10T19:29:40.097457+0000","last_deep_scrub":"132'119","last_deep_scrub_stamp":"2023-10-10T19:29:40.097457+0000","last_clean_scrub_stamp":"2023-10-10T19:29:40.097457+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:06:14.657014+0000","scrub_duration":0.0047236919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"132'934","reported_seq":3569,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:44.310730+0000","last_change":"2023-10-10T19:29:44.310686+0000","last_active":"2023-10-10T19:29:44.310730+0000","last_peered":"2023-10-10T19:29:44.310730+0000","last_clean":"2023-10-10T19:29:44.310730+0000","last_became_active":"2023-10-10T18:45:11.430049+0000","last_became_peered":"2023-10-10T18:45:11.430049+0000","last_unstale":"2023-10-10T19:29:44.310730+0000","last_undegraded":"2023-10-10T19:29:44.310730+0000","last_fullsized":"2023-10-10T19:29:44.310730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'934","last_scrub_stamp":"2023-10-10T19:29:44.310602+0000","last_deep_scrub":"132'934","last_deep_scrub_stamp":"2023-10-10T19:29:44.310602+0000","last_clean_scrub_stamp":"2023-10-10T19:29:44.310602+0000","objects_scrubbed":39,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:47:39.883204+0000","scrub_duration":0.034571985999999999,"objects_trimmed":0,"snaptrim_duration":0.010927930000000001,"stat_sum":{"num_bytes":3454752,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":55,"num_read_kb":20626,"num_write":663,"num_write_kb":30766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"132'990","reported_seq":1152,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:41.209314+0000","last_change":"2023-10-10T19:29:41.209289+0000","last_active":"2023-10-10T19:29:41.209314+0000","last_peered":"2023-10-10T19:29:41.209314+0000","last_clean":"2023-10-10T19:29:41.209314+0000","last_became_active":"2023-10-10T18:45:10.412777+0000","last_became_peered":"2023-10-10T18:45:10.412777+0000","last_unstale":"2023-10-10T19:29:41.209314+0000","last_undegraded":"2023-10-10T19:29:41.209314+0000","last_fullsized":"2023-10-10T19:29:41.209314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'990","last_scrub_stamp":"2023-10-10T19:29:41.209202+0000","last_deep_scrub":"132'990","last_deep_scrub_stamp":"2023-10-10T19:29:41.209202+0000","last_clean_scrub_stamp":"2023-10-10T19:29:41.209202+0000","objects_scrubbed":17,"log_size":990,"log_dups_size":0,"ondisk_log_size":990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:08.926653+0000","scrub_duration":0.069287455999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":38,"num_read_kb":89926,"num_write":914,"num_write_kb":29269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70568,"num_omap_keys":125,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"132'839","reported_seq":1003,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:42.127383+0000","last_change":"2023-10-10T19:29:42.127330+0000","last_active":"2023-10-10T19:29:42.127383+0000","last_peered":"2023-10-10T19:29:42.127383+0000","last_clean":"2023-10-10T19:29:42.127383+0000","last_became_active":"2023-10-10T18:45:10.415225+0000","last_became_peered":"2023-10-10T18:45:10.415225+0000","last_unstale":"2023-10-10T19:29:42.127383+0000","last_undegraded":"2023-10-10T19:29:42.127383+0000","last_fullsized":"2023-10-10T19:29:42.127383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'839","last_scrub_stamp":"2023-10-10T19:29:42.127227+0000","last_deep_scrub":"132'839","last_deep_scrub_stamp":"2023-10-10T19:29:42.127227+0000","last_clean_scrub_stamp":"2023-10-10T19:29:42.127227+0000","objects_scrubbed":14,"log_size":839,"log_dups_size":0,"ondisk_log_size":839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:46:38.680084+0000","scrub_duration":0.027082854999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":45,"num_read_kb":45594,"num_write":766,"num_write_kb":12847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":79898,"num_omap_keys":140,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"132'1056","reported_seq":3681,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:48.647554+0000","last_change":"2023-10-10T19:29:48.647516+0000","last_active":"2023-10-10T19:29:48.647554+0000","last_peered":"2023-10-10T19:29:48.647554+0000","last_clean":"2023-10-10T19:29:48.647554+0000","last_became_active":"2023-10-10T18:45:11.418714+0000","last_became_peered":"2023-10-10T18:45:11.418714+0000","last_unstale":"2023-10-10T19:29:48.647554+0000","last_undegraded":"2023-10-10T19:29:48.647554+0000","last_fullsized":"2023-10-10T19:29:48.647554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1056","last_scrub_stamp":"2023-10-10T19:29:48.647374+0000","last_deep_scrub":"132'1056","last_deep_scrub_stamp":"2023-10-10T19:29:48.647374+0000","last_clean_scrub_stamp":"2023-10-10T19:29:48.647374+0000","objects_scrubbed":38,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:49:57.030894+0000","scrub_duration":0.028559443,"objects_trimmed":0,"snaptrim_duration":0.0075449510000000003,"stat_sum":{"num_bytes":524227,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":65,"num_read_kb":49387,"num_write":726,"num_write_kb":52412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'292","reported_seq":459,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.731119+0000","last_change":"2023-10-10T19:29:29.731097+0000","last_active":"2023-10-10T19:29:29.731119+0000","last_peered":"2023-10-10T19:29:29.731119+0000","last_clean":"2023-10-10T19:29:29.731119+0000","last_became_active":"2023-10-10T18:45:10.405763+0000","last_became_peered":"2023-10-10T18:45:10.405763+0000","last_unstale":"2023-10-10T19:29:29.731119+0000","last_undegraded":"2023-10-10T19:29:29.731119+0000","last_fullsized":"2023-10-10T19:29:29.731119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'292","last_scrub_stamp":"2023-10-10T19:29:29.731010+0000","last_deep_scrub":"132'292","last_deep_scrub_stamp":"2023-10-10T19:29:29.731010+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.731010+0000","objects_scrubbed":18,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:44:14.879878+0000","scrub_duration":0.021479405999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":81,"num_read_kb":20675,"num_write":479,"num_write_kb":4749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":103356,"num_omap_keys":179,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"132'1000","reported_seq":3638,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:58.722286+0000","last_change":"2023-10-10T19:29:58.722260+0000","last_active":"2023-10-10T19:29:58.722286+0000","last_peered":"2023-10-10T19:29:58.722286+0000","last_clean":"2023-10-10T19:29:58.722286+0000","last_became_active":"2023-10-10T18:46:01.267092+0000","last_became_peered":"2023-10-10T18:46:01.267092+0000","last_unstale":"2023-10-10T19:29:58.722286+0000","last_undegraded":"2023-10-10T19:29:58.722286+0000","last_fullsized":"2023-10-10T19:29:58.722286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1000","last_scrub_stamp":"2023-10-10T19:29:58.722178+0000","last_deep_scrub":"132'1000","last_deep_scrub_stamp":"2023-10-10T19:29:58.722178+0000","last_clean_scrub_stamp":"2023-10-10T19:29:58.722178+0000","objects_scrubbed":43,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:10:33.521494+0000","scrub_duration":0.046338531000000002,"objects_trimmed":0,"snaptrim_duration":0.0083609700000000006,"stat_sum":{"num_bytes":2285567,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":68,"num_read_kb":33081,"num_write":691,"num_write_kb":42143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"132'166","reported_seq":324,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:29.151113+0000","last_change":"2023-10-10T19:29:29.151081+0000","last_active":"2023-10-10T19:29:29.151113+0000","last_peered":"2023-10-10T19:29:29.151113+0000","last_clean":"2023-10-10T19:29:29.151113+0000","last_became_active":"2023-10-10T18:45:10.410773+0000","last_became_peered":"2023-10-10T18:45:10.410773+0000","last_unstale":"2023-10-10T19:29:29.151113+0000","last_undegraded":"2023-10-10T19:29:29.151113+0000","last_fullsized":"2023-10-10T19:29:29.151113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'166","last_scrub_stamp":"2023-10-10T19:29:29.150987+0000","last_deep_scrub":"132'166","last_deep_scrub_stamp":"2023-10-10T19:29:29.150987+0000","last_clean_scrub_stamp":"2023-10-10T19:29:29.150987+0000","objects_scrubbed":12,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:52:58.469784+0000","scrub_duration":0.0090081499999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":8269,"num_write":94,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64552,"num_omap_keys":116,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"132'914","reported_seq":3586,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:43.257893+0000","last_change":"2023-10-10T19:29:43.257858+0000","last_active":"2023-10-10T19:29:43.257893+0000","last_peered":"2023-10-10T19:29:43.257893+0000","last_clean":"2023-10-10T19:29:43.257893+0000","last_became_active":"2023-10-10T18:45:11.416026+0000","last_became_peered":"2023-10-10T18:45:11.416026+0000","last_unstale":"2023-10-10T19:29:43.257893+0000","last_undegraded":"2023-10-10T19:29:43.257893+0000","last_fullsized":"2023-10-10T19:29:43.257893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'914","last_scrub_stamp":"2023-10-10T19:29:43.257766+0000","last_deep_scrub":"132'914","last_deep_scrub_stamp":"2023-10-10T19:29:43.257766+0000","last_clean_scrub_stamp":"2023-10-10T19:29:43.257766+0000","objects_scrubbed":42,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:41.946443+0000","scrub_duration":0.029919424999999999,"objects_trimmed":0,"snaptrim_duration":0.012721872,"stat_sum":{"num_bytes":482904,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":85,"num_read_kb":32652,"num_write":648,"num_write_kb":36187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"132'241","reported_seq":395,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.155442+0000","last_change":"2023-10-10T19:29:57.155415+0000","last_active":"2023-10-10T19:29:57.155442+0000","last_peered":"2023-10-10T19:29:57.155442+0000","last_clean":"2023-10-10T19:29:57.155442+0000","last_became_active":"2023-10-10T18:45:10.415140+0000","last_became_peered":"2023-10-10T18:45:10.415140+0000","last_unstale":"2023-10-10T19:29:57.155442+0000","last_undegraded":"2023-10-10T19:29:57.155442+0000","last_fullsized":"2023-10-10T19:29:57.155442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'241","last_scrub_stamp":"2023-10-10T19:29:57.155334+0000","last_deep_scrub":"132'241","last_deep_scrub_stamp":"2023-10-10T19:29:57.155334+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.155334+0000","objects_scrubbed":13,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:40:22.341488+0000","scrub_duration":0.030119740999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":57389,"num_write":173,"num_write_kb":12477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":92062,"num_omap_keys":147,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"132'971","reported_seq":3622,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.000295+0000","last_change":"2023-10-10T19:29:45.000264+0000","last_active":"2023-10-10T19:29:45.000295+0000","last_peered":"2023-10-10T19:29:45.000295+0000","last_clean":"2023-10-10T19:29:45.000295+0000","last_became_active":"2023-10-10T18:45:11.424311+0000","last_became_peered":"2023-10-10T18:45:11.424311+0000","last_unstale":"2023-10-10T19:29:45.000295+0000","last_undegraded":"2023-10-10T19:29:45.000295+0000","last_fullsized":"2023-10-10T19:29:45.000295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'971","last_scrub_stamp":"2023-10-10T19:29:45.000170+0000","last_deep_scrub":"132'971","last_deep_scrub_stamp":"2023-10-10T19:29:45.000170+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.000170+0000","objects_scrubbed":32,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:53:24.335195+0000","scrub_duration":0.025238327000000001,"objects_trimmed":0,"snaptrim_duration":0.0089059480000000003,"stat_sum":{"num_bytes":820479,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":67,"num_read_kb":26489,"num_write":692,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"132'387","reported_seq":550,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:28.747339+0000","last_change":"2023-10-10T19:29:28.747314+0000","last_active":"2023-10-10T19:29:28.747339+0000","last_peered":"2023-10-10T19:29:28.747339+0000","last_clean":"2023-10-10T19:29:28.747339+0000","last_became_active":"2023-10-10T18:45:10.412248+0000","last_became_peered":"2023-10-10T18:45:10.412248+0000","last_unstale":"2023-10-10T19:29:28.747339+0000","last_undegraded":"2023-10-10T19:29:28.747339+0000","last_fullsized":"2023-10-10T19:29:28.747339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'387","last_scrub_stamp":"2023-10-10T19:29:28.747246+0000","last_deep_scrub":"132'387","last_deep_scrub_stamp":"2023-10-10T19:29:28.747246+0000","last_clean_scrub_stamp":"2023-10-10T19:29:28.747246+0000","objects_scrubbed":19,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:01:46.340517+0000","scrub_duration":0.020997531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":70,"num_read_kb":20656,"num_write":333,"num_write_kb":8747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":128481,"num_omap_keys":203,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"132'951","reported_seq":3597,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:03.064395+0000","last_change":"2023-10-10T19:30:03.064365+0000","last_active":"2023-10-10T19:30:03.064395+0000","last_peered":"2023-10-10T19:30:03.064395+0000","last_clean":"2023-10-10T19:30:03.064395+0000","last_became_active":"2023-10-10T18:45:11.413722+0000","last_became_peered":"2023-10-10T18:45:11.413722+0000","last_unstale":"2023-10-10T19:30:03.064395+0000","last_undegraded":"2023-10-10T19:30:03.064395+0000","last_fullsized":"2023-10-10T19:30:03.064395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'951","last_scrub_stamp":"2023-10-10T19:30:03.064283+0000","last_deep_scrub":"132'951","last_deep_scrub_stamp":"2023-10-10T19:30:03.064283+0000","last_clean_scrub_stamp":"2023-10-10T19:30:03.064283+0000","objects_scrubbed":43,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:39:24.338470+0000","scrub_duration":0.041115824000000002,"objects_trimmed":0,"snaptrim_duration":0.012052205,"stat_sum":{"num_bytes":635543,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":63,"num_read_kb":40064,"num_write":667,"num_write_kb":56874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"132'1325","reported_seq":1491,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:50.083396+0000","last_change":"2023-10-10T19:29:31.691013+0000","last_active":"2023-10-10T19:29:50.083396+0000","last_peered":"2023-10-10T19:29:50.083396+0000","last_clean":"2023-10-10T19:29:50.083396+0000","last_became_active":"2023-10-10T18:45:10.405795+0000","last_became_peered":"2023-10-10T18:45:10.405795+0000","last_unstale":"2023-10-10T19:29:50.083396+0000","last_undegraded":"2023-10-10T19:29:50.083396+0000","last_fullsized":"2023-10-10T19:29:50.083396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1319","last_scrub_stamp":"2023-10-10T19:29:31.690927+0000","last_deep_scrub":"132'1319","last_deep_scrub_stamp":"2023-10-10T19:29:31.690927+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.690927+0000","objects_scrubbed":14,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:38:48.082901+0000","scrub_duration":0.043989502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15286709,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":48,"num_read_kb":53317,"num_write":1726,"num_write_kb":25012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40110,"num_omap_keys":72,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"132'933","reported_seq":3577,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:57.037397+0000","last_change":"2023-10-10T19:29:57.037345+0000","last_active":"2023-10-10T19:29:57.037397+0000","last_peered":"2023-10-10T19:29:57.037397+0000","last_clean":"2023-10-10T19:29:57.037397+0000","last_became_active":"2023-10-10T18:45:11.410490+0000","last_became_peered":"2023-10-10T18:45:11.410490+0000","last_unstale":"2023-10-10T19:29:57.037397+0000","last_undegraded":"2023-10-10T19:29:57.037397+0000","last_fullsized":"2023-10-10T19:29:57.037397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'933","last_scrub_stamp":"2023-10-10T19:29:57.037218+0000","last_deep_scrub":"132'933","last_deep_scrub_stamp":"2023-10-10T19:29:57.037218+0000","last_clean_scrub_stamp":"2023-10-10T19:29:57.037218+0000","objects_scrubbed":40,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:12:09.618486+0000","scrub_duration":0.037571087000000003,"objects_trimmed":0,"snaptrim_duration":0.015827366999999998,"stat_sum":{"num_bytes":178682,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":58,"num_read_kb":28965,"num_write":652,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"132'1215","reported_seq":1380,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:45.115778+0000","last_change":"2023-10-10T19:29:45.115751+0000","last_active":"2023-10-10T19:29:45.115778+0000","last_peered":"2023-10-10T19:29:45.115778+0000","last_clean":"2023-10-10T19:29:45.115778+0000","last_became_active":"2023-10-10T18:45:10.410877+0000","last_became_peered":"2023-10-10T18:45:10.410877+0000","last_unstale":"2023-10-10T19:29:45.115778+0000","last_undegraded":"2023-10-10T19:29:45.115778+0000","last_fullsized":"2023-10-10T19:29:45.115778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1215","last_scrub_stamp":"2023-10-10T19:29:45.115661+0000","last_deep_scrub":"132'1215","last_deep_scrub_stamp":"2023-10-10T19:29:45.115661+0000","last_clean_scrub_stamp":"2023-10-10T19:29:45.115661+0000","objects_scrubbed":16,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:37:01.423877+0000","scrub_duration":0.048047320999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":49,"num_read_kb":89747,"num_write":1128,"num_write_kb":25455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":70296,"num_omap_keys":124,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"132'974","reported_seq":3614,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:10.000282+0000","last_change":"2023-10-10T19:30:10.000245+0000","last_active":"2023-10-10T19:30:10.000282+0000","last_peered":"2023-10-10T19:30:10.000282+0000","last_clean":"2023-10-10T19:30:10.000282+0000","last_became_active":"2023-10-10T18:45:11.425435+0000","last_became_peered":"2023-10-10T18:45:11.425435+0000","last_unstale":"2023-10-10T19:30:10.000282+0000","last_undegraded":"2023-10-10T19:30:10.000282+0000","last_fullsized":"2023-10-10T19:30:10.000282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'974","last_scrub_stamp":"2023-10-10T19:30:10.000164+0000","last_deep_scrub":"132'974","last_deep_scrub_stamp":"2023-10-10T19:30:10.000164+0000","last_clean_scrub_stamp":"2023-10-10T19:30:10.000164+0000","objects_scrubbed":36,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:04:49.255703+0000","scrub_duration":0.039024960999999997,"objects_trimmed":0,"snaptrim_duration":0.0065052069999999998,"stat_sum":{"num_bytes":479125,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":64,"num_read_kb":10527,"num_write":680,"num_write_kb":20069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"132'323","reported_seq":474,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:34.384672+0000","last_change":"2023-10-10T19:29:34.384643+0000","last_active":"2023-10-10T19:29:34.384672+0000","last_peered":"2023-10-10T19:29:34.384672+0000","last_clean":"2023-10-10T19:29:34.384672+0000","last_became_active":"2023-10-10T18:45:10.405591+0000","last_became_peered":"2023-10-10T18:45:10.405591+0000","last_unstale":"2023-10-10T19:29:34.384672+0000","last_undegraded":"2023-10-10T19:29:34.384672+0000","last_fullsized":"2023-10-10T19:29:34.384672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'323","last_scrub_stamp":"2023-10-10T19:29:34.384547+0000","last_deep_scrub":"132'323","last_deep_scrub_stamp":"2023-10-10T19:29:34.384547+0000","last_clean_scrub_stamp":"2023-10-10T19:29:34.384547+0000","objects_scrubbed":12,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:06:08.523893+0000","scrub_duration":0.015112752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194858,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":28,"num_read_kb":40985,"num_write":233,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":71992,"num_omap_keys":129,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"132'967","reported_seq":3629,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:59.149786+0000","last_change":"2023-10-10T19:29:59.149750+0000","last_active":"2023-10-10T19:29:59.149786+0000","last_peered":"2023-10-10T19:29:59.149786+0000","last_clean":"2023-10-10T19:29:59.149786+0000","last_became_active":"2023-10-10T18:45:11.423548+0000","last_became_peered":"2023-10-10T18:45:11.423548+0000","last_unstale":"2023-10-10T19:29:59.149786+0000","last_undegraded":"2023-10-10T19:29:59.149786+0000","last_fullsized":"2023-10-10T19:29:59.149786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'967","last_scrub_stamp":"2023-10-10T19:29:59.149633+0000","last_deep_scrub":"132'967","last_deep_scrub_stamp":"2023-10-10T19:29:59.149633+0000","last_clean_scrub_stamp":"2023-10-10T19:29:59.149633+0000","objects_scrubbed":48,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:53:06.235612+0000","scrub_duration":0.046782177000000001,"objects_trimmed":0,"snaptrim_duration":0.016889265000000001,"stat_sum":{"num_bytes":1346662,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":67,"num_read_kb":24822,"num_write":699,"num_write_kb":32529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"132'357","reported_seq":512,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:55.078895+0000","last_change":"2023-10-10T19:29:37.661226+0000","last_active":"2023-10-10T19:29:55.078895+0000","last_peered":"2023-10-10T19:29:55.078895+0000","last_clean":"2023-10-10T19:29:55.078895+0000","last_became_active":"2023-10-10T18:45:10.404858+0000","last_became_peered":"2023-10-10T18:45:10.404858+0000","last_unstale":"2023-10-10T19:29:55.078895+0000","last_undegraded":"2023-10-10T19:29:55.078895+0000","last_fullsized":"2023-10-10T19:29:55.078895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'352","last_scrub_stamp":"2023-10-10T19:29:37.661098+0000","last_deep_scrub":"132'352","last_deep_scrub_stamp":"2023-10-10T19:29:37.661098+0000","last_clean_scrub_stamp":"2023-10-10T19:29:37.661098+0000","objects_scrubbed":11,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:14:09.979735+0000","scrub_duration":0.032683222999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11246969,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":27,"num_read_kb":50704,"num_write":277,"num_write_kb":15297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":68490,"num_omap_keys":115,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"132'1013","reported_seq":3652,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:12.601512+0000","last_change":"2023-10-10T19:30:12.601485+0000","last_active":"2023-10-10T19:30:12.601512+0000","last_peered":"2023-10-10T19:30:12.601512+0000","last_clean":"2023-10-10T19:30:12.601512+0000","last_became_active":"2023-10-10T18:45:11.421579+0000","last_became_peered":"2023-10-10T18:45:11.421579+0000","last_unstale":"2023-10-10T19:30:12.601512+0000","last_undegraded":"2023-10-10T19:30:12.601512+0000","last_fullsized":"2023-10-10T19:30:12.601512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1013","last_scrub_stamp":"2023-10-10T19:30:12.601400+0000","last_deep_scrub":"132'1013","last_deep_scrub_stamp":"2023-10-10T19:30:12.601400+0000","last_clean_scrub_stamp":"2023-10-10T19:30:12.601400+0000","objects_scrubbed":43,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:35:53.052503+0000","scrub_duration":0.035780855,"objects_trimmed":0,"snaptrim_duration":0.012585610000000001,"stat_sum":{"num_bytes":222806,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":60,"num_read_kb":19964,"num_write":709,"num_write_kb":27205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"132'322","reported_seq":481,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:31.316606+0000","last_change":"2023-10-10T19:29:31.316564+0000","last_active":"2023-10-10T19:29:31.316606+0000","last_peered":"2023-10-10T19:29:31.316606+0000","last_clean":"2023-10-10T19:29:31.316606+0000","last_became_active":"2023-10-10T18:45:10.409045+0000","last_became_peered":"2023-10-10T18:45:10.409045+0000","last_unstale":"2023-10-10T19:29:31.316606+0000","last_undegraded":"2023-10-10T19:29:31.316606+0000","last_fullsized":"2023-10-10T19:29:31.316606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'322","last_scrub_stamp":"2023-10-10T19:29:31.316447+0000","last_deep_scrub":"132'322","last_deep_scrub_stamp":"2023-10-10T19:29:31.316447+0000","last_clean_scrub_stamp":"2023-10-10T19:29:31.316447+0000","objects_scrubbed":12,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:27:45.094335+0000","scrub_duration":0.025432287000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":49,"num_read_kb":57404,"num_write":260,"num_write_kb":8464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":65503,"num_omap_keys":117,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"132'1049","reported_seq":3690,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:05.097815+0000","last_change":"2023-10-10T19:30:05.097774+0000","last_active":"2023-10-10T19:30:05.097815+0000","last_peered":"2023-10-10T19:30:05.097815+0000","last_clean":"2023-10-10T19:30:05.097815+0000","last_became_active":"2023-10-10T18:45:11.425189+0000","last_became_peered":"2023-10-10T18:45:11.425189+0000","last_unstale":"2023-10-10T19:30:05.097815+0000","last_undegraded":"2023-10-10T19:30:05.097815+0000","last_fullsized":"2023-10-10T19:30:05.097815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1049","last_scrub_stamp":"2023-10-10T19:30:05.097679+0000","last_deep_scrub":"132'1049","last_deep_scrub_stamp":"2023-10-10T19:30:05.097679+0000","last_clean_scrub_stamp":"2023-10-10T19:30:05.097679+0000","objects_scrubbed":34,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:17:53.287113+0000","scrub_duration":0.033390343000000003,"objects_trimmed":0,"snaptrim_duration":0.016837566000000002,"stat_sum":{"num_bytes":394703,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":67,"num_read_kb":28062,"num_write":753,"num_write_kb":23722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"132'941","reported_seq":3613,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:10.992034+0000","last_change":"2023-10-10T19:30:10.992007+0000","last_active":"2023-10-10T19:30:10.992034+0000","last_peered":"2023-10-10T19:30:10.992034+0000","last_clean":"2023-10-10T19:30:10.992034+0000","last_became_active":"2023-10-10T18:45:11.416643+0000","last_became_peered":"2023-10-10T18:45:11.416643+0000","last_unstale":"2023-10-10T19:30:10.992034+0000","last_undegraded":"2023-10-10T19:30:10.992034+0000","last_fullsized":"2023-10-10T19:30:10.992034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'941","last_scrub_stamp":"2023-10-10T19:30:10.991930+0000","last_deep_scrub":"132'941","last_deep_scrub_stamp":"2023-10-10T19:30:10.991930+0000","last_clean_scrub_stamp":"2023-10-10T19:30:10.991930+0000","objects_scrubbed":31,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:40:51.594438+0000","scrub_duration":0.030546936,"objects_trimmed":0,"snaptrim_duration":0.0074078640000000001,"stat_sum":{"num_bytes":519476,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":77,"num_read_kb":25056,"num_write":683,"num_write_kb":32109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"132'1017","reported_seq":3650,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:41.182730+0000","last_change":"2023-10-10T19:29:41.182702+0000","last_active":"2023-10-10T19:29:41.182730+0000","last_peered":"2023-10-10T19:29:41.182730+0000","last_clean":"2023-10-10T19:29:41.182730+0000","last_became_active":"2023-10-10T18:45:11.424171+0000","last_became_peered":"2023-10-10T18:45:11.424171+0000","last_unstale":"2023-10-10T19:29:41.182730+0000","last_undegraded":"2023-10-10T19:29:41.182730+0000","last_fullsized":"2023-10-10T19:29:41.182730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1017","last_scrub_stamp":"2023-10-10T19:29:41.182614+0000","last_deep_scrub":"132'1017","last_deep_scrub_stamp":"2023-10-10T19:29:41.182614+0000","last_clean_scrub_stamp":"2023-10-10T19:29:41.182614+0000","objects_scrubbed":32,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:38:55.992116+0000","scrub_duration":0.028001608000000001,"objects_trimmed":0,"snaptrim_duration":0.0050938800000000003,"stat_sum":{"num_bytes":416129,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":69,"num_read_kb":25013,"num_write":702,"num_write_kb":37226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"132'1001","reported_seq":3660,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:47.132565+0000","last_change":"2023-10-10T19:29:47.132524+0000","last_active":"2023-10-10T19:29:47.132565+0000","last_peered":"2023-10-10T19:29:47.132565+0000","last_clean":"2023-10-10T19:29:47.132565+0000","last_became_active":"2023-10-10T18:46:00.606420+0000","last_became_peered":"2023-10-10T18:46:00.606420+0000","last_unstale":"2023-10-10T19:29:47.132565+0000","last_undegraded":"2023-10-10T19:29:47.132565+0000","last_fullsized":"2023-10-10T19:29:47.132565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1001","last_scrub_stamp":"2023-10-10T19:29:47.132433+0000","last_deep_scrub":"132'1001","last_deep_scrub_stamp":"2023-10-10T19:29:47.132433+0000","last_clean_scrub_stamp":"2023-10-10T19:29:47.132433+0000","objects_scrubbed":41,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:35:01.992844+0000","scrub_duration":0.032812747000000003,"objects_trimmed":0,"snaptrim_duration":0.0064370249999999999,"stat_sum":{"num_bytes":103727,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":90,"num_read_kb":45570,"num_write":716,"num_write_kb":52505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"132'973","reported_seq":3611,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:49.279974+0000","last_change":"2023-10-10T19:29:49.279944+0000","last_active":"2023-10-10T19:29:49.279974+0000","last_peered":"2023-10-10T19:29:49.279974+0000","last_clean":"2023-10-10T19:29:49.279974+0000","last_became_active":"2023-10-10T18:45:11.412138+0000","last_became_peered":"2023-10-10T18:45:11.412138+0000","last_unstale":"2023-10-10T19:29:49.279974+0000","last_undegraded":"2023-10-10T19:29:49.279974+0000","last_fullsized":"2023-10-10T19:29:49.279974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'973","last_scrub_stamp":"2023-10-10T19:29:49.279855+0000","last_deep_scrub":"132'973","last_deep_scrub_stamp":"2023-10-10T19:29:49.279855+0000","last_clean_scrub_stamp":"2023-10-10T19:29:49.279855+0000","objects_scrubbed":37,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:15:38.764535+0000","scrub_duration":0.031787211000000003,"objects_trimmed":0,"snaptrim_duration":0.014096529,"stat_sum":{"num_bytes":456616,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":70,"num_read_kb":16093,"num_write":668,"num_write_kb":20834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"132'959","reported_seq":3608,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:00.042069+0000","last_change":"2023-10-10T19:30:00.042044+0000","last_active":"2023-10-10T19:30:00.042069+0000","last_peered":"2023-10-10T19:30:00.042069+0000","last_clean":"2023-10-10T19:30:00.042069+0000","last_became_active":"2023-10-10T18:45:11.429256+0000","last_became_peered":"2023-10-10T18:45:11.429256+0000","last_unstale":"2023-10-10T19:30:00.042069+0000","last_undegraded":"2023-10-10T19:30:00.042069+0000","last_fullsized":"2023-10-10T19:30:00.042069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'959","last_scrub_stamp":"2023-10-10T19:30:00.041947+0000","last_deep_scrub":"132'959","last_deep_scrub_stamp":"2023-10-10T19:30:00.041947+0000","last_clean_scrub_stamp":"2023-10-10T19:30:00.041947+0000","objects_scrubbed":41,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:38:23.770402+0000","scrub_duration":0.034907344,"objects_trimmed":0,"snaptrim_duration":0.016808101999999998,"stat_sum":{"num_bytes":708082,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":68,"num_read_kb":48841,"num_write":679,"num_write_kb":56033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"132'1009","reported_seq":3623,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:53.644317+0000","last_change":"2023-10-10T19:29:53.644291+0000","last_active":"2023-10-10T19:29:53.644317+0000","last_peered":"2023-10-10T19:29:53.644317+0000","last_clean":"2023-10-10T19:29:53.644317+0000","last_became_active":"2023-10-10T18:45:11.413072+0000","last_became_peered":"2023-10-10T18:45:11.413072+0000","last_unstale":"2023-10-10T19:29:53.644317+0000","last_undegraded":"2023-10-10T19:29:53.644317+0000","last_fullsized":"2023-10-10T19:29:53.644317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1009","last_scrub_stamp":"2023-10-10T19:29:53.644211+0000","last_deep_scrub":"132'1009","last_deep_scrub_stamp":"2023-10-10T19:29:53.644211+0000","last_clean_scrub_stamp":"2023-10-10T19:29:53.644211+0000","objects_scrubbed":30,"log_size":1009,"log_dups_size":0,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:15:02.264513+0000","scrub_duration":0.035309926999999998,"objects_trimmed":0,"snaptrim_duration":0.0068654069999999996,"stat_sum":{"num_bytes":747011,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":59,"num_read_kb":41116,"num_write":675,"num_write_kb":48421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"132'1014","reported_seq":3648,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:08.003144+0000","last_change":"2023-10-10T19:30:08.003117+0000","last_active":"2023-10-10T19:30:08.003144+0000","last_peered":"2023-10-10T19:30:08.003144+0000","last_clean":"2023-10-10T19:30:08.003144+0000","last_became_active":"2023-10-10T18:45:11.427548+0000","last_became_peered":"2023-10-10T18:45:11.427548+0000","last_unstale":"2023-10-10T19:30:08.003144+0000","last_undegraded":"2023-10-10T19:30:08.003144+0000","last_fullsized":"2023-10-10T19:30:08.003144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1014","last_scrub_stamp":"2023-10-10T19:30:08.003025+0000","last_deep_scrub":"132'1014","last_deep_scrub_stamp":"2023-10-10T19:30:08.003025+0000","last_clean_scrub_stamp":"2023-10-10T19:30:08.003025+0000","objects_scrubbed":36,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:14:37.238471+0000","scrub_duration":0.041397751000000003,"objects_trimmed":0,"snaptrim_duration":0.0081260870000000006,"stat_sum":{"num_bytes":4820268,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":70,"num_read_kb":15871,"num_write":698,"num_write_kb":28703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"132'968","reported_seq":3616,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:01.048137+0000","last_change":"2023-10-10T19:30:01.048111+0000","last_active":"2023-10-10T19:30:01.048137+0000","last_peered":"2023-10-10T19:30:01.048137+0000","last_clean":"2023-10-10T19:30:01.048137+0000","last_became_active":"2023-10-10T18:45:11.425190+0000","last_became_peered":"2023-10-10T18:45:11.425190+0000","last_unstale":"2023-10-10T19:30:01.048137+0000","last_undegraded":"2023-10-10T19:30:01.048137+0000","last_fullsized":"2023-10-10T19:30:01.048137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'968","last_scrub_stamp":"2023-10-10T19:30:01.048031+0000","last_deep_scrub":"132'968","last_deep_scrub_stamp":"2023-10-10T19:30:01.048031+0000","last_clean_scrub_stamp":"2023-10-10T19:30:01.048031+0000","objects_scrubbed":35,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:51:18.799692+0000","scrub_duration":0.030600175,"objects_trimmed":0,"snaptrim_duration":0.017981724000000001,"stat_sum":{"num_bytes":834726,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":72,"num_read_kb":44513,"num_write":669,"num_write_kb":47263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"132'945","reported_seq":3600,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:54.601048+0000","last_change":"2023-10-10T19:29:54.600997+0000","last_active":"2023-10-10T19:29:54.601048+0000","last_peered":"2023-10-10T19:29:54.601048+0000","last_clean":"2023-10-10T19:29:54.601048+0000","last_became_active":"2023-10-10T18:45:11.421430+0000","last_became_peered":"2023-10-10T18:45:11.421430+0000","last_unstale":"2023-10-10T19:29:54.601048+0000","last_undegraded":"2023-10-10T19:29:54.601048+0000","last_fullsized":"2023-10-10T19:29:54.601048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'945","last_scrub_stamp":"2023-10-10T19:29:54.600850+0000","last_deep_scrub":"132'945","last_deep_scrub_stamp":"2023-10-10T19:29:54.600850+0000","last_clean_scrub_stamp":"2023-10-10T19:29:54.600850+0000","objects_scrubbed":34,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:46:40.344869+0000","scrub_duration":0.031231462000000001,"objects_trimmed":0,"snaptrim_duration":0.010756557999999999,"stat_sum":{"num_bytes":807796,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":71,"num_read_kb":21097,"num_write":685,"num_write_kb":25593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"132'924","reported_seq":3573,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:52.038445+0000","last_change":"2023-10-10T19:29:52.038418+0000","last_active":"2023-10-10T19:29:52.038445+0000","last_peered":"2023-10-10T19:29:52.038445+0000","last_clean":"2023-10-10T19:29:52.038445+0000","last_became_active":"2023-10-10T18:45:11.428558+0000","last_became_peered":"2023-10-10T18:45:11.428558+0000","last_unstale":"2023-10-10T19:29:52.038445+0000","last_undegraded":"2023-10-10T19:29:52.038445+0000","last_fullsized":"2023-10-10T19:29:52.038445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'924","last_scrub_stamp":"2023-10-10T19:29:52.038297+0000","last_deep_scrub":"132'924","last_deep_scrub_stamp":"2023-10-10T19:29:52.038297+0000","last_clean_scrub_stamp":"2023-10-10T19:29:52.038297+0000","objects_scrubbed":28,"log_size":924,"log_dups_size":0,"ondisk_log_size":924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:34:00.960972+0000","scrub_duration":0.031392050999999997,"objects_trimmed":0,"snaptrim_duration":0.014012482999999999,"stat_sum":{"num_bytes":684647,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":69,"num_read_kb":49258,"num_write":638,"num_write_kb":56905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"132'993","reported_seq":3638,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:07.632893+0000","last_change":"2023-10-10T19:30:07.632853+0000","last_active":"2023-10-10T19:30:07.632893+0000","last_peered":"2023-10-10T19:30:07.632893+0000","last_clean":"2023-10-10T19:30:07.632893+0000","last_became_active":"2023-10-10T18:45:11.426656+0000","last_became_peered":"2023-10-10T18:45:11.426656+0000","last_unstale":"2023-10-10T19:30:07.632893+0000","last_undegraded":"2023-10-10T19:30:07.632893+0000","last_fullsized":"2023-10-10T19:30:07.632893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'993","last_scrub_stamp":"2023-10-10T19:30:07.632760+0000","last_deep_scrub":"132'993","last_deep_scrub_stamp":"2023-10-10T19:30:07.632760+0000","last_clean_scrub_stamp":"2023-10-10T19:30:07.632760+0000","objects_scrubbed":34,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T06:20:31.200915+0000","scrub_duration":0.035667892,"objects_trimmed":0,"snaptrim_duration":0.0097177140000000006,"stat_sum":{"num_bytes":571513,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":59,"num_read_kb":20609,"num_write":714,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"132'927","reported_seq":3598,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:30:00.406510+0000","last_change":"2023-10-10T19:30:00.406485+0000","last_active":"2023-10-10T19:30:00.406510+0000","last_peered":"2023-10-10T19:30:00.406510+0000","last_clean":"2023-10-10T19:30:00.406510+0000","last_became_active":"2023-10-10T18:45:11.426542+0000","last_became_peered":"2023-10-10T18:45:11.426542+0000","last_unstale":"2023-10-10T19:30:00.406510+0000","last_undegraded":"2023-10-10T19:30:00.406510+0000","last_fullsized":"2023-10-10T19:30:00.406510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'927","last_scrub_stamp":"2023-10-10T19:30:00.406402+0000","last_deep_scrub":"132'927","last_deep_scrub_stamp":"2023-10-10T19:30:00.406402+0000","last_clean_scrub_stamp":"2023-10-10T19:30:00.406402+0000","objects_scrubbed":40,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:36:31.446310+0000","scrub_duration":0.034815272000000001,"objects_trimmed":0,"snaptrim_duration":0.013935637000000001,"stat_sum":{"num_bytes":366510,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":77,"num_read_kb":51992,"num_write":664,"num_write_kb":56578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"132'1043","reported_seq":3697,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-10T19:29:50.062490+0000","last_change":"2023-10-10T19:29:50.062453+0000","last_active":"2023-10-10T19:29:50.062490+0000","last_peered":"2023-10-10T19:29:50.062490+0000","last_clean":"2023-10-10T19:29:50.062490+0000","last_became_active":"2023-10-10T18:45:11.429346+0000","last_became_peered":"2023-10-10T18:45:11.429346+0000","last_unstale":"2023-10-10T19:29:50.062490+0000","last_undegraded":"2023-10-10T19:29:50.062490+0000","last_fullsized":"2023-10-10T19:29:50.062490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1043","last_scrub_stamp":"2023-10-10T19:29:50.062332+0000","last_deep_scrub":"132'1043","last_deep_scrub_stamp":"2023-10-10T19:29:50.062332+0000","last_clean_scrub_stamp":"2023-10-10T19:29:50.062332+0000","objects_scrubbed":42,"log_size":1043,"log_dups_size":0,"ondisk_log_size":1043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:41:49.718435+0000","scrub_duration":0.035570905999999999,"objects_trimmed":0,"snaptrim_duration":0.0088155260000000006,"stat_sum":{"num_bytes":278227,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":77,"num_read_kb":37312,"num_write":744,"num_write_kb":49182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":50927477,"num_objects":2366,"num_object_clones":0,"num_object_copies":4732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":4408,"num_read_kb":1953419,"num_write":44061,"num_write_kb":2517898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":76832768,"data_stored":101854954,"data_compressed":30508016,"data_compressed_allocated":33341440,"data_compressed_original":70082560,"omap_allocated":0,"internal_metadata":0},"log_size":62851,"ondisk_log_size":62851,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64528,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":336884642,"num_objects":493,"num_object_clones":0,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":2858,"num_read_kb":1531013,"num_write":19196,"num_write_kb":469942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":353,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2518057,"num_omap_keys":4386,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183304192,"data_stored":673769284,"data_compressed":87276636,"data_compressed_allocated":112345088,"data_compressed_original":603062272,"omap_allocated":0,"internal_metadata":0},"log_size":20286,"ondisk_log_size":20286,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575397,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348880,"kb_used_data":47496,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94022960,"statfs":{"total":96636764160,"available":96279511040,"internally_reserved":0,"allocated":48635904,"data_stored":134725171,"data_compressed":20528468,"data_compressed_allocated":25550848,"data_compressed_original":114077696,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51800000000000002,"15min":0.54000000000000004},"min":{"1min":0.32700000000000001,"5min":0.28499999999999998,"15min":0.27400000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":1.018},"last":0.629},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.70699999999999996,"5min":0.80700000000000005,"15min":2.153},"last":0.57499999999999996}]},{"osd":1,"last update":"Tue Oct 10 19:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60199999999999998,"15min":0.58599999999999997},"min":{"1min":0.42099999999999999,"5min":0.318,"15min":0.314},"max":{"1min":0.77300000000000002,"5min":0.86799999999999999,"15min":1.802},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57799999999999996,"15min":0.59699999999999998},"min":{"1min":0.34699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68899999999999995,"5min":0.82399999999999995,"15min":2.2170000000000001},"last":0.61199999999999999}]},{"osd":2,"last update":"Tue Oct 10 19:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63,"15min":0.61899999999999999},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.76200000000000001,"5min":2.871,"15min":2.871},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64300000000000002,"15min":0.622},"min":{"1min":0.45000000000000001,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.93000000000000005,"15min":2.1989999999999998},"last":0.64600000000000002}]},{"osd":3,"last update":"Tue Oct 10 19:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.34699999999999998,"15min":0.317},"max":{"1min":0.67300000000000004,"5min":0.874,"15min":1.0349999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.439,"5min":0.33400000000000002,"15min":0.30499999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.7649999999999999},"last":0.66500000000000004}]},{"osd":4,"last update":"Tue Oct 10 19:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48299999999999998,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.28000000000000003},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.089},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55600000000000005,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":1.9419999999999999},"last":0.46999999999999997}]},{"osd":5,"last update":"Tue Oct 10 19:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.501,"15min":0.51400000000000001},"min":{"1min":0.34200000000000003,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":0.69199999999999995,"5min":0.81799999999999995,"15min":1.71},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.57499999999999996},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.311},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":1.177},"last":0.59099999999999997}]},{"osd":6,"last update":"Tue Oct 10 19:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.31},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56499999999999995,"15min":0.57599999999999996},"min":{"1min":0.41899999999999998,"5min":0.33900000000000002,"15min":0.29699999999999999},"max":{"1min":0.76400000000000001,"5min":0.84499999999999997,"15min":2.2559999999999998},"last":0.51900000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575396,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337504,"kb_used_data":36120,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94034336,"statfs":{"total":96636764160,"available":96291160064,"internally_reserved":0,"allocated":36986880,"data_stored":92212231,"data_compressed":11950270,"data_compressed_allocated":14790656,"data_compressed_original":71962624,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.80600000000000005,"15min":0.81100000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.216},"max":{"1min":0.68300000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51100000000000001}]},{"osd":1,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58699999999999997,"15min":0.56000000000000005},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.85299999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.56499999999999995},"min":{"1min":0.41799999999999998,"5min":0.379,"15min":0.222},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.86799999999999999},"last":0.56599999999999995}]},{"osd":2,"last update":"Tue Oct 10 19:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.56200000000000006},"min":{"1min":0.48699999999999999,"5min":0.40400000000000003,"15min":0.30499999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90500000000000003},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.55300000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.29999999999999999},"max":{"1min":0.67500000000000004,"5min":0.76500000000000001,"15min":0.97599999999999998},"last":0.57899999999999996}]},{"osd":3,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57999999999999996,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.32900000000000001},"max":{"1min":0.628,"5min":0.82799999999999996,"15min":0.88200000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.437,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.85699999999999998,"15min":0.95099999999999996},"last":0.82499999999999996}]},{"osd":4,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48199999999999998,"15min":0.46700000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.248},"max":{"1min":0.749,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54900000000000004,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.80300000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68600000000000005}]},{"osd":5,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73599999999999999,"15min":0.879},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51900000000000002,"15min":0.53400000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.26400000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.99299999999999999},"last":0.79100000000000004}]},{"osd":7,"last update":"Tue Oct 10 19:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.31},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.88900000000000001},"last":0.75},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.38,"15min":0.27700000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.86199999999999999},"last":0.72199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575395,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342436,"kb_used_data":32860,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94029404,"statfs":{"total":96636764160,"available":96286109696,"internally_reserved":0,"allocated":33648640,"data_stored":116978587,"data_compressed":17811264,"data_compressed_allocated":21970944,"data_compressed_original":107048960,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.31,"15min":0.315},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17699999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.56899999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41999999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":0.66700000000000004,"15min":0.70799999999999996},"last":0.53900000000000003}]},{"osd":2,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.34899999999999998,"15min":0.35399999999999998},"min":{"1min":0.246,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.55200000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38800000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.19900000000000001},"max":{"1min":0.56699999999999995,"5min":0.64900000000000002,"15min":0.68899999999999995},"last":0.51400000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56299999999999994,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.27700000000000002},"max":{"1min":0.66700000000000004,"5min":0.72899999999999998,"15min":0.84299999999999997},"last":0.58699999999999997}]},{"osd":4,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.41999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.175},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.875,"5min":0.875,"15min":2.2519999999999998},"last":0.65000000000000002}]},{"osd":5,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.90900000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.501,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.628}]},{"osd":6,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.46100000000000002,"5min":0.40500000000000003,"15min":0.37},"max":{"1min":0.81200000000000006,"5min":0.86199999999999999,"15min":1.038},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.38900000000000001,"15min":0.36299999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.123},"last":0.55500000000000005}]},{"osd":7,"last update":"Tue Oct 10 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.621},"min":{"1min":0.46700000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":5.2720000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61899999999999999,"15min":0.64800000000000002},"min":{"1min":0.47899999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":5.2599999999999998},"last":0.57699999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575397,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338068,"kb_used_data":32588,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94033772,"statfs":{"total":96636764160,"available":96290582528,"internally_reserved":0,"allocated":33370112,"data_stored":88805479,"data_compressed":14490668,"data_compressed_allocated":17825792,"data_compressed_original":75632640,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.66500000000000004},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42399999999999999,"15min":0.41699999999999998},"min":{"1min":0.307,"5min":0.25800000000000001,"15min":0.20300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66000000000000003,"15min":0.66500000000000004},"last":0.42699999999999999}]},{"osd":2,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35999999999999999,"15min":0.372},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.52000000000000002,"5min":0.52500000000000002,"15min":2.028},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40799999999999997},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53400000000000003,"5min":0.61299999999999999,"15min":0.77300000000000002},"last":0.38400000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.39600000000000002},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.79100000000000004},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.31,"15min":0.27800000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.80300000000000005},"last":0.40100000000000002}]},{"osd":4,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.629,"15min":0.63100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.26000000000000001},"max":{"1min":0.84399999999999997,"5min":0.97599999999999998,"15min":1.9610000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.38400000000000001},"max":{"1min":0.877,"5min":0.999,"15min":0.999},"last":0.69799999999999995}]},{"osd":5,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.35299999999999998},"max":{"1min":0.81799999999999995,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.622},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72699999999999998,"5min":0.873,"15min":0.96799999999999997},"last":0.71399999999999997}]},{"osd":6,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":0.85699999999999998,"5min":0.96299999999999997,"15min":2.0019999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.627,"15min":0.62},"min":{"1min":0.43099999999999999,"5min":0.39700000000000002,"15min":0.38},"max":{"1min":0.73599999999999999,"5min":0.94099999999999995,"15min":0.95399999999999996},"last":0.63600000000000001}]},{"osd":7,"last update":"Tue Oct 10 19:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.63100000000000001,"15min":0.63400000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.63700000000000001,"15min":0.64900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.80100000000000005,"5min":0.89500000000000002,"15min":2.0539999999999998},"last":0.73299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575397,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319288,"kb_used_data":17904,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94052552,"statfs":{"total":96636764160,"available":96309813248,"internally_reserved":0,"allocated":18333696,"data_stored":57320502,"data_compressed":8516380,"data_compressed_allocated":10579968,"data_compressed_original":51261440,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.33900000000000002},"min":{"1min":0.254,"5min":0.219,"15min":0.19700000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42399999999999999,"15min":0.40400000000000003},"min":{"1min":0.28899999999999998,"5min":0.215,"15min":0.215},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.32200000000000001}]},{"osd":1,"last update":"Tue Oct 10 19:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.378,"15min":0.36299999999999999},"min":{"1min":0.32500000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.45900000000000002,"15min":0.44400000000000001},"min":{"1min":0.40799999999999997,"5min":0.27100000000000002,"15min":0.26600000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.38100000000000001}]},{"osd":3,"last update":"Tue Oct 10 19:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41499999999999998,"15min":0.41699999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.24399999999999999},"max":{"1min":0.60599999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.42799999999999999},"min":{"1min":0.33600000000000002,"5min":0.30199999999999999,"15min":0.23899999999999999},"max":{"1min":0.62,"5min":0.67900000000000005,"15min":0.70099999999999996},"last":0.36699999999999999}]},{"osd":4,"last update":"Tue Oct 10 19:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59999999999999998,"15min":0.77200000000000002},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.35899999999999999},"max":{"1min":0.82899999999999996,"5min":0.95599999999999996,"15min":48.729999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60799999999999998,"15min":0.60099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84699999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.60499999999999998}]},{"osd":5,"last update":"Tue Oct 10 19:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63600000000000001,"15min":0.63100000000000001},"min":{"1min":0.47999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89800000000000002,"5min":1.02,"15min":1.02},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65500000000000003,"15min":0.64100000000000001},"min":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.36399999999999999},"max":{"1min":0.88,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.503}]},{"osd":6,"last update":"Tue Oct 10 19:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.622,"15min":0.78900000000000003},"min":{"1min":0.42899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.91000000000000003,"5min":1.0760000000000001,"15min":48.685000000000002},"last":0.628},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.627,"15min":0.61099999999999999},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.38800000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.97899999999999998},"last":0.53100000000000003}]},{"osd":7,"last update":"Tue Oct 10 19:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":0.80900000000000005},"min":{"1min":0.46600000000000003,"5min":0.42699999999999999,"15min":0.38700000000000001},"max":{"1min":0.94099999999999995,"5min":1.052,"15min":48.710000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.65500000000000003},"min":{"1min":0.45000000000000001,"5min":0.443,"15min":0.42499999999999999},"max":{"1min":0.89900000000000002,"5min":1.034,"15min":1.034},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575396,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335672,"kb_used_data":38384,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94036168,"statfs":{"total":96636764160,"available":96293036032,"internally_reserved":0,"allocated":39305216,"data_stored":100336269,"data_compressed":14072432,"data_compressed_allocated":17829888,"data_compressed_original":81338368,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:55 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35099999999999998,"15min":0.37},"min":{"1min":0.21099999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.58799999999999997,"5min":0.64900000000000002,"15min":5.0670000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.374,"15min":0.39000000000000001},"min":{"1min":0.223,"5min":0.20899999999999999,"15min":0.189},"max":{"1min":0.49099999999999999,"5min":0.59899999999999998,"15min":2.7799999999999998},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.373,"15min":0.379},"min":{"1min":0.25900000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.54600000000000004,"5min":0.628,"15min":2.758},"last":0.498},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39300000000000002,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.64500000000000002},"last":0.47999999999999998}]},{"osd":2,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.25},"max":{"1min":0.60099999999999998,"5min":0.68400000000000005,"15min":1.333},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.41999999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.24199999999999999},"max":{"1min":0.56100000000000005,"5min":0.65400000000000003,"15min":0.98699999999999999},"last":0.53300000000000003}]},{"osd":4,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.623,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79600000000000004,"5min":0.89600000000000002,"15min":0.95499999999999996},"last":0.872},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63400000000000001,"15min":0.63700000000000001},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":1.0169999999999999,"15min":1.23},"last":0.79000000000000004}]},{"osd":5,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.84199999999999997,"5min":0.97699999999999998,"15min":1.2929999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.44800000000000001,"5min":0.40500000000000003,"15min":0.36199999999999999},"max":{"1min":0.875,"5min":0.93600000000000005,"15min":1.103},"last":0.69099999999999995}]},{"osd":6,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.64100000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40300000000000002},"max":{"1min":0.90600000000000003,"5min":0.94199999999999995,"15min":1.198},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.88500000000000001,"5min":0.94899999999999995,"15min":2.7309999999999999},"last":0.80000000000000004}]},{"osd":7,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.39400000000000002},"max":{"1min":0.85699999999999998,"5min":0.97999999999999998,"15min":2.6619999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.49299999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86399999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.85099999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575396,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344800,"kb_used_data":35224,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94027040,"statfs":{"total":96636764160,"available":96283688960,"internally_reserved":0,"allocated":36069376,"data_stored":105586308,"data_compressed":16193482,"data_compressed_allocated":19931136,"data_compressed_original":91820032,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33400000000000002,"5min":0.27600000000000002,"15min":0.21199999999999999},"max":{"1min":0.69599999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59899999999999998,"15min":0.58299999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.23699999999999999},"max":{"1min":0.74399999999999999,"5min":0.82899999999999996,"15min":0.86799999999999999},"last":0.77800000000000002}]},{"osd":1,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.35599999999999998,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":0.67600000000000005,"5min":0.71199999999999997,"15min":1.5920000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"max":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":1.0169999999999999},"last":0.71499999999999997}]},{"osd":2,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.42599999999999999,"5min":0.41999999999999998,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":1.615},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.40100000000000002,"15min":0.36699999999999999},"max":{"1min":0.77800000000000002,"5min":0.85799999999999998,"15min":0.98199999999999998},"last":0.80500000000000005}]},{"osd":3,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.57999999999999996},"min":{"1min":0.376,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.746,"5min":0.85199999999999998,"15min":0.85799999999999998},"last":0.627},{"interface":"front","average":{"1min":0.626,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.28199999999999997},"max":{"1min":0.78100000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.66700000000000004}]},{"osd":5,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.47099999999999997},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.26300000000000001},"max":{"1min":0.71599999999999997,"5min":0.84199999999999997,"15min":1.087},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.82399999999999995,"5min":0.88,"15min":1.111},"last":0.65300000000000002}]},{"osd":6,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.29799999999999999},"max":{"1min":0.82599999999999996,"5min":0.83099999999999996,"15min":1.5580000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.98799999999999999},"last":0.55200000000000005}]},{"osd":7,"last update":"Tue Oct 10 19:29:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.45200000000000001,"5min":0.37,"15min":0.312},"max":{"1min":0.86099999999999999,"5min":1.016,"15min":1.0620000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.64200000000000002},"min":{"1min":0.503,"5min":0.40000000000000002,"15min":0.33700000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":1.1160000000000001},"last":0.68799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834575396,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331424,"kb_used_data":25944,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94040416,"statfs":{"total":96636764160,"available":96297385984,"internally_reserved":0,"allocated":26566656,"data_stored":87927915,"data_compressed":14286216,"data_compressed_allocated":17297408,"data_compressed_original":81149952,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.249},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.82199999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.752,"5min":0.76700000000000002,"15min":0.85099999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58499999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.314},"max":{"1min":0.78600000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.57499999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.28100000000000003},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81899999999999995},"last":0.76200000000000001}]},{"osd":2,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.54200000000000004},"min":{"1min":0.442,"5min":0.35499999999999998,"15min":0.30499999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.28100000000000003},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.77900000000000003}]},{"osd":3,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57199999999999995,"15min":0.55500000000000005},"min":{"1min":0.39600000000000002,"5min":0.377,"15min":0.32600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.76600000000000001,"5min":0.81799999999999995,"15min":0.94899999999999995},"last":0.69099999999999995}]},{"osd":4,"last update":"Tue Oct 10 19:29:47 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.45800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57699999999999996,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.23000000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.47099999999999997}]},{"osd":6,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.502,"15min":0.496},"min":{"1min":0.316,"5min":0.316,"15min":0.26700000000000002},"max":{"1min":0.75800000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.497},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.77800000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.42399999999999999}]},{"osd":7,"last update":"Tue Oct 10 19:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79600000000000004,"5min":0.80000000000000004,"15min":0.89400000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61099999999999999,"15min":0.60199999999999998},"min":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.27700000000000002},"max":{"1min":0.82199999999999995,"5min":0.82599999999999996,"15min":0.90800000000000003},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32264,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20922368,"data_stored":71948432,"data_compressed":9641340,"data_compressed_allocated":12496896,"data_compressed_original":63553536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25841664,"data_stored":106226300,"data_compressed":13852450,"data_compressed_allocated":17678336,"data_compressed_original":98091008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":51052075,"data_compressed":6450195,"data_compressed_allocated":8323072,"data_compressed_original":46788608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26312704,"data_stored":85394164,"data_compressed":10438632,"data_compressed_allocated":13819904,"data_compressed_original":72937472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24768512,"data_stored":91918527,"data_compressed":12211424,"data_compressed_allocated":15597568,"data_compressed_original":82771968,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16703488,"data_stored":74680123,"data_compressed":9765423,"data_compressed_allocated":12320768,"data_compressed_original":70340608,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":22110208,"data_stored":76551778,"data_compressed":9536145,"data_compressed_allocated":12132352,"data_compressed_original":66600960,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":34029568,"data_stored":115997885,"data_compressed":15381027,"data_compressed_allocated":19976192,"data_compressed_original":101978112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10801152,"data_stored":15380743,"data_compressed":4817064,"data_compressed_allocated":5283840,"data_compressed_original":11505664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6225920,"data_stored":9866351,"data_compressed":3958814,"data_compressed_allocated":4292608,"data_compressed_original":8957952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4136960,"data_stored":5382491,"data_compressed":2066185,"data_compressed_allocated":2256896,"data_compressed_original":4472832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11411456,"data_stored":14056169,"data_compressed":3633800,"data_compressed_allocated":4009984,"data_compressed_original":8400896,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9719808,"data_stored":12781845,"data_compressed":3982058,"data_compressed_allocated":4333568,"data_compressed_original":9048064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8282112,"data_stored":12361856,"data_compressed":4520793,"data_compressed_allocated":4976640,"data_compressed_original":10809344,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13295616,"data_stored":14774517,"data_compressed":2414125,"data_compressed_allocated":2658304,"data_compressed_original":5361664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12959744,"data_stored":17250982,"data_compressed":5115177,"data_compressed_allocated":5529600,"data_compressed_original":11526144,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T19:30:30.762 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-10T19:30:30.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.906+0000 7f18be7aa700 1 Processor -- start 2023-10-10T19:30:30.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.906+0000 7f18be7aa700 1 -- start start 2023-10-10T19:30:30.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18be7aa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b80c39d0 0x7f18b80c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18be7aa700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18b8068a00 0x7f18b8068e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18be7aa700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b8156dd0 con 0x7f18b8068630 2023-10-10T19:30:30.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18be7aa700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b8156620 con 0x7f18b8068a00 2023-10-10T19:30:30.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18be7aa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b80c3e00 con 0x7f18b80c39d0 2023-10-10T19:30:30.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bdfa9700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f18b8068a00 0x7f18b8068e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bd7a8700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f18b8068630 0x7f18b8155f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52090/0 (socket says 172.21.15.175:52090) 2023-10-10T19:30:30.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bd7a8700 1 -- 172.21.15.175:0/1298854827 learned_addr learned my addr 172.21.15.175:0/1298854827 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:30:30.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bcfa7700 1 --2- 172.21.15.175:0/1298854827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b80c39d0 0x7f18b80c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bdfa9700 1 -- 172.21.15.175:0/1298854827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b80c39d0 msgr2=0x7f18b80c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:30.912 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bdfa9700 1 --2- 172.21.15.175:0/1298854827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b80c39d0 0x7f18b80c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bdfa9700 1 -- 172.21.15.175:0/1298854827 >> v1:172.21.15.175:6789/0 conn(0x7f18b8068630 legacy=0x7f18b8155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18affff700 1 -- 172.21.15.175:0/1298854827 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102074850 0 0) 0x7f18b8156dd0 con 0x7f18b8068630 2023-10-10T19:30:30.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bdfa9700 1 -- 172.21.15.175:0/1298854827 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b80c7650 con 0x7f18b8068a00 2023-10-10T19:30:30.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18bdfa9700 1 --2- 172.21.15.175:0/1298854827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b8068a00 0x7f18b8068e30 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009e60 tx=0x7f18b0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2bfbb4a2845b585 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:30.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.907+0000 7f18affff700 1 -- 172.21.15.175:0/1298854827 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b000e070 con 0x7f18b8068a00 2023-10-10T19:30:30.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18affff700 1 -- 172.21.15.175:0/1298854827 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b000bc30 con 0x7f18b8068a00 2023-10-10T19:30:30.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18affff700 1 -- 172.21.15.175:0/1298854827 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0012430 con 0x7f18b8068a00 2023-10-10T19:30:30.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18be7aa700 1 -- 172.21.15.175:0/1298854827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b8068a00 msgr2=0x7f18b8068e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18be7aa700 1 --2- 172.21.15.175:0/1298854827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b8068a00 0x7f18b8068e30 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009e60 tx=0x7f18b0009780 comp rx=0 tx=0).stop 2023-10-10T19:30:30.915 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18be7aa700 1 -- 172.21.15.175:0/1298854827 shutdown_connections 2023-10-10T19:30:30.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18be7aa700 1 --2- 172.21.15.175:0/1298854827 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b8068a00 0x7f18b8068e30 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18be7aa700 1 --2- 172.21.15.175:0/1298854827 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b80c39d0 0x7f18b80c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18be7aa700 1 -- 172.21.15.175:0/1298854827 >> 172.21.15.175:0/1298854827 conn(0x7f18b8063230 msgr2=0x7f18b8063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:30.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18be7aa700 1 -- 172.21.15.175:0/1298854827 shutdown_connections 2023-10-10T19:30:30.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.908+0000 7f18be7aa700 1 -- 172.21.15.175:0/1298854827 wait complete. 2023-10-10T19:30:30.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.909+0000 7f18be7aa700 1 Processor -- start 2023-10-10T19:30:30.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.909+0000 7f18be7aa700 1 -- start start 2023-10-10T19:30:30.920 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18be7aa700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b8068630 0x7f18b8166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18be7aa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b8068a00 0x7f18b8167620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18be7aa700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b80c39d0 0x7f18b814fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18be7aa700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8156070 con 0x7f18b80c39d0 2023-10-10T19:30:30.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18be7aa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b81502f0 con 0x7f18b8068a00 2023-10-10T19:30:30.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18be7aa700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8153220 con 0x7f18b8068630 2023-10-10T19:30:30.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bdfa9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b80c39d0 0x7f18b814fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bdfa9700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b80c39d0 0x7f18b814fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60144/0 (socket says 172.21.15.175:60144) 2023-10-10T19:30:30.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bdfa9700 1 -- 172.21.15.175:0/3807688607 learned_addr learned my addr 172.21.15.175:0/3807688607 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:30.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bd7a8700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b8068630 0x7f18b8166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.923 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bcfa7700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b8068a00 0x7f18b8167620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bdfa9700 1 -- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b8068630 msgr2=0x7f18b8166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bdfa9700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b8068630 0x7f18b8166cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bdfa9700 1 -- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b8068a00 msgr2=0x7f18b8167620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:30.924 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bdfa9700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b8068a00 0x7f18b8167620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:30.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bdfa9700 1 -- 172.21.15.175:0/3807688607 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b0009a20 con 0x7f18b80c39d0 2023-10-10T19:30:30.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.910+0000 7f18bd7a8700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b8068630 0x7f18b8166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:30:30.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.911+0000 7f18bdfa9700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b80c39d0 0x7f18b814fdb0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f18b8089810 tx=0x7f18b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:30.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.911+0000 7f18ad7fa700 1 -- 172.21.15.175:0/3807688607 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0012ea0 con 0x7f18b80c39d0 2023-10-10T19:30:30.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.911+0000 7f18ad7fa700 1 -- 172.21.15.175:0/3807688607 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b001d630 con 0x7f18b80c39d0 2023-10-10T19:30:30.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.911+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b80c22d0 con 0x7f18b80c39d0 2023-10-10T19:30:30.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.911+0000 7f18ad7fa700 1 -- 172.21.15.175:0/3807688607 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b001d9f0 con 0x7f18b80c39d0 2023-10-10T19:30:30.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.911+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b8153710 con 0x7f18b80c39d0 2023-10-10T19:30:30.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.912+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b80715d0 con 0x7f18b80c39d0 2023-10-10T19:30:30.927 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.912+0000 7f18ad7fa700 1 -- 172.21.15.175:0/3807688607 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f18b0024070 con 0x7f18b80c39d0 2023-10-10T19:30:30.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.912+0000 7f18ad7fa700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f189c066f60 0x7f189c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:30.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.912+0000 7f18ad7fa700 1 -- 172.21.15.175:0/3807688607 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f18b0087720 con 0x7f18b80c39d0 2023-10-10T19:30:30.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.913+0000 7f18bd7a8700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f189c066f60 0x7f189c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:30.928 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.913+0000 7f18bd7a8700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f189c066f60 0x7f189c069410 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f18b40018d0 tx=0x7f18b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:30.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:30.917+0000 7f18ad7fa700 1 -- 172.21.15.175:0/3807688607 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f18b0054110 con 0x7f18b80c39d0 2023-10-10T19:30:31.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.100+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f18b8068ff0 con 0x7f18b80c39d0 2023-10-10T19:30:31.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.103+0000 7f18ad7fa700 1 -- 172.21.15.175:0/3807688607 <== mon.0 v2:172.21.15.175:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f18b0054110 con 0x7f18b80c39d0 2023-10-10T19:30:31.104 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.104+0000 7f18ad7fa700 1 -- 172.21.15.175:0/3807688607 <== mon.0 v2:172.21.15.175:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7f18b0057730 con 0x7f18b80c39d0 2023-10-10T19:30:31.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f189c066f60 msgr2=0x7f189c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f189c066f60 0x7f189c069410 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f18b40018d0 tx=0x7f18b4009040 comp rx=0 tx=0).stop 2023-10-10T19:30:31.110 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b80c39d0 msgr2=0x7f18b814fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b80c39d0 0x7f18b814fdb0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f18b8089810 tx=0x7f18b0019000 comp rx=0 tx=0).stop 2023-10-10T19:30:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 shutdown_connections 2023-10-10T19:30:31.111 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f189c066f60 0x7f189c069410 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18b80c39d0 0x7f18b814fdb0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b8068630 0x7f18b8166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.112 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 --2- 172.21.15.175:0/3807688607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b8068a00 0x7f18b8167620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 >> 172.21.15.175:0/3807688607 conn(0x7f18b8063230 msgr2=0x7f18b815bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 shutdown_connections 2023-10-10T19:30:31.113 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.106+0000 7f18be7aa700 1 -- 172.21.15.175:0/3807688607 wait complete. 2023-10-10T19:30:31.119 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-10T19:30:31.119 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:30:31.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.234+0000 7f2555563700 1 Processor -- start 2023-10-10T19:30:31.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.234+0000 7f2555563700 1 -- start start 2023-10-10T19:30:31.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2555563700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255006f950 0x7f2550063580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2555563700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2550159db0 0x7f25500c4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2555563700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25500c2e70 con 0x7f25501599e0 2023-10-10T19:30:31.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2555563700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25500632a0 con 0x7f2550159db0 2023-10-10T19:30:31.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2555563700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25500c4d50 con 0x7f255006f950 2023-10-10T19:30:31.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2554d62700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2550159db0 0x7f25500c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2554d62700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2550159db0 0x7f25500c4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60158/0 (socket says 172.21.15.175:60158) 2023-10-10T19:30:31.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2554d62700 1 -- 172.21.15.175:0/3135938719 learned_addr learned my addr 172.21.15.175:0/3135938719 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:31.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f254f7fe700 1 --2- 172.21.15.175:0/3135938719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255006f950 0x7f2550063580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2554d62700 1 -- 172.21.15.175:0/3135938719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255006f950 msgr2=0x7f2550063580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:31.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2554d62700 1 --2- 172.21.15.175:0/3135938719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255006f950 0x7f2550063580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2554d62700 1 -- 172.21.15.175:0/3135938719 >> v1:172.21.15.191:6789/0 conn(0x7f25501599e0 legacy=0x7f255006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:31.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2554d62700 1 -- 172.21.15.175:0/3135938719 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25500c5030 con 0x7f2550159db0 2023-10-10T19:30:31.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.235+0000 7f2554d62700 1 --2- 172.21.15.175:0/3135938719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f25500c4130 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f2548009f20 tx=0x7f2548009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a4700826e0486a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f254effd700 1 -- 172.21.15.175:0/3135938719 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254800e070 con 0x7f2550159db0 2023-10-10T19:30:31.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f254effd700 1 -- 172.21.15.175:0/3135938719 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f254800ba80 con 0x7f2550159db0 2023-10-10T19:30:31.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f254effd700 1 -- 172.21.15.175:0/3135938719 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25480092e0 con 0x7f2550159db0 2023-10-10T19:30:31.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f2555563700 1 -- 172.21.15.175:0/3135938719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 msgr2=0x7f25500c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f2555563700 1 --2- 172.21.15.175:0/3135938719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f25500c4130 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f2548009f20 tx=0x7f2548009840 comp rx=0 tx=0).stop 2023-10-10T19:30:31.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f2555563700 1 -- 172.21.15.175:0/3135938719 shutdown_connections 2023-10-10T19:30:31.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f2555563700 1 --2- 172.21.15.175:0/3135938719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f25500c4130 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f2555563700 1 --2- 172.21.15.175:0/3135938719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f255006f950 0x7f2550063580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f2555563700 1 -- 172.21.15.175:0/3135938719 >> 172.21.15.175:0/3135938719 conn(0x7f25500c5510 msgr2=0x7f25500c5910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:31.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f2555563700 1 -- 172.21.15.175:0/3135938719 shutdown_connections 2023-10-10T19:30:31.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.236+0000 7f2555563700 1 -- 172.21.15.175:0/3135938719 wait complete. 2023-10-10T19:30:31.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.237+0000 7f2555563700 1 Processor -- start 2023-10-10T19:30:31.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.237+0000 7f2555563700 1 -- start start 2023-10-10T19:30:31.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2555563700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f255006f950 0x7f25500c16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2555563700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25501599e0 0x7f255014e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2555563700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f2550168f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2555563700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2550165e00 con 0x7f2550159db0 2023-10-10T19:30:31.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2555563700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25500c1240 con 0x7f255006f950 2023-10-10T19:30:31.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2555563700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25500c49e0 con 0x7f25501599e0 2023-10-10T19:30:31.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2554d62700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f2550168f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2554d62700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f2550168f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60168/0 (socket says 172.21.15.175:60168) 2023-10-10T19:30:31.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f2554d62700 1 -- 172.21.15.175:0/1760496228 learned_addr learned my addr 172.21.15.175:0/1760496228 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:31.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f254f7fe700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25501599e0 0x7f255014e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.238+0000 7f254ffff700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f255006f950 0x7f25500c16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2554d62700 1 -- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25501599e0 msgr2=0x7f255014e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2554d62700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25501599e0 0x7f255014e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2554d62700 1 -- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f255006f950 msgr2=0x7f25500c16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2554d62700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f255006f950 0x7f25500c16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2554d62700 1 -- 172.21.15.175:0/1760496228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2548009ae0 con 0x7f2550159db0 2023-10-10T19:30:31.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2554d62700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f2550168f90 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f2548008950 tx=0x7f2548009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2537fff700 1 -- 172.21.15.175:0/1760496228 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254800be40 con 0x7f2550159db0 2023-10-10T19:30:31.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2537fff700 1 -- 172.21.15.175:0/1760496228 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2548008d40 con 0x7f2550159db0 2023-10-10T19:30:31.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25500c1c10 con 0x7f2550159db0 2023-10-10T19:30:31.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2537fff700 1 -- 172.21.15.175:0/1760496228 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25480256d0 con 0x7f2550159db0 2023-10-10T19:30:31.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.239+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25500c3b10 con 0x7f2550159db0 2023-10-10T19:30:31.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.240+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2550067570 con 0x7f2550159db0 2023-10-10T19:30:31.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.240+0000 7f2537fff700 1 -- 172.21.15.175:0/1760496228 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2548003040 con 0x7f2550159db0 2023-10-10T19:30:31.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.241+0000 7f2537fff700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f253c066f60 0x7f253c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.241+0000 7f2537fff700 1 -- 172.21.15.175:0/1760496228 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f2548086b30 con 0x7f2550159db0 2023-10-10T19:30:31.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.241+0000 7f254ffff700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f253c066f60 0x7f253c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.241+0000 7f254ffff700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f253c066f60 0x7f253c069410 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2540003a80 tx=0x7f2540007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.245+0000 7f2537fff700 1 -- 172.21.15.175:0/1760496228 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f25480535a0 con 0x7f2550159db0 2023-10-10T19:30:31.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.468+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25500c1380 con 0x7f2550159db0 2023-10-10T19:30:31.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.468+0000 7f2537fff700 1 -- 172.21.15.175:0/1760496228 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 324 v324) v1 ==== 95+0+6200 (secure 0 0 0) 0x7f2548020020 con 0x7f2550159db0 2023-10-10T19:30:31.469 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:30:31.470 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":324,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8188,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1993477403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1993477403},{"type":"v1","addr":"172.21.15.191:6837","nonce":1993477403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":320}],"filesystems":[{"mdsmap":{"epoch":324,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:29:30.149242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:30:31.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f253c066f60 msgr2=0x7f253c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f253c066f60 0x7f253c069410 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2540003a80 tx=0x7f2540007ec0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 msgr2=0x7f2550168f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f2550168f90 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f2548008950 tx=0x7f2548009840 comp rx=0 tx=0).stop 2023-10-10T19:30:31.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 shutdown_connections 2023-10-10T19:30:31.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f253c066f60 0x7f253c069410 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2550159db0 0x7f2550168f90 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f255006f950 0x7f25500c16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 --2- 172.21.15.175:0/1760496228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25501599e0 0x7f255014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 >> 172.21.15.175:0/1760496228 conn(0x7f25500c5510 msgr2=0x7f255014f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:31.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 shutdown_connections 2023-10-10T19:30:31.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.470+0000 7f2555563700 1 -- 172.21.15.175:0/1760496228 wait complete. 2023-10-10T19:30:31.475 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 324 2023-10-10T19:30:31.483 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T19:30:31.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.598+0000 7fbcad1ea700 1 Processor -- start 2023-10-10T19:30:31.599 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.598+0000 7fbcad1ea700 1 -- start start 2023-10-10T19:30:31.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbcad1ea700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbca814cdc0 0x7fbca80c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbcad1ea700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 0x7fbca80c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbcad1ea700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbca806f620 0x7fbca8156300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbcad1ea700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca806e940 con 0x7fbca814cdc0 2023-10-10T19:30:31.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbcad1ea700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca80c1c90 con 0x7fbca806f620 2023-10-10T19:30:31.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbcad1ea700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80c6e20 con 0x7fbca814d190 2023-10-10T19:30:31.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca6d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbca814cdc0 0x7fbca80c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca6d9d700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbca814cdc0 0x7fbca80c26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60172/0 (socket says 172.21.15.175:60172) 2023-10-10T19:30:31.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca6d9d700 1 -- 172.21.15.175:0/2066039076 learned_addr learned my addr 172.21.15.175:0/2066039076 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:31.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbc9e59c700 1 --2- 172.21.15.175:0/2066039076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 0x7fbca80c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca759e700 1 --2- 172.21.15.175:0/2066039076 >> v2:172.21.15.191:3300/0 conn(0x7fbca806f620 0x7fbca8156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca6d9d700 1 -- 172.21.15.175:0/2066039076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 msgr2=0x7fbca80c2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:31.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca6d9d700 1 --2- 172.21.15.175:0/2066039076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 0x7fbca80c2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca6d9d700 1 -- 172.21.15.175:0/2066039076 >> v2:172.21.15.191:3300/0 conn(0x7fbca806f620 msgr2=0x7fbca8156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:31.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca6d9d700 1 --2- 172.21.15.175:0/2066039076 >> v2:172.21.15.191:3300/0 conn(0x7fbca806f620 0x7fbca8156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca6d9d700 1 -- 172.21.15.175:0/2066039076 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca806ca40 con 0x7fbca814cdc0 2023-10-10T19:30:31.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.599+0000 7fbca759e700 1 --2- 172.21.15.175:0/2066039076 >> v2:172.21.15.191:3300/0 conn(0x7fbca806f620 0x7fbca8156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:30:31.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbca6d9d700 1 --2- 172.21.15.175:0/2066039076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca814cdc0 0x7fbca80c26e0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fbca80c19d0 tx=0x7fbc9800b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86641c3d1365b607 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbca659c700 1 -- 172.21.15.175:0/2066039076 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98005c10 con 0x7fbca814cdc0 2023-10-10T19:30:31.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbca659c700 1 -- 172.21.15.175:0/2066039076 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbc98005d70 con 0x7fbca814cdc0 2023-10-10T19:30:31.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbca659c700 1 -- 172.21.15.175:0/2066039076 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9800dba0 con 0x7fbca814cdc0 2023-10-10T19:30:31.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 -- 172.21.15.175:0/2066039076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca814cdc0 msgr2=0x7fbca80c26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/2066039076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca814cdc0 0x7fbca80c26e0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fbca80c19d0 tx=0x7fbc9800b4c0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 -- 172.21.15.175:0/2066039076 shutdown_connections 2023-10-10T19:30:31.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/2066039076 >> v2:172.21.15.191:3300/0 conn(0x7fbca806f620 0x7fbca8156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/2066039076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca814cdc0 0x7fbca80c26e0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.612 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/2066039076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 0x7fbca80c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 -- 172.21.15.175:0/2066039076 >> 172.21.15.175:0/2066039076 conn(0x7fbca8068900 msgr2=0x7fbca8068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:31.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 -- 172.21.15.175:0/2066039076 shutdown_connections 2023-10-10T19:30:31.613 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.600+0000 7fbcad1ea700 1 -- 172.21.15.175:0/2066039076 wait complete. 2023-10-10T19:30:31.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.601+0000 7fbcad1ea700 1 Processor -- start 2023-10-10T19:30:31.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.601+0000 7fbcad1ea700 1 -- start start 2023-10-10T19:30:31.614 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbcad1ea700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca806f620 0x7fbca8165c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbcad1ea700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbca814cdc0 0x7fbca81529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbcad1ea700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 0x7fbca8152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbcad1ea700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8150420 con 0x7fbca806f620 2023-10-10T19:30:31.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbcad1ea700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8152500 con 0x7fbca814cdc0 2023-10-10T19:30:31.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbcad1ea700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80c5340 con 0x7fbca814d190 2023-10-10T19:30:31.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbca6d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca806f620 0x7fbca8165c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbca6d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca806f620 0x7fbca8165c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60174/0 (socket says 172.21.15.175:60174) 2023-10-10T19:30:31.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbca6d9d700 1 -- 172.21.15.175:0/98077988 learned_addr learned my addr 172.21.15.175:0/98077988 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:31.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbc9e59c700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbca814cdc0 0x7fbca81529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.617 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.602+0000 7fbca759e700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 0x7fbca8152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbca6d9d700 1 -- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 msgr2=0x7fbca8152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbca6d9d700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 0x7fbca8152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbca6d9d700 1 -- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbca814cdc0 msgr2=0x7fbca81529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbca6d9d700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbca814cdc0 0x7fbca81529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbca6d9d700 1 -- 172.21.15.175:0/98077988 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9800b720 con 0x7fbca806f620 2023-10-10T19:30:31.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbca6d9d700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca806f620 0x7fbca8165c50 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800d5d0 tx=0x7fbc9800b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbc9f7fe700 1 -- 172.21.15.175:0/98077988 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9801e070 con 0x7fbca806f620 2023-10-10T19:30:31.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbc9f7fe700 1 -- 172.21.15.175:0/98077988 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbc9800d950 con 0x7fbca806f620 2023-10-10T19:30:31.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca81666f0 con 0x7fbca806f620 2023-10-10T19:30:31.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbc9f7fe700 1 -- 172.21.15.175:0/98077988 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98019430 con 0x7fbca806f620 2023-10-10T19:30:31.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.603+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca80c04e0 con 0x7fbca806f620 2023-10-10T19:30:31.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.604+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca8067570 con 0x7fbca806f620 2023-10-10T19:30:31.621 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.604+0000 7fbc9f7fe700 1 -- 172.21.15.175:0/98077988 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbc98019860 con 0x7fbca806f620 2023-10-10T19:30:31.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.604+0000 7fbc9f7fe700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbc88066f20 0x7fbc880693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.605+0000 7fbc9f7fe700 1 -- 172.21.15.175:0/98077988 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7fbc98014030 con 0x7fbca806f620 2023-10-10T19:30:31.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.605+0000 7fbc9e59c700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbc88066f20 0x7fbc880693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.623 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.605+0000 7fbc9e59c700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbc88066f20 0x7fbc880693d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbca8163540 tx=0x7fbc900091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.609+0000 7fbc9f7fe700 1 -- 172.21.15.175:0/98077988 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc98052f60 con 0x7fbca806f620 2023-10-10T19:30:31.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.833+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbca806eb50 con 0x7fbca806f620 2023-10-10T19:30:31.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.833+0000 7fbc9f7fe700 1 -- 172.21.15.175:0/98077988 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 324 v324) v1 ==== 95+0+6200 (secure 0 0 0) 0x7fbc98027020 con 0x7fbca806f620 2023-10-10T19:30:31.834 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:30:31.835 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":324,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8188,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1993477403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":1993477403},{"type":"v1","addr":"172.21.15.191:6837","nonce":1993477403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":320}],"filesystems":[{"mdsmap":{"epoch":324,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T18:45:10.774833+0000","modified":"2023-10-10T19:29:30.149242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":8202,"mds_1":8733,"mds_2":8049,"mds_3":7771,"mds_4":7057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7057":{"gid":7057,"name":"b","rank":4,"incarnation":304,"state":"up:active","state_seq":139,"addr":"172.21.15.191:6836/437857850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":437857850},{"type":"v1","addr":"172.21.15.191:6836","nonce":437857850}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_7771":{"gid":7771,"name":"f","rank":3,"incarnation":297,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/3029933182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3029933182},{"type":"v1","addr":"172.21.15.191:6839","nonce":3029933182}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8049":{"gid":8049,"name":"c","rank":2,"incarnation":285,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6836/2026792975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":2026792975},{"type":"v1","addr":"172.21.15.175:6836","nonce":2026792975}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8202":{"gid":8202,"name":"e","rank":0,"incarnation":291,"state":"up:active","state_seq":21,"addr":"172.21.15.175:6837/74990995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6835","nonce":74990995},{"type":"v1","addr":"172.21.15.175:6837","nonce":74990995}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8733":{"gid":8733,"name":"a","rank":1,"incarnation":314,"state":"up:active","state_seq":19,"addr":"172.21.15.175:6834/3954290407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3954290407},{"type":"v1","addr":"172.21.15.175:6834","nonce":3954290407}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-10T19:30:31.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.835+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbc88066f20 msgr2=0x7fbc880693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.835+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbc88066f20 0x7fbc880693d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbca8163540 tx=0x7fbc900091a0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.835+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca806f620 msgr2=0x7fbca8165c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.835+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca806f620 0x7fbca8165c50 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800d5d0 tx=0x7fbc9800b4c0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.836+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 shutdown_connections 2023-10-10T19:30:31.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.836+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7fbc88066f20 0x7fbc880693d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.836+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbca814d190 0x7fbca8152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.836+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbca806f620 0x7fbca8165c50 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.836+0000 7fbcad1ea700 1 --2- 172.21.15.175:0/98077988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbca814cdc0 0x7fbca81529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.836+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 >> 172.21.15.175:0/98077988 conn(0x7fbca8068900 msgr2=0x7fbca80c26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:31.839 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.836+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 shutdown_connections 2023-10-10T19:30:31.840 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.836+0000 7fbcad1ea700 1 -- 172.21.15.175:0/98077988 wait complete. 2023-10-10T19:30:31.840 INFO:teuthology.orchestra.run.smithi175.stderr:dumped fsmap epoch 324 2023-10-10T19:30:31.848 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T19:30:31.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.962+0000 7f00a53c0700 1 Processor -- start 2023-10-10T19:30:31.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.962+0000 7f00a53c0700 1 -- start start 2023-10-10T19:30:31.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f00a53c0700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f00a00c30c0 0x7f00a014db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.966 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f00a53c0700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00a00c0270 0x7f00a014e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f00a53c0700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a006ec10 con 0x7f00a00c2cf0 2023-10-10T19:30:31.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f00a53c0700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a014e600 con 0x7f00a00c0270 2023-10-10T19:30:31.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f00a53c0700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a0153f70 con 0x7f00a00c30c0 2023-10-10T19:30:31.967 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f009f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00a00c0270 0x7f00a014e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f009f7fe700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00a00c0270 0x7f00a014e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60180/0 (socket says 172.21.15.175:60180) 2023-10-10T19:30:31.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f009f7fe700 1 -- 172.21.15.175:0/3412334046 learned_addr learned my addr 172.21.15.175:0/3412334046 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:31.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.963+0000 7f009e7fc700 1 --2- 172.21.15.175:0/3412334046 >> v2:172.21.15.191:3300/0 conn(0x7f00a00c30c0 0x7f00a014db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.968 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f009f7fe700 1 -- 172.21.15.175:0/3412334046 >> v2:172.21.15.191:3300/0 conn(0x7f00a00c30c0 msgr2=0x7f00a014db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:31.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f009f7fe700 1 --2- 172.21.15.175:0/3412334046 >> v2:172.21.15.191:3300/0 conn(0x7f00a00c30c0 0x7f00a014db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f009f7fe700 1 -- 172.21.15.175:0/3412334046 >> v1:172.21.15.191:6789/0 conn(0x7f00a00c2cf0 legacy=0x7f00a006e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f009f7fe700 1 -- 172.21.15.175:0/3412334046 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a0067bd0 con 0x7f00a00c0270 2023-10-10T19:30:31.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f009f7fe700 1 --2- 172.21.15.175:0/3412334046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a014e0c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f00a006dfb0 tx=0x7f0098009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed9e9b80d9a7e53 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f009dffb700 1 -- 172.21.15.175:0/3412334046 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098009e70 con 0x7f00a00c0270 2023-10-10T19:30:31.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f009dffb700 1 -- 172.21.15.175:0/3412334046 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f009800b8f0 con 0x7f00a00c0270 2023-10-10T19:30:31.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f009dffb700 1 -- 172.21.15.175:0/3412334046 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009800bcb0 con 0x7f00a00c0270 2023-10-10T19:30:31.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f00a53c0700 1 -- 172.21.15.175:0/3412334046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 msgr2=0x7f00a014e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f00a53c0700 1 --2- 172.21.15.175:0/3412334046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a014e0c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f00a006dfb0 tx=0x7f0098009250 comp rx=0 tx=0).stop 2023-10-10T19:30:31.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f00a53c0700 1 -- 172.21.15.175:0/3412334046 shutdown_connections 2023-10-10T19:30:31.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f00a53c0700 1 --2- 172.21.15.175:0/3412334046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a014e0c0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.964+0000 7f00a53c0700 1 --2- 172.21.15.175:0/3412334046 >> v2:172.21.15.191:3300/0 conn(0x7f00a00c30c0 0x7f00a014db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.965+0000 7f00a53c0700 1 -- 172.21.15.175:0/3412334046 >> 172.21.15.175:0/3412334046 conn(0x7f00a0068900 msgr2=0x7f00a0068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:31.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.965+0000 7f00a53c0700 1 -- 172.21.15.175:0/3412334046 shutdown_connections 2023-10-10T19:30:31.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.965+0000 7f00a53c0700 1 -- 172.21.15.175:0/3412334046 wait complete. 2023-10-10T19:30:31.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f00a53c0700 1 Processor -- start 2023-10-10T19:30:31.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f00a53c0700 1 -- start start 2023-10-10T19:30:31.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f00a53c0700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a0146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f00a53c0700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00a00c2cf0 0x7f00a01451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f00a53c0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a00c30c0 0x7f00a0167030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f00a53c0700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a01494c0 con 0x7f00a00c0270 2023-10-10T19:30:31.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f00a53c0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0148a80 con 0x7f00a00c30c0 2023-10-10T19:30:31.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f00a53c0700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a01644e0 con 0x7f00a00c2cf0 2023-10-10T19:30:31.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.966+0000 7f009effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a0146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009effd700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a0146c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60194/0 (socket says 172.21.15.175:60194) 2023-10-10T19:30:31.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009effd700 1 -- 172.21.15.175:0/2571928357 learned_addr learned my addr 172.21.15.175:0/2571928357 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:31.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009f7fe700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a00c30c0 0x7f00a0167030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009e7fc700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00a00c2cf0 0x7f00a01451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009effd700 1 -- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00a00c2cf0 msgr2=0x7f00a01451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009effd700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00a00c2cf0 0x7f00a01451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009effd700 1 -- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a00c30c0 msgr2=0x7f00a0167030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:31.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009effd700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a00c30c0 0x7f00a0167030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:31.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009effd700 1 -- 172.21.15.175:0/2571928357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00980094f0 con 0x7f00a00c0270 2023-10-10T19:30:31.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009f7fe700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a00c30c0 0x7f00a0167030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T19:30:31.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f009effd700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a0146c10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f009000cd60 tx=0x7f0090001b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f0096ffd700 1 -- 172.21.15.175:0/2571928357 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009000ab30 con 0x7f00a00c0270 2023-10-10T19:30:31.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f0096ffd700 1 -- 172.21.15.175:0/2571928357 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f009000ac90 con 0x7f00a00c0270 2023-10-10T19:30:31.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f0096ffd700 1 -- 172.21.15.175:0/2571928357 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090012690 con 0x7f00a00c0270 2023-10-10T19:30:31.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a0164760 con 0x7f00a00c0270 2023-10-10T19:30:31.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.967+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a0148430 con 0x7f00a00c0270 2023-10-10T19:30:31.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.968+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a0067570 con 0x7f00a00c0270 2023-10-10T19:30:31.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.969+0000 7f0096ffd700 1 -- 172.21.15.175:0/2571928357 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0090012ac0 con 0x7f00a00c0270 2023-10-10T19:30:31.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.969+0000 7f0096ffd700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0080066f00 0x7f00800693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:31.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.969+0000 7f0096ffd700 1 -- 172.21.15.175:0/2571928357 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f0090085770 con 0x7f00a00c0270 2023-10-10T19:30:31.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.969+0000 7f009e7fc700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0080066f00 0x7f00800693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:31.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.969+0000 7f009e7fc700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0080066f00 0x7f00800693b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f00a0161660 tx=0x7f0088008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:31.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:31.973+0000 7f0096ffd700 1 -- 172.21.15.175:0/2571928357 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00900521e0 con 0x7f00a00c0270 2023-10-10T19:30:32.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.160+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00a00694a0 con 0x7f00a00c0270 2023-10-10T19:30:32.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.161+0000 7f0096ffd700 1 -- 172.21.15.175:0/2571928357 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+18765 (secure 0 0 0) 0x7f0090055800 con 0x7f00a00c0270 2023-10-10T19:30:32.162 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-10-10T19:30:32.163 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":132,"fsid":"7f0fe9e3-2088-4ba2-8a54-92a2be521afd","created":"2023-10-10T18:44:53.631350+0000","modified":"2023-10-10T19:28:33.906285+0000","last_up_change":"2023-10-10T18:45:04.052774+0000","last_in_change":"2023-10-10T18:44:55.999291+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T18:45:05.161495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T18:45:08.855952+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T18:45:09.742056+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":629,"snap_epoch":123,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"65c64bb9-f0f1-438a-aa17-8dc2335c7070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":90849},{"type":"v1","addr":"172.21.15.175:6809","nonce":90849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":90849},{"type":"v1","addr":"172.21.15.175:6811","nonce":90849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":90849},{"type":"v1","addr":"172.21.15.175:6816","nonce":90849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":90849},{"type":"v1","addr":"172.21.15.175:6813","nonce":90849}]},"public_addr":"172.21.15.175:6809/90849","cluster_addr":"172.21.15.175:6811/90849","heartbeat_back_addr":"172.21.15.175:6816/90849","heartbeat_front_addr":"172.21.15.175:6813/90849","state":["exists","up"]},{"osd":1,"uuid":"a02d3af7-057b-403b-bb65-aca12bdb262b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":90850},{"type":"v1","addr":"172.21.15.175:6801","nonce":90850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":90850},{"type":"v1","addr":"172.21.15.175:6803","nonce":90850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":90850},{"type":"v1","addr":"172.21.15.175:6807","nonce":90850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":90850},{"type":"v1","addr":"172.21.15.175:6805","nonce":90850}]},"public_addr":"172.21.15.175:6801/90850","cluster_addr":"172.21.15.175:6803/90850","heartbeat_back_addr":"172.21.15.175:6807/90850","heartbeat_front_addr":"172.21.15.175:6805/90850","state":["exists","up"]},{"osd":2,"uuid":"dd47f5f1-b7f1-4002-a8f7-9fc10dd41d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":90853},{"type":"v1","addr":"172.21.15.175:6817","nonce":90853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":90853},{"type":"v1","addr":"172.21.15.175:6819","nonce":90853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":90853},{"type":"v1","addr":"172.21.15.175:6823","nonce":90853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":90853},{"type":"v1","addr":"172.21.15.175:6821","nonce":90853}]},"public_addr":"172.21.15.175:6817/90853","cluster_addr":"172.21.15.175:6819/90853","heartbeat_back_addr":"172.21.15.175:6823/90853","heartbeat_front_addr":"172.21.15.175:6821/90853","state":["exists","up"]},{"osd":3,"uuid":"65e7c04a-f10b-494c-8c18-17f8ecb9b485","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":90874},{"type":"v1","addr":"172.21.15.175:6825","nonce":90874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":90874},{"type":"v1","addr":"172.21.15.175:6827","nonce":90874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":90874},{"type":"v1","addr":"172.21.15.175:6831","nonce":90874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":90874},{"type":"v1","addr":"172.21.15.175:6829","nonce":90874}]},"public_addr":"172.21.15.175:6825/90874","cluster_addr":"172.21.15.175:6827/90874","heartbeat_back_addr":"172.21.15.175:6831/90874","heartbeat_front_addr":"172.21.15.175:6829/90874","state":["exists","up"]},{"osd":4,"uuid":"0d263e6f-e89f-4f2f-9f54-b867769fe8f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":91481},{"type":"v1","addr":"172.21.15.191:6801","nonce":91481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":91481},{"type":"v1","addr":"172.21.15.191:6803","nonce":91481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":91481},{"type":"v1","addr":"172.21.15.191:6807","nonce":91481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":91481},{"type":"v1","addr":"172.21.15.191:6805","nonce":91481}]},"public_addr":"172.21.15.191:6801/91481","cluster_addr":"172.21.15.191:6803/91481","heartbeat_back_addr":"172.21.15.191:6807/91481","heartbeat_front_addr":"172.21.15.191:6805/91481","state":["exists","up"]},{"osd":5,"uuid":"98e51172-fbcf-474f-9862-d39dc746a151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":91495},{"type":"v1","addr":"172.21.15.191:6817","nonce":91495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":91495},{"type":"v1","addr":"172.21.15.191:6819","nonce":91495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":91495},{"type":"v1","addr":"172.21.15.191:6823","nonce":91495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":91495},{"type":"v1","addr":"172.21.15.191:6821","nonce":91495}]},"public_addr":"172.21.15.191:6817/91495","cluster_addr":"172.21.15.191:6819/91495","heartbeat_back_addr":"172.21.15.191:6823/91495","heartbeat_front_addr":"172.21.15.191:6821/91495","state":["exists","up"]},{"osd":6,"uuid":"5f3ece06-0c14-4bf5-b346-bba2525a7167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":91482},{"type":"v1","addr":"172.21.15.191:6809","nonce":91482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":91482},{"type":"v1","addr":"172.21.15.191:6811","nonce":91482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":91482},{"type":"v1","addr":"172.21.15.191:6815","nonce":91482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":91482},{"type":"v1","addr":"172.21.15.191:6813","nonce":91482}]},"public_addr":"172.21.15.191:6809/91482","cluster_addr":"172.21.15.191:6811/91482","heartbeat_back_addr":"172.21.15.191:6815/91482","heartbeat_front_addr":"172.21.15.191:6813/91482","state":["exists","up"]},{"osd":7,"uuid":"194ceb5a-d817-4197-bb72-672205ca3331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":91512},{"type":"v1","addr":"172.21.15.191:6825","nonce":91512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":91512},{"type":"v1","addr":"172.21.15.191:6827","nonce":91512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":91512},{"type":"v1","addr":"172.21.15.191:6831","nonce":91512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":91512},{"type":"v1","addr":"172.21.15.191:6829","nonce":91512}]},"public_addr":"172.21.15.191:6825/91512","cluster_addr":"172.21.15.191:6827/91512","heartbeat_back_addr":"172.21.15.191:6831/91512","heartbeat_front_addr":"172.21.15.191:6829/91512","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.193647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.171521+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.212205+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.182291+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.253471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.232530+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.148180+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T18:45:01.192429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:6834/840318079":"2023-10-11T19:28:33.906197+0000","172.21.15.175:6834/3457422611":"2023-10-11T19:27:28.893896+0000","172.21.15.191:6834/2914004704":"2023-10-11T19:25:43.872713+0000","172.21.15.191:6839/2393527600":"2023-10-11T19:24:58.863855+0000","172.21.15.175:6836/972555107":"2023-10-11T19:22:23.833708+0000","172.21.15.191:6838/2393527600":"2023-10-11T19:24:58.863855+0000","172.21.15.175:6833/972555107":"2023-10-11T19:22:23.833708+0000","172.21.15.175:6832/3874110823":"2023-10-11T19:00:33.566369+0000","172.21.15.175:6837/1881948738":"2023-10-11T19:11:03.690607+0000","172.21.15.191:6836/3199867402":"2023-10-11T18:59:08.550308+0000","172.21.15.191:6838/1598171176":"2023-10-11T19:03:13.596165+0000","172.21.15.191:6835/3199867402":"2023-10-11T18:59:08.550308+0000","172.21.15.175:6835/2966961140":"2023-10-11T18:55:38.509904+0000","172.21.15.175:6834/3874110823":"2023-10-11T19:00:33.566369+0000","172.21.15.191:6838/246221790":"2023-10-11T18:53:33.481931+0000","172.21.15.191:6837/2914004704":"2023-10-11T19:25:43.872713+0000","172.21.15.191:6837/1851345295":"2023-10-11T19:21:03.818037+0000","172.21.15.191:6837/3870390806":"2023-10-11T18:57:08.527329+0000","172.21.15.191:6839/246221790":"2023-10-11T18:53:33.481931+0000","172.21.15.191:6834/1851345295":"2023-10-11T19:21:03.818037+0000","172.21.15.175:6832/3457422611":"2023-10-11T19:27:28.893896+0000","172.21.15.191:6835/3007475551":"2023-10-11T18:47:43.409173+0000","172.21.15.175:6836/647425682":"2023-10-11T18:52:23.468892+0000","172.21.15.175:6834/688515950":"2023-10-11T18:49:33.433082+0000","172.21.15.175:6833/647425682":"2023-10-11T18:52:23.468892+0000","172.21.15.175:6837/2966961140":"2023-10-11T18:55:38.509904+0000","172.21.15.175:6832/688515950":"2023-10-11T18:49:33.433082+0000","172.21.15.191:6835/1945747421":"2023-10-11T19:05:28.622478+0000","172.21.15.191:6839/1598171176":"2023-10-11T19:03:13.596165+0000","172.21.15.191:6836/1945747421":"2023-10-11T19:05:28.622478+0000","172.21.15.175:6833/2375829435":"2023-10-11T19:07:23.643743+0000","172.21.15.191:6838/1910071809":"2023-10-11T19:08:28.655326+0000","172.21.15.191:6837/840318079":"2023-10-11T19:28:33.906197+0000","172.21.15.191:6839/1910071809":"2023-10-11T19:08:28.655326+0000","172.21.15.191:6834/824552134":"2023-10-11T19:09:53.672299+0000","172.21.15.175:6832/1872591144":"2023-10-11T19:14:43.733216+0000","172.21.15.175:6835/3849771254":"2023-10-11T19:23:38.848907+0000","172.21.15.175:6834/1872591144":"2023-10-11T19:14:43.733216+0000","172.21.15.175:6837/3849771254":"2023-10-11T19:23:38.848907+0000","172.21.15.191:6837/824552134":"2023-10-11T19:09:53.672299+0000","172.21.15.191:6834/3870390806":"2023-10-11T18:57:08.527329+0000","172.21.15.175:6835/1881948738":"2023-10-11T19:11:03.690607+0000","172.21.15.191:6835/4084912544":"2023-10-11T19:17:13.770352+0000","172.21.15.191:6836/3007475551":"2023-10-11T18:47:43.409173+0000","172.21.15.175:6836/2375829435":"2023-10-11T19:07:23.643743+0000","172.21.15.191:6836/4084912544":"2023-10-11T19:17:13.770352+0000","172.21.15.191:6835/62572703":"2023-10-11T19:13:48.722600+0000","172.21.15.191:6836/62572703":"2023-10-11T19:13:48.722600+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T19:30:32.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0080066f00 msgr2=0x7f00800693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.164 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0080066f00 0x7f00800693b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f00a0161660 tx=0x7f0088008040 comp rx=0 tx=0).stop 2023-10-10T19:30:32.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 msgr2=0x7f00a0146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a0146c10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f009000cd60 tx=0x7f0090001b40 comp rx=0 tx=0).stop 2023-10-10T19:30:32.165 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 shutdown_connections 2023-10-10T19:30:32.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0080066f00 0x7f00800693b0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a00c30c0 0x7f00a0167030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00a00c0270 0x7f00a0146c10 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 --2- 172.21.15.175:0/2571928357 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f00a00c2cf0 0x7f00a01451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 >> 172.21.15.175:0/2571928357 conn(0x7f00a0068900 msgr2=0x7f00a014cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:32.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 shutdown_connections 2023-10-10T19:30:32.168 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.163+0000 7f00a53c0700 1 -- 172.21.15.175:0/2571928357 wait complete. 2023-10-10T19:30:32.176 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-10T19:30:32.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.290+0000 7f9571e94700 1 Processor -- start 2023-10-10T19:30:32.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.290+0000 7f9571e94700 1 -- start start 2023-10-10T19:30:32.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f9571e94700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f956c0c1f60 0x7f956c159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f9571e94700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956c0c74b0 con 0x7f956c0696a0 2023-10-10T19:30:32.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f9571e94700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956c0683c0 con 0x7f956c1597f0 2023-10-10T19:30:32.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f9571e94700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956c068f40 con 0x7f956c0c1f60 2023-10-10T19:30:32.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f956bfff700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f956c0696a0 0x7f956c14fc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52100/0 (socket says 172.21.15.175:52100) 2023-10-10T19:30:32.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f956bfff700 1 -- 172.21.15.175:0/2106083253 learned_addr learned my addr 172.21.15.175:0/2106083253 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:30:32.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f956affd700 1 --2- 172.21.15.175:0/2106083253 >> v2:172.21.15.191:3300/0 conn(0x7f956c0c1f60 0x7f956c159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f956a7fc700 1 -- 172.21.15.175:0/2106083253 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911547818 0 0) 0x7f956c0c74b0 con 0x7f956c0696a0 2023-10-10T19:30:32.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.291+0000 7f956a7fc700 1 -- 172.21.15.175:0/2106083253 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9558003570 con 0x7f956c0696a0 2023-10-10T19:30:32.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956a7fc700 1 -- 172.21.15.175:0/2106083253 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715991198 0 0) 0x7f956c0683c0 con 0x7f956c1597f0 2023-10-10T19:30:32.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956a7fc700 1 -- 172.21.15.175:0/2106083253 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f956c0c74b0 con 0x7f956c1597f0 2023-10-10T19:30:32.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956affd700 1 -- 172.21.15.175:0/2106083253 >> v1:172.21.15.191:6789/0 conn(0x7f956c1597f0 legacy=0x7f956c0c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956affd700 1 -- 172.21.15.175:0/2106083253 >> v1:172.21.15.175:6789/0 conn(0x7f956c0696a0 legacy=0x7f956c14fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956affd700 1 -- 172.21.15.175:0/2106083253 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c0693d0 con 0x7f956c0c1f60 2023-10-10T19:30:32.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956affd700 1 --2- 172.21.15.175:0/2106083253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0c1f60 0x7f956c159bc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f95540098e0 tx=0x7f95540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e148be77589b0fd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:32.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956a7fc700 1 -- 172.21.15.175:0/2106083253 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554011630 con 0x7f956c0c1f60 2023-10-10T19:30:32.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956a7fc700 1 -- 172.21.15.175:0/2106083253 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9554011790 con 0x7f956c0c1f60 2023-10-10T19:30:32.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f956a7fc700 1 -- 172.21.15.175:0/2106083253 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554011b50 con 0x7f956c0c1f60 2023-10-10T19:30:32.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f9571e94700 1 -- 172.21.15.175:0/2106083253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0c1f60 msgr2=0x7f956c159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.292+0000 7f9571e94700 1 --2- 172.21.15.175:0/2106083253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0c1f60 0x7f956c159bc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f95540098e0 tx=0x7f95540091a0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.293+0000 7f9571e94700 1 -- 172.21.15.175:0/2106083253 shutdown_connections 2023-10-10T19:30:32.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.293+0000 7f9571e94700 1 --2- 172.21.15.175:0/2106083253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0c1f60 0x7f956c159bc0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.293+0000 7f9571e94700 1 -- 172.21.15.175:0/2106083253 >> 172.21.15.175:0/2106083253 conn(0x7f956c063230 msgr2=0x7f956c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:32.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.293+0000 7f9571e94700 1 -- 172.21.15.175:0/2106083253 shutdown_connections 2023-10-10T19:30:32.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.293+0000 7f9571e94700 1 -- 172.21.15.175:0/2106083253 wait complete. 2023-10-10T19:30:32.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.294+0000 7f9571e94700 1 Processor -- start 2023-10-10T19:30:32.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.294+0000 7f9571e94700 1 -- start start 2023-10-10T19:30:32.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.294+0000 7f9571e94700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0696a0 0x7f956c1668e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.294+0000 7f9571e94700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f956c0c1f60 0x7f956c158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f9571e94700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f956c1597f0 0x7f956c14a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f9571e94700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c155930 con 0x7f956c1597f0 2023-10-10T19:30:32.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f9571e94700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c14a1f0 con 0x7f956c0696a0 2023-10-10T19:30:32.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f9571e94700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c1567a0 con 0x7f956c0c1f60 2023-10-10T19:30:32.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f956c1597f0 0x7f956c14a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f956c1597f0 0x7f956c14a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60206/0 (socket says 172.21.15.175:60206) 2023-10-10T19:30:32.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 -- 172.21.15.175:0/947938015 learned_addr learned my addr 172.21.15.175:0/947938015 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:32.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956affd700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f956c0c1f60 0x7f956c158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956b7fe700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0696a0 0x7f956c1668e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 -- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f956c0c1f60 msgr2=0x7f956c158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f956c0c1f60 0x7f956c158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 -- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0696a0 msgr2=0x7f956c1668e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0696a0 0x7f956c1668e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 -- 172.21.15.175:0/947938015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9554009420 con 0x7f956c1597f0 2023-10-10T19:30:32.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956affd700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f956c0c1f60 0x7f956c158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:30:32.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956b7fe700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0696a0 0x7f956c1668e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T19:30:32.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f956bfff700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f956c1597f0 0x7f956c14a680 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f956400c590 tx=0x7f9564007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:32.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f95637fe700 1 -- 172.21.15.175:0/947938015 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9564005c00 con 0x7f956c1597f0 2023-10-10T19:30:32.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f95637fe700 1 -- 172.21.15.175:0/947938015 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9564005d60 con 0x7f956c1597f0 2023-10-10T19:30:32.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c14b7a0 con 0x7f956c1597f0 2023-10-10T19:30:32.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f95637fe700 1 -- 172.21.15.175:0/947938015 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9564007340 con 0x7f956c1597f0 2023-10-10T19:30:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.295+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c156a10 con 0x7f956c1597f0 2023-10-10T19:30:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.296+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c071720 con 0x7f956c1597f0 2023-10-10T19:30:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.297+0000 7f95637fe700 1 -- 172.21.15.175:0/947938015 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f956401b3b0 con 0x7f956c1597f0 2023-10-10T19:30:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.297+0000 7f95637fe700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f954c066f30 0x7f954c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.297+0000 7f95637fe700 1 -- 172.21.15.175:0/947938015 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7885+0+0 (secure 0 0 0) 0x7f95640843e0 con 0x7f956c1597f0 2023-10-10T19:30:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.297+0000 7f956b7fe700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f954c066f30 0x7f954c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.298+0000 7f956b7fe700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f954c066f30 0x7f954c0693e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f955c002fc0 tx=0x7f955c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:32.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.301+0000 7f95637fe700 1 -- 172.21.15.175:0/947938015 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9564050dd0 con 0x7f956c1597f0 2023-10-10T19:30:32.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.525+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f956c067570 con 0x7f956c1597f0 2023-10-10T19:30:32.536 INFO:tasks.ceph.mds.e.smithi175.stderr: -9842> 2023-10-10T19:29:07.085+0000 7f7f74406700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T19:30:32.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.531+0000 7f95637fe700 1 -- 172.21.15.175:0/947938015 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v325) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9564017020 con 0x7f956c1597f0 2023-10-10T19:30:32.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.536+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f954c066f30 msgr2=0x7f954c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.536+0000 7f9571e94700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f954c066f30 0x7f954c0693e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f955c002fc0 tx=0x7f955c009040 comp rx=0 tx=0).stop 2023-10-10T19:30:32.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.536+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f956c1597f0 msgr2=0x7f956c14a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.536+0000 7f9571e94700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f956c1597f0 0x7f956c14a680 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f956400c590 tx=0x7f9564007dd0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.540+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 shutdown_connections 2023-10-10T19:30:32.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.540+0000 7f9571e94700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f954c066f30 0x7f954c0693e0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.540+0000 7f9571e94700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f956c1597f0 0x7f956c14a680 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.540+0000 7f9571e94700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f956c0696a0 0x7f956c1668e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.540+0000 7f9571e94700 1 --2- 172.21.15.175:0/947938015 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f956c0c1f60 0x7f956c158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.540+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 >> 172.21.15.175:0/947938015 conn(0x7f956c063230 msgr2=0x7f956c0c81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:32.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.540+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 shutdown_connections 2023-10-10T19:30:32.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.540+0000 7f9571e94700 1 -- 172.21.15.175:0/947938015 wait complete. 2023-10-10T19:30:32.544 INFO:teuthology.orchestra.run.smithi175.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-10T19:30:32.558 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-10T19:30:32.568 INFO:tasks.ceph.mds.c.smithi175.stdout:starting mds.c at 2023-10-10T19:30:32.579 INFO:tasks.ceph.mds.e.smithi175.stdout:starting mds.e at 2023-10-10T19:30:32.579 INFO:tasks.ceph.mds.a.smithi175.stdout:starting mds.a at 2023-10-10T19:30:32.582 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-10T19:30:32.583 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-10T19:30:32.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.680+0000 7f51f6431700 1 Processor -- start 2023-10-10T19:30:32.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.680+0000 7f51f6431700 1 -- start start 2023-10-10T19:30:32.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51f6431700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f0156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51f6431700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f51f00c4f70 0x7f51f01565b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51f6431700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f00c10c0 con 0x7f51f006bf50 2023-10-10T19:30:32.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51f6431700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f00689a0 con 0x7f51f00c4f70 2023-10-10T19:30:32.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51f6431700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0068c80 con 0x7f51f006a5e0 2023-10-10T19:30:32.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51ef7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f0156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51ef7fe700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f0156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:44854/0 (socket says 172.21.15.175:44854) 2023-10-10T19:30:32.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51ef7fe700 1 -- 172.21.15.175:0/1255544865 learned_addr learned my addr 172.21.15.175:0/1255544865 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:32.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51f49ce700 1 --2- 172.21.15.175:0/1255544865 >> v2:172.21.15.191:3300/0 conn(0x7f51f00c4f70 0x7f51f01565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51ef7fe700 1 -- 172.21.15.175:0/1255544865 >> v2:172.21.15.191:3300/0 conn(0x7f51f00c4f70 msgr2=0x7f51f01565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:32.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51ef7fe700 1 --2- 172.21.15.175:0/1255544865 >> v2:172.21.15.191:3300/0 conn(0x7f51f00c4f70 0x7f51f01565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51ef7fe700 1 -- 172.21.15.175:0/1255544865 >> v1:172.21.15.191:6789/0 conn(0x7f51f006bf50 legacy=0x7f51f0153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51ef7fe700 1 -- 172.21.15.175:0/1255544865 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f00c1360 con 0x7f51f006a5e0 2023-10-10T19:30:32.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51ef7fe700 1 --2- 172.21.15.175:0/1255544865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f0156070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f51e40099b0 tx=0x7f51e4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a479c0fd58f5f608 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:32.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.681+0000 7f51eeffd700 1 -- 172.21.15.175:0/1255544865 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4009ea0 con 0x7f51f006a5e0 2023-10-10T19:30:32.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51eeffd700 1 -- 172.21.15.175:0/1255544865 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f51e400b860 con 0x7f51f006a5e0 2023-10-10T19:30:32.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51eeffd700 1 -- 172.21.15.175:0/1255544865 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e400bc20 con 0x7f51f006a5e0 2023-10-10T19:30:32.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51f6431700 1 -- 172.21.15.175:0/1255544865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 msgr2=0x7f51f0156070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51f6431700 1 --2- 172.21.15.175:0/1255544865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f0156070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f51e40099b0 tx=0x7f51e4009250 comp rx=0 tx=0).stop 2023-10-10T19:30:32.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51f6431700 1 -- 172.21.15.175:0/1255544865 shutdown_connections 2023-10-10T19:30:32.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51f6431700 1 --2- 172.21.15.175:0/1255544865 >> v2:172.21.15.191:3300/0 conn(0x7f51f00c4f70 0x7f51f01565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51f6431700 1 --2- 172.21.15.175:0/1255544865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f0156070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51f6431700 1 -- 172.21.15.175:0/1255544865 >> 172.21.15.175:0/1255544865 conn(0x7f51f0063230 msgr2=0x7f51f0063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:32.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51f6431700 1 -- 172.21.15.175:0/1255544865 shutdown_connections 2023-10-10T19:30:32.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.682+0000 7f51f6431700 1 -- 172.21.15.175:0/1255544865 wait complete. 2023-10-10T19:30:32.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.683+0000 7f51f6431700 1 Processor -- start 2023-10-10T19:30:32.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.683+0000 7f51f6431700 1 -- start start 2023-10-10T19:30:32.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51f6431700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f014dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51f6431700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51f006bf50 0x7f51f00aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51f6431700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51f00c4f70 0x7f51f0144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51f6431700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00ab7f0 con 0x7f51f006bf50 2023-10-10T19:30:32.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51f6431700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00ab360 con 0x7f51f00c4f70 2023-10-10T19:30:32.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51f6431700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f01579a0 con 0x7f51f006a5e0 2023-10-10T19:30:32.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51ef7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51f006bf50 0x7f51f00aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51ef7fe700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51f006bf50 0x7f51f00aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60244/0 (socket says 172.21.15.175:60244) 2023-10-10T19:30:32.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51ef7fe700 1 -- 172.21.15.175:0/4111496318 learned_addr learned my addr 172.21.15.175:0/4111496318 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:32.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51effff700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f014dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51f49ce700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51f00c4f70 0x7f51f0144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51ef7fe700 1 -- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 msgr2=0x7f51f014dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51ef7fe700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f014dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51ef7fe700 1 -- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51f00c4f70 msgr2=0x7f51f0144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:32.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51ef7fe700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51f00c4f70 0x7f51f0144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:32.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.684+0000 7f51ef7fe700 1 -- 172.21.15.175:0/4111496318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e40094f0 con 0x7f51f006bf50 2023-10-10T19:30:32.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.685+0000 7f51ef7fe700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51f006bf50 0x7f51f00aeda0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f51e40090b0 tx=0x7f51e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:32.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.685+0000 7f51d7fff700 1 -- 172.21.15.175:0/4111496318 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4000d10 con 0x7f51f006bf50 2023-10-10T19:30:32.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.685+0000 7f51d7fff700 1 -- 172.21.15.175:0/4111496318 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f51e401f070 con 0x7f51f006bf50 2023-10-10T19:30:32.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.685+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f0167890 con 0x7f51f006bf50 2023-10-10T19:30:32.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.685+0000 7f51d7fff700 1 -- 172.21.15.175:0/4111496318 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e40266f0 con 0x7f51f006bf50 2023-10-10T19:30:32.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.685+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f00aad10 con 0x7f51f006bf50 2023-10-10T19:30:32.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.686+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f00ab4d0 con 0x7f51f006bf50 2023-10-10T19:30:32.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.686+0000 7f51d7fff700 1 -- 172.21.15.175:0/4111496318 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f51e4026a60 con 0x7f51f006bf50 2023-10-10T19:30:32.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.686+0000 7f51d7fff700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51dc066f60 0x7f51dc069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:32.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.686+0000 7f51d7fff700 1 -- 172.21.15.175:0/4111496318 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 8315+0+0 (secure 0 0 0) 0x7f51e4021070 con 0x7f51f006bf50 2023-10-10T19:30:32.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.687+0000 7f51effff700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51dc066f60 0x7f51dc069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:32.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.687+0000 7f51effff700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51dc066f60 0x7f51dc069410 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f51e00039e0 tx=0x7f51e0007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:32.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.691+0000 7f51d7fff700 1 -- 172.21.15.175:0/4111496318 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51e4053080 con 0x7f51f006bf50 2023-10-10T19:30:32.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:32.916+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f51f0067570 con 0x7f51f006bf50 2023-10-10T19:30:33.531 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.531+0000 7f51d7fff700 1 -- 172.21.15.175:0/4111496318 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v326) v1 ==== 104+0+0 (secure 0 0 0) 0x7f51e400bbe0 con 0x7f51f006bf50 2023-10-10T19:30:33.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51dc066f60 msgr2=0x7f51dc069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:33.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51dc066f60 0x7f51dc069410 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f51e00039e0 tx=0x7f51e0007ec0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.535 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51f006bf50 msgr2=0x7f51f00aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:33.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51f006bf50 0x7f51f00aeda0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f51e40090b0 tx=0x7f51e4009250 comp rx=0 tx=0).stop 2023-10-10T19:30:33.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 shutdown_connections 2023-10-10T19:30:33.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f51dc066f60 0x7f51dc069410 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51f00c4f70 0x7f51f0144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51f006a5e0 0x7f51f014dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 --2- 172.21.15.175:0/4111496318 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51f006bf50 0x7f51f00aeda0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 >> 172.21.15.175:0/4111496318 conn(0x7f51f0063230 msgr2=0x7f51f00c7050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:33.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 shutdown_connections 2023-10-10T19:30:33.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.534+0000 7f51f6431700 1 -- 172.21.15.175:0/4111496318 wait complete. 2023-10-10T19:30:33.547 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-10T19:30:33.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.706+0000 7f772bfff700 1 Processor -- start 2023-10-10T19:30:33.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.706+0000 7f772bfff700 1 -- start start 2023-10-10T19:30:33.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c1567e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:33.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772bfff700 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772c0c82b0 con 0x7f772c0c6490 2023-10-10T19:30:33.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772bfff700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772c0c3e10 con 0x7f772c0c60c0 2023-10-10T19:30:33.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772bfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772c156d20 con 0x7f772c0c6f00 2023-10-10T19:30:33.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772b7fe700 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f772c0c6490 0x7f772c14fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:52116/0 (socket says 172.21.15.175:52116) 2023-10-10T19:30:33.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772b7fe700 1 -- 172.21.15.175:0/859600861 learned_addr learned my addr 172.21.15.175:0/859600861 (peer_addr_for_me v1:172.21.15.175:0/0) 2023-10-10T19:30:33.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772a7fc700 1 --2- 172.21.15.175:0/859600861 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c1567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:33.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f7729ffb700 1 -- 172.21.15.175:0/859600861 <== mon.0 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233172262 0 0) 0x7f772c0c82b0 con 0x7f772c0c6490 2023-10-10T19:30:33.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f7729ffb700 1 -- 172.21.15.175:0/859600861 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7718003570 con 0x7f772c0c6490 2023-10-10T19:30:33.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f7729ffb700 1 -- 172.21.15.175:0/859600861 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118854609 0 0) 0x7f772c0c3e10 con 0x7f772c0c60c0 2023-10-10T19:30:33.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f7729ffb700 1 -- 172.21.15.175:0/859600861 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f772c0c82b0 con 0x7f772c0c60c0 2023-10-10T19:30:33.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772a7fc700 1 -- 172.21.15.175:0/859600861 >> v1:172.21.15.191:6789/0 conn(0x7f772c0c60c0 legacy=0x7f772c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:33.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f7729ffb700 1 -- 172.21.15.175:0/859600861 <== mon.0 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2761638143 0 0) 0x7f7718003570 con 0x7f772c0c6490 2023-10-10T19:30:33.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772a7fc700 1 -- 172.21.15.175:0/859600861 >> v1:172.21.15.175:6789/0 conn(0x7f772c0c6490 legacy=0x7f772c14fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:33.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.707+0000 7f772a7fc700 1 -- 172.21.15.175:0/859600861 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c0c49b0 con 0x7f772c0c6f00 2023-10-10T19:30:33.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f772a7fc700 1 --2- 172.21.15.175:0/859600861 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c1567e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7714009910 tx=0x7f77140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ff5f2f2986af5ae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:33.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f7729ffb700 1 -- 172.21.15.175:0/859600861 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724001470 con 0x7f772c0c6f00 2023-10-10T19:30:33.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f7729ffb700 1 -- 172.21.15.175:0/859600861 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f7714011420 con 0x7f772c0c6f00 2023-10-10T19:30:33.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f7729ffb700 1 -- 172.21.15.175:0/859600861 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140117a0 con 0x7f772c0c6f00 2023-10-10T19:30:33.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f772bfff700 1 -- 172.21.15.175:0/859600861 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 msgr2=0x7f772c1567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:33.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f772bfff700 1 --2- 172.21.15.175:0/859600861 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c1567e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7714009910 tx=0x7f77140091a0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f772bfff700 1 -- 172.21.15.175:0/859600861 shutdown_connections 2023-10-10T19:30:33.716 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f772bfff700 1 --2- 172.21.15.175:0/859600861 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c1567e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f772bfff700 1 -- 172.21.15.175:0/859600861 >> 172.21.15.175:0/859600861 conn(0x7f772c063230 msgr2=0x7f772c063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:33.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f772bfff700 1 -- 172.21.15.175:0/859600861 shutdown_connections 2023-10-10T19:30:33.717 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.708+0000 7f772bfff700 1 -- 172.21.15.175:0/859600861 wait complete. 2023-10-10T19:30:33.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.709+0000 7f772bfff700 1 Processor -- start 2023-10-10T19:30:33.718 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772bfff700 1 -- start start 2023-10-10T19:30:33.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772bfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f772c0c60c0 0x7f772c068460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:33.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772bfff700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c0c6490 0x7f772c157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:33.721 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772bfff700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:33.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772bfff700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c152f40 con 0x7f772c0c6490 2023-10-10T19:30:33.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772bfff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c157950 con 0x7f772c0c60c0 2023-10-10T19:30:33.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772bfff700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772c0cd160 con 0x7f772c0c6f00 2023-10-10T19:30:33.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772a7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c0c6490 0x7f772c157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:33.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772a7fc700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c0c6490 0x7f772c157de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:60246/0 (socket says 172.21.15.175:60246) 2023-10-10T19:30:33.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772a7fc700 1 -- 172.21.15.175:0/2579027229 learned_addr learned my addr 172.21.15.175:0/2579027229 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:33.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772affd700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f772c0c60c0 0x7f772c068460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:33.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.710+0000 7f772b7fe700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:33.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f772a7fc700 1 -- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 msgr2=0x7f772c158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:33.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f772a7fc700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f772a7fc700 1 -- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f772c0c60c0 msgr2=0x7f772c068460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:33.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f772a7fc700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f772c0c60c0 0x7f772c068460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:33.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f772a7fc700 1 -- 172.21.15.175:0/2579027229 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7714009420 con 0x7f772c0c6490 2023-10-10T19:30:33.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f772a7fc700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c0c6490 0x7f772c157de0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f7714002000 tx=0x7f77140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:33.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f7722ffd700 1 -- 172.21.15.175:0/2579027229 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771401caf0 con 0x7f772c0c6490 2023-10-10T19:30:33.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f7722ffd700 1 -- 172.21.15.175:0/2579027229 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f771401cc50 con 0x7f772c0c6490 2023-10-10T19:30:33.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772c0c1cd0 con 0x7f772c0c6490 2023-10-10T19:30:33.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f7722ffd700 1 -- 172.21.15.175:0/2579027229 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771401a450 con 0x7f772c0c6490 2023-10-10T19:30:33.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.711+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772c0c2290 con 0x7f772c0c6490 2023-10-10T19:30:33.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.712+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772c0c2480 con 0x7f772c0c6490 2023-10-10T19:30:33.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.712+0000 7f7722ffd700 1 -- 172.21.15.175:0/2579027229 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f771401a7c0 con 0x7f772c0c6490 2023-10-10T19:30:33.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.713+0000 7f7722ffd700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f770c066f60 0x7f770c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:33.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.713+0000 7f7722ffd700 1 -- 172.21.15.175:0/2579027229 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 8315+0+0 (secure 0 0 0) 0x7f7714086750 con 0x7f772c0c6490 2023-10-10T19:30:33.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.713+0000 7f772affd700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f770c066f60 0x7f770c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:33.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.713+0000 7f772affd700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f770c066f60 0x7f770c069410 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f771c0040a0 tx=0x7f771c007f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:33.729 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.717+0000 7f7722ffd700 1 -- 172.21.15.175:0/2579027229 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7714052fd0 con 0x7f772c0c6490 2023-10-10T19:30:33.898 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:33.897+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f772c157a90 con 0x7f772c0c6490 2023-10-10T19:30:34.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.538+0000 7f7722ffd700 1 -- 172.21.15.175:0/2579027229 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v134) v1 ==== 181+0+0 (secure 0 0 0) 0x7f77140565f0 con 0x7f772c0c6490 2023-10-10T19:30:34.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f770c066f60 msgr2=0x7f770c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:34.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f770c066f60 0x7f770c069410 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f771c0040a0 tx=0x7f771c007f90 comp rx=0 tx=0).stop 2023-10-10T19:30:34.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c0c6490 msgr2=0x7f772c157de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:34.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c0c6490 0x7f772c157de0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f7714002000 tx=0x7f77140091a0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 shutdown_connections 2023-10-10T19:30:34.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f770c066f60 0x7f770c069410 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f772c0c6f00 0x7f772c158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f772c0c60c0 0x7f772c068460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 --2- 172.21.15.175:0/2579027229 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c0c6490 0x7f772c157de0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 >> 172.21.15.175:0/2579027229 conn(0x7f772c063230 msgr2=0x7f772c0c42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:34.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 shutdown_connections 2023-10-10T19:30:34.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.541+0000 7f772bfff700 1 -- 172.21.15.175:0/2579027229 wait complete. 2023-10-10T19:30:34.545 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'cephfs_metadata' removed 2023-10-10T19:30:34.558 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-10T19:30:34.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.679+0000 7f044c9e8700 1 Processor -- start 2023-10-10T19:30:34.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.679+0000 7f044c9e8700 1 -- start start 2023-10-10T19:30:34.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f044c9e8700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f04480c6240 0x7f04480690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:34.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f044c9e8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04480c6610 0x7f04480c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f044c9e8700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04480c4b60 con 0x7f0448069620 2023-10-10T19:30:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f044c9e8700 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0448068c50 con 0x7f04480c6240 2023-10-10T19:30:34.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f044c9e8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0448157420 con 0x7f04480c6610 2023-10-10T19:30:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f044659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f04480c6240 0x7f04480690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f044659c700 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f04480c6240 0x7f04480690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40334/0 (socket says 172.21.15.175:40334) 2023-10-10T19:30:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f044659c700 1 -- 172.21.15.175:0/954748685 learned_addr learned my addr 172.21.15.175:0/954748685 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.680+0000 7f0446d9d700 1 --2- 172.21.15.175:0/954748685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04480c6610 0x7f04480c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044659c700 1 -- 172.21.15.175:0/954748685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04480c6610 msgr2=0x7f04480c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T19:30:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044659c700 1 --2- 172.21.15.175:0/954748685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04480c6610 0x7f04480c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044659c700 1 -- 172.21.15.175:0/954748685 >> v1:172.21.15.191:6789/0 conn(0x7f0448069620 legacy=0x7f04480c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044659c700 1 -- 172.21.15.175:0/954748685 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04480c4880 con 0x7f04480c6240 2023-10-10T19:30:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044659c700 1 --2- 172.21.15.175:0/954748685 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6240 0x7f04480690e0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f0438009f20 tx=0x7f0438009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4de0cc528e0d83 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:34.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044559a700 1 -- 172.21.15.175:0/954748685 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043800e070 con 0x7f04480c6240 2023-10-10T19:30:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044559a700 1 -- 172.21.15.175:0/954748685 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f043800ba60 con 0x7f04480c6240 2023-10-10T19:30:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044559a700 1 -- 172.21.15.175:0/954748685 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04380092e0 con 0x7f04480c6240 2023-10-10T19:30:34.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044c9e8700 1 -- 172.21.15.175:0/954748685 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6240 msgr2=0x7f04480690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.681+0000 7f044c9e8700 1 --2- 172.21.15.175:0/954748685 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6240 0x7f04480690e0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f0438009f20 tx=0x7f0438009840 comp rx=0 tx=0).stop 2023-10-10T19:30:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.682+0000 7f044c9e8700 1 -- 172.21.15.175:0/954748685 shutdown_connections 2023-10-10T19:30:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.682+0000 7f044c9e8700 1 --2- 172.21.15.175:0/954748685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04480c6610 0x7f04480c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.682+0000 7f044c9e8700 1 --2- 172.21.15.175:0/954748685 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6240 0x7f04480690e0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.682+0000 7f044c9e8700 1 -- 172.21.15.175:0/954748685 >> 172.21.15.175:0/954748685 conn(0x7f0448063230 msgr2=0x7f0448063630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:34.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.682+0000 7f044c9e8700 1 -- 172.21.15.175:0/954748685 shutdown_connections 2023-10-10T19:30:34.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.682+0000 7f044c9e8700 1 -- 172.21.15.175:0/954748685 wait complete. 2023-10-10T19:30:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.683+0000 7f044c9e8700 1 Processor -- start 2023-10-10T19:30:34.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.683+0000 7f044c9e8700 1 -- start start 2023-10-10T19:30:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.683+0000 7f044c9e8700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0448069620 0x7f04480ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.683+0000 7f044c9e8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04480c6240 0x7f04480cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:34.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.683+0000 7f044c9e8700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6610 0x7f04480cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.683+0000 7f044c9e8700 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0448152ce0 con 0x7f04480c6610 2023-10-10T19:30:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.683+0000 7f044c9e8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480cab40 con 0x7f04480c6240 2023-10-10T19:30:34.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.683+0000 7f044c9e8700 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04480cc030 con 0x7f0448069620 2023-10-10T19:30:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0446d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6610 0x7f04480cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f044659c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0448069620 0x7f04480ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f044659c700 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0448069620 0x7f04480ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.175:43310/0 (socket says 172.21.15.175:43310) 2023-10-10T19:30:34.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f044659c700 1 -- 172.21.15.175:0/405135825 learned_addr learned my addr 172.21.15.175:0/405135825 (peer_addr_for_me v2:172.21.15.175:0/0) 2023-10-10T19:30:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0446d9d700 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6610 0x7f04480cb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40344/0 (socket says 172.21.15.175:40344) 2023-10-10T19:30:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0445d9b700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04480c6240 0x7f04480cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:34.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0446d9d700 1 -- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0448069620 msgr2=0x7f04480ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:34.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0446d9d700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0448069620 0x7f04480ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0446d9d700 1 -- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04480c6240 msgr2=0x7f04480cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:34.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0446d9d700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04480c6240 0x7f04480cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:34.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0446d9d700 1 -- 172.21.15.175:0/405135825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438009ae0 con 0x7f04480c6610 2023-10-10T19:30:34.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f044659c700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0448069620 0x7f04480ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T19:30:34.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f0446d9d700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6610 0x7f04480cb510 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f044000e980 tx=0x7f04400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:34.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f043e7fc700 1 -- 172.21.15.175:0/405135825 <== mon.0 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440013680 con 0x7f04480c6610 2023-10-10T19:30:34.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f043e7fc700 1 -- 172.21.15.175:0/405135825 <== mon.0 v2:172.21.15.175:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0440013800 con 0x7f04480c6610 2023-10-10T19:30:34.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f043e7fc700 1 -- 172.21.15.175:0/405135825 <== mon.0 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440013be0 con 0x7f04480c6610 2023-10-10T19:30:34.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0448154fc0 con 0x7f04480c6610 2023-10-10T19:30:34.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.684+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04480cc280 con 0x7f04480c6610 2023-10-10T19:30:34.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.685+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04480cacf0 con 0x7f04480c6610 2023-10-10T19:30:34.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.685+0000 7f043e7fc700 1 -- 172.21.15.175:0/405135825 <== mon.0 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f04400123d0 con 0x7f04480c6610 2023-10-10T19:30:34.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.686+0000 7f043e7fc700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0428066f60 0x7f0428069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T19:30:34.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.686+0000 7f043e7fc700 1 -- 172.21.15.175:0/405135825 <== mon.0 v2:172.21.15.175:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 7877+0+0 (secure 0 0 0) 0x7f0440084bc0 con 0x7f04480c6610 2023-10-10T19:30:34.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.686+0000 7f044659c700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0428066f60 0x7f0428069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T19:30:34.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.686+0000 7f044659c700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0428066f60 0x7f0428069410 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0448089f80 tx=0x7f0438009840 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T19:30:34.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.690+0000 7f043e7fc700 1 -- 172.21.15.175:0/405135825 <== mon.0 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04480cacf0 con 0x7f04480c6610 2023-10-10T19:30:34.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:34.871+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f04480cacf0 con 0x7f04480c6610 2023-10-10T19:30:35.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.541+0000 7f043e7fc700 1 -- 172.21.15.175:0/405135825 <== mon.0 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) v1 ==== 169+0+0 (secure 0 0 0) 0x7f04480cacf0 con 0x7f04480c6610 2023-10-10T19:30:35.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0428066f60 msgr2=0x7f0428069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:35.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0428066f60 0x7f0428069410 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0448089f80 tx=0x7f0438009840 comp rx=0 tx=0).stop 2023-10-10T19:30:35.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6610 msgr2=0x7f04480cb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T19:30:35.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6610 0x7f04480cb510 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f044000e980 tx=0x7f04400091a0 comp rx=0 tx=0).stop 2023-10-10T19:30:35.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 shutdown_connections 2023-10-10T19:30:35.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:6832/90896,v1:172.21.15.191:6833/90896] conn(0x7f0428066f60 0x7f0428069410 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:35.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04480c6610 0x7f04480cb510 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:35.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0448069620 0x7f04480ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:35.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 --2- 172.21.15.175:0/405135825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04480c6240 0x7f04480cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T19:30:35.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.546+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 >> 172.21.15.175:0/405135825 conn(0x7f0448063230 msgr2=0x7f04481553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T19:30:35.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.548+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 shutdown_connections 2023-10-10T19:30:35.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-10-10T19:30:35.548+0000 7f044c9e8700 1 -- 172.21.15.175:0/405135825 wait complete. 2023-10-10T19:30:35.550 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'cephfs_data' removed 2023-10-10T19:30:35.564 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-10T19:30:35.565 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-10T19:30:35.565 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.576 INFO:tasks.ceph.mds.a:Stopped 2023-10-10T19:30:35.577 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-10T19:30:35.577 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.626 INFO:tasks.ceph.mds.c:Stopped 2023-10-10T19:30:35.626 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-10T19:30:35.627 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.675 INFO:tasks.ceph.mds.e:Stopped 2023-10-10T19:30:35.675 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-10T19:30:35.676 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.684 INFO:tasks.ceph.mds.b:Stopped 2023-10-10T19:30:35.685 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-10T19:30:35.685 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.734 INFO:tasks.ceph.mds.d:Stopped 2023-10-10T19:30:35.734 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-10T19:30:35.735 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.783 INFO:tasks.ceph.mds.f:Stopped 2023-10-10T19:30:35.784 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-10T19:30:35.784 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-10T19:30:35.785 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.819 INFO:tasks.ceph.osd.0:Stopped 2023-10-10T19:30:35.820 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-10T19:30:35.820 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.891 INFO:tasks.ceph.osd.1:Stopped 2023-10-10T19:30:35.892 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-10T19:30:35.892 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:35.962 INFO:tasks.ceph.osd.2:Stopped 2023-10-10T19:30:35.963 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-10T19:30:35.963 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.040 INFO:tasks.ceph.osd.3:Stopped 2023-10-10T19:30:36.040 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-10T19:30:36.041 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.090 INFO:tasks.ceph.osd.4:Stopped 2023-10-10T19:30:36.090 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-10T19:30:36.091 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.130 INFO:tasks.ceph.osd.5:Stopped 2023-10-10T19:30:36.131 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-10T19:30:36.131 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.216 INFO:tasks.ceph.osd.6:Stopped 2023-10-10T19:30:36.216 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-10T19:30:36.217 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.267 INFO:tasks.ceph.osd.7:Stopped 2023-10-10T19:30:36.267 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-10T19:30:36.268 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-10T19:30:36.268 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.290 INFO:tasks.ceph.mgr.y:Stopped 2023-10-10T19:30:36.291 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-10T19:30:36.291 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.317 INFO:tasks.ceph.mgr.x:Stopped 2023-10-10T19:30:36.317 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-10T19:30:36.318 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-10T19:30:36.318 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.342 INFO:tasks.ceph.mon.a:Stopped 2023-10-10T19:30:36.343 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-10T19:30:36.343 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.369 INFO:tasks.ceph.mon.b:Stopped 2023-10-10T19:30:36.369 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-10T19:30:36.370 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T19:30:36.421 INFO:tasks.ceph.mon.c:Stopped 2023-10-10T19:30:36.421 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-10T19:30:36.422 DEBUG:teuthology.orchestra.run.smithi175:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-10T19:30:36.452 INFO:teuthology.orchestra.run.smithi175.stdout:1696965769.847696 mds.a (mds.1) 29 : cluster [WRN] client.4773 isn't responding to mclientcaps(revoke), ino 0x2000000138c pending pAsLsXsFc issued pAsLsXsFsc, sent 302.058060 seconds ago 2023-10-10T19:30:36.453 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-10T19:30:36.453 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T19:30:36.454 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-10T19:30:36.998 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T19:30:36.998 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-10T19:30:37.095 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T19:30:37.096 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-10T19:30:37.173 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T19:30:37.174 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-10T19:30:37.249 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T19:30:37.249 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-10T19:30:37.891 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T19:30:37.892 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-10T19:30:37.969 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T19:30:37.969 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-10T19:30:38.049 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T19:30:38.050 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-10T19:30:38.127 INFO:tasks.ceph:Archiving mon data... 2023-10-10T19:30:38.129 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/data/mon.a.tgz 2023-10-10T19:30:38.130 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2023-10-10T19:30:38.146 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.0o8MSfcOWv 2023-10-10T19:30:38.146 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0o8MSfcOWv 2023-10-10T19:30:38.288 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.0o8MSfcOWv 2023-10-10T19:30:38.369 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.0o8MSfcOWv is 427KB 2023-10-10T19:30:38.465 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.0o8MSfcOWv 2023-10-10T19:30:38.480 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/data/mon.b.tgz 2023-10-10T19:30:38.480 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-10-10T19:30:38.497 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.tcPqf5KUSM 2023-10-10T19:30:38.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.tcPqf5KUSM 2023-10-10T19:30:38.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.tcPqf5KUSM 2023-10-10T19:30:38.738 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.tcPqf5KUSM is 435KB 2023-10-10T19:30:38.835 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.tcPqf5KUSM 2023-10-10T19:30:38.850 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/data/mon.c.tgz 2023-10-10T19:30:38.851 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-10-10T19:30:38.904 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.87H6U5x33M 2023-10-10T19:30:38.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.87H6U5x33M 2023-10-10T19:30:39.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.87H6U5x33M 2023-10-10T19:30:39.134 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.87H6U5x33M is 435KB 2023-10-10T19:30:39.202 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.87H6U5x33M 2023-10-10T19:30:39.218 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-10T19:30:39.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-10T19:30:39.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-10T19:30:39.281 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-10T19:30:39.282 DEBUG:teuthology.orchestra.run.smithi175:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-10T19:30:39.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-10T19:30:39.310 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-10T19:30:39.348 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-10T19:30:39.353 INFO:tasks.ceph:Archiving crash dumps... 2023-10-10T19:30:39.355 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/remote/smithi175/crash 2023-10-10T19:30:39.357 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-10T19:30:39.381 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/remote/smithi191/crash 2023-10-10T19:30:39.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-10T19:30:39.417 INFO:tasks.ceph:Shutting down logrotate 2023-10-10T19:30:39.417 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:30:39.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-10T19:30:39.514 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-10T19:30:39.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-10T19:30:39.562 INFO:tasks.ceph:Compressing logs... 2023-10-10T19:30:39.563 DEBUG:teuthology.orchestra.run.smithi175:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-10T19:30:39.581 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-10T19:30:39.603 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131446.log 2023-10-10T19:30:39.604 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114069.log 2023-10-10T19:30:39.604 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119328.log 2023-10-10T19:30:39.636 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91342.log 2023-10-10T19:30:39.636 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91129.log 2023-10-10T19:30:39.637 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-10T19:30:39.637 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.91342.log: /var/log/ceph/ceph-client.admin.91129.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-10T19:30:39.637 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91342.log.gz 2023-10-10T19:30:39.637 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91129.log.gz 2023-10-10T19:30:39.638 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-10T19:30:39.638 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91089.log 2023-10-10T19:30:39.638 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-10T19:30:39.639 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.91089.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-10T19:30:39.639 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose /var/log/ceph/ceph.audit.log: -- /var/log/ceph/ceph-mds.d.log 2023-10-10T19:30:39.639 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91089.log.gz 2023-10-10T19:30:39.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-10T19:30:39.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mds.d.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-10T19:30:39.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91213.log 2023-10-10T19:30:39.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.4.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-10T19:30:39.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.91213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91301.log 2023-10-10T19:30:39.643 INFO:teuthology.orchestra.run.smithi191.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91213.log.gz 2023-10-10T19:30:39.643 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91385.log 2023-10-10T19:30:39.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.91301.log: /var/log/ceph/ceph-client.admin.91385.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91301.log.gz 2023-10-10T19:30:39.644 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91385.log.gz 2023-10-10T19:30:39.644 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-10T19:30:39.644 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-10T19:30:39.653 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-10T19:30:39.666 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91170.log 2023-10-10T19:30:39.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.91170.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91170.log.gz 2023-10-10T19:30:39.667 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-10T19:30:39.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91255.log 2023-10-10T19:30:39.696 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.91255.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91255.log.gz 2023-10-10T19:30:39.709 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-10T19:30:39.795 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114069.log: gzip -5 88.2%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.131446.log.gz/var/log/ceph/ceph-client.admin.111837.log 2023-10-10T19:30:39.795 INFO:teuthology.orchestra.run.smithi175.stderr: 2023-10-10T19:30:39.795 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114069.log.gz 2023-10-10T19:30:39.796 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.119328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123261.log 2023-10-10T19:30:39.796 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111837.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.119328.log.gz 2023-10-10T19:30:39.796 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110096.log 2023-10-10T19:30:39.797 INFO:teuthology.orchestra.run.smithi175.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111837.log.gz 2023-10-10T19:30:39.797 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122208.log 2023-10-10T19:30:39.797 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110096.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.123261.log.gz 2023-10-10T19:30:39.798 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110844.log 2023-10-10T19:30:39.798 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122208.log: 89.1%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.110096.log.gz--verbose 2023-10-10T19:30:39.798 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.112203.log 2023-10-10T19:30:39.798 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.122208.log.gz 2023-10-10T19:30:39.799 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96079.log 2023-10-10T19:30:39.799 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112203.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110844.log.gz 2023-10-10T19:30:39.799 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122604.log 2023-10-10T19:30:39.800 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96079.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112203.log.gz 2023-10-10T19:30:39.800 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104100.log 2023-10-10T19:30:39.800 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122604.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96079.log.gz 2023-10-10T19:30:39.801 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124360.log 2023-10-10T19:30:39.801 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112682.log 2023-10-10T19:30:39.801 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.122604.log.gz 2023-10-10T19:30:39.801 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124360.log: gzip 89.5%-5 -- replaced with /var/log/ceph/ceph-client.admin.104100.log.gz--verbose 2023-10-10T19:30:39.802 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.132009.log 2023-10-10T19:30:39.802 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112682.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124360.log.gz 2023-10-10T19:30:39.802 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100680.log 2023-10-10T19:30:39.803 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132009.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112682.log.gz 2023-10-10T19:30:39.803 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121577.log 2023-10-10T19:30:39.803 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95379.log 2023-10-10T19:30:39.803 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.132009.log.gz 2023-10-10T19:30:39.804 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2%/var/log/ceph/ceph-client.admin.121577.log: -- replaced with /var/log/ceph/ceph-client.admin.100680.log.gz 2023-10-10T19:30:39.804 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95421.log 2023-10-10T19:30:39.804 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95379.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121577.log.gz 2023-10-10T19:30:39.805 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.84589.log 2023-10-10T19:30:39.805 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95421.log: gzip 89.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.95379.log.gz--verbose 2023-10-10T19:30:39.805 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.126692.log 2023-10-10T19:30:39.806 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph.tmp-client.admin.84589.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95421.log.gz 2023-10-10T19:30:39.806 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114325.log 2023-10-10T19:30:39.806 INFO:teuthology.orchestra.run.smithi175.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.84589.log.gz 2023-10-10T19:30:39.806 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126777.log 2023-10-10T19:30:39.807 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114325.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126692.log.gz 2023-10-10T19:30:39.807 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127232.log 2023-10-10T19:30:39.807 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110558.log 2023-10-10T19:30:39.808 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114325.log.gz 2023-10-10T19:30:39.808 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127232.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126777.log.gz 2023-10-10T19:30:39.808 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129478.log 2023-10-10T19:30:39.808 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110558.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.127232.log.gz 2023-10-10T19:30:39.809 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129041.log 2023-10-10T19:30:39.809 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112959.log 2023-10-10T19:30:39.809 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110558.log.gz 2023-10-10T19:30:39.809 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129041.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.129478.log.gz 2023-10-10T19:30:39.810 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116629.log 2023-10-10T19:30:39.810 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.129041.log.gz 2023-10-10T19:30:39.810 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126945.log 2023-10-10T19:30:39.811 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116629.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112959.log.gz 2023-10-10T19:30:39.811 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90580.log 2023-10-10T19:30:39.811 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126945.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116629.log.gz 2023-10-10T19:30:39.811 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114129.log 2023-10-10T19:30:39.812 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126945.log.gz 2023-10-10T19:30:39.812 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.90580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113666.log 2023-10-10T19:30:39.812 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113355.log 2023-10-10T19:30:39.812 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90580.log.gz 2023-10-10T19:30:39.813 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99369.log 2023-10-10T19:30:39.813 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114129.log.gz 2023-10-10T19:30:39.813 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95287.log 2023-10-10T19:30:39.814 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113666.log.gz 2023-10-10T19:30:39.814 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99369.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113355.log.gz 2023-10-10T19:30:39.814 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115410.log 2023-10-10T19:30:39.814 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4%/var/log/ceph/ceph-client.admin.95287.log: -- replaced with /var/log/ceph/ceph-client.admin.99369.log.gz 2023-10-10T19:30:39.815 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107503.log 2023-10-10T19:30:39.815 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95287.log.gz 2023-10-10T19:30:39.815 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97214.log 2023-10-10T19:30:39.816 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107503.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115410.log.gz 2023-10-10T19:30:39.816 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130064.log 2023-10-10T19:30:39.816 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97214.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107503.log.gz 2023-10-10T19:30:39.816 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121532.log 2023-10-10T19:30:39.817 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97214.log.gz/var/log/ceph/ceph-client.admin.130064.log: 2023-10-10T19:30:39.817 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131194.log 2023-10-10T19:30:39.817 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111439.log 2023-10-10T19:30:39.818 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.130064.log.gz 2023-10-10T19:30:39.818 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131194.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.121532.log.gz 2023-10-10T19:30:39.818 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97853.log 2023-10-10T19:30:39.818 INFO:teuthology.orchestra.run.smithi175.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.131194.log.gz/var/log/ceph/ceph-client.admin.111439.log: 2023-10-10T19:30:39.819 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129969.log 2023-10-10T19:30:39.819 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109150.log 89.2% 2023-10-10T19:30:39.819 INFO:teuthology.orchestra.run.smithi175.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111439.log.gz 2023-10-10T19:30:39.820 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129969.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97853.log.gz 2023-10-10T19:30:39.820 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123671.log 2023-10-10T19:30:39.820 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129969.log.gz 2023-10-10T19:30:39.820 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113456.log /var/log/ceph/ceph-client.admin.109150.log: 2023-10-10T19:30:39.821 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127798.log 2023-10-10T19:30:39.821 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109150.log.gz 2023-10-10T19:30:39.821 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113456.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123671.log.gz 2023-10-10T19:30:39.822 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127442.log 2023-10-10T19:30:39.822 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127798.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.113456.log.gz 2023-10-10T19:30:39.822 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116755.log 2023-10-10T19:30:39.822 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124318.log 2023-10-10T19:30:39.823 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.127798.log.gz 2023-10-10T19:30:39.823 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116755.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.127442.log.gz 2023-10-10T19:30:39.823 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130340.log 2023-10-10T19:30:39.824 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116755.log.gz 2023-10-10T19:30:39.824 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132262.log 2023-10-10T19:30:39.824 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115776.log 2023-10-10T19:30:39.825 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.124318.log.gz 2023-10-10T19:30:39.825 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132262.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.130340.log.gz 2023-10-10T19:30:39.825 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121930.log 2023-10-10T19:30:39.825 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.132262.log.gz 2023-10-10T19:30:39.826 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121315.log 2023-10-10T19:30:39.826 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122730.log 2023-10-10T19:30:39.826 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115776.log.gz 2023-10-10T19:30:39.827 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103824.log 2023-10-10T19:30:39.827 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121930.log.gz 2023-10-10T19:30:39.827 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.121315.log.gz 2023-10-10T19:30:39.828 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107016.log 2023-10-10T19:30:39.828 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99067.log 2023-10-10T19:30:39.828 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122730.log.gz 2023-10-10T19:30:39.828 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107016.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103824.log.gz 2023-10-10T19:30:39.829 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99537.log 2023-10-10T19:30:39.829 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.107016.log.gz 2023-10-10T19:30:39.829 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115091.log 2023-10-10T19:30:39.830 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99537.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99067.log.gz 2023-10-10T19:30:39.830 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122545.log 2023-10-10T19:30:39.830 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115091.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.99537.log.gz 2023-10-10T19:30:39.830 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107962.log 2023-10-10T19:30:39.831 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115091.log.gz 2023-10-10T19:30:39.831 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108392.log 2023-10-10T19:30:39.831 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107962.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122545.log.gz 2023-10-10T19:30:39.831 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103967.log 2023-10-10T19:30:39.832 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108392.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107962.log.gz 2023-10-10T19:30:39.832 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124024.log 2023-10-10T19:30:39.832 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103967.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108392.log.gz 2023-10-10T19:30:39.833 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-10T19:30:39.833 INFO:teuthology.orchestra.run.smithi175.stderr: 87.5%/var/log/ceph/ceph-client.admin.124024.log: -- replaced with /var/log/ceph/ceph-client.admin.103967.log.gz 2023-10-10T19:30:39.833 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130595.log 2023-10-10T19:30:39.833 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-mds.a.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.124024.log.gz 2023-10-10T19:30:39.834 INFO:teuthology.orchestra.run.smithi175.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.115175.log 2023-10-10T19:30:39.834 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117657.log 2023-10-10T19:30:39.834 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115175.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.130595.log.gz 2023-10-10T19:30:39.835 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95330.log 2023-10-10T19:30:39.835 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96585.log 2023-10-10T19:30:39.835 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115175.log.gz 2023-10-10T19:30:39.835 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95330.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117657.log.gz 2023-10-10T19:30:39.836 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127054.log 2023-10-10T19:30:39.836 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96585.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95330.log.gz 2023-10-10T19:30:39.836 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95978.log 2023-10-10T19:30:39.837 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96585.log.gz 2023-10-10T19:30:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118757.log 2023-10-10T19:30:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96583.log 2023-10-10T19:30:39.837 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.127054.log.gz 2023-10-10T19:30:39.838 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118757.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95978.log.gz 2023-10-10T19:30:39.838 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117790.log 2023-10-10T19:30:39.838 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.118757.log.gz 2023-10-10T19:30:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130957.log 2023-10-10T19:30:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117790.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96583.log.gz 2023-10-10T19:30:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128050.log 2023-10-10T19:30:39.839 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130957.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117790.log.gz 2023-10-10T19:30:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109587.log 2023-10-10T19:30:39.840 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.130957.log.gz 2023-10-10T19:30:39.840 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112472.log 2023-10-10T19:30:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113313.log 2023-10-10T19:30:39.841 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.128050.log.gz 2023-10-10T19:30:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112472.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.109587.log.gz 2023-10-10T19:30:39.841 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102306.log 2023-10-10T19:30:39.842 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112472.log.gz 2023-10-10T19:30:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102562.log 2023-10-10T19:30:39.842 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102306.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113313.log.gzgzip 2023-10-10T19:30:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.97299.log 2023-10-10T19:30:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102562.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102306.log.gz 2023-10-10T19:30:39.843 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96587.log 2023-10-10T19:30:39.843 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.102562.log.gz 2023-10-10T19:30:39.844 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123732.log 2023-10-10T19:30:39.844 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96587.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97299.log.gz 2023-10-10T19:30:39.844 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129696.log 2023-10-10T19:30:39.845 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123732.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96587.log.gz 2023-10-10T19:30:39.845 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117748.log 2023-10-10T19:30:39.845 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.123732.log.gz 2023-10-10T19:30:39.845 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104009.log 2023-10-10T19:30:39.846 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117748.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.129696.log.gzgzip 2023-10-10T19:30:39.846 INFO:teuthology.orchestra.run.smithi175.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.127610.log 2023-10-10T19:30:39.846 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104009.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.117748.log.gz 2023-10-10T19:30:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101515.log 2023-10-10T19:30:39.847 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104009.log.gz 2023-10-10T19:30:39.847 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95887.log 2023-10-10T19:30:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101515.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.127610.log.gz 2023-10-10T19:30:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102821.log 2023-10-10T19:30:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95887.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.101515.log.gz 2023-10-10T19:30:39.848 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104313.log 2023-10-10T19:30:39.849 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95887.log.gz 2023-10-10T19:30:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122503.log 2023-10-10T19:30:39.849 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124948.log 2023-10-10T19:30:39.850 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102821.log.gz 2023-10-10T19:30:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122503.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104313.log.gz 2023-10-10T19:30:39.850 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109192.log 2023-10-10T19:30:39.851 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122503.log.gz 2023-10-10T19:30:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125344.log 2023-10-10T19:30:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109192.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.124948.log.gz-5 2023-10-10T19:30:39.851 INFO:teuthology.orchestra.run.smithi175.stderr:--verbose -- /var/log/ceph/ceph-client.admin.131320.log 2023-10-10T19:30:39.852 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125344.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109192.log.gz 2023-10-10T19:30:39.852 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111264.log 2023-10-10T19:30:39.852 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.125344.log.gz 2023-10-10T19:30:39.853 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119076.log 2023-10-10T19:30:39.853 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111264.log: 88.5%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.131320.log.gz--verbose 2023-10-10T19:30:39.853 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.122250.log 2023-10-10T19:30:39.854 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.119076.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111264.log.gz 2023-10-10T19:30:39.854 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128915.log 2023-10-10T19:30:39.854 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119076.log.gz 2023-10-10T19:30:39.855 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95167.log 2023-10-10T19:30:39.855 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128915.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122250.log.gz 2023-10-10T19:30:39.855 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108493.log 2023-10-10T19:30:39.855 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95167.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128915.log.gz 2023-10-10T19:30:39.856 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130298.log 2023-10-10T19:30:39.856 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95167.log.gz 2023-10-10T19:30:39.856 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103182.log 2023-10-10T19:30:39.857 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124545.log 2023-10-10T19:30:39.857 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108493.log.gz 2023-10-10T19:30:39.857 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103182.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.130298.log.gz 2023-10-10T19:30:39.857 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118154.log 2023-10-10T19:30:39.858 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103182.log.gz 2023-10-10T19:30:39.858 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109108.log 2023-10-10T19:30:39.858 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118154.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124545.log.gz 2023-10-10T19:30:39.859 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111880.log 2023-10-10T19:30:39.859 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109108.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118154.log.gz 2023-10-10T19:30:39.859 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125007.log 2023-10-10T19:30:39.860 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109108.log.gz 2023-10-10T19:30:39.860 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116545.log 2023-10-10T19:30:39.860 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125007.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.111880.log.gz 2023-10-10T19:30:39.860 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107648.log 2023-10-10T19:30:39.861 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116545.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125007.log.gz 2023-10-10T19:30:39.861 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112116.log 2023-10-10T19:30:39.861 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2%/var/log/ceph/ceph-client.admin.107648.log: -- replaced with /var/log/ceph/ceph-client.admin.116545.log.gz 2023-10-10T19:30:39.862 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100764.log 2023-10-10T19:30:39.862 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112116.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107648.log.gz 2023-10-10T19:30:39.862 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129125.log 2023-10-10T19:30:39.863 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100764.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112116.log.gz 2023-10-10T19:30:39.863 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105287.log 2023-10-10T19:30:39.863 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100764.log.gz 2023-10-10T19:30:39.863 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116352.log 2023-10-10T19:30:39.864 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105287.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.129125.log.gz 2023-10-10T19:30:39.864 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97980.log 2023-10-10T19:30:39.864 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116352.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105287.log.gz 2023-10-10T19:30:39.865 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128092.log 2023-10-10T19:30:39.865 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116352.log.gz 2023-10-10T19:30:39.865 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112346.log 2023-10-10T19:30:39.865 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114426.log 2023-10-10T19:30:39.866 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112346.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.128092.log.gz 2023-10-10T19:30:39.866 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132220.log 2023-10-10T19:30:39.866 INFO:teuthology.orchestra.run.smithi175.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.97980.log.gz 2023-10-10T19:30:39.867 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112346.log.gz 2023-10-10T19:30:39.867 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107184.log 2023-10-10T19:30:39.867 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132220.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114426.log.gz 2023-10-10T19:30:39.867 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100806.log 2023-10-10T19:30:39.868 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130256.log 2023-10-10T19:30:39.868 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.132220.log.gz 2023-10-10T19:30:39.868 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107184.log.gz 2023-10-10T19:30:39.869 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123024.log 2023-10-10T19:30:39.869 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100806.log.gz 2023-10-10T19:30:39.869 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107710.log 2023-10-10T19:30:39.869 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123024.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.130256.log.gz 2023-10-10T19:30:39.870 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124402.log 2023-10-10T19:30:39.870 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.123024.log.gz 2023-10-10T19:30:39.870 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104941.log 2023-10-10T19:30:39.871 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107710.log.gz 2023-10-10T19:30:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99411.log 2023-10-10T19:30:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104941.log: gzip 88.2%-5 -- replaced with /var/log/ceph/ceph-client.admin.124402.log.gz--verbose 2023-10-10T19:30:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.103496.log 2023-10-10T19:30:39.872 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104941.log.gz 2023-10-10T19:30:39.872 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111922.log 2023-10-10T19:30:39.872 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99411.log.gz 2023-10-10T19:30:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108350.log 2023-10-10T19:30:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111922.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.103496.log.gz 2023-10-10T19:30:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108221.log 2023-10-10T19:30:39.873 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111922.log.gz 2023-10-10T19:30:39.874 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116115.log 2023-10-10T19:30:39.874 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108350.log.gz 2023-10-10T19:30:39.874 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121105.log 2023-10-10T19:30:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116115.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108221.log.gz 2023-10-10T19:30:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132529.log 2023-10-10T19:30:39.875 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.116115.log.gz 2023-10-10T19:30:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115133.log 2023-10-10T19:30:39.876 INFO:teuthology.orchestra.run.smithi175.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.121105.log.gz 2023-10-10T19:30:39.876 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95761.log 2023-10-10T19:30:39.876 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115133.log: gzip 87.7%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.132529.log.gz-- 2023-10-10T19:30:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131841.log 2023-10-10T19:30:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95761.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115133.log.gz 2023-10-10T19:30:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132304.log 2023-10-10T19:30:39.877 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95761.log.gz 2023-10-10T19:30:39.878 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95551.log 2023-10-10T19:30:39.878 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114646.log 2023-10-10T19:30:39.878 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.131841.log.gz 2023-10-10T19:30:39.879 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.132304.log.gz 2023-10-10T19:30:39.879 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119450.log 2023-10-10T19:30:39.879 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114646.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95551.log.gz 2023-10-10T19:30:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95593.log 2023-10-10T19:30:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.119450.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114646.log.gz 2023-10-10T19:30:39.880 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127882.log 2023-10-10T19:30:39.880 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119450.log.gz 2023-10-10T19:30:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123217.log 2023-10-10T19:30:39.881 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113918.log 2023-10-10T19:30:39.881 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.95593.log.gz 2023-10-10T19:30:39.882 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123217.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127882.log.gz 2023-10-10T19:30:39.882 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104271.log 2023-10-10T19:30:39.882 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123217.log.gz 2023-10-10T19:30:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110667.log 2023-10-10T19:30:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99243.log 2023-10-10T19:30:39.883 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113918.log.gz 2023-10-10T19:30:39.883 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110667.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104271.log.gz 2023-10-10T19:30:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109911.log 2023-10-10T19:30:39.884 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110667.log.gz 2023-10-10T19:30:39.884 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99495.log 2023-10-10T19:30:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109911.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.99243.log.gz 2023-10-10T19:30:39.885 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103073.log 2023-10-10T19:30:39.885 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109911.log.gz 2023-10-10T19:30:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130385.log 2023-10-10T19:30:39.886 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99495.log.gz 2023-10-10T19:30:39.886 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102772.log 2023-10-10T19:30:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130385.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.103073.log.gz 2023-10-10T19:30:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104541.log 2023-10-10T19:30:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102772.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.130385.log.gz 2023-10-10T19:30:39.887 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99025.log 2023-10-10T19:30:39.888 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102772.log.gz 2023-10-10T19:30:39.888 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130553.log 2023-10-10T19:30:39.888 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99025.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104541.log.gz 2023-10-10T19:30:39.889 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114772.log 2023-10-10T19:30:39.889 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130553.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99025.log.gz 2023-10-10T19:30:39.889 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132121.log 2023-10-10T19:30:39.889 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130553.log.gz 2023-10-10T19:30:39.890 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113103.log 2023-10-10T19:30:39.890 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132121.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114772.log.gz 2023-10-10T19:30:39.890 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116965.log 2023-10-10T19:30:39.891 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113103.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.132121.log.gz 2023-10-10T19:30:39.891 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127145.log 2023-10-10T19:30:39.891 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113103.log.gz 2023-10-10T19:30:39.891 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98344.log 2023-10-10T19:30:39.892 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116965.log.gz 2023-10-10T19:30:39.892 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114940.log 2023-10-10T19:30:39.892 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.98344.log: 87.8%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.127145.log.gz--verbose 2023-10-10T19:30:39.892 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.124108.log 2023-10-10T19:30:39.893 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.98344.log.gz 2023-10-10T19:30:39.893 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127103.log 2023-10-10T19:30:39.893 INFO:teuthology.orchestra.run.smithi175.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.114940.log.gz 2023-10-10T19:30:39.894 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124587.log 2023-10-10T19:30:39.894 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127103.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.124108.log.gz 2023-10-10T19:30:39.894 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116839.log 2023-10-10T19:30:39.895 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113540.log 2023-10-10T19:30:39.895 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.127103.log.gz 2023-10-10T19:30:39.895 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116839.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.124587.log.gz 2023-10-10T19:30:39.895 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99453.log 2023-10-10T19:30:39.896 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116839.log.gz 2023-10-10T19:30:39.896 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129268.log 2023-10-10T19:30:39.896 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99453.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113540.log.gz 2023-10-10T19:30:39.896 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112640.log 2023-10-10T19:30:39.897 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99453.log.gz 2023-10-10T19:30:39.897 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108046.log 2023-10-10T19:30:39.897 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.129268.log.gz 2023-10-10T19:30:39.898 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128873.log 2023-10-10T19:30:39.898 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129604.log 2023-10-10T19:30:39.898 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.112640.log.gz 2023-10-10T19:30:39.899 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128873.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108046.log.gz 2023-10-10T19:30:39.899 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115368.log 2023-10-10T19:30:39.899 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128873.log.gz 2023-10-10T19:30:39.899 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129646.log 2023-10-10T19:30:39.900 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115368.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129604.log.gz 2023-10-10T19:30:39.900 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131572.log 2023-10-10T19:30:39.900 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129646.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115368.log.gz 2023-10-10T19:30:39.900 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109629.log 2023-10-10T19:30:39.901 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129646.log.gz 2023-10-10T19:30:39.901 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129394.log 2023-10-10T19:30:39.901 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109629.log: gzip 89.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.131572.log.gz--verbose 2023-10-10T19:30:39.902 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.111180.log 2023-10-10T19:30:39.902 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129394.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109629.log.gz 2023-10-10T19:30:39.902 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111138.log 2023-10-10T19:30:39.903 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.129394.log.gz 2023-10-10T19:30:39.903 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111180.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-10T19:30:39.903 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111138.log: gzip -5 88.2%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.111180.log.gz/var/log/ceph/ceph-client.admin.130687.log 2023-10-10T19:30:39.903 INFO:teuthology.orchestra.run.smithi175.stderr: 2023-10-10T19:30:39.904 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111138.log.gz 2023-10-10T19:30:39.904 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130746.log 2023-10-10T19:30:39.904 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111484.log 2023-10-10T19:30:39.905 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130687.log: /var/log/ceph/ceph-client.admin.130746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127316.log 2023-10-10T19:30:39.905 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130687.log.gz 2023-10-10T19:30:39.905 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123322.log 2023-10-10T19:30:39.906 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130746.log.gz 2023-10-10T19:30:39.906 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127316.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.111484.log.gz 2023-10-10T19:30:39.906 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104735.log 2023-10-10T19:30:39.906 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.127316.log.gz 2023-10-10T19:30:39.907 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101305.log 2023-10-10T19:30:39.907 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123322.log: /var/log/ceph/ceph-client.admin.104735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112514.log 2023-10-10T19:30:39.907 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.123322.log.gz 2023-10-10T19:30:39.908 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101305.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104735.log.gz 2023-10-10T19:30:39.908 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112917.log 2023-10-10T19:30:39.908 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101305.log.gz 2023-10-10T19:30:39.908 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122419.log 2023-10-10T19:30:39.909 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113498.log 2023-10-10T19:30:39.909 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112917.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112514.log.gz 2023-10-10T19:30:39.909 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130123.log 2023-10-10T19:30:39.910 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112917.log.gz 2023-10-10T19:30:39.910 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122419.log.gz 2023-10-10T19:30:39.910 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103698.log 2023-10-10T19:30:39.910 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113498.log: /var/log/ceph/ceph-client.admin.130123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118883.log 2023-10-10T19:30:39.911 INFO:teuthology.orchestra.run.smithi175.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113498.log.gz 2023-10-10T19:30:39.911 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103698.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.130123.log.gz 2023-10-10T19:30:39.911 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98941.log 2023-10-10T19:30:39.912 INFO:teuthology.orchestra.run.smithi175.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103698.log.gz 2023-10-10T19:30:39.912 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125617.log 2023-10-10T19:30:39.912 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128705.log 2023-10-10T19:30:39.912 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.98941.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.118883.log.gz 2023-10-10T19:30:39.913 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113271.log 2023-10-10T19:30:39.913 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98941.log.gz 2023-10-10T19:30:39.913 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.125617.log.gz 2023-10-10T19:30:39.914 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105371.log 2023-10-10T19:30:39.914 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128705.log: /var/log/ceph/ceph-client.admin.113271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128957.log 2023-10-10T19:30:39.914 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.128705.log.gz 2023-10-10T19:30:39.914 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105371.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113271.log.gz 2023-10-10T19:30:39.915 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118378.log 2023-10-10T19:30:39.915 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105371.log.gz 2023-10-10T19:30:39.915 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131757.log 2023-10-10T19:30:39.916 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118028.log 2023-10-10T19:30:39.916 INFO:teuthology.orchestra.run.smithi175.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-10T19:30:39.916 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118378.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.128957.log.gz 2023-10-10T19:30:39.917 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128226.log 2023-10-10T19:30:39.917 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118378.log.gz 2023-10-10T19:30:39.917 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97475.log 2023-10-10T19:30:39.917 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.131757.log.gz 2023-10-10T19:30:39.918 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118028.log.gz 2023-10-10T19:30:39.918 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124864.log 2023-10-10T19:30:39.918 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.128226.log.gz 2023-10-10T19:30:39.919 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122856.log 2023-10-10T19:30:39.919 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111355.log 2023-10-10T19:30:39.919 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.97475.log.gz 2023-10-10T19:30:39.919 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122856.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.124864.log.gz 2023-10-10T19:30:39.920 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129878.log 2023-10-10T19:30:39.920 INFO:teuthology.orchestra.run.smithi175.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122856.log.gz 2023-10-10T19:30:39.920 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127924.log 2023-10-10T19:30:39.920 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111355.log: /var/log/ceph/ceph-client.admin.129878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119286.log 2023-10-10T19:30:39.921 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.111355.log.gz 2023-10-10T19:30:39.921 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.129878.log.gz 2023-10-10T19:30:39.921 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113834.log 2023-10-10T19:30:39.921 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.127924.log.gz 2023-10-10T19:30:39.922 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.119286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111627.log 2023-10-10T19:30:39.922 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131967.log 2023-10-10T19:30:39.922 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119286.log.gz 2023-10-10T19:30:39.923 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113834.log.gz 2023-10-10T19:30:39.923 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126861.log 2023-10-10T19:30:39.923 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111627.log.gz 2023-10-10T19:30:39.923 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130011.log 2023-10-10T19:30:39.924 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126861.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.131967.log.gz 2023-10-10T19:30:39.924 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101623.log 2023-10-10T19:30:39.924 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130011.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.126861.log.gz 2023-10-10T19:30:39.925 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115625.log 2023-10-10T19:30:39.925 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130011.log.gz 2023-10-10T19:30:39.925 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99159.log 2023-10-10T19:30:39.926 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115625.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.101623.log.gz-5 2023-10-10T19:30:39.926 INFO:teuthology.orchestra.run.smithi175.stderr:--verbose -- /var/log/ceph/ceph-client.admin.96254.log 2023-10-10T19:30:39.926 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99159.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115625.log.gz 2023-10-10T19:30:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127274.log 2023-10-10T19:30:39.927 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99159.log.gz 2023-10-10T19:30:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111397.log 2023-10-10T19:30:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127274.log: 88.2%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.96254.log.gz--verbose 2023-10-10T19:30:39.928 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.98178.log 2023-10-10T19:30:39.928 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127274.log.gz 2023-10-10T19:30:39.928 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121762.log 2023-10-10T19:30:39.929 INFO:teuthology.orchestra.run.smithi175.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.111397.log.gz 2023-10-10T19:30:39.929 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.98178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128327.log 2023-10-10T19:30:39.929 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117150.log 2023-10-10T19:30:39.930 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.121762.log.gz 2023-10-10T19:30:39.930 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130830.log 2023-10-10T19:30:39.930 INFO:teuthology.orchestra.run.smithi175.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.98178.log.gz 2023-10-10T19:30:39.930 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104583.log 2023-10-10T19:30:39.931 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117150.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.128327.log.gz 2023-10-10T19:30:39.931 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112766.log 2023-10-10T19:30:39.931 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117150.log.gz 2023-10-10T19:30:39.931 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131799.log 2023-10-10T19:30:39.932 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130830.log.gz 2023-10-10T19:30:39.932 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112766.log: gzip -5 89.3%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.104583.log.gz/var/log/ceph/ceph-client.admin.130872.log 2023-10-10T19:30:39.932 INFO:teuthology.orchestra.run.smithi175.stderr: 2023-10-10T19:30:39.933 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131799.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112766.log.gz 2023-10-10T19:30:39.933 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122335.log 2023-10-10T19:30:39.933 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130872.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131799.log.gz 2023-10-10T19:30:39.934 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111711.log 2023-10-10T19:30:39.934 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122335.log: gzip -5 88.6%--verbose -- /var/log/ceph/ceph-client.admin.128579.log -- replaced with /var/log/ceph/ceph-client.admin.130872.log.gz 2023-10-10T19:30:39.934 INFO:teuthology.orchestra.run.smithi175.stderr: 2023-10-10T19:30:39.935 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111711.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122335.log.gz 2023-10-10T19:30:39.935 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127840.log 2023-10-10T19:30:39.935 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111711.log.gz 2023-10-10T19:30:39.935 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121448.log 2023-10-10T19:30:39.936 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127840.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.128579.log.gz 2023-10-10T19:30:39.936 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122688.log 2023-10-10T19:30:39.936 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121448.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.127840.log.gz 2023-10-10T19:30:39.937 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115988.log 2023-10-10T19:30:39.937 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.121448.log.gz 2023-10-10T19:30:39.937 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119034.log 2023-10-10T19:30:39.938 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115988.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122688.log.gz 2023-10-10T19:30:39.938 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102905.log 2023-10-10T19:30:39.938 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115988.log.gz 2023-10-10T19:30:39.938 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.119034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110264.log 2023-10-10T19:30:39.939 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119034.log.gz 2023-10-10T19:30:39.939 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115304.log 2023-10-10T19:30:39.939 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110264.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102905.log.gz 2023-10-10T19:30:39.940 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95635.log 2023-10-10T19:30:39.940 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115304.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110264.log.gz 2023-10-10T19:30:39.940 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112724.log 2023-10-10T19:30:39.940 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115304.log.gz 2023-10-10T19:30:39.941 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126734.log 2023-10-10T19:30:39.941 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112724.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95635.log.gz 2023-10-10T19:30:39.941 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126510.log 2023-10-10T19:30:39.942 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126734.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112724.log.gz 2023-10-10T19:30:39.942 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107920.log 2023-10-10T19:30:39.942 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126734.log.gz 2023-10-10T19:30:39.942 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115541.log 2023-10-10T19:30:39.943 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116881.log 2023-10-10T19:30:39.943 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.126510.log.gz 2023-10-10T19:30:39.943 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115541.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.107920.log.gz 2023-10-10T19:30:39.944 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-10T19:30:39.944 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.115541.log.gz 2023-10-10T19:30:39.944 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95509.log 2023-10-10T19:30:39.944 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph.audit.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116881.log.gz 2023-10-10T19:30:39.945 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100596.log 2023-10-10T19:30:39.945 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107100.log 2023-10-10T19:30:39.945 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95509.log.gz 2023-10-10T19:30:39.946 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101179.log 2023-10-10T19:30:39.946 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100596.log.gz 2023-10-10T19:30:39.946 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114241.log 2023-10-10T19:30:39.946 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107100.log: /var/log/ceph/ceph-client.admin.101179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119118.log 2023-10-10T19:30:39.947 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107100.log.gz 2023-10-10T19:30:39.947 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114241.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101179.log.gz 2023-10-10T19:30:39.947 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97391.log 2023-10-10T19:30:39.948 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114241.log.gz 2023-10-10T19:30:39.948 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.119118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116031.log 2023-10-10T19:30:39.948 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124150.log 2023-10-10T19:30:39.949 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97391.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119118.log.gz 2023-10-10T19:30:39.949 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-10T19:30:39.949 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95067.log 2023-10-10T19:30:39.949 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97391.log.gz 2023-10-10T19:30:39.950 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116031.log.gz 2023-10-10T19:30:39.950 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122940.log 2023-10-10T19:30:39.950 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.124150.log.gz 2023-10-10T19:30:39.951 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104142.log 2023-10-10T19:30:39.951 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7%/var/log/ceph/ceph-client.admin.122940.log: -- replaced with /var/log/ceph/ceph-client.admin.95067.log.gz 2023-10-10T19:30:39.951 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105839.log 2023-10-10T19:30:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114898.log 2023-10-10T19:30:39.952 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.122940.log.gz 2023-10-10T19:30:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105839.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104142.log.gz 2023-10-10T19:30:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128134.log 2023-10-10T19:30:39.953 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105839.log.gz 2023-10-10T19:30:39.953 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117573.log 2023-10-10T19:30:39.953 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128134.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114898.log.gz 2023-10-10T19:30:39.954 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104815.log 2023-10-10T19:30:39.954 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117573.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.128134.log.gz 2023-10-10T19:30:39.954 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108619.log 2023-10-10T19:30:39.954 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117573.log.gz 2023-10-10T19:30:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101025.log 2023-10-10T19:30:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108619.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104815.log.gz 2023-10-10T19:30:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126426.log 2023-10-10T19:30:39.956 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101025.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108619.log.gz 2023-10-10T19:30:39.956 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125175.log 2023-10-10T19:30:39.956 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101025.log.gz 2023-10-10T19:30:39.956 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98112.log 2023-10-10T19:30:39.957 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109318.log 2023-10-10T19:30:39.957 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.126426.log.gz 2023-10-10T19:30:39.957 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.98112.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.125175.log.gz 2023-10-10T19:30:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104983.log 2023-10-10T19:30:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132346.log 2023-10-10T19:30:39.958 INFO:teuthology.orchestra.run.smithi175.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.98112.log.gz 2023-10-10T19:30:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104983.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109318.log.gz 2023-10-10T19:30:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98046.log 2023-10-10T19:30:39.959 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104983.log.gz 2023-10-10T19:30:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118070.log 2023-10-10T19:30:39.960 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132346.log: /var/log/ceph/ceph-client.admin.98046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131362.log 2023-10-10T19:30:39.960 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.132346.log.gz 2023-10-10T19:30:39.960 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108898.log 2023-10-10T19:30:39.960 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131362.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.98046.log.gz 2023-10-10T19:30:39.961 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118070.log.gz 2023-10-10T19:30:39.961 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100848.log 2023-10-10T19:30:39.961 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99201.log 2023-10-10T19:30:39.961 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108898.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.131362.log.gz 2023-10-10T19:30:39.962 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112556.log 2023-10-10T19:30:39.962 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108898.log.gz 2023-10-10T19:30:39.962 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99201.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100848.log.gz 2023-10-10T19:30:39.963 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119160.log 2023-10-10T19:30:39.963 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112556.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99201.log.gz 2023-10-10T19:30:39.963 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97348.log 2023-10-10T19:30:39.964 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8%/var/log/ceph/ceph-client.admin.119160.log: -- replaced with /var/log/ceph/ceph-client.admin.112556.log.gz 2023-10-10T19:30:39.964 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131614.log 2023-10-10T19:30:39.964 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.119160.log.gz 2023-10-10T19:30:39.964 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97129.log 2023-10-10T19:30:39.965 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124192.log 2023-10-10T19:30:39.965 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97348.log.gz 2023-10-10T19:30:39.965 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97129.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131614.log.gz 2023-10-10T19:30:39.965 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108661.log 2023-10-10T19:30:39.966 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97129.log.gz 2023-10-10T19:30:39.966 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98899.log 2023-10-10T19:30:39.966 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108661.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124192.log.gz 2023-10-10T19:30:39.967 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131488.log 2023-10-10T19:30:39.967 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.98899.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108661.log.gz 2023-10-10T19:30:39.967 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105472.log 2023-10-10T19:30:39.967 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98899.log.gz 2023-10-10T19:30:39.968 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109419.log 2023-10-10T19:30:39.968 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105472.log: 88.6%gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.131488.log.gz-- 2023-10-10T19:30:39.968 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109545.log 2023-10-10T19:30:39.969 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109419.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.105472.log.gz 2023-10-10T19:30:39.969 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117108.log 2023-10-10T19:30:39.969 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109419.log.gz 2023-10-10T19:30:39.969 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128831.log 2023-10-10T19:30:39.970 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117108.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.109545.log.gz-5 2023-10-10T19:30:39.970 INFO:teuthology.orchestra.run.smithi175.stderr:--verbose -- /var/log/ceph/ceph-client.admin.122461.log 2023-10-10T19:30:39.970 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128831.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117108.log.gz 2023-10-10T19:30:39.971 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97517.log 2023-10-10T19:30:39.971 INFO:teuthology.orchestra.run.smithi175.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.128831.log.gz 2023-10-10T19:30:39.971 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122461.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-10T19:30:39.971 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97517.log: gzip -5 --verbose -- 88.2%/var/log/ceph/ceph-client.admin.118505.log 2023-10-10T19:30:39.972 INFO:teuthology.orchestra.run.smithi175.stderr: -- replaced with /var/log/ceph/ceph-client.admin.122461.log.gz 2023-10-10T19:30:39.972 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.97517.log.gz 2023-10-10T19:30:39.972 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112388.log 2023-10-10T19:30:39.973 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110390.log 2023-10-10T19:30:39.973 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112388.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118505.log.gz 2023-10-10T19:30:39.973 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108577.log 2023-10-10T19:30:39.973 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126061.log 2023-10-10T19:30:39.974 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110390.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.112388.log.gz 2023-10-10T19:30:39.974 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108577.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110390.log.gz 2023-10-10T19:30:39.974 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129520.log 2023-10-10T19:30:39.975 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108577.log.gz 2023-10-10T19:30:39.975 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113750.log 2023-10-10T19:30:39.975 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126061.log: /var/log/ceph/ceph-client.admin.129520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96918.log 2023-10-10T19:30:39.975 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126061.log.gz 2023-10-10T19:30:39.976 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113750.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.129520.log.gz 2023-10-10T19:30:39.976 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117944.log 2023-10-10T19:30:39.976 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.113750.log.gz 2023-10-10T19:30:39.977 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130915.log 2023-10-10T19:30:39.977 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121678.log 2023-10-10T19:30:39.977 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117944.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96918.log.gz 2023-10-10T19:30:39.977 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112161.log 2023-10-10T19:30:39.978 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.117944.log.gz 2023-10-10T19:30:39.978 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130915.log.gz 2023-10-10T19:30:39.978 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101263.log 2023-10-10T19:30:39.979 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121678.log: /var/log/ceph/ceph-client.admin.112161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100890.log 2023-10-10T19:30:39.979 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121678.log.gz 2023-10-10T19:30:39.979 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101263.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.112161.log.gz 2023-10-10T19:30:39.979 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125386.log 2023-10-10T19:30:39.980 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101263.log.gz 2023-10-10T19:30:39.980 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109839.log 2023-10-10T19:30:39.980 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113708.log 2023-10-10T19:30:39.980 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125386.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100890.log.gz 2023-10-10T19:30:39.981 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128999.log 2023-10-10T19:30:39.981 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.125386.log.gz 2023-10-10T19:30:39.981 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109839.log.gz 2023-10-10T19:30:39.982 INFO:teuthology.orchestra.run.smithi175.stderr:gzip /var/log/ceph/ceph-client.admin.113708.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.118715.log 2023-10-10T19:30:39.982 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105200.log 2023-10-10T19:30:39.982 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.113708.log.gz 2023-10-10T19:30:39.983 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118715.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.128999.log.gz 2023-10-10T19:30:39.983 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114730.log 2023-10-10T19:30:39.983 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118715.log.gz 2023-10-10T19:30:39.983 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115583.log 2023-10-10T19:30:39.984 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130165.log 2023-10-10T19:30:39.984 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114730.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105200.log.gz 2023-10-10T19:30:39.984 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107588.log 2023-10-10T19:30:39.985 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114730.log.gz 2023-10-10T19:30:39.985 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.115583.log.gz 2023-10-10T19:30:39.985 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108451.log 2023-10-10T19:30:39.985 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130165.log: /var/log/ceph/ceph-client.admin.107588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118673.log 2023-10-10T19:30:39.986 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130165.log.gz 2023-10-10T19:30:39.986 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108451.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107588.log.gz 2023-10-10T19:30:39.986 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122377.log 2023-10-10T19:30:39.987 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108451.log.gz 2023-10-10T19:30:39.987 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130645.log 2023-10-10T19:30:39.987 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114601.log 2023-10-10T19:30:39.987 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122377.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.118673.log.gz 2023-10-10T19:30:39.988 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96920.log 2023-10-10T19:30:39.988 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.122377.log.gz 2023-10-10T19:30:39.988 INFO:teuthology.orchestra.run.smithi175.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.130645.log.gz 2023-10-10T19:30:39.989 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107461.log 2023-10-10T19:30:39.989 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114601.log: /var/log/ceph/ceph-client.admin.96920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100554.log 2023-10-10T19:30:39.989 INFO:teuthology.orchestra.run.smithi175.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.114601.log.gz 2023-10-10T19:30:39.989 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107461.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96920.log.gz 2023-10-10T19:30:39.990 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110625.log 2023-10-10T19:30:39.990 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107461.log.gz 2023-10-10T19:30:39.990 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110054.log 2023-10-10T19:30:39.991 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128537.log 2023-10-10T19:30:39.991 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110625.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100554.log.gz 2023-10-10T19:30:39.991 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109461.log 2023-10-10T19:30:39.991 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110625.log.gz 2023-10-10T19:30:39.992 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110054.log.gz 2023-10-10T19:30:39.992 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128789.log 2023-10-10T19:30:39.992 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128537.log: /var/log/ceph/ceph-client.admin.109461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129816.log 2023-10-10T19:30:39.993 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.128537.log.gz 2023-10-10T19:30:39.993 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128789.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109461.log.gz 2023-10-10T19:30:39.993 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122898.log 2023-10-10T19:30:39.994 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.128789.log.gz 2023-10-10T19:30:39.994 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103400.log 2023-10-10T19:30:39.994 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116461.log 2023-10-10T19:30:39.994 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122898.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.129816.log.gz 2023-10-10T19:30:39.995 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101473.log 2023-10-10T19:30:39.995 INFO:teuthology.orchestra.run.smithi175.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.122898.log.gz 2023-10-10T19:30:39.995 INFO:teuthology.orchestra.run.smithi175.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103400.log.gz 2023-10-10T19:30:39.996 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123839.log 2023-10-10T19:30:39.996 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116461.log: /var/log/ceph/ceph-client.admin.101473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108535.log 2023-10-10T19:30:39.996 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116461.log.gz 2023-10-10T19:30:39.996 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123839.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.101473.log.gz 2023-10-10T19:30:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112245.log 2023-10-10T19:30:39.997 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123839.log.gz 2023-10-10T19:30:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97433.log 2023-10-10T19:30:39.998 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118462.log 2023-10-10T19:30:39.998 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112245.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108535.log.gz 2023-10-10T19:30:39.998 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131152.log 2023-10-10T19:30:39.999 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.112245.log.gz 2023-10-10T19:30:39.999 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97433.log.gz 2023-10-10T19:30:39.999 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111753.log 2023-10-10T19:30:39.999 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118462.log: /var/log/ceph/ceph-client.admin.131152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115862.log 2023-10-10T19:30:40.000 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118462.log.gz 2023-10-10T19:30:40.000 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111753.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.131152.log.gz 2023-10-10T19:30:40.000 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97171.log 2023-10-10T19:30:40.000 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.111753.log.gz 2023-10-10T19:30:40.001 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118547.log 2023-10-10T19:30:40.001 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116157.log 2023-10-10T19:30:40.001 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97171.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115862.log.gz 2023-10-10T19:30:40.002 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118841.log 2023-10-10T19:30:40.002 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97171.log.gz 2023-10-10T19:30:40.002 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118547.log.gz 2023-10-10T19:30:40.002 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100941.log 2023-10-10T19:30:40.003 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116157.log: /var/log/ceph/ceph-client.admin.118841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105025.log 2023-10-10T19:30:40.003 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116157.log.gz 2023-10-10T19:30:40.003 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100941.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118841.log.gz 2023-10-10T19:30:40.004 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108179.log 2023-10-10T19:30:40.004 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100941.log.gz 2023-10-10T19:30:40.004 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113187.log 2023-10-10T19:30:40.004 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110138.log 2023-10-10T19:30:40.005 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108179.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105025.log.gz 2023-10-10T19:30:40.005 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116310.log 2023-10-10T19:30:40.005 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108179.log.gz 2023-10-10T19:30:40.006 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113187.log.gz 2023-10-10T19:30:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:gzip /var/log/ceph/ceph-client.admin.110138.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.95213.log 2023-10-10T19:30:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95677.log 2023-10-10T19:30:40.006 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110138.log.gz 2023-10-10T19:30:40.007 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95213.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116310.log.gz 2023-10-10T19:30:40.007 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114283.log 2023-10-10T19:30:40.007 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95213.log.gz 2023-10-10T19:30:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96923.log 2023-10-10T19:30:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123461.log 2023-10-10T19:30:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114283.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95677.log.gz 2023-10-10T19:30:40.008 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101087.log 2023-10-10T19:30:40.009 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114283.log.gz 2023-10-10T19:30:40.009 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96923.log.gz 2023-10-10T19:30:40.009 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115692.log 2023-10-10T19:30:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123461.log: /var/log/ceph/ceph-client.admin.101087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125428.log 2023-10-10T19:30:40.010 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.123461.log.gz 2023-10-10T19:30:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115692.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.101087.log.gz 2023-10-10T19:30:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113792.log 2023-10-10T19:30:40.011 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115692.log.gz 2023-10-10T19:30:40.011 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126146.log 2023-10-10T19:30:40.011 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117615.log 2023-10-10T19:30:40.012 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113792.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125428.log.gz 2023-10-10T19:30:40.012 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95845.log 2023-10-10T19:30:40.012 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113792.log.gz 2023-10-10T19:30:40.012 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126146.log.gz 2023-10-10T19:30:40.013 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111306.log 2023-10-10T19:30:40.013 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117615.log: /var/log/ceph/ceph-client.admin.95845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110348.log 2023-10-10T19:30:40.013 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117615.log.gz 2023-10-10T19:30:40.014 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111306.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95845.log.gz 2023-10-10T19:30:40.014 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115049.log 2023-10-10T19:30:40.014 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111306.log.gz 2023-10-10T19:30:40.015 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131656.log 2023-10-10T19:30:40.015 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98983.log 2023-10-10T19:30:40.015 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115049.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110348.log.gz 2023-10-10T19:30:40.015 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131656.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115049.log.gzgzip 2023-10-10T19:30:40.016 INFO:teuthology.orchestra.run.smithi175.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.116587.log 2023-10-10T19:30:40.016 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.131656.log.gz 2023-10-10T19:30:40.016 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124822.log 2023-10-10T19:30:40.017 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.98983.log: /var/log/ceph/ceph-client.admin.116587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103266.log 2023-10-10T19:30:40.017 INFO:teuthology.orchestra.run.smithi175.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.98983.log.gz 2023-10-10T19:30:40.017 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124822.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116587.log.gz 2023-10-10T19:30:40.017 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114384.log 2023-10-10T19:30:40.018 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124822.log.gz 2023-10-10T19:30:40.018 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113582.log 2023-10-10T19:30:40.018 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125785.log 2023-10-10T19:30:40.019 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103266.log.gz 2023-10-10T19:30:40.019 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114384.log: /var/log/ceph/ceph-client.admin.113582.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114384.log.gz 2023-10-10T19:30:40.019 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121147.log 2023-10-10T19:30:40.019 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113582.log.gz 2023-10-10T19:30:40.020 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118281.log 2023-10-10T19:30:40.020 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125785.log: /var/log/ceph/ceph-client.admin.121147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99285.log 2023-10-10T19:30:40.020 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125785.log.gz 2023-10-10T19:30:40.021 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118281.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.121147.log.gz 2023-10-10T19:30:40.021 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124780.log 2023-10-10T19:30:40.021 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.118281.log.gz 2023-10-10T19:30:40.021 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101431.log 2023-10-10T19:30:40.022 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114814.log 2023-10-10T19:30:40.022 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124780.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99285.log.gz 2023-10-10T19:30:40.022 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96572.log 2023-10-10T19:30:40.023 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.124780.log.gz 2023-10-10T19:30:40.023 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101431.log.gz 2023-10-10T19:30:40.023 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114002.log 2023-10-10T19:30:40.023 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129562.log 2023-10-10T19:30:40.024 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114814.log.gz 2023-10-10T19:30:40.024 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114002.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96572.log.gz 2023-10-10T19:30:40.024 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124276.log 2023-10-10T19:30:40.024 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114002.log.gz 2023-10-10T19:30:40.025 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111096.log 2023-10-10T19:30:40.025 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121888.log 2023-10-10T19:30:40.025 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.129562.log.gz 2023-10-10T19:30:40.026 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124276.log: /var/log/ceph/ceph-client.admin.111096.log: gzip -5 88.6%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.124276.log.gz/var/log/ceph/ceph-client.admin.108004.log 2023-10-10T19:30:40.026 INFO:teuthology.orchestra.run.smithi175.stderr: 2023-10-10T19:30:40.026 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111096.log.gz 2023-10-10T19:30:40.026 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98244.log 2023-10-10T19:30:40.027 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121888.log: /var/log/ceph/ceph-client.admin.108004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131925.log 2023-10-10T19:30:40.027 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121888.log.gz 2023-10-10T19:30:40.027 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.98244.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.108004.log.gz 2023-10-10T19:30:40.027 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110474.log 2023-10-10T19:30:40.028 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108263.log 2023-10-10T19:30:40.028 INFO:teuthology.orchestra.run.smithi175.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.98244.log.gz 2023-10-10T19:30:40.028 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110474.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.131925.log.gz 2023-10-10T19:30:40.029 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97685.log 2023-10-10T19:30:40.029 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108263.log: gzip 89.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.110474.log.gz--verbose 2023-10-10T19:30:40.029 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.126233.log 2023-10-10T19:30:40.029 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97643.log 2023-10-10T19:30:40.030 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.108263.log.gz 2023-10-10T19:30:40.030 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97685.log.gz 2023-10-10T19:30:40.030 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115734.log 2023-10-10T19:30:40.031 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114199.log 2023-10-10T19:30:40.031 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97643.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126233.log.gz 2023-10-10T19:30:40.031 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115734.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97643.log.gz 2023-10-10T19:30:40.031 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131883.log 2023-10-10T19:30:40.032 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115734.log.gz 2023-10-10T19:30:40.032 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96575.log 2023-10-10T19:30:40.032 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114199.log: /var/log/ceph/ceph-client.admin.131883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116260.log 2023-10-10T19:30:40.033 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114199.log.gz 2023-10-10T19:30:40.033 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131883.log.gz 2023-10-10T19:30:40.033 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122056.log 2023-10-10T19:30:40.033 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96575.log.gz 2023-10-10T19:30:40.034 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127669.log 2023-10-10T19:30:40.034 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118112.log 2023-10-10T19:30:40.034 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116260.log.gz 2023-10-10T19:30:40.035 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122056.log: /var/log/ceph/ceph-client.admin.127669.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122056.log.gz 2023-10-10T19:30:40.035 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121972.log 2023-10-10T19:30:40.035 INFO:teuthology.orchestra.run.smithi175.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.127669.log.gz 2023-10-10T19:30:40.035 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96925.log 2023-10-10T19:30:40.036 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118112.log: /var/log/ceph/ceph-client.admin.121972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96245.log 2023-10-10T19:30:40.036 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118112.log.gz 2023-10-10T19:30:40.036 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96925.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.121972.log.gz 2023-10-10T19:30:40.037 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115457.log 2023-10-10T19:30:40.037 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96925.log.gz 2023-10-10T19:30:40.037 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108745.log 2023-10-10T19:30:40.038 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116503.log 2023-10-10T19:30:40.038 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115457.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96245.log.gz 2023-10-10T19:30:40.038 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108745.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.115457.log.gz 2023-10-10T19:30:40.038 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117024.log 2023-10-10T19:30:40.039 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108745.log.gz 2023-10-10T19:30:40.039 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115904.log 2023-10-10T19:30:40.039 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116503.log: /var/log/ceph/ceph-client.admin.117024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110180.log 2023-10-10T19:30:40.040 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116503.log.gz 2023-10-10T19:30:40.040 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115904.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117024.log.gz 2023-10-10T19:30:40.040 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100638.log 2023-10-10T19:30:40.041 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115904.log.gz 2023-10-10T19:30:40.041 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123587.log 2023-10-10T19:30:40.041 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107226.log 2023-10-10T19:30:40.041 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100638.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110180.log.gz 2023-10-10T19:30:40.042 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128747.log 2023-10-10T19:30:40.042 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100638.log.gz 2023-10-10T19:30:40.042 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123587.log.gz 2023-10-10T19:30:40.043 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115946.log 2023-10-10T19:30:40.043 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107226.log: /var/log/ceph/ceph-client.admin.128747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125259.log 2023-10-10T19:30:40.043 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107226.log.gz 2023-10-10T19:30:40.043 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115946.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128747.log.gz 2023-10-10T19:30:40.044 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105329.log 2023-10-10T19:30:40.044 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115946.log.gz 2023-10-10T19:30:40.044 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105413.log 2023-10-10T19:30:40.045 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122098.log 2023-10-10T19:30:40.045 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105329.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125259.log.gz 2023-10-10T19:30:40.045 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105413.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105329.log.gz 2023-10-10T19:30:40.045 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-10T19:30:40.046 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105413.log.gz 2023-10-10T19:30:40.046 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104633.log 2023-10-10T19:30:40.046 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122098.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121273.log 2023-10-10T19:30:40.047 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122098.log.gz 2023-10-10T19:30:40.047 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101129.log 2023-10-10T19:30:40.047 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104633.log.gz 2023-10-10T19:30:40.047 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117489.log 2023-10-10T19:30:40.048 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117066.log 2023-10-10T19:30:40.048 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101129.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121273.log.gz 2023-10-10T19:30:40.048 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110306.log 2023-10-10T19:30:40.049 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101129.log.gz 2023-10-10T19:30:40.049 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123066.log 2023-10-10T19:30:40.049 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117489.log.gz 2023-10-10T19:30:40.049 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117066.log.gz 2023-10-10T19:30:40.050 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101221.log 2023-10-10T19:30:40.050 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126104.log 2023-10-10T19:30:40.050 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123066.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110306.log.gz 2023-10-10T19:30:40.051 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101221.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-10T19:30:40.051 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.123066.log.gz 2023-10-10T19:30:40.051 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126104.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101221.log.gz 2023-10-10T19:30:40.051 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95803.log 2023-10-10T19:30:40.052 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-osd.1.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.126104.log.gz 2023-10-10T19:30:40.052 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123545.log 2023-10-10T19:30:40.052 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132484.log 2023-10-10T19:30:40.053 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123545.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95803.log.gz 2023-10-10T19:30:40.053 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132484.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123545.log.gz 2023-10-10T19:30:40.053 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.132484.log.gz 2023-10-10T19:30:40.053 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126571.log 2023-10-10T19:30:40.054 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105245.log 2023-10-10T19:30:40.054 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125091.log 2023-10-10T19:30:40.054 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105245.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126571.log.gz 2023-10-10T19:30:40.055 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125091.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105245.log.gz 2023-10-10T19:30:40.055 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125091.log.gz 2023-10-10T19:30:40.055 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125555.log 2023-10-10T19:30:40.056 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123898.log 2023-10-10T19:30:40.056 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123940.log 2023-10-10T19:30:40.057 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123898.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.125555.log.gz 2023-10-10T19:30:40.057 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123940.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123898.log.gz 2023-10-10T19:30:40.057 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123940.log.gz 2023-10-10T19:30:40.058 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107058.log 2023-10-10T19:30:40.058 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97811.log 2023-10-10T19:30:40.058 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130511.log 2023-10-10T19:30:40.059 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97811.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.107058.log.gz 2023-10-10T19:30:40.059 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130511.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.97811.log.gz 2023-10-10T19:30:40.059 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104187.log 2023-10-10T19:30:40.059 INFO:teuthology.orchestra.run.smithi175.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.130511.log.gz 2023-10-10T19:30:40.060 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114517.log 2023-10-10T19:30:40.060 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130788.log 2023-10-10T19:30:40.060 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114517.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104187.log.gz 2023-10-10T19:30:40.061 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130788.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114517.log.gz 2023-10-10T19:30:40.061 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.130788.log.gz 2023-10-10T19:30:40.061 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116797.log 2023-10-10T19:30:40.062 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127187.log 2023-10-10T19:30:40.062 INFO:teuthology.orchestra.run.smithi175.stderr:gzip /var/log/ceph/ceph-client.admin.116797.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.110782.log 2023-10-10T19:30:40.062 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127187.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116797.log.gz 2023-10-10T19:30:40.063 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110782.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127187.log.gz 2023-10-10T19:30:40.063 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110782.log.gz 2023-10-10T19:30:40.063 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123133.log 2023-10-10T19:30:40.064 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110012.log 2023-10-10T19:30:40.064 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125701.log 2023-10-10T19:30:40.064 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 /var/log/ceph/ceph-client.admin.110012.log: --verbose -- /var/log/ceph/ceph-client.admin.111795.log 2023-10-10T19:30:40.065 INFO:teuthology.orchestra.run.smithi175.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.123133.log.gz 2023-10-10T19:30:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125701.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110012.log.gz 2023-10-10T19:30:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111795.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.125701.log.gz 2023-10-10T19:30:40.065 INFO:teuthology.orchestra.run.smithi175.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.111795.log.gz 2023-10-10T19:30:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109671.log 2023-10-10T19:30:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114688.log 2023-10-10T19:30:40.066 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131278.log 2023-10-10T19:30:40.067 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114688.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109671.log.gz 2023-10-10T19:30:40.067 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131278.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.114688.log.gz 2023-10-10T19:30:40.067 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.131278.log.gz 2023-10-10T19:30:40.067 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118420.log 2023-10-10T19:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131530.log 2023-10-10T19:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116394.log 2023-10-10T19:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131530.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.118420.log.gz 2023-10-10T19:30:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116394.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.131530.log.gz 2023-10-10T19:30:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104675.log 2023-10-10T19:30:40.069 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116394.log.gz 2023-10-10T19:30:40.069 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90848.log 2023-10-10T19:30:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131698.log 2023-10-10T19:30:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.90848.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104675.log.gz 2023-10-10T19:30:40.070 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131698.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90848.log.gz 2023-10-10T19:30:40.071 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131698.log.gz 2023-10-10T19:30:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101681.log 2023-10-10T19:30:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96924.log 2023-10-10T19:30:40.071 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118589.log 2023-10-10T19:30:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96924.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.101681.log.gz 2023-10-10T19:30:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118589.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96924.log.gz 2023-10-10T19:30:40.072 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117404.log 2023-10-10T19:30:40.073 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118589.log.gz 2023-10-10T19:30:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122014.log 2023-10-10T19:30:40.073 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110970.log 2023-10-10T19:30:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122014.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117404.log.gz 2023-10-10T19:30:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.110970.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.122014.log.gz 2023-10-10T19:30:40.074 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110970.log.gz 2023-10-10T19:30:40.074 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109360.log 2023-10-10T19:30:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106974.log 2023-10-10T19:30:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126633.log 2023-10-10T19:30:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.106974.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109360.log.gz 2023-10-10T19:30:40.076 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126633.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106974.log.gz 2023-10-10T19:30:40.076 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126633.log.gz 2023-10-10T19:30:40.076 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95465.log 2023-10-10T19:30:40.077 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96037.log 2023-10-10T19:30:40.077 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.95465.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-10T19:30:40.077 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96037.log: 88.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.95465.log.gz--verbose 2023-10-10T19:30:40.077 INFO:teuthology.orchestra.run.smithi175.stderr:-- /var/log/ceph/ceph-client.admin.117531.log 2023-10-10T19:30:40.078 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-mon.a.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.96037.log.gz 2023-10-10T19:30:40.078 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117531.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.117531.log.gz 2023-10-10T19:30:40.078 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104397.log 2023-10-10T19:30:40.079 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114559.log 2023-10-10T19:30:40.079 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97601.log 2023-10-10T19:30:40.079 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114559.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.104397.log.gz 2023-10-10T19:30:40.080 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114559.log.gz 2023-10-10T19:30:40.080 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97601.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97601.log.gz 2023-10-10T19:30:40.080 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97769.log 2023-10-10T19:30:40.080 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107419.log 2023-10-10T19:30:40.081 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97769.log: /var/log/ceph/ceph-client.admin.107419.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97769.log.gz 2023-10-10T19:30:40.081 INFO:teuthology.orchestra.run.smithi175.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.107419.log.gz 2023-10-10T19:30:40.081 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105522.log 2023-10-10T19:30:40.081 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106890.log 2023-10-10T19:30:40.082 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105522.log: /var/log/ceph/ceph-client.admin.106890.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105522.log.gz 2023-10-10T19:30:40.082 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106890.log.gz 2023-10-10T19:30:40.082 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122646.log 2023-10-10T19:30:40.083 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110516.log 2023-10-10T19:30:40.083 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122646.log: /var/log/ceph/ceph-client.admin.110516.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.122646.log.gz 2023-10-10T19:30:40.083 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110516.log.gz 2023-10-10T19:30:40.083 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118950.log 2023-10-10T19:30:40.084 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110432.log 2023-10-10T19:30:40.084 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118950.log: /var/log/ceph/ceph-client.admin.110432.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118950.log.gz 2023-10-10T19:30:40.084 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110432.log.gz 2023-10-10T19:30:40.084 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131236.log 2023-10-10T19:30:40.085 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119388.log 2023-10-10T19:30:40.085 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131236.log: /var/log/ceph/ceph-client.admin.119388.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.131236.log.gz 2023-10-10T19:30:40.085 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119388.log.gz 2023-10-10T19:30:40.085 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126819.log 2023-10-10T19:30:40.086 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95936.log 2023-10-10T19:30:40.086 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126819.log: /var/log/ceph/ceph-client.admin.95936.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126819.log.gz 2023-10-10T19:30:40.086 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95936.log.gz 2023-10-10T19:30:40.086 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96919.log 2023-10-10T19:30:40.087 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111543.log 2023-10-10T19:30:40.087 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96919.log: /var/log/ceph/ceph-client.admin.111543.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96919.log.gz 2023-10-10T19:30:40.087 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111543.log.gz 2023-10-10T19:30:40.087 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99327.log 2023-10-10T19:30:40.088 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123377.log 2023-10-10T19:30:40.088 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99327.log: /var/log/ceph/ceph-client.admin.123377.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99327.log.gz 2023-10-10T19:30:40.088 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.123377.log.gz 2023-10-10T19:30:40.089 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98857.log 2023-10-10T19:30:40.089 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102459.log 2023-10-10T19:30:40.089 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.98857.log: /var/log/ceph/ceph-client.admin.102459.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98857.log.gz 2023-10-10T19:30:40.090 INFO:teuthology.orchestra.run.smithi175.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.102459.log.gz 2023-10-10T19:30:40.090 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124696.log 2023-10-10T19:30:40.090 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116216.log 2023-10-10T19:30:40.090 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123776.log 2023-10-10T19:30:40.091 INFO:teuthology.orchestra.run.smithi175.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.124696.log.gz/var/log/ceph/ceph-client.admin.116216.log: 2023-10-10T19:30:40.091 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123776.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116216.log.gz 2023-10-10T19:30:40.091 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.123776.log.gz 2023-10-10T19:30:40.092 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129167.log 2023-10-10T19:30:40.092 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125886.log 2023-10-10T19:30:40.092 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128495.log 2023-10-10T19:30:40.092 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125886.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129167.log.gz 2023-10-10T19:30:40.093 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125886.log.gz 2023-10-10T19:30:40.093 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128495.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.128495.log.gz 2023-10-10T19:30:40.093 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118799.log 2023-10-10T19:30:40.094 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113145.log 2023-10-10T19:30:40.094 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118799.log: /var/log/ceph/ceph-client.admin.113145.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.118799.log.gz 2023-10-10T19:30:40.094 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113145.log.gz 2023-10-10T19:30:40.094 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96244.log 2023-10-10T19:30:40.095 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100983.log 2023-10-10T19:30:40.095 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96244.log: /var/log/ceph/ceph-client.admin.100983.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96244.log.gz 2023-10-10T19:30:40.095 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100983.log.gz 2023-10-10T19:30:40.096 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112304.log 2023-10-10T19:30:40.096 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124503.log 2023-10-10T19:30:40.096 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112304.log: /var/log/ceph/ceph-client.admin.124503.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112304.log.gz 2023-10-10T19:30:40.096 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124503.log.gz 2023-10-10T19:30:40.097 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117446.log 2023-10-10T19:30:40.097 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115217.log 2023-10-10T19:30:40.097 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117986.log 2023-10-10T19:30:40.097 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115217.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117446.log.gz 2023-10-10T19:30:40.098 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117986.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115217.log.gz 2023-10-10T19:30:40.098 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127756.log 2023-10-10T19:30:40.098 INFO:teuthology.orchestra.run.smithi175.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.117986.log.gz 2023-10-10T19:30:40.099 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119244.log 2023-10-10T19:30:40.099 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127756.log: /var/log/ceph/ceph-client.admin.119244.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.127756.log.gz 2023-10-10T19:30:40.099 INFO:teuthology.orchestra.run.smithi175.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.119244.log.gz 2023-10-10T19:30:40.099 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106814.log 2023-10-10T19:30:40.100 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127400.log 2023-10-10T19:30:40.100 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.106814.log: /var/log/ceph/ceph-client.admin.127400.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106814.log.gz 2023-10-10T19:30:40.100 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.127400.log.gz 2023-10-10T19:30:40.101 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102863.log 2023-10-10T19:30:40.101 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129083.log 2023-10-10T19:30:40.101 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102863.log: /var/log/ceph/ceph-client.admin.129083.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102863.log.gz 2023-10-10T19:30:40.101 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.129083.log.gz 2023-10-10T19:30:40.102 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126367.log 2023-10-10T19:30:40.102 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101389.log 2023-10-10T19:30:40.102 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126367.log: /var/log/ceph/ceph-client.admin.101389.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126367.log.gz 2023-10-10T19:30:40.103 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117213.log 2023-10-10T19:30:40.103 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101389.log.gz 2023-10-10T19:30:40.103 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111669.log 2023-10-10T19:30:40.103 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117213.log: /var/log/ceph/ceph-client.admin.111669.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117213.log.gz 2023-10-10T19:30:40.104 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111669.log.gz 2023-10-10T19:30:40.104 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126191.log 2023-10-10T19:30:40.104 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125302.log 2023-10-10T19:30:40.105 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126191.log: /var/log/ceph/ceph-client.admin.125302.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.126191.log.gz 2023-10-10T19:30:40.105 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125302.log.gz 2023-10-10T19:30:40.105 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112867.log 2023-10-10T19:30:40.105 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115818.log 2023-10-10T19:30:40.106 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112867.log: /var/log/ceph/ceph-client.admin.115818.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112867.log.gz 2023-10-10T19:30:40.106 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115818.log.gz 2023-10-10T19:30:40.106 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99109.log 2023-10-10T19:30:40.107 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109024.log 2023-10-10T19:30:40.107 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.99109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129738.log 2023-10-10T19:30:40.107 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109024.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99109.log.gz 2023-10-10T19:30:40.108 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129738.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109024.log.gz 2023-10-10T19:30:40.108 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129738.log.gz 2023-10-10T19:30:40.108 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109713.log 2023-10-10T19:30:40.108 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121490.log 2023-10-10T19:30:40.109 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.98774.log 2023-10-10T19:30:40.109 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121490.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109713.log.gz 2023-10-10T19:30:40.109 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.0.98774.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121490.log.gz 2023-10-10T19:30:40.109 INFO:teuthology.orchestra.run.smithi175.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.98774.log.gz 2023-10-10T19:30:40.110 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96793.log 2023-10-10T19:30:40.110 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121357.log 2023-10-10T19:30:40.110 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96793.log: /var/log/ceph/ceph-client.admin.121357.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96793.log.gz 2023-10-10T19:30:40.111 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121357.log.gz 2023-10-10T19:30:40.111 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125970.log 2023-10-10T19:30:40.111 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125470.log 2023-10-10T19:30:40.111 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125970.log: /var/log/ceph/ceph-client.admin.125470.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.125970.log.gz 2023-10-10T19:30:40.112 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.125470.log.gz 2023-10-10T19:30:40.112 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96249.log 2023-10-10T19:30:40.112 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102604.log 2023-10-10T19:30:40.113 INFO:teuthology.orchestra.run.smithi175.stderr:gzip /var/log/ceph/ceph-client.admin.96249.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.103740.log 2023-10-10T19:30:40.113 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102604.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.96249.log.gz 2023-10-10T19:30:40.113 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102604.log.gz 2023-10-10T19:30:40.114 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103740.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103740.log.gz 2023-10-10T19:30:40.114 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112074.log 2023-10-10T19:30:40.114 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111222.log 2023-10-10T19:30:40.115 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112074.log: /var/log/ceph/ceph-client.admin.111222.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112074.log.gz 2023-10-10T19:30:40.115 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.111222.log.gz 2023-10-10T19:30:40.115 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97256.log 2023-10-10T19:30:40.116 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113229.log 2023-10-10T19:30:40.116 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97256.log: /var/log/ceph/ceph-client.admin.113229.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97256.log.gz 2023-10-10T19:30:40.116 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113229.log.gz 2023-10-10T19:30:40.117 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102989.log 2023-10-10T19:30:40.117 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132571.log 2023-10-10T19:30:40.117 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104857.log 2023-10-10T19:30:40.117 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132571.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.102989.log.gz 2023-10-10T19:30:40.118 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104857.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.132571.log.gz 2023-10-10T19:30:40.118 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104857.log.gz 2023-10-10T19:30:40.118 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121804.log 2023-10-10T19:30:40.119 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109066.log 2023-10-10T19:30:40.119 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102646.log 2023-10-10T19:30:40.119 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109066.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.121804.log.gz 2023-10-10T19:30:40.120 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102646.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109066.log.gz 2023-10-10T19:30:40.120 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102646.log.gz 2023-10-10T19:30:40.120 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130427.log 2023-10-10T19:30:40.120 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122814.log 2023-10-10T19:30:40.121 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130427.log: /var/log/ceph/ceph-client.admin.122814.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.130427.log.gz 2023-10-10T19:30:40.121 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.122814.log.gz 2023-10-10T19:30:40.121 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125844.log 2023-10-10T19:30:40.122 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97559.log 2023-10-10T19:30:40.122 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125844.log: /var/log/ceph/ceph-client.admin.97559.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125844.log.gz 2023-10-10T19:30:40.122 INFO:teuthology.orchestra.run.smithi175.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97559.log.gz 2023-10-10T19:30:40.122 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101347.log 2023-10-10T19:30:40.123 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124234.log 2023-10-10T19:30:40.123 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.101347.log: /var/log/ceph/ceph-client.admin.124234.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101347.log.gz 2023-10-10T19:30:40.123 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.124234.log.gz 2023-10-10T19:30:40.124 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113624.log 2023-10-10T19:30:40.124 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105644.log 2023-10-10T19:30:40.124 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113624.log: /var/log/ceph/ceph-client.admin.105644.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.113624.log.gz 2023-10-10T19:30:40.124 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96590.log 2023-10-10T19:30:40.125 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105644.log.gz 2023-10-10T19:30:40.125 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-10T19:30:40.126 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96590.log: /var/log/ceph/ceph-osd.0.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96590.log.gz 2023-10-10T19:30:40.126 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97938.log 2023-10-10T19:30:40.126 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112430.log 2023-10-10T19:30:40.127 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97938.log: /var/log/ceph/ceph-client.admin.112430.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97938.log.gz 2023-10-10T19:30:40.127 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112430.log.gz 2023-10-10T19:30:40.127 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102688.log 2023-10-10T19:30:40.127 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109797.log 2023-10-10T19:30:40.128 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102688.log: /var/log/ceph/ceph-client.admin.109797.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102688.log.gz 2023-10-10T19:30:40.128 INFO:teuthology.orchestra.run.smithi175.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109797.log.gz 2023-10-10T19:30:40.128 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124444.log 2023-10-10T19:30:40.129 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117860.log 2023-10-10T19:30:40.129 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124444.log: /var/log/ceph/ceph-client.admin.117860.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124444.log.gz 2023-10-10T19:30:40.129 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.117860.log.gz 2023-10-10T19:30:40.129 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112032.log 2023-10-10T19:30:40.154 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110222.log 2023-10-10T19:30:40.154 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112032.log: /var/log/ceph/ceph-client.admin.110222.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112032.log.gz 2023-10-10T19:30:40.154 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110222.log.gz 2023-10-10T19:30:40.155 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115499.log 2023-10-10T19:30:40.155 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90984.log 2023-10-10T19:30:40.155 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115499.log: /var/log/ceph/ceph-client.admin.90984.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115499.log.gz 2023-10-10T19:30:40.156 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90984.log.gz 2023-10-10T19:30:40.156 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108308.log 2023-10-10T19:30:40.156 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117902.log 2023-10-10T19:30:40.156 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108308.log: /var/log/ceph/ceph-client.admin.117902.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108308.log.gz 2023-10-10T19:30:40.157 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117902.log.gz 2023-10-10T19:30:40.157 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128369.log 2023-10-10T19:30:40.157 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103132.log 2023-10-10T19:30:40.158 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128369.log: /var/log/ceph/ceph-client.admin.103132.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.128369.log.gz 2023-10-10T19:30:40.158 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103132.log.gz 2023-10-10T19:30:40.158 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129226.log 2023-10-10T19:30:40.159 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129927.log 2023-10-10T19:30:40.159 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129226.log: /var/log/ceph/ceph-client.admin.129927.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.129226.log.gz 2023-10-10T19:30:40.159 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129927.log.gz 2023-10-10T19:30:40.159 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118992.log 2023-10-10T19:30:40.160 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96586.log 2023-10-10T19:30:40.160 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118992.log: /var/log/ceph/ceph-client.admin.96586.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118992.log.gz 2023-10-10T19:30:40.160 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96586.log.gz 2023-10-10T19:30:40.161 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103031.log 2023-10-10T19:30:40.161 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129310.log 2023-10-10T19:30:40.161 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103031.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.103031.log.gz 2023-10-10T19:30:40.161 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129310.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129310.log.gz 2023-10-10T19:30:40.162 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122140.log 2023-10-10T19:30:40.162 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116713.log 2023-10-10T19:30:40.162 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122140.log: /var/log/ceph/ceph-client.admin.116713.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122140.log.gz 2023-10-10T19:30:40.163 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116713.log.gz 2023-10-10T19:30:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118238.log 2023-10-10T19:30:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96253.log 2023-10-10T19:30:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118238.log: /var/log/ceph/ceph-client.admin.96253.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118238.log.gz 2023-10-10T19:30:40.164 INFO:teuthology.orchestra.run.smithi175.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96253.log.gz 2023-10-10T19:30:40.164 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122772.log 2023-10-10T19:30:40.164 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125928.log 2023-10-10T19:30:40.164 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122772.log: /var/log/ceph/ceph-client.admin.125928.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.122772.log.gz 2023-10-10T19:30:40.165 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125928.log.gz 2023-10-10T19:30:40.165 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126995.log 2023-10-10T19:30:40.165 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112808.log 2023-10-10T19:30:40.166 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126995.log: /var/log/ceph/ceph-client.admin.112808.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.126995.log.gz 2023-10-10T19:30:40.166 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112808.log.gz 2023-10-10T19:30:40.166 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109953.log 2023-10-10T19:30:40.166 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121720.log 2023-10-10T19:30:40.167 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109953.log: /var/log/ceph/ceph-client.admin.121720.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109953.log.gz 2023-10-10T19:30:40.167 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121720.log.gz 2023-10-10T19:30:40.167 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124066.log 2023-10-10T19:30:40.167 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107369.log 2023-10-10T19:30:40.168 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124066.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124066.log.gz 2023-10-10T19:30:40.168 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107369.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107369.log.gz 2023-10-10T19:30:40.168 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100722.log 2023-10-10T19:30:40.169 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129436.log 2023-10-10T19:30:40.169 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.100722.log: /var/log/ceph/ceph-client.admin.129436.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100722.log.gz 2023-10-10T19:30:40.169 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129436.log.gz 2023-10-10T19:30:40.169 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125743.log 2023-10-10T19:30:40.170 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108703.log 2023-10-10T19:30:40.170 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125743.log: /var/log/ceph/ceph-client.admin.108703.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125743.log.gz 2023-10-10T19:30:40.170 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108703.log.gz 2023-10-10T19:30:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104355.log 2023-10-10T19:30:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102730.log 2023-10-10T19:30:40.171 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104355.log: /var/log/ceph/ceph-client.admin.102730.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104355.log.gz 2023-10-10T19:30:40.171 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102730.log.gz 2023-10-10T19:30:40.172 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126317.log 2023-10-10T19:30:40.172 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105158.log 2023-10-10T19:30:40.172 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126317.log: /var/log/ceph/ceph-client.admin.105158.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126317.log.gz 2023-10-10T19:30:40.173 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105158.log.gz 2023-10-10T19:30:40.173 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126903.log 2023-10-10T19:30:40.173 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96926.log 2023-10-10T19:30:40.173 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126903.log: /var/log/ceph/ceph-client.admin.96926.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.126903.log.gz 2023-10-10T19:30:40.174 INFO:teuthology.orchestra.run.smithi175.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96926.log.gz 2023-10-10T19:30:40.174 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124906.log 2023-10-10T19:30:40.174 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107546.log 2023-10-10T19:30:40.174 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124906.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124906.log.gz 2023-10-10T19:30:40.175 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107546.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.107546.log.gz 2023-10-10T19:30:40.175 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113397.log 2023-10-10T19:30:40.175 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111012.log 2023-10-10T19:30:40.176 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113397.log: /var/log/ceph/ceph-client.admin.111012.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113397.log.gz 2023-10-10T19:30:40.176 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111012.log.gz 2023-10-10T19:30:40.176 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121189.log 2023-10-10T19:30:40.176 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123175.log 2023-10-10T19:30:40.177 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121189.log: /var/log/ceph/ceph-client.admin.123175.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121189.log.gz 2023-10-10T19:30:40.177 INFO:teuthology.orchestra.run.smithi175.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.123175.log.gz 2023-10-10T19:30:40.177 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114468.log 2023-10-10T19:30:40.178 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125049.log 2023-10-10T19:30:40.178 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114468.log: /var/log/ceph/ceph-client.admin.125049.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114468.log.gz 2023-10-10T19:30:40.178 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125049.log.gz 2023-10-10T19:30:40.178 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128184.log 2023-10-10T19:30:40.179 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105564.log 2023-10-10T19:30:40.179 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128184.log: /var/log/ceph/ceph-client.admin.105564.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.128184.log.gz 2023-10-10T19:30:40.179 INFO:teuthology.orchestra.run.smithi175.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105564.log.gz 2023-10-10T19:30:40.180 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127714.log 2023-10-10T19:30:40.180 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116923.log 2023-10-10T19:30:40.180 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127714.log: /var/log/ceph/ceph-client.admin.116923.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127714.log.gz 2023-10-10T19:30:40.180 INFO:teuthology.orchestra.run.smithi175.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116923.log.gz 2023-10-10T19:30:40.181 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126468.log 2023-10-10T19:30:40.181 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-10T19:30:40.181 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126468.log: /var/log/ceph/ceph-mgr.y.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.126468.log.gz 2023-10-10T19:30:40.182 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97727.log 2023-10-10T19:30:40.182 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97727.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.97727.log.gz 2023-10-10T19:30:40.182 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104899.log 2023-10-10T19:30:40.182 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104899.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.104899.log.gz 2023-10-10T19:30:40.183 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116671.log 2023-10-10T19:30:40.183 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116671.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116671.log.gz 2023-10-10T19:30:40.183 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123629.log 2023-10-10T19:30:40.184 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123629.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123629.log.gz 2023-10-10T19:30:40.184 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131110.log 2023-10-10T19:30:40.184 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131110.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.131110.log.gz 2023-10-10T19:30:40.184 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130207.log 2023-10-10T19:30:40.185 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130207.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130207.log.gz 2023-10-10T19:30:40.185 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111054.log 2023-10-10T19:30:40.185 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.111054.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111054.log.gz 2023-10-10T19:30:40.186 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104481.log 2023-10-10T19:30:40.186 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104481.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104481.log.gz 2023-10-10T19:30:40.186 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106932.log 2023-10-10T19:30:40.186 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.106932.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106932.log.gz 2023-10-10T19:30:40.187 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132052.log 2023-10-10T19:30:40.187 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132052.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.132052.log.gz 2023-10-10T19:30:40.187 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103908.log 2023-10-10T19:30:40.188 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103908.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103908.log.gz 2023-10-10T19:30:40.188 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129352.log 2023-10-10T19:30:40.188 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.129352.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.129352.log.gz 2023-10-10T19:30:40.188 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125659.log 2023-10-10T19:30:40.189 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125659.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125659.log.gz 2023-10-10T19:30:40.189 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108088.log 2023-10-10T19:30:40.189 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108088.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108088.log.gz 2023-10-10T19:30:40.190 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121399.log 2023-10-10T19:30:40.190 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121399.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.121399.log.gz 2023-10-10T19:30:40.190 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113876.log 2023-10-10T19:30:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113876.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113876.log.gz 2023-10-10T19:30:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107142.log 2023-10-10T19:30:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107142.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107142.log.gz 2023-10-10T19:30:40.191 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96589.log 2023-10-10T19:30:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96589.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96589.log.gz 2023-10-10T19:30:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104229.log 2023-10-10T19:30:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104229.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104229.log.gz 2023-10-10T19:30:40.192 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132388.log 2023-10-10T19:30:40.193 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.132388.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.132388.log.gz 2023-10-10T19:30:40.193 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126012.log 2023-10-10T19:30:40.193 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126012.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126012.log.gz 2023-10-10T19:30:40.194 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127568.log 2023-10-10T19:30:40.194 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127568.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127568.log.gz 2023-10-10T19:30:40.194 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103224.log 2023-10-10T19:30:40.194 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103224.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.103224.log.gz 2023-10-10T19:30:40.195 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108130.log 2023-10-10T19:30:40.195 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108130.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108130.log.gz 2023-10-10T19:30:40.195 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128411.log 2023-10-10T19:30:40.196 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128411.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128411.log.gz 2023-10-10T19:30:40.196 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123982.log 2023-10-10T19:30:40.196 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123982.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.123982.log.gz 2023-10-10T19:30:40.196 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124629.log 2023-10-10T19:30:40.197 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.124629.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.124629.log.gz 2023-10-10T19:30:40.197 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127484.log 2023-10-10T19:30:40.197 INFO:teuthology.orchestra.run.smithi175.stderr: 95.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-10T19:30:40.197 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127484.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127484.log.gz 2023-10-10T19:30:40.198 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105917.log 2023-10-10T19:30:40.198 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107769.log 2023-10-10T19:30:40.198 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105917.log: /var/log/ceph/ceph-client.admin.107769.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105917.log.gz 2023-10-10T19:30:40.199 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107769.log.gz 2023-10-10T19:30:40.199 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117699.log 2023-10-10T19:30:40.199 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121231.log 2023-10-10T19:30:40.199 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.117699.log: /var/log/ceph/ceph-client.admin.121231.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.117699.log.gz 2023-10-10T19:30:40.200 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.121231.log.gz 2023-10-10T19:30:40.200 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116073.log 2023-10-10T19:30:40.200 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128453.log 2023-10-10T19:30:40.201 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.116073.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116073.log.gz 2023-10-10T19:30:40.201 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122982.log 2023-10-10T19:30:40.201 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128453.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.128453.log.gz 2023-10-10T19:30:40.201 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107870.log 2023-10-10T19:30:40.202 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.122982.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.122982.log.gz 2023-10-10T19:30:40.202 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121619.log 2023-10-10T19:30:40.202 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107870.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107870.log.gz 2023-10-10T19:30:40.203 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127966.log 2023-10-10T19:30:40.203 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.121619.log: 88.4%/var/log/ceph/ceph-client.admin.127966.log: -- replaced with /var/log/ceph/ceph-client.admin.121619.log.gz 2023-10-10T19:30:40.203 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127966.log.gz 2023-10-10T19:30:40.203 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128285.log 2023-10-10T19:30:40.204 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121846.log 2023-10-10T19:30:40.204 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128285.log: /var/log/ceph/ceph-client.admin.121846.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.128285.log.gz 2023-10-10T19:30:40.204 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121846.log.gz 2023-10-10T19:30:40.204 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103782.log 2023-10-10T19:30:40.205 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98815.log 2023-10-10T19:30:40.205 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103782.log: /var/log/ceph/ceph-client.admin.98815.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.103782.log.gz 2023-10-10T19:30:40.205 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98815.log.gz 2023-10-10T19:30:40.206 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96247.log 2023-10-10T19:30:40.206 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104058.log 2023-10-10T19:30:40.206 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96247.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96247.log.gz 2023-10-10T19:30:40.207 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119202.log 2023-10-10T19:30:40.207 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104058.log: /var/log/ceph/ceph-client.admin.119202.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104058.log.gz 2023-10-10T19:30:40.207 INFO:teuthology.orchestra.run.smithi175.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119202.log.gz 2023-10-10T19:30:40.207 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97895.log 2023-10-10T19:30:40.208 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123503.log 2023-10-10T19:30:40.208 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.97895.log: /var/log/ceph/ceph-client.admin.123503.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97895.log.gz 2023-10-10T19:30:40.208 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123503.log.gz 2023-10-10T19:30:40.208 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105067.log 2023-10-10T19:30:40.209 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96251.log 2023-10-10T19:30:40.209 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105067.log: /var/log/ceph/ceph-client.admin.96251.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.105067.log.gz 2023-10-10T19:30:40.209 INFO:teuthology.orchestra.run.smithi175.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96251.log.gz 2023-10-10T19:30:40.210 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104439.log 2023-10-10T19:30:40.210 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125133.log 2023-10-10T19:30:40.210 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.104439.log: /var/log/ceph/ceph-client.admin.125133.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104439.log.gz 2023-10-10T19:30:40.210 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125133.log.gz 2023-10-10T19:30:40.211 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127526.log 2023-10-10T19:30:40.211 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108856.log 2023-10-10T19:30:40.211 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127526.log: /var/log/ceph/ceph-client.admin.108856.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.127526.log.gz 2023-10-10T19:30:40.212 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108856.log.gz 2023-10-10T19:30:40.212 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123419.log 2023-10-10T19:30:40.212 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111585.log 2023-10-10T19:30:40.213 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.123419.log: /var/log/ceph/ceph-client.admin.111585.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.123419.log.gz 2023-10-10T19:30:40.213 INFO:teuthology.orchestra.run.smithi175.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.111585.log.gz 2023-10-10T19:30:40.213 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115262.log 2023-10-10T19:30:40.213 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95719.log 2023-10-10T19:30:40.214 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115262.log: /var/log/ceph/ceph-client.admin.95719.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115262.log.gz 2023-10-10T19:30:40.214 INFO:teuthology.orchestra.run.smithi175.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95719.log.gz 2023-10-10T19:30:40.214 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115007.log 2023-10-10T19:30:40.215 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113960.log 2023-10-10T19:30:40.215 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.115007.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115007.log.gz 2023-10-10T19:30:40.215 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.113960.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113960.log.gz 2023-10-10T19:30:40.215 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107268.log 2023-10-10T19:30:40.216 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108982.log 2023-10-10T19:30:40.216 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107268.log: /var/log/ceph/ceph-client.admin.108982.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107268.log.gz 2023-10-10T19:30:40.216 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108982.log.gz 2023-10-10T19:30:40.217 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114856.log 2023-10-10T19:30:40.217 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132180.log 2023-10-10T19:30:40.217 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.114856.log: /var/log/ceph/ceph-client.admin.132180.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114856.log.gz 2023-10-10T19:30:40.217 INFO:teuthology.orchestra.run.smithi175.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.132180.log.gz 2023-10-10T19:30:40.218 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108940.log 2023-10-10T19:30:40.218 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103866.log 2023-10-10T19:30:40.218 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.108940.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108940.log.gz 2023-10-10T19:30:40.219 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124738.log 2023-10-10T19:30:40.219 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.103866.log: /var/log/ceph/ceph-client.admin.124738.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.103866.log.gz 2023-10-10T19:30:40.219 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124738.log.gz 2023-10-10T19:30:40.219 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109234.log 2023-10-10T19:30:40.220 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118631.log 2023-10-10T19:30:40.220 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109234.log: /var/log/ceph/ceph-client.admin.118631.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109234.log.gz 2023-10-10T19:30:40.220 INFO:teuthology.orchestra.run.smithi175.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118631.log.gz 2023-10-10T19:30:40.220 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.98766.log 2023-10-10T19:30:40.221 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130469.log 2023-10-10T19:30:40.221 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.0.98766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.98766.log.gz 2023-10-10T19:30:40.221 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.130469.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.130469.log.gz 2023-10-10T19:30:40.222 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112598.log 2023-10-10T19:30:40.222 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94956.log 2023-10-10T19:30:40.222 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.112598.log: /var/log/ceph/ceph-client.admin.94956.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112598.log.gz 2023-10-10T19:30:40.222 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94956.log.gz 2023-10-10T19:30:40.223 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-10T19:30:40.223 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110709.log 2023-10-10T19:30:40.223 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-mds.e.log: /var/log/ceph/ceph-client.admin.110709.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110709.log.gz 2023-10-10T19:30:40.223 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127358.log 2023-10-10T19:30:40.224 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.127358.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.127358.log.gz 2023-10-10T19:30:40.224 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126275.log 2023-10-10T19:30:40.224 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.126275.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.126275.log.gz 2023-10-10T19:30:40.225 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125217.log 2023-10-10T19:30:40.225 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.125217.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125217.log.gz 2023-10-10T19:30:40.225 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109503.log 2023-10-10T19:30:40.225 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109503.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109503.log.gz 2023-10-10T19:30:40.226 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128008.log 2023-10-10T19:30:40.226 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128008.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128008.log.gz 2023-10-10T19:30:40.226 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107310.log 2023-10-10T19:30:40.227 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107310.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107310.log.gz 2023-10-10T19:30:40.227 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-10T19:30:40.227 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102947.log 2023-10-10T19:30:40.227 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.102947.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102947.log.gz 2023-10-10T19:30:40.228 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105116.log 2023-10-10T19:30:40.228 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.105116.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105116.log.gz 2023-10-10T19:30:40.228 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128663.log 2023-10-10T19:30:40.229 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.128663.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.128663.log.gz 2023-10-10T19:30:40.229 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109755.log 2023-10-10T19:30:40.229 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109755.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.109755.log.gz 2023-10-10T19:30:40.229 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118196.log 2023-10-10T19:30:40.230 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.118196.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.118196.log.gz 2023-10-10T19:30:40.230 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131404.log 2023-10-10T19:30:40.230 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.131404.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131404.log.gz 2023-10-10T19:30:40.231 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96243.log 2023-10-10T19:30:40.231 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.96243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109276.log 2023-10-10T19:30:40.231 INFO:teuthology.orchestra.run.smithi175.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96243.log.gz 2023-10-10T19:30:40.231 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.109276.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109276.log.gz 2023-10-10T19:30:40.232 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107828.log 2023-10-10T19:30:40.232 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.107828.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107828.log.gz 2023-10-10T19:30:40.342 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mon.b.log: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-10T19:30:40.688 INFO:teuthology.orchestra.run.smithi191.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-10T19:30:41.127 INFO:teuthology.orchestra.run.smithi191.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-10T19:30:42.201 INFO:teuthology.orchestra.run.smithi175.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-10T19:30:47.589 INFO:teuthology.orchestra.run.smithi175.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-10T19:30:48.265 INFO:teuthology.orchestra.run.smithi175.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-10T19:30:51.273 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-10T19:30:51.405 INFO:teuthology.orchestra.run.smithi175.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-10T19:30:51.546 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-10T19:30:51.891 INFO:teuthology.orchestra.run.smithi191.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-10T19:30:51.936 INFO:teuthology.orchestra.run.smithi175.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-10T19:30:54.417 INFO:teuthology.orchestra.run.smithi175.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-10T19:30:54.521 INFO:teuthology.orchestra.run.smithi191.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-10T19:31:00.461 INFO:teuthology.orchestra.run.smithi191.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-10T19:31:00.522 INFO:teuthology.orchestra.run.smithi175.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-10T19:31:00.609 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-10T19:31:00.897 INFO:teuthology.orchestra.run.smithi175.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-10T19:31:00.898 INFO:teuthology.orchestra.run.smithi175.stderr: 2023-10-10T19:31:00.899 INFO:teuthology.orchestra.run.smithi175.stderr:real 0m21.306s 2023-10-10T19:31:00.899 INFO:teuthology.orchestra.run.smithi175.stderr:user 1m36.346s 2023-10-10T19:31:00.900 INFO:teuthology.orchestra.run.smithi175.stderr:sys 0m3.331s 2023-10-10T19:31:06.010 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-10T19:31:06.011 INFO:teuthology.orchestra.run.smithi191.stderr: 2023-10-10T19:31:06.011 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m26.394s 2023-10-10T19:31:06.012 INFO:teuthology.orchestra.run.smithi191.stderr:user 1m55.829s 2023-10-10T19:31:06.012 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m3.504s 2023-10-10T19:31:06.013 INFO:tasks.ceph:Archiving logs... 2023-10-10T19:31:06.013 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/remote/smithi175/log 2023-10-10T19:31:06.015 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-10T19:31:18.787 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/remote/smithi191/log 2023-10-10T19:31:18.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-10T19:31:33.211 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-10T19:31:33.221 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-10T19:31:33.222 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-10T19:31:33.224 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-10T19:31:33.303 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-10T19:31:33.316 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-10T19:31:33.317 DEBUG:teuthology.orchestra.run.smithi175:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T19:31:33.320 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T19:31:33.332 INFO:teuthology.orchestra.run.smithi175.stderr:bash: ntpq: command not found 2023-10-10T19:31:33.334 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-10-10T19:31:33.336 INFO:teuthology.orchestra.run.smithi175.stdout:210 Number of sources = 4 2023-10-10T19:31:33.336 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T19:31:33.336 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2023-10-10T19:31:33.337 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 41 -449us[ -449us] +/- 92ms 2023-10-10T19:31:33.337 INFO:teuthology.orchestra.run.smithi175.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 +137us[ +136us] +/- 50ms 2023-10-10T19:31:33.337 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +186us[ +186us] +/- 76ms 2023-10-10T19:31:33.337 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T19:31:33.339 INFO:teuthology.orchestra.run.smithi191.stdout:210 Number of sources = 4 2023-10-10T19:31:33.339 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T19:31:33.340 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-10-10T19:31:33.340 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 106 -493us[ -453us] +/- 91ms 2023-10-10T19:31:33.340 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 7 377 45 +178us[ +218us] +/- 50ms 2023-10-10T19:31:33.341 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +184us[ +184us] +/- 76ms 2023-10-10T19:31:33.341 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T19:31:33.341 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-10T19:31:33.352 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-10T19:31:33.352 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-10T19:31:33.361 DEBUG:teuthology.orchestra.run.smithi175:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-10T19:31:33.485 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-10T19:31:33.577 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T19:31:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T19:31:33.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T19:31:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T19:31:33.638 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-10T19:31:33.648 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-10T19:31:33.658 INFO:teuthology.task.internal:Duration was 3446.020195 seconds 2023-10-10T19:31:33.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-10T19:31:33.669 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-10T19:31:33.669 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-10T19:31:33.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-10T19:31:33.713 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T19:31:33.721 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T19:31:33.954 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-10T19:31:33.955 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi175.front.sepia.ceph.com 2023-10-10T19:31:33.955 DEBUG:teuthology.orchestra.run.smithi175:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-10T19:31:34.013 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2023-10-10T19:31:34.013 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-10T19:31:34.032 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-10T19:31:34.033 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-10T19:31:34.055 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-10T19:31:34.110 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-10T19:31:34.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-10T19:31:34.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-10T19:31:34.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-10T19:31:34.249 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-10T19:31:34.250 DEBUG:teuthology.orchestra.run.smithi175:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-10T19:31:34.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-10T19:31:34.299 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-10T19:31:34.309 DEBUG:teuthology.orchestra.run.smithi175:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-10T19:31:34.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-10T19:31:34.342 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = core 2023-10-10T19:31:34.364 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2023-10-10T19:31:34.377 DEBUG:teuthology.orchestra.run.smithi175:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-10T19:31:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T19:31:34.406 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-10T19:31:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T19:31:34.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-10T19:31:34.442 INFO:teuthology.task.internal:Transferring archived files... 2023-10-10T19:31:34.444 DEBUG:teuthology.misc:Transferring archived files from smithi175:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/remote/smithi175 2023-10-10T19:31:34.444 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-10T19:31:34.484 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370/remote/smithi191 2023-10-10T19:31:34.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-10T19:31:34.520 INFO:teuthology.task.internal:Removing archive directory... 2023-10-10T19:31:34.521 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-10T19:31:34.523 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-10T19:31:34.568 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-10T19:31:34.581 INFO:teuthology.task.internal:Not uploading archives. 2023-10-10T19:31:34.582 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-10T19:31:34.593 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-10T19:31:34.593 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-10T19:31:34.597 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-10T19:31:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: 262891 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 10 19:31 /home/ubuntu/cephtest 2023-10-10T19:31:34.621 INFO:teuthology.orchestra.run.smithi191.stdout: 262149 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 10 19:31 /home/ubuntu/cephtest 2023-10-10T19:31:34.623 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-10T19:31:34.633 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-10T19:31:34.655 INFO:teuthology.nuke:Checking targets against current locks 2023-10-10T19:31:34.684 DEBUG:teuthology.nuke:shortname: smithi175 2023-10-10T19:31:34.685 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-10T19:31:34.708 DEBUG:teuthology.nuke:shortname: smithi191 2023-10-10T19:31:34.709 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-10T19:31:34.712 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 18:26:52.470722', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcuCFH2aQPp6AqsxooAWa9L+2D7lkIlT3KYhmELk9TKiw8Yutr9DYrlPL+/hGDwU2wBsKtoqN/WB7RivMb1/j0='} 2023-10-10T19:31:34.732 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-10_18:16:52-fs-wip-yuri3-testing-2023-10-10-0720-quincy-distro-default-smithi/7420370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 18:26:52.469006', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKAmsrXlRHhVO40XzsNzCCvWbKIzgovODvyFDIx1r5u6lgIzDMu/D40byN68X9lvtywNnIV6lA/+vo2O0KVoaWw='} 2023-10-10T19:31:34.760 INFO:teuthology.orchestra.console.smithi175:Power off 2023-10-10T19:31:34.761 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-10T19:31:34.791 DEBUG:teuthology.orchestra.console.smithi175:power off output: Chassis Power Control: Down/Off 2023-10-10T19:31:34.792 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T19:31:34.813 INFO:teuthology.orchestra.console.smithi191:Power off 2023-10-10T19:31:34.814 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-10T19:31:34.835 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2023-10-10T19:31:34.841 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2023-10-10T19:31:34.841 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T19:31:34.867 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-10-10T19:31:38.837 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T19:31:38.868 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T19:31:38.950 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2023-10-10T19:31:38.979 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-10-10T19:31:42.954 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T19:31:42.981 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T19:31:43.072 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is off 2023-10-10T19:31:43.093 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2023-10-10T19:31:43.173 INFO:teuthology.orchestra.console.smithi175:Power off completed 2023-10-10T19:31:43.193 INFO:teuthology.orchestra.console.smithi191:Power off completed 2023-10-10T19:31:43.301 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 3446.0201947689056 failure_reason: 'Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi175 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d1f59b549730ba7f93d940a3d32b94c45a4cf7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=85ea02d3d92f4d5d92f33221cbd2708d status: fail success: false 2023-10-10T19:31:43.302 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-10T19:31:43.396 INFO:teuthology.run:FAIL